How often is the AI right?

The problem

An eval run before launch tells you how a feature did on last month's data. It doesn't tell you how it's doing today, on the emails that came in this morning. And a small business can't afford someone whose job is to check.

So the question I wanted the app to answer by itself was: when it suggested something, did the person keep it?

Three ways to grade

Every suggestion the app makes ends up as one row in one table. What decides whether it was right depends on the feature:

The table is filled only by database triggers on tables the app already writes. Nobody has to remember to log feedback from a new screen, and the same rules backfilled the history on day one.

-- No app code writes feedback. Postgres does, from rows the app
-- already writes — so a new screen can't forget to report.
create trigger feedback_from_bill_match
after update of status on bill_job_matches
for each row
when (old.status = 'suggested' and new.status <> old.status)
execute function record_ai_feedback('bills.job_match');

Each row also carries the model and the git commit of the prompt that made it, so a prompt change shows up as a before-and-after bar, not a feeling.

Every miss is a test case

Opening a feature lists its misses, each with what the app suggested next to what the person ended up with, as a line diff. Someone sorts the miss into one of five reasons — and can tick"make it a test case", which adds it to the export the offline evals run against.

That closes the loop: production misses become the eval set, the eval set gates the next prompt, and the scorecard shows whether it helped. There's also a house rule now — a feature with a person in the loop isn't done until it's on the scorecard.

The one that can't be wrong

Most misses cost a click. One doesn't: reading a client's reply to "can we go ahead?" as a yes when it wasn't. That books a crew. So the scorecard counts a false yes as a critical miss of its own, and the feature is built so the model alone can't produce one.

// The model reads the client's reply. Then plain rules get a veto —
// they can only make the answer less sure, never more.
function decideApproval(ai, reply) {
  const core = stripQuotedThread(reply);

  const hedge = core.match(HEDGES);          // "if", "?", "how much", …
  if (hedge) return unclear(`The reply has "${hedge[0]}" in it`);

  if (ai.answer === "yes") {
    if (!YES_WORDS.test(core)) return unclear("No yes word in the reply");
    const no = core.match(NO_WORDS);
    if (no) return unclear(`The reply also says "${no[0]}"`);
    if (mentionsOtherDuration(core, ai)) return unclear("Different timeframe");
  }
  return ai;                                 // silence is never a yes
}

Twelve misses that cost a person a second look is a fair price. One wrong booking isn't.

Honest numbers

The point of a scorecard is that it shows the weak spots too. From the evals so far:

• 98%
  of tonnes and amounts read right across 853 tip dockets (backlog run through the Batch API at half price)
• 78 / 78
  fields read right from bills, permits and title searches
• 13 / 13
  email-to-job guesses right when it says it's sure — which is 68% of emails
• 47%
  client update drafts kept as written. Not good enough yet, and the scorecard says so

The scorecard runs on staging today, and real users haven't started on these features yet. The next step depends on it: once an action stays unchanged nine times in ten, it's allowed to stop asking. That's what thejob assistantis waiting on.

Simplified sketches for this page — names and details changed, none of the original code.