The model passed the demo. Did the workflow?

An assistant writes a perfectly polite reply to a customer and gets the order number wrong. On tone, clarity, and relevance, the response looks good. From the customer’s point of view, the system has failed.

Maybe the order number is correct, but an employee spends seven minutes checking a reply that saved five minutes of writing. We have passed a different test and still made the work slower. The demo won’t show either problem unless we follow the request beyond the model response.

The useful shift in AI evaluation is to treat the completed piece of work as the object being tested, including the data that entered the system, the tools it called, the person who reviewed it, the message that was sent, and the record that changed afterwards. A good response is part of that path (sometimes the most uncertain part), but it isn’t the outcome.

Follow one request all the way through

For a support request, the path may begin with a customer message, continue through an account and order lookup, ask a model to draft the answer, pause for review, send the reply, and update the customer record. If we only save the prompt and response, we can’t tell whether the lookup found the right customer, whether the reviewer silently rewrote the draft, or whether the final record reflects what the customer was told (which is what you will need to know when investigating a complaint).

We’d describe success in business language before choosing a grader. “The answer is relevant” is too loose. “The customer receives the correct return instructions, no unsupported promise is made, and the request moves to the right queue” gives us several things we can actually inspect.

Moreover, this exposes an architectural question. If a predictable route with one model call produces the result, an open-ended agent has added failure modes without adding much value. Anthropic makes a similar case for increasing complexity only when it improves the outcome. The evaluation shouldn’t assume the architecture is fixed; sometimes the most useful result is evidence that the system can be simpler.

Begin with cases people recognize

You probably don’t need a hundred polished prompts to learn something. A small first set can include the work people handle every day, a few awkward inputs with missing or contradictory information, and the cases where a bad action would matter most. The examples should retain the untidy shape of real work (with sensitive details removed), because misspellings, duplicate records, and incomplete notes are part of the environment the system will meet.

Anthropic’s research team began evaluating its multi-agent research system with roughly twenty queries that represented real usage, then expanded the suite as new behavior appeared. Its account of that process is useful because the test set wasn’t treated as a finished benchmark. It became a memory of what the team had learned.

Nevertheless, we’d do the same with production failures. When a user finds a new problem, a redacted version joins the test set before the fix is made. Otherwise it’s very easy to repair one visible example and lose the lesson a month later.

The suite therefore grows with the system because it can’t anticipate every real request on day one.

Let the question choose the grader

Moreover, some checks don’t need another model. For example, code can verify that the correct record changed, that required fields are present, that a tool was allowed, and that a retry didn’t send the same message twice. These checks are cheap, consistent, and close to the final state, so handing them to a language model would make them less reliable for no good reason.

Other judgments are harder to express as exact rules. A model grader can help assess whether a summary is faithful or whether an explanation covers an agreed rubric, particularly when the volume is too large for people to read every example. However, we’d first compare that grader with domain experts, inspect disagreements, and keep some human review in the loop. A beautifully consistent grader can still be consistently wrong about what the business values.

Furthermore, people are especially important when consequence and context matter. They notice that an answer is technically correct but awkward for the customer, or that a five-minute review estimate ignores the time spent opening three systems. NIST’s AI Risk Management Framework playbook treats measurement in this broader way, bringing performance, risk, impact, and monitoring into the same decision.

Evaluation should change what the system may do

In practice, a drafting assistant that occasionally produces a weak sentence can probably be sampled and corrected. A system that sends a refund, changes a customer record, or writes to a regulated file needs a much harder boundary, especially when the action can’t be reversed or nobody will notice the mistake before it matters.

Consequently, one accuracy number can’t make a release decision. We also need to see how much review remains, whether people use the workflow, how long the whole task takes, what each successful outcome costs, and which failures are unacceptable. A slightly less accurate system may perhaps be safer if it knows when to stop and routes uncertain work cleanly to a person.

The practical output of evaluation is therefore a boundary: what the system may do today, what still needs review, and what evidence would justify giving it more room later. Every important failure that moves this boundary should remain in the test set.

Nevertheless, a launch checklist gets older every day.

A collection of real failures gets more useful.