The model passed the demo. Did the workflow?

A convincing AI demo proves that the system can work once. Before release, the useful question is whether the whole workflow works often enough, fails safely, and leaves people with less work than before.

If an assistant writes a good reply but invents the order number, the model produced fluent text and the workflow failed. If it saves five minutes but adds a seven-minute review, the output may be accurate and the workflow still failed. If nobody trusts it enough to use it after the launch week, the benchmark does not matter.

The unit of evaluation should be the completed piece of work, not the model response.

Evaluate from the trigger to the changed record

Take one real task and follow it from trigger to consequence.

For a customer-support workflow, that might be:

1. A customer sends a message.
2. The system finds the account and recent orders.
3. The model decides what information matters.
4. A reply is drafted or an action is proposed.
5. A person reviews exceptions.
6. The reply is sent and the customer record changes.

The AI is only steps three and four. The rest of the system determines whether it receives the right context, has permission to act, exposes uncertainty, and records what happened.

This view also makes architecture easier to challenge. Anthropic recommends increasing complexity only when it demonstrably improves the outcome. A fixed route with one model call is easier to test than an open-ended agent, and often better for work with predictable steps. The workflow architecture map helps separate those mechanisms before evaluation begins.

Write the intended result in business language. “The answer is relevant” is not enough. “The customer receives the correct return instructions, with no unsupported promise, and the request is assigned to the right queue” can be tested.

Build the first test set from real work

Do not begin with a hundred invented prompts. Start with a small sample of work the system will actually see.

Anthropic’s research team began evaluating its multi-agent research system with roughly twenty queries representing real usage patterns. Early changes had large enough effects that a small set made problems visible; the test suite expanded as the team learned more. The same post describes a rubric covering factual accuracy, citation accuracy, completeness, source quality, and tool efficiency.

For an SME workflow, the first set should contain three kinds of cases:

• Ordinary work. The requests people handle every day.
• Hard work. Ambiguous inputs, missing fields, conflicting instructions, unusual language, and requests that require several systems.
• Costly failures. The cases where a wrong answer, wrong action, or data leak would matter most.

Sample cases should preserve the shape of the real work. Remove or replace sensitive data, but do not clean every input until it looks like a product demo. Misspellings, incomplete notes, duplicate records, and contradictory documents are part of the system’s environment.

Keep the set versioned. When a user finds a new failure, add a redacted version before fixing it. Otherwise the team can repair the visible example and accidentally reintroduce the same class of problem later.

An accuracy score cannot make the release decision

Accuracy matters, but a single percentage hides the trade-offs a process owner needs to make. Keep the dimensions separate so a good average cannot conceal an unacceptable failure.

A system with slightly lower average quality may be preferable if it identifies uncertain cases and routes them cleanly to a person. A faster system may be worse if its rare failures are irreversible.

If cost or time saved will support an investment decision, trace the operational result to financial value rather than treating model activity as the return.

The NIST AI Risk Management Framework playbook similarly treats measurement as contextual: performance, risks, impacts, and monitoring belong together. The acceptable bar depends on who is affected and what the system is allowed to do.

Use the cheapest evaluator that can judge the criterion

Different checks need different evaluators.

Use code when the answer is deterministic. Did the workflow call an approved tool? Did it change the correct record? Is every required field present? Did a citation URL exist? These checks are cheap, consistent, and should not be delegated to another model.

Use a model grader for judgments that need scale. A rubric-based evaluator can assess whether a summary is faithful, a response follows policy, or an explanation covers the required points. Google’s evaluation guidance distinguishes model-based, computation-based, and human evaluation: each answers a different kind of question.

Use people for calibration and consequence. A domain expert should examine whether the rubric represents good work, review disagreements, and test unusual behavior. Anthropic’s team found that human testers noticed source-selection biases that automated evaluations missed. Automation made evaluation broader; people made it grounded.

Do not ask one model to grade “overall quality” from one to ten and call the work evaluated. Define the decision criteria, show the grader the evidence it needs, and compare a sample of its judgments with the people accountable for the outcome.

“A human checks it” still needs a design

“A human checks it” is not a safety design.

Review fails when the interface hides the source, displays a confident answer without uncertainty, or makes the reviewer reconstruct the whole case from scratch. After enough apparently correct outputs, people also stop checking carefully.

Measure the reviewer as part of the workflow:

• Can they see the source material beside the proposed output?
• Does the system point to the uncertain or consequential part?
• Can they correct it without leaving the workflow?
• How long does review take for ordinary and difficult cases?
• What percentage of AI suggestions do people change, reject, or send back?
• Are people approving because the result is good, or because the queue is long?

Sometimes the right response to a weak review experience is less autonomy. Drafting, classifying, and retrieving information can still return time without allowing the system to act.

Decide the release boundary from consequence

Not every AI feature needs a laboratory. A tool that reformats an internal note can begin with a small representative set, clear disclosure, and periodic sampling. A system that sends prices, changes a clinical recommendation, approves a refund, or writes to a regulated record needs a much harder boundary.

Ask two questions:

1. Can the action be reversed?
2. Will someone reliably notice a bad result before it matters?

If both answers are no, the system should not take that action autonomously. Improve the workflow, add a deterministic guardrail, reduce its permissions, or keep a person in the decision.

For agents, the release boundary also needs process-level controls such as narrow tools, idempotent writes, and final-state checks; the agent-readiness deep dive covers those details.

The output of evaluation is a release boundary: what the system may do now, what must be reviewed, and what evidence would justify giving it more room later. Keep every production failure that changes that boundary as a redacted regression case. A launch checklist goes stale; a growing set of real failures gets more useful.