Evals before workflows: building a reliable AI evaluation harness

AI workflows need evals because model behavior is probabilistic, and manual review does not scale with the number of cases and changes. Without an evaluation harness, teams cannot reliably tell whether a prompt change or model upgrade improved the workflow, and regressions may remain invisible until users encounter them.

The practical consequence is an order of operations: define success before implementation, derive cases from real usage and observed failures, and run them continuously. A common failure mode is to build the workflow, demo it on a handful of friendly inputs, ship it, and learn through support tickets that there is no baseline for deciding whether it is getting better or worse. OpenAI's evaluation guidance calls this anti-pattern "vibe-based evals": judging changes from a few favorable examples rather than a repeatable test set.

Manual review does not measure reliability

A vibe check is a small, informal sample: run a few familiar prompts, read the outputs, decide they look acceptable, and ship. It can be useful during early prototyping, when the system is simple and changes are large, but it does not estimate performance across the workflow's input distribution. Anthropic's engineering guide to agent evals observes that early in a project, "each change to the system often has a clear, noticeable impact." At that stage, direct review can reveal obvious effects; it becomes less informative as differences narrow and the system grows more complex.

Consider an agent with ten independent steps, each with the same probability of success. At 95 percent reliability per step, the probability that all ten succeed is 0.95^10, about 60 percent. At 92 percent, it is 0.92^10, about 43 percent. These calculations assume both independence and equal per-step reliability; correlated failures or unequal steps require a different model. Even under this simplified model, small per-step shifts have a large end-to-end effect that a person reading a dozen transcripts is unlikely to estimate reliably.

These compounding effects are one reason agent regressions can be difficult to notice in production. Behavior can shift with a prompt edit, model version bump, tool schema change, or retrieval index refresh. Without a fixed measurement, drift can accumulate unobserved. Anthropic describes the resulting reactive loop: teams catch issues only in production, "where fixing one failure creates others," until "users report the agent feels worse after changes, and the team is flying blind with no way to verify except to guess and check." Hamel Husain, who has spent years consulting on these systems, puts the same diagnosis more bluntly: "Teams invest weeks building complex AI systems, but can't tell me if their changes are helping or hurting."

I run an AI engineering studio. Early in a workflow engagement, I ask both "what should the agent do?" and "how will we know it did it?" A useful implementation plan needs an answer to the second question.

Write the evals before the workflow

For a new product, "evals first" means before workflow implementation, not before customer and domain discovery. Initial criteria should reflect what the team has learned about the job and should change as discovery continues.

An eval is an executable specification. Writing one before implementation forces you to state what success means, which inputs matter, what a correct output looks like, and what failure looks like. Anthropic makes the timing argument explicitly: early on, "evals force product teams to specify what success means for the agent." The guide recommends eval-driven development: "build evals to define planned capabilities before agents can fulfill them." This resembles test-driven development: the test states the contract, and the workflow implements it.

The initial suite can be smaller than teams often expect. Anthropic suggests starting with 20 to 50 simple tasks drawn from real usage rather than waiting for hundreds. Treat that range as a practical starting point, not a universal minimum; the suite still needs to represent the workflow's important tasks and failure modes. Anthropic's test for task quality is clear: "a good task is one where two domain experts would independently reach the same pass/fail verdict." Where feasible, pair each task with a reference solution, a known working output that passes all graders, to verify that the task is solvable and the graders are configured correctly.

That collection becomes your golden dataset: in Braintrust's definition, a curated eval set that serves as the canonical benchmark for your use case, covering critical functionality and known failure modes. Version it alongside the code so changes to the benchmark are reviewable.

If a team struggles to write even a small set of representative cases with defensible verdicts, that can signal a need for more product or domain discovery. It does not establish that the team must have exactly twenty cases before implementation. Eval design is part of discovering the specification, not overhead separate from it. Deferring it can make later work harder; Anthropic warns that "evals get harder to build the longer you wait" because early failures go undocumented, definitions remain implicit, and existing behavior starts to constrain the specification.

Build cases from observed failures

Initial cases can come from customer discovery, domain experts, existing manual processes, and known requirements. Once a system is in use, its failures are often the highest-signal additions because they expose inputs and interactions the initial suite missed.

Husain calls error analysis "the single most valuable activity in AI development and consistently the highest-ROI activity," and his field guide includes a useful case study. NurtureBoss, an apartment-industry assistant, was struggling with scheduling. Reading its conversation logs revealed that date handling accounted for 66 percent of failures, a category the team had not anticipated. A targeted fix moved task success from 33 percent to 95 percent. A speculative test suite could easily have missed that concentration of failures; the logs made it visible.

Anthropic's sourcing advice follows the same progression: start with the manual checks you already run before each release, then mine your bug tracker and support queue to convert user-reported failures into test cases. LangChain's writeup on production evals argues that the bottleneck in LLM evaluation is not the scoring technique but "building the operational workflows that turn production failures into reproducible test cases." Its tooling can turn a flagged production trace into a dataset entry in one click, illustrating the goal: make observed failures inexpensive to reproduce.

This creates a durable loop. A production failure becomes an eval case, the fix is verified against it offline, and the case remains as a regression guard. Each observed failure can then become easier to detect if it recurs. Synthetic data also has a place. OpenAI's guidance endorses generated examples for edge-case coverage. Generated cases can broaden a sparse initial suite, but they reflect the assumptions used to create them and should be reviewed against domain knowledge and real usage as those signals become available. Use synthetic cases to complement observed cases, not automatically replace them.

Calibrate LLM judges against human labels

Grading large suites entirely by hand is expensive, so many harnesses use LLM-as-judge: a model grades model outputs. Zheng et al.'s MT-Bench and Chatbot Arena study found that strong LLM judges agreed with human experts more than 80 percent of the time, on par with the rate at which human experts agreed with each other. That result supports the technique, but it does not justify trusting a judge without calibration.

The same line of research catalogs important failure modes. Wang et al. showed that pairwise rankings "can be easily hacked by simply altering their order of appearance." In their experiments, reordering responses let Vicuna-13B beat ChatGPT on 66 of 80 test queries with ChatGPT as the evaluator. Zheng et al. documented verbosity bias, in which judges prefer longer answers, and self-enhancement bias, in which models favor their own outputs. A 2024 study on self-preference found a more specific mechanism: judges assign higher scores to lower-perplexity text that feels familiar to them, regardless of who wrote it. A judge from the same model family as the generator may therefore share some of its blind spots.

Several mitigations are straightforward. Validate the judge against human labels before scaling. OpenAI's guidance is to scale up only once the judge "consistently agrees with human annotations." Prefer explicit binary criteria with written critiques over ambiguous Likert scales when a criterion can be judged that way; Husain notes that a 10 percent increase in passing outputs is immediately meaningful, while a half-point move on a five-point scale requires interpretation. For multidimensional tasks, aggregate several binary criteria to preserve partial-credit and diagnostic signals. Score both orderings in pairwise comparisons. Give each rubric dimension its own isolated judge with specific criteria, as Anthropic recommends, because vague rubrics produce inconsistent judgments. Finally, read the transcripts. Anthropic writes, "you won't know if your graders are working well unless you read the transcripts and grades from many trials." Without calibration and transcript review, judge scores are not reliable evidence.

Score what the job actually requires

Metrics encode product assumptions, so the right choice depends on how users consume the result. Anthropic's guide distinguishes pass@k from pass^k. Pass@k asks whether at least one of k attempts succeeds; it fits settings where one working artifact is enough, such as code generation with human review. Pass^k asks whether all k trials succeed; it fits customer-facing settings where each user receives one attempt and consistency matters. Assuming independent trials with a constant 75 percent single-trial success rate, pass@10 is 1 - (1 - 0.75)^10, about 99.9999 percent, while pass^10 is 0.75^10, about 5.6 percent. The base rate and independence assumption matter; change either and the values change. Reporting pass@k alone for a support agent can therefore overstate the reliability customers experience.

Grading granularity matters too. Partial credit is compatible with binary checks: define separate pass/fail criteria for identifying the problem, verifying the customer, and processing the refund, then aggregate those criteria into a task score. This preserves evidence of partial progress while still allowing a required outcome, such as a processed refund, to act as a hard gate. Code-based graders are fast and objective, but they can be, in Anthropic's words, "brittle to valid variations that don't match expected patterns exactly." Another trap is over-specifying the path because agents may find valid approaches the eval designer did not anticipate. Grade outcomes rather than choreography.

A capability suite eventually saturates. When the agent passes every solvable task, the suite has no headroom to measure further capability gains, so Anthropic recommends adding harder cases. The saturated cases can still remain useful as a regression suite, where a later failure signals lost behavior. Add harder tasks to the capability suite; retain stable cases for continuous regression testing.

Run evaluations continuously

A harness that runs only before launch provides no ongoing regression signal. LangChain's framing describes two complementary modes. Offline evals run against curated datasets with reference outputs and ask, "Is this correct?" They can gate releases and catch known failure modes. Online evals run against production traces that lack ground truth and ask, "Is this helpful?" They can surface novel failures and distribution shifts the curated suite did not anticipate. Neither is a substitute for the other. Offline evaluation alone emphasizes known cases; online evaluation alone detects problems only after deployment.

Tooling can reduce the infrastructure work for both modes. Braintrust ships a GitHub Action that runs evals on pull requests and can block a merge when scores fall below a threshold. Promptfoo runs evals in CI through an open-source CLI. The UK AI Security Institute open-sourced Inspect, the framework it uses for frontier model evaluations, with composable datasets, agents, tools, and scorers, plus more than 200 prebuilt evals. These tools reduce setup costs, but teams still have to choose representative tasks, valid graders, and meaningful thresholds.

What remains is the operating habit. Anthropic's standard is that "owning and iterating on evaluations should be as routine as maintaining unit tests," and OpenAI's guidance is direct: "evaluation is a continuous process." Evals are not a one-time project phase; the cases and graders need maintenance as the workflow, models, and usage change.

Modern models and agent frameworks can make a narrow workflow quick to prototype. A one-week prototype may be feasible in some cases, but it is not a general estimate for production delivery; scope, integrations, data, and risk change the timeline. Reliability depends on the harness around the workflow: a failure-derived golden set, calibrated judges, a CI gate, and production traces that feed new cases back into the suite. A team that can quantify whether a change helped can make release decisions with more evidence than one relying on prompt review alone. After enough customer and domain discovery to define the job, build the initial evals, implement the workflow against them, and keep expanding the suite from production evidence.