A Practical Guide to Building the Right Agent Evals

First, Choose the Right Kind of Metric

Before choosing an LLM evaluation metric, it helps to define the shape of the evaluation. Four dimensions narrow the decision.

1. Metric output

A metric can return:

• Binary: pass or fail. Best for release gates and hard requirements such as "did the agent expose sensitive data?"
• Scalar: a score on a range. Best for qualities such as relevance, completeness, or tone, where performance has degrees.
• Categorical: one of several labels. Best for routing, failure analysis, or identifying the type of issue.

Binary metrics are easier to act on, but they can hide nuance. Scalar metrics preserve nuance, but require a carefully chosen threshold before they can block a release. In practice, binary results suit hard product requirements, while scalar scores are more useful for diagnosis and comparison.

2. Modality

The evaluation should cover the experience users actually receive. A text metric cannot tell you whether a voice agent interrupted the caller, whether an image answer grounded its claims in the image, or whether a generated video matched the prompt.

Text, audio, image, and multimodal applications each require test cases and evaluators that can access the relevant modality. Transcribing everything into text may be useful, but it also removes information that can contain the failure.

3. Interaction type

Single-turn and multi-turn evals answer different questions.

A single-turn metric judges one input and output. It works well for extraction, summarization, RAG answers, and individual agent steps. A multi-turn metric judges a conversation or scenario as a whole. It is necessary when success depends on context retention, role adherence, recovery, or whether the agent eventually resolved the user's request.

Splitting a conversational failure into isolated turns may make the evaluation easier to run, but it changes the behavior being measured when the issue only becomes visible across the full thread.

4. Evaluation method

Most useful eval suites combine two methods:

• Code-based metrics are deterministic and cheap. Use them for exact constraints: schema validity, required fields, tool arguments, latency, regex matches, or known-answer checks. You can implement these as Code-Eval metrics.
• LLM-as-a-judge metrics handle subjective or semantic criteria: helpfulness, tone, policy adherence, completeness, and whether a complex task was completed. A framework such as G-Eval turns those criteria into a reusable judge.

A practical decision rule is to use code when it can evaluate the issue reliably, and an LLM judge when the issue requires human-like interpretation. Many important failures need both.

Find the Issues Before Defining the Metrics

The clearest lesson I've taken from watching teams build evals is that generic metrics are a reasonable starting point, but production behavior should determine what gets added next.

Instrument your application with LLM tracing, then inspect real traces and threads for failed tasks, repeated questions, poor tool choices, escalations, negative feedback, unusual latency, and new use cases. Production is where users reveal the inputs and behaviors your original dataset did not anticipate.

Manual sampling helps, but I've found that it tends to confirm the categories a team already knows. Your system should also attempt to discover new issues automatically. Signals can classify production interactions, surface changes in label frequency, and link each occurrence back to the underlying trace. With auto-classification enabled, new traces that fit none of the existing labels can produce recommended categories instead of disappearing into an "other" bucket.

That is the key shift: observability should not only measure known failure modes. It should expand the list of failure modes.

Turn Production Failures Into Evals

The workflow I recommend looks like this:

1. Automatically detect candidate issues. Sample production traces and threads, classify known problems, and surface new clusters or anomalies.
2. Add human review. Route high-signal examples into an annotation queue. Domain experts confirm whether each candidate is a real failure and record why.
3. Bucket confirmed failures. Build a practical taxonomy such as incorrect retrieval, incomplete resolution, policy violation, wrong tool, context loss, or poor tone.
4. Assign each issue to a metric. Choose a code-based check, an LLM judge, or a combination. One metric may cover several closely related issues; one complex issue may require several metrics.
5. Promote examples into datasets. Every important confirmed failure should become a regression case. Workflows can continuously ingest matching traces into a dataset or send them to a queue for review first.

The goal is not one metric per category. The goal is enough independent evidence to detect the issue reliably. For example, a failed support interaction might require a multi-turn task-completion judge, a code-based check that the correct tool was called, and a policy-adherence judge.

Human annotation acts as the quality layer between automatic discovery and automated evaluation. It prevents noisy clusters, harmless edge cases, and model guesses from becoming permanent test requirements. It also provides labeled cases for checking whether your metrics agree with human judgment.

Work in Eval Cycles

Here, I use eval cycle to mean a stable benchmarking period during which results are compared using the same dataset, issue taxonomy, metric definitions, thresholds, and judge models.

Production monitoring and dataset ingestion can run continuously, while the evaluation system is reviewed on a deliberate cadence. I recommend starting with a quarterly review—not as a universal requirement, but because it gives the benchmark time to produce meaningful comparisons without letting it fall too far behind changing production behavior. Teams with rapid releases or major traffic shifts may need shorter cycles; stable products may need longer ones. At the start of each new eval cycle:

1. Review newly discovered production issues and changes in use-case distribution.
2. Re-run human annotation on a representative sample.
3. Merge, split, add, or retire issue categories.
4. Update the metrics mapped to those categories.
5. Re-align the metrics against human labels.
6. Freeze the new configuration for the next cycle.

Between cycles, automatically ingest useful traces into datasets or queue them for annotation. This creates a record of how user behavior, failure modes, and metric performance drift over time without constantly moving the benchmark.

Metric drift matters too. An LLM judge can appear to change because its prompt, rubric, threshold, or underlying evaluation model changed. Do not change the evaluation model in the middle of an eval cycle. This is a measurement-control requirement: otherwise, you cannot tell whether the application improved or the judge simply moved the goalposts. If you need to adopt a new judge model, benchmark it against the current one on the same human-labeled set, then switch at the boundary of the next cycle.

The Right Evals Are a Moving Target

There is no final metric collection that captures every future failure. Products change, models change, and users find new ways to use—or break—the system.

The durable solution is a loop:

production traces → automatic issue discovery → human validation → issue categories → metrics → regression datasets → production traces

Build that loop well, and your eval suite stops being a static report. To me, that is what a good eval system should be: a living model of what quality means for your product right now.