LLM error analysis is the process of turning a bad evaluation score into a shipped fix: reading why the metric failed the output, categorizing the failure mode, localizing the root cause to a specific pipeline step using trace data, fixing that layer, and converting the failure into a permanent regression test. It is the step between "we run evals" and "evals improve the product."
Most teams stall exactly here. They build a dataset, wire up metrics, run the suite — and then stare at a dashboard that says 71% pass rate. The score is not the value. Nobody ships a fix because a number went down; they ship a fix because they know which failure mode, in which pipeline step, for which kind of input. Error analysis is the methodology that produces that knowledge, and it is almost entirely mechanical once you have the steps.
This guide walks the six steps with the decision rules for each, the failure-category vocabulary that makes step 3 fast, and the automation that removes the manual labor. It assumes evaluation results exist; if you are still setting up metrics, start with what makes a good eval.
What are the six steps of LLM error analysis?
Error analysis is a fixed sequence: collect failures, read the reasoning, categorize the failure mode, localize the root cause in the trace, fix the right layer, and lock in the regression test. Skipping a step produces the two classic pathologies — prompt-tweaking without diagnosis (skipped steps 3–4) and fixes that silently regress later (skipped step 6).
Step 1: Collect the failures in one place
Gather every failing test case from the run — and, in production, every trace flagged by online evals, signals, or human annotation — into a single review queue. The unit of review is the failing case with its full context attached: input, output, expected output if one exists, retrieval context, tool calls, and the metric verdicts with their reasoning.
The anti-pattern here is reviewing failures inside whatever surface they happened to occur in (a CI log, a Slack alert, a spreadsheet export). Fragmented review means no cross-case pattern detection, and patterns are the whole point of the next two steps.
Step 2: Read the judge's reasoning, not just the score
Every serious evaluation metric — criteria-based LLM-as-a-judge metrics especially — returns a reasoning string alongside the score: which claims were unsupported, which part of the answer was irrelevant, what the output omitted. Read it before forming any hypothesis. The reasoning converts a 0.4 into a specific allegation you can verify in seconds.
Two things to check while reading. First, is the metric right? A percentage of "failures" in any run are metric errors — the judge misread the context, or your criteria are ambiguous. These are not application bugs; they are metric bugs, and they route to a different fix (tighten the criteria, adjust the threshold, revalidate against human labels). Second, is the failure one instance or a family? If three reasoning strings make the same allegation, you are looking at a pattern, and patterns get prioritized over one-offs.
Step 3: Categorize the failure mode
Assign every confirmed failure to a category from a fixed vocabulary. Categorization is what turns a pile of bad outputs into a prioritized work queue — "60% of our failures are retrieval misses" is actionable in a way that sixty individual bug reports are not.
The vocabulary that covers most single-turn and agentic applications:
Failure category | What it looks like | Root cause usually lives in |
|---|---|---|
Retrieval miss | Right answer exists in the corpus; wrong or no chunks retrieved | Index, chunking, query construction, embeddings |
Grounding failure | Good context retrieved; output contradicts or ignores it | Prompt, model choice, context formatting |
Instruction-following failure | Output violates format, length, tone, or policy instructions | Prompt structure, competing instructions, model |
Tool misuse | Wrong tool, wrong arguments, missing tool call | Tool descriptions, agent prompt, routing logic |
Reasoning error | Right facts, wrong conclusion or calculation | Model choice, task decomposition, missing intermediate steps |
Edge-case input | Input outside the distribution the app was designed for | Input handling, guardrails, dataset coverage |
Metric error | The output is actually fine; the judge is wrong | Metric criteria, threshold, judge model |
Keep the vocabulary stable across runs. The categories become trend lines — a use case whose grounding failures double after a model swap is a diagnosis delivered for free.
This diagnostic vocabulary is intentionally different from the production failure taxonomy: production categories describe what users experience (context loss, goal drift, policy violation), while error-analysis categories identify the engineering layer likely to require a fix (retrieval, grounding, tool use, metric). Use the production mode to triage impact and the diagnostic category to route remediation.
Step 4: Localize the root cause in the trace
Open the trace behind the failing case and walk the span tree to find the first step where things went wrong. The category from step 3 tells you where to look: a retrieval miss sends you to the retrieval span's query and returned chunks; tool misuse sends you to the tool span's arguments; a grounding failure sends you to the generation span's full prompt as the model actually received it.
The decision rule that resolves most ambiguity: find the first span whose output is wrong given its input. Everything downstream of that span is contaminated; everything upstream is exonerated. A wrong answer built faithfully on bad chunks is a retrieval problem, not a prompt problem — teams that skip this step routinely tune prompts to compensate for broken retrieval, which works just long enough to make the real bug harder to find. (If your application is not traced yet, this step is guesswork; see What Is LLM Tracing? for the instrumentation this methodology assumes.)
Step 5: Fix the layer the diagnosis names
Make the fix at the layer step 4 identified, and only that layer. The mapping in the step-3 table is the guide: chunking and query construction for retrieval misses, prompt and context formatting for grounding failures, tool descriptions for tool misuse, model or decomposition for reasoning errors.
One discipline point: fix one category at a time, highest-frequency first, and re-run the full suite after each fix. Batching five fixes into one change means the re-run cannot tell you which fix worked, which regressed something, and which did nothing.
Step 6: Lock it in as a regression test
Every confirmed failure becomes a dataset entry before the fix ships — input, expected behavior, and the failure category as a tag. The re-run that validates the fix also validates that the new test passes, and from then on the case runs in every CI cycle and every future model or prompt change. This is the compounding step: teams that do it accumulate coverage exactly where their application actually breaks; teams that skip it re-discover the same failures quarterly.
Prioritizing: not all failures are equal
Review order is severity times frequency, with metric errors handled first. Metric errors go first not because they are severe but because they poison the denominator — until the judge is trustworthy, every other count is suspect. After that, a failure mode affecting 30% of a high-traffic use case outranks a rare edge case, with one exception: safety-class failures (leaked data, harmful advice, unauthorized actions) jump the queue at any frequency.
A practical cadence: after every significant eval run, spend the first pass confirming or dismissing metric errors (fast — the reasoning strings make most verdicts obvious), the second pass categorizing what remains, and only then open traces for the top two or three categories. Resist the urge to root-cause everything; a category with two instances can wait for more data.
What to automate
Steps 1–3 automate well; steps 4–5 stay human. The manual version of error analysis dies of labor cost — exporting failures, pasting transcripts into documents, hand-tagging categories — so the teams that sustain the practice are the ones that automate the clerical layers and spend human attention only on diagnosis and fixes.
What the automated version looks like, and what to look for in tooling:
- Collection: failing test cases and flagged production traces flow into annotation queues automatically — no exports. Human annotations (thumbs-down, reviewer notes) land in the same queue as metric failures.
- Clustering: failures are grouped automatically by similarity into candidate failure modes, so step 3 starts from proposed categories rather than a blank page. Reviewer judgment confirms or corrects the clusters instead of creating them.
- Metric recommendation: recurring failure patterns that no current metric catches become suggested metrics — closing the loop from "humans keep flagging this" to "a judge now catches it automatically." This is the mechanism that turns qualitative review into growing automated coverage.
- Trace linkage: every failure links directly to its trace, so step 4 is one click, not a log search.
- Dataset conversion: confirmed failures convert to dataset entries in place, with tags — step 6 without the copy-paste.
Confident AI implements this pipeline end to end — annotation queues that ingest both metric failures and human feedback, automatic failure clustering, metric recommendations from confirmed patterns, and one-click conversion of failures into regression datasets — with each failure linked to its full trace. Whatever stack you use, the automation checklist above is the bar: any step still requiring manual export is a step that will quietly stop happening.
Frequently Asked Questions
What is LLM error analysis?
LLM error analysis is the structured process of converting failing evaluation results into fixes: collecting failures, reading the judge's reasoning, categorizing each failure into a fixed vocabulary of modes (retrieval miss, grounding failure, tool misuse, and so on), localizing the root cause to a specific span in the trace, fixing that layer, and converting the failure into a permanent regression test.
How do I find the root cause of a failing LLM test case?
Open the trace behind the case and find the first span whose output is wrong given its input. Everything downstream is contaminated by that span; everything upstream is fine. A wrong answer built on bad retrieved chunks is a retrieval problem; a wrong answer built on good chunks is a generation problem. Without trace data this distinction is guesswork, which is why error analysis presumes instrumentation.
What are the most common LLM failure categories?
Seven categories cover most applications: retrieval misses (wrong chunks fetched), grounding failures (output ignores good context), instruction-following failures (format, tone, or policy violations), tool misuse (wrong tool or arguments), reasoning errors (right facts, wrong conclusion), edge-case inputs (out-of-distribution requests), and metric errors (the output is fine; the judge is wrong). Keeping the vocabulary stable across runs turns categories into trend lines.
How do I know if a failure is a metric error rather than a real bug?
Read the metric's reasoning string and verify its allegation against the actual output and context. If the judge claims an unsupported statement that is in fact supported, or penalizes something your criteria never prohibited, the failure is a metric bug — route it to metric fixes (tighter criteria, adjusted threshold, revalidation against human labels) rather than application changes. Triage metric errors first, because they distort every other failure count.
Should every failed test case become a regression test?
Every confirmed failure should — after you have verified it is a real application bug rather than a metric error. Add the case to the evaluation dataset with its failure category as a tag before shipping the fix, so the fix is validated against it and every future change runs it. Failures that die in a dashboard get re-discovered later at production prices.
How much of error analysis can be automated?
The clerical layers: collecting failures into queues, clustering them into candidate failure modes, linking each to its trace, and converting confirmed failures into dataset entries. Platforms can also recommend new metrics from recurring human-flagged patterns. The genuinely human parts are confirming the diagnosis and choosing the fix — automation's job is to make sure human attention is spent only there.
Read next
- What Makes a Good Eval — metric design and validation, the prerequisite for trusting the failures you analyze.
- What Is LLM Tracing? Traces, Spans, and Threads Explained — the instrumentation that makes root-cause localization possible.
- AI Production Issue Detection: A Failure Taxonomy and Detection Framework — the production-side detection system that feeds this methodology its inputs.