Introducing Confident Voice: Watch the Launch Video Now

Error Analysis

Discover failure patterns from your annotations and turn them into metrics that catch the same issues automatically.

Overview

Error Analysis turns the qualitative feedback your team leaves on annotation queue items — explanations, expected outputs, expected outcomes — into structured failure modes, then suggests a concrete metric for each one. Instead of reading hundreds of annotations to figure out what's going wrong, you get a short list of named patterns ("Policy Hallucination", "Premature Escalation", etc.) and, for each, a recommendation to create a metric, use one you have, or update one, that you can act on with one click.

Error Analysis for traces

Open it from Annotate → Analyze. The top of the page shows your project's Failure Modes: a strip with each one and how many runs found it, and a treemap sizing each failure mode and its sub-modes by how many items were annotated with them, colored by certainty. Click a block to see the annotations behind it. Add your own with New Failure Mode.

Below, the Traces, Spans and Threads tabs each list the analysis runs for that kind of queue, newest first and searchable by queue, next to the selected run.

Run an Analysis

Click Run Analysis at the top right to open Run Failure Analysis:

Run Failure Analysis
  • Queue Type — Trace, Span or Thread.
  • Annotation Queue — the queue whose annotations to analyze.
  • Allow Recommendation — let the analysis suggest new failure modes and sub-modes beyond the ones you already have.
  • Review Before Saving — save the failure modes it assigns as drafts on the queue items, for a reviewer to accept, instead of adding them to the annotations directly.

The History on the left lists past runs; click one to reuse its configuration.

A run needs annotations with feedback — an explanation, or an expected output or outcome — since the analysis needs the why, not just a thumbs or star rating. If the queue has fewer than 3, the dialog says so: Not enough annotations with feedback.

The run happens in the background, through three stages:

  1. Categorizing failure modes

    The model reads every eligible annotation and groups them into top-level failure modes — recurring patterns of what went wrong.

  2. Generating sub-patterns

    Each failure mode is expanded with sub-modes: more specific manifestations, each tagged with a certainty (high, moderate or low) so you know which patterns are well-evidenced and which are speculative.

  3. Suggesting metrics

    For every failure mode, the model proposes one of three actions:

    • Create — a brand-new metric (name, criteria, evaluation steps) tuned to detect this pattern.
    • Use existing — one of your project's metrics already covers this pattern.
    • Update existing — an existing metric is close, but its criteria need tweaking.

    Each suggestion gets a priority (high, medium or low) reflecting how strongly the data supports it.

Reading a Run

Failure modes and metric suggestions in a run

A run's header names its queue (Go to queue opens it), when it ran and its status, with Re-run Analysis to run it again. A summary line counts its failure modes, how many came with a metric suggestion, the metrics linked to them, and how many are high priority.

Each failure mode card has its name and description, a priority badge, its sub-modes with their certainty, and the metric suggestion: the recommended action, the model's rationale, and the proposed metric (name, criteria and evaluation steps to create; the existing metric to use; the proposed changes to an existing one). The button does the right thing for the suggestion:

SuggestionButtonWhat happens
CreateCreate MetricOpens the metric editor pre-filled with the suggested criteria. Save to link it to the mode.
Use existingUse This MetricLinks the failure mode to the recommended existing metric.
UpdateReview & UpdateOpens the existing metric with the proposed changes applied for review.

Once a failure mode has its metric, the button gives way to a status pill (Metric created, Metric available or Metric updated). Failure modes the model added on its own carry a Recommended pill until you accept them.

Suggestion History

If you've run the analysis several times, each failure mode keeps its suggestion history: page through older runs' recommendations for the same pattern with the arrows on its card. Older ones are read-only and labelled Past suggestion so you don't act on stale advice.

Re-running

Run the analysis again whenever your team has added meaningful new annotations. Each run is independent — older runs stay in the list — but failure modes and their linked metrics carry forward, so you build a picture of how well your metric suite covers the issues humans have flagged.

Coordinating annotators across teams?Keep annotation work organized as reviewers, queues, and datasets growBook a demo

Last updated on

Built byConfident AI