Introducing Confident Voice: Watch the Launch Video Now

Eval Alignment

Compare your metric evals to human annotations and validations to see how well your judges agree with your team.

Overview

Alignment measures how often your project's metrics agree with the people reviewing the same items. Wherever a metric eval and a human judgment exist for the same trace, span, thread or test case, the page treats the human as ground truth and rolls up agreement per metric, down to a confusion matrix (true positive, too strict, too lenient, true negative).

Use Alignment to:

  • Spot metrics that consistently disagree with humans, and tune their judge prompts.
  • Measure improvement in alignment over time as you iterate on metrics.
  • Decide which metrics are reliable enough to gate releases or trigger alerts.
Alignment for traces, from annotations

Open it from Annotate → Alignment. Pick what to compare with the tabs (Traces, Spans, Threads, Test cases), narrow the date range (all time by default), and use Select annotation queue... to only count one queue's annotations.

Human judgments come from two sources, switched at the top right:

  • Annotations — ratings reviewers leave with an annotation form, in a queue or on a single item.
  • Validations — a reviewer marking one metric result or classifier label right or wrong with Validate (see Validate a Verdict).

Each tab shows its count, so you can see at a glance which source has data.

Pass / Fail Convention

To produce a single agreement signal, both sides are normalized to a binary pass / fail:

SourcePassFail
Human annotationThumbs up, or 3-5 starsThumbs down, or 1-2 stars
ValidationMarked Correct: the metric's verdict standsMarked Incorrect: the opposite verdict
Metric evalScore ≥ metric's thresholdScore < metric's threshold

Only thumbs and star ratings count as annotations here; other field types (choices, text, yes/no) don't produce a pass/fail. A comparison is aligned when human and metric agree; otherwise it's misaligned.

Annotations

Annotation Alignment

The Annotation Alignment card at the top plots each annotated item as a point: its Metric score against the reviewer's rating, for the metrics and annotation type you pick. Its Spearman ρ says how closely the ratings follow the scores. Switch to Label to plot ratings against classifier labels instead. Click a point to open that item.

Comparisons and Disagreements

Below the card, a summary line gives the headline (comparisons, how many annotations they came from, percentage aligned, misaligned count, annotation criteria), and Where metrics disagree splits the misaligned comparisons into too lenient (the metric passed what the human failed) and too strict (the metric failed what the human passed).

If reviewers use more than one rating field, tabs let you see All of them or one criterion at a time; alignment is computed independently per criterion.

The table lists every metric with its comparisons, alignment rate, disagreements (lenient vs. strict) and the criteria it was compared on. Click a metric to open its Confusion matrix (human outcome by metric outcome) and Confusion matrix breakdown:

A metric's confusion matrix against human annotations
CellMeaning
True positiveHuman said pass, metric said pass. (Aligned.)
Too strictHuman said pass, metric said fail. (A false negative.)
Too lenientHuman said fail, metric said pass. (A false positive.)
True negativeHuman said fail, metric said fail. (Aligned.)

The breakdown splits the comparisons by the human's rating (thumbs down or 1-2 stars, thumbs up or 3-5 stars), so you can see whether disagreements come from lukewarm ratings or strong dissent.

Validate a Verdict

Annotating a whole item isn't the only way to check a metric. Wherever a metric result has a pass/fail verdict, a Validate button sits next to it: on the Evaluations tab of a trace, span or thread, and on a test case's metrics. Classifier labels have one too, on the Classifications tab of a trace or thread.

Validate opens a small panel:

  • For a metric, Verdict asks whether the pass (or fail) was right: Correct or Incorrect.
  • For a classifier, Classification asks whether the label is right. Mark it Incorrect and pick the Expected label.
  • A Note explains why. It's optional when you mark the result correct and required when you mark it incorrect.
Validate panel for a trace's Faithfulness pass, marked Incorrect with a note that the drafted reply promised a refund the policy doesn't offer
Validating a metric result on a trace
Validate panel for a trace's Issues label Resolution Confirmation, marked Incorrect with Escalation to Human as the expected label and a note
Correcting a classifier's label

Once saved, the button reads Edit validation and its icon turns green or red depending on whether you agreed. Validations are scored on the Validations source of the Alignment page: Metrics for metric results, Signals for classifier labels.

Metric agreement from validations

When There's Nothing to Show

If nothing in range has both a metric eval and a human judgment, the page shows No Comparisons Available. To populate it:

  1. Run online evals on your traces, spans or threads (or evaluate test cases), then
  2. annotate the same items with a thumbs or star rating, or Validate their metric results.
Coordinating annotators across teams?Keep annotation work organized as reviewers, queues, and datasets growBook a demo

Last updated on

Built byConfident AI