Eval Alignment
Compare your metric evals to human annotations and validations to see how well your judges agree with your team.
Overview
Alignment measures how often your project's metrics agree with the people reviewing the same items. Wherever a metric eval and a human judgment exist for the same trace, span, thread or test case, the page treats the human as ground truth and rolls up agreement per metric, down to a confusion matrix (true positive, too strict, too lenient, true negative).
Use Alignment to:
- Spot metrics that consistently disagree with humans, and tune their judge prompts.
- Measure improvement in alignment over time as you iterate on metrics.
- Decide which metrics are reliable enough to gate releases or trigger alerts.
Open it from Annotate → Alignment. Pick what to compare with the tabs (Traces, Spans, Threads, Test cases), narrow the date range (all time by default), and use Select annotation queue... to only count one queue's annotations.
Human judgments come from two sources, switched at the top right:
- Annotations — ratings reviewers leave with an annotation form, in a queue or on a single item.
- Validations — a reviewer marking one metric result or classifier label right or wrong with Validate (see Validate a Verdict).
Each tab shows its count, so you can see at a glance which source has data.
Pass / Fail Convention
To produce a single agreement signal, both sides are normalized to a binary pass / fail:
| Source | Pass | Fail |
|---|---|---|
| Human annotation | Thumbs up, or 3-5 stars | Thumbs down, or 1-2 stars |
| Validation | Marked Correct: the metric's verdict stands | Marked Incorrect: the opposite verdict |
| Metric eval | Score ≥ metric's threshold | Score < metric's threshold |
Only thumbs and star ratings count as annotations here; other field types (choices, text, yes/no) don't produce a pass/fail. A comparison is aligned when human and metric agree; otherwise it's misaligned.
Annotations
Annotation Alignment
The Annotation Alignment card at the top plots each annotated item as a point: its Metric score against the reviewer's rating, for the metrics and annotation type you pick. Its Spearman ρ says how closely the ratings follow the scores. Switch to Label to plot ratings against classifier labels instead. Click a point to open that item.
Comparisons and Disagreements
Below the card, a summary line gives the headline (comparisons, how many annotations they came from, percentage aligned, misaligned count, annotation criteria), and Where metrics disagree splits the misaligned comparisons into too lenient (the metric passed what the human failed) and too strict (the metric failed what the human passed).
If reviewers use more than one rating field, tabs let you see All of them or one criterion at a time; alignment is computed independently per criterion.
The table lists every metric with its comparisons, alignment rate, disagreements (lenient vs. strict) and the criteria it was compared on. Click a metric to open its Confusion matrix (human outcome by metric outcome) and Confusion matrix breakdown:
| Cell | Meaning |
|---|---|
| True positive | Human said pass, metric said pass. (Aligned.) |
| Too strict | Human said pass, metric said fail. (A false negative.) |
| Too lenient | Human said fail, metric said pass. (A false positive.) |
| True negative | Human said fail, metric said fail. (Aligned.) |
The breakdown splits the comparisons by the human's rating (thumbs down or 1-2 stars, thumbs up or 3-5 stars), so you can see whether disagreements come from lukewarm ratings or strong dissent.
Validate a Verdict
Annotating a whole item isn't the only way to check a metric. Wherever a metric result has a pass/fail verdict, a Validate button sits next to it: on the Evaluations tab of a trace, span or thread, and on a test case's metrics. Classifier labels have one too, on the Classifications tab of a trace or thread.
Validate opens a small panel:
- For a metric, Verdict asks whether the pass (or fail) was right: Correct or Incorrect.
- For a classifier, Classification asks whether the label is right. Mark it Incorrect and pick the Expected label.
- A Note explains why. It's optional when you mark the result correct and required when you mark it incorrect.
Once saved, the button reads Edit validation and its icon turns green or red depending on whether you agreed. Validations are scored on the Validations source of the Alignment page: Metrics for metric results, Signals for classifier labels.
When There's Nothing to Show
If nothing in range has both a metric eval and a human judgment, the page shows No Comparisons Available. To populate it:
- Run online evals on your traces, spans or threads (or evaluate test cases), then
- annotate the same items with a thumbs or star rating, or Validate their metric results.
Annotation Queues
Set up queues, assign reviewers, and add items manually or via auto-ingestion.
Error Analysis
Turn human annotations into failure modes and metric suggestions.
Last updated on

