Introduction to Human-in-the-Loop

Overview

Confident AI offers end users and internal annotators to leave human annotations on traces, spans, threads, and test cases. Confident AI provides a centralized place for even non-technical teams to:

• Annotate datasets
• Keep track of end user feedback
• Align metrics with human judgement
• Leave annotations for other stakeholders to internall review

Without real humans giving feedback to an LLM system, evals are no better than vibe-coding.

Ratings and Forms

Reviewers annotate with an annotation form: a set of questions you define once and reuse across queues. Two rating types carry a pass/fail that metrics can be aligned against:

Forms can also ask choices, yes/no, numbers and free text, and collect an expected output (or outcome) and an explanation per field. See Annotation Options to build forms; end users rate with thumbs or stars through the Confident API.

Two Ways to Leave Annotations

There are two ways to leave annotations on Confident AI:

The Annotate Area

Everything your team does with annotations lives under Annotate in the navigation, in four tabs:

• Queues — assign items to reviewers and annotate them one by one.
• Annotations — every annotation left, over time and as a searchable table.
• Alignment — how often your metrics agree with your reviewers, from annotations and from validations.
• Error Analysis — failure patterns found in your reviewers' feedback, with a metric suggested for each.

Single vs Multi-Turn

Single-turn annotation refers to an annotation that is left on a trace or span, while multi-turn refers to annotations on a thread. The only difference between a single and multi-turn annotation is single-turn annotation accepts an optional expected output, while a multi-turn one accepts an optional expected outcome.

Last updated on October 6, 2026