Annotation Options
Define and manage annotation criteria and reusable annotation forms for human-in-the-loop evaluation workflows in your project.
Enable or disable different human annotation options for your project. These criteria are used for all annotations within your project on Confident AI.
Disabling criteria helps reduce confusion for users or team members by limiting the annotation options shown in the UI.
Configure Annotation Option
By default, both annotation systems are enabled:
- Five Star Rating — Rate responses on a 1-5 star scale
- Thumbs Up/Down — Simple binary feedback
You can disable either system (but not both) if you only want to use one. To configure annotation criteria:
- Navigate to Project Settings → Annotation Criteria
- Enable or disable the scoring system (5 star or thumbs up/down) you want to use
- Click Save to apply your changes
Disabling a criteria only hides it from the UI—it does not delete existing annotations for that system. It also does not block API calls or cause existing integrations to error.
Custom Criteria
You can create custom annotation criteria tailored to your specific evaluation needs. By default, no custom criteria are configured.
To add a custom criteria:
- Click Add Custom Criteria
- Enter a unique name for the criteria
- Select a criteria type (Five Star Rating or Thumbs Up/Down)
- Click Save
To delete a custom criteria, click the trash icon next to the criteria you want to remove.
Annotation Forms
Annotation criteria apply to every annotation in your project. Annotation forms go a step further: they let you assemble a reusable set of fields — criteria plus custom questions — and attach them to specific annotation queues. When a queue has a form attached, annotators fill in that form instead of the project’s default annotation options.
A form can mix two kinds of fields:
- Criteria fields — score one of your project’s annotation options (a default rating or a custom criterion). Answers are saved as regular annotations.
- Custom fields — collect extra structured input such as text, numbers, or choices. Answers are saved as form responses alongside the annotated item.
Field Types
Create a Form
Open the Forms section
On this Annotation settings page, scroll to Forms and click Create form. Give the form a Name (e.g. Response quality review) and click Create to open the form editor.
Add and configure fields
Click Add to append a field, then configure it:
- Pick a Type from the table above.
- For Criteria fields, choose which criterion to score (Default ratings or one of your custom criteria) and toggle the optional inputs you want to collect: Explanation, Expected Output, and Expected Outcome.
- For Single choice / Multiple choice fields, define the selectable Options.
- Add an optional Description to guide annotators, and toggle Required to make the field mandatory.
Drag fields to reorder them, or use the trash icon to remove one.
Attach a Form to a Queue
When creating an annotation queue, use the Form (optional) dropdown to pick a form. This replaces the queue’s default fields with those defined on the form. Leave it set to Default to use your project’s annotation options instead.
Criteria answers are stored as normal annotations and appear wherever annotations are shown — the Observatory, test case details, and CSV exports. Custom field answers are stored as form responses and shown alongside the annotated trace, span, thread, or test case.