Annotation Options

Define and manage annotation criteria and reusable annotation forms for human-in-the-loop evaluation workflows in your project.

Included on the Enterprise plan. Book a demo, opens in a new tab. Included on the Team plan. Included on the Starter plan. Not included on the Free plan.

Enable or disable different human annotation options for your project. These criteria are used for all annotations within your project on Confident AI.

Configure Annotation Option

By default, both annotation systems are enabled:

Five Star Rating — Rate responses on a 1-5 star scale
Thumbs Up/Down — Simple binary feedback

You can disable either system (but not both) if you only want to use one. To configure annotation criteria:

Navigate to Project Settings → Annotation Criteria
Enable or disable the scoring system (5 star or thumbs up/down) you want to use
Click Save to apply your changes

Custom Criteria

You can create custom annotation criteria tailored to your specific evaluation needs. By default, no custom criteria are configured.

To add a custom criteria:

Click Add Custom Criteria
Enter a unique name for the criteria
Select a criteria type (Five Star Rating or Thumbs Up/Down)
Click Save

To delete a custom criteria, click the trash icon next to the criteria you want to remove.

Annotation Forms

Annotation criteria apply to every annotation in your project. Annotation forms go a step further: they let you assemble a reusable set of fields — criteria plus custom questions — and attach them to specific annotation queues. When a queue has a form attached, annotators fill in that form instead of the project's default annotation options.

A form can mix two kinds of fields:

Criteria fields — score one of your project's annotation options (a default rating or a custom criterion). Answers are saved as regular annotations.
Custom fields — collect extra structured input such as text, numbers, or choices. Answers are saved as form responses alongside the annotated item.

Field Types

Type	Description
Criteria	Score a single annotation criterion (a default rating or custom one)
Text	Free-form text answer
Number	Whole number answer
Decimal	Decimal number answer
Yes / No	A yes / no toggle
Single choice	Pick one option from a list you define
Multiple choice	Pick one or more options from a list you define

Create a Form

Open the Forms section
On this Annotation settings page, scroll to Forms and click Create form. Give the form a Name (e.g. Response quality review) and click Create to open the form editor.
Add and configure fields
Click Add to append a field, then configure it:
- Pick a Type from the table above.
- For Criteria fields, choose which criterion to score (Default ratings or one of your custom criteria) and toggle the optional inputs you want to collect: Explanation, Expected Output, and Expected Outcome.
- For Single choice / Multiple choice fields, define the selectable Options.
- Add an optional Description to guide annotators, and toggle Required to make the field mandatory.
Drag fields to reorder them, or use the trash icon to remove one.
Configuring fields in the annotation form editor
Preview and save
Switch to the Preview tab to see exactly what annotators will fill in, then click Save.

Attach a Form to a Queue

When creating an annotation queue, use the Form (optional) dropdown to pick a form. This replaces the queue's default fields with those defined on the form. Leave it set to Default to use your project's annotation options instead.

Setting this up for your organization?Set up the controls your team needs before a wider rolloutTalk to us

Last updated on July 27, 2026