Annotations
Define annotation fields and reusable annotation forms for human-in-the-loop evaluation workflows in your project.
Configure the human annotation fields and forms for your project on Confident AI. Fields are the reusable questions annotators answer, such as a star rating or a thumbs up/down, and forms decide which fields annotators fill in. Both live on the Annotations page under Project Settings.
Fields
Fields are reusable questions for your project. Each field has a label, a type, and an optional description, and can be added to any form.
To create a field:
- Navigate to Project Settings → Annotations
- Under Fields, click Create field
- Enter a Label (what annotators are asked, e.g. Helpfulness)
- Select a Type (see the table below), and for Single choice / Multiple choice fields, define the Options
- Optionally add a Description to guide annotators
- Click Create
Field Types
| Type | Description |
|---|---|
| Thumbs Up/Down | A thumbs up or thumbs down |
| Five Star Rating | A rating from 1 to 5 stars |
| Text | Free-form text answer |
| Number | Whole number answer |
| Decimal | Decimal number answer |
| Yes / No | A yes / no toggle |
| Single choice | Pick one option from a list you define |
| Multiple choice | Pick one or more options from a list you define |
Edit or Delete a Field
Click a field (or ⋮ → Edit) to change its label, description, or options. A field's type can't be changed after it's created; create a new field to use a different type.
Changing a choice field's options creates a new version of the field, shown next to its type in the list. Graphs show the latest version by default, so annotations on previous versions only appear when you pick that version from the graph's version selector.
To delete a field, click ⋮ → Delete. The field is removed from every form that uses it.
Annotation Forms
Forms assemble a reusable set of your project's fields and decide what annotators fill in. Attach a form to specific annotation queues, and mark one form as your project's default form, used for annotations made outside of a queue, such as annotating a trace directly. A project has at most one default form.
The Forms table lists each form with how many fields it has and how many queues use it.
Create a Form
Open the Forms section
On the Annotations settings page, scroll to Forms and click Create form. Give the form a Name (e.g. Response quality review) and click Create to open the form editor.
Add and configure fields
Click Add to append a field, then configure it:
- Pick a Field from your project's fields (each field can appear on a form once).
- Toggle Required if annotators must fill it in before they can submit.
- Toggle Collect explanation to let annotators explain their annotation.
Above the fields, toggle Collect Expected Output/Outcome to ask for an expected output (traces and spans) or expected outcome (threads) once per submission. Use the Default Form switch under the form's name to make it the project's default.
Drag fields to reorder them, or use the trash icon to remove one.
The form editor Preview and save
Switch to the Preview tab to see exactly what annotators will fill in, then click Save.
Attach a Form to a Queue
When creating an annotation queue, use the Form dropdown to pick the form annotators fill in for each item. It starts on your project's default form.
Deleting a form that's attached to queues reverts those queues to your project's default annotation settings.
Setting this up for your organization?Set up the controls your team needs before a wider rolloutTalk to usLast updated on