For AI agents: a documentation index is available at the root level at /llms.txt and /llms-full.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page.
Trust CenterStatusSupportGet a demoPlatform
DocumentationEvals API ReferenceIntegrations & OTELPlatform SettingsSelf-HostingChangelog
DocumentationEvals API ReferenceIntegrations & OTELPlatform SettingsSelf-HostingChangelog
    • Platform Settings
    • Data Residency
    • RBAC
  • Project Settings
    • API Keys
    • Team Members
    • Roles & Permissions
    • Transformers
    • Integrations
    • Alerts
    • AI Connections
    • Model Costs
    • Data Usage
    • Evaluation Models
    • Evaluation Rules
    • Annotation Options
    • Classifiers
    • Data Sources
    • Data Retention
    • Audit Logs
  • Organization Settings
    • Projects
    • Users
    • Roles & Permissions
    • Model Credentials
    • SSO
    • Data Retention
    • Audit Logs
    • Feature Access
LogoLogo
Trust CenterStatusSupportGet a demoPlatform
On this page
  • Configure Annotation Option
  • Custom Criteria
Project Settings

Annotation Options

Define and manage annotation criteria for human-in-the-loop evaluation workflows in your project.

Was this page helpful?
Previous

Classifiers

Configure trace and thread classifiers that auto-label your data with the labels you define.

Next
Built with

Enable or disable different human annotation options for your project. These criteria are used for all annotations within your project on Confident AI.

Manage Annotation Options

Disabling criteria helps reduce confusion for users or team members by limiting the annotation options shown in the UI.

Configure Annotation Option

By default, both annotation systems are enabled:

  • Five Star Rating — Rate responses on a 1-5 star scale
  • Thumbs Up/Down — Simple binary feedback

You can disable either system (but not both) if you only want to use one. To configure annotation criteria:

  1. Navigate to Project Settings → Annotation Criteria
  2. Enable or disable the scoring system (5 star or thumbs up/down) you want to use
  3. Click Save to apply your changes

Disabling a criteria only hides it from the UI—it does not delete existing annotations for that system. It also does not block API calls or cause existing integrations to error.

Custom Criteria

You can create custom annotation criteria tailored to your specific evaluation needs. By default, no custom criteria are configured.

To add a custom criteria:

  1. Click Add Custom Criteria
  2. Enter a unique name for the criteria
  3. Select a criteria type (Five Star Rating or Thumbs Up/Down)
  4. Click Save

To delete a custom criteria, click the trash icon next to the criteria you want to remove.