Annotation Queues

Every Annotation Queues method in the Confident AI Python and TypeScript SDKs.

Overview

The Confident AI SDK exposes every Annotation Queue method on the platform. This page documents how to call these methods in all supported languages. See the introduction to install the SDK and set your API key.

Methods

List Annotation Queues

Lists the annotation queues in your Confident AI project one page at a time, newest first. Each queue is returned as a summary with its progress; retrieve a queue by id for the per-reviewer breakdown.

from confident_ai import ConfidentAI
from confident_ai.annotation_queues import AnnotationQueueType

client = ConfidentAI()

result = client.annotation_queues.list(
    page=1,
    page_size=25,
    type=AnnotationQueueType.TRACE,
    search_term="geography",
)

For async mode, call a_list and await it as shown below:

result = await client.annotation_queues.a_list(...)

Parameters

Parameter	Type	Description
`page`	`Optional[int]`	The page of queues to return. Defaults to 1.
`page_size`	`Optional[int]`	The number of queues per page, at most 100. Defaults to 25.
`type`	`Optional[AnnotationQueueType]`	Returns only queues holding this kind of item. See AnnotationQueueType.
`search_term`	`Optional[str]`	Returns only queues whose name contains this text.

import { ConfidentAI } from "confident-ai";
import { AnnotationQueueType } from "confident-ai/annotation-queues";

const client = new ConfidentAI();

const result = await client.annotationQueues.list(
  {
    page: 1,
    pageSize: 25,
    type: AnnotationQueueType.TRACE,
    searchTerm: "geography"
  },
);

Parameters

Parameter	Type	Description
`page`	`number`	The page of queues to return. Defaults to 1.
`pageSize`	`number`	The number of queues per page, at most 100. Defaults to 25.
`type`	`AnnotationQueueType`	Returns only queues holding this kind of item. See AnnotationQueueType.
`searchTerm`	`string`	Returns only queues whose name contains this text.

Returns

This method returns an object of type AnnotationQueueList.

Create Annotation Queue

Creates an annotation queue in your Confident AI project and returns its id. The type you give it decides what can be added to it and cannot be changed afterwards.

from confident_ai import ConfidentAI
from confident_ai.annotation_queues import AnnotationQueueType

client = ConfidentAI()

result = client.annotation_queues.create(
    name="Failed geography answers",
    type=AnnotationQueueType.TRACE,
    form_id="<ANNOTATION-FORM-ID>",
    tags=["geography", "week-12"],
)

For async mode, call a_create and await it as shown below:

result = await client.annotation_queues.a_create(...)

Parameters

Parameter	Type	Description
`name`	`str`	Required. The name of the queue, which must be unique in the project.
`type`	`AnnotationQueueType`	Required. See AnnotationQueueType.
`form_id`	`Optional[str]`	The id of an annotation form in this project to ask of every item in the queue. Forms are created and managed in the Confident AI platform.
`tags`	`Optional[List[str]]`	Tags to put on the queue. A tag that does not exist in the project yet is created.

import { ConfidentAI } from "confident-ai";
import { AnnotationQueueType } from "confident-ai/annotation-queues";

const client = new ConfidentAI();

const result = await client.annotationQueues.create(
  "Failed geography answers",
  AnnotationQueueType.TRACE,
  { formId: "<ANNOTATION-FORM-ID>", tags: ["geography", "week-12"] },
);

Parameters

Parameter	Type	Description
`name`	`string`	Required. The name of the queue, which must be unique in the project.
`type`	`AnnotationQueueType`	Required. See AnnotationQueueType.
`formId`	`string`	The id of an annotation form in this project to ask of every item in the queue. Forms are created and managed in the Confident AI platform.
`tags`	`string[]`	Tags to put on the queue. A tag that does not exist in the project yet is created.

Returns

This method returns an object of type AnnotationQueueRef.

Get Annotation Queue

Retrieves an annotation queue by id from your Confident AI project, with how far through it your team is and how many items each reviewer holds.

from confident_ai import ConfidentAI

client = ConfidentAI()

result = client.annotation_queues.get(
    annotation_queue_id="<ANNOTATION-QUEUE-ID>",
)

For async mode, call a_get and await it as shown below:

result = await client.annotation_queues.a_get(...)

Parameters

Parameter	Type	Description
`annotation_queue_id`	`str`	Required. The id of the annotation queue.

import { ConfidentAI } from "confident-ai";

const client = new ConfidentAI();

const result = await client.annotationQueues.get("<ANNOTATION-QUEUE-ID>");

Parameters

Parameter	Type	Description
`annotationQueueId`	`string`	Required. The id of the annotation queue.

Returns

This method returns an object of type AnnotationQueue.

Update Annotation Queue

Renames an annotation queue, replaces its tags, or attaches a different annotation form to it, and returns the queue. Send formId: null to detach the current form.

from confident_ai import ConfidentAI

client = ConfidentAI()

result = client.annotation_queues.update(
    annotation_queue_id="<ANNOTATION-QUEUE-ID>",
    name="Failed geography answers (2025)",
    form_id="<ANNOTATION-FORM-ID>",
    tags=["geography", "week-13"],
)

For async mode, call a_update and await it as shown below:

result = await client.annotation_queues.a_update(...)

Parameters

Parameter	Type	Description
`annotation_queue_id`	`str`	Required. The id of the annotation queue.
`name`	`Optional[str]`	The new name of the queue, which must be unique in the project.
`form_id`	`Optional[str]`	The id of an annotation form in this project to ask of every item in the queue. Forms are created and managed in the Confident AI platform. Send null to detach the current form.
`tags`	`Optional[List[str]]`	Replaces the tags on the queue. Send an empty array to remove them all; omit it to leave them as they are.

import { ConfidentAI } from "confident-ai";

const client = new ConfidentAI();

const result = await client.annotationQueues.update(
  "<ANNOTATION-QUEUE-ID>",
  {
    name: "Failed geography answers (2025)",
    formId: "<ANNOTATION-FORM-ID>",
    tags: ["geography", "week-13"]
  },
);

Parameters

Parameter	Type	Description
`annotationQueueId`	`string`	Required. The id of the annotation queue.
`name`	`string`	The new name of the queue, which must be unique in the project.
`formId`	`string \| null`	The id of an annotation form in this project to ask of every item in the queue. Forms are created and managed in the Confident AI platform. Send null to detach the current form.
`tags`	`string[]`	Replaces the tags on the queue. Send an empty array to remove them all; omit it to leave them as they are.

Returns

This method returns an object of type AnnotationQueue.

Delete Annotation Queue

Permanently deletes an annotation queue and the items waiting in it. Annotations your team already recorded are kept.

from confident_ai import ConfidentAI

client = ConfidentAI()

result = client.annotation_queues.delete(
    annotation_queue_id="<ANNOTATION-QUEUE-ID>",
)

For async mode, call a_delete and await it as shown below:

result = await client.annotation_queues.a_delete(...)

Parameters

Parameter	Type	Description
`annotation_queue_id`	`str`	Required. The id of the annotation queue.

import { ConfidentAI } from "confident-ai";

const client = new ConfidentAI();

const result = await client.annotationQueues.delete(
  "<ANNOTATION-QUEUE-ID>",
);

Parameters

Parameter	Type	Description
`annotationQueueId`	`string`	Required. The id of the annotation queue.

Returns

This method returns an object of type AnnotationQueueRef.

Batch Annotate Items

Records annotations for several items of one queue in a single call. Each entry is applied on its own, so the response carries one result per entry and a failure on one item does not stop the rest. annotatorEmail and markAsCompleted given at the top level apply to every entry that does not set its own.

from confident_ai import ConfidentAI
from confident_ai.annotation_queues import BatchAnnotateItem
from confident_ai.annotation_queues import QueueItemAnnotation
from confident_ai.annotation_queues import QueueItemFormResponse

client = ConfidentAI()

result = client.annotation_queues.batch_annotate_items(
    annotation_queue_id="<ANNOTATION-QUEUE-ID>",
    items=[
        BatchAnnotateItem(
            queue_item_id="<QUEUE-ITEM-ID>",
            annotations=[
                QueueItemAnnotation(
                    field_type="THUMBS_RATING",
                    value=True,
                    name="Helpfulness",
                    explanation="Answered the question and cited the right source.",
                    expected_output="Mount Everest is 8,848 metres tall.",
                    expected_outcome="The user learns how tall Mount Everest is.",
                    images_mapping={}
                )
            ],
            form_responses=[
                QueueItemFormResponse(
                    label="How helpful was the answer?",
                    value="Very helpful"
                )
            ],
            annotator_email="jane@acme.com",
            flagged=False,
            mark_as_completed=True
        )
    ],
    annotator_email="jane@acme.com",
    mark_as_completed=True,
)

For async mode, call a_batch_annotate_items and await it as shown below:

result = await client.annotation_queues.a_batch_annotate_items(...)

Parameters

Parameter	Type	Description
`annotation_queue_id`	`str`	Required. The id of the annotation queue.
`items`	`List[BatchAnnotateItem]`	Required. The items to annotate. Each is processed on its own, so one failure does not stop the rest. See BatchAnnotateItem.
`annotator_email`	`Optional[str]`	The email address credited for every entry that does not name its own annotator.
`mark_as_completed`	`Optional[bool]`	Whether to mark the items annotated, for every entry that does not say otherwise. Defaults to true.

import { ConfidentAI } from "confident-ai";

const client = new ConfidentAI();

const result = await client.annotationQueues.batchAnnotateItems(
  "<ANNOTATION-QUEUE-ID>",
  [
    {
      queueItemId: "<QUEUE-ITEM-ID>",
      annotations: [
          {
            fieldType: "THUMBS_RATING",
            value: true,
            name: "Helpfulness",
            explanation: "Answered the question and cited the right source.",
            expectedOutput: "Mount Everest is 8,848 metres tall.",
            expectedOutcome: "The user learns how tall Mount Everest is.",
            imagesMapping: {}
          }
        ],
      formResponses: [
          {
            label: "How helpful was the answer?",
            value: "Very helpful"
          }
        ],
      annotatorEmail: "jane@acme.com",
      flagged: false,
      markAsCompleted: true
    }
  ],
  { annotatorEmail: "jane@acme.com", markAsCompleted: true },
);

Parameters

Parameter	Type	Description
`annotationQueueId`	`string`	Required. The id of the annotation queue.
`items`	`BatchAnnotateItem[]`	Required. The items to annotate. Each is processed on its own, so one failure does not stop the rest. See BatchAnnotateItem.
`annotatorEmail`	`string`	The email address credited for every entry that does not name its own annotator.
`markAsCompleted`	`boolean`	Whether to mark the items annotated, for every entry that does not say otherwise. Defaults to true.

Returns

This method returns an object of type BatchAnnotateResult.

Types

AnnotationQueue

A queue of production data your team annotates, with its progress and how the work is shared out.

class AnnotationQueue:
    id: str
    name: str
    type: AnnotationQueueType
    created_at: str = Field(alias="createdAt")
    updated_at: str = Field(alias="updatedAt")
    test_run_id: Optional[str] = Field(alias="testRunId")
    form_id: Optional[str] = Field(alias="formId")
    tags: List[str]
    total_items: int = Field(alias="totalItems")
    completed_items: int = Field(alias="completedItems")
    pending_items: int = Field(alias="pendingItems")
    completion_percentage: int = Field(alias="completionPercentage")
    assigned_items: int = Field(alias="assignedItems")
    assignment_breakdown: Dict[str, AssignmentBreakdownEntry] = Field(alias="assignmentBreakdown")

idstrRequired

The id of the queue, generated by Confident AI.

Example: "<ANNOTATION-QUEUE-ID>"

namestrRequired

The name of the queue, unique within the project.

Example: "Failed geography answers"

typeAnnotationQueueTypeRequired

See AnnotationQueueType.

created_atstrRequired

When the queue was created.

Example: "2025-01-15T10:30:00+00:00"

updated_atstrRequired

When the queue was last changed.

Example: "2025-01-16T09:00:00+00:00"

test_run_idOptional[str]Required

The id of the test run this queue reviews, for a queue Confident AI created from a test run. Null for a queue you created.

form_idOptional[str]Required

The id of the annotation form asked of every item in this queue, or null when the queue has no form.

Example: "<ANNOTATION-FORM-ID>"

tagsList[str]Required

The tags on the queue, for grouping and filtering the queues in your project.

Example: ["geography","week-12"]

total_itemsintRequired

How many items the queue holds.

Example: 40

completed_itemsintRequired

How many of those items have been annotated.

Example: 30

pending_itemsintRequired

How many items are still waiting to be annotated.

Example: 10

completion_percentageintRequired

The share of the queue that has been annotated, from 0 to 100, rounded to a whole number.

Example: 75

assigned_itemsintRequired

How many items in the queue are assigned to a reviewer.

Example: 24

assignment_breakdownDict[str, AssignmentBreakdownEntry]Required

Each reviewer's share of the queue, keyed by their email address. Empty when no item is assigned.

See AssignmentBreakdownEntry.

Example: {"jane@acme.com":{"assigned":12,"completed":9}}

interface AnnotationQueue {
  id: string;
  name: string;
  type: AnnotationQueueType;
  createdAt: string;
  updatedAt: string;
  testRunId: string | null;
  formId: string | null;
  tags: string[];
  totalItems: number;
  completedItems: number;
  pendingItems: number;
  completionPercentage: number;
  assignedItems: number;
  assignmentBreakdown: Record<string, AssignmentBreakdownEntry>;
}

idstringRequired

The id of the queue, generated by Confident AI.

Example: "<ANNOTATION-QUEUE-ID>"

namestringRequired

The name of the queue, unique within the project.

Example: "Failed geography answers"

typeAnnotationQueueTypeRequired

See AnnotationQueueType.

createdAtstringRequired

When the queue was created.

Example: "2025-01-15T10:30:00+00:00"

updatedAtstringRequired

When the queue was last changed.

Example: "2025-01-16T09:00:00+00:00"

testRunIdstring | nullRequired

The id of the test run this queue reviews, for a queue Confident AI created from a test run. Null for a queue you created.

formIdstring | nullRequired

The id of the annotation form asked of every item in this queue, or null when the queue has no form.

Example: "<ANNOTATION-FORM-ID>"

tagsstring[]Required

The tags on the queue, for grouping and filtering the queues in your project.

Example: ["geography","week-12"]

totalItemsnumberRequired

How many items the queue holds.

Example: 40

completedItemsnumberRequired

How many of those items have been annotated.

Example: 30

pendingItemsnumberRequired

How many items are still waiting to be annotated.

Example: 10

completionPercentagenumberRequired

The share of the queue that has been annotated, from 0 to 100, rounded to a whole number.

Example: 75

assignedItemsnumberRequired

How many items in the queue are assigned to a reviewer.

Example: 24

assignmentBreakdownRecord<string, AssignmentBreakdownEntry>Required

Each reviewer's share of the queue, keyed by their email address. Empty when no item is assigned.

See AssignmentBreakdownEntry.

Example: {"jane@acme.com":{"assigned":12,"completed":9}}

AnnotationQueueList

One page of annotation queues, with the total across all pages.

class AnnotationQueueList:
    annotation_queues: List[AnnotationQueueSummary] = Field(alias="annotationQueues")
    total_annotation_queues: int = Field(alias="totalAnnotationQueues")
    page: int
    page_size: int = Field(alias="pageSize")

annotation_queuesList[AnnotationQueueSummary]Required

The queues for the current page, newest first.

See AnnotationQueueSummary.

total_annotation_queuesintRequired

The total number of queues matching the query.

Example: 3

pageintRequired

The page this response covers.

Example: 1

page_sizeintRequired

The number of queues per page.

Example: 25

interface AnnotationQueueList {
  annotationQueues: AnnotationQueueSummary[];
  totalAnnotationQueues: number;
  page: number;
  pageSize: number;
}

annotationQueuesAnnotationQueueSummary[]Required

The queues for the current page, newest first.

See AnnotationQueueSummary.

totalAnnotationQueuesnumberRequired

The total number of queues matching the query.

Example: 3

pagenumberRequired

The page this response covers.

Example: 1

pageSizenumberRequired

The number of queues per page.

Example: 25

AnnotationQueueRef

A reference to an annotation queue by its id.

class AnnotationQueueRef:
    id: str

idstrRequired

The id of the queue, generated by Confident AI.

Example: "<ANNOTATION-QUEUE-ID>"

interface AnnotationQueueRef {
  id: string;
}

idstringRequired

The id of the queue, generated by Confident AI.

Example: "<ANNOTATION-QUEUE-ID>"

AnnotationQueueSummary

A queue as it appears in a list: what it holds and how far through it your team is, without the per-reviewer breakdown.

class AnnotationQueueSummary:
    id: str
    name: str
    type: AnnotationQueueType
    created_at: str = Field(alias="createdAt")
    updated_at: str = Field(alias="updatedAt")
    test_run_id: Optional[str] = Field(alias="testRunId")
    form_id: Optional[str] = Field(alias="formId")
    tags: List[str]
    total_items: int = Field(alias="totalItems")
    completed_items: int = Field(alias="completedItems")
    pending_items: int = Field(alias="pendingItems")
    completion_percentage: int = Field(alias="completionPercentage")

idstrRequired

The id of the queue, generated by Confident AI.

Example: "<ANNOTATION-QUEUE-ID>"

namestrRequired

The name of the queue, unique within the project.

Example: "Failed geography answers"

typeAnnotationQueueTypeRequired

See AnnotationQueueType.

created_atstrRequired

When the queue was created.

Example: "2025-01-15T10:30:00+00:00"

updated_atstrRequired

When the queue was last changed.

Example: "2025-01-16T09:00:00+00:00"

test_run_idOptional[str]Required

The id of the test run this queue reviews, for a queue Confident AI created from a test run. Null for a queue you created.

form_idOptional[str]Required

The id of the annotation form asked of every item in this queue, or null when the queue has no form.

Example: "<ANNOTATION-FORM-ID>"

tagsList[str]Required

The tags on the queue, for grouping and filtering the queues in your project.

Example: ["geography","week-12"]

total_itemsintRequired

How many items the queue holds.

Example: 40

completed_itemsintRequired

How many of those items have been annotated.

Example: 30

pending_itemsintRequired

How many items are still waiting to be annotated.

Example: 10

completion_percentageintRequired

The share of the queue that has been annotated, from 0 to 100, rounded to a whole number.

Example: 75

interface AnnotationQueueSummary {
  id: string;
  name: string;
  type: AnnotationQueueType;
  createdAt: string;
  updatedAt: string;
  testRunId: string | null;
  formId: string | null;
  tags: string[];
  totalItems: number;
  completedItems: number;
  pendingItems: number;
  completionPercentage: number;
}

idstringRequired

The id of the queue, generated by Confident AI.

Example: "<ANNOTATION-QUEUE-ID>"

namestringRequired

The name of the queue, unique within the project.

Example: "Failed geography answers"

typeAnnotationQueueTypeRequired

See AnnotationQueueType.

createdAtstringRequired

When the queue was created.

Example: "2025-01-15T10:30:00+00:00"

updatedAtstringRequired

When the queue was last changed.

Example: "2025-01-16T09:00:00+00:00"

testRunIdstring | nullRequired

The id of the test run this queue reviews, for a queue Confident AI created from a test run. Null for a queue you created.

formIdstring | nullRequired

The id of the annotation form asked of every item in this queue, or null when the queue has no form.

Example: "<ANNOTATION-FORM-ID>"

tagsstring[]Required

The tags on the queue, for grouping and filtering the queues in your project.

Example: ["geography","week-12"]

totalItemsnumberRequired

How many items the queue holds.

Example: 40

completedItemsnumberRequired

How many of those items have been annotated.

Example: 30

pendingItemsnumberRequired

How many items are still waiting to be annotated.

Example: 10

completionPercentagenumberRequired

The share of the queue that has been annotated, from 0 to 100, rounded to a whole number.

Example: 75

AnnotationQueueType

What a queue holds, fixed when it is created: TRACE, SPAN or THREAD queues are filled from your production data, while GOLDEN and TEST_RUN queues are filled by Confident AI.

class AnnotationQueueType(Enum):
    TRACE = "TRACE"
    SPAN = "SPAN"
    THREAD = "THREAD"
    GOLDEN = "GOLDEN"
    TEST_RUN = "TEST_RUN"

enum AnnotationQueueType {
  TRACE = "TRACE",
  SPAN = "SPAN",
  THREAD = "THREAD",
  GOLDEN = "GOLDEN",
  TEST_RUN = "TEST_RUN",
}

TRACE · SPAN · THREAD · GOLDEN · TEST_RUN

AssignmentBreakdownEntry

One reviewer's share of a queue.

class AssignmentBreakdownEntry:
    assigned: int
    completed: int

assignedintRequired

How many items in the queue are assigned to this reviewer.

Example: 12

completedintRequired

How many of those items the reviewer has annotated.

Example: 9

interface AssignmentBreakdownEntry {
  assigned: number;
  completed: number;
}

assignednumberRequired

How many items in the queue are assigned to this reviewer.

Example: 12

completednumberRequired

How many of those items the reviewer has annotated.

Example: 9

BatchAnnotateFailure

An entry that could not be annotated.

class BatchAnnotateFailure:
    queue_item_id: str = Field(alias="queueItemId")
    success: bool
    error: str

queue_item_idstrRequired

The id of the queue item this result is for.

Example: "<QUEUE-ITEM-ID>"

successboolRequired

False when the entry could not be annotated.

Example: false

errorstrRequired

Why this entry failed. The rest of the batch still applied.

Example: "Queue item not found in this queue."

interface BatchAnnotateFailure {
  queueItemId: string;
  success: boolean;
  error: string;
}

queueItemIdstringRequired

The id of the queue item this result is for.

Example: "<QUEUE-ITEM-ID>"

successbooleanRequired

False when the entry could not be annotated.

Example: false

errorstringRequired

Why this entry failed. The rest of the batch still applied.

Example: "Queue item not found in this queue."

BatchAnnotateItem

One item's annotation within a batch. Anything it omits falls back to the batch-level value.

class BatchAnnotateItem:
    queue_item_id: str = Field(alias="queueItemId")
    annotations: Optional[List[QueueItemAnnotation]] = None
    form_responses: Optional[List[QueueItemFormResponse]] = Field(default=None, alias="formResponses")
    annotator_email: Optional[str] = Field(default=None, alias="annotatorEmail")
    flagged: Optional[bool] = None
    mark_as_completed: Optional[bool] = Field(default=None, alias="markAsCompleted")

queue_item_idstrRequired

The id of the queue item this entry annotates.

Example: "<QUEUE-ITEM-ID>"

annotationsOptional[List[QueueItemAnnotation]]

The criteria ratings to record on the item, one entry per criterion.

See QueueItemAnnotation.

form_responsesOptional[List[QueueItemFormResponse]]

The annotations for the fields of the queue's annotation form. Sending them requires annotatorEmail.

See QueueItemFormResponse.

annotator_emailOptional[str]

The email address of the project member the work is credited to. Required when formResponses are sent, and what makes the annotation visible on the platform.

Example: "jane@acme.com"

flaggedOptional[bool]

Whether to flag the item for a second opinion.

Example: false

mark_as_completedOptional[bool]

Whether to mark the item annotated, taking it out of the pending list. Defaults to true.

Example: true

interface BatchAnnotateItem {
  queueItemId: string;
  annotations?: QueueItemAnnotation[];
  formResponses?: QueueItemFormResponse[];
  annotatorEmail?: string;
  flagged?: boolean;
  markAsCompleted?: boolean;
}

queueItemIdstringRequired

The id of the queue item this entry annotates.

Example: "<QUEUE-ITEM-ID>"

annotationsQueueItemAnnotation[]

The criteria ratings to record on the item, one entry per criterion.

See QueueItemAnnotation.

formResponsesQueueItemFormResponse[]

The annotations for the fields of the queue's annotation form. Sending them requires annotatorEmail.

See QueueItemFormResponse.

annotatorEmailstring

The email address of the project member the work is credited to. Required when formResponses are sent, and what makes the annotation visible on the platform.

Example: "jane@acme.com"

flaggedboolean

Whether to flag the item for a second opinion.

Example: false

markAsCompletedboolean

Whether to mark the item annotated, taking it out of the pending list. Defaults to true.

Example: true

BatchAnnotateResult

One result per entry sent to a batch annotation.

class BatchAnnotateResult:
    results: List[Union[BatchAnnotateSuccess, BatchAnnotateFailure]]

resultsList[Union[BatchAnnotateSuccess, BatchAnnotateFailure]]Required

One result per entry sent, in the order they were sent. Read success on each to tell the two shapes apart.

See BatchAnnotateSuccess, BatchAnnotateFailure.

interface BatchAnnotateResult {
  results: (BatchAnnotateSuccess | BatchAnnotateFailure)[];
}

results(BatchAnnotateSuccess | BatchAnnotateFailure)[]Required

One result per entry sent, in the order they were sent. Read success on each to tell the two shapes apart.

See BatchAnnotateSuccess, BatchAnnotateFailure.

BatchAnnotateSuccess

An entry that was annotated.

class BatchAnnotateSuccess:
    queue_item_id: str = Field(alias="queueItemId")
    success: bool
    annotation_ids: List[str] = Field(alias="annotationIds")
    form_response_ids: List[str] = Field(alias="formResponseIds")

queue_item_idstrRequired

The id of the queue item this result is for.

Example: "<QUEUE-ITEM-ID>"

successboolRequired

True when the item was annotated.

Example: true

annotation_idsList[str]Required

The ids of the annotations recorded for this item.

Example: ["<ANNOTATION-ID>"]

form_response_idsList[str]Required

The ids of the form answers recorded for this item.

Example: ["<ANNOTATION-FORM-RESPONSE-ID>"]

interface BatchAnnotateSuccess {
  queueItemId: string;
  success: boolean;
  annotationIds: string[];
  formResponseIds: string[];
}

queueItemIdstringRequired

The id of the queue item this result is for.

Example: "<QUEUE-ITEM-ID>"

successbooleanRequired

True when the item was annotated.

Example: true

annotationIdsstring[]Required

The ids of the annotations recorded for this item.

Example: ["<ANNOTATION-ID>"]

formResponseIdsstring[]Required

The ids of the form answers recorded for this item.

Example: ["<ANNOTATION-FORM-RESPONSE-ID>"]

MLLMImage

An image referenced from a text field by a [DEEPEVAL:IMAGE:<key>] marker. Send either a public url or the bytes in base64.

class MLLMImage:
    url: str
    local: bool
    base64: Optional[str] = None
    filename: Optional[str] = None
    mime_type: Optional[str] = Field(default=None, alias="mimeType")
    data_base64: Optional[str] = Field(default=None, alias="dataBase64")

urlstrRequired

This is the URL of the image.

Example: "https://example.com/everest.png"

localboolRequired

This is true when the image is your local file.

Example: false

base64Optional[str]

The base64 data of the image.

Example: "iVBORw0KGgo="

filenameOptional[str]

The original file name.

Example: "everest.png"

mime_typeOptional[str]

The image's MIME type.

Example: "image/png"

data_base64Optional[str]

The image encoded as a base64 data URL.

Example: "data:image/png;base64,iVBORw0KGgo="

interface MLLMImage {
  url: string;
  local: boolean;
  base64?: string;
  filename?: string;
  mimeType?: string;
  dataBase64?: string;
}

urlstringRequired

This is the URL of the image.

Example: "https://example.com/everest.png"

localbooleanRequired

This is true when the image is your local file.

Example: false

base64string

The base64 data of the image.

Example: "iVBORw0KGgo="

filenamestring

The original file name.

Example: "everest.png"

mimeTypestring

The image's MIME type.

Example: "image/png"

dataBase64string

The image encoded as a base64 data URL.

Example: "data:image/png;base64,iVBORw0KGgo="

QueueItemAnnotation

One rating, as an annotator would leave it on the item. fieldType defaults to THUMBS_RATING.

class QueueItemAnnotation:
    field_type: Optional[Literal["THUMBS_RATING", "FIVE_STAR_RATING"]] = Field(default=None, alias="fieldType")
    value: Union[bool, int]
    name: Optional[str] = None
    explanation: Optional[str] = None
    expected_output: Optional[str] = Field(default=None, alias="expectedOutput")
    expected_outcome: Optional[str] = Field(default=None, alias="expectedOutcome")
    images_mapping: Optional[Dict[str, MLLMImage]] = Field(default=None, alias="imagesMapping")

field_typeOptional[Literal["THUMBS_RATING", "FIVE_STAR_RATING"]]

The kind of rating. Defaults to THUMBS_RATING.

valueUnion[bool, int]Required

The rating to record: true or false for a THUMBS_RATING, 1 to 5 for a FIVE_STAR_RATING.

Example: true

nameOptional[str]

A name for what this rating is for. On a queue with a form, it matches a rating field's label; on a queue without one, omit it for the default rating.

Example: "Helpfulness"

explanationOptional[str]

Why the rating was given.

Example: "Answered the question and cited the right source."

expected_outputOptional[str]

The output the target should have produced. Only for an item holding a trace or span.

Example: "Mount Everest is 8,848 metres tall."

expected_outcomeOptional[str]

The outcome the conversation should have reached. Only for an item holding a thread.

Example: "The user learns how tall Mount Everest is."

images_mappingOptional[Dict[str, MLLMImage]]

Images referenced by [DEEPEVAL:IMAGE:<key>] markers in the text fields, keyed by that marker's key.

See MLLMImage.

interface QueueItemAnnotation {
  fieldType?: "THUMBS_RATING" | "FIVE_STAR_RATING";
  value: boolean | number;
  name?: string;
  explanation?: string;
  expectedOutput?: string;
  expectedOutcome?: string;
  imagesMapping?: Record<string, MLLMImage>;
}

fieldType"THUMBS_RATING" | "FIVE_STAR_RATING"

The kind of rating. Defaults to THUMBS_RATING.

valueboolean | numberRequired

The rating to record: true or false for a THUMBS_RATING, 1 to 5 for a FIVE_STAR_RATING.

Example: true

namestring

A name for what this rating is for. On a queue with a form, it matches a rating field's label; on a queue without one, omit it for the default rating.

Example: "Helpfulness"

explanationstring

Why the rating was given.

Example: "Answered the question and cited the right source."

expectedOutputstring

The output the target should have produced. Only for an item holding a trace or span.

Example: "Mount Everest is 8,848 metres tall."

expectedOutcomestring

The outcome the conversation should have reached. Only for an item holding a thread.

Example: "The user learns how tall Mount Everest is."

imagesMappingRecord<string, MLLMImage>

Images referenced by [DEEPEVAL:IMAGE:<key>] markers in the text fields, keyed by that marker's key.

See MLLMImage.

QueueItemFormResponse

One annotation on a field of the queue's annotation form.

class QueueItemFormResponse:
    label: str
    value: Optional[Any] = None

labelstrRequired

The label of the form field being answered, exactly as the form spells it.

Example: "How helpful was the answer?"

valueOptional[Any]

The annotation value, in the shape the field's type expects: a string for TEXT, a number for NUMBER or FLOAT, a boolean for BOOLEAN, one of the options in the field's config for CHOICE, and a list of them for MULTIPLE_CHOICE.

Example: "Very helpful"

interface QueueItemFormResponse {
  label: string;
  value?: unknown;
}

labelstringRequired

The label of the form field being answered, exactly as the form spells it.

Example: "How helpful was the answer?"

valueunknown

Example: "Very helpful"

Building a production pipeline?Design a scalable API workflow for evals, datasets, traces, and promptsTalk to an engineer

Last updated on September 30, 2026

Built byConfident AI