Annotations

Overview

The Confident AI SDK exposes every Annotation method on the platform. This page documents how to call these methods in all supported languages. See the introduction to install the SDK and set your API key.

Methods

List Annotations

Lists the annotations in your Confident AI project one page at a time, newest first by default. Filter by the trace, span or thread they were left on, by rating scale, and by time window.

from confident_ai import ConfidentAI
from confident_ai.common import AnnotationFieldType
from confident_ai.annotations import AnnotationSortBy

client = ConfidentAI()

result = client.annotations.list(
    page=1,
    page_size=25,
    start="2025-01-01T00:00:00+00:00",
    end="2025-01-31T23:59:59+00:00",
    sort_by=AnnotationSortBy.CREATEDAT,
    ascending="false",
    trace_uuid="3f9c2a1e-5b7d-4c8e-9f01-2a3b4c5d6e7f",
    span_uuid="9d2c6f7e-1a3b-4c5d-8e9f-0a1b2c3d4e5f",
    thread_id="thread-42",
    field_type=AnnotationFieldType.TEXT,
)

For async mode, call a_list and await it as shown below:

result = await client.annotations.a_list(...)

Parameters

Parameter	Type	Description
`page`	`Optional[int]`	The page of annotations to return. Defaults to 1.
`page_size`	`Optional[int]`	The number of annotations per page, at most 100. Defaults to 25.
`start`	`Optional[str]`	Returns only annotations left at or after this ISO 8601 datetime. Defaults to 60 days ago.
`end`	`Optional[str]`	Returns only annotations left before this ISO 8601 datetime. Defaults to the current time.
`sort_by`	`Optional[AnnotationSortBy]`	This determines the field to sort by. Defaults to `createdAt`. See AnnotationSortBy.
`ascending`	`Optional[Literal['true', 'false']]`	This determines if the field specified in `sortBy` should be in ascending order. Defaults to `false`, which returns the newest annotations first.
`trace_uuid`	`Optional[str]`	Returns only annotations left on this trace.
`span_uuid`	`Optional[str]`	Returns only annotations left on this span.
`thread_id`	`Optional[str]`	Returns only annotations left on this thread.
`field_type`	`Optional[AnnotationFieldType]`	Returns only annotations of this field type. See AnnotationFieldType.

import { ConfidentAI } from "confident-ai";
import { AnnotationSortBy } from "confident-ai/annotations";
import { AnnotationFieldType } from "confident-ai/common";

const client = new ConfidentAI();

const result = await client.annotations.list(
  {
    page: 1,
    pageSize: 25,
    start: "2025-01-01T00:00:00+00:00",
    end: "2025-01-31T23:59:59+00:00",
    sortBy: AnnotationSortBy.CREATEDAT,
    ascending: "false",
    traceUuid: "3f9c2a1e-5b7d-4c8e-9f01-2a3b4c5d6e7f",
    spanUuid: "9d2c6f7e-1a3b-4c5d-8e9f-0a1b2c3d4e5f",
    threadId: "thread-42",
    fieldType: AnnotationFieldType.TEXT
  },
);

Parameters

Parameter	Type	Description
`page`	`number`	The page of annotations to return. Defaults to 1.
`pageSize`	`number`	The number of annotations per page, at most 100. Defaults to 25.
`start`	`string`	Returns only annotations left at or after this ISO 8601 datetime. Defaults to 60 days ago.
`end`	`string`	Returns only annotations left before this ISO 8601 datetime. Defaults to the current time.
`sortBy`	`AnnotationSortBy`	This determines the field to sort by. Defaults to `createdAt`. See AnnotationSortBy.
`ascending`	`"true" \| "false"`	This determines if the field specified in `sortBy` should be in ascending order. Defaults to `false`, which returns the newest annotations first.
`traceUuid`	`string`	Returns only annotations left on this trace.
`spanUuid`	`string`	Returns only annotations left on this span.
`threadId`	`string`	Returns only annotations left on this thread.
`fieldType`	`AnnotationFieldType`	Returns only annotations of this field type. See AnnotationFieldType.

Returns

This method returns an object of type AnnotationList.

Trace Annotation

Records a rating against exactly one trace, span or thread, and returns its id. The target must already exist in your project. A rating on a THUMBS_RATING scale is 0 or 1; on a FIVE_STAR_RATING scale it is 1 to 5.

An annotation left on a trace.

from confident_ai import ConfidentAI
from confident_ai.common import AnnotationFieldType
from confident_ai.annotations import TraceAnnotationRequest

client = ConfidentAI()

result = client.annotations.create(
    annotation=TraceAnnotationRequest(
        trace_uuid="3f9c2a1e-5b7d-4c8e-9f01-2a3b4c5d6e7f",
        expected_output="Mount Everest is 8,848 metres tall.",
        field_type=AnnotationFieldType.TEXT,
        value={},
        name="Helpfulness",
        explanation="Answered the question and cited the right source.",
        user_id="end-user-42",
        images_mapping={}
    ),
)

For async mode, call a_create and await it as shown below:

result = await client.annotations.a_create(...)

Parameters

Parameter	Type	Description
`annotation`	`CreateAnnotationRequest`	Required. An annotation to record against exactly one target. Send `traceUuid`, `spanUuid` or `threadId` — never more than one — and the field that goes with it: `expectedOutput` for a trace or span, `expectedOutcome` for a thread. `fieldType` defaults to THUMBS_RATING. Pass one of TraceAnnotationRequest, SpanAnnotationRequest, ThreadAnnotationRequest. See CreateAnnotationRequest.

import { ConfidentAI } from "confident-ai";
import { AnnotationFieldType } from "confident-ai/common";

const client = new ConfidentAI();

const result = await client.annotations.create(
  {
    traceUuid: "3f9c2a1e-5b7d-4c8e-9f01-2a3b4c5d6e7f",
    expectedOutput: "Mount Everest is 8,848 metres tall.",
    fieldType: AnnotationFieldType.TEXT,
    value: {},
    name: "Helpfulness",
    explanation: "Answered the question and cited the right source.",
    userId: "end-user-42",
    imagesMapping: {}
  },
);

Parameters

Parameter	Type	Description
`annotation`	`CreateAnnotationRequest`	Required. An annotation to record against exactly one target. Send `traceUuid`, `spanUuid` or `threadId` — never more than one — and the field that goes with it: `expectedOutput` for a trace or span, `expectedOutcome` for a thread. `fieldType` defaults to THUMBS_RATING. Pass one of TraceAnnotationRequest, SpanAnnotationRequest, ThreadAnnotationRequest. See CreateAnnotationRequest.

Returns

This method returns an object of type AnnotationRef.

Span Annotation

An annotation left on a span.

from confident_ai import ConfidentAI
from confident_ai.common import AnnotationFieldType
from confident_ai.annotations import SpanAnnotationRequest

client = ConfidentAI()

result = client.annotations.create(
    annotation=SpanAnnotationRequest(
        span_uuid="9d2c6f7e-1a3b-4c5d-8e9f-0a1b2c3d4e5f",
        expected_output="Mount Everest is 8,848 metres tall.",
        field_type=AnnotationFieldType.TEXT,
        value={},
        name="Helpfulness",
        explanation="Answered the question and cited the right source.",
        user_id="end-user-42",
        images_mapping={}
    ),
)

For async mode, call a_create and await it as shown below:

result = await client.annotations.a_create(...)

Parameters

Parameter	Type	Description
`annotation`	`CreateAnnotationRequest`	Required. An annotation to record against exactly one target. Send `traceUuid`, `spanUuid` or `threadId` — never more than one — and the field that goes with it: `expectedOutput` for a trace or span, `expectedOutcome` for a thread. `fieldType` defaults to THUMBS_RATING. Pass one of TraceAnnotationRequest, SpanAnnotationRequest, ThreadAnnotationRequest. See CreateAnnotationRequest.

import { ConfidentAI } from "confident-ai";
import { AnnotationFieldType } from "confident-ai/common";

const client = new ConfidentAI();

const result = await client.annotations.create(
  {
    spanUuid: "9d2c6f7e-1a3b-4c5d-8e9f-0a1b2c3d4e5f",
    expectedOutput: "Mount Everest is 8,848 metres tall.",
    fieldType: AnnotationFieldType.TEXT,
    value: {},
    name: "Helpfulness",
    explanation: "Answered the question and cited the right source.",
    userId: "end-user-42",
    imagesMapping: {}
  },
);

Parameters

Parameter	Type	Description
`annotation`	`CreateAnnotationRequest`	Required. An annotation to record against exactly one target. Send `traceUuid`, `spanUuid` or `threadId` — never more than one — and the field that goes with it: `expectedOutput` for a trace or span, `expectedOutcome` for a thread. `fieldType` defaults to THUMBS_RATING. Pass one of TraceAnnotationRequest, SpanAnnotationRequest, ThreadAnnotationRequest. See CreateAnnotationRequest.

Returns

This method returns an object of type AnnotationRef.

Thread Annotation

An annotation left on a thread.

from confident_ai import ConfidentAI
from confident_ai.common import AnnotationFieldType
from confident_ai.annotations import ThreadAnnotationRequest

client = ConfidentAI()

result = client.annotations.create(
    annotation=ThreadAnnotationRequest(
        thread_id="thread-42",
        expected_outcome="The user learns how tall Mount Everest is.",
        field_type=AnnotationFieldType.TEXT,
        value={},
        name="Helpfulness",
        explanation="Answered the question and cited the right source.",
        user_id="end-user-42",
        images_mapping={}
    ),
)

For async mode, call a_create and await it as shown below:

result = await client.annotations.a_create(...)

Parameters

Parameter	Type	Description
`annotation`	`CreateAnnotationRequest`	Required. An annotation to record against exactly one target. Send `traceUuid`, `spanUuid` or `threadId` — never more than one — and the field that goes with it: `expectedOutput` for a trace or span, `expectedOutcome` for a thread. `fieldType` defaults to THUMBS_RATING. Pass one of TraceAnnotationRequest, SpanAnnotationRequest, ThreadAnnotationRequest. See CreateAnnotationRequest.

import { ConfidentAI } from "confident-ai";
import { AnnotationFieldType } from "confident-ai/common";

const client = new ConfidentAI();

const result = await client.annotations.create(
  {
    threadId: "thread-42",
    expectedOutcome: "The user learns how tall Mount Everest is.",
    fieldType: AnnotationFieldType.TEXT,
    value: {},
    name: "Helpfulness",
    explanation: "Answered the question and cited the right source.",
    userId: "end-user-42",
    imagesMapping: {}
  },
);

Parameters

Parameter	Type	Description
`annotation`	`CreateAnnotationRequest`	Required. An annotation to record against exactly one target. Send `traceUuid`, `spanUuid` or `threadId` — never more than one — and the field that goes with it: `expectedOutput` for a trace or span, `expectedOutcome` for a thread. `fieldType` defaults to THUMBS_RATING. Pass one of TraceAnnotationRequest, SpanAnnotationRequest, ThreadAnnotationRequest. See CreateAnnotationRequest.

Returns

This method returns an object of type AnnotationRef.

Get Annotation

Retrieves an annotation by id from your Confident AI project, with the ids of the trace, span or thread it was left on and the team member who left it.

from confident_ai import ConfidentAI

client = ConfidentAI()

result = client.annotations.get(annotation_id="<ANNOTATION-ID>")

For async mode, call a_get and await it as shown below:

result = await client.annotations.a_get(...)

Parameters

Parameter	Type	Description
`annotation_id`	`str`	Required. The id of the annotation.

import { ConfidentAI } from "confident-ai";

const client = new ConfidentAI();

const result = await client.annotations.get("<ANNOTATION-ID>");

Parameters

Parameter	Type	Description
`annotationId`	`string`	Required. The id of the annotation.

Returns

This method returns an object of type Annotation.

Update Annotation

Updates the rating, scale or text of an annotation and returns its id. The target it was left on cannot be changed: expectedOutput is rejected on a thread annotation, and expectedOutcome on a trace or span annotation.

from confident_ai import ConfidentAI
from confident_ai.common import AnnotationFieldType

client = ConfidentAI()

result = client.annotations.update(
    annotation_id="<ANNOTATION-ID>",
    field_type=AnnotationFieldType.TEXT,
    value={},
    explanation="On reflection the answer omitted the source.",
    expected_output="Mount Everest is 8,848 metres tall.",
    expected_outcome="The user learns how tall Mount Everest is.",
    images_mapping={},
)

For async mode, call a_update and await it as shown below:

result = await client.annotations.a_update(...)

Parameters

Parameter	Type	Description
`annotation_id`	`str`	Required. The id of the annotation.
`field_type`	`Optional[AnnotationFieldType]`	See AnnotationFieldType.
`value`	`Optional[AnnotationValue]`	See AnnotationValue.
`explanation`	`Optional[str]`	Why the annotation was given.
`expected_output`	`Optional[str]`	The output the target should have produced. Only for an annotation left on a trace or span.
`expected_outcome`	`Optional[str]`	The outcome the conversation should have reached. Only for an annotation left on a thread.
`images_mapping`	`Optional[Dict[str, MLLMImage]]`	Images referenced by `[DEEPEVAL:IMAGE:<key>]` markers in the text fields, keyed by that marker's key. See MLLMImage.

import { ConfidentAI } from "confident-ai";
import { AnnotationFieldType } from "confident-ai/common";

const client = new ConfidentAI();

const result = await client.annotations.update(
  "<ANNOTATION-ID>",
  {
    fieldType: AnnotationFieldType.TEXT,
    value: {},
    explanation: "On reflection the answer omitted the source.",
    expectedOutput: "Mount Everest is 8,848 metres tall.",
    expectedOutcome: "The user learns how tall Mount Everest is.",
    imagesMapping: {}
  },
);

Parameters

Parameter	Type	Description
`annotationId`	`string`	Required. The id of the annotation.
`fieldType`	`AnnotationFieldType`	See AnnotationFieldType.
`value`	`AnnotationValue`	See AnnotationValue.
`explanation`	`string`	Why the annotation was given.
`expectedOutput`	`string`	The output the target should have produced. Only for an annotation left on a trace or span.
`expectedOutcome`	`string`	The outcome the conversation should have reached. Only for an annotation left on a thread.
`imagesMapping`	`Record<string, MLLMImage>`	Images referenced by `[DEEPEVAL:IMAGE:<key>]` markers in the text fields, keyed by that marker's key. See MLLMImage.

Returns

This method returns an object of type AnnotationRef.

Types

Annotation

A human annotation left on a trace, span or thread, with the ids of what it was left on.

class Annotation:
    id: str
    field_type: AnnotationFieldType = Field(alias="fieldType")
    value: Optional[Union[str, float, bool, List[str]]]
    name: Optional[str]
    explanation: Optional[str]
    expected_outcome: Optional[str] = Field(alias="expectedOutcome")
    expected_output: Optional[str] = Field(alias="expectedOutput")
    created_at: str = Field(alias="createdAt")
    user: Optional[UserReference]
    trace_uuid: Optional[str] = Field(alias="traceUuid")
    span_uuid: Optional[str] = Field(alias="spanUuid")
    thread_id: Optional[str] = Field(alias="threadId")
    test_case_id: Optional[str] = Field(alias="testCaseId")

idstrRequired

This is the id of the annotation generated by Confident AI.

Example: "<ANNOTATION-ID>"

field_typeAnnotationFieldTypeRequired

See AnnotationFieldType.

valueOptional[Union[str, float, bool, List[str]]]Required

The annotated value, in the shape its fieldType expects: a boolean for THUMBS_RATING (true is thumbs up) and BOOLEAN, a whole number from 1 to 5 for FIVE_STAR_RATING, a number for NUMBER or FLOAT, a string for TEXT and CHOICE, and a list of strings for MULTIPLE_CHOICE. Null when the annotation was left without a value.

Example: true

nameOptional[str]Required

The name of the annotation.

explanationOptional[str]Required

This is the explanation for the annotation.

Example: "Correct and concise."

expected_outcomeOptional[str]Required

This is the annotated expected outcome, for conversation annotations.

expected_outputOptional[str]Required

This is the annotated expected output, for span and trace annotations.

Example: "The capital of France is Paris."

created_atstrRequired

The timestamp when the annotation was created.

Example: "2025-01-15T11:00:00+00:00"

userOptional[UserReference]Required

See UserReference.

trace_uuidOptional[str]Required

The uuid of the trace this annotation was left on, or null when it was left on a span or thread.

Example: "3f9c2a1e-5b7d-4c8e-9f01-2a3b4c5d6e7f"

span_uuidOptional[str]Required

The uuid of the span this annotation was left on, or null when it was left on a trace or thread.

thread_idOptional[str]Required

The id of the thread this annotation was left on. It is also set for an annotation on a trace that belongs to a thread.

test_case_idOptional[str]Required

The id of the test case the annotated trace formed, for a trace ingested into a test run.

interface Annotation {
  id: string;
  fieldType: AnnotationFieldType;
  value: string | number | boolean | string[] | null;
  name: string | null;
  explanation: string | null;
  expectedOutcome: string | null;
  expectedOutput: string | null;
  createdAt: string;
  user: UserReference | null;
  traceUuid: string | null;
  spanUuid: string | null;
  threadId: string | null;
  testCaseId: string | null;
}

idstringRequired

This is the id of the annotation generated by Confident AI.

Example: "<ANNOTATION-ID>"

fieldTypeAnnotationFieldTypeRequired

See AnnotationFieldType.

valuestring | number | boolean | string[] | nullRequired

Example: true

namestring | nullRequired

The name of the annotation.

explanationstring | nullRequired

This is the explanation for the annotation.

Example: "Correct and concise."

expectedOutcomestring | nullRequired

This is the annotated expected outcome, for conversation annotations.

expectedOutputstring | nullRequired

This is the annotated expected output, for span and trace annotations.

Example: "The capital of France is Paris."

createdAtstringRequired

The timestamp when the annotation was created.

Example: "2025-01-15T11:00:00+00:00"

userUserReference | nullRequired

See UserReference.

traceUuidstring | nullRequired

The uuid of the trace this annotation was left on, or null when it was left on a span or thread.

Example: "3f9c2a1e-5b7d-4c8e-9f01-2a3b4c5d6e7f"

spanUuidstring | nullRequired

The uuid of the span this annotation was left on, or null when it was left on a trace or thread.

threadIdstring | nullRequired

The id of the thread this annotation was left on. It is also set for an annotation on a trace that belongs to a thread.

testCaseIdstring | nullRequired

The id of the test case the annotated trace formed, for a trace ingested into a test run.

AnnotationFieldType

The kind of value an annotation holds: TEXT, NUMBER, FLOAT or BOOLEAN for a free value, CHOICE or MULTIPLE_CHOICE for a choice from the options in the field's config, and THUMBS_RATING or FIVE_STAR_RATING for a rating.

class AnnotationFieldType(Enum):
    TEXT = "TEXT"
    NUMBER = "NUMBER"
    FLOAT = "FLOAT"
    BOOLEAN = "BOOLEAN"
    CHOICE = "CHOICE"
    MULTIPLE_CHOICE = "MULTIPLE_CHOICE"
    FIVE_STAR_RATING = "FIVE_STAR_RATING"
    THUMBS_RATING = "THUMBS_RATING"

enum AnnotationFieldType {
  TEXT = "TEXT",
  NUMBER = "NUMBER",
  FLOAT = "FLOAT",
  BOOLEAN = "BOOLEAN",
  CHOICE = "CHOICE",
  MULTIPLE_CHOICE = "MULTIPLE_CHOICE",
  FIVE_STAR_RATING = "FIVE_STAR_RATING",
  THUMBS_RATING = "THUMBS_RATING",
}

TEXT · NUMBER · FLOAT · BOOLEAN · CHOICE · MULTIPLE_CHOICE · FIVE_STAR_RATING · THUMBS_RATING

AnnotationList

One page of annotations, with the total across all pages.

class AnnotationList:
    annotations: List[Annotation]
    total_annotations: int = Field(alias="totalAnnotations")
    page: int
    page_size: int = Field(alias="pageSize")

annotationsList[Annotation]Required

The annotations for the current page.

See Annotation.

total_annotationsintRequired

The total number of annotations matching the query across all pages.

Example: 1

pageintRequired

The page this response covers.

Example: 1

page_sizeintRequired

The number of annotations per page.

Example: 25

interface AnnotationList {
  annotations: Annotation[];
  totalAnnotations: number;
  page: number;
  pageSize: number;
}

annotationsAnnotation[]Required

The annotations for the current page.

See Annotation.

totalAnnotationsnumberRequired

The total number of annotations matching the query across all pages.

Example: 1

pagenumberRequired

The page this response covers.

Example: 1

pageSizenumberRequired

The number of annotations per page.

Example: 25

AnnotationRef

A reference to an annotation by its id.

class AnnotationRef:
    id: str

idstrRequired

The id of the annotation, generated by Confident AI.

Example: "<ANNOTATION-ID>"

interface AnnotationRef {
  id: string;
}

idstringRequired

The id of the annotation, generated by Confident AI.

Example: "<ANNOTATION-ID>"

AnnotationSortBy

The annotation field to sort by: createdAt orders by when the annotation was left, rating by the score a THUMBS_RATING or FIVE_STAR_RATING carries.

class AnnotationSortBy(Enum):
    CREATEDAT = "createdAt"
    RATING = "rating"

enum AnnotationSortBy {
  CREATEDAT = "createdAt",
  RATING = "rating",
}

CREATEDAT · RATING

AnnotationValue

AnnotationValue = Union[str, float, bool, List[str]]

type AnnotationValue = string | number | boolean | string[];

One of .

CreateAnnotationRequest

An annotation to record against exactly one target. Send traceUuid, spanUuid or threadId — never more than one — and the field that goes with it: expectedOutput for a trace or span, expectedOutcome for a thread. fieldType defaults to THUMBS_RATING.

CreateAnnotationRequest = Union[
    TraceAnnotationRequest,
    SpanAnnotationRequest,
    ThreadAnnotationRequest,
]

type CreateAnnotationRequest =
  | TraceAnnotationRequest
  | SpanAnnotationRequest
  | ThreadAnnotationRequest;

A CreateAnnotationRequest is one of the shapes below. Send the fields of one of them, never a mix of both.

An annotation left on a trace.

class TraceAnnotationRequest:
    trace_uuid: str = Field(alias="traceUuid")
    expected_output: Optional[str] = Field(default=None, alias="expectedOutput")
    field_type: Optional[AnnotationFieldType] = Field(default=None, alias="fieldType")
    value: AnnotationValue
    name: Optional[str] = None
    explanation: Optional[str] = None
    user_id: Optional[str] = Field(default=None, alias="userId")
    images_mapping: Optional[Dict[str, MLLMImage]] = Field(default=None, alias="imagesMapping")

trace_uuidstrRequired

The uuid of the trace being annotated.

Example: "3f9c2a1e-5b7d-4c8e-9f01-2a3b4c5d6e7f"

expected_outputOptional[str]

The output the trace should have produced.

Example: "Mount Everest is 8,848 metres tall."

field_typeOptional[AnnotationFieldType]

See AnnotationFieldType.

valueAnnotationValueRequired

See AnnotationValue.

nameOptional[str]

A name for what this annotation is for. Omit it for the default annotation.

Example: "Helpfulness"

explanationOptional[str]

Why the annotation was given.

Example: "Answered the question and cited the right source."

user_idOptional[str]

The id of the end user this annotation came from, when it was collected from your own users rather than your team.

Example: "end-user-42"

images_mappingOptional[Dict[str, MLLMImage]]

Images referenced by [DEEPEVAL:IMAGE:<key>] markers in the text fields, keyed by that marker's key.

See MLLMImage.

interface TraceAnnotationRequest {
  traceUuid: string;
  expectedOutput?: string;
  fieldType?: AnnotationFieldType;
  value: AnnotationValue;
  name?: string;
  explanation?: string;
  userId?: string;
  imagesMapping?: Record<string, MLLMImage>;
}

traceUuidstringRequired

The uuid of the trace being annotated.

Example: "3f9c2a1e-5b7d-4c8e-9f01-2a3b4c5d6e7f"

expectedOutputstring

The output the trace should have produced.

Example: "Mount Everest is 8,848 metres tall."

fieldTypeAnnotationFieldType

See AnnotationFieldType.

valueAnnotationValueRequired

See AnnotationValue.

namestring

A name for what this annotation is for. Omit it for the default annotation.

Example: "Helpfulness"

explanationstring

Why the annotation was given.

Example: "Answered the question and cited the right source."

userIdstring

The id of the end user this annotation came from, when it was collected from your own users rather than your team.

Example: "end-user-42"

imagesMappingRecord<string, MLLMImage>

Images referenced by [DEEPEVAL:IMAGE:<key>] markers in the text fields, keyed by that marker's key.

See MLLMImage.

An annotation left on a span.

class SpanAnnotationRequest:
    span_uuid: str = Field(alias="spanUuid")
    expected_output: Optional[str] = Field(default=None, alias="expectedOutput")
    field_type: Optional[AnnotationFieldType] = Field(default=None, alias="fieldType")
    value: AnnotationValue
    name: Optional[str] = None
    explanation: Optional[str] = None
    user_id: Optional[str] = Field(default=None, alias="userId")
    images_mapping: Optional[Dict[str, MLLMImage]] = Field(default=None, alias="imagesMapping")

span_uuidstrRequired

The uuid of the span being annotated.

Example: "9d2c6f7e-1a3b-4c5d-8e9f-0a1b2c3d4e5f"

expected_outputOptional[str]

The output the span should have produced.

Example: "Mount Everest is 8,848 metres tall."

field_typeOptional[AnnotationFieldType]

See AnnotationFieldType.

valueAnnotationValueRequired

See AnnotationValue.

nameOptional[str]

A name for what this annotation is for. Omit it for the default annotation.

Example: "Helpfulness"

explanationOptional[str]

Why the annotation was given.

Example: "Answered the question and cited the right source."

user_idOptional[str]

The id of the end user this annotation came from, when it was collected from your own users rather than your team.

Example: "end-user-42"

images_mappingOptional[Dict[str, MLLMImage]]

Images referenced by [DEEPEVAL:IMAGE:<key>] markers in the text fields, keyed by that marker's key.

See MLLMImage.

interface SpanAnnotationRequest {
  spanUuid: string;
  expectedOutput?: string;
  fieldType?: AnnotationFieldType;
  value: AnnotationValue;
  name?: string;
  explanation?: string;
  userId?: string;
  imagesMapping?: Record<string, MLLMImage>;
}

spanUuidstringRequired

The uuid of the span being annotated.

Example: "9d2c6f7e-1a3b-4c5d-8e9f-0a1b2c3d4e5f"

expectedOutputstring

The output the span should have produced.

Example: "Mount Everest is 8,848 metres tall."

fieldTypeAnnotationFieldType

See AnnotationFieldType.

valueAnnotationValueRequired

See AnnotationValue.

namestring

A name for what this annotation is for. Omit it for the default annotation.

Example: "Helpfulness"

explanationstring

Why the annotation was given.

Example: "Answered the question and cited the right source."

userIdstring

The id of the end user this annotation came from, when it was collected from your own users rather than your team.

Example: "end-user-42"

imagesMappingRecord<string, MLLMImage>

Images referenced by [DEEPEVAL:IMAGE:<key>] markers in the text fields, keyed by that marker's key.

See MLLMImage.

An annotation left on a thread.

class ThreadAnnotationRequest:
    thread_id: str = Field(alias="threadId")
    expected_outcome: Optional[str] = Field(default=None, alias="expectedOutcome")
    field_type: Optional[AnnotationFieldType] = Field(default=None, alias="fieldType")
    value: AnnotationValue
    name: Optional[str] = None
    explanation: Optional[str] = None
    user_id: Optional[str] = Field(default=None, alias="userId")
    images_mapping: Optional[Dict[str, MLLMImage]] = Field(default=None, alias="imagesMapping")

thread_idstrRequired

The id of the thread being annotated.

Example: "thread-42"

expected_outcomeOptional[str]

The outcome the conversation should have reached.

Example: "The user learns how tall Mount Everest is."

field_typeOptional[AnnotationFieldType]

See AnnotationFieldType.

valueAnnotationValueRequired

See AnnotationValue.

nameOptional[str]

A name for what this annotation is for. Omit it for the default annotation.

Example: "Helpfulness"

explanationOptional[str]

Why the annotation was given.

Example: "Answered the question and cited the right source."

user_idOptional[str]

The id of the end user this annotation came from, when it was collected from your own users rather than your team.

Example: "end-user-42"

images_mappingOptional[Dict[str, MLLMImage]]

Images referenced by [DEEPEVAL:IMAGE:<key>] markers in the text fields, keyed by that marker's key.

See MLLMImage.

interface ThreadAnnotationRequest {
  threadId: string;
  expectedOutcome?: string;
  fieldType?: AnnotationFieldType;
  value: AnnotationValue;
  name?: string;
  explanation?: string;
  userId?: string;
  imagesMapping?: Record<string, MLLMImage>;
}

threadIdstringRequired

The id of the thread being annotated.

Example: "thread-42"

expectedOutcomestring

The outcome the conversation should have reached.

Example: "The user learns how tall Mount Everest is."

fieldTypeAnnotationFieldType

See AnnotationFieldType.

valueAnnotationValueRequired

See AnnotationValue.

namestring

A name for what this annotation is for. Omit it for the default annotation.

Example: "Helpfulness"

explanationstring

Why the annotation was given.

Example: "Answered the question and cited the right source."

userIdstring

The id of the end user this annotation came from, when it was collected from your own users rather than your team.

Example: "end-user-42"

imagesMappingRecord<string, MLLMImage>

Images referenced by [DEEPEVAL:IMAGE:<key>] markers in the text fields, keyed by that marker's key.

See MLLMImage.

MLLMImage

An image referenced from a text field by a [DEEPEVAL:IMAGE:<key>] marker. Send either a public url or the bytes in base64.

class MLLMImage:
    url: str
    local: bool
    base64: Optional[str] = None
    filename: Optional[str] = None
    mime_type: Optional[str] = Field(default=None, alias="mimeType")
    data_base64: Optional[str] = Field(default=None, alias="dataBase64")

urlstrRequired

This is the URL of the image.

Example: "https://example.com/everest.png"

localboolRequired

This is true when the image is your local file.

Example: false

base64Optional[str]

The base64 data of the image.

Example: "iVBORw0KGgo="

filenameOptional[str]

The original file name.

Example: "everest.png"

mime_typeOptional[str]

The image's MIME type.

Example: "image/png"

data_base64Optional[str]

The image encoded as a base64 data URL.

Example: "data:image/png;base64,iVBORw0KGgo="

interface MLLMImage {
  url: string;
  local: boolean;
  base64?: string;
  filename?: string;
  mimeType?: string;
  dataBase64?: string;
}

urlstringRequired

This is the URL of the image.

Example: "https://example.com/everest.png"

localbooleanRequired

This is true when the image is your local file.

Example: false

base64string

The base64 data of the image.

Example: "iVBORw0KGgo="

filenamestring

The original file name.

Example: "everest.png"

mimeTypestring

The image's MIME type.

Example: "image/png"

dataBase64string

The image encoded as a base64 data URL.

Example: "data:image/png;base64,iVBORw0KGgo="

UserReference

A Confident AI user, as referenced by the records they created.

class UserReference:
    id: str
    email: str
    name: Optional[str]
    image: Optional[str]

idstrRequired

This is the id of the user.

Example: "<USER-ID>"

emailstrRequired

This is the email address of the user.

Example: "jane@acme.com"

nameOptional[str]Required

This is the display name of the user, or null when they have not set one.

Example: "Jane Doe"

imageOptional[str]Required

This is the URL of the user's avatar, or null when they have none.

interface UserReference {
  id: string;
  email: string;
  name: string | null;
  image: string | null;
}

idstringRequired

This is the id of the user.

Example: "<USER-ID>"

emailstringRequired

This is the email address of the user.

Example: "jane@acme.com"

namestring | nullRequired

This is the display name of the user, or null when they have not set one.

Example: "Jane Doe"

imagestring | nullRequired

This is the URL of the user's avatar, or null when they have none.

Building a production pipeline?Design a scalable API workflow for evals, datasets, traces, and promptsTalk to an engineer

Last updated on September 30, 2026

Built byConfident AI