Managing Datasets via Confident AI SDK

Every Datasets method in the Confident AI Python and TypeScript SDKs.

Overview

The Confident AI SDK exposes every Dataset method on the platform. This page documents how to call these methods in all supported languages. See the introduction to install the SDK and set your API key.

Dataset

client.dataset() returns a Dataset object that stands for one dataset. This object stores the fields listed below, and passes the dataset's id to every method called on it, so you don't need to pass the id nor the stored fields as arguments.

push is the exception, because it sends the stored fields rather than calling a route that names the id. Open the object with alias to call it.

from confident_ai import ConfidentAI

client = ConfidentAI()

dataset = client.dataset(dataset_id="<DATASET-ID>")

Properties

These are the fields a Dataset stores. A method that loads the dataset fills them in, and a method that saves it sends whichever of them you have set, so set them before you save and read them after you load.

Parameter	Type	Description
`dataset_id`	`Optional[str]`	The unique id of the dataset.
`id`	`Optional[str]`	This is the unique id of the dataset.
`alias`	`Optional[str]`	This is the alias of the dataset, which is unique within your project.
`multi_turn`	`Optional[bool]`	This is `true` if the dataset is multi-turn, which contains multi-turn goldens. Single-turn datasets have `multiTurn` set to `false` and contain single-turn goldens.
`version`	`Optional[str]`	The version number of the goldens returned, or `null` when the dataset has no versions.
`goldens`	`Optional[List[Golden]]`	The goldens in the dataset, oldest first. Every golden is single-turn or multi-turn according to the dataset's `multiTurn`. See Golden.

import { ConfidentAI } from "confident-ai";

const client = new ConfidentAI();

const dataset = client.dataset("<DATASET-ID>");

Properties

Parameter	Type	Description
`datasetId`	`string`	The unique id of the dataset.
`id`	`string`	This is the unique id of the dataset.
`alias`	`string`	This is the alias of the dataset, which is unique within your project.
`multiTurn`	`boolean`	This is `true` if the dataset is multi-turn, which contains multi-turn goldens. Single-turn datasets have `multiTurn` set to `false` and contain single-turn goldens.
`version`	`string \| null`	The version number of the goldens returned, or `null` when the dataset has no versions.
`goldens`	`Golden[]`	The goldens in the dataset, oldest first. Every golden is single-turn or multi-turn according to the dataset's `multiTurn`. See Golden.

Methods

Delete Dataset

Permanently deletes the dataset and everything in it: its goldens, versions, tags and custom columns. This action cannot be undone.

from confident_ai import ConfidentAI

client = ConfidentAI()

dataset = client.dataset(dataset_id="<DATASET-ID>")
result = dataset.delete()

For async mode, call a_delete and await it as shown below:

result = await dataset.a_delete(...)

import { ConfidentAI } from "confident-ai";

const client = new ConfidentAI();

const dataset = client.dataset("<DATASET-ID>");
const result = await dataset.delete();

Returns

This method returns an object of type DatasetRef.

Run Evaluation

Starts an evaluation of the dataset's finalized goldens against a metric collection and returns the test run it is evaluated in. It runs asynchronously, so this returns as soon as the run is created. By default the goldens' stored actual outputs are evaluated; supply aiConnectionId or promptAlias, never both, to generate them first.

from confident_ai import ConfidentAI
from confident_ai.common import GenerationMode

client = ConfidentAI()

dataset = client.dataset(dataset_id="<DATASET-ID>")
result = dataset.run_evaluation(
    metric_collection="Answer Quality",
    identifier="Nightly regression",
    version="00.00.01",
    ai_connection_id="<AI-CONNECTION-ID>",
    voice_connection_id="<VOICE-CONNECTION-ID>",
    prompt_alias="capital-lookup",
    prompt_commit="bab04ce",
    generation_mode=GenerationMode.AI_CONNECTION,
    variables_mapping={"question": "Input"},
    include_simulation=False,
    max_concurrent_generation=5,
    generation_timeout=60,
    num_generations=1,
    mcp_server_ids=["<MCP-SERVER-ID>"],
)

For async mode, call a_run_evaluation and await it as shown below:

result = await dataset.a_run_evaluation(...)

Parameters

Parameter	Type	Description
`metric_collection`	`str`	Required. The name of the metric collection to evaluate against. Names come from the list metric collections endpoint.
`identifier`	`Optional[str]`	A label for the resulting test run, used to recognise it in the test runs list.
`version`	`Optional[str]`	The dataset version to evaluate. Omit this field to evaluate the latest version.
`ai_connection_id`	`Optional[str]`	The id of the AI connection used to generate the actual outputs before evaluating them. Required when `generationMode` is `AI_CONNECTION`, and not allowed together with `promptAlias`.
`voice_connection_id`	`Optional[str]`	The id of the voice connection to call while simulating each golden's conversation. Required when `generationMode` is `VOICE_CONNECTION`, needs a multi-turn dataset with `includeSimulation` enabled, and is not allowed together with `aiConnectionId` or `promptAlias`.
`prompt_alias`	`Optional[str]`	The alias of the prompt used to generate the actual outputs before evaluating them. Required when `generationMode` is `PROMPT`, and not allowed together with `aiConnectionId`.
`prompt_commit`	`Optional[str]`	The prompt commit hash to generate with. Requires `promptAlias`. Omit this field to generate with the latest commit on the prompt's main branch.
`generation_mode`	`Optional[GenerationMode]`	See GenerationMode.
`variables_mapping`	`Optional[Dict[str, str]]`	Maps each variable in the prompt to the golden field it is interpolated with, such as `Input` or `Expected Output`, or to a dataset custom column key. This field applies only when generating from a prompt.
`include_simulation`	`Optional[bool]`	Whether to simulate a conversation for each golden before evaluating it, for multi-turn datasets. Every golden needs a `scenario` when this is enabled, and `turns` when it is disabled.
`max_concurrent_generation`	`Optional[int]`	The maximum number of generation calls to run in parallel. An AI connection's own `maxConcurrency` takes precedence over this value.
`generation_timeout`	`Optional[int]`	The number of seconds to wait for a single generation before it is marked as errored.
`num_generations`	`Optional[int]`	How many times to run each golden, so a single outlier response does not skew the results. Omit this field to use the AI connection's `defaultNumGenerations`, which is 1 when the connection does not set one.
`mcp_server_ids`	`Optional[List[str]]`	The ids of the MCP servers to attach to the run. A tool call whose name matches a tool exposed by one of these servers is labeled an MCP tool call rather than a function call.

import { ConfidentAI } from "confident-ai";
import { GenerationMode } from "confident-ai/common";

const client = new ConfidentAI();

const dataset = client.dataset("<DATASET-ID>");
const result = await dataset.runEvaluation(
  "Answer Quality",
  {
    identifier: "Nightly regression",
    version: "00.00.01",
    aiConnectionId: "<AI-CONNECTION-ID>",
    voiceConnectionId: "<VOICE-CONNECTION-ID>",
    promptAlias: "capital-lookup",
    promptCommit: "bab04ce",
    generationMode: GenerationMode.AI_CONNECTION,
    variablesMapping: { question: "Input" },
    includeSimulation: false,
    maxConcurrentGeneration: 5,
    generationTimeout: 60,
    numGenerations: 1,
    mcpServerIds: ["<MCP-SERVER-ID>"]
  },
);

Parameters

Parameter	Type	Description
`metricCollection`	`string`	Required. The name of the metric collection to evaluate against. Names come from the list metric collections endpoint.
`identifier`	`string`	A label for the resulting test run, used to recognise it in the test runs list.
`version`	`string`	The dataset version to evaluate. Omit this field to evaluate the latest version.
`aiConnectionId`	`string`	The id of the AI connection used to generate the actual outputs before evaluating them. Required when `generationMode` is `AI_CONNECTION`, and not allowed together with `promptAlias`.
`voiceConnectionId`	`string`	The id of the voice connection to call while simulating each golden's conversation. Required when `generationMode` is `VOICE_CONNECTION`, needs a multi-turn dataset with `includeSimulation` enabled, and is not allowed together with `aiConnectionId` or `promptAlias`.
`promptAlias`	`string`	The alias of the prompt used to generate the actual outputs before evaluating them. Required when `generationMode` is `PROMPT`, and not allowed together with `aiConnectionId`.
`promptCommit`	`string`	The prompt commit hash to generate with. Requires `promptAlias`. Omit this field to generate with the latest commit on the prompt's main branch.
`generationMode`	`GenerationMode`	See GenerationMode.
`variablesMapping`	`Record<string, string>`	Maps each variable in the prompt to the golden field it is interpolated with, such as `Input` or `Expected Output`, or to a dataset custom column key. This field applies only when generating from a prompt.
`includeSimulation`	`boolean`	Whether to simulate a conversation for each golden before evaluating it, for multi-turn datasets. Every golden needs a `scenario` when this is enabled, and `turns` when it is disabled.
`maxConcurrentGeneration`	`number`	The maximum number of generation calls to run in parallel. An AI connection's own `maxConcurrency` takes precedence over this value.
`generationTimeout`	`number`	The number of seconds to wait for a single generation before it is marked as errored.
`numGenerations`	`number`	How many times to run each golden, so a single outlier response does not skew the results. Omit this field to use the AI connection's `defaultNumGenerations`, which is 1 when the connection does not set one.
`mcpServerIds`	`string[]`	The ids of the MCP servers to attach to the run. A tool call whose name matches a tool exposed by one of these servers is labeled an MCP tool call rather than a function call.

Returns

This method returns an object of type RunDatasetEvaluationResult.

Single Turn Golden

Adds goldens to the dataset as unfinalized goldens, for review on the platform before they are used in evaluations. They go into the dataset's latest version, or among its unversioned goldens when it has no versions.

A single-turn golden as stored in the dataset.

from confident_ai import ConfidentAI
from confident_ai.common import ToolCall, ToolCallType
from confident_ai.datasets import Expectations, SingleTurnGolden

client = ConfidentAI()

dataset = client.dataset(dataset_id="<DATASET-ID>")
result = dataset.queue_goldens(
    goldens=[
        SingleTurnGolden(
            input="What is the capital of France?",
            actual_output="The capital of France is Paris.",
            expected_output="Paris.",
            context=["Paris is the capital of France."],
            retrieval_context=[
                "Paris is the capital and largest city of France."
            ],
            tools_called=[
                ToolCall(
                    name="get_landmark_info",
                    type=ToolCallType.FUNCTION,
                    description="This tool gives information about a mountain.",
                    input_parameters={"mountain": "Everest"},
                    output="8,848 metres",
                    reasoning="The user asked for the height of a mountain."
                )
            ],
            expected_tools=[
                ToolCall(
                    name="get_landmark_info",
                    type=ToolCallType.FUNCTION,
                    description="This tool gives information about a mountain.",
                    input_parameters={"mountain": "Everest"},
                    output="8,848 metres",
                    reasoning="The user asked for the height of a mountain."
                )
            ],
            token_cost=0.002,
            input_token_count=12,
            output_token_count=3,
            id="<GOLDEN-ID>",
            additional_metadata={"source": "faq"},
            expectations=Expectations(
                must=["Cite a source"],
                must_not=["Mention competitors"]
            ),
            comments="Reviewed by the support team.",
            source_file="capitals.csv",
            source_files=["capitals.csv"],
            finalized=True,
            custom_column_key_values={"difficulty": "easy"},
            tags=["geography"],
            images_mapping={
                "map": {"url": "https://example.com/paris.png", "local": False}
            }
        )
    ],
)

For async mode, call a_queue_goldens and await it as shown below:

result = await dataset.a_queue_goldens(...)

Parameters

Parameter	Type	Description
`goldens`	`List[Golden]`	Required. The goldens to queue for review. Every golden in one request must be of the same kind and match the dataset's `multiTurn`. They are stored unfinalized, whatever each golden's own `finalized` says. See Golden.

import { ConfidentAI } from "confident-ai";
import { ToolCallType } from "confident-ai/common";

const client = new ConfidentAI();

const dataset = client.dataset("<DATASET-ID>");
const result = await dataset.queueGoldens(
  [
    {
      input: "What is the capital of France?",
      actualOutput: "The capital of France is Paris.",
      expectedOutput: "Paris.",
      context: ["Paris is the capital of France."],
      retrievalContext: [
          "Paris is the capital and largest city of France."
        ],
      toolsCalled: [
          {
            name: "get_landmark_info",
            type: ToolCallType.FUNCTION,
            description: "This tool gives information about a mountain.",
            inputParameters: { mountain: "Everest" },
            output: "8,848 metres",
            reasoning: "The user asked for the height of a mountain."
          }
        ],
      expectedTools: [
          {
            name: "get_landmark_info",
            type: ToolCallType.FUNCTION,
            description: "This tool gives information about a mountain.",
            inputParameters: { mountain: "Everest" },
            output: "8,848 metres",
            reasoning: "The user asked for the height of a mountain."
          }
        ],
      tokenCost: 0.002,
      inputTokenCount: 12,
      outputTokenCount: 3,
      id: "<GOLDEN-ID>",
      additionalMetadata: { source: "faq" },
      expectations: {
          must: ["Cite a source"],
          mustNot: ["Mention competitors"]
        },
      comments: "Reviewed by the support team.",
      sourceFile: "capitals.csv",
      sourceFiles: ["capitals.csv"],
      finalized: true,
      customColumnKeyValues: { difficulty: "easy" },
      tags: ["geography"],
      imagesMapping: {
          map: { url: "https://example.com/paris.png", local: false }
        }
    }
  ],
);

Parameters

Parameter	Type	Description
`goldens`	`Golden[]`	Required. The goldens to queue for review. Every golden in one request must be of the same kind and match the dataset's `multiTurn`. They are stored unfinalized, whatever each golden's own `finalized` says. See Golden.

Returns

This method returns an object of type DatasetRef.

Multi Turn Golden

A multi-turn golden as stored in the dataset.

from confident_ai import ConfidentAI
from confident_ai.common import Turn, TurnRole
from confident_ai.datasets import Expectations, MultiTurnGolden

client = ConfidentAI()

dataset = client.dataset(dataset_id="<DATASET-ID>")
result = dataset.queue_goldens(
    goldens=[
        MultiTurnGolden(
            scenario="A traveller wants to book a hotel in Paris.",
            expected_outcome="The assistant confirms a reservation near the Louvre.",
            user_description="A traveller planning a weekend in Paris.",
            turns=[
                Turn(
                    role=TurnRole.USER,
                    content="I need a hotel in Paris near the Louvre."
                ),
                Turn(
                    role=TurnRole.ASSISTANT,
                    content="Hôtel du Louvre has rooms available. Which dates?"
                )
            ],
            context=[
                "Hôtel du Louvre is a five-minute walk from the museum."
            ],
            id="<GOLDEN-ID>",
            additional_metadata={"source": "faq"},
            expectations=Expectations(
                must=["Cite a source"],
                must_not=["Mention competitors"]
            ),
            comments="Reviewed by the support team.",
            source_file="capitals.csv",
            source_files=["capitals.csv"],
            finalized=True,
            custom_column_key_values={"difficulty": "easy"},
            tags=["geography"],
            images_mapping={
                "map": {"url": "https://example.com/paris.png", "local": False}
            }
        )
    ],
)

For async mode, call a_queue_goldens and await it as shown below:

result = await dataset.a_queue_goldens(...)

Parameters

Parameter	Type	Description
`goldens`	`List[Golden]`	Required. The goldens to queue for review. Every golden in one request must be of the same kind and match the dataset's `multiTurn`. They are stored unfinalized, whatever each golden's own `finalized` says. See Golden.

import { ConfidentAI } from "confident-ai";
import { TurnRole } from "confident-ai/common";

const client = new ConfidentAI();

const dataset = client.dataset("<DATASET-ID>");
const result = await dataset.queueGoldens(
  [
    {
      scenario: "A traveller wants to book a hotel in Paris.",
      expectedOutcome: "The assistant confirms a reservation near the Louvre.",
      userDescription: "A traveller planning a weekend in Paris.",
      turns: [
          {
            role: TurnRole.USER,
            content: "I need a hotel in Paris near the Louvre."
          },
          {
            role: TurnRole.ASSISTANT,
            content: "Hôtel du Louvre has rooms available. Which dates?"
          }
        ],
      context: ["Hôtel du Louvre is a five-minute walk from the museum."],
      id: "<GOLDEN-ID>",
      additionalMetadata: { source: "faq" },
      expectations: {
          must: ["Cite a source"],
          mustNot: ["Mention competitors"]
        },
      comments: "Reviewed by the support team.",
      sourceFile: "capitals.csv",
      sourceFiles: ["capitals.csv"],
      finalized: true,
      customColumnKeyValues: { difficulty: "easy" },
      tags: ["geography"],
      imagesMapping: {
          map: { url: "https://example.com/paris.png", local: false }
        }
    }
  ],
);

Parameters

Parameter	Type	Description
`goldens`	`Golden[]`	Required. The goldens to queue for review. Every golden in one request must be of the same kind and match the dataset's `multiTurn`. They are stored unfinalized, whatever each golden's own `finalized` says. See Golden.

Returns

This method returns an object of type DatasetRef.

Pull Dataset

Retrieves the dataset with its goldens, oldest first. Pass version to pull a specific version, and finalized=false to pull the goldens still awaiting review instead of the finalized ones. Requires an active trial or paid plan, and the Team plan or above to pull a version.

from confident_ai import ConfidentAI

client = ConfidentAI()

dataset = client.dataset(dataset_id="<DATASET-ID>")
result = dataset.pull(version="00.00.01", finalized="true")

For async mode, call a_pull and await it as shown below:

result = await dataset.a_pull(...)

Parameters

Parameter	Type	Description
`version`	`Optional[str]`	The version to pull. Defaults to the latest version, or to the unversioned goldens when the dataset has no versions. Requires the Team plan or above.
`finalized`	`Optional[Literal['true', 'false']]`	Whether to pull the finalized goldens, or with `false` the goldens still awaiting review. Defaults to `true`.

import { ConfidentAI } from "confident-ai";

const client = new ConfidentAI();

const dataset = client.dataset("<DATASET-ID>");
const result = await dataset.pull(
  { version: "00.00.01", finalized: "true" },
);

Parameters

Parameter	Type	Description
`version`	`string`	The version to pull. Defaults to the latest version, or to the unversioned goldens when the dataset has no versions. Requires the Team plan or above.
`finalized`	`"true" \| "false"`	Whether to pull the finalized goldens, or with `false` the goldens still awaiting review. Defaults to `true`.

Returns

This method returns an object of type Dataset.

Push Dataset

Adds goldens to a dataset, creating it when the alias names none yet, and returns the dataset's id. Pushing to a version requires the Team plan or above.

from confident_ai import ConfidentAI
from confident_ai.common import ToolCall, ToolCallType
from confident_ai.datasets import Expectations, SingleTurnGolden

client = ConfidentAI()

dataset = client.dataset(alias="capitals")
dataset.goldens = [
    SingleTurnGolden(
        input="What is the capital of France?",
        actual_output="The capital of France is Paris.",
        expected_output="Paris.",
        context=["Paris is the capital of France."],
        retrieval_context=[
            "Paris is the capital and largest city of France."
        ],
        tools_called=[
            ToolCall(
                name="get_landmark_info",
                type=ToolCallType.FUNCTION,
                description="This tool gives information about a mountain.",
                input_parameters={"mountain": "Everest"},
                output="8,848 metres",
                reasoning="The user asked for the height of a mountain."
            )
        ],
        expected_tools=[
            ToolCall(
                name="get_landmark_info",
                type=ToolCallType.FUNCTION,
                description="This tool gives information about a mountain.",
                input_parameters={"mountain": "Everest"},
                output="8,848 metres",
                reasoning="The user asked for the height of a mountain."
            )
        ],
        token_cost=0.002,
        input_token_count=12,
        output_token_count=3,
        id="<GOLDEN-ID>",
        additional_metadata={"source": "faq"},
        expectations=Expectations(
            must=["Cite a source"],
            must_not=["Mention competitors"]
        ),
        comments="Reviewed by the support team.",
        source_file="capitals.csv",
        source_files=["capitals.csv"],
        finalized=True,
        custom_column_key_values={"difficulty": "easy"},
        tags=["geography"],
        images_mapping={
            "map": {"url": "https://example.com/paris.png", "local": False}
        }
    )
]
result = dataset.push(finalized=True)

For async mode, call a_push and await it as shown below:

result = await dataset.a_push(...)

Parameters

Parameter	Type	Description
`finalized`	`Optional[bool]`	Whether the goldens pushed are finalized, that is ready to use in evaluations. Applies to every golden in this request.

import { ConfidentAI } from "confident-ai";
import { ToolCallType } from "confident-ai/common";

const client = new ConfidentAI();

const dataset = client.dataset(undefined, { alias: "capitals" });
dataset.goldens = [
  {
    input: "What is the capital of France?",
    actualOutput: "The capital of France is Paris.",
    expectedOutput: "Paris.",
    context: ["Paris is the capital of France."],
    retrievalContext: [
        "Paris is the capital and largest city of France."
      ],
    toolsCalled: [
        {
          name: "get_landmark_info",
          type: ToolCallType.FUNCTION,
          description: "This tool gives information about a mountain.",
          inputParameters: { mountain: "Everest" },
          output: "8,848 metres",
          reasoning: "The user asked for the height of a mountain."
        }
      ],
    expectedTools: [
        {
          name: "get_landmark_info",
          type: ToolCallType.FUNCTION,
          description: "This tool gives information about a mountain.",
          inputParameters: { mountain: "Everest" },
          output: "8,848 metres",
          reasoning: "The user asked for the height of a mountain."
        }
      ],
    tokenCost: 0.002,
    inputTokenCount: 12,
    outputTokenCount: 3,
    id: "<GOLDEN-ID>",
    additionalMetadata: { source: "faq" },
    expectations: {
        must: ["Cite a source"],
        mustNot: ["Mention competitors"]
      },
    comments: "Reviewed by the support team.",
    sourceFile: "capitals.csv",
    sourceFiles: ["capitals.csv"],
    finalized: true,
    customColumnKeyValues: { difficulty: "easy" },
    tags: ["geography"],
    imagesMapping: {
        map: { url: "https://example.com/paris.png", local: false }
      }
  }
];
const result = await dataset.push({ finalized: true });

Parameters

Parameter	Type	Description
`finalized`	`boolean`	Whether the goldens pushed are finalized, that is ready to use in evaluations. Applies to every golden in this request.

Returns

This method returns an object of type DatasetRef.

Methods (Stateless)

These methods take every argument themselves, so a caller reaches them through client.datasets without opening a Dataset first.

List Datasets

Lists all the datasets in your Confident AI project, newest first, without their goldens.

from confident_ai import ConfidentAI

client = ConfidentAI()

result = client.datasets.list()

For async mode, call a_list and await it as shown below:

result = await client.datasets.a_list(...)

import { ConfidentAI } from "confident-ai";

const client = new ConfidentAI();

const result = await client.datasets.list();

Returns

This method returns an object of type DatasetList.

Push Dataset

Adds goldens to a dataset, creating it when the alias names none yet, and returns the dataset's id. Pushing to a version requires the Team plan or above.

from confident_ai import ConfidentAI
from confident_ai.common import ToolCall, ToolCallType
from confident_ai.datasets import Expectations, SingleTurnGolden

client = ConfidentAI()

result = client.datasets.push(
    alias="capitals",
    goldens=[
        SingleTurnGolden(
            input="What is the capital of France?",
            actual_output="The capital of France is Paris.",
            expected_output="Paris.",
            context=["Paris is the capital of France."],
            retrieval_context=[
                "Paris is the capital and largest city of France."
            ],
            tools_called=[
                ToolCall(
                    name="get_landmark_info",
                    type=ToolCallType.FUNCTION,
                    description="This tool gives information about a mountain.",
                    input_parameters={"mountain": "Everest"},
                    output="8,848 metres",
                    reasoning="The user asked for the height of a mountain."
                )
            ],
            expected_tools=[
                ToolCall(
                    name="get_landmark_info",
                    type=ToolCallType.FUNCTION,
                    description="This tool gives information about a mountain.",
                    input_parameters={"mountain": "Everest"},
                    output="8,848 metres",
                    reasoning="The user asked for the height of a mountain."
                )
            ],
            token_cost=0.002,
            input_token_count=12,
            output_token_count=3,
            id="<GOLDEN-ID>",
            additional_metadata={"source": "faq"},
            expectations=Expectations(
                must=["Cite a source"],
                must_not=["Mention competitors"]
            ),
            comments="Reviewed by the support team.",
            source_file="capitals.csv",
            source_files=["capitals.csv"],
            finalized=True,
            custom_column_key_values={"difficulty": "easy"},
            tags=["geography"],
            images_mapping={
                "map": {"url": "https://example.com/paris.png", "local": False}
            }
        )
    ],
    finalized=True,
    version="00.00.01",
)

For async mode, call a_push and await it as shown below:

result = await client.datasets.a_push(...)

Parameters

Parameter	Type	Description
`alias`	`str`	Required. The alias of the dataset, unique within your project. A new dataset is created when no dataset with this alias exists.
`goldens`	`List[Golden]`	Required. The goldens to push. A golden carrying an `id` updates the golden already in the dataset; one without is added. Every golden in one request must be of the same kind — all single- turn, or all multi-turn — and match the dataset's `multiTurn`. A new dataset takes its kind from them. See Golden.
`finalized`	`Optional[bool]`	Whether the goldens pushed are finalized, that is ready to use in evaluations. Applies to every golden in this request.
`version`	`Optional[str]`	The dataset version to push the goldens onto, for example `00.00.01`. When the dataset has versions, omitting it pushes to the latest version; when it has none, omitting it leaves the goldens unversioned. A version cannot be given for a dataset that does not exist yet. Requires the Team plan or above.

import { ConfidentAI } from "confident-ai";
import { ToolCallType } from "confident-ai/common";

const client = new ConfidentAI();

const result = await client.datasets.push(
  "capitals",
  [
    {
      input: "What is the capital of France?",
      actualOutput: "The capital of France is Paris.",
      expectedOutput: "Paris.",
      context: ["Paris is the capital of France."],
      retrievalContext: [
          "Paris is the capital and largest city of France."
        ],
      toolsCalled: [
          {
            name: "get_landmark_info",
            type: ToolCallType.FUNCTION,
            description: "This tool gives information about a mountain.",
            inputParameters: { mountain: "Everest" },
            output: "8,848 metres",
            reasoning: "The user asked for the height of a mountain."
          }
        ],
      expectedTools: [
          {
            name: "get_landmark_info",
            type: ToolCallType.FUNCTION,
            description: "This tool gives information about a mountain.",
            inputParameters: { mountain: "Everest" },
            output: "8,848 metres",
            reasoning: "The user asked for the height of a mountain."
          }
        ],
      tokenCost: 0.002,
      inputTokenCount: 12,
      outputTokenCount: 3,
      id: "<GOLDEN-ID>",
      additionalMetadata: { source: "faq" },
      expectations: {
          must: ["Cite a source"],
          mustNot: ["Mention competitors"]
        },
      comments: "Reviewed by the support team.",
      sourceFile: "capitals.csv",
      sourceFiles: ["capitals.csv"],
      finalized: true,
      customColumnKeyValues: { difficulty: "easy" },
      tags: ["geography"],
      imagesMapping: {
          map: { url: "https://example.com/paris.png", local: false }
        }
    }
  ],
  true,
  "00.00.01",
);

Parameters

Parameter	Type	Description
`alias`	`string`	Required. The alias of the dataset, unique within your project. A new dataset is created when no dataset with this alias exists.
`goldens`	`Golden[]`	Required. The goldens to push. A golden carrying an `id` updates the golden already in the dataset; one without is added. Every golden in one request must be of the same kind — all single- turn, or all multi-turn — and match the dataset's `multiTurn`. A new dataset takes its kind from them. See Golden.
`finalized`	`boolean`	Whether the goldens pushed are finalized, that is ready to use in evaluations. Applies to every golden in this request.
`version`	`string`	The dataset version to push the goldens onto, for example `00.00.01`. When the dataset has versions, omitting it pushes to the latest version; when it has none, omitting it leaves the goldens unversioned. A version cannot be given for a dataset that does not exist yet. Requires the Team plan or above.

Returns

This method returns an object of type DatasetRef.

Pull Dataset

from confident_ai import ConfidentAI

client = ConfidentAI()

result = client.datasets.pull(
    dataset_id="<DATASET-ID>",
    version="00.00.01",
    finalized="true",
)

For async mode, call a_pull and await it as shown below:

result = await client.datasets.a_pull(...)

Parameters

Parameter	Type	Description
`dataset_id`	`str`	Required. The unique id of the dataset.
`version`	`Optional[str]`	The version to pull. Defaults to the latest version, or to the unversioned goldens when the dataset has no versions. Requires the Team plan or above.
`finalized`	`Optional[Literal['true', 'false']]`	Whether to pull the finalized goldens, or with `false` the goldens still awaiting review. Defaults to `true`.

import { ConfidentAI } from "confident-ai";

const client = new ConfidentAI();

const result = await client.datasets.pull(
  "<DATASET-ID>",
  "00.00.01",
  "true",
);

Parameters

Parameter	Type	Description
`datasetId`	`string`	Required. The unique id of the dataset.
`version`	`string`	The version to pull. Defaults to the latest version, or to the unversioned goldens when the dataset has no versions. Requires the Team plan or above.
`finalized`	`"true" \| "false"`	Whether to pull the finalized goldens, or with `false` the goldens still awaiting review. Defaults to `true`.

Returns

This method returns an object of type Dataset.

Delete Dataset

Permanently deletes the dataset and everything in it: its goldens, versions, tags and custom columns. This action cannot be undone.

from confident_ai import ConfidentAI

client = ConfidentAI()

result = client.datasets.delete(dataset_id="<DATASET-ID>")

For async mode, call a_delete and await it as shown below:

result = await client.datasets.a_delete(...)

Parameters

Parameter	Type	Description
`dataset_id`	`str`	Required. The unique id of the dataset.

import { ConfidentAI } from "confident-ai";

const client = new ConfidentAI();

const result = await client.datasets.delete("<DATASET-ID>");

Parameters

Parameter	Type	Description
`datasetId`	`string`	Required. The unique id of the dataset.

Returns

This method returns an object of type DatasetRef.

Single Turn Golden

A single-turn golden as stored in the dataset.

from confident_ai import ConfidentAI
from confident_ai.common import ToolCall, ToolCallType
from confident_ai.datasets import Expectations, SingleTurnGolden

client = ConfidentAI()

result = client.datasets.queue_goldens(
    dataset_id="<DATASET-ID>",
    goldens=[
        SingleTurnGolden(
            input="What is the capital of France?",
            actual_output="The capital of France is Paris.",
            expected_output="Paris.",
            context=["Paris is the capital of France."],
            retrieval_context=[
                "Paris is the capital and largest city of France."
            ],
            tools_called=[
                ToolCall(
                    name="get_landmark_info",
                    type=ToolCallType.FUNCTION,
                    description="This tool gives information about a mountain.",
                    input_parameters={"mountain": "Everest"},
                    output="8,848 metres",
                    reasoning="The user asked for the height of a mountain."
                )
            ],
            expected_tools=[
                ToolCall(
                    name="get_landmark_info",
                    type=ToolCallType.FUNCTION,
                    description="This tool gives information about a mountain.",
                    input_parameters={"mountain": "Everest"},
                    output="8,848 metres",
                    reasoning="The user asked for the height of a mountain."
                )
            ],
            token_cost=0.002,
            input_token_count=12,
            output_token_count=3,
            id="<GOLDEN-ID>",
            additional_metadata={"source": "faq"},
            expectations=Expectations(
                must=["Cite a source"],
                must_not=["Mention competitors"]
            ),
            comments="Reviewed by the support team.",
            source_file="capitals.csv",
            source_files=["capitals.csv"],
            finalized=True,
            custom_column_key_values={"difficulty": "easy"},
            tags=["geography"],
            images_mapping={
                "map": {"url": "https://example.com/paris.png", "local": False}
            }
        )
    ],
)

For async mode, call a_queue_goldens and await it as shown below:

result = await client.datasets.a_queue_goldens(...)

Parameters

Parameter	Type	Description
`dataset_id`	`str`	Required. The unique id of the dataset.
`goldens`	`List[Golden]`	Required. The goldens to queue for review. Every golden in one request must be of the same kind and match the dataset's `multiTurn`. They are stored unfinalized, whatever each golden's own `finalized` says. See Golden.

import { ConfidentAI } from "confident-ai";
import { ToolCallType } from "confident-ai/common";

const client = new ConfidentAI();

const result = await client.datasets.queueGoldens(
  "<DATASET-ID>",
  [
    {
      input: "What is the capital of France?",
      actualOutput: "The capital of France is Paris.",
      expectedOutput: "Paris.",
      context: ["Paris is the capital of France."],
      retrievalContext: [
          "Paris is the capital and largest city of France."
        ],
      toolsCalled: [
          {
            name: "get_landmark_info",
            type: ToolCallType.FUNCTION,
            description: "This tool gives information about a mountain.",
            inputParameters: { mountain: "Everest" },
            output: "8,848 metres",
            reasoning: "The user asked for the height of a mountain."
          }
        ],
      expectedTools: [
          {
            name: "get_landmark_info",
            type: ToolCallType.FUNCTION,
            description: "This tool gives information about a mountain.",
            inputParameters: { mountain: "Everest" },
            output: "8,848 metres",
            reasoning: "The user asked for the height of a mountain."
          }
        ],
      tokenCost: 0.002,
      inputTokenCount: 12,
      outputTokenCount: 3,
      id: "<GOLDEN-ID>",
      additionalMetadata: { source: "faq" },
      expectations: {
          must: ["Cite a source"],
          mustNot: ["Mention competitors"]
        },
      comments: "Reviewed by the support team.",
      sourceFile: "capitals.csv",
      sourceFiles: ["capitals.csv"],
      finalized: true,
      customColumnKeyValues: { difficulty: "easy" },
      tags: ["geography"],
      imagesMapping: {
          map: { url: "https://example.com/paris.png", local: false }
        }
    }
  ],
);

Parameters

Parameter	Type	Description
`datasetId`	`string`	Required. The unique id of the dataset.
`goldens`	`Golden[]`	Required. The goldens to queue for review. Every golden in one request must be of the same kind and match the dataset's `multiTurn`. They are stored unfinalized, whatever each golden's own `finalized` says. See Golden.

Returns

This method returns an object of type DatasetRef.

Multi Turn Golden

A multi-turn golden as stored in the dataset.

from confident_ai import ConfidentAI
from confident_ai.common import Turn, TurnRole
from confident_ai.datasets import Expectations, MultiTurnGolden

client = ConfidentAI()

result = client.datasets.queue_goldens(
    dataset_id="<DATASET-ID>",
    goldens=[
        MultiTurnGolden(
            scenario="A traveller wants to book a hotel in Paris.",
            expected_outcome="The assistant confirms a reservation near the Louvre.",
            user_description="A traveller planning a weekend in Paris.",
            turns=[
                Turn(
                    role=TurnRole.USER,
                    content="I need a hotel in Paris near the Louvre."
                ),
                Turn(
                    role=TurnRole.ASSISTANT,
                    content="Hôtel du Louvre has rooms available. Which dates?"
                )
            ],
            context=[
                "Hôtel du Louvre is a five-minute walk from the museum."
            ],
            id="<GOLDEN-ID>",
            additional_metadata={"source": "faq"},
            expectations=Expectations(
                must=["Cite a source"],
                must_not=["Mention competitors"]
            ),
            comments="Reviewed by the support team.",
            source_file="capitals.csv",
            source_files=["capitals.csv"],
            finalized=True,
            custom_column_key_values={"difficulty": "easy"},
            tags=["geography"],
            images_mapping={
                "map": {"url": "https://example.com/paris.png", "local": False}
            }
        )
    ],
)

For async mode, call a_queue_goldens and await it as shown below:

result = await client.datasets.a_queue_goldens(...)

Parameters

Parameter	Type	Description
`dataset_id`	`str`	Required. The unique id of the dataset.
`goldens`	`List[Golden]`	Required. The goldens to queue for review. Every golden in one request must be of the same kind and match the dataset's `multiTurn`. They are stored unfinalized, whatever each golden's own `finalized` says. See Golden.

import { ConfidentAI } from "confident-ai";
import { TurnRole } from "confident-ai/common";

const client = new ConfidentAI();

const result = await client.datasets.queueGoldens(
  "<DATASET-ID>",
  [
    {
      scenario: "A traveller wants to book a hotel in Paris.",
      expectedOutcome: "The assistant confirms a reservation near the Louvre.",
      userDescription: "A traveller planning a weekend in Paris.",
      turns: [
          {
            role: TurnRole.USER,
            content: "I need a hotel in Paris near the Louvre."
          },
          {
            role: TurnRole.ASSISTANT,
            content: "Hôtel du Louvre has rooms available. Which dates?"
          }
        ],
      context: ["Hôtel du Louvre is a five-minute walk from the museum."],
      id: "<GOLDEN-ID>",
      additionalMetadata: { source: "faq" },
      expectations: {
          must: ["Cite a source"],
          mustNot: ["Mention competitors"]
        },
      comments: "Reviewed by the support team.",
      sourceFile: "capitals.csv",
      sourceFiles: ["capitals.csv"],
      finalized: true,
      customColumnKeyValues: { difficulty: "easy" },
      tags: ["geography"],
      imagesMapping: {
          map: { url: "https://example.com/paris.png", local: false }
        }
    }
  ],
);

Parameters

Parameter	Type	Description
`datasetId`	`string`	Required. The unique id of the dataset.
`goldens`	`Golden[]`	Required. The goldens to queue for review. Every golden in one request must be of the same kind and match the dataset's `multiTurn`. They are stored unfinalized, whatever each golden's own `finalized` says. See Golden.

Returns

This method returns an object of type DatasetRef.

Run Evaluation

from confident_ai import ConfidentAI
from confident_ai.common import GenerationMode

client = ConfidentAI()

result = client.datasets.run_evaluation(
    dataset_id="<DATASET-ID>",
    metric_collection="Answer Quality",
    identifier="Nightly regression",
    version="00.00.01",
    ai_connection_id="<AI-CONNECTION-ID>",
    voice_connection_id="<VOICE-CONNECTION-ID>",
    prompt_alias="capital-lookup",
    prompt_commit="bab04ce",
    generation_mode=GenerationMode.AI_CONNECTION,
    variables_mapping={"question": "Input"},
    include_simulation=False,
    max_concurrent_generation=5,
    generation_timeout=60,
    num_generations=1,
    mcp_server_ids=["<MCP-SERVER-ID>"],
)

For async mode, call a_run_evaluation and await it as shown below:

result = await client.datasets.a_run_evaluation(...)

Parameters

Parameter	Type	Description
`dataset_id`	`str`	Required. The unique id of the dataset.
`metric_collection`	`str`	Required. The name of the metric collection to evaluate against. Names come from the list metric collections endpoint.
`identifier`	`Optional[str]`	A label for the resulting test run, used to recognise it in the test runs list.
`version`	`Optional[str]`	The dataset version to evaluate. Omit this field to evaluate the latest version.
`ai_connection_id`	`Optional[str]`	The id of the AI connection used to generate the actual outputs before evaluating them. Required when `generationMode` is `AI_CONNECTION`, and not allowed together with `promptAlias`.
`voice_connection_id`	`Optional[str]`	The id of the voice connection to call while simulating each golden's conversation. Required when `generationMode` is `VOICE_CONNECTION`, needs a multi-turn dataset with `includeSimulation` enabled, and is not allowed together with `aiConnectionId` or `promptAlias`.
`prompt_alias`	`Optional[str]`	The alias of the prompt used to generate the actual outputs before evaluating them. Required when `generationMode` is `PROMPT`, and not allowed together with `aiConnectionId`.
`prompt_commit`	`Optional[str]`	The prompt commit hash to generate with. Requires `promptAlias`. Omit this field to generate with the latest commit on the prompt's main branch.
`generation_mode`	`Optional[GenerationMode]`	See GenerationMode.
`variables_mapping`	`Optional[Dict[str, str]]`	Maps each variable in the prompt to the golden field it is interpolated with, such as `Input` or `Expected Output`, or to a dataset custom column key. This field applies only when generating from a prompt.
`include_simulation`	`Optional[bool]`	Whether to simulate a conversation for each golden before evaluating it, for multi-turn datasets. Every golden needs a `scenario` when this is enabled, and `turns` when it is disabled.
`max_concurrent_generation`	`Optional[int]`	The maximum number of generation calls to run in parallel. An AI connection's own `maxConcurrency` takes precedence over this value.
`generation_timeout`	`Optional[int]`	The number of seconds to wait for a single generation before it is marked as errored.
`num_generations`	`Optional[int]`	How many times to run each golden, so a single outlier response does not skew the results. Omit this field to use the AI connection's `defaultNumGenerations`, which is 1 when the connection does not set one.
`mcp_server_ids`	`Optional[List[str]]`	The ids of the MCP servers to attach to the run. A tool call whose name matches a tool exposed by one of these servers is labeled an MCP tool call rather than a function call.

import { ConfidentAI } from "confident-ai";
import { GenerationMode } from "confident-ai/common";

const client = new ConfidentAI();

const result = await client.datasets.runEvaluation(
  "<DATASET-ID>",
  "Answer Quality",
  {
    identifier: "Nightly regression",
    version: "00.00.01",
    aiConnectionId: "<AI-CONNECTION-ID>",
    voiceConnectionId: "<VOICE-CONNECTION-ID>",
    promptAlias: "capital-lookup",
    promptCommit: "bab04ce",
    generationMode: GenerationMode.AI_CONNECTION,
    variablesMapping: { question: "Input" },
    includeSimulation: false,
    maxConcurrentGeneration: 5,
    generationTimeout: 60,
    numGenerations: 1,
    mcpServerIds: ["<MCP-SERVER-ID>"]
  },
);

Parameters

Parameter	Type	Description
`datasetId`	`string`	Required. The unique id of the dataset.
`metricCollection`	`string`	Required. The name of the metric collection to evaluate against. Names come from the list metric collections endpoint.
`identifier`	`string`	A label for the resulting test run, used to recognise it in the test runs list.
`version`	`string`	The dataset version to evaluate. Omit this field to evaluate the latest version.
`aiConnectionId`	`string`	The id of the AI connection used to generate the actual outputs before evaluating them. Required when `generationMode` is `AI_CONNECTION`, and not allowed together with `promptAlias`.
`voiceConnectionId`	`string`	The id of the voice connection to call while simulating each golden's conversation. Required when `generationMode` is `VOICE_CONNECTION`, needs a multi-turn dataset with `includeSimulation` enabled, and is not allowed together with `aiConnectionId` or `promptAlias`.
`promptAlias`	`string`	The alias of the prompt used to generate the actual outputs before evaluating them. Required when `generationMode` is `PROMPT`, and not allowed together with `aiConnectionId`.
`promptCommit`	`string`	The prompt commit hash to generate with. Requires `promptAlias`. Omit this field to generate with the latest commit on the prompt's main branch.
`generationMode`	`GenerationMode`	See GenerationMode.
`variablesMapping`	`Record<string, string>`	Maps each variable in the prompt to the golden field it is interpolated with, such as `Input` or `Expected Output`, or to a dataset custom column key. This field applies only when generating from a prompt.
`includeSimulation`	`boolean`	Whether to simulate a conversation for each golden before evaluating it, for multi-turn datasets. Every golden needs a `scenario` when this is enabled, and `turns` when it is disabled.
`maxConcurrentGeneration`	`number`	The maximum number of generation calls to run in parallel. An AI connection's own `maxConcurrency` takes precedence over this value.
`generationTimeout`	`number`	The number of seconds to wait for a single generation before it is marked as errored.
`numGenerations`	`number`	How many times to run each golden, so a single outlier response does not skew the results. Omit this field to use the AI connection's `defaultNumGenerations`, which is 1 when the connection does not set one.
`mcpServerIds`	`string[]`	The ids of the MCP servers to attach to the run. A tool call whose name matches a tool exposed by one of these servers is labeled an MCP tool call rather than a function call.

Returns

This method returns an object of type RunDatasetEvaluationResult.

Types

Dataset

A pulled dataset with the goldens of one version.

class Dataset:
    id: str
    alias: str
    multi_turn: bool = Field(alias="multiTurn")
    version: Optional[str]
    goldens: List[Golden]

idstrRequired

This is the unique id of the dataset.

Example: "<DATASET-ID>"

aliasstrRequired

This is the alias of the dataset, which is unique within your project.

Example: "capitals"

multi_turnboolRequired

This is true if the dataset is multi-turn, which contains multi-turn goldens. Single-turn datasets have multiTurn set to false and contain single-turn goldens.

Example: false

versionOptional[str]Required

The version number of the goldens returned, or null when the dataset has no versions.

Example: "00.00.01"

goldensList[Golden]Required

The goldens in the dataset, oldest first. Every golden is single-turn or multi-turn according to the dataset's multiTurn.

See Golden.

interface Dataset {
  id: string;
  alias: string;
  multiTurn: boolean;
  version: string | null;
  goldens: Golden[];
}

idstringRequired

This is the unique id of the dataset.

Example: "<DATASET-ID>"

aliasstringRequired

This is the alias of the dataset, which is unique within your project.

Example: "capitals"

multiTurnbooleanRequired

This is true if the dataset is multi-turn, which contains multi-turn goldens. Single-turn datasets have multiTurn set to false and contain single-turn goldens.

Example: false

versionstring | nullRequired

The version number of the goldens returned, or null when the dataset has no versions.

Example: "00.00.01"

goldensGolden[]Required

The goldens in the dataset, oldest first. Every golden is single-turn or multi-turn according to the dataset's multiTurn.

See Golden.

DatasetList

class DatasetList:
    datasets: List[DatasetSummary]

datasetsList[DatasetSummary]Required

This is the list of datasets in your project, newest first.

See DatasetSummary.

interface DatasetList {
  datasets: DatasetSummary[];
}

datasetsDatasetSummary[]Required

This is the list of datasets in your project, newest first.

See DatasetSummary.

DatasetRef

class DatasetRef:
    id: str

idstrRequired

This is the unique id of the dataset.

Example: "<DATASET-ID>"

interface DatasetRef {
  id: string;
}

idstringRequired

This is the unique id of the dataset.

Example: "<DATASET-ID>"

DatasetSummary

A dataset as it appears in your project's dataset list, without any goldens.

class DatasetSummary:
    id: str
    alias: str
    multi_turn: bool = Field(alias="multiTurn")

idstrRequired

This is the unique id of the dataset.

Example: "<DATASET-ID>"

aliasstrRequired

This is the alias of the dataset, which is unique within your project.

Example: "capitals"

multi_turnboolRequired

This is true if the dataset is multi-turn, which contains multi-turn goldens. Single-turn datasets have multiTurn set to false and contain single-turn goldens.

Example: false

interface DatasetSummary {
  id: string;
  alias: string;
  multiTurn: boolean;
}

idstringRequired

This is the unique id of the dataset.

Example: "<DATASET-ID>"

aliasstringRequired

This is the alias of the dataset, which is unique within your project.

Example: "capitals"

multiTurnbooleanRequired

This is true if the dataset is multi-turn, which contains multi-turn goldens. Single-turn datasets have multiTurn set to false and contain single-turn goldens.

Example: false

Expectations

What the LLM system must and must not do for a golden: must lists what it has to do, and mustNot what it must avoid.

class Expectations:
    must: List[str]
    must_not: List[str] = Field(alias="mustNot")

mustList[str]Required

must_notList[str]Required

interface Expectations {
  must: string[];
  mustNot: string[];
}

muststring[]Required

mustNotstring[]Required

GenerationMode

Where the actual outputs come from when running a dataset: AI_CONNECTION generates them with an AI connection, PROMPT with a prompt, and VOICE_CONNECTION by calling a voice connection during simulation. Omit it when you supply at most one of aiConnectionId, promptAlias or voiceConnectionId, and Confident AI infers the mode from whichever you sent.

class GenerationMode(Enum):
    AI_CONNECTION = "AI_CONNECTION"
    PROMPT = "PROMPT"
    VOICE_CONNECTION = "VOICE_CONNECTION"

enum GenerationMode {
  AI_CONNECTION = "AI_CONNECTION",
  PROMPT = "PROMPT",
  VOICE_CONNECTION = "VOICE_CONNECTION",
}

AI_CONNECTION · PROMPT · VOICE_CONNECTION

Golden

A golden in the dataset: single-turn when it carries input, multi-turn when it carries scenario. The dataset's multiTurn decides which kind every golden in it is.

Golden = Union[
    SingleTurnGolden,
    MultiTurnGolden,
]

type Golden =
  | SingleTurnGolden
  | MultiTurnGolden;

A Golden is one of the shapes below. Send the fields of one of them, never a mix of both.

A single-turn golden as stored in the dataset.

class SingleTurnGolden:
    input: str
    actual_output: Optional[str] = Field(default=None, alias="actualOutput")
    expected_output: Optional[str] = Field(default=None, alias="expectedOutput")
    context: Optional[List[str]] = None
    retrieval_context: Optional[List[str]] = Field(default=None, alias="retrievalContext")
    tools_called: Optional[List[ToolCall]] = Field(default=None, alias="toolsCalled")
    expected_tools: Optional[List[ToolCall]] = Field(default=None, alias="expectedTools")
    token_cost: Optional[float] = Field(default=None, alias="tokenCost")
    input_token_count: Optional[int] = Field(default=None, alias="inputTokenCount")
    output_token_count: Optional[int] = Field(default=None, alias="outputTokenCount")
    id: Optional[str] = None
    additional_metadata: Optional[Dict[str, Any]] = Field(default=None, alias="additionalMetadata")
    expectations: Optional[Expectations] = None
    comments: Optional[str] = None
    source_file: Optional[str] = Field(default=None, alias="sourceFile")
    source_files: Optional[List[str]] = Field(default=None, alias="sourceFiles")
    finalized: Optional[bool] = None
    custom_column_key_values: Optional[Dict[str, str]] = Field(default=None, alias="customColumnKeyValues")
    tags: Optional[List[str]] = None
    images_mapping: Optional[Dict[str, MLLMImageInput]] = Field(default=None, alias="imagesMapping")

inputstrRequired

This is the input to your LLM application.

Example: "What is the capital of France?"

actual_outputOptional[str]

This is the actual output of your LLM application.

Example: "The capital of France is Paris."

expected_outputOptional[str]

This is the expected output of your LLM application, which is the ideal actual output.

Example: "Paris."

contextOptional[List[str]]

This is the ideal retrieval context of your LLM application.

Example: ["Paris is the capital of France."]

retrieval_contextOptional[List[str]]

This is the retrieval context of your LLM application.

Example: ["Paris is the capital and largest city of France."]

tools_calledOptional[List[ToolCall]]

This is the tools called by your LLM application.

See ToolCall.

expected_toolsOptional[List[ToolCall]]

This is the expected tools to be called by the LLM application.

See ToolCall.

token_costOptional[float]

This is the cost of the tokens used to produce the actual output.

Example: 0.002

input_token_countOptional[int]

This is the number of input tokens passed to the LLM model.

Example: 12

output_token_countOptional[int]

This is the number of output tokens generated by the LLM model.

Example: 3

idOptional[str]

The id of the golden assigned by Confident AI. Use it to get, update or delete this golden.

Example: "<GOLDEN-ID>"

additional_metadataOptional[Dict[str, Any]]

This is any additional metadata associated with the golden.

Example: {"source":"faq"}

expectationsOptional[Expectations]

These are what the LLM system must and must not do for this golden.

See Expectations.

Example: {"must":["Cite a source"],"mustNot":["Mention competitors"]}

commentsOptional[str]

This is any comments associated with the golden.

Example: "Reviewed by the support team."

source_fileOptional[str]

This is the source file from which the golden was retrieved.

Example: "capitals.csv"

source_filesOptional[List[str]]

These are the source files the golden was retrieved from.

Example: ["capitals.csv"]

finalizedOptional[bool]

This is true when the golden is finalized and ready to use in evaluations.

Example: true

custom_column_key_valuesOptional[Dict[str, str]]

Key-value pairs representing custom table column data for this golden. Keys correspond to the custom column keys defined in the dataset. Absent when the golden has no custom column values.

Example: {"difficulty":"easy"}

tagsOptional[List[str]]

These are the tags associated with the golden.

Example: ["geography"]

images_mappingOptional[Dict[str, MLLMImageInput]]

The media this golden refers to, keyed by the id inside each placeholder. Put [DEEPEVAL:IMAGE:<id>] or [DEEPEVAL:PDF:<id>] in a text field where the media belongs, and the platform substitutes the entry with a matching key.

See MLLMImageInput.

Example: {"map":{"url":"https://example.com/paris.png","local":false}}

interface SingleTurnGolden {
  input: string;
  actualOutput?: string | null;
  expectedOutput?: string | null;
  context?: string[] | null;
  retrievalContext?: string[] | null;
  toolsCalled?: ToolCall[] | null;
  expectedTools?: ToolCall[] | null;
  tokenCost?: number | null;
  inputTokenCount?: number | null;
  outputTokenCount?: number | null;
  id?: string;
  additionalMetadata?: Record<string, unknown> | null;
  expectations?: Expectations | null;
  comments?: string | null;
  sourceFile?: string | null;
  sourceFiles?: string[];
  finalized?: boolean;
  customColumnKeyValues?: Record<string, string>;
  tags?: string[];
  imagesMapping?: Record<string, MLLMImageInput>;
}

inputstringRequired

This is the input to your LLM application.

Example: "What is the capital of France?"

actualOutputstring | null

This is the actual output of your LLM application.

Example: "The capital of France is Paris."

expectedOutputstring | null

This is the expected output of your LLM application, which is the ideal actual output.

Example: "Paris."

contextstring[] | null

This is the ideal retrieval context of your LLM application.

Example: ["Paris is the capital of France."]

retrievalContextstring[] | null

This is the retrieval context of your LLM application.

Example: ["Paris is the capital and largest city of France."]

toolsCalledToolCall[] | null

This is the tools called by your LLM application.

See ToolCall.

expectedToolsToolCall[] | null

This is the expected tools to be called by the LLM application.

See ToolCall.

tokenCostnumber | null

This is the cost of the tokens used to produce the actual output.

Example: 0.002

inputTokenCountnumber | null

This is the number of input tokens passed to the LLM model.

Example: 12

outputTokenCountnumber | null

This is the number of output tokens generated by the LLM model.

Example: 3

idstring

The id of the golden assigned by Confident AI. Use it to get, update or delete this golden.

Example: "<GOLDEN-ID>"

additionalMetadataRecord<string, unknown> | null

This is any additional metadata associated with the golden.

Example: {"source":"faq"}

expectationsExpectations | null

These are what the LLM system must and must not do for this golden.

See Expectations.

Example: {"must":["Cite a source"],"mustNot":["Mention competitors"]}

commentsstring | null

This is any comments associated with the golden.

Example: "Reviewed by the support team."

sourceFilestring | null

This is the source file from which the golden was retrieved.

Example: "capitals.csv"

sourceFilesstring[]

These are the source files the golden was retrieved from.

Example: ["capitals.csv"]

finalizedboolean

This is true when the golden is finalized and ready to use in evaluations.

Example: true

customColumnKeyValuesRecord<string, string>

Key-value pairs representing custom table column data for this golden. Keys correspond to the custom column keys defined in the dataset. Absent when the golden has no custom column values.

Example: {"difficulty":"easy"}

tagsstring[]

These are the tags associated with the golden.

Example: ["geography"]

imagesMappingRecord<string, MLLMImageInput>

See MLLMImageInput.

Example: {"map":{"url":"https://example.com/paris.png","local":false}}

A multi-turn golden as stored in the dataset.

class MultiTurnGolden:
    scenario: str
    expected_outcome: Optional[str] = Field(default=None, alias="expectedOutcome")
    user_description: Optional[str] = Field(default=None, alias="userDescription")
    turns: Optional[List[Turn]] = None
    context: Optional[List[str]] = None
    id: Optional[str] = None
    additional_metadata: Optional[Dict[str, Any]] = Field(default=None, alias="additionalMetadata")
    expectations: Optional[Expectations] = None
    comments: Optional[str] = None
    source_file: Optional[str] = Field(default=None, alias="sourceFile")
    source_files: Optional[List[str]] = Field(default=None, alias="sourceFiles")
    finalized: Optional[bool] = None
    custom_column_key_values: Optional[Dict[str, str]] = Field(default=None, alias="customColumnKeyValues")
    tags: Optional[List[str]] = None
    images_mapping: Optional[Dict[str, MLLMImageInput]] = Field(default=None, alias="imagesMapping")

scenariostrRequired

This is a description of the conversation context.

Example: "A traveller wants to book a hotel in Paris."

expected_outcomeOptional[str]

This describes the expected outcome, or ideal conversation flow, of the conversation.

Example: "The assistant confirms a reservation near the Louvre."

user_descriptionOptional[str]

This is the description of the user in the conversation.

Example: "A traveller planning a weekend in Paris."

turnsOptional[List[Turn]]

This is the list of turns in the conversation.

See Turn.

Example: [{"role":"user","content":"I need a hotel in Paris near the Louvre."},{"role":"assistant","content":"Hôtel du Louvre has rooms available. Which dates?"}]

contextOptional[List[str]]

This is the context of the conversation.

Example: ["Hôtel du Louvre is a five-minute walk from the museum."]

idOptional[str]

The id of the golden assigned by Confident AI. Use it to get, update or delete this golden.

Example: "<GOLDEN-ID>"

additional_metadataOptional[Dict[str, Any]]

This is any additional metadata associated with the golden.

Example: {"source":"faq"}

expectationsOptional[Expectations]

These are what the LLM system must and must not do for this golden.

See Expectations.

Example: {"must":["Cite a source"],"mustNot":["Mention competitors"]}

commentsOptional[str]

This is any comments associated with the golden.

Example: "Reviewed by the support team."

source_fileOptional[str]

This is the source file from which the golden was retrieved.

Example: "capitals.csv"

source_filesOptional[List[str]]

These are the source files the golden was retrieved from.

Example: ["capitals.csv"]

finalizedOptional[bool]

This is true when the golden is finalized and ready to use in evaluations.

Example: true

custom_column_key_valuesOptional[Dict[str, str]]

Key-value pairs representing custom table column data for this golden. Keys correspond to the custom column keys defined in the dataset. Absent when the golden has no custom column values.

Example: {"difficulty":"easy"}

tagsOptional[List[str]]

These are the tags associated with the golden.

Example: ["geography"]

images_mappingOptional[Dict[str, MLLMImageInput]]

See MLLMImageInput.

Example: {"map":{"url":"https://example.com/paris.png","local":false}}

interface MultiTurnGolden {
  scenario: string;
  expectedOutcome?: string | null;
  userDescription?: string | null;
  turns?: Turn[] | null;
  context?: string[] | null;
  id?: string;
  additionalMetadata?: Record<string, unknown> | null;
  expectations?: Expectations | null;
  comments?: string | null;
  sourceFile?: string | null;
  sourceFiles?: string[];
  finalized?: boolean;
  customColumnKeyValues?: Record<string, string>;
  tags?: string[];
  imagesMapping?: Record<string, MLLMImageInput>;
}

scenariostringRequired

This is a description of the conversation context.

Example: "A traveller wants to book a hotel in Paris."

expectedOutcomestring | null

This describes the expected outcome, or ideal conversation flow, of the conversation.

Example: "The assistant confirms a reservation near the Louvre."

userDescriptionstring | null

This is the description of the user in the conversation.

Example: "A traveller planning a weekend in Paris."

turnsTurn[] | null

This is the list of turns in the conversation.

See Turn.

Example: [{"role":"user","content":"I need a hotel in Paris near the Louvre."},{"role":"assistant","content":"Hôtel du Louvre has rooms available. Which dates?"}]

contextstring[] | null

This is the context of the conversation.

Example: ["Hôtel du Louvre is a five-minute walk from the museum."]

idstring

The id of the golden assigned by Confident AI. Use it to get, update or delete this golden.

Example: "<GOLDEN-ID>"

additionalMetadataRecord<string, unknown> | null

This is any additional metadata associated with the golden.

Example: {"source":"faq"}

expectationsExpectations | null

These are what the LLM system must and must not do for this golden.

See Expectations.

Example: {"must":["Cite a source"],"mustNot":["Mention competitors"]}

commentsstring | null

This is any comments associated with the golden.

Example: "Reviewed by the support team."

sourceFilestring | null

This is the source file from which the golden was retrieved.

Example: "capitals.csv"

sourceFilesstring[]

These are the source files the golden was retrieved from.

Example: ["capitals.csv"]

finalizedboolean

This is true when the golden is finalized and ready to use in evaluations.

Example: true

customColumnKeyValuesRecord<string, string>

Key-value pairs representing custom table column data for this golden. Keys correspond to the custom column keys defined in the dataset. Absent when the golden has no custom column values.

Example: {"difficulty":"easy"}

tagsstring[]

These are the tags associated with the golden.

Example: ["geography"]

imagesMappingRecord<string, MLLMImageInput>

See MLLMImageInput.

Example: {"map":{"url":"https://example.com/paris.png","local":false}}

MLLMImageInput

An image referenced from a text field by a [DEEPEVAL:IMAGE:<key>] marker. Send either a public url or the bytes in base64.

class MLLMImageInput:
    url: str
    local: bool
    base64: Optional[str] = None
    filename: Optional[str] = None
    mime_type: Optional[str] = Field(default=None, alias="mimeType")
    data_base64: Optional[str] = Field(default=None, alias="dataBase64")

urlstrRequired

This is the URL of the image.

Example: "https://example.com/everest.png"

localboolRequired

This is true when the image is your local file.

Example: false

base64Optional[str]

The base64 data of the image.

Example: "iVBORw0KGgo="

filenameOptional[str]

The original file name.

Example: "everest.png"

mime_typeOptional[str]

The image's MIME type.

Example: "image/png"

data_base64Optional[str]

The image encoded as a base64 data URL.

Example: "data:image/png;base64,iVBORw0KGgo="

interface MLLMImageInput {
  url: string;
  local: boolean;
  base64?: string;
  filename?: string;
  mimeType?: string;
  dataBase64?: string;
}

urlstringRequired

This is the URL of the image.

Example: "https://example.com/everest.png"

localbooleanRequired

This is true when the image is your local file.

Example: false

base64string

The base64 data of the image.

Example: "iVBORw0KGgo="

filenamestring

The original file name.

Example: "everest.png"

mimeTypestring

The image's MIME type.

Example: "image/png"

dataBase64string

The image encoded as a base64 data URL.

Example: "data:image/png;base64,iVBORw0KGgo="

RunDatasetEvaluationResult

class RunDatasetEvaluationResult:
    id: str
    test_case_count: int = Field(alias="testCaseCount")

idstrRequired

This is the unique id of the test run the dataset is evaluated in, generated by Confident AI and not to be confused with the identifier you supplied.

Example: "<TEST-RUN-ID>"

test_case_countintRequired

The number of test cases the evaluation was started with.

Example: 42

interface RunDatasetEvaluationResult {
  id: string;
  testCaseCount: number;
}

idstringRequired

This is the unique id of the test run the dataset is evaluated in, generated by Confident AI and not to be confused with the identifier you supplied.

Example: "<TEST-RUN-ID>"

testCaseCountnumberRequired

The number of test cases the evaluation was started with.

Example: 42

ToolCall

A tool your LLM application invoked, with what it passed in and what came back.

class ToolCall:
    name: str
    type: Optional[ToolCallType] = None
    description: Optional[str] = None
    input_parameters: Optional[Dict[str, Any]] = Field(default=None, alias="inputParameters")
    output: Optional[Any] = None
    reasoning: Optional[str] = None

namestrRequired

This is the name of the tool.

Example: "get_landmark_info"

typeOptional[ToolCallType]

See ToolCallType.

descriptionOptional[str]

This is the description of the tool.

Example: "This tool gives information about a mountain."

input_parametersOptional[Dict[str, Any]]

This is the input parameters that are passed to the tool.

Example: {"mountain":"Everest"}

outputOptional[Any]

This is the output of the tool.

Example: "8,848 metres"

reasoningOptional[str]

This is the reasoning your LLM provided for the tool call.

Example: "The user asked for the height of a mountain."

interface ToolCall {
  name: string;
  type?: ToolCallType;
  description?: string;
  inputParameters?: Record<string, unknown> | null;
  output?: unknown;
  reasoning?: string;
}

namestringRequired

This is the name of the tool.

Example: "get_landmark_info"

typeToolCallType

See ToolCallType.

descriptionstring

This is the description of the tool.

Example: "This tool gives information about a mountain."

inputParametersRecord<string, unknown> | null

This is the input parameters that are passed to the tool.

Example: {"mountain":"Everest"}

outputunknown

This is the output of the tool.

Example: "8,848 metres"

reasoningstring

This is the reasoning your LLM provided for the tool call.

Example: "The user asked for the height of a mountain."

ToolCallType

The type of the tool call, either a function or an MCP tool.

class ToolCallType(Enum):
    FUNCTION = "FUNCTION"
    MCP = "MCP"

enum ToolCallType {
  FUNCTION = "FUNCTION",
  MCP = "MCP",
}

FUNCTION · MCP

Turn

One message in a conversation, from either the user or the assistant, with the context and tools behind an assistant reply.

class Turn:
    id: Optional[str] = None
    role: TurnRole
    content: str
    user_id: Optional[str] = Field(default=None, alias="userId")
    retrieval_context: Optional[List[str]] = Field(default=None, alias="retrievalContext")
    tools_called: Optional[List[ToolCall]] = Field(default=None, alias="toolsCalled")
    metadata: Optional[Dict[str, Any]] = None

idOptional[str]

The id of a turn assigned by Confident AI.

Example: "<TURN-ID>"

roleTurnRoleRequired

See TurnRole.

contentstrRequired

The message content of the turn.

Example: "How tall is Mount Everest?"

user_idOptional[str]

The user ID associated with the turn.

Example: "end-user-42"

retrieval_contextOptional[List[str]]

The contexts retrieved to generate the LLM response for this turn.

Example: ["Everest is 8,848 metres tall."]

tools_calledOptional[List[ToolCall]]

The tools called to generate the LLM response for this turn.

See ToolCall.

metadataOptional[Dict[str, Any]]

Additional metadata to associate with this turn.

Example: {"channel":"web"}

interface Turn {
  id?: string;
  role: TurnRole;
  content: string;
  userId?: string;
  retrievalContext?: string[] | null;
  toolsCalled?: ToolCall[] | null;
  metadata?: Record<string, unknown> | null;
}

idstring

The id of a turn assigned by Confident AI.

Example: "<TURN-ID>"

roleTurnRoleRequired

See TurnRole.

contentstringRequired

The message content of the turn.

Example: "How tall is Mount Everest?"

userIdstring

The user ID associated with the turn.

Example: "end-user-42"

retrievalContextstring[] | null

The contexts retrieved to generate the LLM response for this turn.

Example: ["Everest is 8,848 metres tall."]

toolsCalledToolCall[] | null

The tools called to generate the LLM response for this turn.

See ToolCall.

metadataRecord<string, unknown> | null

Additional metadata to associate with this turn.

Example: {"channel":"web"}

TurnRole

The role of the turn, either user or assistant.

class TurnRole(Enum):
    USER = "user"
    ASSISTANT = "assistant"

enum TurnRole {
  USER = "user",
  ASSISTANT = "assistant",
}

USER · ASSISTANT

Building a production pipeline?Design a scalable API workflow for evals, datasets, traces, and promptsTalk to an engineer

Last updated on October 19, 2026

Built byConfident AI