Datasets

Every Datasets method in the Confident AI Python and TypeScript SDKs.

Overview

The Confident AI SDK exposes every Dataset method on the platform. This page documents how to call these methods in all supported languages. See the introduction to install the SDK and set your API key.

Dataset

client.dataset() returns a Dataset object that stands for one dataset. This object stores the fields listed below, and passes the dataset's id to every method called on it, so you don't need to pass the id nor the stored fields as arguments.

push is the exception, because it sends the stored fields rather than calling a route that names the id. Open the object with alias to call it.

from confident_ai import ConfidentAI

client = ConfidentAI()

dataset = client.dataset(dataset_id="<DATASET-ID>")

Properties

These are the fields a Dataset stores. A method that loads the dataset fills them in, and a method that saves it sends whichever of them you have set, so set them before you save and read them after you load.

Parameter	Type	Description
`dataset_id`	`Optional[str]`	The unique id of the dataset.
`id`	`Optional[str]`	This is the unique id of the dataset.
`alias`	`Optional[str]`	This is the alias of the dataset, which is unique within your project.
`multi_turn`	`Optional[bool]`	This is true if the dataset is multi-turn, which contains multi-turn goldens. Single-turn datasets have `multiTurn` set to false and contain single-turn goldens.
`version`	`Optional[str]`	The version number of the goldens returned, or null when the dataset has no versions.
`goldens`	`Optional[List[Golden]]`	The goldens in the dataset, oldest first. Every golden is single-turn or multi-turn according to the dataset's `multiTurn`. See Golden.

import { ConfidentAI } from "confident-ai";

const client = new ConfidentAI();

const dataset = client.dataset("<DATASET-ID>");

Properties

Parameter	Type	Description
`datasetId`	`string`	The unique id of the dataset.
`id`	`string`	This is the unique id of the dataset.
`alias`	`string`	This is the alias of the dataset, which is unique within your project.
`multiTurn`	`boolean`	This is true if the dataset is multi-turn, which contains multi-turn goldens. Single-turn datasets have `multiTurn` set to false and contain single-turn goldens.
`version`	`string \| null`	The version number of the goldens returned, or null when the dataset has no versions.
`goldens`	`Golden[]`	The goldens in the dataset, oldest first. Every golden is single-turn or multi-turn according to the dataset's `multiTurn`. See Golden.

Methods

Delete Dataset

Permanently deletes the dataset and everything in it: its goldens, versions, tags and custom columns. This action cannot be undone.

from confident_ai import ConfidentAI

client = ConfidentAI()

dataset = client.dataset(dataset_id="<DATASET-ID>")
result = dataset.delete()

For async mode, call a_delete and await it as shown below:

result = await dataset.a_delete(...)

import { ConfidentAI } from "confident-ai";

const client = new ConfidentAI();

const dataset = client.dataset("<DATASET-ID>");
const result = await dataset.delete();

Returns

This method returns an object of type DatasetRef.

Run Evaluation

Starts an evaluation of the dataset's finalized goldens against a metric collection and returns the test run it is evaluated in. It runs asynchronously, so this returns as soon as the run is created. By default the goldens' stored actual outputs are evaluated; supply aiConnectionId or promptAlias, never both, to generate them first.

from confident_ai import ConfidentAI
from confident_ai.common import GenerationMode

client = ConfidentAI()

dataset = client.dataset(dataset_id="<DATASET-ID>")
result = dataset.run_evaluation(
    metric_collection="Answer Quality",
    identifier="Nightly regression",
    version="00.00.01",
    ai_connection_id="<AI-CONNECTION-ID>",
    prompt_alias="capital-lookup",
    prompt_commit="bab04ce",
    generation_mode=GenerationMode.AI_CONNECTION,
    variables_mapping={"question": "Input"},
    include_simulation=False,
    max_concurrent_generation=5,
    generation_timeout=60,
    num_generations=1,
    mcp_server_ids=["<MCP-SERVER-ID>"],
)

For async mode, call a_run_evaluation and await it as shown below:

result = await dataset.a_run_evaluation(...)

Parameters

Parameter	Type	Description
`metric_collection`	`str`	Required. The name of the metric collection to evaluate against. Names come from the list metric collections endpoint.
`identifier`	`Optional[str]`	A label for the resulting test run, used to recognise it in the test runs list.
`version`	`Optional[str]`	The dataset version to evaluate. Omit this field to evaluate the latest version.
`ai_connection_id`	`Optional[str]`	The id of the AI connection used to generate the actual outputs before evaluating them. Required when `generationMode` is AI_CONNECTION, and not allowed together with `promptAlias`.
`prompt_alias`	`Optional[str]`	The alias of the prompt used to generate the actual outputs before evaluating them. Required when `generationMode` is PROMPT, and not allowed together with `aiConnectionId`.
`prompt_commit`	`Optional[str]`	The prompt commit hash to generate with. Requires `promptAlias`. Omit this field to generate with the latest commit on the prompt's main branch.
`generation_mode`	`Optional[GenerationMode]`	See GenerationMode.
`variables_mapping`	`Optional[Dict[str, str]]`	Maps each variable in the prompt to the golden field it is interpolated with, such as `Input` or `Expected Output`, or to a dataset custom column key. This field applies only when generating from a prompt.
`include_simulation`	`Optional[bool]`	Whether to simulate a conversation for each golden before evaluating it, for multi-turn datasets. Every golden needs a `scenario` when this is enabled, and `turns` when it is disabled.
`max_concurrent_generation`	`Optional[int]`	The maximum number of generation calls to run in parallel. An AI connection's own `maxConcurrency` takes precedence over this value.
`generation_timeout`	`Optional[int]`	The number of seconds to wait for a single generation before it is marked as errored.
`num_generations`	`Optional[int]`	How many times to run each golden, so a single outlier response does not skew the results. Omit this field to use the AI connection's `defaultNumGenerations`, which is 1 when the connection does not set one.
`mcp_server_ids`	`Optional[List[str]]`	The ids of the MCP servers to attach to the run. A tool call whose name matches a tool exposed by one of these servers is labeled an MCP tool call rather than a function call.

import { ConfidentAI } from "confident-ai";
import { GenerationMode } from "confident-ai/common";

const client = new ConfidentAI();

const dataset = client.dataset("<DATASET-ID>");
const result = await dataset.runEvaluation(
  "Answer Quality",
  {
    identifier: "Nightly regression",
    version: "00.00.01",
    aiConnectionId: "<AI-CONNECTION-ID>",
    promptAlias: "capital-lookup",
    promptCommit: "bab04ce",
    generationMode: GenerationMode.AI_CONNECTION,
    variablesMapping: { question: "Input" },
    includeSimulation: false,
    maxConcurrentGeneration: 5,
    generationTimeout: 60,
    numGenerations: 1,
    mcpServerIds: ["<MCP-SERVER-ID>"]
  },
);

Parameters

Parameter	Type	Description
`metricCollection`	`string`	Required. The name of the metric collection to evaluate against. Names come from the list metric collections endpoint.
`identifier`	`string`	A label for the resulting test run, used to recognise it in the test runs list.
`version`	`string`	The dataset version to evaluate. Omit this field to evaluate the latest version.
`aiConnectionId`	`string`	The id of the AI connection used to generate the actual outputs before evaluating them. Required when `generationMode` is AI_CONNECTION, and not allowed together with `promptAlias`.
`promptAlias`	`string`	The alias of the prompt used to generate the actual outputs before evaluating them. Required when `generationMode` is PROMPT, and not allowed together with `aiConnectionId`.
`promptCommit`	`string`	The prompt commit hash to generate with. Requires `promptAlias`. Omit this field to generate with the latest commit on the prompt's main branch.
`generationMode`	`GenerationMode`	See GenerationMode.
`variablesMapping`	`Record<string, string>`	Maps each variable in the prompt to the golden field it is interpolated with, such as `Input` or `Expected Output`, or to a dataset custom column key. This field applies only when generating from a prompt.
`includeSimulation`	`boolean`	Whether to simulate a conversation for each golden before evaluating it, for multi-turn datasets. Every golden needs a `scenario` when this is enabled, and `turns` when it is disabled.
`maxConcurrentGeneration`	`number`	The maximum number of generation calls to run in parallel. An AI connection's own `maxConcurrency` takes precedence over this value.
`generationTimeout`	`number`	The number of seconds to wait for a single generation before it is marked as errored.
`numGenerations`	`number`	How many times to run each golden, so a single outlier response does not skew the results. Omit this field to use the AI connection's `defaultNumGenerations`, which is 1 when the connection does not set one.
`mcpServerIds`	`string[]`	The ids of the MCP servers to attach to the run. A tool call whose name matches a tool exposed by one of these servers is labeled an MCP tool call rather than a function call.

Returns

This method returns an object of type RunDatasetEvaluationResult.

Single Turn Golden

Adds goldens to the dataset as unfinalized goldens, for review on the platform before they are used in evaluations.

A single-turn golden to write: one input to your LLM application and the outputs expected of it.

from confident_ai import ConfidentAI
from confident_ai.datasets import SingleTurnGoldenRequest
from confident_ai.common import ToolCall
from confident_ai.common import ToolCallType

client = ConfidentAI()

dataset = client.dataset(dataset_id="<DATASET-ID>")
result = dataset.queue_goldens(
    goldens=[
        SingleTurnGoldenRequest(
            input="What is the capital of France?",
            actual_output="The capital of France is Paris.",
            expected_output="Paris.",
            context=["Paris is the capital of France."],
            retrieval_context=[
                "Paris is the capital and largest city of France."
            ],
            tools_called=[
                ToolCall(
                    name="get_landmark_info",
                    type=ToolCallType.FUNCTION,
                    description="This tool gives information about a mountain.",
                    input_parameters={"mountain": "Everest"},
                    output="8,848 metres",
                    reasoning="The user asked for the height of a mountain."
                )
            ],
            expected_tools=[
                ToolCall(
                    name="get_landmark_info",
                    type=ToolCallType.FUNCTION,
                    description="This tool gives information about a mountain.",
                    input_parameters={"mountain": "Everest"},
                    output="8,848 metres",
                    reasoning="The user asked for the height of a mountain."
                )
            ],
            token_cost=0.002,
            input_token_count=12,
            output_token_count=3,
            additional_metadata={"source": "faq"},
            comments="Reviewed by the support team.",
            source_file="capitals.csv",
            source_files=["capitals.csv"],
            finalized=True,
            custom_column_key_values={"difficulty": "easy"},
            images_mapping={
                "map": {"url": "https://example.com/paris.png", "local": False}
            },
            tags=["geography"]
        )
    ],
)

For async mode, call a_queue_goldens and await it as shown below:

result = await dataset.a_queue_goldens(...)

Parameters

Parameter	Type	Description
`goldens`	`List[GoldenRequest]`	Required. The goldens to queue for review. Every golden in one request must be of the same kind and match the dataset's `multiTurn`. They are stored unfinalized, whatever each golden's own `finalized` says. See GoldenRequest.

import { ConfidentAI } from "confident-ai";
import { ToolCallType } from "confident-ai/common";

const client = new ConfidentAI();

const dataset = client.dataset("<DATASET-ID>");
const result = await dataset.queueGoldens(
  [
    {
      input: "What is the capital of France?",
      actualOutput: "The capital of France is Paris.",
      expectedOutput: "Paris.",
      context: ["Paris is the capital of France."],
      retrievalContext: [
          "Paris is the capital and largest city of France."
        ],
      toolsCalled: [
          {
            name: "get_landmark_info",
            type: ToolCallType.FUNCTION,
            description: "This tool gives information about a mountain.",
            inputParameters: { mountain: "Everest" },
            output: "8,848 metres",
            reasoning: "The user asked for the height of a mountain."
          }
        ],
      expectedTools: [
          {
            name: "get_landmark_info",
            type: ToolCallType.FUNCTION,
            description: "This tool gives information about a mountain.",
            inputParameters: { mountain: "Everest" },
            output: "8,848 metres",
            reasoning: "The user asked for the height of a mountain."
          }
        ],
      tokenCost: 0.002,
      inputTokenCount: 12,
      outputTokenCount: 3,
      additionalMetadata: { source: "faq" },
      comments: "Reviewed by the support team.",
      sourceFile: "capitals.csv",
      sourceFiles: ["capitals.csv"],
      finalized: true,
      customColumnKeyValues: { difficulty: "easy" },
      imagesMapping: {
          map: { url: "https://example.com/paris.png", local: false }
        },
      tags: ["geography"]
    }
  ],
);

Parameters

Parameter	Type	Description
`goldens`	`GoldenRequest[]`	Required. The goldens to queue for review. Every golden in one request must be of the same kind and match the dataset's `multiTurn`. They are stored unfinalized, whatever each golden's own `finalized` says. See GoldenRequest.

Returns

This method returns an object of type DatasetRef.

Multi Turn Golden

Adds goldens to the dataset as unfinalized goldens, for review on the platform before they are used in evaluations.

A multi-turn golden to write: the scenario of a conversation with your LLM application and, optionally, its turns.

from confident_ai import ConfidentAI
from confident_ai.datasets import MultiTurnGoldenRequest

client = ConfidentAI()

dataset = client.dataset(dataset_id="<DATASET-ID>")
result = dataset.queue_goldens(
    goldens=[
        MultiTurnGoldenRequest(
            scenario="A traveller wants to book a hotel in Paris.",
            expected_outcome="The assistant confirms a reservation near the Louvre.",
            user_description="A traveller planning a weekend in Paris.",
            turns=[
                {
                    "role": "user",
                    "content": "I need a hotel in Paris near the Louvre."
                },
                {
                    "role": "assistant",
                    "content": "Hôtel du Louvre has rooms available. Which dates?"
                }
            ],
            context=[
                "Hôtel du Louvre is a five-minute walk from the museum."
            ],
            additional_metadata={"source": "faq"},
            comments="Reviewed by the support team.",
            source_file="capitals.csv",
            source_files=["capitals.csv"],
            finalized=True,
            custom_column_key_values={"difficulty": "easy"},
            images_mapping={
                "map": {"url": "https://example.com/paris.png", "local": False}
            },
            tags=["geography"]
        )
    ],
)

For async mode, call a_queue_goldens and await it as shown below:

result = await dataset.a_queue_goldens(...)

Parameters

Parameter	Type	Description
`goldens`	`List[GoldenRequest]`	Required. The goldens to queue for review. Every golden in one request must be of the same kind and match the dataset's `multiTurn`. They are stored unfinalized, whatever each golden's own `finalized` says. See GoldenRequest.

import { ConfidentAI } from "confident-ai";

const client = new ConfidentAI();

const dataset = client.dataset("<DATASET-ID>");
const result = await dataset.queueGoldens(
  [
    {
      scenario: "A traveller wants to book a hotel in Paris.",
      expectedOutcome: "The assistant confirms a reservation near the Louvre.",
      userDescription: "A traveller planning a weekend in Paris.",
      turns: [
          {
            role: "user",
            content: "I need a hotel in Paris near the Louvre."
          },
          {
            role: "assistant",
            content: "Hôtel du Louvre has rooms available. Which dates?"
          }
        ],
      context: ["Hôtel du Louvre is a five-minute walk from the museum."],
      additionalMetadata: { source: "faq" },
      comments: "Reviewed by the support team.",
      sourceFile: "capitals.csv",
      sourceFiles: ["capitals.csv"],
      finalized: true,
      customColumnKeyValues: { difficulty: "easy" },
      imagesMapping: {
          map: { url: "https://example.com/paris.png", local: false }
        },
      tags: ["geography"]
    }
  ],
);

Parameters

Parameter	Type	Description
`goldens`	`GoldenRequest[]`	Required. The goldens to queue for review. Every golden in one request must be of the same kind and match the dataset's `multiTurn`. They are stored unfinalized, whatever each golden's own `finalized` says. See GoldenRequest.

Returns

This method returns an object of type DatasetRef.

Pull Dataset

Retrieves the dataset with its goldens, oldest first. Pass version to pull a specific version, and finalized=false to pull the goldens still awaiting review instead of the finalized ones. Requires an active trial or paid plan, and the Team plan or above to pull a version.

from confident_ai import ConfidentAI

client = ConfidentAI()

dataset = client.dataset(dataset_id="<DATASET-ID>")
result = dataset.pull(version="00.00.01", finalized="true")

For async mode, call a_pull and await it as shown below:

result = await dataset.a_pull(...)

Parameters

Parameter	Type	Description
`version`	`Optional[str]`	The version to pull. Defaults to the latest version, or to the unversioned goldens when the dataset has no versions. Requires the Team plan or above.
`finalized`	`Optional[Literal['true', 'false']]`	Whether to pull the finalized goldens, or with `false` the goldens still awaiting review. Defaults to `true`.

import { ConfidentAI } from "confident-ai";

const client = new ConfidentAI();

const dataset = client.dataset("<DATASET-ID>");
const result = await dataset.pull(
  { version: "00.00.01", finalized: "true" },
);

Parameters

Parameter	Type	Description
`version`	`string`	The version to pull. Defaults to the latest version, or to the unversioned goldens when the dataset has no versions. Requires the Team plan or above.
`finalized`	`"true" \| "false"`	Whether to pull the finalized goldens, or with `false` the goldens still awaiting review. Defaults to `true`.

Returns

This method returns an object of type Dataset.

Push Dataset

Adds goldens to a dataset, creating it when the alias names none yet, and returns the dataset's id. Pushing to a version requires the Team plan or above.

from confident_ai import ConfidentAI

client = ConfidentAI()

dataset = client.dataset(alias="capitals")
result = dataset.push(finalized=True)

For async mode, call a_push and await it as shown below:

result = await dataset.a_push(...)

Parameters

Parameter	Type	Description
`finalized`	`Optional[bool]`	Whether the goldens pushed are finalized, that is ready to use in evaluations. Applies to every golden in this request.

import { ConfidentAI } from "confident-ai";

const client = new ConfidentAI();

const dataset = client.dataset(undefined, { alias: "capitals" });
const result = await dataset.push({ finalized: true });

Parameters

Parameter	Type	Description
`finalized`	`boolean`	Whether the goldens pushed are finalized, that is ready to use in evaluations. Applies to every golden in this request.

Returns

This method returns an object of type DatasetRef.

Methods (Stateless)

These methods take every argument themselves, so a caller reaches them through client.datasets without opening a Dataset first.

List Datasets

Lists all the datasets in your Confident AI project, newest first, without their goldens.

from confident_ai import ConfidentAI

client = ConfidentAI()

result = client.datasets.list()

For async mode, call a_list and await it as shown below:

result = await client.datasets.a_list(...)

import { ConfidentAI } from "confident-ai";

const client = new ConfidentAI();

const result = await client.datasets.list();

Returns

This method returns an object of type DatasetList.

Push Dataset

Adds goldens to a dataset, creating it when the alias names none yet, and returns the dataset's id. Pushing to a version requires the Team plan or above.

from confident_ai import ConfidentAI
from confident_ai.datasets import PushSingleTurnGolden
from confident_ai.common import ToolCall
from confident_ai.common import ToolCallType

client = ConfidentAI()

result = client.datasets.push(
    alias="capitals",
    goldens=[
        PushSingleTurnGolden(
            input="What is the capital of France?",
            actual_output="The capital of France is Paris.",
            expected_output="Paris.",
            context=["Paris is the capital of France."],
            retrieval_context=[
                "Paris is the capital and largest city of France."
            ],
            tools_called=[
                ToolCall(
                    name="get_landmark_info",
                    type=ToolCallType.FUNCTION,
                    description="This tool gives information about a mountain.",
                    input_parameters={"mountain": "Everest"},
                    output="8,848 metres",
                    reasoning="The user asked for the height of a mountain."
                )
            ],
            expected_tools=[
                ToolCall(
                    name="get_landmark_info",
                    type=ToolCallType.FUNCTION,
                    description="This tool gives information about a mountain.",
                    input_parameters={"mountain": "Everest"},
                    output="8,848 metres",
                    reasoning="The user asked for the height of a mountain."
                )
            ],
            token_cost=0.002,
            input_token_count=12,
            output_token_count=3,
            additional_metadata={"source": "faq"},
            comments="Reviewed by the support team.",
            source_file="capitals.csv",
            source_files=["capitals.csv"],
            finalized=True,
            custom_column_key_values={"difficulty": "easy"},
            images_mapping={
                "map": {"url": "https://example.com/paris.png", "local": False}
            },
            tags=["geography"],
            id="<GOLDEN-ID>"
        )
    ],
    finalized=True,
    version="00.00.01",
)

For async mode, call a_push and await it as shown below:

result = await client.datasets.a_push(...)

Parameters

Parameter	Type	Description
`alias`	`str`	Required. The alias of the dataset, unique within your project. A new dataset is created when no dataset with this alias exists.
`goldens`	`List[PushGolden]`	Required. The goldens to push. A golden carrying an `id` updates the golden already in the dataset; one without is added. Every golden in one request must be of the same kind — all single- turn, or all multi-turn — and match the dataset's `multiTurn`. A new dataset takes its kind from them. See PushGolden.
`finalized`	`Optional[bool]`	Whether the goldens pushed are finalized, that is ready to use in evaluations. Applies to every golden in this request.
`version`	`Optional[str]`	The dataset version to push the goldens onto, for example `00.00.01`. When the dataset has versions, omitting it pushes to the latest version; when it has none, omitting it leaves the goldens unversioned. A version cannot be given for a dataset that does not exist yet. Requires the Team plan or above.

import { ConfidentAI } from "confident-ai";
import { ToolCallType } from "confident-ai/common";

const client = new ConfidentAI();

const result = await client.datasets.push(
  "capitals",
  [
    {
      input: "What is the capital of France?",
      actualOutput: "The capital of France is Paris.",
      expectedOutput: "Paris.",
      context: ["Paris is the capital of France."],
      retrievalContext: [
          "Paris is the capital and largest city of France."
        ],
      toolsCalled: [
          {
            name: "get_landmark_info",
            type: ToolCallType.FUNCTION,
            description: "This tool gives information about a mountain.",
            inputParameters: { mountain: "Everest" },
            output: "8,848 metres",
            reasoning: "The user asked for the height of a mountain."
          }
        ],
      expectedTools: [
          {
            name: "get_landmark_info",
            type: ToolCallType.FUNCTION,
            description: "This tool gives information about a mountain.",
            inputParameters: { mountain: "Everest" },
            output: "8,848 metres",
            reasoning: "The user asked for the height of a mountain."
          }
        ],
      tokenCost: 0.002,
      inputTokenCount: 12,
      outputTokenCount: 3,
      additionalMetadata: { source: "faq" },
      comments: "Reviewed by the support team.",
      sourceFile: "capitals.csv",
      sourceFiles: ["capitals.csv"],
      finalized: true,
      customColumnKeyValues: { difficulty: "easy" },
      imagesMapping: {
          map: { url: "https://example.com/paris.png", local: false }
        },
      tags: ["geography"],
      id: "<GOLDEN-ID>"
    }
  ],
  { finalized: true, version: "00.00.01" },
);

Parameters

Parameter	Type	Description
`alias`	`string`	Required. The alias of the dataset, unique within your project. A new dataset is created when no dataset with this alias exists.
`goldens`	`PushGolden[]`	Required. The goldens to push. A golden carrying an `id` updates the golden already in the dataset; one without is added. Every golden in one request must be of the same kind — all single- turn, or all multi-turn — and match the dataset's `multiTurn`. A new dataset takes its kind from them. See PushGolden.
`finalized`	`boolean`	Whether the goldens pushed are finalized, that is ready to use in evaluations. Applies to every golden in this request.
`version`	`string`	The dataset version to push the goldens onto, for example `00.00.01`. When the dataset has versions, omitting it pushes to the latest version; when it has none, omitting it leaves the goldens unversioned. A version cannot be given for a dataset that does not exist yet. Requires the Team plan or above.

Returns

This method returns an object of type DatasetRef.

Pull Dataset

from confident_ai import ConfidentAI

client = ConfidentAI()

result = client.datasets.pull(
    dataset_id="<DATASET-ID>",
    version="00.00.01",
    finalized="true",
)

For async mode, call a_pull and await it as shown below:

result = await client.datasets.a_pull(...)

Parameters

Parameter	Type	Description
`dataset_id`	`str`	Required. The unique id of the dataset.
`version`	`Optional[str]`	The version to pull. Defaults to the latest version, or to the unversioned goldens when the dataset has no versions. Requires the Team plan or above.
`finalized`	`Optional[Literal['true', 'false']]`	Whether to pull the finalized goldens, or with `false` the goldens still awaiting review. Defaults to `true`.

import { ConfidentAI } from "confident-ai";

const client = new ConfidentAI();

const result = await client.datasets.pull(
  "<DATASET-ID>",
  { version: "00.00.01", finalized: "true" },
);

Parameters

Parameter	Type	Description
`datasetId`	`string`	Required. The unique id of the dataset.
`version`	`string`	The version to pull. Defaults to the latest version, or to the unversioned goldens when the dataset has no versions. Requires the Team plan or above.
`finalized`	`"true" \| "false"`	Whether to pull the finalized goldens, or with `false` the goldens still awaiting review. Defaults to `true`.

Returns

This method returns an object of type Dataset.

Delete Dataset

Permanently deletes the dataset and everything in it: its goldens, versions, tags and custom columns. This action cannot be undone.

from confident_ai import ConfidentAI

client = ConfidentAI()

result = client.datasets.delete(dataset_id="<DATASET-ID>")

For async mode, call a_delete and await it as shown below:

result = await client.datasets.a_delete(...)

Parameters

Parameter	Type	Description
`dataset_id`	`str`	Required. The unique id of the dataset.

import { ConfidentAI } from "confident-ai";

const client = new ConfidentAI();

const result = await client.datasets.delete("<DATASET-ID>");

Parameters

Parameter	Type	Description
`datasetId`	`string`	Required. The unique id of the dataset.

Returns

This method returns an object of type DatasetRef.

Single Turn Golden

Adds goldens to the dataset as unfinalized goldens, for review on the platform before they are used in evaluations.

A single-turn golden to write: one input to your LLM application and the outputs expected of it.

from confident_ai import ConfidentAI
from confident_ai.datasets import SingleTurnGoldenRequest
from confident_ai.common import ToolCall
from confident_ai.common import ToolCallType

client = ConfidentAI()

result = client.datasets.queue_goldens(
    dataset_id="<DATASET-ID>",
    goldens=[
        SingleTurnGoldenRequest(
            input="What is the capital of France?",
            actual_output="The capital of France is Paris.",
            expected_output="Paris.",
            context=["Paris is the capital of France."],
            retrieval_context=[
                "Paris is the capital and largest city of France."
            ],
            tools_called=[
                ToolCall(
                    name="get_landmark_info",
                    type=ToolCallType.FUNCTION,
                    description="This tool gives information about a mountain.",
                    input_parameters={"mountain": "Everest"},
                    output="8,848 metres",
                    reasoning="The user asked for the height of a mountain."
                )
            ],
            expected_tools=[
                ToolCall(
                    name="get_landmark_info",
                    type=ToolCallType.FUNCTION,
                    description="This tool gives information about a mountain.",
                    input_parameters={"mountain": "Everest"},
                    output="8,848 metres",
                    reasoning="The user asked for the height of a mountain."
                )
            ],
            token_cost=0.002,
            input_token_count=12,
            output_token_count=3,
            additional_metadata={"source": "faq"},
            comments="Reviewed by the support team.",
            source_file="capitals.csv",
            source_files=["capitals.csv"],
            finalized=True,
            custom_column_key_values={"difficulty": "easy"},
            images_mapping={
                "map": {"url": "https://example.com/paris.png", "local": False}
            },
            tags=["geography"]
        )
    ],
)

For async mode, call a_queue_goldens and await it as shown below:

result = await client.datasets.a_queue_goldens(...)

Parameters

Parameter	Type	Description
`dataset_id`	`str`	Required. The unique id of the dataset.
`goldens`	`List[GoldenRequest]`	Required. The goldens to queue for review. Every golden in one request must be of the same kind and match the dataset's `multiTurn`. They are stored unfinalized, whatever each golden's own `finalized` says. See GoldenRequest.

import { ConfidentAI } from "confident-ai";
import { ToolCallType } from "confident-ai/common";

const client = new ConfidentAI();

const result = await client.datasets.queueGoldens(
  "<DATASET-ID>",
  [
    {
      input: "What is the capital of France?",
      actualOutput: "The capital of France is Paris.",
      expectedOutput: "Paris.",
      context: ["Paris is the capital of France."],
      retrievalContext: [
          "Paris is the capital and largest city of France."
        ],
      toolsCalled: [
          {
            name: "get_landmark_info",
            type: ToolCallType.FUNCTION,
            description: "This tool gives information about a mountain.",
            inputParameters: { mountain: "Everest" },
            output: "8,848 metres",
            reasoning: "The user asked for the height of a mountain."
          }
        ],
      expectedTools: [
          {
            name: "get_landmark_info",
            type: ToolCallType.FUNCTION,
            description: "This tool gives information about a mountain.",
            inputParameters: { mountain: "Everest" },
            output: "8,848 metres",
            reasoning: "The user asked for the height of a mountain."
          }
        ],
      tokenCost: 0.002,
      inputTokenCount: 12,
      outputTokenCount: 3,
      additionalMetadata: { source: "faq" },
      comments: "Reviewed by the support team.",
      sourceFile: "capitals.csv",
      sourceFiles: ["capitals.csv"],
      finalized: true,
      customColumnKeyValues: { difficulty: "easy" },
      imagesMapping: {
          map: { url: "https://example.com/paris.png", local: false }
        },
      tags: ["geography"]
    }
  ],
);

Parameters

Parameter	Type	Description
`datasetId`	`string`	Required. The unique id of the dataset.
`goldens`	`GoldenRequest[]`	Required. The goldens to queue for review. Every golden in one request must be of the same kind and match the dataset's `multiTurn`. They are stored unfinalized, whatever each golden's own `finalized` says. See GoldenRequest.

Returns

This method returns an object of type DatasetRef.

Multi Turn Golden

Adds goldens to the dataset as unfinalized goldens, for review on the platform before they are used in evaluations.

A multi-turn golden to write: the scenario of a conversation with your LLM application and, optionally, its turns.

from confident_ai import ConfidentAI
from confident_ai.datasets import MultiTurnGoldenRequest

client = ConfidentAI()

result = client.datasets.queue_goldens(
    dataset_id="<DATASET-ID>",
    goldens=[
        MultiTurnGoldenRequest(
            scenario="A traveller wants to book a hotel in Paris.",
            expected_outcome="The assistant confirms a reservation near the Louvre.",
            user_description="A traveller planning a weekend in Paris.",
            turns=[
                {
                    "role": "user",
                    "content": "I need a hotel in Paris near the Louvre."
                },
                {
                    "role": "assistant",
                    "content": "Hôtel du Louvre has rooms available. Which dates?"
                }
            ],
            context=[
                "Hôtel du Louvre is a five-minute walk from the museum."
            ],
            additional_metadata={"source": "faq"},
            comments="Reviewed by the support team.",
            source_file="capitals.csv",
            source_files=["capitals.csv"],
            finalized=True,
            custom_column_key_values={"difficulty": "easy"},
            images_mapping={
                "map": {"url": "https://example.com/paris.png", "local": False}
            },
            tags=["geography"]
        )
    ],
)

For async mode, call a_queue_goldens and await it as shown below:

result = await client.datasets.a_queue_goldens(...)

Parameters

Parameter	Type	Description
`dataset_id`	`str`	Required. The unique id of the dataset.
`goldens`	`List[GoldenRequest]`	Required. The goldens to queue for review. Every golden in one request must be of the same kind and match the dataset's `multiTurn`. They are stored unfinalized, whatever each golden's own `finalized` says. See GoldenRequest.

import { ConfidentAI } from "confident-ai";

const client = new ConfidentAI();

const result = await client.datasets.queueGoldens(
  "<DATASET-ID>",
  [
    {
      scenario: "A traveller wants to book a hotel in Paris.",
      expectedOutcome: "The assistant confirms a reservation near the Louvre.",
      userDescription: "A traveller planning a weekend in Paris.",
      turns: [
          {
            role: "user",
            content: "I need a hotel in Paris near the Louvre."
          },
          {
            role: "assistant",
            content: "Hôtel du Louvre has rooms available. Which dates?"
          }
        ],
      context: ["Hôtel du Louvre is a five-minute walk from the museum."],
      additionalMetadata: { source: "faq" },
      comments: "Reviewed by the support team.",
      sourceFile: "capitals.csv",
      sourceFiles: ["capitals.csv"],
      finalized: true,
      customColumnKeyValues: { difficulty: "easy" },
      imagesMapping: {
          map: { url: "https://example.com/paris.png", local: false }
        },
      tags: ["geography"]
    }
  ],
);

Parameters

Parameter	Type	Description
`datasetId`	`string`	Required. The unique id of the dataset.
`goldens`	`GoldenRequest[]`	Required. The goldens to queue for review. Every golden in one request must be of the same kind and match the dataset's `multiTurn`. They are stored unfinalized, whatever each golden's own `finalized` says. See GoldenRequest.

Returns

This method returns an object of type DatasetRef.

Run Evaluation

from confident_ai import ConfidentAI
from confident_ai.common import GenerationMode

client = ConfidentAI()

result = client.datasets.run_evaluation(
    dataset_id="<DATASET-ID>",
    metric_collection="Answer Quality",
    identifier="Nightly regression",
    version="00.00.01",
    ai_connection_id="<AI-CONNECTION-ID>",
    prompt_alias="capital-lookup",
    prompt_commit="bab04ce",
    generation_mode=GenerationMode.AI_CONNECTION,
    variables_mapping={"question": "Input"},
    include_simulation=False,
    max_concurrent_generation=5,
    generation_timeout=60,
    num_generations=1,
    mcp_server_ids=["<MCP-SERVER-ID>"],
)

For async mode, call a_run_evaluation and await it as shown below:

result = await client.datasets.a_run_evaluation(...)

Parameters

Parameter	Type	Description
`dataset_id`	`str`	Required. The unique id of the dataset.
`metric_collection`	`str`	Required. The name of the metric collection to evaluate against. Names come from the list metric collections endpoint.
`identifier`	`Optional[str]`	A label for the resulting test run, used to recognise it in the test runs list.
`version`	`Optional[str]`	The dataset version to evaluate. Omit this field to evaluate the latest version.
`ai_connection_id`	`Optional[str]`	The id of the AI connection used to generate the actual outputs before evaluating them. Required when `generationMode` is AI_CONNECTION, and not allowed together with `promptAlias`.
`prompt_alias`	`Optional[str]`	The alias of the prompt used to generate the actual outputs before evaluating them. Required when `generationMode` is PROMPT, and not allowed together with `aiConnectionId`.
`prompt_commit`	`Optional[str]`	The prompt commit hash to generate with. Requires `promptAlias`. Omit this field to generate with the latest commit on the prompt's main branch.
`generation_mode`	`Optional[GenerationMode]`	See GenerationMode.
`variables_mapping`	`Optional[Dict[str, str]]`	Maps each variable in the prompt to the golden field it is interpolated with, such as `Input` or `Expected Output`, or to a dataset custom column key. This field applies only when generating from a prompt.
`include_simulation`	`Optional[bool]`	Whether to simulate a conversation for each golden before evaluating it, for multi-turn datasets. Every golden needs a `scenario` when this is enabled, and `turns` when it is disabled.
`max_concurrent_generation`	`Optional[int]`	The maximum number of generation calls to run in parallel. An AI connection's own `maxConcurrency` takes precedence over this value.
`generation_timeout`	`Optional[int]`	The number of seconds to wait for a single generation before it is marked as errored.
`num_generations`	`Optional[int]`	How many times to run each golden, so a single outlier response does not skew the results. Omit this field to use the AI connection's `defaultNumGenerations`, which is 1 when the connection does not set one.
`mcp_server_ids`	`Optional[List[str]]`	The ids of the MCP servers to attach to the run. A tool call whose name matches a tool exposed by one of these servers is labeled an MCP tool call rather than a function call.

import { ConfidentAI } from "confident-ai";
import { GenerationMode } from "confident-ai/common";

const client = new ConfidentAI();

const result = await client.datasets.runEvaluation(
  "<DATASET-ID>",
  "Answer Quality",
  {
    identifier: "Nightly regression",
    version: "00.00.01",
    aiConnectionId: "<AI-CONNECTION-ID>",
    promptAlias: "capital-lookup",
    promptCommit: "bab04ce",
    generationMode: GenerationMode.AI_CONNECTION,
    variablesMapping: { question: "Input" },
    includeSimulation: false,
    maxConcurrentGeneration: 5,
    generationTimeout: 60,
    numGenerations: 1,
    mcpServerIds: ["<MCP-SERVER-ID>"]
  },
);

Parameters

Parameter	Type	Description
`datasetId`	`string`	Required. The unique id of the dataset.
`metricCollection`	`string`	Required. The name of the metric collection to evaluate against. Names come from the list metric collections endpoint.
`identifier`	`string`	A label for the resulting test run, used to recognise it in the test runs list.
`version`	`string`	The dataset version to evaluate. Omit this field to evaluate the latest version.
`aiConnectionId`	`string`	The id of the AI connection used to generate the actual outputs before evaluating them. Required when `generationMode` is AI_CONNECTION, and not allowed together with `promptAlias`.
`promptAlias`	`string`	The alias of the prompt used to generate the actual outputs before evaluating them. Required when `generationMode` is PROMPT, and not allowed together with `aiConnectionId`.
`promptCommit`	`string`	The prompt commit hash to generate with. Requires `promptAlias`. Omit this field to generate with the latest commit on the prompt's main branch.
`generationMode`	`GenerationMode`	See GenerationMode.
`variablesMapping`	`Record<string, string>`	Maps each variable in the prompt to the golden field it is interpolated with, such as `Input` or `Expected Output`, or to a dataset custom column key. This field applies only when generating from a prompt.
`includeSimulation`	`boolean`	Whether to simulate a conversation for each golden before evaluating it, for multi-turn datasets. Every golden needs a `scenario` when this is enabled, and `turns` when it is disabled.
`maxConcurrentGeneration`	`number`	The maximum number of generation calls to run in parallel. An AI connection's own `maxConcurrency` takes precedence over this value.
`generationTimeout`	`number`	The number of seconds to wait for a single generation before it is marked as errored.
`numGenerations`	`number`	How many times to run each golden, so a single outlier response does not skew the results. Omit this field to use the AI connection's `defaultNumGenerations`, which is 1 when the connection does not set one.
`mcpServerIds`	`string[]`	The ids of the MCP servers to attach to the run. A tool call whose name matches a tool exposed by one of these servers is labeled an MCP tool call rather than a function call.

Returns

This method returns an object of type RunDatasetEvaluationResult.

Types

Dataset

A pulled dataset with the goldens of one version.

class Dataset:
    id: str
    alias: str
    multi_turn: bool = Field(alias="multiTurn")
    version: Optional[str]
    goldens: List[Golden]

idstrRequired

This is the unique id of the dataset.

Example: "<DATASET-ID>"

aliasstrRequired

This is the alias of the dataset, which is unique within your project.

Example: "capitals"

multi_turnboolRequired

This is true if the dataset is multi-turn, which contains multi-turn goldens. Single-turn datasets have multiTurn set to false and contain single-turn goldens.

Example: false

versionOptional[str]Required

The version number of the goldens returned, or null when the dataset has no versions.

Example: "00.00.01"

goldensList[Golden]Required

The goldens in the dataset, oldest first. Every golden is single-turn or multi-turn according to the dataset's multiTurn.

See Golden.

interface Dataset {
  id: string;
  alias: string;
  multiTurn: boolean;
  version: string | null;
  goldens: Golden[];
}

idstringRequired

This is the unique id of the dataset.

Example: "<DATASET-ID>"

aliasstringRequired

This is the alias of the dataset, which is unique within your project.

Example: "capitals"

multiTurnbooleanRequired

This is true if the dataset is multi-turn, which contains multi-turn goldens. Single-turn datasets have multiTurn set to false and contain single-turn goldens.

Example: false

versionstring | nullRequired

The version number of the goldens returned, or null when the dataset has no versions.

Example: "00.00.01"

goldensGolden[]Required

The goldens in the dataset, oldest first. Every golden is single-turn or multi-turn according to the dataset's multiTurn.

See Golden.

DatasetList

class DatasetList:
    datasets: List[DatasetSummary]

datasetsList[DatasetSummary]Required

This is the list of datasets in your project, newest first.

See DatasetSummary.

interface DatasetList {
  datasets: DatasetSummary[];
}

datasetsDatasetSummary[]Required

This is the list of datasets in your project, newest first.

See DatasetSummary.

DatasetRef

class DatasetRef:
    id: str

idstrRequired

This is the unique id of the dataset.

Example: "<DATASET-ID>"

interface DatasetRef {
  id: string;
}

idstringRequired

This is the unique id of the dataset.

Example: "<DATASET-ID>"

DatasetSummary

A dataset as it appears in your project's dataset list, without any goldens.

class DatasetSummary:
    id: str
    alias: str
    multi_turn: bool = Field(alias="multiTurn")

idstrRequired

This is the unique id of the dataset.

Example: "<DATASET-ID>"

aliasstrRequired

This is the alias of the dataset, which is unique within your project.

Example: "capitals"

multi_turnboolRequired

This is true if the dataset is multi-turn, which contains multi-turn goldens. Single-turn datasets have multiTurn set to false and contain single-turn goldens.

Example: false

interface DatasetSummary {
  id: string;
  alias: string;
  multiTurn: boolean;
}

idstringRequired

This is the unique id of the dataset.

Example: "<DATASET-ID>"

aliasstringRequired

This is the alias of the dataset, which is unique within your project.

Example: "capitals"

multiTurnbooleanRequired

This is true if the dataset is multi-turn, which contains multi-turn goldens. Single-turn datasets have multiTurn set to false and contain single-turn goldens.

Example: false

GenerationMode

Where the actual outputs come from when running a dataset: AI_CONNECTION generates them with an AI connection, PROMPT with a prompt. Omit it when you supply at most one of aiConnectionId or promptAlias, and Confident AI infers the mode from whichever you sent.

class GenerationMode(Enum):
    AI_CONNECTION = "AI_CONNECTION"
    PROMPT = "PROMPT"

enum GenerationMode {
  AI_CONNECTION = "AI_CONNECTION",
  PROMPT = "PROMPT",
}

AI_CONNECTION · PROMPT

Golden

A golden in the dataset: single-turn when it carries input, multi-turn when it carries scenario. The dataset's multiTurn decides which kind every golden in it is.

Golden = Union[
    SingleTurnGolden,
    MultiTurnGolden,
]

type Golden =
  | SingleTurnGolden
  | MultiTurnGolden;

A Golden is one of the shapes below. Send the fields of one of them, never a mix of both.

A single-turn golden as stored in the dataset.

class SingleTurnGolden:
    input: str
    actual_output: Optional[str] = Field(default=None, alias="actualOutput")
    expected_output: Optional[str] = Field(default=None, alias="expectedOutput")
    context: Optional[List[str]] = None
    retrieval_context: Optional[List[str]] = Field(default=None, alias="retrievalContext")
    tools_called: Optional[List[ToolCall]] = Field(default=None, alias="toolsCalled")
    expected_tools: Optional[List[ToolCall]] = Field(default=None, alias="expectedTools")
    token_cost: Optional[float] = Field(default=None, alias="tokenCost")
    input_token_count: Optional[int] = Field(default=None, alias="inputTokenCount")
    output_token_count: Optional[int] = Field(default=None, alias="outputTokenCount")
    id: Optional[str] = None
    additional_metadata: Optional[Dict[str, Any]] = Field(default=None, alias="additionalMetadata")
    comments: Optional[str] = None
    source_file: Optional[str] = Field(default=None, alias="sourceFile")
    source_files: Optional[List[str]] = Field(default=None, alias="sourceFiles")
    finalized: Optional[bool] = None
    custom_column_key_values: Optional[Dict[str, str]] = Field(default=None, alias="customColumnKeyValues")
    tags: Optional[List[str]] = None

inputstrRequired

This is the input to your LLM application.

Example: "What is the capital of France?"

actual_outputOptional[str]

This is the actual output of your LLM application.

Example: "The capital of France is Paris."

expected_outputOptional[str]

This is the expected output of your LLM application, which is the ideal actual output.

Example: "Paris."

contextOptional[List[str]]

This is the ideal retrieval context of your LLM application.

Example: ["Paris is the capital of France."]

retrieval_contextOptional[List[str]]

This is the retrieval context of your LLM application.

Example: ["Paris is the capital and largest city of France."]

tools_calledOptional[List[ToolCall]]

This is the tools called by your LLM application.

See ToolCall.

expected_toolsOptional[List[ToolCall]]

This is the expected tools to be called by the LLM application.

See ToolCall.

token_costOptional[float]

This is the cost of the tokens used to produce the actual output.

Example: 0.002

input_token_countOptional[int]

This is the number of input tokens passed to the LLM model.

Example: 12

output_token_countOptional[int]

This is the number of output tokens generated by the LLM model.

Example: 3

idOptional[str]

The id of the golden assigned by Confident AI. Use it to get, update or delete this golden.

Example: "<GOLDEN-ID>"

additional_metadataOptional[Dict[str, Any]]

This is any additional metadata associated with the golden.

Example: {"source":"faq"}

commentsOptional[str]

This is any comments associated with the golden.

Example: "Reviewed by the support team."

source_fileOptional[str]

This is the source file from which the golden was retrieved.

Example: "capitals.csv"

source_filesOptional[List[str]]

These are the source files the golden was retrieved from.

Example: ["capitals.csv"]

finalizedOptional[bool]

This is true when the golden is finalized and ready to use in evaluations.

Example: true

custom_column_key_valuesOptional[Dict[str, str]]

Key-value pairs representing custom table column data for this golden. Keys correspond to the custom column keys defined in the dataset. Absent when the golden has no custom column values.

Example: {"difficulty":"easy"}

tagsOptional[List[str]]

These are the tags associated with the golden.

Example: ["geography"]

interface SingleTurnGolden {
  input: string;
  actualOutput?: string | null;
  expectedOutput?: string | null;
  context?: string[] | null;
  retrievalContext?: string[] | null;
  toolsCalled?: ToolCall[] | null;
  expectedTools?: ToolCall[] | null;
  tokenCost?: number | null;
  inputTokenCount?: number | null;
  outputTokenCount?: number | null;
  id?: string;
  additionalMetadata?: Record<string, unknown> | null;
  comments?: string | null;
  sourceFile?: string | null;
  sourceFiles?: string[];
  finalized?: boolean;
  customColumnKeyValues?: Record<string, string>;
  tags?: string[];
}

inputstringRequired

This is the input to your LLM application.

Example: "What is the capital of France?"

actualOutputstring | null

This is the actual output of your LLM application.

Example: "The capital of France is Paris."

expectedOutputstring | null

This is the expected output of your LLM application, which is the ideal actual output.

Example: "Paris."

contextstring[] | null

This is the ideal retrieval context of your LLM application.

Example: ["Paris is the capital of France."]

retrievalContextstring[] | null

This is the retrieval context of your LLM application.

Example: ["Paris is the capital and largest city of France."]

toolsCalledToolCall[] | null

This is the tools called by your LLM application.

See ToolCall.

expectedToolsToolCall[] | null

This is the expected tools to be called by the LLM application.

See ToolCall.

tokenCostnumber | null

This is the cost of the tokens used to produce the actual output.

Example: 0.002

inputTokenCountnumber | null

This is the number of input tokens passed to the LLM model.

Example: 12

outputTokenCountnumber | null

This is the number of output tokens generated by the LLM model.

Example: 3

idstring

The id of the golden assigned by Confident AI. Use it to get, update or delete this golden.

Example: "<GOLDEN-ID>"

additionalMetadataRecord<string, unknown> | null

This is any additional metadata associated with the golden.

Example: {"source":"faq"}

commentsstring | null

This is any comments associated with the golden.

Example: "Reviewed by the support team."

sourceFilestring | null

This is the source file from which the golden was retrieved.

Example: "capitals.csv"

sourceFilesstring[]

These are the source files the golden was retrieved from.

Example: ["capitals.csv"]

finalizedboolean

This is true when the golden is finalized and ready to use in evaluations.

Example: true

customColumnKeyValuesRecord<string, string>

Key-value pairs representing custom table column data for this golden. Keys correspond to the custom column keys defined in the dataset. Absent when the golden has no custom column values.

Example: {"difficulty":"easy"}

tagsstring[]

These are the tags associated with the golden.

Example: ["geography"]

A multi-turn golden as stored in the dataset.

class MultiTurnGolden:
    scenario: str
    expected_outcome: Optional[str] = Field(default=None, alias="expectedOutcome")
    user_description: Optional[str] = Field(default=None, alias="userDescription")
    turns: Optional[List[Turn]] = None
    context: Optional[List[str]] = None
    id: Optional[str] = None
    additional_metadata: Optional[Dict[str, Any]] = Field(default=None, alias="additionalMetadata")
    comments: Optional[str] = None
    source_file: Optional[str] = Field(default=None, alias="sourceFile")
    source_files: Optional[List[str]] = Field(default=None, alias="sourceFiles")
    finalized: Optional[bool] = None
    custom_column_key_values: Optional[Dict[str, str]] = Field(default=None, alias="customColumnKeyValues")
    tags: Optional[List[str]] = None

scenariostrRequired

This is a description of the conversation context.

Example: "A traveller wants to book a hotel in Paris."

expected_outcomeOptional[str]

This describes the expected outcome, or ideal conversation flow, of the conversation.

Example: "The assistant confirms a reservation near the Louvre."

user_descriptionOptional[str]

This is the description of the user in the conversation.

Example: "A traveller planning a weekend in Paris."

turnsOptional[List[Turn]]

This is the list of turns in the conversation.

See Turn.

Example: [{"role":"user","content":"I need a hotel in Paris near the Louvre."},{"role":"assistant","content":"Hôtel du Louvre has rooms available. Which dates?"}]

contextOptional[List[str]]

This is the context of the conversation.

Example: ["Hôtel du Louvre is a five-minute walk from the museum."]

idOptional[str]

The id of the golden assigned by Confident AI. Use it to get, update or delete this golden.

Example: "<GOLDEN-ID>"

additional_metadataOptional[Dict[str, Any]]

This is any additional metadata associated with the golden.

Example: {"source":"faq"}

commentsOptional[str]

This is any comments associated with the golden.

Example: "Reviewed by the support team."

source_fileOptional[str]

This is the source file from which the golden was retrieved.

Example: "capitals.csv"

source_filesOptional[List[str]]

These are the source files the golden was retrieved from.

Example: ["capitals.csv"]

finalizedOptional[bool]

This is true when the golden is finalized and ready to use in evaluations.

Example: true

custom_column_key_valuesOptional[Dict[str, str]]

Key-value pairs representing custom table column data for this golden. Keys correspond to the custom column keys defined in the dataset. Absent when the golden has no custom column values.

Example: {"difficulty":"easy"}

tagsOptional[List[str]]

These are the tags associated with the golden.

Example: ["geography"]

interface MultiTurnGolden {
  scenario: string;
  expectedOutcome?: string | null;
  userDescription?: string | null;
  turns?: Turn[] | null;
  context?: string[] | null;
  id?: string;
  additionalMetadata?: Record<string, unknown> | null;
  comments?: string | null;
  sourceFile?: string | null;
  sourceFiles?: string[];
  finalized?: boolean;
  customColumnKeyValues?: Record<string, string>;
  tags?: string[];
}

scenariostringRequired

This is a description of the conversation context.

Example: "A traveller wants to book a hotel in Paris."

expectedOutcomestring | null

This describes the expected outcome, or ideal conversation flow, of the conversation.

Example: "The assistant confirms a reservation near the Louvre."

userDescriptionstring | null

This is the description of the user in the conversation.

Example: "A traveller planning a weekend in Paris."

turnsTurn[] | null

This is the list of turns in the conversation.

See Turn.

Example: [{"role":"user","content":"I need a hotel in Paris near the Louvre."},{"role":"assistant","content":"Hôtel du Louvre has rooms available. Which dates?"}]

contextstring[] | null

This is the context of the conversation.

Example: ["Hôtel du Louvre is a five-minute walk from the museum."]

idstring

The id of the golden assigned by Confident AI. Use it to get, update or delete this golden.

Example: "<GOLDEN-ID>"

additionalMetadataRecord<string, unknown> | null

This is any additional metadata associated with the golden.

Example: {"source":"faq"}

commentsstring | null

This is any comments associated with the golden.

Example: "Reviewed by the support team."

sourceFilestring | null

This is the source file from which the golden was retrieved.

Example: "capitals.csv"

sourceFilesstring[]

These are the source files the golden was retrieved from.

Example: ["capitals.csv"]

finalizedboolean

This is true when the golden is finalized and ready to use in evaluations.

Example: true

customColumnKeyValuesRecord<string, string>

Key-value pairs representing custom table column data for this golden. Keys correspond to the custom column keys defined in the dataset. Absent when the golden has no custom column values.

Example: {"difficulty":"easy"}

tagsstring[]

These are the tags associated with the golden.

Example: ["geography"]

GoldenRequest

One golden to write: single-turn when it carries input, multi-turn when it carries scenario. A golden cannot be both, and its kind must match the dataset's multiTurn.

GoldenRequest = Union[
    SingleTurnGoldenRequest,
    MultiTurnGoldenRequest,
]

type GoldenRequest =
  | SingleTurnGoldenRequest
  | MultiTurnGoldenRequest;

A GoldenRequest is one of the shapes below. Send the fields of one of them, never a mix of both.

A single-turn golden to write: one input to your LLM application and the outputs expected of it.

class SingleTurnGoldenRequest:
    input: str
    actual_output: Optional[str] = Field(default=None, alias="actualOutput")
    expected_output: Optional[str] = Field(default=None, alias="expectedOutput")
    context: Optional[List[str]] = None
    retrieval_context: Optional[List[str]] = Field(default=None, alias="retrievalContext")
    tools_called: Optional[List[ToolCall]] = Field(default=None, alias="toolsCalled")
    expected_tools: Optional[List[ToolCall]] = Field(default=None, alias="expectedTools")
    token_cost: Optional[float] = Field(default=None, alias="tokenCost")
    input_token_count: Optional[int] = Field(default=None, alias="inputTokenCount")
    output_token_count: Optional[int] = Field(default=None, alias="outputTokenCount")
    additional_metadata: Optional[Dict[str, Any]] = Field(default=None, alias="additionalMetadata")
    comments: Optional[str] = None
    source_file: Optional[str] = Field(default=None, alias="sourceFile")
    source_files: Optional[List[str]] = Field(default=None, alias="sourceFiles")
    finalized: Optional[bool] = None
    custom_column_key_values: Optional[Dict[str, str]] = Field(default=None, alias="customColumnKeyValues")
    images_mapping: Optional[Dict[str, MLLMImage]] = Field(default=None, alias="imagesMapping")
    tags: Optional[List[str]] = None

inputstrRequired

This is the input to your LLM application.

Example: "What is the capital of France?"

actual_outputOptional[str]

This is the actual output of your LLM application.

Example: "The capital of France is Paris."

expected_outputOptional[str]

This is the expected output of your LLM application, which is the ideal actual output.

Example: "Paris."

contextOptional[List[str]]

This is the ideal retrieval context of your LLM application.

Example: ["Paris is the capital of France."]

retrieval_contextOptional[List[str]]

This is the retrieval context of your LLM application.

Example: ["Paris is the capital and largest city of France."]

tools_calledOptional[List[ToolCall]]

This is the tools called by your LLM application.

See ToolCall.

expected_toolsOptional[List[ToolCall]]

This is the expected tools to be called by the LLM application.

See ToolCall.

token_costOptional[float]

This is the cost of the tokens used to produce the actual output.

Example: 0.002

input_token_countOptional[int]

This is the number of input tokens passed to the LLM model.

Example: 12

output_token_countOptional[int]

This is the number of output tokens generated by the LLM model.

Example: 3

additional_metadataOptional[Dict[str, Any]]

Additional metadata to associate with the golden.

Example: {"source":"faq"}

commentsOptional[str]

Comments to associate with the golden.

Example: "Reviewed by the support team."

source_fileOptional[str]

The source file the golden was retrieved from. Like tags and customColumnKeyValues, this is left unchanged when the request omits it; send null to clear it.

Example: "capitals.csv"

source_filesOptional[List[str]]

The source files the golden was retrieved from. Like tags and customColumnKeyValues, these are left unchanged when the request omits them; send an empty array to clear them.

Example: ["capitals.csv"]

finalizedOptional[bool]

Whether the golden is ready to use in evaluations. When pushing or queueing a list of goldens the request decides this for every golden and this field is ignored.

Example: true

custom_column_key_valuesOptional[Dict[str, str]]

Custom dataset column values keyed by column name. A column that does not exist in the dataset yet is created.

Example: {"difficulty":"easy"}

images_mappingOptional[Dict[str, MLLMImage]]

The media this golden refers to, keyed by the id inside each placeholder. Put [DEEPEVAL:IMAGE:<id>] or [DEEPEVAL:PDF:<id>] in a text field where the media belongs, and the platform substitutes the entry with a matching key.

See MLLMImage.

Example: {"map":{"url":"https://example.com/paris.png","local":false}}

tagsOptional[List[str]]

Tags to associate with the golden, which is useful for grouping and filtering goldens. A tag that does not exist in the dataset yet is created.

Example: ["geography"]

interface SingleTurnGoldenRequest {
  input: string;
  actualOutput?: string | null;
  expectedOutput?: string | null;
  context?: string[] | null;
  retrievalContext?: string[] | null;
  toolsCalled?: ToolCall[] | null;
  expectedTools?: ToolCall[] | null;
  tokenCost?: number | null;
  inputTokenCount?: number | null;
  outputTokenCount?: number | null;
  additionalMetadata?: Record<string, unknown> | null;
  comments?: string | null;
  sourceFile?: string | null;
  sourceFiles?: string[];
  finalized?: boolean;
  customColumnKeyValues?: Record<string, string>;
  imagesMapping?: Record<string, MLLMImage>;
  tags?: string[];
}

inputstringRequired

This is the input to your LLM application.

Example: "What is the capital of France?"

actualOutputstring | null

This is the actual output of your LLM application.

Example: "The capital of France is Paris."

expectedOutputstring | null

This is the expected output of your LLM application, which is the ideal actual output.

Example: "Paris."

contextstring[] | null

This is the ideal retrieval context of your LLM application.

Example: ["Paris is the capital of France."]

retrievalContextstring[] | null

This is the retrieval context of your LLM application.

Example: ["Paris is the capital and largest city of France."]

toolsCalledToolCall[] | null

This is the tools called by your LLM application.

See ToolCall.

expectedToolsToolCall[] | null

This is the expected tools to be called by the LLM application.

See ToolCall.

tokenCostnumber | null

This is the cost of the tokens used to produce the actual output.

Example: 0.002

inputTokenCountnumber | null

This is the number of input tokens passed to the LLM model.

Example: 12

outputTokenCountnumber | null

This is the number of output tokens generated by the LLM model.

Example: 3

additionalMetadataRecord<string, unknown> | null

Additional metadata to associate with the golden.

Example: {"source":"faq"}

commentsstring | null

Comments to associate with the golden.

Example: "Reviewed by the support team."

sourceFilestring | null

The source file the golden was retrieved from. Like tags and customColumnKeyValues, this is left unchanged when the request omits it; send null to clear it.

Example: "capitals.csv"

sourceFilesstring[]

The source files the golden was retrieved from. Like tags and customColumnKeyValues, these are left unchanged when the request omits them; send an empty array to clear them.

Example: ["capitals.csv"]

finalizedboolean

Whether the golden is ready to use in evaluations. When pushing or queueing a list of goldens the request decides this for every golden and this field is ignored.

Example: true

customColumnKeyValuesRecord<string, string>

Custom dataset column values keyed by column name. A column that does not exist in the dataset yet is created.

Example: {"difficulty":"easy"}

imagesMappingRecord<string, MLLMImage>

See MLLMImage.

Example: {"map":{"url":"https://example.com/paris.png","local":false}}

tagsstring[]

Tags to associate with the golden, which is useful for grouping and filtering goldens. A tag that does not exist in the dataset yet is created.

Example: ["geography"]

A multi-turn golden to write: the scenario of a conversation with your LLM application and, optionally, its turns.

class MultiTurnGoldenRequest:
    scenario: str
    expected_outcome: Optional[str] = Field(default=None, alias="expectedOutcome")
    user_description: Optional[str] = Field(default=None, alias="userDescription")
    turns: Optional[List[Turn]] = None
    context: Optional[List[str]] = None
    additional_metadata: Optional[Dict[str, Any]] = Field(default=None, alias="additionalMetadata")
    comments: Optional[str] = None
    source_file: Optional[str] = Field(default=None, alias="sourceFile")
    source_files: Optional[List[str]] = Field(default=None, alias="sourceFiles")
    finalized: Optional[bool] = None
    custom_column_key_values: Optional[Dict[str, str]] = Field(default=None, alias="customColumnKeyValues")
    images_mapping: Optional[Dict[str, MLLMImage]] = Field(default=None, alias="imagesMapping")
    tags: Optional[List[str]] = None

scenariostrRequired

This is a description of the conversation context.

Example: "A traveller wants to book a hotel in Paris."

expected_outcomeOptional[str]

This describes the expected outcome, or ideal conversation flow, of the conversation.

Example: "The assistant confirms a reservation near the Louvre."

user_descriptionOptional[str]

This is the description of the user in the conversation.

Example: "A traveller planning a weekend in Paris."

turnsOptional[List[Turn]]

This is the list of turns in the conversation.

See Turn.

Example: [{"role":"user","content":"I need a hotel in Paris near the Louvre."},{"role":"assistant","content":"Hôtel du Louvre has rooms available. Which dates?"}]

contextOptional[List[str]]

This is the context of the conversation.

Example: ["Hôtel du Louvre is a five-minute walk from the museum."]

additional_metadataOptional[Dict[str, Any]]

Additional metadata to associate with the golden.

Example: {"source":"faq"}

commentsOptional[str]

Comments to associate with the golden.

Example: "Reviewed by the support team."

source_fileOptional[str]

The source file the golden was retrieved from. Like tags and customColumnKeyValues, this is left unchanged when the request omits it; send null to clear it.

Example: "capitals.csv"

source_filesOptional[List[str]]

The source files the golden was retrieved from. Like tags and customColumnKeyValues, these are left unchanged when the request omits them; send an empty array to clear them.

Example: ["capitals.csv"]

finalizedOptional[bool]

Whether the golden is ready to use in evaluations. When pushing or queueing a list of goldens the request decides this for every golden and this field is ignored.

Example: true

custom_column_key_valuesOptional[Dict[str, str]]

Custom dataset column values keyed by column name. A column that does not exist in the dataset yet is created.

Example: {"difficulty":"easy"}

images_mappingOptional[Dict[str, MLLMImage]]

See MLLMImage.

Example: {"map":{"url":"https://example.com/paris.png","local":false}}

tagsOptional[List[str]]

Tags to associate with the golden, which is useful for grouping and filtering goldens. A tag that does not exist in the dataset yet is created.

Example: ["geography"]

interface MultiTurnGoldenRequest {
  scenario: string;
  expectedOutcome?: string | null;
  userDescription?: string | null;
  turns?: Turn[] | null;
  context?: string[] | null;
  additionalMetadata?: Record<string, unknown> | null;
  comments?: string | null;
  sourceFile?: string | null;
  sourceFiles?: string[];
  finalized?: boolean;
  customColumnKeyValues?: Record<string, string>;
  imagesMapping?: Record<string, MLLMImage>;
  tags?: string[];
}

scenariostringRequired

This is a description of the conversation context.

Example: "A traveller wants to book a hotel in Paris."

expectedOutcomestring | null

This describes the expected outcome, or ideal conversation flow, of the conversation.

Example: "The assistant confirms a reservation near the Louvre."

userDescriptionstring | null

This is the description of the user in the conversation.

Example: "A traveller planning a weekend in Paris."

turnsTurn[] | null

This is the list of turns in the conversation.

See Turn.

Example: [{"role":"user","content":"I need a hotel in Paris near the Louvre."},{"role":"assistant","content":"Hôtel du Louvre has rooms available. Which dates?"}]

contextstring[] | null

This is the context of the conversation.

Example: ["Hôtel du Louvre is a five-minute walk from the museum."]

additionalMetadataRecord<string, unknown> | null

Additional metadata to associate with the golden.

Example: {"source":"faq"}

commentsstring | null

Comments to associate with the golden.

Example: "Reviewed by the support team."

sourceFilestring | null

The source file the golden was retrieved from. Like tags and customColumnKeyValues, this is left unchanged when the request omits it; send null to clear it.

Example: "capitals.csv"

sourceFilesstring[]

The source files the golden was retrieved from. Like tags and customColumnKeyValues, these are left unchanged when the request omits them; send an empty array to clear them.

Example: ["capitals.csv"]

finalizedboolean

Whether the golden is ready to use in evaluations. When pushing or queueing a list of goldens the request decides this for every golden and this field is ignored.

Example: true

customColumnKeyValuesRecord<string, string>

Custom dataset column values keyed by column name. A column that does not exist in the dataset yet is created.

Example: {"difficulty":"easy"}

imagesMappingRecord<string, MLLMImage>

See MLLMImage.

Example: {"map":{"url":"https://example.com/paris.png","local":false}}

tagsstring[]

Tags to associate with the golden, which is useful for grouping and filtering goldens. A tag that does not exist in the dataset yet is created.

Example: ["geography"]

MLLMImage

An image referenced from a text field by a [DEEPEVAL:IMAGE:<key>] marker. Send either a public url or the bytes in base64.

class MLLMImage:
    url: str
    local: bool
    base64: Optional[str] = None
    filename: Optional[str] = None
    mime_type: Optional[str] = Field(default=None, alias="mimeType")
    data_base64: Optional[str] = Field(default=None, alias="dataBase64")

urlstrRequired

This is the URL of the image.

Example: "https://example.com/everest.png"

localboolRequired

This is true when the image is your local file.

Example: false

base64Optional[str]

The base64 data of the image.

Example: "iVBORw0KGgo="

filenameOptional[str]

The original file name.

Example: "everest.png"

mime_typeOptional[str]

The image's MIME type.

Example: "image/png"

data_base64Optional[str]

The image encoded as a base64 data URL.

Example: "data:image/png;base64,iVBORw0KGgo="

interface MLLMImage {
  url: string;
  local: boolean;
  base64?: string;
  filename?: string;
  mimeType?: string;
  dataBase64?: string;
}

urlstringRequired

This is the URL of the image.

Example: "https://example.com/everest.png"

localbooleanRequired

This is true when the image is your local file.

Example: false

base64string

The base64 data of the image.

Example: "iVBORw0KGgo="

filenamestring

The original file name.

Example: "everest.png"

mimeTypestring

The image's MIME type.

Example: "image/png"

dataBase64string

The image encoded as a base64 data URL.

Example: "data:image/png;base64,iVBORw0KGgo="

PushGolden

One golden to push: single-turn when it carries input, multi-turn when it carries scenario. A golden cannot be both, and its kind must match the dataset's multiTurn. Carrying an id updates that golden instead of adding one.

PushGolden = Union[
    PushSingleTurnGolden,
    PushMultiTurnGolden,
]

type PushGolden =
  | PushSingleTurnGolden
  | PushMultiTurnGolden;

A PushGolden is one of the shapes below. Send the fields of one of them, never a mix of both.

A single-turn golden to push: one input to your LLM application and the outputs expected of it, updating the golden named by id when there is one.

class PushSingleTurnGolden:
    input: str
    actual_output: Optional[str] = Field(default=None, alias="actualOutput")
    expected_output: Optional[str] = Field(default=None, alias="expectedOutput")
    context: Optional[List[str]] = None
    retrieval_context: Optional[List[str]] = Field(default=None, alias="retrievalContext")
    tools_called: Optional[List[ToolCall]] = Field(default=None, alias="toolsCalled")
    expected_tools: Optional[List[ToolCall]] = Field(default=None, alias="expectedTools")
    token_cost: Optional[float] = Field(default=None, alias="tokenCost")
    input_token_count: Optional[int] = Field(default=None, alias="inputTokenCount")
    output_token_count: Optional[int] = Field(default=None, alias="outputTokenCount")
    additional_metadata: Optional[Dict[str, Any]] = Field(default=None, alias="additionalMetadata")
    comments: Optional[str] = None
    source_file: Optional[str] = Field(default=None, alias="sourceFile")
    source_files: Optional[List[str]] = Field(default=None, alias="sourceFiles")
    finalized: Optional[bool] = None
    custom_column_key_values: Optional[Dict[str, str]] = Field(default=None, alias="customColumnKeyValues")
    images_mapping: Optional[Dict[str, MLLMImage]] = Field(default=None, alias="imagesMapping")
    tags: Optional[List[str]] = None
    id: Optional[str] = None

inputstrRequired

This is the input to your LLM application.

Example: "What is the capital of France?"

actual_outputOptional[str]

This is the actual output of your LLM application.

Example: "The capital of France is Paris."

expected_outputOptional[str]

This is the expected output of your LLM application, which is the ideal actual output.

Example: "Paris."

contextOptional[List[str]]

This is the ideal retrieval context of your LLM application.

Example: ["Paris is the capital of France."]

retrieval_contextOptional[List[str]]

This is the retrieval context of your LLM application.

Example: ["Paris is the capital and largest city of France."]

tools_calledOptional[List[ToolCall]]

This is the tools called by your LLM application.

See ToolCall.

expected_toolsOptional[List[ToolCall]]

This is the expected tools to be called by the LLM application.

See ToolCall.

token_costOptional[float]

This is the cost of the tokens used to produce the actual output.

Example: 0.002

input_token_countOptional[int]

This is the number of input tokens passed to the LLM model.

Example: 12

output_token_countOptional[int]

This is the number of output tokens generated by the LLM model.

Example: 3

additional_metadataOptional[Dict[str, Any]]

Additional metadata to associate with the golden.

Example: {"source":"faq"}

commentsOptional[str]

Comments to associate with the golden.

Example: "Reviewed by the support team."

source_fileOptional[str]

The source file the golden was retrieved from. Like tags and customColumnKeyValues, this is left unchanged when the request omits it; send null to clear it.

Example: "capitals.csv"

source_filesOptional[List[str]]

The source files the golden was retrieved from. Like tags and customColumnKeyValues, these are left unchanged when the request omits them; send an empty array to clear them.

Example: ["capitals.csv"]

finalizedOptional[bool]

Whether the golden is ready to use in evaluations. When pushing or queueing a list of goldens the request decides this for every golden and this field is ignored.

Example: true

custom_column_key_valuesOptional[Dict[str, str]]

Custom dataset column values keyed by column name. A column that does not exist in the dataset yet is created.

Example: {"difficulty":"easy"}

images_mappingOptional[Dict[str, MLLMImage]]

See MLLMImage.

Example: {"map":{"url":"https://example.com/paris.png","local":false}}

tagsOptional[List[str]]

Tags to associate with the golden, which is useful for grouping and filtering goldens. A tag that does not exist in the dataset yet is created.

Example: ["geography"]

idOptional[str]

The id of a golden already in this dataset, as returned when the dataset is pulled. That golden is updated in place, staying in the version it is already in. Omit it to add a new golden.

Example: "<GOLDEN-ID>"

interface PushSingleTurnGolden {
  input: string;
  actualOutput?: string | null;
  expectedOutput?: string | null;
  context?: string[] | null;
  retrievalContext?: string[] | null;
  toolsCalled?: ToolCall[] | null;
  expectedTools?: ToolCall[] | null;
  tokenCost?: number | null;
  inputTokenCount?: number | null;
  outputTokenCount?: number | null;
  additionalMetadata?: Record<string, unknown> | null;
  comments?: string | null;
  sourceFile?: string | null;
  sourceFiles?: string[];
  finalized?: boolean;
  customColumnKeyValues?: Record<string, string>;
  imagesMapping?: Record<string, MLLMImage>;
  tags?: string[];
  id?: string;
}

inputstringRequired

This is the input to your LLM application.

Example: "What is the capital of France?"

actualOutputstring | null

This is the actual output of your LLM application.

Example: "The capital of France is Paris."

expectedOutputstring | null

This is the expected output of your LLM application, which is the ideal actual output.

Example: "Paris."

contextstring[] | null

This is the ideal retrieval context of your LLM application.

Example: ["Paris is the capital of France."]

retrievalContextstring[] | null

This is the retrieval context of your LLM application.

Example: ["Paris is the capital and largest city of France."]

toolsCalledToolCall[] | null

This is the tools called by your LLM application.

See ToolCall.

expectedToolsToolCall[] | null

This is the expected tools to be called by the LLM application.

See ToolCall.

tokenCostnumber | null

This is the cost of the tokens used to produce the actual output.

Example: 0.002

inputTokenCountnumber | null

This is the number of input tokens passed to the LLM model.

Example: 12

outputTokenCountnumber | null

This is the number of output tokens generated by the LLM model.

Example: 3

additionalMetadataRecord<string, unknown> | null

Additional metadata to associate with the golden.

Example: {"source":"faq"}

commentsstring | null

Comments to associate with the golden.

Example: "Reviewed by the support team."

sourceFilestring | null

The source file the golden was retrieved from. Like tags and customColumnKeyValues, this is left unchanged when the request omits it; send null to clear it.

Example: "capitals.csv"

sourceFilesstring[]

The source files the golden was retrieved from. Like tags and customColumnKeyValues, these are left unchanged when the request omits them; send an empty array to clear them.

Example: ["capitals.csv"]

finalizedboolean

Whether the golden is ready to use in evaluations. When pushing or queueing a list of goldens the request decides this for every golden and this field is ignored.

Example: true

customColumnKeyValuesRecord<string, string>

Custom dataset column values keyed by column name. A column that does not exist in the dataset yet is created.

Example: {"difficulty":"easy"}

imagesMappingRecord<string, MLLMImage>

See MLLMImage.

Example: {"map":{"url":"https://example.com/paris.png","local":false}}

tagsstring[]

Tags to associate with the golden, which is useful for grouping and filtering goldens. A tag that does not exist in the dataset yet is created.

Example: ["geography"]

idstring

The id of a golden already in this dataset, as returned when the dataset is pulled. That golden is updated in place, staying in the version it is already in. Omit it to add a new golden.

Example: "<GOLDEN-ID>"

A multi-turn golden to push: the scenario of a conversation with your LLM application and, optionally, its turns, updating the golden named by id when there is one.

class PushMultiTurnGolden:
    scenario: str
    expected_outcome: Optional[str] = Field(default=None, alias="expectedOutcome")
    user_description: Optional[str] = Field(default=None, alias="userDescription")
    turns: Optional[List[Turn]] = None
    context: Optional[List[str]] = None
    additional_metadata: Optional[Dict[str, Any]] = Field(default=None, alias="additionalMetadata")
    comments: Optional[str] = None
    source_file: Optional[str] = Field(default=None, alias="sourceFile")
    source_files: Optional[List[str]] = Field(default=None, alias="sourceFiles")
    finalized: Optional[bool] = None
    custom_column_key_values: Optional[Dict[str, str]] = Field(default=None, alias="customColumnKeyValues")
    images_mapping: Optional[Dict[str, MLLMImage]] = Field(default=None, alias="imagesMapping")
    tags: Optional[List[str]] = None
    id: Optional[str] = None

scenariostrRequired

This is a description of the conversation context.

Example: "A traveller wants to book a hotel in Paris."

expected_outcomeOptional[str]

This describes the expected outcome, or ideal conversation flow, of the conversation.

Example: "The assistant confirms a reservation near the Louvre."

user_descriptionOptional[str]

This is the description of the user in the conversation.

Example: "A traveller planning a weekend in Paris."

turnsOptional[List[Turn]]

This is the list of turns in the conversation.

See Turn.

Example: [{"role":"user","content":"I need a hotel in Paris near the Louvre."},{"role":"assistant","content":"Hôtel du Louvre has rooms available. Which dates?"}]

contextOptional[List[str]]

This is the context of the conversation.

Example: ["Hôtel du Louvre is a five-minute walk from the museum."]

additional_metadataOptional[Dict[str, Any]]

Additional metadata to associate with the golden.

Example: {"source":"faq"}

commentsOptional[str]

Comments to associate with the golden.

Example: "Reviewed by the support team."

source_fileOptional[str]

The source file the golden was retrieved from. Like tags and customColumnKeyValues, this is left unchanged when the request omits it; send null to clear it.

Example: "capitals.csv"

source_filesOptional[List[str]]

The source files the golden was retrieved from. Like tags and customColumnKeyValues, these are left unchanged when the request omits them; send an empty array to clear them.

Example: ["capitals.csv"]

finalizedOptional[bool]

Whether the golden is ready to use in evaluations. When pushing or queueing a list of goldens the request decides this for every golden and this field is ignored.

Example: true

custom_column_key_valuesOptional[Dict[str, str]]

Custom dataset column values keyed by column name. A column that does not exist in the dataset yet is created.

Example: {"difficulty":"easy"}

images_mappingOptional[Dict[str, MLLMImage]]

See MLLMImage.

Example: {"map":{"url":"https://example.com/paris.png","local":false}}

tagsOptional[List[str]]

Tags to associate with the golden, which is useful for grouping and filtering goldens. A tag that does not exist in the dataset yet is created.

Example: ["geography"]

idOptional[str]

The id of a golden already in this dataset, as returned when the dataset is pulled. That golden is updated in place, staying in the version it is already in. Omit it to add a new golden.

Example: "<GOLDEN-ID>"

interface PushMultiTurnGolden {
  scenario: string;
  expectedOutcome?: string | null;
  userDescription?: string | null;
  turns?: Turn[] | null;
  context?: string[] | null;
  additionalMetadata?: Record<string, unknown> | null;
  comments?: string | null;
  sourceFile?: string | null;
  sourceFiles?: string[];
  finalized?: boolean;
  customColumnKeyValues?: Record<string, string>;
  imagesMapping?: Record<string, MLLMImage>;
  tags?: string[];
  id?: string;
}

scenariostringRequired

This is a description of the conversation context.

Example: "A traveller wants to book a hotel in Paris."

expectedOutcomestring | null

This describes the expected outcome, or ideal conversation flow, of the conversation.

Example: "The assistant confirms a reservation near the Louvre."

userDescriptionstring | null

This is the description of the user in the conversation.

Example: "A traveller planning a weekend in Paris."

turnsTurn[] | null

This is the list of turns in the conversation.

See Turn.

Example: [{"role":"user","content":"I need a hotel in Paris near the Louvre."},{"role":"assistant","content":"Hôtel du Louvre has rooms available. Which dates?"}]

contextstring[] | null

This is the context of the conversation.

Example: ["Hôtel du Louvre is a five-minute walk from the museum."]

additionalMetadataRecord<string, unknown> | null

Additional metadata to associate with the golden.

Example: {"source":"faq"}

commentsstring | null

Comments to associate with the golden.

Example: "Reviewed by the support team."

sourceFilestring | null

The source file the golden was retrieved from. Like tags and customColumnKeyValues, this is left unchanged when the request omits it; send null to clear it.

Example: "capitals.csv"

sourceFilesstring[]

The source files the golden was retrieved from. Like tags and customColumnKeyValues, these are left unchanged when the request omits them; send an empty array to clear them.

Example: ["capitals.csv"]

finalizedboolean

Whether the golden is ready to use in evaluations. When pushing or queueing a list of goldens the request decides this for every golden and this field is ignored.

Example: true

customColumnKeyValuesRecord<string, string>

Custom dataset column values keyed by column name. A column that does not exist in the dataset yet is created.

Example: {"difficulty":"easy"}

imagesMappingRecord<string, MLLMImage>

See MLLMImage.

Example: {"map":{"url":"https://example.com/paris.png","local":false}}

tagsstring[]

Tags to associate with the golden, which is useful for grouping and filtering goldens. A tag that does not exist in the dataset yet is created.

Example: ["geography"]

idstring

The id of a golden already in this dataset, as returned when the dataset is pulled. That golden is updated in place, staying in the version it is already in. Omit it to add a new golden.

Example: "<GOLDEN-ID>"

RunDatasetEvaluationResult

class RunDatasetEvaluationResult:
    id: str
    test_case_count: int = Field(alias="testCaseCount")

idstrRequired

This is the unique id of the test run the dataset is evaluated in, generated by Confident AI and not to be confused with the identifier you supplied.

Example: "<TEST-RUN-ID>"

test_case_countintRequired

The number of test cases the evaluation was started with.

Example: 42

interface RunDatasetEvaluationResult {
  id: string;
  testCaseCount: number;
}

idstringRequired

This is the unique id of the test run the dataset is evaluated in, generated by Confident AI and not to be confused with the identifier you supplied.

Example: "<TEST-RUN-ID>"

testCaseCountnumberRequired

The number of test cases the evaluation was started with.

Example: 42

ToolCall

A tool your LLM application invoked, with what it passed in and what came back.

class ToolCall:
    name: str
    type: Optional[ToolCallType] = None
    description: Optional[str] = None
    input_parameters: Optional[Dict[str, Any]] = Field(default=None, alias="inputParameters")
    output: Optional[Any] = None
    reasoning: Optional[str] = None

namestrRequired

This is the name of the tool.

Example: "get_landmark_info"

typeOptional[ToolCallType]

See ToolCallType.

descriptionOptional[str]

This is the description of the tool.

Example: "This tool gives information about a mountain."

input_parametersOptional[Dict[str, Any]]

This is the input parameters that are passed to the tool.

Example: {"mountain":"Everest"}

outputOptional[Any]

This is the output of the tool.

Example: "8,848 metres"

reasoningOptional[str]

This is the reasoning your LLM provided for the tool call.

Example: "The user asked for the height of a mountain."

interface ToolCall {
  name: string;
  type?: ToolCallType;
  description?: string;
  inputParameters?: Record<string, unknown> | null;
  output?: unknown;
  reasoning?: string;
}

namestringRequired

This is the name of the tool.

Example: "get_landmark_info"

typeToolCallType

See ToolCallType.

descriptionstring

This is the description of the tool.

Example: "This tool gives information about a mountain."

inputParametersRecord<string, unknown> | null

This is the input parameters that are passed to the tool.

Example: {"mountain":"Everest"}

outputunknown

This is the output of the tool.

Example: "8,848 metres"

reasoningstring

This is the reasoning your LLM provided for the tool call.

Example: "The user asked for the height of a mountain."

ToolCallType

The type of the tool call, either a function or an MCP tool.

class ToolCallType(Enum):
    FUNCTION = "FUNCTION"
    MCP = "MCP"

enum ToolCallType {
  FUNCTION = "FUNCTION",
  MCP = "MCP",
}

FUNCTION · MCP

Turn

One message in a conversation, from either the user or the assistant, with the context and tools behind an assistant reply.

class Turn:
    id: Optional[str] = None
    role: TurnRole
    content: str
    user_id: Optional[str] = Field(default=None, alias="userId")
    retrieval_context: Optional[List[str]] = Field(default=None, alias="retrievalContext")
    tools_called: Optional[List[ToolCall]] = Field(default=None, alias="toolsCalled")

idOptional[str]

The id of a turn assigned by Confident AI.

Example: "<TURN-ID>"

roleTurnRoleRequired

See TurnRole.

contentstrRequired

The message content of the turn.

Example: "How tall is Mount Everest?"

user_idOptional[str]

The user ID associated with the turn.

Example: "end-user-42"

retrieval_contextOptional[List[str]]

The contexts retrieved to generate the LLM response for this turn.

Example: ["Everest is 8,848 metres tall."]

tools_calledOptional[List[ToolCall]]

The tools called to generate the LLM response for this turn.

See ToolCall.

interface Turn {
  id?: string;
  role: TurnRole;
  content: string;
  userId?: string;
  retrievalContext?: string[] | null;
  toolsCalled?: ToolCall[] | null;
}

idstring

The id of a turn assigned by Confident AI.

Example: "<TURN-ID>"

roleTurnRoleRequired

See TurnRole.

contentstringRequired

The message content of the turn.

Example: "How tall is Mount Everest?"

userIdstring

The user ID associated with the turn.

Example: "end-user-42"

retrievalContextstring[] | null

The contexts retrieved to generate the LLM response for this turn.

Example: ["Everest is 8,848 metres tall."]

toolsCalledToolCall[] | null

The tools called to generate the LLM response for this turn.

See ToolCall.

TurnRole

The role of the turn, either user or assistant.

class TurnRole(Enum):
    USER = "user"
    ASSISTANT = "assistant"

enum TurnRole {
  USER = "user",
  ASSISTANT = "assistant",
}

USER · ASSISTANT

Building a production pipeline?Design a scalable API workflow for evals, datasets, traces, and promptsTalk to an engineer

Last updated on September 30, 2026

Built byConfident AI