Run Dataset Evaluation

POSThttps://api.confident-ai.com/v1/datasets/{alias}/run

Starts an evaluation of a dataset's finalized goldens against a metric collection asynchronously, then returns a link to the test run.

Optionally generate the outputs on the fly with either an AI connection or a prompt (by alias and commit) — provide at most one. When not provided, the goldens' stored actual outputs are evaluated by default.

POST/v1/datasets/{alias}/run

curl -X POST "https://api.confident-ai.com/v1/datasets/{alias}/run" \
  -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>" \
  -H "Content-Type: application/json" \
  -d '{
  "metricCollection": "Answer Quality",
  "identifier": "Nightly regression"
}'

200

{
  "success": true,
  "data": {
    "id": "TEST-RUN-ID",
    "testCaseCount": 42
  },
  "link": "https://app.confident-ai.com/project/<PROJECT-ID>/test-runs/<TEST-RUN-ID>/test-cases"
}

Headers

CONFIDENT_API_KEYstringRequired
The API key of your Confident AI project.

Path parameters

aliasstringRequired
The unique alias of the dataset to evaluate.

Request body

metricCollectionstringRequired
The name of the metric collection to evaluate against.
identifierstring
An optional label for the resulting test run.
versionstring
The dataset version to evaluate. Defaults to the latest version.
aiConnectionIdstring
The ID of the AI connection used to generate outputs, you can find this in the Project Settings → AI Connections on the platform. Required for AI_CONNECTION mode.
promptAliasstring
The alias of the prompt used to generate outputs. Required for PROMPT mode.
promptCommitstring
The prompt commit hash to generate with. Defaults to the latest commit on the prompt's main branch.
generationModeenum
The generation source. Optional when at most one of aiConnectionId or promptAlias is provided.
Show 2 enum valuesHide 2 enum values
- AI_CONNECTION
- PROMPT
variablesMappingobject
Maps each variable in the prompt to the golden field it is interpolated with, such as Input or Expected Output, or a dataset custom column key. Only applies when generating from a prompt.
includeSimulationboolean
Whether to simulate a conversation per golden before evaluating, for multi-turn datasets. Goldens need a scenario when this is enabled, and turns when it is disabled.
maxConcurrentGenerationinteger
The maximum number of generation calls to run in parallel. An AI connection's own concurrency limit takes precedence over this.
generationTimeoutinteger
The number of seconds to wait for a generation before it is marked as errored.
numGenerationsinteger
The number of times to run each golden. Defaults to the AI connection's configured default, otherwise 1.
mcpServerIdslist of strings
The IDs of the MCP servers to attach. A tool call whose name matches a tool exposed by one of these servers is labeled an MCP tool call instead of a function call.

Response

successboolean
Indicates if the evaluation was started.
dataobject
Show 2 propertiesHide 2 properties
- idstring
  The unique identifier of the created test run.
- testCaseCountinteger
  The number of test cases the evaluation was started with.
linkstring
A link to view the test run.

Built byConfident AI