Run Evaluation

POSThttps://api.confident-ai.com/v2/datasets/{datasetId}/run

Starts an evaluation of the dataset's finalized goldens against a metric collection and returns the test run it is evaluated in. The evaluation runs asynchronously, so this returns as soon as the run is created. By default the goldens' stored actual outputs are evaluated; supply aiConnectionId or promptAlias, never both, to generate them first.

POST/v2/datasets/{datasetId}/run

curl -X POST "https://api.confident-ai.com/v2/datasets/{datasetId}/run" \
  -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>" \
  -H "Content-Type: application/json" \
  -d '{
  "metricCollection": "Answer Quality",
  "identifier": "Nightly regression",
  "version": "00.00.01",
  "aiConnectionId": "<AI-CONNECTION-ID>",
  "promptAlias": "capital-lookup",
  "promptCommit": "bab04ce",
  "generationMode": "AI_CONNECTION",
  "variablesMapping": {
    "question": "Input"
  },
  "includeSimulation": false,
  "maxConcurrentGeneration": 5,
  "generationTimeout": 60,
  "numGenerations": 1,
  "mcpServerIds": [
    "<MCP-SERVER-ID>"
  ]
}'

200

{
  "success": true,
  "data": {
    "id": "<TEST-RUN-ID>",
    "testCaseCount": 42
  },
  "link": "https://app.confident-ai.com/project/<PROJECT-ID>/test-runs/<TEST-RUN-ID>/test-cases",
  "deprecated": false
}

Headers

CONFIDENT_API_KEYstringRequired
The API key of your Confident AI project.

Path parameters

datasetIdstringRequired
The unique id of the dataset.

Request body

metricCollectionstringRequired
The name of the metric collection to evaluate against. Names come from the list metric collections endpoint.
identifierstring
A label for the resulting test run, used to recognise it in the test runs list.
versionstring
The dataset version to evaluate. Omit this field to evaluate the latest version.
aiConnectionIdstring
The id of the AI connection used to generate the actual outputs before evaluating them. Required when generationMode is AI_CONNECTION, and not allowed together with promptAlias.
promptAliasstring
The alias of the prompt used to generate the actual outputs before evaluating them. Required when generationMode is PROMPT, and not allowed together with aiConnectionId.
promptCommitstring
The prompt commit hash to generate with. Requires promptAlias. Omit this field to generate with the latest commit on the prompt's main branch.
generationModeenum
Where the actual outputs come from when running a dataset: AI_CONNECTION generates them with an AI connection, PROMPT with a prompt. Omit it when you supply at most one of aiConnectionId or promptAlias, and Confident AI infers the mode from whichever you sent.
Show 2 enum valuesHide 2 enum values
- AI_CONNECTION
- PROMPT
variablesMappingobject
Maps each variable in the prompt to the golden field it is interpolated with, such as Input or Expected Output, or to a dataset custom column key. This field applies only when generating from a prompt.
includeSimulationboolean
Whether to simulate a conversation for each golden before evaluating it, for multi-turn datasets. Every golden needs a scenario when this is enabled, and turns when it is disabled.
maxConcurrentGenerationinteger
The maximum number of generation calls to run in parallel. An AI connection's own maxConcurrency takes precedence over this value.
generationTimeoutinteger
The number of seconds to wait for a single generation before it is marked as errored.
numGenerationsinteger
How many times to run each golden, so a single outlier response does not skew the results. Omit this field to use the AI connection's defaultNumGenerations, which is 1 when the connection does not set one.
mcpServerIdslist of strings
The ids of the MCP servers to attach to the run. A tool call whose name matches a tool exposed by one of these servers is labeled an MCP tool call rather than a function call.

Response

Run Evaluation succeeded.

successboolean
Indicates if the request was successful.
dataobject
Show 2 propertiesHide 2 properties
- idstring
  This is the unique id of the test run the dataset is evaluated in, generated by Confident AI and not to be confused with the identifier you supplied.
- testCaseCountinteger
  The number of test cases the evaluation was started with.
linkstring
This is the URL of the resource on the Confident AI platform.
deprecatedboolean
Indicates if this endpoint is deprecated.

Built byConfident AI