Get Test Run

(v2)

GET

Retrieves a test run by id, with its aggregated scores and the results of each test case it covers. Requires an active trial or paid plan.

GET/v2/test-runs/{testRunId}

curl -X GET "https://api.confident-ai.com/v2/test-runs/{testRunId}" \
  -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>"

200

{
  "success": true,
  "data": {
    "id": "<TEST-RUN-ID>",
    "createdAt": "2025-01-01T12:00:00.000Z",
    "identifier": "run-399-102",
    "status": "IN_PROGRESS",
    "multiTurn": false,
    "testsPassed": 8,
    "testsFailed": 2,
    "totalTests": 10,
    "metricsScores": [
      {
        "metric": "Answer Correctness",
        "scores": [
          0.9,
          1
        ],
        "passes": 8,
        "fails": 2,
        "errors": 0,
        "errorType": "AI_CONNECTION_ERROR"
      }
    ],
    "runDuration": 15.2,
    "evaluationCost": 0.254,
    "datasetAlias": "geography-goldens",
    "testFile": "test_geography.py",
    "summary": {
      "summaryOverview": {
        "summary": [
          "8 of 10 test cases passed."
        ],
        "actionItems": [
          "Add goldens for lesser-known peaks."
        ]
      },
      "topicSummaries": [
        {
          "topic": "Mountain heights",
          "summaryPoints": [
            {
              "content": "Answers about mountain heights were correct and cited the retrieved context.",
              "testCaseIds": [
                "<TEST-CASE-ID>"
              ],
              "grade": 0.9
            }
          ],
          "testCaseIds": [
            "<TEST-CASE-ID>"
          ]
        }
      ]
    },
    "testCases": [
      {
        "input": "How tall is Mount Everest?",
        "actualOutput": "Mount Everest is 8,848 metres tall.",
        "expectedOutput": "Mount Everest is 8,848 metres tall.",
        "context": [
          "Everest is 8,848 metres tall."
        ],
        "retrievalContext": [
          "Everest is 8,848 metres tall."
        ],
        "toolsCalled": [
          {
            "name": "get_landmark_info",
            "type": "FUNCTION",
            "description": "This tool gives information about a mountain.",
            "inputParameters": {
              "mountain": "Everest"
            },
            "output": "8,848 metres",
            "reasoning": "The user asked for the height of a mountain."
          }
        ],
        "expectedTools": [
          {
            "name": "get_landmark_info",
            "type": "FUNCTION",
            "description": "This tool gives information about a mountain.",
            "inputParameters": {
              "mountain": "Everest"
            },
            "output": "8,848 metres",
            "reasoning": "The user asked for the height of a mountain."
          }
        ],
        "id": "<TEST-CASE-ID>",
        "name": "everest-height",
        "success": true,
        "runDuration": 1.2,
        "evaluationCost": 0.001,
        "comments": "Reviewed by the geography team.",
        "additionalMetadata": {
          "region": "Nepal"
        },
        "metricsData": [
          {
            "id": "<METRIC-DATA-ID>",
            "name": "Answer Relevancy",
            "score": 0.95,
            "reason": "The answer directly states the capital of France.",
            "success": true,
            "threshold": 0.5,
            "strictMode": false,
            "skipped": false,
            "flaky": false,
            "evaluationModel": "gpt-4o",
            "evaluationCost": 0.0004,
            "error": null,
            "errorType": "AI_CONNECTION_ERROR",
            "createdAt": "2025-01-15T10:30:06.000Z",
            "evaluatedAt": "2025-01-15T10:30:09.000Z",
            "traceUuid": "3f9c2a1e-5b7d-4c8e-9f01-2a3b4c5d6e7f",
            "spanUuid": null
          }
        ]
      }
    ]
  },
  "deprecated": false
}

Headers

CONFIDENT_API_KEYstringRequired
The API key of your Confident AI project.

Path parameters

testRunIdstringRequired
The id of the test run.

Response

Get Test Run succeeded.

successboolean
Indicates if the request was successful.
dataobject
A test run with its aggregated metric scores and every test case in it.
Show 15 propertiesHide 15 properties
- idstring
  This is the unique ID for the test run, generated by Confident AI and not to be confused with the identifier provided by the user.
- createdAtstring
  The time the test run was created.
- identifierstring | null
  The human-readable identifier you gave the test run, if any.
- statusenum
  The status of the test run: IN_PROGRESS while test cases are still being evaluated, then COMPLETED, ERRORED or CANCELLED.
  Show 4 enum valuesHide 4 enum values
  - IN_PROGRESS
  - COMPLETED
  - ERRORED
  - CANCELLED
- multiTurnboolean
  Whether this test run contains multi-turn test cases.
- testsPassedinteger
  The number of test cases that passed.
- testsFailedinteger
  The number of test cases that failed.
- totalTestsinteger
  The total number of test cases in this test run.
- metricsScoreslist of objects
  The aggregated metric scores across all test cases.
  Show 6 propertiesHide 6 properties
  - metricstring
    This is the name of the metric.
  - scoreslist of numbers
    This is an array of scores for the metric across test cases, one per test case that produced a score.
  - passesinteger
    This is the number of times this metric passed the threshold.
  - failsinteger
    This is the number of times this metric failed to pass the threshold.
  - errorsinteger
    This is the number of times this metric errored during evaluation.
  - errorTypeenum | null
    Why an evaluation errored: the AI connection or a transformer failed, the evaluation model failed, the test case lacked the parameters the metric needs, or an internal error occurred.
    Show 5 enum valuesHide 5 enum values
    AI_CONNECTION_ERROR
    TRANSFORMER_ERROR
    EVALUATION_MODEL_ERROR
    INVALID_TEST_CASE_PARAMETERS
    INTERNAL_ERROR
- runDurationnumber
  The total duration of the test run in seconds.
- evaluationCostnumber | null
  The cost of evaluating every test case in the test run.
- datasetAliasstring | null
  The alias of the dataset the test run was evaluated on, if any.
- testFilestring | null
  The test file the test run was started from, if any.
- summaryobject | null
  The AI-generated summary of a test run, produced on the Confident AI platform. Null until it has been generated.
  Show 2 propertiesHide 2 properties
  - summaryOverviewobject
    The headline findings and action items of a test run.
    Show 2 propertiesHide 2 properties
    summarylist of strings
    The headline findings across every topic.
    actionItemslist of strings
    What to change to improve the next test run.
  - topicSummarieslist of objects
    The findings for each topic the test cases were grouped into.
    Show 3 propertiesHide 3 properties
    topicstring
    The topic the test cases were grouped under.
    summaryPointslist of objects
    The findings for this topic.
    Show 3 propertiesHide 3 properties
    contentstring
    One finding about the test cases in this topic.
    testCaseIdslist of strings
    The ids of the test cases this finding is drawn from.
    gradenumber
    How well the test cases behind this finding performed, from 0 to 1.
    testCaseIdslist of strings
    The ids of the test cases grouped under this topic.
- testCaseslist of object | object | object
  The test cases in this test run. Every test case is of the same kind: single-turn, multi-turn or trace-based.
  Show 3 variantsHide 3 variants
  - Single-Turn Test Caseobject
    A test case evaluated as a single exchange with your LLM application.
    Show 15 propertiesHide 15 properties
    inputstring | null
    This is the input of the test case.
    actualOutputstring | null
    This is the actual output of the test case.
    expectedOutputstring | null
    This is the expected output of the test case.
    contextarray | null
    This is the context of the test case.
    retrievalContextarray | null
    This is the retrieval context of the test case.
    toolsCalledarray | null
    This is the tools called of the test case.
    Show 6 propertiesHide 6 properties
    namestring
    This is the name of the tool.
    typeenum
    The type of the tool call, either a function or an MCP tool.
    Show 2 enum valuesHide 2 enum values
    FUNCTION
    MCP
    descriptionstring
    This is the description of the tool.
    inputParametersobject | null
    This is the input parameters that are passed to the tool.
    outputany
    This is the output of the tool.
    reasoningstring
    This is the reasoning your LLM provided for the tool call.
    expectedToolsarray | null
    This is the expected tools of the test case.
    Show 6 propertiesHide 6 properties
    namestring
    This is the name of the tool.
    typeenum
    The type of the tool call, either a function or an MCP tool.
    Show 2 enum valuesHide 2 enum values
    FUNCTION
    MCP
    descriptionstring
    This is the description of the tool.
    inputParametersobject | null
    This is the input parameters that are passed to the tool.
    outputany
    This is the output of the tool.
    reasoningstring
    This is the reasoning your LLM provided for the tool call.
    idstring
    This is the id of the test case generated by Confident AI.
    namestring
    This is the name of the test case.
    successboolean | null
    Whether this test case passed all metric thresholds, or null while it is still being evaluated.
    runDurationnumber | null
    The duration of the test case evaluation in seconds.
    evaluationCostnumber | null
    The cost of evaluating this test case.
    commentsstring | null
    Any comments associated with this test case.
    additionalMetadataobject | null
    Additional metadata associated with this test case.
    metricsDatalist of objects
    The metric evaluation results for this test case.
    Show 17 propertiesHide 17 properties
    idstring
    The unique identifier of the metric data entry.
    namestring
    The name of the metric.
    scorenumber | null
    The final metric score, or null when the metric errored or was skipped.
    reasonstring | null
    The reason for the metric score, generated by the evaluation model at evaluation time.
    successboolean | null
    Whether the metric score is above the threshold, or null while the evaluation is still running.
    thresholdnumber | null
    The threshold for the metric, which determines if the metric is passing or failing.
    strictModeboolean
    Whether the metric was run in strict mode, which outputs a binary score of 0 or 1.
    skippedboolean
    Whether the metric evaluation was skipped.
    flakyboolean
    Whether the metric's verdict was non-deterministic across runs.
    evaluationModelstring | null
    The evaluation model used to run the evaluation.
    evaluationCostnumber | null
    The cost of running the evaluation in USD.
    errorstring | null
    The error message if the evaluation failed.
    errorTypeenum | null
    Why an evaluation errored: the AI connection or a transformer failed, the evaluation model failed, the test case lacked the parameters the metric needs, or an internal error occurred.
    Show 5 enum valuesHide 5 enum values
    AI_CONNECTION_ERROR
    TRANSFORMER_ERROR
    EVALUATION_MODEL_ERROR
    INVALID_TEST_CASE_PARAMETERS
    INTERNAL_ERROR
    createdAtstring
    The time the metric data was created.
    evaluatedAtstring | null
    The time the metric was evaluated, or null while it is still running.
    traceUuidstring | null
    The uuid of the trace this metric was evaluated on, for test cases formed from traces.
    spanUuidstring | null
    The uuid of the span this metric was evaluated on, for component-level metrics.
  - OR
  - Multi-Turn Test Caseobject
    A test case evaluated as a conversation with your LLM application.
    Show 13 propertiesHide 13 properties
    turnslist of objects
    The list of turns in the conversation.
    Show 6 propertiesHide 6 properties
    idstring
    The id of a turn assigned by Confident AI.
    roleenum
    The role of the turn, either user or assistant.
    Show 2 enum valuesHide 2 enum values
    user
    assistant
    contentstring
    The message content of the turn.
    userIdstring
    The user ID associated with the turn.
    retrievalContextarray | null
    The contexts retrieved to generate the LLM response for this turn.
    toolsCalledarray | null
    The tools called to generate the LLM response for this turn.
    Show 6 propertiesHide 6 properties
    namestring
    This is the name of the tool.
    typeenum
    The type of the tool call, either a function or an MCP tool.
    Show 2 enum valuesHide 2 enum values
    FUNCTION
    MCP
    descriptionstring
    This is the description of the tool.
    inputParametersobject | null
    This is the input parameters that are passed to the tool.
    outputany
    This is the output of the tool.
    reasoningstring
    This is the reasoning your LLM provided for the tool call.
    scenariostring | null
    A description of the conversation context.
    expectedOutcomestring | null
    The expected outcome or ideal conversation flow.
    userDescriptionstring | null
    A description of the user in the conversation.
    contextarray | null
    The context provided for the conversation.
    idstring
    This is the id of the test case generated by Confident AI.
    namestring
    This is the name of the test case.
    successboolean | null
    Whether this test case passed all metric thresholds, or null while it is still being evaluated.
    runDurationnumber | null
    The duration of the test case evaluation in seconds.
    evaluationCostnumber | null
    The cost of evaluating this test case.
    commentsstring | null
    Any comments associated with this test case.
    additionalMetadataobject | null
    Additional metadata associated with this test case.
    metricsDatalist of objects
    The metric evaluation results for this test case.
    Show 17 propertiesHide 17 properties
    idstring
    The unique identifier of the metric data entry.
    namestring
    The name of the metric.
    scorenumber | null
    The final metric score, or null when the metric errored or was skipped.
    reasonstring | null
    The reason for the metric score, generated by the evaluation model at evaluation time.
    successboolean | null
    Whether the metric score is above the threshold, or null while the evaluation is still running.
    thresholdnumber | null
    The threshold for the metric, which determines if the metric is passing or failing.
    strictModeboolean
    Whether the metric was run in strict mode, which outputs a binary score of 0 or 1.
    skippedboolean
    Whether the metric evaluation was skipped.
    flakyboolean
    Whether the metric's verdict was non-deterministic across runs.
    evaluationModelstring | null
    The evaluation model used to run the evaluation.
    evaluationCostnumber | null
    The cost of running the evaluation in USD.
    errorstring | null
    The error message if the evaluation failed.
    errorTypeenum | null
    Why an evaluation errored: the AI connection or a transformer failed, the evaluation model failed, the test case lacked the parameters the metric needs, or an internal error occurred.
    Show 5 enum valuesHide 5 enum values
    AI_CONNECTION_ERROR
    TRANSFORMER_ERROR
    EVALUATION_MODEL_ERROR
    INVALID_TEST_CASE_PARAMETERS
    INTERNAL_ERROR
    createdAtstring
    The time the metric data was created.
    evaluatedAtstring | null
    The time the metric was evaluated, or null while it is still running.
    traceUuidstring | null
    The uuid of the trace this metric was evaluated on, for test cases formed from traces.
    spanUuidstring | null
    The uuid of the span this metric was evaluated on, for component-level metrics.
  - OR
  - Trace Test Caseobject
    A test case formed from an ingested trace and evaluated component by component. Its span-level metrics are in metricsData.
    Show 9 propertiesHide 9 properties
    traceobject
    The trace a component-level test case was formed from, without its spans. Fetch the trace by uuid for the full span tree.
    Show 17 propertiesHide 17 properties
    uuidstring
    This is the unique identifier of the trace.
    namestring | null
    This is the name of the trace.
    inputstring | null
    This is the input to the trace.
    outputstring | null
    This is the output of the trace.
    startTimestring
    This is the time the trace started.
    endTimestring
    This is the time the trace ended.
    environmentstring
    This is the environment where your trace was posted, which helps with separating and debugging traces from different environments on the Confident AI platform. Any name up to 64 characters is accepted, for example production, staging or eu-prod.
    metadataobject | null
    This is any additional metadata associated with the trace.
    tagsarray | null
    This is any tags associated with the trace, which helps with grouping traces and filtering them on the Confident AI platform.
    threadIdstring | null
    This is the unique identifier of the thread associated with the trace.
    userIdstring | null
    This is the unique identifier for your end user for the trace.
    metricCollectionNamestring | null
    This is the name of the metric collection the trace was evaluated with.
    retrievalContextarray | null
    This is the retrieval context of your trace, which is to be used for evaluation.
    contextarray | null
    This is the ideal retrieval context of your trace, which is to be used for evaluation.
    expectedOutputstring | null
    This is the expected output of your trace, which is the ideal actual output and to be used for evaluation.
    toolsCalledarray | null
    This is the tools called by your trace, which is to be used for evaluation.
    Show 6 propertiesHide 6 properties
    namestring
    This is the name of the tool.
    typeenum
    The type of the tool call, either a function or an MCP tool.
    Show 2 enum valuesHide 2 enum values
    FUNCTION
    MCP
    descriptionstring
    This is the description of the tool.
    inputParametersobject | null
    This is the input parameters that are passed to the tool.
    outputany
    This is the output of the tool.
    reasoningstring
    This is the reasoning your LLM provided for the tool call.
    expectedToolsarray | null
    This is the expected tools to be called by the trace, which is to be used for evaluation.
    Show 6 propertiesHide 6 properties
    namestring
    This is the name of the tool.
    typeenum
    The type of the tool call, either a function or an MCP tool.
    Show 2 enum valuesHide 2 enum values
    FUNCTION
    MCP
    descriptionstring
    This is the description of the tool.
    inputParametersobject | null
    This is the input parameters that are passed to the tool.
    outputany
    This is the output of the tool.
    reasoningstring
    This is the reasoning your LLM provided for the tool call.
    idstring
    This is the id of the test case generated by Confident AI.
    namestring
    This is the name of the test case.
    successboolean | null
    Whether this test case passed all metric thresholds, or null while it is still being evaluated.
    runDurationnumber | null
    The duration of the test case evaluation in seconds.
    evaluationCostnumber | null
    The cost of evaluating this test case.
    commentsstring | null
    Any comments associated with this test case.
    additionalMetadataobject | null
    Additional metadata associated with this test case.
    metricsDatalist of objects
    The metric evaluation results for this test case.
    Show 17 propertiesHide 17 properties
    idstring
    The unique identifier of the metric data entry.
    namestring
    The name of the metric.
    scorenumber | null
    The final metric score, or null when the metric errored or was skipped.
    reasonstring | null
    The reason for the metric score, generated by the evaluation model at evaluation time.
    successboolean | null
    Whether the metric score is above the threshold, or null while the evaluation is still running.
    thresholdnumber | null
    The threshold for the metric, which determines if the metric is passing or failing.
    strictModeboolean
    Whether the metric was run in strict mode, which outputs a binary score of 0 or 1.
    skippedboolean
    Whether the metric evaluation was skipped.
    flakyboolean
    Whether the metric's verdict was non-deterministic across runs.
    evaluationModelstring | null
    The evaluation model used to run the evaluation.
    evaluationCostnumber | null
    The cost of running the evaluation in USD.
    errorstring | null
    The error message if the evaluation failed.
    errorTypeenum | null
    Why an evaluation errored: the AI connection or a transformer failed, the evaluation model failed, the test case lacked the parameters the metric needs, or an internal error occurred.
    Show 5 enum valuesHide 5 enum values
    AI_CONNECTION_ERROR
    TRANSFORMER_ERROR
    EVALUATION_MODEL_ERROR
    INVALID_TEST_CASE_PARAMETERS
    INTERNAL_ERROR
    createdAtstring
    The time the metric data was created.
    evaluatedAtstring | null
    The time the metric was evaluated, or null while it is still running.
    traceUuidstring | null
    The uuid of the trace this metric was evaluated on, for test cases formed from traces.
    spanUuidstring | null
    The uuid of the span this metric was evaluated on, for component-level metrics.
deprecatedboolean
Indicates if this endpoint is deprecated.

Built byConfident AI