Retrieve Test Run

GEThttps://api.confident-ai.com/v1/test-runs/{testRunId}

Retrieves a list of test cases and their respective metrics from a test run.

GET/v1/test-runs/{testRunId}

curl -X GET "https://api.confident-ai.com/v1/test-runs/{testRunId}" \
  -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>"

200

{
  "success": true,
  "data": {
    "metricsScores": [
      {
        "metric": "Answer Correctness",
        "scores": [
          0
        ],
        "passes": 0,
        "fails": 1,
        "errors": 0
      }
    ],
    "traceMetricsScores": [],
    "conversational": false,
    "identifier": "test-run-001",
    "status": "COMPLETED",
    "testsPassed": 0,
    "testsFailed": 1,
    "totalTests": 1,
    "runDuration": 1.5,
    "testCases": [
      {
        "id": "TEST-CASE-ID",
        "name": "Test Case 1",
        "input": "What's the capital of France?",
        "expectedOutput": "Paris",
        "actualOutput": "San Francisco",
        "success": false,
        "context": [
          "The capital of France is Paris."
        ],
        "retrievalContext": [
          "The capital of France is Paris."
        ],
        "runDuration": 1.2,
        "evaluationCost": 0.001,
        "metricsData": [
          {
            "id": "METRIC-ID",
            "score": 0,
            "reason": "The capital of France is Paris.",
            "success": false,
            "threshold": 0.5,
            "evaluationModel": "gpt-4.1",
            "strictMode": false,
            "evaluationCost": 0.001,
            "name": "Answer Correctness",
            "verboseLogs": "..."
          }
        ]
      }
    ]
  },
  "deprecated": false
}

Headers

CONFIDENT_API_KEYstringRequired
The API key of your Confident AI project.

Path parameters

testRunIdstringRequired
The id of the test run you want to retrieve.

Response

successboolean
This is true if the test run was successfully retrieved.
dataobject
Show 9 propertiesHide 9 properties
- metricsScoreslist of objects
  The aggregated metric scores across all test cases.
  Show 5 propertiesHide 5 properties
  - failsnumber
    This is the number of times this metric failed to pass the threshold.
  - errorsnumber
    This is the number of times this metric errored during evaluation.
  - metricstring
    This is the name of the metric.
  - passesnumber
    This is the number of times this metric passed the threshold.
  - scoreslist of numbers
    This is an array of scores for the metric across test cases.
- testCaseslist of object | object
  The test cases in this test run. Will contain either single-turn test cases or conversational test cases, but not both.
  Show 2 variantsHide 2 variants
  - LLMTestCaseobject
    Show 16 propertiesHide 16 properties
    idstring
    This is the id of the test case generated by Confident AI.
    namestring
    This is the name of the test case.
    inputstring
    This is the input of the test case.
    actualOutputstring
    This is the actual output of the test case.
    contextlist of strings
    This is the context of the test case.
    retrievalContextlist of strings
    This is the retrieval context of the test case.
    expectedOutputstring
    This is the expected output of the test case.
    toolsCalledlist of objects
    This is the tools called of the test case.
    Show 5 propertiesHide 5 properties
    namestring
    This is the name of the tool.
    descriptionstring
    This is the description of the tool.
    inputParametersobject
    This is the input parameters that are passed to the tool.
    outputstring
    This is the output of the tool.
    reasoningstring
    This is the reasoning your LLM provided for the tool call.
    expectedToolslist of objects
    This is the expected tools of the test case.
    Show 5 propertiesHide 5 properties
    namestring
    This is the name of the tool.
    descriptionstring
    This is the description of the tool.
    inputParametersobject
    This is the input parameters that are passed to the tool.
    outputstring
    This is the output of the tool.
    reasoningstring
    This is the reasoning your LLM provided for the tool call.
    metricsDatalist of objects
    The metric evaluation results for this test case.
    Show 21 propertiesHide 21 properties
    idstring
    The unique identifier of the metrics data entry.
    projectIdstring
    The project this metric data belongs to.
    traceUuidstring
    The UUID of the trace this metric data is associated with, if any.
    spanUuidstring
    The UUID of the span this metric data is associated with, if any.
    testCaseIdstring
    The ID of the test case this metric data is associated with, if any.
    testRunIdstring
    The ID of the test run this metric data is associated with, if any.
    threadIdstring
    The ID of the thread this metric data is associated with, if any.
    namestring
    The name of the metric.
    multiTurnboolean
    Whether this metric was evaluated on a multi-turn conversation.
    scorenumber
    The final metric score.
    reasonstring
    The reason for the metric score, generated by the evaluation model at evaluation time.
    successboolean
    Whether the metric score is above the threshold.
    createdAtstring
    The time the metric data was created.
    evaluatedAtstring
    The time the metric was evaluated.
    thresholdnumber
    The threshold for the metric, which determines if the metric is passing or failing.
    strictModeboolean
    Whether the metric was run in strict mode, which outputs a binary score of 0 or 1.
    skippedboolean
    Whether the metric evaluation was skipped.
    evaluationModelstring
    The evaluation model used to run the evaluation.
    errorstring
    The error message if the evaluation failed.
    evaluationCostnumber
    The cost of running the evaluation.
    verboseLogsstring
    The verbose logs of the evaluation, which breaks down the metric score calculation into individual steps.
    commentsstring
    Any comments associated with this test case.
    additionalMetadataobject
    Additional metadata associated with this test case.
    successboolean
    Whether this test case passed all metric thresholds.
    runDurationnumber
    The duration of the test case evaluation in seconds.
    evaluationCostnumber
    The cost of evaluating this test case.
    traceobject
    This is the trace dictionary of a test case.
    Show 21 propertiesHide 21 properties
    uuidstring
    This is the unique identifier of the trace.
    namestring
    This is the name of the trace.
    inputstring
    This is the input to the trace.
    outputstring
    This is the output of the trace.
    startTimestring
    This is the time the trace started.
    endTimestring
    This is the time the trace ended.
    environmentenum
    This is the environment where your trace was posted, which helps with separating and debugging traces from different environments on the Confident AI platform.
    Show 4 enum valuesHide 4 enum values
    production
    development
    staging
    testing
    metadataobject
    This is any additional metadata associated with the trace.
    tagslist of strings
    This is any tags associated with the trace, which helps with grouping traces and filtering them on the Confident AI platform.
    spanslist of object | object | object | object | object
    This is the list of base spans associated with the trace.
    Show 5 variantsHide 5 variants
    BaseSpanobject
    OR
    LlmSpanobject
    OR
    RetrieverSpanobject
    OR
    ToolSpanobject
    OR
    AgentSpanobject
    threadIdstring
    This is the unique identifier of the thread associated with the trace.
    threadobject
    Thread-level fields applied to the thread record. thread.id is an alternate way to specify the thread (must match top-level threadId if both are provided). metadata and tags only take effect when a thread id is resolvable; successive ingestions merge metadata keys, while tags replace any prior value.
    Show 3 propertiesHide 3 properties
    idstring
    The thread id. Equivalent to top-level threadId; if both are set they must match.
    metadataobject
    Custom key/value metadata to attach to the thread. Values can be any JSON-serializable type and are stringified server-side. Successive ingestions for the same thread merge metadata keys.
    tagslist of strings
    Tags to set on the thread. Replaces any previously stored tags.
    userIdstring
    This is the unique identifier for your end user for the trace.
    metricCollectionstring
    This is the metric collection you wish to use to evaluate the trace.
    testRunIdstring
    This is the unique identifier of the test run to associate the trace with. When set, the trace becomes one test case in that test run, and metricCollection is required. Create a test run with the POST /v1/test-runs endpoint to get this id.
    retrievalContextlist of strings
    This is the retrieval context of your trace, which is to be used for evaluation.
    contextlist of strings
    This is the ideal retrieval context of your trace, which is to be used for evaluation.
    expectedOutputstring
    This is the expected output of your trace, which is the ideal actual output and to be used for evaluation.
    toolsCalledlist of objects
    This is the tools called by your trace, which is to be used for evaluation.
    Show 5 propertiesHide 5 properties
    namestring
    This is the name of the tool.
    descriptionstring
    This is the description of the tool.
    inputParametersobject
    This is the input parameters that are passed to the tool.
    outputstring
    This is the output of the tool.
    reasoningstring
    This is the reasoning your LLM provided for the tool call.
    expectedToolslist of objects
    This is the expected tools to be called by the trace, which is to be used for evaluation.
    Show 5 propertiesHide 5 properties
    namestring
    This is the name of the tool.
    descriptionstring
    This is the description of the tool.
    inputParametersobject
    This is the input parameters that are passed to the tool.
    outputstring
    This is the output of the tool.
    reasoningstring
    This is the reasoning your LLM provided for the tool call.
    attachmentsobject
    Map of attachment ids to payloads for all [DEEPEVAL:IMAGE:…] and [DEEPEVAL:PDF:…] markers in this trace. Define attachments at the trace level with same ids for same instances.
  - OR
  - ConversationalTestCaseobject
    Show 13 propertiesHide 13 properties
    idstring
    This is the id of the conversational test case generated by Confident AI.
    namestring
    This is the name of the conversational test case.
    turnslist of objects
    The list of turns in the conversation.
    Show 4 propertiesHide 4 properties
    roleenum
    The role of the turn, either user or assistant.
    Show 2 enum valuesHide 2 enum values
    user
    assistant
    contentstring
    The message content of the turn.
    toolsCalledlist of objects
    The tools called to generate the LLM response for this turn.
    Show 5 propertiesHide 5 properties
    namestring
    This is the name of the tool.
    descriptionstring
    This is the description of the tool.
    inputParametersobject
    This is the input parameters that are passed to the tool.
    outputstring
    This is the output of the tool.
    reasoningstring
    This is the reasoning your LLM provided for the tool call.
    retrievalContextlist of strings
    The contexts retrieved to generate the LLM response for this turn.
    scenariostring
    A description of the conversation context.
    expectedOutcomestring
    The expected outcome or ideal conversation flow.
    userDescriptionstring
    A description of the user in the conversation.
    contextlist of strings
    The context provided for the conversation.
    commentsstring
    Any comments associated with this test case.
    additionalMetadataobject
    Additional metadata associated with this test case.
    metricsDatalist of objects
    The metric evaluation results for this test case.
    Show 21 propertiesHide 21 properties
    idstring
    The unique identifier of the metrics data entry.
    projectIdstring
    The project this metric data belongs to.
    traceUuidstring
    The UUID of the trace this metric data is associated with, if any.
    spanUuidstring
    The UUID of the span this metric data is associated with, if any.
    testCaseIdstring
    The ID of the test case this metric data is associated with, if any.
    testRunIdstring
    The ID of the test run this metric data is associated with, if any.
    threadIdstring
    The ID of the thread this metric data is associated with, if any.
    namestring
    The name of the metric.
    multiTurnboolean
    Whether this metric was evaluated on a multi-turn conversation.
    scorenumber
    The final metric score.
    reasonstring
    The reason for the metric score, generated by the evaluation model at evaluation time.
    successboolean
    Whether the metric score is above the threshold.
    createdAtstring
    The time the metric data was created.
    evaluatedAtstring
    The time the metric was evaluated.
    thresholdnumber
    The threshold for the metric, which determines if the metric is passing or failing.
    strictModeboolean
    Whether the metric was run in strict mode, which outputs a binary score of 0 or 1.
    skippedboolean
    Whether the metric evaluation was skipped.
    evaluationModelstring
    The evaluation model used to run the evaluation.
    errorstring
    The error message if the evaluation failed.
    evaluationCostnumber
    The cost of running the evaluation.
    verboseLogsstring
    The verbose logs of the evaluation, which breaks down the metric score calculation into individual steps.
    successboolean
    Whether this test case passed all metric thresholds.
    runDurationnumber
    The duration of the test case evaluation in seconds.
    evaluationCostnumber
    The cost of evaluating this test case.
- multiTurnboolean
  Whether this test run contains multi-turn test cases.
- identifierstring
  A unique identifier for the test run.
- statusenum
  The current status of the test run.
  Show 4 enum valuesHide 4 enum values
  - IN_PROGRESS
  - COMPLETED
  - ERRORED
  - CANCELLED
- testsPassednumber
  The number of test cases that passed.
- testsFailednumber
  The number of test cases that failed.
- totalTestsnumber
  The total number of test cases in this test run.
- runDurationnumber
  The total duration of the test run in seconds.
deprecatedboolean
This is true if this endpoint is deprecated.

Built byConfident AI