Run Evals

(v2)

POST

Runs the metrics in metricCollection against your test cases and returns the test run id they were evaluated in. Send either single-turn test cases or multi-turn test cases, not both.

POST/v2/evaluate

curl -X POST "https://api.confident-ai.com/v2/evaluate" \
  -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>" \
  -H "Content-Type: application/json" \
  -d '{
  "metricCollection": "Collection Name",
  "testCases": [
    {
      "input": "How tall is mount everest?",
      "actualOutput": "No clue, pretty tall I guess?",
      "expectedOutput": "Mount Everest is 8,848 metres tall.",
      "retrievalContext": [
        "Everest is 8,848 metres tall."
      ],
      "toolsCalled": [
        {
          "name": "get_landmark_info",
          "type": "FUNCTION",
          "description": "This tool gives information about a mountain.",
          "inputParameters": {
            "mountain": "Everest"
          },
          "output": "8,848 metres",
          "reasoning": "The user asked for the height of a mountain."
        }
      ],
      "expectedTools": [
        {
          "name": "get_landmark_info",
          "type": "FUNCTION",
          "description": "This tool gives information about a mountain.",
          "inputParameters": {
            "mountain": "Everest"
          },
          "output": "8,848 metres",
          "reasoning": "The user asked for the height of a mountain."
        }
      ],
      "context": [
        "Everest is 8,848 metres tall."
      ],
      "tokenCost": 0.002,
      "inputTokenCount": 24,
      "outputTokenCount": 12,
      "name": "everest-height",
      "flaky": false,
      "imagesMapping": {
        "summit": {
          "url": "https://example.com/everest.png",
          "local": false
        }
      },
      "additionalMetadata": {
        "region": "Nepal"
      },
      "customColumnKeyValues": {
        "team": "search"
      },
      "tags": [
        "geography"
      ]
    }
  ],
  "hyperparameters": {
    "model": "gpt-4o-mini"
  },
  "identifier": "run-399-102"
}'

200

{
  "success": true,
  "data": {
    "id": "<TEST-RUN-ID>"
  },
  "link": "https://app.confident-ai.com/project/<PROJECT-ID>/test-runs/<TEST-RUN-ID>/test-cases",
  "deprecated": false
}

Headers

CONFIDENT_API_KEYstringRequired
The API key of your Confident AI project.

Request body

metricCollectionstringRequired
The name of the metric collection you wish to use for evaluation.
testCaseslist of object | objectRequired
This is the list of test cases to evaluate. Every test case in one request must be of the same kind — all single-turn, or all multi-turn.
Show 2 variantsHide 2 variants
- Single-Turn Test Case Inputobject
  A test case for a single exchange with your LLM application.
  Show 16 propertiesHide 16 properties
  - inputstringRequired
    This is the input to your LLM application.
  - actualOutputstring
    This is the actual output of your LLM application.
  - expectedOutputstring
    This is the expected output of your LLM application, which is the ideal actual output.
  - retrievalContextlist of strings
    This is the retrieval context of your LLM application.
  - toolsCalledlist of objects
    This is the tools called by your LLM application.
    Show 6 propertiesHide 6 properties
    namestringRequired
    This is the name of the tool.
    typeenum
    The type of the tool call, either a function or an MCP tool.
    Show 2 enum valuesHide 2 enum values
    FUNCTION
    MCP
    descriptionstring
    This is the description of the tool.
    inputParametersobject | null
    This is the input parameters that are passed to the tool.
    outputany
    This is the output of the tool.
    reasoningstring
    This is the reasoning your LLM provided for the tool call.
  - expectedToolslist of objects
    This is the expected tools to be called by the LLM application.
    Show 6 propertiesHide 6 properties
    namestringRequired
    This is the name of the tool.
    typeenum
    The type of the tool call, either a function or an MCP tool.
    Show 2 enum valuesHide 2 enum values
    FUNCTION
    MCP
    descriptionstring
    This is the description of the tool.
    inputParametersobject | null
    This is the input parameters that are passed to the tool.
    outputany
    This is the output of the tool.
    reasoningstring
    This is the reasoning your LLM provided for the tool call.
  - contextlist of strings
    This is the ideal retrieval context of your LLM application.
  - tokenCostnumber
    This is the cost of the tokens used by the LLM model.
  - inputTokenCountinteger
    This is the number of input tokens passed to the LLM model.
  - outputTokenCountinteger
    This is the number of output tokens generated by the LLM model.
  - namestring
    This is the name of your test case, it allows you to search and match test cases across different test runs.
  - flakyboolean
    This is true if the test case's verdict was non-deterministic across runs.
  - imagesMappingobject
    This is the mapping of image placeholders in your test case to the images they refer to.
  - additionalMetadataobject
    Additional metadata associated with this test case.
  - customColumnKeyValuesobject
    This is the custom column key values of the LLM application.
  - tagslist of strings
    This is the list of tags associated with the test case, which is useful for grouping and filtering for test cases.
- OR
- Multi-Turn Test Case Inputobject
  A test case for a conversation with your LLM application.
  Show 15 propertiesHide 15 properties
  - turnslist of objectsRequired
    This is the list of turns in the conversation.
    Show 7 propertiesHide 7 properties
    idstring
    The id of a turn assigned by Confident AI.
    roleenumRequired
    The role of the turn, either user or assistant.
    Show 2 enum valuesHide 2 enum values
    user
    assistant
    contentstringRequired
    The message content of the turn.
    userIdstring
    The user ID associated with the turn.
    retrievalContextarray | null
    The contexts retrieved to generate the LLM response for this turn.
    toolsCalledarray | null
    The tools called to generate the LLM response for this turn.
    Show 6 propertiesHide 6 properties
    namestringRequired
    This is the name of the tool.
    typeenum
    The type of the tool call, either a function or an MCP tool.
    Show 2 enum valuesHide 2 enum values
    FUNCTION
    MCP
    descriptionstring
    This is the description of the tool.
    inputParametersobject | null
    This is the input parameters that are passed to the tool.
    outputany
    This is the output of the tool.
    reasoningstring
    This is the reasoning your LLM provided for the tool call.
    metadataobject | null
    Additional metadata to associate with this turn.
  - scenariostring
    This is a description of the conversation context.
  - expectedOutcomestring
    This describes the expected outcome, or ideal conversation flow, of the conversation.
  - userDescriptionstring
    This is the description of the user in the conversation.
  - chatbotRolestring
    This is the role of the chatbot in the conversation.
  - contextlist of strings
    This is the ideal retrieval context of your LLM application.
  - tokenCostnumber
    This is the cost of the tokens used by the LLM model.
  - inputTokenCountinteger
    This is the number of input tokens passed to the LLM model.
  - outputTokenCountinteger
    This is the number of output tokens generated by the LLM model.
  - namestring
    This is the name of your test case, it allows you to search and match test cases across different test runs.
  - flakyboolean
    This is true if the test case's verdict was non-deterministic across runs.
  - imagesMappingobject
    This is the mapping of image placeholders in your test case to the images they refer to.
  - additionalMetadataobject
    Additional metadata associated with this test case.
  - customColumnKeyValuesobject
    This is the custom column key values of the LLM application.
  - tagslist of strings
    This is the list of tags associated with the test case, which is useful for grouping and filtering for test cases.
hyperparametersobject
This is any hyperparameters like model or prompt you wish to associate with the test run.
identifierstring
A unique identifier for the test run.

Response

Run Evals succeeded.

successboolean
Indicates if the request was successful.
dataobject
Show 1 propertyHide 1 property
- idstring
  This is the unique ID for the test run. This ID is generated by Confident AI and is not to be confused with the identifier provided by the user.
linkstring
This is the URL of the resource on the Confident AI platform.
deprecatedboolean
Indicates if this endpoint is deprecated.

Built byConfident AI