Run LLM Evals

POSThttps://api.confident-ai.com/v1/evaluate

Run online evals for your test cases using the metrics in metricCollection.

POST/v1/evaluate

curl -X POST "https://api.confident-ai.com/v1/evaluate" \
  -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>" \
  -H "Content-Type: application/json" \
  -d '{
  "metricCollection": "Collection Name",
  "llmTestCases": [
    {
      "input": "How tall is mount everest?",
      "actualOutput": "No clue, pretty tall I guess?"
    }
  ]
}'

200

{
  "success": true,
  "data": {
    "id": "TEST-RUN-ID"
  },
  "deprecated": false
}

Headers

CONFIDENT_API_KEYstringRequired
The API key of your Confident AI project.

Request body

metricCollectionstringRequired
The name of the metric collection you wish to use for evaluation.
llmTestCaseslist of objects
This is a list of single-turn test cases to evaluate. If you are evaluating multi-turn test cases, this should be null.
Show 8 propertiesHide 8 properties
- inputstringRequired
  This is the input to your LLM application.
- actualOutputstringRequired
  This is the actual output of your LLM application.
- namestring
  This is the name of your test case, it allows you to search and match test cases across different test runs.
- expectedOutputstring
  This is the expected output of your LLM application, which is the ideal actual output.
- retrievalContextlist of strings
  This is the retrieval context of your LLM application.
- contextlist of strings
  This is the ideal retrieval context of your LLM application.
- toolsCalledlist of objects
  This is the tools called by your LLM application.
  Show 5 propertiesHide 5 properties
  - namestringRequired
    This is the name of the tool.
  - descriptionstringRequired
    This is the description of the tool.
  - inputParametersobject
    This is the input parameters that are passed to the tool.
  - outputstring
    This is the output of the tool.
  - reasoningstring
    This is the reasoning your LLM provided for the tool call.
- expectedToolslist of objects
  This is the expected tools to be called by the LLM application.
  Show 5 propertiesHide 5 properties
  - namestringRequired
    This is the name of the tool.
  - descriptionstringRequired
    This is the description of the tool.
  - inputParametersobject
    This is the input parameters that are passed to the tool.
  - outputstring
    This is the output of the tool.
  - reasoningstring
    This is the reasoning your LLM provided for the tool call.
conversationalTestCaseslist of objects
This is a list of multi-turn test cases to evaluate. If you are evaluating single-turn test cases, this should be null.
Show 6 propertiesHide 6 properties
- turnslist of objectsRequired
  This is the list of turns in the conversation.
  Show 5 propertiesHide 5 properties
  - roleenumRequired
    The role of the turn, either user or assistant.
    Show 2 enum valuesHide 2 enum values
    user
    assistant
  - contentstringRequired
    The message content of the turn.
  - userIdstring
    The user ID associated with the turn.
  - retrievalContextlist of strings
    The contexts retrieved to generate the LLM response for this turn.
  - toolsCalledlist of objects
    The tools called to generate the LLM response for this turn.
    Show 5 propertiesHide 5 properties
    namestringRequired
    This is the name of the tool.
    descriptionstringRequired
    This is the description of the tool.
    inputParametersobject
    This is the input parameters that are passed to the tool.
    outputstring
    This is the output of the tool.
    reasoningstring
    This is the reasoning your LLM provided for the tool call.
- scenariostring
  This is a description of the conversation context.
- namestring
  This is the name of your test case, it allows you to search and match test cases across different test runs.
- expectedOutcomestring
  This describes the expected outcome, or ideal conversation flow, of the conversation.
- userDescriptionstring
  This is the description of the user in the conversation.
- chatbotRolestring
  This is the role of the chatbot in the conversation.
hyperparametersobject
This is any hyperparameters like model or prompt you wish to associate with the test run.
identifierstring
A unique identifier for the test run.

Response

successboolean
This is true if the test cases were successfully evaluated.
dataobject
Show 1 propertyHide 1 property
- idstring
  This is the unique ID for the test run. This ID is generated by Confident AI and is not to be confused with the identifier provided by the user.
deprecatedboolean
This is true if this endpoint is deprecated.

Built byConfident AI