Batch Create Metrics

(v2)

POST

Creates several GEVAL metrics at once and returns the ones created. Metrics whose name already exists in the project are skipped. DAG metrics must be created one at a time.

POST/v2/metrics-batch

curl -X POST "https://api.confident-ai.com/v2/metrics-batch" \
  -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>" \
  -H "Content-Type: application/json" \
  -d '{
  "metrics": [
    {
      "name": "Correctness",
      "multiTurn": false,
      "criteria": "Determine if the actual output is correct based on the expected output.",
      "evaluationSteps": [
        "Compare the actual output with the expected output.",
        "Penalise any factual contradiction."
      ],
      "evaluationParams": [
        "actualOutput",
        "expectedOutput"
      ],
      "rubric": [
        {
          "scoreRange": [
            8,
            10
          ],
          "expectedOutcome": "The answer is factually correct and complete."
        }
      ],
      "algorithm": "DEFAULT",
      "dag": {
        "nodes": {
          "root": {
            "type": "BinaryJudgementNode",
            "criteria": "Does the actual output answer the input?",
            "evaluation_params": [
              "input",
              "actual_output"
            ],
            "children": [
              "pass",
              "fail"
            ]
          },
          "pass": {
            "type": "VerdictNode",
            "verdict": true,
            "score": 10
          },
          "fail": {
            "type": "VerdictNode",
            "verdict": false,
            "score": 0
          }
        }
      },
      "questions": [
        {
          "type": "noul",
          "question": "Does the actual output answer the input?",
          "weight": 2
        }
      ],
      "promptConfig": {
        "messages": [
          {
            "role": "system",
            "content": "You grade support answers."
          },
          {
            "role": "user",
            "content": "QUESTION: {{question}}\nANSWER: {{answer}}"
          }
        ],
        "variables": {
          "question": {
            "field": "input",
            "path": [
              "messages",
              0,
              "content"
            ]
          },
          "answer": {
            "field": "output",
            "path": [
              "messages",
              -1,
              "content"
            ]
          }
        }
      }
    }
  ]
}'

200

{
  "success": true,
  "data": {
    "metrics": [
      {
        "id": "<METRIC-ID>",
        "name": "Correctness",
        "algorithm": "DEFAULT",
        "criteria": "Determine if the actual output is correct based on the expected output.",
        "evaluationSteps": null,
        "rubric": [
          {
            "scoreRange": [
              8,
              10
            ],
            "expectedOutcome": "The answer is factually correct and complete."
          }
        ],
        "dag": {
          "nodes": {
            "root": {
              "type": "BinaryJudgementNode",
              "criteria": "Does the actual output answer the input?",
              "evaluation_params": [
                "input",
                "actual_output"
              ],
              "children": [
                "pass",
                "fail"
              ]
            },
            "pass": {
              "type": "VerdictNode",
              "verdict": true,
              "score": 10
            },
            "fail": {
              "type": "VerdictNode",
              "verdict": false,
              "score": 0
            }
          }
        },
        "questions": [
          {
            "type": "noul",
            "question": "Does the actual output answer the input?",
            "weight": 2
          }
        ],
        "promptConfig": {
          "messages": [
            {
              "role": "system",
              "content": "You grade support answers."
            },
            {
              "role": "user",
              "content": "QUESTION: {{question}}\nANSWER: {{answer}}"
            }
          ],
          "variables": {
            "question": {
              "field": "input",
              "path": [
                "messages",
                0,
                "content"
              ]
            },
            "answer": {
              "field": "output",
              "path": [
                "messages",
                -1,
                "content"
              ]
            }
          }
        },
        "multiTurn": false,
        "requiredParameters": [
          "actualOutput",
          "expectedOutput"
        ]
      }
    ]
  },
  "deprecated": false
}

Headers

CONFIDENT_API_KEYstringRequired
The API key of your Confident AI project.

Request body

metricslist of objectsRequired
The metrics to create. Names must be unique within the batch for the same multiTurn, and DAG metrics are not accepted here.
Show 10 propertiesHide 10 properties
- namestringRequired
  The name of the metric, unique within your project.
- multiTurnboolean
  This is true when the metric evaluates conversations rather than single test cases. It decides which evaluationParams are valid and cannot be changed later.
- criteriastring
  The criteria the metric scores against. A GEVAL metric needs criteria or evaluationSteps.
- evaluationStepslist of strings
  The steps the metric follows to score, as an alternative to criteria.
- evaluationParamslist of enums
  The test case fields the metric evaluates. A single-turn metric needs at least one, and every field must match multiTurn.
  Show 14 enum valuesHide 14 enum values
  - input
  - actualOutput
  - expectedOutput
  - context
  - expectedTools
  - content
  - role
  - scenario
  - expectedOutcome
  - turns
  - toolsCalled
  - retrievalContext
  - metadata
  - tags
- rubriclist of objects
  Score ranges that anchor how the metric scores.
  Show 2 propertiesHide 2 properties
  - scoreRangelist of anyRequired
    The inclusive start and end of the score range this outcome describes, each between 0 and 10, with the start no greater than the end.
  - expectedOutcomestringRequired
    What a response scoring in this range looks like.
- algorithmenum
  The algorithm the metric is evaluated with. GEVAL scores against criteria or evaluation steps, DAG walks a decision graph, JEVAL asks a decision model bounded questions, PROMPT sends your own judge prompt to the evaluation model, CODE runs your own code, and DEFAULT is a built-in metric.
  Show 6 enum valuesHide 6 enum values
  - DEFAULT
  - DAG
  - GEVAL
  - JEVAL
  - CODE
  - PROMPT
- dagobject
  The decision graph of a DAG metric. It is validated on write, and metric references are resolved against the metrics in your project.
  Show 1 propertyHide 1 property
  - nodesobjectRequired
    The graph's nodes keyed by node id, as serialized by deepeval's DAG metric. Judgement nodes list their children by id; verdict nodes carry a verdict and a score, or point at another metric by metric_name.
- questionslist of object | object | object
  The questions a JEVAL metric asks the decision model. Required when algorithm is JEVAL.
  Show 3 variantsHide 3 variants
  - object
    Show 5 propertiesHide 5 properties
    typeenumRequired
    Show 1 enum valueHide 1 enum value
    noul
    questionstringRequired
    weightnumber
    trueDescriptionstring
    falseDescriptionstring
  - OR
  - object
    Show 4 propertiesHide 4 properties
    typeenumRequired
    Show 1 enum valueHide 1 enum value
    score
    questionstringRequired
    weightnumber
    levelslist of stringsRequired
    Between 2 and 10 ordered levels, worst first.
  - OR
  - object
    Show 4 propertiesHide 4 properties
    typeenumRequired
    Show 1 enum valueHide 1 enum value
    choice
    questionstringRequired
    weightnumber
    optionsobjectRequired
    Each option maps to a credit between 0 and 1, or null to mark the question not applicable when that option is likely.
- promptConfigobject
  The judge prompt of a PROMPT metric, and how its variables are filled. The judge scores from 0 to 10 and the score is normalized to 0 to 1.
  Show 2 propertiesHide 2 properties
  - messageslist of objectsRequired
    The judge prompt, as one or more messages.
    Show 2 propertiesHide 2 properties
    roleenumRequired
    Who the message is from. A system message is only allowed first.
    Show 3 enum valuesHide 3 enum values
    system
    user
    assistant
    contentstringRequired
    The message text. Use {{name}} to insert a mapped variable.
  - variablesobjectRequired
    Maps every {{name}} used in the messages to the data it is filled with.

Response

Batch Create Metrics succeeded.

successboolean
Indicates if the request was successful.
dataobject
Show 1 propertyHide 1 property
- metricslist of objects
  This is the list of metrics.
  Show 11 propertiesHide 11 properties
  - idstring
    This is the unique id of the metric.
  - namestring
    This is the name of the metric, unique within your project.
  - algorithmenum | null
    The algorithm the metric is evaluated with. GEVAL scores against criteria or evaluation steps, DAG walks a decision graph, JEVAL asks a decision model bounded questions, PROMPT sends your own judge prompt to the evaluation model, CODE runs your own code, and DEFAULT is a built-in metric.
    Show 6 enum valuesHide 6 enum values
    DEFAULT
    DAG
    GEVAL
    JEVAL
    CODE
    PROMPT
  - criteriastring | null
    This is the criteria the metric scores against, or null when it uses evaluation steps.
  - evaluationStepsarray | null
    These are the steps the metric follows to score, or null when it uses criteria.
  - rubricarray | null
    These are the score ranges that anchor how the metric scores, or null.
    Show 2 propertiesHide 2 properties
    scoreRangelist of any
    The inclusive start and end of the score range this outcome describes, each between 0 and 10, with the start no greater than the end.
    expectedOutcomestring
    What a response scoring in this range looks like.
  - dagobject | null
    The decision graph of a DAG metric. It is validated on write, and metric references are resolved against the metrics in your project.
    Show 1 propertyHide 1 property
    nodesobject
    The graph's nodes keyed by node id, as serialized by deepeval's DAG metric. Judgement nodes list their children by id; verdict nodes carry a verdict and a score, or point at another metric by metric_name.
  - questionslist of object | object | object | null
    Show 3 variantsHide 3 variants
    object
    Show 5 propertiesHide 5 properties
    typeenum
    Show 1 enum valueHide 1 enum value
    noul
    questionstring
    weightnumber
    trueDescriptionstring
    falseDescriptionstring
    OR
    object
    Show 4 propertiesHide 4 properties
    typeenum
    Show 1 enum valueHide 1 enum value
    score
    questionstring
    weightnumber
    levelslist of strings
    Between 2 and 10 ordered levels, worst first.
    OR
    object
    Show 4 propertiesHide 4 properties
    typeenum
    Show 1 enum valueHide 1 enum value
    choice
    questionstring
    weightnumber
    optionsobject
    Each option maps to a credit between 0 and 1, or null to mark the question not applicable when that option is likely.
  - promptConfigobject | null
    The judge prompt of a PROMPT metric, and how its variables are filled. The judge scores from 0 to 10 and the score is normalized to 0 to 1.
    Show 2 propertiesHide 2 properties
    messageslist of objects
    The judge prompt, as one or more messages.
    Show 2 propertiesHide 2 properties
    roleenum
    Who the message is from. A system message is only allowed first.
    Show 3 enum valuesHide 3 enum values
    system
    user
    assistant
    contentstring
    The message text. Use {{name}} to insert a mapped variable.
    variablesobject
    Maps every {{name}} used in the messages to the data it is filled with.
  - multiTurnboolean
    This is true when the metric evaluates conversations rather than single test cases.
  - requiredParameterslist of enums
    The test case fields the metric needs to run.
    Show 14 enum valuesHide 14 enum values
    input
    actualOutput
    expectedOutput
    context
    expectedTools
    content
    role
    scenario
    expectedOutcome
    turns
    toolsCalled
    retrievalContext
    metadata
    tags
deprecatedboolean
Indicates if this endpoint is deprecated.

Built byConfident AI