Create Metric

(v2)

POST

Creates a custom metric in your Confident AI project and returns it. A GEVAL metric scores against criteria or evaluationSteps; a DAG metric needs algorithm set to DAG and a dag.

POST/v2/metrics

curl -X POST "https://api.confident-ai.com/v2/metrics" \
  -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>" \
  -H "Content-Type: application/json" \
  -d '{
  "name": "Correctness",
  "multiTurn": false,
  "criteria": "Determine if the actual output is correct based on the expected output.",
  "evaluationParams": [
    "actualOutput",
    "expectedOutput"
  ]
}'

200

{
  "success": true,
  "data": {
    "id": "<METRIC-ID>",
    "name": "Correctness",
    "algorithm": "DEFAULT",
    "criteria": "Determine if the actual output is correct based on the expected output.",
    "evaluationSteps": null,
    "rubric": [
      {
        "scoreRange": [
          8,
          10
        ],
        "expectedOutcome": "The answer is factually correct and complete."
      }
    ],
    "dag": {
      "nodes": {
        "root": {
          "type": "BinaryJudgementNode",
          "criteria": "Does the actual output answer the input?",
          "evaluation_params": [
            "input",
            "actual_output"
          ],
          "children": [
            "pass",
            "fail"
          ]
        },
        "pass": {
          "type": "VerdictNode",
          "verdict": true,
          "score": 10
        },
        "fail": {
          "type": "VerdictNode",
          "verdict": false,
          "score": 0
        }
      }
    },
    "questions": [
      {
        "type": "noul",
        "question": "Does the actual output answer the input?",
        "weight": 2
      }
    ],
    "promptConfig": {
      "messages": [
        {
          "role": "system",
          "content": "You grade support answers."
        },
        {
          "role": "user",
          "content": "QUESTION: {{question}}\nANSWER: {{answer}}"
        }
      ],
      "variables": {
        "question": {
          "field": "input",
          "path": [
            "messages",
            0,
            "content"
          ]
        },
        "answer": {
          "field": "output",
          "path": [
            "messages",
            -1,
            "content"
          ]
        }
      }
    },
    "multiTurn": false,
    "requiredParameters": [
      "actualOutput",
      "expectedOutput"
    ]
  },
  "deprecated": false
}

Headers

CONFIDENT_API_KEYstringRequired
The API key of your Confident AI project.

Request body

namestringRequired
The name of the metric, unique within your project.
multiTurnboolean
This is true when the metric evaluates conversations rather than single test cases. It decides which evaluationParams are valid and cannot be changed later.
criteriastring
The criteria the metric scores against. A GEVAL metric needs criteria or evaluationSteps.
evaluationStepslist of strings
The steps the metric follows to score, as an alternative to criteria.
evaluationParamslist of enums
The test case fields the metric evaluates. A single-turn metric needs at least one, and every field must match multiTurn.
Show 15 enum valuesHide 15 enum values
- input
- actualOutput
- expectedOutput
- context
- expectedTools
- content
- role
- scenario
- expectedOutcome
- audio
- turns
- toolsCalled
- retrievalContext
- metadata
- tags
rubriclist of objects
Score ranges that anchor how the metric scores.
Show 2 propertiesHide 2 properties
- scoreRangelist of anyRequired
  The inclusive start and end of the score range this outcome describes, each between 0 and 10, with the start no greater than the end.
- expectedOutcomestringRequired
  What a response scoring in this range looks like.
algorithmenum
The algorithm the metric is evaluated with. GEVAL scores against criteria or evaluation steps, DAG walks a decision graph, JEVAL asks a decision model bounded questions, PROMPT sends your own judge prompt to the evaluation model, CODE runs your own code, and DEFAULT is a built-in metric.
Show 6 enum valuesHide 6 enum values
- DEFAULT
- DAG
- GEVAL
- JEVAL
- CODE
- PROMPT
dagobject
The decision graph of a DAG metric. It is validated on write, and metric references are resolved against the metrics in your project.
Show 1 propertyHide 1 property
- nodesobjectRequired
  The graph's nodes keyed by node id, as serialized by deepeval's DAG metric. Judgement nodes list their children by id; verdict nodes carry a verdict and a score, or point at another metric by metric_name.
questionslist of object | object | object
The questions a JEVAL metric asks the decision model. Required when algorithm is JEVAL.
Show 3 variantsHide 3 variants
- Jev Noul Questionobject
  A question the decision model answers true or false. trueDescription and falseDescription tell it what each answer means.
  Show 5 propertiesHide 5 properties
  - typeenumRequired
    Marks this as a noul question.
    Show 1 enum valueHide 1 enum value
    noul
  - questionstringRequired
    The question the decision model answers.
  - weightnumber
    How heavily this question counts towards the metric's score, relative to the metric's other questions.
  - trueDescriptionstring
    What a true answer to this question means.
  - falseDescriptionstring
    What a false answer to this question means.
- OR
- Jev Score Questionobject
  A question the decision model answers by picking one of levels, which are ordered worst first.
  Show 4 propertiesHide 4 properties
  - typeenumRequired
    Marks this as a score question.
    Show 1 enum valueHide 1 enum value
    score
  - questionstringRequired
    The question the decision model answers.
  - weightnumber
    How heavily this question counts towards the metric's score, relative to the metric's other questions.
  - levelslist of stringsRequired
    Between 2 and 10 ordered levels, worst first.
- OR
- Jev Choice Questionobject
  A question the decision model answers by picking one of options, each carrying the credit that answer earns.
  Show 4 propertiesHide 4 properties
  - typeenumRequired
    Marks this as a choice question.
    Show 1 enum valueHide 1 enum value
    choice
  - questionstringRequired
    The question the decision model answers.
  - weightnumber
    How heavily this question counts towards the metric's score, relative to the metric's other questions.
  - optionsobjectRequired
    Each option maps to a credit between 0 and 1, or null to mark the question not applicable when that option is likely.
promptConfigobject
The judge prompt of a PROMPT metric, and how its variables are filled. The judge scores from 0 to 10 and the score is normalized to 0 to 1.
Show 2 propertiesHide 2 properties
- messageslist of objectsRequired
  The judge prompt, as one or more messages.
  Show 2 propertiesHide 2 properties
  - roleenumRequired
    Who the message is from. A system message is only allowed first.
    Show 3 enum valuesHide 3 enum values
    system
    user
    assistant
  - contentstringRequired
    The message text. Use {{name}} to insert a mapped variable.
- variablesobjectRequired
  Maps every {{name}} used in the messages to the data it is filled with.

Response

Create Metric succeeded.

successboolean
Indicates if the request was successful.
dataobject
A custom metric: how it scores, and which test case fields it needs.
Show 11 propertiesHide 11 properties
- idstring
  This is the unique id of the metric.
- namestring
  This is the name of the metric, unique within your project.
- algorithmenum | null
  The algorithm the metric is evaluated with. GEVAL scores against criteria or evaluation steps, DAG walks a decision graph, JEVAL asks a decision model bounded questions, PROMPT sends your own judge prompt to the evaluation model, CODE runs your own code, and DEFAULT is a built-in metric.
  Show 6 enum valuesHide 6 enum values
  - DEFAULT
  - DAG
  - GEVAL
  - JEVAL
  - CODE
  - PROMPT
- criteriastring | null
  This is the criteria the metric scores against, or null when it uses evaluation steps.
- evaluationStepsarray | null
  These are the steps the metric follows to score, or null when it uses criteria.
- rubricarray | null
  These are the score ranges that anchor how the metric scores, or null.
  Show 2 propertiesHide 2 properties
  - scoreRangelist of any
    The inclusive start and end of the score range this outcome describes, each between 0 and 10, with the start no greater than the end.
  - expectedOutcomestring
    What a response scoring in this range looks like.
- dagobject | null
  The decision graph of a DAG metric. It is validated on write, and metric references are resolved against the metrics in your project.
  Show 1 propertyHide 1 property
  - nodesobject
    The graph's nodes keyed by node id, as serialized by deepeval's DAG metric. Judgement nodes list their children by id; verdict nodes carry a verdict and a score, or point at another metric by metric_name.
- questionslist of object | object | object | null
  Show 3 variantsHide 3 variants
  - Jev Noul Questionobject
    A question the decision model answers true or false. trueDescription and falseDescription tell it what each answer means.
    Show 5 propertiesHide 5 properties
    typeenum
    Marks this as a noul question.
    Show 1 enum valueHide 1 enum value
    noul
    questionstring
    The question the decision model answers.
    weightnumber
    How heavily this question counts towards the metric's score, relative to the metric's other questions.
    trueDescriptionstring
    What a true answer to this question means.
    falseDescriptionstring
    What a false answer to this question means.
  - OR
  - Jev Score Questionobject
    A question the decision model answers by picking one of levels, which are ordered worst first.
    Show 4 propertiesHide 4 properties
    typeenum
    Marks this as a score question.
    Show 1 enum valueHide 1 enum value
    score
    questionstring
    The question the decision model answers.
    weightnumber
    How heavily this question counts towards the metric's score, relative to the metric's other questions.
    levelslist of strings
    Between 2 and 10 ordered levels, worst first.
  - OR
  - Jev Choice Questionobject
    A question the decision model answers by picking one of options, each carrying the credit that answer earns.
    Show 4 propertiesHide 4 properties
    typeenum
    Marks this as a choice question.
    Show 1 enum valueHide 1 enum value
    choice
    questionstring
    The question the decision model answers.
    weightnumber
    How heavily this question counts towards the metric's score, relative to the metric's other questions.
    optionsobject
    Each option maps to a credit between 0 and 1, or null to mark the question not applicable when that option is likely.
- promptConfigobject | null
  The judge prompt of a PROMPT metric, and how its variables are filled. The judge scores from 0 to 10 and the score is normalized to 0 to 1.
  Show 2 propertiesHide 2 properties
  - messageslist of objects
    The judge prompt, as one or more messages.
    Show 2 propertiesHide 2 properties
    roleenum
    Who the message is from. A system message is only allowed first.
    Show 3 enum valuesHide 3 enum values
    system
    user
    assistant
    contentstring
    The message text. Use {{name}} to insert a mapped variable.
  - variablesobject
    Maps every {{name}} used in the messages to the data it is filled with.
- multiTurnboolean
  This is true when the metric evaluates conversations rather than single test cases.
- requiredParameterslist of enums
  The test case fields the metric needs to run.
  Show 15 enum valuesHide 15 enum values
  - input
  - actualOutput
  - expectedOutput
  - context
  - expectedTools
  - content
  - role
  - scenario
  - expectedOutcome
  - audio
  - turns
  - toolsCalled
  - retrievalContext
  - metadata
  - tags
deprecatedboolean
Indicates if this endpoint is deprecated.

Built byConfident AI