Create Metric Collection

(v2)

POST

Creates a metric collection from the name and metricsSettings you specify and returns it. A metric that does not exist in the project, or does not match multiTurn, rejects the whole request.

POST/v2/metric-collections

curl -X POST "https://api.confident-ai.com/v2/metric-collections" \
  -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>" \
  -H "Content-Type: application/json" \
  -d '{
  "name": "RAG Quality",
  "multiTurn": false,
  "metricsSettings": [
    {
      "metric": {
        "name": "Answer Relevancy"
      },
      "activated": true,
      "threshold": 0.8,
      "includeReason": true,
      "strictMode": false,
      "sampleRate": 1,
      "evaluationModelProvider": "OPEN_AI",
      "evaluationModelName": "gpt-4o",
      "evaluationModelCredentialId": "<MODEL-CREDENTIAL-ID>",
      "decisionModelProvider": "OPEN_AI",
      "decisionModelName": "jev-latest",
      "evalMode": "LLM"
    }
  ],
  "sampleRate": 1,
  "inputTransformerId": "<TRANSFORMER-ID>",
  "outputTransformerId": null
}'

200

{
  "success": true,
  "data": {
    "id": "<METRIC-COLLECTION-ID>",
    "name": "RAG Quality",
    "multiTurn": false,
    "sampleRate": 1,
    "inputTransformerId": "<TRANSFORMER-ID>",
    "outputTransformerId": null,
    "metricsSettings": [
      {
        "metric": {
          "id": 1,
          "name": "Answer Relevancy"
        },
        "activated": true,
        "threshold": 0.8,
        "includeReason": true,
        "strictMode": false,
        "sampleRate": 1,
        "evaluationModelProvider": "OPEN_AI",
        "evaluationModelName": "gpt-4o",
        "evaluationModelCredentialId": null,
        "decisionModelProvider": "OPEN_AI",
        "decisionModelName": "jev-latest",
        "evalMode": "LLM"
      }
    ]
  },
  "deprecated": false
}

Headers

CONFIDENT_API_KEYstringRequired
The API key of your Confident AI project.

Request body

namestringRequired
The name of the metric collection, which must be unique within your project.
multiTurnboolean
This is true if the collection is multi-turn, which contains only multi-turn metrics. It cannot be changed once the collection exists.
metricsSettingslist of objects
The metrics in the collection with their settings. Each metric must exist in your project and match multiTurn.
Show 12 propertiesHide 12 properties
- metricobjectRequired
  A metric referenced by its name.
  Show 1 propertyHide 1 property
  - namestringRequired
    The name of the metric, which must match a metric in your project or one of Confident AI's built-in metrics.
- activatedboolean
  Whether this metric is activated. Only activated metrics are run during an evaluation.
- thresholdnumber
  The threshold this metric is scored against. A metric passes when its score is equal to or greater than the threshold.
- includeReasonboolean
  Whether a written reason explaining the metric's score is generated during evaluation.
- strictModeboolean
  Whether this metric runs in strict mode, which outputs a binary score of 0 or 1 instead of a continuous score.
- sampleRatenumber
  The probability that this metric is run for any given evaluation, between 0 and 1. Applied on top of the collection's own sampleRate.
- evaluationModelProviderenum | null
  This is the provider of the model.
  Show 20 enum valuesHide 20 enum values
  - OPEN_AI
  - CUSTOM
  - CONFIDENT_AI
  - BEDROCK
  - ANTHROPIC
  - GEMINI
  - X_AI
  - DEEPSEEK
  - MOONSHOT_AI
  - VERTEX_AI
  - AZURE
  - MISTRAL
  - PERPLEXITY
  - OPEN_ROUTER
  - PORTKEY
  - LITE_LLM
  - TRUE_FOUNDRY
  - HUGGING_FACE
  - TYPE_SAFE
  - FAL
- evaluationModelNamestring | null
  The name of the model this metric is evaluated with. Required whenever evaluationModelProvider is set to anything other than CONFIDENT_AI, and has no effect without a provider.
- evaluationModelCredentialIdstring | null
  For BEDROCK and AZURE, the id of the stored credential this metric is evaluated with. Omit it or send null to use the provider's default credential.
- decisionModelProviderenum | null
  This is the provider of the model.
  Show 20 enum valuesHide 20 enum values
  - OPEN_AI
  - CUSTOM
  - CONFIDENT_AI
  - BEDROCK
  - ANTHROPIC
  - GEMINI
  - X_AI
  - DEEPSEEK
  - MOONSHOT_AI
  - VERTEX_AI
  - AZURE
  - MISTRAL
  - PERPLEXITY
  - OPEN_ROUTER
  - PORTKEY
  - LITE_LLM
  - TRUE_FOUNDRY
  - HUGGING_FACE
  - TYPE_SAFE
  - FAL
- decisionModelNamestring | null
  The name of the decision model this metric decides with, used in the HYBRID and DECISION eval modes. Required whenever decisionModelProvider is set, and has no effect without a provider.
- evalModeenum | null
  Who decides in an LLM-as-a-judge metric. LLM: the evaluation model runs the whole metric. HYBRID: the evaluation model extracts and explains, and the decision model answers each decision, falling back to the evaluation model if a decision fails. DECISION: the decision model runs the whole metric in one request, with no evaluation model.
  Show 3 enum valuesHide 3 enum values
  - LLM
  - HYBRID
  - DECISION
sampleRatenumber
The share of eligible entities the whole collection is run against, between 0 and 1. Applied on top of each metric's own sampleRate. Defaults to 1 on create; omit it on an update to leave it unchanged.
inputTransformerIdstring | null
The id of a transformer that reshapes the payload before evaluation. Send null to unset it.
outputTransformerIdstring | null
The id of a transformer that reshapes the result after evaluation. Send null to unset it.

Response

Create Metric Collection succeeded.

successboolean
Indicates if the request was successful.
dataobject
A metric collection: its name, sampling and transformer configuration, and the settings for every metric inside it.
Show 7 propertiesHide 7 properties
- idstring
  This is the id of the metric collection.
- namestring
  This is the name of the metric collection, which you supply to the evals API to run evaluations remotely.
- multiTurnboolean
  Whether this is a multi-turn collection. Multi-turn collections contain only multi-turn metrics and evaluate conversations rather than single test cases.
- sampleRatenumber
  The share of eligible entities this collection is run against, between 0 and 1. Applied on top of each metric's own sampleRate.
- inputTransformerIdstring | null
  The id of the transformer that reshapes the payload before evaluation, or null when the collection does not use one.
- outputTransformerIdstring | null
  The id of the transformer that reshapes the result after evaluation, or null when the collection does not use one.
- metricsSettingslist of objects
  The metrics in the collection with their settings.
  Show 12 propertiesHide 12 properties
  - metricobject
    A metric as it appears inside a collection, by id and name.
    Show 2 propertiesHide 2 properties
    idinteger
    This is the id of the metric.
    namestring
    This is the name of the metric.
  - activatedboolean
    Whether this metric is activated. Only activated metrics are run during an evaluation.
  - thresholdnumber
    The threshold this metric is scored against. A metric passes when its score is equal to or greater than the threshold.
  - includeReasonboolean
    Whether a written reason explaining the metric's score is generated during evaluation.
  - strictModeboolean
    Whether this metric runs in strict mode, which outputs a binary score of 0 or 1 instead of a continuous score.
  - sampleRatenumber
    The probability that this metric is run for any given evaluation, between 0 and 1. Applied on top of the collection's own sampleRate.
  - evaluationModelProviderenum | null
    This is the provider of the model.
    Show 20 enum valuesHide 20 enum values
    OPEN_AI
    CUSTOM
    CONFIDENT_AI
    BEDROCK
    ANTHROPIC
    GEMINI
    X_AI
    DEEPSEEK
    MOONSHOT_AI
    VERTEX_AI
    AZURE
    MISTRAL
    PERPLEXITY
    OPEN_ROUTER
    PORTKEY
    LITE_LLM
    TRUE_FOUNDRY
    HUGGING_FACE
    TYPE_SAFE
    FAL
  - evaluationModelNamestring | null
    The name of the model this metric is evaluated with, or null when the project's default evaluation model is used.
  - evaluationModelCredentialIdstring | null
    For BEDROCK and AZURE, the id of the stored credential this metric is evaluated with, or null when it uses the provider's default credential.
  - decisionModelProviderenum | null
    This is the provider of the model.
    Show 20 enum valuesHide 20 enum values
    OPEN_AI
    CUSTOM
    CONFIDENT_AI
    BEDROCK
    ANTHROPIC
    GEMINI
    X_AI
    DEEPSEEK
    MOONSHOT_AI
    VERTEX_AI
    AZURE
    MISTRAL
    PERPLEXITY
    OPEN_ROUTER
    PORTKEY
    LITE_LLM
    TRUE_FOUNDRY
    HUGGING_FACE
    TYPE_SAFE
    FAL
  - decisionModelNamestring | null
    The name of the decision model this metric decides with, or null when the project's default decision model is used.
  - evalModeenum | null
    Who decides in an LLM-as-a-judge metric. LLM: the evaluation model runs the whole metric. HYBRID: the evaluation model extracts and explains, and the decision model answers each decision, falling back to the evaluation model if a decision fails. DECISION: the decision model runs the whole metric in one request, with no evaluation model.
    Show 3 enum valuesHide 3 enum values
    LLM
    HYBRID
    DECISION
deprecatedboolean
Indicates if this endpoint is deprecated.

Built byConfident AI