Create Collection

POSThttps://api.confident-ai.com/v2/metric-collections

Creates a metric collection from the name and metricsSettings you specify and returns it. A metric that does not exist in the project, or does not match multiTurn, rejects the whole request.

POST/v2/metric-collections

curl -X POST "https://api.confident-ai.com/v2/metric-collections" \
  -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>" \
  -H "Content-Type: application/json" \
  -d '{
  "name": "RAG Quality",
  "multiTurn": false,
  "metricsSettings": [
    {
      "metric": {
        "name": "Answer Relevancy"
      },
      "activated": true,
      "threshold": 0.8,
      "includeReason": true,
      "strictMode": false,
      "sampleRate": 1,
      "evaluationModelProvider": "OPEN_AI",
      "evaluationModelName": "gpt-4o"
    }
  ],
  "sampleRate": 1,
  "inputTransformerId": "<TRANSFORMER-ID>",
  "outputTransformerId": null
}'

200

{
  "success": true,
  "data": {
    "id": "<METRIC-COLLECTION-ID>",
    "name": "RAG Quality",
    "multiTurn": false,
    "sampleRate": 1,
    "inputTransformerId": "<TRANSFORMER-ID>",
    "outputTransformerId": null,
    "metricsSettings": [
      {
        "metric": {
          "id": 1,
          "name": "Answer Relevancy"
        },
        "activated": true,
        "threshold": 0.8,
        "includeReason": true,
        "strictMode": false,
        "sampleRate": 1,
        "evaluationModelProvider": "OPEN_AI",
        "evaluationModelName": "gpt-4o"
      }
    ]
  },
  "deprecated": false
}

Headers

CONFIDENT_API_KEYstringRequired
The API key of your Confident AI project.

Request body

namestringRequired
The name of the metric collection, which must be unique within your project.
multiTurnboolean
This is true if the collection is multi-turn, which contains only multi-turn metrics. It cannot be changed once the collection exists.
metricsSettingslist of objects
The metrics in the collection with their settings. Each metric must exist in your project and match multiTurn.
Show 8 propertiesHide 8 properties
- metricobjectRequired
  A metric referenced by its name.
  Show 1 propertyHide 1 property
  - namestringRequired
    The name of the metric, which must match a metric in your project or one of Confident AI's built-in metrics.
- activatedboolean
  Whether this metric is activated. Only activated metrics are run during an evaluation.
- thresholdnumber
  The threshold this metric is scored against. A metric passes when its score is equal to or greater than the threshold.
- includeReasonboolean
  Whether a written reason explaining the metric's score is generated during evaluation.
- strictModeboolean
  Whether this metric runs in strict mode, which outputs a binary score of 0 or 1 instead of a continuous score.
- sampleRatenumber
  The probability that this metric is run for any given evaluation, between 0 and 1. Applied on top of the collection's own sampleRate.
- evaluationModelProviderenum | null
  This is the provider of the model.
  Show 18 enum valuesHide 18 enum values
  - OPEN_AI
  - CUSTOM
  - CONFIDENT_AI
  - BEDROCK
  - ANTHROPIC
  - GEMINI
  - X_AI
  - DEEPSEEK
  - MOONSHOT_AI
  - VERTEX_AI
  - AZURE
  - MISTRAL
  - PERPLEXITY
  - OPEN_ROUTER
  - PORTKEY
  - LITE_LLM
  - TRUE_FOUNDRY
  - HUGGING_FACE
- evaluationModelNamestring | null
  The name of the model this metric is evaluated with. Required whenever evaluationModelProvider is set to anything other than CONFIDENT_AI, and has no effect without a provider.
sampleRatenumber
The share of eligible entities the whole collection is run against, between 0 and 1. Applied on top of each metric's own sampleRate. Defaults to 1.
inputTransformerIdstring | null
The id of a transformer that reshapes the payload before evaluation. Send null to unset it.
outputTransformerIdstring | null
The id of a transformer that reshapes the result after evaluation. Send null to unset it.

Response

Create Collection succeeded.

successboolean
Indicates if the request was successful.
dataobject
A metric collection: its name, sampling and transformer configuration, and the settings for every metric inside it.
Show 7 propertiesHide 7 properties
- idstring
  This is the id of the metric collection.
- namestring
  This is the name of the metric collection, which you supply to the evals API to run evaluations remotely.
- multiTurnboolean
  Whether this is a multi-turn collection. Multi-turn collections contain only multi-turn metrics and evaluate conversations rather than single test cases.
- sampleRatenumber
  The share of eligible entities this collection is run against, between 0 and 1. Applied on top of each metric's own sampleRate.
- inputTransformerIdstring | null
  The id of the transformer that reshapes the payload before evaluation, or null when the collection does not use one.
- outputTransformerIdstring | null
  The id of the transformer that reshapes the result after evaluation, or null when the collection does not use one.
- metricsSettingslist of objects
  The metrics in the collection with their settings.
  Show 8 propertiesHide 8 properties
  - metricobject
    A metric as it appears inside a collection, by id and name.
    Show 2 propertiesHide 2 properties
    idinteger
    This is the id of the metric.
    namestring
    This is the name of the metric.
  - activatedboolean
    Whether this metric is activated. Only activated metrics are run during an evaluation.
  - thresholdnumber
    The threshold this metric is scored against. A metric passes when its score is equal to or greater than the threshold.
  - includeReasonboolean
    Whether a written reason explaining the metric's score is generated during evaluation.
  - strictModeboolean
    Whether this metric runs in strict mode, which outputs a binary score of 0 or 1 instead of a continuous score.
  - sampleRatenumber
    The probability that this metric is run for any given evaluation, between 0 and 1. Applied on top of the collection's own sampleRate.
  - evaluationModelProviderenum | null
    This is the provider of the model.
    Show 18 enum valuesHide 18 enum values
    OPEN_AI
    CUSTOM
    CONFIDENT_AI
    BEDROCK
    ANTHROPIC
    GEMINI
    X_AI
    DEEPSEEK
    MOONSHOT_AI
    VERTEX_AI
    AZURE
    MISTRAL
    PERPLEXITY
    OPEN_ROUTER
    PORTKEY
    LITE_LLM
    TRUE_FOUNDRY
    HUGGING_FACE
  - evaluationModelNamestring | null
    The name of the model this metric is evaluated with, or null when the project's default evaluation model is used.
deprecatedboolean
Indicates if this endpoint is deprecated.

Built byConfident AI