Create Evaluation Rule

POSThttps://api.confident-ai.com/v1/evaluation-rules

Creates a standing rule that runs a metric collection against matching production traces, spans, or threads as they arrive. Running metrics consumes LLM usage.

The metric collection's turn type must match: THREAD rules require a multi-turn collection, TRACE and SPAN rules require a single-turn one. Only one enabled THREAD rule may target a given metric collection. Requires the Starter plan or above.

POST/v1/evaluation-rules

curl -X POST "https://api.confident-ai.com/v1/evaluation-rules" \
  -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>" \
  -H "Content-Type: application/json" \
  -d '{
  "name": "Score production answers",
  "dataModel": "TRACE",
  "metricCollection": "Answer Quality",
  "sampleRate": 0.2
}'

200

{
  "success": true,
  "data": {
    "id": "EVALUATION-RULE-ID"
  },
  "link": "https://app.confident-ai.com/project/<PROJECT-ID>/workflows"
}

Headers

CONFIDENT_API_KEYstringRequired
The API key of your Confident AI project.

Request body

descriptionstring
A note about what the rule checks.
enabledboolean
Whether the rule runs. Defaults to true on create.
sampleRatenumber
The fraction of matching items to evaluate. Defaults to 1 (all of them).
spanTypeenum
Only evaluate spans of this type. Allowed only when dataModel is SPAN, and cleared automatically if the rule moves off SPAN.
Show 5 enum valuesHide 5 enum values
- SPAN
- AGENT
- TOOL
- RETRIEVER
- LLM
filtersobject
A set of filter groups combined by a top-level operator.
Show 2 propertiesHide 2 properties
- operatorenumRequired
  How filters or groups are combined.
  Show 2 enum valuesHide 2 enum values
  - AND
  - OR
- groupslist of objectsRequired
  The filter groups.
  Show 2 propertiesHide 2 properties
  - operatorenumRequired
    How filters or groups are combined.
    Show 2 enum valuesHide 2 enum values
    AND
    OR
  - filterslist of objectsRequired
    The filter rows in this group.
    Show 4 propertiesHide 4 properties
    categorystringRequired
    The property a filter row matches on (e.g. "Name", "User Id", "Model", "Metadata"). The set of valid values depends on the line's dataModel.
    conditionenumRequired
    The comparison a filter row applies. Valid conditions depend on the category.
    Show 18 enum valuesHide 18 enum values
    Is
    Is not
    Is equal to
    Does not equal
    Is less than
    Is equal or less than
    Is greater than
    Is equal or greater than
    Has
    Has not
    Contains
    Contains only
    Does not contain
    Has increased by more than
    Has increased by less than
    Has decreased by more than
    Has decreased by less than
    Has changed from
    valuestring | number | list of stringsRequired
    The value to match against.
    Show 3 variantsHide 3 variants
    string
    OR
    number
    OR
    list of strings
    keystring
    The property key. Auto-populated from category when omitted; required for Metadata, Metric, and Classifier filters.
threadTimelimitinteger
For THREAD rules, the seconds of inactivity to wait before evaluating a thread, so an in-progress conversation is not scored halfway. Defaults to 300.
overwriteEvalsboolean
Re-evaluate items that already have results for this metric collection instead of skipping them. Defaults to false.
namestringRequired
A name for the rule, unique within the project.
dataModelenumRequired
What kind of item to evaluate.
Show 3 enum valuesHide 3 enum values
- TRACE
- SPAN
- THREAD
metricCollectionstringRequired
The name of the metric collection to run. Must be multi-turn for THREAD rules and single-turn for TRACE and SPAN rules.

Response

successboolean
Indicates if the evaluation rule was successfully created.
dataobject
Show 1 propertyHide 1 property
- idstring
  The unique identifier of the created evaluation rule.
linkstring
A link to the workflows page.

Built byConfident AI