MCP Tools

This tool registers an LLM application endpoint as an AI connection in the Confident AI project, so Confident AI can call it for red-teaming and platform evaluations. Supply the whole configuration in one call.

On create, Confident AI calls the endpoint once to work out whether the connection is usable. Only the new id comes back — read active with get_ai_connection, or call ping_ai_connection to see why it failed.

AI connection guide:

• An AI connection is your LLM app registered as a callable endpoint, so Confident AI can drive it directly — it's what run_rt_framework runs against, and what the platform's dataset evaluations and experiments use.
• To become usable (active true) a connection needs three things: a reachable endpoint (https://, or wss:// when responseMode is WEBSOCKET), a payload body template, and a way to read the answer out of the response.
• Reading the answer: either actualOutputKeyPath — a path into the JSON response like ["choices", 0, "message", "content"] — or actualOutputTransformerId from list_transformers when a path isn't enough. One or the other per output; sending both is rejected.
• active is computed by Confident AI actually calling your endpoint. It is not writable, and any change to how the endpoint is called or parsed re-runs that check — so create and update will hit your service and can take a few seconds.
• Omit anything you are not changing — that is how a value is left alone. Pass null to clear it.
• Any field you do send is a full replacement, so a single entry or sub-field cannot be changed in isolation: headers, queryParams, authentication and cloudProvider must be resent whole, including the parts you are keeping. Read the connection first with get_ai_connection to see what is currently stored.
• asyncResponse requires responseMode HTTP_RESPONSE.
• prompts attaches managed prompts to the payload, keyed by payload key. Each reference names exactly one of version, label, branch, or hash; label and branch refs are live — every run re-resolves them to the label's current version / the branch's current head commit, while version and hash pin a snapshot. Full replacement like headers: resend every entry you want to keep.
• AI connections require the Starter plan or above.

Recipes:

• Plain JSON endpoint: endpoint "https://api.example.com/chat", payload {"query": "{{input}}"}, headers [{key: "Authorization", value: "Bearer ..."}, {key: "Content-Type", value: "application/json"}], actualOutputKeyPath ["choices", 0, "message", "content"].
• Server-sent events: responseMode "SSE_STREAMING", actualOutputEvent the event name carrying tokens, actualOutputAccumulate true to concatenate them.
• RAG app, also capturing context: add retrievalContextKeyPath ["retrieved_documents"] alongside the output path.
• Response shape a path can't express: actualOutputTransformerId "<id from list_transformers>" instead of actualOutputKeyPath.
• Rotate a credential: update with headers [{key: "Authorization", value: "Bearer NEW-TOKEN"}] — every other header must be included to be kept.
• Managed prompt that follows the production label: prompts {"system": {alias: "system-prompt", label: "production"}} — moving the label later updates what runs, with no connection edit.

Calls POST /v2/ai-connections.

This tool permanently deletes an AI connection from the Confident AI project. Past test runs and scheduled tasks that used it are kept, but they stop pointing at it, and anything scheduled against it will no longer run. This cannot be undone.

Calls DELETE /v2/ai-connections/{aiConnectionId}.

This tool fetches one AI connection from the Confident AI project with its full configuration. Use it to inspect the current state before updating, since every field update_ai_connection accepts is a full replacement.

Calls GET /v2/ai-connections/{aiConnectionId}.

This tool lists the AI connections configured in the Confident AI project — the registered LLM application endpoints Confident AI can call directly. Use it to discover the aiConnectionId values accepted by run_rt_framework, and to see at a glance which connections are usable.

Calls GET /v2/ai-connections.

This tool calls an AI connection's endpoint once with a sample test case and reports whether Confident AI could reach it and read the answer out of the response. Use it to check a connection is usable before running a simulation or red-teaming, and to debug one that came back inactive.

This reaches out to your own service and can take a few seconds. The verdict replaces the connection's stored active.

A connection that fails the ping is still a successful call — read active rather than treating the result as an error.

To fix a failing connection: error gives the category, statusCode and response give the cause (a refused connection, a timeout, or the endpoint's own error body all report 'Invalid Endpoint'), and response also shows the real response shape when a key path pulled out the wrong thing. Correct the configuration with update_ai_connection and ping again.

Calls POST /v2/ai-connections/{aiConnectionId}/ping.

This tool updates an AI connection in the Confident AI project. Only the fields you send change — omit a field to leave it alone, pass null to clear it. headers, queryParams and prompts are the exception: they replace the whole collection, so fetch the current state with get_ai_connection and resend every entry the connection should keep.

Changing how the endpoint is called or parsed makes Confident AI call it again to recompute active.

AI connection guide:

• An AI connection is your LLM app registered as a callable endpoint, so Confident AI can drive it directly — it's what run_rt_framework runs against, and what the platform's dataset evaluations and experiments use.
• To become usable (active true) a connection needs three things: a reachable endpoint (https://, or wss:// when responseMode is WEBSOCKET), a payload body template, and a way to read the answer out of the response.
• Reading the answer: either actualOutputKeyPath — a path into the JSON response like ["choices", 0, "message", "content"] — or actualOutputTransformerId from list_transformers when a path isn't enough. One or the other per output; sending both is rejected.
• active is computed by Confident AI actually calling your endpoint. It is not writable, and any change to how the endpoint is called or parsed re-runs that check — so create and update will hit your service and can take a few seconds.
• Omit anything you are not changing — that is how a value is left alone. Pass null to clear it.
• Any field you do send is a full replacement, so a single entry or sub-field cannot be changed in isolation: headers, queryParams, authentication and cloudProvider must be resent whole, including the parts you are keeping. Read the connection first with get_ai_connection to see what is currently stored.
• asyncResponse requires responseMode HTTP_RESPONSE.
• prompts attaches managed prompts to the payload, keyed by payload key. Each reference names exactly one of version, label, branch, or hash; label and branch refs are live — every run re-resolves them to the label's current version / the branch's current head commit, while version and hash pin a snapshot. Full replacement like headers: resend every entry you want to keep.
• AI connections require the Starter plan or above.

Recipes:

• Plain JSON endpoint: endpoint "https://api.example.com/chat", payload {"query": "{{input}}"}, headers [{key: "Authorization", value: "Bearer ..."}, {key: "Content-Type", value: "application/json"}], actualOutputKeyPath ["choices", 0, "message", "content"].
• Server-sent events: responseMode "SSE_STREAMING", actualOutputEvent the event name carrying tokens, actualOutputAccumulate true to concatenate them.
• RAG app, also capturing context: add retrievalContextKeyPath ["retrieved_documents"] alongside the output path.
• Response shape a path can't express: actualOutputTransformerId "<id from list_transformers>" instead of actualOutputKeyPath.
• Rotate a credential: update with headers [{key: "Authorization", value: "Bearer NEW-TOKEN"}] — every other header must be included to be kept.
• Managed prompt that follows the production label: prompts {"system": {alias: "system-prompt", label: "production"}} — moving the label later updates what runs, with no connection edit.

Calls PUT /v2/ai-connections/{aiConnectionId}.

This tool adds items to an annotation queue in the Confident AI project by reference. Provide at least one non-empty list matching the queue's type; items already in the queue are skipped.

Calls POST /v2/annotation-queues/{annotationQueueId}/items.

This tool submits annotations (and/or custom form-field responses) for a single annotation queue item in the Confident AI project. Provide at least one annotation or form response. The item is marked completed unless markAsCompleted is set to false.

Calls POST /v2/annotation-queues/{annotationQueueId}/items/{queueItemId}/annotate.

This tool annotates multiple items of an annotation queue (addressed by the queue's id) in one call. Items are processed best-effort: a failing item is reported in the results without aborting the rest. Batch-level annotatorEmail and markAsCompleted are defaults that per-item values override.

Calls POST /v2/annotation-queues/{annotationQueueId}/batch-annotate.

This tool creates an annotation queue in the Confident AI project. A queue holds items of one type (TRACE, SPAN, THREAD, GOLDEN, or TEST_RUN) for human reviewers to annotate. Queue names are unique within a project.

What annotators are asked depends on formId. With a form attached, the queue presents exactly that form's questions. Without one, it falls back to the project's annotation settings — the built-in Default criterion plus any custom annotation options — and form responses are rejected.

Calls POST /v2/annotation-queues.

This tool creates an ingestion task on an annotation queue: a standing rule that routes matching production traces, spans, or threads into the queue for humans to review. The task starts running as soon as it is created with enabled true.

Task names are unique within a queue. Only TRACE, SPAN and THREAD tasks are supported — GOLDEN queues are filled from a dataset rather than by ingestion.

Note: queue ingestion tasks are available on the Starter plan and above.

Calls POST /v2/annotation-queues/{annotationQueueId}/queue-ingestion-tasks.

This tool permanently deletes an annotation queue (and all of its queue items) from the Confident AI project. Annotations already submitted are not deleted.

Calls DELETE /v2/annotation-queues/{annotationQueueId}.

This tool permanently deletes an ingestion task from an annotation queue and unschedules its routing job. Items the task already queued stay in the queue.

Note: queue ingestion tasks are available on the Starter plan and above.

Calls DELETE /v2/annotation-queues/{annotationQueueId}/queue-ingestion-tasks/{queueIngestionTaskId}.

This tool fetches a single annotation queue from the Confident AI project, including completion statistics and the per-annotator assignment breakdown.

Calls GET /v2/annotation-queues/{annotationQueueId}.

This tool fetches a single ingestion task on an annotation queue, with its full configuration and the reviewers items are assigned to.

Note: queue ingestion tasks are available on the Starter plan and above.

Calls GET /v2/annotation-queues/{annotationQueueId}/queue-ingestion-tasks/{queueIngestionTaskId}.

This tool lists the items in an annotation queue in the Confident AI project, oldest first.

Calls GET /v2/annotation-queues/{annotationQueueId}/items.

This tool lists annotation queues in the Confident AI project, with completion statistics for each queue.

Calls GET /v2/annotation-queues.

This tool lists the ingestion tasks on an annotation queue — the rules that continuously route matching production traces, spans, or threads into that queue for human review. Use it to discover the queueIngestionTaskId values the get, update, and delete tools accept.

Note: queue ingestion tasks are available on the Starter plan and above.

Calls GET /v2/annotation-queues/{annotationQueueId}/queue-ingestion-tasks.

This tool renames an annotation queue, or attaches and detaches its annotation form. At least one of name or formId must be sent; only what you send is changed.

Attaching a form switches the queue to asking that form's questions. Detaching it (form_id null) falls back to the project's annotation settings. Either way, annotations already submitted to the queue are untouched.

Calls PUT /v2/annotation-queues/{annotationQueueId}.

This tool updates an ingestion task on an annotation queue. Only the fields you send are changed; anything omitted keeps its current value. Sending an explicit null for maxItems removes the cap. Toggling enabled schedules or unschedules the routing job.

Changing assignmentStrategy requires sending reviewerEmails in the same call, otherwise reviewers from the previous strategy would be left stranded. Re-sending the same reviewers on a ROUND_ROBIN or RANDOM pool preserves their existing assignment counts, so load balancing carries over the edit.

Note: queue ingestion tasks are available on the Starter plan and above.

Calls PUT /v2/annotation-queues/{annotationQueueId}/queue-ingestion-tasks/{queueIngestionTaskId}.

This tool creates a new annotation (human feedback) on one trace, span, or thread.

The body goes in annotation, and which target you are annotating decides its shape — send exactly one of the three:

• a trace: traceUuid, plus expectedOutput for the output it should have produced.
• a span: spanUuid, plus expectedOutput.
• a thread: threadId, plus expectedOutcome for the outcome the conversation should have reached. A thread takes no expectedOutput.

Every target then takes the feedback itself: fieldType for the kind of rating and value for the rating, whose shape fieldType decides — see the value field's own description for which shape each kind expects.

Calls POST /v2/annotations.

This tool retrieves the complete details of a specific annotation using its unique ID. Use this when you need the full context, explanation, and expected outcome of a single piece of feedback.

Calls GET /v2/annotations/{annotationId}.

This tool retrieves a paginated list of annotations (human feedback) from Confident AI. Use this to find user feedback, ratings, and corrections applied to specific traces, spans, or threads. Narrow it with traceUuid, spanUuid or threadId for one target, fieldType for one kind of rating, and start / end for a time range.

Calls GET /v2/annotations.

This tool updates an existing annotation. Every field is optional and whatever you omit is left as it was, so send only what you are changing: value and fieldType for the rating itself, or explanation for the reasoning behind it.

Which expected field applies depends on what the annotation is attached to, and the tool cannot tell you which from its shape — use expectedOutcome for an annotation on a thread and expectedOutput for one on a trace or span.

Calls PUT /v2/annotations/{annotationId}.

This tool fetches one attack method from the Confident AI project, including its parameters — the schema of what the method accepts, with each parameter's type, whether it is required, its allowed options when it is an enum, and a default that carries this project's configured value when one is set. Call this before update_attack_method to learn the parameter names and the current values.

Calls GET /v2/attack-methods/{attackMethodId}.

This tool lists the red-teaming attack methods available to the Confident AI project. An attack method is the technique used to deliver a probe for a vulnerability — encoding it, roleplaying around it, or escalating over several turns. They come from the deepteam catalog and cannot be created or deleted; what a project owns is each method's configuration. Use this to discover the ids and names to reference when building risk categories.

Note: red teaming through the API is available on the Enterprise plan only.

Calls GET /v2/attack-methods.

This tool discards the Confident AI project's configuration of an attack method, so it falls back to the deepteam catalog's defaults everywhere the project uses it. The attack method itself is not deleted — it cannot be, since the catalog owns it — and it stays selectable in every risk category that already references it.

Resetting a method that was never configured succeeds and changes nothing.

Calls DELETE /v2/attack-methods/{attackMethodId}.

This tool sets how an attack method behaves for the Confident AI project — every risk assessment the project runs uses these values wherever the method appears. Other projects are unaffected. The attack method itself is not created or changed; only the project's configuration of it is.

parameters replaces the stored configuration wholesale, so fetch the method with get_attack_method first and resend every value it should keep. Each value is checked against the method's own schema: an unknown parameter name, a value of the wrong type, or a value outside an enum's options is rejected and nothing is stored. A method whose required parameters are not all set cannot be attached to a risk category, so send them here rather than leaving them blank.

Note: red teaming through the API is available on the Enterprise plan only.

Calls PUT /v2/attack-methods/{attackMethodId}.

This tool creates a classifier: a rule that tags incoming traces or threads with labels. Classifiers only run on TRACE and THREAD items, never spans, and a name is unique per data model within the project.

Passing a preset seeds the classifier with a description, a generation config, and a starting set of labels — SENTIMENT (positive/negative/neutral), TOPICS, USE_CASES, or ISSUES. Use CUSTOM (or omit it) to start empty. Any field you pass explicitly overrides what the preset would have set. SENTIMENT arrives with its labels ready; TOPICS, USE_CASES and ISSUES ship with none, and expect you to call generate_classifier_labels next to populate them from real traffic.

Note: classifiers are available on the Starter plan and above.

Calls POST /v2/classifiers.

This tool adds a label to a classifier. The description is what the classifying model matches against, so write it as a clear statement of when the label applies rather than a bare synonym of the name. Label names are unique within a classifier.

Note: classifiers are available on the Starter plan and above.

Calls POST /v2/classifiers/{classifierId}/labels.

This tool permanently deletes a classifier and all of its labels. Classifications already applied to traces or threads are not removed.

Note: classifiers are available on the Starter plan and above.

Calls DELETE /v2/classifiers/{classifierId}.

This tool permanently deletes a label from a classifier.

Note: classifiers are available on the Starter plan and above.

Calls DELETE /v2/classifiers/{classifierId}/labels/{labelId}.

This tool discovers labels for a classifier from the project's real traffic: it samples recent traces (or threads), clusters them using the classifier's autoGenerationConfig, and writes the themes it finds back as labels with status RECOMMENDED for a human to review and promote.

Side effects worth knowing before calling:

• It reads production traces and runs an LLM over the sample, so it costs usage.
• It first deletes every existing RECOMMENDED label on the classifier. Labels already promoted to ACTIVE are kept, and are passed to the generator so it does not propose them again.
• It runs asynchronously. A started true response means the run was dispatched, not that labels exist yet — poll list_classifier_labels for the RECOMMENDED rows it produces.

The classifier's autoGenerationConfig must already have summaryPrompt and nClusters set, or the call fails. A started false response is not an error: it means the project had too little traffic to sample, or sampling was briefly unavailable — the message says which.

Note: classifiers are available on the Starter plan and above.

Calls POST /v2/classifiers/{classifierId}/generate.

This tool fetches a single classifier with its full configuration and all of its labels.

Note: classifiers are available on the Starter plan and above.

Calls GET /v2/classifiers/{classifierId}.

This tool fetches a single label on a classifier.

Note: classifiers are available on the Starter plan and above.

Calls GET /v2/classifiers/{classifierId}/labels/{labelId}.

This tool lists a classifier's labels, alphabetically. This is also how you read the results of generate_classifier_labels — generated suggestions arrive with status RECOMMENDED, and become usable once promoted to ACTIVE.

Note: classifiers are available on the Starter plan and above.

Calls GET /v2/classifiers/{classifierId}/labels.

This tool lists the classifiers in the Confident AI project — the rules that tag incoming traces or threads with labels, so traffic can be grouped by sentiment, use case, or issue. Use it to discover the classifierId values the other classifier tools accept.

Note: classifiers are available on the Starter plan and above.

Calls GET /v2/classifiers.

This tool updates a classifier. Only the fields you send are changed; anything omitted keeps its current value. Sending an explicit null clears a field.

A classifier's dataModel cannot be changed after creation, and a preset can only be applied when creating one.

Note: classifiers are available on the Starter plan and above.

Calls PUT /v2/classifiers/{classifierId}.

This tool updates a label on a classifier. Only the fields you send are changed. Promoting a generated suggestion is an update to status ACTIVE.

Note: classifiers are available on the Starter plan and above.

Calls PUT /v2/classifiers/{classifierId}/labels/{labelId}.

This tool creates a dashboard in the Confident AI project, optionally with its widgets in one call (widgets are auto-laid-out in order). Give the dashboard and every widget a clear name and description so the dashboard is self-explanatory.

Use query_ad_hoc_widget first to validate a widget definition and check it returns meaningful data before committing it to a dashboard.

Widget composition guide:

• A widget = chart type + mode + one or more lines (each line is one data series: a data_model + an aggregation, optionally filtered).
• mode TIME_SERIES: values over time. Use LINE/AREA for trends, BAR for volumes, BIG_NUMBER (with bucket_mode RANGE) for a single headline stat.
• mode DIMENSION_SERIES: values grouped by dimension (required), e.g. per model or per trace name. Use BAR/STACKED_BAR/GROUPED_BAR/TABLE and topK to bound how many groups appear.
• Match unit to the aggregation: USD for costs, PERCENT for rates, SECONDS/MILLISECONDS for latency, SCORE for scores, COUNT otherwise.

Recipes:

• LLM cost over time: type LINE, mode TIME_SERIES, unit USD, lines: [{name: 'Cost', data_model: 'LLM_SPAN', aggregation: 'TOTAL_COST'}]
• Top 10 models by usage: type BAR, mode DIMENSION_SERIES, dimension model, unit COUNT, top_k: {limit: 10}, lines: [{name: 'Calls', data_model: 'LLM_SPAN', aggregation: 'COUNT'}]
• Trace error rate: type LINE, mode TIME_SERIES, unit PERCENT, lines: [{name: 'Error rate', data_model: 'TRACE', aggregation: 'ERROR_RATE'}]
• Eval pass rate headline: type BIG_NUMBER, bucket_mode RANGE, unit PERCENT, lines: [{name: 'Pass rate', data_model: 'METRIC_DATA', aggregation: 'PASS_RATE'}]
• p90 latency, one line per trace name: type LINE, mode TIME_SERIES, unit MILLISECONDS, multiple lines each with data_model TRACE, aggregation P90_LATENCY, and a filters set on 'Trace Name'.
• Daily active users: type BAR, mode TIME_SERIES, unit COUNT, lines: [{name: 'Users', data_model: 'TRACE', aggregation: 'UNIQUE_END_USERS'}]

Calls POST /v2/dashboards.

This tool adds a widget to an existing dashboard in the Confident AI project. The widget is auto-placed on the next free row unless layout is provided. Use query_ad_hoc_widget first to validate the definition returns meaningful data.

Widget composition guide:

• A widget = chart type + mode + one or more lines (each line is one data series: a data_model + an aggregation, optionally filtered).
• mode TIME_SERIES: values over time. Use LINE/AREA for trends, BAR for volumes, BIG_NUMBER (with bucket_mode RANGE) for a single headline stat.
• mode DIMENSION_SERIES: values grouped by dimension (required), e.g. per model or per trace name. Use BAR/STACKED_BAR/GROUPED_BAR/TABLE and topK to bound how many groups appear.
• Match unit to the aggregation: USD for costs, PERCENT for rates, SECONDS/MILLISECONDS for latency, SCORE for scores, COUNT otherwise.

Recipes:

• LLM cost over time: type LINE, mode TIME_SERIES, unit USD, lines: [{name: 'Cost', data_model: 'LLM_SPAN', aggregation: 'TOTAL_COST'}]
• Top 10 models by usage: type BAR, mode DIMENSION_SERIES, dimension model, unit COUNT, top_k: {limit: 10}, lines: [{name: 'Calls', data_model: 'LLM_SPAN', aggregation: 'COUNT'}]
• Trace error rate: type LINE, mode TIME_SERIES, unit PERCENT, lines: [{name: 'Error rate', data_model: 'TRACE', aggregation: 'ERROR_RATE'}]
• Eval pass rate headline: type BIG_NUMBER, bucket_mode RANGE, unit PERCENT, lines: [{name: 'Pass rate', data_model: 'METRIC_DATA', aggregation: 'PASS_RATE'}]
• p90 latency, one line per trace name: type LINE, mode TIME_SERIES, unit MILLISECONDS, multiple lines each with data_model TRACE, aggregation P90_LATENCY, and a filters set on 'Trace Name'.
• Daily active users: type BAR, mode TIME_SERIES, unit COUNT, lines: [{name: 'Users', data_model: 'TRACE', aggregation: 'UNIQUE_END_USERS'}]

Calls POST /v2/dashboards/{dashboardId}/widgets.

This tool permanently deletes a dashboard from the Confident AI project.

Calls DELETE /v2/dashboards/{dashboardId}.

This tool removes a widget from a dashboard in the Confident AI project.

Calls DELETE /v2/dashboards/{dashboardId}/widgets/{widgetId}.

This tool fetches a dashboard from the Confident AI project, including all of its widgets (with their ids, definitions, lines, and layout). Use it to inspect a dashboard before updating widgets or querying data.

Calls GET /v2/dashboards/{dashboardId}.

This tool lists all dashboards in the Confident AI project, newest first, with each dashboard's widget count.

Calls GET /v2/dashboards.

This tool executes a dashboard's widgets and returns their data — what a user would see rendered on the dashboard. Widgets are queried independently; a failing widget reports an error entry without failing the rest.

Calls POST /v2/dashboards/{dashboardId}/query.

This tool executes a single dashboard widget and returns its data.

Calls POST /v2/dashboards/{dashboardId}/widgets/{widgetId}/query.

This tool updates a dashboard's metadata (name, description, privacy) in the Confident AI project. At least one field must be provided. Widgets are managed with create_widget / update_widget / delete_widget.

Calls PUT /v2/dashboards/{dashboardId}.

This tool replaces a widget's definition on a dashboard in the Confident AI project. The provided definition is a full replacement: when lines is included, the widget's existing lines are deleted and recreated, so include every line the widget should keep (fetch the current state with get_dashboard).

Widget composition guide:

• A widget = chart type + mode + one or more lines (each line is one data series: a data_model + an aggregation, optionally filtered).
• mode TIME_SERIES: values over time. Use LINE/AREA for trends, BAR for volumes, BIG_NUMBER (with bucket_mode RANGE) for a single headline stat.
• mode DIMENSION_SERIES: values grouped by dimension (required), e.g. per model or per trace name. Use BAR/STACKED_BAR/GROUPED_BAR/TABLE and topK to bound how many groups appear.
• Match unit to the aggregation: USD for costs, PERCENT for rates, SECONDS/MILLISECONDS for latency, SCORE for scores, COUNT otherwise.

Recipes:

• LLM cost over time: type LINE, mode TIME_SERIES, unit USD, lines: [{name: 'Cost', data_model: 'LLM_SPAN', aggregation: 'TOTAL_COST'}]
• Top 10 models by usage: type BAR, mode DIMENSION_SERIES, dimension model, unit COUNT, top_k: {limit: 10}, lines: [{name: 'Calls', data_model: 'LLM_SPAN', aggregation: 'COUNT'}]
• Trace error rate: type LINE, mode TIME_SERIES, unit PERCENT, lines: [{name: 'Error rate', data_model: 'TRACE', aggregation: 'ERROR_RATE'}]
• Eval pass rate headline: type BIG_NUMBER, bucket_mode RANGE, unit PERCENT, lines: [{name: 'Pass rate', data_model: 'METRIC_DATA', aggregation: 'PASS_RATE'}]
• p90 latency, one line per trace name: type LINE, mode TIME_SERIES, unit MILLISECONDS, multiple lines each with data_model TRACE, aggregation P90_LATENCY, and a filters set on 'Trace Name'.
• Daily active users: type BAR, mode TIME_SERIES, unit COUNT, lines: [{name: 'Users', data_model: 'TRACE', aggregation: 'UNIQUE_END_USERS'}]

Calls PUT /v2/dashboards/{dashboardId}/widgets/{widgetId}.

This tool creates an ingestion task on an evaluation dataset: a standing rule that harvests matching production traces, spans, or threads into the dataset as goldens. The task starts running as soon as it is created with enabled true.

dataModel must match the dataset's type — a multi-turn dataset only accepts THREAD tasks, and a single-turn dataset only accepts TRACE or SPAN. Task names are unique within a dataset.

Note: dataset ingestion tasks are available on the Starter plan and above.

Calls POST /v2/datasets/{datasetId}/dataset-ingestion-tasks.

Snapshot the current state of an evaluation dataset as a new version. Versions are immutable point-in-time copies of the dataset's goldens. If the dataset has no prior versions, all existing unversioned goldens are backfilled onto the new version.

Calls POST /v2/datasets/{datasetId}/versions.

This tool adds a single golden to an existing evaluation dataset in Confident AI. For single-turn datasets provide input/output fields; for multi-turn (conversational) datasets provide scenario/turns/expectedOutcome.

Calls POST /v2/datasets/{datasetId}/goldens.

This tool permanently deletes an evaluation dataset (and all of its goldens and versions) from Confident AI.

Calls DELETE /v2/datasets/{datasetId}.

This tool permanently deletes an ingestion task from an evaluation dataset and unschedules its harvesting job. Goldens the task already created stay in the dataset.

Note: dataset ingestion tasks are available on the Starter plan and above.

Calls DELETE /v2/datasets/{datasetId}/dataset-ingestion-tasks/{datasetIngestionTaskId}.

This tool permanently deletes a single golden from an evaluation dataset in Confident AI.

Calls DELETE /v2/datasets/{datasetId}/goldens/{goldenId}.

This tool fetches a single ingestion task on an evaluation dataset, with its full configuration and how many goldens it has created.

Note: dataset ingestion tasks are available on the Starter plan and above.

Calls GET /v2/datasets/{datasetId}/dataset-ingestion-tasks/{datasetIngestionTaskId}.

List all versions of an evaluation dataset, newest first.

Calls GET /v2/datasets/{datasetId}/versions.

This tool fetches a single golden from an evaluation dataset in Confident AI, including all of its fields (inputs/outputs or turns, context, metadata, custom columns, and tags).

Calls GET /v2/datasets/{datasetId}/goldens/{goldenId}.

This tool lists the ingestion tasks on an evaluation dataset — the rules that continuously harvest production traces, spans, or threads into that dataset as goldens. Use it to discover the datasetIngestionTaskId values the get, update, and delete tools accept.

Note: dataset ingestion tasks are available on the Starter plan and above.

Calls GET /v2/datasets/{datasetId}/dataset-ingestion-tasks.

This tool can be used to retrieve a list of all datasets available in Confident AI.

Calls GET /v2/datasets.

This tool fetches one evaluation dataset from Confident AI by its id, with all of its goldens, oldest first. Ids come from list_datasets. Pass version to pull a specific version instead of the latest, and finalized false to pull the goldens still awaiting review rather than the finalized ones.

Calls GET /v2/datasets/{datasetId}.

This tool can be used to push an entire evaluation dataset to Confident AI. This will either create a new dataset or append goldens to an existing dataset.

If the target dataset already has versions, goldens are appended to the latest version unless version is provided.

Calls POST /v2/datasets.

This tool queues goldens onto an existing evaluation dataset in Confident AI for later annotation, addressing the dataset by its id — the goldens are added unfinalized whatever each one's own finalized says. Every golden in one call must be of the same kind and must match the dataset's multiTurn.

Calls POST /v2/datasets/{datasetId}/queue.

This tool starts an evaluation of a dataset's finalized goldens against a metric collection — the same run the Confident AI platform starts from a dataset page. The run executes asynchronously; the response contains the id of the created test run and a link to watch it. The dataset must have at least one finalized golden.

How the outputs being evaluated are produced depends on the target given:

• neither aiConnectionId nor promptAlias: the goldens' stored actual outputs are evaluated as they are, and nothing is generated.
• aiConnectionId: Confident AI calls that AI connection — i.e. it reaches out to the customer's own service — to generate an output per golden.
• promptAlias: outputs are generated from that prompt commit, using the model and settings stored on the commit. The project must hold credentials for that model's provider. The two targets are mutually exclusive. Generating outputs and running metrics both consume LLM usage.

Calls POST /v2/datasets/{datasetId}/run.

This tool updates an ingestion task on an evaluation dataset. Only the fields you send are changed; anything omitted keeps its current value. Sending an explicit null clears a field — that is how you detach a transformer or remove a maxGoldens cap.

Toggling enabled schedules or unschedules the harvesting job. Changing dataModel still has to match the dataset's type.

Note: dataset ingestion tasks are available on the Starter plan and above.

Calls PUT /v2/datasets/{datasetId}/dataset-ingestion-tasks/{datasetIngestionTaskId}.

This tool updates a single golden in an evaluation dataset in Confident AI. The golden's fields are replaced with the provided values (fields omitted from golden are cleared), so include every field the golden should keep.

Calls PUT /v2/datasets/{datasetId}/goldens/{goldenId}.

This tool can be used to trigger a remote evaluation for a specific span within a trace on Confident AI. Use this when you want to evaluate a granular step of your LLM application (like a retrieval step or a specific tool call) against a predefined metric collection.

Calls POST /v2/evaluate/spans/{spanUuid}.

This tool can be used to trigger a remote evaluation for an entire conversation thread on Confident AI. Use this to evaluate multi-turn interactions and ensure consistency and quality across a full session.

Calls POST /v2/evaluate/threads/{threadId}.

This tool can be used to trigger a remote evaluation for an entire trace on Confident AI. A trace represents a single end-to-end request in your LLM application. Use this to evaluate the overall performance of a specific execution.

Calls POST /v2/evaluate/traces/{traceUuid}.

This tool triggers an online evaluation for a batch of raw, local test cases on Confident AI.

Calls POST /v2/evaluate.

This tool creates an evaluation rule: a standing rule that runs a metric collection against matching production traces, spans, or threads as they arrive. The rule starts evaluating immediately unless enabled is false. Running metrics consumes LLM usage.

The metric collection's turn type must match the rule: THREAD rules require a multi-turn collection, TRACE and SPAN rules require a single-turn one. Rule names are unique per project, and only one enabled THREAD rule may target a given metric collection — a second one would double-write metric results.

Note: evaluation rules are available on the Starter plan and above.

Calls POST /v2/evaluation-rules.

This tool permanently deletes an evaluation rule. Metric results it already produced are kept; only the rule stops running.

Note: evaluation rules are available on the Starter plan and above.

Calls DELETE /v2/evaluation-rules/{evaluationRuleId}.

This tool fetches a single evaluation rule with its full configuration and the metric collection it runs.

Note: evaluation rules are available on the Starter plan and above.

Calls GET /v2/evaluation-rules/{evaluationRuleId}.

This tool lists the evaluation rules in the Confident AI project — the standing rules that automatically run a metric collection against incoming traces, spans, or threads. Use it to discover the evaluationRuleId values the get, update, and delete tools accept.

Note: evaluation rules are available on the Starter plan and above.

Calls GET /v2/evaluation-rules.

This tool updates an evaluation rule. Only the fields you send are changed; anything omitted keeps its current value. Sending an explicit null clears a field — that is how you remove filters or a span type.

Every constraint is re-checked against the rule's resulting state, not just what you sent: switching a rule to THREAD still requires its metric collection to be multi-turn, and enabling a THREAD rule still conflicts with another enabled one on the same collection.

Note: evaluation rules are available on the Starter plan and above.

Calls PUT /v2/evaluation-rules/{evaluationRuleId}.

This tool registers an S3 bucket or a Snowflake warehouse as an export destination, so the Confident AI project can send scheduled exports to it. Creating a destination does not export anything on its own — an export schedule has to point at it.

The credentials are stored as given and are not verified here, so a wrong key or a missing permission surfaces later, when a schedule runs. Only the new id comes back.

Export destination guide:

• An export destination is where the project sends scheduled exports. It does not decide what gets exported or when — an export schedule does that and points at a destination.
• type is S3 (default) or SNOWFLAKE, and can't be changed after creation.
• S3: bucket, region, accessKeyId and secretAccessKey are all required to create one, and the credentials need write access to the bucket. Each run uploads one gzipped file.
• SNOWFLAKE: send snowflakeConfig with account, username, role, warehouse, database, schema and privateKey (PEM, key-pair auth; plus privateKeyPassphrase for an encrypted key). The role needs USAGE on the warehouse, database and schema, and CREATE TABLE on the schema. Each run upserts rows into CONFIDENT_TRACES or CONFIDENT_ANNOTATIONS, which Confident AI creates. Only TRACES and ANNOTATIONS schedules can use a Snowflake destination. S3 fields are rejected for it.
• pathPrefix is an optional S3 key prefix inside the bucket. It is normalized on write — a leading slash is stripped and a trailing one added — so exports and '/exports/' are stored the same way. It is the one field you clear by passing null.
• Credentials are secrets, so the read tools mask them (S3 keys, and Snowflake privateKey / privateKeyPassphrase). On update, resending a masked value means 'keep what is stored', which is what lets you change the bucket or rotate one key without knowing the other. For Snowflake, send only the snowflakeConfig fields you want to change. On create a masked value is rejected, because there is nothing stored to keep.
• Every other field is left alone by omitting it.
• enabled false pauses uploads without deleting the destination.
• A project may hold at most 3 export destinations.

Recipes:

• Add a destination: create with name, bucket "my-exports", region "us-east-1", accessKeyId and secretAccessKey.
• Rotate the secret: update with secretAccessKey only — the access key id, bucket and region are untouched because they are omitted.
• Move to a different bucket, same credentials: update with bucket and region only.
• Nest exports under a folder: update with pathPrefix "confident/exports".
• Add a Snowflake destination: create with name, type "SNOWFLAKE" and snowflakeConfig {account "myorg-myaccount", username "CONFIDENT_EXPORTER", role "CONFIDENT_EXPORT_ROLE", warehouse "CONFIDENT_WH", database "ANALYTICS", schema "CONFIDENT", privateKey "-----BEGIN PRIVATE KEY-----..."}.
• Point a Snowflake destination at another schema: update with snowflakeConfig {schema "EXPORTS"} only.
• Stop uploads but keep the config: update with enabled false.

Calls POST /v2/export-destinations.

This tool permanently deletes an export destination from the Confident AI project and destroys the stored credentials, so restoring it means re-entering them.

It also disables every export schedule currently uploading to this destination, which is deliberate: a schedule left enabled would quietly stop delivering to the destination. Those schedules are not deleted — re-enable them once they point at another destination. Files already uploaded to a bucket, and rows already loaded into Snowflake, are untouched.

To stop uploads reversibly, call update_export_destination with enabled false instead.

Calls DELETE /v2/export-destinations/{exportDestinationId}.

This tool fetches one export destination from the Confident AI project with its full configuration. Use it to check the bucket, region and path prefix a schedule will upload to, or the Snowflake account, database and schema it loads into.

accessKeyId, secretAccessKey and the Snowflake privateKey / privateKeyPassphrase come back masked. Sending a masked value back through update_export_destination keeps the stored credential, so a read-modify-write round trip does not require knowing it.

Calls GET /v2/export-destinations/{exportDestinationId}.

This tool lists the export destinations configured in the Confident AI project — the S3 buckets and Snowflake warehouses scheduled exports are sent to. Use it to discover the exportDestinationId values that export schedules and the other destination tools accept.

Calls GET /v2/export-destinations.

This tool updates an export destination in the Confident AI project. Only the fields you send change — omit a field to leave it alone. pathPrefix is the one field you can clear, by passing null.

Resending a masked credential keeps the one already stored, so you can change the bucket, or rotate one key, without resending the other.

Schedules already pointing at this destination pick up the change on their next run; nothing is re-uploaded.

Export destination guide:

• An export destination is where the project sends scheduled exports. It does not decide what gets exported or when — an export schedule does that and points at a destination.
• type is S3 (default) or SNOWFLAKE, and can't be changed after creation.
• S3: bucket, region, accessKeyId and secretAccessKey are all required to create one, and the credentials need write access to the bucket. Each run uploads one gzipped file.
• SNOWFLAKE: send snowflakeConfig with account, username, role, warehouse, database, schema and privateKey (PEM, key-pair auth; plus privateKeyPassphrase for an encrypted key). The role needs USAGE on the warehouse, database and schema, and CREATE TABLE on the schema. Each run upserts rows into CONFIDENT_TRACES or CONFIDENT_ANNOTATIONS, which Confident AI creates. Only TRACES and ANNOTATIONS schedules can use a Snowflake destination. S3 fields are rejected for it.
• pathPrefix is an optional S3 key prefix inside the bucket. It is normalized on write — a leading slash is stripped and a trailing one added — so exports and '/exports/' are stored the same way. It is the one field you clear by passing null.
• Credentials are secrets, so the read tools mask them (S3 keys, and Snowflake privateKey / privateKeyPassphrase). On update, resending a masked value means 'keep what is stored', which is what lets you change the bucket or rotate one key without knowing the other. For Snowflake, send only the snowflakeConfig fields you want to change. On create a masked value is rejected, because there is nothing stored to keep.
• Every other field is left alone by omitting it.
• enabled false pauses uploads without deleting the destination.
• A project may hold at most 3 export destinations.

Recipes:

• Add a destination: create with name, bucket "my-exports", region "us-east-1", accessKeyId and secretAccessKey.
• Rotate the secret: update with secretAccessKey only — the access key id, bucket and region are untouched because they are omitted.
• Move to a different bucket, same credentials: update with bucket and region only.
• Nest exports under a folder: update with pathPrefix "confident/exports".
• Add a Snowflake destination: create with name, type "SNOWFLAKE" and snowflakeConfig {account "myorg-myaccount", username "CONFIDENT_EXPORTER", role "CONFIDENT_EXPORT_ROLE", warehouse "CONFIDENT_WH", database "ANALYTICS", schema "CONFIDENT", privateKey "-----BEGIN PRIVATE KEY-----..."}.
• Point a Snowflake destination at another schema: update with snowflakeConfig {schema "EXPORTS"} only.
• Stop uploads but keep the config: update with enabled false.

Calls PUT /v2/export-destinations/{exportDestinationId}.

This tool creates an export schedule in the Confident AI project: a recurring export of traces or conversations, uploaded to an export destination. The schedule starts running as soon as it is created unless you pass enabled false.

Only the new id comes back. Read it with get_export_schedule to confirm the cadence, and again after the first window to see runCount advance.

Export schedule guide:

• An export schedule runs an export of the project's traces or conversations on a cadence and uploads the file to an export destination. It is the recurring counterpart to a one-off export.
• exportType decides what each run produces and is fixed at creation: TRACES (flat CSV of traces), TRACES_WITH_SPANS (JSONL with full spans), CONVERSATIONS (JSONL of whole threads), CONVERSATION_METRICS (CSV of each thread's metric results), ANNOTATIONS (CSV), TEST_RUNS (JSONL). AUDIT_LOGS is organization-scoped and cannot be scheduled. To export something else, create a second schedule — update_export_schedule rejects exportType.
• destinationId comes from list_export_destinations. A schedule without one still runs on time, but the file is not delivered anywhere: a scheduled run has no email recipient, unlike a manual export. Set a destination.
• Cadence: recurrence INTERVAL (the default) repeats every repeatEvery x repeatUnit; recurrence ONCE runs a single time at startAt. Each run covers the window since the previous one — a ONCE schedule covers the last 24 hours.
• Stopping: maxRuns and endAt both retire a schedule, which then disables itself. Re-enabling an already-exhausted schedule is rejected — raise the run cap or push out the end date in the same call.
• enabled false pauses a schedule without deleting it, and is the reversible way to stop it.
• Everything except exportType is editable. Omit a field to leave it alone; filters replaces the whole set.
• A project may hold at most 3 schedules per export type.

Recipes:

• Nightly trace CSV to S3: create with exportType TRACES, repeatEvery 1, repeatUnit DAY, destinationId from list_export_destinations.
• Production traffic only: add filters {"operator": "AND", "groups": [{"operator": "AND", "filters": [{"category": "Environment", "condition": "Is", "value": "production"}]}]}.
• Weekly conversation dump: exportType CONVERSATIONS, repeatEvery 1, repeatUnit WEEK.
• A single backfill-style run: recurrence ONCE with a startAt.
• Pause without losing the config: update with enabled false.
• Retire after 10 runs: update with maxRuns 10.

Calls POST /v2/export-schedules.

This tool permanently deletes an export schedule from the Confident AI project. No further exports run for it. Files already uploaded to the destination are untouched, and the destination itself is not affected.

To stop a schedule reversibly, call update_export_schedule with enabled false instead.

Calls DELETE /v2/export-schedules/{exportScheduleId}.

This tool fetches one export schedule from the Confident AI project with its cadence, filters, destination and run history. Read runCount and lastRunAt to see whether it is actually firing, and enabled to see whether it is paused or has retired itself.

Use it before updating, since filters replaces the whole set.

Calls GET /v2/export-schedules/{exportScheduleId}.

This tool lists the export schedules configured in the Confident AI project — the recurring trace and conversation exports. Use it to discover the exportScheduleId values the other schedule tools accept, and to see which schedules are currently running.

Calls GET /v2/export-schedules.

This tool updates an export schedule in the Confident AI project. Only the fields you send change — omit a field to leave it alone, and pass null to clear description, destinationId, maxRuns or endAt. filters is the exception: it replaces the whole set, so fetch the current one with get_export_schedule and resend every group the schedule should keep.

exportType cannot be changed and is rejected: create a second schedule instead. Changing the cadence re-registers the schedule, so the next run is measured from now rather than from the previous run.

Export schedule guide:

• An export schedule runs an export of the project's traces or conversations on a cadence and uploads the file to an export destination. It is the recurring counterpart to a one-off export.
• exportType decides what each run produces and is fixed at creation: TRACES (flat CSV of traces), TRACES_WITH_SPANS (JSONL with full spans), CONVERSATIONS (JSONL of whole threads), CONVERSATION_METRICS (CSV of each thread's metric results), ANNOTATIONS (CSV), TEST_RUNS (JSONL). AUDIT_LOGS is organization-scoped and cannot be scheduled. To export something else, create a second schedule — update_export_schedule rejects exportType.
• destinationId comes from list_export_destinations. A schedule without one still runs on time, but the file is not delivered anywhere: a scheduled run has no email recipient, unlike a manual export. Set a destination.
• Cadence: recurrence INTERVAL (the default) repeats every repeatEvery x repeatUnit; recurrence ONCE runs a single time at startAt. Each run covers the window since the previous one — a ONCE schedule covers the last 24 hours.
• Stopping: maxRuns and endAt both retire a schedule, which then disables itself. Re-enabling an already-exhausted schedule is rejected — raise the run cap or push out the end date in the same call.
• enabled false pauses a schedule without deleting it, and is the reversible way to stop it.
• Everything except exportType is editable. Omit a field to leave it alone; filters replaces the whole set.
• A project may hold at most 3 schedules per export type.

Recipes:

• Nightly trace CSV to S3: create with exportType TRACES, repeatEvery 1, repeatUnit DAY, destinationId from list_export_destinations.
• Production traffic only: add filters {"operator": "AND", "groups": [{"operator": "AND", "filters": [{"category": "Environment", "condition": "Is", "value": "production"}]}]}.
• Weekly conversation dump: exportType CONVERSATIONS, repeatEvery 1, repeatUnit WEEK.
• A single backfill-style run: recurrence ONCE with a startAt.
• Pause without losing the config: update with enabled false.
• Retire after 10 runs: update with maxRuns 10.

Calls PUT /v2/export-schedules/{exportScheduleId}.

This tool registers an OTLP collector as a forwarding connector, so the Confident AI project starts mirroring its traces there as they arrive. Forwarding begins as soon as the connector is created unless you pass enabled false.

Only the new id comes back. Read the connector with get_forwarding_connector to check delivery is succeeding once traces have flowed.

Forwarding guide:

• A forwarding connector continuously mirrors the project's traces to your own OTLP/HTTP collector (Datadog, Grafana, Honeycomb, an OpenTelemetry Collector) as they arrive. It is a live stream, not a scheduled export, and it copies rather than moves — traces stay in Confident AI either way.
• endpoint must be an https:// URL that resolves to a public address. localhost, private ranges and link-local addresses are rejected, so a collector behind your VPN needs a public ingress first.
• headers is how the collector authenticates you, and it is a full replacement: to add one header you must resend the others, or they are deleted. Read the connector first with get_forwarding_connector.
• Header values are secrets, so every read masks them. A masked value sent back unchanged means 'keep what is stored' — that is what makes the read-modify-write above safe. A masked value under a header key that is not stored is dropped, since there is no secret to recover.
• environments scopes what is forwarded; an empty list means everything.
• enabled false pauses a connector without deleting it, which is the reversible way to stop forwarding.
• A project may hold at most 3 forwarding connectors.

Recipes:

• Start forwarding to a collector: create with endpoint "https://otlp.example.com/v1/traces" and headers [{key: "Authorization", value: "Bearer <token>"}].
• Production only: environments ["production"].
• Rotate the collector's token: update with headers [{key: "Authorization", value: "Bearer NEW-TOKEN"}] — plus every other header the connector should keep, masked values included.
• Pause without losing the config: update with enabled false.
• Diagnose a broken connector: get_forwarding_connector and read lastError, failureCount and lastForwardedAt.

Calls POST /v2/forwarding-connectors.

This tool permanently deletes a forwarding connector from the Confident AI project. Forwarding to that collector stops immediately and the stored headers are destroyed, so restoring it means re-entering the credentials. Traces already delivered to the collector are not affected, and nothing in Confident AI is lost.

To stop forwarding reversibly, call update_forwarding_connector with enabled false instead.

Calls DELETE /v2/forwarding-connectors/{forwardingConnectorId}.

This tool fetches one forwarding connector from the Confident AI project with its full configuration and its delivery health. Use it before updating, since headers is a full replacement and you need to know what is already stored.

Header values come back masked. Sending a masked value back unchanged keeps the stored secret, so the read-modify-write round trip does not require knowing it.

Calls GET /v2/forwarding-connectors/{forwardingConnectorId}.

This tool lists the forwarding connectors configured in the Confident AI project — the OTLP collectors that traces are continuously mirrored to. Use it to discover the forwardingConnectorId values the other forwarding tools accept, and to see at a glance which connectors are currently forwarding.

Calls GET /v2/forwarding-connectors.

This tool updates a forwarding connector in the Confident AI project. Only the fields you send change — omit a field to leave it alone. headers and environments are the exception: each replaces the whole list, so fetch the current state with get_forwarding_connector and resend every entry the connector should keep.

Resending a masked header value keeps the secret that is already stored, so you can rotate one credential without knowing the others.

Forwarding guide:

• A forwarding connector continuously mirrors the project's traces to your own OTLP/HTTP collector (Datadog, Grafana, Honeycomb, an OpenTelemetry Collector) as they arrive. It is a live stream, not a scheduled export, and it copies rather than moves — traces stay in Confident AI either way.
• endpoint must be an https:// URL that resolves to a public address. localhost, private ranges and link-local addresses are rejected, so a collector behind your VPN needs a public ingress first.
• headers is how the collector authenticates you, and it is a full replacement: to add one header you must resend the others, or they are deleted. Read the connector first with get_forwarding_connector.
• Header values are secrets, so every read masks them. A masked value sent back unchanged means 'keep what is stored' — that is what makes the read-modify-write above safe. A masked value under a header key that is not stored is dropped, since there is no secret to recover.
• environments scopes what is forwarded; an empty list means everything.
• enabled false pauses a connector without deleting it, which is the reversible way to stop forwarding.
• A project may hold at most 3 forwarding connectors.

Recipes:

• Start forwarding to a collector: create with endpoint "https://otlp.example.com/v1/traces" and headers [{key: "Authorization", value: "Bearer <token>"}].
• Production only: environments ["production"].
• Rotate the collector's token: update with headers [{key: "Authorization", value: "Bearer NEW-TOKEN"}] — plus every other header the connector should keep, masked values included.
• Pause without losing the config: update with enabled false.
• Diagnose a broken connector: get_forwarding_connector and read lastError, failureCount and lastForwardedAt.

Calls PUT /v2/forwarding-connectors/{forwardingConnectorId}.

This tool re-assesses the governance controls of the policy the Confident AI project belongs to, and reports whether the project currently passes all of them. Fails with an error if the project is not assigned to any governance policy.

This is the project-scoped gate, meant for a deploy check on one project. To re-assess an entire policy across every project enrolled in it, use assess_governance_policy instead.

Calls POST /v2/governance/assess.

This tool opens a connection to an MCP server and lists the tools it exposes. Use it to verify a server after creating or updating one, and to refresh the tool list after the customer's server changes.

This reaches out to the customer's own MCP server — over the network for HTTP, or by launching the command as a subprocess for STDIO — and can take a few seconds. The verdict replaces the server's stored connected and availableTools.

A server that fails to connect is still a successful call — read connected rather than treating the result as an error.

To fix a failing connection: error carries the reason — an unreachable url, a rejected credential, a command that could not be launched, or "MCP server is not fully configured" when the field its transport requires is missing. Correct it with update_mcp_server and connect again.

Calls POST /v2/mcp-servers/{mcpServerId}/connect.

This tool registers one of the customer's MCP servers with the Confident AI project, so evaluations can give the app under test its tools. Supply the whole configuration in one call.

Registering does not connect to the server — the new server starts disconnected with no known tools. Call connect_mcp_server afterwards to verify it and discover what it exposes.

MCP server guide:

• An MCP server here is the customer's own MCP server registered with Confident AI, so evaluations can give the app under test real tools. Ids from list_mcp_servers are the mcpServerIds accepted by run_dataset_evaluation.
• transport decides which fields matter, and the two sets are mutually exclusive — the ones belonging to the transport you did not pick are cleared: HTTP needs url (plus optional headers/auth), STDIO needs command (plus optional args).
• Auth applies to HTTP only. authType defaults to HEADERS, meaning the static headers map carries the credential. OAUTH_CLIENT_CREDENTIALS and AZURE_AD use authConfig instead, and selecting either clears headers.
• Required in authConfig: clientId and clientSecret for OAUTH_CLIENT_CREDENTIALS; tenantId, clientId, scope and clientSecret for AZURE_AD.
• Secrets are write-only. clientSecret is never returned — get_mcp_server shows a masked clientSecretPreview instead — so omit it to keep the stored one, and never try to send the preview back. Changing authType discards the stored secret, so a new one must be supplied.
• Registering or changing a server does not connect to it. connected and availableTools come only from connect_mcp_server, and any config change resets connected to false — so connect again after every update.
• Names must be unique, and uniqueness is not scoped to your project: a name another project already took is rejected.

Recipes:

• Public HTTP server with a token: transport "HTTP", url "https://mcp.example.com/mcp", headers {"Authorization": "Bearer ..."}.
• Azure-AD-protected server: transport "HTTP", url "...", auth_type "AZURE_AD", auth_config {tenant_id, client_id, client_secret, scope}.
• Local subprocess server: transport "STDIO", command "npx", args ["-y", "@modelcontextprotocol/server-github"].
• Rotate a credential: update with auth_config {"client_secret": "NEW"} — the other auth fields are kept, unlike headers which must be resent whole.
• Move an HTTP server to STDIO: send transport "STDIO" and command; the url, headers and auth config are cleared for you.

Calls POST /v2/mcp-servers.

This tool permanently deletes an MCP server from the Confident AI project. Any AI connection or scheduled evaluation that attached this server stops using it. This cannot be undone.

Calls DELETE /v2/mcp-servers/{mcpServerId}.

This tool fetches one MCP server from the Confident AI project with its full configuration, including the tools discovered by the last successful connection. Use it to inspect the current state before updating.

The stored OAuth clientSecret is not returned — authConfig carries a masked clientSecretPreview in its place, which cannot be sent back. Static headers are returned as stored, credentials included.

Calls GET /v2/mcp-servers/{mcpServerId}.

This tool lists the MCP servers registered in the Confident AI project — the customer's own MCP servers that Confident AI knows how to reach. Use it to discover the mcpServerIds values accepted by run_dataset_evaluation.

Credentials (headers, environment variables, OAuth config) are never returned.

Calls GET /v2/mcp-servers.

This tool updates an MCP server in the Confident AI project. Only the fields you send change — omit a field to leave it alone, pass null to clear it. What you send is merged onto the stored server and the result must be valid on its own, so switching transport means supplying that transport's required field in the same call.

headers and args are full replacements: resend every entry you want to keep. authConfig is the exception and merges, so one credential field can be changed on its own.

Any change resets connected to false and does not reconnect, so the returned availableTools still lists what the server exposed under its previous configuration. Call connect_mcp_server afterwards to refresh both.

MCP server guide:

• An MCP server here is the customer's own MCP server registered with Confident AI, so evaluations can give the app under test real tools. Ids from list_mcp_servers are the mcpServerIds accepted by run_dataset_evaluation.
• transport decides which fields matter, and the two sets are mutually exclusive — the ones belonging to the transport you did not pick are cleared: HTTP needs url (plus optional headers/auth), STDIO needs command (plus optional args).
• Auth applies to HTTP only. authType defaults to HEADERS, meaning the static headers map carries the credential. OAUTH_CLIENT_CREDENTIALS and AZURE_AD use authConfig instead, and selecting either clears headers.
• Required in authConfig: clientId and clientSecret for OAUTH_CLIENT_CREDENTIALS; tenantId, clientId, scope and clientSecret for AZURE_AD.
• Secrets are write-only. clientSecret is never returned — get_mcp_server shows a masked clientSecretPreview instead — so omit it to keep the stored one, and never try to send the preview back. Changing authType discards the stored secret, so a new one must be supplied.
• Registering or changing a server does not connect to it. connected and availableTools come only from connect_mcp_server, and any config change resets connected to false — so connect again after every update.
• Names must be unique, and uniqueness is not scoped to your project: a name another project already took is rejected.

Recipes:

• Public HTTP server with a token: transport "HTTP", url "https://mcp.example.com/mcp", headers {"Authorization": "Bearer ..."}.
• Azure-AD-protected server: transport "HTTP", url "...", auth_type "AZURE_AD", auth_config {tenant_id, client_id, client_secret, scope}.
• Local subprocess server: transport "STDIO", command "npx", args ["-y", "@modelcontextprotocol/server-github"].
• Rotate a credential: update with auth_config {"client_secret": "NEW"} — the other auth fields are kept, unlike headers which must be resent whole.
• Move an HTTP server to STDIO: send transport "STDIO" and command; the url, headers and auth config are cleared for you.

Calls PUT /v2/mcp-servers/{mcpServerId}.

This tool creates a metric collection in the Confident AI project. A collection groups existing metrics (referenced by name) with per-metric settings, and is what cloud evaluations (evaluate_trace/evaluate_span/evaluate_thread) run. Every referenced metric must already exist and match the collection's multiTurn type, otherwise the call is rejected and nothing is created.

Calls POST /v2/metric-collections.

This tool permanently deletes a metric collection from the Confident AI project, along with the evaluation rules that run it. Past test runs, eval tasks and scheduled dataset tasks are kept but stop pointing at it, so they no longer evaluate against these metrics. This cannot be undone.

Calls DELETE /v2/metric-collections/{metricCollectionId}.

This tool fetches one metric collection from the Confident AI project with its metrics and their settings. Use it to inspect the current state before updating — particularly before sending metricsSettings, which replaces the whole list.

Calls GET /v2/metric-collections/{metricCollectionId}.

This tool retrieves a list of all available metric collections in your Confident AI project. Use this to discover what metric collections exist before triggering a cloud evaluation (e.g., using evaluate_trace, evaluate_span, or evaluate_thread). It tells you the exact metricCollection names you can use.

Calls GET /v2/metric-collections.

This tool updates a metric collection in the Confident AI project. Send at least one field; omit anything you are not changing. metricsSettings is a full replacement — the existing settings are deleted and recreated from the list you send, so fetch the collection with get_metric_collection first and resend every metric it should keep. A metric name that does not resolve rejects the whole call, leaving the collection as it was.

A collection's multiTurn cannot be changed, because it decides which metrics are eligible to be in it.

Calls PUT /v2/metric-collections/{metricCollectionId}.

This tool creates a custom LLM-as-a-judge metric in the Confident AI project. At least one of criteria or evaluationSteps must be provided.

Calls POST /v2/metrics.

This tool fetches a single custom metric from the Confident AI project by its id.

Calls GET /v2/metrics/{metricId}.

This tool lists all custom metrics defined in the Confident AI project.

Calls GET /v2/metrics.

This tool updates a custom metric in the Confident AI project. Note that criteria, evaluationSteps, and rubric are full replacements — omitting one clears it, and the metric must always keep at least one of criteria or evaluationSteps. Provide at least one of criteria, evaluationSteps, or evaluationParams.

Calls PUT /v2/metrics/{metricId}.

This tool creates multiple custom metrics in the Confident AI project in one call. Metrics whose name already exists are skipped. Each metric follows the same rules as create_metric (at least one of criteria or evaluationSteps; no duplicate name + multi_turn combinations within the batch).

Calls POST /v2/metrics-batch.

This tool lists metric data (online evaluation results) recorded in the Confident AI project, paginated and optionally filtered by time range.

Calls GET /v2/metrics-data.

This tool adds a custom model cost to the Confident AI project, so LLM spans matching the pattern are priced at the given rates instead of Confident AI's pre-configured defaults. It takes effect on spans ingested from then on; spans already stored keep the cost they were recorded with.

A match pattern and provider pair must be unique within the project — creating a second entry for the same pair is rejected. The same pattern with a different provider, or with no provider, is a separate entry and is allowed.

This fails while the project inherits model costs from its organization; call list_model_costs to check inherit first.

Calls POST /v2/model-costs.

This tool permanently deletes a custom model cost. Spans matching it fall back to Confident AI's pre-configured pricing for the model from then on; costs already recorded on stored spans are not recalculated.

This fails while the project inherits model costs from its organization.

Calls DELETE /v2/model-costs/{modelCostId}.

This tool lists the custom model costs configured for the Confident AI project. These are the per-model token prices used to estimate the cost of LLM spans whose provider did not report one, overriding Confident AI's pre-configured defaults. Use it to discover the modelCostId values the update and delete tools accept.

A cost sent explicitly with a span through DeepEval or the API always wins — these entries are the fallback for spans that arrive with token counts but no cost.

Check inherit on the response before trying to write: when it is true the project reads its model costs from the organization, and create, update and delete will all be rejected until inheritance is turned off in the project's model costs settings.

Calls GET /v2/model-costs.

This tool replaces a custom model cost's configuration.

This is a full replacement, not a merge: every field you leave out is cleared, so send the whole entry as you want it to end up. Omitting provider turns a provider-specific entry into a catch-all, and omitting a cost unprices that direction. Call list_model_costs first and resend the values you want to keep.

The resulting match pattern and provider pair must still be unique within the project. This fails while the project inherits model costs from its organization.

Calls PUT /v2/model-costs/{modelCostId}.

This tool re-runs one governance control against every project it governs and records fresh verdicts. This costs real compute, so prefer reading the existing verdicts with list_governance_control_assessments unless you need them recomputed now.

To re-assess a whole policy instead of one control, use assess_governance_policy.

Scope: this tool acts on the whole Confident AI organization, not one project, so it takes no injected projectId. It requires the caller to have sufficient permissions in the organization. (either the owner or an admin)

Calls POST /v2/organization/governance-controls/{controlId}/assess.

This tool re-assesses a governance policy: it re-runs every control the policy applies against every project enrolled in it, and records fresh verdicts. This costs real compute, proportional to controls x projects, so prefer reading the existing verdicts with get_governance_policy unless you specifically need them recomputed now.

This is the organization-scoped counterpart to assess_governance, which gates a single project with a project API key. Use that one for a deploy check; use this one to refresh a whole policy.

Scope: this tool acts on the whole Confident AI organization, not one project, so it takes no injected projectId. It requires the caller to have sufficient permissions in the organization. (either the owner or an admin)

Calls POST /v2/organization/governance-policies/{policyId}/assess.

This tool enrolls projects into a governance policy, so the policy's controls start gating them. A project belongs to at most one policy, so a project already on a different one is moved to this policy rather than added to both.

Partial success: ids that exist in the organization are assigned, and ids that do not are reported back instead of failing the whole call.

Scope: this tool acts on the whole Confident AI organization, not one project, so it takes no injected projectId. It requires the caller to have sufficient permissions in the organization. (either the owner or an admin)

Calls POST /v2/organization/governance-policies/{policyId}/assign.

This tool creates a governance control. Supplying the config matching the control's type snapshots its first version at the same time; omitting it creates the control with no definition, which you can add later with create_governance_control_version.

Control names are unique per organization, so reusing one fails with a 409. OPERATIONAL controls cannot be created — they are seeded from the platform registry.

Scope: this tool acts on the whole Confident AI organization, not one project, so it takes no injected projectId. It requires the caller to have sufficient permissions in the organization. (either the owner or an admin)

Control composition guide:

• A control's definition lives on its versions, not on the control itself. Creating or editing a definition appends a new immutable version; past verdicts keep pointing at the version they were computed against.
• Which config applies is decided by the control's type: RUNTIME -> runtime config, measured over live data PRE_DEPLOYMENT_EVALS -> pre-deployment config, gates on a test run PRE_DEPLOYMENT_RED_TEAMING -> pre-deployment config, gates on a red-team run OPERATIONAL -> seeded from the platform registry; only severity and filters mean anything
• Runtime config requires all of dataModel, aggregation, thresholdSettings, filters and severity to be present, though any of them may be null. Pre-deployment config requires identifier and window, plus filters and severity (nullable).
• aggregation is only valid for certain data models, and the API rejects the rest: TRACE : Count, Error rate, Pass rate, Unique end users, Unique threads, Unique metadata values, Avg latency, P50/P90/P99 latency, Total cost, Avg cost SPAN : Count, Error rate, Error count, Avg latency, P50/P90/P99 latency, Total cost, Input cost, Output cost, Avg cost, Input tokens, Output tokens, Total tokens, Unique metadata values THREAD : Count, Unique end users, Unique metadata values METRIC_DATA: Count, Avg score, Median score, Pass rate, Failure rate ANNOTATION : Count, Avg rating, Avg value Note SPAN has no Pass rate and TRACE has no token aggregations, so use METRIC_DATA to gate on eval scores and SPAN to gate on tokens.
• 'threshold_settings.direction' is which side fails: above fails when the measured value exceeds value, below when it falls under it.
• severity sets how much a failure matters. LOW never blocks a deploy gate; every other value blocks, and so does leaving it null.
• A control governs nothing until it is attached to a policy — use update_governance_policy_controls.

Calls POST /v2/organization/governance-controls.

This tool changes what a governance control checks, by appending a new version of its definition. It does not edit the current version: past verdicts keep pointing at the definition they were computed against, and the new definition applies from the next assessment onward.

Any field you leave out is carried over from the current version, so send the complete config for the control's type rather than a partial edit.

Scope: this tool acts on the whole Confident AI organization, not one project, so it takes no injected projectId. It requires the caller to have sufficient permissions in the organization. (either the owner or an admin)

Control composition guide:

• A control's definition lives on its versions, not on the control itself. Creating or editing a definition appends a new immutable version; past verdicts keep pointing at the version they were computed against.
• Which config applies is decided by the control's type: RUNTIME -> runtime config, measured over live data PRE_DEPLOYMENT_EVALS -> pre-deployment config, gates on a test run PRE_DEPLOYMENT_RED_TEAMING -> pre-deployment config, gates on a red-team run OPERATIONAL -> seeded from the platform registry; only severity and filters mean anything
• Runtime config requires all of dataModel, aggregation, thresholdSettings, filters and severity to be present, though any of them may be null. Pre-deployment config requires identifier and window, plus filters and severity (nullable).
• aggregation is only valid for certain data models, and the API rejects the rest: TRACE : Count, Error rate, Pass rate, Unique end users, Unique threads, Unique metadata values, Avg latency, P50/P90/P99 latency, Total cost, Avg cost SPAN : Count, Error rate, Error count, Avg latency, P50/P90/P99 latency, Total cost, Input cost, Output cost, Avg cost, Input tokens, Output tokens, Total tokens, Unique metadata values THREAD : Count, Unique end users, Unique metadata values METRIC_DATA: Count, Avg score, Median score, Pass rate, Failure rate ANNOTATION : Count, Avg rating, Avg value Note SPAN has no Pass rate and TRACE has no token aggregations, so use METRIC_DATA to gate on eval scores and SPAN to gate on tokens.
• 'threshold_settings.direction' is which side fails: above fails when the measured value exceeds value, below when it falls under it.
• severity sets how much a failure matters. LOW never blocks a deploy gate; every other value blocks, and so does leaving it null.
• A control governs nothing until it is attached to a policy — use update_governance_policy_controls.

Calls POST /v2/organization/governance-controls/{controlId}/versions.

This tool creates a governance policy. It starts with no controls and no projects — attach controls with update_governance_policy_controls and enroll projects with assign_projects_to_governance_policy.

Policy names are unique per organization, so reusing one fails with a 409.

Inheritance is two levels deep only: a policy may extend policies that themselves extend nothing, and naming one that already extends another fails with a 400.

Scope: this tool acts on the whole Confident AI organization, not one project, so it takes no injected projectId. It requires the caller to have sufficient permissions in the organization. (either the owner or an admin)

Calls POST /v2/organization/governance-policies.

This tool permanently deletes a governance control, along with every version of its definition and every verdict ever recorded for it. Any policy holding it simply loses a member and stops applying that check. This cannot be undone.

To stop a control gating one policy without destroying it, detach it with remove_governance_policy_controls instead.

Scope: this tool acts on the whole Confident AI organization, not one project, so it takes no injected projectId. It requires the caller to have sufficient permissions in the organization. (either the owner or an admin)

Calls DELETE /v2/organization/governance-controls/{controlId}.

This tool permanently deletes a governance policy, along with the assessment history recorded under it. Projects enrolled in it become ungoverned — they are not deleted, but nothing assesses them until they are enrolled elsewhere. The controls themselves are not deleted. This cannot be undone.

A policy that other policies extend cannot be deleted; the call fails with a 409 naming them, because its controls are live members of theirs.

Scope: this tool acts on the whole Confident AI organization, not one project, so it takes no injected projectId. It requires the caller to have sufficient permissions in the organization. (either the owner or an admin)

Calls DELETE /v2/organization/governance-policies/{policyId}.

This tool removes a governance policy's agent skill. Coding agents working on projects governed by the policy stop receiving its instructions. The policy and its controls are untouched. Fails with a 404 if the policy has no skill to remove.

Scope: this tool acts on the whole Confident AI organization, not one project, so it takes no injected projectId. It requires the caller to have sufficient permissions in the organization. (either the owner or an admin)

Calls DELETE /v2/organization/governance-policies/{policyId}/skill.

This tool reads one governance control with its health and the time it was last assessed. For the control's definition, read its versions with list_governance_control_versions; for its per-project verdicts, use list_governance_control_assessments.

Scope: this tool acts on the whole Confident AI organization, not one project, so it takes no injected projectId. It requires the caller to have sufficient permissions in the organization. (either the owner or an admin)

Calls GET /v2/organization/governance-controls/{controlId}.

This tool reads one governance policy in full: its effective controls with each one's latest verdict per project, the projects enrolled in it, the policies it extends and the policies that extend it, and its agent skill.

Scope: this tool acts on the whole Confident AI organization, not one project, so it takes no injected projectId. It requires the caller to have sufficient permissions in the organization. (either the owner or an admin)

Calls GET /v2/organization/governance-policies/{policyId}.

This tool reads a governance policy's agent skill — the instructions Confident AI serves to coding agents working on projects governed by that policy, so they know what the policy requires of the code they write.

Scope: this tool acts on the whole Confident AI organization, not one project, so it takes no injected projectId. It requires the caller to have sufficient permissions in the organization. (either the owner or an admin)

Calls GET /v2/organization/governance-policies/{policyId}/skill.

This tool reads one project's full governance view: the controls its policy applies (including those inherited from base policies) and its assessment history over the last 30 days. Use it to explain why a project is failing, once list_governance_projects has shown that it is.

Fails with a 404 if the project is not enrolled in a governance policy — there are no controls to assess it against. list_governance_projects shows which projects are enrolled.

Scope: this tool acts on the whole Confident AI organization, not one project, so it takes no injected projectId. It requires the caller to have sufficient permissions in the organization. (either the owner or an admin)

Calls GET /v2/organization/governance-projects/{projectId}.

This tool reads a control's verdicts per project. Verdicts belong to the definition version they were computed against, so history is read one version at a time; omitting version reads the current one.

Scope: this tool acts on the whole Confident AI organization, not one project, so it takes no injected projectId. It requires the caller to have sufficient permissions in the organization. (either the owner or an admin)

Calls GET /v2/organization/governance-controls/{controlId}/assessments.

This tool lists a control's definition history, newest first. Versions are append-only, so this is the audit trail of how the check has changed and which definition each past verdict was computed against.

Scope: this tool acts on the whole Confident AI organization, not one project, so it takes no injected projectId. It requires the caller to have sufficient permissions in the organization. (either the owner or an admin)

Calls GET /v2/organization/governance-controls/{controlId}/versions.

This tool lists the governance controls in the Confident AI organization — the individual checks that policies apply to projects — with each one's health across the projects it governs. Use it to discover the controlId values the other control tools accept, and to find which checks are failing.

Scope: this tool acts on the whole Confident AI organization, not one project, so it takes no injected projectId. It requires the caller to have sufficient permissions in the organization. (either the owner or an admin)

Calls GET /v2/organization/governance-controls.

This tool lists the governance policies in the Confident AI organization. A policy is the unit that governs: it holds controls, and every project enrolled in it is assessed against those controls. Start here to discover the policyId values every other policy tool accepts.

Scope: this tool acts on the whole Confident AI organization, not one project, so it takes no injected projectId. It requires the caller to have sufficient permissions in the organization. (either the owner or an admin)

Calls GET /v2/organization/governance-policies.

This tool lists the controls a governance policy applies — its own plus those inherited from the policies it extends — with each control's latest verdict per enrolled project.

Scope: this tool acts on the whole Confident AI organization, not one project, so it takes no injected projectId. It requires the caller to have sufficient permissions in the organization. (either the owner or an admin)

Calls GET /v2/organization/governance-policies/{policyId}/controls.

This tool lists the projects enrolled in a governance policy. For a project's own governance standing across the organization, including the ones enrolled in nothing, use list_governance_projects.

Scope: this tool acts on the whole Confident AI organization, not one project, so it takes no injected projectId. It requires the caller to have sufficient permissions in the organization. (either the owner or an admin)

Calls GET /v2/organization/governance-policies/{policyId}/projects.

This tool lists every project in the Confident AI organization with its governance standing — which policy governs it, how many controls apply, and whether those controls are currently passing. Start here to answer "is the organization compliant, and what needs attention", then call get_governance_project for the failing ones.

Scope: this tool acts on the whole Confident AI organization, not one project, so it takes no injected projectId. It requires the caller to have sufficient permissions in the organization. (either the owner or an admin)

Calls GET /v2/organization/governance-projects.

This tool detaches specific controls from a governance policy, leaving its other controls in place. The controls themselves are not deleted — they stay in the organization and can be attached to other policies. Use this rather than update_governance_policy_controls when you do not want to restate the controls the policy should keep.

Scope: this tool acts on the whole Confident AI organization, not one project, so it takes no injected projectId. It requires the caller to have sufficient permissions in the organization. (either the owner or an admin)

Calls DELETE /v2/organization/governance-policies/{policyId}/controls.

This tool removes projects from a governance policy. They become ungoverned — nothing assesses them until they are enrolled in a policy again. The projects themselves are untouched.

Partial success: only projects currently on this policy are removed, and anything else (unknown, or on another policy) is reported as skipped rather than failing the call.

Scope: this tool acts on the whole Confident AI organization, not one project, so it takes no injected projectId. It requires the caller to have sufficient permissions in the organization. (either the owner or an admin)

Calls POST /v2/organization/governance-policies/{policyId}/unassign.

This tool renames a governance control or changes its description. These live on the control itself, so editing them does not create a new version and does not change what the control checks — use create_governance_control_version for the definition.

Only the fields you send are touched, and sending none is a 400. An OPERATIONAL control's name and description come from the platform registry and cannot be edited, so the call fails with a 400.

Scope: this tool acts on the whole Confident AI organization, not one project, so it takes no injected projectId. It requires the caller to have sufficient permissions in the organization. (either the owner or an admin)

Calls PUT /v2/organization/governance-controls/{controlId}.

This tool updates a governance policy's name, description, owner, or the policies it extends. Every field is optional and only the ones you send are touched; sending none at all is a 400 rather than a silent no-op.

description and ownerEmail distinguish omitted from null: omit to leave the current value alone, pass null to clear it.

Scope: this tool acts on the whole Confident AI organization, not one project, so it takes no injected projectId. It requires the caller to have sufficient permissions in the organization. (either the owner or an admin)

Calls PUT /v2/organization/governance-policies/{policyId}.

This tool sets which controls a governance policy owns directly. It replaces the whole set: send every control id you want the policy to own, and any id you leave out is detached. Pass an empty list to detach all of them. To remove specific controls without listing the rest, use remove_governance_policy_controls instead.

Changing a policy's controls changes what gates every project enrolled in it, and takes effect at the next assessment.

Scope: this tool acts on the whole Confident AI organization, not one project, so it takes no injected projectId. It requires the caller to have sufficient permissions in the organization. (either the owner or an admin)

Calls PUT /v2/organization/governance-policies/{policyId}/controls.

This tool sets a governance policy's agent skill, creating it if the policy has none and replacing it outright otherwise. The skill is served to coding agents working on projects governed by this policy, so its body is read by a model rather than a person — state the policy's requirements concretely enough to act on.

Both fields are required and must be non-empty; to remove a skill use delete_governance_policy_skill.

Scope: this tool acts on the whole Confident AI organization, not one project, so it takes no injected projectId. It requires the caller to have sufficient permissions in the organization. (either the owner or an admin)

Calls PUT /v2/organization/governance-policies/{policyId}/skill.

This tool creates a persona in the Confident AI project. Write characteristics as the description a simulator needs to act as this user — who they are, what they want, how they speak — since that text is what replaces the user description of the multi-turn goldens the persona is attached to.

Calls POST /v2/personas.

This tool permanently deletes a persona from the Confident AI project. This cannot be undone.

Goldens attached to the persona are kept, but they lose it — a simulation of one of those goldens falls back to the golden's own user description, which may be empty.

Calls DELETE /v2/personas/{personaId}.

This tool fetches one persona with its full characteristics text. Use it to read the stored wording before editing it, since characteristics is replaced whole rather than appended to.

Calls GET /v2/personas/{personaId}.

This tool lists the personas in the Confident AI project. A persona describes the user that a multi-turn simulation plays: its characteristics stand in as the user description of any multi-turn golden it is attached to, so the simulated conversation is driven by that persona rather than the golden's own description.

Goldens are attached to personas in the Confident AI platform, not through the API, so use this tool to author and inspect the personas themselves and to discover the personaId values the get, update, and delete tools accept.

Calls GET /v2/personas.

This tool updates a persona. Send at least one of name and characteristics; whatever you omit keeps its stored value. characteristics replaces the stored text outright rather than being appended to, so send the whole description — read the current one with get_persona first when you are editing rather than replacing.

The new characteristics apply to every multi-turn golden already attached to this persona, changing the user that future simulations of those goldens play. Past test runs are unaffected.

Calls PUT /v2/personas/{personaId}.

This tool lists the Confident AI projects this connection can act on — the projects the authorizing user has access to. Every project-scoped tool requires a project_id, so call this first when you do not already have one, and ask the user which project to use if more than one is returned. The organization tools act across the whole organization and take no project_id.

Calls GET /v2/projects.

This tool creates a new branch on a prompt in Confident AI, diverging from the head commit of the main branch.

Calls POST /v2/prompts/{promptId}/branches.

This tool can be used to assign a formal version string to a specific commit hash of a prompt. Versions are automatically created on Confident AI and returned here.

Calls POST /v2/prompts/{promptId}/versions.

This tool deletes a branch from a prompt in Confident AI. The main branch cannot be deleted, and a branch with open pull requests must have them closed or merged on the platform first.

Calls DELETE /v2/prompts/{promptId}/branches/{branchId}.

This tool lists all branches of a prompt in Confident AI (every prompt has at least a main branch).

Calls GET /v2/prompts/{promptId}/branches.

This tool fetches a prompt from the Confident AI project at one commit, which is how you pin to an exact revision rather than whatever is current. The commit is looked up on main unless you name a branch. Call list_prompts for the promptId values, and get_prompt_commits to see the hashes on a branch.

Calls GET /v2/prompts/{promptId}/commits/{hash}.

This tool fetches whichever version of a prompt currently carries a label, so an application can follow a moving target like production without being redeployed when the label is moved. The answer changes as soon as somebody labels a different version. Call list_prompts for the promptId values.

Calls GET /v2/prompts/{promptId}/labels/{label}.

This tool fetches the commit of a prompt that was released as a given version, which pins to a published revision rather than whatever is current. Call list_prompts for the promptId values, and get_prompt_versions to see which versions exist.

Calls GET /v2/prompts/{promptId}/versions/{version}.

This tool retrieves a chronological list of all the commits associated with a prompt, addressing it by its id. Use this to view the history of changes, find specific commit hashes, and read commit messages. Pass branch to list one branch's commits rather than every branch's.

Calls GET /v2/prompts/{promptId}/commits.

This tool retrieves a list of all formal versions associated with a prompt, addressing it by its id. Use this to see which versions of a prompt have been released to production or tagged.

Calls GET /v2/prompts/{promptId}/versions.

This tool can be used to retrieve a list of all prompts available in Confident AI.

Calls GET /v2/prompts.

This tool creates a prompt, or saves modifications to an existing prompt's template or configuration, back to Confident AI. The body goes in prompt and carries the template as either text or messages, never both, under the alias the prompt is pushed to — a prompt is created if that alias is new.

Every push creates a new commit in the prompt's history, so you can track changes over time. It does not create a version string: call create_version afterwards for that.

Calls POST /v2/prompts.

This tool renames a branch of a prompt in Confident AI. The main branch cannot be renamed.

Calls PUT /v2/prompts/{promptId}/branches/{branchId}.

This tool creates a report template in the Confident AI project — a recurring executive report that Confident AI researches and writes from your project's own data, on a schedule. Give it a description written as the question the report answers; optionally supply templateSections to fix the report's exact structure, and a cadence to control when it generates.

The guide below is the whole contract — what the generator can retrieve, what it will not do, and what each section type accepts.

Report template guide:

• A report template is a recurring report: a description (the question the report answers) plus, optionally, the exact templateSections it renders.

What the generator can draw on:

• Observability: traces (end-to-end requests), the spans inside them (LLM, retriever, tool, agent), threads grouping traces into conversations, and the end users behind them — volume, error rate, latency percentiles, cost.
• Evaluation: datasets, test runs, and individual metric scores. Scores cover live traffic too, not just test runs.
• Human feedback: reviews and ratings on traces/spans/threads with reviewer identity, and how far each automated metric agrees with them.
• Diagnostics: row-level detail behind failures — failing traces and failing test cases, and annotation contents.

What it will NOT do — asking for these wastes a section:

• Invent or recompute figures. It reports what the platform measures; the only thing it derives is the delta between a current and previous value. Ask for the underlying numbers, not a ratio it would have to compute.
• Return complete row-level data. Failing traces and test cases come back as a curated sample (at most 10-20 items), annotation contents at most 50. Ask for aggregates when the report needs totals.
• Compare periods unless asked. See the window rule below.
• Reach anything outside the project's own observability and evaluation data — no external sources, no repo or deployment data.
• Fail loudly on thin data: an unrelated description, or a window with nothing in it, yields a completed report with one section explaining why.

Writing the template:

• Two modes. With NO sections, the generator picks the structure itself from the description — fastest path, good when you know the question but not the shape. With sections, the report follows yours exactly (same order, same headings).
• The description matters in BOTH modes: it drives the single data fetch that serves every section. A vague description starves well-written sections.
• Section types: CONTENT (prose), ADMONITION (severity callout), STAT_CARDS (headline numbers), TABLE (ranking/breakdown), GRAPH (one chart). STAT_CARDS/TABLE/GRAPH must set use_ai true; CONTENT/ADMONITION can instead be hardcoded with static content.
• Each AI section takes one prompt directive — what THAT section must cover. One directive per section; a prompt asking for two unrelated things produces a section that does neither well.
• Default window is the last 7 days. Comparative wording in the description ("versus last week", "how did this change") is what triggers period-over-period deltas — without it the report only describes the current window.

Scheduling (a template generates on its own cadence):

• A new template defaults to INTERVAL every 1 DAY, enabled. Set recurrence, repeatEvery and repeatUnit for a different cadence — recurrence INTERVAL with repeat_every 1 and repeat_unit WEEK for a weekly report.
• An INTERVAL schedule needs BOTH repeat_every and repeat_unit; one without the other is rejected. Use recurrence ONCE for a single report, with startAt for when (omitted means immediately).
• Bound it with maxRuns and/or endAt; enabled false pauses generation. Once a schedule has hit its cap or end date it cannot be re-enabled until the cap is raised or the date pushed out in the same call.

Recipes:

• Production health, generator-structured: description "Give me an overall health check for the last week: request volume, error rate, latency (average and p99), total cost, top models, and user activity.", no sections.
• Failure diagnosis, explicit structure: sections [{type STAT_CARDS, use_ai true, prompt "Headline the error rate, request volume and p99 latency."}, {type CONTENT, heading "What's Failing", use_ai true, prompt "Summarize the dominant failure modes in 2-4 sentences, citing error counts."}, {type TABLE, heading "Recent Failures", use_ai true, prompt "Table the recent failing traces with their error message and latency."}, {type CONTENT, heading "Recommendations", use_ai true, prompt "A bullet list of concrete fixes grounded in the errors above."}]
• Cost review with a chart: sections [{type GRAPH, heading "Daily LLM Cost", use_ai true, prompt "Chart total LLM cost per day over the window."}, {type TABLE, heading "Cost by Model", use_ai true, prompt "Rank models by total cost with call count and average latency."}]
• Fixed preamble: {type CONTENT, heading "About This Report", use_ai false, content {text "Generated weekly for the platform team."}}

Calls POST /v2/report-templates.

This tool permanently deletes a report template from the Confident AI project. Every report ever generated by it becomes unreachable, forever — this cannot be undone.

Warn the user before calling this, and tell them that disabling the template instead is usually the better option: call update_report_template with enabled false, and generation stops while every past report stays readable.

Calls DELETE /v2/report-templates/{reportTemplateId}.

This tool fetches one report template from the Confident AI project, including every section with its id, prompt or static content, and order. Use it to inspect a template's current sections before replacing them with update_report_template.

Calls GET /v2/report-templates/{reportTemplateId}.

This tool lists the report templates in the Confident AI project, oldest first, with whether each one's schedule is enabled.

Calls GET /v2/report-templates.

This tool updates a report template in the Confident AI project. At least one field must be provided. Passing templateSections REPLACES the section list wholesale — fetch the current state with get_report_template and include every section the template should keep, or the omitted ones are removed.

Set enabled false to pause scheduled generation without deleting the template or its report history. Cadence fields can be retimed here too; omitting one leaves it as it was, and passing it as null clears it.

Report template guide:

• A report template is a recurring report: a description (the question the report answers) plus, optionally, the exact templateSections it renders.

What the generator can draw on:

• Observability: traces (end-to-end requests), the spans inside them (LLM, retriever, tool, agent), threads grouping traces into conversations, and the end users behind them — volume, error rate, latency percentiles, cost.
• Evaluation: datasets, test runs, and individual metric scores. Scores cover live traffic too, not just test runs.
• Human feedback: reviews and ratings on traces/spans/threads with reviewer identity, and how far each automated metric agrees with them.
• Diagnostics: row-level detail behind failures — failing traces and failing test cases, and annotation contents.

What it will NOT do — asking for these wastes a section:

• Invent or recompute figures. It reports what the platform measures; the only thing it derives is the delta between a current and previous value. Ask for the underlying numbers, not a ratio it would have to compute.
• Return complete row-level data. Failing traces and test cases come back as a curated sample (at most 10-20 items), annotation contents at most 50. Ask for aggregates when the report needs totals.
• Compare periods unless asked. See the window rule below.
• Reach anything outside the project's own observability and evaluation data — no external sources, no repo or deployment data.
• Fail loudly on thin data: an unrelated description, or a window with nothing in it, yields a completed report with one section explaining why.

Writing the template:

• Two modes. With NO sections, the generator picks the structure itself from the description — fastest path, good when you know the question but not the shape. With sections, the report follows yours exactly (same order, same headings).
• The description matters in BOTH modes: it drives the single data fetch that serves every section. A vague description starves well-written sections.
• Section types: CONTENT (prose), ADMONITION (severity callout), STAT_CARDS (headline numbers), TABLE (ranking/breakdown), GRAPH (one chart). STAT_CARDS/TABLE/GRAPH must set use_ai true; CONTENT/ADMONITION can instead be hardcoded with static content.
• Each AI section takes one prompt directive — what THAT section must cover. One directive per section; a prompt asking for two unrelated things produces a section that does neither well.
• Default window is the last 7 days. Comparative wording in the description ("versus last week", "how did this change") is what triggers period-over-period deltas — without it the report only describes the current window.

Scheduling (a template generates on its own cadence):

• A new template defaults to INTERVAL every 1 DAY, enabled. Set recurrence, repeatEvery and repeatUnit for a different cadence — recurrence INTERVAL with repeat_every 1 and repeat_unit WEEK for a weekly report.
• An INTERVAL schedule needs BOTH repeat_every and repeat_unit; one without the other is rejected. Use recurrence ONCE for a single report, with startAt for when (omitted means immediately).
• Bound it with maxRuns and/or endAt; enabled false pauses generation. Once a schedule has hit its cap or end date it cannot be re-enabled until the cap is raised or the date pushed out in the same call.

Recipes:

• Production health, generator-structured: description "Give me an overall health check for the last week: request volume, error rate, latency (average and p99), total cost, top models, and user activity.", no sections.
• Failure diagnosis, explicit structure: sections [{type STAT_CARDS, use_ai true, prompt "Headline the error rate, request volume and p99 latency."}, {type CONTENT, heading "What's Failing", use_ai true, prompt "Summarize the dominant failure modes in 2-4 sentences, citing error counts."}, {type TABLE, heading "Recent Failures", use_ai true, prompt "Table the recent failing traces with their error message and latency."}, {type CONTENT, heading "Recommendations", use_ai true, prompt "A bullet list of concrete fixes grounded in the errors above."}]
• Cost review with a chart: sections [{type GRAPH, heading "Daily LLM Cost", use_ai true, prompt "Chart total LLM cost per day over the window."}, {type TABLE, heading "Cost by Model", use_ai true, prompt "Rank models by total cost with call count and average latency."}]
• Fixed preamble: {type CONTENT, heading "About This Report", use_ai false, content {text "Generated weekly for the platform team."}}

Calls PUT /v2/report-templates/{reportTemplateId}.

This tool writes a report into the Confident AI project — YOU author the content and it is stored and rendered as given. Use it to publish an analysis you have already done into the platform, where a team can read, page through and export it.

This is not the AI report generator: nothing here is retrieved, verified or written for you. If you want Confident AI to research and write a recurring report from your project's own data instead, use create_report_template.

Section guide (content must match the section's type):

• CONTENT -> {kind: 'narrative', narrative: '...'}: prose. Plain text only, no markdown or code fences, and don't repeat the heading. Lists go one item per line starting with '- '.
• ADMONITION -> {severity, text}: a callout of 1-3 sentences. severity is INFO, SUCCESS, WARNING or DANGER.
• STAT_CARDS -> {cards: [{label, value, caption?}], highlights?: [{label, value}]}: 3-5 cards, at most 3 highlights. Labels are short Title Case phrases (2-4 words), never raw column names. Round numbers to 2 decimals.
• TABLE -> {headers: [...], rows: [[...]]}: every row needs EXACTLY as many cells as there are headers, in the same order. Short cell text.
• GRAPH -> {type: 'snapshot', graphType, categories: [...], series: [{name, values}]}: one chart with its data baked in. Every series' 'values' must be the same length as categories. LINE/AREA for trends, BAR/STACKED_BAR to compare.

Composition:

• Sections render in the order given; put headline numbers first, then the summary, then detail, then recommendations.
• Every report belongs to a report template (report_template_id) — that's how it is reached in the platform UI. Create one with create_report_template first, or reuse an existing one from list_report_templates.
• Ground every figure in data you actually retrieved. This tool writes exactly what you give it; nothing is verified or recomputed.
• Default status is COMPLETED, which makes the report immediately viewable. Use IN_PROGRESS if you intend to add sections over several update_report calls, then set COMPLETED when done.

Calls POST /v2/reports.

This tool permanently deletes a report and its sections from the Confident AI project. The report template it belonged to is left untouched.

Calls DELETE /v2/reports/{reportId}.

This tool fetches one report from the Confident AI project with all of its sections and their content — what a reader sees rendered. Use it to read a report the AI generator produced, or to inspect a report's current sections before replacing them with update_report.

Calls GET /v2/reports/{reportId}.

This tool lists generated reports in the Confident AI project, newest first, without their section content. Filter by template, status, or created-at window. Fetch one report's sections with get_report.

Calls GET /v2/reports.

This tool updates a report in the Confident AI project. At least one field must be provided. Passing sections REPLACES the whole list — fetch the current state with get_report and include every section the report should keep, or the omitted ones are removed.

Section guide (content must match the section's type):

• CONTENT -> {kind: 'narrative', narrative: '...'}: prose. Plain text only, no markdown or code fences, and don't repeat the heading. Lists go one item per line starting with '- '.
• ADMONITION -> {severity, text}: a callout of 1-3 sentences. severity is INFO, SUCCESS, WARNING or DANGER.
• STAT_CARDS -> {cards: [{label, value, caption?}], highlights?: [{label, value}]}: 3-5 cards, at most 3 highlights. Labels are short Title Case phrases (2-4 words), never raw column names. Round numbers to 2 decimals.
• TABLE -> {headers: [...], rows: [[...]]}: every row needs EXACTLY as many cells as there are headers, in the same order. Short cell text.
• GRAPH -> {type: 'snapshot', graphType, categories: [...], series: [{name, values}]}: one chart with its data baked in. Every series' 'values' must be the same length as categories. LINE/AREA for trends, BAR/STACKED_BAR to compare.

Composition:

• Sections render in the order given; put headline numbers first, then the summary, then detail, then recommendations.
• Every report belongs to a report template (report_template_id) — that's how it is reached in the platform UI. Create one with create_report_template first, or reuse an existing one from list_report_templates.
• Ground every figure in data you actually retrieved. This tool writes exactly what you give it; nothing is verified or recomputed.
• Default status is COMPLETED, which makes the report immediately viewable. Use IN_PROGRESS if you intend to add sections over several update_report calls, then set COMPLETED when done.

Calls PUT /v2/reports/{reportId}.

This tool adds a risk category to a red-teaming framework, optionally with the vulnerability types and attack methods it should probe. A category with no vulnerability types selected produces no attacks when the framework runs.

Ids come from list_vulnerabilities (its vulnerability types) and list_attack_methods. An attack method whose required parameters are not configured is rejected, so set them with update_attack_method first.

Note: red teaming through the API is available on the Enterprise plan only.

Calls POST /v2/rt-frameworks/{rtFrameworkId}/risk-categories.

This tool creates a red-teaming framework in the Confident AI project, either empty or prefilled from a built-in template. A template brings its own risk categories with the vulnerabilities and attack methods each one covers, so it is the fastest way to a runnable framework; an empty framework needs risk categories added with create_risk_category before it can run.

Note: red teaming through the API is available on the Enterprise plan only.

Calls POST /v2/rt-frameworks.

This tool permanently deletes a risk category from a red-teaming framework, along with its vulnerability and attack method selections and its remediation priorities. The vulnerabilities and attack methods themselves are not deleted. Risk assessments already run against this category are kept. This cannot be undone.

Calls DELETE /v2/rt-frameworks/{rtFrameworkId}/risk-categories/{riskCategoryId}.

This tool permanently deletes a red-teaming framework from the Confident AI project, along with every risk category in it and any schedule set up to run it. Risk assessments already produced by the framework are kept, but stop pointing at it. This cannot be undone.

Calls DELETE /v2/rt-frameworks/{rtFrameworkId}.

This tool fetches one risk category with everything it selects: the vulnerability types it probes for, the attack methods it probes with, and the remediation priority set per vulnerability. Each entry carries the id update_risk_category takes, but under a different field, so collect them into vulnerabilityTypeIds and attackMethodIds rather than sending this response back unchanged. vulnerabilityIdToPriorityLevel is the one field already in the shape that tool expects.

Calls GET /v2/rt-frameworks/{rtFrameworkId}/risk-categories/{riskCategoryId}.

This tool fetches one red-teaming framework from the Confident AI project in full, with every risk category and, for each, the vulnerabilities it probes for (their types, criteria and evaluation settings) and the attack methods it probes with. Use it to see what a framework will actually run before running it.

Calls GET /v2/rt-frameworks/{rtFrameworkId}.

This tool lists the risk categories in a red-teaming framework. A risk category is what a run is scoped to: it holds the vulnerability types to probe for and the attack methods to probe them with.

Note: red teaming through the API is available on the Enterprise plan only.

Calls GET /v2/rt-frameworks/{rtFrameworkId}/risk-categories.

This tool lists the red-teaming frameworks in the Confident AI project. A framework groups the risk categories a risk assessment can run, and each risk category selects the vulnerabilities to probe for and the attack methods to probe them with. Use this to find the framework id and risk category names to run.

Note: red teaming through the API is available on the Enterprise plan only.

Calls GET /v2/rt-frameworks.

This tool dispatches a red-teaming risk assessment for a framework in the Confident AI project. The run executes asynchronously and attacks your own application, so it costs usage and sends generated attacks to the target you name. The response contains the id of the created risk assessment and a link to watch it.

Note: red teaming through the API is available on the Enterprise plan only.

Calls POST /v2/rt-frameworks/{rtFrameworkId}/run.

This tool changes what a risk category probes for. Send at least one field; omit anything you are not changing. vulnerabilityTypeIds, attackMethodIds and vulnerabilityIdToPriorityLevel are each full replacements — fetch the category with get_risk_category first and resend every id it should keep, since a partial list drops the rest.

An attack method whose required parameters are not configured is rejected and nothing is changed, so set them with update_attack_method first.

Note: red teaming through the API is available on the Enterprise plan only.

Calls PUT /v2/rt-frameworks/{rtFrameworkId}/risk-categories/{riskCategoryId}.

This tool renames a red-teaming framework or changes its description. Send at least one field; omit anything you are not changing. The framework's risk categories are not touched, so use the risk category tools to change what it runs.

Note: red teaming through the API is available on the Enterprise plan only.

Calls PUT /v2/rt-frameworks/{rtFrameworkId}.

This tool creates a scheduled alert in the Confident AI project. Give it a clear name and a description saying what the alert means and what to do when it fires, since both appear in the notification a human receives.

Base the threshold on what the project's data actually looks like — query the Observatory data first rather than guessing a number, or the alert will fire constantly or never.

Alert composition guide:

• type picks what the alert watches, and defaults to THRESHOLD: THRESHOLD : an aggregate crosses a value. REGRESSION : a trace's newest version, measured on the traces since the previous run, performs worse than its baseline version over the last 30 days. ANOMALY : a trace's metric in a time bucket that completed since the previous run breaks sharply from its own history (buckets are 30 minutes, an hour or a day, the largest that fits the repeat interval). SIGNAL_SPIKE : a classifier label's count since the previous run at least doubles against the interval before, or appears for the first time. REGRESSION, ANOMALY and SIGNAL_SPIKE take detectionSettings instead of aggregation and thresholdSettings, need data_model TRACE (SIGNAL_SPIKE also accepts THREAD), and notify on every run while something is detected inside filters.
• detectionSettings always takes direction (WORSENED or ANY) and minAffectedCount (the fewest affected traces, or threads for a THREAD SIGNAL_SPIKE). REGRESSION and ANOMALY add optional traceName (omit for every trace) and metrics (omit for all of avg_score, error_rate, avg_latency, avg_cost, negative_label_rate); REGRESSION also requires significantOnly. SIGNAL_SPIKE requires classifierIds.
• A THRESHOLD alert re-runs an Observatory aggregate query on a schedule and notifies when the result crosses a threshold: dataModel + aggregation (what to measure), filters (over which slice), thresholdSettings (when to fire), and the schedule fields (how often).
• Each dataModel accepts a different set of aggregation tokens, and the API rejects a pairing it does not support. Read the aggregation field's own description for the tokens each data model takes: it is derived from the alert engine, so it covers every data model and lists only tokens that actually evaluate.
• Latency thresholds are in SECONDS, cost thresholds in USD, and rates (ERROR_RATE, PASS_RATE) are fractions between 0 and 1 — not percentages.
• For every type the measurement window is the repeat interval: an alert repeating every 1 HOUR looks at the last hour of data. A ONCE alert measures the last 24 hours.
• Delivery is configured per project, not per alert. An alert notifies through the project's integrations (Slack, email, PagerDuty, ...) that have alerting enabled for its severity, so an alert in a project with no such integration evaluates on schedule but reaches nobody.

Recipes:

• Error-rate spike: data_model TRACE, aggregation ERROR_RATE, threshold_settings {value: 0.05, direction: 'above'}, recurrence INTERVAL, repeat_every 1, repeat_unit HOUR, severity ERROR
• Latency regression: data_model TRACE, aggregation P90_LATENCY, threshold_settings {value: 5, direction: 'above'}, repeat_every 15, repeat_unit MINUTE
• Daily cost blowout: data_model SPAN, aggregation TOTAL_COST, threshold_settings {value: 100, direction: 'above'}, repeat_every 1, repeat_unit DAY, severity CRITICAL
• Traffic drop: data_model TRACE, aggregation COUNT, threshold_settings {value: 10, direction: 'below'}, repeat_every 1, repeat_unit HOUR
• Version regressions on any trace: type REGRESSION, data_model TRACE, detection_settings {direction: 'WORSENED', significant_only: true, min_affected_count: 20}, repeat_every 1, repeat_unit DAY, severity ERROR

Calls POST /v2/scheduled-alerts.

This tool permanently deletes a scheduled alert from the Confident AI project and unregisters its next run. To stop an alert temporarily, prefer update_scheduled_alert with enabled false, which keeps the definition.

Calls DELETE /v2/scheduled-alerts/{scheduledAlertId}.

This tool retrieves a single scheduled alert in the Confident AI project by id, including its schedule state (runCount, lastRunAt). Fetch the alert with this tool before updating it, so unchanged fields can be left out of the update.

Calls GET /v2/scheduled-alerts/{scheduledAlertId}.

This tool lists the scheduled alerts in the Confident AI project — the standing checks that run on a schedule and notify when an aggregate crosses a threshold, or while a regression, anomaly or signal spike is detected. Use it to discover the scheduledAlertId values the get, update, and delete tools accept, and to check what a project is already alerting on before adding more.

Calls GET /v2/scheduled-alerts.

This tool updates a scheduled alert in the Confident AI project. It is a partial update: only the fields you send are changed, and at least one is required. Send enabled false to pause an alert rather than deleting it.

Two fields need care:

• filters is replaced wholesale, not merged — send the complete set you want, or null to clear it. Omitting it leaves the stored filters untouched.
• Changing dataModel can strand the existing aggregation, since each data model accepts a different set. Send both together when moving an alert between data models.

Alert composition guide:

• type picks what the alert watches, and defaults to THRESHOLD: THRESHOLD : an aggregate crosses a value. REGRESSION : a trace's newest version, measured on the traces since the previous run, performs worse than its baseline version over the last 30 days. ANOMALY : a trace's metric in a time bucket that completed since the previous run breaks sharply from its own history (buckets are 30 minutes, an hour or a day, the largest that fits the repeat interval). SIGNAL_SPIKE : a classifier label's count since the previous run at least doubles against the interval before, or appears for the first time. REGRESSION, ANOMALY and SIGNAL_SPIKE take detectionSettings instead of aggregation and thresholdSettings, need data_model TRACE (SIGNAL_SPIKE also accepts THREAD), and notify on every run while something is detected inside filters.
• detectionSettings always takes direction (WORSENED or ANY) and minAffectedCount (the fewest affected traces, or threads for a THREAD SIGNAL_SPIKE). REGRESSION and ANOMALY add optional traceName (omit for every trace) and metrics (omit for all of avg_score, error_rate, avg_latency, avg_cost, negative_label_rate); REGRESSION also requires significantOnly. SIGNAL_SPIKE requires classifierIds.
• A THRESHOLD alert re-runs an Observatory aggregate query on a schedule and notifies when the result crosses a threshold: dataModel + aggregation (what to measure), filters (over which slice), thresholdSettings (when to fire), and the schedule fields (how often).
• Each dataModel accepts a different set of aggregation tokens, and the API rejects a pairing it does not support. Read the aggregation field's own description for the tokens each data model takes: it is derived from the alert engine, so it covers every data model and lists only tokens that actually evaluate.
• Latency thresholds are in SECONDS, cost thresholds in USD, and rates (ERROR_RATE, PASS_RATE) are fractions between 0 and 1 — not percentages.
• For every type the measurement window is the repeat interval: an alert repeating every 1 HOUR looks at the last hour of data. A ONCE alert measures the last 24 hours.
• Delivery is configured per project, not per alert. An alert notifies through the project's integrations (Slack, email, PagerDuty, ...) that have alerting enabled for its severity, so an alert in a project with no such integration evaluates on schedule but reaches nobody.

Recipes:

• Error-rate spike: data_model TRACE, aggregation ERROR_RATE, threshold_settings {value: 0.05, direction: 'above'}, recurrence INTERVAL, repeat_every 1, repeat_unit HOUR, severity ERROR
• Latency regression: data_model TRACE, aggregation P90_LATENCY, threshold_settings {value: 5, direction: 'above'}, repeat_every 15, repeat_unit MINUTE
• Daily cost blowout: data_model SPAN, aggregation TOTAL_COST, threshold_settings {value: 100, direction: 'above'}, repeat_every 1, repeat_unit DAY, severity CRITICAL
• Traffic drop: data_model TRACE, aggregation COUNT, threshold_settings {value: 10, direction: 'below'}, repeat_every 1, repeat_unit HOUR
• Version regressions on any trace: type REGRESSION, data_model TRACE, detection_settings {direction: 'WORSENED', significant_only: true, min_affected_count: 20}, repeat_every 1, repeat_unit DAY, severity ERROR

Calls PUT /v2/scheduled-alerts/{scheduledAlertId}.

This tool retrieves the complete details of a specific execution span using its UUID. Unlike the list_spans tool, this includes deep details such as the full input/output payload, evaluation metrics data, and human annotations.

Calls GET /v2/spans/{spanUuid}.

This tool retrieves a paginated list of spans from Confident AI. Spans represent individual steps (e.g., an LLM call, a tool execution, or a retriever step) within a larger trace. Use this to filter specific span types, environments, or prompt versions.

Calls GET /v2/spans.

This tool retrieves the complete details of a specific test run using its unique ID. Use this to deeply inspect exactly which test cases failed, view the specific metric scores (and reasoning) for each test case, and examine the tools called during the execution.

Calls GET /v2/test-runs/{testRunId}.

This tool retrieves a paginated list of test runs (evaluations) from Confident AI. Use this to see recent experiments, check if a test run passed/failed, or filter for specific environments or multi-turn conversational tests.

Calls GET /v2/test-runs.

This tool can be used to retrieve the full details of a specific conversational thread. A thread groups multiple traces together into a single conversation and includes thread-level metrics and annotations.

Calls GET /v2/threads/{threadId}.

This tool can be used to retrieve a paginated list of conversation threads from Confident AI. Use this to discover recent multi-turn conversations, filter by environment, or find specific thread IDs for further inspection with get_thread.

Calls GET /v2/threads.

This tool can be used to retrieve the full details of a specific execution trace. A trace includes the overall input/output and a list of all internal spans (steps).

Calls GET /v2/traces/{traceUuid}.

This tool can be used to retrieve a paginated list of traces from Confident AI. Use this to discover recent executions, filter by environment, or find specific trace UUIDs for further inspection with get_trace.

Calls GET /v2/traces.

This tool creates a transformer in the Confident AI project. Creating one does not run it and does not attach it to anything: point an AI connection or metric collection at the returned id to put it to use, and call test_transformer_code to check it does what you expect.

The code is not checked for syntax at create time, so a broken transformer is accepted here and only fails when something runs it.

Transformer guide:

• A transformer is a piece of Python that reshapes one value, for the cases a plain key path can't express. Its id is what the AI connection fields actualOutputTransformerId, retrievalContextTransformerId, toolsCalledTransformerId and stateTransformerId accept, and what a metric collection's or dataset ingestion task's inputTransformerId and outputTransformerId accept.
• The code must define a function named transformer taking a single argument and returning the transformed value. 'async def' works. A function named transform_response is also accepted, but prefer transformer — it is what the platform's editor scaffolds and what the batch evaluation path wraps.
• The argument is the JSON value being transformed: your endpoint's response body when extracting an AI connection's output, or the input / actual output when it is a metric collection's input / output transformer. Both the argument and the return value must be JSON serializable.
• language is PYTHON; nothing else is supported yet.
• name is unique within the project.
• Omit anything you are not changing. codeDefinition is replaced whole, so send code and language together — read the current code with get_transformer first.
• test_transformer_code runs the code that is stored, so save an edit with update_transformer before testing it.

Recipes:

• Starting point, the same one the platform scaffolds: "from typing import Any\n\ndef transformer(data: Any):\n return data\n"
• Reach a value no key path can: def transformer(data): return data["choices"][0]["message"]["content"].strip()
• Flatten retrieved documents into the list of strings a metric expects: def transformer(data): return [doc["text"] for doc in data["documents"]]
• Parse a stringified payload before reading it: def transformer(data): return json.loads(data)["answer"] — with 'import json' at the top.

Calls POST /v2/transformers.

This tool permanently deletes a transformer from the Confident AI project, along with its code. This cannot be undone.

Anything pointing at it is not deleted but silently stops using it: an AI connection's transformer field and a metric collection's or dataset ingestion task's input/output transformer are cleared, so they fall back to their key paths — and an AI connection that had no key path for that output stops being able to read it. Check what references the transformer with list_ai_connections and list_metric_collections before deleting it.

Calls DELETE /v2/transformers/{transformerId}.

This tool fetches one transformer from the Confident AI project, including its code. Use it to read the current code before updating, since codeDefinition is replaced whole rather than merged.

Calls GET /v2/transformers/{transformerId}.

This tool lists the transformers in the Confident AI project. A transformer is a piece of code that pulls a value out of an endpoint's response when a plain key path can't express it.

Use it to resolve the ids that the AI connection tools accept for actualOutputTransformerId and its retrieval-context, tools-called, and state counterparts, and the ones metric collections accept for inputTransformerId and outputTransformerId.

Calls GET /v2/transformers.

This tool runs a stored transformer's code against a sample value in a sandbox and reports what it returned. Use it to check a transformer before pointing an AI connection or metric collection at it, and to debug one that is producing the wrong value — pass a realistic inputData, such as the response body your endpoint actually returns.

It runs the code that is stored, not code you pass in, so save an edit with update_transformer first. Execution is sandboxed and time-limited, and nothing about the transformer is changed by testing it.

Code that raises is still a successful call — read success rather than treating the result as an error. On failure, error gives the category, reason a sentence, and verboseLogs the Python traceback.

Transformer guide:

• A transformer is a piece of Python that reshapes one value, for the cases a plain key path can't express. Its id is what the AI connection fields actualOutputTransformerId, retrievalContextTransformerId, toolsCalledTransformerId and stateTransformerId accept, and what a metric collection's or dataset ingestion task's inputTransformerId and outputTransformerId accept.
• The code must define a function named transformer taking a single argument and returning the transformed value. 'async def' works. A function named transform_response is also accepted, but prefer transformer — it is what the platform's editor scaffolds and what the batch evaluation path wraps.
• The argument is the JSON value being transformed: your endpoint's response body when extracting an AI connection's output, or the input / actual output when it is a metric collection's input / output transformer. Both the argument and the return value must be JSON serializable.
• language is PYTHON; nothing else is supported yet.
• name is unique within the project.
• Omit anything you are not changing. codeDefinition is replaced whole, so send code and language together — read the current code with get_transformer first.
• test_transformer_code runs the code that is stored, so save an edit with update_transformer before testing it.

Recipes:

• Starting point, the same one the platform scaffolds: "from typing import Any\n\ndef transformer(data: Any):\n return data\n"
• Reach a value no key path can: def transformer(data): return data["choices"][0]["message"]["content"].strip()
• Flatten retrieved documents into the list of strings a metric expects: def transformer(data): return [doc["text"] for doc in data["documents"]]
• Parse a stringified payload before reading it: def transformer(data): return json.loads(data)["answer"] — with 'import json' at the top.

Calls POST /v2/transformers/{transformerId}/test-code.

This tool updates a transformer in the Confident AI project. Send at least one field; omit anything you are not changing. codeDefinition is replaced whole, so send code and language together — read the current code with get_transformer first if you are editing rather than rewriting.

Everything already pointing at this transformer picks the new code up on its next run, so an edit changes how existing AI connections and metric collections behave.

Transformer guide:

• A transformer is a piece of Python that reshapes one value, for the cases a plain key path can't express. Its id is what the AI connection fields actualOutputTransformerId, retrievalContextTransformerId, toolsCalledTransformerId and stateTransformerId accept, and what a metric collection's or dataset ingestion task's inputTransformerId and outputTransformerId accept.
• The code must define a function named transformer taking a single argument and returning the transformed value. 'async def' works. A function named transform_response is also accepted, but prefer transformer — it is what the platform's editor scaffolds and what the batch evaluation path wraps.
• The argument is the JSON value being transformed: your endpoint's response body when extracting an AI connection's output, or the input / actual output when it is a metric collection's input / output transformer. Both the argument and the return value must be JSON serializable.
• language is PYTHON; nothing else is supported yet.
• name is unique within the project.
• Omit anything you are not changing. codeDefinition is replaced whole, so send code and language together — read the current code with get_transformer first.
• test_transformer_code runs the code that is stored, so save an edit with update_transformer before testing it.

Recipes:

• Starting point, the same one the platform scaffolds: "from typing import Any\n\ndef transformer(data: Any):\n return data\n"
• Reach a value no key path can: def transformer(data): return data["choices"][0]["message"]["content"].strip()
• Flatten retrieved documents into the list of strings a metric expects: def transformer(data): return [doc["text"] for doc in data["documents"]]
• Parse a stringified payload before reading it: def transformer(data): return json.loads(data)["answer"] — with 'import json' at the top.

Calls PUT /v2/transformers/{transformerId}.

This tool creates a custom vulnerability in the Confident AI project — one the deepteam catalog does not cover — so risk categories can select it like any built-in. To customise a built-in instead, use update_vulnerability: creating one under a catalog name is rejected rather than silently shadowing it.

Note: red teaming through the API is available on the Enterprise plan only.

Calls POST /v2/vulnerabilities.

This tool permanently deletes a vulnerability from the Confident AI project, along with its vulnerability types and the risk category selections that referenced them. Past risk assessment results are kept — they record vulnerabilities by name. This cannot be undone.

Only what the project owns can be deleted: a custom vulnerability is removed outright, while deleting a customised built-in discards the project's copy and reverts it to the catalog entry. A built-in that was never customised has nothing to delete and is rejected.

Calls DELETE /v2/vulnerabilities/{vulnerabilityId}.

This tool fetches one vulnerability from the Confident AI project in full: its vulnerability types, the pass/fail criteria used to judge an attack against it, and the evaluation guidelines and examples that steer that judgement. Use it to inspect the current state before updating — particularly before sending vulnerabilityTypes, evaluationGuidelines or evaluationExamples, each of which replaces the whole list.

Calls GET /v2/vulnerabilities/{vulnerabilityId}.

This tool lists the red-teaming vulnerabilities available to the Confident AI project — the built-in ones from the deepteam catalog together with any custom ones the project has created. A vulnerability is what a risk assessment probes for; its 'vulnerability types' are the specific variants a risk category selects. Use this to discover the ids and names to reference when building risk categories.

Note: red teaming through the API is available on the Enterprise plan only.

Calls GET /v2/vulnerabilities.

This tool updates a vulnerability in the Confident AI project. It works on custom and built-in vulnerabilities alike: updating a built-in stores a copy scoped to this project, which then shadows the catalog entry everywhere the project uses it. Other projects are unaffected.

Send at least one field; omit anything you are not changing, and pass null to clear description. vulnerabilityTypes, evaluationGuidelines and evaluationExamples are each full replacements — fetch the vulnerability with get_vulnerability first and resend every entry it should keep. Types are matched by name, so an unchanged one stays attached to the risk categories that selected it and a renamed one is detached from them.

A built-in cannot be renamed: risk assessment results record vulnerabilities by name, so the catalog name is what past runs are reported under.

Note: red teaming through the API is available on the Enterprise plan only.

Calls PUT /v2/vulnerabilities/{vulnerabilityId}.

This tool executes a widget definition WITHOUT saving it — the recommended way to iterate: preview the widget, check the data is meaningful (non-empty, sane values), adjust, and only then add it to a dashboard with create_dashboard or create_widget. Limits: at most 20 lines, top_k.limit at most 100, time range at most 366 days.

Widget composition guide:

• A widget = chart type + mode + one or more lines (each line is one data series: a data_model + an aggregation, optionally filtered).
• mode TIME_SERIES: values over time. Use LINE/AREA for trends, BAR for volumes, BIG_NUMBER (with bucket_mode RANGE) for a single headline stat.
• mode DIMENSION_SERIES: values grouped by dimension (required), e.g. per model or per trace name. Use BAR/STACKED_BAR/GROUPED_BAR/TABLE and topK to bound how many groups appear.
• Match unit to the aggregation: USD for costs, PERCENT for rates, SECONDS/MILLISECONDS for latency, SCORE for scores, COUNT otherwise.

Recipes:

• LLM cost over time: type LINE, mode TIME_SERIES, unit USD, lines: [{name: 'Cost', data_model: 'LLM_SPAN', aggregation: 'TOTAL_COST'}]
• Top 10 models by usage: type BAR, mode DIMENSION_SERIES, dimension model, unit COUNT, top_k: {limit: 10}, lines: [{name: 'Calls', data_model: 'LLM_SPAN', aggregation: 'COUNT'}]
• Trace error rate: type LINE, mode TIME_SERIES, unit PERCENT, lines: [{name: 'Error rate', data_model: 'TRACE', aggregation: 'ERROR_RATE'}]
• Eval pass rate headline: type BIG_NUMBER, bucket_mode RANGE, unit PERCENT, lines: [{name: 'Pass rate', data_model: 'METRIC_DATA', aggregation: 'PASS_RATE'}]
• p90 latency, one line per trace name: type LINE, mode TIME_SERIES, unit MILLISECONDS, multiple lines each with data_model TRACE, aggregation P90_LATENCY, and a filters set on 'Trace Name'.
• Daily active users: type BAR, mode TIME_SERIES, unit COUNT, lines: [{name: 'Users', data_model: 'TRACE', aggregation: 'UNIQUE_END_USERS'}]

Calls POST /v2/widgets/query.