AI Connections

Overview

The Confident AI SDK exposes every AI Connection method on the platform. This page documents how to call these methods in all supported languages. See the introduction to install the SDK and set your API key.

Methods

List AI Connections

Lists the AI connections in your Confident AI project one page at a time, ordered by name. Each connection is returned with just its endpoint and whether it is active; retrieve one by id for its full configuration. Requires the Starter plan or above.

from confident_ai import ConfidentAI

client = ConfidentAI()

result = client.ai_connections.list(page=1, page_size=25)

For async mode, call a_list and await it as shown below:

result = await client.ai_connections.a_list(...)

Parameters

Parameter	Type	Description
`page`	`Optional[int]`	The page to return. Defaults to 1.
`page_size`	`Optional[int]`	The number of results per page, at most 100. Defaults to 25.

import { ConfidentAI } from "confident-ai";

const client = new ConfidentAI();

const result = await client.aiConnections.list({ page: 1, pageSize: 25 });

Parameters

Parameter	Type	Description
`page`	`number`	The page to return. Defaults to 1.
`pageSize`	`number`	The number of results per page, at most 100. Defaults to 25.

Returns

This method returns an object of type AIConnectionList.

Create AI Connection

Registers your LLM application with Confident AI and returns the id of the connection. The endpoint is called once as the connection is created to work out whether it is active, which you read back by retrieving the connection or pinging it. Requires the Starter plan or above.

from confident_ai import ConfidentAI
from confident_ai.ai_connections import AIConnectionResponseMode
from confident_ai.ai_connections import AIConnectionType

client = ConfidentAI()

result = client.ai_connections.create(
    name="Production Chatbot",
    type=AIConnectionType.ENDPOINT,
    endpoint="https://api.example.com/chat",
    response_mode=AIConnectionResponseMode.HTTP_RESPONSE,
    async_response=False,
    timeout=60,
    max_concurrency=5,
    max_retries=3,
    default_num_generations=1,
    headers=[
        {"key": "Authorization", "value": "Bearer YOUR-TOKEN"},
        {"key": "Content-Type", "value": "application/json"}
    ],
    query_params=[{"key": "stream", "value": "false"}],
    payload={"query": "{{input}}"},
    hyperparameters={"model": "gpt-4o", "temperature": 0.2},
    authentication={
        "type": "AUTH0",
        "domain": "acme.us.auth0.com",
        "clientId": "YOUR-CLIENT-ID",
        "clientSecret": "YOUR-CLIENT-SECRET"
    },
    cloud_provider={
        "provider": "AWS",
        "region": "us-east-1",
        "secretName": "chatbot/api-key"
    },
    actual_output_key_path=["choices", 0, "message", "content"],
    retrieval_context_key_path=["retrieval", "documents"],
    tools_called_key_path=["tool_calls"],
    state_key_path=["session", "state"],
    input_token_count_key_path=["usage", "prompt_tokens"],
    output_token_count_key_path=["usage", "completion_tokens"],
    token_cost_key_path=["usage", "cost"],
    actual_output_event="token",
    retrieval_context_event="retrieval",
    tools_called_event="tool_call",
    state_event="state",
    actual_output_accumulate=True,
    actual_output_transformer_id="<TRANSFORMER-ID>",
    retrieval_context_transformer_id="<TRANSFORMER-ID>",
    tools_called_transformer_id="<TRANSFORMER-ID>",
    state_transformer_id="<TRANSFORMER-ID>",
    input_token_count_transformer_id="<TRANSFORMER-ID>",
    output_token_count_transformer_id="<TRANSFORMER-ID>",
    token_cost_transformer_id="<TRANSFORMER-ID>",
    prompts={
        "systemPrompt": {"alias": "customer-support", "label": "production"}
    },
)

For async mode, call a_create and await it as shown below:

result = await client.ai_connections.a_create(...)

Parameters

Parameter	Type	Description
`name`	`str`	Required. The name of the AI connection, unique within the project.
`type`	`Optional[AIConnectionType]`	See AIConnectionType.
`endpoint`	`Optional[str]`	The `https://` URL Confident AI calls to reach your LLM application, or a `wss://` URL when `responseMode` is WEBSOCKET. An AGENT_HANDLER connection needs none. Send null to clear it.
`response_mode`	`Optional[AIConnectionResponseMode]`	How your application replies. Send null to clear it, which reads the answer out of a completed HTTP response body. See AIConnectionResponseMode.
`async_response`	`Optional[bool]`	Whether your application acknowledges the request and posts the result back later instead of answering inline. Only a non-streaming `responseMode` supports this.
`timeout`	`Optional[int]`	How many seconds to wait for your application to answer before giving up on a request. Defaults to 60 when the connection is created. Send null to clear it.
`max_concurrency`	`Optional[int]`	The most requests Confident AI sends to your application at the same time. Send null to leave it unbounded.
`max_retries`	`Optional[int]`	How many times a failed request is retried before the test case is recorded as errored. Send null to clear it.
`default_num_generations`	`Optional[int]`	How many times your application is called per test case, so one unlucky output does not decide the result. At most 50.
`headers`	`Optional[List[AIConnectionKeyValue]]`	The headers sent with every request. The list replaces the stored headers rather than merging into them, so include every header the connection should keep. Send null to clear them. See AIConnectionKeyValue.
`query_params`	`Optional[List[AIConnectionKeyValue]]`	The query parameters appended to every request. The list replaces the stored parameters rather than merging into them. Send null to clear them. See AIConnectionKeyValue.
`payload`	`Optional[Dict[str, Any]]`	The request body template Confident AI sends. Placeholders such as `{{input}}` are filled from the test case, and any key in `prompts` is filled with that prompt's text. Send null to clear it.
`hyperparameters`	`Optional[Dict[str, Any]]`	Free-form settings recorded against every test run made through this connection, so results can be compared across configurations. They are not sent to your application. Send null to clear them.
`authentication`	`Optional[Dict[str, Any]]`	The authentication configuration Confident AI applies when calling your application, such as Auth0, HMAC or Azure AD settings. Its shape follows the scheme you configure, and it is stored as sent and read back as stored. Send null to clear it.
`cloud_provider`	`Optional[Dict[str, Any]]`	The cloud vault configuration Confident AI uses to pull credentials at call time instead of holding them itself. Its shape follows the provider you configure, and it is stored as sent and read back as stored. Send null to clear it.
`actual_output_key_path`	`Optional[List[Union[str, int]]]`	Where your application's answer sits in its response. Each element is an object key or an array index, walked in order, so `["choices", 0, "message", "content"]` reads `choices[0].message.content`. A connection needs this or `actualOutputTransformerId` before it can be used. Send null or an empty list to clear it.
`retrieval_context_key_path`	`Optional[List[Union[str, int]]]`	Where the retrieved context sits in your application's response, walked the same way as `actualOutputKeyPath`. Set it for RAG applications so retrieval metrics have something to score. Send null or an empty list to clear it.
`tools_called_key_path`	`Optional[List[Union[str, int]]]`	Where the list of tools your application called sits in its response, walked the same way as `actualOutputKeyPath`. Set it for agents so tool-use metrics have something to score. Send null or an empty list to clear it.
`state_key_path`	`Optional[List[Union[str, int]]]`	Where the conversation state sits in your application's response, walked the same way as `actualOutputKeyPath`. Confident AI reads it after each simulated turn and sends it back on the next one. Send null or an empty list to clear it.
`input_token_count_key_path`	`Optional[List[Union[str, int]]]`	Where the prompt token count sits in your application's response, walked the same way as `actualOutputKeyPath`. Send null or an empty list to clear it.
`output_token_count_key_path`	`Optional[List[Union[str, int]]]`	Where the completion token count sits in your application's response, walked the same way as `actualOutputKeyPath`. Send null or an empty list to clear it.
`token_cost_key_path`	`Optional[List[Union[str, int]]]`	Where the cost of the call sits in your application's response, walked the same way as `actualOutputKeyPath`. Send null or an empty list to clear it.
`actual_output_event`	`Optional[str]`	For a streaming `responseMode`, the name of the event carrying your application's answer. Confident AI reads the value out of the events with this name instead of out of a completed body, applying `actualOutputKeyPath` to each one. Send null to clear it.
`retrieval_context_event`	`Optional[str]`	For a streaming `responseMode`, the name of the event carrying the retrieved context, read the same way as `actualOutputEvent`. Send null to clear it.
`tools_called_event`	`Optional[str]`	For a streaming `responseMode`, the name of the event carrying the tools called, read the same way as `actualOutputEvent`. Send null to clear it.
`state_event`	`Optional[str]`	For a streaming `responseMode`, the name of the event carrying the conversation state, read the same way as `actualOutputEvent`. Send null to clear it.
`actual_output_accumulate`	`Optional[bool]`	For a streaming `responseMode`, whether the chunks arriving on `actualOutputEvent` are joined into one answer. Set it to false when each event already carries the whole answer and only the last one counts.
`actual_output_transformer_id`	`Optional[str]`	The id of a transformer that extracts the answer by running your code over the response, for shapes a key path cannot reach. Send this or `actualOutputKeyPath`, never both. The transformer must belong to this project. Send null to clear it.
`retrieval_context_transformer_id`	`Optional[str]`	The id of a transformer that extracts the retrieved context. Send this or `retrievalContextKeyPath`, never both. Send null to clear it.
`tools_called_transformer_id`	`Optional[str]`	The id of a transformer that extracts the tools called. Send this or `toolsCalledKeyPath`, never both. Send null to clear it.
`state_transformer_id`	`Optional[str]`	The id of a transformer that extracts the conversation state. Send this or `stateKeyPath`, never both. Send null to clear it.
`input_token_count_transformer_id`	`Optional[str]`	The id of a transformer that extracts the prompt token count. Send this or `inputTokenCountKeyPath`, never both. Send null to clear it.
`output_token_count_transformer_id`	`Optional[str]`	The id of a transformer that extracts the completion token count. Send this or `outputTokenCountKeyPath`, never both. Send null to clear it.
`token_cost_transformer_id`	`Optional[str]`	The id of a transformer that extracts the cost of the call. Send this or `tokenCostKeyPath`, never both. Send null to clear it.
`prompts`	`Optional[Dict[str, AIConnectionPromptRef]]`	The prompts to substitute into the request body, keyed by the placeholder they fill in `payload`. The map replaces the connection's current prompts rather than merging into them. See AIConnectionPromptRef.

import { ConfidentAI } from "confident-ai";
import {
  AIConnectionResponseMode,
  AIConnectionType,
} from "confident-ai/ai-connections";

const client = new ConfidentAI();

const result = await client.aiConnections.create(
  "Production Chatbot",
  {
    type: AIConnectionType.ENDPOINT,
    endpoint: "https://api.example.com/chat",
    responseMode: AIConnectionResponseMode.HTTP_RESPONSE,
    asyncResponse: false,
    timeout: 60,
    maxConcurrency: 5,
    maxRetries: 3,
    defaultNumGenerations: 1,
    headers: [
      { key: "Authorization", value: "Bearer YOUR-TOKEN" },
      { key: "Content-Type", value: "application/json" }
    ],
    queryParams: [{ key: "stream", value: "false" }],
    payload: { query: "{{input}}" },
    hyperparameters: { model: "gpt-4o", temperature: 0.2 },
    authentication: {
      type: "AUTH0",
      domain: "acme.us.auth0.com",
      clientId: "YOUR-CLIENT-ID",
      clientSecret: "YOUR-CLIENT-SECRET"
    },
    cloudProvider: {
      provider: "AWS",
      region: "us-east-1",
      secretName: "chatbot/api-key"
    },
    actualOutputKeyPath: ["choices", 0, "message", "content"],
    retrievalContextKeyPath: ["retrieval", "documents"],
    toolsCalledKeyPath: ["tool_calls"],
    stateKeyPath: ["session", "state"],
    inputTokenCountKeyPath: ["usage", "prompt_tokens"],
    outputTokenCountKeyPath: ["usage", "completion_tokens"],
    tokenCostKeyPath: ["usage", "cost"],
    actualOutputEvent: "token",
    retrievalContextEvent: "retrieval",
    toolsCalledEvent: "tool_call",
    stateEvent: "state",
    actualOutputAccumulate: true,
    actualOutputTransformerId: "<TRANSFORMER-ID>",
    retrievalContextTransformerId: "<TRANSFORMER-ID>",
    toolsCalledTransformerId: "<TRANSFORMER-ID>",
    stateTransformerId: "<TRANSFORMER-ID>",
    inputTokenCountTransformerId: "<TRANSFORMER-ID>",
    outputTokenCountTransformerId: "<TRANSFORMER-ID>",
    tokenCostTransformerId: "<TRANSFORMER-ID>",
    prompts: {
      systemPrompt: { alias: "customer-support", label: "production" }
    }
  },
);

Parameters

Parameter	Type	Description
`name`	`string`	Required. The name of the AI connection, unique within the project.
`type`	`AIConnectionType`	See AIConnectionType.
`endpoint`	`string \| null`	The `https://` URL Confident AI calls to reach your LLM application, or a `wss://` URL when `responseMode` is WEBSOCKET. An AGENT_HANDLER connection needs none. Send null to clear it.
`responseMode`	`AIConnectionResponseMode \| null`	How your application replies. Send null to clear it, which reads the answer out of a completed HTTP response body. See AIConnectionResponseMode.
`asyncResponse`	`boolean`	Whether your application acknowledges the request and posts the result back later instead of answering inline. Only a non-streaming `responseMode` supports this.
`timeout`	`number \| null`	How many seconds to wait for your application to answer before giving up on a request. Defaults to 60 when the connection is created. Send null to clear it.
`maxConcurrency`	`number \| null`	The most requests Confident AI sends to your application at the same time. Send null to leave it unbounded.
`maxRetries`	`number \| null`	How many times a failed request is retried before the test case is recorded as errored. Send null to clear it.
`defaultNumGenerations`	`number`	How many times your application is called per test case, so one unlucky output does not decide the result. At most 50.
`headers`	`AIConnectionKeyValue[] \| null`	The headers sent with every request. The list replaces the stored headers rather than merging into them, so include every header the connection should keep. Send null to clear them. See AIConnectionKeyValue.
`queryParams`	`AIConnectionKeyValue[] \| null`	The query parameters appended to every request. The list replaces the stored parameters rather than merging into them. Send null to clear them. See AIConnectionKeyValue.
`payload`	`Record<string, unknown> \| null`	The request body template Confident AI sends. Placeholders such as `{{input}}` are filled from the test case, and any key in `prompts` is filled with that prompt's text. Send null to clear it.
`hyperparameters`	`Record<string, unknown> \| null`	Free-form settings recorded against every test run made through this connection, so results can be compared across configurations. They are not sent to your application. Send null to clear them.
`authentication`	`Record<string, unknown> \| null`	The authentication configuration Confident AI applies when calling your application, such as Auth0, HMAC or Azure AD settings. Its shape follows the scheme you configure, and it is stored as sent and read back as stored. Send null to clear it.
`cloudProvider`	`Record<string, unknown> \| null`	The cloud vault configuration Confident AI uses to pull credentials at call time instead of holding them itself. Its shape follows the provider you configure, and it is stored as sent and read back as stored. Send null to clear it.
`actualOutputKeyPath`	`(string \| number)[] \| null`	Where your application's answer sits in its response. Each element is an object key or an array index, walked in order, so `["choices", 0, "message", "content"]` reads `choices[0].message.content`. A connection needs this or `actualOutputTransformerId` before it can be used. Send null or an empty list to clear it.
`retrievalContextKeyPath`	`(string \| number)[] \| null`	Where the retrieved context sits in your application's response, walked the same way as `actualOutputKeyPath`. Set it for RAG applications so retrieval metrics have something to score. Send null or an empty list to clear it.
`toolsCalledKeyPath`	`(string \| number)[] \| null`	Where the list of tools your application called sits in its response, walked the same way as `actualOutputKeyPath`. Set it for agents so tool-use metrics have something to score. Send null or an empty list to clear it.
`stateKeyPath`	`(string \| number)[] \| null`	Where the conversation state sits in your application's response, walked the same way as `actualOutputKeyPath`. Confident AI reads it after each simulated turn and sends it back on the next one. Send null or an empty list to clear it.
`inputTokenCountKeyPath`	`(string \| number)[] \| null`	Where the prompt token count sits in your application's response, walked the same way as `actualOutputKeyPath`. Send null or an empty list to clear it.
`outputTokenCountKeyPath`	`(string \| number)[] \| null`	Where the completion token count sits in your application's response, walked the same way as `actualOutputKeyPath`. Send null or an empty list to clear it.
`tokenCostKeyPath`	`(string \| number)[] \| null`	Where the cost of the call sits in your application's response, walked the same way as `actualOutputKeyPath`. Send null or an empty list to clear it.
`actualOutputEvent`	`string \| null`	For a streaming `responseMode`, the name of the event carrying your application's answer. Confident AI reads the value out of the events with this name instead of out of a completed body, applying `actualOutputKeyPath` to each one. Send null to clear it.
`retrievalContextEvent`	`string \| null`	For a streaming `responseMode`, the name of the event carrying the retrieved context, read the same way as `actualOutputEvent`. Send null to clear it.
`toolsCalledEvent`	`string \| null`	For a streaming `responseMode`, the name of the event carrying the tools called, read the same way as `actualOutputEvent`. Send null to clear it.
`stateEvent`	`string \| null`	For a streaming `responseMode`, the name of the event carrying the conversation state, read the same way as `actualOutputEvent`. Send null to clear it.
`actualOutputAccumulate`	`boolean`	For a streaming `responseMode`, whether the chunks arriving on `actualOutputEvent` are joined into one answer. Set it to false when each event already carries the whole answer and only the last one counts.
`actualOutputTransformerId`	`string \| null`	The id of a transformer that extracts the answer by running your code over the response, for shapes a key path cannot reach. Send this or `actualOutputKeyPath`, never both. The transformer must belong to this project. Send null to clear it.
`retrievalContextTransformerId`	`string \| null`	The id of a transformer that extracts the retrieved context. Send this or `retrievalContextKeyPath`, never both. Send null to clear it.
`toolsCalledTransformerId`	`string \| null`	The id of a transformer that extracts the tools called. Send this or `toolsCalledKeyPath`, never both. Send null to clear it.
`stateTransformerId`	`string \| null`	The id of a transformer that extracts the conversation state. Send this or `stateKeyPath`, never both. Send null to clear it.
`inputTokenCountTransformerId`	`string \| null`	The id of a transformer that extracts the prompt token count. Send this or `inputTokenCountKeyPath`, never both. Send null to clear it.
`outputTokenCountTransformerId`	`string \| null`	The id of a transformer that extracts the completion token count. Send this or `outputTokenCountKeyPath`, never both. Send null to clear it.
`tokenCostTransformerId`	`string \| null`	The id of a transformer that extracts the cost of the call. Send this or `tokenCostKeyPath`, never both. Send null to clear it.
`prompts`	`Record<string, AIConnectionPromptRef> \| null`	The prompts to substitute into the request body, keyed by the placeholder they fill in `payload`. The map replaces the connection's current prompts rather than merging into them. See AIConnectionPromptRef.

Returns

This method returns an object of type AIConnectionRef.

Get AI Connection

Retrieves an AI connection by id with its full configuration. The stored headers, queryParams, authentication and cloudProvider are returned exactly as they were saved, so treat the response as carrying credentials.

from confident_ai import ConfidentAI

client = ConfidentAI()

result = client.ai_connections.get(ai_connection_id="<AI-CONNECTION-ID>")

For async mode, call a_get and await it as shown below:

result = await client.ai_connections.a_get(...)

Parameters

Parameter	Type	Description
`ai_connection_id`	`str`	Required. The id of the AI connection.

import { ConfidentAI } from "confident-ai";

const client = new ConfidentAI();

const result = await client.aiConnections.get("<AI-CONNECTION-ID>");

Parameters

Parameter	Type	Description
`aiConnectionId`	`string`	Required. The id of the AI connection.

Returns

This method returns an object of type AIConnection.

Update AI Connection

Changes an AI connection and returns it. Only the fields you send are touched, and headers, queryParams and prompts each replace the stored collection rather than merging into it. Changing anything that affects how your application is called re-tests the connection, so active in the response is the fresh verdict.

from confident_ai import ConfidentAI
from confident_ai.ai_connections import AIConnectionResponseMode
from confident_ai.ai_connections import AIConnectionType

client = ConfidentAI()

result = client.ai_connections.update(
    ai_connection_id="<AI-CONNECTION-ID>",
    name="Production Chatbot",
    type=AIConnectionType.ENDPOINT,
    endpoint="https://api.example.com/chat",
    response_mode=AIConnectionResponseMode.HTTP_RESPONSE,
    async_response=False,
    timeout=60,
    max_concurrency=5,
    max_retries=3,
    default_num_generations=1,
    headers=[
        {"key": "Authorization", "value": "Bearer YOUR-TOKEN"},
        {"key": "Content-Type", "value": "application/json"}
    ],
    query_params=[{"key": "stream", "value": "false"}],
    payload={"query": "{{input}}"},
    hyperparameters={"model": "gpt-4o", "temperature": 0.2},
    authentication={
        "type": "AUTH0",
        "domain": "acme.us.auth0.com",
        "clientId": "YOUR-CLIENT-ID",
        "clientSecret": "YOUR-CLIENT-SECRET"
    },
    cloud_provider={
        "provider": "AWS",
        "region": "us-east-1",
        "secretName": "chatbot/api-key"
    },
    actual_output_key_path=["choices", 0, "message", "content"],
    retrieval_context_key_path=["retrieval", "documents"],
    tools_called_key_path=["tool_calls"],
    state_key_path=["session", "state"],
    input_token_count_key_path=["usage", "prompt_tokens"],
    output_token_count_key_path=["usage", "completion_tokens"],
    token_cost_key_path=["usage", "cost"],
    actual_output_event="token",
    retrieval_context_event="retrieval",
    tools_called_event="tool_call",
    state_event="state",
    actual_output_accumulate=True,
    actual_output_transformer_id="<TRANSFORMER-ID>",
    retrieval_context_transformer_id="<TRANSFORMER-ID>",
    tools_called_transformer_id="<TRANSFORMER-ID>",
    state_transformer_id="<TRANSFORMER-ID>",
    input_token_count_transformer_id="<TRANSFORMER-ID>",
    output_token_count_transformer_id="<TRANSFORMER-ID>",
    token_cost_transformer_id="<TRANSFORMER-ID>",
    prompts={
        "systemPrompt": {"alias": "customer-support", "label": "production"}
    },
)

For async mode, call a_update and await it as shown below:

result = await client.ai_connections.a_update(...)

Parameters

Parameter	Type	Description
`ai_connection_id`	`str`	Required. The id of the AI connection.
`name`	`Optional[str]`	The name of the AI connection, unique within the project.
`type`	`Optional[AIConnectionType]`	See AIConnectionType.
`endpoint`	`Optional[str]`	The `https://` URL Confident AI calls to reach your LLM application, or a `wss://` URL when `responseMode` is WEBSOCKET. An AGENT_HANDLER connection needs none. Send null to clear it.
`response_mode`	`Optional[AIConnectionResponseMode]`	How your application replies. Send null to clear it, which reads the answer out of a completed HTTP response body. See AIConnectionResponseMode.
`async_response`	`Optional[bool]`	Whether your application acknowledges the request and posts the result back later instead of answering inline. Only a non-streaming `responseMode` supports this.
`timeout`	`Optional[int]`	How many seconds to wait for your application to answer before giving up on a request. Defaults to 60 when the connection is created. Send null to clear it.
`max_concurrency`	`Optional[int]`	The most requests Confident AI sends to your application at the same time. Send null to leave it unbounded.
`max_retries`	`Optional[int]`	How many times a failed request is retried before the test case is recorded as errored. Send null to clear it.
`default_num_generations`	`Optional[int]`	How many times your application is called per test case, so one unlucky output does not decide the result. At most 50.
`headers`	`Optional[List[AIConnectionKeyValue]]`	The headers sent with every request. The list replaces the stored headers rather than merging into them, so include every header the connection should keep. Send null to clear them. See AIConnectionKeyValue.
`query_params`	`Optional[List[AIConnectionKeyValue]]`	The query parameters appended to every request. The list replaces the stored parameters rather than merging into them. Send null to clear them. See AIConnectionKeyValue.
`payload`	`Optional[Dict[str, Any]]`	The request body template Confident AI sends. Placeholders such as `{{input}}` are filled from the test case, and any key in `prompts` is filled with that prompt's text. Send null to clear it.
`hyperparameters`	`Optional[Dict[str, Any]]`	Free-form settings recorded against every test run made through this connection, so results can be compared across configurations. They are not sent to your application. Send null to clear them.
`authentication`	`Optional[Dict[str, Any]]`	The authentication configuration Confident AI applies when calling your application, such as Auth0, HMAC or Azure AD settings. Its shape follows the scheme you configure, and it is stored as sent and read back as stored. Send null to clear it.
`cloud_provider`	`Optional[Dict[str, Any]]`	The cloud vault configuration Confident AI uses to pull credentials at call time instead of holding them itself. Its shape follows the provider you configure, and it is stored as sent and read back as stored. Send null to clear it.
`actual_output_key_path`	`Optional[List[Union[str, int]]]`	Where your application's answer sits in its response. Each element is an object key or an array index, walked in order, so `["choices", 0, "message", "content"]` reads `choices[0].message.content`. A connection needs this or `actualOutputTransformerId` before it can be used. Send null or an empty list to clear it.
`retrieval_context_key_path`	`Optional[List[Union[str, int]]]`	Where the retrieved context sits in your application's response, walked the same way as `actualOutputKeyPath`. Set it for RAG applications so retrieval metrics have something to score. Send null or an empty list to clear it.
`tools_called_key_path`	`Optional[List[Union[str, int]]]`	Where the list of tools your application called sits in its response, walked the same way as `actualOutputKeyPath`. Set it for agents so tool-use metrics have something to score. Send null or an empty list to clear it.
`state_key_path`	`Optional[List[Union[str, int]]]`	Where the conversation state sits in your application's response, walked the same way as `actualOutputKeyPath`. Confident AI reads it after each simulated turn and sends it back on the next one. Send null or an empty list to clear it.
`input_token_count_key_path`	`Optional[List[Union[str, int]]]`	Where the prompt token count sits in your application's response, walked the same way as `actualOutputKeyPath`. Send null or an empty list to clear it.
`output_token_count_key_path`	`Optional[List[Union[str, int]]]`	Where the completion token count sits in your application's response, walked the same way as `actualOutputKeyPath`. Send null or an empty list to clear it.
`token_cost_key_path`	`Optional[List[Union[str, int]]]`	Where the cost of the call sits in your application's response, walked the same way as `actualOutputKeyPath`. Send null or an empty list to clear it.
`actual_output_event`	`Optional[str]`	For a streaming `responseMode`, the name of the event carrying your application's answer. Confident AI reads the value out of the events with this name instead of out of a completed body, applying `actualOutputKeyPath` to each one. Send null to clear it.
`retrieval_context_event`	`Optional[str]`	For a streaming `responseMode`, the name of the event carrying the retrieved context, read the same way as `actualOutputEvent`. Send null to clear it.
`tools_called_event`	`Optional[str]`	For a streaming `responseMode`, the name of the event carrying the tools called, read the same way as `actualOutputEvent`. Send null to clear it.
`state_event`	`Optional[str]`	For a streaming `responseMode`, the name of the event carrying the conversation state, read the same way as `actualOutputEvent`. Send null to clear it.
`actual_output_accumulate`	`Optional[bool]`	For a streaming `responseMode`, whether the chunks arriving on `actualOutputEvent` are joined into one answer. Set it to false when each event already carries the whole answer and only the last one counts.
`actual_output_transformer_id`	`Optional[str]`	The id of a transformer that extracts the answer by running your code over the response, for shapes a key path cannot reach. Send this or `actualOutputKeyPath`, never both. The transformer must belong to this project. Send null to clear it.
`retrieval_context_transformer_id`	`Optional[str]`	The id of a transformer that extracts the retrieved context. Send this or `retrievalContextKeyPath`, never both. Send null to clear it.
`tools_called_transformer_id`	`Optional[str]`	The id of a transformer that extracts the tools called. Send this or `toolsCalledKeyPath`, never both. Send null to clear it.
`state_transformer_id`	`Optional[str]`	The id of a transformer that extracts the conversation state. Send this or `stateKeyPath`, never both. Send null to clear it.
`input_token_count_transformer_id`	`Optional[str]`	The id of a transformer that extracts the prompt token count. Send this or `inputTokenCountKeyPath`, never both. Send null to clear it.
`output_token_count_transformer_id`	`Optional[str]`	The id of a transformer that extracts the completion token count. Send this or `outputTokenCountKeyPath`, never both. Send null to clear it.
`token_cost_transformer_id`	`Optional[str]`	The id of a transformer that extracts the cost of the call. Send this or `tokenCostKeyPath`, never both. Send null to clear it.
`prompts`	`Optional[Dict[str, AIConnectionPromptRef]]`	The prompts to substitute into the request body, keyed by the placeholder they fill in `payload`. The map replaces the connection's current prompts rather than merging into them. See AIConnectionPromptRef.

import { ConfidentAI } from "confident-ai";
import {
  AIConnectionResponseMode,
  AIConnectionType,
} from "confident-ai/ai-connections";

const client = new ConfidentAI();

const result = await client.aiConnections.update(
  "<AI-CONNECTION-ID>",
  {
    name: "Production Chatbot",
    type: AIConnectionType.ENDPOINT,
    endpoint: "https://api.example.com/chat",
    responseMode: AIConnectionResponseMode.HTTP_RESPONSE,
    asyncResponse: false,
    timeout: 60,
    maxConcurrency: 5,
    maxRetries: 3,
    defaultNumGenerations: 1,
    headers: [
      { key: "Authorization", value: "Bearer YOUR-TOKEN" },
      { key: "Content-Type", value: "application/json" }
    ],
    queryParams: [{ key: "stream", value: "false" }],
    payload: { query: "{{input}}" },
    hyperparameters: { model: "gpt-4o", temperature: 0.2 },
    authentication: {
      type: "AUTH0",
      domain: "acme.us.auth0.com",
      clientId: "YOUR-CLIENT-ID",
      clientSecret: "YOUR-CLIENT-SECRET"
    },
    cloudProvider: {
      provider: "AWS",
      region: "us-east-1",
      secretName: "chatbot/api-key"
    },
    actualOutputKeyPath: ["choices", 0, "message", "content"],
    retrievalContextKeyPath: ["retrieval", "documents"],
    toolsCalledKeyPath: ["tool_calls"],
    stateKeyPath: ["session", "state"],
    inputTokenCountKeyPath: ["usage", "prompt_tokens"],
    outputTokenCountKeyPath: ["usage", "completion_tokens"],
    tokenCostKeyPath: ["usage", "cost"],
    actualOutputEvent: "token",
    retrievalContextEvent: "retrieval",
    toolsCalledEvent: "tool_call",
    stateEvent: "state",
    actualOutputAccumulate: true,
    actualOutputTransformerId: "<TRANSFORMER-ID>",
    retrievalContextTransformerId: "<TRANSFORMER-ID>",
    toolsCalledTransformerId: "<TRANSFORMER-ID>",
    stateTransformerId: "<TRANSFORMER-ID>",
    inputTokenCountTransformerId: "<TRANSFORMER-ID>",
    outputTokenCountTransformerId: "<TRANSFORMER-ID>",
    tokenCostTransformerId: "<TRANSFORMER-ID>",
    prompts: {
      systemPrompt: { alias: "customer-support", label: "production" }
    }
  },
);

Parameters

Parameter	Type	Description
`aiConnectionId`	`string`	Required. The id of the AI connection.
`name`	`string`	The name of the AI connection, unique within the project.
`type`	`AIConnectionType`	See AIConnectionType.
`endpoint`	`string \| null`	The `https://` URL Confident AI calls to reach your LLM application, or a `wss://` URL when `responseMode` is WEBSOCKET. An AGENT_HANDLER connection needs none. Send null to clear it.
`responseMode`	`AIConnectionResponseMode \| null`	How your application replies. Send null to clear it, which reads the answer out of a completed HTTP response body. See AIConnectionResponseMode.
`asyncResponse`	`boolean`	Whether your application acknowledges the request and posts the result back later instead of answering inline. Only a non-streaming `responseMode` supports this.
`timeout`	`number \| null`	How many seconds to wait for your application to answer before giving up on a request. Defaults to 60 when the connection is created. Send null to clear it.
`maxConcurrency`	`number \| null`	The most requests Confident AI sends to your application at the same time. Send null to leave it unbounded.
`maxRetries`	`number \| null`	How many times a failed request is retried before the test case is recorded as errored. Send null to clear it.
`defaultNumGenerations`	`number`	How many times your application is called per test case, so one unlucky output does not decide the result. At most 50.
`headers`	`AIConnectionKeyValue[] \| null`	The headers sent with every request. The list replaces the stored headers rather than merging into them, so include every header the connection should keep. Send null to clear them. See AIConnectionKeyValue.
`queryParams`	`AIConnectionKeyValue[] \| null`	The query parameters appended to every request. The list replaces the stored parameters rather than merging into them. Send null to clear them. See AIConnectionKeyValue.
`payload`	`Record<string, unknown> \| null`	The request body template Confident AI sends. Placeholders such as `{{input}}` are filled from the test case, and any key in `prompts` is filled with that prompt's text. Send null to clear it.
`hyperparameters`	`Record<string, unknown> \| null`	Free-form settings recorded against every test run made through this connection, so results can be compared across configurations. They are not sent to your application. Send null to clear them.
`authentication`	`Record<string, unknown> \| null`	The authentication configuration Confident AI applies when calling your application, such as Auth0, HMAC or Azure AD settings. Its shape follows the scheme you configure, and it is stored as sent and read back as stored. Send null to clear it.
`cloudProvider`	`Record<string, unknown> \| null`	The cloud vault configuration Confident AI uses to pull credentials at call time instead of holding them itself. Its shape follows the provider you configure, and it is stored as sent and read back as stored. Send null to clear it.
`actualOutputKeyPath`	`(string \| number)[] \| null`	Where your application's answer sits in its response. Each element is an object key or an array index, walked in order, so `["choices", 0, "message", "content"]` reads `choices[0].message.content`. A connection needs this or `actualOutputTransformerId` before it can be used. Send null or an empty list to clear it.
`retrievalContextKeyPath`	`(string \| number)[] \| null`	Where the retrieved context sits in your application's response, walked the same way as `actualOutputKeyPath`. Set it for RAG applications so retrieval metrics have something to score. Send null or an empty list to clear it.
`toolsCalledKeyPath`	`(string \| number)[] \| null`	Where the list of tools your application called sits in its response, walked the same way as `actualOutputKeyPath`. Set it for agents so tool-use metrics have something to score. Send null or an empty list to clear it.
`stateKeyPath`	`(string \| number)[] \| null`	Where the conversation state sits in your application's response, walked the same way as `actualOutputKeyPath`. Confident AI reads it after each simulated turn and sends it back on the next one. Send null or an empty list to clear it.
`inputTokenCountKeyPath`	`(string \| number)[] \| null`	Where the prompt token count sits in your application's response, walked the same way as `actualOutputKeyPath`. Send null or an empty list to clear it.
`outputTokenCountKeyPath`	`(string \| number)[] \| null`	Where the completion token count sits in your application's response, walked the same way as `actualOutputKeyPath`. Send null or an empty list to clear it.
`tokenCostKeyPath`	`(string \| number)[] \| null`	Where the cost of the call sits in your application's response, walked the same way as `actualOutputKeyPath`. Send null or an empty list to clear it.
`actualOutputEvent`	`string \| null`	For a streaming `responseMode`, the name of the event carrying your application's answer. Confident AI reads the value out of the events with this name instead of out of a completed body, applying `actualOutputKeyPath` to each one. Send null to clear it.
`retrievalContextEvent`	`string \| null`	For a streaming `responseMode`, the name of the event carrying the retrieved context, read the same way as `actualOutputEvent`. Send null to clear it.
`toolsCalledEvent`	`string \| null`	For a streaming `responseMode`, the name of the event carrying the tools called, read the same way as `actualOutputEvent`. Send null to clear it.
`stateEvent`	`string \| null`	For a streaming `responseMode`, the name of the event carrying the conversation state, read the same way as `actualOutputEvent`. Send null to clear it.
`actualOutputAccumulate`	`boolean`	For a streaming `responseMode`, whether the chunks arriving on `actualOutputEvent` are joined into one answer. Set it to false when each event already carries the whole answer and only the last one counts.
`actualOutputTransformerId`	`string \| null`	The id of a transformer that extracts the answer by running your code over the response, for shapes a key path cannot reach. Send this or `actualOutputKeyPath`, never both. The transformer must belong to this project. Send null to clear it.
`retrievalContextTransformerId`	`string \| null`	The id of a transformer that extracts the retrieved context. Send this or `retrievalContextKeyPath`, never both. Send null to clear it.
`toolsCalledTransformerId`	`string \| null`	The id of a transformer that extracts the tools called. Send this or `toolsCalledKeyPath`, never both. Send null to clear it.
`stateTransformerId`	`string \| null`	The id of a transformer that extracts the conversation state. Send this or `stateKeyPath`, never both. Send null to clear it.
`inputTokenCountTransformerId`	`string \| null`	The id of a transformer that extracts the prompt token count. Send this or `inputTokenCountKeyPath`, never both. Send null to clear it.
`outputTokenCountTransformerId`	`string \| null`	The id of a transformer that extracts the completion token count. Send this or `outputTokenCountKeyPath`, never both. Send null to clear it.
`tokenCostTransformerId`	`string \| null`	The id of a transformer that extracts the cost of the call. Send this or `tokenCostKeyPath`, never both. Send null to clear it.
`prompts`	`Record<string, AIConnectionPromptRef> \| null`	The prompts to substitute into the request body, keyed by the placeholder they fill in `payload`. The map replaces the connection's current prompts rather than merging into them. See AIConnectionPromptRef.

Returns

This method returns an object of type AIConnection.

Delete AI Connection

Permanently deletes an AI connection. Anything scheduled against it, such as a dataset run or a risk assessment, stops running.

from confident_ai import ConfidentAI

client = ConfidentAI()

result = client.ai_connections.delete(ai_connection_id="<AI-CONNECTION-ID>")

For async mode, call a_delete and await it as shown below:

result = await client.ai_connections.a_delete(...)

Parameters

Parameter	Type	Description
`ai_connection_id`	`str`	Required. The id of the AI connection.

import { ConfidentAI } from "confident-ai";

const client = new ConfidentAI();

const result = await client.aiConnections.delete("<AI-CONNECTION-ID>");

Parameters

Parameter	Type	Description
`aiConnectionId`	`string`	Required. The id of the AI connection.

Returns

This method returns an object of type AIConnectionRef.

Ping AI Connection

Calls your LLM application once with a sample test case and reports what came back, including what each configured key path or transformer managed to extract. A ping that fails is still a 200: read active and error in the body rather than the status code.

from confident_ai import ConfidentAI

client = ConfidentAI()

result = client.ai_connections.ping(
    ai_connection_id="<AI-CONNECTION-ID>",
    multiturn=False,
)

For async mode, call a_ping and await it as shown below:

result = await client.ai_connections.a_ping(...)

Parameters

Parameter	Type	Description
`ai_connection_id`	`str`	Required. The id of the AI connection.
`multiturn`	`Optional[bool]`	Whether to test the connection over a simulated multi- turn conversation instead of a single call. Defaults to false.

import { ConfidentAI } from "confident-ai";

const client = new ConfidentAI();

const result = await client.aiConnections.ping(
  "<AI-CONNECTION-ID>",
  { multiturn: false },
);

Parameters

Parameter	Type	Description
`aiConnectionId`	`string`	Required. The id of the AI connection.
`multiturn`	`boolean`	Whether to test the connection over a simulated multi- turn conversation instead of a single call. Defaults to false.

Returns

This method returns an object of type AIConnectionPingResult.

Types

AIConnection

An LLM application registered with Confident AI: how to call it, and where in its response each value being evaluated lives. headers, queryParams, authentication and cloudProvider come back exactly as they were stored, credentials included.

class AIConnection:
    id: str
    name: str
    type: AIConnectionType
    active: bool
    endpoint: Optional[str]
    response_mode: Optional[AIConnectionResponseMode] = Field(alias="responseMode")
    async_response: bool = Field(alias="asyncResponse")
    timeout: Optional[int]
    max_concurrency: Optional[int] = Field(alias="maxConcurrency")
    max_retries: Optional[int] = Field(alias="maxRetries")
    default_num_generations: Optional[int] = Field(alias="defaultNumGenerations")
    headers: Optional[List[AIConnectionKeyValue]]
    query_params: Optional[List[AIConnectionKeyValue]] = Field(alias="queryParams")
    payload: Optional[Dict[str, Any]]
    payload_mode: AIConnectionPayloadMode = Field(alias="payloadMode")
    hyperparameters: Optional[Dict[str, Any]]
    authentication: Optional[Dict[str, Any]]
    cloud_provider: Optional[Dict[str, Any]] = Field(alias="cloudProvider")
    actual_output_key_path: List[Union[str, int]] = Field(alias="actualOutputKeyPath")
    retrieval_context_key_path: List[Union[str, int]] = Field(alias="retrievalContextKeyPath")
    tools_called_key_path: List[Union[str, int]] = Field(alias="toolsCalledKeyPath")
    state_key_path: List[Union[str, int]] = Field(alias="stateKeyPath")
    input_token_count_key_path: List[Union[str, int]] = Field(alias="inputTokenCountKeyPath")
    output_token_count_key_path: List[Union[str, int]] = Field(alias="outputTokenCountKeyPath")
    token_cost_key_path: List[Union[str, int]] = Field(alias="tokenCostKeyPath")
    actual_output_transformer_id: Optional[str] = Field(alias="actualOutputTransformerId")
    retrieval_context_transformer_id: Optional[str] = Field(alias="retrievalContextTransformerId")
    tools_called_transformer_id: Optional[str] = Field(alias="toolsCalledTransformerId")
    state_transformer_id: Optional[str] = Field(alias="stateTransformerId")
    input_token_count_transformer_id: Optional[str] = Field(alias="inputTokenCountTransformerId")
    output_token_count_transformer_id: Optional[str] = Field(alias="outputTokenCountTransformerId")
    token_cost_transformer_id: Optional[str] = Field(alias="tokenCostTransformerId")
    actual_output_event: Optional[str] = Field(alias="actualOutputEvent")
    retrieval_context_event: Optional[str] = Field(alias="retrievalContextEvent")
    tools_called_event: Optional[str] = Field(alias="toolsCalledEvent")
    state_event: Optional[str] = Field(alias="stateEvent")
    actual_output_accumulate: bool = Field(alias="actualOutputAccumulate")
    prompts: Optional[Dict[str, AIConnectionPromptRef]]

idstrRequired

The id of the AI connection, generated by Confident AI.

Example: "<AI-CONNECTION-ID>"

namestrRequired

The name of the AI connection, unique within the project.

Example: "Production Chatbot"

typeAIConnectionTypeRequired

See AIConnectionType.

activeboolRequired

Whether Confident AI could last reach your application and read an answer out of its response. Computed by Confident AI whenever the configuration changes or the connection is pinged, not writable.

Example: true

endpointOptional[str]Required

The URL Confident AI calls, or null when no endpoint has been configured.

Example: "https://api.example.com/chat"

response_modeOptional[AIConnectionResponseMode]Required

How your application replies, or null when the answer is read out of a completed HTTP response body.

See AIConnectionResponseMode.

async_responseboolRequired

Whether your application acknowledges the request and posts the result back later instead of answering inline.

Example: false

timeoutOptional[int]Required

How many seconds Confident AI waits for your application to answer, or null when no timeout is set.

Example: 60

max_concurrencyOptional[int]Required

The most requests Confident AI sends at the same time, or null when it is unbounded.

Example: 5

max_retriesOptional[int]Required

How many times a failed request is retried, or null when it is not retried.

Example: 3

default_num_generationsOptional[int]Required

How many times your application is called per test case, or null when it is called once.

Example: 1

headersOptional[List[AIConnectionKeyValue]]Required

The headers sent with every request, returned with their values exactly as stored, or null when none are configured.

See AIConnectionKeyValue.

Example: [{"key":"Authorization","value":"Bearer YOUR-TOKEN"},{"key":"Content-Type","value":"application/json"}]

query_paramsOptional[List[AIConnectionKeyValue]]Required

The query parameters appended to every request, returned with their values exactly as stored, or null when none are configured.

See AIConnectionKeyValue.

Example: [{"key":"stream","value":"false"}]

payloadOptional[Dict[str, Any]]Required

The request body template sent to your application, with its placeholders unresolved, or null when none is configured.

Example: {"query":"{{input}}"}

payload_modeAIConnectionPayloadModeRequired

See AIConnectionPayloadMode.

hyperparametersOptional[Dict[str, Any]]Required

The settings recorded against every test run made through this connection, or null when none are configured.

Example: {"model":"gpt-4o","temperature":0.2}

authenticationOptional[Dict[str, Any]]Required

The authentication configuration applied when calling your application, returned exactly as stored, or null when none is configured.

Example: {"type":"AUTH0","domain":"acme.us.auth0.com","clientId":"YOUR-CLIENT-ID","clientSecret":"YOUR-CLIENT-SECRET"}

cloud_providerOptional[Dict[str, Any]]Required

The cloud vault configuration used to pull credentials at call time, returned exactly as stored, or null when none is configured.

Example: {"provider":"AWS","region":"us-east-1","secretName":"chatbot/api-key"}

actual_output_key_pathList[Union[str, int]]Required

The path walked through your application's response to find its answer, each element an object key or an array index. Empty when a transformer extracts the answer instead, or when nothing is configured.

Example: ["choices",0,"message","content"]

retrieval_context_key_pathList[Union[str, int]]Required

The path walked to find the retrieved context. Empty when a transformer extracts it instead, or when nothing is configured.

Example: ["retrieval","documents"]

tools_called_key_pathList[Union[str, int]]Required

The path walked to find the tools your application called. Empty when a transformer extracts them instead, or when nothing is configured.

Example: ["tool_calls"]

state_key_pathList[Union[str, int]]Required

The path walked to find the conversation state carried between simulated turns. Empty when a transformer extracts it instead, or when nothing is configured.

Example: ["session","state"]

input_token_count_key_pathList[Union[str, int]]Required

The path walked to find the prompt token count. Empty when a transformer extracts it instead, or when nothing is configured.

Example: ["usage","prompt_tokens"]

output_token_count_key_pathList[Union[str, int]]Required

The path walked to find the completion token count. Empty when a transformer extracts it instead, or when nothing is configured.

Example: ["usage","completion_tokens"]

token_cost_key_pathList[Union[str, int]]Required

The path walked to find the cost of the call. Empty when a transformer extracts it instead, or when nothing is configured.

Example: ["usage","cost"]

actual_output_transformer_idOptional[str]Required

The id of the transformer that extracts the answer, or null when a key path does it instead.

retrieval_context_transformer_idOptional[str]Required

The id of the transformer that extracts the retrieved context, or null when a key path does it instead.

tools_called_transformer_idOptional[str]Required

The id of the transformer that extracts the tools called, or null when a key path does it instead.

state_transformer_idOptional[str]Required

The id of the transformer that extracts the conversation state, or null when a key path does it instead.

input_token_count_transformer_idOptional[str]Required

The id of the transformer that extracts the prompt token count, or null when a key path does it instead.

output_token_count_transformer_idOptional[str]Required

The id of the transformer that extracts the completion token count, or null when a key path does it instead.

token_cost_transformer_idOptional[str]Required

The id of the transformer that extracts the cost of the call, or null when a key path does it instead.

actual_output_eventOptional[str]Required

For a streaming responseMode, the event whose data carries the answer, or null when the answer is read out of a completed body.

retrieval_context_eventOptional[str]Required

For a streaming responseMode, the event whose data carries the retrieved context, or null when it is read out of a completed body.

tools_called_eventOptional[str]Required

For a streaming responseMode, the event whose data carries the tools called, or null when they are read out of a completed body.

state_eventOptional[str]Required

For a streaming responseMode, the event whose data carries the conversation state, or null when it is read out of a completed body.

actual_output_accumulateboolRequired

For a streaming responseMode, whether the chunks arriving on actualOutputEvent are joined into one answer rather than only the last one being kept.

Example: true

promptsOptional[Dict[str, AIConnectionPromptRef]]Required

The prompts substituted into the request body, keyed by the placeholder they fill, or null when the connection references none.

See AIConnectionPromptRef.

Example: {"systemPrompt":{"alias":"customer-support","label":"production"}}

interface AIConnection {
  id: string;
  name: string;
  type: AIConnectionType;
  active: boolean;
  endpoint: string | null;
  responseMode: AIConnectionResponseMode | null;
  asyncResponse: boolean;
  timeout: number | null;
  maxConcurrency: number | null;
  maxRetries: number | null;
  defaultNumGenerations: number | null;
  headers: AIConnectionKeyValue[] | null;
  queryParams: AIConnectionKeyValue[] | null;
  payload: Record<string, unknown> | null;
  payloadMode: AIConnectionPayloadMode;
  hyperparameters: Record<string, unknown> | null;
  authentication: Record<string, unknown> | null;
  cloudProvider: Record<string, unknown> | null;
  actualOutputKeyPath: (string | number)[];
  retrievalContextKeyPath: (string | number)[];
  toolsCalledKeyPath: (string | number)[];
  stateKeyPath: (string | number)[];
  inputTokenCountKeyPath: (string | number)[];
  outputTokenCountKeyPath: (string | number)[];
  tokenCostKeyPath: (string | number)[];
  actualOutputTransformerId: string | null;
  retrievalContextTransformerId: string | null;
  toolsCalledTransformerId: string | null;
  stateTransformerId: string | null;
  inputTokenCountTransformerId: string | null;
  outputTokenCountTransformerId: string | null;
  tokenCostTransformerId: string | null;
  actualOutputEvent: string | null;
  retrievalContextEvent: string | null;
  toolsCalledEvent: string | null;
  stateEvent: string | null;
  actualOutputAccumulate: boolean;
  prompts: Record<string, AIConnectionPromptRef> | null;
}

idstringRequired

The id of the AI connection, generated by Confident AI.

Example: "<AI-CONNECTION-ID>"

namestringRequired

The name of the AI connection, unique within the project.

Example: "Production Chatbot"

typeAIConnectionTypeRequired

See AIConnectionType.

activebooleanRequired

Whether Confident AI could last reach your application and read an answer out of its response. Computed by Confident AI whenever the configuration changes or the connection is pinged, not writable.

Example: true

endpointstring | nullRequired

The URL Confident AI calls, or null when no endpoint has been configured.

Example: "https://api.example.com/chat"

responseModeAIConnectionResponseMode | nullRequired

How your application replies, or null when the answer is read out of a completed HTTP response body.

See AIConnectionResponseMode.

asyncResponsebooleanRequired

Whether your application acknowledges the request and posts the result back later instead of answering inline.

Example: false

timeoutnumber | nullRequired

How many seconds Confident AI waits for your application to answer, or null when no timeout is set.

Example: 60

maxConcurrencynumber | nullRequired

The most requests Confident AI sends at the same time, or null when it is unbounded.

Example: 5

maxRetriesnumber | nullRequired

How many times a failed request is retried, or null when it is not retried.

Example: 3

defaultNumGenerationsnumber | nullRequired

How many times your application is called per test case, or null when it is called once.

Example: 1

headersAIConnectionKeyValue[] | nullRequired

The headers sent with every request, returned with their values exactly as stored, or null when none are configured.

See AIConnectionKeyValue.

Example: [{"key":"Authorization","value":"Bearer YOUR-TOKEN"},{"key":"Content-Type","value":"application/json"}]

queryParamsAIConnectionKeyValue[] | nullRequired

The query parameters appended to every request, returned with their values exactly as stored, or null when none are configured.

See AIConnectionKeyValue.

Example: [{"key":"stream","value":"false"}]

payloadRecord<string, unknown> | nullRequired

The request body template sent to your application, with its placeholders unresolved, or null when none is configured.

Example: {"query":"{{input}}"}

payloadModeAIConnectionPayloadModeRequired

See AIConnectionPayloadMode.

hyperparametersRecord<string, unknown> | nullRequired

The settings recorded against every test run made through this connection, or null when none are configured.

Example: {"model":"gpt-4o","temperature":0.2}

authenticationRecord<string, unknown> | nullRequired

The authentication configuration applied when calling your application, returned exactly as stored, or null when none is configured.

Example: {"type":"AUTH0","domain":"acme.us.auth0.com","clientId":"YOUR-CLIENT-ID","clientSecret":"YOUR-CLIENT-SECRET"}

cloudProviderRecord<string, unknown> | nullRequired

The cloud vault configuration used to pull credentials at call time, returned exactly as stored, or null when none is configured.

Example: {"provider":"AWS","region":"us-east-1","secretName":"chatbot/api-key"}

actualOutputKeyPath(string | number)[]Required

Example: ["choices",0,"message","content"]

retrievalContextKeyPath(string | number)[]Required

The path walked to find the retrieved context. Empty when a transformer extracts it instead, or when nothing is configured.

Example: ["retrieval","documents"]

toolsCalledKeyPath(string | number)[]Required

The path walked to find the tools your application called. Empty when a transformer extracts them instead, or when nothing is configured.

Example: ["tool_calls"]

stateKeyPath(string | number)[]Required

The path walked to find the conversation state carried between simulated turns. Empty when a transformer extracts it instead, or when nothing is configured.

Example: ["session","state"]

inputTokenCountKeyPath(string | number)[]Required

The path walked to find the prompt token count. Empty when a transformer extracts it instead, or when nothing is configured.

Example: ["usage","prompt_tokens"]

outputTokenCountKeyPath(string | number)[]Required

The path walked to find the completion token count. Empty when a transformer extracts it instead, or when nothing is configured.

Example: ["usage","completion_tokens"]

tokenCostKeyPath(string | number)[]Required

The path walked to find the cost of the call. Empty when a transformer extracts it instead, or when nothing is configured.

Example: ["usage","cost"]

actualOutputTransformerIdstring | nullRequired

The id of the transformer that extracts the answer, or null when a key path does it instead.

retrievalContextTransformerIdstring | nullRequired

The id of the transformer that extracts the retrieved context, or null when a key path does it instead.

toolsCalledTransformerIdstring | nullRequired

The id of the transformer that extracts the tools called, or null when a key path does it instead.

stateTransformerIdstring | nullRequired

The id of the transformer that extracts the conversation state, or null when a key path does it instead.

inputTokenCountTransformerIdstring | nullRequired

The id of the transformer that extracts the prompt token count, or null when a key path does it instead.

outputTokenCountTransformerIdstring | nullRequired

The id of the transformer that extracts the completion token count, or null when a key path does it instead.

tokenCostTransformerIdstring | nullRequired

The id of the transformer that extracts the cost of the call, or null when a key path does it instead.

actualOutputEventstring | nullRequired

For a streaming responseMode, the event whose data carries the answer, or null when the answer is read out of a completed body.

retrievalContextEventstring | nullRequired

For a streaming responseMode, the event whose data carries the retrieved context, or null when it is read out of a completed body.

toolsCalledEventstring | nullRequired

For a streaming responseMode, the event whose data carries the tools called, or null when they are read out of a completed body.

stateEventstring | nullRequired

For a streaming responseMode, the event whose data carries the conversation state, or null when it is read out of a completed body.

actualOutputAccumulatebooleanRequired

For a streaming responseMode, whether the chunks arriving on actualOutputEvent are joined into one answer rather than only the last one being kept.

Example: true

promptsRecord<string, AIConnectionPromptRef> | nullRequired

The prompts substituted into the request body, keyed by the placeholder they fill, or null when the connection references none.

See AIConnectionPromptRef.

Example: {"systemPrompt":{"alias":"customer-support","label":"production"}}

AIConnectionKeyValue

One header or query parameter Confident AI sends with every request to your application.

class AIConnectionKeyValue:
    key: str
    value: str

keystrRequired

The header or query parameter name.

Example: "Authorization"

valuestrRequired

The value sent under this name on every request. It is stored as sent and read back as stored.

Example: "Bearer YOUR-TOKEN"

interface AIConnectionKeyValue {
  key: string;
  value: string;
}

keystringRequired

The header or query parameter name.

Example: "Authorization"

valuestringRequired

The value sent under this name on every request. It is stored as sent and read back as stored.

Example: "Bearer YOUR-TOKEN"

AIConnectionList

One page of AI connections, with the total across all pages.

class AIConnectionList:
    ai_connections: List[AIConnectionSummary] = Field(alias="aiConnections")
    total_ai_connections: int = Field(alias="totalAIConnections")
    page: int
    page_size: int = Field(alias="pageSize")

ai_connectionsList[AIConnectionSummary]Required

The AI connections for the current page, ordered by name.

See AIConnectionSummary.

total_ai_connectionsintRequired

The total number of AI connections in this project.

Example: 3

pageintRequired

The page this response covers.

Example: 1

page_sizeintRequired

The number of AI connections per page.

Example: 25

interface AIConnectionList {
  aiConnections: AIConnectionSummary[];
  totalAIConnections: number;
  page: number;
  pageSize: number;
}

aiConnectionsAIConnectionSummary[]Required

The AI connections for the current page, ordered by name.

See AIConnectionSummary.

totalAIConnectionsnumberRequired

The total number of AI connections in this project.

Example: 3

pagenumberRequired

The page this response covers.

Example: 1

pageSizenumberRequired

The number of AI connections per page.

Example: 25

AIConnectionPayloadMode

Where the request body sent to your application comes from: JSON when it is the stored payload template, CODE when it is built by a code definition authored on the Confident AI platform. Sending payload through the API sets this to JSON.

class AIConnectionPayloadMode(Enum):
    JSON = "JSON"
    CODE = "CODE"

enum AIConnectionPayloadMode {
  JSON = "JSON",
  CODE = "CODE",
}

JSON · CODE

AIConnectionPingResult

What one test call to your application produced: whether it answered, what it sent back, and what Confident AI managed to extract from it. A ping that failed is reported here with active false rather than as an error status.

class AIConnectionPingResult:
    active: bool
    error: Optional[str]
    status_code: Optional[int] = Field(alias="statusCode")
    time_taken: Optional[float] = Field(alias="timeTaken")
    request: Optional[Dict[str, Any]]
    response: Optional[Dict[str, Any]]
    raw_response: Optional[str] = Field(alias="rawResponse")
    actual_output: Optional[str] = Field(alias="actualOutput")
    retrieval_context: Optional[List[str]] = Field(alias="retrievalContext")
    tools_called: Optional[List[Dict[str, Any]]] = Field(alias="toolsCalled")
    state: Optional[Dict[str, Any]]
    input_token_count: Optional[float] = Field(alias="inputTokenCount")
    output_token_count: Optional[float] = Field(alias="outputTokenCount")
    token_cost: Optional[float] = Field(alias="tokenCost")
    invalid_actual_output: bool = Field(alias="invalidActualOutput")
    invalid_retrieval_context: bool = Field(alias="invalidRetrievalContext")
    invalid_tools_called: bool = Field(alias="invalidToolsCalled")
    invalid_state: bool = Field(alias="invalidState")
    invalid_input_token_count: bool = Field(alias="invalidInputTokenCount")
    invalid_output_token_count: bool = Field(alias="invalidOutputTokenCount")
    invalid_token_cost: bool = Field(alias="invalidTokenCost")

activeboolRequired

Whether your application answered and Confident AI could read the configured values out of its response. This verdict replaces the connection's stored active.

Example: true

errorOptional[str]Required

Why the ping failed, or null when it succeeded. A failed ping is still a 200 response with active false, so read this rather than the status code.

status_codeOptional[int]Required

The HTTP status your application returned: 408 when it timed out, 500 when the call could not be made at all, and null when no call was attempted.

Example: 200

time_takenOptional[float]Required

How long the call took in seconds, or null when no call was attempted.

Example: 1.42

requestOptional[Dict[str, Any]]Required

The body that was sent, with the payload placeholders and prompts resolved, or null when no call was attempted.

Example: {"query":"How tall is Mount Everest?"}

responseOptional[Dict[str, Any]]Required

Your application's parsed response body. This is where to look for the real response shape behind a key path that read nothing, and it carries the reason when the call itself failed.

Example: {"choices":[{"message":{"content":"Mount Everest is 8,848 metres tall."}}]}

raw_responseOptional[str]Required

The unparsed response body, for applications that do not answer with JSON. Null when the response parsed.

actual_outputOptional[str]Required

What actualOutputKeyPath or the actual output transformer pulled out of the response. Check it to confirm the connection reads the field you expect.

Example: "Mount Everest is 8,848 metres tall."

retrieval_contextOptional[List[str]]Required

What was pulled out as the retrieved context, or null when the connection extracts none.

tools_calledOptional[List[Dict[str, Any]]]Required

What was pulled out as the tools called, or null when the connection extracts none.

stateOptional[Dict[str, Any]]Required

What was pulled out as the conversation state, or null when the connection extracts none.

input_token_countOptional[float]Required

What was pulled out as the prompt token count, or null when the connection extracts none.

Example: 18

output_token_countOptional[float]Required

What was pulled out as the completion token count, or null when the connection extracts none.

Example: 9

token_costOptional[float]Required

What was pulled out as the cost of the call, or null when the connection extracts none.

Example: 0.002

invalid_actual_outputboolRequired

Whether the answer was found at its key path but is not a string, which means the path points at the wrong field.

Example: false

invalid_retrieval_contextboolRequired

Whether the retrieved context was found but is not a list of strings.

Example: false

invalid_tools_calledboolRequired

Whether the tools called were found but are not a list of tool calls.

Example: false

invalid_stateboolRequired

Whether the conversation state was found but could not be read.

Example: false

invalid_input_token_countboolRequired

Whether the prompt token count was found but is not a number.

Example: false

invalid_output_token_countboolRequired

Whether the completion token count was found but is not a number.

Example: false

invalid_token_costboolRequired

Whether the cost of the call was found but is not a number.

Example: false

interface AIConnectionPingResult {
  active: boolean;
  error: string | null;
  statusCode: number | null;
  timeTaken: number | null;
  request: Record<string, unknown> | null;
  response: Record<string, unknown> | null;
  rawResponse: string | null;
  actualOutput: string | null;
  retrievalContext: string[] | null;
  toolsCalled: Record<string, unknown>[] | null;
  state: Record<string, unknown> | null;
  inputTokenCount: number | null;
  outputTokenCount: number | null;
  tokenCost: number | null;
  invalidActualOutput: boolean;
  invalidRetrievalContext: boolean;
  invalidToolsCalled: boolean;
  invalidState: boolean;
  invalidInputTokenCount: boolean;
  invalidOutputTokenCount: boolean;
  invalidTokenCost: boolean;
}

activebooleanRequired

Whether your application answered and Confident AI could read the configured values out of its response. This verdict replaces the connection's stored active.

Example: true

errorstring | nullRequired

Why the ping failed, or null when it succeeded. A failed ping is still a 200 response with active false, so read this rather than the status code.

statusCodenumber | nullRequired

The HTTP status your application returned: 408 when it timed out, 500 when the call could not be made at all, and null when no call was attempted.

Example: 200

timeTakennumber | nullRequired

How long the call took in seconds, or null when no call was attempted.

Example: 1.42

requestRecord<string, unknown> | nullRequired

The body that was sent, with the payload placeholders and prompts resolved, or null when no call was attempted.

Example: {"query":"How tall is Mount Everest?"}

responseRecord<string, unknown> | nullRequired

Your application's parsed response body. This is where to look for the real response shape behind a key path that read nothing, and it carries the reason when the call itself failed.

Example: {"choices":[{"message":{"content":"Mount Everest is 8,848 metres tall."}}]}

rawResponsestring | nullRequired

The unparsed response body, for applications that do not answer with JSON. Null when the response parsed.

actualOutputstring | nullRequired

What actualOutputKeyPath or the actual output transformer pulled out of the response. Check it to confirm the connection reads the field you expect.

Example: "Mount Everest is 8,848 metres tall."

retrievalContextstring[] | nullRequired

What was pulled out as the retrieved context, or null when the connection extracts none.

toolsCalledRecord<string, unknown>[] | nullRequired

What was pulled out as the tools called, or null when the connection extracts none.

stateRecord<string, unknown> | nullRequired

What was pulled out as the conversation state, or null when the connection extracts none.

inputTokenCountnumber | nullRequired

What was pulled out as the prompt token count, or null when the connection extracts none.

Example: 18

outputTokenCountnumber | nullRequired

What was pulled out as the completion token count, or null when the connection extracts none.

Example: 9

tokenCostnumber | nullRequired

What was pulled out as the cost of the call, or null when the connection extracts none.

Example: 0.002

invalidActualOutputbooleanRequired

Whether the answer was found at its key path but is not a string, which means the path points at the wrong field.

Example: false

invalidRetrievalContextbooleanRequired

Whether the retrieved context was found but is not a list of strings.

Example: false

invalidToolsCalledbooleanRequired

Whether the tools called were found but are not a list of tool calls.

Example: false

invalidStatebooleanRequired

Whether the conversation state was found but could not be read.

Example: false

invalidInputTokenCountbooleanRequired

Whether the prompt token count was found but is not a number.

Example: false

invalidOutputTokenCountbooleanRequired

Whether the completion token count was found but is not a number.

Example: false

invalidTokenCostbooleanRequired

Whether the cost of the call was found but is not a number.

Example: false

AIConnectionPromptRef

Which prompt to substitute into the payload, and how to pick the text of it. Name the prompt by alias, then choose it with exactly one of version, label, branch or hash.

AIConnectionPromptRef = Union[
    AIConnectionPromptVersionRef,
    AIConnectionPromptLabelRef,
    AIConnectionPromptBranchRef,
    AIConnectionPromptCommitRef,
]

type AIConnectionPromptRef =
  | AIConnectionPromptVersionRef
  | AIConnectionPromptLabelRef
  | AIConnectionPromptBranchRef
  | AIConnectionPromptCommitRef;

A AIConnectionPromptRef is one of the shapes below. Send the fields of one of them, never a mix of both.

A prompt pinned to one of its published versions.

class AIConnectionPromptVersionRef:
    alias: str
    version: str

aliasstrRequired

The alias of the prompt in this project.

Example: "customer-support"

versionstrRequired

The version of the prompt to send, pinned exactly.

Example: "00.00.01"

interface AIConnectionPromptVersionRef {
  alias: string;
  version: string;
}

aliasstringRequired

The alias of the prompt in this project.

Example: "customer-support"

versionstringRequired

The version of the prompt to send, pinned exactly.

Example: "00.00.01"

A prompt followed through one of its labels.

class AIConnectionPromptLabelRef:
    alias: str
    label: str

aliasstrRequired

The alias of the prompt in this project.

Example: "customer-support"

labelstrRequired

The label on the prompt to follow. Whatever the label points at when the connection is called is what gets sent.

Example: "production"

interface AIConnectionPromptLabelRef {
  alias: string;
  label: string;
}

aliasstringRequired

The alias of the prompt in this project.

Example: "customer-support"

labelstringRequired

The label on the prompt to follow. Whatever the label points at when the connection is called is what gets sent.

Example: "production"

A prompt followed through the head commit of one branch.

class AIConnectionPromptBranchRef:
    alias: str
    branch: str

aliasstrRequired

The alias of the prompt in this project.

Example: "customer-support"

branchstrRequired

The branch of the prompt to follow. The branch's head commit is what gets sent, so it moves as the branch moves.

Example: "main"

interface AIConnectionPromptBranchRef {
  alias: string;
  branch: string;
}

aliasstringRequired

The alias of the prompt in this project.

Example: "customer-support"

branchstringRequired

The branch of the prompt to follow. The branch's head commit is what gets sent, so it moves as the branch moves.

Example: "main"

A prompt pinned to one of its commits.

class AIConnectionPromptCommitRef:
    alias: str
    hash: str

aliasstrRequired

The alias of the prompt in this project.

Example: "customer-support"

hashstrRequired

The hash of the prompt commit to send, pinned exactly.

Example: "bab04ce"

interface AIConnectionPromptCommitRef {
  alias: string;
  hash: string;
}

aliasstringRequired

The alias of the prompt in this project.

Example: "customer-support"

hashstringRequired

The hash of the prompt commit to send, pinned exactly.

Example: "bab04ce"

AIConnectionRef

A reference to an AI connection by its id.

class AIConnectionRef:
    id: str

idstrRequired

The id of the AI connection, generated by Confident AI.

Example: "<AI-CONNECTION-ID>"

interface AIConnectionRef {
  id: string;
}

idstringRequired

The id of the AI connection, generated by Confident AI.

Example: "<AI-CONNECTION-ID>"

AIConnectionResponseMode

How your application replies. HTTP_RESPONSE returns one completed body. SSE_STREAMING, HTTP_STREAMING and WEBSOCKET stream the answer, so the values are read from named events as they arrive rather than out of a finished body, and WEBSOCKET requires a wss:// endpoint. PHONE and SIP are voice connections placed over a telephony provider.

class AIConnectionResponseMode(Enum):
    HTTP_RESPONSE = "HTTP_RESPONSE"
    SSE_STREAMING = "SSE_STREAMING"
    HTTP_STREAMING = "HTTP_STREAMING"
    WEBSOCKET = "WEBSOCKET"
    PHONE = "PHONE"
    SIP = "SIP"
    WEBRTC = "WEBRTC"

enum AIConnectionResponseMode {
  HTTP_RESPONSE = "HTTP_RESPONSE",
  SSE_STREAMING = "SSE_STREAMING",
  HTTP_STREAMING = "HTTP_STREAMING",
  WEBSOCKET = "WEBSOCKET",
  PHONE = "PHONE",
  SIP = "SIP",
  WEBRTC = "WEBRTC",
}

HTTP_RESPONSE · SSE_STREAMING · HTTP_STREAMING · WEBSOCKET · PHONE · SIP · WEBRTC

AIConnectionSummary

An AI connection as it appears in a list: enough to pick one out. Retrieve it by id for its request, response and authentication configuration.

class AIConnectionSummary:
    id: str
    name: str
    endpoint: Optional[str]
    active: bool

idstrRequired

The id of the AI connection, generated by Confident AI.

Example: "<AI-CONNECTION-ID>"

namestrRequired

The name of the AI connection, unique within the project.

Example: "Production Chatbot"

endpointOptional[str]Required

The URL Confident AI calls, or null when no endpoint has been configured.

Example: "https://api.example.com/chat"

activeboolRequired

Whether Confident AI could last reach your application and read an answer out of its response. Computed by Confident AI, not writable.

Example: true

interface AIConnectionSummary {
  id: string;
  name: string;
  endpoint: string | null;
  active: boolean;
}

idstringRequired

The id of the AI connection, generated by Confident AI.

Example: "<AI-CONNECTION-ID>"

namestringRequired

The name of the AI connection, unique within the project.

Example: "Production Chatbot"

endpointstring | nullRequired

The URL Confident AI calls, or null when no endpoint has been configured.

Example: "https://api.example.com/chat"

activebooleanRequired

Whether Confident AI could last reach your application and read an answer out of its response. Computed by Confident AI, not writable.

Example: true

AIConnectionType

How Confident AI reaches your LLM application: ENDPOINT calls the endpoint you registered, RELAY_ENDPOINT calls it through the Confident AI relay so it never leaves your network boundary, and AGENT_HANDLER expects your own runner to pull work rather than being called, so it needs no endpoint. Defaults to ENDPOINT.

class AIConnectionType(Enum):
    ENDPOINT = "ENDPOINT"
    RELAY_ENDPOINT = "RELAY_ENDPOINT"
    AGENT_HANDLER = "AGENT_HANDLER"

enum AIConnectionType {
  ENDPOINT = "ENDPOINT",
  RELAY_ENDPOINT = "RELAY_ENDPOINT",
  AGENT_HANDLER = "AGENT_HANDLER",
}

ENDPOINT · RELAY_ENDPOINT · AGENT_HANDLER

Building a production pipeline?Design a scalable API workflow for evals, datasets, traces, and promptsTalk to an engineer

Last updated on September 30, 2026

Built byConfident AI