AI Connections

AI Connections let you run evaluations directly on the platform by connecting to your AI app via an HTTPS endpoint. Instead of writing code, you can trigger evaluations with a click of a button—Confident AI will call your endpoint with data from your goldens and parse the response.

Setting Up an AI Connection

To create an AI connection:

1. Navigate to Project Settings → AI Connections
2. Click New AI Connection
3. Give it a unique identifying name
4. Click Save

Configuring Your Endpoint

Point your AI connection at your AI app’s HTTPS endpoint. It must accept POST requests and return a response containing the actual output of your AI app.

Choose a response mode based on how your endpoint responds:

• HTTP Response: returns a single response containing the actual output (default).
• HTTP Streaming: returns a stream of newline-delimited chunks.
• SSE Streaming: returns a stream of Server-Sent Events.

For streaming endpoints, see Streaming to configure chunk formats, SSE event names, and accumulate mode.

Payload

The payload is the request body Confident AI sends to your endpoint when it calls it. JSON mode lets you map available variables into a JSON structure, while the Code editor lets you write a Python function for conditional logic, data transformation, or full programmatic control over the request body.

JSON

Code

JSON mode lets you define a payload using available variables. You can nest values to match your endpoint’s expected structure.

Available variables:

Variable	Description	Type
golden.input	The input from your golden	string
golden.actual_output	The actual output from your golden	string
golden.expected_output	The expected output from your golden	string
golden.retrieval_context	The retrieval context from your golden	string[]
golden.context	The context from your golden	string[]
golden.expected_tools	The expected tools from your golden	ToolCall[]
golden.tools_called	The tools called from your golden	ToolCall[]
golden.additional_metadata	Additional metadata from your golden	object
conversationalGolden.turns	Turn history for multi-turn evals	Turn[]
conversationalGolden.context	Context for conversational goldens	string[]
conversationalGolden.scenario	Scenario for conversational goldens	string
conversationalGolden.expected_outcome	Expected outcome for conversational goldens	string
conversationalGolden.user_description	User description for conversational goldens	string
conversationalGolden.additional_metadata	Additional metadata for conversational goldens	object
prompts	A dictionary of prompts	object
hyperparameters	A dictionary of hyperparameter key-value pairs	object
testCaseId	Unique identifier for linking traces to test cases	string
turnId	Unique identifier for linking traces to turns	string
state	An object to keep state for multi-turn simulations	object

Use golden.* variables for single-turn evaluations and conversationalGolden.* variables for multi-turn evaluations. See Prompts for details on how to use the prompts dictionary, and Hyperparameters for passing hyperparameters to your endpoint.

Example payload:

1
{
2
  "input": golden.input,
3
  "context": golden.context,
4
  "conversationalContext": conversationalGolden.context,
5
  "prompts": prompts,
6
  "hyperparameters": hyperparameters,
7
  "turns": conversationalGolden.turns
8
}

The custom payload feature lets you structure the request to match your existing API contract—no need to modify your AI app to accept a specific format.

Output Parsing

Once your endpoint returns a response, Confident AI needs to know how to pull the relevant values out of it. Use key paths to point at specific values in your JSON response, or a transformer when you need custom logic to extract them:

• Actual Output Key Path: where to find the actual output (required)
• Retrieval Context Key Path: where to find the retrieval context (optional, for RAG metrics)
• Tool Call Key Path: where to find the tools called (optional, for tool-related metrics)

Actual Output Key Path

A list of strings or integers representing the path to the actual_output value in your JSON response. Use strings for JSON keys and integers for array indices. This is required for evaluation to work.

For example, if your endpoint returns:

1
{
2
  "response": {
3
    "output": "Hello, world!"
4
  }
5
}

Set the key path to ["response", "output"].

For nested arrays, use integers to specify the array index. For example, if your endpoint returns:

1
{
2
  "response": {
3
    "output": {
4
      "content": [{ "text": "Hello, world!" }]
5
    }
6
  }
7
}

Set the key path to ["response", "output", "content", 0, "text"].

Retrieval Context Key Path

A list of strings or integers representing the path to the retrieval_context value in your JSON response. Use strings for JSON keys and integers for array indices. This is optional and only needed if you’re using RAG metrics. The value must be a list of strings.

For example, if your endpoint returns:

1
{
2
  "response": {
3
    ...
4
    "retrieval_context": ["context1", "context2"]
5
  }
6
}

Set the key path to ["response", "retrieval_context"].

Tool Call Key Path

A list of strings or integers representing the path to the tools_called value in your JSON response. Use strings for JSON keys and integers for array indices. This is optional and only needed if you’re using metrics that require a tool call parameter. The value must be a list of ToolCall.

For example, if your endpoint returns:

1
{
2
  "response": {
3
    ...
4
    "tools_called": [
5
      {
6
        "name": "get_weather",
7
        "description": "Get weather for a location",
8
        "reasoning": "User asked about the weather in San Francisco",
9
        "output": "Sunny, 72°F",
10
        "inputParameters": {"location": "San Francisco"}
11
      }
12
    ]
13
  }
14
}

Set the key path to ["response", "tools_called"].

Transformers

When a key path isn’t enough—for example, your endpoint returns a non-standard format that needs custom logic—use a transformer to extract the actual output with your own Python code.

Switch any parser from JSON Key Path to Transformer to select a transformer instead of a key path:

Add your own transformers by navigating to Project Settings → Transformers and clicking Create Transformer. See Transformers for details.

Headers

Add any custom headers your endpoint requires as key-value pairs—such as API keys, bearer tokens, or a Content-Type. Whatever you add here is sent with every request Confident AI makes to your AI app.

Common headers you might set:

• Authorization — a static API key or bearer token (e.g. Bearer sk-...)
• Content-Type — the format of the request body (e.g. application/json)
• A custom header your endpoint expects (e.g. X-API-Key)

Testing Your Connection

Click Ping Endpoint to verify everything is set up correctly. You should receive a 200 status response—if not, check the error message and adjust your configuration accordingly.

✅ Done. Your AI connection is ready to run evaluations.

Next Steps

Now that your AI connection is set up, dive into the pieces that make it production-ready: