AI Connections

AI Connections let you run evaluations directly on the platform by connecting to your AI app via an HTTPS endpoint. Instead of writing code, you can trigger evaluations with a click of a button—Confident AI will call your endpoint with data from your goldens and parse the response.

Setting Up an AI Connection

To create an AI connection:

1. Navigate to Project Settings → AI Connections
2. Click New AI Connection
3. Give it a unique identifying name
4. Click Save

Configuration Parameters

There are several parameters you’ll need to configure in order for your AI connection to work.

Name

Give your AI connection a unique name to identify it within your project.

AI App Endpoint

Your AI app must be accessible via an HTTPS endpoint that accepts POST requests and returns a JSON response containing the actual output of your AI app.

Payload

Configure the JSON payload that Confident AI sends to your endpoint. You can customize this to match your API’s expected structure using values from your goldens.

Available variables:

Example payload:

1
{
2
  "input": golden.input,
3
  "context": golden.context,
4
  "conversationalContext": conversationalGolden.context,
5
  "prompts": prompts,
6
  "turns": conversationalGolden.turns
7
}

Use golden.* variables for single-turn evaluations and conversationalGolden.* variables for multi-turn evaluations. See Prompts for details on how to use the prompts dictionary.

Headers

Add any headers required by your endpoint, such as API keys, authentication tokens, or content type specifications. These headers are sent with every request to your AI app.

Prompts

Associate prompt versions with your AI connection. When running evaluations, these prompts will be attributed to each test run, letting you trace results back to the prompts used.

The prompts variable in your payload is a dictionary where each key maps to an object containing alias and version:

1
{
2
  "system": { "alias": "system-prompt", "version": "1.0.0" },
3
  "assistant": { "alias": "assistant-prompt", "version": "2.1.0" }
4
}

Here’s an example of how your Python endpoint might handle the prompts dictionary:

1
from deepeval.prompt import Prompt
2
3
@app.post("/generate")
4
def generate(request: dict):
5
    # Pull different prompt versions using their keys
6
    system_info = request["prompts"]["system"]
7
    assistant_info = request["prompts"]["assistant"]
8
9
    system_prompt = Prompt(alias=system_info["alias"]).pull(version=system_info["version"])
10
    assistant_prompt = Prompt(alias=assistant_info["alias"]).pull(version=assistant_info["version"])
11
12
    # Use the prompts in your generation
13
    response = llm.generate(
14
        system=system_prompt.text,
15
        assistant=assistant_prompt.text,
16
        user=request["input"]
17
    )
18
19
    return {"output": response}

For more details on working with prompts, see Prompt Versioning.

Actual Output Key Path

A list of strings representing the path to the actual_output value in your JSON response. This is required for evaluation to work.

For example, if your endpoint returns:

1
{
2
  "response": {
3
    "output": "Hello, world!"
4
  }
5
}

Set the key path to ["response", "output"].

Retrieval Context Key Path

A list of strings representing the path to the retrieval_context value in your JSON response. This is optional and only needed if you’re using RAG metrics. The value must be a list of strings.

Tool Call Key Path

A list of ToolCall representing the path to the tools_called value in your JSON response. This is optional and only needed if you’re using metrics that require a tool call parameter. The value must be a list of ToolCall.

Testing Your Connection

After configuring your AI connection, click Ping Endpoint to verify everything is set up correctly. You should receive a 200 status response. If not, check the error message and adjust your configuration accordingly.