AI Connections

AI Connections let you run evaluations directly on the platform by connecting to your AI app via an HTTPS endpoint. Instead of writing code, you can trigger evaluations with a click of a button—Confident AI will call your endpoint with data from your goldens and parse the response.

Setting Up an AI Connection

To create an AI connection:

1. Navigate to Project Settings → AI Connections
2. Click New AI Connection
3. Give it a unique identifying name
4. Click Save

Configuration Parameters

There are several parameters you’ll need to configure in order for your AI connection to work.

Name

Give your AI connection a unique name to identify it within your project.

AI App Endpoint

Your AI app must be accessible via an HTTPS endpoint that accepts POST requests and returns any response containing the actual output of your AI app.

You can choose between HTTP Response, HTTP Streaming, and SSE Streaming modes.

• HTTP Response: The endpoint returns a HTTP response containing the actual output.
• HTTP Streaming: The endpoint returns a stream of HTTP responses containing the actual output.
• SSE Streaming: The endpoint returns a stream of SSE events containing the actual output.

Streaming Mode

For SSE Streaming and HTTP Streaming modes, your endpoints can return either strings, JSON objects, or custom formats that can be processed using transformers for each chunks.

["Hello", "World", "!"]

"Hello World!"

Payload

Configure the payload that gets sent to your endpoint when Confident AI calls it. JSON mode lets you map available variables into a JSON structure, while the Code editor lets you write a Python function for conditional logic, data transformation, or full programmatic control over the request body.

1
{
2
  "input": golden.input,
3
  "context": golden.context,
4
  "conversationalContext": conversationalGolden.context,
5
  "prompts": prompts,
6
  "hyperparameters": hyperparameters,
7
  "turns": conversationalGolden.turns
8
}

Headers

Add any custom headers required by your endpoint as key-value pairs, such as API keys or content type specifications. These headers are sent with every request to your AI app.

Authorization

Configure authentication for requests to your AI app endpoint. The Authorization tab has two sections: Secrets Manager and Authentication.

Secrets Manager

A secrets manager lets you securely retrieve authentication credentials at runtime from a cloud vault, instead of storing them directly on the platform.

To enable a secrets manager:

1. Toggle the secrets manager on
2. Select a provider (e.g. Azure Key Vault)
3. Enter your Vault URL (e.g., https://your-vault.vault.azure.net)
4. Enter your Tenant ID, Client ID, and Client Secret to authenticate to the vault

Authentication

Select an authentication type from the dropdown:

Auth0 requires the following fields:

HMAC requires the following fields:

Prompts

Associate prompt versions with your AI connection. When running evaluations, these prompts will be attributed to each test run, letting you trace results back to the prompts used.

The prompts variable in your payload is a dictionary where each key maps to an object containing alias and version:

1
{
2
  "system": { "alias": "system-prompt", "version": "1.0.0" },
3
  "assistant": { "alias": "assistant-prompt", "version": "2.1.0" }
4
}

Here’s an example of how your Python endpoint might handle the prompts dictionary:

1
from deepeval.prompt import Prompt
2
3
@app.post("/generate")
4
def generate(request: dict):
5
    # Pull different prompt versions using their keys
6
    system_info = request["prompts"]["system"]
7
    assistant_info = request["prompts"]["assistant"]
8
9
    system_prompt = Prompt(alias=system_info["alias"]).pull(version=system_info["version"])
10
    assistant_prompt = Prompt(alias=assistant_info["alias"]).pull(version=assistant_info["version"])
11
12
    # Use the prompts in your generation
13
    response = llm.generate(
14
        system=system_prompt.text,
15
        assistant=assistant_prompt.text,
16
        user=request["input"]
17
    )
18
19
    return {"output": response}

For more details on working with prompts, see Prompt Versioning.

Hyperparameters

Define optional hyperparameters as string key-value pairs. These are sent to your endpoint as part of the payload and are also logged in test runs and experiments, making it easy to track which configuration was used for each evaluation.

Hyperparameters are useful for passing model configuration values like temperature, model_name, or max_tokens to your AI app without hardcoding them into your endpoint. Since they’re logged alongside test run and experiment results, you can compare how different hyperparameter values affect evaluation outcomes.

The hyperparameters variable in your payload is a dictionary where both keys and values are strings:

1
{
2
  "temperature": "0.7",
3
  "model": "gpt-4o",
4
  "max_tokens": "1024"
5
}

Here’s an example of how your Python endpoint might use hyperparameters:

1
@app.post("/generate")
2
def generate(request: dict):
3
    hyperparameters = request.get("hyperparameters", {})
4
5
    response = llm.generate(
6
        model=hyperparameters.get("model", "gpt-4o"),
7
        temperature=float(hyperparameters.get("temperature", "0.7")),
8
        max_tokens=int(hyperparameters.get("max_tokens", "1024")),
9
        user=request["input"]
10
    )
11
12
    return {"output": response}

Transformers

Transformers are a powerful way to apply custom logic to the actual output of your AI app. You can use transformers to extract the actual output from a custom response format using custom Python code.

You can add your own transformers by navigating to Project Settings → Transformers and clicking Create Transformer.

Actual Output Key Path

A list of strings or integers representing the path to the actual_output value in your JSON response. Use strings for JSON keys and integers for array indices. This is required for evaluation to work.

For example, if your endpoint returns:

1
{
2
  "response": {
3
    "output": "Hello, world!"
4
  }
5
}

Set the key path to ["response", "output"].

For nested arrays, use integers to specify the array index. For example, if your endpoint returns:

1
{
2
  "response": {
3
    "output": {
4
      "content": [{ "text": "Hello, world!" }]
5
    }
6
  }
7
}

Set the key path to ["response", "output", "content", 0, "text"].

Retrieval Context Key Path

A list of strings or integers representing the path to the retrieval_context value in your JSON response. Use strings for JSON keys and integers for array indices. This is optional and only needed if you’re using RAG metrics. The value must be a list of strings.

For example, if your endpoint returns:

1
{
2
  "response": {
3
    ...
4
    "retrieval_context": ["context1", "context2"]
5
  }
6
}

Set the key path to ["response", "retrieval_context"].

Tool Call Key Path

A list of strings or integers representing the path to the tools_called value in your JSON response. Use strings for JSON keys and integers for array indices. This is optional and only needed if you’re using metrics that require a tool call parameter. The value must be a list of ToolCall.

For example, if your endpoint returns:

1
{
2
  "response": {
3
    ...
4
    "tools_called": [
5
      {
6
        "name": "get_weather",
7
        "description": "Get weather for a location",
8
        "reasoning": "User asked about the weather in San Francisco",
9
        "output": "Sunny, 72°F",
10
        "inputParameters": {"location": "San Francisco"}
11
      }
12
    ]
13
  }
14
}

Set the key path to ["response", "tools_called"].

Request Timeout

Set the maximum time (in seconds) that Confident AI will wait for your endpoint to respond before timing out. This helps prevent evaluations from hanging indefinitely if your AI connection is slow or unresponsive.

• Minimum: 1 second
• Default: 60 seconds

Max Concurrency

Set the maximum number of concurrent requests that Confident AI will send to your endpoint at the same time. This helps prevent overwhelming your AI app during large evaluation runs.

• Minimum: 1
• Default: 20

Max Retries

Set the maximum number of times Confident AI will retry a failed request to your endpoint. This helps handle transient errors without failing the entire evaluation.

• Minimum: 0
• Default: 0

Configure Multiturn State

During multi-turn simulations, Confident AI calls your endpoint once per turn. You can use state to persist information—like a thread ID or session—across turns so your AI app can maintain context throughout the conversation.

On the first turn, the state variable in your payload will be empty since no prior state exists. If your endpoint returns a state object and the state key path successfully extracts it, that state will be included in the state payload variable from the second turn onwards.

Payload

To enable multiturn state, include state in your payload configuration so it gets sent to your endpoint on each turn:

1
{
2
  "input": golden.input,
3
  "state": state
4
}

Here’s an example of how your endpoint might handle state:

1
@app.post("/generate")
2
def generate(request: dict):
3
    state = request.get("state", {})
4
5
    if not state:
6
        thread_id = create_new_thread()
7
    else:
8
        thread_id = state["threadId"]
9
10
    response = llm.generate(
11
        thread_id=thread_id,
12
        user=request["input"]
13
    )
14
15
    return {
16
        "output": response,
17
        "state": {"threadId": thread_id}
18
    }

State Key Path

The state key path works just like the actual output key path—a list of strings or integers representing the path to the state object in your JSON response. This tells Confident AI where to extract state from your endpoint’s response so it can be passed back on the next turn.

For example, if your endpoint returns:

1
{
2
  "output": "Hello! How can I help?",
3
  "state": {
4
    "threadId": "abc-123"
5
  }
6
}

Set the state key path to ["state"].

Linking Test Cases to Traces

For single-turn evaluations, you can link each test case to its corresponding trace for full observability. This is done by including testCaseId in your payload (enabled by default) and passing it to your tracing setup.

Include testCaseId in your payload configuration and ensure your AI connection is configured to accept it.

1
{
2
  "input": golden.input,
3
  "testCaseId": testCaseId
4
}

Then, pass the testCaseId to your tracing implementation:

1
from deepeval.tracing import observe, update_current_trace
2
3
@observe()
4
def llm_app(input: str, test_case_id: str):
5
    update_current_trace(test_case_id=test_case_id)
6
7
    response = llm.generate(user=input)
8
    return response
9
10
@app.post("/generate")
11
def generate(request: dict):
12
    test_case_id = request.get("testCaseId")
13
    response = llm_app(request["input"], test_case_id)
14
    return {"output": response}

Linking Turns to Traces

For multi-turn evaluations, Confident AI calls your endpoint once per turn. Each turn has its own turnId that you can pass to your tracing setup. This links each turn’s trace to the specific turn in the conversation, letting you view traces per-turn from the evaluation results.

Include turnId in your payload configuration:

1
{
2
  "input": golden.input,
3
  "turnId": turnId,
4
  "state": state
5
}

Then, pass the turnId to your tracing implementation:

1
from deepeval.tracing import observe, update_current_trace
2
3
@observe()
4
def llm_app(input: str, turn_id: str):
5
    update_current_trace(turn_id=turn_id)
6
7
    response = llm.generate(user=input)
8
    return response
9
10
@app.post("/generate")
11
def generate(request: dict):
12
    turn_id = request.get("turnId")
13
    response = llm_app(request["input"], turn_id)
14
    return {"output": response}

Linking Risk Assessments to Threads

For multi-turn red-team attack methods (e.g. Linear Jailbreaking, Crescendo Jailbreaking, Tree Jailbreaking, Sequential Jailbreak, Bad Likert Judge), Confident AI calls your endpoint once per turn — every call produces its own trace. To group all of those turn-traces into a single conversation in the assessment view, the platform sends a threadId in the payload. Stamp it onto your traces and the assessment side drawer will open them as one thread.

Include threadId in your payload configuration:

1
{
2
  "input": golden.input,
3
  "threadId": threadId
4
}

Then, pass the threadId to your tracing implementation:

1
from deepeval.tracing import observe, update_current_trace
2
3
@observe()
4
def llm_app(input: str, thread_id: str):
5
    update_current_trace(thread_id=thread_id)
6
7
    response = llm.generate(user=input)
8
    return response
9
10
@app.post("/generate")
11
def generate(request: dict):
12
    thread_id = request.get("threadId")
13
    response = llm_app(request["input"], thread_id)
14
    return {"output": response}

Testing Your Connection

After configuring your AI connection, click Ping Endpoint to verify everything is set up correctly. You should receive a 200 status response. If not, check the error message and adjust your configuration accordingly.