Set Up Long-Running AI Connections

Configure your AI connection

If you haven’t already, create an AI connection under Project Settings → AI Connections and point it at your endpoint. See AI Connections for the full setup.

The one thing that matters for long-running mode: your payload must include testCaseId, since your agent needs to echo it back when posting results. In JSON payload mode, map the testCaseId variable into your request body:

1
{
2
  "input": golden.input,
3
  "testCaseId": testCaseId
4
}

In Code mode, generate_payload receives testCaseId as a parameter — include it in the returned dict the same way.

Toggle Async Responses

Open your AI connection’s General tab and switch on Async Responses.

Once enabled, the status text under the toggle changes to “Results are posted back via the Evals API results endpoint” — Confident AI will now close the connection after dispatching each request instead of waiting for an output.

Acknowledge fast, work in the background

Your endpoint should return a 2xx immediately and hand the actual work off to a background job. Confident AI treats the acknowledgement as “request received” — nothing in the response body is parsed.

1
from fastapi import BackgroundTasks, FastAPI
2
3
app = FastAPI()
4
5
@app.post("/generate")
6
async def generate(request: dict, background_tasks: BackgroundTasks):
7
    background_tasks.add_task(run_agent, request["input"], request["testCaseId"])
8
    return {"status": "accepted"}

Click Ping Endpoint on the connection to verify — for async connections, a successful ping only checks that your endpoint acknowledges the request.

Run an evaluation

Trigger a single-turn evaluation with a dataset and select your async AI connection as the output generation method. The evaluate dialog shows a notice confirming that this connection responds asynchronously and that results must be posted back through the public endpoint.

The test run is created immediately and stays in progress while it waits for results.

Post results back for evaluation

When your agent finishes a test case, post its result to the results endpoint using the testCaseId from that request’s payload, authenticated with your Project API Key. The SDKs read your key from CONFIDENT_API_KEY (set it via deepeval login or the environment).

1
from deepeval import send_test_case_response
2
3
send_test_case_response(
4
    test_case_id="<TEST-CASE-ID>",
5
    actual_output="The capital of France is Paris.",
6
)

All fields are optional — anything you leave out falls back to the value from the golden:

• actualOutput — the output your agent produced (string).
• retrievalContext — retrieved documents, for RAG metrics (string[]).
• toolsCalled — tools your agent called, for tool metrics (ToolCall[]).
• expectedTools — the tool calls you expected (ToolCall[]).
• metadata — arbitrary metadata to attach to the test case (object).

A successful submission returns "status": "accepted" and the test case is evaluated right away. Once every test case’s result has arrived, the test run finalizes and results appear on your dashboard as usual. See the API reference for the full schema.