Handler Mode

Connect an AI app that has no HTTP endpoint by implementing a handler function the Confident Agent runs locally.

Included on the Enterprise plan. Book a demo, opens in a new tab. Not included on the Team plan. Not included on the Starter plan. Not included on the Free plan.

Handler Mode lets you connect an AI app that has no HTTP endpoint to Confident AI. Instead of pointing an AI Connection at a URL, you implement a small handler function that the Confident Agent runs inside your own environment. Confident AI sends inputs down the agent's existing outbound tunnel, your function produces the output locally, and the agent relays it back.

When to Use Handler Mode

Reach for Handler Mode when you want to evaluate or red team an app that isn't exposed as a callable service:

The pipeline is an internal script, notebook, or library with no API in front of it.
Standing up and hosting an endpoint (even internally) is more work than you want for an evaluation.
Security or compliance requires that inputs are processed in place, with nothing inbound and no code hosted by Confident AI.

If your app already has an HTTP endpoint, use a standard AI Connection — or, for a private endpoint behind a firewall, the Confident Agent in its default forwarding mode.

How It Works

The agent connects outbound over WebSocket Secure (WSS) exactly as in forwarding mode. The only difference is what happens when a request arrives: instead of forwarding it to an internal URL, the agent calls your handler function and relays whatever it returns.

sequenceDiagram
    participant P as Your Pipeline (imported)
    participant H as handler()
    participant A as Confident Agent
    participant C as Confident AI

    A->>C: Connect outbound (WSS/443)
    Note over A,C: WebSocket tunnel established

    C->>A: Send evaluation input
    A->>H: handler(request)
    H->>P: Call your app in-process
    P-->>H: Output
    H-->>A: return output
    A-->>C: Relay output back
    Note over C: Evaluation continues

Your handler is imported once when the agent starts, so expensive setup — loading a model, creating clients — happens a single time. The function is then called once per input, over the same long-lived connection.

Requirements

Outbound internet access on port 443 (WSS) from the machine running the agent
A runtime where your pipeline is importable (its dependencies installed)
No inbound ports need to be opened

Quick Start

A handler runs in the same runtime as the app it calls, so pick the tab for the language your pipeline is written in. Python is available today; TypeScript is coming soon; any other language connects through a small local endpoint and the agent's forwarding mode.

Install the agent
Install the Confident Agent into the same environment as your pipeline. It ships both the runner and the handler decorator.
```
pip install confident-agent
```
Write your handler
Create a file — for example confident_handler.py — and decorate one function with @handler. It receives a Request and returns your app's output. Import and call your existing pipeline directly; you only write the glue.
```
from confident_agent import handler, Request
from my_company.pipeline import summarize  # your existing code

@handler
def summarize_handler(request: Request) -> str:
    return summarize(request.input, request.context)
```
Returning a plain string is enough for most apps — the string becomes the actual output. For richer responses (retrieval context, tool calls, multi-turn state), return a Response instead.
Run the agent
Start the agent pointing at your handler file, authenticated with a project API key. Because the handler imports your pipeline, run this in the same Python environment as your pipeline — the same virtualenv, conda env, or container your app already uses. This is the recommended way to run handler mode:
```
CONFIDENT_API_KEY=<your-api-key> confident-agent --handler confident_handler.py
```
That's the whole command — there is no URL or port to configure. The agent dials out to Confident AI, and when a test run needs your app's output, it calls your handler function directly.
Running with Docker instead
Only use Docker if your team deploys everything containerized. The stock confidentai/confident-agent image contains just the agent — not your pipeline's dependencies — so mounting your handler into it will fail on import (ModuleNotFoundError on your own packages). Instead, build your own image on top of ours, adding your code and dependencies:
FROM confidentai/confident-agent USER root COPY . /app RUN pip install -r /app/requirements.txt USER confident
Build it, then run it. In Docker, pass the same two settings as environment variables instead of CLI flags:
docker build -t my-company/confident-handler . docker run -d \ -e CONFIDENT_API_KEY=<your-api-key> \ -e CONFIDENT_HANDLER=/app/confident_handler.py \ my-company/confident-handler
When the agent connects, the connection status flips to Agent connected in the AI Connection settings.
Create the AI Connection
In Project Settings → AI Connections, click New AI Connection, give it a name, and set the Target to Confident Agent (Handler). There is no endpoint URL to configure. Save, then test it.

Install the agent
Install the Node agent into the same project as your pipeline.
```
npm install @confident-ai/agent
```
Write your handler
Create a file — for example handler.ts — and register one handler. It receives a Request and returns your app's output. Import and call your existing pipeline directly.
```
import { handler, Request, Response } from "@confident-ai/agent";
import { summarize } from "./pipeline"; // your existing code

handler((request: Request): string => {
  return summarize(request.input, request.context);
});
```
Return a string for the actual output, or a Response for richer results ({ output, retrievalContext, toolsCalled, state }).
Run the agent
Start the agent pointing at your handler file, authenticated with a project API key.
```
CONFIDENT_API_KEY=<your-api-key> npx @confident-ai/agent --handler ./handler.ts
```
Create the AI Connection
In Project Settings → AI Connections, click New AI Connection, give it a name, and set the Target to Confident Agent (Handler). There is no endpoint URL to configure. Save, then test it.

Expose a local endpoint

Wrap your pipeline in a small HTTP handler — for example with Ring. It only needs to listen on localhost.

(require '[ring.adapter.jetty :as jetty]
         '[ring.middleware.json :refer [wrap-json-body wrap-json-response]]
         '[ring.util.response :refer [response]])

(defn handler [req]
  (let [{:strs [input context]} (:body req)]
    (response {:output (summarize input context)})))

(jetty/run-jetty
  (-> handler wrap-json-body wrap-json-response)
  {:port 8000})

Run the agent in forwarding mode

Start the Confident Agent in the same network — no handler file needed, only outbound access.

docker run -d \
  -e CONFIDENT_API_KEY=<your-api-key> \
  -e CONFIDENT_WS_BASE_URL=wss://deepeval.confident-ai.com/ws/relay \
  confidentai/confident-agent

Point an AI Connection at the local endpoint
Create an AI Connection, enable Internal, and set the endpoint to your local URL (e.g. http://localhost:8000). Map the payload and set the Actual Output Key Path to ["output"].

The Handler

A handler is any function registered with the SDK — @handler in Python, handler(...) in TypeScript. The agent discovers it at startup; the function name is yours. Exactly one handler per agent process. (For languages that use the HTTP fallback, this section doesn't apply — your endpoint defines the contract instead.)

The Request

Your handler receives a single Request object. Read only the fields you need; unused fields are safe to ignore.

Field	Description	Type
`request.input`	The input for a single-turn golden	`str`
`request.context`	Context from the golden	`list[str]`
`request.retrieval_context`	Retrieval context from the golden	`list[str]`
`request.expected_output`	Expected output, when present	`str \| None`
`request.turns`	Turn history for multi-turn evaluations	`list[Turn]`
`request.scenario`	Scenario for conversational goldens	`str \| None`
`request.state`	Mutable state carried across multi-turn simulations	`Any`
`request.prompts`	Prompt versions attached to the connection	`dict \| None`
`request.hyperparameters`	Hyperparameters attached to the connection	`dict \| None`
`request.test_case_id`	Identifier for linking the test case to a trace	`str \| None`
`request.turn_id`	Identifier for linking an individual turn to a trace	`str \| None`

The Return

Return a string and it is used directly as the actual output — all most single-turn apps need. Return a Response when you need to surface more than the output, such as retrieval context for RAG metrics, tool calls, or updated multi-turn state.

from confident_agent import handler, Request, Response

@handler
def rag_handler(request: Request) -> Response:
    result = pipeline(request.input)
    return Response(
        output=result.answer,
        retrieval_context=result.chunks,   # optional, for RAG metrics
        tools_called=result.tools,         # optional, for tool metrics
    )

Field	Description	Type
`output`	The actual output of your app (required)	`str`
`retrieval_context`	Retrieved chunks, for RAG metrics	`list[str]`
`tools_called`	Tools your app invoked, for tool metrics	`list[ToolCall]`
`state`	Updated state to carry into the next turn	`Any`

Handling Errors

If your handler raises, the agent reports the failure for that single input and the run continues — one bad input won't abort the whole evaluation. The exception message is surfaced on the test case so you can debug it.

@handler
def summarize_handler(request: Request) -> str:
    if not request.input:
        raise ValueError("empty input")  # this test case is marked errored
    return summarize(request.input, request.context)

Configuration

Variable	Description	Required
`CONFIDENT_API_KEY`	Your project API key	Yes
`CONFIDENT_HANDLER`	Path to the file containing your `@handler` function	Yes, for handler mode
`CONFIDENT_WS_BASE_URL`	WebSocket relay URL	No — defaults to `wss://deepeval.confident-ai.com/ws/relay`

The --handler CLI flag and the CONFIDENT_HANDLER environment variable are equivalent; use whichever fits how you run the agent.

Testing Your Handler

Click Ping on the connection. Confident AI sends one sample input through the tunnel, runs your handler, and shows what it returned — a green result confirms the handler is discovered, runs, and produces an output. If the handler isn't found, raises, or returns nothing, the error is shown so you can fix it before running a full evaluation or assessment.

✅ Done. Your AI app is connected with no endpoint — ready to run evaluations and red teaming.

Multi-Turn

For multi-turn evaluations, request.turns carries the conversation so far and request.state carries anything you returned from the previous turn. Return a Response with an updated state to thread information forward. See Multi-Turn State for the full model.

Next Steps

Confident Agent

How the agent's outbound tunnel works, and forwarding mode for apps that do have an internal endpoint.

AI Connections

The full AI Connection model — payloads, output parsing, headers, and more.

No-Code Red Teaming

Run adversarial assessments against your handler-connected app.

Single-Turn Evals Without Code

Run dataset evaluations on the platform against your connection.

Setting this up for your organization?Set up the controls your team needs before a wider rolloutTalk to us

Last updated on August 2, 2026