Single-Turn Evals (No-Code)
Evaluate one-shot interactions like Q&A, summarization, and classification.
Overview
Single-turn evaluations test one input → one output interactions. These are use cases where each request is independent and doesn't rely on conversation history:
- Q&A systems — answering questions from documents or knowledge bases
- Summarization — condensing long content into key points
- Classification — categorizing text into predefined labels
- RAG pipelines — retrieval-augmented generation with context
Single-turn evals treat your AI app as a black box — only the output, tools called, and retrieval context matter for evaluation.
Requirements
To run a single-turn evaluation, you need:
- A single-turn dataset — goldens with
inputand optionallyexpected_output,context, etc. - A single-turn metric collection — the metrics you want to evaluate against
How it works
No-code evals follow a simple 4-step process:
- Define metrics — choose what aspects of quality to measure (e.g., relevancy, faithfulness)
- Create dataset — build goldens with parameters such as inputs and expected outputs
- Generate AI output — provide actual outputs from your AI app
- Evaluate — run metrics against your test cases and view results
Here's a visual representation on the data flow during evaluation:
sequenceDiagram
participant User as You
participant Platform as Confident AI
participant AI as Your AI App
participant Metrics as Metric Collection
User->>Platform: Start Evaluation
loop For each golden in dataset
Platform->>AI: Send input
AI-->>Platform: Generate output
Platform->>Metrics: Run Metrics on test case
Metrics-->>Platform: Metric scores
end
Platform-->>User: Test Run Produced
Note over User,Platform: View results on Dashboard
Run an Evaluation
You can evaluate on a dataset by clicking on the Evaluate button on the top right of a dataset page.
Select your dataset and metrics
- Navigate to Project > Datasets, and select your single-turn dataset to evaluate
- Click Evaluate
- Select your single-turn Metric Collection
Configure output generation
Select how to generate actual outputs:
Configure prompt for single-turn evaluation For single-prompt systems, select a prompt template that Confident AI will use to call your configured LLM provider.
- Select your desired prompt and the version of the prompt as your output generation method
- Map any golden fields from your current dataset to any variables defined within your prompt
- Confident AI calls your prompt for each golden and generates outputs automatically
Configure AI Connection for single-turn evaluation For deployed AI systems, connect Confident AI directly to your HTTP endpoint.
- Go to Settings → AI Connections and create a connection
- Configure your endpoint URL, request payload mapping, response parsing, and headers
- In the evaluation setup, select this AI Connection as your output generation method
Run and view results
Click Run Evaluation and wait for it to complete. You'll be redirected to your test run dashboard showing:
- Score distributions — average, median, and percentiles for each metric
- Pass/fail results — a test case passes only if all metrics meet their thresholds
- AI-generated summary — automated analysis of patterns and issues
- Individual test cases — drill down into specific failures
Single-turn test run results
Long-Running Agents
Some agents might have a long response time, for such agents, Confident AI supports Long-Running Agent mode. Instead of holding the connection open until your agent responds, Confident AI sends each golden to your AI connection, immediately closes the connection, and waits for your agent to post its result back when it's ready.
How it works
- Enable Async Responses on your AI connection's General tab. The evaluate dialog shows a notice whenever you select a connection with async responses enabled.
- For each golden, Confident AI sends the payload to your endpoint with a unique
testCaseId, then closes the connection — it does not wait for a response. - Once your agent finishes, you can post the result to the
POST /v1/test-runs/evaluate/{testCaseId}endpoint with that sametestCaseIdand the actual output. - Confident AI evaluates each test case as its result arrives, and finalizes the test run once every result has been received.
sequenceDiagram
participant User as You
participant Platform as Confident AI
participant AI as Your Agent
User->>Platform: Start Evaluation (Long-Running Agent)
loop For each golden in dataset
Platform->>AI: Send input + confident.testCaseId
Platform-->>Platform: Close connection (no wait)
end
loop When each agent finishes (minutes later)
AI->>Platform: POST /v1/test-runs/evaluate/{testCaseId}
Platform->>Platform: Evaluate test case
end
Platform-->>User: Test Run finalized once all results arrive
Note over User,Platform: View results on Dashboard
Posting results back
Read confident.testCaseId from the payload Confident AI sends to your endpoint, then post your result back for that test case with your project API key:
from confident_ai import ConfidentAI
client = ConfidentAI()
result = client.test_runs.submit_test_case_result(
test_case_id="<TEST-CASE-ID>",
)Send actualOutput along with any other single-turn test case fields you have, such as retrievalContext, toolsCalled or expectedTools. When your agent fails, send error instead of actualOutput: the test case is marked errored and isn't evaluated, so the test run can finish without it. See the Set Up Long-Running AI Connections guide for an end-to-end walkthrough.
Regression Testing
Once you have two or more test runs, you can compare them side-by-side to identify regressions.
Open Regressions
Open Regressions under Test in the sidebar. Choose a run in Select First Test Run, then the run to compare it with in Select Test Run to Compare Against. Test cases are matched across the two runs by Input or Name.
Analyze regressions
The comparison shows both runs' results side by side:
- Regressions — test cases that pass in the run you compare against but fail in the first run
- Improvements — test cases that fail in the run you compare against but pass in the first run
- All test cases — every matched test case, with both runs' metric scores
Regression testing
Next Steps
Multi-Turn Evals
Evaluate conversational AI where context builds across multiple exchanges.
Arena
Compare prompts and models side-by-side in real-time.
Last updated on