Single-Turn Evals (No-Code)

Overview

Single-turn evaluations test one input → one output interactions. These are use cases where each request is independent and doesn't rely on conversation history:

• Q&A systems — answering questions from documents or knowledge bases
• Summarization — condensing long content into key points
• Classification — categorizing text into predefined labels
• RAG pipelines — retrieval-augmented generation with context

Single-turn evals treat your AI app as a black box — only the output, tools called, and retrieval context matter for evaluation.

Requirements

To run a single-turn evaluation, you need:

1. A single-turn dataset — goldens with input and optionally expected_output, context, etc.
2. A single-turn metric collection — the metrics you want to evaluate against

How it works

No-code evals follow a simple 4-step process:

1. Define metrics — choose what aspects of quality to measure (e.g., relevancy, faithfulness)
2. Create dataset — build goldens with parameters such as inputs and expected outputs
3. Generate AI output — provide actual outputs from your AI app
4. Evaluate — run metrics against your test cases and view results

Here's a visual representation on the data flow during evaluation:

sequenceDiagram
    participant User as You
    participant Platform as Confident AI
    participant AI as Your AI App
    participant Metrics as Metric Collection

    User->>Platform: Start Evaluation

    loop For each golden in dataset
        Platform->>AI: Send input
        AI-->>Platform: Generate output
        Platform->>Metrics: Run Metrics on test case
        Metrics-->>Platform: Metric scores
    end

    Platform-->>User: Test Run Produced
    Note over User,Platform: View results on Dashboard

Run an Evaluation

You can evaluate on a dataset by clicking on the Evaluate button on the top right of a dataset page.

1. Select your dataset and metrics

   1. Navigate to Project > Datasets, and select your single-turn dataset to evaluate
   2. Click Evaluate
   3. Select your single-turn Metric Collection

2. Configure output generation

   Select how to generate actual outputs:

   PromptAI Connection

   

   For single-prompt systems, select a prompt template that Confident AI will use to call your configured LLM provider.

   1. Select your desired prompt and the version of the prompt as your output generation method
   2. Map any golden fields from your current dataset to any variables defined within your prompt
   3. Confident AI calls your prompt for each golden and generates outputs automatically

3. Run and view results

   Click Run Evaluation and wait for it to complete. You'll be redirected to your test run dashboard showing:

   • Score distributions — average, median, and percentiles for each metric
   • Pass/fail results — a test case passes only if all metrics meet their thresholds
   • AI-generated summary — automated analysis of patterns and issues
   • Individual test cases — drill down into specific failures

Long-Running Agents

Some agents might have a long response time, for such agents, Confident AI supports Long-Running Agent mode. Instead of holding the connection open until your agent responds, Confident AI sends each golden to your AI connection, immediately closes the connection, and waits for your agent to post its result back when it's ready.

How it works

1. Enable Async Responses on your AI connection's General tab. The evaluate dialog shows a notice whenever you select a connection with async responses enabled.
2. For each golden, Confident AI sends the payload to your endpoint with a unique testCaseId, then closes the connection — it does not wait for a response.
3. Once your agent finishes, you can post the result to the POST /v1/test-runs/evaluate/{testCaseId} endpoint with that same testCaseId and the actual output.
4. Confident AI evaluates each test case as its result arrives, and finalizes the test run once every result has been received.

sequenceDiagram
    participant User as You
    participant Platform as Confident AI
    participant AI as Your Agent

    User->>Platform: Start Evaluation (Long-Running Agent)

    loop For each golden in dataset
        Platform->>AI: Send input + confident.testCaseId
        Platform-->>Platform: Close connection (no wait)
    end

    loop When each agent finishes (minutes later)
        AI->>Platform: POST /v1/test-runs/evaluate/{testCaseId}
        Platform->>Platform: Evaluate test case
    end

    Platform-->>User: Test Run finalized once all results arrive
    Note over User,Platform: View results on Dashboard

Posting results back

Read confident.testCaseId from the payload Confident AI sends to your endpoint, put it in the URL, then post your result back with your project API key:

curl -X POST https://api.confident-ai.com/v1/test-runs/evaluate/<TEST-CASE-ID> \
  -H "Content-Type: application/json" \
  -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>" \
  -d '{
    "actualOutput": "The capital of France is Paris."
  }'

You can send any single-turn test case field alongside actualOutput — for example retrievalContext, toolsCalled, or expectedTools. See the API reference for the full schema, or the Set Up Long-Running AI Connections guide for an end-to-end walkthrough.

Regression Testing

Once you have two or more test runs, you can compare them side-by-side to identify regressions.

1. Open regression testing

   1. Go to your test run's A|B Regression Test tab
   2. Click New Regression Test
   3. Select the test runs you want to compare

2. Analyze regressions

   The comparison view highlights:

   • Regressions (red) — test cases that got worse
   • Improvements (green) — test cases that got better
   • Side-by-side scores — metric comparisons across runs

Next Steps

Last updated on July 27, 2026