Launch Week 02 wrapped — explore all five launches
Confident AI
Docs
Pricing
Products
LLM Evaluation
LLM Evaluation

Benchmark LLM systems with research-backed metrics.

LLM Observability
LLM Observability

Trace, monitor, and alert on production LLM systems.

AI Red Teaming
AI Red Teaming

Stress-test LLM apps against adversarial attacks.

AI Governance
AI Governance

Enforce AI standards and controls across teams.

Book a DemoSign Up
Book a DemoSign Up

Multi-Turn Evals

Simulate conversations and run end-to-end testing for multi-turn use cases

Overview

Multi-turn evaluation requires:

  • A multi-turn dataset of conversational goldens
  • A callback function that wraps around your chatbot to generate conversation turns
  • A list of multi-turn metrics you wish to evaluate with

How It Works

  1. Pull your multi-turn dataset from Confident AI
  2. Define a callback that invokes your chatbot to generate conversation turns
  3. Simulate conversations for each golden in your dataset
  4. Run evaluation on the resulting test cases

Define Your Callback

Define a callback that wraps around your chatbot and generates the next conversation turn:

callback.py
from deepeval.test_case import Turn
from typing import List

def chatbot_callback(input: str, turns: List[Turn], thread_id: str) -> Turn:
    messages = [{"role": turn.role, "content": turn.content} for turn in turns]
    messages.append({"role": "user", "content": input})
    response = your_chatbot(messages) # Replace with your chatbot
    return Turn(role="assistant", content=response)

Run Evals Locally

Running evals locally is only possible with the Python deepeval library. For Typescript or other languages, skip to remote evals.

  1. Pull dataset

    Pull your multi-turn dataset (and create one if you haven't already):

    main.py
    from deepeval.dataset import EvaluationDataset

dataset = EvaluationDataset()
dataset.pull(alias="YOUR-DATASET-ALIAS")

  2. Simulate conversations

    Create a simulator with your callback and generate test cases from your goldens:

    main.py
    from deepeval.simulator import ConversationSimulator

simulator = ConversationSimulator(model_callback=chatbot_callback)
for golden in dataset.goldens:
    test_case = simulator.simulator(golden)
    dataset.add_test_case(test_case)

  3. Run evaluation

    The evaluate() function runs your test suite and uploads results to Confident AI:

    main.py
    from deepeval.metrics import TurnRelevancyMetric
from deepeval import evaluate

# Replace with your metrics
evaluate(test_cases=dataset.test_cases, metrics=[TurnRelevancyMetric()])

    Done! You should see a link to your newly created sharable testing report.

    • Each metric is applied to every test case (e.g., 10 test cases × 2 metrics = 20 evaluations)
    • A test case passes only if all metrics for it pass
    • The test run's pass rate is the proportion of test cases that pass
    Multi-Turn Testing Reports

Run Evals Remotely

  1. Create metric collection

    Go to Project > Metric > Collections:

    Metric Collection for Remote Evals

  2. Pull dataset and simulate conversations

    Set run_remote to true to run simulations remotely:

    main.py
    from deepeval.simulator import ConversationSimulator
from deepeval.dataset import EvaluationDataset

dataset = EvaluationDataset()
dataset.pull(alias="YOUR-DATASET-ALIAS")

simulator = ConversationSimulator(model_callback=chatbot_callback, run_remote=True)
for golden in dataset.goldens:
    test_case = simulator.simulator(golden)
    dataset.add_test_case(test_case)

  3. Run evaluation

    main.py
    from deepeval import evaluate

evaluate(test_case=dataset.test_cases, metric_collection="YOUR-COLLECTION-NAME")

Advanced Usage

Early Stopping

To stop a simulation naturally before it reaches the maximum number of turns, provide an expected_outcome for each golden. The conversation will end automatically after the expected outcome has been reached.

main.py
from deepeval.dataset import ConversationalGolden

conversation_golden = ConversationalGolden(
    scenario="Andy Byron wants to purchase a VIP ticket to a cold play concert.",
    expected_outcome="Successful purchase of a ticket.",
    user_description="Andy Byron is the CEO of Astronomer.",
)

Extend Existing Turns

You can extend existing conversations by providing existing Turns to each golden. The simulator will automatically detect and continue simulating from the existing turns.

main.py
from deepeval.dataset import ConversationalGolden
from deepeval.test_case import Turn

conversation_golden = ConversationalGolden(
    scenario="Andy Byron wants to purchase a VIP ticket to a cold play concert.",
    user_description="Andy Byron is the CEO of Astronomer.",
    turns=[
        Turn(role="user", content="Hi"),
        Turn(role="assistant", content="Hello! How can I help you today?"),
        Turn(role="user", content="I want to purchase a VIP ticket to a cold play concert."),
    ]
)
Setting up evals in CI/CD?Catch regressions before they ship with release-ready eval gatesTalk to an engineer
Built byConfident AI