LLM EvaluationMulti-Turn Evals

Simulate User Turns

Mock user interactions with your multi-turn use cases

Overview

Simulating user turns require:

  • A multi-turn dataset of conversational goldens
  • A callback function that wraps around your chatbot to the next conversation turn

Each conversational golden must have a scenario before you can simulate user turns. It is also highly recommended to provide a user description for higher quality simulations.

How it works

  1. Pull your multi-turn dataset from Confident AI
  2. Define a callback that invokes your chatbot to generate the next conversation turn
  3. Run a simulation for each golden in your dataset

Run Simulations Locally

1

Pull dataset

Pull your multi-turn dataset (and create one if you haven’t already):

main.py
1from deepeval.dataset import EvaluationDataset
2
3dataset = EvaluationDataset()
4dataset.pull(alias="YOUR-DATASET-ALIAS")
2

Define callback

Define a callback that wraps around your chatbot and generates the next conversation turn.

main.py
1from deepeval.test_case import Turn
2from typing import List
3
4def chatbot_callback(input: str, turns: List[Turn], thread_id: str) -> Turn:
5    messages = [{"role": turn.role, "content": turn.content} for turn in turns]
6    messages.append({"role": "user", "content": input})
7    response = your_chatbot(messages) # Replace with your chatbot
8    return Turn(role="assistant", content=response)

The callback should accept an input, and optionally a list of Turns and the thread id. It should return the next Turn in the conversation.

3

Run simulation

Create a simulator with your callback and run simulations for the goldens in your dataset.

main.py
1from deepeval.simulator import ConversationSimulator
2
3simulator = ConversationSimulator(model_callback=chatbot_callback)
4conversational_test_cases = simulator.simulate(conversational_goldens=dataset.goldens)

The conversational test cases should now contain test cases with simulated conversation turns for each golden in your dataset.

Run Simulations Remotely

1

Pull dataset

Pull your multi-turn dataset as you would with local simulations:

main.py
1from deepeval.dataset import EvaluationDataset
2
3dataset = EvaluationDataset()
4dataset.pull(alias="YOUR-DATASET-ALIAS")
2

Simulate turns

Set run_remote to true to run simulations remotely.

main.py
1from deepeval.simulator import ConversationSimulator
2
3simulator = ConversationSimulator(model_callback=chatbot_callback, run_remote=True)
4conversational_test_cases = simulator.simulate(conversational_goldens=dataset.goldens)

Advanced Usage

Early stopping

To stop a simulation naturally before it reaches the maximum number of turns, you can provide the expected_outcome for each golden, which will end the conversation automatically after the expected outcome has been reached.

main.py
1from deepeval.dataset import ConversationalGolden
2
3conversation_golden = ConversationalGolden(
4    scenario="Andy Byron wants to purchase a VIP ticket to a cold play concert.",
5    expected_outcome="Successful purchase of a ticket.",
6    user_description="Andy Byron is the CEO of Astronomer.",
7)

If expected_outcome is unavalaible, the max_user_simulations parameter in the simluate method will stop simulation after a certain number of user turns, which is defaulted to 10.

Extending existing conversations

You can extend existing conversations by providing existing Turns to each golden in your dataset. The simulator will automatically detect and continue simulating from the existing turns.

main.py
1from deepeval.dataset import ConversationalGolden
2from deepeval.test_case import Turn
3
4conversation_golden = ConversationalGolden(
5    scenario="Andy Byron wants to purchase a VIP ticket to a cold play concert.",
6    user_description="Andy Byron is the CEO of Astronomer.",
7    turns=[
8        Turn(role="user", content="Hi"),
9        Turn(role="assistant", content="Hello! How can I help you today?"),
10        Turn(role="user", content="I want to purchase a VIP ticket to a cold play concert."),
11    ]
12)