1.IV: Chapter 1

Evaluating Multi-Turn Chatbots

A chatbot can be polite, on-brand, and technically correct on every individual reply, and still fail the user twelve turns later because nothing actually got resolved. Trace-level metrics light up green. The conversation, taken as a whole, is a failure.

This is the gap multi-turn evaluation is designed to close. Single-turn evaluation grades a prompt-response pair. Conversation evaluation grades the thread: whether the bot remembered what the user said three turns ago, whether it stayed in role across the whole exchange, whether the conversation actually reached the goal. Those failures are invisible at the turn level and dominant in real production traffic.

If you are building a customer support agent, a voice agent, a sales copilot, or any application where the user types more than once, evaluate at both layers, and treat the conversation as the unit of quality. This chapter is the evaluation-side companion to multi-turn observability, which covers what to instrument and watch in production.

Multi-turn failures live across turns, not in any single reply

Multi-turn chatbots fail in patterns that single-turn evaluation cannot see.

Context is lost — the bot forgets information the user provided three turns ago and asks for it again. The reply is technically a good question. The conversation experience is broken. Contradictions slip through — turn 4 contradicts turn 2. Each turn taken alone is plausible. Together, they undermine trust. The bot resolves prematurely — declares the task complete before the user actually got what they needed. From the bot's perspective the conversation is wrapped up. From the user's perspective nothing happened. Role drifts — the bot starts as a patient intake assistant and slowly slides into giving medical advice over the course of the conversation. Topic drifts — the user asked about refunds, the bot helpfully starts answering questions about loyalty programs, and the original request quietly disappears. Escalation requests get ignored — the user asks for a human three times, the bot keeps answering, sentiment goes from neutral to frustrated to gone.

None of these show up if you only score the latest reply. They only show up when you evaluate the whole conversation as a single unit.

This is also why "multi-turn evaluation by averaging single-turn metrics" produces misleading scores. A bot can score 95% on trace-level faithfulness, 92% on trace-level relevancy, 90% on trace-level tone — and still produce conversations that completely fail to resolve. The aggregate hides the structural failures that live across turns.

Trace-level metrics versus conversation-level metrics

A useful chatbot evaluation strategy uses both layers, deliberately.

Trace-level metrics catch failures that live inside a single turn. In Confident AI, each input/output pair is one trace, so these metrics evaluate the full request path for that turn: prompt, retrieved context, tool calls, spans, response, latency, and cost. Answer relevancy — does the response address what the user asked in this turn. Faithfulness — if the response is grounded in retrieved context or tool output, is it actually faithful or did the bot embellish. Retrieval quality and tool correctness — did the turn use the right evidence, tool, and arguments. Tone and helpfulness — does the response sound the way the brand wants. PII and policy adherence — did this single message leak personal information or violate organizational rules. These are necessary, but they cannot see across turns.

Conversation-level metrics catch the failures that span turns. Turn relevancy — does each turn make sense given the conversation so far, not just the immediately preceding message. Role adherence — did the bot stay in role across the whole conversation. Conversation completeness — did the conversation actually accomplish what the user came for. Knowledge retention — did the bot keep facts straight across turns. Context recall — did the bot use information from earlier turns when it should have. Sentiment trajectory — did the user's sentiment improve, stay flat, or get worse over the conversation.

A worked example. A patient intake assistant takes the user through symptom screening, history, and triage routing. A trace-level relevancy metric scores each individual turn as clear and on-topic. A conversation-level role-adherence metric checks whether the bot stayed inside intake (it cannot give medical advice). On turn seven the user asked "do you think this is serious?" and the bot answered with a soft prediction. Trace-level relevancy was high. Role adherence dropped. A conversation-completeness metric checks whether the user reached the right escalation path — they did, eventually, but the bot took fourteen turns instead of six.

Trace-level evaluation says "this turn looks good." Conversation-level evaluation says "the thread failed its job." Both are important. Conversation-level metrics tell you whether the conversation worked; trace-level metrics tell you which turn, retrieval, tool call, or response to fix.

Pick a small set, calibrate against humans, expand later

The instinct on day one is to turn on every conversation-level metric in the catalog. Twelve metrics, all live, all gating CI. Three weeks later the team is arguing about whether "context recall" is even measuring the right thing because it disagrees with human judgment a third of the time. Nobody trusts the suite, the suite gets ignored, and evaluation goes back to "did the demo look ok."

The pattern that works: start with a small set of trace-level and conversation-level metrics that map directly to your product. For most chatbots, that means trace-level answer relevancy and faithfulness, plus conversation-level turn relevancy, role adherence, and conversation completeness. Add retrieval quality or tool correctness if the chatbot uses RAG or tools. Add knowledge retention if you have memory, or sentiment trajectory if customer support is the use case. Calibrate each one against a labeled sample of fifty to a hundred threads. Aim for combined false positive and false negative rates below 5% before any metric gates a release. Then add metrics one at a time as production failures point you to the next gap.

Confident AI ships both trace-level and conversation-level metrics as part of multi-turn evaluation — answer relevancy, faithfulness, retrieval quality, tool correctness, turn relevancy, role adherence, conversation completeness, knowledge retention, context recall — with 50+ research-backed metrics open-source through DeepEval. They are configurable so teams can grade the full thread and still inspect the trace-level scores behind each turn.

Simulate multi-turn data, do not replay it

The biggest practical challenge in multi-turn evaluation is data. You cannot evaluate a conversational bot by feeding it a static prompt and grading the response — because in production it talks to users across multiple exchanges that respond, push back, change topics, and improvise.

Two failed approaches teams reach for first.

Replay historical conversations — rerun yesterday's chats against the new bot and grade the new responses. This works as a smoke test, but it tests behavior you have already seen. The user side of the replayed conversation was responding to your old bot, not your new one. Your new bot will say different things, and the replayed user will not push back, change topic, or escalate the way a real user would. Hand-write a few golden conversations — five carefully scripted dialogues. Useful for sanity-checking but nowhere near coverage. Production traffic does not behave like the five conversations a PM wrote on a Friday.

The right approach is scenario-based simulation. Define each test case as a "golden": a scenario description (what the user is trying to do), a user persona (their style, urgency, emotional state, expertise), and an expected outcome. The evaluation platform plays the user side of the conversation dynamically, generating realistic turns that respond to whatever your bot actually says. The agent under test runs against the simulated user, and metrics score the resulting thread.

Simulation tests how the bot behaves under pressure — frustrated users, contradictory instructions, ambiguous requests, off-topic interjections — in scenarios you have not yet seen in production. What used to take 2-3 hours of manual prompting per scenario takes minutes per scenario, and you can run hundreds of scenarios in parallel. Confident AI runs scenario-based multi-turn simulation natively, with trace-level and conversation-level metrics built in.

Coverage: happy-path, edge, use-case, regression

A multi-turn evaluation suite needs coverage across four categories. Skipping any of them leaves a class of failures unprotected.

Happy-path scenarios. The conversations the bot is supposed to handle when everything goes right. Patient intake from a cooperative patient. Refund request with all the right information up front. Order status lookup with a valid order number. These set the baseline for "is the bot doing its job."

Edge scenarios. Boundary conditions and ambiguity. A user who provides incomplete information across multiple turns. A user who asks two questions at once. A user whose first message is off-topic before they get to the actual request. A user who switches context halfway through. These are where most bots leak quality — not on the happy path, on the conversations that look slightly off.

Use-case-specific scenarios. The actual conversations your product gets in production, categorized by intent. Refund flows. Order status. Account access. Shipping questions. Whatever the categorization is for your product. Coverage here matters more than volume — fifty scenarios across the top use cases, well-designed, will tell you more than five hundred random ones.

Regression scenarios. The conversations that failed in production or during review and should never fail the same way again. These are usually the highest-leverage cases because they come from real user behavior, real prompt versions, and real product constraints.

The healthy ratio over time is roughly 40% happy-path, 30% edge, 20% use-case-specific, and 10% production-regression deep cuts — but the right ratio depends on the product. The thing that does not vary: every scenario should map to a failure mode the team actually cares about. Coverage is not volume. It is confidence that the suite represents the conversations your chatbot is judged on.

CI gates: a thread regression is a real regression

Every prompt change is a small experiment. Most are improvements. Some quietly break things in ways nobody notices until a customer complains. CI gates on multi-turn evaluation are how you catch the bad changes before they ship.

A working CI flow:

Maintain a curated multi-turn evaluation dataset alongside your code, versioned in the same repo. Goldens defined as scenario, persona, expected outcome.
Run scenario-based simulation on every pull request that touches agent logic, prompts, retrieval, tool schemas, or model configuration.
Score the resulting threads on both layers: trace-level metrics for each generated turn, and conversation-level metrics like turn relevancy, role adherence, and conversation completeness for the full thread.
Block merges that regress past tolerance on top-line metrics. Block merges that introduce new failures on critical scenarios regardless of average movement.

Threshold calibration is where teams get stuck. Set them too low and bad changes slip through. Set them too high and every release gets blocked. A useful default to start with: gate on no regressions worse than 5% relative on conversation completeness and role adherence, plus zero new failures on critical production or compliance scenarios. Tune from there based on what you actually see in the first few weeks.

CI gates are necessary but not sufficient. Production traffic still surfaces failure modes your test set does not cover. The complete picture ties production back into evaluation — failing threads in production convert into goldens, the suite grows from real failures, and the next CI run includes them. The observability side of that loop is covered in multi-turn observability.

Where teams stall on multi-turn evaluation

Three failure patterns we see most often.

Treating multi-turn as a feature, not a separate concern. Teams turn on trace-level metrics, group requests by thread, and call it multi-turn evaluation. It is not. Multi-turn datasets, trace-level metrics, conversation-level metrics, and multi-turn test runs need to be first-class concepts, not properties of a single-turn workflow with a thread tab.

Skipping simulation. Teams either evaluate multi-turn behavior on hand-written scripts that do not represent real users, or skip it entirely until production complaints force the issue. Simulation is the unlock — define scenarios as goldens once and the platform generates fresh dynamic conversations every run.

Letting trace-level scores carry the report-out. Aggregate trace-level faithfulness goes up, leadership thinks the bot is improving, and conversation-completeness keeps dropping because the new prompt is faster but worse at resolving threads. Always show conversation-level scores alongside trace-level ones. When they conflict, treat the conversation-level score as the product outcome and use the trace-level scores to debug the bad turns.

The fix in all three cases is the same: pick a workflow that treats multi-turn as the unit of quality, run scenario-based simulation in development and CI, and treat conversation-level metrics as primary — not as a tab on a single-turn dashboard.

A phased rollout for evaluation

Most teams do not need a 200-scenario simulation suite on day one. They need a credible eval baseline, a small set of trace-level and conversation-level metrics, and a clear path to grow.

A practical adoption path:

Week 1: Define goldens and metrics. Pick the top 10-20 scenarios across happy-path, edge, and priority use cases. Define each as scenario plus persona plus expected outcome. Pick a small set of trace-level and conversation-level metrics. Calibrate each against a labeled sample of threads.
Week 2: Run simulation in development. Run the suite locally on every meaningful change. Read failing threads by hand. Confirm the metrics agree with your judgment. If they do not, fix the metric prompt or swap to a different one.
Month 1: Move simulation into CI. Block merges that regress past tolerance. Tune thresholds based on the first two weeks of release data. The threshold should catch real regressions and not block normal release noise.
Month 2: Expand the suite. Add scenarios from production failures (see the multi-turn observability chapter for how to convert failing threads into goldens automatically). Add use-case-specific deep cuts and scenarios for high-volume or high-risk flows.
Quarter 1: Close the loop. Every failing production thread that meets your criteria becomes a golden. The next simulation run includes it. The suite grows weekly without manual export. At this point your evaluation is alive — it improves every cycle without anyone having to maintain it.

The point is not to do everything at once. The point is to never settle for "we score the final reply." Conversation-level evaluation is the difference between knowing your chatbot works and hoping it does.

What "good" looks like, six months in

Six months after rolling this out, a healthy multi-turn chatbot evaluation setup looks like:

The eval suite has 100-300 scenario goldens across happy-path, edge, use-case-specific, and production-regression cases, refreshed weekly from production failures.
Trace-level metrics — answer relevancy, faithfulness, retrieval quality, tool correctness — and conversation-level metrics — turn relevancy, role adherence, conversation completeness, knowledge retention — are calibrated against humans with combined error rates below 5%.
Multi-turn simulation runs on every prompt, model, and retrieval change in CI, blocking merges on conversation-level regressions and any new failure on critical scenarios.
PMs and domain experts contribute scenarios and annotate threads without filing engineering tickets.
When a customer complains, the failing thread converts into a golden in one click, and the next simulation run guarantees the regression cannot ship again.

If you are missing any of these, the gap is usually not engineering capacity — it is the workflow. Most multi-turn evaluation problems get solved when the team stops treating threads as logs and starts treating them as the unit of quality the chatbot is actually judged on.

Why Confident AI

Confident AI is built for multi-turn evaluation as a first-class workflow: separate multi-turn datasets, scenario-based simulation, trace-level metrics, conversation-level metrics, human review, CI gates, and production thread feedback. That is the difference between testing a chatbot and replaying a few old transcripts.

Use Confident AI when you need to evaluate the conversation, not just the latest reply. Teams define goldens as scenarios, personas, and expected outcomes; Confident AI simulates the user side dynamically; then trace-level metrics score each turn while conversation-level metrics score the resulting thread for turn relevancy, role adherence, conversation completeness, knowledge retention, context recall, and escalation behavior. Production failures can become new goldens, so the suite grows from real conversations instead of staying frozen at launch.

Frequently Asked Questions

How do I evaluate a multi-turn chatbot?

Evaluate a multi-turn chatbot at two levels: trace-level turn quality and conversation-level outcome. Trace-level metrics catch individual turn problems across the request trace: response quality, retrieval quality, tool correctness, faithfulness, latency, and cost. Conversation-level metrics score whether the chatbot retained context, stayed in role, completed the task, handled escalation, and achieved the user's goal across the full thread. Confident AI supports both layers with multi-turn datasets, simulation, trace-level scores, and conversation-level metrics.

What are the best LLM evaluation tools for testing multi-turn conversations?

Use a tool that treats multi-turn evaluation as a first-class workflow: multi-turn datasets, scenario-based simulation, trace-level metrics, conversation-level metrics, human review, CI gates, and production-thread feedback. Confident AI is built for this workflow, with simulated user conversations, trace-level scores for each turn, and conversation-level metrics that grade the whole thread.

How do I evaluate whether a chatbot completed the user's request across the whole thread?

Use a conversation completeness metric, then validate it against human-labeled threads. The metric should judge whether the user reached the intended outcome, not whether the final reply sounded good. Confident AI pairs conversation completeness with role adherence, context recall, escalation handling, sentiment trajectory, and human review.

Why is averaging single-turn metrics not enough for multi-turn chatbots?

Averaging single-turn metrics hides structural conversation failures. A chatbot can be relevant and faithful on each individual reply while still repeating itself, forgetting earlier context, drifting roles, or ending without resolving the user's request. Confident AI evaluates the thread as a whole so conversation-level failures are not hidden inside green trace-level averages.

Do multi-turn chatbots need trace-level metrics?

Yes. Conversation-level metrics tell you whether the full thread worked, but trace-level metrics tell you which turn and execution path caused the problem. Each input/output pair is one trace, so trace-level metrics can score answer relevancy, faithfulness, retrieval quality, tool correctness, latency, and cost for that turn. Confident AI connects those trace-level scores to the conversation-level result so teams can debug bad threads without guessing.

What are the best metrics for multi-turn chatbot evaluation?

Start with trace-level answer relevancy and faithfulness, then add conversation-level turn relevancy, role adherence, and conversation completeness. Add retrieval quality or tool correctness when the chatbot uses RAG or tools. Add knowledge retention or context recall when the chatbot relies on memory, and add sentiment trajectory or escalation handling for user-facing support, sales, healthcare, finance, or voice-agent workflows. Confident AI provides these trace-level and multi-turn metrics and supports human alignment before they gate releases.

Should I replay historical conversations or simulate new conversations?

Use historical conversations as smoke tests, not as your main evaluation method. Replay tests how the new bot responds to user turns created for the old bot. Confident AI's scenario-based simulation is better for evaluation because the simulated user responds dynamically to the bot under test, including pushback, ambiguity, escalation, and topic shifts.

How do I run multi-turn chatbot evaluation in CI?

Define multi-turn goldens as scenario, persona, and expected outcome. Run simulation on every pull request that changes prompts, models, retrieval, tools, or agent logic. Confident AI runs the generated threads through trace-level and conversation-level metrics with CI reporting so teams can block regressions before release.

Resources and Next Steps

Start with 10-20 multi-turn scenarios across happy-path, edge, and priority use cases. Pick a small set of trace-level and conversation-level metrics, calibrate them against human-labeled threads, run simulation locally, then move the suite into CI. Once the chatbot is live, convert failing production threads into new goldens every week.