Evaluate Threads

Run evaluations on multi-turn conversations by evaluating entire threads

Included on the Enterprise plan. Book a demo, opens in a new tab. Included on the Team plan. Included on the Starter plan. Not included on the Free plan.

Overview

Thread evaluations let you evaluate an entire multi-turn conversation as a single unit, rather than evaluating individual traces or spans in isolation. This is essential for conversational AI apps where quality depends on the full context of a conversation.

How It Works

Thread evaluations follow these steps:

You create a multi-turn metric collection on Confident AI with the conversational metrics you want to run.
Your app creates traces with a shared thread ID, setting input and output on each trace to represent conversation turns.
Once the conversation is complete, you call the evaluate thread function with the thread ID and metric collection name.
Confident AI builds a conversational test case from the trace I/O values — each trace's input becomes a user turn, and each output becomes an assistant turn.
Your multi-turn metrics run against the full conversation and results appear on the thread in the dashboard.

Only multi-turn metric collections work for thread evaluations. Using a single-turn collection will not produce results.

sequenceDiagram
    participant App as Your App
    participant SDK as Evals API/DeepEval
    participant CAI as Confident AI

    loop Each conversation turn
        App->>SDK: Call observed function
        App->>SDK: Set thread ID, input, and output on trace
        SDK->>CAI: Send trace
    end

    App->>SDK: Conversation complete
    SDK->>CAI: Evaluate thread (thread ID + metric collection)
    CAI->>CAI: Collect all traces in thread
    CAI->>CAI: Build conversational test case from trace I/O
    CAI->>CAI: Run multi-turn metrics
    CAI->>CAI: Store results on thread

How Thread Evals Differ

	Trace & Span Evals	Thread Evals
Scope	Single request/response	Entire multi-turn conversation
Metric collection	Single-turn metrics	Multi-turn metrics
When to run	Real-time or retrospectively	Retrospectively only (after conversation ends)
Data source	Test case parameters you set on spans/traces	Trace `input`/`output` values become conversation turns

The key difference is that you don't set test case parameters for thread evals — instead, Confident AI automatically constructs the conversation from trace I/O:

Trace input → user message
Trace output → assistant message

This is why setting trace I/O correctly is critical for thread evaluations.

Evaluate a Thread

Thread evaluations must be triggered manually after a conversation has completed, since Confident AI cannot automatically know when a multi-turn conversation is finished.

Call the evaluate thread function once the conversation is done:

main.py

from openai import OpenAI
from deepeval.tracing import observe, update_current_trace, evaluate_thread

client = OpenAI()
your_thread_id = "your-thread-id"

@observe()
def llm_app(query: str):
    res = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": query}]
    ).choices[0].message.content
    update_current_trace(thread_id=your_thread_id, input=query, output=res)
    return res

llm_app("What's the weather in SF?")
llm_app("What about tomorrow?")

evaluate_thread(thread_id=your_thread_id, metric_collection="My Multi-Turn Collection")

index.ts

import { observe, updateCurrentTrace, evaluateThread } from "deepeval/tracing";
import OpenAI from "openai";

const yourThreadId = "your-thread-id";

const llmApp = async (query: string) => {
  const openai = new OpenAI();
  const res = await openai.chat.completions.create({
    model: "gpt-4o",
    messages: [{ role: "user", content: query }],
  });
  const data = res.choices[0].message.content;
  updateCurrentTrace({ threadId: yourThreadId, input: query, output: data });
  return data;
};
const observedLlmApp = observe({ fn: llmApp });

await observedLlmApp("What's the weather in SF?");
await observedLlmApp("What about tomorrow?");

await evaluateThread({
  threadId: yourThreadId,
  metricCollection: "My Multi-Turn Collection",
});

Add Turn Context

You can optionally enrich each turn with tools called and retrieval context. This gives multi-turn metrics additional context about how each response was generated.

main.py

from deepeval.tracing import observe, update_current_trace
from deepeval.test_case import ToolCall

@observe()
def llm_app(query: str):
    chunks = retrieve(query)
    res = generate(query, chunks)
    update_current_trace(
        thread_id="your-thread-id",
        input=query,
        output=res,
        retrieval_context=[chunk.text for chunk in chunks],
        tools_called=[ToolCall(name="WebSearch")]
    )
    return res

index.ts

import { observe, updateCurrentTrace } from "deepeval/tracing";

const llmApp = async (query: string) => {
  const chunks = await retrieve(query);
  const res = await generate(query, chunks);
  updateCurrentTrace({
    threadId: "your-thread-id",
    input: query,
    output: res,
    retrievalContext: chunks.map((c) => c.text),
    toolsCalled: [{ name: "WebSearch" }],
  });
  return res;
};
const observedLlmApp = observe({ fn: llmApp });

Examples

Quick quiz: Given the code below, will Confident AI successfully evaluate the thread?

main.py

from deepeval.tracing import observe, update_current_trace, evaluate_thread

your_thread_id = "your-thread-id"

@observe()
def llm_app(query: str):
    update_current_trace(
      thread_id=your_thread_id,
      metric_collection="Collection 1"
    )

llm_app("Hello")
evaluate_thread(thread_id=your_thread_id, metric_collection="Collection 2")

index.ts

import { observe, updateCurrentTrace, evaluateThread } from "deepeval/tracing";

const yourThreadId = "your-thread-id";

const llmApp = (query: string) => {
  updateCurrentTrace({ threadId: yourThreadId });
};
const observedLlmApp = observe({
  metricCollection: "Collection 1",
  fn: llmApp,
});

observedLlmApp("Hello");
await evaluateThread({
  threadId: yourThreadId,
  metricCollection: "Collection 2",
});

Answer: No — the thread evaluation will produce no results because neither input nor output has been set on the trace. Without these, Confident AI has no conversation turns to evaluate.

Next Steps

Thread Traces

Learn how to create threads, set I/O, and log tools called and retrieval context per turn.

Customize Traces

Add tags, metadata, and user info to your traces for filtering and analysis.

Ready to monitor AI in production?Connect traces, alerts, dashboards, and evals in one production workflowBook a demo

Last updated on July 27, 2026