Evaluate Threads

Overview

Thread evaluations let you evaluate an entire multi-turn conversation as a single unit, rather than evaluating individual traces or spans in isolation. This is essential for conversational AI apps where quality depends on the full context of a conversation — did the assistant stay on topic, remember what the user said three turns ago, and eventually resolve their problem?

Like trace and span evals, thread evals run server-side on Confident AI. In your application, use confident-trace to group each request's trace into a thread with a shared thread ID and set the trace's input and output so Confident AI can reconstruct the conversation. You can then trigger the evaluation automatically with an Evaluation Rule or manually with DeepEval's evaluate_thread function.

How It Works

Thread evaluations follow these steps:

1. You create a multi-turn metric collection on Confident AI with the conversational metrics you want to run.
2. Your app creates traces with a shared thread ID, setting input and output on each trace to represent conversation turns.
3. You trigger the evaluation either with a Thread Evaluation Rule or by calling DeepEval's evaluate_thread function when the conversation is complete.
4. Confident AI builds a conversational test case from the trace I/O values — each trace's input becomes a user turn, and each output becomes an assistant turn.
5. Your multi-turn metrics run against the full conversation and results appear on the thread in the dashboard.

Only multi-turn metric collections work for thread evaluations. Using a single-turn collection will not produce results.

sequenceDiagram
    participant App as Your App
    participant SDK as confident-trace
    participant CAI as Confident AI

    loop Each conversation turn
        App->>SDK: Enter span / turn()
        App->>SDK: update_trace(thread_id, input, output)
        SDK->>CAI: Export trace
    end

    CAI->>CAI: Thread idle for the rule's time limit
    CAI->>CAI: Collect all traces in thread
    CAI->>CAI: Build conversational test case from trace I/O
    CAI->>CAI: Run multi-turn metrics
    CAI->>CAI: Store results on thread

How Thread Evals Differ

The key difference is that you don't set a separate test case for thread evals — instead, Confident AI automatically constructs the conversation from trace I/O:

• Trace input → user message
• Trace output → assistant message

This is why setting trace I/O correctly is critical for thread evaluations. Set them to the raw user text and the final assistant reply — not your internal prompt template or a JSON blob — because that's what the conversational metrics will read as the dialogue.

Evaluate a Thread

Use confident-trace to instrument each turn. Set the thread ID and trace I/O with update_trace / updateTrace inside your entry span:

from openai import OpenAI
from confident_trace import init, span, update_trace, shutdown

init()
client = OpenAI()
your_thread_id = "your-thread-id"

def llm_app(query: str):
    with span("llm_app", type="agent"):
        res = client.chat.completions.create(
            model="gpt-4o",
            messages=[{"role": "user", "content": query}]
        ).choices[0].message.content
        update_trace(thread_id=your_thread_id, input=query, output=res)
        return res

try:
    llm_app("What's the weather in SF?")
    llm_app("What about tomorrow?")
finally:
    shutdown()

import OpenAI from "openai";
import { init, withSpan, updateTrace } from "confident-trace";

const runtime = init();
const openai = new OpenAI();
const yourThreadId = "your-thread-id";

const llmApp = async (query: string) => {
  return withSpan({ name: "llm_app", type: "agent" }, async () => {
    const res = await openai.chat.completions.create({
      model: "gpt-4o",
      messages: [{ role: "user", content: query }],
    });
    const data = res.choices[0].message.content ?? "";
    updateTrace({ threadId: yourThreadId, input: query, output: data });
    return data;
  });
};

try {
  await llmApp("What's the weather in SF?");
  await llmApp("What about tomorrow?");
} finally {
  await runtime.shutdown();
}

With a Thread Evaluation Rule enabled, once the second trace has been idle for the rule's time limit, Confident AI evaluates the two-turn conversation and the results appear on the thread in the Observatory.

Trigger an Evaluation from Code

confident-trace is responsible for creating and exporting the thread's traces; it does not include evaluation functions. To trigger the evaluation explicitly when your conversation ends, use evaluate_thread from DeepEval and pass the same thread ID plus the name of your multi-turn metric collection:

from deepeval.tracing import evaluate_thread

# Run each conversation turn using the confident-trace instrumentation above.
try:
    llm_app("What's the weather in SF?")
    llm_app("What about tomorrow?")
finally:
    shutdown()  # Flush the traces before requesting the evaluation.

evaluate_thread(
    thread_id=your_thread_id,
    metric_collection="My Multi-Turn Collection",
)

Call evaluate_thread only after all traces in the conversation have been exported. The asynchronous a_evaluate_thread function is also available in Python.

For a conversation that has already finished, you can also start an evaluation from the Observatory or use the evaluate_thread tool exposed by the Confident AI MCP server.

Add Turn Context

You can optionally enrich each turn with tools called and retrieval context. This gives multi-turn metrics additional context about how each response was generated — for example, whether the assistant actually looked something up before answering.

from confident_trace import span, update_trace

def llm_app(query: str):
    with span("llm_app", type="agent"):
        chunks = retrieve(query)
        results = web_search(query)
        res = generate(query, chunks, results)
        update_trace(
            thread_id="your-thread-id",
            input=query,
            output=res,
            retrieval_context=[chunk.text for chunk in chunks],
            tools_called=[{"name": "WebSearch", "input": {"query": query}, "output": results}],
        )
        return res

import { withSpan, updateTrace } from "confident-trace";

const llmApp = async (query: string) => {
  return withSpan({ name: "llm_app", type: "agent" }, async () => {
    const chunks = await retrieve(query);
    const results = await webSearch(query);
    const res = await generate(query, chunks, results);
    updateTrace({
      threadId: "your-thread-id",
      input: query,
      output: res,
      retrievalContext: chunks.map((c) => c.text),
      toolsCalled: [{ name: "WebSearch", input: { query }, output: results }],
    });
    return res;
  });
};

Tool calls are plain JSON objects with a name and optional input / output — no imports required.

Examples

Quick quiz: Given the code below, with a Thread Evaluation Rule enabled, will Confident AI successfully evaluate the thread?

from confident_trace import span, update_span, update_trace

your_thread_id = "your-thread-id"

def llm_app(query: str):
    with span("llm_app", type="agent"):
        res = generate(query)
        update_span(input=query, output=res)
        update_trace(thread_id=your_thread_id)
        return res

llm_app("Hello")
llm_app("Can you help me with my order?")

import { withSpan, updateSpan, updateTrace } from "confident-trace";

const yourThreadId = "your-thread-id";

const llmApp = async (query: string) => {
  return withSpan({ name: "llm_app", type: "agent" }, async () => {
    const res = await generate(query);
    updateSpan({ input: query, output: res });
    updateTrace({ threadId: yourThreadId });
    return res;
  });
};

await llmApp("Hello");
await llmApp("Can you help me with my order?");

Answer: No — the traces are correctly grouped into a thread, but the thread evaluation will produce no results because neither input nor output has been set on the trace. They were set on the span via update_span, which is what trace and span evals read — not what thread evals read. Without trace-level I/O, Confident AI has no conversation turns to evaluate.

Next Steps

Last updated on September 10, 2026