Evaluate Traces & Spans

Overview

Online evaluations let you run metrics on traces and spans on-the-fly as they’re ingested into Confident AI, giving you real-time production monitoring of your AI’s quality.

You can also trigger evaluations retrospectively on historical traces and spans.

How It Works

Online evaluations for traces and spans follow these steps:

1. You create a metric collection on Confident AI with the single-turn metrics you want to run.
2. You reference that metric collection by name via the metric_collection parameter — on the observe decorator/wrapper for span-level evals, or via update_current_trace for trace-level evals.
3. Inside your observed function, you set test case parameters on the span or trace using the update current span or update current trace function.
4. When the trace is sent to Confident AI, it runs the metrics in your collection against the test case data you provided.
5. Results appear on the trace/span in the Confident AI dashboard.

Map Test Case Parameters

To run evaluations, you first need to understand how trace span parameters map to test case parameters, which is what metrics use for evaluation. These parameters provide the data that metrics evaluate against.

The parameters you pass to the update current span or update current trace function map directly to test case parameters that metrics evaluate against:

All parameters are optional — you only need to provide the ones required by the metrics in your collection.

Evaluate Spans Online

Provide a metric collection on the span’s observe decorator/wrapper and set test case parameters via the update current span function:

1
from deepeval.tracing import observe, update_current_span
2
from openai import OpenAI
3
4
client = OpenAI()
5
6
@observe(metric_collection="My Collection")
7
def llm_app(query: str) -> str:
8
    res = client.chat.completions.create(
9
        model="gpt-4o",
10
        messages=[{"role": "user", "content": query}]
11
    ).choices[0].message.content
12
13
    update_current_span(input=query, output=res)
14
    return res
15
16
llm_app("Write me a poem.")

Evaluate Traces Online

Similar to spans, but use the update_current_trace/updateCurrentTrace function to set both the metric collection and test case parameters on the trace.

1
from deepeval.tracing import observe, update_current_trace
2
from openai import OpenAI
3
4
client = OpenAI()
5
6
@observe()
7
def llm_app(query: str) -> str:
8
    res = client.chat.completions.create(
9
        model="gpt-4o",
10
        messages=[{"role": "user", "content": query}]
11
    ).choices[0].message.content
12
13
    update_current_trace(
14
      input=query,
15
      output=res,
16
      metric_collection="My Collection"
17
    )
18
    return res
19
20
llm_app("Write me a poem.")

If you specify a metric collection but don’t provide sufficient test case parameters for a metric, it will show up as an error on Confident AI but won’t block or cause issues in your code.

Run Evals Offline

You can also trigger evaluations on traces and spans that have already been ingested. This is useful for re-evaluating with new metrics or running evals on historical data.

Evaluate a trace

1
from deepeval.tracing import evaluate_trace
2
3
evaluate_trace(trace_uuid="your-trace-uuid", metric_collection="Collection Name")

Your trace must already contain the necessary test case parameters — you cannot update them when evaluating retrospectively.

Evaluate a span

1
from deepeval.tracing import evaluate_span
2
3
evaluate_span(span_uuid="your-span-uuid", metric_collection="Collection Name")

The metric collection you provide must be a single-turn collection.

Examples

Quick quiz: Given the code below, which metric collection will Confident AI use for the trace, and which for the span?

1
from deepeval.tracing import observe, update_current_span, update_current_trace
2
3
@observe()
4
def outer_function():
5
6
    @observe(metric_collection="Collection 2")
7
    def inner_function():
8
        update_current_span(input="...", output="...")
9
        update_current_trace(
10
          input="...",
11
          output="...",
12
          metric_collection="Collection 1"
13
        )

Answer: "Collection 1" runs for the trace, and "Collection 2" runs for the span.

This is because:

1. The trace-level metric collection is set via update_current_trace(metric_collection=...), which can be called from any span
2. The inner function’s observe decorator sets "Collection 2" as the metric collection for that span
3. The update_current_span call updates the innermost active span (the “inner” span)
4. The update_current_trace call always updates the trace regardless of where it’s called

Next Steps

Now that you can evaluate individual traces and spans, learn how to evaluate entire conversations.