Evaluate Traces & Spans

Overview

Online evaluations let you run metrics on traces and spans on-the-fly as they're ingested into Confident AI, giving you real-time production monitoring of your AI's quality.

You can also trigger evaluations retrospectively on historical traces and spans.

How It Works

Online evaluations for traces and spans follow these steps:

1. You create a metric collection on Confident AI with the single-turn metrics you want to run.
2. You reference that metric collection by name via the metric_collection parameter — on the observe decorator/wrapper for span-level evals, or via update_current_trace for trace-level evals.
3. Inside your observed function, you set test case parameters on the span or trace using the update current span or update current trace function.
4. When the trace is sent to Confident AI, it runs the metrics in your collection against the test case data you provided.
5. Results appear on the trace/span in the Confident AI dashboard.

sequenceDiagram
    participant App as Your App
    participant SDK as Evals API/DeepEval
    participant CAI as Confident AI

    App->>SDK: Call observed function
    SDK->>SDK: Create trace & span(s)
    App->>SDK: Set test case parameters (input, output, etc.)
    App->>SDK: Function returns
    SDK->>CAI: Send trace with test case data
    CAI->>CAI: Look up metric collection
    CAI->>CAI: Run referenceless metrics against test case
    CAI->>CAI: Store results on trace/span

Map Test Case Parameters

To run evaluations, you first need to understand how trace span parameters map to test case parameters, which is what metrics use for evaluation. These parameters provide the data that metrics evaluate against.

The parameters you pass to the update current span or update current trace function map directly to test case parameters that metrics evaluate against:

All parameters are optional — you only need to provide the ones required by the metrics in your collection.

Evaluate Spans Online

Provide a metric collection on the span's observe decorator/wrapper and set test case parameters via the update current span function:

from deepeval.tracing import observe, update_current_span
from openai import OpenAI

client = OpenAI()

@observe(metric_collection="My Collection")
def llm_app(query: str) -> str:
    res = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": query}]
    ).choices[0].message.content

    update_current_span(input=query, output=res)
    return res

llm_app("Write me a poem.")

import { observe, updateCurrentSpan } from "deepeval/tracing";

const generate = async (prompt: string): Promise<string> => {
  updateCurrentSpan({ input: prompt, output: "LLM response" });
  return "LLM response";
};

const observedGenerate = observe({
  type: "llm",
  metricCollection: "My Collection",
  fn: generate,
});

Evaluate Traces Online

Similar to spans, but use the update_current_trace/updateCurrentTrace function to set both the metric collection and test case parameters on the trace.

from deepeval.tracing import observe, update_current_trace
from openai import OpenAI

client = OpenAI()

@observe()
def llm_app(query: str) -> str:
    res = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": query}]
    ).choices[0].message.content

    update_current_trace(
      input=query,
      output=res,
      metric_collection="My Collection"
    )
    return res

llm_app("Write me a poem.")

import { observe, updateCurrentTrace } from "deepeval/tracing";

const generate = async (prompt: string): Promise<string> => {
  updateCurrentTrace({
    input: prompt,
    output: "LLM response",
    metricCollection: "My Collection",
  });
  return "LLM response";
};

const observedGenerate = observe({
  type: "llm",
  fn: generate,
});

If you specify a metric collection but don't provide sufficient test case parameters for a metric, it will show up as an error on Confident AI but won't block or cause issues in your code.

Run Evals Offline

You can also trigger evaluations on traces and spans that have already been ingested. This is useful for re-evaluating with new metrics or running evals on historical data.

Evaluate a trace

from deepeval.tracing import evaluate_trace

evaluate_trace(trace_uuid="your-trace-uuid", metric_collection="Collection Name")

import { evaluateTrace } from "deepeval/tracing";

await evaluateTrace({
  traceUuid: "your-trace-uuid",
  metricCollection: "Collection Name",
});

Your trace must already contain the necessary test case parameters — you cannot update them when evaluating retrospectively.

Evaluate a span

from deepeval.tracing import evaluate_span

evaluate_span(span_uuid="your-span-uuid", metric_collection="Collection Name")

import { evaluateSpan } from "deepeval/tracing";

await evaluateSpan({
  spanUuid: "your-span-uuid",
  metricCollection: "Collection Name",
});

The metric collection you provide must be a single-turn collection.

Examples

Quick quiz: Given the code below, which metric collection will Confident AI use for the trace, and which for the span?

from deepeval.tracing import observe, update_current_span, update_current_trace

@observe()
def outer_function():

    @observe(metric_collection="Collection 2")
    def inner_function():
        update_current_span(input="...", output="...")
        update_current_trace(
          input="...",
          output="...",
          metric_collection="Collection 1"
        )

import {
  observe,
  updateCurrentSpan,
  updateCurrentTrace,
} from "deepeval/tracing";

const innerFunction = () => {
  updateCurrentSpan({ input: "...", output: "..." });
  updateCurrentTrace({ input: "...", output: "..." });
};
const observedInner = observe({
  metricCollection: "Collection 2",
  fn: innerFunction,
});

const outerFunction = () => {
  observedInner();
};
const observedOuter = observe({
  metricCollection: "Collection 1",
  fn: outerFunction,
});

Answer: "Collection 1" runs for the trace, and "Collection 2" runs for the span.

This is because:

1. The trace-level metric collection is set via update_current_trace(metric_collection=...), which can be called from any span
2. The inner function's observe decorator sets "Collection 2" as the metric collection for that span
3. The update_current_span call updates the innermost active span (the "inner" span)
4. The update_current_trace call always updates the trace regardless of where it's called

Next Steps

Now that you can evaluate individual traces and spans, learn how to evaluate entire conversations.

Last updated on July 27, 2026