Introducing Confident Voice: Watch the Launch Video Now

Evaluate Traces & Spans

Run online and offline evaluations on individual traces and spans on the fly

Included on the Enterprise plan. Book a demo, opens in a new tab. Included on the Team plan. Included on the Starter plan. Not included on the Free plan.

Overview

Online evaluations let you run metrics on traces and spans on-the-fly as they're ingested into Confident AI, giving you real-time production monitoring of your AI's quality.

Online Evaluations on Confident AI

Evaluations run server-side on Confident AI, and you set them up there with an evaluation task in Workflows: pick traces or spans, a metric collection, and optionally filters and a sample rate (say 5% of production traffic and 100% of staging). Change which metrics run at any time, without redeploying. Your code only has to log the data the metrics read, with confident-trace.

An evaluation task on Traces

How It Works

Online evaluations for traces and spans follow these steps:

  1. You create a metric collection on Confident AI with the single-turn metrics you want to run.
  2. You create an evaluation task for Traces or Spans that runs the collection.
  3. Inside your span, you set test case parameters on the span or trace using the span and trace update helpers.
  4. When the trace is ingested, Confident AI matches the trace and its spans against your evaluation tasks.
  5. Results appear on the trace/span in the Confident AI dashboard.
sequenceDiagram
    participant App as Your App
    participant SDK as confident-trace
    participant CAI as Confident AI

    App->>SDK: Enter span
    SDK->>SDK: Create trace & span(s)
    App->>SDK: update_span / update_trace (input, output, etc.)
    App->>SDK: Span ends
    SDK->>CAI: Export trace with test case data
    CAI->>CAI: Match evaluation tasks
    CAI->>CAI: Run referenceless metrics against test case
    CAI->>CAI: Store results on trace/span

Map Test Case Parameters

To run evaluations, you first need to understand how trace and span parameters map to test case parameters, which is what metrics use for evaluation. These parameters provide the data that metrics evaluate against.

The parameters you pass to update_span / update_trace (or updateSpan / updateTrace) map directly to test case parameters:

Trace/Span ParameterTest Case ParameterDescription
inputinputThe input to your AI app
outputactual_outputThe output of your AI app
expected_outputexpected_outputThe expected output of your AI app
retrieval_contextretrieval_contextList of retrieved text chunks from a retrieval system
contextcontextList of ideal retrieved text chunks
tools_calledtools_calledList of tool call objects ({"name", "input", "output"}) actually used
expected_toolsexpected_toolsList of tool call objects you expected to be used

All parameters are optional — you only need to provide the ones required by the metrics in your collection. Tool calls are plain JSON objects, so you don't need to import anything to construct them.

Evaluate Spans Online

Create a Span evaluation task and restrict its Span Type to the spans you care about. Retriever spans are a natural fit: a task on Retriever spans running a contextual relevancy metric catches bad retrievals before they ever reach the LLM. Your code sets retrieval_context to the chunks you retrieved:

main.py
from openai import OpenAI
from confident_trace import init, span, update_span, shutdown

init()
client = OpenAI()

@span(type="retriever")
def retriever(query: str) -> list[str]:
    chunks = vector_store.search(query, top_k=3)
    update_span(
        input=query,
        output=chunks,
        retrieval_context=chunks,
    )
    return chunks

def llm_app(query: str) -> str:
    with span("llm_app", type="agent"):
        chunks = retriever(query)
        return client.chat.completions.create(
            model="gpt-4o",
            messages=[{"role": "user", "content": f"{query}\n\n{chunks}"}]
        ).choices[0].message.content

try:
    llm_app("Write me a poem.")
finally:
    shutdown()

A span task runs against every span it matches. A task matching all span types (Any) would evaluate the agent span, retriever span, and auto-instrumented LLM span, so use its Span Type to target only the spans you care about.

Evaluate Traces Online

Create a Trace evaluation task, and set the trace's test case parameters with update_trace / updateTrace. Trace-level evals are the right choice for end-to-end quality — "did the user get a good answer?" — since the trace input and output represent the whole request:

main.py
from openai import OpenAI
from confident_trace import init, span, update_trace, shutdown

init()
client = OpenAI()

def llm_app(query: str) -> str:
    with span("llm_app", type="agent"):
        chunks = retrieve(query)
        res = client.chat.completions.create(
            model="gpt-4o",
            messages=[{"role": "user", "content": f"{query}\n\n{chunks}"}]
        ).choices[0].message.content

        update_trace(
            input=query,
            output=res,
            retrieval_context=chunks,
        )
        return res

try:
    llm_app("Write me a poem.")
finally:
    shutdown()

If a task matches a trace or span but you haven't provided sufficient test case parameters for one of its metrics, that metric shows up as an error on Confident AI. It won't block or cause issues in your code — evaluation happens after export, entirely on the platform.

Select a Collection in Code

When a specific trace or span must always run the same collection, whatever your evaluation tasks say, set metric_collection / metricCollection in update_span / update_trace:

update_span(metric_collection="Retrieval Quality", input=query, retrieval_context=chunks)

A collection set in code takes precedence: evaluation tasks are skipped for that trace or span. This follows the same pattern as model cost resolution: the explicit OpenTelemetry-level value wins, while the project configuration provides the fallback. Prefer evaluation tasks otherwise, since changing a collection in code means redeploying.

Examples

Quick quiz: Given the code below, with one Trace evaluation task and one Span evaluation task (span type Any) both enabled, what does each task evaluate?

main.py
from confident_trace import span, update_span, update_trace

@span(type="tool")
def inner_function(query: str):
    result = lookup(query)
    update_span(input=query, output=result)
    update_trace(input=query, output="final answer")
    return result

def outer_function(query: str):
    with span("outer_function", type="agent"):
        return inner_function(query)

Answer: The Trace task evaluates one test case — input=query, actual_output="final answer" — because update_trace always writes to the trace regardless of where it's called. The Span task evaluates two spans: inner_function with input=query and actual_output=result, and outer_function with no test case parameters at all (which will error for any metric that needs them).

This is because:

  1. update_trace sets trace-level fields from anywhere inside the trace — it doesn't matter that it was called from a child span.
  2. update_span updates the innermost active span (inner_function), not its parent.
  3. A span task with type Any matches every span in the trace, including outer_function, which never had update_span called inside it. Restrict the task to Tool spans (or call update_span in outer_function) to fix this.

Next Steps

Now that you can evaluate individual traces and spans, learn how to evaluate entire conversations.

Ready to monitor AI in production?Connect traces, alerts, dashboards, and evals in one production workflowBook a demo

Last updated on

Built byConfident AI