# Evaluate Traces & Spans

Run online and offline evaluations on individual traces and spans on the fly

## Overview

Online evaluations let you run metrics on traces and spans on-the-fly as they're ingested into Confident AI, giving you real-time production monitoring of your AI's quality.

Product preview: Support Agent Health dashboard with metric data by metric, trace pass rate and count, ratings by end user, sentiment, span cost and trace names

*Online Evaluations on Confident AI*

Evaluations run **server-side** on Confident AI, and you set them up there with an [evaluation task](/docs/llm-tracing/workflows/evaluation) in [Workflows](/docs/llm-tracing/workflows): pick traces or spans, a metric collection, and optionally filters and a sample rate (say 5% of production traffic and 100% of staging). Change which metrics run at any time, without redeploying. Your code only has to log the data the metrics read, with [`confident-trace`](https://github.com/confident-ai/confident-trace).

Product preview: Edit evaluation rule panel for Support quality: enabled, Trace data model, the Support Agent Metrics collection, and optional configs

*An evaluation task on Traces*

> For evaluating multi-turn conversations (threads), see [Evaluate
> Threads](/docs/llm-tracing/evaluate-threads).

## How It Works

Online evaluations for traces and spans follow these steps:

1. You [create a metric collection](/docs/metrics/metric-collections) on Confident AI with the **single-turn** metrics you want to run.
2. You create an [evaluation task](/docs/llm-tracing/workflows/evaluation) for **Traces** or **Spans** that runs the collection.
3. Inside your `span`, you set [test case parameters](/docs/resources/core-concepts/test-cases-goldens-datasets) on the span or trace using the span and trace update helpers.
4. When the trace is ingested, Confident AI matches the trace and its spans against your evaluation tasks.
5. Results appear on the trace/span in the Confident AI dashboard.

```mermaid
sequenceDiagram
    participant App as Your App
    participant SDK as confident-trace
    participant CAI as Confident AI

    App->>SDK: Enter span
    SDK->>SDK: Create trace & span(s)
    App->>SDK: update_span / update_trace (input, output, etc.)
    App->>SDK: Span ends
    SDK->>CAI: Export trace with test case data
    CAI->>CAI: Match evaluation tasks
    CAI->>CAI: Run referenceless metrics against test case
    CAI->>CAI: Store results on trace/span
```

> Only **referenceless** metrics in your metric collection will run during
> tracing. [Referenceless metrics](/docs/metrics/introduction)
> can evaluate your LLM's performance without requiring reference data (like
> `expected_output` or `expected_tools`). Non-referenceless metrics are silently
> skipped.

## Map Test Case Parameters

To run evaluations, you first need to understand how trace and span parameters map to [test case parameters](/docs/resources/core-concepts/test-cases-goldens-datasets), which is what metrics use for evaluation. These parameters provide the data that metrics evaluate against.

The parameters you pass to `update_span` / `update_trace` (or `updateSpan` / `updateTrace`) map directly to test case parameters:

#### Python

| Trace/Span Parameter | Test Case Parameter | Description                                                             |
| -------------------- | ------------------- | ----------------------------------------------------------------------- |
| `input`              | `input`             | The input to your AI app                                                |
| `output`             | `actual_output`     | The output of your AI app                                               |
| `expected_output`    | `expected_output`   | The expected output of your AI app                                      |
| `retrieval_context`  | `retrieval_context` | List of retrieved text chunks from a retrieval system                   |
| `context`            | `context`           | List of ideal retrieved text chunks                                     |
| `tools_called`       | `tools_called`      | List of tool call objects (`{"name", "input", "output"}`) actually used |
| `expected_tools`     | `expected_tools`    | List of tool call objects you expected to be used                       |

#### TypeScript

| Trace/Span Parameter | Test Case Parameter | Description                                                         |
| -------------------- | ------------------- | ------------------------------------------------------------------- |
| `input`              | `input`             | The input to your AI app                                            |
| `output`             | `actualOutput`      | The output of your AI app                                           |
| `expectedOutput`     | `expectedOutput`    | The expected output of your AI app                                  |
| `retrievalContext`   | `retrievalContext`  | List of retrieved text chunks from a retrieval system               |
| `context`            | `context`           | List of ideal retrieved text chunks                                 |
| `toolsCalled`        | `toolsCalled`       | List of tool call objects (`{ name, input, output }`) actually used |
| `expectedTools`      | `expectedTools`     | List of tool call objects you expected to be used                   |

All parameters are **optional** — you only need to provide the ones required by the metrics in your collection. Tool calls are plain JSON objects, so you don't need to import anything to construct them.

> Each metric requires different test case parameters. For details on what each
> metric needs, refer to the [official DeepEval
> documentation](https://deepeval.com/docs/metrics-introduction).

## Evaluate Spans Online

Create a **Span** evaluation task and restrict its **Span Type** to the spans you care about. Retriever spans are a natural fit: a task on **Retriever** spans running a contextual relevancy metric catches bad retrievals before they ever reach the LLM. Your code sets `retrieval_context` to the chunks you retrieved:

#### Python

```python title="main.py" {10-14}
from openai import OpenAI
from confident_trace import init, span, update_span, shutdown

init()
client = OpenAI()

@span(type="retriever")
def retriever(query: str) -> list[str]:
    chunks = vector_store.search(query, top_k=3)
    update_span(
        input=query,
        output=chunks,
        retrieval_context=chunks,
    )
    return chunks

def llm_app(query: str) -> str:
    with span("llm_app", type="agent"):
        chunks = retriever(query)
        return client.chat.completions.create(
            model="gpt-4o",
            messages=[{"role": "user", "content": f"{query}\n\n{chunks}"}]
        ).choices[0].message.content

try:
    llm_app("Write me a poem.")
finally:
    shutdown()
```

#### TypeScript

```typescript title="src/index.ts" {9-13} maxLines={0}
import OpenAI from "openai";
import { init, span, withSpan, updateSpan } from "confident-trace";

const runtime = init();
const openai = new OpenAI();

const retriever = span({ name: "retriever", type: "retriever" }, async (query: string) => {
  const chunks = await vectorStore.search(query, { topK: 3 });
  updateSpan({
    input: query,
    output: chunks,
    retrievalContext: chunks,
  });
  return chunks;
});

const llmApp = async (query: string) => {
  return withSpan({ name: "llm_app", type: "agent" }, async () => {
    const chunks = await retriever(query);
    const res = await openai.chat.completions.create({
      model: "gpt-4o",
      messages: [{ role: "user", content: `${query}\n\n${chunks.join("\n")}` }],
    });
    return res.choices[0].message.content ?? "";
  });
};

try {
  await llmApp("Write me a poem.");
} finally {
  await runtime.shutdown();
}
```

A span task runs against **every span** it matches. A task matching all span types (**Any**) would evaluate the agent span, retriever span, and auto-instrumented LLM span, so use its **Span Type** to target only the spans you care about.

> You don't need to wrap provider calls yourself. The OpenAI integration already
> records the prompt and completion as the LLM span's input and output, so a span
> task restricted to the **LLM** type works with no extra code — and adding your
> own `llm` span around the same call would duplicate it. See [span
> types](/docs/llm-tracing/features/span-types) for the type-specific fields.

## Evaluate Traces Online

Create a **Trace** evaluation task, and set the trace's test case parameters with `update_trace` / `updateTrace`. Trace-level evals are the right choice for end-to-end quality — "did the user get a good answer?" — since the trace input and output represent the whole request:

#### Python

```python title="main.py" {15-19}
from openai import OpenAI
from confident_trace import init, span, update_trace, shutdown

init()
client = OpenAI()

def llm_app(query: str) -> str:
    with span("llm_app", type="agent"):
        chunks = retrieve(query)
        res = client.chat.completions.create(
            model="gpt-4o",
            messages=[{"role": "user", "content": f"{query}\n\n{chunks}"}]
        ).choices[0].message.content

        update_trace(
            input=query,
            output=res,
            retrieval_context=chunks,
        )
        return res

try:
    llm_app("Write me a poem.")
finally:
    shutdown()
```

#### TypeScript

```typescript title="src/index.ts" {16-20} maxLines={0}
import OpenAI from "openai";
import { init, withSpan, updateTrace } from "confident-trace";

const runtime = init();
const openai = new OpenAI();

const llmApp = async (query: string) => {
  return withSpan({ name: "llm_app", type: "agent" }, async () => {
    const chunks = await retrieve(query);
    const res = await openai.chat.completions.create({
      model: "gpt-4o",
      messages: [{ role: "user", content: `${query}\n\n${chunks.join("\n")}` }],
    });
    const data = res.choices[0].message.content ?? "";

    updateTrace({
      input: query,
      output: data,
      retrievalContext: chunks,
    });
    return data;
  });
};

try {
  await llmApp("Write me a poem.");
} finally {
  await runtime.shutdown();
}
```

\> You can run online evals on traces and spans at the same time: create one evaluation task for each data model.

If a task matches a trace or span but you haven't provided sufficient test case parameters for one of its metrics, that metric shows up as an error on Confident AI. It won't block or cause issues in your code — evaluation happens after export, entirely on the platform.

> `update_span` and `update_trace` need an active span to write to — outside of
> any `span` (or after it has ended) they silently do nothing, so if your evals
> are erroring with "missing input", first check that the call is inside the span
> body. If you have no span to call `update_trace` from, open a `trace_context` /
> `traceContext` around the call to set trace fields known up front (though the
> `output` still needs a span); see [set trace attributes without a
> span](/docs/llm-tracing/features/trace-context#update-trace-properties). See
> [`update_trace` vs
> `update_span`](/docs/llm-tracing/troubleshooting#update_trace-vs-update_span)
> if you're unsure which one to use.

## Select a Collection in Code

When a specific trace or span must always run the same collection, whatever your evaluation tasks say, set `metric_collection` / `metricCollection` in `update_span` / `update_trace`:

#### Python

```python
update_span(metric_collection="Retrieval Quality", input=query, retrieval_context=chunks)
```

#### TypeScript

```typescript
updateSpan({ metricCollection: "Retrieval Quality", input: query, retrievalContext: chunks });
```

A collection set in code takes precedence: evaluation tasks are skipped for that trace or span. This follows the same pattern as [model cost resolution](/docs/llm-tracing/features/token-usage-cost): the explicit OpenTelemetry-level value wins, while the project configuration provides the fallback. Prefer evaluation tasks otherwise, since changing a collection in code means redeploying.

## Examples

**Quick quiz:** Given the code below, with one **Trace** evaluation task and one **Span** evaluation task (span type **Any**) both enabled, what does each task evaluate?

#### Python

```python title="main.py"
from confident_trace import span, update_span, update_trace

@span(type="tool")
def inner_function(query: str):
    result = lookup(query)
    update_span(input=query, output=result)
    update_trace(input=query, output="final answer")
    return result

def outer_function(query: str):
    with span("outer_function", type="agent"):
        return inner_function(query)
```

#### TypeScript

```typescript title="src/index.ts" maxLines={0}
import { span, withSpan, updateSpan, updateTrace } from "confident-trace";

const innerFunction = span({ name: "inner_function", type: "tool" }, async (query: string) => {
  const result = await lookup(query);
  updateSpan({ input: query, output: result });
  updateTrace({ input: query, output: "final answer" });
  return result;
});

const outerFunction = async (query: string) => {
  return withSpan({ name: "outer_function", type: "agent" }, async () => {
    return innerFunction(query);
  });
};
```

**Answer:** The **Trace** task evaluates one test case — `input=query`, `actual_output="final answer"` — because `update_trace` always writes to the trace regardless of where it's called. The **Span** task evaluates **two** spans: `inner_function` with `input=query` and `actual_output=result`, and `outer_function` with no test case parameters at all (which will error for any metric that needs them).

This is because:

1. `update_trace` sets trace-level fields from anywhere inside the trace — it doesn't matter that it was called from a child span.
2. `update_span` updates the innermost active span (`inner_function`), not its parent.
3. A span task with type **Any** matches every span in the trace, including `outer_function`, which never had `update_span` called inside it. Restrict the task to **Tool** spans (or call `update_span` in `outer_function`) to fix this.

## Next Steps

Now that you can evaluate individual traces and spans, learn how to evaluate entire conversations.

#### [Evaluate Threads](/docs/llm-tracing/evaluate-threads)

Run evaluations on multi-turn conversations and understand how thread evals
differ from trace evals.

#### [Evaluation Task](/docs/llm-tracing/workflows/evaluation)

Every field of an evaluation task: data model, span type, metric collection,
filters, sample rate, and thread settings.