Build Test Runs from Traces

Overview

Before you ship a change, you want to know your app still clears the bar — that no regression is slipping through to production. That’s what a test run is for: a batch of test cases, each evaluated by the metrics you care about, with one pass-or-fail read at the end. This guide builds one from something your app already produces on every request — its traces.

The whole idea fits in one sentence: point a trace at an open test run, and it becomes an evaluated test case. Confident AI scores it against your metrics, files it under the run, and closes the run out when the last case lands.

Traces reach Confident AI two ways — over the API or through OpenTelemetry — so you can build a run in your most comfortable stack.

In this guide, you will:

• Open a test run and capture your traces as test cases inside this run.
• Send traces with the run id and a metric collection to evaluate them on cloud and automatically convert them to test cases.
• Evaluate the steps inside each trace — the retriever, the LLM call — not just the final answer with component level evals.
• Send test cases over OpenTelemetry with span attributes, for users who already have instrumentation in place.

By the end, you’ll have a repeatable way to benchmark quality before every release, built entirely from your app’s traces.

Prerequisites

Two things need to exist before your first test case lands:

• A Project API Key — CONFIDENT_API_KEY (e.g. confident_us_proj_...). Retrieve yours here.
• A single-turn metric collection in your project. This is what “good” means for your test cases — pass rates, quality thresholds, the metrics you’d gate a release on. You’ll refer to it by name, so note the exact name.

Send Traces via the API

Each step calls the Evals API — every snippet links straight to its reference, where you can try the call live. Prefer to instrument with OpenTelemetry instead? The OpenTelemetry section covers the same flow.

1
curl -X POST https://api.confident-ai.com/v1/test-runs \
2
     -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>" \
3
     -H "Content-Type: application/json" \
4
     -d '{
5
  "metricCollection": "Agent Quality",
6
  "identifier": "my-test-run"
7
}'

1
{
2
  "success": true,
3
  "data": {
4
    "id": "<TEST-RUN-ID>"
5
  },
6
  "link": "https://app.confident-ai.com/project/<PROJECT-ID>/test-runs/<TEST-RUN-ID>",
7
  "deprecated": false
8
}

1
curl -X POST https://api.confident-ai.com/v1/traces \
2
     -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>" \
3
     -H "Content-Type: application/json" \
4
     -d '{
5
  "uuid": "<TRACE-UUID>",
6
  "startTime": "2025-01-15T10:30:00Z",
7
  "endTime": "2025-01-15T10:30:05Z",
8
  "input": "What is the capital of France?",
9
  "output": "The capital of France is Paris.",
10
  "metricCollection": "Collection Name",
11
  "testRunId": "<TEST-RUN-ID>",
12
  "baseSpans": [
13
    {
14
      "endTime": "2025-01-15T10:30:02Z",
15
      "input": "What is the capital of France?",
16
      "name": "Agent",
17
      "output": "Let me look that up for you.",
18
      "startTime": "2025-01-15T10:30:00Z",
19
      "uuid": "<SPAN-UUID>"
20
    }
21
  ]
22
}'

1
curl -X POST https://api.confident-ai.com/v1/traces \
2
     -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>" \
3
     -H "Content-Type: application/json" \
4
     -d '{
5
  "uuid": "<TRACE-UUID>",
6
  "startTime": "2025-01-15T10:30:00Z",
7
  "endTime": "2025-01-15T10:30:05Z",
8
  "input": "What is the capital of France?",
9
  "output": "The capital of France is Paris.",
10
  "metricCollection": "Collection Name",
11
  "testRunId": "<TEST-RUN-ID>",
12
  "llmSpans": [
13
    {
14
      "endTime": "2025-01-15T10:30:05Z",
15
      "input": "Answer using the retrieved context.",
16
      "metricCollection": "LLM Collection Name",
17
      "model": "gpt-4o",
18
      "name": "generate_answer",
19
      "output": "The capital of France is Paris.",
20
      "parentUuid": "<RETRIEVER-SPAN-UUID>",
21
      "startTime": "2025-01-15T10:30:01Z",
22
      "uuid": "<LLM-SPAN-UUID>"
23
    }
24
  ],
25
  "retrieverSpans": [
26
    {
27
      "embedder": "text-embedding-3-small",
28
      "endTime": "2025-01-15T10:30:01Z",
29
      "input": "capital of France",
30
      "metricCollection": "Retriever Collection Name",
31
      "name": "retrieve_context",
32
      "retrievalContext": [
33
        "Paris is the capital and most populous city of France."
34
      ],
35
      "startTime": "2025-01-15T10:30:00Z",
36
      "uuid": "<RETRIEVER-SPAN-UUID>"
37
    }
38
  ]
39
}'

1
curl https://api.confident-ai.com/v1/test-runs/testRunId \
2
     -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>"

Send Traces over OpenTelemetry

If your app already emits OpenTelemetry spans, you can easily carry the same fields as span attributes and export the spans to Confident AI’s OTel endpoint.

The moves are identical to the walkthrough above. You still open the run over the API to get a testRunId; from there, everything rides on the spans you already emit. Point your OTLP/HTTP exporter at Confident AI’s OpenTelemetry endpoint, authenticated with the x-confident-api-key header:

$
https://otel.confident-ai.com/v1/traces

Then set two groups of attributes:

• On the root span — the test case itself: confident.trace.test_run_id, confident.trace.metric_collection, confident.trace.input, confident.trace.output, and confident.trace.name.
• On each child span — a component: confident.span.type, confident.span.name, confident.span.input, confident.span.output, and confident.span.metric_collection.

Attribute values are strings, so JSON-encode anything structured — a list of retrieved chunks, for instance.

OpenTelemetry is language agnostic, so you can always send the traces from your app natively and still get the same test runs as the API or regular SDK users. Here are a few examples in different languages:

1
use opentelemetry::{global, trace::{Tracer, TraceContextExt}, KeyValue};
2
use opentelemetry_otlp::WithExportConfig;
3
use opentelemetry_sdk::runtime;
4
use std::collections::HashMap;
5
6
fn init_tracer() {
7
    let mut headers = HashMap::new();
8
    headers.insert(
9
        "x-confident-api-key".to_string(),
10
        std::env::var("CONFIDENT_API_KEY").expect("CONFIDENT_API_KEY not set"),
11
    );
12
13
    opentelemetry_otlp::new_pipeline()
14
        .tracing()
15
        .with_exporter(
16
            opentelemetry_otlp::new_exporter()
17
                .http()
18
                .with_endpoint("https://otel.confident-ai.com/v1/traces")
19
                .with_headers(headers),
20
        )
21
        .install_batch(runtime::Tokio)
22
        .expect("failed to install tracer");
23
}
24
25
#[tokio::main]
26
async fn main() {
27
    init_tracer();
28
    let tracer = global::tracer("confident-test-run");
29
    let test_run_id = "your-test-run-id"; // the id from step 1
30
31
    let input = "Can I get a refund on my annual plan after two months?";
32
    let output = "Annual plans are refundable on a prorated basis within the first 30 days...";
33
34
    // Root span → the test case, evaluated end to end.
35
    tracer.in_span("Refund policy question", |cx| {
36
        let root = cx.span();
37
        root.set_attribute(KeyValue::new("confident.trace.test_run_id", test_run_id));
38
        root.set_attribute(KeyValue::new("confident.trace.metric_collection", "Agent Quality"));
39
        root.set_attribute(KeyValue::new("confident.trace.name", "Refund policy question"));
40
        root.set_attribute(KeyValue::new("confident.trace.input", input));
41
        root.set_attribute(KeyValue::new("confident.trace.output", output));
42
43
        // Child span → a component, evaluated on its own.
44
        tracer.in_span("generate_answer", |cx| {
45
            let span = cx.span();
46
            span.set_attribute(KeyValue::new("confident.span.type", "llm"));
47
            span.set_attribute(KeyValue::new("confident.span.name", "generate_answer"));
48
            span.set_attribute(KeyValue::new("confident.span.input", "Answer using the policy context..."));
49
            span.set_attribute(KeyValue::new("confident.span.output", output));
50
            span.set_attribute(KeyValue::new("confident.span.metric_collection", "Answer Quality"));
51
        });
52
    });
53
54
    global::shutdown_tracer_provider(); // flush before the process exits
55
}

Once the spans land, they join the same run and finalize just like the cases you sent over the API. Read the results the same way.

Field & Attribute Reference

The API body and the OpenTelemetry attributes carry the same test case — one names its fields in JSON, the other on spans. Use this to move between them:

Concepts

The walkthrough is enough to run a benchmark. This section covers what’s happening underneath — how a trace turns into a test case, and how a run knows when it’s done.

How a trace becomes a test case

A trace, on its own, is a record of something your app did in production. Two things promote it to a test case: a testRunId pointing at an open run, and a metricCollection to evaluate it with.

With both present, Confident AI derives a test case from the trace’s input and output, evaluates it with the collection you named, and files the result under the run. Everything else about the trace — its spans, its timings, its metadata — is kept intact, so opening a test case shows you the exact trace behind its score.

Evaluating the whole vs. the parts

The two levels answer different questions, and they run together on the same case:

• The whole — the metricCollection on the trace — asks: given this input, was the final answer good?
• The parts — a metricCollection on a span — asks: did this one step do its job? Was the retrieved context relevant, did the tool get the right arguments, did the model stay on policy?

Trace-level scores tell you whether a case failed. Component-level scores tell you where. Together they turn a red test case into a diagnosis.

When a run finishes

There’s no “done” call. A run stays open and keeps accepting test cases for as long as they keep arriving — which is what lets you stream cases in over the life of a test suite.

Confident AI closes the run automatically once it has been idle for 4 hours — four hours with no new or updated test case. At that point it computes the run’s pass and fail totals and marks it complete — which means it no longer accepts new test cases. Each case you send resets the idle clock, so an active run never closes mid-suite; when your suite stops, the run settles on its own about four hours after the last case lands.

Best Practices

A few habits keep trace-based runs trustworthy as your app and test suite grow:

• Keep one metric collection per level, and reuse it. Evaluate every test case in a run with the same trace-level collection so scores are comparable, and reserve dedicated collections for the components you care about.
• Own your trace uuids. Generate them yourself so you can tie a test case back to your own logs and re-send it deterministically when you need to.
• JSON-encode structured OpenTelemetry values. Attributes are strings — encode lists and objects (like a span’s retrieved context) as JSON so they come through intact.
• Always flush before exit. Batch exporters drop spans when a process dies first. Shut the tracer down at the end of a test script so every case is sent.
• Keep benchmarks out of production data. Run pre-deployment benchmarks in a dedicated project or environment so test cases don’t blur into your production observability.

FAQ

Next Steps

You can now benchmark quality before every release, built entirely from your app’s traces. To go further: