Compare Models on One Agent

Overview

You can compare one deployed agent across multiple models by logging the model choice as trace metadata. Confident AI can then run the same online metrics for every trace and build dashboards that filter, split, and trend results by that metadata.

This guide shows the pattern across OpenAI Agents, LangGraph, Vercel AI SDK, and Strands Agents. The core idea is always the same: every request emits a trace, every trace includes a stable model_variant, and every model variant is scored by the same metric collection.

Here’s the thing: the model name captured on an LLM span is useful for debugging, but it is often too provider-specific to analyze — and it lives on the span, not the trace. Promoting a stable model_variant to the trace gives every dashboard one clean, product-level dimension to break down, filter, and trend by, even if the underlying provider model ID changes.

This same pattern compares far more than models. Anything you can label on a trace — prompt versions, temperature, retrievers, tool sets — can be compared the exact same way. See Compare Any Parameter to repeat this guide for a different variable.

What You’ll Build

By the end, you will have:

• A traced agent that records model_variant and model_id on every trace.
• A repeatable command to generate comparison traffic for each model variant.
• A metric collection that scores every variant with the same criteria.
• A dashboard that compares quality, trace volume, and latency across variants.
• A clear read of which model wins, not just on one lucky slice of traffic.

Prerequisites

You need a Confident AI project, a project API key, and credentials for whichever model provider your agent calls. For OpenAI-based examples, set OPENAI_API_KEY. For the Strands example, configure AWS credentials with access to the Bedrock model IDs you use.

Install the packages for the integration you are using:

$
python -m venv .venv
$
source .venv/bin/activate
$
pip install -U deepeval openai-agents

Then configure your project and provider credentials for that same integration:

$
export CONFIDENT_API_KEY="confident_us..."
$
export OPENAI_API_KEY="sk-..."

For EU projects, point OpenTelemetry export to the EU endpoint:

$
export CONFIDENT_OTEL_URL="https://eu.otel.confident-ai.com"

Set Up Tracing

Tracing is what feeds every dashboard in this guide. In three steps, you’ll instrument the agent so each request emits a trace tagged with model_variant, attach the metric collection that scores every variant, and verify the data shape before building any widgets.

Instrument the Agent

The most important implementation detail is where you attach metadata. Add model_variant to the trace, not just the LLM span, because dashboards commonly aggregate at the trace level: average trace score, trace count, trace latency, and trace-level online eval results.

1
import os
2
import sys
3
4
from agents import Agent, Runner, add_trace_processor, trace
5
from deepeval.openai_agents import DeepEvalTracingProcessor
6
from deepeval.tracing import update_current_trace
7
8
add_trace_processor(DeepEvalTracingProcessor())
9
10
MODEL_MAP = {"gpt-4o-mini": "gpt-4o-mini", "gpt-4o": "gpt-4o", "gpt-4.1": "gpt-4.1"}
11
12
13
def run_agent(user_input: str, model_variant: str) -> str:
14
    model_id = MODEL_MAP[model_variant]
15
    agent = Agent(
16
        name="Support Agent",
17
        instructions="Answer support questions clearly and safely.",
18
        model=model_id,
19
    )
20
21
    with trace(workflow_name="support-agent"):
22
        output = Runner.run_sync(agent, user_input).final_output
23
        update_current_trace(
24
            input=user_input,
25
            output=output,
26
            metric_collection="Agent Quality",
27
            metadata={
28
                "agent": "support-agent",
29
                "agent_version": os.getenv("AGENT_VERSION", "v2"),
30
                "model_variant": model_variant,
31
                "model_id": model_id,
32
                "rollout": os.getenv("ROLLOUT_NAME", "model-comparison"),
33
                "environment": os.getenv("APP_ENV", "production"),
34
            },
35
        )
36
        return output
37
38
39
if __name__ == "__main__":
40
    print(run_agent(sys.argv[2], sys.argv[1]))

$
export CONFIDENT_TRACE_FLUSH=1
$
export AGENT_VERSION="v2"
$
export ROLLOUT_NAME="model-comparison-smoke-test"
$
export APP_ENV="production"
$
$
for model in gpt-4o-mini gpt-4o gpt-4.1; do
$
  python openai_agents_model_compare.py "$model" \
>
    "A customer says their invoice doubled after upgrading. Explain what to check first."
$
done

Create Metrics

Use the same metric collection for every model variant so each is scored against identical criteria. Your project’s evaluation model — the LLM judge — is shared across every collection, so the judge itself is already consistent. The trap is scoring gpt-4o-mini and gpt-4o with different collections: you would be trending scores from two different rubrics, so the dashboard is no longer an apples-to-apples comparison.

1
update_current_trace(
2
    input=user_input,
3
    output=output,
4
    metric_collection="Agent Quality",
5
)

$
export CONFIDENT_TRACE_FLUSH=1
$
$
prompts=(
>
  "A customer cannot access invoices after changing teams. Help them troubleshoot."
>
  "Summarize why a trial user should upgrade, but do not mention unavailable features."
>
  "The integration failed with an OAuth callback error. Explain the likely cause."
>
  "A user asks for a refund after annual renewal. Give a careful support response."
>
)
$
$
for model in gpt-4o-mini gpt-4o gpt-4.1; do
$
  for prompt in "${prompts[@]}"; do
$
    python openai_agents_model_compare.py "$model" "$prompt"
$
  done
$
done

Verify the Traces

Before building dashboards, verify that the data shape is right. It is much easier to fix metadata and metric-collection names before you create five widgets around them.

1

Open the Observatory

In Confident AI, go to Observatory and filter for your agent or rollout:

• metadata.agent = support-agent
• metadata.rollout = model-comparison
• metadata.model_variant = gpt-4o for OpenAI-based examples, or metadata.model_variant = nova-pro for Strands

2

Inspect one trace

Open a trace and confirm four things:

• The trace input and output are populated.
• The trace metadata includes model_variant, model_id, agent_version, and rollout.
• The LLM span captured the provider model details from your integration.
• The trace has online eval results from Agent Quality, or shows a clear metric error you can fix.

Create the Dashboard

The traces and scores from the previous step feed the dashboard. Build it either way — pick Platform to click through the Confident AI UI, or CLI to run a reproducible script against the Dashboards API. Your choice sticks across every step below, so you only pick once.

Interpret Results

A model that scores higher is only the better choice if it also handled enough traffic to trust and kept latency acceptable. Read the three widgets together:

• Quality: Is the candidate’s average score higher over a meaningful date range?
• Volume: Does each variant have enough traces to trust the result? Low-volume variants can win by chance.
• Latency: Is P90 latency still acceptable for your product?

Compare Any Parameter

Here’s the key insight: nothing in this guide is actually model-specific. model_variant is just the metadata key every dashboard breaks down by. Swap it for any variable you want to A/B and the exact same workflow — one agent, one metric collection, three widgets — still applies. You’re not comparing models, you’re comparing whatever you label on the trace.

To compare something else, repeat the guide and change only two things:

• The metadata key you attach on the trace. Log prompt_version (or temperature, retriever, …) instead of, or alongside, model_variant.
• The dimension each widget breaks down by. Point the same dashboard filters at the new key.

Everything else stays identical. The new key is attached exactly like model_variant — as trace metadata:

1
update_current_trace(
2
    input=user_input,
3
    output=output,
4
    metric_collection="Agent Quality",
5
    metadata={
6
        "agent": "support-agent",
7
        "prompt_version": prompt_variant,  # the dimension you're now comparing
8
    },
9
)

Common parameters teams compare this way:

• Prompt versions — prompt_version: v3 vs v4.
• Decoding settings — temperature: 0.2 vs 0.7.
• Retrieval strategy — retriever: bm25 vs hybrid, or chunk_size: 512 vs 1024.
• Tool sets — toolset: minimal vs full.
• Agent versions — agent_version: v2 vs v3.

Chart Any Measure

And just like the breakdown dimension is swappable, so is the measure each widget plots. This guide charts quality (AVG_SCORE), trace volume (COUNT), and P90 latency (P90_LATENCY) — but that’s only three of many. Add a line with a different aggregation and you have a new comparison from the exact same traffic.

Measures you can break down by any dimension:

• Quality — AVG_SCORE, PASS_RATE, FAILURE_RATE, or AVG_RATING for any metric in your collection.
• Latency — AVG_LATENCY, P50_LATENCY, P90_LATENCY, P99_LATENCY.
• Cost & tokens — TOTAL_COST, AVG_COST, AVG_COST_PER_USER, INPUT_TOKENS, OUTPUT_TOKENS, TOTAL_TOKENS.
• Volume & users — COUNT, UNIQUE_USERS, UNIQUE_THREADS.
• Reliability — ERROR_COUNT, ERROR_RATE.

Best Practices

These are optional deep-dives once the core comparison is working.

• Compare one thing at a time. If the prompt, tools, retriever, and model all change at once, the dashboard cannot tell you what caused the difference.
• Keep metadata names consistent. Dashboards depend on exact metadata keys, so do not alternate between model, model_name, and model_variant.
• Separate product labels from provider IDs. Use model_variant for the decision people understand and model_id for exact reproducibility.
• Use enough traffic before deciding. Low-volume variants can look better or worse by chance. Compare over a stable time range.
• Watch quality and operations together. A model with a higher score but much worse latency may not be the better choice.

What to Track

The dashboard depends on consistent metadata. Start with these keys:

• model_variant — the comparison dimension, such as nova-lite or claude-sonnet.
• model_id — the exact provider model ID used for the request.
• agent — the stable application or agent name, such as support-agent.
• agent_version — the deployed agent version.
• rollout — the rollout, canary, or A/B test name.
• environment — production, staging, development, or testing.

Keep metadata values boring and predictable. model_variant="nova-pro" is easier to query than model_variant="Nova Pro - July canary (fast)". Put temporary rollout context in rollout, not in the model name.

Rollout Patterns

Three common ways to route traffic across models while comparing them:

• Shadow compare — send production traffic to the current model and run a copy through candidate models off the user-facing path. Log shadow traces with rollout=shadow-model-compare. High signal, but every request may call multiple models.
• Canary release — send a small percentage of real traffic to the candidate and label it rollout=canary-v3. The simplest production rollout; watch trace volume, since a 5% canary looks noisy until it has enough traffic.
• Segment routing — route a model to a specific segment such as internal users, one tenant, or one task type, and add metadata for that segment. Useful when the best model depends on the request.

Troubleshooting

Next Steps

Use this setup to compare model variants under one agent, then roll out the winner once quality, volume, and latency all look healthy.