Evaluate MCP Servers

Overview

Model Context Protocol (MCP) is an open standard for connecting AI models to external tools, data sources, and services. An MCP setup has a host (your agent) that orchestrates calls to one or more MCP servers, each exposing a set of tools.

Evaluating MCP comes down to one question: did your agent call the right tools, with the right inputs, to produce a good result? This guide walks through benchmarking your agent before you ship, then adding tracing for test runs and production evaluations.

Evaluating MCP

Before deployment, benchmark your MCP agent against a dataset of goldens. Confident AI sends each input to your AI Connection, captures the generated output and MCP tool interactions, then scores the run with your metric collection.

You’ll need two things set up on Confident AI first:

• A dataset of goldens — what you want to test your MCP agent on.
• An AI Connection — pointed at your deployed MCP host. Confident AI uses it to run the agent and capture the generated output, tool calls, and tool arguments.

With both in place, set up your metrics, connect your MCP server, and run the evaluation:

1

Create your metric collection

Under Project > Metrics > Collections, create a metric collection — the set of metrics for the run. Start with MCP Use or Argument Correctness, then add an output-quality metric if final-answer quality matters.

Not sure what to add? See Choosing Your Metrics.

2

Connect your MCP server

In Project Settings > MCP Servers, click Add server and enter its name and connection details. Confident AI fetches the tool definitions for that server version and uses them to label matching MCP tools during evaluation.

3

Run the evaluation from your dataset

Navigate to Project > Datasets, open your dataset, and click Evaluate. Select the metric collection you created, choose the AI Connection that runs your agent, attach the server you connected under MCP Servers, then click Run Evaluation.

Like AI Connections and prompts, attached MCP servers are saved as test-run hyperparameters. Confident AI also labels matching tool calls as MCP tools, so you do not have to mark them manually.

4

View your MCP eval results

Open the test run and select a test case. Tool calls appear under Tools Used, and matching calls are tagged MCP with their inputs and output inline. The metric scores show how the run performed.

Done ✅. You now have a repeatable pre-deployment benchmark for your MCP agent.

Advanced Usage

Once your pre-deployment benchmark is in place, add tracing for full tool-path visibility. Traces link test runs to the exact tool path behind each result and power live production evaluations.

Tracing MCP

Your host calls MCP servers through MCP clients. In most deployments, each server runs in a separate process:

For tracing, that client/server boundary matters: the host owns the active trace, while each server needs the trace context passed to it explicitly.

The host and each server talk over JSON-RPC 2.0 — a lightweight “call a named method with some params, get a result back” protocol. Calling a tool is a JSON-RPC request with the method tools/call:

1
{
2
  "jsonrpc": "2.0",
3
  "method": "tools/call",
4
  "params": {
5
    "name": "search_docs",
6
    "arguments": { "query": "refund policy for annual plans" }
7
  },
8
  "id": 1
9
}

Here method is the operation, params.name is the tool, and params.arguments are the tool inputs. In this example, the host asks a docs MCP server for context before the agent answers. The top-level id is only the JSON-RPC request ID — it is not the trace ID.

These messages travel over a transport — stdio for local servers or HTTP for remote ones. Either way, the host and server are usually different processes, so the server cannot automatically see the host’s active trace. The host has to send trace context with the tools/call request.

To see every tool call from a single request unified under one trace, you need distributed tracing, which carries trace context across those process boundaries.

At a high level, the host injects W3C trace context into each MCP call’s metadata, each server extracts it to create child spans, and all processes export to Confident AI using the same CONFIDENT_API_KEY.

$
export CONFIDENT_API_KEY="your-project-api-key"
$
export OTEL_EXPORTER_OTLP_ENDPOINT="https://otel.confident-ai.com"

1
{
2
  "jsonrpc": "2.0",
3
  "method": "tools/call",
4
  "params": {
5
    "name": "search_docs",
6
    "arguments": { "query": "refund policy for annual plans" },
7
    "_meta": {
8
      "traceparent": "00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01"
9
    }
10
  },
11
  "id": 1
12
}

1
async def call_tool_with_tracing(session, tool_name: str, arguments: dict):
2
    with tracer.start_as_current_span(f"mcp-tool-{tool_name}") as span:
3
        span.set_attribute("confident.span.type", "tool")
4
        span.set_attribute("confident.tool.name", tool_name)
5
        span.set_attribute("confident.span.input", json.dumps(arguments))
6
7
        # Inject trace context so the MCP server joins this trace
8
        trace_meta = {}
9
        propagator.inject(trace_meta)
10
11
        result = await session.call_tool(tool_name, arguments=arguments, _meta=trace_meta)
12
13
        span.set_attribute("confident.span.output", json.dumps(result.content))
14
        return result

Once your host and servers emit unified traces, link them to your AI Connection. During an evaluation, each generated test case can include a link to the trace that produced it, so you can inspect the exact tool path behind the score.

Evaluate MCP in Production

Pre-deployment benchmarks catch regressions before you ship, but real users send inputs you never tested. Online evaluations close that loop by scoring MCP traces as Confident AI ingests them, so tool-usage quality is monitored continuously in production.

Once your MCP host and servers are tracing to Confident AI, turn on metrics like MCP Use for tool spans and Task Completion for the whole trace. Every production run is scored automatically — no dataset required — and thresholds can alert you when tool-selection quality drops.

Concepts

The workflow above is enough to run an evaluation. This section explains why MCP evaluation matters, how a run works end to end, which metrics to choose, and the practices that keep it reliable.

Why Evaluate MCP?

MCP turns your agent into a tool-using system. The model still writes the final answer, but answer quality depends on the path it took: which tool it chose, what arguments it passed, and whether it used the returned data correctly.

MCP evaluation has to look beyond “was the final answer good?” What you measure depends on how much work your MCP agent does — from one tool call to a multi-step workflow:

• Tool calling — did the agent call the right tool? Tool Correctness compares the tools called against the ones you expected. Best when your MCP is essentially a tool-calling interface.
• Tool usage — how well the LLM uses those tools. Argument Correctness checks whether each call’s inputs fit the request, and MCP Use scores tool selection and argument quality together — no labels needed. Most MCP agents start here.
• Workflow — for MCPs that do more than wrap a tool call, like summarizers or multi-step agents. Task Completion checks whether the run met the user’s goal, and Step Efficiency whether it got there without detours.

That’s the what — see Choosing Your Metrics to turn these categories into a metric collection.

How It Works

When you start a dataset run, Confident AI does the following:

An MCP evaluation does not just check the final answer. It can also score how the agent interacted with the MCP server:

• Connected MCP server provides the tool definitions Confident AI uses to label matching calls as MCP tool calls.
• AI Connection runs your agent and returns the generated output plus tools_called.
• Metrics score the generated test case based on what they measure: tool selection, arguments, final output, or the full workflow.

For referenceless MCP metrics, you do not label expected_tools for every golden. The dataset provides inputs, the AI Connection generates outputs and tool interactions, and the connected server provides MCP tool definitions.

Choosing Your Metrics

A metric collection is the set of metrics every run is scored with. It defines what “good” means for your MCP agent. The Why Evaluate MCP? section grouped metrics by what they measure; here’s how to choose what belongs in the collection.

Start with one referenceless, MCP-native metric and add more only when they cover a real risk:

• MCP Use — the default. It scores tool selection and argument quality using the connected server’s tool definitions, with no labels required.
• Argument Correctness — a sharper lens on inputs: did each call pass arguments that fit the request? Pair it with MCP Use when argument quality is your main risk.
• Task Completion — did the run accomplish the user’s goal? Add it when your agent chains multiple calls and you need to score the outcome.
• Step Efficiency — did it get there without redundant calls or detours? Add it when cost or latency matters.
• An output-quality metric like Answer Relevancy — useful when the final response matters as much as the tool path.

Have labeled data? Add Tool Correctness. It compares the tools called against a per-golden expected_tools list. Everything above is referenceless, so most teams start there and layer Tool Correctness on top once they have labels.

Best Practices

A few practices that keep MCP evaluation trustworthy as agents, tools, and servers evolve:

• Isolate evals from real side effects. MCP tools take real actions — writing to databases, sending emails, moving money. Point evaluation runs at sandboxed servers, mock tools, or read-only credentials so benchmarking never mutates production systems.
• Pin server versions and re-baseline on every upgrade. A new MCP server release can rename tools or change schemas. Evaluate against a fixed version, and treat a server upgrade like a code change: re-run your benchmark before it reaches production.
• Seed your dataset from production traces. Hand-written goldens miss how users actually behave. Promote real edge cases and past failures from traces into the dataset so fixed regressions stay fixed.
• Sample each input multiple times. Agents take different tool paths on identical inputs. Use multi-generation and judge on pass rates, not a single lucky — or unlucky — run.
• Gate releases on your benchmark. Attach thresholds to your metrics and run the benchmark in CI so a drop in tool-selection quality blocks the deploy.
• Track cost and latency, not just correctness. An agent that reaches the right answer through twenty redundant tool calls is still a production problem. Watch step count and latency alongside quality.

FAQ

Next Steps

You benchmarked your MCP agent before deployment, then saw how to trace it for test runs and production evals. To go deeper: