Evaluating Your MCP Client in Code

Overview

An MCP client is the part of your agent that connects to MCP servers and decides which of their tools to call. When something goes wrong in an MCP setup, it is usually not the server. It is the client picking the wrong tool, passing the wrong arguments, or ignoring a tool that was right there the whole time.

This guide shows you how to catch that in code, before you ship. You describe the MCP servers your client talks to, record the tool calls it made during a run, and let deepeval score them. Results land on Confident AI as a test run.

You'll need:

• An MCP client you can invoke from Python, connected to at least one MCP server
• A list of metrics to evaluate with, such as MCP Use
• A CONFIDENT_API_KEY so the test run gets uploaded

How It Works

The key idea: you hand evaluate() the MCP servers your client can reach, and it works out which of the recorded tool calls were MCP calls and which were plain local functions.

1. Ask your MCP server what it exposes, so deepeval knows the tool surface
2. Run your client and record every tool call it made
3. Build a test case from the input, the output, and those tool calls
4. Pass the servers to evaluate(), which tags each tool call as MCP or FUNCTION and uploads the run

sequenceDiagram
    participant Your Code
    participant MCP Server
    participant MCP Client
    participant deepeval
    participant Confident AI

    Your Code->>MCP Server: list_tools()
    MCP Server-->>Your Code: Available tools

    Your Code->>MCP Client: Invoke with input
    loop For each tool the client picks
        MCP Client->>MCP Server: call_tool(name, args)
        MCP Server-->>MCP Client: Tool result
    end
    MCP Client-->>Your Code: actual_output + tools called

    Your Code->>Your Code: Build test case (input + actual_output + tools_called)
    Your Code->>deepeval: evaluate(test_cases, metrics, mcp_servers)
    deepeval->>deepeval: Tag each tool call MCP or FUNCTION
    deepeval->>Confident AI: Upload test run
    Confident AI-->>Your Code: Testing report link

That tagging step is the part worth understanding. Your agent probably calls a mix of MCP tools and ordinary local functions, and they all land in the same tools_called list. deepeval matches each call by name against the tools your MCP servers advertise, so MCP tool calls show up labelled separately on Confident AI instead of being lumped in with everything else.

Describe Your MCP Servers

How you do this depends on whether you own the server or are calling someone else's.

from mcp.server import MCPServer

server = MCPServer(name="GitHub")

@server.tool()
def search_issues(query: str) -> str:
    ...

@server.tool()
def create_issue(title: str, body: str) -> str:
    ...

from mcp import ClientSession
from deepeval.test_case import MCPServer

session = ClientSession(...)
await session.initialize()

tool_list = await session.list_tools()

mcp_server = MCPServer(
    server_name="GitHub",
    transport="streamable-http",
    available_tools=tool_list.tools,
)

Record What Your Client Called

As your client works through a request, capture each tool call as a ToolCall. Name, arguments, and result are what the metrics reason over:

from deepeval.test_case import ToolCall

tools_called = []

result = await session.call_tool(tool_name, tool_args)

tools_called.append(
    ToolCall(
        name=tool_name,
        input_parameters=tool_args,
        output=result.content,
    )
)

Do not worry about marking which ones were MCP calls. That happens automatically in the next step. Record local function calls into the same list and they will be sorted out for you.

Then build your test case as usual:

from deepeval.test_case import LLMTestCase

test_case = LLMTestCase(
    input="Find the open bug about rate limiting and file a follow-up",
    actual_output=agent_response,
    tools_called=tools_called,
)

Run the Evaluation

Pass your servers to evaluate() once and every test case in the run picks them up:

from deepeval import evaluate
from deepeval.metrics import MCPUseMetric

evaluate(
    test_cases=[test_case],
    metrics=[MCPUseMetric()],
    mcp_servers=[server],
)

MCPUseMetric scores two things: whether your client used the primitives available to it sensibly, and whether it passed the right arguments. Both need to know the tool surface, which is exactly what mcp_servers gives them.

Open the report link printed at the end of the run. Your MCP tool calls are labelled as MCP, and anything your agent did locally shows up as a plain function call, so you can see at a glance whether the client reached for the right surface.

Multi-Turn MCP Clients

For a chatbot or agent that holds a conversation, put the tool calls on the turn where they happened and use MultiTurnMCPUseMetric:

from deepeval import evaluate
from deepeval.metrics import MultiTurnMCPUseMetric
from deepeval.test_case import ConversationalTestCase, Turn, ToolCall

test_case = ConversationalTestCase(
    turns=[
        Turn(role="user", content="Any open bugs about rate limiting?"),
        Turn(
            role="assistant",
            content="Found one, issue #42.",
            tools_called=[ToolCall(name="search_issues", input_parameters={"query": "rate limiting"})],
        ),
        Turn(role="user", content="File a follow-up for it"),
        Turn(
            role="assistant",
            content="Filed issue #43.",
            tools_called=[ToolCall(name="create_issue", input_parameters={"title": "Follow-up to #42"})],
        ),
    ],
)

evaluate(
    test_cases=[test_case],
    metrics=[MultiTurnMCPUseMetric()],
    mcp_servers=[server],
)

Tagging works per turn, so a five turn conversation where the client only reached for MCP tools twice shows exactly which two.

Second Option: No-Code with an AI Connection

Everything above assumes you want to drive the evaluation from your own code. You don't have to.

If your MCP client is deployed and reachable, you can point Confident AI at it with an AI Connection, register your MCP servers in project settings, and run the whole thing from the platform against a dataset. Confident AI invokes your client, captures the output and the tool calls, and scores the run with your metric collection. No test case construction, no evaluate() call, nothing to wire up in Python.

That path is worth a walkthrough of its own, so we've kept it separate:

• Evaluate MCP Servers covers the full AI Connection setup for MCP, including tracing and production evals
• Run an evaluation covers the generic no-code flow, from picking a dataset to reading the report

Pick whichever fits how you work. Code-driven gives you tighter control and fits naturally into CI. No-code gets you a report without touching your app.

Next Steps

Last updated on August 13, 2026