MCP Servers

Why Connect to MCP Servers?

Connect your Model Context Protocol (MCP) servers to make their available tools known to Confident AI. During an evaluation, Confident AI uses these tool definitions to identify MCP tool calls and run tool-related metrics against your agent's behavior. That lets you:

• Evaluate MCP clients against the servers they really use. Connect the same MCP servers your agent has access to, and evaluate it with the tools it has in production. This works with AI Connections too: attach the servers on the connection's Resources tab to send them to a model provider such as the OpenAI Responses API, which calls their tools directly, or evaluate a third-party SaaS agent that connects to the same MCP servers.
• See whether your agent called the right tools. Knowing every tool a server offers, Confident AI recognizes MCP tool calls and scores tool selection and arguments with metrics such as MCP Use and Argument Correctness.
• Compare what the agent could do with what it did. Each test case shows the tools available to your agent alongside the tool calls it made.

Connect an MCP server

Open MCP Servers from the project sidebar (under More in Build), then click Add server. Give the server a recognizable Name, an optional Description, and select the Transport it supports, then click Create.

HTTP

Use HTTP for a remote MCP server and enter its endpoint URL. Under Authentication, choose:

• Headers — Provide static request headers as a JSON object, such as an Authorization header.
• Client Credentials — Provide an OAuth 2.0 client ID, client secret, and optional scopes.
• Azure AD — Provide a Microsoft Entra ID tenant ID, client ID, client secret, and scope from an app registration (Team plan and above).

STDIO

Use STDIO when Confident AI should start a local MCP server process. Enter the executable command and any space-separated arguments required to launch the server.

Available tools

After the server connects, its row shows Connected and the number of tools it exposes (for example, 30 tools enabled). Click the tool count to expand the list and verify that the expected names and definitions were synced.

If you change a server's tools, click the server's refresh button to reconnect, or the pencil icon to update it, before running your next evaluation so the available definitions remain current.

Use an MCP server in evaluations

When starting an evaluation from a dataset, attach the connected MCP server to the test run. Confident AI uses its tool definitions to:

• Recognize matching calls as MCP tool calls.
• Evaluate tool selection and argument quality with metrics such as MCP Use and Argument Correctness.
• Show the tools available to your agent alongside its observed tool interactions.

When an AI connection generates the outputs, attach the servers to the connection instead, on its Resources tab: every run through that connection then uses them. Only HTTP servers can be attached to an AI connection.

For the complete evaluation workflow and recommended metrics, see Evaluate MCP Servers.

Last updated on October 6, 2026