Eval Setup Skill

Overview

The confident-eval-setup Agent Skill teaches your coding agent how to get your LLM app ready for evaluation on Confident AI through the MCP server. Its main job is the fiddly part: registering your app as an AI connection, so Confident AI can call it directly. The agent reads how your endpoint is called from your code, creates the connection, and pings it until it works, then uploads a dataset and sends you to the dataset's page to run the evaluation.

Nothing is added to your codebase. The AI connection and the dataset live in your Confident AI project, so anyone on your team can evaluate the app from the dashboard from then on, without an agent or a local setup.

When It Triggers

The skill activates on prompts like:

Set up evals for our support agent on Confident AI.
Connect our /chat endpoint to Confident AI as an AI connection.
Our AI connection is inactive, figure out why and fix it.
Upload goldens.csv as a dataset called "support-faq".

Installation

npx skills add confident-ai/confident-skills --skill "confident-eval-setup"

/plugin marketplace add confident-ai/confident-skills
/plugin install confident-eval-setup@confident-skills
/reload-plugins

git clone https://github.com/confident-ai/confident-skills
cp -r confident-skills/plugins/confident-eval-setup/skills/confident-eval-setup .claude/skills/

Prerequisites

• The Confident AI MCP server connected to your coding agent. It signs in with OAuth, so no API key is needed.
• An app endpoint Confident AI can reach over https:// (or wss:// for WebSocket apps). For an app behind a firewall, run the Confident Agent.

What Happens in Your Project

1. Ask for evals

   The agent calls list_projects to find the project to work in, then checks what's already there with list_ai_connections and list_datasets, so it reuses existing resources instead of creating duplicates.

   Set up evals for our support agent on Confident AI.

2. Connect your app

   The agent reads your app's request and response shapes from the code, then registers the endpoint with create_ai_connection: the URL, a payload template with "golden.input" where each golden's input goes, any headers, and the path to the answer in the response.

   See what the agent sends

   {
     "name": "Support agent (staging)",
     "endpoint": "https://staging.example.com/chat",
     "payload": { "message": "golden.input" },
     "headers": [
       { "key": "Authorization", "value": "Bearer ..." },
       { "key": "Content-Type", "value": "application/json" }
     ],
     "actualOutputKeyPath": ["reply"]
   }

   For streaming apps it sets the server-sent event that carries tokens, and for RAG apps it also maps the retrieved documents, so retrieval metrics have context to score.

3. Verify it with a ping

   The agent calls ping_ai_connection, which sends your endpoint one sample test case and reports whether Confident AI could reach it and read the answer out of the response. If the connection comes back inactive, the agent reads the error, status code, and the real response body, corrects the configuration with update_ai_connection, and pings again until it passes.

   Our AI connection is inactive, figure out why and fix it.

4. Upload the dataset

   The agent turns your test cases (a CSV, a JSON file, a folder of examples, or ones it drafts with you) into goldens and uploads them with push_dataset, which creates the dataset the first time. Single-turn goldens carry an input and, optionally, an expectedOutput; multi-turn goldens carry a scenario and an expectedOutcome.

   See what the agent sends

   {
     "alias": "support-faq",
     "finalized": true,
     "goldens": [
       {
         "input": "How do I reset my password?",
         "expectedOutput": "Go to Settings → Security and choose Reset password."
       },
       {
         "input": "Can I change my billing date?",
         "expectedOutput": "Yes, from Billing → Plan, once per billing cycle."
       }
     ]
   }

   Only finalized goldens are evaluated, so the agent marks them finalized once you've confirmed they're right.

5. Run the evaluation from the dataset page

   The agent finishes by giving you the dataset's URL, which push_dataset returns, such as https://app.confident-ai.com/project/<projectId>/datasets/<datasetId>. Open it and press Evaluate, then choose the AI connection the agent created and the metrics to score with. Confident AI calls your app once per finalized golden and scores every output.

FAQs

Next Steps

Last updated on October 20, 2026