Launch Week 02 wrapped — explore all five launches
Confident AI
Docs
Pricing
Products
LLM Evaluation
LLM Evaluation

Benchmark LLM systems with research-backed metrics.

LLM Observability
LLM Observability

Trace, monitor, and alert on production LLM systems.

AI Red Teaming
AI Red Teaming

Stress-test LLM apps against adversarial attacks.

AI Governance
AI Governance

Enforce AI standards and controls across teams.

Book a DemoSign Up
Book a DemoSign Up

DeepEval Evals Skill

Teach your agent to build pytest eval suites, generate datasets, and iterate on failures with DeepEval.

Overview

The deepeval Agent Skill teaches your coding agent how to add a full evaluation loop to an AI application: classify the app (agent, RAG pipeline, or multi-turn chatbot), generate or reuse a dataset, write a committed pytest eval suite, run it with deepeval test run, and iterate on the failures. It is the main skill of the three that ship in the confident-ai/deepeval repository.

Without the skill, agents tend to hand-write throwaway eval scripts, invent goldens, and call raw pytest. With it, the agent follows the same workflow our docs prescribe — deepeval generate for synthetic data, metrics kept in a separate metrics.py module, traced single-turn evals where possible, and a suite you can rerun without an agent in the room.

When It Triggers

The skill activates on prompts like:

Prompts that trigger the skill
Add evals to my customer support agent.
Generate a dataset of goldens from our docs folder.
Why is my RAG pipeline hallucinating? Set up metrics to catch it.
Run the eval suite and fix the failures.

Installation

Works with Cursor, Claude Code, Codex, Windsurf, OpenCode, and any other Skills-compatible assistant:

npx skills add confident-ai/deepeval --skill "deepeval"

Prerequisites

  • Python 3.9+ with pip install deepeval
  • Model credentials for metrics (e.g. OPENAI_API_KEY)
  • CONFIDENT_API_KEY for hosted reports and traces

What Changes in Your Codebase

  1. Ask for evals

    Describe what you want evaluated. The agent inspects your codebase, picks a use case (multi-turn chatbot, agent, or RAG), and asks a short set of intake questions — evaluation model, dataset source, tracing, and how many improvement rounds to run.

    Prompt
    Add evals to my customer support agent and iterate until they pass.

  2. Let it generate a dataset

    If you don't already have a dataset — local or pulled from Confident AI — the agent generates roughly 30–50 goldens from your docs or knowledge base instead of hand-writing them. You can also ask for one directly:

    Prompt
    Generate a dataset of goldens from the ./docs folder.
    See what the agent runs
    deepeval generate --method docs --variation single-turn \
  --documents ./docs --output-dir ./tests/evals --file-name .dataset

  3. Review the committed eval suite

    The agent starts from the skill's templates and commits a pytest suite you can rerun without an agent in the room, with metric instances kept in a shared metrics.py module.

    See what the agent commits
    tests/evals/test_ai_app.py
    import pytest
from deepeval import assert_test
from deepeval.dataset import EvaluationDataset, Golden

from metrics import SINGLE_TURN_TRACE_METRICS
import ai_app

dataset = EvaluationDataset()
dataset.add_goldens_from_json_file(file_path="tests/evals/.dataset.json")


@pytest.mark.parametrize("golden", dataset.goldens)
def test_single_turn_tracing(golden: Golden):
    ai_app.run_traced_ai_app(golden.input)
    assert_test(golden=golden, metrics=SINGLE_TURN_TRACE_METRICS)

  4. Run and iterate

    Evals run through deepeval test run (not raw pytest). The agent inspects failures — and traces, when tracing is on — makes targeted changes to prompts, tools, or retrieval, and reruns for the agreed number of rounds (five by default):

    Prompt
    Run the eval suite and fix the failures until every metric passes.
    See what the agent runs
    deepeval test run tests/evals/test_ai_app.py \
  --num-processes 5 --identifier "iterating-round-1"

    When Confident AI is enabled, each run lands as a test run in your project, and deepeval view opens the latest hosted report.

FAQs

Do I need a Confident AI account to use this skill?

No — evals run locally with just pip install deepeval and model credentials. A CONFIDENT_API_KEY (or deepeval login) adds hosted reports, traces, production monitoring, and online evals on top.

Can it use a dataset I already have?

Yes. Existing datasets — local files or datasets pulled from Confident AI — are reused as-is, and existing metrics and thresholds are kept unless you change them. The skill only reaches for deepeval generate when no dataset exists, and it never hand-writes goldens.

Why does it run 'deepeval test run' instead of pytest?

The suite is standard pytest under the hood, but the deepeval test run command adds what evals need: parallel execution with --num-processes, run identifiers for tracking iterations, and automatic reporting to Confident AI when enabled.

Which model grades the metrics?

Your choice — the evaluation model is one of the intake questions the skill asks before writing anything, and it uses your own model credentials (e.g. OPENAI_API_KEY) rather than assuming a default.

Next Steps

LLM Evaluation Quickstart

See the underlying evaluation workflow the skill automates.

DeepEval Tracing Skill

Instrument your app so the eval suite can run traced evals.

Scaling beyond prototype?For teams evaluating Confident AI in productionTalk to us

Last updated on

Built byConfident AI