LLM Evaluation Quickstart

5 min quickstart guide for a code-driven LLM evaluation workflow

Overview

Confident AI offers a variety of features for you to test AI apps using code for a pre-deployment workflow, offering a wide range of features for:

Single-turn evaluation: Input-output as distinct AI interactions.
- End-to-end: Treats your AI app as a black box.
- Component-level: Built for agentic use cases—debug each agent step and component (planner, tools, memory, retriever, prompts) with granular assertions.
Multi-turn evaluation: Validate full conversations for consistency, state/memory retention, etc.

You can either run evals via code locally or remotely on Confident AI, both of which gives you the same functionality:

Local Evals

Run evaluations locally using deepeval with full control over metrics
Support for custom metrics, DAG, and advanced evaluation algorithms

Suitable for: Python users, development, and pre-deployment workflows

Remote Evals

Run evaluations on Confident AI platform with pre-built metrics
Integrated with monitoring, datasets, and team collaboration features

Suitable for: Non-python users, online + offline evals for tracing in prod

Vibe Code Your Evals

Let your coding agent build the eval suite for you — datasets, metrics, pytest files, and shareable Confident AI reports. Better yet, use DeepEval as your build-loop ground truth: your agent runs the evals, reads the failures and reason strings, makes the smallest app change, and re-runs to confirm. Choose the install method for your agent below.

Run these four commands in Claude Code:

/plugin marketplace add confident-ai/deepeval
/plugin install deepeval@deepeval-plugins
/reload-plugins
/plugins

The /plugins command should list DeepEval Plugin under your installed plugins.

Install the deepeval Agent Skill with any Skills-compatible installer. This works with Cursor, Claude Code, Codex, Windsurf, OpenCode, and any other assistant that supports the Skills standard:

npx skills add confident-ai/deepeval --skill deepeval

The skill teaches your agent how to pick the right test shape (single-turn / multi-turn / component-level), reuse or generate goldens, write a committed tests/evals/ suite, run deepeval test run, read failures, and iterate. It triggers automatically on prompts like the ones below.

Once installed, open the project you want to evaluate and tell your agent what you need. Example prompts:

"Create a DeepEval pytest eval suite for this app, generate ~30 goldens, and push results to Confident AI."
"My app is a RAG pipeline — set up DeepEval evals with retrieval-focused metrics."
"Generate a dataset from the docs in ./knowledge and run them through DeepEval."

Your agent will run the intake questions, pick metrics, generate goldens with deepeval generate, and produce a committed pytest suite you can rerun in CI.

Run Your First Eval

This examples goes through a single-turn, end-to-end evaluation example in code.

export CONFIDENT_API_KEY="confident_us..."

Create a dataset
It is mandatory to create a dataset for a proper evaluation workflow.
If a dataset is not possible for your team at this point, setup LLM tracing to run ad-hoc evaluations without a dataset instead. Confident AI will generate datasets for you automatically this way.
main.py
from deepeval.dataset import EvaluationDataset, Golden # goldens are what makes up your dataset goldens = [Golden(input="What's the weather like in SF?")] # create dataset dataset = EvaluationDataset(goldens=goldens) # save to Confident AI dataset.push(alias="YOUR-DATASET-ALIAS")
Done ✅. You should now see your dataset on the platform.
You can create one in the UI under Project > Datasets, and upload goldens to your dataset via CSV:
Create Dataset on Confident AI

Create a metric

Create a metric locally in deepeval. Here, we're using the AnswerRelevancyMetric() for demo purposes.

main.py

from deepeval.metrics import AnswerRelevancyMetric

relevancy = AnswerRelevancyMetric() # Using this for the sake of simplicity

Configure evaluation model
Since all metrics in deepeval uses LLM-as-a-Judge, you will also need to configure your LLM judge provider. To use OpenAI for evals:
```
export OPENAI_API_KEY="sk-..."
```
You can also use any model provider since deepeval integrates with all of them.

Create a test run

A test run is a benchmark/snapshot of your AI app's performance at any point in time. You'll need to:

Convert all goldens in your dataset into test cases, then
Use the metric you've created to evaluate each test case

main.py

from deepeval.dataset import EvaluationDataset
from deepeval.test_case import LLMTestCase
from deepeval.metrics import AnswerRelevancyMetric
from deepeval import evaluate

# Pull from Confident AI
dataset = EvaluationDataset()
dataset.pull(alias="YOUR-DATASET-ALIAS")

# Create test cases
for golden in dataset.goldens:
    test_case = LLMTestCase(
        input=golden.input,
        actual_output=llm_app(golden.input) # Replace with your AI app
    )
    dataset.add_test_case(test_case)

# Run an evaluation
evaluate(test_cases=dataset.test_cases, metrics=[AnswerRelevancyMetric()])

Lastly, run main.py to run your first single-turn, end-to-end evaluation:

python main.py

✅ Done. You just created a first test run with a sharable testing report auto-generated on Confident AI.

Login with API key
The TypeScript examples use the Confident AI SDK, which reads your project API key from CONFIDENT_PROJ_API_KEY:
```
npm install confident-ai
export CONFIDENT_PROJ_API_KEY="confident_us..."
```
Create a dataset
It is mandatory to create a dataset for a proper evaluation workflow.
If a dataset is not possible for your team at this point, setup LLM tracing to run ad-hoc evaluations without a dataset instead. Confident AI will generate datasets for you automatically this way.
index.ts
import { ConfidentAI } from "confident-ai"; async function createDataset() { const client = new ConfidentAI(); // goldens are what makes up your dataset const goldens = [{ input: "What's the weather like in SF?" }]; // save to Confident AI, which creates the dataset const dataset = await client.datasets.push("YOUR-DATASET-ALIAS", goldens); console.log(dataset.id); } createDataset().catch(console.error);
Done ✅. You should now see your dataset on the platform. Keep the id it prints: you'll pull the dataset with it next.
You can create one in the UI under Project > Datasets, and upload goldens to your dataset via CSV:
Create Dataset on Confident AI
Create a metric collection
Create a metric collection under Project > Metrics > Collections with the metrics you wish to use for evals.

Create a test run

A test run is a benchmark/snapshot of your AI app's performance at any point in time. You'll need to:

Convert all goldens in your dataset into test cases, then
Use the metric collection you've created to evaluate each test case

index.ts

import { ConfidentAI } from "confident-ai";
import { SingleTurnGolden } from "confident-ai/datasets";
import { SingleTurnTestCaseInput } from "confident-ai/evaluate";

async function runEvaluation() {
  const client = new ConfidentAI();

  // Pull from Confident AI
  const dataset = await client.dataset("<DATASET-ID>").pull();

  // Create test cases
  const testCases: SingleTurnTestCaseInput[] = [];
  for (const golden of dataset.goldens as SingleTurnGolden[]) {
    testCases.push({
      input: golden.input,
      actualOutput: await llmApp(golden.input), // Replace with your AI app
    });
  }

  // Run an evaluation on Confident AI
  await client.evaluate.runEvals("YOUR-METRIC-COLLECTION-NAME", testCases);
}

runEvaluation().catch(console.error);

Lastly, run index.ts to run your first single-turn, end-to-end evaluation:

tsx index.ts

✅ Done. You just created a first test run with a sharable testing report auto-generated on Confident AI.

Testing Report on Confident AI

There are two main pages in a testing report:

Overview - Shows metadata of your test run such as the dataset that was used for testing, average, median, and distribution of each of the metric(s)
Test Cases - Shows all the test cases in your test run, including AI generated summaries of your test bench, and metric data for in-depth debugging and analysis.

When you have two or more test runs, you can also start running A|B regression tests.

Next Steps

Now that you've run your first evaluation, dive deeper into single-turn testing:

End-to-End Evals

Treat your AI app as a black box. Learn how to use LLM tracing for better debugging, run remote evals, and log hyperparameters for A|B testing.

Component-Level Evals

Test individual components like retrievers, generators, and tools. Built for agentic use cases where you need granular assertions.

Scaling beyond prototype?Move from local tests to a repeatable evaluation process for your teamTalk to us

Last updated on October 19, 2026

Overview

Vibe Code Your Evals

Run Your First Eval

Login with API key

Create a dataset

Create a metric

Configure evaluation model

Create a test run

Login with API key

Create a dataset

Create a metric collection

Create a test run

Next Steps