LLM Evaluation Quickstart

Overview

Confident AI offers a variety of features for you to test AI apps using code for a pre-deployment workflow, offering a wide range of features for:

• Single-turn evaluation: Input-output as distinct AI interactions.
  • End-to-end: Treats your AI app as a black box.
  • Component-level: Built for agentic use cases—debug each agent step and component (planner, tools, memory, retriever, prompts) with granular assertions.
• Multi-turn evaluation: Validate full conversations for consistency, state/memory retention, etc.

You can either run evals via code locally or remotely on Confident AI, both of which gives you the same functionality:

Vibe Code Your Evals

Let your coding agent build the eval suite for you — datasets, metrics, pytest files, and shareable Confident AI reports. Better yet, use DeepEval as your build-loop ground truth: your agent runs the evals, reads the failures and reason strings, makes the smallest app change, and re-runs to confirm. Choose the install method for your agent below.

$
/plugin marketplace add confident-ai/deepeval
$
/plugin install deepeval@confident-ai-plugins
$
/reload-plugins
$
/plugins

Once installed, open the project you want to evaluate and tell your agent what you need. Example prompts:

• “Create a DeepEval pytest eval suite for this app, generate ~30 goldens, and push results to Confident AI.”
• “My app is a RAG pipeline — set up DeepEval evals with retrieval-focused metrics.”
• “Generate a dataset from the docs in ./knowledge and run them through DeepEval.”

Your agent will run the intake questions, pick metrics, generate goldens with deepeval generate, and produce a committed pytest suite you can rerun in CI.

Run Your First Eval

This examples goes through a single-turn, end-to-end evaluation example in code.

$
export CONFIDENT_API_KEY="confident_us..."

1
from deepeval.dataset import EvaluationDataset, Golden
2
# goldens are what makes up your dataset
3
goldens = [Golden(input="What's the weather like in SF?")]
4
# create dataset
5
dataset = EvaluationDataset(goldens=goldens)
6
# save to Confident AI
7
dataset.push(alias="YOUR-DATASET-ALIAS")

1
from deepeval.metrics import AnswerRelevancyMetric
2
3
relevancy = AnswerRelevancyMetric() # Using this for the sake of simplicity

$
export OPENAI_API_KEY="sk-..."

1
from deepeval.dataset import EvaluationDataset
2
from deepeval.test_case import LLMTestCase
3
from deepeval.metrics import AnswerRelevancyMetric
4
from deepeval import evaluate
5
6
# Pull from Confident AI
7
dataset = EvaluationDataset()
8
dataset.pull(alias="YOUR-DATASET-ALIAS")
9
10
# Create test cases
11
for golden in dataset.goldens:
12
    test_case = LLMTestCase(
13
        input=golden.input,
14
        actual_output=llm_app(golden.input) # Replace with your AI app
15
    )
16
    dataset.add_test_case(test_case)
17
18
# Run an evaluation
19
evaluate(test_cases=dataset.test_cases, metrics=[AnswerRelevancyMetric()])

$
python main.py

There are two main pages in a testing report:

• Overview - Shows metadata of your test run such as the dataset that was used for testing, average, median, and distribution of each of the metric(s)
• Test Cases - Shows all the test cases in your test run, including AI generated summaries of your test bench, and metric data for in-depth debugging and analysis.

When you have two or more test runs, you can also start running A|B regression tests.

Next Steps

Now that you’ve run your first evaluation, dive deeper into single-turn testing: