Introducing Confident Monitor
Confident AI
Docs
Pricing
Products
LLM Evaluation
LLM Evaluation

Benchmark LLM systems with research-backed metrics.

LLM Observability
LLM Observability

Trace, monitor, and alert on production LLM systems.

AI Red Teaming
AI Red Teaming

Stress-test LLM apps against adversarial attacks.

AI Governance
AI Governance

Enforce AI standards and controls across teams.

Contact salesSign Up
Contact salesSign Up

Single-Turn, E2E Evals

Learn how to run end-to-end testing for single-turn use cases

Overview

End-to-end testing treats your LLM application as a black box — you provide an input and evaluate the final output. This is ideal for simple pipelines, RAG systems, or when you only care about the end result rather than intermediate steps.

Requirements:

  • A dataset of goldens containing golden inputs
  • A list of metrics to evaluate with
  • Construction of an LLMTestCase at runtime (mapping input → actual_output)

How It Works

  1. Pull your dataset from Confident AI
  2. Loop through goldens in your dataset, for each golden:
    • Invoke your LLM app using golden inputs to generate test case parameters such as actual output, tools called, and
    • Map golden fields to test case parameters
    • Add test case back to your dataset
  3. Run evaluation on test cases, which can happen locally or on Confident AI

Here's a visual representation of where the data-flows throughout the process:

sequenceDiagram
    participant Your Code
    participant Confident AI
    participant LLM App
    participant Metrics

    Your Code->>Confident AI: Pull dataset
    Confident AI-->>Your Code: List of goldens

    loop For each golden
        Your Code->>LLM App: Invoke with golden.input
        LLM App-->>Your Code: actual_output
        Your Code->>Your Code: Create test case (input + actual_output)
    end

    Your Code->>Metrics: Run evaluation locally
    Metrics-->>Confident AI: Upload results & generate testing report

Run E2E Tests Locally

Running evals locally is only possible if you are using the Python deepeval library. If you're working with Typescript or any other language, skip to the remote end-to-end evals section instead.

For this section, we'll be using this mock LLM app, that is a simple RAG pipeline:

See Mock LLM App
main.py
from openai import OpenAI

def llm_app(query: str) -> str:
    # Retriever for your vector db
    def retriever(query: str) -> list[str]:
        return ["List", "of", "text", "chunks"]
    # Generator that combines retrieved context with user query
    def generator(query: str, text_chunks: list[str]) -> str:
        return OpenAI().chat.completions.create(
            model="gpt-4o",
            messages=[
                {"role": "user", "content": query}
            ]
        ).choices[0].message.content
    # Calls retriever then generator
    return generator(query, retriever(query))

  1. Pull dataset

    Pull your dataset (and create one if you haven't already):

    main.py
    from deepeval.dataset import EvaluationDataset

dataset = EvaluationDataset()
dataset.pull(alias="YOUR-DATASET-ALIAS")

  2. Loop through goldens to create test cases

    A native for-loop calling your LLM app would do for this step:

    main.py
    from deepeval.test_case import LLMTestCase
from deepeval.dataset import EvaluationDataset

dataset = EvaluationDataset()
dataset.pull(alias="YOUR-DATASET-ALIAS")

for golden in dataset.goldens:
    test_case = LLMTestCase(
        input=golden.input,
        actual_output=llm_app(input)
    )
    dataset.add_test_case(test_case)

  3. Run evaluation using `evaluate()`

    The evaluate() function allows you to create test runs and uploads the data to Confident AI once evaluations have completed locally.

    main.py
    from deepeval.metrics import AnswerRelevancyMetric
from deepeval import evaluate

# Replace with your metrics
evaluate(test_cases=dataset.test_cases, metrics=[AnswerRelevancyMetric()])

    Done ✅. You should see a link to your newly created sharable testing report.

    • The evaluate() function runs your test suite across all test cases and metrics
    • Each metric is applied to every test case (e.g., 10 test cases × 2 metrics = 20 evaluations)
    • A test case passes only if all metrics for it pass
    • The test run’s pass rate is the proportion of test cases that pass
    Single-Turn Testing Reports

The evaluate() function is extremely unopinionated and non-instrusive, which means it is great for teams looking for a lightweight approach for running LLM evaluations. However, it also means that:

  • You have to handle a lot of the ETL yourself to map test case fields, even rewriting your LLM app at times to return the correct data
  • No visibility - you will still want to be able to debug your LLM app even if it is an end-to-end evaluation

In the next section, we'll show how you can avoid this ETL hellhole and bring LLM traces to end-to-end testing.

LLM Tracing for E2E Evals

LLM tracing solves all problems associated with constructing test cases.

  1. Setup LLM tracing

    All you need is to add a few lines of code to your existing LLM app (we'll be using the example from above):

    main.py
    from openai import OpenAI
from deepeval.tracing import observe, update_current_trace

@observe()
def llm_app(query: str) -> str:

    @observe()
    def retriever(query: str) -> list[str]:
        chunks = ["List", "of", "text", "chunks"]
        update_current_trace(retrieval_context=chunks)
        return chunks

    @observe()
    def generator(query: str, text_chunks: list[str]) -> str:
        res = OpenAI().chat.completions.create(model="gpt-4o", messages=[{"role": "user", "content": query}]
        ).choices[0].message.content
        update_current_trace(input=query, output=res)
        return res

    return generator(query, retriever(query))

    The example above shows how we are tracing our LLM app by simply adding a few @observe decorators:

    • Each @observe decorator creates a span, which represents components
    • A trace on the other hand is created by the top-level @observe decorator, and is made up of many spans/components
    • When you run end-to-end testing, you can call the update_current_trace function inside anywhere in your traced app to set test case parameters

    Don't worry too much about learning everything you can about LLM tracing for now. We'll go through it in in a dedicated LLM tracing section.

  2. Pull dataset, and loop through goldens

    Pull your dataset in the same way as before, and use the .evals_iterator() to loop through your goldens. You will use data in your goldens (most likely the input) to call your LLM app:

    main.py
    from deepeval.metrics import AnswerRelevancyMetric
from deepeval.dataset import EvaluationDataset

dataset = EvaluationDataset()
dataset.pull(alias="YOUR-DATASET-ALIAS")

for golden in dataset.evals_iterator(metrics=[AnswerRelevancyMetric()]):
    llm_app(golden.input) # Replace with your LLM app

    Done ✅. You should see a link to your newly created sharable testing report. This is literally all it takes to run end-to-end evaluations, with the added benefit of a full testing report with tracing included on Confident AI.

    Single-Turn Testing Reports (with Tracing)

Run E2E Tests Remotely

Remote end-to-end evals offer no tracibility for debugging but is great because:

  • Team members can build metrics without going through code
  • Supported through the Confident API, for any language

This is possible via the Confident AI SDK or the Confident API, which read your project key from CONFIDENT_PROJ_API_KEY.

  1. Create metric collection

    Go to Project > Metric > Collections:

    Metric Collection for Remote Evals

  2. Pull dataset and construct test cases

    Pull the dataset by its id, which you get back whenever you push to it. Then call your LLM app with each golden's input, and build a test case from the golden and your app's output.

    main.py
    from confident_ai import ConfidentAI

client = ConfidentAI()

result = client.datasets.pull(dataset_id="<DATASET-ID>")

  3. Run evals

    Send the test cases along with your metric collection's name. Set each test case's actual_output to your app's output. Every field of a test case is listed under TestCaseInput.

    main.py
    from confident_ai import ConfidentAI
from confident_ai.evaluate import SingleTurnTestCaseInput

client = ConfidentAI()

result = client.evaluate.run_evals(
    metric_collection="Collection Name",
    test_cases=[
        SingleTurnTestCaseInput(
            input="How tall is mount everest?"
        )
    ],
)

Advanced Usage

Now you've learnt how to run a single-turn, end-to-end evaluation, here are a few things you should also do. The Python examples run evals locally with deepeval, and the TypeScript and cURL examples run them remotely through the Confident AI SDK and the Confident API.

Log prompts and models

Tell Confident AI the configurations used in your LLM app during the evaluation.

Simply add a free-form key-value pair to the hyperparameters argument in the evaluate() function:

from deepeval.prompt import Prompt

prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
prompt.pull()

evaluate(
    hyperparameters={
        "Model": "YOUR-MODEL",
        "Prompt Version": prompt # An instance of your Prompt
    },
    test_cases=[...],
    metrics=[...]
)

Add identifier to test runs

The identifier argument allows you to name test runs, which will come in extremely handy when you're trying to run regression tests on them on the platform.

evaluate(
    identifier="Any custom string",
    test_cases=[...],
    metrics=[...]
)

Name test cases

Similar to the identifier, naming test cases allows you to search and match test cases across different test runs during regression testing.

evaluate(
    test_cases=[LLMTestCase(name="Any custom string", ...)],
    metrics=[...]
)

By default, Confident AI will match test cases based on matching inputs, so naming test cases is not strictly required for regression testing.

Scaling beyond prototype?For teams evaluating Confident AI in productionTalk to us

Last updated on

Built byConfident AI