Single-Turn, Component-Level Evals

Overview

Component-level testing lets you evaluate individual parts of your LLM application — retrievers, generators, tools, planners — rather than just the final output. This is essential for debugging complex pipelines where you need to pinpoint exactly which component is failing.

Requirements:

• A dataset of goldens — determines how many times your app runs
• LLM tracing setup with @observe decorators
• Metrics defined per component via the metrics parameter in @observe()

How It Works

1. Setup LLM tracing with @observe decorators and define metrics for each component
2. Pull your dataset from Confident AI
3. Loop through goldens using the evals_iterator() and invoke your LLM app

sequenceDiagram
    participant Your Code
    participant Confident AI
    participant LLM App
    participant Components

    Your Code->>Confident AI: Pull dataset
    Confident AI-->>Your Code: List of goldens (count = N)

    loop N times via evals_iterator()
        Your Code->>LLM App: Invoke app
        LLM App->>Components: @observe captures spans
        Components->>Components: update_current_span() sets test case fields
        Components-->>LLM App: Component metric(s) executes
    end

    LLM App-->>Confident AI: Upload traces & run component metrics
    Note over Confident AI: Generate testing report

Run Component-Level Tests Locally

This section is nearly identical to this part of the previous section, where we use LLM tracing to run end-to-end evals. This is because LLM tracing is just so convenient to evaluate everything and anything.

We're also using the same mock LLM app in the previous section to demonstrate LLM tracing:

from openai import OpenAI

def llm_app(query: str) -> str:
    # Retriever for your vector db
    def retriever(query: str) -> list[str]:
        return ["List", "of", "text", "chunks"]
    # Generator that combines retrieved context with user query
    def generator(query: str, text_chunks: list[str]) -> str:
        return OpenAI().chat.completions.create(
            model="gpt-4o",
            messages=[
                {"role": "user", "content": query}
            ]
        ).choices[0].message.content
    # Calls retriever then generator
    return generator(query, retriever(query))

1. Setup LLM tracing, and define metrics

   Decorate your application with the @observe decorator, and provide metrics for components that you wish to evaluate:

   from openai import OpenAI
   from deepeval.metrics import AnswerRelevancyMetric, ContextualRelevancyMetric
   from deepeval.tracing import observe, update_current_span
   
   @observe()
   def llm_app(query: str) -> str:
   
       @observe(metrics=[ContextualRelevancyMetric()], embedder="your-embedding-model-name")
       def retriever(query: str) -> list[str]:
           chunks = ["List", "of", "text", "chunks"]
           update_current_span(input=query, retrieval_context=chunks)
           return chunks
   
       @observe(metrics=[AnswerRelevancyMetric()])
       def generator(query: str, text_chunks: list[str]) -> str:
           res = OpenAI().chat.completions.create(model="gpt-4o", messages=[{"role": "user", "content": query}]
           ).choices[0].message.content
           update_current_span(input=query, output=res)
           return res
   
       return generator(query, retriever(query))

   The example above shows how we are tracing our LLM app by simply adding a few @observe decorators:

   • Each @observe decorator creates a span, which represents components
   • A trace on the other hand is created by the top-level @observe decorator, and is made up of many spans/components
   • You include a list of metrics in @observe() for components you wish to evaluate, and call the update_current_span function inside said components to create test cases for evaluation

2. Pull dataset, and loop through goldens

   Pull your dataset and use the .evals_iterator() to iterate. The iterator controls how many times your LLM app runs — once per golden in your dataset.

   from deepeval.dataset import EvaluationDataset
   
   dataset = EvaluationDataset()
   dataset.pull(alias="YOUR-DATASET-ALIAS")
   
   for _ in dataset.evals_iterator():
       llm_app("any input") # golden.input is optional for component-level testing

   Since test case fields are populated via update_current_span() inside your components, you can pass any input to your LLM app — or use golden.input if your test scenarios require specific inputs.

   Done ✅. You should see a link to your newly created sharable testing report.

When you call your LLM app inside a dataset's evals_iterator(), deepeval automatically captures invocations of your LLM app and creates test cases dynamically based on the @observeed component's hierarchy. Here are some more info about component-level evals:

• For components that are @observeed but with no metrics attached, Confident AI will simply not test those components and display them as regular spans instead
• You would generally not use reference-based metrics for component-level testing. This is because goldens are designed to map 1-to-1 to test cases, which makes arguments such as expected_output redundant