Online & Offline Evaluations

Overview

You can run evaluations during LLM tracing, by running metrics on individual:

1. Spans,
2. Traces, and
3. Threads

For online evaluations, you can do this by providing the name of the metric collection you’ve created on Confident AI in the @obesrve decorators during tracing. For offline evaluations, you can call the evaluate_thread function to manually trigger an evaluation.

Prerequisites

Create Metric Collections

To enable referenceless metrics to run, you need to create a metric collection on Confident AI. Your metric collection will determine the set of metrics for you to run evaluations on.

If you’re planning to run offline evals on threads, create a multi-turn metric collection, and vice versa. This is really important as you’ll needs the metric collection names to match as you’ll see in the next section.

Online Evaluations

Online evaluations refer to evals that are ran in real-time, as traces are ingested and sent to Confident AI.

Spans

Simply provide the name of the metric collection to tell Confident AI the specify set of referenceless metrics you wish to run in the @observe decorator.

You’ll also need to use update_current_span with an LLMTestCase at runtime to actually trigger an online evaluation on the server-side:

1
from deepeval.test_case import LLMTestCase
2
from deepeval.tracing import observe, update_current_span
3
from openai import OpenAI
4
5
client = OpenAI()
6
7
@observe(metric_collection="My Collection")
8
def llm_app(query: str) -> str:
9
    res = client.chat.completions.create(
10
        model="gpt-4o",
11
        messages=[
12
            {"role": "user", "content": query}
13
        ]
14
    ).choices[0].message.content
15
16
    update_current_span(test_case=LLMTestCase(input=query, actual_output=res))
17
    return res
18
19
llm_app("Write me a poem.")

The metricCollection argument is an optional strings that determines which metrics in your online metric collection will be ran for this current span.

If you specify a metricCollection list but don’t update your current span with the sufficient test case parameters for metric execution, it will simply show up as an error on Confident AI, and won’t block or cause issues in your code.

Traces

Running evals on traces are akin to running end-to-end evals, where you disregard the performance of individual spans within the trace and treat your application as a black-box.

Similar to evals for spans, you would also provide a metricCollection name, but this time call the update_current_trace() function instead:

1
from deepeval.test_case import LLMTestCase
2
from deepeval.tracing import observe, update_current_trace
3
from openai import OpenAI
4
5
client = OpenAI()
6
7
@observe(metric_collection="My Collection")
8
def llm_app(query: str) -> str:
9
    res = client.chat.completions.create(
10
        model="gpt-4o",
11
        messages=[
12
            {"role": "user", "content": query}
13
        ]
14
    ).choices[0].message.content
15
16
    update_current_trace(test_case=LLMTestCase(input=query, actual_output=res))
17
    return res
18
19
llm_app("Write me a poem.")

Also note that unlike evals on spans, the metricCollection MUST BE DEFINED at the top-level/root span level. You can call update_current_trace anywhere in your observed application though.

Offline Evaluations

Offline evaluations refer to evals that are ran retrospectively, after traces are ingested into Confident AI.

Threads

All threads are offline evaluations because you should only run evals on threads once a multi-turn interaction has completed.

Since it is impossible for Confident AI to automatically know whether a multi-turn conversation has completed or not, you’ll have to trigger an offline evaluation using the evaluate_thread() method ONLY AFTER once you’re certain a conversation has completed:

1
from openai import OpenAI
2
from deepeval.tracing import observe, update_current_trace, evaluate_thread
3
4
client = OpenAI()
5
your_thread_id = "your-thread-id"
6
7
@observe()
8
def llm_app(query: str):
9
    res = client.chat.completions.create(
10
        model="gpt-4o",
11
        messages=[
12
            {"role": "user", "content": query}
13
        ]
14
    ).choices[0].message.content
15
    update_current_trace(thread_id=your_thread_id, input=query, output=res)
16
    return res
17
18
if __name__ == "__main__":
19
    user_query = input("💬 Enter your prompt: ")
20
    response = llm_app(user_query, true)
21
    print(f"\n🤖 Response:\n{response}")
22
    evaluate_thread(thread_id=your_thread_id, metric_collection="Collection Name")

You can also use a_evaluate_thread, the async version of evaluate_thread():

1
from deepeval.tracing import evaluate_thread
2
3
a_evaluate_thread(thread_id=your_thread_id, metric_collection="Collection Name")

You MUST set the input/output of individual traces in a thread for multi-turn evaluations to work. To recap, DeepEval uses the input of a trace as the "user" role content and the output of a trace as the "assistant" role content as turns in your thread.

You can also set the retrieval context and tools called (if any) for each assistant/AI turn as part of the output setting, which would allow you to see the context of which an LLM generation was invoked in:

1
from deepeval.test_case import ToolCall
2
from deepeval.tracing import TurnContext
3
4
update_current_trace(
5
    thread_id=your_thread_id,
6
    output=res,
7
    turnContext=TurnContext(
8
        retrieval_context=["..."], # optional list of strings
9
        tools_called=[ToolCall(...)] # optional list of tools called
10
    )
11
)

To learn more, visit the threads section.

Trace

Although less common since you can run online evals on traces, you can also run single-turn, offline evals on trace retrospectively:

1
from deepeval.tracing import evaluate_trace
2
3
evaluate_trace(trace_uuid=<your-trace-uuid>, metric_collection="Collection Name")

Note that your trace must already contain the necessary test case parameters as you cannot update the test case parameters using the evaluate_trace function.

Span

The same offline evals can also be ran in the same say for spans:

1
from deepeval.tracing import evaluate_span
2
3
evaluate_span(span_uuid=<your-span-uuid>, metric_collection="Collection Name")

Remember, the metric collection you provide must be a single-turn one.

Examples

Trace and span evals

Quick quiz: Given the code below, will Confidnet AI run online evaluations on the trace using metrics in "Collection 2" or "Collection 1"?

1
from deepeval.tracing import observe, update_current_span, update_current_trace
2
3
@observe(metric_collection="Collection 1")
4
def outer_function():
5
    @observe(metric_collection="Collection 2")
6
    def inner_function():
7
        update_current_span(test_case=...)
8
        update_current_trace(test_case=...)

Answer: This will run "Collection 1" for traces, and "Collection 2" for spans.

This is because in this example:

1. When outer_function starts, it creates the “outer” span
2. When inner_function is called, it creates the “inner” span on top
3. Any calls to update_current_span() during inner_function’s execution will update the “inner” span, not the “outer” one
4. Any calls to update_current_trace() during any point inside outer_function will update the entire trace and online evals for traces MUST BE SET on the root level span.

Thread and trace evals

Quick quiz: Given the code below, will Confident AI run online evaluations on the thread using metrics in "Collection 2" or "Collection 1"?

1
from deepeval.tracing import observe, update_current_trace, evaluate_thread
2
3
your_thread_id = "your-thread-id"
4
5
@observe(metric_collection="Collection 1")
6
def outer_function():
7
    update_current_trace(thread_id=your_thread_id, test_case=...)
8
9
evaluate_thread(thread_id=your_thread_id, metric_collection="Collection 2")

Answer: This will NOT run "Collection 1" or "Collection 2", because neither the input or output has been specified in update_current_trace. This means Confident AI will have no turns to evaluate using metrics in your metric collection.