LLM Tracing Quickstart

Overview

Confident AI allows anyone building with any framework, language, and LLMs to setup LLM observability through tracing. Every interaction traced is evals-native, powered by DeepEval. This means apart form tracking latency, cost, and error rates, you can also run evals on:

• Traces
• Spans, and
• Threads

The demo below demonstrates what it looks like on the platform:

How It Works

You would setup LLM tracing either through one of our integrations, or by decorating your LLM app for Python and Typescript users. Either way, Confident AI will create an execution hierarchy of your LLM app, as well as log all components that were called for each LLM invocation:

• Trace: The overall process of tracking and visualizing the execution flow of your LLM application
• Span: Individual units of work within your application (e.g., LLM calls, tool executions, retrievals)

Each observed function CREATES A SPAN, and MANY SPANS MAKE UP A TRACE. When you have tracing setup, you can run evaluations on both the trace and span level.

Trace Your First LLM Call

$
pip install -U deepeval

$
export CONFIDENT_API_KEY=YOUR-API-KEY

1
from openai import OpenAI
2
from deepeval.tracing import observe
3
4
client = OpenAI()
5
6
@observe()
7
def llm_app(query: str) -> str:
8
    return client.chat.completions.create(
9
        model="gpt-4o",
10
        messages=[
11
            {"role": "user", "content": query}
12
        ]
13
    ).choices[0].message.content
14
    return
15
16
# Call app to send trace to Confident AI
17
llm_app("Write me a poem.")

Congratulations! 🎉 Now whenever you run your LLM app, all traces will be logged AND evaluated on Confident AI. Go to the the Observatory section on Confident AI to check it out.

$
CONFIDENT_TRACE_FLUSH=YES

In the next section, we will learn how to run evals for LLM tracing.