LLM Tracing Quickstart

Overview

This guide shows you how to instrument your LLM app using the @observe decorator for Python or the observe wrapper for TypeScript.

How it works

Tracing works through instrumentation, which can either be manual or through one of Confident AI's integrations:

1. Decorate or wrap your functions with @observe (Python) or observe (TypeScript)
2. Each observed function becomes a span
3. The outermost observed function becomes the trace — all nested spans roll up into it (see troubleshooting if spans are creating separate traces instead of nesting)
4. Traces are sent to Confident AI asynchronously with zero latency impact
5. Once ingested, traces can be evaluated automatically using your configured metrics

You should also understand the terminology for tracing:

Vibe Code Your Tracing

Let your coding agent instrument your app for you — it picks the right path (DeepEval @observe, a framework integration, or OpenTelemetry) based on what your project uses and what you ask. Choose the install method for your agent below.

/plugin marketplace add confident-ai/deepeval
/plugin install deepeval@deepeval-plugins
/reload-plugins
/plugins

npx skills add confident-ai/deepeval --skill deepeval-tracing

Once installed, open the project you want to trace and tell your agent what you need. Example prompts:

• "Instrument this app with DeepEval tracing and send traces to Confident AI."
• "Add the OpenAI integration so my LLM calls show up on Confident AI."
• "I'm using LangGraph — wire up DeepEval tracing for it."

Your agent will read the codebase, choose between a native integration and manual @observe, and confirm traces land in the Observatory.

Instrument Your AI App

1. Install DeepEval

   Instrumentation must be done via code, so first install DeepEval, Confident AI's official open-source SDK:

   PythonTypeScript

   pip install -U deepeval

2. Set Your API Key

   Get your Confident AI Project API key and login:

   PythonTypeScript

   export CONFIDENT_API_KEY=YOUR-API-KEY

   from deepeval.tracing import trace_manager
   
   trace_manager.configure(
       confident_api_key="YOUR-API-KEY"
   )

3. Instrument Your App

   Decorate or wrap your functions to automatically capture inputs, outputs, and execution flow. Note that each observe decorator/wrapper creates a span on the UI.

   PythonTypeScript

   from openai import OpenAI
   from deepeval.tracing import observe
   
   client = OpenAI()
   
   @observe()
   def llm_app(query: str) -> str:
       return client.chat.completions.create(
           model="gpt-4o",
           messages=[
               {"role": "user", "content": query}
           ]
       ).choices[0].message.content
   
   # Call app to send trace to Confident AI
   llm_app("Write me a poem.")

   Done ✅. You just created a trace with a span inside it. Go to the Observatory to see your traces there.

Update traces & spans

Once inside an observed function, you can enrich the current trace or span with additional data using update_current_trace / update_current_span (Python) or updateCurrentTrace / updateCurrentSpan (TypeScript).

from deepeval.tracing import observe, update_current_trace, update_current_span

@observe(type="retriever")
def retriever(query: str):
    chunks = retrieve(query)
    update_current_span(input=query, output=chunks)
    return chunks

@observe()
def llm_app(query: str):
    context = retriever(query)
    res = generate(query, context)
    update_current_trace(
        input=query,
        output=res,
        tags=["production"],
        metadata={"app_version": "1.2.3"}
    )
    return res

import {
  observe,
  updateCurrentTrace,
  updateCurrentSpan,
} from "deepeval/tracing";

const retriever = (query: string) => {
  const chunks = retrieve(query);
  updateCurrentSpan({ input: query, output: chunks });
  return chunks;
};
const observedRetriever = observe({ type: "retriever", fn: retriever });

const llmApp = async (query: string) => {
  const context = await observedRetriever(query);
  const res = await generate(query, context);
  updateCurrentTrace({
    input: query,
    output: res,
    tags: ["production"],
    metadata: { appVersion: "1.2.3" },
  });
  return res;
};
const observedLlmApp = observe({ fn: llmApp });

• update_current_trace / updateCurrentTrace sets data on the trace (the outermost observed function) — use it for input/output, tags, metadata, threads, and users.
• update_current_span / updateCurrentSpan sets data on the current span — use it for span-level input/output, metadata, and online eval test case parameters.

Both can be called multiple times from anywhere inside an observed function — values are merged, with later calls overriding earlier ones. Make sure to use the right one — see update_current_trace vs update_current_span in the troubleshooting page.

Using context manager

For pyhton users, if you prefer not to use the @observe decorator, DeepEval also supports the Observer context manager with the same arguments:

from deepeval.tracing import Observer, update_current_span

def generate(prompt: str) -> str:
    with Observer(type="llm", model="gpt-4"):
        res = call_llm(prompt)
        update_current_span(input=prompt, output=res)
        return res

As you learn more about the @observe decorator later on - you can rest assured that everything will automatically apply to context managers as well. This is useful when you can't modify a function's definition or need to instrument a specific code block rather than an entire function.

Instrument Multi-Turn Apps

If your app handles conversations or multi-turn interactions, you can group traces into a thread by providing a thread ID. Each call to your app creates a trace, and traces with the same thread ID are grouped together as a conversation.

from openai import OpenAI
from deepeval.tracing import observe, update_current_trace

client = OpenAI()

@observe()
def llm_app(query: str):
    res = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": query}]
    ).choices[0].message.content

    update_current_trace(thread_id="your-thread-id", input=query, output=res)
    return res

llm_app("What's the weather in SF?")
llm_app("What about tomorrow?")

import OpenAI from "openai";
import { observe, updateCurrentTrace } from "deepeval/tracing";

const llmApp = async (query: string) => {
  const openai = new OpenAI();
  const res = await openai.chat.completions.create({
    model: "gpt-4o",
    messages: [{ role: "user", content: query }],
  });
  const data = res.choices[0].message.content;
  updateCurrentTrace({
    threadId: "your-thread-id",
    input: query,
    output: data,
  });
  return data;
};

const observedLlmApp = observe({ fn: llmApp });
await observedLlmApp("What's the weather in SF?");
await observedLlmApp("What about tomorrow?");

The thread ID can be any string (e.g., a session ID from your app). The input and output is recommended to be the raw user text and LLM response respectively — Confident AI uses these as the conversation turns for display and thread evaluations.

Next steps

Now that you've learnt the very basics of instrumenting your AI app, dive deeper into: