Introduction to LLM Observability & Tracing
Detect anomalies and regressions across instrumented AI apps, then trace them to their root cause.
Overview
Confident AI helps teams detect anomalies, regressions, and emerging failure modes across instrumented AI apps. It combines evaluation-driven observability with end-to-end tracing, so you can monitor quality, behavior, latency, and cost—and then inspect the exact execution behind a change.
Why LLM Observability?
AI apps can regress even when the code and infrastructure stay healthy. Model behavior, retrieval quality, tool use, user inputs, latency, and cost can all change in production.
With Confident AI, you can:
- Continuously evaluate production traffic using 50+ metrics and online evaluations.
- Detect anomalies and regressions across quality, reliability, latency, cost, and classifier outcomes with Monitors.
- Alert your team when quality or operational metrics cross a threshold using Alerts.
- Find the segments driving a change through breakdowns and Signals.
- Investigate the root cause by opening the traces behind a change and inspecting the responsible model call, retrieval, tool execution, or conversation.
How LLM Tracing Works
Tracing enables observability by capturing each execution of your AI app. When monitoring detects a change, traces provide the context needed to find its root cause—down to the model call, retrieval, tool execution, or conversation responsible.
There are three ways to instrument and trace your app:
| Approach | Best For | Language Support |
|---|---|---|
observe decorator/wrapper | Full control over spans, attributes, and trace structure | Python, TypeScript |
| Third-party integrations | Auto-instrument popular frameworks (OpenAI, LangChain, Pydantic AI, Vercel AI SDK, etc.) | Python and/or Typescript |
| OpenTelemetry (OTEL) | Language-agnostic, standards-based instrumentation | Python, TypeScript, Go, Ruby, C#, and more |
There are three core data types to be aware of:
- Traces — a single end-to-end execution of your LLM app
- Spans — individual components within a trace (e.g., LLM calls, retrievals, tool executions)
- Threads — a group of traces representing a multi-turn conversation
By instrumenting your application, every execution is captured. Traces, spans, and threads can then be automatically evaluated against your metrics and monitored for changes over time.
Get started
Instrument your LLM application with confident-trace automatic instrumentation in Python and TypeScript, or existing OpenTelemetry (OTEL) instrumentation:
5 Min Quickstart
Instrument your LLM app and start tracing in minutes.
Integrations & OTEL
Auto-instrument with one-line integrations for OpenAI, LangChain, and more — or use OpenTelemetry for any language.
Key capabilities
Monitors
Detect anomalies and regressions across quality, reliability, latency, cost, and classifier outcomes.
Alerts
Notify your team when monitored metrics cross a threshold so issues can be investigated quickly.
Online Evals
Run evaluations on traces, spans, and threads in real-time as they're ingested, or retrospectively.
Signals
Auto-classify traces and threads with LLM-driven labels you define, and surface spikes, breakdowns, and trends.
Latency & Cost Monitoring
Monitor execution time and token costs across your application and identify unexpected changes.
Root-Cause Investigation
Drill into the traces behind a regression to find the model call, retrieval, tool execution, or conversation responsible.
Learn the fundamentals
New to LLM observability and tracing? These concepts will help you get the most out of your setup:
- Span Types — classify spans as LLM, retriever, tool, or agent
- Input/Output — control what data is captured on traces and spans
- Threads — group traces into multi-turn conversations
How will tracing affect my app?
Confident AI tracing is designed to be completely non-intrusive to your application. It:
- Can be disabled/enabled anytime through the
OTEL_SDK_DISABLED="true"/"false"environment variable — handy for CI and local development. - Requires no rewrite of your existing code — call
init()once and your installed LLM SDKs and frameworks are instrumented automatically. - Exports asynchronously in the background in batches, so it won't add latency to your LLM calls.
- Fails silently if there are any issues, ensuring your app keeps running.
- Works with any function signature — you can set input/output at runtime.
What languages and frameworks are supported?
confident-trace supports Python (3.10+) and TypeScript (Node.js 22+), and auto-instruments OpenAI, Anthropic, LangChain, LangGraph, and many more out of the box. Via OpenTelemetry, you can instrument in any language — including Python, TypeScript, Go, Ruby, and C#. See the Integrations & OTEL page for the full list.
Last updated on