LLM Tracing

Introduction to LLM Tracing

Instrument your LLM applications for observability and evaluate them on a component level.

Overview

Confident AI offers LLM tracing for teams to instrument, observe, and quality-monitor AI apps. Think Datadog for AI apps, but with an additional suite of 50+ evaluation metrics to monitor AI quality over time.

There are three ways to instrument your app for tracing on Confident AI:

ApproachBest ForLanguage Support
observe decorator/wrapperFull control over spans, attributes, and trace structurePython, TypeScript
Third-party integrationsAuto-instrument popular frameworks (OpenAI, LangChain, Pydantic AI, Vercel AI SDK, etc.)Python and/or Typescript
OpenTelemetry (OTEL)Language-agnostic, standards-based instrumentationPython, TypeScript, Go, Ruby, C#, and more

Why LLM Tracing?

Tracing is the underlying infrastructure that powers online evals, which allows you to monitor AI quality in production. and there are three core data types to be aware of:

  • Traces — a single end-to-end execution of your LLM app
  • Spans — individual components within a trace (e.g., LLM calls, retrievals, tool executions)
  • Threads — a group of traces representing a multi-turn conversation

By instrumenting your application, every execution is captured and all three can be automatically evaluated against your metrics.

LLM Tracing: Traces with Evals

Get started

Instrument your LLM application using the @observe decorator, one-line integrations, or OpenTelemetry (OTEL):

5 Min Quickstart

Manually instrument your LLM app start tracing in minutes, in either Python or Typescript.

Integrations & OTEL

Auto-instrument with one-line integrations for OpenAI, LangChain, and more — or use OpenTelemetry for any language.

Key capabilities

Online Evals

Run evaluations on traces, spans, and threads in real-time as they’re ingested, or retrospectively.

Latency & Cost Tracking

Automatically track execution time and token costs across every span in your application.

Customize Traces

Enrich traces with tags, metadata, user IDs, and thread groupings for filtering and attribution.

Trace Management

Route traces to projects, separate by environment, mask PII, and control sampling rates.

Learn the fundamentals

New to LLM tracing? These concepts will help you get the most out of your setup:

  • Span Types — classify spans as LLM, retriever, tool, or agent
  • Input/Output — control what data is captured on traces and spans
  • Threads — group traces into multi-turn conversations

Confident AI tracing is designed to be completely non-intrusive to your application. It:

  • Can be disabled/enabled anytime through the CONFIDENT_TRACING_ENABLED="YES"/"NO" environment variable.
  • Requires no rewrite of your existing code - just add the @observe decorator.
  • Runs asynchronously in the background with zero impact on latency.
  • Fails silently if there are any issues, ensuring your app keeps running.
  • Works with any function signature - you can set input/output at runtime.

For the @observe decorator, Python and TypeScript are supported. For integrations, Python is supported with TypeScript coming soon. Via OpenTelemetry, you can instrument in any language — including Python, TypeScript, Go, Ruby, and C#. See the Integrations & OTEL page for the full list.