LLM tracing is the practice of recording every step an LLM application takes to produce an output — model calls, retrievals, tool executions, agent decisions — as structured, timestamped records you can inspect after the fact. A trace captures one request end to end; spans capture the individual operations inside it.
If you have used distributed tracing in a microservice architecture, the mechanics will feel familiar. The reason LLM tracing exists as its own discipline is that the failure modes are different: an LLM application can return a 200 OK with a confidently wrong answer, and no status code, exception, or latency chart will tell you. Tracing is the layer that makes those failures inspectable.
This guide covers the data model (traces, spans, threads), the OpenTelemetry GenAI semantic conventions that standardize it, the instrumentation approaches available in 2026, and what tracing feeds once it exists. It is written to be useful whichever tracing platform you use.
How is LLM tracing different from APM tracing?
Traditional application performance monitoring answers "did the system run correctly?" — latency, throughput, error rates. LLM tracing must additionally answer "did the system produce a good output?", which requires capturing semantic payloads (prompts, completions, retrieved context, tool arguments) that APM tools deliberately discard.
The differences are structural, not cosmetic:
Dimension | Traditional APM tracing | LLM tracing |
|---|---|---|
Primary failure signal | Exceptions, timeouts, 5xx responses | Wrong, ungrounded, or off-policy outputs that return successfully |
Span payload | Metadata only (duration, status, service name) | Full semantic content: prompts, completions, retrieved documents, tool arguments and results |
Unit of analysis | The request | The request and the conversation (thread) it belongs to |
Cost dimension | CPU, memory, network | Token usage and per-call model spend |
Determinism | Same input, same code path | Same input can produce different outputs and different execution paths |
Downstream consumer | Dashboards and alerts | Dashboards and alerts, plus evaluation, dataset curation, and human review |
The last row matters most. In mature LLM engineering practice, traces are not just a debugging artifact — they are the raw material for detecting production issues and for building regression datasets. A tracing setup that only renders waterfall charts captures a fraction of the value.
The data model: traces, spans, and threads
An LLM tracing data model has three levels. A span is one operation (an LLM call, a retrieval, a tool execution). A trace is the ordered tree of spans produced by one request. A thread is the ordered collection of traces that make up one multi-turn conversation or agent session.
Getting the hierarchy right is the single most important instrumentation decision, because each level answers a different question.
Spans: one operation
A span records a single unit of work with a start time, end time, and a parent. In an LLM application, spans typically fall into a handful of types:
- LLM spans — a call to a model. Should capture the model name, the full prompt (or a redacted version), the completion, token counts, temperature and other parameters, and finish reason.
- Retrieval spans — a query against a vector store or search index. Should capture the query text, the returned chunks, and their scores. Without this, retrieval failures are indistinguishable from generation failures.
- Tool spans — a function or API the model invoked. Should capture the tool name, the arguments the model generated, and the result returned. This is where tool-misuse failures become visible.
- Agent spans — a planning or routing step that contains child spans. In multi-agent systems, these capture handoffs between agents.
- Custom spans — application-specific logic: guardrail checks, cache lookups, post-processing.
Spans nest. A single agent span may contain a retrieval span, three LLM spans, and two tool spans. The tree structure is what lets you answer "which step went wrong?" instead of only "something went wrong."
Traces: one request
A trace is the complete execution record of one input/output pair — a user message in, an application response out, and every span in between. The trace is the unit you debug: when an output is wrong, you open its trace and walk the span tree to find whether the failure originated in retrieval, generation, tool use, or orchestration.
Trace-level metadata should include the application version, prompt version, model version, environment, and user or session identifiers. Version metadata is what turns a trace store into a diagnostic tool — "the failures started with prompt v14" is only answerable if the prompt version is on every trace.
Threads: one conversation
A thread groups traces that share a conversation or session ID. For chatbots, copilots, and agents that handle follow-ups, the thread is the unit of quality: a bot can produce ten individually reasonable replies and still fail the conversation by forgetting facts, contradicting itself, or never resolving the user's goal.
Thread-level failures are invisible at the trace level by construction. If your application is multi-turn, tagging every trace with a stable thread ID is non-negotiable — the reasons are covered in depth in our multi-turn observability playbook.
OpenTelemetry GenAI semantic conventions
OpenTelemetry's GenAI semantic conventions define standard attribute names for LLM spans — gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, and events for prompts and completions — so traces emitted by one SDK can be read by any compliant backend.
The conventions matter for one practical reason: portability. If your instrumentation emits OTel-compliant spans, you can switch tracing backends, or send the same traces to multiple backends, without re-instrumenting your application. Most serious tracing platforms in 2026 — Langfuse, LangSmith, Arize Phoenix, Confident AI, and others — accept OTel GenAI spans, and instrumentation layers like OpenLLMetry and OpenInference build on the same foundation.
Two caveats worth knowing. First, the GenAI conventions are still marked as in development, so attribute names have shifted between versions; pin your semconv version and check release notes before upgrading. Second, the conventions standardize span-level attributes but say little about threads — conversation grouping remains a backend-specific concern, which is worth checking when you evaluate platforms for multi-turn use cases.
Four ways to instrument, compared
There are four instrumentation approaches, and most production teams end up combining two of them: auto-instrumentation for coverage, manual spans for the logic that matters most.
Approach | How it works | Strengths | Weaknesses |
|---|---|---|---|
Manual SDK instrumentation | You wrap functions with decorators or context managers from a tracing SDK | Full control over span boundaries, names, and attributes; captures business logic | Effort scales with codebase; easy to leave gaps |
Auto-instrumentation | A library patches LLM clients (OpenAI, Anthropic, Bedrock) to emit spans automatically | One-line setup; consistent coverage of model calls | Only sees model calls — retrieval, tools, and orchestration logic stay dark |
OTel / OpenInference collectors | Your app emits standard OTel spans; a collector routes them to one or more backends | Vendor-neutral; multi-backend export; reuses existing OTel pipelines | More moving parts; GenAI conventions still stabilizing |
Framework integrations | LangChain, LlamaIndex, OpenAI Agents SDK, Vercel AI SDK emit traces via built-in callbacks | Near-zero setup inside the framework; spans match framework abstractions | Coverage ends at the framework boundary; span granularity is fixed by the framework |
A practical rule: auto-instrumentation and framework integrations get you to "we can see model calls" in an afternoon. Manual spans are what get you to "we can see why the agent chose the wrong tool." Plan for both, and prefer platforms that let the approaches coexist in one trace tree.
On the question of when to do this work: instrument before you think you need to. Teams that wait until their first production incident to add tracing have no history to debug against — the argument is laid out in when should I start tracing?
What tracing feeds: debugging, evaluation, and datasets
Tracing on its own is expensive logging. Its value compounds when traces feed three downstream workflows: debugging individual failures, running evaluations on production behavior, and curating regression datasets from real traffic.
Debugging is the obvious one. A user reports a wrong answer; you open the trace; the span tree shows the retriever returned the right documents but the generation span ignored them. That is a prompt problem, not a retrieval problem — a distinction you cannot make from logs alone.
Evaluation is where LLM tracing departs furthest from APM. Because spans carry full semantic payloads, evaluation metrics — faithfulness, answer relevancy, tool correctness — can run directly on production traces, either on every trace or on sampled subsets. Gartner projects that by 2028, half of enterprises deploying generative AI will invest in LLM observability tooling, and evaluation-on-traces is the capability driving much of that investment: it converts a trace store from a forensic archive into a live quality signal.
Dataset curation closes the loop. Traces that fail evaluation — or that humans flag during review — become test cases in your evaluation dataset, so the next prompt change is tested against real failures rather than synthetic guesses. This trace-to-dataset flow is the mechanism behind the evaluation feedback loop that separates teams whose eval suites grow from teams whose eval suites go stale.
When you evaluate tracing platforms, this is the axis to compare on. Capturing spans is table stakes; the differences show up in whether the platform can run evaluations on traces, group traces into threads, and convert failures into datasets without manual export. Confident AI's tracing is built around that loop, and this guide's criteria apply equally when comparing it against any alternative.
Common tracing mistakes
Four mistakes account for most tracing setups that fail to earn their cost:
- Capturing metadata but not payloads. Spans with durations but no prompts or completions can only answer latency questions. If compliance requires it, redact or mask payloads — do not drop them.
- No version metadata. Traces without prompt, model, and application versions cannot attribute regressions to changes. Every trace should answer "what exact configuration produced this?"
- Ignoring threads. Request-level tracing on a multi-turn product measures the wrong unit. Tag thread IDs from day one, even if you do nothing with them yet.
- Tracing without a consumer. If no evaluation, alerting, or review workflow reads the traces, the store becomes a write-only archive. Decide what will consume traces before deciding what to capture.
Frequently Asked Questions
What is LLM tracing?
LLM tracing is the practice of recording every step an LLM application takes — model calls, retrievals, tool executions, agent decisions — as structured spans grouped into traces. Unlike traditional APM tracing, spans carry full semantic payloads (prompts, completions, retrieved context), because LLM failures are usually wrong outputs that return successfully rather than exceptions.
What is the difference between a trace, a span, and a thread?
A span is one operation, such as a single LLM call or retrieval query. A trace is the ordered tree of spans produced by one request — one input/output pair. A thread is the collection of traces that make up one multi-turn conversation. Spans answer "which step failed," traces answer "what happened on this request," and threads answer "did the conversation succeed."
Do I need OpenTelemetry for LLM tracing?
No, but it is the safest default. OpenTelemetry's GenAI semantic conventions standardize span attributes for model calls, which keeps your instrumentation portable across backends. Most tracing platforms in 2026 accept OTel spans alongside their native SDKs, so emitting OTel-compliant spans preserves the option to switch or multi-home later.
What should an LLM span capture?
At minimum: the operation type (LLM, retrieval, tool, agent), start and end time, parent span, and the semantic payload — prompt and completion for LLM spans, query and returned chunks for retrieval spans, arguments and results for tool spans. Token counts, model parameters, and version metadata (prompt, model, app version) turn spans from logs into diagnostic evidence.
How is LLM tracing different from LLM observability?
Tracing is the capture layer: recording what the application did. Observability is the broader practice built on top of it — dashboards, evaluation on traces, drift detection, alerting, and dataset curation. Tracing without those consumers is just structured logging; observability is what makes the captured data answer quality questions.
Should I trace in development or only in production?
Both, and development first. Traces in development show you what your application actually does before users see it, and they are the substrate for component-level debugging when evals fail. In production, the same instrumentation feeds monitoring, evaluation, and dataset curation. Instrumenting once and reusing it across environments is the whole point of doing it early.
Read next
- When should I start tracing? — the case for instrumenting before your first incident.
- Setting Up AI Agent Observability — the agent-specific trace model: tools, handoffs, and span propagation.
- Setting Up Multi-Turn Agent Observability — why threads, not traces, are the unit of conversational quality.