LLM Tracing Troubleshooting

Overview

This page covers the most common tracing issues with confident-trace — most of them come down to process lifecycle, how tracing context propagates through your app, or how the SDK hooks into the libraries you call. If you're experiencing any of the following, check the relevant sections below:

• No traces appear on the Confident AI dashboard after your app runs.
• Traces cut off in serverless functions or short-lived scripts — the first few show up, the rest never do.
• Traces are incomplete because expected job, session, or parent spans never arrive.
• Unexpected new traces appearing instead of spans nesting under a parent trace.
• Trace attributes set on the wrong thing — you called update_trace expecting to update a span (or vice versa).
• Missing or truncated input/output on spans.
• The same LLM call shows up twice in a trace.
• Stacking @span with another tracing decorator and unsure if it's safe or what order to use.

No Traces Appear

Confident AI uses batch ingestion for traces, so it is normal for a trace to take up to 30 seconds to appear on the dashboard after it has been sent. If your traces still don't show up after that window, work through this list — 99% of the time it's one of the first two.

Your process exited before the queue drained

confident-trace exports spans in batches from a background worker. If your program returns before that worker gets a chance to post, the tail end of your traces is lost — for a short script, that's often all of them. Call shutdown() before your process exits (or flush() in a long-running process):

from confident_trace import init, shutdown

init()

try:
    llm_app("Write me a poem.")
finally:
    shutdown()  # flushes the queue, then tears down the exporter

import { init } from "confident-trace";

const runtime = init();

try {
  await llmApp("Write me a poem.");
} finally {
  await runtime.shutdown(); // flushes the queue, then tears down the exporter
}

See flush and shutdown for the full picture, and the serverless section below if you're on Lambda or similar.

You're missing the preload

# ❌ Broken — init() runs, but nothing is instrumented
node --import tsx src/index.ts

# ✅ Fixed — the preload hooks packages as Node loads them
node --import tsx --import confident-trace/register src/index.ts

# Compiled JavaScript — drop --import tsx and point at your build output
node --import confident-trace/register dist/index.js

init() isn't running first

init() has to run before the SDKs and frameworks you want traced make their first call — clients constructed before init() may already hold un-patched references. Call it at the top of your entry point, before creating your OpenAI client, LangChain graph, and so on.

Configuration is wrong or tracing is disabled

Check that:

• CONFIDENT_API_KEY is set (or passed to init(api_key=...) / init({ apiKey })).
• You're pointing at the right endpoint. The default is our US region — EU and self-hosted users need CONFIDENT_OTEL_ENDPOINT set, otherwise traces go to the wrong deployment and fail to authenticate. See configure init().
• OTEL_SDK_DISABLED isn't set to true in this environment (it's a handy CI switch that's easy to forget about).

Your endpoint uses an untrusted certificate

If CONFIDENT_OTEL_ENDPOINT points at a deployment whose certificate your machines don't trust, such as a POC environment behind a load balancer with a self-signed certificate, every export fails with a certificate verification error like CERTIFICATE_VERIFY_FAILED. Set CONFIDENT_OTEL_TLS_SKIP_VERIFY to skip certificate verification for the SDK's connections to Confident AI:

export CONFIDENT_OTEL_TLS_SKIP_VERIFY=true

Only the exact value true turns verification off, and any other value keeps it on. It covers trace exports and LiveKit call recording uploads. Only do this inside a trusted network.

Another OpenTelemetry provider is already registered

If your app already owns a global TracerProvider (for example via another observability vendor's SDK), confident-trace needs to plug into that provider rather than compete with it for the global:

See existing OpenTelemetry provider for the full setup.

You're exporting over gRPC

Confident AI's OTEL server accepts OTLP over HTTP only. Several OpenTelemetry SDKs and the Collector default to gRPC, so if you configured the exporter yourself (rather than through confident-trace's init()) and see connection resets, UNAVAILABLE / UNIMPLEMENTED status codes, or HTTP 415 errors, you're almost certainly on the wrong protocol.

Switch to the HTTP/protobuf exporter for your runtime:

Also check that OTEL_EXPORTER_OTLP_ENDPOINT has no port suffix like :4317 — that's the gRPC convention, and the OTEL server listens on standard HTTPS.

Full exporter setup per language is on the manual instrumentation page.

Short-Lived Processes and Serverless Functions

Serverless runtimes (AWS Lambda, Google Cloud Functions, Vercel, etc.) freeze or tear down the environment as soon as your handler returns. Because spans are exported asynchronously, a handler that returns immediately after the LLM call will often lose its trace — the background worker never gets scheduled.

The fix is to call flush() at the end of the handler, after any streams have finished, so the queue is drained before the environment freezes:

from confident_trace import init, flush

init()  # once, at module load — not inside the handler

def handler(event, context):
    result = llm_app(event["query"])
    # ✅ drain the queue before the runtime freezes
    flush(timeout_millis=30000)
    return result

import { init } from "confident-trace";

const runtime = init(); // once, at module load — not inside the handler

export const handler = async (event: { query: string }) => {
  const result = await llmApp(event.query);
  // ✅ drain the queue before the runtime freezes
  await runtime.flush(30000);
  return result;
};

Use flush() per invocation and shutdown() only when the process is genuinely terminating. Calling shutdown() in a handler tears down the exporter, and the next warm invocation will have nothing to export with.

Incomplete Traces

If child spans arrive but their expected job, session, or root spans do not, the backend received only part of the trace. OpenTelemetry does not have a separate "root ID": spans belong together through a shared trace ID, and each child references its parent's span ID. A trace looks incomplete when that referenced parent span is never exported.

The Tracer Provider Is Recreated Per Job

In Python, this can happen when a long-lived worker creates a TracerProvider and calls init() for every job, session, or request. init() configures tracing once per process, so later calls return the existing runtime and ignore the new tracer_provider:

from confident_trace import init
from opentelemetry.sdk.trace import TracerProvider

# ❌ This creates a new provider for every job.
def setup_job_tracing():
    provider = TracerProvider()
    init(tracer_provider=provider)  # Later calls ignore this provider.
    return provider

Nothing errors, but the provider created for a later job does not have Confident AI's exporter attached. The result can look inconsistent:

• Job, session, or root spans created by the new provider never reach Confident AI.
• Child LLM or tool spans still using the original provider arrive without their expected parents.

For example, a reused LiveKit worker can create its job and session spans with a new provider while confident-trace integrations continue using the provider from the first init() call. LiveKit uses a separate process per job by default, but shared thread executors, dev or console modes, and custom worker configurations can reuse a process.

Call init() once with a process-wide provider, then configure every job to use that same provider:

from confident_trace import init
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider

provider = TracerProvider()
trace.set_tracer_provider(provider)
init(tracer_provider=provider)

The framework must use this provider for its parent spans. Unlike a context-propagation problem, the parent context can be correct while the provider that created the parent has no Confident AI exporter. For LiveKit, keep init() at module scope as shown in the LiveKit setup.

The Root Span Is Never Exported

A child span can end and export while its manually created root span remains open. OpenTelemetry exports ended spans, so forgetting to end the root leaves the backend with the child spans but no root:

from opentelemetry import trace

tracer = trace.get_tracer(__name__)

# ❌ This root becomes current but does not end when the block exits.
root = tracer.start_span("request")
with trace.use_span(root):
    call_model()

# ✅ Both the root and its child spans end and can be exported.
with tracer.start_as_current_span("request"):
    call_model()

The same symptom occurs when a process exits before the root ends, an OpenTelemetry Collector filter drops the root span, or custom export logic sends only a subset of spans. Make sure the root ends before flushing, and check that any Collector or exporter filters apply consistently to the whole trace.

Missing Parent Spans in Threads

If spans that should nest under a parent are showing up as separate, top-level traces, the tracing context isn't reaching the code that creates them. Where that happens depends on your runtime:

from concurrent.futures import ThreadPoolExecutor
from contextvars import copy_context
from confident_trace import span

@span(type="tool")
def child_task(item):
    ...

# ❌ Broken — child_task creates a separate trace
@span(type="agent")
def parent(items):
    with ThreadPoolExecutor() as executor:
        futures = [executor.submit(child_task, item) for item in items]

# ✅ Fixed — child_task nests under parent
@span(type="agent")
def parent(items):
    ctx = copy_context()
    with ThreadPoolExecutor() as executor:
        futures = [executor.submit(ctx.run, child_task, item) for item in items]

import asyncio
from contextvars import copy_context
from confident_trace import span

@span(type="tool")
def child_task(item):
    ...

# ❌ Broken — child_task creates a separate trace
@span(type="agent")
async def parent(item):
    loop = asyncio.get_event_loop()
    await loop.run_in_executor(None, child_task, item)

# ✅ Fixed — child_task nests under parent
@span(type="agent")
async def parent(item):
    loop = asyncio.get_event_loop()
    ctx = copy_context()
    await loop.run_in_executor(None, ctx.run, child_task, item)

Undecorated Parent Function

If the outermost function that kicks off your pipeline isn't wrapped in a span, there is no parent trace for child spans to nest under — each @span-decorated function (and each auto-instrumented LLM call) becomes its own independent trace.

from confident_trace import span

@span(type="retriever")
def retrieve(query):
    ...

@span(type="llm", model="gpt-4o")
def generate(query, context):
    ...

# ❌ Broken — retrieve and generate each create separate traces
def handle_request(query):
    context = retrieve(query)
    return generate(query, context)

# ✅ Fixed — both nest under handle_request
@span(type="agent")
def handle_request(query):
    context = retrieve(query)
    return generate(query, context)

This is easy to miss on entry points like Flask route handlers, FastAPI endpoints, or task-queue workers — make sure the top-level function that starts your pipeline is the one with the span. Auto-instrumentation captures your LLM calls, but it has no way of knowing where your request begins.

update_trace vs update_span

update_trace() updates the trace — the outermost span, i.e. the whole request — no matter where you call it from. update_span() updates whichever span is currently active, i.e. the component you're inside right now. Mixing them up is the most common reason "my span output is empty" or "my trace name keeps getting overwritten":

from confident_trace import span, update_span, update_trace

# ❌ Broken — sets the *trace* output from inside a child span,
#    so the trace shows the tool result and the span shows nothing
@span(type="tool")
def lookup_order(order_id):
    result = db.get(order_id)
    update_trace(output=result)
    return result

# ✅ Fixed — each helper writes to the level it's named after
@span(type="tool")
def lookup_order(order_id):
    result = db.get(order_id)
    update_span(input=order_id, output=result)
    return result

@span(type="agent")
def handle_message(query):
    result = lookup_order(query)
    res = generate(query, result)
    update_trace(
        input=query,
        output=res,
        tags=["production"],
        thread_id="your-thread-id",
    )
    return res

As a rule of thumb:

• update_trace — request-level facts: input/output as the user saw them, tags, metadata, thread ID, user identity, customer identity, name, environment. It's fine to call it from a child span — it always targets the trace — but do it once, in the entry point, so you know what the final values are.
• update_span — the component you're in: this span's input/output, model, retrieval context, tools called, and other online eval test case parameters.

Both can be called multiple times; values are merged, with later calls overriding earlier ones.

There's also a third helper, trace_context / traceContext, which sets trace-level defaults for traces that start inside its scope — without creating a span — so you can attach tags, a thread ID, user identity, or customer identity to a fully auto-instrumented call. See set trace attributes without a span.

Missing or Truncated Content

If a span exists but its input/output is blank, [REDACTED], or cut short, one of the SDK's content controls is doing its job:

• Capture is off. init(capture_content=False) / init({ captureContent: false }) records structure and timing but no message bodies.
• A redactor rewrote or dropped it. Custom redact functions run before export; if the redacted result isn't a valid message shape, the SDK omits it rather than sending something malformed.
• It exceeded a configured size cap. Size limits are disabled by default, but content over a max_content_bytes limit you've set is truncated.

These controls apply to content the SDK manages. For framework integrations, what gets captured is governed by that framework's own settings. Images and PDFs are captured up to the media size limits, while audio and video are recorded without their content. See masking for how to tune each of these.

Streaming responses missing output

When a span-wrapped function yields its response (e.g. a FastAPI StreamingResponse), the return value is a generator — not the final assembled text — so there's nothing for the SDK to record as output. Collect the chunks and set the output explicitly:

from fastapi.responses import StreamingResponse
from confident_trace import span, update_trace

@span(type="agent")
def generate_stream(query):
    chunks = []
    for chunk in llm.stream(query):
        chunks.append(chunk)
        yield chunk
    update_trace(input=query, output="".join(chunks))

@app.post("/chat")
async def chat(query: str):
    return StreamingResponse(generate_stream(query))

Without this, the trace will appear on Confident AI with no output.

Duplicate Spans

If every LLM call shows up twice in a trace, two instrumentors are attached to the same client or provider. This usually happens when you combine confident-trace's automatic instrumentation with a manual adapter (or another vendor's instrumentor) for the same library.

Stacking @span with Other Tracing Decorators

If your code already uses another tracing decorator from a different observability system, you can leave it in place. @span is a plain decorator (it uses functools.wraps under the hood) and stacks on top of any other decorator without conflict — e.g. MLflow's @mlflow.trace or an OpenTelemetry @tracer.start_as_current_span.

Order does not change correctness, but it controls which wrapper sits closest to your function. Default to putting @span innermost so your existing dashboard keeps showing the outer call as the root:

from confident_trace import span

@tracer.trace      # your existing tracing decorator stays on top
@span(type="agent")
def run_my_ai_app(query: str) -> str:
    ...

What you'll see depends on whether the other tracer is OpenTelemetry-based:

• Non-OTel tracers (MLflow, LangSmith, etc.) emit to their own backend, @span emits to Confident AI. They don't share span or trace IDs, so seeing the same function logged on both platforms is expected — not a duplicate bug.
• OpenTelemetry tracers — confident-trace is OpenTelemetry, so a @span nested inside an OTel span joins the same trace rather than starting a separate one. That's usually what you want, but it does mean the two aren't independent: if your app already owns an OTel TracerProvider, wire confident-trace into it (see existing OpenTelemetry provider) rather than calling init() on its own, or you'll end up with two providers competing for the global.

Diagnostics

When none of the above explains what you're seeing, the SDK can tell you what it did (and didn't) hook.

Check instrumentation status

import { init } from "confident-trace";

const runtime = init();
console.log(runtime.getInstrumentationStatus());

Enable diagnostics logging

import logging

logging.basicConfig(level=logging.WARNING)
logging.getLogger("confident_trace.diagnostics").setLevel(logging.DEBUG)

import { diag, DiagConsoleLogger, DiagLogLevel } from "@opentelemetry/api";

diag.setLogger(new DiagConsoleLogger(), DiagLogLevel.DEBUG);

If diagnostics look healthy but traces still aren't arriving, the problem is on the transport side — check your exporter's logs for HTTP errors (a 401 almost always means the wrong region or API key).

Next Steps

Last updated on October 9, 2026