Troubleshooting

Overview

This page covers common tracing issues caused by Python’s concurrency model and process lifecycle. If you’re experiencing any of the following, check the relevant sections below:

• Unexpected new traces appearing instead of spans nesting under a parent trace.
• Traces not showing up on the Confident AI dashboard after execution.
• Trace attributes not set correctly — output, name, or metadata reflecting the wrong values.
• Missing output on streamed responses — trace appears but with no output.

Using @observe with ThreadPoolExecutor

Python’s concurrent.futures.ThreadPoolExecutor spawns new threads that do not inherit ContextVar values from the calling thread. Since deepeval tracing relies on ContextVar to track the active span, submitting an @observe-decorated function directly to an executor produces a separate, orphaned trace instead of nesting under the parent.

The fix is to snapshot the caller’s context with contextvars.copy_context() and use ctx.run when submitting work:

1
from concurrent.futures import ThreadPoolExecutor
2
from contextvars import copy_context
3
from deepeval.tracing import observe
4
5
@observe()
6
def child_task(item):
7
    ...
8
9
# ❌ Broken — child_task creates a separate trace
10
@observe()
11
def parent():
12
    with ThreadPoolExecutor() as executor:
13
        future = executor.submit(child_task, item)
14
15
# ✅ Fixed — child_task nests under parent
16
@observe()
17
def parent():
18
    ctx = copy_context()
19
    with ThreadPoolExecutor() as executor:
20
        future = executor.submit(ctx.run, child_task, item)

Traces Not Showing Up

Confident AI uses batch ingestion for traces, so it is normal for a trace to take up to 30 seconds to appear on the dashboard after it has been posted. If your traces still don’t show up after that window, the most likely cause is your process exiting before the background worker finishes posting — common in serverless functions (AWS Lambda, Google Cloud Functions, etc.) and short-lived scripts.

To fix this, set the CONFIDENT_TRACE_FLUSH environment variable to force DeepEval to flush traces synchronously before the function returns:

$
export CONFIDENT_TRACE_FLUSH=1

Or set it inline when running a script:

$
CONFIDENT_TRACE_FLUSH=1 python main.py

Using @observe with asyncio.run_in_executor()

loop.run_in_executor() delegates work to a thread pool under the hood, so it has the same ContextVar propagation issue as ThreadPoolExecutor — child spans will create orphaned traces instead of nesting under the parent.

Apply the same copy_context() fix:

1
import asyncio
2
from contextvars import copy_context
3
from deepeval.tracing import observe
4
5
@observe()
6
async def child_task(item):
7
    ...
8
9
# ❌ Broken — child_task creates a separate trace
10
@observe()
11
async def parent():
12
    loop = asyncio.get_event_loop()
13
    await loop.run_in_executor(None, child_task, item)
14
15
# ✅ Fixed — child_task nests under parent
16
@observe()
17
async def parent():
18
    loop = asyncio.get_event_loop()
19
    ctx = copy_context()
20
    await loop.run_in_executor(None, ctx.run, child_task, item)

Undecorated Parent Function

If the outermost calling function is not decorated with @observe, there is no parent trace for child spans to nest under. Each @observe-decorated function called inside it will create its own independent trace.

1
from deepeval.tracing import observe
2
3
@observe()
4
def retrieve(query):
5
    ...
6
7
@observe()
8
def generate(query, context):
9
    ...
10
11
# ❌ Broken — retrieve and generate each create separate traces
12
def handle_request(query):
13
    context = retrieve(query)
14
    return generate(query, context)
15
16
# ✅ Fixed — both nest under handle_request
17
@observe()
18
def handle_request(query):
19
    context = retrieve(query)
20
    return generate(query, context)

This is easy to miss on entry points like Flask route handlers, FastAPI endpoints, or task-queue workers — make sure the top-level function that kicks off your pipeline is decorated.

Using @observe with multiprocessing

multiprocessing.Process and concurrent.futures.ProcessPoolExecutor spawn entirely separate OS processes that do not share memory with the parent. Unlike threads, contextvars.copy_context() cannot propagate tracing context across process boundaries.

Traces created inside child processes will always be independent, top-level traces. There is no workaround for this — if you need child processes to produce spans that nest under a parent, consider switching to ThreadPoolExecutor with the copy_context() fix described above.

update_current_trace vs update_current_span

update_current_trace() updates the trace (the top-level unit), not the span of the function it’s called in. If you call it from a child @observe-decorated function expecting it to set that child’s span data, it will set the trace-level fields instead.

To update a child function’s own span, use update_current_span():

1
from deepeval.tracing import observe, update_current_span, update_current_trace
2
3
@observe()
4
def get_chat_answer(query):
5
    result = ...
6
    update_current_span(output=result, name="Get Chat Answer")
7
8
@observe()
9
def get_related_questions(query):
10
    result = ...
11
    update_current_span(output=result, name="Get Related Questions")
12
13
@observe()
14
def handle_message(query):
15
    get_chat_answer(query)
16
    get_related_questions(query)
17
    update_current_trace(name="Handle Message")

Use update_current_trace() in the top-level function to set trace-level fields like name, tags, or metadata. Use update_current_span() everywhere else.

Streaming Functions Missing Trace Output

When an @observe-decorated function uses yield to stream its response (e.g. a FastAPI StreamingResponse), the trace output won’t be captured automatically because the return value is a generator — not the final assembled text.

To fix this, collect the streamed output and set it explicitly with update_current_trace():

1
from fastapi.responses import StreamingResponse
2
from deepeval.tracing import observe, update_current_trace
3
4
@observe()
5
def generate_stream(query):
6
    chunks = []
7
    for chunk in llm.stream(query):
8
        chunks.append(chunk)
9
        yield chunk
10
    update_current_trace(output="".join(chunks))
11
12
@app.post("/chat")
13
async def chat(query: str):
14
    return StreamingResponse(generate_stream(query))

Without this, the trace will appear on Confident AI with no output.