LlamaIndex

Overview

LlamaIndex is an LLM framework that makes it easy to build knowledge agents from complex data. Confident AI allows you to trace and evaluate LlamaIndex agents in just a few lines of code — every agent run, retrieval, and tool call shows up in the Observatory with its full hierarchy, so you can see what the agent retrieved, which tools it picked, and what each model call cost.

Auto-Instrument

1. Install Dependencies

   Run the following command to install confident-trace along with LlamaIndex and the model provider package your agent uses (the example below uses OpenAI):

   pip install confident-trace llama-index-core llama-index-llms-openai

2. Setup Confident AI Key

   Get your Confident AI Project API key and set it as an environment variable, or pass it to init() directly:

   export CONFIDENT_API_KEY="<your-confident-api-key>"
   export OPENAI_API_KEY="<your-openai-key>"

3. Instrument LlamaIndex

   Call init() once at startup, before defining or running agents. Every subsequent LlamaIndex call in your application will automatically be traced and sent to Confident AI — keep your normal agent.run call.

   import asyncio
   
   from confident_trace import init, shutdown
   from llama_index.core.agent.workflow import FunctionAgent
   from llama_index.core.tools import FunctionTool
   from llama_index.llms.openai import OpenAI
   
   
   def multiply(a: float, b: float) -> float:
       """Useful for multiplying two numbers."""
       return a * b
   
   
   async def main():
       init()
       try:
           agent = FunctionAgent(
               name="assistant",
               llm=OpenAI(model="gpt-4o-mini"),
               tools=[FunctionTool.from_defaults(multiply)],
               system_prompt="You are a helpful assistant that can perform calculations.",
           )
           print(await agent.run("What is 3 * 12?"))
       finally:
           shutdown()
   
   
   if __name__ == "__main__":
       asyncio.run(main())

4. Run LlamaIndex

   Run your agent by executing the script:

   python main.py

   Done ✅. Open the Observatory in your Confident AI project to inspect the trace and its child spans.

What Gets Captured

LlamaIndex agent and workflow runs are exported with their hierarchy intact, so you can follow an execution from the entry point through its steps, retrieval, model calls, and tools.

• Agent and workflow execution — operation names, timing, status, and the parent-child relationships between steps.
• Model calls — messages, model details, tool requests, finish reasons, and token usage.
• Tool calls — tool names and their input/output.
• Retrieval and workflow steps — the intermediate operations LlamaIndex reports while processing the request.
• Custom spans — any custom application spans created inside an operation remain nested under it.

Captured inputs, outputs, messages, and retrieved content follow the content policy.

Set Trace Span Properties

Use a trace context to add properties you know before the call starts. It creates no extra span; the trace started by agent.run() inherits the tags, metadata, user ID, and customer ID.

from confident_trace import init, trace_context

init()

async with trace_context(
    tags=["support"],
    metadata={"release": "2026-09"},
    user_id="user-42",
    customer_id="customer-7",
):
    result = await agent.run("Explain OpenTelemetry in one sentence.")

See users and customers to set an optional display name alongside each ID, and trace context for every supported trace property and update behavior.

Instrumenting Multi-Turn

You do not need turn() when one LlamaIndex entry-point call is already one conversational turn—the integration creates that turn's trace automatically. Use turn() when you want to define the boundary yourself, such as grouping two sequential LlamaIndex calls into one turn. Reuse the same thread ID on later turns to group them into one conversation.

from confident_trace import init, turn

init()

async with turn("support-turn", thread_id="chat-42"):
    context = await agent.run("Find the relevant account details.")
    answer = await agent.run(f"Summarize these details: {context}")

See threads for thread I/O, turn IDs, and user IDs.

Disable LlamaIndex Instrumentation

Pass init() a list of integration identifiers to opt in to only those integrations. The identifier for LlamaIndex is "llamaindex" in Python; omit it to disable this integration. An empty list disables all automatic instrumentation:

from confident_trace import init
init(instrumentations=())
# Use ("llamaindex",) to opt in; omit "llamaindex" to disable it.

This turns off Confident AI's automatic instrumentation; calls made after initialization are not instrumented by this integration.

Next Steps

Now that your agent is traced, dive deeper into:

Last updated on October 23, 2026