LlamaIndex | Confident AI Docs

Overview

LlamaIndex is an LLM framework that makes it easy to build knowledge agents from complex data. Confident AI allows you to trace and evaluate LlamaIndex agents in just a few lines of code.

Tracing Quickstart

Install Dependencies

Run the following command to install the required packages:

$ pip install -U deepeval llama-index

Setup Confident AI Key

$ deepeval login

Instrument LlamaIndex

Call instrument_llama_index once at startup, passing LlamaIndex’s root dispatcher. Every subsequent LlamaIndex call in your application will automatically be traced and sent to Confident AI.

main.py

1 import asyncio
2 from llama_index.llms.openai import OpenAI
3 from llama_index.core.agent import FunctionAgent
4 import llama_index.core.instrumentation as instrument
5 
6 from deepeval.integrations.llama_index import instrument_llama_index
7 instrument_llama_index(instrument.get_dispatcher())
8 
9 def multiply(a: float, b: float) -> float:
10     """Useful for multiplying two numbers."""
11     return a * b
12 
13 agent = FunctionAgent(
14     tools=[multiply],
15     llm=OpenAI(model="gpt-4o-mini"),
16     system_prompt="You are a helpful assistant that can perform calculations.",
17 )
18 
19 async def main():
20     return await agent.run("What is 3 * 12?")
21 
22 asyncio.run(main())

instrument_llama_index registers DeepEval’s handler with LlamaIndex’s instrumentation dispatcher. From that point on, all LlamaIndex spans and events are captured automatically — no other code changes are required.

You can directly view the traces on Confident AI by clicking on the link printed in the console output.

What Gets Traced

The integration captures the following span types automatically:

Span type	When it is created
Agent	Any `Workflow.run()` or `FunctionAgent.run()` call
LLM	Each `LLMChatStartEvent` / `LLMChatEndEvent` pair — includes input messages, model name, and the inferred provider (e.g. `OpenAI`, `Anthropic`)
Tool	Any `call_tool` / `acall_tool` / `acall` invocation — captures tool name, inputs, and outputs
Generic	All other instrumented LlamaIndex methods

Retrieval context from RetrievalEndEvent is automatically attached to the enclosing span, making it available for retrieval-based metrics.

Each span is tagged with integration: "LlamaIndex" so you can filter by framework on Confident AI.

Advanced Features

Set trace attributes

You can attach metadata, user identifiers, and other attributes to a trace by wrapping your LlamaIndex call inside the trace context manager.

main.py

1 import asyncio
2 from llama_index.core.agent import FunctionAgent
3 from llama_index.llms.openai import OpenAI
4 import llama_index.core.instrumentation as instrument
5 from deepeval.integrations.llama_index import instrument_llama_index
6 from deepeval.tracing import trace
7 
8 instrument_llama_index(instrument.get_dispatcher())
9 
10 agent = FunctionAgent(
11     tools=[],
12     llm=OpenAI(model="gpt-4o-mini"),
13     system_prompt="You are a helpful assistant.",
14 )
15 
16 async def handle_request(user_input: str, user_id: str, thread_id: str):
17     with trace(
18         user_id=user_id,
19         thread_id=thread_id,
20         tags=["production", "v2"],
21         metadata={"environment": "prod"},
22     ):
23         return await agent.run(user_input)
24 
25 asyncio.run(handle_request("Hello!", user_id="user-42", thread_id="conv-99"))

View Trace Attributes

name

str

The name of the trace. Learn more.

Evals Usage

Online evals

You can run online evals on your LlamaIndex application to evaluate all incoming traces on Confident AI’s servers. This approach is recommended when your application is in production.

Create metric collection

Create a metric collection on Confident AI with the metrics you wish to use to evaluate your LlamaIndex application.

Create metric collection

The LlamaIndex integration automatically captures input and actual_output for Agent and LLM spans. Use metrics that only require those fields (e.g. Answer Relevancy, Task Completion) unless you also supply retrieval_context, context, expected_output, or expected_tools explicitly via the trace context manager or span context objects.

Run evals

Pass metric_collection to the trace context manager to evaluate the entire trace with your chosen metric collection.

main.py

1 import asyncio
2 from llama_index.llms.openai import OpenAI
3 from llama_index.core.agent import FunctionAgent
4 import llama_index.core.instrumentation as instrument
5 from deepeval.integrations.llama_index import instrument_llama_index
6 from deepeval.tracing import trace
7 
8 instrument_llama_index(instrument.get_dispatcher())
9 
10 def multiply(a: float, b: float) -> float:
11     """Useful for multiplying two numbers."""
12     return a * b
13 
14 agent = FunctionAgent(
15     tools=[multiply],
16     llm=OpenAI(model="gpt-4o-mini"),
17     system_prompt="You are a helpful assistant.",
18 )
19 
20 async def llm_app(user_input: str):
21     with trace(metric_collection="my_metric_collection"):
22         return await agent.run(user_input)
23 
24 asyncio.run(llm_app("What is 3 * 12?"))

All incoming traces will now be evaluated using metrics from your metric collection.

Span-level evals

For finer-grained control, you can attach metrics or a metric collection directly to individual Agent or LLM spans using AgentSpanContext or LlmSpanContext. This lets you evaluate specific spans independently.

main.py

1 import asyncio
2 from llama_index.llms.openai import OpenAI
3 from llama_index.core.agent import FunctionAgent
4 import llama_index.core.instrumentation as instrument
5 from deepeval.integrations.llama_index import instrument_llama_index
6 from deepeval.tracing import trace
7 from deepeval.tracing.trace_context import AgentSpanContext
8 from deepeval.metrics import AnswerRelevancyMetric
9 
10 instrument_llama_index(instrument.get_dispatcher())
11 
12 def multiply(a: float, b: float) -> float:
13     """Useful for multiplying two numbers."""
14     return a * b
15 
16 agent = FunctionAgent(
17     tools=[multiply],
18     llm=OpenAI(model="gpt-4o-mini"),
19     system_prompt="You are a helpful assistant that can perform calculations.",
20 )
21 
22 answer_relevancy = AnswerRelevancyMetric()
23 
24 async def llm_app(user_input: str):
25     agent_span_context = AgentSpanContext(
26         metrics=[answer_relevancy],
27     )
28     with trace(agent_span_context=agent_span_context):
29         return await agent.run(user_input)
30 
31 asyncio.run(llm_app("What is 3 * 12?"))

View Span Context Parameters

AgentSpanContext — applied to Agent spans (i.e. workflow / agent .run() calls):

metrics

List[BaseMetric]

A list of DeepEval metric instances to evaluate this agent span with.

metric_collection

str

Name of a metric collection on Confident AI to use for evaluation.

expected_output

str

The expected output for the agent span.

expected_tools

List[ToolCall]

The expected tools that should have been called.

context

List[str]

Contextual information for the span.

retrieval_context

List[str]

Retrieved documents or chunks for the span.

LlmSpanContext — applied to LLM spans (i.e. individual LLM calls):

metrics

List[BaseMetric]

A list of DeepEval metric instances to evaluate this LLM span with.

metric_collection

str

Name of a metric collection on Confident AI to use for evaluation.

prompt

Prompt

A Prompt object from deepeval.prompt to associate a managed prompt with this LLM span.

expected_output

str

The expected output for the LLM span.

expected_tools

List[ToolCall]

The expected tools that should have been called.

context

List[str]

Contextual information for the span.

retrieval_context

List[str]

Retrieved documents or chunks for the span.

View on Confident AI

You can view the evals on Confident AI by clicking on the link in the output printed in the console.