LlamaIndex

Overview

LlamaIndex is an LLM framework that makes it easy to build knowledge agents from complex data. Confident AI allows you to trace and evaluate LlamaIndex agents in just a few lines of code.

Tracing Quickstart

1. Install Dependencies

   Run the following command to install the required packages:

   pip install -U deepeval llama-index

2. Setup Confident AI Key

   Login to Confident AI using your Confident API key.

   deepeval login

   import deepeval
   
   deepeval.login("<your-confident-api-key>")

3. Instrument LlamaIndex

   Call instrument_llama_index once at startup, passing LlamaIndex's root dispatcher. Every subsequent LlamaIndex call in your application will automatically be traced and sent to Confident AI.

   import asyncio
   from llama_index.llms.openai import OpenAI
   from llama_index.core.agent import FunctionAgent
   import llama_index.core.instrumentation as instrument
   
   from deepeval.integrations.llama_index import instrument_llama_index
   instrument_llama_index(instrument.get_dispatcher())
   
   def multiply(a: float, b: float) -> float:
       """Useful for multiplying two numbers."""
       return a * b
   
   agent = FunctionAgent(
       tools=[multiply],
       llm=OpenAI(model="gpt-4o-mini"),
       system_prompt="You are a helpful assistant that can perform calculations.",
   )
   
   async def main():
       return await agent.run("What is 3 * 12?")
   
   asyncio.run(main())

   You can directly view the traces on Confident AI by clicking on the link printed in the console output.

What Gets Traced

The integration captures the following span types automatically:

Retrieval context from RetrievalEndEvent is automatically attached to the enclosing span, making it available for retrieval-based metrics.

Each span is tagged with integration: "LlamaIndex" so you can filter by framework on Confident AI.

Advanced Features

Set trace attributes

You can attach metadata, user identifiers, and other attributes to a trace by wrapping your LlamaIndex call inside the trace context manager.

import asyncio
from llama_index.core.agent import FunctionAgent
from llama_index.llms.openai import OpenAI
import llama_index.core.instrumentation as instrument
from deepeval.integrations.llama_index import instrument_llama_index
from deepeval.tracing import trace

instrument_llama_index(instrument.get_dispatcher())

agent = FunctionAgent(
    tools=[],
    llm=OpenAI(model="gpt-4o-mini"),
    system_prompt="You are a helpful assistant.",
)

async def handle_request(user_input: str, user_id: str, thread_id: str):
    with trace(
        user_id=user_id,
        thread_id=thread_id,
        tags=["production", "v2"],
        metadata={"environment": "prod"},
    ):
        return await agent.run(user_input)

asyncio.run(handle_request("Hello!", user_id="user-42", thread_id="conv-99"))

Evals Usage

Online evals

You can run online evals on your LlamaIndex application to evaluate all incoming traces on Confident AI's servers. This approach is recommended when your application is in production.

1. Create metric collection

   Create a metric collection on Confident AI with the metrics you wish to use to evaluate your LlamaIndex application.

2. Run evals

   Pass metric_collection to the trace context manager to evaluate the entire trace with your chosen metric collection.

   import asyncio
   from llama_index.llms.openai import OpenAI
   from llama_index.core.agent import FunctionAgent
   import llama_index.core.instrumentation as instrument
   from deepeval.integrations.llama_index import instrument_llama_index
   from deepeval.tracing import trace
   
   instrument_llama_index(instrument.get_dispatcher())
   
   def multiply(a: float, b: float) -> float:
       """Useful for multiplying two numbers."""
       return a * b
   
   agent = FunctionAgent(
       tools=[multiply],
       llm=OpenAI(model="gpt-4o-mini"),
       system_prompt="You are a helpful assistant.",
   )
   
   async def llm_app(user_input: str):
       with trace(metric_collection="my_metric_collection"):
           return await agent.run(user_input)
   
   asyncio.run(llm_app("What is 3 * 12?"))

Span-level evals

For finer-grained control, you can attach metrics or a metric collection directly to individual Agent or LLM spans using AgentSpanContext or LlmSpanContext. This lets you evaluate specific spans independently.

import asyncio
from llama_index.llms.openai import OpenAI
from llama_index.core.agent import FunctionAgent
import llama_index.core.instrumentation as instrument
from deepeval.integrations.llama_index import instrument_llama_index
from deepeval.tracing import trace
from deepeval.tracing.trace_context import AgentSpanContext
from deepeval.metrics import AnswerRelevancyMetric

instrument_llama_index(instrument.get_dispatcher())

def multiply(a: float, b: float) -> float:
    """Useful for multiplying two numbers."""
    return a * b

agent = FunctionAgent(
    tools=[multiply],
    llm=OpenAI(model="gpt-4o-mini"),
    system_prompt="You are a helpful assistant that can perform calculations.",
)

answer_relevancy = AnswerRelevancyMetric()

async def llm_app(user_input: str):
    agent_span_context = AgentSpanContext(
        metrics=[answer_relevancy],
    )
    with trace(agent_span_context=agent_span_context):
        return await agent.run(user_input)

asyncio.run(llm_app("What is 3 * 12?"))

View on Confident AI

You can view the evals on Confident AI by clicking on the link in the output printed in the console.