Pydantic AI

Overview

Pydantic AI is a Python-native LLM agent framework built on the foundations of Pydantic validation. Confident AI allows you to trace and evaluate Pydantic AI agents using an OpenTelemetry-based integration.

Tracing Quickstart

1. Install Dependencies

   Run the following command to install the required packages:

   pip install -U deepeval pydantic-ai

2. Setup Confident AI Key

   Login to Confident AI using your Confident API key.

   export CONFIDENT_API_KEY="<your-confident-api-key>"

   deepeval login

   import deepeval
   
   deepeval.login("<your-confident-api-key>")

3. Configure Pydantic AI

   Pass DeepEvalInstrumentationSettings to your agent's instrument parameter. This sets up the full OpenTelemetry pipeline — including span classification, trace context wiring, and export to Confident AI — in a single step.

   SynchronousAsynchronousStreaming

   from pydantic_ai import Agent
   from deepeval.integrations.pydantic_ai import DeepEvalInstrumentationSettings
   
   agent = Agent(
       "openai:gpt-4o-mini",
       system_prompt="Be concise, reply with one sentence.",
       name="my_agent",
       instrument=DeepEvalInstrumentationSettings(),
   )
   
   result = agent.run_sync("What are LLMs?")
   print(result.output)

4. Run Pydantic AI

   Invoke your agent by executing the script:

   python main.py

   You can view the traces on Confident AI by clicking on the link printed in the console.

Advanced Usage

Logging threads

Threads group related traces together and are useful for chat apps, agents, or any multi-turn interactions. You can learn more about threads here. Pass thread_id to DeepEvalInstrumentationSettings to associate every trace from that agent with a thread.

from pydantic_ai import Agent
from deepeval.integrations.pydantic_ai import DeepEvalInstrumentationSettings

agent = Agent(
    model="openai:gpt-4o-mini",
    system_prompt="Be concise, reply with one sentence.",
    instrument=DeepEvalInstrumentationSettings(
        thread_id="thread_id_1",
    ),
)

result = agent.run_sync("What are LLMs?")

import asyncio
from pydantic_ai import Agent
from deepeval.integrations.pydantic_ai import DeepEvalInstrumentationSettings

agent = Agent(
    model="openai:gpt-4o-mini",
    system_prompt="Be concise, reply with one sentence.",
    instrument=DeepEvalInstrumentationSettings(
        thread_id="thread_id_1",
    ),
)

async def main():
    return await agent.run("What are LLMs?")

result = asyncio.run(main())

import asyncio
from pydantic_ai import Agent
from deepeval.integrations.pydantic_ai import DeepEvalInstrumentationSettings

agent = Agent(
    model="openai:gpt-4o-mini",
    system_prompt="Be concise, reply with one sentence.",
    instrument=DeepEvalInstrumentationSettings(
        thread_id="thread_id_1",
    ),
)

async def main():
    async with agent.run_stream("What are LLMs?") as result:
        async for chunk in result.stream_text(delta=True):
            print(chunk, end="", flush=True)
        final = await result.get_output()
        print("\n\nFinal:", final)

asyncio.run(main())

Trace attributes

You can attach trace-level attributes such as name, tags, metadata, and user ID to every trace produced by the agent. These are baked into DeepEvalInstrumentationSettings as static defaults. They can be overridden at runtime using update_current_trace(...) from inside a tool body.

from pydantic_ai import Agent
from deepeval.integrations.pydantic_ai import DeepEvalInstrumentationSettings

agent = Agent(
    model="openai:gpt-4o-mini",
    system_prompt="Be concise, reply with one sentence.",
    instrument=DeepEvalInstrumentationSettings(
        name="My Agent Trace",
        tags=["production", "v2"],
        metadata={"env": "production"},
        user_id="user_123",
        thread_id="thread_id_1",
    ),
)

result = agent.run_sync("What are LLMs?")

import asyncio
from pydantic_ai import Agent
from deepeval.integrations.pydantic_ai import DeepEvalInstrumentationSettings

agent = Agent(
    model="openai:gpt-4o-mini",
    system_prompt="Be concise, reply with one sentence.",
    instrument=DeepEvalInstrumentationSettings(
        name="My Agent Trace",
        tags=["production", "v2"],
        metadata={"env": "production"},
        user_id="user_123",
        thread_id="thread_id_1",
    ),
)

async def main():
    return await agent.run("What are LLMs?")

result = asyncio.run(main())

Update trace attributes

You can enrich a trace mid-flight from inside a tool body using update_current_trace. This is useful when trace metadata depends on information only available during execution, such as a user ID resolved by a lookup tool.

from pydantic_ai import Agent
from deepeval.tracing import update_current_trace
from deepeval.integrations.pydantic_ai import DeepEvalInstrumentationSettings

agent = Agent(
    "openai:gpt-4o-mini",
    instrument=DeepEvalInstrumentationSettings(),
)

@agent.tool_plain
def lookup_user(user_id: str) -> str:
    # Enrich the trace with data resolved at runtime
    update_current_trace(
        user_id=user_id,
        metadata={"plan": "pro", "region": "us-east"},
    )
    return f"User {user_id} profile loaded."

result = agent.run_sync("Load my profile for user_42.")

Update span attributes

You can attach span-level attributes such as metadata or a metric collection from inside a tool body using update_current_span. This is the primary way to configure per-tool evaluation behavior.

from pydantic_ai import Agent
from deepeval.tracing import update_current_span
from deepeval.integrations.pydantic_ai import DeepEvalInstrumentationSettings

agent = Agent(
    "openai:gpt-4o-mini",
    instrument=DeepEvalInstrumentationSettings(),
)

@agent.tool_plain
def get_weather(city: str) -> str:
    update_current_span(
        metadata={"city": city, "source": "mock"},
        metric_collection="weather-tool-evals",
    )
    return f"{city}: sunny, 22°C"

result = agent.run_sync("What is the weather in Tokyo?")

Logging prompts

If you are managing prompts on Confident AI and wish to log them, use next_llm_span to associate a Prompt with the next LLM span before calling your agent.

from pydantic_ai import Agent
from deepeval.prompt import Prompt
from deepeval.tracing import next_llm_span
from deepeval.integrations.pydantic_ai import DeepEvalInstrumentationSettings

agent = Agent(
    "openai:gpt-4o-mini",
    instrument=DeepEvalInstrumentationSettings(),
)

prompt = Prompt(alias="<prompt-alias>")
prompt.pull(version="00.00.01")

with next_llm_span(prompt=prompt):
    result = agent.run_sync(prompt.interpolate())

Per-call trace context

To set per-call trace attributes (such as a different user_id per request), wrap each agent invocation in with trace(...). This also switches routing to Confident AI's REST transport.

from pydantic_ai import Agent
from deepeval.tracing import trace
from deepeval.integrations.pydantic_ai import DeepEvalInstrumentationSettings

agent = Agent(
    "openai:gpt-4o-mini",
    instrument=DeepEvalInstrumentationSettings(),
)

with trace(user_id="user_42", thread_id="thread_1", name="my-trace"):
    result = agent.run_sync("What are LLMs?")

Sending annotations

Send human annotations on traces or threads on Confident AI. Learn more about sending annotations.

from deepeval.tracing import trace
from deepeval.annotation import send_annotation
from deepeval.integrations.pydantic_ai import DeepEvalInstrumentationSettings

agent = Agent(
    "openai:gpt-4o-mini",
    instrument=DeepEvalInstrumentationSettings(),
)

TRACE_UUID = None
with trace() as current_trace:
    result = agent.run_sync("What are LLMs?")
    TRACE_UUID = current_trace.uuid

send_annotation(
    trace_uuid=TRACE_UUID,
    rating=1,
)

from deepeval.tracing import trace
from deepeval.annotation import send_annotation
from deepeval.integrations.pydantic_ai import DeepEvalInstrumentationSettings

agent = Agent(
    "openai:gpt-4o-mini",
    instrument=DeepEvalInstrumentationSettings(),
)

THREAD_ID = "thread_id_1"
with trace(thread_id=THREAD_ID):
    result = agent.run_sync("What are LLMs?")

send_annotation(
    thread_id=THREAD_ID,
    rating=1,
)

Evals Usage

Online evals

You can run online evals on your Pydantic AI agent. Online evals run evaluations on all incoming traces on Confident AI's servers and are the recommended approach for production agents.

1. Create metric collection

   Create a metric collection on Confident AI with the metrics you want to use to evaluate your agent.

2. Run evals

   You can run online evals at the trace level or the span level. Pass the metric_collection parameter to the appropriate target.

   TraceAgent SpanTool Span

   Pass metric_collection to DeepEvalInstrumentationSettings to evaluate every trace produced by the agent.

   from pydantic_ai import Agent
   from deepeval.integrations.pydantic_ai import DeepEvalInstrumentationSettings
   
   agent = Agent(
       model="openai:gpt-4o-mini",
       system_prompt="Be concise, reply with one sentence.",
       instrument=DeepEvalInstrumentationSettings(
           metric_collection="my_trace_collection",
       ),
   )
   
   result = agent.run_sync("What are LLMs?")

You can view eval results on Confident AI by clicking on the link printed in the console.