Pydantic AI

Overview

Pydantic AI is a Python-native LLM agent framework built on the foundations of Pydantic validation. Confident AI allows you to trace and evaluate Pydantic AI agents using an OpenTelemetry-based integration.

Tracing Quickstart

$
export CONFIDENT_OTEL_URL="https://eu.otel.confident-ai.com"

$
pip install -U deepeval pydantic-ai

$
export CONFIDENT_API_KEY="<your-confident-api-key>"

1
from pydantic_ai import Agent
2
from deepeval.integrations.pydantic_ai import DeepEvalInstrumentationSettings
3
4
agent = Agent(
5
    "openai:gpt-4o-mini",
6
    system_prompt="Be concise, reply with one sentence.",
7
    name="my_agent",
8
    instrument=DeepEvalInstrumentationSettings(),
9
)
10
11
result = agent.run_sync("What are LLMs?")
12
print(result.output)

$
python main.py

Advanced Usage

Logging threads

Threads group related traces together and are useful for chat apps, agents, or any multi-turn interactions. You can learn more about threads here. Pass thread_id to DeepEvalInstrumentationSettings to associate every trace from that agent with a thread.

1
from pydantic_ai import Agent
2
from deepeval.integrations.pydantic_ai import DeepEvalInstrumentationSettings
3
4
agent = Agent(
5
    model="openai:gpt-4o-mini",
6
    system_prompt="Be concise, reply with one sentence.",
7
    instrument=DeepEvalInstrumentationSettings(
8
        thread_id="thread_id_1",
9
    ),
10
)
11
12
result = agent.run_sync("What are LLMs?")

Trace attributes

You can attach trace-level attributes such as name, tags, metadata, and user ID to every trace produced by the agent. These are baked into DeepEvalInstrumentationSettings as static defaults. They can be overridden at runtime using update_current_trace(...) from inside a tool body.

1
from pydantic_ai import Agent
2
from deepeval.integrations.pydantic_ai import DeepEvalInstrumentationSettings
3
4
agent = Agent(
5
    model="openai:gpt-4o-mini",
6
    system_prompt="Be concise, reply with one sentence.",
7
    instrument=DeepEvalInstrumentationSettings(
8
        name="My Agent Trace",
9
        tags=["production", "v2"],
10
        metadata={"env": "production"},
11
        user_id="user_123",
12
        thread_id="thread_id_1",
13
    ),
14
)
15
16
result = agent.run_sync("What are LLMs?")

Update trace attributes

You can enrich a trace mid-flight from inside a tool body using update_current_trace. This is useful when trace metadata depends on information only available during execution, such as a user ID resolved by a lookup tool.

1
from pydantic_ai import Agent
2
from deepeval.tracing import update_current_trace
3
from deepeval.integrations.pydantic_ai import DeepEvalInstrumentationSettings
4
5
agent = Agent(
6
    "openai:gpt-4o-mini",
7
    instrument=DeepEvalInstrumentationSettings(),
8
)
9
10
@agent.tool_plain
11
def lookup_user(user_id: str) -> str:
12
    # Enrich the trace with data resolved at runtime
13
    update_current_trace(
14
        user_id=user_id,
15
        metadata={"plan": "pro", "region": "us-east"},
16
    )
17
    return f"User {user_id} profile loaded."
18
19
result = agent.run_sync("Load my profile for user_42.")

Update span attributes

You can attach span-level attributes such as metadata or a metric collection from inside a tool body using update_current_span. This is the primary way to configure per-tool evaluation behavior.

1
from pydantic_ai import Agent
2
from deepeval.tracing import update_current_span
3
from deepeval.integrations.pydantic_ai import DeepEvalInstrumentationSettings
4
5
agent = Agent(
6
    "openai:gpt-4o-mini",
7
    instrument=DeepEvalInstrumentationSettings(),
8
)
9
10
@agent.tool_plain
11
def get_weather(city: str) -> str:
12
    update_current_span(
13
        metadata={"city": city, "source": "mock"},
14
        metric_collection="weather-tool-evals",
15
    )
16
    return f"{city}: sunny, 22°C"
17
18
result = agent.run_sync("What is the weather in Tokyo?")

Logging prompts

If you are managing prompts on Confident AI and wish to log them, use next_llm_span to associate a Prompt with the next LLM span before calling your agent.

1
from pydantic_ai import Agent
2
from deepeval.prompt import Prompt
3
from deepeval.tracing import next_llm_span
4
from deepeval.integrations.pydantic_ai import DeepEvalInstrumentationSettings
5
6
agent = Agent(
7
    "openai:gpt-4o-mini",
8
    instrument=DeepEvalInstrumentationSettings(),
9
)
10
11
prompt = Prompt(alias="<prompt-alias>")
12
prompt.pull(version="00.00.01")
13
14
with next_llm_span(prompt=prompt):
15
    result = agent.run_sync(prompt.interpolate())

Per-call trace context

To set per-call trace attributes (such as a different user_id per request), wrap each agent invocation in with trace(...). This also switches routing to Confident AI’s REST transport.

1
from pydantic_ai import Agent
2
from deepeval.tracing import trace
3
from deepeval.integrations.pydantic_ai import DeepEvalInstrumentationSettings
4
5
agent = Agent(
6
    "openai:gpt-4o-mini",
7
    instrument=DeepEvalInstrumentationSettings(),
8
)
9
10
with trace(user_id="user_42", thread_id="thread_1", name="my-trace"):
11
    result = agent.run_sync("What are LLMs?")

Sending annotations

Send human annotations on traces or threads on Confident AI. Learn more about sending annotations.

1
from deepeval.tracing import trace
2
from deepeval.annotation import send_annotation
3
from deepeval.integrations.pydantic_ai import DeepEvalInstrumentationSettings
4
5
agent = Agent(
6
    "openai:gpt-4o-mini",
7
    instrument=DeepEvalInstrumentationSettings(),
8
)
9
10
TRACE_UUID = None
11
with trace() as current_trace:
12
    result = agent.run_sync("What are LLMs?")
13
    TRACE_UUID = current_trace.uuid
14
15
send_annotation(
16
    trace_uuid=TRACE_UUID,
17
    rating=1,
18
)

Evals Usage

Online evals

You can run online evals on your Pydantic AI agent. Online evals run evaluations on all incoming traces on Confident AI’s servers and are the recommended approach for production agents.

1
from pydantic_ai import Agent
2
from deepeval.integrations.pydantic_ai import DeepEvalInstrumentationSettings
3
4
agent = Agent(
5
    model="openai:gpt-4o-mini",
6
    system_prompt="Be concise, reply with one sentence.",
7
    instrument=DeepEvalInstrumentationSettings(
8
        metric_collection="my_trace_collection",
9
    ),
10
)
11
12
result = agent.run_sync("What are LLMs?")

You can view eval results on Confident AI by clicking on the link printed in the console.