Launch Week 02 wrapped — explore all five launches
Confident AI
Docs
Pricing
Products
LLM Evaluation
LLM Evaluation

Benchmark LLM systems with research-backed metrics.

LLM Observability
LLM Observability

Trace, monitor, and alert on production LLM systems.

AI Red Teaming
AI Red Teaming

Stress-test LLM apps against adversarial attacks.

AI Governance
AI Governance

Enforce AI standards and controls across teams.

Book a DemoSign Up
Book a DemoSign Up

DeepEval Tracing Skill

Teach your agent to instrument AI apps with DeepEval's native tracing — @observe, span types, and framework integrations.

Overview

The deepeval-tracing Agent Skill teaches your coding agent how to instrument an AI application with DeepEval's native tracing, so every LLM call, retrieval, tool call, and agent step shows up span by span in Confident AI's Observatory. It ships in the confident-ai/deepeval repository alongside the deepeval and deepeval-otel skills.

Its scope is deliberately narrow: producing well-formed traces. The skill picks a supported framework integration when one exists, falls back to manual @observe when it doesn't, gives each span a meaningful type, and adds tags and metadata — then stops. Attaching metrics and running evals belongs to the deepeval skill.

When It Triggers

The skill activates on prompts like:

Prompts that trigger the skill
Add DeepEval tracing to my LangGraph agent.
Instrument this RAG pipeline with @observe.
Send my chatbot's traces to Confident AI.
Should I use the LlamaIndex integration or manual tracing here?

Installation

Works with Cursor, Claude Code, Codex, Windsurf, OpenCode, and any other Skills-compatible assistant:

npx skills add confident-ai/deepeval --skill "deepeval-tracing"

Prerequisites

  • A Python AI application with pip install deepeval
  • deepeval login or an exported CONFIDENT_API_KEY

What Changes in Your Codebase

  1. Ask for tracing

    The agent detects the framework, model provider, agent SDK, and vector database in use, then wires the matching integration — LangGraph, LangChain, OpenAI Agents, LlamaIndex, Pydantic AI, CrewAI, and others. A supported integration always beats manual instrumentation.

    Prompt
    Add DeepEval tracing to my agent so I can see every step in Confident AI.

  2. Manual @observe where no integration fits

    For unsupported frameworks and app-owned wrapper boundaries, the agent falls back to @observe, assigning each span one of the four types — llm, retriever, tool, agent — and capturing inputs and outputs.

    See what the agent writes
    from deepeval.tracing import observe, update_current_span

@observe(type="retriever")
def retrieve_context(query: str):
    documents = retriever.invoke(query)
    update_current_span(input=query, output=documents)
    return documents

@observe(type="llm")
def call_model(prompt: str) -> str:
    output = llm.invoke(prompt)
    update_current_span(input=prompt, output=output)
    return output

  3. Enrich with tags and metadata

    Ask for tags and metadata where they'd help diagnose failure patterns — the agent adds them at the trace level, never tracing secrets or raw sensitive data:

    Prompt
    Tag every trace with the app version and whether the user is on the enterprise tier.
    See what the agent writes
    from deepeval.tracing import observe, update_current_trace

@observe(type="agent")
def run_my_ai_app(user_input: str):
    update_current_trace(
        tags=["rag", "support-chat"],
        metadata={"user_tier": "enterprise", "app_version": "1.2.3"},
    )
    return my_ai_app(user_input)

  4. Verify traces in the Observatory

    With the key set, run your app and confirm traces appear span by span in the Observatory:

    export CONFIDENT_API_KEY="confident_us_proj_..."
python main.py

FAQs

Which frameworks does it support out of the box?

The skill carries an integration index covering frameworks, model providers, agent SDKs, and vector databases — LangGraph, LangChain, OpenAI Agents, LlamaIndex, Pydantic AI, CrewAI, and more. It always prefers a native integration and only falls back to manual @observe when nothing fits.

Will it trace my whole backend?

No. It instruments AI components only — agent loops, LLM calls, retrieval, and tool calls — and refuses to trace web servers, CRUD backends, or infrastructure. It also never puts secrets, credentials, or raw sensitive user data on a trace.

My app isn't Python — can I still trace it?

Not with this skill: it uses the DeepEval Python SDK. For TypeScript, Go, or any other language, use the deepeval-otel skill, which exports raw OpenTelemetry to the same Observatory.

Does it attach metrics or run evals on the traces?

No — its job ends at producing well-formed traces. Attaching span-level metrics and running evals is the deepeval skill's job, which builds traced evals on top of the instrumentation this skill adds.

Next Steps

LLM Tracing Quickstart

See the underlying tracing setup the skill automates.

DeepEval Evals Skill

Build a pytest eval suite on top of your instrumented app.

Scaling beyond prototype?For teams evaluating Confident AI in productionTalk to us

Last updated on

Built byConfident AI