OpenAI Agents

Overview

OpenAI Agents is a lightweight framework for creating agentic workflows using agent swarms, handoffs, and tool use. Confident AI lets you trace OpenAI Agent workflows with one line of code.

Tracing Quickstart

1. Install Dependencies

   Run the following command to install the required packages:

   pip install -U deepeval openai-agents

   npm install deepeval @openai/agents

2. Setup Confident AI Key

   Login to Confident AI using your Confident API key.

   export CONFIDENT_API_KEY="<your-confident-api-key>"

   deepeval login

   import deepeval
   
   deepeval.login("<your-confident-api-key>")
   

3. Configure OpenAI Agents

   Add DeepEval's trace processor to OpenAI Agents.

   from deepeval.openai_agents import DeepEvalTracingProcessor
   from agents import add_trace_processor, Agent, Runner
   
   add_trace_processor(DeepEvalTracingProcessor())
   
   agent = Agent(name="Assistant", instructions="You are a helpful assistant")
   result = Runner.run_sync(agent, "Write a haiku about recursion in programming.")

   import { DeepEvalTracingProcessor } from "deepeval/integrations/openai-agents";
   import { Agent, run, addTraceProcessor } from "@openai/agents";
   
   const processor = new DeepEvalTracingProcessor();
   addTraceProcessor(processor);
   
   const agent = new Agent({ name: "Assistant", instructions: "You are a helpful assistant" });
   const result = await run(agent, "Write a haiku about recursion in programming.");

4. Run OpenAI Agents

   Invoke your agent by executing the script:

   python main.py

   npx ts-node main.ts

   You can directly view the traces on Confident AI by clicking on the link in the output printed in the console.

What Gets Traced

The DeepEvalTracingProcessor automatically captures the following span types from OpenAI Agents:

The LLM span provider is inferred automatically from the model name and normalized to the Confident AI platform format. For response span types, the following invocation parameters are also captured when present: temperature, top_p, max_output_tokens, tool_choice, tools, parallel_tool_calls, reasoning, text, and truncation.

Advanced Usage

Logging threads

Threads are used to group related traces together, and are useful for chat apps, agents, or any multi-turn interactions. You can learn more about threads here.

from deepeval.openai_agents import DeepEvalTracingProcessor
from agents import add_trace_processor, Agent, Runner, trace

add_trace_processor(DeepEvalTracingProcessor())
agent = Agent(name="Assistant", instructions="You are a helpful assistant")

with trace(workflow_name="test_workflow_1", group_id="test_group_id_1"):
      result = Runner.run_sync(agent, "Write a haiku about recursion in programming.")

import { DeepEvalTracingProcessor } from "deepeval/integrations/openai-agents";
import { Agent, run, addTraceProcessor } from "@openai/agents";
import { setTracingContext } from "deepeval/tracing";

const processor = new DeepEvalTracingProcessor();
addTraceProcessor(processor);

const agent = new Agent({ name: "Assistant", instructions: "You are a helpful assistant" });

const main = async () => {
  await setTracingContext(
    {
      threadId: "test_thread_id_1",
      userId: "new_user_id",
    },
    async () => {
      const result = await run(agent, "Write a haiku about recursion in programming.");
    }
  );
};

main();

Logging metadata

You can attach arbitrary metadata to a trace using setTracingContext.

from deepeval.openai_agents import DeepEvalTracingProcessor
from agents import add_trace_processor, Agent, Runner, trace

add_trace_processor(DeepEvalTracingProcessor())
agent = Agent(name="Assistant", instructions="You are a helpful assistant")

with trace(workflow_name="test_workflow_1", metadata={"test_metadata_1": "test_metadata_1"}):
      result = Runner.run_sync(agent, "Write a haiku about recursion in programming.")

import { DeepEvalTracingProcessor } from "deepeval/integrations/openai-agents";
import { Agent, run, addTraceProcessor } from "@openai/agents";
import { setTracingContext } from "deepeval/tracing";

const processor = new DeepEvalTracingProcessor();
addTraceProcessor(processor);

const agent = new Agent({ name: "Assistant", instructions: "You are a helpful assistant" });

const main = async () => {
  await setTracingContext(
    {
      metadata: { test_metadata_1: "test_metadata_1" },
    },
    async () => {
      const result = await run(agent, "Write a haiku about recursion in programming.");
    }
  );
};

main();

Streaming responses

Confident AI handles both asynchronous workflows and streamed responses. The following example shows how to trace streamed responses with OpenAI Agents.

from deepeval.openai_agents import DeepEvalTracingProcessor
from agents import add_trace_processor, Agent, Runner
import asyncio

add_trace_processor(DeepEvalTracingProcessor())

weather_agent = Agent(
    name="Weather Agent",
    instructions="You are a weather agent. You are given a question about the weather and you need to answer it.",
)

async def main():
    result = Runner.run_streamed(weather_agent, "What's the weather in UK?")
    async for chunk in result.stream_events():
        print(chunk, end="", flush=True)

asyncio.run(main())

import { DeepEvalTracingProcessor } from "deepeval/integrations/openai-agents";
import { Agent, run, addTraceProcessor } from "@openai/agents";
import { setTracingContext } from "deepeval/tracing";

const processor = new DeepEvalTracingProcessor();
addTraceProcessor(processor);

const weatherAgent = new Agent({
  name: "Weather Agent",
  instructions: "You are a weather agent. You are given a question about the weather and you need to answer it.",
});

const main = async () => {
  await setTracingContext(
    {
      threadId: "thread_1",
    },
    async () => {
      const result = await run(weatherAgent, "What's the weather in UK?", { stream: true });

      for await (const chunk of result) {
        console.log(chunk);
      }
    }
  );
};

main();

Overwrite trace attributes

By default, the trace input is taken from the first agent span's input and the output from the last agent span's output. If you want to override the input, output, or any other trace attribute, use update_current_trace (Python) or updateCurrentTrace (TypeScript):

from deepeval.openai_agents import DeepEvalTracingProcessor
from agents import add_trace_processor, Agent, Runner, trace
from deepeval.tracing.context import update_current_trace

add_trace_processor(DeepEvalTracingProcessor())
agent = Agent(name="Assistant", instructions="You are a helpful assistant")

with trace(workflow_name="test_workflow_1", metadata={"test_metadata_1": "test_metadata_1"}):
    response_1 = Runner.run_sync(agent, "Hola, ¿cómo estás?")
    update_current_trace(
        name="New name",
        input="New input",
        output="New output",
        metadata={"New key": "New value"}
    )

import { DeepEvalTracingProcessor } from "deepeval/integrations/openai-agents";
import { Agent, run, addTraceProcessor } from "@openai/agents";
import { setTracingContext, updateCurrentTrace } from "deepeval/tracing";

const processor = new DeepEvalTracingProcessor();
addTraceProcessor(processor);

const agent = new Agent({ name: "Assistant", instructions: "You are a helpful assistant" });

const main = async () => {
  await setTracingContext(
    {
      metadata: { test_metadata_1: "test_metadata_1" },
    },
    async () => {
      const result = await run(agent, "Hola, ¿cómo estás?");
      updateCurrentTrace({
        name: "New name",
        input: "New input",
        output: "New output",
        metadata: { newKey: "New value" },
      });
    }
  );
};

main();

Logging prompts

If you are managing prompts on Confident AI and wish to log them, pass your Prompt object via the llmSpanContext in the setTracingContext call (TypeScript) or via the confident_prompt parameter on DeepEval's Agent wrapper (Python).

from agents import Runner, add_trace_processor
from deepeval.prompt import Prompt
from deepeval.openai_agents import DeepEvalTracingProcessor, Agent

add_trace_processor(DeepEvalTracingProcessor())

prompt = Prompt(alias="<prompt-alias>")
prompt.pull(version="00.00.01")

spanish_agent = Agent(
    name="Spanish agent",
    instructions=prompt.interpolate(),
    confident_prompt=prompt,
)
Runner.run_sync(spanish_agent, "¿Cómo estás?")

import { DeepEvalTracingProcessor } from "deepeval/integrations/openai-agents";
import { Agent, run, addTraceProcessor } from "@openai/agents";
import { setTracingContext } from "deepeval/tracing";
import { Prompt } from "deepeval";

const processor = new DeepEvalTracingProcessor();
addTraceProcessor(processor);

const prompt = new Prompt({ alias: "my-prompt" });
await prompt.pull({ version: "00.00.01" });

const agent = new Agent({
  name: "Spanish agent",
  instructions: prompt.interpolate({ name: "John" }),
});

const main = async () => {
  await setTracingContext(
    {
      llmSpanContext: {
        prompt: prompt,
      },
    },
    async () => {
      const result = await run(agent, prompt.interpolate({ name: "John" }));
    }
  );
};

main();

Evals Usage

Online evals

You can run online evals on your OpenAI Agent, which will run evaluations on all incoming traces on Confident AI's servers. This is the recommended approach, especially if your agent is in production.

1. Create metric collection

   Create a metric collection on Confident AI with the metrics you wish to use to evaluate your OpenAI Agent.

   Click to see supported metrics for OpenAI Agents

   Confident AI supports evaluating the input-output pairs of OpenAI Agent spans and traces, which means your metric collections must only contain metrics that only require the input and output for evaluation. These metrics include:

   • Answer Relevancy
   • Task Completion
   • GEval
   • Bias
   • Toxicity

2. Run evals

   You can run evals at both the trace and span level. We recommend creating separate metric collections for each component, since each requires its own evaluation criteria and metrics.

   Agent SpanLLM SpanTool Span

   Replace your Agent with DeepEval's and supply the metric collection name to run evals on the agent span level.

   import asyncio
   from agents import Runner, add_trace_processor
   from deepeval.openai_agents import Agent, DeepEvalTracingProcessor
   
   add_trace_processor(DeepEvalTracingProcessor())
   
   weather_agent = Agent(
       name="Weather Agent",
       instructions="You are a weather agent. You are given a question about the weather and you need to answer it.",
       agent_metric_collection="test_collection_1",
   )
   
   async def main():
       result = await Runner.run(weather_agent, "What's the weather in UK?")
       print(result.final_output)
   
   asyncio.run(main())

   import { DeepEvalTracingProcessor } from "deepeval/integrations/openai-agents";
   import { Agent, run, addTraceProcessor } from "@openai/agents";
   import { setTracingContext } from "deepeval/tracing";
   
   const processor = new DeepEvalTracingProcessor();
   addTraceProcessor(processor);
   
   const weatherAgent = new Agent({
     name: "Weather Agent",
     instructions: "You are a weather agent. You are given a question about the weather and you need to answer it.",
   });
   
   const main = async () => {
     await setTracingContext(
       {
         agentSpanContext: {
           metricCollection: "agent-metric-collection",
         },
       },
       async () => {
         const result = await run(weatherAgent, "What's the weather in UK?");
       }
     );
   };
   
   main();