LangGraph

Overview

LangGraph is a framework for building reactive, multi-agent systems. Confident AI provides a CallbackHandler in both the python and typescript SDKs to trace and evaluate your LangGraph agents automatically.

The callback handler captures the following spans from your LangGraph agent:

• LLM spans — model name, provider, input messages, output content, tool calls made by the model, and token usage
• Tool spans — tool name, input parameters, and output; also aggregated at the trace level as tools_called
• Retriever spans — query input and retrieved document output
• Agent spans — inputs and outputs for the top-level graph (used to set trace-level input/output)

Tracing Quickstart

1. Install Dependencies

   Run the following command to install the required packages:

   pip install -U deepeval langgraph langchain langchain-openai

   npm install deepeval langchain @langchain/langgraph @langchain/openai @langchain/core

2. Setup Confident AI Key

   Login to Confident AI using your Confident API key.

   export CONFIDENT_API_KEY="<your-confident-api-key>"

   deepeval login

   import deepeval
   
   deepeval.login("<your-confident-api-key>")

3. Configure LangGraph

   Provide DeepEval's CallbackHandler to your LangGraph agent's invoke method.

   from langchain.agents import create_agent
   from langchain_openai import ChatOpenAI
   
   from deepeval.integrations.langchain import CallbackHandler
   
   def get_weather(city: str) -> str:
       """Returns the weather in a city"""
       return f"It's always sunny in {city}!"
   
   llm = ChatOpenAI(model="gpt-4o-mini")
   
   agent = create_agent(
       model=llm,
       tools=[get_weather],
       system_prompt="You are a helpful assistant",
   )
   
   result = agent.invoke(
       input={"messages": [{"role": "user", "content": "what is the weather in sf"}]},
       config={"callbacks": [CallbackHandler()]},
   )

   import { createAgent } from "langchain";
   import { ChatOpenAI } from "@langchain/openai";
   import { tool } from "@langchain/core/tools";
   import { z } from "zod";
   
   import { DeepEvalCallbackHandler } from "deepeval/integrations/langchain";
   
   const getWeather = tool(
     async ({ city }: { city: string }) => `It's always sunny in ${city}!`,
     {
       name: "get_weather",
       description: "Returns the weather in a city",
       schema: z.object({ city: z.string() }),
     },
   );
   
   const agent = createAgent({
     model: new ChatOpenAI({ model: "gpt-4o-mini" }),
     tools: [getWeather],
     systemPrompt: "You are a helpful assistant",
   });
   
   const result = await agent.invoke(
     { messages: [{ role: "user", content: "what is the weather in sf" }] },
     { callbacks: [new DeepEvalCallbackHandler({})] },
   );

4. Run LangGraph

   Invoke your agent by executing the script:

   python main.py

   npx ts-node main.ts

   You can directly view the traces on Confident AI by clicking on the link in the output printed in the console.

Trace a LangGraph server deployment

Confident AI allows you to trace your LangGraph applications deployed in a server by passing the CallbackHandler in the exported graph's config and using it in langgraph.json files.

1. Attach DeepEval to your graph

   Attach the CallbackHandler to the graph with with_config and export it. The handler is then applied to every run the server executes.

   from langchain.agents import create_agent
   from langchain_openai import ChatOpenAI
   
   from deepeval.integrations.langchain import CallbackHandler
   
   def get_weather(city: str) -> str:
       """Returns the weather in a city"""
       return f"It's always sunny in {city}!"
   
   llm = ChatOpenAI(model="gpt-4o-mini")
   
   graph = create_agent(
       model=llm,
       tools=[get_weather],
       system_prompt="You are a helpful assistant",
   ).with_config(callbacks=[CallbackHandler()])

   import { createAgent } from "langchain";
   import { ChatOpenAI } from "@langchain/openai";
   import { tool } from "@langchain/core/tools";
   import { z } from "zod";
   
   import { DeepEvalCallbackHandler } from "deepeval/integrations/langchain";
   
   const getWeather = tool(
     async ({ city }: { city: string }) => `It's always sunny in ${city}!`,
     {
       name: "get_weather",
       description: "Returns the weather in a city",
       schema: z.object({ city: z.string() }),
     },
   );
   
   export const graph = createAgent({
     model: new ChatOpenAI({ model: "gpt-4o-mini" }),
     tools: [getWeather],
     systemPrompt: "You are a helpful assistant",
   }).withConfig({ callbacks: [new DeepEvalCallbackHandler({})] });

2. Register the graph in langgraph.json

   Point the graphs entry at the exported graph variable.

   {
     "dependencies": ["."],
     "graphs": { "agent": "./agent.py:graph" },
     "env": ".env"
   }

   {
     "node_version": "20",
     "dependencies": ["."],
     "graphs": { "agent": "./agent.ts:graph" },
     "env": ".env"
   }

3. Start the LangGraph server

   Run the server. Every request it runs against the graph is traced to Confident AI.

   pip install -U "langgraph-cli[inmem]"
   langgraph dev

   npx @langchain/langgraph-cli dev

Advanced Features

Set trace attributes

Confident AI's LLM tracing advanced features provide teams with the ability to set certain attributes for each trace when invoking your LangGraph agent.

For example, thread_id and user_id are used to group related traces together, and are useful for chat apps, agents, or any multi-turn interactions. You can learn more about threads here.

You can set these attributes in the CallbackHandler.

result = agent.invoke(
    input={"messages": [{"role": "user", "content": "what is the weather in sf"}]},
    config={"callbacks": [CallbackHandler(thread_id="123")]},
)

const handler = new DeepEvalCallbackHandler({
  threadId: "123",
});

Logging prompts

If you are managing prompts on Confident AI and wish to log them, pass your Prompt object to the language model instance's metadata parameter.

from langchain_openai import ChatOpenAI
from deepeval.prompt import Prompt

prompt = Prompt(alias="<prompt-alias>")
prompt.pull(version="00.00.01")

llm = ChatOpenAI(
    model="gpt-4o-mini",
    metadata={"prompt": prompt}
)

import { ChatOpenAI } from "@langchain/openai";
import { Prompt } from "deepeval";

const prompt = new Prompt({ alias: "<prompt-alias>" });
prompt.pull({ version: "00.00.01" });

const llm = new ChatOpenAI({
  model: "gpt-4o-mini",
  metadata: {
    prompt: prompt,
  },
});

Evals Usage

Online evals

If your LangGraph agent is in production, and you still want to run evaluations on your traces, use online evals. It lets you run evaluations on all incoming traces on Confident AI's server.

1. Create metric collection

   Create a metric collection on Confident AI with the metrics you wish to use to evaluate your LangGraph agent. Copy the name of the metric collection.

2. Run evals

   Set the metric_collection name to evaluate various components of your LangGraph agent.

   Agent SpanLLM SpanTool Span

   This is the top level component of your LangGraph agent. Also a very ideal component to evaluate with the Task Completion metric.

   from langchain.agents import create_agent
   from langchain_openai import ChatOpenAI
   
   from deepeval.integrations.langchain import CallbackHandler
   
   def get_weather(city: str) -> str:
       """Returns the weather in a city"""
       return f"It's always sunny in {city}!"
   
   agent = create_agent(
       model=ChatOpenAI(model="gpt-4o-mini"),
       tools=[get_weather],
       system_prompt="You are a helpful assistant",
   )
   
   result = agent.invoke(
       input={"messages": [{"role": "user", "content": "what is the weather in sf"}]},
       config={
           "callbacks": [CallbackHandler(metric_collection="<metric_collection_name>")]
       },
   )

   import { createAgent } from "langchain";
   import { ChatOpenAI } from "@langchain/openai";
   import { tool } from "@langchain/core/tools";
   import { z } from "zod";
   
   import { DeepEvalCallbackHandler } from "deepeval/integrations/langchain";
   
   const getWeather = tool(
     async ({ city }: { city: string }) => `It's always sunny in ${city}!`,
     {
       name: "get_weather",
       description: "Returns the weather in a city",
       schema: z.object({ city: z.string() }),
     },
   );
   
   const agent = createAgent({
     model: new ChatOpenAI({ model: "gpt-4o-mini" }),
     tools: [getWeather],
     systemPrompt: "You are a helpful assistant",
   });
   
   const result = await agent.invoke(
     { messages: [{ role: "user", content: "what is the weather in sf" }] },
     {
       callbacks: [
         new DeepEvalCallbackHandler({ metricCollection: "<metric_collection_name>" }),
       ],
     },
   );

View on Confident AI

You can view the evals on Confident AI by clicking on the link in the output printed in the console.