LangChain

Overview

LangChain is a framework for building LLM applications. Confident AI provides a callback handlers in both python and typescript SDKs to trace and evaluate your LangChain applications automatically.

The callback handler captures the following spans from your LangChain application:

• LLM spans — model name, provider, input messages, output content, tool calls made by the model, and token usage
• Tool spans — tool name, input parameters, and output; also aggregated at the trace level as tools_called
• Retriever spans — query input and retrieved document output
• Chain spans — inputs and outputs for top-level chains (used to set trace-level input/output)

Tracing Quickstart

1. Install Dependencies

   Run the following command to install the required packages:

   pip install -U deepeval langchain langchain-openai

   npm install deepeval langchain @langchain/core

2. Setup Confident AI Key

   Login to Confident AI using your Confident API key.

   export CONFIDENT_API_KEY="<your-confident-api-key>"

   deepeval login

   import deepeval
   
   deepeval.login("<your-confident-api-key>")

3. Configure LangChain

   Provide DeepEval's CallbackHandler to your LangChain application's invoke method.

   from langchain_core.tools import tool
   from langchain_openai import ChatOpenAI
   from langchain_core.prompts import ChatPromptTemplate
   from langchain.agents import create_tool_calling_agent, AgentExecutor
   from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
   
   from deepeval.integrations.langchain import CallbackHandler
   
   @tool
   def multiply(a: int, b: int) -> int:
       """Returns the product of two numbers"""
       return a * b
   
   llm = ChatOpenAI(model="gpt-4o-mini")
   
   agent_prompt = ChatPromptTemplate.from_messages(
       [
           ("system", "You are a helpful assistant that can perform mathematical operations."),
           ("human", "{input}"),
           MessagesPlaceholder("agent_scratchpad"),
       ]
   )
   
   agent = create_tool_calling_agent(llm, [multiply], agent_prompt)
   agent_executor = AgentExecutor(agent=agent, tools=[multiply], verbose=True)
   
   result = agent_executor.invoke(
       {"input": "What is 8 multiplied by 6?"},
       config={"callbacks": [CallbackHandler()]},
   )

   import { ChatOpenAI } from "@langchain/openai";
   import {
     ChatPromptTemplate,
     MessagesPlaceholder,
   } from "@langchain/core/prompts";
   import { DynamicStructuredTool } from "@langchain/core/tools";
   import { createToolCallingAgent, AgentExecutor } from "langchain/agents";
   
   import { DeepEvalCallbackHandler } from "deepeval/integrations/langchain";
   
   const handler = new DeepEvalCallbackHandler({});
   
   const multiplyTool = new DynamicStructuredTool({
     name: "multiply",
     description: "Returns the product of two numbers",
     schema: {
       a: { type: "number", description: "The first number" },
       b: { type: "number", description: "The second number" },
     },
     func: async ({ a, b }: { a: number; b: number }) => a * b,
   });
   
   const llm = new ChatOpenAI({
     model: "gpt-4o-mini",
     temperature: 0,
   });
   
   const agentPrompt = ChatPromptTemplate.fromMessages([
     [
       "system",
       "You are a helpful assistant that can perform mathematical operations.",
     ],
     ["human", "{input}"],
     new MessagesPlaceholder("agent_scratchpad"),
   ]);
   
   const agent = await createToolCallingAgent({
     llm,
     tools: [multiplyTool],
     prompt: agentPrompt,
   });
   
   const agentExecutor = new AgentExecutor({
     agent,
     tools: [multiplyTool],
     verbose: true,
   });
   
   const main = async () => {
     const result = await agentExecutor.invoke(
       { input: "What is 8 multiplied by 6?" },
       { callbacks: [handler] }
     );
   };

4. Run LangChain

   Invoke your application by executing the script:

   python main.py

   npx ts-node main.ts

   You can directly view the traces on Confident AI by clicking on the link in the output printed in the console.

Advanced Features

Set trace attributes

Confident AI's LLM tracing advanced features provide teams with the ability to set certain attributes for each trace when invoking your LangChain application.

For example, thread_id and user_id are used to group related traces together, and are useful for chat apps, agents, or any multi-turn interactions. You can learn more about threads here.

You can set these attributes in the CallbackHandler when invoking your LangChain application.

result = agent_executor.invoke(
    {"input": "What is 8 multiplied by 6?"},
    config={
        "callbacks": [CallbackHandler(thread_id="123")]
    },
)

const handler = new DeepEvalCallbackHandler({
  threadId: "123",
});

Logging prompts

If you are managing prompts on Confident AI and wish to log them, pass your Prompt object to the language model instance's metadata parameter.

from langchain_openai import ChatOpenAI
from deepeval.prompt import Prompt

prompt = Prompt(alias="<prompt-alias>")
prompt.pull(version="00.00.01")

llm = ChatOpenAI(
    model="gpt-4o-mini",
    metadata={"prompt": prompt}
)

import { ChatOpenAI } from "@langchain/openai";
import { Prompt } from "deepeval";

const prompt = new Prompt({ alias: "<prompt-alias>" });
prompt.pull({ version: "00.00.01" });

const llm = new ChatOpenAI({
  model: "gpt-4o-mini",
  metadata: {
    prompt: prompt,
  },
});

Evals Usage

Online evals

If your LangChain application is in production, and you still want to run evaluations on your traces, use online evals. It lets you run evaluations on all incoming traces on Confident AI's server.

1. Create metric collection

   Create a metric collection on Confident AI with the metrics you wish to use to evaluate your LangChain application. Copy the name of the metric collection.

2. Run evals

   Set the metric_collection name to evaluate various components of your LangChain application.

   Agent SpanLLM SpanTool Span

   This is the top level component of your LangChain application. Also a very ideal component to evaluate with the Task Completion metric.

   from langchain_core.tools import tool
   from langchain_openai import ChatOpenAI
   from langchain_core.prompts import ChatPromptTemplate
   from langchain.agents import create_tool_calling_agent, AgentExecutor
   from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
   
   from deepeval.integrations.langchain import CallbackHandler
   
   @tool
   def multiply(a: int, b: int) -> int:
       """Returns the product of two numbers"""
       return a * b
   
   llm = ChatOpenAI(model="gpt-4o-mini")
   agent_prompt = ChatPromptTemplate.from_messages([("system", "You are a helpful assistant that can perform mathematical operations."), ("human", "{input}"), MessagesPlaceholder("agent_scratchpad")])
   agent = create_tool_calling_agent(llm, [multiply], agent_prompt)
   agent_executor = AgentExecutor(agent=agent, tools=[multiply], verbose=True)
   
   result = agent_executor.invoke(
       {"input": "What is 8 multiplied by 6?"},
       config={
           "callbacks": [
             CallbackHandler(metric_collection="<metric_collection_name>")
           ]
       },
   )

   import { ChatOpenAI } from "@langchain/openai";
   import {
     ChatPromptTemplate,
     MessagesPlaceholder,
   } from "@langchain/core/prompts";
   import { DynamicStructuredTool } from "@langchain/core/tools";
   import { createToolCallingAgent, AgentExecutor } from "langchain/agents";
   
   import { DeepEvalCallbackHandler } from "deepeval/integrations/langchain";
   
   const handler = new DeepEvalCallbackHandler({
     metricCollection: "<metric_collection_name>",
   });
   
   const multiplyTool = new DynamicStructuredTool({
     name: "multiply",
     description: "Returns the product of two numbers",
     schema: {
       a: { type: "number", description: "The first number" },
       b: { type: "number", description: "The second number" },
     },
     func: async ({ a, b }: { a: number; b: number }) => a * b,
   });
   
   const llm = new ChatOpenAI({
     model: "gpt-4o-mini",
   });
   
   const agentPrompt = ChatPromptTemplate.fromMessages([
     [
       "system",
       "You are a helpful assistant that can perform mathematical operations.",
     ],
     ["human", "{input}"],
     new MessagesPlaceholder("agent_scratchpad"),
   ]);
   
   const agent = await createToolCallingAgent({
     llm,
     tools: [multiplyTool],
     prompt: agentPrompt,
   });
   
   const agentExecutor = new AgentExecutor({
     agent,
     tools: [multiplyTool],
     verbose: true,
   });
   
   const main = async () => {
     const result = await agentExecutor.invoke(
       { input: "What is 8 multiplied by 6?" },
       { callbacks: [handler] }
     );
   
     console.log(result);
   };

View on Confident AI

You can view the evals on Confident AI by clicking on the link in the output printed in the console.