Vercel AI SDK

The AI SDK by Vercel is a powerful TypeScript framework that allows you to use various LLM providers and models for AI-based applications. Confident AI allows you to trace and evaluate AI SDK based LLM applications in just a few lines of code.

Choose Your Integration Mode

DeepEval supports two ways to integrate tracing with the Vercel AI SDK.

import { generateText } from "ai";
import { configureAiSdkTracing } from "deepeval";

const tracer = configureAiSdkTracing();

await generateText({
  model: "openai/gpt-4o",
  prompt: "Explain how neural networks work",
  experimental_telemetry: {
    isEnabled: true,
    tracer: tracer,
  },
});

import { registerOTel } from "@vercel/otel";
import { createDeepEvalProcessors } from "deepeval/integrations/ai-sdk"; 

const deepevalProcessors = createDeepEvalProcessors();

export function register() {
  registerOTel({
    serviceName: "deepeval-next-sandbox",
    spanProcessors: [
      // Your existing processors
      ...deepevalProcessors,
    ],
  });
}

import { NodeTracerProvider } from '@opentelemetry/sdk-trace-node';
import { createDeepEvalProcessors } from "deepeval/integrations/ai-sdk"; 

const deepevalProcessors = createDeepEvalProcessors();

const tracerProvider = new NodeTracerProvider({
  spanProcessors: [
    // Existing processors
    ...deepevalProcessors
  ],
});

tracerProvider.register();

Tracing Quickstart

1. Install Dependencies

   Run the following command to install the required packages:

   npm install ai deepeval

2. Configure AI SDK

   Use DeepEval's configureAiSdkTracing to trace LLM operations.

   Standard SetupExisting OTel

   Generate TextStream TextTool CallingGenerate Structured DataEmbedding Text

   import { generateText } from "ai";
   import { configureAiSdkTracing } from "deepeval";
   
   const tracer = configureAiSdkTracing();
   
   const { text } = await generateText({
       model: "openai/gpt-4o",
       prompt: "How to make the best coffee?",
       experimental_telemetry: {
           isEnabled: true,
           tracer: tracer,
       },
   });
   
   console.log(text);

3. Run AI SDK Generation

   Run your LLM application. You can directly view the traces on Confident AI's traces page inside the observatory.

Advanced Usage

Configuration options

configureAiSdkTracing accepts an optional options object to control tracing behavior:

import { configureAiSdkTracing } from "deepeval";

const tracer = configureAiSdkTracing({
  apiKey: "your-confident-api-key",     // defaults to CONFIDENT_API_KEY env var
  environment: "production",             // defaults to "development"
  name: "my-ai-app",                    // optional default trace name
  traceMetricCollection: "my-metrics",  // optional default metric collection for all traces
  debug: false,                          // set true to enable verbose logging
});

Logging prompts

If you are managing prompts on Confident AI and wish to log them, pass your Prompt object to the llmSpanContext using the setTracingContext function:

import { generateText } from "ai";
import { configureAiSdkTracing, Prompt } from "deepeval";
import { setTracingContext } from "deepeval/tracing";

const prompt = new Prompt({ alias: "PROMPT_ALIAS" });
prompt.pull();

const tracer = configureAiSdkTracing();

await setTracingContext(
  {
    llmSpanContext: {
      prompt: prompt
    }
  }, 
  async () => {
    const { text } = await generateText({
      model: "openai/gpt-4o",
      prompt: "How to make the best coffee?",
      experimental_telemetry: {
        isEnabled: true,
        tracer: tracer,
      },
    });
    console.log(text);
  }
);

import { generateText } from "ai";
import { Prompt } from "deepeval";

const prompt = new Prompt({ alias: "PROMPT_ALIAS" });
prompt.pull();

const { text } = await generateText({
  model: "openai/gpt-4o",
  prompt: "How to make the best coffee?",
  experimental_telemetry: {
    isEnabled: true,
    metadata: {
      promptAlias: prompt.alias,
      promptCommitHash: prompt.hash
    }
  },
});
console.log(text);

Setting trace attributes

Confident AI's LLM tracing advanced features provide teams with the ability to set certain attributes for each trace when invoking your AI SDK applications.

For example, threadId and userId are used to group related traces together, and are useful for chat apps, agents, or any multi-turn interactions. You can learn more about threads here.

You can set these attributes using the setTracingContext function from deepeval/tracing:

import { generateText } from "ai";
import { configureAiSdkTracing } from "deepeval";
import { setTracingContext } from "deepeval/tracing";

const tracer = configureAiSdkTracing();

await setTracingContext(
  {
    threadId: "thread-123",
    userId: "user-456",
    testCaseId: "tc-789",   // optional: link trace to a test case
    turnId: "turn-1",        // optional: identify a specific turn in a conversation
  }, 
  async () => {
    const { text } = await generateText({
      model: "openai/gpt-4o",
      prompt: "How to make the best coffee?",
      experimental_telemetry: {
        isEnabled: true,
        tracer: tracer,
      },
    });
    console.log(text);
  }
);

import { generateText } from "ai";

const { text } = await generateText({
  model: "openai/gpt-4o",
  prompt: "How to make the best coffee?",
  experimental_telemetry: {
    isEnabled: true,
    metadata: {
      threadId: "thread-123",
      userId: "user-456",
      traceName: "Trace name here",
      testCaseId: "tc-789",   // optional: link trace to a test case
      turnId: "turn-1",        // optional: identify a specific turn in a conversation
    }
  },
});
console.log(text);

Evals Usage

Online evals

You can run online evals on your AI SDK application by setting a metricCollection which will run evaluations on all incoming traces on Confident AI's servers. This approach is recommended if your agent is in production.

1. Create metric collection

   Create a metric collection on Confident AI with the metrics you wish to use to evaluate your AI SDK based application.

2. Run evals

   You can run evals at both the trace and span level. We recommend creating separate metric collections for each component, since each requires its own evaluation criteria and metrics.

   Pass different metric collections as trace attributes: metricCollection applies to the entire trace, while llmSpanContext.metricCollection applies to individual LLM spans, and llmSpanContext.toolsMetricCollection applies to tool call spans.

   Standard SetupExisting OTel

   import { generateText } from "ai";
   import { configureAiSdkTracing } from "deepeval";
   import { setTracingContext } from "deepeval/tracing";
   
   const tracer = configureAiSdkTracing();
   
   await setTracingContext(
     {
       metricCollection: "trace-metric-collection-name",
       llmSpanContext: {
         metricCollection: "llm-metric-collection-name",
         toolsMetricCollection: "tool-metric-collection-name"
       }
     }, 
     async () => {
       const { text } = await generateText({
         model: "openai/gpt-4o",
         prompt: "How to make the best coffee?",
         experimental_telemetry: {
           isEnabled: true,
           tracer: tracer,
         },
       });
       console.log(text);
     }
   );

You can view evals on Confident AI by visiting the traces pages inside the observatory on Confident AI platform.

Last updated on July 27, 2026