Vercel AI SDK | Confident AI Docs

The AI SDK by Vercel is a powerful TypeScript framework that allows you to use various LLM providers and models for AI-based applications. Confident AI allows you to trace and evaluate AI SDK based LLM applications in just a few lines of code.

Choose Your Integration Mode

DeepEval supports two ways to integrate tracing with the Vercel AI SDK.

Standard Setup (No Existing OTel)

Existing OpenTelemetry Setup

If you are not already using OpenTelemetry, simply use configureAiSdkTracing and pass the tracer to the AI SDK telemetry:

1 import { generateText } from "ai";
2 import { configureAiSdkTracing } from "deepeval";
3 
4 const tracer = configureAiSdkTracing();
5 
6 await generateText({
7   model: "openai/gpt-4o",
8   prompt: "Explain how neural networks work",
9   experimental_telemetry: {
10     isEnabled: true,
11     tracer: tracer,
12   },
13 });

This is the easiest way to enable tracing.

Tracing Quickstart

For users in the EU region, please set the OTEL endpoint to the EU version as shown below:

$ export CONFIDENT_OTEL_URL="https://eu.otel.confident-ai.com"

Install Dependencies

Run the following command to install the required packages:

$ npm install ai deepeval

Configure AI SDK

Use DeepEval’s configureAiSdkTracing to trace LLM operations.

Standard Setup

Existing OTel

Generate Text

Stream Text

Tool Calling

Generate Structured Data

Embedding Text

1 import { generateText } from "ai";
2 import { configureAiSdkTracing } from "deepeval";
3 
4 const tracer = configureAiSdkTracing();
5 
6 const { text } = await generateText({
7     model: "openai/gpt-4o",
8     prompt: "How to make the best coffee?",
9     experimental_telemetry: {
10         isEnabled: true,
11         tracer: tracer,
12     },
13 });
14 
15 console.log(text);

Run AI SDK Generation

Run your LLM application. You can directly view the traces on Confident AI’s traces page inside the observatory.

Advanced Usage

Configuration options

configureAiSdkTracing accepts an optional options object to control tracing behavior:

1 import { configureAiSdkTracing } from "deepeval";
2 
3 const tracer = configureAiSdkTracing({
4   apiKey: "your-confident-api-key",     // defaults to CONFIDENT_API_KEY env var
5   environment: "production",             // defaults to "development"
6   name: "my-ai-app",                    // optional default trace name
7   traceMetricCollection: "my-metrics",  // optional default metric collection for all traces
8   debug: false,                          // set true to enable verbose logging
9 });

View configureAiSdkTracing Options

apiKey

string

Your Confident AI API key. Defaults to the CONFIDENT_API_KEY environment variable.

environment

string

The deployment environment label attached to all traces (e.g. "production", "staging"). Defaults to "development". Can also be set via the CONFIDENT_TRACE_ENVIRONMENT environment variable.

name

string

A default name applied to every trace created by this tracer. Can be overridden per-trace using setTracingContext or via ai.telemetry.metadata.traceName.

traceMetricCollection

string

A default metric collection applied to every trace for online evaluation. Can be overridden per-trace using setTracingContext or via ai.telemetry.metadata.traceMetricCollection.

otelEndpoint

string

Custom OTLP endpoint URL. Defaults to https://otel.confident-ai.com. Use https://eu.otel.confident-ai.com for the EU region.

debug

boolean

When true, logs tracing configuration details and flush events to the console. Defaults to false.

Logging prompts

If you are managing prompts on Confident AI and wish to log them, pass your Prompt object to the llmSpanContext using the setTracingContext function:

Standard Setup

Existing OTel

1 import { generateText } from "ai";
2 import { configureAiSdkTracing, Prompt } from "deepeval";
3 import { setTracingContext } from "deepeval/tracing";
4 
5 const prompt = new Prompt({ alias: "PROMPT_ALIAS" });
6 prompt.pull();
7 
8 const tracer = configureAiSdkTracing();
9 
10 await setTracingContext(
11   {
12     llmSpanContext: {
13       prompt: prompt
14     }
15   }, 
16   async () => {
17     const { text } = await generateText({
18       model: "openai/gpt-4o",
19       prompt: "How to make the best coffee?",
20       experimental_telemetry: {
21         isEnabled: true,
22         tracer: tracer,
23       },
24     });
25     console.log(text);
26   }
27 );

Logging prompts lets you attribute specific prompts to AI SDK LLM spans. Be sure to pull the prompt before logging it, otherwise the prompt will not be visible on Confident AI.

Setting trace attributes

Confident AI’s LLM tracing advanced features provide teams with the ability to set certain attributes for each trace when invoking your AI SDK applications.

For example, threadId and userId are used to group related traces together, and are useful for chat apps, agents, or any multi-turn interactions. You can learn more about threads here.

You can set these attributes using the setTracingContext function from deepeval/tracing:

Standard Setup

Existing OTel

1 import { generateText } from "ai";
2 import { configureAiSdkTracing } from "deepeval";
3 import { setTracingContext } from "deepeval/tracing";
4 
5 const tracer = configureAiSdkTracing();
6 
7 await setTracingContext(
8   {
9     threadId: "thread-123",
10     userId: "user-456",
11     testCaseId: "tc-789",   // optional: link trace to a test case
12     turnId: "turn-1",        // optional: identify a specific turn in a conversation
13   }, 
14   async () => {
15     const { text } = await generateText({
16       model: "openai/gpt-4o",
17       prompt: "How to make the best coffee?",
18       experimental_telemetry: {
19         isEnabled: true,
20         tracer: tracer,
21       },
22     });
23     console.log(text);
24   }
25 );

View Trace Attributes

name

string

The name of the trace. Learn more.

Evals Usage

Online evals

You can run online evals on your AI SDK application by setting a metricCollection which will run evaluations on all incoming traces on Confident AI’s servers. This approach is recommended if your agent is in production.

Create metric collection

Create a metric collection on Confident AI with the metrics you wish to use to evaluate your AI SDK based application.

Create metric collection

Your metric collection must only contain metrics that only evaluate the input and actual output of your AI SDK application.

Run evals

You can run evals at both the trace and span level. We recommend creating separate metric collections for each component, since each requires its own evaluation criteria and metrics.

Pass different metric collections as trace attributes: metricCollection applies to the entire trace, while llmSpanContext.metricCollection applies to individual LLM spans, and llmSpanContext.toolsMetricCollection applies to tool call spans.

Standard Setup

Existing OTel

1 import { generateText } from "ai";
2 import { configureAiSdkTracing } from "deepeval";
3 import { setTracingContext } from "deepeval/tracing";
4 
5 const tracer = configureAiSdkTracing();
6 
7 await setTracingContext(
8   {
9     metricCollection: "trace-metric-collection-name",
10     llmSpanContext: {
11       metricCollection: "llm-metric-collection-name",
12       toolsMetricCollection: "tool-metric-collection-name"
13     }
14   }, 
15   async () => {
16     const { text } = await generateText({
17       model: "openai/gpt-4o",
18       prompt: "How to make the best coffee?",
19       experimental_telemetry: {
20         isEnabled: true,
21         tracer: tracer,
22       },
23     });
24     console.log(text);
25   }
26 );

All incoming traces will now be evaluated using metrics from your metric collection.

You can view evals on Confident AI by visiting the traces pages inside the observatory on Confident AI platform.