Vercel AI SDK

Use Confident AI for LLM observability and evals for Vercel AI SDK on typescript

Vercel AI SDK

Use Confident AI for LLM observability and evals for Vercel AI SDK on typescript

The AI SDK by vercel is a powerful typescript framework that allows you to use various LLM providers and models for AI based applications. Confident AI allows you to trace and evaluate AI SDK based LLM applications in just a few lines of code.

Choose Your Integration Mode

DeepEval supports two ways to integrate tracing with the Vercel AI SDK.

Standard Setup (No Existing Otel)

Existing OpenTelemetry Setup

If you are not already using OpenTelemetry, simply use configureAiSdkTracing and pass the tracer to the AI SDK telemetry:

1 import { generateText } from "ai";
2 import { configureAiSdkTracing } from "deepeval";
3 
4 const tracer = configureAiSdkTracing();
5 
6 await generateText({
7   model: "openai/gpt-4o",
8   prompt: "Explain how neural networks work",
9   experimental_telemetry: {
10     isEnabled: true,
11     tracer: tracer,
12   },
13 });

This is the easiest way to enable tracing.

Tracing Quickstart

For users in the EU region, please set the OTEL endpoint to the EU version as shown below:

$ export CONFIDENT_OTEL_URL="https://eu.otel.confident-ai.com"

Install Dependencies

Run the following command to install the required packages:

$ npm install ai deepeval

Configure AI SDK

Use DeepEval’s configureAiSdkTracing to trace the LLM operations.

Standard Setup

Existing Otel

Generate Text

Stream Text

Tool Calling

Generate Structured Data

Embedding Text

1 import { generateText } from "ai";
2 import { configureAiSdkTracing } from "deepeval";
3 
4 const tracer = configureAiSdkTracing();
5 
6 const { text } = await generateText({
7     model: "openai/gpt-4o",
8     prompt: "How to make the best coffee?",
9     experimental_telemetry: {
10         isEnabled: true,
11         tracer: tracer,
12     },
13 });
14 
15 console.log(text);

Run AI SDK Generation

Run your LLM application by executing the following script:

$ npx ts-node

You can directly view the traces on Confident AI’s traces page inside the observatory.

Advanced Usage

Logging prompts

If you are managing prompts on Confident AI and wish to log them, pass your Prompt object to the llmSpanContext using the setTracingContext function:

Standard Setup

Existing Otel

1 import { generateText } from "ai";
2 import { configureAiSdkTracing, Prompt } from "deepeval";
3 import { setTracingContext } from "deepeval/tracing";
4 
5 const prompt = new Prompt({ alias: "PROMPT_ALIAS" });
6 prompt.pull();
7 
8 const tracer = configureAiSdkTracing();
9 
10 await setTracingContext(
11   {
12     llmSpanContext: {
13       prompt: prompt
14     }
15   }, 
16   async () => {
17     const { text } = await generateText({
18       model: "openai/gpt-4o",
19       prompt: "How to make the best coffee?",
20       experimental_telemetry: {
21         isEnabled: true,
22         tracer: tracer,
23       },
24     });
25     console.log(text);
26   }
27 );

Logging prompts lets you attribute specific prompts to AI SDK LLM spans. Be sure to pull the prompt before logging it, otherwise the prompt will not be visible on Confident AI.

Setting trace attributes

Confident AI’s LLM tracing advanced features provide teams with the ability to set certain attributes for each trace when invoking your AI SDK applications.

For example, thread_id and user_id are used to group related traces together, and are useful for chat apps, agents, or any multi-turn interactions. You can learn more about threads here.

You can set these attributes using the setTracingContext function from deepeval:

Standard Setup

Existing Otel

1 import { generateText } from "ai";
2 import { configureAiSdkTracing } from "deepeval";
3 import { setTracingContext } from "deepeval/tracing";
4 
5 const tracer = configureAiSdkTracing();
6 
7 await setTracingContext(
8   {
9     threadId: "123",
10     userId: "456"
11   }, 
12   async () => {
13     const { text } = await generateText({
14       model: "openai/gpt-4o",
15       prompt: "How to make the best coffee?",
16       experimental_telemetry: {
17         isEnabled: true,
18         tracer: tracer,
19       },
20     });
21     console.log(text);
22   }
23 );

View Trace Attributes

name

str

The name of the trace. Learn more.

Evals Usage

Online evals

You can run online evals on your AI SDK application by setting a metricCollection which will run evaluations on all incoming traces on Confident AI’s servers. This approach is recommended if your agent is in production.

Create metric collection

Create a metric collection on Confident AI with the metrics you wish to use to evaluate your AI SDK based application.

Create metric collection

Your metric collection must only contain metrics that only evaluate the input and actual output of your AI SDK application.

Run evals

You can run evals at both the trace and span level. We recommend creating separate metric collections for each component, since each requires its own evaluation criteria and metrics. You can pass different metric collections like metricCollection (for entire trace), llmMetricCollection, toolMetricCollection in the configureAiSdkTracing as shown below:

Standard Setup

Existing Otel

1 import { generateText } from "ai";
2 import { configureAiSdkTracing } from "deepeval";
3 import { setTracingContext } from "deepeval/tracing";
4 
5 const tracer = configureAiSdkTracing();
6 
7 await setTracingContext(
8   {
9     metricCollection: "trace-metric-collection-name",
10     llmSpanContext: {
11       metricCollection: "llm-metric-collection-name",
12       toolMetricCollection: "tool-metric-collection-name"
13     }
14   }, 
15   async () => {
16     const { text } = await generateText({
17       model: "openai/gpt-4o",
18       prompt: "How to make the best coffee?",
19       experimental_telemetry: {
20         isEnabled: true,
21         tracer: tracer,
22       },
23     });
24     console.log(text);
25   }
26 );

All incoming traces will now be evaluated using metrics from your metric collection.

You can view evals on Confident AI by visiting the traces pages inside the observatory on Confident AI platform.

Choose Your Integration Mode

DeepEval supports two ways to integrate tracing with the Vercel AI SDK.

Standard Setup (No Existing Otel)

Existing OpenTelemetry Setup

If you are not already using OpenTelemetry, simply use configureAiSdkTracing and pass the tracer to the AI SDK telemetry:

1 import { generateText } from "ai";
2 import { configureAiSdkTracing } from "deepeval";
3 
4 const tracer = configureAiSdkTracing();
5 
6 await generateText({
7   model: "openai/gpt-4o",
8   prompt: "Explain how neural networks work",
9   experimental_telemetry: {
10     isEnabled: true,
11     tracer: tracer,
12   },
13 });

This is the easiest way to enable tracing.

Tracing Quickstart

For users in the EU region, please set the OTEL endpoint to the EU version as shown below:

$ export CONFIDENT_OTEL_URL="https://eu.otel.confident-ai.com"

Install Dependencies

Run the following command to install the required packages:

$ npm install ai deepeval

Configure AI SDK

Use DeepEval’s configureAiSdkTracing to trace the LLM operations.

Standard Setup

Existing Otel

Generate Text

Stream Text

Tool Calling

Generate Structured Data

Embedding Text

1 import { generateText } from "ai";
2 import { configureAiSdkTracing } from "deepeval";
3 
4 const tracer = configureAiSdkTracing();
5 
6 const { text } = await generateText({
7     model: "openai/gpt-4o",
8     prompt: "How to make the best coffee?",
9     experimental_telemetry: {
10         isEnabled: true,
11         tracer: tracer,
12     },
13 });
14 
15 console.log(text);

Run AI SDK Generation

Run your LLM application by executing the following script:

$ npx ts-node

You can directly view the traces on Confident AI’s traces page inside the observatory.

Advanced Usage

Logging prompts

If you are managing prompts on Confident AI and wish to log them, pass your Prompt object to the llmSpanContext using the setTracingContext function:

Standard Setup

Existing Otel

1 import { generateText } from "ai";
2 import { configureAiSdkTracing, Prompt } from "deepeval";
3 import { setTracingContext } from "deepeval/tracing";
4 
5 const prompt = new Prompt({ alias: "PROMPT_ALIAS" });
6 prompt.pull();
7 
8 const tracer = configureAiSdkTracing();
9 
10 await setTracingContext(
11   {
12     llmSpanContext: {
13       prompt: prompt
14     }
15   }, 
16   async () => {
17     const { text } = await generateText({
18       model: "openai/gpt-4o",
19       prompt: "How to make the best coffee?",
20       experimental_telemetry: {
21         isEnabled: true,
22         tracer: tracer,
23       },
24     });
25     console.log(text);
26   }
27 );

Logging prompts lets you attribute specific prompts to AI SDK LLM spans. Be sure to pull the prompt before logging it, otherwise the prompt will not be visible on Confident AI.

Setting trace attributes

Confident AI’s LLM tracing advanced features provide teams with the ability to set certain attributes for each trace when invoking your AI SDK applications.

For example, thread_id and user_id are used to group related traces together, and are useful for chat apps, agents, or any multi-turn interactions. You can learn more about threads here.

You can set these attributes using the setTracingContext function from deepeval:

Standard Setup

Existing Otel

1 import { generateText } from "ai";
2 import { configureAiSdkTracing } from "deepeval";
3 import { setTracingContext } from "deepeval/tracing";
4 
5 const tracer = configureAiSdkTracing();
6 
7 await setTracingContext(
8   {
9     threadId: "123",
10     userId: "456"
11   }, 
12   async () => {
13     const { text } = await generateText({
14       model: "openai/gpt-4o",
15       prompt: "How to make the best coffee?",
16       experimental_telemetry: {
17         isEnabled: true,
18         tracer: tracer,
19       },
20     });
21     console.log(text);
22   }
23 );

View Trace Attributes

name

str

The name of the trace. Learn more.

Evals Usage

Online evals

Create metric collection

Create a metric collection on Confident AI with the metrics you wish to use to evaluate your AI SDK based application.

Create metric collection

Your metric collection must only contain metrics that only evaluate the input and actual output of your AI SDK application.

Run evals

Standard Setup

Existing Otel

1 import { generateText } from "ai";
2 import { configureAiSdkTracing } from "deepeval";
3 import { setTracingContext } from "deepeval/tracing";
4 
5 const tracer = configureAiSdkTracing();
6 
7 await setTracingContext(
8   {
9     metricCollection: "trace-metric-collection-name",
10     llmSpanContext: {
11       metricCollection: "llm-metric-collection-name",
12       toolMetricCollection: "tool-metric-collection-name"
13     }
14   }, 
15   async () => {
16     const { text } = await generateText({
17       model: "openai/gpt-4o",
18       prompt: "How to make the best coffee?",
19       experimental_telemetry: {
20         isEnabled: true,
21         tracer: tracer,
22       },
23     });
24     console.log(text);
25   }
26 );

All incoming traces will now be evaluated using metrics from your metric collection.

You can view evals on Confident AI by visiting the traces pages inside the observatory on Confident AI platform.

1	import { generateText } from "ai";
2	import { configureAiSdkTracing } from "deepeval";
3
4	const tracer = configureAiSdkTracing();
5
6	await generateText({
7	model: "openai/gpt-4o",
8	prompt: "Explain how neural networks work",
9	experimental_telemetry: {
10	isEnabled: true,
11	tracer: tracer,
12	},
13	});

1	import { generateText } from "ai";
2	import { configureAiSdkTracing, Prompt } from "deepeval";
3	import { setTracingContext } from "deepeval/tracing";
4
5	const prompt = new Prompt({ alias: "PROMPT_ALIAS" });
6	prompt.pull();
7
8	const tracer = configureAiSdkTracing();
9
10	await setTracingContext(
11	{
12	llmSpanContext: {
13	prompt: prompt
14	}
15	},
16	async () => {
17	const { text } = await generateText({
18	model: "openai/gpt-4o",
19	prompt: "How to make the best coffee?",
20	experimental_telemetry: {
21	isEnabled: true,
22	tracer: tracer,
23	},
24	});
25	console.log(text);
26	}
27	);