Mastra

Mastra is a TypeScript framework for building AI agents and workflows. Confident AI traces and evaluates Mastra applications through a single observability exporter, with no changes to your agent code.

Mastra emits structured spans for every agent run, model call, tool call, and workflow step. The DeepEvalExporter turns those spans into Confident AI traces and sends them to the Observatory.

Tracing Quickstart

1. Install Dependencies

   Add the deepeval package to your Mastra project:

   npm install deepeval

2. Set your API key

   Set your Confident AI API key as an environment variable. Your region (US/EU) is detected automatically from the key, so no endpoint configuration is needed.

   export CONFIDENT_API_KEY="your-confident-api-key"

3. Register the exporter

   Import DeepEvalExporter from deepeval/integrations/mastra and add it to your Mastra observability config:

   import { Mastra } from "@mastra/core";
   import { Observability } from "@mastra/observability";
   import { DeepEvalExporter } from "deepeval/integrations/mastra";
   
   export const mastra = new Mastra({
     observability: new Observability({
       configs: {
         deepeval: {
           serviceName: "my-service",
           exporters: [new DeepEvalExporter()],
         },
       },
     }),
   });

   You can also pass credentials directly instead of using environment variables:

   new DeepEvalExporter({
     apiKey: "your-confident-api-key",
     environment: "production",
   });

4. Run your agent

   Run any agent or workflow from your configured Mastra instance. Traces appear on Confident AI's traces page inside the Observatory.

   const agent = mastra.getAgent("myAgent");
   await agent.generate("How do I make the best coffee?");

Span type mapping

Mastra produces many span types. The exporter maps each one to a Confident AI span type, defaulting to Custom:

Each span carries its input, output, and metadata. LLM spans also capture the model name and token counts, and tool spans are collected into the trace's tool calls. Streaming event spans (for example model_chunk) are dropped to keep traces readable.

Advanced Usage

Configuration options

DeepEvalExporter accepts an optional options object to control tracing behavior:

import { DeepEvalExporter } from "deepeval/integrations/mastra";

new DeepEvalExporter({
  apiKey: "your-confident-api-key",      // defaults to CONFIDENT_API_KEY env var
  environment: "production",             // defaults to "development"
  name: "my-agent",                      // optional default trace name
  metricCollection: "my-metrics",        // optional metric collection for all traces
  debug: false,                          // set true to enable verbose logging
});

Setting trace attributes

Confident AI's tracing features let you attach attributes such as threadId, userId, tags, and metadata to your traces. threadId and userId group related traces together, which is useful for chat apps, agents, and multi-turn interactions. You can learn more about threads here.

Attributes can be set at two levels. Exporter-level options apply to every trace as defaults:

import { DeepEvalExporter } from "deepeval/integrations/mastra";

new DeepEvalExporter({
  name: "support-agent",
  tags: ["production", "support"],
  metadata: { version: "1.2.0" },
});

Per-request attributes are set on each generate / stream call via Mastra's tracingOptions (and memory for thread and user identity), and override the exporter defaults for that trace:

await agent.generate("What's the weather in Tokyo?", {
  // thread + user identity (also sets the span's conversationId)
  memory: { thread: "thread-123", resource: "user-456" },
  tracingOptions: {
    metadata: {
      userId: "user-456", // maps to the trace's user
      threadId: "thread-123", // maps to the trace's thread
      testCaseId: "tc-1", // optional: link to a Confident AI test case
      turnId: "turn-1", // optional: identify a conversation turn
      team: "growth", // any other keys land in trace metadata
    },
    tags: ["beta"],
  },
});

The exporter reads these from the root span: tracingOptions.metadata keys userId, threadId (or sessionId), testCaseId, turnId, and traceName map to the matching trace fields; any other keys go to trace metadata; tracingOptions.tags become trace tags; and memory.thread / memory.resource populate the thread and user identity.

Logging prompts

If you are managing prompts on Confident AI and wish to log them, pass your Prompt object to the exporter. It's linked to every LLM span the exporter produces:

import { Prompt } from "deepeval";
import { DeepEvalExporter } from "deepeval/integrations/mastra";

const prompt = new Prompt({ alias: "PROMPT_ALIAS" });
await prompt.pull();

new DeepEvalExporter({ prompt });

Evals Usage

Online evals

You can run online evals on your Mastra application by setting a metricCollection, which evaluates all incoming traces on Confident AI's servers. This approach is recommended if your agent is in production.

1. Create metric collection

   Create a metric collection on Confident AI with the metrics you wish to use to evaluate your Mastra application.

2. Run evals

   You can run evals at both the trace and span level. We recommend creating separate metric collections for each component, since each requires its own evaluation criteria and metrics.

   Pass the metric collections to the exporter. metricCollection applies to the entire trace, while llmMetricCollection, agentMetricCollection, and toolMetricCollectionMap apply to individual spans by type.

   import { DeepEvalExporter } from "deepeval/integrations/mastra";
   
   new DeepEvalExporter({
     metricCollection: "trace-metric-collection-name",
     llmMetricCollection: "llm-metric-collection-name",
     agentMetricCollection: "agent-metric-collection-name",
     toolMetricCollectionMap: {
       search: "tool-metric-collection-name",
     },
   });

You can view evals on Confident AI by visiting the traces pages inside the Observatory.