Introduction to SDK Reference

What is the Confident AI SDK?

The Confident AI SDK is the official Python and TypeScript client for the Confident API. It wraps every endpoint in typed methods, so you work with objects in your own language instead of building requests by hand. It allows you to:

• Run evaluations, pull datasets, and version prompts from inside your application code
• Ingest LLM traces and spans, and read them back for debugging and monitoring
• Manage metrics, annotations, alerts, and the rest of your project data programmatically
• Administer your organization — projects, members, roles, and API keys

Every SDK method maps onto a documented API endpoint, so anything you can do over HTTP you can do here in your native language.

Prerequisites

• Python 3.9+ or Node.js 18+
• A Confident AI account. If you don't have one, sign up here.
• A project or organization API key

1. Install the SDK

   Install the Confident AI SDK package for your language.

   PythonTypeScript

   pip install confident-ai

2. Set your API key

   There are two kinds of API key, and which one you need depends on the resource you are reaching for. Get them from Confident AI, then set them as environment variables in the shell.

   macOS / LinuxWindows (PowerShell)

   export CONFIDENT_PROJ_API_KEY="<PROJECT-API-KEY>"
   export CONFIDENT_ORG_API_KEY="<ORGANIZATION-API-KEY>"

   A project API key authenticates everything scoped to one project — datasets, prompts, traces, metrics, test runs, and the rest. An organization API key authenticates organization administration only: organization and projects. Most code needs the project key alone.

3. Create a client

   One client serves every resource. It reads both environment variables when it is constructed, and each method reaches for whichever key it needs.

   PythonTypeScript

   from confident_ai import ConfidentAI
   
   client = ConfidentAI()

Resources

A resource maps to a single feature or entity in Confident AI — a dataset, metric, trace, etc. The SDK provides various methods for each resource under its own name on the client: client.datasets, client.metrics, client.traces. The SDK reference documents each resource's methods, parameters, and types in detail with examples on how to use them.

Resources come in two shapes, stateless and stateful, and the shape decides how you call them.

Stateless resources

Stateless resources offer methods that are a one-time operation. You can pass the required arguments to the method and get the result back without storing any data in memory.

annotations = client.annotations.list(page=1, page_size=25)
metric = client.metrics.get(metric_id="<METRIC-ID>")
trace = client.traces.get(trace_uuid="<TRACE-UUID>")

const annotations = await client.annotations.list({ page: 1, pageSize: 25 });
const metric = await client.metrics.get("<METRIC-ID>");
const trace = await client.traces.get("<TRACE-UUID>");

Stateful resources

Stateful resources are objects you can create and mutate in code. They hold the data in memory, and you can call methods directly on the object.

Here's an example of how you can create a stateful dataset and edit it in memory using the Confident AI SDK and push the changes back to the platform:

dataset = client.dataset(dataset_id="<DATASET-ID>")

dataset.pull()                  # fills the object's fields from the platform
dataset.create_golden(...)      # no dataset_id argument needed
dataset.push()                  # sends the fields the object is holding

const dataset = client.dataset("<DATASET-ID>");

await dataset.pull();            // fills the object's fields from the platform
await dataset.createGolden(...); // no datasetId argument needed
await dataset.push();            // sends the fields the object is holding

The Confident AI SDK supports three stateful resources:

These are the only resources that support stateful operations, they also provide stateless methods to perform the same operations on different entities without having to create an object for each one.

Stateful vs stateless resources

Use the stateless methods when you want a single operation or you are working across records, like listing them or creating one before you have an id. Use the stateful object when you want to make several calls against the same record, or when a method sends the fields the object is holding, like push.

You can tell the two apart by the name. The stateless methods sit under the plural name, and the stateful object comes from the singular one.

client.datasets.list()          # across datasets
client.dataset(dataset_id="<DATASET-ID>").pull()   # within one dataset

client.datasets.list();         // across datasets
client.dataset("<DATASET-ID>").pull(); // within one dataset

Asynchronous methods

Every method is available asynchronously. In Python each method has an a_-prefixed twin that you can await for async operations. TypeScript is natively asynchronous, so every method already returns a promise.

result = await client.annotations.a_list(page=1)
await dataset.a_pull()

const result = await client.annotations.list({ page: 1 });
await dataset.pull();

Images

Goldens, test cases and annotations can carry images. Put a Media where the image belongs in any of their text fields: formatting it into a string leaves a [CONFIDENT:IMAGE:…] marker, and the SDK sends the image with the request.

from confident_ai import Media
from confident_ai.datasets import SingleTurnGolden

photo = Media.from_file("receipt.png")
client.datasets.push(
    alias="receipts",
    goldens=[SingleTurnGolden(input=f"What is the total? {photo}")],
)

import { Media } from "confident-ai";

const photo = Media.fromFile("receipt.png");
await client.datasets.push("receipts", [
  { input: `What is the total? ${photo}` },
]);

Create a Media from wherever the image is:

• Images only, for now. The mime type is guessed from a URL's or file's extension, and is required for bytes and base64.
• Every call that takes images fills in their imagesMapping for you: pushing goldens, creating and updating a golden, running evals, and creating or updating annotations. Markers in a multi-turn golden's turns belong to the golden.
• Local images stay in memory until the process exits, and only resolve in the process that made them. URL images work anywhere.

Text you pull back carries its images as markers too, each holding a signed link. parse_multimodal splits it into text and Media in order, ready to hand to a model:

from confident_ai import parse_multimodal

parts = parse_multimodal(golden.input)   # ["What is the total? ", Media(...)]

import { parseMultimodal } from "confident-ai";

const parts = parseMultimodal(golden.input); // ["What is the total? ", Media]

Command line

Both packages install a confident-ai command. confident-ai gate checks the project against its governance policy and exits non-zero when it doesn't pass, so a CI step can stop a deployment:

confident-ai gate

npx confident-ai gate

It reads CONFIDENT_PROJ_API_KEY, and exits 0 when the policy passes, 1 when it fails or can't be assessed, and 2 on bad usage. Pass -q or --quiet to print nothing and leave the verdict to the exit code.

FAQs

Last updated on October 18, 2026