Manage Datasets

Overview

A dataset, which is either single or multi-turn one, is a list of goldens and forms the basis of any evaluation workflow in development. In this section, you'll learn to manipulate goldens in datasets, including:

• Understanding the golden structure for single and multi-turn datasets
• Uploading goldens via CSV on the platform
• Ingesting goldens from production traces, spans, and threads
• Assigning different team members to review and finalize goldens

Create A Dataset

A dataset can be created one under Project > Datasets (select either the single or multi-turn tab based on the type of dataset you wish to create):

Golden Structure

Understanding the golden structure is essential before uploading your data. Goldens are the building blocks of datasets, and their structure differs slightly between single-turn and multi-turn evaluations:

Upload Goldens via CSV

You can upload both single and multi-turn goldens stored in CSVs to datasets. The fields that you will be mapping to CSV headers will just be slightly different.

Ingest Goldens from Production

To grow a dataset from real traffic, add a dataset ingestion task in Workflows. It adds matching traces or spans to a single-turn dataset, or threads to a multi-turn one, as goldens every five minutes, with filters, a sample rate, and a cap on how many goldens it produces. Chain it after a classifier to ingest only traces with particular labels.

Other Actions

Beyond creating and uploading, you can also:

• Add Images — drag and drop images into text fields for multi-modal goldens
• Edit Non-Text Columns — modify structured fields like Context, Expected Tools, and Tools Called
• Add Custom Columns — extend goldens with additional metadata fields
• Assign Goldens — delegate review to team members
• (Un)finalize Goldens — enable or disable goldens for testing
• Duplicate Dataset — create a copy of an existing dataset
• Tag Datasets — organize datasets with tags
• Delete Dataset — permanently remove a dataset

Adding Images

Datasets on Confident AI are multi-modal by nature — images are natively supported alongside text. You can add images to goldens by dragging and dropping them directly into any text field, including Input, Expected Output, Context, and other list-of-text fields.

When you upload an image, Confident AI stores it and generates a public URL. This URL is embedded in your golden's text fields using a special format: [DEEPEVAL:IMAGE:uuid]. When you pull the dataset for evaluation, you can parse these into an evaluatable format.

Edit Non-Text Columns

Some golden fields require structured data rather than plain text. This is mostly relevant for single-turn datasets — multi-turn datasets only have Context.

A ToolCall object has the following structure:

{
  "name": "get_weather",
  "description": "Get weather for a location",
  "reasoning": "User asked about the weather in San Francisco",
  "output": "Sunny, 72°F",
  "input_parameters": { "location": "San Francisco" }
}

Add Custom Columns

Add custom columns to your dataset to store additional metadata. Custom columns appear as new fields on each golden and can be used for passing dynamic values during evaluation.

Assign Goldens

Assign goldens to different team members for review and annotation.

(Un)finalize Goldens

A golden's Finalized switch decides whether it's pulled for evaluation. Switch it on once you've reviewed and approved a golden, and leave it off while it's still being worked on.

Duplicate Dataset

Create a copy of an existing dataset. Useful when you want to create variations or preserve a snapshot before making changes.

In the dataset editor, click More actions → Duplicate → Dataset. Give the copy a New Dataset Alias (it must be unique to your project), choose under Fields to include which fields to carry over and from which source field, then click Duplicate.

Tag Datasets

Use tags to organize your datasets, for example by use case or owner. Tags show in the Tags column of the datasets list and under the dataset editor's heading.

• From the list — click a dataset's tags cell, then check existing tags or type a new one and press Enter. Changes save when the dropdown closes.
• From the dataset editor — click Add tag, type a name, and press Enter. Click × on a tag to remove it.

Dataset tags are separate from the tags on individual goldens.

Delete Dataset

Remove a dataset permanently on the platform:

Schedule Dataset Evals

Confident AI allows you to schedule automated evals on your datasets. Here's how you can schedule automated evals for your datasets:

1. Choose a Dataset

   1. Navigate to the Datasets tab in the sidebar
   2. Choose any single-turn or multi-turn dataset you wish to schedule evals for

   You'll be redirected to the dataset editor page where you can review and edit your dataset and it's goldens.

2. Create a Schedule

   1. Navigate to the Automations tab in the sidebar.
   2. Click Add Schedule and choose your configuration
   3. Click Create Schedule. It stays disabled until the schedule has an identifier, a metric collection, and an AI connection or prompt commit.

   This will now create a schedule with the specified configuration and run the evals for you with the same configuration at every X interval you've specified in the configuration.

Next Steps

Now that you know how to manage datasets on the platform, learn how to use them for evaluations or work with them programmatically in your code.

Last updated on October 6, 2026