Manage Datasets

Overview

A dataset, which is either single or multi-turn one, is a list of goldens and forms the basis of any evaluation workflow in development. In this section, you’ll learn to manipulate goldens in datasets, including:

• Understanding the golden structure for single and multi-turn datasets
• Uploading goldens via CSV on the platform
• Assigning different team members to review and finalize goldens

Create A Dataset

A dataset can be created one under Project > Datasets (select either the single or multi-turn tab based on the type of dataset you wish to create):

Golden Structure

Understanding the golden structure is essential before uploading your data. Goldens are the building blocks of datasets, and their structure differs slightly between single-turn and multi-turn evaluations:

Upload Goldens via CSV

You can upload both single and multi-turn goldens stored in CSVs to datasets. The fields that you will be mapping to CSV headers will just be slightly different.

Other Actions

Beyond creating and uploading, you can also:

• Add Images — drag and drop images into text fields for multi-modal goldens
• Edit Non-Text Columns — modify structured fields like Context, Expected Tools, and Tools Called
• Add Custom Columns — extend goldens with additional metadata fields
• Assign Goldens — delegate review to team members
• (Un)finalize Goldens — enable or disable goldens for testing
• Duplicate Dataset — create a copy of an existing dataset
• Delete Dataset — permanently remove a dataset

Adding Images

Datasets on Confident AI are multi-modal by nature — images are natively supported alongside text. You can add images to goldens by dragging and dropping them directly into any text field, including Input, Expected Output, Context, and other list-of-text fields.

When you upload an image, Confident AI stores it and generates a public URL. This URL is embedded in your golden’s text fields using a special format: [DEEPEVAL:IMAGE:uuid]. When you pull the dataset for evaluation, you can parse these into an evaluatable format.

Edit Non-Text Columns

Some golden fields require structured data rather than plain text. This is mostly relevant for single-turn datasets — multi-turn datasets only have Context.

A ToolCall object has the following structure:

1
{
2
  "name": "get_weather",
3
  "description": "Get weather for a location",
4
  "reasoning": "User asked about the weather in San Francisco",
5
  "output": "Sunny, 72°F",
6
  "input_parameters": { "location": "San Francisco" }
7
}

Add Custom Columns

Add custom columns to your dataset to store additional metadata. Custom columns appear as new fields on each golden and can be used for passing dynamic values during evaluation.

Assign Goldens

Assign goldens to different team members for review and annotation.

(Un)finalize Goldens

Mark goldens as finalized to lock them from further edits, or unfinalize to allow changes. Finalizing is useful when you’ve reviewed and approved goldens for use in evaluations.

Duplicate Dataset

Create a copy of an existing dataset. Useful when you want to create variations or preserve a snapshot before making changes.

Delete Dataset

Remove a dataset permanently on the platform:

Schedule Dataset Evals

Confident AI allows you to schedule automated evals on your datasets. Here’s how you can schedule automated evals for your datasets:

1

Choose a Dataset

1. Navigate to the Datasets tab in the sidebar
2. Choose any single-turn or multi-turn dataset you wish to schedule evals for

You’ll be redirected to the dataset editor page where you can review and edit your dataset and it’s goldens.

2

Create a Schedule

1. Navigate to the Automations tab in the sidebar.
2. Click Add Schedule and choose your configuration
3. Click Create Schedule.

This will now create a schedule with the specified configuration and run the evals for you with the same configuration at every X interval you’ve specified in the configuration.

Next Steps

Now that you know how to manage datasets on the platform, learn how to use them for evaluations or work with them programmatically in your code.