Automate Dataset Management

Programmatically push goldens to datasets via the Evals API.

Overview

This section covers how to programmatically manage goldens in datasets using the Evals API:

Push single and multi-turn goldens to datasets
Set finalized=True to make goldens available for evaluation, or finalized=False to queue for review
Include custom column values when pushing goldens
Update or delete an individual golden by its id
Delete datasets programmatically

Push Goldens

Push goldens to a dataset. If the dataset does not already exist, Confident AI will create it for you.

For single-turn datasets:

main.py

from deepeval.dataset import EvaluationDataset, Golden

goldens = [Golden(input="How tall is Mt. Everest?")]
dataset = EvaluationDataset(goldens=goldens)

# Push as finalized (ready for evaluation)
dataset.push(alias="YOUR-DATASET-ALIAS", finalized=True)

# Or push as unfinalized (queued for review)
dataset.push(alias="YOUR-DATASET-ALIAS", finalized=False)

For multi-turn datasets:

main.py

from deepeval.dataset import EvaluationDataset, ConversationalGolden
from deepeval.test_case import Turn

goldens = [
ConversationalGolden(
scenario="Angry user asking for a refund.",
turns=[Turn(role="user", content="Give me my money!")]
)
]
dataset = EvaluationDataset(goldens=goldens)

dataset.push(alias="YOUR-DATASET-ALIAS", finalized=True)

main.py

from deepeval.dataset import EvaluationDataset, ConversationalGolden

goldens = [ConversationalGolden(scenario="Angry user asking for a refund.")]
dataset = EvaluationDataset(goldens=goldens)

dataset.push(alias="YOUR-DATASET-ALIAS", finalized=True)

For single-turn datasets:

index.ts

import { EvaluationDataset, Golden } from "deepeval";

const goldens = [new Golden({ input: "How tall is Mt. Everest?" })];
const dataset = new EvaluationDataset({ goldens: goldens });

// Push as finalized (ready for evaluation)
dataset.push({ alias: "YOUR-DATASET-ALIAS", finalized: true });

// Or push as unfinalized (queued for review)
dataset.push({ alias: "YOUR-DATASET-ALIAS", finalized: false });

For multi-turn datasets:

index.ts

import { EvaluationDataset, ConversationalGolden, Turn } from "deepeval";

const firstTurn = new Turn({
  role: "user",
  content: "Where's my money!?",
});

const goldens = [
  new ConversationalGolden({
    scenario: "Angry user asking for a refund.",
    turns: [firstTurn],
  }),
];
const dataset = new EvaluationDataset({ goldens: goldens });

dataset.push({ alias: "YOUR-DATASET-ALIAS", finalized: true });

index.ts

import { EvaluationDataset, ConversationalGolden } from "deepeval";

const conversationalGolden = new ConversationalGolden({
  scenario: "Angry user asking for a refund.",
});

const goldens = [conversationalGolden];
const dataset = new EvaluationDataset({ goldens: goldens });

dataset.push({ alias: "YOUR-DATASET-ALIAS", finalized: true });

Set finalized to true for evaluation-ready goldens, or false to queue for review.

POST/v1/datasets/{alias}

curl -X POST "https://api.confident-ai.com/v1/datasets/{alias}" \
  -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>" \
  -H "Content-Type: application/json" \
  -d '{
  "finalized": true,
  "goldens": [
    {
      "input": "How is the weather like in NYC?",
      "expectedOutput": "No idea"
    }
  ]
}'

For multi-turn datasets:

POST/v1/datasets/{alias}

curl -X POST "https://api.confident-ai.com/v1/datasets/{alias}" \
  -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>" \
  -H "Content-Type: application/json" \
  -d '{
  "finalized": true,
  "conversationalGoldens": [
    {
      "scenario": "Booking a hotel",
      "expectedOutcome": "Successfully booked",
      "userDescription": "Finds hotels in NYC"
    }
  ]
}'

Add Custom Columns

You can include custom column values when pushing goldens. Custom columns must already exist on the dataset, or Confident AI will create them for you.

main.py

from deepeval.dataset import Golden, ConversationalGolden

golden = Golden(
    input="How tall is Mt. Everest?",
    custom_column_key_values={"difficulty": "easy", "category": "geography"}
)

multiturn_golden = ConversationalGolden(
    scenario="User asking for a refund.",
    custom_column_key_values={"sentiment": "angry", "priority": "high"}
)

index.ts

import { Golden, ConversationalGolden } from "deepeval";

const golden = new Golden({
  input: "How tall is Mt. Everest?",
  customColumnKeyValues: { difficulty: "easy", category: "geography" },
});

const multiturnGolden = new ConversationalGolden({
  scenario: "User asking for a refund.",
  customColumnKeyValues: { sentiment: "angry", priority: "high" },
});

POST/v1/datasets/{alias}

curl -X POST "https://api.confident-ai.com/v1/datasets/{alias}" \
  -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>" \
  -H "Content-Type: application/json" \
  -d '{
  "finalized": true,
  "goldens": [
    {
      "input": "What is 2 + 2?",
      "customColumnKeyValues": {
        "key": "value"
      }
    }
  ]
}'

Versioning Datasets

Datasets support immutable, named versions so you can pin evaluation runs to a specific snapshot of goldens.

Create a version to snapshot the current state of the dataset.
Push without specifying version to add goldens to the latest version (or unversioned, if the dataset has no versions yet).
Push with version=... to add goldens to a specific version.
Pull without version to read the latest version. Pull with version=... to read a specific version.
Get versions to list all snapshots, newest first.

Create a version

main.py

from deepeval.dataset import EvaluationDataset

dataset = EvaluationDataset()
version = dataset.create_version(alias="YOUR-DATASET-ALIAS")
# version -> "00.00.01"

index.ts

import { EvaluationDataset } from "deepeval";

const dataset = new EvaluationDataset();
const { version } = await dataset.createVersion({ alias: "YOUR-DATASET-ALIAS" });
// version -> "00.00.01"

POST/v1/datasets/{alias}/versions

curl -X POST "https://api.confident-ai.com/v1/datasets/{alias}/versions" \
  -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>"

The first call to create_version backfills every existing unversioned golden onto the new version. Subsequent calls snapshot all goldens from the previous version (with new IDs) and auto-increment the version number.

List versions

main.py

from deepeval.dataset import EvaluationDataset

dataset = EvaluationDataset()
versions = dataset.get_versions(alias="YOUR-DATASET-ALIAS")
for v in versions:
    print(v.version, v.id)

index.ts

import { EvaluationDataset } from "deepeval";

const dataset = new EvaluationDataset();
const versions = await dataset.getVersions({ alias: "YOUR-DATASET-ALIAS" });
for (const v of versions) {
  console.log(v.version, v.id);
}

GET/v1/datasets/{alias}/versions

curl -X GET "https://api.confident-ai.com/v1/datasets/{alias}/versions" \
  -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>"

Push and pull a specific version

main.py

from deepeval.dataset import EvaluationDataset, Golden

dataset = EvaluationDataset(goldens=[Golden(input="...", expected_output="...")])

# Push goldens onto version 00.00.01
dataset.push(alias="YOUR-DATASET-ALIAS", version="00.00.01")

# Pull a specific version
dataset.pull(alias="YOUR-DATASET-ALIAS", version="00.00.01")
print(dataset._version)  # -> "00.00.01"

index.ts

import { EvaluationDataset, Golden } from "deepeval";

const dataset = new EvaluationDataset();
dataset.addGolden(new Golden({ input: "...", expectedOutput: "..." }));

// Push goldens onto version 00.00.01
await dataset.push({ alias: "YOUR-DATASET-ALIAS", version: "00.00.01" });

// Pull a specific version
await dataset.pull({ alias: "YOUR-DATASET-ALIAS", version: "00.00.01" });

Update a Golden

Update a single golden in place by its id. Pull the dataset first so each golden carries a stable id, edit the fields you want, then send the update.

The golden's fields are replaced with the values you send. tags and custom columns are only changed when you include them.

main.py

from deepeval.dataset import EvaluationDataset

dataset = EvaluationDataset()
dataset.pull(alias="YOUR-DATASET-ALIAS")

# Each pulled golden carries a stable id
golden = dataset.goldens[0]
golden.input = "How tall is Mt. Everest, in meters?"

dataset.update_golden(golden=golden)

index.ts

import { EvaluationDataset } from "deepeval";

const dataset = new EvaluationDataset();
await dataset.pull({ alias: "YOUR-DATASET-ALIAS" });

// Each pulled golden carries a stable id
const golden = dataset.goldens[0];
golden.input = "How tall is Mt. Everest, in meters?";

await dataset.updateGolden({ golden });

For single-turn datasets:

PUT/v1/datasets/{alias}/goldens/{goldenId}

curl -X PUT "https://api.confident-ai.com/v1/datasets/{alias}/goldens/{goldenId}" \
  -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>" \
  -H "Content-Type: application/json" \
  -d '{
  "input": "How is the weather like in NYC?",
  "expectedOutput": "Sunny with a chance of rain.",
  "finalized": true
}'

For multi-turn datasets:

PUT/v1/datasets/{alias}/goldens/{goldenId}

curl -X PUT "https://api.confident-ai.com/v1/datasets/{alias}/goldens/{goldenId}" \
  -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>" \
  -H "Content-Type: application/json" \
  -d '{
  "scenario": "Booking a hotel",
  "expectedOutcome": "Successfully booked",
  "userDescription": "Finds hotels in NYC",
  "finalized": true
}'

Delete a Golden

Remove a single golden from a dataset by its id. Only that golden is removed; the rest of the dataset is unchanged.

main.py

from deepeval.dataset import EvaluationDataset

dataset = EvaluationDataset()
dataset.pull(alias="YOUR-DATASET-ALIAS")

golden = dataset.goldens[0]
dataset.delete_golden(golden=golden)

# Or delete by id directly
dataset.delete_golden(golden="GOLDEN-ID")

index.ts

import { EvaluationDataset } from "deepeval";

const dataset = new EvaluationDataset();
await dataset.pull({ alias: "YOUR-DATASET-ALIAS" });

const golden = dataset.goldens[0];
await dataset.deleteGolden({ golden });

// Or delete by id directly
await dataset.deleteGolden({ golden: "GOLDEN-ID" });

DELETE/v1/datasets/{alias}/goldens/{goldenId}

curl -X DELETE "https://api.confident-ai.com/v1/datasets/{alias}/goldens/{goldenId}" \
  -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>"

Delete Dataset

Delete a dataset programmatically via the Evals API.

main.py

from deepeval.dataset import EvaluationDataset

dataset = EvaluationDataset()
dataset.delete(alias="YOUR-DATASET-ALIAS")

index.ts

import { EvaluationDataset } from "deepeval";

const dataset = new EvaluationDataset();
dataset.delete({ alias: "YOUR-DATASET-ALIAS" });

DELETE/v1/datasets/{alias}

curl -X DELETE "https://api.confident-ai.com/v1/datasets/{alias}" \
  -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>"

Switching Projects

You can push or manage datasets in any project by configuring a CONFIDENT_API_KEY.

For default usage, set CONFIDENT_API_KEY as an environment variable.
To target a specific project, pass a confident_api_key directly when creating the EvaluationDataset.

main.py

from deepeval.dataset import EvaluationDataset

dataset = EvaluationDataset(confident_api_key="confident_us...")

index.ts

import { EvaluationDataset } from "deepeval";

const dataset = new EvaluationDataset({ confidenApiKey: "confident_us..." });

When both are provided, the confident_api_key passed to EvaluationDataset always takes precedence over the environment variable.

Next Steps

Now that you know how to push goldens, learn how to pull them for evaluation.

Pull Datasets

Pull datasets locally to use them in code-driven evaluations.

Curating datasets across your team?Keep collaboration, annotation, and dataset quality organized at scaleBook a demo

Last updated on July 29, 2026