Automate Dataset Management

Overview

This section covers how to programmatically manage goldens in datasets using the Evals API:

• Push single and multi-turn goldens to datasets
• Set finalized=True to make goldens available for evaluation, or finalized=False to queue for review
• Include custom column values when pushing goldens
• Delete datasets programmatically

Push Goldens

Push goldens to a dataset. If the dataset does not already exist, Confident AI will create it for you.

1
from deepeval.dataset import EvaluationDataset, Golden
2
3
goldens = [Golden(input="How tall is Mt. Everest?")]
4
dataset = EvaluationDataset(goldens=goldens)
5
6
# Push as finalized (ready for evaluation)
7
dataset.push(alias="YOUR-DATASET-ALIAS", finalized=True)
8
9
# Or push as unfinalized (queued for review)
10
dataset.push(alias="YOUR-DATASET-ALIAS", finalized=False)

1
from deepeval.dataset import EvaluationDataset, ConversationalGolden
2
from deepeval.test_case import Turn
3
4
goldens = [
5
ConversationalGolden(
6
scenario="Angry user asking for a refund.",
7
turns=[Turn(role="user", content="Give me my money!")]
8
)
9
]
10
dataset = EvaluationDataset(goldens=goldens)
11
12
dataset.push(alias="YOUR-DATASET-ALIAS", finalized=True)

Add Custom Columns

You can include custom column values when pushing goldens. Custom columns must already exist on the dataset, or Confident AI will create them for you.

1
from deepeval.dataset import Golden, ConversationalGolden
2
3
golden = Golden(
4
    input="How tall is Mt. Everest?",
5
    custom_column_key_values={"difficulty": "easy", "category": "geography"}
6
)
7
8
multiturn_golden = ConversationalGolden(
9
    scenario="User asking for a refund.",
10
    custom_column_key_values={"sentiment": "angry", "priority": "high"}
11
)

Delete Dataset

Delete a dataset programmatically via the Evals API.

1
from deepeval.dataset import EvaluationDataset
2
3
dataset = EvaluationDataset()
4
dataset.delete(alias="YOUR-DATASET-ALIAS")

Switching Projects

You can push or manage datasets in any project by configuring a CONFIDENT_API_KEY.

• For default usage, set CONFIDENT_API_KEY as an environment variable.
• To target a specific project, pass a confident_api_key directly when creating the EvaluationDataset.

1
from deepeval.dataset import EvaluationDataset
2
3
dataset = EvaluationDataset(confident_api_key="confident_us...")

When both are provided, the confident_api_key passed to EvaluationDataset always takes precedence over the environment variable.

Next Steps

Now that you know how to push goldens, learn how to pull them for evaluation.