Automate Dataset Management
Programmatically push goldens to datasets via the Confident API.
Overview
This section covers how to programmatically manage goldens in datasets using the Confident API:
- Push single and multi-turn goldens to datasets
- Set
finalized=Trueto make goldens available for evaluation, orfinalized=Falseto queue for review - Include custom column values when pushing goldens
- Update or delete an individual golden by its
id - Delete datasets programmatically
Push Goldens
Push goldens to a dataset by its alias. If no dataset has that alias yet, Confident AI creates it for you.
For single-turn datasets, each golden has an input:
from confident_ai import ConfidentAI
from confident_ai.datasets import SingleTurnGolden
client = ConfidentAI()
result = client.datasets.push(
alias="capitals",
goldens=[
SingleTurnGolden(
input="What is the capital of France?"
)
],
)For multi-turn datasets, each golden has a scenario, and optionally the conversation's turns:
from confident_ai import ConfidentAI
from confident_ai.datasets import MultiTurnGolden
client = ConfidentAI()
result = client.datasets.push(
alias="capitals",
goldens=[
MultiTurnGolden(
scenario="A traveller wants to book a hotel in Paris."
)
],
)- One kind per push: every golden in a push must be the same kind as the dataset.
- Finalized or queued for review:
finalized=Truemakes the goldens available for evaluation, andfinalized=Falsequeues them for review. It applies to every golden in the push. - Custom columns: set values with
custom_column_key_values. Columns the dataset doesn't have yet are created for you. - Updating instead of adding: a golden carrying an
idupdates that golden in place, so you can pull a dataset, edit it and push it back without duplicating it. Leaveidout to add a new golden. - Versions: leave
versionout to push to the latest version. It can't be set for a dataset that doesn't exist yet; see Versioning Datasets.
Every field of both golden kinds is listed under Golden in the SDK reference.
Images in goldens
Goldens can carry images. Put a Media where the image belongs in any of a golden's text fields, including a multi-turn golden's turns, and the SDK sends the image with the push:
from confident_ai import ConfidentAI, Media
from confident_ai.datasets import SingleTurnGolden
client = ConfidentAI()
receipt = Media.from_file("receipt.png")
client.datasets.push(
alias="receipts",
goldens=[SingleTurnGolden(input=f"What is the total on this receipt? {receipt}")],
)import { ConfidentAI, Media } from "confident-ai";
const client = new ConfidentAI();
const receipt = Media.fromFile("receipt.png");
await client.datasets.push("receipts", [
{ input: `What is the total on this receipt? ${receipt}` },
]);Media.from_url sends a public link as it is, and from_file, from_bytes and from_base64 send the image itself. Over the API, put a [CONFIDENT:IMAGE:<id>] marker in the text and send the image under the same id in the golden's imagesMapping: { "url": "", "local": true, "mimeType": "image/png", "dataBase64": "<BASE64>" } for the image itself, with the bare base64 and no data: prefix, or { "url": "<URL>", "local": false } for a link.
Versioning Datasets
Datasets support immutable, named versions so you can pin evaluation runs to a specific snapshot of goldens.
- Create a version to snapshot the current state of the dataset.
- Push without specifying
versionto add goldens to the latest version (or unversioned, if the dataset has no versions yet). - Push with
version=...to add goldens to a specific version. - Pull without
versionto read the latest version. Pull withversion=...to read a specific version. - Get versions to list all snapshots, newest first.
Create a version
from confident_ai import ConfidentAI
client = ConfidentAI()
result = client.datasets.create_version(dataset_id="<DATASET-ID>")The first call to create_version backfills every existing unversioned golden onto the new version. Subsequent calls snapshot all goldens from the previous version (with new IDs) and auto-increment the version number.
List versions
from confident_ai import ConfidentAI
client = ConfidentAI()
result = client.datasets.get_versions(dataset_id="<DATASET-ID>")Push and pull a specific version
To push onto a specific version, pass version when you push goldens. To pull one, pass the same version along with the dataset's id, which every push returns:
from confident_ai import ConfidentAI
client = ConfidentAI()
result = client.datasets.pull(dataset_id="<DATASET-ID>")finalized picks which goldens come back: the finalized ones by default, or with "false" the ones still awaiting review.
Update a Golden
Update a single golden in place by its id. Pull the dataset first so each golden carries a stable id, edit the fields you want, then send the update.
The golden's fields are replaced with the values you send. tags and custom columns are only changed when you include them.
For single-turn datasets:
from confident_ai import ConfidentAI
from confident_ai.datasets import SingleTurnGolden
client = ConfidentAI()
result = client.datasets.update_golden(
dataset_id="<DATASET-ID>",
golden_id="<GOLDEN-ID>",
golden=SingleTurnGolden(
input="What is the capital of France?"
),
)For multi-turn datasets:
from confident_ai import ConfidentAI
from confident_ai.datasets import MultiTurnGolden
client = ConfidentAI()
result = client.datasets.update_golden(
dataset_id="<DATASET-ID>",
golden_id="<GOLDEN-ID>",
golden=MultiTurnGolden(
scenario="A traveller wants to book a hotel in Paris."
),
)Delete a Golden
Remove a single golden from a dataset by its id. Only that golden is removed; the rest of the dataset is unchanged.
from confident_ai import ConfidentAI
client = ConfidentAI()
result = client.datasets.delete_golden(
dataset_id="<DATASET-ID>",
golden_id="<GOLDEN-ID>",
)Delete Dataset
Delete a dataset programmatically via the Confident API.
from confident_ai import ConfidentAI
client = ConfidentAI()
result = client.datasets.delete(dataset_id="<DATASET-ID>")Switching Projects
You can push or manage datasets in any project by using that project's API key.
- For default usage, set
CONFIDENT_PROJ_API_KEYas an environment variable. - To target a specific project, pass its key as
project_api_keywhen creating the client.
from confident_ai import ConfidentAI
client = ConfidentAI(project_api_key="<PROJECT-API-KEY>")import { ConfidentAI } from "confident-ai";
const client = new ConfidentAI({ projectApiKey: "<PROJECT-API-KEY>" });A client keeps the key it was created with, so to work across several projects in one script, create one client per project.
Next Steps
Now that you know how to push goldens, learn how to pull them for evaluation.
Pull Datasets
Pull datasets locally to use them in code-driven evaluations.
Last updated on