Introducing Confident Voice: Watch the Launch Video Now

Automate Dataset Management

Programmatically push goldens to datasets via the Confident API.

Overview

This section covers how to programmatically manage goldens in datasets using the Confident API:

  • Push single and multi-turn goldens to datasets
  • Set finalized=True to make goldens available for evaluation, or finalized=False to queue for review
  • Include custom column values when pushing goldens
  • Update or delete an individual golden by its id
  • Delete datasets programmatically

Push Goldens

Push goldens to a dataset by its alias. If no dataset has that alias yet, Confident AI creates it for you.

For single-turn datasets, each golden has an input:

main.py
from confident_ai import ConfidentAI
from confident_ai.datasets import SingleTurnGolden

client = ConfidentAI()

result = client.datasets.push(
    alias="capitals",
    goldens=[
        SingleTurnGolden(
            input="What is the capital of France?"
        )
    ],
)

For multi-turn datasets, each golden has a scenario, and optionally the conversation's turns:

main.py
from confident_ai import ConfidentAI
from confident_ai.datasets import MultiTurnGolden

client = ConfidentAI()

result = client.datasets.push(
    alias="capitals",
    goldens=[
        MultiTurnGolden(
            scenario="A traveller wants to book a hotel in Paris."
        )
    ],
)
  • One kind per push: every golden in a push must be the same kind as the dataset.
  • Finalized or queued for review: finalized=True makes the goldens available for evaluation, and finalized=False queues them for review. It applies to every golden in the push.
  • Custom columns: set values with custom_column_key_values. Columns the dataset doesn't have yet are created for you.
  • Updating instead of adding: a golden carrying an id updates that golden in place, so you can pull a dataset, edit it and push it back without duplicating it. Leave id out to add a new golden.
  • Versions: leave version out to push to the latest version. It can't be set for a dataset that doesn't exist yet; see Versioning Datasets.

Every field of both golden kinds is listed under Golden in the SDK reference.

Images in goldens

Goldens can carry images. Put a Media where the image belongs in any of a golden's text fields, including a multi-turn golden's turns, and the SDK sends the image with the push:

main.py
from confident_ai import ConfidentAI, Media
from confident_ai.datasets import SingleTurnGolden

client = ConfidentAI()

receipt = Media.from_file("receipt.png")
client.datasets.push(
    alias="receipts",
    goldens=[SingleTurnGolden(input=f"What is the total on this receipt? {receipt}")],
)

Media.from_url sends a public link as it is, and from_file, from_bytes and from_base64 send the image itself. Over the API, put a [CONFIDENT:IMAGE:<id>] marker in the text and send the image under the same id in the golden's imagesMapping: { "url": "", "local": true, "mimeType": "image/png", "dataBase64": "<BASE64>" } for the image itself, with the bare base64 and no data: prefix, or { "url": "<URL>", "local": false } for a link.

Versioning Datasets

Datasets support immutable, named versions so you can pin evaluation runs to a specific snapshot of goldens.

  • Create a version to snapshot the current state of the dataset.
  • Push without specifying version to add goldens to the latest version (or unversioned, if the dataset has no versions yet).
  • Push with version=... to add goldens to a specific version.
  • Pull without version to read the latest version. Pull with version=... to read a specific version.
  • Get versions to list all snapshots, newest first.

Create a version

main.py
from confident_ai import ConfidentAI

client = ConfidentAI()

result = client.datasets.create_version(dataset_id="<DATASET-ID>")

The first call to create_version backfills every existing unversioned golden onto the new version. Subsequent calls snapshot all goldens from the previous version (with new IDs) and auto-increment the version number.

List versions

main.py
from confident_ai import ConfidentAI

client = ConfidentAI()

result = client.datasets.get_versions(dataset_id="<DATASET-ID>")

Push and pull a specific version

To push onto a specific version, pass version when you push goldens. To pull one, pass the same version along with the dataset's id, which every push returns:

main.py
from confident_ai import ConfidentAI

client = ConfidentAI()

result = client.datasets.pull(dataset_id="<DATASET-ID>")

finalized picks which goldens come back: the finalized ones by default, or with "false" the ones still awaiting review.

Update a Golden

Update a single golden in place by its id. Pull the dataset first so each golden carries a stable id, edit the fields you want, then send the update.

The golden's fields are replaced with the values you send. tags and custom columns are only changed when you include them.

For single-turn datasets:

main.py
from confident_ai import ConfidentAI
from confident_ai.datasets import SingleTurnGolden

client = ConfidentAI()

result = client.datasets.update_golden(
    dataset_id="<DATASET-ID>",
    golden_id="<GOLDEN-ID>",
    golden=SingleTurnGolden(
        input="What is the capital of France?"
    ),
)

For multi-turn datasets:

main.py
from confident_ai import ConfidentAI
from confident_ai.datasets import MultiTurnGolden

client = ConfidentAI()

result = client.datasets.update_golden(
    dataset_id="<DATASET-ID>",
    golden_id="<GOLDEN-ID>",
    golden=MultiTurnGolden(
        scenario="A traveller wants to book a hotel in Paris."
    ),
)

Delete a Golden

Remove a single golden from a dataset by its id. Only that golden is removed; the rest of the dataset is unchanged.

main.py
from confident_ai import ConfidentAI

client = ConfidentAI()

result = client.datasets.delete_golden(
    dataset_id="<DATASET-ID>",
    golden_id="<GOLDEN-ID>",
)

Delete Dataset

Delete a dataset programmatically via the Confident API.

main.py
from confident_ai import ConfidentAI

client = ConfidentAI()

result = client.datasets.delete(dataset_id="<DATASET-ID>")

Switching Projects

You can push or manage datasets in any project by using that project's API key.

  • For default usage, set CONFIDENT_PROJ_API_KEY as an environment variable.
  • To target a specific project, pass its key as project_api_key when creating the client.
main.py
from confident_ai import ConfidentAI

client = ConfidentAI(project_api_key="<PROJECT-API-KEY>")

A client keeps the key it was created with, so to work across several projects in one script, create one client per project.

Next Steps

Now that you know how to push goldens, learn how to pull them for evaluation.

Pull Datasets

Pull datasets locally to use them in code-driven evaluations.

Curating datasets across your team?Keep collaboration, annotation, and dataset quality organized at scaleBook a demo

Last updated on

Built byConfident AI