Dataset Ingestion Task
Turn matching traces and spans into dataset goldens automatically.
Overview
A dataset ingestion task turns production data into goldens: it continuously adds matching traces, spans, or threads to a dataset, so the cases your app actually sees end up in your evaluations. On the Workflows page, these tasks are listed under Dataset Ingestion.
Single-turn datasets ingest traces or spans; multi-turn datasets ingest threads as multi-turn goldens. The task runs every five minutes, and new goldens appear in the dataset shortly after the matching data arrives.
Create a Dataset Ingestion Task
- Navigate to Workflows and select Traces, Spans, or Threads
- Expand Dataset Ingestion in the panel beside the graph
- Click New ingestion task
- Configure the task in the side drawer (see fields below)
- Save the task
To start it from a classifier instead, click Add action under the classifier in the graph; the task then only ingests traces with the labels you pick (see Chain Tasks).
Fields
| Field | Description |
|---|---|
| Dataset | The dataset to ingest goldens into |
| Name | A name to identify the task |
| Description | Optional context about the task's purpose |
| Data Model | Trace or Span for a single-turn dataset; a multi-turn dataset always ingests threads |
| Runs after | Optional, traces only: a classifier and the labels a trace must get to be ingested |
| Golden Selection Logic | Which fields map onto the golden: input (always), actual output, expected output, retrieval context, context, tools called, and expected tools. Single-turn datasets only. |
| Filters | Narrow which items are ingested, with the same filters as the Observatory |
| Sample Rate | Fraction of matching items to ingest (0 to 1) |
| Max Goldens | Cap on how many goldens the task produces; leave empty for no limit |
| Input Transformer / Output Transformer | Optional transformers applied to inputs and outputs before they're ingested. Single-turn datasets only. |
Filters can use metric results: when a trace's or thread's evaluation lands after the task's sweep, tasks that filter on metrics check it again, so ingest every trace that failed Faithfulness works without chaining.
Each task row shows its name, data model, target dataset and golden count, and the workflows it belongs to. Use the toggle to enable or disable a task without deleting it. Click the row to edit it, or use its menu to manage its workflows or delete it.
Next Steps
Manage Datasets
Review, finalize, and edit the goldens a task ingests.
Observability Workflows
Triggers, the other tasks, and chaining tasks after a classifier.
Last updated on