Pull Datasets

Pull datasets locally to use them for evaluation.

Overview

In the previous section, we learnt how to push and queue goldens via the Confident API. In this section, we will learn how to:

Pull single and multi-turn datasets for evaluation
Access custom column values from goldens
Parse multi-modal goldens (images) into an evaluatable format
Use the evals_iterator to run evals on single-turn datasets (Python only)
Run evals on Confident AI instead, with the Confident AI SDK

How it works

Code-driven evals follow a similar process to no-code evals, but you control the evaluation loop:

Pull dataset — fetch goldens from Confident AI using the Confident API
Invoke AI app — call your AI app with each golden's input
Create test cases — map golden fields and AI outputs into test cases
Run evaluation — execute metrics on your test cases and push results

Here's a visual representation of the data flow:

sequenceDiagram
    participant You as Your Code
    participant Platform as Confident AI
    participant AI as Your AI App
    participant Metrics as Local/Remote Metrics

    You->>Platform: Pull dataset (goldens)
    Platform-->>You: Return goldens

    loop For each golden in dataset
        You->>AI: Invoke with golden.input
        AI-->>You: Generate output
        You->>You: Create test case from golden + output
    end

    You->>Metrics: Run evaluation on test cases
    Metrics-->>You: Metric scores
    You->>Platform: Push test run results
    Platform-->>You: Test run created

The key difference from no-code evals is that you control the evaluation loop — pulling goldens, invoking your AI app, and constructing test cases all happen in your code.

You can pull datasets from any project by configuring a CONFIDENT_API_KEY.

For default usage, set CONFIDENT_API_KEY as an environment variable.
To target a specific project, pass a confident_api_key directly when creating the EvaluationDataset.

from deepeval.dataset import EvaluationDataset

dataset = EvaluationDataset(confident_api_key="confident_us...")
dataset.pull(alias="YOUR-DATASET-ALIAS")

When both are provided, the confident_api_key passed to EvaluationDataset always takes precedence over the environment variable.

Pull Goldens via the Confident API

Datasets are either single or multi-turn, and you should know that pulling a single-turn dataset will give you single-turn goldens, and vice versa.

Pulling goldens via the Confident API will only pull finalized goldens by default. Below is a single-turn dataset example (click here for multi-turn usage of datasets):

Pull goldens

First use the .pull() method:

main.py

from deepeval.dataset import EvaluationDataset

dataset = EvaluationDataset()
dataset.pull(alias="YOUR-DATASET-ALIAS")

print(dataset.goldens) # Check it's pulled correctly

Construct test cases

Then loop through your dataset of goldens to create a list of test cases:

main.py

from deepeval.dataset import EvaluationDataset
from deepeval.test_case import LLMTestCase

dataset = EvaluationDataset()
dataset.pull(alias="YOUR-DATASET-ALIAS")

for golden in dataset.goldens:
    test_case = LLMTestCase(
        input=golden.input,
        actual_output=llm_app(golden.input),
        # map any additional fields here
    )
    dataset.add_test_case(test_case)

For multi-turn datasets, you will create ConversationalTestCases instead:

main.py

from deepeval.test_case import ConversationalTestCase

for golden in dataset.goldens:
  test_case = simulate(golden) # simulate conversation
  dataset.add_test_case(test_case)

Run an evaluation
By calling .add_test_case() in the previous step, each time you run evaluate Confident AI will automatically associate any created test run with your dataset:
```
from deepeval import evaluate

evaluate(test_cases=dataset.test_cases, metrics=[...])
```

Pull goldens

First use the .pull() method:

index.ts

import { EvaluationDataset } from "deepeval";

const dataset = new EvaluationDataset();
dataset.pull({ alias: "YOUR-DATASET-ALIAS" });

console.log(dataset.goldens);

Construct test cases

Then loop through your dataset of goldens to create a list of test cases:

index.ts

import { EvaluationDataset, Golden, LLMTestCase } from "deepeval";

const dataset = new EvaluationDataset();
dataset.pull({ alias: "YOUR-DATASET-ALIAS" });

for (const golden of dataset.goldens as Golden[]) {
  const testCase = new LLMTestCase({
    input: golden.input,
    actualOutput: llmApp(golden.input),
    // map any additional fields here
  });

  dataset.addTestCase(testCase);
}

For multi-turn datasets, you will create ConversationalTestCases instead:

index.ts

import {
  ConversationalGolden,
  ConversationSimulator,
  EvaluationDataset,
} from "deepeval";

const dataset = new EvaluationDataset();
dataset.pull({ alias: "YOUR-DATASET-ALIAS" });

const simulator = new ConversationSimulator({ modelCallback: chatbotCallback });
const testCases = await simulator.simulate({
  conversationalGoldens: dataset.goldens as ConversationalGolden[],
});

for (const testCase of testCases) {
  dataset.addTestCase(testCase);
}

Run an evaluation

By calling .addTestCase() in the previous step, each time you run evaluate Confident AI will automatically associate any created test run with your dataset:

import { ConversationalTestCase, EvaluationDataset, evaluate } from "deepeval";

const dataset = new EvaluationDataset();
dataset.pull({ alias: "YOUR-DATASET-ALIAS" });

evaluate({
  conversationalTestCases: dataset.testCases as ConversationalTestCase[],
  metrics: [...],
});

Run Evals on Confident AI

The steps above run metrics locally with deepeval. To have Confident AI run them instead, start an evaluation of the dataset against a metric collection. Confident AI evaluates the dataset's finalized goldens, using the actual outputs stored on them or ones it generates for you. The SDK reads your project key from CONFIDENT_PROJ_API_KEY (more on project keys).

Pass the dataset's id, which you get back whenever you push to it, and the name of the metric collection to evaluate against. Create a metric collection if you haven't yet:

main.py

from confident_ai import ConfidentAI

client = ConfidentAI()

result = client.datasets.run_evaluation(
    dataset_id="<DATASET-ID>",
    metric_collection="Answer Quality",
)

index.ts

import { ConfidentAI } from "confident-ai";

const client = new ConfidentAI();

const result = await client.datasets.runEvaluation(
  "<DATASET-ID>",
  "Answer Quality",
);

POST/v2/datasets/{datasetId}/run

curl -X POST "https://api.confident-ai.com/v2/datasets/{datasetId}/run" \
  -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>" \
  -H "Content-Type: application/json" \
  -d '{
  "metricCollection": "Answer Quality"
}'

This evaluates the actual outputs already stored on the goldens. To generate them first, also pass an AI connection's id or a prompt's alias, never both. The call returns as soon as the test run is created, and the evaluation carries on in the background.

Using Custom Columns

If your dataset has custom columns, you can access them via the custom_column_key_values field on each golden:

main.py

from confident_ai import ConfidentAI

client = ConfidentAI()

dataset = client.dataset(dataset_id="<DATASET-ID>")
dataset.pull()

for golden in dataset.goldens:
    columns = golden.custom_column_key_values or {}
    difficulty = columns.get("difficulty")
    category = columns.get("category")

    actual_output = llm_app(golden.input, difficulty=difficulty)

index.ts

import { ConfidentAI } from "confident-ai";

const client = new ConfidentAI();

const dataset = client.dataset("<DATASET-ID>");
await dataset.pull();

for (const golden of dataset.goldens ?? []) {
  if (!("input" in golden)) continue;
  const difficulty = golden.customColumnKeyValues?.difficulty;
  const category = golden.customColumnKeyValues?.category;

  const actualOutput = await llmApp(golden.input, { difficulty });
}

Using Images

Images in golden text fields (such as input or scenario) come back as [CONFIDENT:IMAGE:<url>] markers, each holding a signed link to the image. parse_multimodal splits a field into strings and Media in order, so you can build the content parts your MLLM of choice expects. Here's how you'd build them for openai:

main.py

from confident_ai import ConfidentAI, Media, parse_multimodal

client = ConfidentAI()

dataset = client.dataset(dataset_id="<DATASET-ID>")
dataset.pull()

for golden in dataset.goldens:
    content = []
    for part in parse_multimodal(golden.input):
        if isinstance(part, Media):
            content.append({"type": "image_url", "image_url": {"url": part.url}})
        else:
            content.append({"type": "text", "text": part})

index.ts

import { ConfidentAI, Media, parseMultimodal } from "confident-ai";

const client = new ConfidentAI();

const dataset = client.dataset("<DATASET-ID>");
await dataset.pull();

for (const golden of dataset.goldens ?? []) {
  if (!("input" in golden)) continue;
  const content = parseMultimodal(golden.input).map((part) =>
    part instanceof Media
      ? { type: "image_url", image_url: { url: part.url } }
      : { type: "text", text: part },
  );
}

GET/v2/datasets/{datasetId}

curl -X GET "https://api.confident-ai.com/v2/datasets/{datasetId}" \
  -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>"

Golden text fields carry their images as [CONFIDENT:IMAGE:<url>] markers. Parse the markers to fetch each url, and use it soon after pulling, since the links are signed and expire.

Without an SDK, split each field on the markers yourself. Goldens written before the CONFIDENT marker may still carry [DEEPEVAL:IMAGE:<url>], so match both:

import re

IMAGE_MARKER = re.compile(r"\[(?:CONFIDENT|DEEPEVAL):IMAGE:(.*?)\]")

def parse_multimodal_string(text: str):
    parts, last_end = [], 0
    for match in IMAGE_MARKER.finditer(text):
        if match.start() > last_end:
            parts.append(text[last_end:match.start()])
        parts.append({"url": match.group(1)})
        last_end = match.end()
    if last_end < len(text):
        parts.append(text[last_end:])
    return parts

content = [
    {"type": "image_url", "image_url": {"url": part["url"]}}
    if isinstance(part, dict)
    else {"type": "text", "text": part}
    for part in parse_multimodal_string(golden["input"])
]

This example is for single-turn goldens. For multi-turn goldens, parse scenario and each turn's content the same way.

Using Evals Iterator

Typically, you would just provide your dataset as a list of test cases for evaluation. However, if you're running single-turn, end-to-end OR component-level evaluations and using deepeval in Python, you can use the evals_iterator() instead:

main.py

from deepeval.dataset import EvaluationDataset

dataset = EvaluationDataset()
dataset.pull(alias="YOUR-DATASET-ALIAS")

for golden in dataset.evals_iterator():
    llm_app(golden.input) # Replace with your LLM app

# Async version
# import asyncio
#
# for golden in dataset.evals_iterator():
#    task = asyncio.create_task(a_llm_app(golden.input))
#    dataset.evaluate(task)

You'll need to trace your LLM app to make this work. Read this section on running single-turn end-to-end evals with tracing to learn more.

Datasets in CI/CD

Using datasets in CI/CD follows the same pattern as local evaluation — pull your dataset, create test cases, and run evaluation. The only difference is that you use assert_test() instead of evaluate() to integrate with pytest:

test_llm_app.py

import pytest
from deepeval.test_case import LLMTestCase
from deepeval.dataset import EvaluationDataset
from deepeval.metrics import AnswerRelevancyMetric
from deepeval import assert_test

dataset = EvaluationDataset()
dataset.pull(alias="YOUR-DATASET-ALIAS")

for golden in dataset.goldens:
    test_case = LLMTestCase(input=golden.input, actual_output=llm_app(golden.input))
    dataset.add_test_case(test_case)

@pytest.mark.parametrize("test_case", dataset.test_cases)
def test_llm_app(test_case: LLMTestCase):
    assert_test(test_case, metrics=[AnswerRelevancyMetric()])

Then run with deepeval test run test_llm_app.py to execute your tests. Learn more about setting up automated testing in the Unit-Testing in CI/CD section.

Next Steps

Now that you're familiar with the full dataset lifecycle, time to dive into running evaluations end to end.

Single-Turn Evals

Run end-to-end or component-level evaluations on single-turn interactions.

Multi-Turn Evals

Evaluate conversational AI with multi-turn test cases.

Curating datasets across your team?Keep collaboration, annotation, and dataset quality organized at scaleBook a demo

Last updated on October 19, 2026