Pull Datasets

Overview

In the previous section, we learnt how to push and queue goldens via Confident’s Evals API. In this section, we will learn how to:

• Pull single and multi-turn datasets for evaluation
• Access custom column values from goldens
• Parse multi-modal goldens (images) into an evaluatable format
• Use the evals_iterator to run evals on single-turn datasets (Python only)

How it works

Code-driven evals follow a similar process to no-code evals, but you control the evaluation loop:

1. Pull dataset — fetch goldens from Confident AI using the Evals API
2. Invoke AI app — call your AI app with each golden’s input
3. Create test cases — map golden fields and AI outputs into test cases
4. Run evaluation — execute metrics on your test cases and push results

Here’s a visual representation of the data flow:

The key difference from no-code evals is that you control the evaluation loop — pulling goldens, invoking your AI app, and constructing test cases all happen in your code.

1
from deepeval.dataset import EvaluationDataset
2
3
dataset = EvaluationDataset(confident_api_key="confident_us...")
4
dataset.delete(alias="YOUR-DATASET-ALIAS")

Pull Goldens via Evals API

Datasets are either single or multi-turn, and you should know that pulling a single-turn dataset will give you single-turn goldens, and vice versa.

Pulling goldens via the Evals API will only pull finalized goldens by default. Below is a single-turn dataset example (click here for multi-turn usage of datasets):

1
from deepeval.dataset import EvaluationDataset
2
3
dataset = EvaluationDataset()
4
dataset.pull(alias="YOUR-DATASET-ALIAS")
5
6
print(dataset.goldens) # Check it's pulled correctly

1
from deepeval.dataset import EvaluationDataset
2
from deepeval.test_case import LLMTestCase
3
4
dataset = EvaluationDataset()
5
dataset.pull(alias="YOUR-DATASET-ALIAS")
6
7
for golden in dataset.goldens:
8
    test_case = LLMTestCase(
9
        input=golden.input,
10
        actual_output=llm_app(golden.input),
11
        # map any additional fields here
12
    )
13
    dataset.add_test_case(test_case)

1
from deepeval.test_case import ConversationalTestCase
2
3
for golden in dataset.goldens:
4
  test_case = simulate(golden) # simulate conversation
5
  dataset.add_test_case(test_case)

1
from deepeval import evaluate
2
3
evaluate(test_cases=dataset.test_cases, metrics=[...])

Using Custom Columns

If your dataset has custom columns, you can access them via the custom_column_key_values field on each golden:

1
from deepeval.dataset import EvaluationDataset
2
from deepeval.test_case import LLMTestCase
3
4
dataset = EvaluationDataset()
5
dataset.pull(alias="YOUR-DATASET-ALIAS")
6
7
for golden in dataset.goldens:
8
    # Access custom column values
9
    difficulty = golden.custom_column_key_values.get("difficulty")
10
    category = golden.custom_column_key_values.get("category")
11
12
    # Use them in your test case or LLM app invocation
13
    test_case = LLMTestCase(
14
        input=golden.input,
15
        actual_output=llm_app(golden.input, difficulty=difficulty),
16
    )
17
    dataset.add_test_case(test_case)

Using Images

Any (list of) golden text fields (such as input, scenario, etc.) that contains an image will be in the format of [DEEPEVAL:IMAGE:url]. The url inside the [DEEPEVAL:IMAGE:url] format is a public url that can be accessed by anyone.

For goldens containing images, here you can parse and use it accordingly as follows:

1
from deepeval.dataset import EvaluationDataset
2
from deepeval.utils import convert_to_multi_modal_array
3
4
dataset = EvaluationDataset()
5
dataset.pull(alias="My Evals Dataset")
6
7
for golden in dataset.goldens:
8
    multimodal_array = convert_to_multi_modal_array(golden.input)

1
messages = []
2
for element in multimodal_array:
3
    if isinstance(element, str):
4
        messages.append({"type": "text", "text": element})
5
    elif isinstance(element, MLLMImage):
6
      if element.url:
7
          messages.append(
8
              {
9
                  "type": "image_url",
10
                  "image_url": {"url": element.url},
11
              }
12
          )

1
from deepeval.models import GPTModel
2
from deepeval.dataset import EvaluationDataset
3
4
dataset = EvaluationDataset()
5
dataset.pull(alias="My Evals Dataset")
6
7
model = GPTModel(model="gpt-5.2")
8
9
for golden in dataset.goldens:
10
  print(model.generate(golden.input)) # Images are automatically handled by deepeval

Using Evals Iterator

Typically, you would just provide your dataset as a list of test cases for evaluation. However, if you’re running single-turn, end-to-end OR component-level evaluations and using deepeval in Python, you can use the evals_iterator() instead:

1
from deepeval.dataset import EvaluationDataset
2
3
dataset = EvaluationDataset()
4
dataset.pull(alias="YOUR-DATASET-ALIAS")
5
6
for golden in dataset.evals_iterator():
7
    llm_app(golden.input) # Replace with your LLM app
8
9
# Async version
10
# import asyncio
11
#
12
# for golden in dataset.evals_iterator():
13
#    task = asyncio.create_task(a_llm_app(golden.input))
14
#    dataset.evaluate(task)

You’ll need to trace your LLM app to make this work. Read this section on running single-turn end-to-end evals with tracing to learn more.

Datasets in CI/CD

Using datasets in CI/CD follows the same pattern as local evaluation — pull your dataset, create test cases, and run evaluation. The only difference is that you use assert_test() instead of evaluate() to integrate with pytest:

1
import pytest
2
from deepeval.test_case import LLMTestCase
3
from deepeval.dataset import EvaluationDataset
4
from deepeval.metrics import AnswerRelevancyMetric
5
from deepeval import assert_test
6
7
dataset = EvaluationDataset()
8
dataset.pull(alias="YOUR-DATASET-ALIAS")
9
10
for golden in dataset.goldens:
11
    test_case = LLMTestCase(input=golden.input, actual_output=llm_app(golden.input))
12
    dataset.add_test_case(test_case)
13
14
@pytest.mark.parametrize("test_case", dataset.test_cases)
15
def test_llm_app(test_case: LLMTestCase):
16
    assert_test(test_case, metrics=[AnswerRelevancyMetric()])

Then run with deepeval test run test_llm_app.py to execute your tests. Learn more about setting up automated testing in the Unit-Testing in CI/CD section.

Next Steps

Now that you’re familiar with the full dataset lifecycle, time to dive into running evaluations end to end.