Pull Datasets

Pull datasets locally to use them for evaluation.

Overview

In the previous section, we learnt how to push and queue goldens via Confident's Evals API. In this section, we will learn how to:

Pull single and multi-turn datasets for evaluation
Access custom column values from goldens
Parse multi-modal goldens (images) into an evaluatable format
Use the evals_iterator to run evals on single-turn datasets (Python only)

How it works

Code-driven evals follow a similar process to no-code evals, but you control the evaluation loop:

Pull dataset — fetch goldens from Confident AI using the Evals API
Invoke AI app — call your AI app with each golden's input
Create test cases — map golden fields and AI outputs into test cases
Run evaluation — execute metrics on your test cases and push results

Here's a visual representation of the data flow:

sequenceDiagram
    participant You as Your Code
    participant Platform as Confident AI
    participant AI as Your AI App
    participant Metrics as Local/Remote Metrics

    You->>Platform: Pull dataset (goldens)
    Platform-->>You: Return goldens

    loop For each golden in dataset
        You->>AI: Invoke with golden.input
        AI-->>You: Generate output
        You->>You: Create test case from golden + output
    end

    You->>Metrics: Run evaluation on test cases
    Metrics-->>You: Metric scores
    You->>Platform: Push test run results
    Platform-->>You: Test run created

The key difference from no-code evals is that you control the evaluation loop — pulling goldens, invoking your AI app, and constructing test cases all happen in your code.

You can manage your datasets in any project by configuring a CONFIDENT_API_KEY.

For default usage, set CONFIDENT_API_KEY as an environment variable.
To target a specific project, pass a confident_api_key directly when creating the EvaluationDataset.

from deepeval.dataset import EvaluationDataset

dataset = EvaluationDataset(confident_api_key="confident_us...")
dataset.delete(alias="YOUR-DATASET-ALIAS")

When both are provided, the confident_api_key passed to EvaluationDataset always takes precedence over the environment variable.

Pull Goldens via Evals API

Datasets are either single or multi-turn, and you should know that pulling a single-turn dataset will give you single-turn goldens, and vice versa.

Pulling goldens via the Evals API will only pull finalized goldens by default. Below is a single-turn dataset example (click here for multi-turn usage of datasets):

Pull goldens

First use the .pull() method:

main.py

from deepeval.dataset import EvaluationDataset

dataset = EvaluationDataset()
dataset.pull(alias="YOUR-DATASET-ALIAS")

print(dataset.goldens) # Check it's pulled correctly

Construct test cases

Then loop through your dataset of goldens to create a list of test cases:

main.py

from deepeval.dataset import EvaluationDataset
from deepeval.test_case import LLMTestCase

dataset = EvaluationDataset()
dataset.pull(alias="YOUR-DATASET-ALIAS")

for golden in dataset.goldens:
    test_case = LLMTestCase(
        input=golden.input,
        actual_output=llm_app(golden.input),
        # map any additional fields here
    )
    dataset.add_test_case(test_case)

For multi-turn datasets, you will create ConversationalTestCases instead:

main.py

from deepeval.test_case import ConversationalTestCase

for golden in dataset.goldens:
  test_case = simulate(golden) # simulate conversation
  dataset.add_test_case(test_case)

Run an evaluation
By calling .add_test_case() in the previous step, each time you run evaluate Confident AI will automatically associate any created test run with your dataset:
```
from deepeval import evaluate

evaluate(test_cases=dataset.test_cases, metrics=[...])
```

Pull goldens

First use the .pull() method:

index.ts

import { EvaluationDataset } from "deepeval";

const dataset = new EvaluationDataset();
dataset.pull({ alias: "YOUR-DATASET-ALIAS" });

console.log(dataset.goldens);

Construct test cases

Then loop through your dataset of goldens to create a list of test cases:

index.ts

import { EvaluationDataset, Golden, LLMTestCase } from "deepeval";

const dataset = new EvaluationDataset();
dataset.pull({ alias: "YOUR-DATASET-ALIAS" });

for (const golden of dataset.goldens as Golden[]) {
  const testCase = new LLMTestCase({
    input: golden.input,
    actualOutput: llmApp(golden.input),
    // map any additional fields here
  });

  dataset.addTestCase(testCase);
}

For multi-turn datasets, you will create ConversationalTestCases instead:

index.ts

import {
  ConversationalGolden,
  ConversationSimulator,
  EvaluationDataset,
} from "deepeval";

const dataset = new EvaluationDataset();
dataset.pull({ alias: "YOUR-DATASET-ALIAS" });

const simulator = new ConversationSimulator({ modelCallback: chatbotCallback });
const testCases = await simulator.simulate({
  conversationalGoldens: dataset.goldens as ConversationalGolden[],
});

for (const testCase of testCases) {
  dataset.addTestCase(testCase);
}

Run an evaluation

By calling .addTestCase() in the previous step, each time you run evaluate Confident AI will automatically associate any created test run with your dataset:

import { ConversationalTestCase, EvaluationDataset, evaluate } from "deepeval";

const dataset = new EvaluationDataset();
dataset.pull({ alias: "YOUR-DATASET-ALIAS" });

evaluate({
  conversationalTestCases: dataset.testCases as ConversationalTestCase[],
  metrics: [...],
});

Pull goldens

First, pull goldens using the /v1/datasets endpoint.

GET/v1/datasets/{alias}

curl -X GET "https://api.confident-ai.com/v1/datasets/{alias}" \
  -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>"

Construct test cases
Construct a JSON array of test cases from the goldens you pulled, preserving the golden fields.
[ { "input": "How tall is Mount Everest?", // Replace with your LLM app output "actualOutput": "Mount Everest is 9K meters tall." } ]
Click here to see the parameters for creating a single-turn test case
Parameters of `LLMTestCase`

input: string

actualOutput: string

name: string

expectedOutput: string

retrievalContext: list of strings

context: list of strings

toolsCalled: list of ToolCall

expectedTools: list of ToolCall
[ { "scenario": "User asking about Mount Everest height.", "turns": [ { "role": "user", "content": "How tall is Mount Everest?" }, { "role": "assistant", "content": "Mount Everest is 9K meters tall." } // Replace with your LLM app outputs ], } ]
Click here to see the parameters for creating a multi-turn test case
Parameters of `ConversationalTestCase`

turns: list of Turn

scenario: string

name: string

expectedOutput: string

userDescription: string

chatbotRole: string

Create metric collection

Create a metric collection through v1/metric-collections.

POST/v1/metric-collections

curl -X POST "https://api.confident-ai.com/v1/metric-collections" \
  -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>" \
  -H "Content-Type: application/json" \
  -d '{
  "name": "string",
  "multiTurn": true,
  "metricSettings": [
    {
      "metric": {
        "name": "string"
      },
      "activated": true,
      "threshold": 0.5,
      "includeReason": true,
      "strictMode": false,
      "sampleRate": 1
    }
  ]
}'

Run an evaluation

Run an evaluation using the test cases you constructed and metric collection you created using /v1/evaluate.

POST/v1/evaluate

curl -X POST "https://api.confident-ai.com/v1/evaluate" \
  -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>" \
  -H "Content-Type: application/json" \
  -d '{
  "metricCollection": "string",
  "llmTestCases": [
    {
      "input": "string",
      "actualOutput": "string",
      "name": "string",
      "expectedOutput": "string",
      "retrievalContext": [
        "string"
      ],
      "context": [
        "string"
      ],
      "toolsCalled": [
        {
          "name": "string",
          "description": "string",
          "inputParameters": {},
          "output": "string",
          "reasoning": "string"
        }
      ],
      "expectedTools": [
        {
          "name": "string",
          "description": "string",
          "inputParameters": {},
          "output": "string",
          "reasoning": "string"
        }
      ]
    }
  ],
  "conversationalTestCases": [
    {
      "turns": [
        {
          "role": "user",
          "content": "string",
          "userId": "string",
          "retrievalContext": [
            "string"
          ],
          "toolsCalled": [
            {
              "name": null,
              "description": null,
              "inputParameters": null,
              "output": null,
              "reasoning": null
            }
          ]
        }
      ],
      "scenario": "string",
      "name": "string",
      "expectedOutcome": "string",
      "userDescription": "string",
      "chatbotRole": "string"
    }
  ],
  "hyperparameters": {},
  "identifier": "string"
}'

Using Custom Columns

If your dataset has custom columns, you can access them via the custom_column_key_values field on each golden:

main.py

from deepeval.dataset import EvaluationDataset
from deepeval.test_case import LLMTestCase

dataset = EvaluationDataset()
dataset.pull(alias="YOUR-DATASET-ALIAS")

for golden in dataset.goldens:
    # Access custom column values
    difficulty = golden.custom_column_key_values.get("difficulty")
    category = golden.custom_column_key_values.get("category")

    # Use them in your test case or LLM app invocation
    test_case = LLMTestCase(
        input=golden.input,
        actual_output=llm_app(golden.input, difficulty=difficulty),
    )
    dataset.add_test_case(test_case)

index.ts

import { EvaluationDataset, Golden, LLMTestCase } from "deepeval";

const dataset = new EvaluationDataset();
await dataset.pull({ alias: "YOUR-DATASET-ALIAS" });

for (const golden of dataset.goldens as Golden[]) {
  // Access custom column values
  const difficulty = golden.customColumnKeyValues?.difficulty;
  const category = golden.customColumnKeyValues?.category;

  // Use them in your test case or LLM app invocation
  const testCase = new LLMTestCase({
    input: golden.input,
    actualOutput: await llmApp(golden.input, { difficulty }),
  });
  dataset.addTestCase(testCase);
}

Using Images

Any (list of) golden text fields (such as input, scenario, etc.) that contains an image will be in the format of [DEEPEVAL:IMAGE:url]. The url inside the [DEEPEVAL:IMAGE:url] format is a public url that can be accessed by anyone.

For goldens containing images, here you can parse and use it accordingly as follows:

The deepeval python SDK offers a utility method called convert_to_multi_modal_array. This method is useful for converting a string containing images in the [DEEPEVAL:IMAGE:url] format into a list of strings and MLLMImage items.

from deepeval.dataset import EvaluationDataset
from deepeval.utils import convert_to_multi_modal_array

dataset = EvaluationDataset()
dataset.pull(alias="My Evals Dataset")

for golden in dataset.goldens:
    multimodal_array = convert_to_multi_modal_array(golden.input)

The multimodal_array here is a list containing strings and MLLMImages, you can loop over this array to construct a messages array with images to pass to your MLLM. Here's an example showing how to construct messages array for openai:

messages = []
for element in multimodal_array:
    if isinstance(element, str):
        messages.append({"type": "text", "text": element})
    elif isinstance(element, MLLMImage):
      if element.url:
          messages.append(
              {
                  "type": "image_url",
                  "image_url": {"url": element.url},
              }
          )

You can use a custom method to parse strings with images in [DEEPEVAL:IMAGE:url] format to convert them into an array of strings and URLs

const parseMultimodalString = (s: string) => {
  const PATTERN = /\[DEEPEVAL:IMAGE:(.*?)\]/g;

  const result = [];
  let lastEnd = 0;
  let match;

  while ((match = PATTERN.exec(s)) !== null) {
    const start = match.index;
    const end = PATTERN.lastIndex;

    if (start > lastEnd) {
      result.push(s.slice(lastEnd, start));
    }

    const imageUrl = match[1];
    result.push({ url: imageUrl });

    lastEnd = end;
  }

  if (lastEnd < s.length) {
    result.push(s.slice(lastEnd));
  }

  return result;
}

You can now use this method to get multimodalArray and construct messages array to pass it to your MLLM. Here's an example on how to use it to construct openai format messages:

const multimodalArray = parseMultimodalString(golden.input);

const messages = [];

for (const element of multimodalArray) {
  if (typeof element === "string") {
    messages.push({
      type: "text",
      text: element,
    });
  } else if (element.url) {
    messages.push({
      type: "image_url",
      image_url: { url: element.url },
    });
  }
}

GET/v1/datasets/{alias}

curl -X GET "https://api.confident-ai.com/v1/datasets/{alias}" \
  -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>"

The dataset pulled here has images inside golden fields with the pattern [DEEPEVAL:IMAGE:url]. Please parse the fields to fetch the public url and use it as necessary.

You can use a custom method to parse strings with images in [DEEPEVAL:IMAGE:url] format to convert them into an array of strings and URLs

def parse_multimodal_string(s: str):
    PATTERN = r"\[DEEPEVAL:IMAGE:(.*?)\]"
    matches = list(re.finditer(pattern, s))

    result = []
    last_end = 0
    for m in matches:
        start, end = m.span()
        if start > last_end:
          result.append(s[last_end:start])

        image_url = m.group(1)

        result.append({"url": image_url})
        last_end = end

    if last_end < len(s):
        result.append(s[last_end:])
        return result

You can use this method to get multimodal_array and construct messages array to pass it to your MLLM. Here's an example on how to use it:

multimodal_array = parse_multimodal_string(golden.input)

messages = []
for element in multimodal_array:
    if isinstance(element, str):
        messages.append({"type": "text", "text": element})
    else:
        if element.get("url") is not None:
          messages.append(
              {
                  "type": "image_url",
                  "image_url": {"url": element.url},
              }
          )

deepeval's native models like GPTModel, GeminiModel automatically parse the images inside the [DEEPEVAL:IMAGE:url] formats for you, you can simply pass any golden field with images inside the .generate() or .a_generate() methods and deepeval automatically handles images for you internally!

from deepeval.models import GPTModel
from deepeval.dataset import EvaluationDataset

dataset = EvaluationDataset()
dataset.pull(alias="My Evals Dataset")

model = GPTModel(model="gpt-5.2")

for golden in dataset.goldens:
  print(model.generate(golden.input)) # Images are automatically handled by deepeval

Using Evals Iterator

Typically, you would just provide your dataset as a list of test cases for evaluation. However, if you're running single-turn, end-to-end OR component-level evaluations and using deepeval in Python, you can use the evals_iterator() instead:

main.py

from deepeval.dataset import EvaluationDataset

dataset = EvaluationDataset()
dataset.pull(alias="YOUR-DATASET-ALIAS")

for golden in dataset.evals_iterator():
    llm_app(golden.input) # Replace with your LLM app

# Async version
# import asyncio
#
# for golden in dataset.evals_iterator():
#    task = asyncio.create_task(a_llm_app(golden.input))
#    dataset.evaluate(task)

You'll need to trace your LLM app to make this work. Read this section on running single-turn end-to-end evals with tracing to learn more.

Datasets in CI/CD

Using datasets in CI/CD follows the same pattern as local evaluation — pull your dataset, create test cases, and run evaluation. The only difference is that you use assert_test() instead of evaluate() to integrate with pytest:

test_llm_app.py

import pytest
from deepeval.test_case import LLMTestCase
from deepeval.dataset import EvaluationDataset
from deepeval.metrics import AnswerRelevancyMetric
from deepeval import assert_test

dataset = EvaluationDataset()
dataset.pull(alias="YOUR-DATASET-ALIAS")

for golden in dataset.goldens:
    test_case = LLMTestCase(input=golden.input, actual_output=llm_app(golden.input))
    dataset.add_test_case(test_case)

@pytest.mark.parametrize("test_case", dataset.test_cases)
def test_llm_app(test_case: LLMTestCase):
    assert_test(test_case, metrics=[AnswerRelevancyMetric()])

Then run with deepeval test run test_llm_app.py to execute your tests. Learn more about setting up automated testing in the Unit-Testing in CI/CD section.

Next Steps

Now that you're familiar with the full dataset lifecycle, time to dive into running evaluations end to end.

Single-Turn Evals

Run end-to-end or component-level evaluations on single-turn interactions.

Multi-Turn Evals

Evaluate conversational AI with multi-turn test cases.

Curating datasets across your team?Keep collaboration, annotation, and dataset quality organized at scaleBook a demo