Pull Prompts

Learn how to test and use prompts in your LLM app

Overview

You can pull a prompt version from Confident AI like how you would pull a dataset. It works by:

Providing Confident AI with the alias and optionally version of the prompt you wish to retrieve
Confident AI will provide the non-interpolated version of the prompt
You will then interpolate the variables in code

You should pull prompts once and save it in memory instead of pulling it everytime you need to use it.

Pull Prompt By Version

Pull prompt with alias
Pull your prompt version by providing the alias you've defined:
from deepeval.prompt import Prompt prompt = Prompt(alias="YOUR-PROMPT-ALIAS") prompt.pull(version="latest")
By passing latest instead of a specific version, Confident AI will return the most recent version of your prompt. However, you can also specify the version to override this behavior.
prompt.pull(version="00.00.01")
import { Prompt } from "deepeval"; const prompt = new Prompt({ alias: "YOUR-PROMPT-ALIAS" }); await prompt.pull({ version: "latest" });
By passing latest instead of a specific version, Confident AI will return the most recent version of your prompt. However, you can also specify the version to override this behavior.
await prompt.pull({ version: "00.00.01" });
GET/v1/prompts/{alias}/versions/{version}
curl -X GET "https://api.confident-ai.com/v1/prompts/{alias}/versions/{version}" \ -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>"

Interpolate variables

Now that you have your prompt template, interpolate any dynamic variables you may have defined in your prompt version.

interpolated_prompt = prompt.interpolate(name="Joe")

const interpolatedPrompt = prompt.interpolate({ name: "Joe" });

# Interpolation is done client-side after pulling the prompt
# The API response includes an "interpolationType" field indicating the format:
# - "FSTRING": Use {{ variable }} format (default)
# - "HANDLEBARS": Use {{variable}} format
# Replace variables manually based on the interpolationType in your application code

For example, if this is your prompt version:

{
  "role": "system",
  "content": "You are a helpful assistant called {{ name }}. Speak normally like a human."
}

And your interpolation type is {{ variable }}, interpolating the name (e.g. "Joe") would give you this prompt that is ready for use:

{
  "role": "system",
  "content": "You are a helpful assistant called Joe. Speak normally like a human."
}

You are a helpful assistant called {{ name }}. Speak normally like a human.

And your interpolation type is {{ variable }}, interpolating the name (e.g. “Joe”) would give you this prompt that is ready for use:

You are a helpful assistant called Joe. Speak normally like a human.

interpolated_prompt = prompt.interpolate(name="Joe")

const interpolatedPrompt = prompt.interpolate({ name: "Joe" });

# Interpolation is done client-side after pulling the prompt
# The API response includes an "interpolationType" field indicating the format:
# - "FSTRING": Use {{ variable }} format (default)
# - "HANDLEBARS": Use {{variable}} format
# Replace variables manually based on the interpolationType in your application code

And if you don't have any variables, you must still use the interpolate() method to create a copy of your prompt template to be used in your LLM application.

Use interpolated prompt

By now you should have an interpolated prompt version, for example:

{
  "role": "system",
  "content": "You are a helpful assistant called Joe. Speak normally like a human."
}

Which you can use to generate text from your LLM provider of choice. Here are some examples with OpenAI:

main.py

from deepeval.prompt import Prompt
from openai import OpenAI

prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
prompt.pull()
interpolated_prompt = prompt.interpolate() # interpolate prompt

response = OpenAI().chat.completions.create(
    model="gpt-4o-mini",
    messages=interpolated_prompt
)

print(response.choices[0].message.content)

index.ts

import { Prompt } from "deepeval";
import { OpenAI } from "openai";

const prompt = new Prompt({ alias: "YOUR-PROMPT-ALIAS" });
await prompt.pull();
const interpolatedPrompt = prompt.interpolate(); // interpolate prompt

const openai = new OpenAI();
const response = await openai.chat.completions.create({
  model: "gpt-4o",
  messages: interpolatedPrompt as any[],
});

console.log(response.choices[0].message.content);

First, pull the prompt from Confident AI:

GET/v1/prompts

curl -X GET "https://api.confident-ai.com/v1/prompts" \
  -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>"

Then, interpolate the variables and use the interpolated prompt with OpenAI:

curl -X POST "https://api.openai.com/v1/chat/completions" \
  -H "Authorization: Bearer YOUR_OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4.1",
    "messages": <YOUR-INTERPOLATED-PROMPT>
  }'

You are a helpful assistant called Joe. Speak normally like a human.

Which you can use to generate text from your LLM provider of choice. Here are some examples with OpenAI:

main.py

from deepeval.prompt import Prompt
from openai import OpenAI

prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
prompt.pull()
interpolated_prompt = prompt.interpolate() # interpolate prompt

response = OpenAI().chat.completions.create(
    model="gpt-4o-mini",
    messages={"role": "system", "content": interpolated_prompt}
)

print(response.choices[0].message.content)

index.ts

import { Prompt } from "deepeval";
import { OpenAI } from "openai";

const prompt = new Prompt({ alias: "YOUR-PROMPT-ALIAS" });
await prompt.pull();
const interpolatedPrompt = prompt.interpolate(); // interpolate prompt

const openai = new OpenAI();
const response = await openai.chat.completions.create({
  model: "gpt-4o-mini",
  messages: [{ role: "system", content: interpolatedPrompt }],
});

console.log(response.choices[0].message.content);

First, pull the prompt from Confident AI:

GET/v1/prompts

curl -X GET "https://api.confident-ai.com/v1/prompts" \
  -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>"

Then, interpolate the variables and use the interpolated prompt with OpenAI:

curl -X POST "https://api.openai.com/v1/chat/completions" \
  -H "Authorization: Bearer YOUR_OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4.1",
    "messages": <YOUR-INTERPOLATED-PROMPT>
  }'

You can also fetch all the versions associated with a prompt as shown below:

from deepeval.prompt import Prompt

prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
commits = prompt._get_versions()

import { Prompt } from "deepeval";

const prompt = new Prompt({ alias: "YOUR-PROMPT-ALIAS" });
const commits = await prompt.getVersions();

GET/v1/prompts/{alias}/versions

curl -X GET "https://api.confident-ai.com/v1/prompts/{alias}/versions" \
  -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>"

Pull Prompts By Label

You can also pull specific versions of prompts using the label you assign to the versions on the platform.

from deepeval.prompt import Prompt

prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
prompt.pull(label="staging")

import { Prompt } from "deepeval";

const prompt = new Prompt({ alias: "YOUR-PROMPT-ALIAS" });
await prompt.pull({ label: "staging" });

GET/v1/prompts/{alias}/labels/{label}

curl -X GET "https://api.confident-ai.com/v1/prompts/{alias}/labels/{label}" \
  -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>"

You must manually label each prompt version before pulling it. Click here to learn how to do so.

Pull Prompts By Commit

You can also pull specific snapshots of prompts using it's alias and the commit hash.

from deepeval.prompt import Prompt

prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
prompt.pull()

import { Prompt } from "deepeval";

const prompt = new Prompt({ alias: "YOUR-PROMPT-ALIAS" });
await prompt.pull();

GET/v1/prompts/{alias}/commits/{hash}

curl -X GET "https://api.confident-ai.com/v1/prompts/{alias}/commits/{hash}" \
  -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>"

You can also fetch all the commits associated with a prompt as shown below:

from deepeval.prompt import Prompt

prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
commits = prompt._get_commits()

import { Prompt } from "deepeval";

const prompt = new Prompt({ alias: "YOUR-PROMPT-ALIAS" });
const commits = await prompt.getCommits();

GET/v1/prompts/{alias}/commits

curl -X GET "https://api.confident-ai.com/v1/prompts/{alias}/commits" \
  -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>"

Pull Prompts By Branch

You can also pull specific snapshots of prompts using it's alias and the branch name.

from deepeval.prompt import Prompt

prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
prompt.pull(branch="MY-BRANCH-NAME")

import { Prompt } from "deepeval";

const prompt = new Prompt({ alias: "YOUR-PROMPT-ALIAS" });
await prompt.pull({ branch: "MY-BRANCH-NAME"});

GET/v1/prompts/{alias}/commits/{hash}

curl -X GET "https://api.confident-ai.com/v1/prompts/{alias}/commits/{hash}" \
  -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>"

You can pass the branch query parameter to specify the branch you want to pull the latest commit from.

Branch Operations

You can also perform branch operations such as creating a new branch, updating a branch name, deleting a branch and listing all branches.

from deepeval.prompt import Prompt

prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
prompt.get_branches()

import { Prompt } from "deepeval";

const prompt = new Prompt({ alias: "YOUR-PROMPT-ALIAS" });
await prompt.getBranches();

  curl https://api.confident-ai.com/v1/prompts/PROMPT-ALIAS/branches \
    -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>"

from deepeval.prompt import Prompt

prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
prompt.create_branch(branch="NEW-BRANCH")

import { Prompt } from "deepeval";

const prompt = new Prompt({ alias: "YOUR-PROMPT-ALIAS"});
await prompt.createBranch({ branch: "NEW-BRANCH"});

  curl -X POST https://api.confident-ai.com/v1/prompts/PROMPT-ALIAS/branches \
    -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>" \
    -H "Content-Type: application/json"

from deepeval.prompt import Prompt

prompt = Prompt(alias="YOUR-PROMPT-ALIAS", branch="OLD-BRANCH-NAME")
prompt.update_branch(name="NEW-BRANCH-NAME")

import { Prompt } from "deepeval";

const prompt = new Prompt({ alias: "YOUR-PROMPT-ALIAS", branch: "OLD-BRANCH-NAME" });
await prompt.updateBranch({ name: "NEW-BRANCH-NAME"});

  curl -X PUT https://api.confident-ai.com/v1/prompts/PROMPT-ALIAS/branches/BRANCH-ID \
    -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>" \
    -H "Content-Type: application/json" \
    -d '{
      "name": "RenamedBranch"
    }'

from deepeval.prompt import Prompt

prompt = Prompt(alias="YOUR-PROMPT-ALIAS", branch="OLD-BRANCH-NAME")
prompt.delete_branch(branch="NEW-BRANCH-NAME")

import { Prompt } from "deepeval";

const prompt = new Prompt({ alias: "YOUR-PROMPT-ALIAS", branch: "OLD-BRANCH-NAME" });
await prompt.deleteBranch({ branch: "NEW-BRANCH-NAME"});

  curl -X DELETE https://api.confident-ai.com/v1/prompts/PROMPT-ALIAS/branches/BRANCH-ID \
    -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>"

How Are Prompts Pulled?

Confident AI automatically caches prompts on the client side to minimize API call latency and ensure prompt availability, which is especially useful in production environments.

sequenceDiagram
    participant App as Your Application
    participant DeepEval as DeepEval
    participant Cache as DeepEval Cache
    participant API as Confident AI API

    App->>DeepEval: Pull Prompt
    DeepEval->>Cache: Check cache
    Cache-->>DeepEval: Cached Prompt Data
    DeepEval-->>App: Prompt Data
    Note over Cache,API: Refetching (every 60s)
    Cache->>API: GET /v1/prompts
    API-->>Cache: Update Prompt Cache

sequenceDiagram
    participant App as Your Application
    participant DeepEval as DeepEval
    participant Cache as DeepEval Cache
    participant API as Confident AI API

    Note over App,API: Caching Disabled (refresh=0)
    App->>DeepEval: Pull Prompt
    DeepEval->>Cache: Bypass cache
    Cache->>API: GET /v1/prompts
    API-->>Cache: Prompt Data
    Cache-->>DeepEval: Prompt Data
    DeepEval-->>App: Prompt Data
    Note over Cache,API: Direct API call every time

Customize refresh rate

By default, the cache is refetched every 60 seconds, where DeepEval will automatically update the cached prompt with the up-to-date version from Confident AI. This can be overridden by setting the refresh parameter to a different value. Fetching is done asynchronously, so it will not block your application.

main.py

from deepeval.prompt import Prompt

prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
prompt.pull(refresh=60)
interpolated_prompt = prompt.interpolate(name="Joe")

index.ts

import { Prompt } from "deepeval";

const prompt = new Prompt({ alias: "YOUR-PROMPT-ALIAS" });
await prompt.pull({ refresh: 60 });
const interpolatedPrompt = prompt.interpolate({ name: "Joe" });

Configure cache

To disable caching, you can set refresh=0. This will force an API call every time you pull the prompt, which is particularly useful for development and testing.

main.py

from deepeval.prompt import Prompt

prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
prompt.pull(refresh=0)
interpolated_prompt = prompt.interpolate(name="Joe")

index.ts

import { Prompt } from "deepeval";

const prompt = new Prompt({ alias: "YOUR-PROMPT-ALIAS" });
await prompt.pull({ refresh: 0 });
const interpolatedPrompt = prompt.interpolate({ name: "Joe" });

Advanced Usage

As you learnt in earlier sections, prompts have the additional option to not just version text/messages but also model settings (provider, name, parameters), output type, and tools.

Using model settings

After pulling a prompt, you can access any model settings that were configured for the prompt version via the model_settings property. Model settings include the provider, model name, and model parameters (temperature, max_tokens, etc.).

main.py

from deepeval.prompt import Prompt
from openai import OpenAI

prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
prompt.pull()
interpolated_prompt = prompt.interpolate()
settings = prompt.model_settings

# Use model settings (provider, name, parameters) in your OpenAI call
response = OpenAI().chat.completions.create(
    model=settings.name,
    messages=interpolated_prompt,
    temperature=settings.temperature,
    max_tokens=settings.max_tokens,
    top_p=settings.top_p,
    frequency_penalty=settings.frequency_penalty,
    presence_penalty=settings.presence_penalty,
    stop=settings.stop_sequence,
)

print(response.choices[0].message.content)

index.ts

import { Prompt } from "deepeval";
import { OpenAI } from "openai";

const prompt = new Prompt({ alias: "YOUR-PROMPT-ALIAS" });
await prompt.pull();
const interpolatedPrompt = prompt.interpolate();
const settings = prompt.modelSettings;

// Use model settings (provider, name, parameters) in your OpenAI call
const openai = new OpenAI();
const response = await openai.chat.completions.create({
  model: settings.name,
  messages: interpolatedPrompt as any[],
  temperature: settings.temperature,
  max_tokens: settings.maxTokens,
  top_p: settings.topP,
  frequency_penalty: settings.frequencyPenalty,
  presence_penalty: settings.presencePenalty,
  stop: settings.stopSequence,
});

console.log(response.choices[0].message.content);

GET/v1/prompts/{alias}/versions/{version}

curl -X GET "https://api.confident-ai.com/v1/prompts/{alias}/versions/{version}" \
  -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>"

The response includes a modelSettings object with the configured model settings.

Using output type

After pulling a prompt, you can access the output configuration via the output_type and output_schema properties. This is useful when you want to enforce structured outputs from your LLM.

main.py

from deepeval.prompt import Prompt
from deepeval.prompt.api import OutputType
from openai import OpenAI

prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
prompt.pull()
interpolated_prompt = prompt.interpolate()

# Build response_format based on output type
response_format = None
if prompt.output_type == OutputType.JSON:
    response_format = {"type": "json_object"}
elif prompt.output_type == OutputType.SCHEMA:
    response_format = {
        "type": "json_schema",
        "json_schema": prompt.output_schema
    }

# Use output type in your OpenAI call
response = OpenAI().chat.completions.create(
    model="gpt-4o",
    messages=interpolated_prompt,
    response_format=response_format,
)

print(response.choices[0].message.content)

index.ts

import { Prompt, OutputType } from "deepeval";
import { OpenAI } from "openai";

const prompt = new Prompt({ alias: "YOUR-PROMPT-ALIAS" });
await prompt.pull();
const interpolatedPrompt = prompt.interpolate();

// Build response_format based on output type
let responseFormat: any = undefined;
if (prompt.outputType === OutputType.JSON) {
  responseFormat = { type: "json_object" };
} else if (prompt.outputType === OutputType.SCHEMA) {
  responseFormat = {
    type: "json_schema",
    json_schema: prompt.outputSchema,
  };
}

// Use output type in your OpenAI call
const openai = new OpenAI();
const response = await openai.chat.completions.create({
  model: "gpt-4o",
  messages: interpolatedPrompt as any[],
  response_format: responseFormat,
});

console.log(response.choices[0].message.content);

GET/v1/prompts/{alias}/versions/{version}

curl -X GET "https://api.confident-ai.com/v1/prompts/{alias}/versions/{version}" \
  -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>"

The response includes outputType and outputSchema fields for the output configuration. The outputSchema field contains the output schema definition when outputType is SCHEMA.

The available output types are:

TEXT - Plain text output (default)
JSON - JSON formatted output (maps to {"type": "json_object"})
SCHEMA - Structured output validated against a schema (maps to {"type": "json_schema", ...})

Using tools

After pulling a prompt, you can access any tools that were defined in the prompt version via the tools property. Each tool contains:

name: The name of the tool
description: A description of what the tool does
input_schema: The JSON schema defining the tool's input parameters
mode: The tool mode (ALLOW_ADDITIONAL, NO_ADDITIONAL, or STRICT)

main.py

from deepeval.prompt import Prompt
from openai import OpenAI

prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
prompt.pull()
interpolated_prompt = prompt.interpolate()

# Convert prompt tools to OpenAI format
openai_tools = [
    {
        "type": "function",
        "function": {
            "name": tool.name,
            "description": tool.description,
            "parameters": tool.input_schema,
            "strict": tool.mode == "STRICT",
        },
    }
    for tool in prompt.tools
]

# Use tools in your OpenAI call
response = OpenAI().chat.completions.create(
    model="gpt-4o",
    messages=interpolated_prompt,
    tools=openai_tools,
)

print(response.choices[0].message)

index.ts

import { Prompt } from "deepeval";
import { OpenAI } from "openai";

const prompt = new Prompt({ alias: "YOUR-PROMPT-ALIAS" });
await prompt.pull();
const interpolatedPrompt = prompt.interpolate();

// Convert prompt tools to OpenAI format
const openaiTools = prompt.tools.map((tool) => ({
  type: "function" as const,
  function: {
    name: tool.name,
    description: tool.description,
    parameters: tool.inputSchema,
    strict: tool.mode === "STRICT",
  },
}));

// Use tools in your OpenAI call
const openai = new OpenAI();
const response = await openai.chat.completions.create({
  model: "gpt-4o",
  messages: interpolatedPrompt as any[],
  tools: openaiTools,
});

console.log(response.choices[0].message);

GET/v1/prompts/{alias}/versions/{version}

curl -X GET "https://api.confident-ai.com/v1/prompts/{alias}/versions/{version}" \
  -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>"

The response includes a tools array with each tool's name, description, inputSchema, and mode.

Using images

For prompts containing images, here's how you would parse it and pass it for use in your MLLM of choice:

The deepeval python SDK offers a utility method called convert_to_multi_modal_array. This method is useful for converting a string containing images in the [DEEPEVAL:IMAGE:url] format into a list of strings and MLLMImage items.

from deepeval.prompt import Prompt
from deepeval.utils import convert_to_multi_modal_array

prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
prompt.pull()

multimodal_array = convert_to_multi_modal_array(prompt.text)

The multimodal_array here is a list containing strings and MLLMImages, you can loop over this list to construct a messages array with images to pass to your MLLM. Here's an example showing how to construct messages array for openai:

messages = []
for element in multimodal_array:
    if isinstance(element, str):
        messages.append({"type": "text", "text": element})
    elif isinstance(element, MLLMImage):
      if element.url:
          messages.append(
              {
                  "type": "image_url",
                  "image_url": {"url": element.url},
              }
          )

You can use a custom method to parse strings with images in [DEEPEVAL:IMAGE:url] format to convert them into an array of strings and URLs

const parseMultimodalString = (s: string) => {
  const PATTERN = /\[DEEPEVAL:IMAGE:(.*?)\]/g;

  const result = [];
  let lastEnd = 0;
  let match;

  while ((match = PATTERN.exec(s)) !== null) {
    const start = match.index;
    const end = PATTERN.lastIndex;

    if (start > lastEnd) {
      result.push(s.slice(lastEnd, start));
    }

    const imageUrl = match[1];
    result.push({ url: imageUrl });

    lastEnd = end;
  }

  if (lastEnd < s.length) {
    result.push(s.slice(lastEnd));
  }

  return result;
}

You can now use this method to get multimodalArray from prompt text and construct messages array to pass it to your MLLM. Here's an example on how to use it to construct openai format messages:

import { Prompt } from "deepeval";

const prompt = new Prompt({ alias: "YOUR-PROMPT-ALIAS" });
await prompt.pull();

const multimodalArray = parseMultimodalString(prompt.text);

const messages = [];

for (const element of multimodalArray) {
  if (typeof element === "string") {
    messages.push({
      type: "text",
      text: element,
    });
  } else if (element.url) {
    messages.push({
      type: "image_url",
      image_url: { url: element.url },
    });
  }
}

GET/v1/prompts/{alias}/versions/{version}

curl -X GET "https://api.confident-ai.com/v1/prompts/{alias}/versions/{version}" \
  -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>"

The prompt pulled here has images in text / messages fields with the pattern [DEEPEVAL:IMAGE:url]. Please parse the fields to fetch the public url and use it as necessary.

You can use a custom method to parse strings with images in [DEEPEVAL:IMAGE:url] format to convert them into an array of strings and URLs

def parse_multimodal_string(s: str):
    PATTERN = r"\[DEEPEVAL:IMAGE:(.*?)\]"
    matches = list(re.finditer(pattern, s))

    result = []
    last_end = 0
    for m in matches:
        start, end = m.span()
        if start > last_end:
          result.append(s[last_end:start])

        image_url = m.group(1)

        result.append({"url": image_url})
        last_end = end

    if last_end < len(s):
        result.append(s[last_end:])
        return result

You can now use this method to get multimodal_array and construct messages array to pass it to your MLLM. Here's an example on how to use it:

multimodal_array = parse_multimodal_string(prompt.text)

messages = []
for element in multimodal_array:
    if isinstance(element, str):
        messages.append({"type": "text", "text": element})
    else:
        if element.get("url") is not None:
          messages.append(
              {
                  "type": "image_url",
                  "image_url": {"url": element.url},
              }
          )

Prompt Association

You can and should definitely associate prompt versions with test runs and traced data for Confident AI to let you know which version of your prompt performs best.

Evals

You can associate a prompt with your evals to get detailed insights on how each prompt and their versions are performing. It works by:

Pulling prompt via the Evals API
Logging prompts as a hyperparameter during evaluation

Simply add the pulled prompt instance as a free-form key-value pair to the hyperparameters argument in the evaluate() function

evaluate(
    ...
    hyperparameters={
        "Model": "YOUR-MODEL",
        "Prompt": prompt,
    },
)

Simply add the pulled prompt instance as a free-form key-value pair to the hyperparameters argument in the evaluate() function

evaluate({
  ...
  hyperparameters: {
    Model: "YOUR-MODEL",
    "Prompt": prompt,
  },
});

POST/v1/evaluate

curl -X POST "https://api.confident-ai.com/v1/evaluate" \
  -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>" \
  -H "Content-Type: application/json" \
  -d '{
  "metricCollection": "string",
  "llmTestCases": [
    {
      "input": "string",
      "actualOutput": "string",
      "name": "string",
      "expectedOutput": "string",
      "retrievalContext": [
        "string"
      ],
      "context": [
        "string"
      ],
      "toolsCalled": [
        {
          "name": "string",
          "description": "string",
          "inputParameters": {},
          "output": "string",
          "reasoning": "string"
        }
      ],
      "expectedTools": [
        {
          "name": "string",
          "description": "string",
          "inputParameters": {},
          "output": "string",
          "reasoning": "string"
        }
      ]
    }
  ],
  "conversationalTestCases": [
    {
      "turns": [
        {
          "role": "user",
          "content": "string",
          "userId": "string",
          "retrievalContext": [
            "string"
          ],
          "toolsCalled": [
            {
              "name": null,
              "description": null,
              "inputParameters": null,
              "output": null,
              "reasoning": null
            }
          ]
        }
      ],
      "scenario": "string",
      "name": "string",
      "expectedOutcome": "string",
      "userDescription": "string",
      "chatbotRole": "string"
    }
  ],
  "hyperparameters": {},
  "identifier": "string"
}'

This will automatically attribute the prompt used during this test run, which will allow you get detailed insights in the Confident AI platform.

Tracing

Associating prompts with LLM traces and spans is a great way to determine which prompts performs best in production.

Setup tracing
Attach the @observe decorator to functions/methods that make up your agent, and specify type llm for your LLM-calling functions.
main.py
```
from deepeval.tracing import observe

@observe(type="llm", model="gpt-4.1")
def your_llm_component():
    ...
```
Specifying the type is necessary because logging prompts is only available for LLM spans.

Pull and interpolate prompt

Pull and interpolate the prompt version to use it for LLM generation.

main.py

from deepeval.tracing import observe
from deepeval.prompt import Prompt
from openai import OpenAI

@observe(type="llm", model="gpt-4.1")
def your_llm_component():
    prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
    prompt.pull()
    interpolated_prompt = prompt.interpolate(name="Joe")
    response = OpenAI().chat.completions.create(model="gpt-4o-mini", messages=interpolated_prompt)
    return response.choices[0].message.content

Execute your function

Then simply provide the prompt to the update_llm_span function.

main.py

from deepeval.tracing import observe, update_llm_span
from deepeval.prompt import Prompt
from openai import OpenAI

@observe(type="llm", model="gpt-4.1")
def your_llm_component():
    prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
    prompt.pull()
    interpolated_prompt = prompt.interpolate(name="Joe")
    response = OpenAI().chat.completions.create(model="gpt-4o-mini", messages=interpolated_prompt)
    update_llm_span(prompt=prompt)
    return response.choices[0].message.content

This will automatically attribute the prompt used to the LLM span.

Switching Projects

You can pull and manage your prompts in any project by configuring a CONFIDENT_API_KEY.

For default usage, set CONFIDENT_API_KEY as an environment variable.
To target a specific project, pass a confident_api_key directly when creating the Prompt object.

from deepeval.prompt import Prompt, PromptMessage

prompt = Prompt(
  alias="YOUR-PROMPT-ALIAS",
  confident_api_key="confident_us...",
)

import { Prompt } from "deepeval";

const prompt = new Prompt({
  alias: "YOUR-PROMPT-ALIAS",
  confidentApiKey: "confident_us...",
});

When both are provided, the confident_api_key passed to Prompt always takes precedence over the environment variable.

Next Steps

Now that you can pull prompts into your app, learn how to push them programmatically or run evaluations.

Automate Prompt Management

Push and manage prompts programmatically via the Evals API.

Run Evaluations

Evaluate your LLM app with metrics and datasets.

Managing prompts in production?Prompt versioning, A/B testing, and rollbacks for production teamsBook a demo

Overview

Pull Prompt By Version

Pull prompt with alias

Interpolate variables

Use interpolated prompt

Pull Prompts By Label

Pull Prompts By Commit

Pull Prompts By Branch

Branch Operations

How Are Prompts Pulled?

Customize refresh rate

Configure cache

Advanced Usage

Using model settings

Using output type

Using tools

Using images

Prompt Association

Evals

Tracing

Setup tracing

Pull and interpolate prompt

Execute your function

Switching Projects

Next Steps