Pull Prompts

Overview

You can pull a prompt version from Confident AI like how you would pull a dataset. It works by:

• Providing Confident AI with the alias and optionally version of the prompt you wish to retrieve
• Confident AI will provide the non-interpolated version of the prompt
• You will then interpolate the variables in code

You should pull prompts once and save it in memory instead of pulling it everytime you need to use it.

Pull Prompt By Version

1
from deepeval.prompt import Prompt
2
3
prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
4
prompt.pull(version="latest")

1
prompt.pull(version="00.00.01")

1
interpolated_prompt = prompt.interpolate(name="Joe")

1
{
2
  "role": "system",
3
  "content": "You are a helpful assistant called {{ name }}. Speak normally like a human."
4
}

1
{
2
  "role": "system",
3
  "content": "You are a helpful assistant called Joe. Speak normally like a human."
4
}

1
{
2
  "role": "system",
3
  "content": "You are a helpful assistant called Joe. Speak normally like a human."
4
}

1
from deepeval.prompt import Prompt
2
from openai import OpenAI
3
4
prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
5
prompt.pull()
6
interpolated_prompt = prompt.interpolate() # interpolate prompt
7
8
response = OpenAI().chat.completions.create(
9
    model="gpt-4o-mini",
10
    messages=interpolated_prompt
11
)
12
13
print(response.choices[0].message.content)

You can also fetch all the versions associated with a prompt as shown below:

1
from deepeval.prompt import Prompt
2
3
prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
4
commits = prompt._get_versions()

Pull Prompts By Label

You can also pull specific versions of prompts using the label you assign to the versions on the platform.

1
from deepeval.prompt import Prompt
2
3
prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
4
prompt.pull(label="staging")

You must manually label each prompt version before pulling it. Click here to learn how to do so.

Pull Prompts By Commit

You can also pull specific snapshots of prompts using it’s alias and the commit hash.

1
from deepeval.prompt import Prompt
2
3
prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
4
prompt.pull()

1
prompt.pull(hash="bab04ce")

You can also fetch all the commits associated with a prompt as shown below:

1
from deepeval.prompt import Prompt
2
3
prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
4
commits = prompt._get_commits()

How Are Prompts Pulled?

Confident AI automatically caches prompts on the client side to minimize API call latency and ensure prompt availability, which is especially useful in production environments.

Customize refresh rate

By default, the cache is refetched every 60 seconds, where DeepEval will automatically update the cached prompt with the up-to-date version from Confident AI. This can be overridden by setting the refresh parameter to a different value. Fetching is done asynchronously, so it will not block your application.

1
from deepeval.prompt import Prompt
2
3
prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
4
prompt.pull(refresh=60)
5
interpolated_prompt = prompt.interpolate(name="Joe")

Configure cache

To disable caching, you can set refresh=0. This will force an API call every time you pull the prompt, which is particularly useful for development and testing.

1
from deepeval.prompt import Prompt
2
3
prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
4
prompt.pull(refresh=0)
5
interpolated_prompt = prompt.interpolate(name="Joe")

Advanced Usage

As you learnt in earlier sections, prompts have the additional option to not just version text/messages but also model settings (provider, name, parameters), output type, and tools.

Using model settings

After pulling a prompt, you can access any model settings that were configured for the prompt version via the model_settings property. Model settings include the provider, model name, and model parameters (temperature, max_tokens, etc.).

1
from deepeval.prompt import Prompt
2
from openai import OpenAI
3
4
prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
5
prompt.pull()
6
interpolated_prompt = prompt.interpolate()
7
settings = prompt.model_settings
8
9
# Use model settings (provider, name, parameters) in your OpenAI call
10
response = OpenAI().chat.completions.create(
11
    model=settings.name,
12
    messages=interpolated_prompt,
13
    temperature=settings.temperature,
14
    max_tokens=settings.max_tokens,
15
    top_p=settings.top_p,
16
    frequency_penalty=settings.frequency_penalty,
17
    presence_penalty=settings.presence_penalty,
18
    stop=settings.stop_sequence,
19
)
20
21
print(response.choices[0].message.content)

Using output type

After pulling a prompt, you can access the output configuration via the output_type and output_schema properties. This is useful when you want to enforce structured outputs from your LLM.

1
from deepeval.prompt import Prompt
2
from deepeval.prompt.api import OutputType
3
from openai import OpenAI
4
5
prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
6
prompt.pull()
7
interpolated_prompt = prompt.interpolate()
8
9
# Build response_format based on output type
10
response_format = None
11
if prompt.output_type == OutputType.JSON:
12
    response_format = {"type": "json_object"}
13
elif prompt.output_type == OutputType.SCHEMA:
14
    response_format = {
15
        "type": "json_schema",
16
        "json_schema": prompt.output_schema
17
    }
18
19
# Use output type in your OpenAI call
20
response = OpenAI().chat.completions.create(
21
    model="gpt-4o",
22
    messages=interpolated_prompt,
23
    response_format=response_format,
24
)
25
26
print(response.choices[0].message.content)

The available output types are:

• TEXT - Plain text output (default)
• JSON - JSON formatted output (maps to {"type": "json_object"})
• SCHEMA - Structured output validated against a schema (maps to {"type": "json_schema", ...})

Using tools

After pulling a prompt, you can access any tools that were defined in the prompt version via the tools property. Each tool contains:

• name: The name of the tool
• description: A description of what the tool does
• input_schema: The JSON schema defining the tool’s input parameters
• mode: The tool mode (ALLOW_ADDITIONAL, NO_ADDITIONAL, or STRICT)

1
from deepeval.prompt import Prompt
2
from openai import OpenAI
3
4
prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
5
prompt.pull()
6
interpolated_prompt = prompt.interpolate()
7
8
# Convert prompt tools to OpenAI format
9
openai_tools = [
10
    {
11
        "type": "function",
12
        "function": {
13
            "name": tool.name,
14
            "description": tool.description,
15
            "parameters": tool.input_schema,
16
            "strict": tool.mode == "STRICT",
17
        },
18
    }
19
    for tool in prompt.tools
20
]
21
22
# Use tools in your OpenAI call
23
response = OpenAI().chat.completions.create(
24
    model="gpt-4o",
25
    messages=interpolated_prompt,
26
    tools=openai_tools,
27
)
28
29
print(response.choices[0].message)

Using images

For prompts containing images, here’s how you would parse it and pass it for use in your MLLM of choice:

1
from deepeval.prompt import Prompt
2
from deepeval.utils import convert_to_multi_modal_array
3
4
prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
5
prompt.pull()
6
7
multimodal_array = convert_to_multi_modal_array(prompt.text)

1
messages = []
2
for element in multimodal_array:
3
    if isinstance(element, str):
4
        messages.append({"type": "text", "text": element})
5
    elif isinstance(element, MLLMImage):
6
      if element.url:
7
          messages.append(
8
              {
9
                  "type": "image_url",
10
                  "image_url": {"url": element.url},
11
              }
12
          )

Prompt Association

You can and should definitely associate prompt versions with test runs and traced data for Confident AI to let you know which version of your prompt performs best.

Evals

You can associate a prompt with your evals to get detailed insights on how each prompt and their versions are performing. It works by:

• Pulling prompt via the Evals API
• Logging prompts as a hyperparameter during evaluation

1
evaluate(
2
    ...
3
    hyperparameters={
4
        "Model": "YOUR-MODEL",
5
        "Prompt": prompt,
6
    },
7
)

This will automatically attribute the prompt used during this test run, which will allow you get detailed insights in the Confident AI platform.

Tracing

Associating prompts with LLM traces and spans is a great way to determine which prompts performs best in production.

1
from deepeval.tracing import observe
2
3
@observe(type="llm", model="gpt-4.1")
4
def your_llm_component():
5
    ...

1
from deepeval.tracing import observe
2
from deepeval.prompt import Prompt
3
from openai import OpenAI
4
5
@observe(type="llm", model="gpt-4.1")
6
def your_llm_component():
7
    prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
8
    prompt.pull()
9
    interpolated_prompt = prompt.interpolate(name="Joe")
10
    response = OpenAI().chat.completions.create(model="gpt-4o-mini", messages=interpolated_prompt)
11
    return response.choices[0].message.content

1
from deepeval.tracing import observe, update_llm_span
2
from deepeval.prompt import Prompt
3
from openai import OpenAI
4
5
@observe(type="llm", model="gpt-4.1")
6
def your_llm_component():
7
    prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
8
    prompt.pull()
9
    interpolated_prompt = prompt.interpolate(name="Joe")
10
    response = OpenAI().chat.completions.create(model="gpt-4o-mini", messages=interpolated_prompt)
11
    update_llm_span(prompt=prompt)
12
    return response.choices[0].message.content

Switching Projects

You can pull and manage your prompts in any project by configuring a CONFIDENT_API_KEY.

• For default usage, set CONFIDENT_API_KEY as an environment variable.
• To target a specific project, pass a confident_api_key directly when creating the Prompt object.

1
from deepeval.prompt import Prompt, PromptMessage
2
3
prompt = Prompt(
4
  alias="YOUR-PROMPT-ALIAS",
5
  confident_api_key="confident_us...",
6
)

When both are provided, the confident_api_key passed to Prompt always takes precedence over the environment variable.

Next Steps

Now that you can pull prompts into your app, learn how to push them programmatically or run evaluations.