Working with Prompts

Overview

You can pull a prompt version from Confident AI like how you would pull a dataset. It works by:

• Providing Confident AI with the alias and optionally version of the prompt you wish to retrieve
• Confident AI will provide the non-interpolated version of the prompt
• You will then interpolate the variables in code

You should pull prompts once and save it in memory instead of pulling it everytime you need to use it.

1
from deepeval.prompt import Prompt, PromptMessage
2
3
prompt = Prompt(
4
  alias="YOUR-PROMPT-ALIAS",
5
  confident_api_key="confident_us...",
6
)

Using Prompt Versions

1
from deepeval.prompt import Prompt
2
3
prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
4
prompt.pull()

1
prompt.pull(version="00.00.01")

1
interpolated_prompt = prompt.interpolate(name="Joe")

1
{
2
  "role": "system",
3
  "content": "You are a helpful assistant called {{ name }}. Speak normally like a human."
4
}

1
{
2
  "role": "system",
3
  "content": "You are a helpful assistant called Joe. Speak normally like a human."
4
}

1
{
2
  "role": "system",
3
  "content": "You are a helpful assistant called Joe. Speak normally like a human."
4
}

1
from deepeval.prompt import Prompt
2
from openai import OpenAI
3
4
prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
5
prompt.pull()
6
interpolated_prompt = prompt.interpolate() # interpolate prompt
7
8
response = OpenAI().chat.completions.create(
9
    model="gpt-4o-mini",
10
    messages=interpolated_prompt
11
)
12
13
print(response.choices[0].message.content)

Accessing Tools from Prompts

After pulling a prompt, you can access any tools that were defined in the prompt version via the tools property. Each tool contains:

• name: The name of the tool
• description: A description of what the tool does
• input schema: The JSON schema defining the tool’s input parameters

1
from deepeval.prompt import Prompt
2
3
prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
4
prompt.pull()
5
6
# Access tools from the prompt
7
for tool in prompt.tools:
8
    print(f"Tool Name: {tool.name}")
9
    print(f"Description: {tool.description}")
10
    print(f"Input Schema: {tool.input_schema}")

Pull Prompts By Label

You can pull a prompt from Confident AI using its alias, you can also pull specific versions of prompts using verion and label.

1
from deepeval.prompt import Prompt
2
3
prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
4
prompt.pull(version="00.00.01")

How Are Prompts Pulled?

Confident AI automatically caches prompts on the client side to minimize API call latency and ensure prompt availability, which is especially useful in production environments.

Customize refresh rate

By default, the cache is refetched every 60 seconds, where DeepEval will automatically update the cached prompt with the up-to-date version from Confident AI. This can be overridden by setting the refresh parameter to a different value. Fetching is done asynchronously, so it will not block your application.

1
from deepeval.prompt import Prompt
2
3
prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
4
prompt.pull(refresh=60)
5
interpolated_prompt = prompt.interpolate(name="Joe")

Disable Caching

To disable caching, you can set refresh=0. This will force an API call every time you pull the prompt, which is particularly useful for development and testing.

1
from deepeval.prompt import Prompt
2
3
prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
4
prompt.pull(refresh=0)
5
interpolated_prompt = prompt.interpolate(name="Joe")

Log Prompts During Evals

You can associate a prompt with your evals to get detailed insights on how each prompt and their versions are performing. It works by:

• Pulling or creating prompts in deepeval using the Prompt object
• Logging prompts as hyperparameters in your evaluations

1
from deepeval.prompt import Prompt
2
3
prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
4
prompt.pull(version="00.00.01")

1
evaluate(
2
    ...
3
    hyperparameters={
4
        "Model": "YOUR-MODEL",
5
        "Prompt": prompt,
6
    },
7
)

Log Prompts During Tracing

Associating prompts with LLM traces and spans is a great way to determine which prompts performs best in production.

1
from deepeval.tracing import observe
2
3
@observe(type="llm", model="gpt-4.1")
4
def your_llm_component():
5
    ...

1
from deepeval.tracing import observe
2
from deepeval.prompt import Prompt
3
from openai import OpenAI
4
5
@observe(type="llm", model="gpt-4.1")
6
def your_llm_component():
7
    prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
8
    prompt.pull()
9
    interpolated_prompt = prompt.interpolate(name="Joe")
10
    response = OpenAI().chat.completions.create(model="gpt-4o-mini", messages=interpolated_prompt)
11
    return response.choices[0].message.content

1
from deepeval.tracing import observe, update_llm_span
2
from deepeval.prompt import Prompt
3
from openai import OpenAI
4
5
@observe(type="llm", model="gpt-4.1")
6
def your_llm_component():
7
    prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
8
    prompt.pull()
9
    interpolated_prompt = prompt.interpolate(name="Joe")
10
    response = OpenAI().chat.completions.create(model="gpt-4o-mini", messages=interpolated_prompt)
11
    update_llm_span(prompt=prompt)
12
    return response.choices[0].message.content