Prompt Versioning

Overview

Prompt versioning allows you to optimize and test different versions of your prompts. Managing prompts on Confident AI allows you to:

1. Collaborate and centralize where prompt is stored and edited, even for non-technical team members
2. Pinpoint which version, or even combination of your prompt versions, performed best

There are a million places you can keep your prompts - on GitHub, CSV files, in memory in code, Google Sheets, Notion, or even written in a diary hidden under your table drawer. But only by keeping on prompts on Confident AI can you fully leverage Confident AI’s evaluation features.

Two Types of Prompts

There are two types of prompts you can create:

• (Single) Text Prompt: Use this when you need a straightforward, one-off prompt for simple completions.
• Prompt Message List: Use this when you need to define multiple messages with specific roles (system, user, assistant) in an OpenAI messages format. This format is ideal for few-shot prompting, where you can start with a system message that sets the context.

Prompts vs Prompt Versions

In Confident AI, each prompt is identified by a unique alias. This alias acts as a unique identifier and refers to a single, specific prompt. Different aliases refer to completely separate prompts.

A prompt version represents a different variation of the same prompt. Each version is an iteration or improvement of the original prompt, serving the same purpose but with adjustments—similar to how software versions evolve over time.

Create a Prompt Version

You can create a prompt version in Project > Prompt Studio through two simple steps:

1. Create a text or messages prompt
2. Edit and commit an initial prompt version in the prompt editor

Don’t forget to commit an initial version of your prompt after you’re done editing it. Alternately, you can create a prompt version from code.

1
from deepeval.prompt import Prompt, PromptMessage
2
3
prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
4
prompt.push(messages=[PromptMessage(role="...", content="...")])

Setting Dynamic Variables

You can include variables that can be interpolated dynamically in your LLM application later on. There are four ways currently for you to include variables:

• {variable}
• {{ variable }}
• {{variable}}
• ${variable}

For example, the {variable} interpolation type must follow these rules:

1. No spaces in the variable name
2. Exactly one space between the braces and the variable name

For example:

1
# ✅ Correct usage:
2
"Hi, my name is {name}."
3
"The temperature is {temperature} degrees."
4
"User input: {user_input}"
5
6
# ❌ Incorrect usage:
7
8
"Hi, my name is {variable name}." # Spaces in variable name
9
"Hi, my name is {variable_name }." # Extra spaces around variable name

Using Conditional Logic

Apart from variables, Confident AI also supports jinja templates, which would allow you to render more complex logic such as conditional if/else blocks:

{% if is_admin %}
Welcome back, mighty admin {{ name }}!
{% else %}
Hello {{ name }}, you have regular access.
{% endif %}

As well as for loops:

Shopping List:
{% for item in items %}
- {{ item }}
{% endfor %}

In the next section, we’ll show you how to work with prompts you’ve created on Confident AI.

Labelling Prompt Versions

You can also label different versions of your prompt in the Version History page so no code changes are required to “deploy” a new version into a certain environment.

1
from deepeval.prompt import Prompt
2
3
prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
4
prompt.pull(label="staging")

Update Prompts

You can update your prompt templates in code by specifying the prompt alias and version.

1
from deepeval.prompt import Prompt, PromptVersion
2
3
prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
4
prompt.update(version="00.00.01", messages=[PromptVersion(role="...", content="...")])

You can also update a prompt’s model settings, interpolation type, output type, and output schema through the update method.

1
from deepeval.prompt import Prompt, ModelSettings, InterpolationType, OutputType
2
3
prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
4
prompt.update(
5
    version="00.00.01",
6
    model_settings=ModelSettings(model="gpt-4o-mini"),
7
    interpolation_type=InterpolationType.FSTRING,
8
    output_type=OutputType.TEXT,
9
)