Version Prompts

Overview

Prompt versioning allows you to optimize and test different versions of your prompts. Managing prompts on Confident AI allows you to:

1. Collaborate and centralize where prompts are stored and edited, even for non-technical team members
2. Pinpoint which version, or even combination of your prompt versions, performed best
3. Optionally co-locate model settings, output type, and tools with prompts to version them as a single unit

There are a million places you can keep your prompts - on GitHub, CSV files, in memory in code, Google Sheets, Notion, or even written in a diary hidden under your table drawer. But only by keeping prompts on Confident AI can you fully leverage Confident AI’s evaluation features.

Types of Prompts

There are two types of prompts you can create:

• (Single) Text Prompt: Use this when you need a straightforward, one-off prompt for simple completions.
• Prompt Message List: Use this when you need to define multiple messages with specific roles (system, user, assistant) in an OpenAI messages format. This format is ideal for few-shot prompting, where you can start with a system message that sets the context.

Understanding Prompt Versioning

In Confident AI, each prompt is identified by a unique alias. This alias acts as a unique identifier and refers to a single, specific prompt. Different aliases refer to completely separate prompts.

Every change you make to a prompt is tracked as a commit. This ensures complete history and traceability of all prompt modifications. When you’re ready to mark a commit as a stable release, you can promote it to a version.

• Commit: Every change to a prompt creates a new commit. Commits are automatically tracked and provide a complete history of all modifications.
• Version: A promoted commit that represents a stable release. Version numbers are controlled by Confident AI in the format 00.00.0X (e.g., 00.00.01, 00.00.02).
• Label: Labels (like staging or production) can only be assigned to versions, not commits. This ensures that only stable, versioned prompts are deployed to different environments.

Commit a New Prompt

You can create a prompt in Project > Prompt Studio through two simple steps:

1. Create a text or messages prompt
2. Edit and commit your changes in the prompt editor

Don’t forget to commit your changes after you’re done editing. Every commit is tracked, and you can later promote any commit to a version. You can also create commits from code.

Templating Options

Dynamic variables

You can include variables that can be interpolated dynamically in your LLM application later on. There are five interpolation types available:

Variable names must not contain spaces:

1
# ✅ Correct usage:
2
"Hi, my name is {name}."
3
"The temperature is {temperature} degrees."
4
"User input: {user_input}"
5
6
# ❌ Incorrect usage:
7
"Hi, my name is {variable name}." # Spaces in variable name

Conditional logic

Conditional logic can be added when using JINJA interpolation. JINJA supports jinja templates, which allows you to render more complex logic such as conditional if/else blocks:

{% if is_admin %}
Welcome back, mighty admin {{ name }}!
{% else %}
Hello {{ name }}, you have regular access.
{% endif %}

As well as for loops:

Shopping List:
{% for item in items %}
- {{ item }}
{% endfor %}

Including images

You can also include images simply by dragging and dropping something into the text areas.

Model Configs

Beyond creating and editing prompts, you can also configure model settings, output type, and tools associated with your prompt. These configurations are included in each commit, allowing you to:

• Track not just prompt changes, but also model configuration changes
• Use directly in code when pulling prompts in code
• Compare the impact of models on the same prompt (and vice versa) when running experiments on your AI app

Model settings

You can configure the model provider, model name, and model parameters for each prompt. These settings are tracked with each commit, ensuring that when you pull a prompt in code, you also get the exact model configuration needed to run it.

Output type

You can specify the expected output format for your prompt by selecting an output type. This configuration is tracked with each commit:

• Text Output: Standard text response (default)
• JSON Output: Structured JSON response
• Schema Output: Structured response conforming to a defined schema

When using Schema Output, you can define a custom schema that your LLM response should conform to. This is useful for ensuring structured, predictable outputs.

To configure a schema:

1. Click on Schema Output in the output type dropdown
2. Enter a Schema Name (required)
3. Add Schema Fields with their property names and types (String, Number, Boolean, etc.)
4. Click Save Schema

The schema will be previewed as a Pydantic BaseModel class, making it easy to visualize how your structured output will look.

Attach tools

You can attach tools to your prompt for function calling capabilities. This allows your LLM to invoke external tools like web search, APIs, or custom functions. Tool configurations are tracked with each commit.

To attach tools:

1. Click on the Tools button in the prompt editor
2. Search for available tools
3. Select the tools you want to enable for this prompt version

Assign Prompt Labels

Labels can only be assigned to versions, not commits. This ensures that only stable, versioned prompts are deployed to different environments. You can assign labels in the Version History page so no code changes are required to “deploy” a new version into a certain environment.

To assign a label, first promote a commit to a version, then assign the desired label (e.g., staging, production) to that version.

Only users with sufficient permissions are able to modify prompt labels.

1
from deepeval.prompt import Prompt
2
3
prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
4
prompt.pull(label="staging")

Next Steps

Now that you know how to version prompts on the platform, put them to work in evaluations.