Automate Prompt Management

Overview

Instead of manually creating and updating prompts on the platform, you can automate prompt management via the Evals API. This allows you to:

• Push new prompt commits from your codebase or CI/CD pipeline
• Promote commits to versions programmatically
• Optionally configure model settings, output type, and tools to track alongside prompts
• Integrate prompt management into your development workflow

Push Prompt Commits

Push a new commit of a prompt to Confident AI. If the prompt alias doesn’t exist, it will be created automatically. Every push creates a new commit that tracks your changes.

1
from deepeval.prompt import Prompt
2
from deepeval.prompt.api import PromptMessage
3
4
prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
5
prompt.push(
6
    messages=[
7
        PromptMessage(role="system", content="You are a helpful assistant called {name}."),
8
    ]
9
)

1
from deepeval.prompt import Prompt
2
3
prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
4
prompt.push(text="You are a helpful assistant called {name}.")

1
from deepeval.prompt import Prompt
2
from deepeval.prompt.api import PromptInterpolationType
3
4
prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
5
prompt.push(
6
    text="You are a helpful assistant called {{name}}.",
7
    interpolation_type=PromptInterpolationType.MUSTACHE
8
)

Each push creates a new commit automatically. When you’re ready to mark a commit as a stable release, you can promote it to a version. Version numbers are controlled by Confident AI in the format 00.00.0X.

Create a Version

When you’re ready to mark a commit as a stable release, you can promote it to a version. Version numbers are automatically assigned by Confident AI in the format 00.00.0X (e.g., 00.00.01, 00.00.02).

1
from deepeval.prompt import Prompt
2
3
prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
4
5
# Create a version from the latest commit
6
7
prompt.create_version()
8
9
# Or create a version from a specific commit
10
11
prompt.create_version(hash="COMMIT-HASH")

Once a commit is promoted to a version, you can assign labels (like staging or production) to it. Labels can only exist on versions, not commits.

Adding Model Configs

If you want to manage your model configuration alongside your prompts — tracking the prompt + model together in each commit — you can include model_settings and output_type when pushing.

This is useful when:

• You want to ensure a specific prompt always runs with a specific model and parameters
• You’re A/B testing different prompt + model combinations together
• You want to centralize both prompt and model configuration in one place

1
from deepeval.prompt import Prompt
2
from deepeval.prompt.api import (
3
    PromptMessage,
4
    ModelSettings,
5
    ModelProvider,
6
    OutputType,
7
    ReasoningEffort,
8
    Verbosity,
9
)
10
from pydantic import BaseModel
11
12
class ResponseSchema(BaseModel):
13
    answer: str
14
    confidence: float
15
16
prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
17
18
# Use with push() to create a new commit
19
prompt.push(
20
    messages=[
21
        PromptMessage(role="system", content="You are a helpful assistant."),
22
    ],
23
    model_settings=ModelSettings(
24
        provider=ModelProvider.OPEN_AI,
25
        name="gpt-4o",
26
        temperature=0.7,
27
        max_tokens=1000,
28
        top_p=0.9,
29
        frequency_penalty=0.1,
30
        presence_penalty=0.1,
31
        stop_sequence=["END"],
32
        reasoning_effort=ReasoningEffort.MINIMAL,
33
        verbosity=Verbosity.LOW,
34
    ),
35
    output_type=OutputType.SCHEMA,
36
    output_schema=ResponseSchema,
37
)

Reference

Model settings

Model settings include the provider, model name, and model parameters:

Parameters

Here are all the available parameters you could set:

Providers

Here are the list of available model providers:

Output configurations

You can also configure structured outputs:

Output Types:

• TEXT - Plain text output
• JSON - JSON formatted output
• SCHEMA - Structured output validated against a Pydantic schema

Interpolation types

Specify how variables are interpolated in your prompts:

What about Tools?

You can create and update tools using prompts by pushing prompts with tools. Tools in Confident AI are identified using their names — passing a tool with a new name creates a tool and passing a tool with an existing name updates the tool on the platform. Each push creates a new commit that tracks the tool configuration. Here’s how you can create / update tools:

1
from deepeval.prompt import Prompt, Tool
2
from deepeval.prompt.api import ToolMode
3
from pydantic import BaseModel
4
5
class ToolInputSchema(BaseModel):
6
    result: str
7
    confidence: float
8
9
prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
10
tool = Tool(
11
    name="SearchTool",
12
    description="Search functionality",
13
    mode=ToolMode.STRICT,
14
    structured_schema=ToolInputSchema,
15
)
16
17
# Use with push() to create a new commit with the tool
18
prompt.push(
19
    text="This a prompt for a tool using agent",
20
    tools=[tool]
21
)
22
23
tool_2 = Tool(
24
    name="SearchTool",
25
    description="New search functionality",
26
    mode=ToolMode.STRICT,
27
    structured_schema=ToolInputSchema,
28
)
29
30
# Create a new commit with the new updated tool using 'push'
31
prompt.push(
32
    text="This a prompt for a tool using agent",
33
    tools=[tool_2]
34
)

Prompts in CI/CD

Automate prompt tracking as part of your CI/CD pipeline. A common pattern is to push prompt commits whenever your prompt files change:

1
name: Push Prompt Commits
2
3
on:
4
  push:
5
    paths:
6
      - "prompts/**"
7
8
jobs:
9
  push-prompts:
10
    runs-on: ubuntu-latest
11
    steps:
12
      - uses: actions/checkout@v3
13
14
      - name: Set up Python
15
        uses: actions/setup-python@v4
16
        with:
17
          python-version: "3.11"
18
19
      - name: Install dependencies
20
        run: pip install deepeval
21
22
      - name: Push prompts
23
        env:
24
          CONFIDENT_API_KEY: ${{ secrets.CONFIDENT_API_KEY }}
25
        run: python scripts/push_prompts.py

Your push_prompts.py script can read prompt files and push them:

1
from deepeval.prompt import Prompt
2
3
# Read your prompt content from file or config
4
with open("prompts/assistant.txt") as f:
5
    prompt_text = f.read()
6
7
prompt = Prompt(alias="assistant-prompt")
8
prompt.push(text=prompt_text)
9
10
print("Prompt commit pushed successfully!")

Next Steps

Now that you can push prompts programmatically, learn how to pull them into your app for usage.