Launch Week 02 wrapped — explore all five launches
Confident AI
Docs
Pricing
Products
LLM Evaluation
LLM Evaluation

Benchmark LLM systems with research-backed metrics.

LLM Observability
LLM Observability

Trace, monitor, and alert on production LLM systems.

AI Red Teaming
AI Red Teaming

Stress-test LLM apps against adversarial attacks.

AI Governance
AI Governance

Enforce AI standards and controls across teams.

Book a DemoSign Up
Book a DemoSign Up

Automate Prompt Management

Build automated prompt management pipelines via the Evals API

Overview

Instead of manually creating and updating prompts on the platform, you can automate prompt management via the Evals API. This allows you to:

  • Push new prompt commits from your codebase or CI/CD pipeline
  • Promote commits to versions programmatically
  • Optionally configure model settings, output type, and tools to track alongside prompts
  • Integrate prompt management into your development workflow

Push Prompt Commits

Push a new commit of a prompt to Confident AI. If the prompt alias doesn't exist, it will be created automatically. Every push creates a new commit that tracks your changes.

For message prompts:

main.py
from deepeval.prompt import Prompt
from deepeval.prompt.api import PromptMessage

prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
prompt.push(
    messages=[
        PromptMessage(role="system", content="You are a helpful assistant called {name}."),
    ]
)

For text prompts:

main.py
from deepeval.prompt import Prompt

prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
prompt.push(text="You are a helpful assistant called {name}.")

You can also specify the interpolation type:

main.py
from deepeval.prompt import Prompt
from deepeval.prompt.api import PromptInterpolationType

prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
prompt.push(
    text="You are a helpful assistant called {{name}}.",
    interpolation_type=PromptInterpolationType.MUSTACHE
)

Each push creates a new commit automatically. When you're ready to mark a commit as a stable release, you can promote it to a version. Version numbers are controlled by Confident AI in the format 00.00.0X.

Create a Version

When you're ready to mark a commit as a stable release, you can promote it to a version. Version numbers are automatically assigned by Confident AI in the format 00.00.0X (e.g., 00.00.01, 00.00.02).

main.py
from deepeval.prompt import Prompt

prompt = Prompt(alias="YOUR-PROMPT-ALIAS")

# Create a version from the latest commit

prompt.create_version()

# Or create a version from a specific commit

prompt.create_version(hash="COMMIT-HASH")

Once a commit is promoted to a version, you can assign labels (like staging or production) to it. Labels can only exist on versions, not commits.

Adding Model Configs

If you want to manage your model configuration alongside your prompts — tracking the prompt + model together in each commit — you can include model_settings and output_type when pushing.

This is useful when:

  • You want to ensure a specific prompt always runs with a specific model and parameters
  • You're A/B testing different prompt + model combinations together
  • You want to centralize both prompt and model configuration in one place
main.py
from deepeval.prompt import Prompt
from deepeval.prompt.api import (
    PromptMessage,
    ModelSettings,
    ModelProvider,
    OutputType,
    ReasoningEffort,
    Verbosity,
)
from pydantic import BaseModel

class ResponseSchema(BaseModel):
    answer: str
    confidence: float

prompt = Prompt(alias="YOUR-PROMPT-ALIAS")

# Use with push() to create a new commit
prompt.push(
    messages=[
        PromptMessage(role="system", content="You are a helpful assistant."),
    ],
    model_settings=ModelSettings(
        provider=ModelProvider.OPEN_AI,
        name="gpt-4o",
        temperature=0.7,
        max_tokens=1000,
        top_p=0.9,
        frequency_penalty=0.1,
        presence_penalty=0.1,
        stop_sequence=["END"],
        reasoning_effort=ReasoningEffort.MINIMAL,
        verbosity=Verbosity.LOW,
    ),
    output_type=OutputType.SCHEMA,
    output_schema=ResponseSchema,
)

Reference

Model settings

Model settings include the provider, model name, and model parameters:

FieldTypeDefaultDescription
providerModelProviderOPEN_AIThe model provider (see supported providers below)
namestrNoneThe model name (e.g., "gpt-4o", "claude-3-opus")

Parameters

Here are all the available parameters you could set:

FieldTypeDefaultDescription
temperaturefloat0Controls randomness (0-2)
max_tokensintNoneMaximum tokens in the response
top_pfloat1Nucleus sampling parameter
frequency_penaltyfloat0Penalize repeated tokens (-2 to 2)
presence_penaltyfloat0Penalize tokens based on presence (-2 to 2)
stop_sequenceList[str][]Sequences that stop generation
reasoning_effortReasoningEffortMEDIUMReasoning effort level (MINIMAL, LOW, MEDIUM, HIGH)
verbosityVerbosityMEDIUMOutput verbosity (LOW, MEDIUM, HIGH)

Providers

Here are the list of available model providers:

ProviderDescription
OPEN_AIOpenAI (GPT-4, GPT-4o, etc.)
ANTHROPICAnthropic (Claude models)
GEMINIGoogle Gemini
VERTEX_AIGoogle Vertex AI
BEDROCKAmazon Bedrock
AZUREAzure OpenAI
MISTRALMistral AI
DEEPSEEKDeepSeek
X_AIxAI (Grok)
MOONSHOT_AIMoonshot AI
PERPLEXITYPerplexity
PORTKEYPortkey (gateway)
LITE_LLMLiteLLM (gateway)

Output types

You can optionally set an output type when pushing a prompt. This controls what format the LLM response should follow:

TypeDescription
TEXTPlain text output (default)
JSONJSON formatted output
SCHEMAStructured output validated against a defined schema
prompt.push(
    text="You are a helpful assistant.",
    output_type=OutputType.JSON,
)

Output schema

When output_type is set to SCHEMA, you can define a structured schema that the LLM response should conform to. This is useful when you need typed, validated responses from your LLM.

main.py
from deepeval.prompt import Prompt
from deepeval.prompt.api import PromptMessage, OutputType
from pydantic import BaseModel
from typing import List

class Source(BaseModel):
    url: str
    title: str

class ResponseSchema(BaseModel):
    answer: str
    confidence: float
    tags: List[str]
    sources: List[Source]

prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
prompt.push(
    messages=[
        PromptMessage(role="system", content="You are a helpful assistant."),
    ],
    output_type=OutputType.SCHEMA,
    output_schema=ResponseSchema,
)

The output_schema parameter accepts any Pydantic BaseModel class. Supported field types include primitives (str, int, float, bool), nested BaseModel classes, and List[...] for arrays of any supported type.

Interpolation types

Specify how variables are interpolated in your prompts:

TypeSyntaxExample
FSTRING{variable}Hello, {name}!
MUSTACHE{{variable}}Hello, {{name}}!
MUSTACHE_WITH_SPACE{{ variable }}Hello, {{ name }}!
DOLLAR_BRACKETS${variable}Hello, ${name}!
JINJA{% ... %}{% if admin %}Hello!{% endif %}

What about Tools?

You can create and update tools using prompts by pushing prompts with tools. Tools in Confident AI are identified using their names — passing a tool with a new name creates a tool and passing a tool with an existing name updates the tool on the platform. Each push creates a new commit that tracks the tool configuration. Here's how you can create / update tools:

from deepeval.prompt import Prompt, Tool
from deepeval.prompt.api import ToolMode
from pydantic import BaseModel

class ToolInputSchema(BaseModel):
    result: str
    confidence: float

prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
tool = Tool(
    name="SearchTool",
    description="Search functionality",
    mode=ToolMode.STRICT,
    structured_schema=ToolInputSchema,
)

# Use with push() to create a new commit with the tool
prompt.push(
    text="This a prompt for a tool using agent",
    tools=[tool]
)

tool_2 = Tool(
    name="SearchTool",
    description="New search functionality",
    mode=ToolMode.STRICT,
    structured_schema=ToolInputSchema,
)

# Create a new commit with the new updated tool using 'push'
prompt.push(
    text="This a prompt for a tool using agent",
    tools=[tool_2]
)

Prompts in CI/CD

Automate prompt tracking as part of your CI/CD pipeline. A common pattern is to push prompt commits whenever your prompt files change:

prompts-ci.yml
name: Push Prompt Commits

on:
  push:
    paths:
      - "prompts/**"

jobs:
  push-prompts:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: "3.11"

      - name: Install dependencies
        run: pip install deepeval

      - name: Push prompts
        env:
          CONFIDENT_API_KEY: ${{ secrets.CONFIDENT_API_KEY }}
        run: python scripts/push_prompts.py

Your push_prompts.py script can read prompt files and push them:

scripts/push_prompts.py
from deepeval.prompt import Prompt

# Read your prompt content from file or config
with open("prompts/assistant.txt") as f:
    prompt_text = f.read()

prompt = Prompt(alias="assistant-prompt")
prompt.push(text=prompt_text)

print("Prompt commit pushed successfully!")

Next Steps

Now that you can push prompts programmatically, learn how to pull them into your app for usage.

Pull Prompts

Pull prompt versions into your code for use in your LLM app.

Managing prompts in production?Prompt versioning, A/B testing, and rollbacks for production teamsBook a demo
Built byConfident AI