Introducing Confident Monitor
Confident AI
Docs
Pricing
Products
LLM Evaluation
LLM Evaluation

Benchmark LLM systems with research-backed metrics.

LLM Observability
LLM Observability

Trace, monitor, and alert on production LLM systems.

AI Red Teaming
AI Red Teaming

Stress-test LLM apps against adversarial attacks.

AI Governance
AI Governance

Enforce AI standards and controls across teams.

Contact salesSign Up
Contact salesSign Up

Pull Prompts

Learn how to test and use prompts in your LLM app

Overview

You can pull a prompt version from Confident AI like how you would pull a dataset. It works by:

  • Providing Confident AI with the id and optionally version of the prompt you wish to retrieve
  • Confident AI will provide the non-interpolated version of the prompt
  • You will then interpolate the variables in code

You should pull prompts once and save it in memory instead of pulling it everytime you need to use it.

The examples on this page use the Confident AI SDK. Pulling takes the prompt's id, which you get back whenever you push to the prompt.

Pull Prompt By Version

  1. Pull prompt with its id

    Pull your prompt version by providing the prompt's id and the version:

    from confident_ai import ConfidentAI

client = ConfidentAI()

prompt = client.prompt(prompt_id="<PROMPT-ID>")
prompt.pull(version="00.00.01")

  2. Interpolate variables

    Now that you have your prompt template, interpolate any dynamic variables you may have defined in your prompt version.

    interpolated_prompt = prompt.interpolate(name="Joe")

    For example, if this is your prompt version:

    {
  "role": "system",
  "content": "You are a helpful assistant called {{ name }}. Speak normally like a human."
}

    And your interpolation type is {{ variable }}, interpolating the name (e.g. "Joe") would give you this prompt that is ready for use:

    {
  "role": "system",
  "content": "You are a helpful assistant called Joe. Speak normally like a human."
}

    And if you don't have any variables, you must still use the interpolate() method to create a copy of your prompt template to be used in your LLM application.

  3. Use interpolated prompt

    By now you should have an interpolated prompt version, for example:

    {
  "role": "system",
  "content": "You are a helpful assistant called Joe. Speak normally like a human."
}

    Which you can use to generate text from your LLM provider of choice. Here are some examples with OpenAI:

    main.py
    from confident_ai import ConfidentAI
from openai import OpenAI

client = ConfidentAI()

prompt = client.prompt(prompt_id="<PROMPT-ID>")
prompt.pull()
interpolated_prompt = prompt.interpolate() # interpolate prompt

response = OpenAI().chat.completions.create(
    model="gpt-4o-mini",
    messages=[message.model_dump(exclude_none=True) for message in interpolated_prompt],
)

print(response.choices[0].message.content)

You can also fetch all the versions associated with a prompt as shown below:

main.py
from confident_ai import ConfidentAI

client = ConfidentAI()

result = client.prompts.get_versions(prompt_id="<PROMPT-ID>")

Pull Prompts By Label

You can also pull specific versions of prompts using the label you assign to the versions on the platform.

from confident_ai import ConfidentAI

client = ConfidentAI()

prompt = client.prompt(prompt_id="<PROMPT-ID>")
prompt.pull(label="staging")

You must manually label each prompt version before pulling it. Click here to learn how to do so.

Pull Prompts By Commit

You can also pull specific snapshots of prompts using its id and the commit hash.

from confident_ai import ConfidentAI

client = ConfidentAI()

prompt = client.prompt(prompt_id="<PROMPT-ID>")
prompt.pull()

You can also fetch all the commits associated with a prompt as shown below:

main.py
from confident_ai import ConfidentAI

client = ConfidentAI()

result = client.prompts.get_commits(prompt_id="<PROMPT-ID>")

Pull Prompts By Branch

You can also pull specific snapshots of prompts using its id and the branch name.

from confident_ai import ConfidentAI

client = ConfidentAI()

prompt = client.prompt(prompt_id="<PROMPT-ID>")
prompt.pull(branch="MY-BRANCH-NAME")

Branch Operations

You can also perform branch operations such as creating a new branch, updating a branch name, deleting a branch and listing all branches.

main.py
from confident_ai import ConfidentAI

client = ConfidentAI()

result = client.prompts.get_branches(prompt_id="<PROMPT-ID>")

How Are Prompts Pulled?

Confident AI automatically caches prompts on the client side to minimize API call latency and ensure prompt availability, which is especially useful in production environments.

sequenceDiagram
    participant App as Your Application
    participant SDK as Confident AI SDK
    participant Cache as SDK Cache
    participant API as Confident AI API

    App->>SDK: Pull Prompt
    SDK->>Cache: Check cache
    Cache-->>SDK: Cached Prompt Data
    SDK-->>App: Prompt Data
    Note over Cache,API: Refetching (every 60s)
    Cache->>API: Pull prompt
    API-->>Cache: Update Prompt Cache

Customize refresh rate

By default, the cache is refetched every 60 seconds, where the SDK will automatically update the cached prompt with the up-to-date version from Confident AI. This can be overridden by setting the refresh parameter to a different value. Fetching is done asynchronously, so it will not block your application.

main.py
from confident_ai import ConfidentAI

client = ConfidentAI()

prompt = client.prompt(prompt_id="<PROMPT-ID>")
prompt.pull(refresh=60)
interpolated_prompt = prompt.interpolate(name="Joe")

Configure cache

To disable caching, you can set refresh=0. This will force an API call every time you pull the prompt, which is particularly useful for development and testing.

main.py
from confident_ai import ConfidentAI

client = ConfidentAI()

prompt = client.prompt(prompt_id="<PROMPT-ID>")
prompt.pull(refresh=0)
interpolated_prompt = prompt.interpolate(name="Joe")

Advanced Usage

As you learnt in earlier sections, prompts have the additional option to not just version text/messages but also model settings (provider, name, parameters), output type, and tools.

Using model settings

After pulling a prompt, you can access any model settings that were configured for the prompt version via the model_settings property. Model settings include the provider, model name, and model parameters (temperature, max_tokens, etc.).

main.py
from confident_ai import ConfidentAI
from openai import OpenAI

client = ConfidentAI()

prompt = client.prompt(prompt_id="<PROMPT-ID>")
prompt.pull()
interpolated_prompt = prompt.interpolate()
settings = prompt.model_settings

# Use model settings (provider, name, parameters) in your OpenAI call
response = OpenAI().chat.completions.create(
    model=settings.name,
    messages=[message.model_dump(exclude_none=True) for message in interpolated_prompt],
    temperature=settings.temperature,
    max_tokens=settings.max_tokens,
    top_p=settings.top_p,
    frequency_penalty=settings.frequency_penalty,
    presence_penalty=settings.presence_penalty,
    stop=settings.stop_sequence,
)

print(response.choices[0].message.content)

Using output type

After pulling a prompt, you can access the output configuration via the output_type and output_schema properties. output_schema is JSON Schema, so you can pass it straight to your model provider's structured output option. This is useful when you want to enforce structured outputs from your LLM.

main.py
from confident_ai import ConfidentAI
from confident_ai.prompts import PromptOutputType
from openai import OpenAI

client = ConfidentAI()

prompt = client.prompt(prompt_id="<PROMPT-ID>")
prompt.pull()
interpolated_prompt = prompt.interpolate()

response_format = None
if prompt.output_type == PromptOutputType.JSON:
    response_format = {"type": "json_object"}
elif prompt.output_type == PromptOutputType.SCHEMA:
    response_format = {
        "type": "json_schema",
        "json_schema": {"name": "output", "schema": prompt.output_schema},
    }

response = OpenAI().chat.completions.create(
    model="gpt-4o",
    messages=[message.model_dump(exclude_none=True) for message in interpolated_prompt],
    response_format=response_format,
)

print(response.choices[0].message.content)

The available output types are:

  • TEXT - Plain text output (default)
  • JSON - JSON formatted output (maps to {"type": "json_object"})
  • SCHEMA - Structured output validated against a schema (maps to {"type": "json_schema", ...}, with output_schema as the schema)

Using tools

After pulling a prompt, you can access the tools its commit pins via the tools property. Each tool contains:

  • name: The name of the tool
  • description: A description of what the tool does
  • schema: The JSON Schema of the tool's input (schema_ in Python)
  • mode: The tool mode (ALLOW_ADDITIONAL, NO_ADDITIONAL, or STRICT)
  • id and version: The tool, and the version of it the commit pins
main.py
from confident_ai import ConfidentAI
from confident_ai.common import ToolMode
from openai import OpenAI

client = ConfidentAI()

prompt = client.prompt(prompt_id="<PROMPT-ID>")
prompt.pull()
interpolated_prompt = prompt.interpolate()

# Convert prompt tools to OpenAI format
openai_tools = [
    {
        "type": "function",
        "function": {
            "name": tool.name,
            "description": tool.description,
            "parameters": tool.schema_,
            "strict": tool.mode == ToolMode.STRICT,
        },
    }
    for tool in prompt.tools or []
]

# Use tools in your OpenAI call
response = OpenAI().chat.completions.create(
    model="gpt-4o",
    messages=[message.model_dump(exclude_none=True) for message in interpolated_prompt],
    tools=openai_tools,
)

print(response.choices[0].message)

Using images

Images in a prompt come back as [CONFIDENT:IMAGE:<url>] markers, each holding a signed link. parse_multimodal splits the text into strings and Media in order, so you can build the content parts your MLLM of choice expects. Here's how you'd build them for openai:

main.py
from confident_ai import ConfidentAI, Media, parse_multimodal

client = ConfidentAI()

prompt = client.prompt(prompt_id="<PROMPT-ID>")
prompt.pull()

content = []
for part in parse_multimodal(prompt.interpolate()):
    if isinstance(part, Media):
        content.append({"type": "image_url", "image_url": {"url": part.url}})
    else:
        content.append({"type": "text", "text": part})

This example is for a text prompt. For a messages prompt, parse each message's content the same way.

Prompt Association

You can and should definitely associate prompt versions with test runs and traced data for Confident AI to let you know which version of your prompt performs best.

Evals

You can associate a prompt with your evals to get detailed insights on how each prompt and their versions are performing. It works by:

  • Pulling prompt via the Confident API
  • Logging prompts as a hyperparameter during evaluation

In Python these evals run locally with deepeval, so pull the prompt with deepeval's Prompt (as in the tracing example below) rather than the Confident AI SDK. In TypeScript and over the API they run remotely, so pull the prompt with the Confident AI SDK and log it by its id and type.

Simply add the pulled prompt instance as a free-form key-value pair to the hyperparameters argument in the evaluate() function

evaluate(
    ...
    hyperparameters={
        "Model": "YOUR-MODEL",
        "Prompt": prompt,
    },
)

This will automatically attribute the prompt used during this test run, which will allow you get detailed insights in the Confident AI platform.

Tracing

Associating prompts with LLM traces and spans is a great way to determine which prompts performs best in production.

  1. Setup tracing

    Attach the @observe decorator to functions/methods that make up your agent, and specify type llm for your LLM-calling functions.

    main.py
    from deepeval.tracing import observe

@observe(type="llm", model="gpt-4.1")
def your_llm_component():
    ...

  2. Pull and interpolate prompt

    Pull and interpolate the prompt version to use it for LLM generation.

    main.py
    from deepeval.tracing import observe
from deepeval.prompt import Prompt
from openai import OpenAI

@observe(type="llm", model="gpt-4.1")
def your_llm_component():
    prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
    prompt.pull()
    interpolated_prompt = prompt.interpolate(name="Joe")
    response = OpenAI().chat.completions.create(model="gpt-4o-mini", messages=interpolated_prompt)
    return response.choices[0].message.content

  3. Execute your function

    Then simply provide the prompt to the update_llm_span function.

    main.py
    from deepeval.tracing import observe, update_llm_span
from deepeval.prompt import Prompt
from openai import OpenAI

@observe(type="llm", model="gpt-4.1")
def your_llm_component():
    prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
    prompt.pull()
    interpolated_prompt = prompt.interpolate(name="Joe")
    response = OpenAI().chat.completions.create(model="gpt-4o-mini", messages=interpolated_prompt)
    update_llm_span(prompt=prompt)
    return response.choices[0].message.content

    This will automatically attribute the prompt used to the LLM span.

Switching Projects

You can pull and manage your prompts in any project by using that project's API key.

  • For default usage, set CONFIDENT_PROJ_API_KEY as an environment variable.
  • To target a specific project, pass its key as project_api_key when creating the client.
main.py
from confident_ai import ConfidentAI

client = ConfidentAI(project_api_key="<PROJECT-API-KEY>")

A client keeps the key it was created with, so to work across several projects in one script, create one client per project.

Next Steps

Now that you can pull prompts into your app, learn how to push them programmatically or run evaluations.

Automate Prompt Management

Push and manage prompts programmatically via the Confident API.

Run Evaluations

Evaluate your LLM app with metrics and datasets.

Managing prompts in production?Prompt versioning, A/B testing, and rollbacks for production teamsBook a demo

Last updated on

Built byConfident AI