Provision Projects for Agents on the Fly

Overview

This guide is for teams that want to create Confident AI projects programmatically instead of creating them manually in the platform UI. It is especially useful for internal agent-building platforms, multi-tenant products, and proofs of concept where each agent (or tenant or customer) should have its own isolated Confident AI project.

For example, imagine an enterprise platform where every user builds their own agents. Each agent should get its own Confident AI project so its traces and evaluations stay isolated from every other agent on the platform.

In this guide, you will:

• Configure the Admin SDK with one Organization API Key.
• Create a project in code for each agent the moment it is built.
• Store the returned Project API Key with the project ID for later trace routing.
• Route traces to the correct project by setting the Project API Key on each trace.

By the end, your platform will be able to provision an isolated Confident AI project per agent on demand and send each agent’s traces, datasets, and evaluations to the right workspace.

Build It

$
pip install confidentai deepeval

$
export CONFIDENT_ORG_API_KEY="confident_us_org_..."

1
from confidentai import ConfidentAI
2
3
confident_ai = ConfidentAI()

1
from app.confident import confident_ai
2
from app.storage import save_application_project
3
4
def onboard_application(application_slug: str):
5
    new_project = confident_ai.projects.create(name=application_slug)
6
7
    save_application_project(
8
        application_slug,
9
        project_id=new_project.project.id,
10
        project_api_key=new_project.api_key.value,
11
    )
12
    return new_project.project.id

1
from openai import OpenAI
2
from deepeval.tracing import observe, update_current_trace
3
from app.storage import load_application_project
4
5
client = OpenAI()
6
7
@observe()
8
def run_for_application(application_slug: str, query: str):
9
    application = load_application_project(application_slug)
10
    update_current_trace(confident_api_key=application.project_api_key)
11
12
    return client.chat.completions.create(
13
        model="gpt-4o",
14
        messages=[{"role": "user", "content": query}],
15
    ).choices[0].message.content

1
from app.onboarding import onboard_application
2
from app.agent import run_for_application
3
4
onboard_application("support-bot")
5
run_for_application("support-bot", "What's on my agenda today?")

Next Steps

Now that each tenant or application can route traces to its own project, use these sections to extend the workflow: