Provision Projects for Agents on the Fly

Overview

This guide is for teams that want to create Confident AI projects programmatically instead of creating them manually in the platform UI. It is especially useful for internal agent-building platforms, multi-tenant products, and proofs of concept where each agent (or tenant or customer) should have its own isolated Confident AI project.

For example, imagine an enterprise platform where every user builds their own agents. Each agent should get its own Confident AI project so its traces and evaluations stay isolated from every other agent on the platform.

In this guide, you will:

• Configure the Admin SDK with one Organization API Key.
• Create a project in code for each agent the moment it is built.
• Store the returned Project API Key with the project ID for later trace routing.
• Route traces to the correct project by scoping each request to its Project API Key with confident-trace.

By the end, your platform will be able to provision an isolated Confident AI project per agent on demand and send each agent's traces, datasets, and evaluations to the right workspace.

flowchart TB
    Org["Your Confident AI Organization"]

    Org --> AppA["Application A Project"]
    Org --> AppB["Application B Project"]
    Org --> AppC["Application C Project"]

    AppA --> AppAData["Application A traces<br/>Application A datasets<br/>Application A evaluations"]
    AppB --> AppBData["Application B traces<br/>Application B datasets<br/>Application B evaluations"]
    AppC --> AppCData["Application C traces<br/>Application C datasets<br/>Application C evaluations"]

    classDef org fill:#f8fafc,stroke:#334155,stroke-width:2px
    classDef project fill:#eef2ff,stroke:#4f46e5,stroke-width:1px
    classDef data fill:#f0fdf4,stroke:#16a34a,stroke-width:1px
    class Org org
    class AppA,AppB,AppC project
    class AppAData,AppBData,AppCData data

Build It

1. Install SDKs

   The Admin SDK is available in both Python and TypeScript through confident-ai. You also need confident-trace to route each application's traces into its project.

   PythonTypeScript

   pip install confident-ai confident-trace

2. Configure Admin SDK

   Set CONFIDENT_ORG_API_KEY to your Organization API Key. The Admin SDK reads this variable by default when you create a client.

   export CONFIDENT_ORG_API_KEY="confident_us_org_..."

   PythonTypeScript

   from confident_ai import ConfidentAI
   
   confident_ai = ConfidentAI()

3. Provision Project

   Create a project for each application. The projects.create(...) call returns the new project and its first Project API Key. Store both values with that application so traces can be routed to the same project later.

   The storage helpers in this example represent your own database or persistence layer.

   PythonTypeScript

   from app.confident import confident_ai
   from app.storage import save_application_project
   
   def onboard_application(application_slug: str):
       new_project = confident_ai.projects.create(name=application_slug)
   
       save_application_project(
           application_slug,
           project_id=new_project.project.id,
           project_api_key=new_project.api_key.value,
       )
       return new_project.project.id

4. Route Traces

   The Project API Key determines which project receives a trace. Call init() once when your server starts, then — for each request — load the tenant's stored key and wrap the traced work in a project scope. Every span created inside that scope is exported to the tenant's project.

   PythonTypeScript

   from openai import OpenAI
   from confident_trace import init, span, project_context
   from app.storage import load_application_project
   
   init()
   client = OpenAI()
   
   @span(type="agent")
   def support_agent(query: str) -> str:
       return client.chat.completions.create(
           model="gpt-4o",
           messages=[{"role": "user", "content": query}],
       ).choices[0].message.content
   
   def run_for_application(application_slug: str, query: str):
       application = load_application_project(application_slug)
       with project_context(api_key=application.project_api_key):
           return support_agent(query)

5. Verify Routing

   Create a project for an application, execute the application, and confirm that the trace appears in the correct project. Send requests for two tenants back-to-back to confirm they don't leak into each other.

   PythonTypeScript

   from confident_trace import shutdown
   from app.onboarding import onboard_application
   from app.agent import run_for_application
   
   onboard_application("support-bot")
   onboard_application("sales-bot")
   
   try:
       run_for_application("support-bot", "What's on my agenda today?")
       run_for_application("sales-bot", "Draft a follow-up for the Acme deal.")
   finally:
       shutdown()

   Open the Observatory and switch to the support-bot project. The trace appears in that project, isolated from every other application — and the sales-bot project shows only its own trace.

   Done ✅. The workflow now creates a dedicated project per application and routes traces using a single Organization API Key.

Next Steps

Now that each tenant or application can route traces to its own project, use these sections to extend the workflow:

Last updated on October 2, 2026