Portkey

Overview

Portkey is an AI gateway for routing model requests, managing provider access, and applying retries, fallbacks, and caching. Confident AI traces and evaluates your Portkey calls through confident-trace, Confident AI's OpenTelemetry-native tracing SDK for Python and TypeScript.

The integration captures the following data from supported calls:

• LLM spans — model names, timing, status, and token usage
• Messages — input/output messages and tool-call data returned by the model
• Streaming output — response content as your application consumes it

Auto-Instrument

1. Install Dependencies

   Install confident-trace alongside the client used in the examples:

   PythonTypeScript

   pip install confident-trace

2. Set Your API Keys

   Get your project API key from Confident AI and set the credentials for your model client:

   export CONFIDENT_API_KEY="<your-confident-project-key>"
   export PORTKEY_API_KEY="<your-gateway-key>"

3. Instrument Portkey

   Call init() once before making model calls. It instruments the installed native SDK; keep using your client normally. Replace @openai-prod/gpt-4o-mini with a provider slug and model configured in your Portkey Model Catalog.

   PythonTypeScript

   import os
   from confident_trace import init, shutdown
   from portkey_ai import Portkey
   
   init()
   client = Portkey(api_key=os.environ["PORTKEY_API_KEY"])
   
   try:
       response = client.chat.completions.create(
           model="@openai-prod/gpt-4o-mini",
           messages=[
               {"role": "user", "content": "Explain OpenTelemetry in one sentence."}
           ],
       )
       print(response.choices[0].message.content)
   finally:
       shutdown()

4. Run Portkey

   PythonTypeScript

   python main.py

   Done ✅. Open the Observatory in your Confident AI project to inspect the trace and its model-call spans.

OpenAI-Compatible Clients

If you already use the OpenAI SDK, keep it and point the client at the gateway. Install openai>=1.109,<4 for Python or openai>=7.10.0 <8 for TypeScript alongside confident-trace.

The public endpoint below is detected automatically. For a custom endpoint, also register the exact URL using portkey_proxy_urls / portkeyProxyUrls.

import os
from openai import OpenAI
from confident_trace import init

base_url = "https://api.portkey.ai/v1"
init()
client = OpenAI(base_url=base_url, api_key=os.environ["PORTKEY_API_KEY"])

import OpenAI from "openai";
import { init } from "confident-trace";

const baseURL = "https://api.portkey.ai/v1";
const runtime = init();
const client = new OpenAI({ baseURL, apiKey: process.env.PORTKEY_API_KEY! });

These setup snippets replace the quickstart’s client setup; keep its shutdown handling. Use client.chat.completions.create(...) or client.responses.create(...), including streaming. Gateway routing still depends on valid model names and credentials. Matching ignores trailing slashes but includes the scheme, host, port, and full base path.

For manual TypeScript setup, initialize export and use instrumentPortkey(client) from confident-trace/portkey with the native client, or instrumentOpenAI(client) from confident-trace/openai with an OpenAI client. Manual adapters do not require the preload.

What Gets Captured

Each supported application call creates a model-call span. It nests under the active span; without a parent, it starts a new trace.

• Supported APIs — Native Chat Completions and Responses create, including streaming; Python supports Portkey and AsyncPortkey. OpenAI proxy clients use those same API surfaces. Prompt-management APIs, embeddings, and separate SDK stream/parse helpers are outside this integration.
• Model and usage — the requested model or alias, the returned model when available, response ID, finish reasons, and token counts supplied by the gateway
• Messages and tools — normalized input/output messages and tool-call data; executing a tool is separate work and is not traced by this gateway integration
• Gateway identity — Native SDK spans use the Portkey integration label. OpenAI proxy spans retain OpenAI and add confident.gateway.name=portkey. Both record provider name portkey.

Messages follow the content policy, including capture opt-out, redaction, and configured size limits. Binary multimodal payloads are omitted.

Streaming

Streaming uses the same setup. Each example includes initialization and shutdown:

import os
from confident_trace import init, shutdown
from portkey_ai import Portkey

init()
client = Portkey(api_key=os.environ["PORTKEY_API_KEY"])

try:
    stream = client.chat.completions.create(
        model="@openai-prod/gpt-4o-mini",
        messages=[{"role": "user", "content": "Tell me a short story."}],
        stream=True,
    )
    try:
        for chunk in stream:
            print(chunk)
    finally:
        stream.close()
finally:
    shutdown()

import { Portkey } from "portkey-ai";
import { init } from "confident-trace";

const runtime = init();
const client = new Portkey({ apiKey: process.env.PORTKEY_API_KEY! });

try {
  const stream = await client.chat.completions.create({
    model: "@openai-prod/gpt-4o-mini",
    messages: [{ role: "user", content: "Tell me a short story." }],
    stream: true,
  });
  for await (const chunk of stream) {
    console.log(chunk);
  }
} finally {
  await runtime.shutdown();
}

For Python async code, use AsyncPortkey and await its chat.completions.create or responses.create call; consume streams with async for.

Set Trace Span Properties

Use a trace context to add properties you know before the call starts. It creates no extra span. Each example initializes tracing and creates its client before making the call.

import os
from confident_trace import init, shutdown, trace_context
from portkey_ai import Portkey

init()
client = Portkey(api_key=os.environ["PORTKEY_API_KEY"])

try:
    with trace_context(
        tags=["support"],
        metadata={"gateway": "portkey"},
        user_id="user-42",
    ):
        response = client.chat.completions.create(
            model="@openai-prod/gpt-4o-mini",
            messages=[
                {"role": "user", "content": "Explain OpenTelemetry in one sentence."}
            ],
        )
finally:
    shutdown()

import { Portkey } from "portkey-ai";
import { init, traceContext } from "confident-trace";

const runtime = init();
const client = new Portkey({ apiKey: process.env.PORTKEY_API_KEY! });

try {
  const response = await traceContext(
    { tags: ["support"], metadata: { gateway: "portkey" }, userId: "user-42" },
    () =>
      client.chat.completions.create({
        model: "@openai-prod/gpt-4o-mini",
        messages: [{ role: "user", content: "Explain OpenTelemetry in one sentence." }],
      }),
  );
} finally {
  await runtime.shutdown();
}

See trace context for every supported property.

Instrumenting Multi-Turn

You do not need turn() just to trace a single model call. Use it when you want to define a conversational boundary, such as grouping two calls into one turn. Reuse the same thread ID on later turns to group them into one conversation.

Each example includes initialization and shutdown. Use the same environment variables as the quickstart.

import os
from confident_trace import init, shutdown, turn
from portkey_ai import Portkey

init()
client = Portkey(api_key=os.environ["PORTKEY_API_KEY"])

try:
    with turn("support-turn", thread_id="chat-42"):
        context = client.chat.completions.create(
            model="@openai-prod/gpt-4o-mini",
            messages=[
                {
                    "role": "user",
                    "content": "List two useful facts about OpenTelemetry.",
                }
            ],
        )
        answer = client.chat.completions.create(
            model="@openai-prod/gpt-4o-mini",
            messages=[
                {
                    "role": "user",
                    "content": f"Summarize these facts: {context.choices[0].message.content}",
                }
            ],
        )
finally:
    shutdown()

import { Portkey } from "portkey-ai";
import { init, turn } from "confident-trace";

const runtime = init();
const client = new Portkey({ apiKey: process.env.PORTKEY_API_KEY! });

try {
  const answer = await turn({ name: "support-turn", threadId: "chat-42" }, async () => {
    const context = await client.chat.completions.create({
      model: "@openai-prod/gpt-4o-mini",
      messages: [
        { role: "user", content: "List two useful facts about OpenTelemetry." },
      ],
    });
    return client.chat.completions.create({
      model: "@openai-prod/gpt-4o-mini",
      messages: [
        { role: "user", content: `Summarize these facts: ${JSON.stringify(context)}` },
      ],
    });
  });
} finally {
  await runtime.shutdown();
}

See threads for conversation grouping and turn properties.

Troubleshooting

• Missing content: check capture settings and the supported API list above. Gateway-side exporters have separate content controls.
• Duplicate records: avoid combining multiple instrumentors for the same client call. Client instrumentation and gateway-side export can also report separate views of one request.

For general setup issues, see troubleshooting.

Disable Portkey Instrumentation

Pass init() a list of integration identifiers to opt in to only those integrations. The quickstart uses "portkey" in Python and "portkey" in TypeScript. OpenAI proxy clients use "openai", independently of the native gateway integration. An empty list disables all automatic instrumentation:

from confident_trace import init

init(instrumentations=())
# To enable only the quickstart integration: instrumentations=("portkey",)

import { init } from "confident-trace";

init({ instrumentations: [] });
// To enable only the quickstart integration: instrumentations: ["portkey"]

Apply the selection to your startup init() call. It controls SDK hooks, not individual gateway hosts, and does not change a gateway’s own export settings. Manually installed TypeScript adapters have their own restoration function.

Next Steps

Last updated on September 11, 2026