LiteLLM

Overview

LiteLLM is a Python SDK and proxy server for calling models from multiple providers through an OpenAI-compatible interface. Confident AI traces and evaluates your LiteLLM calls through confident-trace, Confident AI's OpenTelemetry-native tracing SDK for Python and TypeScript.

The integration captures the following data from supported calls:

• LLM spans — model names, timing, status, and token usage
• Messages — input/output messages and tool-call data returned by the model
• Streaming output — response content as your application consumes it

The Python quickstart uses the native LiteLLM SDK. TypeScript uses an OpenAI client connected to a LiteLLM proxy; there is no native TypeScript LiteLLM hook.

Auto-Instrument

1. Install Dependencies

   Install confident-trace alongside the client used in the examples:

   PythonTypeScript

   pip install confident-trace

2. Set Your API Keys

   Get your project API key from Confident AI and set the credentials for your model client:

   export CONFIDENT_API_KEY="<your-confident-project-key>"
   
   # Python native SDK: credentials for the provider you call
   export OPENAI_API_KEY="<your-openai-key>"
   
   # TypeScript / OpenAI proxy clients
   export LITELLM_API_KEY="<your-proxy-key>"
   export LITELLM_BASE_URL="http://localhost:4000/v1"

3. Instrument LiteLLM

   Call init() once before making model calls. Initialize before importing LiteLLM function aliases so the calls use the instrumented functions. In the TypeScript example, replace gateway-model with an alias from your proxy configuration.

   PythonTypeScript

   import os
   from confident_trace import init, shutdown
   
   init()
   import litellm
   
   
   try:
       response = litellm.completion(
           model="openai/gpt-4o-mini",
           messages=[
               {"role": "user", "content": "Explain OpenTelemetry in one sentence."}
           ],
       )
       print(response.choices[0].message.content)
   finally:
       shutdown()

4. Run LiteLLM

   PythonTypeScript

   python main.py

   Done ✅. Open the Observatory in your Confident AI project to inspect the trace and its model-call spans.

OpenAI-Compatible Clients

If you already use the OpenAI SDK, keep it and point the client at the gateway. Install openai>=1.109,<4 for Python or openai>=7.10.0 <8 for TypeScript alongside confident-trace.

Register the exact proxy base URL in both languages; the TypeScript quickstart already demonstrates this path.

import os
from openai import OpenAI
from confident_trace import init

base_url = "http://localhost:4000/v1"
init(litellm_proxy_urls=[base_url])
client = OpenAI(base_url=base_url, api_key=os.environ["LITELLM_API_KEY"])

import OpenAI from "openai";
import { init } from "confident-trace";

const baseURL = "http://localhost:4000/v1";
const runtime = init({ litellmProxyUrls: [baseURL] });
const client = new OpenAI({ baseURL, apiKey: process.env.LITELLM_API_KEY! });

These setup snippets replace the quickstart’s client setup; keep its shutdown handling. Use client.chat.completions.create(...) or client.responses.create(...), including streaming. Gateway routing still depends on valid model names and credentials. Matching ignores trailing slashes but includes the scheme, host, port, and full base path.

Existing LiteLLM callbacks remain untouched. The older "deepeval" callback integration is a separate export path and is not required for this setup. Python native calls use the LiteLLM label; proxy calls use OpenAI with gateway identity litellm.

What Gets Captured

Each supported application call creates a model-call span. It nests under the active span; without a parent, it starts a new trace.

• Supported APIs — Python completion, acompletion, Router.completion, and Router.acompletion, including streaming. OpenAI proxy clients support Chat Completions and Responses create. Embeddings, legacy text completions, and other native LiteLLM APIs are outside this integration.
• Model and usage — the requested model or alias, the returned model when available, response ID, finish reasons, and token counts supplied by the gateway
• Messages and tools — normalized input/output messages and tool-call data; executing a tool is separate work and is not traced by this gateway integration
• Gateway identity — Native SDK spans use the LiteLLM integration label. OpenAI proxy spans retain OpenAI and add confident.gateway.name=litellm. Both record provider name litellm.

Messages follow the content policy, including capture opt-out, redaction, and configured size limits. Binary multimodal payloads are omitted.

Streaming

Streaming uses the same setup. Each example includes initialization and shutdown:

import os
from confident_trace import init, shutdown

init()
import litellm

try:
    stream = litellm.completion(
        model="openai/gpt-4o-mini",
        messages=[{"role": "user", "content": "Tell me a short story."}],
        stream=True,
    )
    try:
        for chunk in stream:
            print(chunk)
    finally:
        stream.close()
finally:
    shutdown()

import OpenAI from "openai";
import { init } from "confident-trace";

const baseURL = process.env.LITELLM_BASE_URL!;
const runtime = init({ litellmProxyUrls: [baseURL] });
const client = new OpenAI({ baseURL, apiKey: process.env.LITELLM_API_KEY! });

try {
  const stream = await client.chat.completions.create({
    model: "gateway-model",
    messages: [{ role: "user", content: "Tell me a short story." }],
    stream: true,
  });
  for await (const chunk of stream) {
    console.log(chunk);
  }
} finally {
  await runtime.shutdown();
}

For Python async code, use await litellm.acompletion(...) or await router.acompletion(...); consume streams with async for.

Set Trace Span Properties

Use a trace context to add properties you know before the call starts. It creates no extra span. Each example initializes tracing and creates its client before making the call.

import os
from confident_trace import init, shutdown, trace_context

init()
import litellm

try:
    with trace_context(
        tags=["support"],
        metadata={"gateway": "litellm"},
        user_id="user-42",
    ):
        response = litellm.completion(
            model="openai/gpt-4o-mini",
            messages=[
                {"role": "user", "content": "Explain OpenTelemetry in one sentence."}
            ],
        )
finally:
    shutdown()

import OpenAI from "openai";
import { init, traceContext } from "confident-trace";

const baseURL = process.env.LITELLM_BASE_URL!;
const runtime = init({ litellmProxyUrls: [baseURL] });
const client = new OpenAI({ baseURL, apiKey: process.env.LITELLM_API_KEY! });

try {
  const response = await traceContext(
    { tags: ["support"], metadata: { gateway: "litellm" }, userId: "user-42" },
    () =>
      client.chat.completions.create({
        model: "gateway-model",
        messages: [{ role: "user", content: "Explain OpenTelemetry in one sentence." }],
      }),
  );
} finally {
  await runtime.shutdown();
}

See trace context for every supported property.

Instrumenting Multi-Turn

You do not need turn() just to trace a single model call. Use it when you want to define a conversational boundary, such as grouping two calls into one turn. Reuse the same thread ID on later turns to group them into one conversation.

Each example includes initialization and shutdown. Use the same environment variables as the quickstart.

import os
from confident_trace import init, shutdown, turn

init()
import litellm

try:
    with turn("support-turn", thread_id="chat-42"):
        context = litellm.completion(
            model="openai/gpt-4o-mini",
            messages=[
                {
                    "role": "user",
                    "content": "List two useful facts about OpenTelemetry.",
                }
            ],
        )
        answer = litellm.completion(
            model="openai/gpt-4o-mini",
            messages=[
                {
                    "role": "user",
                    "content": f"Summarize these facts: {context.choices[0].message.content}",
                }
            ],
        )
finally:
    shutdown()

import OpenAI from "openai";
import { init, turn } from "confident-trace";

const baseURL = process.env.LITELLM_BASE_URL!;
const runtime = init({ litellmProxyUrls: [baseURL] });
const client = new OpenAI({ baseURL, apiKey: process.env.LITELLM_API_KEY! });

try {
  const answer = await turn({ name: "support-turn", threadId: "chat-42" }, async () => {
    const context = await client.chat.completions.create({
      model: "gateway-model",
      messages: [
        { role: "user", content: "List two useful facts about OpenTelemetry." },
      ],
    });
    return client.chat.completions.create({
      model: "gateway-model",
      messages: [
        { role: "user", content: `Summarize these facts: ${JSON.stringify(context)}` },
      ],
    });
  });
} finally {
  await runtime.shutdown();
}

See threads for conversation grouping and turn properties.

Troubleshooting

• Missing content: check capture settings and the supported API list above. Gateway-side exporters have separate content controls.
• Duplicate records: avoid combining multiple instrumentors for the same client call. Client instrumentation and gateway-side export can also report separate views of one request.

For general setup issues, see troubleshooting.

Disable LiteLLM Instrumentation

Use init() to select which client SDKs to instrument. An empty list disables all automatic instrumentation:

from confident_trace import init

init(instrumentations=())
# To enable only the quickstart integration: instrumentations=("litellm",)

import { init } from "confident-trace";

init({ instrumentations: [] });
// To enable only the quickstart integration: instrumentations: ["openai"]

Apply the selection to your startup init() call. It controls SDK hooks, not individual gateway hosts, and does not change a gateway’s own export settings.

Next Steps

Last updated on September 11, 2026