# Track LLM Costs

Track the token usage and cost of your LLM calls

## Overview

Confident AI tracks the token usage and cost of your LLM calls, helping you identify high-cost models and heavy usage patterns across your application.

> Cost tracking only applies to [LLM
> spans](/docs/llm-tracing/features/span-types#llm-spans). If you haven't
> already, learn how to [configure span
> types](/docs/llm-tracing/features/span-types) first.

[Video](https://confident-docs.s3.us-east-1.amazonaws.com/llm-tracing:cost-tracking.mp4)

*LLM Cost Tracking*

## How It Works

Confident AI resolves token usage and cost for each LLM span in the following
order of precedence, and separately for both input and output tokens:

1. **Per-token costs and counts set in `confident-trace`** (via `span(...)` or `update_span()` / `updateSpan()`) take the **highest priority** and will always override any other source.
   - Integrations may provide the token count, but cost calculation happens the same way.
2. **Custom set model costs** — if you provide token counts but not per-token costs, Confident AI will use the pricing you've configured in your [Model Costs settings](/docs/llm-tracing/features/model-costs) to calculate the cost.
3. **Automatic inference** — if neither per-token costs nor project-level costs are available, Confident AI tokenizes the span's input/output text using a provider-specific tokenizer and internally looks up pricing based on the `model`.

> Automatic inference is only available for **OpenAI**, **Anthropic**, and
> **Gemini** models. For all other providers, supply token counts and costs
> manually or configure [Model Costs](/docs/llm-tracing/features/model-costs) for
> your project.

## Automatic Usage Capture

If you're using the [OpenAI integration](/docs/integrations/third-party/openai) or any other supported provider integration, you don't need to do anything: `init()` instruments the client and each call's `input_token_count` and `output_token_count` are captured on the LLM span using the standard OpenTelemetry GenAI attributes.

> **Streaming?** Usage for streamed responses depends on what the provider returns and only becomes available once the stream completes, so make sure you consume the whole stream. Some providers also need usage reporting switched on explicitly for streams — for OpenAI that's `stream_options={"include_usage": True}`.

## Track Token Usage Count

You can manually set the input and output token counts on an LLM span using `update_span()` / `updateSpan()`. This is useful when your provider returns token usage in the response and you want to log it precisely, or when you're calling a model no integration covers.

#### Python

```python title="main.py" {6-9}
from confident_trace import span, update_span

@span(type="llm", model="gpt-4o", provider="openai")
def generate_response(prompt: str) -> str:
    response = call_llm(prompt)
    update_span(
        input_token_count=response.usage.prompt_tokens,
        output_token_count=response.usage.completion_tokens,
    )
    return response.text
```

#### TypeScript

```typescript title="src/index.ts" {7-10} maxLines={0}
import { span, updateSpan } from "confident-trace";

const generateResponse = span(
  { name: "generate_response", type: "llm", model: "gpt-4o", provider: "openai" },
  async (prompt: string) => {
    const response = await callLlm(prompt);
    updateSpan({
      inputTokenCount: response.usage.promptTokens,
      outputTokenCount: response.usage.completionTokens,
    });
    return response.text;
  },
);
```

Token counts must be non-negative integers, and an explicit `0` is kept as `0` rather than treated as unset.

> Don't add a manual `llm` span around a call that an integration already instruments — you'll end up with two LLM spans (and double the cost) for one request. Use manual token counts only for calls no integration covers.

If you don't provide token counts and aren't using an integration, Confident AI will attempt to infer them by tokenizing the span's input and output text using the appropriate provider tokenizer. The table below summarizes each supported provider and its tokenization method.

| Provider  | Tokenizer                                                                        | Example Models                                       | Token Counting Method                                   |
| --------- | -------------------------------------------------------------------------------- | ---------------------------------------------------- | ------------------------------------------------------- |
| OpenAI    | [tiktoken](https://github.com/openai/tiktoken)                                   | `gpt-4o`, `gpt-4.1`, `o1`, `o3`                      | Client-side tokenization using model-specific encodings |
| Anthropic | [@anthropic-ai/tokenizer](https://www.npmjs.com/package/@anthropic-ai/tokenizer) | `claude-3.5-sonnet`, `claude-3.7-sonnet`, `claude-4` | Claude-specific tokenization algorithm                  |
| Google    | Gemini API                                                                       | `gemini-2.0-flash`, `gemini-2.5-pro`                 | Server-side token counting via API call                 |

See the [OpenAI documentation](https://platform.openai.com/docs/guides/text-generation/token-counting), [Anthropic documentation](https://docs.anthropic.com/claude/references/token-counting), or [Google documentation](https://ai.google.dev/pricing) for the most up-to-date pricing.

> Note that the input and output are calculated separately — you don't have to provide both to set the cost for either.

## Track Token Usage Cost

Once token counts are available (either set manually, captured by an integration, or inferred automatically), Confident AI resolves the per-token cost using the following precedence:

1. **Per-token costs set in code** — if you provide cost per input/output tokens directly via `span(...)` or `update_span()` / `updateSpan()`, these always take priority.
2. **Custom set model costs** — if per-token costs aren't set in code, Confident AI uses the pricing configured in your [Model Costs settings](/docs/llm-tracing/features/model-costs).
3. **Automatic price lookup** — if no project-level costs are configured, Confident AI looks up the per-token pricing internally based on the `model`. This is only available for **OpenAI**, **Anthropic**, and **Gemini** models.

If none of the above resolve a per-token cost, the cost for that side (input or output) is **not logged**.

### Explicit Cost Setting

Set the per-token costs explicitly on the span alongside your token counts. This spares you for provider models not supported by automatic price lookup — self-hosted models, custom gateways, or negotiated enterprise pricing.

> Explicit cost setting is best for teams that want programmatic control over cost. For teams wanting to set model costs on the platform directly, see [custom price lookup.](/docs/llm-tracing/features/token-usage-cost#custom-price-lookup)

#### Python

```python title="main.py" {5-6,11-12}
from confident_trace import span, update_span

@span(
    type="llm", model="my-model", provider="custom",
    cost_per_input_token=0.000001,
    cost_per_output_token=0.000002,
)
def generate_response(prompt: str) -> str:
    response = call_llm(prompt)
    update_span(
        input_token_count=response.usage.prompt_tokens,
        output_token_count=response.usage.completion_tokens,
    )
    return response.text
```

#### TypeScript

```typescript title="src/index.ts" {6-7,12-13} maxLines={0}
import { span, updateSpan } from "confident-trace";

const generateResponse = span(
  {
    name: "generate_response", type: "llm", model: "my-model", provider: "custom",
    costPerInputToken: 0.000001,
    costPerOutputToken: 0.000002,
  },
  async (prompt: string) => {
    const response = await callLlm(prompt);
    updateSpan({
      inputTokenCount: response.usage.promptTokens,
      outputTokenCount: response.usage.completionTokens,
    });
    return response.text;
  },
);
```

> Rates are **USD per token**, not per million tokens. A model priced at \$1.00 per million input tokens is `cost_per_input_token=0.000001`. Rates must be finite and non-negative.

You can pass the cost fields either on the span options (as above) or to `update_span()` / `updateSpan()` — whichever is more convenient. Either way they only apply to an LLM span; on any other span type the general fields are still applied but the LLM fields are skipped with a one-time warning.

### Custom Price Lookup

If you provide token counts but don't set per-token costs in code, Confident AI will use the pricing you've configured in your project's [Model Costs settings](/docs/llm-tracing/features/model-costs). This is useful when you want to manage pricing centrally without changing any code.

Model costs are matched against the `model` name on your LLM span using case-insensitive regex patterns, which match anywhere in the name unless anchored. For example:

- `^gpt-4o$` — matches only `gpt-4o`
- `gpt-4.*` — matches `gpt-4o`, `gpt-4o-mini`, `gpt-4-turbo`, etc.
- `claude-.*` — matches all Claude model variants

You can optionally restrict a cost rule to a specific provider, and set input and output costs independently per million tokens. See the full [Model Costs settings](/docs/llm-tracing/features/model-costs) page for setup instructions.

Product preview: Model Costs page with the inherit toggle off, a custom gpt-4.1 cost at $2 input and $8 output, and pre-configured pricing tabs by provider

*Configure Model Costs*

### Automatic Price Lookup

If you provide a supported `model` on your LLM span and neither SDK-level nor project-level costs are configured, Confident AI will automatically look up the per-token pricing and calculate the cost — no additional code needed.

#### Python

```python title="main.py" {3}
from confident_trace import span, update_span

@span(type="llm", model="gpt-4o", provider="openai")
def generate_response(prompt: str) -> str:
    output = call_llm(prompt)
    update_span(input=prompt, output=output)
    return output
```

#### TypeScript

```typescript title="src/index.ts" {4} maxLines={0}
import { span, updateSpan } from "confident-trace";

const generateResponse = span(
  { name: "generate_response", type: "llm", model: "gpt-4o", provider: "openai" },
  async (prompt: string) => {
    const output = await callLlm(prompt);
    updateSpan({ input: prompt, output });
    return output;
  },
);
```

> Automatic price lookup is only available for **OpenAI**, **Anthropic**, and
> **Gemini** models. For all other providers, set per-token costs manually or
> configure [Model Costs](/docs/llm-tracing/features/model-costs) for your
> project.

## Cost on Traces

Cost on traces are automatically set by summing up the cost of all LLM spans in said trace. Similar to LLM spans, trace cost defaults to null values if no LLM spans have non-null values.

## Next Steps

With cost tracking configured, continue setting up the rest of your instrumentation.

#### [Set Input/Output](/docs/llm-tracing/features/input-output)

Override the default input and output on traces and spans for better
visualization and evaluation.

#### [Thread Traces](/docs/llm-tracing/features/threads)

Group traces into threads to track multi-turn conversations and evaluate
entire workflows.