Launch Week 02 wrapped — explore all five launches
Confident AI
Docs
Pricing
Products
LLM Evaluation
LLM Evaluation

Benchmark LLM systems with research-backed metrics.

LLM Observability
LLM Observability

Trace, monitor, and alert on production LLM systems.

AI Red Teaming
AI Red Teaming

Stress-test LLM apps against adversarial attacks.

AI Governance
AI Governance

Enforce AI standards and controls across teams.

Book a DemoSign Up
Book a DemoSign Up

Trace Broadcasting

Trace broadcasting lets you send the same OpenTelemetry traces to multiple destinations at once — for example, your own data warehouse for long-term storage, plus Confident AI for LLM observability and online evaluations.

Because Confident AI accepts standard OTLP/HTTP, any pipeline that produces OTLP can broadcast a copy of every trace to https://otel.confident-ai.com/v1/traces. No proprietary protocol or wrapper SDK is required.

Overview

Common reasons teams broadcast traces:

  • Compliance / data residency — keep a copy of every trace in an internal warehouse before anything leaves their network.
  • Vendor independence — keep raw spans in their own infrastructure so they can switch or add observability vendors later.
  • Specialized backends — use a general-purpose APM (Datadog, Tempo, Jaeger) for service monitoring, and Confident AI for LLM-specific evaluation.
  • Sampling separation — keep 100% of traces locally for debugging, but only send a sampled subset externally.

There are two equivalent ways to broadcast: configure an OpenTelemetry Collector (a small standalone binary) that fans out traces, or attach multiple exporters directly inside your application. Both produce the same result.

Architecture

Via the Collector

sequenceDiagram
    participant App as Your Application
    participant Collector as OpenTelemetry Collector
    participant Warehouse as Data Warehouse
    participant Confident as Confident AI

    App->>Collector: Export OTLP spans

    par Broadcast
        Collector->>Warehouse: Send via OTLP / Kafka / file exporter
    and
        Collector->>Confident: Send via OTLP/HTTP<br/>(x-confident-api-key)
    end

Recommended for production — buffering, retries, sampling, and PII scrubbing all live in one centralized place.

Via the SDK

sequenceDiagram
    participant Code as Your Code
    participant Provider as TracerProvider
    participant ProcA as BatchSpanProcessor A
    participant ProcB as BatchSpanProcessor B
    participant Warehouse as Data Warehouse
    participant Confident as Confident AI

    Code->>Provider: span ends

    par Independent export
        Provider->>ProcA: onEnd(span)
        ProcA->>Warehouse: OTLP export
    and
        Provider->>ProcB: onEnd(span)
        ProcB->>Confident: OTLP/HTTP export
    end

Simpler — good for single-service apps. Each BatchSpanProcessor batches and retries independently, so a failure on one destination doesn't affect the other.

Setup

Pick whichever flavor fits your stack — both achieve the same broadcast.

otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      http:
      grpc:

exporters:
  otlphttp/warehouse:
    endpoint: https://traces.internal.yourcompany.com
    headers:
      authorization: Bearer ${env:WAREHOUSE_API_KEY}

  otlphttp/confident:
    endpoint: https://otel.confident-ai.com
    headers:
      x-confident-api-key: ${env:CONFIDENT_API_KEY}

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlphttp/warehouse, otlphttp/confident]

Load this into a running OpenTelemetry Collector — see the official Collector docs for deployment options. Listing both exporters in the same pipeline is all that's needed; every span goes to both.

After this, emit spans as you normally would — every span flows to both destinations.

Advanced Collector Features

These features are unique to the Collector path. They let you change broadcast behavior without touching application code.

Selective broadcast

To send only LLM-tagged spans to Confident AI while keeping 100% in the warehouse, use the routing connector:

otel-collector-config.yaml
connectors:
  routing:
    default_pipelines: [traces/warehouse]
    table:
      - context: span
        statement: route() where attributes["confident.span.type"] != nil
        pipelines: [traces/warehouse, traces/confident]

Sampling

To keep 100% locally but only sample 10% (plus all errors) to Confident AI:

otel-collector-config.yaml
processors:
  tail_sampling/confident:
    decision_wait: 10s
    policies:
      - name: errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: random
        type: probabilistic
        probabilistic: { sampling_percentage: 10 }

Apply it to the Confident AI pipeline only, leaving the warehouse pipeline unsampled.

PII scrubbing

Strip or hash sensitive fields before they leave your network:

otel-collector-config.yaml
processors:
  attributes/redact:
    actions:
      - { key: user.email, action: hash }
      - { key: http.request.header.authorization, action: delete }

Then add attributes/redact to the pipeline's processors list.

Combining with Distributed Tracing

If you already use distributed tracing across multiple services, point all services at a shared Collector and let it handle the broadcast:

sequenceDiagram
    participant ServiceA as Service A
    participant ServiceB as Service B
    participant ServiceC as Service C
    participant Collector as OpenTelemetry Collector
    participant Warehouse as Data Warehouse
    participant Confident as Confident AI

    ServiceA->>ServiceB: Request + traceparent
    ServiceB->>ServiceC: Request + traceparent

    par
        ServiceA->>Collector: OTLP spans
    and
        ServiceB->>Collector: OTLP spans
    and
        ServiceC->>Collector: OTLP spans
    end

    par
        Collector->>Warehouse: All spans
    and
        Collector->>Confident: All spans
    end

Because traceparent is propagated end-to-end, every destination receives a complete, unified trace.

Best Practices

Set Confident AI attributes

Broadcasting only changes where spans go, not what they contain. Spans must still carry the confident.* attributes (e.g. confident.span.type, confident.span.input, confident.llm.model) to render correctly in Observatory. See Span-Level Attribute Mappings.

Prefer the Collector in production

Once you have more than one service, a Collector is strongly recommended:

  • A single buffer absorbs spikes instead of every app holding its own queue.
  • Network blips to either destination only affect the Collector — your apps stay snappy.
  • You can change destinations, sampling, or PII rules without redeploying app code.

Use HTTP, not gRPC

Confident AI's OTLP endpoint accepts HTTP only. Use otlphttp in the Collector and OTLPSpanExporter from opentelemetry-exporter-otlp-proto-http in the SDK.

Set environment per pipeline

Use OTEL_RESOURCE_ATTRIBUTES to control which Confident AI environment traces land in:

OTEL_RESOURCE_ATTRIBUTES="confident.trace.environment=production"

For different environments per destination, run two Collector pipelines with different resource processors.

Debug sinks in isolation

When traces look wrong, disable one exporter at a time to confirm whether the issue is upstream or specific to one destination.

Need help wiring this into your stack?Bring traces and evals into the tools your team already usesTalk to an expert

Last updated on

Built byConfident AI