Trace Broadcasting

Trace broadcasting lets you send the same OpenTelemetry traces to multiple destinations at once — for example, your own data warehouse for long-term storage, plus Confident AI for LLM observability and online evaluations.

Because Confident AI accepts standard OTLP/HTTP, any pipeline that produces OTLP can broadcast a copy of every trace to https://otel.confident-ai.com/v1/traces. No proprietary protocol or wrapper SDK is required.

Overview

Common reasons teams broadcast traces:

• Compliance / data residency — keep a copy of every trace in an internal warehouse before anything leaves their network.
• Vendor independence — keep raw spans in their own infrastructure so they can switch or add observability vendors later.
• Specialized backends — use a general-purpose APM (Datadog, Tempo, Jaeger) for service monitoring, and Confident AI for LLM-specific evaluation.
• Sampling separation — keep 100% of traces locally for debugging, but only send a sampled subset externally.

There are two equivalent ways to broadcast: configure an OpenTelemetry Collector (a small standalone binary) that fans out traces, or attach multiple exporters directly inside your application. Both produce the same result.

Architecture

Via the Collector

Recommended for production — buffering, retries, sampling, and PII scrubbing all live in one centralized place.

Via the SDK

Simpler — good for single-service apps. Each BatchSpanProcessor batches and retries independently, so a failure on one destination doesn’t affect the other.

Setup

Pick whichever flavor fits your stack — both achieve the same broadcast.

1
receivers:
2
  otlp:
3
    protocols:
4
      http:
5
      grpc:
6
7
exporters:
8
  otlphttp/warehouse:
9
    endpoint: https://traces.internal.yourcompany.com
10
    headers:
11
      authorization: Bearer ${env:WAREHOUSE_API_KEY}
12
13
  otlphttp/confident:
14
    endpoint: https://otel.confident-ai.com
15
    headers:
16
      x-confident-api-key: ${env:CONFIDENT_API_KEY}
17
18
service:
19
  pipelines:
20
    traces:
21
      receivers: [otlp]
22
      exporters: [otlphttp/warehouse, otlphttp/confident]

After this, emit spans as you normally would — every span flows to both destinations.

Advanced Collector Features

These features are unique to the Collector path. They let you change broadcast behavior without touching application code.

Selective broadcast

To send only LLM-tagged spans to Confident AI while keeping 100% in the warehouse, use the routing connector:

1
connectors:
2
  routing:
3
    default_pipelines: [traces/warehouse]
4
    table:
5
      - context: span
6
        statement: route() where attributes["confident.span.type"] != nil
7
        pipelines: [traces/warehouse, traces/confident]

Sampling

To keep 100% locally but only sample 10% (plus all errors) to Confident AI:

1
processors:
2
  tail_sampling/confident:
3
    decision_wait: 10s
4
    policies:
5
      - name: errors
6
        type: status_code
7
        status_code: { status_codes: [ERROR] }
8
      - name: random
9
        type: probabilistic
10
        probabilistic: { sampling_percentage: 10 }

Apply it to the Confident AI pipeline only, leaving the warehouse pipeline unsampled.

PII scrubbing

Strip or hash sensitive fields before they leave your network:

1
processors:
2
  attributes/redact:
3
    actions:
4
      - { key: user.email, action: hash }
5
      - { key: http.request.header.authorization, action: delete }

Then add attributes/redact to the pipeline’s processors list.

Combining with Distributed Tracing

If you already use distributed tracing across multiple services, point all services at a shared Collector and let it handle the broadcast:

Because traceparent is propagated end-to-end, every destination receives a complete, unified trace.

Best Practices

Set Confident AI attributes

Broadcasting only changes where spans go, not what they contain. Spans must still carry the confident.* attributes (e.g. confident.span.type, confident.span.input, confident.llm.model) to render correctly in Observatory. See Span-Level Attribute Mappings.

Prefer the Collector in production

Once you have more than one service, a Collector is strongly recommended:

• A single buffer absorbs spikes instead of every app holding its own queue.
• Network blips to either destination only affect the Collector — your apps stay snappy.
• You can change destinations, sampling, or PII rules without redeploying app code.

Use HTTP, not gRPC

Confident AI’s OTLP endpoint accepts HTTP only. Use otlphttp in the Collector and OTLPSpanExporter from opentelemetry-exporter-otlp-proto-http in the SDK.

Set environment per pipeline

Use OTEL_RESOURCE_ATTRIBUTES to control which Confident AI environment traces land in:

$
OTEL_RESOURCE_ATTRIBUTES="confident.trace.environment=production"

For different environments per destination, run two Collector pipelines with different resource processors.

Debug sinks in isolation

When traces look wrong, disable one exporter at a time to confirm whether the issue is upstream or specific to one destination.