OpenTelemetry

OpenTelemetry is an open-source observability framework that allows teams to collect, analyze, and visualize telemetry data.

Overview

Confident AI can recieve traces on https://otel.confident-ai.com. To export traces using the OpenTelemetry SDK, you can configure your Collector with the official open-telemetry library.

Quickstart

Given below is the quickstart for exporting traces to Confident AI OTLP endpoint (which will then published to observatory) for different languages.

Set Environment Variables
First set your CONFIDENT_API_KEY and OTEL_EXPORTER_OTLP_ENDPOINT as an enviornment variable:
Bash
```
export CONFIDENT_API_KEY="confident_us..."
export OTEL_EXPORTER_OTLP_ENDPOINT="https://otel.confident-ai.com"
```

Trace your first LLM application

Install opentelemetry dependencies:

Bash

pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp-proto-http

Run the following code:

main.py

import os

from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

OTLP_ENDPOINT = os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT")
CONFIDENT_API_KEY = os.getenv("CONFIDENT_API_KEY")

trace_provider = TracerProvider()
exporter = OTLPSpanExporter(
    endpoint=f"{OTLP_ENDPOINT}/v1/traces",
    headers={"x-confident-api-key": CONFIDENT_API_KEY},
)
span_processor = BatchSpanProcessor(span_exporter=exporter)
trace_provider.add_span_processor(span_processor)
tracer = trace_provider.get_tracer("deepeval_tracer")

# Start a span
with tracer.start_as_current_span("confident-llm-span") as span:

    # Set attributes
    span.set_attribute("confident.trace.name", "example-trace")
    span.set_attribute("confident.span.type", "llm")
    span.set_attribute("confident.llm.model", "gpt-4o")
    span.set_attribute("confident.span.input", "What is the capital of France?")
    span.set_attribute("confident.span.output", "Paris")

trace_provider.force_flush()
print("Traces posted successfully to https://otel.confident-ai.com")

Run the code:

python main.py

The above example creates a new OpenTelemetry trace provider and sets OTLPSpanExporter to it so that the spans are exported to the Confident AI OTLP endpoint ONLY. If you wish to set OTLPSpanExporter to the existing OpenTelemetry trace provider, refer to the following example.

main.py

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter


OTLP_ENDPOINT = os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT")
CONFIDENT_API_KEY = os.getenv("CONFIDENT_API_KEY")

# Setup OpenTelemetry
if not isinstance(trace.get_tracer_provider(), TracerProvider):
    tracer_provider = TracerProvider()
    trace.set_tracer_provider(tracer_provider)
else:
    tracer_provider = trace.get_tracer_provider()

exporter = OTLPSpanExporter(
    endpoint=f"{OTLP_ENDPOINT}/v1/traces",
    headers={"x-confident-api-key": CONFIDENT_API_KEY},
)

span_processor = BatchSpanProcessor(span_exporter=exporter)
tracer_provider.add_span_processor(span_processor)
tracer = trace.get_tracer("deepeval_tracer")

Install Node.js dependencies

npm init -y
npm install @opentelemetry/api @opentelemetry/sdk-trace-node @opentelemetry/sdk-trace-base @opentelemetry/exporter-trace-otlp-proto dotenv

Install TypeScript and ts-node

npm install -D typescript ts-node @types/node

Create index.ts file. This file contains the code for creating an LLM span.

index.ts

import * as opentelemetry from '@opentelemetry/api';
import { NodeTracerProvider } from '@opentelemetry/sdk-trace-node';
import { BatchSpanProcessor } from '@opentelemetry/sdk-trace-base';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-proto';

// Environment variables (similar to Python's os.getenv)
const OTLP_ENDPOINT = process.env.OTEL_EXPORTER_OTLP_ENDPOINT;
const CONFIDENT_API_KEY = process.env.CONFIDENT_API_KEY;

// Add validation for required environment variables
if (!OTLP_ENDPOINT) {
    throw new Error('OTEL_EXPORTER_OTLP_ENDPOINT environment variable is required');
}

// Create OTLP exporter with HTTPS support
const otlpExporter = new OTLPTraceExporter({
    url: `${OTLP_ENDPOINT}/v1/traces`,
    headers: {
        'x-confident-api-key': CONFIDENT_API_KEY || ''
    },
});

// Set up the tracer provider with the batch span processor
const provider = new NodeTracerProvider({
    spanProcessors: [new BatchSpanProcessor(otlpExporter)]
});

// Register the provider globally
opentelemetry.trace.setGlobalTracerProvider(provider);

// Create a tracer
const tracer = opentelemetry.trace.getTracer('confident-llm-tracer');

async function main() {
    // Start a span
    tracer.startActiveSpan('confident-llm-span-typescript', (span) => {
        // Set attributes
        span.setAttributes({
            'confident.trace.name': 'example-trace',
            'confident.span.type': 'llm',
            'confident.llm.model': 'gpt-4o',
            'confident.span.input': 'What is the capital of France?',
            'confident.span.output': 'Paris'
        });

        // Simulate some work here
        console.log('Processing LLM request...');

        // End the span
        span.end();
    });

    // Shut down the provider to ensure traces are flushed before the script exits
    await provider.shutdown();
    console.log(`Trace posted successfully to ${OTLP_ENDPOINT}.`);
}

main().catch((error) => {
    console.error('Error sending traces:', error);
    process.exit(1);
});

Create a basic tsconfig.json file:

tsconfig.json

{
    "compilerOptions": {
        "target": "ES2020",
        "module": "commonjs",
        "esModuleInterop": true,
        "skipLibCheck": true,
        "forceConsistentCasingInFileNames": true,
        "outDir": "./dist"
    }
}

Run the code:

npx ts-node index.ts

Install Go (version 1.19 or later recommended):

Set up environment variables:

export OTLP_ENDPOINT="otel.confident-ai.com"
export CONFIDENT_API_KEY="<your-confident-api-key>"

Initialize Go Module:

go mod init go-example

Create main.go file.

main.go

package main

import (
    "context"
    "fmt"
    "log"
    "os"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
    "go.opentelemetry.io/otel/propagation"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
)

func main() {
    endpoint := os.Getenv("OTLP_ENDPOINT")
    confidentApiKey := os.Getenv("CONFIDENT_API_KEY")

    exporter, err := otlptracehttp.New(context.Background(),
        otlptracehttp.WithEndpoint(endpoint),
        otlptracehttp.WithHeaders(map[string]string{"x-confident-api-key": confidentApiKey}),
    )
    if err != nil {
        log.Fatalf("failed to create OTLP exporter: %v", err)
    }

    tp := sdktrace.NewTracerProvider(sdktrace.WithBatcher(exporter))
    otel.SetTracerProvider(tp)
    otel.SetTextMapPropagator(propagation.TraceContext{})
    defer func() {
        fmt.Println("Shutting down tracer provider...")
        _ = tp.Shutdown(context.Background())
    }()

    _, span := otel.Tracer("example.com/otel-openai").Start(context.Background(), "chat gpt-4o")
    defer func() {
        span.End()
        fmt.Println("Span ended - Trace posted successfully to:", endpoint)
    }()

    span.SetAttributes(
        attribute.String("confident.span.type", "llm"),
        attribute.String("confident.llm.model", "gpt-4o"),
        attribute.String("confident.span.input", "input"),
        attribute.String("confident.span.output", "output"),
    )

}

Install dependencies:

go mod tidy

Run the code:

go run main.go

Create Gemfile file. This file contains the dependencies for the Ruby application.

Gemfile

source 'https://rubygems.org'

gem 'opentelemetry-sdk'
gem 'opentelemetry-exporter-otlp'

Install dependencies:

bundle install

Create example.rb file. This file contains the code for creating an LLM span.

example.rb

require 'opentelemetry/sdk'
require 'opentelemetry/exporter/otlp'

# Ensure OTLP endpoint and API key are set
OTLP_ENDPOINT    = ENV.fetch('OTEL_EXPORTER_OTLP_ENDPOINT') { abort 'Set OTEL_EXPORTER_OTLP_ENDPOINT' }
CONFIDENT_API_KEY = ENV.fetch('CONFIDENT_API_KEY')       { abort 'Set CONFIDENT_API_KEY' }

OpenTelemetry::SDK.configure do |c|
c.add_span_processor(
    OpenTelemetry::SDK::Trace::Export::BatchSpanProcessor.new(
    OpenTelemetry::Exporter::OTLP::Exporter.new(
        endpoint: "#{OTLP_ENDPOINT}/v1/traces",
        headers:  { 'x-confident-api-key' => CONFIDENT_API_KEY },
    )
    )
)
end

tracer = OpenTelemetry.tracer_provider.tracer(__FILE__)

tracer.in_span('confident-llm-span-ruby') do |span|
    span.set_attribute('confident.trace.name', 'example-trace')
    span.set_attribute('confident.span.type', 'llm')
    span.set_attribute('confident.llm.model', 'gpt-4o')
    span.set_attribute('confident.span.input', 'What is the capital of France?')
    span.set_attribute('confident.span.output', 'Paris')

    puts 'Span created successfully!'
end

# Flush and allow time for HTTP export
OpenTelemetry.tracer_provider.shutdown
puts "Traces posted successfully to #{OTLP_ENDPOINT}."
sleep 2

Run the code:

ruby example.rb

Create a New Console App

dotnet new console -n ConfidentLLMExample
cd ConfidentLLMExample

Add Required NuGet Packages

dotnet add package OpenTelemetry
dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol

Create Program.cs file. This file contains the code for creating an LLM span.

Program.cs

using System;
using OpenTelemetry;
using OpenTelemetry.Trace;
using OpenTelemetry.Resources;
using OpenTelemetry.Exporter;
using System.Threading.Tasks;

class Program
{
    static async Task Main(string[] args)
    {
        var otlpEndpoint = Environment.GetEnvironmentVariable("OTEL_EXPORTER_OTLP_ENDPOINT");
        var confidentApiKey = Environment.GetEnvironmentVariable("CONFIDENT_API_KEY");

        Console.WriteLine($"OTLP Endpoint: {otlpEndpoint}");
        Console.WriteLine($"API Key configured: {!string.IsNullOrEmpty(confidentApiKey)}");

        using var tracerProvider = Sdk.CreateTracerProviderBuilder()
            .SetResourceBuilder(ResourceBuilder.CreateDefault()
                .AddService("ConfidentLLMService"))
            .AddSource("ConfidentLLMTracer")
            .AddOtlpExporter(options =>
            {
                options.Endpoint = new Uri($"{otlpEndpoint}/v1/traces");
                options.Headers = $"x-confident-api-key={confidentApiKey}";
                options.Protocol = OtlpExportProtocol.HttpProtobuf;
                // Add timeout and retry configuration
                options.TimeoutMilliseconds = 30000;
            })
            .Build();

        var tracer = tracerProvider.GetTracer("ConfidentLLMTracer");

        Console.WriteLine("Starting span...");
        using (var currentSpan = tracer.StartActiveSpan("confident-llm-span-csharp"))
        {
            currentSpan.SetAttribute("confident.trace.name", "example-trace");
            currentSpan.SetAttribute("confident.span.type", "llm");
            currentSpan.SetAttribute("confident.llm.model", "gpt-4o");
            currentSpan.SetAttribute("confident.span.input", "What is the capital of France?");
            currentSpan.SetAttribute("confident.span.output", "Paris");

            Console.WriteLine("Span created with attributes. It will end after 5 seconds.");
            await Task.Delay(5000);
        }

        Console.WriteLine("Span ended. Flushing traces...");

        // Force flush traces before exiting
        tracerProvider.ForceFlush();

        // Wait a bit to ensure traces are sent
        await Task.Delay(2000);
        Console.WriteLine($"Trace posted successfully to {otlpEndpoint}.");
    }
}

Build and Run

dotnet run

🎉 Congratulations! You have successfully sent traces. Go to the the Observatory section on Confident AI to check it out.

Click to see native python implementation using DeepEval

DeepEval provides a native ConfidentSpanExporter that directly exports traces to Confident AI observatory.

example.py

import time
import json
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

from deepeval.tracing.otel.exporter import ConfidentSpanExporter

# Set up tracer provider
tracer_provider = trace.get_tracer_provider()
if not isinstance(tracer_provider, TracerProvider):
    trace.set_tracer_provider(TracerProvider())

# Add confident span exporter wrapped in batch span processor
tracer_provider.add_span_processor(BatchSpanProcessor(ConfidentSpanExporter()))

# Get tracer
tracer = trace.get_tracer("deepeval_tracer")

# set attributes
with tracer.start_as_current_span("confident-llm-span") as span:
    span.set_attribute("confident.trace.name", "example-trace")
    span.set_attribute("confident.span.type", "llm")
    span.set_attribute("confident.llm.model", "gpt-4o")
    span.set_attribute("confident.span.input", "What is the capital of France?")
    span.set_attribute("confident.span.output", "Paris")

Understanding OTEL with Confident AI

In this section, we will mainly discuss:

gen_ai attribute and event conventions
Confident AI specific conventions
Advanced configurations

OTEL endpoints

Confident AI offers the https://otel.confident-ai.com endpoint that accepts OpenTelemetry traces in the OTLP format. Please note that Confident AI does not support GRPC for the OpenTelemetry endpoint. Please use HTTP instead.

Attributes

Confident AI adheres to the GenAI semantic convention and adds an extra layer on top of it to capture additional data about the LLM applications. Confident AI uses confident.* namespace to map specific attributes with llm tracing data model. These specific attributes always take precedence over gen_ai.* conventions and are recommended for all users that are manually instrumenting their applications.

Advanced configurations

The enviornment allows you to configure where your trace belongs on Confident AI (defaulted to "development"). You can configure the enviornment and sampling rate of traces by setting the following two enviornment variables:

OTEL_RESOURCE_ATTRIBUTES="confident.trace.environment=production"

Trace-Level Attribute Mappings

These are the attributes specific to Confident AI traces similar to tracing features. The trace level attributes are set in the span attributes using the confident.trace.* namespace.

Name

The trace name is displayed in the UI. You can customize it based on your liking for better UI display using the following attribute:

"confident.trace.name" (of type str) used for updating trace name

with tracer.start_as_current_span("custom_span") as span:
    span.set_attribute("confident.trace.name", "test_trace")

span.setAttributes({
  "confident.trace.name": "example-trace",
});

span.SetAttributes(
    attribute.String("confident.trace.name", "example-trace"),
)

span.setAttributes({
    "confident.trace.name": "example-trace",
});

span.SetAttribute("confident.trace.name", "example-trace");

Input/Output

You can set trace input and output at runtime using the following attributes:

"confident.trace.input" (of type Any) used for updating trace input
"confident.trace.output" (of type Any) used for updating trace output

with tracer.start_as_current_span("custom_span") as span:
    span.set_attribute("confident.trace.input", input)
    span.set_attribute("confident.trace.output", output)

span.setAttributes({
  "confident.trace.input": input,
  "confident.trace.output": output,
});

span.SetAttributes(
    attribute.String("confident.trace.input", input),
    attribute.String("confident.trace.output", output),
)

span.setAttributes({
    "confident.trace.input": input,
    "confident.trace.output": output,
});

span.SetAttribute("confident.trace.input", input);
span.SetAttribute("confident.trace.output", output);

Metric Collection

Metric collection allows you to run metrics on cloud and publish results to the observatory.

"confident.trace.metric_collection" (of type str) update the name of the metric collection for the span

with tracer.start_as_current_span("custom_span") as span:
    span.set_attribute("confident.trace.metric_collection", "<your_metric_collection>")

span.setAttributes({
  "confident.trace.metric_collection": "<your_metric_collection>",
});

span.SetAttributes(
    attribute.String("confident.trace.metric_collection", "<your_metric_collection>"),
)

span.setAttributes({
    "confident.trace.metric_collection": "<your_metric_collection>",
});

span.SetAttribute("confident.trace.metric_collection", "<your_metric_collection>");

Test Case

LLM test case parameters can be used to unit test interactions within your LLM application. They can be set in the span as trace level attributes using the confident.trace.* namespace.

Given below is the example of running online evaluation for a span.

with tracer.start_as_current_span("confident_evaluation") as span:
    input = "What is the capital of France?"
    output = my_llm_app(input) # your LLM application

    span.set_attribute('confident.trace.metric_collection', "<your_metric_collection>")
    span.set_attribute('confident.trace.input', input)
    span.set_attribute('confident.trace.output', output)
    span.set_attribute('confident.trace.retrieval_context', ["context1", "context2"])
    span.set_attribute('confident.trace.expected_output', "Paris")

span.setAttributes({
  "confident.trace.metric_collection": "<your_metric_collection>",
  "confident.trace.llm_test_case.input": input,
  "confident.trace.llm_test_case.actual_output": output,
});

span.SetAttributes(
    attribute.String("confident.trace.metric_collection", "<your_metric_collection>"),
    attribute.String("confident.trace.input", input),
    attribute.String("confident.trace.output", output),
    attribute.String("confident.trace.retrieval_context", ["context1", "context2"]),
    attribute.String("confident.trace.expected_output", "Paris"),
)

span.setAttributes({
    "confident.trace.metric_collection": "<your_metric_collection>",
    "confident.trace.input": input,
    "confident.trace.output": output,
    "confident.trace.retrieval_context": ["context1", "context2"],
    "confident.trace.expected_output": "Paris",
});

span.SetAttribute("confident.trace.metric_collection", "<your_metric_collection>");
span.SetAttribute("confident.trace.input", input);
span.SetAttribute("confident.trace.output", output);
span.SetAttribute("confident.trace.retrieval_context", ["context1", "context2"]);
span.SetAttribute("confident.trace.expected_output", "Paris");

LLM test case attributes mapping:

"confident.trace.input" (of type str) used for updating test case input
"confident.trace.output" (of type str) used for updating test case actual output
[Optional] "confident.trace.expected_output" (of type str) used for updating test case expected output
[Optional] "confident.trace.context" (of type list[str]) used for updating test case context
[Optional] "confident.trace.retrieval_context" (of type list[str]) used for updating test case retrieval context
[Optional] "confident.trace.tools_called" (of type ToolCall) used for updating test case tools called
[Optional] "confident.trace.expected_tools" (of type ToolCall) used for updating test case expected tools

Metadata

Attach metadata to the trace. This information can be used for filtering, grouping, and analyzing your traces in the observatory.

"confident.trace.metadata" (of type str) used for updating trace metadata

This attribute is a JSON string which is parsed into a dictionary.

import json

with tracer.start_as_current_span("custom_span") as span:
    span.set_attribute("confident.trace.metadata", json.dumps({"key": "value"}))

span.setAttributes({
  "confident.trace.metadata": JSON.stringify({ key: "value" }),
});

attribute.String("confident.trace.metadata", `{"key": "value"}`)

span.setAttributes({
    "confident.trace.metadata": `{"key": "value"}`,
});

span.set_attribute('confident.trace.metadata', '{"key": "value"}');

Thread Id

A thread on Confident AI is a collection of one or more traces, letting you view full conversations — perfect for chat apps, agents, or any multi-turn interactions.

"confident.trace.thread_id" (of type str) used for updating trace thread id

with tracer.start_as_current_span("custom_span") as span:
    span.set_attribute("confident.trace.thread_id", "123")

span.setAttributes({
  "confident.trace.thread_id": "123",
});

span.SetAttributes(
    attribute.String("confident.trace.thread_id", "123"),
)

span.setAttributes({
    "confident.trace.thread_id": "123",
});

span.SetAttribute("confident.trace.thread_id", "123");

User Id

Track user interactions by setting user id in a trace — useful for monitoring token usage, identifying top users, and managing costs.

"confident.trace.user_id" (of type str) used for updating trace user id

with tracer.start_as_current_span("custom_span") as span:
    span.set_attribute("confident.trace.user_id", "123")

span.setAttributes({
  "confident.trace.user_id": "123",
});

span.SetAttributes(
    attribute.String("confident.trace.user_id", "123"),
)

span.setAttributes({
    "confident.trace.user_id": "123",
});

span.SetAttribute("confident.trace.user_id", "123");

Test Case Id

For single-turn evaluations via AI Connections, Confident AI sends a testCaseId in the payload to your endpoint. Pass it as the test_case_id attribute on your trace to link the trace back to its test case — letting you click through to the full trace directly from evaluation results.

"confident.trace.test_case_id" (of type str) used for linking the trace to a test case in evaluation results

with tracer.start_as_current_span("custom_span") as span:
    span.set_attribute("confident.trace.test_case_id", test_case_id)

span.setAttributes({
  "confident.trace.test_case_id": testCaseId,
});

span.SetAttributes(
    attribute.String("confident.trace.test_case_id", testCaseId),
)

span.setAttributes({
    "confident.trace.test_case_id": testCaseId,
});

span.SetAttribute("confident.trace.test_case_id", testCaseId);

Turn Id

For multi-turn evaluations via AI Connections, Confident AI sends a turnId in the payload for each turn. Pass it as the turn_id attribute on your trace to link each turn's trace to the specific turn in the conversation.

"confident.trace.turn_id" (of type str) used for linking the trace to a specific turn in multi-turn evaluation results

with tracer.start_as_current_span("custom_span") as span:
    span.set_attribute("confident.trace.turn_id", turn_id)

span.setAttributes({
  "confident.trace.turn_id": turnId,
});

span.SetAttributes(
    attribute.String("confident.trace.turn_id", turnId),
)

span.setAttributes({
    "confident.trace.turn_id": turnId,
});

span.SetAttribute("confident.trace.turn_id", turnId);

Span-Level Attribute Mappings

These are the attributes specific to Confident AI spans similar to tracing features. The span level attributes are set in the span attributes using the confident.span.* namespace.

Name

The name of the span is displayed in the UI. You can customize it based on your liking for better UI display using the following attribute:

"confident.span.name" (of type str) used for updating span name

with tracer.start_as_current_span("custom_span") as span:
    span.set_attribute("confident.span.name", "custom_span")

span.setAttributes({
  "confident.span.name": "custom_span",
});

span.SetAttributes(
    attribute.String("confident.span.name", "custom_span"),
)

span.setAttributes({
    "confident.span.name": "custom_span",
});

span.SetAttribute("confident.span.name", "custom_span");

Input/Output

You can set span input and output at runtime using the following attributes:

"confident.span.input" (of type Any) used for updating span input
"confident.span.output" (of type Any) used for updating span output

with tracer.start_as_current_span("custom_span") as span:
    span.set_attribute("confident.span.input", input)
    span.set_attribute("confident.span.output", output)

span.setAttributes({
  "confident.span.input": input,
  "confident.span.output": output,
});

span.SetAttributes(
    attribute.String("confident.span.input", input),
    attribute.String("confident.span.output", output),
)

span.setAttributes({
    "confident.span.input": input,
    "confident.span.output": output,
});

span.SetAttribute("confident.span.input", input);
span.SetAttribute("confident.span.output", output);

Metric Collection

Metric collection allows you to run metrics on cloud and publish results to the observatory.

"confident.span.metric_collection" (of type str) update the name of the metric collection for the span

with tracer.start_as_current_span("custom_span") as span:
    span.set_attribute("confident.span.metric_collection", "<your_metric_collection>")

span.setAttributes({
  "confident.span.metric_collection": "<your_metric_collection>",
});

span.SetAttributes(
    attribute.String("confident.span.metric_collection", "<your_metric_collection>"),
)

span.setAttributes({
    "confident.span.metric_collection": "<your_metric_collection>",
});

span.SetAttribute("confident.span.metric_collection", "<your_metric_collection>");

Test Case

LLM test case parameters can be used to unit test interactions within your LLM application. They can be set on the span level in any span type using the confident.span.* namespace.

Given below is the example of running online evaluation for a span.

with tracer.start_as_current_span("confident_evaluation") as span:
    input = "What is the capital of France?"
    output = my_llm_app(input) # your LLM application

    span.set_attribute('confident.span.metric_collection', "<your_metric_collection>")
    span.set_attribute('confident.span.input', input)
    span.set_attribute('confident.span.output', output)
    span.set_attribute('confident.span.retrieval_context', ["context1", "context2"])
    span.set_attribute('confident.span.expected_output', "Paris")

span.setAttributes({
  "confident.span.metric_collection": "<your_metric_collection>",
  "confident.span.llm_test_case.input": input,
  "confident.span.llm_test_case.actual_output": output,
});

span.SetAttributes(
    attribute.String("confident.span.metric_collection", "<your_metric_collection>"),
    attribute.String("confident.span.input", input),
    attribute.String("confident.span.output", output),
    attribute.String("confident.span.retrieval_context", "context1"),
    attribute.String("confident.span.expected_output", "Paris"),
)

span.setAttributes({
    "confident.span.metric_collection": "<your_metric_collection>",
    "confident.span.input": input,
    "confident.span.output": output,
    "confident.span.retrieval_context": ["context1", "context2"],
    "confident.span.expected_output": "Paris",
});

span.SetAttribute("confident.span.metric_collection", "<your_metric_collection>");
span.SetAttribute("confident.span.input", input);
span.SetAttribute("confident.span.output", output);
span.SetAttribute("confident.span.retrieval_context", "context1");
span.SetAttribute("confident.span.expected_output", "Paris");

LLM test case attributes mapping:

"confident.span.input" (of type str) used for updating test case input
"confident.span.output" (of type str) used for updating test case actual output
[Optional] "confident.span.expected_output" (of type str) used for updating test case expected output
[Optional] "confident.span.context" (of type list[str]) used for updating test case context
[Optional] "confident.span.retrieval_context" (of type list[str]) used for updating test case retrieval context
[Optional] "confident.span.tools_called" (of type ToolCall) used for updating test case tools called
[Optional] "confident.span.expected_tools" (of type ToolCall) used for updating test case expected tools

Metadata

Metadata can be attached to the span. This information can be used for filtering, grouping, and analyzing your spans in the observatory.

"confident.span.metadata" (of type str) used for updating span metadata

This attribute is a JSON string which is parsed into a dictionary.

import json
with tracer.start_as_current_span("custom_span") as span:
    span.set_attribute("confident.span.metadata", json.dumps({"key": "value"}))

span.setAttributes({
  "confident.span.metadata": JSON.stringify({ key: "value" }),
});

span.SetAttributes(
    attribute.String("confident.span.metadata", `{"key": "value"}`),
)

span.setAttributes({
    "confident.span.metadata": `{"key": "value"}`,
});

span.SetAttribute("confident.span.metadata", "{\"key\": \"value\"}");

Type specific attributes

Span types are optional but allow you to classify the most common types of components in LLM applications, which includes these 4 default span types:

llm
agent
retriever
tool

You can set the span type using the following attribute:

"confident.span.type" (of type str) used for updating span type

with tracer.start_as_current_span("custom_span") as span:
    span.set_attribute("confident.span.type", "llm")

span.setAttributes({
  "confident.span.type": "llm",
});

span.SetAttributes(
    attribute.String("confident.span.type", "llm"),
)

span.setAttributes({
    "confident.span.type": "llm",
});

span.SetAttribute("confident.span.type", "llm");

Span-Level Attributes for Specific Span Types

Given below are attributes for specific span types. It is recommended to set these attributes in the span attributes using the confident.{span_type}.* namespace.

Custom

This is the default span type. All the attributes that we used above with confident.span.* namespace are applicable to this span type.

LLM

To create a LLM span, set the confident.span.type to llm. After that refer to the table below for more LLM span attributes.

"confident.llm.model" (of type str) used for updating LLM model
[Optional] "confident.llm.cost_per_input_token" (of type float) used for updating cost per input token
[Optional] "confident.llm.cost_per_output_token" (of type float) used for updating cost per output token
[Optional] "confident.llm.input_token_count" (of type int) used for updating LLM Span input token count
[Optional] "confident.llm.output_token_count" (of type int) used for updating LLM Span output token count

You can also attribute Confident AI prompts to your LLM spans using prompt specific attributes:

[Optional] "confident.llm.prompt_alias" (of type string) used for identifying your prompt on Confident AI
[Optional] "confident.llm.prompt_commit_hash" (of type string) used for tracking which commit of prompt was used
[Optional] "confident.llm.prompt_label" (of type string) used for identifying your prompt by label
[Optional] "confident.llm.prompt_version" (of type string) used for identifying your prompt by version

Given below is the sample code for setting attributes for LLM span type.

with tracer.start_as_current_span("llm_span") as span:
    span.set_attribute("confident.span.type", "llm")
    span.set_attribute("confident.llm.model", "gpt-3.5-turbo")
    span.set_attribute("confident.span.prompt_alias", prompt.alias) # Must be pulled from Confident AI
    span.set_attribute("confident.span.prompt_commit_hash", prompt.hash)
    span.set_attribute("confident.span.prompt_label", prompt.label)
    span.set_attribute("confident.span.prompt_version", prompt.version)
    span.set_attribute("confident.span.input", [
        json.dumps({"role": "system", "content": "You are a helpful assistant."}),
        json.dumps({"role": "user", "content": input})
    ])
    time.sleep(0.5)
    span.set_attribute("confident.span.output", "Hello world")

span.setAttributes({
  "confident.span.type": "llm",
  "confident.span.prompt_alias": prompt._alias, // Must match the alias from Confident AI
  "confident.span.prompt_commit_hash": prompt.hash,
  "confident.span.prompt_label": prompt.label,
  "confident.span.prompt_version": prompt.version,
  "confident.llm.model": "gpt-3.5-turbo",
  "confident.span.input": [
    JSON.stringify({ role: "system", content: "You are a helpful assistant." }),
    JSON.stringify({ role: "user", content: "What is the capital of France?" }),
  ],
  "confident.span.output": "Hello world",
});

span.SetAttributes(
    attribute.String("confident.span.type", "llm"),
    attribute.String("confident.llm.model", "gpt-3.5-turbo"),
    attribute.String("confident.span.prompt_alias", "PROMPT-ALIAS"), // Must match alias on Confident AI
    attribute.String("confident.span.prompt_commit_hash", "HASH"),
    attribute.String("confident.span.prompt_label", "LABE;"),
    attribute.String("confident.span.prompt_version", "VERSION"),
    attribute.StringSlice("confident.span.input", []string{
        `{"role": "system", "content": "You are a helpful assistant."}`,
        `{"role": "user", "content": "What is the capital of France?"}`,
    }),
    attribute.String("confident.span.output", "Hello world"),
)

span.setAttributes({
    "confident.span.type": "llm",
    "confident.llm.model": "gpt-3.5-turbo",
    "confident.span.prompt_alias": "PROMPT-ALIAS", # Must match alias on Confident AI
    "confident.span.prompt_commit_hash": "HASH",
    "confident.span.prompt_label": "LABEL",
    "confident.span.prompt_version": "VERSION",
    "confident.span.input": [
        `{"role": "system", "content": "You are a helpful assistant."}`,
        `{"role": "user", "content": "What is the capital of France?"}`
    ],
    "confident.span.output": "Hello world",
});

span.SetAttribute("confident.span.type", "llm");
span.SetAttribute("confident.llm.model", "gpt-3.5-turbo");
span.SetAttribute("confident.span.prompt_alias", "PROMPT-ALIAS");
span.SetAttribute("confident.span.prompt_commit_hash", "HASH");
span.SetAttribute("confident.span.prompt_label", "LABEL");
span.SetAttribute("confident.span.prompt_version", "VERSION");
span.SetAttribute("confident.span.input", [
    `{"role": "system", "content": "You are a helpful assistant."}`,
    `{"role": "user", "content": "What is the capital of France?"}`
]);
span.SetAttribute("confident.span.output", "Hello world");

Agent

To create a Agent span, set the confident.span.type to agent. After that refer to the table below for more Agent span attributes.

"confident.agent.name" (of type str) used for updating Agent span name
[Optional] "confident.agent.available_tools" (of type list[str]) used for updating Agent span available tools
[Optional] "confident.agent.agent_handoffs" (of type list[str]) used for updating Agent span agent handoffs

Given below is the sample code for setting attributes for Agent span type.

with tracer.start_as_current_span("agent_span") as span:
    span.set_attribute("confident.span.type", "agent")
    span.set_attribute("confident.agent.name", "agent_span")
    span.set_attribute("confident.agent.available_tools", ["llm_agent", "retriever_span", "tool_span"])
    span.set_attribute("confident.agent.agent_handoffs", ["llm_agent", "retriever_span", "tool_span"])

    span.set_attribute("confident.span.input", json.dumps({"input": "input"}))
    span.set_attribute("confident.span.output", json.dumps({"output": "output"}))

span.setAttributes({
  "confident.span.type": "agent",
  "confident.agent.name": "agent_span",
  "confident.agent.available_tools": [
    "llm_agent",
    "retriever_span",
    "tool_span",
  ],
  "confident.agent.agent_handoffs": [
    "llm_agent",
    "retriever_span",
    "tool_span",
  ],
  "confident.span.input": `{"input": input}`,
  "confident.span.output": `{"output": input}`,
});

span.SetAttributes(
    attribute.String("confident.agent.name", "agent_span"),
    attribute.String("confident.agent.available_tools", []string{"llm_agent", "retriever_span", "tool_span"}),
    attribute.String("confident.agent.agent_handoffs", []string{"llm_agent", "retriever_span", "tool_span"}),
    attribute.String("confident.span.input", `{"input": "input"}`),
    attribute.String("confident.span.output", `{"output": "output"}`),
    attribute.String("confident.span.type", "agent"),
)

span.setAttributes({
    "confident.span.type": "agent",
    "confident.agent.name": "agent_span",
    "confident.agent.available_tools": ["llm_agent", "retriever_span", "tool_span"],
    "confident.agent.agent_handoffs": ["llm_agent", "retriever_span", "tool_span"],
    "confident.span.input": `{"input": "input"}`,
    "confident.span.output": `{"output": "output"}`,
});

span.SetAttribute("confident.span.type", "agent");
span.SetAttribute("confident.agent.name", "agent_span");
span.SetAttribute("confident.agent.available_tools", ["llm_agent", "retriever_span", "tool_span"]);
span.SetAttribute("confident.agent.agent_handoffs", ["llm_agent", "retriever_span", "tool_span"]);
span.SetAttribute("confident.span.input", `{"input": "input"}`);
span.SetAttribute("confident.span.output", `{"output": "output"}`);

Tool

To create a Tool span, set the confident.span.type to tool. After that refer to the table below for more Tool span attributes.

"confident.tool.name" (of type str) used for updating Tool span name
[Optional] "confident.tool.description" (of type str) used for updating Tool span description

Given below is the sample code for setting attributes for Tool span type.

with tracer.start_as_current_span("tool_span") as span:
    span.set_attribute("confident.span.type", "tool")
    span.set_attribute("confident.tool.name", "tool name")

    span.set_attribute("confident.tool.description", "tool description")
    span.set_attribute("confident.span.input", json.dumps({"input": "input"}))
    span.set_attribute("confident.span.output", json.dumps({"output": "output"}))

span.setAttributes({
  "confident.span.type": "tool",
  "confident.tool.name": "tool name",
  "confident.tool.description": "tool description",
  "confident.span.input": `{"input": "input"}`,
  "confident.span.output": `{"output": "output"}`,
});

span.SetAttributes(
    attribute.String("confident.tool.name", "tool name"),
    attribute.String("confident.tool.description", "tool description"),
    attribute.String("confident.span.input", `{"input": "input"}`),
    attribute.String("confident.span.output", `{"output": "output"}`),
    attribute.String("confident.span.type", "tool"),
)

span.setAttributes({
    "confident.span.type": "tool",
    "confident.tool.name": "tool name",
    "confident.tool.description": "tool description",
    "confident.span.input": `{"input": "input"}`,
    "confident.span.output": `{"output": "output"}`,
});

span.SetAttribute("confident.span.type", "tool");
span.SetAttribute("confident.tool.name", "tool name");
span.SetAttribute("confident.tool.description", "tool description");
span.SetAttribute("confident.span.input", `{"input": "input"}`);
span.SetAttribute("confident.span.output", `{"output": "output"}`);

Retriever

To create a Retriever span, set the confident.span.type to retriever. After that refer to the table below for more Retriever span attributes.

"confident.retriever.embedder" (of type str) used for updating Retrieval span embedder model
[Optional] "confident.retriever.top_k" (of type int) used for updating Retrieval span top k
[Optional] "confident.retriever.chunk_size" (of type int) used for updating Retrieval span chunk size

Given below is the sample code for setting attributes for Retriever span type.

with tracer.start_as_current_span("retriever_span") as span:
    span.set_attribute("confident.span.type", "retriever")
    span.set_attribute("confident.retriever.embedder", "embedder")

    span.set_attribute("confident.span.input", input)
    span.set_attribute("confident.retriever.retrieval_context", ["asd", "asd"])
    span.set_attribute("confident.retriever.top_k", 10)
    span.set_attribute("confident.retriever.chunk_size", 10)

span.setAttributes({
  "confident.span.type": "retriever",
  "confident.retriever.embedder": "embedder",
  "confident.span.input": input,
  "confident.retriever.retrieval_context": ["asd", "asd"],
  "confident.retriever.top_k": 10,
  "confident.retriever.chunk_size": 10,
});

span.SetAttributes(
    attribute.String("confident.retriever.embedder", "embedder"),
    attribute.String("confident.span.input", input),
    attribute.StringSlice("confident.retriever.retrieval_context", []string{"asd", "asd"}),
    attribute.Int("confident.retriever.top_k", 10),
    attribute.Int("confident.retriever.chunk_size", 10),
    attribute.String("confident.span.type", "retriever"),
)

span.setAttributes({
    "confident.span.type": "retriever",
    "confident.retriever.embedder": "embedder",
    "confident.span.input": input,
    "confident.retriever.retrieval_context": new string[] { "asd", "asd" },
    "confident.retriever.top_k": 10,
    "confident.retriever.chunk_size": 10,
});

span.SetAttribute("confident.retriever.embedder", "embedder");
span.SetAttribute("confident.span.input", input);
span.SetAttribute("confident.retriever.retrieval_context", new string[] { "asd", "asd" });
span.SetAttribute("confident.retriever.top_k", 10);
span.SetAttribute("confident.retriever.chunk_size", 10);

Need help wiring this into your stack?Bring traces and evals into the tools your team already usesTalk to an expert

Overview

Quickstart

Set Environment Variables

Trace your first LLM application

Understanding OTEL with Confident AI

OTEL endpoints

Attributes

Advanced configurations

Trace-Level Attribute Mappings

Name

Input/Output

Metric Collection

Test Case

Tags

Metadata

Thread Id

User Id

Test Case Id

Turn Id

Span-Level Attribute Mappings

Name

Input/Output

Metric Collection

Test Case

Metadata

Type specific attributes

Span-Level Attributes for Specific Span Types

Custom

LLM

Agent

Tool

Retriever