OpenTelemetry

OpenTelemetry is an open-source observability framework that allows teams to collect, analyze, and visualize telemetry data.

Overview

Confident AI can recieve traces on https://otel.confident-ai.com. To export traces using the OpenTelemetry SDK, you can configure your Collector with the official open-telemetry library.

Quickstart

Given below is the quickstart for exporting traces to Confident AI OTLP endpoint (which will then published to observatory) for different languages.

$
export CONFIDENT_API_KEY="confident_us..."
>
export OTEL_EXPORTER_OTLP_ENDPOINT="https://otel.confident-ai.com"

$
pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp-proto-http

1
import os
2
3
from opentelemetry.sdk.trace import TracerProvider
4
from opentelemetry.sdk.trace.export import BatchSpanProcessor
5
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
6
7
OTLP_ENDPOINT = os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT")
8
CONFIDENT_API_KEY = os.getenv("CONFIDENT_API_KEY")
9
10
trace_provider = TracerProvider()
11
exporter = OTLPSpanExporter(
12
    endpoint=f"{OTLP_ENDPOINT}/v1/traces",
13
    headers={"x-confident-api-key": CONFIDENT_API_KEY},
14
)
15
span_processor = BatchSpanProcessor(span_exporter=exporter)
16
trace_provider.add_span_processor(span_processor)
17
tracer = trace_provider.get_tracer("deepeval_tracer")
18
19
# Start a span
20
with tracer.start_as_current_span("confident-llm-span") as span:
21
22
    # Set attributes
23
    span.set_attribute("confident.trace.name", "example-trace")
24
    span.set_attribute("confident.span.type", "llm")
25
    span.set_attribute("confident.llm.model", "gpt-4o")
26
    span.set_attribute("confident.span.input", "What is the capital of France?")
27
    span.set_attribute("confident.span.output", "Paris")
28
29
trace_provider.force_flush()
30
print("Traces posted successfully to https://otel.confident-ai.com")

$
python main.py

1
from opentelemetry import trace
2
from opentelemetry.sdk.trace import TracerProvider
3
from opentelemetry.sdk.trace.export import BatchSpanProcessor
4
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
5
6
7
OTLP_ENDPOINT = os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT")
8
CONFIDENT_API_KEY = os.getenv("CONFIDENT_API_KEY")
9
10
# Setup OpenTelemetry
11
if not isinstance(trace.get_tracer_provider(), TracerProvider):
12
    tracer_provider = TracerProvider()
13
    trace.set_tracer_provider(tracer_provider)
14
else:
15
    tracer_provider = trace.get_tracer_provider()
16
17
exporter = OTLPSpanExporter(
18
    endpoint=f"{OTLP_ENDPOINT}/v1/traces",
19
    headers={"x-confident-api-key": CONFIDENT_API_KEY},
20
)
21
22
span_processor = BatchSpanProcessor(span_exporter=exporter)
23
tracer_provider.add_span_processor(span_processor)
24
tracer = trace.get_tracer("deepeval_tracer")

🎉 Congratulations! You have successfully sent traces. Go to the the Observatory section on Confident AI to check it out.

1
import time
2
import json
3
from opentelemetry import trace
4
from opentelemetry.sdk.trace import TracerProvider
5
from opentelemetry.sdk.trace.export import BatchSpanProcessor
6
7
from deepeval.tracing.otel.exporter import ConfidentSpanExporter
8
9
# Set up tracer provider
10
tracer_provider = trace.get_tracer_provider()
11
if not isinstance(tracer_provider, TracerProvider):
12
    trace.set_tracer_provider(TracerProvider())
13
14
# Add confident span exporter wrapped in batch span processor
15
tracer_provider.add_span_processor(BatchSpanProcessor(ConfidentSpanExporter()))
16
17
# Get tracer
18
tracer = trace.get_tracer("deepeval_tracer")
19
20
# set attributes
21
with tracer.start_as_current_span("confident-llm-span") as span:
22
    span.set_attribute("confident.trace.name", "example-trace")
23
    span.set_attribute("confident.span.type", "llm")
24
    span.set_attribute("confident.llm.model", "gpt-4o")
25
    span.set_attribute("confident.span.input", "What is the capital of France?")
26
    span.set_attribute("confident.span.output", "Paris")

Understanding OTEL with Confident AI

In this section, we will mainly discuss:

• gen_ai attribute and event conventions
• Confident AI specific conventions
• Advanced configurations

OTEL endpoints

Confident AI offers the https://otel.confident-ai.com endpoint that accepts OpenTelemetry traces in the OTLP format. Please note that Confident AI does not support GRPC for the OpenTelemetry endpoint. Please use HTTP instead.

Attributes

Confident AI adheres to the GenAI semantic convention and adds an extra layer on top of it to capture additional data about the LLM applications. Confident AI uses confident.* namespace to map specific attributes with llm tracing data model. These specific attributes always take precedence over gen_ai.* conventions and are recommended for all users that are manually instrumenting their applications.

Advanced configurations

The enviornment allows you to configure where your trace belongs on Confident AI (defaulted to "development"). You can configure the enviornment and sampling rate of traces by setting the following two enviornment variables:

$
OTEL_RESOURCE_ATTRIBUTES="confident.trace.environment=production"

Trace-Level Attribute Mappings

These are the attributes specific to Confident AI traces similar to tracing features. The trace level attributes are set in the span attributes using the confident.trace.* namespace.

Name

The trace name is displayed in the UI. You can customize it based on your liking for better UI display using the following attribute:

• "confident.trace.name" (of type str) used for updating trace name

1
with tracer.start_as_current_span("custom_span") as span:
2
    span.set_attribute("confident.trace.name", "test_trace")

Input/Output

You can set trace input and output at runtime using the following attributes:

• "confident.trace.input" (of type Any) used for updating trace input
• "confident.trace.output" (of type Any) used for updating trace output

1
with tracer.start_as_current_span("custom_span") as span:
2
    span.set_attribute("confident.trace.input", input)
3
    span.set_attribute("confident.trace.output", output)

Metric Collection

Metric collection allows you to run metrics on cloud and publish results to the observatory.

• "confident.trace.metric_collection" (of type str) update the name of the metric collection for the span

1
with tracer.start_as_current_span("custom_span") as span:
2
    span.set_attribute("confident.trace.metric_collection", "<your_metric_collection>")

Test Case

LLM test case parameters can be used to unit test interactions within your LLM application. They can be set in the span as trace level attributes using the confident.trace.* namespace.

Given below is the example of running online evaluation for a span.

1
with tracer.start_as_current_span("confident_evaluation") as span:
2
    input = "What is the capital of France?"
3
    output = my_llm_app(input) # your LLM application
4
5
    span.set_attribute('confident.trace.metric_collection', "<your_metric_collection>")
6
    span.set_attribute('confident.trace.input', input)
7
    span.set_attribute('confident.trace.output', output)
8
    span.set_attribute('confident.trace.retrieval_context', ["context1", "context2"])
9
    span.set_attribute('confident.trace.expected_output', "Paris")

LLM test case attributes mapping:

• "confident.trace.input" (of type str) used for updating test case input
• "confident.trace.output" (of type str) used for updating test case actual output
• [Optional] "confident.trace.expected_output" (of type str) used for updating test case expected output
• [Optional] "confident.trace.context" (of type list[str]) used for updating test case context
• [Optional] "confident.trace.retrieval_context" (of type list[str]) used for updating test case retrieval context
• [Optional] "confident.trace.tools_called" (of type ToolCall) used for updating test case tools called
• [Optional] "confident.trace.expected_tools" (of type ToolCall) used for updating test case expected tools

Tags

Tags are simple string labels that make it easy to group related traces together, and cannot be applied to spans.

• "confident.trace.tags" (of type list[str]) used for updating trace tags

1
with tracer.start_as_current_span("custom_span") as span:
2
    span.set_attribute("confident.trace.tags", ["tag1", "tag2"])

Metadata

Attach metadata to the trace. This information can be used for filtering, grouping, and analyzing your traces in the observatory.

• "confident.trace.metadata" (of type str) used for updating trace metadata

This attribute is a JSON string which is parsed into a dictionary.

1
import json
2
3
with tracer.start_as_current_span("custom_span") as span:
4
    span.set_attribute("confident.trace.metadata", json.dumps({"key": "value"}))

Thread Id

A thread on Confident AI is a collection of one or more traces, letting you view full conversations — perfect for chat apps, agents, or any multi-turn interactions.

• "confident.trace.thread_id" (of type str) used for updating trace thread id

1
with tracer.start_as_current_span("custom_span") as span:
2
    span.set_attribute("confident.trace.thread_id", "123")

User Id

Track user interactions by setting user id in a trace — useful for monitoring token usage, identifying top users, and managing costs.

• "confident.trace.user_id" (of type str) used for updating trace user id

1
with tracer.start_as_current_span("custom_span") as span:
2
    span.set_attribute("confident.trace.user_id", "123")

Span-Level Attribute Mappings

These are the attributes specific to Confident AI spans similar to tracing features. The span level attributes are set in the span attributes using the confident.span.* namespace.

Name

The name of the span is displayed in the UI. You can customize it based on your liking for better UI display using the following attribute:

• "confident.span.name" (of type str) used for updating span name

1
with tracer.start_as_current_span("custom_span") as span:
2
    span.set_attribute("confident.span.name", "custom_span")

Input/Output

You can set span input and output at runtime using the following attributes:

• "confident.span.input" (of type Any) used for updating span input
• "confident.span.output" (of type Any) used for updating span output

1
with tracer.start_as_current_span("custom_span") as span:
2
    span.set_attribute("confident.span.input", input)
3
    span.set_attribute("confident.span.output", output)

Metric Collection

Metric collection allows you to run metrics on cloud and publish results to the observatory.

• "confident.span.metric_collection" (of type str) update the name of the metric collection for the span

1
with tracer.start_as_current_span("custom_span") as span:
2
    span.set_attribute("confident.span.metric_collection", "<your_metric_collection>")

Test Case

LLM test case parameters can be used to unit test interactions within your LLM application. They can be set on the span level in any span type using the confident.span.* namespace.

Given below is the example of running online evaluation for a span.

1
with tracer.start_as_current_span("confident_evaluation") as span:
2
    input = "What is the capital of France?"
3
    output = my_llm_app(input) # your LLM application
4
5
    span.set_attribute('confident.span.metric_collection', "<your_metric_collection>")
6
    span.set_attribute('confident.span.input', input)
7
    span.set_attribute('confident.span.output', output)
8
    span.set_attribute('confident.span.retrieval_context', ["context1", "context2"])
9
    span.set_attribute('confident.span.expected_output', "Paris")

LLM test case attributes mapping:

• "confident.span.input" (of type str) used for updating test case input
• "confident.span.output" (of type str) used for updating test case actual output
• [Optional] "confident.span.expected_output" (of type str) used for updating test case expected output
• [Optional] "confident.span.context" (of type list[str]) used for updating test case context
• [Optional] "confident.span.retrieval_context" (of type list[str]) used for updating test case retrieval context
• [Optional] "confident.span.tools_called" (of type ToolCall) used for updating test case tools called
• [Optional] "confident.span.expected_tools" (of type ToolCall) used for updating test case expected tools

Metadata

Metadata can be attached to the span. This information can be used for filtering, grouping, and analyzing your spans in the observatory.

• "confident.span.metadata" (of type str) used for updating span metadata

This attribute is a JSON string which is parsed into a dictionary.

1
import json
2
with tracer.start_as_current_span("custom_span") as span:
3
    span.set_attribute("confident.span.metadata", json.dumps({"key": "value"}))

Type specific attributes

Span types are optional but allow you to classify the most common types of components in LLM applications, which includes these 4 default span types:

• llm
• agent
• retriever
• tool

You can set the span type using the following attribute:

• "confident.span.type" (of type str) used for updating span type

1
with tracer.start_as_current_span("custom_span") as span:
2
    span.set_attribute("confident.span.type", "llm")

Span-Level Attributes for Specific Span Types

Given below are attributes for specific span types. It is recommended to set these attributes in the span attributes using the confident.{span_type}.* namespace.

Custom

This is the default span type. All the attributes that we used above with confident.span.* namespace are applicable to this span type.

LLM

To create a LLM span, set the confident.span.type to llm. After that refer to the table below for more LLM span attributes.

• "confident.llm.model" (of type str) used for updating LLM model
• [Optional] "confident.llm.cost_per_input_token" (of type float) used for updating cost per input token
• [Optional] "confident.llm.cost_per_output_token" (of type float) used for updating cost per output token
• [Optional] "confident.llm.input_token_count" (of type int) used for updating LLM Span input token count
• [Optional] "confident.llm.output_token_count" (of type int) used for updating LLM Span output token count

Given below is the sample code for setting attributes for LLM span type.

1
with tracer.start_as_current_span("llm_span") as span:
2
    span.set_attribute("confident.span.type", "llm")
3
    span.set_attribute("confident.llm.model", "gpt-3.5-turbo")
4
    span.set_attribute("confident.span.input", [
5
        json.dumps({"role": "system", "content": "You are a helpful assistant."}),
6
        json.dumps({"role": "user", "content": input})
7
    ])
8
    time.sleep(0.5)
9
    span.set_attribute("confident.span.output", "Hello world")

Agent

To create a Agent span, set the confident.span.type to agent. After that refer to the table below for more Agent span attributes.

• "confident.agent.name" (of type str) used for updating Agent span name
• [Optional] "confident.agent.available_tools" (of type list[str]) used for updating Agent span available tools
• [Optional] "confident.agent.agent_handoffs" (of type list[str]) used for updating Agent span agent handoffs

Given below is the sample code for setting attributes for Agent span type.

1
with tracer.start_as_current_span("agent_span") as span:
2
    span.set_attribute("confident.span.type", "agent")
3
    span.set_attribute("confident.agent.name", "agent_span")
4
    span.set_attribute("confident.agent.available_tools", ["llm_agent", "retriever_span", "tool_span"])
5
    span.set_attribute("confident.agent.agent_handoffs", ["llm_agent", "retriever_span", "tool_span"])
6
7
    span.set_attribute("confident.span.input", json.dumps({"input": "input"}))
8
    span.set_attribute("confident.span.output", json.dumps({"output": "output"}))

Tool

To create a Tool span, set the confident.span.type to tool. After that refer to the table below for more Tool span attributes.

• "confident.tool.name" (of type str) used for updating Tool span name
• [Optional] "confident.tool.description" (of type str) used for updating Tool span description

Given below is the sample code for setting attributes for Tool span type.

1
with tracer.start_as_current_span("tool_span") as span:
2
    span.set_attribute("confident.span.type", "tool")
3
    span.set_attribute("confident.tool.name", "tool name")
4
5
    span.set_attribute("confident.tool.description", "tool description")
6
    span.set_attribute("confident.span.input", json.dumps({"input": "input"}))
7
    span.set_attribute("confident.span.output", json.dumps({"output": "output"}))

Retriever

To create a Retriever span, set the confident.span.type to retriever. After that refer to the table below for more Retriever span attributes.

• "confident.retriever.embedder" (of type str) used for updating Retrieval span embedder model
• [Optional] "confident.retriever.top_k" (of type int) used for updating Retrieval span top k
• [Optional] "confident.retriever.chunk_size" (of type int) used for updating Retrieval span chunk size

Given below is the sample code for setting attributes for Retriever span type.

1
with tracer.start_as_current_span("retriever_span") as span:
2
    span.set_attribute("confident.span.type", "retriever")
3
    span.set_attribute("confident.retriever.embedder", "embedder")
4
5
    span.set_attribute("confident.span.input", input)
6
    span.set_attribute("confident.retriever.retrieval_context", ["asd", "asd"])
7
    span.set_attribute("confident.retriever.top_k", 10)
8
    span.set_attribute("confident.retriever.chunk_size", 10)