Open Inference | Confident AI Docs

Overview

OpenInference is an open standard for capturing and storing AI model inferences. Confident AI allows you to trace and evaluate any application instrumented with OpenInference in just a few lines of code.

Tracing Quickstart

For users in the EU region, please set the OTEL endpoint to the EU version as shown below:

$ export CONFIDENT_OTEL_URL="https://eu.otel.confident-ai.com"

Install Dependencies

Run the following command to install the required packages:

1 pip install -U deepeval opentelemetry-sdk opentelemetry-exporter-otlp-proto-http

Instrument OpenInference

Call instrument_openinference once at startup, before your agent runs. It attaches to the active OpenTelemetry TracerProvider and begins forwarding spans to Confident AI.

You will also need the specific OpenInference instrumentor for your framework, such as openinference-instrumentation-langchain, openinference-instrumentation-openai or @arizeai/openinference-instrumentation-anthropic, etc.

Python

TypeScript

main.py

1 import os
2 from openinference.instrumentation.langchain import LangChainInstrumentor
3 from langchain_openai import ChatOpenAI
4 from deepeval.integrations.openinference import instrument_openinference
5 
6 LangChainInstrumentor().instrument()
7 
8 instrument_openinference()
9 
10 llm = ChatOpenAI(model="gpt-4o-mini")
11 result = llm.invoke("What are LLMs?")

Confident AI strictly adheres to OpenInference semantic conventions and will only capture the telemetry data your instrumentors explicitly expose. To get full traces of your entire application flow, please ensure you instrument your application in nested layers for full observability from OpenInference.

Run your code

Get your traces by running your code as shown here:

$ python main.py

You can directly view the traces on Confident AI’s observatory page

Advanced Usage

Logging prompts

If you are managing prompts on Confident AI and wish to log them, pass your Prompt object to instrument_openinference.

Python

TypeScript

main.py

1 import os
2 from deepeval.prompt import Prompt
3 from langchain_openai import ChatOpenAI
4 from openinference.instrumentation.langchain import LangChainInstrumentor
5 from deepeval.integrations.openinference import instrument_openinference
6 
7 prompt = Prompt(alias="my-prompt")
8 prompt.pull(version="00.00.01")
9 
10 system_prompt = prompt.interpolate()
11 
12 LangChainInstrumentor().instrument()
13 
14 instrument_openinference(
15     confident_prompt=prompt, 
16 )
17 
18 llm = ChatOpenAI(model="gpt-4o-mini")
19 result = llm.invoke(system_prompt)

Logging prompts lets you attribute specific prompts to OpenInference LLM spans. Be sure to pull the prompt before logging it, otherwise the prompt will not be visible on Confident AI.

Logging threads

Threads are used to group related traces together, and are useful for chat apps, agents, or any multi-turn interactions. You can learn more about threads here. Pass the thread_id to instrument_openinference.

Python

TypeScript

main.py

1 import os
2 from openinference.instrumentation.langchain import LangChainInstrumentor
3 from langchain_openai import ChatOpenAI
4 from deepeval.integrations.openinference import instrument_openinference
5 
6 LangChainInstrumentor().instrument()
7 
8 instrument_openinference(
9     thread_id="thread_1",
10     user_id="user_1"
11 )
12 
13 llm = ChatOpenAI(model="gpt-4o-mini")
14 result = llm.invoke("What are LLMs?")

Trace attributes

Other trace attributes can also be passed to instrument_openinference.

Python

TypeScript

main.py

1 import os
2 from openinference.instrumentation.langchain import LangChainInstrumentor
3 from langchain_openai import ChatOpenAI
4 from deepeval.integrations.openinference import instrument_openinference
5 
6 LangChainInstrumentor().instrument()
7 
8 instrument_openinference(
9     name="Name of Trace",
10     tags=["Tag 1", "Tag 2"],
11     metadata={"Key": "Value"},
12     user_id="user_1",
13     thread_id="conversation-abc123",
14     environment="production",
15 )
16 
17 llm = ChatOpenAI(model="gpt-4o-mini")
18 result = llm.invoke("What are LLMs?")

View Trace Attributes

name

str

The name of the trace. Learn more.

Evals Usage

Online evals

You can run online evals on your OpenInference instrumentation, which will run evaluations on all incoming traces on Confident AI’s servers.

Create metric collection

Create a metric collection on Confident AI with the metrics you wish to use to evaluate your AgentCore agent.

Create metric collection

Your metric collection must only contain metrics that evaluate the input and actual output of the component it is assigned to.

Run evals

You can run evals at both the trace and span level. We recommend creating separate metric collections for each component, since each requires its own evaluation criteria and metrics.

After instrumenting your AgentCore agent, pass the metric collection name to the respective component:

Trace

LLM Span

Tool Span

1 import os
2 from openinference.instrumentation.langchain import LangChainInstrumentor
3 from langchain_openai import ChatOpenAI
4 from deepeval.integrations.openinference import instrument_openinference
5 
6 LangChainInstrumentor().instrument()
7 
8 instrument_openinference(
9     name="openinference_application",
10     trace_metric_collection="TRACE-METRIC-COLLECTION"
11 )
12 
13 llm = ChatOpenAI(model="gpt-4o-mini")
14 result = llm.invoke("What are LLMs?")

All incoming traces will now be evaluated using metrics from your metric collection.

You can view evals on Confident AI by clicking on the link in the output printed in the console.