AgentCore
Use Confident AI for LLM observability and evals for Amazon AgentCore
Overview
Amazon Bedrock AgentCore is AWS's managed runtime for deploying and scaling AI agents. Confident AI allows you to trace and evaluate agents running on AgentCore in just a few lines of code — each /invocations request shows up in the Observatory as one trace, with the model, tool, and agent spans from whatever framework you run inside it nested underneath, so you can see every turn your deployed agent handles and how it performed.
| Runtime | Requirements | Setup |
|---|---|---|
| Python | Python 3.10+, bedrock-agentcore (tested with 1.22.0), OTel ASGI instrumentation (tested with 0.63b1) | Call init() explicitly before creating the app |
| TypeScript | Not supported | — |
Auto-Instrument
Install Dependencies
Run the following command to install
confident-tracewith itsagentcoreextra, along with the AgentCore runtime and your model SDK:pip install 'confident-trace[agentcore]' bedrock-agentcore boto3The
agentcoreextra pulls in OpenTelemetry's ASGI middleware, which is what letsconfident-tracewrap the AgentCore HTTP server and open a span for each incoming/invocationsrequest. If you are using AgentCore with Strands, also install:pip install 'strands-agents[otel]'Setup Confident AI Key
Get your Confident AI Project API key and set it as an environment variable. Boto3 uses your normal AWS credential chain.
export CONFIDENT_API_KEY="<your-confident-api-key>" export BEDROCK_MODEL="<bedrock-model-id>" # plus AWS_REGION and AWS credentials for BedrockInstrument AgentCore
Call
init()once at startup, before constructingBedrockAgentCoreApp. Every request that hits your entrypoint will then produce a trace with the Bedrockconversespan underneath the server span.main.py import asyncio import os import boto3 from bedrock_agentcore.runtime import BedrockAgentCoreApp from confident_trace import init, shutdown init( endpoint="https://otel.confident-ai.com/v1/traces", protocol="http/protobuf", api_key=os.environ["CONFIDENT_API_KEY"], ) app = BedrockAgentCoreApp() client = boto3.client("bedrock-runtime") @app.entrypoint async def invoke(payload, context): # Offloading to a thread keeps the trace context intact for the sync Boto3 call. return await asyncio.to_thread( client.converse, modelId=os.environ["BEDROCK_MODEL"], messages=[{"role": "user", "content": [{"text": payload["prompt"]}]}], ) if __name__ == "__main__": try: app.run() finally: shutdown()Run your agent
Start the app locally and send it a request:
python main.py curl -X POST http://localhost:8080/invocations \ -H "Content-Type: application/json" \ -d '{"prompt": "Explain OpenTelemetry in one sentence."}'Done ✅. Open the Observatory in your Confident AI project to inspect the trace. The same file runs unchanged when deployed to AgentCore.
What Gets Captured
Each AgentCore invocation is exported as a trace that connects the incoming request to the model, agent, and tool operations performed while handling it.
- Invocation lifecycle — request timing, status, streaming duration, and the operations nested inside the entry point.
- Conversation identity — the AgentCore session is used as the trace's thread ID, grouping invocations from the same session.
- Distributed context — incoming W3C trace context is preserved so AgentCore work can remain connected to an upstream trace.
- Model calls — model details, messages, tool requests and results, finish reasons, and token usage when provided by the enabled model integration.
- Agent and tool calls — operations emitted by enabled framework integrations remain nested under the invocation.
- Custom spans — custom application spans created while handling the request retain their place in the trace.
The AgentCore request boundary itself doesn't inspect or record request and response bodies. Content captured by nested integrations follows their respective content policies.
Set Trace Span Properties
Use a trace context to add properties you know before the call starts. It creates no extra span; the trace started by the @app.entrypoint handler inherits the tags, metadata, user ID, and customer ID.
from confident_trace import init, trace_context
init()
@app.entrypoint
async def invoke(payload, context):
async with trace_context(
tags=["support"],
metadata={"release": "2026-09"},
user_id=payload.get("user_id"),
customer_id=payload.get("customer_id"),
):
return await asyncio.to_thread(
client.converse,
modelId=os.environ["BEDROCK_MODEL"],
messages=[{"role": "user", "content": [{"text": payload["prompt"]}]}],
)This request-driven example uses user_id and customer_id because the payload supplies only IDs.
See users and customers to set an optional display name alongside each ID, and trace context for every supported trace property and update behavior.
Instrumenting Multi-Turn
You do not need turn() when one AgentCore entry-point call is already one conversational turn—the integration creates that turn's trace automatically. Use turn() when you want to define the boundary yourself, such as grouping two sequential AgentCore calls into one turn. Reuse the same thread ID on later turns to group them into one conversation.
from confident_trace import init, turn
init()
@app.entrypoint
async def invoke(payload, context):
async with turn("support-turn", thread_id=context.session_id):
first = await call_agent(payload["prompt"])
return await call_agent(f"Summarize this result: {first}")See threads for thread I/O, turn IDs, and user IDs.
Disable AgentCore Instrumentation
Pass init() a list of integration identifiers to opt in to only those integrations. The identifier for AgentCore is "agentcore" in Python; omit it to disable this integration. An empty list disables all automatic instrumentation:
from confident_trace import init
init(instrumentations=())
# Use ("agentcore",) to opt in; omit "agentcore" to disable it.This turns off Confident AI's automatic instrumentation; calls made after initialization are not instrumented by this integration.
Next Steps
Now that your deployed agent is traced, dive deeper into:
Online Evals
Run evaluations on traces and spans in real-time as they're ingested into Confident AI to monitor your agent's quality.
Threads
Every AgentCore session is already a thread — view and evaluate the whole conversation as one unit.
Last updated on