Introducing Customer Monitoring: Product Analytics for B2B AI Agents

B2B agents are sold to customers, not users

Confident AI has always tracked users. For a B2B agent, that is only half the picture.

Your contracts, renewals, and escalations happen at the account level. When a customer's champion says the agent "isn't working for us," they are describing the experience of everyone on their team, not a single user.

Customer Monitoring adds that layer. Traces are grouped into the customer they belong to, and each customer's users are grouped under the account. You can move from how is Acme doing? to which Acme users are struggling? to which trace went wrong? without leaving the page.

Every account, in aggregate and one by one

Each customer gets a page summarizing how their users are doing:

• Cost: what the account costs you over the selected window
• Activity: when the account was last seen and how consistently it uses the agent
• Sentiment: how the account's users feel about the agent's responses
• Issues: the most common problems the account runs into
• Use cases: what the account actually uses the agent for

Below the summary, a breakdown of every user in the account shows their sentiment, top issue, trace and thread counts, cost, and last activity. Switch to the traces or threads tab to inspect the underlying interactions directly.

Grounded in traces and evals

Product analytics tools can tell you that an account is less active. They cannot tell you why the answers it received were wrong.

On Confident AI, the account view is built from the same data your evaluations run on. Sentiment, issues, and use cases come from classifiers on the account's traces and threads. Quality comes from the evals already scoring your production traffic. When a customer's sentiment drops, you can open the exact conversations behind it and see whether the cause was a retrieval failure, a broken tool call, or a gap in what the agent was built to do.

That gives each team what it needs from the same place:

• PMs see which accounts are healthy, which are at risk, and what each one uses the agent for
• Customer success teams walk into a call knowing exactly what went wrong for that account
• Engineers get the traces needed to fix the problem

One line to set up

Customers come from your application. If you already instrument with confident-trace, attach the customer to the trace context alongside the user:

from confident_trace import trace_context

with trace_context(
    customer={"id": "customer-42", "name": "Acme Corp"},
    user={"id": "user-7", "name": "Marta"},
):
    agent.invoke(query)

Every trace inside that context is attributed to the customer, and the account page fills in as traffic arrives. See the customers documentation for TypeScript and other options.

Get started

Customer Monitoring is live on Confident AI now.

Add a customer ID to your traces and see how every account experiences your agent. Book a demo with the Confident AI team to see Customer Monitoring in action.