Launch Week 02 wrapped — explore all five launches
Confident AI
Docs
Pricing
Products
LLM Evaluation
LLM Evaluation

Benchmark LLM systems with research-backed metrics.

LLM Observability
LLM Observability

Trace, monitor, and alert on production LLM systems.

AI Red Teaming
AI Red Teaming

Stress-test LLM apps against adversarial attacks.

AI Governance
AI Governance

Enforce AI standards and controls across teams.

Book a DemoSign Up
Book a DemoSign Up

August 21, 2026

  • Evals
  • Datasets
  • Security
  • Integrations

Test Runs: Brand New Page

TGIF! Thank god it's features, here's what we shipped this week:

Test runs finally got a page of their own. The new overview surfaces failing topics, names the issues they have in common, and aggregates metric scores so you can actually diagnose a bad run instead of scrolling until your eyes glaze over—MCP included, for the deep dives. While we were turning that page: personas hit beta so you can reuse a user across multi-turn goldens, audit logs export their entire history (APIs included), AI connections can carry MCP server context (yes, including the OpenAI Responses API), and the model catalog grew up—latest OpenAI models on Bedrock via Mantle, Claude on Vertex in every region. Same tests. Brand new page.

Changelog August 21, 2026

Added

  • Test Runs Overview Page - Test runs grew a dedicated overview that surfaces failing topics, identifies the issues they have in common, and aggregates metric scores so a bad run comes with a diagnosis instead of a homework assignment. MCP is wired in for the deep analysis, which means your coding agent can poke at the same numbers you can. Turn the page, skip the autopsy.
  • Public Model Configuration Routes - Evaluation, platform, and simulation models now have public routes. Configure the judge, the platform brain, and the simulator programmatically—same settings, now with an API that doesn't require a scavenger hunt through the sidebar. Models, now in public.
  • Personas (Beta) - Personas are in beta. Define a user once—tone, temperament, the whole character sheet—and reuse it across multi-turn goldens instead of rewriting "frustrated customer who types in lowercase" on every row. Same person, every conversation. It's not you, it's your persona.
  • Full Audit Log Export - The entire history of audit logs can now be exported in one go, and public APIs will do it for you if clicking "Save as CSV" is beneath your compliance pipeline. Every action, one file, no paging through 2019 by hand. Log off, file in hand.
  • MCP Context on AI Connections - AI connections can now carry MCP server context, and the agent endpoint can call the MCP server directly—which means you can evaluate setups like the OpenAI Responses API without standing up a wrapper just to fetch tools. The connection makes the call. Context, fully connected.
  • Hyperparameter Keys in AI Connection Payloads - Hyperparameter keys are now first-class in the AI connection payload. Reference a hyperparameter.key token in your JSON, and the saved values fill in at run time instead of living as a blob you hope someone typed correctly. Keys, not vibes. Hyper specific.

Changed

  • Latest Models on Bedrock & Vertex - Bedrock now serves the latest OpenAI models through Mantle—API keys, the whole thing—and Vertex AI finally runs Claude in every region. "We don't have that model in that region" is officially retired as an excuse. The catalog caught up with the press releases. Model citizens, globally.

That's the drop for this week—see you next Friday.

Built byConfident AI