Generating Reports for Stakeholders

Overview

The goal of this guide is simple: get a curated report into your stakeholders’ hands, automatically, on a schedule — no one manually pulling numbers the night before a leadership sync.

On Confident AI you do that with an Executive Report: an AI-written summary of your project — quality, cost, latency, eval health — generated from a template you control and emailed out when it’s ready. The winning workflow is three steps:

Reports read straight from your project’s existing data — traces, test runs, metrics, and annotations — so the more you’ve logged, the richer they get.

Planning the Report

Before you touch the builder, decide what your stakeholders actually need to know. A great leadership report answers a handful of standing questions — not “here’s every metric we have.”

Every report is built from a small set of section types — the primitives you’ll assemble in the builder. Get familiar with them first:

Stat cards

Graph

Table

Content

Admonition

Stat cards are a row of headline metrics, where each card carries a label, a value, and an optional caption. They work best for the numbers leadership scans first, such as pass rate, volume, and cost.

With the building blocks in mind, plan around what Confident AI can actually measure. Reports draw on four kinds of project data:

• Observability captures how your agent behaves in production, across traces, spans, threads, and end users.
• Evaluation measures how your agent scores against your benchmarks, from datasets to test runs.
• Diagnostics surfaces what’s actually going wrong, in failing traces and failing test cases.
• Correlation ties quality signals to human judgment, through online metric scores and annotations.

Pick the areas that map to what leadership asks, then turn each question into a section. Here are concrete report ideas for each — and the section type that fits:

For example, a single cost report can weave several section types together. The one below came from one prompt: “Show me some insights on our cost usage on average lately. In particular, I’d like to know our total trace and model costs as well as which user(s) are spending the most money.” Confident AI answered it with three sections — a stat cards row (model cost, total trace cost, trace events, distinct end users, average cost per trace), a table of the most active day and highest-cost model, and a content overview that narrates the numbers.

Keep it tight — five or six sections that answer real questions beat twenty no one reads. Then cap it with a caveat admonition noting what the data doesn’t cover, so leadership doesn’t over-read it.

If a section can’t be answered from this data, drop it or rephrase it. With the plan in hand, you’re ready to build.

Building the Report

You’ll find your generated Executive Reports on the Reports page, opened from the project sidebar. But you don’t create reports there directly — every report is produced from a Report Template, a reusable, scheduled definition that lives in Project Settings → Report Templates.

Think of a template as the recipe and a report as the dish: each scheduled run cooks a fresh report from your project’s latest data. Every template gets its own tab on the Reports page, so a Weekly Leadership Update template builds a running history you can flip through.

Both paths live in the same editor — the Use custom template toggle switches between them.

To create a template, go to Project Settings → Report Templates → New Template, give it a Name (e.g. Weekly Leadership Update) and a starting Description, and you’ll land in the template editor. Now pick your path:

• Generate the entire report — you write a description and Confident AI decides the sections and writes the whole report. Fastest to set up, but the structure can vary run to run.
• Build a custom template — the report is still AI-generated, but instead of letting the model decide the layout, you define the exact sections and their order. Confident AI then fills each one from your live data every run. More upfront work, but you get the same consistent structure each time.

The rule of thumb: reach for a custom template whenever this is a recurring report going to stakeholders — consistency is what makes it trustworthy. Let Confident AI generate the entire report for quick, exploratory, or one-off reports.

Generate the Entire Report

Leave Use custom template off and simply fill in the Description — a plain-English prompt of what the report should analyze:

“What are the trends in error rates for all types of evaluations in the past month?”

That’s it. Confident AI plans which data to query, picks the sections that fit, and writes the whole report for you — an overview, key findings, stat cards, an optional graph and tables, recommendations, and a caveat. You don’t choose sections; you describe the question and let the model do the rest.

This is the fastest path and a great default. The better your description — naming data types, metrics, and time windows — the sharper the report.

Build a Custom Template

Turn Use custom template on when stakeholders need a specific, consistent structure every run (the usual case for a recurring leadership report). The section builder opens, and you assemble exactly the sections you planned earlier.

Here’s the key thing to understand: a custom template doesn’t turn off AI — it just fixes the structure. You decide which sections appear and in what order, and Confident AI still generates the content of each one from your live project data at run time (except for any sections you deliberately hardcode). Think of it as handing the model an outline instead of a blank page: you own the skeleton, the model writes the body.

1

Add and configure sections

Click Add to create a section, then set its Heading (“shown above the section”) and its Type. See the section types below.

2

Reorder and preview

Drag sections into the order stakeholders should read them. Open the Preview tab to see the rendered layout with placeholder data.

3

Save

When you have unsaved changes, a Save / Discard controller appears at the bottom — click Save to commit.

✅ Done. You now have a template that produces exactly the report you planned.

Report Section Types

You’ve already met the five section types. In the builder, the only new decision is how each one gets filled — by hand or by AI. Content and Admonition support both; Stat cards, Table, and Graph are AI-only, since their numbers are pulled live from your data at generation time. Here’s how to fill each:

• Content — type the exact prose into the Content box, or flip Generate with AI on and add a Prompt so the model drafts it from your data each run (e.g. “Summarize agent health in two short paragraphs and list three recommendations.”).
• Admonition — hand-author it by choosing a Severity (Info, Success, Warning, or Danger) and writing the Content, or flip Generate with AI on and let a Prompt drive it — the model writes a 1–3 sentence callout and sets the severity to match (Success for wins, Warning/Danger for risks).
• Stat cards — always AI-generated, so your Prompt names the figures to surface: “Pass rate, total test runs, and average cost per run for the last 7 days, each vs. the prior week.”
• Table — always AI-generated, so your Prompt describes the columns and rows: “The five metrics with the highest failure rates, with pass rate and test-run count.”
• Graph — always AI-generated, so your Prompt says what to plot: “Pass rate by day over the last 7 days.” The model then picks the chart style and either runs a live query or plots the exact numbers pulled from your data.

Delivering the Report

A curated report is only useful if it actually reaches people. Automation has two parts: when it generates and who it emails.

Schedule generation

Every template runs on a daily schedule. On the Report Templates list:

• Enable / disable — the switch on each row controls whether the template generates on its schedule.
• Generate now — the ⋮ menu runs it on demand. Use this to test your template without waiting for the next day.

Turn on report emails

Now wire up delivery so a fresh report lands in the right inboxes the moment it’s generated:

When a report finishes generating, each selected recipient gets an email titled “Your executive report is ready” with the date range and a link straight to the report on the platform.

Best Practices

Once your reports are generating on their own, a few details help you get the most out of them — exporting by hand, choosing the model that writes them, sharpening weak sections, and the separate report type built for security stakeholders.

Exporting Reports

Beyond email, you can always read reports in-app. Open Reports from the project sidebar — one tab per template, newest first, with Report N of M arrows to step through history.

Each report’s toolbar has two actions for manual sharing:

• Download as PDF exports the document exactly as rendered, charts and tables included. This is your handoff for stakeholders who aren’t on the platform.
• Expand opens a full-screen, print-quality view so you can proof the report before you export it.

Selecting the Generation Model

Reports are written by your project’s platform model — the model that powers Confident AI’s own AI features, like classification, summaries, and report generation. This is a separate setting from the evaluation model that scores your LLM-as-a-judge metrics, so changing one never touches the other.

Set or change it in Project Settings → Platform Model. If you haven’t set a project-specific one, reports use your organization’s default platform model.

Optimizing Report Content

The sharpness of a report comes down to how precisely each prompt names the data type, metric, and time window to pull. Vague prompts get vague sections, so lead with the specifics you want the model to surface.

When a prompt can’t be mapped onto data the project actually has, the section comes back as an “irrelevant query” instead of inventing numbers. That usually means the plan drifted from reality:

• The section asks about a feature the project has no data for (e.g. red-team scores with no red-teaming).
• The time window has no traffic.
• The prompt is too abstract to turn into a query.

Go back to your plan — tighten the section to name the data type, metric, and window — and the next run produces a real report.

Next Steps

You can now plan a report, build it with a custom template, and automate delivery to stakeholders. To go deeper on the data behind them: