Introducing Confident Monitor
Confident AI
Docs
Pricing
Products
LLM Evaluation
LLM Evaluation

Benchmark LLM systems with research-backed metrics.

LLM Observability
LLM Observability

Trace, monitor, and alert on production LLM systems.

AI Red Teaming
AI Red Teaming

Stress-test LLM apps against adversarial attacks.

AI Governance
AI Governance

Enforce AI standards and controls across teams.

Contact salesSign Up
Contact salesSign Up

Summarization

Summarization is a single-turn metric to determine if your summarizer is generating facutally correct summaries.

Overview

The summarization metric is a single-turn metric that uses LLM-as-a-judge to evaluate an LLM's ability to summarize text. It generates close-ended questions from the original text and checks if the summary can answer them accurately.

Here's a good read on how our summarization metric was developed.

Required Parameters

These are the parameters you must supply in your test case to run evaluations for summarization metric:

inputstringRequired

The text sent to your summarization agent to summarize.

actual_outputstringRequired

The summary generated by your summarization agent for the given input.

Metric Parameters

These are the parameters you can configure for summarization metric when adding it to a metric collection:

assessment_questionslist of strings

A list of strings specifying the assessment questions. For example, Is the coverage score based on a percentage of 'yes' answers?.

truths_extraction_limitnumber

An integer specifying the maximum number of truths to extract from the actual output.

How Is It Calculated?

The summarization metric breaks the score into alignment_score and coverage_score.


Summarization=min(Alignement Score, Coverage Score)\text{Summarization} = \text{min(\text{Alignement Score}, \text{Coverage Score})}

The final score is the minumum of:

  • alignment_score which determines whether the summary contains hallucinated or contradictory information to the original text.
  • coverage_score which determines whether the summary contains the necessary information from the original text.

Usage

To run the summarization metric on Confident AI, add it to a single-turn metric collection. This will allow you to use summarization metric for:

  • Single-turn E2E testing
  • Single-turn component-level testing
  • Online and offline evals for traces and spans

Last updated on

Built byConfident AI