Image Helpfulness
Image Helpfulness is a single-turn image metric to determine if images help a reader understand the text
Overview
The image helpfulness metric is a single-turn image metric that uses MLLM-as-a-judge to assess whether each image in your LLM application's output helps a reader understand the text it sits alongside.
Required Parameters
These are the parameters you must supply in your test case to run evaluations for image helpfulness metric:
inputlist of string | MLLMImageRequired
The input you supplied to your LLM application.
actual_outputlist of string | MLLMImageRequired
The final output your LLM application generates, as the text and images it returned in order.
Metric Parameters
These are the parameters you can configure for image helpfulness metric when adding it to a metric collection:
max_context_sizenumber
An integer specifying the maximum number of characters of the text above and
below the image to consider. For example, 100.
How Is It Calculated?
The image helpfulness metric scores each image in the actual output against the text directly above and below it using an MLLM, then averages those scores across all of the images.
The final score is the average helpfulness score of the images found in the actual output.
Usage
To run the image helpfulness metric on Confident AI, add it to a single-turn metric collection. This will allow you to use image helpfulness metric for:
- Single-turn E2E testing
- Single-turn component-level testing
- Online and offline evals for traces and spans
Last updated on