Image Reference
Image Reference is a single-turn image metric to determine if images are accurately referred to in text
Overview
The image reference metric is a single-turn image metric that uses MLLM-as-a-judge to assess whether each image in your LLM application's output is accurately referred to and explained by the text it sits alongside.
Required Parameters
These are the parameters you must supply in your test case to run evaluations for image reference metric:
inputlist of string | MLLMImageRequired
The input you supplied to your LLM application.
actual_outputlist of string | MLLMImageRequired
The final output your LLM application generates, as the text and images it returned in order.
Metric Parameters
These are the parameters you can configure for image reference metric when adding it to a metric collection:
max_context_sizenumber
An integer specifying the maximum number of characters of the text above and
below the image to consider. For example, 100.
How Is It Calculated?
The image reference metric scores each image in the actual output against the text directly above and below it using an MLLM, then averages those scores across all of the images.
The final score is the average reference score of the images found in the actual output.
Usage
To run the image reference metric on Confident AI, add it to a single-turn metric collection. This will allow you to use image reference metric for:
- Single-turn E2E testing
- Single-turn component-level testing
- Online and offline evals for traces and spans
Last updated on