Launch Week 02 wrapped — explore all five launches
Confident AI
Docs
Pricing
Products
LLM Evaluation
LLM Evaluation

Benchmark LLM systems with research-backed metrics.

LLM Observability
LLM Observability

Trace, monitor, and alert on production LLM systems.

AI Red Teaming
AI Red Teaming

Stress-test LLM apps against adversarial attacks.

AI Governance
AI Governance

Enforce AI standards and controls across teams.

Book a DemoSign Up
Book a DemoSign Up

Contextual Precision

Contextual Precision is a single-turn metric used to evaluate a RAG retriever

Overview

The contextual precision metric is a single-turn RAG metric that uses LLM-as-a-judge to evaluate how well your retriever ranks the retrieved context based on the input query.

Required Parameters

These are the parameters you must supply in your test case to run evaluations for contextual precision metric:

inputstringrequired

The input query you supply to your RAG application.

expected_outputstringrequired

The expected output your RAG application has to generate for a given input.

retrieval_contextlist of stringrequired

The retrieved context your retriever outputs for a given input sorted by their rank.

How Is It Calculated?

The contextual precision metric evaluates each retrieved node using an LLM to check if it is correctly ranked for relevance to the input. It then calculates the final score using the following equation:


\text{Contextual Precision} = \frac{1}{\text{Num of Relevant Nodes}}\sum_{k=1}^{n}(\frac{\text{Num of Relevant Nodes Upto position k}}{k} \times r_k)

A high contextual precison score indicates that all the retrieved nodes are in the order of their relevance to the input.

Create Locally

You can create the ContextualPrecisionMetric in deepeval as follows:

from deepeval.metrics import ContextualPrecisionMetric

metric = ContextualPrecisionMetric()

Here's a list of parameters you can configure when creating a ContextualPrecisionMetric:

thresholdnumberdefault: 0.5

A float to represent the minimum passing threshold.

modelstring | Objectdefault: gpt-4.1

A string specifying which of OpenAI's GPT models to use OR any custom LLM model of type DeepEvalBaseLLM.

include_reasonbooleandefault: true

A boolean to enable the inclusion a reason for its evaluation score.

async_modebooleandefault: true

A boolean to enable concurrent execution within the measure() method.

strict_modebooleandefault: false

A boolean to enforce a binary metric score: 0 for perfection, 1 otherwise.

verbose_modebooleandefault: false

A boolean to print the intermediate steps used to calculate the metric score.

evaluation_templateContextualPrecisionTemplatedefault: deepeval's template

An instance of ContextualPrecisionTemplate object, which allows you to override the default prompts used to compute the ContextualPrecisionMetric score.

Create Remotely

For users not using deepeval python, or want to run evals remotely on Confident AI, you can use the contextual precision metric by adding it to a single-turn metric collection. This will allow you to use contextual precision metric for:

  • Single-turn E2E testing
  • Single-turn component-level testing
  • Online and offline evals for traces and spans
Built byConfident AI