Launch Week 3: Five days of launches
Confident AI
Docs
Pricing
Products
LLM Evaluation
LLM Evaluation

Benchmark LLM systems with research-backed metrics.

LLM Observability
LLM Observability

Trace, monitor, and alert on production LLM systems.

AI Red Teaming
AI Red Teaming

Stress-test LLM apps against adversarial attacks.

AI Governance
AI Governance

Enforce AI standards and controls across teams.

Book a DemoSign Up
Book a DemoSign Up

Pattern Match

Pattern Match is a single-turn metric to determine if your LLM output matches a regular expression

Overview

The pattern match metric is a single-turn metric that checks whether your LLM application's output matches a regular expression you supply. Unlike other metrics, it does not use an LLM for evaluation.

Required Parameters

These are the parameters you must supply in your test case to run evaluations for pattern match metric:

inputstringRequired

The input you supplied to your LLM application.

actual_outputstringRequired

The final output your LLM application generates.

Metric Parameters

These are the parameters you can configure for pattern match metric when adding it to a metric collection:

patternstringRequired

A string specifying the regular expression pattern the actual output must match. For example, ^[\\w\\.-]+@[\\w\\.-]+\\.\\w+$.

ignore_caseboolean

A boolean specifying whether the pattern matching should be case-insensitive.

How Is It Calculated?

The pattern match metric uses a deterministic approach to calculate the score by matching the actual output against the supplied regular expression using Python's built-in re module.


Pattern Match={1if the actual output fully matches the pattern0otherwise\text{Pattern Match} = \begin{cases} 1 & \text{if the actual output fully matches the pattern} \\ 0 & \text{otherwise} \end{cases}

The final score is binary, the actual output either matches the pattern in full or it does not.

Usage

To run the pattern match metric on Confident AI, add it to a single-turn metric collection. This will allow you to use pattern match metric for:

  • Single-turn E2E testing
  • Single-turn component-level testing
  • Online and offline evals for traces and spans

Last updated on

Built byConfident AI