Launch Week 3: Five days of launches
Confident AI
Docs
Pricing
Products
LLM Evaluation
LLM Evaluation

Benchmark LLM systems with research-backed metrics.

LLM Observability
LLM Observability

Trace, monitor, and alert on production LLM systems.

AI Red Teaming
AI Red Teaming

Stress-test LLM apps against adversarial attacks.

AI Governance
AI Governance

Enforce AI standards and controls across teams.

Book a DemoSign Up
Book a DemoSign Up

Topic Adherence

Topic Adherence is a multi-turn metric to determine if your chatbot stays on its relevant topics

Overview

The topic adherence metric is a multi-turn metric that uses LLM-as-a-judge to evaluate whether your chatbot answers only the questions that fall within the topics it is meant to cover.

Required Parameters

These are the parameters you must supply in your test case to run evaluations for topic adherence metric:

turnslist of TurnRequired

A list of Turns as exchanges between user and assistant.

Parameters of Turn:

roleuser | assistantRequired

The role of the person speaking, it's either user or assistant

contentstringRequired

The content provided by the role for the turn

Metric Parameters

These are the parameters you can configure for topic adherence metric when adding it to a metric collection:

relevant_topicslist of stringsRequired

A list of strings that define what topics your LLM agent can answer. For example, financial.

How Is It Calculated?

The topic adherence metric first extracts the question and answer pairs from the conversation, then uses an LLM to classify each pair against the relevant topics you supplied.


Topic Adherence=Number of True Positives and True NegativesTotal Number of QA Pairs\text{Topic Adherence} = \frac{\text{Number of True Positives and True Negatives}}{\text{Total Number of QA Pairs}}

The final score is the proportion of question and answer pairs your chatbot handled correctly in the conversation.

Usage

To run the topic adherence metric on Confident AI, add it to a multi-turn metric collection. This will allow you to use topic adherence metric for:

  • Multi-turn E2E testing
  • Online and offline evals for traces and spans

Last updated on

Built byConfident AI