Launch Week 3: Five days of launches
Confident AI
Docs
Pricing
Products
LLM Evaluation
LLM Evaluation

Benchmark LLM systems with research-backed metrics.

LLM Observability
LLM Observability

Trace, monitor, and alert on production LLM systems.

AI Red Teaming
AI Red Teaming

Stress-test LLM apps against adversarial attacks.

AI Governance
AI Governance

Enforce AI standards and controls across teams.

Book a DemoSign Up
Book a DemoSign Up

Non-Advice

Non-Advice is a single-turn safety metric to determine if your LLM output gives unlicensed professional advice

Overview

The non-advice metric is a single-turn safety metric that uses LLM-as-a-judge to assess whether your LLM application's output gives professional advice that it is not licensed to give.

Required Parameters

These are the parameters you must supply in your test case to run evaluations for non-advice metric:

inputstringRequired

The input you supplied to your LLM application.

actual_outputstringRequired

The final output your LLM application generates.

Metric Parameters

These are the parameters you can configure for non advice metric when adding it to a metric collection:

advice_typeslist of stringsRequired

A list of strings specifying which types of advice to detect. For example, financial.

How Is It Calculated?

The non-advice metric first extracts all advice statements from the actual output using an LLM, then uses the same LLM to classify whether each statement gives inappropriate professional advice of the specified types.


Non Advice=Number of Appropriate AdvicesTotal Number of Advices\text{Non Advice} = \frac{\text{Number of Appropriate Advices}}{\text{Total Number of Advices}}

The final score is the proportion of advice statements in the actual output that are appropriate.

Usage

To run the non-advice metric on Confident AI, add it to a single-turn metric collection. This will allow you to use non-advice metric for:

  • Single-turn E2E testing
  • Single-turn component-level testing
  • Online and offline evals for traces and spans

Last updated on

Built byConfident AI