Create Metrics via Confident AI SDK
Overview
The Confident AI SDK exposes every Metrics Batch method on the platform. This page documents how to call these methods in all supported languages. See the introduction to install the SDK and set your API key.
Methods
Create Metrics Batch
Creates several GEVAL metrics at once and returns the ones created. Metrics whose name already exists in the project are skipped. DAG metrics must be created one at a time.
from confident_ai import ConfidentAI
from confident_ai.common import MetricEvaluationParam, MetricInput
client = ConfidentAI()
result = client.metrics_batch.create(
metrics=[
MetricInput(
name="Correctness",
multi_turn=False,
criteria="Determine if the actual output is correct based on the expected output.",
evaluation_params=[
MetricEvaluationParam.ACTUALOUTPUT,
MetricEvaluationParam.EXPECTEDOUTPUT
]
)
],
)For async mode, call a_create and await it as shown below:
result = await client.metrics_batch.a_create(...)Parameters
| Parameter | Type | Description |
|---|---|---|
metrics | List[MetricInput] | Required. The metrics to create. Names must be unique within the batch for the same multiTurn, and DAG metrics are not accepted here. See MetricInput. |
import { ConfidentAI } from "confident-ai";
import { MetricEvaluationParam } from "confident-ai/common";
const client = new ConfidentAI();
const result = await client.metricsBatch.create(
[
{
name: "Correctness",
multiTurn: false,
criteria: "Determine if the actual output is correct based on the expected output.",
evaluationParams: [
MetricEvaluationParam.ACTUALOUTPUT,
MetricEvaluationParam.EXPECTEDOUTPUT
]
}
],
);Parameters
| Parameter | Type | Description |
|---|---|---|
metrics | MetricInput[] | Required. The metrics to create. Names must be unique within the batch for the same multiTurn, and DAG metrics are not accepted here. See MetricInput. |
Returns
This method returns an object of type MetricList.
Types
JevQuestion
A bounded question a JEVAL metric asks the decision model: a noul (true/false) question, a score question over ordered levels (worst first), or a choice question over options with credits.
JevQuestion = Union[
JevNoulQuestion,
JevScoreQuestion,
JevChoiceQuestion,
]type JevQuestion =
| JevNoulQuestion
| JevScoreQuestion
| JevChoiceQuestion;A JevQuestion is one of the shapes below. Send the fields of one of them, never a mix of both.
A question the decision model answers true or false. trueDescription and falseDescription tell it what each answer means.
class JevNoulQuestion:
type: Literal["noul"]
question: str
weight: Optional[float] = None
true_description: Optional[str] = Field(default=None, alias="trueDescription")
false_description: Optional[str] = Field(default=None, alias="falseDescription")typeLiteral["noul"]Required
Marks this as a noul question.
questionstrRequired
The question the decision model answers.
weightOptional[float]
How heavily this question counts towards the metric's score, relative to the metric's other questions.
true_descriptionOptional[str]
What a true answer to this question means.
false_descriptionOptional[str]
What a false answer to this question means.
interface JevNoulQuestion {
type: "noul";
question: string;
weight?: number;
trueDescription?: string;
falseDescription?: string;
}type"noul"Required
Marks this as a noul question.
questionstringRequired
The question the decision model answers.
weightnumber
How heavily this question counts towards the metric's score, relative to the metric's other questions.
trueDescriptionstring
What a true answer to this question means.
falseDescriptionstring
What a false answer to this question means.
A question the decision model answers by picking one of levels, which are ordered worst first.
class JevScoreQuestion:
type: Literal["score"]
question: str
weight: Optional[float] = None
levels: List[str]typeLiteral["score"]Required
Marks this as a score question.
questionstrRequired
The question the decision model answers.
weightOptional[float]
How heavily this question counts towards the metric's score, relative to the metric's other questions.
levelsList[str]Required
Between 2 and 10 ordered levels, worst first.
interface JevScoreQuestion {
type: "score";
question: string;
weight?: number;
levels: string[];
}type"score"Required
Marks this as a score question.
questionstringRequired
The question the decision model answers.
weightnumber
How heavily this question counts towards the metric's score, relative to the metric's other questions.
levelsstring[]Required
Between 2 and 10 ordered levels, worst first.
A question the decision model answers by picking one of options, each carrying the credit that answer earns.
class JevChoiceQuestion:
type: Literal["choice"]
question: str
weight: Optional[float] = None
options: Dict[str, Optional[float]]typeLiteral["choice"]Required
Marks this as a choice question.
questionstrRequired
The question the decision model answers.
weightOptional[float]
How heavily this question counts towards the metric's score, relative to the metric's other questions.
optionsDict[str, Optional[float]]Required
Each option maps to a credit between 0 and 1, or null to mark the question not applicable when that option is likely.
interface JevChoiceQuestion {
type: "choice";
question: string;
weight?: number;
options: Record<string, number | null>;
}type"choice"Required
Marks this as a choice question.
questionstringRequired
The question the decision model answers.
weightnumber
How heavily this question counts towards the metric's score, relative to the metric's other questions.
optionsRecord<string, number | null>Required
Each option maps to a credit between 0 and 1, or null to mark the question not applicable when that option is likely.
Metric
A custom metric: how it scores, and which test case fields it needs.
class Metric:
id: str
name: str
algorithm: Optional[MetricAlgorithm]
criteria: Optional[str]
evaluation_steps: Optional[List[str]] = Field(alias="evaluationSteps")
rubric: Optional[List[Rubric]]
dag: Optional[MetricDag]
questions: Optional[List[JevQuestion]]
prompt_config: Optional[PromptMetricConfig] = Field(alias="promptConfig")
multi_turn: bool = Field(alias="multiTurn")
required_parameters: List[MetricEvaluationParam] = Field(alias="requiredParameters")idstrRequired
This is the unique id of the metric.
Example: "<METRIC-ID>"
namestrRequired
This is the name of the metric, unique within your project.
Example: "Correctness"
algorithmOptional[MetricAlgorithm]Required
See MetricAlgorithm.
criteriaOptional[str]Required
This is the criteria the metric scores against, or null when it uses evaluation steps.
Example: "Determine if the actual output is correct based on the expected output."
evaluation_stepsOptional[List[str]]Required
These are the steps the metric follows to score, or null when it uses criteria.
rubricOptional[List[Rubric]]Required
These are the score ranges that anchor how the metric scores, or null.
See Rubric.
dagOptional[MetricDag]Required
See MetricDag.
questionsOptional[List[JevQuestion]]Required
These are the questions a JEVAL metric asks the decision model, or null for other algorithms.
See JevQuestion.
prompt_configOptional[PromptMetricConfig]Required
This is the judge prompt of a PROMPT metric, or null for other algorithms.
See PromptMetricConfig.
multi_turnboolRequired
This is true when the metric evaluates conversations rather than single test cases.
Example: false
required_parametersList[MetricEvaluationParam]Required
The test case fields the metric needs to run.
Example: ["actualOutput","expectedOutput"]
interface Metric {
id: string;
name: string;
algorithm: MetricAlgorithm | null;
criteria: string | null;
evaluationSteps: string[] | null;
rubric: Rubric[] | null;
dag: MetricDag | null;
questions: JevQuestion[] | null;
promptConfig: PromptMetricConfig | null;
multiTurn: boolean;
requiredParameters: MetricEvaluationParam[];
}idstringRequired
This is the unique id of the metric.
Example: "<METRIC-ID>"
namestringRequired
This is the name of the metric, unique within your project.
Example: "Correctness"
algorithmMetricAlgorithm | nullRequired
See MetricAlgorithm.
criteriastring | nullRequired
This is the criteria the metric scores against, or null when it uses evaluation steps.
Example: "Determine if the actual output is correct based on the expected output."
evaluationStepsstring[] | nullRequired
These are the steps the metric follows to score, or null when it uses criteria.
rubricRubric[] | nullRequired
These are the score ranges that anchor how the metric scores, or null.
See Rubric.
dagMetricDag | nullRequired
See MetricDag.
questionsJevQuestion[] | nullRequired
These are the questions a JEVAL metric asks the decision model, or null for other algorithms.
See JevQuestion.
promptConfigPromptMetricConfig | nullRequired
This is the judge prompt of a PROMPT metric, or null for other algorithms.
See PromptMetricConfig.
multiTurnbooleanRequired
This is true when the metric evaluates conversations rather than single test cases.
Example: false
requiredParametersMetricEvaluationParam[]Required
The test case fields the metric needs to run.
Example: ["actualOutput","expectedOutput"]
MetricAlgorithm
The algorithm the metric is evaluated with. GEVAL scores against criteria or evaluation steps, DAG walks a decision graph, JEVAL asks a decision model bounded questions, PROMPT sends your own judge prompt to the evaluation model, CODE runs your own code, and DEFAULT is a built-in metric.
class MetricAlgorithm(Enum):
DEFAULT = "DEFAULT"
DAG = "DAG"
GEVAL = "GEVAL"
JEVAL = "JEVAL"
CODE = "CODE"
PROMPT = "PROMPT"enum MetricAlgorithm {
DEFAULT = "DEFAULT",
DAG = "DAG",
GEVAL = "GEVAL",
JEVAL = "JEVAL",
CODE = "CODE",
PROMPT = "PROMPT",
}DEFAULT · DAG · GEVAL · JEVAL · CODE · PROMPT
MetricDag
The decision graph of a DAG metric. It is validated on write, and metric references are resolved against the metrics in your project.
class MetricDag:
nodes: Dict[str, Any]nodesDict[str, Any]Required
The graph's nodes keyed by node id, as serialized by deepeval's DAG metric. Judgement nodes list their children by id; verdict nodes carry a verdict and a score, or point at another metric by metric_name.
Example: {"root":{"type":"BinaryJudgementNode","criteria":"Does the actual output answer the input?","evaluation_params":["input","actual_output"],"children":["pass","fail"]},"pass":{"type":"VerdictNode","verdict":true,"score":10},"fail":{"type":"VerdictNode","verdict":false,"score":0}}
interface MetricDag {
nodes: Record<string, unknown>;
}nodesRecord<string, unknown>Required
The graph's nodes keyed by node id, as serialized by deepeval's DAG metric. Judgement nodes list their children by id; verdict nodes carry a verdict and a score, or point at another metric by metric_name.
Example: {"root":{"type":"BinaryJudgementNode","criteria":"Does the actual output answer the input?","evaluation_params":["input","actual_output"],"children":["pass","fail"]},"pass":{"type":"VerdictNode","verdict":true,"score":10},"fail":{"type":"VerdictNode","verdict":false,"score":0}}
MetricEvaluationParam
A test case field a metric evaluates. Single-turn metrics take input, actualOutput, expectedOutput, context and expectedTools; multi-turn metrics take content, role, scenario, expectedOutcome, audio (GEVAL only) and turns; toolsCalled, retrievalContext, metadata and tags apply to both.
class MetricEvaluationParam(Enum):
INPUT = "input"
ACTUALOUTPUT = "actualOutput"
EXPECTEDOUTPUT = "expectedOutput"
CONTEXT = "context"
EXPECTEDTOOLS = "expectedTools"
CONTENT = "content"
ROLE = "role"
SCENARIO = "scenario"
EXPECTEDOUTCOME = "expectedOutcome"
AUDIO = "audio"
TURNS = "turns"
TOOLSCALLED = "toolsCalled"
RETRIEVALCONTEXT = "retrievalContext"
METADATA = "metadata"
TAGS = "tags"enum MetricEvaluationParam {
INPUT = "input",
ACTUALOUTPUT = "actualOutput",
EXPECTEDOUTPUT = "expectedOutput",
CONTEXT = "context",
EXPECTEDTOOLS = "expectedTools",
CONTENT = "content",
ROLE = "role",
SCENARIO = "scenario",
EXPECTEDOUTCOME = "expectedOutcome",
AUDIO = "audio",
TURNS = "turns",
TOOLSCALLED = "toolsCalled",
RETRIEVALCONTEXT = "retrievalContext",
METADATA = "metadata",
TAGS = "tags",
}INPUT · ACTUALOUTPUT · EXPECTEDOUTPUT · CONTEXT · EXPECTEDTOOLS · CONTENT · ROLE · SCENARIO · EXPECTEDOUTCOME · AUDIO · TURNS · TOOLSCALLED · RETRIEVALCONTEXT · METADATA · TAGS
MetricInput
A metric to create. GEVAL metrics need criteria or evaluationSteps; a DAG metric needs algorithm set to DAG and a dag; a JEVAL metric needs algorithm set to JEVAL and questions; a PROMPT metric needs algorithm set to PROMPT and a promptConfig.
class MetricInput:
name: str
multi_turn: Optional[bool] = Field(default=None, alias="multiTurn")
criteria: Optional[str] = None
evaluation_steps: Optional[List[str]] = Field(default=None, alias="evaluationSteps")
evaluation_params: Optional[List[MetricEvaluationParam]] = Field(default=None, alias="evaluationParams")
rubric: Optional[List[Rubric]] = None
algorithm: Optional[MetricAlgorithm] = None
dag: Optional[MetricDag] = None
questions: Optional[List[JevQuestion]] = None
prompt_config: Optional[PromptMetricConfig] = Field(default=None, alias="promptConfig")namestrRequired
The name of the metric, unique within your project.
Example: "Correctness"
multi_turnOptional[bool]
This is true when the metric evaluates conversations rather than single test cases. It decides which evaluationParams are valid and cannot be changed later.
Example: false
criteriaOptional[str]
The criteria the metric scores against. A GEVAL metric needs criteria or evaluationSteps.
Example: "Determine if the actual output is correct based on the expected output."
evaluation_stepsOptional[List[str]]
The steps the metric follows to score, as an alternative to criteria.
Example: ["Compare the actual output with the expected output.","Penalise any factual contradiction."]
evaluation_paramsOptional[List[MetricEvaluationParam]]
The test case fields the metric evaluates. A single-turn metric needs at least one, and every field must match multiTurn.
Example: ["actualOutput","expectedOutput"]
rubricOptional[List[Rubric]]
Score ranges that anchor how the metric scores.
See Rubric.
algorithmOptional[MetricAlgorithm]
See MetricAlgorithm.
dagOptional[MetricDag]
See MetricDag.
questionsOptional[List[JevQuestion]]
The questions a JEVAL metric asks the decision model. Required when algorithm is JEVAL.
See JevQuestion.
prompt_configOptional[PromptMetricConfig]
See PromptMetricConfig.
interface MetricInput {
name: string;
multiTurn?: boolean;
criteria?: string;
evaluationSteps?: string[];
evaluationParams?: MetricEvaluationParam[];
rubric?: Rubric[];
algorithm?: MetricAlgorithm;
dag?: MetricDag;
questions?: JevQuestion[];
promptConfig?: PromptMetricConfig;
}namestringRequired
The name of the metric, unique within your project.
Example: "Correctness"
multiTurnboolean
This is true when the metric evaluates conversations rather than single test cases. It decides which evaluationParams are valid and cannot be changed later.
Example: false
criteriastring
The criteria the metric scores against. A GEVAL metric needs criteria or evaluationSteps.
Example: "Determine if the actual output is correct based on the expected output."
evaluationStepsstring[]
The steps the metric follows to score, as an alternative to criteria.
Example: ["Compare the actual output with the expected output.","Penalise any factual contradiction."]
evaluationParamsMetricEvaluationParam[]
The test case fields the metric evaluates. A single-turn metric needs at least one, and every field must match multiTurn.
Example: ["actualOutput","expectedOutput"]
rubricRubric[]
Score ranges that anchor how the metric scores.
See Rubric.
algorithmMetricAlgorithm
See MetricAlgorithm.
dagMetricDag
See MetricDag.
questionsJevQuestion[]
The questions a JEVAL metric asks the decision model. Required when algorithm is JEVAL.
See JevQuestion.
promptConfigPromptMetricConfig
See PromptMetricConfig.
MetricList
class MetricList:
metrics: List[Metric]metricsList[Metric]Required
This is the list of metrics.
See Metric.
interface MetricList {
metrics: Metric[];
}metricsMetric[]Required
This is the list of metrics.
See Metric.
PromptMetricConfig
The judge prompt of a PROMPT metric, and how its variables are filled. The judge scores from 0 to 10 and the score is normalized to 0 to 1.
class PromptMetricConfig:
messages: List[PromptMetricConfigMessage]
variables: Dict[str, PromptMetricConfigVariables]messagesList[PromptMetricConfigMessage]Required
The judge prompt, as one or more messages.
variablesDict[str, PromptMetricConfigVariables]Required
Maps every {{name}} used in the messages to the data it is filled with.
interface PromptMetricConfig {
messages: PromptMetricConfigMessage[];
variables: Record<string, PromptMetricConfigVariables>;
}messagesPromptMetricConfigMessage[]Required
The judge prompt, as one or more messages.
variablesRecord<string, PromptMetricConfigVariables>Required
Maps every {{name}} used in the messages to the data it is filled with.
Rubric
A score range and the outcome it stands for. Ranges must not overlap.
class Rubric:
score_range: Tuple[int, int] = Field(alias="scoreRange")
expected_outcome: str = Field(alias="expectedOutcome")score_rangeTuple[int, int]Required
The inclusive start and end of the score range this outcome describes, each between 0 and 10, with the start no greater than the end.
Example: [8,10]
expected_outcomestrRequired
What a response scoring in this range looks like.
Example: "The answer is factually correct and complete."
interface Rubric {
scoreRange: [number, number];
expectedOutcome: string;
}scoreRange[number, number]Required
The inclusive start and end of the score range this outcome describes, each between 0 and 10, with the start no greater than the end.
Example: [8,10]
expectedOutcomestringRequired
What a response scoring in this range looks like.
Example: "The answer is factually correct and complete."
Last updated on