DAG
DAG is a custom metric that lets you build a deterministic decision tree for evaluation
Overview
DAG (deep acyclic graph) is a custom metric that lets you build a deterministic decision tree for evaluation. An LLM judge answers one focused question at each node, and you decide what every outcome scores.
It gives you more control over scoring than G-Eval by breaking a complex criteria into a series of small decisions, then mapping the outcome of those decisions to scores you define yourself.
Why DAG?
G-Eval asks an LLM judge for one holistic judgement and lets it generate the score. DAG splits that judgement into steps and takes the scoring away from the judge:
- Deterministic score mapping — you assign the score to every terminal outcome, so the same decision path always produces the same score
- Conditional rules and gates — you can express criteria such as "fail immediately if a required condition is failing, otherwise carry on evaluating quality"
- Focused decisions — each node judges one thing, which is more reliable than asking a judge to weigh everything at once
Use DAG when you can write your rubric out as rules. Use G-Eval when one holistic judgement over a subjective quality is enough.
DAG Node Types
A node is one step in the graph. There are three types, and the difference between them is what a node produces: a task node produces evidence, and the two judgement nodes produce a decision.
Task Node
A task node transforms the test case parameters, or the output of a task node above it, into structured evidence for the nodes below it. It asks the LLM to do a piece of work rather than to make a decision, which means it never assigns a score and never ends a path.
Use one when a decision needs something that is not directly present in the test case. Judging whether a summary covers every section is hard to do in one step, but extracting the list of sections first and then judging that list is two easy steps.
instructionsstringRequired
What the node should do with the parameters available to it, such as
Extract every heading from the actual output.
output_labelstringRequired
The name this node's output is presented under to the nodes below it, such as
Extracted headings.
evaluation_paramslist
The test case parameters the node can read, such as the input, the actual output, or the retrieval context.
labelstring
A name for the node, shown when you inspect how a score was reached.
Binary Judgement Node
A binary judgement node answers one yes-or-no question and follows either its True branch or its False branch. It has exactly two outcomes, no more and no fewer.
This is the node to reach for when you want a gate. A missing required field, a refusal, or a broken output format are all things that should decide the score on their own, before any question of quality is considered.
criteriastringRequired
The yes-or-no question the judge has to answer, such as
Are all required headings present?.
evaluation_paramslist
The test case parameters the node can read, in addition to the evidence passed down from its parent nodes.
labelstring
A name for the node, shown when you inspect how a score was reached.
Non-Binary Judgement Node
A non-binary judgement node classifies what it is given into one of several named outcomes that you define, and follows the branch matching the outcome it picks. The judge can only answer with one of your outcomes, so the branches are always exhaustive.
Use it for graded rather than pass-or-fail judgements. Where a binary node forces "correct or not", a non-binary node lets you separate Correct order, Partially out of order, and Incorrect order and score each one differently.
criteriastringRequired
The classification question the judge has to answer, such as
Classify the ordering of the headings.
evaluation_paramslist
The test case parameters the node can read, in addition to the evidence passed down from its parent nodes.
labelstring
A name for the node, shown when you inspect how a score was reached.
How a Path Ends
A DAG has no single exit node. Every branch of a judgement node carries a verdict, and the verdict decides whether the path stops there or carries on:
verdictboolean | stringRequired
The outcome this branch belongs to, True or False for a binary judgement,
and one of your named outcomes for a non-binary judgement.
scorenumber
An integer from 0 to 10 that ends the path and becomes the metric's score.
thennode | metric
The node to continue to instead of scoring, which can be another task node, another judgement node, or a whole other metric such as G-Eval.
Every branch sets exactly one of score or then. A branch that continues into another metric also ends the path, because that metric's score is adopted as the final score. Only a branch that continues into another node keeps the graph going, which is what guarantees that every path ends at a score.
Building the Graph
Three rules govern how the nodes fit together:
- Root nodes are where evaluation starts.
- A node can have several parents. Where two paths need the same evidence or the same decision, point both at the same node rather than duplicating it, and the work is only done once.
- The graph is validated as a whole. Cycles, a binary judgement missing one of its two branches, a non-binary judgement with repeated outcome names, and a branch that sets both
scoreandthenor neither are all rejected, so finish wiring every outcome before running the metric.
Create a DAG Metric
DAG metrics are built visually rather than written out. Navigate to Metrics → Library → Create Metric → DAG to open the DAG metric builder, where you add nodes, set each one's criteria and parameters, and connect their outcomes.
Once saved, a DAG metric behaves like any other metric on Confident AI. Add it to a metric collection to use it in evaluations.
Scaling beyond prototype?For teams evaluating Confident AI in productionTalk to usLast updated on