Task Completion

Task Completion is a single-turn metric to determine an agent's task completion score

Overview

The task completion metric is a single-turn metric that uses LLM-as-a-judge to assess whether your LLM agent successfully completes the given task based on its entire trace.

How Is It Calculated?

The task completion metric uses an LLM to extract the task and outcome from each step in the trace, then uses the same LLM to determine if the task was satisfied based on the outcome.

\text{Task Completion} = \text{Alignment Score}(\text{Task}, \text{Outcome})

The final score is the alignment of task and outcome as extracted from the trace.

Create Locally

You can create the TaskCompletionMetric in deepeval as follows:

from deepeval.metrics import TaskCompletionMetric

metric = TaskCompletionMetric()

Here's a list of parameters you can configure when creating a TaskCompletionMetric:

thresholdnumberdefault: 0.5

A float to represent the minimum passing threshold.

taskstring

A string representing the task to be completed. If no task is supplied, it is automatically inferred from the trace.

modelstring | Objectdefault: gpt-4.1

A string specifying which of OpenAI's GPT models to use OR any custom LLM model of type DeepEvalBaseLLM.

include_reasonbooleandefault: true

A boolean to enable the inclusion a reason for its evaluation score.

async_modebooleandefault: true

A boolean to enable concurrent execution within the measure() method.

strict_modebooleandefault: false

A boolean to enforce a binary metric score: 0 for perfection, 1 otherwise.

verbose_modebooleandefault: false

A boolean to print the intermediate steps used to calculate the metric score.

Create Remotely

For users not using deepeval python, or want to run evals remotely on Confident AI, you can use the task completion metric by adding it to a single-turn metric collection. This will allow you to use task completion metric for:

Single-turn E2E testing
Online and offline evals for traces