Classifier Task

Overview

A classifier task assigns a label to traces and threads as they are ingested, based on a description and a set of labels you define. The labels it produces surface as Signals and as filterable dimensions across the Observatory and Dashboards. On the Workflows page, classifier tasks are listed under Classifiers.

How a Classifier Thinks

When a classifier runs, the underlying LLM receives:

1. The classifier's description — what is this classifier looking for?
2. The list of labels with each label's description — when should this label be assigned?
3. The trace or thread payload — input, output, metadata, error, tags, and (for threads) the conversation turns

The model picks one label or returns "no match." There is no rule engine, no metadata-based pre-filtering, and no regex — everything depends on how the descriptions read against the data.

Create a Classifier Task

1. Navigate to Workflows and select Traces or Threads
2. Expand Classifiers in the panel beside the graph
3. Click New classifier
4. Give the classifier a name and description
5. Save the classifier

After creating, click its row to open the classifier editor in a side drawer. This is where you manage labels and configure generation settings.

Labels

Each classifier has one or more labels. Add labels manually with New Label (Name + Description) or auto-suggest them in bulk via Generate Labels. Each label has its own enable toggle — disabled labels are not assigned to new items but remain in the classifier's history.

Generate Labels

If you don't yet know what labels you need, Generate Labels proposes a set from your recent traces or threads. Click Configure Generation first to set the prompt and clustering parameters, then Generate Labels to run the three-stage pipeline:

1. Summarizing — the model summarizes a sample of your recent traces or threads using the configured summary prompt
2. Clustering — summaries are grouped into the configured number of clusters using K-means
3. Labeling — each cluster is turned into a candidate label (name + description) and shown on the row as Recommended

Recommended labels show Accept (✓) and Decline (✕) actions instead of the regular edit menu. Accepted labels become regular labels and start running on the next ingestion tick. Declined labels are deleted. Re-running generation while recommendations are still pending discards the old ones first.

Auto Classify

The Auto Classify toggle in the classifier editor is separate from the top-level Enabled toggle:

• Enabled — turns the classifier on or off entirely
• Auto Classify — when on, the classifier may propose new labels (saved as Recommended on the labels list) when none of your existing labels fit a trace or thread; when off, it can only pick from the labels you've already defined or return no match

Leave Auto Classify on if you want to keep discovering edge cases, and off if you want a fixed taxonomy.

Sample Rate

The Sample Rate below the classifier list controls what fraction of incoming traces (or threads) are sent for classification — 1.0 classifies everything, 0.1 classifies roughly one in ten. This is a project-wide setting shared across all enabled classifiers for that data model.

Time Limit (threads only)

For thread classifiers, Time Limit defines how many seconds of inactivity must pass before a thread is eligible for classification. The classification runs once no new trace has arrived for that period. Set this long enough that follow-up turns have stopped arriving, but not so long that you miss the conversation window.

Cost

Each classification logs a usage event for billing. See Project Settings → Data Usage under the Signals line for live usage and projected cost.

Next Steps

Last updated on October 6, 2026