Runtime Controls
Assess production observability metrics against a threshold.
Runtime controls assess production observability data against a threshold. They use the same metric model as alerts, but turn the result into an auditable governance requirement.
Each runtime control evaluates a trailing 24-hour window.
When should you use runtime controls?
- When a requirement depends on measurable production behavior rather than project configuration.
- When you need to continuously enforce reliability, quality, latency, cost, traffic, or human-feedback thresholds.
- When the requirement should apply to a precisely filtered set of traces, spans, threads, metric results, or annotations.
Configure a runtime control
A runtime control consists of four parts:
- Data model — select Trace, Span, Thread, Metric data, or Annotation.
- Aggregation — select the value to compute. The available aggregations depend on the data model.
- Threshold — select Above or Below, then enter a numeric value.
- Filters — optionally narrow the evaluated data by environment, tags, metadata, or other available properties.
Data models and aggregations
Trace
Trace controls aggregate end-to-end LLM application requests. Use them to govern request volume, quality, latency, cost, and user activity.
| Aggregation | What it measures |
|---|---|
| Trace count | Total number of matching traces |
| Error rate | Percentage of matching traces that contain an error |
| Pass rate | Percentage of matching traces that pass evaluation |
| Unique end users | Number of distinct end users represented by matching traces |
| Unique threads | Number of distinct threads represented by matching traces |
| Avg latency | Mean latency across matching traces |
| P50 latency | Median latency across matching traces |
| P90 latency | 90th-percentile latency across matching traces |
| P99 latency | 99th-percentile latency across matching traces |
| Total cost | Sum of LLM cost across matching traces |
| Avg cost | Mean LLM cost per matching trace |
| Unique metadata values | Number of distinct values for selected metadata |
Span
Span controls aggregate individual operations within traces. Use them to isolate the behavior of a model call, tool call, retriever, or another instrumented operation.
| Aggregation | What it measures |
|---|---|
| Span count | Total number of matching spans |
| Error rate | Percentage of matching spans that contain an error |
| Error count | Total number of matching spans that contain an error |
| Avg latency | Mean latency across matching spans |
| P50 latency | Median latency across matching spans |
| P90 latency | 90th-percentile latency across matching spans |
| P99 latency | 99th-percentile latency across matching spans |
| Total cost | Sum of LLM cost across matching spans |
| Input cost | Sum of input-token cost across matching spans |
| Output cost | Sum of output-token cost across matching spans |
| Avg cost | Mean LLM cost per matching span |
| Unique metadata values | Number of distinct values for selected metadata |
| Input tokens | Total input-token usage across matching spans |
| Output tokens | Total output-token usage across matching spans |
| Total tokens | Total input and output tokens across matching spans |
Thread
Thread controls aggregate multi-turn conversations.
| Aggregation | What it measures |
|---|---|
| Thread count | Total number of matching threads |
| Unique end users | Number of distinct end users represented by matching threads |
| Unique metadata values | Number of distinct values for selected metadata |
Metric data
Metric data controls aggregate evaluation metric results. First select the metric data source—Trace, Span, Thread, or Test run—then choose the aggregation.
| Aggregation | What it measures |
|---|---|
| Metric count | Total number of matching metric results |
| Avg score | Mean score across matching metric results |
| Median score | Median score across matching metric results |
| Pass rate | Percentage of matching metric results that pass |
| Failure rate | Percentage of matching metric results that fail |
Annotation
Annotation controls aggregate human feedback attached to your data.
| Aggregation | What it measures |
|---|---|
| Annotation count | Total number of matching annotations |
| Avg rating | Mean rating across matching annotations |
Threshold behavior
The threshold direction describes the condition that causes the control to fail:
| Direction | The control fails when… | Example |
|---|---|---|
| Above | The aggregated value is greater than the threshold | Fail when error rate is above 5% |
| Below | The aggregated value is less than the threshold | Fail when successful trace count is below 1,000 |
If the aggregated value remains within the required bound, the control passes.
Common runtime controls
- Reliability — require error rate to remain below an agreed limit.
- Latency — require average or percentile latency to remain below an SLA.
- Cost — require token cost to remain below a daily budget.
- Traffic — require trace, span, thread, metric, or annotation volume to remain above a minimum.
- Adoption — require the number of unique end users to remain above a target.
- Quality — require evaluation pass rate, metric scores, or human ratings to remain above a target.
Last updated on