Operational Controls
Verify that projects meet required configuration standards.
Operational controls verify that a project is configured according to your standards. They are static checks assessed from the project's current state, with no thresholds to configure.
These controls ship with Confident AI and can't be created manually. Add the controls your organization requires to a governance policy, then assign that policy to projects.
When should you use operational controls?
- When a requirement can be verified by checking whether a project has a specific capability configured or enabled.
- When you need to enforce a consistent governance baseline across every governed project.
- When projects must have scheduled evaluations, recent tracing activity, notification integrations, reusable metrics, or threat detection.
Alert controls
Alert controls verify that teams are notified when monitored behavior needs attention.
| Control | Passes when the project has… |
|---|---|
| Has scheduled alerts | At least one scheduled alert |
| Has scheduled alerts on traces | A scheduled alert that evaluates traces |
| Has scheduled alerts on spans | A scheduled alert that evaluates spans |
| Has scheduled alerts on threads | A scheduled alert that evaluates threads |
Scheduled job controls
Scheduled job controls verify that evaluations and security assessments run repeatedly instead of only on demand.
| Control | Passes when the project has… |
|---|---|
| Has scheduled eval test runs | A recurring evaluation test run |
| Has scheduled risk assessments | A recurring red teaming risk assessment |
Dataset controls
Dataset controls verify that test cases are stored and versioned for repeatable evaluation.
| Control | Passes when the project has… |
|---|---|
| Has dataset ingestion | At least one dataset |
| Has single-turn datasets | A dataset containing single-turn test cases |
| Has multi-turn datasets | A dataset containing multi-turn test cases |
| Has dataset versions | At least one versioned dataset |
Tracing controls
Tracing controls verify that production activity is visible and can be routed to human review.
| Control | Passes when the project has… |
|---|---|
| Has logged traces | Traces logged in the last 30 days |
| Has logged threads | Threads logged in the last 30 days |
| Has queue ingestion | An annotation queue receiving items |
Integration controls
Integration controls verify that Confident AI can notify the right teams and create follow-up work.
| Control | Passes when the project has… |
|---|---|
| Has alert integrations | A Slack, Discord, Email, PagerDuty, or Microsoft Teams notification integration |
| Has ticketing integrations | A Linear or GitHub Issues integration |
Classifier controls
Classifier controls verify that automated classification is enabled for incoming production data.
| Control | Passes when the project has… |
|---|---|
| Has trace classifiers enabled | Trace classifiers enabled |
| Has thread classifiers enabled | Thread classifiers enabled |
Metric controls
Metric controls verify that the project has reusable measurements for evaluation and reporting.
| Control | Passes when the project has… |
|---|---|
| Has custom metrics | At least one custom metric |
| Has metric collections | At least one metric collection |
Threat detection controls
Threat detection controls verify that production traffic is monitored for security threats.
| Control | Passes when the project has… |
|---|---|
| Has trace threat detection enabled | Threat detection enabled for traces |
| Has thread threat detection enabled | Threat detection enabled for threads |
Last updated on