Confident AI provides comprehensive settings to help you configure your workspace exactly how you need it. Settings are organized into two categories based on their scope and impact.
Settings in Confident AI are divided into two levels:
Teams focused on org-wide AI adoption—such as Centers of Excellence, innovation teams, or platform teams in large enterprises—will find Organization Settings particularly relevant. These settings help manage resources and governance for multiple AI teams across the company.
When you create an account on Confident AI, an organization and a project within that organization are automatically created for you. Each organization can have multiple projects and users.
Organization — The top-level container for all your Confident AI resources. Subscription plans and billing are managed at the organization level.
Project — A workspace where data and access permissions are isolated. All data (test cases, metrics, datasets, traces, etc.) is separated at the project level.
Project settings allow you to customize configurations for individual projects, including team access, evaluation models, and integrations.
Add team members to your project and manage their access.
Generate and manage API keys for authentication.
Configure the models used for LLM-as-a-judge evaluations.
Automatically run metric collections on incoming traces, spans, and threads.
Configure LLM model costs for usage tracking and cost estimation.
View data consumption, ingestion metrics, and cost insights.
Set up Slack, Discord, Teams, or email notifications.
Connect your AI provider API keys for evaluations.
Configure alerts for important project events.
Define criteria for human-in-the-loop annotations.
Manage retention policies for your project data.
View a log of all actions performed in your project.
Control access levels for team members.
Organization settings control resources and configurations that apply to all projects within your organization.
View and manage all projects in your organization.
Manage users and their access across the organization.
Configure Single Sign-On for secure authentication.
Manage API credentials for AI models shared across projects.
Set default retention policies for all projects.
View a log of all actions across your organization.
Control which features are enabled for projects.