Platform Settings
Confident AI provides platform settings for configuring projects, managing organization-wide resources, and automating administrative workflows through the Admin SDK.
Overview
Settings in Confident AI are organized into three areas:
- Project Settings configure settings that apply to a specific project.
- Organization Settings manage settings that apply across all projects in your organization.
- Admin SDK manages organization and project resources programmatically.
Teams focused on org-wide AI adoption—such as Centers of Excellence, innovation teams, or platform teams in large enterprises—will find Organization Settings particularly relevant. These settings help manage resources and access for multiple AI teams across the company.
Project vs Organization
When you create an account on Confident AI, an organization and a project within that organization are automatically created for you. Each organization can have multiple projects and users.
-
Organization — The top-level container for all your Confident AI resources. Subscription plans and billing are managed at the organization level.
-
Project — A workspace where data and access permissions are isolated. All data (test cases, metrics, datasets, traces, etc.) is separated at the project level.
Project Settings
Project settings allow you to customize configurations for individual projects, including team access, evaluation models, and integrations.
Add team members to your project and manage their access.
Generate and manage API keys for authentication.
Configure the models used for LLM-as-a-judge evaluations.
Configure LLM model costs for usage tracking and cost estimation.
View data consumption, ingestion metrics, and cost insights.
Connect Slack, Discord, Teams, PagerDuty, email, Linear, or GitHub Issues.
Continuously scan incoming traces and threads for security vulnerabilities.
Connect external data sources or upload documents for dataset generation.
Connect your AI provider API keys for evaluations.
Define criteria for human-in-the-loop annotations.
Manage retention policies for your project data.
View a log of all actions performed in your project.
Control access levels for team members.
Organization Settings
Organization settings control resources and configurations that apply to all projects within your organization.
View and manage all projects in your organization.
Manage users and their access across the organization.
Configure Single Sign-On for secure authentication.
Manage API credentials for AI models shared across projects.
Set default retention policies for all projects.
View a log of all actions across your organization.
Control which features are enabled for projects.
Admin SDK
The Admin SDK lets you manage organization and project resources programmatically. Use it to create projects, invite members, manage roles, and provision API keys without configuring each resource manually in the platform UI.