Platform Settings
Confident AI provides platform settings for configuring projects, managing organization-wide resources, and automating administrative workflows through the Admin SDK.
Overview
Settings in Confident AI are organized into three areas:
- Project Settings configure settings that apply to a specific project.
- Organization Settings manage settings that apply across all projects in your organization.
- Admin SDK manages organization and project resources programmatically.
Project vs Organization
When you create an account on Confident AI, an organization and a project within that organization are automatically created for you. Each organization can have multiple projects and users.
-
Organization — The top-level container for all your Confident AI resources. Subscription plans and billing are managed at the organization level.
-
Project — A workspace where data and access permissions are isolated. All data (test cases, metrics, datasets, traces, etc.) is separated at the project level.
Learn more about projects
See how to structure your projects and why data separation matters.
Project Settings
Project settings allow you to customize configurations for individual projects, including team access, evaluation models, and integrations.
Manage Team Members
Add team members to your project and manage their access.
API Keys
Generate and manage API keys for authentication.
Evaluation Models
Configure the models used for LLM-as-a-judge evaluations.
Model Costs
Configure LLM model costs for usage tracking and cost estimation.
Data Usage
View data consumption, ingestion metrics, and cost insights.
Integrations
Connect Slack, Discord, Teams, PagerDuty, email, Linear, or GitHub Issues.
Threat Detection
Continuously scan incoming traces and threads for security vulnerabilities.
Knowledge Base
Connect external data sources or upload documents for dataset generation.
AI Connections
Connect your AI provider API keys for evaluations.
Annotation Options
Define criteria for human-in-the-loop annotations.
Data Retention
Manage retention policies for your project data.
Audit Logs
View a log of all actions performed in your project.
Roles & Permissions
Control access levels for team members.
Organization Settings
Organization settings control resources and configurations that apply to all projects within your organization.
Projects
View and manage all projects in your organization.
Users
Manage users and their access across the organization.
SSO
Configure Single Sign-On for secure authentication.
Model Credentials
Manage API credentials for AI models shared across projects.
Data Retention
Set default retention policies for all projects.
Audit Logs
View a log of all actions across your organization.
Feature Access
Control which features are enabled for projects.
Admin SDK
The Admin SDK lets you manage organization and project resources programmatically. Use it to create projects, invite members, manage roles, and provision API keys without configuring each resource manually in the platform UI.
Admin SDK Introduction
Learn what the Admin SDK manages and when to use it.
Admin SDK Quickstart
Install the SDK, configure an Organization API Key, and make your first request.
Projects
Create, update, and delete projects programmatically.
Last updated on