Roles & Permissions

Roles and permissions let you control what each team member can do within a project.

Default Roles

Every project comes with three preset roles. Each preset role includes all project permissions unless noted otherwise.

Owner

Full access to all resources in the project. No permission exclusions.

Manager

Includes all permissions. Managers have the same access as Owners.

Member

Includes everything except project:delete, retentionConfig:manage, user:manage, user:delete, and iam:manage. Members cannot delete the project, manage retention settings, assign roles to project members, remove members from the project, or manage roles and policies.

Custom Roles

You can create custom roles to fit your team’s needs. To create a new role:

1. Navigate to Project Settings → Roles & Permissions
2. Click New Role
3. Enter a name and description for the role
4. Assign a policy to the role
5. Click Save

Common custom roles include “Annotator” roles that only allow a certain group of users for read and write access to datasets.

Custom Policies

Policies define the specific permissions a role has. Each permission controls access to a particular action, like dataset:read, dataset:create, golden:update, or trace:delete.

To create a custom policy:

1. Navigate to Project Settings → Roles & Permissions
2. Scroll to Custom Policies and click New Policy
3. Enter a name for the policy
4. Select the permissions you want to include
5. Click Save

Once created, you can assign your custom policy to any role.

Permission Syntax

Permissions follow a resource:action format. For example, dataset:read grants read access to datasets, while trace:evaluate allows running evaluations on traces.

Actions:

• create — Create new resources
• read — View resources
• update — Modify existing resources
• delete — Remove resources
• evaluate — Run evaluations on the resource
• assign — Assign resources to users or queues
• manage — Includes create, update, and delete (varies by resource)

Permission resources:

• dataset, golden — Datasets and their goldens
• metric, metricCollection — Metric scores and collections
• evaluationRule — Rules that automatically run metric collections on incoming traces, spans, and threads
• trace, span, thread, endUser — Observability data
• testRun, testCase, experiment — Evaluation runs
• prompt, promptVersion, promptLabel — Prompts and their versions
• annotationQueue, queue_item — Annotation queues and their items
• project, apiKey, modelCredential, modelCost, evaluationModel — Project settings and configuration
• retentionConfig — Data retention settings for traces, spans, test runs, datasets, and prompts (how long each is kept)
• iam — Project roles and policies
• transformer, aiConnection, alertConfig, integration — Integrations and tools
• user — Team member management; user:manage controls assigning roles to members, while user:delete controls removing members from the project

Not every resource will have all actions. For example, dataset doens’t have project_member, while annotation_queue doesn’t have evaluate. You can find the full list of permissions on the roles & permissions page.