Deploy with Helm | Confident AI Docs

With the infrastructure in place, install the confident-ai Helm chart using the Terraform outputs. The chart pulls its images from Confident AI’s registry and installs the app plus in-cluster ClickHouse. Because the app uses EKS Pod Identity, its ServiceAccount needs no annotation.

The recommended setup keeps app secrets in AWS Secrets Manager (synced by the External Secrets Operator) and runs Redis on ElastiCache. Both are provisioned by the Terraform module. In-cluster Redis and a Kubernetes Secret are supported as a simpler alternative, see Simpler option.

Two things come from Confident AI with your Enterprise license:

Image pull credentials: an AWS access key ID and secret that let the cluster pull the first-party images (hosted in a separate ECR account). The chart uses them to mint and refresh the pull secret automatically.
License key (CONFIDENT_LICENSE_KEY): the signed key that enables your plan’s features.

About OPENAI_API_KEY: it backs the built-in Confident AI evaluation provider, which runs on OpenAI. It is optional. Omit it and connect your own model provider or an LLM gateway per project from AI Connections in the app instead.

Create the namespace

$ kubectl create namespace confident-ai

Put the secrets in Secrets Manager

DATABASE_URL comes straight from Terraform. Each key in this JSON object becomes an app secret:

$ SECRET_NAME=$(terraform output -raw secrets_manager_secret_name)
$ REGION=$(terraform output -raw region)
$ 
$ aws secretsmanager put-secret-value --secret-id "$SECRET_NAME" --region "$REGION" \
>   --secret-string "{
>     \"DATABASE_URL\":\"$(terraform output -raw database_url)\",
>     \"BETTER_AUTH_SECRET\":\"$(openssl rand -hex 32)\",
>     \"OPENAI_API_KEY\":\"sk-...\",
>     \"CONFIDENT_LICENSE_KEY\":\"...\"
>   }"

This uses the secret Terraform created with confident_create_secrets_manager = true. If you did not enable it, either turn it on and re-apply, or use the simpler option.

Install the External Secrets Operator

Install it into the confident-ai namespace as the external-secrets-sa account, the exact account Terraform gave the read role to, so ESO inherits the AWS access through Pod Identity with no keys:

$ helm repo add external-secrets https://charts.external-secrets.io && helm repo update
$ helm install external-secrets external-secrets/external-secrets \
>   -n confident-ai \
>   --set installCRDs=true \
>   --set serviceAccount.name=external-secrets-sa

Set up ALB ingress (for HTTPS)

To serve the app on a domain over HTTPS, install the AWS Load Balancer Controller (it provisions an ALB from your Ingress) and request an ACM certificate in your region covering all four subdomains, app., api., evals., and otel. of yourdomain.com (a wildcard *.yourdomain.com also works). You will reference the certificate ARN in the values file below.

Only trying it out? Set ingress.enabled: false in the values file and reach the app with kubectl port-forward -n confident-ai svc/confident-frontend 3000:3000.

Write the values file

Save this as values.aws.yaml. Fill the bracketed values from your Terraform outputs and the credentials Confident AI gave you.

1 image:
2   tag: "v2.0.18"   # use the version Confident AI gives you
3 
4 # The chart mints and refreshes the ECR pull secret from these credentials.
5 imagePullSecrets:
6   - name: ecr-registry-credentials
7 imagePullSecretRefresh:
8   enabled: true
9   region: us-east-1
10   awsAccessKeyId: "<from Confident AI>"
11   awsSecretAccessKey: "<from Confident AI>"
12 
13 config:
14   cloudProvider: AWS
15   frontendUrl: https://app.yourdomain.com
16   backendUrl: https://api.yourdomain.com
17   subdomain: yourdomain.com
18 
19 serviceAccount:
20   create: true          # Pod Identity is already wired to this SA, no annotation needed
21 
22 storage:
23   testCasesBucket: <test_cases_bucket>
24   payloadsBucket: <payloads_bucket>
25   aws:
26     region: <region>
27 
28 # Recommended: app secrets come from AWS Secrets Manager via ESO.
29 secrets:
30   externalSecrets:
31     enabled: true
32     provider: aws
33     createStore: true
34     remoteKey: <secrets_manager_secret_name>
35     aws:
36       region: <region>
37 
38 clickhouse:
39   internal: true
40   password: "<choose-a-password>"
41   storageClass: gp3
42   keeper:
43     storageClass: gp3
44 
45 # Recommended: managed Redis (ElastiCache) from Terraform.
46 redis:
47   internal: false
48   externalUrl: <redis_url>
49 
50 # Required for code-based and transformer metrics (Lambda sandbox from Terraform).
51 codeExecutor:
52   provider: AWS_LAMBDA
53   aws:
54     lambdaFunctionName: <code_executor_function_name>
55     lambdaRegion: <region>
56 
57 ingress:
58   enabled: true
59   className: alb
60   annotations:
61     alb.ingress.kubernetes.io/scheme: internet-facing
62     alb.ingress.kubernetes.io/target-type: ip
63     alb.ingress.kubernetes.io/group.name: confident
64     alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
65     alb.ingress.kubernetes.io/certificate-arn: <your-acm-cert-arn>
66   hosts:
67     evals: evals.yourdomain.com
68     otel: otel.yourdomain.com

Install the chart

The chart is published to GHCR as an OCI artifact:

$ helm install confident-ai \
>   oci://ghcr.io/confident-ai/charts/confident-ai \
>   --version 0.1.0 \
>   -n confident-ai \
>   -f values.aws.yaml
$ 
$ kubectl get pods -n confident-ai -w

The ClickHouse operator starts first, then a migrations job runs, then the app pods come up. This takes a few minutes.

Verify

$ kubectl get externalsecret -n confident-ai   # STATUS should be SecretSynced
$ kubectl get ingress -n confident-ai          # ADDRESS is the ALB hostname

Create app., api., evals., and otel. DNS records (CNAME, or a Route 53 alias) pointing at the ALB hostname, then open https://app.yourdomain.com and sign in.

Simpler option: in-cluster Redis and a Kubernetes Secret

If you would rather not run a cloud secret store or managed Redis, the chart can hold secrets in a Kubernetes Secret and run Redis in the cluster. This is less production-hardened (secrets live in the cluster, Redis has no managed backups), but it removes the ESO and ElastiCache steps. Skip steps 2 and 3 above, and replace the secrets and redis blocks in the values file with:

1 secrets:
2   data:
3     DATABASE_URL: "<database_url>"
4     BETTER_AUTH_SECRET: "<openssl rand -hex 32>"
5     OPENAI_API_KEY: "sk-..."
6     CONFIDENT_LICENSE_KEY: "<your license key>"
7 
8 redis:
9   internal: true
10   storageClass: gp3

Back up ClickHouse (recommended)

For production, enable the nightly ClickHouse backup to S3. Provision the backup bucket (confident_clickhouse_backup_bucket_enabled = true); the backup pod writes with Pod Identity, so set serviceAccountName to a service account whose role can write to that bucket. Full detail is on the Disaster Recovery page. Add this under your existing clickhouse: block and helm upgrade:

1 clickhouse:
2   backup:
3     enabled: true
4     provider: s3
5     schedule: "0 2 * * *"          # nightly at 02:00 UTC
6     serviceAccountName: confident
7     s3:
8       bucket: <clickhouse_backup_bucket>
9       region: <region>

Troubleshooting

Symptom	Cause and fix
`externalsecret` never reaches `SecretSynced`	ESO must run as `external-secrets-sa` in the `confident-ai` namespace so it inherits the Pod Identity read role. Recheck the `helm install` flags and that `confident_create_secrets_manager = true`.
Pods stuck `Pending`, PVCs unbound	EKS ships no default StorageClass. Create the gp3 default from the Infrastructure page; the values file sets `storageClass: gp3` on ClickHouse.
`ImagePullBackOff` on the app images	The images live in a separate ECR account, so node IAM cannot pull them. Confirm `imagePullSecretRefresh` is enabled and the AWS keys from Confident AI are correct.
Ingress never gets an `ADDRESS`	The AWS Load Balancer Controller is not installed, or the subnets are not tagged. Public subnets need `kubernetes.io/role/elb=1`; the network step tags them for you.
Frontend returns 500 with `ENOTFOUND confident-backend`	The frontend resolves backend services by their chart-prefixed names. Keep `fullnameOverride: confident` (the chart default); do not change it.
S3 access denied from the app	Pod Identity is not associated. Keep `serviceAccount.create: true` with the default name so it matches the association Terraform created.
ClickHouse Keeper logs `Not authenticated`	Stale PersistentVolumeClaims from a previous failed install. `helm uninstall`, `kubectl delete pvc -n confident-ai --all`, then reinstall.
`broken pipe` errors during `kubectl port-forward`	Harmless connection resets. Re-run the port-forward.

Updating and tearing down

Change the app: edit values.aws.yaml, then helm upgrade confident-ai oci://ghcr.io/confident-ai/charts/confident-ai --version 0.1.0 -n confident-ai -f values.aws.yaml.
Rotate a secret: run aws secretsmanager put-secret-value again. ESO re-syncs it, then restart the pods to pick up the change (they hold secrets as env vars until they restart): kubectl rollout restart deployment -n confident-ai.
Change infrastructure: edit the Terraform config and terraform apply.
Remove everything: helm uninstall confident-ai -n confident-ai, then terraform destroy, then delete the network from the Infrastructure page if Terraform did not own it.