Deploy on AWS with Helm

With the infrastructure in place, install the confident-ai Helm chart using the Terraform outputs. The chart pulls its images from Confident AI's registry and installs the app plus in-cluster ClickHouse. Because the app uses EKS Pod Identity, its ServiceAccount needs no annotation.

The recommended setup keeps app secrets in AWS Secrets Manager (synced by the External Secrets Operator) and runs Redis on ElastiCache. Both are provisioned by the Terraform module. In-cluster Redis and a Kubernetes Secret are supported as a simpler alternative, see Simpler option.

1. Create the namespace

   kubectl create namespace confident-ai

2. Put the secrets in Secrets Manager

   DATABASE_URL comes straight from Terraform. Each key in this JSON object becomes an app secret:

   SECRET_NAME=$(terraform output -raw secrets_manager_secret_name)
   REGION=$(terraform output -raw region)
   
   aws secretsmanager put-secret-value --secret-id "$SECRET_NAME" --region "$REGION" \
     --secret-string "{
       \"DATABASE_URL\":\"$(terraform output -raw database_url)\",
       \"BETTER_AUTH_SECRET\":\"$(openssl rand -hex 32)\",
       \"OPENAI_API_KEY\":\"sk-...\",
       \"CONFIDENT_LICENSE_KEY\":\"...\"
     }"

3. Install the External Secrets Operator

   Install it into the confident-ai namespace as the external-secrets-sa account, the exact account Terraform gave the read role to, so ESO inherits the AWS access through Pod Identity with no keys:

   helm repo add external-secrets https://charts.external-secrets.io && helm repo update
   helm install external-secrets external-secrets/external-secrets \
     -n confident-ai \
     --set installCRDs=true \
     --set serviceAccount.name=external-secrets-sa

4. Set up ALB ingress (for HTTPS)

   To serve the app on a domain over HTTPS, install the AWS Load Balancer Controller (it provisions an ALB from your Ingress) and request an ACM certificate in your region covering all four subdomains, app., api., evals., and otel. of yourdomain.com (a wildcard *.yourdomain.com also works). You will reference the certificate ARN in the values file below.

5. Write the values file

   Save this as values.aws.yaml. Fill the bracketed values from your Terraform outputs and the credentials Confident AI gave you.

   # The chart mints and refreshes the ECR pull secret from these credentials.
   imagePullSecrets:
     - name: ecr-registry-credentials
   imagePullSecretRefresh:
     enabled: true
     region: us-east-1
     awsAccessKeyId: "<from Confident AI>"
     awsSecretAccessKey: "<from Confident AI>"
   
   config:
     cloudProvider: AWS
     frontendUrl: https://app.yourdomain.com
     backendUrl: https://api.yourdomain.com
     subdomain: yourdomain.com
   
   serviceAccount:
     create: true          # Pod Identity is already wired to this SA, no annotation needed
   
   storage:
     testCasesBucket: <test_cases_bucket>
     payloadsBucket: <payloads_bucket>
     aws:
       region: <region>
   
   # Recommended: app secrets come from AWS Secrets Manager via ESO.
   secrets:
     externalSecrets:
       enabled: true
       provider: aws
       createStore: true
       remoteKey: <secrets_manager_secret_name>
       aws:
         region: <region>
   
   clickhouse:
     internal: true
     password: "<choose-a-password>"
     storageClass: gp3
     keeper:
       storageClass: gp3
   
   # Recommended: managed Redis (ElastiCache) from Terraform.
   redis:
     internal: false
     externalUrl: <redis_url>
   
   # Required for code-based and transformer metrics (Lambda sandbox from Terraform).
   codeExecutor:
     provider: AWS_LAMBDA
     aws:
       lambdaFunctionName: <code_executor_function_name>
       lambdaRegion: <region>
   
   ingress:
     enabled: true
     className: alb
     annotations:
       alb.ingress.kubernetes.io/scheme: internet-facing
       alb.ingress.kubernetes.io/target-type: ip
       alb.ingress.kubernetes.io/group.name: confident
       alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
       alb.ingress.kubernetes.io/certificate-arn: <your-acm-cert-arn>
     hosts:
       evals: evals.yourdomain.com
       otel: otel.yourdomain.com

6. Install the chart

   The chart is published to GHCR as an OCI artifact:

   helm install confident-ai \
     oci://ghcr.io/confident-ai/charts/confident-ai \
     --version 0.2.0 \
     -n confident-ai \
     -f values.aws.yaml
   
   kubectl get pods -n confident-ai -w

   The ClickHouse operator starts first, then a migrations job runs, then the app pods come up. This takes a few minutes.

7. Verify

   kubectl get externalsecret -n confident-ai   # STATUS should be SecretSynced
   kubectl get ingress -n confident-ai          # ADDRESS is the ALB hostname

   Create app., api., evals., and otel. DNS records (CNAME, or a Route 53 alias) pointing at the ALB hostname, then open https://app.yourdomain.com and sign in.

Simpler option: in-cluster Redis and a Kubernetes Secret

If you would rather not run a cloud secret store or managed Redis, the chart can hold secrets in a Kubernetes Secret and run Redis in the cluster. This is less production-hardened (secrets live in the cluster, Redis has no managed backups), but it removes the ESO and ElastiCache steps. Skip steps 2 and 3 above, and replace the secrets and redis blocks in the values file with:

secrets:
  data:
    DATABASE_URL: "<database_url>"
    BETTER_AUTH_SECRET: "<openssl rand -hex 32>"
    OPENAI_API_KEY: "sk-..."
    CONFIDENT_LICENSE_KEY: "<your license key>"

redis:
  internal: true
  storageClass: gp3

Back up ClickHouse (recommended)

For production, enable the nightly ClickHouse backup to S3. Provision the backup bucket (confident_clickhouse_backup_bucket_enabled = true); the backup pod writes with Pod Identity, so set serviceAccountName to a service account whose role can write to that bucket. Full detail is on the Disaster Recovery page. Add this under your existing clickhouse: block and helm upgrade:

clickhouse:
  backup:
    enabled: true
    provider: s3
    schedule: "0 2 * * *"          # nightly at 02:00 UTC
    serviceAccountName: confident
    s3:
      bucket: <clickhouse_backup_bucket>
      region: <region>

Troubleshooting

Updating and tearing down

• Change the app: edit values.aws.yaml, then helm upgrade confident-ai oci://ghcr.io/confident-ai/charts/confident-ai --version 0.2.0 -n confident-ai -f values.aws.yaml.
• Rotate a secret: run aws secretsmanager put-secret-value again. ESO re-syncs it, then restart the pods to pick up the change (they hold secrets as env vars until they restart): kubectl rollout restart deployment -n confident-ai.
• Change infrastructure: edit the Terraform config and terraform apply.
• Remove everything: helm uninstall confident-ai -n confident-ai, then terraform destroy, then delete the network from the Infrastructure page if Terraform did not own it.

Last updated on July 31, 2026