Deploy with Helm | Confident AI Docs

With the infrastructure in place, install the confident-ai Helm chart using the Terraform outputs, then expose it over HTTPS with a Google-managed certificate. The chart pulls its images from Confident AI’s registry and installs the app plus in-cluster ClickHouse.

The recommended setup keeps app secrets in Google Secret Manager (synced by the External Secrets Operator) and runs Redis on Memorystore. Both are provisioned by the Terraform module. In-cluster Redis and a Kubernetes Secret are supported as a simpler alternative, see Simpler option.

Two things come from Confident AI with your Enterprise license:

Image pull credentials: an AWS access key ID and secret that let the cluster pull the first-party images. The chart uses them to mint and refresh the pull secret automatically.
License key (CONFIDENT_LICENSE_KEY): the signed key that enables your plan’s features.

About OPENAI_API_KEY: it backs the built-in Confident AI evaluation provider, which runs on OpenAI. It is optional. Omit it and connect your own model provider or an LLM gateway per project from AI Connections in the app instead.

Set your domain once. You will expose four subdomains on it: app. and api. for the dashboard and API, and evals. and otel. for the evaluation and trace-ingestion endpoints:

$ export DOMAIN=yourdomain.com

Reserve a static IP and point DNS

$ gcloud compute addresses create confident-ip --global
$ gcloud compute addresses describe confident-ip --global --format='value(address)'

At your DNS provider, create A records for app.$DOMAIN, api.$DOMAIN, evals.$DOMAIN, and otel.$DOMAIN, all pointing at that IP. The managed certificate only goes Active once these resolve, so set them all now.

Create the namespace and managed certificate

$ kubectl create namespace confident-ai
$ 
$ kubectl apply -f - <<EOF
$ apiVersion: networking.gke.io/v1
$ kind: ManagedCertificate
$ metadata:
$   name: confident-cert
$   namespace: confident-ai
$ spec:
$   domains:
$     - app.$DOMAIN
$     - api.$DOMAIN
$     - evals.$DOMAIN
$     - otel.$DOMAIN
$ EOF

Put the secrets in Secret Manager

DATABASE_URL comes straight from Terraform. Each key in this JSON object becomes an app secret:

$ printf '{"DATABASE_URL":"%s","BETTER_AUTH_SECRET":"%s","OPENAI_API_KEY":"sk-...","CONFIDENT_LICENSE_KEY":"..."}' \
>   "$(terraform output -raw database_url)" "$(openssl rand -hex 32)" \
>   | gcloud secrets versions add "$(terraform output -raw secret_manager_secret_id)" --data-file=-

This uses the secret Terraform created with confident_create_secret_manager = true. If you did not enable it, either turn it on and re-apply, or use the simpler option.

Install the External Secrets Operator

Terraform already Workload-Identity-bound the ESO service account to confident-ai/external-secrets-sa:

$ helm repo add external-secrets https://charts.external-secrets.io && helm repo update
$ helm install external-secrets external-secrets/external-secrets \
>   -n external-secrets --create-namespace --set installCRDs=true
$ 
$ kubectl create serviceaccount external-secrets-sa -n confident-ai
$ kubectl annotate serviceaccount external-secrets-sa -n confident-ai \
>   iam.gke.io/gcp-service-account=$(terraform output -raw eso_service_account_email)

Write the values file

Save this as values.gcp.yaml. Fill the bracketed values from your Terraform outputs and the credentials Confident AI gave you. The image.tag is pinned to the chart’s current app version.

1 image:
2   tag: "v2.0.18"   # use the version Confident AI gives you
3 
4 # The chart mints and refreshes the ECR pull secret from these credentials.
5 imagePullSecrets:
6   - name: ecr-registry-credentials
7 imagePullSecretRefresh:
8   enabled: true
9   region: us-east-1
10   awsAccessKeyId: "<from Confident AI>"
11   awsSecretAccessKey: "<from Confident AI>"
12 
13 config:
14   cloudProvider: GCP
15   frontendUrl: https://app.yourdomain.com
16   backendUrl: https://api.yourdomain.com
17   subdomain: yourdomain.com
18 
19 serviceAccount:
20   create: true
21   annotations:
22     iam.gke.io/gcp-service-account: <app_service_account_email>
23 
24 storage:
25   testCasesBucket: <test_cases_bucket>
26   payloadsBucket: <payloads_bucket>
27   gcp:
28     projectId: <your-gcp-project>
29     region: us-central1
30 
31 # Recommended: app secrets come from Google Secret Manager via ESO.
32 secrets:
33   externalSecrets:
34     enabled: true
35     provider: gcpsm
36     createStore: true
37     remoteKey: <secret_manager_secret_id>
38     serviceAccountRef:
39       name: external-secrets-sa
40     gcp:
41       projectId: <your-gcp-project>
42       clusterLocation: us-central1
43       clusterName: <cluster_name>
44 
45 clickhouse:
46   internal: true
47   password: "<choose-a-password>"
48 
49 # Recommended: managed Redis (Memorystore) from Terraform.
50 redis:
51   internal: false
52   externalUrl: "<redis_url>"
53 
54 # Required for code-based and transformer metrics (Cloud Run sandbox from Terraform).
55 codeExecutor:
56   provider: GCP_CLOUD_FUNCTIONS
57   gcp:
58     functionUrl: <code_executor_function_url>
59 
60 ingress:
61   enabled: true
62   # GKE's L7 controller claims Ingresses via the ingress.class annotation, not
63   # ingressClassName (this cluster has no "gce" IngressClass). Leave className
64   # empty, or the Ingress is ignored and never gets an ADDRESS.
65   className: ""
66   annotations:
67     kubernetes.io/ingress.class: gce
68     kubernetes.io/ingress.global-static-ip-name: confident-ip
69     networking.gke.io/managed-certificates: confident-cert
70   hosts:
71     evals: evals.yourdomain.com
72     otel: otel.yourdomain.com

Install the chart

The chart is published to GHCR as an OCI artifact:

$ helm install confident-ai \
>   oci://ghcr.io/confident-ai/charts/confident-ai \
>   --version 0.1.0 \
>   -n confident-ai \
>   -f values.gcp.yaml
$ 
$ kubectl get pods -n confident-ai -w

The ClickHouse operator starts first, then a migrations job runs, then the app pods come up. This takes a few minutes.

Wait for the certificate, then verify

The Google-managed certificate takes 15 to 60 minutes to provision after DNS resolves:

$ kubectl get managedcertificate -n confident-ai   # STATUS moves Provisioning -> Active
$ kubectl get ingress -n confident-ai              # ADDRESS should match your static IP
$ kubectl get externalsecret -n confident-ai       # STATUS should be SecretSynced

Once the certificate is Active, open https://app.$DOMAIN and sign in. The cookie is set on .$DOMAIN, so all subdomains share it.

Simpler option: in-cluster Redis and a Kubernetes Secret

If you would rather not run a cloud secret store or managed Redis, the chart can hold secrets in a Kubernetes Secret and run Redis in the cluster. This is less production-hardened (secrets live in the cluster, Redis has no managed backups), but it removes the ESO and Memorystore steps. Skip steps 3 and 4 above, and replace the secrets and redis blocks in the values file with:

1 secrets:
2   data:
3     DATABASE_URL: "<database_url>"
4     BETTER_AUTH_SECRET: "<openssl rand -hex 32>"
5     OPENAI_API_KEY: "sk-..."
6     CONFIDENT_LICENSE_KEY: "<your license key>"
7 
8 redis:
9   internal: true

Back up ClickHouse (recommended)

For production, enable the nightly ClickHouse backup to GCS. It needs the backup bucket and a Secret with GCS HMAC keys; that one-time setup is on the Disaster Recovery page. Once they exist, add this under your existing clickhouse: block and helm upgrade:

1 clickhouse:
2   backup:
3     enabled: true
4     provider: gcs
5     schedule: "0 2 * * *"          # nightly at 02:00 UTC
6     gcs:
7       bucket: <clickhouse_backup_bucket>
8     credentialsSecret: clickhouse-backup-creds

Troubleshooting

Symptom	Cause and fix
`externalsecret` never reaches `SecretSynced`	ESO cannot read Secret Manager. Recheck the `external-secrets-sa` annotation and that `confident_create_secret_manager = true`. `kubectl describe externalsecret -n confident-ai`.
App pods briefly `CreateContainerConfigError`	ESO has not synced the secret yet. It self-heals once `externalsecret` shows `SecretSynced`.
Ingress never gets an `ADDRESS`	GKE’s L7 controller claims Ingresses through the `kubernetes.io/ingress.class: gce` annotation, not `ingressClassName`. Keep `className: ""` as shown.
Certificate stuck `Provisioning`	DNS is not resolving to the static IP yet, or the load balancer is not serving. Confirm the `A` records and that `kubectl get ingress` shows the static IP.
Backends `UNHEALTHY`, browser shows 502	GKE health-checks each Service. On a fresh install the chart’s readiness probes set the health-check paths automatically, give it a few minutes. If you added probes to an already-built load balancer, GKE does not re-derive the path, update it with `gcloud compute health-checks update http <name> --request-path=/health`.
ClickHouse pod crashes with `Listen [::]... Address family not supported`	GKE nodes are IPv4-only. The chart sets ClickHouse to listen on `0.0.0.0`; only relevant if you overrode `clickhouse.extraConfig`.
ClickHouse Keeper logs `Not authenticated`	Stale PersistentVolumeClaims from a previous failed install. `helm uninstall`, `kubectl delete pvc -n confident-ai --all`, then reinstall.
Frontend returns 500 with `ENOTFOUND confident-backend`	The frontend resolves backend services by their chart-prefixed names. Keep `fullnameOverride: confident` (the chart default); do not change it.
Code metric fails with 403	The app service account lacks `run.invoker` on the Cloud Run sandbox. Confirm the `iam.gke.io/gcp-service-account` annotation and that Terraform granted the invoker binding.
Opening dataset goldens returns 500 with `iam.serviceAccounts.signBlob` denied	The app signs GCS URLs, which under Workload Identity calls the IAM Credentials API. The app service account needs `roles/iam.serviceAccountTokenCreator` on itself and the `iamcredentials.googleapis.com` API enabled. The Terraform module grants this; if you provisioned before that change, add it manually with `gcloud iam service-accounts add-iam-policy-binding`.

Updating and tearing down

Change the app: edit values.gcp.yaml, then helm upgrade confident-ai oci://ghcr.io/confident-ai/charts/confident-ai --version 0.1.0 -n confident-ai -f values.gcp.yaml.
Rotate a secret: write a new version to Secret Manager. ESO re-syncs it, then restart the pods to pick up the change (they hold secrets as env vars until they restart): kubectl rollout restart deployment -n confident-ai.
Change infrastructure: edit the Terraform config and terraform apply.
Remove everything: helm uninstall confident-ai -n confident-ai, helm uninstall external-secrets -n external-secrets, then terraform destroy, then delete the static IP and (if Terraform did not own it) the network from the Infrastructure page.