Deploy on Azure with Helm

With the infrastructure in place, install the confident-ai Helm chart using the Terraform outputs. The chart pulls its images from Confident AI's registry and installs the app plus in-cluster ClickHouse. On Azure, set config.isAzureEnvironment: true and reach Blob storage through the connection string.

The recommended setup keeps app secrets in Azure Key Vault (synced by the External Secrets Operator) and runs Redis on Azure Managed Redis. Both are provisioned by the Terraform module. In-cluster Redis and a Kubernetes Secret are supported as a simpler alternative, see Simpler option.

1. Create the namespace

   kubectl create namespace confident-ai

2. Put the secrets in Key Vault

   Store each value as its own Key Vault secret. Key Vault names cannot contain _, so use -; the chart rewrites - back to _ on the way in (for example DATABASE-URL becomes DATABASE_URL). Include the storage connection string here too, since Key Vault owns the whole secret set:

   KV_URI=$(terraform output -raw key_vault_uri)
   KV_NAME=$(echo "$KV_URI" | sed -E 's#https://([^.]+).*#\1#')
   
   az keyvault secret set --vault-name "$KV_NAME" --name DATABASE-URL                     --value "$(terraform output -raw database_url)"
   az keyvault secret set --vault-name "$KV_NAME" --name AZURE-STORAGE-CONNECTION-STRING  --value "$(terraform output -raw storage_connection_string)"
   az keyvault secret set --vault-name "$KV_NAME" --name BETTER-AUTH-SECRET               --value "$(openssl rand -hex 32)"
   az keyvault secret set --vault-name "$KV_NAME" --name OPENAI-API-KEY                   --value "sk-..."
   az keyvault secret set --vault-name "$KV_NAME" --name CONFIDENT-LICENSE-KEY            --value "..."

3. Link ESO's identity with a federated credential

   Terraform creates the managed identity for ESO but not the federated credential that lets it act as the external-secrets-sa Kubernetes account. AKS already has Workload Identity enabled, so you only add the link.

   Resolve the cluster's OIDC issuer and the ESO identity Terraform created:

   RESOURCE_GROUP=confident-prod-rg
   CLUSTER_NAME=$(terraform output -raw cluster_name)
   OIDC_ISSUER_URL=$(az aks show -g $RESOURCE_GROUP -n $CLUSTER_NAME --query oidcIssuerProfile.issuerUrl -o tsv)
   ESO_IDENTITY_NAME=$(az identity list -g $RESOURCE_GROUP --query "[?ends_with(name,'eso-identity')].name | [0]" -o tsv)
   ESO_CLIENT_ID=$(az identity show -g $RESOURCE_GROUP -n "$ESO_IDENTITY_NAME" --query clientId -o tsv)

   Create the federated credential that binds the identity to the service account:

   az identity federated-credential create --name eso-confident \
     --identity-name "$ESO_IDENTITY_NAME" --resource-group "$RESOURCE_GROUP" \
     --issuer "$OIDC_ISSUER_URL" \
     --subject system:serviceaccount:confident-ai:external-secrets-sa \
     --audience api://AzureADTokenExchange

4. Install the External Secrets Operator

   helm repo add external-secrets https://charts.external-secrets.io && helm repo update
   helm install external-secrets external-secrets/external-secrets \
     -n external-secrets --create-namespace --set installCRDs=true
   
   kubectl create serviceaccount external-secrets-sa -n confident-ai
   kubectl annotate serviceaccount external-secrets-sa -n confident-ai \
     azure.workload.identity/client-id=$ESO_CLIENT_ID

5. Set up ingress (for HTTPS)

   Turn on the AKS application routing add-on, which runs a managed NGINX ingress controller:

   az aks approuting enable --resource-group confident-prod-rg --name <cluster_name>

   The values file below uses className: webapprouting.kubernetes.io and exposes all four subdomains (app., api., evals., otel.). Add TLS with a Kubernetes secret or cert-manager that covers all four.

6. Write the values file

   Save this as values.azure.yaml. Fill the bracketed values from your Terraform outputs and the credentials Confident AI gave you.

   # The chart mints and refreshes the ECR pull secret from these credentials.
   imagePullSecrets:
     - name: ecr-registry-credentials
   imagePullSecretRefresh:
     enabled: true
     region: us-east-1
     awsAccessKeyId: "<from Confident AI>"
     awsSecretAccessKey: "<from Confident AI>"
   
   config:
     cloudProvider: AZURE
     isAzureEnvironment: true
     frontendUrl: https://app.yourdomain.com
     backendUrl: https://api.yourdomain.com
     subdomain: yourdomain.com
   
   serviceAccount:
     create: true
   
   storage:
     testCasesBucket: <test_cases_container>
     payloadsBucket: <payloads_container>
     azure:
       storageAccountName: <storage_account_name>
   
   # Recommended: all app secrets (including the storage connection string) come
   # from Azure Key Vault via ESO.
   secrets:
     externalSecrets:
       enabled: true
       provider: azurekv
       createStore: true
       serviceAccountRef:
         name: external-secrets-sa
       azure:
         vaultUrl: <key_vault_uri>
         tenantId: <your-tenant-id>
   
   clickhouse:
     internal: true
     password: "<choose-a-password>"
   
   # Recommended: managed Redis (Azure Managed Redis) from Terraform.
   redis:
     internal: false
     externalUrl: <redis_url>
   
   # Required for code-based and transformer metrics (Azure Function sandbox from Terraform).
   codeExecutor:
     provider: AZURE_FUNCTIONS
     azure:
       functionUrl: <code_executor_function_url>/api/execute
   
   ingress:
     enabled: true
     className: webapprouting.kubernetes.io
     hosts:
       evals: evals.yourdomain.com
       otel: otel.yourdomain.com

7. Install the chart

   The chart is published to GHCR as an OCI artifact:

   helm install confident-ai \
     oci://ghcr.io/confident-ai/charts/confident-ai \
     --version 0.2.0 \
     -n confident-ai \
     -f values.azure.yaml
   
   kubectl get pods -n confident-ai -w

   The ClickHouse operator starts first, then a migrations job runs, then the app pods come up. This takes a few minutes.

8. Verify

   kubectl get externalsecret -n confident-ai   # STATUS should be SecretSynced
   kubectl get svc -n app-routing-system        # note the EXTERNAL-IP

   Create app., api., evals., and otel. DNS records for that IP, then open https://app.yourdomain.com and sign in.

Simpler option: in-cluster Redis and a Kubernetes Secret

If you would rather not run Key Vault or managed Redis, the chart can hold secrets in a Kubernetes Secret and run Redis in the cluster. This is less production-hardened (secrets live in the cluster, Redis has no managed backups), but it removes the Key Vault, federated-credential, and ESO steps. Skip steps 2 through 4 above, and replace the secrets and redis blocks in the values file with:

secrets:
  data:
    DATABASE_URL: "<database_url>"
    AZURE_STORAGE_CONNECTION_STRING: "<storage_connection_string>"
    BETTER_AUTH_SECRET: "<openssl rand -hex 32>"
    OPENAI_API_KEY: "sk-..."
    CONFIDENT_LICENSE_KEY: "<your license key>"

redis:
  internal: true

Code executor key

The Azure Function sandbox is protected by a function key, which you read from the Azure portal after Terraform creates the Function. Add it as a secret so the app can call the sandbox:

• Key Vault (recommended): az keyvault secret set --vault-name "$KV_NAME" --name CODE-EXECUTOR-AZURE-FUNCTION-KEY --value "<function key>"
• Simpler option: add CODE_EXECUTOR_AZURE_FUNCTION_KEY: "<function key>" to secrets.data.

Back up ClickHouse (recommended)

For production, enable the nightly ClickHouse backup to Blob storage, authenticated with the storage connection string. Provision the backup container and see the Disaster Recovery page for the credential detail. Add this under your existing clickhouse: block and helm upgrade:

clickhouse:
  backup:
    enabled: true
    provider: azure
    schedule: "0 2 * * *"          # nightly at 02:00 UTC
    azure:
      container: <clickhouse_backup_container>

Troubleshooting

Updating and tearing down

• Change the app: edit values.azure.yaml, then helm upgrade confident-ai oci://ghcr.io/confident-ai/charts/confident-ai --version 0.2.0 -n confident-ai -f values.azure.yaml.
• Rotate a secret: write a new version to Key Vault. ESO re-syncs it, then restart the pods to pick up the change (they hold secrets as env vars until they restart): kubectl rollout restart deployment -n confident-ai.
• Change infrastructure: edit the Terraform config and terraform apply.
• Remove everything: helm uninstall confident-ai -n confident-ai, helm uninstall external-secrets -n external-secrets, then terraform destroy, then az group delete --name confident-prod-rg --yes if you want the Step 1 resources gone.

Last updated on July 31, 2026