Kubernetes Deployment

Overview

With infrastructure and cluster access ready, you now deploy the Confident AI application services. This step covers:

• Preparing Kubernetes manifests provided by Confident AI
• Updating container image tags to the versions provided by Confident AI
• Configuring the ingress with your domain names and TLS settings
• Setting up External Secrets to sync credentials from Azure Key Vault
• Deploying services in the correct order: base config → Redis → application services
• Monitoring deployment status and troubleshooting common issues

After completion, all Confident AI services will be running in your AKS cluster.

How Kubernetes deployments work

Kubernetes uses YAML manifests to describe what you want to run. A Deployment tells Kubernetes:

• What container image to use
• How many replicas (copies) to run
• What environment variables and secrets to inject
• Resource limits (CPU, memory)

When you kubectl apply a manifest, Kubernetes:

1. Reads your desired state
2. Compares it to current state
3. Creates, updates, or deletes resources to match
4. Continuously ensures the desired state is maintained

Manifest structure

The Kubernetes manifests are organized by service:

base/                    # Shared configuration
├── namespace.yaml       # The confident-ai namespace (already created)
├── network/
│   └── ingress.yaml     # Load balancer and routing rules
└── common/
    └── external-secrets/ # Secrets sync configuration
confident-backend/       # API service
├── deployment.yaml
└── service.yaml
confident-frontend/      # Web dashboard
confident-evals/         # Evaluation service
confident-otel/          # OpenTelemetry collector
redis/                   # Cache service

Prepare manifests

Update image tags

Container images are stored in Confident AI’s AWS ECR. Your Confident AI representative will provide the specific version tags to use.

Open each deployment file and update the image field:

Example change in confident-backend/deployment.yaml:

1
containers:
2
  - name: confident-backend
3
    image: 128045499490.dkr.ecr.us-east-1.amazonaws.com/confidentai/confident-backend:v1.2.3

Configure ingress

The ingress defines how external traffic reaches your services. For Azure, this uses NGINX Ingress (not AWS ALB). Edit base/network/ingress.yaml:

1
apiVersion: networking.k8s.io/v1
2
kind: Ingress
3
metadata:
4
  name: confident-ingress
5
  namespace: confident-ai
6
  annotations:
7
    cert-manager.io/cluster-issuer: letsencrypt-prod
8
spec:
9
  ingressClassName: nginx

1
spec:
2
  tls:
3
    - hosts:
4
        - app.yourdomain.com
5
        - api.yourdomain.com
6
        - deepeval.yourdomain.com
7
        - otel.yourdomain.com
8
      secretName: confident-tls
9
  rules:
10
    - host: app.yourdomain.com
11
      http:
12
        paths:
13
          - path: /
14
            pathType: Prefix
15
            backend:
16
              service:
17
                name: confident-frontend
18
                port:
19
                  number: 3000
20
21
    - host: api.yourdomain.com
22
      # ...
23
24
    - host: deepeval.yourdomain.com
25
      # ...
26
27
    - host: otel.yourdomain.com
28
      # ...

1
metadata:
2
  annotations:
3
    cert-manager.io/cluster-issuer: letsencrypt-prod  # or your ClusterIssuer name
4
spec:
5
  tls:
6
    - hosts:
7
        - app.yourdomain.com
8
        - api.yourdomain.com
9
      secretName: confident-tls

Configure External Secrets

External Secrets syncs credentials from Azure Key Vault into Kubernetes. The configuration needs to match your deployment.

Update the secret store

Edit base/common/external-secrets/secret-store.yaml:

The vaultUrl must match your Key Vault URI (from terraform output key_vault_uri):

1
spec:
2
  provider:
3
    azurekv:
4
      vaultUrl: "https://confidentaisconfidentkv.vault.azure.net/"
5
      authType: WorkloadIdentity
6
      serviceAccountRef:
7
        name: external-secrets-sa

Update the secret references

Edit base/common/external-secrets/external-secrets.yaml:

The key field in each secret reference should match the Key Vault secret names created by Terraform. The secrets use hyphenated names (e.g., DATABASE-URL, BETTER-AUTH-SECRET):

1
data:
2
  - secretKey: DATABASE_URL
3
    remoteRef:
4
      key: DATABASE-URL
5
  - secretKey: REDIS_URL
6
    remoteRef:
7
      key: REDIS-URL

Deploy services

Deploy in order to ensure dependencies are available when needed.

$
kubectl apply -f base/network/
$
kubectl apply -f base/common/external-secrets/

$
kubectl get externalsecret -n confident-ai -w

NAME                       STORE                          REFRESH   STATUS
confident-externalsecret   confident-clustersecretstore   1h        SecretSynced

$
kubectl get secret confident-externalsecret -n confident-ai

$
kubectl apply -f redis/

$
kubectl get pods -n confident-ai -l app=redis

$
kubectl apply -f confident-backend/
$
kubectl apply -f confident-frontend/
$
kubectl apply -f confident-evals/
$
kubectl apply -f confident-otel/

Monitor deployment

Watch pod status

$
kubectl get pods -n confident-ai -w

All pods should eventually reach Running status with all containers ready:

NAME                                  READY   STATUS    RESTARTS   AGE
confident-backend-xxx-yyy             1/1     Running   0          2m
confident-frontend-xxx-yyy            1/1     Running   0          2m
confident-evals-xxx-yyy               1/1     Running   0          2m
confident-otel-xxx-yyy                1/1     Running   0          2m
confident-redis-xxx-yyy               1/1     Running   0          3m

Check events

If pods aren’t starting, check recent events:

$
kubectl get events -n confident-ai --sort-by='.lastTimestamp' | tail -20

Common deployment issues

ImagePullBackOff

confident-backend-xxx   0/1   ImagePullBackOff   0   2m

The cluster can’t pull the container image. Causes:

1. ECR credentials not synced: The CronJob that refreshes ECR tokens hasn’t run yet
2. Wrong image tag: The specified version doesn’t exist
3. Network issues: Nodes can’t reach ECR

Fix: Manually trigger ECR credential sync:

$
kubectl create job --from=cronjob/ecr-credentials-sync manual-ecr-sync -n confident-ai

Wait 30 seconds, then check if pods start pulling images.

CrashLoopBackOff

confident-backend-xxx   0/1   CrashLoopBackOff   5   10m

The container starts but crashes. Check logs to see why:

$
kubectl logs deployment/confident-backend -n confident-ai

Common causes:

External Secrets not syncing

$
kubectl describe externalsecret confident-externalsecret -n confident-ai

Look at the Status section for error messages. Common issues:

Ingress not getting external IP

After applying the ingress, check if it has an address:

$
kubectl get ingress -n confident-ai

The ADDRESS column should show the NGINX Ingress load balancer IP. If it’s empty:

$
kubectl get svc -n ingress-nginx ingress-nginx-controller

The EXTERNAL-IP should show the Azure Load Balancer IP. If it shows <pending>:

• NSG blocking: The NSG may not allow inbound traffic on ports 80/443
• Quota issues: Public IP quota may be exhausted
• Azure Policy: Policies may block public IP or Load Balancer creation

Scaling services

Once everything is running, you can scale based on load:

$
# Scale backend for more API capacity
$
kubectl scale deployment confident-backend --replicas=3 -n confident-ai
$
$
# Scale frontend if dashboard is slow
$
kubectl scale deployment confident-frontend --replicas=2 -n confident-ai
$
$
# Scale evals for more concurrent evaluations
$
kubectl scale deployment confident-evals --replicas=3 -n confident-ai

Next steps

After all services are running and healthy, proceed to Verification to test the deployment end-to-end.