Verification

Overview

You’ve provisioned infrastructure and deployed the application. This final step verifies everything works correctly. You will:

• Confirm all infrastructure components (GKE, Cloud SQL, GCS, secrets) are healthy
• Configure DNS to point to your load balancer
• Test application access via the frontend and backend URLs
• Run functional tests (user login, project creation, SDK connectivity)
• Check health endpoints for each service
• Complete a production readiness checklist

After this step, your deployment is verified and ready for users.

Infrastructure verification

Before testing the application, verify all infrastructure components are healthy.

GKE cluster health

$
kubectl get nodes

All nodes should show Ready:

NAME                                                STATUS   ROLES    AGE    VERSION
gke-confidentai-stage-system-12345678-aaaa          Ready    <none>   1h     v1.31.x
gke-confidentai-stage-system-12345678-bbbb          Ready    <none>   1h     v1.31.x
gke-confidentai-stage-workers-12345678-cccc         Ready    <none>   1h     v1.31.x
gke-confidentai-stage-workers-12345678-dddd         Ready    <none>   1h     v1.31.x

$
kubectl describe node <node-name>

Cloud SQL connectivity

Verify the database is accessible from the cluster by checking backend logs:

$
kubectl logs deployment/confident-backend -n confident-ai | grep -i database

You should see successful connection messages, not connection refused errors.

Check Cloud SQL status in GCP Console:

1. Go to GCP Console → SQL → Instances
2. Find your instance
3. State should be “Runnable”

Secret Manager secrets

Verify secrets exist and External Secrets can read them:

$
# Check External Secrets synced to Kubernetes
$
kubectl get secret confident-externalsecret -n confident-ai
$
$
# Check Secret Manager via CLI
$
gcloud secrets list --project=$(terraform output -raw gcp_project_id) --format="value(name)"

Cloud Storage

Verify the buckets exist:

$
gcloud storage ls --project=$(terraform output -raw gcp_project_id)

You should see three buckets (e.g., confidentai-stage-testcases, confidentai-stage-payloads, and confidentai-stage-chbackups).

DNS configuration

The NGINX Ingress controller has a GCP Network Load Balancer IP. DNS records must point to it.

Get the Load Balancer IP

$
kubectl get svc -n ingress-nginx ingress-nginx-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

Example output:

34.85.123.45

Create DNS records

Add A records for each hostname you configured in the ingress:

Verify DNS propagation

After adding records, verify they resolve correctly:

$
nslookup app.yourdomain.com
$
nslookup api.yourdomain.com

You should see the Load Balancer IP in the response. If you see “NXDOMAIN” or your old values, wait for DNS propagation (typically 5-30 minutes, up to 48 hours for some providers).

Application verification

Frontend access

Open your frontend URL in a browser:

https://app.yourdomain.com

What you should see:

1. HTTPS (padlock icon) — certificate is working
2. Confident AI login page — application is serving
3. Google OAuth button (if configured) — SSO is set up

Backend health check

The backend exposes a health endpoint:

$
curl -s https://api.yourdomain.com/health

Expected response:

1
{ "status": "ok", "version": "1.2.3" }

Evals health check

$
curl -s https://deepeval.yourdomain.com/health

Expected: {"status":"healthy"}

OTEL collector health

$
curl -s https://otel.yourdomain.com/health

Expected: {"status":"ok"}

Functional testing

Test 1: User login

1. Navigate to https://app.yourdomain.com
2. Click “Sign in with Google” (or your configured auth provider)
3. Complete authentication
4. Expected: Redirected to dashboard, user session created

Test 2: Create a project

1. From the dashboard, click “New Project”
2. Enter a project name
3. Click Create
4. Expected: Project created successfully, appears in list

This verifies database connectivity and basic write operations.

Test 3: API key generation

1. Go to Project Settings → API Keys
2. Click “Generate API Key”
3. Copy the generated key
4. Expected: API key displayed (save it—it won’t be shown again)

Test 4: SDK connectivity

From your local machine (or any machine that can reach the backend):

1
import deepeval
2
3
deepeval.login(
4
    api_key="<your-generated-api-key>",
5
    confident_api_endpoint="https://api.yourdomain.com"
6
)
7
8
print(deepeval.check_connection())

Expected: True or success message

Test 5: Run a simple evaluation

1
from deepeval import evaluate
2
from deepeval.test_case import LLMTestCase
3
from deepeval.metrics import AnswerRelevancyMetric
4
5
test_case = LLMTestCase(
6
    input="What is the capital of France?",
7
    actual_output="Paris is the capital of France.",
8
)
9
10
metric = AnswerRelevancyMetric()
11
evaluate([test_case], [metric])

Expected: Evaluation runs and results appear in the dashboard.

Service health checks

Run comprehensive health checks on all services:

$
# All pods should be Running
$
kubectl get pods -n confident-ai
$
$
# No recent crashes
$
kubectl get events -n confident-ai --field-selector type=Warning
$
$
# Check each deployment is ready
$
kubectl rollout status deployment/confident-backend -n confident-ai
$
kubectl rollout status deployment/confident-frontend -n confident-ai
$
kubectl rollout status deployment/confident-evals -n confident-ai
$
kubectl rollout status deployment/confident-otel -n confident-ai

View logs for troubleshooting

$
# Backend logs
$
kubectl logs -f deployment/confident-backend -n confident-ai
$
$
# Frontend logs
$
kubectl logs -f deployment/confident-frontend -n confident-ai
$
$
# Evals logs
$
kubectl logs -f deployment/confident-evals -n confident-ai

Production readiness checklist

Before announcing the deployment is ready for users, verify:

Security

• HTTPS working on all endpoints (padlock icon in browser)
• TLS certificate valid and not expiring soon
• Firewall rules restrict access appropriately
• Cloud SQL not publicly accessible (Private Service Access only)
• GCS buckets have uniform access and no public objects
• Secret Manager has access controls configured
• terraform.tfvars not committed to version control

High availability

• At least 2 worker nodes running
• At least 2 replicas for backend service (scale if not)
• Cloud SQL has regional HA enabled (if required)
• Worker pool autoscaling configured for traffic spikes

Monitoring (recommended)

• Cloud Monitoring configured for GKE
• Cloud SQL metrics and alerts enabled
• GKE Container logging enabled
• Alerts configured for node health, pod restarts, errors

Operations

• Team members have cluster access (see Cluster Access page)
• Runbook documented for common issues
• Backup strategy confirmed for Cloud SQL
• Upgrade path understood for future releases

What to do if verification fails

Don’t panic. Most issues have straightforward fixes:

1. Identify the failing component: Use the checks above to isolate which part isn’t working
2. Check logs: kubectl logs for pod issues, GCP Console for service-level issues
3. Review configuration: Typos in URLs, missing certificates, wrong secret names
4. Check network: Firewall rules, DNS propagation, VPN connectivity
5. Contact support: If stuck, reach out to your Confident AI representative with:
   • What step failed
   • Exact error messages
   • Results of relevant kubectl get commands

Summary

You’ve completed the Confident AI deployment on GCP:

1. Prerequisites — Installed tools and gathered credentials
2. Configuration — Set up Terraform variables
3. Provisioning — Created GCP infrastructure
4. TLS Certificates — Configured cert-manager and ClusterIssuer
5. Cluster Access — Configured kubectl access
6. Kubernetes Deployment — Deployed application services
7. Verification — Tested everything works

Your deployment is now ready for users. Welcome to Confident AI!

Need help? Contact your Confident AI representative or email support@confident-ai.com with details about your deployment and any issues encountered.