Environment_Infrastructure/roadmap/test-env/08-verify.md

08 — Verification Checklist (Test)

Context

Run these checks after a successful pipeline deployment to the test environment.

1 — Swarm services are up

docker service ls --filter label=project=co.iklim

All services should show REPLICAS 1/1.

docker service ps iklimco_swag
docker service ps iklimco_cert-reloader
docker service ps iklimco_vault
docker service ps iklimco_apisix

No tasks in Failed or Rejected state.

2 — SWAG obtained the cert

docker exec $(docker ps -q -f name=iklimco_swag) \
  certbot certificates

Expected: certificate for *.iklim.co, VALID: XX days.

docker exec $(docker ps -q -f name=iklimco_swag) \
  ls /config/etc/letsencrypt/live/iklim.co/

Expected: fullchain.pem, privkey.pem, cert.pem, chain.pem.

3 — Nginx config is valid

docker exec $(docker ps -q -f name=iklimco_swag) nginx -t

Expected: syntax is ok and test is successful.

4 — Public API endpoint

curl -si https://api-test.iklim.co/health

Expected: HTTP 2xx or APISIX response (not a cert error, not a 502).

TLS cert check:

echo | openssl s_client -connect api-test.iklim.co:443 -servername api-test.iklim.co 2>/dev/null \
  | openssl x509 -noout -subject -dates

Expected: subject=CN=*.iklim.co, dates valid, notAfter > today.

5 — IP-restricted subdomains block non-whitelisted IPs

From a non-whitelisted IP:

curl -si https://grafana-test.iklim.co

Expected: HTTP 403.

From a whitelisted IP (78.187.87.109 or 95.70.151.248):

curl -si https://grafana-test.iklim.co

Expected: HTTP 200 (Grafana login page).

6 — Vault is reachable internally (not externally)

From outside the server:

curl -sk https://vault.iklim.co:8200/v1/sys/health
# or
curl -sk https://<server-public-ip>:8200/v1/sys/health

Expected: connection refused or timeout — Vault must not be reachable externally.

From inside the Swarm (exec into any service container):

docker exec $(docker ps -q -f name=iklimco_apisix | head -1) \
  curl -sk https://vault.iklim.co:8200/v1/sys/health

Expected: JSON response {"sealed":false,...}.

7 — cert-reloader is watching

docker service logs iklimco_cert-reloader --tail 10

Expected: [cert-reloader] started — no errors.

8 — Vault cert path is correct

VAULT_CTR=$(docker ps -q -f name=iklimco_vault)
docker exec "$VAULT_CTR" ls /vault/certs/

Expected: STAR.iklim.co.full.crt and STAR.iklim.co_key.txt.

9 — fail2ban is active (SWAG)

docker exec $(docker ps -q -f name=iklimco_swag) \
  fail2ban-client status

Expected: list of jails including nginx-http-auth, nginx-botsearch, etc.

10 — No services have published unexpected ports

docker service ls --format "{{.Name}}\t{{.Ports}}" \
  --filter label=project=co.iklim

Only iklimco_swag should have published ports (*:80->80, *:443->443). All other services should show empty ports column.