Environment_Infrastructure/roadmap/prod-env/09-verify.md

09 — Verification Checklist (Prod)

Context

Run these checks after a successful production pipeline deployment. The current setup source is ../../setup/09-prod-runner-ha-and-swarm.md.

1 — Swarm Cluster Health

docker node ls

Expected: 3 managers (Leader + 2 Reachable) for iklim-app-01/02/03, and 3 workers (Ready) for iklim-db-01/02/03.

docker node inspect iklim-app-01 --format '{{.Spec.Labels}}'
docker node inspect iklim-db-01 --format '{{.Spec.Labels}}'

Expected: app nodes have type=service; DB nodes have role=db and db-index=01/02/03.

2 — Infra, DB, and Vault Services

docker service ls --filter label=project=co.iklim
docker service ps iklimco_vault
docker service ps iklimco_rabbitmq
docker service ps iklimco_apisix

Expected: all current services show their desired replica counts.

Vault is deployed by docker-stack-vault.yml; the main infra and DB services are deployed by docker-stack-infra_db-prod.yml.

3 — DB Node Placement

docker service ps iklimco_patroni-01
docker service ps iklimco_patroni-02
docker service ps iklimco_patroni-03
docker service ps iklimco_mongodb-01
docker service ps iklimco_mongodb-02
docker service ps iklimco_mongodb-03
docker service ps iklimco_etcd-01
docker service ps iklimco_etcd-02
docker service ps iklimco_etcd-03

Expected: tasks run on their matching iklim-db-0X hostnames according to the stack placement constraints.

4 — Service-Node Infrastructure Placement

docker service ps iklimco_redis
docker service ps iklimco_redis-sentinel
docker service ps iklimco_rabbitmq
docker service ps iklimco_swag
docker service ps iklimco_cert-reloader
docker service ps iklimco_cert-distributor

Expected: Redis, Sentinel, RabbitMQ, SWAG, and cert services run on app/service nodes, not DB nodes.

5 — SWAG Certificate Is Valid

docker exec $(docker ps -q -f name=iklimco_swag | head -1) certbot certificates

Expected: certificate for *.iklim.co, valid and issued by Let's Encrypt.

TLS check from outside:

echo | openssl s_client -connect api.iklim.co:443 -servername api.iklim.co 2>/dev/null \
  | openssl x509 -noout -subject -dates

Expected: CN=*.iklim.co and a current notAfter date.

6 — Public API and Restricted Subdomains

curl -si https://api.iklim.co/health

Expected: HTTP 2xx or an APISIX response, with no TLS error.

From a non-whitelisted IP:

curl -si https://grafana.iklim.co
curl -si https://apigw.iklim.co
curl -si https://rabbitmq.iklim.co

Expected: HTTP 403.

From a whitelisted IP:

curl -si https://grafana.iklim.co
curl -si https://apigw.iklim.co
curl -si https://rabbitmq.iklim.co

Expected: HTTP 200 or the expected login/management page.

7 — Vault Is Not Publicly Reachable

From outside:

curl -sk --connect-timeout 5 https://<iklim-app-01-public-ip>:8200/v1/sys/health

Expected: connection refused or timeout.

From inside overlay:

docker exec $(docker ps -q -f name=iklimco_apisix | head -1) \
  curl -sk https://vault.iklim.co:8200/v1/sys/health

Expected: JSON response with "sealed":false.

8 — Certificate Reload Chain

docker service logs iklimco_cert-reloader --tail 10
docker service ps iklimco_cert-distributor
docker exec $(docker ps -q -f name=iklimco_vault | head -1) ls /vault/certs/

Expected: cert-reloader has no errors, cert-distributor is running, and Vault sees STAR.iklim.co.full.crt plus STAR.iklim.co_key.pem.

9 — No Unexpected Published Ports

docker service ls --format "{{.Name}}\t{{.Ports}}" --filter label=project=co.iklim

Expected: only services intentionally published by the stack expose ports. Redis and RabbitMQ must not appear as DB-node host-mode services.

10 — Microservice Health

After microservices are deployed by their separate production workflows:

curl -si https://api.iklim.co/v1/weather/current?lat=39&lon=35

Expected: valid JSON response.

Historical / Superseded by Setup

Older verification snippets that used iklim-patroni, iklim-etcd, or separate DB stack names are superseded. Current prod DB services are part of the iklimco stack deployed from docker-stack-infra_db-prod.yml.