Murat ÖZDEMİR
8875af8e8a
Remove setup runbook references from prod roadmap docs so roadmap remains design intent only. Keep setup-to-roadmap links, but normalize them to explicit relative paths.
4.3 KiB
09 — Verification Checklist (Prod)
Context
Run these checks after a successful production pipeline deployment. This roadmap keeps verification intent; exact execution belongs to the current production workflow and operational runbooks.
1 — Swarm Cluster Health
docker node ls
Expected: 3 managers (Leader + 2 Reachable) for iklim-app-01/02/03, and 3 workers (Ready) for iklim-db-01/02/03.
docker node inspect iklim-app-01 --format '{{.Spec.Labels}}'
docker node inspect iklim-db-01 --format '{{.Spec.Labels}}'
Expected: app nodes have type=service; DB nodes have role=db and db-index=01/02/03.
2 — Infra, DB, and Vault Services
docker service ls --filter label=project=co.iklim
docker service ps iklimco_vault
docker service ps iklimco_rabbitmq
docker service ps iklimco_apisix
Expected: all current services show their desired replica counts.
Vault is deployed by docker-stack-vault.yml; the main infra and DB services are deployed by docker-stack-infra_db-prod.yml.
3 — DB Node Placement
docker service ps iklimco_patroni-01
docker service ps iklimco_patroni-02
docker service ps iklimco_patroni-03
docker service ps iklimco_mongodb-01
docker service ps iklimco_mongodb-02
docker service ps iklimco_mongodb-03
docker service ps iklimco_etcd-01
docker service ps iklimco_etcd-02
docker service ps iklimco_etcd-03
Expected: tasks run on their matching iklim-db-0X hostnames according to the stack placement constraints.
4 — Service-Node Infrastructure Placement
docker service ps iklimco_redis
docker service ps iklimco_redis-sentinel
docker service ps iklimco_rabbitmq
docker service ps iklimco_swag
docker service ps iklimco_cert-reloader
docker service ps iklimco_cert-distributor
Expected: Redis, Sentinel, RabbitMQ, SWAG, and cert services run on app/service nodes, not DB nodes.
5 — SWAG Certificate Is Valid
docker exec $(docker ps -q -f name=iklimco_swag | head -1) certbot certificates
Expected: certificate for *.iklim.co, valid and issued by Let's Encrypt.
TLS check from outside:
echo | openssl s_client -connect api.iklim.co:443 -servername api.iklim.co 2>/dev/null \
| openssl x509 -noout -subject -dates
Expected: CN=*.iklim.co and a current notAfter date.
6 — Public API and Restricted Subdomains
curl -si https://api.iklim.co/health
Expected: HTTP 2xx or an APISIX response, with no TLS error.
From a non-whitelisted IP:
curl -si https://grafana.iklim.co
curl -si https://apigw.iklim.co
curl -si https://rabbitmq.iklim.co
Expected: HTTP 403.
From a whitelisted IP:
curl -si https://grafana.iklim.co
curl -si https://apigw.iklim.co
curl -si https://rabbitmq.iklim.co
Expected: HTTP 200 or the expected login/management page.
7 — Vault Is Not Publicly Reachable
From outside:
curl -sk --connect-timeout 5 https://<iklim-app-01-public-ip>:8200/v1/sys/health
Expected: connection refused or timeout.
From inside overlay:
docker exec $(docker ps -q -f name=iklimco_apisix | head -1) \
curl -sk https://vault.iklim.co:8200/v1/sys/health
Expected: JSON response with "sealed":false.
8 — Certificate Reload Chain
docker service logs iklimco_cert-reloader --tail 10
docker service ps iklimco_cert-distributor
docker exec $(docker ps -q -f name=iklimco_vault | head -1) ls /vault/certs/
Expected: cert-reloader has no errors, cert-distributor is running, and Vault sees STAR.iklim.co.full.crt plus STAR.iklim.co_key.pem.
9 — No Unexpected Published Ports
docker service ls --format "{{.Name}}\t{{.Ports}}" --filter label=project=co.iklim
Expected: only services intentionally published by the stack expose ports. Redis and RabbitMQ must not appear as DB-node host-mode services.
10 — Microservice Health
After microservices are deployed by their separate production workflows:
curl -si https://api.iklim.co/v1/weather/current?lat=39&lon=35
Expected: valid JSON response.
Historical / Superseded by Setup
Older verification snippets that used iklim-patroni, iklim-etcd, or separate DB stack names are superseded. Current prod DB services are part of the iklimco stack deployed from docker-stack-infra_db-prod.yml.