Murat ÖZDEMİR
8875af8e8a
Remove setup runbook references from prod roadmap docs so roadmap remains design intent only. Keep setup-to-roadmap links, but normalize them to explicit relative paths.
159 lines
4.3 KiB
Markdown
159 lines
4.3 KiB
Markdown
# 09 — Verification Checklist (Prod)
|
|
|
|
## Context
|
|
|
|
Run these checks after a successful production pipeline deployment. This roadmap keeps verification intent; exact execution belongs to the current production workflow and operational runbooks.
|
|
|
|
## 1 — Swarm Cluster Health
|
|
|
|
```bash
|
|
docker node ls
|
|
```
|
|
|
|
Expected: 3 managers (`Leader` + 2 `Reachable`) for `iklim-app-01/02/03`, and 3 workers (`Ready`) for `iklim-db-01/02/03`.
|
|
|
|
```bash
|
|
docker node inspect iklim-app-01 --format '{{.Spec.Labels}}'
|
|
docker node inspect iklim-db-01 --format '{{.Spec.Labels}}'
|
|
```
|
|
|
|
Expected: app nodes have `type=service`; DB nodes have `role=db` and `db-index=01/02/03`.
|
|
|
|
## 2 — Infra, DB, and Vault Services
|
|
|
|
```bash
|
|
docker service ls --filter label=project=co.iklim
|
|
docker service ps iklimco_vault
|
|
docker service ps iklimco_rabbitmq
|
|
docker service ps iklimco_apisix
|
|
```
|
|
|
|
Expected: all current services show their desired replica counts.
|
|
|
|
Vault is deployed by `docker-stack-vault.yml`; the main infra and DB services are deployed by `docker-stack-infra_db-prod.yml`.
|
|
|
|
## 3 — DB Node Placement
|
|
|
|
```bash
|
|
docker service ps iklimco_patroni-01
|
|
docker service ps iklimco_patroni-02
|
|
docker service ps iklimco_patroni-03
|
|
docker service ps iklimco_mongodb-01
|
|
docker service ps iklimco_mongodb-02
|
|
docker service ps iklimco_mongodb-03
|
|
docker service ps iklimco_etcd-01
|
|
docker service ps iklimco_etcd-02
|
|
docker service ps iklimco_etcd-03
|
|
```
|
|
|
|
Expected: tasks run on their matching `iklim-db-0X` hostnames according to the stack placement constraints.
|
|
|
|
## 4 — Service-Node Infrastructure Placement
|
|
|
|
```bash
|
|
docker service ps iklimco_redis
|
|
docker service ps iklimco_redis-sentinel
|
|
docker service ps iklimco_rabbitmq
|
|
docker service ps iklimco_swag
|
|
docker service ps iklimco_cert-reloader
|
|
docker service ps iklimco_cert-distributor
|
|
```
|
|
|
|
Expected: Redis, Sentinel, RabbitMQ, SWAG, and cert services run on app/service nodes, not DB nodes.
|
|
|
|
## 5 — SWAG Certificate Is Valid
|
|
|
|
```bash
|
|
docker exec $(docker ps -q -f name=iklimco_swag | head -1) certbot certificates
|
|
```
|
|
|
|
Expected: certificate for `*.iklim.co`, valid and issued by Let's Encrypt.
|
|
|
|
TLS check from outside:
|
|
|
|
```bash
|
|
echo | openssl s_client -connect api.iklim.co:443 -servername api.iklim.co 2>/dev/null \
|
|
| openssl x509 -noout -subject -dates
|
|
```
|
|
|
|
Expected: `CN=*.iklim.co` and a current `notAfter` date.
|
|
|
|
## 6 — Public API and Restricted Subdomains
|
|
|
|
```bash
|
|
curl -si https://api.iklim.co/health
|
|
```
|
|
|
|
Expected: HTTP 2xx or an APISIX response, with no TLS error.
|
|
|
|
From a non-whitelisted IP:
|
|
|
|
```bash
|
|
curl -si https://grafana.iklim.co
|
|
curl -si https://apigw.iklim.co
|
|
curl -si https://rabbitmq.iklim.co
|
|
```
|
|
|
|
Expected: HTTP 403.
|
|
|
|
From a whitelisted IP:
|
|
|
|
```bash
|
|
curl -si https://grafana.iklim.co
|
|
curl -si https://apigw.iklim.co
|
|
curl -si https://rabbitmq.iklim.co
|
|
```
|
|
|
|
Expected: HTTP 200 or the expected login/management page.
|
|
|
|
## 7 — Vault Is Not Publicly Reachable
|
|
|
|
From outside:
|
|
|
|
```bash
|
|
curl -sk --connect-timeout 5 https://<iklim-app-01-public-ip>:8200/v1/sys/health
|
|
```
|
|
|
|
Expected: connection refused or timeout.
|
|
|
|
From inside overlay:
|
|
|
|
```bash
|
|
docker exec $(docker ps -q -f name=iklimco_apisix | head -1) \
|
|
curl -sk https://vault.iklim.co:8200/v1/sys/health
|
|
```
|
|
|
|
Expected: JSON response with `"sealed":false`.
|
|
|
|
## 8 — Certificate Reload Chain
|
|
|
|
```bash
|
|
docker service logs iklimco_cert-reloader --tail 10
|
|
docker service ps iklimco_cert-distributor
|
|
docker exec $(docker ps -q -f name=iklimco_vault | head -1) ls /vault/certs/
|
|
```
|
|
|
|
Expected: cert-reloader has no errors, cert-distributor is running, and Vault sees `STAR.iklim.co.full.crt` plus `STAR.iklim.co_key.pem`.
|
|
|
|
## 9 — No Unexpected Published Ports
|
|
|
|
```bash
|
|
docker service ls --format "{{.Name}}\t{{.Ports}}" --filter label=project=co.iklim
|
|
```
|
|
|
|
Expected: only services intentionally published by the stack expose ports. Redis and RabbitMQ must not appear as DB-node host-mode services.
|
|
|
|
## 10 — Microservice Health
|
|
|
|
After microservices are deployed by their separate production workflows:
|
|
|
|
```bash
|
|
curl -si https://api.iklim.co/v1/weather/current?lat=39&lon=35
|
|
```
|
|
|
|
Expected: valid JSON response.
|
|
|
|
## Historical / Superseded by Setup
|
|
|
|
Older verification snippets that used `iklim-patroni`, `iklim-etcd`, or separate DB stack names are superseded. Current prod DB services are part of the `iklimco` stack deployed from `docker-stack-infra_db-prod.yml`.
|