# 09 — Verification Checklist (Prod)

## Context

Run these checks after a successful production pipeline deployment. This roadmap keeps verification intent; exact execution belongs to the current production workflow and operational runbooks.

## 1 — Swarm Cluster Health

```bash
docker node ls
```

Expected: 3 managers (`Leader` + 2 `Reachable`) for `iklim-app-01/02/03`, and 3 workers (`Ready`) for `iklim-db-01/02/03`.

```bash
docker node inspect iklim-app-01 --format '{{.Spec.Labels}}'
docker node inspect iklim-db-01 --format '{{.Spec.Labels}}'
```

Expected: app nodes have `type=service`; DB nodes have `role=db` and `db-index=01/02/03`.

## 2 — Infra, DB, and Vault Services

```bash
docker service ls --filter label=project=co.iklim
docker service ps iklimco_vault
docker service ps iklimco_rabbitmq
docker service ps iklimco_apisix
```

Expected: all current services show their desired replica counts.

Vault is deployed by `docker-stack-vault.yml`; the main infra and DB services are deployed by `docker-stack-infra_db-prod.yml`.

## 3 — DB Node Placement

```bash
docker service ps iklimco_patroni-01
docker service ps iklimco_patroni-02
docker service ps iklimco_patroni-03
docker service ps iklimco_mongodb-01
docker service ps iklimco_mongodb-02
docker service ps iklimco_mongodb-03
docker service ps iklimco_etcd-01
docker service ps iklimco_etcd-02
docker service ps iklimco_etcd-03
```

Expected: tasks run on their matching `iklim-db-0X` hostnames according to the stack placement constraints.

## 4 — Service-Node Infrastructure Placement

```bash
docker service ps iklimco_redis
docker service ps iklimco_redis-sentinel
docker service ps iklimco_rabbitmq
docker service ps iklimco_swag
docker service ps iklimco_cert-reloader
docker service ps iklimco_cert-distributor
```

Expected: Redis, Sentinel, RabbitMQ, SWAG, and cert services run on app/service nodes, not DB nodes.

## 5 — SWAG Certificate Is Valid

```bash
docker exec $(docker ps -q -f name=iklimco_swag | head -1) certbot certificates
```

Expected: certificate for `*.iklim.co`, valid and issued by Let's Encrypt.

TLS check from outside:

```bash
echo | openssl s_client -connect api.iklim.co:443 -servername api.iklim.co 2>/dev/null \
  | openssl x509 -noout -subject -dates
```

Expected: `CN=*.iklim.co` and a current `notAfter` date.

## 6 — Public API and Restricted Subdomains

```bash
curl -si https://api.iklim.co/health
```

Expected: HTTP 2xx or an APISIX response, with no TLS error.

From a non-whitelisted IP:

```bash
curl -si https://grafana.iklim.co
curl -si https://apigw.iklim.co
curl -si https://rabbitmq.iklim.co
```

Expected: HTTP 403.

From a whitelisted IP:

```bash
curl -si https://grafana.iklim.co
curl -si https://apigw.iklim.co
curl -si https://rabbitmq.iklim.co
```

Expected: HTTP 200 or the expected login/management page.

## 7 — Vault Is Not Publicly Reachable

From outside:

```bash
curl -sk --connect-timeout 5 https://<iklim-app-01-public-ip>:8200/v1/sys/health
```

Expected: connection refused or timeout.

From inside overlay:

```bash
docker exec $(docker ps -q -f name=iklimco_apisix | head -1) \
  curl -sk https://vault.iklim.co:8200/v1/sys/health
```

Expected: JSON response with `"sealed":false`.

## 8 — Certificate Reload Chain

```bash
docker service logs iklimco_cert-reloader --tail 10
docker service ps iklimco_cert-distributor
docker exec $(docker ps -q -f name=iklimco_vault | head -1) ls /vault/certs/
```

Expected: cert-reloader has no errors, cert-distributor is running, and Vault sees `STAR.iklim.co.full.crt` plus `STAR.iklim.co_key.pem`.

## 9 — No Unexpected Published Ports

```bash
docker service ls --format "{{.Name}}\t{{.Ports}}" --filter label=project=co.iklim
```

Expected: only services intentionally published by the stack expose ports. Redis and RabbitMQ must not appear as DB-node host-mode services.

## 10 — Microservice Health

After microservices are deployed by their separate production workflows:

```bash
curl -si https://api.iklim.co/v1/weather/current?lat=39&lon=35
```

Expected: valid JSON response.

## Historical / Superseded by Setup

Older verification snippets that used `iklim-patroni`, `iklim-etcd`, or separate DB stack names are superseded. Current prod DB services are part of the `iklimco` stack deployed from `docker-stack-infra_db-prod.yml`.