# 08 — Verification Checklist (Test)

## Context
Run these checks after a successful pipeline deployment to the test environment.

## 1 — Swarm services are up

```bash
docker service ls --filter label=project=co.iklim
```

All services should show `REPLICAS 1/1`.

```bash
docker service ps iklimco_swag
docker service ps iklimco_cert-reloader
docker service ps iklimco_vault
docker service ps iklimco_apisix
```

No tasks in `Failed` or `Rejected` state.

## 2 — SWAG obtained the cert

```bash
docker exec $(docker ps -q -f name=iklimco_swag) \
  certbot certificates
```

Expected: certificate for `*.iklim.co`, `VALID: XX days`.

```bash
docker exec $(docker ps -q -f name=iklimco_swag) \
  ls /config/etc/letsencrypt/live/iklim.co/
```

Expected: `fullchain.pem`, `privkey.pem`, `cert.pem`, `chain.pem`.

## 3 — Nginx config is valid

```bash
docker exec $(docker ps -q -f name=iklimco_swag) nginx -t
```

Expected: `syntax is ok` and `test is successful`.

## 4 — Public API endpoint

```bash
curl -si https://api-test.iklim.co/health
```

Expected: HTTP 2xx or APISIX response (not a cert error, not a 502).

TLS cert check:
```bash
echo | openssl s_client -connect api-test.iklim.co:443 -servername api-test.iklim.co 2>/dev/null \
  | openssl x509 -noout -subject -dates
```

Expected: `subject=CN=*.iklim.co`, dates valid, `notAfter` > today.

## 5 — IP-restricted subdomains block non-whitelisted IPs

From a non-whitelisted IP:
```bash
curl -si https://grafana-test.iklim.co
```
Expected: HTTP 403.

From a whitelisted IP (78.187.87.109 or 95.70.151.248):
```bash
curl -si https://grafana-test.iklim.co
```
Expected: HTTP 200 (Grafana login page).

## 6 — Vault is reachable internally (not externally)

From outside the server:
```bash
curl -sk https://vault.iklim.co:8200/v1/sys/health
# or
curl -sk https://<server-public-ip>:8200/v1/sys/health
```
Expected: **connection refused** or **timeout** — Vault must not be reachable externally.

From inside the Swarm (exec into any service container):
```bash
docker exec $(docker ps -q -f name=iklimco_apisix | head -1) \
  curl -sk https://vault.iklim.co:8200/v1/sys/health
```
Expected: JSON response `{"sealed":false,...}`.

## 7 — cert-reloader is watching

```bash
docker service logs iklimco_cert-reloader --tail 10
```
Expected: `[cert-reloader] started` — no errors.

## 8 — Vault cert path is correct

```bash
VAULT_CTR=$(docker ps -q -f name=iklimco_vault)
docker exec "$VAULT_CTR" ls /vault/certs/
```
Expected: `STAR.iklim.co.full.crt` and `STAR.iklim.co_key.txt`.

## 9 — fail2ban is active (SWAG)

```bash
docker exec $(docker ps -q -f name=iklimco_swag) \
  fail2ban-client status
```
Expected: list of jails including `nginx-http-auth`, `nginx-botsearch`, etc.

## 10 — No services have published unexpected ports

```bash
docker service ls --format "{{.Name}}\t{{.Ports}}" \
  --filter label=project=co.iklim
```

Only `iklimco_swag` should have published ports (`*:80->80`, `*:443->443`).
All other services should show empty ports column.