Environment_Infrastructure/roadmap/prod-env/06-cert-reloader.md

06 — cert-reloader Sidecar Service (Prod)

Context

Same service definition as test (see test-env-setup/06-cert-reloader.md). Prod-specific consideration: Vault is single-instance on the manager node (same as SWAG), so the cert copy to /opt/iklimco/ssl/ works without cross-node distribution.

When Vault is expanded to a 3-node Raft cluster (see 07-vault-raft-plan.md), the cert-reloader must be updated to distribute the cert to the other Vault nodes.

Current behavior (single-Vault prod)

SWAG (manager) renews cert → swag-vl
cert-reloader (manager) detects change → copies to /opt/iklimco/ssl/ → reloads Vault
Vault (manager) reads /opt/iklimco/ssl/ → serves new cert

No cross-node distribution needed.

Future behavior (3-node Vault Raft — see step 07)

When Vault runs on service-1, service-2, service-3:

cert-reloader detects cert change
→ copies cert to /opt/iklimco/ssl/ on service-1 (local)
→ SSH copy to service-2:/opt/iklimco/ssl/
→ SSH copy to service-3:/opt/iklimco/ssl/
→ docker service update --force iklimco_vault  (restarts all 3 replicas)

This requires:

• An SSH key that cert-reloader can use to reach service-2 and service-3
• That key mounted as a Docker secret into cert-reloader
• Known_hosts for service-2 and service-3 pre-configured

Script update for this phase is tracked in 07-vault-raft-plan.md.

Verification

docker service ps iklimco_cert-reloader
docker service logs iklimco_cert-reloader --tail 20

Expected: [cert-reloader] started, no error lines.

Confirm Vault cert is current after SWAG renewal:

# Check cert expiry on Vault's TLS endpoint from inside the overlay
docker exec $(docker ps -q -f name=iklimco_vault) \
  sh -c 'echo | openssl s_client -connect vault.iklim.co:8200 2>/dev/null \
         | openssl x509 -noout -dates'

notAfter should match the cert in /opt/iklimco/ssl/STAR.iklim.co.full.crt.