06 — cert-reloader Sidecar Service (Test)

Context

Purpose: Watches SWAG's certificate volume for changes; copies renewed certs to /opt/iklimco/ssl/ on the host; forces Vault to reload its TLS cert.
Replaces: ops/vault-reload-after-swag-renewal.sh (which was designed for manual use). The sidecar automates this after every SWAG renewal.
Runs on: manager node (same node as SWAG and Vault, ensuring volume + socket access).

How it works

SWAG renews cert
  → writes new fullchain.pem to swag-vl:/config/etc/letsencrypt/live/iklim.co/
cert-reloader wakes every 3600s
  → detects MD5 change on fullchain.pem
  → copies fullchain.pem + privkey.pem to /opt/iklimco/ssl/ (host bind mount)
  → docker service update --force iklimco_vault
Vault restarts
  → reads new cert from /opt/iklimco/ssl/ (already mounted as /vault/certs)

Step 1 — Service definition (already in `03-infra-stack-changes.md`)

The cert-reloader service is added to docker-stack-infra.yml as documented in step 03. No separate action needed here beyond that file change.

Step 2 — Ensure `/opt/iklimco/ssl/` exists on the host

The Prepare Init Files step in the pipeline already creates this directory and copies the initial cert. The cert-reloader handles subsequent renewals.

On first deploy, the bootstrap cert (copied during pipeline init) is used until SWAG obtains its first Let's Encrypt cert (see 07-deploy-pipeline-update.md).

Step 3 — Verify cert-reloader is running

docker service ps iklimco_cert-reloader
docker service logs iklimco_cert-reloader --tail 20

Expected log on startup:

[cert-reloader] started

Step 4 — Trigger a manual test (optional, for verification)

Force a cert copy and Vault reload without waiting for renewal:

SWAG_VOL=$(docker volume inspect iklimco_swag-vl --format '{{.Mountpoint}}')
CERT="$SWAG_VOL/etc/letsencrypt/live/iklim.co/fullchain.pem"

if [ -f "$CERT" ]; then
  cp "$CERT" /opt/iklimco/ssl/STAR.iklim.co.full.crt
  KEYF="$SWAG_VOL/etc/letsencrypt/live/iklim.co/privkey.pem"
  cp "$KEYF" /opt/iklimco/ssl/STAR.iklim.co_key.txt
  docker service update --force iklimco_vault
  echo "✅ Manual reload triggered"
else
  echo "⚠️  Cert not yet obtained by SWAG"
fi

Notes

Docker socket (/var/run/docker.sock) is mounted into cert-reloader — this is intentional and necessary. The service is pinned to manager and is minimal (docker:27-cli image).
cert-reloader checks every 3600s (1 hour). Let's Encrypt certs renew every ~60 days; the 1-hour check window is more than sufficient.
If Vault restarts (due to cert reload), it may need to be unsealed automatically. Vault's healthcheck in docker-stack-infra.yml already handles auto-unseal via the vault_unseal_key Docker secret. Verify this works after a cert reload.

Future — Multi-node Vault (prod)

When Vault runs as a 3-node Raft cluster on different physical machines, cert-reloader must also SSH-copy the cert to the other nodes' /opt/iklimco/ssl/. This is handled in prod-env-setup/06-cert-reloader.md.

3.1 KiB Raw Blame History