Environment_Infrastructure/roadmap/prod-env/07-vault-raft-plan.md

07 — Vault Raft Stack and Bootstrap Automation (Prod)

Context

Production Vault is a 3-node Raft cluster, but it is no longer initialized through a manual post-deploy runbook.

Current references:

• Stack file: root docker-stack-vault.yml
• Bootstrap script: root init/vault/vault-bootstrap.sh
• Template: root init/vault/vault-template-v2.json

Current Model

Vault is deployed separately from docker-stack-infra_db-prod.yml.

The Vault stack uses:

• 3 replicas, one per service node when placement allows it.
• Docker volumes such as vault-data-vl and vault-logs-vl.
• /opt/iklimco/ssl:/vault/certs:ro for TLS certificates.
• iklimco-net as an external overlay network.
• vault_unseal_key as a Docker secret.

The production workflow calls init-infra-prod.sh, which calls init/vault/vault-bootstrap.sh. The bootstrap script handles stack deploy, initialization, unseal key secret rotation, peer join, and peer unseal.

Certificate Flow

Vault does not read TLS certificates directly from /mnt/storagebox/ssl.

The current flow is:

SWAG renews certificate
cert-reloader copies renewed files to /mnt/storagebox/ssl
cert-distributor syncs certificate files to /opt/iklimco/ssl on service nodes
Vault reads /opt/iklimco/ssl through the /vault/certs mount

Bootstrap Flow

Normal production bootstrap is automated:

1. Create or refresh the placeholder vault_unseal_key secret when needed.
2. Deploy docker-stack-vault.yml.
3. Initialize Vault with one key share and one threshold if it is not initialized.
4. Replace the placeholder vault_unseal_key secret with the real unseal key.
5. Unseal the leader.
6. Join peers to the Raft cluster.
7. Unseal peers.
8. Verify Raft peers and service health.

These operations belong to vault-bootstrap.sh, not to a manual operator checklist.

Verification

Use the current setup verification flow:

docker service ps iklimco_vault
docker exec $(docker ps -q -f name=iklimco_vault | head -1) vault status
docker exec $(docker ps -q -f name=iklimco_vault | head -1) vault operator raft list-peers

Expected state:

• Vault service has 3 running tasks.
• vault status reports Sealed false.
• Raft list shows one leader and two followers.

Historical / Superseded by Setup

The previous manual procedure is superseded:

• Deploying Vault through docker-stack-infra.yml + docker-stack-infra.prod.yml.
• Creating /opt/iklimco/vault/data host-path directories on each app node.
• Running vault operator init manually.
• Manually copying/storing unseal keys.
• Manually running vault operator raft join on peers.
• Manually unsealing each peer after join.

Keep those notes only as historical context. For current prod, use docker-stack-vault.yml and init/vault/vault-bootstrap.sh.