Environment_Infrastructure/setup/09-prod-runner-ha-and-swarm.md

09 - Prod Runner HA and Swarm Deploy Model

The purpose of this phase is to set up Gitea Actions runners in prod so they run in HA mode and define the prerequisites for distributing services across 3 nodes on Swarm.

Runner Count

A single runner is functionally enough, but it is not HA. Because the prod target is HA, act_runner will be installed as a systemd service on all 3 Swarm manager nodes:

Host	Runner
iklim-app-01	act_runner systemd
iklim-app-02	act_runner systemd
iklim-app-03	act_runner systemd

In this model, if any manager/runner is lost, the other runners can pick up pipeline jobs.

Runner Installation Model

The runner will not run as a Docker container. It runs as a systemd service on the app nodes. Job containers start on Docker bridge; deploy workflows connect the job container to iklimco-net after the stack creates that network.

Installation:

• gitea-runner sistem kullanicisi
• /usr/local/bin/act_runner
• /etc/gitea-act-runner/config.yaml
• /var/lib/gitea-runner
• gitea-act-runner.service

If runner jobs use Docker CLI for deploy, the gitea-runner user needs access to the Docker daemon. Docker group membership is considered close to root-level permission; only trusted repos/jobs should use these runner labels.

Runner Label Policy

Shared labels on all prod runners:

prod-runner:docker://catthehacker/ubuntu:act-22.04
ubuntu-24.04

Node-specific labels (hostname of each app node):

iklim-app-01
iklim-app-02
iklim-app-03

If existing prod workflows use runs-on: prod-runner, any of the 3 runners can pick up the job. If pinning to a specific node is required, use a node-specific label.

Deploy Race Risk

When there is more than one runner, multiple deploy jobs can run at the same time. This is good for HA, but it can create race risk on shared resources.

Risk areas:

• Concurrent docker stack deploy on the same stack
• Concurrent docker service update for the same service
• Concurrent updates to the same .env or manifest file on StorageBox
• Root infrastructure pipeline and microservice deploy pipeline running at the same time

Required measure:

• Prod root infrastructure deploy should run manually or with approval.
• Prod deploy for the same service must not be triggered more than once at the same time.
• All prod deploy workflows are queued with the Gitea Actions concurrency: group: prod-deploy block; concurrent execution is prevented by Gitea.

Prerequisites — StorageBox Secrets

Before the deploy pipeline runs, the following files must exist on StorageBox. These files are not created automatically; they are created manually during the initial setup.

SWAG / GoDaddy Credentials

prod/secrets/iklim.co/.env.secrets.swag

GODADDY_KEY=<api-key>
GODADDY_SECRET=<api-secret>

For the GoDaddy API key: https://developer.godaddy.com/keys — create a Production key. If an existing key is known to have been shared in any chat, Slack, or email, revoke it before use and create a new one.

.env.secrets.swag contains only SWAG/GoDaddy credentials. .env.secrets.shared contains AppRole IDs, DB passwords, and other runtime secrets — do not mix these two files.

Gitea PROD_FLOATING_IP Variable

For DNS automation, PROD_FLOATING_IP must be defined as a Gitea project variable. See the "Gitea Variable: PROD_FLOATING_IP" step in 06-prod-terraform-iac.md.

Docker Secrets

Before the infra stack is deployed, rabbitmq_erlang_cookie must exist as a Docker secret. The current prod workflow creates it in the Create Infrastructure Docker Secrets step if it is missing.

# RabbitMQ Erlang cluster cookie; must be the same on all RabbitMQ nodes.
# The workflow does this automatically if the secret is missing:
openssl rand -hex 32 | docker secret create rabbitmq_erlang_cookie -

The vault_unseal_key secret is managed by init/vault/vault-bootstrap.sh. The bootstrap script creates a placeholder on first deploy, deploys docker-stack-vault.yml, initializes/unseals Vault, and rotates the secret to the real unseal key.

Verify secrets:

docker secret ls
# rabbitmq_erlang_cookie row must appear

SWAG Nginx Configuration Templates

Before the deploy pipeline runs, the following template files must exist in the repo:

• template/swag/site-confs/default.conf
• template/swag/site-confs/api.conf.tpl
• template/swag/site-confs/apigw.conf.tpl
• template/swag/site-confs/rabbitmq.conf.tpl
• template/swag/site-confs/grafana.conf.tpl

These files are created in the test environment (test-env/04-swag-nginx-configs.md); they are not created separately for prod. Template files are shared by both environments; prod-specific values are injected with environment variables during deploy.

Verify that the prod/secrets/iklim.co/.env file on StorageBox contains the following variables:

API_SUBDOMAIN=api.iklim.co
APIGW_SUBDOMAIN=apigw.iklim.co
RABBITMQ_SUBDOMAIN=rabbitmq.iklim.co
GRAFANA_SUBDOMAIN=grafana.iklim.co
RESTRICTED_IPS="78.187.87.109/32,95.70.151.248/32"
SWAG_CERT_DIR=/mnt/storagebox/ssl
SWAG_DNS_CONFIG_DIR=/mnt/storagebox/swag/dns-conf
SWAG_SITE_CONFS_DIR=/mnt/storagebox/swag/site-confs
SWAG_PROXY_CONFS_DIR=/mnt/storagebox/swag/proxy-confs

The pipeline sources these variables and renders the template files into the $SWAG_SITE_CONFS_DIR (/mnt/storagebox/swag/site-confs) directory. Because StorageBox is mounted commonly on all app nodes, even if the configuration is created on a single runner, SWAG containers on other nodes access the same files.

APISIX Configuration

The following prerequisites must be satisfied before deploy.

init.sh SSL Block

The ssls/1 PUT block and the dev SSL block inside init/apisix-core/init.sh must be removed. This change is made in the test environment (test-env/05-apisix-remove-ssl.md); the same init.sh file is also used in prod, so no separate change is required for prod.

Custom APISIX Image

The prod stack uses the registry.tarla.io/iklimco/custom-apisix:3.12.0 image. This image's config.yaml must contain real IP header configuration for the overlay CIDR:

nginx_config:
  http:
    real_ip_header: "X-Real-IP"
    real_ip_recursive: "on"
    set_real_ip_from:
      - "10.0.0.0/8"
      - "172.16.0.0/12"
      - "192.168.0.0/16"

These three CIDR ranges cover all typical Docker Swarm overlay subnet allocations. APISIX reads the real client IP from SWAG's X-Real-IP header instead of the overlay container IP.

If the image requires a rebuild because config.yaml changed, run from the project root:

bash ops/push-harbor-custom-images.sh

During deploy, init/apisix-core/init.sh is run once by the pipeline. It writes the APISIX configuration to Patroni etcd with the /apisix prefix; the 3 replicas in prod read this etcd state commonly, so no separate init per replica is required. Detail: roadmap/prod-env/05-apisix-remove-ssl.md.

Deploy Serialization with Gitea Concurrency

Because 3 runners run in prod, more than one deploy job can be triggered at the same time. Instead of a StorageBox-based mkdir/rmdir lock mechanism, the Gitea Actions concurrency feature is used.

Add the following block to the pipeline file (deploy-prod.yml):

concurrency:
  group: prod-deploy
  cancel-in-progress: false

With cancel-in-progress: false, a new run in the same group is queued by Gitea until the previous one finishes. It appears as "queued" in the UI and is not shown as an error. There is no stale lock risk: even if the runner crashes or the job is canceled, Gitea handles state management.

All prod deploy workflows, including infra and microservices, must use the same group: prod-deploy value so infra deploy and microservice deploy cannot overlap.

Deploy Pipeline

.gitea/workflows/deploy-prod.yml is the full step order of the prod deploy pipeline. Steps marked with * are prod-specific and do not exist in the test pipeline.

#	Step	Note
1	Checkout Branch
2	Prepare Folders
3	Set up SSH Key and Add to known_hosts
4	Update Apt Repository and Install Required Tools	gettext tree jq — jq is required for the GoDaddy DNS API
5	Fetch Prod Env From Storagebox	Fetch .env and .env.secrets.shared
6	Fetch Service Secret Files	Fetch .env.secrets.<svc> and .env.secrets.swag
7	Prepare Database Init Files	Render PostgreSQL/MongoDB init templates
8	Docker Login to Harbor
9	Prepare SWAG Directories	Render dns-conf and site-confs; reload node-local SWAG if present
10	Bootstrap Vault TLS Placeholder	Creates temporary cert only if missing
11	Create Infrastructure Docker Secrets	Creates rabbitmq_erlang_cookie if missing
12	Deploy Swarm Stacks	docker-stack-infra_db-prod.yml
13	Connect Runner to Overlay Network	Connects job container to iklimco-net
14	Initialize Production Infrastructure	Runs init-infra-prod.sh; this triggers Vault bootstrap and RabbitMQ setup
15	Wait for Infrastructure Services	Waits for iklimco_vault and iklimco_rabbitmq
16	Provision Vault AppRole IDs and Docker Secrets	Downloads service vault-files, runs init/provision-all-services.sh
17	Upload Updated Secrets to Storagebox	Uploads .env.secrets.* and .env
18	Wait for etcd	Waits for etcd health
19	Run APISIX Init	SPRING_PROFILES_ACTIVE=prod
20	Bootstrap SWAG Certificate	Waits for SWAG and cert-reloader output in SWAG_CERT_DIR
21	Initialize MongoDB Replica Set	rs.initiate() or missing-member rs.add()
22	Run Database Init Scripts	Patroni primary + MongoDB replica set; SQL+JS init
23	Update DNS Records	GoDaddy API; api/apigw/rabbitmq/grafana A records
24	Review Environment

Stack Placement Boundary

docker-stack-infra_db-prod.yml is intentionally a mixed infrastructure stack. The DB/cluster services in that file are placed on DB nodes and expose host-mode cluster ports:

• Patroni/PostgreSQL, MongoDB, and etcd run on iklim-db-* workers.

The service-node infrastructure in the same file remains overlay-only unless a reverse proxy or explicit published port is defined by the stack:

• Redis, Redis Sentinel, and RabbitMQ run on node.labels.type == service app/service nodes.
• Redis and RabbitMQ must not be treated as DB-node host-mode services.

Historical Note: Removed Cert Scp Lines

Older workflow versions copied certificate files manually in an Initialize Workspace step. That step no longer exists in the current root prod workflow. The removed lines are kept here only as a historical reference:

# REMOVED — manual cert copy with scp is no longer required:
scp -P 23 ${{ vars.STORAGEBOX_USER }}@${{ vars.STORAGEBOX_USER }}.your-storagebox.de:prod/app/iklim.co/ssl/STAR.iklim.co.full.crt ./STAR.iklim.co.full.crt
scp -P 23 ${{ vars.STORAGEBOX_USER }}@${{ vars.STORAGEBOX_USER }}.your-storagebox.de:prod/app/iklim.co/ssl/STAR.iklim.co_key.pem ./STAR.iklim.co_key.pem

This line was also removed from the old Prepare Init Files step:

# REMOVED:
sudo cp STAR.iklim.co.full.crt STAR.iklim.co_key.pem /opt/iklimco/ssl/

The certificate is now obtained by SWAG from Let's Encrypt and written to the SWAG_CERT_DIR (/mnt/storagebox/ssl/) directory in the Bootstrap SWAG Certificate step. Later renewals are handled automatically by cert-reloader.

Bootstrap SWAG Certificate (Step 20)

On the first deploy, SWAG obtains the Let's Encrypt certificate with the GoDaddy DNS-01 challenge. The current step waits for the Swarm iklimco_swag service to be running, then waits for cert-reloader to write STAR.iklim.co.full.crt to SWAG_CERT_DIR.

- name: Bootstrap SWAG Certificate
  run: |
    set -a; . ./.env; set +a
    echo "Waiting for SWAG service..."
    docker service ps iklimco_swag --filter 'desired-state=running'
    echo "Waiting for cert-reloader output in ${SWAG_CERT_DIR}..."
    docker run --rm -v "${SWAG_CERT_DIR}:/ssl:ro" alpine \
      test -f /ssl/STAR.iklim.co.full.crt
  working-directory: /workspace/iklim.co

After this step, certificate files exist inside SWAG_CERT_DIR (/mnt/storagebox/ssl/). cert-distributor syncs these files to node-local /opt/iklimco/ssl, where Vault reads them. Later renewals are handled automatically by SWAG, cert-reloader, and cert-distributor.

Run Database Init Scripts (Step 22)

PostgreSQL and MongoDB init scripts run after Patroni primary and MongoDB replica set readiness:

- name: Run Database Init Scripts
  run: |
    set -a; . ./.env; . ./.env.secrets.shared; set +a

    PG_URI="postgresql://${DATABASE_POSTGRES_ROOT_USER}@${DATABASE_POSTGRES_HOST}/postgres?connect_timeout=5&target_session_attrs=read-write"
    MONGO_URI="mongodb://${DATABASE_MONGODB_ROOT_USER}:${DATABASE_MONGODB_ROOT_PASSWD}@${DATABASE_MONGODB_HOST}/admin?${DATABASE_MONGODB_PARAMS}"
    for sql_file in $(ls ./init/postgresql/*.sql 2>/dev/null | sort); do
      echo "▶ $(basename "$sql_file")"
      docker run --rm -i --network iklimco-net \
        -e PGPASSWORD="${DATABASE_POSTGRES_ROOT_PASSWD}" \
        postgis/postgis:18-3.6 \
        psql "$PG_URI" < "$sql_file"
    done

    for js_file in $(ls ./init/mongodb/*.js 2>/dev/null | sort); do
      echo "▶ $(basename "$js_file")"
      docker run --rm -i --network iklimco-net "${IMAGE_MONGODB}" \
        sh -c 'cat > /tmp/init.js && mongosh "$MONGO_INIT_URI" --quiet --file /tmp/init.js' \
        < "$js_file"
    done
    echo "✅ Database init scripts completed"
  working-directory: /workspace/iklim.co

• DATABASE_POSTGRES_HOST: multi-host Patroni target; the workflow uses target_session_attrs=read-write to reach the primary
• DATABASE_MONGODB_HOST: MongoDB replica set host list
• SQL files ./init/postgresql/*.sql and JS files ./init/mongodb/*.js are created in the Prepare Init Files step by the init_postgresql/init_mongodb functions in common-functions-prod.sh
• Idempotent: CREATE IF NOT EXISTS / createCollection semantics; runs safely again on later deploys

Swarm Service Distribution

In prod, all 3 app nodes are manager + app worker, so services can be distributed across 3 nodes.

Microservices

Prod microservice workflows do not rebuild application images. They read deploy/prod.env, promote the tested Harbor digest to a stable prod tag, and call swarm_service_update with deploy/docker-stack-service.yml.

For first deploy, swarm_service_update exports SERVICE_IMAGE and runs:

docker stack deploy --with-registry-auth -c deploy/docker-stack-service.yml iklimco

For existing services it performs docker service update with --update-order start-first and --update-failure-action rollback.

Infra Services

The current prod infra stack is docker-stack-infra_db-prod.yml. Vault is not inside this stack; it is deployed separately by vault-bootstrap.sh using docker-stack-vault.yml.

cert-reloader and Vault Auto-Unseal

The cert-reloader sidecar service runs as replicas: 1 inside the infra stack. It detects the Let's Encrypt certificate renewed by SWAG and distributes it to Vault. Because prod uses the shared StorageBox mount, SSH-based distribution is not required.

Certificate renewal flow:

SWAG renews the certificate -> stores it inside the SWAG named volume
cert-reloader detects the MD5 change
  -> copies it to /mnt/storagebox/ssl/ directory  (StorageBox)
cert-distributor syncs it to /opt/iklimco/ssl on service nodes
  -> runs docker service update --force iklimco_vault
Vault (3 replicas) restarts
  -> each instance reads the new certificate from /opt/iklimco/ssl
  -> entrypoint retry-unseal loop reads from the vault_unseal_key Docker secret and unseals

The 3 Vault replicas run their own retry-unseal loop independently. The certificate renewal -> distribution -> restart -> unseal chain requires no manual intervention after bootstrap.

Vault Raft Configuration

Vault is defined as 3 replicas with Raft storage in docker-stack-vault.yml:

vault:
  volumes:
    - vault-data-vl:/vault/file
    - vault-logs-vl:/vault/logs
    - /opt/iklimco/ssl:/vault/certs:ro
  deploy:
    mode: replicated
    replicas: 3
    placement:
      max_replicas_per_node: 1
      constraints:
        - node.labels.type == service

The Vault stack uses vault-template-v2.json, vault_unseal_key, and the iklimco-net external network. Bootstrap and unseal are handled by init/vault/vault-bootstrap.sh.

Vault Raft Cluster Initial Setup

Vault Raft cluster setup is no longer a manual post-deploy procedure. It is handled by init/vault/vault-bootstrap.sh, called through init-infra-prod.sh by the root prod workflow.

Step 1 — Stack Deploy

The bootstrap script deploys:

docker stack deploy --with-registry-auth -c docker-stack-vault.yml iklimco

Step 2 — Vault Initialize (iklim-app-01)

The script runs vault operator init -key-shares=1 -key-threshold=1 if Vault is not initialized. It stores bootstrap output under /tmp/vault-bootstrap/main-vault-init.txt during the run.

echo "bootstrap" | docker secret create vault_unseal_key -

Then it rotates vault_unseal_key to the real unseal key and unseals the leader and peers.

Step 3 — Unseal the Leader

No manual unseal command is required in the normal path.

Step 4 — Join the Other Nodes to the Raft Cluster

Peer join and peer unseal are handled by vault-bootstrap.sh.

Step 5 — Verify the Cluster

docker exec "$VAULT_CTR" vault operator raft list-peers

Expected: 3 peers — one leader, two follower.

Gateway and Public Traffic

Public internet enters only through SWAG on 80/tcp and 443/tcp. SWAG is pinned to iklim-app-01, where the Floating IP is located. APISIX admin ports (9180) and other service ports are not opened publicly; SWAG forwards all public traffic to APISIX as a reverse proxy.

Subdomain Routing

Subdomain	Target Service	Restriction
api.iklim.co	APISIX :9080	Public
apigw.iklim.co	APISIX Dashboard :9000	IP restricted with RESTRICTED_IPS
rabbitmq.iklim.co	RabbitMQ Management :15672	IP restricted with RESTRICTED_IPS
grafana.iklim.co	Grafana :3000	IP restricted with RESTRICTED_IPS

IP restriction is done with the RESTRICTED_IPS_BLOCK nginx allow block derived from the RESTRICTED_IPS variable; it is applied in SWAG nginx configuration, not in the Hetzner firewall.

SWAG -> APISIX Load Distribution

SWAG connects to APISIX through the Docker Swarm service name with proxy_pass http://apisix:9080;. Swarm resolves the apisix service name to a VIP (Virtual IP); the IPVS load balancer distributes incoming connections round-robin across the 3 replicas in prod. No additional upstream or load balancer configuration is required on the SWAG side; load distribution happens transparently at the overlay network layer.

Prometheus is intentionally not exposed externally through SWAG. Access uses Grafana, whose internal connection is http://prometheus:9090, or an SSH tunnel.

Detay: Environment_Infrastructure/roadmap/prod-env/04-swag-nginx-configs.md.

Post-Deploy Verification

After a successful prod pipeline deploy, run the following checks.

Swarm Health

docker node ls

Expected: 3 managers (Leader + 2 Reachable) — iklim-app-01/02/03; 3 workers (Ready) — iklim-db-01/02/03.

docker service ls --filter label=project=co.iklim

All services must show REPLICAS X/X; target met.

Precipitation Image Directory

ls -ld /mnt/storagebox/precipitation/images

The directory must exist; it is required before iklimco_precipitation-service is deployed.

docker volume inspect iklimco_image-data

Expected: Options.device -> /mnt/storagebox/precipitation/images.

SWAG Certificate

docker exec $(docker ps -q -f name=iklimco_swag) certbot certificates

Expected: *.iklim.co, VALID: XX days (Let's Encrypt — not the old manual cert).

TLS check from outside:

echo | openssl s_client -connect api.iklim.co:443 -servername api.iklim.co 2>/dev/null \
  | openssl x509 -noout -subject -dates

Expected: CN=*.iklim.co, notAfter > 2026-07-15.

Warning: The old manual *.iklim.co certificate expires on 2026-07-15. After SWAG's Let's Encrypt certificate is verified for the first time, the old cert on StorageBox can be archived and is no longer used.

Public API Access

curl -si https://api.iklim.co/health

It must return HTTP 2xx; there must be no TLS error.

IP Restriction

From a disallowed IP:

curl -si https://grafana.iklim.co
curl -si https://apigw.iklim.co
curl -si https://rabbitmq.iklim.co

All must return HTTP 403.

From an allowed IP (78.187.87.109 or 95.70.151.248):

curl -si https://grafana.iklim.co    # HTTP 200 Grafana
curl -si https://apigw.iklim.co      # HTTP 200 APISIX Dashboard
curl -si https://rabbitmq.iklim.co   # HTTP 200 RabbitMQ Management

Vault Access Control

Must not be reachable from outside:

# Expected: connection refused or timeout
curl -sk --connect-timeout 5 https://<iklim-app-01-public-ip>:8200/v1/sys/health

Must be reachable from inside the overlay:

# Expected: {"sealed":false,...}
docker exec $(docker ps -q -f name=iklimco_apisix | head -1) \
  curl -sk https://vault.iklim.co:8200/v1/sys/health

No Unexpected Ports

docker service ls --format "{{.Name}}\t{{.Ports}}" \
  --filter label=project=co.iklim

Only iklimco_swag -> *:80->80/tcp, *:443->443/tcp should publish ports; other services must not publish ports.

APISIX Replica Distribution

docker service ps iklimco_apisix

Expected: 3 tasks, all Running, on different nodes.

fail2ban (SWAG Container)

docker exec $(docker ps -q -f name=iklimco_swag) fail2ban-client status

Expected: a list with more than one jail.

Microservice Health (After Microservices Are Deployed)

After microservices are deployed with a separate pipeline:

curl -si "https://api.iklim.co/v1/weather/current?lat=39&lon=35"

Expected: valid JSON weather response.

Acceptance Criteria

• 3 prod runners appear online in the Gitea UI.
• Every runner has the prod-runner label.
• Any runner can run a simple Docker command.
• docker node ls shows 3 managers.
• When one runner/node is shut down, another runner can pick up a new job.
• All prod deploy workflows (concurrency: group: prod-deploy) are queued by Gitea; concurrent execution is prevented.
• Public ingress is limited to only 22, 80, and 443.
• prod/secrets/iklim.co/.env.secrets.swag exists on StorageBox and contains valid GoDaddy credentials.
• PROD_FLOATING_IP project variable is defined in Gitea.
• rabbitmq_erlang_cookie appears in docker secret ls.
• The ssl, swag/config, swag/site-confs, grafana/data, and precipitation/images directories exist on StorageBox; see 07-prod-ansible-bootstrap.md — StorageBox Directory Structure.
• The template/swag/site-confs/default.conf, api.conf.tpl, apigw.conf.tpl, rabbitmq.conf.tpl, and grafana.conf.tpl template files exist in the repo.
• StorageBox prod/secrets/iklim.co/.env has correct values for API_SUBDOMAIN, APIGW_SUBDOMAIN, RABBITMQ_SUBDOMAIN, GRAFANA_SUBDOMAIN, RESTRICTED_IPS, SWAG_CERT_DIR, SWAG_DNS_CONFIG_DIR, SWAG_SITE_CONFS_DIR, and SWAG_PROXY_CONFS_DIR.
• After the first deploy, docker exec $(docker ps -q -f name=iklimco_swag) nginx -t succeeds and returns syntax is ok.
• The output of cat /mnt/storagebox/swag/site-confs/api.conf | grep server_name contains server_name api.iklim.co;.
• The ssls/1 PUT block does not exist inside init/apisix-core/init.sh.
• The registry.tarla.io/iklimco/custom-apisix:3.12.0 image exists in Harbor and its config.yaml contains real_ip_header, real_ip_recursive, and set_real_ip_from (covering 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16) configuration.
• After the first deploy, real client IP appears in APISIX access logs, not the SWAG overlay IP: docker exec $(docker ps -q -f name=iklimco_apisix | head -1) tail -5 /usr/local/apisix/logs/access.log
• docker service ps iklimco_cert-reloader shows that the service is running.
• docker service ls contains the current prod infra services from docker-stack-infra_db-prod.yml and the separate iklimco_vault service from docker-stack-vault.yml; deprecated base-stack services such as iklimco_postgresql, iklimco_mongodb, iklimco_pg-proxy, and iklimco_mongo-proxy are not present.
• The output of docker service logs iklimco_cert-reloader --tail 20 contains [cert-reloader] started and has no error lines.
• The notAfter date of the Vault TLS endpoint certificate matches /mnt/storagebox/ssl/STAR.iklim.co.full.crt: docker exec $(docker ps -q -f name=iklimco_vault | head -1) sh -c 'echo | openssl s_client -connect vault.iklim.co:8200 2>/dev/null | openssl x509 -noout -dates'
• vault operator raft list-peers returns 3 peers: 1 leader, 2 followers.
• The vault_unseal_key Docker secret exists and appears in docker secret ls.
• 3 Vault containers are not sealed: docker exec $(docker ps -q -f name=iklimco_vault | head -1) vault status | grep Sealed -> Sealed false.
• The first deploy pipeline successfully completes all current root workflow steps; the Review Environment step succeeds.
• After the Bootstrap SWAG Certificate step, ls /mnt/storagebox/ssl/ -> STAR.iklim.co.full.crt and STAR.iklim.co_key.pem exist.
• The Run Database Init Scripts step completes without error; PostgreSQL and MongoDB are healthy and init scripts are applied.
• In the output of docker service ls --filter label=project=co.iklim, all infra services show X/X.
• docker volume inspect iklimco_image-data → Options.device=/mnt/storagebox/precipitation/images.
• docker exec $(docker ps -q -f name=iklimco_swag) certbot certificates -> *.iklim.co Let's Encrypt certificate is valid; it is not the old manual cert.
• echo | openssl s_client -connect api.iklim.co:443 2>/dev/null | openssl x509 -noout -subject -dates → CN=*.iklim.co, notAfter > 2026-07-15.
• curl -si https://api.iklim.co/health -> HTTP 2xx; no TLS error.
• https://grafana.iklim.co, https://apigw.iklim.co, https://rabbitmq.iklim.co — returns HTTP 403 from a disallowed IP and HTTP 200 from an allowed IP.
• curl --connect-timeout 5 https://<public-ip>:8200 -> connection refused or timeout; Vault is not reachable from outside.
• docker exec $(docker ps -q -f name=iklimco_apisix | head -1) curl -sk https://vault.iklim.co:8200/v1/sys/health -> {"sealed":false,...}; reachable from inside the overlay.
• docker service ls --format "{{.Name}}\t{{.Ports}}" --filter label=project=co.iklim -> only iklimco_swag publishes ports.
• docker service ps iklimco_apisix -> 3 tasks, Running, on different nodes.
• docker exec $(docker ps -q -f name=iklimco_swag) fail2ban-client status -> more than one jail appears.