Environment_Infrastructure/roadmap/prod-env/08-deploy-pipeline-update.md

08 — Production Deploy Pipeline Model

Context

The production deploy pipeline is no longer a pending set of step additions. The current source of truth is the root .gitea/workflows/deploy-prod.yml, with the operational explanation in ../../setup/09-prod-runner-ha-and-swarm.md and root prod_env-ci_dc-pipeline.md.

Current Pipeline Order

The current root production workflow runs in this order:

#	Step	Note
1	Checkout Branch
2	Prepare Folders
3	Set up SSH Key and Add to known_hosts
4	Update Apt Repository and Install Required Tools	gettext tree jq; jq is required for the GoDaddy DNS API
5	Fetch Prod Env From Storagebox	Fetch .env and .env.secrets.shared
6	Fetch Service Secret Files	Fetch .env.secrets.<svc> and .env.secrets.swag
7	Prepare Database Init Files	Render PostgreSQL/MongoDB init templates
8	Docker Login to Harbor
9	Prepare SWAG Directories	Render dns-conf and site-confs; reload node-local SWAG if present
10	Bootstrap Vault TLS Placeholder	Creates a temporary cert only if missing
11	Create Infrastructure Docker Secrets	Creates rabbitmq_erlang_cookie if missing
12	Deploy Swarm Stacks	Deploys docker-stack-infra_db-prod.yml
13	Connect Runner to Overlay Network	Connects the job container to iklimco-net
14	Initialize Production Infrastructure	Runs init-infra-prod.sh; this triggers Vault bootstrap and RabbitMQ setup
15	Wait for Infrastructure Services	Waits for iklimco_vault and iklimco_rabbitmq
16	Provision Vault AppRole IDs and Docker Secrets	Downloads service vault-files, runs init/provision-all-services.sh
17	Upload Updated Secrets to Storagebox	Uploads .env.secrets.* and .env
18	Wait for etcd	Waits for etcd health
19	Run APISIX Init	SPRING_PROFILES_ACTIVE=prod
20	Bootstrap SWAG Certificate	Waits for SWAG and cert-reloader output in SWAG_CERT_DIR
21	Initialize MongoDB Replica Set	Runs rs.initiate() or missing-member rs.add()
22	Run Database Init Scripts	Patroni primary + MongoDB replica set; SQL and JS init
23	Update DNS Records	GoDaddy API; api, apigw, rabbitmq, and grafana A records
24	Review Environment

All production deploy workflows must share concurrency.group: prod-deploy so infra and microservice deploys cannot overlap.

Current Environment Files

The production StorageBox env file is prod/secrets/iklim.co/.env. The old .env.prod name is superseded.

Current SWAG-related variables include:

SWAG_CERT_DIR=/mnt/storagebox/ssl
SWAG_DNS_CONFIG_DIR=/mnt/storagebox/swag/dns-conf
SWAG_SITE_CONFS_DIR=/mnt/storagebox/swag/site-confs
SWAG_PROXY_CONFS_DIR=/mnt/storagebox/swag/proxy-confs

Current Stack Deployment

The pipeline deploys the current production infra/DB stack:

docker stack deploy --with-registry-auth -c docker-stack-infra_db-prod.yml iklimco

Vault is not part of that stack. Vault is deployed and bootstrapped by init/vault/vault-bootstrap.sh through init-infra-prod.sh using:

docker stack deploy --with-registry-auth -c docker-stack-vault.yml iklimco

Database Initialization

MongoDB replica set initialization is a dedicated workflow step. It runs rs.initiate() when the replica set is uninitialized and rs.add() when members from DATABASE_MONGODB_HOST are missing.

Database init scripts run after Patroni primary and MongoDB replica set readiness. PostgreSQL uses the multi-host Patroni connection with target_session_attrs=read-write; MongoDB uses the replica set host list from DATABASE_MONGODB_HOST.

Microservice Deploy Model

Prod microservice workflows do not use a separate docker-stack-service.prod.yml overlay anymore.

The current model is:

• read deploy/prod.env;
• promote the tested Harbor digest to the stable prod tag;
• call swarm_service_update with deploy/docker-stack-service.yml;
• use docker service update with --update-order start-first and rollback behavior for existing services.

Historical / Superseded by Setup

The following earlier plan items are superseded:

• Removing cert scp lines from an Initialize Workspace step as a live action; those lines are already gone.
• Creating prod deploy steps around docker-stack-infra.yml + docker-stack-infra.prod.yml.
• Waiting for a legacy etcd:2379 service from a base stack.
• Using docker-stack-db.prod.yml as the DB stack reference.
• Writing SWAG DNS files through SWAG_CONFIG_DIR/dns-conf.
• Storing prod env in prod/secrets/iklim.co/.env.prod.
• Deploying microservices with docker-stack-service.yml plus docker-stack-service.prod.yml.

Keep this file as a roadmap summary. For exact commands, use the root workflow and ../../setup/09-prod-runner-ha-and-swarm.md.