# 08 — Production Deploy Pipeline Model

## Context

The production deploy pipeline is no longer a pending set of step additions. The current source of truth is the root `.gitea/workflows/deploy-prod.yml`, with the operational overview in root `prod_env-ci_dc-pipeline.md`.

## Current Pipeline Order

The current root production workflow runs in this order:

| # | Step | Note |
| --- | --- | --- |
| 1 | Checkout Branch | |
| 2 | Prepare Folders | |
| 3 | Set up SSH Key and Add to known_hosts | |
| 4 | Update Apt Repository and Install Required Tools | `gettext tree jq`; `jq` is required for the GoDaddy DNS API |
| 5 | Fetch Prod Env From Storagebox | Fetch `.env` and `.env.secrets.shared` |
| 6 | Fetch Service Secret Files | Fetch `.env.secrets.<svc>` and `.env.secrets.swag` |
| 7 | Prepare Database Init Files | Render PostgreSQL/MongoDB init templates |
| 8 | Docker Login to Harbor | |
| 9 | Prepare SWAG Directories | Render `dns-conf` and `site-confs`; reload node-local SWAG if present |
| 10 | Bootstrap Vault TLS Placeholder | Creates a temporary cert only if missing |
| 11 | Create Infrastructure Docker Secrets | Creates `rabbitmq_erlang_cookie` if missing |
| 12 | Deploy Swarm Stacks | Deploys `docker-stack-infra_db-prod.yml` |
| 13 | Connect Runner to Overlay Network | Connects the job container to `iklimco-net` |
| 14 | Initialize Production Infrastructure | Runs `init-infra-prod.sh`; this triggers Vault bootstrap and RabbitMQ setup |
| 15 | Wait for Infrastructure Services | Waits for `iklimco_vault` and `iklimco_rabbitmq` |
| 16 | Provision Vault AppRole IDs and Docker Secrets | Downloads service `vault-files`, runs `init/provision-all-services.sh` |
| 17 | Upload Updated Secrets to Storagebox | Uploads `.env.secrets.*` and `.env` |
| 18 | Wait for etcd | Waits for etcd health |
| 19 | Run APISIX Init | `SPRING_PROFILES_ACTIVE=prod` |
| 20 | Bootstrap SWAG Certificate | Waits for SWAG and cert-reloader output in `SWAG_CERT_DIR` |
| 21 | Initialize MongoDB Replica Set | Runs `rs.initiate()` or missing-member `rs.add()` |
| 22 | Run Database Init Scripts | Patroni primary + MongoDB replica set; SQL and JS init |
| 23 | Update DNS Records | GoDaddy API; `api`, `apigw`, `rabbitmq`, and `grafana` A records |
| 24 | Review Environment | |

All production deploy workflows must share `concurrency.group: prod-deploy` so infra and microservice deploys cannot overlap.

## Current Environment Files

The production StorageBox env file is `prod/secrets/iklim.co/.env`. The old `.env.prod` name is superseded.

Current SWAG-related variables include:

```bash
SWAG_CERT_DIR=/mnt/storagebox/ssl
SWAG_DNS_CONFIG_DIR=/mnt/storagebox/swag/dns-conf
SWAG_SITE_CONFS_DIR=/mnt/storagebox/swag/site-confs
SWAG_PROXY_CONFS_DIR=/mnt/storagebox/swag/proxy-confs
```

## Current Stack Deployment

The pipeline deploys the current production infra/DB stack:

```bash
docker stack deploy --with-registry-auth -c docker-stack-infra_db-prod.yml iklimco
```

Vault is not part of that stack. Vault is deployed and bootstrapped by `init/vault/vault-bootstrap.sh` through `init-infra-prod.sh` using:

```bash
docker stack deploy --with-registry-auth -c docker-stack-vault.yml iklimco
```

## Database Initialization

MongoDB replica set initialization is a dedicated workflow step. It runs `rs.initiate()` when the replica set is uninitialized and `rs.add()` when members from `DATABASE_MONGODB_HOST` are missing.

Database init scripts run after Patroni primary and MongoDB replica set readiness. PostgreSQL uses the multi-host Patroni connection with `target_session_attrs=read-write`; MongoDB uses the replica set host list from `DATABASE_MONGODB_HOST`.

## Microservice Deploy Model

Prod microservice workflows do not use a separate `docker-stack-service.prod.yml` overlay anymore.

The current model is:

- read `deploy/prod.env`;
- promote the tested Harbor digest to the stable prod tag;
- call `swarm_service_update` with `deploy/docker-stack-service.yml`;
- use `docker service update` with `--update-order start-first` and rollback behavior for existing services.

## Historical / Superseded by Setup

The following earlier plan items are superseded:

- Removing cert `scp` lines from an `Initialize Workspace` step as a live action; those lines are already gone.
- Creating prod deploy steps around `docker-stack-infra.yml` + `docker-stack-infra.prod.yml`.
- Waiting for a legacy `etcd:2379` service from a base stack.
- Using `docker-stack-db.prod.yml` as the DB stack reference.
- Writing SWAG DNS files through `SWAG_CONFIG_DIR/dns-conf`.
- Storing prod env in `prod/secrets/iklim.co/.env.prod`.
- Deploying microservices with `docker-stack-service.yml` plus `docker-stack-service.prod.yml`.

Keep this file as a roadmap summary. For exact commands, use the root workflow and root operational documentation.