Murat ÖZDEMİR
8875af8e8a
Remove setup runbook references from prod roadmap docs so roadmap remains design intent only. Keep setup-to-roadmap links, but normalize them to explicit relative paths.
97 lines
4.7 KiB
Markdown
97 lines
4.7 KiB
Markdown
# 08 — Production Deploy Pipeline Model
|
|
|
|
## Context
|
|
|
|
The production deploy pipeline is no longer a pending set of step additions. The current source of truth is the root `.gitea/workflows/deploy-prod.yml`, with the operational overview in root `prod_env-ci_dc-pipeline.md`.
|
|
|
|
## Current Pipeline Order
|
|
|
|
The current root production workflow runs in this order:
|
|
|
|
| # | Step | Note |
|
|
| --- | --- | --- |
|
|
| 1 | Checkout Branch | |
|
|
| 2 | Prepare Folders | |
|
|
| 3 | Set up SSH Key and Add to known_hosts | |
|
|
| 4 | Update Apt Repository and Install Required Tools | `gettext tree jq`; `jq` is required for the GoDaddy DNS API |
|
|
| 5 | Fetch Prod Env From Storagebox | Fetch `.env` and `.env.secrets.shared` |
|
|
| 6 | Fetch Service Secret Files | Fetch `.env.secrets.<svc>` and `.env.secrets.swag` |
|
|
| 7 | Prepare Database Init Files | Render PostgreSQL/MongoDB init templates |
|
|
| 8 | Docker Login to Harbor | |
|
|
| 9 | Prepare SWAG Directories | Render `dns-conf` and `site-confs`; reload node-local SWAG if present |
|
|
| 10 | Bootstrap Vault TLS Placeholder | Creates a temporary cert only if missing |
|
|
| 11 | Create Infrastructure Docker Secrets | Creates `rabbitmq_erlang_cookie` if missing |
|
|
| 12 | Deploy Swarm Stacks | Deploys `docker-stack-infra_db-prod.yml` |
|
|
| 13 | Connect Runner to Overlay Network | Connects the job container to `iklimco-net` |
|
|
| 14 | Initialize Production Infrastructure | Runs `init-infra-prod.sh`; this triggers Vault bootstrap and RabbitMQ setup |
|
|
| 15 | Wait for Infrastructure Services | Waits for `iklimco_vault` and `iklimco_rabbitmq` |
|
|
| 16 | Provision Vault AppRole IDs and Docker Secrets | Downloads service `vault-files`, runs `init/provision-all-services.sh` |
|
|
| 17 | Upload Updated Secrets to Storagebox | Uploads `.env.secrets.*` and `.env` |
|
|
| 18 | Wait for etcd | Waits for etcd health |
|
|
| 19 | Run APISIX Init | `SPRING_PROFILES_ACTIVE=prod` |
|
|
| 20 | Bootstrap SWAG Certificate | Waits for SWAG and cert-reloader output in `SWAG_CERT_DIR` |
|
|
| 21 | Initialize MongoDB Replica Set | Runs `rs.initiate()` or missing-member `rs.add()` |
|
|
| 22 | Run Database Init Scripts | Patroni primary + MongoDB replica set; SQL and JS init |
|
|
| 23 | Update DNS Records | GoDaddy API; `api`, `apigw`, `rabbitmq`, and `grafana` A records |
|
|
| 24 | Review Environment | |
|
|
|
|
All production deploy workflows must share `concurrency.group: prod-deploy` so infra and microservice deploys cannot overlap.
|
|
|
|
## Current Environment Files
|
|
|
|
The production StorageBox env file is `prod/secrets/iklim.co/.env`. The old `.env.prod` name is superseded.
|
|
|
|
Current SWAG-related variables include:
|
|
|
|
```bash
|
|
SWAG_CERT_DIR=/mnt/storagebox/ssl
|
|
SWAG_DNS_CONFIG_DIR=/mnt/storagebox/swag/dns-conf
|
|
SWAG_SITE_CONFS_DIR=/mnt/storagebox/swag/site-confs
|
|
SWAG_PROXY_CONFS_DIR=/mnt/storagebox/swag/proxy-confs
|
|
```
|
|
|
|
## Current Stack Deployment
|
|
|
|
The pipeline deploys the current production infra/DB stack:
|
|
|
|
```bash
|
|
docker stack deploy --with-registry-auth -c docker-stack-infra_db-prod.yml iklimco
|
|
```
|
|
|
|
Vault is not part of that stack. Vault is deployed and bootstrapped by `init/vault/vault-bootstrap.sh` through `init-infra-prod.sh` using:
|
|
|
|
```bash
|
|
docker stack deploy --with-registry-auth -c docker-stack-vault.yml iklimco
|
|
```
|
|
|
|
## Database Initialization
|
|
|
|
MongoDB replica set initialization is a dedicated workflow step. It runs `rs.initiate()` when the replica set is uninitialized and `rs.add()` when members from `DATABASE_MONGODB_HOST` are missing.
|
|
|
|
Database init scripts run after Patroni primary and MongoDB replica set readiness. PostgreSQL uses the multi-host Patroni connection with `target_session_attrs=read-write`; MongoDB uses the replica set host list from `DATABASE_MONGODB_HOST`.
|
|
|
|
## Microservice Deploy Model
|
|
|
|
Prod microservice workflows do not use a separate `docker-stack-service.prod.yml` overlay anymore.
|
|
|
|
The current model is:
|
|
|
|
- read `deploy/prod.env`;
|
|
- promote the tested Harbor digest to the stable prod tag;
|
|
- call `swarm_service_update` with `deploy/docker-stack-service.yml`;
|
|
- use `docker service update` with `--update-order start-first` and rollback behavior for existing services.
|
|
|
|
## Historical / Superseded by Setup
|
|
|
|
The following earlier plan items are superseded:
|
|
|
|
- Removing cert `scp` lines from an `Initialize Workspace` step as a live action; those lines are already gone.
|
|
- Creating prod deploy steps around `docker-stack-infra.yml` + `docker-stack-infra.prod.yml`.
|
|
- Waiting for a legacy `etcd:2379` service from a base stack.
|
|
- Using `docker-stack-db.prod.yml` as the DB stack reference.
|
|
- Writing SWAG DNS files through `SWAG_CONFIG_DIR/dns-conf`.
|
|
- Storing prod env in `prod/secrets/iklim.co/.env.prod`.
|
|
- Deploying microservices with `docker-stack-service.yml` plus `docker-stack-service.prod.yml`.
|
|
|
|
Keep this file as a roadmap summary. For exact commands, use the root workflow and root operational documentation.
|