feat(infra): Standardize StorageBox permissions and refactor DB stack name

- Ensure consistent directory and file permissions on StorageBox mounts for improved container access across application and database services.
- Introduce application-specific `storagebox_uid`/`gid` variables for more granular ownership control.
- Enhance StorageBox mount reliability by adding systemd reload and remount handlers for configuration changes.
- Add root credentials to Patroni's etcd configuration for authenticated communication.
- Update all relevant documentation and deployment scripts to use the `iklimco` Docker stack name for database services.
- Re-encrypt production vault secrets to include the new etcd password.
This commit is contained in:
Murat ÖZDEMİR 2026-05-23 18:11:01 +03:00
parent f23835a30a
commit 6f9d0d1588
16 changed files with 114 additions and 110 deletions

View File

@ -382,7 +382,7 @@ ansible-vault encrypt group_vars/all/vault.yml
Şifre çözme (düzenleme için):
```bash
ansible-vault edit group_vars/all/vault.yml
ansible-vault edit group_vars/all/vault.yml --vault-password-file=../.vault_pass
```
## Vault Kullanımı

View File

@ -4,16 +4,18 @@ storagebox_url: "https://{{ storagebox_user }}.your-storagebox.de/"
storagebox_mount_point: "/mnt/storagebox"
storagebox_password: "{{ vault_storagebox_password }}"
storagebox_managed_directories:
- path: "{{ storagebox_mount_point }}/db"
mode: "0777"
- path: "{{ storagebox_mount_point }}/ssl"
mode: "0755"
mode: "0777"
- path: "{{ storagebox_mount_point }}/swag/config"
mode: "0755"
mode: "0777"
- path: "{{ storagebox_mount_point }}/swag/site-confs"
mode: "0755"
- path: "{{ storagebox_mount_point }}/grafana/data"
mode: "0755"
mode: "0777"
- path: "{{ storagebox_managed_directories_grafana_path | default(storagebox_mount_point ~ '/grafana/data') }}"
mode: "0777"
- path: "{{ storagebox_mount_point }}/precipitation/images"
mode: "0755"
mode: "0777"
iklim_password: "{{ vault_iklim_password }}"
act_runner_labels: "prod-runner:docker://catthehacker/ubuntu:act-22.04,ubuntu-24.04,{{ inventory_hostname }}"

View File

@ -1,18 +1,21 @@
$ANSIBLE_VAULT;1.1;AES256
65346532343639643339393034653934623131356161666233303537643731346264313262613362
3034643838306533303631356537613438316430373733310a643766326231353065643263643039
31663634346663623137396237313663313332663666363437373935656235353530393735383434
3730343864333365390a313030386439653438386337666162623264333832363766306161323230
66613534326166323365656133376535623738633738353361363430336139643261326638393265
34376261613566343139346639363731353331333563333263396530376537646261336137636562
61633234646132313337343064353537623232613734636131316536333432393236363633366539
62633138346463316433613433343265313831643562366661313934306534663930333539363136
65386538346262306637626261323066366364346364316232663865356165383335626536353764
64346431313231643963383633326266313135653436363634623939373739326665663865366439
38636364363631303632363566323239336438303337323934353365653531383833363239323865
64656635626265313761636239356135326237383931623534633361373632613234313265613730
37396436656162656466386136316338316537343730623364353239346336313931663864623363
66663432363332393134386130643530653163343563353336666135313065383762666131633239
64353935376439313334326238373336653233386135333831383831643737356231313435323765
30353130333530356334653864376635636634653262333936396234653264323830333935616532
38393335306362323031643563643636393464636635633435373334393563656531
35633932313336623165666132313361616531343730333232386161653237373532393462303835
3465343432663961666662323261336166396263303638390a626234663263636431396561333365
33343736343831626539646564343436366264663564306137653331643133666136316133303165
3931303233666137660a363264623062656564363039313238623563643539313163383235633531
35353037616434663163643764633737636664393430613563353039626163366361336264653634
66663134626537346130643231646665346434313333343938353034643738323432653530373463
32636231343335663734366536646538366331323463373366323665306565663635393035323138
39616435303639303135393635363531613064636163326563353532633630333366623736663836
36366565336138326533343935643661393736393238353430333934323533323037613631306331
31353963623165333130633636323938313437666433366638626435333337613136663335393861
34656436313037353632363062326530383339626161623830316435393962306463653039623031
63376639646462306263393063383233376564643262666332366439353766386330633962323738
30383938346139636636363636643236323464386133643936373562383561633065373163356436
39623761626465643638663533306539663039666234366433333264363035393734623535343335
63353132663735336530643330646464343030663361613235376435303839333934373432666333
34653733333561633838323861636233623139353834303439646165653731303361376462333566
33653862356234353436383666613135353935366433623766613739343239356437353163393933
62653665643533326262666462326437313664363266306337333132343339326339306130363339
66386362633735656165393265363161313062386362643634343732336435666437666134643761
30613465306633303531

View File

@ -1,3 +1,5 @@
---
storagebox_uid: "1000" # SWAG kullanıcısı
storagebox_gid: "1000"
storagebox_dir_mode: "0777"
storagebox_file_mode: "0666"

View File

@ -1,4 +1,6 @@
# DB node'larında StorageBox uid/gid=999 (mongodb ve postgres container user)
# davfs2 dosyaları uid 999 sahibi gösterir; container içi erişim açılır.
storagebox_dir_mode: "0777"
storagebox_file_mode: "0666"
storagebox_uid: "999"
storagebox_gid: "999"

View File

@ -3,52 +3,41 @@
ansible.builtin.file:
path: "{{ storagebox_mount_point }}/db/mongodb-{{ inventory_hostname.split('-')[-1] }}/config"
state: directory
mode: '0755'
mode: '0777'
- name: Create StorageBox PostgreSQL config directory
ansible.builtin.file:
path: "{{ storagebox_mount_point }}/db/postgresql-{{ inventory_hostname.split('-')[-1] }}/config"
state: directory
mode: '0755'
- name: Sync StorageBox after directory creation
ansible.builtin.command: sync
mode: '0777'
- name: Deploy mongod.conf to StorageBox
ansible.builtin.template:
src: mongod.conf.j2
dest: "{{ storagebox_mount_point }}/db/mongodb-{{ inventory_hostname.split('-')[-1] }}/config/mongod.conf"
mode: '0644'
mode: '0666'
- name: Deploy patroni.yml to StorageBox
ansible.builtin.template:
src: patroni.yml.j2
dest: "{{ storagebox_mount_point }}/db/postgresql-{{ inventory_hostname.split('-')[-1] }}/config/patroni.yml"
mode: '0644'
- name: Sync StorageBox after config file writes
ansible.builtin.command: sync
mode: '0666'
- name: Generate MongoDB replica set keyfile on db-01
when: inventory_hostname == 'iklim-db-01'
ansible.builtin.shell: |
openssl rand -base64 756 > {{ storagebox_mount_point }}/db/mongodb-01/config/rs-auth.key
chmod 400 {{ storagebox_mount_point }}/db/mongodb-01/config/rs-auth.key
cp {{ storagebox_mount_point }}/db/mongodb-01/config/rs-auth.key \
{{ storagebox_mount_point }}/db/mongodb-02/config/rs-auth.key
cp {{ storagebox_mount_point }}/db/mongodb-01/config/rs-auth.key \
{{ storagebox_mount_point }}/db/mongodb-03/config/rs-auth.key
chmod 400 {{ storagebox_mount_point }}/db/mongodb-02/config/rs-auth.key
chmod 400 {{ storagebox_mount_point }}/db/mongodb-03/config/rs-auth.key
sync
args:
creates: "{{ storagebox_mount_point }}/db/mongodb-01/config/rs-auth.key"
- name: Wait for MongoDB keyfile on this node's StorageBox mount
ansible.builtin.wait_for:
path: "{{ storagebox_mount_point }}/db/mongodb-{{ inventory_hostname.split('-')[-1] }}/config/rs-auth.key"
timeout: 60
- name: Fix MongoDB keyfile permissions on this node
ansible.builtin.shell: |
chmod 400 {{ storagebox_mount_point }}/db/mongodb-{{ inventory_hostname.split('-')[-1] }}/config/rs-auth.key
ansible.builtin.file:
path: "{{ storagebox_mount_point }}/db/mongodb-{{ inventory_hostname.split('-')[-1] }}/config/rs-auth.key"
mode: '0400'
owner: "{{ storagebox_uid }}"
group: "{{ storagebox_gid }}"

View File

@ -14,6 +14,8 @@ etcd3:
- etcd-01:2379
- etcd-02:2379
- etcd-03:2379
username: root
password: "{{ vault_etcd_root_password }}"
bootstrap:
dcs:

View File

@ -19,6 +19,15 @@
- /opt/iklimco/vault/data
when: inventory_hostname in groups['app']
- name: Set vault data directory ownership (vault container runs as uid 100)
ansible.builtin.file:
path: /opt/iklimco/vault/data
state: directory
owner: '100'
group: '100'
mode: '0750'
when: inventory_hostname in groups['app']
- name: Create db specific directories
ansible.builtin.file:
path: "{{ item }}"

View File

@ -0,0 +1,10 @@
---
- name: Reload systemd
ansible.builtin.systemd:
daemon_reload: yes
- name: Remount storagebox
ansible.builtin.shell: |
umount {{ storagebox_mount_point }} || true
mount {{ storagebox_mount_point }}
listen: "refresh storagebox mount"

View File

@ -25,6 +25,9 @@
line: "{{ storagebox_url }} {{ storagebox_mount_point }} davfs _netdev,auto,user,rw,uid={{ storagebox_uid | default('root') }},gid={{ storagebox_gid | default('root') }}{% if storagebox_dir_mode is defined %},dir_mode={{ storagebox_dir_mode }}{% endif %}{% if storagebox_file_mode is defined %},file_mode={{ storagebox_file_mode }}{% endif %} 0 0"
regexp: "^{{ storagebox_url | regex_escape() }}"
state: present
notify:
- Reload systemd
- refresh storagebox mount
- name: Mount StorageBox
ansible.builtin.mount:
@ -33,6 +36,8 @@
fstype: davfs
opts: "_netdev,auto,user,rw,uid={{ storagebox_uid | default('root') }},gid={{ storagebox_gid | default('root') }}{% if storagebox_dir_mode is defined %},dir_mode={{ storagebox_dir_mode }}{% endif %}{% if storagebox_file_mode is defined %},file_mode={{ storagebox_file_mode }}{% endif %}"
state: mounted
notify:
- refresh storagebox mount
- name: Write mount marker
ansible.builtin.copy:
@ -48,3 +53,4 @@
group: "{{ item.group | default(omit) }}"
mode: "{{ item.mode | default('0755') }}"
loop: "{{ storagebox_managed_directories | default([]) }}"
notify: "refresh storagebox mount"

View File

@ -190,7 +190,7 @@ chmod 600 /tmp/.env
export $(grep -v '^\s*#' /tmp/.env.secrets.shared | grep -v '^\s*$' | xargs)
export $(grep -v '^\s*#' /tmp/.env | grep -v '^\s*$' | xargs)
docker stack deploy --with-registry-auth -c docker-stack-db.prod.yml iklim-db
docker stack deploy --with-registry-auth -c docker-stack-db-prod.yml iklimco
# deploy başarılı bir şekilde tamamlanınca
rm /tmp/.env
@ -207,14 +207,14 @@ ssh root@<db-01-ip>
scp -P 23 u469968@u469968.your-storagebox.de:prod/secrets/iklim.co/.env.secrets.shared \
/tmp/.env.secrets.shared
chmod 600 /tmp/.env.secrets.shared
scp -P 23 u469968@u469968.your-storagebox.de:prod/secrets/iklim.co/.env.secrets \
scp -P 23 u469968@u469968.your-storagebox.de:prod/secrets/iklim.co/.env \
/tmp/.env
chmod 600 /tmp/.env
export $(grep -v '^\s*#' /tmp/.env.secrets.shared | grep -v '^\s*$' | xargs)
export $(grep -v '^\s*#' /tmp/.env | grep -v '^\s*$' | xargs)
MONGO_CID=$(docker ps --filter name=iklim-db_mongodb-01 --format "{{.ID}}" | head -1)
MONGO_CID=$(docker ps --filter name=iklimco_mongodb-01 --format "{{.ID}}" | head -1)
docker exec -it $MONGO_CID mongosh \
-u "$DATABASE_MONGODB_ROOT_USER" \
-p "$DATABASE_MONGODB_ROOT_PASSWD" \
@ -228,6 +228,12 @@ rs.initiate({
]
})'
# bir süre sonra
docker exec -it $MONGO_CID mongosh \
-u "$DATABASE_MONGODB_ROOT_USER" \
-p "$DATABASE_MONGODB_ROOT_PASSWD" \
--authenticationDatabase admin --eval 'rs.status()'
rm /tmp/.env
rm /tmp/.env.secrets.shared
history -c && history -w

View File

@ -43,7 +43,7 @@ Prometheus and Grafana run as single instances, but their storage profiles are d
Grafana uses the `GRAFANA_DATA_DIR` env var with a named-volume fallback for test. Prometheus continues to use the named Docker volume. See Step 9 for implementation details.
**Note:** PostgreSQL and MongoDB are not in `docker-stack-infra.yml`. They run in separate stacks on DB nodes (`iklim-db` and `iklim-patroni`). See `08-prod-db-cluster-kurulum.md`.
**Note:** PostgreSQL and MongoDB are not in `docker-stack-infra.yml`. See `08-prod-db-cluster-kurulum.md`.
## Step 1 — Apply all test-env changes first
@ -622,27 +622,6 @@ services:
labels:
project: co.iklim
# ── Disabled in prod ─────────────────────────────────────────────────────────
etcd:
deploy:
replicas: 0
postgresql:
deploy:
replicas: 0
mongodb:
deploy:
replicas: 0
pg-proxy:
deploy:
replicas: 0
mongo-proxy:
deploy:
replicas: 0
secrets:
rabbitmq_erlang_cookie:
external: true
@ -706,7 +685,7 @@ In the production environment, the `pg-proxy` and `mongo-proxy` services (socat-
## Placement and Replica Summary — prod
| Service | File | Replicas | Placement | HA Note |
|---------|------|----------|-----------|---------|
| ---------------- | ------------ | -------- | ------------------------------------------- | ------------------------------------------------------------------------------------- |
| swag | base | 1 | `node.hostname == iklim-app-01` | No clustering support; Floating IP pinned to node |
| cert-reloader | base | 1 | `node.hostname == iklim-app-01` | Cron-style task; duplicate would be problematic |
| vault | prod overlay | 3 | `node.labels.type == service`; max 1/node | Raft cluster — see `07-vault-raft-plan.md` |
@ -716,14 +695,8 @@ In the production environment, the `pg-proxy` and `mongo-proxy` services (socat-
| redis-replica | prod overlay | 2 | `node.labels.type == service`; max 1/node | Sentinel replica; spread:hostname |
| redis-sentinel | prod overlay | 3 | `node.labels.type == service`; max 1/node | Quorum=2; failover automatic |
| rabbitmq | prod overlay | 3 | `node.labels.type == service`; max 1/node | Erlang cluster; quorum queues |
| etcd | prod overlay | 0 | — | Disabled (`replicas: 0`); APISIX uses Patroni etcd on DB nodes |
| postgresql | prod overlay | 0 | — | Disabled (`replicas: 0`); Patroni HA runs as `iklim-db` stack on DB nodes; port 5432 conflict |
| mongodb | prod overlay | 0 | — | Disabled (`replicas: 0`); MongoDB replica set runs as `iklim-db` stack on DB nodes; port 27017 conflict |
| pg-proxy | prod overlay | 0 | — | Deprecated; microservices use multi-host JDBC with native Patroni failover |
| mongo-proxy | prod overlay | 0 | — | Deprecated; microservices use multi-host MongoClient with native replica set failover |
| prometheus | base | 1 | `node.labels.type == service` | No native HA; Thanos is overkill at this scale |
| grafana | base | 1 | `node.labels.type == service` | Not critical |
> PostgreSQL and MongoDB run in separate DB stacks on `iklim-db-*` nodes. See `08-prod-db-cluster-kurulum.md`.
> PostgreSQL and MongoDB run in separate DB stacks on `iklimco-*` nodes. See `08-prod-db-cluster-kurulum.md`.
> etcd: 3-node cluster on DB nodes — APISIX shares it via `/apisix` prefix.
> Disabled services (`replicas: 0`) are removed from `docker service ls` by a post-deploy step in `deploy-prod.yml`.

View File

@ -115,7 +115,7 @@ APISIX reads its entire configuration from etcd; init script will fail silently
echo "⏳ Waiting for Patroni etcd..."
for i in $(seq 1 30); do
if docker run --rm --network iklimco-net alpine \
sh -c "wget -qO- http://etcd-01:2379/health 2>/dev/null | grep -q '\"health\":\"true\"'"; then
sh -c "wget -qO- http://etcd:2379/health 2>/dev/null | grep -q '\"health\":\"true\"'"; then
echo "✅ Patroni etcd ready"
break
fi
@ -125,7 +125,7 @@ APISIX reads its entire configuration from etcd; init script will fail silently
done
```
> **Note:** In prod, APISIX uses the 3-node Patroni etcd cluster on DB nodes (`etcd-01/02/03:2379`) via the `/apisix` prefix — resolved through `iklimco-net` overlay DNS aliases defined in `docker-stack-db.prod.yml`. The standalone `etcd` service from the base stack is disabled (`replicas: 0` in the prod overlay) and removed from the service list by a post-deploy step. This step waits for Patroni etcd (`etcd-01:2379`) to be healthy before running the APISIX init script.
> **Note:** In prod, APISIX uses the 3-node Patroni etcd cluster on DB nodes (`etcd/02/03:2379`) via the `/apisix` prefix — resolved through `iklimco-net` overlay DNS aliases defined in `docker-stack-db.prod.yml`. The standalone `etcd` service from the base stack is disabled (`replicas: 0` in the prod overlay) and removed from the service list by a post-deploy step. This step waits for Patroni etcd (`etcd:2379`) to be healthy before running the APISIX init script.
## Step 5 — Add `Run APISIX Init` step
@ -308,7 +308,7 @@ With `cancel-in-progress: false`, a new run waits in the queue until the previou
14. **Prepare SWAG Directories** ← NEW (`$SWAG_CONFIG_DIR/dns-conf`; renders nginx conf templates)
15. Bootstrap Vault TLS Placeholder
16. Deploy Swarm Stack
17. **Wait for etcd** ← NEW (Patroni etcd `etcd-01:2379` overlay DNS)
17. **Wait for etcd** ← NEW (Patroni etcd `etcd:2379` overlay DNS)
18. **Run APISIX Init** ← NEW (`SPRING_PROFILES_ACTIVE=prod`)
19. **Bootstrap SWAG Certificate** ← NEW
20. **Run Database Init Scripts** ← NEW (`postgresql`, `mongodb`)

View File

@ -110,10 +110,10 @@ docker service ps iklim-patroni_patroni-03
docker stack services iklim-etcd
# MongoDB replica set
docker stack services iklim-db
docker service ps iklim-db_mongodb-01
docker service ps iklim-db_mongodb-02
docker service ps iklim-db_mongodb-03
docker stack services iklimco
docker service ps iklimco_mongodb-01
docker service ps iklimco_mongodb-02
docker service ps iklimco_mongodb-03
```
All tasks should show node names matching `iklim-db-01`, `iklim-db-02`, or `iklim-db-03` with placement constraint `role=db`.

View File

@ -85,6 +85,6 @@ MongoDB logs are written to stdout and can be watched with `docker logs`. Config
## 5. Acceptance Criteria
- `iklim-db-01` appears as Ready and Active in the `docker node ls` command.
- `docker stack services iklim-db` shows both services with 1/1 replicas.
- `docker stack services iklimco` shows both services with 1/1 replicas.
- Access from the application node is available through the `iklim-db_postgresql` and `iklim-db_mongodb` DNS names.
- Data is preserved from named volumes after reboot; verify with `docker volume ls`.

View File

@ -486,7 +486,7 @@ Volumes `postgresql-01-data`, `postgresql-02-data`, `postgresql-03-data` are dec
```bash
# On iklim-app-01 — Patroni cluster status:
docker exec -it $(docker ps -q -f name=iklim-db_patroni-01 | head -1) \
docker exec -it $(docker ps -q -f name=iklimco_patroni-01 | head -1) \
patronictl -c /etc/patroni/patroni.yml list
```
@ -502,7 +502,7 @@ docker run --rm --network iklimco-net alpine \
```bash
# Find the current primary:
docker exec -it $(docker ps -q -f name=iklim-db_patroni-01 | head -1) \
docker exec -it $(docker ps -q -f name=iklimco_patroni-01 | head -1) \
patronictl -c /etc/patroni/patroni.yml topology
```
@ -528,7 +528,7 @@ set -a; . /tmp/.env.secrets.shared; set +a
# Automatic ETCD_INITIAL_CLUSTER_STATE detection:
DEPLOY_FILE="docker-stack-db.prod.yml"
if docker service ls --filter name=iklim-db_etcd-01 -q 2>/dev/null | grep -q .; then
if docker service ls --filter name=iklimco_etcd-01 -q 2>/dev/null | grep -q .; then
echo " etcd services mevcut, 'existing' ile deploy ediliyor..."
DEPLOY_FILE=$(mktemp /tmp/docker-stack-db.XXXXXX.yml)
sed "s/ETCD_INITIAL_CLUSTER_STATE: new/ETCD_INITIAL_CLUSTER_STATE: existing/g" \
@ -540,7 +540,7 @@ fi
docker stack deploy \
--with-registry-auth \
-c "$DEPLOY_FILE" \
iklim-db
iklimco
[ "$DEPLOY_FILE" != "docker-stack-db.prod.yml" ] && rm -f "$DEPLOY_FILE"
@ -557,15 +557,15 @@ for i in $(seq 1 18); do
sleep 10
done
docker stack services iklim-db
docker stack services iklimco
```
### DB Node Placement Check
```bash
docker service ps iklim-db_etcd-01
docker service ps iklim-db_mongodb-01
docker service ps iklim-db_patroni-01
docker service ps iklimco_etcd-01
docker service ps iklimco_mongodb-01
docker service ps iklimco_patroni-01
```
All tasks must run on the expected `iklim-db-*` nodes.
@ -592,7 +592,7 @@ rs.initiate({
## 7. Access from App Services
App containers connect to DB services through the `iklimco-net` overlay network by **overlay DNS name**. Because the `iklim-db` stack shares the `iklimco-net` external network, service names and aliases are resolved through overlay DNS.
App containers connect to DB services through the `iklimco-net` overlay network by **overlay DNS name**. Because the `iklimco` stack shares the `iklimco-net` external network, service names and aliases are resolved through overlay DNS.
### MongoDB Replica Set Connection String
@ -660,10 +660,10 @@ Modern veritabanı araçları (DBeaver, Compass vb.) küme farkındalıklı bağ
## Acceptance Criteria
- `docker stack services iklim-db` — 9 services visible (etcd-01/02/03, mongodb-01/02/03, patroni-01/02/03), all `1/1`
- `docker service ps iklim-db_patroni-01/02/03` — each task runs on its expected `iklim-db-*` node
- `docker service ps iklim-db_mongodb-01/02/03` — each task runs on its expected `iklim-db-*` node
- `docker service ps iklim-db_etcd-01/02/03` — each task runs on its expected `iklim-db-*` node
- `docker stack services iklimco` — 9 services visible (etcd-01/02/03, mongodb-01/02/03, patroni-01/02/03), all `1/1`
- `docker service ps iklimco_patroni-01/02/03` — each task runs on its expected `iklim-db-*` node
- `docker service ps iklimco_mongodb-01/02/03` — each task runs on its expected `iklim-db-*` node
- `docker service ps iklimco_etcd-01/02/03` — each task runs on its expected `iklim-db-*` node
- `patronictl list` — 1 `Leader`, 2 `Replica`, all `running`
- etcd health endpoint returns `"health":"true"` on all three nodes via overlay
- `rs.status()` — 1 PRIMARY, 2 SECONDARY