Environment_Infrastructure/roadmap/test-env/07-deploy-pipeline-update.md

# 07 — Deploy Pipeline Update (Test)

## Context
- **File:** `.gitea/workflows/deploy-test.yml`
- Changes:
  1. Remove manual `scp STAR.iklim.co.full.crt` steps (SWAG now owns cert lifecycle).
  2. Add SWAG host directories preparation (dns-conf, nginx site-confs).
  3. Add cert bootstrap step: on first deploy, wait for SWAG to obtain cert, then copy
     to `/opt/iklimco/ssl/` so Vault can start.
  4. Ensure `GODADDY_KEY` and `GODADDY_SECRET` are available from `.env.secrets.swag`.

## Step 1 — Update `Initialize Servers` step

**Remove** the two `scp` lines that copy the TLS cert files:

```yaml
# DELETE these two lines from the "Initialize Servers" step:
          scp -P 23 ${{ vars.STORAGEBOX_USER }}@${{ vars.STORAGEBOX_USER }}.your-storagebox.de:test/app/iklim.co/ssl/STAR.iklim.co.full.crt ./STAR.iklim.co.full.crt
          scp -P 23 ${{ vars.STORAGEBOX_USER }}@${{ vars.STORAGEBOX_USER }}.your-storagebox.de:test/app/iklim.co/ssl/STAR.iklim.co_key.pem ./STAR.iklim.co_key.pem
```

Also remove any references to `STAR.iklim.co.full.crt` and `STAR.iklim.co_key.pem` in
the `Prepare Init Files` step's `sudo cp` commands:

```yaml
# DELETE or make conditional:
          sudo cp STAR.iklim.co.full.crt STAR.iklim.co_key.pem /opt/iklimco/ssl/ 2>/dev/null || true
```

## Step 2 — Add `Prepare SWAG Directories` step

Insert this step **before** `Deploy Swarm Stack`:

```yaml
      - name: Prepare SWAG Directories
        run: |
          set -a; . ./.env; . ./.env.secrets.swag; set +a

          docker run --rm -v /opt/iklimco/swag:/output alpine \
            mkdir -p /output/dns-conf /output/site-confs

          envsubst < swag/dns-conf/godaddy.ini.tpl | docker run --rm -i \
            -v /opt/iklimco/swag/dns-conf:/output \
            alpine sh -c "cat > /output/godaddy.ini && chmod 600 /output/godaddy.ini"
          echo "✅ godaddy.ini written"

          # RESTRICTED_IPS → multi-line allow block (indented 8 spaces per nginx style)
          export RESTRICTED_IPS_BLOCK="$(echo "$RESTRICTED_IPS" | tr ',' '\n' | sed 's|.*|        allow &;|')"

          # Explicit var list prevents nginx $upstream_* from being substituted by envsubst
          SWAG_VARS='${API_SUBDOMAIN}${APIGW_SUBDOMAIN}${GRAFANA_SUBDOMAIN}${RABBITMQ_SUBDOMAIN}${RESTRICTED_IPS_BLOCK}'
          for tpl in swag/site-confs/*.conf.tpl; do
            fname=$(basename "${tpl%.tpl}")
            envsubst "$SWAG_VARS" < "$tpl" | docker run --rm -i \
              -v /opt/iklimco/swag/site-confs:/output \
              alpine sh -c "cat > /output/${fname}"
            echo "✅ ${fname}"
          done

          cat swag/site-confs/default.conf | docker run --rm -i \
            -v /opt/iklimco/swag/site-confs:/output \
            alpine sh -c "cat > /output/default.conf"

          echo "✅ SWAG directories ready"

          SWAG_CTR=$(docker ps -q -f name=iklimco_swag 2>/dev/null | head -1)
          if [ -n "$SWAG_CTR" ]; then
            docker exec "$SWAG_CTR" nginx -t && docker exec "$SWAG_CTR" nginx -s reload
            echo "✅ SWAG nginx reloaded"
          fi
        working-directory: /workspace/iklim.co
```

> `GODADDY_KEY` ve `GODADDY_SECRET` `.env.secrets.swag` içinde olmalı (bkz. step 02).
> `API_SUBDOMAIN`, `APIGW_SUBDOMAIN` vb. `.env` içinde olmalı (bkz. step 04).
> Dosyalar `docker run alpine` ile yazılır — host'a `sudo` erişimi gerekmez.
> `RESTRICTED_IPS` comma-separated CIDR listesinden (`env-test/.env`) her satıra `allow` direktifi üretilir.

## Step 3 — Add `Bootstrap SWAG Certificate` step

Insert this step **after** `Deploy Swarm Stack` and **before** any step that depends on
Vault being accessible (e.g., `Provision Vault AppRole IDs`):

```yaml
      - name: Bootstrap SWAG Certificate
        run: |
          echo "Waiting for SWAG container to start..."
          SWAG_CTR=""
          for i in $(seq 1 24); do
            SWAG_CTR=$(docker ps -q -f name=iklimco_swag 2>/dev/null | head -1)
            [ -n "$SWAG_CTR" ] && break
            sleep 10
          done

          if [ -z "$SWAG_CTR" ]; then
            echo "❌ SWAG container did not start in time"
            exit 1
          fi

          CERT_PATH="/config/etc/letsencrypt/live/iklim.co/fullchain.pem"
          echo "Waiting for SWAG to obtain Let's Encrypt cert (up to 10 min)..."
          for i in $(seq 1 20); do
            if docker exec "$SWAG_CTR" test -f "$CERT_PATH" 2>/dev/null; then
              echo "✅ Cert obtained by SWAG"
              break
            fi
            echo "  attempt $i/20 — waiting 30s..."
            sleep 30
          done

          if ! docker exec "$SWAG_CTR" test -f "$CERT_PATH" 2>/dev/null; then
            echo "❌ SWAG did not obtain cert in time. Check logs:"
            docker service logs iklimco_swag --tail 50
            exit 1
          fi

          # Copy cert to host for Vault bootstrap
          sudo mkdir -p /opt/iklimco/ssl
          docker exec "$SWAG_CTR" cat "$CERT_PATH" | \
            sudo tee /opt/iklimco/ssl/STAR.iklim.co.full.crt > /dev/null
          docker exec "$SWAG_CTR" cat "/config/etc/letsencrypt/live/iklim.co/privkey.pem" | \
            sudo tee /opt/iklimco/ssl/STAR.iklim.co_key.pem > /dev/null
          echo "✅ Cert bootstrapped to /opt/iklimco/ssl/"
        working-directory: /workspace/iklim.co
```

> **First deploy only:** SWAG contacts Let's Encrypt via GoDaddy DNS challenge.
> This step waits up to 10 minutes. On subsequent deploys the cert is already in
> `swag-vl` (persisted volume) and SWAG starts immediately — wait loop exits fast.

## Step 4 — Re-order steps

Final step order in the pipeline:

1. Checkout Branch
2. Prepare Folders
3. Set up SSH Key
4. Update Apt / Install Tools
5. Fetch Service Secret Files
6. Initialize Docker Swarm
7. Initialize Servers
8. Upload Updated Secrets to Storagebox
9. Provision Vault AppRole IDs and Docker Secrets
10. Upload Updated Env to Storagebox
11. Prepare Init Files ← `sudo cp STAR.iklim.co.*.crt` lines removed
12. Stop Docker Compose Services
13. Docker Login to Harbor
14. **Prepare SWAG Directories** ← NEW
15. Bootstrap Vault TLS Placeholder
16. Deploy Swarm Stack
17. **Wait for etcd** ← zaten pipeline'da mevcut; etcd health endpoint'i kontrol eder
18. **Run APISIX Init** ← NEW (`SPRING_PROFILES_ACTIVE=test`)
19. **Bootstrap SWAG Certificate** ← NEW
20. **Run Database Init Scripts** ← NEW
21. Review Environment

> Steps 8 (Provision Vault) runs before SWAG because it creates Docker secrets and
> AppRole IDs — Vault must be reachable for this. On re-deploys, Vault is already
> running with the previous cert. On first deploy, step 16 handles the cert wait before
> any further Vault interaction is needed post-deploy.
>
> If Vault provisioning (step 8) fails on first deploy because Vault has no cert yet,
> move step 16 before step 8. Adjust based on observed behavior.

## Notes
- `.env` must contain the subdomain env vars added in step 04. Add them to storagebox
  `test/secrets/iklim.co/.env` before the first deploy.
- `RESTRICTED_IP_1` and `RESTRICTED_IP_2` are hardcoded in the pipeline step above.
  Move to `.env` if they change often.
- Precipitation service expects its image-data bind mount at
  `/mnt/storagebox/precipitation/images`. This directory is provisioned by the
  test Ansible bootstrap through `storagebox_managed_directories`; do not rely on
  the deploy pipeline to create it.