diff --git a/roadmap/prod-env/03-infra-stack-changes.md b/roadmap/prod-env/03-infra-stack-changes.md index ef979f6..0a4ad2b 100644 --- a/roadmap/prod-env/03-infra-stack-changes.md +++ b/roadmap/prod-env/03-infra-stack-changes.md @@ -371,7 +371,59 @@ docker exec "$CTR" rabbitmqctl cluster_status > '{"queue-type":"quorum"}' --apply-to queues > ``` -## Step 7 — Create `docker-stack-infra.prod.yml` +## Step 7 — RabbitMQ WebSocket Sticky Sessions (Consistent Hash) + +RabbitMQ Web STOMP (over WebSocket) requires a persistent connection. In a 3-node RabbitMQ cluster, if an APISIX instance uses the default Swarm VIP for the `rabbitmq` upstream, it may cause unnecessary inter-node traffic or connection drops if the session doesn't persist on the same node. + +To optimize this, we implement **Consistent Hashing (chash)** at the APISIX layer based on the client's IP address (`remote_addr`). + +### 1. Update APISIX Upstream Configuration (init.sh) + +Update the `rabbitmq` upstream definition in `init/apisix-core/init.sh` to target specific cluster nodes instead of the generic service name, enabling the `chash` algorithm for prod. + +```bash +# Update upstream rabbitmq block in init.sh +if [[ "$PROFILE" == "prod" ]]; then + # Direct node DNS names to bypass Swarm VIP and allow chash to work effectively + RABBITMQ_NODES='{"rabbitmq-iklim-app-01:15674":1, "rabbitmq-iklim-app-02:15674":1, "rabbitmq-iklim-app-03:15674":1}' + LB_TYPE="chash" + HASH_KEY="remote_addr" +else + RABBITMQ_NODES='{"rabbitmq:15674":1}' + LB_TYPE="roundrobin" + HASH_KEY="" +fi + +call_api "upstream rabbitmq" -X PUT "$APISIX_ADMIN_URL/upstreams/rabbitmq-upstream" \ + -H "X-API-KEY: $API_KEY" -H "Content-Type: application/json" \ + -d '{ + "name": "rabbitmq-upstream", + "type": "'"$LB_TYPE"'", + "key": "'"$HASH_KEY"'", + "nodes": '"$RABBITMQ_NODES"', + "timeout": {"connect": 10, "send": 3600, "read": 3600}, + "scheme": "http", + '"$HC"' + }' +``` + +### 2. Enable Real IP Detection in APISIX + +Consistent hashing by `remote_addr` requires APISIX to see the actual client IP, not the internal IP of the SWAG (Nginx) proxy. + +Update `template/apisix-core/config.yaml.template`: + +```yaml +nginx_config: + http: + real_ip_header: "X-Forwarded-For" + real_ip_from: + - 10.0.0.0/8 + - 172.16.0.0/12 + - 192.168.0.0/16 +``` + +## Step 8 — Create `docker-stack-infra.prod.yml` Create this file in the repo root alongside `docker-stack-infra.yml`. It combines all prod-specific overrides from Steps 2–6: