5 Commits

Author SHA1 Message Date
03a3c5fb70 feat(monitoring): add BE microservice health monitoring with dedup and readable crash alerts
Adds two-layer BE-* microservice monitoring: an aggregate Stack Services replica check across the iklimco and monitoring Swarm stacks (checks/swarm_services.py), and per-service actuator/DNS health checks driven by a new microservice_monitors section in monitors.yml (checks/actuator.py). BE-* services are excluded from the Stack Services aggregate so a single service incident produces exactly one Slack alert instead of two.

Fixes found during QA of the initial implementation: RabbitMQ host derivation generated nonexistent numbered DNS names instead of using the real single-service topology, RabbitMQ and Patroni cluster checks did not push down status on node/member shortfall, and the etcd cluster check still used a hardcoded node list and quorum threshold instead of deriving them from CLUSTER_SIZE_ETCD.

Also hardens Docker event crash alerting: excludes ephemeral Gitea Actions runner containers from crash notifications, skips alerts for containers Swarm intentionally stopped (rolling update/scale-down, detected via task DesiredState) instead of alerting on every deploy, and adds human-readable exit code descriptions to the Slack message.

Uptime Kuma monitor names are now prefixed with "iklim [env]" since test and prod share one Kuma instance and unprefixed names collided. setup_uptime_kuma.py gained an ensure_push_monitor helper and a microservice push-monitor creation pass. Removed state.py and the unused restart_threshold config field (dead code, no longer referenced anywhere).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-10 17:31:11 +03:00
969c4a2301 fix(monitoring): resolve health-agent bugs and flapping monitors
- Vault flapping: Fix resp evaluation on HTTP 429
- Storagebox block: Move mount check to a daemon thread
- Push monitors: Increase interval to 75s and restore 60s sleep
- Redis Sentinel: Fix authentication in sentinel_kwargs
- Ext Https Api: Update URL to /health
2026-06-26 22:51:15 +03:00
bc8b3d0934 refactor: convert all monitor names to Title Case and update health-agent digest 2026-06-26 20:47:31 +03:00
72a91072fb feat(health-agent): add README, workflows, and translate monitors.yml to English
- Add health-agent README with architecture, config, and deployment docs
- Add deploy-monitoring-test.yml workflow (mirrors prod, test-runner, test storagebox paths)
- Add health-agent service to docker-stack-monitoring.yml
- Add .env.example with all runtime variables and .gitignore for generated files
- Add config/generated/.gitkeep to track empty generated directory
- Translate all Turkish group names and status page titles in monitors.yml to English
- Remove users.yml.example (Dozzle was removed in previous commit)
2026-06-25 19:20:25 +03:00
f742bfdd11 feat(health-agent): add monitors.yml with env-aware node IP mapping from Ansible inventory 2026-06-25 18:59:14 +03:00