Environment_Monitoring/common-functions-base.sh
Murat ÖZDEMİR 58d5c24f41
Some checks failed
Deploy Environment Monitoring to Production Environment / deploy (push) Failing after 10s
feat(health-agent): add CI/CD pipeline, Uptime Kuma setup, and runtime configuration
Deploy workflows:
- Integrate health-agent build (test) and image promotion (prod) into monitoring stack workflows
- Add storagebox download of health-agent runtime (.env.monitoring.health-agent-runtime → health-agent/.env) and setup (.env.monitoring.health-agent-setup → health-agent/.env.setup) env files
- Add "Run Uptime Kuma Setup" step: runs setup_uptime_kuma.py inside the built image only when uk_tokens.yml is missing, writes tokens to HEALTH_AGENT_CONFIG_GENERATED_DIR (/mnt/storagebox/monitoring/uk_generated)
- Add health-agent/** and health-agent/deploy/prod.env path triggers to test and prod workflows respectively
- Add HARBOR_CI_TOKEN login and HARBOR_PULL_TOKEN login before stack deploy in both workflows
- Source health-agent/.env before docker stack deploy to expose HEALTH_AGENT_CONFIG_GENERATED_DIR

Dockerfile:
- Copy config/ and scripts/ into image so setup_uptime_kuma.py can run inside the container

setup_uptime_kuma.py:
- Load .env and .env.setup automatically via python-dotenv (no manual export needed)
- Write uk_tokens.yml to config/generated/ (aligned with container volume mount)

Health checks:
- PATRONI_HOSTS and VAULT_HOSTS are now configurable via env vars (comma-separated host:port); no code change needed when node count changes
- REDIS_SENTINEL_HOSTS now correctly parses host:port format; default updated to redis-sentinel:26379
- Fix NameError in check_patroni_cluster() caused by leftover node variable after loop refactor
- Remove verify_ssl=False from Vault check; vault.iklim.co has a valid certificate

Ops:
- Add ops/build-and-push-health-agent.sh for manual bypass of CI pipeline
- Add health-agent/deploy/prod.env template for prod image promotion manifest

Project structure:
- Move .env.example and .env.setup.example to health-agent/env-example/ (root .gitignore excludes health-agent/.env*)
- Add root .gitignore: excludes uk_tokens.yml, __pycache__, .venv, and env files
- Remove health-agent/.gitignore (superseded by root .gitignore)
2026-06-26 18:45:17 +03:00

240 lines
7.2 KiB
Bash
Raw Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

#!/bin/bash
# ==============================================================================
# 🛠️ BASE UTILITY FUNCTIONS (iklim.co)
# ==============================================================================
# Tüm ortamlar (Dev, Test, Prod) tarafından ortak kullanılan çekirdek fonksiyonlar.
# Bu dosya doğrudan çalıştırılmaz, ortam scriptleri tarafından 'source' edilir.
# Belirtilen env dosyasını sisteme yükler (export eder).
source_env_file() {
local path="$1"
if [ -f "$path" ]; then
set -o allexport
source "$path"
set +o allexport
fi
}
# Klasördeki tüm .env.secrets.* dosyalarını otomatik bulur ve yükler.
# (.example ve .shared dosyalarını atlar, onları ana akış yönetir).
source_service_secret_files() {
local file
for file in .env.secrets.*; do
[ -f "$file" ] || continue
[[ "$file" == *.example ]] && continue
[ "$file" = ".env.secrets.shared" ] && continue
source_env_file "$file"
done
}
# Kritik bir env dosyasının varlığını kontrol eder, yoksa scripti durdurur.
require_env_file() {
local path="$1"
local description="$2"
if [ ! -f "$path" ]; then
log_message "ERROR" "$description not found at $path"
exit 1
fi
}
# Env dosyalarındaki değişken isimlerini tarayıp 'envsubst' için liste oluşturur.
# Template dosyalarını doldururken hangi değişkenlerin çözüleceğini belirler.
envsubst_vars_from_files() {
local file
for file in "$ENV_PATH" "$ENV_SECRETS_SHARED_PATH" .env.secrets.*; do
[ -f "$file" ] || continue
[[ "$file" == *.example ]] && continue
grep -E '^[A-Za-z_][A-Za-z_0-9]*=' "$file" | cut -d= -f1
done | sort -u | sed 's/^/\$/' | tr '\n' ' '
}
# Belirtilen bir değişkenin değerini hiyerarşik olarak (env -> shared -> secrets) arar.
lookup_env_value() {
local name="$1"
local file
local value=""
for file in "$ENV_PATH" "$ENV_SECRETS_SHARED_PATH" .env.secrets.*; do
[ -f "$file" ] || continue
[[ "$file" == *.example ]] && continue
if grep -q "^${name}=" "$file"; then
value="$(grep "^${name}=" "$file" | tail -n1 | cut -d '=' -f2-)"
fi
done
printf '%s' "$value"
}
# Matematiksel veya mantıksal işlem gerektiren env değerlerini hesaplar.
refresh_calculated_env_vars() {
}
# Tüm çevre dosyalarını (ana env, ortak sırlar ve servis sırları) tazeleyerek yükler.
refresh_env_vars() {
source_env_file "$ENV_PATH"
source_env_file "$ENV_SECRETS_SHARED_PATH"
source_service_secret_files
refresh_calculated_env_vars
log_message "INFO" "Environment variables refreshed from all .env and .env.secrets.* files 🔄"
}
# Bir değişkeni hem shell'e export eder hem de (Dev ortamında) terminale bilgi basar.
export_variable() {
local name="$1"
local value
if [ $# -ge 2 ]; then
value="$2"
else
log_message "DEBUG" "Looking for ${name} value in env files"
value="$(lookup_env_value "$name")"
fi
export "${name}=${value}"
if [[ "$ENVIRONMENT" == "dev" ]]; then
log_message "DEBUG" "Env variable ${name} is set to: ${value:0:5}... 🌎"
fi
}
# --- 🔐 Ortak Vault Yardımcıları ---
# Vault kilidini açar (Unseal). Dev, Test ve tek-node kurulumlar için VIP path kullanır.
# Prod HA Raft cluster için common-functions-prod.sh bu fonksiyonu override eder.
unseal_vault() {
local vault_addr=$1
local _curl="curl -s"
[[ "${VAULT_SKIP_VERIFY:-false}" == "true" ]] && _curl="curl -sk"
local RESPONSE_JSON ERROR_MSG sealed
RESPONSE_JSON=$($_curl $vault_addr/v1/sys/health)
ERROR_MSG=$(echo "$RESPONSE_JSON" | jq -r '.errors[]?')
if [[ -n "$ERROR_MSG" ]]; then
log_message "ERROR" "$ERROR_MSG"
exit 1
fi
sealed=$(echo $RESPONSE_JSON | jq .sealed)
if [ "$sealed" = "true" ]; then
log_message "INFO" "🔓🗝️ Unsealing Vault ($vault_addr)..."
RESPONSE_JSON=$($_curl --request PUT -H "Content-Type: application/json" \
--data "{\"key\": \"$VAULT_UNSEAL_KEY\"}" $vault_addr/v1/sys/unseal)
ERROR_MSG=$(echo "$RESPONSE_JSON" | jq -r '.errors[]?')
if [[ -n "$ERROR_MSG" ]]; then
log_message "ERROR" "$ERROR_MSG"
exit 1
fi
log_message "SUCCESS" "Vault unsealed successfully"
else
log_message "INFO" "Vault is already unsealed"
fi
}
# --- 📝 Ortak Log Yardımcıları ---
# Log seviyesini numerik değere çevirir
_get_log_level_num() {
case "${1^^}" in
TRACE) echo 1 ;;
DEBUG) echo 2 ;;
INFO) echo 3 ;;
SUCCESS) echo 4 ;;
WARN) echo 5 ;;
ERROR) echo 6 ;;
FATAL) echo 7 ;;
NONE) echo 99 ;;
*) echo 3 ;; # Default to INFO
esac
}
# Log seviyesine uygun emojiyi döndürür
_get_log_level_emoji() {
case "${1^^}" in
TRACE) echo "🔎" ;;
DEBUG) echo "🪲" ;;
INFO) echo "" ;;
SUCCESS) echo "✅" ;;
WARN) echo "⚠️" ;;
ERROR) echo "❌" ;;
FATAL) echo "☠️" ;;
*) echo "🤔" ;;
esac
}
# Timestamp'li log fonksiyonu. GLOBAL_LOG_LEVEL'a göre filtreler.
# Kullanım: log_message "INFO" "Bu bir log mesajıdır"
log_message() {
local level="${1^^}"
local message="$2"
local current_level_num=$(_get_log_level_num "$level")
local global_level_str="${GLOBAL_LOG_LEVEL:-INFO}"
local global_level_num=$(_get_log_level_num "$global_level_str")
local emoji=$(_get_log_level_emoji "$level")
local env_name="${SPRING_PROFILES_ACTIVE:-UNKNOWN}"
env_name="${env_name^^}"
if [ "$current_level_num" -ge "$global_level_num" ]; then
#local timestamp=$(TZ="Europe/Istanbul" date +"%Y-%m-%dT%H:%M:%S%:z")
local timestamp=$(TZ="Europe/Istanbul" date +"%H:%M:%S")
if [ "$current_level_num" -ge 5 ]; then
# ERROR ve FATAL hata akışına (stderr) basılır
echo "[$timestamp] [$env_name] $emoji [$level] $message" >&2
else
echo "[$timestamp] [$env_name] $emoji [$level] $message"
fi
fi
}
# Kayıtlı tüm log dosyalarının son satırlarını basar ve dosyaları temizler.
log_tail() {
for logfile in "${LOG_FILES[@]}"; do
tail -n 5 "$logfile"
rm -f "$logfile"
done
}
# Bir log dosyası oluşana kadar bekler ve başından (veya bir pattern'den) kesit basar.
log_head() {
local logfile=$1
local pattern=$2
local lines=5
while [ ! -f "$logfile" ] || [ "$(wc -l < "$logfile")" -lt $lines ]; do
sleep 0.5
done
if [ -z "$pattern" ]; then
head -n $lines "$logfile"
else
local start_line
start_line=$(grep -nm1 "$pattern" "$logfile" | cut -d: -f1)
if [ -z "$start_line" ]; then
head -n $lines "$logfile"
else
sed -n "${start_line},$((start_line + lines - 1))p" "$logfile"
fi
fi
}
# --- 🐳 Ortak Swarm Yardımcıları ---
# Docker Swarm üzerindeki bir servisi rolling-restart (güncelleme) yöntemiyle tazeler.
swarm_service_update() {
local stack="$1"
local service="$2"
local image="$3"
local full_name="${stack}_${service}"
if docker service inspect "$full_name" >/dev/null 2>&1; then
log_message "INFO" "🔄 Updating $full_name$image"
docker service update \
--image "$image" \
--with-registry-auth \
--update-order start-first \
--update-failure-action rollback \
"$full_name"
log_message "SUCCESS" "$full_name updated"
else
log_message "ERROR" "Service $full_name does not exist. Manual stack deploy required."
return 1
fi
}