2 commits

Author	SHA1	Message	Date
justin	7670608c1a	fix(monitoring): render alertmanager.yml from template at deploy (fixes crash loop) ... Alertmanager does not expand ${ENV} in its YAML, so the committed config with ${TELEGRAM_BOT_TOKEN}/${TELEGRAM_CHAT_ID} crash-looped it (line 24: cannot unmarshal !!str `${TELEG...` into int64) - 11k+ restarts on prod, alerting dead. - rename alertmanager.yml -> alertmanager.yml.template (keeps ${} placeholders) - deploy.sh: envsubst the template into the (gitignored) alertmanager.yml from .env, scoped to the two TELEGRAM vars so the {{ }} Go-template message survives - gitignore the rendered file (contains the bot token) - warns if the vars are unset	2026-06-07 04:49:53 -05:00
justin	a4a5500bfc	Add Prometheus + Grafana + Alertmanager monitoring stack ... Full observability stack with Telegram alerting: Components: - Prometheus: metrics collection, 90-day retention - Grafana: dashboards at monitoring.performancewest.net - Alertmanager: routes alerts to Telegram bot - node-exporter: OS metrics (CPU, RAM, disk, network) - cAdvisor: container metrics (CPU, memory, restarts) - postgres-exporter: PostgreSQL connection/query metrics - nginx-exporter: request rate, 5xx errors, connections - blackbox-exporter: HTTP/TCP endpoint probing + SSL cert checks Alert rules: - Service down (HTTP probe, TCP port, container missing) - Container restart loops - High CPU/memory/disk/load - PostgreSQL down or high connections - SSL cert expiring (14d warning, 3d critical) - Slow HTTP responses, high 5xx rate Blackbox probes all public endpoints: performancewest.net, api, dev, crm, lists, analytics, minio, crypto, pay Telegram alerts: critical=1h repeat, warning=6h repeat, auto-resolve notifications Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-01 02:08:39 -05:00