e318f12e36
On 2026-06-27 / filled to 100% and crash-looped Postgres ('No space left on
device'), taking down Listmonk mid-send. Cause: an orphaned 15GB
/tmp/forgejo-dump.zip (interrupted backup) + uncapped Docker json-file logs
(forgejo container log alone was ~1GB), with NO disk monitoring to warn first.
- pw-disk-space-alert.sh + cron (every 15m): Telegram warn at 80%, auto-reclaim
build cache + orphaned forgejo dump at 88%. Silent when healthy.
- ansible docker role: write /etc/docker/daemon.json with 50m x 3 log cap
(150MB/container max) + non-disruptive Reload docker handler.
4 lines
327 B
Text
4 lines
327 B
Text
# Disk-space guardrail: alert via Telegram before / fills up and crashes
|
|
# Postgres/Listmonk; auto-reclaims build cache + orphaned forgejo dump at CRIT.
|
|
# Every 15 minutes. Runs as root (needs du on /var/lib/docker + docker socket).
|
|
*/15 * * * * root /usr/local/bin/pw-disk-space-alert >> /var/log/pw-disk-space-alert.log 2>&1
|