Commit graph

2 commits

Author SHA1 Message Date
justin
6b2cf5a07b infra: raise disk alert thresholds to 90% warn / 94% auto-clean
80% was too chatty for a box that idles ~72%. 90% warn / 94% reclaim still
leaves ~14GB headroom before the danger zone (Postgres needs free space to
checkpoint).
2026-06-27 10:00:15 -05:00
justin
e318f12e36 infra: disk-space guardrail + Docker log rotation (prevent disk-full crash)
On 2026-06-27 / filled to 100% and crash-looped Postgres ('No space left on
device'), taking down Listmonk mid-send. Cause: an orphaned 15GB
/tmp/forgejo-dump.zip (interrupted backup) + uncapped Docker json-file logs
(forgejo container log alone was ~1GB), with NO disk monitoring to warn first.

- pw-disk-space-alert.sh + cron (every 15m): Telegram warn at 80%, auto-reclaim
  build cache + orphaned forgejo dump at 88%. Silent when healthy.
- ansible docker role: write /etc/docker/daemon.json with 50m x 3 log cap
  (150MB/container max) + non-disruptive Reload docker handler.
2026-06-27 09:47:29 -05:00