On 2026-06-27 / filled to 100% and crash-looped Postgres ('No space left on
device'), taking down Listmonk mid-send. Cause: an orphaned 15GB
/tmp/forgejo-dump.zip (interrupted backup) + uncapped Docker json-file logs
(forgejo container log alone was ~1GB), with NO disk monitoring to warn first.
- pw-disk-space-alert.sh + cron (every 15m): Telegram warn at 80%, auto-reclaim
build cache + orphaned forgejo dump at 88%. Silent when healthy.
- ansible docker role: write /etc/docker/daemon.json with 50m x 3 log cap
(150MB/container max) + non-disruptive Reload docker handler.
Three bugs the owner hit:
1. Per-operator reputation alert (06:10 cron, mail_reputation_monitor --alert)
silently never ran: it redirected to /var/log/pw-mail-reputation.log but
/var/log is root-only and that file was never pre-created, so the deploy
user's >> redirect failed and cron aborted before the command. Repointed
both mail-alert crons to deploy-writable /opt/performancewest/logs/.
2. IP reputation alert (20:00 cron) still referenced the removed rehab pool
(.91-.93) and used 8.8.8.8 for Spamhaus (which returns the open-resolver
error, not a real answer). Dropped the rehab section, relabeled to the two
live IPs (.94/.107), and switched the DNSBL check to Control D (76.76.2.0)
which returns real Spamhaus ZEN data. (It was correctly SILENT lately
because delivery is healthy -- silent-on-healthy is by design.)
3. DMARC daily digest was pure noise: it alerted on ANY external IP with >=20
failing msgs, but those are legit recipient-side forwarders/security
gateways (inkyphishfence, cloud-sec-av, Proofpoint, Mimecast, ...) that
re-send our mail and naturally break SPF/DKIM alignment -- benign under
p=reject. Added PTR-based forwarder detection (FORWARDER_PTR_HINTS) so the
digest tags them [fwd] and only alerts on (a) OUR IP <95% pass or (b) an
UNKNOWN non-forwarder external IP with >=100 failing msgs = real spoofing.
Verified: all 4 currently-flagged external IPs now classify as forwarder=True.
Tracks the rehab pool (rehab02-04 / .91-.93) delivery + bounce + Spamhaus ZEN
DNSBL status in the daily report and alert body. Alerts only if a rehab IP lands
on a DNSBL or rehab delivery drops <40% with real volume (recipient quality
slipped) -- a recovering IP naturally bounces more so the threshold is lenient.
End-of-day (20:00 Central) check of campaign deliverability across both sending
pools (main out05-09 + healthcare hcout). Sends a Telegram alert ONLY when there
is a reputation problem -- delivery below 65% or a spam/policy-block (550-5.7.1)
spike above 150/day -- so healthy days stay silent. Reuses the existing
TELEGRAM_BOT_TOKEN/CHAT_ID from /opt/performancewest/.env. Logs every run to
/var/log/pw-warmup-healthcheck.log for history. Excludes internal/probe noise so
the delivery figure reflects real external recipients.
