justin/new-site
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(monitoring): repair both dead mail-alert crons + de-noise DMARC digest

Browse source 
Three bugs the owner hit:
1. Per-operator reputation alert (06:10 cron, mail_reputation_monitor --alert)
   silently never ran: it redirected to /var/log/pw-mail-reputation.log but
   /var/log is root-only and that file was never pre-created, so the deploy
   user's >> redirect failed and cron aborted before the command. Repointed
   both mail-alert crons to deploy-writable /opt/performancewest/logs/.
2. IP reputation alert (20:00 cron) still referenced the removed rehab pool
   (.91-.93) and used 8.8.8.8 for Spamhaus (which returns the open-resolver
   error, not a real answer). Dropped the rehab section, relabeled to the two
   live IPs (.94/.107), and switched the DNSBL check to Control D (76.76.2.0)
   which returns real Spamhaus ZEN data. (It was correctly SILENT lately
   because delivery is healthy -- silent-on-healthy is by design.)
3. DMARC daily digest was pure noise: it alerted on ANY external IP with >=20
   failing msgs, but those are legit recipient-side forwarders/security
   gateways (inkyphishfence, cloud-sec-av, Proofpoint, Mimecast, ...) that
   re-send our mail and naturally break SPF/DKIM alignment -- benign under
   p=reject. Added PTR-based forwarder detection (FORWARDER_PTR_HINTS) so the
   digest tags them [fwd] and only alerts on (a) OUR IP <95% pass or (b) an
   UNKNOWN non-forwarder external IP with >=100 failing msgs = real spoofing.

Verified: all 4 currently-flagged external IPs now classify as forwarder=True.
This commit is contained in:
justin 2026-06-24 06:28:50 -05:00
parent c20edb28cd
commit ae68edbc58
4 changed files with 74 additions and 31 deletions
Download patch file Download diff file Expand all files Collapse all files

42
infra/monitoring/pw-warmup-tg-alert.sh
View file

@ -4,7 +4,7 @@
set -uo pipefail
LOG=/var/log/mail.log
TODAY=$(date '+%b %d')
REPORT=/var/log/pw-warmup-healthcheck.log
REPORT=/opt/performancewest/logs/pw-warmup-healthcheck.log
MIN_DELIVERY=65
MAX_SPAMBLOCK=150
MIN_SENT=50
@ -27,20 +27,9 @@ HBOUNCE=$(mlog 'hcout[0-9]/smtp' | grep -c 'status=bounced')
HSPAM=$(mlog 'hcout[0-9]/smtp' | grep 'status=bounced' | grep -c '550-5.7.1')
HTOTAL=$((HSENT + HBOUNCE)); HDEL=0
[ "$HTOTAL" -gt 0 ] && HDEL=$(python3 -c "print(round(100*$HSENT/$HTOTAL))")
# IP rehab pool (.91-.93 / rehab02-04) — recovering after the May 30-31 blast.
# A recovering IP naturally bounces more on a cold list, so the rehab bounce
# threshold is lenient; we only flag if it is alarmingly high (a sign the rehab
# recipient quality regressed) or if a rehab IP lands on a DNSBL.
RSENT=$(mlog 'rehab0[234]/smtp' | grep -c 'status=sent')
RBOUNCE=$(mlog 'rehab0[234]/smtp' | grep -c 'status=bounced')
RTOTAL=$((RSENT + RBOUNCE)); RDEL=0
[ "$RTOTAL" -gt 0 ] && RDEL=$(python3 -c "print(round(100*$RSENT/$RTOTAL))")
# DNSBL check for the rehab IPs (reuse Spamhaus ZEN — the one that matters most).
RBL=''
for ip in 91 92 93; do
hit=$(dig +short +time=3 +tries=1 ${ip}.124.174.207.zen.spamhaus.org 2>/dev/null | head -1)
[ -n "$hit" ] && RBL="${RBL}.${ip} on Spamhaus ZEN ($hit); "
done
# NOTE: the IP rehab pool (.91-.93 / rehab02-04) and the multi-IP rotation were
# REMOVED 2026-06-23 (snowshoe cleanup, see docs/deliverability.md). Only the two
# warm sending IPs remain: .94 (trucking / out05) and .107 (HC / hcout1).
PROBLEMS=''
if [ "$MSENT" -ge "$MIN_SENT" ]; then
[ "$MDEL" -lt "$MIN_DELIVERY" ] && PROBLEMS="${PROBLEMS}- Main pool delivery ${MDEL}% (below ${MIN_DELIVERY}%)\n"
@ -50,20 +39,23 @@ if [ "$HSENT" -ge "$MIN_SENT" ]; then
[ "$HDEL" -lt "$MIN_DELIVERY" ] && PROBLEMS="${PROBLEMS}- HC stream delivery ${HDEL}%\n"
[ "$HSPAM" -gt "$MAX_SPAMBLOCK" ] && PROBLEMS="${PROBLEMS}- HC stream spam/policy blocks: ${HSPAM}\n"
fi
# Rehab problems: DNSBL listing is always a problem; bounce >60% with real
# volume means the recipient quality slipped (rehab should be on clean domains).
[ -n "$RBL" ] && PROBLEMS="${PROBLEMS}- IP rehab DNSBL: ${RBL}\n"
if [ "$RSENT" -ge 10 ] && [ "$RDEL" -lt 40 ]; then
PROBLEMS="${PROBLEMS}- IP rehab delivery ${RDEL}% (recipient quality slipped)\n"
fi
# DNSBL check for the two live sending IPs (.94 trucking, .107 HC). 8.8.8.8 is
# blocked by Spamhaus (returns 127.255.255.254 = "open resolver"), so query a
# resolver that returns real ZEN data (Control D 76.76.2.0; cross-check Neustar).
RBL=''
for ip in 94 107; do
ans=$(dig +short +time=3 +tries=1 ${ip}.124.174.207.zen.spamhaus.org @76.76.2.0 2>/dev/null | grep -E '^127\.0\.0\.' | head -1)
[ -n "$ans" ] && RBL="${RBL}.${ip} on Spamhaus ZEN ($ans); "
done
[ -n "$RBL" ] && PROBLEMS="${PROBLEMS}- Sending IP on DNSBL: ${RBL}\n"
{
echo "==== TG WARMUP CHECK $(date) ===="
echo "MAIN: sent=$MSENT bounced=$MBOUNCE delivery=${MDEL}% spamblock=$MSPAM"
echo "HC: sent=$HSENT bounced=$HBOUNCE delivery=${HDEL}% spamblock=$HSPAM"
echo "REHAB(.91-.93): sent=$RSENT bounced=$RBOUNCE delivery=${RDEL}% dnsbl=${RBL:-clean}"
echo "MAIN(.94): sent=$MSENT bounced=$MBOUNCE delivery=${MDEL}% spamblock=$MSPAM"
echo "HC(.107): sent=$HSENT bounced=$HBOUNCE delivery=${HDEL}% spamblock=$HSPAM"
echo "dnsbl: ${RBL:-clean}"
echo "problems: ${PROBLEMS:-none}"
} >> "$REPORT" 2>&1
if [ -n "$PROBLEMS" ]; then
MSG=$(printf '⚠️ Performance West IP reputation alert (%s)\n\nMain pool: %d%% delivery, %d sent, %d bounced, %d spam-blocks\nHC stream: %d%% delivery, %d sent, %d spam-blocks\nRehab (.91-.93): %d%% delivery, %d sent, %d bounced (dnsbl: %s)\n\nIssues:\n%b' "$TODAY" "$MDEL" "$MSENT" "$MBOUNCE" "$MSPAM" "$HDEL" "$HSENT" "$HSPAM" "$RDEL" "$RSENT" "$RBOUNCE" "${RBL:-clean}" "$PROBLEMS")
MSG=$(printf '⚠️ Performance West IP reputation alert (%s)\n\nMain pool (.94): %d%% delivery, %d sent, %d bounced, %d spam-blocks\nHC stream (.107): %d%% delivery, %d sent, %d spam-blocks\nDNSBL: %s\n\nIssues:\n%b' "$TODAY" "$MDEL" "$MSENT" "$MBOUNCE" "$MSPAM" "$HDEL" "$HSENT" "$HSPAM" "${RBL:-clean}" "$PROBLEMS")
tg "$MSG"
fi