justin/new-site
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(deliverability): mail reputation monitor (SNDS-equivalent from postfix logs)

Browse source 
Adds scripts/mail_reputation_monitor.py + migration 101 (mail_reputation_daily).
Sender reputation is judged by the RECEIVING operator (Microsoft/Google/Yahoo/
Proofpoint), and the provider portals (SNDS/Postmaster/CFL) need a login and lag
24-48h. Our postfix logs already carry the ground truth in real time: every send
records the receiving host + SMTP response, and the response classifies WHY:
  250            -> accepted
  451 4.7.500    -> throttled (Microsoft rate-limiting a cold IP)
  550 5.7.x      -> reject_reputation (spam/reputation)
  550 5.1.1/5.4.1-> reject_recipient (dead mailbox / access denied = list hygiene)
  550 ...SPAM    -> reject_content (SpamAssassin)

The parser classifies each egress delivery (out0x/hcout/relay) by (sending_ip,
receiver, outcome, reason_code) and upserts ONE daily aggregate row per bucket
(idempotent ON CONFLICT), so a nightly cron over the rotated log gives a queryable
trend without re-parse double-counting. --alert prints a per-operator summary and
Telegram-alerts on regressions (>=10% reputation rejects, or Microsoft >=70%
throttled). Reads stdin ("-") so the host-owned /var/log/mail.log can be piped
into the DB-connected workers container.

Motivation: 2026-06-19 audit found ~80% of Microsoft sends were getting 451 4.7.500
throttles on the warming IPs -- this makes that trend visible as reputation recovers.
This commit is contained in:
justin 2026-06-19 08:35:45 -05:00
parent bd7ba23841
commit 08f651dc1e
2 changed files with 409 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

39
api/migrations/101_mail_reputation_daily.sql Normal file
View file

@ -0,0 +1,39 @@
-- Daily mail-reputation snapshots parsed from the postfix logs.
--
-- WHY: Sender reputation lives at the RECEIVING operator (Microsoft 365, Google,
-- Yahoo, ...), not at the recipient domain. The provider portals (SNDS, Postmaster,
-- Yahoo CFL) show this but each needs a login and lags 24-48h. Our own postfix logs
-- already contain the ground truth in real time: every send to a Microsoft tenant
-- returns 250 (accepted) / 451 4.7.500 (throttled, "server busy") / 5xx (rejected),
-- and the reject text tells us WHY (reputation 5.7.1, recipient unknown 5.1.1, etc.).
-- A 2026-06-19 audit found 80% of Microsoft sends were getting 451 4.7.500 throttles
-- on the warming IPs -- exactly the signal we need to watch ease as reputation builds.
--
-- This table stores ONE row per (log_date, sending_ip, receiver_operator, outcome
-- class) so we can chart accepted% / deferred% / reject-reason mix per operator over
-- time without re-parsing logs (which rotate fast). Populated by
-- scripts/mail_reputation_monitor.py (idempotent upsert per day).
CREATE TABLE IF NOT EXISTS mail_reputation_daily (
id BIGSERIAL PRIMARY KEY,
log_date DATE NOT NULL, -- the calendar day of the log lines (server TZ)
sending_ip TEXT NOT NULL, -- our egress IP (207.174.124.94 / .107 / ...)
receiver TEXT NOT NULL, -- normalized receiving operator: microsoft|google|yahoo|proofpoint|other
outcome TEXT NOT NULL, -- accepted|throttled|reject_reputation|reject_recipient|reject_content|reject_other|deferred_other
reason_code TEXT, -- representative enhanced status / code (e.g. 4.7.500, 5.7.1, 5.1.1)
msg_count INTEGER NOT NULL DEFAULT 0,
sample_text TEXT, -- one representative raw response, for debugging
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
UNIQUE (log_date, sending_ip, receiver, outcome, reason_code)
);
CREATE INDEX IF NOT EXISTS idx_mail_rep_daily_date ON mail_reputation_daily (log_date);
CREATE INDEX IF NOT EXISTS idx_mail_rep_daily_receiver ON mail_reputation_daily (receiver, log_date);
COMMENT ON TABLE mail_reputation_daily IS
'Daily per-IP per-receiving-operator mail delivery outcome counts parsed from postfix logs (SNDS-equivalent reputation trend, no provider login needed). Populated by scripts/mail_reputation_monitor.py.';
COMMENT ON COLUMN mail_reputation_daily.receiver IS
'Normalized receiving operator: microsoft|google|yahoo|proofpoint|mimecast|other';
COMMENT ON COLUMN mail_reputation_daily.outcome IS
'accepted|throttled|reject_reputation|reject_recipient|reject_content|reject_other|deferred_other';