From a05b97196a0219350a6bb8983967677424d470c6 Mon Sep 17 00:00:00 2001 From: justin Date: Thu, 2 Jul 2026 15:42:44 -0500 Subject: [PATCH] runbook: document 2026-07-02 clean-pool decay + bounce->source reconcile fix --- docs/email-deliverability-runbook.md | 63 ++++++++++++++++++++++++++++ 1 file changed, 63 insertions(+) diff --git a/docs/email-deliverability-runbook.md b/docs/email-deliverability-runbook.md index 41f7dbd..666c213 100644 --- a/docs/email-deliverability-runbook.md +++ b/docs/email-deliverability-runbook.md @@ -363,3 +363,66 @@ Consider migrating ifupdown -> netplan with all addresses, or pin `unattended-upgrades` to skip auto-reboot. The `mail_reputation_monitor.py` attributes egress to `.71` as "transactional default" -- after this incident, a spike of `.71` egress in the bulk streams is itself an alarm. + +## Incident: 2026-07-02 — the "clean" pool silently decayed (bounce → source loop missing) + +**Symptom.** Even after the audience queries were fixed to send only to the +verified-clean pool (`email_verify_result IN ('smtp_valid','send_confirmed')`), +the fuel-tax newsletters still ran **10-15% hard bounce** (06-30: 163/1584, +07-01: 212/1583, 07-02: 239/1589) and no sales were landing. Newsletters were +being blamed on content/reputation; the real cause was list rot. + +**Diagnosis (measured).** Joined the 7-day newsletter hard bounces back to +`fmcsa_carriers.email_verify_result`: **873 of the bounced addresses were +`smtp_valid`** — i.e. the "clean" pool itself. Then checked the newsletter's +staging table directly: + +``` +staged_total | staged_hard_bounced | staged_blocklisted + 11551 | 1371 | 1249 +``` + +**12% of the "clean" staged pool had ALREADY hard-bounced** in Listmonk, and +1,249 were blocklisted, yet they were still being re-selected every send. The +audience filter was correct; the gap was that **nothing demoted a source row +after it hard-bounced on a real send.** Trucking mailboxes churn fast (brokers +fold, drivers switch carriers, free mailboxes get deleted), so `smtp_valid` +verified weeks ago silently rots, but `pw_smtp_valid_stage` (and the builders) +kept trusting it. `listmonk-bounce-sync.py` blocklists the *Listmonk subscriber* +after 3 strikes but never touched `fmcsa_carriers`, so the source of truth — +and every list rebuilt from it — stayed dirty. + +**Fix — close the bounce → source feedback loop.** New +`scripts/reconcile_bounces_to_source.py` (host script, stdlib-only, runs via +`docker exec` like `coupon_ab_scoreboard.py`): +1. Reads every HARD bounce Listmonk recorded in a lookback window (trucking +, + with `--include-hc`, healthcare). +2. Demotes the matching `fmcsa_carriers` row to `email_verify_result = + 'hard_bounced'` (`email_verified = FALSE`) — the same terminal, never-sendable + state `burner_list_verify.py` uses. Worst signal wins; idempotent. +3. Rebuilds `pw_smtp_valid_stage` from the now-cleaned source, excluding anything + blocklisted or hard-bounced in Listmonk, so the newsletter's join is clean. + +First apply (2026-07-02, `--days 60`): **demoted 21,033** dead source rows, +pruned the stage **11,551 → 9,631**. Post-run the staged pool had **0** +hard-bounced and **0** blocklisted addresses. + +**Guardrail — now recurring.** `/etc/cron.d/pw-bounce-reconcile` (repo +`infra/cron/pw-bounce-reconcile`, wired through the `mail-pipeline` Ansible role) +runs it daily at **06:20 UTC** — after the every-5-min `listmonk-bounce-sync` +has ingested overnight bounces, before the 06:30 UTC consumer scrub and the +08:00 UTC trucking builder. The clean pool now self-heals; a verified address +that dies is demoted within a day instead of being re-mailed indefinitely. + +**Note on growing the clean pool.** The SMTP-probe verifier +(`scripts/workers/email_verifier.py`) only checks rows with `email_verified IS +NULL`, and as of this incident there are **0** left — the whole 2M-row census is +probed. The remaining pools are **428K `mx_probe_blocked`** (big ISPs that +tarpit port-25 probes — Comcast/AT&T/Verizon/etc.) and **1.09M +`catch_all_domain`**. Neither can be cleared by another SMTP probe. The only +ground truth is a **real send**: either the guarded catch-all rollout in the +builder (`catch_all_enabled()` folds catch-all in once warmup day ≥ 21 and the +live bounce rate is proven low — currently day 30, 5-day bounce ~0.04%, so this +is safe to widen), or a **burner-domain verification send** +(`burner_list_verify.py`) for the `mx_probe_blocked` pool. Do NOT try to grow +the pool by re-running the SMTP probe; it is exhausted.