runbook: document 2026-07-02 clean-pool decay + bounce->source reconcile fix

docs/email-deliverability-runbook.md

@@ -363,3 +363,66 @@ Consider migrating ifupdown -> netplan with all addresses, or pin
 `unattended-upgrades` to skip auto-reboot. The `mail_reputation_monitor.py`
 attributes egress to `.71` as "transactional default" -- after this incident, a
 spike of `.71` egress in the bulk streams is itself an alarm.
+
+## Incident: 2026-07-02 — the "clean" pool silently decayed (bounce → source loop missing)
+
+**Symptom.** Even after the audience queries were fixed to send only to the
+verified-clean pool (`email_verify_result IN ('smtp_valid','send_confirmed')`),
+the fuel-tax newsletters still ran **10-15% hard bounce** (06-30: 163/1584,
+07-01: 212/1583, 07-02: 239/1589) and no sales were landing. Newsletters were
+being blamed on content/reputation; the real cause was list rot.
+
+**Diagnosis (measured).** Joined the 7-day newsletter hard bounces back to
+`fmcsa_carriers.email_verify_result`: **873 of the bounced addresses were
+`smtp_valid`** — i.e. the "clean" pool itself. Then checked the newsletter's
+staging table directly:
+
+```
+staged_total | staged_hard_bounced | staged_blocklisted
+       11551 |                1371 |               1249
+```
+
+**12% of the "clean" staged pool had ALREADY hard-bounced** in Listmonk, and
+1,249 were blocklisted, yet they were still being re-selected every send. The
+audience filter was correct; the gap was that **nothing demoted a source row
+after it hard-bounced on a real send.** Trucking mailboxes churn fast (brokers
+fold, drivers switch carriers, free mailboxes get deleted), so `smtp_valid`
+verified weeks ago silently rots, but `pw_smtp_valid_stage` (and the builders)
+kept trusting it. `listmonk-bounce-sync.py` blocklists the *Listmonk subscriber*
+after 3 strikes but never touched `fmcsa_carriers`, so the source of truth —
+and every list rebuilt from it — stayed dirty.
+
+**Fix — close the bounce → source feedback loop.** New
+`scripts/reconcile_bounces_to_source.py` (host script, stdlib-only, runs via
+`docker exec` like `coupon_ab_scoreboard.py`):
+1. Reads every HARD bounce Listmonk recorded in a lookback window (trucking +,
+   with `--include-hc`, healthcare).
+2. Demotes the matching `fmcsa_carriers` row to `email_verify_result =
+   'hard_bounced'` (`email_verified = FALSE`) — the same terminal, never-sendable
+   state `burner_list_verify.py` uses. Worst signal wins; idempotent.
+3. Rebuilds `pw_smtp_valid_stage` from the now-cleaned source, excluding anything
+   blocklisted or hard-bounced in Listmonk, so the newsletter's join is clean.
+
+First apply (2026-07-02, `--days 60`): **demoted 21,033** dead source rows,
+pruned the stage **11,551 → 9,631**. Post-run the staged pool had **0**
+hard-bounced and **0** blocklisted addresses.
+
+**Guardrail — now recurring.** `/etc/cron.d/pw-bounce-reconcile` (repo
+`infra/cron/pw-bounce-reconcile`, wired through the `mail-pipeline` Ansible role)
+runs it daily at **06:20 UTC** — after the every-5-min `listmonk-bounce-sync`
+has ingested overnight bounces, before the 06:30 UTC consumer scrub and the
+08:00 UTC trucking builder. The clean pool now self-heals; a verified address
+that dies is demoted within a day instead of being re-mailed indefinitely.
+
+**Note on growing the clean pool.** The SMTP-probe verifier
+(`scripts/workers/email_verifier.py`) only checks rows with `email_verified IS
+NULL`, and as of this incident there are **0** left — the whole 2M-row census is
+probed. The remaining pools are **428K `mx_probe_blocked`** (big ISPs that
+tarpit port-25 probes — Comcast/AT&T/Verizon/etc.) and **1.09M
+`catch_all_domain`**. Neither can be cleared by another SMTP probe. The only
+ground truth is a **real send**: either the guarded catch-all rollout in the
+builder (`catch_all_enabled()` folds catch-all in once warmup day ≥ 21 and the
+live bounce rate is proven low — currently day 30, 5-day bounce ~0.04%, so this
+is safe to widen), or a **burner-domain verification send**
+(`burner_list_verify.py`) for the `mx_probe_blocked` pool. Do NOT try to grow
+the pool by re-running the SMTP probe; it is exhausted.