justin/new-site
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(deliverability): document Jun 22 re-send of never-delivered DKIM-window audience

Browse source 
Records the MAIN_EXCLUDE_OPERATORS=google override, the resend_dkim_backup_20260622
rollback table, the past-send_at HTTP 400 gotcha (use --send-hour for same-day
re-runs), and the exact revert SQL. 6461-row backup; ~2999 re-sent Jun 22, rest
drain on subsequent daily runs (Gmail excluded, Microsoft/Hotmail included).
This commit is contained in:
justin 2026-06-22 11:59:29 -05:00
parent 5a3063ecb3
commit 62292b96af
1 changed files with 30 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

30
docs/email-deliverability-runbook.md
View file

@ -171,6 +171,36 @@ sudo opendkim-testkey -d performancewest.net -s mail -vvv # expect "key OK"
- Dead M365 tenant scrub: HC defers are mostly `451 4.4.4` against dead M365 - Dead M365 tenant scrub: HC defers are mostly `451 4.4.4` against dead M365
tenants + `421` LuxSci throttle. Identify and suppress dead tenants. tenants + `421` LuxSci throttle. Identify and suppress dead tenants.
### Re-send of the never-delivered (unsigned) audience — Jun 22 2026
The ~79k cold sends made during the broken window (Jun 1 - Jun 18 00:31 UTC) were
stamped `listmonk_sent_at` at send time, so the builder permanently excluded them
even though they were junked/blocked unsigned. With DKIM now fixed we re-send to
the now-deliverable subset, **excluding Gmail** (Google consumer reputation is
still recovering) but **including Microsoft/Hotmail** (the bulk of the list).
What was done (all reversible):
1. `MAIN_EXCLUDE_OPERATORS` env override added to the builder (commit `5a3063e`):
when set it REPLACES the default `WARMUP_EXCLUDE_OPERATORS`. Set to `google` in
the `workers` service env so cold sends go to everything except Google, driving
both the SQL exclude and the per-operator daily cap (google cap=0, others 120).
2. Backed up the reset target to `performancewest.resend_dkim_backup_20260622`
(6,461 rows = broken-window AND `email_verify_result IN (smtp_valid,
send_confirmed)` AND `mx_provider <> google`), then `UPDATE fmcsa_carriers SET
listmonk_sent_at = NULL` for exactly those rows so the builder re-queues them.
3. Ran the builder with `--send-hour 17 --send-minute 30` (the default per-tz hours
09-12 UTC were already past; **Listmonk rejects a past `send_at` with HTTP 400
"Scheduled date should be in the future"** — always override the hour for a
same-day manual re-run after the normal window). Result: 30 campaigns,
queued_recipients=3000 (warmup cap), ~2,999 re-stamped. Provider mix: Microsoft
1,272 / Comcast / Charter / Proofpoint / long-tail; **zero Google**.
The remaining ~3.5k of the 6,461 backup set drain on subsequent daily runs under
the same cap. To revert a row: `UPDATE fmcsa_carriers c SET listmonk_sent_at =
b.old_listmonk_sent_at FROM resend_dkim_backup_20260622 b WHERE c.dot_number =
b.dot_number;`. To resume normal warmup exclusion later, unset
`MAIN_EXCLUDE_OPERATORS` (reverts to Google+Microsoft+consumer-MX held to day 30).
### Follow-up hardening — DONE (Jun 17-18 2026) ### Follow-up hardening — DONE (Jun 17-18 2026)
All discovered during the post-incident technical audit; each fix is codified. All discovered during the post-incident technical audit; each fix is codified.