# Email Deliverability - Incident & Issue Timeline (May–June 2026)

Listmonk + Postfix (self-hosted MTA) cold-outreach for trucking (IP .94) and
healthcare (IP .107) on host 207.174.124.71. Dates are when the issue was
identified/fixed; root causes often predate the fix.

---

## May

- **2026-05-21** - Rebuilt the Listmonk bounce-sync after unreliable webhook
  delivery (Listmonk silently drops bounces it can't FK-match to a subscriber).
  Switched to log-scraping `/var/log/mail.log` and inserting with real
  subscriber IDs. (commit `ba2f6eb`)

- **2026-05-30** - ⭐ **The DKIM disaster blast.** A large trucking blast went
  out with **broken DKIM signing**, so receivers applied DMARC auth-policy
  rejection. **7,634 "hard bounces" in one day** - but ~6,604 were DSN `5.7.1`
  (DMARC/policy failures, *not* bad mailboxes); only ~221 were real dead
  mailboxes (`5.1.1`). This is the event that poisoned the carrier list.

- **2026-05-31** → following weeks - Fallout: Listmonk auto-blocklisted on the
  **first** hard bounce, and the bounce-sync's own SQL also blocklisted on the
  first 5xx of *any* DSN. Result: **~17,000 carriers wrongly blocklisted** (88%
  of the list) over the broken-DKIM window. Not discovered as a false-positive
  until late June.

---

## June - root-cause fixes to the sending stack

- **2026-06-14** - Per-MX-operator throttling added; Google / Microsoft 365
  (Workspace) excluded from warmup sends. HC warmup corrected to run **daily**
  for the full 21-day ramp (was weekdays-only, stretching the ramp). (`9e40965`,
  `2caab6a`)

- **2026-06-16** - Stopped blasting trucking to `mx_unreachable` dead domains;
  the verifier was mislabeling live big-ISP mailboxes as unreachable. Suppressed
  defunct/legacy/satellite ISP domains in cold sends. (`1652a3b`, `1eb29f8`,
  `c183957`)

- **2026-06-17** - ⭐ **Root DKIM fix.** Found OpenDKIM was **not signing**
  campaign mail (the Docker-injected path bypassed signing); fixed and codified
  in Ansible. Also: added a `text/plain` MIME part to every email (spam-filter
  requirement), stable Message-ID hostname, Postfix `mail.log` logrotate,
  decommissioned SMTP2GO (local MTA only). (`4d59019`, `a32a3b0`, `b375385`,
  `2e4388a`, `a04ecf7`)

- **2026-06-18** - ⭐ Moved bulk campaigns to a **dedicated subdomain**
  `send.performancewest.net` (protects the root domain's reputation); Ansible
  signs it. **Killed the snowshoe IP pattern** now that DKIM works (consolidated
  sending IPs). Excluded Apple/iCloud consumer mail; began scrubbing stale
  consumer subscribers from Listmonk. Catch-all pool auto-rollout gated by
  warmup-day + live bounce rate. (`5c3b429`, `545e6f7`, `b40fc7e`, `40da017`)

- **2026-06-19** - Removed 18 dormant **snowshoe IPs** from Postfix + host.
  Built a **mail-reputation monitor** (SNDS-equivalent from Postfix logs) +
  nightly snapshot cron. Stood up **DMARC aggregate-report ingestion** (dedicated
  `dmarc@` mailbox + parser); classified the whole `207.174.124.0/24` as ours.
  (`9dd6f53`, `08f651d`, `b45332b`, `8e5590b`, `707d538`)

- **2026-06-20** - Bounded the untagged (NULL `mx_provider`) bucket in the
  selector and closed **MX-exclusion gaps** (consumer MX operators were leaking
  into cold sends); added an MX-tagging cron. (`9eeed47`, `bc93d93`)

- **2026-06-21** - Fixed the **Reply-To header shape** - Listmonk was silently
  dropping a malformed Reply-To. (`e414ec4`)

- **2026-06-22** - ⭐ **Post-DKIM re-send** to the list, with a **Gmail-only
  exclusion** (Gmail still distrusted the warming domain). Stepped the trucking
  rate cap back up to 400/h (day 19–20), 500/h ceiling. (`5a3063e`, `1e9dcfc`)

- **2026-06-22/23** - Fixed broken CTAs in trucking email: a recurring
  `@TrackLink` **404** + link-collapse bug, and order CTAs pointing at the wrong
  ($399 catch-all) service page. (`3325259`, `e3f4392`, `a90cdc9`)

- **2026-06-24** - ⭐ **Sending-IP outage.** The warmed sending IPs **dropped
  off interface `ens18` on reboot**, so mail stopped/misrouted. Fixed to persist
  across reboots. Also repaired two dead mail-alert crons + de-noised the DMARC
  digest. (`4276ada`, `ae68edb`)

- **2026-06-26** - ⭐ **Volume whipsaw fixed.** The catch-all guardrail used a
  **2-day** bounce window; one bad batch (Jun 24: 465 sent / 10.75%) flipped
  catch-all OFF, starving volume so badly it couldn't gather a 300-send sample to
  re-enable - a self-reinforcing trap. Widened the window **2d → 5d**. Also fixed
  the HC cron **re-mailing the whole list daily** (added per-day send lists).
  (`f344287`, `b350a13`)

- **2026-06-26** - ⭐⭐ **The re-blocklist bomb.** Discovered `listmonk-bounce-sync`
  (root cron, every 5 min) was blocklisting carriers on the **first hard bounce
  of *any* 5xx DSN** via direct SQL - bypassing Listmonk's own threshold. *This*
  is the mechanism that wrongly killed ~17,000 carriers in May. Rewrote it: only
  genuine bad-mailbox DSNs (5.1.1/5.1.10/5.1.0/5.0.0/5.4.1/5.5.0) count, and it
  now requires **≥3 distinct hard bounces**. Reputation/policy 5.7.x and
  quota/greylist 5.2.x never trigger a blocklist. (`bfdbf8f`)

- **2026-06-27** - ⭐ **Wrongly-blocklisted recovery send (campaign 727).**
  Un-blocklisted 4,317 false-positive carriers (excluding the ~688 real dead
  mailboxes), re-sent with a fresh 30%-off coupon. Verified the bounce-sync fix
  held live: 727 took ~61 hard bounces but **0 carriers re-blocklisted**.

- **2026-06-27** - ⭐⭐⭐ **Disk-full Postgres crash, mid-send.** `/` hit **100%**
  (orphaned 15GB forgejo backup dump + uncapped Docker logs), Postgres
  crash-looped on "No space left on device", and the Listmonk container was
  destroyed mid-campaign. Recovered (pruned build cache + dumps + orphan volumes:
  100% → 72%, 62GB free), recreated Listmonk, campaigns auto-resumed. Added a
  **Docker log cap** (50m×3) and a **disk-space monitor** (Telegram warn at 90%,
  auto-reclaim at 94%) - neither existed before. (`e318f12`, `6b2cf5a`)

- **2026-06-27** - ⭐ **/24 RBL listing.** The whole `207.174.124.0/24` block
  got listed on **invaluement** (ivmSIP + ivmSIP/24) - affects ~11% of
  recipients (Intermedia/securence business domains); **Spamhaus / Barracuda /
  SpamCop all clean**, so Gmail/Microsoft/Yahoo unaffected. Dialed catch-all back
  to smtp_valid-only and submitted a delist request (propagation pending). Also
  noted ~73 "very low reputation" rejects are **Google-Workspace custom domains**
  the `@gmail.com` filter misses.

---

## Cross-cutting root causes (the recurring themes)

1. **DKIM not signing (May)** → DMARC rejections misread as hard bounces →
   mass false-positive blocklisting. The single most damaging issue.
2. **Over-aggressive blocklisting logic** (both Listmonk's count:1 default *and*
   the bounce-sync's own SQL) turned transient/policy bounces into permanent
   list death.
3. **Reputation/warmup fragility** - snowshoe IPs, missing dedicated subdomain,
   consumer-domain leakage, big-MX (Google/MS) sensitivity during warmup.
4. **Operational guardrails too twitchy or missing** - 2-day catch-all window
   whipsaw; no disk monitoring; uncapped Docker logs; IPs dropping off NIC on
   reboot; dead alert crons.