justin/new-site
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
new-site/docs/email-deliverability-runbook.md

5.6 KiB
Raw Blame History

Email Deliverability & IP Warmup Runbook

Performance West self-hosts its outbound MTA (Postfix on the app server) because transactional relays (SES, Postmark, SendGrid) forbid the cold prospecting email our FMCSA trucking and telecom campaigns depend on. That means we own our sending-IP reputation and must manage it manually. This doc is the operational guide for keeping it healthy.

Infrastructure layout

  • Host Postfix on the app server (207.174.124.71), reached by Listmonk via SMTP at 172.18.0.1:25.
  • Sending IPs: 207.174.124.90 through .109 (20 IPs), each with valid FCrDNS (mtaNN.performancewest.net) and authorized in SPF (-all).
    • .90 / mta01: historically a dedicated Yahoo trickle IP. We no longer mail Yahoo at all, so it is idle.
    • .91-.109 / mta02-mta20: rotation pool, selected via transport_maps = hash:/etc/postfix/transport, randmap:{<active pool>}.
  • Warmup scheduler: /usr/local/bin/pw-mta-warmup (daily cron /etc/cron.d/pw-mta-warmup, 07:17 UTC). Recomputes the active rotation pool from a start date stamped in /etc/postfix/pw-warmup-start. Ramp schedule: day 0-3 -> 3 IPs, 4-7 -> 5, 8-11 -> 8, 12-17 -> 12, 18-24 -> 16, 25+ -> 19. The pool only ever grows. It picks IPs from the front of the ALL=(...) array.

What we do NOT mail

The Yahoo / Verizon-Media family is excluded entirely (yahoo, aol, att, verizon, frontier, sbcglobal, bellsouth, pacbell, ameritech, ymail, rocketmail, aim, netscape, compuserve, etc.). They aggressively defer cold senders with 421 4.7.0 [TSS04] ... unexpected volume or user complaints, and that deferral poisons the sending IP for Gmail and Microsoft too.

Enforced in two layers:

  1. Audience build (authoritative): scripts/_email_exclusions.py (BLOCKED_EMAIL_DOMAINS), imported by build_trucking_campaigns.py and populate_new_carrier_startup_campaign.py. New campaigns never include them.
  2. Postfix backstop: /etc/postfix/transport maps every Yahoo-family domain to hold:. If any leak into the queue they are parked, never sent from a rotation IP.

Incident: May 30-31 2026 reputation collapse

A campaign blast pushed ~29k sends in a day across cold IPs .91/.92/.93 with no daily volume cap. Result:

  • Gmail: 550-5.7.1 ... likely unsolicited mail (hard spam block).
  • Yahoo: 421 TSS04 on the rotation IPs.
  • Steady state afterward: ~13% delivery (10k sent vs 68k deferred + 7k bounced in a day). Listmonk open rate ~4%, clicks ~0.

Remediation (Jun 02 2026)

  • Retired the 3 burned IPs (.91/.92/.93 = out02/03/04) from rotation. Confirmed .94-.109 had never sent outbound (only inbound port-scan noise), so they are pristine.
  • Swapped rotation to fresh .94/.95/.96 (out05/06/07) and reset the warmup start date to day 0.
  • Patched pw-mta-warmup ALL array to start at out05 so the daily cron never reverts to the burned IPs.
  • Rewrote /etc/postfix/transport to hold: the full Yahoo family (was a partial list with buggy duplicate keys routing to yahooslow).
  • Flushed the entire stale queue (1,846 blast-era messages, mostly dead satellite ISPs) so fresh IPs start clean.
  • Enabled Listmonk sliding-window rate limit so no campaign can blast again: app.message_sliding_window=true, duration 1h, rate 50, message_rate=2.
  • Paused 19 trucking campaigns (IDs 275-293, ~13k recipients) that were scheduled to fire Jun 03; they were built before the exclusion fix and would have re-torched the fresh IPs. Rebuild them small/clean before resending.

Fresh-IP warmup discipline (the rules)

The historical mail.log proves these IPs sustain ~2,500 sends/day at 68-76% delivery once warm (May 19-21). Collapses only ever came from 17k-29k spikes. So we ramp ASSERTIVELY but never spike. The Listmonk sliding-window cap (/usr/local/bin/pw-listmonk-rampcap, daily cron 07:20 UTC, driven off the same /etc/postfix/pw-warmup-start stamp) enforces this automatically:

warmup day hourly cap ~daily total
0-1 50/h ~500
2-3 150/h ~1,500
4-6 250/h ~2,500
7+ 300/h ~3,000 (hard ceiling)

Hard rule from the data: never exceed ~4k/day, never spike.

Other rules:

  1. Best recipients first. Gmail + Microsoft + clean ISPs only (Yahoo family already excluded). Send small focused batches, e.g. build_trucking_campaigns --only-segment mcs150 --max-per-segment 100 --date <today> --send-hour <H>.
  2. Scrub hard bounces immediately. 550 5.1.1, full mailbox, "not our customer" all hurt reputation signals.
  3. Watch the signals daily (see commands below). If Gmail 550-5.7.1 or Yahoo 421 TSS04 reappear, STOP and hold for several days.

Monitoring commands

# delivery mix today
sudo grep "^$(date '+%b %d')" /var/log/mail.log | grep -oE 'status=(sent|deferred|bounced)' | sort | uniq -c

# per-IP outbound volume today (catch a runaway blast early)
for ip in 94 95 96; do echo -n ".$ip: "; sudo grep "^$(date '+%b %d')" /var/log/mail.log | grep -c "207.174.124.$ip"; done

# top deferral / bounce reasons today
sudo grep "^$(date '+%b %d')" /var/log/mail.log | grep status=deferred | grep -oE 'said: [0-9]{3}[^)]{0,50}' | sort | uniq -c | sort -rn | head

# queue size
sudo postqueue -p | tail -1

# active rotation pool + warmup day
sudo postconf -h transport_maps
echo $(( ($(date +%s) - $(sudo cat /etc/postfix/pw-warmup-start)) / 86400 ))

Backups left on the server (Jun 02 2026 remediation)

  • /etc/postfix/main.cf.bak.*
  • /etc/postfix/transport.bak.*
  • /usr/local/bin/pw-mta-warmup.bak.*