new-site/docs/email-deliverability-runbook.md

Email Deliverability & IP Warmup Runbook

Performance West self-hosts its outbound MTA (Postfix on the app server) because transactional relays (SES, Postmark, SendGrid) forbid the cold prospecting email our FMCSA trucking and telecom campaigns depend on. That means we own our sending-IP reputation and must manage it manually. This doc is the operational guide for keeping it healthy.

Infrastructure layout

• Host Postfix on the app server (207.174.124.71), reached by Listmonk via SMTP at 172.18.0.1:25.
• Sending IPs: 207.174.124.90 through .109 (20 IPs), each with valid FCrDNS (mtaNN.performancewest.net) and authorized in SPF (-all).
  • .90 / mta01: historically a dedicated Yahoo trickle IP. We no longer mail Yahoo at all, so it is idle.
  • .91-.109 / mta02-mta20: rotation pool, selected via transport_maps = hash:/etc/postfix/transport, randmap:{<active pool>}.
• Warmup scheduler: /usr/local/bin/pw-mta-warmup (daily cron /etc/cron.d/pw-mta-warmup, 07:17 UTC). Recomputes the active rotation pool from a start date stamped in /etc/postfix/pw-warmup-start. Ramp schedule: day 0-3 -> 3 IPs, 4-7 -> 5, 8-11 -> 8, 12-17 -> 12, 18-24 -> 16, 25+ -> 19. The pool only ever grows. It picks IPs from the front of the ALL=(...) array.

What we do NOT mail

The Yahoo / Verizon-Media family is excluded entirely (yahoo, aol, att, verizon, frontier, sbcglobal, bellsouth, pacbell, ameritech, ymail, rocketmail, aim, netscape, compuserve, etc.). They aggressively defer cold senders with 421 4.7.0 [TSS04] ... unexpected volume or user complaints, and that deferral poisons the sending IP for Gmail and Microsoft too.

Enforced in two layers:

1. Audience build (authoritative): scripts/_email_exclusions.py (BLOCKED_EMAIL_DOMAINS), imported by build_trucking_campaigns.py and populate_new_carrier_startup_campaign.py. New campaigns never include them.
2. Postfix backstop: /etc/postfix/transport maps every Yahoo-family domain to hold:. If any leak into the queue they are parked, never sent from a rotation IP.

Incident: May 30-31 2026 reputation collapse

A campaign blast pushed ~29k sends in a day across cold IPs .91/.92/.93 with no daily volume cap. Result:

• Gmail: 550-5.7.1 ... likely unsolicited mail (hard spam block).
• Yahoo: 421 TSS04 on the rotation IPs.
• Steady state afterward: ~13% delivery (10k sent vs 68k deferred + 7k bounced in a day). Listmonk open rate ~4%, clicks ~0.

Remediation (Jun 02 2026)

• Retired the 3 burned IPs (.91/.92/.93 = out02/03/04) from rotation. Confirmed .94-.109 had never sent outbound (only inbound port-scan noise), so they are pristine.
• Swapped rotation to fresh .94/.95/.96 (out05/06/07) and reset the warmup start date to day 0.
• Patched pw-mta-warmup ALL array to start at out05 so the daily cron never reverts to the burned IPs.
• Rewrote /etc/postfix/transport to hold: the full Yahoo family (was a partial list with buggy duplicate keys routing to yahooslow).
• Flushed the entire stale queue (1,846 blast-era messages, mostly dead satellite ISPs) so fresh IPs start clean.
• Enabled Listmonk sliding-window rate limit so no campaign can blast again: app.message_sliding_window=true, duration 1h, rate 50, message_rate=2.
• Paused 19 trucking campaigns (IDs 275-293, ~13k recipients) that were scheduled to fire Jun 03; they were built before the exclusion fix and would have re-torched the fresh IPs. Rebuild them small/clean before resending.

Fresh-IP warmup discipline (the rules)

The historical mail.log proves these IPs sustain ~2,500 sends/day at 68-76% delivery once warm (May 19-21). Collapses only ever came from 17k-29k spikes. So we ramp ASSERTIVELY but never spike. The Listmonk sliding-window cap (/usr/local/bin/pw-listmonk-rampcap, daily cron 07:20 UTC, driven off the same /etc/postfix/pw-warmup-start stamp) enforces this automatically:

warmup day	hourly cap	~daily total
0-1	50/h	~500
2-3	150/h	~1,500
4-6	250/h	~2,500
7+	300/h	~3,000 (hard ceiling)

Hard rule from the data: never exceed ~4k/day, never spike.

Other rules:

1. Best recipients first. Gmail + Microsoft + clean ISPs only (Yahoo family already excluded). Send small focused batches, e.g. build_trucking_campaigns --only-segment mcs150 --max-per-segment 100 --date <today> --send-hour <H>.
2. Scrub hard bounces immediately. 550 5.1.1, full mailbox, "not our customer" all hurt reputation signals.
3. Watch the signals daily (see commands below). If Gmail 550-5.7.1 or Yahoo 421 TSS04 reappear, STOP and hold for several days.

Monitoring commands

# delivery mix today
sudo grep "^$(date '+%b %d')" /var/log/mail.log | grep -oE 'status=(sent|deferred|bounced)' | sort | uniq -c

# per-IP outbound volume today (catch a runaway blast early)
for ip in 94 95 96; do echo -n ".$ip: "; sudo grep "^$(date '+%b %d')" /var/log/mail.log | grep -c "207.174.124.$ip"; done

# top deferral / bounce reasons today
sudo grep "^$(date '+%b %d')" /var/log/mail.log | grep status=deferred | grep -oE 'said: [0-9]{3}[^)]{0,50}' | sort | uniq -c | sort -rn | head

# queue size
sudo postqueue -p | tail -1

# active rotation pool + warmup day
sudo postconf -h transport_maps
echo $(( ($(date +%s) - $(sudo cat /etc/postfix/pw-warmup-start)) / 86400 ))

Backups left on the server (Jun 02 2026 remediation)

• /etc/postfix/main.cf.bak.*
• /etc/postfix/transport.bak.*
• /usr/local/bin/pw-mta-warmup.bak.*

Incident: Jun 17 2026 — campaign mail sent UNSIGNED (no DKIM)

Symptom: "no new sales." Campaigns were sending (~3-4k/day) but delivery was ~23% (sent 1,802 vs deferred 5,143 + bounced 580), Gmail returned 550-5.7.1 likely unsolicited mail, and there were zero clicks since Jun 8 despite ~600 opens/day.

Root cause: OpenDKIM was signing nothing that came from Listmonk. /etc/opendkim.conf was in single-key mode with no InternalHosts, so it defaulted to signing only 127.0.0.1. Cron/transactional mail is injected locally (127.0.0.1) so it WAS signed — but campaign mail is injected over the Docker bridge from the Listmonk containers (172.18.0.5 trucking, 172.18.0.25 healthcare). Those clients were not "internal," so OpenDKIM verified (instead of signed) them: every cold email went out unsigned. Since Feb 2024 Gmail/Yahoo require DKIM on bulk mail, so unsigned campaigns were junked/blocked. Proof: 2,620 campaign messages that day, 0 "DKIM-Signature field added" events, while the every-5-min cron mail was signed.

The correct table files already existed (/etc/opendkim/{key.table, signing.table,trusted.hosts}, and trusted.hosts already listed 172.16.0.0/12) — they were simply never wired into opendkim.conf.

Fix (now codified in Ansible roles/mail): point opendkim.conf at the tables and set the signing scope —

KeyTable           refile:/etc/opendkim/key.table
SigningTable       refile:/etc/opendkim/signing.table
InternalHosts      /etc/opendkim/trusted.hosts   # includes 172.16.0.0/12 (Docker)
ExternalIgnoreList /etc/opendkim/trusted.hosts
OversignHeaders    From

then systemctl restart opendkim. This fixes BOTH streams at once: the healthcare submission instances (ports 2526-2528) inherit the global smtpd_milters and the *@performancewest.net signing table covers compliance@. Verified by injecting a message from a Docker IP through both port 25 and port 2526 and confirming "DKIM-Signature field added" for each.

Verify DKIM is actually signing campaign mail:

# Should be NON-ZERO and roughly track campaign volume:
sudo journalctl -u opendkim --since today | grep -c 'DKIM-Signature field added'
# Cross-check: campaign cleanup events today (should be similar order of magnitude)
sudo grep "^$(date '+%b %e')" /var/log/mail.log | grep -c postfix/cleanup
# Key still matches published DNS:
sudo opendkim-testkey -d performancewest.net -s mail -vvv   # expect "key OK"

Still TODO from this incident (list quality + content, not yet done):

• Throttle/pause Gmail until reputation recovers (550-5.7.1 was still firing). The trucking ramp/cap (pw-listmonk-rampcap) currently holds at 200/h and the builder excludes the big-MX operators (Google/Microsoft/...) until warmup day 30; revisit once reputation recovers.
• Dead M365 tenant scrub: HC defers are mostly 451 4.4.4 against dead M365 tenants + 421 LuxSci throttle. Identify and suppress dead tenants.

Re-send of the never-delivered (unsigned) audience — Jun 22 2026

The ~79k cold sends made during the broken window (Jun 1 - Jun 18 00:31 UTC) were stamped listmonk_sent_at at send time, so the builder permanently excluded them even though they were junked/blocked unsigned. With DKIM now fixed we re-send to the now-deliverable subset, excluding Gmail (Google consumer reputation is still recovering) but including Microsoft/Hotmail (the bulk of the list).

What was done (all reversible):

1. MAIN_EXCLUDE_OPERATORS env override added to the builder (commit 5a3063e): when set it REPLACES the default WARMUP_EXCLUDE_OPERATORS. Set to google in the workers service env so cold sends go to everything except Google, driving both the SQL exclude and the per-operator daily cap (google cap=0, others 120).
2. Backed up the reset target to performancewest.resend_dkim_backup_20260622 (6,461 rows = broken-window AND email_verify_result IN (smtp_valid, send_confirmed) AND mx_provider <> google), then UPDATE fmcsa_carriers SET listmonk_sent_at = NULL for exactly those rows so the builder re-queues them.
3. Ran the builder with --send-hour 17 --send-minute 30 (the default per-tz hours 09-12 UTC were already past; Listmonk rejects a past send_at with HTTP 400 "Scheduled date should be in the future" — always override the hour for a same-day manual re-run after the normal window). Result: 30 campaigns, queued_recipients=3000 (warmup cap), ~2,999 re-stamped. Provider mix: Microsoft 1,272 / Comcast / Charter / Proofpoint / long-tail; zero Google.

The remaining ~3.5k of the 6,461 backup set drain on subsequent daily runs under the same cap. To revert a row: UPDATE fmcsa_carriers c SET listmonk_sent_at = b.old_listmonk_sent_at FROM resend_dkim_backup_20260622 b WHERE c.dot_number = b.dot_number;. To resume normal warmup exclusion later, unset MAIN_EXCLUDE_OPERATORS (reverts to Google+Microsoft+consumer-MX held to day 30).

Incident: Jun 22 2026 — @TrackLink on per-subscriber CTAs = 404 + collapse

Symptom. The trucking "deficiency" CTA buttons (the primary order link and the secondary DOT-check link) rendered as Listmonk tracking redirects (https://lists.performancewest.net/link/<uuid>/...) that 404'd. The redirect target (registered in links.url) was https://performancewest.net/order/boc3-filing&utm_source=... — note the & with no ? — an invalid URL.

Root cause. Listmonk's @TrackLink marker registers one static URL per tracked link and points every recipient's /link/<uuid> redirect at that single row. This is fundamentally incompatible with a per-subscriber href such as {{ .Subscriber.Attribs.lp_link }}&utm_source=...:

• The registered links.url was captured with the {{ lp_link }} token dropped, yielding /order/slug&utm_source=... (first &, no ?) → 404 for everyone.
• Even if the URL had been valid, a static registration collapses every carrier onto the first subscriber's ?dot= (or ?npi=/?clia=) value — wrong order pre-fill for the entire blast.

By contrast, a static CTA (same URL for all recipients, e.g. the CRTC ?code=... order link) tracks correctly — keep @TrackLink there.

Why removing tracking loses nothing. Real human clicks are already attributed via Umami's campaign-click event (bot-filtered by pw-bot-filter.js). Listmonk's own click counters were already established as unreliable for this stream. So Listmonk link tracking on per-subscriber CTAs is both redundant and destructive.

Fix — live (already-sent + in-flight mail). Rewrote the 10 broken registered rows in place (replace the first & with ?) so the baked /link/<uuid> redirects resolve. Backup table listmonk.pw_links_dkim_fix_bak_20260622 holds the old urls. Verified the exact redirect that 404'd now returns 200 → lands on the (generic, DOT-not-prefilled but fully functional) order page. To revert:

UPDATE links l SET url = b.url
FROM pw_links_dkim_fix_bak_20260622 b WHERE l.id = b.id;  -- in the `listmonk` DB

Fix — source (future builds, the real fix). Stripped @TrackLink from every per-subscriber / per-provider CTA so each row renders its own direct link (no redirect, no collapse). Files changed:

• scripts/create_deficiency_source_campaigns.py — _cta() (lp_link order button) and _dot_check_cta() (per-DOT tools link).
• data/trucking_campaigns/{ucr_annual_reminder,ifta_quarterly_reminder}.html (per-carrier lp_link).
• data/hc_campaigns/*.html (10 templates, per-provider ?npi=/?clia=). lp_link already starts its query with ?dot= (see lp_link_with_coupon()), so {{ lp_link }}&utm... renders to a valid per-carrier URL once the redirect is gone.

Healthcare note. The HC Listmonk DB (listmonk_hc) had 0 registered links despite 13,425 sent — @TrackLink was not being stripped there at all, so the literal @TrackLink shipped as harmless trailing text in utm_campaign and the hrefs still 200'd (per-provider ?npi= was present literally in the template, not via lp_link). No live HC breakage; source templates cleaned anyway to remove the collapse risk on the next send.

Guardrail. Never put @TrackLink on an href containing a {{ .Subscriber... }} token. Per-subscriber links must render directly; rely on Umami campaign-click for human-click attribution.

Follow-up hardening — DONE (Jun 17-18 2026)

All discovered during the post-incident technical audit; each fix is codified.

1. OpenDKIM not signing — fixed + codified in Ansible roles/mail (commit 4d59019). Foundational fix above.
2. mail.log unbounded (~1 GB, no logrotate) — this host logs via Postfix's built-in postlogd (no rsyslog), so a rename+create would strand the open inode. Added a copytruncate logrotate rule (daily, 14-day, compressed) to roles/mail (commit 2e4388a). Applied live, 1 GB archive compressed.
3. Plaintext (altbody) MIME part — all campaigns were HTML-only (a spam-score signal; Listmonk only emits multipart/alternative when altbody is set). New scripts/_email_plaintext.py renders a text/plain part from the HTML body (preserves Listmonk template tags, links -> "text (url)"); wired into the trucking builder (and thus UCR + IFTA) and the healthcare builder. Tests: scripts/test_email_plaintext.py. Commits a32a3b0, 4664601.
4. @localhost.localdomain Message-IDs — Listmonk derived the Message-ID from the random Docker container id. Pinned both listmonk + listmonk-hc hostname: perfwest.performancewest.net in docker-compose.yml (matches the SMTP hello_hostname). Commit a32a3b0.
5. Dead/legacy/satellite ISP scrub — added DEAD_ISP_DOMAINS (52 domains, identified from our own Listmonk bounce table) to BLOCKED_EMAIL_DOMAINS in _email_exclusions.py, so every builder that imports it stops cold-mailing them. Deliberately keeps still-active large consumer ISPs (comcast/charter/ cox/centurylink) — their bounces were the no-DKIM problem, not dead mailboxes. Commit c183957.
6. deploy@performancewest.net self-bounce — the deploy user's crontab held 3 jobs (payment_reminder, amb_location_scraper, renewal_worker) that are EXACT duplicates of systemd timers in the worker-crons role AND redirected to /var/log (which deploy cannot write), so they failed and cron mailed the error to deploy@ (no mailbox -> self-bounce). Removed the redundant deploy crontab (backed up to logs/deploy-crontab.bak.*); the systemd timers carry the work. No IaC change needed (Ansible never created that crontab).
7. Entire campaign pipeline was not in IaC — the campaign cron builders, IP warmup/ramp helpers, and bounce watchers lived ONLY on the host. New Ansible mail-pipeline role + playbooks/deploy-mail-pipeline.yml deploy them all from the canonical repo copies (infra/cron/, infra/postfix/, infra/monitoring/, infra/systemd/, scripts/*bounce*). Commit 4dc5690.
8. Telecom + transactional email was also HTML-only — the campaign-builder plaintext fix (#3) only covered Listmonk mass-mail. The telecom/filing/ customer-transactional path (499-Q reminders, RMD/FCC filing review links, intake/completion/delivery/commission emails, order confirmations) builds its own MIMEMultipart / nodemailer messages, and ~17 of them attached ONLY an HTML part — a malformed single-part multipart/alternative and a spam signal. Fixed at the source so all callers are covered:
   • scripts/workers/worker_email.py send_worker_email() now auto-derives the text/plain part from HTML via _email_plaintext.html_to_text when the caller omits text=.
   • 16 rolled-their-own Python senders (scripts/workers/**, scripts/formation/ document_delivery.py) attach an html_to_text(...) plaintext sibling before the HTML part (job_server + document_delivery wrap text+html in an alternative sub-part so PDF/DOCX still attach to the mixed root).
   • api/src/email.ts gained a dependency-free htmlToText() and sendEmail now defaults text to it (covers checkout/webhook HTML-only sends). NB: telecom campaigns themselves are still manually created+sent in the Listmonk UI (no send automation; compliance_alert_list.py / rmd_deficiency_campaign.py only populate lists). The one telecom send to date — campaign 407 "FCC Deficiency Report - FREEDOM249", Jun 08 — was HTML-only AND sent inside the DKIM-broken window: 384 sent / 343 views / 0 clicks (the same junked-mail signature as the trucking blasts). Any future telecom UI campaign should set an altbody (Listmonk "Plain text" toggle) and run through the same dead-ISP/suppression hygiene. Commit b375385.

INCIDENT 2026-06-24: warmed sending IPs dropped off the interface after reboot

Impact: ~37h of degraded deliverability + a near-zero sales day (Jun 24 04:04 -> Jun 25 17:25). Root cause was infrastructure, not reputation.

What happened. An unattended kernel upgrade rebooted the host at Jun 24 04:04 (6.12.90 -> 6.12.94). The warmed sending IPs .94 (trucking/out05) and .107 (HC/hcout1) are defined in /etc/network/interfaces, but classic ifupdown (0.8.44) only applies the FIRST address line per stanza -- so only .71 (the primary) came back up. Postfix's smtp_bind_address=.94/.107 then failed with warning: smtp_connect_addr: bind ...: Cannot assign requested address and silently fell back to egressing from .71. .71 is (a) NOT in the SPF record (v=spf1 ... ip4:.94 ip4:.107 -all) so every fallback message failed SPF, and (b) listed on RLR621 + Trend Micro ERS-QIL, so receivers deferred them (451 ... blacklisted - RLR621 - ip=<207.174.124.71>). Net: the IP warming was bypassed and mail either failed SPF or got reputation-deferred.

Detection. Tail of /var/log/mail.log showed Cannot assign requested address (16,993 in one log) + deferrals citing ip=<207.174.124.71>. ip -4 addr show ens18 showed only .71 bound (missing .72/.94/.107). last reboot pinned the start to the 04:04 boot. Major RBLs (Spamhaus ZEN/DBL, Barracuda, SpamCop, SORBS) were still clean for .94/.107 and the domain -- RLR621/ERS-QIL are proprietary soft listings keyed off .71/HELO and age off.

Fix (all applied 2026-06-25 ~17:25 CDT).

1. Re-bound live: ip addr add 207.174.124.{72,94,107}/23 dev ens18, then postqueue -f.
2. Reboot-persistence in /etc/network/interfaces: added explicit up/down ip addr add/del ... hooks for the 3 secondaries (classic ifupdown ignores 2nd+ address lines; the hooks are honored). Backup at /etc/network/interfaces.bak-*.
3. Belt-and-suspenders systemd oneshot pw-mail-ips.service (in repo at infra/mail/pw-mail-ips.service) re-binds the IPs + flushes the queue on boot.
4. Watchdog cron */5 pw-mail-ip-watchdog (repo infra/mail/) re-binds any missing sending IP and flushes if it had to act or sees Cannot assign lines.

Lesson / TODO. The host does unattended-upgrade reboots ~weekly (seen 05-25, 05-30, 06-24, all ~04:04). Any IP/transport change must be reboot-tested. Consider migrating ifupdown -> netplan with all addresses, or pin unattended-upgrades to skip auto-reboot. The mail_reputation_monitor.py attributes egress to .71 as "transactional default" -- after this incident, a spike of .71 egress in the bulk streams is itself an alarm.