117 lines
5.6 KiB
Markdown
117 lines
5.6 KiB
Markdown
# Email Deliverability & IP Warmup Runbook
|
|
|
|
Performance West self-hosts its outbound MTA (Postfix on the app server) because
|
|
transactional relays (SES, Postmark, SendGrid) forbid the cold prospecting email
|
|
our FMCSA trucking and telecom campaigns depend on. That means **we own our
|
|
sending-IP reputation** and must manage it manually. This doc is the operational
|
|
guide for keeping it healthy.
|
|
|
|
## Infrastructure layout
|
|
|
|
- **Host Postfix** on the app server (`207.174.124.71`), reached by Listmonk via
|
|
SMTP at `172.18.0.1:25`.
|
|
- **Sending IPs:** `207.174.124.90` through `.109` (20 IPs), each with valid
|
|
FCrDNS (`mtaNN.performancewest.net`) and authorized in SPF (`-all`).
|
|
- `.90` / `mta01`: historically a dedicated Yahoo trickle IP. We no longer mail
|
|
Yahoo at all, so it is idle.
|
|
- `.91-.109` / `mta02-mta20`: rotation pool, selected via
|
|
`transport_maps = hash:/etc/postfix/transport, randmap:{<active pool>}`.
|
|
- **Warmup scheduler:** `/usr/local/bin/pw-mta-warmup` (daily cron
|
|
`/etc/cron.d/pw-mta-warmup`, 07:17 UTC). Recomputes the active rotation pool
|
|
from a start date stamped in `/etc/postfix/pw-warmup-start`. Ramp schedule:
|
|
day 0-3 -> 3 IPs, 4-7 -> 5, 8-11 -> 8, 12-17 -> 12, 18-24 -> 16, 25+ -> 19.
|
|
The pool only ever grows. It picks IPs from the front of the `ALL=(...)` array.
|
|
|
|
## What we do NOT mail
|
|
|
|
The **Yahoo / Verizon-Media family** is excluded entirely (yahoo, aol, att,
|
|
verizon, frontier, sbcglobal, bellsouth, pacbell, ameritech, ymail, rocketmail,
|
|
aim, netscape, compuserve, etc.). They aggressively defer cold senders with
|
|
`421 4.7.0 [TSS04] ... unexpected volume or user complaints`, and that deferral
|
|
poisons the sending IP for Gmail and Microsoft too.
|
|
|
|
Enforced in two layers:
|
|
1. **Audience build** (authoritative): `scripts/_email_exclusions.py`
|
|
(`BLOCKED_EMAIL_DOMAINS`), imported by `build_trucking_campaigns.py` and
|
|
`populate_new_carrier_startup_campaign.py`. New campaigns never include them.
|
|
2. **Postfix backstop:** `/etc/postfix/transport` maps every Yahoo-family domain
|
|
to `hold:`. If any leak into the queue they are parked, never sent from a
|
|
rotation IP.
|
|
|
|
## Incident: May 30-31 2026 reputation collapse
|
|
|
|
A campaign blast pushed ~29k sends in a day across cold IPs `.91/.92/.93` with no
|
|
daily volume cap. Result:
|
|
- Gmail: `550-5.7.1 ... likely unsolicited mail` (hard spam block).
|
|
- Yahoo: `421 TSS04` on the rotation IPs.
|
|
- Steady state afterward: ~13% delivery (10k sent vs 68k deferred + 7k bounced
|
|
in a day). Listmonk open rate ~4%, clicks ~0.
|
|
|
|
### Remediation (Jun 02 2026)
|
|
- **Retired the 3 burned IPs** (`.91/.92/.93` = out02/03/04) from rotation.
|
|
Confirmed `.94-.109` had never sent outbound (only inbound port-scan noise),
|
|
so they are pristine.
|
|
- **Swapped rotation to fresh `.94/.95/.96`** (out05/06/07) and reset the warmup
|
|
start date to day 0.
|
|
- **Patched `pw-mta-warmup`** `ALL` array to start at `out05` so the daily cron
|
|
never reverts to the burned IPs.
|
|
- **Rewrote `/etc/postfix/transport`** to `hold:` the full Yahoo family (was a
|
|
partial list with buggy duplicate keys routing to `yahooslow`).
|
|
- **Flushed the entire stale queue** (1,846 blast-era messages, mostly dead
|
|
satellite ISPs) so fresh IPs start clean.
|
|
- **Enabled Listmonk sliding-window rate limit** so no campaign can blast again:
|
|
`app.message_sliding_window=true`, duration `1h`, rate `50`, `message_rate=2`.
|
|
- **Paused 19 trucking campaigns** (IDs 275-293, ~13k recipients) that were
|
|
scheduled to fire Jun 03; they were built before the exclusion fix and would
|
|
have re-torched the fresh IPs. Rebuild them small/clean before resending.
|
|
|
|
## Fresh-IP warmup discipline (the rules)
|
|
|
|
The historical mail.log proves these IPs sustain ~2,500 sends/day at 68-76%
|
|
delivery once warm (May 19-21). Collapses only ever came from 17k-29k spikes.
|
|
So we ramp ASSERTIVELY but never spike. The Listmonk sliding-window cap
|
|
(`/usr/local/bin/pw-listmonk-rampcap`, daily cron 07:20 UTC, driven off the same
|
|
`/etc/postfix/pw-warmup-start` stamp) enforces this automatically:
|
|
|
|
| warmup day | hourly cap | ~daily total |
|
|
|-----------:|-----------:|-------------:|
|
|
| 0-1 | 50/h | ~500 |
|
|
| 2-3 | 150/h | ~1,500 |
|
|
| 4-6 | 250/h | ~2,500 |
|
|
| 7+ | 300/h | ~3,000 (hard ceiling) |
|
|
|
|
Hard rule from the data: **never exceed ~4k/day, never spike.**
|
|
|
|
Other rules:
|
|
1. **Best recipients first.** Gmail + Microsoft + clean ISPs only (Yahoo family
|
|
already excluded). Send small focused batches, e.g.
|
|
`build_trucking_campaigns --only-segment mcs150 --max-per-segment 100 --date <today> --send-hour <H>`.
|
|
2. **Scrub hard bounces immediately.** `550 5.1.1`, full mailbox, "not our
|
|
customer" all hurt reputation signals.
|
|
3. **Watch the signals daily** (see commands below). If Gmail `550-5.7.1` or
|
|
Yahoo `421 TSS04` reappear, STOP and hold for several days.
|
|
|
|
## Monitoring commands
|
|
|
|
```bash
|
|
# delivery mix today
|
|
sudo grep "^$(date '+%b %d')" /var/log/mail.log | grep -oE 'status=(sent|deferred|bounced)' | sort | uniq -c
|
|
|
|
# per-IP outbound volume today (catch a runaway blast early)
|
|
for ip in 94 95 96; do echo -n ".$ip: "; sudo grep "^$(date '+%b %d')" /var/log/mail.log | grep -c "207.174.124.$ip"; done
|
|
|
|
# top deferral / bounce reasons today
|
|
sudo grep "^$(date '+%b %d')" /var/log/mail.log | grep status=deferred | grep -oE 'said: [0-9]{3}[^)]{0,50}' | sort | uniq -c | sort -rn | head
|
|
|
|
# queue size
|
|
sudo postqueue -p | tail -1
|
|
|
|
# active rotation pool + warmup day
|
|
sudo postconf -h transport_maps
|
|
echo $(( ($(date +%s) - $(sudo cat /etc/postfix/pw-warmup-start)) / 86400 ))
|
|
```
|
|
|
|
## Backups left on the server (Jun 02 2026 remediation)
|
|
- `/etc/postfix/main.cf.bak.*`
|
|
- `/etc/postfix/transport.bak.*`
|
|
- `/usr/local/bin/pw-mta-warmup.bak.*`
|