Isolates bulk sending reputation onto a dedicated subdomain so the root domain
stays clean for transactional/verification mail (and recovers faster). Replies
still go to the root domain via Reply-To, so the customer-facing reply experience
is unchanged.
- build_trucking_campaigns.py: add env-overridable FROM_EMAIL
(noreply@send.performancewest.net); use it for both scheduled + test sends
instead of inheriting base["from_email"] from the DB base campaign.
- build_healthcare_campaigns_cron.py: FROM_EMAIL ->
compliance@send.performancewest.net (env-overridable).
- bounce-watcher.sh / hc-bounce-watcher.sh: track the new subdomain envelope
sender (keep legacy root-domain sender so the pre-cutover queue still drains;
HC also tracks by hcout transport regardless of sender).
Infra already live (separate, non-code): subdomain DNS (A/MX/SPF/DKIM
selector=send/DMARC p=reject) on the Hestia master, OpenDKIM signs
d=send.performancewest.net (verified end-to-end), egress .94/.107. Root SPF
trimmed to the real IPs; pointless IP-rehab cron disabled.
Replaces the panic-era burner-domain verification plan with an in-house
automatic catch-all rollout in the trucking/IFTA/UCR builders. Root-cause
classification of the 75k pre-DKIM-fix bounces showed ~55% were reputation/
auth (now fixed by DKIM signing) and only ~29% genuinely-dead mailboxes;
catch-all domains accept at RCPT time so they do not user-unknown bounce at
send, making a controlled in-house bleed safer than warming a separate burner.
catch_all_enabled() adds catch-all results only when warmup_day >=
CAMPAIGN_CATCH_ALL_MIN_DAY (21) AND the recent 2-day live bounce rate is below
CAMPAIGN_CATCH_ALL_MAX_BOUNCE_PCT (8%) on a >=300-sent sample; auto-reverts to
the clean smtp_valid/send_confirmed pool on the next run if bounces spike.
Short window so a past disaster cannot block the rollout forever and a fresh
spike trips fast. CAMPAIGN_INCLUDE_CATCH_ALL=1/0 still hard-overrides.
USABLE_FILTER (static) -> usable_filter() (per-run, memoized, one DB probe).
IFTA/UCR SELECT_SQL -> _select_sql() so tc.usable_filter() resolves at call
time, not import. 13 logic unit tests pass; live dry-run decision = OFF
(day 15 < 21 and recent 2d bounce 42% from the aging-out Jun-16 disaster).
The daily 40%-off coupon was being merged into every trucking/UCR/IFTA/OTC
send, but those discount sends were not actually being delivered (the
DKIM-broken window). Now that deliverability is fixed, re-test whether
normal-price offers convert before giving margin away.
New CAMPAIGN_ENABLE_COUPON env flag (default OFF) gates daily-coupon
minting in build_trucking_campaigns + the UCR/IFTA/OTC builders (which
import it as tc.COUPON_ENABLED). With it off, no code is minted and an
empty coupon_code is merged -> the campaign templates' existing
{{ if .Subscriber.Attribs.coupon_code }} guard falls through to the
normal-price {{ else }} branch and landing-page links carry no ?code=.
No template or DB changes; fully reversible (set CAMPAIGN_ENABLE_COUPON=1).
Verified: COUPON_ENABLED defaults False, coupon_attribs(None) -> empty,
lp_link drops ?code= when no coupon, all 4 builders compile.
Two deliverability hardening fixes from the email audit:
1. Plaintext (altbody): all campaigns were HTML-only. Listmonk only emits
multipart/alternative when altbody is set, and HTML-only bulk mail is a
spam-score signal. New scripts/_email_plaintext.py renders a readable
text/plain part from the HTML body (dependency-free; preserves Listmonk
{{ .Subscriber }}/{{ UnsubscribeURL }} template tags, turns links into
'text (url)'). Wired into the trucking builder (and thus UCR + IFTA, which
reuse create_and_schedule_campaign) and the healthcare builder.
2. Stable container hostname: Listmonk derived its Message-ID from the random
docker container id -> @localhost.localdomain (spam-score signal). Pin both
listmonk + listmonk-hc hostname to perfwest.performancewest.net, matching
Listmonk's SMTP hello_hostname.
Part of the email-deliverability incident hardening.
The smtp_valid pool is only ~3k unsent — too small to sustain campaigns. SMTP
probing can't confirm catch-all/mx_unreachable deliverability; only a REAL send
can. burner_list_verify.py reconciles a verification send from a DISPOSABLE burner
domain (isolated from PW/carrierone reputation):
- hard bounce -> fmcsa_carriers.email_verify_result='hard_bounced' (excluded)
- delivered -> 'send_confirmed' (proven deliverable; PW campaigns send to it)
It tails the burner MTA mail.log (reuses bounce-watcher's status= pattern) and
writes back idempotently. The PW trucking filter now treats smtp_valid +
send_confirmed as sendable. docs/campaign-deliverability-plan.md captures the full
diagnosis, the burner design, and CAN-SPAM guardrails.
Remaining (needs a domain + isolated MTA identity — operator/infra decision):
stand up the burner domain, the verification-send worker, and a writeback cron.
Root cause of zero conversions since Jun 9 + the Gmail/Outlook block storm:
the send filter was '(email_verified IS TRUE OR result IN ...)'. The verifier
sets email_verified=TRUE optimistically for mx_unreachable (domain exists but
its mail server never answered the RCPT probe) — 438,163 such rows. Those HARD
BOUNCE on send, producing ~1,100 bounces/day (~47% rate) and blocklisting half
the 120k subscriber base, so real prospects never saw the offer.
Fix: key the send filter ONLY off email_verify_result, never the broken boolean.
Recovery mode (default): send only 'smtp_valid' to drive bounce rate to ~0 and
rebuild reputation; set CAMPAIGN_INCLUDE_CATCH_ALL=1 to re-add catch-all domains
once recovered. Mirrors the healthcare list-cleaning approach (HC bounces ~2-3%,
which proves the fix). Note: only ~3k smtp_valid unsent remain — list growth via
real-send bounce verification (separate burner domain) is the follow-up.
Main pool is calendar-day 12 but reputation is wrecked (54% delivery, Gmail+
Outlook blocks) -- NOT warmed. MX tagging confirmed the cause: 702k carriers on
Google + 135k on Microsoft = the warmup was hammering exactly the two operators
blocking us. Hold Google/MS/Proofpoint/etc. OUT entirely until day 30 (configurable),
sending only to the long-tail operators (yahoo/comcast/charter/centurylink/etc.)
that don't bot-throttle, so reputation can recover; then re-introduce big
operators gradually via mx_daily_caps. 1.24M/1.49M carriers now tagged.
The Jun 13-14 Gmail+Outlook block storm came from the main/trucking pool having
NO per-MX throttling (only HC had it) -- it concentrated warmup volume on
Google/Microsoft-Workspace-hosted business domains. Port the HC fix:
- migration 097: fmcsa_carriers.mx_provider column.
- mx_tag_carriers.py: resolve MX once per distinct domain (reuses the verifier's
classifier+cache), tag every carrier with that domain's operator. Bounded per
run, prioritizes unsent verified carriers.
- build_trucking_campaigns: during warmup (day<=6) EXCLUDE tagged Google/MS/
Proofpoint/etc. carriers in fetch_carriers; per-MX cap in select_sendable_
carriers so known operators never dominate the quota. Untagged carriers pass
(not collapsed onto one bucket) until tagging fills in. mx_daily_caps ramps
with the main warmup day; MAIN_SKIP_BIG_MX=0 disables once warmed.
add_subscriber only attached existing subscribers to the new list without
updating attribs, so a carrier emailed in a prior campaign kept STALE attribs --
meaning the daily coupon code and IFTA merge fields (ifta_due_date, ifta_quarter,
lp_link) rendered BLANK for any repeat recipient. Now merge+PUT the fresh attribs
on the existing subscriber before attaching. Affects all trucking campaigns, not
just IFTA. Verified: IFTA preview now persists ifta_quarter/ifta_due_date/lp_link.
The sales we got came at $79 + a 24hr coupon; cutting MCS-150 to $39 flat
removed urgency and conversions did NOT improve (a permanent low price sets a
new anchor and lets people defer). Restore the higher anchor and let an
expiring discount create the now-or-lose-it decision.
- Restore MCS-150 anchor $39 -> $79 (catalog single source + regenerated).
- build_trucking_campaigns.py: mint ONE random 5-letter coupon per send-day
(40% off, valid through 23:59:59 ET that day) into the existing discount_codes
table; inject coupon_code/pct/expires + a ?code= LP link into every email.
Idempotent per day; service-fee-only scope (gov/pass-through fees never cut).
- Listmonk MCS-150 (186) + Inactive USDOT (188) templates: lead with the
struck-through anchor + sale price + code + 'expires tonight', and point the
primary CTA at the order page (with code) instead of the 'free check' tool.
- OrderPriceBanner: validates ?code= via /api/v1/discount and shows
was/now + expiry; Wizard forwards the code to order creation.
- Verified: code gen, expiry math, scope enforcement, discount API
(40% off $79 = $47.40), site+api builds clean.
- Wizard Finish button: for visitors with no token/order (e.g. arriving via a
campaign ?dot= link), create the compliance order from collected intake data
and redirect to Stripe Checkout, instead of silently doing nothing.
- StateTrucking: Operating States no longer required; single-state/intrastate
carriers can finish (relabeled 'Other Operating States (if any)').
- build_trucking_campaigns: per-state programs (weight_tax/emissions) now derive
the CTA landing page from the deficiency flag's state suffix (e.g.
state_weight_tax_OR -> OR), not the carrier base state, so a GA-based carrier
flagged for OR weight-mile tax links to the OR page (not a mismatched one).
send_test() replaced company/dot/state placeholders but not
{{ .Subscriber.Attribs.lp_link }}, so the CTA button (Check My Emissions
Status, Register My Tax Account, etc.) rendered as a bare '?dot=...' that
linked to nowhere in every owner test/approval email. Real subscribers were
unaffected (their lp_link attrib is populated). Now the test mirrors the real
audience link via build_lp_link(campaign_type, state).
Lets us fire small, controlled batches (e.g. MCS-150 only, 100/tz, sent today)
while the new sending IPs warm up, instead of the full multi-segment schedule.
Yahoo operates a large family of consumer domains (AOL, AT&T, Verizon,
Frontier, sbcglobal, bellsouth, etc.) that aggressively defer cold senders
with 421 'unexpected volume / user complaints', which poisons our self-hosted
sending IP for every other provider. Previously we only excluded
aol.com/yahoo.com/ymail.com/rocketmail.com.
Centralize the authoritative block list in scripts/_email_exclusions.py and
import it from both audience builders so they stay in sync.
Add 6 flag-based campaign segments to build_trucking_campaigns.py keyed off
fmcsa_carriers.deficiency_flags: for_hire_boc3, irp_ifta, intrastate_authority,
state_weight_tax (per-state LP), state_emissions (CA->ca-mcp-carb), hazmat.
Each injects an order-LP link into subscriber attribs (lp_link) and only
schedules when its CAMPAIGN_*_ID source template env is set (nightly run never
breaks on unconfigured templates). Adds --list-segments audience report and a
synthetic-data segment test (fixed a real psycopg2 % escaping bug in LIKE).
- send_test no longer overwrites {{ UnsubscribeURL }} with a dead static URL;
Listmonk renders it into a working per-subscriber unsubscribe link.
- dot-compliance hero grid: 4 columns (minmax 150px, max-width 920px) instead
of 3 to reduce vertical space.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- import_subscribers: was POSTing wrong bulk shape AND fallback used 'list_ids'
(ignored by Listmonk) instead of 'lists' -> subscribers never attached to the
list -> real sends would go to an empty list. Now single-adds with 'lists',
handles already-exists, returns a count, logs if 0 added.
- send_test: passed base['lists'] (objects) instead of IDs -> test send rejected.
Now extracts list IDs.
- create_and_schedule_campaign: add schedule= flag (preview makes drafts).
- --preview: 1 sample carrier/campaign, only owner email, drafts not scheduled,
test sends immediately, never marks real carriers sent.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Each of the 8 daily campaigns gets a test send immediately after creation
using the first row's real carrier data as the sample. TEST_EMAIL env var
overrides the default (carrierone@gmx.com).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- build_trucking_campaigns.py: nightly script that creates 8 Listmonk campaigns
per day (4 TZ x 2 types: MCS-150 overdue 2k/TZ, inactive USDOT 1k/TZ)
at 4AM ET / 5AM ET (CT) / 6AM ET (MT) / 7AM ET (PT). Deduplicates via
listmonk_sent_at column.
- migration 083: add listmonk_sent_at + listmonk_campaign_type to fmcsa_carriers
- email_verifier.py: bump max_workers from 5 to 20 for 4x faster throughput
- cron: daily pw-trucking-campaigns at 08:00 UTC (3 AM EST)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
