Two routing bugs that sent carriers to wrong/dead order pages:
1. MCS-150 + Inactive campaigns linked to /order/dot-full-compliance ($399)
instead of their actual service: build_lp_link()/lp_slug_for() fell through
to the dot-full-compliance catch-all for any campaign_type not in
DEFICIENCY_SEGMENTS, ignoring the existing PRICE_SLUG_BY_CAMPAIGN map. So
MCS-150 carriers (should be mcs150-update $79) and Inactive carriers (should
be usdot-reactivation $149) were both quoted a 5x-priced bundle they never
asked for — a severe conversion killer on the two highest-volume segments.
Fix: lp_slug_for() now checks PRICE_SLUG_BY_CAMPAIGN first; build_lp_link()
delegates to it (single source of truth).
2. IFTA-quarterly + UCR-annual builders set lp_link to a BARE path when no
coupon was active (LP_LINK with no query). The body appends '&utm_source=...'
so the CTA rendered as /order/ifta-quarterly&utm... (no '?') = 404. Fix:
both now always emit a leading '?' query carrying ?dot= (and ?code= when a
coupon is on), mirroring the main builder's lp_link_with_coupon().
Audited every campaign_type: all 14 order slugs now resolve 200 and match the
intended service/price. Compliance-check secondary links (/tools/dot-compliance-
check) verified correct and intentionally kept where a 'check status' CTA fits.
Root cause of the order-CTA 404s recurring after the prior live fix: the
builder clones email bodies from STORED Listmonk source campaigns (ids
186/188/271-274/309/310/469/473), not from the edited source files. Those
stored bodies still carried @TrackLink on the per-subscriber order CTA, so
every nightly build re-registered a single static /order/<slug>&utm... link
(no '?') that 404s for every recipient. This morning's 3,000 real sends AND
the owner spot-check both went out with dead order links.
Two durable guards:
1. get_base_campaign() now strips @TrackLink from any cloned body (with a
warning), so a stale/re-edited source campaign can never reach recipients
broken again. Human clicks are already attributed via Umami.
2. The owner test-send now builds the CTA via lp_link_with_coupon(dot=...)
(leading '?') instead of build_lp_link() (bare path).
Also fixed live: stripped @TrackLink from the 10 stored source campaign
bodies; rewrote the 12 already-registered broken links. Backups in listmonk:
pw_source_tracklink_bak_20260623 + pw_links_tracklink_bak_20260623.
After the Jun 2026 no-DKIM incident (campaign mail went out unsigned ->
junked/blocked, ~23% delivery), DKIM is fixed and we must re-send to the
now-signed audience. The builder previously held Google AND Microsoft AND
consumer-MX out until warmup day 30; that blocks the re-send of the Microsoft-
hosted business domains that are most of the list.
Add MAIN_EXCLUDE_OPERATORS (comma-separated mx_provider labels) to override
WARMUP_EXCLUDE_OPERATORS. Set it to 'google' in the workers env so we send to
everything EXCEPT Google's consumer inboxes (still recovering reputation),
including Microsoft/Hotmail. Drives both the SQL exclude and the per-operator
daily cap consistently. Unset => prior default; '' => exclude nobody.
Listmonk applies campaign headers as `for hdr,val := range set { h.Add(hdr,val) }`
(internal/manager/manager.go v6.1.0): each map's KEY is the literal header name.
The trucking/CRTC/deficiency builders wrote {"name":"Reply-To","value":..} (and
{"key":..,"value":..}), which emits junk `name:`/`value:` headers and NO real
Reply-To, so replies fell back to the From address (noreply@send.performancewest.net)
instead of info@performancewest.net. HC builder already used the correct
{"Reply-To": value} shape; match it everywhere. Verified against listmonk source.
Impact: outbound only; no customer replies were lost (noreply@ is a real mailbox),
but reply UX pointed at a no-reply address. Live campaign headers re-patched separately.
CAMPAIGN_COUPON_AB_PCTS="20,30,0" now means 20% / 30% / full-price. The 0 arm
mints no code; pick_coupon_for_email returns ("","") so it renders identically
to a normal-price send, while carriers are still deterministically hash-bucketed
into it (re-hash a converter's email to recover their arm). Even ~33/33/33 split
incl. the control verified over 30k. Adds test_full_price_control_arm; 8/8 pass.
Two correctness fixes that gate enabling the coupon test:
1. On-the-fly pricing. The coupon block hardcoded '$79 $47' (only true at 40%
off) — a false claim on the 20/30% arms. Now build_trucking_campaigns.py
reads api/src/service-catalog.ts (same source checkout uses) and computes
coupon_price_full / coupon_price_deal per recipient as full - round(full*pct/100),
exactly matching the server. Service-fee-only; non-discountable services
(boc3-filing passthrough) get NO price and fall back to percent-only copy.
Quotes the service the email is ABOUT (mcs150 $79, reactivation $149), not the
bundle the CTA happens to link to. service-catalog.ts now ships in the worker
image; helper degrades to percent-only if it can't be read.
2. CTA URL bug (likely a big driver of the zero-click problem). Main campaign
CTAs render '/order/slug&utm_source=...' (no '?') -> HTTP 404, verified live.
Deficiency CTAs would double-'?' once a coupon added '?code='. lp_link now
owns the query (?dot=...&code=...) so every template appends with a leading
'&' and is valid in all 4 states (main/deficiency x coupon on/off), verified
against live URLs returning 200.
Deficiency _deal_box now shows real was/now prices (percent-only for boc3).
Tests: 7/7 pass (adds URL-wellformed + price-matches-checkout cases).
- CAMPAIGN_COUPON_AB_PCTS="20,30,40" mints one daily code per arm; each
carrier is bucketed by a stable sha256(email) hash so the split is even
(~33/33/33 verified over 30k) and stable across re-sends (no arm-hopping).
- Each arm's code stores its own percent in discount_codes, so the advertised
discount always matches what checkout applies; redemptions are countable per
code (marker campaign-daily:<date>:<pct>).
- Empty/unset keeps legacy single-arm behavior (COUPON_PCT, legacy marker).
- coupon_attribs() now takes per-recipient pct.
- Tests: scripts/tests/test_coupon_ab.py (5 pass). SpamAssassin: both main
campaigns (186/188) score 0.0 HAM across all 3 arms, coupon block renders
clean; harness saved for re-runs.
Completes the MX-exclusion plan. Untagged carriers can't be excluded (the big-MX
gate is MX-based, so an unresolved Google/Yahoo domain would slip through), and
were previously UNCAPPED in select_sendable_carriers -- a flood of freshly-imported,
never-resolved domains could dominate a run before pw-mx-tag resolves them.
Added a single shared untagged_cap (env MAIN_UNTAGGED_MX_CAP, default max(quota,200))
so untagged sends are bounded without starving the pool: at the default the bucket
can still fill an entire run's quota (no behavior change today), but the cap can be
tightened to a fraction once pw-mx-tag has drained the backlog -- which is fast,
since only ~3,035 distinct *verified-sendable* untagged domains remain (< one
20k/day tag run). Tagged carriers keep their per-operator caps unchanged.
Verified: compiles; cap logic never starves at default, enforces the limit when
set lower.
Fix 1 (build_trucking_campaigns.py): the warmup big-MX exclusion only covered the
clean-label operators (google/microsoft/proofpoint/...). Consumer mailbox
operators that mx_tag_carriers.py labels with an "mx:" prefix slipped BOTH the
exclusion and the per-MX throttle -- notably mx:yahoodns.net (283k sendable
carriers = Yahoo Small Business/AOL custom domains) and mx:icloud.com (25k), plus
comcast/charter/centurylink/windstream/tds/earthlink. These are custom domains
whose MX points at a consumer provider, invisible to the literal-domain blocklist.
Added CONSUMER_MX_OPERATORS, folded into WARMUP_EXCLUDE_OPERATORS used by both the
fetch_carriers() exclusion SQL and mx_daily_caps() (same day-30 ramp). Behind the
existing MAIN_SKIP_BIG_MX switch.
Validated read-only: after the fix the warmup-eligible pool is 353,909 carriers
(315,892 untagged + ~38k genuinely small/self-hosted operators), so the long tail
still sustains the daily quota -- not starved -- while 0 consumer-MX carriers are
selected during warmup.
Fix 3 (infra/cron/pw-mx-tag): mx_tag_carriers.py was on no cron, so the untagged
(NULL) backlog (~316k) never drained and new FMCSA imports stayed untagged,
slowly re-opening the gap. Added a daily 05:45 UTC cron (--only-unsent
--limit-domains 20000), before the 08:00 builder. Idempotent/bounded (only tags
mx_provider IS NULL). Verified live: a 200-domain test run tagged 216 domains.
(Fix 2 -- bounding the NULL bucket cap -- deferred; the cron will drain it.)
Isolates bulk sending reputation onto a dedicated subdomain so the root domain
stays clean for transactional/verification mail (and recovers faster). Replies
still go to the root domain via Reply-To, so the customer-facing reply experience
is unchanged.
- build_trucking_campaigns.py: add env-overridable FROM_EMAIL
(noreply@send.performancewest.net); use it for both scheduled + test sends
instead of inheriting base["from_email"] from the DB base campaign.
- build_healthcare_campaigns_cron.py: FROM_EMAIL ->
compliance@send.performancewest.net (env-overridable).
- bounce-watcher.sh / hc-bounce-watcher.sh: track the new subdomain envelope
sender (keep legacy root-domain sender so the pre-cutover queue still drains;
HC also tracks by hcout transport regardless of sender).
Infra already live (separate, non-code): subdomain DNS (A/MX/SPF/DKIM
selector=send/DMARC p=reject) on the Hestia master, OpenDKIM signs
d=send.performancewest.net (verified end-to-end), egress .94/.107. Root SPF
trimmed to the real IPs; pointless IP-rehab cron disabled.
Replaces the panic-era burner-domain verification plan with an in-house
automatic catch-all rollout in the trucking/IFTA/UCR builders. Root-cause
classification of the 75k pre-DKIM-fix bounces showed ~55% were reputation/
auth (now fixed by DKIM signing) and only ~29% genuinely-dead mailboxes;
catch-all domains accept at RCPT time so they do not user-unknown bounce at
send, making a controlled in-house bleed safer than warming a separate burner.
catch_all_enabled() adds catch-all results only when warmup_day >=
CAMPAIGN_CATCH_ALL_MIN_DAY (21) AND the recent 2-day live bounce rate is below
CAMPAIGN_CATCH_ALL_MAX_BOUNCE_PCT (8%) on a >=300-sent sample; auto-reverts to
the clean smtp_valid/send_confirmed pool on the next run if bounces spike.
Short window so a past disaster cannot block the rollout forever and a fresh
spike trips fast. CAMPAIGN_INCLUDE_CATCH_ALL=1/0 still hard-overrides.
USABLE_FILTER (static) -> usable_filter() (per-run, memoized, one DB probe).
IFTA/UCR SELECT_SQL -> _select_sql() so tc.usable_filter() resolves at call
time, not import. 13 logic unit tests pass; live dry-run decision = OFF
(day 15 < 21 and recent 2d bounce 42% from the aging-out Jun-16 disaster).
The daily 40%-off coupon was being merged into every trucking/UCR/IFTA/OTC
send, but those discount sends were not actually being delivered (the
DKIM-broken window). Now that deliverability is fixed, re-test whether
normal-price offers convert before giving margin away.
New CAMPAIGN_ENABLE_COUPON env flag (default OFF) gates daily-coupon
minting in build_trucking_campaigns + the UCR/IFTA/OTC builders (which
import it as tc.COUPON_ENABLED). With it off, no code is minted and an
empty coupon_code is merged -> the campaign templates' existing
{{ if .Subscriber.Attribs.coupon_code }} guard falls through to the
normal-price {{ else }} branch and landing-page links carry no ?code=.
No template or DB changes; fully reversible (set CAMPAIGN_ENABLE_COUPON=1).
Verified: COUPON_ENABLED defaults False, coupon_attribs(None) -> empty,
lp_link drops ?code= when no coupon, all 4 builders compile.
Two deliverability hardening fixes from the email audit:
1. Plaintext (altbody): all campaigns were HTML-only. Listmonk only emits
multipart/alternative when altbody is set, and HTML-only bulk mail is a
spam-score signal. New scripts/_email_plaintext.py renders a readable
text/plain part from the HTML body (dependency-free; preserves Listmonk
{{ .Subscriber }}/{{ UnsubscribeURL }} template tags, turns links into
'text (url)'). Wired into the trucking builder (and thus UCR + IFTA, which
reuse create_and_schedule_campaign) and the healthcare builder.
2. Stable container hostname: Listmonk derived its Message-ID from the random
docker container id -> @localhost.localdomain (spam-score signal). Pin both
listmonk + listmonk-hc hostname to perfwest.performancewest.net, matching
Listmonk's SMTP hello_hostname.
Part of the email-deliverability incident hardening.
The smtp_valid pool is only ~3k unsent — too small to sustain campaigns. SMTP
probing can't confirm catch-all/mx_unreachable deliverability; only a REAL send
can. burner_list_verify.py reconciles a verification send from a DISPOSABLE burner
domain (isolated from PW/carrierone reputation):
- hard bounce -> fmcsa_carriers.email_verify_result='hard_bounced' (excluded)
- delivered -> 'send_confirmed' (proven deliverable; PW campaigns send to it)
It tails the burner MTA mail.log (reuses bounce-watcher's status= pattern) and
writes back idempotently. The PW trucking filter now treats smtp_valid +
send_confirmed as sendable. docs/campaign-deliverability-plan.md captures the full
diagnosis, the burner design, and CAN-SPAM guardrails.
Remaining (needs a domain + isolated MTA identity — operator/infra decision):
stand up the burner domain, the verification-send worker, and a writeback cron.
Root cause of zero conversions since Jun 9 + the Gmail/Outlook block storm:
the send filter was '(email_verified IS TRUE OR result IN ...)'. The verifier
sets email_verified=TRUE optimistically for mx_unreachable (domain exists but
its mail server never answered the RCPT probe) — 438,163 such rows. Those HARD
BOUNCE on send, producing ~1,100 bounces/day (~47% rate) and blocklisting half
the 120k subscriber base, so real prospects never saw the offer.
Fix: key the send filter ONLY off email_verify_result, never the broken boolean.
Recovery mode (default): send only 'smtp_valid' to drive bounce rate to ~0 and
rebuild reputation; set CAMPAIGN_INCLUDE_CATCH_ALL=1 to re-add catch-all domains
once recovered. Mirrors the healthcare list-cleaning approach (HC bounces ~2-3%,
which proves the fix). Note: only ~3k smtp_valid unsent remain — list growth via
real-send bounce verification (separate burner domain) is the follow-up.
Main pool is calendar-day 12 but reputation is wrecked (54% delivery, Gmail+
Outlook blocks) -- NOT warmed. MX tagging confirmed the cause: 702k carriers on
Google + 135k on Microsoft = the warmup was hammering exactly the two operators
blocking us. Hold Google/MS/Proofpoint/etc. OUT entirely until day 30 (configurable),
sending only to the long-tail operators (yahoo/comcast/charter/centurylink/etc.)
that don't bot-throttle, so reputation can recover; then re-introduce big
operators gradually via mx_daily_caps. 1.24M/1.49M carriers now tagged.
The Jun 13-14 Gmail+Outlook block storm came from the main/trucking pool having
NO per-MX throttling (only HC had it) -- it concentrated warmup volume on
Google/Microsoft-Workspace-hosted business domains. Port the HC fix:
- migration 097: fmcsa_carriers.mx_provider column.
- mx_tag_carriers.py: resolve MX once per distinct domain (reuses the verifier's
classifier+cache), tag every carrier with that domain's operator. Bounded per
run, prioritizes unsent verified carriers.
- build_trucking_campaigns: during warmup (day<=6) EXCLUDE tagged Google/MS/
Proofpoint/etc. carriers in fetch_carriers; per-MX cap in select_sendable_
carriers so known operators never dominate the quota. Untagged carriers pass
(not collapsed onto one bucket) until tagging fills in. mx_daily_caps ramps
with the main warmup day; MAIN_SKIP_BIG_MX=0 disables once warmed.
add_subscriber only attached existing subscribers to the new list without
updating attribs, so a carrier emailed in a prior campaign kept STALE attribs --
meaning the daily coupon code and IFTA merge fields (ifta_due_date, ifta_quarter,
lp_link) rendered BLANK for any repeat recipient. Now merge+PUT the fresh attribs
on the existing subscriber before attaching. Affects all trucking campaigns, not
just IFTA. Verified: IFTA preview now persists ifta_quarter/ifta_due_date/lp_link.
The sales we got came at $79 + a 24hr coupon; cutting MCS-150 to $39 flat
removed urgency and conversions did NOT improve (a permanent low price sets a
new anchor and lets people defer). Restore the higher anchor and let an
expiring discount create the now-or-lose-it decision.
- Restore MCS-150 anchor $39 -> $79 (catalog single source + regenerated).
- build_trucking_campaigns.py: mint ONE random 5-letter coupon per send-day
(40% off, valid through 23:59:59 ET that day) into the existing discount_codes
table; inject coupon_code/pct/expires + a ?code= LP link into every email.
Idempotent per day; service-fee-only scope (gov/pass-through fees never cut).
- Listmonk MCS-150 (186) + Inactive USDOT (188) templates: lead with the
struck-through anchor + sale price + code + 'expires tonight', and point the
primary CTA at the order page (with code) instead of the 'free check' tool.
- OrderPriceBanner: validates ?code= via /api/v1/discount and shows
was/now + expiry; Wizard forwards the code to order creation.
- Verified: code gen, expiry math, scope enforcement, discount API
(40% off $79 = $47.40), site+api builds clean.
- Wizard Finish button: for visitors with no token/order (e.g. arriving via a
campaign ?dot= link), create the compliance order from collected intake data
and redirect to Stripe Checkout, instead of silently doing nothing.
- StateTrucking: Operating States no longer required; single-state/intrastate
carriers can finish (relabeled 'Other Operating States (if any)').
- build_trucking_campaigns: per-state programs (weight_tax/emissions) now derive
the CTA landing page from the deficiency flag's state suffix (e.g.
state_weight_tax_OR -> OR), not the carrier base state, so a GA-based carrier
flagged for OR weight-mile tax links to the OR page (not a mismatched one).
send_test() replaced company/dot/state placeholders but not
{{ .Subscriber.Attribs.lp_link }}, so the CTA button (Check My Emissions
Status, Register My Tax Account, etc.) rendered as a bare '?dot=...' that
linked to nowhere in every owner test/approval email. Real subscribers were
unaffected (their lp_link attrib is populated). Now the test mirrors the real
audience link via build_lp_link(campaign_type, state).
Lets us fire small, controlled batches (e.g. MCS-150 only, 100/tz, sent today)
while the new sending IPs warm up, instead of the full multi-segment schedule.
Yahoo operates a large family of consumer domains (AOL, AT&T, Verizon,
Frontier, sbcglobal, bellsouth, etc.) that aggressively defer cold senders
with 421 'unexpected volume / user complaints', which poisons our self-hosted
sending IP for every other provider. Previously we only excluded
aol.com/yahoo.com/ymail.com/rocketmail.com.
Centralize the authoritative block list in scripts/_email_exclusions.py and
import it from both audience builders so they stay in sync.
Add 6 flag-based campaign segments to build_trucking_campaigns.py keyed off
fmcsa_carriers.deficiency_flags: for_hire_boc3, irp_ifta, intrastate_authority,
state_weight_tax (per-state LP), state_emissions (CA->ca-mcp-carb), hazmat.
Each injects an order-LP link into subscriber attribs (lp_link) and only
schedules when its CAMPAIGN_*_ID source template env is set (nightly run never
breaks on unconfigured templates). Adds --list-segments audience report and a
synthetic-data segment test (fixed a real psycopg2 % escaping bug in LIKE).
- send_test no longer overwrites {{ UnsubscribeURL }} with a dead static URL;
Listmonk renders it into a working per-subscriber unsubscribe link.
- dot-compliance hero grid: 4 columns (minmax 150px, max-width 920px) instead
of 3 to reduce vertical space.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- import_subscribers: was POSTing wrong bulk shape AND fallback used 'list_ids'
(ignored by Listmonk) instead of 'lists' -> subscribers never attached to the
list -> real sends would go to an empty list. Now single-adds with 'lists',
handles already-exists, returns a count, logs if 0 added.
- send_test: passed base['lists'] (objects) instead of IDs -> test send rejected.
Now extracts list IDs.
- create_and_schedule_campaign: add schedule= flag (preview makes drafts).
- --preview: 1 sample carrier/campaign, only owner email, drafts not scheduled,
test sends immediately, never marks real carriers sent.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Each of the 8 daily campaigns gets a test send immediately after creation
using the first row's real carrier data as the sample. TEST_EMAIL env var
overrides the default (carrierone@gmx.com).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- build_trucking_campaigns.py: nightly script that creates 8 Listmonk campaigns
per day (4 TZ x 2 types: MCS-150 overdue 2k/TZ, inactive USDOT 1k/TZ)
at 4AM ET / 5AM ET (CT) / 6AM ET (MT) / 7AM ET (PT). Deduplicates via
listmonk_sent_at column.
- migration 083: add listmonk_sent_at + listmonk_campaign_type to fmcsa_carriers
- email_verifier.py: bump max_workers from 5 to 20 for 4x faster throughput
- cron: daily pw-trucking-campaigns at 08:00 UTC (3 AM EST)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
