The smtp_valid pool is only ~3k unsent — too small to sustain campaigns. SMTP
probing can't confirm catch-all/mx_unreachable deliverability; only a REAL send
can. burner_list_verify.py reconciles a verification send from a DISPOSABLE burner
domain (isolated from PW/carrierone reputation):
- hard bounce -> fmcsa_carriers.email_verify_result='hard_bounced' (excluded)
- delivered -> 'send_confirmed' (proven deliverable; PW campaigns send to it)
It tails the burner MTA mail.log (reuses bounce-watcher's status= pattern) and
writes back idempotently. The PW trucking filter now treats smtp_valid +
send_confirmed as sendable. docs/campaign-deliverability-plan.md captures the full
diagnosis, the burner design, and CAN-SPAM guardrails.
Remaining (needs a domain + isolated MTA identity — operator/infra decision):
stand up the burner domain, the verification-send worker, and a writeback cron.
Root cause of zero conversions since Jun 9 + the Gmail/Outlook block storm:
the send filter was '(email_verified IS TRUE OR result IN ...)'. The verifier
sets email_verified=TRUE optimistically for mx_unreachable (domain exists but
its mail server never answered the RCPT probe) — 438,163 such rows. Those HARD
BOUNCE on send, producing ~1,100 bounces/day (~47% rate) and blocklisting half
the 120k subscriber base, so real prospects never saw the offer.
Fix: key the send filter ONLY off email_verify_result, never the broken boolean.
Recovery mode (default): send only 'smtp_valid' to drive bounce rate to ~0 and
rebuild reputation; set CAMPAIGN_INCLUDE_CATCH_ALL=1 to re-add catch-all domains
once recovered. Mirrors the healthcare list-cleaning approach (HC bounces ~2-3%,
which proves the fix). Note: only ~3k smtp_valid unsent remain — list growth via
real-send bounce verification (separate burner domain) is the follow-up.
Main pool is calendar-day 12 but reputation is wrecked (54% delivery, Gmail+
Outlook blocks) -- NOT warmed. MX tagging confirmed the cause: 702k carriers on
Google + 135k on Microsoft = the warmup was hammering exactly the two operators
blocking us. Hold Google/MS/Proofpoint/etc. OUT entirely until day 30 (configurable),
sending only to the long-tail operators (yahoo/comcast/charter/centurylink/etc.)
that don't bot-throttle, so reputation can recover; then re-introduce big
operators gradually via mx_daily_caps. 1.24M/1.49M carriers now tagged.
The Jun 13-14 Gmail+Outlook block storm came from the main/trucking pool having
NO per-MX throttling (only HC had it) -- it concentrated warmup volume on
Google/Microsoft-Workspace-hosted business domains. Port the HC fix:
- migration 097: fmcsa_carriers.mx_provider column.
- mx_tag_carriers.py: resolve MX once per distinct domain (reuses the verifier's
classifier+cache), tag every carrier with that domain's operator. Bounded per
run, prioritizes unsent verified carriers.
- build_trucking_campaigns: during warmup (day<=6) EXCLUDE tagged Google/MS/
Proofpoint/etc. carriers in fetch_carriers; per-MX cap in select_sendable_
carriers so known operators never dominate the quota. Untagged carriers pass
(not collapsed onto one bucket) until tagging fills in. mx_daily_caps ramps
with the main warmup day; MAIN_SKIP_BIG_MX=0 disables once warmed.
add_subscriber only attached existing subscribers to the new list without
updating attribs, so a carrier emailed in a prior campaign kept STALE attribs --
meaning the daily coupon code and IFTA merge fields (ifta_due_date, ifta_quarter,
lp_link) rendered BLANK for any repeat recipient. Now merge+PUT the fresh attribs
on the existing subscriber before attaching. Affects all trucking campaigns, not
just IFTA. Verified: IFTA preview now persists ifta_quarter/ifta_due_date/lp_link.
The sales we got came at $79 + a 24hr coupon; cutting MCS-150 to $39 flat
removed urgency and conversions did NOT improve (a permanent low price sets a
new anchor and lets people defer). Restore the higher anchor and let an
expiring discount create the now-or-lose-it decision.
- Restore MCS-150 anchor $39 -> $79 (catalog single source + regenerated).
- build_trucking_campaigns.py: mint ONE random 5-letter coupon per send-day
(40% off, valid through 23:59:59 ET that day) into the existing discount_codes
table; inject coupon_code/pct/expires + a ?code= LP link into every email.
Idempotent per day; service-fee-only scope (gov/pass-through fees never cut).
- Listmonk MCS-150 (186) + Inactive USDOT (188) templates: lead with the
struck-through anchor + sale price + code + 'expires tonight', and point the
primary CTA at the order page (with code) instead of the 'free check' tool.
- OrderPriceBanner: validates ?code= via /api/v1/discount and shows
was/now + expiry; Wizard forwards the code to order creation.
- Verified: code gen, expiry math, scope enforcement, discount API
(40% off $79 = $47.40), site+api builds clean.
- Wizard Finish button: for visitors with no token/order (e.g. arriving via a
campaign ?dot= link), create the compliance order from collected intake data
and redirect to Stripe Checkout, instead of silently doing nothing.
- StateTrucking: Operating States no longer required; single-state/intrastate
carriers can finish (relabeled 'Other Operating States (if any)').
- build_trucking_campaigns: per-state programs (weight_tax/emissions) now derive
the CTA landing page from the deficiency flag's state suffix (e.g.
state_weight_tax_OR -> OR), not the carrier base state, so a GA-based carrier
flagged for OR weight-mile tax links to the OR page (not a mismatched one).
send_test() replaced company/dot/state placeholders but not
{{ .Subscriber.Attribs.lp_link }}, so the CTA button (Check My Emissions
Status, Register My Tax Account, etc.) rendered as a bare '?dot=...' that
linked to nowhere in every owner test/approval email. Real subscribers were
unaffected (their lp_link attrib is populated). Now the test mirrors the real
audience link via build_lp_link(campaign_type, state).
Lets us fire small, controlled batches (e.g. MCS-150 only, 100/tz, sent today)
while the new sending IPs warm up, instead of the full multi-segment schedule.
Yahoo operates a large family of consumer domains (AOL, AT&T, Verizon,
Frontier, sbcglobal, bellsouth, etc.) that aggressively defer cold senders
with 421 'unexpected volume / user complaints', which poisons our self-hosted
sending IP for every other provider. Previously we only excluded
aol.com/yahoo.com/ymail.com/rocketmail.com.
Centralize the authoritative block list in scripts/_email_exclusions.py and
import it from both audience builders so they stay in sync.
Add 6 flag-based campaign segments to build_trucking_campaigns.py keyed off
fmcsa_carriers.deficiency_flags: for_hire_boc3, irp_ifta, intrastate_authority,
state_weight_tax (per-state LP), state_emissions (CA->ca-mcp-carb), hazmat.
Each injects an order-LP link into subscriber attribs (lp_link) and only
schedules when its CAMPAIGN_*_ID source template env is set (nightly run never
breaks on unconfigured templates). Adds --list-segments audience report and a
synthetic-data segment test (fixed a real psycopg2 % escaping bug in LIKE).
- send_test no longer overwrites {{ UnsubscribeURL }} with a dead static URL;
Listmonk renders it into a working per-subscriber unsubscribe link.
- dot-compliance hero grid: 4 columns (minmax 150px, max-width 920px) instead
of 3 to reduce vertical space.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- import_subscribers: was POSTing wrong bulk shape AND fallback used 'list_ids'
(ignored by Listmonk) instead of 'lists' -> subscribers never attached to the
list -> real sends would go to an empty list. Now single-adds with 'lists',
handles already-exists, returns a count, logs if 0 added.
- send_test: passed base['lists'] (objects) instead of IDs -> test send rejected.
Now extracts list IDs.
- create_and_schedule_campaign: add schedule= flag (preview makes drafts).
- --preview: 1 sample carrier/campaign, only owner email, drafts not scheduled,
test sends immediately, never marks real carriers sent.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Each of the 8 daily campaigns gets a test send immediately after creation
using the first row's real carrier data as the sample. TEST_EMAIL env var
overrides the default (carrierone@gmx.com).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- build_trucking_campaigns.py: nightly script that creates 8 Listmonk campaigns
per day (4 TZ x 2 types: MCS-150 overdue 2k/TZ, inactive USDOT 1k/TZ)
at 4AM ET / 5AM ET (CT) / 6AM ET (MT) / 7AM ET (PT). Deduplicates via
listmonk_sent_at column.
- migration 083: add listmonk_sent_at + listmonk_campaign_type to fmcsa_carriers
- email_verifier.py: bump max_workers from 5 to 20 for 4x faster throughput
- cron: daily pw-trucking-campaigns at 08:00 UTC (3 AM EST)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
