Replaces the panic-era burner-domain verification plan with an in-house
automatic catch-all rollout in the trucking/IFTA/UCR builders. Root-cause
classification of the 75k pre-DKIM-fix bounces showed ~55% were reputation/
auth (now fixed by DKIM signing) and only ~29% genuinely-dead mailboxes;
catch-all domains accept at RCPT time so they do not user-unknown bounce at
send, making a controlled in-house bleed safer than warming a separate burner.
catch_all_enabled() adds catch-all results only when warmup_day >=
CAMPAIGN_CATCH_ALL_MIN_DAY (21) AND the recent 2-day live bounce rate is below
CAMPAIGN_CATCH_ALL_MAX_BOUNCE_PCT (8%) on a >=300-sent sample; auto-reverts to
the clean smtp_valid/send_confirmed pool on the next run if bounces spike.
Short window so a past disaster cannot block the rollout forever and a fresh
spike trips fast. CAMPAIGN_INCLUDE_CATCH_ALL=1/0 still hard-overrides.
USABLE_FILTER (static) -> usable_filter() (per-run, memoized, one DB probe).
IFTA/UCR SELECT_SQL -> _select_sql() so tc.usable_filter() resolves at call
time, not import. 13 logic unit tests pass; live dry-run decision = OFF
(day 15 < 21 and recent 2d bounce 42% from the aging-out Jun-16 disaster).
Reframe away from 'escape the FCC' optics that would draw enforcement attention:
- Header/flagbar: 'Move your VoIP home to Canada' / 'US obligations ride on your
upstream' (was 'no FCC reporting, no USAC, no S/S to run')
- Recast claims to 'CRTC regulatory home, not FCC' and scope the no-USF/no-499/
no-RMD claims to the Canadian-jurisdiction traffic (accurate for US-number
traffic, which rides on the compliant US upstream)
- STIR/SHAKEN bullet now explicitly pro-compliance: 'we don't help anyone dodge
call-authentication; upstream partners are fully S/S compliant'
- Drop 'outside the FCC's reach'
- Add honest caveat: Canada is not for short-duration/dialer traffic; Canadian
carriers are more stringent on ACD/ASR than anywhere; this is for real
conversational voice (UCaaS/PBX/business/residential/live-agent)
Pivot from the hedge/second-entity framing to the consolidation pitch: one CRTC
carrier as the home base, nexus in Canada, customers onboarded from anywhere.
Lead value props with the three concrete reseller realities:
- No FCC reporting (no 499-A/Q, no RMD recert)
- No USAC/USF on your revenue (contribution sits upstream)
- No STIR/SHAKEN to set up or run (reseller can't get a US token; upstream signs)
Add: No FCC Section 214 / no ongoing 214 burden -- CRTC BITS is a cheap,
low-burden notification by comparison. Header/subject reworked; keeps the honest
US-termination + upstream-signing explanation.
Address the two most common objections truthfully (researched against CRTC,
FCC 2025 Third-Party Authentication Order, and STIR/SHAKEN cross-border docs):
- US-based long-distance termination operators routinely accept traffic from
Canadian carriers (cross-border voice is a standard interconnect).
- STIR/SHAKEN: a Canadian reseller cannot get a US SPC token (US-carrier-only),
so US-bound calls are signed by the upstream US-number provider that assigns
the DIDs -- exactly how most small US carriers already rely on upstream
signing. Canadian-origin traffic falls under the lighter CRTC regime, handled
by the upstream Canadian carrier. Does NOT claim S/S disappears -- it moves to
the upstream, off the carrier's day-to-day operation.
Address the obvious 'but I need US numbers' objection: several Canadian
wholesale carriers (Fibernetics, Iristel, VoIP.ms, Telnyx, Bandwidth, Twilio,
Frontier) provision US DIDs to CRTC-registered carriers, so they can keep
serving US customers from the Canadian entity. Adds a Canada-advantage bullet
and updates the guide block to call out both US + Canadian DIDs.
The FCC's 2025 Robocall Mitigation Order (47 CFR 64.1200(n)(4), FCC 25-6)
requires collecting + authenticating a government-issued photo ID for every
new customer before turning up voice service. Add it to the US-carrier burden
list and the matching 'does not apply in Canada' advantage.
- campaign_helpers.py: extract the branded Listmonk HTML helpers (hdr/flagbar/
stats/cta/footer/P/UL/etc.) + create_campaign() from create_campaigns.py into
a side-effect-free shared module; create_campaign() now takes an altbody so
every campaign ships a plaintext alternative (deliverability).
- create_crtc_usf_campaign.py: build the one-off CRTC email hooked on the Q3
2026 USF factor (38.8%, +1.8pts, eff Jul 1), with a $200-off CANADA200 banner
(expires Fri 23:59 ET, CTA links carry ?code= for auto-apply), the full US
carrier burden vs Canada advantage, BC/ON incorporation, and a hosted
carrier-guide PDF download. Creates a DRAFT only; sending stays manual.
- checkout.ts: generalize ensureCompliancePortalUser -> ensurePortalUser and
call it in the CRTC post-payment path so PayPal/crypto/webhook-confirmed CRTC
orders always get an ERPNext Customer + Website User (the single source of
truth for portal login/password), matching the compliance fix from the
PayPal incident. Also flip portal_user_created for canada_crtc/formation.
- canada-crtc.ts: enforce discount active+start/expiry windows, global usage
limit and applies_to scope server-side at checkout (was active-only), so a
promo like CANADA200 actually stops working after its expiry.
- scripts/generate_canada_carrier_guide_pdf.py: render the public Canadian
wholesale carrier/vendor guide PDF (reuses the canonical VENDORS list) to
site/public/guides/canada-carrier-guide.pdf for the CRTC campaign lead magnet.
The daily 40%-off coupon was being merged into every trucking/UCR/IFTA/OTC
send, but those discount sends were not actually being delivered (the
DKIM-broken window). Now that deliverability is fixed, re-test whether
normal-price offers convert before giving margin away.
New CAMPAIGN_ENABLE_COUPON env flag (default OFF) gates daily-coupon
minting in build_trucking_campaigns + the UCR/IFTA/OTC builders (which
import it as tc.COUPON_ENABLED). With it off, no code is minted and an
empty coupon_code is merged -> the campaign templates' existing
{{ if .Subscriber.Attribs.coupon_code }} guard falls through to the
normal-price {{ else }} branch and landing-page links carry no ?code=.
No template or DB changes; fully reversible (set CAMPAIGN_ENABLE_COUPON=1).
Verified: COUPON_ENABLED defaults False, coupon_attribs(None) -> empty,
lp_link drops ?code= when no coupon, all 4 builders compile.
SMTP2GO is no longer used: Listmonk relays through the local Postfix MTA
(172.18.0.1:25 from the Docker network), which DKIM-signs and delivers
direct-to-recipient-MX; transactional mail goes through Carbonio. Verified
zero smtp2go in any live container env + postfix has no external relayhost.
Removed the stale references so a rebuild/new dev can't re-introduce it:
- api/src/config.ts: SMTP_HOST default mail.smtp2go.com -> co.carrierone.com
- scripts/workers/crypto_payment_worker.py: same default fix
- infra/ansible all.yml: listmonk_smtp_* now 172.18.0.1:25, no auth (+comment)
- app.env.j2 / email.ts / crm.md / go-live-todo.md / architecture.svg: docs
All transactional/worker senders built multipart/alternative (or mixed)
messages with ONLY an HTML part. A single-part multipart/alternative is
malformed and HTML-only mail is a spam-score signal -- the same class of
deliverability bug that hurt the campaign pipeline, but on the telecom /
filing / customer-transactional path (499-Q reminders, RMD/FCC filing
review links, intake/completion/delivery emails, commissions, etc).
- worker_email.send_worker_email: auto-derive plaintext from HTML when
caller omits text= (fixes the shared helper for all current+future use)
- 16 rolled-their-own senders in scripts/workers/** + scripts/formation/
document_delivery.py: attach html_to_text(...) plaintext sibling before
the HTML part (job_server + document_delivery wrap text+html in an
alternative sub-part so PDFs still attach to the mixed root)
- api/src/email.ts: add dependency-free htmlToText() and default
sendEmail text to it (fixes checkout/webhook HTML-only sends)
Verified: all py files compile + import at runtime, api tsc passes,
htmlToText handles hrefs/lists/entities, 11 plaintext unit tests pass.
Telecom campaign 407 (Jun 8) was HTML-only + sent in the DKIM-broken
window -> 384 sent / 0 clicks (same junked-mail signature).
The FMCSA census was a one-time snapshot (last loaded ~May 30) with NO refresh
timer -- carriers newly falling out of MCS-150/UCR compliance were never picked
up. New scripts/workers/fmcsa_source_refresh.py orchestrates the full pipeline
(census download -> enrichment -> deficiency flag -> verify new emails ->
MX-tag new) and runs weekly via cron pw-fmcsa-refresh (Sun 09:00 UTC), codified
in the mail-pipeline Ansible role.
Idempotent + incremental: the census upsert preserves email_verified /
listmonk_sent_at / deficiency_flags, so existing carriers keep their send state
and only census fields refresh; new DOTs flow into verification then campaigns.
A carrier who refiled gets a fresh mcs150_parsed, so the builder's overdue
WHERE clause stops targeting them automatically. Verify is capped per run
(20k) so it never stalls on millions of rows.
(Healthcare already auto-catches newly-revalidation-overdue providers within
its 63k institutional pool via pw-hc-refresh Mon/Wed/Fri.)
The anchor regex only matched quoted hrefs; unquoted (href=URL) dropped the
URL from the plaintext part. Now handles double/single/unquoted. Added
scripts/test_email_plaintext.py (11 cases: link forms, mailto, template-tag
preservation, tag stripping, entity unescape, blank-line collapse).
Added DEAD_ISP_DOMAINS (52 domains) to BLOCKED_EMAIL_DOMAINS, so every
campaign builder that imports the shared exclusions (trucking, UCR, IFTA via
create_and_schedule_campaign, and the healthcare importer) stops cold-mailing
them. Domains were identified from our own Listmonk bounce table (top bounced
recipient domains) cross-checked against ISP status: defunct dial-up brands
(earthlink, netzero, juno, mindspring...), Qwest/Embarq legacy, satellite
(hughes, wildblue, dishmail), Altice/Suddenlink rural, WOW!/Knology, small
rural ISPs (windstream, tds, iowatelecom...) and Alaska regional.
Deliberately keeps still-active large consumer ISPs (comcast/charter/cox/
centurylink) -- their bounces were the cold-IP/no-DKIM reputation problem
(now fixed), not dead mailboxes, and they carry real prospects.
Part of the email-deliverability incident hardening.
Two deliverability hardening fixes from the email audit:
1. Plaintext (altbody): all campaigns were HTML-only. Listmonk only emits
multipart/alternative when altbody is set, and HTML-only bulk mail is a
spam-score signal. New scripts/_email_plaintext.py renders a readable
text/plain part from the HTML body (dependency-free; preserves Listmonk
{{ .Subscriber }}/{{ UnsubscribeURL }} template tags, turns links into
'text (url)'). Wired into the trucking builder (and thus UCR + IFTA, which
reuse create_and_schedule_campaign) and the healthcare builder.
2. Stable container hostname: Listmonk derived its Message-ID from the random
docker container id -> @localhost.localdomain (spam-score signal). Pin both
listmonk + listmonk-hc hostname to perfwest.performancewest.net, matching
Listmonk's SMTP hello_hostname.
Part of the email-deliverability incident hardening.
The verifier returned (True, 'mx_unreachable') when it couldn't complete a port-25
probe to ANY MX — marking 438,163 addresses email_verified=TRUE. But these are NOT
dead: they're dominated by Comcast (13.7k), AT&T/SBCGlobal (13.5k), Verizon, Cox,
Charter, Frontier, etc. — major ISPs that deliberately tarpit/refuse probes from
unknown IPs. Confirmed from prod: comcast MX connects + returns 220. The probe
failure ≠ undeliverable.
Fix: return (False, 'mx_probe_blocked') — MX exists, deliverability UNKNOWN, must
be confirmed by a real send. Excluded from PW campaigns; prime burner-verification
target (burner_list_verify upgrades it to send_confirmed on delivery). Existing
438,163 mx_unreachable rows reclassified in prod to mx_probe_blocked / verified=FALSE.
The smtp_valid pool is only ~3k unsent — too small to sustain campaigns. SMTP
probing can't confirm catch-all/mx_unreachable deliverability; only a REAL send
can. burner_list_verify.py reconciles a verification send from a DISPOSABLE burner
domain (isolated from PW/carrierone reputation):
- hard bounce -> fmcsa_carriers.email_verify_result='hard_bounced' (excluded)
- delivered -> 'send_confirmed' (proven deliverable; PW campaigns send to it)
It tails the burner MTA mail.log (reuses bounce-watcher's status= pattern) and
writes back idempotently. The PW trucking filter now treats smtp_valid +
send_confirmed as sendable. docs/campaign-deliverability-plan.md captures the full
diagnosis, the burner design, and CAN-SPAM guardrails.
Remaining (needs a domain + isolated MTA identity — operator/infra decision):
stand up the burner domain, the verification-send worker, and a writeback cron.
Root cause of zero conversions since Jun 9 + the Gmail/Outlook block storm:
the send filter was '(email_verified IS TRUE OR result IN ...)'. The verifier
sets email_verified=TRUE optimistically for mx_unreachable (domain exists but
its mail server never answered the RCPT probe) — 438,163 such rows. Those HARD
BOUNCE on send, producing ~1,100 bounces/day (~47% rate) and blocklisting half
the 120k subscriber base, so real prospects never saw the offer.
Fix: key the send filter ONLY off email_verify_result, never the broken boolean.
Recovery mode (default): send only 'smtp_valid' to drive bounce rate to ~0 and
rebuild reputation; set CAMPAIGN_INCLUDE_CATCH_ALL=1 to re-add catch-all domains
once recovered. Mirrors the healthcare list-cleaning approach (HC bounces ~2-3%,
which proves the fix). Note: only ~3k smtp_valid unsent remain — list growth via
real-send bounce verification (separate burner domain) is the follow-up.
The MCS-150 intake-completion email linked customers to /order/dot-compliance,
which is the sales/checkout page -- it ignores ?order= and asks the customer to
re-pick services and pay again, so they 'cannot enter any data' (Paul Wilson's
report). Link to the per-service intake wizard /order/<slug>?order=... instead,
which loads the paid order, pre-fills from the FMCSA census, and drops payment.
Also add a Trailers field to the DOT intake fleet section and wire it through to
the MCS-150 PDF Q26 trailer row, so carriers can update trucks AND trailers.
- Added a line asking them to call their insurance agent to confirm Form E
ability before clicking yes/no, so we pick the right path first time.
- Reply-To now routes to info@performancewest.net (monitored), overridable via
SC_COC_REPLY_TO env.
The Dockerfile copies form PDFs explicitly by name; the SC COC template was
missing, so fill_sc_coc() would FileNotFoundError in the container. Added it.
Routes SC intrastate-authority orders to the real SCDMV COC product instead of a
PSC certificate (which doesn't apply to property carriers):
- sc_coc_filing.py: emails the carrier a one-click yes/no — does your insurer
have / can they file a Form E (SC intrastate liability, $750k or $300k by
GVWR) with SCDMV? Records the answer; builds the filled COC package.
- state_trucking._handle_sc_coc_gate: SC intrastate gate —
no answer -> email the question once, HOLD
answered no -> broker referral opened, HOLD (ops todo)
answered yes-> proceed to bill the exact $25 SCDMV COC fee (at cost) + file
- API POST /compliance-orders/:id/sc-insurance: records yes/no in intake_data
(no schema change); NO opens an insurance_lead broker-referral ticket +
Telegram; YES re-dispatches the worker to bill the $25 + file.
- site/order/sc-insurance: customer one-click yes/no page (auto-submits when
the email links straight to ?have=yes|no).
Non-SC intrastate still uses the PSC/PUC email path or a manual todo.
SC for-hire PROPERTY carriers (not passenger/HHG/hazwaste) register intrastate
via the SCDMV Certificate of Compliance (COC), not a PSC certificate. This adds:
- sc_coc_pdf_filler.fill_sc_coc(): fills the official SCDMV Form COC from
intake (business name, officers, physical/mailing address, phone), picks
New vs Renewal, and stamps the coverage class (E-L low-value / E-LC).
Field names in the source PDF are auto-generated + offset from their labels;
mapped here by verified on-page geometry. Verified by render.
- suggest_coverage_class(): E-L for low-value cargo (scrap/dump/aggregate),
else E-LC (safer default).
- gov_fee: SC intrastate fee corrected from $0 placeholder to the real $25
COC new-application fee (renewals $0), billed at cost.
The carrier's INSURER files the Form E (liability) + Form H (cargo, E-LC only)
directly with SCDMV; we collect the COC app + $25 and submit it.
Intrastate operating authority is state-specific + application-based like IRP, so
it reuses the same email/POA + invoice-reconciliation flow:
- intrastate_filing.send_intrastate_submission: emails the state PSC/PUC the
authority application with the signed POA attached (subject tag [PW-ISA CO-..]),
reusing irp_filing's MinIO download + census enrich helpers.
- The shared poller (irp_invoice_poller) now matches BOTH [PW-IRP] and [PW-ISA]
tags, parses the fee, Telegram-alerts, and bills the customer the exact amount
with the correct service slug.
- state_trucking gov-fee gate routes intrastate-authority to the PSC/PUC email
path; if no submission email is configured for the base state it falls back
to a manual todo (safe default — no emailing guessed agency addresses).
Per-state ISA_<ST>_EMAIL env (blank until the exact agency address is verified).
SC/GA/TX scaffolded. Customer still only sees an exact-fee payment link; you only
approve the final filing.
- send_irp_submission now REQUIRES and ATTACHES the signed Power of Attorney PDF
(downloaded from MinIO) — the state won't act on a third-party filing without
it, and 'on file, available on request' stalls the request. If the POA isn't
available we don't email and fall back to a manual todo.
- Backfill missing legal_name + registered address from the FMCSA census so the
submission isn't sent with a blank address (root cause of the empty
'Legal/registered address: , ,' line). Customer-supplied values win.
- state_trucking passes signed_auth_key through to the IRP submitter.
- Fix 'Object of type date is not JSON serializable' when creating the admin
todo (json.dumps(..., default=str)) — broke the intrastate (bash-fee) path.
The gov-fee email now lists exactly what the amount covers (full breakdown) so
the customer can check it for accuracy, with two clear actions: a ✅ pay link and
a ❓ 'something looks wrong' link to /order/dispute.
New /order/dispute page shows the fee breakdown and lets the customer describe
what's wrong; it opens an 'issue' support ticket pre-tagged with the order
(amount + label + their note) via /api/v1/tickets, so ops corrects the fee
before any payment is taken. The /order/pay page also shows the itemized
breakdown and a dispute link.
- gov_fee: add AGENCY_PROCESSING_FEE (per-service card/convenience fee passed
through so the customer pays the true all-in cost); estimate_gov_fee now folds
it into the billed total. IFTA/intrastate/UCR fees are published/near-exact.
- IRP fees can't be looked up — only the base state computes them. New
irp_filing.py: emails the base-state IRP unit a Schedule A/B request (Reply-To
the IRP filings mailbox, [PW-IRP CO-...] subject tag), and a 15-min cron
(irp_invoice_poller) scans the mailbox for the state's invoice reply, parses
the exact apportioned fee, Telegram-alerts you, and bills the customer the
EXACT amount via a gov-fee child order + payment link. Then it proceeds to
ready_to_file for your final approval.
- state_trucking gov-fee gate now routes IRP to the email/invoice path and
IFTA/intrastate to immediate exact-fee billing.
- Mailbox is configurable (IRP_FILINGS_IMAP_* in app.env.j2); falls back to
OPS_IMAP_* filtered by the [PW-IRP] tag until a dedicated mailbox exists.
Telegram alerts fire on IRP submission sent, invoice received (billed), and
un-parseable replies (so you can read + enter the fee manually).
At-cost services (IRP/IFTA/intrastate) only collected our service fee at
checkout; the variable state fee was never billed, so orders stalled at
authorization_signed and the filing card would have had to front large IRP fees.
New end-to-end, hands-off flow (you only approve the final filing):
1. After authorization is signed, state_trucking auto-estimates the gov fee
from intake (base/op states, power units, weight) via gov_fee.estimate_gov_fee.
2. Creates a CHILD compliance order (CG-..., service_fee=0, gov_fee=estimate,
parent_order_number set, migration 099) that flows through the EXISTING
checkout/payment/webhook machinery.
3. Emails the customer a payment link to /order/pay (new self-contained page)
showing every method with correct surcharges — ACH 0% (Stripe 0.8%/ cap
absorbed, no GoCardless needed), card/PayPal 3%, Klarna 6%, crypto 0%.
4. Order holds at awaiting_government_fee_approval until paid.
5. On payment, handlePaymentComplete detects the child (parent_order_number)
and re-dispatches the PARENT with gov_fee_paid=true, which proceeds to
prepare + queue the filing and stops at ready_to_file for your approval.
IRP fees are estimates billed at cost (refund overage / rebill shortfall); IFTA
decals + most intrastate fees are near-exact. Tunable via env.
relay_integration.py line 34 called logging.getenv (no such attr), which threw
AttributeError on import -> load_card_from_erpnext() crashed for every caller
(BOC-3 and now UCR filing payment). Drop the bogus line; LOG is set correctly on
the next line. Present since the initial commit.
Adds scripts/workers/services/ucr_playwright.py — a UCR.gov National Registration
System automation that, given a USDOT + fleet size, runs the register/pay flow,
pays the federal UCR fee with the matched PW filing card (Relay/Stripe Issuing),
and captures a confirmation screenshot + number. Conventions match
boc3_playwright / fmcsa_web_submitter: dev-mode dry-run guard, undetected
(patchright) browser, CAPTCHA detection, screenshot evidence, dataclass result.
Safety: verifies the displayed fee against the federal schedule before paying and
refuses to auto-charge a surprising amount (UCR_MAX_AUTO_FEE_USD) — falls back to
manual filing instead.
Wires it into MCS150UpdateHandler: when an approved (admin_approved) order has
slug ucr-registration, _file_ucr_registration runs the automation, uploads the
confirmation screenshot to MinIO, records filing_status + confirmation, and sets
fulfillment_status=completed on success. On CAPTCHA / fee-mismatch / failure it
reverts to ready_to_file with a high-priority 'file manually' todo. This replaces
the old behavior where approving a UCR just sat at authorization_signed.
UCR (and other admin-assisted DOT services) route through MCS150UpdateHandler,
which hardcoded 'MCS-150' and self.SERVICE_SLUG in the admin todo, the Telegram
fulfillment notification, and the customer status email -- so approving Paul's
UCR produced an 'MCS-150 Review / mcs150-update / PDF: not generated' alert and
an 'MCS-150 biennial update' customer email, both wrong.
Add SERVICE_DISPLAY_NAMES + _service_label(slug); use the actual slug everywhere.
Admin-assisted services now show 'UCR Annual Registration — FILE NOW ... file
manually on the portal (no auto-generated form)' instead of MCS-150/PDF wording,
and the customer email names the right service.
Admin-assisted DOT services (UCR, BOC-3) routed to this handler were marked
ready_to_file with whatever intake existed -- e.g. a UCR with only a DOT number,
missing legal name / state / fleet-size bracket (which sets the UCR fee tier).
That made the admin 'ready to file' status dishonest and unfileable.
Now, for ADMIN_ASSISTED_REQUIRED services we first enrich intake from the FMCSA
census (legal_name, address_state, power_units) + the order email, and derive
the UCR fleet_size_bracket from power units (UCR_FLEET_BRACKETS). If every
required field is then present we persist it and mark intake validated (falls
through to the admin review gate -> ready_to_file). If anything is still
missing, we persist what we have, set fulfillment_status=awaiting_intake, and
email the customer to complete intake -- instead of falsely showing ready_to_file.
Two of our three real paid customers (Mark Adams / mark@adamslumber.com and
Paul Wilson / synthetic@pipeline.com) never completed intake. They each hit the
old hard cap of 10 daily reminders (last sent Jun 12 / Jun 13) and the worker
then went permanently silent -- the last two daily runs reminded 0 orders even
though both still owe us intake on paid work. (The third, mitchell allen /
mitchell@allenscrapmetal.com, did complete intake; his orders are dispatched.)
Replace the dead-stop cap with a two-phase cadence:
- daily for the first DAILY_PHASE (10) nudges -- the initial burst,
- then weekly (WEEKLY_INTERVAL_DAYS) up to an absolute MAX_REMINDERS (60),
so a paid order with missing intake keeps getting a gentle nudge instead of
being dropped. Tunable via INTAKE_REMINDER_DAILY_PHASE /
INTAKE_REMINDER_WEEKLY_INTERVAL_DAYS / INTAKE_REMINDER_MAX. Clearing
intake_reminder_last_at re-arms an order immediately (documented in the
module docstring).
Main pool is calendar-day 12 but reputation is wrecked (54% delivery, Gmail+
Outlook blocks) -- NOT warmed. MX tagging confirmed the cause: 702k carriers on
Google + 135k on Microsoft = the warmup was hammering exactly the two operators
blocking us. Hold Google/MS/Proofpoint/etc. OUT entirely until day 30 (configurable),
sending only to the long-tail operators (yahoo/comcast/charter/centurylink/etc.)
that don't bot-throttle, so reputation can recover; then re-introduce big
operators gradually via mx_daily_caps. 1.24M/1.49M carriers now tagged.
The real bottleneck was the write, not DNS: each per-domain UPDATE full-scanned
fmcsa_carriers (no functional index on lower(split_part(email,'@',2))). Resolve
all domains concurrently into a list, load a temp table, then ONE join-UPDATE =
single table scan. Tags ~12k domains -> hundreds of thousands of carriers fast.
The serial path (verifier's 8s+6s lifetime per domain) was far too slow for
bulk tagging -- 0 tagged in 3 min on dead domains. Self-contained fast resolver
+ ThreadPoolExecutor(40) resolves thousands of domains in minutes.
The Jun 13-14 Gmail+Outlook block storm came from the main/trucking pool having
NO per-MX throttling (only HC had it) -- it concentrated warmup volume on
Google/Microsoft-Workspace-hosted business domains. Port the HC fix:
- migration 097: fmcsa_carriers.mx_provider column.
- mx_tag_carriers.py: resolve MX once per distinct domain (reuses the verifier's
classifier+cache), tag every carrier with that domain's operator. Bounded per
run, prioritizes unsent verified carriers.
- build_trucking_campaigns: during warmup (day<=6) EXCLUDE tagged Google/MS/
Proofpoint/etc. carriers in fetch_carriers; per-MX cap in select_sendable_
carriers so known operators never dominate the quota. Untagged carriers pass
(not collapsed onto one bucket) until tagging fills in. mx_daily_caps ramps
with the main warmup day; MAIN_SKIP_BIG_MX=0 disables once warmed.
The HC warmup crons were '* * 1-5' (Mon-Fri), silently skipping weekends -- but a
proper warmup needs CONTINUOUS daily volume for 21 days (mailbox providers reward
consistency; gaps stall reputation). The Jun 14 'HC 0 sent' alert was just a
skipped Sunday, but the weekend skips also broke ramp continuity.
- pw-hc-campaign + pw-hc-nppes: '* * 1-5' -> '* * *' (daily), vendored + applied live.
- Re-aligned the warmup start stamp from calendar-day 9 to send-day 5 so the
volume ramp matches reputation actually built (it had skipped ~4 weekend days,
running the ramp ahead of real history).
- Fixed the stale 'Mon-Fri only' comment in daily_slice().
- Vendored nppes cron now carries the enriched-CSV + 4-segment config.
otc_reincorporation.html: redomesticate-to-Texas hook (Business Court + TXSE +
DE franchise-tax cost) personalized by state_inc_name/company/ticker, cross-sell
RA/foreign-qual/annual-report/franchise-tax, same-day coupon, lead-capture CTA to
/contact?service=reincorporation (high-touch corporate service, not self-serve),
careful 'not a law firm / not legal advice' disclaimers + CAN-SPAM address.
build_otc_campaign.py: emails only verified-email issuers from the harvest+scrape
+verify pipeline, --de-nv-only for the best reincorp fit, reuses trucking sender
plumbing + coupon. Per-deal value is high so capped modestly (400/run default).
scrape_otc_emails.py: fetch each issuer domain's IR/contact pages (gzip,
HTML-only, early-abort, prefer ir@/investor@/info@), extract a contact email.
Skip filing-agent domains (DFN/Donnelley/Broadridge/etc.) that leak into the
extracted domain -- those are not the issuer's site. Same filter added to the
harvester's DOMAIN_NOISE for future runs. Phone (100%) is the fallback channel
for email misses.
Pilot -> production: harvest_otc_issuers.py pulls the OTC/None universe (2,771),
keeps US-domestic (requires BOTH a US state-of-incorporation AND a US-state
business address -- disambiguates the 'DE'=Delaware-vs-Germany trap that leaked
Infineon etc.), and extracts each issuer's website DOMAIN directly from its
latest 10-K/8-K/DEF-14A filing (free, no scrape; ~58-60% find rate in testing).
Outputs cik,name,ticker,state_inc,phone,city,state,zip,domain -- ready for the
domain->email scrape + verify step. Phone is 100% (clean fallback call channel).
Reincorporation-to-TX / RA / foreign-qual / franchise-tax / annual-report fit.
The OIG-screening + NPPES-update segments were effectively limited to ~1,437
providers because the warmup 'any' selector excluded not-on-reval-list rows as a
deliverability proxy -- but that excludes almost the ENTIRE institutional list
(org NPIs aren't individual Medicare enrollees). Since we already SMTP-verified
all 63k inboxes, add an 'institutional_verified' selector that trusts our own
verification instead of reval-list presence. Result: OIG + NPPES-update now
address 62,422 (43x more), giving multiple broad offers to test engagement on.
- enrich_institutional_revalidation.py: fast local join of the institutional
list to the CMS Revalidation Due Date List bulk file (revalidation_base.csv)
by NPI -> adds reval_due_date/days_overdue/reval_status. ~1,437 are genuine
Medicare enrollees (197 overdue / 164 due-soon) -> flagship $599 reval pitch.
- npi_reactivation stays on leie_or_deactivated (only REAL deactivations -- no
false 'your NPI is deactivated' claims to active orgs).
UCR (Unified Carrier Registration) is annual: opens Oct 1, due Dec 31, mandatory
for interstate carriers (op A, same ~628k pool as IFTA) -> recurring revenue.
- build_ucr_annual_campaign.py: 3-touch business-day cadence (30/12/4 bd before
Dec 31, wider than IFTA since it's once a year), escalating tone, same-day
coupon, 'I already did it' suppression. Reuses build_trucking_campaigns +
IFTA business-day/token helpers (DRY). Per-year cycle reset.
- ucr_annual_reminder.html: deadline + fines/OOS risk + 'we figure out your fee
tier' + coupon + filed link + CAN-SPAM. Source campaign 473.
- migration 096: ucr_reminded_at / ucr_touch_no / ucr_self_filed_at.
- ifta.ts: add GET /api/v1/ucr/filed (shares the HMAC token scheme).
- checkout.ts: order-placement Telegram now shows 'Source: campaign (code X)'
when a discount code is present, so IFTA/UCR/CLIA conversions are visible.
(Confirmed order-alert Telegram already fires from handlePaymentComplete for
all compliance orders via both webhook + session paths.)
The handler is an API route; performancewest.net/api/* proxies to the Astro site
(:4322), only api.performancewest.net serves the API. Build the filed link from
PUBLIC_API_URL (default https://api.performancewest.net). Verified end-to-end:
valid token -> 200 confirmation page, invalid -> 403.
- Multi-touch reminders at 10/7/4 BUSINESS days before each deadline (weekends
skipped; biz-day math so a touch never lands purely on a weekend with no
runway). Escalating tone soft -> urgent -> last-chance, with the 'almost too
late to DIY, we can still file it' angle so it's a convenience sale, not a free
reminder service. ifta_touch_no tracks the highest touch sent so each touch
hits only carriers below that level; never repeats a touch.
- 'I already filed it' one-click link: HMAC-tokenized GET /api/v1/ifta/filed
(token matches between Python builder and api/src/routes/ifta.ts -- verified
identical output), records ifta_self_filed_at, friendly confirmation page,
stops further touches this cycle + gives DIY-vs-prospect signal. Builder
excludes self-filed carriers.
- migration 094 (ifta_touch_no) + 095 (ifta_self_filed_at); cycle reset clears
both each new quarter. Verified: biz-day touch schedule, token cross-match.
add_subscriber only attached existing subscribers to the new list without
updating attribs, so a carrier emailed in a prior campaign kept STALE attribs --
meaning the daily coupon code and IFTA merge fields (ifta_due_date, ifta_quarter,
lp_link) rendered BLANK for any repeat recipient. Now merge+PUT the fresh attribs
on the existing subscriber before attaching. Affects all trucking campaigns, not
just IFTA. Verified: IFTA preview now persists ifta_quarter/ifta_due_date/lp_link.
IFTA returns are due on fixed dates (Apr30/Jul31/Oct31/Jan31) and every
interstate carrier (op code A, ~628k sendable) files 4x/year forever -- pure
recurring revenue, no per-carrier deadline data needed.
- build_ifta_quarterly_campaign.py: self-gates to the reminder window (~21d
before each deadline), selects interstate carriers, mints the same-day coupon,
builds+schedules the campaign reusing build_trucking_campaigns plumbing (DRY:
one source of truth for sending/suppression/coupon). Per-quarter cycle reset
(ifta_reminder_cycle marker) so each quarter re-reminds the full pool; marks
ifta_reminded_at to avoid double-sends within a cycle.
- ifta_quarterly_reminder.html: deadline + penalties + 'we do the math' + coupon
+ CAN-SPAM. Listmonk source campaign id 469.
- migration 094: fmcsa_carriers.ifta_reminded_at column + partial index.
Verified: deadline/window logic correct, imports reuse tc helpers, migration
applied on prod.
