The payment_reminder worker (15min/1day/2day recovery emails) ran daily but sent
0 emails for months because:
1. ORDER_TABLES omitted compliance_orders entirely -- where ALL trucking/renewal/
telecom orders + every abandoned cart actually live. It only checked
canada_crtc/formation/bundle (near-empty). Added compliance_orders.
2. The reminder_15m/1d/2d_sent_at dedup columns didn't exist on compliance_orders
(added via migration).
3. The systemd timer ran once daily (16:00 UTC) -- can't catch the 15min or 1day
windows. Changed to every 30 min.
Verified live: a backdated pending order triggered its 1-day reminder and a real
abandoned cart (Simpson & Co) got a recovery email, both delivered.
Was deep-linking ?name={{QueryEscape company}} which (a) risked the @TrackLink
per-subscriber 404 bug and (b) QueryEscape isn't a guaranteed Listmonk template
func + company names have commas/spaces that break the query. Now a clean static
?state=CT link (pre-selects CT, @TrackLink-safe) and the body names the company
in text so the recipient knows to look it up. Funnel verified live end-to-end.
The corporation-check overdue result now links to /order/annual-report-filing
(or /order/entity-reinstatement for dissolved) with ?legal_name=&state=&entity_number=,
instead of dead-ending at /contact. The order Wizard reads those params and
seeds the entity step so the funnel opens pre-filled. Completes the tool->order->
checkout path proven e2e on dev.
The catalog already priced annual-report-filing ($149), registered-agent ($99),
entity-reinstatement ($299) but they had no intake spec, manifest entry, or
order page, so the corporation-check tool's overdue result dead-ended at a
contact form. Now a real self-serve funnel on the proven compliance-order
pipeline:
- REQUIRED_FIELDS intake specs (legal_name/address_state/email via the entity step)
- INTAKE_MANIFEST + CORPORATE_SLUGS entries (entity->review->payment)
- /order/annual-report-filing, /order/registered-agent, /order/entity-reinstatement
API tsc clean; site builds all 3 pages. E2E on dev next.
The tool treated any ACTIVE entity as 'good standing, no action' - so an
overdue-but-active CT business (the CT campaign's target) saw a false all-clear.
Now: overdue-but-active -> 'REPORT OVERDUE' badge + past-due message with the
due date + days overdue + cost-to-fix, matching the corp-status API signal.
entity_cache had CT status but no annual-report clock, so the corporation-check
tool told overdue-but-active CT businesses they were fine. Now:
- backfill_ct_annual_report_due.py: pulls annual_report_due_date from the live
CT Socrata registry into entity_cache.annual_report_due_date (keyed by
accountnumber=entity_number). Ran: 577,562 CT rows populated, 235,875 overdue.
- corp-status API: reads annual_report_due_date, returns annual_report_overdue +
days_overdue, and bumps years_behind to >=1 for an overdue-but-active entity so
the remediation cost isn't $0. This makes the tool match the CT campaign hook.
- create_ct_annual_report_campaign.py: personalized CT annual-report overdue
email (company/due_date/days_overdue attribs), from filings.* subdomain, soft
free-compliance-check CTA. HTML+plaintext, List-Unsubscribe, @TrackLink.
- pw-mta-warmup.sh: when the rotation pool is a single IP, write plain
transport_maps (no catch-all randmap) + default_transport, so the filings
sender_dependent map (-> outfilings/.90) survives the daily warmup run.
Previously randmap:{out05:} clobbered filings routing every day at 07:17 UTC.
Verified end-to-end: Listmonk CT test egresses .90 (outfilings), DKIM
d=filings.performancewest.net s=filings, multipart HTML+plaintext, attribs
render (Riverside Landscaping LLC / due 2026-06-15 / 16 days ago), 0 stray
tokens, real per-subscriber unsubscribe link.
Added a fresh sending IP (.90/mta01) bound to the filings cold-outreach stream,
isolated from the reputation-damaged .94 trucking IP:
- .90 bound on ens18, persisted 3 ways (interfaces up/down hooks + pw-mail-ips
service + watchdog cron, all updated for the Jun-24-outage safety pattern)
- postfix 'outfilings' transport binds .90 / HELO mta01
- sender_dependent_default_transport_maps routes @filings.* -> outfilings; moved
the randmap:{out05:} catch-all to default_transport=out05 so the sender map
can win (recipient transport_maps outranks it otherwise)
- filings SPF updated to include .90
TEST RESULT: fresh IP fixed the reputation signal (Gmail error changed from
'low domain reputation' on .94 to 'likely unsolicited' cold-start on .90).
Non-Google (GMX/Microsoft) deliver fine; Gmail cold-start remains a slow wall.
Plan: CT campaign to non-Google first, warm .90, trickle Gmail over weeks.
Gmail/Microsoft rejected send.performancewest.net mail for DOMAIN reputation
(not IP), so a fresh subdomain resets that to neutral. Fully separate from
send.* to avoid the trucking cold-send history. First use: the CT SOS
annual-report leads + a Gmail warm-up test.
- DNS (A/SPF/DKIM/DMARC/MX) published in HestiaCP -> HE slaves, all resolve.
- 2048-bit DKIM key (selector 'filings'); opendkim-testkey -> key OK.
- OpenDKIM key.table/signing.table/trusted.hosts updated + reloaded; live send
to GMX signed + accepted.
- Codified in the mail role's opendkim_signing_domains; setup doc in infra/mail.
Section 10 of new-sector-compliance-targets.md rewritten with live-pulled,
verified results for the 'which verticals have 500k+ email leads' question:
- USPTO trademarks (#1) DISPROVEN: owner email is not in any free bulk file
(Daily XML DTD + Case Files research dataset have no email column) and the
TSDR/ODP API is now ID.me-auth-walled (2026-06-18). Demoted to needs-append.
- FMCSA Company Census is the real 500k+ winner: 2.93M rows w/ email_address,
1.66M active+email (1.59M distinct), 636,278 overdue MCS-150. 92% verify.
- Connecticut SOS (#3) is the email-native state flagship: 787k w/ business
email, 439,763 active+email, 97,466 overdue annual reports. 96% verify.
(No other state business registry on Socrata exposes a business email.)
New scripts/build_registry_outreach_lists.py: one extensible engine with
due_date and staleness clock types; CT + FMCSA adapters; emits overdue,
due_soon, active_emailable CSVs compatible with verify_csv_emails.py.
Adds --telegram mode to coupon_ab_scoreboard.py + a daily cron (18:00 UTC).
Pushes the per-arm clicks+conversions table to Telegram ONLY when there was at
least one PAID sale in the window; stays silent otherwise (per request, no
all-zero spam). Creds read from /opt/performancewest/.env like the other
monitors. Verified live: silent at 0 sales (14d), sends at 5 sales (30d).
So we can see whether a discount helps the CTA (clicks) AND the sale (paid
orders), per arm:
1. pw-analytics.js: include the ?code= (the price-test arm) on the campaign-click
Umami event, and fire it when EITHER a utm OR a code is present (coupon links
often carry code with no utm, so those human clicks were being dropped). Also
strip a stray @TrackLink suffix Listmonk glues onto the last query value.
2. coupon_ab_scoreboard.py: one report combining HUMAN clicks (Umami, already
bot-filtered) and PAID conversions (compliance_orders) per arm. Discount arms
map by code->pct (campaign-daily:<date>:<pct> marker); the no-code control arm
is recovered by re-hashing customer_email the same way the builder bucketed it.
Prints clicks, paid orders, revenue, and click->order per arm.
Two bugs that invalidated the price A/B/C test (20/30/0):
1. OrderPriceBanner.astro shipped with data-api="" (PUBLIC_API_URL was never
set at build), so the ?code= discount lookup hit a relative /api/... path on
the static host -> 502. The struck-through discounted price NEVER rendered
for anyone who clicked a coupon link; every visitor saw full price. Fixed to
fall back to window.__PW_API (set in Base.astro) then the prod API host, the
same pattern the rest of the site uses.
2. Daily coupons expired 23:59 ET of the SEND day. The builder runs in the
morning ET and carriers read mail across ~36h, so most discount arms saw an
already-expired code by open time. Extended to 23:59 ET of the day AFTER
send (~1.5 days), and updated the email + banner wording to match.
Net: the A/B/C arms can now actually be redeemed, so the test can produce a
real signal. Measured by paid Stripe orders keyed to the arm (re-hash of the
converter email), not bot-polluted opens/clicks.
First rollout (2026-06-28) targeted ALL enabled trucking subs and hit a 34%
bounce rate: ~80% of 'enabled' are catch_all_domain (accept MX probe, bounce at
the real mailbox). Now joins pw_smtp_valid_stage (verified smtp_valid pool, same
as the drip) and excludes lists 563/564 (already-mailed by the aborted run).
Default batches 14->5 for the smaller ~7.9k clean pool. Engagement on the clean
deliveries was strong (30% open, 28% click-of-openers), so content stays.
Splits the ~46.6k warm trucking list (enabled, dot_number) into 14 daily
batches via subscribers.id % 14, each a private list filled by Listmonk's
query-based bulk add and its own campaign cloned from draft 750, scheduled on
successive days at 14:00 UTC. Dry-run by default; --commit to create+schedule.
Spreads volume so the invaluement /24 delist propagates before bulk send, and
the engagement-positive content warms domains + dilutes the sales-pitch ratio.
One-off industry-news newsletter: FreightWaves story on Prime suing the IRS
over reefer-diesel excise tax, broadened to cover PTO/off-highway fuel use
(dump hydraulics, mixers, boom/bucket trucks, etc.) since the Form 4136
off-highway credit isn't reefer-specific. CTA: keep fuel records + free DOT
check. Includes not-a-law-firm/not-a-tax-advisor fine print. @TrackLink on
both static links (safe: not per-subscriber). Creates as DRAFT.
80% was too chatty for a box that idles ~72%. 90% warn / 94% reclaim still
leaves ~14GB headroom before the danger zone (Postgres needs free space to
checkpoint).
On 2026-06-27 / filled to 100% and crash-looped Postgres ('No space left on
device'), taking down Listmonk mid-send. Cause: an orphaned 15GB
/tmp/forgejo-dump.zip (interrupted backup) + uncapped Docker json-file logs
(forgejo container log alone was ~1GB), with NO disk monitoring to warn first.
- pw-disk-space-alert.sh + cron (every 15m): Telegram warn at 80%, auto-reclaim
build cache + orphaned forgejo dump at 88%. Silent when healthy.
- ansible docker role: write /etc/docker/daemon.json with 50m x 3 log cap
(150MB/container max) + non-disruptive Reload docker handler.
This script ran every 5 min and blocklisted on the FIRST hard bounce of ANY
5xx DSN via direct SQL, bypassing Listmonk's count-based bounce.actions rule.
That is the actual mechanism that wrongly killed ~17,000 good carriers during
the broken-DKIM window (their mail got 5.7.1 DMARC-reject, not bad-mailbox).
Fix: only genuine bad-mailbox DSNs (5.1.1/5.1.0/5.0.0/5.4.1/5.5.0) count toward
a blocklist, and a subscriber must accumulate >=3 such hard bounces (matching
Listmonk's threshold) before being blocklisted. Reputation/policy 5.7.x and
quota/greylist 5.2.x never trigger a blocklist.
A 2-day guardrail window let one bad daily batch (Jun-24: 465 sent / 10.75%)
flip catch-all OFF, starving daily volume to ~200/day -- which then could not
reach the 300-sent min sample to ever re-enable (self-reinforcing trap). The
same period over 5 days is a true ~3.8% (7,995 sent / 303 bounced), well under
the 8% ceiling, so catch-all stays ON and steady ~3k/day flows. Still trips
fast on a sustained real spike.
The institutional NPPES pool is ~98% not_on_list org NPIs, which land on a
soft/green result in the public NPI tool ('you're basically fine') -- so they
click and leave with no buy trigger. The providers who get a RED 'revalidation
PAST DUE, CMS may deactivate' result (a real $599 reason to act) are the CMS
revalidation-overdue NPIs.
This joins the CMS Revalidation Due Date List to NPPES endpoint emails and
applies the warmup deliverability gates in the correct order, critically
re-filtering Google by REAL MX *after* SMTP verification populates mx_provider
(a one-off manual run on 2026-06-26 filtered on the pre-verify empty mx_provider
column and leaked 214 sends to Gmail, earning a low-reputation bounce).
The HC warmup imported ~1000 fresh providers/day into a persistent segment list
(list 10), but each day's campaign targeted that WHOLE cumulative list -- so the
cohort imported on day 1 received the identical 'Free NPI Check' email every
subsequent day (verified: subscriber 3410 got campaigns 38-42, 5x). Program-wide
that was 23,843 sends to 6,587 people (3.6x avg, max 30x) and blocklisted ~12%
of the list -- a Yahoo 'user complaints' deferral now confirms the burn.
Fix: import each day's slice into a dedicated dated SEND list and bind that day's
campaign to ONLY that list, so every provider gets exactly one send. The
persistent segment list is still used for dedup/records. ensure_campaign now
matches the exact dated name (never reuses a prior day's campaign/slice).
Unattended kernel-upgrade reboot (Jun 24 04:04) left only .71 bound because
classic ifupdown applies just the first 'address' line. Postfix then failed to
bind .94/.107 ('Cannot assign requested address') and silently egressed from
.71 -- which is NOT in SPF (every fallback msg failed SPF) and is on RLR621 +
Trend ERS-QIL. ~37h of bypassed IP-warming + a near-zero sales day.
Fixes:
- /etc/network/interfaces: explicit up/down ip-addr hooks for .72/.94/.107
- pw-mail-ips.service: systemd oneshot re-binds IPs + flushes queue on boot
- pw-mail-ip-watchdog: */5 cron re-binds missing IPs + flushes, also catches
'Cannot assign' bind failures
- runbook: full incident writeup + reboot-test lesson
Host already remediated live; this commits the host artifacts + docs.
The dev deploy.sh is an identical copy of prod's, so its hardcoded
`cd /opt/performancewest` meant running /opt/performancewest-dev/deploy.sh
silently rebuilt and recreated PROD's containers instead of dev's. Resolve the
script's real directory (readlink -f handles symlinks/relative paths) so the
same tracked script is correct in every clone; docker compose then derives the
project name + auto-loads docker-compose.override.yml from that directory,
keeping dev (project performancewest-dev) and prod isolated.
Conversion fix for the checkout drop-off (54 sessions reached an /order/ page
over 3 days, 0 advanced to payment). Root cause was friction, not a bug: every
order page dropped a cold email-click straight into a 28-field intake Wizard
before showing any payment option.
- New ExpressCheckout.astro: payment-first entry. Shows price + the minimal
fields the API needs (prefilled from public records: ?dot= FMCSA census for
trucking, ?npi= NPPES lookup for healthcare) + Continue to payment. Creates a
single-service batch-of-one (POST /compliance-orders/batch, which does NOT
gate Stripe on intake_data_validated) then create-session -> Stripe. Full
intake is collected AFTER payment via the per-service 'Complete Your Intake
Form' email the webhook already sends (links to /order/<slug>?order=CO-xxx,
which re-enters the Wizard in paid-intake mode).
- New OrderFlow.astro: single source of truth replacing ~50 near-identical thin
Wizard wrappers. Trucking + healthcare default to payment-first (express on
top, marketing hero moved BELOW the CTA). Telecom + corporate keep Wizard-first
(rich pre-payment FCC/499 intake, no public-records prefill). Paid-intake
re-entry (?order=/?token=) always renders the full Wizard.
- Rewrote all 50 /order/*.astro pages to use OrderFlow (foreign-qualification
keeps its multi-state toggle via slotted content).
- Fixed the dead Tawk.to live-chat widget site-wide: the snippet set an invalid
crossorigin='*' attribute, forcing the browser into anonymous CORS mode and
blocking the script (0 chat requests fired anywhere). Removed it to match
Tawk's official snippet (footer partial + 73 static public/*.html files).
Verified: build clean; express on top with hero below; ?dot=/?npi= prefill;
paid-intake re-entry swaps to Wizard; telecom stays wizard-first; batch-of-one
-> live Stripe URL; both POST endpoints allow the prod origin via CORS.
Three bugs the owner hit:
1. Per-operator reputation alert (06:10 cron, mail_reputation_monitor --alert)
silently never ran: it redirected to /var/log/pw-mail-reputation.log but
/var/log is root-only and that file was never pre-created, so the deploy
user's >> redirect failed and cron aborted before the command. Repointed
both mail-alert crons to deploy-writable /opt/performancewest/logs/.
2. IP reputation alert (20:00 cron) still referenced the removed rehab pool
(.91-.93) and used 8.8.8.8 for Spamhaus (which returns the open-resolver
error, not a real answer). Dropped the rehab section, relabeled to the two
live IPs (.94/.107), and switched the DNSBL check to Control D (76.76.2.0)
which returns real Spamhaus ZEN data. (It was correctly SILENT lately
because delivery is healthy -- silent-on-healthy is by design.)
3. DMARC daily digest was pure noise: it alerted on ANY external IP with >=20
failing msgs, but those are legit recipient-side forwarders/security
gateways (inkyphishfence, cloud-sec-av, Proofpoint, Mimecast, ...) that
re-send our mail and naturally break SPF/DKIM alignment -- benign under
p=reject. Added PTR-based forwarder detection (FORWARDER_PTR_HINTS) so the
digest tags them [fwd] and only alerts on (a) OUR IP <95% pass or (b) an
UNKNOWN non-forwarder external IP with >=100 failing msgs = real spoofing.
Verified: all 4 currently-flagged external IPs now classify as forwarder=True.
When we resume Gmail sends, the front-loaded-inject + slow-drain pattern
buries mail: Listmonk stamps Date at injection (verified live: queued msg
Date matched postfix arrival, deferred 4h47m later), and Gmail sorts the
inbox by the Date header. So a msg injected at 08:00 but accepted at 14:00
files 6h down a Gmail inbox.
Documents: why NOT to future-date the Date header (spam signal + breaks our
DKIM which signs Date + doesn't help Outlook's received-time sort), and the
real fix -- pace Listmonk injection to match Gmail's accept rate (just-in-time
Date) via a dedicated Gmail stream on its own IP + low sliding-window rate +
queue-age guard. Outlook/M365 (current audience) sorts by received time so the
burial is cosmetic there and not worth fixing.
Procedure only; Gmail still excluded in _email_exclusions.py until re-enabled.
Consolidate the outbound mail footprint to match the SPF intent (already
trimmed to .94/.107 on 2026-06-19). A 20-IP sending footprint reads as
snowshoe spam to receivers and was contributing to domain-reputation
throttling (Microsoft 451 4.7.500, Gmail low-reputation).
Removed from /etc/postfix/master.cf: transports yahooslow, out02-04,
out06-20, rehab02-04, HC submission ports 2527/2528, hcout2/hcout3.
Removed from /etc/network/interfaces (+ live ip addr del): host bindings
.90-.93, .95-.106, .108-.109. Kept: .94 (trucking/out05), .107 (HC/hcout1),
.71/.72 (infra).
Verified live: postfix check OK, both streams still status=sent post-change,
SSH session on .71 unaffected, transport_maps still routes via out05.
Snapshots: infra/postfix/live-snapshots/master.cf, infra/network/interfaces.
Live backups on server: /root/{master.cf,interfaces}.bak_snowshoe_*.
api.performancewest.net uses an explicit per-path allowlist; everything else
falls through to a trusted-IP-only catch-all that returns 403. Six browser-
facing routes had no location block, so they 403'd for every public visitor:
/api/v1/npi/ <- THE healthcare sales killer. The 'Free NPI
Compliance Check' tool (top of the HC funnel,
where every HC campaign sends traffic) fetches
/api/v1/npi/lookup. It 403'd -> CORS error in
the browser -> the tool never rendered results
or the upsell CTAs (Revalidation $399 / NPPES
$149 / Bundle $899) -> 0 HC sales despite 17
sessions reaching it in 30d and 0 HC orders
EVER created in the compliance DB.
/api/v1/cdr/ telecom CDR profile tool
/api/v1/icc/ intrastate/ICC profile tool
/api/v1/corp/ corporate foreign-qual check
/api/v1/foreign-qualification/ foreign qualification quote/jurisdictions
/api/v1/lnpa-regions LNPA region lookup
Added explicit proxy_pass blocks (mirroring the existing entities/identity
pattern) before the catch-all. Verified live: all six now reach the app with
proper CORS; the NPI tool renders results + order CTAs end-to-end via a real
browser; npi-revalidation order page -> Stripe confirmed.
The live /etc/nginx/sites-enabled/pw-api.conf was hand-edited and untracked;
committing the current state here so it is version-controlled. (Live backup:
/root/pw-api.conf.bak_20260623.)
Two routing bugs that sent carriers to wrong/dead order pages:
1. MCS-150 + Inactive campaigns linked to /order/dot-full-compliance ($399)
instead of their actual service: build_lp_link()/lp_slug_for() fell through
to the dot-full-compliance catch-all for any campaign_type not in
DEFICIENCY_SEGMENTS, ignoring the existing PRICE_SLUG_BY_CAMPAIGN map. So
MCS-150 carriers (should be mcs150-update $79) and Inactive carriers (should
be usdot-reactivation $149) were both quoted a 5x-priced bundle they never
asked for — a severe conversion killer on the two highest-volume segments.
Fix: lp_slug_for() now checks PRICE_SLUG_BY_CAMPAIGN first; build_lp_link()
delegates to it (single source of truth).
2. IFTA-quarterly + UCR-annual builders set lp_link to a BARE path when no
coupon was active (LP_LINK with no query). The body appends '&utm_source=...'
so the CTA rendered as /order/ifta-quarterly&utm... (no '?') = 404. Fix:
both now always emit a leading '?' query carrying ?dot= (and ?code= when a
coupon is on), mirroring the main builder's lp_link_with_coupon().
Audited every campaign_type: all 14 order slugs now resolve 200 and match the
intended service/price. Compliance-check secondary links (/tools/dot-compliance-
check) verified correct and intentionally kept where a 'check status' CTA fits.
Root cause of the order-CTA 404s recurring after the prior live fix: the
builder clones email bodies from STORED Listmonk source campaigns (ids
186/188/271-274/309/310/469/473), not from the edited source files. Those
stored bodies still carried @TrackLink on the per-subscriber order CTA, so
every nightly build re-registered a single static /order/<slug>&utm... link
(no '?') that 404s for every recipient. This morning's 3,000 real sends AND
the owner spot-check both went out with dead order links.
Two durable guards:
1. get_base_campaign() now strips @TrackLink from any cloned body (with a
warning), so a stale/re-edited source campaign can never reach recipients
broken again. Human clicks are already attributed via Umami.
2. The owner test-send now builds the CTA via lp_link_with_coupon(dot=...)
(leading '?') instead of build_lp_link() (bare path).
Also fixed live: stripped @TrackLink from the 10 stored source campaign
bodies; rewrote the 12 already-registered broken links. Backups in listmonk:
pw_source_tracklink_bak_20260623 + pw_links_tracklink_bak_20260623.
The NPI/healthcare intake step persists provider email + name only into
intake_data (not the top-level state.email/state.name that the DOT/?dot=
flow sets). ReviewStep's order-create POST therefore sent empty
customer_email/customer_name -> API 400 'service_slug, customer_email, and
customer_name are required', blocking EVERY healthcare checkout at the
review step (explains 0 HC sales despite 13,425 sends).
ReviewStep now falls back to intake_data.{email,provider_name,
organization_name,legal_name,entity_name}; the Wizard cold-visitor create
path also now recognizes provider_name/organization_name. Verified the
trucking path is unaffected (it already populated top-level state).
The cold-visitor review-step path POSTed /validate with no Content-Type, so
the API returned 415 and validation silently failed — the user could create
the order but never advance from review to payment (the last blocker in the
trucking/HC checkout funnel). The Wizard's own validate call already set the
header; ReviewStep now matches. Completes the checkout repair in 5546c58.
Diagnosed via live browser E2E why campaign clicks (25 checkout-page-views,
36h) produced 0 conversions. Four bugs, all blocking checkout:
1. DOTIntakeStep: a missing `});` (DFWP hydration block, commit 9718ab9
Jun 2) left the pw:step-shown listener unclosed -> 'missing ) after
argument list' SYNTAX ERROR killed the whole DOT intake script. Effect:
?dot= prefill silently failed for ~3 weeks (exactly the campaign window),
so every carrier had to re-type all their details.
2. ReviewStep: service slug read from `.pw-step[data-slug]` (first match),
which on trucking/HC is the INTAKE step's slug ('dot-intake'/'npi-intake'),
not the order. The cold-visitor order-create POST sent
service_slug='dot-intake' -> API 501/400 -> 'Could not validate order',
blocking checkout at the review step on EVERY multi-step vertical. Now
reads `.pw-wizard[data-service]` (authoritative). Confirmed against prod:
bad slug=400, correct slug=201.
3. Shared-bundle null derefs: every step's <script> is bundled onto every
order page, so steps whose anchor element is absent threw at top level and
could abort siblings:
- ClassificationWizard: top-level renderQuestion(0) -> appendChild on
null (errored on 47/67 order pages)
- BDCDataStep: (querySelector as HTMLElement).getAttribute on null
- STIRShakenStep / EarthStationStep: top-level addEventListener on null
- ForeignQualStep: many top-level getElementById(...)! lookups
Each now guarded to no-op when its step isn't present.
Verified by browser E2E: full flow dot-intake -> review -> payment ->
live Stripe Checkout session, and a 67-page scan now reports 0 JS errors
(was 47 pages erroring). Real human clicks are tracked via Umami; these
were pure functional breakages of the conversion path.
Listmonk @TrackLink registers ONE static URL per tracked link and points
every recipient's /link/<uuid> redirect at it. On per-subscriber hrefs
({{ lp_link }}, ?dot=, ?npi=, ?clia=) this is doubly broken:
- the registered links.url was captured before the {{ lp_link }} token
rendered, yielding /order/slug&utm_source=... (first &, no ?) -> 404
- even when valid it collapses every carrier/provider onto the first
subscriber's dot/npi/clia value
Real human clicks are already tracked via Umami campaign-click (bot
filtered), so Listmonk link tracking here is redundant and destructive.
Stripped @TrackLink from per-subscriber CTAs:
- scripts/create_deficiency_source_campaigns.py (_cta, _dot_check_cta)
- data/trucking_campaigns/{ucr,ifta}_*.html
- data/hc_campaigns/*.html (10 templates)
Static CTAs (e.g. CRTC ?code= order link) keep @TrackLink (safe).
Live fix to the 10 broken registered links.url rows applied separately
(first & -> ?), backup in listmonk.pw_links_dkim_fix_bak_20260622.
Docs: new runbook incident section + corrected the disproven
'use @TrackLink on all CTAs' guidance in fmcsa/hc plans.
The day-9 Gmail block that forced the 200/h hold is resolved: per-MX throttling
shipped, Google is excluded entirely (MAIN_EXCLUDE_OPERATORS=google), and the
OpenDKIM signing bug is fixed. With Google out of the mix, 400/h (~4k/day) is
within the envelope these IPs cleanly sustained at 68-76% delivery with zero
blocks. Lets the post-DKIM re-send backlog drain in ~1 day instead of ~3.
Records the MAIN_EXCLUDE_OPERATORS=google override, the resend_dkim_backup_20260622
rollback table, the past-send_at HTTP 400 gotcha (use --send-hour for same-day
re-runs), and the exact revert SQL. 6461-row backup; ~2999 re-sent Jun 22, rest
drain on subsequent daily runs (Gmail excluded, Microsoft/Hotmail included).
After the Jun 2026 no-DKIM incident (campaign mail went out unsigned ->
junked/blocked, ~23% delivery), DKIM is fixed and we must re-send to the
now-signed audience. The builder previously held Google AND Microsoft AND
consumer-MX out until warmup day 30; that blocks the re-send of the Microsoft-
hosted business domains that are most of the list.
Add MAIN_EXCLUDE_OPERATORS (comma-separated mx_provider labels) to override
WARMUP_EXCLUDE_OPERATORS. Set it to 'google' in the workers env so we send to
everything EXCEPT Google's consumer inboxes (still recovering reputation),
including Microsoft/Hotmail. Drives both the SQL exclude and the per-operator
daily cap consistently. Unset => prior default; '' => exclude nobody.
Listmonk applies campaign headers as `for hdr,val := range set { h.Add(hdr,val) }`
(internal/manager/manager.go v6.1.0): each map's KEY is the literal header name.
The trucking/CRTC/deficiency builders wrote {"name":"Reply-To","value":..} (and
{"key":..,"value":..}), which emits junk `name:`/`value:` headers and NO real
Reply-To, so replies fell back to the From address (noreply@send.performancewest.net)
instead of info@performancewest.net. HC builder already used the correct
{"Reply-To": value} shape; match it everywhere. Verified against listmonk source.
Impact: outbound only; no customer replies were lost (noreply@ is a real mailbox),
but reply UX pointed at a no-reply address. Live campaign headers re-patched separately.
CAMPAIGN_COUPON_AB_PCTS="20,30,0" now means 20% / 30% / full-price. The 0 arm
mints no code; pick_coupon_for_email returns ("","") so it renders identically
to a normal-price send, while carriers are still deterministically hash-bucketed
into it (re-hash a converter's email to recover their arm). Even ~33/33/33 split
incl. the control verified over 30k. Adds test_full_price_control_arm; 8/8 pass.
Two correctness fixes that gate enabling the coupon test:
1. On-the-fly pricing. The coupon block hardcoded '$79 $47' (only true at 40%
off) — a false claim on the 20/30% arms. Now build_trucking_campaigns.py
reads api/src/service-catalog.ts (same source checkout uses) and computes
coupon_price_full / coupon_price_deal per recipient as full - round(full*pct/100),
exactly matching the server. Service-fee-only; non-discountable services
(boc3-filing passthrough) get NO price and fall back to percent-only copy.
Quotes the service the email is ABOUT (mcs150 $79, reactivation $149), not the
bundle the CTA happens to link to. service-catalog.ts now ships in the worker
image; helper degrades to percent-only if it can't be read.
2. CTA URL bug (likely a big driver of the zero-click problem). Main campaign
CTAs render '/order/slug&utm_source=...' (no '?') -> HTTP 404, verified live.
Deficiency CTAs would double-'?' once a coupon added '?code='. lp_link now
owns the query (?dot=...&code=...) so every template appends with a leading
'&' and is valid in all 4 states (main/deficiency x coupon on/off), verified
against live URLs returning 200.
Deficiency _deal_box now shows real was/now prices (percent-only for boc3).
Tests: 7/7 pass (adds URL-wellformed + price-matches-checkout cases).
- CAMPAIGN_COUPON_AB_PCTS="20,30,40" mints one daily code per arm; each
carrier is bucketed by a stable sha256(email) hash so the split is even
(~33/33/33 verified over 30k) and stable across re-sends (no arm-hopping).
- Each arm's code stores its own percent in discount_codes, so the advertised
discount always matches what checkout applies; redemptions are countable per
code (marker campaign-daily:<date>:<pct>).
- Empty/unset keeps legacy single-arm behavior (COUPON_PCT, legacy marker).
- coupon_attribs() now takes per-recipient pct.
- Tests: scripts/tests/test_coupon_ab.py (5 pass). SpamAssassin: both main
campaigns (186/188) score 0.0 HAM across all 3 arms, coupon block renders
clean; harness saved for re-runs.
Found during a bug-review pass of the one-email-per-provider work:
1. assign_all overwrite bug: an email on MULTIPLE rows (shared practice inbox /
multiple NPIs -- 2,592 such emails, 299 with mixed status) was assigned by
the LAST row, so a less-urgent row could clobber an urgent one (overdue ->
free check). Now keeps the most-urgent (lowest-priority) assignment.
2. warm_segment double-import + wrong-row render: all of an email's rows passed
the candidate filter, so it could be imported twice (over-counting the slice)
and attribs_for could render a sibling row's blank due-date in the overdue
email. Now requires row_matches(seg) for the specific row AND dedupes by
email (one row per email).
3. free-check email rendered broken text ('last updated on -- about years
ago', 'Last updated . ~ yrs ago') for any provider whose NPPES date isn't
cached yet (the free check goes to everyone, and the fill is gradual). Wrapped
the example sentence + official-record card in listmonk {{ if
.nppes_last_updated }}...{{ else }}...{{ end }}; added a date-free else
branch. altbody keeps the conditionals (listmonk evaluates body+altbody), and
the test/preview renderer gained a minimal {{ if/else/end }} evaluator so
previews match real sends. Verified both branches render with zero unfilled
tokens.
4. cross-cron double-send: pw-hc-campaign (warmup file) and pw-hc-nppes (63k
file) share state but tracked imports per-segment; 312 emails overlap both
files, so a provider could get an urgent email from one cron AND the free
check from the other. Added load_all_imported() global guard (union of all
segment state) so each provider gets exactly one healthcare email overall.
All verified: assignment regression test (10 cases) + new dup-email/guard checks
pass; all 6 templates render clean.
Make the free NPI compliance check the catch-all for ALL verified institutional
providers, but route anyone with a more important/time-sensitive issue to THAT
email instead -- each provider gets exactly one email, their most urgent.
- SEGMENTS gain a 'priority' (lower=more urgent): reactivation 10, revalidation
overdue 20, due-soon 30, bundle 45, free-NPI-check 100 (catch-all).
- assign_segment()/assign_all(): route each provider to the single
highest-priority active segment whose selector matches; warm_segment() takes
the assignment map and only claims its assigned providers (disjoint pools, no
double-mailing). main() now splits the daily slice by priority order, serving
urgent segments fully before the broad free-check consumes the remainder.
- nppes_outdated selector -> 'institutional_default' (every verified, non-
deactivated row), since the free check's value no longer depends on staleness;
list/campaign renamed 'HC Warmup - Free NPI Check'.
- FIX latent bug: reactivation selector treated 'not on CMS reval list' as
deactivated -- false for org NPIs (would mis-tell active practices they're
deactivated). Now uses the REAL nppes_deactivated flag (or OIG/SAM exclusion).
- Drop blanket oig_screening from the default rotation: it matched every row and
would starve the catch-all, and the free check already screens OIG/SAM and
routes to the paid fix on a hit. Still runnable via --segments.
- Add scripts/test_segment_assignment.py (10 cases incl. 'overdue AND stale ->
overdue wins'); all pass.
Pivot the weakest healthcare email from an 'your record is out of date -> buy an
update' sell into a free, value-first compliance check (the funnel already
exists: /tools/npi-compliance-check + /api/v1/npi/lookup run 5 live gov checks --
NPI status, Medicare revalidation, OIG/SAM exclusions, NPPES freshness -- and
deep-link to the right paid fix).
- Subject: 'A free compliance check for your NPI' (was 'may be out of date').
- Header: 'Free NPI Compliance Check' covering NPPES/revalidation/exclusions/NPI.
- Body: keep the REAL last_updated date as a credibility hook ('we pulled your
public records'), but frame it honestly ('that's usually fine') and pivot to
the broader free check. Adds a 4-item 'your free check covers' card.
- CTA now -> /tools/npi-compliance-check?npi={npi} (prefills + auto-runs their
own check on landing) with @TrackLink + UTM; dropped the straight-to-order
NPPES CTA and the redundant 'look up on NPPES' button.
- Reassurance reframed to free-first ('the check is completely free; a fix is
optional, flat-fee'). cta_path updated in the segment registry.
- Verified: render + plaintext + headless screenshot, CTA tracked, no stray
order link, zero unfilled tokens.
