The HC warmup imported ~1000 fresh providers/day into a persistent segment list
(list 10), but each day's campaign targeted that WHOLE cumulative list -- so the
cohort imported on day 1 received the identical 'Free NPI Check' email every
subsequent day (verified: subscriber 3410 got campaigns 38-42, 5x). Program-wide
that was 23,843 sends to 6,587 people (3.6x avg, max 30x) and blocklisted ~12%
of the list -- a Yahoo 'user complaints' deferral now confirms the burn.
Fix: import each day's slice into a dedicated dated SEND list and bind that day's
campaign to ONLY that list, so every provider gets exactly one send. The
persistent segment list is still used for dedup/records. ensure_campaign now
matches the exact dated name (never reuses a prior day's campaign/slice).
Unattended kernel-upgrade reboot (Jun 24 04:04) left only .71 bound because
classic ifupdown applies just the first 'address' line. Postfix then failed to
bind .94/.107 ('Cannot assign requested address') and silently egressed from
.71 -- which is NOT in SPF (every fallback msg failed SPF) and is on RLR621 +
Trend ERS-QIL. ~37h of bypassed IP-warming + a near-zero sales day.
Fixes:
- /etc/network/interfaces: explicit up/down ip-addr hooks for .72/.94/.107
- pw-mail-ips.service: systemd oneshot re-binds IPs + flushes queue on boot
- pw-mail-ip-watchdog: */5 cron re-binds missing IPs + flushes, also catches
'Cannot assign' bind failures
- runbook: full incident writeup + reboot-test lesson
Host already remediated live; this commits the host artifacts + docs.
The dev deploy.sh is an identical copy of prod's, so its hardcoded
`cd /opt/performancewest` meant running /opt/performancewest-dev/deploy.sh
silently rebuilt and recreated PROD's containers instead of dev's. Resolve the
script's real directory (readlink -f handles symlinks/relative paths) so the
same tracked script is correct in every clone; docker compose then derives the
project name + auto-loads docker-compose.override.yml from that directory,
keeping dev (project performancewest-dev) and prod isolated.
Conversion fix for the checkout drop-off (54 sessions reached an /order/ page
over 3 days, 0 advanced to payment). Root cause was friction, not a bug: every
order page dropped a cold email-click straight into a 28-field intake Wizard
before showing any payment option.
- New ExpressCheckout.astro: payment-first entry. Shows price + the minimal
fields the API needs (prefilled from public records: ?dot= FMCSA census for
trucking, ?npi= NPPES lookup for healthcare) + Continue to payment. Creates a
single-service batch-of-one (POST /compliance-orders/batch, which does NOT
gate Stripe on intake_data_validated) then create-session -> Stripe. Full
intake is collected AFTER payment via the per-service 'Complete Your Intake
Form' email the webhook already sends (links to /order/<slug>?order=CO-xxx,
which re-enters the Wizard in paid-intake mode).
- New OrderFlow.astro: single source of truth replacing ~50 near-identical thin
Wizard wrappers. Trucking + healthcare default to payment-first (express on
top, marketing hero moved BELOW the CTA). Telecom + corporate keep Wizard-first
(rich pre-payment FCC/499 intake, no public-records prefill). Paid-intake
re-entry (?order=/?token=) always renders the full Wizard.
- Rewrote all 50 /order/*.astro pages to use OrderFlow (foreign-qualification
keeps its multi-state toggle via slotted content).
- Fixed the dead Tawk.to live-chat widget site-wide: the snippet set an invalid
crossorigin='*' attribute, forcing the browser into anonymous CORS mode and
blocking the script (0 chat requests fired anywhere). Removed it to match
Tawk's official snippet (footer partial + 73 static public/*.html files).
Verified: build clean; express on top with hero below; ?dot=/?npi= prefill;
paid-intake re-entry swaps to Wizard; telecom stays wizard-first; batch-of-one
-> live Stripe URL; both POST endpoints allow the prod origin via CORS.
Three bugs the owner hit:
1. Per-operator reputation alert (06:10 cron, mail_reputation_monitor --alert)
silently never ran: it redirected to /var/log/pw-mail-reputation.log but
/var/log is root-only and that file was never pre-created, so the deploy
user's >> redirect failed and cron aborted before the command. Repointed
both mail-alert crons to deploy-writable /opt/performancewest/logs/.
2. IP reputation alert (20:00 cron) still referenced the removed rehab pool
(.91-.93) and used 8.8.8.8 for Spamhaus (which returns the open-resolver
error, not a real answer). Dropped the rehab section, relabeled to the two
live IPs (.94/.107), and switched the DNSBL check to Control D (76.76.2.0)
which returns real Spamhaus ZEN data. (It was correctly SILENT lately
because delivery is healthy -- silent-on-healthy is by design.)
3. DMARC daily digest was pure noise: it alerted on ANY external IP with >=20
failing msgs, but those are legit recipient-side forwarders/security
gateways (inkyphishfence, cloud-sec-av, Proofpoint, Mimecast, ...) that
re-send our mail and naturally break SPF/DKIM alignment -- benign under
p=reject. Added PTR-based forwarder detection (FORWARDER_PTR_HINTS) so the
digest tags them [fwd] and only alerts on (a) OUR IP <95% pass or (b) an
UNKNOWN non-forwarder external IP with >=100 failing msgs = real spoofing.
Verified: all 4 currently-flagged external IPs now classify as forwarder=True.
When we resume Gmail sends, the front-loaded-inject + slow-drain pattern
buries mail: Listmonk stamps Date at injection (verified live: queued msg
Date matched postfix arrival, deferred 4h47m later), and Gmail sorts the
inbox by the Date header. So a msg injected at 08:00 but accepted at 14:00
files 6h down a Gmail inbox.
Documents: why NOT to future-date the Date header (spam signal + breaks our
DKIM which signs Date + doesn't help Outlook's received-time sort), and the
real fix -- pace Listmonk injection to match Gmail's accept rate (just-in-time
Date) via a dedicated Gmail stream on its own IP + low sliding-window rate +
queue-age guard. Outlook/M365 (current audience) sorts by received time so the
burial is cosmetic there and not worth fixing.
Procedure only; Gmail still excluded in _email_exclusions.py until re-enabled.
Consolidate the outbound mail footprint to match the SPF intent (already
trimmed to .94/.107 on 2026-06-19). A 20-IP sending footprint reads as
snowshoe spam to receivers and was contributing to domain-reputation
throttling (Microsoft 451 4.7.500, Gmail low-reputation).
Removed from /etc/postfix/master.cf: transports yahooslow, out02-04,
out06-20, rehab02-04, HC submission ports 2527/2528, hcout2/hcout3.
Removed from /etc/network/interfaces (+ live ip addr del): host bindings
.90-.93, .95-.106, .108-.109. Kept: .94 (trucking/out05), .107 (HC/hcout1),
.71/.72 (infra).
Verified live: postfix check OK, both streams still status=sent post-change,
SSH session on .71 unaffected, transport_maps still routes via out05.
Snapshots: infra/postfix/live-snapshots/master.cf, infra/network/interfaces.
Live backups on server: /root/{master.cf,interfaces}.bak_snowshoe_*.
api.performancewest.net uses an explicit per-path allowlist; everything else
falls through to a trusted-IP-only catch-all that returns 403. Six browser-
facing routes had no location block, so they 403'd for every public visitor:
/api/v1/npi/ <- THE healthcare sales killer. The 'Free NPI
Compliance Check' tool (top of the HC funnel,
where every HC campaign sends traffic) fetches
/api/v1/npi/lookup. It 403'd -> CORS error in
the browser -> the tool never rendered results
or the upsell CTAs (Revalidation $399 / NPPES
$149 / Bundle $899) -> 0 HC sales despite 17
sessions reaching it in 30d and 0 HC orders
EVER created in the compliance DB.
/api/v1/cdr/ telecom CDR profile tool
/api/v1/icc/ intrastate/ICC profile tool
/api/v1/corp/ corporate foreign-qual check
/api/v1/foreign-qualification/ foreign qualification quote/jurisdictions
/api/v1/lnpa-regions LNPA region lookup
Added explicit proxy_pass blocks (mirroring the existing entities/identity
pattern) before the catch-all. Verified live: all six now reach the app with
proper CORS; the NPI tool renders results + order CTAs end-to-end via a real
browser; npi-revalidation order page -> Stripe confirmed.
The live /etc/nginx/sites-enabled/pw-api.conf was hand-edited and untracked;
committing the current state here so it is version-controlled. (Live backup:
/root/pw-api.conf.bak_20260623.)
Two routing bugs that sent carriers to wrong/dead order pages:
1. MCS-150 + Inactive campaigns linked to /order/dot-full-compliance ($399)
instead of their actual service: build_lp_link()/lp_slug_for() fell through
to the dot-full-compliance catch-all for any campaign_type not in
DEFICIENCY_SEGMENTS, ignoring the existing PRICE_SLUG_BY_CAMPAIGN map. So
MCS-150 carriers (should be mcs150-update $79) and Inactive carriers (should
be usdot-reactivation $149) were both quoted a 5x-priced bundle they never
asked for — a severe conversion killer on the two highest-volume segments.
Fix: lp_slug_for() now checks PRICE_SLUG_BY_CAMPAIGN first; build_lp_link()
delegates to it (single source of truth).
2. IFTA-quarterly + UCR-annual builders set lp_link to a BARE path when no
coupon was active (LP_LINK with no query). The body appends '&utm_source=...'
so the CTA rendered as /order/ifta-quarterly&utm... (no '?') = 404. Fix:
both now always emit a leading '?' query carrying ?dot= (and ?code= when a
coupon is on), mirroring the main builder's lp_link_with_coupon().
Audited every campaign_type: all 14 order slugs now resolve 200 and match the
intended service/price. Compliance-check secondary links (/tools/dot-compliance-
check) verified correct and intentionally kept where a 'check status' CTA fits.
Root cause of the order-CTA 404s recurring after the prior live fix: the
builder clones email bodies from STORED Listmonk source campaigns (ids
186/188/271-274/309/310/469/473), not from the edited source files. Those
stored bodies still carried @TrackLink on the per-subscriber order CTA, so
every nightly build re-registered a single static /order/<slug>&utm... link
(no '?') that 404s for every recipient. This morning's 3,000 real sends AND
the owner spot-check both went out with dead order links.
Two durable guards:
1. get_base_campaign() now strips @TrackLink from any cloned body (with a
warning), so a stale/re-edited source campaign can never reach recipients
broken again. Human clicks are already attributed via Umami.
2. The owner test-send now builds the CTA via lp_link_with_coupon(dot=...)
(leading '?') instead of build_lp_link() (bare path).
Also fixed live: stripped @TrackLink from the 10 stored source campaign
bodies; rewrote the 12 already-registered broken links. Backups in listmonk:
pw_source_tracklink_bak_20260623 + pw_links_tracklink_bak_20260623.
The NPI/healthcare intake step persists provider email + name only into
intake_data (not the top-level state.email/state.name that the DOT/?dot=
flow sets). ReviewStep's order-create POST therefore sent empty
customer_email/customer_name -> API 400 'service_slug, customer_email, and
customer_name are required', blocking EVERY healthcare checkout at the
review step (explains 0 HC sales despite 13,425 sends).
ReviewStep now falls back to intake_data.{email,provider_name,
organization_name,legal_name,entity_name}; the Wizard cold-visitor create
path also now recognizes provider_name/organization_name. Verified the
trucking path is unaffected (it already populated top-level state).
The cold-visitor review-step path POSTed /validate with no Content-Type, so
the API returned 415 and validation silently failed — the user could create
the order but never advance from review to payment (the last blocker in the
trucking/HC checkout funnel). The Wizard's own validate call already set the
header; ReviewStep now matches. Completes the checkout repair in 5546c58.
Diagnosed via live browser E2E why campaign clicks (25 checkout-page-views,
36h) produced 0 conversions. Four bugs, all blocking checkout:
1. DOTIntakeStep: a missing `});` (DFWP hydration block, commit 9718ab9
Jun 2) left the pw:step-shown listener unclosed -> 'missing ) after
argument list' SYNTAX ERROR killed the whole DOT intake script. Effect:
?dot= prefill silently failed for ~3 weeks (exactly the campaign window),
so every carrier had to re-type all their details.
2. ReviewStep: service slug read from `.pw-step[data-slug]` (first match),
which on trucking/HC is the INTAKE step's slug ('dot-intake'/'npi-intake'),
not the order. The cold-visitor order-create POST sent
service_slug='dot-intake' -> API 501/400 -> 'Could not validate order',
blocking checkout at the review step on EVERY multi-step vertical. Now
reads `.pw-wizard[data-service]` (authoritative). Confirmed against prod:
bad slug=400, correct slug=201.
3. Shared-bundle null derefs: every step's <script> is bundled onto every
order page, so steps whose anchor element is absent threw at top level and
could abort siblings:
- ClassificationWizard: top-level renderQuestion(0) -> appendChild on
null (errored on 47/67 order pages)
- BDCDataStep: (querySelector as HTMLElement).getAttribute on null
- STIRShakenStep / EarthStationStep: top-level addEventListener on null
- ForeignQualStep: many top-level getElementById(...)! lookups
Each now guarded to no-op when its step isn't present.
Verified by browser E2E: full flow dot-intake -> review -> payment ->
live Stripe Checkout session, and a 67-page scan now reports 0 JS errors
(was 47 pages erroring). Real human clicks are tracked via Umami; these
were pure functional breakages of the conversion path.
Listmonk @TrackLink registers ONE static URL per tracked link and points
every recipient's /link/<uuid> redirect at it. On per-subscriber hrefs
({{ lp_link }}, ?dot=, ?npi=, ?clia=) this is doubly broken:
- the registered links.url was captured before the {{ lp_link }} token
rendered, yielding /order/slug&utm_source=... (first &, no ?) -> 404
- even when valid it collapses every carrier/provider onto the first
subscriber's dot/npi/clia value
Real human clicks are already tracked via Umami campaign-click (bot
filtered), so Listmonk link tracking here is redundant and destructive.
Stripped @TrackLink from per-subscriber CTAs:
- scripts/create_deficiency_source_campaigns.py (_cta, _dot_check_cta)
- data/trucking_campaigns/{ucr,ifta}_*.html
- data/hc_campaigns/*.html (10 templates)
Static CTAs (e.g. CRTC ?code= order link) keep @TrackLink (safe).
Live fix to the 10 broken registered links.url rows applied separately
(first & -> ?), backup in listmonk.pw_links_dkim_fix_bak_20260622.
Docs: new runbook incident section + corrected the disproven
'use @TrackLink on all CTAs' guidance in fmcsa/hc plans.
The day-9 Gmail block that forced the 200/h hold is resolved: per-MX throttling
shipped, Google is excluded entirely (MAIN_EXCLUDE_OPERATORS=google), and the
OpenDKIM signing bug is fixed. With Google out of the mix, 400/h (~4k/day) is
within the envelope these IPs cleanly sustained at 68-76% delivery with zero
blocks. Lets the post-DKIM re-send backlog drain in ~1 day instead of ~3.
Records the MAIN_EXCLUDE_OPERATORS=google override, the resend_dkim_backup_20260622
rollback table, the past-send_at HTTP 400 gotcha (use --send-hour for same-day
re-runs), and the exact revert SQL. 6461-row backup; ~2999 re-sent Jun 22, rest
drain on subsequent daily runs (Gmail excluded, Microsoft/Hotmail included).
After the Jun 2026 no-DKIM incident (campaign mail went out unsigned ->
junked/blocked, ~23% delivery), DKIM is fixed and we must re-send to the
now-signed audience. The builder previously held Google AND Microsoft AND
consumer-MX out until warmup day 30; that blocks the re-send of the Microsoft-
hosted business domains that are most of the list.
Add MAIN_EXCLUDE_OPERATORS (comma-separated mx_provider labels) to override
WARMUP_EXCLUDE_OPERATORS. Set it to 'google' in the workers env so we send to
everything EXCEPT Google's consumer inboxes (still recovering reputation),
including Microsoft/Hotmail. Drives both the SQL exclude and the per-operator
daily cap consistently. Unset => prior default; '' => exclude nobody.
Listmonk applies campaign headers as `for hdr,val := range set { h.Add(hdr,val) }`
(internal/manager/manager.go v6.1.0): each map's KEY is the literal header name.
The trucking/CRTC/deficiency builders wrote {"name":"Reply-To","value":..} (and
{"key":..,"value":..}), which emits junk `name:`/`value:` headers and NO real
Reply-To, so replies fell back to the From address (noreply@send.performancewest.net)
instead of info@performancewest.net. HC builder already used the correct
{"Reply-To": value} shape; match it everywhere. Verified against listmonk source.
Impact: outbound only; no customer replies were lost (noreply@ is a real mailbox),
but reply UX pointed at a no-reply address. Live campaign headers re-patched separately.
CAMPAIGN_COUPON_AB_PCTS="20,30,0" now means 20% / 30% / full-price. The 0 arm
mints no code; pick_coupon_for_email returns ("","") so it renders identically
to a normal-price send, while carriers are still deterministically hash-bucketed
into it (re-hash a converter's email to recover their arm). Even ~33/33/33 split
incl. the control verified over 30k. Adds test_full_price_control_arm; 8/8 pass.
Two correctness fixes that gate enabling the coupon test:
1. On-the-fly pricing. The coupon block hardcoded '$79 $47' (only true at 40%
off) — a false claim on the 20/30% arms. Now build_trucking_campaigns.py
reads api/src/service-catalog.ts (same source checkout uses) and computes
coupon_price_full / coupon_price_deal per recipient as full - round(full*pct/100),
exactly matching the server. Service-fee-only; non-discountable services
(boc3-filing passthrough) get NO price and fall back to percent-only copy.
Quotes the service the email is ABOUT (mcs150 $79, reactivation $149), not the
bundle the CTA happens to link to. service-catalog.ts now ships in the worker
image; helper degrades to percent-only if it can't be read.
2. CTA URL bug (likely a big driver of the zero-click problem). Main campaign
CTAs render '/order/slug&utm_source=...' (no '?') -> HTTP 404, verified live.
Deficiency CTAs would double-'?' once a coupon added '?code='. lp_link now
owns the query (?dot=...&code=...) so every template appends with a leading
'&' and is valid in all 4 states (main/deficiency x coupon on/off), verified
against live URLs returning 200.
Deficiency _deal_box now shows real was/now prices (percent-only for boc3).
Tests: 7/7 pass (adds URL-wellformed + price-matches-checkout cases).
- CAMPAIGN_COUPON_AB_PCTS="20,30,40" mints one daily code per arm; each
carrier is bucketed by a stable sha256(email) hash so the split is even
(~33/33/33 verified over 30k) and stable across re-sends (no arm-hopping).
- Each arm's code stores its own percent in discount_codes, so the advertised
discount always matches what checkout applies; redemptions are countable per
code (marker campaign-daily:<date>:<pct>).
- Empty/unset keeps legacy single-arm behavior (COUPON_PCT, legacy marker).
- coupon_attribs() now takes per-recipient pct.
- Tests: scripts/tests/test_coupon_ab.py (5 pass). SpamAssassin: both main
campaigns (186/188) score 0.0 HAM across all 3 arms, coupon block renders
clean; harness saved for re-runs.
Found during a bug-review pass of the one-email-per-provider work:
1. assign_all overwrite bug: an email on MULTIPLE rows (shared practice inbox /
multiple NPIs -- 2,592 such emails, 299 with mixed status) was assigned by
the LAST row, so a less-urgent row could clobber an urgent one (overdue ->
free check). Now keeps the most-urgent (lowest-priority) assignment.
2. warm_segment double-import + wrong-row render: all of an email's rows passed
the candidate filter, so it could be imported twice (over-counting the slice)
and attribs_for could render a sibling row's blank due-date in the overdue
email. Now requires row_matches(seg) for the specific row AND dedupes by
email (one row per email).
3. free-check email rendered broken text ('last updated on -- about years
ago', 'Last updated . ~ yrs ago') for any provider whose NPPES date isn't
cached yet (the free check goes to everyone, and the fill is gradual). Wrapped
the example sentence + official-record card in listmonk {{ if
.nppes_last_updated }}...{{ else }}...{{ end }}; added a date-free else
branch. altbody keeps the conditionals (listmonk evaluates body+altbody), and
the test/preview renderer gained a minimal {{ if/else/end }} evaluator so
previews match real sends. Verified both branches render with zero unfilled
tokens.
4. cross-cron double-send: pw-hc-campaign (warmup file) and pw-hc-nppes (63k
file) share state but tracked imports per-segment; 312 emails overlap both
files, so a provider could get an urgent email from one cron AND the free
check from the other. Added load_all_imported() global guard (union of all
segment state) so each provider gets exactly one healthcare email overall.
All verified: assignment regression test (10 cases) + new dup-email/guard checks
pass; all 6 templates render clean.
Make the free NPI compliance check the catch-all for ALL verified institutional
providers, but route anyone with a more important/time-sensitive issue to THAT
email instead -- each provider gets exactly one email, their most urgent.
- SEGMENTS gain a 'priority' (lower=more urgent): reactivation 10, revalidation
overdue 20, due-soon 30, bundle 45, free-NPI-check 100 (catch-all).
- assign_segment()/assign_all(): route each provider to the single
highest-priority active segment whose selector matches; warm_segment() takes
the assignment map and only claims its assigned providers (disjoint pools, no
double-mailing). main() now splits the daily slice by priority order, serving
urgent segments fully before the broad free-check consumes the remainder.
- nppes_outdated selector -> 'institutional_default' (every verified, non-
deactivated row), since the free check's value no longer depends on staleness;
list/campaign renamed 'HC Warmup - Free NPI Check'.
- FIX latent bug: reactivation selector treated 'not on CMS reval list' as
deactivated -- false for org NPIs (would mis-tell active practices they're
deactivated). Now uses the REAL nppes_deactivated flag (or OIG/SAM exclusion).
- Drop blanket oig_screening from the default rotation: it matched every row and
would starve the catch-all, and the free check already screens OIG/SAM and
routes to the paid fix on a hit. Still runnable via --segments.
- Add scripts/test_segment_assignment.py (10 cases incl. 'overdue AND stale ->
overdue wins'); all pass.
Pivot the weakest healthcare email from an 'your record is out of date -> buy an
update' sell into a free, value-first compliance check (the funnel already
exists: /tools/npi-compliance-check + /api/v1/npi/lookup run 5 live gov checks --
NPI status, Medicare revalidation, OIG/SAM exclusions, NPPES freshness -- and
deep-link to the right paid fix).
- Subject: 'A free compliance check for your NPI' (was 'may be out of date').
- Header: 'Free NPI Compliance Check' covering NPPES/revalidation/exclusions/NPI.
- Body: keep the REAL last_updated date as a credibility hook ('we pulled your
public records'), but frame it honestly ('that's usually fine') and pivot to
the broader free check. Adds a 4-item 'your free check covers' card.
- CTA now -> /tools/npi-compliance-check?npi={npi} (prefills + auto-runs their
own check on landing) with @TrackLink + UTM; dropped the straight-to-order
NPPES CTA and the redundant 'look up on NPPES' button.
- Reassurance reframed to free-first ('the check is completely free; a fix is
optional, flat-fee'). cta_path updated in the segment registry.
- Verified: render + plaintext + headless screenshot, CTA tracked, no stray
order link, zero unfilled tokens.
An old NPPES last_updated date does NOT mean the practice closed or that CMS
penalizes them: an NPI never expires and there is no NPPES login schedule. Many
records are stale precisely because nothing changed. Removed the overclaim that
an old record 'has almost certainly drifted' and the false 'attest periodically'
duty. Now states the real rule (correct NPPES within 30 days of a change) and
makes the harm conditional ('if anything has changed since then, your record is
now out of date'). Keeps NPPES distinct from Medicare revalidation/PECOS, which
is the separate segment that actually carries deactivation stakes.
- Add NPPES_STALE_MAX_YEARS (default 10): a record untouched for many years is
a stronger signal the practice closed/moved, and a bounce burns the warming
IP. Observed institutional distribution clusters 3-7yrs with ~0 beyond 8, so
10 is a safe ceiling that mails the whole real pool while excluding any
outlier ancient record. MIN stays 3 (keeps the 'out of date' claim credible).
- Restore the SMTP-verification gate (verify_ok) that the shared
institutional_verified selector had -- the swap to nppes_stale dropped it; we
only mail inboxes we already proved live.
- enrich: process the re-fetch queue STALEST-FIRST so a bounded (--limit) or
--max-age refresh spends its budget on the most-overdue cache entries (and new
NPIs) first, never starving them behind merely-aging ones.
- Selector unit-tested (10 cases incl. window edges, verify gate, deactivated).
The NPPES 'may be out of date' email previously asserted staleness with no
per-record evidence (softened earlier to a generic 'periodic review required').
NPPES is fully public and every record carries basic.last_updated, so we now
cite the actual government date the provider can verify on the registry.
- enrich_nppes_last_updated.py: joins real basic.last_updated /
enumeration_date / deactivated onto the institutional list via a cached,
resumable per-NPI crawl (no batch endpoint exists). Adds nppes_last_updated,
nppes_enumeration, nppes_years_stale, nppes_deactivated.
- cron: new 'nppes_stale' selector mails ONLY records >= 3yrs stale (env
HC_NPPES_STALE_MIN_YEARS) and excludes deactivated NPIs; empty date => no
match, so we never claim staleness without the government date to back it.
- template: headline + official-record card now show the real last_updated
date and ~N-years-ago, sourced to npiregistry.cms.hhs.gov.
- attribs + test SAMPLE expose the new fields; verified render + plaintext.
Diagnosing zero healthcare sales (11k sent, 5479 opens, 0 clicks, 0 orders).
Root cause of clicks=0: Listmonk only registers a link for tracking when the
href ends with the literal @TrackLink marker; all 10 hc templates lacked it
(trucking/CRTC have it). So the entire funnel was unmeasurable below 'open'.
Changes:
- Click tracking: append @TrackLink + UTM to every /order/ CTA across all 10
templates (external gov self-verify links left untracked on purpose).
- Remove all service prices from emails (99/49/49/99yr/9mo). Price is
now revealed on the order page after value is established; catalog
(api/src/service-catalog.ts) stays source of truth. Kept the 0,000 OIG
penalty stat (regulatory fact, not our price). Added a neutral 'flat fee shown
up front' reassurance block where the fee table used to be.
- Compliance/honesty: the nppes_outdated email asserted a per-record
'FLAGGED OUT OF DATE / detected' status, but its selector only checks
deliverability and the data has no NPPES last-updated field -> unsubstantiated
for every recipient. Reframed to a generally-true periodic-attestation message
('PERIODIC REVIEW REQUIRED', 'most practices drift out of date'). Same hedging
applied to npi_reactivation ('may be deactivated ... confirm on official
sources'). Substantiated reval 'past due' claims (backed by the public CMS
Revalidation list) were kept.
- Fixed stale $299 OIG metadata in build script -> $79/mo (reference only).
Docs: docs/healthcare-competitive-pricing.md (benchmark research) and
docs/healthcare-email-compliance-review.md (CAN-SPAM / FTC / impersonation pass;
flags SOC2/HIPAA/PCI badge claims for owner confirmation).
Verified headless: all 10 render with 0 JS errors, exactly 1 tracked CTA each,
no price leaks.
- New 'Choose your province: BC or Ontario' comparison card (entity name,
registered office city, fees, annual return, portal, area codes, corp tax)
inserted above the carriers banner. Previously Ontario was only mentioned
in a buried FAQ; BC outnumbered ON 53:12.
- Tax-comparison H2 + collapse-menu label now read 'British Columbia / Ontario'
and the key-takeaway notes ON is ~12.2% (within ~1pt of BC).
- Made hero chip, 'what we deliver' (registered office + file corporation),
and banking copy province-aware (BC or Ontario) instead of BC-only.
- Verified headless: province card renders, H2 visible (not auto-collapsed),
13 accordions + proof expander intact, 28 Ontario mentions, no new JS errors.
Completes the MX-exclusion plan. Untagged carriers can't be excluded (the big-MX
gate is MX-based, so an unresolved Google/Yahoo domain would slip through), and
were previously UNCAPPED in select_sendable_carriers -- a flood of freshly-imported,
never-resolved domains could dominate a run before pw-mx-tag resolves them.
Added a single shared untagged_cap (env MAIN_UNTAGGED_MX_CAP, default max(quota,200))
so untagged sends are bounded without starving the pool: at the default the bucket
can still fill an entire run's quota (no behavior change today), but the cap can be
tightened to a fraction once pw-mx-tag has drained the backlog -- which is fast,
since only ~3,035 distinct *verified-sendable* untagged domains remain (< one
20k/day tag run). Tagged carriers keep their per-operator caps unchanged.
Verified: compiles; cap logic never starves at default, enforces the limit when
set lower.
Fix 1 (consumer mx: exclusion) and Fix 3 (pw-mx-tag cron) live as of 9eeed47.
Verified: warmup pool 353,909 after fix (not starved), mx:yahoodns.net cap=0
during warmup, cron tags idempotently. Fix 2 (NULL bucket cap) deferred.
Fix 1 (build_trucking_campaigns.py): the warmup big-MX exclusion only covered the
clean-label operators (google/microsoft/proofpoint/...). Consumer mailbox
operators that mx_tag_carriers.py labels with an "mx:" prefix slipped BOTH the
exclusion and the per-MX throttle -- notably mx:yahoodns.net (283k sendable
carriers = Yahoo Small Business/AOL custom domains) and mx:icloud.com (25k), plus
comcast/charter/centurylink/windstream/tds/earthlink. These are custom domains
whose MX points at a consumer provider, invisible to the literal-domain blocklist.
Added CONSUMER_MX_OPERATORS, folded into WARMUP_EXCLUDE_OPERATORS used by both the
fetch_carriers() exclusion SQL and mx_daily_caps() (same day-30 ramp). Behind the
existing MAIN_SKIP_BIG_MX switch.
Validated read-only: after the fix the warmup-eligible pool is 353,909 carriers
(315,892 untagged + ~38k genuinely small/self-hosted operators), so the long tail
still sustains the daily quota -- not starved -- while 0 consumer-MX carriers are
selected during warmup.
Fix 3 (infra/cron/pw-mx-tag): mx_tag_carriers.py was on no cron, so the untagged
(NULL) backlog (~316k) never drained and new FMCSA imports stayed untagged,
slowly re-opening the gap. Added a daily 05:45 UTC cron (--only-unsent
--limit-domains 20000), before the 08:00 builder. Idempotent/bounded (only tags
mx_provider IS NULL). Verified live: a 200-domain test run tagged 216 domains.
(Fix 2 -- bounding the NULL bucket cap -- deferred; the cron will drain it.)
Analysis-only plan (no code shipped). The trucking builder's warmup excludes
big receiving operators (Google/MS/Proofpoint/...) by mx_provider, but three
holes let throttling/consumer MX through during the day<=30 window:
1. Consumer operators tagged with the "mx:" prefix (mx:yahoodns.net = 283,113
sendable carriers, mx:icloud.com = 24,985, comcast/charter/centurylink/...)
are NOT in BIG_MX_OPERATORS, so they slip both the exclusion and the throttle.
These are custom domains whose MX points at Yahoo/iCloud -- invisible to the
literal-domain blocklist, only catchable via MX tagging. Biggest hole.
2. 315,892 untagged (NULL) sendable carriers are sent to unvetted (kept by design
for anti-starvation, but uncapped).
3. mx_tag_carriers.py is on no cron, so the NULL backlog never drains and new
FMCSA imports stay untagged -- slowly re-opening gaps 1 and 2.
Plan proposes: CONSUMER_MX_OPERATORS set folded into exclusion+throttle (behind
the existing MAIN_SKIP_BIG_MX switch), a bounded cap on the NULL bucket, and a
daily pw-mx-tag cron. Includes live numbers, validation steps (dry-run selector
diff, no sends), and open decisions (re-introduction ramp, permanent vs warmup-
only exclusion for Yahoo/iCloud custom domains).
Campaign 509 (CRTC USF Q3, 4,156 sent) shipped with raw <a href> URLs, so
Listmonk never registered the links and recorded ZERO clicks -- even though
Umami logged the real order-page visits AND a carrier phoned in after clicking.
Same mistake docs/fmcsa-trucking-plan.md already flagged ("Use @TrackLink on all
CTAs"); the trucking campaigns do it, the CRTC one didn't.
Listmonk only tracks a link when its href ends with the literal @TrackLink marker
(it strips it and rewrites through lists.performancewest.net/link/). Added a
_track() helper that appends UTM params (so Umami attributes the visit too) +
@TrackLink, applied to both the primary order CTA and the guide-PDF download.
The running campaign 509's body was also patched live in the DB (same two links)
so its remaining sends record clicks. Future CRTC campaigns get it from source.
First live ingest (28 reports) showed our warmup rotation pool (.91-.109, out0x)
mislabeled EXTERNAL because OUR_IPS only listed 4 specific IPs -- every one was
100% DMARC-passing, clearly ours, and would have generated false spoofing alerts.
Replace the literal-IP set with an ipaddress subnet check on 207.174.124.0/24
(our whole block). The only genuinely-external failing sender is 35.174.145.124
(AWS, 32 msgs spoofing us, SPF-fail/no-DKIM, all correctly rejected by p=reject) --
exactly the signal the --alert path is meant to surface.
Tool 2 of the deliverability monitoring pair (Tool 1 = mail_reputation_monitor).
DMARC rua reports from dozens of operators (Google, Yahoo, Comcast, Cox, Bell,
Mimecast, Cisco ESA, GMX, mail.com, ...) were landing in ops@ (dmarc@ was a DL),
burying real mail and never parsed. Now ingested + queryable:
- dmarc@performancewest.net converted DL -> dedicated Carbonio mailbox; isolated
IMAP creds in server .env, surfaced to workers in docker-compose.yml (mirrors
OPS_IMAP_*). 29 historical reports moved ops@ -> dmarc@ via IMAP.
- scripts/dmarc_report_parser.py: IMAP fetch unseen -> decompress .gz/.zip/.xml
(namespace-agnostic: classic + urn:ietf:params:xml:ns:dmarc-2.0 GMX/mail.com) ->
parse aggregate XML -> upsert dmarc_report (keyed (org_name,report_id), no-op on
re-parse) + dmarc_record per source IP. dmarc_pass = dkim_aligned OR spf_aligned.
Marks \Seen. --dry-run/--all/--alert (7d per-IP summary + Telegram if one of OUR
IPs <95% pass, or EXTERNAL IP sends >=20 failing msgs as us = spoofing under
p=reject). psycopg2 imported lazily so --dry-run runs without the driver.
- api/migrations/102_dmarc_aggregate.sql: dmarc_report + dmarc_record tables.
- infra/cron/pw-dmarc-parser: 06:20 UTC daily --alert (after reputation, before scrub).
- docs/deliverability.md: DMARC section DONE; query examples.
Verified: dry-run --all parses all 28 reports (1 non-report test probe), 0 unknown
after the namespace fix.
Runs mail_reputation_monitor --alert at 06:10 UTC, piping the day's postfix log
(sudo cat, same pattern as pw-warmup-tg-alert) into the DB-connected workers
container. Builds the daily SNDS-equivalent reputation trend and Telegram-alerts
on operator regressions. Installed to /etc/cron.d/pw-mail-reputation.
Adds scripts/mail_reputation_monitor.py + migration 101 (mail_reputation_daily).
Sender reputation is judged by the RECEIVING operator (Microsoft/Google/Yahoo/
Proofpoint), and the provider portals (SNDS/Postmaster/CFL) need a login and lag
24-48h. Our postfix logs already carry the ground truth in real time: every send
records the receiving host + SMTP response, and the response classifies WHY:
250 -> accepted
451 4.7.500 -> throttled (Microsoft rate-limiting a cold IP)
550 5.7.x -> reject_reputation (spam/reputation)
550 5.1.1/5.4.1-> reject_recipient (dead mailbox / access denied = list hygiene)
550 ...SPAM -> reject_content (SpamAssassin)
The parser classifies each egress delivery (out0x/hcout/relay) by (sending_ip,
receiver, outcome, reason_code) and upserts ONE daily aggregate row per bucket
(idempotent ON CONFLICT), so a nightly cron over the rotated log gives a queryable
trend without re-parse double-counting. --alert prints a per-operator summary and
Telegram-alerts on regressions (>=10% reputation rejects, or Microsoft >=70%
throttled). Reads stdin ("-") so the host-owned /var/log/mail.log can be piped
into the DB-connected workers container.
Motivation: 2026-06-19 audit found ~80% of Microsoft sends were getting 451 4.7.500
throttles on the warming IPs -- this makes that trend visible as reputation recovers.
performancewest.net + send.performancewest.net both show Enrolled in the Yahoo
Sender Hub, reporting email fbl@. All three FBLs (Google Postmaster, MS SNDS+JMRP,
Yahoo CFL) now complete.
Added yahoo-verification-key TXT records via Hestia for performancewest.net
(apex) and send.performancewest.net; both propagated to all HE.net slaves +
public resolvers. Ready to click Verify in the Yahoo CFL form, complaint dest fbl@.
SNDS access requested/granted for 207.174.124.94 + .107; JMRP feeds registered
with complaint dest fbl@. Section marked complete. SNDS data populates in ~24-48h.
Note JMRP delivers ARF complaints to the signed-in MS account's email, not
automatically to fbl@; set a forward if that account isn't fbl@performancewest.net.
