The DB can record a pdf_minio_path before the object is uploaded (e.g. a
prepared-filing path written for an order whose prep never completed -- Paul
Wilson / Mark Adams MCS-150s). The documents list now HEAD-checks each key and
returns an exists flag; the UI shows 'not generated yet' instead of a dead View
button, and the stream endpoint returns a clean 404 for a missing object.
Adds a Documents section to the compliance-order detail drawer so you can
review the actual filing PDFs before approving an order:
GET /api/v1/admin/compliance-orders/:id/documents list viewable objects
GET /api/v1/admin/compliance-orders/:id/document?key=&token= stream one
Key discovery pulls from esign_records (unsigned + signed docs per order),
intake_data.filing_status (pdf_minio_path, attested_pdf, evidence/*), and the
order's engagement_letter / rmd_packet columns.
Rather than hand out presigned URLs (MinIO's public host is IP-allowlisted to a
few office IPs, so links break elsewhere), the API streams the object through
itself from internal minio:9000, gated by the admin JWT. The stream endpoint
accepts the token via ?token= (new middleware requireAdminQueryOrHeader) so a
PDF opens in a new tab, and refuses any key that isn't one of the order's own
documents.
The shared security snippet blocked any path matching /(admin|administrator|
login.action|struts) with 'return 444', which drops the connection. That bare
'admin' token also matched our own operations dashboard at /admin and the new
/admin/compliance-orders, so the browser showed 'This site can't be reached'.
Dropped the bare 'admin' token; administrator/login.action/struts stay blocked.
Applied live on prod (sudo edit + nginx reload); this updates the source of
truth so the ansible nginx role won't reintroduce it.
The admin SPA only managed formation_orders; compliance service orders
(telecom/DOT/healthcare) had no admin surface, so you couldn't see what was
paid, what was stuck on intake, or approve a prepared filing for submission.
API (api/src/routes/admin.ts), all requireAdmin:
GET /api/v1/admin/compliance-orders list, grouped by batch, filters
GET /api/v1/admin/compliance-orders/stats queue overview counts
GET /api/v1/admin/compliance-orders/:id full detail + audit log
POST /api/v1/admin/compliance-orders/:id/approve approve ready_to_file + dispatch worker
POST /api/v1/admin/compliance-orders/:id/rearm-intake clear reminder stamp so daily nudge resumes
UI: new static page /admin/compliance-orders/ (self-contained, CSP-safe inline
CSS, no external JS framework) reusing the existing pw_admin_token session.
Cards group multi-service batches, flag paid+intake-incomplete in red, show
reminder counts, and expose Approve & Re-arm buttons. Linked from the main
/admin top bar. Every approve/re-arm writes an order_audit_log entry.
Companion to the worker MinIO-retry fix. Makes the worker auto-recover from
process death (crash, manual kill, missed boot trigger), not just MinIO outages.
- start_worker.bat: propagate Python's exit code (exit /b %rc%) so Task
Scheduler can actually detect a failed run (it previously always exited 0).
- reconfigure_task.ps1 (new): re-registers PW-DocserverWorker with
RestartCount=99 / 1-min interval, StartWhenAvailable, and two triggers —
AtStartup plus a 5-min repeating trigger with MultipleInstances=IgnoreNew, so
a dead worker relaunches within ~5 min and never double-runs. Idempotent.
- install.ps1: same self-healing settings for fresh installs.
- Verified on the box: killed the worker -> task relaunched it; firing again
while running stayed at one instance.
Docs updated to match reality:
- docserver/README.md: new 'Reliability / self-healing' section.
- document-generation.md: corrected the stale 'Flask DocServer :5050 / HTTP'
description to the actual MinIO outbound-only transport.
- e2e-test-plan.md: removed the outdated 'Word COM fails under SYSTEM / requires
RDP after every reboot' limitation; now self-healing under SYSTEM session 0.
- infrastructure.md: fixed VM spec (Win Server 2019, Word 16.0, Python 3.13,
SSH port 22422) + self-healing note.
- architecture.md / formation-system.md: trigger + self-healing details.
The worker called sys.exit(1) on any MinIO connection error, so a single
transient 502 from MinIO/its reverse proxy left it dead until a manual restart
or reboot (its scheduled task only runs at system startup). It had been dead
~5 weeks after a 502 on May 9.
- _connect_minio_forever(): retry the initial MinIO connect indefinitely with
capped exponential backoff (5s..120s) instead of exiting.
- main loop: wrap each poll cycle; on any error, log + rebuild the client and
keep polling rather than crashing.
Verified on the box: normal DOCX->PDF still works (~11s e2e); a bogus endpoint
now retries forever without ever calling sys.exit (was the exact May-9 failure).
Two of our three real paid customers (Mark Adams / mark@adamslumber.com and
Paul Wilson / synthetic@pipeline.com) never completed intake. They each hit the
old hard cap of 10 daily reminders (last sent Jun 12 / Jun 13) and the worker
then went permanently silent -- the last two daily runs reminded 0 orders even
though both still owe us intake on paid work. (The third, mitchell allen /
mitchell@allenscrapmetal.com, did complete intake; his orders are dispatched.)
Replace the dead-stop cap with a two-phase cadence:
- daily for the first DAILY_PHASE (10) nudges -- the initial burst,
- then weekly (WEEKLY_INTERVAL_DAYS) up to an absolute MAX_REMINDERS (60),
so a paid order with missing intake keeps getting a gentle nudge instead of
being dropped. Tunable via INTAKE_REMINDER_DAILY_PHASE /
INTAKE_REMINDER_WEEKLY_INTERVAL_DAYS / INTAKE_REMINDER_MAX. Clearing
intake_reminder_last_at re-arms an order immediately (documented in the
module docstring).
Main pool is calendar-day 12 but reputation is wrecked (54% delivery, Gmail+
Outlook blocks) -- NOT warmed. MX tagging confirmed the cause: 702k carriers on
Google + 135k on Microsoft = the warmup was hammering exactly the two operators
blocking us. Hold Google/MS/Proofpoint/etc. OUT entirely until day 30 (configurable),
sending only to the long-tail operators (yahoo/comcast/charter/centurylink/etc.)
that don't bot-throttle, so reputation can recover; then re-introduce big
operators gradually via mx_daily_caps. 1.24M/1.49M carriers now tagged.
The real bottleneck was the write, not DNS: each per-domain UPDATE full-scanned
fmcsa_carriers (no functional index on lower(split_part(email,'@',2))). Resolve
all domains concurrently into a list, load a temp table, then ONE join-UPDATE =
single table scan. Tags ~12k domains -> hundreds of thousands of carriers fast.
The serial path (verifier's 8s+6s lifetime per domain) was far too slow for
bulk tagging -- 0 tagged in 3 min on dead domains. Self-contained fast resolver
+ ThreadPoolExecutor(40) resolves thousands of domains in minutes.
The Jun 13-14 Gmail+Outlook block storm came from the main/trucking pool having
NO per-MX throttling (only HC had it) -- it concentrated warmup volume on
Google/Microsoft-Workspace-hosted business domains. Port the HC fix:
- migration 097: fmcsa_carriers.mx_provider column.
- mx_tag_carriers.py: resolve MX once per distinct domain (reuses the verifier's
classifier+cache), tag every carrier with that domain's operator. Bounded per
run, prioritizes unsent verified carriers.
- build_trucking_campaigns: during warmup (day<=6) EXCLUDE tagged Google/MS/
Proofpoint/etc. carriers in fetch_carriers; per-MX cap in select_sendable_
carriers so known operators never dominate the quota. Untagged carriers pass
(not collapsed onto one bucket) until tagging fills in. mx_daily_caps ramps
with the main warmup day; MAIN_SKIP_BIG_MX=0 disables once warmed.
The HC warmup crons were '* * 1-5' (Mon-Fri), silently skipping weekends -- but a
proper warmup needs CONTINUOUS daily volume for 21 days (mailbox providers reward
consistency; gaps stall reputation). The Jun 14 'HC 0 sent' alert was just a
skipped Sunday, but the weekend skips also broke ramp continuity.
- pw-hc-campaign + pw-hc-nppes: '* * 1-5' -> '* * *' (daily), vendored + applied live.
- Re-aligned the warmup start stamp from calendar-day 9 to send-day 5 so the
volume ramp matches reputation actually built (it had skipped ~4 weekend days,
running the ramp ahead of real history).
- Fixed the stale 'Mon-Fri only' comment in daily_slice().
- Vendored nppes cron now carries the enriched-CSV + 4-segment config.
otc_reincorporation.html: redomesticate-to-Texas hook (Business Court + TXSE +
DE franchise-tax cost) personalized by state_inc_name/company/ticker, cross-sell
RA/foreign-qual/annual-report/franchise-tax, same-day coupon, lead-capture CTA to
/contact?service=reincorporation (high-touch corporate service, not self-serve),
careful 'not a law firm / not legal advice' disclaimers + CAN-SPAM address.
build_otc_campaign.py: emails only verified-email issuers from the harvest+scrape
+verify pipeline, --de-nv-only for the best reincorp fit, reuses trucking sender
plumbing + coupon. Per-deal value is high so capped modestly (400/run default).
scrape_otc_emails.py: fetch each issuer domain's IR/contact pages (gzip,
HTML-only, early-abort, prefer ir@/investor@/info@), extract a contact email.
Skip filing-agent domains (DFN/Donnelley/Broadridge/etc.) that leak into the
extracted domain -- those are not the issuer's site. Same filter added to the
harvester's DOMAIN_NOISE for future runs. Phone (100%) is the fallback channel
for email misses.
Pilot -> production: harvest_otc_issuers.py pulls the OTC/None universe (2,771),
keeps US-domestic (requires BOTH a US state-of-incorporation AND a US-state
business address -- disambiguates the 'DE'=Delaware-vs-Germany trap that leaked
Infineon etc.), and extracts each issuer's website DOMAIN directly from its
latest 10-K/8-K/DEF-14A filing (free, no scrape; ~58-60% find rate in testing).
Outputs cik,name,ticker,state_inc,phone,city,state,zip,domain -- ready for the
domain->email scrape + verify step. Phone is 100% (clean fallback call channel).
Reincorporation-to-TX / RA / foreign-qual / franchise-tax / annual-report fit.
Ran the email-findability pilot we should have run for CLIA. SEC/OTC is viable:
~940 US-domestic OTC issuers, domain recoverable from the 10-K/8-K filing itself
at ~100% (free, no scrape), email via site scrape ~25-50%, phone 100%. High
per-deal value (reincorporation/RA/foreign-qual/franchise tax). Documented the
build plan.
The OIG-screening + NPPES-update segments were effectively limited to ~1,437
providers because the warmup 'any' selector excluded not-on-reval-list rows as a
deliverability proxy -- but that excludes almost the ENTIRE institutional list
(org NPIs aren't individual Medicare enrollees). Since we already SMTP-verified
all 63k inboxes, add an 'institutional_verified' selector that trusts our own
verification instead of reval-list presence. Result: OIG + NPPES-update now
address 62,422 (43x more), giving multiple broad offers to test engagement on.
- enrich_institutional_revalidation.py: fast local join of the institutional
list to the CMS Revalidation Due Date List bulk file (revalidation_base.csv)
by NPI -> adds reval_due_date/days_overdue/reval_status. ~1,437 are genuine
Medicare enrollees (197 overdue / 164 due-soon) -> flagship $599 reval pitch.
- npi_reactivation stays on leie_or_deactivated (only REAL deactivations -- no
false 'your NPI is deactivated' claims to active orgs).
Synthesize this session's findings into a ranking of candidate verticals by the
one thing that actually gates cold email: a reliable public bulk source of
deliverable emails. Tier 1 (email native): FCC, FMCSA. Tier 2 (one free hop):
healthcare ORG NPIs (already harvested 63k verified), SEC/OTC corporate. Tier 3
(domain-scrape): FMC OTI, state business entities by trigger. Tier 4 (phone/mail
only, NOT email): CLIA (0.3% match proven), EPA RCRA, individual NPIs.
UCR (Unified Carrier Registration) is annual: opens Oct 1, due Dec 31, mandatory
for interstate carriers (op A, same ~628k pool as IFTA) -> recurring revenue.
- build_ucr_annual_campaign.py: 3-touch business-day cadence (30/12/4 bd before
Dec 31, wider than IFTA since it's once a year), escalating tone, same-day
coupon, 'I already did it' suppression. Reuses build_trucking_campaigns +
IFTA business-day/token helpers (DRY). Per-year cycle reset.
- ucr_annual_reminder.html: deadline + fines/OOS risk + 'we figure out your fee
tier' + coupon + filed link + CAN-SPAM. Source campaign 473.
- migration 096: ucr_reminded_at / ucr_touch_no / ucr_self_filed_at.
- ifta.ts: add GET /api/v1/ucr/filed (shares the HMAC token scheme).
- checkout.ts: order-placement Telegram now shows 'Source: campaign (code X)'
when a discount code is present, so IFTA/UCR/CLIA conversions are visible.
(Confirmed order-alert Telegram already fires from handlePaymentComplete for
all compliance orders via both webhook + session paths.)
The handler is an API route; performancewest.net/api/* proxies to the Astro site
(:4322), only api.performancewest.net serves the API. Build the filed link from
PUBLIC_API_URL (default https://api.performancewest.net). Verified end-to-end:
valid token -> 200 confirmation page, invalid -> 403.
- Multi-touch reminders at 10/7/4 BUSINESS days before each deadline (weekends
skipped; biz-day math so a touch never lands purely on a weekend with no
runway). Escalating tone soft -> urgent -> last-chance, with the 'almost too
late to DIY, we can still file it' angle so it's a convenience sale, not a free
reminder service. ifta_touch_no tracks the highest touch sent so each touch
hits only carriers below that level; never repeats a touch.
- 'I already filed it' one-click link: HMAC-tokenized GET /api/v1/ifta/filed
(token matches between Python builder and api/src/routes/ifta.ts -- verified
identical output), records ifta_self_filed_at, friendly confirmation page,
stops further touches this cycle + gives DIY-vs-prospect signal. Builder
excludes self-filed carriers.
- migration 094 (ifta_touch_no) + 095 (ifta_self_filed_at); cycle reset clears
both each new quarter. Verified: biz-day touch schedule, token cross-match.
add_subscriber only attached existing subscribers to the new list without
updating attribs, so a carrier emailed in a prior campaign kept STALE attribs --
meaning the daily coupon code and IFTA merge fields (ifta_due_date, ifta_quarter,
lp_link) rendered BLANK for any repeat recipient. Now merge+PUT the fresh attribs
on the existing subscriber before attaching. Affects all trucking campaigns, not
just IFTA. Verified: IFTA preview now persists ifta_quarter/ifta_due_date/lp_link.
IFTA returns are due on fixed dates (Apr30/Jul31/Oct31/Jan31) and every
interstate carrier (op code A, ~628k sendable) files 4x/year forever -- pure
recurring revenue, no per-carrier deadline data needed.
- build_ifta_quarterly_campaign.py: self-gates to the reminder window (~21d
before each deadline), selects interstate carriers, mints the same-day coupon,
builds+schedules the campaign reusing build_trucking_campaigns plumbing (DRY:
one source of truth for sending/suppression/coupon). Per-quarter cycle reset
(ifta_reminder_cycle marker) so each quarter re-reminds the full pool; marks
ifta_reminded_at to avoid double-sends within a cycle.
- ifta_quarterly_reminder.html: deadline + penalties + 'we do the math' + coupon
+ CAN-SPAM. Listmonk source campaign id 469.
- migration 094: fmcsa_carriers.ifta_reminded_at column + partial index.
Verified: deadline/window logic correct, imports reuse tc helpers, migration
applied on prod.
Capture the full decision trail and chosen approach for making CLIA labs
emailable: why NPI->NPPES (0.3%) and DirectTrust failed, datacenter-IP search
blocking, the $99 B2B-list -> email-domain -> scrape-current-email -> verify
pipeline (durable domain even when the mailbox is stale), hard rules protecting
the warming mail pool, gzip/HTML-only bandwidth optimization, residential proxy
options, the sample-validation gate before committing, what's already built
(harvest, service, order page, email template), and the postcard fallback.
Set up the CLIA recurring-renewal vein (every clinical lab renews its CLIA cert
on a 2-year cycle; CMS publishes the full lab file with expiration dates):
- service-catalog: clia-renewal ($449, discountable) + order page (npi-intake
steps) + intake manifest entry.
- harvest_clia_renewals.py: parse the CMS Provider-of-Services CLIA file, filter
to labs expiring within a window (default 120d), emit name/address/phone/expiry.
676k labs -> ~70k expiring in the next ~4 months.
- match_clia_to_nppes.py: CLIA has no NPI/email, so bridge to emailable NPPES
orgs by normalized name+zip to recover NPI+email (yield TBD; labs that do not
match still have clean phone+postal for a phone/mail channel).
- hc_clia_renewal.html: warm turnover-safety-net email with the striped official-
record card (CLIA #, expiry, status), verify-on-CMS-QCOR, founder guarantee
card, full CAN-SPAM address.
- Upgrade the plain teal record banner to the authoritative barber-pole 'Official
record' banner in the personal/turnover/overdue-personal templates (the switch
to personal templates had dropped the striped look from live revalidation sends).
- nppes_outdated: replace plain info table with the striped 'Official record -
NPPES NPI Registry' card (status honestly labeled as our compliance flag).
- Wire revalidation_overdue -> hc_revalidation_overdue_personal.html with a direct
past-due subject ('Your Medicare revalidation is past due - let's get it filed')
and PAST DUE status + days-overdue in the record card; due_soon stays warm.
- Striped card now on all 7 templates that show a real record; oig_screening and
compliance_bundle correctly omit it (no specific record to display).
Match the authoritative 'official record' look of the revalidation emails on the
deactivated/NPI-reactivation template: striped banner + structured NPPES record
card. Kept it accurate -- NPI/name are labeled NPPES (the real public source);
the deactivation status is labeled as our compliance flag (not an NPPES field),
since deactivation is not a single public dataset, with a 'confirm via official
sources' footnote.
Per your call: add the same personal founder card (headshot linked to /about,
service-neutral satisfaction-guarantee quote, signature, title) to the four
remaining HC templates for a consistent trust signal across all healthcare
outreach. Kept the factually-direct subjects where the situation IS past-due/
deactivated (npi_reactivation) -- only the framing softens, not the facts.
All HC templates now use the v2 signature.
CAN-SPAM requires a valid physical postal address in every commercial email.
All 8 HC campaign templates and the FCC campaign_template.html only had
'Cheyenne, WY' (no street) -- added the full
'525 Randall Ave Ste 100-1195, Cheyenne, WY 82001' to match the (already-correct)
trucking templates. Audited every Listmonk source/sent campaign + wrapper
templates: all active sends carry address + unsubscribe.
Also: revalidation segments now use hc_revalidation_personal.html with subject
'Let's make sure your Medicare revalidation is handled in time'.
Adds hc_revalidation_personal.html: the turnover safety-net email plus a
personal guarantee card from Justin Hannah -- round headshot (links to /about so
readers can confirm a real person stands behind it), an italic satisfaction-
guarantee quote ('I will personally make it right... that is my promise'), a
rendered 'Justin Hannah' signature (Dancing Script, SIL OFL), and his title
(Founder & Principal Consultant). Signature image generated via PIL and added to
site/public/images/justin-signature.png. Test sent to justin@.
New HC template (hc_revalidation_turnover.html) that gets the warm, 'someone who
has our back' feel WITHOUT falsely claiming a prior business relationship (which
would be a deceptive practice under FTC/UDAP and is especially risky with
compliance-minded healthcare admins). Instead it leans on:
- the real staff-turnover insight ('whoever last handled this may have moved on')
- genuine relevance (their actual NPI + CMS revalidation due date)
- the safety-net positioning ('we keep an eye on this so it does not become your
problem' / 'we will make sure it gets done right no matter who handled it')
- true social proof (trusted by providers nationwide) + verify-on-CMS.gov
Every claim is true and defensible. Test sent to justin@.
The sales we got came at $79 + a 24hr coupon; cutting MCS-150 to $39 flat
removed urgency and conversions did NOT improve (a permanent low price sets a
new anchor and lets people defer). Restore the higher anchor and let an
expiring discount create the now-or-lose-it decision.
- Restore MCS-150 anchor $39 -> $79 (catalog single source + regenerated).
- build_trucking_campaigns.py: mint ONE random 5-letter coupon per send-day
(40% off, valid through 23:59:59 ET that day) into the existing discount_codes
table; inject coupon_code/pct/expires + a ?code= LP link into every email.
Idempotent per day; service-fee-only scope (gov/pass-through fees never cut).
- Listmonk MCS-150 (186) + Inactive USDOT (188) templates: lead with the
struck-through anchor + sale price + code + 'expires tonight', and point the
primary CTA at the order page (with code) instead of the 'free check' tool.
- OrderPriceBanner: validates ?code= via /api/v1/discount and shows
was/now + expiry; Wizard forwards the code to order creation.
- Verified: code gen, expiry math, scope enforcement, discount API
(40% off $79 = $47.40), site+api builds clean.
Day 9 (2026-06-13) alert: main pool 54% delivery, 202 Gmail spam-blocks
(550-5.7.1 'Gmail has detected') on warming IPs .94-.98. The 4k/day (400/h)
ramp was too aggressive AND the trucking pool lacks the per-MX throttling the HC
pool got -- Google-Workspace-hosted business domains (weberfarms.net, uatruck.com,
etc.) concentrated and Gmail blocked us. Held at 200/h (~2k/day) through day 20 to
recover, then slow step to 300/h. Applied live (cap already set to 200/h).
Adds /etc/cron.d/pw-hc-nppes (weekdays 07:30) that imports the verified NPPES
institutional general-compliance base into the OIG screening segment, throttled
per MX operator. Separate from the 07:00 reval-segment run so the two pipelines
stay independent. Vendored the cron file under infra/cron/.
Reputation is tracked per receiving mail operator, not per recipient domain, so
the daily warmup slice is now distributed across MX operators with per-operator
daily caps (ramping with the warmup day): Microsoft/Google/Proofpoint/etc. capped
individually, long-tail operators each get a generous default. This lets total
daily volume be much higher than a flat cap without hammering any single system.
mx_throttled() respects the mx_provider column the verifier now writes; falls back
to flat slicing if absent.
The saved Goal + Funnel reports used a pre-v3.1.0 parameters shape (urls/dateRange)
that umami 3.1.0's report schema rejects (400 -> 'Something went wrong'). Migrated
both reports in the umami DB to the current funnelReportSchema/goalReportSchema
(steps[], top-level dates, goal type/value). Verified funnel route now passes schema
validation. TODO updated/closed.
Reputation is tracked per receiving mail operator (Microsoft 365, Google
Workspace, Proofpoint, etc.), not per recipient domain -- so warmup can safely
send far more total volume if it's spread across many MX operators and throttled
per-operator. The verifier now classifies each domain's (already-cached) MX into
a provider label and writes an mx_provider column, so the warmup importer can
cap sends per operator per day. NPPES institutional sample distribution:
Microsoft 33%, Google 11%, Proofpoint ~16%, long tail across dozens of others.
Extracts cold-mailable provider inboxes from the NPPES endpoint_pfile, dropping
Direct/HISP gateway domains (not deliverable from a normal MTA). From the
June 2026 NPPES file: 88,728 institutional + 19,355 consumer mailable
candidates. Institutional is the warmup-safe slice (consumer webmail is held
back -- aggressive filtering would hurt the warming IP).
