Commit graph

799 commits

Author SHA1 Message Date
justin
147657d82d fix(docker): COPY SC COC Form.pdf into workers image
The Dockerfile copies form PDFs explicitly by name; the SC COC template was
missing, so fill_sc_coc() would FileNotFoundError in the container. Added it.
2026-06-16 09:23:43 -05:00
justin
c46efe5730 feat(sc-coc): SC intrastate Certificate of Compliance flow (insurance gate -> $25 fee -> file)
Routes SC intrastate-authority orders to the real SCDMV COC product instead of a
PSC certificate (which doesn't apply to property carriers):

  - sc_coc_filing.py: emails the carrier a one-click yes/no — does your insurer
    have / can they file a Form E (SC intrastate liability, $750k or $300k by
    GVWR) with SCDMV? Records the answer; builds the filled COC package.
  - state_trucking._handle_sc_coc_gate: SC intrastate gate —
      no answer  -> email the question once, HOLD
      answered no -> broker referral opened, HOLD (ops todo)
      answered yes-> proceed to bill the exact $25 SCDMV COC fee (at cost) + file
  - API POST /compliance-orders/:id/sc-insurance: records yes/no in intake_data
    (no schema change); NO opens an insurance_lead broker-referral ticket +
    Telegram; YES re-dispatches the worker to bill the $25 + file.
  - site/order/sc-insurance: customer one-click yes/no page (auto-submits when
    the email links straight to ?have=yes|no).

Non-SC intrastate still uses the PSC/PUC email path or a manual todo.
2026-06-16 09:15:55 -05:00
justin
dae9603808 fix(erpnext): remove default 'BC' from Sales Order incorporation_province
The custom_incorporation_province field had default='BC', which stamped 'BC'
on EVERY Sales Order (US trucking, formation, compliance) — not just Canadian
CRTC orders. This leaked a meaningless 'BC' onto e.g. an SC scrap-metal carrier's
order. Removed the default and added a blank option so it's empty unless it's an
actual Canadian incorporation. Existing non-canada_crtc orders cleared in prod
via db_set (13 fixed; the 2 real canada_crtc orders keep BC).
2026-06-16 09:12:49 -05:00
justin
ad590aab7c feat(sc-coc): SCDMV Certificate of Compliance PDF filler + correct $25 state fee
SC for-hire PROPERTY carriers (not passenger/HHG/hazwaste) register intrastate
via the SCDMV Certificate of Compliance (COC), not a PSC certificate. This adds:
  - sc_coc_pdf_filler.fill_sc_coc(): fills the official SCDMV Form COC from
    intake (business name, officers, physical/mailing address, phone), picks
    New vs Renewal, and stamps the coverage class (E-L low-value / E-LC).
    Field names in the source PDF are auto-generated + offset from their labels;
    mapped here by verified on-page geometry. Verified by render.
  - suggest_coverage_class(): E-L for low-value cargo (scrap/dump/aggregate),
    else E-LC (safer default).
  - gov_fee: SC intrastate fee corrected from $0 placeholder to the real $25
    COC new-application fee (renewals $0), billed at cost.

The carrier's INSURER files the Form E (liability) + Form H (cargo, E-LC only)
directly with SCDMV; we collect the COC app + $25 and submit it.
2026-06-16 09:08:50 -05:00
justin
01b3e1d234 chore(env): scaffold ISA_SC_DMS_USER/PASS for SC PSC MyDMS e-file portal
Non-attorney 'Service' filer account registered under Performance West
(filings@performancewest.net). Credentials live only in the server .env
(blank default in template, never committed). Consumed by the upcoming SC
intrastate Playwright e-filer.
2026-06-16 08:19:17 -05:00
justin
c27cfd3242 docs(crons): note IRP invoice poller now also handles intrastate [PW-ISA] replies 2026-06-16 07:59:38 -05:00
justin
b125d46663 feat(intrastate): automate state PUC/PSC authority filing (email + invoice + auto-bill)
Intrastate operating authority is state-specific + application-based like IRP, so
it reuses the same email/POA + invoice-reconciliation flow:
  - intrastate_filing.send_intrastate_submission: emails the state PSC/PUC the
    authority application with the signed POA attached (subject tag [PW-ISA CO-..]),
    reusing irp_filing's MinIO download + census enrich helpers.
  - The shared poller (irp_invoice_poller) now matches BOTH [PW-IRP] and [PW-ISA]
    tags, parses the fee, Telegram-alerts, and bills the customer the exact amount
    with the correct service slug.
  - state_trucking gov-fee gate routes intrastate-authority to the PSC/PUC email
    path; if no submission email is configured for the base state it falls back
    to a manual todo (safe default — no emailing guessed agency addresses).

Per-state ISA_<ST>_EMAIL env (blank until the exact agency address is verified).
SC/GA/TX scaffolded. Customer still only sees an exact-fee payment link; you only
approve the final filing.
2026-06-16 07:57:57 -05:00
justin
42b433db5a deploy: reset generated site files before pull (fixes silently-stranded commits)
deploy.sh ran sync_nav.py / gen-service-catalog.py which dirty site/public +
site/src in place; that made 'git pull' abort, so recent commits never reached
prod until pulled manually. Reset those generated paths before pulling so deploys
always fast-forward. Also document the IRP POA signer-name/title follow-up.
2026-06-16 05:28:45 -05:00
justin
a74516a255 irp: attach signed POA + census-enrich address; fix date JSON crash
- send_irp_submission now REQUIRES and ATTACHES the signed Power of Attorney PDF
  (downloaded from MinIO) — the state won't act on a third-party filing without
  it, and 'on file, available on request' stalls the request. If the POA isn't
  available we don't email and fall back to a manual todo.
- Backfill missing legal_name + registered address from the FMCSA census so the
  submission isn't sent with a blank address (root cause of the empty
  'Legal/registered address: , ,' line). Customer-supplied values win.
- state_trucking passes signed_auth_key through to the IRP submitter.
- Fix 'Object of type date is not JSON serializable' when creating the admin
  todo (json.dumps(..., default=str)) — broke the intrastate (bash-fee) path.
2026-06-16 05:18:23 -05:00
justin
1d6693adb9 govfee: itemize the estimate in the email + add a 'fix my fee' dispute path
The gov-fee email now lists exactly what the amount covers (full breakdown) so
the customer can check it for accuracy, with two clear actions: a  pay link and
a  'something looks wrong' link to /order/dispute.

New /order/dispute page shows the fee breakdown and lets the customer describe
what's wrong; it opens an 'issue' support ticket pre-tagged with the order
(amount + label + their note) via /api/v1/tickets, so ops corrects the fee
before any payment is taken. The /order/pay page also shows the itemized
breakdown and a dispute link.
2026-06-16 05:00:31 -05:00
justin
ea695d6828 feat(govfee): exact fees + agency processing fees; IRP email/invoice reconciliation
- gov_fee: add AGENCY_PROCESSING_FEE (per-service card/convenience fee passed
  through so the customer pays the true all-in cost); estimate_gov_fee now folds
  it into the billed total. IFTA/intrastate/UCR fees are published/near-exact.

- IRP fees can't be looked up — only the base state computes them. New
  irp_filing.py: emails the base-state IRP unit a Schedule A/B request (Reply-To
  the IRP filings mailbox, [PW-IRP CO-...] subject tag), and a 15-min cron
  (irp_invoice_poller) scans the mailbox for the state's invoice reply, parses
  the exact apportioned fee, Telegram-alerts you, and bills the customer the
  EXACT amount via a gov-fee child order + payment link. Then it proceeds to
  ready_to_file for your final approval.

- state_trucking gov-fee gate now routes IRP to the email/invoice path and
  IFTA/intrastate to immediate exact-fee billing.

- Mailbox is configurable (IRP_FILINGS_IMAP_* in app.env.j2); falls back to
  OPS_IMAP_* filtered by the [PW-IRP] tag until a dedicated mailbox exists.

Telegram alerts fire on IRP submission sent, invoice received (billed), and
un-parseable replies (so you can read + enter the fee manually).
2026-06-16 04:58:14 -05:00
justin
861f2fbfd4 feat(govfee): auto-quote + collect state fees for at-cost trucking services
At-cost services (IRP/IFTA/intrastate) only collected our service fee at
checkout; the variable state fee was never billed, so orders stalled at
authorization_signed and the filing card would have had to front large IRP fees.

New end-to-end, hands-off flow (you only approve the final filing):
  1. After authorization is signed, state_trucking auto-estimates the gov fee
     from intake (base/op states, power units, weight) via gov_fee.estimate_gov_fee.
  2. Creates a CHILD compliance order (CG-..., service_fee=0, gov_fee=estimate,
     parent_order_number set, migration 099) that flows through the EXISTING
     checkout/payment/webhook machinery.
  3. Emails the customer a payment link to /order/pay (new self-contained page)
     showing every method with correct surcharges — ACH 0% (Stripe 0.8%/ cap
     absorbed, no GoCardless needed), card/PayPal 3%, Klarna 6%, crypto 0%.
  4. Order holds at awaiting_government_fee_approval until paid.
  5. On payment, handlePaymentComplete detects the child (parent_order_number)
     and re-dispatches the PARENT with gov_fee_paid=true, which proceeds to
     prepare + queue the filing and stops at ready_to_file for your approval.

IRP fees are estimates billed at cost (refund overage / rebill shortfall); IFTA
decals + most intrastate fees are near-exact. Tunable via env.
2026-06-16 04:35:45 -05:00
justin
3e13b722f6 fix(relay): logging.getenv -> crash on import (card loading was broken)
relay_integration.py line 34 called logging.getenv (no such attr), which threw
AttributeError on import -> load_card_from_erpnext() crashed for every caller
(BOC-3 and now UCR filing payment). Drop the bogus line; LOG is set correctly on
the next line. Present since the initial commit.
2026-06-16 03:30:40 -05:00
justin
aadf9f5bc1 feat(ucr): Playwright auto-filing for UCR registration on approval
Adds scripts/workers/services/ucr_playwright.py — a UCR.gov National Registration
System automation that, given a USDOT + fleet size, runs the register/pay flow,
pays the federal UCR fee with the matched PW filing card (Relay/Stripe Issuing),
and captures a confirmation screenshot + number. Conventions match
boc3_playwright / fmcsa_web_submitter: dev-mode dry-run guard, undetected
(patchright) browser, CAPTCHA detection, screenshot evidence, dataclass result.

Safety: verifies the displayed fee against the federal schedule before paying and
refuses to auto-charge a surprising amount (UCR_MAX_AUTO_FEE_USD) — falls back to
manual filing instead.

Wires it into MCS150UpdateHandler: when an approved (admin_approved) order has
slug ucr-registration, _file_ucr_registration runs the automation, uploads the
confirmation screenshot to MinIO, records filing_status + confirmation, and sets
fulfillment_status=completed on success. On CAPTCHA / fee-mismatch / failure it
reverts to ready_to_file with a high-priority 'file manually' todo. This replaces
the old behavior where approving a UCR just sat at authorization_signed.
2026-06-16 03:29:05 -05:00
justin
bf69960e8c admin: mark-filed action to advance manual/admin-assisted orders to completed
Admin-assisted services (UCR, MC authority, etc.) have no automated submission,
so approving them only flips to authorization_signed and then sits there -- there
was no way to advance to completed. Add POST /mark-filed (filed_waiting_state |
completed, optional confirmation #, transactional + audit-logged) and drawer
buttons 'Mark as filed (waiting on agency)' / 'Mark completed' shown for orders in
authorization_signed / ready_to_file / filed_waiting_state. Confirmation number
is recorded into intake_data.filing_status.manual_confirmation.
2026-06-16 03:12:57 -05:00
justin
6c10c6a6cd mcs150 handler: service-aware todos/notifications/emails (stop mislabeling UCR as MCS-150)
UCR (and other admin-assisted DOT services) route through MCS150UpdateHandler,
which hardcoded 'MCS-150' and self.SERVICE_SLUG in the admin todo, the Telegram
fulfillment notification, and the customer status email -- so approving Paul's
UCR produced an 'MCS-150 Review / mcs150-update / PDF: not generated' alert and
an 'MCS-150 biennial update' customer email, both wrong.

Add SERVICE_DISPLAY_NAMES + _service_label(slug); use the actual slug everywhere.
Admin-assisted services now show 'UCR Annual Registration — FILE NOW ... file
manually on the portal (no auto-generated form)' instead of MCS-150/PDF wording,
and the customer email names the right service.
2026-06-16 03:02:53 -05:00
justin
326aee7714 admin: inline filing screenshots + atomic approve transaction
- Documents now flag is_image and the drawer renders screenshots / confirmation
  images as inline clickable thumbnails (click to open full size); PDFs keep the
  View link. Evidence keys are labeled (Filing confirmation screenshot, etc.),
  the worker-temp screenshot_path (not a MinIO key) is dropped in favor of the
  durable evidence copy, and non-file evidence (fax_log_id) is skipped.
- Wrap approve's status-update + audit-insert in a transaction so a failure can
  no longer leave an order out of ready_to_file without dispatching (the earlier
  audit CHECK violation did exactly that to Paul's UCR; it has been reset).
2026-06-16 02:57:24 -05:00
justin
73c27c75b1 migration 098: allow compliance order types in order_audit_log
The admin compliance-orders approve/re-arm actions write order_audit_log rows
with order_type='compliance', but the CHECK constraint (from migration 004)
only allowed formation/service/quote -- so every approve failed with a 500
('Approve failed.'). Expand the constraint to include compliance + compliance_batch.
2026-06-16 02:49:46 -05:00
justin
3df3a08221 mcs150 handler: derive admin-assisted intake from census; gate ready_to_file
Admin-assisted DOT services (UCR, BOC-3) routed to this handler were marked
ready_to_file with whatever intake existed -- e.g. a UCR with only a DOT number,
missing legal name / state / fleet-size bracket (which sets the UCR fee tier).
That made the admin 'ready to file' status dishonest and unfileable.

Now, for ADMIN_ASSISTED_REQUIRED services we first enrich intake from the FMCSA
census (legal_name, address_state, power_units) + the order email, and derive
the UCR fleet_size_bracket from power units (UCR_FLEET_BRACKETS). If every
required field is then present we persist it and mark intake validated (falls
through to the admin review gate -> ready_to_file). If anything is still
missing, we persist what we have, set fulfillment_status=awaiting_intake, and
email the customer to complete intake -- instead of falsely showing ready_to_file.
2026-06-16 02:46:10 -05:00
justin
8e1e2f16bf admin docs: only list objects that actually exist (drop dead/phantom rows)
Filter the documents list to objects that exist in storage, so stray keys (a
template pdf_minio_path, or a phantom mcs150 esign_records row on a UCR order
from the shared remediation pipeline) no longer surface as dead rows. The UI
drops the now-unreachable 'not generated yet' branch.
2026-06-16 02:37:33 -05:00
justin
c8e0065729 admin docs: hide phantom prepared-filing PDF for non-form services
The dot-compliance-remediation pipeline seeds filing_status.pdf_minio_path on
every order in a batch, but only MCS-150-producing slugs (mcs150-update,
dot-registration, usdot-reactivation, dot-full-compliance) ever generate it.
For admin-assisted services like UCR it was a phantom 'Prepared filing PDF /
not generated yet' row. Gate the prepared-filing artifacts on FORM_PRODUCING_SLUGS
(mirrors the worker's MCS150_FORM_SLUGS) and give the empty state a clearer
explanation.
2026-06-16 02:35:29 -05:00
justin
d18de006d8 admin approve: block filing when intake incomplete (force override + warning)
Paul Wilson's UCR (CO-FE07212A) sat at fulfillment_status=ready_to_file with
intake_data_validated=false, so the Approve & File button would have dispatched
it for government submission with incomplete intake and no document to review.

Backend: /approve now refuses an order whose intake_data_validated is false
unless {force:true} is passed (409 code=intake_incomplete); the override is
recorded in order_audit_log. The fulfillment_status=ready_to_file requirement
is unchanged, so awaiting_intake orders (e.g. Mitchell's MCS-150s) still 409.

UI: the drawer shows an amber 'intake not complete' warning above the approve
button, and approving an intake-incomplete order triggers an explicit
override confirmation before sending force=true.
2026-06-16 00:33:22 -05:00
justin
aa498fdfdf admin docs: probe existence with ranged GET (HEAD fails presigned-URL sig) 2026-06-16 00:23:40 -05:00
justin
1f3b36b29e admin docs: verify object existence, mark dead links, cleaner 404
The DB can record a pdf_minio_path before the object is uploaded (e.g. a
prepared-filing path written for an order whose prep never completed -- Paul
Wilson / Mark Adams MCS-150s). The documents list now HEAD-checks each key and
returns an exists flag; the UI shows 'not generated yet' instead of a dead View
button, and the stream endpoint returns a clean 404 for a missing object.
2026-06-16 00:22:35 -05:00
justin
bce5db4a09 admin: view order PDFs from MinIO (signed forms, prepared filings, evidence)
Adds a Documents section to the compliance-order detail drawer so you can
review the actual filing PDFs before approving an order:
  GET /api/v1/admin/compliance-orders/:id/documents  list viewable objects
  GET /api/v1/admin/compliance-orders/:id/document?key=&token=  stream one

Key discovery pulls from esign_records (unsigned + signed docs per order),
intake_data.filing_status (pdf_minio_path, attested_pdf, evidence/*), and the
order's engagement_letter / rmd_packet columns.

Rather than hand out presigned URLs (MinIO's public host is IP-allowlisted to a
few office IPs, so links break elsewhere), the API streams the object through
itself from internal minio:9000, gated by the admin JWT. The stream endpoint
accepts the token via ?token= (new middleware requireAdminQueryOrHeader) so a
PDF opens in a new tab, and refuses any key that isn't one of the order's own
documents.
2026-06-16 00:20:15 -05:00
justin
d65f5ea279 nginx: stop blocking /admin (bot-scan rule matched our own dashboard)
The shared security snippet blocked any path matching /(admin|administrator|
login.action|struts) with 'return 444', which drops the connection. That bare
'admin' token also matched our own operations dashboard at /admin and the new
/admin/compliance-orders, so the browser showed 'This site can't be reached'.
Dropped the bare 'admin' token; administrator/login.action/struts stay blocked.
Applied live on prod (sudo edit + nginx reload); this updates the source of
truth so the ansible nginx role won't reintroduce it.
2026-06-16 00:05:54 -05:00
justin
48fab25840 docs: document the admin compliance-orders surface in the runbook 2026-06-16 00:00:12 -05:00
justin
2296566e85 admin: compliance-orders dashboard (view, approve-to-file, re-arm intake)
The admin SPA only managed formation_orders; compliance service orders
(telecom/DOT/healthcare) had no admin surface, so you couldn't see what was
paid, what was stuck on intake, or approve a prepared filing for submission.

API (api/src/routes/admin.ts), all requireAdmin:
  GET  /api/v1/admin/compliance-orders            list, grouped by batch, filters
  GET  /api/v1/admin/compliance-orders/stats      queue overview counts
  GET  /api/v1/admin/compliance-orders/:id        full detail + audit log
  POST /api/v1/admin/compliance-orders/:id/approve       approve ready_to_file + dispatch worker
  POST /api/v1/admin/compliance-orders/:id/rearm-intake  clear reminder stamp so daily nudge resumes

UI: new static page /admin/compliance-orders/ (self-contained, CSP-safe inline
CSS, no external JS framework) reusing the existing pw_admin_token session.
Cards group multi-service batches, flag paid+intake-incomplete in red, show
reminder counts, and expose Approve & Re-arm buttons. Linked from the main
/admin top bar. Every approve/re-arm writes an order_audit_log entry.
2026-06-15 23:57:05 -05:00
justin
b48d0cb799 docserver: self-healing Task Scheduler config + docs
Companion to the worker MinIO-retry fix. Makes the worker auto-recover from
process death (crash, manual kill, missed boot trigger), not just MinIO outages.

- start_worker.bat: propagate Python's exit code (exit /b %rc%) so Task
  Scheduler can actually detect a failed run (it previously always exited 0).
- reconfigure_task.ps1 (new): re-registers PW-DocserverWorker with
  RestartCount=99 / 1-min interval, StartWhenAvailable, and two triggers —
  AtStartup plus a 5-min repeating trigger with MultipleInstances=IgnoreNew, so
  a dead worker relaunches within ~5 min and never double-runs. Idempotent.
- install.ps1: same self-healing settings for fresh installs.
- Verified on the box: killed the worker -> task relaunched it; firing again
  while running stayed at one instance.

Docs updated to match reality:
- docserver/README.md: new 'Reliability / self-healing' section.
- document-generation.md: corrected the stale 'Flask DocServer :5050 / HTTP'
  description to the actual MinIO outbound-only transport.
- e2e-test-plan.md: removed the outdated 'Word COM fails under SYSTEM / requires
  RDP after every reboot' limitation; now self-healing under SYSTEM session 0.
- infrastructure.md: fixed VM spec (Win Server 2019, Word 16.0, Python 3.13,
  SSH port 22422) + self-healing note.
- architecture.md / formation-system.md: trigger + self-healing details.
2026-06-15 22:49:21 -05:00
justin
7929413eeb docserver: survive MinIO outages instead of exiting
The worker called sys.exit(1) on any MinIO connection error, so a single
transient 502 from MinIO/its reverse proxy left it dead until a manual restart
or reboot (its scheduled task only runs at system startup). It had been dead
~5 weeks after a 502 on May 9.

- _connect_minio_forever(): retry the initial MinIO connect indefinitely with
  capped exponential backoff (5s..120s) instead of exiting.
- main loop: wrap each poll cycle; on any error, log + rebuild the client and
  keep polling rather than crashing.

Verified on the box: normal DOCX->PDF still works (~11s e2e); a bogus endpoint
now retries forever without ever calling sys.exit (was the exact May-9 failure).
2026-06-15 22:40:27 -05:00
justin
ef3b7a96f0 intake-reminder: weekly fallback so capped paid orders aren't abandoned
Two of our three real paid customers (Mark Adams / mark@adamslumber.com and
Paul Wilson / synthetic@pipeline.com) never completed intake. They each hit the
old hard cap of 10 daily reminders (last sent Jun 12 / Jun 13) and the worker
then went permanently silent -- the last two daily runs reminded 0 orders even
though both still owe us intake on paid work. (The third, mitchell allen /
mitchell@allenscrapmetal.com, did complete intake; his orders are dispatched.)

Replace the dead-stop cap with a two-phase cadence:
  - daily for the first DAILY_PHASE (10) nudges -- the initial burst,
  - then weekly (WEEKLY_INTERVAL_DAYS) up to an absolute MAX_REMINDERS (60),
so a paid order with missing intake keeps getting a gentle nudge instead of
being dropped. Tunable via INTAKE_REMINDER_DAILY_PHASE /
INTAKE_REMINDER_WEEKLY_INTERVAL_DAYS / INTAKE_REMINDER_MAX. Clearing
intake_reminder_last_at re-arms an order immediately (documented in the
module docstring).
2026-06-15 22:13:27 -05:00
justin
ba6f171c9d docs: record trucking per-MX throttling fix + the 702k-google smoking gun 2026-06-14 21:39:14 -05:00
justin
b9b963f87b trucking: extend big-operator exclusion to day 30 (reputation recovery)
Main pool is calendar-day 12 but reputation is wrecked (54% delivery, Gmail+
Outlook blocks) -- NOT warmed. MX tagging confirmed the cause: 702k carriers on
Google + 135k on Microsoft = the warmup was hammering exactly the two operators
blocking us. Hold Google/MS/Proofpoint/etc. OUT entirely until day 30 (configurable),
sending only to the long-tail operators (yahoo/comcast/charter/centurylink/etc.)
that don't bot-throttle, so reputation can recover; then re-introduce big
operators gradually via mx_daily_caps. 1.24M/1.49M carriers now tagged.
2026-06-14 21:35:58 -05:00
justin
3fbfcbfaab mx_tag: bulk UPDATE via temp-table join (per-domain UPDATE full-scanned 1.49M rows each time)
The real bottleneck was the write, not DNS: each per-domain UPDATE full-scanned
fmcsa_carriers (no functional index on lower(split_part(email,'@',2))). Resolve
all domains concurrently into a list, load a temp table, then ONE join-UPDATE =
single table scan. Tags ~12k domains -> hundreds of thousands of carriers fast.
2026-06-14 21:29:52 -05:00
justin
fd1522abee mx_tag: per-call Resolver (dns Resolver not thread-safe -- was deadlocking) 2026-06-14 21:26:38 -05:00
justin
5fd187a001 mx_tag: incremental write via as_completed (no straggler hold-up) 2026-06-14 21:24:46 -05:00
justin
60e6dc5d19 mx_tag: concurrent DNS resolution (40 workers, 3s timeout) for bulk speed
The serial path (verifier's 8s+6s lifetime per domain) was far too slow for
bulk tagging -- 0 tagged in 3 min on dead domains. Self-contained fast resolver
+ ThreadPoolExecutor(40) resolves thousands of domains in minutes.
2026-06-14 21:16:17 -05:00
justin
9e40965092 trucking: per-MX-operator throttling + Google/MS-Workspace warmup exclusion
The Jun 13-14 Gmail+Outlook block storm came from the main/trucking pool having
NO per-MX throttling (only HC had it) -- it concentrated warmup volume on
Google/Microsoft-Workspace-hosted business domains. Port the HC fix:

- migration 097: fmcsa_carriers.mx_provider column.
- mx_tag_carriers.py: resolve MX once per distinct domain (reuses the verifier's
  classifier+cache), tag every carrier with that domain's operator. Bounded per
  run, prioritizes unsent verified carriers.
- build_trucking_campaigns: during warmup (day<=6) EXCLUDE tagged Google/MS/
  Proofpoint/etc. carriers in fetch_carriers; per-MX cap in select_sendable_
  carriers so known operators never dominate the quota. Untagged carriers pass
  (not collapsed onto one bucket) until tagging fills in. mx_daily_caps ramps
  with the main warmup day; MAIN_SKIP_BIG_MX=0 disables once warmed.
2026-06-14 21:11:23 -05:00
justin
2caab6aa69 hc: warmup must run DAILY for the full 21-day ramp (not weekdays-only)
The HC warmup crons were '* * 1-5' (Mon-Fri), silently skipping weekends -- but a
proper warmup needs CONTINUOUS daily volume for 21 days (mailbox providers reward
consistency; gaps stall reputation). The Jun 14 'HC 0 sent' alert was just a
skipped Sunday, but the weekend skips also broke ramp continuity.

- pw-hc-campaign + pw-hc-nppes: '* * 1-5' -> '* * *' (daily), vendored + applied live.
- Re-aligned the warmup start stamp from calendar-day 9 to send-day 5 so the
  volume ramp matches reputation actually built (it had skipped ~4 weekend days,
  running the ramp ahead of real history).
- Fixed the stale 'Mon-Fri only' comment in daily_slice().
- Vendored nppes cron now carries the enriched-CSV + 4-segment config.
2026-06-14 21:02:08 -05:00
justin
134a2611f6 otc: reincorporation email template + campaign builder
otc_reincorporation.html: redomesticate-to-Texas hook (Business Court + TXSE +
DE franchise-tax cost) personalized by state_inc_name/company/ticker, cross-sell
RA/foreign-qual/annual-report/franchise-tax, same-day coupon, lead-capture CTA to
/contact?service=reincorporation (high-touch corporate service, not self-serve),
careful 'not a law firm / not legal advice' disclaimers + CAN-SPAM address.

build_otc_campaign.py: emails only verified-email issuers from the harvest+scrape
+verify pipeline, --de-nv-only for the best reincorp fit, reuses trucking sender
plumbing + coupon. Per-deal value is high so capped modestly (400/run default).
2026-06-14 06:58:43 -05:00
justin
4d3af2aeae otc: domain->email scraper + filing-agent domain filtering
scrape_otc_emails.py: fetch each issuer domain's IR/contact pages (gzip,
HTML-only, early-abort, prefer ir@/investor@/info@), extract a contact email.
Skip filing-agent domains (DFN/Donnelley/Broadridge/etc.) that leak into the
extracted domain -- those are not the issuer's site. Same filter added to the
harvester's DOMAIN_NOISE for future runs. Phone (100%) is the fallback channel
for email misses.
2026-06-14 06:56:45 -05:00
justin
fdea97e57e otc: EDGAR harvester for US-domestic OTC issuers + domain-from-filings
Pilot -> production: harvest_otc_issuers.py pulls the OTC/None universe (2,771),
keeps US-domestic (requires BOTH a US state-of-incorporation AND a US-state
business address -- disambiguates the 'DE'=Delaware-vs-Germany trap that leaked
Infineon etc.), and extracts each issuer's website DOMAIN directly from its
latest 10-K/8-K/DEF-14A filing (free, no scrape; ~58-60% find rate in testing).
Outputs cik,name,ticker,state_inc,phone,city,state,zip,domain -- ready for the
domain->email scrape + verify step. Phone is 100% (clean fallback call channel).
Reincorporation-to-TX / RA / foreign-qual / franchise-tax / annual-report fit.
2026-06-14 01:24:56 -05:00
justin
591e387513 docs: SEC/OTC pilot results - viable (domain free from EDGAR filings, 100%)
Ran the email-findability pilot we should have run for CLIA. SEC/OTC is viable:
~940 US-domestic OTC issuers, domain recoverable from the 10-K/8-K filing itself
at ~100% (free, no scrape), email via site scrape ~25-50%, phone 100%. High
per-deal value (reincorporation/RA/foreign-qual/franchise tax). Documented the
build plan.
2026-06-14 01:22:04 -05:00
justin
1465690832 docs: record HC org-email diversification + CMS-URL maintenance note 2026-06-14 01:10:31 -05:00
justin
b73edadb89 hc: unlock the full 62k verified institutional pool for broad offers
The OIG-screening + NPPES-update segments were effectively limited to ~1,437
providers because the warmup 'any' selector excluded not-on-reval-list rows as a
deliverability proxy -- but that excludes almost the ENTIRE institutional list
(org NPIs aren't individual Medicare enrollees). Since we already SMTP-verified
all 63k inboxes, add an 'institutional_verified' selector that trusts our own
verification instead of reval-list presence. Result: OIG + NPPES-update now
address 62,422 (43x more), giving multiple broad offers to test engagement on.

- enrich_institutional_revalidation.py: fast local join of the institutional
  list to the CMS Revalidation Due Date List bulk file (revalidation_base.csv)
  by NPI -> adds reval_due_date/days_overdue/reval_status. ~1,437 are genuine
  Medicare enrollees (197 overdue / 164 due-soon) -> flagship $599 reval pitch.
- npi_reactivation stays on leie_or_deactivated (only REAL deactivations -- no
  false 'your NPI is deactivated' claims to active orgs).
2026-06-14 01:07:40 -05:00
justin
792f5e948f docs: vertical lead-source analysis ranked by email-source reliability
Synthesize this session's findings into a ranking of candidate verticals by the
one thing that actually gates cold email: a reliable public bulk source of
deliverable emails. Tier 1 (email native): FCC, FMCSA. Tier 2 (one free hop):
healthcare ORG NPIs (already harvested 63k verified), SEC/OTC corporate. Tier 3
(domain-scrape): FMC OTI, state business entities by trigger. Tier 4 (phone/mail
only, NOT email): CLIA (0.3% match proven), EPA RCRA, individual NPIs.
2026-06-14 00:56:27 -05:00
justin
a2665c22c2 ucr: annual-renewal reminder campaign + order-alert campaign source
UCR (Unified Carrier Registration) is annual: opens Oct 1, due Dec 31, mandatory
for interstate carriers (op A, same ~628k pool as IFTA) -> recurring revenue.

- build_ucr_annual_campaign.py: 3-touch business-day cadence (30/12/4 bd before
  Dec 31, wider than IFTA since it's once a year), escalating tone, same-day
  coupon, 'I already did it' suppression. Reuses build_trucking_campaigns +
  IFTA business-day/token helpers (DRY). Per-year cycle reset.
- ucr_annual_reminder.html: deadline + fines/OOS risk + 'we figure out your fee
  tier' + coupon + filed link + CAN-SPAM. Source campaign 473.
- migration 096: ucr_reminded_at / ucr_touch_no / ucr_self_filed_at.
- ifta.ts: add GET /api/v1/ucr/filed (shares the HMAC token scheme).
- checkout.ts: order-placement Telegram now shows 'Source: campaign (code X)'
  when a discount code is present, so IFTA/UCR/CLIA conversions are visible.
  (Confirmed order-alert Telegram already fires from handlePaymentComplete for
  all compliance orders via both webhook + session paths.)
2026-06-14 00:30:23 -05:00
justin
2b361a83a8 fix(ifta): 'I already filed' link must use the API host, not the site host
The handler is an API route; performancewest.net/api/* proxies to the Astro site
(:4322), only api.performancewest.net serves the API. Build the filed link from
PUBLIC_API_URL (default https://api.performancewest.net). Verified end-to-end:
valid token -> 200 confirmation page, invalid -> 403.
2026-06-13 23:49:05 -05:00
justin
3d4226e95c ifta: 3-touch business-day cadence + 'I already filed it' suppression
- Multi-touch reminders at 10/7/4 BUSINESS days before each deadline (weekends
  skipped; biz-day math so a touch never lands purely on a weekend with no
  runway). Escalating tone soft -> urgent -> last-chance, with the 'almost too
  late to DIY, we can still file it' angle so it's a convenience sale, not a free
  reminder service. ifta_touch_no tracks the highest touch sent so each touch
  hits only carriers below that level; never repeats a touch.
- 'I already filed it' one-click link: HMAC-tokenized GET /api/v1/ifta/filed
  (token matches between Python builder and api/src/routes/ifta.ts -- verified
  identical output), records ifta_self_filed_at, friendly confirmation page,
  stops further touches this cycle + gives DIY-vs-prospect signal. Builder
  excludes self-filed carriers.
- migration 094 (ifta_touch_no) + 095 (ifta_self_filed_at); cycle reset clears
  both each new quarter. Verified: biz-day touch schedule, token cross-match.
2026-06-13 23:41:14 -05:00
justin
872154ebf7 fix(trucking): refresh subscriber attribs for existing carriers on re-import
add_subscriber only attached existing subscribers to the new list without
updating attribs, so a carrier emailed in a prior campaign kept STALE attribs --
meaning the daily coupon code and IFTA merge fields (ifta_due_date, ifta_quarter,
lp_link) rendered BLANK for any repeat recipient. Now merge+PUT the fresh attribs
on the existing subscriber before attaching. Affects all trucking campaigns, not
just IFTA. Verified: IFTA preview now persists ifta_quarter/ifta_due_date/lp_link.
2026-06-13 23:26:47 -05:00