Commit graph

624 commits

Author SHA1 Message Date
justin
c8c9a04c1d hc: add 'revalidation due soon' warmup segment (proactive, grows supply)
The HC warmup pool is supply-constrained (~400 verified providers, all fed by
the same narrow 'revalidation 1-90 days OVERDUE' slice). This adds a mirror-image
proactive segment that targets providers whose Medicare revalidation is UPCOMING
within the next 1-90 days, drawn from the same CMS Revalidation Due Date List --
no new data source needed. 'Handle it before your deadline' is a strong pitch and
roughly doubles the deliverable pool.

- New selector reval_due_soon (status=upcoming, days_until in [HC_DUE_SOON_MIN,
  HC_DUE_SOON_MAX] default 1-90).
- New segment revalidation_due_soon reusing the existing /order/npi-revalidation
  service ($599) with template hc_revalidation_due_soon.html.
- attribs_for now exposes days_until (positive days to due date).
- Added to ACTIVE_SEGMENTS.
2026-06-12 19:33:49 -05:00
justin
773c443079 legal: permanent do-not-contact for dataspindle.com + close re-import gap
David Sgro (PA OAG complaint BCP-26-05-025816) opted out 2026-04-13; response
emailed to the AG 2026-06-11. To make the suppression bulletproof and keep the
response's representations true:
- Added a legal do-not-contact list (DO_NOT_CONTACT_DOMAINS/_EMAILS) to
  _email_exclusions.py with dataspindle.com / dave@dataspindle.com; folded into
  BLOCKED_EMAIL_DOMAINS and is_blocked().
- listmonk_import.upsert_subscriber now refuses to import/re-confirm any
  suppressed address. This closes the exact gap that re-added him on 2026-04-26:
  the duplicate-import branch re-added an existing unsubscribed subscriber to
  lists with status=confirmed, overriding the opt-out.
2026-06-11 13:24:10 -05:00
justin
32623d36b8 legal: draft PA AG response re David Sgro complaint BCP-26-05-025816 (docx + md)
Draft response to PA OAG Bureau of Consumer Protection mediation request.
Core arguments: (1) address came from his own public FCC RMD filing, not
scraping; (2) commercial email is governed by CAN-SPAM (opt-out, permits B2B),
not the fax/telemarketing 'Unsolicited Telecommunication Advertisement Act' he
cites; (3) opt-out honored same day (manual suppression Apr 13), now permanent;
(4) no purchase/harm; (5) the post-opt-out 'emails' he complains of were our
replies to HIS own argumentative emails, not solicitations. Marked DRAFT FOR
ATTORNEY REVIEW with bracketed items to confirm before sending.
2026-06-11 12:40:48 -05:00
justin
887bf9a14a warmup: grow main (trucking) pool faster -- 3k -> 4k/day now, 5k at day 14
The main sending IPs are cleanly warmed: today 3,845 sent at 0.18% bounce,
ZERO deferrals, ZERO ISP rate-limit/blocklist/Spamhaus hits. The script's own
note records these IPs historically sustained ~2,500/day at 68-76% delivery;
collapses only ever came from 17k-29k spikes. So we have ample headroom to
accelerate the trucking ramp safely:
  day 7-13: 300/h -> 400/h (~4,000/day)   [applied now, day 8]
  day 14+:  new    500/h    (~5,000/day)   [hard ceiling, well under ~17k]

Also vendored pw-listmonk-rampcap into the repo (infra/postfix/) -- it
previously lived only on the server at /usr/local/bin. Live script updated and
applied (listmonk cap now 400/h).
2026-06-11 00:13:41 -05:00
justin
c8a0824143 firewall: allow ezstorehost (207.174.124.51) to reach Forgejo SSH
Add ezstorehost to trusted_admin in both layers — the nft input set and
the DOCKER-USER iptables chain (Forgejo is containerised; DNAT means the
post-DNAT dport 22 rule applies). Required for static-tenant deploys from
ezStorehost-infra to clone repos over ssh://.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-10 22:45:43 -05:00
justin
a1db921c71 mcs150/workers: don't fill MCS-150 for non-form services; quiet ERPNext workflow advance
- MCS150UpdateHandler is the catch-all for many admin-assisted DOT services
  (UCR, MC authority, audit prep, ETA, name reservation, registered agent,
  annual report). It was filling an MCS-150 PDF for ALL of them -- e.g. a UCR
  order produced a wrong MCS-150 PDF. Now only MCS150_FORM_SLUGS fill the form;
  others get an admin-review todo (PDF 'not generated') for manual handling.
  Signature flow was already correctly scoped (UCR is not in DOT_SIGNING).
- handle_process_compliance_service forced the Sales Order workflow_state to
  'Review' via set_value, which bypasses ERPNext's allowed transitions and
  threw WorkflowPermissionError (Received -> Review) on every run. The Postgres
  fulfillment_status is the source of truth; the ERPNext workflow_state is a
  cosmetic mirror. Now try the proper apply_workflow action and stay quiet
  (debug, not warning) when no valid Review transition exists.
2026-06-10 17:22:38 -05:00
justin
a04146da2b crtc: remove Canadian accountant/accounting-setup service (no longer offered)
We no longer offer Canadian accountant/accounting setup. Removed all
service-offering content:
- Marketing page (services/telecom/canada-crtc): the 'Set Up Canadian
  Accounting (we help)' next-steps card, the '3 hours of complimentary
  accounting consultation' deliverable bullet, and the whole 'Accounting
  Support' section (assigned accountant, portal chat, $75/hr, 3 complimentary
  hours).
- Order page (order/canada-crtc): the '3 hrs Canadian accounting support'
  included-feature bullet and the 'Preferred accounting software'
  (Xero/QuickBooks) form field + its accounting-hours helper text.
- Fulfillment (canada_crtc.py): dropped the bank-setup email line offering
  '3 hours of Canadian accounting consultation'.

Kept factual GST/HST tax advisories and the bank's QuickBooks/Xero
transaction-sync feature (third-party bank capability, not our service).
2026-06-10 16:51:33 -05:00
justin
7d8a08d9d3 mcs150: scope intake-completion email to actual MCS-150-form services
MCS150UpdateHandler is the catch-all for many admin-assisted DOT services
(UCR, MC authority, audit prep, ETA, name reservation, registered agent,
annual report). My new intake-completeness gate was firing the 'confirm your
MCS-150 details' email for ALL of them -- e.g. a UCR order wrongly emailed the
customer about MCS-150 details. Scope the gate to MCS150_FORM_SLUGS (the
services that actually file an MCS-150: mcs150-update, dot-registration,
usdot-reactivation, dot-full-compliance).
2026-06-10 14:52:36 -05:00
justin
1ff8b88ac8 fix: stop suppressing synthetic@pipeline.com (real customer address)
Paul Wilson (Compound Technologies) signed up with synthetic@pipeline.com,
which is a genuine, deliverable EarthLink address (pipeline.com MX ->
earthlink-vadesecure.net; he confirmed receipt by phone). Our code had
hardcoded pipeline.com + the synthetic@ prefix as a 'non-deliverable
FMCSA-census placeholder' and silently suppressed every automated email to
him (checkout provisioning, order-creation validation, intake reminders,
set-password invites). Nothing in the codebase actually generates that
address, so the placeholder rationale was wrong. Removed pipeline.com and the
synthetic@ rule from all four suppression sites; only RFC-reserved
example.com/test.com/invalid remain blocked.
2026-06-10 14:41:19 -05:00
justin
983c732baf migration 093: add awaiting_intake fulfillment status for MCS-150 intake gate 2026-06-10 14:14:45 -05:00
justin
a3aeedd716 mcs150: census-prefilled intake-completion flow + completeness gate
Closes the data gap for orders that bypass the full intake (e.g. the DOT
compliance-remediation pipeline) and for all MCS-150 variants:

- Worker intake-completeness gate (mcs150_update): before filling, check the
  customer-required operational fields the FMCSA census cannot supply
  (operation classification, cargo, CURRENT annual mileage, email; plus
  signer/address for new-registration/reactivation, and states-of-operation
  for 150B hazmat). If missing, email the customer a census-pre-filled intake
  link and hold the order at fulfillment_status='awaiting_intake' with an admin
  todo, instead of fabricating a blank filing. The existing intake PUT endpoint
  already re-dispatches the worker on submit, so filing auto-resumes.
- Intake wizard (Wizard.astro): when resuming ?order=CO-xxx for a DOT/MCS order,
  seed still-empty fields from the FMCSA census (name/address/fleet/interstate)
  so the customer only confirms the operational details.
- /api/v1/dot/census now also returns total_drivers + a normalized
  carrier_operation_code for the prefill.
- MCS150Step.astro extended to collect every field the filler needs across all
  variants: mailing address, cdl_drivers, primary_vehicle_type,
  reason_for_filing, usdot_revoked, cell/fax, hazmat-safety-permit block
  (needs_hmsp, operating states, security plan), and intermodal-equipment
  provider counts; all prefill from intake_data.

verify_mcs150_variants.py covers 150/150B/150C end-to-end (ALL PASS).
2026-06-10 14:03:28 -05:00
justin
38739e023c mcs150: complete all-variant field mapping (150/150B/150C)
Adds the previously-unmapped fields so every variant fills fully:
- Q25 hazmat C/S/B/NB matrix (HAZMAT_ROW_MAP x HAZMAT_COL_MAP, 156 boxes)
- MCS-150B states-of-operation checkboxes (full name or 2-letter code), HMSP
  Hazard/Permit/Security radios, and accident count (32accidentNumber)
- MCS-150C intermodal equipment counts (20owned/leased/serviced) + correct
  field renumbering (17dunbrad/18irs/19eMail) + USDOT Button + named-export
  Reason/Mailing radios
- Structured fleet via intake['vehicles'] = {vehicle_type: {owned, term_leased,
  trip_leased}} across all Q26 vehicle rows; non-CMV count; cell/fax; second
  officer
- _set_button now resolves a candidate tuple against each field's actual export
  states, so numeric (/0../4) and named (/Yes,/Biennial...) radios both work

verify_mcs150_variants.py exercises all three variants end-to-end: ALL PASS.
2026-06-10 13:55:55 -05:00
justin
96f31e7c31 mcs150: only check Q29 passenger-cert box for passenger carriers
certifyBox is the Q29 Passenger Carrier Compliance Certification YES box
(page 3, y=530), not a general perjury checkbox. It was being checked
unconditionally, which wrongly marked freight/property carriers as passenger
carriers. Now only check it when the carrier is a passenger carrier; the
Q31 perjury declaration is made via the signature.
2026-06-10 13:44:34 -05:00
justin
b95ee04752 mcs150: fill all checkboxes/radios correctly + stamp explicit checkmarks
Fixes a batch of missing fields the FMCSA census does not provide and the
filler was mis-mapping:

- Corrected the question->field mapping to match the actual form: Q22 =
  COMPANY OPERATIONS (interstate/intrastate, 22xBox), Q23 = OPERATION
  CLASSIFICATIONS (for-hire/private/govt, 23xBox). These were swapped, and
  the bogus entity-type->23xBox map (no entity-type question exists on this
  form revision) was removed.
- Added proper radio-group handling for Reason for Filing (Biennial Update),
  Mailing-address (Same as principal vs below), and Q28 USDOT-revoked, with
  correct option indices (these are /0../n radios, not /Yes checkboxes; the
  old code set them to /Yes and never selected the right option).
- Map interstate/intrastate from the FMCSA census carrierOperationCode, and
  populate email/phone/mileage/cargo from intake.
- AcroForm checkbox/radio appearances use a ZapfDingbats glyph that
  poppler/Preview fail to render (value set but box looks empty). Now stamp
  an explicit X overlay into the page content for every 'on' box so it shows
  in every viewer and in the faxed output.
2026-06-10 13:41:48 -05:00
justin
386467bedf mcs150: trim FMCSA instruction pages from form templates
The official MCS-150/150B/150C PDFs ship with 8 (150/150B) or 4 (150C)
FMCSA instruction/example pages before the actual fillable form. We were
generating + faxing/submitting all of them. Trimmed the source templates
down to the FORM pages only:
  MCS-150  11 -> 3 pages (289 fields preserved)
  MCS-150B 12 -> 4 pages (349 fields preserved)
  MCS-150C  6 -> 2 pages (33 fields preserved)

The filler iterates writer.pages (no absolute index) and signature
anchors are derived dynamically via enumerate(reader.pages), so no
page-specific markup needed fixing. Removed one-off diag script.
2026-06-10 13:25:07 -05:00
justin
4447905864 workers: don't re-upload handler-returned MinIO keys as local files
handle_process_compliance_service assumed handlers return local temp
paths and re-uploaded each to MinIO. The MCS-150 handler uploads itself
and returns the MinIO key, so the re-upload tried to read a nonexistent
local file and logged a 'File not found' error after the order was
already correctly held at the admin gate. Now we skip files that don't
exist locally and keep the returned key as-is.
2026-06-10 12:47:16 -05:00
justin
71404810c4 mcs150: backfill signer name from signed cert + re-stamp signed form
- When intake lacks signer_name, backfill it from the name the client
  typed when signing the perjury certification (that name is exactly what
  belongs in the form's print/type-name field, certifyName).
- After a client-approved re-dispatch, re-point the signed esign record at
  the freshly filled form and re-stamp the signature, so the signed PDF an
  admin reviews reflects the current complete form (not a stale earlier
  fill). Field layout (and thus signature anchors) is unchanged across
  fills, so the recorded anchor coordinates stay valid.
2026-06-10 12:41:27 -05:00
justin
bd0d8c0e2c verify_mcs150: correct handler class name 2026-06-10 12:39:41 -05:00
justin
79c460ef25 mcs150: render verification harness + auto-generate appearance streams
- fill_mcs150 now uses auto_regenerate=True so pypdf writes appearance
  streams for every text field. Preview/Chrome ignore /NeedAppearances and
  were showing blank widgets over the values; generated /AP streams make
  the text render in all viewers.
- New verify_mcs150.py reads each widget's /AP /N appearance stream (the
  literal drawn glyphs) to confirm expected values actually render, since
  the container has no OCR/raster tooling. Exits non-zero on any miss.
2026-06-10 12:37:25 -05:00
justin
534f13e480 gitignore: stray personal image 2026-06-10 12:35:56 -05:00
justin
d5e66786a2 mcs150: enrich intake from FMCSA carrier census before PDF fill
The MCS-150 biennial update re-confirms the carrier's existing FMCSA
record. Previously the PDF filler only had whatever the intake form
collected; rescued/sparse orders (or orders where the carrier's data
lives in FMCSA, not the intake) produced near-empty forms. Now we pull
the carrier census (legal name, address, EIN, fleet counts) from the
FMCSA carrier API and merge it under any customer-provided intake values
(customer edits win), so the form is pre-filled with the carrier's
current registered data. Refactored the FMCSA fetch into a shared
_fetch_fmcsa_carrier helper used by both enrichment and status check.
2026-06-10 12:35:52 -05:00
justin
7e5946d65a fix(mcs150 pdf): set /NeedAppearances so filled field values actually render
Customer saw the MCS-150 looking blank / 'data covered by the form fields': the
values were correctly written to the AcroForm /V, but pypdf left the template's
empty /AP appearance streams in place and NeedAppearances was false, so viewers
rendered the blank widget over the value. Setting AcroForm /NeedAppearances=true
makes viewers regenerate appearances from the values. (The missing signature was
a downstream effect of the separate fobj_put MinIO-upload bug, now fixed -- with
no PDF in MinIO the anchor extraction + signature stamping both failed.)
2026-06-10 12:20:45 -05:00
justin
b28dda7c5a feat(telegram): include a presigned PDF view link in the admin-review alert
When an MCS-150/USDOT order hits the pre-submission admin-verification gate, the
Telegram FULFILLMENT NEEDED alert now appends a presigned link to the prepared
PDF (via the public minio.performancewest.net endpoint, IP-allowlisted to admin)
so you can review the document straight from the alert before approving. Added
notify_fulfillment_todo(view_url=...) + a _presigned_view_url helper (public
endpoint + explicit region to avoid the region-probe that 403s from the worker).
2026-06-10 12:13:43 -05:00
justin
09d928a582 fix(mcs150): MinIO upload used wrong method fobj_put -> fput_object
The MCS-150/USDOT PDF was generated fine but the MinIO upload threw 'Minio object
has no attribute fobj_put' (wrong method name + signature), so the prepared filing
PDF was never persisted -- nothing for an admin to review at the verification gate,
and the esign-completed re-dispatch failed with 'File not found'. Use the correct
minio fput_object(bucket, key, file_path). Affects every MCS-150/USDOT filing.
2026-06-10 12:08:12 -05:00
justin
1fe942c109 fix(checkout): don't skip ERPNext SO for synthetic@pipeline.com (real customers use it)
ensureComplianceSalesOrder skipped the FMCSA-census placeholder email, but a real
paying customer (Paul Wilson) genuinely uses synthetic@pipeline.com, so his SO
never got created/regenerated. A Sales Order is internal bookkeeping, not an
outbound email, so the placeholder skip is unnecessary here (the email/portal
guard in ensureCompliancePortalUser still protects actual sends).
2026-06-10 07:05:21 -05:00
justin
7708086130 Fix batch payment totals in Telegram and ERPNext invoice flow 2026-06-10 07:04:06 -05:00
justin
6827aafdbc fix(checkout): batch surcharge 5x over-count + ERPNext SO missing discount
Mitchell's batch CB-95BA6C90: Stripe correctly charged $450.88 ($437.75 net +
$13.13 surcharge), but the DB + Telegram showed $503.40 with a $65.65 surcharge.
Two bugs:

1) On Stripe session creation, the per-row surcharge UPDATE wrote the FULL batch
   surcharge ($13.13) to EVERY row via WHERE batch_id, so anything summing the
   per-row field (the Telegram order notification) over-counted Nx (5 x $13.13 =
   $65.65). Now the single surcharge is split across the rows so they sum to the
   true total. Stripe was always charged correctly (one surcharge line item).

2) ensureComplianceSalesOrder built the ERPNext SO from full line-item prices but
   applied NO discount, so the SO grand total over-stated what the customer paid.
   Now applies the promo/bundle discount via apply_discount_on=Grand Total +
   discount_amount on both the primary and fallback SO create.
2026-06-10 06:57:59 -05:00
justin
058d4d426a feat(compliance): admin verification gate + durable submission evidence
Per request: after the customer signs but BEFORE we submit to the government, hold
the order for a human to verify the prepared filing is correct.

- MCS-150 handler (mcs150-update + usdot-reactivation): new admin-verification gate
  after the signature gate -- if not admin_approved, set fulfillment_status=
  'ready_to_file', create a HIGH-priority 'VERIFY before filing' admin todo, and
  STOP (no FMCSA submission). job_server injects admin_approved from the dispatch
  payload (mirrors client_approved).
- New admin endpoint POST /api/v1/admin/compliance-orders/:id/approve-submit
  (requireAdmin): verifies status=ready_to_file, re-dispatches the worker with
  admin_approved=true to proceed to actual submission.
- Durable submission EVIDENCE: the web/fax submitters only wrote confirmation
  screenshots to an ephemeral temp dir. Now _upload_submission_evidence copies the
  FMCSA confirmation screenshot + attested PDF + fax_log_id to MinIO under
  filings/<slug>/<order>/evidence/ and records the keys on the order, so we keep
  proof of every government submission.

(state-trucking + the FCC handlers already gate via admin todos / auto_filing.py;
this brings MCS-150 to parity and adds evidence retention.)
2026-06-09 22:50:30 -05:00
justin
e87715aee7 fix(portal): onboarding/login links last 7 days, not 60 min
The rescue onboarding emails hardcoded a 60-minute expiry -- way too short for a
paid customer who hasn't engaged yet (they may not check email for hours/days),
so Paul's and Mitchell's links expired before they used them. Onboarding links
now last 7 days (ONBOARDING_TTL_MINUTES); the standard security password-RESET
window bumped 30min -> 2h. Re-issued fresh 7-day links to all 3 affected
customers (none had set a password yet) via reissue-onboarding-links.mjs, cc'd.
2026-06-09 22:50:09 -05:00
justin
a6d2f10149 chore: Mark Adams rescue (3rd real customer stuck on the login bug)
Paid Jun 1 for MCS-150 (card), no customers row -> couldn't log in -> no intake ->
filing stuck 'NEEDS MANUAL FILING'. Created his customers row + sent login +
intake email (cc justin). All 3 real paying customers now rescued; the underlying
card/PayPal login bug is fixed so new orders won't hit this.
2026-06-09 22:34:52 -05:00
justin
1854753c70 monitoring: add .91-.93 IP rehab to daily Telegram warmup alert
Tracks the rehab pool (rehab02-04 / .91-.93) delivery + bounce + Spamhaus ZEN
DNSBL status in the daily report and alert body. Alerts only if a rehab IP lands
on a DNSBL or rehab delivery drops <40% with real volume (recipient quality
slipped) -- a recovering IP naturally bounces more so the threshold is lenient.
2026-06-09 20:34:41 -05:00
justin
1c2e263bb7 warmup(ip-rehab): bias recipients to multi-subscriber business domains (cut bounce)
Day-0 batch saw ~45% bounce because 'no listmonk bounce record' is a weak
liveness signal. Now require the recipient's domain to have >=2 enabled
subscribers (a real org, not a one-off typo'd address), which materially lowers
the dead-mailbox bounce rate on the recovering IPs.
2026-06-09 20:31:45 -05:00
justin
25f4a7503b warmup: IP rehab for .91-.93 so they can be reallocated
The 3 IPs (mta02-04 / .91-.93) retired after the May 30-31 over-volume blast are
NOT on any DNSBL (Spamhaus/Barracuda/SpamCop/SORBS all clean) and have clean PTRs
+ SPF/DKIM/DMARC -- the damage was provider-internal reputation, which recovers
with slow clean sending. scripts/ip_rehab.py sends a tiny ramping trickle
(10/IP/day -> cap 60) of genuine CAN-SPAM-compliant compliance check-in mail to
clean business-domain, never-bounced recipients via dedicated heavily-throttled
postfix transports rehab02/03/04 (30s/msg, bound to .91/.92/.93). Routing uses an
X-PW-Rehab-IP header + header_checks FILTER to override the transport_maps randmap
warmup rotation (verified: mail routes via rehab transports, status=sent). Daily
cron pw-ip-rehab. After ~2-3 weeks of clean sending the IPs can be reallocated.
2026-06-09 20:27:47 -05:00
justin
6f361d307d warmup: add Microsoft consumer (hotmail/outlook/live/msn) to cold-mail exclusions
Main-pool delivery was stuck ~60-67% (897+ daily spam-blocks). Audit of the main
listmonk found ~47k consumer mailboxes still enabled in OLD cold trucking lists
(built before the exclusion existed): 19k gmail, 27.5k yahoo-family + Microsoft
consumer. gmail alone was 101 of today's 909 550-5.7.1 blocks. Blocklisted all
~47k consumer addresses NOT in the protected FCC lists 3/6 (per the keep-gmails
decision for those warm lists). Also added Microsoft consumer domains to
BLOCKED_EMAIL_DOMAINS so the daily trucking builder stops re-adding them
(Microsoft SmartScreen silently junks/defers cold B2B mail -- a reputation drag
even though it doesn't 550 like Google). Enabled base 95455 -> 48707 (real
business/ISP domains).
2026-06-09 20:10:54 -05:00
justin
9fa2c86f01 fix(warmup): HC cron logged to /var/log (deploy can't write) -> cron silently died
The HC warmup builder ran from cron at 07:00 but the >> /var/log/pw-hc-campaign.log
redirect failed (deploy user cannot write /var/log), and a failed output redirect
makes cron abort the command BEFORE it runs -> HC sent 0/day since the log file was
removed. Route HC cron logs to /opt/performancewest/logs/ (deploy-owned) so the
redirect always succeeds. Builder itself was fine (verified: imports + sends work,
0 bounces). Also removed the stale 'campaign-warmup.sh 122' root-cron line that
pointed at a finished campaign + no longer existed.
2026-06-09 16:06:28 -05:00
justin
3dbf5b3bb2 chore: Mitchell Allen rescue scripts (customers row, SO backfill, re-dispatch, login+signature email) 2026-06-09 14:58:52 -05:00
justin
8d301a1ab7 fix(checkout): SO creation falls back to generic item if a catalog Item is missing
Tracing Mitchell's batch, SO creation 404'd on a missing ERPNext Item
(INTRASTATE-AUTHORITY) -- 30 of 70 catalog erpnext_item codes were missing in
ERPNext. Created all 30. Added a safety net: if createResource('Sales Order')
404s on a missing Item, retry once with every line remapped to the generic
COMPLIANCE-SERVICE item so a missing catalog Item never strands a paid order's SO.
2026-06-09 14:50:06 -05:00
justin
e2467752dc chore: export ensureComplianceSalesOrder for rescue/backfill use 2026-06-09 14:44:28 -05:00
justin
68e6b60951 fix: worker emails (localhost:25 -> SMTP relay) + create ERPNext SO on webhook payment
Two bugs found tracing Mitchell Allen's batch CB-95BA6C90 (5 DOT services, card):

1) Worker authorization/signing-link/status emails were sent via
   smtplib.SMTP('localhost', 25), which has no MTA in the workers container ->
   every send failed '[Errno 111] Connection refused', so customers never got
   their e-sign links and orders sat 'awaiting client signature' forever. Routed
   all 9 hardcoded localhost:25 sites (state_trucking, mcs150_update, boc3_filing,
   hazmat_phmsa, mailbox_setup, dot_esign, completion_emails) through the
   authenticated SMTP relay (SMTP_HOST/PORT/STARTTLS/login) + added a shared
   worker_email.send_worker_email helper.

2) The ERPNext Sales Order for compliance/compliance_batch was only created in
   the /checkout/create-session endpoint, but CARD orders confirm via the Stripe
   WEBHOOK -> handlePaymentComplete, which never created the SO. Result: every
   webhook-confirmed order had erpnext_sales_order=NULL and workers logged
   'Sales Order not found 404' then built from PG. Added idempotent
   ensureComplianceSalesOrder() to handlePaymentComplete so ALL payment methods
   (card-webhook, PayPal, crypto) create + link the SO.
2026-06-09 14:40:46 -05:00
justin
220f301453 test(e2e): fix compliance_orders seed columns (no total_cents); regression PASS
e2e-paypal-portal-fix.mjs now passes against live prod: completing a compliance
order creates the customers row (id, name=E2E Tester, company from intake_data,
no password) -> customer can register/reset + log in. PayPal login bug fixed.
2026-06-09 14:35:04 -05:00
justin
3c65dd8748 fix(checkout): pull company from intake_data (compliance has no customer_company col)
compliance_orders stores company in intake_data JSON, not a column; read it from
there (company/legal_name/entity_name) with graceful fallback. Fix e2e test seed
accordingly.
2026-06-09 14:31:42 -05:00
justin
9987b1e30d fix(checkout): create Postgres customers row on order completion (PayPal login bug)
Portal login + forgot-password read the Postgres customers table (bcrypt), NOT
ERPNext. ensureCompliancePortalUser (the common path for Stripe/PayPal/crypto via
handlePaymentComplete) only provisioned the ERPNext customer/website-user and
never created the customers row -- so customers (notably PayPal, who reach this
path directly) had no account to log into or reset a password against. Now upserts
the customers row (no password; ON CONFLICT keeps any existing hash) with name +
company so they can register/reset and log in immediately.

Also: narrowed the placeholder-email skip from 'any synthetic@ or pipeline.com' to
exactly 'synthetic@pipeline.com' (the FMCSA-census placeholder) so real customers
on those real consumer domains aren't wrongly skipped -- which is what bit Paul
Wilson. Added cc support to sendEmail. e2e-paypal-portal-fix.mjs is the regression
test (seeds a compliance order, runs handlePaymentComplete, asserts the customers
row is created). Rescue scripts for the affected customer included.
2026-06-09 14:28:19 -05:00
justin
b437f66bc8 docs(dexit): name search fixed (TX open-data API) / honest (NV unknown); rm probes
E2E harness re-run = ALL PASS: TX returns real availability via the open-data API,
NV returns unknown (available=None) to flag manual verification, both flow through
to correct ERPNext Sales Orders. Removed one-off portal-probe scripts.
2026-06-09 08:46:28 -05:00
justin
20c11e6180 fix(formation/NV): name search returns unknown (admin-verify), not a fake result
Nevada's entity search (esos.nv.gov/SilverFlume) is behind Imperva Incapsula bot
protection that blocks headless + the residential proxy IPs, and NV has no public
open-data API/bulk dataset. The old adapter scraped the blocked challenge page and
returned available=False for everything (would tell customers a free name is taken)
and also had a NameError bug (f"${CODE}..."). Now NV detects the Incapsula
challenge and returns available=None ('could not determine') with a note to verify
manually on SilverFlume -- never a false 'taken', so it never wrongly blocks an
order. TX remains fully automated via the open-data API.
2026-06-09 08:41:22 -05:00
justin
f94ad1682b fix(formation/TX): name search via Texas open-data API, not scraping
The TX Comptroller web search is now a JS form (old input#entityName selector
dead) and SOSDirect is login-gated, so the scraper returned garbage. Replaced
search_name with the Texas Socrata 'Active Franchise Taxpayers' dataset
(data.texas.gov/resource/9cir-efmm.json) over SoQL -- free, no-auth, no-login,
no bot-blocks. Exact normalized match => unavailable; no rows => available; API
error => available=None (never a false 'taken'). Verified: unique name = 0 rows
(available), 'APPLE INC.' = exact match (taken).
2026-06-09 08:34:37 -05:00
justin
561ad78ea8 docs(dexit): note NV adapter also mis-parses happy path (not just errors) 2026-06-09 08:14:09 -05:00
justin
76c4d55603 fix(formation): name-search returns null (not false) on adapter error
E2E harness exposed that the NV name-search adapter times out on a stale input
selector and search_name() swallowed the error and returned available=False --
i.e. it would tell customers an AVAILABLE name is taken. Now returns
available=None ('could not determine') on adapter error / unknown state, which the
API already maps to null. The NV/TX portal selectors still need a scraping fix
(separate task; e2e harness is the acceptance test) before enabling a self-serve
formation checkout. Documented full e2e results + the bugs caught (missing ERPNext
Items, entity_type case, missing /name-search route) in the readiness doc.
2026-06-09 08:06:43 -05:00
justin
4c0decd175 fix(formation): add working /name-search worker route + e2e harness
Two latent bugs the e2e harness caught:
1. api entities.ts GET /states/:code/name-search calls WORKER_URL/name-search,
   but job_server had NO such route -> 404 -> silently fell back to stale
   entity_cache on every live name check. Added a synchronous /name-search route
   returning {available,exact_match,similar_names,state}.
2. both the new route AND the existing async handle_name_search imported a
   nonexistent search_name_sync(); fixed to drive the real async search_name()
   via an event loop (same pattern as /entity-status).

scripts/e2e-formation-order.mjs: replays the real formation order flow (live
name search -> formation_orders insert -> ERPNext customer + Sales Order with
BUSINESS-FORMATION + STATE-FILING-FEE line items -> verify SO total + DB linkage
-> cleanup) without a real Stripe charge or state filing. Run in the api container.
Also created the missing ERPNext Items (BUSINESS-FORMATION, STATE-FILING-FEE,
FOREIGN-QUAL-SINGLE/MULTI) that the formation SO references.
2026-06-09 07:51:54 -05:00
justin
c0344769a0 docs(dexit): handle foreign-qualification complication on a move
A DE corp foreign-qualified in CA/NY/etc must update EVERY foreign registration
when it domesticates (withdraw in the destination state since it becomes domestic
there; amend or re-file in the rest to reflect the new home state) -- getting it
wrong = doing business unregistered (default judgments/penalties). We already have
the building block (foreign-qualification-single/multi SKU + ForeignQualificationHandler
that fans out per state + migration 066/073 schema); missing is amend/withdraw modes
+ an intake step capturing the list of qualified states. Product = multi-part, priced
per state touched (revenue multiplier), scoped at intake.
2026-06-09 07:39:50 -05:00
justin
b5b2e6e6c3 site: add DEXIT corporate-services page + readiness assessment + cited filings
New page /services/corporate/dexit-reincorporation (matches CRTC service-page
structure): explains DEXIT, the DE franchise-tax dollar driver (real Oracle Health
proxy: $23,600 -> ~$1,000), NV/TX/FL destination guidance, 6-step how-it-works,
3 cited real SEC reincorporation filings (Oracle Health, FG Financial, LogicMark)
with verbatim quotes + EDGAR links, honesty callout, and a lead-gen CTA ('Get my
DEXIT estimate' -> /contact?topic=dexit, NOT a buy-now checkout). Linked from the
corporate services index (new card) + the global Services dropdown across the site.

docs/dexit-cited-filings.md: the filing excerpts + verified gov/statute links.
docs/dexit-readiness-assessment.md: HONEST e2e readiness -- new NV/TX formation is
built (checkout order_type=formation -> formation_orders -> ERPNext SO ->
formation_worker -> TX/NV adapters) but unverified e2e; the 'move a company'
(conversion/domestication) flow + corporate annual-report automation are NOT built;
EIN is kept on a conversion (our ein_worker does NEW EINs only). Page stays lead-gen
until the generic entity-conversion SKU + admin-assisted handler are built+tested.
2026-06-09 07:35:12 -05:00