SMTP2GO is no longer used: Listmonk relays through the local Postfix MTA
(172.18.0.1:25 from the Docker network), which DKIM-signs and delivers
direct-to-recipient-MX; transactional mail goes through Carbonio. Verified
zero smtp2go in any live container env + postfix has no external relayhost.
Removed the stale references so a rebuild/new dev can't re-introduce it:
- api/src/config.ts: SMTP_HOST default mail.smtp2go.com -> co.carrierone.com
- scripts/workers/crypto_payment_worker.py: same default fix
- infra/ansible all.yml: listmonk_smtp_* now 172.18.0.1:25, no auth (+comment)
- app.env.j2 / email.ts / crm.md / go-live-todo.md / architecture.svg: docs
All transactional/worker senders built multipart/alternative (or mixed)
messages with ONLY an HTML part. A single-part multipart/alternative is
malformed and HTML-only mail is a spam-score signal -- the same class of
deliverability bug that hurt the campaign pipeline, but on the telecom /
filing / customer-transactional path (499-Q reminders, RMD/FCC filing
review links, intake/completion/delivery emails, commissions, etc).
- worker_email.send_worker_email: auto-derive plaintext from HTML when
caller omits text= (fixes the shared helper for all current+future use)
- 16 rolled-their-own senders in scripts/workers/** + scripts/formation/
document_delivery.py: attach html_to_text(...) plaintext sibling before
the HTML part (job_server + document_delivery wrap text+html in an
alternative sub-part so PDFs still attach to the mixed root)
- api/src/email.ts: add dependency-free htmlToText() and default
sendEmail text to it (fixes checkout/webhook HTML-only sends)
Verified: all py files compile + import at runtime, api tsc passes,
htmlToText handles hrefs/lists/entities, 11 plaintext unit tests pass.
Telecom campaign 407 (Jun 8) was HTML-only + sent in the DKIM-broken
window -> 384 sent / 0 clicks (same junked-mail signature).
The FMCSA census was a one-time snapshot (last loaded ~May 30) with NO refresh
timer -- carriers newly falling out of MCS-150/UCR compliance were never picked
up. New scripts/workers/fmcsa_source_refresh.py orchestrates the full pipeline
(census download -> enrichment -> deficiency flag -> verify new emails ->
MX-tag new) and runs weekly via cron pw-fmcsa-refresh (Sun 09:00 UTC), codified
in the mail-pipeline Ansible role.
Idempotent + incremental: the census upsert preserves email_verified /
listmonk_sent_at / deficiency_flags, so existing carriers keep their send state
and only census fields refresh; new DOTs flow into verification then campaigns.
A carrier who refiled gets a fresh mcs150_parsed, so the builder's overdue
WHERE clause stops targeting them automatically. Verify is capped per run
(20k) so it never stalls on millions of rows.
(Healthcare already auto-catches newly-revalidation-overdue providers within
its 63k institutional pool via pw-hc-refresh Mon/Wed/Fri.)
The verifier returned (True, 'mx_unreachable') when it couldn't complete a port-25
probe to ANY MX — marking 438,163 addresses email_verified=TRUE. But these are NOT
dead: they're dominated by Comcast (13.7k), AT&T/SBCGlobal (13.5k), Verizon, Cox,
Charter, Frontier, etc. — major ISPs that deliberately tarpit/refuse probes from
unknown IPs. Confirmed from prod: comcast MX connects + returns 220. The probe
failure ≠ undeliverable.
Fix: return (False, 'mx_probe_blocked') — MX exists, deliverability UNKNOWN, must
be confirmed by a real send. Excluded from PW campaigns; prime burner-verification
target (burner_list_verify upgrades it to send_confirmed on delivery). Existing
438,163 mx_unreachable rows reclassified in prod to mx_probe_blocked / verified=FALSE.
The MCS-150 intake-completion email linked customers to /order/dot-compliance,
which is the sales/checkout page -- it ignores ?order= and asks the customer to
re-pick services and pay again, so they 'cannot enter any data' (Paul Wilson's
report). Link to the per-service intake wizard /order/<slug>?order=... instead,
which loads the paid order, pre-fills from the FMCSA census, and drops payment.
Also add a Trailers field to the DOT intake fleet section and wire it through to
the MCS-150 PDF Q26 trailer row, so carriers can update trucks AND trailers.
- Added a line asking them to call their insurance agent to confirm Form E
ability before clicking yes/no, so we pick the right path first time.
- Reply-To now routes to info@performancewest.net (monitored), overridable via
SC_COC_REPLY_TO env.
Routes SC intrastate-authority orders to the real SCDMV COC product instead of a
PSC certificate (which doesn't apply to property carriers):
- sc_coc_filing.py: emails the carrier a one-click yes/no — does your insurer
have / can they file a Form E (SC intrastate liability, $750k or $300k by
GVWR) with SCDMV? Records the answer; builds the filled COC package.
- state_trucking._handle_sc_coc_gate: SC intrastate gate —
no answer -> email the question once, HOLD
answered no -> broker referral opened, HOLD (ops todo)
answered yes-> proceed to bill the exact $25 SCDMV COC fee (at cost) + file
- API POST /compliance-orders/:id/sc-insurance: records yes/no in intake_data
(no schema change); NO opens an insurance_lead broker-referral ticket +
Telegram; YES re-dispatches the worker to bill the $25 + file.
- site/order/sc-insurance: customer one-click yes/no page (auto-submits when
the email links straight to ?have=yes|no).
Non-SC intrastate still uses the PSC/PUC email path or a manual todo.
SC for-hire PROPERTY carriers (not passenger/HHG/hazwaste) register intrastate
via the SCDMV Certificate of Compliance (COC), not a PSC certificate. This adds:
- sc_coc_pdf_filler.fill_sc_coc(): fills the official SCDMV Form COC from
intake (business name, officers, physical/mailing address, phone), picks
New vs Renewal, and stamps the coverage class (E-L low-value / E-LC).
Field names in the source PDF are auto-generated + offset from their labels;
mapped here by verified on-page geometry. Verified by render.
- suggest_coverage_class(): E-L for low-value cargo (scrap/dump/aggregate),
else E-LC (safer default).
- gov_fee: SC intrastate fee corrected from $0 placeholder to the real $25
COC new-application fee (renewals $0), billed at cost.
The carrier's INSURER files the Form E (liability) + Form H (cargo, E-LC only)
directly with SCDMV; we collect the COC app + $25 and submit it.
Intrastate operating authority is state-specific + application-based like IRP, so
it reuses the same email/POA + invoice-reconciliation flow:
- intrastate_filing.send_intrastate_submission: emails the state PSC/PUC the
authority application with the signed POA attached (subject tag [PW-ISA CO-..]),
reusing irp_filing's MinIO download + census enrich helpers.
- The shared poller (irp_invoice_poller) now matches BOTH [PW-IRP] and [PW-ISA]
tags, parses the fee, Telegram-alerts, and bills the customer the exact amount
with the correct service slug.
- state_trucking gov-fee gate routes intrastate-authority to the PSC/PUC email
path; if no submission email is configured for the base state it falls back
to a manual todo (safe default — no emailing guessed agency addresses).
Per-state ISA_<ST>_EMAIL env (blank until the exact agency address is verified).
SC/GA/TX scaffolded. Customer still only sees an exact-fee payment link; you only
approve the final filing.
- send_irp_submission now REQUIRES and ATTACHES the signed Power of Attorney PDF
(downloaded from MinIO) — the state won't act on a third-party filing without
it, and 'on file, available on request' stalls the request. If the POA isn't
available we don't email and fall back to a manual todo.
- Backfill missing legal_name + registered address from the FMCSA census so the
submission isn't sent with a blank address (root cause of the empty
'Legal/registered address: , ,' line). Customer-supplied values win.
- state_trucking passes signed_auth_key through to the IRP submitter.
- Fix 'Object of type date is not JSON serializable' when creating the admin
todo (json.dumps(..., default=str)) — broke the intrastate (bash-fee) path.
The gov-fee email now lists exactly what the amount covers (full breakdown) so
the customer can check it for accuracy, with two clear actions: a ✅ pay link and
a ❓ 'something looks wrong' link to /order/dispute.
New /order/dispute page shows the fee breakdown and lets the customer describe
what's wrong; it opens an 'issue' support ticket pre-tagged with the order
(amount + label + their note) via /api/v1/tickets, so ops corrects the fee
before any payment is taken. The /order/pay page also shows the itemized
breakdown and a dispute link.
- gov_fee: add AGENCY_PROCESSING_FEE (per-service card/convenience fee passed
through so the customer pays the true all-in cost); estimate_gov_fee now folds
it into the billed total. IFTA/intrastate/UCR fees are published/near-exact.
- IRP fees can't be looked up — only the base state computes them. New
irp_filing.py: emails the base-state IRP unit a Schedule A/B request (Reply-To
the IRP filings mailbox, [PW-IRP CO-...] subject tag), and a 15-min cron
(irp_invoice_poller) scans the mailbox for the state's invoice reply, parses
the exact apportioned fee, Telegram-alerts you, and bills the customer the
EXACT amount via a gov-fee child order + payment link. Then it proceeds to
ready_to_file for your final approval.
- state_trucking gov-fee gate now routes IRP to the email/invoice path and
IFTA/intrastate to immediate exact-fee billing.
- Mailbox is configurable (IRP_FILINGS_IMAP_* in app.env.j2); falls back to
OPS_IMAP_* filtered by the [PW-IRP] tag until a dedicated mailbox exists.
Telegram alerts fire on IRP submission sent, invoice received (billed), and
un-parseable replies (so you can read + enter the fee manually).
At-cost services (IRP/IFTA/intrastate) only collected our service fee at
checkout; the variable state fee was never billed, so orders stalled at
authorization_signed and the filing card would have had to front large IRP fees.
New end-to-end, hands-off flow (you only approve the final filing):
1. After authorization is signed, state_trucking auto-estimates the gov fee
from intake (base/op states, power units, weight) via gov_fee.estimate_gov_fee.
2. Creates a CHILD compliance order (CG-..., service_fee=0, gov_fee=estimate,
parent_order_number set, migration 099) that flows through the EXISTING
checkout/payment/webhook machinery.
3. Emails the customer a payment link to /order/pay (new self-contained page)
showing every method with correct surcharges — ACH 0% (Stripe 0.8%/ cap
absorbed, no GoCardless needed), card/PayPal 3%, Klarna 6%, crypto 0%.
4. Order holds at awaiting_government_fee_approval until paid.
5. On payment, handlePaymentComplete detects the child (parent_order_number)
and re-dispatches the PARENT with gov_fee_paid=true, which proceeds to
prepare + queue the filing and stops at ready_to_file for your approval.
IRP fees are estimates billed at cost (refund overage / rebill shortfall); IFTA
decals + most intrastate fees are near-exact. Tunable via env.
relay_integration.py line 34 called logging.getenv (no such attr), which threw
AttributeError on import -> load_card_from_erpnext() crashed for every caller
(BOC-3 and now UCR filing payment). Drop the bogus line; LOG is set correctly on
the next line. Present since the initial commit.
Adds scripts/workers/services/ucr_playwright.py — a UCR.gov National Registration
System automation that, given a USDOT + fleet size, runs the register/pay flow,
pays the federal UCR fee with the matched PW filing card (Relay/Stripe Issuing),
and captures a confirmation screenshot + number. Conventions match
boc3_playwright / fmcsa_web_submitter: dev-mode dry-run guard, undetected
(patchright) browser, CAPTCHA detection, screenshot evidence, dataclass result.
Safety: verifies the displayed fee against the federal schedule before paying and
refuses to auto-charge a surprising amount (UCR_MAX_AUTO_FEE_USD) — falls back to
manual filing instead.
Wires it into MCS150UpdateHandler: when an approved (admin_approved) order has
slug ucr-registration, _file_ucr_registration runs the automation, uploads the
confirmation screenshot to MinIO, records filing_status + confirmation, and sets
fulfillment_status=completed on success. On CAPTCHA / fee-mismatch / failure it
reverts to ready_to_file with a high-priority 'file manually' todo. This replaces
the old behavior where approving a UCR just sat at authorization_signed.
UCR (and other admin-assisted DOT services) route through MCS150UpdateHandler,
which hardcoded 'MCS-150' and self.SERVICE_SLUG in the admin todo, the Telegram
fulfillment notification, and the customer status email -- so approving Paul's
UCR produced an 'MCS-150 Review / mcs150-update / PDF: not generated' alert and
an 'MCS-150 biennial update' customer email, both wrong.
Add SERVICE_DISPLAY_NAMES + _service_label(slug); use the actual slug everywhere.
Admin-assisted services now show 'UCR Annual Registration — FILE NOW ... file
manually on the portal (no auto-generated form)' instead of MCS-150/PDF wording,
and the customer email names the right service.
Admin-assisted DOT services (UCR, BOC-3) routed to this handler were marked
ready_to_file with whatever intake existed -- e.g. a UCR with only a DOT number,
missing legal name / state / fleet-size bracket (which sets the UCR fee tier).
That made the admin 'ready to file' status dishonest and unfileable.
Now, for ADMIN_ASSISTED_REQUIRED services we first enrich intake from the FMCSA
census (legal_name, address_state, power_units) + the order email, and derive
the UCR fleet_size_bracket from power units (UCR_FLEET_BRACKETS). If every
required field is then present we persist it and mark intake validated (falls
through to the admin review gate -> ready_to_file). If anything is still
missing, we persist what we have, set fulfillment_status=awaiting_intake, and
email the customer to complete intake -- instead of falsely showing ready_to_file.
Two of our three real paid customers (Mark Adams / mark@adamslumber.com and
Paul Wilson / synthetic@pipeline.com) never completed intake. They each hit the
old hard cap of 10 daily reminders (last sent Jun 12 / Jun 13) and the worker
then went permanently silent -- the last two daily runs reminded 0 orders even
though both still owe us intake on paid work. (The third, mitchell allen /
mitchell@allenscrapmetal.com, did complete intake; his orders are dispatched.)
Replace the dead-stop cap with a two-phase cadence:
- daily for the first DAILY_PHASE (10) nudges -- the initial burst,
- then weekly (WEEKLY_INTERVAL_DAYS) up to an absolute MAX_REMINDERS (60),
so a paid order with missing intake keeps getting a gentle nudge instead of
being dropped. Tunable via INTAKE_REMINDER_DAILY_PHASE /
INTAKE_REMINDER_WEEKLY_INTERVAL_DAYS / INTAKE_REMINDER_MAX. Clearing
intake_reminder_last_at re-arms an order immediately (documented in the
module docstring).
David Sgro (PA OAG complaint BCP-26-05-025816) opted out 2026-04-13; response
emailed to the AG 2026-06-11. To make the suppression bulletproof and keep the
response's representations true:
- Added a legal do-not-contact list (DO_NOT_CONTACT_DOMAINS/_EMAILS) to
_email_exclusions.py with dataspindle.com / dave@dataspindle.com; folded into
BLOCKED_EMAIL_DOMAINS and is_blocked().
- listmonk_import.upsert_subscriber now refuses to import/re-confirm any
suppressed address. This closes the exact gap that re-added him on 2026-04-26:
the duplicate-import branch re-added an existing unsubscribed subscriber to
lists with status=confirmed, overriding the opt-out.
- MCS150UpdateHandler is the catch-all for many admin-assisted DOT services
(UCR, MC authority, audit prep, ETA, name reservation, registered agent,
annual report). It was filling an MCS-150 PDF for ALL of them -- e.g. a UCR
order produced a wrong MCS-150 PDF. Now only MCS150_FORM_SLUGS fill the form;
others get an admin-review todo (PDF 'not generated') for manual handling.
Signature flow was already correctly scoped (UCR is not in DOT_SIGNING).
- handle_process_compliance_service forced the Sales Order workflow_state to
'Review' via set_value, which bypasses ERPNext's allowed transitions and
threw WorkflowPermissionError (Received -> Review) on every run. The Postgres
fulfillment_status is the source of truth; the ERPNext workflow_state is a
cosmetic mirror. Now try the proper apply_workflow action and stay quiet
(debug, not warning) when no valid Review transition exists.
We no longer offer Canadian accountant/accounting setup. Removed all
service-offering content:
- Marketing page (services/telecom/canada-crtc): the 'Set Up Canadian
Accounting (we help)' next-steps card, the '3 hours of complimentary
accounting consultation' deliverable bullet, and the whole 'Accounting
Support' section (assigned accountant, portal chat, $75/hr, 3 complimentary
hours).
- Order page (order/canada-crtc): the '3 hrs Canadian accounting support'
included-feature bullet and the 'Preferred accounting software'
(Xero/QuickBooks) form field + its accounting-hours helper text.
- Fulfillment (canada_crtc.py): dropped the bank-setup email line offering
'3 hours of Canadian accounting consultation'.
Kept factual GST/HST tax advisories and the bank's QuickBooks/Xero
transaction-sync feature (third-party bank capability, not our service).
MCS150UpdateHandler is the catch-all for many admin-assisted DOT services
(UCR, MC authority, audit prep, ETA, name reservation, registered agent,
annual report). My new intake-completeness gate was firing the 'confirm your
MCS-150 details' email for ALL of them -- e.g. a UCR order wrongly emailed the
customer about MCS-150 details. Scope the gate to MCS150_FORM_SLUGS (the
services that actually file an MCS-150: mcs150-update, dot-registration,
usdot-reactivation, dot-full-compliance).
Paul Wilson (Compound Technologies) signed up with synthetic@pipeline.com,
which is a genuine, deliverable EarthLink address (pipeline.com MX ->
earthlink-vadesecure.net; he confirmed receipt by phone). Our code had
hardcoded pipeline.com + the synthetic@ prefix as a 'non-deliverable
FMCSA-census placeholder' and silently suppressed every automated email to
him (checkout provisioning, order-creation validation, intake reminders,
set-password invites). Nothing in the codebase actually generates that
address, so the placeholder rationale was wrong. Removed pipeline.com and the
synthetic@ rule from all four suppression sites; only RFC-reserved
example.com/test.com/invalid remain blocked.
Closes the data gap for orders that bypass the full intake (e.g. the DOT
compliance-remediation pipeline) and for all MCS-150 variants:
- Worker intake-completeness gate (mcs150_update): before filling, check the
customer-required operational fields the FMCSA census cannot supply
(operation classification, cargo, CURRENT annual mileage, email; plus
signer/address for new-registration/reactivation, and states-of-operation
for 150B hazmat). If missing, email the customer a census-pre-filled intake
link and hold the order at fulfillment_status='awaiting_intake' with an admin
todo, instead of fabricating a blank filing. The existing intake PUT endpoint
already re-dispatches the worker on submit, so filing auto-resumes.
- Intake wizard (Wizard.astro): when resuming ?order=CO-xxx for a DOT/MCS order,
seed still-empty fields from the FMCSA census (name/address/fleet/interstate)
so the customer only confirms the operational details.
- /api/v1/dot/census now also returns total_drivers + a normalized
carrier_operation_code for the prefill.
- MCS150Step.astro extended to collect every field the filler needs across all
variants: mailing address, cdl_drivers, primary_vehicle_type,
reason_for_filing, usdot_revoked, cell/fax, hazmat-safety-permit block
(needs_hmsp, operating states, security plan), and intermodal-equipment
provider counts; all prefill from intake_data.
verify_mcs150_variants.py covers 150/150B/150C end-to-end (ALL PASS).
Fixes a batch of missing fields the FMCSA census does not provide and the
filler was mis-mapping:
- Corrected the question->field mapping to match the actual form: Q22 =
COMPANY OPERATIONS (interstate/intrastate, 22xBox), Q23 = OPERATION
CLASSIFICATIONS (for-hire/private/govt, 23xBox). These were swapped, and
the bogus entity-type->23xBox map (no entity-type question exists on this
form revision) was removed.
- Added proper radio-group handling for Reason for Filing (Biennial Update),
Mailing-address (Same as principal vs below), and Q28 USDOT-revoked, with
correct option indices (these are /0../n radios, not /Yes checkboxes; the
old code set them to /Yes and never selected the right option).
- Map interstate/intrastate from the FMCSA census carrierOperationCode, and
populate email/phone/mileage/cargo from intake.
- AcroForm checkbox/radio appearances use a ZapfDingbats glyph that
poppler/Preview fail to render (value set but box looks empty). Now stamp
an explicit X overlay into the page content for every 'on' box so it shows
in every viewer and in the faxed output.
handle_process_compliance_service assumed handlers return local temp
paths and re-uploaded each to MinIO. The MCS-150 handler uploads itself
and returns the MinIO key, so the re-upload tried to read a nonexistent
local file and logged a 'File not found' error after the order was
already correctly held at the admin gate. Now we skip files that don't
exist locally and keep the returned key as-is.
- When intake lacks signer_name, backfill it from the name the client
typed when signing the perjury certification (that name is exactly what
belongs in the form's print/type-name field, certifyName).
- After a client-approved re-dispatch, re-point the signed esign record at
the freshly filled form and re-stamp the signature, so the signed PDF an
admin reviews reflects the current complete form (not a stale earlier
fill). Field layout (and thus signature anchors) is unchanged across
fills, so the recorded anchor coordinates stay valid.
The MCS-150 biennial update re-confirms the carrier's existing FMCSA
record. Previously the PDF filler only had whatever the intake form
collected; rescued/sparse orders (or orders where the carrier's data
lives in FMCSA, not the intake) produced near-empty forms. Now we pull
the carrier census (legal name, address, EIN, fleet counts) from the
FMCSA carrier API and merge it under any customer-provided intake values
(customer edits win), so the form is pre-filled with the carrier's
current registered data. Refactored the FMCSA fetch into a shared
_fetch_fmcsa_carrier helper used by both enrichment and status check.
When an MCS-150/USDOT order hits the pre-submission admin-verification gate, the
Telegram FULFILLMENT NEEDED alert now appends a presigned link to the prepared
PDF (via the public minio.performancewest.net endpoint, IP-allowlisted to admin)
so you can review the document straight from the alert before approving. Added
notify_fulfillment_todo(view_url=...) + a _presigned_view_url helper (public
endpoint + explicit region to avoid the region-probe that 403s from the worker).
The MCS-150/USDOT PDF was generated fine but the MinIO upload threw 'Minio object
has no attribute fobj_put' (wrong method name + signature), so the prepared filing
PDF was never persisted -- nothing for an admin to review at the verification gate,
and the esign-completed re-dispatch failed with 'File not found'. Use the correct
minio fput_object(bucket, key, file_path). Affects every MCS-150/USDOT filing.
Per request: after the customer signs but BEFORE we submit to the government, hold
the order for a human to verify the prepared filing is correct.
- MCS-150 handler (mcs150-update + usdot-reactivation): new admin-verification gate
after the signature gate -- if not admin_approved, set fulfillment_status=
'ready_to_file', create a HIGH-priority 'VERIFY before filing' admin todo, and
STOP (no FMCSA submission). job_server injects admin_approved from the dispatch
payload (mirrors client_approved).
- New admin endpoint POST /api/v1/admin/compliance-orders/:id/approve-submit
(requireAdmin): verifies status=ready_to_file, re-dispatches the worker with
admin_approved=true to proceed to actual submission.
- Durable submission EVIDENCE: the web/fax submitters only wrote confirmation
screenshots to an ephemeral temp dir. Now _upload_submission_evidence copies the
FMCSA confirmation screenshot + attested PDF + fax_log_id to MinIO under
filings/<slug>/<order>/evidence/ and records the keys on the order, so we keep
proof of every government submission.
(state-trucking + the FCC handlers already gate via admin todos / auto_filing.py;
this brings MCS-150 to parity and adds evidence retention.)
Two bugs found tracing Mitchell Allen's batch CB-95BA6C90 (5 DOT services, card):
1) Worker authorization/signing-link/status emails were sent via
smtplib.SMTP('localhost', 25), which has no MTA in the workers container ->
every send failed '[Errno 111] Connection refused', so customers never got
their e-sign links and orders sat 'awaiting client signature' forever. Routed
all 9 hardcoded localhost:25 sites (state_trucking, mcs150_update, boc3_filing,
hazmat_phmsa, mailbox_setup, dot_esign, completion_emails) through the
authenticated SMTP relay (SMTP_HOST/PORT/STARTTLS/login) + added a shared
worker_email.send_worker_email helper.
2) The ERPNext Sales Order for compliance/compliance_batch was only created in
the /checkout/create-session endpoint, but CARD orders confirm via the Stripe
WEBHOOK -> handlePaymentComplete, which never created the SO. Result: every
webhook-confirmed order had erpnext_sales_order=NULL and workers logged
'Sales Order not found 404' then built from PG. Added idempotent
ensureComplianceSalesOrder() to handlePaymentComplete so ALL payment methods
(card-webhook, PayPal, crypto) create + link the SO.
Two latent bugs the e2e harness caught:
1. api entities.ts GET /states/:code/name-search calls WORKER_URL/name-search,
but job_server had NO such route -> 404 -> silently fell back to stale
entity_cache on every live name check. Added a synchronous /name-search route
returning {available,exact_match,similar_names,state}.
2. both the new route AND the existing async handle_name_search imported a
nonexistent search_name_sync(); fixed to drive the real async search_name()
via an event loop (same pattern as /entity-status).
scripts/e2e-formation-order.mjs: replays the real formation order flow (live
name search -> formation_orders insert -> ERPNext customer + Sales Order with
BUSINESS-FORMATION + STATE-FILING-FEE line items -> verify SO total + DB linkage
-> cleanup) without a real Stripe charge or state filing. Run in the api container.
Also created the missing ERPNext Items (BUSINESS-FORMATION, STATE-FILING-FEE,
FOREIGN-QUAL-SINGLE/MULTI) that the formation SO references.
Follow-on to the trucking new-carrier slug fix:
- success page: add dot-new-carrier-bundle to DOT_SLUGS + NEW_CARRIER_SLUGS so
the order-confirmation 'what to expect' messaging classifies it as trucking.
- pipeline_orchestrator: the trucking onboarding PIPELINE was keyed under the
bare 'new-carrier-bundle' slug, which is the TELECOM bundle's slug (also a
collision at the worker layer). Re-keyed to 'dot-new-carrier-bundle' so a
trucking bundle never runs the telecom pipeline (and vice versa).
Client-facing and website code now describes only a generic per-document signing
authorization; nothing visible to signers or recorded in the website/API code or
DB schema references ink, paper, reproduction, or any fulfillment mechanics.
- rename esign-ink-consent.ts -> esign-sign-consent.ts; INK_CONSENT_TEXT ->
SIGN_CONSENT_TEXT (generic: 'use my signature to complete and submit this
single filing', no ink/paper/reproduce language); helpers ink* -> sign*
- portal-esign-generic.ts: API field ink_reproduction -> require_sign_consent,
ink_consent_text -> sign_consent_text, request field ink_consent -> sign_consent
- signing page (site/public/portal/esign): all ids/vars/comments ink* -> sign*;
no 'ink' string remains
- npi_provider metadata flag ink_reproduction -> require_sign_consent
- migration 090/092 + live DB column comments rewritten to drop ink/plotter
wording (DB column names kept as ink_consent* for compat, internal only)
- order-timeline.ts buffer comments neutralized
- tests: 37 checks, consent text asserted to omit ink/plotter/paper/reproduce/etc
DB columns ink_consent* retained (internal, never sent to clients) to avoid a
risky rename of already-applied prod columns.
Consent gate (the legal linchpin from the wet-signature memo):
- migration 092 adds ink_consent/ink_consent_at/ink_consent_text to esign_records
- extract pure, unit-tested gate logic into esign-ink-consent.ts (DRY single
source for route + signing page): isInkReproduction / inkConsentRequired /
inkConsentSatisfied + verbatim client-safe INK_CONSENT_TEXT
- portal-esign-generic.ts: GET surfaces ink_reproduction + consent text; POST
gates DRAWN signatures on ink-path docs on explicit consent, stores it
- signing page locks the signature block until consent is checked (drawn only)
- npi_provider marks cms855/cms10114 esign metadata ink_reproduction=true
- 33 unit checks: gate truth table + consent text omits all internal mechanics
(plotter/machine/CMS/MAC/etc) and keeps required legal reassurances
Patent-risk memo (docs/legal/patent-risk-mechanical-wet-signature.md):
- prior-art-dated risk analysis (autopen 1803/1942, plotters, CNC = public domain
=> low risk on core concept; e-sign workflow space litigious)
- firsthand recent-grant sweep (1.58M USPTO grants 2021-2025, queried via DuckDB):
ZERO patents on machine-applies-signature-in-ink; e-sign players hold only
electronic-workflow patents. Not an FTO; flags where attorney search is needed
Adds a second machine class (small fan-shaped reach arm) alongside the
CR-10/AxiDraw rectangular-bed plotters, so wet signatures can be produced
while away from the home station.
ink_signature_plotter.py:
- PlotterConfig gains dialect (marlin|lineus) + name; new LineUsConfig
(native units, pen height = per-move Z, reach annulus from shoulder pivot).
- Named machine profiles (cr10 default, axidraw, lineus) via load_profile().
- bed_mm_to_lineus_units(), check_reach() (annulus for lineus, rectangle for
marlin), compute_jig_offset_for_box() (solves jig from the ACTUAL fitted ink
extent so a wide cell line doesn't over-constrain a small arm).
- emit_gcode() dispatches to emit_marlin_gcode()/emit_lineus_gcode().
- send_lineus(): WiFi TCP 1337 (NUL-terminated, ok-acked) or USB serial,
dry_run=True default (same gating as the CR-10 path).
ink_signature_cli.py: --profile, --solve-jig (auto-applies jig offset),
--lineus-host/--lineus-usb, reach-check that refuses to --plot out-of-reach
on Line-us.
Tests: 43 checks (was 30) covering profiles, reach check, jig solve, lineus
emitter, dry-run sender. Docs updated with profiles + portable workflow.
Telegram notifications:
- Add shared scripts/workers/telegram_notify.py (send_telegram, notify_fulfillment_todo,
create_admin_todo) so every worker alerts the operator the same way; fire-and-forget.
- Fire notify_fulfillment_todo after each admin_todos insert across all 8 service
handlers (9 sites) so no fulfillment task waits unseen.
(Orders + quotes + tickets already notified via checkout/quotes/tickets routes.)
Client portal order progress:
- order-timeline: derive real per-step status from live signals (payment paid,
e-signature signed, fulfillment_status) instead of a static template; add
current_step to the response.
- Extract pure applyLiveStatus into order-timeline-status.ts (DB-free) + unit test
(api/test/test_timeline_status.ts, 8 cases).
- portal /me now returns compliance_orders.fulfillment_status.
- Dashboard renders a client-safe Progress badge (In progress / Action needed /
Filed-awaiting-confirmation / Completed); batches show the most actionable status.
No back-office mechanics exposed.
ERPNext sync parity:
- Create a Sales Order for formation and fcc_carrier_registration orders (previously
only canada_crtc + compliance synced); write erpnext_sales_order back to each table.
Non-blocking, matches existing pattern.
Verified: API tsc clean, timeline unit tests 8/8, Astro build 58 pages,
cms10114/ink/paper_batch Python tests still green, no mechanics leaks.
The Standard no-login CMS path needs an ORIGINAL ink signature on paper
(CMS-10114: 'Stamped, faxed or copied signatures will not be accepted'). This
adds a pipeline to redraw the provider's own captured strokes in real ink with a
pen on a CR-10 V2 (or any Marlin/GRBL machine) — original, in ink, never copied.
- migration 090: esign_records.signature_vector (JSONB stroke paths, 0..1).
- signing page now captures normalized stroke paths alongside the PNG; API
stores a size-bounded vector for drawn signatures.
- ink_signature_plotter.py (hardware-independent): fit strokes to the signature
anchor box, PDF-pt -> bed-mm via jig offset, emit Marlin/GRBL G-code (Z pen or
M280 servo/BLTouch), SVG toolpath preview, and render_signature_on_pdf (a
digital twin that proves the toolpath lands on the cert line). Gated serial
sender (dry_run default).
- ink_signature_cli.py: end-to-end load-record -> gcode+preview, --test-box jig
calibration, --plot to stream over USB.
- Corrected CMS-10114 signature anchor to sit inside the Section 4A signing cell
(above the bottom rule, below the label).
- docs/ink-signature-plotter.md documents the CR-10 retrofit + interpretive risk.
Tests: test_ink_signature.py 30/30, test_cms10114.py 27/27, test_paper_batch.py
15/15, API tsc clean, Astro build 58 pages.
Verified firsthand against the live CMS-10114 (Rev. 02/25, OMB 0938-0931):
- Section 1A confirms paper is valid for Change of Information (#2) AND
Reactivation (#4), not just initial enumeration. Resolves the UNCERTAIN flag.
- Current mailing address is CMS NPI Enumerator Services, Mail Stop DO-01-51,
7500 Security Blvd, Baltimore MD 21244. The old Fargo PO Box 6059 is retired;
corrected in mac_routing.NPI_ENUMERATOR + all docs.
- No electronic no-login equivalent exists for CMS (NPI Registry API is
read-only; PECOS/NPPES-IA require login), unlike FMCSA's ask.fmcsa ticket form.
So tiers stay: Standard=paper CMS-10114 (no login), Expedited=NPPES surrogate.
New: cms10114_pdf_filler.py fills the flat official form via text overlay
(reason checkbox + NPI + Section 2A identity + Section 4A cert name + signature
anchor); wired into npi_provider._generate_10114_for_signing for nppes-update.
Signed forms route to the NPI Enumerator via the existing daily batch.
Tests: test_cms10114.py 27/27, test_paper_batch.py 15/15, Astro build 58 pages.
- NpiIntakeStep: add positively-framed 'can you grant electronic I&A Surrogate
access?' question for all filing slugs (reval/reactivation/nppes-update/
enrollment/bundle). Optional, never required, never mentions paper; captured
as intake_data.surrogate_access (yes/no/blank). Astro build green (58 pages).
- npi_provider.py: surface the surrogate answer in the admin todo so fulfillment
knows EXPEDITED (online via surrogate) vs STANDARD (e-sign + daily mail batch).
Standard (no-login) CMS filings are mailed in one Priority Mail envelope per
destination agency, batched each postal working-day morning to save postage.
- migration 089: paper_filing_batches table + esign_records.paper_batch_id /
filing_destination_key (idempotent: a filing is batched at most once).
- batch_cover_sheet.py: per-agency cover sheet (sender/dest/date/manifest) +
merged print-job PDF (cover + all enclosed signed filings).
- daily_paper_batch.py worker: gather signed+unbatched cms855/cms10114 filings,
group by destination (MAC by state via mac_routing; Fargo for CMS-10114),
build cover+merged PDF per agency, persist batch, mark filings batched.
Self-gates on postal working days (skips weekends + federal/USPS holidays).
Phase 1 = human prints+mails; phase 2 = wire print-mail API.
- worker-crons: pw-paper-batch systemd timer (Mon-Fri 13:30 UTC, self-gated).
- test_paper_batch.py: 15/15 pass (working-day gating, routing, cover+merge).
- Verified Standard(no-login)/Expedited(surrogate) matrix from official CMS-855
PDFs (docs/healthcare-filing-tiers-verified.md): reactivation+revalidation are
855I paper-to-MAC reasons, original-signature, routed by state; sig may not be
delegated; 855B needs PECOS app fee.
- Add scripts/workers/mac_routing.py: state->MAC routing (all 56 jurisdictions,
12 destinations) for envelope addressing + daily batch grouping. Addresses
marked VERIFY before live mail.
- npi_provider.py: fix access strings to two-tier framing; NPPES update/reactivation
no longer 'online-only'; note 855B fee.
- checkout.ts + service pages: strip client-facing mechanics & the paper-vs-tier
choice; surrogate is the only optional, positively-framed ask (faster, never
required, never share password).
The residential proxy password contains a '#', which urlparse() misreads as a
URL fragment and corrupts the port (ValueError: Port could not be cast...).
Parse scheme://creds@host:port manually and percent-decode user/pass so both
raw ('#') and encoded ('%23') passwords work. Verified against the live
credential.
CMS healthcare portals (NPPES, PECOS, I&A) block datacenter IPs, so the
healthcare browser automation needs to egress via the residential proxy on
hg409y7ez04.sn.mynetname.net (username 'performancewest').
- undetected_browser: use_proxy now accepts an env-var name, so callers can
select a domain-specific proxy. _proxy_config(proxy_env) reads it and falls
back to UNDETECTED_PROXY_URL. Healthcare uses 'HEALTHCARE_PROXY_URL'.
- probe_npi_undetected: launches with use_proxy='HEALTHCARE_PROXY_URL' when set.
- npi_provider: documents that the (future) automated NPPES/PECOS flows must
use the healthcare proxy.
- Plumb HEALTHCARE_PROXY_URL (+ UNDETECTED_PROXY_URL fallback) through the
ansible env template and docker-compose workers env.
The credential itself is NOT in the repo. Set the full URL in the ansible
vault as vault_healthcare_proxy_url:
socks5://performancewest:<password>@hg409y7ez04.sn.mynetname.net:<port>
Verified parsing + Playwright proxy-dict wiring with a unit test.
Copy: drop paper/electronic/fax framing across the revalidation + enrollment
marketing pages and the order-confirmation email; present two service tiers:
- Standard filing (no CMS account; we prepare CMS-855, you sign, we submit to MAC)
- Expedited filing (CMS I&A surrogate access; same-day PECOS filing + tracking)
Internal worker todos + the _STANDARD_FILING_SLUGS identifier updated to match.
New scripts/test_healthcare_e2e.py validates the whole order line (slug
consistency x6 places, price agreement, intake field collection+enforcement,
worker dispatch, handler execution producing CMS-855 PDF+anchor, free-tool
action_urls). 45 checks.
Bugs found + fixed by the test:
- medicare-enrollment requires practice_state server-side but the wizard never
enforced it -> orders could be paid then stall. Wizard now requires it.
- determine_form_type defaulted org NPIs to the individual 855I because
enumeration_type is never collected -> wrong form, CMS rejection. Now does a
live NPPES lookup (safe 855I fallback).
- cms855_pdf_filler.py: fills official CMS-855I/B/O/A AcroForms from intake
(name, NPI, DOB, cert-page printed name) and records the signature anchor at
the form's official /Sig box so the e-sign stamper lands on the cert line.
- npi_provider handlers (revalidation/reactivation/enrollment) now generate the
paper CMS-855, upload it to MinIO, request_esign with anchors, and email the
signing link. Human completes/verifies + USPS Priority Mails to the MAC.
- scripts/Dockerfile: copy the official CMS-855I/B/O/A forms into the image.
- order-confirmation email presents both filing methods: paper CMS-855 (no
account needed, client e-signs one page, we print+mail to their MAC) and
I&A surrogacy (faster, needs CMS account). NPPES-only services note that
surrogacy is required (web-only).
- npi_provider handlers record the access model per service in admin todos.
- marketing copy leads with the lowest-friction paper option.
Adds a systemd-timed worker that nudges customers who paid but never completed
their intake form (which stalls fulfillment).
- migration 087: intake_reminder_count + intake_reminder_last_at on
compliance_orders (makes the daily run idempotent and bounded), plus a
partial index for the paid-order eligibility scan.
- scripts/workers/intake_reminder.py: each run emails any paid order with
intake_data_validated != TRUE, capped at 10 reminders/order, at most one
consolidated email per customer per day (groups a customer's incomplete
services into one email). Reuses the post-payment intake URL format
(/order/{slug}?order={n}) and the API's email validation, skipping
placeholder/invalid addresses (synthetic@, pipeline.com, etc.). Sends via
smtplib with SMTP_PASS (verified working in the worker container).
- worker-crons: pw-intake-reminder timer, daily ~noon ET (16:00 UTC).
