Adds scripts/workers/services/ucr_playwright.py — a UCR.gov National Registration
System automation that, given a USDOT + fleet size, runs the register/pay flow,
pays the federal UCR fee with the matched PW filing card (Relay/Stripe Issuing),
and captures a confirmation screenshot + number. Conventions match
boc3_playwright / fmcsa_web_submitter: dev-mode dry-run guard, undetected
(patchright) browser, CAPTCHA detection, screenshot evidence, dataclass result.
Safety: verifies the displayed fee against the federal schedule before paying and
refuses to auto-charge a surprising amount (UCR_MAX_AUTO_FEE_USD) — falls back to
manual filing instead.
Wires it into MCS150UpdateHandler: when an approved (admin_approved) order has
slug ucr-registration, _file_ucr_registration runs the automation, uploads the
confirmation screenshot to MinIO, records filing_status + confirmation, and sets
fulfillment_status=completed on success. On CAPTCHA / fee-mismatch / failure it
reverts to ready_to_file with a high-priority 'file manually' todo. This replaces
the old behavior where approving a UCR just sat at authorization_signed.
UCR (and other admin-assisted DOT services) route through MCS150UpdateHandler,
which hardcoded 'MCS-150' and self.SERVICE_SLUG in the admin todo, the Telegram
fulfillment notification, and the customer status email -- so approving Paul's
UCR produced an 'MCS-150 Review / mcs150-update / PDF: not generated' alert and
an 'MCS-150 biennial update' customer email, both wrong.
Add SERVICE_DISPLAY_NAMES + _service_label(slug); use the actual slug everywhere.
Admin-assisted services now show 'UCR Annual Registration — FILE NOW ... file
manually on the portal (no auto-generated form)' instead of MCS-150/PDF wording,
and the customer email names the right service.
Admin-assisted DOT services (UCR, BOC-3) routed to this handler were marked
ready_to_file with whatever intake existed -- e.g. a UCR with only a DOT number,
missing legal name / state / fleet-size bracket (which sets the UCR fee tier).
That made the admin 'ready to file' status dishonest and unfileable.
Now, for ADMIN_ASSISTED_REQUIRED services we first enrich intake from the FMCSA
census (legal_name, address_state, power_units) + the order email, and derive
the UCR fleet_size_bracket from power units (UCR_FLEET_BRACKETS). If every
required field is then present we persist it and mark intake validated (falls
through to the admin review gate -> ready_to_file). If anything is still
missing, we persist what we have, set fulfillment_status=awaiting_intake, and
email the customer to complete intake -- instead of falsely showing ready_to_file.
Two of our three real paid customers (Mark Adams / mark@adamslumber.com and
Paul Wilson / synthetic@pipeline.com) never completed intake. They each hit the
old hard cap of 10 daily reminders (last sent Jun 12 / Jun 13) and the worker
then went permanently silent -- the last two daily runs reminded 0 orders even
though both still owe us intake on paid work. (The third, mitchell allen /
mitchell@allenscrapmetal.com, did complete intake; his orders are dispatched.)
Replace the dead-stop cap with a two-phase cadence:
- daily for the first DAILY_PHASE (10) nudges -- the initial burst,
- then weekly (WEEKLY_INTERVAL_DAYS) up to an absolute MAX_REMINDERS (60),
so a paid order with missing intake keeps getting a gentle nudge instead of
being dropped. Tunable via INTAKE_REMINDER_DAILY_PHASE /
INTAKE_REMINDER_WEEKLY_INTERVAL_DAYS / INTAKE_REMINDER_MAX. Clearing
intake_reminder_last_at re-arms an order immediately (documented in the
module docstring).
Main pool is calendar-day 12 but reputation is wrecked (54% delivery, Gmail+
Outlook blocks) -- NOT warmed. MX tagging confirmed the cause: 702k carriers on
Google + 135k on Microsoft = the warmup was hammering exactly the two operators
blocking us. Hold Google/MS/Proofpoint/etc. OUT entirely until day 30 (configurable),
sending only to the long-tail operators (yahoo/comcast/charter/centurylink/etc.)
that don't bot-throttle, so reputation can recover; then re-introduce big
operators gradually via mx_daily_caps. 1.24M/1.49M carriers now tagged.
The real bottleneck was the write, not DNS: each per-domain UPDATE full-scanned
fmcsa_carriers (no functional index on lower(split_part(email,'@',2))). Resolve
all domains concurrently into a list, load a temp table, then ONE join-UPDATE =
single table scan. Tags ~12k domains -> hundreds of thousands of carriers fast.
The serial path (verifier's 8s+6s lifetime per domain) was far too slow for
bulk tagging -- 0 tagged in 3 min on dead domains. Self-contained fast resolver
+ ThreadPoolExecutor(40) resolves thousands of domains in minutes.
The Jun 13-14 Gmail+Outlook block storm came from the main/trucking pool having
NO per-MX throttling (only HC had it) -- it concentrated warmup volume on
Google/Microsoft-Workspace-hosted business domains. Port the HC fix:
- migration 097: fmcsa_carriers.mx_provider column.
- mx_tag_carriers.py: resolve MX once per distinct domain (reuses the verifier's
classifier+cache), tag every carrier with that domain's operator. Bounded per
run, prioritizes unsent verified carriers.
- build_trucking_campaigns: during warmup (day<=6) EXCLUDE tagged Google/MS/
Proofpoint/etc. carriers in fetch_carriers; per-MX cap in select_sendable_
carriers so known operators never dominate the quota. Untagged carriers pass
(not collapsed onto one bucket) until tagging fills in. mx_daily_caps ramps
with the main warmup day; MAIN_SKIP_BIG_MX=0 disables once warmed.
The HC warmup crons were '* * 1-5' (Mon-Fri), silently skipping weekends -- but a
proper warmup needs CONTINUOUS daily volume for 21 days (mailbox providers reward
consistency; gaps stall reputation). The Jun 14 'HC 0 sent' alert was just a
skipped Sunday, but the weekend skips also broke ramp continuity.
- pw-hc-campaign + pw-hc-nppes: '* * 1-5' -> '* * *' (daily), vendored + applied live.
- Re-aligned the warmup start stamp from calendar-day 9 to send-day 5 so the
volume ramp matches reputation actually built (it had skipped ~4 weekend days,
running the ramp ahead of real history).
- Fixed the stale 'Mon-Fri only' comment in daily_slice().
- Vendored nppes cron now carries the enriched-CSV + 4-segment config.
otc_reincorporation.html: redomesticate-to-Texas hook (Business Court + TXSE +
DE franchise-tax cost) personalized by state_inc_name/company/ticker, cross-sell
RA/foreign-qual/annual-report/franchise-tax, same-day coupon, lead-capture CTA to
/contact?service=reincorporation (high-touch corporate service, not self-serve),
careful 'not a law firm / not legal advice' disclaimers + CAN-SPAM address.
build_otc_campaign.py: emails only verified-email issuers from the harvest+scrape
+verify pipeline, --de-nv-only for the best reincorp fit, reuses trucking sender
plumbing + coupon. Per-deal value is high so capped modestly (400/run default).
scrape_otc_emails.py: fetch each issuer domain's IR/contact pages (gzip,
HTML-only, early-abort, prefer ir@/investor@/info@), extract a contact email.
Skip filing-agent domains (DFN/Donnelley/Broadridge/etc.) that leak into the
extracted domain -- those are not the issuer's site. Same filter added to the
harvester's DOMAIN_NOISE for future runs. Phone (100%) is the fallback channel
for email misses.
Pilot -> production: harvest_otc_issuers.py pulls the OTC/None universe (2,771),
keeps US-domestic (requires BOTH a US state-of-incorporation AND a US-state
business address -- disambiguates the 'DE'=Delaware-vs-Germany trap that leaked
Infineon etc.), and extracts each issuer's website DOMAIN directly from its
latest 10-K/8-K/DEF-14A filing (free, no scrape; ~58-60% find rate in testing).
Outputs cik,name,ticker,state_inc,phone,city,state,zip,domain -- ready for the
domain->email scrape + verify step. Phone is 100% (clean fallback call channel).
Reincorporation-to-TX / RA / foreign-qual / franchise-tax / annual-report fit.
The OIG-screening + NPPES-update segments were effectively limited to ~1,437
providers because the warmup 'any' selector excluded not-on-reval-list rows as a
deliverability proxy -- but that excludes almost the ENTIRE institutional list
(org NPIs aren't individual Medicare enrollees). Since we already SMTP-verified
all 63k inboxes, add an 'institutional_verified' selector that trusts our own
verification instead of reval-list presence. Result: OIG + NPPES-update now
address 62,422 (43x more), giving multiple broad offers to test engagement on.
- enrich_institutional_revalidation.py: fast local join of the institutional
list to the CMS Revalidation Due Date List bulk file (revalidation_base.csv)
by NPI -> adds reval_due_date/days_overdue/reval_status. ~1,437 are genuine
Medicare enrollees (197 overdue / 164 due-soon) -> flagship $599 reval pitch.
- npi_reactivation stays on leie_or_deactivated (only REAL deactivations -- no
false 'your NPI is deactivated' claims to active orgs).
UCR (Unified Carrier Registration) is annual: opens Oct 1, due Dec 31, mandatory
for interstate carriers (op A, same ~628k pool as IFTA) -> recurring revenue.
- build_ucr_annual_campaign.py: 3-touch business-day cadence (30/12/4 bd before
Dec 31, wider than IFTA since it's once a year), escalating tone, same-day
coupon, 'I already did it' suppression. Reuses build_trucking_campaigns +
IFTA business-day/token helpers (DRY). Per-year cycle reset.
- ucr_annual_reminder.html: deadline + fines/OOS risk + 'we figure out your fee
tier' + coupon + filed link + CAN-SPAM. Source campaign 473.
- migration 096: ucr_reminded_at / ucr_touch_no / ucr_self_filed_at.
- ifta.ts: add GET /api/v1/ucr/filed (shares the HMAC token scheme).
- checkout.ts: order-placement Telegram now shows 'Source: campaign (code X)'
when a discount code is present, so IFTA/UCR/CLIA conversions are visible.
(Confirmed order-alert Telegram already fires from handlePaymentComplete for
all compliance orders via both webhook + session paths.)
The handler is an API route; performancewest.net/api/* proxies to the Astro site
(:4322), only api.performancewest.net serves the API. Build the filed link from
PUBLIC_API_URL (default https://api.performancewest.net). Verified end-to-end:
valid token -> 200 confirmation page, invalid -> 403.
- Multi-touch reminders at 10/7/4 BUSINESS days before each deadline (weekends
skipped; biz-day math so a touch never lands purely on a weekend with no
runway). Escalating tone soft -> urgent -> last-chance, with the 'almost too
late to DIY, we can still file it' angle so it's a convenience sale, not a free
reminder service. ifta_touch_no tracks the highest touch sent so each touch
hits only carriers below that level; never repeats a touch.
- 'I already filed it' one-click link: HMAC-tokenized GET /api/v1/ifta/filed
(token matches between Python builder and api/src/routes/ifta.ts -- verified
identical output), records ifta_self_filed_at, friendly confirmation page,
stops further touches this cycle + gives DIY-vs-prospect signal. Builder
excludes self-filed carriers.
- migration 094 (ifta_touch_no) + 095 (ifta_self_filed_at); cycle reset clears
both each new quarter. Verified: biz-day touch schedule, token cross-match.
add_subscriber only attached existing subscribers to the new list without
updating attribs, so a carrier emailed in a prior campaign kept STALE attribs --
meaning the daily coupon code and IFTA merge fields (ifta_due_date, ifta_quarter,
lp_link) rendered BLANK for any repeat recipient. Now merge+PUT the fresh attribs
on the existing subscriber before attaching. Affects all trucking campaigns, not
just IFTA. Verified: IFTA preview now persists ifta_quarter/ifta_due_date/lp_link.
IFTA returns are due on fixed dates (Apr30/Jul31/Oct31/Jan31) and every
interstate carrier (op code A, ~628k sendable) files 4x/year forever -- pure
recurring revenue, no per-carrier deadline data needed.
- build_ifta_quarterly_campaign.py: self-gates to the reminder window (~21d
before each deadline), selects interstate carriers, mints the same-day coupon,
builds+schedules the campaign reusing build_trucking_campaigns plumbing (DRY:
one source of truth for sending/suppression/coupon). Per-quarter cycle reset
(ifta_reminder_cycle marker) so each quarter re-reminds the full pool; marks
ifta_reminded_at to avoid double-sends within a cycle.
- ifta_quarterly_reminder.html: deadline + penalties + 'we do the math' + coupon
+ CAN-SPAM. Listmonk source campaign id 469.
- migration 094: fmcsa_carriers.ifta_reminded_at column + partial index.
Verified: deadline/window logic correct, imports reuse tc helpers, migration
applied on prod.
Set up the CLIA recurring-renewal vein (every clinical lab renews its CLIA cert
on a 2-year cycle; CMS publishes the full lab file with expiration dates):
- service-catalog: clia-renewal ($449, discountable) + order page (npi-intake
steps) + intake manifest entry.
- harvest_clia_renewals.py: parse the CMS Provider-of-Services CLIA file, filter
to labs expiring within a window (default 120d), emit name/address/phone/expiry.
676k labs -> ~70k expiring in the next ~4 months.
- match_clia_to_nppes.py: CLIA has no NPI/email, so bridge to emailable NPPES
orgs by normalized name+zip to recover NPI+email (yield TBD; labs that do not
match still have clean phone+postal for a phone/mail channel).
- hc_clia_renewal.html: warm turnover-safety-net email with the striped official-
record card (CLIA #, expiry, status), verify-on-CMS-QCOR, founder guarantee
card, full CAN-SPAM address.
- Upgrade the plain teal record banner to the authoritative barber-pole 'Official
record' banner in the personal/turnover/overdue-personal templates (the switch
to personal templates had dropped the striped look from live revalidation sends).
- nppes_outdated: replace plain info table with the striped 'Official record -
NPPES NPI Registry' card (status honestly labeled as our compliance flag).
- Wire revalidation_overdue -> hc_revalidation_overdue_personal.html with a direct
past-due subject ('Your Medicare revalidation is past due - let's get it filed')
and PAST DUE status + days-overdue in the record card; due_soon stays warm.
- Striped card now on all 7 templates that show a real record; oig_screening and
compliance_bundle correctly omit it (no specific record to display).
CAN-SPAM requires a valid physical postal address in every commercial email.
All 8 HC campaign templates and the FCC campaign_template.html only had
'Cheyenne, WY' (no street) -- added the full
'525 Randall Ave Ste 100-1195, Cheyenne, WY 82001' to match the (already-correct)
trucking templates. Audited every Listmonk source/sent campaign + wrapper
templates: all active sends carry address + unsubscribe.
Also: revalidation segments now use hc_revalidation_personal.html with subject
'Let's make sure your Medicare revalidation is handled in time'.
The sales we got came at $79 + a 24hr coupon; cutting MCS-150 to $39 flat
removed urgency and conversions did NOT improve (a permanent low price sets a
new anchor and lets people defer). Restore the higher anchor and let an
expiring discount create the now-or-lose-it decision.
- Restore MCS-150 anchor $39 -> $79 (catalog single source + regenerated).
- build_trucking_campaigns.py: mint ONE random 5-letter coupon per send-day
(40% off, valid through 23:59:59 ET that day) into the existing discount_codes
table; inject coupon_code/pct/expires + a ?code= LP link into every email.
Idempotent per day; service-fee-only scope (gov/pass-through fees never cut).
- Listmonk MCS-150 (186) + Inactive USDOT (188) templates: lead with the
struck-through anchor + sale price + code + 'expires tonight', and point the
primary CTA at the order page (with code) instead of the 'free check' tool.
- OrderPriceBanner: validates ?code= via /api/v1/discount and shows
was/now + expiry; Wizard forwards the code to order creation.
- Verified: code gen, expiry math, scope enforcement, discount API
(40% off $79 = $47.40), site+api builds clean.
Reputation is tracked per receiving mail operator, not per recipient domain, so
the daily warmup slice is now distributed across MX operators with per-operator
daily caps (ramping with the warmup day): Microsoft/Google/Proofpoint/etc. capped
individually, long-tail operators each get a generous default. This lets total
daily volume be much higher than a flat cap without hammering any single system.
mx_throttled() respects the mx_provider column the verifier now writes; falls back
to flat slicing if absent.
Reputation is tracked per receiving mail operator (Microsoft 365, Google
Workspace, Proofpoint, etc.), not per recipient domain -- so warmup can safely
send far more total volume if it's spread across many MX operators and throttled
per-operator. The verifier now classifies each domain's (already-cached) MX into
a provider label and writes an mx_provider column, so the warmup importer can
cap sends per operator per day. NPPES institutional sample distribution:
Microsoft 33%, Google 11%, Proofpoint ~16%, long tail across dozens of others.
Extracts cold-mailable provider inboxes from the NPPES endpoint_pfile, dropping
Direct/HISP gateway domains (not deliverable from a normal MTA). From the
June 2026 NPPES file: 88,728 institutional + 19,355 consumer mailable
candidates. Institutional is the warmup-safe slice (consumer webmail is held
back -- aggressive filtering would hurt the warming IP).
add_subscriber only attached an already-existing subscriber to the new list
without updating attribs, so the due-soon template's days_until merge field was
blank for providers already imported by another segment. Now PUT the merged
attribs (existing + this segment's npi/practice/due-date/days_until) before
adding to the list.
The HC warmup pool is supply-constrained (~400 verified providers, all fed by
the same narrow 'revalidation 1-90 days OVERDUE' slice). This adds a mirror-image
proactive segment that targets providers whose Medicare revalidation is UPCOMING
within the next 1-90 days, drawn from the same CMS Revalidation Due Date List --
no new data source needed. 'Handle it before your deadline' is a strong pitch and
roughly doubles the deliverable pool.
- New selector reval_due_soon (status=upcoming, days_until in [HC_DUE_SOON_MIN,
HC_DUE_SOON_MAX] default 1-90).
- New segment revalidation_due_soon reusing the existing /order/npi-revalidation
service ($599) with template hc_revalidation_due_soon.html.
- attribs_for now exposes days_until (positive days to due date).
- Added to ACTIVE_SEGMENTS.
David Sgro (PA OAG complaint BCP-26-05-025816) opted out 2026-04-13; response
emailed to the AG 2026-06-11. To make the suppression bulletproof and keep the
response's representations true:
- Added a legal do-not-contact list (DO_NOT_CONTACT_DOMAINS/_EMAILS) to
_email_exclusions.py with dataspindle.com / dave@dataspindle.com; folded into
BLOCKED_EMAIL_DOMAINS and is_blocked().
- listmonk_import.upsert_subscriber now refuses to import/re-confirm any
suppressed address. This closes the exact gap that re-added him on 2026-04-26:
the duplicate-import branch re-added an existing unsubscribed subscriber to
lists with status=confirmed, overriding the opt-out.
- MCS150UpdateHandler is the catch-all for many admin-assisted DOT services
(UCR, MC authority, audit prep, ETA, name reservation, registered agent,
annual report). It was filling an MCS-150 PDF for ALL of them -- e.g. a UCR
order produced a wrong MCS-150 PDF. Now only MCS150_FORM_SLUGS fill the form;
others get an admin-review todo (PDF 'not generated') for manual handling.
Signature flow was already correctly scoped (UCR is not in DOT_SIGNING).
- handle_process_compliance_service forced the Sales Order workflow_state to
'Review' via set_value, which bypasses ERPNext's allowed transitions and
threw WorkflowPermissionError (Received -> Review) on every run. The Postgres
fulfillment_status is the source of truth; the ERPNext workflow_state is a
cosmetic mirror. Now try the proper apply_workflow action and stay quiet
(debug, not warning) when no valid Review transition exists.
We no longer offer Canadian accountant/accounting setup. Removed all
service-offering content:
- Marketing page (services/telecom/canada-crtc): the 'Set Up Canadian
Accounting (we help)' next-steps card, the '3 hours of complimentary
accounting consultation' deliverable bullet, and the whole 'Accounting
Support' section (assigned accountant, portal chat, $75/hr, 3 complimentary
hours).
- Order page (order/canada-crtc): the '3 hrs Canadian accounting support'
included-feature bullet and the 'Preferred accounting software'
(Xero/QuickBooks) form field + its accounting-hours helper text.
- Fulfillment (canada_crtc.py): dropped the bank-setup email line offering
'3 hours of Canadian accounting consultation'.
Kept factual GST/HST tax advisories and the bank's QuickBooks/Xero
transaction-sync feature (third-party bank capability, not our service).
MCS150UpdateHandler is the catch-all for many admin-assisted DOT services
(UCR, MC authority, audit prep, ETA, name reservation, registered agent,
annual report). My new intake-completeness gate was firing the 'confirm your
MCS-150 details' email for ALL of them -- e.g. a UCR order wrongly emailed the
customer about MCS-150 details. Scope the gate to MCS150_FORM_SLUGS (the
services that actually file an MCS-150: mcs150-update, dot-registration,
usdot-reactivation, dot-full-compliance).
Paul Wilson (Compound Technologies) signed up with synthetic@pipeline.com,
which is a genuine, deliverable EarthLink address (pipeline.com MX ->
earthlink-vadesecure.net; he confirmed receipt by phone). Our code had
hardcoded pipeline.com + the synthetic@ prefix as a 'non-deliverable
FMCSA-census placeholder' and silently suppressed every automated email to
him (checkout provisioning, order-creation validation, intake reminders,
set-password invites). Nothing in the codebase actually generates that
address, so the placeholder rationale was wrong. Removed pipeline.com and the
synthetic@ rule from all four suppression sites; only RFC-reserved
example.com/test.com/invalid remain blocked.
Closes the data gap for orders that bypass the full intake (e.g. the DOT
compliance-remediation pipeline) and for all MCS-150 variants:
- Worker intake-completeness gate (mcs150_update): before filling, check the
customer-required operational fields the FMCSA census cannot supply
(operation classification, cargo, CURRENT annual mileage, email; plus
signer/address for new-registration/reactivation, and states-of-operation
for 150B hazmat). If missing, email the customer a census-pre-filled intake
link and hold the order at fulfillment_status='awaiting_intake' with an admin
todo, instead of fabricating a blank filing. The existing intake PUT endpoint
already re-dispatches the worker on submit, so filing auto-resumes.
- Intake wizard (Wizard.astro): when resuming ?order=CO-xxx for a DOT/MCS order,
seed still-empty fields from the FMCSA census (name/address/fleet/interstate)
so the customer only confirms the operational details.
- /api/v1/dot/census now also returns total_drivers + a normalized
carrier_operation_code for the prefill.
- MCS150Step.astro extended to collect every field the filler needs across all
variants: mailing address, cdl_drivers, primary_vehicle_type,
reason_for_filing, usdot_revoked, cell/fax, hazmat-safety-permit block
(needs_hmsp, operating states, security plan), and intermodal-equipment
provider counts; all prefill from intake_data.
verify_mcs150_variants.py covers 150/150B/150C end-to-end (ALL PASS).
Adds the previously-unmapped fields so every variant fills fully:
- Q25 hazmat C/S/B/NB matrix (HAZMAT_ROW_MAP x HAZMAT_COL_MAP, 156 boxes)
- MCS-150B states-of-operation checkboxes (full name or 2-letter code), HMSP
Hazard/Permit/Security radios, and accident count (32accidentNumber)
- MCS-150C intermodal equipment counts (20owned/leased/serviced) + correct
field renumbering (17dunbrad/18irs/19eMail) + USDOT Button + named-export
Reason/Mailing radios
- Structured fleet via intake['vehicles'] = {vehicle_type: {owned, term_leased,
trip_leased}} across all Q26 vehicle rows; non-CMV count; cell/fax; second
officer
- _set_button now resolves a candidate tuple against each field's actual export
states, so numeric (/0../4) and named (/Yes,/Biennial...) radios both work
verify_mcs150_variants.py exercises all three variants end-to-end: ALL PASS.
certifyBox is the Q29 Passenger Carrier Compliance Certification YES box
(page 3, y=530), not a general perjury checkbox. It was being checked
unconditionally, which wrongly marked freight/property carriers as passenger
carriers. Now only check it when the carrier is a passenger carrier; the
Q31 perjury declaration is made via the signature.
Fixes a batch of missing fields the FMCSA census does not provide and the
filler was mis-mapping:
- Corrected the question->field mapping to match the actual form: Q22 =
COMPANY OPERATIONS (interstate/intrastate, 22xBox), Q23 = OPERATION
CLASSIFICATIONS (for-hire/private/govt, 23xBox). These were swapped, and
the bogus entity-type->23xBox map (no entity-type question exists on this
form revision) was removed.
- Added proper radio-group handling for Reason for Filing (Biennial Update),
Mailing-address (Same as principal vs below), and Q28 USDOT-revoked, with
correct option indices (these are /0../n radios, not /Yes checkboxes; the
old code set them to /Yes and never selected the right option).
- Map interstate/intrastate from the FMCSA census carrierOperationCode, and
populate email/phone/mileage/cargo from intake.
- AcroForm checkbox/radio appearances use a ZapfDingbats glyph that
poppler/Preview fail to render (value set but box looks empty). Now stamp
an explicit X overlay into the page content for every 'on' box so it shows
in every viewer and in the faxed output.
The official MCS-150/150B/150C PDFs ship with 8 (150/150B) or 4 (150C)
FMCSA instruction/example pages before the actual fillable form. We were
generating + faxing/submitting all of them. Trimmed the source templates
down to the FORM pages only:
MCS-150 11 -> 3 pages (289 fields preserved)
MCS-150B 12 -> 4 pages (349 fields preserved)
MCS-150C 6 -> 2 pages (33 fields preserved)
The filler iterates writer.pages (no absolute index) and signature
anchors are derived dynamically via enumerate(reader.pages), so no
page-specific markup needed fixing. Removed one-off diag script.
handle_process_compliance_service assumed handlers return local temp
paths and re-uploaded each to MinIO. The MCS-150 handler uploads itself
and returns the MinIO key, so the re-upload tried to read a nonexistent
local file and logged a 'File not found' error after the order was
already correctly held at the admin gate. Now we skip files that don't
exist locally and keep the returned key as-is.
- When intake lacks signer_name, backfill it from the name the client
typed when signing the perjury certification (that name is exactly what
belongs in the form's print/type-name field, certifyName).
- After a client-approved re-dispatch, re-point the signed esign record at
the freshly filled form and re-stamp the signature, so the signed PDF an
admin reviews reflects the current complete form (not a stale earlier
fill). Field layout (and thus signature anchors) is unchanged across
fills, so the recorded anchor coordinates stay valid.
- fill_mcs150 now uses auto_regenerate=True so pypdf writes appearance
streams for every text field. Preview/Chrome ignore /NeedAppearances and
were showing blank widgets over the values; generated /AP streams make
the text render in all viewers.
- New verify_mcs150.py reads each widget's /AP /N appearance stream (the
literal drawn glyphs) to confirm expected values actually render, since
the container has no OCR/raster tooling. Exits non-zero on any miss.
The MCS-150 biennial update re-confirms the carrier's existing FMCSA
record. Previously the PDF filler only had whatever the intake form
collected; rescued/sparse orders (or orders where the carrier's data
lives in FMCSA, not the intake) produced near-empty forms. Now we pull
the carrier census (legal name, address, EIN, fleet counts) from the
FMCSA carrier API and merge it under any customer-provided intake values
(customer edits win), so the form is pre-filled with the carrier's
current registered data. Refactored the FMCSA fetch into a shared
_fetch_fmcsa_carrier helper used by both enrichment and status check.
Customer saw the MCS-150 looking blank / 'data covered by the form fields': the
values were correctly written to the AcroForm /V, but pypdf left the template's
empty /AP appearance streams in place and NeedAppearances was false, so viewers
rendered the blank widget over the value. Setting AcroForm /NeedAppearances=true
makes viewers regenerate appearances from the values. (The missing signature was
a downstream effect of the separate fobj_put MinIO-upload bug, now fixed -- with
no PDF in MinIO the anchor extraction + signature stamping both failed.)
When an MCS-150/USDOT order hits the pre-submission admin-verification gate, the
Telegram FULFILLMENT NEEDED alert now appends a presigned link to the prepared
PDF (via the public minio.performancewest.net endpoint, IP-allowlisted to admin)
so you can review the document straight from the alert before approving. Added
notify_fulfillment_todo(view_url=...) + a _presigned_view_url helper (public
endpoint + explicit region to avoid the region-probe that 403s from the worker).
The MCS-150/USDOT PDF was generated fine but the MinIO upload threw 'Minio object
has no attribute fobj_put' (wrong method name + signature), so the prepared filing
PDF was never persisted -- nothing for an admin to review at the verification gate,
and the esign-completed re-dispatch failed with 'File not found'. Use the correct
minio fput_object(bucket, key, file_path). Affects every MCS-150/USDOT filing.
Per request: after the customer signs but BEFORE we submit to the government, hold
the order for a human to verify the prepared filing is correct.
- MCS-150 handler (mcs150-update + usdot-reactivation): new admin-verification gate
after the signature gate -- if not admin_approved, set fulfillment_status=
'ready_to_file', create a HIGH-priority 'VERIFY before filing' admin todo, and
STOP (no FMCSA submission). job_server injects admin_approved from the dispatch
payload (mirrors client_approved).
- New admin endpoint POST /api/v1/admin/compliance-orders/:id/approve-submit
(requireAdmin): verifies status=ready_to_file, re-dispatches the worker with
admin_approved=true to proceed to actual submission.
- Durable submission EVIDENCE: the web/fax submitters only wrote confirmation
screenshots to an ephemeral temp dir. Now _upload_submission_evidence copies the
FMCSA confirmation screenshot + attested PDF + fax_log_id to MinIO under
filings/<slug>/<order>/evidence/ and records the keys on the order, so we keep
proof of every government submission.
(state-trucking + the FCC handlers already gate via admin todos / auto_filing.py;
this brings MCS-150 to parity and adds evidence retention.)
The rescue onboarding emails hardcoded a 60-minute expiry -- way too short for a
paid customer who hasn't engaged yet (they may not check email for hours/days),
so Paul's and Mitchell's links expired before they used them. Onboarding links
now last 7 days (ONBOARDING_TTL_MINUTES); the standard security password-RESET
window bumped 30min -> 2h. Re-issued fresh 7-day links to all 3 affected
customers (none had set a password yet) via reissue-onboarding-links.mjs, cc'd.
