Commit graph

72 commits

Author SHA1 Message Date
justin
e40f359693 fix photo upload: add synchronous /jobs/presign and /jobs/minio-upload endpoints to workers 2026-05-30 19:13:51 -05:00
justin
a3a546abff add FMCSA confirmation email monitor: IMAP watcher links confirmations to orders, Telegram notify 2026-05-30 19:01:42 -05:00
justin
b90d443667 add submit_filing(): 3x web retry then fax fallback, fix success detection, shared attestation helper 2026-05-30 18:58:14 -05:00
justin
4a4bbd048e fix FMCSA form: wait for lvl2 dropdown, update selectors 2026-05-30 18:54:34 -05:00
justin
99a53ad970 add FMCSA web submitter: Playwright automation for ask.fmcsa.dot.gov ticket form 2026-05-30 18:47:48 -05:00
justin
1f1113d63c add fax filing pipeline: VitalPBX sender, attestation cover page with digital signature, compliance checker pending filing override
- filing_attestation.py: generates cover page attesting PW submitted document
  to recipient with date/time stamp, contact info, and digital signature
- fax_sender.py: sends PDFs via VitalPBX API, polls for delivery, generates
  attested copy for customer records
- dot-lookup.ts: if DOT has pending MCS-150 order, show green 'UPDATE SUBMITTED'
  instead of red 'OVERDUE' in compliance checker
- requirements.txt: add pyhanko + cryptography for PDF digital signatures
2026-05-30 18:32:01 -05:00
justin
7ef509c247 fix photo ID upload: use workers for MinIO storage + public presigned URLs
- id-upload.ts: replace broken direct minio import with workers presign/upload
- job_server.py: add minio-upload handler for API to store files via workers
- rewrite presigned URLs from internal minio:9000 to public minio.performancewest.net
- fixes: thumbnail not showing after phone upload, base64 fallback storage
2026-05-30 18:12:06 -05:00
justin
72d1b336c5 Add --where filter to email verifier for targeted scrubs
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-30 15:06:02 -05:00
justin
97f6a08183 Bind email verifier to secondary IP (.72) for SMTP probes
Campaign emails send from .71 via Postfix (now explicitly bound).
Verification RCPT TO probes go from .72 to protect sending reputation.
Configurable via VERIFY_SOURCE_IP env var.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-30 14:52:22 -05:00
justin
2dacf1ea0e FMCSA enrichment: OOS orders bulk download + authority/insurance API lookup
- Downloads 389K OOS orders from Socrata, merges into fmcsa_carriers
- Batch enriches authority status + insurance filing via FMCSA API
- Adds columns: oos_active, authority_status, insurance_*_on_file, etc.
- Rate limited to 1 req/sec for API calls
- Prioritizes campaign-eligible for-hire carriers

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-30 14:46:40 -05:00
justin
1cfda2c119 Fix census download crash at 100K: integer out of range
safe_int now clamps values to PostgreSQL INTEGER max (2.1B) and
handles scientific notation. Mileage columns changed to BIGINT
on prod since carriers can have >2B annual miles.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-30 13:41:24 -05:00
justin
d32ef991b8 Fix Dockerfile: use JSON array syntax for filenames with spaces
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-30 13:07:26 -05:00
justin
02b65fc37a Fix Dockerfile: quote MCS-150 filenames with spaces
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-30 13:06:58 -05:00
justin
8ab5768606 Add MCS-150 PDF forms to workers Docker image
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-30 13:05:26 -05:00
justin
b2a4a48610 Add pypdf to requirements for MCS-150 form filling
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-30 13:00:20 -05:00
justin
1f24255358 MCS-150 official PDF filler — fills actual FMCSA fillable form
Uses pypdf to fill the official MCS-150/150B/150C fillable PDFs.
Maps intake data to 289 form fields (text + checkboxes).
Supports form type detection (standard vs hazmat vs intermodal).
Produces ready-to-fax PDF from intake data.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-30 12:59:15 -05:00
justin
e82aa0b8c2 Fix PayPal capture for compliance orders + MCS-150 form generator
PayPal capture was defaulting to canada_crtc_orders table for all
non-formation orders. Now properly routes compliance_batch orders
to compliance_orders table with batch_id lookup. Also infers
order type from ID prefix (CB-=batch, CO-=compliance, FO-=formation).

MCS-150 form generator: produces DOCX with fax cover sheet + filled
MCS-150 form for faxing to FMCSA at 202-366-3477.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-30 11:34:16 -05:00
justin
763100f664 Fix bounce watcher: pass campaign_uuid to Listmonk webhook
Bounces showed 0 in campaigns because the webhook didn't include
campaign_uuid. Now fetches UUID of running campaign via API and
includes it in bounce reports. Refreshes every 100 messages.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-29 15:32:48 -05:00
justin
4cfcb6a50c Add dnspython to requirements for email verifier
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-29 14:37:36 -05:00
justin
ed0a44645e Email verifier: add catch-all domain detection via random probe
Before checking the real address, sends a random 20-char address to
the domain. If the server accepts it (250), the domain is catch-all
and individual verification is meaningless. Result cached per domain.
Existing known catch-all list (gmail, outlook, etc.) still bypassed.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-29 14:34:23 -05:00
justin
258e384f95 Fix state list import: use individual subscriber creates instead of bulk
Listmonk bulk import API returns 400. Switch to individual POST
/api/subscribers with 409 conflict handling (update existing subs
to add them to the new list). Progress logging every 200 records.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-29 14:26:44 -05:00
justin
4df73337b8 Add state-targeted campaign creator (CA, OR, NY)
Script creates Listmonk campaigns with state-specific content:
- California: MCP + CARB + CHP CA Number + insurance
- Oregon: Weight-Mile Tax + IRP + ODOT authority
- New York: HUT + insurance ($1.5M NYC) + intrastate authority

Usage: python3 scripts/create_state_campaigns.py california

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-29 14:25:12 -05:00
justin
4aff121c0b Fix interstate detection: use carrier_operation code A, not text match
FMCSA census carrier_operation is single-letter: A=Interstate,
B=Intrastate Hazmat, C=Intrastate Non-Hazmat. Previous code searched
for "interstate" in text which never matched. Now 22,089 interstate
carriers will be properly flagged for IRP/IFTA.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-29 14:01:18 -05:00
justin
46411c09c6 Add Playwright automation for BOC-3 filing on processagent.com
- boc3_playwright.py: 4-step form automation (company info, contact,
  account, payment) using patchright/undetected Playwright
- Payment with PW company card ($25/filing), credentials from env
- CAPTCHA detection — falls back to admin todo if reCAPTCHA triggers
- boc3_filing.py: process() tries Playwright first, falls back to
  manual admin todo on failure
- Env vars needed: PW_CARD_NUMBER, PW_CARD_CVC, PW_CARD_EXP_MONTH,
  PW_CARD_EXP_YEAR, BOC3_ACCOUNT_PASSWORD

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-29 13:24:12 -05:00
justin
584f887f82 Extend FMCSA flagger with state-level deficiency flags
- Interstate carriers flagged for IRP/IFTA needs
- Weight-distance tax flags (OR/NY/KY/NM/CT)
- State carrier permit flags (CA MCP, etc.)
- Emissions flags (CA CARB, ACT states)
- Intrastate authority flags (29 states, for-hire only)
- New --state-lists flag creates state-targeted Listmonk lists
  (CA, OR, NY, KY, NM, CT, TX, FL)
- Stats now include state-level counts

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-29 13:03:49 -05:00
justin
e473953fef Update BOC-3 handler: Process Agent LLC (Registered Agents Inc)
ProcessAgent.com is subsidiary of Registered Agents Inc. $25/yr
blanket BOC-3, no API, will automate via Playwright. Updated
partner details, admin todo steps, removed dead API stub.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-29 13:00:31 -05:00
justin
33da00fd89 50-state trucking compliance: services, checker, order page, CA landing
- Migration 079: state_trucking_requirements table seeded for all 51 jurisdictions
  (IRP, IFTA, weight-distance taxes, MCP/CARB, intrastate authority, state DOT)
- Migration 080: carrier_operating_states tracking table
- 13 new state trucking services in catalog ($99-$599)
- StateTruckingHandler with state-specific admin todos
- DOT compliance checker: 7 new state-level checks (IRP, IFTA, weight tax,
  MCP/CARB, emissions, intrastate authority, state DOT number)
- New API endpoint: GET /api/v1/dot/state-requirements
- DOT order page: state compliance service cards with auto-preselect
- California trucking landing page (MCP + CARB + IRP + IFTA)
- Fix: DOT checker nav missing Trucking/DOT section
- Fix: All 8 DOT intake pages missing style block (dangling text)
- Fix: DOT confirmation email now says "Order Confirmed" not "Action Required"
- Fix: MCS150/BOC3/StateTrucking handlers missing async process() method
- Fix: StateTruckingHandler connection leak + slug resolution

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-29 12:46:33 -05:00
justin
e67df3c4c3 DOT intake: review-only (no telecom entity step), email verifier,
updated flagger excluding 4+ year stale carriers

- Intake manifest: DOT services use ["review"] only, skipping the
  telecom entity step with FRN/USAC fields
- Flagger: excludes 4+ year overdue carriers from campaign (spam
  trap risk). 18,277 safe targets from 100K records.
- Email verifier: self-hosted MX + SMTP verification tool

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-29 00:29:28 -05:00
justin
5bf364aced Enhance FMCSA flagger: for-hire as deficiency, zero fleet detection
- For-hire carriers now flagged with issues (BOC-3/UCR/insurance needed)
- Zero trucks/drivers flagged as stale data
- For-hire + MCS-150 overdue = critical severity
- Actionable flags count excludes zero_fleet (informational only)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-29 00:19:00 -05:00
justin
e510cefb88 Fix: bool(email) instead of email for campaign_eligible flag
Python 'and' returns the string, not True. Changed to bool().

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-29 00:17:18 -05:00
justin
563df6c5d4 Add FMCSA carrier deficiency flagger for campaign targeting
Analyzes census data and flags carriers with:
- MCS-150 overdue (2+ years) / severely overdue (4+ years)
- For-hire status (needs BOC-3, UCR, insurance)
- Hazmat (needs PHMSA registration)

Adds columns to fmcsa_carriers: deficiency_flags, deficiency_count,
deficiency_severity, issues_summary, campaign_eligible.

Can populate Listmonk subscriber list with --listmonk flag.
From 100K test: 20,039 campaign-eligible carriers with email.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-29 00:15:52 -05:00
justin
8149996107 Add 8 DOT/FMCSA services to catalog and handler registry
Service catalog (compliance-orders.ts):
- mcs150-update: $79 (MCS-150 biennial update)
- boc3-filing: $149 (BOC-3 process agent)
- ucr-registration: $79 + $59 gov fee (UCR annual)
- dot-registration: $149 (new USDOT number)
- mc-authority: $499 + $300 gov fee (operating authority)
- dot-drug-alcohol: $199 (D&A compliance program)
- dot-audit-prep: $399 (new entrant safety audit prep)
- dot-full-compliance: $499 (bundle)

Handler registry (__init__.py):
- MCS150UpdateHandler for admin-assisted filings
- BOC3FilingHandler for process agent designations
- Other DOT services use MCS150 handler pattern (admin todo)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-28 22:43:57 -05:00
justin
8441c6f0c0 Add MCS-150 and BOC-3 service handlers for trucking compliance
MCS-150 Biennial Update ($79):
- Admin-assisted filing (FMCSA Portal requires Login.gov MFA)
- Creates admin todo with intake data and filing steps
- Checks current MCS-150 status via FMCSA API
- Sends status email to client

BOC-3 Process Agent Filing ($149):
- Partners with blanket process agent (NWRA or similar)
- Collects carrier info, submits designation to partner
- Partner files electronically with FMCSA
- Stub for future process agent API integration
- Sends status/confirmation emails

Both follow the same handler pattern as FCC services (admin todo
with structured data when full automation isn't possible).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-28 22:37:47 -05:00
justin
dfee4fc6c0 Add FMCSA motor carrier census table and Socrata data downloader
New vertical: FMCSA/DOT motor carrier compliance services.
- Migration 078: fmcsa_carriers table with 31 fields (DOT#, name,
  email, phone, address, fleet size, MCS-150 date, carrier type)
- Downloader: Socrata API ingest for 2M+ carriers with upsert
- Data source: data.transportation.gov (free, public)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-28 21:05:46 -05:00
justin
27a1a1f7ed Add Frontier Networks + Twilio to vendor guide (12 vendors total)
Skipped Comwave (acquired). Full list: Fibernetics, Iristel,
Flowroute/BCM One, VoIP.ms, Telnyx, SkySwitch/Sangoma, Distributel,
Allstream/Zayo, Bandwidth, IXICA, Frontier Networks, Twilio.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-27 11:21:07 -05:00
justin
79375c11b3 Add Bandwidth and IXICA to vendor guide from CRTC registry data
Scraped CRTC registered providers list (4,669 records, 1,843 companies).
Added Bandwidth Inc. (CLEC, wholesale DID/voice API) and IXICA
Communications (Toronto CLEC, wholesale voice/data). Now 10 vendors.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-27 11:20:35 -05:00
justin
9919947440 Fix: Flowroute (Intrado) -> Flowroute (BCM One)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-27 11:17:13 -05:00
justin
15cd006260 Vendor guide: Calibri font, US carrier note, cleanup
- Set Calibri as explicit font on all runs via _run() helper
- Added note that many large US carriers work with Canadian carriers
  and to ask about specific regulatory requirements
- BCM One ownership for Flowroute
- Cleaned up leftover manual font assignments

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-27 11:16:02 -05:00
justin
3bac30510e Update Flowroute ownership: now BCM One (acquired from Intrado)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-27 11:14:12 -05:00
justin
32c1e57c5c Add Canadian Wholesale Vendor Reference Guide to CRTC binder
New DOCX generator with 8 recommended upstream providers:
Fibernetics, Iristel, Flowroute, VoIP.ms, Telnyx, SkySwitch,
Distributel, Allstream. Each with services, website, and notes.

Wired into CRTC handler Step 6a (generates before eSign pause)
and added to binder compiler default sections.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-27 11:11:47 -05:00
justin
7139630fa8 Add Accessible Canada Act (ACA) compliance tracking for CRTC orders
- CRTC handler: new compliance calendar entry for ACA accessibility
  plan/progress report (June 1 annual deadline), with 2-month advance
  reminder. Covers new entrant feedback process and existing carrier
  plans/reports. Notes dual notification requirement (Commissioner + CRTC).
- Order page: amber advisory banner about June 1, 2026 ACA deadline
  with key obligations and penalty warning
- "What's included" list: added ACA accessibility compliance tracking

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-27 10:09:33 -05:00
justin
ba2f6eb667 UPL-proof document templates + reliable bounce sync
Templates (22 files):
- Replace "Reviewed By" with "Document prepared by" + consulting disclaimer
- Add "not a law firm / not legal advice" footer to all CPNI, CALEA, RMD docs
- Change "on behalf of" to "at the direction of" in discontinuance letter
- Reframe RMD penalty language as client acknowledgment

Bounce sync:
- New listmonk-bounce-sync.py replaces unreliable bash tail watcher
- Scans full mail.log, matches QIDs to campaign senders, inserts directly
  into Listmonk DB with proper subscriber_id foreign keys
- Idempotent, runs via cron every 5 minutes

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-21 15:06:29 -05:00
justin
71d466c922 Wire createCommission() into compliance batch checkout
Compliance batch orders now create commission ledger entries when
a discount code (agent referral) is used. Tracks total order amount,
discount applied, and links to the agent for payout processing.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-11 11:00:38 -05:00
justin
97dd08c821 Fix flagged items: CRTC email submission, BITS todo, selector docs, stale plans
- CRTC letter now auto-emailed to secretary.general@crtc.gc.ca after eSign
- BITS admin todo updated to reference electronic + physical submission
- COLIN selectors.py: documented verification status per step
- BC config: added CRTC Secretary General email address
- plan.md: marked completed items (eSign, portal auth, CRTC email)
- go-live-todo.md: marked Compliance Calendar DocType as imported

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-04 11:33:45 -05:00
justin
d6da7f4d7b Use Word VM PDF converter instead of raw LibreOffice in base handler
_convert_to_pdf() now calls pdf_converter.convert_to_pdf() which tries
the Windows Word VM via MinIO first (pixel-perfect), falling back to
LibreOffice headless automatically when the VM is unavailable.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-04 11:02:49 -05:00
justin
9ca6094984 Fix MinioStorage.upload_file → .upload + suppress instant delivery during eSign pause
- upload_file doesn't exist on MinioStorage — the method is upload()
- Return [] when pausing for eSign so job_server doesn't trigger instant delivery

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-04 10:59:45 -05:00
justin
ff47f47a37 Wire CPNI, CALEA, 499-A engagement, and discontinuance to generic eSign
Each handler now pauses for officer signature via the eSign portal
before filing/submitting. esign_completed callback re-dispatches
through standard pipeline with client_approved=true.

- CPNI: officer signs certification before ECFS submission (perjury)
- CALEA SSI: officer signs plan before delivery
- 499-A engagement: replaced custom JWT/email with request_esign()
- Discontinuance: officer signs deactivation letter before USAC email
- job_server: injects client_approved + order_number into order_data

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-04 10:53:59 -05:00
justin
40844b2aff Add generic eSign portal for all compliance document types
Reusable signing flow: service handler generates document → inserts
esign_records row → emails JWT link → client reviews PDF + signs →
API stores signature + resumes pipeline. Works for RMD, CPNI, CALEA,
499-A engagement, discontinuance, CRTC, and any future doc types.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-04 10:45:37 -05:00
justin
37a22cf474 Delete 499-A checklist generator — intake wizard handles data collection
The checklist was a manual-process artifact that listed what info the
client needed to gather. Since all data is now collected through the
intake wizard and CDR upload, the checklist is unnecessary. Removed
from the 499-A handler's prep packet and deleted the 1,326-line
generator file.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-04 09:10:50 -05:00
justin
463c180444 Add shared DOCX style module + campaign tools
_styles.py: Centralized typography, spacing, and formatting for all
26 DOCX generators. Calibri 9.5pt body, 1.15 line spacing, navy
headings, consistent signature blocks, page numbers, PW footer.
All generators will be migrated to use this instead of defining
their own styles.

Campaign tools:
- campaign_template.html: Styled email template for Listmonk campaigns
- populate_deficiency_list.py: Populates Listmonk with FCC deficiency data
- send_test_campaigns.py: Sends test emails with real carrier data

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-04 08:52:07 -05:00