Commit graph

84 commits

Author SHA1 Message Date
justin
d3bf5b3520 preview test send: keep {{ UnsubscribeURL }} (real link); hero 4-col to save vertical space
- send_test no longer overwrites {{ UnsubscribeURL }} with a dead static URL;
  Listmonk renders it into a working per-subscriber unsubscribe link.
- dot-compliance hero grid: 4 columns (minmax 150px, max-width 920px) instead
  of 3 to reduce vertical space.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-05-31 17:38:57 -05:00
justin
60312e5201 campaign builder: add --preview mode + fix subscriber-attach + test-send list bugs
- import_subscribers: was POSTing wrong bulk shape AND fallback used 'list_ids'
  (ignored by Listmonk) instead of 'lists' -> subscribers never attached to the
  list -> real sends would go to an empty list. Now single-adds with 'lists',
  handles already-exists, returns a count, logs if 0 added.
- send_test: passed base['lists'] (objects) instead of IDs -> test send rejected.
  Now extracts list IDs.
- create_and_schedule_campaign: add schedule= flag (preview makes drafts).
- --preview: 1 sample carrier/campaign, only owner email, drafts not scheduled,
  test sends immediately, never marks real carriers sent.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-05-31 17:28:09 -05:00
justin
03702dfbb7 campaign builder: send test email to carrierone@gmx.com per campaign before real blast
Each of the 8 daily campaigns gets a test send immediately after creation
using the first row's real carrier data as the sample. TEST_EMAIL env var
overrides the default (carrierone@gmx.com).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-31 17:19:28 -05:00
justin
0b7a35a58e trucking campaigns: daily builder + MX verifier concurrency + tracking column
- build_trucking_campaigns.py: nightly script that creates 8 Listmonk campaigns
  per day (4 TZ x 2 types: MCS-150 overdue 2k/TZ, inactive USDOT 1k/TZ)
  at 4AM ET / 5AM ET (CT) / 6AM ET (MT) / 7AM ET (PT). Deduplicates via
  listmonk_sent_at column.
- migration 083: add listmonk_sent_at + listmonk_campaign_type to fmcsa_carriers
- email_verifier.py: bump max_workers from 5 to 20 for 4x faster throughput
- cron: daily pw-trucking-campaigns at 08:00 UTC (3 AM EST)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-31 10:07:44 -05:00
justin
b25d1f5fd3 trucking wrap-up: close-out becomes a paid order + workflow
- Checker closing mode now pitches a done-for-you 'Trucking Wrap-Up' ($199)
  with a buy button to /order/dot-compliance?services=carrier-closeout, instead
  of a lead form. DIY checklist replaced by what's-included list.
- Entity dissolution offered as a paid add-on with the lawsuits/liens/judgments
  warning before dissolving.
- New catalog services: carrier-closeout ($199), entity-dissolution ($199).
- CarrierCloseoutHandler orchestrates the sequential shutdown workflow
  (final MCS-150 out-of-business, MC revoke, UCR cancel, IFTA/IRP + state
  closures; dissolution branch for the add-on) as admin-tracked tasks.
- Sell-your-trucks: single shared form with quick-cash / marketplace / both;
  name field is now a real first+last name (no corp-name prefill).
- tickets categories: add truck_sale_both, drop business_closeout (now an order).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-05-31 01:01:02 -05:00
justin
e0ba8acc90 add pipeline orchestrator, mailbox 1583 flow, EIN + virtual-mailbox services
- Pipeline orchestrator: chains sequential fulfillment for new carrier bundles
  (formation → EIN → USDOT → MC → BOC-3 → MCS-150 → D&A → UCR)
- Mailbox setup: Anytime Mailbox provisioning with USPS 1583 e-sign + online notarization
- New services: ein-application ($79), virtual-mailbox ($149/yr)
- Registered all new handlers in SERVICE_HANDLERS
- Pipeline cron: every 5 minutes
2026-05-30 22:56:54 -05:00
justin
ad41de817c BOC-3: use relay_integration.load_card_from_erpnext() for proper card matching 2026-05-30 22:46:52 -05:00
justin
6ca835b1b4 BOC-3: load matching PW card from ERPNext (stripe/paypal/crypto per payment method)
EIN: add handler with IRS hours check (Mon-Fri 7am-10pm ET), dev mode guard
2026-05-30 22:43:30 -05:00
justin
6dd311f375 add dev mode guards: skip FMCSA submission, fax, and web filing in non-production environments 2026-05-30 22:28:59 -05:00
justin
aa7ed5efe9 wire MCS-150 handler to full pipeline: PDF fill → MinIO → e-sign → web/fax submit → attestation
- Fills official MCS-150 PDF with intake data (pypdf)
- Uploads to MinIO for storage
- Creates esign_records row with perjury declaration
- Sends e-sign link to customer (JWT, 7-day expiry)
- After sign: submits via ask.fmcsa.dot.gov (3x) → fax fallback
- Generates attestation cover page + digital signature
- Updates order with filing status, method, screenshots
- Creates admin todo for verification
2026-05-30 22:13:18 -05:00
justin
ad3d189b2b post-completion flow: survey, referral program, review ask
- Migration 081: referral_codes, referral_uses, exit_surveys tables
- API: POST /api/v1/survey, POST /api/v1/referral/check, GET /api/v1/referral/:email
- Worker: completion_emails.py — sends completion + 24h follow-up (survey + referral)
- Survey page: /survey/?order=X&rating=N — star rating, feedback, Google review ask
- Referral: REF-FIRSTNAME codes, $25 credit per referred order, no limit
- Low ratings (1-3 stars) trigger Telegram alert for admin follow-up
- Cron: every 15 minutes
2026-05-30 21:22:14 -05:00
justin
d51a2f61b4 fix Iowa: remove wrong Socrata dataset (was liquor stores), mark SOS as WAF-blocked 2026-05-30 19:28:41 -05:00
justin
e40f359693 fix photo upload: add synchronous /jobs/presign and /jobs/minio-upload endpoints to workers 2026-05-30 19:13:51 -05:00
justin
a3a546abff add FMCSA confirmation email monitor: IMAP watcher links confirmations to orders, Telegram notify 2026-05-30 19:01:42 -05:00
justin
b90d443667 add submit_filing(): 3x web retry then fax fallback, fix success detection, shared attestation helper 2026-05-30 18:58:14 -05:00
justin
4a4bbd048e fix FMCSA form: wait for lvl2 dropdown, update selectors 2026-05-30 18:54:34 -05:00
justin
99a53ad970 add FMCSA web submitter: Playwright automation for ask.fmcsa.dot.gov ticket form 2026-05-30 18:47:48 -05:00
justin
1f1113d63c add fax filing pipeline: VitalPBX sender, attestation cover page with digital signature, compliance checker pending filing override
- filing_attestation.py: generates cover page attesting PW submitted document
  to recipient with date/time stamp, contact info, and digital signature
- fax_sender.py: sends PDFs via VitalPBX API, polls for delivery, generates
  attested copy for customer records
- dot-lookup.ts: if DOT has pending MCS-150 order, show green 'UPDATE SUBMITTED'
  instead of red 'OVERDUE' in compliance checker
- requirements.txt: add pyhanko + cryptography for PDF digital signatures
2026-05-30 18:32:01 -05:00
justin
7ef509c247 fix photo ID upload: use workers for MinIO storage + public presigned URLs
- id-upload.ts: replace broken direct minio import with workers presign/upload
- job_server.py: add minio-upload handler for API to store files via workers
- rewrite presigned URLs from internal minio:9000 to public minio.performancewest.net
- fixes: thumbnail not showing after phone upload, base64 fallback storage
2026-05-30 18:12:06 -05:00
justin
72d1b336c5 Add --where filter to email verifier for targeted scrubs
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-30 15:06:02 -05:00
justin
97f6a08183 Bind email verifier to secondary IP (.72) for SMTP probes
Campaign emails send from .71 via Postfix (now explicitly bound).
Verification RCPT TO probes go from .72 to protect sending reputation.
Configurable via VERIFY_SOURCE_IP env var.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-30 14:52:22 -05:00
justin
2dacf1ea0e FMCSA enrichment: OOS orders bulk download + authority/insurance API lookup
- Downloads 389K OOS orders from Socrata, merges into fmcsa_carriers
- Batch enriches authority status + insurance filing via FMCSA API
- Adds columns: oos_active, authority_status, insurance_*_on_file, etc.
- Rate limited to 1 req/sec for API calls
- Prioritizes campaign-eligible for-hire carriers

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-30 14:46:40 -05:00
justin
1cfda2c119 Fix census download crash at 100K: integer out of range
safe_int now clamps values to PostgreSQL INTEGER max (2.1B) and
handles scientific notation. Mileage columns changed to BIGINT
on prod since carriers can have >2B annual miles.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-30 13:41:24 -05:00
justin
d32ef991b8 Fix Dockerfile: use JSON array syntax for filenames with spaces
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-30 13:07:26 -05:00
justin
02b65fc37a Fix Dockerfile: quote MCS-150 filenames with spaces
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-30 13:06:58 -05:00
justin
8ab5768606 Add MCS-150 PDF forms to workers Docker image
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-30 13:05:26 -05:00
justin
b2a4a48610 Add pypdf to requirements for MCS-150 form filling
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-30 13:00:20 -05:00
justin
1f24255358 MCS-150 official PDF filler — fills actual FMCSA fillable form
Uses pypdf to fill the official MCS-150/150B/150C fillable PDFs.
Maps intake data to 289 form fields (text + checkboxes).
Supports form type detection (standard vs hazmat vs intermodal).
Produces ready-to-fax PDF from intake data.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-30 12:59:15 -05:00
justin
e82aa0b8c2 Fix PayPal capture for compliance orders + MCS-150 form generator
PayPal capture was defaulting to canada_crtc_orders table for all
non-formation orders. Now properly routes compliance_batch orders
to compliance_orders table with batch_id lookup. Also infers
order type from ID prefix (CB-=batch, CO-=compliance, FO-=formation).

MCS-150 form generator: produces DOCX with fax cover sheet + filled
MCS-150 form for faxing to FMCSA at 202-366-3477.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-30 11:34:16 -05:00
justin
763100f664 Fix bounce watcher: pass campaign_uuid to Listmonk webhook
Bounces showed 0 in campaigns because the webhook didn't include
campaign_uuid. Now fetches UUID of running campaign via API and
includes it in bounce reports. Refreshes every 100 messages.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-29 15:32:48 -05:00
justin
4cfcb6a50c Add dnspython to requirements for email verifier
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-29 14:37:36 -05:00
justin
ed0a44645e Email verifier: add catch-all domain detection via random probe
Before checking the real address, sends a random 20-char address to
the domain. If the server accepts it (250), the domain is catch-all
and individual verification is meaningless. Result cached per domain.
Existing known catch-all list (gmail, outlook, etc.) still bypassed.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-29 14:34:23 -05:00
justin
258e384f95 Fix state list import: use individual subscriber creates instead of bulk
Listmonk bulk import API returns 400. Switch to individual POST
/api/subscribers with 409 conflict handling (update existing subs
to add them to the new list). Progress logging every 200 records.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-29 14:26:44 -05:00
justin
4df73337b8 Add state-targeted campaign creator (CA, OR, NY)
Script creates Listmonk campaigns with state-specific content:
- California: MCP + CARB + CHP CA Number + insurance
- Oregon: Weight-Mile Tax + IRP + ODOT authority
- New York: HUT + insurance ($1.5M NYC) + intrastate authority

Usage: python3 scripts/create_state_campaigns.py california

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-29 14:25:12 -05:00
justin
4aff121c0b Fix interstate detection: use carrier_operation code A, not text match
FMCSA census carrier_operation is single-letter: A=Interstate,
B=Intrastate Hazmat, C=Intrastate Non-Hazmat. Previous code searched
for "interstate" in text which never matched. Now 22,089 interstate
carriers will be properly flagged for IRP/IFTA.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-29 14:01:18 -05:00
justin
46411c09c6 Add Playwright automation for BOC-3 filing on processagent.com
- boc3_playwright.py: 4-step form automation (company info, contact,
  account, payment) using patchright/undetected Playwright
- Payment with PW company card ($25/filing), credentials from env
- CAPTCHA detection — falls back to admin todo if reCAPTCHA triggers
- boc3_filing.py: process() tries Playwright first, falls back to
  manual admin todo on failure
- Env vars needed: PW_CARD_NUMBER, PW_CARD_CVC, PW_CARD_EXP_MONTH,
  PW_CARD_EXP_YEAR, BOC3_ACCOUNT_PASSWORD

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-29 13:24:12 -05:00
justin
584f887f82 Extend FMCSA flagger with state-level deficiency flags
- Interstate carriers flagged for IRP/IFTA needs
- Weight-distance tax flags (OR/NY/KY/NM/CT)
- State carrier permit flags (CA MCP, etc.)
- Emissions flags (CA CARB, ACT states)
- Intrastate authority flags (29 states, for-hire only)
- New --state-lists flag creates state-targeted Listmonk lists
  (CA, OR, NY, KY, NM, CT, TX, FL)
- Stats now include state-level counts

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-29 13:03:49 -05:00
justin
e473953fef Update BOC-3 handler: Process Agent LLC (Registered Agents Inc)
ProcessAgent.com is subsidiary of Registered Agents Inc. $25/yr
blanket BOC-3, no API, will automate via Playwright. Updated
partner details, admin todo steps, removed dead API stub.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-29 13:00:31 -05:00
justin
33da00fd89 50-state trucking compliance: services, checker, order page, CA landing
- Migration 079: state_trucking_requirements table seeded for all 51 jurisdictions
  (IRP, IFTA, weight-distance taxes, MCP/CARB, intrastate authority, state DOT)
- Migration 080: carrier_operating_states tracking table
- 13 new state trucking services in catalog ($99-$599)
- StateTruckingHandler with state-specific admin todos
- DOT compliance checker: 7 new state-level checks (IRP, IFTA, weight tax,
  MCP/CARB, emissions, intrastate authority, state DOT number)
- New API endpoint: GET /api/v1/dot/state-requirements
- DOT order page: state compliance service cards with auto-preselect
- California trucking landing page (MCP + CARB + IRP + IFTA)
- Fix: DOT checker nav missing Trucking/DOT section
- Fix: All 8 DOT intake pages missing style block (dangling text)
- Fix: DOT confirmation email now says "Order Confirmed" not "Action Required"
- Fix: MCS150/BOC3/StateTrucking handlers missing async process() method
- Fix: StateTruckingHandler connection leak + slug resolution

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-29 12:46:33 -05:00
justin
e67df3c4c3 DOT intake: review-only (no telecom entity step), email verifier,
updated flagger excluding 4+ year stale carriers

- Intake manifest: DOT services use ["review"] only, skipping the
  telecom entity step with FRN/USAC fields
- Flagger: excludes 4+ year overdue carriers from campaign (spam
  trap risk). 18,277 safe targets from 100K records.
- Email verifier: self-hosted MX + SMTP verification tool

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-29 00:29:28 -05:00
justin
5bf364aced Enhance FMCSA flagger: for-hire as deficiency, zero fleet detection
- For-hire carriers now flagged with issues (BOC-3/UCR/insurance needed)
- Zero trucks/drivers flagged as stale data
- For-hire + MCS-150 overdue = critical severity
- Actionable flags count excludes zero_fleet (informational only)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-29 00:19:00 -05:00
justin
e510cefb88 Fix: bool(email) instead of email for campaign_eligible flag
Python 'and' returns the string, not True. Changed to bool().

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-29 00:17:18 -05:00
justin
563df6c5d4 Add FMCSA carrier deficiency flagger for campaign targeting
Analyzes census data and flags carriers with:
- MCS-150 overdue (2+ years) / severely overdue (4+ years)
- For-hire status (needs BOC-3, UCR, insurance)
- Hazmat (needs PHMSA registration)

Adds columns to fmcsa_carriers: deficiency_flags, deficiency_count,
deficiency_severity, issues_summary, campaign_eligible.

Can populate Listmonk subscriber list with --listmonk flag.
From 100K test: 20,039 campaign-eligible carriers with email.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-29 00:15:52 -05:00
justin
8149996107 Add 8 DOT/FMCSA services to catalog and handler registry
Service catalog (compliance-orders.ts):
- mcs150-update: $79 (MCS-150 biennial update)
- boc3-filing: $149 (BOC-3 process agent)
- ucr-registration: $79 + $59 gov fee (UCR annual)
- dot-registration: $149 (new USDOT number)
- mc-authority: $499 + $300 gov fee (operating authority)
- dot-drug-alcohol: $199 (D&A compliance program)
- dot-audit-prep: $399 (new entrant safety audit prep)
- dot-full-compliance: $499 (bundle)

Handler registry (__init__.py):
- MCS150UpdateHandler for admin-assisted filings
- BOC3FilingHandler for process agent designations
- Other DOT services use MCS150 handler pattern (admin todo)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-28 22:43:57 -05:00
justin
8441c6f0c0 Add MCS-150 and BOC-3 service handlers for trucking compliance
MCS-150 Biennial Update ($79):
- Admin-assisted filing (FMCSA Portal requires Login.gov MFA)
- Creates admin todo with intake data and filing steps
- Checks current MCS-150 status via FMCSA API
- Sends status email to client

BOC-3 Process Agent Filing ($149):
- Partners with blanket process agent (NWRA or similar)
- Collects carrier info, submits designation to partner
- Partner files electronically with FMCSA
- Stub for future process agent API integration
- Sends status/confirmation emails

Both follow the same handler pattern as FCC services (admin todo
with structured data when full automation isn't possible).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-28 22:37:47 -05:00
justin
dfee4fc6c0 Add FMCSA motor carrier census table and Socrata data downloader
New vertical: FMCSA/DOT motor carrier compliance services.
- Migration 078: fmcsa_carriers table with 31 fields (DOT#, name,
  email, phone, address, fleet size, MCS-150 date, carrier type)
- Downloader: Socrata API ingest for 2M+ carriers with upsert
- Data source: data.transportation.gov (free, public)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-28 21:05:46 -05:00
justin
27a1a1f7ed Add Frontier Networks + Twilio to vendor guide (12 vendors total)
Skipped Comwave (acquired). Full list: Fibernetics, Iristel,
Flowroute/BCM One, VoIP.ms, Telnyx, SkySwitch/Sangoma, Distributel,
Allstream/Zayo, Bandwidth, IXICA, Frontier Networks, Twilio.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-27 11:21:07 -05:00
justin
79375c11b3 Add Bandwidth and IXICA to vendor guide from CRTC registry data
Scraped CRTC registered providers list (4,669 records, 1,843 companies).
Added Bandwidth Inc. (CLEC, wholesale DID/voice API) and IXICA
Communications (Toronto CLEC, wholesale voice/data). Now 10 vendors.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-27 11:20:35 -05:00
justin
9919947440 Fix: Flowroute (Intrado) -> Flowroute (BCM One)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-27 11:17:13 -05:00
justin
15cd006260 Vendor guide: Calibri font, US carrier note, cleanup
- Set Calibri as explicit font on all runs via _run() helper
- Added note that many large US carriers work with Canadian carriers
  and to ask about specific regulatory requirements
- BCM One ownership for Flowroute
- Cleaned up leftover manual font assignments

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-27 11:16:02 -05:00