justin 899b880e7f trucking: weekly FMCSA source refresh so new non-compliant carriers are caught

The FMCSA census was a one-time snapshot (last loaded ~May 30) with NO refresh
timer -- carriers newly falling out of MCS-150/UCR compliance were never picked
up. New scripts/workers/fmcsa_source_refresh.py orchestrates the full pipeline
(census download -> enrichment -> deficiency flag -> verify new emails ->
MX-tag new) and runs weekly via cron pw-fmcsa-refresh (Sun 09:00 UTC), codified
in the mail-pipeline Ansible role.

Idempotent + incremental: the census upsert preserves email_verified /
listmonk_sent_at / deficiency_flags, so existing carriers keep their send state
and only census fields refresh; new DOTs flow into verification then campaigns.
A carrier who refiled gets a fresh mcs150_parsed, so the builder's overdue
WHERE clause stops targeting them automatically. Verify is capped per run
(20k) so it never stalls on millions of rows.

(Healthcare already auto-catches newly-revalidation-overdue providers within
its 63k institutional pool via pw-hc-refresh Mon/Wed/Fri.)

2026-06-17 20:44:54 -05:00

9.5 KiB

Raw Blame History

Trucking / DOT Compliance System

Last updated: 2026-05-29

Overview

Performance West's trucking vertical provides DOT/FMCSA compliance services to 2M+ US motor carriers. The system covers federal FMCSA requirements and state-level obligations across all 50 states + DC.

Data Pipeline

FMCSA Socrata API (data.transportation.gov)
    |
    v
fmcsa_census_downloader.py ──> fmcsa_carriers table (2M+ records)
    |                              - dot_number, legal_name, email
    |                              - phy_state, fleet size, MCS-150 date
    |                              - carrier_operation (A=Interstate, B=Intra HM, C=Intra)
    v
fmcsa_deficiency_flagger.py ──> deficiency_flags, campaign_eligible
    |                              - MCS-150 overdue/severe
    |                              - For-hire, hazmat, zero fleet
    |                              - Interstate IRP/IFTA needs
    |                              - State weight tax, carrier permit, emissions
    |                              - Intrastate authority
    v
Listmonk lists ──> Email campaigns (state-segmented)
    - FMCSA MCS-150 Overdue Carriers
    - FMCSA State - California (MCP/CARB)
    - FMCSA State - Oregon (Weight-Mile Tax)
    - FMCSA State - New York (HUT)
    - FMCSA State - Kentucky (KYU)
    - FMCSA State - Texas
    - FMCSA State - Florida
    - FMCSA State - Connecticut (HUF)
    - FMCSA State - New Mexico (WDT)

Compliance Checker

Endpoint: GET /api/v1/dot/lookup?dot=XXXXXX

Federal Checks (1-7) — from FMCSA QCMobile API

#	Check	Source	Colors
1	Operating Status	FMCSA API `allowedToOperate`	Green/Red
2	MCS-150 Biennial Update	Census + API `mcs150Outdated`	Green/Yellow/Red
3	Insurance (BIPD/Cargo/Bond)	FMCSA API `*InsuranceOnFile`	Green/Red per type
4	Safety Rating	FMCSA API `safetyRating` (S/C/U)	Green/Yellow/Red
5	Operating Authority	FMCSA API `*AuthorityStatus`	Green/Red
6	Out-of-Service Rate	FMCSA API `*OosRate` vs national avg	Green/Yellow/Red
7	Crash Record	FMCSA API `crashTotal`/`fatalCrash`	Green/Yellow/Red

State Checks (8-14) — from state_trucking_requirements table

#	Check	Trigger	Color
8	IRP Registration	Interstate (carrier_op=A) + IRP state	Yellow
9	IFTA Fuel Tax	Interstate + IFTA state	Yellow
10	Weight/Distance Tax	phy_state in OR/NY/KY/NM/CT	Yellow
11	State Carrier Permit	State requires (CA MCP, etc.)	Yellow
12	Emissions (CARB/ACT)	State has emissions program	Yellow
13	Intrastate Authority	For-hire + state requires	Yellow
14	State DOT Number	State requires (CA, TX)	Yellow

State Requirements API: GET /api/v1/dot/state-requirements?base_state=CA&operating_states=CA,OR,NV

Returns structured requirements per state + recommended services with pricing.

Services Catalog

Federal DOT Services

Slug	Service	Price	Handler
`mcs150-update`	MCS-150 Biennial Update	$69	MCS150UpdateHandler (admin todo)
`boc3-filing`	BOC-3 Process Agent Filing	$89	BOC3FilingHandler (Playwright → processagent.com)
`ucr-registration`	UCR Annual Registration	$69 + gov	MCS150UpdateHandler
`dot-registration`	New USDOT Number	$89	MCS150UpdateHandler
`mc-authority`	MC Operating Authority	$349 + $300	MCS150UpdateHandler
`dot-drug-alcohol`	D&A Compliance Program	$149/yr	MCS150UpdateHandler
`dot-audit-prep`	Safety Audit Prep	$399	MCS150UpdateHandler
`dot-full-compliance`	DOT Full Bundle	$499	MCS150UpdateHandler

State Trucking Services

Slug	Service	Price	Handler
`irp-registration`	IRP Registration	$199	StateTruckingHandler
`ifta-application`	IFTA Application + Decals	$149	StateTruckingHandler
`ifta-quarterly`	IFTA Quarterly Filing	$99	StateTruckingHandler
`or-weight-mile-tax`	Oregon Weight-Mile Tax	$199	StateTruckingHandler
`ny-hut-registration`	NY Highway Use Tax	$199	StateTruckingHandler
`ky-kyu-registration`	KY Weight-Distance Tax	$199	StateTruckingHandler
`nm-weight-distance`	NM Weight-Distance Tax	$199	StateTruckingHandler
`ct-highway-use-fee`	CT Highway Use Fee	$199	StateTruckingHandler
`ca-mcp-carb`	California MCP + CARB	$349	StateTruckingHandler
`state-dot-registration`	State DOT Registration	$149	StateTruckingHandler
`intrastate-authority`	Intrastate Authority	$249	StateTruckingHandler
`osow-permit`	Oversize/Overweight Permit	$99	StateTruckingHandler
`state-trucking-bundle`	State Compliance Bundle	$599	StateTruckingHandler

Vendors

Vendor	Service	Cost	Integration
Process Agent LLC (Registered Agents Inc)	BOC-3 blanket filing	$25/yr	Playwright on processagent.com
Northwest Registered Agent (CorpTools API)	DC registered agent, formations	Varies	REST API (JWT auth)
National Drug Screening	D&A testing programs	TBD	Reseller program (pending)
Tivly	Insurance referrals	Commission	Affiliate (pending)

Database Tables

Table	Migration	Purpose
`fmcsa_carriers`	078	2M+ carrier census records
`state_trucking_requirements`	079	51-jurisdiction state requirements
`carrier_operating_states`	080	Per-carrier operating state tracking
`compliance_orders`	(existing)	All DOT + state orders
`admin_todos`	(existing)	Admin filing tasks

Key Indexes on fmcsa_carriers

dot_number (PK)
email_address
phy_state
mcs150_parsed
authorized_for_hire
campaign_eligible

Frontend Pages

Page	Path	Type
DOT Compliance Check	`/tools/dot-compliance-check/`	Static HTML
DOT Order Page	`/order/dot-compliance/`	Static HTML
Trucking Services	`/services/trucking/`	Static HTML
California Compliance	`/services/trucking/california/`	Static HTML
Texas Compliance	`/services/trucking/texas/`	Static HTML
Oregon Compliance	`/services/trucking/oregon/`	Static HTML
New York Compliance	`/services/trucking/new-york/`	Static HTML
Florida Compliance	`/services/trucking/florida/`	Static HTML
Pricing (Trucking section)	`/pricing/`	Static HTML
8 DOT intake pages	`/order/{slug}/`	Astro (review-only)

Order Flow

1. Carrier visits /tools/dot-compliance-check
   |
2. Enters DOT# → API returns 14 checks (7 federal + 7 state)
   |
3. CTA: "Fix My DOT Filings" → /order/dot-compliance?dot=X&services=Y&state=Z
   |
4. Selects services (federal + state pre-selected from checker)
   |
5. Stripe checkout ($0 bypass if promo covers total)
   |
6. Confirmation email: "Order Confirmed — we're on it"
   |
7. Worker dispatches to handler:
   ├── BOC3FilingHandler → Playwright on processagent.com ($25)
   ├── MCS150UpdateHandler → admin todo (Login.gov MFA)
   └── StateTruckingHandler → admin todo (state-specific steps)
   |
8. Admin completes filing → sends confirmation to customer

Campaign Segmentation

Segment	Count	Criteria
All with email	49,186	email_address IS NOT NULL
Campaign eligible	29,792	Has actionable flags, not stale
Interstate (IRP/IFTA)	19,761	carrier_operation = 'A'
MCS-150 overdue	19,176	mcs150_parsed < 2 years ago
For-hire carriers	19,811	authorized_for_hire = TRUE
Intrastate authority needed	14,081	For-hire in 29 states
Emissions states	12,424	CARB/ACT states
Weight-tax states	6,289	OR/NY/KY/NM/CT
CA MCP/CARB needed	3,418	phy_state = CA

Note: Census currently has 100K of 2M records. Full download will 5-10x these numbers.

Key Files

api/
  src/routes/dot-lookup.ts          — compliance checker API (14 checks + state reqs)
  src/routes/compliance-orders.ts   — 21 DOT service slugs in COMPLIANCE_SERVICES
  src/routes/checkout.ts            — DOT_SLUGS set for "Order Confirmed" email
  migrations/078_fmcsa_carriers.sql
  migrations/079_state_trucking_requirements.sql
  migrations/080_carrier_operating_states.sql

scripts/workers/
  fmcsa_census_downloader.py        — Socrata API bulk download
  fmcsa_source_refresh.py           — WEEKLY refresh orchestrator (census ->
                                      enrichment -> flag -> verify -> mx-tag);
                                      cron pw-fmcsa-refresh (Sun 09:00 UTC), so
                                      newly-non-compliant carriers are auto-caught
  fmcsa_deficiency_flagger.py       — flag carriers + populate Listmonk lists
  services/mcs150_update.py         — MCS-150 handler (admin todo)
  services/boc3_filing.py           — BOC-3 handler (Playwright + admin fallback)
  services/boc3_playwright.py       — processagent.com Playwright automation
  services/state_trucking.py        — all state services (admin todo)

site/
  public/tools/dot-compliance-check/  — compliance checker frontend
  public/order/dot-compliance/        — batch order page
  public/services/trucking/           — landing pages (main + 5 states)
  src/lib/intake_manifest.ts          — DOT + state service manifests

9.5 KiB Raw Blame History