justin/new-site
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

trucking: weekly FMCSA source refresh so new non-compliant carriers are caught

Browse source 
The FMCSA census was a one-time snapshot (last loaded ~May 30) with NO refresh
timer -- carriers newly falling out of MCS-150/UCR compliance were never picked
up. New scripts/workers/fmcsa_source_refresh.py orchestrates the full pipeline
(census download -> enrichment -> deficiency flag -> verify new emails ->
MX-tag new) and runs weekly via cron pw-fmcsa-refresh (Sun 09:00 UTC), codified
in the mail-pipeline Ansible role.

Idempotent + incremental: the census upsert preserves email_verified /
listmonk_sent_at / deficiency_flags, so existing carriers keep their send state
and only census fields refresh; new DOTs flow into verification then campaigns.
A carrier who refiled gets a fresh mcs150_parsed, so the builder's overdue
WHERE clause stops targeting them automatically. Verify is capped per run
(20k) so it never stalls on millions of rows.

(Healthcare already auto-catches newly-revalidation-overdue providers within
its 63k institutional pool via pw-hc-refresh Mon/Wed/Fri.)
This commit is contained in:
justin 2026-06-17 20:44:54 -05:00
parent 4171f48736
commit 899b880e7f
4 changed files with 144 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

4
docs/trucking-system.md
View file

@ -196,6 +196,10 @@ api/
scripts/workers/
fmcsa_census_downloader.py — Socrata API bulk download
fmcsa_source_refresh.py — WEEKLY refresh orchestrator (census ->
enrichment -> flag -> verify -> mx-tag);
cron pw-fmcsa-refresh (Sun 09:00 UTC), so
newly-non-compliant carriers are auto-caught
fmcsa_deficiency_flagger.py — flag carriers + populate Listmonk lists
services/mcs150_update.py — MCS-150 handler (admin todo)
services/boc3_filing.py — BOC-3 handler (Playwright + admin fallback)