justin/new-site
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Add verified 500k+ email registry pipeline (FMCSA + CT SOS)

Browse source 
Section 10 of new-sector-compliance-targets.md rewritten with live-pulled,
verified results for the 'which verticals have 500k+ email leads' question:

- USPTO trademarks (#1) DISPROVEN: owner email is not in any free bulk file
  (Daily XML DTD + Case Files research dataset have no email column) and the
  TSDR/ODP API is now ID.me-auth-walled (2026-06-18). Demoted to needs-append.
- FMCSA Company Census is the real 500k+ winner: 2.93M rows w/ email_address,
  1.66M active+email (1.59M distinct), 636,278 overdue MCS-150. 92% verify.
- Connecticut SOS (#3) is the email-native state flagship: 787k w/ business
  email, 439,763 active+email, 97,466 overdue annual reports. 96% verify.
  (No other state business registry on Socrata exposes a business email.)

New scripts/build_registry_outreach_lists.py: one extensible engine with
due_date and staleness clock types; CT + FMCSA adapters; emits overdue,
due_soon, active_emailable CSVs compatible with verify_csv_emails.py.
This commit is contained in:
justin 2026-06-30 17:39:20 -05:00
parent 9abd08c305
commit 59f320ae0e
2 changed files with 533 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

176
docs/new-sector-compliance-targets.md
View file

@ -511,3 +511,179 @@ high-trust second wave.
Output CSVs: `npi_all_cold_emailable.csv`, `npi_overdue_cold_emailable.csv`,
`npi_overdue_direct_secure.csv` (NPI, email, name, specialty, state, due date,
days overdue, LEIE flag, opt-out ending).
---
## 10. Verticals With 500k+ EMAIL Leads (the volume question) — PULLED & VERIFIED
Honest framing: NPPES, our other verified free email source, only yields ~120k
*cold-emailable* inboxes. To clear **500,000+ real, public, deadline-keyed email
addresses**, you have to move to registries where the email is **published in a
free bulk/Socrata feed**. We pulled the candidates live this session. The verdict
changed materially from the first draft of this section — see §10.1.
> Status legend: ✅ pulled + verified this session · ❌ disproven this session.
> All counts below are from live pulls on the session date.
### 10.1 USPTO Trademarks — ❌ DOES NOT QUALIFY (correction)
The earlier draft called this the strongest 500k+ email vertical. **That was
wrong, and the pull proves it.** Two independent blockers:
1. **Owner email is NOT in any free bulk file.** The Trademark Daily XML DTD's
`correspondent` group is `address-1 … address-5` only — there is **no email
element**. The Trademark Case Files research dataset
(`owner.dta` / `correspondent_domrep_attorney.dta`, freely downloadable from
`data.uspto.gov/ui/datasets/products/files/TRCFECO2/...`) was checked against
its 2023 schema + variable tables: **zero email columns anywhere.** The
mandatory-email rule put the email in the *application record*, not the *bulk
distribution*. Owner email lives only in **per-record TSDR**.
2. **TSDR is now auth-walled.** As of **2026-06-18** the USPTO Open Data Portal
requires a USPTO.gov account, and the API key requires **ID.me identity
verification**; the legacy open `bulkdata.uspto.gov` is decommissioned and the
TSDR bulk API now returns "register for an API key." So even per-record email
scraping needs an ID.me-verified account + key.
Net: USPTO trademarks remain a great *audience* with a clean §8/§9 renewal clock,
but there is **no free, bulk, email-native path** to them. It belongs in the
"address/phone-only, needs email-append" bucket (like NPPES/EPA), not the 500k+
email bucket. The slot is replaced by FMCSA below.
### 10.2 FMCSA Company Census — ✅ THE 500k+ WINNER (`email_address`)
- **Source:** `data.transportation.gov/resource/az4n-8mr2` (Company Census File),
free Socrata, no auth. **4,459,640 rows total.**
- **Email (VERIFIED):** **2,927,276 rows carry `email_address` (2,791,522
distinct).** Filtered to **`status_code = 'A'` (active) + email = 1,668,135**.
Our live build (active + usable-syntax email) produced **1,661,813 rows,
1,588,732 distinct emails.**
- **Dateable clock (VERIFIED):** `mcs150_date` (format `YYYYMMDD HHMM`) + **24
months** = biennial MCS-150 due date. Live build:
- **636,278 OVERDUE** MCS-150 updates (hottest leads, alone >500k).
- **73,781 due within 90 days** (pre-emptive).
- **Deliverability spot-check:** most-overdue 500-row sample → **459/500 (92%)
sendable** after MX/SMTP. (This was the *worst* cohort — mcs150 dates back to
2003 with dead domains like `alltel.net`; fresher overdue cohorts verify higher.
CT below hit 96% on the same test.)
- **Offer:** DOT compliance renewal bundle — already in our wheelhouse (we run
MCS-150 update, UCR, BOC-3 today). This just bolts the **email channel** onto
the existing trucking playbook; previously we leaned on phone/mail.
- **Bonus columns:** `phone`, `dot_number`, physical + mailing address,
`carrier_operation`, fleet size, `hm_ind` (hazmat) — rich segmentation.
### 10.3 Connecticut SOS Business Registry — ✅ QUALIFIES on its own (`business_email_address`)
- **Source:** `data.ct.gov/resource/n7gp-d28j` (CT Business Registry – Business
Master), free Socrata, no auth. **1,283,770 rows total.** CT is the standout: a
full Socrata sweep of `.gov` open-data portals found **no other state
business-entity registry that publishes a business email** (CO, MD, NY, WA, IA
master files have address only; CT is unique in exposing the email).
- **Email (VERIFIED):** **787,674 rows carry `business_email_address`.** Filtered
to `status = 'Active'` + email: live build = **439,763 rows, 303,299 distinct
emails.** (Distinct alone is below 500k — CT clears the *row* bar and is the
email-native SOS flagship, but pair it with FMCSA for the combined 500k+
distinct.)
- **Dateable clock (VERIFIED):** `annual_report_due_date` (literal column). Live
build: **97,466 OVERDUE annual reports**, **16,235 due within 90 days.**
- **Deliverability spot-check:** most-overdue 500-row sample → **482/500 (96%)
sendable** after MX/SMTP (18 hard-fails).
- **Offer:** annual-report filing + registered-agent + reinstatement (core
Performance West products).
- **Bonus columns:** `naics_code` (vertical targeting), registration date,
business type, woman/veteran/minority-owned flags.
### 10.4 Combined verified email universe (this session)
| Registry | Active + email rows | Distinct emails | Overdue (hottest) | Due ≤90d | Verify rate |
|---|---|---|---|---|---|
| FMCSA Company Census | **1,661,813** | **1,588,732** | **636,278** | 73,781 | 92% (worst cohort) |
| Connecticut SOS | **439,763** | 303,299 | **97,466** | 16,235 | 96% |
| **Combined** | **~2.10M** | **~1.89M** | **~733k** | ~90k | — |
Either one of **FMCSA alone (1.59M distinct)** or **FMCSA+CT (~1.89M distinct)**
decisively clears the 500k+ goal with a real, dateable deficiency hook on each
record. The **overdue segments alone (~733k)** already exceed 500k.
### 10.5 Honorable mentions that DON'T clear 500k email
- **USPTO trademarks:** email not in free bulk; API ID.me-walled (§10.1). ❌
- **NPPES (NPI):** ~120k cold inboxes only (§8). Great audience, wrong scale. ✅verified earlier.
- **FAA airmen/aircraft (~300k):** email is opt-in only — small usable volume.
- **ATF FFL, FMC OTI:** tiny universes (thousands). Quality, not volume.
- **EPA RCRA/ECHO:** **no email at all** in bulk (§3). Volume but zero native email.
- **Other state SOS master files (CO/MD/NY/WA/IA):** address only, no email column.
### 10.6 The pipeline (built this session)
`scripts/build_registry_outreach_lists.py` — one extensible engine for any free
registry that exposes **email + a dateable clock**. Two clock types:
- `due_date` : a column already holds the obligation's due date (CT annual report).
- `staleness` : a column holds last-filed date; due = last + `interval_days`
(FMCSA `mcs150_date` + 730d).
Adding a registry = one `Registry(...)` entry (Socrata host, resource id, column
map). It server-side-filters to active + email-present, computes overdue/due-soon,
and writes three campaign-ready CSVs per registry (every row has an `email`
column, matching `verify_csv_emails.py`'s contract):
```
<key>_overdue.csv obligation PAST DUE (lead with the deadline)
<key>_due_soon.csv due within --soon-days (pre-emptive offer)
<key>_active_emailable.csv all active + email (broad campaign)
```
Run:
```
python3 scripts/build_registry_outreach_lists.py --registries fmcsa,ct
python3 scripts/verify_csv_emails.py --in out/fmcsa_overdue.csv --out out/fmcsa_overdue
```
Then import the `_verified.csv` into the campaign tool. MX/SMTP verification runs
on the NON-sending IP (port 25 + MX confirmed working on our infra) so it never
touches the warmed MTA reputation.
### 10.7 Recommended sequencing
1. **FMCSA overdue (636k)** — biggest single email-native deadline pool, and it
reuses our existing DOT-compliance fulfillment. Start here.
2. **CT overdue annual reports (97k)** — highest verify rate (96%), clean
annual-report + reinstatement offer, NAICS for sub-segmenting.
3. **FMCSA due-soon + CT due-soon (~90k)** — pre-emptive "due in <90 days" offers.
4. Broaden SOS coverage only if another state is found to expose a business email
(none did this session beyond CT).
---
## 11. Extra High-Volume Public-Email Candidates to Verify
These are second-ring ideas after USPTO, FMCSA, and SOS files. They may not each clear 500k alone, but they can contribute to a national email-native compliance lead graph.
| Candidate | Email likelihood | Deadline/trigger | Product angle | Notes |
|---|---|---|---|---|
| Professional license mega-boards | Medium-high in some states | Renewal, CE, bond/insurance, status | License renewal calendar | Prioritize establishment/company records over individual professionals. |
| City/county business license portals | Medium, fragmented | Annual local license or business tax certificate | Local filing support | Huge aggregate universe, but ETL-heavy. |
| State insurance producer/agency files | Often email-rich | License renewal, CE, appointment status | Renewal/CE monitoring | Paperwork familiarity is higher, so focus on agencies/small shops. |
| Real estate brokerage firm files | Sometimes email-rich | Firm/branch/DB renewal | Firm compliance calendar | Avoid individual agents first. |
| Cosmetology/barber establishment files | Sometimes email-rich | Shop license renewal, inspection status | Shop renewal support | Strong small-operator fit. |
| Motor vehicle dealer/repair facility files | Sometimes email-rich | Dealer/repair license, bond | Renewal/bond packet | Good business-critical lapse angle. |
| Local health/food permits | Sometimes email-rich | Food permit renewal, inspection | Permit calendar | Lower ticket unless bundled. |
**Verification rule:** do not count a registry as email-native until a current bulk/API pull confirms (1) email column present, (2) enough distinct deliverable emails after MX/SMTP verification, (3) a computable recurring deadline, and (4) the email belongs to the operator rather than a filing intermediary.
### 11.1 Additional verification queue
If USPTO/FMCSA/SOS are already being pulled, verify these next because they may expose public email or become strong email-enriched compliance lists:
1. **FCC ULS wireless licensees** — expiring private land mobile/business licenses, construction deadlines, ULS contact cleanup.
2. **FCC ASR tower owners** — smaller but high-value tower compliance and owner/contact update work.
3. **FCC broadcast station owners** — public-file and renewal admin audits, high value but counsel-heavy.
4. **State insurance agencies/producers** — huge volume, better at agency level than individual producers.
5. **Real estate brokerage firm/branch licenses** — firm/branch renewal and designated-broker cleanup.
6. **Umbrella occupational license portals** — find states whose detail APIs expose emails, then pick the best boards.
7. **Pest-control/pesticide businesses** — strong field-service fit with license/CE/roster deadlines.
8. **Dealer/repair/dismantler/inspection-station licenses** — business-critical renewals and bond/insurance tracking.
9. **Childcare provider directories** — public-facing emails plus license/inspection status, careful tone required.
10. **CLIA/state lab licenses** — two-year certificate renewal, contact-light federally but enrichable through state/provider directories.
11. **State controlled-substance registrations** — high-consequence renewal calendar, sensitive messaging.
12. **State Medicaid/DMEPOS provider lists** — extend NPI/PECOS revalidation into provider-enrollment monitoring.
13. **Tobacco retailer/product registries** — recurring retail permits and compliance-check outcomes.
14. **Cannabis/hemp/product registries** — email-rich in some states, high compliance pain, platform restrictions.
15. **Energy/solar installer directories** — public emails in customer-facing lists plus contractor/vendor renewals.
16. **Public procurement/vendor certification lists** — email-rich, but filter to trades/operators rather than professional grant/SAM filers.
17. **Lottery/gaming/amusement licenses** — recurring permits for retailers, route operators, and venues.
18. **Public-facing facility directories joined to licensing data** — use directory email plus regulator deficiency signal.
19. **RDAP/website/domain enrichment** — not a compliance list by itself, but a domain discovery layer for SOS/USPTO/local permits.
20. **MSB/NMLS/state financial licenses** — smaller but high-value renewal, bond, branch, and delegate-update work.
Prioritize any source where the first sample shows: public email, explicit expiration date, business/facility record type, and a filing we can complete administratively for a fixed price.