Add verified 500k+ email registry pipeline (FMCSA + CT SOS)
Section 10 of new-sector-compliance-targets.md rewritten with live-pulled, verified results for the 'which verticals have 500k+ email leads' question: - USPTO trademarks (#1) DISPROVEN: owner email is not in any free bulk file (Daily XML DTD + Case Files research dataset have no email column) and the TSDR/ODP API is now ID.me-auth-walled (2026-06-18). Demoted to needs-append. - FMCSA Company Census is the real 500k+ winner: 2.93M rows w/ email_address, 1.66M active+email (1.59M distinct), 636,278 overdue MCS-150. 92% verify. - Connecticut SOS (#3) is the email-native state flagship: 787k w/ business email, 439,763 active+email, 97,466 overdue annual reports. 96% verify. (No other state business registry on Socrata exposes a business email.) New scripts/build_registry_outreach_lists.py: one extensible engine with due_date and staleness clock types; CT + FMCSA adapters; emits overdue, due_soon, active_emailable CSVs compatible with verify_csv_emails.py.
This commit is contained in:
justin
parent
9abd08c305
commit
59f320ae0e
2 changed files with 533 additions and 0 deletions
|
|
@ -511,3 +511,179 @@ high-trust second wave.
|
|||
Output CSVs: `npi_all_cold_emailable.csv`, `npi_overdue_cold_emailable.csv`,
|
||||
`npi_overdue_direct_secure.csv` (NPI, email, name, specialty, state, due date,
|
||||
days overdue, LEIE flag, opt-out ending).
|
||||
|
||||
---
|
||||
|
||||
## 10. Verticals With 500k+ EMAIL Leads (the volume question) — PULLED & VERIFIED
|
||||
|
||||
Honest framing: NPPES, our other verified free email source, only yields ~120k
|
||||
*cold-emailable* inboxes. To clear **500,000+ real, public, deadline-keyed email
|
||||
addresses**, you have to move to registries where the email is **published in a
|
||||
free bulk/Socrata feed**. We pulled the candidates live this session. The verdict
|
||||
changed materially from the first draft of this section — see §10.1.
|
||||
|
||||
> Status legend: ✅ pulled + verified this session · ❌ disproven this session.
|
||||
> All counts below are from live pulls on the session date.
|
||||
|
||||
### 10.1 USPTO Trademarks — ❌ DOES NOT QUALIFY (correction)
|
||||
|
||||
The earlier draft called this the strongest 500k+ email vertical. **That was
|
||||
wrong, and the pull proves it.** Two independent blockers:
|
||||
|
||||
1. **Owner email is NOT in any free bulk file.** The Trademark Daily XML DTD's
|
||||
`correspondent` group is `address-1 … address-5` only — there is **no email
|
||||
element**. The Trademark Case Files research dataset
|
||||
(`owner.dta` / `correspondent_domrep_attorney.dta`, freely downloadable from
|
||||
`data.uspto.gov/ui/datasets/products/files/TRCFECO2/...`) was checked against
|
||||
its 2023 schema + variable tables: **zero email columns anywhere.** The
|
||||
mandatory-email rule put the email in the *application record*, not the *bulk
|
||||
distribution*. Owner email lives only in **per-record TSDR**.
|
||||
2. **TSDR is now auth-walled.** As of **2026-06-18** the USPTO Open Data Portal
|
||||
requires a USPTO.gov account, and the API key requires **ID.me identity
|
||||
verification**; the legacy open `bulkdata.uspto.gov` is decommissioned and the
|
||||
TSDR bulk API now returns "register for an API key." So even per-record email
|
||||
scraping needs an ID.me-verified account + key.
|
||||
|
||||
Net: USPTO trademarks remain a great *audience* with a clean §8/§9 renewal clock,
|
||||
but there is **no free, bulk, email-native path** to them. It belongs in the
|
||||
"address/phone-only, needs email-append" bucket (like NPPES/EPA), not the 500k+
|
||||
email bucket. The slot is replaced by FMCSA below.
|
||||
|
||||
### 10.2 FMCSA Company Census — ✅ THE 500k+ WINNER (`email_address`)
|
||||
- **Source:** `data.transportation.gov/resource/az4n-8mr2` (Company Census File),
|
||||
free Socrata, no auth. **4,459,640 rows total.**
|
||||
- **Email (VERIFIED):** **2,927,276 rows carry `email_address` (2,791,522
|
||||
distinct).** Filtered to **`status_code = 'A'` (active) + email = 1,668,135**.
|
||||
Our live build (active + usable-syntax email) produced **1,661,813 rows,
|
||||
1,588,732 distinct emails.**
|
||||
- **Dateable clock (VERIFIED):** `mcs150_date` (format `YYYYMMDD HHMM`) + **24
|
||||
months** = biennial MCS-150 due date. Live build:
|
||||
- **636,278 OVERDUE** MCS-150 updates (hottest leads, alone >500k).
|
||||
- **73,781 due within 90 days** (pre-emptive).
|
||||
- **Deliverability spot-check:** most-overdue 500-row sample → **459/500 (92%)
|
||||
sendable** after MX/SMTP. (This was the *worst* cohort — mcs150 dates back to
|
||||
2003 with dead domains like `alltel.net`; fresher overdue cohorts verify higher.
|
||||
CT below hit 96% on the same test.)
|
||||
- **Offer:** DOT compliance renewal bundle — already in our wheelhouse (we run
|
||||
MCS-150 update, UCR, BOC-3 today). This just bolts the **email channel** onto
|
||||
the existing trucking playbook; previously we leaned on phone/mail.
|
||||
- **Bonus columns:** `phone`, `dot_number`, physical + mailing address,
|
||||
`carrier_operation`, fleet size, `hm_ind` (hazmat) — rich segmentation.
|
||||
|
||||
### 10.3 Connecticut SOS Business Registry — ✅ QUALIFIES on its own (`business_email_address`)
|
||||
- **Source:** `data.ct.gov/resource/n7gp-d28j` (CT Business Registry – Business
|
||||
Master), free Socrata, no auth. **1,283,770 rows total.** CT is the standout: a
|
||||
full Socrata sweep of `.gov` open-data portals found **no other state
|
||||
business-entity registry that publishes a business email** (CO, MD, NY, WA, IA
|
||||
master files have address only; CT is unique in exposing the email).
|
||||
- **Email (VERIFIED):** **787,674 rows carry `business_email_address`.** Filtered
|
||||
to `status = 'Active'` + email: live build = **439,763 rows, 303,299 distinct
|
||||
emails.** (Distinct alone is below 500k — CT clears the *row* bar and is the
|
||||
email-native SOS flagship, but pair it with FMCSA for the combined 500k+
|
||||
distinct.)
|
||||
- **Dateable clock (VERIFIED):** `annual_report_due_date` (literal column). Live
|
||||
build: **97,466 OVERDUE annual reports**, **16,235 due within 90 days.**
|
||||
- **Deliverability spot-check:** most-overdue 500-row sample → **482/500 (96%)
|
||||
sendable** after MX/SMTP (18 hard-fails).
|
||||
- **Offer:** annual-report filing + registered-agent + reinstatement (core
|
||||
Performance West products).
|
||||
- **Bonus columns:** `naics_code` (vertical targeting), registration date,
|
||||
business type, woman/veteran/minority-owned flags.
|
||||
|
||||
### 10.4 Combined verified email universe (this session)
|
||||
| Registry | Active + email rows | Distinct emails | Overdue (hottest) | Due ≤90d | Verify rate |
|
||||
|---|---|---|---|---|---|
|
||||
| FMCSA Company Census | **1,661,813** | **1,588,732** | **636,278** | 73,781 | 92% (worst cohort) |
|
||||
| Connecticut SOS | **439,763** | 303,299 | **97,466** | 16,235 | 96% |
|
||||
| **Combined** | **~2.10M** | **~1.89M** | **~733k** | ~90k | — |
|
||||
|
||||
Either one of **FMCSA alone (1.59M distinct)** or **FMCSA+CT (~1.89M distinct)**
|
||||
decisively clears the 500k+ goal with a real, dateable deficiency hook on each
|
||||
record. The **overdue segments alone (~733k)** already exceed 500k.
|
||||
|
||||
### 10.5 Honorable mentions that DON'T clear 500k email
|
||||
- **USPTO trademarks:** email not in free bulk; API ID.me-walled (§10.1). ❌
|
||||
- **NPPES (NPI):** ~120k cold inboxes only (§8). Great audience, wrong scale. ✅verified earlier.
|
||||
- **FAA airmen/aircraft (~300k):** email is opt-in only — small usable volume.
|
||||
- **ATF FFL, FMC OTI:** tiny universes (thousands). Quality, not volume.
|
||||
- **EPA RCRA/ECHO:** **no email at all** in bulk (§3). Volume but zero native email.
|
||||
- **Other state SOS master files (CO/MD/NY/WA/IA):** address only, no email column.
|
||||
|
||||
### 10.6 The pipeline (built this session)
|
||||
`scripts/build_registry_outreach_lists.py` — one extensible engine for any free
|
||||
registry that exposes **email + a dateable clock**. Two clock types:
|
||||
- `due_date` : a column already holds the obligation's due date (CT annual report).
|
||||
- `staleness` : a column holds last-filed date; due = last + `interval_days`
|
||||
(FMCSA `mcs150_date` + 730d).
|
||||
|
||||
Adding a registry = one `Registry(...)` entry (Socrata host, resource id, column
|
||||
map). It server-side-filters to active + email-present, computes overdue/due-soon,
|
||||
and writes three campaign-ready CSVs per registry (every row has an `email`
|
||||
column, matching `verify_csv_emails.py`'s contract):
|
||||
```
|
||||
<key>_overdue.csv obligation PAST DUE (lead with the deadline)
|
||||
<key>_due_soon.csv due within --soon-days (pre-emptive offer)
|
||||
<key>_active_emailable.csv all active + email (broad campaign)
|
||||
```
|
||||
Run:
|
||||
```
|
||||
python3 scripts/build_registry_outreach_lists.py --registries fmcsa,ct
|
||||
python3 scripts/verify_csv_emails.py --in out/fmcsa_overdue.csv --out out/fmcsa_overdue
|
||||
```
|
||||
Then import the `_verified.csv` into the campaign tool. MX/SMTP verification runs
|
||||
on the NON-sending IP (port 25 + MX confirmed working on our infra) so it never
|
||||
touches the warmed MTA reputation.
|
||||
|
||||
### 10.7 Recommended sequencing
|
||||
1. **FMCSA overdue (636k)** — biggest single email-native deadline pool, and it
|
||||
reuses our existing DOT-compliance fulfillment. Start here.
|
||||
2. **CT overdue annual reports (97k)** — highest verify rate (96%), clean
|
||||
annual-report + reinstatement offer, NAICS for sub-segmenting.
|
||||
3. **FMCSA due-soon + CT due-soon (~90k)** — pre-emptive "due in <90 days" offers.
|
||||
4. Broaden SOS coverage only if another state is found to expose a business email
|
||||
(none did this session beyond CT).
|
||||
|
||||
---
|
||||
|
||||
## 11. Extra High-Volume Public-Email Candidates to Verify
|
||||
|
||||
These are second-ring ideas after USPTO, FMCSA, and SOS files. They may not each clear 500k alone, but they can contribute to a national email-native compliance lead graph.
|
||||
|
||||
| Candidate | Email likelihood | Deadline/trigger | Product angle | Notes |
|
||||
|---|---|---|---|---|
|
||||
| Professional license mega-boards | Medium-high in some states | Renewal, CE, bond/insurance, status | License renewal calendar | Prioritize establishment/company records over individual professionals. |
|
||||
| City/county business license portals | Medium, fragmented | Annual local license or business tax certificate | Local filing support | Huge aggregate universe, but ETL-heavy. |
|
||||
| State insurance producer/agency files | Often email-rich | License renewal, CE, appointment status | Renewal/CE monitoring | Paperwork familiarity is higher, so focus on agencies/small shops. |
|
||||
| Real estate brokerage firm files | Sometimes email-rich | Firm/branch/DB renewal | Firm compliance calendar | Avoid individual agents first. |
|
||||
| Cosmetology/barber establishment files | Sometimes email-rich | Shop license renewal, inspection status | Shop renewal support | Strong small-operator fit. |
|
||||
| Motor vehicle dealer/repair facility files | Sometimes email-rich | Dealer/repair license, bond | Renewal/bond packet | Good business-critical lapse angle. |
|
||||
| Local health/food permits | Sometimes email-rich | Food permit renewal, inspection | Permit calendar | Lower ticket unless bundled. |
|
||||
|
||||
**Verification rule:** do not count a registry as email-native until a current bulk/API pull confirms (1) email column present, (2) enough distinct deliverable emails after MX/SMTP verification, (3) a computable recurring deadline, and (4) the email belongs to the operator rather than a filing intermediary.
|
||||
|
||||
### 11.1 Additional verification queue
|
||||
|
||||
If USPTO/FMCSA/SOS are already being pulled, verify these next because they may expose public email or become strong email-enriched compliance lists:
|
||||
|
||||
1. **FCC ULS wireless licensees** — expiring private land mobile/business licenses, construction deadlines, ULS contact cleanup.
|
||||
2. **FCC ASR tower owners** — smaller but high-value tower compliance and owner/contact update work.
|
||||
3. **FCC broadcast station owners** — public-file and renewal admin audits, high value but counsel-heavy.
|
||||
4. **State insurance agencies/producers** — huge volume, better at agency level than individual producers.
|
||||
5. **Real estate brokerage firm/branch licenses** — firm/branch renewal and designated-broker cleanup.
|
||||
6. **Umbrella occupational license portals** — find states whose detail APIs expose emails, then pick the best boards.
|
||||
7. **Pest-control/pesticide businesses** — strong field-service fit with license/CE/roster deadlines.
|
||||
8. **Dealer/repair/dismantler/inspection-station licenses** — business-critical renewals and bond/insurance tracking.
|
||||
9. **Childcare provider directories** — public-facing emails plus license/inspection status, careful tone required.
|
||||
10. **CLIA/state lab licenses** — two-year certificate renewal, contact-light federally but enrichable through state/provider directories.
|
||||
11. **State controlled-substance registrations** — high-consequence renewal calendar, sensitive messaging.
|
||||
12. **State Medicaid/DMEPOS provider lists** — extend NPI/PECOS revalidation into provider-enrollment monitoring.
|
||||
13. **Tobacco retailer/product registries** — recurring retail permits and compliance-check outcomes.
|
||||
14. **Cannabis/hemp/product registries** — email-rich in some states, high compliance pain, platform restrictions.
|
||||
15. **Energy/solar installer directories** — public emails in customer-facing lists plus contractor/vendor renewals.
|
||||
16. **Public procurement/vendor certification lists** — email-rich, but filter to trades/operators rather than professional grant/SAM filers.
|
||||
17. **Lottery/gaming/amusement licenses** — recurring permits for retailers, route operators, and venues.
|
||||
18. **Public-facing facility directories joined to licensing data** — use directory email plus regulator deficiency signal.
|
||||
19. **RDAP/website/domain enrichment** — not a compliance list by itself, but a domain discovery layer for SOS/USPTO/local permits.
|
||||
20. **MSB/NMLS/state financial licenses** — smaller but high-value renewal, bond, branch, and delegate-update work.
|
||||
|
||||
Prioritize any source where the first sample shows: public email, explicit expiration date, business/facility record type, and a filing we can complete administratively for a fixed price.
|
||||
|
|
|
|||
357
scripts/build_registry_outreach_lists.py
Normal file
357
scripts/build_registry_outreach_lists.py
Normal file
|
|
@ -0,0 +1,357 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Build public-registry outreach lists with EMAIL + a dateable compliance clock.
|
||||
|
||||
Implements the "500k+ email leads" verticals from
|
||||
docs/new-sector-compliance-targets.md sec 10. The winning pattern is always:
|
||||
|
||||
public registry -> recurring obligation -> automated deficiency check
|
||||
-> EMAIL outreach -> fixed-price filing service.
|
||||
|
||||
Unlike NPPES (email-poor) and EPA RCRA (no email at all), these registries expose
|
||||
a real, cold-emailable contact email in a FREE bulk/Socrata feed AND a column we
|
||||
can turn into a dateable "your X is due/overdue" hook:
|
||||
|
||||
ct Connecticut SOS Business Registry -- VERIFIED this session
|
||||
data.ct.gov/n7gp-d28j | email: business_email_address
|
||||
clock: annual_report_due_date (annual report) -> overdue/due-soon
|
||||
~439,763 ACTIVE + email (303k distinct), ~97,466 OVERDUE annual reports.
|
||||
|
||||
fmcsa FMCSA Company Census (motor carriers) -- VERIFIED this session
|
||||
data.transportation.gov/az4n-8mr2 | email: email_address
|
||||
clock: mcs150_date + 24mo (biennial MCS-150 update) -> overdue/due-soon
|
||||
~2,927,276 rows with email (2.79M distinct), ~1,668,135 ACTIVE + email.
|
||||
This is the genuine 500k+ email vertical (replaces USPTO trademarks,
|
||||
whose owner email is NOT in any free bulk file and whose API is now
|
||||
ID.me-auth-walled as of 2026-06-18).
|
||||
|
||||
Two clock types are supported per adapter:
|
||||
* "due_date" : a column already holds the obligation's due date.
|
||||
* "staleness" : a column holds the last-filed date; due = last + interval_days.
|
||||
|
||||
Adding a registry = one Registry entry (Socrata host, resource, column mapping).
|
||||
|
||||
Output (every row has an `email` column, per verify_csv_emails.py's contract):
|
||||
out/<key>_overdue.csv obligation PAST DUE (hottest leads)
|
||||
out/<key>_due_soon.csv due within --soon-days (pre-emptive)
|
||||
out/<key>_active_emailable.csv all active + email (broad campaign)
|
||||
|
||||
Then MX/SMTP-verify on the NON-sending IP before importing:
|
||||
python3 scripts/verify_csv_emails.py --in out/ct_overdue.csv --out out/ct_overdue
|
||||
|
||||
Usage:
|
||||
python3 scripts/build_registry_outreach_lists.py --registries ct,fmcsa
|
||||
python3 scripts/build_registry_outreach_lists.py --registries ct --max-rows 5000
|
||||
python3 scripts/build_registry_outreach_lists.py --list
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import csv
|
||||
import datetime
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
from collections import Counter
|
||||
from dataclasses import dataclass
|
||||
|
||||
UA = "performancewest-registry-outreach/1.0 (+https://performancewest.net)"
|
||||
|
||||
|
||||
def email_ok(e: str) -> bool:
|
||||
e = (e or "").strip()
|
||||
return bool(e) and "@" in e and "." in e.split("@")[-1] and " " not in e
|
||||
|
||||
|
||||
@dataclass
|
||||
class Registry:
|
||||
"""One public registry that exposes a cold-emailable contact + a clock."""
|
||||
|
||||
key: str
|
||||
name: str
|
||||
host: str # Socrata host, e.g. data.ct.gov
|
||||
resource: str # Socrata 4x4, e.g. n7gp-d28j
|
||||
email_field: str # primary cold-emailable email column
|
||||
name_field: str
|
||||
obligation: str # human label, e.g. "annual report"
|
||||
clock: str # "due_date" | "staleness"
|
||||
clock_field: str # due-date col, OR last-filed col for staleness
|
||||
interval_days: int = 0 # for staleness: days from last-filed to due
|
||||
status_field: str | None = None
|
||||
active_values: tuple[str, ...] = ()
|
||||
alt_email_field: str | None = None
|
||||
id_field: str | None = None
|
||||
type_field: str | None = None
|
||||
naics_field: str | None = None
|
||||
city_field: str | None = None
|
||||
state_field: str | None = None
|
||||
phone_field: str | None = None
|
||||
notes: str = ""
|
||||
|
||||
|
||||
REGISTRIES: dict[str, Registry] = {
|
||||
"ct": Registry(
|
||||
key="ct",
|
||||
name="Connecticut SOS Business Registry",
|
||||
host="data.ct.gov",
|
||||
resource="n7gp-d28j",
|
||||
email_field="business_email_address",
|
||||
alt_email_field="category_survey_email_address",
|
||||
name_field="name",
|
||||
obligation="annual report",
|
||||
clock="due_date",
|
||||
clock_field="annual_report_due_date",
|
||||
status_field="status",
|
||||
active_values=("active",),
|
||||
id_field="accountnumber",
|
||||
type_field="business_type",
|
||||
naics_field="naics_code",
|
||||
city_field="billingcity",
|
||||
state_field="billingstate",
|
||||
notes="business_email_address is a real cold-emailable inbox; "
|
||||
"annual_report_due_date is the dateable annual-report clock.",
|
||||
),
|
||||
"fmcsa": Registry(
|
||||
key="fmcsa",
|
||||
name="FMCSA Company Census (motor carriers)",
|
||||
host="data.transportation.gov",
|
||||
resource="az4n-8mr2",
|
||||
email_field="email_address",
|
||||
name_field="legal_name",
|
||||
obligation="MCS-150 biennial update",
|
||||
clock="staleness",
|
||||
clock_field="mcs150_date",
|
||||
interval_days=730, # biennial
|
||||
status_field="status_code",
|
||||
active_values=("a",), # A = active
|
||||
id_field="dot_number",
|
||||
type_field="carrier_operation",
|
||||
city_field="phy_city",
|
||||
state_field="phy_state",
|
||||
phone_field="phone",
|
||||
notes="email_address coverage ~2.9M rows; mcs150_date + 24mo is the "
|
||||
"biennial MCS-150 staleness clock. Replaces USPTO (now auth-walled).",
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def socrata_get(host: str, resource: str, params: dict) -> list[dict]:
|
||||
url = f"https://{host}/resource/{resource}.json?{urllib.parse.urlencode(params)}"
|
||||
req = urllib.request.Request(url, headers={"User-Agent": UA, "Accept": "application/json"})
|
||||
with urllib.request.urlopen(req, timeout=180) as r:
|
||||
return json.loads(r.read().decode("utf-8"))
|
||||
|
||||
|
||||
def socrata_count(host: str, resource: str, where: str | None = None) -> int:
|
||||
params = {"$select": "count(1)"}
|
||||
if where:
|
||||
params["$where"] = where
|
||||
rows = socrata_get(host, resource, params)
|
||||
return int(rows[0].get("count_1", 0)) if rows else 0
|
||||
|
||||
|
||||
def parse_date(val: str | None) -> datetime.date | None:
|
||||
if not val:
|
||||
return None
|
||||
val = val.strip()
|
||||
for fmt in ("%Y-%m-%dT%H:%M:%S.%f", "%Y-%m-%dT%H:%M:%S", "%Y-%m-%d",
|
||||
"%m/%d/%Y", "%m/%d/%Y %H:%M:%S",
|
||||
"%Y%m%d %H%M", "%Y%m%d"): # FMCSA mcs150_date = 'YYYYMMDD HHMM'
|
||||
try:
|
||||
return datetime.datetime.strptime(val, fmt).date()
|
||||
except ValueError:
|
||||
continue
|
||||
try:
|
||||
return datetime.datetime.strptime(val[:10], "%Y-%m-%d").date()
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
|
||||
def due_date_for(reg: Registry, raw: str | None) -> datetime.date | None:
|
||||
"""Resolve the obligation's due date from the registry's clock column."""
|
||||
d = parse_date(raw)
|
||||
if d is None:
|
||||
return None
|
||||
if reg.clock == "staleness":
|
||||
return d + datetime.timedelta(days=reg.interval_days)
|
||||
return d # due_date
|
||||
|
||||
|
||||
def fetch_active_emailable(reg: Registry, max_rows: int | None, page: int = 50000):
|
||||
"""Yield raw rows: active records that carry a usable email (server-filtered)."""
|
||||
cols = [c for c in {
|
||||
reg.id_field, reg.name_field, reg.email_field, reg.alt_email_field,
|
||||
reg.status_field, reg.clock_field, reg.type_field, reg.naics_field,
|
||||
reg.city_field, reg.state_field, reg.phone_field,
|
||||
} if c]
|
||||
|
||||
where = [f"{reg.email_field} IS NOT NULL"]
|
||||
if reg.status_field and reg.active_values:
|
||||
ors = " OR ".join(f"upper({reg.status_field})='{v.upper()}'" for v in reg.active_values)
|
||||
where.append(f"({ors})")
|
||||
where_s = " AND ".join(where)
|
||||
|
||||
offset, pulled = 0, 0
|
||||
while True:
|
||||
params = {
|
||||
"$select": ",".join(cols),
|
||||
"$where": where_s,
|
||||
"$order": (reg.id_field or reg.name_field),
|
||||
"$limit": page,
|
||||
"$offset": offset,
|
||||
}
|
||||
rows = socrata_get(reg.host, reg.resource, params)
|
||||
if not rows:
|
||||
break
|
||||
for r in rows:
|
||||
email = (r.get(reg.email_field) or "").strip()
|
||||
if not email_ok(email) and reg.alt_email_field:
|
||||
email = (r.get(reg.alt_email_field) or "").strip()
|
||||
if not email_ok(email):
|
||||
continue
|
||||
yield r, email.lower()
|
||||
pulled += 1
|
||||
if max_rows and pulled >= max_rows:
|
||||
return
|
||||
offset += len(rows)
|
||||
if len(rows) < page:
|
||||
break
|
||||
|
||||
|
||||
HEADER = ["registry", "entity_id", "name", "email", "phone", "entity_type",
|
||||
"naics", "city", "biz_state", "status", "obligation",
|
||||
"due_date", "days_overdue", "deficiency", "last_filed"]
|
||||
|
||||
|
||||
def enrich(reg: Registry, r: dict, email: str, today: datetime.date) -> dict:
|
||||
raw = r.get(reg.clock_field) if reg.clock_field else None
|
||||
last_filed = parse_date(raw) if reg.clock == "staleness" else None
|
||||
due = due_date_for(reg, raw)
|
||||
if due:
|
||||
days_overdue = (today - due).days
|
||||
deficiency = "overdue" if days_overdue > 0 else "upcoming"
|
||||
else:
|
||||
days_overdue, deficiency = None, "no_date"
|
||||
return {
|
||||
"registry": reg.key,
|
||||
"entity_id": r.get(reg.id_field, "") if reg.id_field else "",
|
||||
"name": r.get(reg.name_field, ""),
|
||||
"email": email,
|
||||
"phone": r.get(reg.phone_field, "") if reg.phone_field else "",
|
||||
"entity_type": r.get(reg.type_field, "") if reg.type_field else "",
|
||||
"naics": r.get(reg.naics_field, "") if reg.naics_field else "",
|
||||
"city": r.get(reg.city_field, "") if reg.city_field else "",
|
||||
"biz_state": r.get(reg.state_field, "") if reg.state_field else "",
|
||||
"status": r.get(reg.status_field, "") if reg.status_field else "",
|
||||
"obligation": reg.obligation,
|
||||
"due_date": due.isoformat() if due else "",
|
||||
"days_overdue": days_overdue if days_overdue is not None else "",
|
||||
"deficiency": deficiency,
|
||||
"last_filed": last_filed.isoformat() if last_filed else "",
|
||||
}
|
||||
|
||||
|
||||
def write_csv(path: str, rows: list[dict]) -> None:
|
||||
with open(path, "w", newline="") as f:
|
||||
w = csv.DictWriter(f, fieldnames=HEADER)
|
||||
w.writeheader()
|
||||
w.writerows(rows)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser(description=__doc__,
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument("--registries", default="ct,fmcsa",
|
||||
help="comma list of registry keys (default: ct,fmcsa)")
|
||||
ap.add_argument("--out-dir", default="/tmp/registry_out")
|
||||
ap.add_argument("--soon-days", type=int, default=90)
|
||||
ap.add_argument("--max-rows", type=int, default=None,
|
||||
help="cap rows per registry (quick test)")
|
||||
ap.add_argument("--dedupe-email", action="store_true",
|
||||
help="keep one row per email (most-overdue first)")
|
||||
ap.add_argument("--list", action="store_true", dest="do_list")
|
||||
args = ap.parse_args()
|
||||
|
||||
if args.do_list:
|
||||
print("Configured registries (email + dateable clock):\n")
|
||||
for k, reg in REGISTRIES.items():
|
||||
print(f" {k:6} {reg.name}")
|
||||
print(f" {reg.host}/{reg.resource} email={reg.email_field!r}")
|
||||
print(f" clock={reg.clock}({reg.clock_field}"
|
||||
+ (f"+{reg.interval_days}d" if reg.clock == 'staleness' else "")
|
||||
+ f") obligation={reg.obligation!r}\n")
|
||||
return 0
|
||||
|
||||
os.makedirs(args.out_dir, exist_ok=True)
|
||||
today = datetime.date.today()
|
||||
keys = [k.strip().lower() for k in args.registries.split(",") if k.strip()]
|
||||
grand = Counter()
|
||||
|
||||
for k in keys:
|
||||
reg = REGISTRIES.get(k)
|
||||
if not reg:
|
||||
print(f"!! unknown registry {k!r} (see --list)", file=sys.stderr)
|
||||
continue
|
||||
|
||||
print(f"\n=== {reg.name} [{k}] :: {reg.host}/{reg.resource} ===")
|
||||
try:
|
||||
print(f" dataset rows: {socrata_count(reg.host, reg.resource):,}")
|
||||
except Exception as e:
|
||||
print(f" (count failed: {e})")
|
||||
|
||||
t0 = time.time()
|
||||
rows: list[dict] = []
|
||||
seen: set[str] = set()
|
||||
for r, email in fetch_active_emailable(reg, args.max_rows):
|
||||
er = enrich(reg, r, email, today)
|
||||
if args.dedupe_email:
|
||||
if email in seen:
|
||||
continue
|
||||
seen.add(email)
|
||||
rows.append(er)
|
||||
if len(rows) % 100000 == 0:
|
||||
print(f" pulled {len(rows):,} ({len(rows)/max(time.time()-t0,1e-6):.0f}/s)")
|
||||
|
||||
overdue = [r for r in rows if r["deficiency"] == "overdue"]
|
||||
overdue.sort(key=lambda r: -(r["days_overdue"] or 0))
|
||||
|
||||
def days_until(r: dict) -> int | None:
|
||||
d = parse_date(r["due_date"])
|
||||
return (d - today).days if d else None
|
||||
|
||||
soon = [r for r in rows if r["deficiency"] == "upcoming"
|
||||
and (du := days_until(r)) is not None and 0 <= du <= args.soon_days]
|
||||
soon.sort(key=days_until)
|
||||
|
||||
rank = {"overdue": 0, "upcoming": 1, "no_date": 2}
|
||||
rows.sort(key=lambda r: (rank.get(r["deficiency"], 3), -(r["days_overdue"] or 0)))
|
||||
|
||||
p_over = os.path.join(args.out_dir, f"{k}_overdue.csv")
|
||||
p_soon = os.path.join(args.out_dir, f"{k}_due_soon.csv")
|
||||
p_all = os.path.join(args.out_dir, f"{k}_active_emailable.csv")
|
||||
write_csv(p_over, overdue)
|
||||
write_csv(p_soon, soon)
|
||||
write_csv(p_all, rows)
|
||||
|
||||
uniq = len({r["email"] for r in rows})
|
||||
grand["rows"] += len(rows); grand["overdue"] += len(overdue); grand["soon"] += len(soon)
|
||||
print(f" active+emailable: {len(rows):,} (distinct emails {uniq:,})")
|
||||
print(f" OVERDUE {reg.obligation}: {len(overdue):,} -> {p_over}")
|
||||
print(f" due soon (<= {args.soon_days}d): {len(soon):,} -> {p_soon}")
|
||||
print(f" all active emailable: {len(rows):,} -> {p_all}")
|
||||
print(f" pulled in {time.time()-t0:.0f}s")
|
||||
|
||||
print("\n=== TOTALS ===")
|
||||
print(f" active+emailable: {grand['rows']:,}")
|
||||
print(f" overdue: {grand['overdue']:,}")
|
||||
print(f" due soon: {grand['soon']:,}")
|
||||
print("\nNext: MX/SMTP-verify on the NON-sending IP before import, e.g.")
|
||||
print(f" python3 scripts/verify_csv_emails.py --in {args.out_dir}/{keys[0]}_overdue.csv "
|
||||
f"--out {args.out_dir}/{keys[0]}_overdue")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Loading…
Add table
Add a link
Reference in a new issue