diff --git a/docs/new-sector-compliance-targets.md b/docs/new-sector-compliance-targets.md index b8843dd..d4cb486 100644 --- a/docs/new-sector-compliance-targets.md +++ b/docs/new-sector-compliance-targets.md @@ -511,3 +511,179 @@ high-trust second wave. Output CSVs: `npi_all_cold_emailable.csv`, `npi_overdue_cold_emailable.csv`, `npi_overdue_direct_secure.csv` (NPI, email, name, specialty, state, due date, days overdue, LEIE flag, opt-out ending). + +--- + +## 10. Verticals With 500k+ EMAIL Leads (the volume question) — PULLED & VERIFIED + +Honest framing: NPPES, our other verified free email source, only yields ~120k +*cold-emailable* inboxes. To clear **500,000+ real, public, deadline-keyed email +addresses**, you have to move to registries where the email is **published in a +free bulk/Socrata feed**. We pulled the candidates live this session. The verdict +changed materially from the first draft of this section — see §10.1. + +> Status legend: ✅ pulled + verified this session · ❌ disproven this session. +> All counts below are from live pulls on the session date. + +### 10.1 USPTO Trademarks — ❌ DOES NOT QUALIFY (correction) + +The earlier draft called this the strongest 500k+ email vertical. **That was +wrong, and the pull proves it.** Two independent blockers: + +1. **Owner email is NOT in any free bulk file.** The Trademark Daily XML DTD's + `correspondent` group is `address-1 … address-5` only — there is **no email + element**. The Trademark Case Files research dataset + (`owner.dta` / `correspondent_domrep_attorney.dta`, freely downloadable from + `data.uspto.gov/ui/datasets/products/files/TRCFECO2/...`) was checked against + its 2023 schema + variable tables: **zero email columns anywhere.** The + mandatory-email rule put the email in the *application record*, not the *bulk + distribution*. Owner email lives only in **per-record TSDR**. +2. **TSDR is now auth-walled.** As of **2026-06-18** the USPTO Open Data Portal + requires a USPTO.gov account, and the API key requires **ID.me identity + verification**; the legacy open `bulkdata.uspto.gov` is decommissioned and the + TSDR bulk API now returns "register for an API key." So even per-record email + scraping needs an ID.me-verified account + key. + +Net: USPTO trademarks remain a great *audience* with a clean §8/§9 renewal clock, +but there is **no free, bulk, email-native path** to them. It belongs in the +"address/phone-only, needs email-append" bucket (like NPPES/EPA), not the 500k+ +email bucket. The slot is replaced by FMCSA below. + +### 10.2 FMCSA Company Census — ✅ THE 500k+ WINNER (`email_address`) +- **Source:** `data.transportation.gov/resource/az4n-8mr2` (Company Census File), + free Socrata, no auth. **4,459,640 rows total.** +- **Email (VERIFIED):** **2,927,276 rows carry `email_address` (2,791,522 + distinct).** Filtered to **`status_code = 'A'` (active) + email = 1,668,135**. + Our live build (active + usable-syntax email) produced **1,661,813 rows, + 1,588,732 distinct emails.** +- **Dateable clock (VERIFIED):** `mcs150_date` (format `YYYYMMDD HHMM`) + **24 + months** = biennial MCS-150 due date. Live build: + - **636,278 OVERDUE** MCS-150 updates (hottest leads, alone >500k). + - **73,781 due within 90 days** (pre-emptive). +- **Deliverability spot-check:** most-overdue 500-row sample → **459/500 (92%) + sendable** after MX/SMTP. (This was the *worst* cohort — mcs150 dates back to + 2003 with dead domains like `alltel.net`; fresher overdue cohorts verify higher. + CT below hit 96% on the same test.) +- **Offer:** DOT compliance renewal bundle — already in our wheelhouse (we run + MCS-150 update, UCR, BOC-3 today). This just bolts the **email channel** onto + the existing trucking playbook; previously we leaned on phone/mail. +- **Bonus columns:** `phone`, `dot_number`, physical + mailing address, + `carrier_operation`, fleet size, `hm_ind` (hazmat) — rich segmentation. + +### 10.3 Connecticut SOS Business Registry — ✅ QUALIFIES on its own (`business_email_address`) +- **Source:** `data.ct.gov/resource/n7gp-d28j` (CT Business Registry – Business + Master), free Socrata, no auth. **1,283,770 rows total.** CT is the standout: a + full Socrata sweep of `.gov` open-data portals found **no other state + business-entity registry that publishes a business email** (CO, MD, NY, WA, IA + master files have address only; CT is unique in exposing the email). +- **Email (VERIFIED):** **787,674 rows carry `business_email_address`.** Filtered + to `status = 'Active'` + email: live build = **439,763 rows, 303,299 distinct + emails.** (Distinct alone is below 500k — CT clears the *row* bar and is the + email-native SOS flagship, but pair it with FMCSA for the combined 500k+ + distinct.) +- **Dateable clock (VERIFIED):** `annual_report_due_date` (literal column). Live + build: **97,466 OVERDUE annual reports**, **16,235 due within 90 days.** +- **Deliverability spot-check:** most-overdue 500-row sample → **482/500 (96%) + sendable** after MX/SMTP (18 hard-fails). +- **Offer:** annual-report filing + registered-agent + reinstatement (core + Performance West products). +- **Bonus columns:** `naics_code` (vertical targeting), registration date, + business type, woman/veteran/minority-owned flags. + +### 10.4 Combined verified email universe (this session) +| Registry | Active + email rows | Distinct emails | Overdue (hottest) | Due ≤90d | Verify rate | +|---|---|---|---|---|---| +| FMCSA Company Census | **1,661,813** | **1,588,732** | **636,278** | 73,781 | 92% (worst cohort) | +| Connecticut SOS | **439,763** | 303,299 | **97,466** | 16,235 | 96% | +| **Combined** | **~2.10M** | **~1.89M** | **~733k** | ~90k | — | + +Either one of **FMCSA alone (1.59M distinct)** or **FMCSA+CT (~1.89M distinct)** +decisively clears the 500k+ goal with a real, dateable deficiency hook on each +record. The **overdue segments alone (~733k)** already exceed 500k. + +### 10.5 Honorable mentions that DON'T clear 500k email +- **USPTO trademarks:** email not in free bulk; API ID.me-walled (§10.1). ❌ +- **NPPES (NPI):** ~120k cold inboxes only (§8). Great audience, wrong scale. ✅verified earlier. +- **FAA airmen/aircraft (~300k):** email is opt-in only — small usable volume. +- **ATF FFL, FMC OTI:** tiny universes (thousands). Quality, not volume. +- **EPA RCRA/ECHO:** **no email at all** in bulk (§3). Volume but zero native email. +- **Other state SOS master files (CO/MD/NY/WA/IA):** address only, no email column. + +### 10.6 The pipeline (built this session) +`scripts/build_registry_outreach_lists.py` — one extensible engine for any free +registry that exposes **email + a dateable clock**. Two clock types: +- `due_date` : a column already holds the obligation's due date (CT annual report). +- `staleness` : a column holds last-filed date; due = last + `interval_days` + (FMCSA `mcs150_date` + 730d). + +Adding a registry = one `Registry(...)` entry (Socrata host, resource id, column +map). It server-side-filters to active + email-present, computes overdue/due-soon, +and writes three campaign-ready CSVs per registry (every row has an `email` +column, matching `verify_csv_emails.py`'s contract): +``` +_overdue.csv obligation PAST DUE (lead with the deadline) +_due_soon.csv due within --soon-days (pre-emptive offer) +_active_emailable.csv all active + email (broad campaign) +``` +Run: +``` +python3 scripts/build_registry_outreach_lists.py --registries fmcsa,ct +python3 scripts/verify_csv_emails.py --in out/fmcsa_overdue.csv --out out/fmcsa_overdue +``` +Then import the `_verified.csv` into the campaign tool. MX/SMTP verification runs +on the NON-sending IP (port 25 + MX confirmed working on our infra) so it never +touches the warmed MTA reputation. + +### 10.7 Recommended sequencing +1. **FMCSA overdue (636k)** — biggest single email-native deadline pool, and it + reuses our existing DOT-compliance fulfillment. Start here. +2. **CT overdue annual reports (97k)** — highest verify rate (96%), clean + annual-report + reinstatement offer, NAICS for sub-segmenting. +3. **FMCSA due-soon + CT due-soon (~90k)** — pre-emptive "due in <90 days" offers. +4. Broaden SOS coverage only if another state is found to expose a business email + (none did this session beyond CT). + +--- + +## 11. Extra High-Volume Public-Email Candidates to Verify + +These are second-ring ideas after USPTO, FMCSA, and SOS files. They may not each clear 500k alone, but they can contribute to a national email-native compliance lead graph. + +| Candidate | Email likelihood | Deadline/trigger | Product angle | Notes | +|---|---|---|---|---| +| Professional license mega-boards | Medium-high in some states | Renewal, CE, bond/insurance, status | License renewal calendar | Prioritize establishment/company records over individual professionals. | +| City/county business license portals | Medium, fragmented | Annual local license or business tax certificate | Local filing support | Huge aggregate universe, but ETL-heavy. | +| State insurance producer/agency files | Often email-rich | License renewal, CE, appointment status | Renewal/CE monitoring | Paperwork familiarity is higher, so focus on agencies/small shops. | +| Real estate brokerage firm files | Sometimes email-rich | Firm/branch/DB renewal | Firm compliance calendar | Avoid individual agents first. | +| Cosmetology/barber establishment files | Sometimes email-rich | Shop license renewal, inspection status | Shop renewal support | Strong small-operator fit. | +| Motor vehicle dealer/repair facility files | Sometimes email-rich | Dealer/repair license, bond | Renewal/bond packet | Good business-critical lapse angle. | +| Local health/food permits | Sometimes email-rich | Food permit renewal, inspection | Permit calendar | Lower ticket unless bundled. | + +**Verification rule:** do not count a registry as email-native until a current bulk/API pull confirms (1) email column present, (2) enough distinct deliverable emails after MX/SMTP verification, (3) a computable recurring deadline, and (4) the email belongs to the operator rather than a filing intermediary. + +### 11.1 Additional verification queue + +If USPTO/FMCSA/SOS are already being pulled, verify these next because they may expose public email or become strong email-enriched compliance lists: + +1. **FCC ULS wireless licensees** — expiring private land mobile/business licenses, construction deadlines, ULS contact cleanup. +2. **FCC ASR tower owners** — smaller but high-value tower compliance and owner/contact update work. +3. **FCC broadcast station owners** — public-file and renewal admin audits, high value but counsel-heavy. +4. **State insurance agencies/producers** — huge volume, better at agency level than individual producers. +5. **Real estate brokerage firm/branch licenses** — firm/branch renewal and designated-broker cleanup. +6. **Umbrella occupational license portals** — find states whose detail APIs expose emails, then pick the best boards. +7. **Pest-control/pesticide businesses** — strong field-service fit with license/CE/roster deadlines. +8. **Dealer/repair/dismantler/inspection-station licenses** — business-critical renewals and bond/insurance tracking. +9. **Childcare provider directories** — public-facing emails plus license/inspection status, careful tone required. +10. **CLIA/state lab licenses** — two-year certificate renewal, contact-light federally but enrichable through state/provider directories. +11. **State controlled-substance registrations** — high-consequence renewal calendar, sensitive messaging. +12. **State Medicaid/DMEPOS provider lists** — extend NPI/PECOS revalidation into provider-enrollment monitoring. +13. **Tobacco retailer/product registries** — recurring retail permits and compliance-check outcomes. +14. **Cannabis/hemp/product registries** — email-rich in some states, high compliance pain, platform restrictions. +15. **Energy/solar installer directories** — public emails in customer-facing lists plus contractor/vendor renewals. +16. **Public procurement/vendor certification lists** — email-rich, but filter to trades/operators rather than professional grant/SAM filers. +17. **Lottery/gaming/amusement licenses** — recurring permits for retailers, route operators, and venues. +18. **Public-facing facility directories joined to licensing data** — use directory email plus regulator deficiency signal. +19. **RDAP/website/domain enrichment** — not a compliance list by itself, but a domain discovery layer for SOS/USPTO/local permits. +20. **MSB/NMLS/state financial licenses** — smaller but high-value renewal, bond, branch, and delegate-update work. + +Prioritize any source where the first sample shows: public email, explicit expiration date, business/facility record type, and a filing we can complete administratively for a fixed price. diff --git a/scripts/build_registry_outreach_lists.py b/scripts/build_registry_outreach_lists.py new file mode 100644 index 0000000..efb5d63 --- /dev/null +++ b/scripts/build_registry_outreach_lists.py @@ -0,0 +1,357 @@ +#!/usr/bin/env python3 +"""Build public-registry outreach lists with EMAIL + a dateable compliance clock. + +Implements the "500k+ email leads" verticals from +docs/new-sector-compliance-targets.md sec 10. The winning pattern is always: + + public registry -> recurring obligation -> automated deficiency check + -> EMAIL outreach -> fixed-price filing service. + +Unlike NPPES (email-poor) and EPA RCRA (no email at all), these registries expose +a real, cold-emailable contact email in a FREE bulk/Socrata feed AND a column we +can turn into a dateable "your X is due/overdue" hook: + + ct Connecticut SOS Business Registry -- VERIFIED this session + data.ct.gov/n7gp-d28j | email: business_email_address + clock: annual_report_due_date (annual report) -> overdue/due-soon + ~439,763 ACTIVE + email (303k distinct), ~97,466 OVERDUE annual reports. + + fmcsa FMCSA Company Census (motor carriers) -- VERIFIED this session + data.transportation.gov/az4n-8mr2 | email: email_address + clock: mcs150_date + 24mo (biennial MCS-150 update) -> overdue/due-soon + ~2,927,276 rows with email (2.79M distinct), ~1,668,135 ACTIVE + email. + This is the genuine 500k+ email vertical (replaces USPTO trademarks, + whose owner email is NOT in any free bulk file and whose API is now + ID.me-auth-walled as of 2026-06-18). + +Two clock types are supported per adapter: + * "due_date" : a column already holds the obligation's due date. + * "staleness" : a column holds the last-filed date; due = last + interval_days. + +Adding a registry = one Registry entry (Socrata host, resource, column mapping). + +Output (every row has an `email` column, per verify_csv_emails.py's contract): + out/_overdue.csv obligation PAST DUE (hottest leads) + out/_due_soon.csv due within --soon-days (pre-emptive) + out/_active_emailable.csv all active + email (broad campaign) + +Then MX/SMTP-verify on the NON-sending IP before importing: + python3 scripts/verify_csv_emails.py --in out/ct_overdue.csv --out out/ct_overdue + +Usage: + python3 scripts/build_registry_outreach_lists.py --registries ct,fmcsa + python3 scripts/build_registry_outreach_lists.py --registries ct --max-rows 5000 + python3 scripts/build_registry_outreach_lists.py --list +""" +from __future__ import annotations + +import argparse +import csv +import datetime +import json +import os +import sys +import time +import urllib.parse +import urllib.request +from collections import Counter +from dataclasses import dataclass + +UA = "performancewest-registry-outreach/1.0 (+https://performancewest.net)" + + +def email_ok(e: str) -> bool: + e = (e or "").strip() + return bool(e) and "@" in e and "." in e.split("@")[-1] and " " not in e + + +@dataclass +class Registry: + """One public registry that exposes a cold-emailable contact + a clock.""" + + key: str + name: str + host: str # Socrata host, e.g. data.ct.gov + resource: str # Socrata 4x4, e.g. n7gp-d28j + email_field: str # primary cold-emailable email column + name_field: str + obligation: str # human label, e.g. "annual report" + clock: str # "due_date" | "staleness" + clock_field: str # due-date col, OR last-filed col for staleness + interval_days: int = 0 # for staleness: days from last-filed to due + status_field: str | None = None + active_values: tuple[str, ...] = () + alt_email_field: str | None = None + id_field: str | None = None + type_field: str | None = None + naics_field: str | None = None + city_field: str | None = None + state_field: str | None = None + phone_field: str | None = None + notes: str = "" + + +REGISTRIES: dict[str, Registry] = { + "ct": Registry( + key="ct", + name="Connecticut SOS Business Registry", + host="data.ct.gov", + resource="n7gp-d28j", + email_field="business_email_address", + alt_email_field="category_survey_email_address", + name_field="name", + obligation="annual report", + clock="due_date", + clock_field="annual_report_due_date", + status_field="status", + active_values=("active",), + id_field="accountnumber", + type_field="business_type", + naics_field="naics_code", + city_field="billingcity", + state_field="billingstate", + notes="business_email_address is a real cold-emailable inbox; " + "annual_report_due_date is the dateable annual-report clock.", + ), + "fmcsa": Registry( + key="fmcsa", + name="FMCSA Company Census (motor carriers)", + host="data.transportation.gov", + resource="az4n-8mr2", + email_field="email_address", + name_field="legal_name", + obligation="MCS-150 biennial update", + clock="staleness", + clock_field="mcs150_date", + interval_days=730, # biennial + status_field="status_code", + active_values=("a",), # A = active + id_field="dot_number", + type_field="carrier_operation", + city_field="phy_city", + state_field="phy_state", + phone_field="phone", + notes="email_address coverage ~2.9M rows; mcs150_date + 24mo is the " + "biennial MCS-150 staleness clock. Replaces USPTO (now auth-walled).", + ), +} + + +def socrata_get(host: str, resource: str, params: dict) -> list[dict]: + url = f"https://{host}/resource/{resource}.json?{urllib.parse.urlencode(params)}" + req = urllib.request.Request(url, headers={"User-Agent": UA, "Accept": "application/json"}) + with urllib.request.urlopen(req, timeout=180) as r: + return json.loads(r.read().decode("utf-8")) + + +def socrata_count(host: str, resource: str, where: str | None = None) -> int: + params = {"$select": "count(1)"} + if where: + params["$where"] = where + rows = socrata_get(host, resource, params) + return int(rows[0].get("count_1", 0)) if rows else 0 + + +def parse_date(val: str | None) -> datetime.date | None: + if not val: + return None + val = val.strip() + for fmt in ("%Y-%m-%dT%H:%M:%S.%f", "%Y-%m-%dT%H:%M:%S", "%Y-%m-%d", + "%m/%d/%Y", "%m/%d/%Y %H:%M:%S", + "%Y%m%d %H%M", "%Y%m%d"): # FMCSA mcs150_date = 'YYYYMMDD HHMM' + try: + return datetime.datetime.strptime(val, fmt).date() + except ValueError: + continue + try: + return datetime.datetime.strptime(val[:10], "%Y-%m-%d").date() + except ValueError: + return None + + +def due_date_for(reg: Registry, raw: str | None) -> datetime.date | None: + """Resolve the obligation's due date from the registry's clock column.""" + d = parse_date(raw) + if d is None: + return None + if reg.clock == "staleness": + return d + datetime.timedelta(days=reg.interval_days) + return d # due_date + + +def fetch_active_emailable(reg: Registry, max_rows: int | None, page: int = 50000): + """Yield raw rows: active records that carry a usable email (server-filtered).""" + cols = [c for c in { + reg.id_field, reg.name_field, reg.email_field, reg.alt_email_field, + reg.status_field, reg.clock_field, reg.type_field, reg.naics_field, + reg.city_field, reg.state_field, reg.phone_field, + } if c] + + where = [f"{reg.email_field} IS NOT NULL"] + if reg.status_field and reg.active_values: + ors = " OR ".join(f"upper({reg.status_field})='{v.upper()}'" for v in reg.active_values) + where.append(f"({ors})") + where_s = " AND ".join(where) + + offset, pulled = 0, 0 + while True: + params = { + "$select": ",".join(cols), + "$where": where_s, + "$order": (reg.id_field or reg.name_field), + "$limit": page, + "$offset": offset, + } + rows = socrata_get(reg.host, reg.resource, params) + if not rows: + break + for r in rows: + email = (r.get(reg.email_field) or "").strip() + if not email_ok(email) and reg.alt_email_field: + email = (r.get(reg.alt_email_field) or "").strip() + if not email_ok(email): + continue + yield r, email.lower() + pulled += 1 + if max_rows and pulled >= max_rows: + return + offset += len(rows) + if len(rows) < page: + break + + +HEADER = ["registry", "entity_id", "name", "email", "phone", "entity_type", + "naics", "city", "biz_state", "status", "obligation", + "due_date", "days_overdue", "deficiency", "last_filed"] + + +def enrich(reg: Registry, r: dict, email: str, today: datetime.date) -> dict: + raw = r.get(reg.clock_field) if reg.clock_field else None + last_filed = parse_date(raw) if reg.clock == "staleness" else None + due = due_date_for(reg, raw) + if due: + days_overdue = (today - due).days + deficiency = "overdue" if days_overdue > 0 else "upcoming" + else: + days_overdue, deficiency = None, "no_date" + return { + "registry": reg.key, + "entity_id": r.get(reg.id_field, "") if reg.id_field else "", + "name": r.get(reg.name_field, ""), + "email": email, + "phone": r.get(reg.phone_field, "") if reg.phone_field else "", + "entity_type": r.get(reg.type_field, "") if reg.type_field else "", + "naics": r.get(reg.naics_field, "") if reg.naics_field else "", + "city": r.get(reg.city_field, "") if reg.city_field else "", + "biz_state": r.get(reg.state_field, "") if reg.state_field else "", + "status": r.get(reg.status_field, "") if reg.status_field else "", + "obligation": reg.obligation, + "due_date": due.isoformat() if due else "", + "days_overdue": days_overdue if days_overdue is not None else "", + "deficiency": deficiency, + "last_filed": last_filed.isoformat() if last_filed else "", + } + + +def write_csv(path: str, rows: list[dict]) -> None: + with open(path, "w", newline="") as f: + w = csv.DictWriter(f, fieldnames=HEADER) + w.writeheader() + w.writerows(rows) + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("--registries", default="ct,fmcsa", + help="comma list of registry keys (default: ct,fmcsa)") + ap.add_argument("--out-dir", default="/tmp/registry_out") + ap.add_argument("--soon-days", type=int, default=90) + ap.add_argument("--max-rows", type=int, default=None, + help="cap rows per registry (quick test)") + ap.add_argument("--dedupe-email", action="store_true", + help="keep one row per email (most-overdue first)") + ap.add_argument("--list", action="store_true", dest="do_list") + args = ap.parse_args() + + if args.do_list: + print("Configured registries (email + dateable clock):\n") + for k, reg in REGISTRIES.items(): + print(f" {k:6} {reg.name}") + print(f" {reg.host}/{reg.resource} email={reg.email_field!r}") + print(f" clock={reg.clock}({reg.clock_field}" + + (f"+{reg.interval_days}d" if reg.clock == 'staleness' else "") + + f") obligation={reg.obligation!r}\n") + return 0 + + os.makedirs(args.out_dir, exist_ok=True) + today = datetime.date.today() + keys = [k.strip().lower() for k in args.registries.split(",") if k.strip()] + grand = Counter() + + for k in keys: + reg = REGISTRIES.get(k) + if not reg: + print(f"!! unknown registry {k!r} (see --list)", file=sys.stderr) + continue + + print(f"\n=== {reg.name} [{k}] :: {reg.host}/{reg.resource} ===") + try: + print(f" dataset rows: {socrata_count(reg.host, reg.resource):,}") + except Exception as e: + print(f" (count failed: {e})") + + t0 = time.time() + rows: list[dict] = [] + seen: set[str] = set() + for r, email in fetch_active_emailable(reg, args.max_rows): + er = enrich(reg, r, email, today) + if args.dedupe_email: + if email in seen: + continue + seen.add(email) + rows.append(er) + if len(rows) % 100000 == 0: + print(f" pulled {len(rows):,} ({len(rows)/max(time.time()-t0,1e-6):.0f}/s)") + + overdue = [r for r in rows if r["deficiency"] == "overdue"] + overdue.sort(key=lambda r: -(r["days_overdue"] or 0)) + + def days_until(r: dict) -> int | None: + d = parse_date(r["due_date"]) + return (d - today).days if d else None + + soon = [r for r in rows if r["deficiency"] == "upcoming" + and (du := days_until(r)) is not None and 0 <= du <= args.soon_days] + soon.sort(key=days_until) + + rank = {"overdue": 0, "upcoming": 1, "no_date": 2} + rows.sort(key=lambda r: (rank.get(r["deficiency"], 3), -(r["days_overdue"] or 0))) + + p_over = os.path.join(args.out_dir, f"{k}_overdue.csv") + p_soon = os.path.join(args.out_dir, f"{k}_due_soon.csv") + p_all = os.path.join(args.out_dir, f"{k}_active_emailable.csv") + write_csv(p_over, overdue) + write_csv(p_soon, soon) + write_csv(p_all, rows) + + uniq = len({r["email"] for r in rows}) + grand["rows"] += len(rows); grand["overdue"] += len(overdue); grand["soon"] += len(soon) + print(f" active+emailable: {len(rows):,} (distinct emails {uniq:,})") + print(f" OVERDUE {reg.obligation}: {len(overdue):,} -> {p_over}") + print(f" due soon (<= {args.soon_days}d): {len(soon):,} -> {p_soon}") + print(f" all active emailable: {len(rows):,} -> {p_all}") + print(f" pulled in {time.time()-t0:.0f}s") + + print("\n=== TOTALS ===") + print(f" active+emailable: {grand['rows']:,}") + print(f" overdue: {grand['overdue']:,}") + print(f" due soon: {grand['soon']:,}") + print("\nNext: MX/SMTP-verify on the NON-sending IP before import, e.g.") + print(f" python3 scripts/verify_csv_emails.py --in {args.out_dir}/{keys[0]}_overdue.csv " + f"--out {args.out_dir}/{keys[0]}_overdue") + return 0 + + +if __name__ == "__main__": + sys.exit(main())