mail: DMARC aggregate-report parser + dedicated dmarc@ mailbox ingestion

Tool 2 of the deliverability monitoring pair (Tool 1 = mail_reputation_monitor).
DMARC rua reports from dozens of operators (Google, Yahoo, Comcast, Cox, Bell,
Mimecast, Cisco ESA, GMX, mail.com, ...) were landing in ops@ (dmarc@ was a DL),
burying real mail and never parsed. Now ingested + queryable:

- dmarc@performancewest.net converted DL -> dedicated Carbonio mailbox; isolated
  IMAP creds in server .env, surfaced to workers in docker-compose.yml (mirrors
  OPS_IMAP_*). 29 historical reports moved ops@ -> dmarc@ via IMAP.
- scripts/dmarc_report_parser.py: IMAP fetch unseen -> decompress .gz/.zip/.xml
  (namespace-agnostic: classic + urn:ietf:params:xml:ns:dmarc-2.0 GMX/mail.com) ->
  parse aggregate XML -> upsert dmarc_report (keyed (org_name,report_id), no-op on
  re-parse) + dmarc_record per source IP. dmarc_pass = dkim_aligned OR spf_aligned.
  Marks \Seen. --dry-run/--all/--alert (7d per-IP summary + Telegram if one of OUR
  IPs <95% pass, or EXTERNAL IP sends >=20 failing msgs as us = spoofing under
  p=reject). psycopg2 imported lazily so --dry-run runs without the driver.
- api/migrations/102_dmarc_aggregate.sql: dmarc_report + dmarc_record tables.
- infra/cron/pw-dmarc-parser: 06:20 UTC daily --alert (after reputation, before scrub).
- docs/deliverability.md: DMARC section DONE; query examples.

Verified: dry-run --all parses all 28 reports (1 non-report test probe), 0 unknown
after the namespace fix.
This commit is contained in:
justin 2026-06-19 08:50:20 -05:00
parent b45332b5f7
commit 8e5590b492
5 changed files with 509 additions and 8 deletions

View file

@ -224,14 +224,42 @@ all HE.net slaves + 8.8.8.8/1.1.1.1/9.9.9.9):
- `send.performancewest.net` TXT `yahoo-verification-key=Ps5hGjVxXgeQcLcxr671YG0/RxzjjL0eqh6vfULubEo=`
(added alongside the existing `send` SPF record; both TXT coexist).
### ✅ DMARC aggregate reports — mailbox FIXED 2026-06-19 (parser still TODO)
Gmail/Yahoo/Microsoft send daily per-IP auth+disposition XML to
`dmarc@performancewest.net` (DMARC record has `rua=mailto:dmarc@...`). **That
mailbox was REJECTING (5.1.1) until 2026-06-19 — we were silently losing every
report.** It's now a Carbonio DL -> ops@ (verified delivering). Next: add IMAP creds
for ops@ (or a dedicated dmarc mailbox) and build a small collector/parser worker to
chart per-IP/per-domain pass-fail without any provider login. Now actually worth
doing since the data finally arrives.
### ✅ DMARC aggregate reports — DONE 2026-06-19 (dedicated mailbox + parser)
Gmail/Yahoo/Microsoft + dozens of operators (Comcast, Cox, Bell, Mimecast, Cisco
ESA, GMX, mail.com, gosecure, ...) send daily per-IP auth+disposition XML to
`dmarc@performancewest.net` (DMARC record: `p=reject; rua=mailto:dmarc@; ruf=mailto:dmarc@; fo=1`).
**That mailbox was REJECTING (5.1.1) until 2026-06-19 — we silently lost every
report.** Now fully wired:
1. **Dedicated mailbox.** `dmarc@performancewest.net` is its own Carbonio account
(was a DL -> ops@, which buried ops@ under report XML). Isolated IMAP credential
in the server `.env` (`DMARC_IMAP_{HOST,PORT,USER,PASS}`), surfaced to the workers
container in `docker-compose.yml` (mirrors the `OPS_IMAP_*` pattern). The 29
historical reports that had landed in ops@ were moved over via IMAP.
2. **Parser worker.** `scripts/dmarc_report_parser.py` IMAP-fetches unseen messages,
decompresses the `.gz`/`.zip`/`.xml` attachment (namespace-agnostic — handles both
the classic and the `urn:ietf:params:xml:ns:dmarc-2.0` GMX/mail.com schema), parses
the aggregate XML, and upserts one `dmarc_report` row (keyed `(org_name, report_id)`,
so re-parsing is a no-op) + one `dmarc_record` row per source IP into the schema from
`api/migrations/102_dmarc_aggregate.sql`. `dmarc_pass = dkim_aligned=pass OR
spf_aligned=pass`. Marks each message `\Seen` so each run only handles new reports.
Flags: `--dry-run`, `--all` (backfill seen), `--alert` (7-day per-IP summary +
Telegram if one of OUR IPs drops below 95% pass, or an EXTERNAL IP sends >=20 failing
msgs as us = spoofing under `p=reject`).
3. **Cron.** `/etc/cron.d/pw-dmarc-parser` (tracked at `infra/cron/pw-dmarc-parser`)
runs `... workers python3 -m scripts.dmarc_report_parser --alert` daily at 06:20 UTC.
Query examples once populated:
```sql
-- who sends as us, and are they aligning? (the payoff of the DKIM/subdomain fixes)
SELECT source_ip, sum(msg_count) total,
sum(msg_count) FILTER (WHERE dmarc_pass) pass,
round(100.0*sum(msg_count) FILTER (WHERE dmarc_pass)/sum(msg_count)) pass_pct
FROM dmarc_record r JOIN dmarc_report rep ON rep.id=r.report_id
WHERE rep.date_begin >= now()-interval '7 days'
GROUP BY source_ip ORDER BY total DESC;
-- any UNKNOWN IP failing alignment = spoofing/forgotten relay (reputation poison)
```
---