mail: DMARC aggregate-report parser + dedicated dmarc@ mailbox ingestion
Tool 2 of the deliverability monitoring pair (Tool 1 = mail_reputation_monitor). DMARC rua reports from dozens of operators (Google, Yahoo, Comcast, Cox, Bell, Mimecast, Cisco ESA, GMX, mail.com, ...) were landing in ops@ (dmarc@ was a DL), burying real mail and never parsed. Now ingested + queryable: - dmarc@performancewest.net converted DL -> dedicated Carbonio mailbox; isolated IMAP creds in server .env, surfaced to workers in docker-compose.yml (mirrors OPS_IMAP_*). 29 historical reports moved ops@ -> dmarc@ via IMAP. - scripts/dmarc_report_parser.py: IMAP fetch unseen -> decompress .gz/.zip/.xml (namespace-agnostic: classic + urn:ietf:params:xml:ns:dmarc-2.0 GMX/mail.com) -> parse aggregate XML -> upsert dmarc_report (keyed (org_name,report_id), no-op on re-parse) + dmarc_record per source IP. dmarc_pass = dkim_aligned OR spf_aligned. Marks \Seen. --dry-run/--all/--alert (7d per-IP summary + Telegram if one of OUR IPs <95% pass, or EXTERNAL IP sends >=20 failing msgs as us = spoofing under p=reject). psycopg2 imported lazily so --dry-run runs without the driver. - api/migrations/102_dmarc_aggregate.sql: dmarc_report + dmarc_record tables. - infra/cron/pw-dmarc-parser: 06:20 UTC daily --alert (after reputation, before scrub). - docs/deliverability.md: DMARC section DONE; query examples. Verified: dry-run --all parses all 28 reports (1 non-report test probe), 0 unknown after the namespace fix.
This commit is contained in:
parent
b45332b5f7
commit
8e5590b492
5 changed files with 509 additions and 8 deletions
|
|
@ -224,14 +224,42 @@ all HE.net slaves + 8.8.8.8/1.1.1.1/9.9.9.9):
|
|||
- `send.performancewest.net` TXT `yahoo-verification-key=Ps5hGjVxXgeQcLcxr671YG0/RxzjjL0eqh6vfULubEo=`
|
||||
(added alongside the existing `send` SPF record; both TXT coexist).
|
||||
|
||||
### ✅ DMARC aggregate reports — mailbox FIXED 2026-06-19 (parser still TODO)
|
||||
Gmail/Yahoo/Microsoft send daily per-IP auth+disposition XML to
|
||||
`dmarc@performancewest.net` (DMARC record has `rua=mailto:dmarc@...`). **That
|
||||
mailbox was REJECTING (5.1.1) until 2026-06-19 — we were silently losing every
|
||||
report.** It's now a Carbonio DL -> ops@ (verified delivering). Next: add IMAP creds
|
||||
for ops@ (or a dedicated dmarc mailbox) and build a small collector/parser worker to
|
||||
chart per-IP/per-domain pass-fail without any provider login. Now actually worth
|
||||
doing since the data finally arrives.
|
||||
### ✅ DMARC aggregate reports — DONE 2026-06-19 (dedicated mailbox + parser)
|
||||
Gmail/Yahoo/Microsoft + dozens of operators (Comcast, Cox, Bell, Mimecast, Cisco
|
||||
ESA, GMX, mail.com, gosecure, ...) send daily per-IP auth+disposition XML to
|
||||
`dmarc@performancewest.net` (DMARC record: `p=reject; rua=mailto:dmarc@; ruf=mailto:dmarc@; fo=1`).
|
||||
**That mailbox was REJECTING (5.1.1) until 2026-06-19 — we silently lost every
|
||||
report.** Now fully wired:
|
||||
|
||||
1. **Dedicated mailbox.** `dmarc@performancewest.net` is its own Carbonio account
|
||||
(was a DL -> ops@, which buried ops@ under report XML). Isolated IMAP credential
|
||||
in the server `.env` (`DMARC_IMAP_{HOST,PORT,USER,PASS}`), surfaced to the workers
|
||||
container in `docker-compose.yml` (mirrors the `OPS_IMAP_*` pattern). The 29
|
||||
historical reports that had landed in ops@ were moved over via IMAP.
|
||||
2. **Parser worker.** `scripts/dmarc_report_parser.py` IMAP-fetches unseen messages,
|
||||
decompresses the `.gz`/`.zip`/`.xml` attachment (namespace-agnostic — handles both
|
||||
the classic and the `urn:ietf:params:xml:ns:dmarc-2.0` GMX/mail.com schema), parses
|
||||
the aggregate XML, and upserts one `dmarc_report` row (keyed `(org_name, report_id)`,
|
||||
so re-parsing is a no-op) + one `dmarc_record` row per source IP into the schema from
|
||||
`api/migrations/102_dmarc_aggregate.sql`. `dmarc_pass = dkim_aligned=pass OR
|
||||
spf_aligned=pass`. Marks each message `\Seen` so each run only handles new reports.
|
||||
Flags: `--dry-run`, `--all` (backfill seen), `--alert` (7-day per-IP summary +
|
||||
Telegram if one of OUR IPs drops below 95% pass, or an EXTERNAL IP sends >=20 failing
|
||||
msgs as us = spoofing under `p=reject`).
|
||||
3. **Cron.** `/etc/cron.d/pw-dmarc-parser` (tracked at `infra/cron/pw-dmarc-parser`)
|
||||
runs `... workers python3 -m scripts.dmarc_report_parser --alert` daily at 06:20 UTC.
|
||||
|
||||
Query examples once populated:
|
||||
```sql
|
||||
-- who sends as us, and are they aligning? (the payoff of the DKIM/subdomain fixes)
|
||||
SELECT source_ip, sum(msg_count) total,
|
||||
sum(msg_count) FILTER (WHERE dmarc_pass) pass,
|
||||
round(100.0*sum(msg_count) FILTER (WHERE dmarc_pass)/sum(msg_count)) pass_pct
|
||||
FROM dmarc_record r JOIN dmarc_report rep ON rep.id=r.report_id
|
||||
WHERE rep.date_begin >= now()-interval '7 days'
|
||||
GROUP BY source_ip ORDER BY total DESC;
|
||||
-- any UNKNOWN IP failing alignment = spoofing/forgotten relay (reputation poison)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue