docs: explain phantom email opens/clicks from security scanners (Proofpoint/Microsoft/etc), plain-English + technical

docs/email-phantom-opens-scanners.md

@@ -0,0 +1,156 @@
+# Phantom Opens & Clicks from Email Security Scanners
+
+Date: 2026-07-03
+
+## In plain English (no tech background needed)
+
+When we send a marketing email, most of the businesses we email use a **security
+guard for their inbox** (Microsoft, Proofpoint, Mimecast, and similar). Before a
+real person ever sees our email, that guard **opens the email and clicks every
+link in it first**, just to make sure nothing is dangerous. Think of it like the
+mailroom in a big office opening and X-raying a package before handing it to the
+employee.
+
+Here is the problem: our tools can't easily tell the difference between the guard
+and a real person. When the guard opens the email, it counts as an "open." When
+the guard clicks the link to inspect it, it can count as a "click." So our reports
+can say *"30% of people opened it and some clicked!"* when in reality it was mostly
+robots checking for viruses, and **no actual human was interested**.
+
+That's why we don't trust "opens" or even raw "clicks" as proof that the campaign
+is working. A robot opening an email doesn't pay us. So instead we look at things a
+robot almost never does:
+
+- Did someone reach the **order page** for a service?
+- Did they **start checkout**?
+- Did they **actually pay**?
+
+Those are the numbers that matter. If a campaign shows lots of opens and clicks but
+nobody reached the order page, that's a strong sign it was mostly the robot
+security guards, not real customers. We also run a **filter** that spots the
+tell-tale signs of these robots and quietly ignores them, so our real numbers only
+count real people.
+
+Bottom line: **"opens" and "clicks" look impressive but are mostly robots. Real
+interest = order-page visits, checkouts, and payments.**
+
+## The short version
+
+Email opens, and even *clicks*, are unreliable signals because **email security
+gateways automatically fetch and render every link in a message** to scan it for
+malware, before the human ever sees the email. Those automated fetches fire the
+same tracking pixels and, for the advanced scanners that drive a real headless
+browser, the same JavaScript pageviews/clicks that a human would. So a campaign
+can show 20-40% "opens" and a handful of "clicks" with **zero real human
+engagement**.
+
+This is why we treat **Listmonk opens as noise**, do our conversion tracking in
+**Umami with a bot filter**, and judge campaigns on **human clicks → order-page
+views → checkout → paid orders**, never on opens.
+
+## Who does this and how
+
+| Gateway | Product | Behavior |
+| --- | --- | --- |
+| Microsoft | Defender for O365 "Safe Links" / ATP detonation | Rewrites every URL, fetches it, and **detonates** suspicious ones in a real browser sandbox. Very common on business/`outlook`/M365 tenants. |
+| Proofpoint | URL Defense (`urldefense.com` / `pphosted.com`) | Rewrites and pre-fetches every link; re-scans on click. |
+| Mimecast | URL Protect / Attachment Protect | Pre-fetches and rewrites links. |
+| Barracuda | Link Protect | Fetches links at delivery. |
+| Cisco / Symantec / MessageLabs | Secure Email / ClickTime | Scan-on-delivery and scan-on-click. |
+| Google | Gmail image proxy (`GoogleImageProxy`) | Pre-fetches images (inflates *opens*, not usually JS clicks). |
+
+Two distinct effects:
+
+1. **Inflated opens.** Any gateway that loads the tracking pixel (or Gmail's
+   image proxy caching it) counts as an "open" in Listmonk. This happens on a
+   large fraction of B2B mail. Opens are therefore close to meaningless.
+2. **Phantom clicks / pageviews.** The advanced scanners (Microsoft ATP,
+   Proofpoint, Mimecast) run a **headless browser** that executes JavaScript.
+   That fires a real Umami pageview and can trip click events, so the visit
+   *looks* human. This is the dangerous one because it corrupts the metric we
+   actually rely on (clicks), not just the one we already ignore (opens).
+
+Extra wrinkles:
+
+- Scanners often fetch **within seconds of delivery**, in bulk, from datacenter
+  IPs, before any human could plausibly read the email.
+- They may fetch **every** link, so a "click" on a deep CTA can be a scanner
+  walking all URLs, not a human choosing one.
+- Microsoft rewrites the URL, so the referrer/host can look like
+  `*.safelinks.protection.outlook.com`; Proofpoint uses `urldefense.com`.
+
+## How we defend against it
+
+### 1. Listmonk click tracking is intentionally OFF on per-subscriber CTAs
+
+Our conversion CTAs are per-subscriber links (`?dot=`, `?npi=`, `{{ lp_link }}`).
+Listmonk's `@TrackLink` was removed from those (it caused a 404 + collapse bug),
+so **"0 clicks" in Listmonk is expected and not a real signal**. Real clicks are
+attributed in Umami via the `campaign-click` event. See the runbook incident
+"Jun 22 2026 — `@TrackLink` on per-subscriber CTAs".
+
+### 2. Umami has a headless/scanner bot filter
+
+`site/public/js/pw-bot-filter.js` (loaded before the Umami script on every page,
+wired via `data-before-send="umamiBeforeSend"`) scores each visit for
+automation/headless signals and **drops the event when it looks like a bot**, so
+scanner traffic never reaches analytics. It is designed to **fail open**: any
+uncertainty counts as human, so we never undercount real people.
+
+Signals it uses (see the file for the full, commented list):
+
+- **Decisive (any one suppresses):**
+  - `navigator.webdriver === true` (Selenium/Puppeteer/Playwright, Chrome headless)
+  - Headless/automation user-agents (`HeadlessChrome`, `PhantomJS`, `Electron`, ...)
+  - Known scanner UAs (`Proofpoint`, `Mimecast`, `Barracuda`, `Cisco`,
+    `Symantec`, `MessageLabs`, `GoogleImageProxy`, `Microsoft Office`, ...)
+  - Zero/collapsed screen or viewport, or a window bigger than the monitor
+    (`outer-gt-screen`) — classic headless geometry
+  - A **software WebGL rasterizer** (`SwiftShader`, `llvmpipe`, `swrast`) on a
+    desktop-class UA — the single hardest signal to spoof; headless/VM scanners
+    fall back to software rendering
+  - Zero logical CPUs (`hardwareConcurrency === 0`)
+- **Soft (need two to corroborate):** tiny screen, low color depth, empty
+  `navigator.languages`, no pointer/hover/touch input surface, no WebGL at all.
+
+The result is exposed as `window.pwIsBot` / `window.pwBotReasons` for debugging.
+
+### 3. Server-side scanner exclusion
+
+Most known scanner user-agents are already excluded server-side, and the MX-tag
+pipeline classifies each recipient's receiving operator
+(`google`/`microsoft`/`proofpoint`/`mimecast`/`barracuda`/...) so campaign
+builders can throttle per operator. See `scripts/mx_tag_carriers.py`,
+`scripts/mail_reputation_monitor.py`, and `infra/cron/pw-mx-tag`.
+
+### 4. We measure conversion, not opens
+
+The A/B scoreboard (`scripts/coupon_ab_scoreboard.py`) explicitly counts **human
+clicks (Umami, bot-filtered) + paid orders**, never opens. The funnel we watch
+is:
+
+```text
+sent → accepted → HUMAN clicks (Umami) → order-page views → checkout starts → paid orders
+```
+
+## What this means in practice
+
+- **Never celebrate an open rate.** On a B2B list heavy with Microsoft/Proofpoint
+  MX, 20-40% opens can be almost entirely scanners.
+- **Even a raw click number can be inflated** if it comes from Listmonk link
+  tracking or an un-filtered analytics setup. Trust the **Umami `campaign-click`**
+  count (bot-filtered) and, above all, **order-page views and paid orders**.
+- **Instant, bulk, datacenter-sourced "engagement" right after send = scanners.**
+  Real humans trickle in over hours/days from residential/mobile networks.
+- If a campaign shows opens/clicks but **zero order-page views**, that is the
+  signature of scanner-only traffic, not a landing-page problem.
+
+## Related docs
+
+- `docs/email-deliverability-runbook.md` — sending infra, warmup, incidents
+  (including the `@TrackLink` click-tracking incident and the clean-pool decay fix).
+- `docs/email-no-sales-action-plan-2026-07-02.md` — conversion playbook; item #2
+  ("opens inflated by bots/scanners") is exactly this problem.
+- `site/public/js/pw-bot-filter.js` — the implementation.
+- `site/public/js/pw-analytics.js` — the `campaign-click` / `checkout-page-view`
+  event definitions.