diff --git a/docs/email-phantom-opens-scanners.md b/docs/email-phantom-opens-scanners.md new file mode 100644 index 0000000..026df00 --- /dev/null +++ b/docs/email-phantom-opens-scanners.md @@ -0,0 +1,156 @@ +# Phantom Opens & Clicks from Email Security Scanners + +Date: 2026-07-03 + +## In plain English (no tech background needed) + +When we send a marketing email, most of the businesses we email use a **security +guard for their inbox** (Microsoft, Proofpoint, Mimecast, and similar). Before a +real person ever sees our email, that guard **opens the email and clicks every +link in it first**, just to make sure nothing is dangerous. Think of it like the +mailroom in a big office opening and X-raying a package before handing it to the +employee. + +Here is the problem: our tools can't easily tell the difference between the guard +and a real person. When the guard opens the email, it counts as an "open." When +the guard clicks the link to inspect it, it can count as a "click." So our reports +can say *"30% of people opened it and some clicked!"* when in reality it was mostly +robots checking for viruses, and **no actual human was interested**. + +That's why we don't trust "opens" or even raw "clicks" as proof that the campaign +is working. A robot opening an email doesn't pay us. So instead we look at things a +robot almost never does: + +- Did someone reach the **order page** for a service? +- Did they **start checkout**? +- Did they **actually pay**? + +Those are the numbers that matter. If a campaign shows lots of opens and clicks but +nobody reached the order page, that's a strong sign it was mostly the robot +security guards, not real customers. We also run a **filter** that spots the +tell-tale signs of these robots and quietly ignores them, so our real numbers only +count real people. + +Bottom line: **"opens" and "clicks" look impressive but are mostly robots. Real +interest = order-page visits, checkouts, and payments.** + +## The short version + +Email opens, and even *clicks*, are unreliable signals because **email security +gateways automatically fetch and render every link in a message** to scan it for +malware, before the human ever sees the email. Those automated fetches fire the +same tracking pixels and, for the advanced scanners that drive a real headless +browser, the same JavaScript pageviews/clicks that a human would. So a campaign +can show 20-40% "opens" and a handful of "clicks" with **zero real human +engagement**. + +This is why we treat **Listmonk opens as noise**, do our conversion tracking in +**Umami with a bot filter**, and judge campaigns on **human clicks → order-page +views → checkout → paid orders**, never on opens. + +## Who does this and how + +| Gateway | Product | Behavior | +| --- | --- | --- | +| Microsoft | Defender for O365 "Safe Links" / ATP detonation | Rewrites every URL, fetches it, and **detonates** suspicious ones in a real browser sandbox. Very common on business/`outlook`/M365 tenants. | +| Proofpoint | URL Defense (`urldefense.com` / `pphosted.com`) | Rewrites and pre-fetches every link; re-scans on click. | +| Mimecast | URL Protect / Attachment Protect | Pre-fetches and rewrites links. | +| Barracuda | Link Protect | Fetches links at delivery. | +| Cisco / Symantec / MessageLabs | Secure Email / ClickTime | Scan-on-delivery and scan-on-click. | +| Google | Gmail image proxy (`GoogleImageProxy`) | Pre-fetches images (inflates *opens*, not usually JS clicks). | + +Two distinct effects: + +1. **Inflated opens.** Any gateway that loads the tracking pixel (or Gmail's + image proxy caching it) counts as an "open" in Listmonk. This happens on a + large fraction of B2B mail. Opens are therefore close to meaningless. +2. **Phantom clicks / pageviews.** The advanced scanners (Microsoft ATP, + Proofpoint, Mimecast) run a **headless browser** that executes JavaScript. + That fires a real Umami pageview and can trip click events, so the visit + *looks* human. This is the dangerous one because it corrupts the metric we + actually rely on (clicks), not just the one we already ignore (opens). + +Extra wrinkles: + +- Scanners often fetch **within seconds of delivery**, in bulk, from datacenter + IPs, before any human could plausibly read the email. +- They may fetch **every** link, so a "click" on a deep CTA can be a scanner + walking all URLs, not a human choosing one. +- Microsoft rewrites the URL, so the referrer/host can look like + `*.safelinks.protection.outlook.com`; Proofpoint uses `urldefense.com`. + +## How we defend against it + +### 1. Listmonk click tracking is intentionally OFF on per-subscriber CTAs + +Our conversion CTAs are per-subscriber links (`?dot=`, `?npi=`, `{{ lp_link }}`). +Listmonk's `@TrackLink` was removed from those (it caused a 404 + collapse bug), +so **"0 clicks" in Listmonk is expected and not a real signal**. Real clicks are +attributed in Umami via the `campaign-click` event. See the runbook incident +"Jun 22 2026 — `@TrackLink` on per-subscriber CTAs". + +### 2. Umami has a headless/scanner bot filter + +`site/public/js/pw-bot-filter.js` (loaded before the Umami script on every page, +wired via `data-before-send="umamiBeforeSend"`) scores each visit for +automation/headless signals and **drops the event when it looks like a bot**, so +scanner traffic never reaches analytics. It is designed to **fail open**: any +uncertainty counts as human, so we never undercount real people. + +Signals it uses (see the file for the full, commented list): + +- **Decisive (any one suppresses):** + - `navigator.webdriver === true` (Selenium/Puppeteer/Playwright, Chrome headless) + - Headless/automation user-agents (`HeadlessChrome`, `PhantomJS`, `Electron`, ...) + - Known scanner UAs (`Proofpoint`, `Mimecast`, `Barracuda`, `Cisco`, + `Symantec`, `MessageLabs`, `GoogleImageProxy`, `Microsoft Office`, ...) + - Zero/collapsed screen or viewport, or a window bigger than the monitor + (`outer-gt-screen`) — classic headless geometry + - A **software WebGL rasterizer** (`SwiftShader`, `llvmpipe`, `swrast`) on a + desktop-class UA — the single hardest signal to spoof; headless/VM scanners + fall back to software rendering + - Zero logical CPUs (`hardwareConcurrency === 0`) +- **Soft (need two to corroborate):** tiny screen, low color depth, empty + `navigator.languages`, no pointer/hover/touch input surface, no WebGL at all. + +The result is exposed as `window.pwIsBot` / `window.pwBotReasons` for debugging. + +### 3. Server-side scanner exclusion + +Most known scanner user-agents are already excluded server-side, and the MX-tag +pipeline classifies each recipient's receiving operator +(`google`/`microsoft`/`proofpoint`/`mimecast`/`barracuda`/...) so campaign +builders can throttle per operator. See `scripts/mx_tag_carriers.py`, +`scripts/mail_reputation_monitor.py`, and `infra/cron/pw-mx-tag`. + +### 4. We measure conversion, not opens + +The A/B scoreboard (`scripts/coupon_ab_scoreboard.py`) explicitly counts **human +clicks (Umami, bot-filtered) + paid orders**, never opens. The funnel we watch +is: + +```text +sent → accepted → HUMAN clicks (Umami) → order-page views → checkout starts → paid orders +``` + +## What this means in practice + +- **Never celebrate an open rate.** On a B2B list heavy with Microsoft/Proofpoint + MX, 20-40% opens can be almost entirely scanners. +- **Even a raw click number can be inflated** if it comes from Listmonk link + tracking or an un-filtered analytics setup. Trust the **Umami `campaign-click`** + count (bot-filtered) and, above all, **order-page views and paid orders**. +- **Instant, bulk, datacenter-sourced "engagement" right after send = scanners.** + Real humans trickle in over hours/days from residential/mobile networks. +- If a campaign shows opens/clicks but **zero order-page views**, that is the + signature of scanner-only traffic, not a landing-page problem. + +## Related docs + +- `docs/email-deliverability-runbook.md` — sending infra, warmup, incidents + (including the `@TrackLink` click-tracking incident and the clean-pool decay fix). +- `docs/email-no-sales-action-plan-2026-07-02.md` — conversion playbook; item #2 + ("opens inflated by bots/scanners") is exactly this problem. +- `site/public/js/pw-bot-filter.js` — the implementation. +- `site/public/js/pw-analytics.js` — the `campaign-click` / `checkout-page-view` + event definitions.