new-site/scripts/ops/carbonio/README.md

Carbonio noreply@ mailbox auto-purge

Server-side maintenance for the noreply@performancewest.net mailbox on the Carbonio (Zextras) mail host co.carrierone.com.

Problem

The noreply@ mailbox accumulated 35,337 messages (~488 MB). A sampled audit showed ~98.6% were machine noise: bounce DSNs (this box's own Postfix backscatter), out-of-office / auto-reply messages, and helpdesk/ticket auto-acknowledgements. Buried in the rest were a small number of genuine human replies to the trucking (DOT#/MCS-150) and telecom/FCC campaigns -- these land here because of the historical Reply-To behaviour -- plus the occasional unsubscribe request.

Policy (explicit)

• DELETE: bounces, ticket/case auto-acknowledgements, out-of-office and auto-reply messages, delivery-status notifications, authentication reports.
• KEEP: genuine human replies (Re:/Fwd:) and unsubscribe/opt-out requests.
• Fail-safe: when a message is not clearly machine-generated, KEEP it.
• Deletions move to /Trash (reversible), never hard-delete.

Why a header/sender classifier, not subject matching

Subject text alone is unreliable: auto-responders frequently reply with a deceptive Re: prefix (e.g. an auto-responder answering our campaign with Re: <our subject>). The classifier therefore uses, in precedence order:

1. Unsubscribe guard (compliance) -- always KEEP, overrides everything.
2. RFC 3834 Auto-Submitted: header -- if present and != no, the sending system has declared the message automatic (bounces = auto-generated, vacation/auto-replies = auto-replied). This is the single most reliable signal and it catches the deceptive Re: auto-responders.
3. Machine From-address -- exact bot localparts (mailer-daemon, postmaster, no-reply, ...), strong tokens anywhere in the localpart (...-bounces@, expense-noreply-...@, auth-results@), and display-name bots (Mail Delivery System, System Administrator, ...).
4. STRONG auto subjects -- unambiguous machine markers no human types (New Ticket Created, (autoresponse), Auto Re:, your request with id ##...##, we're on it, Undeliverable, Authentication Report, ...). Checked before the human Re: guard so ticket auto-acks dressed as Re: are still removed.
5. Human Re:/Fwd: -- KEEP.
6. Ticket tag [##...##] / broad auto-ack subjects -- DELETE.
7. Default -> KEEP (human-safe).

Subjects are RFC 2047 MIME-decoded first (campaign subjects contain an em-dash, so they arrive =?utf-8?Q?...?= encoded and would otherwise evade matching).

The ruleset was validated against a hand-labelled set drawn from the live mailbox: 23/23 cases correct, including keeping the real Re: replies from the same campaigns whose auto-responder twins were deleted.

Execution model

nr_purge.sh runs in three stages so the expensive part stays small:

• Phase 1 -- fast server-side search-delete of from:MAILER-DAEMON bounces (the ~97% bulk), guarded against unsubscribe. No per-message fetch.
• Phase 1.5 -- fast search-delete of the common non-MAILER machine classes (from:postmaster, Undeliverable, automatic reply, out of office, delivery status notification), each hard-guarded with AND NOT (subject:Re OR subject:Fwd OR subject:unsubscribe ...) so anything ambiguous falls through to the accurate classifier.
• Phase 2 -- header-classify the small remainder one message at a time using the full decide() ruleset; KEEP decisions are cached so survivors are not re-fetched on subsequent pages.

On the initial backfill this reduced 35,337 -> 68 messages (67 genuine human replies + 1 unsubscribe), moving ~35,269 machine items to Trash.

Usage

# read-only preview of the N most-recent messages (prints survivors + sample deletes)
bash nr_purge.sh --preview 150

# full purge (move matches to /Trash)
bash nr_purge.sh --apply

# date-bounded purge (only inspect last N days) -- used by the daily cron
bash nr_purge.sh --apply --days 3

# Phase-1-only fast bounce sweep
bash nr_purge.sh --apply --quick

Deployment

The script lives on the Carbonio host at /opt/zextras/nr_purge.sh (and a copy in ~zextras/). It must run as the zextras user (owns zmmailbox).

A daily cron is installed in root's crontab (not the zextras crontab, which Carbonio/zmcontrol regenerates and would wipe):

17 4 * * * su - zextras -c 'bash /opt/zextras/nr_purge.sh --apply --days 3' >> /var/log/nr_purge_cron.log 2>&1

--days 3 keeps the daily run cheap: it only header-inspects mail from the last three days (a few dozen messages), which is more than enough overlap to catch anything that arrived since the previous run.

To (re)deploy after editing this file:

scp -P 22022 nr_purge.sh justin@co.carrierone.com:/tmp/nr_purge.sh
ssh -p 22022 justin@co.carrierone.com \
  'sudo cp /tmp/nr_purge.sh /opt/zextras/nr_purge.sh && sudo chown zextras: /opt/zextras/nr_purge.sh && sudo chmod +x /opt/zextras/nr_purge.sh'

Notes / gotchas

• zmmailbox search -l works up to 1000 results/page; offset paging (-o) does not work reliably and large limits (2000+) silently return empty. The script loops on "delete the top page, re-search" instead of offset paging.
• Trash still counts against mailbox size until emptied. The initial backfill left Trash populated (reversible); emptying it is an optional, irreversible follow-up.