infra: codify the email-campaign pipeline in Ansible (new mail-pipeline role)

The entire outbound campaign pipeline lived ONLY on the host and was never in
IaC -- a fresh rebuild would have silently shipped NO campaigns, NO IP warmup/
ramp, and NO bounce processing. New mail-pipeline role + deploy-mail-pipeline.yml
playbook deploy it from the canonical repo copies:

  cron.d (infra/cron/):
    - pw-trucking-campaign-builder, pw-ifta-campaign, pw-ucr-campaign
    - pw-hc-campaign, pw-hc-nppes, pw-hc-refresh
    - pw-mta-warmup, pw-listmonk-rampcap, pw-hc-rampcap
    - pw-ip-rehab, pw-warmup-tg-alert
  helper scripts (-> /usr/local/bin):
    - pw-mta-warmup, pw-listmonk-rampcap, pw-hc-rampcap, pw-warmup-tg-alert
    - postfix-bounce-notify.sh, postfix-hc-bounce-notify.sh, listmonk-bounce-sync.py
  systemd services:
    - pw-bounce-watcher.service (was missing from repo), pw-hc-bounce-watcher.service

Also creates the deploy-owned {{project_dir}}/logs dir (deploy can't write
/var/log, so a missing dir made cron redirects fail). Added the 6 cron.d files
that existed only on the host, the trucking bounce-watcher unit, and synced
infra/cron/pw-hc-refresh to the live version (revalidation download + enrich
steps). Role wired into site.yml after the mail (OpenDKIM) role.

Part of the email-deliverability incident hardening.

infra/ansible/playbooks/deploy-mail-pipeline.yml

@@ -0,0 +1,11 @@
+---
+# Deploy only the mail-pipeline role (campaign crons, IP warmup/ramp helpers,
+# bounce watchers). Run after changing anything under infra/cron, infra/postfix,
+# infra/monitoring, infra/systemd, or scripts/*bounce*.
+#
+# Usage: ansible-playbook playbooks/deploy-mail-pipeline.yml -i inventory/hosts.yml --ask-vault-pass
+- name: Deploy mail-pipeline (campaign crons + warmup + bounce watchers)
+  hosts: pw
+  become: true
+  roles:
+    - role: "{{ playbook_dir }}/../roles/mail-pipeline"

infra/ansible/playbooks/site.yml

@@ -16,6 +16,7 @@
 #   workers    — Python job server + Ollama LLM
 #   shkeeper   — k3s + Helm + SHKeeper (crypto payments: BTC/ETH/USDC/Polygon/TRX/BNB/LTC)
 #   mail       — OpenDKIM signing for outbound Postfix mail (incl. Listmonk campaigns)
+#   mail-pipeline — campaign cron builders + IP warmup/ramp + bounce watchers
 #   nginx      — nginx + certbot TLS for all domains + fail2ban
 
 - name: Provision Performance West server
@@ -33,6 +34,7 @@
     - worker-crons
     - shkeeper
     - mail
+    - mail-pipeline
     - nginx
     - monitoring
     - security-updates

infra/ansible/roles/mail-pipeline/defaults/main.yml

@@ -0,0 +1,7 @@
+---
+# mail-pipeline role defaults
+#
+# project_dir + deploy_user are normally provided by the common/app roles'
+# group_vars; these defaults keep the role self-contained.
+project_dir: /opt/performancewest
+deploy_user: deploy

infra/ansible/roles/mail-pipeline/handlers/main.yml

@@ -0,0 +1,14 @@
+---
+- name: Reload systemd
+  ansible.builtin.systemd:
+    daemon_reload: true
+
+- name: Restart pw-bounce-watcher
+  ansible.builtin.systemd:
+    name: pw-bounce-watcher.service
+    state: restarted
+
+- name: Restart pw-hc-bounce-watcher
+  ansible.builtin.systemd:
+    name: pw-hc-bounce-watcher.service
+    state: restarted

infra/ansible/roles/mail-pipeline/tasks/main.yml

@@ -0,0 +1,119 @@
+---
+# mail-pipeline role
+#
+# Codifies the outbound email-campaign pipeline that previously lived ONLY on
+# the host (none of this was in IaC before -- a fresh rebuild would have silently
+# shipped NO campaigns, NO IP warmup/ramp, and NO bounce processing):
+#
+#   - /etc/cron.d/pw-*        daily campaign builders + IP-warmup/ramp drivers
+#   - /usr/local/bin/pw-*     warmup/ramp/healthcheck helper scripts
+#   - /usr/local/bin/postfix-*-bounce-notify.sh   bounce watchers
+#   - pw-bounce-watcher / pw-hc-bounce-watcher    systemd watcher services
+#
+# The campaign BUILDER logic (scripts/build_*.py) is synced with the app/workers
+# code; this role only deploys the host-level glue (cron + helper scripts +
+# services). The OpenDKIM signing + mail.log logrotate live in the `mail` role.
+
+# ── log + state dirs ────────────────────────────────────────────────────────
+# The deploy user CANNOT write /var/log, so the deploy-owned cron jobs log to
+# /opt/performancewest/logs. A missing dir makes the `>>` redirect fail before
+# the command runs (cron then mails the error to deploy@ -> self-bounce).
+- name: Ensure deploy-owned cron log directory exists
+  ansible.builtin.file:
+    path: "{{ project_dir }}/logs"
+    state: directory
+    owner: "{{ deploy_user }}"
+    group: "{{ deploy_user }}"
+    mode: "0775"
+
+# ── warmup / ramp helper scripts (run as root: edit main.cf, restart cntrs) ──
+- name: Deploy mail warmup/ramp/healthcheck helper scripts
+  ansible.builtin.copy:
+    src: "{{ playbook_dir }}/../../{{ item.src }}"
+    dest: "/usr/local/bin/{{ item.dest }}"
+    owner: root
+    group: root
+    mode: "0755"
+  loop:
+    - { src: "infra/postfix/pw-mta-warmup.sh",       dest: "pw-mta-warmup" }
+    - { src: "infra/postfix/pw-listmonk-rampcap.sh", dest: "pw-listmonk-rampcap" }
+    - { src: "infra/postfix/pw-hc-rampcap.sh",       dest: "pw-hc-rampcap" }
+    - { src: "infra/monitoring/pw-warmup-tg-alert.sh", dest: "pw-warmup-tg-alert" }
+
+# ── bounce watchers (tail mail.log -> Listmonk bounce webhook) ──────────────
+- name: Deploy bounce-watcher scripts
+  ansible.builtin.copy:
+    src: "{{ playbook_dir }}/../../{{ item.src }}"
+    dest: "/usr/local/bin/{{ item.dest }}"
+    owner: root
+    group: root
+    mode: "0755"
+  loop:
+    - { src: "scripts/bounce-watcher.sh",    dest: "postfix-bounce-notify.sh" }
+    - { src: "scripts/hc-bounce-watcher.sh", dest: "postfix-hc-bounce-notify.sh" }
+  notify:
+    - Restart pw-bounce-watcher
+    - Restart pw-hc-bounce-watcher
+
+- name: Deploy bounce-watcher systemd units
+  ansible.builtin.copy:
+    src: "{{ playbook_dir }}/../../infra/systemd/{{ item }}"
+    dest: "/etc/systemd/system/{{ item }}"
+    owner: root
+    group: root
+    mode: "0644"
+  loop:
+    - pw-bounce-watcher.service
+    - pw-hc-bounce-watcher.service
+  notify:
+    - Reload systemd
+    - Restart pw-bounce-watcher
+    - Restart pw-hc-bounce-watcher
+
+- name: Enable + start bounce-watcher services
+  ansible.builtin.systemd:
+    name: "{{ item }}"
+    enabled: true
+    state: started
+    daemon_reload: true
+  loop:
+    - pw-bounce-watcher.service
+    - pw-hc-bounce-watcher.service
+
+# ── listmonk bounce-sync poller (host python, every 5 min via root crontab) ──
+- name: Deploy listmonk bounce-sync poller
+  ansible.builtin.copy:
+    src: "{{ playbook_dir }}/../../scripts/listmonk-bounce-sync.py"
+    dest: /usr/local/bin/listmonk-bounce-sync.py
+    owner: root
+    group: root
+    mode: "0755"
+
+- name: Schedule listmonk bounce-sync (root crontab, every 5 min)
+  ansible.builtin.cron:
+    name: listmonk-bounce-sync
+    minute: "*/5"
+    job: "/usr/bin/python3 /usr/local/bin/listmonk-bounce-sync.py >> /var/log/bounce-sync.log 2>&1"
+
+# ── campaign + warmup cron.d files ──────────────────────────────────────────
+# These reference scripts/ in {{ project_dir }} and the docker compose stack, so
+# they are deployed verbatim from infra/cron/ (the canonical, reviewed copies).
+- name: Deploy campaign + warmup cron.d files
+  ansible.builtin.copy:
+    src: "{{ playbook_dir }}/../../infra/cron/{{ item }}"
+    dest: "/etc/cron.d/{{ item }}"
+    owner: root
+    group: root
+    mode: "0644"
+  loop:
+    - pw-trucking-campaign-builder
+    - pw-ifta-campaign
+    - pw-ucr-campaign
+    - pw-hc-campaign
+    - pw-hc-nppes
+    - pw-hc-refresh
+    - pw-mta-warmup
+    - pw-listmonk-rampcap
+    - pw-hc-rampcap
+    - pw-ip-rehab
+    - pw-warmup-tg-alert