- build_trucking_campaigns.py: nightly script that creates 8 Listmonk campaigns
per day (4 TZ x 2 types: MCS-150 overdue 2k/TZ, inactive USDOT 1k/TZ)
at 4AM ET / 5AM ET (CT) / 6AM ET (MT) / 7AM ET (PT). Deduplicates via
listmonk_sent_at column.
- migration 083: add listmonk_sent_at + listmonk_campaign_type to fmcsa_carriers
- email_verifier.py: bump max_workers from 5 to 20 for 4x faster throughput
- cron: daily pw-trucking-campaigns at 08:00 UTC (3 AM EST)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- CRTC letter now auto-emailed to secretary.general@crtc.gc.ca after eSign
- BITS admin todo updated to reference electronic + physical submission
- COLIN selectors.py: documented verification status per step
- BC config: added CRTC Secretary General email address
- plan.md: marked completed items (eSign, portal auth, CRTC email)
- go-live-todo.md: marked Compliance Calendar DocType as imported
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When any Playwright submission fails (selector not found, timeout, etc.):
1. Full-page screenshot captured and uploaded to MinIO
2. Telegram alert sent immediately with error details + screenshot link
3. Email alert to ops with same info
4. Admin todo includes screenshot MinIO path for debugging
5. Client order stays pending for manual completion
Proactive selector health check (daily 7am CT cron):
- Navigates to each portal (FCC RMD, USAC E-File, FCC CPNI/ECFS)
- Verifies all critical selectors are still present in the DOM
- If selectors are missing (UI changed): alerts via Telegram + email
BEFORE any real client order fails
- Reports which service slugs are affected
Integrated into:
- RMD filing handler (fccprod.servicenowservices.com)
- Form 499-A handler (forms.universalservice.org)
- Form 499-Q handler (already had error handling)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
After 499-A+Q bundle is filed, the handler now creates actual
compliance_orders for each remaining quarterly 499-Q filing:
Schedule: Q1 due Feb 1, Q2 due May 1, Q3 due Aug 1, Q4 due Nov 1
Each quarterly order:
- Created as paid (covered by bundle price)
- Has due_date, quarter, period_end_date in intake_data
- Links to parent 499-A order
- Tracks reminder status (30d/14d/7d sent flags)
Notification worker (quarterly_499q_notify.py):
- Runs daily at 8am CT via systemd timer
- Sends HTML reminder emails at 30, 14, 7 days before due
- Email includes intake link for client to submit quarterly data
- Late warning at 7 days: "USAC may estimate higher contributions"
- Idempotent: won't re-send same reminder level
Added fcc-499q service slug ($0, not sold standalone).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Comprehensive security update automation:
1. Debian OS (unattended-upgrades) — tightened to security-only:
- Removed general Debian updates (prevents feature/breaking changes)
- Only Debian-Security origins auto-installed
- Email admin on every upgrade via ops@performancewest.net
- Auto-reboot at 4 AM if kernel update requires it
- needrestart auto-restarts services after library updates
2. Docker CE — major version guard:
- Patch updates within pinned major version auto-applied
- Major version jumps held + admin alerted for manual review
- docker-ce, docker-ce-cli, containerd.io all version-guarded
3. Container base images — daily at 3:30 AM:
- Pulls latest base images for all docker-compose services
- Compares image digests — only rebuilds if changed
- Restarts only affected services (not full stack)
- Alerts admin on rebuild failures requiring manual intervention
- Covers both prod and dev compose projects
4. k3s — weekly Sunday at 3:45 AM:
- Patch updates within current minor auto-applied
- Minor/major upgrades alert admin for manual review
- Verifies node Ready status after update
- Alerts on failures with investigation instructions
5. Admin notifications via SMTP:
- [INFO] for successful patches
- [WARNING] for available major upgrades needing review
- [CRITICAL] for failures requiring immediate intervention
- Falls back to syslog if SMTP unavailable
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
