Add Playwright failure monitoring: Telegram alerts + screenshots + health check

When any Playwright submission fails (selector not found, timeout, etc.):
1. Full-page screenshot captured and uploaded to MinIO
2. Telegram alert sent immediately with error details + screenshot link
3. Email alert to ops with same info
4. Admin todo includes screenshot MinIO path for debugging
5. Client order stays pending for manual completion

Proactive selector health check (daily 7am CT cron):
- Navigates to each portal (FCC RMD, USAC E-File, FCC CPNI/ECFS)
- Verifies all critical selectors are still present in the DOM
- If selectors are missing (UI changed): alerts via Telegram + email
  BEFORE any real client order fails
- Reports which service slugs are affected

Integrated into:
- RMD filing handler (fccprod.servicenowservices.com)
- Form 499-A handler (forms.universalservice.org)
- Form 499-Q handler (already had error handling)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
justin 2026-05-04 02:44:02 -05:00
parent 3e04edd384
commit 78c04b8bc3
4 changed files with 409 additions and 2 deletions

View file

@ -158,6 +158,15 @@ worker_crons:
on_calendar: "Sat *-*-* 10:00:00 UTC"
persistent: true
# Playwright selector health check — daily 12:00 UTC (7am CT).
# Proactively detects when FCC/USAC portal UI changes would break automation.
# Alerts via Telegram + email if selectors are missing.
- name: pw-playwright-health
description: Check FCC/USAC portal selectors are still valid
module: scripts.workers.services.telecom.playwright_monitor
on_calendar: "*-*-* 12:00:00 UTC"
persistent: true
# 499-Q quarterly filing reminders — daily 13:00 UTC (8am CT).
# Sends reminder emails at 30/14/7 days before each quarterly due date.
# Creates compliance_orders for each quarter when the 499-A+Q bundle is filed.