Add Playwright failure monitoring: Telegram alerts + screenshots + health check
When any Playwright submission fails (selector not found, timeout, etc.): 1. Full-page screenshot captured and uploaded to MinIO 2. Telegram alert sent immediately with error details + screenshot link 3. Email alert to ops with same info 4. Admin todo includes screenshot MinIO path for debugging 5. Client order stays pending for manual completion Proactive selector health check (daily 7am CT cron): - Navigates to each portal (FCC RMD, USAC E-File, FCC CPNI/ECFS) - Verifies all critical selectors are still present in the DOM - If selectors are missing (UI changed): alerts via Telegram + email BEFORE any real client order fails - Reports which service slugs are affected Integrated into: - RMD filing handler (fccprod.servicenowservices.com) - Form 499-A handler (forms.universalservice.org) - Form 499-Q handler (already had error handling) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
3e04edd384
commit
78c04b8bc3
4 changed files with 409 additions and 2 deletions
|
|
@ -477,10 +477,33 @@ class RMDFilingHandler(BaseServiceHandler):
|
|||
|
||||
except Exception as exc:
|
||||
logger.exception("RMDFilingHandler: portal submission failed: %s", exc)
|
||||
|
||||
# Upload screenshot + alert ops via Telegram
|
||||
screenshot_key = None
|
||||
try:
|
||||
from scripts.workers.services.telecom.playwright_monitor import (
|
||||
upload_failure_screenshot_async, alert_playwright_failure,
|
||||
)
|
||||
screenshot_key = await upload_failure_screenshot_async(
|
||||
page, order_number, "rmd-filing", work_dir,
|
||||
)
|
||||
alert_playwright_failure(
|
||||
order_number=order_number,
|
||||
service_slug="rmd-filing",
|
||||
service_name="RMD Registration",
|
||||
entity_name=entity.get("legal_name", ""),
|
||||
error=exc,
|
||||
screenshot_key=screenshot_key,
|
||||
portal_url="https://fccprod.servicenowservices.com/rmd",
|
||||
)
|
||||
except Exception as alert_exc:
|
||||
logger.warning("Playwright failure alert failed: %s", alert_exc)
|
||||
|
||||
self._create_admin_todo(
|
||||
order_number,
|
||||
f"Automated RMD submission failed for FRN {frn}: {exc}. "
|
||||
"Inspect worker logs for the screenshot/video, then file manually.",
|
||||
f"{'Screenshot: MinIO ' + screenshot_key if screenshot_key else 'No screenshot captured.'}"
|
||||
f"\nFile manually at https://fccprod.servicenowservices.com/rmd",
|
||||
)
|
||||
return None, ""
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue