The custom_contact_email field doesn't exist on Sales Order DocType,
causing the email-based fallback query to crash. Simplified to use
Customer record lookup only (email_id match works).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Disable sidebar (was auto-linking order names as broken relative URLs)
- Remove broken portal_user_name lookup (field doesn't exist)
- Match orders by Customer record OR by custom_contact_email on Sales Order
- Merges both result sets without duplicates
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- CRTC letter now auto-emailed to secretary.general@crtc.gc.ca after eSign
- BITS admin todo updated to reference electronic + physical submission
- COLIN selectors.py: documented verification status per step
- BC config: added CRTC Secretary General email address
- plan.md: marked completed items (eSign, portal auth, CRTC email)
- go-live-todo.md: marked Compliance Calendar DocType as imported
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Compliance Deadline: created by job_server after filing completion,
tracks annual filing deadlines (RMD, CPNI, 499-A, BDC).
Compliance Calendar: used by renewal_worker for billing cycle,
tracks due dates, invoices, and recurring renewal entries.
Both were referenced in code but never created — caused 417 errors.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
_convert_to_pdf() now calls pdf_converter.convert_to_pdf() which tries
the Windows Word VM via MinIO first (pixel-perfect), falling back to
LibreOffice headless automatically when the VM is unavailable.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- upload_file doesn't exist on MinioStorage — the method is upload()
- Return [] when pausing for eSign so job_server doesn't trigger instant delivery
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Each handler now pauses for officer signature via the eSign portal
before filing/submitting. esign_completed callback re-dispatches
through standard pipeline with client_approved=true.
- CPNI: officer signs certification before ECFS submission (perjury)
- CALEA SSI: officer signs plan before delivery
- 499-A engagement: replaced custom JWT/email with request_esign()
- Discontinuance: officer signs deactivation letter before USAC email
- job_server: injects client_approved + order_number into order_data
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Reusable signing flow: service handler generates document → inserts
esign_records row → emails JWT link → client reviews PDF + signs →
API stores signature + resumes pipeline. Works for RMD, CPNI, CALEA,
499-A engagement, discontinuance, CRTC, and any future doc types.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The checklist was a manual-process artifact that listed what info the
client needed to gather. Since all data is now collected through the
intake wizard and CDR upload, the checklist is unnecessary. Removed
from the 499-A handler's prep packet and deleted the 1,326-line
generator file.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
_styles.py: Centralized typography, spacing, and formatting for all
26 DOCX generators. Calibri 9.5pt body, 1.15 line spacing, navy
headings, consistent signature blocks, page numbers, PW footer.
All generators will be migrated to use this instead of defining
their own styles.
Campaign tools:
- campaign_template.html: Styled email template for Listmonk campaigns
- populate_deficiency_list.py: Populates Listmonk with FCC deficiency data
- send_test_campaigns.py: Sends test emails with real carrier data
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sends to the monitoring bot immediately when payment is confirmed:
- Customer name and email
- Service/slug ordered
- Total amount (includes all fees: service + formation + state + addons)
- Payment method
- Order number and type
Fire-and-forget — never blocks the payment flow.
Requires TELEGRAM_BOT_TOKEN and TELEGRAM_CHAT_ID env vars on API container.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Critical #1 — CRTC: Fix undefined 'province' variable (canada_crtc.py:1322)
Crashes every order at Step 6 document generation. Replaced with
order_data.get("custom_incorporation_province", "BC").
Critical #2 — FCC Carrier Reg: Add State PUC state picker
The order page collected "1/few/nationwide" but API expected an array
of state codes. Added a multi-state checkbox grid that appears when
State PUC add-on is checked. Sends puc_states: ["CA","NY",...] in
service_wizard. Price updates per-state ($399 × count).
Critical #3 — Compliance: Add REQUIRED_FIELDS for fcc-499q and
fcc-499a-discontinuance. Without these, intake validation was
completely skipped — invalid data accepted silently.
High #4 — FCC Carrier Reg: Don't mark D.C. Agent complete
prematurely. Was calling _update_step() right after creating the
admin todo. Now waits for admin to confirm NW order is placed.
High #5 — Compliance: Add fcc-499q and fcc-499a-discontinuance to
REQUIRES_ENTITY_FRN set. Both require FRN for USAC filing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When any Playwright submission fails (selector not found, timeout, etc.):
1. Full-page screenshot captured and uploaded to MinIO
2. Telegram alert sent immediately with error details + screenshot link
3. Email alert to ops with same info
4. Admin todo includes screenshot MinIO path for debugging
5. Client order stays pending for manual completion
Proactive selector health check (daily 7am CT cron):
- Navigates to each portal (FCC RMD, USAC E-File, FCC CPNI/ECFS)
- Verifies all critical selectors are still present in the DOM
- If selectors are missing (UI changed): alerts via Telegram + email
BEFORE any real client order fails
- Reports which service slugs are affected
Integrated into:
- RMD filing handler (fccprod.servicenowservices.com)
- Form 499-A handler (forms.universalservice.org)
- Form 499-Q handler (already had error handling)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
job_server.py expects process() to return a list of file paths for
MinIO upload. The 499-Q and discontinuance handlers were returning
dicts like {'status':'admin_review'} which caused the job_server to
iterate dict keys as file paths -> 'File not found: status'.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When AUTO_FILING_ENABLED is explicitly set as an env var, skip the
ERPNext API call entirely. The ERPNext client hangs indefinitely
when the host is unreachable (dev workers can't reach prod ERPNext),
blocking all compliance handlers.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
499-Q Handler:
- Auto-filing toggle integration (same as 499-A)
- Playwright USAC E-File submission for quarterly form
- Revenue field filling (4 categories)
- Confirmation number capture + PDF save
- Client receives "data received" email immediately, then
"filed successfully" email with confirmation number after submission
- Falls back to admin todo if Playwright/session unavailable
Discontinuance Handler:
- Auto-emails deactivation letter to USAC (Form499@usac.org)
with DOCX attachment + entity summary in body
- CC to admin email for records
- Dev mode: redirects USAC email to admin instead
- Client confirmation email with process timeline
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Formal letter addressed to USAC Contributor Operations requesting
deactivation of a 499 Filer ID. Covers:
- Entity identification (name, Filer ID, FRN, EIN)
- Reason for deactivation + termination date
- Final 499-A status (zero-revenue included OR filed separately)
- Successor entity info (if applicable)
- Outstanding balance acknowledgment
- Related filings confirmation (RMD, CPNI, BDC)
- Officer signature block
- Entity summary box
Handler updated to:
- Generate the letter via document_gen template
- Upload DOCX to MinIO (compliance/{order_number}/)
- Reference the letter in admin todo
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Both scrapers held a cursor/transaction open while doing slow HTTP
requests to FCC ServiceNow and company websites, causing
"idle in transaction" for 10+ minutes and triggering the
PostgresSlowQueries alert.
Fix: fetch all row IDs upfront, commit the read transaction
immediately, then process each row with its own short
UPDATE+COMMIT cycle. No long-lived transactions.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When user selects "cancel registration" in the compliance checker:
- Option 1: "499-A Discontinuance (incl. zero-revenue filing)" $299
For carriers with no revenue — includes final zero-revenue 499-A
+ deactivation letter + CORES update
- Option 2: "499-A Filing + Discontinuance" $798 ($499+$299)
For carriers with actual revenue — full 499-A filed separately
+ deactivation process
Standalone discontinuance ($299) is for carriers already current
on filings who just want to close out.
Handler detects whether zero-revenue filing is included vs
handled by a separate full 499-A order.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Per 2026 FCC Form 499-A Instructions:
- Final 499-A CAN have actual revenue (not required to be zero)
— reports revenue for the period the company was in service
- Deactivation is a SEPARATE step: submit letter to USAC with
termination date and successor entity info, within 30 days
- Line 603: check TRS/LNP/NANPA exemption boxes, write
"Not in business as of [date]"
- USAC processing takes 60-90 days
- CORES must be updated to reflect inactive status
Handler now creates admin todo with 4-step process:
1. File final 499-A with actual/zero revenue
2. Submit USAC deactivation letter (Form499@usac.org)
3. Update CORES registration
4. Confirm CPNI/RMD/BDC discontinued
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
After 499-A+Q bundle is filed, the handler now creates actual
compliance_orders for each remaining quarterly 499-Q filing:
Schedule: Q1 due Feb 1, Q2 due May 1, Q3 due Aug 1, Q4 due Nov 1
Each quarterly order:
- Created as paid (covered by bundle price)
- Has due_date, quarter, period_end_date in intake_data
- Links to parent 499-A order
- Tracks reminder status (30d/14d/7d sent flags)
Notification worker (quarterly_499q_notify.py):
- Runs daily at 8am CT via systemd timer
- Sends HTML reminder emails at 30, 14, 7 days before due
- Email includes intake link for client to submit quarterly data
- Late warning at 7 days: "USAC may estimate higher contributions"
- Idempotent: won't re-send same reminder level
Added fcc-499q service slug ($0, not sold standalone).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
New service slug fcc-499a-zero for carriers with no telecom revenue:
- $179 instead of $499 (no revenue analysis needed)
- Minimal intake: entity, officer, filer ID, filing type only
- Skips revenue schedules (blocks 3-4), USF calculations (block 5),
traffic study upload, and revenue workbook generation
- Fills blocks 1-2 and 6 only, all revenue lines left as zero
Compliance checker: shows both options (mutually exclusive checkboxes)
Order page: maps form_499a_zero to fcc-499a-zero slug
Handler: detects slug and skips revenue pipeline
DC Agent shown when either 499-A variant is checked
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
FCC APIs (CORES, ServiceNow) are responding in 28-45 seconds.
The 30s frontend timeout was being hit ~50% of the time. Added
quick=1 to skip slow Playwright corp checks and increased timeout
to 60s as safety margin.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When both voice and broadband are selected, BDC status is red but
the CTA had no case for red — nothing was added to the order.
Now: red = "BDC Broadband + Voice Filing" at $349 (both filings),
yellow = single filing at $199 (unchanged).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The BDC toggle was in the static HTML file (site/public/tools/)
which is overridden by the Astro page. Fixed the ACTUAL page at
site/src/pages/tools/fcc-compliance-check.astro.
Two-step flow: asks about retail voice first, then broadband.
Results: voice+broadband=red (both filings), voice-only=yellow
(BDC Voice), broadband-only=yellow (BDC Broadband), neither=green.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
STIR/SHAKEN card creates confusion with OCN bundling on the order
page — if carried over, it doesn't auto-select OCN. Self-reported
RMD status also doesn't reliably indicate actual compliance.
Reverted to hidden (computed internally but not shown to users).
Removed STIR toggle and pending-question watcher for it.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
BDC Voice Subscription filing is required for all retail voice
providers (VoIP, CLEC, wireline) even without broadband service.
The compliance checker was asking about broadband first, which
caused voice-only carriers to overlook their BDC Voice obligation.
- Renamed card: "BDC Filing (Broadband + Voice Subscription Data)"
- Reversed question order: ask retail voice first, then broadband
- "No" for voice now says "No, or wholesale only" (wholesale exempt)
- Voice-only retail carriers correctly flagged for BDC Voice filing
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Containers without a memory limit have spec_memory_limit_bytes=0,
causing division to produce +Inf which always fires. Added guard:
only alert when a limit is actually set (> 0).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Dashboard queries now use max() to pick UP value when old stale
probe targets coexist with new ones. Prometheus admin API enabled
for future TSDB cleanup of stale series.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- ERPNext: custom blackbox module with Host: performancewest.net header
(ERPNext multitenancy requires site name in Host for routing)
- Forgejo: add extra_hosts to blackbox-exporter so it can resolve
host.docker.internal to reach forgejo on port 3000
- Blackbox http_erpnext module: sets Host header, expects 200
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Workers: use http_internal module (HTTP/1.0 SimpleHTTPServer)
- ERPNext: use /api/method/ping, accept 401/403 (still means alive)
- Listmonk: use /health not /api/health (403 without auth)
- Forgejo: port 3000 not 3030
- Dev API: probe via HTTPS public URL (blackbox can't reach Docker)
- Added http_internal blackbox module accepting HTTP/1.0 + 401/403
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Each service gets its own Prometheus probe verifying actual functionality:
- API: /status endpoint (checks DB connectivity, returns 503 if down)
- Workers: /health endpoint (job server responsive)
- ERPNext: API method call (MariaDB + Redis + app all working)
- MinIO: /minio/health/live (storage accessible)
- Listmonk: /api/health (email service + DB)
- Ollama: root endpoint (LLM inference available)
- Umami: /api/heartbeat (analytics tracking)
- Forgejo: root page (git server accessible)
- PostgreSQL: pg_up metric from postgres-exporter
- All HTTPS endpoints: SSL + reachability from outside
Service-specific alerts with context:
- API down = DB may be unreachable
- Workers down = compliance orders not processing
- ERPNext down = CRM inaccessible
- MinIO down = document storage unavailable
Custom Grafana dashboard: "Performance West — Services Overview"
- Service status grid (UP/DOWN with colors)
- Response time charts (internal + HTTPS)
- SSL certificate expiry gauges
- Container CPU/memory per service
- PostgreSQL connections, nginx req/s, active alerts
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
MinIO returns 403 when accessed via minio.performancewest.net because
it interprets the Host header as a bucket name. Switch blackbox probe
to internal http://minio:9000/minio/health/live which works correctly.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
host network mode prevented Prometheus from reaching the exporter.
Switched back to bridge with extra_hosts + explicit port mapping.
Added timeout flag to prevent hanging on stub_status fetch.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
nginx-exporter couldn't reach host nginx via host.docker.internal
(connection timeout). Switch to network_mode: host so it can access
127.0.0.1:8888 directly. Prometheus scrapes via host.docker.internal
with extra_hosts mapping.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- nginx stub_status moved to port 8888 (port 80 was being caught
by other server blocks and returning 301)
- nginx-exporter updated to scrape :8888
- Added alertmanager scrape job to Prometheus config (was missing,
so alertmanager dashboard had no data)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Community dashboards reference datasource uid=prometheus but the
auto-generated UID was random. Pin to uid=prometheus for compatibility.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Comprehensive security update automation:
1. Debian OS (unattended-upgrades) — tightened to security-only:
- Removed general Debian updates (prevents feature/breaking changes)
- Only Debian-Security origins auto-installed
- Email admin on every upgrade via ops@performancewest.net
- Auto-reboot at 4 AM if kernel update requires it
- needrestart auto-restarts services after library updates
2. Docker CE — major version guard:
- Patch updates within pinned major version auto-applied
- Major version jumps held + admin alerted for manual review
- docker-ce, docker-ce-cli, containerd.io all version-guarded
3. Container base images — daily at 3:30 AM:
- Pulls latest base images for all docker-compose services
- Compares image digests — only rebuilds if changed
- Restarts only affected services (not full stack)
- Alerts admin on rebuild failures requiring manual intervention
- Covers both prod and dev compose projects
4. k3s — weekly Sunday at 3:45 AM:
- Patch updates within current minor auto-applied
- Minor/major upgrades alert admin for manual review
- Verifies node Ready status after update
- Alerts on failures with investigation instructions
5. Admin notifications via SMTP:
- [INFO] for successful patches
- [WARNING] for available major upgrades needing review
- [CRITICAL] for failures requiring immediate intervention
- Falls back to syslog if SMTP unavailable
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Users could skip Q1b (customer type) and Q2 (voice delivery) and
hit Next — the wizard silently defaulted to retail. Now validates:
- Q1b must be answered (customer type selected)
- Q2 must be answered if voice is checked and not wholesale-only
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
STIRShakenStep intake:
- New "Terminate only" option for carriers that only receive pre-signed
calls and don't originate
- Contextual hints for each option explaining requirements
- Show/hide vendor and upstream fields based on selection
RMD letter generator:
- New terminate_only section explaining verification-only posture,
citing 47 CFR § 64.6301 (signing obligation on originating provider)
- Added to needs_exhibit_a list
RMD Exhibit A generator:
- New terminate_only STIR/SHAKEN paragraph with SBC verification language
- Fixed scope paragraph: wholesale/facilities carriers no longer get
"small provider without Class 4 switch" boilerplate
- Fixed OCN paragraph: wholesale carriers get neutral wording instead
of "no OCN required for small retail provider"
RMD filing handler:
- Maps stir_shaken_status to rmd_option for Exhibit A generation
- Passes entity metadata (ocn, wholesale, gateway, contact) to generator
- Maps terminate_only → partial_implementation for FCC RMD form radio
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
