Both scrapers held a cursor/transaction open while doing slow HTTP
requests to FCC ServiceNow and company websites, causing
"idle in transaction" for 10+ minutes and triggering the
PostgresSlowQueries alert.
Fix: fetch all row IDs upfront, commit the read transaction
immediately, then process each row with its own short
UPDATE+COMMIT cycle. No long-lived transactions.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When user selects "cancel registration" in the compliance checker:
- Option 1: "499-A Discontinuance (incl. zero-revenue filing)" $299
For carriers with no revenue — includes final zero-revenue 499-A
+ deactivation letter + CORES update
- Option 2: "499-A Filing + Discontinuance" $798 ($499+$299)
For carriers with actual revenue — full 499-A filed separately
+ deactivation process
Standalone discontinuance ($299) is for carriers already current
on filings who just want to close out.
Handler detects whether zero-revenue filing is included vs
handled by a separate full 499-A order.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Per 2026 FCC Form 499-A Instructions:
- Final 499-A CAN have actual revenue (not required to be zero)
— reports revenue for the period the company was in service
- Deactivation is a SEPARATE step: submit letter to USAC with
termination date and successor entity info, within 30 days
- Line 603: check TRS/LNP/NANPA exemption boxes, write
"Not in business as of [date]"
- USAC processing takes 60-90 days
- CORES must be updated to reflect inactive status
Handler now creates admin todo with 4-step process:
1. File final 499-A with actual/zero revenue
2. Submit USAC deactivation letter (Form499@usac.org)
3. Update CORES registration
4. Confirm CPNI/RMD/BDC discontinued
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
After 499-A+Q bundle is filed, the handler now creates actual
compliance_orders for each remaining quarterly 499-Q filing:
Schedule: Q1 due Feb 1, Q2 due May 1, Q3 due Aug 1, Q4 due Nov 1
Each quarterly order:
- Created as paid (covered by bundle price)
- Has due_date, quarter, period_end_date in intake_data
- Links to parent 499-A order
- Tracks reminder status (30d/14d/7d sent flags)
Notification worker (quarterly_499q_notify.py):
- Runs daily at 8am CT via systemd timer
- Sends HTML reminder emails at 30, 14, 7 days before due
- Email includes intake link for client to submit quarterly data
- Late warning at 7 days: "USAC may estimate higher contributions"
- Idempotent: won't re-send same reminder level
Added fcc-499q service slug ($0, not sold standalone).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
New service slug fcc-499a-zero for carriers with no telecom revenue:
- $179 instead of $499 (no revenue analysis needed)
- Minimal intake: entity, officer, filer ID, filing type only
- Skips revenue schedules (blocks 3-4), USF calculations (block 5),
traffic study upload, and revenue workbook generation
- Fills blocks 1-2 and 6 only, all revenue lines left as zero
Compliance checker: shows both options (mutually exclusive checkboxes)
Order page: maps form_499a_zero to fcc-499a-zero slug
Handler: detects slug and skips revenue pipeline
DC Agent shown when either 499-A variant is checked
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
FCC APIs (CORES, ServiceNow) are responding in 28-45 seconds.
The 30s frontend timeout was being hit ~50% of the time. Added
quick=1 to skip slow Playwright corp checks and increased timeout
to 60s as safety margin.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When both voice and broadband are selected, BDC status is red but
the CTA had no case for red — nothing was added to the order.
Now: red = "BDC Broadband + Voice Filing" at $349 (both filings),
yellow = single filing at $199 (unchanged).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The BDC toggle was in the static HTML file (site/public/tools/)
which is overridden by the Astro page. Fixed the ACTUAL page at
site/src/pages/tools/fcc-compliance-check.astro.
Two-step flow: asks about retail voice first, then broadband.
Results: voice+broadband=red (both filings), voice-only=yellow
(BDC Voice), broadband-only=yellow (BDC Broadband), neither=green.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
STIR/SHAKEN card creates confusion with OCN bundling on the order
page — if carried over, it doesn't auto-select OCN. Self-reported
RMD status also doesn't reliably indicate actual compliance.
Reverted to hidden (computed internally but not shown to users).
Removed STIR toggle and pending-question watcher for it.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
BDC Voice Subscription filing is required for all retail voice
providers (VoIP, CLEC, wireline) even without broadband service.
The compliance checker was asking about broadband first, which
caused voice-only carriers to overlook their BDC Voice obligation.
- Renamed card: "BDC Filing (Broadband + Voice Subscription Data)"
- Reversed question order: ask retail voice first, then broadband
- "No" for voice now says "No, or wholesale only" (wholesale exempt)
- Voice-only retail carriers correctly flagged for BDC Voice filing
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Containers without a memory limit have spec_memory_limit_bytes=0,
causing division to produce +Inf which always fires. Added guard:
only alert when a limit is actually set (> 0).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Dashboard queries now use max() to pick UP value when old stale
probe targets coexist with new ones. Prometheus admin API enabled
for future TSDB cleanup of stale series.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- ERPNext: custom blackbox module with Host: performancewest.net header
(ERPNext multitenancy requires site name in Host for routing)
- Forgejo: add extra_hosts to blackbox-exporter so it can resolve
host.docker.internal to reach forgejo on port 3000
- Blackbox http_erpnext module: sets Host header, expects 200
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Workers: use http_internal module (HTTP/1.0 SimpleHTTPServer)
- ERPNext: use /api/method/ping, accept 401/403 (still means alive)
- Listmonk: use /health not /api/health (403 without auth)
- Forgejo: port 3000 not 3030
- Dev API: probe via HTTPS public URL (blackbox can't reach Docker)
- Added http_internal blackbox module accepting HTTP/1.0 + 401/403
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Each service gets its own Prometheus probe verifying actual functionality:
- API: /status endpoint (checks DB connectivity, returns 503 if down)
- Workers: /health endpoint (job server responsive)
- ERPNext: API method call (MariaDB + Redis + app all working)
- MinIO: /minio/health/live (storage accessible)
- Listmonk: /api/health (email service + DB)
- Ollama: root endpoint (LLM inference available)
- Umami: /api/heartbeat (analytics tracking)
- Forgejo: root page (git server accessible)
- PostgreSQL: pg_up metric from postgres-exporter
- All HTTPS endpoints: SSL + reachability from outside
Service-specific alerts with context:
- API down = DB may be unreachable
- Workers down = compliance orders not processing
- ERPNext down = CRM inaccessible
- MinIO down = document storage unavailable
Custom Grafana dashboard: "Performance West — Services Overview"
- Service status grid (UP/DOWN with colors)
- Response time charts (internal + HTTPS)
- SSL certificate expiry gauges
- Container CPU/memory per service
- PostgreSQL connections, nginx req/s, active alerts
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
MinIO returns 403 when accessed via minio.performancewest.net because
it interprets the Host header as a bucket name. Switch blackbox probe
to internal http://minio:9000/minio/health/live which works correctly.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
host network mode prevented Prometheus from reaching the exporter.
Switched back to bridge with extra_hosts + explicit port mapping.
Added timeout flag to prevent hanging on stub_status fetch.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
nginx-exporter couldn't reach host nginx via host.docker.internal
(connection timeout). Switch to network_mode: host so it can access
127.0.0.1:8888 directly. Prometheus scrapes via host.docker.internal
with extra_hosts mapping.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- nginx stub_status moved to port 8888 (port 80 was being caught
by other server blocks and returning 301)
- nginx-exporter updated to scrape :8888
- Added alertmanager scrape job to Prometheus config (was missing,
so alertmanager dashboard had no data)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Community dashboards reference datasource uid=prometheus but the
auto-generated UID was random. Pin to uid=prometheus for compatibility.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Comprehensive security update automation:
1. Debian OS (unattended-upgrades) — tightened to security-only:
- Removed general Debian updates (prevents feature/breaking changes)
- Only Debian-Security origins auto-installed
- Email admin on every upgrade via ops@performancewest.net
- Auto-reboot at 4 AM if kernel update requires it
- needrestart auto-restarts services after library updates
2. Docker CE — major version guard:
- Patch updates within pinned major version auto-applied
- Major version jumps held + admin alerted for manual review
- docker-ce, docker-ce-cli, containerd.io all version-guarded
3. Container base images — daily at 3:30 AM:
- Pulls latest base images for all docker-compose services
- Compares image digests — only rebuilds if changed
- Restarts only affected services (not full stack)
- Alerts admin on rebuild failures requiring manual intervention
- Covers both prod and dev compose projects
4. k3s — weekly Sunday at 3:45 AM:
- Patch updates within current minor auto-applied
- Minor/major upgrades alert admin for manual review
- Verifies node Ready status after update
- Alerts on failures with investigation instructions
5. Admin notifications via SMTP:
- [INFO] for successful patches
- [WARNING] for available major upgrades needing review
- [CRITICAL] for failures requiring immediate intervention
- Falls back to syslog if SMTP unavailable
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Users could skip Q1b (customer type) and Q2 (voice delivery) and
hit Next — the wizard silently defaulted to retail. Now validates:
- Q1b must be answered (customer type selected)
- Q2 must be answered if voice is checked and not wholesale-only
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
STIRShakenStep intake:
- New "Terminate only" option for carriers that only receive pre-signed
calls and don't originate
- Contextual hints for each option explaining requirements
- Show/hide vendor and upstream fields based on selection
RMD letter generator:
- New terminate_only section explaining verification-only posture,
citing 47 CFR § 64.6301 (signing obligation on originating provider)
- Added to needs_exhibit_a list
RMD Exhibit A generator:
- New terminate_only STIR/SHAKEN paragraph with SBC verification language
- Fixed scope paragraph: wholesale/facilities carriers no longer get
"small provider without Class 4 switch" boilerplate
- Fixed OCN paragraph: wholesale carriers get neutral wording instead
of "no OCN required for small retail provider"
RMD filing handler:
- Maps stir_shaken_status to rmd_option for Exhibit A generation
- Passes entity metadata (ocn, wholesale, gateway, contact) to generator
- Maps terminate_only → partial_implementation for FCC RMD form radio
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Uncomment STIR/SHAKEN check in fcc-lookup.ts — shows self-reported
implementation status from RMD filing
- Add toggle: "Do you originate calls or only terminate?"
- Terminate only → green, signing cert not required, file RMD as
partial implementation
- Originate or both → red, must have own STI certificate as of
June 2025
- Toggle integrates with pending-question system (CTA waits for answer)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Wholesale providers that only receive/terminate pre-signed calls don't
need a STIR/SHAKEN signing certificate. Info box explains: originating
providers must sign with own cert (as of June 2025), but
terminating-only providers just verify signatures (software config)
and file RMD as "partial implementation."
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Q5 was shown immediately on voice/broadband checkbox, before Q1b
(customer type) was answered — always defaulting to retail variant.
Now Q5 only appears after Q1b is answered, and the correct variant
(retail vs wholesale) is set at that point.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Explain that non-interconnected VoIP means voice apps without phone
numbers (e.g. Microsoft Teams, Discord) to distinguish from
interconnected VoIP which connects to the PSTN.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Non-interconnected VoIP, satellite, paging, private radio, and other
specialized services have different registration requirements. Show
a yellow info box under Q1 directing these users to contact us for
a custom assessment instead of using the wizard.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Retail carriers: ask where end-user customers are located, explain
state PUC nexus (registration triggered by customer location, not
incorporation). Wholesale carriers: simplified question about carrier
customer regions, explains they generally don't need state PUC
registration since the retail carrier holds the state obligation.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Wholesale voice carriers run their own switching by definition — the
Q2 options (reseller, UCaaS, own switch) are retail delivery models.
When wholesale is selected in Q1b, Q2 is hidden and Q3 (infrastructure
needs: LCR, DIDs, interconnection, STIR/SHAKEN) is shown directly.
voiceDelivery is auto-set to own_switch for wholesale.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
State PUC registration is triggered by customer location, not
incorporation state. Telecom services use local infrastructure
(switches, numbers, towers) creating attributional nexus. Rewritten
Q5 explains this and provides guidance: ~27 states need full
certification, ~9 registration only, ~5 minimal. Shows PUC info
box for multi-state/nationwide selections.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When carrier selects "Yes" to international services on Q6, show a
Canada CRTC recommendation box explaining: direct Canadian DIDs,
lower termination rates vs US gateway routing, CRTC interconnection.
Links to /order/canada-crtc with pricing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When CALEA SSI card shows red/amber, ask if carrier serves end users
(retail) or only other carriers (wholesale). Wholesale-only carriers
are exempt as "interconnecting carriers" under CALEA — card turns
green. Retail/both keeps the red status. Toggle integrates with
existing pending-question system (CTA waits for all toggles answered).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- CALEA: wholesale-only carriers exempt as "interconnecting carriers" per
FCC rules. Only retail/both customer types trigger CALEA SSI requirement.
- New Q6 "Will you offer international services?" appears after Q5.
Triggers International Section 214 Authorization add-on ($1,499).
- Section 214 info box explains when it's required.
- Customer type question (Q1b) is now visually separate from service
type checkboxes to avoid confusion.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The retail/wholesale radios were visually mixed in with the voice/broadband
checkboxes, making it easy to misread "Wholesale" as a service type.
Moved to a distinct Q1b section "Who are your customers?" that only
appears after checking voice or broadband. Single selection covers
both services (retail / wholesale / both).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Voice and broadband can have independent customer models (e.g. wholesale
voice + retail broadband). Each service type now gets its own inline
retail/wholesale/both radio when checked. Derivation logic updated:
- Voice carriers always need RMD+CPNI regardless of mode
- BDC only required when broadband has retail end users
- CALEA still triggered by voice or facilities-based broadband
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
wizard.addonFee was being subtracted and then recalculated each call,
causing prices to accumulate/subtract randomly on checkbox toggle.
Simplified to just sum base + checked addons + formation fees.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Phase 3-5:
- API: POST /api/v1/fcc-carrier-registration (order creation with pricing)
- API: GET /api/v1/fcc-carrier-registration/:id (status)
- API: GET /api/v1/fcc-carrier-registration/state-fees (formation fees)
- Checkout: fcc_carrier_registration order type with Stripe line items
- Payment handler: dispatch worker + send confirmation email
- Pipeline handler: 8-step CRTC-style pipeline (formation → CORES → 499 →
DC Agent → State PUC → RMD/CPNI/CALEA/BDC → add-ons → final review)
- Job server dispatch map entry
- Service page CTA updated to link to order page
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Base package includes CORES/FRN, Form 499, DC Agent, RMD, CPNI, CALEA,
BDC — all shown as included with checkmarks. Wizard determines which
are relevant (grayed out if not needed for service type).
Only STIR/SHAKEN (+$499), OCN (+$2,650), State PUC (+$399/state),
and formation are itemized add-ons.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Phase 1-2 of the new registration product:
- Migration 075: fcc_carrier_registrations table with full pipeline status,
service wizard answers, entity choice, pricing, idempotency tracking
- Order page with 5-step wizard:
1. Service wizard (voice/broadband/wholesale + delivery method + infra needs)
2. Registration checklist (auto-determined + add-ons with dynamic pricing)
3. Entity choice (existing FRN search OR new formation with nexus guidance)
4. Contact & officer info
5. Review & payment with engagement clickwrap
Still needed: API endpoint, checkout integration, worker pipeline handler.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Updated across 61 static HTML files (nav links), bundles catalog,
service page title/description/heading, and llms.txt.
URL stays /services/telecom/ipes-isp (no redirect needed).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
