- ERPNext: custom blackbox module with Host: performancewest.net header
(ERPNext multitenancy requires site name in Host for routing)
- Forgejo: add extra_hosts to blackbox-exporter so it can resolve
host.docker.internal to reach forgejo on port 3000
- Blackbox http_erpnext module: sets Host header, expects 200
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Workers: use http_internal module (HTTP/1.0 SimpleHTTPServer)
- ERPNext: use /api/method/ping, accept 401/403 (still means alive)
- Listmonk: use /health not /api/health (403 without auth)
- Forgejo: port 3000 not 3030
- Dev API: probe via HTTPS public URL (blackbox can't reach Docker)
- Added http_internal blackbox module accepting HTTP/1.0 + 401/403
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Each service gets its own Prometheus probe verifying actual functionality:
- API: /status endpoint (checks DB connectivity, returns 503 if down)
- Workers: /health endpoint (job server responsive)
- ERPNext: API method call (MariaDB + Redis + app all working)
- MinIO: /minio/health/live (storage accessible)
- Listmonk: /api/health (email service + DB)
- Ollama: root endpoint (LLM inference available)
- Umami: /api/heartbeat (analytics tracking)
- Forgejo: root page (git server accessible)
- PostgreSQL: pg_up metric from postgres-exporter
- All HTTPS endpoints: SSL + reachability from outside
Service-specific alerts with context:
- API down = DB may be unreachable
- Workers down = compliance orders not processing
- ERPNext down = CRM inaccessible
- MinIO down = document storage unavailable
Custom Grafana dashboard: "Performance West — Services Overview"
- Service status grid (UP/DOWN with colors)
- Response time charts (internal + HTTPS)
- SSL certificate expiry gauges
- Container CPU/memory per service
- PostgreSQL connections, nginx req/s, active alerts
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
MinIO returns 403 when accessed via minio.performancewest.net because
it interprets the Host header as a bucket name. Switch blackbox probe
to internal http://minio:9000/minio/health/live which works correctly.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
host network mode prevented Prometheus from reaching the exporter.
Switched back to bridge with extra_hosts + explicit port mapping.
Added timeout flag to prevent hanging on stub_status fetch.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
nginx-exporter couldn't reach host nginx via host.docker.internal
(connection timeout). Switch to network_mode: host so it can access
127.0.0.1:8888 directly. Prometheus scrapes via host.docker.internal
with extra_hosts mapping.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- nginx stub_status moved to port 8888 (port 80 was being caught
by other server blocks and returning 301)
- nginx-exporter updated to scrape :8888
- Added alertmanager scrape job to Prometheus config (was missing,
so alertmanager dashboard had no data)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Community dashboards reference datasource uid=prometheus but the
auto-generated UID was random. Pin to uid=prometheus for compatibility.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Comprehensive security update automation:
1. Debian OS (unattended-upgrades) — tightened to security-only:
- Removed general Debian updates (prevents feature/breaking changes)
- Only Debian-Security origins auto-installed
- Email admin on every upgrade via ops@performancewest.net
- Auto-reboot at 4 AM if kernel update requires it
- needrestart auto-restarts services after library updates
2. Docker CE — major version guard:
- Patch updates within pinned major version auto-applied
- Major version jumps held + admin alerted for manual review
- docker-ce, docker-ce-cli, containerd.io all version-guarded
3. Container base images — daily at 3:30 AM:
- Pulls latest base images for all docker-compose services
- Compares image digests — only rebuilds if changed
- Restarts only affected services (not full stack)
- Alerts admin on rebuild failures requiring manual intervention
- Covers both prod and dev compose projects
4. k3s — weekly Sunday at 3:45 AM:
- Patch updates within current minor auto-applied
- Minor/major upgrades alert admin for manual review
- Verifies node Ready status after update
- Alerts on failures with investigation instructions
5. Admin notifications via SMTP:
- [INFO] for successful patches
- [WARNING] for available major upgrades needing review
- [CRITICAL] for failures requiring immediate intervention
- Falls back to syslog if SMTP unavailable
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Users could skip Q1b (customer type) and Q2 (voice delivery) and
hit Next — the wizard silently defaulted to retail. Now validates:
- Q1b must be answered (customer type selected)
- Q2 must be answered if voice is checked and not wholesale-only
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
STIRShakenStep intake:
- New "Terminate only" option for carriers that only receive pre-signed
calls and don't originate
- Contextual hints for each option explaining requirements
- Show/hide vendor and upstream fields based on selection
RMD letter generator:
- New terminate_only section explaining verification-only posture,
citing 47 CFR § 64.6301 (signing obligation on originating provider)
- Added to needs_exhibit_a list
RMD Exhibit A generator:
- New terminate_only STIR/SHAKEN paragraph with SBC verification language
- Fixed scope paragraph: wholesale/facilities carriers no longer get
"small provider without Class 4 switch" boilerplate
- Fixed OCN paragraph: wholesale carriers get neutral wording instead
of "no OCN required for small retail provider"
RMD filing handler:
- Maps stir_shaken_status to rmd_option for Exhibit A generation
- Passes entity metadata (ocn, wholesale, gateway, contact) to generator
- Maps terminate_only → partial_implementation for FCC RMD form radio
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Uncomment STIR/SHAKEN check in fcc-lookup.ts — shows self-reported
implementation status from RMD filing
- Add toggle: "Do you originate calls or only terminate?"
- Terminate only → green, signing cert not required, file RMD as
partial implementation
- Originate or both → red, must have own STI certificate as of
June 2025
- Toggle integrates with pending-question system (CTA waits for answer)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Wholesale providers that only receive/terminate pre-signed calls don't
need a STIR/SHAKEN signing certificate. Info box explains: originating
providers must sign with own cert (as of June 2025), but
terminating-only providers just verify signatures (software config)
and file RMD as "partial implementation."
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Q5 was shown immediately on voice/broadband checkbox, before Q1b
(customer type) was answered — always defaulting to retail variant.
Now Q5 only appears after Q1b is answered, and the correct variant
(retail vs wholesale) is set at that point.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Explain that non-interconnected VoIP means voice apps without phone
numbers (e.g. Microsoft Teams, Discord) to distinguish from
interconnected VoIP which connects to the PSTN.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Non-interconnected VoIP, satellite, paging, private radio, and other
specialized services have different registration requirements. Show
a yellow info box under Q1 directing these users to contact us for
a custom assessment instead of using the wizard.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Retail carriers: ask where end-user customers are located, explain
state PUC nexus (registration triggered by customer location, not
incorporation). Wholesale carriers: simplified question about carrier
customer regions, explains they generally don't need state PUC
registration since the retail carrier holds the state obligation.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Wholesale voice carriers run their own switching by definition — the
Q2 options (reseller, UCaaS, own switch) are retail delivery models.
When wholesale is selected in Q1b, Q2 is hidden and Q3 (infrastructure
needs: LCR, DIDs, interconnection, STIR/SHAKEN) is shown directly.
voiceDelivery is auto-set to own_switch for wholesale.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
State PUC registration is triggered by customer location, not
incorporation state. Telecom services use local infrastructure
(switches, numbers, towers) creating attributional nexus. Rewritten
Q5 explains this and provides guidance: ~27 states need full
certification, ~9 registration only, ~5 minimal. Shows PUC info
box for multi-state/nationwide selections.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When carrier selects "Yes" to international services on Q6, show a
Canada CRTC recommendation box explaining: direct Canadian DIDs,
lower termination rates vs US gateway routing, CRTC interconnection.
Links to /order/canada-crtc with pricing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When CALEA SSI card shows red/amber, ask if carrier serves end users
(retail) or only other carriers (wholesale). Wholesale-only carriers
are exempt as "interconnecting carriers" under CALEA — card turns
green. Retail/both keeps the red status. Toggle integrates with
existing pending-question system (CTA waits for all toggles answered).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- CALEA: wholesale-only carriers exempt as "interconnecting carriers" per
FCC rules. Only retail/both customer types trigger CALEA SSI requirement.
- New Q6 "Will you offer international services?" appears after Q5.
Triggers International Section 214 Authorization add-on ($1,499).
- Section 214 info box explains when it's required.
- Customer type question (Q1b) is now visually separate from service
type checkboxes to avoid confusion.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The retail/wholesale radios were visually mixed in with the voice/broadband
checkboxes, making it easy to misread "Wholesale" as a service type.
Moved to a distinct Q1b section "Who are your customers?" that only
appears after checking voice or broadband. Single selection covers
both services (retail / wholesale / both).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Voice and broadband can have independent customer models (e.g. wholesale
voice + retail broadband). Each service type now gets its own inline
retail/wholesale/both radio when checked. Derivation logic updated:
- Voice carriers always need RMD+CPNI regardless of mode
- BDC only required when broadband has retail end users
- CALEA still triggered by voice or facilities-based broadband
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
wizard.addonFee was being subtracted and then recalculated each call,
causing prices to accumulate/subtract randomly on checkbox toggle.
Simplified to just sum base + checked addons + formation fees.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Phase 3-5:
- API: POST /api/v1/fcc-carrier-registration (order creation with pricing)
- API: GET /api/v1/fcc-carrier-registration/:id (status)
- API: GET /api/v1/fcc-carrier-registration/state-fees (formation fees)
- Checkout: fcc_carrier_registration order type with Stripe line items
- Payment handler: dispatch worker + send confirmation email
- Pipeline handler: 8-step CRTC-style pipeline (formation → CORES → 499 →
DC Agent → State PUC → RMD/CPNI/CALEA/BDC → add-ons → final review)
- Job server dispatch map entry
- Service page CTA updated to link to order page
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Base package includes CORES/FRN, Form 499, DC Agent, RMD, CPNI, CALEA,
BDC — all shown as included with checkmarks. Wizard determines which
are relevant (grayed out if not needed for service type).
Only STIR/SHAKEN (+$499), OCN (+$2,650), State PUC (+$399/state),
and formation are itemized add-ons.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Phase 1-2 of the new registration product:
- Migration 075: fcc_carrier_registrations table with full pipeline status,
service wizard answers, entity choice, pricing, idempotency tracking
- Order page with 5-step wizard:
1. Service wizard (voice/broadband/wholesale + delivery method + infra needs)
2. Registration checklist (auto-determined + add-ons with dynamic pricing)
3. Entity choice (existing FRN search OR new formation with nexus guidance)
4. Contact & officer info
5. Review & payment with engagement clickwrap
Still needed: API endpoint, checkout integration, worker pipeline handler.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Updated across 61 static HTML files (nav links), bundles catalog,
service page title/description/heading, and llms.txt.
URL stays /services/telecom/ipes-isp (no redirect needed).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
set_value doesn't work for child tables like portal_users. Use
updateResource (PUT /api/resource/Customer) which handles it correctly.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When the user revisits a completed intake (intake_data_validated=true),
shows a success screen with Go to Portal and Revise buttons instead of
the blank form. Revise adds ?revise=1 to bypass the check.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
batch_id was missing from the SELECT, so order_data.batch_id was always
None. This meant the batch email skip in _request_entity_intake never
triggered, causing duplicate intake emails for every batch order.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Yellow banner on each step: "We've pre-filled this from your FCC records.
Please review carefully and correct any information that is inaccurate."
- Only shows when accessed via token/FRN (post-payment intake)
- CORES address: filter by company name suffix (LLC/Inc/Corp) instead of
requiring a number — addresses like "PO Box 123" now work
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
CategoryStep script runs on ALL pages using the Wizard component.
It tried to find #pw-wizard (its inner quiz div) and called
querySelectorAll on it — null on CPNI/RMD/etc pages, crashing the
entire script bundle. This prevented FRN auto-fill, officer
suggestions, and all other intake functionality.
Guard: if #pw-wizard doesn't exist, skip all CategoryStep logic.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Entity cache has no RA/officer data yet. Instead, fetch the FCC lookup
(quick mode) and offer RMD contact name + address and CORES principal
address as clickable suggestions to auto-fill Officer 1.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Removed "How many officers" dropdown — all 3 always visible
- Officers 2 and 3 marked as (optional) in legend
- Only Officer 1 validated (name, title, street, city required)
- Blank optional officers skipped in saved data
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
