new-site/docs/architecture.md

System Architecture

Last updated: 2026-04-17 (15 Docker containers + k3s SHKeeper pods + Windows DocServer VM + dev stack + crypto treasury + foreign qualification + compliance check tool)

Overview

Performance West runs as a multi-service stack on a Debian 13 VM (207.174.124.71):

• Astro static site — marketing pages, service descriptions, CRTC order form, crypto payment page
• Express API — order creation, Stripe/PayPal/SHKeeper checkout, identity verification, webhook receiver, portal setup API
• ERPNext — CRM, orders, invoicing, ticketing, customer portal (v15 + MariaDB, custom image with 4 Frappe apps). Portal at portal.performancewest.net
• Listmonk — email marketing at lists.performancewest.net (Go binary, PostgreSQL-backed). 22 campaigns across 4 lists. Bounce processing via POP3 from Carbonio.
• MinIO — S3-compatible document storage (formation docs, compliance reports, binder PDFs)
• Ollama — local LLM for document generation and email drafting (qwen2.5:7b)
• Workers — Python automation: Anytime Mailbox signup (Playwright + IMAP OTP), BC incorporation, Flowroute DID provisioning, document generation, AMB location scraping, payment reminders, GCKey provisioning (Playwright), compliance calendar renewal lifecycle, client email processing
• SHKeeper — self-hosted crypto payment processor via k3s (BTC, LTC, DOGE, ETH, BNB, MATIC, TRX). TRX uses nginx proxy at :5555 → TronGrid with API key injection. ETH uses proxy at :5556 → publicnode.
• Crypto Treasury — post-payment pipeline: SHKeeper → Bridge (Stripe) offramp → RelayFi bank → Relay debit card → vendor payments. Manual mode until Bridge approval.
• Umami — self-hosted web analytics at analytics.performancewest.net
• PostgreSQL — API state, orders, identity verifications, AMB locations, sessions, jurisdictions, foreign qualifications, crypto ledger (api-postgres; also hosts Listmonk DB). 67 migrations applied.
• Dev stack — separate environment at dev.performancewest.net / api.dev.performancewest.net (port 4323/3002, own PG on 5433). Systemd unit: performancewest-dev.service.

Payment Gateway Architecture

Active gateways: Stripe (card/ACH/Klarna), PayPal Direct (Orders API v2), SHKeeper (crypto).

Gateway	Status	Methods	Surcharge
Stripe	Active	Card (+3%), ACH (0%), Klarna (+6%)	Via Stripe Checkout Sessions
PayPal	Active	PayPal Direct (+3%)	Orders API v2 with capture/tracking/refund
SHKeeper	Active	Crypto (0%) — BTC/LTC/DOGE + EVM chains	Branded /order/crypto-pay page with coin selector
Adyen	Future	Card/ACH/Klarna/CashApp/AmazonPay (frappe_adyen installed, not configured)

Payment card routing for vendor expenses:

Customer paid with	Filing card	ERPNext SID	Funds timing
Card/ACH/Klarna	Stripe Issuing virtual card	SID-0002	T+2 card, T+4 ACH
PayPal	PayPal Mastercard	SID-0001	Instant
Crypto	crypto-filing-card	crypto-filing-card	Manual

Fund availability detection: Stripe balance.available webhook checks settlement timing (T+2 card/Klarna, T+4 ACH). PayPal is instant. Crypto is manual. When funds clear: Stripe Issuing topup → advance to "Client Selection" → email client portal setup link.

ERPNext custom Frappe apps (baked into performancewest-erpnext:latest):

App	Purpose
frappe_crypto	SHKeeper crypto gateway
frappe_adyen	Adyen gateway (future)
frappe_ca_registry	BC Corporate Online incorporation automation
performancewest_erpnext	Surcharge hooks, identity gate, custom DocTypes

Service Architecture

┌─────────────────────────────────────────────────────────────────┐
│  Proxmox Host                                                   │
│                                                                 │
│  ┌── Linux VM (Debian 13) — 207.174.124.71 ───────────────┐   │
│  │                                                          │   │
│  │  Docker Compose (15 containers)                          │   │
│  │    ├── site (Astro → nginx:alpine)           :4322       │   │
│  │    ├── api (Express/TypeScript)              :3001       │   │
│  │    ├── api-postgres (app data)               :5432       │   │
│  │    ├── erpnext (custom image + 4 apps)       :8080       │   │
│  │    ├── erpnext-worker-default                            │   │
│  │    ├── erpnext-worker-short                              │   │
│  │    ├── erpnext-scheduler                                 │   │
│  │    ├── erpnext-mariadb                       :3306       │   │
│  │    ├── erpnext-redis                         :6379       │   │
│  │    ├── listmonk (email marketing)            :9100       │   │
│  │    ├── minio (document storage)              :9000/:9001 │   │
│  │    ├── workers (Python automation)           :8090       │   │
│  │    ├── ollama (local LLM)                   :11434       │   │
│  │    ├── umami (analytics)                     :3100       │   │
│  │    └── umami-postgres                                    │   │
│  │                                                          │   │
│  │  Dev Stack (docker compose at /opt/performancewest-dev)   │   │
│  │    ├── dev-site (Astro → nginx:alpine)       :4323       │   │
│  │    ├── dev-api (Express/TypeScript)           :3002       │   │
│  │    ├── dev-api-postgres                       :5433       │   │
│  │    └── dev-workers (Python automation)                    │   │
│  │                                                          │   │
│  │  k3s / Kubernetes (SHKeeper crypto payments)             │   │
│  │    ├── shkeeper-deployment (Flask API)  NodePort :30723   │   │
│  │    ├── bitcoin-shkeeper (3 containers)                   │   │
│  │    ├── ethereum-shkeeper (3 containers) via :5556 proxy  │   │
│  │    ├── polygon-shkeeper (3 containers)                   │   │
│  │    ├── bnb-shkeeper (3 containers)                       │   │
│  │    ├── tron-shkeeper (3 containers)    via :5555 proxy   │   │
│  │    ├── litecoin-shkeeper (3 containers)                  │   │
│  │    ├── dogecoin-shkeeper (3 containers)                  │   │
│  │    └── mariadb (SHKeeper database)                       │   │
│  │                                                          │   │
│  │  nginx RPC Proxies (inject API keys, strip basic auth)   │   │
│  │    ├── :5555 → api.trongrid.io + TRON-PRO-API-KEY        │   │
│  │    └── :5556 → ethereum-rpc.publicnode.com               │   │
│  └──────────────────────────────────────────────────────────┘   │
│                                                                 │
│  ┌── Windows VM (DocServer) — 108.181.102.34 ─────────────┐   │
│  │  SSH: port 22422 (key auth)                              │   │
│  │  Office 365 Word (COM automation)                        │   │
│  │  Python 3.13 + pywin32 + minio SDK                       │   │
│  │  docserver_worker.py (MinIO poller, 12s interval)        │   │
│  │  Task Scheduler: PW-DocserverWorker (AtLogOn)             │   │
│  │  Auto-logon configured (requires RDP after cold reboot)  │   │
│  │  Private network: 10.4.20.247 → MinIO via nginx          │   │
│  └──────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────┘

DNS

Domain	Target
performancewest.net	site (:4322)
api.performancewest.net	api (:3001)
crm.performancewest.net	ERPNext (:8080)
lists.performancewest.net	Listmonk (:9100) — email marketing
portal.performancewest.net	ERPNext (:8080) — client portal
analytics.performancewest.net	Umami (:3100)
pay.performancewest.net	SHKeeper API (:5000 via k3s NodePort :30723)
crypto.performancewest.net	SHKeeper admin UI (:30723 via nginx)
dev.performancewest.net	Dev site (:4323)
api.dev.performancewest.net	Dev API (:3002)
minio.performancewest.net	MinIO S3 API (:9000)
minio-console.performancewest.net	MinIO Console (:9001)

All A records point to 207.174.124.71. TLS via Let's Encrypt (certbot, 8 certs). Mail records unchanged (HestiaCP at 207.174.124.15).

External panels:

Domain	Service
cp.carrierone.com	HestiaCP — DNS, .ca domain/email provisioning

Data Flow

Browser
  │
  ├── performancewest.net ──────> nginx ──> site (static pages)
  │
  ├── api.performancewest.net ──> nginx ──> Express API
  │                                          ├── api-postgres (state fees, API keys)
  │                                          ├── ERPNext REST API (CRM, orders, tickets)
  │                                          ├── Listmonk API (email marketing)
  │                                          └── Workers HTTP API (job dispatch)
  │
  ├── crm.performancewest.net ──> nginx ──> ERPNext
  │                                          ├── erpnext-mariadb
  │                                          └── erpnext-redis
  │
  ├── pay.performancewest.net ──> nginx ──> SHKeeper (k3s :5000)
  │
  ├── lists.performancewest.net ──> nginx ──> Listmonk (:9100)
  │
  ├── portal.performancewest.net ──> nginx ──> ERPNext (:8080)
  │
  ├── minio.performancewest.net ──> nginx ──> MinIO S3 (:9000)
  └── minio-console.performancewest.net ──> nginx ──> MinIO Console (:9001)

Order Flow (ERPNext BPM)

Customer places order → Express API → ERPNext Sales Invoice + Payment Request
  → ERPNext payment gateway (Adyen or SHKeeper) → Customer pays
  → ERPNext webhook → Express API → Workers job server
  → Workers execute: name search → filing → EIN → doc gen
  → Documents uploaded to MinIO
  → ERPNext status → Review → Admin approves → Delivered

Key Integration Points

From	To	Method	Purpose
Website	Express API	HTTPS	Forms, orders, name search
Express API	ERPNext	REST API	CRM, orders, invoicing, tickets
Express API	Workers	HTTP :8090	Job dispatch (via webhooks)
Express API	Listmonk	REST API	Subscriber onboarding, email campaigns
ERPNext	Express API	Webhooks	Workflow state change notifications
ERPNext	Adyen	Sessions API v71	Card/ACH/Klarna/CashApp/AmazonPay (via frappe_adyen)
ERPNext	SHKeeper	REST API :5000	Crypto payment requests (via frappe_crypto)
Workers	ERPNext	REST API	Status updates, workflow advances
Workers	MinIO	S3 API	Document upload/download
Workers	Ollama	HTTP :11434	LLM doc generation, email drafting
Workers	DocServer	MinIO transport	DOCX-to-PDF conversion (MinIO poller: to-convert/ → converted/)
Workers	State portals	Playwright	Name search, entity filing
Workers	IRS	Playwright	EIN obtainment
ERPNext	Email	SMTP	Invoices, notifications, ticket replies
Relay debit card	State portals	Playwright	Filing fee payment
Workers	HestiaCP	SSH	.ca domain/email provisioning (cp.carrierone.com)
Workers	Flowroute	REST API	Canadian DID provisioning
Workers	Porkbun	REST API	.ca domain registration
Workers	Client mailboxes	IMAP	Monitor/process client .ca email
Website	Anytime Mailbox	External	Vancouver mailbox for Canadian carrier clients
ERPNext	Venn.ca	Referral	Canadian business banking referral
AI Agents	MCP Server	stdio	Service lookup, pricing, tools (npm package)

Worker Processes (Python)

The workers container runs a job server (job_server.py on port 8090) that dispatches jobs to specialized worker modules. Jobs are triggered by ERPNext webhooks via the Express API.

Worker	File	Trigger	Purpose
Job Server	job_server.py	HTTP POST :8090	Central dispatcher — 22 job handlers
CRTC Pipeline	services/canada_crtc.py	Webhook	14-step CRTC carrier registration pipeline
GCKey Provisioner	gckey_provisioner.py	Called by CRTC Step 11	Playwright-based GCKey account creation (5-step wizard, hCaptcha)
Renewal Worker	renewal_worker.py	Daily cron (7 AM)	Compliance calendar lifecycle: upcoming → due soon → invoice → paid → completed → re-calendar
Formation Worker	formation_worker.py	Webhook	US state LLC/Corp filing via Playwright
Binder Compiler	binder_compiler.py	Called by pipeline	PDF merge: certificate + articles + CRTC letter + OA
Document Gen	document_gen/	Called by pipeline	DOCX template fill → MinIO → DocServer PDF conversion
HestiaCP Provisioner	hestia_provisioner.py	Called by pipeline	.ca domain + 14 mailboxes via SSH to cp.carrierone.com
Client Email Processor	client_email_processor.py	Cron (15 min)	IMAP monitor for client .ca mailboxes
AMB Scraper	amb_location_scraper.py	Daily cron	Anytime Mailbox location pricing + sold-out detection
Payment Reminder	payment_reminder.py	Cron (5 min)	Abandoned cart recovery (15min/1d/2d intervals)
Commission Worker	commission_worker.py	Cron (daily 02:00)	Agent commission eligibility check (14-day holdback)
Crypto Payment Worker	crypto_payment_worker.py	Cron (60s)	Treasury state machine: received → sizing → offramping → funds_at_relay → settled
Cold Wallet Sweeper	cold_wallet_sweeper.py	Cron (30 min)	Sweep SHKeeper hot wallet excess to cold storage
Relay Deposit Monitor	relay_deposit_monitor.py	Cron (5 min)	IMAP parser for Relay bank deposit alerts
USF Factor Monitor	usf_factor_monitor.py	Cron (daily 09:00 CT)	Scrape USAC quarterly USF factor, email all FCC carriers
De Minimis Check	deminimis_factor_check.py	Cron (daily 03:00)	Alert if fcc_deminimis_factors missing for current/next year
CDR Retention Sweeper	cdr_retention_sweeper.py	Cron (daily 05:00)	Purge CDR data past retention window
CDR Unlock Nudge	cdr_unlock_nudge.py	Cron (daily 10:00 CT)	Email customers with locked CDR studies behind paywall
FCC RMD Scraper	fcc_rmd_removed_scraper.py	Cron (weekly Wed)	Track FCC RMD carrier removals
Foreign Qual Handler	services/foreign_qualification.py	Webhook	Multi-state Certificate of Authority filings
New Carrier Bundle	services/new_carrier_bundle.py	Webhook	6-handler chain: CORES → DC Agent → 499 Init → RMD → CPNI → CALEA

CRTC Pipeline Steps (14 total)

Step	Name	Key Actions
1	Order Received	Validate, create ERPNext Sales Order
2	Payment Confirmed	Create Sales Invoice + Payment Entry
3	Client Selection	Client picks mailbox unit, DID, domain
4	BC Incorporation	Playwright → BC Corporate Online
5	Domain Registration	Porkbun .ca + HestiaCP provisioning
6a	CRTC Letter Generation	DOCX template → MinIO → DocServer PDF
6b	eSign	Email JWT link → client signs → resume pipeline
7	CRTC Submission	Mail signed letter to Secretary General
8	Anytime Mailbox	Playwright signup + IMAP OTP
9	Binder Compilation	PDF merge of all documents
10	Delivery	Email binder + admin print/ship email
11	BITS Registration	GCKey provisioning + admin ToDo for BITS filing
12	CCTS Membership	Admin ToDo + client obligations email
13	Compliance Calendar	Create 17 compliance entries (regulatory + tax + ATS)
14	Ready for Review	Final admin review before marking Delivered

Compliance Calendar Renewal Lifecycle

The renewal_worker.py runs daily at 7 AM and manages the full lifecycle of recurring compliance obligations for all carriers:

Upcoming → Due Soon (30 days out) → Invoice Sent → Paid → Completed → auto-re-calendar next year

• 17 entries per carrier: BC annual report, CRTC annual maintenance, mailbox renewal, domain renewal, DID renewal, CCTS renewal, T2 tax return, corporate tax payment, GST/HST return, T4/T4A slips, BC PST, WorkSafeBC, CRTC registration update, plus ATS survey forms (REP-T/T1 mandatory for all carriers)
• Billable items generate ERPNext Sales Invoice; entry cannot be completed until paid
• On payment: entry marked Completed, admin ToDo created, next-year entry auto-created
• Webhook-triggered: handle_renewal_payment in job_server for immediate processing

DocServer (Windows VM)

MinIO-based transport — no direct HTTP connection between Linux and Windows VMs.

Workers: upload DOCX to minio://performancewest/to-convert/{uuid}.docx
  → DocServer polls to-convert/ bucket every 12 seconds
  → Word COM converts DOCX → PDF
  → DocServer uploads PDF to minio://performancewest/converted/{uuid}.pdf
  → Workers poll converted/ bucket for result

• Heartbeat: DocServer writes docserver-heartbeat.json to MinIO every 60 seconds
• Fallback: If heartbeat is stale (>5 min), workers auto-switch to LibreOffice headless

Boot Sequence

All services auto-start on reboot via systemd:

Unit	What it starts	Depends on
docker.service	Docker daemon	network
k3s.service	k3s + all SHKeeper pods	docker
nginx.service	Reverse proxy + TLS + RPC proxies (:5555/:5556)	network
performancewest.service	Prod docker compose (15 containers)	docker
performancewest-dev.service	Dev docker compose (4 containers)	docker, performancewest

All containers use restart: unless-stopped. k3s manages pod restarts via deployment specs. The nginx RPC proxy configs for TronGrid (:5555) and ETH (:5556) are in /etc/nginx/conf.d/ and load automatically. UFW rules for ports 5555/5556 are persistent (allow from 10.42.0.0/16 only).

Scheduled Workers (systemd timers)

Deployed via infra/ansible/roles/worker-crons/. Each timer runs docker compose exec -T workers python -m <module>.

Timer	Cadence	Module
pw-usf-factor-monitor	daily 09:00 CT	scripts.workers.usf_factor_monitor
pw-deminimis-factor-check	daily 03:00 UTC	scripts.workers.deminimis_factor_check
pw-cold-wallet-sweep	every 30 min	scripts.workers.cold_wallet_sweeper
pw-crypto-payment-worker	every 60s	scripts.workers.crypto_payment_worker
pw-relay-deposit-monitor	every 5 min	scripts.workers.relay_deposit_monitor
pw-commission-worker	daily 02:00 UTC	scripts.workers.commission_worker
pw-renewal-worker	daily 04:00 UTC	scripts.workers.renewal_worker
pw-cdr-retention	daily 05:00 UTC	scripts.workers.cdr_retention_sweeper
pw-cdr-unlock-nudge	daily 10:00 CT	scripts.workers.cdr_unlock_nudge
pw-payment-reminder	daily 11:00 CT	scripts.workers.payment_reminder
pw-fcc-rmd-removed	weekly Wed 08:00 CT	scripts.workers.fcc_rmd_removed_scraper
pw-client-email-processor	every 15 min	scripts.workers.client_email_processor
pw-amb-location-scraper	daily 06:00 UTC	scripts.workers.amb_location_scraper

Database Schema (67 migrations)

Key tables added in this cycle:

Table	Migration	Purpose
jurisdictions	066	Unified US states + CA provinces (55 rows)
foreign_qualification_registrations	066	Per-state COA filings
state_compliance_obligations	067	Annual report fees/dates for all 51 US jurisdictions
crypto_payment_ledger	062	Immutable append-only money movement ledger
crypto_payment_jobs	065	Treasury state machine (received → settled)
vendor_obligations	063	Sizer: filing fees + commission reserves
cold_wallet_config / cold_wallet_sweeps	064	Cold wallet management

FCC Compliance Check Tool

Public at /tools/fcc-compliance-check. Queries FRN against CORES, RMD, 499 Filer DB, and CPNI records. Displays:

• CORES registration status (red-light check)
• RMD filing + certification date
• STIR/SHAKEN implementation (self-reported from RMD, hidden until STI-PA API access)
• CPNI annual certification (past-due detection with correct deadline logic)
• Form 499-A annual filing (CY-year-aware)
• Form 499-Q quarterly (de minimis trade-off explanation)
• BDC broadband/voice interactive toggle (two-step questionnaire)

Remediation CTA links to /order/fcc-compliance?services=cpni,499a,... for bundled ordering with 15% discount.

Compliance Service Catalog (34 handlers)

Slug	Handler	Price
new-carrier-bundle	NewCarrierBundleHandler (6 sub-handlers)	$1,799
fcc-full-compliance	FullComplianceHandler	$1,499
fcc-499a	Form499AHandler	$499
fcc-499a-499q	Form499ABundleHandler	$599
cpni-certification	CPNIFilingHandler (9 category variants)	$149
rmd-filing	RMDFilingHandler	$219
calea-ssi	CALEASSIHandler (6 category variants)	$299
bdc-filing / bdc-broadband / bdc-voice	BDCFilingHandler	$299/$199/$149
foreign-qualification-single	ForeignQualificationHandler	$149 + state fees
foreign-qualification-multi	ForeignQualificationHandler	$99/state + fees
dc-agent	DCAgentHandler (NWRA wholesale)	$99/yr
cores-frn-registration	CORESFRNRegistrationHandler	$99
fcc-499-initial	Form499InitialHandler	$299
ocn-registration	OCNRegistrationHandler	$799

State Formation Adapters

10 states with real adapters (name search via Playwright + portal config): WY (237 lines), CO (177), DE (119), FL (118), TX (SOSDirect), NV (SilverFlume), UT (DCC), NM, OH, MT. ~40 remaining states have stub adapters that create admin todos for manual filing.

JurisdictionConfig abstraction at scripts/formation/jurisdictions/ reads from the jurisdictions DB table (migration 066) and provides per-state fees, portal URLs, NWRA addresses, and entity type catalogs.

Deployment

Production: ansible-playbook infra/ansible/playbooks/deploy.yml -i infra/ansible/inventory/hosts.yml

Dev: bash scripts/deploy-dev.sh — rsyncs source files + rebuilds Docker containers. Static pages (tools, services, homepage) are in site/public/ and survive Astro rebuilds.

nginx cache policy: _astro/ = immutable (1 year). HTML = no-cache. Images = 30 days.

• Atomic uploads: .tmp_ prefix + copy_object rename prevents partial reads