new-site/docs/architecture.md

# System Architecture

**Last updated:** 2026-05-07 (15 Docker containers + k3s SHKeeper pods + Windows DocServer + Postfix/OpenDKIM + bounce watcher + dev stack)

See also:
- [Business Flow Diagram](business-flow.svg)
- [Technical Architecture Diagram](technical-architecture.svg)
- [Order Processing Flowchart](order-flow.svg)

## Overview

Performance West runs as a multi-service stack on a Debian 13 VM (207.174.124.71):

- **Astro static site** — marketing pages, service descriptions, CRTC order form, crypto payment page
- **Express API** — order creation, Stripe/PayPal/SHKeeper checkout, identity verification, webhook receiver, portal setup API
- **ERPNext** — CRM, orders, invoicing, ticketing, customer portal (v15 + MariaDB, custom image with 4 Frappe apps). Portal at `portal.performancewest.net`
- **Listmonk** — email marketing at `lists.performancewest.net` (Go binary, PostgreSQL-backed). 22 campaigns across 4 lists. Bounce processing via POP3 from Carbonio.
- **MinIO** — S3-compatible document storage (formation docs, compliance reports, binder PDFs)
- **Ollama** — local LLM for document generation and email drafting (qwen2.5:7b)
- **Workers** — Python automation: Anytime Mailbox signup (Playwright + IMAP OTP), BC incorporation, Flowroute DID provisioning, document generation, AMB location scraping, payment reminders, GCKey provisioning (Playwright), compliance calendar renewal lifecycle, client email processing
- **SHKeeper** — self-hosted crypto payment processor via k3s (BTC, LTC, DOGE, ETH, BNB, MATIC, TRX). TRX uses nginx proxy at :5555 → TronGrid with API key injection. ETH uses proxy at :5556 → publicnode.
- **Crypto Treasury** — post-payment pipeline: SHKeeper → Bridge (Stripe) offramp → RelayFi bank → Relay debit card → vendor payments. Manual mode until Bridge approval.
- **Umami** — self-hosted web analytics at `analytics.performancewest.net`
- **PostgreSQL** — API state, orders, identity verifications, AMB locations, sessions, jurisdictions, foreign qualifications, crypto ledger (api-postgres; also hosts Listmonk DB). 67 migrations applied.
- **Dev stack** — separate environment at `dev.performancewest.net` / `api.dev.performancewest.net` (port 4323/3002, own PG on 5433). Systemd unit: `performancewest-dev.service`.

## Payment Gateway Architecture

**Active gateways:** Stripe (card/ACH/Klarna), PayPal Direct (Orders API v2), SHKeeper (crypto).

| Gateway | Status | Methods | Surcharge |
|---------|--------|---------|-----------|
| Stripe | **Active** | Card (+3%), ACH (0%), Klarna (+6%) | Via Stripe Checkout Sessions |
| PayPal | **Active** | PayPal Direct (+3%) | Orders API v2 with capture/tracking/refund |
| SHKeeper | **Active** | Crypto (0%) — BTC/LTC/DOGE + EVM chains | Branded `/order/crypto-pay` page with coin selector |
| Adyen | Future | Card/ACH/Klarna/CashApp/AmazonPay (frappe_adyen installed, not configured) |

**Payment card routing for vendor expenses:**

| Customer paid with | Filing card | ERPNext SID | Funds timing |
|---|---|---|---|
| Card/ACH/Klarna | Stripe Issuing virtual card | SID-0002 | T+2 card, T+4 ACH |
| PayPal | PayPal Mastercard | SID-0001 | Instant |
| Crypto | crypto-filing-card | crypto-filing-card | Manual |

**Fund availability detection:** Stripe `balance.available` webhook checks settlement timing (T+2 card/Klarna, T+4 ACH). PayPal is instant. Crypto is manual. When funds clear: Stripe Issuing topup → advance to "Client Selection" → email client portal setup link.

ERPNext custom Frappe apps (baked into `performancewest-erpnext:latest`):

| App | Purpose |
|-----|---------|
| `frappe_crypto` | SHKeeper crypto gateway |
| `frappe_adyen` | Adyen gateway (future) |
| `frappe_ca_registry` | BC Corporate Online incorporation automation |
| `performancewest_erpnext` | Surcharge hooks, identity gate, custom DocTypes |

## Service Architecture

```
┌─────────────────────────────────────────────────────────────────┐
│  Proxmox Host                                                   │
│                                                                 │
│  ┌── Linux VM (Debian 13) — 207.174.124.71 ───────────────┐   │
│  │                                                          │   │
│  │  Docker Compose (15 containers)                          │   │
│  │    ├── site (Astro → nginx:alpine)           :4322       │   │
│  │    ├── api (Express/TypeScript)              :3001       │   │
│  │    ├── api-postgres (app data)               :5432       │   │
│  │    ├── erpnext (custom image + 4 apps)       :8080       │   │
│  │    ├── erpnext-worker-default                            │   │
│  │    ├── erpnext-worker-short                              │   │
│  │    ├── erpnext-scheduler                                 │   │
│  │    ├── erpnext-mariadb                       :3306       │   │
│  │    ├── erpnext-redis                         :6379       │   │
│  │    ├── listmonk (email marketing)            :9100       │   │
│  │    ├── minio (document storage)              :9000/:9001 │   │
│  │    ├── workers (Python automation)           :8090       │   │
│  │    ├── ollama (local LLM)                   :11434       │   │
│  │    ├── umami (analytics)                     :3100       │   │
│  │    └── umami-postgres                                    │   │
│  │                                                          │   │
│  │  Dev Stack (docker compose at /opt/performancewest-dev)   │   │
│  │    ├── dev-site (Astro → nginx:alpine)       :4323       │   │
│  │    ├── dev-api (Express/TypeScript)           :3002       │   │
│  │    ├── dev-api-postgres                       :5433       │   │
│  │    └── dev-workers (Python automation)                    │   │
│  │                                                          │   │
│  │  k3s / Kubernetes (SHKeeper crypto payments)             │   │
│  │    ├── shkeeper-deployment (Flask API)  NodePort :30723   │   │
│  │    ├── bitcoin-shkeeper (3 containers)                   │   │
│  │    ├── ethereum-shkeeper (3 containers) via :5556 proxy  │   │
│  │    ├── polygon-shkeeper (3 containers)                   │   │
│  │    ├── bnb-shkeeper (3 containers)                       │   │
│  │    ├── tron-shkeeper (3 containers)    via :5555 proxy   │   │
│  │    ├── litecoin-shkeeper (3 containers)                  │   │
│  │    ├── dogecoin-shkeeper (3 containers)                  │   │
│  │    └── mariadb (SHKeeper database)                       │   │
│  │                                                          │   │
│  │  nginx RPC Proxies (inject API keys, strip basic auth)   │   │
│  │    ├── :5555 → api.trongrid.io + TRON-PRO-API-KEY        │   │
│  │    └── :5556 → ethereum-rpc.publicnode.com               │   │
│  └──────────────────────────────────────────────────────────┘   │
│                                                                 │
│  ┌── Windows VM (DocServer) — 108.181.102.34 ─────────────┐   │
│  │  SSH: port 22422 (key auth)                              │   │
│  │  Office 365 Word (COM automation)                        │   │
│  │  Python 3.13 + pywin32 + minio SDK                       │   │
│  │  docserver_worker.py (MinIO poller, 12s interval)        │   │
│  │  Task Scheduler: PW-DocserverWorker (AtStartup + 5-min)   │   │
│  │  Self-healing: restart-on-fail + MinIO-retry (no RDP)     │   │
│  │  Private network: 10.4.20.247 → MinIO via nginx          │   │
│  └──────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────┘
```

## DNS

| Domain | Target |
|--------|--------|
| `performancewest.net` | site (:4322) |
| `api.performancewest.net` | api (:3001) |
| `crm.performancewest.net` | ERPNext (:8080) |
| `lists.performancewest.net` | Listmonk (:9100) — email marketing |
| `portal.performancewest.net` | ERPNext (:8080) — client portal |
| `analytics.performancewest.net` | Umami (:3100) |
| `pay.performancewest.net` | SHKeeper API (:5000 via k3s NodePort :30723) |
| `crypto.performancewest.net` | SHKeeper admin UI (:30723 via nginx) |
| `dev.performancewest.net` | Dev site (:4323) |
| `api.dev.performancewest.net` | Dev API (:3002) |
| `minio.performancewest.net` | MinIO S3 API (:9000) |
| `minio-console.performancewest.net` | MinIO Console (:9001) |

All A records point to `207.174.124.71`. TLS via Let's Encrypt (certbot, 8 certs). Mail records unchanged (HestiaCP at `207.174.124.15`).

**External panels:**

| Domain | Service |
|--------|---------|
| `cp.carrierone.com` | HestiaCP — DNS, .ca domain/email provisioning |

## Data Flow

```
Browser
  │
  ├── performancewest.net ──────> nginx ──> site (static pages)
  │
  ├── api.performancewest.net ──> nginx ──> Express API
  │                                          ├── api-postgres (state fees, API keys)
  │                                          ├── ERPNext REST API (CRM, orders, tickets)
  │                                          ├── Listmonk API (email marketing)
  │                                          └── Workers HTTP API (job dispatch)
  │
  ├── crm.performancewest.net ──> nginx ──> ERPNext
  │                                          ├── erpnext-mariadb
  │                                          └── erpnext-redis
  │
  ├── pay.performancewest.net ──> nginx ──> SHKeeper (k3s :5000)
  │
  ├── lists.performancewest.net ──> nginx ──> Listmonk (:9100)
  │
  ├── portal.performancewest.net ──> nginx ──> ERPNext (:8080)
  │
  ├── minio.performancewest.net ──> nginx ──> MinIO S3 (:9000)
  └── minio-console.performancewest.net ──> nginx ──> MinIO Console (:9001)
```

## Order Flow (ERPNext BPM)

```
Customer places order → Express API → ERPNext Sales Invoice + Payment Request
  → ERPNext payment gateway (Adyen or SHKeeper) → Customer pays
  → ERPNext webhook → Express API → Workers job server
  → Workers execute: name search → filing → EIN → doc gen
  → Documents uploaded to MinIO
  → ERPNext status → Review → Admin approves → Delivered
```

## Key Integration Points

| From | To | Method | Purpose |
|------|----|--------|---------|
| Website | Express API | HTTPS | Forms, orders, name search |
| Express API | ERPNext | REST API | CRM, orders, invoicing, tickets |
| Express API | Workers | HTTP :8090 | Job dispatch (via webhooks) |
| Express API | Listmonk | REST API | Subscriber onboarding, email campaigns |
| ERPNext | Express API | Webhooks | Workflow state change notifications |
| ERPNext | Adyen | Sessions API v71 | Card/ACH/Klarna/CashApp/AmazonPay (via frappe_adyen) |
| ERPNext | SHKeeper | REST API :5000 | Crypto payment requests (via frappe_crypto) |
| Workers | ERPNext | REST API | Status updates, workflow advances |
| Workers | MinIO | S3 API | Document upload/download |
| Workers | Ollama | HTTP :11434 | LLM doc generation, email drafting |
| Workers | DocServer | MinIO transport | DOCX-to-PDF conversion (MinIO poller: to-convert/ → converted/) |
| Workers | State portals | Playwright | Name search, entity filing |
| Workers | IRS | Playwright | EIN obtainment |
| ERPNext | Email | SMTP | Invoices, notifications, ticket replies |
| Relay debit card | State portals | Playwright | Filing fee payment |
| Workers | HestiaCP | SSH | .ca domain/email provisioning (cp.carrierone.com) |
| Workers | Flowroute | REST API | Canadian DID provisioning |
| Workers | Porkbun | REST API | .ca domain registration |
| Workers | Client mailboxes | IMAP | Monitor/process client .ca email |
| Website | Anytime Mailbox | External | Vancouver mailbox for Canadian carrier clients |
| ERPNext | Venn.ca | Referral | Canadian business banking referral |
| AI Agents | MCP Server | stdio | Service lookup, pricing, tools (npm package) |

## Worker Processes (Python)

The `workers` container runs a job server (`job_server.py` on port 8090) that dispatches jobs
to specialized worker modules. Jobs are triggered by ERPNext webhooks via the Express API.

| Worker | File | Trigger | Purpose |
|--------|------|---------|---------|
| Job Server | `job_server.py` | HTTP POST :8090 | Central dispatcher — 22 job handlers |
| CRTC Pipeline | `services/canada_crtc.py` | Webhook | 14-step CRTC carrier registration pipeline |
| GCKey Provisioner | `gckey_provisioner.py` | Called by CRTC Step 11 | Playwright-based GCKey account creation (5-step wizard, hCaptcha) |
| Renewal Worker | `renewal_worker.py` | Daily cron (7 AM) | Compliance calendar lifecycle: upcoming → due soon → invoice → paid → completed → re-calendar |
| Formation Worker | `formation_worker.py` | Webhook | US state LLC/Corp filing via Playwright |
| Binder Compiler | `binder_compiler.py` | Called by pipeline | PDF merge: certificate + articles + CRTC letter + OA |
| Document Gen | `document_gen/` | Called by pipeline | DOCX template fill → MinIO → DocServer PDF conversion |
| HestiaCP Provisioner | `hestia_provisioner.py` | Called by pipeline | .ca domain + 14 mailboxes via SSH to cp.carrierone.com |
| Client Email Processor | `client_email_processor.py` | Cron (15 min) | IMAP monitor for client .ca mailboxes |
| AMB Scraper | `amb_location_scraper.py` | Daily cron | Anytime Mailbox location pricing + sold-out detection |
| Payment Reminder | `payment_reminder.py` | Cron (5 min) | Abandoned cart recovery (15min/1d/2d intervals) |
| Commission Worker | `commission_worker.py` | Cron (daily 02:00) | Agent commission eligibility check (14-day holdback) |
| Crypto Payment Worker | `crypto_payment_worker.py` | Cron (60s) | Treasury state machine: received → sizing → offramping → funds_at_relay → settled |
| Cold Wallet Sweeper | `cold_wallet_sweeper.py` | Cron (30 min) | Sweep SHKeeper hot wallet excess to cold storage |
| Relay Deposit Monitor | `relay_deposit_monitor.py` | Cron (5 min) | IMAP parser for Relay bank deposit alerts |
| USF Factor Monitor | `usf_factor_monitor.py` | Cron (daily 09:00 CT) | Scrape USAC quarterly USF factor, email all FCC carriers |
| De Minimis Check | `deminimis_factor_check.py` | Cron (daily 03:00) | Alert if fcc_deminimis_factors missing for current/next year |
| CDR Retention Sweeper | `cdr_retention_sweeper.py` | Cron (daily 05:00) | Purge CDR data past retention window |
| CDR Unlock Nudge | `cdr_unlock_nudge.py` | Cron (daily 10:00 CT) | Email customers with locked CDR studies behind paywall |
| FCC RMD Scraper | `fcc_rmd_removed_scraper.py` | Cron (weekly Wed) | Track FCC RMD carrier removals |
| Foreign Qual Handler | `services/foreign_qualification.py` | Webhook | Multi-state Certificate of Authority filings |
| New Carrier Bundle | `services/new_carrier_bundle.py` | Webhook | 6-handler chain: CORES → DC Agent → 499 Init → RMD → CPNI → CALEA |

### CRTC Pipeline Steps (14 total)

| Step | Name | Key Actions |
|------|------|-------------|
| 1 | Order Received | Validate, create ERPNext Sales Order |
| 2 | Payment Confirmed | Create Sales Invoice + Payment Entry |
| 3 | Client Selection | Client picks mailbox unit, DID, domain |
| 4 | BC Incorporation | Playwright → BC Corporate Online |
| 5 | Domain Registration | Porkbun .ca + HestiaCP provisioning |
| 6a | CRTC Letter Generation | DOCX template → MinIO → DocServer PDF |
| 6b | eSign | Email JWT link → client signs → resume pipeline |
| 7 | CRTC Submission | Mail signed letter to Secretary General |
| 8 | Anytime Mailbox | Playwright signup + IMAP OTP |
| 9 | Binder Compilation | PDF merge of all documents |
| 10 | Delivery | Email binder + admin print/ship email |
| 11 | BITS Registration | GCKey provisioning + admin ToDo for BITS filing |
| 12 | CCTS Membership | Admin ToDo + client obligations email |
| 13 | Compliance Calendar | Create 17 compliance entries (regulatory + tax + ATS) |
| 14 | Ready for Review | Final admin review before marking Delivered |

### Compliance Calendar Renewal Lifecycle

The `renewal_worker.py` runs daily at 7 AM and manages the full lifecycle of recurring
compliance obligations for all carriers:

```
Upcoming → Due Soon (30 days out) → Invoice Sent → Paid → Completed → auto-re-calendar next year
```

- **17 entries per carrier:** BC annual report, CRTC annual maintenance, mailbox renewal,
  domain renewal, DID renewal, CCTS renewal, T2 tax return, corporate tax payment,
  GST/HST return, T4/T4A slips, BC PST, WorkSafeBC, CRTC registration update,
  plus ATS survey forms (REP-T/T1 mandatory for all carriers)
- **Billable items** generate ERPNext Sales Invoice; entry cannot be completed until paid
- **On payment:** entry marked Completed, admin ToDo created, next-year entry auto-created
- **Webhook-triggered:** `handle_renewal_payment` in job_server for immediate processing

### DocServer (Windows VM)

MinIO-based transport — no direct HTTP connection between Linux and Windows VMs.

```
Workers: upload DOCX to minio://performancewest/to-convert/{uuid}.docx
  → DocServer polls to-convert/ bucket every 12 seconds
  → Word COM converts DOCX → PDF
  → DocServer uploads PDF to minio://performancewest/converted/{uuid}.pdf
  → Workers poll converted/ bucket for result
```

- **Heartbeat:** DocServer writes `docserver-heartbeat.json` to MinIO every 60 seconds
- **Fallback:** If heartbeat is stale (>5 min), workers auto-switch to LibreOffice headless
- **Self-healing:** the worker retries MinIO with backoff instead of exiting on an
  outage; the `PW-DocserverWorker` task restarts on failure (99×/1 min) and has a
  5-min repeating trigger, so a crash/missed boot self-recovers without RDP. See
  `docserver/README.md`.

## Boot Sequence

All services auto-start on reboot via systemd:

| Unit | What it starts | Depends on |
|------|----------------|------------|
| `docker.service` | Docker daemon | network |
| `k3s.service` | k3s + all SHKeeper pods | docker |
| `nginx.service` | Reverse proxy + TLS + RPC proxies (:5555/:5556) | network |
| `performancewest.service` | Prod docker compose (15 containers) | docker |
| `performancewest-dev.service` | Dev docker compose (4 containers) | docker, performancewest |

All containers use `restart: unless-stopped`. k3s manages pod restarts via deployment specs. The nginx RPC proxy configs for TronGrid (:5555) and ETH (:5556) are in `/etc/nginx/conf.d/` and load automatically. UFW rules for ports 5555/5556 are persistent (allow from 10.42.0.0/16 only).

## Scheduled Workers (systemd timers)

Deployed via `infra/ansible/roles/worker-crons/`. Each timer runs `docker compose exec -T workers python -m <module>`.

| Timer | Cadence | Module |
|---|---|---|
| `pw-usf-factor-monitor` | daily 09:00 CT | `scripts.workers.usf_factor_monitor` |
| `pw-deminimis-factor-check` | daily 03:00 UTC | `scripts.workers.deminimis_factor_check` |
| `pw-cold-wallet-sweep` | every 30 min | `scripts.workers.cold_wallet_sweeper` |
| `pw-crypto-payment-worker` | every 60s | `scripts.workers.crypto_payment_worker` |
| `pw-relay-deposit-monitor` | every 5 min | `scripts.workers.relay_deposit_monitor` |
| `pw-commission-worker` | daily 02:00 UTC | `scripts.workers.commission_worker` |
| `pw-renewal-worker` | daily 04:00 UTC | `scripts.workers.renewal_worker` |
| `pw-cdr-retention` | daily 05:00 UTC | `scripts.workers.cdr_retention_sweeper` |
| `pw-cdr-unlock-nudge` | daily 10:00 CT | `scripts.workers.cdr_unlock_nudge` |
| `pw-payment-reminder` | daily 11:00 CT | `scripts.workers.payment_reminder` |
| `pw-fcc-rmd-removed` | weekly Wed 08:00 CT | `scripts.workers.fcc_rmd_removed_scraper` |
| `pw-client-email-processor` | every 15 min | `scripts.workers.client_email_processor` |
| `pw-amb-location-scraper` | daily 06:00 UTC | `scripts.workers.amb_location_scraper` |

## Database Schema (67 migrations)

Key tables added in this cycle:

| Table | Migration | Purpose |
|---|---|---|
| `jurisdictions` | 066 | Unified US states + CA provinces (55 rows) |
| `foreign_qualification_registrations` | 066 | Per-state COA filings |
| `state_compliance_obligations` | 067 | Annual report fees/dates for all 51 US jurisdictions |
| `crypto_payment_ledger` | 062 | Immutable append-only money movement ledger |
| `crypto_payment_jobs` | 065 | Treasury state machine (received → settled) |
| `vendor_obligations` | 063 | Sizer: filing fees + commission reserves |
| `cold_wallet_config` / `cold_wallet_sweeps` | 064 | Cold wallet management |

## FCC Compliance Check Tool

Public at `/tools/fcc-compliance-check`. Queries FRN against CORES, RMD, 499 Filer DB, and CPNI records. Displays:
- CORES registration status (red-light check)
- RMD filing + certification date
- STIR/SHAKEN implementation (self-reported from RMD, hidden until STI-PA API access)
- CPNI annual certification (past-due detection with correct deadline logic)
- Form 499-A annual filing (CY-year-aware)
- Form 499-Q quarterly (de minimis trade-off explanation)
- BDC broadband/voice interactive toggle (two-step questionnaire)

Remediation CTA links to `/order/fcc-compliance?services=cpni,499a,...` for bundled ordering with 15% discount.

## Compliance Service Catalog (34 handlers)

| Slug | Handler | Price |
|---|---|---|
| `new-carrier-bundle` | NewCarrierBundleHandler (6 sub-handlers) | $1,799 |
| `fcc-full-compliance` | FullComplianceHandler | $1,499 |
| `fcc-499a` | Form499AHandler | $499 |
| `fcc-499a-499q` | Form499ABundleHandler | $599 |
| `cpni-certification` | CPNIFilingHandler (9 category variants) | $149 |
| `rmd-filing` | RMDFilingHandler | $219 |
| `calea-ssi` | CALEASSIHandler (6 category variants) | $299 |
| `bdc-filing` / `bdc-broadband` / `bdc-voice` | BDCFilingHandler | $299/$199/$149 |
| `foreign-qualification-single` | ForeignQualificationHandler | $149 + state fees |
| `foreign-qualification-multi` | ForeignQualificationHandler | $99/state + fees |
| `dc-agent` | DCAgentHandler (NWRA wholesale) | $99/yr |
| `cores-frn-registration` | CORESFRNRegistrationHandler | $99 |
| `fcc-499-initial` | Form499InitialHandler | $299 |
| `ocn-registration` | OCNRegistrationHandler | $799 |

## State Formation Adapters

10 states with real adapters (name search via Playwright + portal config):
WY (237 lines), CO (177), DE (119), FL (118), TX (SOSDirect), NV (SilverFlume), UT (DCC), NM, OH, MT.
~40 remaining states have stub adapters that create admin todos for manual filing.

JurisdictionConfig abstraction at `scripts/formation/jurisdictions/` reads from the `jurisdictions` DB table (migration 066) and provides per-state fees, portal URLs, NWRA addresses, and entity type catalogs.

## Deployment

**Production:** `ansible-playbook infra/ansible/playbooks/deploy.yml -i infra/ansible/inventory/hosts.yml`

**Dev:** `bash scripts/deploy-dev.sh` — rsyncs source files + rebuilds Docker containers. Static pages (tools, services, homepage) are in `site/public/` and survive Astro rebuilds.

**nginx cache policy:** `_astro/` = immutable (1 year). HTML = no-cache. Images = 30 days.
- **Atomic uploads:** `.tmp_` prefix + `copy_object` rename prevents partial reads