# Performance West — Go-Live Deployment Checklist

**Last updated:** 2026-04-05

---

## PRIORITY 1 — Infrastructure ~~COMPLETE~~

- [x] Provision Proxmox Linux VM: Debian 13, 32GB RAM, 8 vCPU, 232GB SSD
- [x] Point DNS: performancewest.net, api., portal., crm., lists., analytics., pay., crypto., minio., minio-console., dev., api.dev. → 207.174.124.71
- [x] Run Ansible site.yml playbook to provision full stack
- [x] Run all DB migrations (001–035) against PostgreSQL
- [x] Set up Let's Encrypt TLS certificates for all 12 subdomains
- [x] Configure firewall (UFW): allow 80, 443, SSH 22022 only + trusted IPs
- [x] Configure fail2ban: sshd, nginx-badbots, pw-api jails
- [x] Set up `performancewest.service` systemd unit for auto-start on reboot
- [x] Install k3s with `--docker --disable=traefik`
- [x] Install Helm 3
- [x] Set up `dev.performancewest.net` / `api.dev.performancewest.net` dev stack (ports 4323/3002)
- [x] Provision Windows VM for DocServer (108.181.102.34, Office 365 Word COM, MinIO transport)
  - SSH: port 22422 (key auth), Python 3.13 + pywin32 + minio SDK
  - Task Scheduler: PW-DocserverWorker (auto-start, auto-restart on failure)
  - Private network: 10.4.20.247 → MinIO via nginx
- [x] RAM upgrade: 62GB RAM verified, all 15 containers UP

---

## PRIORITY 2 — ERPNext + Integrations ~~COMPLETE~~

- [x] Configure ERPNext: create site, install erpnext + payments apps
- [x] Run setup wizard (Company: Performance West Inc., USD, America/Chicago)
- [x] Build custom ERPNext image with frappe_crypto + frappe_adyen + performancewest_erpnext
- [x] Install 6 apps: frappe, erpnext, payments, frappe_crypto, frappe_adyen, performancewest_erpnext
- [x] Import 7 custom DocTypes: Formation Order, Compliance Calendar, Sensitive ID, Referral Partner, Compliance Service, Sales Agent, Commission Ledger
- [x] Import 3 workflows: Formation Order (18 states), Canada CRTC (30 steps), Renewal
- [x] Create 16 service Items (CRTC Package $3,899, LLC Basic $179, etc.)
- [ ] Update Subscription Plans to new pricing (RA $99/yr ($49 WY), Annual Report $99/yr, CRTC Maintenance $349/yr)
- [ ] Update ERPNext Item rates: LLC Basic $179 (was $149), RA $99 (was $125), add WY RA $49
- [ ] Create new Items: CA Formation C$449, Formation Maintenance $179/yr, Free DID (stub)
- [x] Add custom fields: Sales Order (30+ fields incl. identity, esign, binder, regulatory)
- [x] Add custom fields: Sales Invoice (surcharge_pct, payment_gateway, stripe_payment_intent)
- [x] Hide 18 irrelevant modules for all users
- [x] Generate API keys (ERPNEXT_API_KEY / ERPNEXT_API_SECRET)
- [x] Configure ERPNext outgoing SMTP (co.carrierone.com:587, noreply@performancewest.net)
- [x] Configure ERPNext incoming email (support@performancewest.net → auto-create Issue)
- [x] Import 18 ERPNext webhooks: 11 CRTC + 7 Formation (all pointing to Express API)
- [x] Create ERPNext roles: Sales Agent (no desk), Accounting Advisor (desk access)
- [x] Import 20 compliance service fixtures (compliance_services.json)
- [x] Configure ERPNext Assignment Rule: Accounting Support → Accounting Advisor role
- [x] Portal Settings: enable Sales Order, Sales Invoice, Issue for customer portal
- [x] Website Settings: PW branding, navy theme, custom CSS
- [x] Create portal.performancewest.net DNS + SSL + nginx vhost → ERPNext :8080
- [x] 6 CRTC workflow notification emails (Received → Delivered) using Carbonio SMTP
- [ ] Configure ERPNext incoming email IMAP password for support@ account
- [ ] Create ERPNext Payment Gateway Account for Crypto-Crypto (SHKeeper)
- [ ] Configure Crypto Payment Settings in ERPNext UI (pay.performancewest.net + SHKeeper API key)
- [ ] Configure ERPNext Assignment Rules: expand for CRTC ticket routing

---

## PRIORITY 3 — Payment Processing

- [x] Deploy SHKeeper via Helm in k3s (7 crypto daemons: BTC, ETH, MATIC, BNB, TRX, LTC, DOGE)
- [x] Set up SHKeeper nginx reverse proxy at pay.performancewest.net + crypto.performancewest.net
- [x] Build frappe_crypto app (SHKeeper gateway, checkout page, webhook receiver)
- [x] Rewrite checkout.ts: Stripe Checkout Sessions (card + ACH + Klarna), PayPal Orders v2, SHKeeper crypto
- [x] Add Stripe webhook handler — events: checkout.session.completed, payment_intent.succeeded, balance.available
- [x] Add SHKeeper webhook handler for crypto payment confirmation
- [x] Add PayPal capture / webhook handler
- [x] Order confirmation email on payment (Carbonio SMTP via email.ts)
- [x] Auto-create Sales Invoice + Payment Entry on payment (createInvoiceFromSalesOrder)
- [x] Fund detection: balance.available webhook → T+2/T+4 business day advance → Stripe Issuing topup
- [x] Abandoned cart recovery (payment_reminder.py cron every 5 min, 15min/1d/2d intervals)
- [x] Payment method selector on order form Step 5 (ACH recommended, expedited → PayPal/crypto only)
- [x] Expedited processing toggle (+$500, restricts to PayPal/crypto only)
- [ ] Set `STRIPE_SECRET_KEY` + `STRIPE_WEBHOOK_SECRET` + `STRIPE_IDENTITY_WEBHOOK_SECRET` in .env
- [ ] Register Stripe webhook at api.performancewest.net/api/v1/webhooks/stripe
  - Events: checkout.session.completed, payment_intent.succeeded, balance.available, identity.verification_session.verified
- [ ] Set `SHKEEPER_API_KEY` in .env
- [ ] Test end-to-end Stripe card payment flow
- [ ] Test end-to-end Stripe ACH payment flow
- [ ] Test end-to-end PayPal payment flow
- [ ] Test end-to-end crypto payment flow
- [ ] Adyen: apply for merchant account (future — not blocking launch)

---

## PRIORITY 4 — Banking & Financial

- [ ] Set up Relay virtual debit card (SID-0002) dedicated to filing fees
- [ ] Store Relay card details in ERPNext Sensitive ID (encrypted)
- [ ] Set up Relay checking account structure (Operating, Filing Fees, Profit, Tax, Commissions, Owner Pay)
- [ ] Store PayPal Mastercard (SID-0001) details in ERPNext Sensitive ID
- [ ] Change NW RA portal password (was shared in chat — URGENT)
- [ ] Create Northwest Registered Agent wholesale orders for each state's RA address

---

## PRIORITY 5 — Document Generation & Workers

- [x] Set up MinIO: create performancewest bucket, configure access
- [x] Upload 9 DOCX templates to MinIO (to-convert/ → converted/ transport buckets created)
- [x] Configure Ollama with qwen2.5:7b model
- [x] Install Playwright browsers (chromium) in worker container
- [x] Add PyJWT to scripts/requirements.txt (for portal eSign JWT generation)
- [x] DocServer: MinIO-based transport (docserver_worker.py polls to-convert/, drops converted/)
  - Worker polls every 12s, heartbeat every 60s at minio://performancewest/docserver-heartbeat.json
  - LibreOffice headless fallback auto-activates when DocServer unavailable
- [x] Test LibreOffice fallback DOCX-to-PDF conversion *(verify manually)*
- [x] Provision Windows VM, run docserver/install.ps1 with MinIO credentials
- [x] Verify Word COM conversion end-to-end (36KB DOCX → 82KB PDF, 12s round-trip)
- [x] MinIO nginx allow-list updated with private network IP 10.4.20.247
- [x] ERPNext CSS fix after reboot (`chmod o+x /var/lib/docker` + `bench build --force`)
- [x] ERPNext "Sent via ERPNext" email footer disabled (`disable_standard_email_footer: 1`)

---

## PRIORITY 6 — External Accounts & Services

- [ ] Claim Google Business Profile at business.google.com for Performance West Inc.
- [ ] Claim Trustpilot business profile at business.trustpilot.com
- [ ] Update Google Reviews link in homepage with actual Google Place ID
- [ ] Create SOSDirect account for Texas state filings
- [ ] Create UtahID account for Utah state filings
- [ ] Sign up for 2captcha or anticaptcha (Delaware CAPTCHA solving)
- [ ] Set up Porkbun account for .ca domain registration (CRTC service)
- [ ] Set `PORKBUN_API_KEY` / `PORKBUN_SECRET_KEY` in .env
- [ ] Set up Flowroute account for Canadian DID provisioning
- [ ] Set `FLOWROUTE_ACCESS_KEY` / `FLOWROUTE_SECRET_KEY` in .env
- [ ] Configure HestiaCP on cp.carrierone.com for .ca domain hosting + email
- [ ] Set `HESTIA_SSH_KEY` path in workers container .env (key must be mounted or baked)
- [ ] Test HestiaCP provisioner: `python -m scripts.workers.hestia_provisioner provision test.ca "Test Company"`
- [ ] Verify all 14 mailbox types create correctly per domain
- [ ] Set up Anytime Mailbox wholesale/partner account
- [ ] Configure ANYTIME_MAILBOX_IMAP_PASS for OTP auto-fetch
- [x] Configure client email processor cron (every 15 min): deployed via `worker-crons` ansible role (`pw-client-email-processor.timer`)

---

## PRIORITY 7 — Listmonk (Email Marketing) ~~COMPLETE~~

- [x] Deploy Listmonk at lists.performancewest.net (Docker, PostgreSQL-backed)
- [x] Configure SMTP2GO outbound email (separate from Carbonio transactional SMTP)
- [x] Import 10,191 FCC RMD contacts into Listmonk (3 subscriber lists)
- [x] Configure bounce processing via POP3 from Carbonio (bounces@performancewest.net)
- [x] Create 22 scheduled campaigns across 4 lists
- [x] nginx vhost at lists.performancewest.net with Let's Encrypt cert
- [x] Remove RMD/Robocall Mitigation Database language from all campaigns

---

## PRIORITY 8 — Analytics & Security

- [x] Deploy Umami analytics container at analytics.performancewest.net
- [x] Umami first-run config: site registered, site ID `55250014-ee15-44ac-a1f6-81dabad3fe0f`
- [x] Add Umami tracking script to Base.astro (`!isDev` guard, production only)
- [x] nginx CSP includes analytics.performancewest.net in script-src + connect-src
- [x] CAA DNS records for Let's Encrypt
- [x] HSTS headers on all domains
- [x] fail2ban pw-api jail (rate-limit abusers at /api/v1/checkout)
- [ ] Verify SSL Labs A+ grade after deployment
- [ ] Verify SecurityHeaders.com A+ grade
- [ ] Verify Mozilla Observatory A+ grade
- [ ] Verify Google Safe Browsing clean status

---

## PRIORITY 9 — State Formation Automation

- [ ] Populate CSS selectors for top 10 states (WY, CO, DE, FL, TX, NV, UT, NM, OH, MT)
- [ ] Test Wyoming name search end-to-end via Playwright
- [ ] Test Colorado name search via Socrata API (confirmed working in isolation)
- [ ] Test full formation order flow: order → name search → payment → filing
- [ ] Implement ~40 remaining state adapters (only ~12 have real implementations)

---

## PRIORITY 10 — Canada CRTC Service

- [x] CRTC order form (5-step, identity verification, AMB location picker, payment methods)
- [x] Stripe Identity gate (Step 4, auto-advance after verified, form snapshot persistence)
- [x] AMB location scraper (daily cron, operator_name extraction, sold-out deactivation)
- [x] Anytime Mailbox signup automation (Playwright + IMAP OTP from Carbonio)
- [x] Flowroute DID search + purchase (area codes 604/778/236/250)
- [x] .ca domain registration via Porkbun (CIRA CPR: BC corp number, category CCO)
- [x] HestiaCP provisioner: 14 mailboxes per domain including regulatory@, abuse@, postmaster@
  - Credentials stored in ERPNext Sensitive ID (encrypted)
  - Email 1 (brief): domain live notification
  - Email 2 (credentials): all 14 passwords, IMAP/SMTP settings, webmail URL
- [x] BC incorporation via COLIN (anonymous filing, no login required)
- [x] CRTC notification letter generation (DOCX template → MinIO transport → Word/LibreOffice PDF)
- [x] eSign portal page (/portal/sign) — canvas drawing pad, letter preview iframe, JWT auth
  - Pipeline pauses at Step 6b, emails client JWT-signed sign link (72h expiry)
  - Signature stored as base64 PNG in PG (esign_signature_b64)
  - On submit: ERPNext workflow advances to "CRTC Submitted", resume_crtc_pipeline job dispatched
- [x] Corporate binder compilation (PDF merge: certificate + articles + CRTC letter + OA)
- [x] Binder delivery email: FROM regulatory@domain.ca (HestiaCP SMTP credentials)
  - AMB binder label: `c/o <operator_name>`, unit number
  - Own-address binder label: company name + attn contact
- [x] Admin print/ship email (PirateShip USPS label instructions)
- [x] Client portal order status page at portal.performancewest.net/orders
  - 10-step visual pipeline with colored chips (completed/active/pending)
  - Action CTAs: "Complete Setup →" (Client Selection), "Sign CRTC Letter →" (Pending eSign)
  - Invoice table with Paid/Partial/Unpaid badges
- [x] Phase 2 customer auth: ERPNext Website User created on order, portal password set from success page
- [x] Phase 4 auto-invoice: Sales Invoice + Payment Entry created on payment
- [x] Phase 5 ERPNext notifications: 6 workflow state emails (Received → Delivered) via Carbonio
- [x] Phase 6 custom portal /orders page in performancewest_erpnext Frappe app
- [x] BITS registration step (Step 11) — GCKey provisioning via Playwright + admin ToDo for BITS filing
  - GCKey signup wizard automated: 5-step flow, hCaptcha handling, credentials to ERPNext Sensitive ID
  - Username format: `pw-{bc_number}`, recovery email: `regulatory@domain.ca`
- [x] CCTS registration step (Step 12) — admin ToDo + client obligations email
- [x] Compliance calendar auto-creation (Step 13) — 17 entries per carrier
  - Regulatory: BC annual report, CRTC maintenance, mailbox, domain, DID, CCTS, CRTC update
  - Tax: T2, corporate tax, GST/HST, T4/T4A, BC PST, WorkSafeBC
  - Surveys: REP-T/T1 (mandatory), REP-U/802a/802j/Facilities/Pricing (>$10M threshold)
- [x] Compliance calendar renewal lifecycle (`renewal_worker.py`)
  - Daily cron: upcoming → due soon → invoice sent → paid → completed → re-calendar
  - Billable items generate ERPNext Sales Invoice; payment required before completion
  - Webhook handler: `handle_renewal_payment` in job_server.py
- [x] CRTC pipeline expanded from 12 to 14 steps (BITS + CCTS + expanded compliance + review)
- [x] BC config expanded: `bits`, `ccts`, `gckey`, `ats` (6 survey forms), `corporate_obligations` (7 tax/filing items)
- [ ] BC COLIN selector verification (Steps 5-12 need live session testing)
- [ ] Verify GCKey Steps 3-5 selectors on first live provisioning run
- [ ] Verify AMB operator_name extraction on next daily scraper run
- [ ] Test full CRTC order end-to-end on dev stack
- [ ] Deploy BITS/CCTS/compliance/renewal code to production
- [x] Add renewal cron — deployed via `worker-crons` ansible role (`pw-renewal-worker.timer` daily 04:00 UTC)
- [x] Add USF factor monitor cron — deployed via `worker-crons` ansible role (`pw-usf-factor-monitor.timer` daily 09:00 CT)
  - Polls https://www.usac.org/service-providers/making-payments/contribution-factors/
  - On detecting a new quarterly factor, emails all FCC-carrier customers (bcc justin@) with the new % and the delta vs. prior quarter so they can update their USF surcharges before the quarter starts
  - Requires migration 049 (`usf_contribution_factors` table) to be applied
- [x] Create ERPNext Items for renewal invoicing: CRTC-MAINT-ANNUAL, MAILBOX-RENEWAL, BC-ANNUAL-REPORT, DOMAIN-RENEWAL-CA, COMPLIANCE-OTHER (fixture: `performancewest_erpnext/performancewest_erpnext/fixtures/item.json`; imports on `bench migrate`)
- [ ] Import updated Compliance Calendar DocType to production ERPNext

---

## PRIORITY 11 — Multi-Province, Canadian Formation & Universal Compliance

> Full plan: [`docs/multi-province-plan.md`](multi-province-plan.md) — 16 sub-phases, 28-36 hours, 5-6 sessions

**Three interconnected features:**
1. Multi-province CRTC ($3,899, Ontario first)
2. Standalone Canadian formation (C$449 + gov fees, separate flow from CRTC)
3. Universal compliance calendar (all order types: US + CA formation + CRTC)

- [ ] Phase 0: OBR + CRA BN Playwright recon (Ontario Business Registry + CRA Business Number)
- [ ] Phase 1: ProvinceConfig abstraction + universal compliance module design
- [ ] Phase 2: DB migrations (CRTC province column + formation entity types + country)
- [ ] Phase 3a-d: Ontario config, adapter, AMB scraper extension, Flowroute ON area codes
- [ ] Phase 3e: CanadianIncorporationHandler (shared pipeline for formation-only + CRTC)
- [ ] Phase 3f: CRA Business Number registration stub ($49 add-on)
- [ ] Phase 3g: Universal compliance entry creator (all order types)
- [ ] Phase 4: Service page province comparison table + FAQ rewrite
- [ ] Phase 5a: CRTC order form multi-province (province selector, dynamic content)
- [ ] Phase 5b: Formation page backend wiring (frontend already supports Canada)
- [ ] Phase 6a: CRTC API route (accept province)
- [ ] Phase 6b: CRTC pipeline refactor (compose with CanadianIncorporationHandler)
- [ ] Phase 6c: Formation API route (accept CA provinces + entity types ltd/inc/corp)
- [ ] Phase 7a: Canadian province compliance configs (BC + ON corporate obligations)
- [ ] Phase 7b: US state compliance configs (all 51 jurisdictions — annual report, franchise tax, RA)
- [ ] Phase 8: Dev E2E tests (CRTC ON + CA formation-only + US formation compliance)

**New products:**
- CA Formation: C$449 + gov fees (incorp + minutes + binder + compliance calendar). AMB mailbox separate.
- CRA BN add-on: $49
- Formation Maintenance Bundle (US or CA): $179/yr (annual report $99 + RA renewal $99)

---

## PRIORITY 12 — Sales & Marketing

- [ ] Configure Google Analytics 4 property and connect to Umami export
- [ ] Set up Google Search Console for performancewest.net
- [ ] Set up Google Ads account (future)
- [ ] Create referral program landing page (/agents)
- [ ] Onboard first 3 sales agents (test referral code flow end-to-end)
- [ ] Create Trustpilot review request automation in post-delivery email

---

## PRIORITY 13 — Content & Video

- [ ] Generate AI-narrated video for Canada CRTC service page
- [ ] Generate marketing videos for US formation services
- [ ] Create vendor directory PDF for CRTC clients (5 categories, delivered via portal)
- [ ] Create US fintech banking guide PDF for CRTC clients (Wise, Airwallex, Payoneer)

---

## PRIORITY 14 — MCP Server & AI Integration

- [ ] Publish MCP server package to npm (`@performancewest/mcp`)
- [ ] Test all 10 MCP tools against live ERPNext
- [ ] Document MCP server usage for AI agent integration

---

## PRIORITY 15 — Client Portal & Testing

- [x] Portal order status page (portal.performancewest.net/orders) — visual 10-step pipeline
- [x] eSign page (/portal/sign) — canvas signature, letter preview, JWT auth
- [x] Client selection page (/portal/setup) — unit picker, DID picker, confirm
- [x] Domain search page (/portal/domain-search)
- [x] Manage services page (/portal/manage-services)
- [ ] Phase 7: SupportWidget → ERPNext Issue direct (remove PG tickets fallback)
- [ ] Phase 8: Deprecate PG customer auth (portal-auth.ts login/register → ERPNext portal only)
- [ ] Phase 9: E2E test — order → payment → invoice → portal login → order status → ticket
- [ ] Load test: simulate 10 concurrent CRTC orders through the full pipeline
- [ ] Mobile test all order form steps on iOS Safari + Android Chrome

---

## PRIORITY 16 — Environment Variables Still Needed

- [ ] `STRIPE_SECRET_KEY` — from Stripe Dashboard (live key)
- [ ] `STRIPE_WEBHOOK_SECRET` — from Stripe Dashboard → Webhooks endpoint
- [ ] `STRIPE_IDENTITY_WEBHOOK_SECRET` — from Stripe Dashboard (identity webhook)
- [x] `CUSTOMER_JWT_SECRET` — generated and set (M0IFZv9ipOxU5juZnf7zwdTfpk_5_5...)
- [x] `SMTP_PASS` — set ($m)}4G8&!yu;{)#]sp, Carbonio noreply@)
- [ ] `SHKEEPER_API_KEY` — from SHKeeper admin panel at crypto.performancewest.net
- [ ] `PORKBUN_API_KEY` / `PORKBUN_SECRET_KEY` — from Porkbun account API settings
- [ ] `FLOWROUTE_ACCESS_KEY` / `FLOWROUTE_SECRET_KEY` — from Flowroute account
- [ ] `HESTIA_SSH_KEY` — path to SSH private key on workers container (mount via volume)
- [ ] `ANYTIME_MAILBOX_IMAP_PASS` — mailbox password for IMAP OTP fetch
- [ ] `PAYPAL_CLIENT_ID` / `PAYPAL_CLIENT_SECRET` — from PayPal developer dashboard
- [ ] `MINIO_ACCESS_KEY` / `MINIO_SECRET_KEY` — already set (verify on server)
- [x] Windows DocServer MinIO credentials — configured in `C:\docserver\docserver.env` on VM

---

## PRIORITY 17 — Low Priority / When Revenue Supports

- [ ] Adyen merchant account application (card/ACH/Klarna/CashApp/AmazonPay via Adyen)
- [ ] Stripe Issuing cardholder setup for SID-0002 (Relay is interim filing card)
- [ ] Automated annual report filing for existing clients
- [ ] Attorney review integration (removed from site; may re-add as premium tier)
- [ ] Multi-language support (French for Quebec clients)
- [ ] Mobile app (React Native) for client portal access
- [ ] Wholesale API for resellers and referring attorneys

---

## POST-LAUNCH MONITORING

- [ ] Set up UptimeRobot (or similar) monitors for:
  - https://performancewest.net (site)
  - https://api.performancewest.net/health (API)
  - https://portal.performancewest.net (ERPNext portal)
  - https://lists.performancewest.net (Listmonk)
  - SHKeeper /health endpoint
- [ ] Set up Grafana + Prometheus for container metrics (future)
- [ ] Set up PagerDuty or similar for on-call alerts
- [ ] Review Umami weekly dashboard every Monday
- [ ] Review ERPNext open Issues weekly (customer support queue)
- [ ] Review abandoned cart report weekly (payment_reminder.py logs)
- [ ] Monthly: review state filing fee table for changes
- [ ] Monthly: review AMB location pricing (daily scraper catches this automatically)
- [ ] Quarterly: rotate API keys (Stripe, SHKeeper, Porkbun, Flowroute)