new-site/docs/go-live-todo.md
justin f8cd37ac8c Initial commit — Performance West telecom compliance platform
Includes: API (Express/TypeScript), Astro site, Python workers,
document generators, FCC compliance tools, Canada CRTC formation,
Ansible infrastructure, and deployment scripts.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-27 06:54:22 -05:00

359 lines
21 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Performance West — Go-Live Deployment Checklist
**Last updated:** 2026-04-05
---
## PRIORITY 1 — Infrastructure ~~COMPLETE~~
- [x] Provision Proxmox Linux VM: Debian 13, 32GB RAM, 8 vCPU, 232GB SSD
- [x] Point DNS: performancewest.net, api., portal., crm., lists., analytics., pay., crypto., minio., minio-console., dev., api.dev. → 207.174.124.71
- [x] Run Ansible site.yml playbook to provision full stack
- [x] Run all DB migrations (001035) against PostgreSQL
- [x] Set up Let's Encrypt TLS certificates for all 12 subdomains
- [x] Configure firewall (UFW): allow 80, 443, SSH 22022 only + trusted IPs
- [x] Configure fail2ban: sshd, nginx-badbots, pw-api jails
- [x] Set up `performancewest.service` systemd unit for auto-start on reboot
- [x] Install k3s with `--docker --disable=traefik`
- [x] Install Helm 3
- [x] Set up `dev.performancewest.net` / `api.dev.performancewest.net` dev stack (ports 4323/3002)
- [x] Provision Windows VM for DocServer (108.181.102.34, Office 365 Word COM, MinIO transport)
- SSH: port 22422 (key auth), Python 3.13 + pywin32 + minio SDK
- Task Scheduler: PW-DocserverWorker (auto-start, auto-restart on failure)
- Private network: 10.4.20.247 → MinIO via nginx
- [x] RAM upgrade: 62GB RAM verified, all 15 containers UP
---
## PRIORITY 2 — ERPNext + Integrations ~~COMPLETE~~
- [x] Configure ERPNext: create site, install erpnext + payments apps
- [x] Run setup wizard (Company: Performance West Inc., USD, America/Chicago)
- [x] Build custom ERPNext image with frappe_crypto + frappe_adyen + performancewest_erpnext
- [x] Install 6 apps: frappe, erpnext, payments, frappe_crypto, frappe_adyen, performancewest_erpnext
- [x] Import 7 custom DocTypes: Formation Order, Compliance Calendar, Sensitive ID, Referral Partner, Compliance Service, Sales Agent, Commission Ledger
- [x] Import 3 workflows: Formation Order (18 states), Canada CRTC (30 steps), Renewal
- [x] Create 16 service Items (CRTC Package $3,899, LLC Basic $179, etc.)
- [ ] Update Subscription Plans to new pricing (RA $99/yr ($49 WY), Annual Report $99/yr, CRTC Maintenance $349/yr)
- [ ] Update ERPNext Item rates: LLC Basic $179 (was $149), RA $99 (was $125), add WY RA $49
- [ ] Create new Items: CA Formation C$449, Formation Maintenance $179/yr, Free DID (stub)
- [x] Add custom fields: Sales Order (30+ fields incl. identity, esign, binder, regulatory)
- [x] Add custom fields: Sales Invoice (surcharge_pct, payment_gateway, stripe_payment_intent)
- [x] Hide 18 irrelevant modules for all users
- [x] Generate API keys (ERPNEXT_API_KEY / ERPNEXT_API_SECRET)
- [x] Configure ERPNext outgoing SMTP (co.carrierone.com:587, noreply@performancewest.net)
- [x] Configure ERPNext incoming email (support@performancewest.net → auto-create Issue)
- [x] Import 18 ERPNext webhooks: 11 CRTC + 7 Formation (all pointing to Express API)
- [x] Create ERPNext roles: Sales Agent (no desk), Accounting Advisor (desk access)
- [x] Import 20 compliance service fixtures (compliance_services.json)
- [x] Configure ERPNext Assignment Rule: Accounting Support → Accounting Advisor role
- [x] Portal Settings: enable Sales Order, Sales Invoice, Issue for customer portal
- [x] Website Settings: PW branding, navy theme, custom CSS
- [x] Create portal.performancewest.net DNS + SSL + nginx vhost → ERPNext :8080
- [x] 6 CRTC workflow notification emails (Received → Delivered) using Carbonio SMTP
- [ ] Configure ERPNext incoming email IMAP password for support@ account
- [ ] Create ERPNext Payment Gateway Account for Crypto-Crypto (SHKeeper)
- [ ] Configure Crypto Payment Settings in ERPNext UI (pay.performancewest.net + SHKeeper API key)
- [ ] Configure ERPNext Assignment Rules: expand for CRTC ticket routing
---
## PRIORITY 3 — Payment Processing
- [x] Deploy SHKeeper via Helm in k3s (7 crypto daemons: BTC, ETH, MATIC, BNB, TRX, LTC, DOGE)
- [x] Set up SHKeeper nginx reverse proxy at pay.performancewest.net + crypto.performancewest.net
- [x] Build frappe_crypto app (SHKeeper gateway, checkout page, webhook receiver)
- [x] Rewrite checkout.ts: Stripe Checkout Sessions (card + ACH + Klarna), PayPal Orders v2, SHKeeper crypto
- [x] Add Stripe webhook handler — events: checkout.session.completed, payment_intent.succeeded, balance.available
- [x] Add SHKeeper webhook handler for crypto payment confirmation
- [x] Add PayPal capture / webhook handler
- [x] Order confirmation email on payment (Carbonio SMTP via email.ts)
- [x] Auto-create Sales Invoice + Payment Entry on payment (createInvoiceFromSalesOrder)
- [x] Fund detection: balance.available webhook → T+2/T+4 business day advance → Stripe Issuing topup
- [x] Abandoned cart recovery (payment_reminder.py cron every 5 min, 15min/1d/2d intervals)
- [x] Payment method selector on order form Step 5 (ACH recommended, expedited → PayPal/crypto only)
- [x] Expedited processing toggle (+$500, restricts to PayPal/crypto only)
- [ ] Set `STRIPE_SECRET_KEY` + `STRIPE_WEBHOOK_SECRET` + `STRIPE_IDENTITY_WEBHOOK_SECRET` in .env
- [ ] Register Stripe webhook at api.performancewest.net/api/v1/webhooks/stripe
- Events: checkout.session.completed, payment_intent.succeeded, balance.available, identity.verification_session.verified
- [ ] Set `SHKEEPER_API_KEY` in .env
- [ ] Test end-to-end Stripe card payment flow
- [ ] Test end-to-end Stripe ACH payment flow
- [ ] Test end-to-end PayPal payment flow
- [ ] Test end-to-end crypto payment flow
- [ ] Adyen: apply for merchant account (future — not blocking launch)
---
## PRIORITY 4 — Banking & Financial
- [ ] Set up Relay virtual debit card (SID-0002) dedicated to filing fees
- [ ] Store Relay card details in ERPNext Sensitive ID (encrypted)
- [ ] Set up Relay checking account structure (Operating, Filing Fees, Profit, Tax, Commissions, Owner Pay)
- [ ] Store PayPal Mastercard (SID-0001) details in ERPNext Sensitive ID
- [ ] Change NW RA portal password (was shared in chat — URGENT)
- [ ] Create Northwest Registered Agent wholesale orders for each state's RA address
---
## PRIORITY 5 — Document Generation & Workers
- [x] Set up MinIO: create performancewest bucket, configure access
- [x] Upload 9 DOCX templates to MinIO (to-convert/ → converted/ transport buckets created)
- [x] Configure Ollama with qwen2.5:7b model
- [x] Install Playwright browsers (chromium) in worker container
- [x] Add PyJWT to scripts/requirements.txt (for portal eSign JWT generation)
- [x] DocServer: MinIO-based transport (docserver_worker.py polls to-convert/, drops converted/)
- Worker polls every 12s, heartbeat every 60s at minio://performancewest/docserver-heartbeat.json
- LibreOffice headless fallback auto-activates when DocServer unavailable
- [x] Test LibreOffice fallback DOCX-to-PDF conversion *(verify manually)*
- [x] Provision Windows VM, run docserver/install.ps1 with MinIO credentials
- [x] Verify Word COM conversion end-to-end (36KB DOCX → 82KB PDF, 12s round-trip)
- [x] MinIO nginx allow-list updated with private network IP 10.4.20.247
- [x] ERPNext CSS fix after reboot (`chmod o+x /var/lib/docker` + `bench build --force`)
- [x] ERPNext "Sent via ERPNext" email footer disabled (`disable_standard_email_footer: 1`)
---
## PRIORITY 6 — External Accounts & Services
- [ ] Claim Google Business Profile at business.google.com for Performance West Inc.
- [ ] Claim Trustpilot business profile at business.trustpilot.com
- [ ] Update Google Reviews link in homepage with actual Google Place ID
- [ ] Create SOSDirect account for Texas state filings
- [ ] Create UtahID account for Utah state filings
- [ ] Sign up for 2captcha or anticaptcha (Delaware CAPTCHA solving)
- [ ] Set up Porkbun account for .ca domain registration (CRTC service)
- [ ] Set `PORKBUN_API_KEY` / `PORKBUN_SECRET_KEY` in .env
- [ ] Set up Flowroute account for Canadian DID provisioning
- [ ] Set `FLOWROUTE_ACCESS_KEY` / `FLOWROUTE_SECRET_KEY` in .env
- [ ] Configure HestiaCP on cp.carrierone.com for .ca domain hosting + email
- [ ] Set `HESTIA_SSH_KEY` path in workers container .env (key must be mounted or baked)
- [ ] Test HestiaCP provisioner: `python -m scripts.workers.hestia_provisioner provision test.ca "Test Company"`
- [ ] Verify all 14 mailbox types create correctly per domain
- [ ] Set up Anytime Mailbox wholesale/partner account
- [ ] Configure ANYTIME_MAILBOX_IMAP_PASS for OTP auto-fetch
- [x] Configure client email processor cron (every 15 min): deployed via `worker-crons` ansible role (`pw-client-email-processor.timer`)
---
## PRIORITY 7 — Listmonk (Email Marketing) ~~COMPLETE~~
- [x] Deploy Listmonk at lists.performancewest.net (Docker, PostgreSQL-backed)
- [x] Configure SMTP2GO outbound email (separate from Carbonio transactional SMTP)
- [x] Import 10,191 FCC RMD contacts into Listmonk (3 subscriber lists)
- [x] Configure bounce processing via POP3 from Carbonio (bounces@performancewest.net)
- [x] Create 22 scheduled campaigns across 4 lists
- [x] nginx vhost at lists.performancewest.net with Let's Encrypt cert
- [x] Remove RMD/Robocall Mitigation Database language from all campaigns
---
## PRIORITY 8 — Analytics & Security
- [x] Deploy Umami analytics container at analytics.performancewest.net
- [x] Umami first-run config: site registered, site ID `55250014-ee15-44ac-a1f6-81dabad3fe0f`
- [x] Add Umami tracking script to Base.astro (`!isDev` guard, production only)
- [x] nginx CSP includes analytics.performancewest.net in script-src + connect-src
- [x] CAA DNS records for Let's Encrypt
- [x] HSTS headers on all domains
- [x] fail2ban pw-api jail (rate-limit abusers at /api/v1/checkout)
- [ ] Verify SSL Labs A+ grade after deployment
- [ ] Verify SecurityHeaders.com A+ grade
- [ ] Verify Mozilla Observatory A+ grade
- [ ] Verify Google Safe Browsing clean status
---
## PRIORITY 9 — State Formation Automation
- [ ] Populate CSS selectors for top 10 states (WY, CO, DE, FL, TX, NV, UT, NM, OH, MT)
- [ ] Test Wyoming name search end-to-end via Playwright
- [ ] Test Colorado name search via Socrata API (confirmed working in isolation)
- [ ] Test full formation order flow: order → name search → payment → filing
- [ ] Implement ~40 remaining state adapters (only ~12 have real implementations)
---
## PRIORITY 10 — Canada CRTC Service
- [x] CRTC order form (5-step, identity verification, AMB location picker, payment methods)
- [x] Stripe Identity gate (Step 4, auto-advance after verified, form snapshot persistence)
- [x] AMB location scraper (daily cron, operator_name extraction, sold-out deactivation)
- [x] Anytime Mailbox signup automation (Playwright + IMAP OTP from Carbonio)
- [x] Flowroute DID search + purchase (area codes 604/778/236/250)
- [x] .ca domain registration via Porkbun (CIRA CPR: BC corp number, category CCO)
- [x] HestiaCP provisioner: 14 mailboxes per domain including regulatory@, abuse@, postmaster@
- Credentials stored in ERPNext Sensitive ID (encrypted)
- Email 1 (brief): domain live notification
- Email 2 (credentials): all 14 passwords, IMAP/SMTP settings, webmail URL
- [x] BC incorporation via COLIN (anonymous filing, no login required)
- [x] CRTC notification letter generation (DOCX template → MinIO transport → Word/LibreOffice PDF)
- [x] eSign portal page (/portal/sign) — canvas drawing pad, letter preview iframe, JWT auth
- Pipeline pauses at Step 6b, emails client JWT-signed sign link (72h expiry)
- Signature stored as base64 PNG in PG (esign_signature_b64)
- On submit: ERPNext workflow advances to "CRTC Submitted", resume_crtc_pipeline job dispatched
- [x] Corporate binder compilation (PDF merge: certificate + articles + CRTC letter + OA)
- [x] Binder delivery email: FROM regulatory@domain.ca (HestiaCP SMTP credentials)
- AMB binder label: `c/o <operator_name>`, unit number
- Own-address binder label: company name + attn contact
- [x] Admin print/ship email (PirateShip USPS label instructions)
- [x] Client portal order status page at portal.performancewest.net/orders
- 10-step visual pipeline with colored chips (completed/active/pending)
- Action CTAs: "Complete Setup →" (Client Selection), "Sign CRTC Letter →" (Pending eSign)
- Invoice table with Paid/Partial/Unpaid badges
- [x] Phase 2 customer auth: ERPNext Website User created on order, portal password set from success page
- [x] Phase 4 auto-invoice: Sales Invoice + Payment Entry created on payment
- [x] Phase 5 ERPNext notifications: 6 workflow state emails (Received → Delivered) via Carbonio
- [x] Phase 6 custom portal /orders page in performancewest_erpnext Frappe app
- [x] BITS registration step (Step 11) — GCKey provisioning via Playwright + admin ToDo for BITS filing
- GCKey signup wizard automated: 5-step flow, hCaptcha handling, credentials to ERPNext Sensitive ID
- Username format: `pw-{bc_number}`, recovery email: `regulatory@domain.ca`
- [x] CCTS registration step (Step 12) — admin ToDo + client obligations email
- [x] Compliance calendar auto-creation (Step 13) — 17 entries per carrier
- Regulatory: BC annual report, CRTC maintenance, mailbox, domain, DID, CCTS, CRTC update
- Tax: T2, corporate tax, GST/HST, T4/T4A, BC PST, WorkSafeBC
- Surveys: REP-T/T1 (mandatory), REP-U/802a/802j/Facilities/Pricing (>$10M threshold)
- [x] Compliance calendar renewal lifecycle (`renewal_worker.py`)
- Daily cron: upcoming → due soon → invoice sent → paid → completed → re-calendar
- Billable items generate ERPNext Sales Invoice; payment required before completion
- Webhook handler: `handle_renewal_payment` in job_server.py
- [x] CRTC pipeline expanded from 12 to 14 steps (BITS + CCTS + expanded compliance + review)
- [x] BC config expanded: `bits`, `ccts`, `gckey`, `ats` (6 survey forms), `corporate_obligations` (7 tax/filing items)
- [ ] BC COLIN selector verification (Steps 5-12 need live session testing)
- [ ] Verify GCKey Steps 3-5 selectors on first live provisioning run
- [ ] Verify AMB operator_name extraction on next daily scraper run
- [ ] Test full CRTC order end-to-end on dev stack
- [ ] Deploy BITS/CCTS/compliance/renewal code to production
- [x] Add renewal cron — deployed via `worker-crons` ansible role (`pw-renewal-worker.timer` daily 04:00 UTC)
- [x] Add USF factor monitor cron — deployed via `worker-crons` ansible role (`pw-usf-factor-monitor.timer` daily 09:00 CT)
- Polls https://www.usac.org/service-providers/making-payments/contribution-factors/
- On detecting a new quarterly factor, emails all FCC-carrier customers (bcc justin@) with the new % and the delta vs. prior quarter so they can update their USF surcharges before the quarter starts
- Requires migration 049 (`usf_contribution_factors` table) to be applied
- [x] Create ERPNext Items for renewal invoicing: CRTC-MAINT-ANNUAL, MAILBOX-RENEWAL, BC-ANNUAL-REPORT, DOMAIN-RENEWAL-CA, COMPLIANCE-OTHER (fixture: `performancewest_erpnext/performancewest_erpnext/fixtures/item.json`; imports on `bench migrate`)
- [ ] Import updated Compliance Calendar DocType to production ERPNext
---
## PRIORITY 11 — Multi-Province, Canadian Formation & Universal Compliance
> Full plan: [`docs/multi-province-plan.md`](multi-province-plan.md) — 16 sub-phases, 28-36 hours, 5-6 sessions
**Three interconnected features:**
1. Multi-province CRTC ($3,899, Ontario first)
2. Standalone Canadian formation (C$449 + gov fees, separate flow from CRTC)
3. Universal compliance calendar (all order types: US + CA formation + CRTC)
- [ ] Phase 0: OBR + CRA BN Playwright recon (Ontario Business Registry + CRA Business Number)
- [ ] Phase 1: ProvinceConfig abstraction + universal compliance module design
- [ ] Phase 2: DB migrations (CRTC province column + formation entity types + country)
- [ ] Phase 3a-d: Ontario config, adapter, AMB scraper extension, Flowroute ON area codes
- [ ] Phase 3e: CanadianIncorporationHandler (shared pipeline for formation-only + CRTC)
- [ ] Phase 3f: CRA Business Number registration stub ($49 add-on)
- [ ] Phase 3g: Universal compliance entry creator (all order types)
- [ ] Phase 4: Service page province comparison table + FAQ rewrite
- [ ] Phase 5a: CRTC order form multi-province (province selector, dynamic content)
- [ ] Phase 5b: Formation page backend wiring (frontend already supports Canada)
- [ ] Phase 6a: CRTC API route (accept province)
- [ ] Phase 6b: CRTC pipeline refactor (compose with CanadianIncorporationHandler)
- [ ] Phase 6c: Formation API route (accept CA provinces + entity types ltd/inc/corp)
- [ ] Phase 7a: Canadian province compliance configs (BC + ON corporate obligations)
- [ ] Phase 7b: US state compliance configs (all 51 jurisdictions — annual report, franchise tax, RA)
- [ ] Phase 8: Dev E2E tests (CRTC ON + CA formation-only + US formation compliance)
**New products:**
- CA Formation: C$449 + gov fees (incorp + minutes + binder + compliance calendar). AMB mailbox separate.
- CRA BN add-on: $49
- Formation Maintenance Bundle (US or CA): $179/yr (annual report $99 + RA renewal $99)
---
## PRIORITY 12 — Sales & Marketing
- [ ] Configure Google Analytics 4 property and connect to Umami export
- [ ] Set up Google Search Console for performancewest.net
- [ ] Set up Google Ads account (future)
- [ ] Create referral program landing page (/agents)
- [ ] Onboard first 3 sales agents (test referral code flow end-to-end)
- [ ] Create Trustpilot review request automation in post-delivery email
---
## PRIORITY 13 — Content & Video
- [ ] Generate AI-narrated video for Canada CRTC service page
- [ ] Generate marketing videos for US formation services
- [ ] Create vendor directory PDF for CRTC clients (5 categories, delivered via portal)
- [ ] Create US fintech banking guide PDF for CRTC clients (Wise, Airwallex, Payoneer)
---
## PRIORITY 14 — MCP Server & AI Integration
- [ ] Publish MCP server package to npm (`@performancewest/mcp`)
- [ ] Test all 10 MCP tools against live ERPNext
- [ ] Document MCP server usage for AI agent integration
---
## PRIORITY 15 — Client Portal & Testing
- [x] Portal order status page (portal.performancewest.net/orders) — visual 10-step pipeline
- [x] eSign page (/portal/sign) — canvas signature, letter preview, JWT auth
- [x] Client selection page (/portal/setup) — unit picker, DID picker, confirm
- [x] Domain search page (/portal/domain-search)
- [x] Manage services page (/portal/manage-services)
- [ ] Phase 7: SupportWidget → ERPNext Issue direct (remove PG tickets fallback)
- [ ] Phase 8: Deprecate PG customer auth (portal-auth.ts login/register → ERPNext portal only)
- [ ] Phase 9: E2E test — order → payment → invoice → portal login → order status → ticket
- [ ] Load test: simulate 10 concurrent CRTC orders through the full pipeline
- [ ] Mobile test all order form steps on iOS Safari + Android Chrome
---
## PRIORITY 16 — Environment Variables Still Needed
- [ ] `STRIPE_SECRET_KEY` — from Stripe Dashboard (live key)
- [ ] `STRIPE_WEBHOOK_SECRET` — from Stripe Dashboard → Webhooks endpoint
- [ ] `STRIPE_IDENTITY_WEBHOOK_SECRET` — from Stripe Dashboard (identity webhook)
- [x] `CUSTOMER_JWT_SECRET` — generated and set (M0IFZv9ipOxU5juZnf7zwdTfpk_5_5...)
- [x] `SMTP_PASS` — set ($m)}4G8&!yu;{)#]sp, Carbonio noreply@)
- [ ] `SHKEEPER_API_KEY` — from SHKeeper admin panel at crypto.performancewest.net
- [ ] `PORKBUN_API_KEY` / `PORKBUN_SECRET_KEY` — from Porkbun account API settings
- [ ] `FLOWROUTE_ACCESS_KEY` / `FLOWROUTE_SECRET_KEY` — from Flowroute account
- [ ] `HESTIA_SSH_KEY` — path to SSH private key on workers container (mount via volume)
- [ ] `ANYTIME_MAILBOX_IMAP_PASS` — mailbox password for IMAP OTP fetch
- [ ] `PAYPAL_CLIENT_ID` / `PAYPAL_CLIENT_SECRET` — from PayPal developer dashboard
- [ ] `MINIO_ACCESS_KEY` / `MINIO_SECRET_KEY` — already set (verify on server)
- [x] Windows DocServer MinIO credentials — configured in `C:\docserver\docserver.env` on VM
---
## PRIORITY 17 — Low Priority / When Revenue Supports
- [ ] Adyen merchant account application (card/ACH/Klarna/CashApp/AmazonPay via Adyen)
- [ ] Stripe Issuing cardholder setup for SID-0002 (Relay is interim filing card)
- [ ] Automated annual report filing for existing clients
- [ ] Attorney review integration (removed from site; may re-add as premium tier)
- [ ] Multi-language support (French for Quebec clients)
- [ ] Mobile app (React Native) for client portal access
- [ ] Wholesale API for resellers and referring attorneys
---
## POST-LAUNCH MONITORING
- [ ] Set up UptimeRobot (or similar) monitors for:
- https://performancewest.net (site)
- https://api.performancewest.net/health (API)
- https://portal.performancewest.net (ERPNext portal)
- https://lists.performancewest.net (Listmonk)
- SHKeeper /health endpoint
- [ ] Set up Grafana + Prometheus for container metrics (future)
- [ ] Set up PagerDuty or similar for on-call alerts
- [ ] Review Umami weekly dashboard every Monday
- [ ] Review ERPNext open Issues weekly (customer support queue)
- [ ] Review abandoned cart report weekly (payment_reminder.py logs)
- [ ] Monthly: review state filing fee table for changes
- [ ] Monthly: review AMB location pricing (daily scraper catches this automatically)
- [ ] Quarterly: rotate API keys (Stripe, SHKeeper, Porkbun, Flowroute)