justin/new-site
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
new-site/docs/go-live-todo.md
justin f8cd37ac8c Initial commit — Performance West telecom compliance platform 
Includes: API (Express/TypeScript), Astro site, Python workers,
document generators, FCC compliance tools, Canada CRTC formation,
Ansible infrastructure, and deployment scripts.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-27 06:54:22 -05:00

21 KiB
Raw Blame History

Performance West — Go-Live Deployment Checklist

Last updated: 2026-04-05

PRIORITY 1 — Infrastructure COMPLETE

  • Provision Proxmox Linux VM: Debian 13, 32GB RAM, 8 vCPU, 232GB SSD
  • Point DNS: performancewest.net, api., portal., crm., lists., analytics., pay., crypto., minio., minio-console., dev., api.dev. → 207.174.124.71
  • Run Ansible site.yml playbook to provision full stack
  • Run all DB migrations (001–035) against PostgreSQL
  • Set up Let's Encrypt TLS certificates for all 12 subdomains
  • Configure firewall (UFW): allow 80, 443, SSH 22022 only + trusted IPs
  • Configure fail2ban: sshd, nginx-badbots, pw-api jails
  • Set up performancewest.service systemd unit for auto-start on reboot
  • Install k3s with --docker --disable=traefik
  • Install Helm 3
  • Set up dev.performancewest.net / api.dev.performancewest.net dev stack (ports 4323/3002)
  • Provision Windows VM for DocServer (108.181.102.34, Office 365 Word COM, MinIO transport)
    • SSH: port 22422 (key auth), Python 3.13 + pywin32 + minio SDK
    • Task Scheduler: PW-DocserverWorker (auto-start, auto-restart on failure)
    • Private network: 10.4.20.247 → MinIO via nginx
  • RAM upgrade: 62GB RAM verified, all 15 containers UP

PRIORITY 2 — ERPNext + Integrations COMPLETE

  • Configure ERPNext: create site, install erpnext + payments apps
  • Run setup wizard (Company: Performance West Inc., USD, America/Chicago)
  • Build custom ERPNext image with frappe_crypto + frappe_adyen + performancewest_erpnext
  • Install 6 apps: frappe, erpnext, payments, frappe_crypto, frappe_adyen, performancewest_erpnext
  • Import 7 custom DocTypes: Formation Order, Compliance Calendar, Sensitive ID, Referral Partner, Compliance Service, Sales Agent, Commission Ledger
  • Import 3 workflows: Formation Order (18 states), Canada CRTC (30 steps), Renewal
  • Create 16 service Items (CRTC Package $3,899, LLC Basic $179, etc.)
  • Update Subscription Plans to new pricing (RA $99/yr ($49 WY), Annual Report $99/yr, CRTC Maintenance $349/yr)
  • Update ERPNext Item rates: LLC Basic $179 (was $149), RA $99 (was $125), add WY RA $49
  • Create new Items: CA Formation C$449, Formation Maintenance $179/yr, Free DID (stub)
  • Add custom fields: Sales Order (30+ fields incl. identity, esign, binder, regulatory)
  • Add custom fields: Sales Invoice (surcharge_pct, payment_gateway, stripe_payment_intent)
  • Hide 18 irrelevant modules for all users
  • Generate API keys (ERPNEXT_API_KEY / ERPNEXT_API_SECRET)
  • Configure ERPNext outgoing SMTP (co.carrierone.com:587, noreply@performancewest.net)
  • Configure ERPNext incoming email (support@performancewest.net → auto-create Issue)
  • Import 18 ERPNext webhooks: 11 CRTC + 7 Formation (all pointing to Express API)
  • Create ERPNext roles: Sales Agent (no desk), Accounting Advisor (desk access)
  • Import 20 compliance service fixtures (compliance_services.json)
  • Configure ERPNext Assignment Rule: Accounting Support → Accounting Advisor role
  • Portal Settings: enable Sales Order, Sales Invoice, Issue for customer portal
  • Website Settings: PW branding, navy theme, custom CSS
  • Create portal.performancewest.net DNS + SSL + nginx vhost → ERPNext :8080
  • 6 CRTC workflow notification emails (Received → Delivered) using Carbonio SMTP
  • Configure ERPNext incoming email IMAP password for support@ account
  • Create ERPNext Payment Gateway Account for Crypto-Crypto (SHKeeper)
  • Configure Crypto Payment Settings in ERPNext UI (pay.performancewest.net + SHKeeper API key)
  • Configure ERPNext Assignment Rules: expand for CRTC ticket routing

PRIORITY 3 — Payment Processing

  • Deploy SHKeeper via Helm in k3s (7 crypto daemons: BTC, ETH, MATIC, BNB, TRX, LTC, DOGE)
  • Set up SHKeeper nginx reverse proxy at pay.performancewest.net + crypto.performancewest.net
  • Build frappe_crypto app (SHKeeper gateway, checkout page, webhook receiver)
  • Rewrite checkout.ts: Stripe Checkout Sessions (card + ACH + Klarna), PayPal Orders v2, SHKeeper crypto
  • Add Stripe webhook handler — events: checkout.session.completed, payment_intent.succeeded, balance.available
  • Add SHKeeper webhook handler for crypto payment confirmation
  • Add PayPal capture / webhook handler
  • Order confirmation email on payment (Carbonio SMTP via email.ts)
  • Auto-create Sales Invoice + Payment Entry on payment (createInvoiceFromSalesOrder)
  • Fund detection: balance.available webhook → T+2/T+4 business day advance → Stripe Issuing topup
  • Abandoned cart recovery (payment_reminder.py cron every 5 min, 15min/1d/2d intervals)
  • Payment method selector on order form Step 5 (ACH recommended, expedited → PayPal/crypto only)
  • Expedited processing toggle (+$500, restricts to PayPal/crypto only)
  • Set STRIPE_SECRET_KEY + STRIPE_WEBHOOK_SECRET + STRIPE_IDENTITY_WEBHOOK_SECRET in .env
  • Register Stripe webhook at api.performancewest.net/api/v1/webhooks/stripe
    • Events: checkout.session.completed, payment_intent.succeeded, balance.available, identity.verification_session.verified
  • Set SHKEEPER_API_KEY in .env
  • Test end-to-end Stripe card payment flow
  • Test end-to-end Stripe ACH payment flow
  • Test end-to-end PayPal payment flow
  • Test end-to-end crypto payment flow
  • Adyen: apply for merchant account (future — not blocking launch)

PRIORITY 4 — Banking & Financial

  • Set up Relay virtual debit card (SID-0002) dedicated to filing fees
  • Store Relay card details in ERPNext Sensitive ID (encrypted)
  • Set up Relay checking account structure (Operating, Filing Fees, Profit, Tax, Commissions, Owner Pay)
  • Store PayPal Mastercard (SID-0001) details in ERPNext Sensitive ID
  • Change NW RA portal password (was shared in chat — URGENT)
  • Create Northwest Registered Agent wholesale orders for each state's RA address

PRIORITY 5 — Document Generation & Workers

  • Set up MinIO: create performancewest bucket, configure access
  • Upload 9 DOCX templates to MinIO (to-convert/ → converted/ transport buckets created)
  • Configure Ollama with qwen2.5:7b model
  • Install Playwright browsers (chromium) in worker container
  • Add PyJWT to scripts/requirements.txt (for portal eSign JWT generation)
  • DocServer: MinIO-based transport (docserver_worker.py polls to-convert/, drops converted/)
    • Worker polls every 12s, heartbeat every 60s at minio://performancewest/docserver-heartbeat.json
    • LibreOffice headless fallback auto-activates when DocServer unavailable
  • Test LibreOffice fallback DOCX-to-PDF conversion (verify manually)
  • Provision Windows VM, run docserver/install.ps1 with MinIO credentials
  • Verify Word COM conversion end-to-end (36KB DOCX → 82KB PDF, 12s round-trip)
  • MinIO nginx allow-list updated with private network IP 10.4.20.247
  • ERPNext CSS fix after reboot (chmod o+x /var/lib/docker + bench build --force)
  • ERPNext "Sent via ERPNext" email footer disabled (disable_standard_email_footer: 1)

PRIORITY 6 — External Accounts & Services

  • Claim Google Business Profile at business.google.com for Performance West Inc.
  • Claim Trustpilot business profile at business.trustpilot.com
  • Update Google Reviews link in homepage with actual Google Place ID
  • Create SOSDirect account for Texas state filings
  • Create UtahID account for Utah state filings
  • Sign up for 2captcha or anticaptcha (Delaware CAPTCHA solving)
  • Set up Porkbun account for .ca domain registration (CRTC service)
  • Set PORKBUN_API_KEY / PORKBUN_SECRET_KEY in .env
  • Set up Flowroute account for Canadian DID provisioning
  • Set FLOWROUTE_ACCESS_KEY / FLOWROUTE_SECRET_KEY in .env
  • Configure HestiaCP on cp.carrierone.com for .ca domain hosting + email
  • Set HESTIA_SSH_KEY path in workers container .env (key must be mounted or baked)
  • Test HestiaCP provisioner: python -m scripts.workers.hestia_provisioner provision test.ca "Test Company"
  • Verify all 14 mailbox types create correctly per domain
  • Set up Anytime Mailbox wholesale/partner account
  • Configure ANYTIME_MAILBOX_IMAP_PASS for OTP auto-fetch
  • Configure client email processor cron (every 15 min): deployed via worker-crons ansible role (pw-client-email-processor.timer)

PRIORITY 7 — Listmonk (Email Marketing) COMPLETE

  • Deploy Listmonk at lists.performancewest.net (Docker, PostgreSQL-backed)
  • Configure SMTP2GO outbound email (separate from Carbonio transactional SMTP)
  • Import 10,191 FCC RMD contacts into Listmonk (3 subscriber lists)
  • Configure bounce processing via POP3 from Carbonio (bounces@performancewest.net)
  • Create 22 scheduled campaigns across 4 lists
  • nginx vhost at lists.performancewest.net with Let's Encrypt cert
  • Remove RMD/Robocall Mitigation Database language from all campaigns

PRIORITY 8 — Analytics & Security

  • Deploy Umami analytics container at analytics.performancewest.net
  • Umami first-run config: site registered, site ID 55250014-ee15-44ac-a1f6-81dabad3fe0f
  • Add Umami tracking script to Base.astro (!isDev guard, production only)
  • nginx CSP includes analytics.performancewest.net in script-src + connect-src
  • CAA DNS records for Let's Encrypt
  • HSTS headers on all domains
  • fail2ban pw-api jail (rate-limit abusers at /api/v1/checkout)
  • Verify SSL Labs A+ grade after deployment
  • Verify SecurityHeaders.com A+ grade
  • Verify Mozilla Observatory A+ grade
  • Verify Google Safe Browsing clean status

PRIORITY 9 — State Formation Automation

  • Populate CSS selectors for top 10 states (WY, CO, DE, FL, TX, NV, UT, NM, OH, MT)
  • Test Wyoming name search end-to-end via Playwright
  • Test Colorado name search via Socrata API (confirmed working in isolation)
  • Test full formation order flow: order → name search → payment → filing
  • Implement ~40 remaining state adapters (only ~12 have real implementations)

PRIORITY 10 — Canada CRTC Service

  • CRTC order form (5-step, identity verification, AMB location picker, payment methods)
  • Stripe Identity gate (Step 4, auto-advance after verified, form snapshot persistence)
  • AMB location scraper (daily cron, operator_name extraction, sold-out deactivation)
  • Anytime Mailbox signup automation (Playwright + IMAP OTP from Carbonio)
  • Flowroute DID search + purchase (area codes 604/778/236/250)
  • .ca domain registration via Porkbun (CIRA CPR: BC corp number, category CCO)
  • HestiaCP provisioner: 14 mailboxes per domain including regulatory@, abuse@, postmaster@
    • Credentials stored in ERPNext Sensitive ID (encrypted)
    • Email 1 (brief): domain live notification
    • Email 2 (credentials): all 14 passwords, IMAP/SMTP settings, webmail URL
  • BC incorporation via COLIN (anonymous filing, no login required)
  • CRTC notification letter generation (DOCX template → MinIO transport → Word/LibreOffice PDF)
  • eSign portal page (/portal/sign) — canvas drawing pad, letter preview iframe, JWT auth
    • Pipeline pauses at Step 6b, emails client JWT-signed sign link (72h expiry)
    • Signature stored as base64 PNG in PG (esign_signature_b64)
    • On submit: ERPNext workflow advances to "CRTC Submitted", resume_crtc_pipeline job dispatched
  • Corporate binder compilation (PDF merge: certificate + articles + CRTC letter + OA)
  • Binder delivery email: FROM regulatory@domain.ca (HestiaCP SMTP credentials)
    • AMB binder label: c/o <operator_name>, unit number
    • Own-address binder label: company name + attn contact
  • Admin print/ship email (PirateShip USPS label instructions)
  • Client portal order status page at portal.performancewest.net/orders
    • 10-step visual pipeline with colored chips (completed/active/pending)
    • Action CTAs: "Complete Setup →" (Client Selection), "Sign CRTC Letter →" (Pending eSign)
    • Invoice table with Paid/Partial/Unpaid badges
  • Phase 2 customer auth: ERPNext Website User created on order, portal password set from success page
  • Phase 4 auto-invoice: Sales Invoice + Payment Entry created on payment
  • Phase 5 ERPNext notifications: 6 workflow state emails (Received → Delivered) via Carbonio
  • Phase 6 custom portal /orders page in performancewest_erpnext Frappe app
  • BITS registration step (Step 11) — GCKey provisioning via Playwright + admin ToDo for BITS filing
    • GCKey signup wizard automated: 5-step flow, hCaptcha handling, credentials to ERPNext Sensitive ID
    • Username format: pw-{bc_number}, recovery email: regulatory@domain.ca
  • CCTS registration step (Step 12) — admin ToDo + client obligations email
  • Compliance calendar auto-creation (Step 13) — 17 entries per carrier
    • Regulatory: BC annual report, CRTC maintenance, mailbox, domain, DID, CCTS, CRTC update
    • Tax: T2, corporate tax, GST/HST, T4/T4A, BC PST, WorkSafeBC
    • Surveys: REP-T/T1 (mandatory), REP-U/802a/802j/Facilities/Pricing (>$10M threshold)
  • Compliance calendar renewal lifecycle (renewal_worker.py)
    • Daily cron: upcoming → due soon → invoice sent → paid → completed → re-calendar
    • Billable items generate ERPNext Sales Invoice; payment required before completion
    • Webhook handler: handle_renewal_payment in job_server.py
  • CRTC pipeline expanded from 12 to 14 steps (BITS + CCTS + expanded compliance + review)
  • BC config expanded: bits, ccts, gckey, ats (6 survey forms), corporate_obligations (7 tax/filing items)
  • BC COLIN selector verification (Steps 5-12 need live session testing)
  • Verify GCKey Steps 3-5 selectors on first live provisioning run
  • Verify AMB operator_name extraction on next daily scraper run
  • Test full CRTC order end-to-end on dev stack
  • Deploy BITS/CCTS/compliance/renewal code to production
  • Add renewal cron — deployed via worker-crons ansible role (pw-renewal-worker.timer daily 04:00 UTC)
  • Add USF factor monitor cron — deployed via worker-crons ansible role (pw-usf-factor-monitor.timer daily 09:00 CT)
  • Create ERPNext Items for renewal invoicing: CRTC-MAINT-ANNUAL, MAILBOX-RENEWAL, BC-ANNUAL-REPORT, DOMAIN-RENEWAL-CA, COMPLIANCE-OTHER (fixture: performancewest_erpnext/performancewest_erpnext/fixtures/item.json; imports on bench migrate)
  • Import updated Compliance Calendar DocType to production ERPNext

PRIORITY 11 — Multi-Province, Canadian Formation & Universal Compliance

Full plan: docs/multi-province-plan.md — 16 sub-phases, 28-36 hours, 5-6 sessions

Three interconnected features:

  1. Multi-province CRTC ($3,899, Ontario first)
  2. Standalone Canadian formation (C$449 + gov fees, separate flow from CRTC)
  3. Universal compliance calendar (all order types: US + CA formation + CRTC)
  • Phase 0: OBR + CRA BN Playwright recon (Ontario Business Registry + CRA Business Number)
  • Phase 1: ProvinceConfig abstraction + universal compliance module design
  • Phase 2: DB migrations (CRTC province column + formation entity types + country)
  • Phase 3a-d: Ontario config, adapter, AMB scraper extension, Flowroute ON area codes
  • Phase 3e: CanadianIncorporationHandler (shared pipeline for formation-only + CRTC)
  • Phase 3f: CRA Business Number registration stub ($49 add-on)
  • Phase 3g: Universal compliance entry creator (all order types)
  • Phase 4: Service page province comparison table + FAQ rewrite
  • Phase 5a: CRTC order form multi-province (province selector, dynamic content)
  • Phase 5b: Formation page backend wiring (frontend already supports Canada)
  • Phase 6a: CRTC API route (accept province)
  • Phase 6b: CRTC pipeline refactor (compose with CanadianIncorporationHandler)
  • Phase 6c: Formation API route (accept CA provinces + entity types ltd/inc/corp)
  • Phase 7a: Canadian province compliance configs (BC + ON corporate obligations)
  • Phase 7b: US state compliance configs (all 51 jurisdictions — annual report, franchise tax, RA)
  • Phase 8: Dev E2E tests (CRTC ON + CA formation-only + US formation compliance)

New products:

  • CA Formation: C$449 + gov fees (incorp + minutes + binder + compliance calendar). AMB mailbox separate.
  • CRA BN add-on: $49
  • Formation Maintenance Bundle (US or CA): $179/yr (annual report $99 + RA renewal $99)

PRIORITY 12 — Sales & Marketing

  • Configure Google Analytics 4 property and connect to Umami export
  • Set up Google Search Console for performancewest.net
  • Set up Google Ads account (future)
  • Create referral program landing page (/agents)
  • Onboard first 3 sales agents (test referral code flow end-to-end)
  • Create Trustpilot review request automation in post-delivery email

PRIORITY 13 — Content & Video

  • Generate AI-narrated video for Canada CRTC service page
  • Generate marketing videos for US formation services
  • Create vendor directory PDF for CRTC clients (5 categories, delivered via portal)
  • Create US fintech banking guide PDF for CRTC clients (Wise, Airwallex, Payoneer)

PRIORITY 14 — MCP Server & AI Integration

  • Publish MCP server package to npm (@performancewest/mcp)
  • Test all 10 MCP tools against live ERPNext
  • Document MCP server usage for AI agent integration

PRIORITY 15 — Client Portal & Testing

  • Portal order status page (portal.performancewest.net/orders) — visual 10-step pipeline
  • eSign page (/portal/sign) — canvas signature, letter preview, JWT auth
  • Client selection page (/portal/setup) — unit picker, DID picker, confirm
  • Domain search page (/portal/domain-search)
  • Manage services page (/portal/manage-services)
  • Phase 7: SupportWidget → ERPNext Issue direct (remove PG tickets fallback)
  • Phase 8: Deprecate PG customer auth (portal-auth.ts login/register → ERPNext portal only)
  • Phase 9: E2E test — order → payment → invoice → portal login → order status → ticket
  • Load test: simulate 10 concurrent CRTC orders through the full pipeline
  • Mobile test all order form steps on iOS Safari + Android Chrome

PRIORITY 16 — Environment Variables Still Needed

  • STRIPE_SECRET_KEY — from Stripe Dashboard (live key)
  • STRIPE_WEBHOOK_SECRET — from Stripe Dashboard → Webhooks endpoint
  • STRIPE_IDENTITY_WEBHOOK_SECRET — from Stripe Dashboard (identity webhook)
  • CUSTOMER_JWT_SECRET — generated and set (M0IFZv9ipOxU5juZnf7zwdTfpk_5_5...)
  • SMTP_PASS — set ($m)}4G8&!yu;{)#]sp, Carbonio noreply@)
  • SHKEEPER_API_KEY — from SHKeeper admin panel at crypto.performancewest.net
  • PORKBUN_API_KEY / PORKBUN_SECRET_KEY — from Porkbun account API settings
  • FLOWROUTE_ACCESS_KEY / FLOWROUTE_SECRET_KEY — from Flowroute account
  • HESTIA_SSH_KEY — path to SSH private key on workers container (mount via volume)
  • ANYTIME_MAILBOX_IMAP_PASS — mailbox password for IMAP OTP fetch
  • PAYPAL_CLIENT_ID / PAYPAL_CLIENT_SECRET — from PayPal developer dashboard
  • MINIO_ACCESS_KEY / MINIO_SECRET_KEY — already set (verify on server)
  • Windows DocServer MinIO credentials — configured in C:\docserver\docserver.env on VM

PRIORITY 17 — Low Priority / When Revenue Supports

  • Adyen merchant account application (card/ACH/Klarna/CashApp/AmazonPay via Adyen)
  • Stripe Issuing cardholder setup for SID-0002 (Relay is interim filing card)
  • Automated annual report filing for existing clients
  • Attorney review integration (removed from site; may re-add as premium tier)
  • Multi-language support (French for Quebec clients)
  • Mobile app (React Native) for client portal access
  • Wholesale API for resellers and referring attorneys

POST-LAUNCH MONITORING

  • Set up UptimeRobot (or similar) monitors for:
  • Set up Grafana + Prometheus for container metrics (future)
  • Set up PagerDuty or similar for on-call alerts
  • Review Umami weekly dashboard every Monday
  • Review ERPNext open Issues weekly (customer support queue)
  • Review abandoned cart report weekly (payment_reminder.py logs)
  • Monthly: review state filing fee table for changes
  • Monthly: review AMB location pricing (daily scraper catches this automatically)
  • Quarterly: rotate API keys (Stripe, SHKeeper, Porkbun, Flowroute)