justin/new-site
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
new-site/docs/e2e-test-plan.md
justin f8cd37ac8c Initial commit — Performance West telecom compliance platform 
Includes: API (Express/TypeScript), Astro site, Python workers,
document generators, FCC compliance tools, Canada CRTC formation,
Ansible infrastructure, and deployment scripts.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-27 06:54:22 -05:00

179 lines
7.4 KiB
Markdown
Raw Blame History

# CRTC Pipeline E2E Test Plan
**Created:** 2026-04-06
**Environment:** Dev stack (site=4323, api=3002, postgres=5433, shared ERPNext/MinIO)
**Scope:** Post-payment pipeline only (order already submitted and paid)
**Vendor strategy:** Test/sandbox APIs where available, mock where not
**eSign:** Inject base64 signature via API
**Test output:** Playwright screenshots at each checkpoint + console report
---
## Vendor Mock/Sandbox Strategy
| Vendor | Strategy | Details |
|--------|----------|---------|
| BC Registry (COLIN) | **Mock** — no sandbox | Patch to return fake BC# `BC1234567` |
| Flowroute DID | **Test mode** | `FLOWROUTE_TEST=true`, get test DID |
| Porkbun .ca | **Mock** — no sandbox | Patch to return `test-e2e-{uuid}.ca` |
| Anytime Mailbox | **Mock** — costs money | Patch to return fake AMB unit ID |
| HestiaCP | **Mock** — don't create real accounts | Patch to return success |
| GCKey | **Mock** — no gov accounts | Patch to return fake credentials |
| SMTP | **Capture to file** | `DRY_RUN_EMAIL=true` or Mailhog |
| MinIO | **Real** | Dev MinIO shared with prod |
| DocServer | **Real** | Windows VM (or LibreOffice fallback) |
| ERPNext | **Real** | Shared instance, test SO cleaned up after |
---
## Phases
### Phase 0: Test Harness (Opus 4.6)
Write `scripts/tests/e2e_crtc_pipeline.py`:
- Playwright + requests + psycopg2 orchestration script
- Connects to dev PG, dev API, dev workers, ERPNext
- Creates test data, runs pipeline, screenshots, verifies, cleans up
- Screenshots to `scripts/tests/screenshots/`
### Phase 1: Create Test Order (Sonnet 4.6)
- Insert PG `canada_crtc_orders` with `payment_status=paid`, `funds_available=TRUE`
- Create ERPNext SO at `"Client Selection"` state
- **Screenshot 1:** ERPNext SO detail page
### Phase 2: Steps 1-4 — Incorporation (Sonnet 4.6)
- Pre-populate mocked vendor results in PG (BC#, DID, AMB)
- Advance ERPNext SO workflow through Steps 1-4
- **Screenshot 2:** ERPNext SO with BC#/DID/AMB fields
- **Screenshot 3:** PG query results
### Phase 3: Step 5 — Domain (Sonnet 4.6)
- Pre-populate domain in PG, advance workflow to `"Domain Ready"`
- **Screenshot 4:** ERPNext SO domain fields
### Phase 4: Step 6 — CRTC Letter DOCX/PDF (Opus 4.6)
- Trigger `generate_crtc_docs` job on worker
- Verify DOCX: entity name, BC#, 5 sections, signature block
- Verify PDF: MinIO object exists, valid PDF header, >0 bytes
- Verify DocServer heartbeat (or LibreOffice fallback noted)
- **Screenshot 5:** MinIO console — PDF listing
- **Screenshot 6:** ERPNext SO — `"Awaiting eSign"` state
### Phase 5: Step 6b — eSign (Sonnet 4.6)
- Generate JWT token for test order
- Screenshot eSign portal page
- POST inject base64 signature
- Verify PG: `esign_signed_at`, ERPNext SO: `"CRTC Submitted"`
- **Screenshot 7:** eSign portal page
- **Screenshot 8:** ERPNext SO post-eSign
### Phase 6: Steps 7-10 — Binder + Delivery (Sonnet 4.6)
- Wait for `resume_crtc_pipeline` job completion
- Verify binder: MinIO object, valid PDF, multi-page
- Verify delivery email (Mailhog or file capture)
- Verify banking referral sent
- **Screenshot 9:** MinIO binder PDF
- **Screenshot 10:** Delivery email
- **Screenshot 11:** ERPNext SO `"Banking Ready"`
### Phase 7: Steps 11-13 — BITS/CCTS/Compliance (Sonnet 4.6)
- Wait for pipeline to reach `"Ready for Review"`
- Verify BITS: ToDo exists, `custom_bits_filed_at` set
- Verify CCTS: ToDo exists, `custom_ccts_filed_at` set
- Verify Compliance Calendar: 12+ entries with correct dates/amounts
- **Screenshot 12:** Compliance Calendar list
- **Screenshot 13:** ToDo list (BITS + CCTS)
- **Screenshot 14:** ERPNext SO final state
### Phase 8: Cleanup (Sonnet 4.6)
- Delete PG test rows, ERPNext SO/Calendar/ToDo, MinIO objects
- Print cleanup report
---
## Screenshot Manifest (14 screenshots)
| # | Filename | Source | Verifies |
|---|----------|--------|----------|
| 1 | `01-so-client-selection.png` | ERPNext SO | Initial post-payment state |
| 2 | `02-so-incorporation-complete.png` | ERPNext SO | BC#, DID, AMB populated |
| 3 | `03-pg-order-fields.png` | Terminal PG | Raw DB row |
| 4 | `04-so-domain-ready.png` | ERPNext SO | Domain + HestiaCP |
| 5 | `05-minio-crtc-letter.png` | MinIO console | Letter PDF exists |
| 6 | `06-so-awaiting-esign.png` | ERPNext SO | Awaiting eSign state |
| 7 | `07-esign-portal-page.png` | Browser | eSign page with preview |
| 8 | `08-so-crtc-submitted.png` | ERPNext SO | Post-eSign state |
| 9 | `09-minio-binder.png` | MinIO console | Binder PDF exists |
| 10 | `10-delivery-email.png` | Mailhog/file | Client delivery email |
| 11 | `11-so-banking-ready.png` | ERPNext SO | Banking referral sent |
| 12 | `12-compliance-calendar.png` | ERPNext list | 12+ compliance entries |
| 13 | `13-todos-bits-ccts.png` | ERPNext list | BITS + CCTS ToDos |
| 14 | `14-so-ready-for-review.png` | ERPNext SO | Final state all fields |
---
## Model Assignment
| Phase | Model | Why |
|-------|-------|-----|
| 0: Test harness | **Opus 4.6** | Complex architecture — mocks, Playwright, error handling |
| 1: Create order | Sonnet 4.6 | Mechanical DB + API calls |
| 2: Steps 1-4 | Sonnet 4.6 | Pre-populate mock data |
| 3: Step 5 | Sonnet 4.6 | Pre-populate mock data |
| 4: Step 6 DOCX/PDF | **Opus 4.6** | Critical — DOCX structure + PDF conversion verification |
| 5: eSign | Sonnet 4.6 | API inject + verify |
| 6: Steps 7-10 | Sonnet 4.6 | File + email verification |
| 7: Steps 11-13 | Sonnet 4.6 | ERPNext entry verification |
| 8: Cleanup | Sonnet 4.6 | Mechanical deletes |
**Total effort:** 8-10 hours (1-2 sessions)
---
## Test Results (2026-04-06)
### Run Summary
| Phase | Result | Time | Details |
|-------|--------|------|---------|
| 1 | PASS | <1s | PG order + ERPNext SO created + submitted + workflow advanced (Received → Mailbox Ready) |
| 2-3 | PASS | <1s | Mock BC# BC1234567, DID +16045551234, domain, AMB |
| 4 | PASS | 2s | DOCX 37.6KB (5/5 content checks), PDF 42KB (LibreOffice), uploaded to MinIO |
| 5 | PASS | 3s | eSign page screenshot captured, JWT simulated in PG |
| 6 | PASS | 10s | PDF verified in MinIO (42KB, valid header) |
| 7 | PASS | 1s | ERPNext SO at "Mailbox Ready" state, no compliance entries (expected) |
| 7b | PASS | 30s | eSign screenshot captured, ERPNext login times out (Docker network) |
| **Total** | **ALL PASS** | **48s** | |
### Binder Compilation (Separate Test)
| Test | Result | Details |
|------|--------|---------|
| DOCX generation | PASS | 37.5KB |
| LibreOffice PDF | PASS | 41.4KB |
| Binder compilation | PASS | 37.3KB, **5 pages** (cover + TOC + divider + letter content) |
### DocServer Investigation
Word COM fails under SYSTEM account and "Run whether user is logged on or not" mode.
Requires interactive desktop session (RDP login). Auto-logon configured (registry keys set)
but blocked by hosting provider's Windows Server 2019 policy.
**Workaround:** RDP into the VM once after reboot → AtLogOn trigger fires → Word COM works.
LibreOffice fallback handles conversions automatically when DocServer is unavailable.
### Known Limitations
1. **DocServer** — requires RDP login after cold reboot (auto-logon blocked by hosting provider)
2. **eSign JWT** — test uses different secret than dev API; falls back to PG simulation
3. **Compliance Calendar** — DocType not imported to ERPNext; 417 error on query
4. **ERPNext screenshots** — Playwright can't log into ERPNext from Docker (login page structure)
5. **Full pipeline** — individual components tested; full 14-step pipeline needs ERPNext workflow + all DocTypes imported
Reference in a new issue View git blame Copy permalink