b48d0cb799
Companion to the worker MinIO-retry fix. Makes the worker auto-recover from process death (crash, manual kill, missed boot trigger), not just MinIO outages. - start_worker.bat: propagate Python's exit code (exit /b %rc%) so Task Scheduler can actually detect a failed run (it previously always exited 0). - reconfigure_task.ps1 (new): re-registers PW-DocserverWorker with RestartCount=99 / 1-min interval, StartWhenAvailable, and two triggers — AtStartup plus a 5-min repeating trigger with MultipleInstances=IgnoreNew, so a dead worker relaunches within ~5 min and never double-runs. Idempotent. - install.ps1: same self-healing settings for fresh installs. - Verified on the box: killed the worker -> task relaunched it; firing again while running stayed at one instance. Docs updated to match reality: - docserver/README.md: new 'Reliability / self-healing' section. - document-generation.md: corrected the stale 'Flask DocServer :5050 / HTTP' description to the actual MinIO outbound-only transport. - e2e-test-plan.md: removed the outdated 'Word COM fails under SYSTEM / requires RDP after every reboot' limitation; now self-healing under SYSTEM session 0. - infrastructure.md: fixed VM spec (Win Server 2019, Word 16.0, Python 3.13, SSH port 22422) + self-healing note. - architecture.md / formation-system.md: trigger + self-healing details.
188 lines
7.9 KiB
Markdown
188 lines
7.9 KiB
Markdown
# CRTC Pipeline E2E Test Plan
|
||
|
||
**Created:** 2026-04-06
|
||
**Environment:** Dev stack (site=4323, api=3002, postgres=5433, shared ERPNext/MinIO)
|
||
**Scope:** Post-payment pipeline only (order already submitted and paid)
|
||
**Vendor strategy:** Test/sandbox APIs where available, mock where not
|
||
**eSign:** Inject base64 signature via API
|
||
**Test output:** Playwright screenshots at each checkpoint + console report
|
||
|
||
---
|
||
|
||
## Vendor Mock/Sandbox Strategy
|
||
|
||
| Vendor | Strategy | Details |
|
||
|--------|----------|---------|
|
||
| BC Registry (COLIN) | **Mock** — no sandbox | Patch to return fake BC# `BC1234567` |
|
||
| Flowroute DID | **Test mode** | `FLOWROUTE_TEST=true`, get test DID |
|
||
| Porkbun .ca | **Mock** — no sandbox | Patch to return `test-e2e-{uuid}.ca` |
|
||
| Anytime Mailbox | **Mock** — costs money | Patch to return fake AMB unit ID |
|
||
| HestiaCP | **Mock** — don't create real accounts | Patch to return success |
|
||
| GCKey | **Mock** — no gov accounts | Patch to return fake credentials |
|
||
| SMTP | **Capture to file** | `DRY_RUN_EMAIL=true` or Mailhog |
|
||
| MinIO | **Real** | Dev MinIO shared with prod |
|
||
| DocServer | **Real** | Windows VM (or LibreOffice fallback) |
|
||
| ERPNext | **Real** | Shared instance, test SO cleaned up after |
|
||
|
||
---
|
||
|
||
## Phases
|
||
|
||
### Phase 0: Test Harness (Opus 4.6)
|
||
|
||
Write `scripts/tests/e2e_crtc_pipeline.py`:
|
||
- Playwright + requests + psycopg2 orchestration script
|
||
- Connects to dev PG, dev API, dev workers, ERPNext
|
||
- Creates test data, runs pipeline, screenshots, verifies, cleans up
|
||
- Screenshots to `scripts/tests/screenshots/`
|
||
|
||
### Phase 1: Create Test Order (Sonnet 4.6)
|
||
|
||
- Insert PG `canada_crtc_orders` with `payment_status=paid`, `funds_available=TRUE`
|
||
- Create ERPNext SO at `"Client Selection"` state
|
||
- **Screenshot 1:** ERPNext SO detail page
|
||
|
||
### Phase 2: Steps 1-4 — Incorporation (Sonnet 4.6)
|
||
|
||
- Pre-populate mocked vendor results in PG (BC#, DID, AMB)
|
||
- Advance ERPNext SO workflow through Steps 1-4
|
||
- **Screenshot 2:** ERPNext SO with BC#/DID/AMB fields
|
||
- **Screenshot 3:** PG query results
|
||
|
||
### Phase 3: Step 5 — Domain (Sonnet 4.6)
|
||
|
||
- Pre-populate domain in PG, advance workflow to `"Domain Ready"`
|
||
- **Screenshot 4:** ERPNext SO domain fields
|
||
|
||
### Phase 4: Step 6 — CRTC Letter DOCX/PDF (Opus 4.6)
|
||
|
||
- Trigger `generate_crtc_docs` job on worker
|
||
- Verify DOCX: entity name, BC#, 5 sections, signature block
|
||
- Verify PDF: MinIO object exists, valid PDF header, >0 bytes
|
||
- Verify DocServer heartbeat (or LibreOffice fallback noted)
|
||
- **Screenshot 5:** MinIO console — PDF listing
|
||
- **Screenshot 6:** ERPNext SO — `"Awaiting eSign"` state
|
||
|
||
### Phase 5: Step 6b — eSign (Sonnet 4.6)
|
||
|
||
- Generate JWT token for test order
|
||
- Screenshot eSign portal page
|
||
- POST inject base64 signature
|
||
- Verify PG: `esign_signed_at`, ERPNext SO: `"CRTC Submitted"`
|
||
- **Screenshot 7:** eSign portal page
|
||
- **Screenshot 8:** ERPNext SO post-eSign
|
||
|
||
### Phase 6: Steps 7-10 — Binder + Delivery (Sonnet 4.6)
|
||
|
||
- Wait for `resume_crtc_pipeline` job completion
|
||
- Verify binder: MinIO object, valid PDF, multi-page
|
||
- Verify delivery email (Mailhog or file capture)
|
||
- Verify banking referral sent
|
||
- **Screenshot 9:** MinIO binder PDF
|
||
- **Screenshot 10:** Delivery email
|
||
- **Screenshot 11:** ERPNext SO `"Banking Ready"`
|
||
|
||
### Phase 7: Steps 11-13 — BITS/CCTS/Compliance (Sonnet 4.6)
|
||
|
||
- Wait for pipeline to reach `"Ready for Review"`
|
||
- Verify BITS: ToDo exists, `custom_bits_filed_at` set
|
||
- Verify CCTS: ToDo exists, `custom_ccts_filed_at` set
|
||
- Verify Compliance Calendar: 12+ entries with correct dates/amounts
|
||
- **Screenshot 12:** Compliance Calendar list
|
||
- **Screenshot 13:** ToDo list (BITS + CCTS)
|
||
- **Screenshot 14:** ERPNext SO final state
|
||
|
||
### Phase 8: Cleanup (Sonnet 4.6)
|
||
|
||
- Delete PG test rows, ERPNext SO/Calendar/ToDo, MinIO objects
|
||
- Print cleanup report
|
||
|
||
---
|
||
|
||
## Screenshot Manifest (14 screenshots)
|
||
|
||
| # | Filename | Source | Verifies |
|
||
|---|----------|--------|----------|
|
||
| 1 | `01-so-client-selection.png` | ERPNext SO | Initial post-payment state |
|
||
| 2 | `02-so-incorporation-complete.png` | ERPNext SO | BC#, DID, AMB populated |
|
||
| 3 | `03-pg-order-fields.png` | Terminal PG | Raw DB row |
|
||
| 4 | `04-so-domain-ready.png` | ERPNext SO | Domain + HestiaCP |
|
||
| 5 | `05-minio-crtc-letter.png` | MinIO console | Letter PDF exists |
|
||
| 6 | `06-so-awaiting-esign.png` | ERPNext SO | Awaiting eSign state |
|
||
| 7 | `07-esign-portal-page.png` | Browser | eSign page with preview |
|
||
| 8 | `08-so-crtc-submitted.png` | ERPNext SO | Post-eSign state |
|
||
| 9 | `09-minio-binder.png` | MinIO console | Binder PDF exists |
|
||
| 10 | `10-delivery-email.png` | Mailhog/file | Client delivery email |
|
||
| 11 | `11-so-banking-ready.png` | ERPNext SO | Banking referral sent |
|
||
| 12 | `12-compliance-calendar.png` | ERPNext list | 12+ compliance entries |
|
||
| 13 | `13-todos-bits-ccts.png` | ERPNext list | BITS + CCTS ToDos |
|
||
| 14 | `14-so-ready-for-review.png` | ERPNext SO | Final state all fields |
|
||
|
||
---
|
||
|
||
## Model Assignment
|
||
|
||
| Phase | Model | Why |
|
||
|-------|-------|-----|
|
||
| 0: Test harness | **Opus 4.6** | Complex architecture — mocks, Playwright, error handling |
|
||
| 1: Create order | Sonnet 4.6 | Mechanical DB + API calls |
|
||
| 2: Steps 1-4 | Sonnet 4.6 | Pre-populate mock data |
|
||
| 3: Step 5 | Sonnet 4.6 | Pre-populate mock data |
|
||
| 4: Step 6 DOCX/PDF | **Opus 4.6** | Critical — DOCX structure + PDF conversion verification |
|
||
| 5: eSign | Sonnet 4.6 | API inject + verify |
|
||
| 6: Steps 7-10 | Sonnet 4.6 | File + email verification |
|
||
| 7: Steps 11-13 | Sonnet 4.6 | ERPNext entry verification |
|
||
| 8: Cleanup | Sonnet 4.6 | Mechanical deletes |
|
||
|
||
**Total effort:** 8-10 hours (1-2 sessions)
|
||
|
||
---
|
||
|
||
## Test Results (2026-04-06)
|
||
|
||
### Run Summary
|
||
|
||
| Phase | Result | Time | Details |
|
||
|-------|--------|------|---------|
|
||
| 1 | PASS | <1s | PG order + ERPNext SO created + submitted + workflow advanced (Received → Mailbox Ready) |
|
||
| 2-3 | PASS | <1s | Mock BC# BC1234567, DID +16045551234, domain, AMB |
|
||
| 4 | PASS | 2s | DOCX 37.6KB (5/5 content checks), PDF 42KB (LibreOffice), uploaded to MinIO |
|
||
| 5 | PASS | 3s | eSign page screenshot captured, JWT simulated in PG |
|
||
| 6 | PASS | 10s | PDF verified in MinIO (42KB, valid header) |
|
||
| 7 | PASS | 1s | ERPNext SO at "Mailbox Ready" state, no compliance entries (expected) |
|
||
| 7b | PASS | 30s | eSign screenshot captured, ERPNext login times out (Docker network) |
|
||
| **Total** | **ALL PASS** | **48s** | |
|
||
|
||
### Binder Compilation (Separate Test)
|
||
|
||
| Test | Result | Details |
|
||
|------|--------|---------|
|
||
| DOCX generation | PASS | 37.5KB |
|
||
| LibreOffice PDF | PASS | 41.4KB |
|
||
| Binder compilation | PASS | 37.3KB, **5 pages** (cover + TOC + divider + letter content) |
|
||
|
||
### DocServer Investigation
|
||
|
||
As of 2026-06, the worker runs fine under the SYSTEM account in session 0 on
|
||
this Windows Server 2019 box (Word COM initialises and converts normally), so the
|
||
old "requires an interactive RDP login" workaround is no longer needed for normal
|
||
operation. It is **self-healing**:
|
||
|
||
- It retries the MinIO connection with backoff instead of `sys.exit(1)`, so a
|
||
transient MinIO 502 / outage no longer kills it (that was the cause of a
|
||
multi-week outage in May 2026).
|
||
- The `PW-DocserverWorker` task restarts on failure (99×/1 min) and has a
|
||
5-minute repeating safety trigger (`MultipleInstances=IgnoreNew`), so a crash
|
||
or missed boot trigger self-recovers within ~5 min without a reboot/RDP.
|
||
|
||
LibreOffice fallback still handles conversions automatically if DocServer is ever
|
||
unavailable. See `docserver/README.md` → "Reliability / self-healing".
|
||
|
||
### Known Limitations
|
||
|
||
1. **DocServer** — self-healing (auto-restarts on MinIO outage/crash); RDP only
|
||
needed if Word COM itself breaks (DCOM misconfig → run `fix_dcom.bat`)
|
||
2. **eSign JWT** — test uses different secret than dev API; falls back to PG simulation
|
||
3. **Compliance Calendar** — DocType not imported to ERPNext; 417 error on query
|
||
4. **ERPNext screenshots** — Playwright can't log into ERPNext from Docker (login page structure)
|
||
5. **Full pipeline** — individual components tested; full 14-step pipeline needs ERPNext workflow + all DocTypes imported
|