Initial commit — Performance West telecom compliance platform
Includes: API (Express/TypeScript), Astro site, Python workers, document generators, FCC compliance tools, Canada CRTC formation, Ansible infrastructure, and deployment scripts. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
justin
commit
f8cd37ac8c
1823 changed files with 145167 additions and 0 deletions
271
docs/infrastructure.md
Normal file
271
docs/infrastructure.md
Normal file
|
|
@ -0,0 +1,271 @@
|
|||
# Infrastructure
|
||||
|
||||
**Last updated:** 2026-04-06
|
||||
|
||||
## Production Server — Linux VM
|
||||
|
||||
| Resource | Spec |
|
||||
|----------|------|
|
||||
| OS | Debian 13 (Trixie) |
|
||||
| IP | 207.174.124.71 |
|
||||
| SSH | `ssh -p 22022 deploy@207.174.124.71` |
|
||||
| vCPU | 8 |
|
||||
| RAM | 32 GB |
|
||||
| Disk | 232 GB SSD |
|
||||
| Network | Bridged, static IP |
|
||||
|
||||
## Proxmox VM — Windows (DocServer) — NOT YET PROVISIONED
|
||||
|
||||
| Resource | Spec |
|
||||
|----------|------|
|
||||
| OS | Windows Server 2022 |
|
||||
| vCPU | 2 |
|
||||
| RAM | 4 GB |
|
||||
| Disk | 40 GB SSD |
|
||||
| Software | Microsoft Office 2021 |
|
||||
| Service | DocServer on port 5050 |
|
||||
|
||||
The Windows VM will provide high-fidelity DOCX-to-PDF conversion via Office 2021. DocServer exposes a REST API on port 5050. LibreOffice on the Linux VM serves as a fallback.
|
||||
|
||||
## External Infrastructure Dependencies
|
||||
|
||||
- **HestiaCP** — `cp.carrierone.com:22022` — DNS management and mail hosting for `performancewest.net` (user: `justin`)
|
||||
- **Nameservers** — `ns1.he.net` through `ns5.he.net` + `ns0.cp.carrierone.com`
|
||||
|
||||
## Auto-Start on Reboot
|
||||
|
||||
All services are configured to restart automatically after a reboot via two complementary mechanisms:
|
||||
|
||||
### 1. systemd `performancewest.service`
|
||||
|
||||
A custom systemd unit at `/etc/systemd/system/performancewest.service` runs `docker compose up -d --remove-orphans` after the Docker daemon and network are ready. This ensures the full compose stack is reconciled on every boot.
|
||||
|
||||
```bash
|
||||
# Status
|
||||
sudo systemctl status performancewest.service
|
||||
|
||||
# Manually start/stop the stack
|
||||
sudo systemctl start performancewest.service
|
||||
sudo systemctl stop performancewest.service
|
||||
|
||||
# Reload (runs docker compose up -d --remove-orphans)
|
||||
sudo systemctl reload performancewest.service
|
||||
```
|
||||
|
||||
Enabled services verified: `performancewest`, `docker`, `nginx`, `fail2ban`, `unattended-upgrades`, `k3s`.
|
||||
|
||||
### 2. Docker `restart: unless-stopped`
|
||||
|
||||
Every container in `docker-compose.yml` has `restart: unless-stopped`. If Docker itself restarts, each container is individually restarted by the Docker daemon before the systemd unit fires.
|
||||
|
||||
### 3. k3s (SHKeeper)
|
||||
|
||||
k3s is a separate systemd service (`k3s.service`) that starts automatically on boot. All SHKeeper pods have `restartPolicy: Always` and are managed by Kubernetes deployments.
|
||||
|
||||
### Boot order
|
||||
|
||||
```
|
||||
Kernel → network-online.target → docker.service → performancewest.service
|
||||
└─ docker compose up -d
|
||||
→ k3s.service → SHKeeper pods auto-reconciled
|
||||
```
|
||||
|
||||
## Docker Compose Orchestration
|
||||
|
||||
All Docker services are defined in `docker-compose.yml` at the project root. Environment variables are sourced from `.env` (see `.env.example` for required values).
|
||||
|
||||
```bash
|
||||
# Start all services
|
||||
cd /opt/performancewest
|
||||
docker compose up -d
|
||||
|
||||
# Rebuild after code changes
|
||||
docker compose build site api && docker compose up -d site api
|
||||
|
||||
# View logs
|
||||
docker compose logs -f site api workers
|
||||
|
||||
# Stop everything
|
||||
docker compose down
|
||||
```
|
||||
|
||||
### Running Containers (13 Docker + k3s pods)
|
||||
|
||||
**Docker Compose:**
|
||||
|
||||
| Container | Image | Port |
|
||||
|-----------|-------|------|
|
||||
| site | performancewest-site (Astro/nginx) | 4322 |
|
||||
| api | performancewest-api (Express/TS) | 3001 |
|
||||
| api-postgres | postgres:16-alpine | 5432 |
|
||||
| erpnext | performancewest-erpnext:latest (custom) | 8080 |
|
||||
| erpnext-worker-default | performancewest-erpnext:latest | — |
|
||||
| erpnext-worker-short | performancewest-erpnext:latest | — |
|
||||
| erpnext-scheduler | performancewest-erpnext:latest | — |
|
||||
| erpnext-mariadb | mariadb:10.6 | 3306 |
|
||||
| erpnext-redis | redis:7-alpine | 6379 |
|
||||
| listmonk | listmonk/listmonk:latest | 9100 |
|
||||
| listmonk-postgres | postgres:16-alpine | — |
|
||||
| minio | minio/minio:latest | 9000/9001 |
|
||||
| workers | performancewest-workers (Python) | 8090 |
|
||||
| ollama | ollama/ollama:latest | 11434 |
|
||||
| umami | ghcr.io/umami-software/umami:postgresql-latest | 3100 |
|
||||
| umami-postgres | postgres:16-alpine | — |
|
||||
|
||||
**k3s / Kubernetes (SHKeeper):**
|
||||
|
||||
| Deployment | Replicas | Port |
|
||||
|------------|----------|------|
|
||||
| shkeeper-deployment | 1 | 5000 (LoadBalancer) |
|
||||
| bitcoin-shkeeper | 3 | — |
|
||||
| ethereum-shkeeper | 3 | — |
|
||||
| polygon-shkeeper | 3 | — |
|
||||
| bnb-shkeeper | 3 | — |
|
||||
| tron-shkeeper | 3 | — |
|
||||
| litecoin-shkeeper | 3 | — |
|
||||
| dogecoin-shkeeper | 3 | — |
|
||||
| mariadb (SHKeeper) | 1 | — |
|
||||
|
||||
### Service Dependencies
|
||||
|
||||
```
|
||||
site (standalone)
|
||||
api → api-postgres
|
||||
erpnext → erpnext-mariadb, erpnext-redis
|
||||
erpnext-worker-default → erpnext
|
||||
erpnext-worker-short → erpnext
|
||||
erpnext-scheduler → erpnext
|
||||
listmonk → listmonk-postgres
|
||||
minio (standalone)
|
||||
workers → erpnext, minio, api-postgres, ollama
|
||||
ollama (standalone)
|
||||
umami → umami-postgres
|
||||
```
|
||||
|
||||
SHKeeper pods are managed by k3s and are independent of Docker Compose.
|
||||
|
||||
### ERPNext Notes
|
||||
|
||||
- Image: `performancewest-erpnext:latest` — custom image built from `erpnext/Dockerfile` extending `frappe/erpnext:version-15`
|
||||
- Baked-in apps: `frappe_crypto` (1.0.0), `frappe_adyen` (1.0.0), `performancewest_erpnext` (1.0.0)
|
||||
- Database: MariaDB 10.6 (NOT PostgreSQL — ERPNext v15 doesn't support Postgres)
|
||||
- 6 apps installed: frappe, erpnext, payments, frappe_crypto, frappe_adyen, performancewest_erpnext
|
||||
- Admin credentials: `Administrator` / stored in Ansible vault
|
||||
- API keys set in `.env`: `ERPNEXT_API_KEY` / `ERPNEXT_API_SECRET`
|
||||
- First-run init: `bench new-site` is run by the Ansible `erpnext` role on first deploy (guarded by sentinel file `erpnext-initialized`)
|
||||
|
||||
### k3s Notes
|
||||
|
||||
- Installed with `--docker --disable=traefik` to avoid port conflicts with host nginx
|
||||
- Helm 3 installed for SHKeeper chart management (`vsys-host/shkeeper`)
|
||||
- SHKeeper exposed via LoadBalancer service at port 5000, proxied through host nginx
|
||||
- k3s uses Docker as its container runtime (not containerd)
|
||||
|
||||
## nginx Reverse Proxy
|
||||
|
||||
Host-level nginx handles TLS termination. Configs live in `/etc/nginx/sites-available/`. All deployed via the Ansible `nginx` role from `infra/ansible/roles/nginx/templates/`.
|
||||
|
||||
| Config file | Domain | Upstream |
|
||||
|---|---|---|
|
||||
| `pw-site.conf` | `performancewest.net`, `www.` | `:4322` |
|
||||
| `pw-api.conf` | `api.performancewest.net` | `:3001` |
|
||||
| `pw-crm.conf` | `crm.performancewest.net` | `:8080` (requires `proxy_set_header Host performancewest.net`) |
|
||||
| `pw-listmonk.conf` | `lists.performancewest.net` | `:9100` |
|
||||
| `pw-portal.conf` | `portal.performancewest.net` | `:8080` (static assets from Docker volumes, Frappe branding replaced via sub_filter) |
|
||||
| `pw-analytics.conf` | `analytics.performancewest.net` | `:3100` |
|
||||
| `pw-btcpay-tls.conf` | `pay.performancewest.net` | SHKeeper `:5000` |
|
||||
| `pw-crypto-tls.conf` | `crypto.performancewest.net` | SHKeeper admin |
|
||||
| `pw-minio.conf` | `minio.performancewest.net` | `:9000` |
|
||||
| `pw-minio.conf` | `minio-console.performancewest.net` | `:9001` |
|
||||
|
||||
### TLS Certificates
|
||||
|
||||
Managed by Certbot with automatic renewal (cron at 3:30 AM daily):
|
||||
|
||||
```bash
|
||||
# Certificates obtained for:
|
||||
performancewest.net + www.performancewest.net
|
||||
api.performancewest.net
|
||||
crm.performancewest.net
|
||||
analytics.performancewest.net
|
||||
pay.performancewest.net
|
||||
crypto.performancewest.net
|
||||
minio.performancewest.net
|
||||
minio-console.performancewest.net
|
||||
# Note: mail.performancewest.net cert lives on HestiaCP (207.174.124.15)
|
||||
```
|
||||
|
||||
## Firewall (UFW)
|
||||
|
||||
```
|
||||
Rule 1: ALLOW IN from 24.162.65.184 # Trusted IP 1
|
||||
Rule 2: ALLOW IN from 76.228.206.147 # Trusted IP 2
|
||||
Rule 3: ALLOW IN 22022/tcp # SSH
|
||||
Rule 4: ALLOW IN 80/tcp # HTTP
|
||||
Rule 5: ALLOW IN 443/tcp # HTTPS
|
||||
Default: DENY incoming, ALLOW outgoing
|
||||
```
|
||||
|
||||
## Fail2ban
|
||||
|
||||
Jails active: `sshd`, `nginx-badbots`, `pw-api` (enabled after API container log exists).
|
||||
|
||||
Trusted IPs whitelisted in `/etc/fail2ban/jail.local`: `24.162.65.184`, `76.228.206.147`.
|
||||
|
||||
## Unattended Security Updates
|
||||
|
||||
Configured via `/etc/apt/apt.conf.d/50unattended-upgrades` and `20auto-upgrades`:
|
||||
- Daily apt update + upgrade
|
||||
- Automatic reboot at 4:00 AM if kernel update requires it
|
||||
- 7-day autoclean
|
||||
|
||||
## Ansible Deployment
|
||||
|
||||
```
|
||||
infra/ansible/
|
||||
inventory/
|
||||
hosts.yml # deploy user, port 22022, 207.174.124.71
|
||||
bootstrap.yml # root user, first-run only
|
||||
group_vars/all.yml # all vars + vault references
|
||||
playbooks/
|
||||
bootstrap.yml # First-run: common + docker (runs as root)
|
||||
site.yml # Full provisioning: all roles
|
||||
deploy.yml # Code deploy only (no infra changes)
|
||||
run-migrations.yml # Run a specific SQL migration
|
||||
roles/
|
||||
common/ # Packages, deploy user, SSH hardening, UFW
|
||||
docker/ # Docker CE + compose plugin + performancewest.service (boot auto-start)
|
||||
postgresql/ # API postgres + migrations + backup cron
|
||||
app/ # Express API container + .env
|
||||
site/ # Astro site container
|
||||
erpnext/ # ERPNext + MariaDB + workers + bench new-site (custom image build)
|
||||
minio/ # MinIO + mc client + bucket creation
|
||||
workers/ # Python workers + Ollama + model pull
|
||||
shkeeper/ # k3s + Helm + SHKeeper deployment (replaces old bitcart role)
|
||||
nginx/ # nginx + certbot TLS (all domains) + fail2ban
|
||||
```
|
||||
|
||||
### Deploy Commands
|
||||
|
||||
```bash
|
||||
# First-time server setup (run as local user with SSH key to root)
|
||||
ansible-playbook -i infra/ansible/inventory/bootstrap.yml infra/ansible/playbooks/bootstrap.yml
|
||||
|
||||
# Full provisioning (run after bootstrap — uses deploy user)
|
||||
ansible-playbook -i infra/ansible/inventory/hosts.yml infra/ansible/playbooks/site.yml --ask-vault-pass
|
||||
|
||||
# Code-only deploy (no infra changes, no vault needed for most)
|
||||
ansible-playbook -i infra/ansible/inventory/hosts.yml infra/ansible/playbooks/deploy.yml
|
||||
|
||||
# Run a specific DB migration
|
||||
ansible-playbook -i infra/ansible/inventory/hosts.yml infra/ansible/playbooks/run-migrations.yml -e "migration=010_canada_crtc.sql"
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
- **Uptime checks** — external HTTP monitoring for all subdomains
|
||||
- **ERPNext alerts** — system errors surfaced as ERPNext Issues
|
||||
- **Docker restart policies** — `unless-stopped` on all containers
|
||||
- **k3s pod management** — Kubernetes ensures SHKeeper pods stay running
|
||||
- **systemd auto-start** — `performancewest.service` (enabled) runs `docker compose up -d` on every boot
|
||||
- **PostgreSQL backups** — `/usr/local/bin/pg-backup.sh` runs at 2 AM daily, 30-day retention, stored in `/opt/backups/postgresql/`
|
||||
Loading…
Add table
Add a link
Reference in a new issue