new-site/docs/infrastructure.md

Infrastructure

Last updated: 2026-04-06

Production Server — Linux VM

Resource	Spec
OS	Debian 13 (Trixie)
IP	207.174.124.71
SSH	ssh -p 22022 deploy@207.174.124.71
vCPU	8
RAM	32 GB
Disk	232 GB SSD
Network	Bridged, static IP

Proxmox VM — Windows (DocServer) — NOT YET PROVISIONED

Resource	Spec
OS	Windows Server 2022
vCPU	2
RAM	4 GB
Disk	40 GB SSD
Software	Microsoft Office 2021
Service	DocServer on port 5050

The Windows VM will provide high-fidelity DOCX-to-PDF conversion via Office 2021. DocServer exposes a REST API on port 5050. LibreOffice on the Linux VM serves as a fallback.

External Infrastructure Dependencies

• HestiaCP — cp.carrierone.com:22022 — DNS management and mail hosting for performancewest.net (user: justin)
• Nameservers — ns1.he.net through ns5.he.net + ns0.cp.carrierone.com

Auto-Start on Reboot

All services are configured to restart automatically after a reboot via two complementary mechanisms:

1. systemd performancewest.service

A custom systemd unit at /etc/systemd/system/performancewest.service runs docker compose up -d --remove-orphans after the Docker daemon and network are ready. This ensures the full compose stack is reconciled on every boot.

# Status
sudo systemctl status performancewest.service

# Manually start/stop the stack
sudo systemctl start performancewest.service
sudo systemctl stop performancewest.service

# Reload (runs docker compose up -d --remove-orphans)
sudo systemctl reload performancewest.service

Enabled services verified: performancewest, docker, nginx, fail2ban, unattended-upgrades, k3s.

2. Docker restart: unless-stopped

Every container in docker-compose.yml has restart: unless-stopped. If Docker itself restarts, each container is individually restarted by the Docker daemon before the systemd unit fires.

3. k3s (SHKeeper)

k3s is a separate systemd service (k3s.service) that starts automatically on boot. All SHKeeper pods have restartPolicy: Always and are managed by Kubernetes deployments.

Boot order

Kernel → network-online.target → docker.service → performancewest.service
                                                     └─ docker compose up -d
                               → k3s.service → SHKeeper pods auto-reconciled

Docker Compose Orchestration

All Docker services are defined in docker-compose.yml at the project root. Environment variables are sourced from .env (see .env.example for required values).

# Start all services
cd /opt/performancewest
docker compose up -d

# Rebuild after code changes
docker compose build site api && docker compose up -d site api

# View logs
docker compose logs -f site api workers

# Stop everything
docker compose down

Running Containers (13 Docker + k3s pods)

Docker Compose:

Container	Image	Port
site	performancewest-site (Astro/nginx)	4322
api	performancewest-api (Express/TS)	3001
api-postgres	postgres:16-alpine	5432
erpnext	performancewest-erpnext:latest (custom)	8080
erpnext-worker-default	performancewest-erpnext:latest	—
erpnext-worker-short	performancewest-erpnext:latest	—
erpnext-scheduler	performancewest-erpnext:latest	—
erpnext-mariadb	mariadb:10.6	3306
erpnext-redis	redis:7-alpine	6379
listmonk	listmonk/listmonk:latest	9100
listmonk-postgres	postgres:16-alpine	—
minio	minio/minio:latest	9000/9001
workers	performancewest-workers (Python)	8090
ollama	ollama/ollama:latest	11434
umami	ghcr.io/umami-software/umami:postgresql-latest	3100
umami-postgres	postgres:16-alpine	—

k3s / Kubernetes (SHKeeper):

Deployment	Replicas	Port
shkeeper-deployment	1	5000 (LoadBalancer)
bitcoin-shkeeper	3	—
ethereum-shkeeper	3	—
polygon-shkeeper	3	—
bnb-shkeeper	3	—
tron-shkeeper	3	—
litecoin-shkeeper	3	—
dogecoin-shkeeper	3	—
mariadb (SHKeeper)	1	—

Service Dependencies

site (standalone)
api → api-postgres
erpnext → erpnext-mariadb, erpnext-redis
erpnext-worker-default → erpnext
erpnext-worker-short → erpnext
erpnext-scheduler → erpnext
listmonk → listmonk-postgres
minio (standalone)
workers → erpnext, minio, api-postgres, ollama
ollama (standalone)
umami → umami-postgres

SHKeeper pods are managed by k3s and are independent of Docker Compose.

ERPNext Notes

• Image: performancewest-erpnext:latest — custom image built from erpnext/Dockerfile extending frappe/erpnext:version-15
• Baked-in apps: frappe_crypto (1.0.0), frappe_adyen (1.0.0), performancewest_erpnext (1.0.0)
• Database: MariaDB 10.6 (NOT PostgreSQL — ERPNext v15 doesn't support Postgres)
• 6 apps installed: frappe, erpnext, payments, frappe_crypto, frappe_adyen, performancewest_erpnext
• Admin credentials: Administrator / stored in Ansible vault
• API keys set in .env: ERPNEXT_API_KEY / ERPNEXT_API_SECRET
• First-run init: bench new-site is run by the Ansible erpnext role on first deploy (guarded by sentinel file erpnext-initialized)

k3s Notes

• Installed with --docker --disable=traefik to avoid port conflicts with host nginx
• Helm 3 installed for SHKeeper chart management (vsys-host/shkeeper)
• SHKeeper exposed via LoadBalancer service at port 5000, proxied through host nginx
• k3s uses Docker as its container runtime (not containerd)

nginx Reverse Proxy

Host-level nginx handles TLS termination. Configs live in /etc/nginx/sites-available/. All deployed via the Ansible nginx role from infra/ansible/roles/nginx/templates/.

Config file	Domain	Upstream
pw-site.conf	performancewest.net, www.	:4322
pw-api.conf	api.performancewest.net	:3001
pw-crm.conf	crm.performancewest.net	:8080 (requires proxy_set_header Host performancewest.net)
pw-listmonk.conf	lists.performancewest.net	:9100
pw-portal.conf	portal.performancewest.net	:8080 (static assets from Docker volumes, Frappe branding replaced via sub_filter)
pw-analytics.conf	analytics.performancewest.net	:3100
pw-btcpay-tls.conf	pay.performancewest.net	SHKeeper :5000
pw-crypto-tls.conf	crypto.performancewest.net	SHKeeper admin
pw-minio.conf	minio.performancewest.net	:9000
pw-minio.conf	minio-console.performancewest.net	:9001

TLS Certificates

Managed by Certbot with automatic renewal (cron at 3:30 AM daily):

# Certificates obtained for:
performancewest.net + www.performancewest.net
api.performancewest.net
crm.performancewest.net
analytics.performancewest.net
pay.performancewest.net
crypto.performancewest.net
minio.performancewest.net
minio-console.performancewest.net
# Note: mail.performancewest.net cert lives on HestiaCP (207.174.124.15)

Firewall (UFW)

Rule 1:  ALLOW IN  from 24.162.65.184    # Trusted IP 1
Rule 2:  ALLOW IN  from 76.228.206.147   # Trusted IP 2
Rule 3:  ALLOW IN  22022/tcp             # SSH
Rule 4:  ALLOW IN  80/tcp                # HTTP
Rule 5:  ALLOW IN  443/tcp               # HTTPS
Default: DENY incoming, ALLOW outgoing

Fail2ban

Jails active: sshd, nginx-badbots, pw-api (enabled after API container log exists).

Trusted IPs whitelisted in /etc/fail2ban/jail.local: 24.162.65.184, 76.228.206.147.

Unattended Security Updates

Configured via /etc/apt/apt.conf.d/50unattended-upgrades and 20auto-upgrades:

• Daily apt update + upgrade
• Automatic reboot at 4:00 AM if kernel update requires it
• 7-day autoclean

Ansible Deployment

infra/ansible/
  inventory/
    hosts.yml           # deploy user, port 22022, 207.174.124.71
    bootstrap.yml       # root user, first-run only
    group_vars/all.yml  # all vars + vault references
  playbooks/
    bootstrap.yml       # First-run: common + docker (runs as root)
    site.yml            # Full provisioning: all roles
    deploy.yml          # Code deploy only (no infra changes)
    run-migrations.yml  # Run a specific SQL migration
  roles/
    common/     # Packages, deploy user, SSH hardening, UFW
    docker/     # Docker CE + compose plugin + performancewest.service (boot auto-start)
    postgresql/ # API postgres + migrations + backup cron
    app/        # Express API container + .env
    site/       # Astro site container
    erpnext/    # ERPNext + MariaDB + workers + bench new-site (custom image build)
    minio/      # MinIO + mc client + bucket creation
    workers/    # Python workers + Ollama + model pull
    shkeeper/   # k3s + Helm + SHKeeper deployment (replaces old bitcart role)
    nginx/      # nginx + certbot TLS (all domains) + fail2ban

Deploy Commands

# First-time server setup (run as local user with SSH key to root)
ansible-playbook -i infra/ansible/inventory/bootstrap.yml infra/ansible/playbooks/bootstrap.yml

# Full provisioning (run after bootstrap — uses deploy user)
ansible-playbook -i infra/ansible/inventory/hosts.yml infra/ansible/playbooks/site.yml --ask-vault-pass

# Code-only deploy (no infra changes, no vault needed for most)
ansible-playbook -i infra/ansible/inventory/hosts.yml infra/ansible/playbooks/deploy.yml

# Run a specific DB migration
ansible-playbook -i infra/ansible/inventory/hosts.yml infra/ansible/playbooks/run-migrations.yml -e "migration=010_canada_crtc.sql"

Monitoring

• Uptime checks — external HTTP monitoring for all subdomains
• ERPNext alerts — system errors surfaced as ERPNext Issues
• Docker restart policies — unless-stopped on all containers
• k3s pod management — Kubernetes ensures SHKeeper pods stay running
• systemd auto-start — performancewest.service (enabled) runs docker compose up -d on every boot
• PostgreSQL backups — /usr/local/bin/pg-backup.sh runs at 2 AM daily, 30-day retention, stored in /opt/backups/postgresql/