new-site/docs/infrastructure.md

# Infrastructure

**Last updated:** 2026-04-06

## Production Server — Linux VM

| Resource | Spec |
|----------|------|
| OS | Debian 13 (Trixie) |
| IP | 207.174.124.71 |
| SSH | `ssh -p 22022 deploy@207.174.124.71` |
| vCPU | 8 |
| RAM | 32 GB |
| Disk | 232 GB SSD |
| Network | Bridged, static IP |

## Proxmox VM — Windows (DocServer) — NOT YET PROVISIONED

| Resource | Spec |
|----------|------|
| OS | Windows Server 2022 |
| vCPU | 2 |
| RAM | 4 GB |
| Disk | 40 GB SSD |
| Software | Microsoft Office 2021 |
| Service | DocServer on port 5050 |

The Windows VM will provide high-fidelity DOCX-to-PDF conversion via Office 2021. DocServer exposes a REST API on port 5050. LibreOffice on the Linux VM serves as a fallback.

## External Infrastructure Dependencies

- **HestiaCP** — `cp.carrierone.com:22022` — DNS management and mail hosting for `performancewest.net` (user: `justin`)
- **Nameservers** — `ns1.he.net` through `ns5.he.net` + `ns0.cp.carrierone.com`

## Auto-Start on Reboot

All services are configured to restart automatically after a reboot via two complementary mechanisms:

### 1. systemd `performancewest.service`

A custom systemd unit at `/etc/systemd/system/performancewest.service` runs `docker compose up -d --remove-orphans` after the Docker daemon and network are ready. This ensures the full compose stack is reconciled on every boot.

```bash
# Status
sudo systemctl status performancewest.service

# Manually start/stop the stack
sudo systemctl start performancewest.service
sudo systemctl stop performancewest.service

# Reload (runs docker compose up -d --remove-orphans)
sudo systemctl reload performancewest.service
```

Enabled services verified: `performancewest`, `docker`, `nginx`, `fail2ban`, `unattended-upgrades`, `k3s`.

### 2. Docker `restart: unless-stopped`

Every container in `docker-compose.yml` has `restart: unless-stopped`. If Docker itself restarts, each container is individually restarted by the Docker daemon before the systemd unit fires.

### 3. k3s (SHKeeper)

k3s is a separate systemd service (`k3s.service`) that starts automatically on boot. All SHKeeper pods have `restartPolicy: Always` and are managed by Kubernetes deployments.

### Boot order

```
Kernel → network-online.target → docker.service → performancewest.service
                                                     └─ docker compose up -d
                               → k3s.service → SHKeeper pods auto-reconciled
```

## Docker Compose Orchestration

All Docker services are defined in `docker-compose.yml` at the project root. Environment variables are sourced from `.env` (see `.env.example` for required values).

```bash
# Start all services
cd /opt/performancewest
docker compose up -d

# Rebuild after code changes
docker compose build site api && docker compose up -d site api

# View logs
docker compose logs -f site api workers

# Stop everything
docker compose down
```

### Running Containers (13 Docker + k3s pods)

**Docker Compose:**

| Container | Image | Port |
|-----------|-------|------|
| site | performancewest-site (Astro/nginx) | 4322 |
| api | performancewest-api (Express/TS) | 3001 |
| api-postgres | postgres:16-alpine | 5432 |
| erpnext | performancewest-erpnext:latest (custom) | 8080 |
| erpnext-worker-default | performancewest-erpnext:latest | — |
| erpnext-worker-short | performancewest-erpnext:latest | — |
| erpnext-scheduler | performancewest-erpnext:latest | — |
| erpnext-mariadb | mariadb:10.6 | 3306 |
| erpnext-redis | redis:7-alpine | 6379 |
| listmonk | listmonk/listmonk:latest | 9100 |
| listmonk-postgres | postgres:16-alpine | — |
| minio | minio/minio:latest | 9000/9001 |
| workers | performancewest-workers (Python) | 8090 |
| ollama | ollama/ollama:latest | 11434 |
| umami | ghcr.io/umami-software/umami:postgresql-latest | 3100 |
| umami-postgres | postgres:16-alpine | — |

**k3s / Kubernetes (SHKeeper):**

| Deployment | Replicas | Port |
|------------|----------|------|
| shkeeper-deployment | 1 | 5000 (LoadBalancer) |
| bitcoin-shkeeper | 3 | — |
| ethereum-shkeeper | 3 | — |
| polygon-shkeeper | 3 | — |
| bnb-shkeeper | 3 | — |
| tron-shkeeper | 3 | — |
| litecoin-shkeeper | 3 | — |
| dogecoin-shkeeper | 3 | — |
| mariadb (SHKeeper) | 1 | — |

### Service Dependencies

```
site (standalone)
api → api-postgres
erpnext → erpnext-mariadb, erpnext-redis
erpnext-worker-default → erpnext
erpnext-worker-short → erpnext
erpnext-scheduler → erpnext
listmonk → listmonk-postgres
minio (standalone)
workers → erpnext, minio, api-postgres, ollama
ollama (standalone)
umami → umami-postgres
```

SHKeeper pods are managed by k3s and are independent of Docker Compose.

### ERPNext Notes

- Image: `performancewest-erpnext:latest` — custom image built from `erpnext/Dockerfile` extending `frappe/erpnext:version-15`
- Baked-in apps: `frappe_crypto` (1.0.0), `frappe_adyen` (1.0.0), `performancewest_erpnext` (1.0.0)
- Database: MariaDB 10.6 (NOT PostgreSQL — ERPNext v15 doesn't support Postgres)
- 6 apps installed: frappe, erpnext, payments, frappe_crypto, frappe_adyen, performancewest_erpnext
- Admin credentials: `Administrator` / stored in Ansible vault
- API keys set in `.env`: `ERPNEXT_API_KEY` / `ERPNEXT_API_SECRET`
- First-run init: `bench new-site` is run by the Ansible `erpnext` role on first deploy (guarded by sentinel file `erpnext-initialized`)

### k3s Notes

- Installed with `--docker --disable=traefik` to avoid port conflicts with host nginx
- Helm 3 installed for SHKeeper chart management (`vsys-host/shkeeper`)
- SHKeeper exposed via LoadBalancer service at port 5000, proxied through host nginx
- k3s uses Docker as its container runtime (not containerd)

## nginx Reverse Proxy

Host-level nginx handles TLS termination. Configs live in `/etc/nginx/sites-available/`. All deployed via the Ansible `nginx` role from `infra/ansible/roles/nginx/templates/`.

| Config file | Domain | Upstream |
|---|---|---|
| `pw-site.conf` | `performancewest.net`, `www.` | `:4322` |
| `pw-api.conf` | `api.performancewest.net` | `:3001` |
| `pw-crm.conf` | `crm.performancewest.net` | `:8080` (requires `proxy_set_header Host performancewest.net`) |
| `pw-listmonk.conf` | `lists.performancewest.net` | `:9100` |
| `pw-portal.conf` | `portal.performancewest.net` | `:8080` (static assets from Docker volumes, Frappe branding replaced via sub_filter) |
| `pw-analytics.conf` | `analytics.performancewest.net` | `:3100` |
| `pw-btcpay-tls.conf` | `pay.performancewest.net` | SHKeeper `:5000` |
| `pw-crypto-tls.conf` | `crypto.performancewest.net` | SHKeeper admin |
| `pw-minio.conf` | `minio.performancewest.net` | `:9000` |
| `pw-minio.conf` | `minio-console.performancewest.net` | `:9001` |

### TLS Certificates

Managed by Certbot with automatic renewal (cron at 3:30 AM daily):

```bash
# Certificates obtained for:
performancewest.net + www.performancewest.net
api.performancewest.net
crm.performancewest.net
analytics.performancewest.net
pay.performancewest.net
crypto.performancewest.net
minio.performancewest.net
minio-console.performancewest.net
# Note: mail.performancewest.net cert lives on HestiaCP (207.174.124.15)
```

## Firewall (UFW)

```
Rule 1:  ALLOW IN  from 24.162.65.184    # Trusted IP 1
Rule 2:  ALLOW IN  from 76.228.206.147   # Trusted IP 2
Rule 3:  ALLOW IN  22022/tcp             # SSH
Rule 4:  ALLOW IN  80/tcp                # HTTP
Rule 5:  ALLOW IN  443/tcp               # HTTPS
Default: DENY incoming, ALLOW outgoing
```

## Fail2ban

Jails active: `sshd`, `nginx-badbots`, `pw-api` (enabled after API container log exists).

Trusted IPs whitelisted in `/etc/fail2ban/jail.local`: `24.162.65.184`, `76.228.206.147`.

## Unattended Security Updates

Configured via `/etc/apt/apt.conf.d/50unattended-upgrades` and `20auto-upgrades`:
- Daily apt update + upgrade
- Automatic reboot at 4:00 AM if kernel update requires it
- 7-day autoclean

## Ansible Deployment

```
infra/ansible/
  inventory/
    hosts.yml           # deploy user, port 22022, 207.174.124.71
    bootstrap.yml       # root user, first-run only
    group_vars/all.yml  # all vars + vault references
  playbooks/
    bootstrap.yml       # First-run: common + docker (runs as root)
    site.yml            # Full provisioning: all roles
    deploy.yml          # Code deploy only (no infra changes)
    run-migrations.yml  # Run a specific SQL migration
  roles/
    common/     # Packages, deploy user, SSH hardening, UFW
    docker/     # Docker CE + compose plugin + performancewest.service (boot auto-start)
    postgresql/ # API postgres + migrations + backup cron
    app/        # Express API container + .env
    site/       # Astro site container
    erpnext/    # ERPNext + MariaDB + workers + bench new-site (custom image build)
    minio/      # MinIO + mc client + bucket creation
    workers/    # Python workers + Ollama + model pull
    shkeeper/   # k3s + Helm + SHKeeper deployment (replaces old bitcart role)
    nginx/      # nginx + certbot TLS (all domains) + fail2ban
```

### Deploy Commands

```bash
# First-time server setup (run as local user with SSH key to root)
ansible-playbook -i infra/ansible/inventory/bootstrap.yml infra/ansible/playbooks/bootstrap.yml

# Full provisioning (run after bootstrap — uses deploy user)
ansible-playbook -i infra/ansible/inventory/hosts.yml infra/ansible/playbooks/site.yml --ask-vault-pass

# Code-only deploy (no infra changes, no vault needed for most)
ansible-playbook -i infra/ansible/inventory/hosts.yml infra/ansible/playbooks/deploy.yml

# Run a specific DB migration
ansible-playbook -i infra/ansible/inventory/hosts.yml infra/ansible/playbooks/run-migrations.yml -e "migration=010_canada_crtc.sql"
```

## Monitoring

- **Uptime checks** — external HTTP monitoring for all subdomains
- **ERPNext alerts** — system errors surfaced as ERPNext Issues
- **Docker restart policies** — `unless-stopped` on all containers
- **k3s pod management** — Kubernetes ensures SHKeeper pods stay running
- **systemd auto-start** — `performancewest.service` (enabled) runs `docker compose up -d` on every boot
- **PostgreSQL backups** — `/usr/local/bin/pg-backup.sh` runs at 2 AM daily, 30-day retention, stored in `/opt/backups/postgresql/`