justin/new-site
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
new-site/docs/vm-security-hardening.md

3.1 KiB
Raw Blame History

VM / host security hardening

Prod app host: 207.174.124.71 (Debian 13 trixie, k3s + Docker, SSH on 22022).

Baseline (already in place before 2026-06)

  • SSH: permitrootlogin no, passwordauthentication no, key-only, port 22022.
  • fail2ban active (sshd, nginx-badbots).
  • unattended-upgrades enabled; 0 pending security updates.
  • TLS: Qualys SSL Labs A+, SecurityHeaders A, full CSP/HSTS-preload (set in /etc/nginx/snippets/pw-security.conf + pw-site.conf).

CRITICAL issue found + fixed (2026-06-06): wide-open host ports

ss/external probing showed a large attack surface reachable from the public internet (no host firewall; iptables INPUT policy ACCEPT; kube-router does not firewall host ports; Docker publishes container ports via 0.0.0.0 DNAT):

  • 5432 Postgres (customer DB!), 6443 k8s API, 10250 kubelet, 3022 Forgejo SSH, 9100/9101 Listmonk admin, 3001/3002 APIs, 8080, 3033, 3100, 5555/5556, 8888, 4322/4323, and the hc submission ports 2526-2528 — all OPEN to the internet. Confirmed via off-network /dev/tcp probes. Only :25 was provider-filtered.

Fix

Two layers, installed as a persistent, boot-enabled systemd service pw-firewall.service (auto-rollback timer was used during rollout):

  1. /etc/pw-firewall/pw-firewall.nft - dedicated inet pw_fw table with an input hook at priority -150 (evaluated before kube-router's ACCEPT). Allows loopback, established/related, internal subnets (127/8, 172.16/12, k3s 10.42/16 + 10.43/16, docker/cni/flannel/veth ifaces), ICMP, then a public allow-list { 22, 22022, 80, 443 } and DROPs all other NEW inbound on ens18. (Port 25 inbound removed - the VM only SENDS bulk mail; inbound mail is handled by Carbonio elsewhere. Outbound :25 egress is unaffected.)

  2. /usr/local/sbin/pw-docker-fw.sh - because Docker-published ports are DNAT'd and traverse FORWARD (not input), this adds DOCKER-USER rules: RETURN established, DROP NEW inbound on ens18 (scoped to the uplink so container<->container + nginx->container loopback are untouched), RETURN. Re-applied on docker restart via docker.service.d/pw-firewall.conf ExecStartPost.

Verified after fix (off-network)

  • Sensitive ports 5432/3022/9100/9101/8080/3001/3002/3100/3033/6443/10250/2526/25 -> blocked. Public 80/443/22022 -> open. Site HTTPS -> 200.
  • Internal intact: nginx->container loopback (listmonk 9100/9101 = 200), api->DB select 1, container->internet DNS egress, k3s node Ready, outbound mail still relays to Gmail MX from the hc IPs.

Files

  • /etc/pw-firewall/pw-firewall.nft
  • /usr/local/sbin/pw-docker-fw.sh
  • /etc/systemd/system/pw-firewall.service (enabled)
  • /etc/systemd/system/docker.service.d/pw-firewall.conf

TODO / follow-ups

  • Consider binding the Docker-published ports to 127.0.0.1 in compose (defence in depth) so they never bind 0.0.0.0 in the first place - the firewall already covers it, but compose-level 127.0.0.1:PORT:PORT is cleaner.
  • k8s API (6443) / kubelet (10250): now firewalled; if remote kubectl is ever needed, allow-list the specific admin source IP rather than reopening.