Andy Alexander

1. Infrastructure as code, live in public

Terraform provisions the hosts on Proxmox and Ansible configures them: a three-node k3s cluster, PostgreSQL, Icinga2, and an nginx/Let's Encrypt reverse proxy on a cloud host. It runs a Flask + PostgreSQL app anyone can use.

Design choice: the public entry point is the cloud host, which reaches home over WireGuard — the home network accepts no inbound connections.

Honest limit: all three k3s nodes share one physical host, so it's a working multi-node cluster, not real fault tolerance.

Live demo · Repo and write-up

2. Kea DHCP high availability lab

A hot-standby Kea DHCPv4 pair on Rocky Linux 9 (SELinux enforcing) across two Proxmox hosts, behind an ISP-style OPNsense relay with option 82. Configs live in Git with a drift check that runs before every deploy.

Design choice: each peer keeps its own PostgreSQL lease database — a shared database would be the single point of failure HA exists to remove.

Result: a scripted failover test stops the primary, gets a real client lease from the standby, and confirms that lease synced back on failback — about 58 s to failover. A perfdhcp load test found saturation near 2,700 leases/s and showed the limit was CPU.

HA runbook · Relay lab · Configs and failover test

3. Storage and backups

Migrated 30 TB from mdadm RAID5 to TrueNAS SCALE with a ZFS RAIDZ1 pool, using PCIe SATA controller passthrough under Proxmox, and moved cross-host storage access to NFS. Off-site backups go to a cloud host with restic; UPS shutdown is handled by NUT.

Design choices: restic runs on TrueNAS itself (the real data source, one less hop to break) and targets the WireGuard overlay address, so the backup endpoint is never exposed to the internet. NUT runs on a dedicated Raspberry Pi, with staggered thresholds so storage always shuts down before the hypervisor.

Result: backups and full restic check --read-data verification run on a schedule; the shutdown sequence was reboot-tested across all three hosts.

Storage migration · Backups · UPS

4. Root-cause analysis

Two investigations, written up in full, including the wrong turns.

• A "bridge bug" that wasn't. One same-host VM pair could ping but not hold a TCP connection. I first concluded the Proxmox bridge was at fault; packet captures later showed a guest configured with a /32 prefix, sending every reply via the gateway. The runbook keeps the original reasoning and explains why the evidence was misread.
• A full hypervisor hang on the primary Proxmox host, traced to a wedged USB controller shared with the UPS. The fix was moving UPS monitoring to its own device.

/32 write-up · USB hang write-up

5. Nested vSphere 8 cluster lab

A two-host ESXi 8 cluster with vCenter, nested on Proxmox: vCenter deployed by the CLI installer from a JSON template, inventory and a distributed switch built with govc, an NFS shared datastore, templates, and a VMware → Proxmox migration with Proxmox's native importer.

Design choices: both ESXi VMs get one CPU model across two Intel generations, so vMotion works without EVC; vCenter stays on local storage, so losing the NFS server can't take the management plane down with it.

Result: vMotion at zero packet loss, and a vSphere HA test by hard-stopping a host — failure detected in 18 s and the VM restarted on the survivor.

Runbook

flowchart TB
    INET(("Internet
XGS-PON fiber"))
    USER(["Public visitors"])

    subgraph SW ["MikroTik CRS310 switch"]
        BW["bridge-WAN
ONT-on-a-stick SFP+"]
        BL["bridge-LAN"]
    end

    subgraph WU ["wu — Proxmox"]
        FW["OPNsense VM
router · firewall · DHCP"]
        PH2["pihole2
secondary DNS + Unbound"]
    end

    PH1["pihole
primary DNS + Unbound"]
    PI["pi2b
NUT UPS server"]

    subgraph SWG ["swearengen — Proxmox"]
        NAS["TrueNAS SCALE
ZFS RAIDZ1"]
        APPS["Plex · Home Assistant"]
    end

    subgraph GAR ["garrett — Proxmox"]
        K3S["k3s · PostgreSQL · Icinga2
Terraform + Ansible"]
    end

    KK1["kk1 — cloud
WireGuard hub · nginx/TLS · backup target"]
    REMOTE["Remote site
Proxmox · WireGuard spoke"]

    INET --- BW
    BW ---|"wu NIC 1"| FW
    FW ---|"wu NIC 2"| BL
    BL --- PH1
    BL --- PI
    BL --- SWG
    BL --- GAR
    FW == "WireGuard" ==> KK1
    REMOTE == "WireGuard" ==> KK1
    USER -->|"HTTPS"| KK1
    KK1 -. "tasks.greenbean.org" .-> K3S
    NAS -. "restic" .-> KK1