Cluster-A 🌐 in cloud (Hetzner control plane) + workers across 3+ home sites 🏡🏚🏠 over VPN

Cluster-B 🧱 local cluster at your main home
Dynamic DNS via Joker.com 🌍 to keep each home site reachable
CloudNativePG 🍃 with:
- Primary DB at your home site (Cluster-A)
- Async replicas at Site-2 & Site-3
Topology-aware reads 📍 (local reads)
Write routing to primary ✍️
Local services per home site 🧩

🌐 High-Level Architecture

        Hetzner Cloud ☁️
    ┌───────────────────────┐
    │ k3s control-plane 🧠   │
    └───────┬───────────────┘
            │ Tailscale/WG VPN 🔐
──────────────────────────────────────
        Multi-Site WAN Mesh

🏡 Home Site (Primary DB)
┌───────────────────────────┐
│ k3s worker nodes          │
│ Longhorn storage 💾       │
│ CNPG primary 🐘           │
│ DDNS updater ↻            │
└───────────────────────────┘

🏚 Site-2
┌───────────────────────────┐
│ k3s worker nodes          │
│ CNPG async replica 📖     │
│ DDNS updater ↻            │
└───────────────────────────┘

🏠 Site-3
┌───────────────────────────┐
│ k3s worker nodes          │
│ CNPG async replica 📖     │
│ DDNS updater ↻            │
└───────────────────────────┘

🏡 Local-only Cluster-B
┌───────────────────────────┐
│ k3s server + workers      │
│ Local apps only 🚪        │
└───────────────────────────┘

✅ Core Principles

Feature	Design
Cluster-A	Distributed across sites via VPN
Cluster-B	Local to main home
Networking	Tailscale or WireGuard
DNS	Joker.com Dynamic DNS
DB	CNPG primary @ home + async replicas at other sites
Storage	Longhorn per site only (no cross-WAN replication)
Routing	Writes → primary; Reads → local

🏗️ Step-by-Step Setup

1️⃣ VPN Everywhere (Tailscale recommended)

curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up --auth-key <KEY> --hostname <NODE_NAME>
ip -br a | grep tailscale0

Use the Tailscale IP (100.x.x.x) for all k3s --node-ip values.

2️⃣ Joker.com Dynamic DNS 🌍

For each site create DNS entries like:

home.example.com
site2.example.com
site3.example.com

Enable Dynamic DNS in Joker control panel (per host).

🔁 Site updater (ddclient method)

sudo apt install -y ddclient

/etc/ddclient.conf per site:

daemon=300
use=web, web=svc.joker.com/nic/checkip
protocol=dyndns2
server=svc.joker.com/nic/update?
ssl=yes

login=YOUR_DDNS_USER
password=YOUR_DDNS_PASS
host=home.example.com

Enable:

sudo systemctl enable --now ddclient

3️⃣ Cluster-A: Cloud control plane ☁️

On the Hetzner VM:

TS_IP_CP=<tailscale ip>

curl -sfL https://get.k3s.io | \
INSTALL_K3S_EXEC="--node-ip ${TS_IP_CP} --disable servicelb --disable local-storage" \
sh -

kubectl taint nodes $(hostname) node-role.kubernetes.io/control-plane=true:NoSchedule

4️⃣ Join Worker Nodes at Each Site 🏡🏚🏠

On each worker:

TS_IP_NODE=<tailscale-ip>
K3S_URL=https://<tailscale-ip-hetzner>:6443
K3S_TOKEN=$(ssh root@<hetzner> "sudo cat /var/lib/rancher/k3s/server/node-token")

curl -sfL https://get.k3s.io | \
K3S_URL=$K3S_URL K3S_TOKEN=$K3S_TOKEN \
INSTALL_K3S_EXEC="--node-ip ${TS_IP_NODE}" sh -

Label nodes:

kubectl label node node-home-1 site=home topology.kubernetes.io/zone=home
kubectl label node node-site2-1 site=site2 topology.kubernetes.io/zone=site2
kubectl label node node-site3-1 site=site3 topology.kubernetes.io/zone=site3

5️⃣ Install MetalLB at Each Site 🎯

Give each site its own LAN range.

Example:

# home LAN LB range
addresses: ["192.168.10.240-192.168.10.250"]

Repeat for site2/site3.

6️⃣ Install Longhorn 🐄 (per site)

kubectl create ns longhorn-system
kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.6.2/deploy/longhorn.yaml

Set replicas only within that site via UI or StorageClass override.

7️⃣ CloudNativePG Setup 🐘

🏡 Primary at Home Site

Apply primary manifest (truncated):

spec:
  instances: 2
  storage: { size: 200Gi, storageClass: longhorn }
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: site
                operator: In
                values: ["home"]

🏚 & 🏠 Replica Clusters (async)

Each replica points to home RW service and stays pinned locally.

8️⃣ Topology-Aware Reads 📍

Patch read-only services:

kubectl -n data annotate svc appdb-home-ro  service.kubernetes.io/topology-mode=auto
kubectl -n data annotate svc appdb-site2-ro service.kubernetes.io/topology-mode=auto
kubectl -n data annotate svc appdb-site3-ro service.kubernetes.io/topology-mode=auto

9️⃣ App DSNs 🎯

Environment variables:

WRITE_DSN=postgres://...@appdb-home-rw...
READ_DSN=postgres://...@appdb-siteX-ro...

Your application router decides read vs write.

🔟 Local-Only Cluster-B at Home 🧱

Install k3s normally.
Optional: run your own CNPG here too.

🎉 At This Point You Have:

✅ Global multi-site k3s mesh
✅ Local-only cluster for homelab apps
✅ Joker DDNS for site discovery
✅ VPN-backed private cluster network
✅ CNPG primary + async replicas
✅ Local read affinity
✅ Cloud ingress + local per-site ingress

💡 Tips

Need	Solution
Local-only site services	MetalLB w/ site pools
Public HTTPS	Traefik + cert-manager
Zero trust	Tailscale ACLs
DR	Promote CNPG replica
GitOps	Flux across clusters

The Daily Kebab

The ramblings of a technomuse

Super K3S internet test setup