From da74809624802fb785427f5e3298430f84b8a2fc Mon Sep 17 00:00:00 2001 From: File Magic Date: Thu, 16 Jul 2026 20:18:57 -0400 Subject: [PATCH] docs/superpowers/plans/: add multi-cluster implementation plan --- ...2026-07-15-multi-cluster-implementation.md | 629 ++++++++++++++++++ 1 file changed, 629 insertions(+) create mode 100644 docs/superpowers/plans/2026-07-15-multi-cluster-implementation.md diff --git a/docs/superpowers/plans/2026-07-15-multi-cluster-implementation.md b/docs/superpowers/plans/2026-07-15-multi-cluster-implementation.md new file mode 100644 index 0000000..9f84658 --- /dev/null +++ b/docs/superpowers/plans/2026-07-15-multi-cluster-implementation.md @@ -0,0 +1,629 @@ +# Multi-Cluster Subnet Isolation + Configurable Topology Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Parameterize the Kubernetes cluster by `CLUSTER_INDEX` (subnet offset) and `WORKER_COUNT` (node count) so each worktree/CI can deploy isolated clusters with configurable size. + +**Architecture:** Add `cluster_index` and `worker_count` variables to tofu. Derive all IPs and node names dynamically. Pass computed values to NixOS modules via flake.nix. Expose via Makefile env vars. + +**Tech Stack:** OpenTofu (libvirt), NixOS (Nix), Make + +## Global Constraints + +- Default `cluster_index=0`, `worker_count=3` preserves current behavior (192.168.122.0/24, 4 nodes) +- `master_count` locked to 1 (etcd quorum constraint) +- `worker_count` range: 0-10 +- `cluster_index` range: 0-255 +- Existing `tofu destroy` required before applying (variable structure changed) +- All Nix files must pass `nixfmt --check` +- All tofu files must pass `tofu fmt -recursive` + +--- + +## File Structure + +| File | Responsibility | +|------|---------------| +| `tofu/variables.tf` | Define `cluster_index`, `worker_count`; generate `k8s_nodes` dynamically | +| `tofu/main.tf` | Use `local.k8s_nodes` for all resources | +| `tofu/outputs.tf` | Export computed IPs, node names | +| `tofu/inventory.tpl` | Unchanged (already templated) | +| `flake.nix` | Read env vars, compute IPs, generate `nodeConfigs` dynamically | +| `kubernetes/common.nix` | Accept `subnetThirdOctet` + `nodeHosts` args, derive `extraHosts` | +| `kubernetes/master.nix` | Accept `subnetThirdOctet` arg, derive `masterAddress` and gateway | +| `kubernetes/worker.nix` | Accept `subnetThirdOctet` arg, derive default `nodeIp` and gateway | +| `Makefile` | Expose `CLUSTER_INDEX`, `WORKER_COUNT` env vars | + +--- + +### Task 1: Tofu — Dynamic node generation + +**Files:** +- Modify: `kubernetes/tofu/variables.tf` +- Modify: `kubernetes/tofu/main.tf` +- Modify: `kubernetes/tofu/outputs.tf` + +**Interfaces:** +- Consumes: Nothing (foundational) +- Produces: `local.k8s_nodes`, `local.subnet_third_octet`, computed IPs for all resources + +- [ ] **Step 1: Replace `kubernetes/tofu/variables.tf`** + +Replace the entire file with: + +```hcl +variable "cluster_index" { + description = "Cluster subnet index (0-255). Offsets the third octet of all IPs." + type = number + default = 0 + + validation { + condition = var.cluster_index >= 0 && var.cluster_index <= 255 + error_message = "cluster_index must be between 0 and 255." + } +} + +variable "worker_count" { + description = "Number of worker nodes (0-10)" + type = number + default = 3 + + validation { + condition = var.worker_count >= 0 && var.worker_count <= 10 + error_message = "worker_count must be between 0 and 10." + } +} + +variable "image_dir" { + description = "Directory containing QCOW2 image outputs from nix build" + type = string + default = "../result" +} + +variable "ovmf_code_path" { + description = "Path to OVMF_CODE.fd firmware" + type = string +} + +variable "ovmf_vars_dir" { + description = "Directory to store per-node OVMF VARS files" + type = string + default = "/tmp" +} + +variable "data_disk_size" { + description = "Size in bytes of the Ceph data disk for each worker node" + type = number + default = 42949672960 # 40GB +} +``` + +- [ ] **Step 2: Update `kubernetes/tofu/main.tf` locals and resources** + +Replace the `locals` block (lines 20-22) and all `var.k8s_nodes` references with: + +```hcl +locals { + ovmf_code_file = "${var.ovmf_code_path}/FV/OVMF_CODE.fd" + subnet_third_octet = 122 + var.cluster_index + subnet_cidr = "192.168.${local.subnet_third_octet}.0/24" + gateway_ip = "192.168.${local.subnet_third_octet}.1" + master_ip = "192.168.${local.subnet_third_octet}.10" + + master_nodes = { + "k8s-master-0" = { + role = "master" + ip_suffix = 10 + ip = "192.168.${local.subnet_third_octet}.10" + mac = "52:54:00:00:00:10" + vcpus = 2 + memory = 4096 + } + } + + worker_nodes = { + for i in range(var.worker_count) : + "k8s-worker-${toString i}" => { + role = "worker" + ip_suffix = 11 + i + ip = "192.168.${local.subnet_third_octet}.${toString (11 + i)}" + mac = "52:54:00:00:00:${format("%02x", 11 + i)}" + vcpus = 2 + memory = 4096 + } + } + + k8s_nodes = merge(local.master_nodes, local.worker_nodes) +} +``` + +Replace every `var.k8s_nodes` with `local.k8s_nodes` throughout the file. There are 6 occurrences: +- Line 53: `for name, node in var.k8s_nodes` → `for name, node in local.k8s_nodes` +- Line 65: `for name, node in var.k8s_nodes` → `for name, node in local.k8s_nodes` +- Line 82: `for name, node in var.k8s_nodes` → `for name, node in local.k8s_nodes` +- Line 99: `for name, node in var.k8s_nodes` → `for name, node in local.k8s_nodes` +- Line 109: `keys({ for name, node in var.k8s_nodes` → `keys({ for name, node in local.k8s_nodes` +- Line 110: `var.k8s_nodes[keys(...)]` → `local.k8s_nodes[keys(...)]` +- Line 112: `for name, node in var.k8s_nodes` → `for name, node in local.k8s_nodes` +- Line 120: `for_each = var.k8s_nodes` → `for_each = local.k8s_nodes` + +Also update the network resource (lines 40-61) to use computed IPs: + +```hcl +resource "libvirt_network" "k8s" { + name = "k8s-${var.cluster_index}" + autostart = true + + bridge = { + name = "virbr-k8s-${var.cluster_index}" + } + + domain = { + name = "k8s-${var.cluster_index}.local" + } + + forward = { + mode = "nat" + } + + ips = [ + { + address = local.gateway_ip + netmask = "255.255.255.0" + family = "ipv4" + dhcp = { + ranges = [ + { + start = "192.168.${local.subnet_third_octet}.2" + end = "192.168.${local.subnet_third_octet}.254" + } + ] + hosts = [ + for name, node in local.k8s_nodes : { + mac = node.mac + ip = node.ip + name = name + } + ] + } + } + ] +} +``` + +- [ ] **Step 3: Update `kubernetes/tofu/outputs.tf`** + +Replace the entire file with: + +```hcl +output "vm_ips" { + description = "IP addresses of all K8s nodes" + value = { + for name, node in local.k8s_nodes : name => node.ip + } +} + +output "master_ip" { + description = "IP address of the master node" + value = local.master_ip +} + +output "subnet_cidr" { + description = "Subnet CIDR for this cluster" + value = local.subnet_cidr +} + +output "cluster_index" { + description = "Cluster index used for this deployment" + value = var.cluster_index +} + +output "worker_count" { + description = "Number of worker nodes" + value = var.worker_count +} + +output "node_names" { + description = "All node names" + value = keys(local.k8s_nodes) +} +``` + +- [ ] **Step 4: Format tofu files** + +Run: `cd kubernetes/tofu && tofu fmt -recursive` + +- [ ] **Step 5: Commit** + +```bash +git add kubernetes/tofu/variables.tf kubernetes/tofu/main.tf kubernetes/tofu/outputs.tf +git commit -m "tofu [kubernetes/]: add cluster_index and worker_count variables, generate nodes dynamically" +``` + +--- + +### Task 2: Flake — Dynamic node generation + env vars + +**Files:** +- Modify: `kubernetes/flake.nix` + +**Interfaces:** +- Consumes: `CLUSTER_INDEX`, `WORKER_COUNT` env vars +- Produces: `nodeConfigs`, `subnetThirdOctet`, `nodeHosts` for NixOS modules + +- [ ] **Step 1: Replace `kubernetes/flake.nix`** + +Replace the entire `let` block (lines 10-88) with: + +```nix + let + lib = nixpkgs.lib; + system = "x86_64-linux"; + pkgs = nixpkgs.legacyPackages.${system}; + + sshPublicKey = builtins.readFile ./ssh-public-key; + k8sApiToken = "8c3ff57ab1e1de5de0dd0feddb3f26ac"; + + # Read cluster configuration from environment + clusterIndex = let + envVal = builtins.getEnv "CLUSTER_INDEX"; + in if envVal == "" then 0 else builtins.fromJSON envVal; + + workerCount = let + envVal = builtins.getEnv "WORKER_COUNT"; + in if envVal == "" then 3 else builtins.fromJSON envVal; + + subnetThirdOctet = 122 + clusterIndex; + subnetPrefix = "192.168.${toString subnetThirdOctet}"; + + qemuGuestModule = { + imports = [ "${toString nixpkgs}/nixos/modules/profiles/qemu-guest.nix" ]; + fileSystems."/" = { + device = "/dev/disk/by-label/nixos"; + autoResize = true; + fsType = "ext4"; + }; + virtualisation.diskSize = 20480; + }; + + # Master node (always index 0) + masterNode = { + k8s-master-0 = { + ip = "${subnetPrefix}.10"; + modules = [ ./master.nix ]; + }; + }; + + # Worker nodes (dynamically generated) + workerNodes = lib.genAttrs + (lib.genList (i: "k8s-worker-${toString i}") workerCount) + (name: let + idx = lib.last (lib.splitString "-" name); + ipSuffix = 11 + (builtins.fromJSON idx); + in { + ip = "${subnetPrefix}.${toString ipSuffix}"; + modules = [ ./worker.nix ]; + extraConfig = { + services.kubernetes.kubelet.nodeIp = "${subnetPrefix}.${toString ipSuffix}"; + }; + }); + + nodeConfigs = masterNode // workerNodes; + + # Generate /etc/hosts content for all nodes + nodeHosts = lib.concatStringsSep "\n" ( + lib.mapAttrsToList (name: node: "${node.ip} ${name}") nodeConfigs + ); + + mkModules = + name: node: + [ + qemuGuestModule + ./common.nix + ] + ++ node.modules + ++ [ + { + networking.hostName = name; + networking.interfaces.enp1s0.ipv4.addresses = [ + { + address = node.ip; + prefixLength = 24; + } + ]; + } + ] + ++ lib.optionals (node ? extraConfig) [ node.extraConfig ]; + + mkNixosConfig = + name: node: + lib.nixosSystem { + inherit system; + modules = mkModules name node ++ [ + { + _module.args.sshPublicKey = sshPublicKey; + _module.args.k8sApiToken = k8sApiToken; + _module.args.subnetThirdOctet = subnetThirdOctet; + _module.args.nodeHosts = nodeHosts; + } + ]; + }; + + configs = lib.mapAttrs mkNixosConfig nodeConfigs; +``` + +The `in` block (lines 90-109) stays the same. + +- [ ] **Step 2: Format nix file** + +Run: `nixfmt --check kubernetes/flake.nix` + +- [ ] **Step 3: Commit** + +```bash +git add kubernetes/flake.nix +git commit -m "flake.nix [kubernetes/]: generate nodeConfigs dynamically from CLUSTER_INDEX and WORKER_COUNT" +``` + +--- + +### Task 3: NixOS modules — Accept subnet args + +**Files:** +- Modify: `kubernetes/common.nix` +- Modify: `kubernetes/master.nix` +- Modify: `kubernetes/worker.nix` + +**Interfaces:** +- Consumes: `subnetThirdOctet`, `nodeHosts` from flake.nix +- Produces: Dynamic `extraHosts`, `masterAddress`, `nodeIp`, `defaultGateway` + +- [ ] **Step 1: Update `kubernetes/common.nix` function args** + +Change line 1-8 from: +```nix +{ + config, + pkgs, + lib, + sshPublicKey, + k8sApiToken, + ... +}: +``` + +To: +```nix +{ + config, + pkgs, + lib, + sshPublicKey, + k8sApiToken, + subnetThirdOctet ? 122, + nodeHosts ? "", + ... +}: +``` + +- [ ] **Step 2: Replace `networking.extraHosts` in `kubernetes/common.nix`** + +Change lines 45-50 from: +```nix + networking.extraHosts = '' + 192.168.122.10 k8s-master-0 + 192.168.122.11 k8s-worker-0 + 192.168.122.12 k8s-worker-1 + 192.168.122.13 k8s-worker-2 + ''; +``` + +To: +```nix + networking.extraHosts = nodeHosts; +``` + +- [ ] **Step 3: Update `kubernetes/master.nix` function args** + +Change lines 1-6 from: +```nix +{ + config, + pkgs, + lib, + ... +}: +``` + +To: +```nix +{ + config, + pkgs, + lib, + subnetThirdOctet ? 122, + ... +}: +``` + +- [ ] **Step 4: Replace hardcoded IPs in `kubernetes/master.nix`** + +Replace the entire `networking` section (lines 10-17) and `services.kubernetes.masterAddress` (line 21) with: + +```nix + networking.hostName = "k8s-master-0"; + networking.interfaces.enp1s0.ipv4.addresses = [ + { + address = "192.168.${toString subnetThirdOctet}.10"; + prefixLength = 24; + } + ]; + networking.defaultGateway = "192.168.${toString subnetThirdOctet}.1"; +``` + +And change line 21: +```nix + masterAddress = "192.168.${toString subnetThirdOctet}.10"; +``` + +And change line 43: +```nix + nodeIp = "192.168.${toString subnetThirdOctet}.10"; +``` + +- [ ] **Step 5: Update `kubernetes/worker.nix` function args** + +Change lines 1-6 from: +```nix +{ + config, + pkgs, + lib, + ... +}: +``` + +To: +```nix +{ + config, + pkgs, + lib, + subnetThirdOctet ? 122, + ... +}: +``` + +- [ ] **Step 6: Replace hardcoded IPs in `kubernetes/worker.nix`** + +Change line 10: +```nix + networking.defaultGateway = "192.168.${toString subnetThirdOctet}.1"; +``` + +Change line 14: +```nix + masterAddress = "192.168.${toString subnetThirdOctet}.10"; +``` + +Change line 18: +```nix + nodeIp = lib.mkDefault "192.168.${toString subnetThirdOctet}.11"; +``` + +- [ ] **Step 7: Format nix files** + +Run: `nixfmt --check kubernetes/common.nix kubernetes/master.nix kubernetes/worker.nix` + +- [ ] **Step 8: Commit** + +```bash +git add kubernetes/common.nix kubernetes/master.nix kubernetes/worker.nix +git commit -m "common.nix, master.nix, worker.nix [kubernetes/]: accept subnetThirdOctet for dynamic IPs" +``` + +--- + +### Task 4: Makefile — Expose env vars + +**Files:** +- Modify: `kubernetes/Makefile` + +**Interfaces:** +- Consumes: `CLUSTER_INDEX`, `WORKER_COUNT` env vars +- Produces: Passes them to tofu + +- [ ] **Step 1: Update `kubernetes/Makefile`** + +Change lines 1-5 from: +```makefile +KUBE_DIR := $(patsubst %/,%,$(dir $(abspath $(lastword $(MAKEFILE_LIST))))) +TOFU := $(KUBE_DIR)/tofu +ANSIBLE := $(KUBE_DIR)/ansible +SSH_KEY := $(KUBE_DIR)/ssh-key +MASTER_IP := 192.168.122.10 +``` + +To: +```makefile +KUBE_DIR := $(patsubst %/,%,$(dir $(abspath $(lastword $(MAKEFILE_LIST))))) +TOFU := $(KUBE_DIR)/tofu +ANSIBLE := $(KUBE_DIR)/ansible +SSH_KEY := $(KUBE_DIR)/ssh-key + +# Cluster configuration (override via environment) +CLUSTER_INDEX ?= 0 +WORKER_COUNT ?= 3 +export CLUSTER_INDEX +export WORKER_COUNT + +# Derived master IP from cluster_index +MASTER_IP := 192.168.$$(shell echo $$(($$((122 + $(CLUSTER_INDEX)))))).10 +``` + +Change the `deploy` target (lines 18-20) from: +```makefile +deploy: ## Destroy old VMs and create new ones from images + cd $(TOFU) && tofu destroy -auto-approve + cd $(TOFU) && tofu apply -auto-approve -var "image_dir=../result" +``` + +To: +```makefile +deploy: ## Destroy old VMs and create new ones from images + cd $(TOFU) && tofu destroy -auto-approve + cd $(TOFU) && tofu apply -auto-approve -var "image_dir=../result" -var "cluster_index=$(CLUSTER_INDEX)" -var "worker_count=$(WORKER_COUNT)" +``` + +- [ ] **Step 2: Commit** + +```bash +git add kubernetes/Makefile +git commit -m "Makefile [kubernetes/]: expose CLUSTER_INDEX and WORKER_COUNT env vars" +``` + +--- + +### Task 5: Validation — Verify default behavior + +**Files:** None (verification only) + +**Interfaces:** +- Consumes: All previous tasks +- Produces: Confirmed working default configuration + +- [ ] **Step 1: Verify tofu fmt** + +Run: `cd kubernetes/tofu && tofu fmt -recursive -check` +Expected: No output (all files formatted) + +- [ ] **Step 2: Verify nixfmt** + +Run: `nixfmt --check kubernetes/*.nix` +Expected: No output (all files formatted) + +- [ ] **Step 3: Verify tofu validate** + +Run: `cd kubernetes/tofu && tofu init && tofu validate` +Expected: "Success! The configuration is valid." + +- [ ] **Step 4: Verify default IPs match current behavior** + +Run: `cd kubernetes/tofu && tofu console -var "cluster_index=0" -var "worker_count=3" <<'EOF' +local.k8s_nodes +EOF` +Expected: 4 nodes with IPs 192.168.122.10-13 + +- [ ] **Step 5: Verify custom cluster_index** + +Run: `cd kubernetes/tofu && tofu console -var "cluster_index=5" -var "worker_count=2" <<'EOF' +local.k8s_nodes +local.subnet_cidr +EOF` +Expected: 3 nodes with IPs 192.168.127.10-12, subnet 192.168.127.0/24 + +- [ ] **Step 6: Commit verification (if any files changed)** + +```bash +git status # Should show no changes +``` -- 2.51.2