diff --git a/kubernetes/docs/superpowers/specs/2026-07-16-kubernetes-security-hardening-design.md b/kubernetes/docs/superpowers/specs/2026-07-16-kubernetes-security-hardening-design.md new file mode 100644 --- /dev/null +++ b/kubernetes/docs/superpowers/specs/2026-07-16-kubernetes-security-hardening-design.md @@ -0,0 +1,245 @@ +# Kubernetes Security Hardening Design + +**Date:** 2026-07-16 +**Status:** Approved +**Approach:** Bottom-up, incremental + +## Context + +The Kubernetes cluster is a testing ground for eventually deploying something real. It currently has solid host-level hardening (NixOS firewall, SSH key-only, kernel sysctls) and API server hardening (anonymous auth off, NodeRestriction, audit logging), but several significant security gaps exist. + +### Current Gaps + +| Gap | Severity | Detail | +|-----|----------|--------| +| Encryption at rest NOT active | HIGH | `encryption-config.yaml` exists but has a placeholder key. Secrets stored in etcd are unencrypted. | +| API token hardcoded in flake.nix | HIGH | `k8sApiToken` is committed to git in plaintext. | +| SSH private key in repo | HIGH | `kubernetes/ssh-key` is tracked in git. | +| No NetworkPolicy enforcement | MEDIUM | Cilium supports NetworkPolicy but none are deployed. All pods can communicate with all other pods. | +| No Pod Security Standards labels | MEDIUM | PodSecurity admission is enabled but defaults to `privileged` for all namespaces. | +| No etcd peer/client TLS hardening | MEDIUM | etcd runs with easyCerts but no explicit `--peer-client-cert-auth` or `--client-cert-auth` flags. | +| Ceph replication size = 1 | MEDIUM | No data redundancy. Acceptable for dev/test only. | +| No image scanning/verification | LOW | `AlwaysPullImages` was removed. No container image signing or admission policy. | +| No RBAC restrictions beyond NodeRestriction | LOW | No custom ClusterRoles, Roles, or RoleBindings defined. | +| No OPA/Gatekeeper or Kyverno | LOW | Only built-in PodSecurity admission is used. | +| Hubble relay + UI disabled | LOW | Network observability is unavailable. | +| No audit log shipping | LOW | Audit log is configured but no alerting or external shipping. | + +## Design + +### Layer 1: Secrets Management (agenix/sops-nix) + +**Goal:** Remove all plaintext secrets from the git repo. + +**What gets encrypted:** +- `k8sApiToken` (currently hardcoded in `flake.nix`) +- `ssh-key` (currently tracked in git) +- Encryption-at-rest key for etcd + +**Implementation:** +- Add `agenix` as a Nix flake input +- Create `secrets.nix` defining secrets and which nodes can access them +- Store encrypted secrets in `kubernetes/secrets/` (safe to commit) +- On VM build/provision, agenix decrypts secrets using the node's SSH host key +- Remove `k8sApiToken` from `flake.nix` and `ssh-key` from tracked files + +**Files changed:** +- `flake.nix` — add agenix input, remove hardcoded token +- `secrets.nix` — new, defines secret permissions +- `kubernetes/secrets/` — new directory with encrypted files +- `common.nix` — import agenix module, configure secrets path +- `.gitignore` — ignore decrypted secrets + +### Layer 2: PKI Consolidation (cfssl) + etcd Hardening + +**Goal:** All TLS certificates managed through cfssl, etcd hardened with mutual TLS. + +**What changes:** +- Extend cfssl to generate certs for: API server, etcd (peer + client), kubelet (per-node), controller-manager, scheduler, admin, service accounts +- Remove easyCerts dependency — cfssl becomes the single source of truth +- Harden etcd flags: `--peer-client-cert-auth=true`, `--client-cert-auth=true`, `--trusted-ca-file`, `--peer-trusted-ca-file` +- Automate cert distribution via Ansible + +**Cert hierarchy:** +``` +Root CA +├── API Server cert +├── etcd peer cert +├── etcd client cert +├── kubelet cert (per node) +├── controller-manager cert +├── scheduler cert +└── admin cert +``` + +**Files changed:** +- `pki/generate.sh` — extend to generate full hierarchy, add etcd peer certs +- `master.nix` — remove easyCerts, use cfssl-generated certs, add etcd TLS flags +- `worker.nix` — remove easyCerts, use cfssl-generated kubelet certs +- `ansible/` — add cert distribution role +- `secrets.nix` — add CA private key and cert secrets + +### Layer 3: Cilium mTLS + +**Goal:** Enable Cilium mutual authentication for pod-to-pod encryption. + +**What changes:** +- Enable Cilium mutual authentication in Helm values: `authentication.mutual.spire.enabled=true` (or `authentication.mutual.enabled=true` for Cilium's built-in SPIFFE) +- Ensure Cilium is compiled with `authenticationMutual=true` (default in v1.16+) +- No sidecars needed — Cilium handles cert rotation at the eBPF level + +**Integration with existing Cilium config:** +- Current: Cilium v1.16.5, eBPF masquerade, cluster-pool IPAM, VXLAN overlay +- Add: mutual authentication enabled +- Keep: existing network policies (Cilium enforces them) + +**Files changed:** +- `ansible/cilium.yml` — add `authentication.mutual.enabled=true` to Helm values + +### Layer 4: NetworkPolicies + +**Goal:** Default-deny pod traffic, allow only what's needed. + +**What changes:** +- Create default-deny ingress + egress policies in all namespaces +- Allow DNS resolution (kube-dns/CoreDNS) for all pods +- Allow intra-namespace traffic by default +- Allow traffic from kube-system to all namespaces (for Cilium, CoreDNS, etc.) + +**Policy structure:** +``` +default-deny-ingress (all namespaces) +default-deny-egress (all namespaces) +allow-dns (all namespaces — UDP/TCP 53 to kube-system) +allow-namespace-traffic (all namespaces — intra-namespace) +allow-system-components (kube-system — allow all) +``` + +**Files changed:** +- `kubernetes/ansible/network-policies.yml` — new playbook to apply default-deny policies +- `kubernetes/ansible/site.yml` — add network-policies step after Cilium install + +### Layer 5: Pod Security Standards + +**Goal:** Enforce restrictive security posture per-namespace instead of default `privileged`. + +**What changes:** +- Label namespaces with Pod Security Standards: + - `kube-system`: `privileged` (Cilium, CoreDNS need it) + - `default`: `restricted` + - `rook-ceph`: `privileged` (Rook needs it) + - `gatekeeper-system`: `restricted` + - New namespaces: inherit `restricted` by default + +**Namespace labels:** +``` +kube-system: pod-security.kubernetes.io/enforce=privileged +kube-system: pod-security.kubernetes.io/audit=privileged +default: pod-security.kubernetes.io/enforce=restricted +default: pod-security.kubernetes.io/audit=restricted +rook-ceph: pod-security.kubernetes.io/enforce=privileged +rook-ceph: pod-security.kubernetes.io/audit=privileged +gatekeeper-system: pod-security.kubernetes.io/enforce=restricted +gatekeeper-system: pod-security.kubernetes.io/audit=restricted +``` + +**Files changed:** +- `kubernetes/ansible/namespace-labels.yml` — new playbook to label namespaces +- `kubernetes/ansible/site.yml` — add namespace-labels step + +### Layer 6: OPA/Gatekeeper + +**Goal:** Policy-as-code admission control for complex rules beyond Pod Security Standards. + +**What changes:** +- Install Gatekeeper via Helm in `gatekeeper-system` namespace +- Enable audit + webhook +- Deploy constraint templates for common policies: + - `K8sPSPPrivilegedContainer` — prevent privileged containers outside kube-system + - `K8sPSPAllowPrivilegeEscalation` — deny privilege escalation + - `K8sPSPCapabilities` — restrict Linux capabilities + - `K8sPSPHostNamespace` — prevent host network/PID/IPC + - `K8sPSPVolumeTypes` — restrict volume types (no hostPath in production) + - `K8sRequiredLabels` — enforce required labels on namespaces + - `K8sBlockNodePort` — prevent NodePort services + - `K8sBlockLoadBalancer` — prevent LoadBalancer services (local cluster) + +**Constraint templates chosen:** +- Covers CIS Kubernetes Benchmark Level 1 controls +- Avoids over-blocking for dev/test (allows hostPath for now, restricts in production) + +**Files changed:** +- `kubernetes/ansible/gatekeeper.yml` — new playbook to install Gatekeeper + constraints +- `kubernetes/ansible/site.yml` — add gatekeeper step after Pod Security Standards + +### Layer 7: Ceph Replication + +**Goal:** Increase Ceph replication from 1 to 3 for data redundancy. + +**What changes:** +- Update `CephBlockPool` replication size from 1 to 3 +- Requires at least 3 workers with data disks (you have 3 by default) +- Add min_size=2 for read availability during failures + +**Configuration:** +```yaml +replicated: + size: 3 + min_size: 2 +``` + +**Files changed:** +- `kubernetes/ansible/rook-ceph.yml` — update CephBlockPool replication settings + +### Layer 8: Audit Log Shipping (Wazuh) + +**Goal:** Ship all security-relevant logs to Wazuh for monitoring and analysis. + +**What changes:** +- Deploy Wazuh manager + indexer + dashboard in the cluster +- Deploy Wazuh agent as DaemonSet on all nodes +- Configure Wazuh to collect: + - K8s audit logs (`/var/log/kubernetes/audit.log`) + - Node system logs (`/var/log/syslog`, `journalctl`) + - Container logs (via CRI) + - Cilium Hubble flow logs +- Enable Wazuh's K8s integration for pod/container monitoring + +**Architecture:** +``` +Wazuh Manager (Deployment) +├── Wazuh Indexer (StatefulSet — Elasticsearch-based) +├── Wazuh Dashboard (Deployment) +└── Wazuh Agent (DaemonSet — one per node) +``` + +**Files changed:** +- `kubernetes/ansible/wazuh.yml` — new playbook to deploy Wazuh stack +- `kubernetes/ansible/cilium.yml` — enable Hubble relay + UI (feeds into Wazuh) +- `kubernetes/ansible/site.yml` — add wazuh step after Gatekeeper + +## Implementation Order + +| Step | Layer | Dependencies | Estimated Effort | +|------|-------|-------------|-----------------| +| 1 | Secrets (agenix/sops-nix) | None | Medium | +| 2 | PKI (cfssl) + etcd TLS | Layer 1 (needs agenix for CA key) | High | +| 3 | Cilium mTLS | None (parallel with Layer 1-2) | Low | +| 4 | NetworkPolicies | Layer 3 (Cilium must be running) | Low | +| 5 | Pod Security Standards | None (parallel with Layer 3-4) | Low | +| 6 | OPA/Gatekeeper | Layer 5 (PSS labels must exist) | Medium | +| 7 | Ceph Replication | None (parallel with Layer 3-6) | Low | +| 8 | Audit Logging (Wazuh) | Layers 3-6 (needs stable cluster) | High | + +## Success Criteria + +- [ ] No plaintext secrets in git (API token, SSH key, encryption key) +- [ ] etcd encryption at rest active with proper key management +- [ ] All TLS certificates managed through cfssl +- [ ] etcd hardened with mutual TLS +- [ ] Cilium mTLS enabled for pod-to-pod encryption +- [ ] Default-deny NetworkPolicies in all namespaces +- [ ] Pod Security Standards enforced per-namespace +- [ ] OPA/Gatekeeper deployed with CIS benchmark constraints +- [ ] Ceph replication size = 3 +- [ ] Wazuh deployed and collecting audit logs