diff --git a/kubernetes/docs/debugging.md b/kubernetes/docs/debugging.md index 37711a9..ce682d5 100644 --- a/kubernetes/docs/debugging.md +++ b/kubernetes/docs/debugging.md @@ -824,6 +824,116 @@ systemd.tmpfiles.rules = [ --- +## 2026-08-13: Boot-blocker diagnosis without a working serial console (13-inch-thin-cannon) + +### Symptom +- Installed 13-inch-thin-cannon boots to a LUKS prompt, passphrase is **accepted** (no + re-prompt), then the serial console goes silent. SSH never comes up (watched to 240 s). +- Serial log frozen at ~3.3 KB (prompt + masked bullets). VGA screendump looks + near-black to a human eye. Looks exactly like a kernel hang. + +### Technique: QEMU monitor screendump + tesseract OCR (the breakthrough) +When the serial console is silent but the kernel is alive, read the **VGA screen** +via the QEMU monitor and OCR it. This is the non-libvirt analogue of the existing +`virsh qemu-monitor-command screendump` trick: + +```bash +# 1. Screendump from the QEMU monitor TCP port (MONITOR_PORT=5555): +printf 'screendump /tmp/opencode/screen.ppm\n' | timeout 3 nc 127.0.0.1 5555 +# (or: bash -c 'exec 3<>/dev/tcp/127.0.0.1/5555; echo "screendump /tmp/opencode/screen.ppm" >&3') + +# 2. OCR with tesseract (path is a pinned nix-store binary): +/nix/store/2ac32mdm2rdcy5fx6pma5jckpnfa2pkz-tesseract-5.5.2/bin/tesseract \ + /tmp/opencode/screen.ppm stdout --psm 6 + +# 3. Optional: scale up 2x first (tesseract needs ~10px glyphs): +/nix/store/clx4xcbrg16dhzyzm2faxjddfk9rz979-imagemagick-7.1.2-27/bin/magick \ + /tmp/opencode/screen.ppm -resize 200% -colorspace Gray /tmp/opencode/screen2x.ppm +``` + +What OCR revealed (the serial console never printed these): +``` +[FAILED] Failed to start Find NixOS closure. +[DEPEND] Dependency failed for Initrd Default Target. +You are in emergency mode. After logging in, type "journalctl -xb"... +``` +The system was **NOT hung** — it had dropped to the initrd emergency shell on VGA. +Root cause chain: the disk had **no `/nix` btrfs subvolume**, so the Nix store lived +inside `/root`, which the impermanence rollback wipes and replaces with `root-blank` +every boot. `initrd-find-nixos-closure` (`RequiresMountsFor = /sysroot/nix/store`, +nixpkgs `initrd.nix:688`) had no mount unit for that path → dependency failed → +emergency shell. Fix: a persistent `/nix` subvolume (disko.nix) + `neededForBoot = true` +(→ `x-initrd.mount`) so the initrd mounts the store before the closure finder runs. + +### Technique: serial SysRq trigger to prove kernel liveness +When "no output after unlock" looks like a hang, send a SysRq over the serial console +via the QEMU monitor to prove the kernel is alive: + +```bash +printf 'sendkey ctrl-alt-sysrq-t\n' | timeout 3 nc 127.0.0.1 5555 +# 40 s later the kernel echoed to serial: "sysrq: Show State" +``` +`sysrq: Show State` is the expected serial-side header. Note the **task dump itself +went to VGA (tty0), not serial** — the `console=ttyS0 console=tty0` kernel param sends +the dump to both, but systemd had already switched the display path, so OCR the VGA +screendump to read the dump. Combined with a QEMU `info registers` (three samples, +`HLT=1` in EFLAGS = idle loop, not a spin/panic), this distinguishes "hung kernel" +from "boot failed, console not echoing". + +### Technique: PPM → ASCII renderer for quick screen eyeball +`/tmp/opencode/ppm2ascii.sh` renders a QEMU screendump as ASCII art with color-coded +glyph blocks, letting you "see" the VGA layout in a terminal without opening the image +(helpful to find the region OCR should target and to confirm text-vs-blank). + +## 2026-08-13: Intermittent initrd watchdog reboot (13-inch-thin-cannon) + +### Symptom +- The **first** cold boot of the installed 13-inch-thin-cannon VM sometimes dies + ~1-2 s into the initrd (right after the LUKS prompt is unlocked). Serial log: + ``` + [ 10.827277] kvm_intel: VMX not supported by CPU 7 + [ 14.599504] watchdog: watchdog0: watchdog did not stop! + [ 14.836988] watchdog: watchdog0: watchdog did not stop! + [ 15.322382] watchdog: watchdog0: watchdog did not stop! + [ 15.328746] reboot: Restarting system + ``` + The VM reboots itself; the next boot always survives. Intermittent — the same + disk/binary sometimes boots straight through to SSH (verified working). + +### Root cause +- QEMU's **q35 machine emulates an Intel ICH9 TCO watchdog** (`iTCO_wdt` driver). + Real 13-inch-thin-cannon hardware has no such device, so this is a harness + artifact, not a config bug. +- Kernel commit `1ae3e78c` ("watchdog: iTCO_wdt: No need to stop the timer in + probe") stopped the driver from clearing NO_REBOOT / stopping the TCO timer at + probe. QEMU ≥ 8.0 enables the TCO reboot-pin strap by default (commit `a6b6414`), + so an armed TCO genuinely resets/reboots the guest. +- During the delayed initrd (the LUKS prompt holds the kernel ~10 s), the watchdog + device gets opened and released before the driver can stop it → kernel watchdog + core prints `watchdog did not stop!` and falls through to `emergency_restart()` + (`reboot: Restarting system`). The LUKS delay is why the *first* boot dies — the + prompt pushes the release past the TCO window. +- Intermittency: the QEMU TCO timer state at power-on varies, and a fast initrd + beats the window (verified: surviving boots log `iTCO_wdt ... initialized. + heartbeat=30 sec`). + +### Fix (status: mitigation under test, 2026-08-13) +- Harness-side: `-global ICH9-LPC.noreboot=on` disables the TCO/chipset reset + (see also generic `-no-reboot` for distinguishing hardware-reset vs + guest-software-reboot). Test run was aborted before the verdict landed. +- Guest-side fallback: `boot.blacklistedKernelModules = [ "iTCO_wdt" ]` so the + driver never probes (RHEL-documented mitigation for the same symptom on real + hardware). Not yet applied — prefer the harness fix so the guest config stays + identical to real hardware. + +### Technique: distinguishing the reboot type +- `-global ICH9-LPC.noreboot=on`: stops TCO/hardware-chipset resets. +- `-no-reboot`: VM exits instead of restarting on a guest-initiated reboot + (kernel `emergency_restart`/reboot syscall). If a reboot survives the `noreboot` + flag but is caught by `-no-reboot`, it is a software reboot, not TCO. + +--- + ## OVMF Path Discovery After NixOS rebuild, the OVMF path changes. Find it: