From 9eb80899e109a248a6b7975f938790faccce6832 Mon Sep 17 00:00:00 2001 From: File Magic Date: Thu, 13 Aug 2026 19:17:24 -0400 Subject: [PATCH] iso-build.md [13-inch-thin-cannon/docs]: save point + start-next checklist (boot verify failures, watchdog verdict) --- 13-inch-thin-cannon/docs/iso-build.md | 68 +++++++++++++++++++++------ 1 file changed, 53 insertions(+), 15 deletions(-) diff --git a/13-inch-thin-cannon/docs/iso-build.md b/13-inch-thin-cannon/docs/iso-build.md index 224dcca..e7cb185 100644 --- a/13-inch-thin-cannon/docs/iso-build.md +++ b/13-inch-thin-cannon/docs/iso-build.md @@ -261,8 +261,10 @@ before the closure finder runs). TCO reset being enabled by default (QEMU ≥ 8.0), a device open/release during the delayed initrd boot can't stop the watchdog → `watchdog: watchdog0: watchdog did not stop!` → kernel `emergency_restart()`. The prompt delays the - driver's init, which is why it's the first boot that dies. Full analysis + - mitigation status in + driver's init, which is why it's the first boot that dies. **Tested verdict + (2026-08-13):** `-global ICH9-LPC.noreboot=on` does **NOT** prevent it — the + reboot is a guest software `emergency_restart()`, not the QEMU TCO hardware + reset. Full analysis + mitigation status in `kubernetes/docs/debugging.md` ("Intermittent initrd watchdog reboot"). - **Secure-boot auto-enroll causes an extra reboot.** First boot after install logs `Enrolling secure boot keys from directory: \loader\keys\auto` → @@ -274,9 +276,18 @@ before the closure finder runs). `persist-persistent-etc-passwd.service`, `persist-persistent-etc-group.service`, `persist-persistent-etc-shadow.service` (persistenced can't bind-mount/link `/etc/passwd`-family into `/persistent/etc`), `home-manager-file_magic.service`, - and `wireguard-wg0.service`. Likely related to the **`persist-files` - activation snippet failing during `nixos-install`** (the impermanence - directories weren't populated at install time). + `wireguard-wg0.service`, and `swap-swapfile.swap` (`/swap/swapfile` missing). + Likely related to the **`persist-files` activation snippet failing during + `nixos-install`** (the impermanence directories weren't populated at install + time). Also: `/etc/nixos` is **not persisted on the installed system** + (`test -f /etc/nixos/configuration.nix` fails) — it WAS verified working in an + earlier session, so this is a regression or an intermittent persist race. + The `boot` phase now dumps per-unit `journalctl` on failure (see "Start next" + below) to root-cause these. +- **Harness: Secure Boot check has a parsing bug.** `sbctl status` prints + `Secure Boot: ✓ Enabled` (tab + checkmark), but the harness greps for + `"Secure Boot: Enabled"` → always FAIL even though Secure Boot is enabled. + Fix: grep for `Secure Boot:.*Enabled`. - **Diagnosis technique that worked:** QEMU monitor `screendump` + tesseract OCR and serial SysRq-trigger to prove kernel liveness — details in `kubernetes/docs/debugging.md` ("Boot-blocker diagnosis without a working @@ -309,22 +320,49 @@ before the closure finder runs). ## Current status & missing work +**Save point (2026-08-13, late):** `just` is installed (`home.nix`), the +justfile now has `build`/`debug`/`live`/`install`/`boot`/`up`/`ovmf`/`help` +targets and a fixed `ovmf` recipe, and `iso-test.sh` accepts `QEMU_EXTRA=...` +for debug flags plus dumps per-unit journals on boot-verification failure. The +watchdog-reboot verdict is in (guest software reboot; see `debugging.md`). +Uncommitted at this save point: `home.nix` (just), `justfile`, `iso-test.sh`, +and the two doc updates (this file + `debugging.md`). + - **Green:** `live` phase, `install` phase (`disko-install succeeded`, exit 0, `RAM=10240`). -- **In progress (2026-08-13):** `boot` phase — the closure-finder blocker is - fixed (persistent `/nix` subvolume + `neededForBoot`) and boot now works - (SSH up, impermanence mounts verified, Secure Boot enabled). Remaining - before green: (1) intermittent initrd watchdog reboot on some cold boots - (QEMU ICH9 TCO harness artifact — mitigation being tested, see - `debugging.md`); (2) failed services on the installed system - (`persist-*-etc-{passwd,group,shadow}`, `home-manager-file_magic`, - `wireguard-wg0` — suspected `persist-files` activation failure at install - time); (3) finish boot-phase verifications and run `just up` end-to-end, - then commit. +- **Boot phase: kernel boots, SSH comes up, but verification FAILS.** The + closure-finder fix works. Remaining failures (all reproducible via + `just boot`): + 1. Failed services: `persist-persistent-etc-{passwd,group,shadow}`, + `home-manager-file_magic`, `wireguard-wg0`, `swap-swapfile.swap`. + 2. `/etc/nixos` not persisted on the installed system (was OK earlier — + regression or persist race). + 3. Harness Secure-Boot check bug (see known issues). + 4. Intermittent initrd watchdog self-reboot (harness survives via retry; + real hardware unaffected). - **Deferred hardening:** `hardened.apparmor.enable` and `hardened.kernel.hardened` are one-liners on the host config (enabled on the ISO, off on the machine "for now"). +### Start next (ordered) + +1. **Gather the failure detail.** Run `just boot`; the harness now prints + `journalctl -n 6` per failed unit plus `/etc/nixos`, `/persistent/etc` and + `/swap` state. (The last run was aborted before this landed.) +2. **Fix the harness Secure Boot check** — `sbctl status` emits + `Secure Boot: ✓ Enabled`; change the grep to `Secure Boot:.*Enabled`. +3. **Root-cause + fix `/etc/nixos` not persisted and the failed services** — + expected to trace back to the `persist-files` activation failure during + `nixos-install` (impermanence directories empty on the installed disk). + Verify the same `persist-files` snippet also handles `/etc/nixos`. +4. **Confirm harness tolerance of the watchdog self-reboot** — run `just boot` + 3-5× cold; it should pass every time via the second boot. If it ever fails, + fall back to `boot.blacklistedKernelModules = [ "iTCO_wdt" ]` on the host + config (documented in `debugging.md`). +5. **Reach green then `just up` end-to-end** (build → live → install → boot). + The pending changes (`home.nix`, `justfile`, `iso-test.sh`, doc updates) + are a natural first commit once the boot phase is green. + ## Useful links - [disko](https://github.com/nix-community/disko) — declarative disk layout + -- 2.51.2