Boot Time Optimization #

Baseline (2026-03-13) #

Total boot to multi-user: ~10.4s (kernel 1.2s + userspace 9.2s)

systemd-analyze blame (top offenders) #

Service	Time	Notes
dhcpcd.service	5.733s	Critical path — blocks network-online.target
growpart.service	1.445s	Partition growing
dev-vda1.device	1.261s	Device detection
sshd-keygen.service	1.048s	Generates host keys on every boot
epi-init.service	1.031s	epi guest initialization
firewall.service	683ms	nftables firewall — unnecessary in VM behind passt
systemd-udev-trigger.service	668ms	Coldplug all udev devices
resolvconf.service	378ms	DNS resolver config
systemd-timesyncd.service	371ms	NTP time sync
systemd-oomd.service	356ms	OOM killer
systemd-journald.service	345ms	Journal

Critical chain to multi-user.target #

multi-user.target @9.172s
└─epi-init-hooks.service @9.124s +47ms
  └─network-online.target @9.121s
    └─dhcpcd.service @3.387s +5.733s     ← BOTTLENECK

Critical chain to sshd.service #

sshd.service +18ms
└─epi-init.service @2.975s +1.031s
  └─basic.target @2.971s

Note: sshd starts at ~4s, but SSH won't succeed until networking (dhcpcd) completes.

Loaded kernel modules: 69 #

Many unnecessary modules loaded: cfg80211 (wifi), atkbd/serio (keyboard), vmw_vsock/vmci (VMware), nftables stack, edac_core, intel_rapl, dm_mod, loop, isofs/cdrom, autofs4, etc.

Active services: 39 #

---

Optimizations Applied #

1. Replace dhcpcd with systemd-networkd (biggest win) #

• Actual savings: ~4.5s (dhcpcd 5.7s → networkd-wait-online 1.3s)
• Why: dhcpcd dominated the critical path due to ARP probing (~5s) and sequential IPv6 negotiation. systemd-networkd acquires DHCP in parallel with other boot tasks and doesn't ARP-probe by default.
• Change: networking.useNetworkd = true; (with networking.useDHCP = true)
• Bonus: Networking is no longer on the critical path — networkd finishes before epi-init, so multi-user.target no longer waits on DHCP.

2. Disable firewall #

• Actual savings: ~700ms (removed from boot entirely)
• Why: The VM runs behind passt (user-mode networking). The host controls port forwarding — only explicitly mapped ports are reachable. A guest firewall is redundant and loads the entire nftables kernel module stack.
• Change: networking.firewall.enable = false;

3. Disable unnecessary services #

• Services disabled:
  • getty@ — no VGA/virtual console needed (serial-getty kept for epi console)
  • logrotate — already disabled before this work

4. Blacklist unnecessary kernel modules #

• Actual savings: ~200ms (69 → ~35 modules loaded)
• Why: The default NixOS kernel config auto-loads dozens of modules the VM will never use. Each module load adds overhead during boot.
• Modules blacklisted: cfg80211/rfkill (wireless), ccp (AMD crypto), atkbd/libps2/serio (keyboard), vmw_vsock/vmw_vmci (VMware), intel_rapl (power mgmt), edac_core (ECC), evdev/mac_hid (input), dmi_sysfs, qemu_fw_cfg, autofs4, dm_mod, loop, efi_pstore, vivaldi_fmap, 8021q (VLANs)
• Kept: kvm/kvm_amd (nested virt), button/tiny_power_button (ACPI shutdown), af_packet (raw sockets), isofs/cdrom (seed ISO)

---

Results #

Metric	Before	After	Improvement
Total boot (kernel + userspace)	~10.4s	~5.0s	52% faster
Kernel boot	1.2s	1.2s	—
Userspace to multi-user	9.2s	3.8s	5.4s faster
DHCP (critical path)	5.7s	1.3s	4.4s faster
firewall.service	683ms	removed	683ms

All 13 e2e tests pass (23s, down from 28s).

New critical chain #

multi-user.target @4.6s
└─epi-init-hooks.service @4.5s +78ms
  └─epi-init.service @3.6s +852ms
    └─basic.target @3.6s

Networking is no longer the bottleneck — epi-init is now the critical path.

---

Shutdown fixes (2026-03-14) #

Two bugs prevented graceful guest shutdown:

1. ExecStop script shebang failed in systemd context #

• Symptom: shutdown.sh exited with status 127 — env: 'sh': No such file or directory
• Cause: #!/usr/bin/env sh requires sh in PATH, but systemd transient services run with a minimal environment where it's not available.
• Fix: Resolve sh to its absolute nix store path at script generation time (e.g. #!/nix/store/.../bin/sh), matching how ch-remote/timeout/tail were already resolved.

2. Stopping the slice bypassed ExecStop #

• Symptom: systemctl --user stop completed in 0.15s — no graceful shutdown.
• Cause: stop_instance stopped the systemd slice (kills all cgroup processes immediately) instead of the VM service (which triggers ExecStop).
• Fix: Stop the VM service first (triggers ACPI power-button → guest shutdown), then stop the slice to clean up helper units (passt, virtiofsd).
• Result: Graceful shutdown now takes ~7s (ACPI power-button → guest systemd shutdown → VM exit → helper cleanup).

3. shutdown-vmm exit code #

• shutdown-vmm returns non-zero when the VM has already exited (successful ACPI shutdown). Added || true to prevent spurious service failure status.

---

Future opportunities #

• Pre-generate SSH host keys: sshd-keygen.service takes ~1s per boot. Host keys could be baked into the image or generated during epi-init (saved in the overlay).
• Reduce initrd: The initrd could be trimmed to only include virtio drivers.
• Custom kernel config: A minimal kernel for cloud-hypervisor VMs (virtio-only, no USB/sound/graphics/SCSI) could reduce kernel boot time further.
• Parallel epi-init: The epi-init script runs sequentially; some operations (user creation, SSH key setup) could potentially run in parallel.