diff --git a/PLAN.md b/PLAN.md new file mode 100644 index 000000000..d9b38813c --- /dev/null +++ b/PLAN.md @@ -0,0 +1,27 @@ +# PLAN: Making Darling Fully Capable of Running Nix + +> **Goal**: Enable Darling (macOS compatibility layer for Linux) to run the Nix +> package manager reliably, so that Linux machines can build, test, and +> cross-compile `x86_64-darwin` Nix derivations — analogous to how Wine enables +> building and testing Windows binaries on Linux. + +The full plan has been split into focused documents to keep context manageable. +See the **[plan/](./plan/)** directory for all details. + +## Quick Navigation + +| Document | Description | +|---|---| +| [plan/README.md](./plan/README.md) | **Start here** — index, priority table, effort estimates | +| [plan/00-background.md](./plan/00-background.md) | Motivation, what works today, what doesn't | +| [plan/01-blockers.md](./plan/01-blockers.md) | Detailed analysis of each blocking issue | +| [plan/02-phase0-packaging.md](./plan/02-phase0-packaging.md) | `flake.nix`, devShell, `.envrc`, NixOS module | +| [plan/03-phase1-syscalls.md](./plan/03-phase1-syscalls.md) | `setattrlist`, `renameatx_np`, `utimensat`, etc. | +| [plan/04-phase2-sandbox.md](./plan/04-phase2-sandbox.md) | `sandbox-exec` passthrough, sandbox API stubs | +| [plan/05-phase3-nix-install.md](./plan/05-phase3-nix-install.md) | Automated installer, verification, wrappers | +| [plan/06-phase4-building.md](./plan/06-phase4-building.md) | Trivial derivations → stdenv → binary substitution | +| [plan/07-phase5-daemon.md](./plan/07-phase5-daemon.md) | Multi-user mode, Directory Services stubs, launchd | +| [plan/08-phase6-ci.md](./plan/08-phase6-ci.md) | NixOS VM tests, regression suite, GitHub Actions | +| [plan/09-phase7-remote-builder.md](./plan/09-phase7-remote-builder.md) | Darling as a `nix.buildMachines` target | +| [plan/10-phase8-stretch.md](./plan/10-phase8-stretch.md) | `aarch64-darwin`, GUI testing, Hydra builder | +| [plan/11-architecture.md](./plan/11-architecture.md) | System diagram, key technical decisions, glossary | \ No newline at end of file diff --git a/plan/00-background.md b/plan/00-background.md new file mode 100644 index 000000000..ce9be953d --- /dev/null +++ b/plan/00-background.md @@ -0,0 +1,104 @@ +# Background & Motivation + +## Why This Matters + +The Nix ecosystem currently has no way to build or test `x86_64-darwin` +derivations without access to real Apple hardware (or a macOS VM that requires +macOS licensing). This is a serious limitation for: + +- **Open-source CI**: Projects that need to verify their Darwin builds cannot do + so on commodity Linux infrastructure. +- **Cross-compilation verification**: Even when cross-compiling *to* Darwin, the + resulting binaries cannot be smoke-tested without macOS. +- **Nixpkgs maintenance**: Darwin breakage often goes unnoticed until a macOS + user reports it. + +[Darling](https://www.darlinghq.org/) is an open-source Darwin/macOS +translation layer for Linux — conceptually the same as Wine, but for macOS +instead of Windows. If Darling can run the Nix package manager and reliably +execute Nix-built Darwin binaries, we unlock the ability to build and test +`x86_64-darwin` packages on Linux. + +## Prior Art + +The [nixie-dev/darling-nix](https://github.com/nixie-dev/darling-nix) project +has already demonstrated that Darling can be packaged with Nix and integrated +with NixOS module tests. Their overlay builds Darling from source with +`clangStdenv` and provides a `darling` package plus an SDK output with `ld64` +and `ar`/`ranlib` from cctools-port. + +A [blog post by ersei](https://ersei.net/en/blog/nix-all-the-way-down) +documented an end-to-end attempt at installing Nix inside Darling, identifying +concrete blockers along the way. Key findings from that effort: + +- The Nix installer fails because `xmllint`, `diskutil info`, and `dseditgroup` + are missing or unimplemented in Darling. +- The installer forces multi-user mode on Darwin; single-user mode requires + manual patching. +- `lchflags` fails with `EINVAL` during `nix-env` profile installation because + `setattrlist` is not implemented. +- Even after binary-patching `libnixstore.dylib` to skip the `lchflags` error + check, `sandbox-exec` is missing so builds fail. +- Setting `_NIX_TEST_NO_SANDBOX=1` gets past that, but then `mv` crashes on + unimplemented syscall 488 (`renameatx_np`) and `touch` segfaults. +- Various other programs (e.g. `fish`) crash with `Illegal instruction` due to + incomplete syscall coverage. +- After extensive workarounds (replacing broken coreutils in the store, removing + docs from home-manager, etc.), Nix + home-manager + neovim were eventually + made to work, but the result was fragile and required many manual + interventions. + +This plan synthesizes those findings with our own code analysis of the Darling +source tree into an actionable roadmap. + +--- + +## Current State of Affairs + +### What Works + +- Darling boots a macOS-like container with `darling shell`. +- Basic command-line utilities (`echo`, `ls`, `cp`, etc.) function. +- The Darling prefix (`~/.darling`) provides an overlayfs-backed macOS-like + filesystem hierarchy. +- DMG/XIP images can be mounted and Xcode command-line tools can be installed. +- Simple C programs can be compiled and executed using Apple's toolchain. +- Darling can be built with Nix via the `nixie-dev/darling-nix` overlay and + ships as part of upstream nixpkgs. +- The `darlingserver` provides userspace syscall translation (no kernel module + required on modern builds). + +### What Does Not Work (for Nix) + +| Issue | Root Cause | Severity | +|---|---|---| +| `lchflags()` returns `EINVAL` | `setattrlist()` not implemented | **Blocker** | +| `/usr/bin/sandbox-exec` missing | Sandbox framework is stubbed | **Blocker** | +| `mv` crashes (`Unimplemented syscall 488`) | `renameatx_np` / `renameat2` not implemented | **Blocker** | +| `touch` segfaults | Likely missing `utimensat` or file-flag syscall | **Blocker** | +| Nix installer forces multi-user on Darwin | Installer script checks `uname` | High | +| `diskutil info` not implemented | `diskutil` is a shell script supporting only `eject` | Medium | +| `xmllint` missing | Not shipped in Darling | Medium | +| `dseditgroup` / Directory Services missing | User/group management unimplemented | Medium | +| `posix_spawn` + `POSIX_SPAWN_SETEXEC` → `ENOEXEC` | Incomplete `posix_spawn` attribute support | High | +| dyld cache load errors | Shared cache not generated for prefix | Medium | +| Sporadic segfaults in various programs | Incomplete syscall/ABI coverage | High | +| Darling reports macOS 10.15 (Catalina) | Newer Nix binaries target ≥ 11.0 | Medium | + +### Relevant Source Locations in This Repo + +| Area | Path | Notes | +|---|---|---| +| Sandbox stubs | `src/sandbox/sandbox.c` | All functions return "Not implemented" or 0 | +| Sandbox library | `src/libsandbox/` | `libsandbox.1.dylib` — thin shim | +| Syscall translation | `src/external/darlingserver/` | Submodule (empty until checked out) | +| libc wrappers | `src/external/libc/` | Darwin libc with BSD syscall wrappers | +| launchd | `src/launchd/` | Process management, uses `posix_spawn` | +| diskutil | `src/diskutil/diskutil` | Shell script, only supports `eject` verb | +| duct tape shims | `src/duct/src/` | Minimal stubs for `acl`, `dns_sd`, etc. | +| Build system | `CMakeLists.txt` | Top-level; deployment target is 11.0 | +| CI (current) | `.github/workflows/actions.yaml` | Debian-only, no Nix | + +--- + +*Next: [Known Blockers →](./01-blockers.md)* \ No newline at end of file diff --git a/plan/01-blockers.md b/plan/01-blockers.md new file mode 100644 index 000000000..72f34b8ac --- /dev/null +++ b/plan/01-blockers.md @@ -0,0 +1,241 @@ +# Known Blockers + +Detailed analysis of each issue that prevents Nix from running inside Darling, +with fix strategies and pointers into the codebase. + +--- + +## B1: `lchflags` / `setattrlist` Failure + +**Symptom**: Running `nix-env` to install a package fails with: +``` +error: clearing flags of path '/nix/store/…-user-environment/bin': Invalid argument +``` + +**What happens**: Nix's store optimisation code (in `libnixstore`) calls +`lchflags(path, 0)` to clear `UF_IMMUTABLE` before garbage collection. The +relevant Nix source: + +```c +#if __APPLE__ + if (lchflags(path.c_str(), 0)) { + if (errno != ENOTSUP) + throw SysError("clearing flags of path '%1%'", path); + } +#endif +``` + +On macOS, `lchflags()` is emulated via `setattrlist(2)`. Darling does not +implement `setattrlist`, so the underlying syscall fails with `EINVAL`. + +**Location in Darling**: +- Syscall translation: `src/external/darlingserver/` (submodule) +- libc wrapper: `src/external/libc/` + +**Fix strategy**: +1. Implement `setattrlist` / `fsetattrlist` in darlingserver's BSD syscall + handler. At minimum, handle `ATTR_CMN_FLAGS` (clearing `UF_IMMUTABLE`). +2. Return success (0) for attribute sets that have no Linux equivalent but are + benign to ignore (e.g., Finder info, extended security). +3. Ensure `lchflags(path, 0)` returns 0 rather than `EINVAL`. +4. Long-term: implement a proper `UF_IMMUTABLE` ↔ `FS_IMMUTABLE_FL` mapping + via `ioctl(FS_IOC_SETFLAGS)`. + +**Workaround (from blog post)**: Binary-patch `libnixstore.dylib` — replace the +`je` (jump-if-equal) after the `lchflags` call with `jmp` (unconditional jump) +to skip the error path. This is fragile and version-specific. + +**Effort**: Medium — needs darlingserver changes + libc verification. + +--- + +## B2: Missing `sandbox-exec` + +**Symptom**: `nix-build` of any derivation fails with: +``` +error: executing '/bin/bash': Bad file descriptor +``` + +**What happens**: Nix on Darwin wraps every builder invocation with: +``` +/usr/bin/sandbox-exec -f -D _GLOBAL_TMP_DIR=... +``` + +The binary `/usr/bin/sandbox-exec` does not exist in the Darling prefix. +`posix_spawn` is called with `sandbox-exec` as the executable, which returns +`ENOEXEC`. Nix reports this misleadingly as "Bad file descriptor". + +The sandbox API in `src/sandbox/sandbox.c` is entirely stubbed: +```c +int sandbox_init(const char *profile, uint64_t flags, char **errorbuf) +{ + *errorbuf = strdup("Not implemented"); + return 0; +} +``` + +**Fix strategy** (two options, not mutually exclusive): + +- **Option A — Stub `sandbox-exec` (MVP)**: Ship a `/usr/bin/sandbox-exec` + shell script or small C program that: + - Parses `-f ` and `-D =` arguments (discards them). + - `exec`s the remaining arguments as the builder command. + - Darling already provides Linux-level isolation via namespaces and the + darlingserver container, so skipping the macOS sandbox is safe. + +- **Option B — Translate to Linux sandboxing (stretch)**: Parse Apple's Sandbox + Profile Language (Scheme-based `.sb` files) and map rules to Linux + equivalents (Landlock, seccomp-bpf, namespaces). Large effort, not needed + for MVP. + +**Workaround (from blog post)**: Set `_NIX_TEST_NO_SANDBOX=1` — this is an +internal Nix environment variable that bypasses sandbox-exec. Works but is not +meant for production use. + +**Effort**: Small for Option A (a few hours), Large for Option B (weeks). + +--- + +## B3: Unimplemented Syscall 488 (`renameatx_np`) + +**Symptom**: `mv` from Nix's coreutils crashes: +``` +Unimplemented syscall (488) +``` +This breaks derivation builds that need to move files (very common). + +**What happens**: macOS syscall 488 is `renameatx_np`, which extends `rename` +with atomic swap and exclusive-create semantics. Modern Darwin coreutils +(fetched from the Nix binary cache) use this syscall. Darling's syscall table +does not have an entry for it. + +**Fix strategy**: Implement `renameatx_np` by translating to Linux's +`renameat2(2)`. The flag mapping: + +| macOS Flag | Value | Linux Equivalent | +|---|---|---| +| `RENAME_SWAP` | `0x00000002` | `RENAME_EXCHANGE` | +| `RENAME_EXCL` | `0x00000004` | `RENAME_NOREPLACE` | + +When no flags are set, fall through to plain `renameat`. + +**Location**: darlingserver syscall table (`src/external/darlingserver/`). + +**Workaround (from blog post)**: Replace the Nix store's `mv` binary with +Darling's built-in `/bin/mv` that uses older syscalls. This is a "Nix crime" +(modifying store paths) and breaks reproducibility. + +**Effort**: Small — straightforward syscall mapping, well-defined semantics. + +--- + +## B4: `touch` / `utimensat` Crash + +**Symptom**: Running `touch` from Nix's coreutils causes: +``` +Segmentation fault: 11 (core dumped) +``` +This breaks derivation builds (e.g., neovim's build script calls `touch`). + +**What happens**: The Nix-provided `touch` (compiled for newer macOS) likely +uses `setattrlistat` or a `utimensat`-related path that is missing or buggy in +Darling. It may also be related to the `setattrlist` gap from B1 — `touch -t` +on macOS can go through `setattrlist` to set modification times. + +**Fix strategy**: +1. Audit the `utimensat` / `futimens` translation in darlingserver. +2. Ensure `UTIME_NOW` and `UTIME_OMIT` sentinel values are correctly handled. +3. If the crash is in `setattrlistat`, fixing B1 may resolve this too. +4. Test with Nix's specific coreutils version. + +**Workaround (from blog post)**: Same as B3 — replace the store's `touch` with +Darling's built-in version. + +**Effort**: Medium — needs debugging to pinpoint exact crash location. + +--- + +## B5: `posix_spawn` with `POSIX_SPAWN_SETEXEC` Returns `ENOEXEC` + +**Symptom**: Even when `sandbox-exec` or other executables exist, `posix_spawn` +with the `POSIX_SPAWN_SETEXEC` flag (which makes it behave like `exec`) can +return `ENOEXEC` for certain binaries. + +**What happens**: The `POSIX_SPAWN_SETEXEC` flag is used by Nix's sandbox setup +and by launchd (`src/launchd/src/core.c:4553`). If darlingserver doesn't fully +support this flag in its `posix_spawn` implementation, the caller gets +`ENOEXEC` and reports confusing errors. + +**Fix strategy**: Verify `POSIX_SPAWN_SETEXEC` handling in darlingserver's +`posix_spawn` implementation. Ensure it correctly replaces the current process +image (like `execve`) rather than spawning a child. + +**Effort**: Medium — requires darlingserver debugging. + +--- + +## B6: macOS Version Mismatch + +**Symptom**: Various subtle failures due to Darling reporting macOS 10.15 +(Catalina) while Nix's pre-built Darwin binaries increasingly target macOS 11.0+ +(Big Sur). + +**What happens**: The Nix binary cache serves binaries built with +`-mmacosx-version-min=11.0` or higher. These binaries may use APIs or syscalls +that were introduced in Big Sur and aren't present in Darling's Catalina-era +libraries. + +**Note**: The `CMakeLists.txt` already sets `CMAKE_OSX_DEPLOYMENT_TARGET` to +`11.0`, but the runtime environment (`sw_vers`, `SystemVersion.plist`) may still +report 10.15. + +**Fix strategy**: +1. Update `sw_vers` / `SystemVersion.plist` in the Darling prefix to report 11.0. +2. Audit `__MAC_OS_X_VERSION_MIN_REQUIRED` availability guards in Darling's + libc, libSystem, and frameworks. +3. Ensure there are no code paths gated on version checks that disable + functionality we need. + +**Effort**: Medium — version bumps can have cascading effects. + +--- + +## B7: dyld Shared Cache + +**Symptom**: Some binaries print on startup: +``` +dyld: dyld cache load error: shared cache file open() failed +``` + +**What happens**: macOS ships a pre-linked shared cache +(`/System/Library/dyld/dyld_shared_cache_x86_64`) that contains all system +libraries. Darling may not generate this cache, forcing `dyld` to fall back to +loading individual `.dylib` files. This usually works but can cause errors with +binaries that assume the cache exists. + +**Fix strategy**: +1. Determine if Darling generates a shared cache during prefix initialization. +2. If not, add a cache generation step or ensure the fallback path works + reliably. +3. This is lower priority — most Nix binaries link against Nix-provided + libraries, not system ones. + +**Effort**: Medium-to-Large depending on root cause. + +--- + +## Blocker Dependency Graph + +``` +B1 (setattrlist) ──→ B4 (touch/utimensat) may share root cause +B2 (sandbox-exec) ──→ B5 (posix_spawn) related but independent +B3 (renameatx_np) standalone +B6 (version) affects everything subtly +B7 (dyld cache) standalone, lower priority +``` + +**Recommended fix order**: B2 (quick win) → B3 (quick win) → B1 → B4 → B5 → B6 → B7 + +--- + +*[← Background](./00-background.md) | [Phase 0 — Packaging →](./02-phase0-packaging.md)* \ No newline at end of file diff --git a/plan/02-phase0-packaging.md b/plan/02-phase0-packaging.md new file mode 100644 index 000000000..7eaecf1bf --- /dev/null +++ b/plan/02-phase0-packaging.md @@ -0,0 +1,207 @@ +# Phase 0 — Nix Packaging + DevShell + +**Priority**: P0 · **Effort**: S (1–2 weeks) · **Depends on**: Nothing + +This is the foundation phase. Before any Darling hacking begins, we need a +reproducible build, a developer shell with all required tools, and editor +integration so that contributors (human and AI) can be productive immediately. + +--- + +## Tasks + +### 0.1 — Add `flake.nix` + +Create a `flake.nix` at the repo root that exposes: + +- `packages.x86_64-linux.darling` — the main Darling binary + prefix +- `packages.x86_64-linux.darling-sdk` — macOS SDK + cctools (`ld64`, `ar`, + `ranlib`) for cross-compilation + +Use the [nixie-dev/darling-nix](https://github.com/nixie-dev/darling-nix) +packaging as a reference. Their `packages/darling/default.nix` demonstrates: + +- Building with `clangStdenv` +- A `ccWrapperBypass` that detects `-target *darwin*` and calls the unwrapped + compiler to avoid `cc-wrapper` interfering with Darwin cross-compilation +- Splitting the SDK into a separate output +- Post-fixup checks that ensure no `/nix/store` paths leak into the Darling + root (which would break the prefix overlay) + +Key decisions: + +- Pin `nixpkgs` input to a recent stable release. +- Use `fetchFromGitHub` with `fetchSubmodules = true` to get all submodules + (there are 100+ in `.gitmodules`). +- Strip large test directories from the source to stay under Hydra output limits + (see the `postFetch` in the reference packaging). + +### 0.2 — Add NixOS Module + +Create `nixosModules.darling` that: + +- Ensures the Darling binary is installed. +- Configures darlingserver's userspace-only mode (no kernel module required on + modern kernels with `overlayfs` + user namespaces). +- Sets up `/etc/darling` configuration if needed. +- Optionally provides a `darling-prefix.service` systemd unit for persistent + prefixes. + +### 0.3 — Set Up Binary Cache + +- Create a [Cachix](https://cachix.org/) cache (or equivalent) for CI-built + artifacts. +- Add `nixConfig.extra-substituters` and `nixConfig.extra-trusted-public-keys` + to `flake.nix` so users automatically use the cache. +- Document the cache setup in the repo README. + +### 0.4 — Pin Submodules + +The `darlingserver` submodule (at `src/external/darlingserver/`) is empty in a +shallow checkout. Ensure the flake's `fetchFromGitHub` with +`fetchSubmodules = true` captures it, so the build is fully reproducible from +a single source fetch. + +Verify all 100+ submodules listed in `.gitmodules` are resolved. If any fail, +pin their commits explicitly. + +### 0.5 — Add `devShell` + +Add `devShells.x86_64-linux.default` to the flake. This shell must provide +every tool needed to build Darling, debug issues, and work comfortably in Zed. + +**Build dependencies** (same as `nativeBuildInputs` for the Darling package): + +- `clang` / `clangStdenv.cc` +- `cmake` +- `ninja` +- `pkg-config` +- `bison` +- `flex` +- `python3` +- `makeWrapper` + +**Runtime & library dependencies** (same as `buildInputs`): + +- `freetype`, `libjpeg`, `libpng`, `libtiff`, `giflib` +- `libX11`, `libXext`, `libXrandr`, `libXcursor`, `libxkbfile` +- `cairo`, `libglvnd`, `fontconfig`, `dbus`, `libGLU` +- `fuse`, `ffmpeg`, `pulseaudio` +- `libbsd`, `openssl` +- Linux headers (`stdenv.cc.libc.linuxHeaders`) + +**Debugging & analysis tools**: + +- `gdb` — for debugging crashes inside Darling / darlingserver +- `strace` — for tracing Linux syscalls made by darlingserver +- `rizin` — for binary analysis / patching (used in the blog post to patch + `libnixstore.dylib`) +- `file` — for identifying binary types (Mach-O vs ELF) + +**Code exploration**: + +- `ripgrep` — fast grep across the large codebase +- `fd` — fast find +- `jq` — JSON processing (useful for Nix evaluation debugging) + +**Nix tooling** (critical for Zed integration): + +- `nil` or `nixd` — Nix language server, so Zed provides completions, + diagnostics, and go-to-definition for `.nix` files +- `nixfmt-rfc-style` — Nix formatter + +**C/C++ tooling** (critical for Zed integration): + +- `clang-tools` — provides `clangd` for C/C++ language server support in Zed +- `bear` or `cmake`'s `CMAKE_EXPORT_COMPILE_COMMANDS` — for generating + `compile_commands.json` so `clangd` understands the build + +**Example structure**: + +```nix +devShells.x86_64-linux.default = pkgs.mkShell.override { stdenv = pkgs.clangStdenv; } { + packages = with pkgs; [ + # Build + cmake ninja pkg-config bison flex python3 makeWrapper + + # Libraries (for cmake to find) + freetype libjpeg libpng libtiff giflib + libX11 libXext libXrandr libXcursor libxkbfile + cairo libglvnd fontconfig dbus libGLU + fuse ffmpeg pulseaudio + libbsd openssl + + # Debug + gdb strace rizin file + + # Code exploration + ripgrep fd jq + + # Nix tooling (for Zed) + nil nixfmt-rfc-style + + # C/C++ tooling (for Zed) + clang-tools + ]; + + CMAKE_EXPORT_COMPILE_COMMANDS = "1"; +}; +``` + +### 0.6 — Add `.envrc` + +Create a `.envrc` at the repo root: + +```bash +use flake +``` + +This single line is all that's needed. When `direnv` is installed (which it +should be on any NixOS or nix-with-direnv setup), entering the project directory +will: + +1. Evaluate the flake's `devShell`. +2. Export all environment variables (paths to tools, library paths, etc.). +3. Make tools available to the shell **and** to Zed (which reads direnv state). + +**Why this matters for Zed**: Zed discovers language servers, formatters, and +other tools through the environment. Without `.envrc` + direnv, Zed won't find +`clangd`, `nil`, or `nixfmt` — meaning no LSP support, no inline errors, and +no formatting. With it, everything works automatically the moment you open the +project. + +Add `.envrc` to `.gitignore` exclusions (make sure it's NOT ignored) and add +`.direnv/` to `.gitignore` (the cache directory should be ignored). + +--- + +## Verification Checklist + +After completing Phase 0, the following should all work: + +- [ ] `nix build .#darling` produces a working Darling installation +- [ ] `nix build .#darling-sdk` produces the SDK with `ld64`, `ar`, `ranlib` +- [ ] `nix develop` drops into a shell with `cmake`, `clang`, `gdb`, `nil`, etc. +- [ ] `cd`-ing into the repo with direnv enabled loads the devShell automatically +- [ ] Opening the repo in Zed shows Nix LSP working (completions in `.nix` files) +- [ ] Opening a `.c` file in Zed shows `clangd` providing diagnostics +- [ ] `darling shell echo Hello` works from the built package +- [ ] `nix flake check` passes + +--- + +## Notes + +- The devShell is intentionally **large**. This is a complex C/C++/Objective-C + project with 100+ submodules, and developers need the full toolkit available + without hunting for dependencies. +- `CMAKE_EXPORT_COMPILE_COMMANDS=1` is set in the devShell so that any cmake + configure run produces `compile_commands.json`, which `clangd` needs. + Alternatively, contributors can run `bear -- cmake --build build/` to + generate it. +- The `.envrc` should be committed to the repo (not gitignored) so every + contributor gets the same experience. Only `.direnv/` (the cache) is ignored. + +--- + +*[← Known Blockers](./01-blockers.md) | [Phase 1 — Syscall Fixes →](./03-phase1-syscalls.md)* \ No newline at end of file diff --git a/plan/03-phase1-syscalls.md b/plan/03-phase1-syscalls.md new file mode 100644 index 000000000..4cd351449 --- /dev/null +++ b/plan/03-phase1-syscalls.md @@ -0,0 +1,333 @@ +# Phase 1 — Core Syscall & API Fixes + +**Priority**: P0 · **Effort**: L (4–8 weeks) · **Depends on**: Phase 0 + +These are the minimum changes needed for Nix binaries to not crash on startup +and for basic Nix operations (eval, install, build) to function inside Darling. + +All syscall work happens in the `darlingserver` submodule +(`src/external/darlingserver/`) and/or the libc wrappers in +`src/external/libc/`. Some fixes may also touch the XNU syscall shim layer in +`src/external/xnu/darling/src/libsystem_kernel/`. + +--- + +## Tasks + +### 1.1 — Implement `setattrlist` / `fsetattrlist` / `getattrlist` + +**Resolves**: [Blocker B1](./01-blockers.md#b1-lchflags--setattrlist-failure), +partially [B4](./01-blockers.md#b4-touch--utimensat-crash) + +**What to do**: + +Add the `setattrlist(2)` and `fsetattrlist(2)` BSD syscalls to darlingserver's +syscall handler. These are the underlying calls that `lchflags`, `chflags`, +`utimes`, and other file-metadata functions use on macOS. + +**Minimum viable implementation**: + +| Attribute Group | Attribute | Action | +|---|---|---| +| `ATTR_CMN_FLAGS` | `UF_IMMUTABLE` | Map to `FS_IMMUTABLE_FL` via `ioctl(FS_IOC_SETFLAGS)`, or silently succeed when clearing (value = 0) | +| `ATTR_CMN_FLAGS` | All other flags | Return 0 (success), ignore silently | +| `ATTR_CMN_MODTIME` | modification time | Translate to `utimensat` on the Linux side | +| `ATTR_CMN_ACCTIME` | access time | Translate to `utimensat` on the Linux side | +| `ATTR_CMN_CRTIME` | creation time | Silently ignore (ext4/btrfs don't expose birth time for writing) | +| Everything else | — | Return `ENOTSUP` or 0 depending on whether ignoring is safe | + +Also implement `getattrlist(2)` at minimum for `ATTR_CMN_FLAGS` so that +programs that read-then-modify flags don't crash. + +**Files to modify**: + +- `src/external/darlingserver/` — syscall dispatch table, new handler +- `src/external/xnu/darling/src/libsystem_kernel/` — ensure the Mach trap / + BSD syscall number is wired through +- `src/external/libc/` — verify the userspace `setattrlist()` wrapper calls the + correct syscall number + +**Testing**: Write a small C program that calls `lchflags(path, 0)` and +`setattrlist()` with `ATTR_CMN_FLAGS`. Run inside `darling shell`. Must return 0. + +--- + +### 1.2 — Fix `lchflags` Return Value + +**Resolves**: [Blocker B1](./01-blockers.md#b1-lchflags--setattrlist-failure) + +**What to do**: + +Once `setattrlist` is implemented (1.1), verify that `lchflags(path, 0)` returns +0 (success). On macOS, `lchflags` is implemented as: + +```c +int lchflags(const char *path, int flags) { + struct attrlist attrlist; + memset(&attrlist, 0, sizeof(attrlist)); + attrlist.bitmapcount = ATTR_BIT_MAP_COUNT; + attrlist.commonattr = ATTR_CMN_FLAGS; + return setattrlist(path, &attrlist, &flags, sizeof(flags), + FSOPT_NOFOLLOW); +} +``` + +Trace through `src/external/libc/` to confirm this is what Darling's libc does, +and that the `FSOPT_NOFOLLOW` option is respected (i.e., the Linux side uses +`fstatat` / `utimensat` with `AT_SYMLINK_NOFOLLOW` rather than following +symlinks). + +**Testing**: Same test program as 1.1. Additionally, copy the exact Nix +`nix-env` invocation from the blog post and verify it completes without the +"clearing flags" error. + +--- + +### 1.3 — Implement `renameatx_np` (Syscall 488) + +**Resolves**: [Blocker B3](./01-blockers.md#b3-unimplemented-syscall-488-renameatx_np) + +**What to do**: + +Add syscall 488 (`renameatx_np`) to darlingserver's BSD syscall table. Translate +directly to Linux's `renameat2(2)`. + +**Signature**: + +```c +int renameatx_np(int fromfd, const char *from, int tofd, const char *to, + unsigned int flags); +``` + +**Flag translation**: + +| macOS Flag | macOS Value | Linux Equivalent | Linux Value | +|---|---|---|---| +| `RENAME_SWAP` | `0x00000002` | `RENAME_EXCHANGE` | `(1 << 1)` | +| `RENAME_EXCL` | `0x00000004` | `RENAME_NOREPLACE` | `(1 << 0)` | +| (none / 0) | `0x00000000` | (none — use plain `renameat`) | `0` | + +**Edge cases**: + +- If both `RENAME_SWAP` and `RENAME_EXCL` are set, return `EINVAL` (same as + macOS behavior). +- If the underlying Linux filesystem doesn't support `renameat2` flags (e.g., + NFS), return `ENOTSUP`. + +**Files to modify**: + +- `src/external/darlingserver/` — add syscall 488 to the dispatch table +- `src/external/xnu/darling/src/libsystem_kernel/` — wire the BSD syscall number + +**Testing**: Write a C program that: +1. Creates two files. +2. Calls `renameatx_np` with `RENAME_SWAP` to atomically swap them. +3. Calls `renameatx_np` with `RENAME_EXCL` to rename with exclusive semantics. +4. Verifies contents are correct after each operation. + +Also verify that Nix's `mv` (from coreutils) no longer crashes. + +--- + +### 1.4 — Audit and Fix `utimensat` / `futimens` + +**Resolves**: [Blocker B4](./01-blockers.md#b4-touch--utimensat-crash) + +**What to do**: + +The Nix-provided `touch` (from coreutils, built for Darwin) segfaults. Debug +this to determine the exact failing call. Likely candidates: + +1. `utimensat` with `UTIME_NOW` or `UTIME_OMIT` sentinel values not being + translated correctly. +2. `setattrlistat` (a variant of `setattrlist` with `at`-style directory fd) not + being implemented. If `touch` uses this path, it will crash since + `setattrlist` is missing (see 1.1). +3. A NULL-pointer dereference in the syscall translation layer when handling + edge cases. + +**Debug approach**: + +```bash +# On the Linux host, trace darlingserver's syscalls: +strace -f -p $(pidof darlingserver) -e trace=utimensat,openat,fstatat 2>&1 | head -100 + +# Inside darling shell, with xtrace: +DARLING_XTRACE=1 /nix/store/.../bin/touch /tmp/testfile +``` + +**Fix**: Ensure the `utimensat` handler in darlingserver: + +- Accepts `UTIME_NOW` (`((1 << 30) - 1)` on macOS, same on Linux) and passes + it through. +- Accepts `UTIME_OMIT` (`((1 << 30) - 2)` on macOS, same on Linux) and passes + it through. +- Handles `AT_FDCWD` correctly as the directory file descriptor. +- Does not dereference NULL `timespec` pointers (which means "set to current + time" on both platforms). + +**Testing**: `touch /tmp/testfile` inside darling shell with Nix's coreutils +must not segfault. Also test `touch -t 202301011200 /tmp/testfile` (explicit +timestamp). + +--- + +### 1.5 — Implement or Stub `clonefile` / `fclonefileat` (Syscall 462) + +**Resolves**: Potential build failures when Nix optimises store copies. + +**What to do**: + +Nix uses `clonefile` on APFS for copy-on-write file duplication (much faster +than `cp`). On Linux, the equivalents are `ioctl(FICLONE)` (for btrfs/XFS) or +`copy_file_range`. + +**Implementation options** (in order of preference): + +1. **Translate to `ioctl(FICLONE)`** if the underlying Linux filesystem supports + it (btrfs, XFS). This preserves the CoW semantics. +2. **Translate to `copy_file_range`** as a fallback — not CoW but still + efficient (kernel-side copy, no userspace buffering). +3. **Return `ENOTSUP`** — Nix will fall back to regular `read`/`write` copy. + This is the simplest option and is perfectly functional, just slower. + +For MVP, option 3 is fine. Nix handles `ENOTSUP` gracefully. + +**Testing**: Call `clonefile("/tmp/src", "/tmp/dst", 0)` inside darling shell. +Verify it either succeeds or returns `ENOTSUP` (not a crash / unimplemented +syscall error). + +--- + +### 1.6 — Implement `getentropy` / `CCRandomGenerateBytes` + +**Resolves**: Potential crashes in crypto / hashing code used by Nix and its +dependencies. + +**What to do**: + +`getentropy(buf, len)` is a simple call to fill a buffer with random bytes. Map +it to Linux's `getrandom(buf, len, 0)`. This may already be implemented in +Darling — verify first. + +`CCRandomGenerateBytes` is part of CommonCrypto and calls `getentropy` under the +hood. If `getentropy` works, this should work too. + +**Verify**: + +```c +#include +int main(void) { + char buf[32]; + return getentropy(buf, sizeof(buf)); +} +``` + +Compile with Apple's clang inside darling shell, run, and check return value. + +--- + +### 1.7 — Triage Unimplemented Syscalls + +**Resolves**: Reduces "Unimplemented syscall (N)" crashes across the board. + +**What to do**: + +1. Run a Nix install + trivial build inside darling shell with syscall tracing + enabled. +2. Collect all "Unimplemented syscall" messages. +3. Map each syscall number to its name (using XNU headers / the macOS syscall + table). +4. Categorize: + - **Must fix**: Causes Nix to crash or fail. + - **Should stub**: Called but return value isn't critical (e.g., `kdebug` + tracing calls). Return 0 or `ENOTSUP`. + - **Can ignore**: Informational, doesn't affect execution. +5. File an issue or task for each "must fix" syscall. + +**Output**: A table in this repo (e.g., `plan/syscall-triage.md`) tracking: + +| Syscall # | Name | Caller | Impact | Status | +|---|---|---|---|---| +| 488 | `renameatx_np` | `mv` (coreutils) | Crash | Fixed (1.3) | +| 462 | `clonefile` | Nix store | Slow fallback | Stubbed (1.5) | +| ... | ... | ... | ... | ... | + +--- + +### 1.8 — Update Emulated macOS Version + +**Resolves**: [Blocker B6](./01-blockers.md#b6-macos-version-mismatch) + +**What to do**: + +Ensure Darling's runtime environment matches or exceeds the macOS version that +Nix's pre-built Darwin binaries target (currently 11.0 / Big Sur for most +Nixpkgs packages). + +**Check current state**: + +```bash +darling shell sw_vers +# Expected: ProductVersion: 10.15.x (Catalina) +# Desired: ProductVersion: 11.0 or higher +``` + +**Steps**: + +1. Update `SystemVersion.plist` in the Darling prefix (likely in + `src/external/files/` or generated during prefix initialization). +2. Verify `CMakeLists.txt` already sets `CMAKE_OSX_DEPLOYMENT_TARGET 11.0` + (it does — confirmed in our code analysis). +3. Audit `__MAC_OS_X_VERSION_MIN_REQUIRED` / `@available` guards in Darling's: + - `src/external/libc/` + - `src/external/corefoundation/` + - `src/external/foundation/` + - `src/external/libdispatch/` +4. Ensure no code paths are gated behind version checks that would disable + functionality Nix needs (e.g., newer filesystem calls, newer POSIX APIs). +5. Test that Nix's pre-built `x86_64-darwin` binaries (from `cache.nixos.org`) + launch without version-related `dyld` errors. + +**Risks**: Bumping the version may expose new codepaths that call unimplemented +APIs. This is acceptable — better to surface those issues now than to paper +over them with an old version number. + +--- + +## Recommended Implementation Order + +``` +1.3 (renameatx_np) — quick win, unblocks mv + ↓ +1.1 (setattrlist) — biggest impact, unblocks lchflags + possibly touch + ↓ +1.2 (lchflags verify) — verification step after 1.1 + ↓ +1.4 (utimensat) — may be resolved by 1.1, debug to confirm + ↓ +1.5 (clonefile stub) — quick, just return ENOTSUP + ↓ +1.6 (getentropy) — verify first, may already work + ↓ +1.7 (triage) — discovery task, informs remaining work + ↓ +1.8 (version bump) — do last, may surface new issues +``` + +--- + +## Verification Checklist + +After completing Phase 1, the following should all work inside `darling shell`: + +- [ ] `lchflags /tmp/testfile 0` returns success (exit code 0) +- [ ] `mv /tmp/a /tmp/b` works with Nix's coreutils `mv` (no "Unimplemented syscall") +- [ ] `touch /tmp/testfile` works with Nix's coreutils `touch` (no segfault) +- [ ] A pre-built Nix binary from `cache.nixos.org` launches without dyld errors +- [ ] `sw_vers` reports macOS 11.0 or higher +- [ ] No "Unimplemented syscall" messages for syscalls in the critical path + +--- + +*[← Phase 0 — Packaging](./02-phase0-packaging.md) | [Phase 2 — Sandbox →](./04-phase2-sandbox.md)* \ No newline at end of file diff --git a/plan/04-phase2-sandbox.md b/plan/04-phase2-sandbox.md new file mode 100644 index 000000000..c9724e923 --- /dev/null +++ b/plan/04-phase2-sandbox.md @@ -0,0 +1,288 @@ +# Phase 2 — Sandbox & Build Isolation + +**Priority**: P0 · **Effort**: S (1 week) · **Depends on**: Nothing (can be done in parallel with Phase 1) + +Nix on Darwin relies heavily on `/usr/bin/sandbox-exec` for build isolation. +Every derivation builder is wrapped with it. Since Darling doesn't ship this +binary and the entire sandbox API is stubbed, this is a hard blocker for any +`nix-build` invocation. + +The good news: this is one of the **quickest wins** in the entire plan. A simple +passthrough stub is sufficient because Darling already runs inside a Linux-level +container with namespace isolation. + +--- + +## Context + +When Nix builds a derivation on Darwin, it does roughly this: + +``` +posix_spawn(NULL, "/usr/bin/sandbox-exec", + { attributes = POSIX_SPAWN_SETEXEC, file_actions = {} }, + {"sandbox-exec", "-f", "/tmp/nix-build-foo.drv-0/.sandbox.sb", + "-D", "_GLOBAL_TMP_DIR=/tmp", + "/bin/bash", "-e", "/nix/store/...-builder.sh"}, + {env...}) +``` + +The sandbox profile (`.sb` file) is a Scheme-based DSL that restricts file +access, network access, and process operations. Example from Nix: + +```scheme +(version 1) +(allow default) +; Disallow creating setuid/setgid binaries +(deny file-write-setugid) +``` + +Nix generates these profiles dynamically per-build. The `sandbox-exec` binary +reads the profile, applies the restrictions via macOS's `sandbox_init` API, then +`exec`s the builder. + +Internally, Nix checks for `_NIX_TEST_NO_SANDBOX` to bypass this entirely, but +that's a testing escape hatch, not a supported configuration. + +--- + +## Tasks + +### 2.1 — Create `/usr/bin/sandbox-exec` Stub + +Create a stub `sandbox-exec` binary that lives in the Darling prefix at +`/usr/bin/sandbox-exec`. This is the **MVP approach**. + +**Behavior**: + +1. Parse command-line arguments matching the real `sandbox-exec` interface: + - `-f ` — path to a `.sb` sandbox profile (ignored) + - `-p ` — inline sandbox profile (ignored) + - `-D =` — parameter definitions for the profile (ignored) + - `-n ` — predefined profile name (ignored) + - Everything after the flags is the command to execute. +2. Ignore all sandbox-related arguments. +3. `exec` the remaining arguments (the builder command). + +**Implementation options** (pick one): + +#### Option A — Shell script (simplest) + +```sh +#!/bin/sh +# sandbox-exec stub for Darling +# Ignores sandbox profiles and exec's the builder directly. +# Darling provides Linux-level isolation via namespaces/darlingserver. + +while [ $# -gt 0 ]; do + case "$1" in + -f) shift 2 ;; # skip -f + -p) shift 2 ;; # skip -p + -n) shift 2 ;; # skip -n + -D) shift 2 ;; # skip -D key=value + -D*) shift ;; # skip -Dkey=value (no space) + *) break ;; + esac +done + +exec "$@" +``` + +Pros: trivial, no compilation needed. +Cons: requires `/bin/sh` to be working (it is in Darling); slight overhead from +shell parse. + +#### Option B — Small C program (more robust) + +```c +#include +#include +#include + +int main(int argc, char *argv[]) { + int i = 1; + while (i < argc) { + if ((strcmp(argv[i], "-f") == 0 || + strcmp(argv[i], "-p") == 0 || + strcmp(argv[i], "-n") == 0 || + strcmp(argv[i], "-D") == 0) && i + 1 < argc) { + i += 2; /* skip flag + argument */ + } else if (strncmp(argv[i], "-D", 2) == 0) { + i += 1; /* skip -Dkey=value */ + } else { + break; + } + } + + if (i >= argc) { + fprintf(stderr, "sandbox-exec: no command specified\n"); + return 1; + } + + execvp(argv[i], &argv[i]); + perror("sandbox-exec: exec"); + return 127; +} +``` + +Pros: no shell dependency, handles edge cases better, tiny binary. +Cons: needs to be compiled as a Mach-O binary and installed into the prefix. + +**Recommendation**: Start with Option A (shell script) for speed. Replace with +Option B later if any issues arise. + +**Installation**: The stub must be installed during Darling's build/prefix setup. +Add it to the CMake install step or to the prefix initialization script. + +**Location in build system**: Create `src/sandbox-exec/` with the stub and a +`CMakeLists.txt` that installs it to `libexec/darling/usr/bin/sandbox-exec`. + +--- + +### 2.2 — Fix Sandbox API Stubs + +**Current state** (`src/sandbox/sandbox.c`): + +```c +int sandbox_init(const char *profile, uint64_t flags, char **errorbuf) +{ + *errorbuf = strdup("Not implemented"); + return 0; +} +``` + +This is subtly wrong: it returns 0 (success) but also sets `*errorbuf` to an +error string. Callers that check `errorbuf != NULL` after a "successful" call +may be confused, or may leak memory expecting `errorbuf` to be NULL on success. + +**Fix**: Set `*errorbuf = NULL` on success: + +```c +int sandbox_init(const char *profile, uint64_t flags, char **errorbuf) +{ + if (errorbuf) + *errorbuf = NULL; + return 0; +} +``` + +Apply the same fix to: + +- `sandbox_init_with_parameters` +- `sandbox_init_with_extensions` +- `sandbox_wakeup_daemon` (currently returns -1; change to return 0 if callers + expect success, or leave as-is if it's genuinely optional) + +**Files to modify**: `src/sandbox/sandbox.c` + +**Also verify**: `src/libsandbox/src/sandbox.c` (the `libsandbox.1.dylib` shim) +doesn't have the same issue. + +--- + +### 2.3 — Ensure `sandbox_check` Always Permits + +**Current state** (`src/sandbox/sandbox.c`): + +```c +int sandbox_check(pid_t pid, const char *operation, + enum sandbox_filter_type type, ...) +{ + return 0; +} +``` + +This is correct — returning 0 means "allowed". Verify that the `_by_audit_token` +variant behaves the same (it does, based on code analysis). No changes needed +unless testing reveals issues. + +--- + +### 2.4 — (Stretch) Basic Sandbox Profile Language Parsing + +> **This is NOT required for Nix support.** It's documented here for +> completeness and for future contributors who want proper sandbox parity. + +Implement basic parsing of Apple's Sandbox Profile Language (`.sb` files) and +translate deny rules to Linux isolation mechanisms: + +| macOS Sandbox Rule | Linux Equivalent | +|---|---| +| `(deny file-write*)` | Read-only bind mounts or Landlock `LANDLOCK_ACCESS_FS_WRITE_FILE` deny | +| `(deny file-read* (subpath "/private"))` | Landlock path-beneath rule | +| `(deny network*)` | Unshare network namespace (`CLONE_NEWNET`) | +| `(deny network-outbound)` | `iptables` / `nftables` OUTPUT DROP, or network namespace | +| `(deny process-exec)` | `seccomp-bpf` filter on `execve` | +| `(deny process-fork)` | `seccomp-bpf` filter on `clone` / `fork` | +| `(deny file-write-setugid)` | `seccomp-bpf` filter on `fchmod` with setuid/setgid bits | +| `(allow default)` | Baseline: allow everything, then layer on denies | + +This would require: + +1. A Scheme parser (or at minimum a purpose-built `.sb` parser — the language is + a small subset of Scheme). +2. Translation logic mapping macOS sandbox operations to Linux syscall filters. +3. Integration with `sandbox-exec` to apply the translated policy before + `exec`-ing the builder. + +**Effort**: Weeks to months. Not recommended until after Phase 4 is working. + +--- + +## Security Considerations + +**Q: Is it safe to skip the macOS sandbox?** + +Yes, for the Darling use case: + +1. **Darling already provides isolation.** The `darlingserver` runs Darling + processes inside a Linux container with namespace isolation (mount, PID, user + namespaces via `overlayfs`). This is comparable to — and arguably stronger + than — macOS's `sandbox-exec` for build isolation purposes. + +2. **Nix's sandbox is defense-in-depth.** Nix's primary isolation comes from the + build environment setup (clean `$PATH`, empty `$HOME`, controlled `$TMPDIR`). + The macOS sandbox adds an extra layer but isn't the only protection. + +3. **The Linux host can add its own sandboxing.** If stronger isolation is + needed, the host can run Darling inside a systemd-nspawn container, a VM, or + with additional seccomp profiles. This provides equivalent-or-better security + to macOS's sandbox. + +4. **No untrusted code.** In the Nix builder context, the code being executed is + from derivations that the user has chosen to build. The sandbox prevents + accidental side effects, not malicious code execution. + +--- + +## Verification Checklist + +After completing Phase 2, the following should all work inside `darling shell`: + +- [ ] `/usr/bin/sandbox-exec` exists and is executable +- [ ] `sandbox-exec -f /dev/null -D _GLOBAL_TMP_DIR=/tmp /bin/echo hello` prints "hello" +- [ ] `sandbox-exec -p '(version 1)(allow default)' /bin/echo hello` prints "hello" +- [ ] `sandbox-exec` with no command argument prints an error and exits non-zero +- [ ] Calling `sandbox_init("no_network", 0, &err)` from C returns 0 with `err == NULL` +- [ ] Nix's builder invocation (`posix_spawn` → `sandbox-exec` → `/bin/bash`) + no longer returns `ENOEXEC` / "Bad file descriptor" +- [ ] A trivial `nix-build` with `_NIX_TEST_NO_SANDBOX` **unset** proceeds past + the sandbox-exec step (it may still fail later due to Phase 1 issues, but it + must not fail at the sandbox stage) + +--- + +## Implementation Order + +``` +2.1 (sandbox-exec stub) — do first, biggest impact + ↓ +2.2 (fix sandbox_init) — quick follow-up, same files + ↓ +2.3 (verify sandbox_check) — no changes expected, just verify + ↓ +2.4 (stretch: SBPL parse) — defer until after Phase 4 +``` + +--- + +*[← Phase 1 — Syscall Fixes](./03-phase1-syscalls.md) | [Phase 3 — Nix Installation →](./05-phase3-nix-install.md)* \ No newline at end of file diff --git a/plan/05-phase3-nix-install.md b/plan/05-phase3-nix-install.md new file mode 100644 index 000000000..67118aeea --- /dev/null +++ b/plan/05-phase3-nix-install.md @@ -0,0 +1,329 @@ +# Phase 3 — Nix Installation Inside Darling + +**Priority**: P0 · **Effort**: M (2–3 weeks) · **Depends on**: Phase 1 (syscall fixes), Phase 2 (sandbox stub) + +With the syscall fixes from Phase 1 and the `sandbox-exec` stub from Phase 2, +the Nix package manager should be installable inside a Darling prefix. This +phase covers automating that installation, verifying core Nix commands, and +providing convenient wrappers for host-side usage. + +--- + +## Context + +The official Nix installer for macOS (`nix-*-x86_64-darwin`) has several +assumptions that conflict with Darling's environment: + +1. **Forces multi-user mode on Darwin.** The installer detects `uname -s` = + `Darwin` and refuses single-user installation. Multi-user mode requires + `dseditgroup`, `sysadminctl`, and a working `launchd` — none of which are + fully functional in Darling yet. + +2. **Requires `diskutil info /`** to check the root filesystem type (APFS vs + HFS+). Darling's `diskutil` is a shell script that only supports `eject`. + +3. **Requires `xmllint`** for parsing plists. Not shipped in Darling. + +4. **Requires Directory Services** (`dseditgroup`, `dscl`) for creating the + `nixbld` group and build users. + +5. **Calls `lchflags`** during profile installation (fixed in Phase 1). + +6. **Root user quirks.** Nix defaults `build-users-group = nixbld` when running + as root. In single-user mode as root, this must be overridden to empty. + +All of these are solvable with a patched installer script and pre-configured +`nix.conf`. + +--- + +## Tasks + +### 3.1 — Create Automated Nix-in-Darling Installer + +Create a script at `scripts/install-nix-in-darling.sh` (run from the Linux +host) that automates the entire Nix installation inside a Darling prefix. + +**Steps the script should perform**: + +1. **Verify prerequisites**: + - Darling is installed and `darling shell echo ok` works. + - The prefix is initialized (`~/.darling` or `$DPREFIX` exists). + - Phase 1 and Phase 2 fixes are in place (check for `/usr/bin/sandbox-exec` + inside the prefix). + +2. **Pre-configure Nix**: + ```bash + darling shell mkdir -p /etc/nix + darling shell tee /etc/nix/nix.conf <<'EOF' + # Single-user mode: no build users group + build-users-group = + # Disable macOS sandbox (we use the sandbox-exec stub) + sandbox = false + # Use the Nix binary cache + substituters = https://cache.nixos.org + trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= + EOF + ``` + +3. **Download the Nix installer**: + - Fetch the latest `nix-*-x86_64-darwin` installer tarball from + `https://releases.nixos.org/nix/`. + - Verify its signature / hash. + - Extract it into a temporary directory inside the prefix. + +4. **Patch the installer**: + - Remove or bypass the `uname`-based multi-user enforcement. + - Remove the `diskutil info` check. + - Remove the `xmllint` dependency (or provide a stub). + - Suppress the "installing as root is not supported" warning. + - Force `--no-daemon` mode. + + The patching should be done with `sed` or a patch file applied to the + extracted `install` script. Keep the patch minimal and well-documented so it + can be updated when Nix releases new installer versions. + +5. **Run the patched installer**: + ```bash + darling shell /tmp/nix-installer/install --no-daemon + ``` + +6. **Post-install verification**: + - Run each command from the verification checklist (see below). + - Source the Nix profile: `. /Users/root/.nix-profile/etc/profile.d/nix.sh` + - Print the installed Nix version. + +7. **Clean up**: + - Remove the temporary installer files. + - Optionally run `nix-collect-garbage` to free space. + +**Error handling**: The script should `set -euo pipefail` and provide clear +error messages at each step, indicating which phase/blocker is likely the cause +if something fails. + +--- + +### 3.2 — Pre-Built Darling Prefix with Nix + +Create a Nix derivation (`packages.x86_64-linux.darling-nix-prefix`) that +produces a Darling prefix tarball with Nix pre-installed. This lets users skip +the installation process entirely. + +**Approach**: + +1. Build Darling in a Nix sandbox. +2. Initialize a fresh prefix. +3. Run the installer script from 3.1 inside the prefix (this requires a + working Darling at build time — may need to be done in a NixOS VM test + context rather than a pure derivation, since Darling needs namespace + capabilities). +4. Snapshot the prefix as a tarball. +5. Users restore with: + ```bash + mkdir -p ~/.darling + tar xf /nix/store/...-darling-nix-prefix.tar -C ~/.darling + ``` + +**Alternative**: If building inside a Nix sandbox is too complex (due to +namespace requirements), provide a script that generates the prefix on the +user's machine and document it as a one-time setup step. + +--- + +### 3.3 — Verify Core Nix Commands + +After installation, the following commands must work without errors inside +`darling shell`. Each one exercises a different subsystem: + +| Command | What It Tests | +|---|---| +| `nix --version` | Binary loads, `dyld` resolves all libraries | +| `nix-env --version` | Same, plus `libnixstore` loads correctly | +| `nix-store --verify` | Store database access, file system operations | +| `nix-instantiate --eval -E '1 + 1'` | Nix evaluator, no build needed | +| `nix eval --expr '1 + 1'` | Flake-enabled CLI, evaluator | +| `nix-store --dump-db` | SQLite database access in `/nix/var/nix/db/` | +| `nix-env -qa hello` | Channel/registry querying, HTTP fetching | + +**Known potential issues at this stage**: + +- **SQLite**: Nix's store database uses SQLite. If Darling's `fcntl` locking + (via `F_SETLK` / `F_GETLK`) is buggy, database operations will fail or hang. + Add SQLite lock testing to the verification. + +- **curl / TLS**: `nix-env -qa` and binary substitution need working HTTPS. + Darling ships its own curl and SSL certificates. If they're outdated or the + TLS handshake uses unimplemented syscalls, fetching will fail. Verify with: + ```bash + darling shell curl -sI https://cache.nixos.org/nix-cache-info + ``` + +- **`/nix` path**: By default, Nix installs to `/nix`. Inside Darling, this is + within the prefix overlay at `~/.darling/nix`. This is fine for isolated + usage. For shared-store mode (Phase 7), we'll need to symlink this to the + host's `/nix` via `/Volumes/SystemRoot/nix`. + +--- + +### 3.4 — Host-Side Wrapper: `darling-nix` + +Create a convenience wrapper script (installed as part of the Darling Nix +package) that runs Nix commands inside Darling from the Linux host: + +```bash +#!/usr/bin/env bash +# darling-nix — run Nix commands inside a Darling prefix +set -euo pipefail + +# Source Nix profile and run the command +exec darling shell bash -lc ' + . /Users/root/.nix-profile/etc/profile.d/nix.sh + exec "$@" +' -- "$@" +``` + +**Usage examples**: + +```bash +# Evaluate an expression +darling-nix nix-instantiate --eval -E '1 + 1' + +# Build a trivial derivation +darling-nix nix-build --expr 'derivation { name = "test"; builder = "/bin/bash"; args = ["-c" "echo ok > $out"]; system = "x86_64-darwin"; }' + +# Install a package +darling-nix nix-env -iA nixpkgs.hello + +# Interactive Nix repl +darling-nix nix repl +``` + +**Install location**: `$out/bin/darling-nix` in the Darling Nix package. + +**Enhancements for later**: + +- Support `--prefix ` to use a non-default Darling prefix. +- Support `--store ` to configure the Nix store location. +- Capture and forward exit codes correctly. +- Handle signals (SIGINT, SIGTERM) and propagate them to the Darling process. + +--- + +### 3.5 — Nix Channel / Registry Setup + +After Nix is installed, set up a usable channel or flake registry so users can +immediately start building packages: + +```bash +# Add the nixpkgs channel (for nix-env / nix-shell) +darling shell nix-channel --add https://nixos.org/channels/nixpkgs-unstable nixpkgs +darling shell nix-channel --update + +# Or, for flakes: +darling shell nix registry add nixpkgs github:NixOS/nixpkgs/nixpkgs-unstable +``` + +This should be part of the installer script (3.1) as an optional post-install +step. + +**Potential issue**: `nix-channel --update` downloads and unpacks a tarball, +which exercises `curl`, `xz`, `tar`, and filesystem operations. Any crash here +points to remaining syscall gaps from Phase 1. + +--- + +## Shared Store Considerations + +For Phase 3, Nix runs with its own store inside the Darling prefix +(`~/.darling/nix/store`). This is the simplest setup and avoids any +interaction with the host's Nix store. + +For later phases (especially Phase 7 — Remote Builder), we'll want to share the +host's `/nix/store` with the Darling prefix. The mechanism: + +```bash +# Inside the Darling prefix, /Volumes/SystemRoot is the host's / +# So /Volumes/SystemRoot/nix/store is the host's /nix/store + +# Option A: Symlink +darling shell ln -sf /Volumes/SystemRoot/nix /nix + +# Option B: Bind mount (if overlayfs allows it) +# Configured in darlingserver / prefix init +``` + +This is NOT part of Phase 3 — just documented here so the installation script +doesn't make assumptions that would conflict with shared-store mode later. In +particular: + +- Don't hardcode paths that assume `/nix` is local to the prefix. +- Make the store location configurable in `nix.conf`. +- Ensure the installer doesn't fail if `/nix` is a symlink. + +--- + +## Debugging Tips + +If installation fails, here are the most useful debugging techniques: + +**Trace the installer script**: +```bash +darling shell bash -x /tmp/nix-installer/install --no-daemon 2>&1 | tee install.log +``` + +**Trace Nix binary startup**: +```bash +# On the host, trace darlingserver while running a Nix command: +strace -f -p $(pidof darlingserver) -e trace=openat,stat,fstat,lstat,readlink 2>&1 | head -200 & +darling shell /nix/store/.../bin/nix --version +``` + +**Trace inside Darling with xtrace**: +```bash +DARLING_XTRACE=1 darling shell /nix/store/.../bin/nix-env --version 2>&1 | head -500 +``` + +**Check for unimplemented syscalls**: +```bash +darling shell /nix/store/.../bin/nix --version 2>&1 | grep -i "unimplemented\|STUB\|not.implemented" +``` + +**Inspect the store database**: +```bash +darling shell sqlite3 /nix/var/nix/db/db.sqlite ".tables" +darling shell sqlite3 /nix/var/nix/db/db.sqlite "SELECT count(*) FROM ValidPaths;" +``` + +--- + +## Verification Checklist + +After completing Phase 3, ALL of the following must pass: + +- [ ] `scripts/install-nix-in-darling.sh` completes without errors +- [ ] `darling shell nix --version` prints the Nix version +- [ ] `darling shell nix-env --version` prints the Nix version +- [ ] `darling shell nix-store --verify` reports no errors +- [ ] `darling shell nix-instantiate --eval -E '1 + 1'` prints `2` +- [ ] `darling shell nix eval --expr '1 + 1'` prints `2` +- [ ] `darling shell curl -sI https://cache.nixos.org/nix-cache-info` returns HTTP 200 +- [ ] `darling-nix nix --version` works from the Linux host +- [ ] `darling-nix nix-instantiate --eval -E 'builtins.currentSystem'` prints `"x86_64-darwin"` +- [ ] No "Unimplemented syscall" messages during any of the above +- [ ] No segfaults during any of the above + +--- + +## Risk Assessment + +| Risk | Likelihood | Impact | Mitigation | +|---|---|---|---| +| SQLite locking doesn't work | Medium | High — store operations fail | Test `fcntl` locking early; if broken, use `PRAGMA locking_mode=EXCLUSIVE` | +| curl/TLS fails | Medium | High — no binary substitution | Test HTTPS early; fall back to `--option substitute false` for offline mode | +| Nix installer changes break our patches | Medium | Medium — need to update patches | Pin a specific Nix version; provide a patch file rather than inline sed | +| `/nix` path conflicts with shared store | Low | Medium — need reconfiguration | Keep store location configurable from the start | +| Nix evaluator hits unimplemented syscalls | Low | Medium — eval works but slowly | Phase 1 triage (1.7) should catch these | + +--- + +*[← Phase 2 — Sandbox](./04-phase2-sandbox.md) | [Phase 4 — Derivation Building →](./06-phase4-building.md)* \ No newline at end of file diff --git a/plan/06-phase4-building.md b/plan/06-phase4-building.md new file mode 100644 index 000000000..4a3d12459 --- /dev/null +++ b/plan/06-phase4-building.md @@ -0,0 +1,452 @@ +# Phase 4 — Derivation Building + +**Priority**: P1 · **Effort**: L (4–8 weeks) · **Depends on**: Phase 3 (Nix installation) + +This is the acid test: can Nix actually *build* derivations inside Darling? Phase +3 got Nix installed and evaluating; this phase gets it building real software. + +We progress from trivial derivations through to full stdenv builds and binary +substitution from the official cache. + +--- + +## Context + +A Nix derivation build on Darwin involves: + +1. Nix creates a temporary build directory (`/tmp/nix-build-.drv-N/`). +2. Nix generates a sandbox profile (`.sb` file) in that directory. +3. Nix calls `posix_spawn` to execute `/usr/bin/sandbox-exec -f `. +4. The builder (usually `/bin/bash`) runs inside the sandbox with a clean + environment (`$PATH`, `$HOME`, `$TMPDIR` all controlled by Nix). +5. The builder script sources `$stdenv/setup` and runs the build phases + (unpack, configure, build, install, fixup, etc.). +6. Build output is written to `$out` (a path in `/nix/store`). +7. Nix registers the output in the store database and makes it read-only. + +Each step exercises different parts of the Darling compatibility layer. This +phase works through them incrementally. + +--- + +## Tasks + +### 4.1 — Build a Trivial Derivation + +The simplest possible derivation — no dependencies, no stdenv, just `/bin/bash` +writing a file: + +```nix +derivation { + name = "hello-darling"; + builder = "/bin/bash"; + args = [ "-c" "echo 'Hello from Darling!' > $out" ]; + system = "x86_64-darwin"; +} +``` + +**Build command**: + +```bash +darling-nix nix-build --expr 'derivation { name = "hello-darling"; builder = "/bin/bash"; args = [ "-c" "echo hello > $out" ]; system = "x86_64-darwin"; }' +``` + +**What this exercises**: + +- `posix_spawn` → `sandbox-exec` stub → `/bin/bash` (Phase 2 must be working) +- File creation in `/nix/store` +- `$out` environment variable propagation +- Basic file I/O (`echo`, redirect) +- Store path registration (SQLite write) +- Setting store path read-only (`chmod`, possibly `lchflags`) + +**Expected failure modes**: + +| Failure | Likely Cause | Fix | +|---|---|---| +| "Bad file descriptor" / ENOEXEC | `sandbox-exec` stub not installed or not executable | Phase 2 — verify installation | +| "clearing flags of path" | `lchflags` still failing | Phase 1.1 / 1.2 | +| Sandbox profile write fails | `/tmp` not writable or path issue | Check prefix `/private/tmp` setup | +| Builder hangs | `posix_spawn` with `POSIX_SPAWN_SETEXEC` broken | Phase 1 — B5 | +| "build failure may have been caused by lack of free disk space" | Generic Nix error wrapping the real issue | Check build log in `/nix/var/log/nix/` | + +**Debugging**: + +```bash +# Verbose build with debug output +darling-nix nix-build --expr '...' -vvvv --debug 2>&1 | tee build.log + +# Check if the builder can be invoked manually +darling shell /usr/bin/sandbox-exec -f /dev/null -D _GLOBAL_TMP_DIR=/tmp /bin/bash -c 'echo ok' + +# Manually run the derivation's builder to isolate the failure +darling shell nix-shell --pure --run 'echo $out' /nix/store/...-hello-darling.drv +``` + +--- + +### 4.2 — Get `bash` Executing Reliably in Build Sandboxes + +Even after 4.1 works, there may be subtle issues with bash inside Nix's build +environment. The build environment is intentionally spartan: + +- `$HOME=/homeless-shelter` (doesn't exist) +- `$PATH=/path-not-set` (intentionally broken) +- `$TMPDIR=/tmp/nix-build-.drv-N/` +- `$NIX_STORE=/nix/store` + +**Requirements for bash to function**: + +- `/dev/null` must exist and be readable/writable +- `/dev/urandom` must exist (some builds need random data) +- `/dev/zero` must exist +- `$TMPDIR` must be writable +- `posix_spawn` with `POSIX_SPAWN_SETEXEC` must work (acts like `exec`) +- Signal handling must work (Nix sends `SIGTERM` to cancel builds) +- `pipe2` / `dup2` must work (for shell redirections) +- `fcntl` with `F_GETFD` / `F_SETFD` must work (for `O_CLOEXEC`) + +**Verification**: + +```bash +# Inside darling shell, simulate a Nix build environment: +env -i HOME=/homeless-shelter PATH=/path-not-set \ + TMPDIR=/tmp/test-build NIX_STORE=/nix/store \ + /bin/bash -c 'echo "PATH=$PATH"; echo "HOME=$HOME"; echo ok > /tmp/test-build/out' +``` + +**Check device nodes**: + +```bash +darling shell ls -la /dev/null /dev/urandom /dev/zero +# These should exist. If not, they need to be created during prefix init +# or symlinked from /Volumes/SystemRoot/dev/ +``` + +--- + +### 4.3 — Build with Nix's `bash` (from the Binary Cache) + +The trivial derivation in 4.1 uses Darling's built-in `/bin/bash`. Real +derivations use Nix's own bash from the store (e.g., +`/nix/store/...-bash-5.2-p26/bin/bash`). This is a pre-built `x86_64-darwin` +Mach-O binary fetched from `cache.nixos.org`. + +**Test**: + +```nix +let + bash = builtins.fetchurl { + url = "https://cache.nixos.org/nar/..."; # or use a pinned store path + }; +in derivation { + name = "test-nix-bash"; + builder = "${bash}/bin/bash"; + args = [ "-c" "echo 'Using Nix bash!' > $out" ]; + system = "x86_64-darwin"; +} +``` + +Or more practically: + +```bash +# Force-fetch bash from the binary cache +darling-nix nix-store -r /nix/store/...-bash-5.2-p26 + +# Then build a derivation that uses it +darling-nix nix-build --expr ' + let pkgs = import { system = "x86_64-darwin"; }; + in derivation { + name = "test-nix-bash"; + builder = "${pkgs.bash}/bin/bash"; + args = [ "-c" "echo ok > \$out" ]; + system = "x86_64-darwin"; + } +' +``` + +**What this additionally exercises**: + +- `dyld` loading the Nix-built bash and all its dependencies (`libSystem`, + `libc++`, etc.) — these are Mach-O binaries that must be translated by Darling +- NAR unpacking (when fetching from the cache) +- Symlink handling in `/nix/store` +- `LC_RPATH` / `@rpath` resolution in Mach-O binaries + +**Expected failure modes**: + +| Failure | Likely Cause | Fix | +|---|---|---| +| `dyld: Symbol not found` | Nix bash built for macOS 11+ but Darling reports 10.15 | Phase 1.8 (version bump) | +| `dyld: Library not loaded` | Missing `libSystem` or `libc++` dylib in Darling prefix | Verify Darling's system libraries cover the needed symbols | +| `Illegal instruction: 4` | Binary uses CPU instruction Darling doesn't translate | Check if SSE/AVX instructions are involved; may need darlingserver fix | +| Segfault during load | `dyld` cache issue or broken mmap translation | See [Blocker B7](./01-blockers.md#b7-dyld-shared-cache) | + +--- + +### 4.4 — Handle Binary Substitution from `cache.nixos.org` + +Binary substitution (downloading pre-built packages instead of building them) is +critical for practical use — building everything from source inside Darling would +be extremely slow. + +**What to test**: + +```bash +# Fetch a simple package from the cache +darling-nix nix-store -r /nix/store/...-hello-2.12.1 + +# Or build with substitution: +darling-nix nix-build '' -A hello --system x86_64-darwin +``` + +**Substitution pipeline**: + +``` +nix-store --realise + → curl HTTPS request to cache.nixos.org + → download .narinfo (package metadata) + → download .nar.xz (compressed archive) + → xz decompress + → NAR unpack to /nix/store/... + → set permissions (chmod, chown) + → register in SQLite database + → clear flags (lchflags — Phase 1) +``` + +**Requirements**: + +- **HTTPS/TLS**: `curl` must successfully connect to `cache.nixos.org`. Test: + ```bash + darling shell curl -sI https://cache.nixos.org/nix-cache-info + ``` + +- **xz decompression**: The `xz` binary from the store must work. If it uses + unimplemented syscalls, we need the host-side `xz` or a fallback. + +- **NAR unpacking**: Nix's NAR format uses `mknod`, `symlink`, `chmod`, + `chown`, `utimes`. All must work. + +- **Large file support**: Some NARs are hundreds of MB. Ensure `mmap`, `ftruncate`, + and large `read`/`write` calls work correctly. + +- **Certificate verification**: Nix verifies the binary cache's signing key, not + TLS certificates for trust. But `curl` still needs working TLS. Darling ships + OpenSSL certificates via `src/external/openssl_certificates/` — verify they're + up to date. + +--- + +### 4.5 — Build a Simple C Program with Darwin stdenv + +This is the first "real" build — compiling C code using Nixpkgs' Darwin stdenv, +which pulls in clang, ld64, Apple SDK headers, and the full build machinery. + +```bash +darling-nix nix-build '' -A hello --system x86_64-darwin +``` + +**What the Darwin stdenv does**: + +1. Sources `$stdenv/setup` (a large bash script). +2. Unpacks the source tarball. +3. Runs `./configure` (or cmake, meson, etc.). +4. Compiles with `clang` targeting `x86_64-apple-darwin`. +5. Links with `ld64` (Apple's linker, from cctools-port). +6. Runs fixup phase: `install_name_tool`, `codesign`, `strip`, etc. +7. Produces a Mach-O executable or library in `$out`. + +**Key binaries that must work** (all from the Nix store, built for Darwin): + +| Binary | Role | Concern | +|---|---|---| +| `bash` | Builder shell | Covered in 4.2/4.3 | +| `coreutils` (`mv`, `cp`, `touch`, `install`, `mkdir`) | Basic file operations | `mv` needs `renameatx_np` (Phase 1.3), `touch` needs `utimensat` (Phase 1.4) | +| `clang` | C/C++/ObjC compiler | May use `posix_spawn` internally; large binary with many dylib deps | +| `ld64` | Apple linker | Writes Mach-O output; may use `fcntl` advisory locks | +| `ar` / `ranlib` | Archive tools | From cctools, should be straightforward | +| `install_name_tool` | Fix dylib paths | Modifies Mach-O headers; needs working `mmap` + `ftruncate` | +| `codesign_allocate` | Code signature space | May fail (no codesign in Darling); needs graceful fallback | +| `strip` | Strip symbols | Modifies Mach-O binaries | +| `xattr` | Extended attributes | `xattr -cr` is run during fixup; needs `removexattr` / `listxattr` | +| `sed`, `grep`, `awk` | Text processing | Usually fine, but check for syscall issues | +| `tar`, `gzip`, `xz` | Archive handling | `tar` may use `fchflags`; `xz` may use newer syscalls | + +**Coreutils crash workaround strategy**: + +If specific coreutils binaries from the Nix store crash due to unimplemented +syscalls, there are two approaches: + +1. **Preferred**: Fix the syscall in darlingserver (Phase 1). +2. **Temporary**: Create wrapper scripts in the prefix that intercept the + crashing commands and redirect to Darling's built-in versions: + ```bash + # In the Darling prefix: + mkdir -p /usr/local/nix-compat/bin + cat > /usr/local/nix-compat/bin/mv << 'EOF' + #!/bin/sh + exec /bin/mv "$@" + EOF + chmod +x /usr/local/nix-compat/bin/mv + # Add /usr/local/nix-compat/bin early in $PATH for builds + ``` + This is a "Nix crime" if done inside the store, but putting it in `$PATH` + via `nix.conf` or a build hook is acceptable as a temporary measure. + +--- + +### 4.6 — Fix Remaining Coreutils / Build-Tool Crashes + +Based on the blog post and analysis, the following specific binaries are known +to crash inside Darling when fetched from the Nix binary cache. Each needs +either a syscall fix or a workaround. + +| Binary | Crash Symptom | Root Cause | Fix | +|---|---|---|---| +| `mv` | `Unimplemented syscall (488)` | `renameatx_np` missing | Phase 1.3 | +| `touch` | `Segmentation fault: 11` | `utimensat` / `setattrlist` | Phase 1.4 / 1.1 | +| `install` | `clearing flags` or crash | `fchflags` / `chflags` | Phase 1.1 | +| `cp` | Possible crash | `clonefile` / `fclonefileat` | Phase 1.5 | +| `tar` | `fchflags` warning or crash | `fchflags` on extracted files | Phase 1.1 | +| `xattr` | `removexattr` failure | xattr syscalls incomplete | New task — implement `removexattr`, `listxattr`, `getxattr` | +| `codesign_allocate` | Likely failure | Code signing not supported | Stub or skip in stdenv fixup phase | +| `fish` | `Illegal instruction: 4` | Uses newer CPU/syscall features | Lower priority — not in the critical build path | + +**Approach**: Work through these in order of build-pipeline criticality. A build +can't succeed if `mv` crashes, so that's fixed first (Phase 1.3). The codesign +tools are less critical — if they fail, we can patch the stdenv fixup phase to +skip codesigning inside Darling. + +**Extended attribute (xattr) handling**: + +The Darwin stdenv fixup phase runs `xattr -cr $out` to clear quarantine +attributes. This requires: + +- `listxattr` — list all xattrs on a file +- `removexattr` — remove a specific xattr +- `getxattr` / `setxattr` — read/write xattr values + +On Linux, these have direct equivalents (`listxattr(2)`, `removexattr(2)`, +etc.). Darlingserver needs to translate the macOS xattr syscalls to the Linux +ones, mapping the `com.apple.*` namespace appropriately. + +If full xattr support is too complex, a minimal approach: + +- `listxattr` → return empty list (no xattrs) +- `removexattr` → return success (nothing to remove) +- `getxattr` → return `ENODATA` (no such xattr) + +This is safe because Darling files won't have real Apple quarantine attributes. + +--- + +### 4.7 — Verify Build Output Correctness + +After a successful `nix-build`, verify the output is correct: + +```bash +# Build hello +darling-nix nix-build '' -A hello --system x86_64-darwin + +# Check the output exists and is a Mach-O binary +darling shell file /nix/store/...-hello-2.12.1/bin/hello + +# Run it +darling shell /nix/store/...-hello-2.12.1/bin/hello +# Expected: "Hello, world!" + +# Verify the store path is valid +darling-nix nix-store --verify-path /nix/store/...-hello-2.12.1 + +# Check closure (all dependencies resolved) +darling-nix nix-store -qR /nix/store/...-hello-2.12.1 +``` + +**Important**: The output hash of a derivation built inside Darling will differ +from the same derivation built on real macOS if the build is not perfectly +reproducible (input-addressed derivations use the same hash regardless of +content, but if there are build failures or different outputs, something is +wrong). + +--- + +### 4.8 — Handle `codesign` in the Fixup Phase + +The Darwin stdenv's fixup phase attempts to ad-hoc codesign all Mach-O binaries. +This calls `codesign_allocate` and/or `codesign` (or `sigtool` in recent +Nixpkgs). Darling is unlikely to support code signing. + +**Options**: + +1. **Stub `codesign`**: Provide a `/usr/bin/codesign` that does nothing and + returns 0. Mach-O binaries will work fine inside Darling without signatures. + +2. **Patch stdenv**: Override the Darwin stdenv to skip the signing fixup phase + when running inside Darling. Detect this via an environment variable + (e.g., `NIX_DARLING=1`). + +3. **Use `sigtool`**: Recent Nixpkgs uses a pure-Nix `sigtool` for ad-hoc + signing that may actually work since it's just modifying Mach-O bytes. + Test before assuming it fails. + +**Recommendation**: Test option 3 first. If it fails, use option 1 (quickest). +Option 2 is cleanest but requires Nixpkgs patching. + +--- + +## Verification Checklist + +After completing Phase 4, ALL of the following must pass: + +- [ ] Trivial derivation (4.1) builds and produces correct output +- [ ] Derivation using Nix's bash from the store (4.3) builds successfully +- [ ] `nix-store -r` fetches packages from `cache.nixos.org` without errors +- [ ] NAR unpacking works for at least 10 different packages +- [ ] `nix-build '' -A hello --system x86_64-darwin` succeeds +- [ ] The built `hello` binary runs and prints "Hello, world!" +- [ ] `nix-store --verify-path` confirms the output is valid +- [ ] No "Unimplemented syscall" messages during the build +- [ ] No segfaults during the build +- [ ] `nix-collect-garbage` runs without errors (exercises store deletion + `lchflags`) + +--- + +## Risk Assessment + +| Risk | Likelihood | Impact | Mitigation | +|---|---|---|---| +| clang crashes inside Darling | Medium | Critical — can't compile anything | Test clang standalone first; may need specific dyld/ABI fixes | +| ld64 produces bad Mach-O output | Low | Critical — binaries won't run | Compare output with real macOS build; use `otool -L` to verify | +| Stdenv setup script uses unsupported shell features | Low | High — all builds fail | Test bash compatibility thoroughly in 4.2 | +| Binary cache signatures fail verification | Low | High — no substitution | Check Nix's ed25519 verification code path; may need `libsodium` to work | +| Build takes hours due to Darling overhead | High | Medium — usable but slow | Focus on binary substitution; only build what can't be fetched | +| Race conditions from incomplete `fcntl` locking | Medium | Medium — intermittent failures | Test concurrent builds only in Phase 5; keep Phase 4 single-threaded | + +--- + +## Performance Expectations + +Darling adds overhead to every syscall (Darwin → Linux translation). Expect: + +- **Evaluation**: ~2–5× slower than native Linux Nix evaluation. The Nix + evaluator is CPU-bound, so the overhead is mostly from dyld and library + translation, not syscall volume. + +- **Binary substitution**: ~1.5–2× slower. Network I/O dominates; the overhead + is in NAR unpacking (filesystem syscalls). + +- **Compilation**: ~3–10× slower. Compilation is both CPU-intensive and makes + many syscalls (file reads, process spawning). The `clang` → `ld64` pipeline + inside Darling will be noticeably slower than on native macOS. + +- **Disk usage**: Each Darling prefix uses overlayfs, so the base system files + are shared. The Nix store will be the main disk consumer. Plan for ~10–20 GB + for a basic set of packages. + +These are rough estimates. Actual performance will depend heavily on the host +hardware and which syscalls are hot. Profiling after Phase 4 is complete will +identify optimisation opportunities. + +--- + +*[← Phase 3 — Nix Installation](./05-phase3-nix-install.md) | [Phase 5 — Nix Daemon →](./07-phase5-daemon.md)* \ No newline at end of file diff --git a/plan/07-phase5-daemon.md b/plan/07-phase5-daemon.md new file mode 100644 index 000000000..d6ed51878 --- /dev/null +++ b/plan/07-phase5-daemon.md @@ -0,0 +1,377 @@ +# Phase 5 — Nix Daemon & Multi-User Mode + +**Priority**: P2 · **Effort**: M (2–4 weeks) · **Depends on**: Phase 4 (derivation building) + +Single-user mode (Phases 0–4) is sufficient for development and testing, but a +production-grade setup benefits from the Nix daemon for concurrent builds, +proper garbage collection, and user isolation. This phase adds multi-user Nix +support inside Darling. + +--- + +## Context + +On a real macOS system, the Nix daemon (`nix-daemon`) runs as a LaunchDaemon +managed by `launchd`. It: + +1. Listens on a Unix domain socket (`/nix/var/nix/daemon-socket/socket`). +2. Accepts build requests from unprivileged users. +3. Spawns builds as dedicated `_nixbldN` users (members of the `nixbld` group). +4. Manages the Nix store exclusively — only the daemon writes to `/nix/store`. +5. Handles garbage collection, signing, and binary cache downloads. + +The multi-user Nix installer on macOS creates: + +- A `nixbld` group (GID 30000 by convention). +- 32 build users `_nixbld1` through `_nixbld32` (UIDs 300–331). +- A LaunchDaemon plist at `/Library/LaunchDaemons/org.nixos.nix-daemon.plist`. +- Nix profile scripts in `/etc/profile.d/` and `/etc/bashrc.d/`. + +All of this relies on Directory Services (`dscl`, `dseditgroup`, `sysadminctl`) +for user/group management and `launchd`/`launchctl` for service management. +Darling has partial `launchd` support but no Directory Services implementation. + +--- + +## Tasks + +### 5.1 — Implement Directory Services Stubs + +The Nix installer uses these commands to create build users and groups: + +```bash +# Create the nixbld group +dseditgroup -o create -q -i 30000 nixbld + +# Create build users +sysadminctl -addUser _nixbld1 -UID 300 -GID 30000 -home /var/empty -shell /usr/bin/false +# ... repeated for _nixbld2 through _nixbld32 + +# Add users to the group +dseditgroup -o edit -a _nixbld1 -t user nixbld +``` + +Darling does not implement these commands. We need thin wrappers that translate +to Linux user/group management operating on the prefix's `/etc/passwd` and +`/etc/group` files. + +#### `dseditgroup` stub + +Create `src/tools/dseditgroup` (or a shell script installed to +`libexec/darling/usr/sbin/dseditgroup`) that handles: + +| Invocation | Translation | +|---|---| +| `dseditgroup -o create -q -i ` | `echo ":x::" >> /etc/group` (if not exists) | +| `dseditgroup -o edit -a -t user ` | Append `` to the group's member list in `/etc/group` | +| `dseditgroup -o delete ` | Remove the group from `/etc/group` | +| `dseditgroup -o checkmember -m ` | Check if user is in the group; exit 0 if yes, non-zero if no | + +Does not need to support the full `dseditgroup` interface — only what the Nix +installer uses. + +#### `sysadminctl` stub + +Create a stub that handles: + +| Invocation | Translation | +|---|---| +| `sysadminctl -addUser -UID -GID -home -shell ` | `echo ":x:::::" >> /etc/passwd` | +| `sysadminctl -deleteUser ` | Remove the user from `/etc/passwd` | + +#### `dscl` stub + +The Nix installer may also use `dscl` in some code paths: + +| Invocation | Translation | +|---|---| +| `dscl . -read /Groups/ PrimaryGroupID` | Parse `/etc/group` and print the GID | +| `dscl . -read /Users/ UniqueID` | Parse `/etc/passwd` and print the UID | +| `dscl . -list /Users` | List all usernames from `/etc/passwd` | +| `dscl . -create /Users/ ...` | Append to `/etc/passwd` | + +**Implementation notes**: + +- These stubs modify files within the Darling prefix (`~/.darling/etc/passwd`, + `~/.darling/etc/group`), not the host's files. This is safe. +- Do NOT use `useradd`/`groupadd` (those operate on the host). Directly + manipulate the prefix's files. +- Add basic input validation (duplicate detection, numeric ranges). +- Make them idempotent — running the installer twice should not create duplicate + entries. + +**Testing**: + +```bash +# Inside darling shell: +dseditgroup -o create -q -i 30000 nixbld +grep nixbld /etc/group +# Expected: nixbld:x:30000: + +sysadminctl -addUser _nixbld1 -UID 300 -GID 30000 -home /var/empty -shell /usr/bin/false +grep _nixbld1 /etc/passwd +# Expected: _nixbld1:x:300:30000::/var/empty:/usr/bin/false +``` + +--- + +### 5.2 — Get `nix-daemon` Running + +Once build users exist, launch the Nix daemon inside Darling. + +**Step 1 — Manual launch (for testing)**: + +```bash +darling shell nix-daemon & +``` + +The daemon should: + +- Create the socket at `/nix/var/nix/daemon-socket/socket`. +- Listen for connections. +- Fork build processes as `_nixbldN` users (requires working `setuid`/`setgid` + inside the Darling prefix). + +**Step 2 — Verify client connectivity**: + +```bash +# In another darling shell, as a non-root user: +darling shell nix-store --version +# This should connect to the daemon over the Unix socket +``` + +**Requirements for the daemon to function**: + +| Requirement | Status in Darling | Notes | +|---|---|---| +| Unix domain sockets | Likely works | Darling maps to Linux AF_UNIX sockets | +| `setuid` / `setgid` | Needs verification | Daemon drops privileges to build users; must work within the namespace | +| `fork` / `posix_spawn` | Partially works | Phase 1/B5 fixes needed for reliability | +| `fcntl` advisory locking | Needs verification | Store database locking; critical for concurrent access | +| `kill` / signal delivery | Likely works | Daemon sends SIGTERM to cancel builds | +| `/var/empty` exists | May need creation | Home directory for build users | + +**Potential issues**: + +- **`setuid` within namespaces**: Darling uses user namespaces. `setuid` inside a + user namespace works differently — the process can only switch to UIDs mapped + in the namespace. The Darling prefix must have the `_nixbldN` UIDs mapped. + This may require changes to darlingserver's namespace setup. + +- **Socket permissions**: The daemon socket must be readable/writable by all + users who should be able to trigger builds. Check that `chmod 0660` on the + socket works and that group membership is respected. + +- **Process isolation**: The daemon expects to be able to create per-build + temporary directories under `/tmp` or `$TMPDIR`, owned by the build user. + Verify that `chown` works for changing file ownership to build users. + +**Debugging**: + +```bash +# Watch daemon logs: +darling shell nix-daemon --debug 2>&1 | tee daemon.log + +# Test socket connectivity: +darling shell ls -la /nix/var/nix/daemon-socket/socket + +# Trace daemon syscalls from the host: +strace -f -p $(pgrep -f nix-daemon) -e trace=socket,bind,listen,accept,clone,setuid,setgid 2>&1 | head -200 +``` + +--- + +### 5.3 — LaunchDaemon Integration + +Make the Nix daemon manageable via `launchctl`, as it would be on real macOS. + +**Step 1 — Install the plist**: + +The Nix installer creates `/Library/LaunchDaemons/org.nixos.nix-daemon.plist`: + +```xml + + + + + Label + org.nixos.nix-daemon + ProgramArguments + + /nix/var/nix/profiles/default/bin/nix-daemon + + KeepAlive + + RunAtLoad + + StandardErrorPath + /var/log/nix-daemon.log + + +``` + +**Step 2 — Load with launchctl**: + +```bash +darling shell launchctl load /Library/LaunchDaemons/org.nixos.nix-daemon.plist +``` + +**Step 3 — Verify**: + +```bash +darling shell launchctl list | grep nix +# Expected: org.nixos.nix-daemon with a PID + +darling shell launchctl print system/org.nixos.nix-daemon +# Expected: status information +``` + +**Known risks**: Darling's `launchd` implementation (`src/launchd/`) is +functional for basic service management but may not support all plist keys. +`KeepAlive` (automatic restart) is the most likely to have issues. If launchd +integration is unreliable, fall back to manual daemon startup or a simple +wrapper script. + +**Fallback — systemd integration on the host**: + +If launchd proves too unreliable, an alternative is to manage the daemon from +the Linux host using systemd: + +```ini +# /etc/systemd/system/darling-nix-daemon.service +[Unit] +Description=Nix Daemon inside Darling +After=network.target + +[Service] +ExecStart=/usr/bin/darling shell /nix/var/nix/profiles/default/bin/nix-daemon +Restart=on-failure +Type=simple + +[Install] +WantedBy=multi-user.target +``` + +This bypasses launchd entirely while still providing reliable daemon management. + +--- + +### 5.4 — Test Concurrent Builds + +Multi-user mode enables parallel builds. Test that multiple derivations can build +simultaneously without interference. + +**Test procedure**: + +```bash +# Start the daemon +darling shell nix-daemon & + +# In parallel, build several independent packages: +darling-nix nix-build '' -A hello --system x86_64-darwin & +darling-nix nix-build '' -A which --system x86_64-darwin & +darling-nix nix-build '' -A yes --system x86_64-darwin & +wait +``` + +**What to watch for**: + +- **Store database locking**: SQLite must handle concurrent reads/writes via + `fcntl` locking. If locking is broken, you'll see `database is locked` errors + or silent corruption. + +- **Build user contention**: Each concurrent build should use a different + `_nixbldN` user. Verify with `ps aux | grep nix-build` inside darling shell. + +- **`/tmp` isolation**: Each build gets its own `$TMPDIR`. Verify no cross- + contamination between concurrent builds. + +- **File descriptor exhaustion**: Darling's fd table is backed by Linux fds. Many + concurrent builds can exhaust the per-process limit. Check `ulimit -n` inside + darling shell and increase if needed. + +- **Deadlocks**: If `posix_spawn` or `fork` has race conditions in Darling's + implementation, concurrent builds may deadlock. Monitor with `strace -f` and + look for stuck processes. + +**Expected outcome**: All three builds complete (possibly via binary +substitution) without errors. If building from source, expect it to be slow but +correct. + +--- + +### 5.5 — Nix Profile Scripts + +The multi-user installer sets up profile scripts so Nix is available to all +users. Verify these work: + +```bash +# /etc/profile.d/nix.sh should be sourced on login +darling shell bash -l -c 'which nix' +# Expected: /nix/var/nix/profiles/default/bin/nix + +# Verify $NIX_PATH is set +darling shell bash -l -c 'echo $NIX_PATH' + +# Verify the daemon socket is used (not direct store access) +darling shell bash -l -c 'nix-store --version' +# Should connect via /nix/var/nix/daemon-socket/socket +``` + +--- + +## Upgrade Path: Single-User → Multi-User + +Users who completed Phase 3 (single-user installation) should be able to +upgrade to multi-user mode. Document a migration procedure: + +1. Stop any running Nix processes. +2. Run the Directory Services stubs to create build users (5.1). +3. Update `/etc/nix/nix.conf`: + ```diff + - build-users-group = + + build-users-group = nixbld + - sandbox = false + + sandbox = true + ``` +4. Start the daemon (5.2 or 5.3). +5. Verify with `nix-store --version` (should connect to daemon). + +The Nix store itself doesn't need migration — it's the same `/nix/store` +regardless of single-user or multi-user mode. Only the access method changes +(direct vs. via daemon). + +--- + +## Verification Checklist + +After completing Phase 5, ALL of the following must pass: + +- [ ] `dseditgroup -o create -q -i 30000 nixbld` succeeds +- [ ] `sysadminctl -addUser _nixbld1 -UID 300 -GID 30000 -home /var/empty -shell /usr/bin/false` succeeds +- [ ] `/etc/group` and `/etc/passwd` inside the prefix contain the expected entries +- [ ] `nix-daemon` starts without errors +- [ ] `/nix/var/nix/daemon-socket/socket` exists after daemon start +- [ ] `nix-store --version` (as non-root) connects to the daemon +- [ ] A derivation build via the daemon completes successfully +- [ ] The build runs as a `_nixbldN` user (not root) +- [ ] `launchctl load` of the nix-daemon plist starts the service (or the systemd fallback works) +- [ ] Two concurrent `nix-build` invocations complete without database errors +- [ ] `nix-collect-garbage -d` works via the daemon + +--- + +## Risk Assessment + +| Risk | Likelihood | Impact | Mitigation | +|---|---|---|---| +| `setuid` doesn't work in Darling's namespace | High | Critical — daemon can't use build users | Test early; may need darlingserver namespace mapping changes | +| `fcntl` locking broken → database corruption | Medium | Critical — store becomes unusable | Test with `PRAGMA integrity_check` after concurrent builds | +| launchd can't manage the daemon reliably | Medium | Medium — use systemd fallback | Have the systemd unit file ready as Plan B | +| Build users can't write to `$TMPDIR` | Medium | High — all daemon builds fail | Verify `chown` and directory permissions for build user UIDs | +| Socket permissions prevent non-root access | Low | Medium — only root can build | Check `chmod`/`chgrp` on the socket; may need a `nix-users` group | + +--- + +*[← Phase 4 — Derivation Building](./06-phase4-building.md) | [Phase 6 — CI & Testing →](./08-phase6-ci.md)* \ No newline at end of file diff --git a/plan/08-phase6-ci.md b/plan/08-phase6-ci.md new file mode 100644 index 000000000..d8d5d3ea3 --- /dev/null +++ b/plan/08-phase6-ci.md @@ -0,0 +1,640 @@ +# Phase 6 — CI & Automated Testing + +**Priority**: P1 · **Effort**: M (2–3 weeks) · **Depends on**: Phase 3 (Nix installation) + +Automated testing is essential to prevent regressions as we add syscall +implementations, sandbox support, and other compatibility fixes. This phase +establishes a comprehensive CI pipeline that verifies Darling builds correctly +and that Nix functions inside it. + +CI work can begin as soon as Phase 3 is working (Nix installs and evaluates +inside Darling). Tests for later phases (daemon, remote builder) are added +incrementally as those phases land. + +--- + +## Context + +The current CI (`.github/workflows/actions.yaml`) only builds Debian packages. +It does not: + +- Build Darling with Nix. +- Run any functional tests. +- Verify Nix compatibility. +- Test inside a NixOS VM (which is needed for namespace/overlay support). + +We need to replace or supplement this with a Nix-native CI pipeline that runs +real integration tests. + +--- + +## Tasks + +### 6.1 — NixOS VM Test: Nix-in-Darling + +Create a NixOS VM test at `tests/nix-in-darling.nix` that exercises the full +Nix-inside-Darling pipeline end-to-end. + +**Test structure** (using `nixos/lib/testing-python.nix`): + +```nix +{ pkgs, ... }: +{ + name = "nix-in-darling"; + + nodes.machine = { config, pkgs, ... }: { + # Import our NixOS module + imports = [ ../nixosModules/darling ]; + + # Enable Darling + programs.darling.enable = true; + + # Give the VM enough resources + virtualisation.memorySize = 4096; + virtualisation.diskSize = 20480; # 20 GB for Nix store + virtualisation.cores = 4; + }; + + testScript = '' + machine.wait_for_unit("default.target") + + # Phase 0: Darling boots + machine.succeed("darling shell echo 'Hello from Darling'") + + # Phase 2: sandbox-exec stub exists + machine.succeed("darling shell test -x /usr/bin/sandbox-exec") + machine.succeed("darling shell /usr/bin/sandbox-exec -f /dev/null /bin/echo ok") + + # Phase 3: Install Nix + machine.succeed("scripts/install-nix-in-darling.sh") + + # Phase 3: Nix commands work + machine.succeed("darling-nix nix --version") + machine.succeed("darling-nix nix-instantiate --eval -E '1 + 1' | grep 2") + machine.succeed("darling-nix nix eval --expr 'builtins.currentSystem' | grep x86_64-darwin") + + # Phase 4: Trivial build + machine.succeed( + "darling-nix nix-build --expr '" + + "'derivation { name = \"test\"; builder = \"/bin/bash\"; " + + "args = [\"-c\" \"echo ok > \\$out\"]; " + + "system = \"x86_64-darwin\"; }'" + ) + + # Phase 4: Verify output + result = machine.succeed( + "darling shell cat $(darling-nix nix-build --no-link --expr '" + + "'derivation { name = \"test\"; builder = \"/bin/bash\"; " + + "args = [\"-c\" \"echo ok > \\$out\"]; " + + "system = \"x86_64-darwin\"; }')" + ) + assert "ok" in result, f"Expected 'ok' in output, got: {result}" + + machine.log("All Nix-in-Darling tests passed!") + ''; +} +``` + +**Key considerations**: + +- The test runs in a NixOS VM, which provides the kernel namespace support + Darling needs. This avoids requiring special privileges on the CI runner. +- The VM needs ample disk space (Darling prefix + Nix store + build artifacts). +- Timeout must be generous — Darling operations are slow, and the first Nix + installation involves downloading and unpacking the installer. +- The test should be structured so that early failures (Darling doesn't boot) + produce clear error messages rather than cryptic timeouts. + +--- + +### 6.2 — Wire Tests into `flake.nix` + +Add the NixOS VM test to the flake's `checks` output: + +```nix +checks.x86_64-linux = { + # Build Darling itself + darling-build = self.packages.x86_64-linux.darling; + + # NixOS integration test + nix-in-darling = import ./tests/nix-in-darling.nix { + inherit pkgs; + }; +}; +``` + +This allows running: + +```bash +# Run all checks +nix flake check + +# Run just the integration test +nix build .#checks.x86_64-linux.nix-in-darling +``` + +--- + +### 6.3 — GitHub Actions Workflow + +Replace or supplement the existing `.github/workflows/actions.yaml` with a +Nix-native workflow. + +**Workflow file**: `.github/workflows/nix-ci.yaml` + +```yaml +name: Nix CI + +on: + push: + branches: [main] + pull_request: + +jobs: + build: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + with: + submodules: recursive + + - uses: cachix/install-nix-action@v27 + with: + extra_nix_config: | + experimental-features = nix-command flakes + + - uses: cachix/cachix-action@v15 + with: + name: darling-nix # our Cachix cache + authToken: '${{ secrets.CACHIX_AUTH_TOKEN }}' + + - name: Build Darling + run: nix build .#darling -L + + - name: Build Darling SDK + run: nix build .#darling-sdk -L + + test-syscalls: + runs-on: ubuntu-latest + needs: build + steps: + - uses: actions/checkout@v4 + with: + submodules: recursive + + - uses: cachix/install-nix-action@v27 + with: + extra_nix_config: | + experimental-features = nix-command flakes + + - uses: cachix/cachix-action@v15 + with: + name: darling-nix + + - name: Run syscall regression tests + run: nix build .#checks.x86_64-linux.syscall-regression -L + + test-nix-integration: + runs-on: ubuntu-latest + needs: build + steps: + - uses: actions/checkout@v4 + with: + submodules: recursive + + - uses: cachix/install-nix-action@v27 + with: + extra_nix_config: | + experimental-features = nix-command flakes + system-features = kvm + + - uses: cachix/cachix-action@v15 + with: + name: darling-nix + + - name: Run Nix-in-Darling integration test + run: nix build .#checks.x86_64-linux.nix-in-darling -L + timeout-minutes: 60 # generous timeout for VM test +``` + +**Notes**: + +- The integration test requires KVM for the NixOS VM. GitHub's `ubuntu-latest` + runners have KVM available. Verify with `system-features = kvm` in the Nix + config. +- The build job runs first and pushes artifacts to Cachix. Subsequent test jobs + pull from the cache, avoiding redundant rebuilds. +- The `timeout-minutes: 60` is important — Darling operations inside a VM inside + CI can be very slow. Adjust as needed based on real-world timings. +- `submodules: recursive` is required because Darling has 100+ submodules. This + checkout step may itself take 5–10 minutes. + +**Alternative: use a self-hosted runner** if GitHub's runners are too slow or +lack KVM. A dedicated NixOS machine with nested virtualisation enabled would +provide the most reliable CI environment. + +--- + +### 6.4 — Syscall Regression Test Suite + +Create a set of small C programs under `tests/syscalls/` that exercise every +syscall we've fixed. These run inside `darling shell` and assert expected +behavior. + +**Directory structure**: + +``` +tests/ +├── syscalls/ +│ ├── test_lchflags.c +│ ├── test_setattrlist.c +│ ├── test_renameatx_np.c +│ ├── test_utimensat.c +│ ├── test_clonefile.c +│ ├── test_getentropy.c +│ ├── test_posix_spawn.c +│ ├── test_xattr.c +│ ├── test_fcntl_locking.c +│ └── run_all.sh +├── sandbox/ +│ ├── test_sandbox_exec.sh +│ ├── test_sandbox_init.c +│ └── run_all.sh +└── nix/ + ├── test_nix_eval.sh + ├── test_nix_build_trivial.sh + ├── test_nix_substitution.sh + └── run_all.sh +``` + +**Example test — `test_lchflags.c`**: + +```c +#include +#include +#include +#include +#include +#include +#include + +#define ASSERT(cond, msg) do { \ + if (!(cond)) { \ + fprintf(stderr, "FAIL: %s (errno=%d: %s)\n", msg, errno, strerror(errno)); \ + exit(1); \ + } \ +} while (0) + +int main(void) { + const char *path = "/tmp/test_lchflags_XXXXXX"; + char tmppath[256]; + strncpy(tmppath, path, sizeof(tmppath)); + + int fd = mkstemp(tmppath); + ASSERT(fd >= 0, "mkstemp failed"); + close(fd); + + /* Clear all flags — this is what Nix does */ + int ret = lchflags(tmppath, 0); + ASSERT(ret == 0, "lchflags(path, 0) should return 0"); + + unlink(tmppath); + printf("PASS: test_lchflags\n"); + return 0; +} +``` + +**Example test — `test_renameatx_np.c`**: + +```c +#include +#include +#include +#include +#include +#include +#include + +/* macOS renameatx_np flags */ +#ifndef RENAME_SWAP +#define RENAME_SWAP 0x00000002 +#endif +#ifndef RENAME_EXCL +#define RENAME_EXCL 0x00000004 +#endif + +extern int renameatx_np(int fromfd, const char *from, + int tofd, const char *to, unsigned int flags); + +#define ASSERT(cond, msg) do { \ + if (!(cond)) { \ + fprintf(stderr, "FAIL: %s (errno=%d: %s)\n", msg, errno, strerror(errno)); \ + exit(1); \ + } \ +} while (0) + +static void write_file(const char *path, const char *content) { + int fd = open(path, O_WRONLY | O_CREAT | O_TRUNC, 0644); + ASSERT(fd >= 0, "open for write failed"); + write(fd, content, strlen(content)); + close(fd); +} + +static void read_file(const char *path, char *buf, size_t len) { + int fd = open(path, O_RDONLY); + ASSERT(fd >= 0, "open for read failed"); + ssize_t n = read(fd, buf, len - 1); + ASSERT(n >= 0, "read failed"); + buf[n] = '\0'; + close(fd); +} + +int main(void) { + const char *a = "/tmp/renameatx_a"; + const char *b = "/tmp/renameatx_b"; + char buf[64]; + + /* Test RENAME_SWAP */ + write_file(a, "AAA"); + write_file(b, "BBB"); + + int ret = renameatx_np(AT_FDCWD, a, AT_FDCWD, b, RENAME_SWAP); + ASSERT(ret == 0, "renameatx_np RENAME_SWAP failed"); + + read_file(a, buf, sizeof(buf)); + ASSERT(strcmp(buf, "BBB") == 0, "after swap, a should contain BBB"); + + read_file(b, buf, sizeof(buf)); + ASSERT(strcmp(buf, "AAA") == 0, "after swap, b should contain AAA"); + + /* Test RENAME_EXCL */ + unlink(b); + ret = renameatx_np(AT_FDCWD, a, AT_FDCWD, b, RENAME_EXCL); + ASSERT(ret == 0, "renameatx_np RENAME_EXCL (target absent) should succeed"); + + write_file(a, "CCC"); + ret = renameatx_np(AT_FDCWD, a, AT_FDCWD, b, RENAME_EXCL); + ASSERT(ret != 0 && errno == EEXIST, + "renameatx_np RENAME_EXCL (target exists) should fail with EEXIST"); + + unlink(a); + unlink(b); + printf("PASS: test_renameatx_np\n"); + return 0; +} +``` + +**Runner script — `tests/syscalls/run_all.sh`**: + +```bash +#!/bin/bash +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +PASS=0 +FAIL=0 +ERRORS="" + +for test_src in "$SCRIPT_DIR"/test_*.c; do + test_name="$(basename "$test_src" .c)" + test_bin="/tmp/$test_name" + + echo "--- $test_name ---" + + # Compile inside Darling using Apple's clang + if ! cc -o "$test_bin" "$test_src" 2>&1; then + echo "FAIL: $test_name (compilation failed)" + FAIL=$((FAIL + 1)) + ERRORS="$ERRORS\n $test_name: compilation failed" + continue + fi + + # Run + if "$test_bin"; then + PASS=$((PASS + 1)) + else + FAIL=$((FAIL + 1)) + ERRORS="$ERRORS\n $test_name: test failed" + fi + + rm -f "$test_bin" +done + +echo "" +echo "=== Results: $PASS passed, $FAIL failed ===" +if [ $FAIL -gt 0 ]; then + echo -e "Failures:$ERRORS" + exit 1 +fi +``` + +**Integration with Nix**: Create a derivation that compiles and runs all tests +inside a Darling prefix (this requires a NixOS VM test context since Darling +needs namespace support): + +```nix +checks.x86_64-linux.syscall-regression = nixosTest { + name = "darling-syscall-regression"; + nodes.machine = { ... }: { + imports = [ ../nixosModules/darling ]; + programs.darling.enable = true; + }; + testScript = '' + machine.wait_for_unit("default.target") + machine.succeed("darling shell bash /path/to/tests/syscalls/run_all.sh") + ''; +}; +``` + +--- + +### 6.5 — Nix Compatibility Test Matrix + +Create a test that attempts to build an expanding set of Nixpkgs packages inside +Darling and tracks pass/fail rates over time. + +**File**: `tests/nix/compatibility-matrix.sh` + +**Approach**: + +```bash +#!/bin/bash +# Test building a set of packages inside Darling +# Tracks pass/fail for each package + +PACKAGES=( + # Tier 1: Must work (no native compilation, just fetch from cache) + "hello" + "which" + "yes" + + # Tier 2: Should work (simple C programs) + "tree" + "jq" + + # Tier 3: Stretch (complex builds) + "curl" + "git" + "python3" +) + +RESULTS_FILE="/tmp/compat-matrix-$(date +%Y%m%d).json" +echo '{"results": [' > "$RESULTS_FILE" + +for pkg in "${PACKAGES[@]}"; do + echo "--- Testing: $pkg ---" + start_time=$(date +%s) + + if darling-nix nix-build '' -A "$pkg" --system x86_64-darwin --no-out-link 2>/tmp/build-$pkg.log; then + status="pass" + else + status="fail" + fi + + end_time=$(date +%s) + duration=$((end_time - start_time)) + + echo " $pkg: $status (${duration}s)" + echo " {\"package\": \"$pkg\", \"status\": \"$status\", \"duration\": $duration}," >> "$RESULTS_FILE" +done + +# Close JSON (remove trailing comma hack) +sed -i '$ s/,$//' "$RESULTS_FILE" +echo ']}' >> "$RESULTS_FILE" + +echo "" +echo "Results written to $RESULTS_FILE" + +# Summary +pass_count=$(grep -c '"pass"' "$RESULTS_FILE" || true) +fail_count=$(grep -c '"fail"' "$RESULTS_FILE" || true) +total=${#PACKAGES[@]} +echo "=== Compatibility: $pass_count/$total passed ($fail_count failed) ===" +``` + +**Tracking over time**: Store the JSON results as CI artifacts. A simple script +can compare results between runs to detect regressions (a package that was +passing now fails) or progress (a package that was failing now passes). + +--- + +### 6.6 — Darling Build Smoke Test + +A lighter-weight test that doesn't need a NixOS VM — just verifies Darling +builds from source with Nix: + +```nix +checks.x86_64-linux.darling-build = self.packages.x86_64-linux.darling; +``` + +This runs as part of `nix flake check` and catches build regressions (missing +dependencies, broken patches, compiler errors) without the overhead of a VM +test. + +--- + +### 6.7 — Test Darling SDK Cross-Compilation + +Verify that the SDK output can be used to cross-compile Darwin binaries from +Linux (without running them inside Darling — just the compilation step): + +```bash +# Use the SDK's clang + ld64 to compile a Darwin binary on Linux +$darling_sdk/bin/x86_64-apple-darwin-ld64 ... # or however the SDK exposes the tools +``` + +This tests the SDK packaging independently of the Darling runtime. + +--- + +## Test Categories + +| Category | Runs In | Needs VM | Frequency | Phase Dependency | +|---|---|---|---|---| +| Build smoke test | Nix sandbox | No | Every PR | Phase 0 | +| SDK cross-compile | Nix sandbox | No | Every PR | Phase 0 | +| Syscall regression | Darling shell (in VM) | Yes | Every PR | Phase 1 | +| Sandbox stub test | Darling shell (in VM) | Yes | Every PR | Phase 2 | +| Nix installation | Darling shell (in VM) | Yes | Every PR | Phase 3 | +| Trivial build | Darling shell (in VM) | Yes | Every PR | Phase 4 | +| Compatibility matrix | Darling shell (in VM) | Yes | Nightly / weekly | Phase 4 | +| Daemon & multi-user | Darling shell (in VM) | Yes | Every PR | Phase 5 | +| Remote builder | NixOS VM with Nix daemon | Yes | Nightly / weekly | Phase 7 | + +--- + +## CI Performance Considerations + +NixOS VM tests are slow. Strategies to keep CI times reasonable: + +1. **Cachix**: Push all build artifacts to a binary cache. Subsequent runs skip + rebuilding Darling (which takes 30+ minutes from scratch). + +2. **Test parallelism**: Run the build smoke test and SDK test in parallel with + the VM-based tests (they're independent). + +3. **Incremental testing**: On PRs that only touch `plan/` or `docs/`, skip the + expensive VM tests. Use path filters in the workflow: + ```yaml + on: + push: + paths-ignore: + - 'plan/**' + - '*.md' + ``` + +4. **Test VM snapshots**: If the NixOS testing framework supports it, take a + snapshot after Darling initialization and restore from it for each test. This + avoids re-bootstrapping Darling's prefix on every test run. + +5. **Split VM tests**: Rather than one monolithic test, split into focused tests + (syscalls, sandbox, Nix install, build). Failed tests give faster feedback + about what broke. + +6. **Timeout management**: Set aggressive but realistic timeouts per test step. + A hanging test should fail fast rather than consume the full CI allocation: + ```python + # In the NixOS test script: + machine.succeed("timeout 300 darling-nix nix-build ...") + ``` + +--- + +## Verification Checklist + +After completing Phase 6, ALL of the following should be true: + +- [ ] `nix flake check` passes (includes build smoke test) +- [ ] `.github/workflows/nix-ci.yaml` exists and runs on PRs +- [ ] Syscall regression tests exist for `lchflags`, `renameatx_np`, `utimensat` (at minimum) +- [ ] Sandbox stub tests verify `sandbox-exec` passthrough works +- [ ] NixOS VM test installs Nix inside Darling and evaluates an expression +- [ ] NixOS VM test builds a trivial derivation inside Darling +- [ ] CI results are visible on GitHub PR checks +- [ ] Cachix cache is populated by CI and speeds up subsequent runs +- [ ] Compatibility matrix script exists and produces JSON output +- [ ] Adding a new syscall implementation has a clear path: implement → add test → CI verifies + +--- + +## Maintenance + +- **Adding new tests**: When a new syscall is implemented (Phase 1), add a + corresponding `test_.c` to `tests/syscalls/`. The runner script + picks it up automatically. + +- **Updating the compatibility matrix**: As more packages start working, add them + to the `PACKAGES` array. The matrix should only grow, never shrink (removing a + package hides regressions). + +- **Flaky tests**: If a test passes intermittently (likely due to Darling's + incomplete implementation), mark it as `@flaky` in the test script and track + it separately. Do not disable it — flaky tests are signals of real issues. + +- **CI costs**: NixOS VM tests are expensive. Monitor CI usage and adjust the + trigger frequency (e.g., move the compatibility matrix to weekly if it's too + costly to run on every PR). + +--- + +*[← Phase 5 — Nix Daemon](./07-phase5-daemon.md) | [Phase 7 — Remote Builder →](./09-phase7-remote-builder.md)* \ No newline at end of file diff --git a/plan/09-phase7-remote-builder.md b/plan/09-phase7-remote-builder.md new file mode 100644 index 000000000..cdbe27068 --- /dev/null +++ b/plan/09-phase7-remote-builder.md @@ -0,0 +1,630 @@ +# Phase 7 — Nixpkgs `x86_64-darwin` Remote Builder + +**Priority**: P2 · **Effort**: L (4–8 weeks) · **Depends on**: Phase 4 (derivation building), Phase 5 (Nix daemon) + +The ultimate goal of this project: use Darling as a **remote builder** so that a +Linux host's Nix daemon can offload `x86_64-darwin` builds to a Darling +instance — just as it would offload to a real macOS machine over SSH. + +This unlocks the ability for any NixOS machine to build and test Darwin packages +without Apple hardware. + +--- + +## Context + +Nix supports remote builds via two mechanisms: + +1. **SSH-based remote builders** (`nix.buildMachines`): The local Nix daemon + connects to a remote machine over SSH, copies the derivation closure, runs + the build remotely, and copies the result back. The remote machine must run + `nix-daemon` and accept SSH connections. + +2. **Build hooks**: A custom `build-hook` program that Nix invokes when it + encounters a derivation for a system it can't build locally. The hook decides + where and how to build it. + +For Darling, the SSH approach is the most natural: run `sshd` inside Darling, +configure the host's Nix daemon to treat it as a remote builder for +`x86_64-darwin`, and let the standard Nix remote-build protocol handle the rest. + +An alternative is a custom build hook that calls `darling shell` directly, +avoiding SSH overhead. Both approaches are covered below. + +--- + +## Architecture + +``` +┌──────────────────────────────────────────────────────────┐ +│ Linux Host (NixOS) │ +│ │ +│ User runs: nix build .#myPackage --system x86_64-darwin │ +│ │ │ +│ ▼ │ +│ ┌──────────────────────────────────┐ │ +│ │ Nix Daemon (Linux) │ │ +│ │ system = x86_64-linux │ │ +│ │ buildMachines includes: │ │ +│ │ { hostName = "darling-vm"; │ │ +│ │ systems = ["x86_64-darwin"];│ │ +│ │ sshKey = "..."; } │ │ +│ └──────────┬───────────────────────┘ │ +│ │ SSH (or darling-exec) │ +│ ▼ │ +│ ┌──────────────────────────────────┐ │ +│ │ Darling Container │ │ +│ │ ┌────────────────────────────┐ │ │ +│ │ │ sshd (port 2222) │ │ │ +│ │ │ nix-daemon │ │ │ +│ │ │ sandbox-exec stub │ │ │ +│ │ │ /nix/store (shared) │──┼── bind mount ──┐ │ +│ │ └────────────────────────────┘ │ │ │ +│ └──────────────────────────────────┘ │ │ +│ │ │ +│ /nix/store ◄────────────────────────────────────────┘ │ +│ │ +└──────────────────────────────────────────────────────────┘ +``` + +The key insight is the **shared `/nix/store`**: by bind-mounting or symlinking +the host's `/nix/store` into the Darling prefix, we avoid the expensive step of +copying store paths back and forth over SSH. The SSH connection is still used for +the build protocol (derivation transfer, build log streaming, result +registration) but the actual store content is shared via the filesystem. + +--- + +## Tasks + +### 7.1 — Run `sshd` Inside Darling + +Set up an SSH server inside the Darling prefix so the host's Nix daemon can +connect to it as a remote builder. + +**Steps**: + +1. **Install sshd**: Darling ships OpenSSH (`src/external/openssh/`). Verify + that `/usr/sbin/sshd` exists in the prefix and is functional. + +2. **Generate host keys**: + ```bash + darling shell ssh-keygen -A + ``` + +3. **Configure sshd** (`/etc/ssh/sshd_config` inside the prefix): + ``` + Port 2222 + ListenAddress 127.0.0.1 + PermitRootLogin yes + PubkeyAuthentication yes + AuthorizedKeysFile .ssh/authorized_keys + PasswordAuthentication no + UsePAM no + Subsystem sftp /usr/libexec/sftp-server + ``` + + Using port 2222 avoids conflict with the host's sshd on port 22. + +4. **Set up SSH keys**: Generate a keypair for the Nix daemon to use: + ```bash + ssh-keygen -t ed25519 -N "" -f /etc/nix/darling-builder-key + darling shell mkdir -p /var/root/.ssh + cat /etc/nix/darling-builder-key.pub | darling shell tee /var/root/.ssh/authorized_keys + darling shell chmod 600 /var/root/.ssh/authorized_keys + ``` + +5. **Start sshd**: + ```bash + darling shell /usr/sbin/sshd -f /etc/ssh/sshd_config + ``` + +6. **Verify connectivity**: + ```bash + ssh -i /etc/nix/darling-builder-key -p 2222 root@127.0.0.1 echo ok + # Expected: ok + ``` + +**Potential issues**: + +- **Network stack**: Darling's network layer needs to support `bind()` on + `127.0.0.1:2222` and `accept()` incoming connections. Since Darling maps to + Linux sockets, this should work, but verify. + +- **PTY allocation**: SSH uses pseudo-terminals. Darling needs working `/dev/ptmx` + and `openpty()`. Non-interactive commands (which is what Nix uses) may not need + a PTY, but the SSH handshake might still require basic PTY support. + +- **`sshd` privilege separation**: OpenSSH's privilege separation uses `fork`, + `setuid`, and `chroot`. If these don't work inside Darling, configure sshd + with `UsePrivilegeSeparation no` (deprecated but functional). + +- **PAM**: Set `UsePAM no` since Darling doesn't implement PAM. + +--- + +### 7.2 — Configure the Host as a Remote Build Client + +Add the Darling instance as a remote builder in the host's Nix configuration. + +**NixOS configuration**: + +```nix +nix.buildMachines = [{ + hostName = "127.0.0.1"; + port = 2222; + systems = [ "x86_64-darwin" ]; + sshUser = "root"; + sshKey = "/etc/nix/darling-builder-key"; + maxJobs = 4; + speedFactor = 1; # lower than native builders; adjust based on benchmarks + supportedFeatures = [ ]; + mandatoryFeatures = [ ]; +}]; + +nix.distributedBuilds = true; + +# Optional: only use the Darling builder for Darwin, not for Linux +nix.settings.extra-platforms = [ "x86_64-darwin" ]; +``` + +**Verification**: + +```bash +# Test that Nix can connect to the builder +nix store ping --store ssh://root@127.0.0.1:2222 + +# Test a remote build +nix build --expr 'derivation { name = "test"; builder = "/bin/bash"; args = ["-c" "echo ok > $out"]; system = "x86_64-darwin"; }' -L + +# The build should be offloaded to the Darling instance +``` + +**Troubleshooting**: + +```bash +# Check if the Nix daemon can reach sshd +sudo -u nix-daemon ssh -i /etc/nix/darling-builder-key -p 2222 root@127.0.0.1 nix --version + +# Check the Nix daemon logs for builder connection errors +journalctl -u nix-daemon -f + +# Verify the Darling sshd is listening +ss -tlnp | grep 2222 +``` + +--- + +### 7.3 — Shared `/nix/store` + +The naive remote-build setup copies store paths over SSH, which is extremely slow +for large closures. Since the Darling instance runs on the same machine, we can +share the store filesystem directly. + +**Mechanism**: Darling mounts the host's root filesystem at `/Volumes/SystemRoot` +inside the prefix. The host's `/nix/store` is therefore accessible at +`/Volumes/SystemRoot/nix/store` from within Darling. + +**Setup**: + +```bash +# Inside the Darling prefix, symlink /nix to the host's /nix +darling shell ln -sf /Volumes/SystemRoot/nix /nix +``` + +Or, if that conflicts with Darling's overlayfs: + +```bash +# Bind mount the host's /nix into the prefix +# This may need to be done during prefix initialization in darlingserver +mount --bind /nix ~/.darling/nix +``` + +**Benefits**: + +- **No copy overhead**: Store paths don't need to be transferred over SSH. The + Nix daemon on both sides sees the same physical files. +- **Shared garbage collection**: The host's GC manages the shared store. +- **Instant result availability**: After a Darwin build completes, its output is + immediately available on the host without copying. + +**Caveats**: + +- **Store database**: Nix's SQLite database (`/nix/var/nix/db/db.sqlite`) must + not be shared between the host and Darling Nix daemons — they're different + Nix instances with potentially different database schemas. Each needs its own + database. + + Solution: Configure the Darling Nix instance to use a different database + location: + ``` + # In /etc/nix/nix.conf inside Darling: + store = /nix + state = /var/nix # Darling-local state, not shared + ``` + Or use a local overlay for `/nix/var` while sharing `/nix/store`. + +- **Concurrent writes**: If both the host and Darling write to `/nix/store` + simultaneously, there's a risk of corruption. Mitigate by: + - Making the Darling Nix daemon the exclusive writer for `x86_64-darwin` paths. + - Using Nix's content-addressed store paths (which are safe for concurrent + writes since paths are determined by content). + - Using file-level locking (`fcntl`) which works across the shared mount. + +- **Permission mapping**: Darling's UID/GID namespace may differ from the host's. + Ensure that files written by Darling's `_nixbldN` users are readable by the + host's Nix daemon. This may require mapping UIDs or using a shared `nixbld` + group. + +**Fallback**: If shared store proves too complex, fall back to SSH-based copying. +It's slower but simpler and guaranteed correct. Use Nix's `--builders` flag with +`ssh-ng://` protocol which has optimised store path transfer. + +--- + +### 7.4 — Alternative: Custom Build Hook (No SSH) + +Instead of SSH, implement a custom Nix build hook that invokes `darling shell` +directly. This avoids the SSH setup entirely and may have lower overhead. + +**How Nix build hooks work**: + +1. Nix calls the `build-hook` program (configured in `nix.conf`) when a build + can't be performed locally. +2. The hook reads the derivation path and system type from stdin. +3. The hook decides whether to accept the build. If yes, it outputs the builder + machine specification. +4. Nix then proceeds to run the build on that machine. + +**Custom hook — `darling-build-hook`**: + +```bash +#!/usr/bin/env bash +# darling-build-hook — Nix build hook that offloads x86_64-darwin builds to Darling + +set -euo pipefail + +# Read build request from Nix +# Protocol: https://nixos.org/manual/nix/stable/advanced-topics/distributed-builds +read -r drv_path system + +if [[ "$system" != "x86_64-darwin" ]]; then + echo "# decline" # Not a Darwin build, let Nix handle it + exit 0 +fi + +echo "# accept" +echo "darling-builder x86_64-darwin /etc/nix/darling-builder-key 4 1" + +# Nix will now SSH to "darling-builder" (which must resolve, or use the +# machines file). Alternatively, this hook could run the build directly: +# +# darling shell nix-store --realise "$drv_path" +# echo "$drv_path" +``` + +**Note**: The build hook protocol is somewhat complex and version-dependent. The +SSH approach (7.1/7.2) is more battle-tested and recommended for initial +implementation. The custom hook is an optimisation for later. + +**Nix configuration for the hook**: + +```nix +nix.settings.build-hook = "/path/to/darling-build-hook"; +``` + +--- + +### 7.5 — NixOS Module for the Darling Builder + +Wrap all the setup (sshd, keys, store sharing, `nix.buildMachines`) into a +reusable NixOS module. + +**Module file**: `nixosModules/darling-builder.nix` + +```nix +{ config, lib, pkgs, ... }: + +with lib; + +let + cfg = config.services.darling-builder; +in { + options.services.darling-builder = { + enable = mkEnableOption "Darling-based x86_64-darwin remote builder"; + + port = mkOption { + type = types.port; + default = 2222; + description = "SSH port for the Darling builder"; + }; + + maxJobs = mkOption { + type = types.int; + default = 4; + description = "Maximum concurrent builds on the Darling builder"; + }; + + speedFactor = mkOption { + type = types.int; + default = 1; + description = "Speed factor (lower = deprioritised vs native builders)"; + }; + + shareStore = mkOption { + type = types.bool; + default = true; + description = "Share /nix/store between host and Darling (avoids copying)"; + }; + + sshKeyPath = mkOption { + type = types.str; + default = "/etc/nix/darling-builder-key"; + description = "Path to the SSH private key for connecting to the builder"; + }; + }; + + config = mkIf cfg.enable { + # Ensure Darling is available + programs.darling.enable = true; + + # Generate SSH keys if they don't exist + system.activationScripts.darling-builder-keys = '' + if [ ! -f ${cfg.sshKeyPath} ]; then + ${pkgs.openssh}/bin/ssh-keygen -t ed25519 -N "" -f ${cfg.sshKeyPath} + chown root:root ${cfg.sshKeyPath} + chmod 600 ${cfg.sshKeyPath} + fi + ''; + + # Set up the Darling prefix with sshd and Nix + systemd.services.darling-builder = { + description = "Darling x86_64-darwin Nix builder"; + wantedBy = [ "multi-user.target" ]; + after = [ "network.target" ]; + + serviceConfig = { + Type = "simple"; + ExecStartPre = [ + # Initialize prefix and install Nix if needed + "${pkgs.writeShellScript "darling-builder-init" '' + darling shell test -x /usr/sbin/sshd || exit 1 + darling shell test -x /usr/bin/sandbox-exec || exit 1 + + # Set up SSH authorized keys + darling shell mkdir -p /var/root/.ssh + cat ${cfg.sshKeyPath}.pub | darling shell tee /var/root/.ssh/authorized_keys > /dev/null + darling shell chmod 600 /var/root/.ssh/authorized_keys + + # Generate host keys if needed + darling shell test -f /etc/ssh/ssh_host_ed25519_key || darling shell ssh-keygen -A + + ${optionalString cfg.shareStore '' + # Symlink /nix to host's /nix via /Volumes/SystemRoot + darling shell ln -sf /Volumes/SystemRoot/nix /nix 2>/dev/null || true + ''} + ''}" + ]; + ExecStart = "${pkgs.darling}/bin/darling shell /usr/sbin/sshd -D -f /etc/ssh/sshd_config -p ${toString cfg.port}"; + Restart = "on-failure"; + RestartSec = 5; + }; + }; + + # Register as a Nix remote builder + nix.buildMachines = [{ + hostName = "127.0.0.1"; + port = cfg.port; + systems = [ "x86_64-darwin" ]; + sshUser = "root"; + sshKey = cfg.sshKeyPath; + maxJobs = cfg.maxJobs; + speedFactor = cfg.speedFactor; + supportedFeatures = [ ]; + mandatoryFeatures = [ ]; + }]; + + nix.distributedBuilds = true; + }; +} +``` + +**Usage** (in a NixOS configuration): + +```nix +{ + imports = [ ./path/to/darling-nix/nixosModules/darling-builder.nix ]; + + services.darling-builder = { + enable = true; + maxJobs = 8; + shareStore = true; + }; +} +``` + +After `nixos-rebuild switch`, the user can immediately build Darwin packages: + +```bash +nix build nixpkgs#hello --system x86_64-darwin +``` + +--- + +### 7.6 — Test Top Nixpkgs Packages + +Once the builder is operational, systematically test building the most +commonly-used `x86_64-darwin` packages from Nixpkgs. + +**Tier 1 — Must pass** (fetch from binary cache, minimal building): + +| Package | Why It Matters | +|---|---| +| `hello` | Simplest C program; validates full stdenv pipeline | +| `which` | Trivial utility; shell script install | +| `coreutils` | Foundation of every build; exercises many syscalls | +| `bash` | Builder shell; must work perfectly | +| `gnugrep` | Used in stdenv setup scripts | +| `gnused` | Used in stdenv setup scripts | +| `gawk` | Used in stdenv setup scripts | + +**Tier 2 — Should pass** (moderate complexity): + +| Package | Why It Matters | +|---|---| +| `curl` | Needed for fetching; exercises TLS + network | +| `git` | Needed for `fetchgit` in derivations | +| `python3` | Common build dependency; complex build | +| `jq` | Used in many CI scripts | +| `openssl` | Crypto library; exercises many low-level APIs | +| `pkg-config` | Build tool; should be straightforward | +| `cmake` | Build tool; complex but well-tested | + +**Tier 3 — Stretch** (complex, many dependencies): + +| Package | Why It Matters | +|---|---| +| `nodejs` | Large build; JavaScript ecosystem foundation | +| `go` | Self-hosting compiler; stresses the runtime | +| `rustc` | Very large build; exercises many syscalls | +| `llvm` | Compiler infrastructure; tests C++ heavily | +| `ghc` | Haskell compiler; extremely complex build | + +**Tracking**: Use the compatibility matrix from [Phase 6](./08-phase6-ci.md) +(task 6.5) to track pass/fail rates. Run this as a nightly CI job and publish +results to a dashboard or markdown file in the repo. + +**When something fails**: For each failure: + +1. Capture the full build log. +2. Identify the first error (often buried under cascading failures). +3. Determine if it's a syscall issue (→ Phase 1), a sandbox issue (→ Phase 2), + a coreutils issue (→ Phase 4.6), or a new category. +4. File an issue with the `[compat]` label. +5. Add it to `plan/syscall-triage.md` if it's a new syscall. + +--- + +### 7.7 — Documentation and Templates + +Create user-facing documentation so others can set up their own Darling builders. + +**Deliverables**: + +1. **NixOS wiki page**: Step-by-step guide for setting up a Darling-based Darwin + builder on NixOS. Cover both the NixOS module approach and the manual setup. + +2. **Flake template** (`templates/darling-builder`): + ```bash + nix flake init -t github:user/darling-nix#darling-builder + ``` + Generates a minimal `flake.nix` + NixOS configuration that sets up the + builder. + +3. **Troubleshooting guide**: Common issues and their solutions: + - "Connection refused" → sshd not running or wrong port + - "Permission denied" → SSH key mismatch + - "Build failed with signal 11" → unimplemented syscall → file an issue + - "Store path not valid" → shared store database mismatch + - "builder for '...' failed with exit code 1" → check the build log + +4. **Performance tuning guide**: Tips for getting the best performance: + - Use binary substitution aggressively (`substituters` in `nix.conf`) + - Set `max-jobs` based on available CPU cores + - Use `--cores N` to limit per-build parallelism + - Enable store sharing to avoid copy overhead + - Put the Nix store on fast storage (SSD/NVMe) + +--- + +## Security Considerations + +Running sshd inside Darling on `127.0.0.1:2222` is relatively safe: + +- **Loopback only**: The SSH server only listens on localhost. It's not reachable + from the network. +- **Key-based auth only**: Password authentication is disabled. Only the specific + key generated for the builder can connect. +- **Contained environment**: The Darling prefix is isolated from the host via + namespaces. Even if an attacker gains access to the Darling sshd, they're + inside a container with limited host access. +- **Shared store risk**: If `/nix/store` is shared, a compromised builder could + write malicious store paths. Mitigate by: + - Only sharing the store read-only from the host side. + - Using Nix's content-addressing to verify outputs. + - Running the Darling builder with minimal host capabilities. + +For production use, consider running the Darling builder inside an additional +isolation layer (systemd-nspawn, VM, or dedicated user namespace) to defense-in- +depth against container escapes. + +--- + +## Performance Expectations + +With store sharing enabled: + +| Operation | Expected Overhead vs Native macOS | +|---|---| +| Binary substitution | ~1.2–1.5× (NAR unpack syscall overhead) | +| Nix evaluation | ~2–5× (CPU-bound, translation overhead) | +| C compilation (clang) | ~3–8× (many syscalls, process spawning) | +| Linking (ld64) | ~2–4× (I/O bound, moderate syscall count) | +| Full `hello` build | ~3–5× (mostly substitution + simple compile) | +| Full `python3` build | ~5–10× (complex build, many phases) | + +Without store sharing (SSH copy): + +- Add ~30 seconds per 100 MB of closure for each copy direction. +- A typical stdenv closure is ~500 MB, so expect ~2.5 minutes overhead per build + just for copying. + +**Recommendation**: Always enable store sharing for local Darling builders. SSH +copy mode is only useful for remote machines running Darling (future work). + +--- + +## Verification Checklist + +After completing Phase 7, ALL of the following must pass: + +- [ ] `sshd` runs inside Darling and accepts SSH connections from the host +- [ ] `ssh -p 2222 root@127.0.0.1 nix --version` returns a Nix version string +- [ ] Host's `nix.buildMachines` includes the Darling builder +- [ ] `nix build --expr '...' --system x86_64-darwin` offloads to the Darling builder +- [ ] Build log is streamed back to the host in real time +- [ ] Build output is available in the host's `/nix/store` after completion +- [ ] `/nix/store` is shared (no SSH copy overhead) when `shareStore = true` +- [ ] `nix build nixpkgs#hello --system x86_64-darwin` succeeds (Tier 1 package) +- [ ] At least 5/7 Tier 1 packages build successfully +- [ ] At least 3/7 Tier 2 packages build successfully +- [ ] The NixOS module (`services.darling-builder`) works end-to-end +- [ ] Documentation exists for manual and module-based setup + +--- + +## What This Enables + +Once Phase 7 is working, any NixOS user can: + +```nix +# flake.nix +{ + outputs = { self, nixpkgs }: { + packages.x86_64-darwin.myApp = nixpkgs.legacyPackages.x86_64-darwin.callPackage ./. {}; + }; +} +``` + +```bash +# Build a Darwin package on a Linux machine +nix build .#packages.x86_64-darwin.myApp +``` + +This is the same workflow they'd use with a real macOS remote builder, but +without needing Apple hardware. The Darling builder is transparent to the user — +they don't need to know or care that it's running inside a compatibility layer. + +--- + +*[← Phase 6 — CI & Testing](./08-phase6-ci.md) | [Phase 8 — Stretch Goals →](./10-phase8-stretch.md)* \ No newline at end of file diff --git a/plan/10-phase8-stretch.md b/plan/10-phase8-stretch.md new file mode 100644 index 000000000..a78fefca3 --- /dev/null +++ b/plan/10-phase8-stretch.md @@ -0,0 +1,412 @@ +# Phase 8 — Long-Term / Stretch Goals + +**Priority**: P3 · **Effort**: XL (months–years) · **Depends on**: Phase 7 (remote builder) + +These are aspirational items that would make the Darling+Nix story truly +compelling but are not required for basic functionality. Each is a significant +project in its own right. They're documented here to provide direction for +future contributors and to ensure the earlier phases don't make architectural +decisions that would preclude these goals. + +--- + +## 8.1 — `aarch64-darwin` Support + +**What**: Build and test Apple Silicon (`aarch64-darwin`) packages on Linux. + +**Why it matters**: Apple has fully transitioned to ARM. The majority of macOS +users now run Apple Silicon. Nixpkgs' `aarch64-darwin` support is growing +rapidly, but CI coverage is limited by hardware availability. + +**Current state**: Darling only supports `x86_64`. The entire codebase — +darlingserver's syscall translation, the Mach-O loader, dyld, and all the Darwin +libraries — is x86_64-only. + +**Approach options**: + +| Option | Complexity | Performance | Notes | +|---|---|---|---| +| QEMU user-mode emulation | Medium | Slow (~10–50×) | Translate AArch64 instructions to x86_64; `qemu-aarch64` already exists but doesn't handle Mach-O | +| Full AArch64 Darling port | Very High | Near-native on aarch64-linux | Requires porting all of darlingserver, dyld, and libSystem to AArch64 | +| Rosetta-like translation | Extremely High | Fast (~1.5–3×) | AOT binary translation from AArch64 Mach-O to x86_64 ELF; research-grade effort | +| FEX-Emu integration | High | Moderate (~3–8×) | FEX-Emu handles x86_64→AArch64 translation; combine with Darling for Mach-O→ELF on AArch64 Linux hosts | + +**Recommended path**: Start with QEMU user-mode for correctness testing (not +performance). A Darling-aware QEMU wrapper that loads Mach-O binaries and +translates Darwin syscalls via darlingserver, with AArch64 instruction emulation +handled by QEMU. + +Long-term, a native AArch64 port of Darling is the right answer if the project +gains enough contributors. + +**Prerequisites**: +- All Phase 1–7 work must be solid on x86_64 first. +- Darlingserver's architecture must be cleanly separated from x86_64 specifics + (register mapping, calling conventions, instruction patching). +- The Mach-O loader must handle `arm64` and `arm64e` slices. + +**Effort**: 6–18 months for QEMU approach; years for native port. + +--- + +## 8.2 — GUI Application Testing + +**What**: Run macOS GUI applications inside Darling on Linux with enough fidelity +for automated screenshot-based testing. + +**Why it matters**: Many Nixpkgs Darwin packages include GUI components (e.g., +Emacs with Cocoa frontend, various `.app` bundles). Currently there's no way to +test these on Linux. + +**Current state**: Darling has partial Cocoa/AppKit support via +[Cocotron](https://github.com/darlinghq/darling-cocotron), which translates +Cocoa drawing calls to X11. Basic windows can be created but most applications +crash or render incorrectly. + +**Approach**: + +1. **Headless rendering**: Run Darling with a virtual X11 server (`Xvfb`) or + Wayland compositor (`wlheadless`). Cocotron renders to the virtual display. + +2. **Screenshot capture**: After launching an app, capture the framebuffer and + compare against reference screenshots using image comparison tools (e.g., + `perceptualdiff`, `pixelmatch`). + +3. **Accessibility-based testing**: If Darling implements enough of the + Accessibility framework, use it for UI testing without screenshots (more + robust to rendering differences). + +**Example test workflow**: + +```bash +# Start Xvfb +Xvfb :99 -screen 0 1920x1080x24 & +export DISPLAY=:99 + +# Launch a Cocoa app inside Darling +darling shell open -a /Applications/TextEdit.app & + +# Wait for window to appear +sleep 5 + +# Capture screenshot +import -window root /tmp/screenshot.png + +# Compare against reference +perceptualdiff /tmp/screenshot.png tests/references/textedit.png +``` + +**Blockers**: +- Cocotron's X11 backend needs significant work for modern Cocoa APIs. +- Core Animation, Metal, and modern AppKit features are unimplemented. +- Font rendering differences between macOS (Core Text) and Linux (FreeType) will + cause pixel-level mismatches — need fuzzy comparison. + +**Effort**: 3–12 months for basic "does the window open and look roughly right" +testing. Much longer for full GUI fidelity. + +--- + +## 8.3 — Nix Flake Integration Library + +**What**: A Nix library function (`buildDarwinWithDarling`) that lets any flake +build Darwin packages using Darling, without the user needing to set up a remote +builder. + +**Why it matters**: The remote builder approach (Phase 7) requires system-level +NixOS configuration. A flake-level library would make Darwin-on-Linux accessible +to any Nix user, even those not running NixOS. + +**Design**: + +```nix +# In any project's flake.nix: +{ + inputs = { + nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable"; + darling-nix.url = "github:user/darling-nix"; + }; + + outputs = { self, nixpkgs, darling-nix }: { + packages.x86_64-linux.hello-darwin = + darling-nix.lib.buildDarwinWithDarling { + inherit nixpkgs; + # Standard mkDerivation arguments: + pname = "hello"; + version = "2.12.1"; + src = ./. ; + buildInputs = [ ]; + # Darling-specific options: + darlingPrefix = "~/.darling"; # optional + shareStore = true; # optional + }; + }; +} +``` + +**Implementation sketch**: + +The `buildDarwinWithDarling` function would: + +1. Build the derivation specification (`.drv` file) using Nixpkgs' Darwin stdenv. +2. Wrap the build invocation in a `darling shell` call. +3. Handle store path management (shared or copied). +4. Return the output path as a normal Nix derivation result. + +**Challenges**: + +- This requires Darling to be runnable inside a Nix sandbox (needs namespace + capabilities). May need `__noChroot = true` or a fixed-output derivation + wrapper. +- Must handle the bootstrap problem: the Darling binary itself needs to be built + for Linux before it can be used to build Darwin packages. +- Nix's build sandbox on Linux may conflict with Darling's namespace usage. + +**Alternative**: Instead of embedding Darling in the build, provide a flake that +sets up the remote builder and let users `nix build --system x86_64-darwin` as +usual. This is simpler and avoids the sandbox-within-sandbox issues. + +**Effort**: 2–4 months for the library; ongoing maintenance as Nixpkgs evolves. + +--- + +## 8.4 — Upstream Contributions + +**What**: Push all syscall fixes, sandbox stubs, and compatibility improvements +back to the [upstream Darling project](https://github.com/darlinghq/darling). + +**Why it matters**: Maintaining a fork is expensive. Upstream contributions +benefit the entire Darling community and reduce our maintenance burden. + +**Strategy**: + +1. **Keep changes modular**: Each syscall fix should be a self-contained commit + with a clear description and test case. This makes upstream review easier. + +2. **Separate Nix-specific changes**: Things like the `sandbox-exec` stub, + Directory Services stubs, and the NixOS module should be kept in our fork/ + overlay. They're useful for the Nix use case but may not align with + upstream's goals. + +3. **Coordinate with upstream**: Open issues/discussions on the Darling GitHub + before submitting large changes. The Darling team may have opinions on + implementation approaches (e.g., they may prefer a different `setattrlist` + implementation than what we propose). + +4. **Contribute tests**: Upstream Darling has minimal testing. Contributing our + syscall regression tests (Phase 6.4) would be valuable even without the + corresponding fixes. + +**Candidates for upstreaming**: + +| Change | Upstream Value | Nix-Specific? | +|---|---|---| +| `setattrlist` / `fsetattrlist` implementation | High — many programs need this | No | +| `renameatx_np` (syscall 488) implementation | High — modern coreutils need this | No | +| `utimensat` fixes | High — affects `touch` and many tools | No | +| `clonefile` stub (returns `ENOTSUP`) | Medium — graceful degradation | No | +| `getentropy` mapping to `getrandom` | Medium — security-related programs need this | No | +| macOS version bump (10.15 → 11.0) | High — unblocks modern software | No | +| `sandbox-exec` stub | Medium — useful but opinionated | Somewhat | +| `sandbox_init` errorbuf fix | Low — cosmetic | No | +| Directory Services stubs | Low — very Nix-specific | Yes | +| NixOS module | None — Nix ecosystem only | Yes | + +**Effort**: Ongoing; each upstream PR takes 1–4 weeks including review cycles. + +--- + +## 8.5 — macOS SDK Management + +**What**: Automate downloading, unpacking, and managing macOS SDKs inside the +Darling prefix via Nix derivations. + +**Why it matters**: Building Darwin software requires Apple's SDK headers and +frameworks. Currently, users must manually download Xcode or the Command Line +Tools and install them. This is a friction point and a licensing grey area. + +**Approach**: + +1. **Use Nixpkgs' existing SDK infrastructure**: Nixpkgs already packages macOS + SDKs (e.g., `apple-sdk_15`, `apple-sdk_14`). These are available as Nix + derivations and can be installed into the Darling prefix. + +2. **Automatic SDK installation**: The Darling builder setup (Phase 7 NixOS + module) should automatically install the appropriate SDK into the prefix: + ```nix + services.darling-builder.sdk = pkgs.darwin.apple_sdk_15; + ``` + +3. **SDK version selection**: Allow users to choose which SDK version to use. + Different Nixpkgs branches may require different SDK versions. + +**Licensing considerations**: + +- Apple's Xcode license allows use on Apple hardware. Using Apple's SDK headers + on Linux (via Darling) is a legal grey area. +- Nixpkgs' SDK packages contain only headers and `.tbd` stub files (not actual + binaries), which may be covered by fair use for interoperability purposes. +- Darling itself ships significant Apple-derived open-source code under APSL. +- **Recommendation**: Document the licensing situation clearly. Do not distribute + Apple proprietary binaries. Use open-source headers where possible and let + users supply their own SDK if needed. + +**Effort**: 2–4 weeks for the Nix integration; legal review is separate. + +--- + +## 8.6 — Binary Cache for `x86_64-darwin` + +**What**: Run a Darling-based build farm (Hydra, Garnix, or custom) that +continuously builds `x86_64-darwin` packages and populates a public binary +cache. + +**Why it matters**: If Darling can reliably build Darwin packages, we can provide +a community binary cache that eliminates the need for Apple hardware for most +users. Even partial coverage (the top 1000 most-used packages) would be +enormously valuable. + +**Architecture**: + +``` +┌──────────────────────────────────────────────┐ +│ Hydra / Build Coordinator (NixOS) │ +│ jobset: nixpkgs x86_64-darwin │ +│ │ +│ ┌────────────────────────────────────────┐ │ +│ │ Builder 1: NixOS + Darling │ │ +│ │ services.darling-builder.enable │ │ +│ │ maxJobs = 8 │ │ +│ └────────────────────────────────────────┘ │ +│ ┌────────────────────────────────────────┐ │ +│ │ Builder 2: NixOS + Darling │ │ +│ │ ... │ │ +│ └────────────────────────────────────────┘ │ +│ │ +│ → pushes NARs to: darling-cache.example.org │ +└──────────────────────────────────────────────┘ +``` + +**Users add the cache**: + +```nix +nix.settings = { + substituters = [ "https://darling-cache.example.org" ]; + trusted-public-keys = [ "darling-cache.example.org-1:AAAA..." ]; +}; +``` + +**Challenges**: + +- **Reproducibility**: Builds inside Darling may not produce bit-for-bit + identical outputs to builds on real macOS. This means the cache serves + "Darling-built" packages that might differ from the official `cache.nixos.org` + Darwin packages. Users need to understand this. + +- **Coverage**: Not all packages will build successfully inside Darling. The + cache must gracefully handle partial coverage — users fall back to building + locally (or on real macOS) for packages that aren't cached. + +- **Maintenance**: A build farm requires ongoing infrastructure maintenance, + monitoring, and storage management. + +- **Trust**: Users must trust the cache operator. Use Nix's content-addressing + and signing to provide integrity guarantees. + +**Effort**: 1–3 months to set up the infrastructure; ongoing maintenance. + +--- + +## 8.7 — Build Reproducibility Verification + +**What**: Ensure that derivation outputs built inside Darling are as close to +bit-for-bit identical as possible to those built on real macOS. + +**Why it matters**: If Darling-built packages differ from real macOS-built +packages, it undermines the value of the compatibility layer. Ideally, a package +built inside Darling should be indistinguishable from one built on real macOS. + +**Approach**: + +1. **Identify sources of non-determinism**: + - Timestamps embedded in binaries (Mach-O headers, `__DATA` segments). + - Hostname / username embedded in build artifacts. + - Random data (UUIDs, build IDs) that differ between builds. + - Filesystem ordering differences (`readdir` order). + - Floating-point rounding differences (unlikely but possible if Darling's FPU + emulation differs). + +2. **Compare build outputs**: + ```bash + # Build on real macOS + real_output=$(nix-build '' -A hello --system x86_64-darwin) + + # Build inside Darling + darling_output=$(darling-nix nix-build '' -A hello --system x86_64-darwin) + + # Compare + diffoscope "$real_output" "$darling_output" + ``` + +3. **Fix divergences**: For each difference, determine if it's a Darling bug + (fix it) or inherent non-determinism (document it). + +4. **Content-addressed derivations**: Nix's experimental content-addressed (CA) + derivation mode hashes outputs by content rather than by input. This means + two builds that produce identical content (regardless of where they were + built) share the same store path. Push for CA derivation support to make + Darling-built and macOS-built outputs interchangeable. + +**Effort**: Ongoing; this is a continuous improvement process rather than a +one-time task. + +--- + +## 8.8 — Container / VM Image Distribution + +**What**: Distribute pre-configured Darling+Nix environments as OCI container +images or VM images for easy adoption. + +**Why it matters**: Not everyone uses NixOS. A Docker/Podman image or a QEMU VM +image with Darling+Nix pre-installed would make Darwin-on-Linux accessible to +the broader developer community. + +**Deliverables**: + +1. **OCI image** (`ghcr.io/user/darling-nix:latest`): + ```dockerfile + FROM nixos/nix:latest + RUN nix build github:user/darling-nix#darling + RUN /path/to/scripts/install-nix-in-darling.sh + ENTRYPOINT ["darling-nix"] + ``` + Requires: Docker-in-Docker or privileged mode for namespaces. + +2. **NixOS VM image**: A QEMU qcow2 image built with `nixos-generators` that + includes the `darling-builder` NixOS module pre-configured. + +3. **GitHub Codespaces / Gitpod integration**: A `.devcontainer.json` that sets + up a development environment with Darling+Nix for cloud-based development. + +**Effort**: 2–4 weeks per distribution format. + +--- + +## Summary: Stretch Goal Prioritization + +If resources allow work beyond Phase 7, prioritize in this order: + +1. **8.4 — Upstream contributions**: Lowest effort, highest community value. +2. **8.5 — SDK management**: Directly improves usability of Phases 4–7. +3. **8.7 — Reproducibility**: Builds confidence in Darling-built packages. +4. **8.6 — Binary cache**: High value but requires infrastructure commitment. +5. **8.3 — Flake library**: Nice developer experience but requires solving hard + sandbox-in-sandbox problems. +6. **8.8 — Container images**: Broadens the audience beyond NixOS users. +7. **8.2 — GUI testing**: Niche but uniquely valuable; depends on Cocotron + maturity. +8. **8.1 — `aarch64-darwin`**: Most impactful long-term, but the most work. + +--- + +*[← Phase 7 — Remote Builder](./09-phase7-remote-builder.md) | [Architecture →](./11-architecture.md)* \ No newline at end of file diff --git a/plan/11-architecture.md b/plan/11-architecture.md new file mode 100644 index 000000000..4a4d102eb --- /dev/null +++ b/plan/11-architecture.md @@ -0,0 +1,403 @@ +# Architecture & Key Technical Decisions + +This document describes the high-level system architecture and records the +rationale behind major technical decisions. It serves as a reference for +contributors who need to understand *why* things are designed the way they are, +not just *what* to build. + +--- + +## System Architecture + +### Overview + +``` +┌──────────────────────────────────────────────────────────────────────┐ +│ Linux Host (NixOS) │ +│ │ +│ ┌────────────────────────────────────────────────────────────────┐ │ +│ │ Host Nix Daemon │ │ +│ │ ┌──────────────────────────────────────────────────────────┐ │ │ +│ │ │ nix.buildMachines = [{ │ │ │ +│ │ │ hostName = "127.0.0.1"; port = 2222; │ │ │ +│ │ │ systems = ["x86_64-darwin"]; │ │ │ +│ │ │ }] │ │ │ +│ │ └────────────────────┬─────────────────────────────────────┘ │ │ +│ └───────────────────────┼────────────────────────────────────────┘ │ +│ │ SSH / darling-exec │ +│ ┌───────────────────────▼────────────────────────────────────────┐ │ +│ │ Darling Container (overlayfs prefix at ~/.darling) │ │ +│ │ │ │ +│ │ ┌──────────────────────────────────────────────────────────┐ │ │ +│ │ │ darlingserver │ │ │ +│ │ │ • Translates Darwin/XNU syscalls → Linux syscalls │ │ │ +│ │ │ • Manages Mach-O loading via mldr + dyld │ │ │ +│ │ │ • Provides namespace isolation (mount, PID, user) │ │ │ +│ │ └──────────────────────────────────────────────────────────┘ │ │ +│ │ │ │ +│ │ ┌─────────────────┐ ┌──────────────────────────────────┐ │ │ +│ │ │ Darwin Userland │ │ Nix (Darwin build) │ │ │ +│ │ │ • dyld │ │ • nix / nix-daemon │ │ │ +│ │ │ • libSystem │ │ • nix-build / nix-store │ │ │ +│ │ │ • libc │ │ • sandbox-exec stub │ │ │ +│ │ │ • CoreFoundation│ │ • curl, bash, coreutils │ │ │ +│ │ │ • libdispatch │ │ • clang, ld64 (from stdenv) │ │ │ +│ │ │ • Obj-C runtime │ │ • Darwin stdenv build machinery │ │ │ +│ │ └─────────────────┘ └──────────────────────────────────┘ │ │ +│ │ │ │ +│ │ /nix/store ──symlink──▶ /Volumes/SystemRoot/nix/store │ │ +│ │ /dev, /proc ──mount──▶ host kernel interfaces │ │ +│ └────────────────────────────────────────────────────────────────┘ │ +│ │ +│ /nix/store (shared filesystem — single source of truth) │ +│ │ +└──────────────────────────────────────────────────────────────────────┘ +``` + +### Component Responsibilities + +| Component | Role | Location | +|---|---|---| +| **darlingserver** | Userspace syscall translator. Intercepts Mach/BSD traps from Darwin binaries and translates them to Linux equivalents. Manages the container namespace. | `src/external/darlingserver/` (submodule) | +| **mldr** | Mach-O loader. Loads Darwin Mach-O executables on Linux, sets up the process image, and hands off to `dyld`. | `src/libelfloader/` | +| **dyld** | Apple's dynamic linker. Resolves `@rpath`, `@loader_path`, loads `.dylib` dependencies. Runs inside the translated environment. | `src/external/dyld/` | +| **libSystem / libc** | Darwin's standard C library. Provides POSIX wrappers (`lchflags`, `setattrlist`, `posix_spawn`, etc.) that ultimately invoke darlingserver-translated syscalls. | `src/external/libc/`, `src/external/libsystem/` | +| **sandbox-exec stub** | Passthrough shim replacing Apple's `sandbox-exec`. Ignores sandbox profiles and directly `exec`s the builder command. | `src/sandbox-exec/` (to be created, Phase 2) | +| **Nix (Darwin)** | The Nix package manager compiled for `x86_64-darwin`, fetched from the official binary cache. Runs inside Darling as a Darwin process. | `/nix/store/...-nix-*/` inside the prefix | +| **Host Nix Daemon** | The Linux-native Nix daemon that orchestrates builds. Offloads `x86_64-darwin` builds to the Darling instance via SSH or a custom build hook. | Standard NixOS `nix-daemon.service` | +| **Darling prefix** | An overlayfs-backed directory (`~/.darling`) that provides a macOS-like filesystem hierarchy. System files are read-only from the Darling installation; user/build files are writable in the upper layer. | `~/.darling/` (runtime) | + +### Data Flow: Building a Darwin Derivation + +``` +1. User: nix build .#myPkg --system x86_64-darwin + │ +2. Host Nix Daemon: Identifies x86_64-darwin → selects Darling builder + │ +3. SSH transport: Connects to sshd inside Darling (port 2222) + │ +4. Darling nix-daemon: Receives build request + │ +5. Nix build setup: Creates /tmp/nix-build-myPkg.drv-0/ + Writes .sandbox.sb profile + │ +6. sandbox-exec stub: Ignores profile, exec's /bin/bash + │ +7. bash builder: Sources $stdenv/setup + Runs unpack → configure → build → install → fixup + │ +8. Syscall translation: Every Darwin syscall (open, stat, mmap, posix_spawn, + lchflags, renameatx_np, ...) goes through darlingserver + and becomes the Linux equivalent + │ +9. Build output: Written to /nix/store/...-myPkg + │ +10. Store registration: nix-daemon registers the path in SQLite + │ +11. Shared store: Output is immediately visible to the host + (shared /nix/store via bind mount / symlink) + │ +12. Host Nix Daemon: Marks the build as complete, returns result to user +``` + +--- + +## Key Technical Decisions + +### Decision 1: Syscall Implementation Depth + +**Decision**: Implement syscalls to the minimum depth required for Nix, not for +general macOS compatibility. + +**Rationale**: Full macOS API coverage is a multi-year effort (and the upstream +Darling project's ongoing goal). We should be surgical about what we implement. +For example: + +- `setattrlist` only needs to handle `ATTR_CMN_FLAGS` for clearing + `UF_IMMUTABLE`. We don't need full Finder-info, resource-fork, or ACL + support through this API. +- `clonefile` can return `ENOTSUP` — Nix gracefully falls back to regular copy. +- `sandbox_init` can return success with a NULL error buffer — Darling's + namespace isolation is already sufficient. + +**Trade-off**: Some non-Nix Darwin programs may still fail. That's acceptable — +this project's scope is Nix support, not universal macOS compatibility. + +**How this affects contributors**: When implementing a syscall, always check +what the *caller* actually needs. Read the Nix source (or whatever Nix-ecosystem +program is calling it) and implement only what's required to make that caller +succeed. Document the scope of the implementation in code comments. + +--- + +### Decision 2: Sandbox Strategy + +**Decision**: Start with a `sandbox-exec` stub that passes through to `exec`. +Do NOT attempt to implement Apple's Sandbox Profile Language initially. + +**Rationale**: Nix's sandbox on Darwin is defense-in-depth. The macOS sandbox +(`sandbox-exec` + `.sb` profiles) restricts file access, network access, and +process operations during builds. Inside Darling, we already have: + +1. **Linux namespace isolation**: The Darling container uses mount namespaces + (overlayfs), PID namespaces, and optionally network namespaces. This provides + equivalent-or-stronger isolation to macOS's sandbox for build purposes. + +2. **Nix's own isolation**: Nix controls `$PATH`, `$HOME`, `$TMPDIR`, and other + environment variables. The build environment is intentionally spartan. The + macOS sandbox adds a second layer, but its absence doesn't fundamentally + compromise build isolation. + +3. **No untrusted code**: In the Nix builder context, the code being executed + comes from derivations the user has chosen to build. The sandbox prevents + accidental side effects (e.g., a build script accidentally writing to `/usr`), + not malicious code execution. + +**When to revisit**: If Darling is ever used to run arbitrary untrusted macOS +software (not just Nix builds), proper sandbox support becomes important. See +[Phase 2, task 2.4](./04-phase2-sandbox.md#24--stretch-basic-sandbox-profile-language-parsing) +for the stretch-goal design. + +--- + +### Decision 3: Single-User vs. Multi-User Nix + +**Decision**: Target single-user mode first (Phase 3), add multi-user later +(Phase 5). + +**Rationale**: Single-user mode has far fewer moving parts: + +| Aspect | Single-User | Multi-User | +|---|---|---| +| Daemon required | No | Yes | +| Build users required | No | Yes (30+ users) | +| Directory Services required | No | Yes (`dseditgroup`, `sysadminctl`) | +| `launchd` integration | No | Yes | +| `setuid` / privilege separation | No | Yes | +| Concurrent builds | No | Yes | +| Suitable for development/testing | Yes | Yes | +| Suitable for production builders | Maybe | Yes | + +Single-user mode is sufficient for the MVP (Phases 0–4). It lets us validate +that Nix works inside Darling without solving the much harder problems of user +management and privilege separation inside a namespace-based container. + +--- + +### Decision 4: Shared vs. Separate Nix Store + +**Decision**: Share the host's `/nix/store` with the Darling prefix via the +existing `/Volumes/SystemRoot` mount. + +**Rationale**: + +- **Avoids duplicating store contents.** A typical Nix closure for building + Darwin packages is 500 MB–2 GB. Duplicating this inside the Darling prefix + wastes disk and slows down builds (SSH copy overhead). + +- **Host Nix daemon can manage garbage collection.** With a shared store, there's + a single GC root set. Without sharing, the Darling-side store accumulates + garbage that's invisible to the host's `nix-collect-garbage`. + +- **Darwin build outputs are immediately available on the host.** No need to + copy results back after a build completes — the output is already in the + shared `/nix/store`. + +**Implementation**: + +``` +# Inside the Darling prefix: +/nix → /Volumes/SystemRoot/nix (symlink) + → /nix/store (host's store, shared) + → /nix/var (Darling-local state, NOT shared) +``` + +The store content (`/nix/store`) is shared, but the state +(`/nix/var/nix/db/db.sqlite`, `/nix/var/nix/daemon-socket/`, etc.) is +Darling-local. This prevents database conflicts between the host and Darling +Nix instances. + +**Caveat**: Darling's overlayfs may interfere with writes to the shared store. +If so, use a direct bind mount (`mount --bind /nix/store ~/.darling/nix/store`) +during prefix initialization, bypassing the overlayfs upper layer for the store +directory. Test this during Phase 3. + +**Fallback**: If shared store causes issues (permission mismatches, locking +conflicts, overlayfs quirks), fall back to a fully separate store inside the +Darling prefix. This is simpler but slower (requires SSH-based store path +transfer for the remote builder in Phase 7). + +--- + +### Decision 5: macOS Version Target + +**Decision**: Target macOS 11.0 (Big Sur) as the emulated version. + +**Rationale**: + +- Darling's `CMakeLists.txt` already sets `CMAKE_OSX_DEPLOYMENT_TARGET` to 11.0. +- Nixpkgs' Darwin stdenv targets macOS 11.0+ for `x86_64-darwin` builds. +- The official Nix binary cache (`cache.nixos.org`) serves binaries built with + `-mmacosx-version-min=11.0` or higher. +- macOS 10.15 (Catalina, which Darling currently reports at runtime) is past + end-of-life and increasingly unsupported by modern software. + +**What this means**: + +- `sw_vers` inside Darling should report `ProductVersion: 11.0`. +- `__MAC_OS_X_VERSION_MIN_REQUIRED` should be `110000` (Big Sur). +- Any `@available(macOS 11.0, *)` checks in Darling's libraries should evaluate + to true. +- APIs introduced in Big Sur (e.g., `os_log` improvements, certain + `posix_spawn` attributes) should be available or gracefully stubbed. + +**Risk**: Bumping the version may expose new code paths in Darling's libraries +that call unimplemented APIs. This is acceptable — it surfaces real issues rather +than papering over them with an artificially old version number. + +--- + +### Decision 6: CI Strategy + +**Decision**: Use NixOS VM tests as the primary CI mechanism, with lighter-weight +build-only tests for fast feedback. + +**Rationale**: Darling requires Linux namespace support (user namespaces, +overlayfs, mount namespaces) that isn't available inside a standard container or +Nix build sandbox. NixOS VM tests provide a full Linux kernel, which guarantees +the necessary capabilities. + +**Trade-off**: VM tests are slow (5–30 minutes). We mitigate this with: + +1. A fast "build smoke test" that just builds Darling (no VM, runs in Nix + sandbox). Catches compilation regressions in ~10 minutes. +2. Binary caching (Cachix) so that the Darling build itself is rarely rebuilt + from scratch in CI. +3. Parallelised test jobs — the build test and VM test run concurrently. +4. Path-based CI triggers — documentation-only changes skip the VM test. + +See [Phase 6](./08-phase6-ci.md) for full CI design. + +--- + +### Decision 7: SSH vs. Custom Build Hook for Remote Builds + +**Decision**: Use SSH-based remote builds as the primary mechanism. A custom +build hook is a secondary optimisation. + +**Rationale**: + +| Aspect | SSH Remote Builder | Custom Build Hook | +|---|---|---| +| Protocol maturity | Battle-tested, standard Nix feature | Custom, must handle edge cases | +| Setup complexity | Moderate (sshd + keys) | Low (single script) | +| Store transfer | Built-in (SSH or shared mount) | Must be implemented | +| Build log streaming | Built-in | Must be implemented | +| Error handling | Built-in | Must be implemented | +| Nix version coupling | Low (protocol is stable) | High (hook interface can change) | + +SSH remote builds are the standard way to offload Nix builds to another machine. +Even though Darling runs on the same host, treating it as a "remote" builder via +SSH reuses all of Nix's existing remote-build infrastructure — derivation +closure transfer, build log streaming, result retrieval, and error handling. + +The custom build hook (calling `darling shell` directly) avoids SSH overhead and +is simpler to set up, but it requires reimplementing protocol details that SSH +remote builds handle automatically. It's better suited as an optimisation after +the SSH approach is proven. + +See [Phase 7](./09-phase7-remote-builder.md) for both approaches. + +--- + +## Subsystem Map + +A quick reference for where to find things in the Darling source tree: + +``` +darling-nix/ +├── plan/ # This planning documentation +├── src/ +│ ├── sandbox/ # sandbox_init, sandbox_check, etc. (stubs) +│ │ └── sandbox.c # ← Fix errorbuf handling (Phase 2.2) +│ ├── libsandbox/ # libsandbox.1.dylib shim +│ ├── diskutil/ # diskutil shell script (eject only) +│ ├── duct/src/ # Minimal shims (acl, dns_sd, os_log) +│ ├── launchd/ # launchd + launchctl implementation +│ │ ├── src/core.c # posix_spawn usage for job management +│ │ └── support/launchctl.c # launchctl CLI, uses lchflags +│ ├── external/ +│ │ ├── darlingserver/ # ← Main syscall translation (submodule) +│ │ ├── libc/ # Darwin libc (lchflags, setattrlist wrappers) +│ │ ├── xnu/ # XNU kernel headers + libsystem_kernel +│ │ │ └── darling/src/libsystem_kernel/ # BSD syscall stubs +│ │ ├── dyld/ # Apple's dynamic linker +│ │ ├── libsystem/ # libSystem umbrella library +│ │ ├── corefoundation/ # CoreFoundation framework +│ │ ├── foundation/ # Foundation framework (NSFileManager, etc.) +│ │ ├── libdispatch/ # Grand Central Dispatch +│ │ ├── objc4/runtime/ # Objective-C runtime +│ │ ├── openssh/ # OpenSSH (sshd for remote builder) +│ │ ├── bash/ # Darling's built-in bash +│ │ ├── cctools-port/ # ld64, ar, ranlib (Apple linker tools) +│ │ ├── swift/ # Swift runtime libraries +│ │ └── ... # ~100 more submodules +│ ├── native/ # Linux-native wrappers (wraps ELF libs for Darwin use) +│ ├── frameworks/ # macOS frameworks (AppKit, CoreGraphics, etc.) +│ └── private-frameworks/ # Private frameworks (Bom, etc.) +├── CMakeLists.txt # Top-level build configuration +├── .gitmodules # Submodule definitions (~100 entries) +├── .github/workflows/ # CI (currently Debian-only) +└── tools/ # Build/install utilities +``` + +### Where Syscall Changes Go + +``` +User code (e.g. Nix) calls lchflags() + │ + ▼ +src/external/libc/ ← Darwin libc wrapper: translates to setattrlist() + │ + ▼ +src/external/xnu/darling/src/libsystem_kernel/ ← BSD syscall stub: + packages args into a trap + │ + ▼ +src/external/darlingserver/ ← Handles the trap on the Linux side: + translates setattrlist → ioctl/utimensat/etc. + │ + ▼ +Linux kernel ← Actual filesystem operation +``` + +Understanding this call chain is essential for debugging. If `lchflags` fails: + +1. Is the libc wrapper calling the right syscall number? → Check `src/external/libc/` +2. Is the syscall number wired in the kernel trap table? → Check `src/external/xnu/.../libsystem_kernel/` +3. Is darlingserver handling it? → Check `src/external/darlingserver/` +4. Is the Linux translation correct? → `strace` on the darlingserver process + +--- + +## Glossary + +| Term | Meaning | +|---|---| +| **Darling prefix** (DPREFIX) | The overlayfs-backed directory (`~/.darling`) that provides the macOS filesystem hierarchy. Analogous to Wine's WINEPREFIX. | +| **darlingserver** | The userspace process that handles Darwin syscall translation. Replaces the earlier LKM (Linux Kernel Module) approach. | +| **mldr** | Mach-O loader — the ELF-side binary that loads a Mach-O executable and sets up the Darling execution environment. | +| **dyld** | Apple's dynamic linker. Handles `@rpath`, `@loader_path`, and `.dylib` loading within the Darwin process. | +| **Mach-O** | The executable format used by macOS (analogous to ELF on Linux). | +| **libSystem** | macOS's umbrella system library (analogous to `libc.so` on Linux but includes more). Contains libc, libm, libpthread, etc. | +| **stdenv** | Nix's standard build environment. The Darwin stdenv provides clang, ld64, Apple SDK headers, and shell scripts for the build phases. | +| **NAR** | Nix Archive — Nix's serialisation format for store paths. Used for binary substitution (downloading pre-built packages). | +| **Binary substitution** | Downloading pre-built packages from a binary cache instead of building from source. Critical for performance inside Darling. | +| **sandbox-exec** | macOS's command-line sandbox tool. Applies a Sandbox Profile (`.sb` file) before executing a command. Nix uses this for build isolation on Darwin. | +| **SBPL** | Sandbox Profile Language — the Scheme-based DSL used in `.sb` files to define sandbox rules. | +| **cctools** | Apple's binary tools suite (`ld64`, `ar`, `ranlib`, `otool`, `install_name_tool`). Darling uses the `cctools-port` fork that builds on Linux. | +| **overlayfs** | Linux filesystem that layers a writable upper directory over a read-only lower directory. Darling uses this for prefixes so the base system is shared and user changes are isolated. | + +--- + +*[← Phase 8 — Stretch Goals](./10-phase8-stretch.md) | [Back to Plan Index](./README.md)* \ No newline at end of file diff --git a/plan/README.md b/plan/README.md new file mode 100644 index 000000000..74e08aa71 --- /dev/null +++ b/plan/README.md @@ -0,0 +1,61 @@ +# PLAN: Making Darling Fully Capable of Running Nix + +> **Goal**: Enable Darling (macOS compatibility layer for Linux) to run the Nix +> package manager reliably, so that Linux machines can build, test, and +> cross-compile `x86_64-darwin` Nix derivations — analogous to how Wine enables +> building and testing Windows binaries on Linux. + +## Plan Documents + +| Document | Description | +|---|---| +| [Background & Current State](./00-background.md) | Motivation, what works today, what doesn't | +| [Known Blockers](./01-blockers.md) | Detailed analysis of each blocking issue with fix strategies | +| [Phase 0 — Nix Packaging + DevShell](./02-phase0-packaging.md) | `flake.nix`, devShell, `.envrc`, NixOS module | +| [Phase 1 — Core Syscall Fixes](./03-phase1-syscalls.md) | `setattrlist`, `renameatx_np`, `utimensat`, etc. | +| [Phase 2 — Sandbox Stub](./04-phase2-sandbox.md) | `sandbox-exec` passthrough, sandbox API stubs | +| [Phase 3 — Nix Installation](./05-phase3-nix-install.md) | Automated installer, verification, wrappers | +| [Phase 4 — Derivation Building](./06-phase4-building.md) | Trivial derivations → stdenv → binary substitution | +| [Phase 5 — Nix Daemon](./07-phase5-daemon.md) | Multi-user mode, Directory Services stubs, launchd | +| [Phase 6 — CI & Testing](./08-phase6-ci.md) | NixOS VM tests, regression suite, GitHub Actions | +| [Phase 7 — Remote Builder](./09-phase7-remote-builder.md) | Darling as a `nix.buildMachines` target | +| [Phase 8 — Stretch Goals](./10-phase8-stretch.md) | `aarch64-darwin`, GUI testing, Hydra builder | +| [Architecture](./11-architecture.md) | System diagram, key technical decisions | + +## Priority & Effort Estimates + +| Phase | Priority | Effort | Depends On | +|-------|----------|--------|------------| +| Phase 0 — Nix packaging + devShell | P0 | S (1–2 weeks) | — | +| Phase 1 — Syscall fixes | P0 | L (4–8 weeks) | Phase 0 | +| Phase 2 — Sandbox stub | P0 | S (1 week) | — | +| Phase 3 — Nix installation | P0 | M (2–3 weeks) | Phases 1, 2 | +| Phase 4 — Derivation building | P1 | L (4–8 weeks) | Phase 3 | +| Phase 5 — Nix daemon | P2 | M (2–4 weeks) | Phase 4 | +| Phase 6 — CI/testing | P1 | M (2–3 weeks) | Phase 3 | +| Phase 7 — Remote builder | P2 | L (4–8 weeks) | Phases 4, 5 | +| Phase 8 — Stretch goals | P3 | XL (months) | Phase 7 | + +**Estimated time to MVP** (Phases 0–3): ~8–14 weeks of focused effort. + +**Estimated time to usable Darwin builder** (through Phase 7): ~6–12 months. + +## How to Contribute + +1. **Pick a task** from any phase document (earlier phases first). +2. **Check upstream** [Darling issues](https://github.com/darlinghq/darling/issues) for existing work. +3. **Write a minimal reproducer** — a small C program or shell command that demonstrates the bug inside `darling shell`. +4. **Fix it** in the appropriate subsystem (`darlingserver` for syscalls, `src/external/libc` for wrappers, `src/sandbox` for sandbox, etc.). +5. **Add a test** to the regression suite (see [Phase 6](./08-phase6-ci.md)). +6. **Submit a PR** to this repo, and consider upstreaming to `darlinghq/darling`. + +## References + +- [Darling Project](https://www.darlinghq.org/) — upstream macOS compatibility layer +- [Darling GitHub](https://github.com/darlinghq/darling) — upstream source +- [nixie-dev/darling-nix](https://github.com/nixie-dev/darling-nix) — Nix overlay for Darling +- [Nix All The Way Down](https://ersei.net/en/blog/nix-all-the-way-down) — blog post documenting Nix-in-Darling attempt +- [Nix Darwin sandbox source](https://github.com/NixOS/nix/blob/master/src/libstore/platform/darwin.cc) — Nix's `sandbox-exec` invocation +- [Apple `setattrlist` docs](https://developer.apple.com/documentation/kernel/1387673-setattrlist) +- [Apple `renameatx_np` docs](https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/renameatx_np.2.html) +- [Darling Docs — Build Instructions](https://docs.darlinghq.org/build-instructions.html) \ No newline at end of file