diff --git a/PLAN.md b/PLAN.md index 257ee748c..8f3abaf68 100644 --- a/PLAN.md +++ b/PLAN.md @@ -19,11 +19,25 @@ See the **[plan/](./plan/)** directory for all details. | Phase 4 β€” Building | 🚧 Tooling ready | `scripts/build-trivial.sh` (new) | | Phase 5 β€” Daemon | 🚧 Stubs done | `src/dirserv/` (new), `tests/dirserv/` (new) | | Phase 6 β€” CI | 🚧 In progress | `.tangled/workflows/ci.yml`, `tests/darling-smoke.nix`, `tests/nix-in-darling.nix`, `tests/nix/compatibility-matrix.sh` (new) | -| Phase 7 β€” Remote Builder | 🚧 Module & hook ready | `nix/darlingBuilderModule.nix` (new), `scripts/darling-build-hook` (new), `tests/darling-builder.nix` (new) | +| Phase 7 β€” Remote Builder | 🚧 Module, hook & docs ready | `nix/darlingBuilderModule.nix` (new), `scripts/darling-build-hook` (new), `tests/darling-builder.nix` (new), `docs/darwin-builder.md` (new), `templates/darling-builder/` (new) | | Phase 8 β€” Stretch | πŸ“‹ Planned | β€” | ### Recently Completed +- **Phase 7.7 β€” Documentation and flake template**: Created + `docs/darwin-builder.md` β€” comprehensive user-facing guide covering + NixOS module quick start, manual setup (sshd, SSH keys, builder + registration), shared `/nix/store` configuration, verification + procedures, custom build hook (no SSH) alternative, performance + tuning (binary substitution, job parallelism, store sharing, storage), + troubleshooting (connection refused, permission denied, unimplemented + syscalls, database errors, sandbox issues, slow builds), security + considerations, and architecture diagram. Created + `templates/darling-builder/` β€” a `nix flake init` template that + generates a ready-to-use NixOS configuration with the Darling builder + module pre-configured. Wired into `flake.nix` as + `templates.darling-builder`. Includes its own README with options + reference, architecture diagram, and troubleshooting section. - **Phase 7.5 β€” NixOS module for Darling builder**: Created `nix/darlingBuilderModule.nix` β€” a full NixOS module that sets up a Darling instance as a `nix.buildMachines` remote builder for @@ -187,6 +201,7 @@ See the **[plan/](./plan/)** directory for all details. | [plan/10-phase8-stretch.md](./plan/10-phase8-stretch.md) | `aarch64-darwin`, GUI testing, Hydra builder | | [plan/11-architecture.md](./plan/11-architecture.md) | System diagram, key technical decisions, glossary | | [plan/syscall-triage.md](./plan/syscall-triage.md) | Tracking table for unimplemented syscalls | +| [docs/darwin-builder.md](./docs/darwin-builder.md) | **User guide** β€” setup, troubleshooting, performance tuning | ## New File Map @@ -195,7 +210,9 @@ Files created or modified as part of this plan: ```text darling-nix/ β”œβ”€β”€ .tangled/workflows/ci.yml # tangled.org CI workflow (Phase 6) -β”œβ”€β”€ flake.nix # Flake with package, devShell, NixOS module, builder (Phase 0, 7) +β”œβ”€β”€ docs/ +β”‚ └── darwin-builder.md # NEW β€” User-facing setup guide, troubleshooting, perf tuning (Phase 7.7) +β”œβ”€β”€ flake.nix # Flake with package, devShell, NixOS module, builder, templates (Phase 0, 7) β”œβ”€β”€ nix/ β”‚ β”œβ”€β”€ package.nix # Darling Nix derivation (Phase 0) β”‚ β”œβ”€β”€ devShell.nix # Developer shell (Phase 0) @@ -235,6 +252,10 @@ darling-nix/ β”‚ β”œβ”€β”€ test_renameatx_np.c # renameatx_np tests (Phase 1) β”‚ β”œβ”€β”€ test_setattrlist_flags.c # setattrlist ATTR_CMN_FLAGS tests (Phase 1) β”‚ └── test_utimensat.c # utimensat/timestamp tests (Phase 1) +β”œβ”€β”€ templates/ +β”‚ └── darling-builder/ # NEW β€” Flake template for Darwin builder setup (Phase 7.7) +β”‚ β”œβ”€β”€ flake.nix # Ready-to-use NixOS config with services.darling-builder +β”‚ └── README.md # Quick start, options reference, troubleshooting └── plan/ β”œβ”€β”€ README.md # Index + priority table β”œβ”€β”€ 00-background.md # Motivation & current state @@ -373,12 +394,13 @@ Step 5 (remote builder) extends the MVP into a **usable Darwin builder**. ./tests/nix/compatibility-matrix.sh --tier 1 ./tests/nix/compatibility-matrix.sh --output results.json --compare previous.json ``` - - 7.7: Write user-facing docs and flake template + - ~~7.7: Write user-facing docs and flake template~~ βœ… Done β€” see `docs/darwin-builder.md` and `templates/darling-builder/` ### Completed Task Summary | Task | Status | Description | |------|--------|-------------| +| 7.7 | βœ… | User-facing docs (`docs/darwin-builder.md`) and flake template (`templates/darling-builder/`) | | 1.1 | βœ… | `setattrlist`/`fsetattrlist`/`getattrlist` β€” ATTR_CMN_FLAGS, CRTIME, CHGTIME | | 1.2 | βœ… | `lchflags` return value β€” verified via 1.1 | | 1.3 | βœ… | `renameatx_np` (syscall 488) β€” maps to Linux `renameat2` | @@ -424,6 +446,8 @@ Step 5 (remote builder) extends the MVP into a **usable Darwin builder**. | `nix build .#checks.x86_64-linux.darling-smoke -L` | NixOS VM smoke test (no network) | After building Darling | | `nix build .#checks.x86_64-linux.nix-in-darling -L` | Full Nix-in-Darling integration test | End-to-end validation | | `nix build .#checks.x86_64-linux.darling-builder -L` | Remote builder VM test (sshd, SSH auth, service) | After editing `nix/darlingBuilderModule.nix` | +| `docs/darwin-builder.md` | User-facing setup guide, troubleshooting, perf tuning | Setting up a Darling builder for the first time | +| `nix flake init -t .#darling-builder` | Generate a ready-to-use NixOS config with the builder | Bootstrapping a new Darling builder project | See [plan/README.md](./plan/README.md) for the full priority table and effort estimates. \ No newline at end of file diff --git a/docs/darwin-builder.md b/docs/darwin-builder.md new file mode 100644 index 000000000..67972803e --- /dev/null +++ b/docs/darwin-builder.md @@ -0,0 +1,713 @@ +# Setting Up a Darling-Based Darwin Builder + +> Build `x86_64-darwin` Nix packages on Linux β€” no Apple hardware required. + +This guide walks you through setting up [Darling](https://www.darlinghq.org/) +as a Nix remote builder so your Linux machine can build macOS (`x86_64-darwin`) +derivations. This is analogous to how Wine enables running Windows binaries on +Linux, but for macOS build toolchains. + +## Table of Contents + +- [Overview](#overview) +- [Prerequisites](#prerequisites) +- [Quick Start (NixOS Module)](#quick-start-nixos-module) +- [Manual Setup](#manual-setup) +- [Shared /nix/store](#shared-nixstore) +- [Verifying the Builder](#verifying-the-builder) +- [Alternative: Custom Build Hook (No SSH)](#alternative-custom-build-hook-no-ssh) +- [Performance Tuning](#performance-tuning) +- [Troubleshooting](#troubleshooting) +- [Security Considerations](#security-considerations) +- [Architecture](#architecture) + +--- + +## Overview + +Darling is a macOS compatibility layer for Linux that translates macOS system +calls into Linux equivalents. By running Nix inside Darling, we get a builder +that: + +- Reports `builtins.currentSystem == "x86_64-darwin"` +- Executes Darwin derivations using macOS-compatible toolchains +- Integrates with Nix's remote builder infrastructure over SSH +- Can optionally share `/nix/store` with the host for zero-copy builds + +**Two approaches are available:** + +| Approach | Pros | Cons | +|----------|------|------| +| **NixOS Module** (SSH-based) | Declarative, integrates with `nix.buildMachines`, works with any Nix client | Requires sshd inside Darling | +| **Custom Build Hook** (no SSH) | Simpler setup, lower overhead | Only works locally, less standard | + +--- + +## Prerequisites + +- **Linux x86_64** host (NixOS recommended, but any Linux with Nix works) +- **Nix** with flakes enabled (`experimental-features = nix-command flakes`) +- **Kernel support**: unprivileged user namespaces and overlayfs + ```bash + # Check user namespace support + sysctl kernel.unprivileged_userns_clone + # Should be: kernel.unprivileged_userns_clone = 1 + + # If not, enable it (NixOS handles this automatically): + sudo sysctl kernel.unprivileged_userns_clone=1 + ``` + +--- + +## Quick Start (NixOS Module) + +The fastest path on NixOS is the declarative module. Add this to your +NixOS configuration: + +```nix +# /etc/nixos/flake.nix (or wherever your system flake is) +{ + inputs = { + nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable"; + darling-nix = { + url = "github:user/darling-nix"; # adjust to actual repo + inputs.nixpkgs.follows = "nixpkgs"; + }; + }; + + outputs = { nixpkgs, darling-nix, ... }: { + nixosConfigurations.myhost = nixpkgs.lib.nixosSystem { + system = "x86_64-linux"; + modules = [ + ./configuration.nix + + # Base Darling support (programs.darling) + darling-nix.nixosModules.nixos + + # Darling builder service (services.darling-builder) + darling-nix.nixosModules.darling-builder + + { + services.darling-builder = { + enable = true; + maxJobs = 4; + shareStore = true; # share /nix/store between host and Darling + }; + } + ]; + }; + }; +} +``` + +Then rebuild and test: + +```bash +# Apply the configuration +sudo nixos-rebuild switch + +# Test connectivity to the Darling builder +darling-builder-test + +# Build a Darwin package from your Linux host! +nix build nixpkgs#hello --system x86_64-darwin +``` + +### Module Options Reference + +| Option | Type | Default | Description | +|--------|------|---------|-------------| +| `enable` | bool | `false` | Enable the Darling builder service | +| `package` | package | `pkgs.darling` | Darling package to use | +| `port` | int | `2222` | SSH port for the builder (inside Darling) | +| `maxJobs` | int | `1` | Maximum concurrent build jobs | +| `speedFactor` | int | `1` | Nix speed factor (lower = deprioritised vs native builders) | +| `shareStore` | bool | `false` | Share `/nix/store` between host and Darling via `/Volumes/SystemRoot` | +| `sshKeyPath` | path | `/etc/nix/darling-builder-key` | Path to the SSH private key for the Nix daemon | +| `prefixPath` | path | `/var/lib/darling-builder` | Path to the Darling prefix directory | +| `supportedFeatures` | list of str | `[]` | Nix supported features for this builder | +| `mandatoryFeatures` | list of str | `[]` | Nix mandatory features for this builder | +| `installNix` | bool | `true` | Automatically install Nix inside the Darling prefix | +| `nixVersion` | str | `"2.24.10"` | Nix version to install inside Darling | + +--- + +## Manual Setup + +If you're not on NixOS or prefer manual control, follow these steps. + +### 1. Install Darling + +```bash +# Using the flake +nix profile install github:user/darling-nix + +# Or build from source +git clone https://github.com/user/darling-nix +cd darling-nix +nix build +export PATH="$(pwd)/result/bin:$PATH" +``` + +### 2. Initialise the Darling Prefix + +```bash +# Boot Darling (creates the prefix at ~/.darling by default) +darling shell echo "Darling is working" + +# Verify macOS identity +darling shell uname -s # β†’ Darwin +darling shell sw_vers # β†’ macOS 11.7.4 (Big Sur) +``` + +### 3. Install Nix Inside Darling + +Use the automated installer script: + +```bash +./scripts/install-nix-in-darling.sh +``` + +Or install manually: + +```bash +# Download the Nix installer inside Darling +darling shell bash -lc ' + curl -L https://nixos.org/nix/install -o /tmp/install-nix + chmod +x /tmp/install-nix + /tmp/install-nix --no-daemon +' +``` + +Verify: + +```bash +./scripts/verify-nix.sh + +# Or manually: +darling shell bash -lc "nix --version" +darling shell bash -lc "nix eval --raw --expr 'builtins.currentSystem'" +# β†’ x86_64-darwin +``` + +### 4. Set Up sshd Inside Darling + +```bash +# Generate host keys +darling shell ssh-keygen -A + +# Write sshd config +darling shell tee /etc/ssh/sshd_config << 'EOF' +Port 2222 +ListenAddress 127.0.0.1 +PermitRootLogin yes +PubkeyAuthentication yes +AuthorizedKeysFile .ssh/authorized_keys +PasswordAuthentication no +ChallengeResponseAuthentication no +UsePAM no +Subsystem sftp /usr/libexec/sftp-server +EOF + +# Generate an SSH keypair for the Nix daemon +sudo ssh-keygen -t ed25519 -N "" -f /etc/nix/darling-builder-key + +# Install the public key inside Darling +darling shell mkdir -p /var/root/.ssh +sudo cat /etc/nix/darling-builder-key.pub | darling shell tee /var/root/.ssh/authorized_keys +darling shell chmod 700 /var/root/.ssh +darling shell chmod 600 /var/root/.ssh/authorized_keys + +# Start sshd +darling shell /usr/sbin/sshd -f /etc/ssh/sshd_config + +# Verify connectivity +ssh -i /etc/nix/darling-builder-key -p 2222 -o StrictHostKeyChecking=no root@127.0.0.1 echo ok +# β†’ ok +``` + +### 5. Register the Builder with Nix + +Add to `/etc/nix/machines` (or use `nix.buildMachines` on NixOS): + +``` +ssh://root@127.0.0.1 x86_64-darwin /etc/nix/darling-builder-key 4 1 - - - +``` + +The fields are: `store-uri system ssh-key max-jobs speed-factor supported-features mandatory-features public-host-key` + +Or in `nix.conf`: + +```ini +builders = ssh://root@127.0.0.1:2222 x86_64-darwin /etc/nix/darling-builder-key 4 1 - - - +``` + +Enable distributed builds: + +```ini +# In /etc/nix/nix.conf +builders-use-substitutes = true +``` + +### 6. Test a Build + +```bash +# Ping the builder +nix store ping --store ssh://root@127.0.0.1:2222 + +# Build a trivial derivation +nix build --expr 'derivation { + name = "hello-darwin"; + builder = "/bin/bash"; + args = ["-c" "echo hello from darwin > $out"]; + system = "x86_64-darwin"; +}' -L + +# Build a real package +nix build nixpkgs#hello --system x86_64-darwin -L +``` + +--- + +## Shared /nix/store + +By default, Nix copies store paths over SSH to the builder and back. Since +Darling runs on the same machine, this is wasteful. You can share +`/nix/store` directly. + +### How It Works + +Darling mounts the host's root filesystem at `/Volumes/SystemRoot` inside the +prefix. This means the host's `/nix/store` is accessible at +`/Volumes/SystemRoot/nix/store` from within Darling. + +### Setup + +```bash +# Inside Darling, symlink /nix to the host's /nix +darling shell ln -sf /Volumes/SystemRoot/nix /nix +``` + +Or, if there's a conflict with Darling's overlay filesystem: + +```bash +# Bind mount the host's /nix into the prefix +sudo mount --bind /nix ~/.darling/nix +``` + +### Important Caveats + +1. **Separate databases**: The Nix SQLite database + (`/nix/var/nix/db/db.sqlite`) must NOT be shared between the host and + Darling. Each Nix instance needs its own database. Configure the Darling + Nix instance to use separate state: + + ```ini + # In /etc/nix/nix.conf inside Darling: + store = /nix + state = /var/nix + ``` + +2. **Concurrent writes**: If both host and Darling write to `/nix/store` + simultaneously, use content-addressed store paths (which are safe for + concurrent writes) or ensure the Darling builder is the exclusive writer + for `x86_64-darwin` paths. + +3. **Permission mapping**: Darling's UID/GID namespace may differ from the + host's. Ensure files written by Darling's `_nixbldN` users are readable + by the host's Nix daemon. + +### NixOS Module + +If you're using the NixOS module, just set: + +```nix +services.darling-builder.shareStore = true; +``` + +The module handles the symlink and state directory separation automatically. + +--- + +## Verifying the Builder + +### Automated Checks + +```bash +# NixOS module: use the built-in connectivity test +darling-builder-test + +# Manual setup: use the build hook check +./scripts/darling-build-hook --check + +# Standalone Nix health-check +./scripts/verify-nix.sh +./scripts/verify-nix.sh --online # also test network/cache access + +# Run the compatibility matrix (after Nix builds work) +./tests/nix/compatibility-matrix.sh --tier 1 +``` + +### Progressive Build Tests + +The `build-trivial.sh` script tests derivation building at five increasing +levels of complexity: + +```bash +# Run all five levels +./scripts/build-trivial.sh + +# Target a specific level with debug output +./scripts/build-trivial.sh --level 1 --debug + +# Levels: +# 1. Echo to $out β€” minimal: sandbox-exec β†’ bash β†’ file creation +# 2. Multi-line builder β€” mkdir, chmod, loops, multiple output files +# 3. Input transformation β€” builtins.toFile, sort, wc +# 4. Derivation dependency β€” one derivation consumes another's output +# 5. Binary substitution β€” fetch pre-built package from cache.nixos.org +``` + +### NixOS VM Tests + +Run the full test suite without needing a live Darling instance: + +```bash +# Smoke test (no network, fast) +nix build .#checks.x86_64-linux.darling-smoke -L + +# Full Nix integration test (needs network) +nix build .#checks.x86_64-linux.nix-in-darling -L + +# Remote builder test (sshd, SSH auth, service lifecycle) +nix build .#checks.x86_64-linux.darling-builder -L + +# Directory Services stubs (pure shell, no Darling needed) +nix build .#checks.x86_64-linux.dirserv-stubs -L + +# Run everything +nix flake check +``` + +--- + +## Alternative: Custom Build Hook (No SSH) + +If you don't want to run sshd inside Darling, you can use the custom build +hook. This invokes `darling shell` directly instead of going through SSH. + +### Setup + +```bash +# Verify the hook environment is ready +./scripts/darling-build-hook --check + +# Build a single derivation directly +./scripts/darling-build-hook --build /nix/store/...-foo.drv + +# Print the machine spec line +./scripts/darling-build-hook --machine-spec +``` + +### Configure Nix to Use the Hook + +In `nix.conf`: + +```ini +builders = /path/to/darling-build-hook x86_64-darwin - 4 1 - - - +``` + +Or on NixOS: + +```nix +nix.settings.builders = [ + "/path/to/darling-build-hook x86_64-darwin - 4 1 - - -" +]; +``` + +### Environment Variables + +| Variable | Default | Description | +|----------|---------|-------------| +| `DARLING_BUILD_HOOK_DARLING` | `darling` | Path to the darling binary | +| `DARLING_BUILD_HOOK_PREFIX` | auto | Darling prefix path | +| `DARLING_BUILD_HOOK_NIX_PROFILE` | `/Users/root/.nix-profile/etc/profile.d/nix.sh` | Nix profile to source inside Darling | +| `DARLING_BUILD_HOOK_MAX_JOBS` | `4` | Maximum concurrent jobs | +| `DARLING_BUILD_HOOK_VERBOSE` | `0` | Verbosity level (0=quiet, 1=debug) | +| `DPREFIX` | auto | Fallback for prefix path | + +--- + +## Performance Tuning + +### Binary Substitution + +The single most important performance optimization is to use binary +substitution aggressively. Most `x86_64-darwin` packages in Nixpkgs are +already built by Hydra and available from `cache.nixos.org`. + +```ini +# In /etc/nix/nix.conf inside Darling: +substituters = https://cache.nixos.org +trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= +``` + +On the host: + +```ini +# In /etc/nix/nix.conf on the host: +builders-use-substitutes = true +``` + +This tells Nix to let the builder download substitutes directly from the +cache, instead of building everything from source. + +### Job Parallelism + +```nix +services.darling-builder = { + maxJobs = 4; # concurrent derivations +}; +``` + +Also limit per-build parallelism to avoid oversubscription: + +```ini +# In /etc/nix/nix.conf inside Darling: +cores = 4 +``` + +A good rule of thumb: `maxJobs Γ— cores ≀ number of CPU cores`. + +### Store Sharing + +If you're building many packages, enable store sharing to eliminate the +SSH copy overhead: + +```nix +services.darling-builder.shareStore = true; +``` + +This makes build outputs instantly available on the host without any +network transfer. + +### Storage + +Put the Darling prefix and Nix store on fast storage (SSD/NVMe). Nix builds +are I/O-heavy, and spinning disks will be a significant bottleneck. + +### Speed Factor + +If you have native Darwin builders (e.g., a Mac mini), set the Darling +builder's speed factor lower so Nix prefers the native machine: + +```nix +services.darling-builder.speedFactor = 1; +# Native builder would be speedFactor = 10 or higher +``` + +--- + +## Troubleshooting + +### "Connection refused" when connecting to the builder + +**Cause**: sshd is not running inside the Darling prefix, or it's listening +on a different port. + +```bash +# Check if sshd is listening +ss -tlnp | grep 2222 + +# Check if the Darling prefix is running +darling shell echo ok + +# Restart sshd inside Darling +darling shell /usr/sbin/sshd -f /etc/ssh/sshd_config + +# Check sshd logs +darling shell cat /var/log/sshd.log 2>/dev/null +``` + +### "Permission denied (publickey)" + +**Cause**: SSH key mismatch between the host and the Darling prefix. + +```bash +# Verify the key exists +ls -la /etc/nix/darling-builder-key + +# Verify the public key is installed in Darling +darling shell cat /var/root/.ssh/authorized_keys + +# Regenerate keys +sudo ssh-keygen -t ed25519 -N "" -f /etc/nix/darling-builder-key +sudo cat /etc/nix/darling-builder-key.pub | darling shell tee /var/root/.ssh/authorized_keys + +# Test manually +ssh -vvv -i /etc/nix/darling-builder-key -p 2222 root@127.0.0.1 echo ok +``` + +### "builder for '...' failed with exit code 1" + +**Cause**: The derivation build itself failed. Check the build log. + +```bash +# Get the full build log +nix log /nix/store/...-failed.drv + +# Or build with verbose output +nix build ... -L --keep-failed + +# Check for unimplemented syscalls +darling shell bash -lc 'nix-build ...' 2>&1 | grep -i "unimplemented\|STUB" +``` + +### "Unimplemented syscall" errors + +**Cause**: The derivation uses a macOS syscall that Darling doesn't implement yet. + +```bash +# Run the syscall triage tool to identify the issue +./scripts/triage-syscalls.sh --output /tmp/triage.md + +# Check the known triage table +cat plan/syscall-triage.md +``` + +If you discover a new unimplemented syscall, please +[file an issue](https://github.com/darlinghq/darling/issues) with: + +1. The exact syscall name and number +2. The derivation that triggered it +3. The full error message + +### "Store path not valid" / database errors + +**Cause**: When using shared `/nix/store`, the host and Darling Nix instances +may have different database states. + +```bash +# Verify the store inside Darling +darling shell bash -lc 'nix-store --verify --check-contents' + +# Re-register a missing path +darling shell bash -lc 'nix-store --register-validity <<< "..."' + +# If all else fails, repair the store +darling shell bash -lc 'nix-store --verify --repair' +``` + +### "sandbox-exec: not found" or sandbox errors + +**Cause**: The sandbox-exec stub is not installed in the prefix. + +```bash +# Check if sandbox-exec is present +darling shell test -x /usr/bin/sandbox-exec && echo ok || echo missing + +# Rebuild Darling with sandbox stubs +nix build .#darling +``` + +### Darling prefix won't start / crashes + +```bash +# Check if darlingserver is running +pgrep darlingserver + +# Try shutting down and re-initialising +darling shutdown +darling shell echo ok + +# Check system requirements +sysctl kernel.unprivileged_userns_clone # must be 1 +``` + +### Build is extremely slow + +See [Performance Tuning](#performance-tuning) above. The most common causes are: + +1. **No binary substitution**: The builder is compiling everything from source. + Make sure `substituters` and `builders-use-substitutes` are configured. +2. **No store sharing**: Large closures are being copied over SSH. Enable + `shareStore = true`. +3. **Slow storage**: The Nix store is on a spinning disk. Move it to SSD. +4. **Oversubscription**: Too many concurrent jobs for available CPU cores. + +--- + +## Security Considerations + +- **sshd inside Darling**: The SSH server only listens on `127.0.0.1` (loopback), + so it's not accessible from the network. Only key-based authentication is + allowed β€” no passwords. + +- **Root inside Darling**: The builder runs as root inside the Darling prefix, + but this is a *virtual* root. Darling uses Linux user namespaces, so the + "root" inside Darling maps to an unprivileged user on the host. + +- **Store integrity**: When sharing `/nix/store`, Nix's content-addressed paths + provide integrity guarantees β€” a path's name includes a hash of its contents, + so tampering is detectable. + +- **Network access**: Derivations built inside Darling can access the network + unless the Nix sandbox restricts it. Fixed-output derivations (fetchers) + need network access by design; regular builds should not. + +- **Upstream trust**: This is an experimental project. It should not be used as + a trusted builder for production deployments without thorough security review. + +--- + +## Architecture + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Linux Host (NixOS) β”‚ +β”‚ β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” SSH (localhost:2222) β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ Nix Daemon β”‚ ──────────────────────── β”‚ sshd β”‚ β”‚ +β”‚ β”‚ β”‚ or build-hook pipe β”‚ (Darling)β”‚ β”‚ +β”‚ β”‚ Offloads β”‚ β”‚ β”‚ β”‚ +β”‚ β”‚ x86_64- β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β” β”‚ β”‚ +β”‚ β”‚ darwin β”‚ β”‚ β”‚ Nix β”‚ β”‚ β”‚ +β”‚ β”‚ builds β”‚ β”‚ β”‚daemonβ”‚ β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β””β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ β”‚ +β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ β”‚ β”‚ +β”‚ β–Ό β”‚ /Volumes/ β”‚ β”‚ β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ SystemRoot/nix ──▢│ /nix β”‚ β”‚ +β”‚ β”‚ /nix/store β”‚β—€β”€β”€β”€β”€β”€β”€β”˜ (shared store) β”‚ (symlink)β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β”‚ Darling Prefix β”‚ +β”‚ (~/.darling or β”‚ +β”‚ /var/lib/darling-builder) β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +**Key components:** + +| Component | Role | +|-----------|------| +| **darlingserver** | Userspace daemon that translates macOS syscalls to Linux | +| **Darling prefix** | Virtual macOS filesystem tree (overlayfs-based) | +| **sandbox-exec stub** | Passes through commands without sandboxing (Darling provides Linux-level isolation) | +| **Directory Services stubs** | `dscl`, `dseditgroup`, `sysadminctl` β€” translate macOS user/group commands to `/etc/passwd` + `/etc/group` | +| **diskutil stub** | Returns expected filesystem info for the Nix installer | +| **darling-build-hook** | Alternative to SSH β€” invokes `darling shell` directly | +| **darlingBuilderModule.nix** | NixOS module that wires everything together declaratively | + +For more technical details, see [plan/11-architecture.md](../plan/11-architecture.md). + +--- + +## Further Reading + +- [Project plan](../plan/README.md) β€” full development plan with phases and tasks +- [Known blockers](../plan/01-blockers.md) β€” detailed analysis of blocking issues +- [Syscall triage](../plan/syscall-triage.md) β€” tracking table for unimplemented syscalls +- [Darling documentation](https://docs.darlinghq.org/) β€” upstream Darling docs +- [Nix remote builders](https://nixos.org/manual/nix/stable/advanced-topics/distributed-builds.html) β€” Nix manual on distributed builds +- [Blog: Nix All The Way Down](https://ersei.net/en/blog/nix-all-the-way-down) β€” early exploration of Nix-in-Darling \ No newline at end of file diff --git a/flake.nix b/flake.nix index c7ce2e3f9..02eb94a1a 100644 --- a/flake.nix +++ b/flake.nix @@ -29,6 +29,17 @@ packages.darling-sdk = pkgs: pkgs.darling.sdk; + # ── Flake Templates ────────────────────────────────────────────── + # + # Initialise a new project with: + # nix flake init -t github:nixie-dev/darling-nix#darling-builder + # + # See: docs/darwin-builder.md, plan/09-phase7-remote-builder.md (Task 7.7) + templates.darling-builder = { + path = ./templates/darling-builder; + description = "NixOS configuration with a Darling-based x86_64-darwin remote builder"; + }; + # ── NixOS Modules ──────────────────────────────────────────────── # # The base module (programs.darling) is autoloaded from diff --git a/plan/09-phase7-remote-builder.md b/plan/09-phase7-remote-builder.md index 67e8ffb52..40c179d59 100644 --- a/plan/09-phase7-remote-builder.md +++ b/plan/09-phase7-remote-builder.md @@ -521,35 +521,53 @@ results to a dashboard or markdown file in the repo. --- -### 7.7 β€” Documentation and Templates +### 7.7 β€” Documentation and Templates βœ… Create user-facing documentation so others can set up their own Darling builders. -**Deliverables**: +**Status**: βœ… Complete β€” see `docs/darwin-builder.md` and `templates/darling-builder/`. -1. **NixOS wiki page**: Step-by-step guide for setting up a Darling-based Darwin - builder on NixOS. Cover both the NixOS module approach and the manual setup. +**Deliverables**: -2. **Flake template** (`templates/darling-builder`): +1. βœ… **User-facing setup guide** (`docs/darwin-builder.md`): Comprehensive + documentation covering NixOS module quick start, manual setup (sshd, SSH + keys, builder registration), shared `/nix/store` configuration, verification + procedures (automated checks, progressive build tests, NixOS VM tests), + custom build hook (no SSH) alternative, performance tuning (binary + substitution, job parallelism, store sharing, storage, speed factor), + troubleshooting (connection refused, permission denied, unimplemented + syscalls, database errors, sandbox issues, slow builds), security + considerations, and architecture diagram with component table. + +2. βœ… **Flake template** (`templates/darling-builder/`): ```bash - nix flake init -t github:user/darling-nix#darling-builder + nix flake init -t github:nixie-dev/darling-nix#darling-builder ``` - Generates a minimal `flake.nix` + NixOS configuration that sets up the - builder. - -3. **Troubleshooting guide**: Common issues and their solutions: + Generates a `flake.nix` with a ready-to-use NixOS configuration that + imports both the base Darling module and the builder module, with all + options documented inline. Includes its own `README.md` with getting + started steps, options reference table, architecture diagram, and + troubleshooting section. Wired into `flake.nix` as + `templates.darling-builder`. + +3. βœ… **Troubleshooting guide** (in `docs/darwin-builder.md`): Covers all + planned scenarios: - "Connection refused" β†’ sshd not running or wrong port - "Permission denied" β†’ SSH key mismatch - "Build failed with signal 11" β†’ unimplemented syscall β†’ file an issue - "Store path not valid" β†’ shared store database mismatch - "builder for '...' failed with exit code 1" β†’ check the build log + - "sandbox-exec: not found" β†’ sandbox stub missing + - Darling prefix crashes β†’ darlingserver / kernel requirements + - Slow builds β†’ substitution, store sharing, storage, oversubscription -4. **Performance tuning guide**: Tips for getting the best performance: +4. βœ… **Performance tuning guide** (in `docs/darwin-builder.md`): Covers: - Use binary substitution aggressively (`substituters` in `nix.conf`) - Set `max-jobs` based on available CPU cores - Use `--cores N` to limit per-build parallelism - Enable store sharing to avoid copy overhead - Put the Nix store on fast storage (SSD/NVMe) + - Speed factor configuration for multi-builder setups --- diff --git a/plan/README.md b/plan/README.md index 6731c1325..10f797e2b 100644 --- a/plan/README.md +++ b/plan/README.md @@ -75,6 +75,8 @@ | `tests/syscall/test_renameatx_np.c` | renameatx_np regression tests (plain rename, SWAP, EXCL, invalid flags) | | `tests/syscall/test_setattrlist_flags.c` | setattrlist/getattrlist ATTR_CMN_FLAGS tests | | `tests/syscall/test_utimensat.c` | utimensat/setattrlistat timestamp handling tests | +| `docs/darwin-builder.md` | User-facing setup guide β€” NixOS module, manual setup, shared store, troubleshooting, perf tuning (Phase 7.7) | +| `templates/darling-builder/` | Flake template β€” `nix flake init -t .#darling-builder` generates a ready-to-use NixOS config (Phase 7.7) | ## References diff --git a/templates/darling-builder/README.md b/templates/darling-builder/README.md new file mode 100644 index 000000000..af0aedf0a --- /dev/null +++ b/templates/darling-builder/README.md @@ -0,0 +1,151 @@ +# Darling Builder Template + +This template sets up a NixOS configuration with a **Darling-based +`x86_64-darwin` remote builder**, allowing your Linux machine to build macOS +packages without Apple hardware. + +## Getting Started + +### 1. Initialise from the template + +```bash +mkdir my-darwin-builder && cd my-darwin-builder +nix flake init -t github:nixie-dev/darling-nix#darling-builder +``` + +### 2. Customise `flake.nix` + +Open `flake.nix` and: + +- Replace `"myhost"` with your machine's hostname. +- Uncomment and add your existing NixOS modules (`hardware-configuration.nix`, + `configuration.nix`, etc.). +- Adjust `services.darling-builder` options to taste (see + [Options](#module-options) below). + +### 3. Apply the configuration + +```bash +sudo nixos-rebuild switch --flake .#myhost +``` + +This will: + +1. Install Darling on the host. +2. Create and initialise a Darling prefix at `/var/lib/darling-builder`. +3. Install Nix inside the Darling prefix. +4. Start an SSH server inside the prefix on `127.0.0.1:2222`. +5. Register the Darling instance as a `nix.buildMachines` entry for + `x86_64-darwin`. + +### 4. Verify + +```bash +# Built-in connectivity check +darling-builder-test + +# Build a Darwin package from your Linux host +nix build nixpkgs#hello --system x86_64-darwin +``` + +## Module Options + +| Option | Type | Default | Description | +|--------|------|---------|-------------| +| `enable` | bool | `false` | Enable the Darling builder service | +| `package` | package | `pkgs.darling` | Darling package to use | +| `port` | int | `2222` | SSH port inside the Darling prefix | +| `maxJobs` | int | `1` | Maximum concurrent build jobs | +| `speedFactor` | int | `1` | Nix builder speed factor (lower = deprioritised) | +| `shareStore` | bool | `false` | Share `/nix/store` between host and Darling via `/Volumes/SystemRoot` | +| `sshKeyPath` | path | `/etc/nix/darling-builder-key` | SSH private key for the Nix daemon | +| `prefixPath` | path | `/var/lib/darling-builder` | Darling prefix directory | +| `supportedFeatures` | list of str | `[]` | Nix supported features | +| `mandatoryFeatures` | list of str | `[]` | Nix mandatory features | +| `installNix` | bool | `true` | Auto-install Nix inside the prefix on first boot | +| `nixVersion` | str | `"2.24.10"` | Nix version to install inside Darling | + +## How It Works + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Linux Host (NixOS) β”‚ +β”‚ β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” SSH (localhost:2222) β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ Nix Daemon β”‚ ──────────────────────▢ β”‚ sshd β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ (Darling) β”‚ β”‚ +β”‚ β”‚ offloads β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ +β”‚ β”‚ x86_64- β”‚ β”‚ β”‚ Nix β”‚ β”‚ β”‚ +β”‚ β”‚ darwin β”‚ β”‚ β”‚ daemon β”‚ β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ β”‚ +β”‚ β–Ό /Volumes/ β”‚ β”‚ β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” SystemRoot/nix ──────▢│ /nix β”‚ β”‚ +β”‚ β”‚/nix/store β”‚ (shared store) β”‚ (symlink) β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β”‚ Darling Prefix β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +Darling is a macOS compatibility layer that translates macOS system calls into +Linux equivalents. By running Nix inside Darling, the builder reports +`builtins.currentSystem == "x86_64-darwin"` and can execute Darwin derivations. + +When `shareStore` is enabled, the host's `/nix/store` is made available inside +the Darling prefix via `/Volumes/SystemRoot`, eliminating the need to copy +store paths over SSH. + +## Troubleshooting + +### "Connection refused" + +The sshd inside Darling isn't running or is on a different port. + +```bash +# Check if the service is running +systemctl status darling-builder + +# Check if sshd is listening +ss -tlnp | grep 2222 + +# Restart the service +sudo systemctl restart darling-builder +``` + +### "Permission denied (publickey)" + +SSH key mismatch between the host and Darling prefix. + +```bash +# Verify the key exists +ls -la /etc/nix/darling-builder-key + +# Verify the public key inside Darling +darling --prefix /var/lib/darling-builder shell cat /var/root/.ssh/authorized_keys + +# Test SSH manually +ssh -vvv -i /etc/nix/darling-builder-key -p 2222 root@127.0.0.1 echo ok +``` + +### Build fails with "Unimplemented syscall" + +The derivation uses a macOS syscall that Darling doesn't support yet. This is +expected for complex packages. Check the +[syscall triage table](https://github.com/nixie-dev/darling-nix/blob/main/plan/syscall-triage.md) +and consider filing an issue upstream. + +### Build is very slow + +1. **Enable binary substitution** β€” most `x86_64-darwin` packages are already + on `cache.nixos.org`. Ensure `builders-use-substitutes = true` is set in + the host's `nix.conf`. +2. **Enable store sharing** β€” set `shareStore = true` to avoid copying store + paths over SSH. +3. **Use fast storage** β€” put the Darling prefix on SSD/NVMe. + +## Further Reading + +- [Full setup guide](https://github.com/nixie-dev/darling-nix/blob/main/docs/darwin-builder.md) +- [Project plan](https://github.com/nixie-dev/darling-nix/blob/main/plan/README.md) +- [Darling documentation](https://docs.darlinghq.org/) +- [Nix distributed builds](https://nixos.org/manual/nix/stable/advanced-topics/distributed-builds.html) \ No newline at end of file diff --git a/templates/darling-builder/flake.nix b/templates/darling-builder/flake.nix new file mode 100644 index 000000000..446ba02ba --- /dev/null +++ b/templates/darling-builder/flake.nix @@ -0,0 +1,87 @@ +{ + description = "NixOS configuration with a Darling-based x86_64-darwin builder"; + + inputs = { + nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable"; + + darling-nix = { + url = "github:nixie-dev/darling-nix"; + inputs.nixpkgs.follows = "nixpkgs"; + }; + }; + + outputs = + { + nixpkgs, + darling-nix, + ... + }: + { + # ── NixOS system configuration ─────────────────────────────────── + # + # Replace "myhost" with your machine's hostname. + # Adjust the module list and options to fit your setup. + # + # Build with: + # sudo nixos-rebuild switch --flake .#myhost + # + nixosConfigurations.myhost = nixpkgs.lib.nixosSystem { + system = "x86_64-linux"; + modules = [ + # Your existing hardware and system configuration: + # ./hardware-configuration.nix + # ./configuration.nix + + # Base Darling support β€” provides `programs.darling` + darling-nix.nixosModules.nixos + + # Darling builder service β€” provides `services.darling-builder` + darling-nix.nixosModules.darling-builder + + # ── Builder settings ──────────────────────────────────────── + { + services.darling-builder = { + # Enable the builder service. After `nixos-rebuild switch`, + # a systemd service will: + # 1. Initialise a Darling prefix + # 2. Install Nix inside the prefix + # 3. Start sshd so the host Nix daemon can connect + # 4. Register as a `nix.buildMachines` entry + enable = true; + + # Maximum number of concurrent build jobs. + # A safe default is half your CPU core count. + maxJobs = 4; + + # Share /nix/store between the host and the Darling prefix + # via /Volumes/SystemRoot. This avoids copying store paths + # over SSH and makes build results instantly available. + # Disable this if you encounter permission or database issues. + shareStore = true; + + # SSH port for the builder (inside the Darling prefix). + # Uses 2222 by default to avoid conflicting with the host's sshd. + # port = 2222; + + # Nix speed factor β€” lower means Nix will prefer other builders + # when available. Set higher if this is your only Darwin builder. + # speedFactor = 1; + + # Path to the SSH private key used by the Nix daemon to connect + # to the Darling builder. Generated automatically on first boot. + # sshKeyPath = "/etc/nix/darling-builder-key"; + + # Path to the Darling prefix directory. + # prefixPath = "/var/lib/darling-builder"; + + # Automatically install Nix inside the Darling prefix on first boot. + # installNix = true; + + # Nix version to install inside Darling. + # nixVersion = "2.24.10"; + }; + } + ]; + }; + }; +}