diff --git a/openspec/changes/add-nix-devshell/.openspec.yaml b/openspec/changes/add-nix-devshell/.openspec.yaml new file mode 100644 index 0000000..5849c2d --- /dev/null +++ b/openspec/changes/add-nix-devshell/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-08-01 diff --git a/openspec/changes/add-nix-devshell/design.md b/openspec/changes/add-nix-devshell/design.md new file mode 100644 index 0000000..e828969 --- /dev/null +++ b/openspec/changes/add-nix-devshell/design.md @@ -0,0 +1,113 @@ +## Context + +Trawler pins Rust 1.94.0 in `rust-toolchain.toml`. Windows CI builds and tests the whole workspace; Tangled's Linux CI intentionally covers only `trawler-core`. On the target NixOS development host, an ephemeral `nix shell` can compile `trawler-core`, but linking the GPUI app fails because XCB/xkbcommon native libraries are not presented through a development-shell linker environment. The host has no `DISPLAY` or Wayland session, so launching and visually inspecting the application also requires an isolated display server. Mesa's lavapipe Vulkan ICD is available and can provide deterministic software rendering. + +Trawler already owns the application-side automation needed for this change: a feature-gated fixture seeder, a doubly gated localhost devtools server, semantic JSON state dumps, keystroke/text injection, and window screenshot capture. This change packages their Linux prerequisites and orchestrates them safely; it does not introduce a second automation protocol. + +Stakeholders are the maintainer on Windows and automated/agent developers on NixOS. The workflow must never open the maintainer's real graph. + +## Goals / Non-Goals + +**Goals:** + +- Make a clean `nix develop` shell sufficient to build and test the full workspace on x86_64 Linux. +- Launch the real GPUI application without an existing desktop session by using a virtual X11 display and software Vulkan. +- Seed and use only a deterministic disposable fixture graph. +- Drive the existing devtools JSONL protocol repeatedly in a long-lived session, capture semantic state, and produce PNG screenshots that can be inspected by an agent. +- Retain a one-shot smoke mode as the deterministic end-to-end acceptance path. +- Fail clearly on missing native capabilities, startup timeout, protocol failure, or screenshot failure. +- Clean up the application, virtual display, and temporary graph reliably. + +**Non-Goals:** + +- Declare Linux a supported end-user release target; Windows remains authoritative for product UX. +- Replace Windows or Tangled CI, or add Nix-based CI in this change. +- Change the devtools protocol, fixture content, or GPUI application behavior unless a verified Linux compatibility bug requires a separately reviewed adjustment. +- Run against, import, or mutate the user's default graph. +- Optimize GPU performance; software rendering is for deterministic development preview. +- Package or install Trawler as a Linux desktop application. +- Add raw mouse/pointer injection to the devtools protocol. Agent automation uses the existing keyboard-first semantic surface; pointer-only flows remain a native-desktop/manual check or a later devtools change. + +## Decisions + +### D1 — A pinned flake supplies a real development environment + +Add `flake.nix` and `flake.lock` with one x86_64-linux development shell. The shell uses a pinned nixpkgs input plus a Rust toolchain overlay capable of consuming `rust-toolchain.toml`, so the Rust version remains defined once rather than duplicated in shell scripts. Native build/runtime inputs include the compiler/linker and pkg-config surface required by GPUI, XCB/X11, xkbcommon, Wayland client libraries, fontconfig/freetype, Vulkan loader/Mesa software drivers, Xvfb, and small protocol-inspection tools. + +`mkShell` is required rather than an ad-hoc `nix shell`: its setup hooks populate `NIX_LDFLAGS`, `PKG_CONFIG_PATH`, and runtime library paths. The current linker failure demonstrated that merely placing library packages on `PATH` is insufficient. + +Alternatives considered: + +- **rustup in the shell:** matches CI and avoids an overlay, but performs a network-managed toolchain install outside the flake closure and weakens reproducibility. +- **A full Nix package derivation now:** useful for distribution, but unnecessary for a development shell and would combine packaging with the preview problem. +- **flake-utils:** avoided for one supported system; plain `nixpkgs.lib` keeps the flake smaller. + +The implementation spike SHALL confirm the exact current nixpkgs attribute names and library hooks rather than copying obsolete Xorg package names. + +### D2 — Xvfb plus lavapipe provides an isolated graphical session + +The preview helper launches a private Xvfb server with a fixed screen size/depth and points the application at it through `DISPLAY`. It selects Mesa's lavapipe ICD explicitly for Vulkan software rendering, so the workflow does not require an attached monitor, access to another user's X authority, or the physical GPU. + +Alternatives considered: + +- **Borrow the host desktop:** unavailable to the service user and creates focus/security coupling. +- **Headless Wayland/Weston:** viable, but screenshot behavior is compositor-dependent and the repository already documents X11 capture as the solid Linux path. +- **Only run GPUI's fake-platform tests:** valuable but insufficient; the requested capability includes launching and visually previewing the real application. + +### D3 — The helper owns both smoke and long-lived preview lifecycles + +A repository helper exposes two workflows over one lifecycle implementation: + +- **Smoke:** start a clean session, execute a deterministic multi-command fixture flow, save dump/log/screenshot artifacts, then stop and clean up. +- **Serve:** start a clean session and leave it running until explicit `stop` or interruption, allowing an agent to interleave reasoning with any number of `send`, `dump`, `bounds`, and `screenshot` operations. + +Session start performs the following: + +1. Create a unique temporary workspace and register cleanup traps. +2. Build Trawler with `--features devtools`. +3. Seed the deterministic fixture graph into the temporary workspace; refuse an existing/non-owned graph path. +4. Start Xvfb and launch Trawler with both `TRAWLER_DEVTOOLS=1` and `TRAWLER_GRAPH_DIR` set. +5. Wait with a bounded timeout for `devtools.port`, checking that child processes remain alive. +6. Write a session manifest containing the owned scratch root, graph path, display, app/Xvfb PIDs, log, and devtools port. + +While a serve session is live, helper subcommands read and validate that manifest before issuing JSONL requests or taking screenshots. `status` distinguishes a healthy session from stale metadata. `stop` terminates child processes, removes the manifest and disposable graph state, and is idempotent. The exact CLI spelling may follow repository conventions, but start/status/send/screenshot/stop behaviors are part of the contract. + +The smoke command uses the same session controls rather than a second code path. Navigation steps that settle asynchronously are sent as separate requests with bounded dump polling, matching the existing dev-loop guidance. + +### D4 — Trawler's screenshot command is the acceptance path + +The primary visual artifact must be produced by the devtools `screenshot` command, exercising the same feature the repository documents. An external X11 root-window capture may be used diagnostically, but it does not by itself satisfy the requirement. + +If xcap cannot capture under Xvfb, implementation pauses to determine whether a small application-side compatibility fix, a different Xvfb invocation, or a requirements change is appropriate. The helper must not silently claim visual success from semantic dumps alone. + +### D5 — Safety is structural, not a warning + +The helper creates its own graph directory and does not accept the default graph path. It always sets `TRAWLER_GRAPH_DIR`, verifies the fixture location is inside its owned temporary root, and refuses unsafe overrides. Cleanup is trap-based and idempotent. The user's graph is therefore unreachable by construction during preview. + +### D6 — Automated preview is isolated from native desktop use + +The Nix devshell does not install or replace Trawler globally. Automated sessions always use a private Xvfb display, lavapipe, a worktree-built binary, and a scratch graph, so they can coexist with a real Linux desktop installation using the user's display, hardware renderer, installed binary, and default graph. + +Documentation also gives an explicit manual native-display development command for a developer running from an interactive graphical terminal, still with a scratch `TRAWLER_GRAPH_DIR`. That path is useful for human pointer checks but is not the deterministic agent acceptance path: the always-on agent service has no desktop credentials, Wayland capture varies by compositor, and the current devtools protocol has no raw click command. + +## Risks / Trade-offs + +- [GPUI/blade cannot initialize Vulkan against Xvfb + lavapipe] → Pin the ICD explicitly, capture startup logs, and spike this before polishing helper UX; evaluate a headless Weston path only with evidence. +- [xcap cannot capture its own window under Xvfb] → Treat screenshot as a blocking acceptance failure; use external X11 capture only for diagnosis and bring any fallback decision back for review. +- [Nixpkgs native dependency names/hooks drift] → Pin `flake.lock`, use `pkg-config` checks in the shell verification, and document update procedure. +- [Rust overlay duplicates `rust-toolchain.toml` semantics imperfectly] → Use the overlay's rustup-toolchain-file parser and verify `rustc --version` plus installed targets in acceptance tests. +- [Preview helper leaks Xvfb/app processes after interruption] → Track PIDs, install EXIT/INT/TERM traps, use bounded waits, and test a forced-failure cleanup path. +- [A long-lived session leaves stale metadata after a hard kill] → Validate PIDs/display/port on every command, make `status` identify stale sessions, and let `stop` clean stale owned state safely. +- [A requested UI path is pointer-only] → Prefer Trawler's keyboard equivalent; document the protocol limitation and use human native-desktop verification rather than brittle coordinate automation in this change. +- [Adding broad Linux dependencies implies product support] → Documentation explicitly calls this a development/automation environment; Windows remains the release target and authoritative visual check. + +## Migration Plan + +This is additive. Developers opt in with `nix develop`; existing Cargo and Windows workflows remain unchanged. Rollback is removal of the flake, helper, and documentation. No graph format, application state, or user data migration exists. + +Implementation order is intentionally risk-first: prove full-workspace linking, then real-window launch, then devtools dump, then screenshot, and only afterward polish documentation and lifecycle behavior. + +## Open Questions + +1. Which current nixpkgs Mesa output exposes the lavapipe ICD most portably inside the flake shell: a store-pinned ICD path or the NixOS `/run/opengl-driver` path? Prefer the store path if available; prove it during the spike. +2. Should a later change add the Nix full-workspace check to Tangled CI, or keep CI split Windows-app/Linux-core as it is today? diff --git a/openspec/changes/add-nix-devshell/proposal.md b/openspec/changes/add-nix-devshell/proposal.md new file mode 100644 index 0000000..14bd73b --- /dev/null +++ b/openspec/changes/add-nix-devshell/proposal.md @@ -0,0 +1,29 @@ +## Why + +Trawler's core tests run on Linux, but the GPUI application and headless UI tests cannot currently be built or launched from a clean NixOS environment without manually discovering and wiring native display, input, font, and Vulkan dependencies. A reproducible Nix development environment and isolated visual-preview workflow will let an agent develop and verify the real application independently without using the maintainer's Windows machine or touching the user's graph. + +## What Changes + +- Add a pinned Nix flake development shell containing the Rust and native Linux dependencies required to build and test the full workspace. +- Add a repeatable headless-preview workflow that launches Trawler's existing `devtools` build against a deterministic scratch fixture graph under an isolated virtual display and software Vulkan renderer. +- Provide both a deterministic one-shot smoke run and a long-lived session for iterative navigation, state inspection, and screenshots across many devtools requests. +- Make semantic automation (`keys`, `type`, `dump`, `bounds`) and PNG screenshot capture available throughout a preview session; document that raw pointer injection is not part of the current devtools protocol. +- Document the development, test, preview, troubleshooting, and cleanup commands, including the boundary that Windows remains the product target. +- Ensure preview tooling refuses to use the default/real graph directory and cleans up its processes and temporary state. + +## Capabilities + +### New Capabilities + +- `nix-development-environment`: Reproducible Nix-based full-workspace builds, tests, and isolated devtools visual preview on Linux/NixOS. + +### Modified Capabilities + +None. The change consumes the existing fixture-graph, UI-test-harness, and dev-automation-server contracts without changing their behavior. + +## Impact + +- New root-level Nix flake and lock file. +- New development helper(s), session metadata, and documentation for virtual-display preview lifecycle and iterative devtools requests. +- Linux development dependencies include X11/XCB, xkbcommon, font libraries, Vulkan loader/software ICD support, and virtual-display/capture tooling. +- CI and shipped release builds remain unchanged unless a later decision explicitly adopts the Nix checks in automation. diff --git a/openspec/changes/add-nix-devshell/specs/nix-development-environment/spec.md b/openspec/changes/add-nix-devshell/specs/nix-development-environment/spec.md new file mode 100644 index 0000000..6cdd3e7 --- /dev/null +++ b/openspec/changes/add-nix-devshell/specs/nix-development-environment/spec.md @@ -0,0 +1,108 @@ +## ADDED Requirements + +### Requirement: Reproducible full-workspace development shell +The repository SHALL provide a pinned Nix flake development shell for x86_64 Linux containing the Rust toolchain and native dependencies required to compile, link, and test both `trawler-core` and the GPUI `trawler` application. + +#### Scenario: Clean shell builds the workspace +- **WHEN** a developer enters the shell from a clean checkout with `nix develop` +- **THEN** the Rust version matches `rust-toolchain.toml` +- **AND** `cargo check --workspace --all-targets` completes without missing native-library or pkg-config errors + +#### Scenario: Full tests link in the shell +- **WHEN** a developer runs `cargo test --workspace` inside the shell +- **THEN** the GPUI test binary links with its Linux XCB, xkbcommon, font, and graphics dependencies +- **AND** the non-ignored workspace tests execute + +### Requirement: Isolated real-application preview +The repository SHALL provide smoke and long-lived session commands that launch the real GPUI Trawler application with the `devtools` feature on a private virtual display and software Vulkan renderer without requiring an existing graphical login session. + +#### Scenario: Preview starts on a headless host +- **WHEN** the preview command runs with no pre-existing `DISPLAY` or Wayland session +- **THEN** it starts an isolated virtual display +- **AND** launches a `--features devtools` Trawler binary with `TRAWLER_DEVTOOLS=1` +- **AND** discovers the localhost JSONL endpoint through the fixture graph's `devtools.port` + +#### Scenario: Startup failure is bounded and diagnosable +- **WHEN** the virtual display, renderer, application, or devtools endpoint fails to start +- **THEN** the command exits non-zero within a bounded timeout +- **AND** reports which stage failed and preserves or prints the relevant application log location + +### Requirement: Long-lived iterative preview session +The preview workflow SHALL support keeping one isolated application session alive across multiple agent operations until explicit stop or interruption. + +#### Scenario: Agent navigates incrementally +- **WHEN** an agent starts a serve session and sends multiple `keys`, `type`, `dump`, `bounds`, or `screenshot` requests over time +- **THEN** every request targets the same running Trawler process and fixture graph +- **AND** state from earlier requests remains visible to later requests + +#### Scenario: Session can be rediscovered +- **WHEN** a serve session is healthy and a later command runs from the repository +- **THEN** the command discovers validated session metadata including the graph, display, process, log, and devtools endpoint +- **AND** `status` reports the session as live without starting another application + +#### Scenario: Session stops explicitly +- **WHEN** the agent requests stop +- **THEN** the application and virtual display terminate +- **AND** disposable session state and metadata are removed while requested artifacts remain + +### Requirement: Deterministic semantic and visual verification +The preview workflow SHALL drive Trawler through its existing devtools protocol and produce both machine-readable semantic state and a PNG captured by the devtools screenshot command. + +#### Scenario: Semantic state is observable +- **WHEN** the preview client sends valid `keys`, `type`, and `dump` requests +- **THEN** each request receives one valid JSON response in order +- **AND** the dump reflects the resulting fixture-graph UI state + +#### Scenario: Screenshot is previewable +- **WHEN** the preview client sends a `screenshot` request with an output path +- **THEN** the application returns a successful response +- **AND** the output is a non-empty, valid PNG containing the Trawler window + +#### Scenario: One-shot smoke uses the session lifecycle +- **WHEN** the smoke command runs +- **THEN** it starts one isolated session, executes a multi-step fixture flow, saves semantic/log/PNG artifacts, and stops through the same lifecycle controls used by serve mode + +### Requirement: Preview cannot touch the user's graph +The preview command MUST create and use a deterministic disposable fixture graph under a temporary directory it owns, and MUST NOT infer or use Trawler's default graph directory. + +#### Scenario: Preview uses disposable fixtures +- **WHEN** a preview session starts +- **THEN** it seeds a fresh graph with Trawler's fixture builder +- **AND** sets `TRAWLER_GRAPH_DIR` explicitly to that graph before application startup + +#### Scenario: Unsafe graph override is refused +- **WHEN** a requested graph path is outside the helper-owned scratch root, already exists unexpectedly, or resolves to the default graph location +- **THEN** the helper exits before launching Trawler +- **AND** does not modify that graph path + +#### Scenario: Real desktop installation remains isolated +- **WHEN** a real Linux desktop installation and an automated preview exist at the same time +- **THEN** the preview uses a different display, renderer, binary, graph directory, and process lifecycle +- **AND** it does not replace global packages or acquire the real graph's lock + +### Requirement: Preview lifecycle is self-cleaning +The preview workflow SHALL clean up application and virtual-display processes and disposable graph state after smoke completion, explicit serve stop, failure, or interruption while preserving explicitly requested output artifacts. + +#### Scenario: Successful run cleans up +- **WHEN** semantic and screenshot verification complete +- **THEN** the application and virtual display are stopped +- **AND** the temporary fixture graph is removed +- **AND** requested dump, log, and screenshot artifacts remain available + +#### Scenario: Interrupted run cleans up +- **WHEN** the preview command receives an interrupt or one child process exits unexpectedly +- **THEN** remaining child processes are terminated +- **AND** cleanup is safe to run more than once + +#### Scenario: Stale session is diagnosable +- **WHEN** session metadata remains after a hard process exit +- **THEN** `status` reports it as stale rather than live +- **AND** `stop` safely removes remaining owned processes and state + +### Requirement: Nix preview workflow is documented +The repository SHALL document how to enter the shell, run the full tests, use smoke and long-lived preview sessions, inspect semantic and PNG outputs, launch manually on a real Linux desktop with a scratch graph, troubleshoot native graphics startup, and clean up, while stating that Windows remains Trawler's product target and raw pointer injection is not available through devtools. + +#### Scenario: New developer follows documented path +- **WHEN** a developer follows the documentation from a clean NixOS checkout +- **THEN** every required command and expected artifact path is stated explicitly +- **AND** no step requires operating on the user's real graph or installing packages globally diff --git a/openspec/changes/add-nix-devshell/tasks.md b/openspec/changes/add-nix-devshell/tasks.md new file mode 100644 index 0000000..e29a43d --- /dev/null +++ b/openspec/changes/add-nix-devshell/tasks.md @@ -0,0 +1,30 @@ +## 1. Reproducible Development Shell + +- [ ] 1.1 Add `flake.nix` and `flake.lock` with the Rust toolchain derived from `rust-toolchain.toml` and the native GPUI Linux build/runtime dependencies for x86_64-linux. +- [ ] 1.2 Add shell-time validation or documentation for the selected Vulkan ICD and native pkg-config/linker paths without relying on globally installed packages. +- [ ] 1.3 Verify from `nix develop` that `cargo check --workspace --all-targets` completes and that `cargo test --workspace` links and runs the non-ignored core and GPUI fake-platform tests. + +## 2. Headless Devtools Preview + +- [ ] 2.1 Add a small standard-library JSONL devtools client that sends ordered requests, validates one JSON response per request, and reports protocol errors clearly. +- [ ] 2.2 Add a shared preview lifecycle that creates a unique temporary root, builds the devtools binary, seeds a fresh fixture graph, and refuses paths outside its owned scratch area. +- [ ] 2.3 Launch a private Xvfb display with lavapipe selected explicitly, start Trawler with `TRAWLER_DEVTOOLS=1` and the scratch `TRAWLER_GRAPH_DIR`, and discover `devtools.port` with a bounded health-checked wait. +- [ ] 2.4 Persist and validate a session manifest containing the owned paths, display, child PIDs, log, and devtools endpoint; implement live/stale `status` reporting. +- [ ] 2.5 Implement long-lived start/send/screenshot/stop behavior so iterative commands target one application and fixture graph until explicit cleanup. +- [ ] 2.6 Implement a one-shot smoke flow on the same lifecycle that navigates the fixture with separate `keys`/`type` requests, polls `dump` for settled state, and saves semantic JSON. +- [ ] 2.7 Capture the Trawler window through the devtools `screenshot` command, validate that the artifact is a non-empty PNG, and visually inspect it. + +## 3. Safety and Lifecycle Hardening + +- [ ] 3.1 Add idempotent EXIT/INT/TERM cleanup that terminates Trawler and Xvfb and removes disposable graph state while preserving requested dump, log, and screenshot outputs. +- [ ] 3.2 Add bounded failure checks for missing display/renderer startup, application exit, missing `devtools.port`, malformed protocol responses, and screenshot failure. +- [ ] 3.3 Exercise at least one forced-failure/interruption path and verify no preview processes remain. +- [ ] 3.4 Exercise stale-session detection and cleanup after a hard child-process exit. +- [ ] 3.5 Verify structurally that the helper always sets `TRAWLER_GRAPH_DIR` to its owned fixture path and cannot open the default/user graph. + +## 4. Documentation and Final Verification + +- [ ] 4.1 Document `nix develop`, full-workspace checks/tests, smoke and serve session commands, generated artifact paths, native-desktop scratch launch, the no-pointer-injection limitation, troubleshooting, cleanup behavior, and the Windows-product/Linux-development boundary. +- [ ] 4.2 Run formatting and lint checks plus the full non-ignored workspace test suite from the devshell. +- [ ] 4.3 Run both the documented long-lived iterative flow and one-shot smoke flow from clean temporary state and retain semantic dumps and screenshots as verification evidence. +- [ ] 4.4 Verify implementation against the `nix-development-environment` requirements and resolve or document every deviation before archive.