Coordinator on folknet #

A design for moving the spec-watch coordinator off the development box onto folknet: the scheduled tick (review, writer launches, landing through bench/land.sh), the primary checkout it lands into, and the scripts it calls. Afterwards the development box is only where a person connects from. It serves the standing rule that heavy work runs on folknet hardware (the roster handoff.md on main). Nothing here is built or moved yet. Hosts are named by role; real addresses stay in $HOME/rig-notes/rig-access.md.

In one paragraph. Create one new small unprivileged LXC, the coordinator guest, with the existing lane scripts (bench/lane-create-lxc.sh, bench/lane-provision.sh). Clone the primary checkout there, fetching from the rig forge's pull mirror. Recreate the gitignored files by hand with rig-network addresses and no jump host. Install the same cron entry with spec-watch-schedule.sh. Only one coordinator can tick because the development box's cron entry is removed and its primary checkout renamed, so a leftover entry or a hand-run tick fails at the tick script's first check. The tick, pi-tracks, bench/land.sh and the lane scripts need no code change. Pushing to the public forge stays a person's step at first; an automatic push with a deploy key is a later, separate decision.

Today and after #

flowchart LR
  subgraph today["today"]
    direction LR
    p1["person"] --> dev1["development box<br/>cron tick, primary checkout,<br/>/tracks, status line"]
    dev1 -- "ssh via jump host<br/>(~18.6 KB/s up)" --> lanes1["2 build lanes<br/>gates + writers"]
  end
  subgraph after["after"]
    direction LR
    p2["person"] --> dev2["development box<br/>or laptop: ssh only"]
    dev2 -- "ssh via jump host" --> coord["coordinator guest<br/>cron tick, primary checkout,<br/>/tracks, status line"]
    coord -- "rig LAN" --> lanes2["2 build lanes<br/>gates + writers"]
  end

A tick uses little CPU: one claude -p session talking to a remote API, git, and node scripts. The heavy parts (gates, landing checks, writers) already run on the lanes through bench/on-lane.sh and bench/lane-writer-launch.sh. Moving the tick matters because every gate and writer launch now crosses the development box's slow uplink as a bundle (docs/infrastructure.md → "Build lane"). A tick on the rig LAN does not.

Where the coordinator runs #

	New LXC (chosen)	One of the two lane guests
Isolation	Only the tick, the primary and its local-only files	Shares a user and home with unattended claude --bg writers that run acceptEdits
CPU contention	None with gates; the tick is light	A 96-thread gate or batch competes with the tick, and a busy tick slows a gate
Blast radius	A writer or a pkill -f on a lane cannot touch the primary (the pkill incident in docs/infrastructure.md → "Rig change log")	bench/lane-apply.sh runs git clean -fdx in copies under the same home; a lane restart stops ticks and gates together
Cost	One more guest to provision and back up	None

The coordinator guest is created with bench/lane-create-lxc.sh's own sizing variables (LANE_CORES=8 LANE_MEMORY_MB=16384 LANE_DISK_GB=60). It goes on a node that carries no lane, so losing one node never takes a lane and the coordinator together. It is provisioned with bench/lane-provision.sh and the writer-host section (LANE_PI_TRACKS_URL, plus a LANE_WRITER_SETUP command that installs the tick's own harness: its CLI, configuration checkout and policy hook). That installs everything the tick needs: git, Node 22, python3, jq, the harness, and pi-tracks at ~/.local/share/pi-tracks, the path the cron entry names. It also installs the Rust toolchain and browsers, which the coordinator uses only when a lane is unreachable (see "Gates when both lanes are down"). LANE_WRITER_DIRS names the primary checkout and its .worktrees instead of the lane roots. Add a weekly backup job beside the two lane jobs, because the primary can hold landed commits no forge has yet.

The primary checkout and its remotes #

flowchart LR
  pub["public forge"] -- "pull mirror, every 10 min" --> rig["rig forge"]
  rig -- "fetch (LAN, no rate limit)" --> prim["primary on the<br/>coordinator guest"]
  prim -- "push main + spec/* tags<br/>(a person, at first)" --> pub
  pub -- "lane mirrors fetch<br/>(NEV_LANE_GIT_URL)" --> lanes["lane mirrors"]
  prim -- "bundle of commits<br/>not on origin/main" --> lanes

• Clone. git clone from the rig forge's pull mirror of the current repository. Then set remote.origin.pushurl to every push URL the development box's primary has (git remote -v; during the rename's step 4 there are two). Fetching from the rig forge avoids the public forge's rate limit.
• Mirror lag is harmless. The mirror is up to 10 minutes behind a push, so a fetch can move origin/main back to an older commit. bench/remote-lane.sh bundles origin/main..HEAD, so the bundle just gets larger. After a push, the forge's POST /repos/{owner}/{repo}/mirror-sync shortens the lag. bench/land.sh reads only local main, and never origin.
• Who pushes. bench/land.sh says "Nothing is pushed", and a person pushes main from the development box; the one exception is jobs/scripts/poll-and-land.sh, which pushes the spec/<name>/<sha> tag of each job it lands to JOBS_TAG_REMOTE (default origin). The first cutover keeps that, with no forge credential on folknet. A person pushes from the laptop or the development box with git fetch <coordinator remote> main followed by git push origin main 'refs/tags/spec/*'. The alternative is git push on the guest over ssh -A, using the person's own key through a forwarded agent.
• Automatic push, later. If landings should reach the public forge unattended:
  • Generate a deploy key on the guest (ssh-keygen). The private half never leaves it.
  • A person adds the public half to the repository as a write deploy key in the forge's web interface. That is the only provisioning step, and nothing is committed.
  • To keep the key away from writers, it belongs to a second unix account that runs no writer and no session. The tick account can only reach it through one sudo rule for a fixed script. That script pushes main fast-forward only, plus spec/* tags.
  • The key is in the guest's backups, so the backup storage must not be the NAS export that is open to the whole subnet.
  • Whether landed code goes public with no person in between is a person's decision.

Exactly one tick owner #

The tick's lock (spec-watch-tick.sh, flock on ~/.local/state/spec-watch/<repo basename>.tick.lock) exists only on one machine. Two coordinators would both see the same free slots and launch duplicate writers into the same lane copies: the lane path is <root>/<worktree basename> on both. They would also land into two diverging main branches. Three measures keep one owner, and none of them needs code:

1. Remove the development box's cron entry. Run spec-watch-schedule.sh remove <primary>. It deletes only the marker line and the entry for that repository, so the other repositories' entries stay.
2. Retire the development box's primary. Rename the directory (mv <primary> <primary>.retired). A leftover cron entry or a tick run by hand then stops at spec-watch-tick.sh's first check (not a primary checkout). A tick already running ends normally, since it holds open files, not the path. Renaming it back is the rollback.
3. Wait out a running tick before the first folknet tick. Both conditions must hold on the development box: flock -n <lock> true succeeds, and <repo basename>.tick.pid names no live process.

A lease that both machines can see, for example an owner ref on the rig forge that spec-watch-tick.sh checks, would enforce this by mechanism instead of by procedure. It is not needed while one person does the cutover. It is listed under open questions.

Writers and gates from the coordinator #

No script changes. The ssh routes to the lanes change, and they live only in local-only files:

• bench/lane.local.json. Copy the development box's lanes[] entries (lanes[0] on the first guest, lanes[1] on the second) with the rig-network lane addresses. Change ssh_opts to drop -J <jump host> and name the coordinator's own key. root stays the same, and so does git_url, whatever the rename's step 4 has reached by then.
• The primary's .env. Its GENERA_LANE_SSH, GENERA_LANE_SSH_OPTS and GENERA_LANE_GIT_URL (or the NEV_* names) take precedence over lane.local.json in bench/on-lane.sh and bench/remote-lane.sh. They must name the same jump-free route, or be left out. Every worktree links this file (tracks.json → install).
• Lane access. Generate an ssh key on the coordinator guest and add its public half to ~lane/.ssh/authorized_keys on both lanes. A lane writer's liveness check (bin/spec-status.mjs writerFor()) reads ssh and sshOpts from the per-track runtime file, which bin/spec-watch-launch.mjs fills from the launch script's output. New launches therefore carry the new route automatically.
• Faster transfers. Bundles, --pull directories and writer fetch-backs (bench/lane-writer-fetch.sh) move over the rig LAN instead of through the jump host.

Gates when both lanes are down #

bench/on-lane.sh falls back to a local run on the coordinator guest. That still runs on folknet hardware, but with 8 cores a cold cargo test --workspace is slow. A cold run peaks at about 1.1 GB more memory. This is acceptable: nothing needs a fast turnaround, and a gate that fails on a reachable lane is never retried locally. Local writers are the same case: bin/spec-watch-launch.mjs launches one on the guest only when every lane refuses.

Authentication #

• Claude. The tick runs claude --add-dir ... -p '/spec-watch' (spec-watch-tick.sh, with ANTHROPIC_MODEL=claude-opus-5) on the same subscription as the lane writers. A person signs in once over ssh with claude auth login, the step already used for the lanes.
• ~/.config/pi-tracks/env. Contains TRACKS_TOOL=claude, the same as on the development box.
• Model gateway. NEV_MODEL_URL (or GENERA_MODEL_URL) in the primary's .env names the gateway's rig-network address, the one each lane keeps in ~lane/.lane-model-env. The development box's value is a loopback port forward and does not work on the guest. The key is copied as it is.
• Metered key. Leave ANTHROPIC_API_KEY out of the guest's .env. No unattended step may spend metered credit (the roster's decision rule), and a key that is absent cannot be spent. Add it back only while a person runs an approved metered spec.
• $HOME/.config/folknet-writer/env (mode 600) is copied only when stage 3 of docs/folknet-writer-design.md needs a writer launched from the guest.

Secret files move by scp from the development box, one file each, mode 600, and never through git.

How a person sees state #

The status line, /tracks (tracks-status.sh), /waiting (waiting.mjs) and spec-status.mjs all read local files: the primary, ~/.local/state/spec-watch/ and its stages/. They therefore run on the coordinator guest. A person runs ssh <coordinator> from the development box or the laptop, then either starts claude in the primary (the ~/.claude checkout supplies the same status line and hooks) or runs the scripts directly. For a quick look without a session: ssh <coordinator> tail -n 40 .local/state/spec-watch/spec-watch.log, or ssh <coordinator> .local/share/pi-tracks/bin/spec-status.mjs <primary>. Hand-started interactive tracks run there as well, so that main moves in only one checkout.

~/.claude and pi-tracks on the guest #

Both are anonymous, read-only clones made by bench/lane-provision.sh, as on the lanes. Changes are still made on the development box or the laptop, then committed and pushed. The guest only pulls. A pull must not rewrite scripts under a running tick, because bash reads a script as it runs. So the guest's crontab gets a second entry that takes the same lock before pulling: flock -n <lock> sh -c 'git -C ~/.claude pull -q --ff-only; git -C ~/chook pull -q --ff-only; git -C ~/.local/share/pi-tracks pull -q --ff-only', every 15 minutes. ~/.claude/settings.local.json is per machine and gitignored, so it is written by hand.

Cutover #

Preconditions.

• No spec track is live: every spec is landed or disabled.
• No hand-started track is live in the development box's .worktrees. Each one has landed, or its branch has been pushed and its worktree removed, so it can be recreated on the guest.
• The development box's main has been pushed.

The per-track runtime state is then not needed. Its files are keyed by a hash of the worktree's absolute path (bin/spec-status.mjs runtimeFile()), so they would not carry over to a new path anyway. Cut over in the daytime, with a person watching the first tick.

sequenceDiagram
  participant D as development box
  participant C as coordinator guest
  participant L as lanes
  C->>C: 1-4 create, provision, clone, local-only files
  C->>L: 5 check: lane ssh + on-lane gate
  D->>D: 6 remove cron, wait for lock, rename primary
  C->>C: 7 install cron, watch first tick
  C->>L: 8 enable one spec, it writes, reviews, lands

1. Create and provision the guest. Run bench/lane-create-lxc.sh with the sizes above, then bench/lane-provision.sh with LANE_PI_TRACKS_URL, LANE_WRITER_SETUP and LANE_WRITER_DIRS set to the primary and its .worktrees. Check: a rerun prints only unchanged: lines.
2. Sign in. Run claude auth login over ssh. Check: claude -p 'reply ok' --model claude-opus-5 prints ok.
3. Clone the primary. Clone from the rig forge mirror and set the push URLs. Check: git -C <primary> rev-parse main equals the development box's main.
4. Write the local-only files. These are .env (lane keys, gateway address, no metered key), bench/lane.local.json, ~/.config/pi-tracks/env and ~/.claude/settings.local.json. Then run the tracks.json install step once in the primary if oracle/node_modules is needed there (bench/oracle-install.sh after a landing expects it). Check: git -C <primary> status --porcelain is empty.
5. Check the lane routes. Add the guest's key on both lanes. Check: ssh -o BatchMode=yes <lane> true succeeds for each entry, and bench/verify.sh in a throwaway worktree prints on-lane: ... running on the build lane.
6. Stop the development box. Run spec-watch-schedule.sh remove <primary>, wait until the lock is free, then mv <primary> <primary>.retired. Check: crontab -l has no entry for genera, and no === tick line for it appears in the development box's log after this point.
7. Start the guest. Run TRACKS_TOOL=claude spec-watch-schedule.sh install <primary>. Check: within 5 minutes the guest's spec-watch.log has a === tick line, then skipped: idle or an === exit 0.
8. First real run. Enable one small spec. Check: it launches on a lane, is fetched back, reviewed and landed, a spec/<name>/<sha> tag appears, and the floors ran on a lane.
9. Push. A person pushes main and the tags. Check: git ls-remote on the public forge shows the landed commit.

Rollback #

On the guest, stop ticking. On the development box, restore the checkout and its cron entry:

ssh <coordinator> .local/share/pi-tracks/bin/spec-watch-schedule.sh remove <guest primary>
mv <primary>.retired <primary> && TRACKS_TOOL=claude ~/.local/share/pi-tracks/bin/spec-watch-schedule.sh install <primary>

That restores today's setup exactly as long as the guest landed nothing. If it did, first wait for the guest's lock to be free, then run git -C <primary> pull --ff-only <guest primary over ssh> main and fetch its spec/* tags. A spec track that is live on the guest has its worktree and runtime state only there. Let it land first, or push its branch and accept that the development box's tick starts it again from that branch.

Other repositories #

The development box's crontab also ticks the sandbox and two other repositories, each with its own marker line. Each one moves by the same steps, one repository at a time, after genera has run cleanly on the guest. The sandbox has no remote, so it is cloned over ssh from the development box.

Open questions #

• Does the public forge offer per-repository deploy keys? If not, an automatic push needs a dedicated forge account's key, or stays a person's step.
• Should landings be pushed without a person? This is a person's decision (see "Who pushes").
• Should there be a lease both machines can see, so that a second coordinator is refused by mechanism and not only by the rename? A pi-tracks change to spec-watch-tick.sh.
• Which node carries the guest? Pick one with no lane and spare memory. Record the choice in the local rig notes.
• Should the guest also hold the gateway port forwards the development box keeps today? On the rig LAN it reaches the gateway directly, so probably not.
• Once genera has moved, the "Two machines" paragraph in ~/.claude/CLAUDE.md (projects live at ~/workspace/<name> on the development box) needs a line for repositories whose primary is on the coordinator guest.

Files and symbols this design relies on #

nev: bench/lane-create-lxc.sh, bench/lane-provision.sh (writer-host section, LANE_WRITER_DIRS), bench/lane.example.json, bench/lane-writer-launch.sh, bench/lane-writer-fetch.sh, bench/remote-lane.sh, bench/lane-apply.sh, bench/on-lane.sh, bench/verify.sh, bench/land.sh, bench/oracle-install.sh, bench/web-features-update.sh, tracks.json (install, specWatch.lanes, landScript, landChecks), docs/infrastructure.md, docs/folknet-writer-design.md.

pi-tracks: bin/spec-watch-tick.sh (the lock, pid file and not a primary checkout check), bin/spec-watch-schedule.sh (install, remove, the cron marker line), bin/spec-status.mjs (runtimeFile(), writerFor()), bin/spec-watch-launch.mjs (lane launch and local fallback), bin/spec-watch-due.mjs, bin/tracks-status.sh, bin/waiting.mjs, prompts/spec-watch.md.

Elsewhere: ~/.claude/docs/working-modes.md → "Mode 3", ~/.claude/statusline.sh (load_stages), $HOME/rig-notes/rig-access.md.