# Coordinator on folknet

A design for moving the spec-watch coordinator off the development box onto folknet:
the scheduled tick (review, writer launches, landing through `bench/land.sh`), the
primary checkout it lands into, and the scripts it calls. Afterwards the development box
is only where a person connects from. It serves the standing rule that heavy work runs
on folknet hardware (the roster `handoff.md` on `main`). Nothing here is built or moved
yet. Hosts are named by role; real addresses stay in `$HOME/rig-notes/rig-access.md`.

**In one paragraph.** Create one new small unprivileged LXC, the **coordinator guest**,
with the existing lane scripts (`bench/lane-create-lxc.sh`, `bench/lane-provision.sh`).
Clone the primary checkout there, fetching from the rig forge's pull mirror. Recreate the
gitignored files by hand with rig-network addresses and no jump host. Install the same
cron entry with `spec-watch-schedule.sh`. Only one coordinator can tick because the
development box's cron entry is removed and its primary checkout renamed, so a leftover
entry or a hand-run tick fails at the tick script's first check. The tick, pi-tracks,
`bench/land.sh` and the lane scripts need no code change. Pushing to the public forge
stays a person's step at first; an automatic push with a deploy key is a later, separate
decision.

## Today and after

```mermaid
flowchart LR
  subgraph today["today"]
    direction LR
    p1["person"] --> dev1["development box<br/>cron tick, primary checkout,<br/>/tracks, status line"]
    dev1 -- "ssh via jump host<br/>(~18.6 KB/s up)" --> lanes1["2 build lanes<br/>gates + writers"]
  end
  subgraph after["after"]
    direction LR
    p2["person"] --> dev2["development box<br/>or laptop: ssh only"]
    dev2 -- "ssh via jump host" --> coord["coordinator guest<br/>cron tick, primary checkout,<br/>/tracks, status line"]
    coord -- "rig LAN" --> lanes2["2 build lanes<br/>gates + writers"]
  end
```

A tick uses little CPU: one `claude -p` session talking to a remote API, `git`, and
node scripts. The heavy parts (gates, landing checks, writers) already run on the lanes
through `bench/on-lane.sh` and `bench/lane-writer-launch.sh`. Moving the tick matters
because every gate and writer launch now crosses the development box's slow uplink as a
bundle (`docs/infrastructure.md` → "Build lane"). A tick on the rig LAN does not.

## Where the coordinator runs

| | New LXC (chosen) | One of the two lane guests |
|---|---|---|
| Isolation | Only the tick, the primary and its local-only files | Shares a user and home with unattended `claude --bg` writers that run `acceptEdits` |
| CPU contention | None with gates; the tick is light | A 96-thread gate or batch competes with the tick, and a busy tick slows a gate |
| Blast radius | A writer or a `pkill -f` on a lane cannot touch the primary (the `pkill` incident in `docs/infrastructure.md` → "Rig change log") | `bench/lane-apply.sh` runs `git clean -fdx` in copies under the same home; a lane restart stops ticks and gates together |
| Cost | One more guest to provision and back up | None |

The coordinator guest is created with `bench/lane-create-lxc.sh`'s own sizing variables
(`LANE_CORES=8 LANE_MEMORY_MB=16384 LANE_DISK_GB=60`). It goes on a node that carries no
lane, so losing one node never takes a lane and the coordinator together. It is
provisioned with `bench/lane-provision.sh` and the writer-host section (`LANE_PI_TRACKS_URL`,
plus a `LANE_WRITER_SETUP` command that installs the tick's own harness: its CLI,
configuration checkout and policy hook). That installs everything the tick needs:
`git`, Node 22, `python3`, `jq`, the harness, and pi-tracks at
`~/.local/share/pi-tracks`, the path the cron entry names. It also installs the Rust
toolchain and browsers, which the coordinator uses only when a lane is unreachable (see
"Gates when both lanes are down"). `LANE_WRITER_DIRS` names the primary checkout and its
`.worktrees` instead of the lane roots. Add a weekly backup job beside the two lane jobs,
because the primary can hold landed commits no forge has yet.

## The primary checkout and its remotes

```mermaid
flowchart LR
  pub["public forge"] -- "pull mirror, every 10 min" --> rig["rig forge"]
  rig -- "fetch (LAN, no rate limit)" --> prim["primary on the<br/>coordinator guest"]
  prim -- "push main + spec/* tags<br/>(a person, at first)" --> pub
  pub -- "lane mirrors fetch<br/>(NEV_LANE_GIT_URL)" --> lanes["lane mirrors"]
  prim -- "bundle of commits<br/>not on origin/main" --> lanes
```

- **Clone.** `git clone` from the rig forge's pull mirror of the current repository. Then
  set `remote.origin.pushurl` to every push URL the development box's primary has
  (`git remote -v`; during the rename's step 4 there are two). Fetching from the rig
  forge avoids the public forge's rate limit.
- **Mirror lag is harmless.** The mirror is up to 10 minutes behind a push, so a fetch
  can move `origin/main` back to an older commit. `bench/remote-lane.sh` bundles
  `origin/main..HEAD`, so the bundle just gets larger. After a push, the forge's
  `POST /repos/{owner}/{repo}/mirror-sync` shortens the lag. `bench/land.sh` reads only
  local `main`, and never `origin`.
- **Who pushes.** `bench/land.sh` says "Nothing is pushed", and a person pushes `main`
  from the development box; the one exception is `jobs/scripts/poll-and-land.sh`, which
  pushes the `spec/<name>/<sha>` tag of each job it lands to `JOBS_TAG_REMOTE` (default
  `origin`). The first cutover keeps that, with no forge
  credential on folknet. A person pushes from the laptop or the development box with
  `git fetch <coordinator remote> main` followed by `git push origin main 'refs/tags/spec/*'`.
  The alternative is `git push` on the guest over `ssh -A`, using the person's own key
  through a forwarded agent.
- **Automatic push, later.** If landings should reach the public forge unattended:
  - Generate a deploy key on the guest (`ssh-keygen`). The private half never leaves it.
  - A person adds the public half to the repository as a write deploy key in the forge's
    web interface. That is the only provisioning step, and nothing is committed.
  - To keep the key away from writers, it belongs to a second unix account that runs no
    writer and no session. The tick account can only reach it through one sudo rule for a
    fixed script. That script pushes `main` fast-forward only, plus `spec/*` tags.
  - The key is in the guest's backups, so the backup storage must not be the NAS export
    that is open to the whole subnet.
  - Whether landed code goes public with no person in between is a person's decision.

## Exactly one tick owner

The tick's lock (`spec-watch-tick.sh`, `flock` on
`~/.local/state/spec-watch/<repo basename>.tick.lock`) exists only on one machine.
Two coordinators would both see the same free slots and launch duplicate writers into
the **same lane copies**: the lane path is `<root>/<worktree basename>` on both. They
would also land into two diverging `main` branches. Three measures keep one owner, and
none of them needs code:

1. **Remove the development box's cron entry.** Run `spec-watch-schedule.sh remove
   <primary>`. It deletes only the marker line and the entry for that repository, so the
   other repositories' entries stay.
2. **Retire the development box's primary.** Rename the directory
   (`mv <primary> <primary>.retired`). A leftover cron entry or a tick run by hand then
   stops at `spec-watch-tick.sh`'s first check (`not a primary checkout`). A tick
   already running ends normally, since it holds open files, not the path. Renaming it
   back is the rollback.
3. **Wait out a running tick before the first folknet tick.** Both conditions must hold
   on the development box: `flock -n <lock> true` succeeds, and `<repo basename>.tick.pid`
   names no live process.

A lease that both machines can see, for example an owner ref on the rig forge that
`spec-watch-tick.sh` checks, would enforce this by mechanism instead of by procedure.
It is not needed while one person does the cutover. It is listed under open questions.

## Writers and gates from the coordinator

No script changes. The ssh routes to the lanes change, and they live only in local-only
files:

- **`bench/lane.local.json`.** Copy the development box's `lanes[]` entries (`lanes[0]`
  on the first guest, `lanes[1]` on the second) with the rig-network lane addresses.
  Change `ssh_opts` to drop `-J <jump host>` and name the coordinator's own key.
  `root` stays the same, and so does `git_url`, whatever the rename's step 4 has
  reached by then.
- **The primary's `.env`.** Its `GENERA_LANE_SSH`, `GENERA_LANE_SSH_OPTS` and
  `GENERA_LANE_GIT_URL` (or the `NEV_*` names) take precedence over `lane.local.json` in
  `bench/on-lane.sh` and `bench/remote-lane.sh`. They must name the same jump-free
  route, or be left out. Every worktree links this file (`tracks.json` → `install`).
- **Lane access.** Generate an ssh key on the coordinator guest and add its public half
  to `~lane/.ssh/authorized_keys` on both lanes. A lane writer's liveness check
  (`bin/spec-status.mjs` `writerFor()`) reads `ssh` and `sshOpts` from the per-track
  runtime file, which `bin/spec-watch-launch.mjs` fills from the launch script's
  output. New launches therefore carry the new route automatically.
- **Faster transfers.** Bundles, `--pull` directories and writer fetch-backs
  (`bench/lane-writer-fetch.sh`) move over the rig LAN instead of through the jump host.

### Gates when both lanes are down

`bench/on-lane.sh` falls back to a local run on the coordinator guest. That still runs on
folknet hardware, but with 8 cores a cold `cargo test --workspace` is slow. A cold run
peaks at about 1.1 GB more memory. This is acceptable: nothing needs a fast turnaround,
and a gate that fails on a reachable lane is never retried locally. Local writers are
the same case: `bin/spec-watch-launch.mjs` launches one on the guest only when every
lane refuses.

## Authentication

- **Claude.** The tick runs `claude --add-dir ... -p '/spec-watch'`
  (`spec-watch-tick.sh`, with `ANTHROPIC_MODEL=claude-opus-5`) on the same subscription
  as the lane writers. A person signs in once over ssh with `claude auth login`, the
  step already used for the lanes.
- **`~/.config/pi-tracks/env`.** Contains `TRACKS_TOOL=claude`, the same as on the
  development box.
- **Model gateway.** `NEV_MODEL_URL` (or `GENERA_MODEL_URL`) in the primary's `.env`
  names the gateway's rig-network address, the one each lane keeps in
  `~lane/.lane-model-env`. The development box's value is a loopback port forward and
  does not work on the guest. The key is copied as it is.
- **Metered key.** Leave `ANTHROPIC_API_KEY` out of the guest's `.env`. No unattended
  step may spend metered credit (the roster's decision rule), and a key that is absent
  cannot be spent. Add it back only while a person runs an approved metered spec.
- **`$HOME/.config/folknet-writer/env`** (mode 600) is copied only when stage 3 of
  `docs/folknet-writer-design.md` needs a writer launched from the guest.

Secret files move by `scp` from the development box, one file each, mode 600, and never
through git.

## How a person sees state

The status line, `/tracks` (`tracks-status.sh`), `/waiting` (`waiting.mjs`) and
`spec-status.mjs` all read local files: the primary, `~/.local/state/spec-watch/` and its
`stages/`. They therefore run on the coordinator guest. A person runs
`ssh <coordinator>` from the development box or the laptop, then either starts `claude`
in the primary (the `~/.claude` checkout supplies the same status line and hooks) or
runs the scripts directly. For a quick look without a session:
`ssh <coordinator> tail -n 40 .local/state/spec-watch/spec-watch.log`, or
`ssh <coordinator> .local/share/pi-tracks/bin/spec-status.mjs <primary>`. Hand-started
interactive tracks run there as well, so that `main` moves in only one checkout.

## `~/.claude` and pi-tracks on the guest

Both are anonymous, read-only clones made by `bench/lane-provision.sh`, as on the lanes.
Changes are still made on the development box or the laptop, then committed and pushed.
The guest only pulls. A pull must not rewrite scripts under a running tick, because bash
reads a script as it runs. So the guest's crontab gets a second entry that takes the same
lock before pulling:
`flock -n <lock> sh -c 'git -C ~/.claude pull -q --ff-only; git -C ~/chook pull -q --ff-only; git -C ~/.local/share/pi-tracks pull -q --ff-only'`,
every 15 minutes. `~/.claude/settings.local.json` is per machine and gitignored, so it
is written by hand.

## Cutover

**Preconditions.**

- No spec track is live: every spec is landed or disabled.
- No hand-started track is live in the development box's `.worktrees`. Each one has
  landed, or its branch has been pushed and its worktree removed, so it can be recreated
  on the guest.
- The development box's `main` has been pushed.

The per-track runtime state is then not needed. Its files are keyed by a hash of the
worktree's absolute path (`bin/spec-status.mjs` `runtimeFile()`), so they would not carry
over to a new path anyway. Cut over in the daytime, with a person watching the first
tick.

```mermaid
sequenceDiagram
  participant D as development box
  participant C as coordinator guest
  participant L as lanes
  C->>C: 1-4 create, provision, clone, local-only files
  C->>L: 5 check: lane ssh + on-lane gate
  D->>D: 6 remove cron, wait for lock, rename primary
  C->>C: 7 install cron, watch first tick
  C->>L: 8 enable one spec, it writes, reviews, lands
```

1. **Create and provision the guest.** Run `bench/lane-create-lxc.sh` with the sizes
   above, then `bench/lane-provision.sh` with `LANE_PI_TRACKS_URL`, `LANE_WRITER_SETUP`
   and `LANE_WRITER_DIRS` set to the primary and its `.worktrees`. *Check:* a rerun prints only `unchanged:`
   lines.
2. **Sign in.** Run `claude auth login` over ssh. *Check:*
   `claude -p 'reply ok' --model claude-opus-5` prints `ok`.
3. **Clone the primary.** Clone from the rig forge mirror and set the push URLs.
   *Check:* `git -C <primary> rev-parse main` equals the development box's `main`.
4. **Write the local-only files.** These are `.env` (lane keys, gateway address, no
   metered key), `bench/lane.local.json`, `~/.config/pi-tracks/env` and
   `~/.claude/settings.local.json`. Then run the `tracks.json` `install` step once in
   the primary if `oracle/node_modules` is needed there (`bench/oracle-install.sh` after
   a landing expects it). *Check:* `git -C <primary> status --porcelain` is empty.
5. **Check the lane routes.** Add the guest's key on both lanes. *Check:*
   `ssh -o BatchMode=yes <lane> true` succeeds for each entry, and `bench/verify.sh` in a
   throwaway worktree prints `on-lane: ... running on the build lane`.
6. **Stop the development box.** Run `spec-watch-schedule.sh remove <primary>`, wait
   until the lock is free, then `mv <primary> <primary>.retired`. *Check:* `crontab -l`
   has no entry for genera, and no `=== tick` line for it appears in the development
   box's log after this point.
7. **Start the guest.** Run `TRACKS_TOOL=claude spec-watch-schedule.sh install
   <primary>`. *Check:* within 5 minutes the guest's `spec-watch.log` has a `=== tick`
   line, then `skipped: idle` or an `=== exit 0`.
8. **First real run.** Enable one small spec. *Check:* it launches on a lane, is fetched
   back, reviewed and landed, a `spec/<name>/<sha>` tag appears, and the floors ran on a
   lane.
9. **Push.** A person pushes `main` and the tags. *Check:* `git ls-remote` on the public
   forge shows the landed commit.

## Rollback

On the guest, stop ticking. On the development box, restore the checkout and its cron
entry:

```
ssh <coordinator> .local/share/pi-tracks/bin/spec-watch-schedule.sh remove <guest primary>
mv <primary>.retired <primary> && TRACKS_TOOL=claude ~/.local/share/pi-tracks/bin/spec-watch-schedule.sh install <primary>
```

That restores today's setup exactly as long as the guest landed nothing. If it did,
first wait for the guest's lock to be free, then run `git -C <primary> pull --ff-only
<guest primary over ssh> main` and fetch its `spec/*` tags. A spec track that is live on
the guest has its worktree and runtime state only there. Let it land first, or push its
branch and accept that the development box's tick starts it again from that branch.

## Other repositories

The development box's crontab also ticks the sandbox and two other repositories, each
with its own marker line. Each one moves by the same steps, one repository at a time,
after genera has run cleanly on the guest. The sandbox has no remote, so it is cloned
over ssh from the development box.

## Open questions

- Does the public forge offer per-repository deploy keys? If not, an automatic push
  needs a dedicated forge account's key, or stays a person's step.
- Should landings be pushed without a person? This is a person's decision (see "Who
  pushes").
- Should there be a lease both machines can see, so that a second coordinator is refused
  by mechanism and not only by the rename? A pi-tracks change to `spec-watch-tick.sh`.
- Which node carries the guest? Pick one with no lane and spare memory. Record the
  choice in the local rig notes.
- Should the guest also hold the gateway port forwards the development box keeps today?
  On the rig LAN it reaches the gateway directly, so probably not.
- Once genera has moved, the "Two machines" paragraph in `~/.claude/CLAUDE.md` (projects
  live at `~/workspace/<name>` on the development box) needs a line for repositories
  whose primary is on the coordinator guest.

## Files and symbols this design relies on

nev: `bench/lane-create-lxc.sh`, `bench/lane-provision.sh` (writer-host section,
`LANE_WRITER_DIRS`), `bench/lane.example.json`, `bench/lane-writer-launch.sh`,
`bench/lane-writer-fetch.sh`, `bench/remote-lane.sh`, `bench/lane-apply.sh`,
`bench/on-lane.sh`, `bench/verify.sh`, `bench/land.sh`, `bench/oracle-install.sh`,
`bench/web-features-update.sh`, `tracks.json` (`install`, `specWatch.lanes`,
`landScript`, `landChecks`), `docs/infrastructure.md`,
`docs/folknet-writer-design.md`.

pi-tracks: `bin/spec-watch-tick.sh` (the lock, pid file and `not a primary checkout`
check), `bin/spec-watch-schedule.sh` (`install`, `remove`, the cron marker line),
`bin/spec-status.mjs` (`runtimeFile()`, `writerFor()`), `bin/spec-watch-launch.mjs`
(lane launch and local fallback), `bin/spec-watch-due.mjs`, `bin/tracks-status.sh`,
`bin/waiting.mjs`, `prompts/spec-watch.md`.

Elsewhere: `~/.claude/docs/working-modes.md` → "Mode 3", `~/.claude/statusline.sh`
(`load_stages`), `$HOME/rig-notes/rig-access.md`.