Folknet inventory #

A design, not yet built, for keeping correct and current facts about what is on folknet, and for handing those facts to a job when it starts, so that a job never picks a model, lane or service that is not there.

The case behind it: a job chose glm-5.3-flash, an alias the gateway still lists although nothing has served it since 2026-09-22, and failed on its first request with a 503. Nothing in the job's path had any way to know. Today the facts are spread over prose (docs/infrastructure.md → "As discovered", docs/rig-capacity.md), gitignored files (bench/lane.local.json, bench/rig-inventory.local.json), the private rig notes on the development box, and ~/.lane-model-env on each lane; nothing compares any of them with the live rig.

In brief #

• Three kinds of fact, one file each. What should exist is declared by hand in a tracked, public file, bench/folknet.json, by id only. How to reach each id (addresses, accounts, key paths) is in a private file on each lane, never in a repository. What is there right now is written by a probe on each lane into a snapshot, ~/folknet/inventory.json, keyed by the same ids.
• Model readiness is a one-token completion, not /v1/models: the gateway lists aliases with no backend, and each pool's HAProxy reports liveness only.
• The snapshot holds no secret and no address, by construction and by a check before it is written, so it can be copied into run records, logs and result.json freely.
• A job gets a frozen copy at start. job-run copies the snapshot into the job's state directory, hands every stage its path as JOB_INVENTORY, and before the first stage refuses to run when a model or service the manifest names is not ready, or when the snapshot is too old.
• Stale means unavailable. When the probe stops, the snapshot ages out and jobs refuse to start, the same rule as "no lane answered, nothing runs".

Decisions #

• The alias list is declared in this repository, in bench/folknet.json. The site file the gateway's config is generated from lives outside this repository and outside the rig, so it cannot be the source a job or a test reads; the probe's undeclared report (listed by the gateway, not declared here) catches drift between the two.
• The discovery collector is not extended for now. This project's probe stands alone, and the collector stays a source of host liveness only. Per-alias readiness in the collector remains a later option; it would shrink the probe, not replace the declared file.
• job-run gains exit code 3, "nothing ran: the rig lacks something", distinct from 1 (a stage failed) and 2 (bad input), so a poller can retry later instead of parking the job. The change lands with the runner work on track/dagu-trial, since the jobs crate exists only there.

What the discovery collector already provides #

The site runs a discovery collector, maintained outside this project: every host advertises itself over mDNS, and the collector aggregates host agents' reports and DNS-SD advertisements. It exists and answers; one read-only look (2026-09-25) found:

• A dashboard, and one JSON document at /api/discovery (schemaVersion 1) with compute.hosts (per host: identity, power, reporting, stale, and tables of CPU, memory and GPU), inventory.capacity (totals from fresh reports only), services (per entry: lastSuccessAt, stale, and data with kind, status, readiness, healthEndpoint, registrations), and a 90-day change journal. freshnessSeconds is 600: an entry not refreshed in ten minutes is marked stale.
• For models it is not enough. It knows the gateway (kind: gateway, status: responding) and two model pools, both from their own DNS-SD descriptors; every one has readiness: "unverified", checked against a liveness or /v1/models endpoint. The retired GLM pool is still advertised (and reports an HTTP error); the four-card VM pool and the DeepSeek pair are not advertised at all. Nothing is per alias.

So the collector is a good source for host liveness and capabilities and for "is the NAS or the forge host up", and not a source for "can this alias answer a completion". The design reads it and does not depend on it for any job gate.

Sources of truth #

Fact	Source	Kept current by	May be this old	When the source fails
Which aliases exist, which pool role serves each, its in-flight limit, whether it is retired	declared, bench/folknet.json	a commit on main, made with the rig change that adds or retires the alias	as old as main	n/a
Whether an alias answers a completion	probed: one-token completion through the gateway, with the key jobs use	the lane's probe, every 5 minutes	15 minutes at job start	alias marked failing with the error; see below
Whether the gateway lists an alias	probed: GET /v1/models	the same probe	15 minutes	listed/not-listed recorded, never used as readiness
Whether the key jobs use is granted an alias	probed: the same completion (the gateway answers 401 for an alias outside the key's allowlist)	the same probe	15 minutes	alias marked not-granted
The lane itself: free disk, load, the NAS share mounted and writable, the git mirror present	probed on the lane	the probe	15 minutes	the lane's snapshot says so; submit skips that lane
Forge reachable	probed: Forgejo's unauthenticated version endpoint (not a git fetch: the forge rate-limits)	the probe	15 minutes	failing; jobs that publish refuse
Host roles, class and capabilities (architecture, cards, memory class)	declared, bench/folknet.json, by role id and class	a commit	as old as main	n/a
Host liveness and measured capacity	the discovery collector, read by the probe	the collector's own cadence, read every 5 minutes	the collector's freshnessSeconds; its stale flag is copied through	recorded as unknown; informational, gates nothing
Remote providers and the rules on them	declared, bench/folknet.json → providers	a commit	as old as main	n/a
Subscription agent logged in on the lane	probed: the harness's own login check, without a model call	the probe	1 hour	failing; writer and review stages that need it refuse
Addresses, accounts, key paths	private bindings file on each lane	by hand, like ~/.lane-model-env today	n/a	the probe cannot run; the snapshot ages out
Card counts and power per pool	private, bench/rig-inventory.local.json (unchanged)	by hand	n/a	n/a

Declared file. bench/folknet.json names roles (gateway, pool:qwen27b-eight-card, pool:deepseek-arm-pair, lane:0, nas, forge, collector), each with its class, capabilities (architecture, card class, memory class) and location as an opaque site and rack id whose real name is in the bindings; aliases with their pool role, max_in_flight (the figures docs/infrastructure.md already states, such as 16 for qwen3.8-flash-next) and a state of serving or retired with a date, and providers. It is under bench/, so bench/land.sh refuses a track that edits it: a job cannot widen the set of things it is checked against. The probe reads it from the lane mirror's main (git show main:bench/folknet.json), never from a job's tree, and records that commit in the snapshot.

The readiness probe. For each declared alias that is not retired: POST /v1/chat/completions with max_tokens: 1, a fixed short prompt, and a 30-second timeout. The result is one of:

• ready — any 200 answer, reasoning-only included;
• failing — an error answer (the 503 of an alias with no deployment, a 5xx from the pool), with the status code and a scrubbed message;
• not-granted — a 401 or 403 naming the key's allowlist;
• slow — the timeout expired. A pool at its in-flight limit queues rather than refuses (the ARM replicas' HAProxy queues up to 30 minutes), so a busy pool looks like a slow one; slow is not failing.

The last result wins; there is no hysteresis. A false "ready" costs a job that fails after starting, a false "not ready" only delays one, so one failure is enough and one success clears it. Each entry keeps observed_at, since (when the status last changed) and last_ready_at. Aliases the gateway lists that bench/folknet.json does not declare are recorded as undeclared, which is how a rig change made without its declaration shows up. The probe costs one output token per alias per five minutes.

When the probe itself fails (lane restarted, bindings missing, gateway unreachable), the snapshot is not rewritten, its generated_at ages, and everything that reads it treats an old snapshot as "nothing is available".

Public and private #

The repository is public, so no real address, host name, account or key enters it.

• Public: bench/folknet.json (ids, classes, alias names, limits, rules), this document, the probe's code, and every snapshot, which by design carries ids and statuses only. Alias names are already public (docs/infrastructure.md).
• Private, on each lane: ~/.config/folknet/bindings.json, mode 600: per role id, the address and port the probe uses (gateway, collector, forge's web port), and the path of the gateway key file. The gateway address already lives in ~/.lane-model-env, and the key in ~/dagu/home/model-key; the bindings file points at those rather than holding second copies. Written by bench/lane-provision.sh from environment variables, the way LANE_MODEL_URL becomes NEV_LANE_MODEL_URL today.
• Private, on the development box: $HOME/rig-notes/rig-roles.json (the hosts bench/rig-snapshot.sh logs into) and bench/*.local.json, unchanged. The design adds nothing there.

Keeping secrets out of what a job sees. Error messages quote URLs (a refused connection names the address it tried). Before writing, the probe replaces every string from the bindings file (address, host name, port-bearing URL, key) that appears anywhere in the snapshot by its role id, then refuses to write the snapshot at all if any bound value, or anything shaped like an IP address or bearer token, remains. The key reaches the probe's HTTP client from its file and is never on a command line. A job therefore reads its context without reading anything secret, and the gateway key keeps reaching a stage only the way it does now (NEV_MODEL_API_KEY in the step's environment).

What a job receives when it starts #

Shape: a file. One JSON document per lane, rewritten atomically by the probe:

{
  "format": 1,
  "generated_at": "2026-09-25T18:00:00Z",
  "lane": "lane:0",
  "declared_commit": "<sha of main that bench/folknet.json was read from>",
  "models": {
    "qwen3.8-27b-fast": { "state": "serving", "status": "ready", "listed": true,
                          "observed_at": "…", "since": "…", "last_ready_at": "…",
                          "max_in_flight": 16 },
    "glm-5.3-flash":    { "state": "retired", "retired_on": "2026-09-22", "listed": true }
  },
  "services": {
    "gateway": { "status": "ready", "observed_at": "…" },
    "nas":     { "status": "ready", "observed_at": "…" },
    "forge":   { "status": "ready", "observed_at": "…" }
  },
  "lane_state": { "disk_free_gb": 0, "load_1m": 0.0, "observed_at": "…" },
  "providers": {
    "gateway":       { "billing": "self-hosted",  "allowed": true },
    "claude-code":   { "billing": "subscription", "allowed": true, "status": "ready" },
    "anthropic-api": { "billing": "metered", "allowed": false,
                       "rule": "no metered Anthropic API without a person's approval" }
  },
  "hosts": { "pool:deepseek-arm-pair": { "reporting": true, "stale": false } }
}

A file rather than a small service on the lane because a file needs no process to be running when the job starts (Dagu, for one, does not come back after a lane restart), it can be frozen with the job, and a stale file is visibly stale.

How it reaches stages.

1. The Dagu run step (bench/dagu/job.yaml) passes --inventory "$HOME/folknet/inventory.json" to job-run. The path is not a secret, so it may be a plain argument; nothing new goes into Dagu's run parameters.
2. job-run (jobs/src/runner.rs run_inner()) copies it to <state>/inventory.json on a job's first start, beside the manifest copy it already keeps, and on every resume copies the current one again (a resumed job must not trust a snapshot from hours ago).
3. Before the first stage that has not passed, job-run checks the manifest against the copy (below). A failure writes the reason into state.json and the result, and exits with a new code, 3, meaning "nothing ran: the rig lacks something", distinct from 1 (a stage failed) and 2 (bad input), so a poller can submit the job again later instead of parking it.
4. Every stage gets JOB_INVENTORY=<state>/inventory.json beside the JOB_* variables it already gets. A stage that runs for hours and wants the live view, such as a batch wrapper choosing among models before each batch, reads JOB_INVENTORY_LIVE, the lane's own snapshot path, and applies the same freshness rule.
5. result.json records the copy's sha256, its generated_at, declared_commit, and the entries for the models and services the job used, so a failed run shows what the job was told.

The check before the first stage (job-check inventory <manifest> <inventory>, a compiled-in check like the others). It divides the work with the requirements check that already runs at the same point (jobs/src/preflight.rs, docs/infrastructure.md -> "Dagu on a lane"): that one asks what the lane itself can answer right now -- a binary on PATH, a socket, a directory -- and this one asks what the rig's own inventory says about a model or a service, which can go stale between one minute and the next in a way a resolvable binary does not.

• the snapshot is at most 15 minutes old;
• every model the manifest names, in agents.*.model and in each model of a models string (alias:count:concurrency), is declared, not retired, and ready, or slow with a last_ready_at within the last 30 minutes;
• every provider an agent uses is allowed and, where it is probed, ready;
• the services the stages need are ready: the gateway when any agent is of kind gateway or names a gateway model, the NAS when a stage has a ledger check, the forge when the run publishes;
• a manifest concurrency above an alias's max_in_flight is a warning in the log, not a refusal: it queues rather than fails.

As built (jobs/src/inventory.rs): job-check inventory <manifest> <snapshot> [--declared <file>] [--now <time>], with the declared file defaulting to bench/folknet.json under the current directory. A claude-code agent's model names a model of the subscription rather than a gateway alias, so it is covered by the provider rule. Today's probe writes models only, so providers' status and services are checked once a snapshot carries those sections, and the forge is not checked: whether a run publishes is not in the manifest.

In job-run (--inventory <snapshot> [--declared <file>], jobs/src/runner.rs): the declared file defaults to {tools}/folknet.json, which the Dagu workflow's tools step lays out from the submitting checkout's bench/folknet.json, so a job's own tree is never read. The check runs after the requirements check (a lane missing a binary exits 9, not 3) and only when a stage is left to run. A missing or unreadable snapshot is a refusal like a stale one. On a refusal the reasons go to stderr and to state.json's inventory field, and a result is committed locally but not published. Refusals come in two kinds, and the exit code says which:

• May clear by the next probe -- a snapshot stale, missing or unreadable, a model not in the snapshot or not ready, a provider or service not ready. The result has status: "not-run", job-run exits 3, and jobs/scripts/poll-and-land.sh resends the job on its next pass without counting the resend.
• Never clears by itself -- a model not declared in bench/folknet.json or retired (there or in the snapshot), a provider not declared or allowed: false. Only a new manifest or an edit to the declared file changes these, so a resend would be refused forever. When any refusal is of this kind the result has status: "failed" with a null failed_stage, job-run exits 10, and the poller records the job failed with the reasons and does not resend it.

Every result carries the inventory field (sha256, generated_at, declared_commit, the entries for the models and services used, the refusals, permanent -- the refusals of the second kind -- and warnings).

The message names the alias and its status: "glm-5.3-flash is retired since 2026-09-22", "deepseek-v4.1-flash failing since 17:40 (503)".

The same check, earlier, at submit. The proposed submit script (docs/autonomous-loop.md → "How it is shipped and started") reads each lane's snapshot over the ssh call it already makes, skips lanes whose snapshot is stale or whose lane state is poor, and runs the same check before sending anything. The runner's check still runs, because minutes or hours can pass between submit and start.

Remote services #

Metered and subscription providers are entries in providers of bench/folknet.json, each with billing (self-hosted, subscription, metered), allowed, and a rule in words. The standing rule, no metered Anthropic API without a person's approval, is allowed: false on the metered entry. The job manifest check on track/dagu-trial already refuses anthropic in code (jobs/src/manifest.rs PROVIDERS); a test asserts that the allowed set declared in the file equals that list, so the code and the declaration cannot drift apart. There is no path by which a job obtains approval: a use that needs it happens outside jobs, by a person.

A model served by a remote OpenAI-compatible endpoint rather than the rig (Kimi, for development and testing while the rig is down) is declared as an alias like the rig's, with a provider naming its own providers entry (kimi-for-coding names kimi, billed by subscription). The check counts a named model under its alias's provider, gateway when there is none (jobs/src/inventory.rs alias_provider()), so a job on such a model is checked and recorded as that provider, not as the self-hosted rig. Such providers are reached only through an alias, never named by a manifest, so the test comparing the allowed set with PROVIDERS leaves them out. Which endpoint a lane's jobs reach is the lane's NEV_LANE_MODEL_URL in ~/.lane-model-env; one that pi talks to differently from what it guesses from the URL also sets NEV_MODEL_COMPAT there (jobs/scripts/pi-session.sh).

Availability of a subscription provider is probed without spending a model call: the harness's own login check on the lane, hourly. Its usage limits are not tracked; a stage that hits one fails like any other stage failure.

What replaces the prose #

• docs/infrastructure.md → "As discovered" keeps a description of the roles and how they fit together, and stops listing aliases and their states. Instead it points at bench/folknet.json for the list, and at bench/folknet-status.sh, which prints a lane's snapshot as a table (id, status, since, last ready).
• A workspace test (crates/harness/tests/) reads bench/folknet.json and fails when a model alias named in docs/infrastructure.md or docs/rig-capacity.md in backticks is not declared, so a doc cannot name an alias the file does not know. Retired aliases stay declared, so their history in those docs still passes.
• "Rig change log" stays prose: it is history. A change that adds or retires an alias edits bench/folknet.json in the same commit, which a person's landing already requires because the file is under bench/.
• bench/rig-inventory.local.json model_pools stays where it is; card counts and power are private by the rule in docs/rig-capacity.md.

Staged plan #

Each stage lands alone and has a check. The jobs crate (job-run, job-check, jobs/src/manifest.rs) is not on main yet; stage 1's first half and stages 2 and 3 do not need it, and every part that edits it waits for it to land. Every stage touches bench/, so each lands by a person's /land, not unattended.

1. Declared file and the check, no probe. bench/folknet.json with today's aliases, roles and providers, and the docs-to-declaration test (needs nothing else); then job-check inventory against a hand-written snapshot fixture and the providers-equal-PROVIDERS test (need the jobs crate). Check: the workspace tests; the fixture job manifest (jobs/fixtures/dagu/) with its models set to glm-5.3-flash is refused with "retired", and with qwen3.8-27b-fast in a fixture marked ready is accepted.
2. The probe, run by hand on one lane. bench/folknet-probe.py (the lane has Python but no requests, so the standard library only) with the bindings file and the scrub-then-refuse step. Check: on a lane, the snapshot shows the serving aliases ready and glm-5.3-flash as retired and listed; a scripted test feeds error messages carrying fake addresses and a fake key and asserts none reaches the file, and that a leftover one makes the probe refuse to write.
3. The probe on a timer on every lane. A cron entry for the lane user, installed by bench/lane-provision.sh (cron rather than a Dagu schedule, because Dagu does not restart with the lane). Check: the snapshot's age stays under ten minutes through a lane restart; with the entry removed, bench/folknet-status.sh reports it stale after fifteen minutes.
4. job-run takes it. --inventory, the copy into the state directory, JOB_INVENTORY and JOB_INVENTORY_LIVE, the check before the first stage, exit 3 (may clear) and exit 10 (never clears), the result fields, and the Dagu workflow passing the path. Check: jobs tests with a stale, a failing, a retired and a ready fixture; one fixture job run through bench/dagu-lane.sh with a retired alias exits 10 before any stage and records why in result.json, and the same job with a ready alias runs.
5. Submit, the collector, and the docs. The submit-side check, the collector's host and service entries copied into hosts, bench/folknet-status.sh, and "As discovered" rewritten to point at the file. Check: a submit with every lane's snapshot stale sends nothing and exits non-zero; the workspace test passes with the rewritten doc.

What it makes harder later. Every alias change on the rig now needs a declaration beside the gateway's own config (already true of the gateway's site file, which is a third place), and that declaration waits for a person's landing. The snapshot format becomes something the runner, the checks and the submit script all read, so changing it means a format bump and readers for both. A job gated on readiness at start can still lose a model an hour in; that case is handled only by stages that choose to read the live snapshot.

Left out #

• A query service. A file is enough while the questions are "is this alias ready" and "is this service up". If jobs come to need live answers mid-stage, a small HTTP front on the same snapshot can come later without changing its shape.
• Choosing a model for the job. The inventory says what is available; the manifest still names what to use. Falling back to another alias automatically is left out because it changes what a job measured.
• Per-replica state. Readiness is per alias, through the gateway. How many replicas behind an alias are up (and so its real in-flight capacity) is not visible there and would need logins to each pool host.
• Writing to the collector. The design only reads it. Adding per-alias readiness to the collector itself would give the whole site one source, but that service is not this project's.
• Usage limits and budgets for subscription providers.