# Rig capacity

Measured rates for both halves of the loop: CPU-side verification (building the oracle
corpus, compiling candidates, scoring them) and GPU-side generation (the sampling and
capacity tiers of `docs/infrastructure.md` → "Three tiers"). The question is which tier
saturates first.

Hardware is described by role and class, never by count, hostname or account. The rig's
real counts live in the gitignored `bench/rig-inventory.local.json` (shape:
`bench/rig-inventory.example.json`); `bench/capacity.py` combines them with the rates here
to print absolute throughput and power draw. Nothing computed from the real counts is
written back here.

```mermaid
flowchart LR
  gpu["GPU: writes candidates<br/>~0.1–0.4 completions/s per pool"] --> cpu["CPU: compile + score<br/>144 candidates/s, 272,930 cases/s"]
  oracle["CPU: browser oracle<br/>~1.5 docs/s/core, once"] --> cpu
```

Writing candidates is the bottleneck by several orders of magnitude.

## CPU tier

Measured on the 16-core, no-GPU development box, shared with other tracks' work; the load
is reported beside each figure.

### Oracle corpus building

`bench/oracle-scaling.sh` runs N `oracle_shard` processes on disjoint seed shards, each
driving all three engines, 8 documents per worker, zero engine errors. Browser memory is
peak resident memory minus a pre-step baseline.

| workers | total docs | wall (s) | docs/s (fleet) | docs/s per core (16-way) | browser RSS, MB total / per worker | load (1/5/15 min, before) |
|---|---|---|---|---|---|---|
| 1  | 8   | 5.4  | 1.48 | 1.48 | 862 / 862   | 43.9 / 39.4 / 26.1 |
| 2  | 16  | 5.9  | 2.73 | 1.36 | 1721 / 860  | 43.1 / 39.3 / 26.1 |
| 4  | 32  | 7.6  | 4.21 | 1.05 | 2783 / 696  | 43.6 / 39.5 / 26.3 |
| 8  | 64  | 11.9 | 5.38 | 0.67 | 5160 / 645  | 45.6 / 40.1 / 26.6 |
| 16 | 128 | 16.5 | 7.74 | 0.48 | 9539 / 596  | 47.5 / 40.6 / 26.9 |

- Memory is not the limit: under 1 GB per worker for all three engines.
- With only 8 documents per worker, browser launch (4–13 s) dominates. One worker given
  40 documents renders at **21.3 docs/s** through all three engines (1.88 s after a 3.8 s
  launch).
- Every step ran beside a WebGPU CTS sweep saturating the box, so the falling per-core
  rate may be contention rather than anything inherent. Open: re-measure in a quiet
  window.

### Batch candidate compilation

`bench/gen_candidates.py` generates N synthetic candidates (five templates, 50–300 lines)
as one crate (`batch`) or N crates (`separate`); `bench/compile-batch.sh` times `cargo
check` and `cargo build --release`, uncached and through the shared sccache (warm, not
from empty). CPU-seconds, under a load average of 24–50:

| N | layout | uncached check (cpu-s) | uncached build (cpu-s) | sccache check (cpu-s) | sccache build (cpu-s) |
|---|---|---|---|---|---|
| 10   | batch    | 0.12  | 0.15  | 0.15  | 0.07 |
| 10   | separate | 0.56  | 0.61  | 0.63  | 0.12 |
| 100  | batch    | 0.31  | 0.50  | 0.33  | 0.07 |
| 100  | separate | 5.11  | 5.52  | 6.05  | 0.71 |
| 1000 | batch    | 2.29  | 4.69  | 2.42  | 0.07 |
| 1000 | separate | 54.57 | 58.65 | 57.98 | 8.74 |

- At N=1000, one crate per candidate costs about **24× more CPU** per candidate to check
  (54.6 ms against 2.29 ms) and 12.5× more to build: per-crate overhead, not code size.
- Wall-clock at N=1000, batch, uncached: **144 candidates/s** on the contended box.
- Normalized-AST deduplication (`bench/src/bin/dedup_cost.rs`): about **2,000
  functions/s**, cheap enough to run before every batch.

### A real batch on a build lane

One `absolute_real_model_run` batch, 32 candidates from `qwen3.8-27b-fast` at 32 in
flight, started from the development box with `bench/absolute-mixed-batch.sh` and run on
a 96-thread lane (`docs/infrastructure.md` → "Build lane"). CPU is the lane container's
own cgroup usage sampled every 2 s (`bench/cpu-sample.sh`), split by phase with
`bench/cpu-phases.py`; "busy" is CPUs kept busy on average. Probes one crate at a time,
as every batch did before `candidate_crate::probe`:

| phase | wall (s) | CPU-s | avg busy | peak busy |
|---|---|---|---|---|
| generation loop (includes the worktree's first release build on the lane) | 216.5 | 1112.2 | 5.1 | 79.3 |
| draw: waiting on the model | 236.0 | 11.7 | 0.0 | 0.9 |
| probe: 32 candidates one crate at a time (first one built the probe dependencies) | 122.3 | 99.4 | 0.8 | 9.3 |
| scoring crate build (8 survivors) | 0.8 | 2.4 | 3.1 | 1.2 |
| scoring run | 0.5 | 0 | 0 | 0 |
| per-candidate rebuild for compiler error files (since removed) | 30.8 | 16.2 | 0.5 | 1.3 |
| **whole run** | **609.1** | **1280.2** | **2.1** | **79.3** |

8 of 32 compiled; best 82/108, equal to the committed candidate. Outside the one-off
builds, one feature run keeps **about one CPU busy** and spends most of its time waiting
on the model; filling the rig's CPUs would take hundreds of such runs at once, so the
CPUs are not what a batch waits on.

The same batch after the change: one wide `--keep-going` probe build with no compiler
wrapper, compiler error files written from that build's messages, nested builds free of
`cargo run`'s variables (`candidate_crate` `nested_cargo()`). Three runs:

| run | draw (s) | probe (s) | scoring build + run (s) | whole run (s) | avg / peak busy | compiled | best |
|---|---|---|---|---|---|---|---|
| after 1: 32 × `qwen3.8-27b-fast` | 283.9 | 19.0 | 1.5 | 323.2 | 0.4 / 8.5 | 12/32 | 82/108 |
| after 2: same | 297.1 | 16.3 | 4.1 | 318.6 | 0.2 / 11.0 | 11/32 | 82/108 |
| two pools: 16 × `qwen3.8-27b-fast`, 16 × `qwen3.8-flash-next` | 146.5 / 703.4 | 5.1 / 4.2 | 0.9 / 1.3 | 729.0 | 0.1 / 9.1 | 5/16, 13/16 | 74/108 |

The probe dropped from 122 s to 4–5 s. Runs 1 and 2 each spent about 15 s of their
probe rebuilding `ring`, `rustls`, `rustls-webpki`, `ureq`, `provenance` and
`generation`: `ring`'s build tracks `CARGO_MANIFEST_DIR`, which the batch binary
inherited from `cargo run` and passed to its nested builds, so any build of the same
target directory started elsewhere marked it dirty. Nested builds now drop those
variables; the two-pool run is the first with that change. The draw is now nearly the
whole run, and it runs on the serving hardware: the lane sits at under one busy CPU
waiting on replies (about 700 completion tokens/s across 32 requests to the fast pool).
The first two-pool run's `flash-next` draw overlapped another batch on the same pool; a
rerun with the pool to itself took 656.9 s for 16 (9 compiled, best 74/108) against
109.4 s for the fast pool's 16 (5 compiled, best 82/108), whole run 660.9 s, probes 4.6
and 1.6 s. Splitting a batch across the two pools does not shorten it: the slower pool
sets the wall time, at about twice a 32-candidate fast-pool batch.

Probing replayed on the same sources with warm dependencies (`bin/probe_replay.rs`,
`bench/probe-replay-measure.sh`), old against new, on one lane:

| candidates | one crate at a time | one wide build through sccache-dist | same, compiled |
|---|---|---|---|
| 32 absolute (the batch above) | 19.0–23.1 s | 14.0 s, avg 3.3 busy, peak 16 | 8 both ways |
| 64 absolute (an earlier mixed batch) | 41.6 s, avg 1.0 busy | 36.6 s, avg 4.0, peak 47.8 | 29 both ways |
| 64 overflow (an earlier mixed run) | 44.9 s, avg 1.0 busy | 42.9 s, avg 4.8, peak 57.0 | 56 both ways |

Both probes admit exactly the same candidates on all three sets. Through sccache-dist the
wide build sends each candidate's library to a builder (16–58 distributed compiles per
replay) but gains little wall time: the per-candidate work is under a second, so shipping
it costs about what it saves.

The same wide build with no compiler wrapper, against sccache-dist run twice back to back
(`PROBE_REPLAY_RUNS='wide-sccache wide-sccache wide-local'`, a fresh
`PROBE_REPLAY_WORK`), on one build lane:

| candidates | sccache-dist, first | sccache-dist, second | no wrapper | compiled |
|---|---|---|---|---|
| 32 absolute | 14.5 s (49 dist compiles) | 14.6 s (16) | 1.4 s | 8 in every run |
| 64 absolute | 35.7 s (82) | 35.8 s (49) | 1.4 s | 29 in every run |
| 64 overflow | 45.5 s (91) | 42.8 s (58) | 1.6 s | 56 in every run |

The second sccache-dist pass hands out fewer compiles (some are cache hits) yet takes the
same time, so the cost is not a cold cache or a first toolchain upload: it is a fixed
delay per compile through the wrapper. Without it the lane's own cores probe a batch
20–30 times faster (peak 7–31 busy CPUs for about a second). An earlier no-wrapper run on
a fresh work directory took 2.2 s for the 32. The probe runs without the wrapper.

### Inner-loop evaluation

`bench/run-inner-loop.sh` runs a stand-in candidate over `bench/fixtures/`, measuring the
harness-side cost every candidate pays (deserialize, clone, compare). Load average 57–59:

| threads | cases/s (fleet) | cases/s per core | load (1 min, before) |
|---|---|---|---|
| 1  | 25,709  | 25,709 | 57.4 |
| 2  | 65,241  | 32,621 | 57.2 |
| 4  | 97,411  | 24,353 | 57.2 |
| 8  | 167,757 | 20,970 | 57.2 |
| 16 | 272,930 | 17,058 | 57.3 |

No plateau through 16 threads: the inner loop is nowhere near the constraint.

### Other CPU workloads

No dedicated benchmarks yet for the conformance suites, the WebGPU CTS on software Vulkan,
or trace requests.

### Core budget formula

```
verification_cores = machines × cores_per_machine
```

Corpus building is occasional (a corpus is fixed per browser-version set); compilation and
scoring run on every sample and get the rest. `bench/capacity.py` reports the pool and
per-hour throughput; a specific split waits on the real candidate arrival rate.

## GPU tier

```
sampling_candidates_per_s  = cards    × per_card_candidates_per_s
capacity_escalations_per_s = machines × per_machine_escalations_per_s
```

All sweeps below use `bench/rig_throughput.py` with the real width-equation prompt
`bin/real_model_run.rs` assembles, one round per concurrency level (N requests at once).

### Measured — the rig's vLLM server

`qwen3.8-27b`: dense 27B, int4 AutoRound, 4-way tensor parallel on four 24 GB cards in one
VM, `max_model_len` 196608, vLLM 0.24 at the time (now 0.27.1). Prompt 4,807 tokens,
output capped at 3000 (since raised to 8000; the concurrency-1 reply hit the old cap).

| concurrency | completions/s | output tok/s | time to first token, mean (s) | latency, mean (s) | latency, max (s) |
|---|---|---|---|---|---|
| 1  | 0.025 | 76.5  | 4.7  | 39.2  | 39.2  |
| 4  | 0.067 | 142.2 | 14.1 | 49.4  | 59.9  |
| 8  | 0.094 | 167.3 | 25.2 | 66.6  | 85.4  |
| 16 | 0.116 | 240.0 | 41.5 | 117.4 | 137.4 |
| 32 | 0.125 | 237.2 | 95.9 | 186.3 | 256.6 |

Aggregate throughput plateaus by 32; latency climbs linearly throughout.

### Measured — the eight-card pool (`qwen3.8-27b-fast`)

Same model on eight cards (aliases `-fast` and `-long`, 131k context). Prompt 5,006
tokens.

| concurrency | completions/s | output tok/s | time to first token, mean (s) | latency, mean (s) | latency, max (s) |
|---|---|---|---|---|---|
| 1  | 0.312 | 21.8  | n/a | 3.2  | 3.2  |
| 2  | 0.112 | 144.3 | 2.5 | 10.5 | 17.8 |
| 4  | 0.207 | 203.2 | 2.7 | 9.1  | 19.3 |
| 8  | 0.240 | 338.5 | 4.2 | 18.0 | 33.4 |
| 16 | 0.342 | 523.0 | 6.7 | 27.0 | 46.7 |

No plateau by 16; about 2.2× the four-card pool at the same concurrency. The
concurrency-1 reply streamed no content before finishing at 70 tokens, so its time to
first token is unmeasured.

### Measured — the four-card VM (`qwen3.8-27b-fast-2`)

The eight-card pool's recipe on four 3090s in a VM, one worker per card, four requests per
worker (`docs/infrastructure.md` → "Where each piece runs"). Prompt 5,040 tokens,
2026-09-24.

| concurrency | completions/s | output tok/s | time to first token, mean (s) | latency, mean (s) | latency, max (s) |
|---|---|---|---|---|---|
| 1  | 0.070 | 106.8 | 5.9 | 14.3 | 14.3 |
| 2  | 0.087 | 254.1 | 5.7 | 21.5 | 22.9 |
| 4  | 0.174 | 317.7 | 5.1 | 15.8 | 22.9 |
| 8  | 0.322 | 685.1 | 4.7 | 18.2 | 24.8 |
| 16 | 0.398 | 819.3 | 8.8 | 27.6 | 40.2 |

Not a controlled comparison with the eight-card sweep (different days, other traffic on
that pool), but it is not the slower alias.

### Measured — GLM-5.3 Flash (2-way tensor parallel, 4-bit EXL3)

Probably mixture-of-experts. Same prompt, 3000-token cap. Stopped at concurrency 8
because every completion ran to the cap.

| concurrency | completions/s | output tok/s | time to first token, mean (s) | latency, mean (s) | latency, max (s) |
|---|---|---|---|---|---|
| 1 | 0.008 | 25.1 | 8.9  | 119.5 | 119.5 |
| 4 | 0.020 | 60.4 | 10.7 | 192.2 | 198.7 |
| 8 | 0.020 | 58.0 | 94.5 | 263.5 | 391.7 |

12 of 13 completions hit the cap. The reason: with `enable_thinking: false` this server
still writes its reasoning into `message.content`, so the budget goes to analysis prose
and code never arrives. That is a serving problem, not a prompt one, and GLM was replaced.

### Attempted — Qwen3.8-Flash-Next

Chosen over DeepSeek-V4-Flash to replace GLM because it shares `qwen3.8-27b`'s lineage,
already proven through the same extraction code, even though published two-Spark figures
rate it slower (42.6 tok/s single-stream against 72.4 for DeepSeek-V4-Flash and 41.1 for
GLM).

```mermaid
flowchart TD
  a["4-bit EXL3 checkpoint, 101 GB<br/>fits one 121 GB node"] --> b{"any vLLM build<br/>on the rig serves it?"}
  b -- "no: fork too old for the<br/>architecture, upstream has no EXL3" --> c["size a new EXL3 backend<br/>for upstream vLLM (not written)"]
  b --> d["serve with the checkpoint's own<br/>engine (TabbyAPI) instead"]
  d --> e["temperature 1.0 default:<br/>0 of 8 compile (word salad)"]
  e --> f["sampler preset 0.7 / top-k 20 / top-p 0.95:<br/>19 of 32 compile, best 140/227"]
```

- **Why vLLM failed.** The rig's EXL3 fork predates the model's architecture
  (`Qwen4ExpForConditionalGeneration`); a newer generic build also predates it; upstream
  vLLM 0.30.0 registers the architecture but has no EXL3 backend. The fork's EXL3 support
  is a one-file overlay for routed experts and one codebook only, while this checkpoint
  quantizes every linear with the newer `mul1` codebook and stores its 39 GB n-gram
  table as a trellis. So either direction needs a new EXL3 backend; the smaller path is
  an overlay onto upstream 0.30.0's arm64 image. The kernel library (1.5.1) was rebuilt
  for this GPU class; the backend is not written and stays queued.
- **Single node, the library's own server (TabbyAPI), 3000-token cap:**

  | concurrency | completions/s | output tok/s | time to first token, mean (s) | latency, mean (s) | latency, max (s) | hit 3000 cap |
  |---|---|---|---|---|---|---|
  | 1 | 0.016 | 24.9 | 9.6 | 64.4  | 64.4  | 0 of 1 |
  | 4 | 0.025 | 56.3 | 3.8 | 127.3 | 159.8 | 1 of 4 |
  | 8 | 0.033 | 85.1 | 5.2 | 212.5 | 242.1 | 4 of 8 |

  Unlike GLM, replies stop on their own. One sequential 8-candidate batch compiled 1 of 8
  (114/227, not admitted); the rest put prose inside the first Rust fence.

**Serving — four single-Spark replicas through the gateway (2026-09-22, evening).** One
replica per node as a systemd unit (`bench/sparks/`), GLM disabled, HAProxy least
connections with eight in flight per replica, gateway alias **`qwen3.8-flash-next`**.
Prompt 4,475 tokens by this engine's tokenizer:

| concurrency | completions/s | output tok/s | TTFT mean (s) | latency mean (s) | latency max (s) | hit 3000 cap |
|---|---|---|---|---|---|---|
| 1 | 0.012 | 25.1 | 7.9 | 84.1 | 84.1 | 0 of 1 |
| 4 | 0.036 | 66.4 | 6.5 | 73.0 | 111.2 | 0 of 4 |
| 8 | 0.064 | 123.1 | 1.7 | 89.5 | 124.2 | 0 of 8 |
| 16 | 0.109 | 161.4 | 2.3 | 82.9 | 146.6 | 0 of 16 |
| 32 | 0.132 | 238.9 | 3.4 | 145.3 | 241.8 | 0 of 32 |

Sixteen in flight is the latency knee. The first batch through the alias compiled 0 of 8:
the client sends no sampling parameters and the engine's fallback was temperature 1.0 with
no truncation, so replies drifted into multilingual word salad. With the checkpoint's own
preset as fallback (`bench/sparks/sampler-genera.yml`: temperature 0.7, top-k 20, top-p
0.95), one 32-candidate batch took 7 m 49 s wall and **compiled 19 of 32, admitted 14,
best 140/227**, replacing the committed `claude-opus-5` width candidate (132/227). Since
2026-09-24 only two replicas serve this alias (`docs/infrastructure.md`).

**Per-replica batch size, one node (2026-09-23, isolated rerun).** One node switched from
`cache_mode: FP16`, `cache_size: 65536`, `max_batch_size: 8` to `Q8`, `131072`, `16`
(same memory), HAProxy `maxconn` 8 → 16. Pool idle before each round. Output tok/s:

| in flight | pool, all baseline | pool, trial node changed | one Spark direct, trial | one Spark direct, baseline |
|---|---|---|---|---|
| 8 | 144.7 tok/s | 155.4 (+7%) | 81.8 | 82.0 |
| 16 | 183.4 | 200.8 (+9%) | 91.4 | 81.7 |
| 32 | 284.3 | 275.3 (−3%) | — | — |
| 64 | 503 errors | 310.9 (after queue fix) | — | 311.0 (after queue fix) |

The 64-in-flight errors came from HAProxy, not the batch size: the backend had no
`timeout queue`, so queued requests failed after 10 s. `timeout queue 30m` fixed it and
stays; everything else was reverted.

**Per-replica batch size, all four nodes (2026-09-23).** Same change on all four, fresh
prompt (15,678 bytes), so comparable only within this table:

| in flight | pool, all baseline | all batch 16, run 1 | all batch 16, run 2 | batch 16 mean vs baseline |
|---|---|---|---|---|
| 8 | 123.9 tok/s | 130.3 (+5.2%) | 135.1 (+9.1%) | +7.1% |
| 16 | 225.8 | 207.8 (−8.0%) | 211.1 (−6.5%) | −7.2% |
| 32 | 309.6 | 310.6 (+0.3%) | 295.5 (−4.6%) | −2.1% |
| 64 | 313.0 | 294.9 (−5.8%) | — | −5.8% |

**Not kept**: the bar was +5% at 16 and 32 in flight. Thirty-two baseline slots already
cover those levels, and the Q8 cache is the likely cost. The rig is at baseline (batch 8,
`maxconn 8`, `timeout queue 30m`), matching `bench/sparks/tabby-config.yml`.

### Measured — real batches through the gateway

Gateway at `127.0.0.1:18002` (tunnelled, bearer key `NEV_MODEL_API_KEY`), vLLM 0.27.1 for
Qwen. Real `bin/real_model_run.rs` batches, `MAX_COMPLETION_TOKENS` 8000:

| model | prompt tokens | output tokens | generation time (s) | output tok/s, batch | output tok/s, per candidate | hit 8000 cap |
|---|---|---|---|---|---|---|
| `qwen3.8-27b` | 4,881 | 16,085 | 147.1 | 109 | 80-139 | 0 of 8 |
| `glm-5.3-flash` | 4,075 | 54,350 | 2,263.4 | 24.0 | 21-36 | 4 of 8 |
| `qwen3.8-27b-fast` | 5,006 | 17,563 | 80.9 | 217.0 | 80.3-114.7 | 1 of 8 |
| `qwen3.8-27b-fast` (32 candidates, concurrency 8, 3 batches, `specs/rig-batch-absolute.md`) | 626,240 | 501,235 | ~687.7 | ~728.7 | 60.4-123.0 | 19 of 96 |
| `qwen3.8-flash-next` (32 candidates, concurrency 32, 5 batches, `specs/rig-batch-absolute.md`) | not reported | not reported | 584.5-599.4 per batch (2,968.2 summed) | not derivable | not derivable | not derivable |

- The first two rows drew sequentially; later rows concurrently (`NEV_CONCURRENCY`,
  default 8; `NEV_CANDIDATES`, default 8). A concurrent batch takes as long as its
  slowest reply: a Qwen batch of 8 drew in about 60 s against 168.0 s summed, a Claude
  batch in about 121 s against 755.2 s summed. Compiling and scoring still run one
  candidate at a time afterwards.
- `qwen3.8-27b-fast`, 32 candidates: generation time is summed latency divided by 8
  (251.6 + 201.2 + 234.9 s); full batch wall 414, 266 and 293 s.
- `qwen3.8-flash-next` returns no `usage` block, so no token figures exist; full batch
  wall 1,158, 1,014, 1,152, 966 and 1,056 s; 75 of 160 compiled, against 10 of 96 at
  `qwen3.8-27b-fast`.
- Null-content replies (a gateway quirk) are excluded from per-candidate ranges.

**Sampling tier — 24 GB consumer cards (modelled, not measured).** Data-parallel replicas
scale near-linearly (~98.5% per card); tensor parallelism over PCIe helps only to two
cards. `per_card_candidates_per_s` ≈ **0.47** (range 0.1–2.2) for a 30B-total/3B-active
mixture-of-experts model; at batch ≥ 24 most experts are touched and the advantage over a
dense model mostly disappears. Dense 14–27B models run 0.7–1.5 candidates/s per card; the
choice between them is about pass rate, not throughput.

**Capacity tier — large unified-memory machines (modelled).**
`per_machine_escalations_per_s` ≈ **0.02** (about 70/hour) for a ~100B/~12B-active model
at an 8k prompt and 1.5k output; prefill dominates.

### Power

```
tier_power_w   = count × watts
total_power_w  = Σ tier_power_w
power_binds    = total_power_w > power_budget_w
```

`bench/capacity.py` computes this from the local inventory. A model, not a control:
nothing enforces the budget. Which hardware the project may hold is decided by
`docs/infrastructure.md` → "Sharing the hardware".

## Combined schedule

Pending: a schedule that keeps every tier busy needs the sampling tier's real candidate
arrival rate first.