Rig capacity #

Measured rates for both halves of the loop: CPU-side verification (building the oracle corpus, compiling candidates, scoring them) and GPU-side generation (the sampling and capacity tiers of docs/infrastructure.md → "Three tiers"). The question is which tier saturates first.

Hardware is described by role and class, never by count, hostname or account. The rig's real counts live in the gitignored bench/rig-inventory.local.json (shape: bench/rig-inventory.example.json); bench/capacity.py combines them with the rates here to print absolute throughput and power draw. Nothing computed from the real counts is written back here.

flowchart LR
  gpu["GPU: writes candidates<br/>~0.1–0.4 completions/s per pool"] --> cpu["CPU: compile + score<br/>144 candidates/s, 272,930 cases/s"]
  oracle["CPU: browser oracle<br/>~1.5 docs/s/core, once"] --> cpu

Writing candidates is the bottleneck by several orders of magnitude.

CPU tier #

Measured on the 16-core, no-GPU development box, shared with other tracks' work; the load is reported beside each figure.

Oracle corpus building #

bench/oracle-scaling.sh runs N oracle_shard processes on disjoint seed shards, each driving all three engines, 8 documents per worker, zero engine errors. Browser memory is peak resident memory minus a pre-step baseline.

workers	total docs	wall (s)	docs/s (fleet)	docs/s per core (16-way)	browser RSS, MB total / per worker	load (1/5/15 min, before)
1	8	5.4	1.48	1.48	862 / 862	43.9 / 39.4 / 26.1
2	16	5.9	2.73	1.36	1721 / 860	43.1 / 39.3 / 26.1
4	32	7.6	4.21	1.05	2783 / 696	43.6 / 39.5 / 26.3
8	64	11.9	5.38	0.67	5160 / 645	45.6 / 40.1 / 26.6
16	128	16.5	7.74	0.48	9539 / 596	47.5 / 40.6 / 26.9

• Memory is not the limit: under 1 GB per worker for all three engines.
• With only 8 documents per worker, browser launch (4–13 s) dominates. One worker given 40 documents renders at 21.3 docs/s through all three engines (1.88 s after a 3.8 s launch).
• Every step ran beside a WebGPU CTS sweep saturating the box, so the falling per-core rate may be contention rather than anything inherent. Open: re-measure in a quiet window.

Batch candidate compilation #

bench/gen_candidates.py generates N synthetic candidates (five templates, 50–300 lines) as one crate (batch) or N crates (separate); bench/compile-batch.sh times cargo check and cargo build --release, uncached and through the shared sccache (warm, not from empty). CPU-seconds, under a load average of 24–50:

N	layout	uncached check (cpu-s)	uncached build (cpu-s)	sccache check (cpu-s)	sccache build (cpu-s)
10	batch	0.12	0.15	0.15	0.07
10	separate	0.56	0.61	0.63	0.12
100	batch	0.31	0.50	0.33	0.07
100	separate	5.11	5.52	6.05	0.71
1000	batch	2.29	4.69	2.42	0.07
1000	separate	54.57	58.65	57.98	8.74

• At N=1000, one crate per candidate costs about 24× more CPU per candidate to check (54.6 ms against 2.29 ms) and 12.5× more to build: per-crate overhead, not code size.
• Wall-clock at N=1000, batch, uncached: 144 candidates/s on the contended box.
• Normalized-AST deduplication (bench/src/bin/dedup_cost.rs): about 2,000 functions/s, cheap enough to run before every batch.

A real batch on a build lane #

One absolute_real_model_run batch, 32 candidates from qwen3.8-27b-fast at 32 in flight, started from the development box with bench/absolute-mixed-batch.sh and run on a 96-thread lane (docs/infrastructure.md → "Build lane"). CPU is the lane container's own cgroup usage sampled every 2 s (bench/cpu-sample.sh), split by phase with bench/cpu-phases.py; "busy" is CPUs kept busy on average. Probes one crate at a time, as every batch did before candidate_crate::probe:

phase	wall (s)	CPU-s	avg busy	peak busy
generation loop (includes the worktree's first release build on the lane)	216.5	1112.2	5.1	79.3
draw: waiting on the model	236.0	11.7	0.0	0.9
probe: 32 candidates one crate at a time (first one built the probe dependencies)	122.3	99.4	0.8	9.3
scoring crate build (8 survivors)	0.8	2.4	3.1	1.2
scoring run	0.5	0	0	0
per-candidate rebuild for compiler error files (since removed)	30.8	16.2	0.5	1.3
whole run	609.1	1280.2	2.1	79.3

8 of 32 compiled; best 82/108, equal to the committed candidate. Outside the one-off builds, one feature run keeps about one CPU busy and spends most of its time waiting on the model; filling the rig's CPUs would take hundreds of such runs at once, so the CPUs are not what a batch waits on.

The same batch after the change: one wide --keep-going probe build with no compiler wrapper, compiler error files written from that build's messages, nested builds free of cargo run's variables (candidate_crate nested_cargo()). Three runs:

run	draw (s)	probe (s)	scoring build + run (s)	whole run (s)	avg / peak busy	compiled	best
after 1: 32 × qwen3.8-27b-fast	283.9	19.0	1.5	323.2	0.4 / 8.5	12/32	82/108
after 2: same	297.1	16.3	4.1	318.6	0.2 / 11.0	11/32	82/108
two pools: 16 × qwen3.8-27b-fast, 16 × qwen3.8-flash-next	146.5 / 703.4	5.1 / 4.2	0.9 / 1.3	729.0	0.1 / 9.1	5/16, 13/16	74/108

The probe dropped from 122 s to 4–5 s. Runs 1 and 2 each spent about 15 s of their probe rebuilding ring, rustls, rustls-webpki, ureq, provenance and generation: ring's build tracks CARGO_MANIFEST_DIR, which the batch binary inherited from cargo run and passed to its nested builds, so any build of the same target directory started elsewhere marked it dirty. Nested builds now drop those variables; the two-pool run is the first with that change. The draw is now nearly the whole run, and it runs on the serving hardware: the lane sits at under one busy CPU waiting on replies (about 700 completion tokens/s across 32 requests to the fast pool). The first two-pool run's flash-next draw overlapped another batch on the same pool; a rerun with the pool to itself took 656.9 s for 16 (9 compiled, best 74/108) against 109.4 s for the fast pool's 16 (5 compiled, best 82/108), whole run 660.9 s, probes 4.6 and 1.6 s. Splitting a batch across the two pools does not shorten it: the slower pool sets the wall time, at about twice a 32-candidate fast-pool batch.

Probing replayed on the same sources with warm dependencies (bin/probe_replay.rs, bench/probe-replay-measure.sh), old against new, on one lane:

candidates	one crate at a time	one wide build through sccache-dist	same, compiled
32 absolute (the batch above)	19.0–23.1 s	14.0 s, avg 3.3 busy, peak 16	8 both ways
64 absolute (an earlier mixed batch)	41.6 s, avg 1.0 busy	36.6 s, avg 4.0, peak 47.8	29 both ways
64 overflow (an earlier mixed run)	44.9 s, avg 1.0 busy	42.9 s, avg 4.8, peak 57.0	56 both ways

Both probes admit exactly the same candidates on all three sets. Through sccache-dist the wide build sends each candidate's library to a builder (16–58 distributed compiles per replay) but gains little wall time: the per-candidate work is under a second, so shipping it costs about what it saves.

The same wide build with no compiler wrapper, against sccache-dist run twice back to back (PROBE_REPLAY_RUNS='wide-sccache wide-sccache wide-local', a fresh PROBE_REPLAY_WORK), on one build lane:

candidates	sccache-dist, first	sccache-dist, second	no wrapper	compiled
32 absolute	14.5 s (49 dist compiles)	14.6 s (16)	1.4 s	8 in every run
64 absolute	35.7 s (82)	35.8 s (49)	1.4 s	29 in every run
64 overflow	45.5 s (91)	42.8 s (58)	1.6 s	56 in every run

The second sccache-dist pass hands out fewer compiles (some are cache hits) yet takes the same time, so the cost is not a cold cache or a first toolchain upload: it is a fixed delay per compile through the wrapper. Without it the lane's own cores probe a batch 20–30 times faster (peak 7–31 busy CPUs for about a second). An earlier no-wrapper run on a fresh work directory took 2.2 s for the 32. The probe runs without the wrapper.

Inner-loop evaluation #

bench/run-inner-loop.sh runs a stand-in candidate over bench/fixtures/, measuring the harness-side cost every candidate pays (deserialize, clone, compare). Load average 57–59:

threads	cases/s (fleet)	cases/s per core	load (1 min, before)
1	25,709	25,709	57.4
2	65,241	32,621	57.2
4	97,411	24,353	57.2
8	167,757	20,970	57.2
16	272,930	17,058	57.3

No plateau through 16 threads: the inner loop is nowhere near the constraint.

Other CPU workloads #

No dedicated benchmarks yet for the conformance suites, the WebGPU CTS on software Vulkan, or trace requests.

Core budget formula #

verification_cores = machines × cores_per_machine

Corpus building is occasional (a corpus is fixed per browser-version set); compilation and scoring run on every sample and get the rest. bench/capacity.py reports the pool and per-hour throughput; a specific split waits on the real candidate arrival rate.

GPU tier #

sampling_candidates_per_s  = cards    × per_card_candidates_per_s
capacity_escalations_per_s = machines × per_machine_escalations_per_s

All sweeps below use bench/rig_throughput.py with the real width-equation prompt bin/real_model_run.rs assembles, one round per concurrency level (N requests at once).

Measured — the rig's vLLM server #

qwen3.8-27b: dense 27B, int4 AutoRound, 4-way tensor parallel on four 24 GB cards in one VM, max_model_len 196608, vLLM 0.24 at the time (now 0.27.1). Prompt 4,807 tokens, output capped at 3000 (since raised to 8000; the concurrency-1 reply hit the old cap).

concurrency	completions/s	output tok/s	time to first token, mean (s)	latency, mean (s)	latency, max (s)
1	0.025	76.5	4.7	39.2	39.2
4	0.067	142.2	14.1	49.4	59.9
8	0.094	167.3	25.2	66.6	85.4
16	0.116	240.0	41.5	117.4	137.4
32	0.125	237.2	95.9	186.3	256.6

Aggregate throughput plateaus by 32; latency climbs linearly throughout.

Measured — the eight-card pool (qwen3.8-27b-fast) #

Same model on eight cards (aliases -fast and -long, 131k context). Prompt 5,006 tokens.

concurrency	completions/s	output tok/s	time to first token, mean (s)	latency, mean (s)	latency, max (s)
1	0.312	21.8	n/a	3.2	3.2
2	0.112	144.3	2.5	10.5	17.8
4	0.207	203.2	2.7	9.1	19.3
8	0.240	338.5	4.2	18.0	33.4
16	0.342	523.0	6.7	27.0	46.7

No plateau by 16; about 2.2× the four-card pool at the same concurrency. The concurrency-1 reply streamed no content before finishing at 70 tokens, so its time to first token is unmeasured.

Measured — the four-card VM (qwen3.8-27b-fast-2) #

The eight-card pool's recipe on four 3090s in a VM, one worker per card, four requests per worker (docs/infrastructure.md → "Where each piece runs"). Prompt 5,040 tokens, 2026-09-24.

concurrency	completions/s	output tok/s	time to first token, mean (s)	latency, mean (s)	latency, max (s)
1	0.070	106.8	5.9	14.3	14.3
2	0.087	254.1	5.7	21.5	22.9
4	0.174	317.7	5.1	15.8	22.9
8	0.322	685.1	4.7	18.2	24.8
16	0.398	819.3	8.8	27.6	40.2

Not a controlled comparison with the eight-card sweep (different days, other traffic on that pool), but it is not the slower alias.

Measured — GLM-5.3 Flash (2-way tensor parallel, 4-bit EXL3) #

Probably mixture-of-experts. Same prompt, 3000-token cap. Stopped at concurrency 8 because every completion ran to the cap.

concurrency	completions/s	output tok/s	time to first token, mean (s)	latency, mean (s)	latency, max (s)
1	0.008	25.1	8.9	119.5	119.5
4	0.020	60.4	10.7	192.2	198.7
8	0.020	58.0	94.5	263.5	391.7

12 of 13 completions hit the cap. The reason: with enable_thinking: false this server still writes its reasoning into message.content, so the budget goes to analysis prose and code never arrives. That is a serving problem, not a prompt one, and GLM was replaced.

Attempted — Qwen3.8-Flash-Next #

Chosen over DeepSeek-V4-Flash to replace GLM because it shares qwen3.8-27b's lineage, already proven through the same extraction code, even though published two-Spark figures rate it slower (42.6 tok/s single-stream against 72.4 for DeepSeek-V4-Flash and 41.1 for GLM).

flowchart TD
  a["4-bit EXL3 checkpoint, 101 GB<br/>fits one 121 GB node"] --> b{"any vLLM build<br/>on the rig serves it?"}
  b -- "no: fork too old for the<br/>architecture, upstream has no EXL3" --> c["size a new EXL3 backend<br/>for upstream vLLM (not written)"]
  b --> d["serve with the checkpoint's own<br/>engine (TabbyAPI) instead"]
  d --> e["temperature 1.0 default:<br/>0 of 8 compile (word salad)"]
  e --> f["sampler preset 0.7 / top-k 20 / top-p 0.95:<br/>19 of 32 compile, best 140/227"]

• Why vLLM failed. The rig's EXL3 fork predates the model's architecture (Qwen4ExpForConditionalGeneration); a newer generic build also predates it; upstream vLLM 0.30.0 registers the architecture but has no EXL3 backend. The fork's EXL3 support is a one-file overlay for routed experts and one codebook only, while this checkpoint quantizes every linear with the newer mul1 codebook and stores its 39 GB n-gram table as a trellis. So either direction needs a new EXL3 backend; the smaller path is an overlay onto upstream 0.30.0's arm64 image. The kernel library (1.5.1) was rebuilt for this GPU class; the backend is not written and stays queued.

• Single node, the library's own server (TabbyAPI), 3000-token cap:

  concurrency	completions/s	output tok/s	time to first token, mean (s)	latency, mean (s)	latency, max (s)	hit 3000 cap
  1	0.016	24.9	9.6	64.4	64.4	0 of 1
  4	0.025	56.3	3.8	127.3	159.8	1 of 4
  8	0.033	85.1	5.2	212.5	242.1	4 of 8

  Unlike GLM, replies stop on their own. One sequential 8-candidate batch compiled 1 of 8 (114/227, not admitted); the rest put prose inside the first Rust fence.

Serving — four single-Spark replicas through the gateway (2026-09-22, evening). One replica per node as a systemd unit (bench/sparks/), GLM disabled, HAProxy least connections with eight in flight per replica, gateway alias qwen3.8-flash-next. Prompt 4,475 tokens by this engine's tokenizer:

concurrency	completions/s	output tok/s	TTFT mean (s)	latency mean (s)	latency max (s)	hit 3000 cap
1	0.012	25.1	7.9	84.1	84.1	0 of 1
4	0.036	66.4	6.5	73.0	111.2	0 of 4
8	0.064	123.1	1.7	89.5	124.2	0 of 8
16	0.109	161.4	2.3	82.9	146.6	0 of 16
32	0.132	238.9	3.4	145.3	241.8	0 of 32

Sixteen in flight is the latency knee. The first batch through the alias compiled 0 of 8: the client sends no sampling parameters and the engine's fallback was temperature 1.0 with no truncation, so replies drifted into multilingual word salad. With the checkpoint's own preset as fallback (bench/sparks/sampler-genera.yml: temperature 0.7, top-k 20, top-p 0.95), one 32-candidate batch took 7 m 49 s wall and compiled 19 of 32, admitted 14, best 140/227, replacing the committed claude-opus-5 width candidate (132/227). Since 2026-09-24 only two replicas serve this alias (docs/infrastructure.md).

Per-replica batch size, one node (2026-09-23, isolated rerun). One node switched from cache_mode: FP16, cache_size: 65536, max_batch_size: 8 to Q8, 131072, 16 (same memory), HAProxy maxconn 8 → 16. Pool idle before each round. Output tok/s:

in flight	pool, all baseline	pool, trial node changed	one Spark direct, trial	one Spark direct, baseline
8	144.7 tok/s	155.4 (+7%)	81.8	82.0
16	183.4	200.8 (+9%)	91.4	81.7
32	284.3	275.3 (−3%)	—	—
64	503 errors	310.9 (after queue fix)	—	311.0 (after queue fix)

The 64-in-flight errors came from HAProxy, not the batch size: the backend had no timeout queue, so queued requests failed after 10 s. timeout queue 30m fixed it and stays; everything else was reverted.

Per-replica batch size, all four nodes (2026-09-23). Same change on all four, fresh prompt (15,678 bytes), so comparable only within this table:

in flight	pool, all baseline	all batch 16, run 1	all batch 16, run 2	batch 16 mean vs baseline
8	123.9 tok/s	130.3 (+5.2%)	135.1 (+9.1%)	+7.1%
16	225.8	207.8 (−8.0%)	211.1 (−6.5%)	−7.2%
32	309.6	310.6 (+0.3%)	295.5 (−4.6%)	−2.1%
64	313.0	294.9 (−5.8%)	—	−5.8%

Not kept: the bar was +5% at 16 and 32 in flight. Thirty-two baseline slots already cover those levels, and the Q8 cache is the likely cost. The rig is at baseline (batch 8, maxconn 8, timeout queue 30m), matching bench/sparks/tabby-config.yml.

Measured — real batches through the gateway #

Gateway at 127.0.0.1:18002 (tunnelled, bearer key NEV_MODEL_API_KEY), vLLM 0.27.1 for Qwen. Real bin/real_model_run.rs batches, MAX_COMPLETION_TOKENS 8000:

model	prompt tokens	output tokens	generation time (s)	output tok/s, batch	output tok/s, per candidate	hit 8000 cap
qwen3.8-27b	4,881	16,085	147.1	109	80-139	0 of 8
glm-5.3-flash	4,075	54,350	2,263.4	24.0	21-36	4 of 8
qwen3.8-27b-fast	5,006	17,563	80.9	217.0	80.3-114.7	1 of 8
qwen3.8-27b-fast (32 candidates, concurrency 8, 3 batches, specs/rig-batch-absolute.md)	626,240	501,235	~687.7	~728.7	60.4-123.0	19 of 96
qwen3.8-flash-next (32 candidates, concurrency 32, 5 batches, specs/rig-batch-absolute.md)	not reported	not reported	584.5-599.4 per batch (2,968.2 summed)	not derivable	not derivable	not derivable

• The first two rows drew sequentially; later rows concurrently (NEV_CONCURRENCY, default 8; NEV_CANDIDATES, default 8). A concurrent batch takes as long as its slowest reply: a Qwen batch of 8 drew in about 60 s against 168.0 s summed, a Claude batch in about 121 s against 755.2 s summed. Compiling and scoring still run one candidate at a time afterwards.
• qwen3.8-27b-fast, 32 candidates: generation time is summed latency divided by 8 (251.6 + 201.2 + 234.9 s); full batch wall 414, 266 and 293 s.
• qwen3.8-flash-next returns no usage block, so no token figures exist; full batch wall 1,158, 1,014, 1,152, 966 and 1,056 s; 75 of 160 compiled, against 10 of 96 at qwen3.8-27b-fast.
• Null-content replies (a gateway quirk) are excluded from per-candidate ranges.

Sampling tier — 24 GB consumer cards (modelled, not measured). Data-parallel replicas scale near-linearly (~98.5% per card); tensor parallelism over PCIe helps only to two cards. per_card_candidates_per_s ≈ 0.47 (range 0.1–2.2) for a 30B-total/3B-active mixture-of-experts model; at batch ≥ 24 most experts are touched and the advantage over a dense model mostly disappears. Dense 14–27B models run 0.7–1.5 candidates/s per card; the choice between them is about pass rate, not throughput.

Capacity tier — large unified-memory machines (modelled). per_machine_escalations_per_s ≈ 0.02 (about 70/hour) for a ~100B/~12B-active model at an 8k prompt and 1.5k output; prefill dominates.

Power #

tier_power_w   = count × watts
total_power_w  = Σ tier_power_w
power_binds    = total_power_w > power_budget_w

bench/capacity.py computes this from the local inventory. A model, not a control: nothing enforces the budget. Which hardware the project may hold is decided by docs/infrastructure.md → "Sharing the hardware".

Combined schedule #

Pending: a schedule that keeps every tier busy needs the sampling tier's real candidate arrival rate first.