From f2bb3f0ca31d5e33f600a72b871ae3dabacf9eee Mon Sep 17 00:00:00 2001 From: "@permadeath.com" Date: Sun, 30 Aug 2026 10:37:06 -0400 Subject: [PATCH] docs(harness): a running measurement owns its inputs, or it does not own its result Three times in one night a file that looked idle from outside was load-bearing to a process that had not announced itself: a compressor gzipping logs the corpus save was between enumerating and opening, a sweep within minutes of gzipping a live log into a valid truncated one, and a target/release deleted under a running benchmark. Care did not prevent any of them; every one used a reasonable proxy for idle. Freeze what the result depends on and fail; rebuild what can be reconstructed and survive. A guard that covers the input somebody thought of and not the one somebody changed is worse than none, because it gets cited as coverage. Also records the two coordination errors either side of a line: a correct git cherry count about the wrong branch, which only domain knowledge catches, and a conflict report that was a claim about patches rather than content, which a file-set intersection settles in seconds. Change-Id: I1ce160560a9ca464dfcffb8ec0f6a96cb7cde23b --- plan/harness.md | 62 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 62 insertions(+) diff --git a/plan/harness.md b/plan/harness.md index c4e6b42..df30584 100644 --- a/plan/harness.md +++ b/plan/harness.md @@ -214,6 +214,68 @@ What follows: reduction, budgets in work units, a seeded stream per unit - but the test is "same observation, same decision", not "same seed, same match". +## A running measurement owns its inputs, or it does not own its result + +Three times in one night a file that looked idle to an outside observer was +load-bearing to a process that had not announced itself: + +- A compressor gzipped decision logs that `sds bench`'s corpus save was between + *enumerating* and *opening*. Both ends read `.gz` perfectly well; the save + died on `FileNotFoundError` because the path it held no longer existed. A + time-to-check/time-to-use race with the run's own housekeeping as the other + writer. +- A sweep compressing logs older than five minutes came within a few minutes of + gzipping one that was still being written. `gzip` replaces atomically, so the + result would have been a **complete, valid, truncated** `.gz` - passing + `gzip -t`, reading cleanly, and silently shortening one match's decisions in a + corpus somebody then fitted against. +- A `target/release/` was deleted to reclaim disk while a benchmark was mid-run + on the binary inside it. + +The tempting rule is "look harder before touching". It does not scale and it did +not work: every one of those was done by somebody being careful, using a signal +that was a reasonable proxy for idle - file age, a merged branch, a build +directory. **No amount of looking from outside fixes this. Declaring from inside +does.** A run should name what it owns, and the two ways to survive a violation +are not the same: + +- **Freeze** what must not change, and *fail* if it does. Right for anything the + result depends on - the bot binary, the bridge jar, the weights file, the + scenario suite. A run continued on changed inputs is worse than a run that + stopped, because its numbers are reported as if nothing happened. +- **Rebuild** what can be reconstructed, and *survive*. Right for a build + artefact: a runner that rebuilds a missing binary makes somebody else's wrong + inference cost a minute instead of a window. + +The choice differs by what the artefact is *for*, and a guard that freezes what +somebody thought of while leaving what somebody actually changed is worse than +none, because it gets cited as coverage. The first version of this checksummed +the binary and the jar and would have passed silently through a rebase that +rewrote `weights/` under a live arm. + +## Two errors either side of a line, and only one is worth tooling + +Both happened while co-ordinating one night's work, and the difference decides +where effort belongs. + +`git cherry` reported **2 unlanded commits of 14** for a branch. The number was +correct. It was about a different branch than the one being discussed - the work +had moved to a fresh branch when its predecessor merged - and the branch the +command named genuinely did have 2 of 14. **Nothing in the output could have +flagged it.** A confident, accurate number about the wrong subject is caught by +knowing the subject well enough to notice it does not fit, and by nothing else. + +A pull request reported as **conflicted** was not, in content. A conflict report +is a claim about whether a *patch series* applies to a moved base, not about +whether two changes touch the same lines. The branch changed 11 files, the base +had moved 46, and **the intersection was empty** - including the two files that +would actually have collided. Resolving it blind costs an hour and invites a +judgement call; the file-set intersection settles it in seconds. + +One is uncatchable by tooling and the other was catchable by a check nobody +thought to run. Knowing which is which is what says where to spend the effort, +and the second kind is the one that recurs. + ## A short run measured its suite's alphabetical prefix `bench` picked its scenario with `game % len(scenarios)`, which covers a suite -- 2.51.2