diff --git a/plan/harness.md b/plan/harness.md index c4e6b42..df30584 100644 --- a/plan/harness.md +++ b/plan/harness.md @@ -214,6 +214,68 @@ What follows: reduction, budgets in work units, a seeded stream per unit - but the test is "same observation, same decision", not "same seed, same match". +## A running measurement owns its inputs, or it does not own its result + +Three times in one night a file that looked idle to an outside observer was +load-bearing to a process that had not announced itself: + +- A compressor gzipped decision logs that `sds bench`'s corpus save was between + *enumerating* and *opening*. Both ends read `.gz` perfectly well; the save + died on `FileNotFoundError` because the path it held no longer existed. A + time-to-check/time-to-use race with the run's own housekeeping as the other + writer. +- A sweep compressing logs older than five minutes came within a few minutes of + gzipping one that was still being written. `gzip` replaces atomically, so the + result would have been a **complete, valid, truncated** `.gz` - passing + `gzip -t`, reading cleanly, and silently shortening one match's decisions in a + corpus somebody then fitted against. +- A `target/release/` was deleted to reclaim disk while a benchmark was mid-run + on the binary inside it. + +The tempting rule is "look harder before touching". It does not scale and it did +not work: every one of those was done by somebody being careful, using a signal +that was a reasonable proxy for idle - file age, a merged branch, a build +directory. **No amount of looking from outside fixes this. Declaring from inside +does.** A run should name what it owns, and the two ways to survive a violation +are not the same: + +- **Freeze** what must not change, and *fail* if it does. Right for anything the + result depends on - the bot binary, the bridge jar, the weights file, the + scenario suite. A run continued on changed inputs is worse than a run that + stopped, because its numbers are reported as if nothing happened. +- **Rebuild** what can be reconstructed, and *survive*. Right for a build + artefact: a runner that rebuilds a missing binary makes somebody else's wrong + inference cost a minute instead of a window. + +The choice differs by what the artefact is *for*, and a guard that freezes what +somebody thought of while leaving what somebody actually changed is worse than +none, because it gets cited as coverage. The first version of this checksummed +the binary and the jar and would have passed silently through a rebase that +rewrote `weights/` under a live arm. + +## Two errors either side of a line, and only one is worth tooling + +Both happened while co-ordinating one night's work, and the difference decides +where effort belongs. + +`git cherry` reported **2 unlanded commits of 14** for a branch. The number was +correct. It was about a different branch than the one being discussed - the work +had moved to a fresh branch when its predecessor merged - and the branch the +command named genuinely did have 2 of 14. **Nothing in the output could have +flagged it.** A confident, accurate number about the wrong subject is caught by +knowing the subject well enough to notice it does not fit, and by nothing else. + +A pull request reported as **conflicted** was not, in content. A conflict report +is a claim about whether a *patch series* applies to a moved base, not about +whether two changes touch the same lines. The branch changed 11 files, the base +had moved 46, and **the intersection was empty** - including the two files that +would actually have collided. Resolving it blind costs an hour and invites a +judgement call; the file-set intersection settles it in seconds. + +One is uncatchable by tooling and the other was catchable by a check nobody +thought to run. Knowing which is which is what says where to spend the effort, +and the second kind is the one that recurs. + ## A short run measured its suite's alphabetical prefix `bench` picked its scenario with `game % len(scenarios)`, which covers a suite