---
enabled: false
touches: [jobs/src/checks.rs, jobs/src/manifest.rs, jobs/src/runner.rs, jobs/src/preflight.rs, jobs/src/bin/job-check.rs, jobs/scripts/reference-check.sh, jobs/scripts/pi-writer.sh, jobs/scripts/pi-review.sh, jobs/tests/checks.rs, jobs/tests/manifest.rs, jobs/tests/reference_check.rs, jobs/tests/runner.rs, jobs/tests/preflight.rs, jobs/tests/pi_writer.rs, jobs/tests/pi_review.rs, jobs/fixtures/reference-check/**]
depends-on: [job-review-rework]
---

# `job-reference-check`: a writer's calls on open spec questions are checked against the browsers

A spec cannot settle every case, and a writer that stops to ask stalls an unattended job. So
the writer decides, and the decision is checked by something other than the writer: this
spec adds the workflow a writer follows when the spec leaves a question open (make a
best-effort call, check it against the browsers and the spec text, record it) and a script
stage that rechecks each call independently, so a call the browsers contradict comes back
to the writer as a finding instead of landing.

It builds on `specs/job-review-rework.md`: the check's stage carries `on_fail: "rework"`, so
a contradicted call starts another writer round with the browsers' answers, and rounds
spent this way count against the same failed-round allowance.

**Where.** `jobs/src/checks.rs` (the new check), `jobs/src/manifest.rs` `CHECKS` and the
check dispatch in `jobs/src/runner.rs`, `jobs/src/bin/job-check.rs` (the same check by
hand), and `jobs/src/preflight.rs`; `jobs/scripts/pi-writer.sh` and
`jobs/scripts/pi-review.sh`; a new `jobs/scripts/reference-check.sh`. No `bench/` file: the
Dagu workflow's `tools` step already copies every file in `jobs/scripts/` into `{tools}`
(`bench/dagu/job.yaml`, step `tools`), so the new script reaches a job without a change
there.

**Manifest.** No new field. A job that wants the check adds a script stage after its writer
and names the check:

```json
"limits": { "wall_clock_seconds": 5400, "writer_relaunches": 1, "failed_rounds": 2 },
"rework_from": "write",
"stages": [
  { "name": "write", "kind": "writer", "agent": "writer", "command": ["{agent}"] },
  { "name": "test", "kind": "script", "on_fail": "rework",
    "command": ["cargo", "test", "-p", "layout", "--bin", "viewport_units_check"] },
  { "name": "reference", "kind": "script", "on_fail": "rework",
    "command": ["{tools}/reference-check.sh"],
    "checks": [{ "check": "reference", "args": ["{state}/reference-check.json", "{state}/decisions.json"] }] },
  { "name": "measure", "kind": "script",
    "command": ["cargo", "run", "--release", "-p", "layout", "--bin", "viewport_units_check"] },
  { "name": "review", "kind": "review", "agent": "review", "command": ["{agent}"] }
]
```

**The workflow.**

1. **Decide.** When the spec, and the CSS or HTML text it cites, leave a question open (an
   edge case, an ambiguous sentence, two readings), the writer picks the reading it thinks
   right and goes on. The brief in `pi-writer.sh` says so, in every round.
2. **Record.** It appends one entry to `<state>/decisions.json` (a JSON list; the state
   directory is outside the checkout, so nothing is committed or has to fit `touches`):
   `id` (letters, digits and `-`), `question`, `spec` (the spec text or URL and anchor the
   question comes from), `choice` (the reading taken), `probe` and `sources`. `probe` is
   `{ "html": <a document>, "expect": [{ "node": <index in the dump's nodes>, "field":
   "bounding.x" | "bounding.y" | "bounding.w" | "bounding.h" | "rects.length" |
   "style.<property>", "value": <number or string> }] }`: a page and what the choice
   predicts the browsers report for it, in the terms of `docs/dump-format.md`. `sources`
   lists anything else consulted (other engines' source or binaries, a spec issue) as
   strings; it is read by the review and never decides a check.
3. **Check it first.** The writer runs its probe through the oracle (`node
   oracle/oracle.mjs`, one request with the three engines) before building on the choice,
   and changes the choice when the reference (below) says something else.
4. **Checked again, independently.** A script stage running `{tools}/reference-check.sh`
   reads `decisions.json`, sends each probe once to `node oracle/oracle.mjs` in the workdir
   with the engines chromium, firefox and webkit, and writes
   `<state>/reference-check.json`: `{ "format": 1, "decisions": [{ "id", "outcome",
   "detail", "fields" }] }`. `fields` holds one entry per expectation: `node`, `field`,
   `expected`, `reported` (each browser's value), `reference` and `rule`.
5. **The reference, per field.** Numbers are equal when within 1/60 px, the comparator's
   tolerance in `docs/dump-format.md`; strings when identical.
   - When at least two of the three browsers report the same value, that value is the
     reference, and `rule` is `majority`. Of the browsers that agree, Chromium's value is
     taken when Chromium is among them, else Firefox's, so the reference is one reported
     value rather than an average.
   - When all three differ, Chromium's value is the reference, and `rule` is
     `chromium-default`. This is the simplest rule that always gives an answer, and it
     stands until a better one exists; recording which rule set each reference lets the
     review see where the check leaned on it.
6. **The outcome, per decision.** `agrees` when every expectation matches its reference;
   `disagrees` otherwise; `unchecked` when fewer than three browsers answered or the probe
   is malformed (not valid HTML the oracle accepts, a node index outside a browser's dump,
   a field no browser reports), since a reference from fewer than three browsers is not
   one the rules above define. `detail` names each field with the expected value, each
   browser's value, the reference and the rule that set it (`majority` or
   `chromium-default`). The script exits 0 when it wrote the file, including when a decision
   disagrees, 2 when it could not start the oracle or read `decisions.json`. No
   `decisions.json`, or an empty one, writes `{ "format": 1, "decisions": [] }` and exits
   0. The oracle command is `JOBS_ORACLE` when set (tests use a stub), else
   `node oracle/oracle.mjs`.
7. **The check.** A new compiled-in check, `reference <reference-check.json>
   <decisions.json>` (`jobs/src/checks.rs` `reference()`, and the same two arguments to
   `job-check reference` by hand): passes when the first file is `format` 1 and no decision
   is `disagrees`; fails, with one message per disagreeing decision naming its `id`,
   `question` and `detail`, when any is; cannot run when the first file is missing or not
   JSON. A decision in `decisions.json` missing from the first file fails too, so a script
   that skipped one cannot pass; a missing `decisions.json` counts as an empty list. A pass
   lists the `unchecked` decisions and every field whose reference came from
   `chromium-default`, for the review.
8. **Back to the writer.** The stage carries `on_fail: "rework"`, so a disagreement starts a
   round (`specs/job-review-rework.md` rules 1 and 2) whose findings are those messages: the
   writer gets the browsers' answers and starts again from them. Rounds spent this way
   count against the failed-round allowance like any other, and a job whose allowance runs
   out on them waits for a person as any other.

**Other engines.** Other engines' source and binaries (Servo and Ladybird among them) are
evidence, not an oracle: they appear in `sources` for the review to weigh, and never pass or
fail a check, because an engine the project does not verify against can be wrong in the
same way the writer is. They are read from `NEV_REFERENCE_DIR` when set, else
`$NEV_LANE_SHARED/reference`; whether anything is on the share there today is not known (no
document under `docs/` describes it), so the brief names the directory and says to go on
without it when it is missing.

**The writer and the review.** `jobs/scripts/pi-writer.sh` adds the workflow above to its
brief in every round, and, in a round, allows a finding to be rebutted by a commit message
backed by a decision whose reference check agrees. `jobs/scripts/pi-review.sh` carries
`<state>/decisions.json` and `<state>/reference-check.json` in its prompt when they exist,
asks the review to weigh every `unchecked` decision and every `chromium-default` reference
on the spec text, and accepts a rebuttal as evidence only when its decision agrees.

**Preflight.** A stage whose command runs `reference-check.sh` needs the oracle's
Playwright browsers, the same requirement a writer stage has (`jobs/src/preflight.rs`
`check_oracle()`), and `node`.

**Not built here.** A better rule than Chromium's value for three-way disagreement;
comparing fields beyond those listed in the probe format; fetching or indexing other
engines' source onto the share; amending `specs/viewport-units.job.json` or any other
manifest to add the stage, which is a person's edit under `specs/`.

**Tests.** In `jobs/tests/`, with no lane, model or network:

- check (`jobs/tests/checks.rs`): `reference` passes on `agrees` and `unchecked`, and its
  message lists the `unchecked` decisions and the `chromium-default` fields; fails on
  `disagrees` with the decision's `id` in the message, and on a decision absent from the
  file; cannot run on a missing file.
- manifest (`jobs/tests/manifest.rs`): `reference` with two arguments is a known check; with
  one it is a violation.
- `reference-check.sh` (`jobs/tests/reference_check.rs`, with `JOBS_ORACLE` a stub in
  `jobs/fixtures/reference-check/` printing canned responses): three browsers agreeing with
  the expectation (`agrees`, rule `majority`); three agreeing against it (`disagrees`); two
  agreeing with the expectation and one not (`agrees`, rule `majority`); two agreeing
  against it (`disagrees`); Firefox and WebKit agreeing while Chromium differs (the
  reference is Firefox's value); all three differing, the expectation equal to Chromium's
  (`agrees`, rule `chromium-default`) and equal to another browser's (`disagrees`); one
  browser erroring (`unchecked`); a node index outside the dump (`unchecked`); a number
  within 1/60 px (`agrees`); no `decisions.json` (empty list, exit 0); and a stub that exits
  at once (exit 2).
- runner (`jobs/tests/runner.rs`): a `reference` stage with `on_fail: "rework"` whose check
  fails once sends the job back with the disagreeing decision's message in the round's
  findings.
- `pi-writer.sh` (`jobs/tests/pi_writer.rs`, with the existing fake pi): the brief names
  `decisions.json`, the probe format and `NEV_REFERENCE_DIR`.
- `pi-review.sh` (`jobs/tests/pi_review.rs`, with the existing stub pi): the prompt carries
  `decisions.json` and `reference-check.json` when they exist.
- preflight (`jobs/tests/preflight.rs`): a stage running `reference-check.sh` without the
  oracle's browsers is unmet.

**Gates.** `cargo test -p jobs` then `cargo test --workspace`, on a build lane
(`bench/on-lane.sh --lane-only`), never on the development box. Not a generated function: no
model batch, provenance record or floor. Whether a real model writer uses `decisions.json`
well is left to the first job that runs with the stage; none is claimed here.

**Files.** `touches` is the check and its dispatch, the preflight, `job-check`, the new
script and the two pi scripts, with their tests and the oracle stub's fixtures. It edits no
`bench/` file and no existing spec. `depends-on` names `job-review-rework`, since the check
is useful only through `on_fail: "rework"`, which that spec adds.