burrito.space / bez
This repository has no description
Star
Fork 0
Watch
bez specs job-reference-check.md
12 kB
Markdown
at track/scripted-e2e

enabled: false touches: [jobs/src/checks.rs, jobs/src/manifest.rs, jobs/src/runner.rs, jobs/src/preflight.rs, jobs/src/bin/job-check.rs, jobs/scripts/reference-check.sh, jobs/scripts/pi-writer.sh, jobs/scripts/pi-review.sh, jobs/tests/checks.rs, jobs/tests/manifest.rs, jobs/tests/reference_check.rs, jobs/tests/runner.rs, jobs/tests/preflight.rs, jobs/tests/pi_writer.rs, jobs/tests/pi_review.rs, jobs/fixtures/reference-check/**] depends-on: [job-review-rework] #

job-reference-check: a writer's calls on open spec questions are checked against the browsers #

A spec cannot settle every case, and a writer that stops to ask stalls an unattended job. So the writer decides, and the decision is checked by something other than the writer: this spec adds the workflow a writer follows when the spec leaves a question open (make a best-effort call, check it against the browsers and the spec text, record it) and a script stage that rechecks each call independently, so a call the browsers contradict comes back to the writer as a finding instead of landing.

It builds on specs/job-review-rework.md: the check's stage carries on_fail: "rework", so a contradicted call starts another writer round with the browsers' answers, and rounds spent this way count against the same failed-round allowance.

Where. jobs/src/checks.rs (the new check), jobs/src/manifest.rs CHECKS and the check dispatch in jobs/src/runner.rs, jobs/src/bin/job-check.rs (the same check by hand), and jobs/src/preflight.rs; jobs/scripts/pi-writer.sh and jobs/scripts/pi-review.sh; a new jobs/scripts/reference-check.sh. No bench/ file: the Dagu workflow's tools step already copies every file in jobs/scripts/ into {tools} (bench/dagu/job.yaml, step tools), so the new script reaches a job without a change there.

Manifest. No new field. A job that wants the check adds a script stage after its writer and names the check:

"limits": { "wall_clock_seconds": 5400, "writer_relaunches": 1, "failed_rounds": 2 },
"rework_from": "write",
"stages": [
  { "name": "write", "kind": "writer", "agent": "writer", "command": ["{agent}"] },
  { "name": "test", "kind": "script", "on_fail": "rework",
    "command": ["cargo", "test", "-p", "layout", "--bin", "viewport_units_check"] },
  { "name": "reference", "kind": "script", "on_fail": "rework",
    "command": ["{tools}/reference-check.sh"],
    "checks": [{ "check": "reference", "args": ["{state}/reference-check.json", "{state}/decisions.json"] }] },
  { "name": "measure", "kind": "script",
    "command": ["cargo", "run", "--release", "-p", "layout", "--bin", "viewport_units_check"] },
  { "name": "review", "kind": "review", "agent": "review", "command": ["{agent}"] }
]

The workflow.

  1. Decide. When the spec, and the CSS or HTML text it cites, leave a question open (an edge case, an ambiguous sentence, two readings), the writer picks the reading it thinks right and goes on. The brief in pi-writer.sh says so, in every round.
  2. Record. It appends one entry to <state>/decisions.json (a JSON list; the state directory is outside the checkout, so nothing is committed or has to fit touches): id (letters, digits and -), question, spec (the spec text or URL and anchor the question comes from), choice (the reading taken), probe and sources. probe is { "html": <a document>, "expect": [{ "node": <index in the dump's nodes>, "field": "bounding.x" | "bounding.y" | "bounding.w" | "bounding.h" | "rects.length" | "style.<property>", "value": <number or string> }] }: a page and what the choice predicts the browsers report for it, in the terms of docs/dump-format.md. sources lists anything else consulted (other engines' source or binaries, a spec issue) as strings; it is read by the review and never decides a check.
  3. Check it first. The writer runs its probe through the oracle (node oracle/oracle.mjs, one request with the three engines) before building on the choice, and changes the choice when the reference (below) says something else.
  4. Checked again, independently. A script stage running {tools}/reference-check.sh reads decisions.json, sends each probe once to node oracle/oracle.mjs in the workdir with the engines chromium, firefox and webkit, and writes <state>/reference-check.json: { "format": 1, "decisions": [{ "id", "outcome", "detail", "fields" }] }. fields holds one entry per expectation: node, field, expected, reported (each browser's value), reference and rule.
  5. The reference, per field. Numbers are equal when within 1/60 px, the comparator's tolerance in docs/dump-format.md; strings when identical.
    • When at least two of the three browsers report the same value, that value is the reference, and rule is majority. Of the browsers that agree, Chromium's value is taken when Chromium is among them, else Firefox's, so the reference is one reported value rather than an average.
    • When all three differ, Chromium's value is the reference, and rule is chromium-default. This is the simplest rule that always gives an answer, and it stands until a better one exists; recording which rule set each reference lets the review see where the check leaned on it.
  6. The outcome, per decision. agrees when every expectation matches its reference; disagrees otherwise; unchecked when fewer than three browsers answered or the probe is malformed (not valid HTML the oracle accepts, a node index outside a browser's dump, a field no browser reports), since a reference from fewer than three browsers is not one the rules above define. detail names each field with the expected value, each browser's value, the reference and the rule that set it (majority or chromium-default). The script exits 0 when it wrote the file, including when a decision disagrees, 2 when it could not start the oracle or read decisions.json. No decisions.json, or an empty one, writes { "format": 1, "decisions": [] } and exits 0. The oracle command is JOBS_ORACLE when set (tests use a stub), else node oracle/oracle.mjs.
  7. The check. A new compiled-in check, reference <reference-check.json> <decisions.json> (jobs/src/checks.rs reference(), and the same two arguments to job-check reference by hand): passes when the first file is format 1 and no decision is disagrees; fails, with one message per disagreeing decision naming its id, question and detail, when any is; cannot run when the first file is missing or not JSON. A decision in decisions.json missing from the first file fails too, so a script that skipped one cannot pass; a missing decisions.json counts as an empty list. A pass lists the unchecked decisions and every field whose reference came from chromium-default, for the review.
  8. Back to the writer. The stage carries on_fail: "rework", so a disagreement starts a round (specs/job-review-rework.md rules 1 and 2) whose findings are those messages: the writer gets the browsers' answers and starts again from them. Rounds spent this way count against the failed-round allowance like any other, and a job whose allowance runs out on them waits for a person as any other.

Other engines. Other engines' source and binaries (Servo and Ladybird among them) are evidence, not an oracle: they appear in sources for the review to weigh, and never pass or fail a check, because an engine the project does not verify against can be wrong in the same way the writer is. They are read from NEV_REFERENCE_DIR when set, else $NEV_LANE_SHARED/reference; whether anything is on the share there today is not known (no document under docs/ describes it), so the brief names the directory and says to go on without it when it is missing.

The writer and the review. jobs/scripts/pi-writer.sh adds the workflow above to its brief in every round, and, in a round, allows a finding to be rebutted by a commit message backed by a decision whose reference check agrees. jobs/scripts/pi-review.sh carries <state>/decisions.json and <state>/reference-check.json in its prompt when they exist, asks the review to weigh every unchecked decision and every chromium-default reference on the spec text, and accepts a rebuttal as evidence only when its decision agrees.

Preflight. A stage whose command runs reference-check.sh needs the oracle's Playwright browsers, the same requirement a writer stage has (jobs/src/preflight.rs check_oracle()), and node.

Not built here. A better rule than Chromium's value for three-way disagreement; comparing fields beyond those listed in the probe format; fetching or indexing other engines' source onto the share; amending specs/viewport-units.job.json or any other manifest to add the stage, which is a person's edit under specs/.

Tests. In jobs/tests/, with no lane, model or network:

  • check (jobs/tests/checks.rs): reference passes on agrees and unchecked, and its message lists the unchecked decisions and the chromium-default fields; fails on disagrees with the decision's id in the message, and on a decision absent from the file; cannot run on a missing file.
  • manifest (jobs/tests/manifest.rs): reference with two arguments is a known check; with one it is a violation.
  • reference-check.sh (jobs/tests/reference_check.rs, with JOBS_ORACLE a stub in jobs/fixtures/reference-check/ printing canned responses): three browsers agreeing with the expectation (agrees, rule majority); three agreeing against it (disagrees); two agreeing with the expectation and one not (agrees, rule majority); two agreeing against it (disagrees); Firefox and WebKit agreeing while Chromium differs (the reference is Firefox's value); all three differing, the expectation equal to Chromium's (agrees, rule chromium-default) and equal to another browser's (disagrees); one browser erroring (unchecked); a node index outside the dump (unchecked); a number within 1/60 px (agrees); no decisions.json (empty list, exit 0); and a stub that exits at once (exit 2).
  • runner (jobs/tests/runner.rs): a reference stage with on_fail: "rework" whose check fails once sends the job back with the disagreeing decision's message in the round's findings.
  • pi-writer.sh (jobs/tests/pi_writer.rs, with the existing fake pi): the brief names decisions.json, the probe format and NEV_REFERENCE_DIR.
  • pi-review.sh (jobs/tests/pi_review.rs, with the existing stub pi): the prompt carries decisions.json and reference-check.json when they exist.
  • preflight (jobs/tests/preflight.rs): a stage running reference-check.sh without the oracle's browsers is unmet.

Gates. cargo test -p jobs then cargo test --workspace, on a build lane (bench/on-lane.sh --lane-only), never on the development box. Not a generated function: no model batch, provenance record or floor. Whether a real model writer uses decisions.json well is left to the first job that runs with the stage; none is claimed here.

Files. touches is the check and its dispatch, the preflight, job-check, the new script and the two pi scripts, with their tests and the oracle stub's fixtures. It edits no bench/ file and no existing spec. depends-on names job-review-rework, since the check is useful only through on_fail: "rework", which that spec adds.