burrito.space / bez
This repository has no description
Star
Fork 0
Watch
bez docs provenance.md
9.0 kB
Markdown
at track/scripted-e2e

Provenance #

The record every generated engine function carries (roadmap.md → "Phase 2 — Generation pipeline" → "Provenance is a first-class artifact, not a comment"). It lets the maintenance lifecycle (roadmap.md → "Ongoing development lifecycle" → "A specification changes") find exactly the functions a spec change affects, without a person reading the diff. Implemented in crates/provenance.

flowchart LR
  change["changed spec section<br/>or definition"] --> norm["normalize + hash"]
  norm --> idx["provenance index<br/>section → files,<br/>definition → files"]
  recs["*.provenance.json<br/>beside each generated file"] --> idx
  webref["webref definitions<br/>(closure)"] --> idx
  idx --> affected["generated functions<br/>to regenerate"]

Record #

One record per generated function, ten fields:

Field Type Meaning
spec_url string The section's URL, anchor included.
section_id string The section identifier ("10.3.3"), stable across the section's own edits so the watcher can key on it.
spec_text_hash hash Of the section's normalized text — see "Normalization".
definition_closure_hash hash Of the transitive closure of definitions the section's text references — see "Definition closure".
spec_level string The one canonical level this algorithm is pinned to ("CSS 2.1"), since CSS carries the same text in several levels (docs/spec-change-flow.md → "One canonical level per algorithm").
spec_revision string A commit hash for a github-sourced spec, or a dated snapshot label; regeneration is debounced to a stable revision (docs/spec-change-flow.md → "Debounce").
model string The model that produced the accepted candidate.
prompt_hash hash Of the exact prompt sent, so a prompt-template change is told apart from a spec change.
sampling_seed u64 The seed of the accepted candidate.
verification struct Cases run, cases passed, and whether the permanent regression set stayed whole.

crates/provenance/src/record.rs — ProvenanceRecord, VerificationResult; JSON on disk.

Normalization #

crates/provenance/src/normalize.rs — normalize_spec_text. Much spec HTML changes nothing about the algorithm: in one three-month window 23 commits touched only <wpt> blocks. The normalizer:

  1. drops <wpt>...</wpt> blocks;
  2. drops elements whose class contains example, note, issue or informative;
  3. drops a trailing "Changes" section (from a heading exactly Changes to the next heading at the same or shallower level);
  4. replaces every remaining tag with a space, decodes &amp; &lt; &gt; &quot; &#39; &nbsp;, and collapses whitespace.

A hand-written tag scan, not an HTML parse: spec fragments are well-formed enough.

Variable-name normalization: <var> names are canonicalized by first-appearance order (canonicalize_variable_names), so a consistent rename (CSS 2.1 §10.3.3's 'avail' → 'containing-block-width') keeps the hash, while a new or moved variable changes it.

Definition closure #

crates/provenance/src/closure.rs — DefinitionIndex, closure_for, closure_hash.

A third of real spec changes alter meaning through a redefined term the algorithm's own text never touches (docs/spec-change-flow.md; for example, html#12421 changed "active document" and 26 call sites with unchanged text changed behaviour). So the record also hashes every definition the section transitively references.

pub trait DefinitionIndex {
    /// Anchor ids of definitions `section_id`'s own text references directly.
    fn references(&self, section_id: &str) -> Vec<String>;
    /// Anchor ids a given definition's own text in turn references.
    fn definition_references(&self, definition_id: &str) -> Vec<String>;
    /// The definition's (already normalized-ready) text, or `None` if this index
    /// doesn't know the id.
    fn definition_text(&self, definition_id: &str) -> Option<String>;
}

closure_for follows references until nothing new appears and returns a BTreeSet. closure_hash normalizes each member, sorts by anchor id and hashes with separators, so the hash depends on membership only, never discovery order.

The production source is webref: crates/provenance/src/webref.rs WebrefIndex reads corpus/webref/definitions.json (gitignored), written by node oracle/suites/webref-dump.mjs from @webref/css listAll() (already the curated data) and @webref/idl parseAll(): {id, kind, name, text, references}, where references come from <type> tokens in a CSS feature's syntax plus its for scope. With @webref/css@8.7.5 and @webref/idl@3.84.0: 5,098 definitions (1,723 CSS features, 3,375 top-level WebIDL names, kept shallow).

Scope boundary. Webref indexes CSS features (properties, types, selectors), not CSS 2.1's prose glossary terms ("used value", "containing block") the four generated functions reference. Those still resolve only through each css21_*.rs Fixture. A real glossary source is unstarted.

Hashing #

crates/provenance/src/hash.rs — Hash::of: SHA-256 (sha2), lowercase hex, serialized as a plain JSON string. Any stable fixed-width fingerprint would do.

Where the record lives #

Beside the generated file: width.rs has width.provenance.json in the same directory. Finding a record is a path transformation, and the record diffs on its own.

The index #

crates/provenance/src/index.rs — ProvenanceIndex::build, from (path, record) pairs and a DefinitionIndex: section_id → paths directly, definition_id → paths by recomputing each record's closure and inverting it (so the record keeps exactly ten fields). crates/provenance/src/loader.rs load_records walks crates/layout/src/generated/ for *.provenance.json, naming each by its sibling file.

cargo run -p provenance --bin provenance_impact -- <id> answers the question against the real records (with WebrefIndex, or a definitions-free stand-in if the dump is missing): -- 10.3.3 finds width.rs.

Live fetch #

crates/provenance/src/fetch.rs. fetch_section(cache_dir, url, anchor) fetches a section and caches its raw HTML under corpus/specs/ (gitignored; keyed by a hash of url#anchor, file name also carries the fetch date). A cached entry is reused whatever its age. extract_section finds the heading carrying the anchor (a nested <a name> or the heading's own id) and takes everything to the next heading at the same or shallower level. load_normalized returns None rather than fetching when nothing is cached, so tests skip cleanly. cargo run -p provenance --bin fetch_specs fills the cache for the four CSS 2.1 generated functions (fetched 2026-09-22):

Function Resolved URL
css21_width https://www.w3.org/TR/CSS21/visudet.html#blockwidth
css21_margin_collapse https://www.w3.org/TR/CSS21/box.html#collapsing-margins
css21_percentage_resolution https://www.w3.org/TR/CSS21/visudet.html#the-width-property
css21_absolute_positioning https://www.w3.org/TR/CSS21/visudet.html#abs-non-replaced-width

Proof against the four hand-trimmed constants. Each live section's normalized hash was compared with the old hand-abbreviated constant's. All four differed, as expected: the constants were elided excerpts, and two folded in extra sections (§10.5 and part of §8.4 into percentage; §9.3.1, §9.3.2 and §10.1 into absolute positioning) that one anchor per function does not.

Function Constant (normalized) hash Live (normalized) hash
css21_width 44ef95f09861892763311add094f0f0496954478937f8571809deb8911bb839a 497e15db083b34a3f140b4a022010b5730f170dab86a8afb5678578c3ea4a3a1
css21_margin_collapse 8f36fe00ecf24e78d919b213293997a6935290a03d3fdbe248655adda2b75968 1023403b87b82ea44a8f7afdc9f166adf3ecc129e68ade7afe662735eb1bb65b
css21_percentage_resolution 60a65367513510a1eb832d656402ccc657ca8ee12fd706e99ccce2d53008b712 f27029ef1561dcbbe8c6b0ec2eb6cbe6e643ea788617f444f34693cdcf54b320
css21_absolute_positioning b726442b4155df72dad350bebbfc0f2cb150891e68354b636f3734ad838f438a 4fec683bbfcb41eaae0e3b64cc3306048ad811c2034825bf3b18d126d682dc4b

Each SPEC_TEXT is now the verbatim fetched HTML, and each file has an idempotent_against_the_cached_live_fetch test comparing it with the cache (skipped when the cache is empty).

What's implemented, what's deferred #

Implemented: the record, normalization with variable canonicalization, closure and hash, the index and loader, provenance_impact, live fetch backing all four generated functions, and the webref-backed DefinitionIndex.

Deferred:

  • A real source for CSS 2.1 glossary terms (the Fixtures stand in meanwhile).
  • Wiring the index into an actual regeneration trigger; the generation loop's admit writes records but nothing yet acts on a detected spec change.