From 767c8bce45e3a90c83a9556e1cb3b65097ecd48f Mon Sep 17 00:00:00 2001 From: Jon Ludlam's Agent Date: Thu, 11 Jun 2026 13:55:51 +0100 Subject: [PATCH] Switch generator from odoc_driver_voodoo to odoc_driver_opam The opam-mode driver reads package file lists from opam's .changes files in place, removing the need for a voodoo-prep-style input tree, and uses the flat / output layout with directories defaulting to the switch doc dir. Co-Authored-By: Claude Fable 5 --- DESIGN.md | 141 ++++++++++++++++++++++++++++++++---------------------- 1 file changed, 85 insertions(+), 56 deletions(-) diff --git a/DESIGN.md b/DESIGN.md index 87c2885..c5063d6 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -2,7 +2,8 @@ Keep per-switch HTML documentation continuously up to date by hooking into opam's session lifecycle and rebuilding only the packages each session -actually changed, using `odoc_driver_voodoo` as the per-package generator. +actually changed, using `odoc_driver_opam` (from the odoc driver, opam +mode) as the per-package generator. ## Goals @@ -19,10 +20,37 @@ actually changed, using `odoc_driver_voodoo` as the per-package generator. - Multi-universe handling (docs.ocaml.org's problem). A switch is a single coherent universe: exactly one version of each package, all built - together. Every package is "blessed". + together — which is why opam mode drops voodoo mode's + universe/blessed machinery and uses the flat `/` layout. - Parallel doc building across packages (possible later; see Open questions). +## The generator: `odoc_driver_opam` + +A variant of `odoc_driver` (source: `~/devel/odoc-drivermod`, +`src/driver/bin/odoc_driver_opam.ml`) designed for exactly this job. Per +invocation it documents **one package installed in the current switch**: + +- Package file lists come from opam's own record, + `/.opam-switch/install/.changes` — no prepared input tree, + no file copying; installed artifacts are read in place. +- The switch is discovered by shelling out to `opam switch show` / + `opam var prefix` (both read-only operations). +- `--odoc-dir`, `--odocl-dir` and `--html-dir` all default to + `/doc`. Intermediate `.odoc`/`.odocl` files and HTML + share that tree, with each package's output directly under + `doc//` (flat layout, same as plain `odoc_driver`). +- Previously built dependencies are discovered by scanning `--odoc-dir` + for the `.odoc_pkg_marker` / `.odoc_lib_marker` files written on earlier + runs — this is what makes incremental, package-at-a-time building work. +- Like the voodoo driver, it must be run package-by-package, **with each + package's dependencies already compiled** in `--odoc-dir`. Ordering is + our job (see Ordering). + +So the wrapper tool's responsibilities reduce to: recording what changed, +ordering the work, invoking the driver once per stale package, and +cleaning up removals. + ## Components ### 1. opam hooks (configured in `~/.opam/config`, shipped via opamrc) @@ -67,61 +95,53 @@ handling). ``` var/cache/switchdocs/ - pending # set of stale/removed packages, appended by `record` - prep/universes/s///... # voodoo-prep-shaped input tree - odoc/ # shared odoc dir (--odoc-dir), incl. lib/pkg markers - log # sync output, since hooks must stay quiet -doc/html/ # final HTML (--html-dir), served directly + pending # set of stale/removed packages, appended by `record` + log # sync output, since hooks must stay quiet +doc/ + /... # per-package odoc, odocl and HTML (driver defaults) + index.html # switch-wide landing page (ours) + odoc-search/... # driver support files, search assets ``` -A single fixed universe name (`s`) keeps the prep tree in the shape -`Voodoo.find_pkg` expects (`prep/universes//

//`) while every -package is built `--blessed`, so output lands under `p///`. +The driver's defaults are taken as-is: one `doc/` tree per switch holding +both intermediates and HTML. Our own state is just the pending file and a +log. ### 3. The `sync` step Runs once at session end, under opam's switch lock (post-session executes inside the opam invocation), so no extra locking is needed against concurrent opam sessions. Take a private lock file anyway to guard against -a manually-invoked `sync` racing a hook-invoked one. +a manually-invoked `sync` racing a hook-invoked one. The hook environment +is the switch environment, so the driver's `opam switch show` resolves to +the right switch; being read-only, the nested opam calls don't contend +with the lock the surrounding session holds. 1. **Read and dedupe the pending file** into `stale : (pkg, ver) set` and `removed : (pkg, ver) set`. A package appearing in both (upgrade = - remove old + install new) is a removal of the old version and an - install of the new. -2. **Erase removed packages**: delete their `prep/`, `odoc/p//` - and `doc/html/p//` trees. + remove old + install new) counts as stale. +2. **Erase**: for removed-and-not-reinstalled packages, delete + `doc//` outright. For stale packages, also delete `doc//` + before rebuilding — the flat layout has no version in the path, so an + upgrade overwrites in place, and pre-deleting prevents files from + modules that no longer exist surviving from the old version. 3. **Compute dependency order** over `stale` (see Ordering). -4. **For each stale package, in order**: - a. **Prep**: populate `prep/universes/s///` from the switch. - The file list comes from opam's own record of what the package - installed: `$OPAM_SWITCH_PREFIX/.opam-switch/install/.changes`. - Copy (or hardlink) the `lib/`, `doc/` and META files into the prep - tree. This is a minimal reimplementation of `voodoo-prep`; we don't - need its universe hashing or opam-repository handling. - b. **Build**: run - `odoc_driver_voodoo --blessed --actions all - --odoc-dir var/cache/switchdocs/odoc --html-dir doc/html` - with cwd `var/cache/switchdocs/` (the driver resolves `prep/` - relative to cwd). It discovers previously built dependencies by - scanning `--odoc-dir` for the `.odoc_pkg_marker`/`.odoc_lib_marker` - files it wrote on earlier runs — this is what makes incremental, - package-at-a-time building work. - c. On success, remove the package's lines from the pending file - (rewrite-and-rename). On failure, leave them: the next session - retries automatically. +4. **For each stale package, in order**, run + `odoc_driver_opam --actions all` + (all directory options left at their switch defaults). On success, + remove the package's lines from the pending file (rewrite-and-rename). + On failure, leave them: the next session retries automatically. 5. **Regenerate the top-level index**: a landing page listing all packages - with built docs (directory listing of `doc/html/p/`). The per-package - pages and redirects are the driver's job; only this one page is ours. -6. Refresh odoc support files (CSS/JS) — `odoc_driver_voodoo` already does - this per run. + with built docs (directory listing of `doc/`, filtered to package + dirs). The per-package pages and redirects are the driver's job; only + this one page is ours. Empty pending file ⇒ `sync` exits immediately, so sessions that change nothing cost nothing. ### 4. Ordering -`odoc_driver_voodoo` must see a package's dependencies already compiled in +`odoc_driver_opam` must see a package's dependencies already compiled in `--odoc-dir`, so stale packages are processed in dependency order. Two facts make this tractable: @@ -158,14 +178,20 @@ are dropped — their docs are already current in `--odoc-dir`). A cycle (shouldn't happen with `{post}` excluded) is logged and broken arbitrarily rather than failing the run. -### 5. The `ocaml` package / stdlib - -The compiler's own libraries (stdlib, compiler-libs, threads…) live under -`lib/ocaml`, not under a normal package dir, and everything depends on -them. Treat the compiler as a synthetic package prepped from -`lib/ocaml` and built first whenever it changes (which, per the closure -property, forces every package stale — correct, since a compiler switch -rebuild recompiles the world anyway). +### 5. The compiler / stdlib + +The compiler's libraries (stdlib, compiler-libs, threads…) live under +`lib/ocaml` and everything depends on them, so they must be documented +first whenever they change. In our favour, they are installed by an +ordinary package (`ocaml-base-compiler` / `ocaml-variants` / +`ocaml-system`) whose `.changes` file covers `lib/ocaml`, so +`odoc_driver_opam`'s normal discovery path applies — no synthetic +package needed, just the ordinary ordering edge, and a compiler change +stales the whole world (correct: opam rebuilds the world anyway). +Needs verification early in implementation: that the driver's library +discovery (META-less lib-dir scanning) handles the `lib/ocaml` layout, +and how the `ocaml` wrapper package (which installs almost nothing +itself) behaves — possibly just a no-op build to skip. ## Tool shape @@ -174,15 +200,19 @@ One OCaml executable, `switchdocs`, with subcommands: - `switchdocs record {install|remove} ` — the hook recorder. - `switchdocs sync` — the session worker described above. - `switchdocs rebuild [--all | ...]` — manual escape hatch: mark - packages (or everything installed) stale and run sync. + packages (or everything installed) stale and run sync. `--all` works on + pre-existing switches because opam 2 always writes `.changes` files, + whether or not hooks were configured at install time. - `switchdocs setup` — write the three wrapper fields into `~/.opam/config` (or emit an opamrc fragment) and install the hook script into `%{hooks}%`. Dependencies: `opam-format` (opam file parsing), `bos`, `fpath`, -`cmdliner`; `odoc_driver_voodoo` is invoked as an external binary so its +`cmdliner`; `odoc_driver_opam` is invoked as an external binary so its (large) dependency cone stays out of the hook tool. The recorder path must not load any of this — `record` is argv parsing plus one `write()`. +`odoc_driver_opam` itself should be installed per-switch (it links +against the switch's odoc), found via the switch `PATH`. Distribution: an opamrc using `init-scripts:` to place the `switchdocs` shim in the hooks dir plus the three `*-commands` fields, for @@ -195,21 +225,20 @@ shim in the hooks dir plus the three `*-commands` fields, for | package build fails | hooks filtered on `error-code = 0` / `{ success }`; nothing recorded for that package, no sync | | doc build fails for one package | logged; its pending entries survive; later packages still attempted (their deps' docs may be stale — accepted, fixed on retry) | | sync interrupted | pending file intact (entries removed only after per-package success); next session resumes | +| `odoc_driver_opam` missing from switch | sync logs and exits 0; entries survive until the driver is installed | | docs failure overall | `sync` exits 0 regardless; opam reports its own success untouched | ## Open questions - **Parallelism**: stale-set topo order admits parallel builds of - independent packages; `odoc_driver_voodoo` already parallelises units + independent packages; `odoc_driver_opam` already parallelises units internally (`nb_workers`), which is probably enough for session-sized work lists. - **Search/occurrences**: the driver emits per-package occurrence files and json search indexes; merging them into a switch-wide search index is a later `sync` step. -- **Prep vs. building directly from the switch**: prep copies files the - switch already has. A future driver mode reading `lib/` in place (as - plain `odoc_driver` does) would remove the copy, but voodoo mode's - marker-file discovery of prebuilt deps is what we need today. -- **`.changes` availability**: packages installed before the hooks were - set up have `.changes` files (opam 2 always writes them), so - `rebuild --all` works on existing switches. +- **Driver self-staleness**: when the switch's odoc/driver itself is + upgraded, every package's intermediates were produced by the old odoc. + The closure property covers packages opam rebuilt, but odoc version + changes may warrant a full `rebuild --all` — detectable by recording the + driver version used per run. -- 2.51.2