diff --git a/experimental/owl/README.md b/experimental/owl/README.md index fb3ece1..923a3b9 100644 --- a/experimental/owl/README.md +++ b/experimental/owl/README.md @@ -15,15 +15,17 @@ Netlify, GitHub Pages, S3, …). Two decoupled components, composed by the flake: ``` -checkout ─▶ owl-filter (Rust/globset) ─▶ pruned tree ─▶ owl-render (Astro/Shiki) ─▶ dist/ - applies owl.fileset.txt one page per file, two views +checkout ─▶ fileset (Rust/globset) ─▶ pruned tree ─▶ owl-render (Astro/Shiki) ─▶ dist/ + applies owl.fileset.txt one page per file, two views ``` -- **`filter/`** — `owl-filter`, a Rust CLI that applies an `owl.fileset.txt` - manifest and copies the *included* files into an output directory. This is a - Nix-level **pre-filter**: excluded paths (notably `secrets/`) never reach the - renderer. See the top of `owl.fileset.txt` (at the checkout root) for the - manifest format — globs parsed by the `globset` crate, `!` negation, `@import`. +- **`fileset`** — a general-purpose Rust CLI + library (its own flake at + `../fileset`) that applies an `owl.fileset.txt` manifest and copies the *included* + files into an output directory. This is owl's Nix-level **pre-filter**: excluded + paths (notably `secrets/`) never reach the renderer. owl keeps no Rust of its own — + it consumes `fileset` as a `path:../fileset` flake input and uses only its binary. + See `../fileset/README.md` for the manifest format — globs parsed by the `globset` + crate, `!` negation, `@import`. - **`web/`** — `owl-render`, a standalone binary wrapping the Astro renderer. It takes the pruned tree as a **runtime argument** (`owl-render `) and drives Astro's programmatic `build()` — so one build artifact renders any tree, @@ -37,7 +39,7 @@ checkout ─▶ owl-filter (Rust/globset) ─▶ pruned tree ─▶ owl-render ( ## Build / run -Commands run from `experimental/owl/`. `owl-filter` always walks the on-disk +Commands run from `experimental/owl/`. `fileset` always walks the on-disk checkout and applies `owl.fileset.txt`; the renderer renders whatever pruned tree it is handed. Two ways to render, sharing one Astro codebase: @@ -49,14 +51,14 @@ is handed. Two ways to render, sharing one Astro codebase: (with `$OWL_TITLE`), for hot-reloading owl's *own* renderer. `gen-manifest.mjs` reads those two env vars; `owl-render` takes them as `` and `--title`. -### 1. Development — `owl-filter` + `npm run dev` +### 1. Development — `fileset` + `npm run dev` The fast inner loop: a hermetic pre-filter feeding Astro's live dev server (HMR). `npm` needs `node` (e.g. `nix develop`, or `nix shell nixpkgs#nodejs`). ```bash -# pre-filter a checkout (or any tree) into a pruned dir -nix run .#owl-filter -- --fileset ../../owl.fileset.txt ../.. /tmp/owl-out +# pre-filter a checkout (or any tree) into a pruned dir (fileset is its own flake) +nix run ../fileset -- --fileset ../../owl.fileset.txt ../.. /tmp/owl-out # live server with hot reload — re-run the filter when browsed files change cd web && OWL_INPUT_DIR=/tmp/owl-out OWL_TITLE=everything npm run dev # http://localhost:4321 ``` @@ -67,8 +69,8 @@ dev` above — renderer hot-reloads, content frozen — the everyday choice for on owl; `run-owl-for-owl-development-with-dynamic-fileset.sh` adds a watcher that re-filters on every repo change, so new/changed/deleted files show up live. -Iterating on the filter itself? `cargo run --manifest-path filter/Cargo.toml --` -rebuilds faster than `nix run`. +Iterating on the filter itself? It lives in the `fileset` crate now: +`cargo run --manifest-path ../fileset/Cargo.toml --` rebuilds faster than `nix run`. > **Do not use `npm run build`.** It is a non-hermetic, non-development static > build (local `node`/npm, no fileset pre-filter, no reproducibility) with no use @@ -109,19 +111,21 @@ after the first re-renders only changed pages), re-runs both on any repo change `watchexec` watcher, and serves the result. Use it to browse a feature in progress; see the script header for details. -### 3. Standalone binaries — `owl-filter` and `owl-render` +### 3. Standalone binaries — `fileset` and `owl-render` ```bash -nix run .#owl-filter -- --fileset # prune a checkout +nix run ../fileset -- --fileset # prune a checkout nix run .#owl-render -- [--incremental] [--title T] # render a pruned tree -nix build .#owl-filter # -> result/bin/owl-filter +nix build ../fileset # -> result/bin/fileset nix build .#owl-render # -> result/bin/owl-render ``` -`owl-render` renders a **pre-filtered** tree (it does no filtering) — a full build by -default; `--incremental` + `--work-dir D` reuse a persistent cache to re-render only -changed pages. The three scripts call `owl-filter` internally to prune the tree they -render; `run-owl-for-general-development.sh` also drives `owl-render`. +The pre-filter (`fileset`) is a **separate, general-purpose flake** at `../fileset`; +owl ships only `owl-render`. `owl-render` renders a **pre-filtered** tree (it does no +filtering) — a full build by default; `--incremental` + `--work-dir D` reuse a +persistent cache to re-render only changed pages. The three scripts call `fileset` +internally to prune the tree they render; `run-owl-for-general-development.sh` also +drives `owl-render`. ### 4. As a library in another flake @@ -132,18 +136,19 @@ site, bypassing owl's own `everything` input: # `title` is optional (default "owl") — the site name shown in owl's UI. owl.lib.${system}.renderCheckout { src = ./some-checkout; title = "my-repo"; } # filter + render owl.lib.${system}.renderTree { tree = pruned-store-path; title = "my-repo"; } # render a pre-filtered tree (runs owl-render) -owl.lib.${system}.filterTree { src = ...; fileset = ...; } # just the pre-filter -owl.packages.${system}.owl-filter # the filter binary +owl.lib.${system}.filterTree { src = ...; fileset = ...; } # just the pre-filter (runs the fileset binary) owl.packages.${system}.owl-render # the renderer binary +fileset.packages.${system}.default # the pre-filter binary (its own flake) ``` **Scripts:** none — this path is for other flakes consuming owl, not local dev. Regenerate `npmDepsHash` on `web/package-lock.json` changes: -`nix run nixpkgs#prefetch-npm-deps -- web/package-lock.json`. `Cargo.lock` and -`flake.lock` must stay committed (crane needs the lock for reproducible builds). -The renderer needs **Astro 7+** (for `experimental.incrementalBuild` + `getStaticPaths` -`cacheKey`); `renderTree`/`.#site` disable incremental so the build is always complete. +`nix run nixpkgs#prefetch-npm-deps -- web/package-lock.json`. owl's `flake.lock` must +stay committed; the `fileset` crate keeps its own `Cargo.lock` + `flake.lock` (crane +needs the lock for reproducible builds). The renderer needs **Astro 7+** (for +`experimental.incrementalBuild` + `getStaticPaths` `cacheKey`); `renderTree`/`.#site` +disable incremental so the build is always complete. ## Notes & limitations (v1) @@ -156,5 +161,5 @@ The renderer needs **Astro 7+** (for `experimental.incrementalBuild` + `getStati only a pure content edit gets the single-page fast path. - Deferred to later: tree-sitter + symbol cross-refs / jump-to-definition, Pagefind search, git blame/history, request-sending functions. -- `Cargo.lock` and `flake.lock` must be committed (crane needs the lock for - reproducible builds). +- owl's `flake.lock` (and the `fileset` crate's `Cargo.lock` + `flake.lock`) must be + committed for reproducible builds (crane needs the lock). diff --git a/experimental/owl/filter/Cargo.lock b/experimental/owl/filter/Cargo.lock deleted file mode 100644 index d634264..0000000 --- a/experimental/owl/filter/Cargo.lock +++ /dev/null @@ -1,304 +0,0 @@ -# This file is automatically @generated by Cargo. -# It is not intended for manual editing. -version = 4 - -[[package]] -name = "aho-corasick" -version = "1.1.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c982642fa9e8606056828ee9a8505737230110bb1099153c79efe865c59d12ba" -dependencies = [ - "memchr", -] - -[[package]] -name = "anstream" -version = "1.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "824a212faf96e9acacdbd09febd34438f8f711fb84e09a8916013cd7815ca28d" -dependencies = [ - "anstyle", - "anstyle-parse", - "anstyle-query", - "anstyle-wincon", - "colorchoice", - "is_terminal_polyfill", - "utf8parse", -] - -[[package]] -name = "anstyle" -version = "1.0.14" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "940b3a0ca603d1eade50a4846a2afffd5ef57a9feac2c0e2ec2e14f9ead76000" - -[[package]] -name = "anstyle-parse" -version = "1.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "52ce7f38b242319f7cabaa6813055467063ecdc9d355bbb4ce0c68908cd8130e" -dependencies = [ - "utf8parse", -] - -[[package]] -name = "anstyle-query" -version = "1.1.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "40c48f72fd53cd289104fc64099abca73db4166ad86ea0b4341abe65af83dadc" -dependencies = [ - "windows-sys", -] - -[[package]] -name = "anstyle-wincon" -version = "3.0.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "291e6a250ff86cd4a820112fb8898808a366d8f9f58ce16d1f538353ad55747d" -dependencies = [ - "anstyle", - "once_cell_polyfill", - "windows-sys", -] - -[[package]] -name = "anyhow" -version = "1.0.104" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "330a5ed07fa54e4702c9d6c4174f74427fc0ef6e214bbd677ae50a5099946470" - -[[package]] -name = "bstr" -version = "1.13.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1f7dc094d718f2e1c1559ad110e27eeaae14a5465d3d56dd6dbd793079fbd530" -dependencies = [ - "memchr", - "serde_core", -] - -[[package]] -name = "clap" -version = "4.6.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "473c7e07f409a8d772161724aa8db6a765a2532a70f9667eeb7b49d3d02fbdca" -dependencies = [ - "clap_builder", - "clap_derive", -] - -[[package]] -name = "clap_builder" -version = "4.6.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7b48fea5a88e9ae728a2dcbedbfc0e730f7d60da42e1cb049a83c9fb8b789889" -dependencies = [ - "anstream", - "anstyle", - "clap_lex", - "strsim", -] - -[[package]] -name = "clap_derive" -version = "4.6.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d012d2b9d65aca7f18f4d9878a045bc17899bba951561ba5ec3c2ba1eed9a061" -dependencies = [ - "heck", - "proc-macro2", - "quote", - "syn", -] - -[[package]] -name = "clap_lex" -version = "1.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c8d4a3bb8b1e0c1050499d1815f5ab16d04f0959b233085fb31653fbfc9d98f9" - -[[package]] -name = "colorchoice" -version = "1.0.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570" - -[[package]] -name = "globset" -version = "0.4.20" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "07c34a9410465b45bd9787443bc7370f37735bad04b0f0cd57ff1a3186c98988" -dependencies = [ - "aho-corasick", - "bstr", - "log", - "regex-automata", - "regex-syntax", -] - -[[package]] -name = "heck" -version = "0.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea" - -[[package]] -name = "is_terminal_polyfill" -version = "1.70.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a6cb138bb79a146c1bd460005623e142ef0181e3d0219cb493e02f7d08a35695" - -[[package]] -name = "log" -version = "0.4.33" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0ceec5bc11778974d1bcb055b18002eba7f4b3518b6a0081b3af5f21666da9ad" - -[[package]] -name = "memchr" -version = "2.8.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98" - -[[package]] -name = "once_cell_polyfill" -version = "1.70.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "384b8ab6d37215f3c5301a95a4accb5d64aa607f1fcb26a11b5303878451b4fe" - -[[package]] -name = "owl-filter" -version = "0.1.0" -dependencies = [ - "anyhow", - "clap", - "globset", - "walkdir", -] - -[[package]] -name = "proc-macro2" -version = "1.0.107" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9" -dependencies = [ - "unicode-ident", -] - -[[package]] -name = "quote" -version = "1.0.47" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001" -dependencies = [ - "proc-macro2", -] - -[[package]] -name = "regex-automata" -version = "0.4.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ad8553b9b26413251cbf30e620595c7a41b3887f03da04579c0e6b0d6a06b4b2" -dependencies = [ - "aho-corasick", - "memchr", - "regex-syntax", -] - -[[package]] -name = "regex-syntax" -version = "0.8.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4" - -[[package]] -name = "same-file" -version = "1.0.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "93fc1dc3aaa9bfed95e02e6eadabb4baf7e3078b0bd1b4d7b6b0b68378900502" -dependencies = [ - "winapi-util", -] - -[[package]] -name = "serde_core" -version = "1.0.229" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "67dca2c9c51e58a4791a4b1ed58308b39c64224d349a935ab5039aa360942a48" -dependencies = [ - "serde_derive", -] - -[[package]] -name = "serde_derive" -version = "1.0.229" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348" -dependencies = [ - "proc-macro2", - "quote", - "syn", -] - -[[package]] -name = "strsim" -version = "0.11.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7da8b5736845d9f2fcb837ea5d9e2628564b3b043a70948a3f0b778838c5fb4f" - -[[package]] -name = "syn" -version = "3.0.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "53e9bae58849f64dfa4f5d5ae372c8341f7305f82a3868709269343628b659a3" -dependencies = [ - "proc-macro2", - "quote", - "unicode-ident", -] - -[[package]] -name = "unicode-ident" -version = "1.0.24" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" - -[[package]] -name = "utf8parse" -version = "0.2.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821" - -[[package]] -name = "walkdir" -version = "2.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "29790946404f91d9c5d06f9874efddea1dc06c5efe94541a7d6863108e3a5e4b" -dependencies = [ - "same-file", - "winapi-util", -] - -[[package]] -name = "winapi-util" -version = "0.1.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22" -dependencies = [ - "windows-sys", -] - -[[package]] -name = "windows-link" -version = "0.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" - -[[package]] -name = "windows-sys" -version = "0.61.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc" -dependencies = [ - "windows-link", -] diff --git a/experimental/owl/filter/Cargo.toml b/experimental/owl/filter/Cargo.toml deleted file mode 100644 index 694e8e9..0000000 --- a/experimental/owl/filter/Cargo.toml +++ /dev/null @@ -1,11 +0,0 @@ -[package] -name = "owl-filter" -version = "0.1.0" -edition = "2021" -license = "MIT" - -[dependencies] -anyhow = "1" -clap = { version = "4", features = ["derive"] } -globset = "0.4" -walkdir = "2" diff --git a/experimental/owl/filter/src/fileset.rs b/experimental/owl/filter/src/fileset.rs deleted file mode 100644 index 4f83d08..0000000 --- a/experimental/owl/filter/src/fileset.rs +++ /dev/null @@ -1,207 +0,0 @@ -//! Parser and matcher for `owl.fileset.txt` — the gitignore-like manifest that -//! decides which files owl renders. Each line is one glob, compiled by `globset` -//! with `literal_separator(true)` and anchored gitignore-style: a pattern with a -//! `/` in it is anchored to the checkout root, a pattern without one matches at -//! any depth, and a trailing `/` matches a directory's whole subtree. A leading -//! `!` re-includes; `@import ` inlines another fileset/.gitignore-style -//! file (path relative to the importing file). Paths are matched checkout-root- -//! relative, later rules win, and an unmatched path defaults to "included". See -//! the header of `owl.fileset.txt` for the source-of-truth format docs. - -use std::path::{Path, PathBuf}; - -use anyhow::{bail, Context, Result}; -use globset::{GlobBuilder, GlobMatcher}; - -/// One rule: the glob(s) a single pattern expands to, plus whether matching them -/// re-includes (`!`) or excludes. The globs are OR'd (any match = the rule fires). -struct Rule { - negated: bool, - matchers: Vec, -} - -/// An ordered list of rules resolved from a fileset file (imports inlined in place). -pub struct Fileset { - rules: Vec, -} - -impl Fileset { - /// Load a fileset file, recursively inlining `@import`ed files. - pub fn load(path: &Path) -> Result { - let mut rules = Vec::new(); - parse_file(path, &mut rules, &mut Vec::new())?; - Ok(Fileset { rules }) - } - - /// Whether a checkout-root-relative path is included. Default include; scan - /// rules in order and let the last match win (exclude drops, `!` re-includes). - pub fn includes(&self, rel: &Path) -> bool { - let mut included = true; - for rule in &self.rules { - if rule.matchers.iter().any(|m| m.is_match(rel)) { - included = rule.negated; - } - } - included - } - - #[cfg(test)] - fn from_str(content: &str) -> Result { - let mut rules = Vec::new(); - parse_lines(content, Path::new("."), &mut rules, &mut Vec::new())?; - Ok(Fileset { rules }) - } -} - -/// Read a fileset file and parse its lines. `visited` holds canonicalized paths -/// already parsed, breaking `@import` cycles. -fn parse_file(path: &Path, rules: &mut Vec, visited: &mut Vec) -> Result<()> { - let canon = path - .canonicalize() - .with_context(|| format!("resolving fileset {}", path.display()))?; - if visited.contains(&canon) { - return Ok(()); - } - visited.push(canon.clone()); - - let content = std::fs::read_to_string(&canon) - .with_context(|| format!("reading fileset {}", canon.display()))?; - let dir = canon.parent().unwrap_or_else(|| Path::new(".")); - parse_lines(&content, dir, rules, visited) -} - -/// Parse the lines of a fileset, resolving `@import` relative to `dir`. -fn parse_lines( - content: &str, - dir: &Path, - rules: &mut Vec, - visited: &mut Vec, -) -> Result<()> { - for (idx, raw) in content.lines().enumerate() { - let line = raw.trim(); - if line.is_empty() || line.starts_with('#') { - continue; - } - if let Some(rest) = line.strip_prefix("@import") { - let target = rest.trim(); - if target.is_empty() { - bail!("line {}: `@import` needs a path", idx + 1); - } - parse_file(&dir.join(target), rules, visited)?; - continue; - } - let (negated, pat) = match line.strip_prefix('!') { - Some(rest) => (true, rest.trim()), - None => (false, line), - }; - let matchers = compile_pattern(pat) - .with_context(|| format!("line {}: invalid pattern {:?}", idx + 1, pat))?; - rules.push(Rule { negated, matchers }); - } - Ok(()) -} - -/// Expand one gitignore-style pattern into the globset globs it should match, then -/// compile them. Anchoring rules mirror gitignore: a `/` anywhere anchors to the -/// root, no `/` matches at any depth, a trailing `/` matches a directory subtree. -fn compile_pattern(raw: &str) -> Result> { - let (dir_only, pat) = match raw.strip_suffix('/') { - Some(p) => (true, p), - None => (false, raw), - }; - let (leading_anchor, pat) = match pat.strip_prefix('/') { - Some(p) => (true, p), - None => (false, pat), - }; - if pat.is_empty() { - bail!("empty pattern"); - } - let anchored = leading_anchor || pat.contains('/'); - - let mut globs: Vec = Vec::new(); - if anchored { - if !dir_only { - globs.push(pat.to_string()); - } - globs.push(format!("{pat}/**")); - } else { - if !dir_only { - globs.push(pat.to_string()); - globs.push(format!("**/{pat}")); - } - globs.push(format!("{pat}/**")); - globs.push(format!("**/{pat}/**")); - } - - globs - .iter() - .map(|g| { - Ok(GlobBuilder::new(g) - .literal_separator(true) - .build()? - .compile_matcher()) - }) - .collect() -} - -#[cfg(test)] -mod tests { - use super::*; - use std::path::Path; - - fn included(fs: &Fileset, p: &str) -> bool { - fs.includes(Path::new(p)) - } - - #[test] - fn default_is_included() { - let fs = Fileset::from_str("secrets/\n").unwrap(); - assert!(included(&fs, "venus/foo.nix")); - } - - #[test] - fn directory_exclude_covers_contents_at_any_depth() { - let fs = Fileset::from_str("secrets/\n").unwrap(); - assert!(!included(&fs, "secrets/passwords.yaml")); - assert!(!included(&fs, "secrets/personal/jupiter.env")); - assert!(!included(&fs, "nested/secrets/x")); // slashless -> any depth - assert!(included(&fs, "secretsX/thing")); // respects the name boundary - } - - #[test] - fn anchored_pattern_respects_literal_separator() { - // A `/` anchors to root; `*` never crosses a path separator. - let fs = Fileset::from_str("a/*.md\n").unwrap(); - assert!(!included(&fs, "a/x.md")); - assert!(included(&fs, "a/b/x.md")); // `*` does not span a directory - assert!(included(&fs, "x.md")); // anchored: only under a/ - } - - #[test] - fn slashless_pattern_matches_any_depth() { - let fs = Fileset::from_str("*.loro\n").unwrap(); - assert!(!included(&fs, "snap.loro")); - assert!(!included(&fs, "deep/dir/snap.loro")); - } - - #[test] - fn negation_re_includes_and_last_match_wins() { - let fs = Fileset::from_str("build/\n!build/keep/\n").unwrap(); - assert!(!included(&fs, "build/out.js")); - assert!(included(&fs, "build/keep/y.js")); - } - - #[test] - fn lockfiles_excluded_but_flake_lock_kept() { - let fs = Fileset::from_str("*.lock\n!flake.lock\n").unwrap(); - assert!(!included(&fs, "Cargo.lock")); - assert!(!included(&fs, "a/b/pnpm-lock.lock")); - assert!(included(&fs, "flake.lock")); - assert!(included(&fs, "experimental/owl/flake.lock")); - } - - #[test] - fn import_directive_needs_a_path() { - assert!(Fileset::from_str("@import\n").is_err()); - } -} diff --git a/experimental/owl/filter/src/main.rs b/experimental/owl/filter/src/main.rs deleted file mode 100644 index 289276c..0000000 --- a/experimental/owl/filter/src/main.rs +++ /dev/null @@ -1,70 +0,0 @@ -//! owl-filter — apply an `owl.fileset.txt` manifest to a checkout, copying the -//! included files into an output directory. This is owl's Nix-level pre-filter: -//! it runs before the Astro renderer, so excluded paths (notably `secrets/`) -//! never reach the render step. The renderer only ever sees the pruned tree. - -mod fileset; - -use std::fs; -use std::path::PathBuf; - -use anyhow::{Context, Result}; -use clap::Parser; -use walkdir::WalkDir; - -use fileset::Fileset; - -/// Copy the files an `owl.fileset.txt` includes from a checkout into an output dir. -#[derive(Parser)] -#[command(name = "owl-filter", version, about)] -struct Args { - /// Path to the fileset manifest (e.g. the checkout's `owl.fileset.txt`). - #[arg(long)] - fileset: PathBuf, - - /// Source checkout root to filter. - src: PathBuf, - - /// Output directory to populate with the included files. - out: PathBuf, -} - -fn main() -> Result<()> { - let args = Args::parse(); - let fileset = Fileset::load(&args.fileset)?; - let src_root = args - .src - .canonicalize() - .with_context(|| format!("resolving source dir {}", args.src.display()))?; - - // Prune VCS metadata dirs before walking them (they're huge and also excluded). - let walker = WalkDir::new(&src_root).into_iter().filter_entry(|e| { - !(e.file_type().is_dir() && matches!(e.file_name().to_str(), Some(".git" | ".jj"))) - }); - - let mut copied = 0usize; - for entry in walker { - let entry = entry?; - // Regular files only: symlinks and directories are skipped. - if !entry.file_type().is_file() { - continue; - } - let rel = entry - .path() - .strip_prefix(&src_root) - .expect("walked path is under src_root"); - if !fileset.includes(rel) { - continue; - } - let dest = args.out.join(rel); - if let Some(parent) = dest.parent() { - fs::create_dir_all(parent).with_context(|| format!("mkdir {}", parent.display()))?; - } - fs::copy(entry.path(), &dest) - .with_context(|| format!("copy {} -> {}", entry.path().display(), dest.display()))?; - copied += 1; - } - - eprintln!("owl-filter: copied {copied} files to {}", args.out.display()); - Ok(()) -} diff --git a/experimental/owl/flake.lock b/experimental/owl/flake.lock index 07c2027..737629e 100644 --- a/experimental/owl/flake.lock +++ b/experimental/owl/flake.lock @@ -3,11 +3,11 @@ "advisory-db": { "flake": false, "locked": { - "lastModified": 1786008882, - "narHash": "sha256-NuiXG+3pR/w7kq+ubJR4FgG7CLEdNm48iuiTw92lp/s=", + "lastModified": 1786446289, + "narHash": "sha256-wHPYNshuOPVU415g2IdIzQD52W05HUdl5O/pCVLBODo=", "owner": "rustsec", "repo": "advisory-db", - "rev": "1237bbe09d2701e14e6593a630fbaf28928df712", + "rev": "d0861df1eab469d3c58d6b836ce48b5766e5f217", "type": "github" }, "original": { @@ -46,6 +46,25 @@ "url": "file:///Users/yuto/src/everything" } }, + "fileset": { + "inputs": { + "advisory-db": "advisory-db", + "crane": "crane", + "flake-utils": "flake-utils", + "nixpkgs": [ + "nixpkgs" + ] + }, + "locked": { + "path": "../fileset", + "type": "path" + }, + "original": { + "path": "../fileset", + "type": "path" + }, + "parent": [] + }, "flake-utils": { "inputs": { "systems": "systems" @@ -64,6 +83,24 @@ "type": "github" } }, + "flake-utils_2": { + "inputs": { + "systems": "systems_2" + }, + "locked": { + "lastModified": 1731533236, + "narHash": "sha256-l0KFg5HjrsfsO/JpG+r7fRrqm12kzFHyUHqHCVpMMbI=", + "owner": "numtide", + "repo": "flake-utils", + "rev": "11707dc2f618dd54ca8739b309ec4fc024de578b", + "type": "github" + }, + "original": { + "owner": "numtide", + "repo": "flake-utils", + "type": "github" + } + }, "nixpkgs": { "locked": { "lastModified": 1785975029, @@ -82,10 +119,9 @@ }, "root": { "inputs": { - "advisory-db": "advisory-db", - "crane": "crane", "everything": "everything", - "flake-utils": "flake-utils", + "fileset": "fileset", + "flake-utils": "flake-utils_2", "nixpkgs": "nixpkgs" } }, @@ -103,6 +139,21 @@ "repo": "default", "type": "github" } + }, + "systems_2": { + "locked": { + "lastModified": 1681028828, + "narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=", + "owner": "nix-systems", + "repo": "default", + "rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e", + "type": "github" + }, + "original": { + "owner": "nix-systems", + "repo": "default", + "type": "github" + } } }, "root": "root", diff --git a/experimental/owl/flake.nix b/experimental/owl/flake.nix index 3199283..10f9e30 100644 --- a/experimental/owl/flake.nix +++ b/experimental/owl/flake.nix @@ -4,13 +4,14 @@ inputs = { nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable"; - crane.url = "github:ipetkov/crane"; - flake-utils.url = "github:numtide/flake-utils"; - advisory-db = { - url = "github:rustsec/advisory-db"; - flake = false; + # The Rust fileset pre-filter, now its own general-purpose flake next door. owl + # consumes only its compiled binary (in filterTree + the run scripts), so a path + # input beats a Cargo dependency — owl keeps no Rust of its own. + fileset = { + url = "path:../fileset"; + inputs.nixpkgs.follows = "nixpkgs"; }; # The monorepo itself, as a plain source tree (the repo root has no flake.nix), @@ -27,9 +28,8 @@ { self, nixpkgs, - crane, flake-utils, - advisory-db, + fileset, everything, ... }: @@ -39,19 +39,8 @@ pkgs = nixpkgs.legacyPackages.${system}; inherit (pkgs) lib; - craneLib = crane.mkLib pkgs; - - # --- owl-filter: the Rust fileset pre-filter (crate in ./filter) --------- - # Pure-Rust deps (globset/walkdir/clap/anyhow), so no cmake/perl; libiconv - # only for the darwin linker. - filterSrc = craneLib.cleanCargoSource ./filter; - filterArgs = { - src = filterSrc; - strictDeps = true; - buildInputs = lib.optionals pkgs.stdenv.isDarwin [ pkgs.libiconv ]; - }; - filterDeps = craneLib.buildDepsOnly filterArgs; - owl-filter = craneLib.buildPackage (filterArgs // { cargoArtifacts = filterDeps; }); + # The fileset pre-filter binary, built by the sibling `fileset` flake. + filterBin = fileset.packages.${system}.default; # --- owl-render: the Astro renderer as a runtime binary ------------------ # buildNpmPackage materializes node_modules from the lock but does NOT run @@ -103,16 +92,16 @@ ''; # --- compose: filter a checkout, then render the pruned tree ------------- - # filterTree runs owl-filter over `src` using `fileset`, producing a tree - # with excluded paths (e.g. secrets/) removed — that tree is the ONLY thing - # the render sandbox sees. + # filterTree runs the fileset binary over `src` using `fileset` (the manifest, + # which shadows the flake input of the same name here), producing a tree with + # excluded paths (e.g. secrets/) removed — the ONLY thing the render sandbox sees. filterTree = { src, fileset, }: pkgs.runCommand "owl-filtered" { } '' - ${owl-filter}/bin/owl-filter --fileset ${fileset} ${src} "$out" + ${filterBin}/bin/fileset --fileset ${fileset} ${src} "$out" ''; renderCheckout = @@ -128,8 +117,8 @@ in { packages = { - default = owl-filter; - inherit owl-filter owl-render; + default = owl-render; + inherit owl-render; # The whole monorepo rendered to a static site (result/ is the deployable # dir). Renders the COMMITTED tree of the `everything` input, so commit @@ -142,7 +131,7 @@ }; apps.default = flake-utils.lib.mkApp { - drv = owl-filter; + drv = owl-render; }; # Composition helpers for consumers (e.g. a top-level flake) that have a @@ -151,34 +140,9 @@ inherit filterTree renderTree renderCheckout; }; - checks = { - inherit owl-filter; - - owl-filter-clippy = craneLib.cargoClippy ( - filterArgs - // { - cargoArtifacts = filterDeps; - cargoClippyExtraArgs = "--all-targets -- --deny warnings"; - } - ); - - owl-filter-fmt = craneLib.cargoFmt { src = filterSrc; }; - - owl-filter-audit = craneLib.cargoAudit { src = filterSrc; inherit advisory-db; }; - - owl-filter-nextest = craneLib.cargoNextest ( - filterArgs - // { - cargoArtifacts = filterDeps; - partitions = 1; - partitionType = "count"; - cargoNextestPartitionsExtraArgs = "--no-tests=pass"; - } - ); - }; - - devShells.default = craneLib.devShell { - # cargo + rustc come from craneLib; add Node for the owl-web dev loop. + devShells.default = pkgs.mkShell { + # owl has no Rust of its own now (the filter moved to the `fileset` flake); + # just Node for the owl-web dev loop. packages = [ pkgs.nodejs ]; }; } diff --git a/experimental/owl/run-owl-for-general-development.sh b/experimental/owl/run-owl-for-general-development.sh index c9f4f2f..5e434dc 100755 --- a/experimental/owl/run-owl-for-general-development.sh +++ b/experimental/owl/run-owl-for-general-development.sh @@ -4,7 +4,7 @@ # For developing a feature OTHER than owl: a stable, correct owl over your current files # that tracks the working tree without restarts. # -# 1. owl-filter (hermetic `nix run`) prunes the live checkout with owl.fileset.txt into a +# 1. fileset (hermetic `nix run`) prunes the live checkout with owl.fileset.txt into a # fresh staging tree — new/untracked files appear, excluded (secrets/) + build junk # never leave, and a fresh tree each pass means deletions propagate. # 2. owl-render (built once, then reused) renders that tree to a static $dist. It reuses a @@ -34,14 +34,14 @@ owl_render=$(readlink -f "$render_link")/bin/owl-render # One pass: filter the live repo into a fresh staging tree, then render it incrementally # into $dist reusing $work's cache. A fresh staging tree each pass makes deletions -# propagate (owl-filter only ever adds/overwrites). Baked into a temp script with concrete +# propagate (fileset only ever adds/overwrites). Baked into a temp script with concrete # paths so the watcher can invoke it without quoting gymnastics. refilter=$(mktemp "${TMPDIR:-/tmp}/owl-refilter.XXXXXX") cat > "$refilter" < "$refilter" </dev/null; rm -rf "$input" "$refilter"' EXIT -echo "==> owl-filter: initial snapshot -> $input" -nix run "$here#owl-filter" -- --fileset "$fileset" "$repo_root" "$input" +echo "==> fileset: initial snapshot -> $input" +nix run "$here/../fileset" -- --fileset "$fileset" "$repo_root" "$input" # Watch the repo; re-filter on change. owl's own outputs (.owl-tree/, the generated # manifest, .jj/) are ignored so our writes can't retrigger the loop; .git and diff --git a/experimental/owl/run-owl-for-owl-development-with-frozen-fileset.sh b/experimental/owl/run-owl-for-owl-development-with-frozen-fileset.sh index 1d17b1c..854782d 100755 --- a/experimental/owl/run-owl-for-owl-development-with-frozen-fileset.sh +++ b/experimental/owl/run-owl-for-owl-development-with-frozen-fileset.sh @@ -17,8 +17,8 @@ fileset="$repo_root/owl.fileset.txt" input=$(mktemp -d "${TMPDIR:-/tmp}/owl-input.XXXXXX") trap 'rm -rf "$input"' EXIT -echo "==> owl-filter: one-time snapshot of the repo -> $input" -nix run "$here#owl-filter" -- --fileset "$fileset" "$repo_root" "$input" +echo "==> fileset: one-time snapshot of the repo -> $input" +nix run "$here/../fileset" -- --fileset "$fileset" "$repo_root" "$input" echo "==> npm run dev: http://localhost:4321 (renderer HMR; content frozen; Ctrl-C to stop)" cd "$here/web" diff --git a/owl.fileset.txt b/owl.fileset.txt index 8d8cf0d..ac5ebfc 100644 --- a/owl.fileset.txt +++ b/owl.fileset.txt @@ -5,6 +5,9 @@ # git tracking is untouched. Everything owl encounters is rendered by default; # the rules below subtract from that set (and `!` can add back). # +# This manifest is applied by the general-purpose `fileset` tool +# (experimental/fileset); see its README for the canonical format reference. +# # FORMAT (gitignore-like) # # Line comment. Blank lines are ignored. # Exclude paths matching this glob.