diff --git a/docs/lineage_and_persistence.md b/docs/lineage_and_persistence.md index 5d07c23..6605870 100644 --- a/docs/lineage_and_persistence.md +++ b/docs/lineage_and_persistence.md @@ -91,4 +91,114 @@ JSON export/import (`LineagePanel`) is a complete offline breeder. atproto is pu selected only on login — it adds Publish, the global feed, and cross-user lineage. The Wild7 anchor is a dev toggle; a clean local ship boots into the random grid (it already does). +## The forest / overview home (implemented — M3.5) + +The app boots into an **overview**: a pannable/zoomable forest of every saved organism, thumbnails +as nodes with lineage edges drawn parent→child. It's the third top-level `view.mode` +(`overview | grid | focus`). Clicking a node loads it into the focus/editor view to inspect, tune, +breed, or continue saving; **+ New random organism** starts a fresh lineage in the random breeder +grid; **⌂ Home** returns from grid/focus. + +How it reuses the engine: the breeder grid already renders by *time-slicing one resident sim across +candidates and caching each converged frame as a static thumbnail texture*, then compositing. The +overview reuses that wholesale — `Engine.setPlacements(rects)` generalizes the compositor from a +regular cols×rows grid to **arbitrary CSS-pixel rectangles**, so panning/zooming only recomputes +placements (cheap) and never rebuilds a thumbnail. Hover-to-animate works identically. + +Module split (mirrors the breeder): `state/overview.svelte.ts` owns plain reactive layout data +(node positions + a `nonce` + a `building` flag) and a non-reactive holder for the compiled rules; +`ui/Overview.svelte` owns the viewport (pan/zoom), draws edges as an SVG overlay, and overlays +hit-targets. `persistence/repo.ts` is the shared `Repository` singleton both the breeder and the +overview read. + +**Layout.** A tidy forest: `y = depth` (already stored, monotonic), `x` packed so a parent centres +over its children (in-order leaf assignment). Lineage is a DAG, so a node is placed once under its +**first in-set parent** (the lineage-panel convention) and every other in-set parent becomes an +extra edge — so crossover (`⋈`) joins render the moment they exist, with no layout change. + +**Known limit (the load-all problem).** The current path is O(n) on every axis: fetch all records, +**compile every CPPN synchronously** (this blocks the main thread — the visible "freeze" at load), +lay out the whole forest, warm every thumbnail one-per-frame. Fine at tens; untenable at the +hundreds–thousands atproto will bring. That's what the next section plans. + +## Lazy loading at scale (design — not built) + +The shift is from **load-all** to **"render a viewport-bounded subgraph, expanded on demand along +edges."** Two parts are genuinely hard; the rest is bookkeeping. + +**Hard part 1 — the link direction is asymmetric** (see §"Save vs publish" above for why). Ancestry +is cheap and needs no index: parent pointers go up, so you resolve `getRecord` and walk. Descendants +are the problem — a child points up at its parent; the parent doesn't know its children, so +cross-user `children()` needs the **AppView backlink index**. Consequence: a lazy explorer expands +**upward for free and downward only as far as an index reaches.** The UI must degrade gracefully — +render a "⋯ N children" stub that resolves when an index exists, never assume downward is always +available. + +**Hard part 2 — layout without global knowledge.** Today's tidy layout needs the *whole* tree, and +positions must be **stable** (an on-screen node must not jump when neighbors load). +- `y` is free and global: `depth` needs no global knowledge and never shifts. +- `x` is local and anchored: lay out only the loaded subgraph; on each expansion, re-run local + layout but **translate so the anchored (focused) node keeps its screen position**. Off-screen + nodes may reflow — fine. Once a node is on screen, **freeze its x** until it leaves the viewport; + new nodes slot into gaps. Trades perfect tidiness for stability — the right trade for an explorer. + +**The model.** A `GraphStore` (pure TS, same boundary as `overview.svelte.ts`): + +``` +nodes: Map +anchor: id +``` + +Edges are known **before** a node loads (loading a node reveals its parent ids), so partial edges to +unloaded parents render as stubs and expansion walks edges (BFS frontier from the anchor), never the +whole set. + +**The loop.** +1. **Entry point** — you open *somewhere*, not "the forest": a deep-linked at-uri, "my roots", + "recent", or a search hit. That id is the anchor. +2. **Expand frontier** — load the anchor's parents (up-walk) and children (index / own repo) one hop; + neighbors become stubs. +3. **Viewport-gate materialization** — *loading a record* (cheap-ish) is separate from + *materializing it* (compile + GPU warmup, expensive). A node is only compiled + thumbnailed when + its position intersects the viewport (+ margin). Stubs near the viewport edge trigger the next hop + as you pan toward them. +4. **Evict** — nodes far outside the viewport release their thumbnail textures (optionally drop back + to stub) to bound memory. + +**Engine changes.** Replace `setGeneration` (replace-all, one texture per spec forever) with an +**LRU thumbnail pool keyed by organism id** (cap ~60–100 textures, recycle on evict), incremental +`requestThumb(id, spec)` / `releaseThumb(id)`, and a **per-frame build budget** (warm ≤K newly-visible +thumbs/frame) so panning into a dense region streams instead of hitching. `setPlacements` already does +arbitrary positioning + off-screen culling, so that half exists. + +**Thumbnail persistence (the real win).** Warming is *the* expensive step. Cache captured thumbnails +to **IndexedDB as blobs** keyed by `id + engineVersion + simPins`; re-simulate only on a miss. This is +M5's "animated atlas thumbnails" generalized, worth pulling earlier because it makes the explorer feel +fast. Later a thumbnail blob could ride as a `blob` ref on the PDS record so cross-user previews skip +re-simulating someone else's genome (re-sim only to verify). + +**Repository evolution** (mostly additive; `children` paging is the one real change): +``` +get(id) // have it (up-walk) +getMany(ids) // batch a frontier +ancestry(id, maxDepth?) // have it; bound depth +children(id, cursor?, limit?) // PAGED. local: scan own repo. atproto: AppView query +roots(cursor?) / recent(cursor?) // entry points +``` + +**Phasing** — steps 1–2 are pure local refactors with immediate payoff (kill the compile freeze, +bound memory) and are exactly what atproto needs later, so not throwaway: +1. **Now, over `LocalRepository`** (tens of organisms is a fine test rig): `GraphStore` + + viewport-gated materialization + engine LRU. Only visible nodes compile, so the load freeze goes. +2. **Thumbnail IndexedDB cache** — instant revisits. +3. **Atproto up-walk** — ancestry cross-user needs zero index; ship upward exploration. +4. **AppView for `children`** — the backlink index unlocks downward + cross-user descendants. Biggest + piece, rightly last. + See `docs/web_architecture.md` §7 (record schema) and `[[milestone-status]]` (M4 = atproto). diff --git a/src/App.svelte b/src/App.svelte index d61a292..95c210b 100644 --- a/src/App.svelte +++ b/src/App.svelte @@ -193,8 +193,12 @@ async function goHome() { if (!engine) return; view.mode = "overview"; - engine.setMode("grid"); // overview reuses grid-mode compositing - await loadOverview(); // bumps overview.nonce -> the overview effect rebuilds thumbnails + // Drop the stale breeder/focus composite NOW, before the async reload: otherwise the engine keeps + // compositing the old thumbnails (and falls back to the regular 3×3 grid on the placement-length + // mismatch) until loadOverview resolves. Empty specs render blank under the loading spinner. + engine.setPlacements(null); + engine.setGeneration([], 1); + await loadOverview(); // building=true + nodes cleared -> spinner; nonce bump rebuilds thumbnails } diff --git a/src/state/overview.svelte.ts b/src/state/overview.svelte.ts index 6ae2903..02dcc70 100644 --- a/src/state/overview.svelte.ts +++ b/src/state/overview.svelte.ts @@ -61,6 +61,9 @@ function paramsFromRecord(r: OrganismRecord): SubstrateParams { /** (Re)load every saved organism, lay out the forest, and compile each to a render spec. */ export async function loadOverview(): Promise { overview.building = true; // shows the loading state through the fetch + (blocking) compile below + overview.nodes = []; // drop the previous forest: the spinner shows (nodes empty) and no stale layout lingers + overview.cols = 0; + overview.rows = 0; const all = await repo.list(); const byId = new Map(all.map((s) => [s.id, s.record])); const inSet = (pid: string) => byId.has(pid);