diff --git a/docs/incident-2026-08-18-empty-mst-node.md b/docs/incident-2026-08-18-empty-mst-node.md index 7a70fe9..ce95ee7 100644 --- a/docs/incident-2026-08-18-empty-mst-node.md +++ b/docs/incident-2026-08-18-empty-mst-node.md @@ -226,9 +226,10 @@ fixed `pruneIfEmpty`. Its served root is now identical to a canonical fresh buil zat.dev has not written since, so it still carries its empty node under `io.atcr.manifest/…/l` and still serves a non-canonical root. -### post-incident sweep +### sweep at mitigation time -All nine repos, walked from their served roots: +All nine repos, walked from their served roots. Note the `empty=1` counts undercount: +this sweep deduped by CID, and every empty node shares one CID (zat.dev actually held three). ``` did:plc:b64lsctzqnzpv6vd4ry3qktw records=54 missing=0 empty=0 @@ -259,16 +260,48 @@ why the same 7 bytes were present under other DIDs while missing under the broke that does not descend into the stranded branch still succeeds, and the collect pass puts the missing block back. Any repo carrying a pre-v0.3.19 artifact repairs itself on its next write. -- **zat.dev needs a canonical rebuild** — new root, new rev, re-signed commit. The defect B - fix restores availability but preserves the non-canonical shape; only a rebuild (or a - delete that trips `pruneIfEmpty`, which is how birds.place healed) makes the root agree - with what other implementations compute. +- **zat.dev is repaired.** It carried **three** stranded empty nodes, not one — an early + sweep of mine deduped by CID, and every empty node shares the same CID, so it only ever + reported the first. Each was cleared by an ordinary forward create-then-delete of a key + landing inside it (see below); its served root now equals a canonical fresh rebuild: + `ee21ba1b…`. No history was rewritten. +- **zds `applyWrites` indexes a batch out of order.** The MST applies ops sequentially, but + the SQL does every delete before every create, so a same-key create+delete in one batch + leaves the record row present while the MST says absent. The reference PDS applies writes + in one ordered loop (`packages/pds/src/actor-store/repo/transactor.ts:193-213`) and + tranquil resolves each delete against the running MST + (`crates/tranquil-api/src/repo/record/batch.rs:180-190`); zds is the outlier and can + reach a state neither produces. Unfixed. - Unrelated, found in the same logs: an httpz worker panic, `access of union field 'http' while field 'retired' is active` in `collectTimedOut` (worker.zig:1076), SIGABRT'd the process at 05:16:47. Separate availability bug, upstream. --- +## repair: the corrective forward commit + +A stranded empty node cannot be pruned by an ordinary delete — `pruneIfEmpty` only fires +after a delete that *descends through* the pointer and finds something, and there is nothing +inside an empty node to find. The fix is a create that lands inside it followed by a delete: + +``` +create io.atcr.manifest/ → the empty node now holds one entry +delete io.atcr.manifest/ → node empties, pruneIfEmpty drops the pointer +``` + +Two commits, never one batch, because of the `applyWrites` ordering defect above. + +The trigger key must satisfy both MST placement rules: `keyHeight(key) == layer of the empty +node`, and it must sort strictly inside the gap the node occupies. Both bounds share the +`io.atcr.manifest` collection prefix, so the key has to live in that collection — there is no +scratch-namespace option. For a node that is its parent's last entry, appending a suffix to +that entry's rkey yields the smallest key greater than it, which is always inside the gap. + +birds.place never needed this: its bot's own create/delete cycle on `place.birds.sighting/*` +happened to land in the right gap and pruned the node as a side effect. + +--- + ## citations Everything below was read at the versions named, not recalled. @@ -290,6 +323,23 @@ Everything below was read at the versions named, not recalled. - `build.zig.zon` pinned `zat v0.3.10` from `29fd5e2` (2026-07-02) until `4cb15aa` (2026-08-12 23:54:39 -0500 = 2026-08-13 04:54 UTC), which bumped it to `v0.3.29` +**atproto spec** (https://atproto.com/specs/repository, read 2026-08-18) +- *"The top of the tree must not be a an empty node which only points to a sub-tree. Empty + intermediate nodes are allowed, as long as they point to a sub-tree which does contain + entries."* — so `{"e":[],"l":null}` as a subtree pointer is invalid +- *"An empty repository with no records is represented as a single MST node with an empty + array of entries."* — the one position where it is valid +- *"The overall structure and shape of the MST is deterministic based on the current + key/value content, regardless of the history of insertions and deletions that lead to the + current contents."* +- rev *"Must increase monotonically"* — the repair is an append, not a history rewrite + +**other implementations** +- reference PDS `bluesky-social/atproto` @ `02f6e227b`: + `packages/pds/src/actor-store/repo/transactor.ts:193-213` +- tranquil `tranquil.farm/tranquil-pds` @ `1dc0c40`: + `crates/tranquil-api/src/repo/record/batch.rs:180-190` + **production** (`zds-pds`, fly.io, `/data/zds.sqlite3`) - birds.place `did:plc:w4p4bumx22vgnjmctoqjj7sy`, commits 8730 / 8732, revs `3msvg56t43sku` / `3msvg63musrkw`