From e3ce79ee7def9cbd2269792ac40f0eb864c54e15 Mon Sep 17 00:00:00 2001 From: zzstoatzz Date: Tue, 18 Aug 2026 01:56:08 -0500 Subject: [PATCH] docs: add a changelog; stop tracking state in the README and incident doc MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The incident writeup had grown a status table, a "resolution" section and an "outstanding" section, all of which needed hand-editing every time something shipped — and all of which went stale between edits. That state now lives in CHANGELOG.md, and the doc keeps only what does not change: the mechanism, the reproduction, and the repair recipe. Also drops the README index entry for it. The README lists durable guides; an incident writeup is not one, and the link was one more place to maintain. Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 17 +++++ README.md | 5 +- docs/incident-2026-08-18-empty-mst-node.md | 82 ++++++++++------------ 3 files changed, 57 insertions(+), 47 deletions(-) create mode 100644 CHANGELOG.md diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..9cff3e6 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,17 @@ +# changelog + +## unreleased + +- **fix**: bump zat to `v0.4.3`, which makes `Mst.collectBlocks` emit the block for an + empty MST node instead of mistaking it for an unloaded stub and skipping it. Two repos + on pds.zat.dev carried empty subtree nodes minted by a pre-`v0.3.19` writer whose blocks + were never stored; the first tree walk to reach one returned `PartialTree`, taking every + `createRecord` to 500 and `getRepo` to 404 regardless of collection. The fix is + self-healing — a write that does not descend into the stranded branch still commits, and + the collect pass restores the missing block. See + `docs/incident-2026-08-18-empty-mst-node.md`. +- **fix**: handle `zat.cbor.Value.float` in the CBOR -> ATProto JSON writer. New in zat + `v0.4.2`; the unhandled switch case broke the build on the version bump. +- **ops**: repaired the three stranded empty MST nodes in the `zat.dev` repo with forward + create/delete commits. Its served root now matches a canonical rebuild of its keyset. No + history was rewritten. diff --git a/README.md b/README.md index e652453..8558995 100644 --- a/README.md +++ b/README.md @@ -16,9 +16,8 @@ Start with the document that matches the work: - Operators: [operator guide](docs/operations.md), [production deployment](docs/deployment.md), [invite codes](docs/invite-codes.md), - [Comail](docs/comail.md), the - [account takedown runbook](docs/account-takedown-runbook.md), and the - [2026-08-18 empty-MST-node incident](docs/incident-2026-08-18-empty-mst-node.md) + [Comail](docs/comail.md), and the + [account takedown runbook](docs/account-takedown-runbook.md) - Account work: [create-account flow](docs/create-account-flow.md), [account security](docs/account-security.md), and [passkeys](docs/passkeys.md) - Protocol work: [architecture](docs/architecture.md), diff --git a/docs/incident-2026-08-18-empty-mst-node.md b/docs/incident-2026-08-18-empty-mst-node.md index ce95ee7..fd62a17 100644 --- a/docs/incident-2026-08-18-empty-mst-node.md +++ b/docs/incident-2026-08-18-empty-mst-node.md @@ -7,10 +7,10 @@ fixed upstream before it ever ran here. Two defects compose. Neither is dangerous alone. -| | defect | where | status | +| | defect | introduced | fixed | |---|---|---|---| -| **A** | delete leaves an emptied subtree node in the tree instead of pruning the pointer | zat ≤ v0.3.18 | fixed in [`f7d0816`](#citations), shipped v0.3.19 | -| **B** | `collectBlocks` never emits an empty node's block, so it is referenced but never stored | zat, current | **still live** | +| **A** | delete leaves an emptied subtree node in the tree instead of pruning the pointer | zat v0.3.10 | zat v0.3.19 (`f7d0816`) | +| **B** | `collectBlocks` never emits an empty node's block, so it is referenced but never stored | always | zat v0.4.3 | A *created* the bad node. B *hid* it, then turned it into an outage. @@ -172,14 +172,22 @@ request was irrelevant. The tree could not be loaded at all. → shipped in v0.3.19; zds still pinned v0.3.10 2026-08-12 15:13:07 commit 8730 + place.birds.sighting/3msvg56t43skv 15:13:37 commit 8732 − same key → EMPTY NODE CREATED (defect A) - ... zat.dev acquires one the same way, under io.atcr.manifest/… 2026-08-13 04:54 zds 4cb15aa bumps zat v0.3.10 → v0.3.29 defect A fixed — 13h41m too late. the bad node persists. 2026-08-18 05:14 / 05:16 last successful commits on both repos ~05:37 first `PartialTree` — a walk finally reaches the empty node - → all writes down on zat.dev and birds.place + → all writes down on zat.dev and birds.place + <05:59 mitigation: the 7-byte block inserted for both DIDs; writes resume + (birds.place commits again at 05:59:39, pruning its own node) + 06:33 zat v0.4.3 deployed (defect B fixed) + later zat.dev's three stranded nodes cleared by forward create/delete + commits; PDS-wide sweep clean ``` +Dated entries are for birds.place, whose history was replayed commit by commit. zat.dev +separately acquired **three** such nodes under `io.atcr.manifest/…` by the same mechanism; +its history was not replayed, so their dates are not established. + The gap is the interesting part. The bad node sat inert for six days because nothing had to traverse that particular branch. Defect B guarantees the block is missing; it just takes a walk that reaches it to convert that into an outage. @@ -221,10 +229,10 @@ Applied to both DIDs. Both repos served again immediately. birds.place then **healed itself**: its bot's normal create/delete cycle on `place.birds.sighting/*` lands in exactly that subtree, and the delete now runs through the -fixed `pruneIfEmpty`. Its served root is now identical to a canonical fresh build. +fixed `pruneIfEmpty`. Its served root returned to a canonical fresh build without intervention. -zat.dev has not written since, so it still carries its empty node under -`io.atcr.manifest/…/l` and still serves a non-canonical root. +zat.dev had no traffic landing in the right place, so it kept its stranded nodes and served +a non-canonical root until they were cleared deliberately — see **repair** below. ### sweep at mitigation time @@ -250,55 +258,41 @@ why the same 7 bytes were present under other DIDs while missing under the broke --- -## outstanding - -- **defect B fix written, not yet released.** `collectNodeBlocks` now separates a stub from - an empty node by comparing the node's cid to the empty-node cid, and emits the 7 bytes when - they match. Two regression tests cover it — one on the closure invariant, one on the - production repair path — and both fail without the fix. Needs a zat release and a zds bump. - The fix is **self-healing**: `nodeCid` returns a stub's cid without loading it, so a write - that does not descend into the stranded branch still succeeds, and the collect pass puts - the missing block back. Any repo carrying a pre-v0.3.19 artifact repairs itself on its - next write. -- **zat.dev is repaired.** It carried **three** stranded empty nodes, not one — an early - sweep of mine deduped by CID, and every empty node shares the same CID, so it only ever - reported the first. Each was cleared by an ordinary forward create-then-delete of a key - landing inside it (see below); its served root now equals a canonical fresh rebuild: - `ee21ba1b…`. No history was rewritten. -- **zds `applyWrites` indexes a batch out of order.** The MST applies ops sequentially, but - the SQL does every delete before every create, so a same-key create+delete in one batch - leaves the record row present while the MST says absent. The reference PDS applies writes - in one ordered loop (`packages/pds/src/actor-store/repo/transactor.ts:193-213`) and - tranquil resolves each delete against the running MST - (`crates/tranquil-api/src/repo/record/batch.rs:180-190`); zds is the outlier and can - reach a state neither produces. Unfixed. -- Unrelated, found in the same logs: an httpz worker panic, - `access of union field 'http' while field 'retired' is active` in `collectTimedOut` - (worker.zig:1076), SIGABRT'd the process at 05:16:47. Separate availability bug, upstream. - ---- - ## repair: the corrective forward commit +Restoring the block fixes availability but not shape: the empty node is still there, still +invalid, still changing every ancestor CID. Clearing it needs a forward commit — **not** a +history rewrite. Past commits are untouched; the repo simply moves to a new commit whose +`data` root is the correctly-shaped tree, which is ordinary appending under the spec's only +requirement that `rev` increase monotonically. + A stranded empty node cannot be pruned by an ordinary delete — `pruneIfEmpty` only fires after a delete that *descends through* the pointer and finds something, and there is nothing -inside an empty node to find. The fix is a create that lands inside it followed by a delete: +inside an empty node to find. So it takes a create that lands inside it, then a delete: ``` create io.atcr.manifest/ → the empty node now holds one entry delete io.atcr.manifest/ → node empties, pruneIfEmpty drops the pointer ``` -Two commits, never one batch, because of the `applyWrites` ordering defect above. +Two commits, never one batch, because of the `applyWrites` ordering defect below. -The trigger key must satisfy both MST placement rules: `keyHeight(key) == layer of the empty -node`, and it must sort strictly inside the gap the node occupies. Both bounds share the -`io.atcr.manifest` collection prefix, so the key has to live in that collection — there is no -scratch-namespace option. For a node that is its parent's last entry, appending a suffix to -that entry's rkey yields the smallest key greater than it, which is always inside the gap. +The trigger key must satisfy both MST placement rules: `keyHeight(key)` must equal the layer +of the empty node, and it must sort strictly inside the gap the node occupies. Both bounds +share the `io.atcr.manifest` collection prefix, so the key has to live in that collection — +there is no scratch-namespace option. For a node that is its parent's last entry, appending +a suffix to that entry's rkey yields the smallest key greater than it, which is always +inside the gap. birds.place never needed this: its bot's own create/delete cycle on `place.birds.sighting/*` -happened to land in the right gap and pruned the node as a side effect. +happened to land in the right gap and pruned its node as a side effect. + +### counting them is harder than it looks + +zat.dev held **three** stranded nodes, not one. Every empty node is the same 7 bytes and +therefore the same CID, so a tree walk that memoises visited CIDs reports only the first one +and silently hides the rest. The first repair looked like it had *moved* the node rather +than removed it. Count by path, never by CID. --- -- 2.51.2