diff --git a/star-lite/readme.md b/star-lite/readme.md index c59a0f2..d6aa207 100644 --- a/star-lite/readme.md +++ b/star-lite/readme.md @@ -34,7 +34,7 @@ STAR-lite is a flat list of every key/record pair in a repository, in lexicograp | name | type | | ----- | --------------------------------------- | | magic | three-byte mark to identify the format | -| cid | atproto-format binary MST node CID link | +| cid | atproto-format binary CID link | | len | unsigned varint | | str | utf-8 bytes | | cbor | cbor bytes | @@ -102,30 +102,66 @@ STAR-lite is intended to be externally compressed with zstd in transport or for TODO: include recommended zstd configs, and tables/graphs showing compression performance. should show vs CAR, and also compare gzip (maybe brotli?) to zstd settings -## STAR-lite algorithms +#### DATAaaaaaaaa -While any atproto MST library can reconstruct a full repo MST by simply inserting each `(key, record)` pair, materializing the entire MST at once costs significant memory or i/o overhead. +Ratios are STAR/CAR (lower is better). "raw" baseline = uncompressed CAR; "coder" baseline = CAR compressed with the same coder. -We exploit the lexicographic key ordering of STAR-lite files (or any stream of lex-ordered key-record pairs) to **walk a fully-reconstructed MST without holding the entire tree in memory**. +#### overall -This enables efficient transformations, like verifying repository integrity, or conversion to stream-ordered atproto CARv1 format archive. +N=4866, raw CAR=2.18 GiB, raw STAR=1.62 GiB. +| setting | mean (raw) | med (raw) | wt (raw) | mean (coder) | med (coder) | wt (coder) | +|---|---:|---:|---:|---:|---:|---:| +| raw | 0.668 | 0.678 | 0.746 | — | — | — | +| gzip | 0.292 | 0.232 | 0.215 | 0.568 | 0.556 | 0.552 | +| zstd --fast 1 | 0.333 | 0.295 | 0.286 | 0.632 | 0.635 | 0.671 | +| zstd 3 | 0.281 | 0.224 | 0.195 | 0.566 | 0.551 | 0.553 | +| zstd 9 | 0.276 | 0.218 | 0.183 | 0.562 | 0.544 | 0.542 | -### MST state: node stack +#### < 10 KiB +N=2168, raw CAR=6.31 MiB, raw STAR=4.04 MiB. +| setting | mean (raw) | med (raw) | wt (raw) | mean (coder) | med (coder) | wt (coder) | +|---|---:|---:|---:|---:|---:|---:| +| raw | 0.614 | 0.611 | 0.640 | — | — | — | +| gzip | 0.395 | 0.370 | 0.301 | 0.624 | 0.640 | 0.585 | +| zstd --fast 1 | 0.413 | 0.402 | 0.342 | 0.653 | 0.663 | 0.628 | +| zstd 3 | 0.383 | 0.363 | 0.295 | 0.621 | 0.630 | 0.578 | +| zstd 9 | 0.381 | 0.361 | 0.292 | 0.622 | 0.634 | 0.578 | -We don't need to materialize the entire MST at once for a depth-first tree-reconstructing walk across it: a narrow stack of MST nodes (one per layer of the tree) is sufficient state. +#### 10 KiB – 1 MiB -When a key's layer is *greater than the previous* key's layer, all in-progress MST nodes from lower layers are complete, and can be **frozen**: encoded in atproto MST node format to compute their CIDs, recursively resolving into a CID link from the current key's node. +N=2346, raw CAR=379.92 MiB, raw STAR=276.84 MiB. -At this point, the newly frozen nodes can be: +| setting | mean (raw) | med (raw) | wt (raw) | mean (coder) | med (coder) | wt (coder) | +|---|---:|---:|---:|---:|---:|---:| +| raw | 0.706 | 0.710 | 0.729 | — | — | — | +| gzip | 0.209 | 0.210 | 0.208 | 0.517 | 0.520 | 0.526 | +| zstd --fast 1 | 0.266 | 0.270 | 0.270 | 0.607 | 0.615 | 0.629 | +| zstd 3 | 0.199 | 0.198 | 0.193 | 0.519 | 0.521 | 0.529 | +| zstd 9 | 0.193 | 0.191 | 0.184 | 0.511 | 0.513 | 0.519 | -- simply discarded, when verifying archive integrity, -- serialized into runs of CAR-format blocks, -- any other transformation +#### 1 MiB – 100 MiB -Once the entire tree has been walked and frozen, the highest-layer MST node can finally be considered frozen to produce the root node CID, which must match the CID in a STAR-lite file's header. +N=352, raw CAR=1.80 GiB, raw STAR=1.35 GiB. + +| setting | mean (raw) | med (raw) | wt (raw) | mean (coder) | med (coder) | wt (coder) | +|---|---:|---:|---:|---:|---:|---:| +| raw | 0.749 | 0.746 | 0.750 | — | — | — | +| gzip | 0.216 | 0.216 | 0.216 | 0.556 | 0.557 | 0.557 | +| zstd --fast 1 | 0.288 | 0.290 | 0.289 | 0.676 | 0.675 | 0.680 | +| zstd 3 | 0.192 | 0.194 | 0.195 | 0.547 | 0.543 | 0.558 | +| zstd 9 | 0.180 | 0.182 | 0.183 | 0.534 | 0.532 | 0.547 | + + +## STAR-lite algorithms + +While any atproto MST library can reconstruct a full repo MST by simply inserting each `(key, record)` pair, materializing the entire MST at once costs significant memory or i/o overhead. + +We exploit the lexicographic key ordering of STAR-lite files (or any stream of lex-ordered key-record pairs) to **walk a fully-reconstructed MST without holding the entire tree in memory**. + +This enables efficient transformations, like verifying repository integrity, or conversion to stream-ordered atproto CARv1 format archive. ### Archive verification @@ -470,3 +506,23 @@ Note: it is **not possible** to know if an atproto CAR is stream-ordered except [rkey]: https://atproto.com/specs/record-key [commit]: https://www.ietf.org/archive/id/draft-holmgren-at-repository-00.html#section-2.4 [commit-sigs]: https://www.ietf.org/archive/id/draft-holmgren-at-repository-00.html#name-commit-signatures + + + + +### MST state: node stack + + + +We don't need to materialize the entire MST at once for a depth-first tree-reconstructing walk across it: a narrow stack of MST nodes (one per layer of the tree) is sufficient state. + +When a key's layer is *greater than the previous* key's layer, all in-progress MST nodes from lower layers are complete, and can be **frozen**: encoded in atproto MST node format to compute their CIDs, recursively resolving into a CID link from the current key's node. + +At this point, the newly frozen nodes can be: + +- simply discarded, when verifying archive integrity, +- serialized into runs of CAR-format blocks, +- any other transformation + +Once the entire tree has been walked and frozen, the highest-layer MST node can finally be considered frozen to produce the root node CID, which must match the CID in a STAR-lite file's header. +