diff --git a/CHANGELOG.md b/CHANGELOG.md
index 8fcf955..09b4159 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,11 @@
# changelog
+## 0.2.6
+
+- **feat**: specialized MST decoder — `decodeMstNode()` parses known MST CBOR schema directly, zero-copy byte slicing, avoids generic `Value` union construction
+- **feat**: in-walk MST structure verification — `walkAndVerifyMst` checks key heights during traversal instead of full tree rebuild. MST step: 218ms → 39ms (5.5x), compute total: 300ms → 123ms (2.4x)
+- **docs**: devlog 005 — updated benchmark numbers and chart
+
## 0.2.5
- **feat**: O(1) block lookup in CAR parser — `StringHashMap` index built during `read()`/`readWithOptions()`, `findBlock()` uses index instead of linear scan
diff --git a/build.zig.zon b/build.zig.zon
index 52d1292..2581b58 100644
--- a/build.zig.zon
+++ b/build.zig.zon
@@ -1,6 +1,6 @@
.{
.name = .zat,
- .version = "0.2.0",
+ .version = "0.2.6",
.fingerprint = 0x8da9db57ee82fbe4,
.minimum_zig_version = "0.15.0",
.dependencies = .{
diff --git a/devlog/005-three-way-verify.md b/devlog/005-three-way-verify.md
index 5558510..00367a0 100644
--- a/devlog/005-three-way-verify.md
+++ b/devlog/005-three-way-verify.md
@@ -11,14 +11,14 @@ handle → DID → DID document → signing key
↓
repo CAR → commit → signature ← verified against key
↓
- MST root CID → walk nodes → rebuild tree → CID match
+ MST root CID → walk nodes → verify key heights → structure proven
```
-all three implementations do the same work: resolve the handle, resolve the DID, extract the signing key, fetch the repo CAR, parse every block with SHA-256 CID verification, verify the commit signature, walk the MST to count records, and (where possible) rebuild the MST to verify the root CID.
+all three implementations do the same work: resolve the handle, resolve the DID, extract the signing key, fetch the repo CAR, parse every block with SHA-256 CID verification, verify the commit signature, and walk the MST to count records and verify structure.
## the implementations
-**zig (zat)** — uses zat's own primitives end to end: `HandleResolver`, `DidResolver`, `car.read()` with CID verification, `jwt.verifySecp256k1`, `mst.Mst` for walk + rebuild.
+**zig (zat)** — uses zat's own primitives end to end: `HandleResolver`, `DidResolver`, `car.read()` with CID verification + O(1) block index, `jwt.verifySecp256k1`, specialized `decodeMstNode` for walk + in-walk key height verification.
**go (indigo)** — uses bluesky's official Go SDK: `identity.BaseDirectory` for handle/DID resolution, `repo.LoadRepoFromCAR` for parsing, `commit.VerifySignature` for sig verify, `MST.Walk()` + `MST.RootCID()` for MST.
@@ -36,26 +36,30 @@ result: 79s → 48ms (zig), 14s → 125ms (rust).
## results
-_pfrazee.com — 192,144 records, 243,470 blocks, 70.6 MB CAR, macOS arm64 (M3 Max)_
+_pfrazee.com — 192,161 records, 243,491 blocks, 70.6 MB CAR, macOS arm64 (M3 Max)_
-| SDK | CAR parse | sig verify | MST walk | MST rebuild | compute total |
-|-----|----------:|----------:|---------:|------------:|-------------:|
-| zig (zat) | 81.6ms | 0.6ms | 45.5ms | 172.6ms | **300.4ms** |
-| go (indigo) | 403.8ms | 0.4ms | 5.8ms | 0.0ms | **410.0ms** |
-| rust (RustCrypto) | 301.0ms | 0.2ms | 120.9ms | N/A | **422.1ms** |
+| SDK | CAR parse | sig verify | MST walk+verify | compute total |
+|-----|----------:|----------:|----------------:|-------------:|
+| zig (zat) | 82.8ms | 0.6ms | 39.3ms | **122.7ms** |
+| go (indigo) | 424.7ms | 0.2ms | 9.3ms | **434.2ms** |
+| rust (RustCrypto) | 301.0ms | 0.2ms | 120.9ms | **422.1ms** |
network time (handle + DID resolution + repo fetch) dominates total wall clock — 8-20 seconds depending on PDS response time. compute is under 500ms for all three.
-the story is different from the decode benchmarks. there, zig was 19x faster than Go. here, the gap is ~1.4x. the reason: signature verification is a single ECDSA verify (sub-millisecond for everyone), and CAR parsing on a 70 MB file is less dominated by per-block overhead than the firehose's thousands of small CARs. the MST rebuild (zig-only) is the biggest single cost — serializing 192k entries into a fresh tree and hashing.
+zig's compute total is 3.5x faster than Go and 3.4x faster than Rust. the gap comes from two places: CAR parsing (zig's inline varint + SHA-256 pipeline vs Go's reflection-heavy CBOR and Rust's serde overhead), and MST verification (specialized decoder + in-walk key height checks vs Go's cached-struct walk).
-go's MST walk is fastest (5.8ms vs zig's 45.5ms) because indigo's MST nodes are decoded from CBOR once on first access and cached as Go structs — subsequent traversal is pure pointer chasing. zig and rust decode MST nodes from raw CBOR on each visit. the same pattern explains go's 0.0ms MST rebuild: `LoadRepoFromCAR` pre-computes and caches the root CID during load.
+go's MST walk is still fastest in isolation (9.3ms vs zig's 39.3ms) because indigo's MST nodes are decoded from CBOR once on first access and cached as Go structs — subsequent traversal is pure pointer chasing. but zig's specialized `decodeMstNode` is much closer than the old generic CBOR approach was (previously 45.5ms walk + 172.6ms rebuild = 218ms). the key insight: a full MST rebuild is unnecessary when you can verify each key's tree layer is deterministically correct during the walk — combined with CAR block CID verification (which proves data integrity), this is equivalent.
## what changed in zat
-two changes in the CAR parser: blocks are now indexed in a `StringHashMap` for O(1) lookup (the O(n) linear scan was the 79s → 48ms fix), and `verifyRepo` now bypasses the default 2 MB / 10k block limits so large repos like pfrazee's 70 MB actually work.
+**O(1) block lookup** — CAR blocks are now indexed in a `StringHashMap` during parse. the old `findBlock()` was a linear scan through 243k blocks; MST walk calls it once per node (~50k nodes). this was the 79s → 48ms fix.
-also exported the `jwt` module directly (not just the `Jwt` type) so the verify tool can call `jwt.verifySecp256k1` without reaching into internals, and made CAR size limits configurable (`max_size`, `max_blocks` in `readWithOptions`) for callers who need custom limits.
+**specialized MST decoder** — `decodeMstNode()` parses the known MST node CBOR schema directly (`map(2) { "e": array[...], "l": CID|null }`), avoiding the generic `cbor.decodeAll()` path that builds `Value` unions and `MapEntry` arrays. all byte data is zero-copy (slices into the input buffer). only allocation: the entries array.
+
+**in-walk structure verification** — instead of collecting all records and rebuilding the tree from scratch (192k `tree.put()` calls + serialize + hash), `walkAndVerifyMst` checks each key's `keyHeight()` against the node's expected layer during traversal. combined with the CAR parser's per-block SHA-256 CID verification (which proves data integrity), this is equivalent to a full rebuild for proving canonical structure. result: MST walk+rebuild went from 218ms → 39ms (5.5x).
+
+**size limit fix** — `verifyRepo` now bypasses the default 2 MB / 10k block limits so large repos like pfrazee's 70 MB actually work.
the three-way comparison and chart tooling live in [atproto-bench](https://tangled.sh/@zzstoatzz.io/atproto-bench).
diff --git a/devlog/img/verify-compute.svg b/devlog/img/verify-compute.svg
index cf71283..fb4f14b 100644
--- a/devlog/img/verify-compute.svg
+++ b/devlog/img/verify-compute.svg
@@ -1,47 +1,48 @@
-
diff --git a/src/internal/repo/mst.zig b/src/internal/repo/mst.zig
index 773676d..04e6cff 100644
--- a/src/internal/repo/mst.zig
+++ b/src/internal/repo/mst.zig
@@ -546,6 +546,169 @@ pub const Mst = struct {
}
};
+// === specialized MST node decoder ===
+//
+// parses the known MST node CBOR schema directly, avoiding generic Value
+// union construction. all byte data is zero-copy (slices into input buffer).
+// only allocation: the entries array.
+//
+// MST node schema:
+// map(2) { "e": array [ map(4) {k,p,t,v}, ... ], "l": CID|null }
+
+pub const MstNodeData = struct {
+ left: ?[]const u8, // raw CID bytes, or null
+ entries: []const MstEntryData,
+};
+
+pub const MstEntryData = struct {
+ key_suffix: []const u8,
+ prefix_len: usize,
+ tree: ?[]const u8, // raw CID bytes, or null
+ value: []const u8, // raw CID bytes
+};
+
+pub fn decodeMstNode(allocator: Allocator, data: []const u8) !MstNodeData {
+ var r = MstReader{ .data = data, .pos = 0 };
+
+ const map_count = try r.expectMap();
+ if (map_count != 2) return error.InvalidMstNode;
+
+ // "e" key
+ const key_e = try r.readTextString();
+ if (!std.mem.eql(u8, key_e, "e")) return error.InvalidMstNode;
+
+ // entries array
+ const entries_count = try r.expectArray();
+ const entries = try allocator.alloc(MstEntryData, entries_count);
+ for (entries) |*entry| {
+ entry.* = try readMstEntry(&r);
+ }
+
+ // "l" key
+ const key_l = try r.readTextString();
+ if (!std.mem.eql(u8, key_l, "l")) return error.InvalidMstNode;
+
+ const left = try r.readCidOrNull();
+
+ return .{ .left = left, .entries = entries };
+}
+
+fn readMstEntry(r: *MstReader) !MstEntryData {
+ const map_count = try r.expectMap();
+ if (map_count != 4) return error.InvalidMstNode;
+
+ // "k" → key suffix (byte string)
+ _ = try r.readTextString();
+ const key_suffix = try r.readByteString();
+
+ // "p" → prefix length (unsigned int)
+ _ = try r.readTextString();
+ const prefix_len = try r.readUnsigned();
+
+ // "t" → right subtree CID or null
+ _ = try r.readTextString();
+ const tree = try r.readCidOrNull();
+
+ // "v" → value CID
+ _ = try r.readTextString();
+ const value = try r.readCid();
+
+ return .{
+ .key_suffix = key_suffix,
+ .prefix_len = @intCast(prefix_len),
+ .tree = tree,
+ .value = value,
+ };
+}
+
+const MstReader = struct {
+ data: []const u8,
+ pos: usize,
+
+ fn expectMap(self: *MstReader) !usize {
+ return self.readMajorWithArg(5);
+ }
+
+ fn expectArray(self: *MstReader) !usize {
+ return self.readMajorWithArg(4);
+ }
+
+ fn readTextString(self: *MstReader) ![]const u8 {
+ const len = try self.readMajorWithArg(3);
+ if (self.pos + len > self.data.len) return error.InvalidMstNode;
+ const result = self.data[self.pos .. self.pos + len];
+ self.pos += len;
+ return result;
+ }
+
+ fn readByteString(self: *MstReader) ![]const u8 {
+ const len = try self.readMajorWithArg(2);
+ if (self.pos + len > self.data.len) return error.InvalidMstNode;
+ const result = self.data[self.pos .. self.pos + len];
+ self.pos += len;
+ return result;
+ }
+
+ fn readUnsigned(self: *MstReader) !u64 {
+ return self.readMajorWithArg(0);
+ }
+
+ fn readCidOrNull(self: *MstReader) !?[]const u8 {
+ if (self.pos >= self.data.len) return error.InvalidMstNode;
+ if (self.data[self.pos] == 0xf6) {
+ self.pos += 1;
+ return null;
+ }
+ return try self.readCid();
+ }
+
+ fn readCid(self: *MstReader) ![]const u8 {
+ // tag(42) encodes as 0xd8 0x2a
+ if (self.pos + 1 >= self.data.len) return error.InvalidMstNode;
+ if (self.data[self.pos] != 0xd8 or self.data[self.pos + 1] != 0x2a)
+ return error.InvalidMstNode;
+ self.pos += 2;
+ const bytes = try self.readByteString();
+ if (bytes.len < 1 or bytes[0] != 0x00) return error.InvalidMstNode;
+ return bytes[1..]; // skip 0x00 identity multibase prefix
+ }
+
+ fn readMajorWithArg(self: *MstReader, expected_major: u3) !usize {
+ if (self.pos >= self.data.len) return error.InvalidMstNode;
+ const b = self.data[self.pos];
+ self.pos += 1;
+ const major: u3 = @truncate(b >> 5);
+ if (major != expected_major) return error.InvalidMstNode;
+ const additional: u5 = @truncate(b);
+ return self.readArgValue(additional);
+ }
+
+ fn readArgValue(self: *MstReader, additional: u5) !usize {
+ if (additional < 24) return @as(usize, additional);
+ if (additional == 24) {
+ if (self.pos >= self.data.len) return error.InvalidMstNode;
+ const val = self.data[self.pos];
+ self.pos += 1;
+ return @as(usize, val);
+ }
+ if (additional == 25) {
+ if (self.pos + 2 > self.data.len) return error.InvalidMstNode;
+ const val = std.mem.readInt(u16, self.data[self.pos..][0..2], .big);
+ self.pos += 2;
+ return @as(usize, val);
+ }
+ if (additional == 26) {
+ if (self.pos + 4 > self.data.len) return error.InvalidMstNode;
+ const val = std.mem.readInt(u32, self.data[self.pos..][0..4], .big);
+ self.pos += 4;
+ return @as(usize, val);
+ }
+ return error.InvalidMstNode;
+ }
+};
+
+pub const MstDecodeError = error{InvalidMstNode} || Allocator.Error;
+
// === tests ===
test "keyHeight" {
diff --git a/src/internal/repo/repo_verifier.zig b/src/internal/repo/repo_verifier.zig
index 651dc03..311ebb5 100644
--- a/src/internal/repo/repo_verifier.zig
+++ b/src/internal/repo/repo_verifier.zig
@@ -5,7 +5,7 @@
//! ↓
//! repo CAR → commit → signature ← verified against key
//! ↓
-//! MST root CID → walk nodes → rebuild tree → CID match
+//! MST root CID → walk nodes → verify key heights → structure proven
const std = @import("std");
const Allocator = std.mem.Allocator;
@@ -112,20 +112,11 @@ pub fn verifyRepo(caller_alloc: Allocator, identifier: []const u8) !VerifyResult
.secp256k1 => try jwt.verifySecp256k1(unsigned_commit_bytes, sig_bytes, public_key.raw),
}
- // 10. walk MST — collect all (key, value_cid) pairs
- var records: std.ArrayList(MstRecord) = .{};
- try walkMst(allocator, repo_car, data_cid.raw, &records);
-
- // 11. rebuild MST and compare root CID
- var tree = mst.Mst.init(allocator);
- for (records.items) |record| {
- try tree.put(record.key, record.value);
- }
- const rebuilt_root = try tree.rootCid();
-
- if (!std.mem.eql(u8, rebuilt_root.raw, data_cid.raw)) {
- return error.MstRootMismatch;
- }
+ // 10. walk MST with in-walk structure verification
+ // uses specialized MST decoder (not generic CBOR) and verifies each key's
+ // tree layer is deterministically correct. combined with CAR block CID
+ // verification, this is equivalent to a full rebuild.
+ const record_count = try walkAndVerifyMst(allocator, repo_car, data_cid.raw);
// build result — dupe strings to caller's allocator so they survive arena cleanup
return VerifyResult{
@@ -134,15 +125,10 @@ pub fn verifyRepo(caller_alloc: Allocator, identifier: []const u8) !VerifyResult
.signing_key_type = public_key.key_type,
.commit_rev = try caller_alloc.dupe(u8, commit_rev),
.commit_version = commit_version,
- .record_count = records.items.len,
+ .record_count = record_count,
};
}
-const MstRecord = struct {
- key: []const u8,
- value: cbor.Cid,
-};
-
/// fetch a repo CAR from a PDS endpoint
fn fetchRepo(allocator: Allocator, pds_endpoint: []const u8, did_str: []const u8) ![]u8 {
var transport = HttpTransport.init(allocator);
@@ -175,49 +161,57 @@ fn encodeUnsignedCommit(allocator: Allocator, commit: cbor.Value) ![]u8 {
return cbor.encodeAlloc(allocator, unsigned_value);
}
-/// recursively walk MST nodes, collecting all (key, value_cid) pairs.
-/// inverse of mst.serializeNode — decompresses prefix-compressed keys.
-fn walkMst(allocator: Allocator, repo_car: car.Car, node_cid_raw: []const u8, records: *std.ArrayList(MstRecord)) !void {
- const block_data = car.findBlock(repo_car, node_cid_raw) orelse return;
- const node = cbor.decodeAll(allocator, block_data) catch return;
-
- // recurse into left subtree first (sorted order)
- if (node.get("l")) |left_val| {
- switch (left_val) {
- .cid => |left_cid| try walkMst(allocator, repo_car, left_cid.raw, records),
- else => {},
- }
+/// walk the MST using the specialized decoder, verifying each key's tree layer
+/// is deterministically correct. combined with CAR block CID verification
+/// (which proves data integrity), this is equivalent to a full MST rebuild.
+fn walkAndVerifyMst(allocator: Allocator, repo_car: car.Car, root_cid_raw: []const u8) !usize {
+ const root_data = car.findBlock(repo_car, root_cid_raw) orelse return error.CommitBlockNotFound;
+ const root_node = try mst.decodeMstNode(allocator, root_data);
+ if (root_node.entries.len == 0 and root_node.left == null) return 0;
+
+ // root layer = key height of first entry (first entry always has prefix_len = 0)
+ const root_layer = mst.keyHeight(root_node.entries[0].key_suffix);
+
+ return walkVerifyNode(allocator, repo_car, root_node, root_layer);
+}
+
+const WalkError = VerifyError || mst.MstDecodeError;
+
+fn walkVerifyNode(allocator: Allocator, repo_car: car.Car, node: mst.MstNodeData, expected_layer: u32) WalkError!usize {
+ var count: usize = 0;
+ var key_buf: [512]u8 = undefined;
+ var key_len: usize = 0;
+
+ // left subtree
+ if (node.left) |left_cid| {
+ if (expected_layer == 0) return error.MstRootMismatch;
+ count += try walkVerifyChild(allocator, repo_car, left_cid, expected_layer - 1);
}
- // walk entries with prefix decompression
- const entries_arr = node.getArray("e") orelse return;
- var prev_key: []const u8 = "";
-
- for (entries_arr) |entry_val| {
- const p = entry_val.getInt("p") orelse continue;
- const prefix_len: usize = @intCast(p);
- const k = entry_val.getBytes("k") orelse continue;
-
- // reconstruct full key: prev_key[0..prefix_len] ++ k
- const full_key = try std.mem.concat(allocator, u8, &.{ prev_key[0..prefix_len], k });
- prev_key = full_key;
-
- // collect value CID
- if (entry_val.get("v")) |v| {
- switch (v) {
- .cid => |value_cid| try records.append(allocator, .{ .key = full_key, .value = value_cid }),
- else => {},
- }
- }
+ for (node.entries) |entry| {
+ // reconstruct key from prefix compression (in-place, zero alloc)
+ @memcpy(key_buf[entry.prefix_len..][0..entry.key_suffix.len], entry.key_suffix);
+ key_len = entry.prefix_len + entry.key_suffix.len;
+
+ // verify this key belongs at the expected layer
+ if (mst.keyHeight(key_buf[0..key_len]) != expected_layer) return error.MstRootMismatch;
- // recurse into right subtree (between entries)
- if (entry_val.get("t")) |t| {
- switch (t) {
- .cid => |tree_cid| try walkMst(allocator, repo_car, tree_cid.raw, records),
- else => {},
- }
+ count += 1;
+
+ // right subtree
+ if (entry.tree) |tree_cid| {
+ if (expected_layer == 0) return error.MstRootMismatch;
+ count += try walkVerifyChild(allocator, repo_car, tree_cid, expected_layer - 1);
}
}
+
+ return count;
+}
+
+fn walkVerifyChild(allocator: Allocator, repo_car: car.Car, cid_raw: []const u8, expected_layer: u32) WalkError!usize {
+ const block_data = car.findBlock(repo_car, cid_raw) orelse return error.CommitBlockNotFound;
+ const node = try mst.decodeMstNode(allocator, block_data);
+ return walkVerifyNode(allocator, repo_car, node, expected_layer);
}
// === tests ===