From 9afa0bf5b6cac79f816c303ce8dfe7deec1ef98a Mon Sep 17 00:00:00 2001 From: zzstoatzz Date: Wed, 8 Jul 2026 23:27:47 -0500 Subject: [PATCH] docs: jss seal + gloom construction spec (upstream review) Co-Authored-By: Claude Fable 5 --- docs/jss-seal-spec.md | 52 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) create mode 100644 docs/jss-seal-spec.md diff --git a/docs/jss-seal-spec.md b/docs/jss-seal-spec.md new file mode 100644 index 0000000..60b4ded --- /dev/null +++ b/docs/jss-seal-spec.md @@ -0,0 +1,52 @@ +# jss v1 sealing + gloom construction (from upstream @ 0c45f29 + gloom v0.1.0) + +what the writer's seal path must produce, bit-exact. complements docs/jss-format-v1.md. + +## gloom filter construction (bit-exact or their reader rejects / false-negatives) +- OptimalParams(n, p): clamp n==0→1, p≤0→0.0001, p≥1→0.99. bitsPerItem = -ln(p)/0.4804530139182014; + totalBits = n·bitsPerItem; baseBlocks = ceil(totalBits/512); maxBlocks = ceil(baseBlocks·4). + numBlocks = baseBlocks; k = bestK; while EstimateFPR(numBlocks,k,n) > p and numBlocks < maxBlocks: + numBlocks = min(numBlocks + numBlocks/8 + 1, maxBlocks) (integer div), re-pick k. +- bestK: scan k=3..17, pick k minimizing EstimateFPR (strict < → lowest k wins). +- EstimateFPR: lambda = n/numBlocks; if lambda>10000 → partitionedBlockFP(primes, lambda); + else Poisson sum j=0..(lambda+10√lambda+20), logProb = -λ + j·ln λ − Σln(j), break when + prob<1e-15 and j>int(λ); Σ prob·partitionedBlockFP(primes,j). partitionedBlockFP(primes,j) = + ∏(1−(1−1/pᵢ)^j). FLOAT-SENSITIVE: golden-test (n=4096,p=0.001) + a sweep vs Go output. +- prime partitions per k (sum 512 each) — copy tables verbatim from gloom params.go:40-56. + offsets[i] = prefix sum of primes. +- Add: h = xxh3_128(did); blockIdx = mulhi64(h.hi, numBlocks) (Lemire, HIGH word); + intra: u32 = trunc(h.lo) ^ trunc(h.lo>>32); for i in 0..k: bit = offsets[i] + intra % primes[i]; + set bit (bitPos/64 word, bitPos%64) in blocks[blockIdx·8..]. count += 1 per Add. +- MarshalBinary (LE): version u8=1 | k u32 | numBlocks u64 | count u64 | numBlocks×64B words | + crc32c(Castagnoli) u32 over all preceding bytes. count = unique DIDs added (dedup'd!). +- constructors coerce numBlocks 0→1; unsupported k → 7; reader rejects version≠1, k∉3..17, + numBlocks 0 or >1<<50, length mismatch, crc mismatch. + +## seal sizing + sequence +- segment bloom: gloom.New(n = unique non-empty DIDs in segment, p=0.001); one AddString per unique DID. + zero DIDs → still written (n coerced to 1, count=0). +- per-block blooms: gloom.New(n = configured MaxEventsPerBlock (4096 default), p=0.001) — all identical + size (region indexable by multiplication); populated with the block's unique non-empty DIDs. + zero blocks → 8-byte region header only (0,0). +- empty-string DIDs excluded from blooms and unique_did_count. +- collection ids: FIRST-SEEN order during the frame walk. real collections intern with count++ per event; + kinds account/identity/sync ALSO intern sentinels "$account"/"$identity"/"$sync" inline (taking the next + first-seen id, interleaved not appended) with count NOT incremented. +- walk: frames from 256 to footer_offset; block decoding to 0 events terminates the walk. +- footer order: block index (at footer_offset == block_index_offset) | segment bloom | per-block bloom + region (8B header + n×size) | collection index (16B header + zstd body: table in id order + [len u8][count u32][nsid], then per-block bitmasks). +- header: min/max seq + witnessed_at all 0 when event_count==0. checksum = xxh3_64 of + header[12..256] (checksum field zeroed) ++ file[footer_offset..EOF] — covers to EOF, no slack + bytes allowed anywhere. +- io order: pwrite footer, fsync, pwrite header, fsync, close. + +## reader validations ours must survive +- offsets monotone: 256 ≤ footer ≤ size; block_index==footer ≤ did_bloom ≤ block_blooms ≤ + collection_index ≤ size; bloom region size must equal collection_index_offset − block_blooms_offset + exactly (8 + n×size). +- blocks: within [256, footer), non-overlapping, offset-monotonic, MaxSeq≥MinSeq, and non-empty blocks + strictly increasing seq ranges (min > previous max). +- VerifySealedMetadata re-derives EVERYTHING from rows: exact counts, envelopes, collection table + (no dups, exact counts, sentinel presence both directions), bitmask sets exactly equal row-derived, + and NO false negatives in either bloom for any row DID. -- 2.51.2