diff --git a/docs/jss-seal-spec.md b/docs/jss-seal-spec.md index 60b4ded..91a4048 100644 --- a/docs/jss-seal-spec.md +++ b/docs/jss-seal-spec.md @@ -1,6 +1,30 @@ # jss v1 sealing + gloom construction (from upstream @ 0c45f29 + gloom v0.1.0) -what the writer's seal path must produce, bit-exact. complements docs/jss-format-v1.md. +what the writer's seal path must produce, bit-exact. complements +[jss-format-v1.md](jss-format-v1.md), which describes the format at rest; this +file describes how to *write* it. + +## if you are reading this cold + +- **jss** is Stream's segment file format — the on-disk columnar container the + archive is made of. A *sealed* segment is one that is finished and immutable; + see [invariants.md](invariants.md). +- **gloom** is a blocked-bloom-filter library ([jcalabro/gloom](https://github.com/jcalabro/gloom), + pinned at v0.1.0). Its serialized filters are embedded in sealed jss footers, + one per segment and one per block, so a reader can ask "might this segment + contain this DID?" without opening the segment body. +- **upstream** throughout this file means Bluesky's Jetstream V2 + (`bluesky-social/jetstream`), pinned at the commit in the title. + +**Why bit-exact and not merely compatible.** A bloom filter's contract is *no +false negatives*: if it says a DID is absent, the reader skips the segment +without looking. Place one bit differently from the Go implementation and that +promise breaks silently — a reader skips a segment that does hold the DID, and +the data looks lost rather than mis-indexed. Nothing in the format detects it. +That is why the constants below are copied verbatim rather than re-derived, and +why the parameter maths is pinned by golden vectors generated from the Go +library: the float arithmetic is sensitive enough that an independent +implementation can land on different block counts. ## gloom filter construction (bit-exact or their reader rejects / false-negatives) - OptimalParams(n, p): clamp n==0→1, p≤0→0.0001, p≥1→0.99. bitsPerItem = -ln(p)/0.4804530139182014;