From 014730e0439350c0e9b3aa5e8e4b23e4c845be35 Mon Sep 17 00:00:00 2001 From: zzstoatzz Date: Thu, 30 Jul 2026 02:57:41 -0500 Subject: [PATCH] docs: give the seal spec a way in The file is bit-exact reference material and worth keeping dense, but it used "gloom" thirty times without ever saying it is a bloom filter library, never said which upstream it tracks, and never said why bit-exactness matters rather than mere compatibility. A reader arriving cold has no entry point. Adds a short orientation block -- what jss, gloom and upstream are, and the reason the constants are copied verbatim: a bloom filter promises no false negatives, so a single misplaced bit makes a reader skip a segment that does hold the DID, and nothing in the format detects it. The spec content is unchanged. Co-Authored-By: Claude Opus 5 (1M context) --- docs/jss-seal-spec.md | 26 +++++++++++++++++++++++++- 1 file changed, 25 insertions(+), 1 deletion(-) diff --git a/docs/jss-seal-spec.md b/docs/jss-seal-spec.md index 60b4ded..91a4048 100644 --- a/docs/jss-seal-spec.md +++ b/docs/jss-seal-spec.md @@ -1,6 +1,30 @@ # jss v1 sealing + gloom construction (from upstream @ 0c45f29 + gloom v0.1.0) -what the writer's seal path must produce, bit-exact. complements docs/jss-format-v1.md. +what the writer's seal path must produce, bit-exact. complements +[jss-format-v1.md](jss-format-v1.md), which describes the format at rest; this +file describes how to *write* it. + +## if you are reading this cold + +- **jss** is Stream's segment file format — the on-disk columnar container the + archive is made of. A *sealed* segment is one that is finished and immutable; + see [invariants.md](invariants.md). +- **gloom** is a blocked-bloom-filter library ([jcalabro/gloom](https://github.com/jcalabro/gloom), + pinned at v0.1.0). Its serialized filters are embedded in sealed jss footers, + one per segment and one per block, so a reader can ask "might this segment + contain this DID?" without opening the segment body. +- **upstream** throughout this file means Bluesky's Jetstream V2 + (`bluesky-social/jetstream`), pinned at the commit in the title. + +**Why bit-exact and not merely compatible.** A bloom filter's contract is *no +false negatives*: if it says a DID is absent, the reader skips the segment +without looking. Place one bit differently from the Go implementation and that +promise breaks silently — a reader skips a segment that does hold the DID, and +the data looks lost rather than mis-indexed. Nothing in the format detects it. +That is why the constants below are copied verbatim rather than re-derived, and +why the parameter maths is pinned by golden vectors generated from the Go +library: the float arithmetic is sensitive enough that an independent +implementation can land on different block counts. ## gloom filter construction (bit-exact or their reader rejects / false-negatives) - OptimalParams(n, p): clamp n==0→1, p≤0→0.0001, p≥1→0.99. bitsPerItem = -ln(p)/0.4804530139182014; -- 2.51.2