jss sealed segment format v1 (byte-exact, from upstream source @ 0c45f29) #

condensed from bluesky-social/jetstream segment/*.go + §3.1–3.4 of upstream's docs/README.md (2026-07-08). Those are upstream Go paths and upstream's README — neither is in this repository. this format is wire-visible contract (getSegment/getBlock serve raw bytes) — implement as-specified. all integers little-endian, all offsets absolute. extension .jss.

This file pins a different upstream commit than the rest of the docs. The title says 0c45f29; semantic-parity.md, upstream-bootstrap-spec.md, configuration-parity.md, upstream-harness.md, bootstrap-semantic-parity.md, opentelemetry.md and grafana-dashboard.md all pin f29815c391fc2644f8a3dd36b899fb3697dd1ea6. jss-seal-spec.md is the only other file on 0c45f29. This spec has not been re-checked against f29815c; every constant below matches storage/segment.zig.

Verified against storage/segment.zig on 2026-07-30: magic "jss0", header_size = 256, the four reader sanity caps (1 << 18, 1 << 20, 1 << 20, 1 << 30), and 4096 events/block all match the values below.

constants #

• magic "jss0"; header 256 bytes; header version 1
• block index entry 52 bytes; default 4096 events/block
• kinds 1..7: create, update, delete, identity, account, sync, create_resync
• reader sanity caps: block event_count ≤ 1<<18; block_count ≤ 1<<20; collection_count ≤ 1<<20; decompressed block ≤ 1 GiB
• field maxes: did ≤ 65535, collection/rkey/rev ≤ 255, payload ≤ u32

header (256 bytes at offset 0) #

checksum = xxh3_64(seed 0) over header[12..256] || file[footer_offset..EOF].

block frames (256 .. footer_offset) #

back-to-back: block_len u64 then block_len bytes = one plain zstd frame (content CRC enabled by writer; NO dictionary). block N via block-index entry N (offset points at the length prefix; frame at offset+8). sequential walk possible without footer. compaction can leave empty blocks: uncompressed body exactly 00 00 00 00 (event_count 0).

columnar block layout (decompressed) #

event_count u32
seq[]            n × u64
witnessed_at[]   n × i64 µs
indexed_at[]     n × i64 µs (0 = unset → display falls back to witnessed_at)
kind[]           n × u8 (1..7)
collection_len[] n × u8
did_len[]        n × u16
rkey_len[]       n × u8
rev_len[]        n × u8
event_len[]      n × u32 (payload)
collections blob | dids blob | rkeys blob | revs blob | payloads blob (raw DAG-CBOR)

strings sliced by prefix sums; total must be exact (trailing bytes = error).

footer (footer_offset .. EOF), four sections, located via header offsets only #

1. block index (at block_index_offset): block_count × 52B: offset u64 (of length prefix), compressed_size u32, uncompressed_size u32, event_count u32, min_seq u64, max_seq u64, min_witnessed_at i64, max_witnessed_at i64
2. segment DID bloom (did_bloom_offset): one gloom v0.1.0 MarshalBinary blob (version u8=1, k u32, num_blocks u64, count u64, blocks n×64B, crc32c u32; xxh3-128 hashing, Lemire block reduction). pinned gloom version = part of the format.
3. per-block DID blooms (block_did_bloom_offset): block_count u32, bloom_size_bytes u32, then block_count × bloom_size_bytes gloom blobs.
4. collection block index (collection_index_offset .. EOF): 16B header (collection_count u32, block_count u32, bitmask_len u32 == ceil(cc/8), uncompressed_size u32) + one zstd frame body: collection table (len u8, count u32, nsid [len]u8 — interleaved) then block bitmasks (block_count × bitmask_len; bit N ⇒ collection id N present in block). sentinel "collections" $account/$identity/$sync tag DID-level marker events; their count stays 0.

minimal reader may skip both blooms and the collection index: header + frame walk (or block index) + zstd + columnar decode iterates everything. blooms/collection index are no-false-negative query accelerators.

sealed vs active #

active: magic + 252 zero bytes, then fully-flushed frames, no footer. checksum field 0 = active. sealing: footer_offset = size; walk frames for stats; write footer; fsync; pwrite header; fsync. per-block integrity = zstd frame CRC only (file xxh3 covers header+footer, not block bytes). Stream decodes through its pinned vendored libzstd, which verifies that content checksum. The upstream-produced fixture mutation test flips only a block checksum byte, proves the JSS metadata checksum still passes, and requires block decode to fail.

ops notes #

• no CLI flag forces small segments; sealed files come from the bootstrap→merge cutover (--max-backfill-repos N gives a small sealed set). steady-state shutdown does NOT seal.
• paths: <data-dir>/segments/seg_<10-char base36>.jss (+ Stream's bootstrap capture at backfill/live/segments/; upstream uses backfill/live_segments/)
• ground-truth tooling: jetstream inspect-segment <file> --blocks=full, inspect-all