# jss sealed segment format v1 (byte-exact, from upstream source @ 0c45f29)

condensed from `bluesky-social/jetstream` `segment/*.go` + §3.1–3.4 of upstream's
`docs/README.md` (2026-07-08). **Those are upstream Go paths and upstream's
README — neither is in this repository.**
this format is wire-visible contract (getSegment/getBlock serve raw bytes) — implement as-specified.
all integers little-endian, all offsets absolute. extension `.jss`.

> **This file pins a different upstream commit than the rest of the docs.** The
> title says `0c45f29`; `semantic-parity.md`, `upstream-bootstrap-spec.md`,
> `configuration-parity.md`, `upstream-harness.md`,
> `bootstrap-semantic-parity.md`, `opentelemetry.md` and
> `grafana-dashboard.md` all pin `f29815c391fc2644f8a3dd36b899fb3697dd1ea6`.
> `jss-seal-spec.md` is the only other file on `0c45f29`. This spec has not
> been re-checked against `f29815c`; every constant below matches
> `storage/segment.zig`.

Verified against `storage/segment.zig` on 2026-07-30: magic `"jss0"`,
`header_size = 256`, the four reader sanity caps (`1 << 18`, `1 << 20`,
`1 << 20`, `1 << 30`), and 4096 events/block all match the values below.

## constants
- magic `"jss0"`; header 256 bytes; header version 1
- block index entry 52 bytes; default 4096 events/block
- kinds 1..7: create, update, delete, identity, account, sync, create_resync
- reader sanity caps: block event_count ≤ 1<<18; block_count ≤ 1<<20; collection_count ≤ 1<<20; decompressed block ≤ 1 GiB
- field maxes: did ≤ 65535, collection/rkey/rev ≤ 255, payload ≤ u32

## header (256 bytes at offset 0)
| off | size | field |
|---|---|---|
| 0 | 4 | magic "jss0" |
| 4 | 8 | checksum u64 xxh3; **0 ⇒ active/unsealed** |
| 12 | 2 | version u16 = 1 |
| 14 | 4 | block_count u32 |
| 18 | 4 | event_count u32 |
| 22 | 4 | unique_did_count u32 |
| 26 | 8 | min_seq u64 |
| 34 | 8 | max_seq u64 |
| 42 | 8 | min_witnessed_at i64 µs |
| 50 | 8 | max_witnessed_at i64 µs |
| 58 | 8 | footer_offset u64 |
| 66 | 8 | did_bloom_offset u64 |
| 74 | 8 | block_did_bloom_offset u64 |
| 82 | 8 | collection_index_offset u64 |
| 90 | 8 | block_index_offset u64 (== footer_offset, enforced) |
| 98 | 158 | reserved, zeroed (hashed!) |

checksum = xxh3_64(seed 0) over `header[12..256] || file[footer_offset..EOF]`.

## block frames (256 .. footer_offset)
back-to-back: `block_len u64` then `block_len` bytes = one plain zstd frame (content CRC
enabled by writer; NO dictionary). block N via block-index entry N (offset points at the
length prefix; frame at offset+8). sequential walk possible without footer.
compaction can leave empty blocks: uncompressed body exactly `00 00 00 00` (event_count 0).

## columnar block layout (decompressed)
```
event_count u32
seq[]            n × u64
witnessed_at[]   n × i64 µs
indexed_at[]     n × i64 µs (0 = unset → display falls back to witnessed_at)
kind[]           n × u8 (1..7)
collection_len[] n × u8
did_len[]        n × u16
rkey_len[]       n × u8
rev_len[]        n × u8
event_len[]      n × u32 (payload)
collections blob | dids blob | rkeys blob | revs blob | payloads blob (raw DAG-CBOR)
```
strings sliced by prefix sums; total must be exact (trailing bytes = error).

## footer (footer_offset .. EOF), four sections, located via header offsets only
1. **block index** (at block_index_offset): block_count × 52B:
   offset u64 (of length prefix), compressed_size u32, uncompressed_size u32, event_count u32,
   min_seq u64, max_seq u64, min_witnessed_at i64, max_witnessed_at i64
2. **segment DID bloom** (did_bloom_offset): one gloom v0.1.0 MarshalBinary blob
   (version u8=1, k u32, num_blocks u64, count u64, blocks n×64B, crc32c u32; xxh3-128 hashing,
   Lemire block reduction). pinned gloom version = part of the format.
3. **per-block DID blooms** (block_did_bloom_offset): block_count u32, bloom_size_bytes u32,
   then block_count × bloom_size_bytes gloom blobs.
4. **collection block index** (collection_index_offset .. EOF):
   16B header (collection_count u32, block_count u32, bitmask_len u32 == ceil(cc/8),
   uncompressed_size u32) + one zstd frame body: collection table (len u8, count u32,
   nsid [len]u8 — interleaved) then block bitmasks (block_count × bitmask_len; bit N ⇒
   collection id N present in block). sentinel "collections" $account/$identity/$sync tag
   DID-level marker events; their count stays 0.

**minimal reader** may skip both blooms and the collection index: header + frame walk (or
block index) + zstd + columnar decode iterates everything. blooms/collection index are
no-false-negative query accelerators.

## sealed vs active
active: magic + 252 zero bytes, then fully-flushed frames, no footer. checksum field 0 = active.
sealing: footer_offset = size; walk frames for stats; write footer; fsync; pwrite header; fsync.
per-block integrity = zstd frame CRC only (file xxh3 covers header+footer, not block bytes).
Stream decodes through its pinned vendored libzstd, which verifies that content checksum.
The upstream-produced fixture mutation test flips only a block checksum byte, proves the
JSS metadata checksum still passes, and requires block decode to fail.

## ops notes
- no CLI flag forces small segments; sealed files come from the bootstrap→merge cutover
  (`--max-backfill-repos N` gives a small sealed set). steady-state shutdown does NOT seal.
- paths: `<data-dir>/segments/seg_<10-char base36>.jss` (+ Stream's bootstrap capture at `backfill/live/segments/`; upstream uses `backfill/live_segments/`)
- ground-truth tooling: `jetstream inspect-segment <file> --blocks=full`, `inspect-all`