# Compression verification

Current behavior: v0.1.5 supports omitted-size archive and live frames.
Sequence resume cursors seed live deduplication; timestamp cursors do not.
The release suite passes 73 tests. See [client.md](client.md) for the current
consumer contract. The dated checks and test counts below record the earlier
review stages, not the current suite size or deployed server state.

## Initial dictionary-compression verification, September 2026

The unified client enables dictionary compression by default and supplies the
refresh callback to both live-only and archive-cutover consumers. Default
negotiation and refresh tests failed before their respective fixes. The 57-test
client suite passes, including replacement dictionary, unchanged dictionary,
invalid/unavailable dictionary, and explicit disable. A ten-event live failover
smoke against stream.waow.tech passes with this candidate.

The archive server's rejection previously used plain text. A real local
WebSocket upgrade regression fails on the old content type and passes with
HTTP 400 and the UnknownZstdDictionary JSON envelope. Its local build passes.
The server fix is commit `d0091f9`. All 21 full admission suites passed on
2026-09-06. It was deployed on 2026-09-08 using the checksummed Docker-save
transfer; its admitted config digest is
`sha256:2841d664a9b5639a6f8430b777e1d280fe2bf301a4dbb3d2bf61022cc81dee68`.
The destination Docker reports the enclosing OCI manifest digest
`sha256:c331e7b502c95f3a07af3b25101ce01b2e4a87ef9c2d5c3c5c3cc6aa608dddd4`;
the manifest, config, and every layer were hash-verified before restart.

The performance workload and raw results belong to atproto-bench under
jetstream-compression. The 20-pair quiet baseline uses a synthetic corpus and
must not be presented as a production compression ratio. No allocation reuse
optimization has been made.

## Omitted live frame sizes, 2026-09-08

The live dictionary decoder now also accepts frames without declared content
size. It uses libzstd's decompression bound, capped by the existing read limit,
and returns only the actual decoded bytes. A conservative bound larger than
the limit does not itself reject a frame: actual output must fit. Declared-size
frames retained their existing fast path; that change left archive decoding
unchanged. Omitted-size archive support subsequently shipped in v0.1.5.

Two regressions failed first with UnknownFrameSize. Tests cover exact-limit
output, excess output, bad checksums, truncation, and decoder reuse after errors.
A third fixture was generated with klauspost/compress v1.19.2's streaming
writer (Write, Flush, Close) and independently decoded with upstream's
DecodeAll configuration. Its 8 MiB window produces only 448 bytes; the Zig
client accepts it at a 448-byte limit and rejects it at 447 bytes.
All 60 tests and the examples build pass. No consumer API change is required.
This closes the omitted-size live-frame gap, not a claim to cover every zstd
encoding or resource-policy difference.

## Production verification, 2026-09-08

Production reports build `d0091f9`. A genuine HTTP/1.1 WebSocket upgrade
with dictionary ID 1 returns HTTP 400, application/json, and
`UnknownZstdDictionary`, naming current dictionary 20260811. The candidate
client received ten consecutive events after live host failover with default
compression enabled (24123007708..24123007717, 2375 ms). Rotation and
same-dictionary fallback are exercised by loopback tests; the production
probe did not rotate the server dictionary.

The client suite passes 57/57 tests; examples build and formatting passes.
`zig zen` review: dictionary ownership and cleanup remain explicit, refresh
uses the selected live host, and this change adds no per-event allocation.
The existing decompressed event allocation is unchanged.

Post-deploy cadence checks each ran 60 seconds: live attach delivered 79,148
messages; ten-second rewind delivered 50,624. Both had one connection, zero
disconnects/errors, and increasing message counts in every interval (the
reporter prints 0/s for its initial baseline). Upstream sequence advanced
from 33,455,790,952 to 33,455,832,751. Observed RSS was 4.19 GB; drop and
compaction-error counters were zero.

## Final live-decoder audit, 2026-09-08

References: official Jetstream `58c4d7f7a9130e53b40348ad3d1f7aafed0e4843`,
its Atmos v0.4.0 dependency, and klauspost/compress v1.19.2. Atmos supplies
HTTP retry policy; Jetstream live.go and klauspost's DecodeAll/framedec.go
supply the live compression contract.

Two additional regressions failed first: concatenated frames were rejected,
and a 2 KiB window with four output bytes was accepted under a 1 KiB limit.
The decoder now walks all zstd/skippable frames, bounds aggregate output,
and checks every window against min(readLimit, 512 MiB), with upstream's
1 KiB minimum effective window. This supersedes the earlier 448-byte-limit
acceptance experiment: upstream also rejects a large window even when actual
output is small. Omitted-size frames remain supported within both limits.

A loopback test verifies malformed compressed messages invoke onError:
accepting delivers the next valid event on the same connection; declining
stops without delivering it. That callback behavior needed no code change.
All 63 tests pass.

Raw-record scope is not a gap: upstream options.go explicitly restricts
WithRawRecords to archive CBOR decoding. Live events remain JSON. The SDK's
archive-only decode_records=false option follows that scope; callback-borrowed
bytes must be copied when retained. Existing malformed-record/delete coverage
passes. No additional public raw-record API is required for this release.

The separate HTTP-library follow-up must compare Atmos v0.4.0's complete
header contract: RateLimit-Reset takes precedence over Retry-After and is a
signed Unix timestamp. The current HTTP helper prioritizes Retry-After and
also interprets small reset values as relative delays. This is broader than
the previously recorded negative-value edge and is not fixed in this release.

The signed-reset follow-up is adopted in SDK 0.1.4 via ZAT 0.5.2 and ZTTP
0.1.2. The SDK already enforced Atmos reset precedence and Unix-time semantics;
its new regression verifies a negative reset remains authoritative over
Retry-After. ZTTP retains its separate generic delay helper policy.

The atproto-bench differential lane identified two additional live API gaps:
resume-cursor boundary filtering and missing dictionary-rejection callbacks.
Both regressions failed before fixes. The sequence-resume fix seeded the
dedup floor from max(explicit floor, resume cursor); v0.1.5 limits that rule
to sequence cursors, excluding timestamp wire cursors. Dictionary rejection
reports UnknownZstdDictionary after refresh, honoring callback refusal
before reconnect. That stage passed 66 tests.