# benchmarks

`bench/` contains local benchmarks for ZDS storage paths. The goal is to track
scaling curves and compare protocol work against reference implementations, not
to produce a single universal score.

The benchmark style follows [`atproto-bench`](https://tangled.org/zzstoatzz.io/atproto-bench):
state the exact work being measured, keep correctness work visible, compare
equivalent operations, and treat run-to-run variance as real.

## run

```sh
just bench all
just bench write
just bench read
just bench repo
just bench blob
just bench blob-gc 512 65536
just bench metastore 10 1000
just bench get-cid 10 1000
just bench get-block 10 1000
just bench decode-record 10 1000
just bench render-record 10 1000
just bench get-record 10 1000
just bench list-records 10 1000
just bench repo-size 5000
just bench get-repo 5000 4 5
just bench firehose 1000 32 100
just bench publish-sync 100 3 3 3145728
just bench fly-upload
just bench repo-large 100000
just bench space 1000
just bench http
just bench startup 5 10000
just bench official-pds
just bench write-profile 10 500
just bench run --scenario write --records 10000
```

`just bench ...` is a Just module declared from the repository root with
`mod bench`.

## current scenarios

- `write`: `applyWrites` creating `app.bsky.feed.post` records.
- `read`: repeated `listRecords` queries against one seeded repo.
- `repo`: full repo CAR materialization through `writeRepoCar`.
- `blob`: blob write/read against disk blobstore plus SQLite metadata.
- `blob-gc`: bounded deletion of untethered blob metadata and disk files.
- `metastore`: Tranquil-shaped apply/get/list benchmark with caller counts and
  latency percentiles.
- `get-cid`: CID-only record lookup, matching Tranquil's `get_record_cid`
  metastore benchmark.
- `get-block`: record index lookup plus `repo_blocks` byte fetch.
- `decode-record`: DAG-CBOR decode of one seeded record block, without SQLite.
- `render-record`: DAG-CBOR decode plus JSON rendering, without SQLite.
- `get-record`: full `com.atproto.repo.getRecord` storage materialization.
- `list-records`: full `com.atproto.repo.listRecords` storage materialization.
- `repo-size`: synthetic small/medium/large repo probes for
  `listRecords(limit=100)` and full repo CAR materialization. The default Just
  target keeps this moderate; use `repo-large` for intentional 100k+ record
  runs.
- `get-repo`: focused `com.atproto.sync.getRepo` storage-pressure probe. It
  measures full repo CAR construction, `since` diff CAR construction after one
  additional commit, and concurrent full exports against one seeded repo. Its
  isolation lane runs account lookups and writes during those exports so a
  throughput improvement cannot hide resident-facing database stalls.
- `firehose`: runs the maximum 32 synthetic `subscribeRepos` pollers while
  measuring resident account-read latency. Event-history reads must remain
  ordinary indexed range reads; migration or repair work in this path causes
  global database-lock contention that stalls login, writes, and blob reads.
- `publish-sync`: mixed workload modeled on continuous media publishers. It
  runs concurrent multi-megabyte blob uploads and same-repo writes alongside
  blob/repo sync reads, while measuring a small metadata lookup for unrelated
  tail-latency stalls.
- `fly-upload`: end-to-end upload/write/read pressure on an isolated,
  production-shaped Fly Machine. A separate load-generator Machine logs in 100
  synthetic accounts, publishes blobs and records at staged concurrency, and
  records collateral authenticated-read latency. The temporary Fly app and all
  of its Machines are deleted after the run.
- `space`: experimental permissioned-data storage probes for space discovery,
  private record writes, private record reads/lists, repo oplog catch-up, and
  blob readback.
- `http`: starts a local ReleaseFast ZDS server and measures a seeded HTTP
  route matrix with ApacheBench. This is the route-level check for the `httpz`
  server boundary and should be tracked separately from storage
  microbenchmarks.
- `official-pds`: copies the official-PDS read probe into a local
  `bluesky-social/atproto` checkout and runs it with Vitest.
- `write-profile`: `applyWrites` stage timing under write concurrency.

Each run uses temporary SQLite and blobstore state.

## permissioned data

Permissioned-data benchmarks are separate from the public repo/appview/sync
benchmarks. The goal is not broad coverage yet; it is to catch obviously bad
access patterns introduced by the experimental `com.atproto.space.*` storage
model.

`just bench space` seeds one self-owned space, one writer repo, and one small
blob, then measures these paths as distinct units of work:

- `listSpaces`
- `createRecord` into a space writer repo
- `getRecord` by `(space, repo, collection, rkey)`
- `listRecords`, limit 50, with values included by default
- full-state `getRepo` CAR construction, including signed commit and record index
- full-state permissioned repo CAR verification for migration and backfill
- `getBlob` exact-space reference authorization plus storage readback
- `listRepoOps` catch-up reads

Comparison against Daniel's permissioned-data branch should live in this
section once we have a repeatable way to run that PDS locally. Until then, keep
ZDS numbers as local trend data and compare only equivalent operations.

## repo sizes

The normal write/read benchmarks use small synthetic repos so they stay fast
enough to run while developing. `just bench repo-size` adds explicit
small/medium/large tiers for repo-size-sensitive paths:

- small: 100 records
- medium: 1,000 records, unless the selected maximum is lower
- large: the `max_records` argument

Use `just bench repo-large 100000` for a deliberate larger run. Large public
accounts are useful reference shapes because they stress repo export, record
indexes, and cursor/list behavior. As of June 15, 2026, the public appview
reports about 38k posts for `pfrazee.com` and about 87k posts for
`jcsalterego.bsky.social`, so a 100k synthetic tier is the right order of
magnitude for large-repo stress. Live network corpora should be captured into
fixtures before they become stable benchmarks. Do not make routine CI or
pre-commit checks depend on live PDS availability.

June 29, 2026 local reachable-export check after switching full
`com.atproto.sync.getRepo` from whole-DID `repo_blocks` scans to current-root
reachability via `zat.mst.collectReachableBlocks`:

| repo tier | records | list | full CAR export |
|---|---:|---:|---:|
| small | 100 | 13.4k ops/s, 15.0 ms | 3.6k ops/s, 72.1 MB/s, 1.4 ms |
| medium | 1,000 | 12.4k ops/s, 16.1 ms | 297 ops/s, 59.6 MB/s, 16.8 ms |
| large | 5,000 | 11.9k ops/s, 16.8 ms | 52 ops/s, 53.1 MB/s, 95.7 ms |

These are synthetic local trend numbers, not cross-PDS comparisons. The old
whole-DID range-scan export path is intentionally no longer the production full
export behavior because it could include retained stale blocks.

`just bench get-repo` is the narrower operator-facing lane for the same sync
surface. It belongs here first because ZDS owns the storage and route behavior
for serving repo exports. Cross-PDS HTTP comparisons, compression behavior, and
live backfill traffic should live in `atproto-bench` once this local seam is
stable enough to compare against other implementations.

## methodology

- Build ZDS with `-Doptimize=ReleaseFast`.
- Use temporary SQLite and blobstore state for every run.
- Seed records before read benchmarks so lookup/list work is separated from
  write setup.
- Report throughput plus p50/p95/p99/max/mean latency for concurrent paths.
- Keep HTTP route benchmarks separate from storage benchmarks. Route numbers
  include request parsing, routing, response serialization, auth/session work,
  and `httpz` connection handling.
- Compare ratios from the same machine and same run window. Absolute numbers
  move with system load, thermal state, compiler version, and storage backend.
- Do not compare paths that perform different correctness work. For example,
  CID-only metadata lookup is separate from full `getRecord` materialization.

## traffic model

For a small hosted PDS, the hot paths are not exotic. A hundred active accounts
mainly means more session checks, appview-proxied feed/profile reads, record
writes from clients and apps, blob uploads, sync subscribers, crawl requests,
and occasional repo export/import. The storage benchmarks should therefore keep
these domains separate:

- record index: current CID lookups, existence checks, and index-only lists.
- record materialization: index lookup, block fetch, DAG-CBOR decode, and JSON
  response rendering.
- write path: validation, repo mutation, commit/block persistence, event
  publication, and crawl notification.
- blob path: upload body ingest, disk write, metadata persistence, and readback.
- sync path: event delivery and repo CAR materialization.

## current local baseline

On this machine, with `--records 1000` unless noted:

| scenario | zds |
|---|---:|
| write records | 144 ops/s |
| list records | 30k ops/s |
| write repo car | 252 ops/s, 440 MB/s |
| blob put+get | 368 ops/s, 46 MB/s |

Current local HTTP route baseline after adopting `httpz`, measured with
`just bench http` using 5000 requests at concurrency 50 for normal routes and
500 requests at concurrency 10 for blob/CAR routes. The harness covers public
metadata, OAuth metadata, session/account reads, preference reads, repo reads,
sync reads, blob reads, and repo CAR reads. Stateful write endpoints and
WebSocket `subscribeRepos` need separate harnesses.

| route group | httpz |
|---|---:|
| static/landing routes | 11.6k-40.2k req/s |
| public identity/server metadata | 31.9k-38.9k req/s |
| OAuth metadata/JWKS | 35.6k-37.0k req/s |
| OAuth authorize JSON | 4.4k req/s, p95 28 ms |
| session/account reads | 20.5k-34.9k req/s |
| `createSession` | 7.4k req/s, p95 20 ms |
| repo reads | 26.2k-39.0k req/s |
| `listRecords`, limit 50 | 26.2k req/s, p95 6 ms |
| sync JSON reads | 24.1k-39.4k req/s |
| blob/CAR reads | 34.5k-37.9k req/s |

The pre-httpz column below is the original narrow route set run against commit
`4841aec`; it should not be treated as coverage for the expanded route matrix.

| route | pre-httpz | httpz |
|---|---:|---:|
| `/xrpc/_health` | 34.1k req/s, p95 3 ms | 34.5k req/s, p95 3 ms |
| `describeServer` | 38.4k req/s, p95 1 ms | 39.3k req/s, p95 1 ms |
| `getSession` | 19.9k req/s, p95 8 ms | 20.8k req/s, p95 6 ms |
| `listRecords`, limit 50 | 19.8k req/s, p95 11 ms | 22.4k req/s, p95 7 ms |

## coverage map

High-level ZDS surface area represented in `bench/`:

| PDS surface | representative benchmark |
|---|---|
| repo writes | `write`, `metastore`, `write-profile` |
| repo reads | `get-cid`, `get-block`, `get-record`, `list-records` |
| repo export/sync snapshots | `repo`, `repo-size`, `get-repo` |
| sync event storage | `metastore` write path plus smoke tests |
| blob storage | `blob`, `publish-sync`, `space` blob readback |
| HTTP routing/serialization/auth overhead | `http` |
| OAuth/session/account routes | `http` route matrix |
| permissioned spaces | `space` |
| official PDS read-path comparison | `official-pds` |

Still intentionally outside the routine benchmark gate:

- passkey ceremony cryptography and browser UX
- mail delivery providers
- live appview proxy latency
- WebSocket subscriber fanout under network backpressure
- live public-repo corpora until captured as stable fixtures

## implementation comparison

The main PDS implementations do not expose identical benchmark seams, so keep
comparisons grouped by equivalent work:

| implementation | comparable bench surface | notes |
|---|---|---|
| ZDS | `bench/main.zig`, `tools/http_bench.sh` | local SQLite/blobstore, storage microbenchmarks, HTTP route matrix, permissioned-space storage |
| Tranquil | `crates/tranquil-store/benches/metastore*.rs`, `blockstore.rs`, `eventlog.rs` | best match for metastore-shaped record/index/event/blockstore comparisons |
| official Bluesky PDS | `bench/official-pds-records.bench.ts` copied into `packages/pds` | best match for actor-store record CID/get/list/write paths |
| Pegasus | `pegasus/bench/bench_repository.ml`, `mist/bench/bench_mst.ml` | best match for repository block/MST internals; not a full HTTP/PDS route comparison yet |

## stage checklist

For each protocol-parity or storage-behavior stage, do this before calling the
stage complete:

- Re-read the relevant atproto spec or lexicon, then cross-check the official
  PDS implementation and Tranquil.
- Decide whether any new parser, codec, verifier, resolver, repository
  primitive, OAuth helper, or JSON-path helper belongs in `zat`; keep it local
  unless the reusable boundary is clear.
- Add or update a focused ZDS benchmark for the changed surface.
- Run the comparable Tranquil benchmark when Tranquil has the same seam.
- Run the comparable official PDS probe when the official PDS has the same
  seam.
- Record any mismatch as either fixed, intentionally local with rationale, or a
  concrete follow-up issue. Do not leave “close enough” behavior unnamed.

Run Tranquil's metastore benchmark:

```sh
cd /tmp/tranquil-pds
SQLX_OFFLINE=true cargo bench -p tranquil-store --bench metastore --features tranquil-store/test-harness
```

Run the matching ZDS scenarios:

```sh
just bench metastore 10 1000
just bench metastore 100 200
just bench get-cid 10 1000
just bench get-cid 100 200
```

This matrix includes only operation/count pairs measured with the same unit of
work. Tranquil's `get_record_cid` is a record-index operation. The official PDS
probe measures its actor-store read paths with Vitest; its single-operation rows
are included for matching internal read/write boundaries.

Summary from the May 31, 2026 local run: ZDS is fastest in the single-caller
CID lookup and full-materialization read probes. Tranquil is fastest in the
10/100 caller CID lookup probes and its index-only `list_records` path is faster
than ZDS's full `listRecords` materialization. For writes, Tranquil and ZDS are
close around the single-caller 1000-op shape, Tranquil is faster with much
tighter tails at 10 callers, and ZDS has higher 100-caller throughput with much
worse tail latency.

| operation | callers | zds | tranquil | official PDS |
|---|---:|---:|---:|---:|
| apply one record commit | 1 | 281 ops/s, p95 8.1 ms | 289 ops/s, p95 4.9 ms | 448 ops/s, p99 4.0 ms |
| current record CID lookup | 1 | 509k ops/s, p95 2 us | 111k ops/s, p95 12.6 us | 3.1k ops/s, p99 511 us |

The direct caller-count concurrency matrix is ZDS and Tranquil only. Official
PDS concurrency probes are kept separate until they use the same caller model.

| operation | callers | ops | zds | tranquil |
|---|---:|---:|---:|---:|
| apply one record commit | 10 | 10000 | 289 ops/s, p95 86.1 ms | 744 ops/s, p95 15.9 ms |
| apply one record commit | 100 | 20000 | 1423 ops/s, p95 65.0 ms, p99 806 ms | 1051 ops/s, p95 100 ms |
| current record CID lookup | 10 | 10000 | 345k ops/s, p95 18 us | 610k ops/s, p95 27 us |
| current record CID lookup | 100 | 20000 | 302k ops/s, p95 273 us | 630k ops/s, p95 212 us |

Official PDS batched CID lookup:

| operation | batch shape | converted throughput | batch p99 |
|---|---|---:|---:|
| current record CID lookup | 10 callers x 100 lookups | 3.3k lookups/s | 317 ms |
| current record CID lookup | 100 callers x 20 lookups | 3.2k lookups/s | 630 ms |

Tranquil's metastore `list_records` benchmark is index-only, while ZDS
`listRecords` joins `repo_blocks` and materializes record data. Keep these rows
as adjacent probes until ZDS grows an explicit index-only list benchmark.

| operation | callers | ops | zds full materialization | tranquil index-only |
|---|---:|---:|---:|---:|
| list records, limit 50 | 1 | 5000 | 22.5k ops/s, p95 58 us | 32.9k ops/s, p95 37 us |
| list records, limit 50 | 10 | 10000 | 18.2k ops/s, p95 97 us | 38.9k ops/s, p95 271 us |
| list records, limit 50 | 100 | 20000 | 21.5k ops/s, p95 60 us | 38.5k ops/s, p95 2.75 ms |

The current Tranquil `metastore` bench panics at its built-in 1000-caller
`get_record_cid` row with `metastore handler backpressure`; treat that as a
backpressure behavior to investigate, not as a throughput result.

## official PDS probe

The official-PDS probe lives at `bench/official-pds-records.bench.ts` and is
run with:

```sh
ATPROTO_REPO=/tmp/bluesky-atproto just bench official-pds
```

It seeds 1000 `app.bsky.feed.post` records through the official test network,
then measures actor-store reads directly:

| operation | official PDS |
|---|---:|
| apply one record commit | 448 ops/s, p99 4.0 ms |
| current record CID lookup | 3.1k ops/s, p99 511 us |
| full getRecord materialization | 3.0k ops/s, p99 525 us |
| full listRecords, limit 50 | 635 ops/s, p99 2.4 ms |

The matching ZDS single-caller rows are:

| operation | zds |
|---|---:|
| apply one record commit | 642 ops/s, p95 2.9 ms |
| current record CID lookup | 414k ops/s, p95 4 us |
| full getRecord materialization | 48.6k ops/s, p95 28 us |
| full listRecords, limit 50 | 21.6k ops/s, p95 60 us |

## local probes

These rows are useful for ZDS tuning but are not direct Tranquil comparisons.

| operation | callers | ops | zds |
|---|---:|---:|---:|
| get record block | 10 | 10000 | 128k ops/s, p95 11 us |
| get record block | 100 | 20000 | 121k ops/s, p95 15 us |
| decode DAG-CBOR | 10 | 10000 | 40.6M ops/s, p95 below timer resolution |
| decode DAG-CBOR | 100 | 20000 | 12.6M ops/s, p95 below timer resolution |
| render record JSON | 10 | 10000 | 24.8M ops/s, p95 below timer resolution |
| render record JSON | 100 | 20000 | 12.4M ops/s, p95 1 us |
| full getRecord | 10 | 10000 | 38.4k ops/s, p95 173 us |
| full getRecord | 100 | 20000 | 38.1k ops/s, p95 11.7 ms |
| full listRecords, limit 50 | 10 | 10000 | 21.1k ops/s, p95 57 us |
| full listRecords, limit 50 | 100 | 20000 | 19.5k ops/s, p95 78 us |

The read-path split says SQLite serialization and row materialization dominate
single-record reads. Decode/render is effectively invisible for tiny benchmark
records; it needs a larger-record probe before we can extrapolate image-heavy or
rich-record traffic.

## write profile

`write-profile` tracks where concurrent writes spend time:

| callers | ops | throughput | lock wait | load repo | build commit | sqlite/event |
|---:|---:|---:|---:|---:|---:|---:|
| 10 | 5000 | 708 ops/s | 25% | 26% | 3% | 46% |
| 100 | 2000 | 3683 ops/s | 64% | 10% | 1% | 25% |

ZDS separates repo ordering from database serialization: writes take a
per-DID lane before loading and mutating the repo, then use the DB mutex for the
shared SQLite connection. Tranquil remains the reference shape for bounded write
queues, explicit backpressure, less full-repo state loading, and a narrower
SQLite persistence phase.

## comparison work

Tranquil benchmark source:

- [`crates/tranquil-store/benches/metastore.rs`](https://tangled.org/tranquil.farm/tranquil-pds/tree/main/crates/tranquil-store/benches/metastore.rs)
- [`crates/tranquil-store/benches/metastore_scale.rs`](https://tangled.org/tranquil.farm/tranquil-pds/tree/main/crates/tranquil-store/benches/metastore_scale.rs)
- [`crates/tranquil-store/benches/blockstore.rs`](https://tangled.org/tranquil.farm/tranquil-pds/tree/main/crates/tranquil-store/benches/blockstore.rs)
- [`crates/tranquil-store/benches/eventlog.rs`](https://tangled.org/tranquil.farm/tranquil-pds/tree/main/crates/tranquil-store/benches/eventlog.rs)

Official PDS source for matching probes:

- [`packages/pds/src/actor-store/record/reader.ts`](https://github.com/bluesky-social/atproto/blob/main/packages/pds/src/actor-store/record/reader.ts)
- [`packages/pds/src/api/com/atproto/repo/applyWrites.ts`](https://github.com/bluesky-social/atproto/blob/main/packages/pds/src/api/com/atproto/repo/applyWrites.ts)
- [`packages/pds/src/actor-store/repo/sql-repo-reader.ts`](https://github.com/bluesky-social/atproto/blob/main/packages/pds/src/actor-store/repo/sql-repo-reader.ts)

Open apples-to-apples comparison gaps:

- single-repo write curve as record count grows
- multi-repo concurrent writes
- record-index list latency without record body materialization, across ZDS and
  Tranquil
- official-PDS direct concurrency harness for the same read/write boundaries
- full repo export cost
- blob/block store throughput

ZDS and Tranquil both report latency percentiles for the metastore-shaped write
and CID lookup rows. The remaining comparison gap is broader coverage of
blockstore, eventlog, full repo export behavior, an official-PDS benchmark
harness, and an explicit ZDS index-only list probe that matches Tranquil's
`list_records`.

## June 18, 2026 write-path check

ZDS now uses `zat.mst.Mst.loadLazy` for repo authoring, backed by SQLite
`repo_blocks`, instead of reading every repo block into an in-memory CAR before
each write. This completes the earlier `zat` lazy-MST adoption for the hot write
path.

Fresh local 10-caller metastore-shaped rows. The ZDS write row was repeated
three times after the change and stayed in the same band: 2312.8-2525.2 ops/s,
p95 11.2-12.2 ms.

| operation | zds | tranquil |
|---|---:|---:|
| apply one record commit | 2312.8-2525.2 ops/s, p95 11.2-12.2 ms | 503 ops/s, p95 26.1 ms |
| current record CID lookup | 319.7k ops/s, p95 8 us | 294.6k ops/s, p95 54 us |
| list records, limit 50 | 21.5k ops/s, p95 53 us | 35.8k ops/s, p95 382 us |

Two `just bench write-profile 10 1000` runs after the lazy-MST change reported
2527.1-2698.4 ops/s, p95 9.3-9.8 ms. Average write time was split roughly
across lock wait 23-24%, lazy repo loading 9-10%, staging 10%, commit build 7%,
and SQLite/event persistence 50%.

### Spaces self-read scope parity (2026-10-02)

`just bench space-scopes` alternates member-list-style `read_self` checks with
writes outside the granted collection. One million checks accept 500,000 reads
and reject all 500,000 writes. Before the fix, ZDS rejected both. On the local
Apple Silicon host, the ReleaseFast loop measured 42 ns/check before and
50–60 ns/check after; the earlier exit represented incorrect authorization.

The official PDS at `5b95b2f2723a2882824fbb0b819fd6aad884ef20` uses
[`read_self` for listMembers](https://github.com/bluesky-social/atproto/blob/5b95b2f2723a2882824fbb0b819fd6aad884ef20/packages/pds/src/api/com/atproto/simplespace/listMembers.ts)
and its scope matcher makes both read actions collection-independent. A local
probe of that revision's `SpacePermission.matches` accepted the same 500,000
reads and rejected all writes (14 ns/check in Node, with an already parsed
grant; this is not directly comparable to ZDS's parse-and-match measurement).
The local Tranquil scope implementation has no Spaces permission matcher, so
there is no equivalent probe there. No storage or protocol primitive changed;
this authorization policy remains local to ZDS.