# benchmarking

Benchmark notes should separate behavior domains and only compare equivalent
work. If a measurement does not exist for the same unit of work, keep it out of
the comparison table instead of filling the cell with a nearby but different
operation.

## references

- ZDS: `bench/main.zig`
- Tranquil: `crates/tranquil-store/benches/metastore.rs`
- Official PDS: `packages/pds/src/actor-store/record/reader.ts`

## current matrix

The active apples-to-apples matrix is intentionally small:

- apply one record commit
- current record CID lookup

Adjacent probes are tracked separately:

- record-index list
- full record materialization
- blob/block put and get
- repo export
- sync event delivery

ZDS, Tranquil, and the official PDS have matching single-caller numbers for
apply-one-record and current-CID lookup. ZDS and Tranquil also have direct
10/100 caller concurrency curves. Official-PDS batch probes stay outside the
main matrix until they are measured through the same unit of work.

Current local comparison. The Tranquil rows are from a fresh local metastore
run on June 18, 2026. The ZDS write row was remeasured after completing the
lazy-MST adoption in the repo write path.

| operation | callers | zds | tranquil | note |
|---|---:|---:|---:|---|
| apply one record commit | 1 | 1598 ops/s, p95 0.65 ms | 267 ops/s, p95 4.13 ms | ZDS no longer eagerly loads the whole repo CAR |
| apply one record commit | 10 | 2313-2525 ops/s, p95 11.2-12.2 ms | 503 ops/s, p95 26.1 ms | ZDS row is repeated-run range |
| current CID lookup | 1 | 523k ops/s, p95 2 us | 116k ops/s, p95 12.6 us | ZDS ahead |
| current CID lookup | 10 | 319.7k ops/s, p95 8 us | 294.6k ops/s, p95 54 us | both use record-index lookup |

Tranquil's built-in 1000-caller metastore row currently trips handler
backpressure in the benchmark harness. Keep it out of the table until the
harness handles backpressure as a measured result.

ZDS write authoring uses `zat.mst.Mst.loadLazy` with a SQLite-backed
`repo_blocks` reader. This avoids reading every stored repo block into a CAR
before each write while preserving the signed commit, MST update, record index,
and sync event behavior.

## read path

ZDS follows the official PDS read shape for `getRecord` and `listRecords`: read
the current record index row, fetch matching `repo_blocks` bytes, decode
DAG-CBOR, and render JSON for the XRPC response. Tranquil's metastore
`list_records` is a different unit of work because it returns only rkeys and
CIDs. Keep those rows separate.

Current ZDS read breakdown on this machine:

| operation | callers | ops | result |
|---|---:|---:|---:|
| current CID lookup | 10 | 10000 | 222k ops/s, p95 225 us |
| current CID lookup | 100 | 20000 | 211k ops/s, p95 887 us |
| index + block fetch | 10 | 10000 | 128k ops/s, p95 11 us |
| index + block fetch | 100 | 20000 | 121k ops/s, p95 15 us |
| full getRecord | 10 | 10000 | 38.4k ops/s, p95 173 us |
| full getRecord | 100 | 20000 | 38.1k ops/s, p95 11.7 ms |
| full listRecords, limit 50 | 10 | 10000 | 21.1k ops/s, p95 57 us |
| full listRecords, limit 50 | 100 | 20000 | 19.5k ops/s, p95 78 us |

Adjacent Tranquil list comparison:

| operation | callers | zds | tranquil | note |
|---|---:|---:|---:|---|
| list records, limit 50 | 1 | 22.5k ops/s, p95 58 us | 32.9k ops/s, p95 37 us | ZDS full materialization; Tranquil index-only |
| list records, limit 50 | 10 | 18.2k ops/s, p95 97 us | 38.9k ops/s, p95 271 us | not same unit of work |
| list records, limit 50 | 100 | 21.5k ops/s, p95 60 us | 38.5k ops/s, p95 2.75 ms | not same unit of work |

The official PDS probe on the same machine reports:

| operation | result |
|---|---:|
| apply one record commit | 448 ops/s, p99 4.0 ms |
| current CID lookup | 3.1k ops/s, p99 511 us |
| full getRecord | 3.0k ops/s, p99 525 us |
| full listRecords, limit 50 | 635 ops/s, p99 2.4 ms |

The official PDS batched CID lookup probe reports:

| batch shape | converted throughput | batch p99 |
|---|---:|---:|
| 10 callers x 100 lookups | 3.3k lookups/s | 317 ms |
| 100 callers x 20 lookups | 3.2k lookups/s | 630 ms |

For tiny records, DAG-CBOR decode and JSON rendering are below useful timer
resolution. Larger records and index-only list probes are tracked as separate
comparison gaps instead of being folded into the current table.

## permissioned data

Permissioned-data measurements should stay separate from the existing repo,
blob, sync, and HTTP route matrices. The experimental `com.atproto.space.*`
surface has its own storage model, actor-local space state, writer repo state,
record oplogs, credentials, and ranged blob reads, so mixing it into the public
repo tables would hide the actual access pattern.

The first benchmark slice should be small and diagnostic rather than exhaustive:

- seed one self-owned private space and one writer repo.
- measure `createSpace`, record write, record read, `listRecords`, full-state
  `getRepo` CAR construction, ranged `getBlob`, and writer repo oplog catch-up
  separately.
- report p50/p95/p99/max for the concurrent paths, but keep write, read, blob,
  and oplog rows separate.
- compare to Daniel's permissioned-data branch only after we can run that PDS
  locally with the same seeded work.

The question for this slice is simply whether any route has an obviously bad
query shape, full-table scan, unnecessary blob materialization, or avoidable
global lock hold. It does not need to become a giant protocol benchmark suite.

## HTTP route path

`just bench http` starts a local ReleaseFast ZDS server and measures a seeded
route matrix with ApacheBench. These numbers include HTTP request parsing,
routing, response serialization, auth/session work, `httpz` connection
handling, and whatever endpoint work the route performs. Keep them separate
from the storage microbenchmarks above.

The current harness covers public metadata, OAuth metadata, session/account
reads, preference reads, repo reads, sync reads, blob reads, and repo CAR reads.
It deliberately does not loop over stateful write endpoints such as
`createRecord`, `applyWrites`, `uploadBlob`, or passkey registration because a
static ApacheBench body would mostly measure duplicate/error paths after the
first request.

`just bench websocket-churn` covers the WebSocket lifecycle separately. It
repeatedly opens and closes cohorts of `subscribeRepos` clients at the server's
live-connection limit, then checks that detached connection workers return to the
baseline thread count. A warm second round also guards against renewed,
unbounded RSS growth while allowing the allocator to retain its first-round
high-water capacity.

Current local route baseline after adopting `httpz`, with 5000 requests at
concurrency 50 for normal routes and 500 requests at concurrency 10 for the
larger blob/CAR routes:

| route | httpz |
|---|---:|
| `/` | 15.0k req/s, p95 10 ms |
| `/favicon.svg` | 40.2k req/s, p95 1 ms |
| `/og-image` | 11.6k req/s, p95 5 ms |
| `/xrpc/_health` | 31.9k req/s, p95 1 ms |
| `/.well-known/did.json` | 38.9k req/s, p95 1 ms |
| `/.well-known/atproto-did` | 38.5k req/s, p95 1 ms |
| OAuth metadata/JWKS | 35.6k-37.0k req/s, p95 2 ms |
| `oauth/authorize` JSON | 4.4k req/s, p95 28 ms |
| `describeServer` | 38.7k req/s, p95 1 ms |
| `createSession` | 7.4k req/s, p95 20 ms |
| `getSession` | 23.8k req/s, p95 6 ms |
| `getServiceAuth` | 24.5k req/s, p95 5 ms |
| account/security reads | 20.5k-34.9k req/s, p95 4-7 ms |
| `getPreferences` | 32.0k req/s, p95 5 ms |
| `resolveHandle` | 23.9k req/s, p95 2 ms |
| `describeRepo` | 39.0k req/s, p95 1 ms |
| `getRecord` | 30.0k req/s, p95 5 ms |
| `listRecords`, limit 50 | 26.2k req/s, p95 6 ms |
| `listMissingBlobs` | 27.6k req/s, p95 6 ms |
| sync JSON reads | 24.1k-39.4k req/s, p95 1-4 ms |
| `sync.getBlob` | 34.5k req/s, p95 1 ms |
| `sync.getBlob` HEAD | 34.8k req/s, p95 1 ms |
| `sync.getRepo` | 37.9k req/s, p95 below 1 ms |
| `sync.getRepo` HEAD | 36.6k req/s, p95 below 1 ms |

The separate permissioned-data scenario (`just bench space`) covers space
discovery, management-policy reads, record writes and reads, writer/oplog
enumeration, full repo CAR construction, referenced-blob enumeration, and blob
serving. Keep it separate from public-repo benchmarks because the storage and
commit formats are intentionally different.

The only pre-httpz route numbers currently available are the original narrow
route set, run against commit `4841aec` with the same four probes:

| route | pre-httpz | httpz |
|---|---:|---:|
| `/xrpc/_health` | 34.1k req/s, p95 3 ms | 34.5k req/s, p95 3 ms |
| `describeServer` | 38.4k req/s, p95 1 ms | 39.3k req/s, p95 1 ms |
| `getSession` | 19.9k req/s, p95 8 ms | 20.8k req/s, p95 6 ms |
| `listRecords`, limit 50 | 19.8k req/s, p95 11 ms | 22.4k req/s, p95 7 ms |

## traffic model

For roughly 100 hosted accounts, useful benchmarks should mirror real traffic:

- clients call `getSession`, preferences, profiles, timelines, and proxied
  appview routes repeatedly while open.
- apps call `createRecord`, `applyWrites`, `getRecord`, and `listRecords`.
- media posts call `uploadBlob` before record creation.
- relays and tools call sync endpoints, repo export, and blob reads.
- the landing page reads residents, record counts, recent records, and profile
  blobs.

This traffic model is the basis for future route-mix benchmarks and per-route
latency instrumentation. Keep those measurements separate from isolated storage
microbenchmarks.

## startup and deployment interruption

Run `just bench startup 5 10000` for a ReleaseSafe build (the production
optimization mode) and five samples per scenario. Build time is excluded.
`tools/startup_bench.py` measures subprocess launch to the first HTTP 200 health
response with `status: ok`. It emits JSON with platform, binary SHA-256,
individual samples and min/median/max. Polling adds up to roughly 5 ms plus HTTP
request overhead; these are warm-filesystem runs, not cold VM boots.

The lanes separate fresh schema creation, empty existing-database restart, and
history-shaped restart. The history fixture has 10,000 commits, 60,000 blocks with 4 KiB payloads,
10,000 sequence events with 4 KiB payloads,
and 2,000 unversioned tree blocks which match neither records nor commits. It
is a synthetic SQLite workload, not a signed repository or an API throughput
benchmark. Increase the second argument to examine scaling.

For an existing local SQLite backup, run:

```sh
python3 tools/startup_bench.py --snapshot /path/to/backup.sqlite3 --runs 5
```

The source is opened read-only and copied with SQLite's backup API before every
sample. The server only opens disposable copies. Runs bind to loopback, disable
crawler notifications, discard inherited `ZDS_*` configuration, and clean up
child processes and temporary data. Output exports stage durations, not server
logs or account data. Use local backups rather than a production database path.

Startup logs report `storage_ms`, `account_setup_ms`, and `http_init_ms`. They
measure storage initialization, account bootstrap/announcement setup, and HTTP
server construction respectively. Storage also reports `migrations_ms` and
`sequence_ms` to separate those two phases. The existing "listening" message precedes
`server.listen()`; HTTP health remains the authoritative readiness measurement.

### 2026-09-05: index commit CID lookups

The revision backfill runs on startup and matches NULL-revision blocks against
commits by `(did, cid)`. Without a matching index, unmatched historical tree
blocks repeatedly scan an account's commit history. A non-unique composite
index removes those scans without skipping backfill or integrity validation.

Five-run medians on macOS arm64, ReleaseSafe, warm filesystem. This initial
fixture had block and commit history but no sequence-event payloads:

| lane | before | indexed |
| --- | ---: | ---: |
| fresh schema | 22.00 ms | 22.34 ms |
| empty restart | 9.11 ms | 9.33 ms |
| history restart | 3147.59 ms | 15.38 ms |
| old history snapshot, including index creation | — | 29.01 ms |

These fixture results are not production startup timings. Before the change,
Fly logs showed about 5 seconds from process launch to the listening message.
An external one-second health probe with a two-second request timeout observed
8 consecutive timeouts spanning roughly 17 seconds during the prior deployment;
that includes machine replacement and proxy routing, not just process startup.

For production rollouts, record both process stage timings and an external
health probe across replacement. Compare failure windows separately from healthy
response latency. A single machine with a local SQLite volume has a deployment
interruption even when application startup is fast; avoid extra restarts solely
to publish benchmark notes.

### sequence recovery

The first indexed production deployment still recorded `storage_ms=5709`.
The fixture improvement alone did not establish a production improvement.
Read-only production probes then identified another scan: computing the next
sequence via `MAX` over a union of every commit and event sequence. That query
scanned the event table (including its large row storage). Computing `MAX(seq)`
inside each branch first preserves empty-table and gap semantics while letting
SQLite seek to each table's final integer primary key. A production read-only
comparison returned 10914 for both queries, taking 556 ms and 1 ms respectively;
these are individual warm query measurements, not full startup measurements.

The history fixture now includes 10,000 events with 4 KiB payloads to exercise
this path. Regression tests cover empty histories, either history being ahead,
gaps, and an empty commit table with retained events.

### narrow revision backfill to missing revisions

Production `EXPLAIN QUERY PLAN` showed `SCAN repo_blocks` for both backfill
updates even after commit lookup was indexed. Unlike the read-only SELECT
probes, UPDATE selected a full scan of payload-bearing block rows. A partial
index on `(did, cid) WHERE repo_rev IS NULL` bounds subsequent backfill work to
unversioned blocks. It is created after the legacy `repo_rev` column migration.
Index creation still scans existing blocks once on upgrade; it does not remove
or bypass migration work. The fixture now includes 4 KiB block payloads as well
as events, exposing the cost of scanning payload-bearing tables.

With payloads in the fixture, five-run macOS ReleaseSafe median readiness fell
from 33.86 ms (commit index only) to 9.22 ms (partial index plus per-table sequence
maxima). Empty restarts measured 9.16 ms. These warm-cache results are distinct
from the initial small-block comparison above.

### September 2026: independent simplespace read/write access

On macOS arm64, Zig 0.16.0, ReleaseFast, three `just bench space 1000`
runs before and after the API/storage update gave these median throughputs:

| storage operation | before ops/s | after ops/s |
| --- | ---: | ---: |
| listSpaces | 56,055 | 54,399 |
| listMembers | 38,596 | 37,099 |
| createRecord | 7,389 | 7,224 |
| getRecord | 180,344 | 174,748 |
| listRecords | 44,609 | 43,957 |
| getRepo | 624 | 628 |

These sequential local samples show small changes (roughly 1–4% on most read
paths), not a production capacity estimate. The member list now returns two
additional flags and space configuration reads return independent policies.
At 10,000 records, a single export probe measured 69.0 ops/s before and 68.6
after; createRecord measured 7,016 before and 7,319 after. Membership size is
not scaled with record count in this fixture.

A ReleaseSafe startup rehearsal on a consistent production SQLite backup
preserved account, session, OAuth, public block, permissioned record, space
configuration, and membership state. The migration stage took 10 ms locally;
first readiness was 336 ms and a subsequent restart reached readiness in
11 ms. These are individual warm-filesystem samples, not Fly replacement
or network interruption measurements.