Historical Jetstream replay #

See Appview traces for optional storage and replay traces in VictoriaTraces.

The native SQLite driver uses the bundled SQLite source and requires CGO. Existing SQLite databases need no format conversion or schema change for the driver switch. The driver switch preserves the cache configuration, WAL/NORMAL durability, index deferral, and atomic replay checkpoints.

Incomplete full all-author replay uses a WAL checkpoint threshold of 65,536 pages, approximately 256 MiB with 4 KiB pages. Larger thresholds combine repeated page writes and reduce checkpoint frequency. The WAL can exceed the threshold by one transaction and retain its allocated size after a checkpoint. Checkpoints still run on the writer, so individual checkpoint stalls can grow.

Replay restores the normal 1,000-page threshold before live readiness. Retries and restarts retain the archive threshold while the source has deferred indexes. Completed sources, disabled replay, subsets, and participant replay use the normal threshold. Schema, atomic checkpoints, and synchronous=NORMAL remain unchanged. The larger interval can require more replay after a power loss because WAL synchronization occurs less often.

Historical replay is optional. The default appview still uses the legacy Microcosm stream. The v2 adapter reconstructs current records, not a complete mutation audit. The provider can compact deleted records from its archive.

Configuration #

1. Supply JETSTREAM_API_KEY through the service environment or a runtime secret.
2. Set PROVENANCE_JETSTREAM_REPLAY=true.
3. For a network-wide soft index, set ENABLE_SOFT_VOUCH=true.
4. Set ENABLE_SOFT_VOUCH_ALL_AUTHORS=true.
5. For blocked moderation lists, set ENABLE_SOFT_VOUCH_LISTS=true.
6. Start the appview.

The default archive origin is https://jetstream.us-west.bsky.network. PROVENANCE_JETSTREAM_REPLAY_URL selects another origin. The API key authenticates archive requests only. It does not reach the public WebSocket. No API key belongs in Git, a Nix expression, or a command argument. Keep secret files outside the repository. A path:. Nix flake can copy ignored files, including .env, into the Nix store. Use Git-filtered flake checks instead of path:. when the repository contains private data. Git-backed flakes exclude untracked files. .gitignore does not exclude a file that Git already tracks. Check sources also exclude .env, .env.*, and .direnv. This filter does not protect the initial copy of a path:. flake.

NixOS exposes services.provenance.appview.jetstreamReplay, jetstreamReplayURL, and jetstreamAPIKeyFile. The module loads the key with systemd credentials. Development services expose jetstreamReplay and jetstreamReplayURL and accept the key through the environment.

CAUTION: Do not enable replay before you assess archive charges and available disk space. The provider meters compressed download bytes. A full import can consume substantial time, disk space, and memory.

Recovery and cutover #

The official Go SDK pins each archive plan, decodes selected collections, and connects the live tail without a gap. It fills an expired live cursor from the archive. SQLite commits each batch and its sequence checkpoint in one transaction.

Full all-author archive replay defers the soft_vouches_pair and soft_vouches_subject query indexes. Primary keys, identity uniqueness, and record-version checks remain active. Graph queries remain correct but can run much slower during archive replay.

At archive cutover, replay rebuilds both indexes in one transaction before it announces live readiness. The rebuilding phase identifies this work in logs, indexing status, and metrics. Index sorts use temporary files, so the rebuild requires free disk space for the indexes and sort files.

Schema 12 retains the deferred state across retries and process restarts. An interrupted rebuild retains the durable checkpoint and deferred state. A completed source keeps its indexes on later starts. Startup without full all-author replay restores deferred indexes before the service accepts requests.

All-author replay groups contiguous soft-record mutations for batch SQL lookups and writes. Each group validates mutations in sequence order and retains the final accepted value for each record key. The group keeps deletion timestamps so that older imports cannot restore deleted records. Replay flushes the group before account events, resets, identity events, and explicit records. These boundaries preserve lifecycle order. Short groups retain the individual-record path. Records, versions, account watermarks, cache invalidation, and the checkpoint still commit together. Per-account sequence guards reject duplicate events and preserve ordering across interleaved accounts. Witnessed timestamps preserve the existing snapshot and legacy replay guards. Sequence order distinguishes record mutations with the same timestamp. Snapshot and deletion watermarks still reject records at or before their timestamp.

Account deletion and sync divergence remove obsolete records. Replacement records after a sync marker restore current state. Network-wide list replay retains list definitions and members before a subscriber references them. Participant-only mode retains referenced-list snapshots. Explicit ingestion never enrolls nonparticipants.

Checkpoints include the archive origin and soft mode. Different origins do not share sequence cursors. Changing the origin or soft mode starts a new archive import. Do not replace a server with an unrelated sequence namespace at the same origin.

Downloads and WebSocket handshakes share the external worker queue and the project User-Agent. Archive downloads follow redirects through the same queue. Redirects to another origin do not receive the archive API key. Replay rejects encrypted-to-unencrypted redirects and embedded credentials. Loopback development origins can redirect to loopback HTTP endpoints. PROVENANCE_EXTERNAL_WORKERS bounds HTTP concurrency and the SDK decode pool. The SDK uses one range stream per segment and at most three attempts per request. The queue retains per-origin Retry-After cooldowns without additional retries. The SDK retains partial range downloads in memory during retries, not across process restarts. A process restart resumes from the last committed event batch and can download part of a segment again. The SDK also prefetches compressed segments. The worker count is not a strict memory limit.

Batch performance #

Replay requests batches of up to 256 events. SQLite retains full transaction durability. The store prepares SQL queries once and reuses them across transactions to avoid repeated SQL parsing. Each batch reads each author's sequence watermark once and writes its final watermark with the cursor. All-author follow and block replay reads snapshot and record watermarks together. Obsolete imported records skip account creation and record writes. Accepted records still invalidate selected cache paths, even when their values do not change.

Local synthetic benchmarks compare new records, active updates, and obsolete imported records with one or 64 authors. The SQL changes and larger batches improved throughput by approximately 1.6–2.1 times for new records and active updates. Obsolete imported records improved by approximately 3.6–4.2 times. These results compare the previous 64-event batches with the new 256-event batches on the same development host. They do not predict production throughput on Misaki or the cost of a full archive import. The 1,024-event batches gave little extra active-update throughput and held write locks for longer.

Prepared queries gave a further improvement with the same 256-event batches and 64-author workloads:

Workload	Before (events/s)	After (events/s)
New records	7,703	23,233
Active updates	7,423	25,552–26,690
Obsolete imported records	33,806	54,361

These local benchmarks use temporary databases, not the production database. Prepared-query wrappers increase temporary allocations per active-update batch from approximately 779 KB to 1,208 KB. The batch size, worker count, schema, and durability settings remain unchanged.

Run the local benchmark:

go test ./internal/storage -run '^$' -bench '^BenchmarkReplayBatch$' -benchmem

Subset imports #

Set PROVENANCE_JETSTREAM_AFTER_SEQ to the exclusive start sequence. Set PROVENANCE_JETSTREAM_BEFORE_SEQ to the inclusive end sequence. The start defaults to zero. A nonzero end requires replay mode and must exceed the start. Both zero values select full replay with live cutover. Nix modules expose jetstreamAfterSequence and jetstreamBeforeSequence.

A subset uses snapshot-only replay and an isolated checkpoint namespace. It does not connect the live stream or report production readiness. Use a separate database for a trial. Isolated checkpoints do not isolate records within a shared database. The adapter stops if the end exceeds the observed sealed archive tip. A successful sweep commits the end checkpoint, including an empty filtered tail.

Status and metrics #

Structured logs report replay phases, periodic progress, retry decisions, and terminal error causes. See Service logs for debug configuration and field definitions.

/healthz reports process health. /readyz reports replay readiness separately. Replay readiness requires a live connection and activity within two minutes. The last committed event must also remain within two minutes of wall time. Disabled replay reports readiness without a replay freshness check. Subsets, retries, terminal errors, and stale replay return HTTP 503 from /readyz.

/v1/indexing requires the service bearer key and X-Provenance-DID. It reports the phase, range, durable checkpoint, observed archive tip, timestamps, observed lag, and sanitized error category. Event timestamps remain unknown after a restart until the first committed event.

/metrics exposes Prometheus text without authentication. Restrict access at the network boundary if system statistics are private. Metrics contain no DIDs, record contents, credentials, URLs, or error text.

Prefix	Statistics
provenance_index_	Phase, checkpoint, archive tip, timestamps, lag, readiness, committed events, retries, errors, response bytes, batch duration
provenance_http_	Request counts and duration with bounded method and status labels
provenance_database_	Allocated page bytes, WAL bytes, connection use, connection waits, scrape errors
go_, process_	Go runtime and process statistics

Counters reset at process start. Checkpoints persist in SQLite. Download bytes count HTTP response bytes, not provider billing units. Database scrapes read allocation metadata instead of global row counts. The archive tip describes a sealed snapshot, not the current live sequence. Archive plan observations parse summary fields incrementally without a second complete plan buffer. Plans larger than the event-size limit still update the archive tip. Observation failures do not change the response that the SDK receives.

Authentication, malformed events, storage errors, and unclassified upstream errors stop replay without a checkpoint advance for the failed batch. The adapter logs SDK warnings for non-object record payloads and continues with valid records. Those records do not enter the index. Other decode errors and mixed error groups still stop replay. Transient errors retain reconnect backoff, jitter, and shared cooldowns. To recover from a terminal error, repair its cause and restart the service. The restart uses the durable checkpoint. No endpoint skips a failed event.

Query limits #

Built-in trust algorithms read outgoing records only for reachable accounts. Unrelated records no longer consume the query edge limit. The 10-hop cap and the state-based breadth-first score calculation remain unchanged. One query shares its neighbor cache, edge budget, and state budget across reputation author checks. Concurrent graph changes prevent stale cache publication and cause bounded retries. Custom algorithms without the neighbor capability retain the complete edge-buffer contract.

The 100,000-record storage fixture allocates about 118 MB for a complete graph read and selection. An outgoing read and selection for one author with 100 edges allocates about 106 KB. These figures exclude traversal state, SDK buffers, peak memory, WAL, and SQLite caches.

Validation #

Local tests cover binary archive decode, live cutover, collection filters, bearer-key scope, atomic rollback, restart, and equal-time events. The adapter does not start an import unless replay is enabled. No full-network import forms part of the test suite.

To verify the key without a segment download, run the opt-in archive-plan probe:

PROVENANCE_REPLAY_SMOKE=true go test -tags=integration ./internal/microcosm -run '^TestReplayLivePlan$' -v

The probe requires JETSTREAM_API_KEY in the process environment and permits one sequence in its archive plan.

For a subset trial with a separate temporary database, run:

PROVENANCE_REPLAY_SUBSET_TEST=true go test -tags=integration ./internal/appview -run '^TestLiveReplaySubset$' -v

The trial requires the runtime key and imports sequences (0,1000] with one external worker. Default tests exclude both live probes. Each probe requires the integration build tag and its explicit environment flag. It also verifies the durable checkpoint after a database restart. This small sequence range does not bound compressed segment download size.

Provider documentation: https://bsky.network/docs/jetstream-replay.