From df0888fdd060abc0f1474539074c37595c7d25df Mon Sep 17 00:00:00 2001 From: dawn <90008@klbr.net> Date: Sat, 18 Jul 2026 16:27:30 +0000 Subject: [PATCH] [docs] document /stream event formats --- docs/api/README.md | 2 +- docs/api/stream.md | 137 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 file(s) changed, 138 insertion(s)(+), 1 deletion(s)(-) diff --git a/docs/api/README.md b/docs/api/README.md --- a/docs/api/README.md +++ b/docs/api/README.md @@ -6,7 +6,7 @@ ## public -- `GET /stream`: subscribe to the event stream. query params: `cursor` (optional, start from a specific event ID). slow consumers may receive a `{"type":"error","error":"ConsumerTooSlow",...}` frame before the connection closes. +- `GET /stream`: [stream API](stream.md), subscribe to the ordered websocket event stream. - `GET /subscribe`: [jetstream API](jetstream.md), subscribe to a filtered, jetstream-compatible websocket event stream. - `GET /stats`: get stats about the database (counts of repos, records, events; sizes of keyspaces on disk). - `GET /health` / `GET /_health`: health check. diff --git a/docs/api/stream.md b/docs/api/stream.md new file mode 100644 --- /dev/null +++ b/docs/api/stream.md @@ -0,0 +1,137 @@ +--- +title: stream +--- + +available in builds compiled with the `indexer_stream` feature. `/stream` is hydrant's ordered websocket event stream. each subscriber receives a full copy of the stream. + +`/stream` is not wire- or protocol-compatible with [`tap`](../concepts/vs-tap.md). tap uses a sharded work queue, server-managed acknowledgements, and redelivery; hydrant uses broadcast delivery and client-managed cursors. + +## GET /stream + +subscribe to the ordered event stream. + +### query parameters + +| param | type | description | +| :--- | :--- | :--- | +| `cursor` | integer | inclusive `u64` event ID from which to replay durable record events. | + +if `cursor` is omitted, the connection starts at the current stream head and receives only new events. if it is present, hydrant replays durable record events whose IDs are greater than or equal to the cursor, up to the head observed when the connection starts, then switches to live delivery. + +`cursor=0` replays all retained record events. IDs are monotonic but may have gaps: live-only identity and account events consume IDs without being persisted, and events that cannot be inflated are skipped. + +### event envelope + +events are JSON text frames with this envelope: + +```json +{ + "id": 42, + "type": "record", + "record": {} +} +``` + +| field | type | description | +| :--- | :--- | :--- | +| `id` | integer | `u64` stream event ID. | +| `type` | string | `record`, `identity`, or `account`. | +| `record` | object | present when `type` is `record`. | +| `identity` | object | present when `type` is `identity`. | +| `account` | object | present when `type` is `account`. | + +only the payload matching `type` is present. + +### record + +record events are durable and available through cursor replay. + +```json +{ + "id": 42, + "type": "record", + "record": { + "live": true, + "did": "did:plc:abc123xyz", + "rev": "3kpjxabc123", + "collection": "app.bsky.feed.post", + "rkey": "3kpjxabc123", + "action": "create", + "record": { + "$type": "app.bsky.feed.post", + "text": "hello, world!", + "createdAt": "2026-05-27T12:00:00.000Z" + }, + "cid": "bafyreihy..." + } +} +``` + +| field | type | description | +| :--- | :--- | :--- | +| `live` | boolean | `true` for live ingestion; `false` for backfill. | +| `did` | string | repository DID. | +| `rev` | string | repository revision TID. | +| `collection` | string | collection NSID. | +| `rkey` | string | record key. | +| `action` | string | `create`, `update`, or `delete`. | +| `record` | object | decoded DAG-CBOR record; omitted when no record content is stored. | +| `cid` | string | CID of `record`; omitted when no record content is stored. | + +`record` and `cid` are omitted for deletes and when hydrant is configured not to store record content, including `HYDRANT_ONLY_INDEX_LINKS=true`. + +### identity + +identity events are live-only and are not available through cursor replay. + +```json +{ + "id": 43, + "type": "identity", + "identity": { + "did": "did:plc:abc123xyz", + "handle": "user.bsky.social" + } +} +``` + +| field | type | description | +| :--- | :--- | :--- | +| `did` | string | repository DID. | +| `handle` | string | current handle; omitted when unavailable. | + +### account + +account events are live-only and are not available through cursor replay. + +```json +{ + "id": 44, + "type": "account", + "account": { + "did": "did:plc:abc123xyz", + "active": false, + "status": "deactivated" + } +} +``` + +| field | type | description | +| :--- | :--- | :--- | +| `did` | string | repository DID. | +| `active` | boolean | whether the repository is active. | +| `status` | string | account status; omitted when unavailable. | + +### websocket behavior + +- the server sends an empty ping every 30 seconds and responds to client ping frames with pong frames. +- client text and binary frames are not accepted; hydrant closes the connection when it receives one. +- if sending blocks for `HYDRANT_STREAM_SEND_TIMEOUT` (30 seconds by default), hydrant sends an error text frame and closes the connection: + +```json +{ + "type": "error", + "error": "ConsumerTooSlow", + "message": "stream socket send blocked for at least 30 seconds" +} +``` -- tangled.sh