diff --git a/docs/README.md b/docs/README.md index c3cdd4b..d741b3d 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,22 +1,26 @@ -`hydrant` is an AT Protocol indexer built on the `fjall` database. it's built to be flexible, supporting both full-network indexing and filtered indexing (e.g., by DID), allowing querying with XRPCs (not only `com.atproto.*`!), providing an ordered event stream, etc. oh and it can also act as a relay! - -you can see [random.wisp.place](https://tangled.org/did:plc:dfl62fgb7wtjj3fcbb72naae/random.wisp.place) (standalone binary using http API) or the [statusphere example](https://tangled.org/did:plc:dfl62fgb7wtjj3fcbb72naae/hydrant/blob/main/examples/statusphere.rs) (hydrant-as-library) for examples. for rust docs look at https://hydrant.klbr.net/ for now. - -**WARNING: *the db format is only partially stable.*** we provide migrations in hydrant itself, so nothing should go wrong! you should still probably keep backups just in case! - -## what's here - -- [getting started](getting-started.md): building, running, reverse proxying -- [configuration](configuration.md): all environment variables -- [build features](build-features.md): optional cargo features (`relay`, `backlinks`, etc.) -- [concepts](concepts/README.md): how the stream works, relay comparison, multi-relay support -- [rest api](api/README.md): management API reference -- [xrpc](xrpc/README.md): data access via XRPC - -## quick start - -```bash -cargo build --release -export HYDRANT_DATABASE_PATH=./hydrant.db -./target/release/hydrant -``` +--- +title: hydrant +--- + +`hydrant` is an AT Protocol indexer built on the `fjall` database. it's built to be flexible, supporting both full-network indexing and filtered indexing (e.g., by DID), allowing querying with XRPCs (not only `com.atproto.*`!), providing an ordered event stream, etc. oh and it can also act as a relay! + +you can see [random.wisp.place](https://tangled.org/did:plc:dfl62fgb7wtjj3fcbb72naae/random.wisp.place) (standalone binary using http API) or the [statusphere example](https://tangled.org/did:plc:dfl62fgb7wtjj3fcbb72naae/hydrant/blob/main/examples/statusphere.rs) (hydrant-as-library) for examples. for rust docs look at https://hydrant.klbr.net/ for now. + +**WARNING: *the db format is only partially stable.*** we provide migrations in hydrant itself, so nothing should go wrong! you should still probably keep backups just in case! + +## what's here + +- [getting started](getting-started.md): building, running, reverse proxying +- [configuration](configuration.md): all environment variables +- [build features](build-features.md): optional cargo features (`relay`, `backlinks`, etc.) +- [concepts](concepts/README.md): how the stream works, relay comparison, multi-relay support +- [rest api](api/README.md): management API reference +- [xrpc](xrpc/README.md): data access via XRPC + +## quick start + +```bash +cargo build --release +export HYDRANT_DATABASE_PATH=./hydrant.db +./target/release/hydrant +``` diff --git a/docs/api/README.md b/docs/api/README.md index 88def89..8ee044c 100644 --- a/docs/api/README.md +++ b/docs/api/README.md @@ -1,4 +1,6 @@ -# rest api +--- +title: rest api +--- hydrant's REST API is split into public endpoints (safe to expose) and management endpoints (keep private). see [getting started](../getting-started.md#reverse-proxying) for guidance on what to expose. diff --git a/docs/api/crawler.md b/docs/api/crawler.md index 3953030..0f113ff 100644 --- a/docs/api/crawler.md +++ b/docs/api/crawler.md @@ -1,4 +1,6 @@ -# crawler management +--- +title: crawler management +--- ## GET /crawler/sources diff --git a/docs/api/database.md b/docs/api/database.md index 7f01d78..f40b182 100644 --- a/docs/api/database.md +++ b/docs/api/database.md @@ -1,4 +1,6 @@ -# database operations +--- +title: database operations +--- - `POST /db/train`: train zstd compression dictionaries for the `repos`, `blocks`, and `events` keyspaces. dictionaries are written to disk; a restart is required to apply them. the crawler, firehose, and backfill worker are paused for the duration and restored on completion. - `POST /db/compact`: trigger a full major compaction of all database keyspaces in parallel. the crawler, firehose, and backfill worker are paused for the duration and restored on completion. diff --git a/docs/api/filter.md b/docs/api/filter.md index 1408fdd..89851f7 100644 --- a/docs/api/filter.md +++ b/docs/api/filter.md @@ -1,4 +1,6 @@ -# filter management +--- +title: filter management +--- - `GET /filter`: get the current filter configuration. - `PATCH /filter`: update the filter configuration. diff --git a/docs/api/firehose.md b/docs/api/firehose.md index 5704c8f..e68fab4 100644 --- a/docs/api/firehose.md +++ b/docs/api/firehose.md @@ -1,4 +1,6 @@ -# firehose management +--- +title: firehose management +--- ## GET /firehose/sources diff --git a/docs/api/ingestion.md b/docs/api/ingestion.md index 77d82fc..ecb26b5 100644 --- a/docs/api/ingestion.md +++ b/docs/api/ingestion.md @@ -1,4 +1,6 @@ -# ingestion control +--- +title: ingestion control +--- ## GET /ingestion diff --git a/docs/api/pds.md b/docs/api/pds.md index a516e6a..ab8bf2d 100644 --- a/docs/api/pds.md +++ b/docs/api/pds.md @@ -1,4 +1,6 @@ -# PDS management +--- +title: PDS management +--- hydrant rate-limits firehose events per PDS. each PDS is assigned to a named rate tier that controls how aggressively hydrant limits events from it. two built-in tiers are always present: `default` (conservative limits for unknown operators) and `trusted` (higher limits for well-behaved operators). additional tiers can be defined via `RATE_TIERS`. diff --git a/docs/api/repos.md b/docs/api/repos.md index 9a17763..92ef337 100644 --- a/docs/api/repos.md +++ b/docs/api/repos.md @@ -1,4 +1,6 @@ -# repository management +--- +title: repository management +--- all `/repos` endpoints that return lists respond with NDJSON by default. send `Accept: application/json` or `Content-Type: application/json` to get a JSON array instead. diff --git a/docs/build-features.md b/docs/build-features.md index d152436..6b98aec 100644 --- a/docs/build-features.md +++ b/docs/build-features.md @@ -1,4 +1,6 @@ -# build features +--- +title: build features +--- `hydrant` has several optional compile-time features: diff --git a/docs/concepts/README.md b/docs/concepts/README.md index f29084a..8ba1d89 100644 --- a/docs/concepts/README.md +++ b/docs/concepts/README.md @@ -1,4 +1,6 @@ -# concepts +--- +title: concepts +--- - [hydrant vs tap](vs-tap.md): design comparison, stream behavior - [relay & seeding](relay.md): multi-relay support, firehose seeding, crawler sources diff --git a/docs/concepts/relay.md b/docs/concepts/relay.md index 2cc15e8..0a1a75e 100644 --- a/docs/concepts/relay.md +++ b/docs/concepts/relay.md @@ -1,4 +1,6 @@ -# relay, seeding & crawler sources +--- +title: relay, seeding & crawler sources +--- ## multiple relay support diff --git a/docs/concepts/vs-tap.md b/docs/concepts/vs-tap.md index 445d970..8b89fda 100644 --- a/docs/concepts/vs-tap.md +++ b/docs/concepts/vs-tap.md @@ -1,4 +1,6 @@ -# hydrant vs tap +--- +title: hydrant vs tap +--- while [`tap`](https://github.com/bluesky-social/indigo/tree/main/cmd/tap) is designed as a firehose consumer and simply just propagates events while handling sync, `hydrant` is flexible, it allows you to directly query the database for records, and it also provides an ordered view of events, allowing the use of a cursor to fetch events from a specific point. it can act as both an indexer or an ephemeral view of some window of events. diff --git a/docs/configuration.md b/docs/configuration.md index 1109c23..9072233 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1,4 +1,6 @@ -# configuration +--- +title: configuration +--- hydrant is configured via environment variables, all prefixed with `HYDRANT_` (except `RUST_LOG`). a `.env` file in the working directory is loaded automatically. diff --git a/docs/getting-started.md b/docs/getting-started.md index 54ffd69..1ce28ca 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -1,4 +1,6 @@ -# getting started +--- +title: getting started +--- ## requirements diff --git a/docs/xrpc/README.md b/docs/xrpc/README.md index 1860b6b..d4625ae 100644 --- a/docs/xrpc/README.md +++ b/docs/xrpc/README.md @@ -1,4 +1,6 @@ -# xrpc +--- +title: xrpc +--- `hydrant` implements the following XRPC endpoints under `/xrpc/`. only expose `/xrpc/*` publicly, see [getting started](../getting-started.md#reverse-proxying) for guidance. diff --git a/docs/xrpc/atproto.md b/docs/xrpc/atproto.md index b7822b5..822449d 100644 --- a/docs/xrpc/atproto.md +++ b/docs/xrpc/atproto.md @@ -1,4 +1,6 @@ -# com.atproto.* +--- +title: com.atproto.* +--- these are standard atproto endpoints. you can look at [the atproto api reference](https://docs.bsky.app/docs/category/http-reference) for more info. diff --git a/docs/xrpc/backlinks.md b/docs/xrpc/backlinks.md index f38e237..fd146a7 100644 --- a/docs/xrpc/backlinks.md +++ b/docs/xrpc/backlinks.md @@ -1,4 +1,6 @@ -# blue.microcosm.links.* +--- +title: blue.microcosm.links.* +--- hydrant implements a subset of [microcosm constellation](https://constellation.microcosm.blue/) when it's built with the `backlinks` cargo feature (`cargo build --features backlinks`). diff --git a/docs/xrpc/hydrant.md b/docs/xrpc/hydrant.md index bc00306..7cb1618 100644 --- a/docs/xrpc/hydrant.md +++ b/docs/xrpc/hydrant.md @@ -1,4 +1,6 @@ -# systems.gaze.hydrant.* +--- +title: systems.gaze.hydrant.* +--- these are some non-standard XRPCs that might be useful.