diff --git a/docs/api/crawler.md b/docs/api/crawler.md index b83fe90..4f40523 100644 --- a/docs/api/crawler.md +++ b/docs/api/crawler.md @@ -1,17 +1,33 @@ # crawler management -- `GET /crawler/sources`: list all currently active crawler sources. - - returns a JSON array of `{ "url": string, "mode": "relay" | "by_collection", "persisted": bool }`. - - `persisted: true` means the source was added via the API and is stored in the database, it will survive a restart. `persisted: false` means the source came from `CRAWLER_URLS` and is not written to the database. -- `POST /crawler/sources`: add a crawler source at runtime. - - body: `{ "url": string, "mode": "relay" | "by_collection" }`. - - the source is written to the database before the producer task is started, so it is safe to add sources and then immediately restart without losing them. - - if a source with the same URL already exists (whether from `CRAWLER_URLS` or a previous `POST`), it is replaced: the running task is stopped and a new one is started with the new mode. any cursor state for that URL is preserved. - - returns `201 Created` on success. -- `DELETE /crawler/sources`: remove a crawler source at runtime. - - body: `{ "url": string }`. - - the producer task is stopped immediately. - - if the source was added via the API (`persisted: true`), it is removed from the database and will not reappear on restart. if it came from `CRAWLER_URLS` (`persisted: false`), only the running task is stopped, the source will reappear on the next restart since `CRAWLER_URLS` is re-applied at startup. - - cursor state is not cleared. use `DELETE /crawler/cursors` separately if you want the source to restart from the beginning when re-added. - - returns `200 OK` if the source was found and removed, `404 Not Found` otherwise. -- `DELETE /crawler/cursors`: reset stored cursors for a given crawler URL. body: `{ "key": "..." }` where key is a URL. clears the list-repos crawler cursor as well as any by-collection cursors associated with that URL. causes the next crawler pass to restart from the beginning. +## GET /crawler/sources + +list all currently active crawler sources. returns a JSON array of `{ "url": string, "mode": "relay" | "by_collection", "persisted": bool }`. + +`persisted: true` means the source was added via the API and is stored in the database; it will survive a restart. `persisted: false` means the source came from `CRAWLER_URLS` and is not written to the database. + +## POST /crawler/sources + +add a crawler source at runtime. body: `{ "url": string, "mode": "relay" | "by_collection" }`. + +the source is written to the database before the producer task is started, so it is safe to add sources and then immediately restart without losing them. + +if a source with the same URL already exists (whether from `CRAWLER_URLS` or a previous `POST`), it is replaced: the running task is stopped and a new one is started with the new mode. any cursor state for that URL is preserved. + +returns `201 Created` on success. + +## DELETE /crawler/sources + +remove a crawler source at runtime. body: `{ "url": string }`. + +the producer task is stopped immediately. + +if the source was added via the API (`persisted: true`), it is removed from the database and will not reappear on restart. if it came from `CRAWLER_URLS` (`persisted: false`), only the running task is stopped; the source will reappear on the next restart since `CRAWLER_URLS` is re-applied at startup. + +cursor state is not cleared. use `DELETE /crawler/cursors` separately if you want the source to restart from the beginning when re-added. + +returns `200 OK` if the source was found and removed, `404 Not Found` otherwise. + +## DELETE /crawler/cursors + +reset stored cursors for a given crawler URL. body: `{ "key": "..." }` where key is a URL. clears the list-repos crawler cursor as well as any by-collection cursors associated with that URL. causes the next crawler pass to restart from the beginning. diff --git a/docs/api/firehose.md b/docs/api/firehose.md index 31dcbdc..b6415e0 100644 --- a/docs/api/firehose.md +++ b/docs/api/firehose.md @@ -1,18 +1,33 @@ # firehose management -- `GET /firehose/sources`: list all currently active firehose sources. - - returns a JSON array of `{ "url": string, "persisted": bool, "is_pds": bool }`. - - `persisted: true` means the source was added via the API and is stored in the database, it will survive a restart. `persisted: false` means the source came from `RELAY_HOSTS` and is not written to the database. - - `is_pds: true` means the source is a direct PDS connection with host authority enforcement enabled. -- `POST /firehose/sources`: add a firehose source at runtime. - - body: `{ "url": string, "is_pds": bool }`. `is_pds` defaults to `false`. - - the source is persisted to the database before the ingestor task is started. - - if a source with the same URL already exists, it is replaced: the running task is stopped and a new one is started. any existing cursor state for that URL is preserved. - - returns `201 Created` on success. -- `DELETE /firehose/sources`: remove a firehose relay at runtime. - - body: `{ "url": string }`. - - the ingestor task is stopped immediately. - - if the source was added via the API (`persisted: true`), it is removed from the database and will not reappear on restart. if it came from `RELAY_HOSTS` (`persisted: false`), only the running task is stopped; the source reappears on the next restart. - - cursor state is not cleared. use `DELETE /firehose/cursors` separately if you want the relay to restart from the beginning when re-added. - - returns `200 OK` if the relay was found and removed, `404 Not Found` otherwise. -- `DELETE /firehose/cursors`: reset the stored cursor for a given firehose relay URL. body: `{ "key": "..." }` where key is a URL. causes the next firehose connection to restart from the beginning. +## GET /firehose/sources + +list all currently active firehose sources. returns a JSON array of `{ "url": string, "persisted": bool, "is_pds": bool }`. + +`persisted: true` means the source was added via the API and is stored in the database; it will survive a restart. `persisted: false` means the source came from `RELAY_HOSTS` and is not written to the database. `is_pds: true` means the source is a direct PDS connection with host authority enforcement enabled. + +## POST /firehose/sources + +add a firehose source at runtime. body: `{ "url": string, "is_pds": bool }`. `is_pds` defaults to `false`. + +the source is persisted to the database before the ingestor task is started. + +if a source with the same URL already exists, it is replaced: the running task is stopped and a new one is started. any existing cursor state for that URL is preserved. + +returns `201 Created` on success. + +## DELETE /firehose/sources + +remove a firehose relay at runtime. body: `{ "url": string }`. + +the ingestor task is stopped immediately. + +if the source was added via the API (`persisted: true`), it is removed from the database and will not reappear on restart. if it came from `RELAY_HOSTS` (`persisted: false`), only the running task is stopped; the source reappears on the next restart. + +cursor state is not cleared. use `DELETE /firehose/cursors` separately if you want the relay to restart from the beginning when re-added. + +returns `200 OK` if the relay was found and removed, `404 Not Found` otherwise. + +## DELETE /firehose/cursors + +reset the stored cursor for a given firehose relay URL. body: `{ "key": "..." }` where key is a URL. causes the next firehose connection to restart from the beginning. diff --git a/docs/api/ingestion.md b/docs/api/ingestion.md index f6037c5..a63a8f7 100644 --- a/docs/api/ingestion.md +++ b/docs/api/ingestion.md @@ -1,7 +1,11 @@ # ingestion control -- `GET /ingestion`: get the current ingestion status. - - returns `{ "crawler": bool, "firehose": bool, "backfill": bool }`. -- `PATCH /ingestion`: enable or disable ingestion components at runtime without restarting. - - body: `{ "crawler"?: bool, "firehose"?: bool, "backfill"?: bool }`. only provided fields are updated. - - when disabled, each component finishes its current task before pausing (e.g. the backfill worker completes any in-flight repo syncs, the firehose finishes processing the current message). they resume immediately when re-enabled. +## GET /ingestion + +get the current ingestion status. returns `{ "crawler": bool, "firehose": bool, "backfill": bool }`. + +## PATCH /ingestion + +enable or disable ingestion components at runtime without restarting. body: `{ "crawler"?: bool, "firehose"?: bool, "backfill"?: bool }`. only provided fields are updated. + +when disabled, each component finishes its current task before pausing (e.g. the backfill worker completes any in-flight repo syncs, the firehose finishes processing the current message). they resume immediately when re-enabled. diff --git a/docs/api/pds.md b/docs/api/pds.md index 951ddc9..68094f0 100644 --- a/docs/api/pds.md +++ b/docs/api/pds.md @@ -6,30 +6,39 @@ the per-second limit scales with the number of active accounts on the PDS: `max( you can also define an optional `account_limit` for a rate tier. if a PDS exceeds this number of active accounts, hydrant will reject any new account creation events from it. -the built-in tiers are defined as follows: -- `default`: `50` per sec (floor), `+0.5` per account. max `3_600_000`/hr, `86_400_000`/day. `100` account limit. -- `trusted`: `5000` per sec (floor), `+10.0` per account. max `18_000_000`/hr, `432_000_000`/day. `10_000_000` account limit. +the built-in tiers are: -tiers are resolved in this order: +| tier | per_second_base | per_second_account_mul | per_hour | per_day | account_limit | +| :--- | :--- | :--- | :--- | :--- | :--- | +| `default` | 50 | +0.5 | 3,600,000 | 86,400,000 | 100 | +| `trusted` | 5000 | +10.0 | 18,000,000 | 432,000,000 | 10,000,000 | -1. **explicit API assignment**, set via `PUT /pds/tiers`, stored in the database, survives restarts. -2. **glob rules**, from `TIER_RULES`, evaluated in order; first match wins. -3. **`default` tier**, applied if no rule or explicit assignment matches. +tiers are resolved in this order: explicit API assignment (set via `PUT /pds/tiers`, stored in the database, survives restarts), then glob rules (from `TIER_RULES`, evaluated in order; first match wins), then the `default` tier (applied if nothing else matches). deleting an API assignment reverts the host to glob-rule resolution, not necessarily back to `default`. if a rule like `*.bsky.network:trusted` matches the host, it will become trusted again without any further action. -- `GET /pds/tiers`: list all current tier assignments alongside the available tier definitions. - - returns `{ "assignments": [{ "host": string, "tier": string }], "rate_tiers": { : { "per_second_base": int, "per_second_account_mul": float, "per_hour": int, "per_day": int } } }`. - - `assignments` only contains PDSes with an explicit API assignment. hosts without one resolve via glob rules or fall back to `default`. -- `PUT /pds/tiers`: assign a PDS to a named rate tier. - - body: `{ "host": string, "tier": string }`. - - `host` is the PDS hostname (e.g. `pds.example.com`). - - `tier` must be one of the configured tier names. returns `400` if unknown. - - assignments are persisted to the database and survive restarts. - - re-assigning the same host updates the tier in place without creating a duplicate. -- `DELETE /pds/tiers`: remove an explicit tier assignment for a PDS. - - query parameter: `?host=` (e.g. `?host=pds.example.com`). - - reverts the host to glob-rule resolution (not necessarily `default`, a matching `TIER_RULES` pattern still applies). - - returns `200` even if no assignment existed. -- `GET /pds/rate-tiers`: list the available rate tier definitions. - - returns a map of tier name to `{ "per_second_base", "per_second_account_mul", "per_hour", "per_day", "account_limit" }`. +## GET /pds/tiers + +list all current tier assignments alongside the available tier definitions. returns `{ "assignments": [{ "host": string, "tier": string }], "rate_tiers": { : { "per_second_base": int, "per_second_account_mul": float, "per_hour": int, "per_day": int } } }`. + +`assignments` only contains PDSes with an explicit API assignment. hosts without one resolve via glob rules or fall back to `default`. + +## PUT /pds/tiers + +assign a PDS to a named rate tier. body: `{ "host": string, "tier": string }`. + +`host` is the PDS hostname (e.g. `pds.example.com`). `tier` must be one of the configured tier names; returns `400` if unknown. + +assignments are persisted to the database and survive restarts. re-assigning the same host updates the tier in place without creating a duplicate. + +## DELETE /pds/tiers + +remove an explicit tier assignment for a PDS. query parameter: `?host=` (e.g. `?host=pds.example.com`). + +reverts the host to glob-rule resolution (not necessarily `default`; a matching `TIER_RULES` pattern still applies). + +returns `200` even if no assignment existed. + +## GET /pds/rate-tiers + +list the available rate tier definitions. returns a map of tier name to `{ "per_second_base", "per_second_account_mul", "per_hour", "per_day", "account_limit" }`. diff --git a/docs/api/repos.md b/docs/api/repos.md index 24b0777..9a17763 100644 --- a/docs/api/repos.md +++ b/docs/api/repos.md @@ -2,10 +2,27 @@ all `/repos` endpoints that return lists respond with NDJSON by default. send `Accept: application/json` or `Content-Type: application/json` to get a JSON array instead. -- `GET /repos`: get a list of repositories and their sync status. supports pagination and filtering: - - `limit`: max results (default 100, max 1000) - - `cursor`: did key for paginating. -- `GET /repos/{did}`: get the sync status and metadata of a specific repository. also returns the handle, PDS URL and the atproto signing key (these won't be available before the repo has been backfilled once at least). -- `PUT /repos`: explicitly track repositories. accepts an NDJSON body of `{"did": "..."}` (or JSON array of the same). only affects repositories that are not known or are untracked. returns a list of the DIDs that were queued for backfill. -- `DELETE /repos`: untrack repositories. accepts an NDJSON body of `{"did": "..."}` (or JSON array of the same). only affects repositories that are currently tracked. returns a list of the DIDs that were untracked. -- `POST /repos/resync`: force a new backfill for one or more repositories. accepts an NDJSON body of `{"did": "..."}` (or JSON array of the same). only affects repositories hydrant already knows about. returns a list of the DIDs that were queued. +## GET /repos + +get a list of repositories and their sync status. supports pagination and filtering: + +| param | description | +| :--- | :--- | +| `limit` | max results (default 100, max 1000) | +| `cursor` | did key for paginating | + +## GET /repos/{did} + +get the sync status and metadata of a specific repository. also returns the handle, PDS URL and the atproto signing key (these won't be available before the repo has been backfilled at least once). + +## PUT /repos + +explicitly track repositories. accepts an NDJSON body of `{"did": "..."}` (or JSON array of the same). only affects repositories that are not known or are untracked. returns a list of the DIDs that were queued for backfill. + +## DELETE /repos + +untrack repositories. accepts an NDJSON body of `{"did": "..."}` (or JSON array of the same). only affects repositories that are currently tracked. returns a list of the DIDs that were untracked. + +## POST /repos/resync + +force a new backfill for one or more repositories. accepts an NDJSON body of `{"did": "..."}` (or JSON array of the same). only affects repositories hydrant already knows about. returns a list of the DIDs that were queued.