title: crawler management #
GET /crawler/sources #
list all currently active crawler sources. returns a JSON array of { "url": string, "mode": "list_repos" | "relay_first" | "by_collection" }.
POST /crawler/sources #
add a crawler source at runtime.
| field | description |
|---|---|
url |
URL of the crawler source |
mode |
"list_repos", "relay_first", or "by_collection". relay_first is the explicit coverage-over-moderation mode described in relay crawler sources |
the source is written to the database before the producer task is started, so it is safe to add sources and then immediately restart without losing them.
if a source with the same URL already exists (whether from CRAWLER_URLS or a previous POST), it is replaced: the running task is stopped and a new one is started with the new mode. any cursor state for that URL is preserved.
returns 201 Created on success.
DELETE /crawler/sources #
remove a crawler source at runtime.
| field | description |
|---|---|
url |
URL of the source to remove |
the producer task is stopped immediately.
if the source was added via the API (persisted: true), it is removed from the database and will not reappear on restart. if it came from CRAWLER_URLS (persisted: false), only the running task is stopped; the source will reappear on the next restart since CRAWLER_URLS is re-applied at startup.
cursor state is not cleared. use DELETE /crawler/cursors separately if you want the source to restart from the beginning when re-added.
returns 200 OK if the source was found and removed, 404 Not Found otherwise.
DELETE /crawler/cursors #
reset stored cursors for a given crawler URL.
| field | description |
|---|---|
key |
URL of the crawler source to reset |
clears the list-repos crawler cursor as well as any by-collection or relay-first pass, host, and per-PDS cursors associated with that URL. causes the next crawler pass to restart from the beginning.