---
title: crawler management
---

## GET /crawler/sources

list all currently active crawler sources. returns a JSON array of `{ "url": string, "mode": "list_repos" | "relay_first" | "by_collection" }`.

## POST /crawler/sources

add a crawler source at runtime.

| field | description |
| :--- | :--- |
| `url` | URL of the crawler source |
| `mode` | `"list_repos"`, `"relay_first"`, or `"by_collection"`. `relay_first` is the explicit coverage-over-moderation mode described in [relay crawler sources](../concepts/relay.md#crawler-sources) |

the source is written to the database before the producer task is started, so it is safe to add sources and then immediately restart without losing them.

if a source with the same URL already exists (whether from `CRAWLER_URLS` or a previous `POST`), it is replaced: the running task is stopped and a new one is started with the new mode. any cursor state for that URL is preserved.

returns `201 Created` on success.

## DELETE /crawler/sources

remove a crawler source at runtime.

| field | description |
| :--- | :--- |
| `url` | URL of the source to remove |

the producer task is stopped immediately.

if the source was added via the API (`persisted: true`), it is removed from the database and will not reappear on restart. if it came from `CRAWLER_URLS` (`persisted: false`), only the running task is stopped; the source will reappear on the next restart since `CRAWLER_URLS` is re-applied at startup.

cursor state is not cleared. use `DELETE /crawler/cursors` separately if you want the source to restart from the beginning when re-added.

returns `200 OK` if the source was found and removed, `404 Not Found` otherwise.

## DELETE /crawler/cursors

reset stored cursors for a given crawler URL.

| field | description |
| :--- | :--- |
| `key` | URL of the crawler source to reset |

clears the list-repos crawler cursor as well as any by-collection or relay-first pass, host, and per-PDS cursors associated with that URL. causes the next crawler pass to restart from the beginning.