diff --git a/packages/docs/content/docs/api-reference/admin/backfill.md b/packages/docs/content/docs/api-reference/admin/backfill.md index f667856..b4a5a2e 100644 --- a/packages/docs/content/docs/api-reference/admin/backfill.md +++ b/packages/docs/content/docs/api-reference/admin/backfill.md @@ -96,10 +96,61 @@ curl -X POST http://127.0.0.1:3000/admin/backfill \ ```json { "id": "550e8400-e29b-41d4-a716-446655440000", - "status": "pending" + "status": "running" } ``` +## Cancel a backfill job + +``` +POST /admin/backfill/{id}/cancel +``` + +Requests cancellation of a running backfill job. The job status transitions to `cancelling` immediately; the background worker will stop at its next checkpoint and set the final status to `cancelled`. See the [Backfill guide](../../guides/backfill.md#cancelling-a-job) for details on the two-phase process. + +```ts tab="TypeScript" tab-group="language" +const response = await fetch( + `http://127.0.0.1:3000/admin/backfill/${jobId}/cancel`, + { method: "POST", headers }, +); +const data = await response.json(); +``` +```js tab="JavaScript" tab-group="language" +const response = await fetch( + `http://127.0.0.1:3000/admin/backfill/${jobId}/cancel`, + { method: "POST", headers }, +); +const data = await response.json(); +``` +```rust tab="Rust" tab-group="language" +let response = client + .post(format!("http://127.0.0.1:3000/admin/backfill/{job_id}/cancel")) + .bearer_auth(token) + .send() + .await?; +let data: serde_json::Value = response.json().await?; +``` +```go tab="Go" tab-group="language" +url := fmt.Sprintf("http://127.0.0.1:3000/admin/backfill/%s/cancel", jobID) +req, _ := http.NewRequest("POST", url, nil) +req.Header.Set("Authorization", "Bearer "+token) +resp, err := http.DefaultClient.Do(req) +``` +```sh tab="cURL" tab-group="language" +curl -X POST "http://127.0.0.1:3000/admin/backfill/$JOB_ID/cancel" -H "$AUTH" +``` + +**Response**: `200 OK` + +```json +{ + "id": "550e8400-e29b-41d4-a716-446655440000", + "status": "cancelling" +} +``` + +Returns `400` if the job is not currently running, or `404` if the job ID is not found. + ## List backfill jobs ``` @@ -112,9 +163,10 @@ interface BackfillJob { collection: string | null; did: string | null; status: string; - total_repos: number; - processed_repos: number; - total_records: number; + stage: string; + total_repos: number | null; + processed_repos: number | null; + total_records: number | null; error: string | null; started_at: string | null; completed_at: string | null; @@ -158,6 +210,7 @@ curl http://127.0.0.1:3000/admin/backfill/status -H "$AUTH" "collection": "xyz.statusphere.status", "did": null, "status": "completed", + "stage": "completed", "total_repos": 42, "processed_repos": 42, "total_records": 1000, @@ -168,3 +221,5 @@ curl http://127.0.0.1:3000/admin/backfill/status -H "$AUTH" } ] ``` + +The `status` field tracks the overall job state (`running`, `cancelling`, `cancelled`, `completed`, `failed`). The `stage` field tracks the current processing phase (`pending`, `discovering_repos`, `resolving_pds`, `fetching_records`, `completed`, `failed`, `cancelled`). diff --git a/packages/docs/content/docs/getting-started/dashboard.md b/packages/docs/content/docs/getting-started/dashboard.md index 324a49b..1e55bb8 100644 --- a/packages/docs/content/docs/getting-started/dashboard.md +++ b/packages/docs/content/docs/getting-started/dashboard.md @@ -49,7 +49,7 @@ Navigate to **Records** to browse all indexed atproto records. Records are group ### Backfill -Navigate to **Backfill** to view and manage backfill jobs. You can start a new backfill for any record-type lexicon to import historical records from the network. The page shows job status, progress (repos processed / total), and record counts. See [Backfill](../guides/backfill.md) for how the process works. +Navigate to **Backfill** to view and manage backfill jobs. You can start a new backfill for any record-type lexicon to import historical records from the network. The table shows each job's collection, DID scope, current stage, and start time. Click a row to open a detail sheet with full metadata and a stage-by-stage progress log that updates in real time. Running jobs can be cancelled from the detail sheet — the job transitions to "cancelling" while the worker finishes its current batch, then to "cancelled". See [Backfill](../guides/backfill.md) for how the process works. ### Dead Letters diff --git a/packages/docs/content/docs/guides/backfill.md b/packages/docs/content/docs/guides/backfill.md index 2414484..da7cc42 100644 --- a/packages/docs/content/docs/guides/backfill.md +++ b/packages/docs/content/docs/guides/backfill.md @@ -13,17 +13,49 @@ See the [admin API](../api-reference/admin/backfill.md) for endpoint details. ## How it works -1. **Determine target collections**: uses the specified collection, or all record lexicons with `backfill: true` -2. **Discover DIDs**: HappyView calls the relay's `com.atproto.sync.listReposByCollection` to find repos that contain records for each target collection (paginated) -3. **Resolve each PDS**: for each discovered DID, HappyView resolves the DID document via PLC to find the user's PDS endpoint -4. **Fetch records**: HappyView calls `com.atproto.repo.listRecords` on each PDS for the target collection (paginated) and upserts each record into the local database -5. **Track progress**: counters for `processed_repos` and `total_records` are updated as the job runs +A backfill job runs through three sequential phases: + +1. **Discovering repos** — HappyView calls the relay's `com.atproto.sync.listReposByCollection` to find repos that contain records for each target collection. Discovered DIDs are stored in a tracking table so progress can be resumed. +2. **Resolving PDS** — For each discovered DID, HappyView resolves the DID document (via PLC directory or `did:web`) to find the user's PDS endpoint. +3. **Fetching records** — HappyView calls `com.atproto.repo.listRecords` on each PDS for the target collection(s), upserting each record into the local database. PDS endpoints are processed concurrently (up to 10 PDS hosts, 3 DIDs per host). + +Progress counters (`total_repos`, `processed_repos`, `total_records`) and the current `stage` are updated in real time. The dashboard's Backfill page shows live progress, and clicking a job opens a detail sheet with a stage-by-stage progress log. + +### Rate limiting + +All three phases handle HTTP 429 responses. HappyView reads the `RateLimit-Reset` header (a Unix timestamp, the AT Protocol convention) to determine how long to wait, falling back to the `retry-after` header, then defaulting to 5 seconds. ## Job lifecycle -A backfill job moves through `pending → running → completed` (or `failed`). Unlike earlier versions of HappyView that relied on Tap, the job is only marked `completed` once every discovered repo has been fully processed — there is no separate downstream queue. Progress is visible in real time on the dashboard's Backfill page. +A backfill job has both a `status` (overall state) and a `stage` (current phase): + +| Status | Description | +| ------------ | ---------------------------------------------------- | +| `running` | Job is actively processing | +| `cancelling` | Cancel requested, waiting for the worker to stop | +| `cancelled` | Worker has stopped and cleaned up | +| `completed` | All repos processed successfully | +| `failed` | An error occurred | + +The `stage` field tracks which phase the job is in: `pending`, `discovering_repos`, `resolving_pds`, `fetching_records`, `completed`, `failed`, or `cancelled`. + +## Cancelling a job + +Running jobs can be cancelled via `POST /admin/backfill/{id}/cancel` or the Cancel button in the dashboard. Cancellation is two-phase: + +1. The endpoint sets the job status to `cancelling`. +2. The worker checks for cancellation at natural checkpoints (between relay pages, every 100 DIDs during resolution, every 100 repos during fetching). When it detects the `cancelling` status, it stops work and sets the final status to `cancelled`. + +This means there may be a short delay between clicking Cancel and the job fully stopping, depending on what the worker is doing at that moment. + +## Resuming after restart + +Backfill jobs survive server restarts. On startup, HappyView checks for jobs that were running when the server last stopped: + +- **Running** jobs are re-spawned and resume from where they left off. Each phase is idempotent — discovery skips already-known DIDs, resolution skips already-resolved endpoints, and fetching skips already-completed repos. +- **Cancelling** jobs (where the cancel was requested but the worker hadn't stopped yet) are immediately finalised as `cancelled`. -If a job fails midway, the `error` field contains the failure reason. Re-running the backfill resumes from scratch but is idempotent (records are upserted by URI). +Per-DID progress is tracked in the database, so a job that was halfway through fetching records will pick up from the next unprocessed repo, not start over. ## Re-running backfills