diff --git a/README.md b/README.md index 359a2f2..bdd313a 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ Building an AppView from scratch means wiring up real-time event streams, record - **Network sync built in:** Real-time record streaming via Jetstream, historical backfill from each user's PDS, and atproto OAuth with DPoP-bound proxy writes back to the PDS. -- **Customize with Lua, hooks, and plugins:** Lua scripts for query and procedure logic, index hooks that fire on every record change, WASM plugins for external platform integration, and labeler subscriptions for content moderation. +- **Customize with Lua scripts and plugins:** Trigger-keyed Lua scripts for XRPC query/procedure logic and record/label event handling, WASM plugins for external platform integration, and labeler subscriptions for content moderation. - **Protocol-native:** Works with any PDS, resolves DIDs through the directory, and fetches network lexicons via DNS authority resolution. diff --git a/packages/docs/content/blog/happyview-2.9.md b/packages/docs/content/blog/happyview-2.9.md index 5421367..b83db6b 100644 --- a/packages/docs/content/blog/happyview-2.9.md +++ b/packages/docs/content/blog/happyview-2.9.md @@ -1,6 +1,6 @@ --- title: "HappyView v2.9" -description: "Backfill concurrency, db.query filters, multi-device DPoP sessions, and a mountain of performance fixes." +description: "Trigger-keyed scripts, backfill concurrency, db.query filters, multi-device DPoP sessions, plus bugs fixes and performance improvements." date: 2026-05-26 author: name: "Trezy" @@ -9,9 +9,32 @@ tags: - announcements --- -While there's not a lot of big, shiny new features this time around, 2.9 is chock full of performance improvements and bug fixes to make everybody's life better. +2.9 is a big one. Scripts got an overhaul, backfill got way faster, and there's a mountain of bug fixes and performance improvements across the board. -## Backfill, but make it concurrent +## Trigger-keyed scripts + +The biggest conceptual change in 2.9: scripts are no longer embedded with lexicons, and lexicons are no longer limited to a single script. + +| Trigger | Fires when... | +| ----------------------- | ----------------------------------------------------------------- | +| `record.index:` | Any record event (create, update, delete) — the wildcard fallback | +| `record.create:` | A record is created | +| `record.update:` | A record is updated | +| `record.delete:` | A record is deleted | +| `xrpc.query:` | An XRPC query is called | +| `xrpc.procedure:` | An XRPC procedure is called | +| `labeler.apply:` | A label arrives on a record of this type | +| `labeler.apply:_actor` | A label arrives on a bare DID (actor-level) | + +For record events, the dispatcher tries the action-specific trigger first (e.g. `record.create:com.example.post`), then falls back to the wildcard `record.index:com.example.post`. This means you can have one general-purpose script that handles everything, or surgical scripts for specific actions — or both. + +Scripts are managed through the dashboard under **Settings > Scripts**, or via the new [`/admin/scripts`](/docs/api-reference/admin/scripts) API endpoints. The lexicon detail page also shows which scripts target each lexicon, with links to create or edit them. + +**If you're upgrading from v2.x to v2.9:** existing index hooks and lexicon scripts will be migrated to the new system automatically. + +Full docs: [Record & Label Scripts](/docs/guides/label-scripts), [Lua Scripting](/docs/guides/lua-scripting), [Admin API — Scripts](/docs/api-reference/admin/scripts). + +## Backfill, but concurrent The biggest change is that PDS resolution and record fetching now run concurrently. Previously, HappyView resolved every DID's PDS endpoint before it started fetching any records. For large backfills with hundreds of thousands of DIDs, that meant the fetcher sat idle for potentially hours. Now fetching starts as soon as the first DIDs are resolved and runs alongside resolution for the rest of the job. @@ -55,11 +78,11 @@ Full docs are in the [Database API reference](/docs/api-reference/lua/database-a ## Auth fixes -There were actually bugs that have been making my life hard, but I _finally_ figured them out. +These were bugs that have been making my life hard, but I _finally_ figured out what was causing them. First, users were basically limited to one auth session per client. If you signed into [Cartridge](https://cartridge.dev) from a second device, it would kill your other auth session. Whoops. -Second, there were scenarios where the PDS may refresh the auth session while a HappyView XRPC was in-progress. IF that happened, HappyView would handle it internally so any other requests in that XRPC worked, but _it didn't return the refreshed tokens to the client._ Follow up requests from the client would break. Double whoops. +Second, there were scenarios where the PDS may refresh the auth session while a HappyView XRPC was in-progress. If that happened, HappyView would handle it internally so any other requests in that XRPC worked, but _it didn't return the refreshed tokens to the client._ Follow up requests from the client would break. Double whoops. Both of these are fixed properly now, AND I added a couple new endpoints so clients can allow users to see and manage their active sessions: @@ -74,7 +97,7 @@ If you tried to use `@happyview/oauth-client` with the latest versions of the `@ ## CI & infrastructure -- **Binary releases** — Rust binaries are now published to GitHub Releases alongside Docker images, so you can grab a prebuilt binary directly. +- **Binary releases** — Rust binaries are now published to GitHub Releases alongside Docker images, so you can grab a prebuilt binary directly if you're not into Docker. ## Go play diff --git a/packages/docs/content/docs/api-reference/admin/lexicons.md b/packages/docs/content/docs/api-reference/admin/lexicons.md index 99c92fa..fddad4d 100644 --- a/packages/docs/content/docs/api-reference/admin/lexicons.md +++ b/packages/docs/content/docs/api-reference/admin/lexicons.md @@ -172,8 +172,7 @@ curl -X POST http://127.0.0.1:3000/admin/lexicons \ | `lexicon_json` | object | yes | Raw lexicon JSON (must have `lexicon: 1` and `id`) | | `backfill` | boolean | no | Whether uploading triggers historical backfill (default `true`) | | `target_collection` | string | no | For query/procedure lexicons, the record collection they operate on | -| `script` | string | no | Lua script for query/procedure endpoints | -| `index_hook` | string | no | [Index hook](../../guides/index-hooks.md) Lua script for record lexicons | +| `token_cost` | integer | no | Token cost for query/procedure endpoints (overrides instance default) | **Response**: `201 Created` (new) or `200 OK` (upsert) diff --git a/packages/docs/content/docs/api-reference/admin/meta.json b/packages/docs/content/docs/api-reference/admin/meta.json index d5851f1..f1e0c09 100644 --- a/packages/docs/content/docs/api-reference/admin/meta.json +++ b/packages/docs/content/docs/api-reference/admin/meta.json @@ -3,6 +3,7 @@ "pages": [ "admin-api", "lexicons", + "scripts", "records", "stats", "backfill", diff --git a/packages/docs/content/docs/api-reference/admin/scripts.md b/packages/docs/content/docs/api-reference/admin/scripts.md new file mode 100644 index 0000000..4f56bb2 --- /dev/null +++ b/packages/docs/content/docs/api-reference/admin/scripts.md @@ -0,0 +1,405 @@ +--- +title: "Scripts" +--- + +Manage trigger-keyed scripts. Scripts run automatically in response to events like record indexing, XRPC calls, or labeler actions. The trigger id (e.g. `record.index:xyz.statusphere.status`) determines when a script fires. + +**Permissions:** `scripts:read` for GET endpoints, `scripts:manage` for mutating endpoints. + +```ts tab="TypeScript" tab-group="language" +const TOKEN = "hv_..."; // your API key +const headers = { Authorization: `Bearer ${TOKEN}` }; +``` +```js tab="JavaScript" tab-group="language" +const TOKEN = "hv_..."; // your API key +const headers = { Authorization: `Bearer ${TOKEN}` }; +``` +```rust tab="Rust" tab-group="language" +let token = "hv_..."; // your API key +``` +```go tab="Go" tab-group="language" +token := "hv_..." // your API key +``` +```sh tab="cURL" tab-group="language" +# All examples assume $TOKEN is an API key (hv_...) +AUTH="Authorization: Bearer $TOKEN" +``` + +## List scripts + +``` +GET /admin/scripts +``` + +Optionally filter by NSID suffix with the `?suffix=` query parameter. + +```ts tab="TypeScript" tab-group="language" +interface Script { + id: string; + script_type: string; + body: string; + description: string | null; + created_at: string; + updated_at: string; +} + +// List all scripts +const response = await fetch("http://127.0.0.1:3000/admin/scripts", { + headers, +}); +const data: Script[] = await response.json(); + +// Filter by NSID suffix +const filtered = await fetch( + "http://127.0.0.1:3000/admin/scripts?suffix=xyz.statusphere.status", + { headers }, +); +const filteredData: Script[] = await filtered.json(); +``` +```js tab="JavaScript" tab-group="language" +// List all scripts +const response = await fetch("http://127.0.0.1:3000/admin/scripts", { + headers, +}); +const data = await response.json(); + +// Filter by NSID suffix +const filtered = await fetch( + "http://127.0.0.1:3000/admin/scripts?suffix=xyz.statusphere.status", + { headers }, +); +const filteredData = await filtered.json(); +``` +```rust tab="Rust" tab-group="language" +// List all scripts +let response = client + .get("http://127.0.0.1:3000/admin/scripts") + .bearer_auth(token) + .send() + .await?; +let data: serde_json::Value = response.json().await?; + +// Filter by NSID suffix +let response = client + .get("http://127.0.0.1:3000/admin/scripts?suffix=xyz.statusphere.status") + .bearer_auth(token) + .send() + .await?; +let filtered: serde_json::Value = response.json().await?; +``` +```go tab="Go" tab-group="language" +// List all scripts +req, _ := http.NewRequest("GET", "http://127.0.0.1:3000/admin/scripts", nil) +req.Header.Set("Authorization", "Bearer "+token) +resp, err := http.DefaultClient.Do(req) + +// Filter by NSID suffix +req, _ = http.NewRequest("GET", "http://127.0.0.1:3000/admin/scripts?suffix=xyz.statusphere.status", nil) +req.Header.Set("Authorization", "Bearer "+token) +resp, err = http.DefaultClient.Do(req) +``` +```sh tab="cURL" tab-group="language" +# List all scripts +curl http://127.0.0.1:3000/admin/scripts -H "$AUTH" + +# Filter by NSID suffix +curl "http://127.0.0.1:3000/admin/scripts?suffix=xyz.statusphere.status" -H "$AUTH" +``` + +| Parameter | Type | Required | Description | +| --------- | ------ | -------- | ---------------------------------------------------------------- | +| `suffix` | string | no | Filter to scripts whose id ends with `:` (query param) | + +**Response**: `200 OK` + +```json +[ + { + "id": "record.index:xyz.statusphere.status", + "script_type": "lua", + "body": "function handle()\n return event\nend", + "description": "Process indexed statuses", + "created_at": "2026-01-01T00:00:00Z", + "updated_at": "2026-01-01T00:00:00Z" + } +] +``` + +## Get a script + +``` +GET /admin/scripts/{id} +``` + +The `{id}` path parameter is the trigger string, URL-encoded (e.g. `record.index%3Axyz.statusphere.status`). + +```ts tab="TypeScript" tab-group="language" +const response = await fetch( + "http://127.0.0.1:3000/admin/scripts/record.index%3Axyz.statusphere.status", + { headers }, +); +const data: Script = await response.json(); +``` +```js tab="JavaScript" tab-group="language" +const response = await fetch( + "http://127.0.0.1:3000/admin/scripts/record.index%3Axyz.statusphere.status", + { headers }, +); +const data = await response.json(); +``` +```rust tab="Rust" tab-group="language" +let response = client + .get("http://127.0.0.1:3000/admin/scripts/record.index%3Axyz.statusphere.status") + .bearer_auth(token) + .send() + .await?; +let data: serde_json::Value = response.json().await?; +``` +```go tab="Go" tab-group="language" +req, _ := http.NewRequest("GET", "http://127.0.0.1:3000/admin/scripts/record.index%3Axyz.statusphere.status", nil) +req.Header.Set("Authorization", "Bearer "+token) +resp, err := http.DefaultClient.Do(req) +``` +```sh tab="cURL" tab-group="language" +curl "http://127.0.0.1:3000/admin/scripts/record.index%3Axyz.statusphere.status" -H "$AUTH" +``` + +**Response**: `200 OK` + +```json +{ + "id": "record.index:xyz.statusphere.status", + "script_type": "lua", + "body": "function handle()\n return event\nend", + "description": "Process indexed statuses", + "created_at": "2026-01-01T00:00:00Z", + "updated_at": "2026-01-01T00:00:00Z" +} +``` + +## Create or replace a script + +``` +POST /admin/scripts +``` + +Creates a new script or replaces an existing one by `id`. The trigger grammar and Lua body are validated at write-time. + +```ts tab="TypeScript" tab-group="language" +const response = await fetch("http://127.0.0.1:3000/admin/scripts", { + method: "POST", + headers: { + ...headers, + "Content-Type": "application/json", + }, + body: JSON.stringify({ + id: "record.index:xyz.statusphere.status", + script_type: "lua", + body: "function handle()\n return event\nend", + description: "Process indexed statuses", + }), +}); +const data: Script = await response.json(); +``` +```js tab="JavaScript" tab-group="language" +const response = await fetch("http://127.0.0.1:3000/admin/scripts", { + method: "POST", + headers: { + ...headers, + "Content-Type": "application/json", + }, + body: JSON.stringify({ + id: "record.index:xyz.statusphere.status", + script_type: "lua", + body: "function handle()\n return event\nend", + description: "Process indexed statuses", + }), +}); +const data = await response.json(); +``` +```rust tab="Rust" tab-group="language" +let response = client + .post("http://127.0.0.1:3000/admin/scripts") + .bearer_auth(token) + .json(&serde_json::json!({ + "id": "record.index:xyz.statusphere.status", + "script_type": "lua", + "body": "function handle()\n return event\nend", + "description": "Process indexed statuses" + })) + .send() + .await?; +let data: serde_json::Value = response.json().await?; +``` +```go tab="Go" tab-group="language" +body := bytes.NewBufferString(`{ + "id": "record.index:xyz.statusphere.status", + "script_type": "lua", + "body": "function handle()\n return event\nend", + "description": "Process indexed statuses" +}`) +req, _ := http.NewRequest("POST", "http://127.0.0.1:3000/admin/scripts", body) +req.Header.Set("Authorization", "Bearer "+token) +req.Header.Set("Content-Type", "application/json") +resp, err := http.DefaultClient.Do(req) +``` +```sh tab="cURL" tab-group="language" +curl -X POST http://127.0.0.1:3000/admin/scripts \ + -H "$AUTH" \ + -H "Content-Type: application/json" \ + -d '{ + "id": "record.index:xyz.statusphere.status", + "script_type": "lua", + "body": "function handle()\n return event\nend", + "description": "Process indexed statuses" + }' +``` + +| Field | Type | Required | Description | +| ------------- | ------ | -------- | -------------------------------------------------------------- | +| `id` | string | yes | Trigger string (e.g. `record.index:xyz.statusphere.status`) | +| `script_type` | string | no | Script language; defaults to `"lua"` | +| `body` | string | yes | The script source code | +| `description` | string | no | Human-readable description (max 300 characters) | + +**Response**: `201 Created` (new) or `200 OK` (update) + +```json +{ + "id": "record.index:xyz.statusphere.status", + "script_type": "lua", + "body": "function handle()\n return event\nend", + "description": "Process indexed statuses", + "created_at": "2026-01-01T00:00:00Z", + "updated_at": "2026-01-01T00:00:00Z" +} +``` + +## Partial update a script + +``` +PATCH /admin/scripts/{id} +``` + +Updates individual fields of an existing script. At least one field must be provided. Setting `description` to `null` in JSON clears it. If `script_type` is changed, `body` must also be provided so validation can run against the new type. + +```ts tab="TypeScript" tab-group="language" +const response = await fetch( + "http://127.0.0.1:3000/admin/scripts/record.index%3Axyz.statusphere.status", + { + method: "PATCH", + headers: { + ...headers, + "Content-Type": "application/json", + }, + body: JSON.stringify({ + description: "Updated description for status processing", + }), + }, +); +const data: Script = await response.json(); +``` +```js tab="JavaScript" tab-group="language" +const response = await fetch( + "http://127.0.0.1:3000/admin/scripts/record.index%3Axyz.statusphere.status", + { + method: "PATCH", + headers: { + ...headers, + "Content-Type": "application/json", + }, + body: JSON.stringify({ + description: "Updated description for status processing", + }), + }, +); +const data = await response.json(); +``` +```rust tab="Rust" tab-group="language" +let response = client + .patch("http://127.0.0.1:3000/admin/scripts/record.index%3Axyz.statusphere.status") + .bearer_auth(token) + .json(&serde_json::json!({ + "description": "Updated description for status processing" + })) + .send() + .await?; +let data: serde_json::Value = response.json().await?; +``` +```go tab="Go" tab-group="language" +body := bytes.NewBufferString(`{ + "description": "Updated description for status processing" +}`) +req, _ := http.NewRequest("PATCH", "http://127.0.0.1:3000/admin/scripts/record.index%3Axyz.statusphere.status", body) +req.Header.Set("Authorization", "Bearer "+token) +req.Header.Set("Content-Type", "application/json") +resp, err := http.DefaultClient.Do(req) +``` +```sh tab="cURL" tab-group="language" +curl -X PATCH "http://127.0.0.1:3000/admin/scripts/record.index%3Axyz.statusphere.status" \ + -H "$AUTH" \ + -H "Content-Type: application/json" \ + -d '{ "description": "Updated description for status processing" }' +``` + +| Field | Type | Required | Description | +| ------------- | ------------ | -------- | ---------------------------------------------------------------- | +| `script_type` | string | no | Script language; requires `body` alongside | +| `body` | string | no | New script source; re-validated against `script_type` | +| `description` | string\|null | no | New description, or `null` to clear | + +**Response**: `200 OK` + +```json +{ + "id": "record.index:xyz.statusphere.status", + "script_type": "lua", + "body": "function handle()\n return event\nend", + "description": "Updated description for status processing", + "created_at": "2026-01-01T00:00:00Z", + "updated_at": "2026-01-01T00:00:00Z" +} +``` + +## Delete a script + +``` +DELETE /admin/scripts/{id} +``` + +```ts tab="TypeScript" tab-group="language" +const response = await fetch( + "http://127.0.0.1:3000/admin/scripts/record.index%3Axyz.statusphere.status", + { + method: "DELETE", + headers, + }, +); +``` +```js tab="JavaScript" tab-group="language" +const response = await fetch( + "http://127.0.0.1:3000/admin/scripts/record.index%3Axyz.statusphere.status", + { + method: "DELETE", + headers, + }, +); +``` +```rust tab="Rust" tab-group="language" +let response = client + .delete("http://127.0.0.1:3000/admin/scripts/record.index%3Axyz.statusphere.status") + .bearer_auth(token) + .send() + .await?; +``` +```go tab="Go" tab-group="language" +req, _ := http.NewRequest("DELETE", "http://127.0.0.1:3000/admin/scripts/record.index%3Axyz.statusphere.status", nil) +req.Header.Set("Authorization", "Bearer "+token) +resp, err := http.DefaultClient.Do(req) +``` +```sh tab="cURL" tab-group="language" +curl -X DELETE "http://127.0.0.1:3000/admin/scripts/record.index%3Axyz.statusphere.status" \ + -H "$AUTH" +``` + +**Response**: `204 No Content` diff --git a/packages/docs/content/docs/api-reference/lua/atproto-api.md b/packages/docs/content/docs/api-reference/lua/atproto-api.md index 379be5f..8146401 100644 --- a/packages/docs/content/docs/api-reference/lua/atproto-api.md +++ b/packages/docs/content/docs/api-reference/lua/atproto-api.md @@ -2,7 +2,7 @@ title: "atproto API" --- -The `atproto` table provides atproto utility functions. Available in queries, procedures, and [index hooks](../../guides/index-hooks.md). +The `atproto` table provides atproto utility functions. Available in all [Lua scripts](../../guides/lua-scripting.md) — queries, procedures, and [record/label scripts](../../guides/label-scripts). ## atproto.resolve_service_endpoint diff --git a/packages/docs/content/docs/api-reference/lua/database-api.md b/packages/docs/content/docs/api-reference/lua/database-api.md index 96c78b3..a8ebbcf 100644 --- a/packages/docs/content/docs/api-reference/lua/database-api.md +++ b/packages/docs/content/docs/api-reference/lua/database-api.md @@ -2,7 +2,7 @@ title: "Database API" --- -The `db` table provides access to the database. Available in queries, procedures, and [index hooks](../../guides/index-hooks.md). +The `db` table provides access to the database. Available in all [Lua scripts](../../guides/lua-scripting.md) — queries, procedures, and [record/label scripts](../../guides/label-scripts). ## db.query diff --git a/packages/docs/content/docs/api-reference/lua/http-api.md b/packages/docs/content/docs/api-reference/lua/http-api.md index 6b65477..16cd635 100644 --- a/packages/docs/content/docs/api-reference/lua/http-api.md +++ b/packages/docs/content/docs/api-reference/lua/http-api.md @@ -2,7 +2,7 @@ title: "HTTP API" --- -The `http` table provides async HTTP client functions. Available in queries, procedures, and [index hooks](../../guides/index-hooks.md). +The `http` table provides async HTTP client functions. Available in all [Lua scripts](../../guides/lua-scripting.md) — queries, procedures, and [record/label scripts](../../guides/label-scripts). ## Methods @@ -21,10 +21,10 @@ http.head(url, opts?) The optional second argument is a table with: -| Field | Type | Description | -| --------- | ------ | ---------------------------------------------- | -| `headers` | table | Request headers as key-value string pairs | -| `body` | string | Request body (ignored for GET and HEAD) | +| Field | Type | Description | +| --------- | ------ | ----------------------------------------- | +| `headers` | table | Request headers as key-value string pairs | +| `body` | string | Request body (ignored for GET and HEAD) | ## Response diff --git a/packages/docs/content/docs/api-reference/lua/json-api.md b/packages/docs/content/docs/api-reference/lua/json-api.md index 794592f..b7b8141 100644 --- a/packages/docs/content/docs/api-reference/lua/json-api.md +++ b/packages/docs/content/docs/api-reference/lua/json-api.md @@ -2,7 +2,7 @@ title: "JSON API" --- -The `json` global provides JSON serialization and deserialization. Available in queries, procedures, and [index hooks](../../guides/index-hooks.md). +The `json` global provides JSON serialization and deserialization. Available in all [Lua scripts](../../guides/lua-scripting.md) — queries, procedures, and [record/label scripts](../../guides/label-scripts). ## json.encode diff --git a/packages/docs/content/docs/api-reference/lua/utility-globals.md b/packages/docs/content/docs/api-reference/lua/utility-globals.md index d9fbde3..42e796b 100644 --- a/packages/docs/content/docs/api-reference/lua/utility-globals.md +++ b/packages/docs/content/docs/api-reference/lua/utility-globals.md @@ -2,7 +2,7 @@ title: "Utility Globals" --- -Global functions available in queries, procedures, and [index hooks](../../guides/index-hooks.md). These don't belong to a specific API table — they're available at the top level of any Lua script. +Global functions available in all [Lua scripts](../../guides/lua-scripting.md) — queries, procedures, and [record/label scripts](../../guides/label-scripts). These don't belong to a specific API table — they're available at the top level of any script. ## now diff --git a/packages/docs/content/docs/api-reference/lua/xrpc-lua-api.md b/packages/docs/content/docs/api-reference/lua/xrpc-lua-api.md index 3db18b7..a5bdbf2 100644 --- a/packages/docs/content/docs/api-reference/lua/xrpc-lua-api.md +++ b/packages/docs/content/docs/api-reference/lua/xrpc-lua-api.md @@ -2,7 +2,7 @@ title: "XRPC Lua API" --- -The `xrpc` table provides cross-endpoint XRPC calls. Available in queries, procedures, and [index hooks](../../guides/index-hooks.md). +The `xrpc` table provides cross-endpoint XRPC calls. Available in all [Lua scripts](../../guides/lua-scripting.md) — queries, procedures, and [record/label scripts](../../guides/label-scripts). ## xrpc.query diff --git a/packages/docs/content/docs/guides/api-keys.md b/packages/docs/content/docs/guides/api-keys.md index 326eb34..4aa84cc 100644 --- a/packages/docs/content/docs/guides/api-keys.md +++ b/packages/docs/content/docs/guides/api-keys.md @@ -45,6 +45,7 @@ interface LexiconsResponse { const data: LexiconsResponse = await response.json(); ``` + ```js tab="JavaScript" tab-group="language" const TOKEN = "hv_a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4"; @@ -54,6 +55,7 @@ const response = await fetch("http://127.0.0.1:3000/admin/lexicons", { const data = await response.json(); ``` + ```rust tab="Rust" tab-group="language" let client = reqwest::Client::new(); let token = "hv_a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4"; @@ -66,6 +68,7 @@ let response = client let data: serde_json::Value = response.json().await?; ``` + ```go tab="Go" tab-group="language" token := "hv_a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4" @@ -74,6 +77,7 @@ req.Header.Set("Authorization", "Bearer "+token) resp, err := http.DefaultClient.Do(req) ``` + ```sh tab="cURL" tab-group="language" curl http://127.0.0.1:3000/admin/lexicons \ -H "Authorization: Bearer hv_a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4" @@ -105,4 +109,4 @@ The **Last Used** column in the API Keys table shows when each key was last used - [Admin API reference](../api-reference/admin/admin-api.md) — full endpoint documentation - [Scripting](./lua-scripting.md) — automate record processing with Lua scripts -- [Index hooks](./index-hooks.md) — push records to external services on write +- [Record & label scripts](./label-scripts) — react to record changes and label events diff --git a/packages/docs/content/docs/guides/attestation-signing.md b/packages/docs/content/docs/guides/attestation-signing.md index 16c81f8..957ec54 100644 --- a/packages/docs/content/docs/guides/attestation-signing.md +++ b/packages/docs/content/docs/guides/attestation-signing.md @@ -39,7 +39,7 @@ If key loading fails for any reason, signing is disabled and `atproto.sign` / `a ## Using in Lua scripts -Available in queries, procedures, and index hooks via the [atproto API](../api-reference/lua/atproto-api.md). +Available in queries, procedures, and record/label scripts via the [atproto API](../api-reference/lua/atproto-api.md). ### Signing a record diff --git a/packages/docs/content/docs/guides/event-logs.md b/packages/docs/content/docs/guides/event-logs.md index ba1265c..38aafec 100644 --- a/packages/docs/content/docs/guides/event-logs.md +++ b/packages/docs/content/docs/guides/event-logs.md @@ -10,29 +10,29 @@ Events follow a `category.action` naming convention. Each event has a severity l ### Lexicon events -| Event Type | Severity | Subject | Detail | -|---|---|---|---| -| `lexicon.created` | info | Lexicon NSID | `revision`, `has_script`, `source` | -| `lexicon.updated` | info | Lexicon NSID | `revision`, `has_script`, `source` | -| `lexicon.deleted` | info | Lexicon NSID | — | +| Event Type | Severity | Subject | Detail | +| ----------------- | -------- | ------------ | ---------------------------------- | +| `lexicon.created` | info | Lexicon NSID | `revision`, `has_script`, `source` | +| `lexicon.updated` | info | Lexicon NSID | `revision`, `has_script`, `source` | +| `lexicon.deleted` | info | Lexicon NSID | — | Logged when lexicons are uploaded, updated, or deleted via the [admin API](../api-reference/admin/lexicons.md). The `actor_did` is the user who performed the action. ### Record events -| Event Type | Severity | Subject | Detail | -|---|---|---|---| -| `record.created` | info | Record AT URI | `collection`, `did`, `rkey` | -| `record.deleted` | info | Record AT URI | `collection`, `did`, `rkey` | +| Event Type | Severity | Subject | Detail | +| ---------------- | -------- | ------------- | --------------------------- | +| `record.created` | info | Record AT URI | `collection`, `did`, `rkey` | +| `record.deleted` | info | Record AT URI | `collection`, `did`, `rkey` | Logged when records are received from Jetstream and stored or removed from the local database. These are system-triggered events (`actor_did` is null). If a database error occurs during the operation, the same event type is logged with `error` severity and the error message is included in the detail. ### Script events -| Event Type | Severity | Subject | Detail | -|---|---|---|---| -| `script.executed` | info | Method NSID | `method`, `caller_did`, `duration_ms` | -| `script.error` | error | Method NSID | `error`, `script_source`, `input`, `caller_did`, `method` | +| Event Type | Severity | Subject | Detail | +| ----------------- | -------- | ----------- | --------------------------------------------------------- | +| `script.executed` | info | Method NSID | `method`, `caller_did`, `duration_ms` | +| `script.error` | error | Method NSID | `error`, `script_source`, `input`, `caller_did`, `method` | Logged when Lua scripts run for XRPC query or procedure endpoints. Script errors capture the full context needed to reproduce and debug the issue: the error message, the complete Lua script source, the input that triggered it, and the caller's DID. @@ -42,63 +42,63 @@ For query scripts (unauthenticated), `caller_did` and `input` are omitted from t ### User events -| Event Type | Severity | Subject | Detail | -|---|---|---|---| -| `user.created` | info | New user DID | `template` (if used) | -| `user.deleted` | info | Removed user ID | — | -| `user.bootstrapped` | info | Bootstrapped user DID | — | -| `user.permissions_updated` | info | User ID | `granted`, `revoked` | -| `user.super_transferred` | warn | New super user ID | `from_user_id` | +| Event Type | Severity | Subject | Detail | +| -------------------------- | -------- | --------------------- | -------------------- | +| `user.created` | info | New user DID | `template` (if used) | +| `user.deleted` | info | Removed user ID | — | +| `user.bootstrapped` | info | Bootstrapped user DID | — | +| `user.permissions_updated` | info | User ID | `granted`, `revoked` | +| `user.super_transferred` | warn | New super user ID | `from_user_id` | The `user.bootstrapped` event is logged when the first user is auto-promoted to super user (see [Auth - Auto-bootstrap](../api-reference/admin/admin-api.md#auth)). ### Auth events -| Event Type | Severity | Subject | Detail | -|---|---|---|---| -| `auth.permission_denied` | error | Endpoint path | `required_permission`, `user_id` | +| Event Type | Severity | Subject | Detail | +| ------------------------ | -------- | ------------- | -------------------------------- | +| `auth.permission_denied` | error | Endpoint path | `required_permission`, `user_id` | Logged when a user attempts to access an endpoint they don't have permission for. ### API Key events -| Event Type | Severity | Subject | Detail | -|---|---|---|---| -| `api_key.created` | info | Key ID | `name`, `permissions` | -| `api_key.revoked` | info | Key ID | `name` | +| Event Type | Severity | Subject | Detail | +| ----------------- | -------- | ------- | --------------------- | +| `api_key.created` | info | Key ID | `name`, `permissions` | +| `api_key.revoked` | info | Key ID | `name` | ### Script Variable events -| Event Type | Severity | Subject | Detail | -|---|---|---|---| -| `script_variable.upserted` | info | Variable key | — | -| `script_variable.deleted` | info | Variable key | — | +| Event Type | Severity | Subject | Detail | +| -------------------------- | -------- | ------------ | ------ | +| `script_variable.upserted` | info | Variable key | — | +| `script_variable.deleted` | info | Variable key | — | ### Hook events -| Event Type | Severity | Subject | Detail | -|---|---|---|---| -| `hook.executed` | info | Record AT URI | `lexicon_id` | -| `hook.dead_lettered` | error | Record AT URI | `lexicon_id`, `error` | +| Event Type | Severity | Subject | Detail | +| ---------------------- | -------- | ---------- | --------------------- | +| `script.executed` | info | Trigger ID | `trigger_id` | +| `script.dead_lettered` | error | Trigger ID | `trigger_id`, `error` | -Logged when [index hooks](./index-hooks.md) run. Dead-lettered events indicate a hook failed all retry attempts. You can manage dead letters from the **Data > Dead Letters** page in the dashboard — see [Dead Letters](#dead-letters) below. +Logged when [record/label scripts](./label-scripts) run. Dead-lettered events indicate a script failed all retry attempts. You can manage dead letters from the **Data > Dead Letters** page in the dashboard — see [Dead Letters](#dead-letters) below. ### Backfill events -| Event Type | Severity | Subject | Detail | -|---|---|---|---| -| `backfill.started` | info | Collection NSID | `job_id` | -| `backfill.completed` | info | Collection NSID | `job_id`, `total_repos` | -| `backfill.failed` | error | Collection NSID | `job_id`, `error` | +| Event Type | Severity | Subject | Detail | +| -------------------- | -------- | --------------- | ----------------------- | +| `backfill.started` | info | Collection NSID | `job_id` | +| `backfill.completed` | info | Collection NSID | `job_id`, `total_repos` | +| `backfill.failed` | error | Collection NSID | `job_id`, `error` | See [Backfill](./backfill.md) for background on backfill jobs. ### Jetstream events -| Event Type | Severity | Subject | Detail | -|---|---|---|---| -| `jetstream.connected` | info | — | `url` | -| `jetstream.disconnected` | warn | — | `reason` | +| Event Type | Severity | Subject | Detail | +| ------------------------ | -------- | ------- | -------- | +| `jetstream.connected` | info | — | `url` | +| `jetstream.disconnected` | warn | — | `reason` | Logged when the WebSocket connection to [Jetstream](https://github.com/bluesky-social/jetstream) is established or lost. @@ -149,6 +149,7 @@ const page: EventsResponse = await fetch( { headers }, ).then((r) => r.json()); ``` + ```js tab="JavaScript" tab-group="language" const TOKEN = "hv_..."; // your API key const headers = { Authorization: `Bearer ${TOKEN}` }; @@ -177,6 +178,7 @@ const page = await fetch( { headers }, ).then((r) => r.json()); ``` + ```rust tab="Rust" tab-group="language" let client = reqwest::Client::new(); let token = "hv_..."; // your API key @@ -224,6 +226,7 @@ let page: serde_json::Value = client .json() .await?; ``` + ```go tab="Go" tab-group="language" token := "hv_..." // your API key @@ -251,6 +254,7 @@ req, _ = http.NewRequest("GET", req.Header.Set("Authorization", "Bearer "+token) page, err := http.DefaultClient.Do(req) ``` + ```sh tab="cURL" tab-group="language" AUTH="Authorization: Bearer hv_..." # your API key @@ -279,11 +283,11 @@ See [Configuration](../getting-started/configuration.md) for all environment var ## Dead Letters -When an index hook fails after all retry attempts, the event is stored in the dead letters queue. You can manage dead letters from the **Data > Dead Letters** page in the dashboard. +When a record or label script fails after all retry attempts, the event is stored in the dead letters queue. You can manage dead letters from the **Data > Dead Letters** page in the dashboard. From the dead letters page you can: -- **Retry Hook** — replay the stored record through the index hook (use after fixing a hook script) +- **Retry Script** — replay the stored event through the script (use after fixing the script) - **Re-index** — fetch the record fresh from the PDS and run it through the full indexing pipeline (use when the record may have changed) - **Dismiss** — mark the dead letter as resolved without retrying diff --git a/packages/docs/content/docs/guides/index-hooks.md b/packages/docs/content/docs/guides/index-hooks.md deleted file mode 100644 index 1f20b68..0000000 --- a/packages/docs/content/docs/guides/index-hooks.md +++ /dev/null @@ -1,227 +0,0 @@ ---- -title: "Index Hooks" ---- - -Index hooks are Lua scripts that run whenever a record in a collection is created, updated, or deleted. They run **before** the record is indexed, giving you the ability to filter out unwanted records, transform record data before storage, or trigger side effects like syncing with external services. - -Index hooks fire on **all** record events for the collection — including records created by HappyView procedure endpoints, not just events from the network. Unlike [query and procedure scripts](./lua-scripting.md) that run in response to XRPC requests, index hooks are triggered by incoming Jetstream events (which include events caused by HappyView's own PDS writes). - -## Attaching a hook - -Each record-type lexicon can have one index hook. You can add it through the [dashboard](../getting-started/dashboard.md) (click "Add Index Hook" on any record lexicon's detail page) or via the [admin API](../api-reference/admin/lexicons.md#upload--upsert-a-lexicon) by including the `index_hook` field when uploading a lexicon. - -## Script structure - -Like query and procedure scripts, index hooks must define a `handle()` function: - -```lua -function handle() - if action == "delete" then - log("deleted " .. uri) - else - log(action .. " " .. uri) - end - return true -end -``` - -The function is called once per record event. The return value controls what happens next: - -| Return value | Effect | -| ------------ | ----------------------------------------------------------- | -| `nil` | The record is **not** indexed (skipped entirely) | -| A table | That table is stored as the record instead | -| `true` | The original record is stored as-is | -| *(no hook)* | The original record is stored as-is | - -On **delete** events, returning `nil` skips the delete (the record stays in the database). - -**Important:** If your hook has side effects (e.g. syncing to a search index) but you want normal indexing to proceed, return `record` or `true` — not nothing. A missing return statement returns `nil`, which **skips indexing**. - -If the hook errors after all retries, the system **fails open** — the original record is stored and the failed event is dead-lettered for later inspection. - -## Context globals - -These globals are set before `handle()` is called: - -| Global | Type | Description | -| ------------ | ------ | -------------------------------------------------- | -| `action` | string | `"create"`, `"update"`, or `"delete"` | -| `uri` | string | The full AT URI (e.g. `at://did:plc:abc/col/rkey`) | -| `did` | string | The repo DID | -| `collection` | string | The collection NSID | -| `rkey` | string | The record key | -| `record` | table? | The full record as a Lua table (nil on delete) | - -Index hooks do **not** have access to `caller_did`, `input`, `params`, `method`, or the `Record` API. They run from the Jetstream event stream, not from a user request. - -## Available APIs - -Index hooks have access to: - -- **[Database API](../api-reference/lua/database-api.md)** — `db.query`, `db.get`, `db.search`, `db.backlinks`, `db.count`, `db.raw` -- **[HTTP API](../api-reference/lua/http-api.md)** — `http.get`, `http.post`, `http.put`, `http.patch`, `http.delete`, `http.head` -- **[XRPC Lua API](../api-reference/lua/xrpc-lua-api.md)** — `xrpc.query`, `xrpc.procedure` -- **[atproto API](../api-reference/lua/atproto-api.md)** — `atproto.resolve_service_endpoint`, `atproto.get_labels`, `atproto.get_labels_batch` -- **[JSON API](../api-reference/lua/json-api.md)** — `json.encode`, `json.decode` -- **[Utility globals](./lua-scripting.md#utility-globals)** — `log()`, `now()`, `TID()`, `toarray()` -- **[Script variables](../api-reference/admin/script-variables.md)** — `env` table with key-value pairs configured in the dashboard - -## Error handling and retries - -Index hooks are designed to be resilient: - -1. If a hook fails, it retries up to **3 times** with exponential backoff (1s, 2s, 4s delays). -2. If all retries are exhausted, the failed event is inserted into the `dead_letter_hooks` table for later inspection. -3. On failure the system **fails open** — the original record is stored as-is so indexing is not permanently blocked. - -Failed hooks are logged as errors. Check the [event logs](./event-logs.md) or query the `dead_letter_hooks` table directly to find and replay failures. - -### Performance considerations - -Because hooks run synchronously before indexing, they block the Jetstream consumer while executing. With retry logic (1s + 2s + 4s backoff), a persistently failing hook could block for ~7 seconds per record. Keep hook scripts fast and ensure external services they depend on are reliable. - -### Dead letter table - -The `dead_letter_hooks` table stores events that failed all retry attempts: - -| Column | Type | Description | -| ------------ | ----------- | --------------------------------------- | -| `id` | UUID | Primary key | -| `lexicon_id` | text | The lexicon NSID | -| `uri` | text | The AT URI of the record | -| `did` | text | The repo DID | -| `collection` | text | The collection NSID | -| `rkey` | text | The record key | -| `action` | text | `create`, `update`, or `delete` | -| `record` | jsonb | The record data (null on delete) | -| `error` | text | The error message from the last attempt | -| `attempts` | int | Total number of attempts made | -| `created_at` | timestamptz | When the failure was recorded | - -## Examples - -### Filter out records missing a required field - -Skip indexing any record that doesn't have a `title` field: - -```lua -function handle() - if action == "delete" then - return record -- allow deletes to proceed - end - - if record.title == nil or record.title == "" then - return nil -- skip: no title - end - - return record -end -``` - -### Transform a record before storage - -Enrich a record with a computed field before it is stored: - -```lua -function handle() - if action == "delete" then - return record - end - - record.slug = string.lower(string.gsub(record.title or "", "%s+", "-")) - return record -end -``` - -### Post to a webhook - -```lua -function handle() - http.post("https://hooks.example.com/records", { - headers = { ["Content-Type"] = "application/json" }, - body = json.encode({ - action = action, - uri = uri, - did = did, - record = record - }) - }) - return record -end -``` - -### Sync to Algolia - -Push records to an Algolia search index on create/update, and remove them on delete: - -```lua -function handle() - local headers = { - ["X-Algolia-API-Key"] = "your-api-key", - ["X-Algolia-Application-Id"] = "your-app-id", - ["Content-Type"] = "application/json" - } - - if action == "delete" then - http.delete("https://YOUR-APP.algolia.net/1/indexes/records/" .. uri, { - headers = headers - }) - else - http.put("https://YOUR-APP.algolia.net/1/indexes/records/" .. uri, { - headers = headers, - body = json.encode({ - objectID = uri, - collection = collection, - did = did, - record = record - }) - }) - end - - return record -end -``` - -See the full [Algolia sync reference](../reference/script-examples/algolia-sync.md) for more detail. - -### Sync to Meilisearch - -Push records to a self-hosted Meilisearch index on create/update, and remove them on delete: - -```lua -function handle() - local headers = { - ["Authorization"] = "Bearer " .. env.MEILISEARCH_API_KEY, - ["Content-Type"] = "application/json" - } - - if action == "delete" then - http.delete(env.MEILISEARCH_URL .. "/indexes/records/documents/" .. uri, { - headers = headers - }) - else - http.post(env.MEILISEARCH_URL .. "/indexes/records/documents", { - headers = headers, - body = json.encode(toarray({ - { - id = uri, - collection = collection, - did = did, - record = record - } - })) - }) - end - - return record -end -``` - -See the full [Meilisearch sync reference](../reference/script-examples/meilisearch-sync.md) for more detail. - -## Next steps - -- [Lua Scripting](./lua-scripting.md): Full reference for the sandbox, APIs, and debugging -- [Lexicons](lexicons.md): Understand how record, query, and procedure lexicons work together -- [Admin API — Lexicons](../api-reference/admin/lexicons.md#upload--upsert-a-lexicon): Upload lexicons with index hooks via the API diff --git a/packages/docs/content/docs/guides/lexicons.md b/packages/docs/content/docs/guides/lexicons.md index 0e10ab7..81d3e9a 100644 --- a/packages/docs/content/docs/guides/lexicons.md +++ b/packages/docs/content/docs/guides/lexicons.md @@ -8,12 +8,12 @@ You don't write route handlers or database queries; you upload a lexicon and Hap ## Supported lexicon types -| Type | Effect | -| ------------- | ------------------------------------------------------------------------------ | -| `record` | Adds the collection to the Jetstream subscription filter and indexes records into the database. Supports [index hooks](index-hooks.md) | -| `query` | Registers a `GET /xrpc/{nsid}` endpoint that queries indexed records | -| `procedure` | Registers a `POST /xrpc/{nsid}` endpoint that proxies writes to the user's PDS | -| `definitions` | Stored but does not generate routes or subscriptions | +| Type | Effect | +| ------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | +| `record` | Adds the collection to the Jetstream subscription filter and indexes records into the database. Supports [record scripts](label-scripts) | +| `query` | Registers a `GET /xrpc/{nsid}` endpoint that queries indexed records | +| `procedure` | Registers a `POST /xrpc/{nsid}` endpoint that proxies writes to the user's PDS | +| `definitions` | Stored but does not generate routes or subscriptions | A typical setup has three lexicons working together: a **record** lexicon that defines the data and triggers indexing, a **query** lexicon that exposes a read endpoint, and a **procedure** lexicon that exposes a write endpoint. The [Statusphere tutorial](../tutorials/statusphere.md) walks through this pattern end-to-end. @@ -100,7 +100,7 @@ In short: if you want to serve an XRPC method on your instance, you need a local ## Next steps - [Lua Scripting](./lua-scripting.md): Add custom query and procedure logic to your endpoints -- [Index Hooks](index-hooks.md): Run Lua scripts when records are indexed from the network +- [Record & Label Scripts](label-scripts): Run Lua scripts when records are indexed or labels arrive - [XRPC API](../api-reference/xrpc-api.md): Understand how the generated endpoints behave - [Backfill](backfill.md): Learn how historical records are indexed - [Admin API](../api-reference/admin/admin-api.md): Full reference for lexicon management endpoints diff --git a/packages/docs/content/docs/guides/lua-scripting.md b/packages/docs/content/docs/guides/lua-scripting.md index 3b75c75..f9d5c53 100644 --- a/packages/docs/content/docs/guides/lua-scripting.md +++ b/packages/docs/content/docs/guides/lua-scripting.md @@ -10,9 +10,9 @@ Without Lua scripts, HappyView's query endpoints return raw records and procedur - Compose multi-record operations - Build entirely custom behavior -Scripts are attached to query and procedure lexicons and run in a sandboxed Lua VM with access to the [Record API](#record-api), a [database API](#database-api), an [HTTP client API](#http-api), a [JSON API](#json-api), and a set of [context globals](#context-globals). +Scripts run in a sandboxed Lua VM with access to the [Record API](#record-api), a [database API](#database-api), an [HTTP client API](#http-api), a [JSON API](#json-api), and a set of [context globals](#context-globals). -For scripts that react to record changes from the network (rather than XRPC requests), see [Index Hooks](index-hooks.md). +For scripts that react to record changes or label events (rather than XRPC requests), see [Record & Label Scripts](label-scripts). ## Script structure @@ -57,24 +57,24 @@ These globals are set automatically before `handle()` is called. ### Query globals -| Global | Type | Description | -| ------------ | ------- | ------------------------------------------------ | -| `method` | string | The XRPC method name | -| `params` | table | Query string parameters (all values are strings) | -| `collection` | string | Target collection NSID | +| Global | Type | Description | +| ------------ | ------- | ------------------------------------------------------ | +| `method` | string | The XRPC method name | +| `params` | table | Query string parameters (all values are strings) | +| `collection` | string | Target collection NSID | | `caller_did` | string? | DID of the authenticated user (nil if unauthenticated) | -| `env` | table | Script variables configured in the dashboard | +| `env` | table | Script variables configured in the dashboard | ## Utility globals Available in both queries and procedures: -| Function | Returns | Description | -| ---------------- | ------- | ------------------------------------------------------------------- | -| `now()` | string | Current UTC timestamp in ISO 8601 format | -| `log(message)` | — | Log a message (appears in server logs at debug level) | +| Function | Returns | Description | +| ---------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `now()` | string | Current UTC timestamp in ISO 8601 format | +| `log(message)` | — | Log a message (appears in server logs at debug level) | | `TID()` | string | Generate a fresh atproto TID (13-character sortable identifier). Also provides conversion methods — see [Utility Globals reference](../api-reference/lua/utility-globals.md#tid). | -| `toarray(table)` | table | Mark a table as a JSON array for serialization (see [below](#toarray)) | +| `toarray(table)` | table | Mark a table as a JSON array for serialization (see [below](#toarray)) | ### toarray @@ -195,6 +195,7 @@ The **full error message** is logged server-side at error level. Check the serve See the example script references for complete, ready-to-use scripts: **Queries:** + - [Get a record](../reference/script-examples/get-record.md) — fetch a single record by AT URI - [Paginated list](../reference/script-examples/paginated-list.md) — list records with cursor-based pagination and DID filtering - [List or fetch](../reference/script-examples/list-or-fetch.md) — combined single-record lookup and paginated listing @@ -202,6 +203,7 @@ See the example script references for complete, ready-to-use scripts: - [Verify signed record](../reference/script-examples/signed-record-verify.md) — fetch a record and verify its attestation signature **Procedures:** + - [Create a record](../reference/script-examples/create-record.md) — simple write that saves input as a record - [Upsert a record](../reference/script-examples/upsert-record.md) — create or update using a deterministic rkey - [Update or delete](../reference/script-examples/update-or-delete.md) — single endpoint handling create, update, and delete @@ -211,12 +213,14 @@ See the example script references for complete, ready-to-use scripts: - [Complex mutations](../reference/script-examples/complex-mutations.md) — load, transform, and save a record with multiple field changes - [Signed record](../reference/script-examples/signed-record.md) — save a record with an attestation signature -**Index Hooks:** +**Record & Label Scripts:** + - [Algolia sync](../reference/script-examples/algolia-sync.md) — push records to an Algolia search index on create/update/delete ## Next steps -- [Index Hooks](index-hooks.md): React to record changes from the network in real time +- [Record & Label Scripts](label-scripts): React to record changes and label events in real time - [Lexicons](lexicons.md): Understand how record, query, and procedure lexicons work together +- [Admin API — Scripts](../api-reference/admin/scripts.md): Manage scripts via the API - [XRPC API](../api-reference/xrpc-api.md): See how endpoints behave with and without Lua scripts - [Dashboard](../getting-started/dashboard.md#lua-editor): Use the web editor with context-aware completions diff --git a/packages/docs/content/docs/guides/meta.json b/packages/docs/content/docs/guides/meta.json index 0cf3422..7bc4499 100644 --- a/packages/docs/content/docs/guides/meta.json +++ b/packages/docs/content/docs/guides/meta.json @@ -4,7 +4,7 @@ "upgrading-to-v2", "lexicons", "backfill", - "index-hooks", + "label-scripts", "lua-scripting", "api-clients", "attestation-signing", diff --git a/packages/docs/content/docs/guides/record-scripts.md b/packages/docs/content/docs/guides/record-scripts.md new file mode 100644 index 0000000..b4af710 --- /dev/null +++ b/packages/docs/content/docs/guides/record-scripts.md @@ -0,0 +1,311 @@ +--- +title: "Record & Label Scripts" +--- + +Record and label scripts are Lua scripts that run in response to events on the AT Protocol network. **Record scripts** fire when a record in a collection is created, updated, or deleted. **Label scripts** fire when a label is applied to a record or actor. Both run **before** the event is indexed, giving you the ability to filter, transform, or trigger side effects. + +These scripts are event-driven -- they react to incoming Jetstream events (which include events caused by HappyView's own PDS writes), not to XRPC requests. For scripts that run in response to XRPC queries and procedures, see [Lua Scripting](./lua-scripting.md). + +> **Migration note:** Prior to v2.9, record scripts were called "index hooks" and were attached directly to lexicons. They now live in their own `scripts` table and are managed separately. Existing index hooks were automatically migrated. + +## Trigger grammar + +Every script is identified by a **trigger string** -- the script's `id` in the `scripts` table IS its trigger binding. There is no separate name or host column; the trigger string determines which events the script receives. + +### Record event triggers + +| Trigger | Fires when | +| -------------------------- | --------------------------------------------- | +| `record.index:` | Any record event (create, update, or delete) | +| `record.create:` | A record is created | +| `record.update:` | A record is updated | +| `record.delete:` | A record is deleted | + +**Cascade rule:** When a record event occurs, the dispatcher tries the action-specific trigger first (e.g. `record.create:`), then falls back to `record.index:` if no action-specific script exists. This means you can use `record.index` as a catch-all and override individual actions when needed. + +### Label event triggers + +| Trigger | Fires when | +| -------------------------- | -------------------------------------------------- | +| `labeler.apply:` | A label arrives whose subject is `at:////` | +| `labeler.apply:_actor` | A label arrives whose subject is a bare DID (actor-level label) | + +There is no cascade for label triggers -- each trigger string must match exactly. + +## Creating scripts + +You can create scripts through the [dashboard](../getting-started/dashboard.md) (Settings > Scripts > New) or via the [admin API](../api-reference/admin/scripts.md) (`POST /admin/scripts`). + +When creating a script, you provide the trigger string as the script's `id`. For example, a script with id `record.index:xyz.statusphere.status` will fire on every record event for the `xyz.statusphere.status` collection. + +## Script structure + +Like query and procedure scripts, record and label scripts must define a `handle()` function: + +```lua +function handle() + if action == "delete" then + log("deleted " .. uri) + else + log(action .. " " .. uri) + end + return true +end +``` + +The function is called once per event. + +### Record script return values + +| Return value | Effect | +| ------------ | ----------------------------------------------------------- | +| `nil` | The record is **not** indexed (skipped entirely) | +| A table | That table is stored as the record instead | +| `true` | The original record is stored as-is | +| *(no script)* | The original record is stored as-is | + +On **delete** events, returning `nil` skips the delete (the record stays in the database). + +**Important:** If your script has side effects (e.g. syncing to a search index) but you want normal indexing to proceed, return `record` or `true` -- not nothing. A missing return statement returns `nil`, which **skips indexing**. + +### Label script return values + +| Return value | Effect | +| ------------ | ------------------------------------------------------------ | +| `nil` | The label is **not** persisted (skipped entirely) | +| A table | The returned fields are merged with the original label | +| `true` | The original label is stored as-is | +| *(no script)* | The original label is stored as-is | + +When a label script returns a table, any field the script omits falls back to the original value. This means `return event` passes the label through unchanged, while `return { val = "new-value" }` rewrites only the `val` field. + +## Context globals + +### Record script globals + +These globals are set before `handle()` is called for record events: + +| Global | Type | Description | +| ------------ | ------ | -------------------------------------------------- | +| `action` | string | `"create"`, `"update"`, or `"delete"` | +| `uri` | string | The full AT URI (e.g. `at://did:plc:abc/col/rkey`) | +| `did` | string | The repo DID | +| `collection` | string | The collection NSID | +| `rkey` | string | The record key | +| `record` | table? | The full record as a Lua table (nil on delete) | +| `event` | table | The full event payload (see below) | + +The `event` table contains the same fields as the individual globals (`action`, `uri`, `did`, `collection`, `rkey`, `record`). New scripts can use either style -- `event.action` or the bare `action` global -- both work. The `event` table corresponds to the `RecordEventPayload` struct in the Rust dispatcher. + +### Label script globals + +These globals are set before `handle()` is called for label events: + +| Global | Type | Description | +| ------- | ------- | ------------------------------------------------ | +| `src` | string | DID of the labeler that issued the label | +| `uri` | string | The label subject (`at://` URI or bare DID) | +| `val` | string | The label value (e.g. `"!hide"`, `"nudity"`) | +| `neg` | boolean | `true` if this is a negation (label removal) | +| `cts` | string | Creation timestamp (ISO 8601) | +| `exp` | string? | Expiration timestamp (nil if the label does not expire) | +| `event` | table | The full label event as a table (same fields) | + +Record and label scripts do **not** have access to `caller_did`, `input`, `params`, `method`, or the `Record` API. They run from the event stream, not from a user request. + +## Available APIs + +Record and label scripts have access to: + +- **[Database API](../api-reference/lua/database-api.md)** -- `db.query`, `db.get`, `db.search`, `db.backlinks`, `db.count`, `db.raw` +- **[HTTP API](../api-reference/lua/http-api.md)** -- `http.get`, `http.post`, `http.put`, `http.patch`, `http.delete`, `http.head` +- **[XRPC Lua API](../api-reference/lua/xrpc-lua-api.md)** -- `xrpc.query`, `xrpc.procedure` +- **[atproto API](../api-reference/lua/atproto-api.md)** -- `atproto.resolve_service_endpoint`, `atproto.get_labels`, `atproto.get_labels_batch` +- **[JSON API](../api-reference/lua/json-api.md)** -- `json.encode`, `json.decode` +- **[Utility globals](./lua-scripting.md#utility-globals)** -- `log()`, `now()`, `TID()`, `toarray()` +- **[Script variables](../api-reference/admin/script-variables.md)** -- `env` table with key-value pairs configured in the dashboard + +## Error handling and retries + +Record and label scripts are designed to be resilient: + +1. If a script fails, it retries up to **4 attempts total** (1 initial + 3 retries) with exponential backoff (1s, 2s, 4s delays). +2. If all attempts are exhausted, the failed event is inserted into the `dead_letter_scripts` table for later inspection. +3. On failure the system **fails open** -- the original record or label is stored as-is so indexing is not permanently blocked. The firehose has no caller to surface errors to. + +Failed scripts are logged as errors. Check the [event logs](./event-logs.md) or query the `dead_letter_scripts` table directly to find and replay failures. + +### Performance considerations + +Because scripts run synchronously before indexing, they block the Jetstream consumer while executing. With retry logic (1s + 2s + 4s backoff), a persistently failing script could block for ~7 seconds per event. Keep scripts fast and ensure external services they depend on are reliable. + +### Dead letter table + +The `dead_letter_scripts` table stores events that failed all retry attempts: + +| Column | Type | Description | +| ------------ | ----------- | ----------------------------------------------------- | +| `id` | BIGSERIAL | Primary key | +| `script_ref` | text | The trigger id of the script that failed | +| `host_kind` | text | `'record'` or `'label'` | +| `host_id` | text | Identifies the specific event source | +| `payload` | jsonb | The full event payload | +| `error` | text | The error message from the last attempt | +| `attempts` | int | Total number of attempts made | +| `created_at` | timestamptz | When the failure was recorded | +| `resolved_at`| timestamptz | When the failure was resolved (null until resolved) | + +## Examples + +### Filter out records missing a required field + +Create a script with trigger `record.index:your.collection.nsid` to skip indexing any record that doesn't have a `title` field: + +```lua +function handle() + if action == "delete" then + return record -- allow deletes to proceed + end + + if record.title == nil or record.title == "" then + return nil -- skip: no title + end + + return record +end +``` + +### Transform a record before storage + +Enrich a record with a computed field before it is stored: + +```lua +function handle() + if action == "delete" then + return record + end + + record.slug = string.lower(string.gsub(record.title or "", "%s+", "-")) + return record +end +``` + +### Post to a webhook + +```lua +function handle() + http.post("https://hooks.example.com/records", { + headers = { ["Content-Type"] = "application/json" }, + body = json.encode({ + action = action, + uri = uri, + did = did, + record = record + }) + }) + return record +end +``` + +### Sync to Algolia + +Push records to an Algolia search index on create/update, and remove them on delete: + +```lua +function handle() + local headers = { + ["X-Algolia-API-Key"] = "your-api-key", + ["X-Algolia-Application-Id"] = "your-app-id", + ["Content-Type"] = "application/json" + } + + if action == "delete" then + http.delete("https://YOUR-APP.algolia.net/1/indexes/records/" .. uri, { + headers = headers + }) + else + http.put("https://YOUR-APP.algolia.net/1/indexes/records/" .. uri, { + headers = headers, + body = json.encode({ + objectID = uri, + collection = collection, + did = did, + record = record + }) + }) + end + + return record +end +``` + +See the full [Algolia sync reference](../reference/script-examples/algolia-sync.md) for more detail. + +### Sync to Meilisearch + +Push records to a self-hosted Meilisearch index on create/update, and remove them on delete: + +```lua +function handle() + local headers = { + ["Authorization"] = "Bearer " .. env.MEILISEARCH_API_KEY, + ["Content-Type"] = "application/json" + } + + if action == "delete" then + http.delete(env.MEILISEARCH_URL .. "/indexes/records/documents/" .. uri, { + headers = headers + }) + else + http.post(env.MEILISEARCH_URL .. "/indexes/records/documents", { + headers = headers, + body = json.encode(toarray({ + { + id = uri, + collection = collection, + did = did, + record = record + } + })) + }) + end + + return record +end +``` + +See the full [Meilisearch sync reference](../reference/script-examples/meilisearch-sync.md) for more detail. + +### Filter labels by value + +Create a script with trigger `labeler.apply:your.collection.nsid` to only persist specific label values: + +```lua +function handle() + local allowed = { ["!hide"] = true, ["nudity"] = true, ["spam"] = true } + + if not allowed[val] then + return nil -- skip: label value not in allowlist + end + + return event +end +``` + +### Rewrite a label field + +Normalize the label value before it is stored: + +```lua +function handle() + return { val = string.lower(val) } +end +``` + +Fields not returned fall back to their original values, so only `val` is changed here. + +## Next steps + +- [Lua Scripting](./lua-scripting.md): Full reference for the sandbox, APIs, and debugging (covers query and procedure scripts) +- [Admin API -- Scripts](../api-reference/admin/scripts.md): Create and manage scripts via the API +- [Lexicons](lexicons.md): Understand how record, query, and procedure lexicons work together diff --git a/packages/docs/content/docs/index.md b/packages/docs/content/docs/index.md index 69bd6c6..917ac20 100644 --- a/packages/docs/content/docs/index.md +++ b/packages/docs/content/docs/index.md @@ -12,7 +12,7 @@ Building an AppView from scratch means wiring up real-time event streams, record - **Network sync built in:** Real-time record streaming via [Jetstream](https://github.com/bluesky-social/jetstream), historical [backfill](guides/backfill.md) from each user's PDS, and atproto OAuth with DPoP-bound proxy writes back to the PDS. -- **Customize with Lua, hooks, and plugins:** [Lua scripts](guides/lua-scripting.md) for query and procedure logic, [index hooks](guides/index-hooks.md) that fire on every record change, WASM [plugins](guides/plugins.md) for external platform integration, and [labeler](guides/labelers.md) subscriptions for content moderation. +- **Customize with Lua scripts and plugins:** Trigger-keyed [Lua scripts](guides/lua-scripting.md) for XRPC query/procedure logic and [record/label event handling](guides/label-scripts), WASM [plugins](guides/plugins.md) for external platform integration, and [labeler](guides/labelers.md) subscriptions for content moderation. - **Protocol-native:** Works with any PDS, resolves DIDs through the directory, and fetches [network lexicons](guides/lexicons.md#network-lexicons) via DNS authority resolution. @@ -33,7 +33,7 @@ Building an AppView from scratch means wiring up real-time event streams, record - [Quickstart](getting-started/deployment/railway.md): Deploy HappyView on Railway or run it locally - [Lexicons](guides/lexicons.md): Upload lexicon schemas and start indexing records - [Lua Scripting](guides/lua-scripting.md): Write custom query and procedure logic -- [Index Hooks](guides/index-hooks.md): React to record changes in real time +- [Record & Label Scripts](guides/label-scripts): React to record changes and label events in real time - [Labelers](guides/labelers.md): Subscribe to external labelers and manage content labels - [Plugins](guides/plugins.md): Integrate with external platforms using WASM plugins - [Event Logs](guides/event-logs.md): Monitor system activity, debug script errors, and audit admin actions diff --git a/packages/docs/content/docs/reference/script-examples/algolia-sync.md b/packages/docs/content/docs/reference/script-examples/algolia-sync.md index 1ae1949..d4594c6 100644 --- a/packages/docs/content/docs/reference/script-examples/algolia-sync.md +++ b/packages/docs/content/docs/reference/script-examples/algolia-sync.md @@ -4,7 +4,7 @@ title: "Algolia Sync" Push records to an Algolia search index whenever they are created, updated, or deleted on the network. -**Lexicon type:** record (index hook) +**Script type:** record event (e.g. `record.index:`) ```lua function handle() diff --git a/packages/docs/content/docs/reference/script-examples/meilisearch-sync.md b/packages/docs/content/docs/reference/script-examples/meilisearch-sync.md index fdb4a7b..a8f4f26 100644 --- a/packages/docs/content/docs/reference/script-examples/meilisearch-sync.md +++ b/packages/docs/content/docs/reference/script-examples/meilisearch-sync.md @@ -4,7 +4,7 @@ title: "Meilisearch Sync" Push records to a Meilisearch search index whenever they are created, updated, or deleted on the network. -**Lexicon type:** record (index hook) +**Script type:** record event (e.g. `record.index:`) ```lua function handle()