diff --git a/docs/README.md b/docs/README.md --- a/docs/README.md +++ b/docs/README.md @@ -11,6 +11,7 @@ - 🔄 **Real-Time Sync**: Records stream in from the AT Protocol network in real-time via [Tap](https://github.com/bluesky-social/indigo/tree/main/cmd/tap), with cryptographic verification and backfill via the admin API - 🔐 **OAuth Built In**: [AIP](https://github.com/graze-social/aip) handles authentication, and writes are proxied back to the user's PDS, so there's no session management needed - 🌙 **Lua Scripting**: Add custom query and procedure logic with Lua scripts that have full access to the record database - 🗄️ **Automatic Indexing**: HappyView indexes relevant records into PostgreSQL as they arrive, ready to query +- 🪝 **Index Hooks**: Attach Lua scripts to record collections that fire on every create, update, or delete — sync to search engines, trigger webhooks, or build materialized views in real time - 🌐 **Network Lexicons**: Fetch lexicon schemas directly from the AT Protocol network via DNS authority resolution - ⚡ **Hot Reloading**: Upload or update lexicons at runtime, and new endpoints are available immediately with no restart - 🛠️ **Admin Dashboard**: Manage lexicons, monitor record stats, and run backfill jobs through a built-in admin API @@ -30,4 +31,5 @@ - [Quickstart](getting-started/deployment/railway.md): Deploy HappyView on Railway or run it locally - [Lexicons](guides/lexicons.md): Upload lexicon schemas and start indexing records - [Lua Scripting](guides/scripting.md): Write custom query and procedure logic +- [Index Hooks](guides/index-hooks.md): React to record changes in real time - [Event Logs](guides/event-logs.md): Monitor system activity, debug script errors, and audit admin actions diff --git a/docs/guides/index-hooks.md b/docs/guides/index-hooks.md new file mode 100644 --- /dev/null +++ b/docs/guides/index-hooks.md @@ -0,0 +1,133 @@ +# Index Hooks + +Index hooks are Lua scripts that run automatically whenever a record in a collection is created, updated, or deleted on the network. They let you react to record changes in real time — push data to search engines, sync with external APIs, send notifications, or build materialized views. + +Unlike [query and procedure scripts](scripting.md) that run in response to XRPC requests, index hooks are triggered by the firehose. They run asynchronously and never block record indexing. + +## Attaching a hook + +Each record-type lexicon can have one index hook. You can add it through the [dashboard](../getting-started/dashboard.md) (click "Add Index Hook" on any record lexicon's detail page) or via the [admin API](../reference/admin-api.md#upload--upsert-a-lexicon) by including the `on_index_script` field when uploading a lexicon. + +## Script structure + +Like query and procedure scripts, index hooks must define a `handle()` function: + +```lua +function handle() + if action == "delete" then + log("deleted " .. uri) + else + log(action .. " " .. uri) + end +end +``` + +The function is called once per record event. There is no return value — index hooks are fire-and-forget from the caller's perspective. + +## Context globals + +These globals are set before `handle()` is called: + +| Global | Type | Description | +| ------------ | ------ | -------------------------------------------------- | +| `action` | string | `"create"`, `"update"`, or `"delete"` | +| `uri` | string | The full AT URI (e.g. `at://did:plc:abc/col/rkey`) | +| `did` | string | The repo DID | +| `collection` | string | The collection NSID | +| `rkey` | string | The record key | +| `record` | table? | The full record as a Lua table (nil on delete) | + +Index hooks do **not** have access to `caller_did`, `input`, `params`, `method`, or the `Record` API. They run from the firehose, not from a user request. + +## Available APIs + +Index hooks have access to: + +- **[Database API](scripting.md#database-api)** — `db.query`, `db.get`, `db.search`, `db.backlinks`, `db.count`, `db.raw` +- **[HTTP API](scripting.md#http-api)** — `http.get`, `http.post`, `http.put`, `http.patch`, `http.delete`, `http.head` +- **[JSON API](scripting.md#json-api)** — `json.encode`, `json.decode` +- **[Utility globals](scripting.md#utility-globals)** — `log()`, `now()`, `TID()`, `toarray()` + +## Error handling and retries + +Index hooks are designed to be resilient: + +1. If a hook fails, it retries up to **3 times** with exponential backoff (1s, 2s, 4s delays). +2. If all retries are exhausted, the failed event is inserted into the `dead_letter_hooks` table for later inspection. +3. Hook failures never block record indexing — the record is always indexed regardless of whether the hook succeeds. + +Failed hooks are logged as errors. Check the [event logs](event-logs.md) or query the `dead_letter_hooks` table directly to find and replay failures. + +### Dead letter table + +The `dead_letter_hooks` table stores events that failed all retry attempts: + +| Column | Type | Description | +| ------------ | ----------- | --------------------------------------- | +| `id` | UUID | Primary key | +| `lexicon_id` | text | The lexicon NSID | +| `uri` | text | The AT URI of the record | +| `did` | text | The repo DID | +| `collection` | text | The collection NSID | +| `rkey` | text | The record key | +| `action` | text | `create`, `update`, or `delete` | +| `record` | jsonb | The record data (null on delete) | +| `error` | text | The error message from the last attempt | +| `attempts` | int | Total number of attempts made | +| `created_at` | timestamptz | When the failure was recorded | + +## Examples + +### Post to a webhook + +```lua +function handle() + http.post("https://hooks.example.com/records", { + headers = { ["Content-Type"] = "application/json" }, + body = json.encode({ + action = action, + uri = uri, + did = did, + record = record + }) + }) +end +``` + +### Sync to Algolia + +Push records to an Algolia search index on create/update, and remove them on delete: + +```lua +function handle() + local headers = { + ["X-Algolia-API-Key"] = "your-api-key", + ["X-Algolia-Application-Id"] = "your-app-id", + ["Content-Type"] = "application/json" + } + + if action == "delete" then + http.delete("https://YOUR-APP.algolia.net/1/indexes/records/" .. uri, { + headers = headers + }) + else + http.put("https://YOUR-APP.algolia.net/1/indexes/records/" .. uri, { + headers = headers, + body = json.encode({ + objectID = uri, + collection = collection, + did = did, + record = record + }) + }) + end +end +``` + +See the full [Algolia sync reference](../reference/scripts/algolia-sync.md) for more detail. + +## Next steps + +- [Lua Scripting](scripting.md): Full reference for the sandbox, APIs, and debugging +- [Lexicons](lexicons.md): Understand how record, query, and procedure lexicons work together +- [Admin API](../reference/admin-api.md#upload--upsert-a-lexicon): Upload lexicons with index hooks via the API diff --git a/docs/guides/lexicons.md b/docs/guides/lexicons.md --- a/docs/guides/lexicons.md +++ b/docs/guides/lexicons.md @@ -8,7 +8,7 @@ ## Supported lexicon types | Type | Effect | | ------------- | ------------------------------------------------------------------------------ | -| `record` | Syncs the collection filter to Tap and indexes records into Postgres | +| `record` | Syncs the collection filter to Tap and indexes records into Postgres. Supports [index hooks](index-hooks.md) | | `query` | Registers a `GET /xrpc/{nsid}` endpoint that queries indexed records | | `procedure` | Registers a `POST /xrpc/{nsid}` endpoint that proxies writes to the user's PDS | | `definitions` | Stored but does not generate routes or subscriptions | @@ -79,6 +79,7 @@ ## Next steps - [Lua Scripting](scripting.md): Add custom query and procedure logic to your endpoints +- [Index Hooks](index-hooks.md): Run Lua scripts when records are indexed from the network - [XRPC API](../reference/xrpc-api.md): Understand how the generated endpoints behave - [Backfill](backfill.md): Learn how historical records are indexed - [Admin API](../reference/admin-api.md): Full reference for lexicon management endpoints diff --git a/docs/guides/scripting.md b/docs/guides/scripting.md --- a/docs/guides/scripting.md +++ b/docs/guides/scripting.md @@ -8,7 +8,9 @@ - Validate input - Compose multi-record operations - Build entirely custom behavior -Scripts are attached to query and procedure lexicons and run in a sandboxed Lua VM with access to the [Record API](#record-api), a [read-only database API](#database-api), an [HTTP client API](#http-api), and a set of [context globals](#context-globals). +Scripts are attached to query and procedure lexicons and run in a sandboxed Lua VM with access to the [Record API](#record-api), a [read-only database API](#database-api), an [HTTP client API](#http-api), a [JSON API](#json-api), and a set of [context globals](#context-globals). + +For scripts that react to record changes from the network (rather than XRPC requests), see [Index Hooks](index-hooks.md). ## Script structure @@ -322,6 +324,28 @@ local resp = http.delete(url, { headers = { ... } }) local resp = http.head(url) ``` +## JSON API + +The `json` global provides JSON serialization and deserialization. Available in queries, procedures, and [index hooks](index-hooks.md). + +### json.encode + +```lua +local str = json.encode({ key = "value", items = { 1, 2, 3 } }) +-- '{"key":"value","items":[1,2,3]}' +``` + +Converts a Lua table to a JSON string. + +### json.decode + +```lua +local tbl = json.decode('{"key": "value"}') +-- tbl.key == "value" +``` + +Parses a JSON string into a Lua table. Returns an error if the input is not valid JSON. + ## Standard libraries The following Lua 5.4 standard library modules are available: @@ -447,8 +471,12 @@ - [Sidecar records](../reference/scripts/sidecar-records.md) — create linked records across collections with a shared rkey - [Cascading delete](../reference/scripts/cascading-delete.md) — delete a record and all related records - [Complex mutations](../reference/scripts/complex-mutations.md) — load, transform, and save a record with multiple field changes +**Index Hooks:** +- [Algolia sync](../reference/scripts/algolia-sync.md) — push records to an Algolia search index on create/update/delete + ## Next steps +- [Index Hooks](index-hooks.md): React to record changes from the network in real time - [Lexicons](lexicons.md): Understand how record, query, and procedure lexicons work together - [XRPC API](../reference/xrpc-api.md): See how endpoints behave with and without Lua scripts - [Dashboard](../getting-started/dashboard.md#lua-editor): Use the web editor with context-aware completions diff --git a/docs/reference/admin-api.md b/docs/reference/admin-api.md --- a/docs/reference/admin-api.md +++ b/docs/reference/admin-api.md @@ -54,6 +54,8 @@ | ------------------- | ------- | -------- | ------------------------------------------------------------------- | | `lexicon_json` | object | yes | Raw lexicon JSON (must have `lexicon: 1` and `id`) | | `backfill` | boolean | no | Whether uploading triggers historical backfill (default `true`) | | `target_collection` | string | no | For query/procedure lexicons, the record collection they operate on | +| `script` | string | no | Lua script for query/procedure endpoints | +| `on_index_script` | string | no | [Index hook](../guides/index-hooks.md) Lua script for record lexicons | **Response**: `201 Created` (new) or `200 OK` (upsert) diff --git a/docs/reference/scripts/algolia-sync.md b/docs/reference/scripts/algolia-sync.md new file mode 100644 --- /dev/null +++ b/docs/reference/scripts/algolia-sync.md @@ -0,0 +1,55 @@ +# Index Hook: Algolia Sync + +Push records to an Algolia search index whenever they are created, updated, or deleted on the network. + +**Lexicon type:** record (index hook) + +```lua +function handle() + local headers = { + ["X-Algolia-API-Key"] = "your-api-key", + ["X-Algolia-Application-Id"] = "your-app-id", + ["Content-Type"] = "application/json" + } + + if action == "delete" then + http.delete("https://YOUR-APP.algolia.net/1/indexes/records/" .. uri, { + headers = headers + }) + else + http.put("https://YOUR-APP.algolia.net/1/indexes/records/" .. uri, { + headers = headers, + body = json.encode({ + objectID = uri, + collection = collection, + did = did, + record = record + }) + }) + end +end +``` + +## How it works + +1. On **create** or **update**: sends a `PUT` request to Algolia's index API with the record data, using the AT URI as the `objectID`. Algolia upserts the object — if it already exists, it's replaced. +2. On **delete**: sends a `DELETE` request to remove the object from the index by its AT URI. + +The `json.encode()` function converts the Lua table into a JSON string for the request body. See [JSON API](../../guides/index-hooks.md#json-api). + +## Configuration + +Replace the placeholder values: + +| Placeholder | Value | +| ------------------------ | --------------------------------------------------------------------- | +| `your-api-key` | Your Algolia Admin API key (with write permissions) | +| `your-app-id` | Your Algolia Application ID | +| `YOUR-APP` | Your Algolia application subdomain (same as the Application ID) | +| `records` | The Algolia index name (choose any name you like) | + +## Use case + +This hook keeps an external search index in sync with your indexed records in real time. Users searching through Algolia get results that reflect the latest state of the network without polling or scheduled jobs. + +Combine this with a [query script](../../guides/scripting.md) that searches Algolia instead of the local database for a full-text search experience that goes beyond what `db.search` offers. diff --git a/sidebars.ts b/sidebars.ts --- a/sidebars.ts +++ b/sidebars.ts @@ -81,6 +81,11 @@ label: "Lua Scripting", }, { type: "doc", + id: "guides/index-hooks", + label: "Index Hooks", + }, + { + type: "doc", id: "guides/backfill", label: "Backfill", }, @@ -163,6 +168,11 @@ { type: "doc", id: "reference/scripts/complex-mutations", label: "Complex Mutations", + }, + { + type: "doc", + id: "reference/scripts/algolia-sync", + label: "Algolia Sync", }, ], },