# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## What this is

Stargazer is a text search over **non-Bluesky ATProto records**. It tails the Bluesky Jetstream firehose, drops anything that matches an NSID blocklist (by default `app.bsky.*`, `chat.bsky.*`, etc.), extracts prose from the remaining lexicons, and indexes it into Meilisearch.

## Commands

Dev shell (provides Go toolchain, Meilisearch, process-compose, delve, golangci-lint):

```
nix develop
```

Run the whole stack (Meilisearch + ingester):

```
process-compose up
```

Run pieces individually:

```
meilisearch --db-path ./.state/meili --http-addr 127.0.0.1:7700
go run ./cmd/ingester [-config config.yaml]
go run ./cmd/search [-config config.yaml] [-collection <nsid|prefix*>] [-limit N] <query...>
```

Build / lint / test:

```
go build ./...
go vet ./...
golangci-lint run
go test ./...
go test ./internal/classify -run TestX   # single test
```

State lives in `./.state/` (Meili DB + cursor). Deleting `./.state/cursor` starts from now (or the beginning of the sealed archive if `jetstream.api_key` is set).

## Configuration

`config.yaml` is non-secret. Secrets come from env and override the file:

- `JETSTREAM_API_KEY` — presence flips the ingester from **live-only** tail to **archive-replay + live** (see below).
- `MEILI_MASTER_KEY` — Meili auth.

## Architecture

Two binaries share `internal/`:

- `cmd/ingester` — long-running firehose consumer.
- `cmd/search` — one-shot CLI that queries Meili directly (no server component).

Ingester pipeline, per batch from `jetstream.Client.Events`:

1. Skip non-commit events.
2. On delete: queue the doc id for removal.
3. On create/update: `classify.Classifier.IsBlocked(nsid)` drops blocked NSIDs early (no decode cost beyond what Jetstream already did).
4. `classify.Classifier.Extract` walks the decoded record and concatenates every string field whose key is in `prose_fields` (matched at any depth, min rune length applied). Also pulls `createdAt` and `langs[0]`.
5. Records with empty extracted text are counted as `notext` and skipped.
6. Surviving docs are pushed to Meili in one `AddDocuments` call, then deletes, then the cursor is saved.

Key design points to preserve when editing:

- **Full firehose, client-side filter.** The ingester intentionally does *not* pass `WithCollections` to Jetstream. It eats full bandwidth (Bluesky-native traffic dominates) in exchange for automatic discovery of new lexicons and a simpler model. If you narrow the subscription, update the comment in `cmd/ingester/main.go` and the blocklist notes in `config.yaml`.
- **Two subscribe modes, chosen by API key presence.** With key → `WithAPIKey + WithAfterSeq(start)` (exclusive; clean resume; cold start replays the sealed archive). Without → `WithLiveCursor(start)` (server's live buffer; `0` == "from now"). Don't collapse these paths.
- **Cursor is written after Meili accepts the batch.** If `AddDocuments` fails, the loop `continue`s without saving the cursor, so the failed range is replayed on the next connect. Preserve this ordering.
- **Meili primary key sanitisation.** `docID` in `cmd/ingester/main.go` maps `did/collection/rkey` → `[A-Za-z0-9_-]` by replacing `:`, `/`, `.` with `_`. The full AT-URI is stored separately as `uri`. Don't change the id scheme without a full reindex.
- **Meili schema is declared once in `internal/meili/meili.go` `Ensure`.** Searchable: `text`. Filterable: `collection`, `did`, `lang`, `createdAt`. Sortable: `createdAt`, `indexedAt`. Any new field the search CLI wants to filter/sort on must be added here.
- **Blocklist wildcards are prefix-only** and require the trailing `.` (`app.bsky.*` → prefix `app.bsky.`), so `app.bskyish` is not blocked. See `classify.New`.
- **Stats invariant:** `seen == blocked + indexed + notext`. The 30 s summary in the ingester relies on this; keep the four increments mutually exclusive per event.