manoo.dev / stargazer
text search for the atmosphere
Star
Fork 0
Watch
stargazer CLAUDE.md
4.0 kB
Markdown
at main

CLAUDE.md #

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

What this is #

Stargazer is a text search over non-Bluesky ATProto records. It tails the Bluesky Jetstream firehose, drops anything that matches an NSID blocklist (by default app.bsky.*, chat.bsky.*, etc.), extracts prose from the remaining lexicons, and indexes it into Meilisearch.

Commands #

Dev shell (provides Go toolchain, Meilisearch, process-compose, delve, golangci-lint):

nix develop

Run the whole stack (Meilisearch + ingester):

process-compose up

Run pieces individually:

meilisearch --db-path ./.state/meili --http-addr 127.0.0.1:7700
go run ./cmd/ingester [-config config.yaml]
go run ./cmd/search [-config config.yaml] [-collection <nsid|prefix*>] [-limit N] <query...>

Build / lint / test:

go build ./...
go vet ./...
golangci-lint run
go test ./...
go test ./internal/classify -run TestX   # single test

State lives in ./.state/ (Meili DB + cursor). Deleting ./.state/cursor starts from now (or the beginning of the sealed archive if jetstream.api_key is set).

Configuration #

config.yaml is non-secret. Secrets come from env and override the file:

  • JETSTREAM_API_KEY — presence flips the ingester from live-only tail to archive-replay + live (see below).
  • MEILI_MASTER_KEY — Meili auth.

Architecture #

Two binaries share internal/:

  • cmd/ingester — long-running firehose consumer.
  • cmd/search — one-shot CLI that queries Meili directly (no server component).

Ingester pipeline, per batch from jetstream.Client.Events:

  1. Skip non-commit events.
  2. On delete: queue the doc id for removal.
  3. On create/update: classify.Classifier.IsBlocked(nsid) drops blocked NSIDs early (no decode cost beyond what Jetstream already did).
  4. classify.Classifier.Extract walks the decoded record and concatenates every string field whose key is in prose_fields (matched at any depth, min rune length applied). Also pulls createdAt and langs[0].
  5. Records with empty extracted text are counted as notext and skipped.
  6. Surviving docs are pushed to Meili in one AddDocuments call, then deletes, then the cursor is saved.

Key design points to preserve when editing:

  • Full firehose, client-side filter. The ingester intentionally does not pass WithCollections to Jetstream. It eats full bandwidth (Bluesky-native traffic dominates) in exchange for automatic discovery of new lexicons and a simpler model. If you narrow the subscription, update the comment in cmd/ingester/main.go and the blocklist notes in config.yaml.
  • Two subscribe modes, chosen by API key presence. With key → WithAPIKey + WithAfterSeq(start) (exclusive; clean resume; cold start replays the sealed archive). Without → WithLiveCursor(start) (server's live buffer; 0 == "from now"). Don't collapse these paths.
  • Cursor is written after Meili accepts the batch. If AddDocuments fails, the loop continues without saving the cursor, so the failed range is replayed on the next connect. Preserve this ordering.
  • Meili primary key sanitisation. docID in cmd/ingester/main.go maps did/collection/rkey → [A-Za-z0-9_-] by replacing :, /, . with _. The full AT-URI is stored separately as uri. Don't change the id scheme without a full reindex.
  • Meili schema is declared once in internal/meili/meili.go Ensure. Searchable: text. Filterable: collection, did, lang, createdAt. Sortable: createdAt, indexedAt. Any new field the search CLI wants to filter/sort on must be added here.
  • Blocklist wildcards are prefix-only and require the trailing . (app.bsky.* → prefix app.bsky.), so app.bskyish is not blocked. See classify.New.
  • Stats invariant: seen == blocked + indexed + notext. The 30 s summary in the ingester relies on this; keep the four increments mutually exclusive per event.