diff --git a/README.md b/README.md index 91496b7..65db518 100644 --- a/README.md +++ b/README.md @@ -1,53 +1,42 @@ # ken -fuzzy find any record in your [atproto](https://atproto.com) repo. semantic search over a [PDS](https://atproto.com/guides/data-repos), with the vector pack written back to your own PDS as a record you can inspect or delete. +semantic search over your [atproto](https://atproto.com) repo. sign in, ken builds a vector index of your records, search by meaning. -running at **[ken.waow.tech](https://ken.waow.tech)**. +**[ken.waow.tech](https://ken.waow.tech)** -## how it works +## lexicons -1. sign in with your handle. oauth goes to your PDS. -2. backend fetches your whole repo in one call via [`com.atproto.sync.getRepo`](https://atproto.com/specs/sync#getrepo), parses the CAR locally via [zat](https://tangled.org/zat.dev/zat) -3. records whose collections have no semantic text (likes, follows, reposts, blocks, listitems, gates, etc.) are dropped before embedding. large repos also get a 2-year time cutoff so pipeline memory stays bounded; the pack-meta line in the UI shows exactly what was cut -4. each surviving record is embedded with [bge-small-en-v1.5](https://huggingface.co/BAAI/bge-small-en-v1.5) running through [llama.cpp](https://github.com/ggerganov/llama.cpp), 16 records per batch -5. optional: click save, and the resulting vector pack is written back to your PDS as a `tech.waow.ken.pack` record plus a few vector blobs. ken's server keeps nothing past the current session — the pack lives on your PDS, and ken just reloads it on your next sign-in. click delete and ken tombstones the record on your repo -6. subsequent sign-ins reuse vectors by `(uri, cid)` — only new or changed records get re-embedded +- [`tech.waow.ken.pack`](lexicons/tech/waow/ken/pack.json) — saved vector index (record + blobs on your PDS) -search is in-memory cosine similarity across whatever the backend currently has cached for you. partial search works from the moment the first batch finishes, so the UI never blocks waiting on a full index. - -### data propagation - -writing a record to a public PDS is a broadcast: the PDS emits a firehose event, and any relay or downstream consumer subscribed to your PDS can ingest the record and the blobs it references. this is how atproto is designed to work, and ken participates in it like every other app that writes records. your pack is not uniquely exposed — it propagates the same way your posts do — but "saved on my PDS" is not the same as "only on my PDS." if you want to minimize your network surface, don't click save; an unsaved pack lives only in ken's in-memory cache and disappears when you sign out or the server restarts. - -## sharing - -a signed-in user can share a specific search via `https://ken.waow.tech/?handle=X&q=Y`. the backend's `GET /` injects per-query OpenGraph tags so link unfurlers render a real preview. a visitor loads the target's saved pack via the same public-read path anyone else could take, and runs the query — no auth needed. - -the records being searched were already publicly readable from the PDS, so sharing a query doesn't expose any individual record that wasn't exposed before. what it does add is a new way to *find* things: semantic search across every record you've indexed is a different discoverability surface than e.g. scrolling a profile. if you have records that are technically public but you'd rather not see surfaced by meaning, don't save the pack. - -## layout +## stack ``` -backend/ zig http server + llama.cpp wrapper + indexer - src/ main source - llama-include/ llama.h headers - llama-bin/ linux x86_64 .so files (docker build) - llama-bin-macos/ arm64 dylibs (local dev) - models/ bge-small.gguf - fly.toml production config - Dockerfile multi-stage build for fly -lexicons/ atproto lexicon specs - tech/waow/ken/pack.json +sync.getRepo → CAR walk + filter → llama.cpp embed → in-memory search + ↓ + (opt-in) save pack to user's PDS ``` -## running locally +- **backend**: [zig](https://ziglang.org) 0.16, [zat](https://tangled.org/zat.dev/zat) (AT Protocol), [llama.cpp](https://github.com/ggerganov/llama.cpp) ([bge-small-en-v1.5](https://huggingface.co/BAAI/bge-small-en-v1.5)) +- **infra**: [fly.io](https://fly.io) (performance-2x, 4 GB) -```bash +## develop + +```sh cd backend zig build OAUTH_CLIENT_SECRET_KEY=... MODEL_PATH=models/bge-small.gguf ./zig-out/bin/embed-on-pds ``` +## deploy + +```sh +just deploy +``` + +## notes + +- [large-repo embedding pipeline](notes/large-repo-embedding.md) — filtering, time cutoff, memory budget + ## license MIT diff --git a/backend/src/assets/main.js b/backend/src/assets/main.js index b506033..3d1a4ed 100644 --- a/backend/src/assets/main.js +++ b/backend/src/assets/main.js @@ -341,16 +341,14 @@ function renderPackActions(j) { if (j.persisted) { packStateEl.textContent = "saved"; - packMenuDescEl.textContent = - "a vector index of your records lives as a record on your PDS. ken reloads it on sign-in so you don't have to re-embed."; + packMenuDescEl.textContent = "on your PDS. ken reloads it next sign-in."; packViewLink.classList.remove("hidden"); packViewLink.href = `https://pdsls.dev/${j.persisted_uri}`; packDeleteBtn.classList.remove("hidden"); packSaveBtn.classList.add("hidden"); } else { packStateEl.textContent = "not saved"; - packMenuDescEl.textContent = - "the index lives in ken's memory only. save it to your PDS to keep it across sessions."; + packMenuDescEl.textContent = "in memory only. save to keep it across sessions."; packViewLink.classList.add("hidden"); packDeleteBtn.classList.add("hidden"); packSaveBtn.classList.remove("hidden");