diff --git a/docs/api/specs/01-architecture.md b/docs/api/specs/01-architecture.md index 6eec7e8..2dbc76c 100644 --- a/docs/api/specs/01-architecture.md +++ b/docs/api/specs/01-architecture.md @@ -64,8 +64,8 @@ The system shall prioritize: - **AT Protocol network** — source of all Tangled content - **Tap** — filtered event delivery from the AT Protocol firehose (deployed on Railway) - **Turso/libSQL** — relational storage, Tantivy-backed FTS, and native vector search -- **Embedding provider** — generates vectors for semantic search -- **Railway** — deployment platform for Twister services and Tap +- **Ollama** — local embedding model server (nomic-embed-text or EmbeddingGemma); deployed as a Railway sidecar service +- **Railway** — deployment platform for Twister services, Tap, and Ollama ## 7. Architecture Summary @@ -106,7 +106,8 @@ ATProto Firehose / PDS | -------------- | -------------------------------------------- | -------------------------- | | `api` | HTTP search, graph summary, and document API | Railway service (public) | | `indexer` | Tap consumer, normalizer, DB writer | Railway service (internal) | -| `embed-worker` | Async embedding generation | Optional Railway service | +| `embed-worker` | Async embedding generation via Ollama | Optional Railway service | +| `ollama` | Local embedding model server | Railway service (internal) | | `tap` | ATProto sync | Railway (already deployed) | ## 9. Repository Structure @@ -142,6 +143,22 @@ twister healthcheck # One-shot health probe ## 11. Technology Choices +### Embedding: Ollama (self-hosted) + +Embeddings are generated locally via Ollama rather than an external API service. This eliminates per-token costs, external service dependencies, and data egress concerns. + +**Recommended models (in order of preference):** + +| Model | Parameters | Dimensions | Quantized Size | Notes | +|-------|-----------|------------|----------------|-------| +| nomic-embed-text-v1.5 | 137M | 768 (Matryoshka: 64–768) | ~262 MB (F16) | 8192 context, battle-tested, Railway template exists | +| EmbeddingGemma | 308M | 768 | <200 MB (quantized) | Best-in-class MTEB for size, released Sept 2025 | +| all-minilm | 23M | 384 | ~46 MB | Budget option, lower quality | + +**Go integration:** Use the official Ollama Go client (`github.com/ollama/ollama/api`) with the `Embed()` method. The embed-worker calls Ollama over Railway's internal network (`ollama.railway.internal:11434`). + +**Railway deployment:** Ollama runs as a separate Railway service (~1–2 GB RAM, 1–2 vCPU, ~$10–30/mo). The nomic-embed Railway template provides a proven starting point. No cold starts on always-on services; model loads in 2–10 seconds on first request after deploy. + ### Language: Go Go is the implementation language for the API server, indexer, embedding worker, and CLI commands. Rationale: straightforward long-running services, excellent HTTP support, good concurrency model, small container footprint. diff --git a/docs/api/specs/03-data-model.md b/docs/api/specs/03-data-model.md index 49f9f76..714de4f 100644 --- a/docs/api/specs/03-data-model.md +++ b/docs/api/specs/03-data-model.md @@ -93,6 +93,17 @@ CREATE INDEX idx_documents_fts ON documents USING fts ( ) WITH (weights='title=3.0,repo_name=2.5,author_handle=2.0,summary=1.5,tags_json=1.2,body=1.0'); ``` +### FTS Maintenance + +Turso's Tantivy-backed FTS uses `NoMergePolicy` — segment count grows with writes and is never automatically compacted. This increases query fan-out over time. + +**Required maintenance:** Run `OPTIMIZE INDEX idx_documents_fts;` periodically (e.g., daily cron or after bulk backfill). This merges segments and reclaims space. + +**Known limitations:** +- No read-your-writes within a transaction — FTS queries see a pre-commit snapshot +- No snippet function (use `fts_highlight()` for highlighting) +- FTS is experimental in Turso; requires the `fts` feature flag + ## 4. Embeddings Table ```sql @@ -108,7 +119,27 @@ CREATE INDEX idx_embeddings_vec ON document_embeddings( ); ``` -The vector dimension (768) is configurable by model. Changing models requires a new column or table migration. +The vector dimension (768) matches nomic-embed-text-v1.5 and EmbeddingGemma defaults. Changing models may require a new column or table migration if the dimension changes. + +### Vector Index Tuning + +The DiskANN index accepts tuning parameters at creation time: + +```sql +CREATE INDEX idx_embeddings_vec ON document_embeddings( + libsql_vector_idx(embedding, 'metric=cosine', 'max_neighbors=50', 'search_l=200') +); +``` + +| Parameter | Default | Description | +|-----------|---------|-------------| +| `max_neighbors` | 3*sqrt(D) | Graph connectivity; higher = better recall, more storage | +| `search_l` | 200 | Neighbors visited during search; higher = better recall, slower | +| `insert_l` | 70 | Neighbors visited during insert | +| `alpha` | 1.2 | Graph sparsity factor | +| `compress_neighbors` | — | Quantize neighbor vectors for storage savings | + +Start with defaults and tune after measuring recall on representative queries. ## 5. Sync State Table diff --git a/docs/api/specs/04-data-pipeline.md b/docs/api/specs/04-data-pipeline.md index 08bdec0..e2f5447 100644 --- a/docs/api/specs/04-data-pipeline.md +++ b/docs/api/specs/04-data-pipeline.md @@ -324,6 +324,7 @@ If embedding generation fails: - Jobs are retried with exponential backoff up to a max attempt count - After max attempts, the job enters `dead` state - The embed-worker exposes failed job count as a metric +- If Ollama is unreachable (sidecar down), all pending jobs pause until connectivity is restored ### DB Failures diff --git a/docs/api/specs/05-search.md b/docs/api/specs/05-search.md index 5d38308..6915416 100644 --- a/docs/api/specs/05-search.md +++ b/docs/api/specs/05-search.md @@ -61,11 +61,17 @@ Use `fts_highlight()` to generate highlighted snippets: fts_highlight(d.body, '', '', ?) AS body_snippet ``` +### FTS Operational Notes + +- **Segment merging:** Turso FTS uses Tantivy's `NoMergePolicy`. Run `OPTIMIZE INDEX idx_documents_fts;` after bulk writes (backfill) and periodically in production to keep query performance stable. +- **Read-your-writes:** FTS queries within the same transaction see a pre-commit snapshot. If a document is written and immediately searched in the same transaction, FTS will not find it. The indexer and API are separate processes, so this is not a concern in normal operation. +- **Feature flag:** Turso FTS requires the `fts` feature flag to be enabled on the database. + ## 3. Semantic Search ### Query Flow -1. Convert user query text to embedding via the configured provider +1. Convert user query text to embedding via Ollama (self-hosted) 2. Query `vector_top_k` for nearest neighbors 3. Join back to `documents` to get metadata 4. Filter out deleted/hidden documents diff --git a/docs/api/specs/06-operations.md b/docs/api/specs/06-operations.md index ab039f6..cfbab69 100644 --- a/docs/api/specs/06-operations.md +++ b/docs/api/specs/06-operations.md @@ -41,16 +41,14 @@ All configuration is via environment variables. | `SEARCH_MAX_LIMIT` | `100` | Maximum results per page | | `SEARCH_DEFAULT_MODE` | `keyword` | Default search mode | -### Embedding +### Embedding (Ollama — self-hosted) -| Variable | Default | Description | -| ---------------------- | ------- | ---------------------------------------------------- | -| `EMBEDDING_PROVIDER` | — | Provider name (e.g., `openai`, `ollama`, `voyageai`) | -| `EMBEDDING_MODEL` | — | Model name (e.g., `text-embedding-3-small`) | -| `EMBEDDING_API_KEY` | — | Provider API key | -| `EMBEDDING_API_URL` | — | Provider base URL (for self-hosted) | -| `EMBEDDING_DIM` | `768` | Vector dimensionality | -| `EMBEDDING_BATCH_SIZE` | `32` | Batch size for embed-worker | +| Variable | Default | Description | +| ---------------------- | ------------------------------------------ | ---------------------------------------------- | +| `OLLAMA_URL` | `http://ollama.railway.internal:11434` | Ollama server URL | +| `EMBEDDING_MODEL` | `nomic-embed-text` | Ollama model name | +| `EMBEDDING_DIM` | `768` | Vector dimensionality (must match model) | +| `EMBEDDING_BATCH_SIZE` | `32` | Documents per embedding batch | ### Hybrid Search @@ -87,10 +85,9 @@ INDEXED_COLLECTIONS=sh.tangled.repo,sh.tangled.repo.issue,sh.tangled.repo.pull,s SEARCH_DEFAULT_LIMIT=20 SEARCH_MAX_LIMIT=100 -# Embedding (Phase 2) -# EMBEDDING_PROVIDER=openai -# EMBEDDING_MODEL=text-embedding-3-small -# EMBEDDING_API_KEY=sk-... +# Embedding — Ollama (Phase 2) +# OLLAMA_URL=http://ollama.railway.internal:11434 +# EMBEDDING_MODEL=nomic-embed-text # EMBEDDING_DIM=768 # Server @@ -285,7 +282,7 @@ Required secrets: | ------------------- | --------------------------------- | | `TURSO_AUTH_TOKEN` | Turso database authentication | | `TAP_AUTH_PASSWORD` | Tap admin API authentication | -| `EMBEDDING_API_KEY` | Embedding provider authentication | +| `OLLAMA_URL` | Ollama sidecar connection (no secret if internal networking) | | `ADMIN_AUTH_TOKEN` | Admin endpoint authentication | ### Admin Endpoints @@ -331,6 +328,7 @@ All Twister services deploy as separate Railway services within the same project | api | `twister api` | `GET /healthz` | yes | | indexer | `twister indexer` | `GET :9090/health` | no | | embed-worker | `twister embed-worker` | `GET :9091/health` | no | +| ollama | (Railway template) | `GET /api/tags` | no | All services share the same Docker image. Railway uses the start command to select the subcommand. @@ -356,10 +354,9 @@ TAP_URL=wss://${{tap.RAILWAY_PUBLIC_DOMAIN}}/channel # Railway service referenc TAP_AUTH_PASSWORD=... INDEXED_COLLECTIONS=sh.tangled.repo,sh.tangled.repo.issue,sh.tangled.repo.pull,sh.tangled.string,sh.tangled.actor.profile -# Embed-worker (Phase 2) -# EMBEDDING_PROVIDER=openai -# EMBEDDING_MODEL=text-embedding-3-small -# EMBEDDING_API_KEY=sk-... +# Embed-worker + Ollama (Phase 2) +# OLLAMA_URL=http://ollama.railway.internal:11434 +# EMBEDDING_MODEL=nomic-embed-text ``` Railway supports referencing other services' variables with `${{service.VAR}}` syntax, which is useful for linking the indexer to Tap's domain. diff --git a/docs/api/specs/09-search-site.md b/docs/api/specs/09-search-site.md new file mode 100644 index 0000000..4a9b71d --- /dev/null +++ b/docs/api/specs/09-search-site.md @@ -0,0 +1,166 @@ +--- +title: "Spec 09 — Search Site" +updated: 2026-03-23 +--- + +A minimal static site that serves as both the public Twister API documentation and a live search showcase. Dark mode only, no framework or build step. + +## 1. Purpose + +- Give developers a browsable reference for the Twister search API +- Give anyone a way to try search against live indexed Tangled content +- Provide a shareable public URL before the mobile app ships + +## 2. Scope + +In scope: + +- Static HTML/CSS/JS (Alpine.js, no bundler) +- API reference pages generated from the spec docs +- Live search input wired to `GET /search` +- Result rendering with type-aware cards (repo, issue, PR, profile, string) +- Filter controls for collection, type, author, language, state +- Pagination +- Responsive layout (mobile-friendly, single breakpoint) + +Out of scope: + +- Auth, OAuth, or any write operations +- Semantic or hybrid mode toggle (keyword only for MVP) +- Server-side rendering or static-site generator +- Analytics or telemetry + +## 3. Pages + +| Route | Content | +| ----------------- | --------------------------------------------------------------------------- | +| `/` | Search input + results (the homepage is the search page) | +| `/docs` | API overview: base URL, auth (none for public), rate limits, response shape | +| `/docs/search` | `GET /search` — parameters, filters, response contract, examples | +| `/docs/documents` | `GET /documents/{id}` — request/response, examples | +| `/docs/health` | `GET /healthz`, `GET /readyz` — purpose and expected responses | + +## 4. Search Page Behavior + +1. Text input with a submit button. No debounce search-as-you-type for MVP. +2. On submit, fetch `GET {API_BASE}/search?q={query}&limit=20` (plus any active filters). +3. Render results as a vertical list of cards. +4. Each card shows: `record_type` badge, `title`, `body_snippet` (with `` highlights preserved), `author_handle`, `repo_name` (when present), `updated_at` relative time. +5. Clicking a result opens the canonical Tangled URL (`https://tangled.org/{handle}/{repo}` for repos, etc.) in a new tab. +6. "Load more" button appends the next page (`offset += limit`). +7. Empty state: "No results" message. +8. Error state: inline message if the API is unreachable. +9. Filter bar above results: dropdowns/inputs for `type`, `language`, `author`. Filters are query params so URLs are shareable. + +## 5. API Docs Pages + +Hand-written HTML mirroring the contracts in spec 05 (search) and spec 08 (app integration). Each page includes: + +- Endpoint signature (method, path) +- Parameter table (name, type, default, description) +- Example request (curl) +- Example response (JSON block with syntax highlighting via `
`)
+
+No generated docs tooling. The pages are static and updated manually when the API changes.
+
+## 6. Styling
+
+Minimal CSS, no utility framework.
+
+### Tokens
+
+```css
+:root {
+  --bg: #0e0e0e;
+  --surface: #1a1a1a;
+  --border: #2a2a2a;
+  --text: #e0e0e0;
+  --text-dim: #888;
+  --accent: #7aa2f7;
+  --mark-bg: #7aa2f733;
+  --mono: "Google Sans Mono", monospace;
+  --sans: "Google Sans", sans-serif;
+  --radius: 6px;
+}
+```
+
+### Rules
+
+- Dark theming.
+- `Google Sans` for body text. `Google Sans Mono` for code, JSON, and badges.
+- Fonts loaded via Google Fonts ``. System fallbacks: `sans-serif`, `monospace`.
+- Max content width: `720px`, centered.
+- Cards: `var(--surface)` background, `var(--border)` border, `var(--radius)` corners.
+- `` tags in snippets styled with `var(--mark-bg)` background and `var(--accent)` text.
+- Code blocks: `var(--surface)` background, horizontal scroll, no wrapping.
+- Links: `var(--accent)`, no underline, underline on hover.
+- Inputs and buttons: `var(--surface)` background, `var(--border)` border, `var(--text)` text.
+- One breakpoint at `640px` for mobile: full-width cards, stacked filter bar.
+
+## 7. Package Design
+
+The site lives in `internal/view/` as a self-contained Go package. It owns the templates, static assets, and HTTP handlers. The `api` package mounts `view.Handler()` into its router — nothing else leaks out.
+
+### Exports
+
+The package exposes a single constructor:
+
+```go
+// Handler returns an http.Handler that serves the site pages and static assets.
+func Handler() http.Handler
+```
+
+The `api` package calls `view.Handler()` and mounts it as a fallback after API routes.
+
+### Package Structure
+
+```text
+internal/view/
+  view.go               # Handler(), route setup, embed directives
+  templates/
+    layout.html         # Shared shell (head, nav, footer)
+    index.html          # Search page
+    docs/
+      index.html        # API overview
+      search.html       # GET /search docs
+      documents.html    # GET /documents/{id} docs
+      health.html       # Health endpoints docs
+  static/
+    style.css           # All styles, single file
+    search.js           # Search fetch, render, pagination, filters
+```
+
+### Embedding
+
+`view.go` uses `//go:embed` to bundle `templates/` and `static/`. Templates are parsed once at init. Static assets are served under `/static/` via `http.FileServer`.
+
+### Routing
+
+`view.Handler()` returns a mux that handles:
+
+| Pattern | Handler |
+| --- | --- |
+| `GET /` | Render `index.html` |
+| `GET /docs` | Render `docs/index.html` |
+| `GET /docs/search` | Render `docs/search.html` |
+| `GET /docs/documents` | Render `docs/documents.html` |
+| `GET /docs/health` | Render `docs/health.html` |
+| `GET /static/*` | Serve embedded CSS/JS files |
+
+## 9. Configuration
+
+Since the site is served by the same origin as the API, search requests use relative paths (`/search?q=...`). No `API_BASE` config needed — the browser's origin is the API.
+
+## 10. Local Development
+
+Run `twister api` locally. The site is served at `http://localhost:8080/` alongside the API endpoints. No separate dev server or file server required.
+
+The API docs pages render without any indexed data. The search page needs a running indexer and populated database to return results.
+
+## 11. Constraints
+
+- No dependencies besides Alpine via CDN.
+- Total site weight target: under 50 KB excluding fonts.
+- Works in modern browsers (last 2 versions of Chrome, Firefox, Safari).
+- All fetch calls include error handling for network failures and non-200 responses.
+- No CORS concerns — the site and API share an origin.
diff --git a/docs/api/specs/README.md b/docs/api/specs/README.md
index 3a4b804..ea8546b 100644
--- a/docs/api/specs/README.md
+++ b/docs/api/specs/README.md
@@ -10,13 +10,14 @@ It ingests records through [Tap](https://github.com/bluesky-social/indigo/tree/m
 
 ## Specifications
 
-| # | Document | Description |
-|---|----------|-------------|
-| 1 | [Architecture](01-architecture.md) | Purpose, goals, design principles, system context, tech choices |
-| 2 | [Tangled Lexicons](02-tangled-lexicons.md) | `sh.tangled.*` record schemas and fields |
-| 3 | [Data Model](03-data-model.md) | Database schema, search documents, sync state |
-| 4 | [Data Pipeline](04-data-pipeline.md) | Tap integration, normalization, failure handling |
-| 5 | [Search](05-search.md) | Search modes, API contract, scoring, filtering |
-| 6 | [Operations](06-operations.md) | Configuration, observability, security, deployment |
-| 7 | [Graph Backfill](07-graph-backfill.md) | Seed-based user discovery and content backfill |
-| 8 | [App Integration](08-app-integration.md) | Mobile-facing contracts for search and graph summaries |
+| #   | Document                                   | Description                                                     |
+| --- | ------------------------------------------ | --------------------------------------------------------------- |
+| 1   | [Architecture](01-architecture.md)         | Purpose, goals, design principles, system context, tech choices |
+| 2   | [Tangled Lexicons](02-tangled-lexicons.md) | `sh.tangled.*` record schemas and fields                        |
+| 3   | [Data Model](03-data-model.md)             | Database schema, search documents, sync state                   |
+| 4   | [Data Pipeline](04-data-pipeline.md)       | Tap integration, normalization, failure handling                |
+| 5   | [Search](05-search.md)                     | Search modes, API contract, scoring, filtering                  |
+| 6   | [Operations](06-operations.md)             | Configuration, observability, security, deployment              |
+| 7   | [Graph Backfill](07-graph-backfill.md)     | Seed-based user discovery and content backfill                  |
+| 8   | [App Integration](08-app-integration.md)   | Mobile-facing contracts for search and graph summaries          |
+| 9   | [Search Site](09-search-site.md)           | Static site for API docs and live search                        |
diff --git a/docs/api/tasks/README.md b/docs/api/tasks/README.md
index fb301ae..c0de2e5 100644
--- a/docs/api/tasks/README.md
+++ b/docs/api/tasks/README.md
@@ -38,3 +38,4 @@ Within MVP, run graph backfill before calling the environment search-ready for u
 - Restart does not lose sync position
 - Reindex exists for repair
 - Graph backfill populates initial content from seed users
+- A static search site with API docs is publicly accessible
diff --git a/docs/api/tasks/phase-1-mvp.md b/docs/api/tasks/phase-1-mvp.md
index ff61666..1dded17 100644
--- a/docs/api/tasks/phase-1-mvp.md
+++ b/docs/api/tasks/phase-1-mvp.md
@@ -16,6 +16,7 @@ Get a searchable product online: ingestion, keyword search, deployment, and oper
 - Restart does not lose sync position
 - Reindex exists for repair
 - Graph backfill populates initial content from seed users
+- A static search site with API docs is publicly accessible
 
 ## M0 — Repository Bootstrap ✅
 
@@ -261,6 +262,62 @@ Expose a usable public search API backed by Turso's Tantivy-backed FTS.
 
 A user can search Tangled content reliably with keyword search.
 
+## M5a — Search Site
+
+refs: [specs/09-search-site.md](../specs/09-search-site.md)
+
+### Goal
+
+Ship a static site that doubles as public API documentation and a live search demo. Alpine.js via CDN for reactivity, no build step.
+
+### Deliverables
+
+- `internal/view/` package exporting `Handler() http.Handler`
+- Embedded templates (`templates/`) and static assets (`static/`) via `//go:embed`
+- Search page (`/`) wired to `GET /search` with result cards, filters, and pagination
+- API docs pages (`/docs/*`) covering search, documents, and health endpoints
+- Dark-mode-only styling with Google Sans fonts and minimal CSS tokens
+
+### Tasks
+
+- [ ] Create `internal/view/` package with `view.go`, `templates/`, and `static/` directories
+- [ ] Implement `Handler()` that returns an `http.Handler` with routes for all pages and `/static/*`
+- [ ] Embed templates and static assets via `//go:embed`; parse templates once at init
+- [ ] Use a shared `layout.html` template for the shell (head, nav, footer)
+- [ ] Mount `view.Handler()` in the `api` package router as a fallback after API routes
+- [ ] Build search page:
+  - Text input + submit
+  - Fetch `GET /search` with relative path (same origin)
+  - Render result cards with type badge, title, snippet (preserve ``), author, repo, relative time
+  - "Load more" pagination via offset
+  - Filter bar: type, language, author (reflected in URL query params)
+  - Empty and error states
+- [ ] Build API docs pages:
+  - `/docs` — overview (base URL, response shape, no auth)
+  - `/docs/search` — `GET /search` params, filters, example curl, example response
+  - `/docs/documents` — `GET /documents/{id}` request/response
+  - `/docs/health` — `GET /healthz`, `GET /readyz`
+- [ ] Implement `style.css` with design tokens (`--bg`, `--surface`, `--border`, `--accent`, etc.)
+- [ ] Load Google Sans and Google Sans Mono via Google Fonts ``
+- [ ] Result card links open canonical Tangled URLs in new tab
+- [ ] Verify total site weight under 50 KB (excluding fonts and Alpine CDN)
+
+### Verification
+
+- [ ] `twister api` serves the search page at `http://localhost:8080/`
+- [ ] API endpoints (`/search`, `/healthz`, etc.) still work alongside the site
+- [ ] Searching a known repo name shows it in results
+- [ ] Filter by type restricts results to that type
+- [ ] "Load more" appends next page of results
+- [ ] API docs pages render correct endpoint signatures, parameter tables, and example JSON
+- [ ] Site works on mobile viewport (stacked layout at 640px)
+- [ ] Site works with API unavailable (error state shown, no crash)
+- [ ] All pages share consistent styling and navigation
+
+### Exit Criteria
+
+A user can search Tangled content and read API docs from a public URL without installing anything.
+
 ## M6 — Railway Deployment
 
 refs: [specs/06-operations.md](../specs/06-operations.md)
@@ -336,7 +393,8 @@ Make the system recoverable and operable with repair tools.
   2. For each document, re-run normalization from stored fields (or re-fetch if source available)
   3. Update FTS-relevant fields
   4. Upsert back to store
-  5. Log progress (N/total, errors)
+  5. Run `OPTIMIZE INDEX idx_documents_fts` after bulk reindex to merge Tantivy segments
+  6. Log progress (N/total, errors)
 - [ ] Implement `POST /admin/reindex` endpoint (behind `ENABLE_ADMIN_ENDPOINTS` + `ADMIN_AUTH_TOKEN`)
 - [ ] Add error summary output on completion
 - [ ] Exit non-zero on unrecoverable failures
diff --git a/docs/api/tasks/phase-2-semantic.md b/docs/api/tasks/phase-2-semantic.md
index 11bc72c..0906c0d 100644
--- a/docs/api/tasks/phase-2-semantic.md
+++ b/docs/api/tasks/phase-2-semantic.md
@@ -1,30 +1,37 @@
 ---
 title: "Phase 2 — Semantic Search"
-updated: 2026-03-22
+updated: 2026-03-23
 ---
 
 # Phase 2 — Semantic Search
 
-Add embedding generation and vector-based retrieval on top of the keyword baseline.
+Add embedding generation and vector-based retrieval on top of the keyword baseline, using self-hosted Ollama for embeddings instead of external API services.
 
-## M8 — Embedding Pipeline
+## M8 — Ollama Sidecar and Embedding Pipeline
 
-refs: [specs/03-data-model.md](../specs/03-data-model.md), [specs/05-search.md](../specs/05-search.md)
+refs: [specs/01-architecture.md](../specs/01-architecture.md), [specs/03-data-model.md](../specs/03-data-model.md), [specs/05-search.md](../specs/05-search.md)
 
 ### Goal
 
-Add asynchronous embedding generation without blocking ingestion.
+Deploy Ollama as a Railway sidecar and add asynchronous embedding generation without blocking ingestion.
 
 ### Deliverables
 
+- Ollama Railway service running nomic-embed-text-v1.5 (or EmbeddingGemma)
 - `embedding_jobs` table operational (schema from M1)
 - `embed-worker` subcommand
-- Embedding provider abstraction (OpenAI, Voyage, Ollama)
+- Ollama-backed embedding provider (with interface for future alternatives)
 - Retry and dead-letter behavior
 - `twister reembed` command
 
 ### Tasks
 
+- [ ] Deploy Ollama on Railway:
+  - Use the nomic-embed Railway template as a starting point
+  - Configure as internal service (no public URL)
+  - Pre-pull `nomic-embed-text` model on startup
+  - Health check: `GET /api/tags` on port 11434
+  - Resource budget: 1–2 GB RAM, 1–2 vCPU
 - [ ] Define embedding provider interface:
 
   ```go
@@ -35,34 +42,79 @@ Add asynchronous embedding generation without blocking ingestion.
   }
   ```
 
-- [ ] Implement OpenAI provider (or preferred provider)
+- [ ] Implement Ollama provider using the official Go client:
+
+  ```go
+  import "github.com/ollama/ollama/api"
+
+  // OllamaProvider calls Ollama's /api/embed endpoint
+  // over Railway internal networking (ollama.railway.internal:11434)
+  type OllamaProvider struct {
+      client *api.Client
+      model  string  // "nomic-embed-text"
+      dim    int     // 768
+  }
+  ```
+
+  - Configure via `OLLAMA_URL` env var (default: `http://ollama.railway.internal:11434`)
+  - Support batch embedding (Ollama accepts multiple inputs per request)
+  - Timeout per request (default: 30s)
+  - Connection health check on startup
 - [ ] Implement embedding input text composition (see spec 04-data-pipeline.md, section 5):
   `title\nrepo_name\nauthor_handle\ntags\nsummary\nbody`
 - [ ] Add job enqueueing: on document upsert, insert `embedding_jobs` row with `status=pending`
 - [ ] Implement `embed-worker` loop:
-  1. Poll for `pending` jobs (batch by `EMBEDDING_BATCH_SIZE`)
+  1. Poll for `pending` jobs (batch by `EMBEDDING_BATCH_SIZE`, default: 32)
   2. Compose input text per document
-  3. Call embedding provider
+  3. Call Ollama provider
   4. Store vectors in `document_embeddings` with `vector32(?)`
   5. Mark job `completed`
   6. On failure: increment `attempts`, set `last_error`, backoff
   7. After max attempts: mark `dead`
-- [ ] Create DiskANN vector index: `CREATE INDEX idx_embeddings_vec ON document_embeddings(libsql_vector_idx(embedding, 'metric=cosine'))`
+- [ ] Create DiskANN vector index (see spec 03 for tuning params):
+  ```sql
+  CREATE INDEX idx_embeddings_vec ON document_embeddings(
+      libsql_vector_idx(embedding, 'metric=cosine')
+  );
+  ```
 - [ ] Implement `reembed` command (re-generate all embeddings, useful for model migration)
 - [ ] Skip deleted documents in embedding pipeline
 - [ ] Add health check endpoint for embed-worker (port 9091)
+- [ ] Add Ollama connectivity check to embed-worker readiness probe
+
+### Model Selection Notes
+
+**nomic-embed-text-v1.5** is the default recommendation:
+- 137M parameters, 768-dimension vectors
+- Matryoshka support (can truncate to 64/128/256/512 dims for storage tradeoff)
+- 8192 token context window
+- ~262 MB at F16 quantization, ~500 MB RAM at runtime
+- Battle-tested with llama.cpp/Ollama, Railway template exists
+
+**EmbeddingGemma** is the quality alternative:
+- 308M parameters, 768-dimension vectors
+- Best MTEB scores for models under 500M parameters
+- <200 MB quantized, similar RAM footprint
+- Released Sept 2025, less deployment track record
+
+**all-minilm** is the budget fallback:
+- 23M parameters, 384-dimension vectors (requires schema change)
+- ~46 MB model, minimal resources
+- Suitable for testing or cost-constrained environments
 
 ### Verification
 
+- [ ] Ollama service starts on Railway and responds to health checks
 - [ ] Creating a new searchable document enqueues an embedding job
 - [ ] Worker processes the job and stores a vector in `document_embeddings`
 - [ ] Failed embedding calls retry with bounded attempts
-- [ ] Keyword search still works when embed-worker is down
+- [ ] Keyword search still works when embed-worker or Ollama is down
 - [ ] `reembed` regenerates embeddings for all eligible documents
+- [ ] Ollama connectivity failure is surfaced in embed-worker health check
 
 ### Exit Criteria
 
-Embeddings are produced asynchronously and stored durably.
+Embeddings are produced asynchronously via self-hosted Ollama and stored durably in Turso.
 
 ## M9 — Semantic Search
 
@@ -75,13 +127,14 @@ Expose vector-based semantic retrieval.
 ### Deliverables
 
 - `GET /search/semantic` endpoint
-- Query-time embedding (convert query text → vector)
+- Query-time embedding (convert query text → vector via Ollama)
 - Vector similarity search via `vector_top_k`
 - Response parity with keyword search
 
 ### Tasks
 
-- [ ] Implement query embedding: call embedding provider with user's query text
+- [ ] Implement query embedding: call Ollama provider with user's query text
+- [ ] Cache query embeddings for identical queries within a short TTL (optional, reduces Ollama load)
 - [ ] Implement semantic search repository:
 
   ```sql
@@ -98,6 +151,7 @@ Expose vector-based semantic retrieval.
 - [ ] Add timeout and cost controls (limit vector search to reasonable K)
 - [ ] Wire `/search/semantic` handler
 - [ ] Return `matched_by: ["semantic"]` in results
+- [ ] Graceful degradation: if Ollama is unreachable, return 503 for semantic search while keyword search remains available
 
 ### Verification
 
@@ -106,7 +160,8 @@ Expose vector-based semantic retrieval.
 - [ ] Semantic search returns the same JSON schema as keyword search
 - [ ] Latency is acceptable under small test load
 - [ ] Filters work correctly with semantic results
+- [ ] Semantic search degrades gracefully when Ollama is down
 
 ### Exit Criteria
 
-The API supports true semantic search over Tangled documents.
+The API supports true semantic search over Tangled documents, powered entirely by self-hosted infrastructure.