diff --git a/docs/api/specs/01-architecture.md b/docs/api/specs/01-architecture.md
index 6eec7e8..2dbc76c 100644
--- a/docs/api/specs/01-architecture.md
+++ b/docs/api/specs/01-architecture.md
@@ -64,8 +64,8 @@ The system shall prioritize:
- **AT Protocol network** — source of all Tangled content
- **Tap** — filtered event delivery from the AT Protocol firehose (deployed on Railway)
- **Turso/libSQL** — relational storage, Tantivy-backed FTS, and native vector search
-- **Embedding provider** — generates vectors for semantic search
-- **Railway** — deployment platform for Twister services and Tap
+- **Ollama** — local embedding model server (nomic-embed-text or EmbeddingGemma); deployed as a Railway sidecar service
+- **Railway** — deployment platform for Twister services, Tap, and Ollama
## 7. Architecture Summary
@@ -106,7 +106,8 @@ ATProto Firehose / PDS
| -------------- | -------------------------------------------- | -------------------------- |
| `api` | HTTP search, graph summary, and document API | Railway service (public) |
| `indexer` | Tap consumer, normalizer, DB writer | Railway service (internal) |
-| `embed-worker` | Async embedding generation | Optional Railway service |
+| `embed-worker` | Async embedding generation via Ollama | Optional Railway service |
+| `ollama` | Local embedding model server | Railway service (internal) |
| `tap` | ATProto sync | Railway (already deployed) |
## 9. Repository Structure
@@ -142,6 +143,22 @@ twister healthcheck # One-shot health probe
## 11. Technology Choices
+### Embedding: Ollama (self-hosted)
+
+Embeddings are generated locally via Ollama rather than an external API service. This eliminates per-token costs, external service dependencies, and data egress concerns.
+
+**Recommended models (in order of preference):**
+
+| Model | Parameters | Dimensions | Quantized Size | Notes |
+|-------|-----------|------------|----------------|-------|
+| nomic-embed-text-v1.5 | 137M | 768 (Matryoshka: 64–768) | ~262 MB (F16) | 8192 context, battle-tested, Railway template exists |
+| EmbeddingGemma | 308M | 768 | <200 MB (quantized) | Best-in-class MTEB for size, released Sept 2025 |
+| all-minilm | 23M | 384 | ~46 MB | Budget option, lower quality |
+
+**Go integration:** Use the official Ollama Go client (`github.com/ollama/ollama/api`) with the `Embed()` method. The embed-worker calls Ollama over Railway's internal network (`ollama.railway.internal:11434`).
+
+**Railway deployment:** Ollama runs as a separate Railway service (~1–2 GB RAM, 1–2 vCPU, ~$10–30/mo). The nomic-embed Railway template provides a proven starting point. No cold starts on always-on services; model loads in 2–10 seconds on first request after deploy.
+
### Language: Go
Go is the implementation language for the API server, indexer, embedding worker, and CLI commands. Rationale: straightforward long-running services, excellent HTTP support, good concurrency model, small container footprint.
diff --git a/docs/api/specs/03-data-model.md b/docs/api/specs/03-data-model.md
index 49f9f76..714de4f 100644
--- a/docs/api/specs/03-data-model.md
+++ b/docs/api/specs/03-data-model.md
@@ -93,6 +93,17 @@ CREATE INDEX idx_documents_fts ON documents USING fts (
) WITH (weights='title=3.0,repo_name=2.5,author_handle=2.0,summary=1.5,tags_json=1.2,body=1.0');
```
+### FTS Maintenance
+
+Turso's Tantivy-backed FTS uses `NoMergePolicy` — segment count grows with writes and is never automatically compacted. This increases query fan-out over time.
+
+**Required maintenance:** Run `OPTIMIZE INDEX idx_documents_fts;` periodically (e.g., daily cron or after bulk backfill). This merges segments and reclaims space.
+
+**Known limitations:**
+- No read-your-writes within a transaction — FTS queries see a pre-commit snapshot
+- No snippet function (use `fts_highlight()` for highlighting)
+- FTS is experimental in Turso; requires the `fts` feature flag
+
## 4. Embeddings Table
```sql
@@ -108,7 +119,27 @@ CREATE INDEX idx_embeddings_vec ON document_embeddings(
);
```
-The vector dimension (768) is configurable by model. Changing models requires a new column or table migration.
+The vector dimension (768) matches nomic-embed-text-v1.5 and EmbeddingGemma defaults. Changing models may require a new column or table migration if the dimension changes.
+
+### Vector Index Tuning
+
+The DiskANN index accepts tuning parameters at creation time:
+
+```sql
+CREATE INDEX idx_embeddings_vec ON document_embeddings(
+ libsql_vector_idx(embedding, 'metric=cosine', 'max_neighbors=50', 'search_l=200')
+);
+```
+
+| Parameter | Default | Description |
+|-----------|---------|-------------|
+| `max_neighbors` | 3*sqrt(D) | Graph connectivity; higher = better recall, more storage |
+| `search_l` | 200 | Neighbors visited during search; higher = better recall, slower |
+| `insert_l` | 70 | Neighbors visited during insert |
+| `alpha` | 1.2 | Graph sparsity factor |
+| `compress_neighbors` | — | Quantize neighbor vectors for storage savings |
+
+Start with defaults and tune after measuring recall on representative queries.
## 5. Sync State Table
diff --git a/docs/api/specs/04-data-pipeline.md b/docs/api/specs/04-data-pipeline.md
index 08bdec0..e2f5447 100644
--- a/docs/api/specs/04-data-pipeline.md
+++ b/docs/api/specs/04-data-pipeline.md
@@ -324,6 +324,7 @@ If embedding generation fails:
- Jobs are retried with exponential backoff up to a max attempt count
- After max attempts, the job enters `dead` state
- The embed-worker exposes failed job count as a metric
+- If Ollama is unreachable (sidecar down), all pending jobs pause until connectivity is restored
### DB Failures
diff --git a/docs/api/specs/05-search.md b/docs/api/specs/05-search.md
index 5d38308..6915416 100644
--- a/docs/api/specs/05-search.md
+++ b/docs/api/specs/05-search.md
@@ -61,11 +61,17 @@ Use `fts_highlight()` to generate highlighted snippets:
fts_highlight(d.body, '', '', ?) AS body_snippet
```
+### FTS Operational Notes
+
+- **Segment merging:** Turso FTS uses Tantivy's `NoMergePolicy`. Run `OPTIMIZE INDEX idx_documents_fts;` after bulk writes (backfill) and periodically in production to keep query performance stable.
+- **Read-your-writes:** FTS queries within the same transaction see a pre-commit snapshot. If a document is written and immediately searched in the same transaction, FTS will not find it. The indexer and API are separate processes, so this is not a concern in normal operation.
+- **Feature flag:** Turso FTS requires the `fts` feature flag to be enabled on the database.
+
## 3. Semantic Search
### Query Flow
-1. Convert user query text to embedding via the configured provider
+1. Convert user query text to embedding via Ollama (self-hosted)
2. Query `vector_top_k` for nearest neighbors
3. Join back to `documents` to get metadata
4. Filter out deleted/hidden documents
diff --git a/docs/api/specs/06-operations.md b/docs/api/specs/06-operations.md
index ab039f6..cfbab69 100644
--- a/docs/api/specs/06-operations.md
+++ b/docs/api/specs/06-operations.md
@@ -41,16 +41,14 @@ All configuration is via environment variables.
| `SEARCH_MAX_LIMIT` | `100` | Maximum results per page |
| `SEARCH_DEFAULT_MODE` | `keyword` | Default search mode |
-### Embedding
+### Embedding (Ollama — self-hosted)
-| Variable | Default | Description |
-| ---------------------- | ------- | ---------------------------------------------------- |
-| `EMBEDDING_PROVIDER` | — | Provider name (e.g., `openai`, `ollama`, `voyageai`) |
-| `EMBEDDING_MODEL` | — | Model name (e.g., `text-embedding-3-small`) |
-| `EMBEDDING_API_KEY` | — | Provider API key |
-| `EMBEDDING_API_URL` | — | Provider base URL (for self-hosted) |
-| `EMBEDDING_DIM` | `768` | Vector dimensionality |
-| `EMBEDDING_BATCH_SIZE` | `32` | Batch size for embed-worker |
+| Variable | Default | Description |
+| ---------------------- | ------------------------------------------ | ---------------------------------------------- |
+| `OLLAMA_URL` | `http://ollama.railway.internal:11434` | Ollama server URL |
+| `EMBEDDING_MODEL` | `nomic-embed-text` | Ollama model name |
+| `EMBEDDING_DIM` | `768` | Vector dimensionality (must match model) |
+| `EMBEDDING_BATCH_SIZE` | `32` | Documents per embedding batch |
### Hybrid Search
@@ -87,10 +85,9 @@ INDEXED_COLLECTIONS=sh.tangled.repo,sh.tangled.repo.issue,sh.tangled.repo.pull,s
SEARCH_DEFAULT_LIMIT=20
SEARCH_MAX_LIMIT=100
-# Embedding (Phase 2)
-# EMBEDDING_PROVIDER=openai
-# EMBEDDING_MODEL=text-embedding-3-small
-# EMBEDDING_API_KEY=sk-...
+# Embedding — Ollama (Phase 2)
+# OLLAMA_URL=http://ollama.railway.internal:11434
+# EMBEDDING_MODEL=nomic-embed-text
# EMBEDDING_DIM=768
# Server
@@ -285,7 +282,7 @@ Required secrets:
| ------------------- | --------------------------------- |
| `TURSO_AUTH_TOKEN` | Turso database authentication |
| `TAP_AUTH_PASSWORD` | Tap admin API authentication |
-| `EMBEDDING_API_KEY` | Embedding provider authentication |
+| `OLLAMA_URL` | Ollama sidecar connection (no secret if internal networking) |
| `ADMIN_AUTH_TOKEN` | Admin endpoint authentication |
### Admin Endpoints
@@ -331,6 +328,7 @@ All Twister services deploy as separate Railway services within the same project
| api | `twister api` | `GET /healthz` | yes |
| indexer | `twister indexer` | `GET :9090/health` | no |
| embed-worker | `twister embed-worker` | `GET :9091/health` | no |
+| ollama | (Railway template) | `GET /api/tags` | no |
All services share the same Docker image. Railway uses the start command to select the subcommand.
@@ -356,10 +354,9 @@ TAP_URL=wss://${{tap.RAILWAY_PUBLIC_DOMAIN}}/channel # Railway service referenc
TAP_AUTH_PASSWORD=...
INDEXED_COLLECTIONS=sh.tangled.repo,sh.tangled.repo.issue,sh.tangled.repo.pull,sh.tangled.string,sh.tangled.actor.profile
-# Embed-worker (Phase 2)
-# EMBEDDING_PROVIDER=openai
-# EMBEDDING_MODEL=text-embedding-3-small
-# EMBEDDING_API_KEY=sk-...
+# Embed-worker + Ollama (Phase 2)
+# OLLAMA_URL=http://ollama.railway.internal:11434
+# EMBEDDING_MODEL=nomic-embed-text
```
Railway supports referencing other services' variables with `${{service.VAR}}` syntax, which is useful for linking the indexer to Tap's domain.
diff --git a/docs/api/specs/09-search-site.md b/docs/api/specs/09-search-site.md
new file mode 100644
index 0000000..4a9b71d
--- /dev/null
+++ b/docs/api/specs/09-search-site.md
@@ -0,0 +1,166 @@
+---
+title: "Spec 09 — Search Site"
+updated: 2026-03-23
+---
+
+A minimal static site that serves as both the public Twister API documentation and a live search showcase. Dark mode only, no framework or build step.
+
+## 1. Purpose
+
+- Give developers a browsable reference for the Twister search API
+- Give anyone a way to try search against live indexed Tangled content
+- Provide a shareable public URL before the mobile app ships
+
+## 2. Scope
+
+In scope:
+
+- Static HTML/CSS/JS (Alpine.js, no bundler)
+- API reference pages generated from the spec docs
+- Live search input wired to `GET /search`
+- Result rendering with type-aware cards (repo, issue, PR, profile, string)
+- Filter controls for collection, type, author, language, state
+- Pagination
+- Responsive layout (mobile-friendly, single breakpoint)
+
+Out of scope:
+
+- Auth, OAuth, or any write operations
+- Semantic or hybrid mode toggle (keyword only for MVP)
+- Server-side rendering or static-site generator
+- Analytics or telemetry
+
+## 3. Pages
+
+| Route | Content |
+| ----------------- | --------------------------------------------------------------------------- |
+| `/` | Search input + results (the homepage is the search page) |
+| `/docs` | API overview: base URL, auth (none for public), rate limits, response shape |
+| `/docs/search` | `GET /search` — parameters, filters, response contract, examples |
+| `/docs/documents` | `GET /documents/{id}` — request/response, examples |
+| `/docs/health` | `GET /healthz`, `GET /readyz` — purpose and expected responses |
+
+## 4. Search Page Behavior
+
+1. Text input with a submit button. No debounce search-as-you-type for MVP.
+2. On submit, fetch `GET {API_BASE}/search?q={query}&limit=20` (plus any active filters).
+3. Render results as a vertical list of cards.
+4. Each card shows: `record_type` badge, `title`, `body_snippet` (with `` highlights preserved), `author_handle`, `repo_name` (when present), `updated_at` relative time.
+5. Clicking a result opens the canonical Tangled URL (`https://tangled.org/{handle}/{repo}` for repos, etc.) in a new tab.
+6. "Load more" button appends the next page (`offset += limit`).
+7. Empty state: "No results" message.
+8. Error state: inline message if the API is unreachable.
+9. Filter bar above results: dropdowns/inputs for `type`, `language`, `author`. Filters are query params so URLs are shareable.
+
+## 5. API Docs Pages
+
+Hand-written HTML mirroring the contracts in spec 05 (search) and spec 08 (app integration). Each page includes:
+
+- Endpoint signature (method, path)
+- Parameter table (name, type, default, description)
+- Example request (curl)
+- Example response (JSON block with syntax highlighting via `
`)
+
+No generated docs tooling. The pages are static and updated manually when the API changes.
+
+## 6. Styling
+
+Minimal CSS, no utility framework.
+
+### Tokens
+
+```css
+:root {
+ --bg: #0e0e0e;
+ --surface: #1a1a1a;
+ --border: #2a2a2a;
+ --text: #e0e0e0;
+ --text-dim: #888;
+ --accent: #7aa2f7;
+ --mark-bg: #7aa2f733;
+ --mono: "Google Sans Mono", monospace;
+ --sans: "Google Sans", sans-serif;
+ --radius: 6px;
+}
+```
+
+### Rules
+
+- Dark theming.
+- `Google Sans` for body text. `Google Sans Mono` for code, JSON, and badges.
+- Fonts loaded via Google Fonts ``. System fallbacks: `sans-serif`, `monospace`.
+- Max content width: `720px`, centered.
+- Cards: `var(--surface)` background, `var(--border)` border, `var(--radius)` corners.
+- `` tags in snippets styled with `var(--mark-bg)` background and `var(--accent)` text.
+- Code blocks: `var(--surface)` background, horizontal scroll, no wrapping.
+- Links: `var(--accent)`, no underline, underline on hover.
+- Inputs and buttons: `var(--surface)` background, `var(--border)` border, `var(--text)` text.
+- One breakpoint at `640px` for mobile: full-width cards, stacked filter bar.
+
+## 7. Package Design
+
+The site lives in `internal/view/` as a self-contained Go package. It owns the templates, static assets, and HTTP handlers. The `api` package mounts `view.Handler()` into its router — nothing else leaks out.
+
+### Exports
+
+The package exposes a single constructor:
+
+```go
+// Handler returns an http.Handler that serves the site pages and static assets.
+func Handler() http.Handler
+```
+
+The `api` package calls `view.Handler()` and mounts it as a fallback after API routes.
+
+### Package Structure
+
+```text
+internal/view/
+ view.go # Handler(), route setup, embed directives
+ templates/
+ layout.html # Shared shell (head, nav, footer)
+ index.html # Search page
+ docs/
+ index.html # API overview
+ search.html # GET /search docs
+ documents.html # GET /documents/{id} docs
+ health.html # Health endpoints docs
+ static/
+ style.css # All styles, single file
+ search.js # Search fetch, render, pagination, filters
+```
+
+### Embedding
+
+`view.go` uses `//go:embed` to bundle `templates/` and `static/`. Templates are parsed once at init. Static assets are served under `/static/` via `http.FileServer`.
+
+### Routing
+
+`view.Handler()` returns a mux that handles:
+
+| Pattern | Handler |
+| --- | --- |
+| `GET /` | Render `index.html` |
+| `GET /docs` | Render `docs/index.html` |
+| `GET /docs/search` | Render `docs/search.html` |
+| `GET /docs/documents` | Render `docs/documents.html` |
+| `GET /docs/health` | Render `docs/health.html` |
+| `GET /static/*` | Serve embedded CSS/JS files |
+
+## 9. Configuration
+
+Since the site is served by the same origin as the API, search requests use relative paths (`/search?q=...`). No `API_BASE` config needed — the browser's origin is the API.
+
+## 10. Local Development
+
+Run `twister api` locally. The site is served at `http://localhost:8080/` alongside the API endpoints. No separate dev server or file server required.
+
+The API docs pages render without any indexed data. The search page needs a running indexer and populated database to return results.
+
+## 11. Constraints
+
+- No dependencies besides Alpine via CDN.
+- Total site weight target: under 50 KB excluding fonts.
+- Works in modern browsers (last 2 versions of Chrome, Firefox, Safari).
+- All fetch calls include error handling for network failures and non-200 responses.
+- No CORS concerns — the site and API share an origin.
diff --git a/docs/api/specs/README.md b/docs/api/specs/README.md
index 3a4b804..ea8546b 100644
--- a/docs/api/specs/README.md
+++ b/docs/api/specs/README.md
@@ -10,13 +10,14 @@ It ingests records through [Tap](https://github.com/bluesky-social/indigo/tree/m
## Specifications
-| # | Document | Description |
-|---|----------|-------------|
-| 1 | [Architecture](01-architecture.md) | Purpose, goals, design principles, system context, tech choices |
-| 2 | [Tangled Lexicons](02-tangled-lexicons.md) | `sh.tangled.*` record schemas and fields |
-| 3 | [Data Model](03-data-model.md) | Database schema, search documents, sync state |
-| 4 | [Data Pipeline](04-data-pipeline.md) | Tap integration, normalization, failure handling |
-| 5 | [Search](05-search.md) | Search modes, API contract, scoring, filtering |
-| 6 | [Operations](06-operations.md) | Configuration, observability, security, deployment |
-| 7 | [Graph Backfill](07-graph-backfill.md) | Seed-based user discovery and content backfill |
-| 8 | [App Integration](08-app-integration.md) | Mobile-facing contracts for search and graph summaries |
+| # | Document | Description |
+| --- | ------------------------------------------ | --------------------------------------------------------------- |
+| 1 | [Architecture](01-architecture.md) | Purpose, goals, design principles, system context, tech choices |
+| 2 | [Tangled Lexicons](02-tangled-lexicons.md) | `sh.tangled.*` record schemas and fields |
+| 3 | [Data Model](03-data-model.md) | Database schema, search documents, sync state |
+| 4 | [Data Pipeline](04-data-pipeline.md) | Tap integration, normalization, failure handling |
+| 5 | [Search](05-search.md) | Search modes, API contract, scoring, filtering |
+| 6 | [Operations](06-operations.md) | Configuration, observability, security, deployment |
+| 7 | [Graph Backfill](07-graph-backfill.md) | Seed-based user discovery and content backfill |
+| 8 | [App Integration](08-app-integration.md) | Mobile-facing contracts for search and graph summaries |
+| 9 | [Search Site](09-search-site.md) | Static site for API docs and live search |
diff --git a/docs/api/tasks/README.md b/docs/api/tasks/README.md
index fb301ae..c0de2e5 100644
--- a/docs/api/tasks/README.md
+++ b/docs/api/tasks/README.md
@@ -38,3 +38,4 @@ Within MVP, run graph backfill before calling the environment search-ready for u
- Restart does not lose sync position
- Reindex exists for repair
- Graph backfill populates initial content from seed users
+- A static search site with API docs is publicly accessible
diff --git a/docs/api/tasks/phase-1-mvp.md b/docs/api/tasks/phase-1-mvp.md
index ff61666..1dded17 100644
--- a/docs/api/tasks/phase-1-mvp.md
+++ b/docs/api/tasks/phase-1-mvp.md
@@ -16,6 +16,7 @@ Get a searchable product online: ingestion, keyword search, deployment, and oper
- Restart does not lose sync position
- Reindex exists for repair
- Graph backfill populates initial content from seed users
+- A static search site with API docs is publicly accessible
## M0 — Repository Bootstrap ✅
@@ -261,6 +262,62 @@ Expose a usable public search API backed by Turso's Tantivy-backed FTS.
A user can search Tangled content reliably with keyword search.
+## M5a — Search Site
+
+refs: [specs/09-search-site.md](../specs/09-search-site.md)
+
+### Goal
+
+Ship a static site that doubles as public API documentation and a live search demo. Alpine.js via CDN for reactivity, no build step.
+
+### Deliverables
+
+- `internal/view/` package exporting `Handler() http.Handler`
+- Embedded templates (`templates/`) and static assets (`static/`) via `//go:embed`
+- Search page (`/`) wired to `GET /search` with result cards, filters, and pagination
+- API docs pages (`/docs/*`) covering search, documents, and health endpoints
+- Dark-mode-only styling with Google Sans fonts and minimal CSS tokens
+
+### Tasks
+
+- [ ] Create `internal/view/` package with `view.go`, `templates/`, and `static/` directories
+- [ ] Implement `Handler()` that returns an `http.Handler` with routes for all pages and `/static/*`
+- [ ] Embed templates and static assets via `//go:embed`; parse templates once at init
+- [ ] Use a shared `layout.html` template for the shell (head, nav, footer)
+- [ ] Mount `view.Handler()` in the `api` package router as a fallback after API routes
+- [ ] Build search page:
+ - Text input + submit
+ - Fetch `GET /search` with relative path (same origin)
+ - Render result cards with type badge, title, snippet (preserve ``), author, repo, relative time
+ - "Load more" pagination via offset
+ - Filter bar: type, language, author (reflected in URL query params)
+ - Empty and error states
+- [ ] Build API docs pages:
+ - `/docs` — overview (base URL, response shape, no auth)
+ - `/docs/search` — `GET /search` params, filters, example curl, example response
+ - `/docs/documents` — `GET /documents/{id}` request/response
+ - `/docs/health` — `GET /healthz`, `GET /readyz`
+- [ ] Implement `style.css` with design tokens (`--bg`, `--surface`, `--border`, `--accent`, etc.)
+- [ ] Load Google Sans and Google Sans Mono via Google Fonts ``
+- [ ] Result card links open canonical Tangled URLs in new tab
+- [ ] Verify total site weight under 50 KB (excluding fonts and Alpine CDN)
+
+### Verification
+
+- [ ] `twister api` serves the search page at `http://localhost:8080/`
+- [ ] API endpoints (`/search`, `/healthz`, etc.) still work alongside the site
+- [ ] Searching a known repo name shows it in results
+- [ ] Filter by type restricts results to that type
+- [ ] "Load more" appends next page of results
+- [ ] API docs pages render correct endpoint signatures, parameter tables, and example JSON
+- [ ] Site works on mobile viewport (stacked layout at 640px)
+- [ ] Site works with API unavailable (error state shown, no crash)
+- [ ] All pages share consistent styling and navigation
+
+### Exit Criteria
+
+A user can search Tangled content and read API docs from a public URL without installing anything.
+
## M6 — Railway Deployment
refs: [specs/06-operations.md](../specs/06-operations.md)
@@ -336,7 +393,8 @@ Make the system recoverable and operable with repair tools.
2. For each document, re-run normalization from stored fields (or re-fetch if source available)
3. Update FTS-relevant fields
4. Upsert back to store
- 5. Log progress (N/total, errors)
+ 5. Run `OPTIMIZE INDEX idx_documents_fts` after bulk reindex to merge Tantivy segments
+ 6. Log progress (N/total, errors)
- [ ] Implement `POST /admin/reindex` endpoint (behind `ENABLE_ADMIN_ENDPOINTS` + `ADMIN_AUTH_TOKEN`)
- [ ] Add error summary output on completion
- [ ] Exit non-zero on unrecoverable failures
diff --git a/docs/api/tasks/phase-2-semantic.md b/docs/api/tasks/phase-2-semantic.md
index 11bc72c..0906c0d 100644
--- a/docs/api/tasks/phase-2-semantic.md
+++ b/docs/api/tasks/phase-2-semantic.md
@@ -1,30 +1,37 @@
---
title: "Phase 2 — Semantic Search"
-updated: 2026-03-22
+updated: 2026-03-23
---
# Phase 2 — Semantic Search
-Add embedding generation and vector-based retrieval on top of the keyword baseline.
+Add embedding generation and vector-based retrieval on top of the keyword baseline, using self-hosted Ollama for embeddings instead of external API services.
-## M8 — Embedding Pipeline
+## M8 — Ollama Sidecar and Embedding Pipeline
-refs: [specs/03-data-model.md](../specs/03-data-model.md), [specs/05-search.md](../specs/05-search.md)
+refs: [specs/01-architecture.md](../specs/01-architecture.md), [specs/03-data-model.md](../specs/03-data-model.md), [specs/05-search.md](../specs/05-search.md)
### Goal
-Add asynchronous embedding generation without blocking ingestion.
+Deploy Ollama as a Railway sidecar and add asynchronous embedding generation without blocking ingestion.
### Deliverables
+- Ollama Railway service running nomic-embed-text-v1.5 (or EmbeddingGemma)
- `embedding_jobs` table operational (schema from M1)
- `embed-worker` subcommand
-- Embedding provider abstraction (OpenAI, Voyage, Ollama)
+- Ollama-backed embedding provider (with interface for future alternatives)
- Retry and dead-letter behavior
- `twister reembed` command
### Tasks
+- [ ] Deploy Ollama on Railway:
+ - Use the nomic-embed Railway template as a starting point
+ - Configure as internal service (no public URL)
+ - Pre-pull `nomic-embed-text` model on startup
+ - Health check: `GET /api/tags` on port 11434
+ - Resource budget: 1–2 GB RAM, 1–2 vCPU
- [ ] Define embedding provider interface:
```go
@@ -35,34 +42,79 @@ Add asynchronous embedding generation without blocking ingestion.
}
```
-- [ ] Implement OpenAI provider (or preferred provider)
+- [ ] Implement Ollama provider using the official Go client:
+
+ ```go
+ import "github.com/ollama/ollama/api"
+
+ // OllamaProvider calls Ollama's /api/embed endpoint
+ // over Railway internal networking (ollama.railway.internal:11434)
+ type OllamaProvider struct {
+ client *api.Client
+ model string // "nomic-embed-text"
+ dim int // 768
+ }
+ ```
+
+ - Configure via `OLLAMA_URL` env var (default: `http://ollama.railway.internal:11434`)
+ - Support batch embedding (Ollama accepts multiple inputs per request)
+ - Timeout per request (default: 30s)
+ - Connection health check on startup
- [ ] Implement embedding input text composition (see spec 04-data-pipeline.md, section 5):
`title\nrepo_name\nauthor_handle\ntags\nsummary\nbody`
- [ ] Add job enqueueing: on document upsert, insert `embedding_jobs` row with `status=pending`
- [ ] Implement `embed-worker` loop:
- 1. Poll for `pending` jobs (batch by `EMBEDDING_BATCH_SIZE`)
+ 1. Poll for `pending` jobs (batch by `EMBEDDING_BATCH_SIZE`, default: 32)
2. Compose input text per document
- 3. Call embedding provider
+ 3. Call Ollama provider
4. Store vectors in `document_embeddings` with `vector32(?)`
5. Mark job `completed`
6. On failure: increment `attempts`, set `last_error`, backoff
7. After max attempts: mark `dead`
-- [ ] Create DiskANN vector index: `CREATE INDEX idx_embeddings_vec ON document_embeddings(libsql_vector_idx(embedding, 'metric=cosine'))`
+- [ ] Create DiskANN vector index (see spec 03 for tuning params):
+ ```sql
+ CREATE INDEX idx_embeddings_vec ON document_embeddings(
+ libsql_vector_idx(embedding, 'metric=cosine')
+ );
+ ```
- [ ] Implement `reembed` command (re-generate all embeddings, useful for model migration)
- [ ] Skip deleted documents in embedding pipeline
- [ ] Add health check endpoint for embed-worker (port 9091)
+- [ ] Add Ollama connectivity check to embed-worker readiness probe
+
+### Model Selection Notes
+
+**nomic-embed-text-v1.5** is the default recommendation:
+- 137M parameters, 768-dimension vectors
+- Matryoshka support (can truncate to 64/128/256/512 dims for storage tradeoff)
+- 8192 token context window
+- ~262 MB at F16 quantization, ~500 MB RAM at runtime
+- Battle-tested with llama.cpp/Ollama, Railway template exists
+
+**EmbeddingGemma** is the quality alternative:
+- 308M parameters, 768-dimension vectors
+- Best MTEB scores for models under 500M parameters
+- <200 MB quantized, similar RAM footprint
+- Released Sept 2025, less deployment track record
+
+**all-minilm** is the budget fallback:
+- 23M parameters, 384-dimension vectors (requires schema change)
+- ~46 MB model, minimal resources
+- Suitable for testing or cost-constrained environments
### Verification
+- [ ] Ollama service starts on Railway and responds to health checks
- [ ] Creating a new searchable document enqueues an embedding job
- [ ] Worker processes the job and stores a vector in `document_embeddings`
- [ ] Failed embedding calls retry with bounded attempts
-- [ ] Keyword search still works when embed-worker is down
+- [ ] Keyword search still works when embed-worker or Ollama is down
- [ ] `reembed` regenerates embeddings for all eligible documents
+- [ ] Ollama connectivity failure is surfaced in embed-worker health check
### Exit Criteria
-Embeddings are produced asynchronously and stored durably.
+Embeddings are produced asynchronously via self-hosted Ollama and stored durably in Turso.
## M9 — Semantic Search
@@ -75,13 +127,14 @@ Expose vector-based semantic retrieval.
### Deliverables
- `GET /search/semantic` endpoint
-- Query-time embedding (convert query text → vector)
+- Query-time embedding (convert query text → vector via Ollama)
- Vector similarity search via `vector_top_k`
- Response parity with keyword search
### Tasks
-- [ ] Implement query embedding: call embedding provider with user's query text
+- [ ] Implement query embedding: call Ollama provider with user's query text
+- [ ] Cache query embeddings for identical queries within a short TTL (optional, reduces Ollama load)
- [ ] Implement semantic search repository:
```sql
@@ -98,6 +151,7 @@ Expose vector-based semantic retrieval.
- [ ] Add timeout and cost controls (limit vector search to reasonable K)
- [ ] Wire `/search/semantic` handler
- [ ] Return `matched_by: ["semantic"]` in results
+- [ ] Graceful degradation: if Ollama is unreachable, return 503 for semantic search while keyword search remains available
### Verification
@@ -106,7 +160,8 @@ Expose vector-based semantic retrieval.
- [ ] Semantic search returns the same JSON schema as keyword search
- [ ] Latency is acceptable under small test load
- [ ] Filters work correctly with semantic results
+- [ ] Semantic search degrades gracefully when Ollama is down
### Exit Criteria
-The API supports true semantic search over Tangled documents.
+The API supports true semantic search over Tangled documents, powered entirely by self-hosted infrastructure.