# embed-stack

Three services that together form a routing embedding API for Qwen3-Embedding-0.6B
on Apple Silicon:

- **orchestrator** (Rust) — tokenizes and routes incoming requests. Short inputs
  (≤ 8192 wrapped tokens) go to a warm Swift/MLX server; long inputs spawn an
  ephemeral Rust/llama.cpp server sized to the exact bucket.
- **mlx-embed-server** (Swift/MLX) — persistently warm, 3 buckets
  `[256, 1024, 8192] × [1]`, ~2 GB idle.
- **llama-embed-server** (Rust/llama.cpp) — ephemeral, spawned by the
  orchestrator for long-doc requests, sized to `ceil(tokens / 64) × 64`.

Both backends use F16 weights so outputs are numerically equivalent.

## Prerequisites

- macOS 14+ with Apple Silicon
- Rust stable (`rustup install stable`)
- Swift 6.1 (Xcode 16+)
- CMake (for the llama.cpp native build)
- A HuggingFace snapshot of `Qwen/Qwen3-Embedding-0.6B` and a
  `Qwen3-Embedding-0.6B-f16.gguf` file locally

## Build

```bash
./scripts/build.sh
```

This runs `cargo build --release --workspace` and `swift build -c release`
inside `mlx-embed-server/`, then copies the Metal libraries next to the Swift
binary.

## Run

```bash
export TOKENIZER_PATH=~/.cache/huggingface/hub/models--Qwen--Qwen3-Embedding-0.6B/snapshots/<hash>/tokenizer.json
export SWIFT_MODEL_DIR=~/.cache/huggingface/hub/models--Qwen--Qwen3-Embedding-0.6B/snapshots/<hash>
export RUST_MODEL_PATH=/path/to/Qwen3-Embedding-0.6B-f16.gguf
./scripts/run-orchestrator.sh
```

Then:

```bash
curl -s -X POST http://localhost:8080/embed \
  -H 'Content-Type: application/json' \
  -d '{"texts":["hello world"],"input_type":"document"}'
```

## Performance notes (M1, 16 GB)

Measured end-to-end through the orchestrator against Qwen3-Embedding-0.6B (F16)
using slices of Project Gutenberg #84 (Frankenstein):

| Request shape                         | Route              | Latency |
|---------------------------------------|--------------------|---------|
| 1 short text (~100 tokens)            | Swift only         | ~0.2 s  |
| 2 mixed short texts (~1k tokens each) | Swift only         | ~0.6 s  |
| 1 long text (~15k tokens)             | Rust spawn → run   | ~29 s   |
| short + long together                 | Swift ∥ Rust       | ~28 s   |
| > 32,768 wrapped tokens               | —                  | HTTP 400 |

Swift cold start (all 3 buckets warmed): ~1.5 s.
Rust spawn cost per long-doc request: ~1 s.

### Idle resource cost

With the orchestrator and Swift backend live and idle (all buckets warmed, no
Rust subprocess):

| Process              | Physical footprint | CPU |
|----------------------|--------------------|-----|
| orchestrator (Rust)  | ~40 MB             | 0%  |
| EmbedServer (Swift)  | ~2.0 GB (wired GPU memory) | 0%  |
| **Total**            | **~2.04 GB**       | 0%  |

The Rust backend contributes **zero** idle cost — it only exists during
long-doc requests and is killed on completion.

## Layout

```
orchestrator/          Rust — public :8080
llama-embed-server/    Rust — ephemeral backend
mlx-embed-server/      Swift — warm backend
scripts/               build + run helpers
```