embed-stack #

Three services that together form a routing embedding API for Qwen3-Embedding-0.6B on Apple Silicon:

• orchestrator (Rust) — tokenizes and routes incoming requests. Short inputs (≤ 8192 wrapped tokens) go to a warm Swift/MLX server; long inputs spawn an ephemeral Rust/llama.cpp server sized to the exact bucket.
• mlx-embed-server (Swift/MLX) — persistently warm, 3 buckets [256, 1024, 8192] × [1], ~2 GB idle.
• llama-embed-server (Rust/llama.cpp) — ephemeral, spawned by the orchestrator for long-doc requests, sized to ceil(tokens / 64) × 64.

Both backends use F16 weights so outputs are numerically equivalent.

Prerequisites #

• macOS 14+ with Apple Silicon
• Rust stable (rustup install stable)
• Swift 6.1 (Xcode 16+)
• CMake (for the llama.cpp native build)
• A HuggingFace snapshot of Qwen/Qwen3-Embedding-0.6B and a Qwen3-Embedding-0.6B-f16.gguf file locally

Build #

./scripts/build.sh

This runs cargo build --release --workspace and swift build -c release inside mlx-embed-server/, then copies the Metal libraries next to the Swift binary.

Run #

export TOKENIZER_PATH=~/.cache/huggingface/hub/models--Qwen--Qwen3-Embedding-0.6B/snapshots/<hash>/tokenizer.json
export SWIFT_MODEL_DIR=~/.cache/huggingface/hub/models--Qwen--Qwen3-Embedding-0.6B/snapshots/<hash>
export RUST_MODEL_PATH=/path/to/Qwen3-Embedding-0.6B-f16.gguf
./scripts/run-orchestrator.sh

Then:

curl -s -X POST http://localhost:8080/embed \
  -H 'Content-Type: application/json' \
  -d '{"texts":["hello world"],"input_type":"document"}'

Performance notes (M1, 16 GB) #

Measured end-to-end through the orchestrator against Qwen3-Embedding-0.6B (F16) using slices of Project Gutenberg #84 (Frankenstein):

Swift cold start (all 3 buckets warmed): ~1.5 s. Rust spawn cost per long-doc request: ~1 s.

Idle resource cost #

With the orchestrator and Swift backend live and idle (all buckets warmed, no Rust subprocess):

The Rust backend contributes zero idle cost — it only exists during long-doc requests and is killed on completion.

Layout #

orchestrator/          Rust — public :8080
llama-embed-server/    Rust — ephemeral backend
mlx-embed-server/      Swift — warm backend
scripts/               build + run helpers