GemTranslate #

A drop-in, API-compatible replacement for LibreTranslate powered by Google's TranslateGemma, inferred in-process via llama-cpp-python.

• FastAPI-based async REST API with LibreTranslate-compatible routes, request parameters, response shapes, error messages and status codes
• Translation work is funneled through a single internal worker on a bounded queue, so the LLM is never overloaded and the event loop is never blocked
• API keys (optional, required-or-not, per-key char limits)
• IP blocklist (IPs and CIDR ranges, X-Forwarded-For aware)
• YAML configuration, including the list of supported languages
• Bundled web UI (LibreTranslate-style translate page)

Quick start #

uv sync
# edit config.yaml (model path, languages, keys, blocklist, ...)
uv run gemtranslate --config config.yaml

Then:

curl http://localhost:5000/languages
curl -X POST http://localhost:5000/translate \
  -H 'Content-Type: application/json' \
  -d '{"q": "Hello world", "source": "en", "target": "de"}'

The server also serves a small web UI at /, Swagger UI at /docs, ReDoc at /redoc, and the OpenAPI document at /spec and /openapi.json.

The model is loaded in-process at startup, so do not run multiple worker processes (e.g. gunicorn -w N); use a single uvicorn process behind your load balancer if you need scale-out (each process loads its own model copy).

Model #

Any TranslateGemma GGUF works (google/translategemma-4b-it or google/translategemma-12b-it, quantized with llama.cpp's convert_hf_to_gguf.py + llama-quantize). Put the file on disk and point model.path at it (relative paths are resolved against the config file).

Recommended settings:

setting	CPU (4B)	GPU (12B)
n_gpu_layers	0	-1
n_threads	0 (auto)	0 (auto)
n_ctx	8192	8192+

GPU acceleration (Vulkan) #

By default uv sync installs the prebuilt CPU-only wheel of llama-cpp-python. To run on a Vulkan GPU (AMD, Intel or NVIDIA), rebuild that one package from source with the Vulkan backend:

# one-time build dependencies (Debian/Ubuntu)
sudo apt install build-essential cmake ninja-build libvulkan-dev

# sanity check: a real GPU must show up (not just llvmpipe)
vulkaninfo --summary

# rebuild llama-cpp-python with the Vulkan backend
CMAKE_ARGS="-DGGML_VULKAN=ON" uv pip install \
    --no-binary llama-cpp-python --reinstall --no-cache \
    llama-cpp-python==0.3.35

Then set model.n_gpu_layers: -1 in config.yaml (or a smaller number to keep a few layers on the CPU). A working Vulkan build announces itself at startup:

ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Graphics (RADV GFX1151) (radv) | uma: 1 | fp16: 1 | ...
llama_prepare_model_devices: using device Vulkan0 (AMD Radeon Graphics (RADV GFX1151))
load_tensors: layer   0 assigned to device Vulkan0, ...

Notes:

• --no-binary is mandatory — without it uv reinstalls the prebuilt CPU wheel. --no-cache matters too, because uv's cached sdist build is not keyed on CMAKE_ARGS; toggling build flags without it can silently reuse the old CPU build.
• Back to CPU: uv sync (restores the prebuilt wheel) and n_gpu_layers: 0.
• With several GPUs, GGML_VULKAN_DEVICE=<n> selects which one llama.cpp uses.
• If n_gpu_layers: -1 fails on your GPU/driver, keep 1–2 layers on the CPU (e.g. n_gpu_layers: 30 for a 31-layer model).

The prompt used for translation follows the official TranslateGemma chat template exactly:

<start_of_turn>user
You are a professional {source} ({src_code}) to {target} ({tgt_code})
translator. Your goal is to accurately convey the meaning and nuances of the
original {source} text while adhering to {target} grammar, vocabulary, and
cultural sensitivities.
Produce only the {target} translation, without any additional explanations or
commentary. Please translate the following {source} text into {target}:

{text}<end_of_turn>
<start_of_turn>model

Greedy decoding (temperature: 0, top_k: 1) is used for the main translation; alternatives are sampled generations at alternative_temperature with unique seeds, de-duplicated.

API compatibility #

Implemented routes (same paths/params/shapes/errors as LibreTranslate):

route	notes
POST /translate	q (string or array for batch), source (code or auto), target, format (text/html), alternatives, api_key. detectedLanguage only when source=auto.
POST /detect	q (string); returns up to 3 {confidence, language} candidates (langdetect + lexilang, same as LibreTranslate).
GET /languages	[{"code", "name", "targets"}] from the YAML config.
GET /health	{"status": "ok"}; 503 while the model is still loading.
GET /frontend/settings	char limit, timeout, key requirements, supported file formats, default languages.
POST /suggest	403 when suggestions are disabled (default); accepts and acks otherwise.
POST /translate_file	multipart upload (.txt, .html), returns translatedFileUrl.
GET /download_file/{filename}	serves the stored translation.
GET /spec, /openapi.json, /docs, /redoc	LibreTranslate-shaped OpenAPI doc + Swagger UI.

Error responses are {"error": "<message>"} with the same status codes and messages as LibreTranslate (Invalid request: missing q parameter, <lang> is not supported, Invalid API key, Please contact the server operator to get an API key, Banned-style blocklist message, Slowdown: ... when the internal queue is full, etc.).

The web UI (/) is a single static page. Its input and output use dir="auto", so RTL languages (Hebrew, Arabic, Persian, Urdu, ...) render right-to-left and mixed-script text keeps correct Unicode bidi ordering.

Known, intentional deviations:

• alternatives are extra LLM generations (LibreTranslate uses argos alternatives) — capped by limits.alternatives_limit, deduplicated.
• Language resolution accepts any BCP-47 code whose primary subtag is configured (e.g. de-DE → de), in addition to the exact/variant fallbacks LibreTranslate has (pt/pb, zh/zt).
• JSON alternatives values that are not numbers return 400 (LibreTranslate crashes with 500 there).

Configuration reference (config.yaml) #

server:
  host: "0.0.0.0"
  port: 5000
  url_prefix: ""            # e.g. "/translate" to mount everything under a path
  trust_forwarded_for: false

model:
  path: "/path/to/translategemma-4b-it.IQ4_XS.gguf"
  n_ctx: 8192
  n_gpu_layers: 0           # -1 to offload everything to a GPU (Vulkan build)
  n_threads: 0              # 0 = auto
  n_batch: 512
  max_tokens: 1024
  temperature: 0.0
  alternative_temperature: 0.7
  timeout: 120.0
  verbose: false

limits:
  char_limit: -1            # max chars per text; -1 = unlimited
  batch_limit: -1           # max items in a batch q; -1 = unlimited
  alternatives_limit: -1    # max alternatives; -1 = unlimited
  frontend_timeout: 15000
  queue_size: 100           # single-worker queue depth; full => 429
  html_max_segments: 64
  file_expiration: 3600
  req_per_minute: -1        # requests/min per API key (or source IP); -1 = off

api_keys:
  enabled: false
  require_api_key: false
  keys:
    - "some-key"
    - key: "another-key"
      char_limit: 5000
    - key: "high"
      req_per_minute: 300   # per-key override of the global limit

ip_blocklist:
  enabled: false
  networks:
    - "203.0.113.7"
    - "198.51.100.0/24"
  message: "Your IP address is blocked"

frontend:
  source: "auto"
  target: "en"
  suggestions: false
  files_translation: true

languages:
  - code: "en"
    name: "English"
  - code: "de"
    name: "German"
  # ... one entry per language you want to expose (see shipped config)

Architecture #

HTTP (async FastAPI)
   │  API key + IP blocklist checks
   ▼
asyncio.Queue (bounded, per-process)
   │
   ▼
single worker task ──▶ dedicated executor thread ──▶ llama_cpp.Llama
                                                           (TranslateGemma GGUF)

• Every translation unit (per text in a batch, per HTML segment) becomes one queue job; the worker runs exactly one job at a time.
• Generation runs off the event loop in a single executor thread, so the async API stays responsive while the model is busy.
• If the queue is full, the request fails fast with 429 {"error": "Slowdown: too many requests queued, try again later"}.

Rate limiting #

limits.req_per_minute enables a per-client fixed 60-second window (LibreTranslate's "requests per minute"). The client is identified by API key when one is supplied (and keys are enabled), otherwise by source IP (X-Forwarded-For when trust_forwarded_for is set). An API key may override the global limit with its own req_per_minute. Rejected attempts are counted, so a client only re-enters at the next window boundary; exceeding the limit returns 429 {"error": "Slowdown: Rate limit reached"}. The counter lives in process memory (single-worker design).

Tests #

uv run pytest

The test suite runs against a fake LLM (no model download needed) and covers endpoint shapes, batch/alternatives/HTML behavior, language detection, char limits, API keys, IP blocklist, file translation and queue-full 429s.

License #

GemTranslate is licensed under the GNU Affero General Public License v3.0 or later (AGPL-3.0-or-later) — the same license as LibreTranslate.

Attribution #

Portions of this codebase are derived from LibreTranslate (AGPL-3.0), notably:

• src/gemtranslate/formatting.py::improve_translation_formatting — port of libretranslate/language.py::improve_translation_formatting
• src/gemtranslate/detect.py — port of the detector from libretranslate/detect.py (langdetect seed 0 + lexilang short-text boost)
• src/gemtranslate/languages.py — variant aliases (pt/pb, zh/zt) from libretranslate/language.py
• The overall endpoint surface, request/response shapes, error messages and status codes, which are intentionally reproduced one-to-one for API compatibility.

AI assistance disclaimer #

This software was produced with the assistance of the Pi coding-agent harness and the Qwen 3.8 27B LLM model, running with local inference.