clee.sh / qxmx
A single-model (Bonsai-27B) inference engine for Intel Arc GPUs
Star
Fork 1
Watch
C++ 95%
C 2%
Meson <1%
Shell <1%
Dockerfile <1%
Python <1%
115 4 0
docs
meson
scripts
src
tests
third_party
tools
.dockerignore
.gitignore
AGENTS.md
Dockerfile
LICENSE-third-party.txt
meson_options.txt
meson.build
plan.md
README.md
ROADMAP.md
README.md

qxmx #

This project contains an experimental single-purpose inference engine with one goal: running the Bonsai-27B ternary-quantized model as fast as possible on Intel Arc GPU hardware.

I have an Arc Pro B70. Once I discovered that Intel's GPU hardware natively supports a 2-bit integer format (and AMD's Radeon Pro R9700 does not!), one which happens to map perfectly in-hardware to the ternary values [-1, 0, +1]... I wanted to see how hard it would be to bootstrap a new inference engine from scratch using modern LLMs (mainly GLM-5.2 and Kimi-K3). KV cache has user-selectable K quantization (fp16, fp8, and q8_0), but V quantization is currently hardcoded to 4-bit TurboQuant. There's an OpenAI-chat-completions-compatible endpoint built into qxmx_serve.

DSpark speculative decoding is supported by the model but not yet by this software. Nor is any vision functionality currently implemented.

One thing I've learned is that despite being compatible with a wide variety of open models, llama.cpp is really well-optimized and hard to beat!

I was inspired by https://github.com/antirez/ds4/ and https://github.com/JustVugg/colibri/ but don't blame them! None of this is their fault.

download the model #

You'll need either of the ternary 27B Q2_0 files from Prism-ML's huggingface; both work, but I mainly developed and tested with Ternary-Bonsai-27B-Q2_g64.gguf specifically. In the future it may make more sense to use hf download to grab the appropriate files once we support speculative decoding or vision features, but for now, just this one GGUF file is all we need.

the easy path #

You can build the Dockerfile and then run docker --device /dev/dri ... to pass your Intel GPU through (right now only one GPU is supported but multiple Arc GPU support is planned for a future release). That's probably the easiest way to get it running locally. An example command:

docker run --rm --name qxmx_serve --device /dev/dri -p 8080:8080 -v ~/models:/models:ro qxmx -- ./build/qxmx_serve --host 0.0.0.0 --port 8080 --slots 2 --ctx-per-slot 131072 --ctk q8_0 /models/Ternary-Bonsai-27B-Q2_g64.gguf

This will get you a local web server that speaks something similar to the OpenAI /chat/completions endpoint at http://localhost:8080/v1/chat/completions, and you can see the single model at http://localhost:8080/v1/models.

Tool-calling in maki seems to work. Performance optimizations are ongoing but speed is now competitive with llama.cpp on B70.

building from source #

The author has only tested this on Debian Trixie with some packages from testing/unstable.

You'll need to ensure you have run apt install meson ninja-build pkg-config libstdc++-14-dev, and then you'll have to set up the oneapi apt repo and install the intel-oneapi-dpcpp-cpp-2026.1 package from it. Once that's all done, you should be able to run:

source /opt/intel/oneapi/setvars.sh
meson setup --native-file meson/icpx.ini build
meson compile -C build

Steal from the Dockerfile. Or if you're on Fedora or Arch or whatever, send me a PR to update these instructions once you figure out the right steps to get it working for you.