local models #

running local LLMs for real agentic work — serving, tool-calling, and the harness around them.

• nvidia-runtime — loaded drivers, CUDA runtime libraries, and Linux binary compatibility on RTX GPUs
• gemma-4-12b-mlx — serving gemma 4 on apple silicon; use mlx-vlm, not mlx-lm
• tool-calling — per-engine tool-call parsing quirks, and why structured tool_calls beat any text-parsed format
• harness-weight — a heavy harness destabilizes tool-calling that the same model handles cleanly under a lean one
• dwarfstar — specialized DeepSeek inference, quantization, memory placement, and interpreting prefill versus generation claims

sources #

• does-it-tool — pre-flight: does this model actually tool-call on this endpoint?
• zigman — the eval loop these notes came from