llama.cpp

Gemma 4 12B features an encoder-free multimodal architecture that runs locally on 16GB VRAM devices with native audio support. By eliminating separate vision and audio encoders, it reduces latency and pairs with a dedicated MTP model for faster inference, marking the first mid-sized multimodal model with a macOS desktop app for fully offline interaction.

本地运行大语言模型（LLMs）可通过 llama.cpp、Ollama 和 LM Studio 等工具实现，兼顾隐私与学习。

Reachy Mini now runs its voice backend locally, eliminating the need for cloud servers.

Julien Chaumond 展示 Qwen3.6-27B 模型通过 Llama.cpp 在 MacBook Pro 上本地运行 Pi 编程代理，处理 Hugging Face 代码库任务时性能逼近 Claude Opus，且完全离线。

With MTP support, llama.cpp improves local model inference speed by 78%, boosting Qwen3.6-27B from 25 to 45 tokens/sec on A10G.

MTP is a new speculative decoding feature built into llama.cpp that can approximately double token generation speed for most use cases, achieving ~30 tok/sec with the Dense 27B model and ~100 tok/sec with the MoE model.

Google announces its model weights are compatible with major open-source ecosystems and can be directly downloaded from Hugging Face and Kaggle, lowering deployment barriers.