Transformers now runs llama.cpp quants

Hugging Face has added initial support for efficiently running GGUF-quantized models through the Transformers API, allowing users to load local, memory-friendly checkpoints with familiar Python and PyTorch workflows. The implementation targets Apple Silicon Macs and initially supports Qwen3.5 dense and mixture-of-experts architectures, using llama.cpp’s ggml Metal kernels for quantized matrix operations, attention, normalization, and related tasks to approach llama.cpp performance while retaining Transformers features such as custom generation logic, evaluation, model inspection, and optional dequantization for fine-tuning. GGUF files package model weights, metadata, tokenizers, and chat templates in a single format, with quantization variants such as Q4_K_M offering reduced memory use at some potential quality cost. Models can be loaded via from_pretrained or served through an OpenAI-compatible Transformers endpoint, enabling use with local clients such as Jan and Pi. Benchmarks on an M2 Max MacBook Pro indicate that Transformers performs close to llama.cpp, although measurements differ because the Transformers tests include prompt processing. The project also improves the general Transformers generation loop by reducing unnecessary attention-mask work and CPU-GPU synchronization, while future work will expand architecture coverage, batching, padding support, hardware availability, and potential reuse of ggml kernels for vision, audio, multimodal, and other model types.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.