BaseRT: Best-in-Class LLM Inference on Apple Silicon via Native Metal

BaseRT is a specialized AI inference runtime designed for Apple's Metal API, eschewing common abstractions found in other runtimes like llama.cpp and MLX, thereby enhancing performance on Apple silicon. By bypassing intermediate frameworks and unnecessary abstractions, BaseRT achieves higher throughput for local large language model (LLM) inference, particularly on Apple M3 and M4 Pro devices, as demonstrated by benchmarks showing up to 1.56× faster decode than llama.cpp and 1.35× faster than MLX. The runtime is built around a few core ideas aimed at reducing overhead, such as a zero-allocation decode loop, hand-fused Metal kernels, and compute-bound prefill, which collectively optimize execution efficiency. BaseRT's architecture allows for consistent, hardware-adaptive performance across different Apple silicon generations and supports various model families like LLaMA, Qwen3, and Gemma, with a focus on minimizing per-token latency and maximizing prefill throughput in mixture-of-experts models. This approach makes BaseRT a leading option for achieving best-in-class LLM inference on Apple devices, and it is available as a CLI, C API, and through various language bindings, encouraging community engagement and further development.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.