Optimizing on-device inference for Apple Silicon

Hybrid Compute divides AI tasks between cloud-based frontier models for research and reasoning and local models on Macs for private files and apps, making fast local inference important for a seamless experience. Lily is a lightweight Rust and Metal inference engine built specifically for Qwen3.6-35B-A3B on Apple silicon, bypassing PyTorch and MLX in favor of model-specific execution plans for prompt processing, or prefill, and token generation, or decode. On an M5 Max MacBook Pro with 128 GB of unified memory, Lily averaged 1.23 times MLX-LM’s prefill throughput and 1.35 times its decode throughput across contexts from 256 to 128K tokens, reaching 5,749.9 prefill tokens per second and 186.6 decode tokens per second at 4K-token workloads. Its optimizations address Qwen’s sparse mixture-of-experts routing, recurrent Gated DeltaNet layers, and grouped-query attention by keeping routing and state on the GPU, dequantizing 4-bit weights during computation, selecting workload-specific matrix or vector paths, fusing operations to reduce memory traffic, and reusing attention-cache data across query heads. Tests found that some techniques, including speculative decoding, reduced overall performance under these conditions, while measurements suggested core operations were already approaching hardware bandwidth and compute limits. Lily’s output remained broadly consistent with MLX-LM, with perplexity only 0.04% higher and matching top-token choices at 96.35% of tested positions, and the project’s authors argue that efficient local inference increasingly requires engines tailored to both a model’s architecture and the hardware platform.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.