Per-tensor layout maps for GGUF quantization

Bartowski describes a data-driven approach to improving GGUF quantization layouts by assigning precision on a per-tensor basis rather than relying only on model-agnostic heuristics in llama.cpp. After running more than 1,000 quantization experiments, primarily on Qwen models and validated across several other model families, the work found that token embeddings are especially sensitive, sensitivity tends to be highest in early and late layers, and tensor types such as attention value/output projections, feed-forward up projections, and SSM outputs often benefit most from additional bits. A solver uses these measured sensitivity priors, model shape, quantization block sizes, and target formats to generate layouts intended to optimize quality per bit while making labels such as Q3_K_S, Q3_K_M, and Q3_K_L more accurately reflect their predominant tensor types. Tests showed that the approach generally transfers to different architectures after adjustments for unusual layouts, although improvements are smaller for dense models below roughly 3 bits and failures remain possible. To manage this uncertainty, new model architectures receive canary tests that compare mapped layouts against the prior heuristic using KLD against BF16 token probabilities, with the system reverting to the older method when the new layout performs worse. The revised scheme changes some available quantization variants and file sizes, discourages embedding-only upgrades in favor of selecting the next quantization tier, and will continue to be evaluated on additional models and specialized tensor structures.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.