Q4_K_M vs Q5_K_M vs Q6_K: Ollama quantization explained

Q4_K_M has established itself as an unofficial standard in the Ollama community. When someone asks which quantization format they should load, the answer is almost reflexively: “Q4_K_M, good compromise.” This is true in many cases, but it is not a universal default. Anyone who processes longer German documents, runs RAG pipelines or has a Mac mini with enough unified memory is giving away quality. This article explains what really separates Q4_K_M, Q5_K_M and Q6_K, where the differences become measurable, and when an upgrade is worth the storage cost.

What separates K-Quants from older formats

GGUF quantization in the llama.cpp ecosystem knows several format families. The older formats Q4_0, Q4_1, Q5_0, Q5_1 and Q8_0 work with classic block integer formats. Q8_0 does not belong to the K-quant family; it is a very high quality 8-bit reference outside of this family. The K-Quants include Q2_K, the Q3_K, Q4_K and Q5_K variants as well as Q6_K. They use superblocks and additional scale or min parameters in order to lose less quality at low bit widths.

The “M” in Q4_K_M does not denote an abstract layer quality. What is meant is a concrete mixture: The original k-quants-PR quantizes selected tensors, in particular attention.wv or attn_v and feed_forward.w2 or ffn_down, higher than the rest. With Q4_K_M, parts of these tensors are treated with Q6_K, while the remaining weights predominantly use Q4_K. This is the practical reason why Q4_K_M is slightly larger than Q4_K_S, but usually more robust.

The result is visible in measurements, but the source must be clearly identified. The original llama.cpp-k-quants-PR shows a perplexity of 5.9601 on LLaMA-7B for Q4_K_M versus 5.9066 on FP16; Q6_K is very close to FP16 at 5.9110. These are smaller deltas than many community tables later suggest. For modern downstream benchmarks, Uygar Kurt’s study (arXiv, January 2026) is more relevant: it tests 13 GGUF configurations on Llama-3.1-8B-Instruct, including K-Quants and legacy formats, on GSM8K, HellaSwag, MMLU, IFEval and TruthfulQA. Restriction: An 8B model was tested in English. It is not directly transferable to other model sizes and other languages.

What really differentiates the stages

The most important levels for Ollama users in comparison. Q8_0 is deliberately placed next to it as a reference, but is not a K-quant:

Q4_K_M is the smallest of the K-quant levels compared here, but not the smallest K-quant overall; Q2_K and Q3_K are smaller and significantly more aggressive. Q5_K_M reduces the perplexity delta compared to Q4_K_M with a moderate increase in memory. Q6_K is so close to Q8_0 that the residual difference is no longer noticeable in many practical setups. Q8_0 is a very high quality reference, but the memory requirement is around 74 percent higher than Q4_K_M.

The benchmark differences on downstream tasks are measurable but not linear. In Kurt’s table, Q4_K_M is on average just below F16, Q5_K_M and Q6_K are also close to the baseline, and Q8_0 is almost exactly there. At the same time, the table shows that perplexity does not explain everything: Q5_0 is a legacy format and achieves the highest average value in this evaluation, while several 5-bit variants show almost the same perplexity but different GSM8K and IFEval values. Kurt’s Pareto analysis names Q5_0 and Q4_K_S as frontier points and formally classifies Q4_K_M and Q5_K_M as dominated; However, the Q5_0 lead is classified in the paper itself as possibly being due to noise, and in practice Ollama’s model library primarily delivers K-quant variants. This supports the actual statement better than a simple ranking list: If you work productively locally, you always need to keep an eye on your own task in addition to the file size and PPL.

Regarding processing throughput: Smaller quants save memory and can be faster, but this does not scale monotonically with bit width. Kurt’s throughput table, for example, shows Q5_K_M sometimes faster than Q4_K_M. On Apple Silicon, the central argument is therefore less the bandwidth than the capacity: a lot of unified memory allows higher quants or larger models without a local setup immediately running to the memory limit. A dedicated GPU whose model resides entirely in VRAM has no PCIe bottleneck; It only becomes problematic when model parts have to be swapped out or moved between memory areas.

Where Q4_K_M is actually enough

Clear use case: short English-language prompts, context under 4K tokens, creative texts or summaries without strict factual fidelity. Here Q4_K_M is a reasonable choice. The model responds more smoothly, fits in less RAM and allows larger models to run on systems with limited memory that would otherwise not be bootable. A good 14B model in Q4_K_M can be better than a weaker 8B model in Q8_0 despite stronger quantization if the larger model is fundamentally better at the task.

This is the real trade-off: increase the quality level within the same model, or increase the number of parameters of the model. If you work on a system with 16 GB of RAM, you can weigh up between an 8B model in Q6_K and a 14B model in Q4_K_M. The latter is often the wiser decision as long as the task involves brief English contexts.

Where Q4_K_M clearly loses: Long-Context, RAG, German

This is where the default becomes problematic. A peer-reviewed study by Mekala et al. (EMNLP 2025) examined quantization on long-context tasks and measured accuracy losses of up to 59 percent for individual 4-bit setups. The study tested five methods: FP8, GPTQ-int8, AWQ-int4, GPTQ-int4 and BNB-nf4. GGUF-K quants have not been tested directly. In addition, the measured long context inputs start at 64K tokens; a threshold of 16K is therefore a cautious practical extrapolation, not a directly measured limit. The evidence still justifies a conservative recommendation: Anyone who runs very long documents or RAG pipelines with long retrievals should prefer Q5_K_M or Q6_K and test their own workload.

The language effect is also proven from the EMNLP study. Non-English tasks lose disproportionately with 4-bit methods. For German-language texts, this means that Q4_K_M gets into drift territory earlier than with English. Anyone who uses Ollama to combine German-language documents on a Mac mini should explicitly take this point into account. The paper on quantization risks in local AI setups describes this risk profile in more detail; more about this in DeepDive on LLM inference and quantization.

Another critical point: RAG pipelines place special demands on instruction fidelity. The model should quote correctly from a given context and not hallucinate. The IFEval values ​​in Kurt’s table are not monotonic: Q4_K_M is not simply below Q6_K, and Q5_0 performs particularly strongly. That’s exactly the point: PPL alone is not enough as a selection criterion. For RAG applications, Q5_K_M is a useful starting point; Q6_K is the more conservative choice when memory is available.

Decision matrix by RAM and task

The decision depends on three parameters: available RAM (or VRAM), task type and language context. As a guide:

16GB RAM or less: Q4_K_M is the pragmatic default for English short contexts. For German texts or RAG, switch to Q5_K_M if possible, even if that forces the model size down.

24GB RAM: Q5_K_M is accessible for many 8B and 14B models. Q6_K for 8B models is within reach and recommended when instructional fidelity or longer contexts are important.

32 GB RAM and more (e.g. Mac mini M4 Pro or Mac Studio with M4 Max): Q6_K for 14B models or Q5_K_M for 30B class models are realistic. Q8_0 is only worthwhile if the memory is not used for other purposes; The increase in quality compared to Q6_K is marginal in most practical tasks. The Setup for Ollama on the Mac mini shows which model sizes run sensibly on which equipment.

Regarding the question Q6_K vs. Q8_0: The perplexity difference between Q6_K (+0.03) and Q8_0 (+0.01) compared to F16 is very small. If you use Q6_K and don’t have any measurable quality problems, you don’t need to switch to Q8_0 unless there is an abundance of memory. Q8_0 is more relevant for benchmarking purposes or as a nearly lossless baseline when comparing models.

Delimitation: K-Quants vs. GPTQ, AWQ, FP8

A common misunderstanding: K-Quants are GGUF formats from the llama.cpp ecosystem and run on CPU or via Metal/CUDA from a single file. GPTQ, AWQ, INT8 and FP8 are other quantization approaches typically deployed on GPU servers using PyTorch-based frameworks such as vLLM or HuggingFace Transformers. The number of papers for GPTQ-int4, AWQ-int4 and FP8 is significantly more extensive than for K-Quants. Anyone working with vLLM in a data center setup makes different decisions than Ollama users on consumer hardware. The architectures are not easily transferable.

IQ4_XS, IQ4_NL and other I-Quants are a separate topic within the GGUF/llama.cpp environment. They can be interesting for very tight storage budgets, but answer a different question than this article: maximum compression instead of robust default selection between Q4_K_M, Q5_K_M and Q6_K. That is why they are deliberately excluded here.

K-Quants are the relevant framework for everyday Ollama on Mac, Linux consumer or Windows PC. Everything else is a different stack. More on the differences between inference frameworks in Comparison of vLLM, llama.cpp, Ollama and TensorRT-LLM.