If you've been running local LLMs on consumer hardware, you've almost certainly used Q4_K_M. It's the default choice for most quantized models on HuggingFace — the format everyone reaches for when FP16 won't fit and Q5 looks like overkill. But most people don't actually know what the letters mean, and that matters more than you'd think.
What Q4_K_M actually is
The format name breaks down into three parts. Q4 means every weight in the model is stored in 4 bits instead of 16. Simple enough. But then there's K_M, and that's where it gets interesting.
K refers to k-quants — a quantization scheme where weights are grouped into blocks and each block gets its own scale factor. This is fundamentally different from simpler quantization approaches that use a single scale for an entire layer. Grouping weights into blocks lets you capture variation within the layer without needing per-weight granularity. That's the key insight: block quantization with local scale factors preserves more precision where it matters.
The M stands for "meta" — it specifies how the scale factors themselves are stored. In Q4_K_M, scales are stored in 6-bit floating point format, which gives you better dynamic range than fixed-point and costs very little extra bits. The combination of block-level 4-bit weights plus 6-bit floating scales gives you something that's closer to true 4-bit effective precision in practice, while retaining enough granularity to avoid the worst quantization artefacts.
The block size is 8 — meaning every group of 8 weights shares one scale. This is a deliberate tradeoff: smaller blocks (like Q4_K_S with block size 4) preserve more detail but add more overhead. Larger blocks save space but introduce more rounding error per block. K_M's block size of 8 hits the sweet spot for most hardware and model sizes.
VRAM footprint: how the formats compare
Here's where it gets practical. A 7B parameter model in FP16 needs roughly 14GB of VRAM just for the weights — that's before you account for activations, KV cache, or any overhead. Q4_K_M gets that down to about 3.8–4.2GB, depending on the model. That's roughly a 3.5x reduction, and it means you can run a 7B model on a 6GB card if you're careful with context length.
Compare that to Q8_0 — the 8-bit format that many consider "reference quality." Q8_0 needs roughly 7–7.5GB for the same 7B model. It's double the VRAM of Q4_K_M for less than a meaningful quality gain in most workloads. Q8_0 is essentially ceiling for quality — the returns from going above 8-bit are minimal for general use, which is why virtually no one runs Q16. The real quality floor is Q4.
Q5_K_M sits in between, typically needing 4.5–5GB for a 7B model. It's a reasonable middle ground if you're GPU-rich but not quite VRAM-rich enough for Q8. You get most of the quality benefits of 8-bit while retaining significant space savings. The penalty is in generation speed — more bits means more data to shuffle — but it's rarely a bottleneck on modern hardware.
For reference, Q3_K_M drops to about 3.2–3.5GB, and that's where you start entering genuinely visible quality degradation for most tasks. Q3 is fine for experimenting but not for anything you care about.
RTX 3060 12GB: real numbers
I ran these formats on a stock RTX 3060 12GB with a Xeon E3-1270 v5 (16 threads), running Llama.cpp with batch size 1 and roughly 2048 context tokens allocated. All tests used the same 7B model architecture for fair comparison. GPU utilization was consistently between 85–95% during generation — the CPU wasn't the bottleneck.
Q4_K_M: 20–24 tokens/second at 2048 context. At 4096 context, it drops to 16–18 t/s due to KV cache pressure on a 12GB card with this architecture. These numbers vary by model (longer model names = slightly more CPU-side overhead in the GGUF format), but the ballpark is consistent.
Q5_K_M: 16–19 t/s at 2048 context. The additional bit-depth slows memory throughput measurably.
Q8_0: 12–15 t/s. Notice the pattern — halving the bit-depth roughly doubles throughput. This holds until you hit compute-bound operations, which typically only happens with very small models or very short contexts.
For the 3060 specifically: Q4_K_M is the format that lets you actually use the full 12GB meaningfully. You can run a 7B model in Q4_K_M, maintain a generous context window, and still have headroom for other processes. Q8_0 at 7.5GB leaves less margin and delivers noticeably slower output.
Where the accuracy hit actually shows up
The standard line on Q4 quantization is that it's "nearly lossless" for general text generation, and that's mostly true — but it's not the whole story.
For tasks like creative writing, general Q&A, summarisation, and most conversational use cases, Q4_K_M is functionally equivalent to FP16 in blind tests. The differences are subtle enough that you'd need side-by-side comparison with specific adversarial prompts to notice them. This is why people say "nearly lossless" — the degradation is below the threshold of practical notice for most users.
But there are three areas where it shows up, and they're worth knowing:
Math and code. LLM quantization errors compound in tasks that require precise, deterministic output. If you're generating Python and a variable name gets quantized to the wrong token, the code is broken. If you're doing multi-step arithmetic, small errors in intermediate values can cascade into wrong final answers. Q4_K_M shows a measurable drop on GSM8K and similar math benchmarks — typically 2–4% versus FP16. For code generation, the degradation is task-dependent but visible on complex functions.
Long, complex context windows. When you're running retrieval-augmented generation or feeding in long documents, quantization error in the attention mechanism becomes more relevant. The model is making more decisions per token, and small errors propagate further. Q4_K_M handles this fine for normal use, but if you're building something where every generation decision matters critically, you may notice a higher rate of subtle context errors.
Instruction following on edge cases. Standard benchmarks like MMLU show Q4_K_M nearly at FP16 parity. But adversarial or unusual prompts — the stuff that's not in training distributions — can expose the quantization noise more. The model loses a little bit of "creative" reasoning capability. Most users won't hit this. If you're building a coding assistant or something where you need consistent, precise instruction adherence, it's worth testing.
The important thing: none of this means Q4_K_M is bad. It means Q4_K_M is lossy, like every quantization format, and the loss is small enough that it's often irrelevant. Know where it matters for your use case.
Which format to actually use
The practical recommendation comes down to three scenarios:
VRAM ≤ 8GB, or running a 13B+ model: Q4_K_M is your only reasonable choice. Q5_K_M won't fit; Q8_0 definitely won't. The quality tradeoffs are acceptable for most use cases, and the alternative — not running the model at all — is worse. If you're really squeezed, Q3_K_M is better than nothing, but test your specific task before committing to it.
VRAM 8–12GB, 7B model: Q4_K_M is still the default, but you have room to experiment. If you're doing coding, math, or anything where precision matters, try Q5_K_M. You might lose 2–4 t/s — on a 3060 that's the difference between 18 and 14 t/s. If the speed hit bothers you, Q4_K_M is rarely wrong. But don't default to Q8_0 here — the VRAM cost is high and the quality gain over Q5 is usually minimal.
VRAM > 12GB or want maximum quality: Q8_0 is the reference choice for a reason. Yes, it's slower — but on modern cards (3090, 4090, or better), the absolute throughput is still high enough that 12 t/s is perfectly usable. If you're building anything production-grade or you care deeply about output quality, Q8_0 is the right call. The cost is VRAM, not money — the format itself is free.
The broader point: Q4_K_M isn't a compromise. It's the format that makes local LLMs actually runnable on consumer hardware. The "M" in the suffix — the meta scale encoding — is specifically what makes it better than naive 4-bit quantization, and understanding why should give you more confidence in the results you're getting. This isn't a hack. It's the deliberate engineering choice that makes local AI practical at all.
Try it. Tweak the context length. Watch the token counter. Then try Q5 and feel the speed difference. You'll learn more from ten minutes of running models than from any amount of theory.