Quantization Explained — Q4, Q5, Q8, and FP16 for Local AI
What Q4_K_M, Q5_K_M, Q8, and FP16 actually mean, how much VRAM each saves, and which quantization to pick for local LLMs without wrecking quality.
📋 This article contains affiliate links. We may earn a small commission at no extra cost to you.
This guide explains the memory math behind every recommendation on this site. It contains affiliate links to GPUs we reference.
Quick answer
Use Q4_K_M (4-bit) as your default — it cuts VRAM by about 75% versus FP16 with quality loss that barely matters for chat, coding, and writing. Step up to Q5 or Q8 only if you have VRAM to spare and want the last few percent of accuracy. Drop to Q3 only to squeeze a bigger model into a smaller card, and accept the quality hit.
Quantization is the single reason a 32-billion-parameter model fits on a 24GB graphics card instead of needing 64GB. If you have ever wondered why local AI guides throw around Q4_K_M, Q5_K_M, Q8, and FP16, this is the explainer. By the end you will know exactly how much memory each level costs, what you give up, and which one to pick for your hardware.
This pairs directly with the VRAM requirements guide and the VRAM calculator, which use the same math you are about to learn.
What Quantization Actually Is
A model’s “weights” are billions of numbers. By default they are stored in 16-bit floating point (FP16/BF16), which takes 2 bytes per parameter. Quantization stores those same numbers with fewer bits — 8-bit, 5-bit, 4-bit — so each weight takes less memory. The model gets smaller and loads into less VRAM, at the cost of some precision in those numbers.
The key insight: large language models are remarkably tolerant of this. Going from 16-bit to 4-bit shrinks the model by roughly 75% while keeping most of its capability, because the model has so much redundancy that small rounding errors in individual weights mostly wash out.
The Levels, From Heaviest to Lightest
| Level | Bytes per parameter | VRAM vs FP16 | Quality | When to use |
|---|---|---|---|---|
| FP16 / BF16 | 2.0 | 100% (baseline) | Maximum | Training, research, max accuracy |
| Q8 | 1.0 | ~50% | ~1–2% loss | When you have VRAM to spare |
| Q5_K_M | ~0.65 | ~35% | Very small loss | Great middle ground |
| Q4_K_M | ~0.5 | ~25% | Noticeable on hard reasoning, fine for daily use | The standard. Start here. |
| Q3_K_M | ~0.4 | ~20% | Degrades on complex tasks | Only to fit a bigger model |
The _K_M suffix (as in Q4_K_M) refers to the GGUF “K-quant, medium” method, which protects the most important weights with slightly higher precision. It is why a modern Q4_K_M holds up far better than naive 4-bit rounding from a few years ago. You will also see AWQ and GPTQ — different 4-bit schemes used more in GPU-only (vLLM) setups; the practical tradeoffs are similar.
The Math You Can Do in Your Head
The formula behind every VRAM estimate on this site:
VRAM ≈ (parameters in billions × bytes per parameter) + ~2GB overhead + context
Worked examples:
- Llama 3.3 8B at Q4_K_M: (8 × 0.5) + 2 = 6GB — fits an 8GB card.
- DeepSeek R1 14B at Q4_K_M: (14 × 0.5) + 2 ≈ 9GB — comfortable on 12–16GB.
- Qwen 2.5 32B at Q4_K_M: (32 × 0.5) + 2 ≈ 18GB — needs 24GB for headroom.
- Same 32B at Q8: (32 × 1.0) + 2 = 34GB — now it needs multi-GPU or a 48GB card.
That last pair is the whole point: the same model swings from “fits a single 24GB GPU” to “needs a workstation” purely on quantization. Context length adds on top — roughly +0.5GB at 4K tokens, +2–4GB at 32K, and +8–16GB at 128K, which is where even 24GB cards get tight.
How to Pick
- Default to Q4_K_M. For chat, coding, summarization, and writing, the quality gap from FP16 is hard to notice and the memory savings are enormous.
- Go Q5_K_M or Q8 if you have spare VRAM and care about the last few percent on reasoning or precise instruction-following. Q8 is effectively lossless for most uses.
- Drop to Q3 only as a fitting trick — to run a 70B-class model on hardware that otherwise could not touch it. Expect weaker reasoning and more mistakes.
- Never trade parameter count away cheaply. A 14B model at Q4 generally beats a 7B model at Q8 of similar size on disk. When in doubt, prefer the bigger model at a lower bit depth — up to a point, more parameters beats more precision.
Common Mistakes
- Running FP16 locally “for quality” and running out of memory. Almost nobody needs FP16 for inference. Q8 gives you near-identical output at half the VRAM.
- Assuming Q4 is “low quality.” Modern K-quants at 4-bit are the mainstream choice for a reason — the loss is real but small.
- Forgetting context overhead. A model that fits at 4K tokens can spill at 128K. Budget memory for the context you actually use.
- Comparing bit depths across different model families. A 14B at Q4 and a 7B at Q8 are not the same thing — parameter count usually matters more than the extra bits.
Turn the Math Into Hardware
Once you know your target model and quantization, you have a VRAM number — and that number picks your GPU tier:
- 8–12GB: 7B–8B models at Q4. Start with the Intel Arc B580 or budget tier.
- 16GB: 14B models comfortably at Q4. The RTX 5060 Ti 16GB is the sweet spot.
- 24GB: 32B models at Q4. The used RTX 3090 is the cheapest path here.
For the full per-model table, see the VRAM requirements guide, or plug your numbers into the VRAM calculator.
Frequently Asked Questions
What does Q4_K_M mean?
It is a 4-bit quantization using the GGUF “K-quant, medium” method, which keeps the most important weights at slightly higher precision. It stores each parameter in about half a byte, cutting VRAM roughly 75% versus FP16 while keeping most of the model’s quality.
Is Q4 or Q8 better for local AI?
Q8 is higher quality (near-lossless) but uses twice the VRAM of Q4. For everyday chat, coding, and writing, Q4_K_M is the practical default; choose Q8 only when you have memory to spare and want maximum accuracy.
Does quantization make models slower?
No — usually the opposite. A quantized model is smaller, so it fits in VRAM and reads from memory faster. The bottleneck for local LLMs is fitting in VRAM, and quantization is what makes that possible.
How much VRAM does quantization save?
Versus FP16: Q8 saves about half, Q5 about 65%, and Q4 about 75%. That is what lets a 32B model run on a 24GB card instead of needing 64GB.
What is GGUF?
GGUF is the model file format used by llama.cpp, Ollama, and LM Studio. It packages quantized weights (like Q4_K_M) so they load directly on consumer hardware. AWQ and GPTQ are alternative 4-bit formats used more in GPU-only serving stacks.
Last updated: May 2026. Byte-per-parameter values and VRAM estimates match the methodology in our VRAM requirements guide and reference llama.cpp GGUF quantization and Hugging Face model cards.