What is quantization? The Q in model files, explained
Why the same model comes in files of very different sizes, what the smaller ones give up and how to read names like Q4_K_M.
Short answer
Storing each of a model's numbers with fewer bits, like rounding. The file gets much smaller; the model gets slightly less exact.
- Size: Qwen3.5 9B drops from 17.9 GB at 16 bits to 5.7 GB as Q4_K_M, 32% of the size.
- In file names: The number after Q is roughly the bits kept: Q8 about 8, Q4 about 4.
- Where to start: Q4_K_M, the usual balance of size and quality.
Published file sizes (unsloth/Qwen3.5-9B-GGUF, read Oct 23, 2026)
The idea
Think of a photo saved at a lower quality: it looks almost the same, and the file is much smaller. Or a recipe that says “a third of a cup” instead of “0.3333 cups”. A model is billions of numbers learned in training. Quantization stores each one with fewer bits, a bit like rounding it, so the whole file shrinks. The model still works, but every number is a little less exact.
A bit is one 0 or 1 in the computer's memory. The original files use 16 bits for each number. The “Q” in a file name says how many bits are left: Q8 keeps about 8, Q4 about 4.
Watch: quantization in 54 seconds
A silent explainer: one number stored with 16, 8 and 4 bits, a group sharing one scale, the real file sizes of Qwen3.5 9B and llama.cpp's published quality loss.
Transcript
- Same model. 32% of the size. How? Quantization.
- A model is billions of numbers. Quantization rounds each one. Fewer bits per number.
- Follow one number: 0.42. Fewer bits, bigger error.
- 4 bits: only 16 values. Each snaps to the nearest. A group shares one scale.
- Qwen3.5 9B: real file sizes. Start at Q4_K_M. 5.7 GB file · ~8 GB to run. Memory to spare? Go up. Short on memory? Q3_K_M.
- What you lose. Perplexity increase · Llama 3 8B · llama.cpp. Below 4 bits, loss climbs fast. Q4_K_M: a small loss.
- Start at Q4_K_M. Then go up if it fits. Local AI Wizard · Learn.
No sound. Every number in the video is on this page, with its source.
One number, fewer bits
Follow one made-up number, 0.42. With 16 bits it barely moves. With 8 bits it moves a little. With 4 bits there are only 16 possible values, so it lands on the nearest one and moves more.
The trick that keeps 4 bits usable is the shared scale. Numbers are stored in small groups. Each group keeps one extra number, its scale (here 0.116), and each number keeps only a whole-number step from −8 to 7. To read it back, the program multiplies the step by the scale. Small numbers in a group can round to zero, like 0.05 here.
| Stored with | Read back as | Off by |
|---|---|---|
| 16 bits | 0.41992 | 0.00008 |
| 8 bits | 0.42094 | 0.00094 |
| 4 bits | 0.46286 | 0.04286 |
Real file sizes
Qwen3.5 9B: one model, six files
File size in GB, from 3 to 16 bits per number
Hub file sizes (unsloth/Qwen3.5-9B-GGUF, read Oct 23, 2026)
To run a file you need a bit more memory than its size: room to work plus the conversation itself. With the same rule as the model pages, Q4_K_M needs about 8 GB and BF16 about 22 GB, before your computer keeps its own share (2 GB on a graphics card, 2 to 8 GB on a Mac or PC). The site's general planning figure for Qwen3.5 9B in Q4 is 7 GB. Switch the chart to the table to see every file. The repository has no plain Q2_K; its 2-bit files are Unsloth Dynamic (see the names below).
What you lose
Quality loss is measured with perplexity: how surprised a model is by real text it has not seen. Lower is better, and quantizing raises it. llama.cpp publishes how much each file type raises it on one model, Llama 3 8B (llama.cpp · quantize options, read Oct 25, 2026).
| File | Perplexity increase (lower is better) |
|---|---|
| Q8_0 | 0.0026 |
| Q6_K | 0.0217 |
| Q5_K_M | 0.0569 |
| Q4_K_M | 0.1754 |
| Q4_0 | 0.4685 |
| Q3_K_M | 0.6569 |
| Q2_K | 3.5199 |
Read it as a curve. From 8 down to 5 bits the change is tiny; when k-quants were introduced, the 6-bit file stayed within 0.1% of the original (llama.cpp PR #1684 · k-quants, read Oct 25, 2026). Q4_K_M is still small. Below 4 bits the loss climbs fast: Q3_K_M loses about 4 times as much as Q4_K_M, and Q2_K far more.
That is why Q4_K_M is the usual sweet spot: about a third of the original size with a small loss, just before the curve bends. llama.cpp's own tool once labelled it “medium, balanced quality - recommended” (llama.cpp · quantize options, 2023, read Oct 25, 2026), and Ollama picks it by default from a Hugging Face repository that has it (Hugging Face · Ollama, read Oct 17, 2026).
Loss depends on the model: small models and some newer ones are more sensitive, and published benchmarks rarely test quantized files. This site does not invent a quality score for a file.
Reading the names
The labels you meet on Hugging Face and in Ollama or LM Studio.
- GGUF
- The file format, not a precision. One file holds the whole model, at any of the precisions below.llama.cpp · Ollama · LM StudioHugging Face · GGUF · read Oct 17, 2026
- Q4_K_M
- Q4: about 4 bits per number. K: “k-quant”, numbers stored in groups of 256, each split into 8 blocks of 32 with its own scale. M: medium mix, where a few sensitive parts are kept at 6 bits.GGUF files (S = small, all parts at 4 bits)llama.cpp PR #1684 · k-quants · read Oct 25, 2026
- Q8_0 · Q4_0
- The older, simpler formats: blocks of 32 numbers that share one scale. Q8_0 stays very close to the original; Q4_0 loses more than Q4_K_M at a similar size.GGUF filesllama.cpp · ggml-common.h · read Oct 25, 2026
- IQ4_XS · IQ3_M · IQ2_XXS
- “i-quants”. Before rounding, the tool reads sample text to learn which numbers matter most (an importance matrix) and keeps those more exact. XS and XXS mean extra small.GGUF files, mostly 2 to 4 bitsllama.cpp · imatrix · read Oct 25, 2026
- UD-Q4_K_XL · UD-IQ2_M
- Unsloth Dynamic: files from the Unsloth team that are “selectively quantized”, giving different parts of the model different precisions. XL is their larger mix.GGUF files from unsloth/…unsloth/DeepSeek-R1-GGUF · read Oct 25, 2026
- MXFP4
- A 4-bit floating-point format in which blocks of numbers share a scale. OpenAI trained gpt-oss with it, which lets gpt-oss-20b run within 16 GB.gpt-oss, as publishedOpenAI · gpt-oss · read Oct 25, 2026
- FP8
- 8-bit floating point (E4M3 or E5M2). Some publishers release an official FP8 copy next to the original.Server graphics cardsMicikevicius et al., 2022 · FP8 · read Oct 25, 2026
- BF16 · F16
- 16 bits per number: the original, not quantized. BF16 keeps 8 bits for the size of the number and 7 for its digits.The publisher's releaseGoogle Cloud · bfloat16 · read Oct 25, 2026
- GPTQ
- A method that rounds a whole model in one pass to 3 or 4 bits, correcting each step's error as it goes.Graphics-card runtimes, not llama.cppFrantar et al., 2022 · GPTQ · read Oct 25, 2026
- AWQ
- “Activation-aware”: finds the about 1% of numbers that matter most and protects them while rounding the rest.Graphics-card runtimes, not llama.cppLin et al., 2023 · AWQ · read Oct 25, 2026
- EXL2
- Mixes precisions inside one model to hit any average between 2 and 8 bits, such as 4.65.ExLlamaV2ExLlamaV2 README · read Oct 25, 2026
- MLX 4-bit · 8-bit
- Apple's format for Macs. By default numbers are rounded in groups of 64, each with its own scale.Macs with Apple silicon (LM Studio, mlx-lm)Apple · mlx-lm · read Oct 25, 2026
How to choose
- 1 · Start here
- Q4_K_MAbout a third of the original size. For Qwen3.5 9B: 5.7 GB.
- 2 · Memory to spare
- Q5_K_M · Q6_K · Q8_0Closer to the original; larger and slower to read.
- 3 · Does not fit
- Q3_K_M · IQ3 · IQ4_XSSmaller files with more loss. For Qwen3.5 9B, Q3_K_M is 4.7 GB.
- 4 · Still does not fit
- A smaller model in Q4Usually a better choice than a bigger model in Q2, where the loss is far larger.
Each model page lists its real files and the memory each needs in Versions to download (Qwen3.5 9B). To see which models and files fit your computer, pick your machine.
Questions
Does quantization make a model worse?
A little, and more the fewer bits it keeps. At 8 and 6 bits the change is very small; at 4 bits (Q4_K_M) it is small; at 2 bits it is large. How much depends on the model, so a quantized file can score lower than the original on a benchmark.
Is Q8_0 always better than Q4_K_M?
It is closer to the original, but for Qwen3.5 9B it is 9.5 GB against 5.7 GB, and a bigger file takes longer to read for every word, so it usually writes more slowly. Pick it when it fits with room to spare.
Can I quantize a model myself?
Yes. llama.cpp's llama-quantize tool turns a 16-bit GGUF file into Q4_K_M or another type. Most people download a file someone already made, because popular models are converted within days.
Do this site's scores apply to Q4 files?
No. Benchmark scores here are for the publisher's original weights. A quantized copy is not tested separately, so its score can be a little lower.