Skip to content
Local AI Wizard
Your machine
Theme

What is quantization? The Q in model files, explained

Why the same model comes in files of very different sizes, what the smaller ones give up and how to read names like Q4_K_M.

Files and versions · updated

Short answer

Storing each of a model's numbers with fewer bits, like rounding. The file gets much smaller; the model gets slightly less exact.

  • Size: Qwen3.5 9B drops from 17.9 GB at 16 bits to 5.7 GB as Q4_K_M, 32% of the size.
  • In file names: The number after Q is roughly the bits kept: Q8 about 8, Q4 about 4.
  • Where to start: Q4_K_M, the usual balance of size and quality.

Published file sizes (unsloth/Qwen3.5-9B-GGUF, read Oct 23, 2026)

The idea

Think of a photo saved at a lower quality: it looks almost the same, and the file is much smaller. Or a recipe that says “a third of a cup” instead of “0.3333 cups”. A model is billions of numbers learned in training. Quantization stores each one with fewer bits, a bit like rounding it, so the whole file shrinks. The model still works, but every number is a little less exact.

A bit is one 0 or 1 in the computer's memory. The original files use 16 bits for each number. The “Q” in a file name says how many bits are left: Q8 keeps about 8, Q4 about 4.

Watch: quantization in 54 seconds

A silent explainer: one number stored with 16, 8 and 4 bits, a group sharing one scale, the real file sizes of Qwen3.5 9B and llama.cpp's published quality loss.

Transcript

  1. Same model. 32% of the size. How? Quantization.
  2. A model is billions of numbers. Quantization rounds each one. Fewer bits per number.
  3. Follow one number: 0.42. Fewer bits, bigger error.
  4. 4 bits: only 16 values. Each snaps to the nearest. A group shares one scale.
  5. Qwen3.5 9B: real file sizes. Start at Q4_K_M. 5.7 GB file · ~8 GB to run. Memory to spare? Go up. Short on memory? Q3_K_M.
  6. What you lose. Perplexity increase · Llama 3 8B · llama.cpp. Below 4 bits, loss climbs fast. Q4_K_M: a small loss.
  7. Start at Q4_K_M. Then go up if it fits. Local AI Wizard · Learn.

No sound. Every number in the video is on this page, with its source.

One number, fewer bits

Follow one made-up number, 0.42. With 16 bits it barely moves. With 8 bits it moves a little. With 4 bits there are only 16 possible values, so it lands on the nearest one and moves more.

One number, 0.42, stored three ways16 bits: read back as 0.41992, off by 0.00008. 8 bits: read back as 0.42094, off by 0.00094. 4 bits: read back as 0.46286, off by 0.04286.One number, 0.42, stored three ways16 bits0.41992off by 0.000088 bits0.42094off by 0.000944 bits0.46286off by 0.042864 bits: 8 numbers share one scale16 allowed values, 0.116 apart, from -0.93 to 0.81. 0.42 → 0.463; -0.17 → -0.116; 0.81 → 0.810; -0.63 → -0.579; 0.05 → 0.000; 0.29 → 0.347; -0.38 → -0.347; 0.71 → 0.694.4 bits: 8 numbers share one scale16 allowed values, 0.116 apart0.420.463-0.930
Left: one square per bit. Right: the 4-bit values a group can take (ticks), each number (hollow) and where it lands (filled). A simplified version of llama.cpp's Q8_0 and Q4_0, which use groups of 32 numbers.

The trick that keeps 4 bits usable is the shared scale. Numbers are stored in small groups. Each group keeps one extra number, its scale (here 0.116), and each number keeps only a whole-number step from −8 to 7. To read it back, the program multiplies the step by the scale. Small numbers in a group can round to zero, like 0.05 here.

The same example as a table. Made-up numbers; the arithmetic is the real rounding.
Stored withRead back asOff by
16 bits0.419920.00008
8 bits0.420940.00094
4 bits0.462860.04286

Real file sizes

Qwen3.5 9B: one model, six files

File size in GB, from 3 to 16 bits per number

4.7 GBQ3_K_MQ3_K_M5.7 GBQ4_K_MQ4_K_M6.6 GBQ5_K_MQ5_K_M7.5 GBQ6_KQ6_K9.5 GBQ8_0Q8_017.9 GBBF16BF16

Hub file sizes (unsloth/Qwen3.5-9B-GGUF, read Oct 23, 2026)

To run a file you need a bit more memory than its size: room to work plus the conversation itself. With the same rule as the model pages, Q4_K_M needs about 8 GB and BF16 about 22 GB, before your computer keeps its own share (2 GB on a graphics card, 2 to 8 GB on a Mac or PC). The site's general planning figure for Qwen3.5 9B in Q4 is 7 GB. Switch the chart to the table to see every file. The repository has no plain Q2_K; its 2-bit files are Unsloth Dynamic (see the names below).

What you lose

Quality loss is measured with perplexity: how surprised a model is by real text it has not seen. Lower is better, and quantizing raises it. llama.cpp publishes how much each file type raises it on one model, Llama 3 8B (llama.cpp · quantize options, read Oct 25, 2026).

Increase over the 16-bit original, Llama 3 8B, as published by llama.cpp. Other models lose more or less.
FilePerplexity increase (lower is better)
Q8_00.0026
Q6_K0.0217
Q5_K_M0.0569
Q4_K_M0.1754
Q4_00.4685
Q3_K_M0.6569
Q2_K3.5199

Read it as a curve. From 8 down to 5 bits the change is tiny; when k-quants were introduced, the 6-bit file stayed within 0.1% of the original (llama.cpp PR #1684 · k-quants, read Oct 25, 2026). Q4_K_M is still small. Below 4 bits the loss climbs fast: Q3_K_M loses about 4 times as much as Q4_K_M, and Q2_K far more.

That is why Q4_K_M is the usual sweet spot: about a third of the original size with a small loss, just before the curve bends. llama.cpp's own tool once labelled it “medium, balanced quality - recommended” (llama.cpp · quantize options, 2023, read Oct 25, 2026), and Ollama picks it by default from a Hugging Face repository that has it (Hugging Face · Ollama, read Oct 17, 2026).

Loss depends on the model: small models and some newer ones are more sensitive, and published benchmarks rarely test quantized files. This site does not invent a quality score for a file.

Reading the names

The labels you meet on Hugging Face and in Ollama or LM Studio.

GGUF
The file format, not a precision. One file holds the whole model, at any of the precisions below.llama.cpp · Ollama · LM StudioHugging Face · GGUF · read Oct 17, 2026
Q4_K_M
Q4: about 4 bits per number. K: “k-quant”, numbers stored in groups of 256, each split into 8 blocks of 32 with its own scale. M: medium mix, where a few sensitive parts are kept at 6 bits.GGUF files (S = small, all parts at 4 bits)llama.cpp PR #1684 · k-quants · read Oct 25, 2026
Q8_0 · Q4_0
The older, simpler formats: blocks of 32 numbers that share one scale. Q8_0 stays very close to the original; Q4_0 loses more than Q4_K_M at a similar size.GGUF filesllama.cpp · ggml-common.h · read Oct 25, 2026
IQ4_XS · IQ3_M · IQ2_XXS
“i-quants”. Before rounding, the tool reads sample text to learn which numbers matter most (an importance matrix) and keeps those more exact. XS and XXS mean extra small.GGUF files, mostly 2 to 4 bitsllama.cpp · imatrix · read Oct 25, 2026
UD-Q4_K_XL · UD-IQ2_M
Unsloth Dynamic: files from the Unsloth team that are “selectively quantized”, giving different parts of the model different precisions. XL is their larger mix.GGUF files from unsloth/…unsloth/DeepSeek-R1-GGUF · read Oct 25, 2026
MXFP4
A 4-bit floating-point format in which blocks of numbers share a scale. OpenAI trained gpt-oss with it, which lets gpt-oss-20b run within 16 GB.gpt-oss, as publishedOpenAI · gpt-oss · read Oct 25, 2026
FP8
8-bit floating point (E4M3 or E5M2). Some publishers release an official FP8 copy next to the original.Server graphics cardsMicikevicius et al., 2022 · FP8 · read Oct 25, 2026
BF16 · F16
16 bits per number: the original, not quantized. BF16 keeps 8 bits for the size of the number and 7 for its digits.The publisher's releaseGoogle Cloud · bfloat16 · read Oct 25, 2026
GPTQ
A method that rounds a whole model in one pass to 3 or 4 bits, correcting each step's error as it goes.Graphics-card runtimes, not llama.cppFrantar et al., 2022 · GPTQ · read Oct 25, 2026
AWQ
“Activation-aware”: finds the about 1% of numbers that matter most and protects them while rounding the rest.Graphics-card runtimes, not llama.cppLin et al., 2023 · AWQ · read Oct 25, 2026
EXL2
Mixes precisions inside one model to hit any average between 2 and 8 bits, such as 4.65.ExLlamaV2ExLlamaV2 README · read Oct 25, 2026
MLX 4-bit · 8-bit
Apple's format for Macs. By default numbers are rounded in groups of 64, each with its own scale.Macs with Apple silicon (LM Studio, mlx-lm)Apple · mlx-lm · read Oct 25, 2026

How to choose

1 · Start here
Q4_K_MAbout a third of the original size. For Qwen3.5 9B: 5.7 GB.
2 · Memory to spare
Q5_K_M · Q6_K · Q8_0Closer to the original; larger and slower to read.
3 · Does not fit
Q3_K_M · IQ3 · IQ4_XSSmaller files with more loss. For Qwen3.5 9B, Q3_K_M is 4.7 GB.
4 · Still does not fit
A smaller model in Q4Usually a better choice than a bigger model in Q2, where the loss is far larger.

Each model page lists its real files and the memory each needs in Versions to download (Qwen3.5 9B). To see which models and files fit your computer, pick your machine.

Questions

Does quantization make a model worse?

A little, and more the fewer bits it keeps. At 8 and 6 bits the change is very small; at 4 bits (Q4_K_M) it is small; at 2 bits it is large. How much depends on the model, so a quantized file can score lower than the original on a benchmark.

Is Q8_0 always better than Q4_K_M?

It is closer to the original, but for Qwen3.5 9B it is 9.5 GB against 5.7 GB, and a bigger file takes longer to read for every word, so it usually writes more slowly. Pick it when it fits with room to spare.

Can I quantize a model myself?

Yes. llama.cpp's llama-quantize tool turns a 16-bit GGUF file into Q4_K_M or another type. Most people download a file someone already made, because popular models are converted within days.

Do this site's scores apply to Q4 files?

No. Benchmark scores here are for the publisher's original weights. A quantized copy is not tested separately, so its score can be a little lower.