Skip to content
Local AI Wizard
Your machine
Theme

Q4_K_M, Q8_0 or F16: which file should I download?

The same model in three precisions: file size, memory and what you give up.

Files and versions · updated

Short answer

Start with Q4_K_M. Move up to Q8_0 only if it fits with room to spare.

  • Q4_K_M: The usual pick: 16.8 GB for Qwen3.6 27B, 31% of the original.
  • Q8_0: Closer to the original, and bigger: 28.6 GB for the same model.
  • F16 · BF16: Full precision (53.8 GB), for converting or fine-tuning, not for everyday use.

Published file sizes (unsloth/Qwen3.6-27B-GGUF, read Oct 16, 2026)

What the names mean

A GGUF file stores every weight of the model with a chosen precision. The name says which one (Hugging Face · GGUF, read Oct 17, 2026):

Q4_K_M
4-bit “k-quant”, medium variantWeights in blocks with their own scales; some layers kept at higher precision. llama.cpp measures 4.89 bits per weight.
Q8_0
8-bit, one scale per block of 32 weightsAn older, simpler scheme that stays close to the original. 8.5 bits per weight.
F16 · BF16
16-bit floating point, no quantizationHow most publishers release the weights. F16 measures 16 bits per weight.

The letter after K is the size within the same bit count: S (small), M (medium), L (large). Files starting with IQ use an importance matrix to keep quality at even fewer bits. Bits per weight from llama.cpp's table for Llama 3.1 8B (llama.cpp · quantize, read Oct 17, 2026).

Three real files

Qwen3.6 27B on Hugging Face. Memory is the site's planning estimate for short conversations.

File sizes read Oct 16, 2026; GB = 10⁹ bytes.
FileSize (GB)32 GB+ Mac or PC (GB, est.)Graphics card (GB, est.)
Q4_K_M16.82923
Q8_028.64539
BF1653.87367

The memory columns use the site's Q4, Q8 and FP16 formula (how memory is estimated). The model also has a separate vision file of about 0.9 GB, needed only to read images.

Largest file per machine

The most precise of the three files whose 27B estimate stays within what the machine offers to a model.

RTX 4090 · 24 GB VRAM
Q4_K_MOffers 24 GB to a model
RTX 5090 · 32 GB VRAM
Q4_K_MOffers 32 GB to a model
Mac mini M4 Pro · 48 GB unified
Q4_K_MOffers 44 GB to a model
Mac Studio M5 Max · 128 GB unified
BF16Offers 104 GB to a model

How to choose

Most people
Q4_K_MOllama picks it by default when a Hugging Face repository has it.Hugging Face · Ollama
Memory to spare
Q8_0 · Q6_K · Q5_K_MCloser to the original, larger and slower to read.
Not enough memory
Q3_K_M · IQ3 · IQ2Smaller, with a larger quality loss; compare with a smaller model in Q4.

Smaller files have less to read for every word. In llama.cpp's example run on one machine, Q8_0 generated text more slowly than Q4_K_M (llama.cpp · quantize, read Oct 17, 2026). The site does not estimate speed. Benchmark scores on this site are for the publisher's original weights; a quantized file can score a little lower.