Local AI basics: what do 9B, Q4 and VRAM mean?
The ideas behind every number on this site, one short section each.
Short articles, one question each, answered with the site's own numbers.
The ideas behind every number on this site, one short section each.
Model weights, working room, the conversation and your system, added up.
Install one app, pick a file that fits your memory and run your first model.
About 29 GB for a 27B model in Q4 on a 32 GB+ Mac or PC, 23 GB on a graphics card.
A graphics card uses only its VRAM; a 128 GB Mac offers about 104 GB to a model.
Yes, small ones: an 8 GB graphics card holds models up to about 8B parameters in Q4.
A KV cache that grows per token: 4.8 GB at 32,768 tokens for Qwen3 8B, almost its Q4 file size.
13 reviewed models run well on 8 GB; Granite 4.2 3B leads by LMArena.
21 reviewed models run well on 16 GB; Granite 4.2 3B leads by LMArena.
35 reviewed models run well on 32 GB; Gemma 4 31B leads by LMArena.
8 of 21 image models with a documented path fit 16 GB; FLUX.2 [klein] 4B needs 13 GB.
Fewer bits per number: Qwen3.5 9B drops from 17.9 GB to 5.7 GB as Q4_K_M.
Start with Q4_K_M: 16.8 GB for Qwen3.6 27B, against 53.8 GB in BF16.
GGUF opens everywhere; MLX only on Apple silicon Macs. Qwen3 4B: 2.5 GB vs 2.3 GB at 4 bits.
GGUF and MLX are file formats of the same model; fine-tunes and abliterated copies are changed models.
One release line in several sizes: in Qwen3.5, 9B needs about 15 GB in Q4, 35B about 35 GB.
Memory follows the total: Qwen3.5 35B A3B needs about 35 GB in Q4, planned from 35B, not 3B.
195 of 260 reviewed models allow commercial use freely; 16 forbid it. Check the weights' licence.
Embeddings power search and RAG; Qwen3-Embedding 0.6B needs about 3 GB, so almost any machine runs one.
6 of 16 rated vision models run well on a 24 GB card; Qwen3.8 27B leads them.
A token is a piece of a word; a Mac mini M4 writes 24.1 per second with a 7B model, faster than you read.
Memory bandwidth: the same 7B file runs at 65 tokens/s on 320 GB/s and 189 on 1,008 GB/s.
It runs slowly from PC RAM: an 8 GB card with 32 GB of RAM reaches models up to 32 GB.
Install one app and pick a file that fits: Qwen3.5 9B in Q4_K_M is a 5.7 GB download.
27B parameters in Q4 take about 17 GB; memory, not speed, decides what runs.
Three sources, three scales, never mixed: 22 models have LiveBench, 25 LMArena and 16 embeddings MTEB.
Card for speed, Mac for size: an RTX 4090 24 GB runs up to 27B in Q4, a 64 GB Mac Studio up to 49B.
Whisper large-v3-turbo needs about 6 GB of GPU memory; large-v3 about 10 GB.
Each topic opens its section of the Local AI basics article.