Skip to content
Local AI Wizard
Your machine
Theme

Why is one computer faster than another at running an LLM?

Memory bandwidth, the file size and what published llama.cpp results show.

Speed · updated

Short answer

Mostly memory bandwidth: to write each word, the computer reads the whole model from memory.

  • Fast memory: RTX 4090 · 24 GB VRAM (1,008 GB/s) writes 189 tokens/s with a 7B file.
  • Slow memory: Radeon RX 9060 XT · 16 GB VRAM (320 GB/s) writes 65 with the same file.
  • In short: 3.2× the bandwidth gave 2.9× the speed.
  • To go faster: A smaller file, the whole model in fast memory, or more bandwidth.

Published community measurements (llama.cpp scoreboards) and manufacturer bandwidth; Methodology

Why bandwidth

A language model writes one token (a word or part of a word) at a time, and to choose each token the computer reads every weight of the model once. The arithmetic is quick; moving the weights from memory to the chip is what takes the time. So the speed at which a machine writes text is set mostly by its memory bandwidth, the gigabytes per second its memory can deliver, divided by the size of the file it reads.

The llama.cpp reference file used below is 3.82 GB (llama-bench reports 3.56 GiB for the 6.74B-parameter Llama 7B in Q4_0). A machine with 1,008 GB/s could at most read it about 264 times a second; that rough ceiling is a teaching bound, not a measurement. Real runs reach 57% to 84% of it, because the program also reads the conversation's cache and the chip never streams memory perfectly. (llama.cpp Apple Silicon summary · community result, read Oct 25, 2026)

Measured results

Token generation (128 tokens, batch 1) of the same 7B Q4_0 file, sorted by bandwidth.

Memory bandwidth and published llama.cpp speed of the 7B Q4_0 reference file, by machine.
MachineBandwidth (GB/s)Measured (tokens/s)Rough ceiling (tokens/s)Share of ceilingBackend
Mac mini M5 Pro 24GB30767.68084%Metal
Radeon RX 9060 XT 16 GB32065.48478%ROCm (HIP)
Mac Studio M5 Max 48GB614119.916175%Metal
Radeon RX 9070 XT 16 GB64097.316758%ROCm (HIP)
RTX 4070 Ti SUPER 16 GB672132.917676%CUDA
RTX 5070 Ti 16 GB896182.423478%CUDA
RTX 3090 24 GB936161.924566%CUDA
Radeon RX 7900 XTX 24 GB960170.125168%ROCm (HIP)
RTX 4090 24 GB1,00818926472%CUDA
Mac Studio M5 Ultra 512GB1,200179.131457%Metal

The order of the speed column follows the order of the bandwidth column almost row for row, across NVIDIA, AMD and Apple chips. Graphics cards and Macs land on the same line: a Mac is not slow because it is a Mac, and a graphics card is not fast because it is a graphics card; the one with more bandwidth writes faster.

Sources: (llama.cpp CUDA scoreboard · community result, read Sep 25, 2026) (llama.cpp Apple Silicon summary · community result, read Oct 25, 2026) Bandwidth is each manufacturer's specification, linked on the machine's page.

Reading vs writing

Writing (token generation)
Limited by memory bandwidthOne token at a time, the whole file read each time. This is the speed you feel while the answer appears.
Reading your prompt (prompt processing)
Limited by computeThe whole prompt is processed in parallel, so GPU cores matter more here; long documents wait on this part.
Bigger file
Proportionally slowerA 14B model in Q4 is about twice the file of a 7B, so it writes at about half the speed on the same machine.
MoE models
Read only the active part per tokenThey still need memory for every expert, but each token reads only the active ones, so they write faster than their total size suggests.

Getting more speed

Three levers follow from the rule. A smaller file (a lower quantization or a smaller model) is read faster. Keeping the whole model in the fast memory avoids the much slower path through PC RAM (offloading). And for the same memory size, a machine with more bandwidth writes faster. The site's planning estimates never turn bandwidth into a speed figure; measured speeds appear only where a source published them, on each machine's page.

llama.cpp runs a model across GPU and CPU when it does not fit: (llama.cpp README, read Oct 17, 2026)