Skip to content
Local AI Wizard
Your machine
Theme

How much memory does a long conversation use?

Context length and the KV cache, computed from real model configurations.

Memory · updated

Short answer

~4.8GB

More than the file suggests: the conversation is kept in a KV cache that grows with every token. For Qwen3 8B, up to about 4.8 GB.

  • Short chat (4,096 tokens): 0.6 GB.
  • Full context (32,768 tokens): 4.8 GB, almost the size of the 5 GB Q4_K_M file.
  • To keep it small: Set a shorter context in your app, or start a new chat.

Computed from the model's config.json at 16-bit cache precision. Methodology

What the KV cache is

The context is everything the model can see at once: your messages, its answers, any document you pasted, counted in tokens (a word or part of a word). To avoid redoing work, the program stores two vectors per token in every attention layer, the keys and values: the KV cache. Its size is the number of layers × the key-value heads × the head size × 2 (keys and values) × 2 bytes per number, for every token in the conversation.

For Qwen3 8B the collected config.json gives: 2 (K and V) x 36 global layers x 8 KV heads x 128 head size x 2 B (FP16) per token, 144 KiB per token. (Qwen/Qwen3-8B config.json, read Oct 23, 2026)

Cache size by context

KV cache in GB at 16-bit precision, from each model's config.json. A dash is beyond the model's published context.

KV cache size by context length for three reviewed models.
Context (tokens)Qwen3 8B (GB)Qwen3 14B (GB)Gemma 4 12B (GB)
4,0960.60.670.4
16,3842.422.680.6
32,7684.835.370.87
131,072Beyond this model's published 32,768-token contextBeyond this model's published 32,768-token context2.48

Gemma 4 12B grows far more slowly because most of its layers only look at a sliding window of the last 1,024 tokens; only its few global layers keep every token. Its config.json formula: 2 (K and V) x 8 global layers x 1 KV heads x 512 head size x 2 B (FP16) per token; sliding layers: 40 x 8 x 256, capped at 1024 tokens. Collected Oct 23, 2026.

A 12 GB card, worked out

Qwen3 8B Q4_K_M on RTX 3060 · 12 GB VRAM: file × 1.2, plus the cache, plus the card's 2 GB reserve, rounded up.

4,096 tokens
Needs 9 GBWithin the card's 12 GB
16,384 tokens
Needs 11 GBWithin the card's 12 GB
32,768 tokens
Needs 13 GBMore than the card's 12 GB
The file
5.03 GBQ4_K_M in Qwen/Qwen3-8B-GGUF, as published on the Hub.Qwen/Qwen3-8B-GGUF

So on RTX 3060 · 12 GB VRAM this file works up to about 16,384 tokens of conversation by the site's estimate; a longer chat needs a smaller file, a smaller cache or more memory.

Keeping it small

Set a shorter context
The cache is reserved for the window you chooseOllama, LM Studio and llama.cpp let you set it; most chats never need the model's maximum.
Start a new chat
An empty context has an empty cacheLong threads slow down and fill memory; a fresh one starts small.
Quantize the cache
Fewer bits per number, a smaller cachellama.cpp can store the cache at a lower precision; the site's figures stay at 16 bits, the default.
Pick a model with sliding layers
Like Gemma 4 12BModel pages use the same config.json formula when they pick the file that fits a context.