How much memory does a long conversation use?
Context length and the KV cache, computed from real model configurations.
Short answer
~4.8GB
More than the file suggests: the conversation is kept in a KV cache that grows with every token. For Qwen3 8B, up to about 4.8 GB.
- Short chat (4,096 tokens): 0.6 GB.
- Full context (32,768 tokens): 4.8 GB, almost the size of the 5 GB Q4_K_M file.
- To keep it small: Set a shorter context in your app, or start a new chat.
Computed from the model's config.json at 16-bit cache precision. Methodology
What the KV cache is
The context is everything the model can see at once: your messages, its answers, any document you pasted, counted in tokens (a word or part of a word). To avoid redoing work, the program stores two vectors per token in every attention layer, the keys and values: the KV cache. Its size is the number of layers × the key-value heads × the head size × 2 (keys and values) × 2 bytes per number, for every token in the conversation.
For Qwen3 8B the collected config.json gives: 2 (K and V) x 36 global layers x 8 KV heads x 128 head size x 2 B (FP16) per token, 144 KiB per token. (Qwen/Qwen3-8B config.json, read Oct 23, 2026)
Cache size by context
KV cache in GB at 16-bit precision, from each model's config.json. A dash is beyond the model's published context.
| Context (tokens) | Qwen3 8B (GB) | Qwen3 14B (GB) | Gemma 4 12B (GB) |
|---|---|---|---|
| 4,096 | 0.6 | 0.67 | 0.4 |
| 16,384 | 2.42 | 2.68 | 0.6 |
| 32,768 | 4.83 | 5.37 | 0.87 |
| 131,072 | Beyond this model's published 32,768-token context | Beyond this model's published 32,768-token context | 2.48 |
Gemma 4 12B grows far more slowly because most of its layers only look at a sliding window of the last 1,024 tokens; only its few global layers keep every token. Its config.json formula: 2 (K and V) x 8 global layers x 1 KV heads x 512 head size x 2 B (FP16) per token; sliding layers: 40 x 8 x 256, capped at 1024 tokens. Collected Oct 23, 2026.
A 12 GB card, worked out
Qwen3 8B Q4_K_M on RTX 3060 · 12 GB VRAM: file × 1.2, plus the cache, plus the card's 2 GB reserve, rounded up.
- 4,096 tokens
- Needs 9 GBWithin the card's 12 GB
- 16,384 tokens
- Needs 11 GBWithin the card's 12 GB
- 32,768 tokens
- Needs 13 GBMore than the card's 12 GB
- The file
- 5.03 GBQ4_K_M in Qwen/Qwen3-8B-GGUF, as published on the Hub.Qwen/Qwen3-8B-GGUF
So on RTX 3060 · 12 GB VRAM this file works up to about 16,384 tokens of conversation by the site's estimate; a longer chat needs a smaller file, a smaller cache or more memory.
Keeping it small
- Set a shorter context
- The cache is reserved for the window you chooseOllama, LM Studio and llama.cpp let you set it; most chats never need the model's maximum.
- Start a new chat
- An empty context has an empty cacheLong threads slow down and fill memory; a fresh one starts small.
- Quantize the cache
- Fewer bits per number, a smaller cachellama.cpp can store the cache at a lower precision; the site's figures stay at 16 bits, the default.
- Pick a model with sliding layers
- Like Gemma 4 12BModel pages use the same config.json formula when they pick the file that fits a context.