Estimate KV-cache memory before loading a model
Compute GQA cache bytes from layer count, KV heads, head dimension and retained tokens. Use a separate latent-cache calculation for MLA models.
For ordinary multi-head or grouped-query attention, estimate the KV cache with 2 × layers × kv_heads × head_dim × bytes × tokens × batch. Count the tokens retained by every live sequence, including the prompt and generated continuation. This estimates cache payload, not the memory required to load and run the whole model.
Start with the model config
Read num_hidden_layers and num_key_value_heads from the exact checkpoint’s config.json. For these Qwen checkpoints, head_dim is hidden_size / num_attention_heads. Do not substitute the query-head count for the KV-head count: grouped-query attention shares keys and values across several query heads.
These worked examples use 8,192 retained tokens per sequence, batch 1 and two bytes per cached element, appropriate for FP16 or BF16:
| Checkpoint | Layers | KV heads | Head dimension | Bytes per token | Cache GiB |
|---|---|---|---|---|---|
| Qwen2.5-7B-Instruct | 28 | 4 | 128 | 57,344 | 0.4375 |
| Qwen2.5-14B-Instruct | 48 | 8 | 128 | 196,608 | 1.5000 |
| DeepSeek-V3, compressed MLA | 61 | Latent cache | 512 + 64 | 70,272 | 0.5361 |
The inputs come from the official 7B config, 14B config, and DeepSeek-V3 config. These are checkpoint-specific examples, not defaults for later members of those model families.
Run a tiny calculator
Save this as cache.py and run python3 cache.py. Change tokens and batch to your planned workload:
# cache.py: cache payload only; GiB means 2**30 bytes.
tokens, batch, element_bytes = 8192, 1, 2
for name, layers, kv_heads, head_dim in [
("Qwen2.5-7B", 28, 4, 128),
("Qwen2.5-14B", 48, 8, 128),
]:
per_token = 2 * layers * kv_heads * head_dim * element_bytes
gib = per_token * tokens * batch / 2**30
print(f"{name}: {per_token} bytes/token; {gib:.4f} GiB")
latent_rank, rope_dim, layers = 512, 64, 61
per_token = layers * (latent_rank + rope_dim) * element_bytes
print(f"DeepSeek-V3 MLA: {per_token} bytes/token; "
f"{per_token * tokens * batch / 2**30:.4f} GiB")
Output:
Qwen2.5-7B: 57344 bytes/token; 0.4375 GiB
Qwen2.5-14B: 196608 bytes/token; 1.5000 GiB
DeepSeek-V3 MLA: 70272 bytes/token; 0.5361 GiB
Treat MLA separately
DeepSeek-V3 can retain one compressed KV latent plus a positional key component per layer. Its compressed payload is layers × (kv_lora_rank + qk_rope_head_dim) × bytes × tokens × batch. There is no leading factor of two: the latent represents both key and value information. The reference implementation also has an expanded attention path; the number above assumes the compressed path and two-byte cache storage. FP8 weights do not establish the cache dtype.
Leave room for the runtime
Add weights, activations, attention workspaces and allocator overhead separately. Paged caches round allocations to blocks; static caches may reserve the configured maximum. Prefix sharing, sliding windows, cache quantization and tensor-parallel placement can change actual allocations. For unequal sequence lengths, replace tokens × batch with the sum of retained sequence lengths before accounting for sharing.
Go deeper into KV-cache mechanics and compressed long-context state.