KV Cache Memory Explained: Bytes per Token and GPU Budget

Why a longer context uses more memory

Longer context is not just a model capability. It is a resource commitment. In a conventional autoregressive transformer, previously computed keys and values can be retained so later decoding steps do not need to recreate those projections.

That retained state is the KV cache. Its size depends on the shape of the cached tensors, not simply on the model’s parameter count.

Derive the memory for one token

For each token, each attention layer retains both a key and a value. If there are L layers, Hkv key/value heads, D elements per head, and B bytes per element:

bytes_per_token = 2 * L * Hkv * D * B

The factor of two is for keys and values. With a hypothetical 32-layer model, 8 KV heads, a head dimension of 128, and two-byte elements:

2 * 32 * 8 * 128 * 2 = 131,072 bytes
                         128 KiB per token

This is a hypothetical geometry, not a claim about a named model or deployed service. A sequence of 8,192 tokens uses 1 GiB of raw KV tensor storage with these inputs.

From tensor geometry to allocated blocks

Count KV heads, not query heads

Grouped-query attention can have more query heads than KV heads. Multiplying by the query-head count overestimates the cache in that case. Multi-query attention reduces the KV-head dimension further.

The calculator assumes every layer has the same geometry. Hybrid models, sliding-window layers, compressed latent representations, and layer-specific cache policies need a different calculation.

Concurrency multiplies the commitment

If S independent sequences each retain T tokens:

aggregate_bytes = bytes_per_token * T * S

Eight independent sequences of the example above need 8 GiB for their raw KV tensors. The formula assumes no shared prefixes. A serving system with prefix reuse may share physical blocks, but sharing must be modeled explicitly rather than assumed.

Allocated is not the same as used

An allocator may reserve tokens in fixed-size blocks. With K tokens per block, a sequence receives:

blocks_per_sequence = ceil(T / K)
allocated_bytes = blocks_per_sequence * K * bytes_per_token

A partially filled final block creates internal slack. Increasing block size can increase that slack; it does not change the raw tensor bytes per live token. The visualizer distinguishes logical usage from this rounded allocation.

Check the calculation in the visualizer

Open the KV Cache Visualizer. Double context length, then double sequence count. Compare the raw and block-allocated values. Finally, choose a context length that is not divisible by the block size.

The implementation has independent unit tests for the tensor formula, block rounding, and invalid inputs.

What to record in a real inference run

The 128 KiB-per-token and 1 GiB-per-sequence figures above are calculated, not measured GPU consumption. Record geometry, precision, context, concurrency, and block size when comparing configurations. Compare allocated bytes with the live-token requirement to isolate internal slack.

To turn the calculated examples into a runtime experiment, also record the model revision, serving engine, GPU, allocator, and observed peak memory.

The budget is not an out-of-memory guarantee

The lab’s budget is memory available to the KV cache, not total GPU capacity. Model weights, activation peaks, temporary buffers, runtime metadata, and allocator overhead must fit elsewhere. FP8 or INT8 scale metadata is also excluded.

That makes this a sizing model, not an out-of-memory predictor or a throughput benchmark. Prefix sharing, sliding windows, hybrid architectures, and sharding require different assumptions. Use the model to understand sensitivity, then measure the actual runtime.

Lower cache precision reduces element bytes but may introduce scales, different kernels, and quality tradeoffs. Quantizing weights does not automatically quantize the KV cache. Check the two configurations separately.

Likewise, the number of active requests is not the same as the configured maximum batch size. Requests arrive with different prompt lengths and generate different numbers of tokens. A useful capacity estimate tracks their live token totals rather than multiplying the longest possible request by every possible slot. Keep worst-case provisioning and observed usage as separate numbers.

Further reading

Primary references behind this explanation. Consult documentation for the version and environment you actually run.

1. Hugging Face: How caching works
2. PagedAttention: the original paper