Lab 01 / AI infrastructure
Context has a cost.
A conventional decoder's memory footprint, one dimension at a time. Change the geometry and watch the budget respond.
Interactive KV cache calculator
Model geometry
A memory budget, made visible
128 KiBPer token
1 GiBPer sequence
8 GiBAll sequences
KV budget utilization
Within budget33.3%
Physical block allocation
8 GiBRounded allocation
0 BInternal slack
Unused space in final blocks
2 × 32 layers × 8 KV heads × 128 dimensions × 8192 tokens × 2 bytes × 8 sequences
The model, not the machine.
M = 2 × layers × KV heads × head dimension × tokens ×
bytes per element × sequences
All capacities use binary units: 1 GiB = 1,073,741,824 bytes. Query heads do not multiply KV storage; they must be divisible by the KV-head count in this grouped-query model.
Assumptions
- All sequences have equal context length; all layers use identical, full-attention cache geometry.
- Blocks are allocated independently per sequence. No prefix sharing, eviction, sliding windows, or cross-device sharding.
- Budget means memory reserved for KV, excluding weights, activations, metadata, runtime buffers, and quantization scales.
- Allocated bytes round each sequence up to a whole block. This models internal slack, not external fragmentation.