LLM Prefix Caching: What Can Be Reused and What Cannot

Many requests begin with the same system instructions or reference document. Recomputing that shared prefix for every request repeats work. Prefix caching can reuse compatible KV state so a later request skips processing the already-known beginning.

The useful unit is a token prefix with matching execution context, not a visually similar string and not a matching paragraph anywhere in the prompt.

Prefix identity is stricter than text resemblance

Tokenization, chat templates, special tokens, whitespace, model revision, adapters, and positional behavior can all affect reuse. If two prompts diverge near the beginning, a large identical suffix is generally not interchangeable with the KV state from the first prompt: that state depends on preceding context.

Request A: shared system tokens | shared document | question A
Request B: shared system tokens | shared document | question B
           reusable prefix                       | new work

This diagram describes the opportunity, not a promise that a particular runtime caches every boundary. Block alignment and model architecture can restrict the reusable portion.

Blocks need identity, ownership, and eviction

A block-oriented cache can associate physical KV blocks with logical prefixes and keep references while requests use them. Prefix identity may incorporate parent-prefix information and additional execution metadata, rather than hashing the block’s tokens in isolation.

An engine must distinguish active references from cached-but-reclaimable state. Retaining reusable blocks can reduce future computation but consume capacity that other requests need. Eviction policy turns expected reuse into a resource decision.

Sharing must also respect tenant isolation and confidentiality requirements. Do not treat cross-request reuse as permission to share state across arbitrary trust boundaries. Follow the engine’s documented isolation mechanisms and threat model.

A hit does not eliminate decode

Reusing prefix KV state primarily avoids repeated prefill for that prefix. The request still needs work for its uncached suffix and generated tokens. If output generation dominates end-to-end latency, a high prefix-hit rate may have a small effect on total time.

Likewise, a request-level hit count can conceal how much was reused. Reusing one block and reusing a long document are both “hits” under some metrics but represent very different saved work. Track matched tokens or blocks and the remaining prefill workload.

Build a controlled reuse experiment

Use an authorized local serving environment and pin model, tokenizer, template, runtime, precision, and hardware versions. Prepare a fixed common prefix with several distinct suffixes.

Measure a cold request, then requests with the same prefix, then requests that intentionally differ near the beginning. Preserve prompt/output lengths and generation settings as much as possible. Record the runtime’s cache statistics rather than assuming reuse occurred because the text looks similar.

Collect first-token latency, remaining prefill tokens, cache occupancy, evictions, throughput, and end-to-end latency. Distinguish engine startup warmup from prefix-cache warmup. A restart that clears the cache is a different experimental condition from an eviction under pressure.

Keep the cold-cache, warm-cache, and changed-prefix conditions separate when reporting the experiment.

Common reasons the expected improvement disappears

• A template or tokenization difference prevents the intended match.
• The reusable prefix is too short relative to decode work.
• KV pressure evicts blocks before the next reuse.
• The workload is already limited elsewhere, such as queueing or communication.
• A hybrid architecture or attention policy has additional reuse constraints.

Do not increase cache retention without checking admission and latency effects under load. Saved computation for future requests can compete with memory needed by current ones.

The KV calculator assumes independent sequences, so it intentionally does not subtract shared-prefix bytes. That conservative model is useful for understanding the baseline. A sharing-aware estimate needs actual prefix relationships, reference lifetimes, and the runtime’s allocation rules, not an arbitrary “cache savings” percentage.

Further reading

Primary references behind this explanation. Consult documentation for the version and environment you actually run.

1. vLLM: automatic prefix caching
2. vLLM: prefix caching design
3. Hugging Face: caching explanations