An inference server can deliver a fast first token and then stream slowly, or spend a long time preparing the prompt and stream smoothly afterward. One throughput number hides that distinction.
Autoregressive inference has two useful phases. Prefill processes the input tokens and builds the state needed for generation. Decode repeatedly generates new tokens, using previously cached keys and values. Both run the model, but their shapes and opportunities for parallel work differ.
The same weights, different work shapes
During prefill, many prompt tokens can participate in relatively large matrix operations. That often offers more arithmetic work per weight read and better hardware utilization than a small decode batch. It is not a guarantee: short prompts, implementation overhead, attention cost, and hardware can change the balance.
During ordinary decode, each active sequence advances incrementally. A small batch may reuse each weight across relatively little work while also reading growing KV state. Memory movement can dominate even though the GPU reports it is busy.
request arrival -> queue -> tokenize -> prefill -> first token
-> decode -> token -> decode -> tokenTime to first token includes more than GPU prefill if measured at the client. Queueing, preprocessing, transport, and scheduling belong to that observation too.
Memory-bound is a condition, not a permanent model label
Arithmetic intensity compares useful operations with bytes moved. A rough roofline model asks whether achievable compute or memory bandwidth is the tighter bound. The relevant traffic includes weights, KV reads/writes, activations, and implementation overhead.
Increasing batch size can reuse weights across more tokens and change the bottleneck. Longer context can increase attention-state traffic. Quantization changes byte counts and kernels, while tensor parallelism adds communication. A workload can move between limiting resources as these variables change.
GPU utilization alone does not separate these cases. A device can be continuously active while doing inefficient, memory-stalled, or communication-heavy work. Useful output under a latency objective is a better measure than a single busy percentage.
Define the metrics before comparing engines
Record client-observed time to first token, inter-token latency or time per output token, end-to-end latency, completed requests, and useful output tokens per second. State whether prompt tokens are included in the throughput number.
Use the same model and tokenizer revision, hardware, precision, serving engine version, attention backend, prompt/output distributions, concurrency, and arrival process. Separate warmup from the measured interval. Include rejected, timed-out, and cancelled requests rather than quietly measuring only successes.
For an exploratory matrix, vary prompt length while holding output length and concurrency stable. Then vary concurrency while holding lengths stable. This isolates more information than changing every setting together.
This comparison method helps separate the phase, workload, and hardware effects in a GPU benchmark.
Scheduling couples the phases
A large prefill can compete with active decode work. Chunked prefill divides prompt processing into smaller pieces so the scheduler can interleave work, though its exact policy is engine-specific. The tradeoff can involve both first-token latency and streaming smoothness.
Disaggregated prefill and decode place the phases on separate resources. That can support specialized scheduling but requires transferring or sharing the necessary state. Transfer bandwidth, latency, memory ownership, failure recovery, and network topology become part of the design. Separation does not make data movement free.
Use capacity estimates without mistaking them for speed estimates
The KV memory calculator predicts retained tensor bytes under explicit assumptions. It does not predict tokens per second. Two configurations with equal KV footprints can have very different compute shapes, communication costs, and kernels.
When a serving change improves throughput, ask which resource was freed and what happened to latency. More batching, less recomputation, better allocation, and faster kernels are different explanations that require different evidence.
Further reading
Primary references behind this explanation. Consult documentation for the version and environment you actually run.