A static batch of requests often finishes unevenly. One sequence ends after a few tokens while another continues for hundreds. If the engine waits for the entire batch before admitting replacements, available capacity can remain unused.
Continuous batching changes the scheduling boundary. Requests can join and leave as generation progresses, rather than forcing every member to share one lifetime. The exact admission policy differs across engines, but the central idea is to reconsider work at finer granularity.
A slot is not a fixed amount of work
Imagine three requests: A has two output tokens remaining, B has six, and C has one. After the next decoding iteration, C can finish; later A finishes. New work can replace them without waiting for B.
That is an illustrative schedule, not a measured trace. Real scheduling must account for prompt work, cache blocks, token limits, model shapes, and execution constraints. A long prefill is not interchangeable with one decode token merely because both belong to one request.
iteration 1: A B C
iteration 2: A B D
iteration 3: B D EThe letters show active requests, not equal execution time or a guaranteed fairness policy.
Memory and token budgets constrain admission
Every admitted request can retain KV state as it generates. An engine may have room for another request now but not enough headroom for all active requests to grow indefinitely. Policies for preemption, recomputation, swapping, or rejection affect both reliability and latency.
The scheduler also needs a work budget. Bounding tokens processed per iteration can limit large prefills, but a token budget is not a universal time budget: token cost depends on phase, context, architecture, and implementation.
Efficient paged allocation can make memory available more flexibly. It does not remove the need to choose whom to admit.
Throughput is not the same as goodput
A configuration that processes more tokens per second can still be worse for an interactive application if first-token or inter-token latency exceeds the service objective. Define useful completed work under the required latency and error constraints, not just aggregate token production.
Observe at least:
- Offered request rate and completed request rate.
- Prompt and output token distributions.
- Time to first token, inter-token latency, and end-to-end latency.
- Waiting queue depth, active requests, and KV occupancy.
- Preemptions, recomputations, cancellations, rejections, and timeouts.
Keep unsuccessful outcomes visible. A run that rejects difficult requests can appear faster when only successful completions are summarized.
Design a comparison that exposes saturation
Use the same engine/model revisions, hardware, precision, and arrival workload. Warm the runtime consistently. Increase offered load in controlled steps while keeping length distributions stable. At each step, collect repeated runs and distributions rather than one average.
A closed-loop client that waits before issuing the next request reduces offered load during slow periods. An open-loop generator preserves the arrival schedule and reveals queue growth more directly. Choose deliberately and document the choice.
This article reports no server measurement. The schedule above teaches resource reuse; it does not quantify a speedup.
Overload still needs a policy
Continuous batching cannot make demand disappear. An unbounded waiting queue can turn a temporary burst into long delays, stale requests, and memory pressure. Deadlines, queue limits, admission control, and cancellation propagation remain necessary.
Larger batches may improve utilization while increasing per-request waiting or iteration time. Long prefills can interfere with decode smoothness; aggressive decode preference can delay new prompts. Fairness and priority choices should be stated, not inferred from a throughput chart.
Finally, benchmark the workload you intend to serve. A homogeneous offline generation test may be appropriate for bulk processing and misleading for a mixed interactive service. The scheduling mechanism is useful precisely because request lifetimes differ—your test should reflect that difference.
Further reading
Primary references behind this explanation. Consult documentation for the version and environment you actually run.