A dependency slows down. Clients time out and retry. The extra attempts increase load on the same dependency, which slows further. What began as a short disturbance can become a self-sustaining retry storm.
Retries are useful when another attempt has a reasonable chance of succeeding. They are harmful when they repeat expensive work that cannot currently complete, exceed the caller’s deadline, or apply a non-idempotent operation twice.
Count attempts across layers
If several layers retry independently, the leaf service can receive many attempts for one original request. A retry count that seems conservative in one component can be excessive across the call chain.
Record attempts separately from logical operations. Preserve a request or idempotency identity across appropriate retries so logs and metrics can explain amplification. A timeout is an uncertain outcome: the original operation may have completed even though the reply did not arrive.
For writes, define idempotency semantics with durable deduplication or an operation-specific protocol. Merely adding a random request ID does not implement deduplication.
Backoff needs a deadline and a budget
A common full-jitter policy chooses a random delay within a capped exponential envelope. The following illustrates the envelope in milliseconds; it is not a production retry implementation:
base_ms = 100
cap_ms = 2_000
for retry_index in range(6):
upper_ms = min(cap_ms, base_ms * 2 ** retry_index)
print(retry_index, upper_ms)
# A full-jitter delay is sampled from [0, upper_ms].The total deadline must include previous work, waiting, and future attempts. Before sleeping or retrying, check whether enough useful time remains. Bound attempts and aggregate retry traffic; a token or ratio-based retry budget can prevent every failing request from independently using its maximum.
Honor server guidance where the protocol provides it, but keep it inside the caller’s overall time budget. Cancellation should reach queued and downstream work so expired requests do not continue consuming resources indefinitely.
Queue capacity is a policy decision
An unbounded queue turns excess arrival rate into retained memory and increasing wait time. A bounded queue forces the system to choose: wait, reject, shed lower-priority work, or route elsewhere under a defined contract.
Blocking a producer can be valid backpressure, but blocking an event-loop thread or a worker needed to drain the same queue can deadlock progress. The bounded queue article makes shutdown and wait predicates explicit.
Load shedding is not an admission of failure. Rejecting work that cannot finish within its deadline can protect useful work and reduce wasted computation. The caller must understand that rejection and avoid converting it into an immediate retry flood.
Test recovery, not only overload
Use a disposable test service and controlled fault injection—not a shared production dependency. Compare baseline, slowed dependency, retry behavior, and recovery after the fault ends.
Observe logical request rate, attempt rate, queue wait, active concurrency, deadline expiry, rejection rate, downstream work, and successful completion latency. Distinguish application retries from transport retransmissions. Preserve failed outcomes in the results.
The purpose of the experiment is to establish whether the system sheds work and recovers, rather than merely surviving with an ever-growing queue.
Watch for hidden feedback loops
Connection pools, thread pools, and inference batch schedulers can all become queues. Increasing one pool’s limit can overload the next dependency. Retries may also synchronize after a common timeout or fixed delay; jitter helps spread attempts but does not create missing capacity.
Prefer a small number of clearly owned retry decisions, explicit deadlines, and observable queue limits. A reliable service knows when not to attempt more work. That is often more important than making its happy-path retry loop clever.
Further reading
Primary references behind this explanation. Consult documentation for the version and environment you actually run.