Linux TCP Buffer Tuning: rmem, wmem, Autotuning, and BDP

A fast NIC and a healthy server do not guarantee that one TCP connection can fill the path. The sender needs enough permitted data in flight to cover the time between transmission and acknowledgment. But enlarging every buffer is not a reliable fix: the connection may be congestion-limited, application-limited, CPU-limited, or paced by the peer.

Start by distinguishing a socket’s allocated buffer, the receiver’s advertised window, and the sender’s congestion window. They interact, but they are not the same number.

Use BDP to ask a question, not to choose a sysctl blindly

The bandwidth-delay product is a useful scale estimate:

# Hypothetical decimal units, not a benchmark.
bits_per_second = 10_000_000_000
rtt_seconds = 0.020
bytes_in_flight = bits_per_second * rtt_seconds / 8
print(int(bytes_in_flight))  # 25,000,000 bytes

This example describes 10 Gbit/s over a 20 ms round trip: 25 MB of data in flight, about 23.84 MiB. It does not say that a 25 MB socket setting will produce 10 Gbit/s. Framing overhead, competing traffic, congestion control, retransmissions, and receiver behavior remain relevant.

For a rough window-limited ceiling, divide usable in-flight bytes by RTT. Use consistent units and remember that RTT varies under queueing. A buffer can also hold unsent data, so allocated memory is not synonymous with bytes on the wire.

Linux has several related controls

tcp_rmem and tcp_wmem provide TCP-specific minimum, default, and maximum values. Receive autotuning is controlled by tcp_moderate_rcvbuf. Generic socket controls such as net.core.rmem_max and net.core.wmem_max are related but have different roles, particularly for application-requested socket buffers.

An application explicitly setting SO_RCVBUF or SO_SNDBUF changes how the corresponding automatic sizing behaves. Linux also accounts for bookkeeping when handling these options; a value returned by getsockopt should not be interpreted as an exact count of application payload bytes.

Window scaling is negotiated during connection establishment. Testing a changed setup with an old connection can therefore answer a different question from testing a newly established connection.

Observe one connection during a representative workload

On your authorized Linux host:

sysctl net.ipv4.tcp_rmem net.ipv4.tcp_wmem
sysctl net.ipv4.tcp_moderate_rcvbuf
sysctl net.core.rmem_max net.core.wmem_max
ss -tinm

The precise ss fields depend on the kernel and iproute2 version. Record RTT, congestion/window information where available, retransmissions, socket memory, and queue occupancy for the target flow. Establish whether the application uses explicit socket options.

If the peer reads slowly, a constrained receive window may be the effect of application backpressure rather than a bad buffer default. If one CPU is saturated processing packets, larger buffers can add waiting time without adding service capacity.

Compare one change at a time

Save the original configuration and use a disposable or controlled environment. Keep payload size, connection count, direction, warmup, and load generation fixed. Record useful throughput, latency distribution, retransmissions, CPU work, and memory footprint before and after the change. Repeat runs; do not select only the best one.

This article intentionally provides no universal sysctl -w values and claims no measured speedup. Defaults and memory accounting vary with kernel version and environment. Inspect the documentation for the version you actually deploy.

Larger buffers have a budget

A large per-socket allowance multiplied by many connections can become a substantial memory commitment. More queued data can also hide overload until latency or memory pressure becomes severe. Admission control and application queue bounds belong in the same discussion as network buffers.

If the observation suggests loss rather than a window limit, use the packet-drop diagnosis guide. If CPU placement is the problem, investigate steering and locality before changing buffer sizes.

Further reading

Primary references behind this explanation. Consult documentation for the version and environment you actually run.

1. Linux kernel: TCP IP sysctl documentation
2. Linux tcp(7): buffer sizes and window scaling
3. Linux socket(7): SO_RCVBUF and SO_SNDBUF