Received by the NIC is not received by the application
The NIC has received traffic, but the application is still waiting. Between those two observations are hardware queues, memory transfers, kernel execution contexts, protocol processing, socket queues, and scheduling.
There is no single CPU that necessarily owns the entire journey. A good diagnosis asks where the work is queued, which context performs it, and what an observation can actually establish.
Explore the receive path
The explorer follows a conventional local receive path for an established IPv4 TCP flow on a physical NIC. Select a stage with a pointer, or use Tab and Enter. Previous and Next move through the same conceptual sequence.
Select a stage to inspect the work, execution context, and observational tools.
NIC & queue selection
An arriving frame is classified by the NIC. On an RSS-capable setup, a flow hash and indirection table choose a receive queue. One busy flow is not automatically spread across every queue.
- Where does the work run?
- NIC hardware; this decision is not a userspace thread running on a CPU.
- What can I inspect?
ethtool -x eth0; ethtool -S eth0- What does this not prove?
- Driver support and statistic names vary. Hardware counters are observations, not proof of where the application stalls.
It is a teaching diagram, not a capture, emulator, or assertion that every Linux packet follows the same call graph. It generates no traffic.
Queues, execution contexts, and handoffs
Arrival is not application delivery
RSS can classify a flow into a receive queue. The device uses DMA to transfer data into buffers arranged by the driver. Descriptor rings track buffer and completion information; treating them as simple queues of whole application messages misses important ownership details.
An interrupt commonly schedules NAPI work rather than processing every packet to completion immediately. NAPI receive polling has a budget and can continue without another device interrupt when work remains. It usually runs in software-interrupt context, but threaded NAPI and busy polling are important exceptions.
Several handoffs can change locality
GRO may combine eligible packets before higher-layer processing. RPS/RFS can steer protocol processing to another CPU. The IP layer determines local delivery, with applicable routing and filtering behavior, before TCP makes accepted in-order data available through the socket.
A wakeup makes a task eligible to run; it does not mean the task ran immediately. IRQ affinity therefore does not pin the application, and an ACK does not prove the application consumed the data. TCP presents a byte stream, so recv() boundaries need not match transmitted segments or application messages.
The ordering in the explorer deliberately groups complex operations. XDP, VLANs, bridges, tunnels, forwarding, encryption, receive offload, and specialized driver paths can introduce additional branches.
Inspect an authorized Linux host
First use the explorer to explain a hypothetical stall at three points: the RX queue, kernel protocol processing, and the application. For each, name one relevant observation and one alternative explanation.
For a real inspection, use an authorized Linux lab. The following commands only read state, but tools, permissions, and driver support vary. Replace eth0 before running them:
ip -s link show dev eth0
ethtool -l eth0
ethtool -k eth0
cat /proc/interrupts
cat /proc/softirqs
ss -tinChannel counts describe configuration, not instantaneous utilization. Interface counters, interrupts, softirqs, and socket information describe different layers. Correlate them over the same workload interval; do not identify a bottleneck from one large counter.
The separate Packet Flow Explorer teaches a locally generated outbound path and optional NAT. It is intentionally not the same direction as this receive-path article.
Build a measurement around one workload
Before comparing receive-path configurations, record the NIC and driver, kernel, CPU topology, queue/IRQ configuration, offloads, connection count, payload distribution, and workload duration.
Measure useful throughput and client-observed latency alongside per-CPU work, drop counters, and retransmission deltas. Distinguish application payload from on-wire bytes. Keep observation overhead and counter scope visible, and retain errors rather than dropping them from the sample.
The explorer’s stage numbers are ordering labels, not time, CPU IDs, or measured costs.
Why the apparent bottleneck can be elsewhere
More queues and more polling are not automatically better. Spreading work can reduce a saturated queue while adding interrupts, cross-CPU handoffs, or remote-memory access.
Offloads can make packet captures look unlike wire traffic. A low aggregate CPU percentage can hide one saturated core. A scheduler delay or application lock can dominate after the network stack has done its work.
Start with a falsifiable hypothesis and change one variable at a time. Never use a diagram as evidence that an unobserved machine follows that exact path.
Further reading
Primary references behind this explanation. Consult documentation for the version and environment you actually run.