Linux Internals / Under the Hood

Linux Memory Reclaim and the OOM Killer: Diagnose the Right Limit

Distinguish reclaim, cache eviction, swapping, cgroup limits, and OOM kills. Use memory and pressure observations without treating low free memory as a diagnosis.

Palash Vij / Published / 3 min read
Source-backed explainer Linux Memory Reclaim Observability

A service slows down before it is killed for memory use. The host still appears to have available memory. Both observations can be true: the service may be constrained by its memory cgroup, and reclaim can consume time before a limit leads to an OOM event.

Begin by identifying the scope of the pressure. Host-wide free memory, a container limit, a process’s resident set, and the memory available for a particular allocation are different quantities.

Low free memory is not automatically bad

Linux uses otherwise idle memory for caches. Some cached pages can be reclaimed when applications need memory, so a small MemFree value alone does not prove trouble. MemAvailable is a more useful estimate for many host-level questions, but it is still not a promise that every allocation will succeed.

Memory differs in reclaimability. Clean file-backed pages can often be discarded and reread. Dirty pages need appropriate writeback. Anonymous memory may require swap to be reclaimed without terminating its owner. Pinned or otherwise unreclaimable memory changes the available options.

The reclaim algorithms and heuristics evolve; avoid treating one historical LRU description as a complete account of every modern kernel.

Reclaim can become application latency

Background reclaim tries to maintain useful free capacity. Under pressure, allocating tasks can enter direct reclaim or other slow allocation paths. Even before an OOM kill, this work can increase request latency or reduce throughput.

Swapping is not identical to an OOM condition, and disabling swap does not prevent memory pressure. It changes which pages can be reclaimed and how the system responds when demand exceeds available resources.

Compaction addresses fragmentation and contiguous allocation needs. A machine can have free pages yet struggle with a particular allocation shape. Total bytes alone do not describe that condition.

A cgroup can fail while the host is healthy

With cgroup v2, memory controls include accounting, pressure behavior, and limits. memory.high and memory.max have different meanings; crossing a high boundary can trigger reclaim pressure without being the same as a hard maximum.

Inspect the workload’s actual cgroup rather than assuming a fixed container path. Events such as high, max, oom, and oom_kill distinguish related but non-identical outcomes. An OOM event count does not always equal a count of processes killed.

cat /proc/self/cgroup
grep -E '^(MemAvailable|Cached|Dirty|Writeback|SwapFree):' /proc/meminfo
cat /proc/pressure/memory
vmstat 1 5

Then inspect memory.current, memory.stat, memory.events, and the configured limits in the identified cgroup. The commands above are read-only; visibility depends on permissions and namespaces.

Pressure measures time lost, not just bytes

Pressure Stall Information can help identify time during which work is delayed by resource pressure. It complements usage gauges: a cache-heavy workload may use plenty of memory without stalling, while a smaller constrained workload suffers repeated reclaim.

Correlate pressure with allocation rate, page faults, application queue depth, and latency over the same interval. Check logs and cgroup events for actual kill evidence. A container exit code is a clue, not a complete root-cause analysis by itself.

Fix the demand or the justified budget

Potential remedies include bounding queues, reducing retained state, correcting a leak, changing workload concurrency, or assigning a larger limit when the system has capacity. Raising a limit without understanding demand can defer the failure and affect other workloads.

Do not reproduce memory exhaustion on a shared machine. Use an isolated cgroup or disposable VM with explicit resource limits and a recovery plan to apply this diagnostic method.

The key distinction is between usage and pressure. Track both, and connect them to the latency distribution before concluding that an impressive-looking memory graph explains the failure.

Further reading

Primary references behind this explanation. Consult documentation for the version and environment you actually run.

  1. Linux kernel: cgroup v2 memory interface
  2. Linux kernel: Pressure Stall Information
  3. Linux kernel: virtual memory controls
Follow the thread

Related reading

Search the notebook

Search articles, projects, and experiments. Everything stays on your device.

palash / notebook

A small doorway into the site. No shell, no remote execution.

Welcome. Type help to find your way around.

lab →experience →contact →Ctrl / Cmd + K