Debugging a Hung NFS Mount: Timeouts, Retries, and D-State Waits

A shell command touches an NFS path and stops returning. Starting more commands against the same path may only create more blocked tasks. Before changing a mount option, identify what is waiting and preserve the evidence needed to distinguish recovery from failure.

This is a diagnostic playbook for an authorized Linux environment, not a procedure for forcefully repairing a production filesystem. Unmounts, process termination, network changes, and cache invalidation can affect data integrity and require workload-specific decisions.

First separate the waiting process from the failing dependency

Establish which operation is blocked and which mount it addresses. A long listing can wait on metadata or name resolution; a buffered write can wait on writeback or throttling. A process in D state indicates an uninterruptible kernel wait, not specifically an NFS fault.

On a lab client, begin with observations that do not walk the unhealthy directory:

findmnt -t nfs,nfs4
nfsstat -m
ps -eo pid,stat,wchan:32,comm
cat /proc/self/mountstats

wchan is a clue about a wait location, not a complete stack trace. Permissions and kernel configuration can hide details. Mount statistics belong to the observing process’s mount namespace; a container may see a different view.

If you already know the affected process, a carefully scoped trace or kernel stack can help. Attaching instrumentation has overhead and may expose paths or data. Avoid recursively listing the filesystem as a “health check” when path operations are already blocked.

Understand what the mount promises

With normal hard behavior, the client keeps retrying requests rather than returning an error merely because the server is temporarily unavailable. That protects a different contract from a soft mount, where requests eventually fail according to retry behavior.

soft is not a universal responsiveness switch. Linux documentation warns of possible silent data corruption in some cases. An application that assumes writes either succeed or remain pending may not handle ambiguous partial progress safely.

timeo is expressed in deciseconds, not seconds. retrans participates in recovery behavior; it is not a simple overall wall-clock deadline for every operation. TCP and UDP have different timeout behavior. The old intr option is ignored by modern Linux kernels, so advice built around it is often obsolete.

Follow a decision tree

1. Is the wait really on NFS? Correlate the process operation and mount. Rule out user/group lookup or unrelated application locks.
2. Is the server endpoint reachable for the required service? A ping response is not proof of a working NFS service. Conversely, blocked ICMP is not proof that NFS is down.
3. Are transport or RPC counters changing? Compare interval deltas. TCP retransmissions and RPC retransmissions are different signals.
4. Does the server receive and complete requests? Correlate server health, storage latency, permissions, and resource pressure through authorized observations.
5. Did identity or state change? A server replacement, export change, or stale file handle can require different recovery from a transient network outage.

Document the current hypothesis and what evidence would disprove it. Multiple failures can produce the same shell symptom.

Use operation-level measurements

nfsiostat and mountstats can expose per-mount and per-operation context such as RPC round-trip and execution times. Interpret execution time and queueing separately from network RTT; those metrics are not automatically equivalent to disk latency.

Record kernel, client utilities, protocol version, mount options, server environment, and workload interval. Do not assume a counter total belongs to one process. A controlled comparison should keep directory size, cache state, concurrency, and operation type stable.

Use this playbook to design a controlled failure experiment. Reproduce faults only in an isolated lab with disposable data and a recovery plan.

Recovery should preserve the application’s contract

A signal may remain pending while a task waits in a kernel path; “kill succeeded” is not the same as “the operation has unwound.” A lazy unmount detaches a namespace entry but does not prove all outstanding work completed. Forceful recovery can leave uncertain writes or application state.

Restore the failed dependency where possible, then verify application progress and data consistency. If changing semantics is necessary, make it an explicit application decision—not a generic command copied from a troubleshooting article.

Further reading

Primary references behind this explanation. Consult documentation for the version and environment you actually run.

1. Linux nfs(5): mount options and recovery
2. Linux proc_pid_stat(5): process state
3. Linux nfsstat(8)
4. Linux mountstats(8)