From: Rik Theys <Rik.Theys@esat.kuleuven.be>
To: Linux Nfs <linux-nfs@vger.kernel.org>
Subject: processes in D state / blocked tasks
Date: Fri, 24 Apr 2026 09:12:03 +0200 [thread overview]
Message-ID: <f32822c8-9910-43a1-8d7f-72df6b79a1e8@esat.kuleuven.be> (raw)
Hi,
Since a few weeks some of our numbercrunchers are periodically logging
blocked tasks and the NFS client seems to (at least partially) hang. The
client is currently running upstream 6.18.20 on a Rocky 9 userland. The
server is a RHEL10 running 6.12.0-124.45.1.el10_1.x86_64.
Most likely the reason we are seeing this come up now is that more users
are starting to use the shared server and it frequently comes under high
memory pressure.
We are seeing similar issues on Rocky 8 servers running
4.18.0-553.117.1.el8_10.x86_64.
On one of the Rocky 9 servers running 6.18.20, the following blocked
tasks are logged:
INFO: task node:1646922 blocked for more than 122 seconds.
Tainted: G E 6.18.20-1.el9.esat.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:node state:D stack:0 pid:1646922 tgid:1646922
ppid:1646730 task_flags:0x400000 flags:0x00080001
Call Trace:
<TASK>
__schedule+0x282/0x600
? select_idle_sibling+0x27/0x4c0
? terminate_walk+0xef/0x100
schedule+0x26/0x90
io_schedule+0x42/0x70
folio_wait_bit_common+0x127/0x360
? _raw_spin_unlock_irqrestore+0x23/0x40
? __pfx_wake_page_function+0x10/0x10
folio_wait_writeback+0x27/0x80
__filemap_get_folio+0x250/0x330
nfs_write_begin+0xa1/0x3e0 [nfs]
generic_perform_write+0x89/0x290
? nfs_ctx_key_to_expire+0xd3/0x120 [nfs]
nfs_file_write+0x1e7/0x2f0 [nfs]
vfs_write+0x31f/0x430
ksys_write+0x61/0xe0
do_syscall_64+0x61/0x12d0
entry_SYSCALL_64_after_hwframe+0x76/0x7e
RIP: 0033:0x7fbe608ff15f
RSP: 002b:00007ffda1c8f410 EFLAGS: 00000293 ORIG_RAX: 0000000000000001
RAX: ffffffffffffffda RBX: 0000000034979e10 RCX: 00007fbe608ff15f
RDX: 00000000000002e3 RSI: 000000002c7b5b50 RDI: 000000000000002c
RBP: 00000000000002e3 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000010 R11: 0000000000000293 R12: 00000000000002e3
R13: 000000002c7b5b50 R14: 00000000000002e3 R15: 00007fbe609f69e0
</TASK>
The taint seems to come from the fuse module.
INFO: task libuv-worker:474227 blocked for more than 122 seconds.
Tainted: G E 6.18.20-1.el9.esat.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:libuv-worker state:D stack:0 pid:474227 tgid:474198
ppid:474192 task_flags:0x400040 flags:0x00080001
Call Trace:
<TASK>
__schedule+0x282/0x600
schedule+0x26/0x90
__wait_on_freeing_inode+0xb9/0x110
? __pfx_var_wake_function+0x10/0x10
? __pfx_nfs_find_actor+0x10/0x10 [nfs]
find_inode+0xa3/0xf0
? __pfx_nfs_find_actor+0x10/0x10 [nfs]
ilookup5_nowait+0x74/0xa0
? __pfx_nfs_find_actor+0x10/0x10 [nfs]
ilookup5+0x44/0x110
? _kstrtoull+0x3a/0x90
? nfs_map_string_to_numeric+0x7f/0xa0 [nfsv4]
? __pfx_nfs_find_actor+0x10/0x10 [nfs]
? __pfx_nfs_init_locked+0x10/0x10 [nfs]
iget5_locked+0x26/0x80
nfs_fhget+0xe6/0x7e0 [nfs]
_nfs4_opendata_to_nfs4_state+0x16b/0x220 [nfsv4]
_nfs4_open_and_get_state+0xc3/0x2c0 [nfsv4]
? __pfx_nfs_alloc_no_seqid+0x10/0x10 [nfsv4]
? nfs4_opendata_alloc+0x26d/0x400 [nfsv4]
_nfs4_do_open.isra.0+0x167/0x470 [nfsv4]
? obj_cgroup_charge_account+0x9f/0x2a0
nfs4_do_open+0xcc/0x210 [nfsv4]
nfs4_atomic_open+0x11b/0x130 [nfsv4]
? alloc_nfs_open_context+0x2a/0x150 [nfs]
nfs_atomic_open+0x227/0x6a0 [nfs]
lookup_open.isra.0+0x241/0x6d0
? __d_lookup+0x72/0xb0
open_last_lookups+0x1f6/0x470
path_openat+0x83/0x270
do_filp_open+0xbd/0x170
? __virt_addr_valid+0xf9/0x170
? check_heap_object+0x33/0x190
? preempt_count_add+0x69/0xa0
? path_get+0x11/0x30
? _raw_spin_unlock+0x15/0x30
? audit_alloc_name+0x12f/0x140
? alloc_fd+0xae/0x110
do_sys_openat2+0x71/0xd0
__x64_sys_openat+0x52/0xa0
do_syscall_64+0x61/0x12d0
entry_SYSCALL_64_after_hwframe+0x76/0x7e
RIP: 0033:0x7f84d0efee54
RSP: 002b:00007f84a97bdb60 EFLAGS: 00000293 ORIG_RAX: 0000000000000101
RAX: ffffffffffffffda RBX: 00007f84a97be470 RCX: 00007f84d0efee54
RDX: 0000000000080000 RSI: 000000001a1d3210 RDI: 00000000ffffff9c
RBP: 000000001a1d3210 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000293 R12: 0000000000080000
R13: 0000000000000019 R14: 000000001a95a4c0 R15: 000000001a95a4c0
</TASK>
The system starts to build up a lot of processes in uninterruptable
sleep (D). When I look at the output of
ps -e -o state,pid,user,cmd,wchan | grep ^D
it seems most of these blocked processes are waiting in
rpc_wait_bit_killable, some in folio_wait_bit_common and a few in
autofs_wait.
# cat /proc/3806883/stack
[<0>] __rpc_execute+0x148/0x2e0 [sunrpc]
[<0>] rpc_execute+0x12b/0x150 [sunrpc]
[<0>] rpc_run_task+0x110/0x180 [sunrpc]
[<0>] nfs4_call_sync_sequence+0x75/0xb0 [nfsv4]
[<0>] _nfs4_proc_access+0x108/0x1a0 [nfsv4]
[<0>] nfs4_proc_access+0x66/0xf0 [nfsv4]
[<0>] nfs_do_access+0xb4/0x270 [nfs]
[<0>] nfs_permission+0x91/0x1f0 [nfs]
[<0>] inode_permission+0xe6/0x190
[<0>] link_path_walk+0x1f1/0x2a0
[<0>] path_lookupat+0x6f/0x1a0
[<0>] filename_lookup+0xdf/0x1e0
[<0>] vfs_statx+0x65/0x150
[<0>] vfs_fstatat+0x64/0xa0
[<0>] __do_sys_newfstatat+0x25/0x60
[<0>] do_syscall_64+0x61/0x12d0
[<0>] entry_SYSCALL_64_after_hwframe+0x76/0x7e
# cat /proc/2949346/stack
[<0>] filemap_fault+0x4b8/0xc50
[<0>] __do_fault+0x34/0x180
[<0>] do_read_fault+0x129/0x220
[<0>] do_fault+0x123/0x270
[<0>] __handle_mm_fault+0x56f/0x6b0
[<0>] handle_mm_fault+0xef/0x310
[<0>] do_user_addr_fault+0x216/0x6d0
[<0>] exc_page_fault+0x66/0x170
[<0>] asm_exc_page_fault+0x22/0x30
# cat /proc/577956/stack
[<0>] autofs_mount_wait+0x46/0xf0
[<0>] autofs_d_automount+0xd3/0x200
[<0>] __traverse_mounts+0x9c/0x260
[<0>] step_into+0x170/0x370
[<0>] link_path_walk+0x139/0x2a0
[<0>] path_openat+0x93/0x270
[<0>] do_filp_open+0xbd/0x170
[<0>] do_sys_openat2+0x71/0xd0
[<0>] __x64_sys_openat+0x52/0xa0
[<0>] do_syscall_64+0x61/0x12d0
[<0>] entry_SYSCALL_64_after_hwframe+0x76/0x7e
What could cause these processes to end up in this state? Is there some
deadlock due to memory pressure? Which information could be more useful
to further debug this and/or make the system recover?
The server seems to be fine: there's no high load, no indications of a
network issue and the disks are not saturated.
On the Rocky 8 system the following blocked tasks are logged:
INFO: task git:4512 blocked for more than 120 seconds.
Not tainted 4.18.0-553.117.1.el8_10.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:git state:D stack:0 pid:4512 ppid:4226
flags:0x80000082
Call Trace:
__schedule+0x2d1/0x870
schedule+0x55/0xf0
io_schedule+0x12/0x40
wait_on_page_bit+0xfd/0x220
? find_get_pages_range_tag+0xd2/0x350
? filemap_fdatawait_keep_errors+0x50/0x50
wait_on_page_writeback+0x2b/0x90
__filemap_fdatawait_range+0x81/0xf0
? memcg_slab_free_hook+0x141/0x1b0
filemap_write_and_wait+0x55/0x90
nfs_wb_all+0x1a/0x120 [nfs]
nfs4_file_flush+0x6f/0xa0 [nfsv4]
filp_close+0x31/0x70
put_files_struct+0x70/0xc0
do_exit+0x32f/0xb10
? syscall_trace_enter+0x1ff/0x2d0
do_group_exit+0x3a/0xa0
__x64_sys_exit_group+0x14/0x20
do_syscall_64+0x5b/0x1d0
entry_SYSCALL_64_after_hwframe+0x66/0xcb
RIP: 0033:0x7fa49031a346
Code: Unable to access opcode bytes at RIP 0x7fa49031a31c.
RSP: 002b:00007ffe099b4f78 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
RAX: ffffffffffffffda RBX: 00007fa4905de860 RCX: 00007fa49031a346
RDX: 0000000000000001 RSI: 000000000000003c RDI: 0000000000000001
RBP: 0000000000000001 R08: 00000000000000e7 R09: ffffffffffffff90
R10: 00007ffe099b4e7f R11: 0000000000000246 R12: 00007fa4905de860
R13: 0000000000000001 R14: 00007fa4905e7548 R15: 0000000000000000
INFO: task git:4513 blocked for more than 120 seconds.
Not tainted 4.18.0-553.117.1.el8_10.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:git state:D stack:0 pid:4513 ppid:4226
flags:0x80000082
Call Trace:
__schedule+0x2d1/0x870
schedule+0x55/0xf0
io_schedule+0x12/0x40
wait_on_page_bit+0xfd/0x220
? find_get_pages_range_tag+0xd2/0x350
? filemap_fdatawait_keep_errors+0x50/0x50
wait_on_page_writeback+0x2b/0x90
__filemap_fdatawait_range+0x81/0xf0
? memcg_slab_free_hook+0x141/0x1b0
filemap_write_and_wait+0x55/0x90
nfs_wb_all+0x1a/0x120 [nfs]
nfs4_file_flush+0x6f/0xa0 [nfsv4]
filp_close+0x31/0x70
put_files_struct+0x70/0xc0
do_exit+0x32f/0xb10
? syscall_trace_enter+0x1ff/0x2d0
do_group_exit+0x3a/0xa0
__x64_sys_exit_group+0x14/0x20
do_syscall_64+0x5b/0x1d0
entry_SYSCALL_64_after_hwframe+0x66/0xcb
RIP: 0033:0x7fcc80c5d346
Code: Unable to access opcode bytes at RIP 0x7fcc80c5d31c.
RSP: 002b:00007ffe25bb62d8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
RAX: ffffffffffffffda RBX: 00007fcc80f21860 RCX: 00007fcc80c5d346
RDX: 0000000000000000 RSI: 000000000000003c RDI: 0000000000000000
RBP: 0000000000000000 R08: 00000000000000e7 R09: ffffffffffffff90
R10: 00007ffe25bb61df R11: 0000000000000246 R12: 00007fcc80f21860
R13: 0000000000000001 R14: 00007fcc80f2a548 R15: 0000000000000000
Regards,
Rik
--
Rik Theys
System Engineer
KU Leuven - Dept. Elektrotechniek (ESAT)
Kasteelpark Arenberg 10 bus 2440 - B-3001 Leuven-Heverlee
+32(0)16/32.11.07
----------------------------------------------------------------
<<Any errors in spelling, tact or fact are transmission errors>>
reply other threads:[~2026-04-24 7:12 UTC|newest]
Thread overview: [no followups] expand[flat|nested] mbox.gz Atom feed
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=f32822c8-9910-43a1-8d7f-72df6b79a1e8@esat.kuleuven.be \
--to=rik.theys@esat.kuleuven.be \
--cc=linux-nfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).