From: "wangzhiqiang (Q)" <wangzhiqiang95@huawei.com>
To: <linux-lvm@redhat.com>
Cc: lixiaokeng@huawei.com, louhongxiang@huawei.com
Subject: [linux-lvm] vgchange acquire an exclusive VG lock for refresh
Date: Tue, 29 Aug 2023 11:53:19 +0800 [thread overview]
Message-ID: <0b5edc8d-6ba6-0c8a-3d5c-84f9f7bc4d13@huawei.com> (raw)
I've got an issue while execute vgchange --refresh concurrently, process vgchange and systemd-udevd
enters the D state.
root 2933 0.0 1.0 54216 31344 pts/1 D<L 11:21 0:00 vgchange --refresh
root 2934 0.0 1.1 54216 32788 pts/1 D<L 11:21 0:00 vgchange --refresh
root 2935 0.0 1.0 54216 31496 pts/1 D<L 11:21 0:00 vgchange --refresh
root 2937 0.0 1.0 54216 31500 pts/1 S<L 11:21 0:00 vgchange --refresh
root 2951 0.0 0.0 0 0 ? D 11:21 0:00 [systemd-udevd]
root 2952 0.0 0.0 0 0 ? D 11:21 0:00 [systemd-udevd]
process stack as follows
[root@localhost ~]# cat /proc/2937/stack
[<0>] dm_wait_for_bios_completion+0x298/0x340 [dm_mod]
[<0>] dm_wait_for_completion+0xfe/0x110 [dm_mod]
[<0>] __dm_suspend+0xd0/0x310 [dm_mod]
[<0>] dm_suspend+0x133/0x180 [dm_mod]
[<0>] dev_suspend+0x115/0x150 [dm_mod]
[<0>] ctl_ioctl+0x320/0x4c0 [dm_mod]
[<0>] dm_ctl_ioctl+0x26/0x40 [dm_mod]
[<0>] __se_sys_ioctl+0xeb/0x120
[<0>] do_syscall_64+0x30/0x40
[<0>] entry_SYSCALL_64_after_hwframe+0x61/0xc6
[root@localhost ~]# cat /proc/2935/stack
[<0>] dm_suspend+0x2b/0x180 [dm_mod]
[<0>] dev_suspend+0x115/0x150 [dm_mod]
[<0>] ctl_ioctl+0x320/0x4c0 [dm_mod]
[<0>] dm_ctl_ioctl+0x26/0x40 [dm_mod]
[<0>] __se_sys_ioctl+0xeb/0x120
[<0>] do_syscall_64+0x30/0x40
[<0>] entry_SYSCALL_64_after_hwframe+0x61/0xc6
[root@localhost ~]# cat /proc/2951/stack
[<0>] __lock_page+0x25a/0x7a0
[<0>] truncate_inode_pages_range+0xc52/0x1370
[<0>] __blkdev_put+0x15b/0x3f0
[<0>] blkdev_close+0x49/0x60
[<0>] __fput+0x1b9/0x480
[<0>] task_work_run+0xad/0xf0
[<0>] do_exit+0x47f/0x810
[<0>] do_group_exit+0x77/0x150
[<0>] get_signal+0x2a2/0xa60
[<0>] arch_do_signal_or_restart+0x1c6/0x310
[<0>] exit_to_user_mode_loop+0xdb/0x100
[<0>] exit_to_user_mode_prepare+0x7e/0x90
[<0>] syscall_exit_to_user_mode+0x12/0x40
[<0>] entry_SYSCALL_64_after_hwframe+0x61/0xc6
I recommend get an exclusive lock when vgchange --refresh.
From 7ddadf2bbbe0383893ab8f5b7cb8d51ebf7a5ed4 Mon Sep 17 00:00:00 2001
From: wangzhiqiang <wangzhiqiang95@huawei.com>
Date: Tue, 29 Aug 2023 10:28:15 +0800
Subject: [PATCH] vgchange acquire an exclusive VG lock for refresh.
Let's run the following test cases, we've got hungtask.
P2 or P3 can't suspend snap and udevd also can't read
device info because real or cow has been suspend.
P1(vgchange) P2(vgchange) P3(vgchange)
--------------------------------------------------------
suspend real
suspend cow
suspend snap
suspend snap
suspend snap
function query()
{
while true
do
vgchange --refresh
done
}
query 1 &> /dev/null &
query 2 &> /dev/null &
query 3 &> /dev/null &
[ 247.250669] INFO: task vgchange:2934 blocked for more than 122 seconds.
[ 247.251935] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 247.252619] task:vgchange state:D stack: 0 pid: 2934 ppid: 2930 flags:0x00000080
[ 247.252626] Call Trace:
[ 247.252633] __schedule+0x547/0xe80
[ 247.252640] ? io_schedule_timeout+0xb0/0xb0
[ 247.252647] ? _raw_spin_lock+0x7a/0xd0
[ 247.252652] schedule+0x6f/0x130
[ 247.252658] schedule_preempt_disabled+0xa/0x10
[ 247.252664] __mutex_lock.constprop.0+0x311/0x700
[ 247.252673] ? kernel_text_address.part.0+0xb9/0xc0
[ 247.252678] ? ww_mutex_unlock+0x50/0x50
[ 247.252685] ? unwind_get_return_address+0x33/0x50
[ 247.252692] ? cred_has_capability.isra.0+0x14b/0x220
[ 247.252698] ? check_nnp_nosuid.isra.0+0x170/0x170
[ 247.252704] mutex_lock+0x114/0x130
[ 247.252710] ? __mutex_lock_slowpath+0x10/0x10
[ 247.252716] ? rwsem_down_write_slowpath+0x830/0x830
[ 247.252733] ? dm_get+0x40/0x70 [dm_mod]
[ 247.252752] ? dm_get_md+0xb3/0x120 [dm_mod]
[ 247.252770] dm_suspend+0x2b/0x180 [dm_mod]
[ 247.252788] dev_suspend+0x115/0x150 [dm_mod]
[ 247.252805] ctl_ioctl+0x320/0x4c0 [dm_mod]
[ 247.252823] ? dev_set_geometry+0x290/0x290 [dm_mod]
[ 247.252841] ? copy_params+0x2e0/0x2e0 [dm_mod]
[ 247.252846] ? locks_dispose_list+0xa0/0xd0
[ 247.252851] ? locks_remove_posix+0x24f/0x340
[ 247.252856] ? locks_remove_flock+0x1e6/0x250
[ 247.252865] ? ioctl_fiemap.isra.0+0x110/0x1e0
[ 247.252872] ? vfs_write+0x2a5/0x440
[ 247.252877] ? selinux_file_ioctl+0x89/0x1c0
[ 247.252895] ? ctl_ioctl+0x4c0/0x4c0 [dm_mod]
[ 247.252912] dm_ctl_ioctl+0x26/0x40 [dm_mod]
[ 247.252918] __se_sys_ioctl+0xeb/0x120
[ 247.252923] do_syscall_64+0x30/0x40
[ 247.252929] entry_SYSCALL_64_after_hwframe+0x61/0xc6
[ 247.252967] INFO: task vgchange:2935 blocked for more than 122 seconds.
[ 247.254242] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 247.254909] task:vgchange state:D stack: 0 pid: 2935 ppid: 2931 flags:0x00000080
[ 247.254922] Call Trace:
[ 247.254929] __schedule+0x547/0xe80
[ 247.254935] ? io_schedule_timeout+0xb0/0xb0
[ 247.254942] ? _raw_spin_lock+0x7a/0xd0
[ 247.254947] schedule+0x6f/0x130
[ 247.254953] schedule_preempt_disabled+0xa/0x10
[ 247.254959] __mutex_lock.constprop.0+0x311/0x700
[ 247.254965] ? kernel_text_address.part.0+0xb9/0xc0
[ 247.254976] ? ww_mutex_unlock+0x50/0x50
[ 247.254981] ? unwind_get_return_address+0x33/0x50
[ 247.254988] ? cred_has_capability.isra.0+0x14b/0x220
[ 247.254995] ? check_nnp_nosuid.isra.0+0x170/0x170
[ 247.255000] mutex_lock+0x114/0x130
[ 247.255006] ? __mutex_lock_slowpath+0x10/0x10
[ 247.255012] ? rwsem_down_write_slowpath+0x830/0x830
[ 247.255029] ? dm_get+0x40/0x70 [dm_mod]
[ 247.255052] ? dm_get_md+0xb3/0x120 [dm_mod]
[ 247.255069] dm_suspend+0x2b/0x180 [dm_mod]
[ 247.255088] dev_suspend+0x115/0x150 [dm_mod]
[ 247.255113] ctl_ioctl+0x320/0x4c0 [dm_mod]
[ 247.255142] ? dev_set_geometry+0x290/0x290 [dm_mod]
[ 247.255160] ? copy_params+0x2e0/0x2e0 [dm_mod]
[ 247.255175] ? nohz_balance_exit_idle+0x2b/0x160
[ 247.255187] ? rb_insert_color+0x2a/0x280
[ 247.255196] ? ioctl_fiemap.isra.0+0x110/0x1e0
[ 247.255202] ? selinux_file_ioctl+0x89/0x1c0
[ 247.255219] ? ctl_ioctl+0x4c0/0x4c0 [dm_mod]
[ 247.255238] dm_ctl_ioctl+0x26/0x40 [dm_mod]
[ 247.255250] __se_sys_ioctl+0xeb/0x120
[ 247.255255] do_syscall_64+0x30/0x40
[ 247.255262] entry_SYSCALL_64_after_hwframe+0x61/0xc6
[ 493.015195] INFO: task systemd-udevd:2951 blocked for more than 122 seconds.
[ 493.016461] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 493.017131] task:systemd-udevd state:D stack: 0 pid: 2951 ppid: 719 flags:0x000043a6
[ 493.017138] Call Trace:
[ 493.017145] __schedule+0x547/0xe80
[ 493.017152] ? io_schedule_timeout+0xb0/0xb0
[ 493.017157] ? orc_find.part.0+0x240/0x240
[ 493.017163] ? get_stack_info_noinstr+0x16/0x140
[ 493.017171] ? bpf_ksym_find+0xc0/0x110
[ 493.017189] schedule+0x6f/0x130
[ 493.017195] io_schedule+0x6d/0xa0
[ 493.017202] __lock_page+0x25a/0x7a0
[ 493.017209] ? wait_on_page_bit_killable+0x6c0/0x6c0
[ 493.017216] ? __lock_page_or_retry+0x3e0/0x3e0
[ 493.017222] ? add_to_page_cache_lru+0x370/0x370
[ 493.017231] truncate_inode_pages_range+0xc52/0x1370
[ 493.017242] ? generic_error_remove_page+0x90/0x90
[ 493.017248] ? __filemap_fdatawait_range+0x179/0x270
[ 493.017256] ? filemap_range_has_page+0x170/0x170
[ 493.017264] ? __bitmap_or_equal+0x89/0x120
[ 493.017271] ? _find_next_bit.constprop.0+0xde/0x120
[ 493.017279] ? cpumask_next+0x2c/0x40
[ 493.017287] ? smp_call_function_many_cond+0x430/0x5f0
[ 493.017295] ? touch_buffer+0xa0/0xa0
[ 493.017305] ? __find_get_block+0x110/0x110
[ 493.017311] ? __find_get_block+0x110/0x110
[ 493.017316] ? touch_buffer+0xa0/0xa0
[ 493.017322] __blkdev_put+0x15b/0x3f0
[ 493.017328] ? blkdev_put+0x1c0/0x1c0
[ 493.017333] blkdev_close+0x49/0x60
[ 493.017339] __fput+0x1b9/0x480
[ 493.017346] task_work_run+0xad/0xf0
[ 493.017354] do_exit+0x47f/0x810
[ 493.017361] ? exit_mm+0x450/0x450
[ 493.017371] ? _raw_spin_lock_irq+0x7b/0xd0
[ 493.017377] do_group_exit+0x77/0x150
[ 493.017384] get_signal+0x2a2/0xa60
[ 493.017393] arch_do_signal_or_restart+0x1c6/0x310
[ 493.017399] ? get_sigframe_size+0x20/0x20
[ 493.017405] ? vfs_read+0x1f6/0x2d0
[ 493.017411] ? unroll_tree_refs+0x18a/0x1e0
[ 493.017424] exit_to_user_mode_loop+0xdb/0x100
[ 493.017430] exit_to_user_mode_prepare+0x7e/0x90
[ 493.017438] syscall_exit_to_user_mode+0x12/0x40
[ 493.017445] entry_SYSCALL_64_after_hwframe+0x61/0xc6
cat /proc/2937/stack
[<0>] dm_wait_for_bios_completion+0x298/0x340 [dm_mod]
[<0>] dm_wait_for_completion+0xfe/0x110 [dm_mod]
[<0>] __dm_suspend+0xd0/0x310 [dm_mod]
[<0>] dm_suspend+0x133/0x180 [dm_mod]
[<0>] dev_suspend+0x115/0x150 [dm_mod]
[<0>] ctl_ioctl+0x320/0x4c0 [dm_mod]
[<0>] dm_ctl_ioctl+0x26/0x40 [dm_mod]
[<0>] __se_sys_ioctl+0xeb/0x120
[<0>] do_syscall_64+0x30/0x40
[<0>] entry_SYSCALL_64_after_hwframe+0x61/0xc6
Signed-off-by: wangzhiqiang <wangzhiqiang95@huawei.com>
---
tools/vgchange.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tools/vgchange.c b/tools/vgchange.c
index e4b57db..eb3096d 100644
--- a/tools/vgchange.c
+++ b/tools/vgchange.c
@@ -837,7 +837,7 @@ int vgchange(struct cmd_context *cmd, int argc, char **argv)
cmd->lockd_vg_enforce_sh = 1;
}
- if (update)
+ if (update || arg_is_set(cmd, refresh_ARG))
flags |= READ_FOR_UPDATE;
else if (arg_is_set(cmd, activate_ARG))
flags |= READ_FOR_ACTIVATE;
--
2.33.0
_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
next reply other threads:[~2023-08-29 4:12 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-08-29 3:53 wangzhiqiang (Q) [this message]
2023-08-29 6:36 ` [linux-lvm] vgchange acquire an exclusive VG lock for refresh wangzhiqiang (Q)
2023-08-29 21:41 ` David Teigland
2023-08-30 2:14 ` wangzhiqiang (Q)
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=0b5edc8d-6ba6-0c8a-3d5c-84f9f7bc4d13@huawei.com \
--to=wangzhiqiang95@huawei.com \
--cc=linux-lvm@redhat.com \
--cc=lixiaokeng@huawei.com \
--cc=louhongxiang@huawei.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).