From: Mathieu Desnoyers via lttng-dev <lttng-dev@lists.lttng.org>
To: Minlan Wang <wangminlan@szsandstone.com>
Cc: lttng-dev <lttng-dev@lists.lttng.org>
Subject: Re: [lttng-dev] urcu workqueue thread uses 99% of cpu while workqueue is empty
Date: Tue, 14 Jun 2022 09:39:27 -0400 (EDT) [thread overview]
Message-ID: <541638875.58723.1655213967957.JavaMail.zimbra@efficios.com> (raw)
In-Reply-To: <20220614035533.GA174967@localhost.localdomain>+C23B4245BB527E95>
----- On Jun 13, 2022, at 11:55 PM, Minlan Wang wangminlan@szsandstone.com wrote:
> Hi, Mathieu,
Hi Minlan,
Thanks for the detailed bug report. Can I ask more precisely which commit ID
of the userspace-rcu stable-2.12 branch you are using ? Typically a "userspace-rcu-latest-0.12.tar.bz2"
gets generated from a git tree at a given point in time, but it does not give
me enough details to know which commit it refers to.
Thanks,
Mathieu
> We are running a CentOS 8.2 os on Intel(R) Xeon(R) CPU E5-2630 v4,
> and using the workqueue interfaces in src/workqueue.h in
> userspace-rcu-latest-0.12.tar.bz2.
> Recently, we found the workqueue thread rushes cpu into 99% usage.
> After some debuging, we found that the futex in struct urcu_workqueue got
> into very big negative value, e.g, -12484; while the qlen, cbs_tail, and
> cbs_head suggest that the workqueue is empty.
> We add a watchpoint of workqueue->futex in workqueue_thread(), and got this
> log when workqueue->futex first get into -2:
> ...
> Old value = -1
> New value = 0
> 0x00007ffff37c1d6d in futex_wake_up (futex=0x55555f74aa40) at workqueue.c:160
> 160 in workqueue.c
> #0 0x00007ffff37c1d6d in futex_wake_up (futex=0x55555f74aa40) at
> workqueue.c:160
> #1 0x00007ffff37c2737 in wake_worker_thread (workqueue=0x55555f74aa00) at
> workqueue.c:324
> #2 0x00007ffff37c29fb in urcu_workqueue_queue_work (workqueue=0x55555f74aa00,
> work=0x555566e05e00, func=0x7ffff7523c90 <write_dirty_finish>) at
> workqueue.c:3
> 67
> #3 0x00007ffff752c520 in aio_complete_cb (ctx=<optimized out>,
> iocb=<optimized out>, res=<optimized out>, res2=<optimized out>) at
> bio/aio_bio_adapter.c:152
> #4 0x00007ffff752c696 in poll_io_complete (arg=0x555562e4f4a0) at
> bio/aio_bio_adapter.c:289
> #5 0x00007ffff72e6ea5 in start_thread () from /usr/lib64/libpthread.so.0
> #6 0x00007ffff415d96d in clone () from /usr/lib64/libc.so.6
> [Switching to Thread 0x7fffde3f3700 (LWP 821768)]
> Hardware watchpoint 4: -location workqueue->futex
>
> Old value = 0
> New value = -1
> 0x00007ffff37c2473 in __uatomic_dec (len=4, addr=0x55555f74aa40) at
> ../include/urcu/uatomic.h:490
> 490 ../include/urcu/uatomic.h: No such file or directory.
> #0 0x00007ffff37c2473 in __uatomic_dec (len=4, addr=0x55555f74aa40) at
> ../include/urcu/uatomic.h:490
> #1 workqueue_thread (arg=0x55555f74aa00) at workqueue.c:250
> #2 0x00007ffff72e6ea5 in start_thread () from /usr/lib64/libpthread.so.0
> #3 0x00007ffff415d96d in clone () from /usr/lib64/libc.so.6
> Hardware watchpoint 4: -location workqueue->futex
>
> Old value = -1
> New value = -2
> 0x00007ffff37c2473 in __uatomic_dec (len=4, addr=0x55555f74aa40) at
> ../include/urcu/uatomic.h:490
> 490 in ../include/urcu/uatomic.h
> #0 0x00007ffff37c2473 in __uatomic_dec (len=4, addr=0x55555f74aa40) at
> ../include/urcu/uatomic.h:490
> #1 workqueue_thread (arg=0x55555f74aa00) at workqueue.c:250
> #2 0x00007ffff72e6ea5 in start_thread () from /usr/lib64/libpthread.so.0
> #3 0x00007ffff415d96d in clone () from /usr/lib64/libc.so.6
> Hardware watchpoint 4: -location workqueue->futex
>
> Old value = -2
> New value = -3
> 0x00007ffff37c2473 in __uatomic_dec (len=4, addr=0x55555f74aa40) at
> ../include/urcu/uatomic.h:490
> 490 in ../include/urcu/uatomic.h
> #0 0x00007ffff37c2473 in __uatomic_dec (len=4, addr=0x55555f74aa40) at
> ../include/urcu/uatomic.h:490
> #1 workqueue_thread (arg=0x55555f74aa00) at workqueue.c:250
> #2 0x00007ffff72e6ea5 in start_thread () from /usr/lib64/libpthread.so.0
> #3 0x00007ffff415d96d in clone () from /usr/lib64/libc.so.6
> Hardware watchpoint 4: -location workqueue->futex
> ...
>
> After this, things went into wild, workqueue->futex got into bigger negative
> value, and workqueue thread eat up the cpu it is using.
> This ends only when workqueue->futex down flew into 0.
>
> Do you have any idea why this is happening, and how to fix it?
>
> B.R
> Minlan Wang
--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
next prev parent reply other threads:[~2022-06-14 13:39 UTC|newest]
Thread overview: 22+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-06-14 3:55 [lttng-dev] urcu workqueue thread uses 99% of cpu while workqueue is empty Minlan Wang via lttng-dev
2022-06-14 13:39 ` Mathieu Desnoyers via lttng-dev [this message]
2022-06-14 13:40 ` Mathieu Desnoyers via lttng-dev
2022-06-14 14:19 ` Mathieu Desnoyers via lttng-dev
2022-06-14 15:53 ` Mathieu Desnoyers via lttng-dev
2022-06-15 3:49 ` Minlan Wang via lttng-dev
2022-06-15 13:35 ` Mathieu Desnoyers via lttng-dev
2022-06-15 14:15 ` Mathieu Desnoyers via lttng-dev
2022-06-16 7:09 ` Minlan Wang via lttng-dev
2022-06-16 8:09 ` Minlan Wang via lttng-dev
2022-06-17 13:37 ` Mathieu Desnoyers via lttng-dev
2022-06-21 3:52 ` Minlan Wang via lttng-dev
2022-06-21 13:12 ` [lttng-dev] ***UNCHECKED*** " Mathieu Desnoyers via lttng-dev
2022-06-22 7:45 ` Minlan Wang via lttng-dev
2022-06-22 13:19 ` [lttng-dev] ***UNCHECKED*** " Mathieu Desnoyers via lttng-dev
2022-06-22 20:28 ` Mathieu Desnoyers via lttng-dev
2022-06-22 20:52 ` Mathieu Desnoyers via lttng-dev
2022-06-23 9:08 ` Minlan Wang via lttng-dev
[not found] ` <20220623034528.GA271179@localhost.localdomain>
2022-06-23 3:57 ` Minlan Wang via lttng-dev
2022-06-23 14:09 ` Mathieu Desnoyers via lttng-dev
2022-06-24 6:21 ` Minlan Wang via lttng-dev
2022-06-23 8:36 ` [lttng-dev] [PART 4/4] " Minlan Wang via lttng-dev
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=541638875.58723.1655213967957.JavaMail.zimbra@efficios.com \
--to=lttng-dev@lists.lttng.org \
--cc=mathieu.desnoyers@efficios.com \
--cc=wangminlan@szsandstone.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).