From: "Paul E. McKenney" <paulmck@kernel.org>
To: Joel Fernandes <joel@joelfernandes.org>
Cc: Frederic Weisbecker <frederic@kernel.org>, rcu@vger.kernel.org
Subject: Re: [BUG] Random intermittent boost failures (Was Re: [BUG] TREE04..)
Date: Fri, 15 Sep 2023 07:45:20 -0700 [thread overview]
Message-ID: <7ecec718-dec8-411f-9dbd-dbe5fa946450@paulmck-laptop> (raw)
In-Reply-To: <20230914215324.GA1972295@google.com>
On Thu, Sep 14, 2023 at 09:53:24PM +0000, Joel Fernandes wrote:
> On Thu, Sep 14, 2023 at 06:56:27PM +0000, Joel Fernandes wrote:
> > On Thu, Sep 14, 2023 at 08:23:38AM -0700, Paul E. McKenney wrote:
> > > On Thu, Sep 14, 2023 at 01:13:51PM +0000, Joel Fernandes wrote:
> > > > On Thu, Sep 14, 2023 at 04:11:26AM -0700, Paul E. McKenney wrote:
> > > > > On Wed, Sep 13, 2023 at 04:30:20PM -0400, Joel Fernandes wrote:
> > > > > > On Mon, Sep 11, 2023 at 4:16 AM Paul E. McKenney <paulmck@kernel.org> wrote:
> > > > > > [..]
> > > > > > > > I am digging deeper to see why the rcu_preempt thread cannot be pushed out
> > > > > > > > and then I'll also look at why is it being pushed out in the first place.
> > > > > > > >
> > > > > > > > At least I have a strong repro now running 5 instances of TREE03 in parallel
> > > > > > > > for several hours.
> > > > > > >
> > > > > > > Very good! Then why not boot with rcutorture.onoff_interval=0 and see if
> > > > > > > the problem still occurs? If yes, then there is definitely some reason
> > > > > > > other than CPU hotplug that makes this happen.
> > > > > >
> > > > > > Hi Paul,
> > > > > > So looks so far like onoff_interval=0 makes the issue disappear. So
> > > > > > likely hotplug related. I am ok with doing the cpus_read_lock during
> > > > > > boost testing and seeing if that fixes it. If it does, I can move on
> > > > > > to the next thing in my backlog.
> > > > > >
> > > > > > What do you think? Or should I spend more time root-causing it? It is
> > > > > > most like runaway RT threads combined with the CPU hotplug threads,
> > > > > > making scheduling of the rcu_preempt thread not happen. But I can't
> > > > > > say for sure without more/better tracing (Speaking of better tracing,
> > > > > > I am adding core-dump support to rcutorture, but it is not there yet).
> > > > >
> > > > > This would not be the first time rcutorture has had trouble with those
> > > > > threads, so I am for adding the cpus_read_lock().
> > > > >
> > > > > Additional root-causing might be helpful, but then again, you might
> > > > > have higher priority things to worry about. ;-)
> > > >
> > > > No worries. Unfortunately putting cpus_read_lock() around the boost test
> > > > causes hangs. I tried something like the following [1]. If you have a diff, I can
> > > > quickly try something to see if the issue goes away as well.
> > >
> > > The other approaches that occur to me are:
> > >
> > > 1. Synchronize with the torture.c CPU-hotplug code. This is a bit
> > > tricky as well.
> > >
> > > 2. Rearrange the testing to convert one of the TREE0* scenarios that
> > > is not in CFLIST (TREE06 or TREE08) to a real-time configuration,
> > > with boosting but without CPU hotplug. Then remove boosting
> > > from TREE04.
> > >
> > > Of these, #2 seems most productive. But is there a better way?
> >
> > We could have the gp thread at higher priority for TREE03. What I see
> > consistently is that the GP thread gets migrated from CPU M to CPU N only to
> > be immediately sent back. Dumping the state showed CPU N is running ksoftirqd
> > which is also a rt priority 2. Making rcu_preempt 3 and ksoftirqd 2 might
> > give less of a run-around to rcu_preempt maybe enough to prevent the grace
> > period from stalling. I am not sure if this will fix it, but I am running a
> > test to see how it goes, will let you know.
>
> That led to a lot of fireworks. :-) I am thinking though, do we really need
> to run a boost kthread on all CPUs? I think that might be the root cause
> because the boost threads run on all CPUs except perhaps the one dying.
>
> We could run them on just the odd, or even ones and still be able to get
> sufficient boost testing. This may be especially important without RT
> throttling. I'll go ahead and queue a test like that.
>
> Thoughts?
The problem with this is that it will often render RCU priority boosting
unnecessary. Any kthread preempted within an RCU read-side critical
section will with high probability quickly be resumed on one of the
even-numbered CPUs.
Or were you also planning to bind the rcu_torture_reader() kthreads to
a specific CPU, preventing such migration? Or am I missing something
here?
Thanx, Paul
prev parent reply other threads:[~2023-09-15 14:45 UTC|newest]
Thread overview: 23+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-09-10 20:14 [BUG] Random intermittent boost failures (Was Re: [BUG] TREE04..) Joel Fernandes
2023-09-10 21:16 ` Paul E. McKenney
2023-09-10 23:37 ` Joel Fernandes
2023-09-11 2:27 ` Joel Fernandes
2023-09-11 8:16 ` Paul E. McKenney
2023-09-11 13:17 ` Joel Fernandes
2023-09-11 13:49 ` Paul E. McKenney
2023-09-11 16:18 ` Joel Fernandes
2023-09-13 20:30 ` Joel Fernandes
2023-09-14 11:11 ` Paul E. McKenney
2023-09-14 13:13 ` Joel Fernandes
2023-09-14 15:23 ` Paul E. McKenney
2023-09-14 18:56 ` Joel Fernandes
2023-09-14 21:53 ` Joel Fernandes
2023-09-15 0:13 ` Joel Fernandes
2023-09-15 11:33 ` Joel Fernandes
2023-09-15 14:53 ` Paul E. McKenney
2023-09-15 16:37 ` Joel Fernandes
2023-09-15 16:57 ` Paul E. McKenney
2023-09-15 21:14 ` Joel Fernandes
2023-09-18 6:05 ` Paul E. McKenney
2023-09-15 14:48 ` Paul E. McKenney
2023-09-15 14:45 ` Paul E. McKenney [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=7ecec718-dec8-411f-9dbd-dbe5fa946450@paulmck-laptop \
--to=paulmck@kernel.org \
--cc=frederic@kernel.org \
--cc=joel@joelfernandes.org \
--cc=rcu@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).