From: Tejun Heo <tj@kernel.org>
To: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mike Galbraith <umgwanakikbuti@gmail.com>,
Michal Hocko <mhocko@kernel.org>, Jiri Slaby <jslaby@suse.cz>,
Thomas Gleixner <tglx@linutronix.de>,
Petr Mladek <pmladek@suse.com>, Jan Kara <jack@suse.cz>,
Ben Hutchings <ben@decadent.org.uk>,
Sasha Levin <sasha.levin@oracle.com>, Shaohua Li <shli@fb.com>,
LKML <linux-kernel@vger.kernel.org>,
stable <stable@vger.kernel.org>,
Daniel Bilik <daniel.bilik@neosystem.cz>,
Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Subject: Re: Crashes with 874bbfe600a6 in 3.18.25
Date: Tue, 9 Feb 2016 12:51:01 -0500 [thread overview]
Message-ID: <20160209175101.GB3741@mtj.duckdns.org> (raw)
In-Reply-To: <CA+55aFy4-tseM84CTTgCo_8KrPPA7JP8yF0kC1zJrmA2u2sq_Q@mail.gmail.com>
Hello,
On Tue, Feb 09, 2016 at 09:04:18AM -0800, Linus Torvalds wrote:
> On Tue, Feb 9, 2016 at 8:50 AM, Tejun Heo <tj@kernel.org> wrote:
> > idk, not doing so is likely to cause subtle bugs which are difficult
> > to track down. The problem with -stable is 874bbfe6 being backported
> > without the matching timer fix.
>
> Well, according to this thread, even witht he timer fix the end result
> then shows odd problems, _and_ has a NO_HZ_FULL regression.
I don't know what that odd problem is indicating but it's likely we're
seeing another issue exposed by these changes or a bug during
backport, but yeah it's problematic.
> I do agree about subtle bugs, but we haven't actually seen any other
> ones than the vmstat breakage so far.
The thing with vmstat is that it's a work item which is most likely to
expose the issue as it runs constantly on all systems and we started
seeing it triggering soon after timer migration becomes more common.
I'd be surprised if we don't discover a lot more subtler ones down the
road. Maybe it's that most of them won't trigger often enough to
matter much but it's a bit scary.
> Also, I suspect that to flush out any bugs, we might want to
>
> (a) actually dequeue timers and work queues that are bound to a
> particular CPU when a CPU goes down.
>
> Sure, we *could* make it a rule that everybody who binds a timer
> to a particular CPU should just register the cpu-down thing, but why
> make a rule that you have to make extra work? People who do per-cpu
> work should have a setup function for when a new CPU comes _up_, but
> why make people do pointless extra crap for the cpu-down case when the
> generic code could just do ti for them.
This goes the same for work items and timers. If we want to do
explicit dequeueing or flushing of cpu-bound stuff on cpu down, we'll
have to either dedicate *_on() interfaces for correctness or introduce
a separate set of interfaces to use for optimization and correctness.
The current situation is that work itmes which are explicitly shut
down on cpu-down are correctness usages while the ones which are not
are optimization usages. I'll try to scan through the usages and see
what the actual proportions are like. Maybe we can get away with
declaring that _on() usages are absolute.
> (b) maybe one of the test-bots could be encouraged to do a lot of cpu
> offlining/onlining as a stress test>
>
> That (a) part is important in that it avoids the subtle bug where some
> timer or workqueue entry ends up being run on the wrong CPU after all,
> just because the target CPU went down.
>
> And the (b) part would hopefully flush out things that didn't start
> things properly when a new cpu comes online.
>
> Hmm? The above is obviously a longer-term thing and a bigger change,
> but I think we should be able to just revert 874bbfe6 without anything
> else going on, since I don't think we ever found anything else than
> vmstat that had issues.
So, how about reverting 874bbfe6 and performing random foreign
queueing during -rc's for a couple cycles so that we can at least find
out the broken ones quickly in devel branch and backport fixes as
they're found?
Thanks.
--
tejun
next prev parent reply other threads:[~2016-02-09 17:51 UTC|newest]
Thread overview: 57+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-01-20 21:19 Crashes with 874bbfe600a6 in 3.18.25 Jan Kara
2016-01-20 21:39 ` Shaohua Li
2016-01-21 9:52 ` Jan Kara
2016-01-21 13:29 ` Sasha Levin
2016-01-22 1:10 ` Sasha Levin
2016-01-22 16:09 ` Tejun Heo
2016-01-23 2:20 ` Ben Hutchings
2016-01-23 16:11 ` Thomas Gleixner
2016-01-26 9:34 ` Jan Kara
2016-01-26 9:49 ` Thomas Gleixner
2016-01-26 11:14 ` Petr Mladek
2016-01-26 13:09 ` Thomas Gleixner
2016-02-03 9:35 ` Jiri Slaby
2016-02-03 10:41 ` Thomas Gleixner
2016-02-03 12:28 ` Michal Hocko
2016-02-03 16:24 ` Tejun Heo
2016-02-03 16:48 ` Michal Hocko
2016-02-03 16:59 ` Tejun Heo
2016-02-04 6:37 ` Michal Hocko
2016-02-04 7:40 ` Michal Hocko
2016-02-03 17:01 ` Mike Galbraith
2016-02-03 17:06 ` Tejun Heo
2016-02-03 17:13 ` Mike Galbraith
2016-02-03 17:15 ` Tejun Heo
2016-02-04 2:00 ` Mike Galbraith
2016-02-05 16:49 ` Tejun Heo
2016-02-05 20:47 ` Mike Galbraith
2016-02-05 20:54 ` Tejun Heo
2016-02-05 20:59 ` Mike Galbraith
2016-02-05 21:06 ` Tejun Heo
2016-02-06 13:07 ` Henrique de Moraes Holschuh
2016-02-07 5:19 ` Mike Galbraith
2016-02-07 5:59 ` Mike Galbraith
2016-02-09 15:31 ` Mike Galbraith
2016-02-09 16:39 ` Linus Torvalds
2016-02-09 16:50 ` Tejun Heo
2016-02-09 17:04 ` Mike Galbraith
2016-02-09 17:54 ` Tejun Heo
2016-02-09 17:56 ` Mike Galbraith
2016-02-09 18:02 ` Mike Galbraith
2016-02-09 18:27 ` Tejun Heo
2016-02-09 17:04 ` Linus Torvalds
2016-02-09 17:51 ` Tejun Heo [this message]
2016-02-09 18:06 ` Linus Torvalds
2016-02-04 10:04 ` Mike Galbraith
2016-02-04 10:46 ` Thomas Gleixner
2016-02-04 11:07 ` Mike Galbraith
2016-02-04 11:20 ` Jan Kara
2016-02-04 16:39 ` Daniel Bilik
2016-02-05 2:40 ` Mike Galbraith
2016-02-05 8:11 ` Daniel Bilik
2016-02-05 8:33 ` Mike Galbraith
2016-02-03 18:46 ` Thomas Gleixner
2016-02-03 19:01 ` Tejun Heo
2016-02-03 19:05 ` Thomas Gleixner
2016-02-03 19:15 ` Tejun Heo
2016-02-05 5:44 ` Mike Galbraith
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20160209175101.GB3741@mtj.duckdns.org \
--to=tj@kernel.org \
--cc=ben@decadent.org.uk \
--cc=daniel.bilik@neosystem.cz \
--cc=gregkh@linuxfoundation.org \
--cc=jack@suse.cz \
--cc=jslaby@suse.cz \
--cc=linux-kernel@vger.kernel.org \
--cc=mhocko@kernel.org \
--cc=pmladek@suse.com \
--cc=sasha.levin@oracle.com \
--cc=shli@fb.com \
--cc=stable@vger.kernel.org \
--cc=tglx@linutronix.de \
--cc=torvalds@linux-foundation.org \
--cc=umgwanakikbuti@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).