From: Jan Kara <jack@suse.cz>
To: Thomas Gleixner <tglx@linutronix.de>
Cc: Mike Galbraith <umgwanakikbuti@gmail.com>,
Tejun Heo <tj@kernel.org>, Michal Hocko <mhocko@kernel.org>,
Jiri Slaby <jslaby@suse.cz>, Petr Mladek <pmladek@suse.com>,
Jan Kara <jack@suse.cz>, Ben Hutchings <ben@decadent.org.uk>,
Sasha Levin <sasha.levin@oracle.com>, Shaohua Li <shli@fb.com>,
LKML <linux-kernel@vger.kernel.org>,
stable@vger.kernel.org, Daniel Bilik <daniel.bilik@neosystem.cz>
Subject: Re: Crashes with 874bbfe600a6 in 3.18.25
Date: Thu, 4 Feb 2016 12:20:44 +0100 [thread overview]
Message-ID: <20160204112044.GE4956@quack.suse.cz> (raw)
In-Reply-To: <alpine.DEB.2.11.1602041130380.25254@nanos>
On Thu 04-02-16 11:46:47, Thomas Gleixner wrote:
> On Thu, 4 Feb 2016, Mike Galbraith wrote:
> > On Wed, 2016-02-03 at 12:06 -0500, Tejun Heo wrote:
> > > On Wed, Feb 03, 2016 at 06:01:53PM +0100, Mike Galbraith wrote:
> > > > Hm, so it's ok to queue work to an offline CPU? What happens if it
> > > > doesn't come back for an eternity or two?
> > >
> > > Right now, it just loses affinity....
> >
> > WRT affinity...
> >
> > Somebody somewhere queues a delayed work, a timer is started on CPUX,
> > work is targeted at CPUX. Now wash/rinse/repeat mod_delayed_work()
> > along with migrations. Should __queue_delayed_work() not refrain from
> > altering dwork->cpu once set?
> >
> > I'm also wondering why 22b886dd only applies to kernels >= 4.2.
> >
> > <quote>
> > Regardless of the previous CPU a timer was on, add_timer_on()
> > currently simply sets timer->flags to the new CPU. As the caller must
> > be seeing the timer as idle, this is locally fine, but the timer
> > leaving the old base while unlocked can lead to race conditions as
> > follows.
> >
> > Let's say timer was on cpu 0.
> >
> > cpu 0 cpu 1
> > -----------------------------------------------------------------------------
> > del_timer(timer) succeeds
> > del_timer(timer)
> > lock_timer_base(timer) locks cpu_0_base
> > add_timer_on(timer, 1)
> > spin_lock(&cpu_1_base->lock)
> > timer->flags set to cpu_1_base
> > operates on @timer operates on @timer
> > </quote>
> >
> > What's the difference between...
> > timer->flags = (timer->flags & ~TIMER_BASEMASK) | cpu;
> > and...
> > timer_set_base(timer, base);
> >
> > ...that makes that fix unneeded prior to 4.2? We take the same locks
> > in < 4.2 kernels, so seemingly both will diddle concurrently above.
>
> Indeed, you are right.
>
> The same can happen on pre 4.2, just the fix does not apply as we changed the
> internals how the base is managed in the timer itself. Backport below.
Thanks for backport Thomas and to Mike for persistence :). I've asked my
friend seeing crashes with 3.18.25 to try whether this patch fixes the
issues. It may take some time so stay tuned...
Honza
> 8<----------------------------
>
> --- a/kernel/time/timer.c
> +++ b/kernel/time/timer.c
> @@ -956,13 +956,26 @@ EXPORT_SYMBOL(add_timer);
> */
> void add_timer_on(struct timer_list *timer, int cpu)
> {
> - struct tvec_base *base = per_cpu(tvec_bases, cpu);
> + struct tvec_base *new_base = per_cpu(tvec_bases, cpu);
> + struct tvec_base *base;
> unsigned long flags;
>
> timer_stats_timer_set_start_info(timer);
> BUG_ON(timer_pending(timer) || !timer->function);
> - spin_lock_irqsave(&base->lock, flags);
> - timer_set_base(timer, base);
> +
> + /*
> + * If @timer was on a different CPU, it must be migrated with the
> + * old base locked to prevent other operations proceeding with the
> + * wrong base locked. See lock_timer_base().
> + */
> + base = lock_timer_base(timer, &flags);
> + if (base != new_base) {
> + timer_set_base(timer, NULL);
> + spin_unlock(&base->lock);
> + base = new_base;
> + spin_lock(&base->lock);
> + timer_set_base(timer, base);
> + }
> debug_activate(timer, timer->expires);
> internal_add_timer(base, timer);
> spin_unlock_irqrestore(&base->lock, flags);
>
>
>
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
next prev parent reply other threads:[~2016-02-04 11:20 UTC|newest]
Thread overview: 57+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-01-20 21:19 Crashes with 874bbfe600a6 in 3.18.25 Jan Kara
2016-01-20 21:39 ` Shaohua Li
2016-01-21 9:52 ` Jan Kara
2016-01-21 13:29 ` Sasha Levin
2016-01-22 1:10 ` Sasha Levin
2016-01-22 16:09 ` Tejun Heo
2016-01-23 2:20 ` Ben Hutchings
2016-01-23 16:11 ` Thomas Gleixner
2016-01-26 9:34 ` Jan Kara
2016-01-26 9:49 ` Thomas Gleixner
2016-01-26 11:14 ` Petr Mladek
2016-01-26 13:09 ` Thomas Gleixner
2016-02-03 9:35 ` Jiri Slaby
2016-02-03 10:41 ` Thomas Gleixner
2016-02-03 12:28 ` Michal Hocko
2016-02-03 16:24 ` Tejun Heo
2016-02-03 16:48 ` Michal Hocko
2016-02-03 16:59 ` Tejun Heo
2016-02-04 6:37 ` Michal Hocko
2016-02-04 7:40 ` Michal Hocko
2016-02-03 17:01 ` Mike Galbraith
2016-02-03 17:06 ` Tejun Heo
2016-02-03 17:13 ` Mike Galbraith
2016-02-03 17:15 ` Tejun Heo
2016-02-04 2:00 ` Mike Galbraith
2016-02-05 16:49 ` Tejun Heo
2016-02-05 20:47 ` Mike Galbraith
2016-02-05 20:54 ` Tejun Heo
2016-02-05 20:59 ` Mike Galbraith
2016-02-05 21:06 ` Tejun Heo
2016-02-06 13:07 ` Henrique de Moraes Holschuh
2016-02-07 5:19 ` Mike Galbraith
2016-02-07 5:59 ` Mike Galbraith
2016-02-09 15:31 ` Mike Galbraith
2016-02-09 16:39 ` Linus Torvalds
2016-02-09 16:50 ` Tejun Heo
2016-02-09 17:04 ` Mike Galbraith
2016-02-09 17:54 ` Tejun Heo
2016-02-09 17:56 ` Mike Galbraith
2016-02-09 18:02 ` Mike Galbraith
2016-02-09 18:27 ` Tejun Heo
2016-02-09 17:04 ` Linus Torvalds
2016-02-09 17:51 ` Tejun Heo
2016-02-09 18:06 ` Linus Torvalds
2016-02-04 10:04 ` Mike Galbraith
2016-02-04 10:46 ` Thomas Gleixner
2016-02-04 11:07 ` Mike Galbraith
2016-02-04 11:20 ` Jan Kara [this message]
2016-02-04 16:39 ` Daniel Bilik
2016-02-05 2:40 ` Mike Galbraith
2016-02-05 8:11 ` Daniel Bilik
2016-02-05 8:33 ` Mike Galbraith
2016-02-03 18:46 ` Thomas Gleixner
2016-02-03 19:01 ` Tejun Heo
2016-02-03 19:05 ` Thomas Gleixner
2016-02-03 19:15 ` Tejun Heo
2016-02-05 5:44 ` Mike Galbraith
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20160204112044.GE4956@quack.suse.cz \
--to=jack@suse.cz \
--cc=ben@decadent.org.uk \
--cc=daniel.bilik@neosystem.cz \
--cc=jslaby@suse.cz \
--cc=linux-kernel@vger.kernel.org \
--cc=mhocko@kernel.org \
--cc=pmladek@suse.com \
--cc=sasha.levin@oracle.com \
--cc=shli@fb.com \
--cc=stable@vger.kernel.org \
--cc=tglx@linutronix.de \
--cc=tj@kernel.org \
--cc=umgwanakikbuti@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).