From: Laura Abbott <labbott@redhat.com>
To: James Bottomley <James.Bottomley@HansenPartnership.com>,
"ksummit-discuss@lists.linuxfoundation.org"
<ksummit-discuss@lists.linuxfoundation.org>
Subject: Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful
Date: Wed, 5 Sep 2018 10:41:30 -0700 [thread overview]
Message-ID: <f5c3a8c7-b1bc-2566-59be-619d637c1293@redhat.com> (raw)
In-Reply-To: <1536142432.8121.6.camel@HansenPartnership.com>
On 09/05/2018 03:13 AM, James Bottomley wrote:
> I'm seeing a lot of wasted effort by our customers on kernel bugs and
> trying to engage the distribution to fix them. As a caveat, I'm
> working in the cloud, so the distributions in question are usually
> community ones not enterprise ones. However, we do have a fair few
> customers on LTS kernels from Distributions.
>
> Mostly they find a cloud performance regression, they try to engage the
> distro, spend ages working on it or submitting bugs and usually end up
> with an unsatisfactory result. By the time they call my team in, we've
> likely only got a week to fix the issue. However, step one is always
> confirming whether upstream works (95% of the time it does) and then
> finding the fix by bisection (usually assisted by knowledge of where
> the bug is). To do the bisection we usually have to build a kernel
> package with our guesses and get them to try it, so it can be a bit
> slow. Once we have the backport, we send it to stable and notify the
> distribution to include it in their next kernel release.
>
> Here's the rub: community distributions (even LTS ones) don't have the
> resources even to triage cloud bugs in environments they likely can't
> reproduce, so we really need to develop assistive tools for customers
> to perform bisections to identify what caused the bug or (in the 95%
> case) what fixed it. Having a bugzilla and using it as first line of
> support implies a service expectation (usually coming from Enterprise)
> that simply isn't met, so distributions need to fix this at the point
> of interaction: bugzilla.
>
> The first suggestion is that kernel builds are pretty much automated
> and we try to make every commit buildable, so could we automate the
> machinery that allows a customer to do bisection simply by installing a
> kernel package? (we here, obviously means the distro, but going from
> git bisect to kernel package would be the useful link).
>
As mentioned further down, I did try having scripts to do this but
it was ultimately fairly fragile. I think a better approach is
leveraging the existing in tree support for building a distro
package and using that with regular git bisect.
> Second suggestion is that the bugzillas need to say much more strongly
> that the reporter really needs to confirm the fix in upstream and do
> the bisection themselves (and ideally request the backport to stable
> themselves).
>
At least in Fedora this is something we hit fairly frequently. We
do strongly encourage people to report bugs and bisect. This runs
into a number of problems:
- Bisecting on local machines is slow and people often don't want
to give up their machine resources.
- If people give me a test case I can reproduce, I'm usually
okay to run a bisect myself but it's pretty rare to get a
test case.
- People are hesitant to run bisections and build kernels.
There's a lot of steps involved. We try and point people to wiki
pages with instructions but many times we end up having to
go back and forth explaining how to do the setup.
- People are hesitant to report bugs to the upstream. I end up
having to explain where to report bugs or run get_maintainer.pl
for people otherwise they just file it against kernel.org
bugzilla or just e-mail LKML. I started a skeleton of a web
project to make a web interface for get_maintainer.pl but it
never got very far.
Tooling can certainly help with a lot of this but some of
this may also just be more documentation and needing to
guide people.
Thanks,
Laura
prev parent reply other threads:[~2018-09-05 17:41 UTC|newest]
Thread overview: 33+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-09-05 10:13 [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful James Bottomley
2018-09-05 11:37 ` Mark Brown
2018-09-05 15:03 ` Paul E. McKenney
2018-09-05 15:50 ` Steven Rostedt
2018-09-05 16:20 ` Paul E. McKenney
2018-09-05 16:45 ` James Bottomley
2018-09-05 17:00 ` Paul E. McKenney
2018-09-05 19:25 ` Jiri Kosina
2018-09-05 19:40 ` James Bottomley
2018-09-06 19:54 ` Jiri Kosina
2018-09-18 13:43 ` Martin K. Petersen
2018-09-18 14:12 ` Geert Uytterhoeven
2018-09-18 15:01 ` Martin K. Petersen
2018-09-18 15:27 ` Christoph Hellwig
2018-09-18 15:34 ` Jens Axboe
2018-09-18 17:08 ` Mark Brown
2018-09-18 16:12 ` Mark Brown
2018-09-18 20:20 ` Takashi Iwai
2018-09-19 0:08 ` Mark Brown
2018-09-18 20:37 ` Takashi Iwai
2018-09-19 6:16 ` Geert Uytterhoeven
2018-09-19 6:31 ` Takashi Iwai
2018-09-19 9:23 ` Jan Kara
2018-09-19 9:27 ` Takashi Iwai
2018-09-05 13:16 ` Takashi Iwai
2018-09-05 13:20 ` Jiri Kosina
2018-09-05 13:39 ` Konstantin Ryabitsev
2018-09-05 15:16 ` Sasha Levin
2018-09-05 16:44 ` Laura Abbott
2018-09-05 20:15 ` Konstantin Ryabitsev
2018-09-05 20:36 ` Takashi Iwai
2018-09-07 20:24 ` Mauro Carvalho Chehab
2018-09-05 17:41 ` Laura Abbott [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=f5c3a8c7-b1bc-2566-59be-619d637c1293@redhat.com \
--to=labbott@redhat.com \
--cc=James.Bottomley@HansenPartnership.com \
--cc=ksummit-discuss@lists.linuxfoundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).