From: Thorsten Leemhuis <linux@leemhuis.info>
To: Greg KH <gregkh@linuxfoundation.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>,
"Rafael J. Wysocki" <rafael@kernel.org>,
regressions@lists.linux.dev,
Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
Konstantin Ryabitsev <konstantin@linuxfoundation.org>,
Pablo Neira Ayuso <pablo@netfilter.org>,
ksummit@lists.linux.dev, workflows@vger.kernel.org
Subject: Re: RFC: building a regression tracking bot for Linux kernel development
Date: Fri, 23 Apr 2021 12:49:25 +0200 [thread overview]
Message-ID: <8a44e6b3-fff4-ea6f-2643-bb2a15ff990a@leemhuis.info> (raw)
In-Reply-To: <YIKdQKJNZOSfdL9B@kroah.com>
On 23.04.21 12:11, Greg KH wrote:
> On Thu, Apr 22, 2021 at 09:16:40AM +0200, Thorsten Leemhuis wrote:
>> Lo! As mentioned a few times recently I'm staring to build a bot for
>> semi-automatic Linux kernel regressions tracking. Find below a rough
>> description of how I imagine it's going to work. That way I want to give
>> everyone a chance to influence things before I'm starting to code for
>> real. Early feedback will help to build something that's acceptable for
>> the Linux kernel developer community and used in practice in the long
>> run, and that's what I aim for.
>>
>> I know, I know, "Talk is cheap. Show me the code.". But I had to think
>> things through and write some of it down anyway, so no harm done in
>> posting it as RFC. I CCed ksummit, as many maintainers hang out there
>> and because this is a follow up to my former regression tracking work we
>> discussed on both kernel and maintainers summit 2017; it fact it
>> hopefully might be something for this year as well, we'll see, too early
>> to tell.
>>
>> So how will the "regzbot" work? The ideal case is simple:
>>
>> Someone reports a regression to the recently created regressions mailing
>> list(regressions@lists.linux.dev). There the user includes a tag like this:
>>> #regzb introduced: 94a632d91ad1 ("usc: xhbi-foo: check bar_params earlier")
>
> That's great, but the ability for most people to track stuff down to the
> commit id feels very low. I would think that the "this no longer works"
> is the bug report / regression, and eventually that can be tracked down
> to "commit XXXX caused this" which is what you want to see above.
>
> Or am I confused?
Sorry, I should have made it more obvious that I leave some stuff out
initially and get to aspect like that later. :-/
[…]
>> That can't be all
>> -----------------
>>
>> Of course the world is more complicated than the simple example scenario
>> above, as the devil is always in the details. The three most obvious
>> problems the initial ideal scenario left aside:
>>
>> * The reporter doesn't specify the #regzb tag at all. Regzbot can't do
>> anything about it, it sadly won't have visionary power and a AI engine
>> any time soon. Some human (for a while that often will be me) thus needs
>> to reply with the tag with a proper reply-to to the report to make
>> regboz track it.
>
> Any specific format/tag we can use to help make this easier? Or is that
> just something that you are going to do "by hand" to start with?
For now making sure regressions@lists.linux.dev is added to the CCed of
any replies to regression reports would be a really big help to already
establish it as central place where they are all visible. Then no human
or bot needs to monitor hundreds of mailing list to find them -- which
obviously is doomed to fail and was one of the pain points when I
tracked regressions manually years ago.
And in the long run of course everyone is free to use those tags
(obviously they won't help before I get that bot running), but for in
the beginning it will be mainly "me doing it by hand"...
>> * regression in stable and longterm kernels sometimes affect multiple
>> versions, for example if a patch that works fine in mainline was
>> backported to the longterm kernel 5.10 and 5.4 – but causes problems in
>> both, as something required by the patch is missing in those lines. How
>> this will be solved exactly remains to be seen, maybe like this:
>>> #regzb Introduced: c39667ddcfd6 e39667ddcfd1 ("usc: xhbi-foo: check bar_params a little later again")
>>
>> Then regzbot can look those commits up and from that determine the
>> affected versions. Obviously the reporter will likely not be aware of
>> it, hence it's likely that the stable maintainer or the developer need
>> to send a mail to make regzbot aware that this regression affects
>> multiple versions.
>
> This shouldn't be that big of an issue, as the stable maintainers have
> simple tools that can show "what releases was this commit in" today,
> which is what we use to track what commits need to be backported where.
>
> I can give you a copy of my tools off-line if you are curious about this
> and think it's worth adding to your tool.
Guess I might get back to that sooner or later, but if you already have
something to handle that I might ignore that case initially and focus on
other aspects to make sure that stuff gets properly off the ground.
> Oh, and many thanks for doing this, I think this looks great.
Thx for taking a look. Feels good to hear that, as I really don't want
to build something which in the end doesn't get used.
Ciao, Thorsten
P.S.: There is one thing I forgot to mention: regzbot will obviously in
the long run also be able to generate reports in text form and send them
to Linus or LKML occasionally (likely once a week before or after a new rc).
next prev parent reply other threads:[~2021-04-23 10:49 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-04-22 7:16 RFC: building a regression tracking bot for Linux kernel development Thorsten Leemhuis
2021-04-22 14:51 ` Mark Brown
2021-04-23 6:34 ` Thorsten Leemhuis
2021-04-23 10:11 ` Greg KH
2021-04-23 10:49 ` Thorsten Leemhuis [this message]
2021-04-23 11:01 ` Takashi Iwai
2021-04-23 11:11 ` Thorsten Leemhuis
2021-04-23 20:45 ` Guillaume Tucker
2021-04-24 9:43 ` Thorsten Leemhuis
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=8a44e6b3-fff4-ea6f-2643-bb2a15ff990a@leemhuis.info \
--to=linux@leemhuis.info \
--cc=gregkh@linuxfoundation.org \
--cc=konstantin@linuxfoundation.org \
--cc=ksummit@lists.linux.dev \
--cc=linux-kernel@vger.kernel.org \
--cc=pablo@netfilter.org \
--cc=rafael@kernel.org \
--cc=regressions@lists.linux.dev \
--cc=torvalds@linux-foundation.org \
--cc=workflows@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.