From: ricardo.canuelo@collabora.com (Ricardo Cañuelo)
To: kernelci@lists.linux.dev, gregkh@linuxfoundation.org,
thorsten@leemhuis.info, regressions@lists.linux.dev
Cc: kernel@collabora.com, linux-kernel@vger.kernel.org,
Gustavo Padovan <gustavo.padovan@collabora.com>,
Shreeya Patel <shreeya.patel@collabora.com>
Subject: Kernel regression tracking/reporting initiatives and KCIDB
Date: Tue, 01 Aug 2023 13:47:16 +0200 [thread overview]
Message-ID: <874jljhtmj.fsf@rcn-XPS-13-9305.i-did-not-set--mail-host-address--so-tickle-me> (raw)
Hi all,
I'm Ricardo from Collabora. In the past months, we’ve been analyzing the
current status of CI regression reporting and tracking in the Linux
kernel: assessing the existing tools, testing their functionalities,
collecting ideas about desirable features that aren’t available yet and
sketching some of them.
As part of this effort, we wrote a Regression Tracker tool [1] as a
proof of concept. It’s a rather simple tool that takes existing
regression data and reports and uses them to show more context on each
reported regression, as well as highlighting the relationships between
them, whether they can be caused by an infrastructure error and other
additional metadata about their current status. We’ve been using it
mostly as a playground for us to explore the current status of the
functionalities provided by CI systems and to test ideas about new
features.
We’re also checking other tools and services provided by the community,
such as regzbot [2], collaborating with them when possible and thinking
about how to combine multiple scattered efforts by different people
towards the same common goal. As a first step, we’ve contributed to
regzbot and partially integrated its results into the Regression Tracker
tool.
So far, we’ve been using the KernelCI regression data and reports as a
data source, we're now wondering if we could tackle the problem with a
more general approach by building on top of what KCIDB already provides.
In general, CI systems tend to define regressions as a low-level concept
which is rather static: a snapshot of a test result at a certain point
in time. When it comes to reporting them to developers, there's much
more info that could be added to them. In particular, the context of it
and the fact that a reported regression has a life cycle:
- did this test also fail on other hardware targets or with other kernel
configurations?
- is it possible that the test failed because of an infrastructure
error?
- does the test fail consistently since that commit or does it show
unstable results?
- does the test output show any traces of already known bugs?
- has this regression been bisected and reported anywhere?
- was the regression reported by anyone? If so, is there someone already
working on it?
Many of these info points can be extracted from the CI results databases
and processed to provide additional regression data. That’s what we’re
trying to do with the Regression Tracker tool, and we think it’d be
interesting to start experimenting with the data in KCIDB to see how
this could be improved and what would be the right way to integrate this
type of functionality.
Please let us know if that's a possibility and if you'd like to add
anything to the ideas proposed above.
Cheers,
Ricardo
[1] https://kernel.pages.collabora.com/kernelci-regressions-tracker/
[2] https://linux-regtracking.leemhuis.info/regzbot/all/
next reply other threads:[~2023-08-01 11:47 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-08-01 11:47 Ricardo Cañuelo [this message]
2023-08-02 8:07 ` Kernel regression tracking/reporting initiatives and KCIDB Thorsten Leemhuis
2023-08-07 8:29 ` Nikolai Kondrashov
2023-08-04 16:06 ` Nikolai Kondrashov
2023-08-08 9:55 ` Ricardo Cañuelo
2023-08-17 13:32 ` Guillaume Tucker
2023-08-18 7:50 ` Ricardo Cañuelo
2023-08-18 20:11 ` Guillaume Tucker
2023-08-21 10:30 ` Ricardo Cañuelo
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=874jljhtmj.fsf@rcn-XPS-13-9305.i-did-not-set--mail-host-address--so-tickle-me \
--to=ricardo.canuelo@collabora.com \
--cc=gregkh@linuxfoundation.org \
--cc=gustavo.padovan@collabora.com \
--cc=kernel@collabora.com \
--cc=kernelci@lists.linux.dev \
--cc=linux-kernel@vger.kernel.org \
--cc=regressions@lists.linux.dev \
--cc=shreeya.patel@collabora.com \
--cc=thorsten@leemhuis.info \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).