From: Nikolai Kondrashov <spbnick@gmail.com>
To: Gustavo Padovan <gus@collabora.com>
Cc: kernelci-results-staging <kernelci-results-staging@groups.io>,
Jeny Sheth <jeny.sadadia@collabora.com>, bot <bot@kernelci.org>,
"shreeya.patel" <shreeya.patel@collabora.com>,
"kernelci@lists.linux.dev" <kernelci@lists.linux.dev>
Subject: Re: KernelCI report for stable-rc: v5.15.164
Date: Tue, 6 Aug 2024 19:32:53 +0300 [thread overview]
Message-ID: <d0b5b62d-9f19-468f-810b-6533fb1c23e4@gmail.com> (raw)
In-Reply-To: <191286e9b09.f4f1c6a4526882.2185642814129237428@collabora.com>
On 8/6/24 7:03 PM, Gustavo Padovan wrote:
>
>
> ---- On Tue, 06 Aug 2024 07:06:00 -0300 Nikolai Kondrashov wrote ---
>
> > On 8/6/24 1:03 PM, Nikolai Kondrashov via groups.io wrote:
> > > On 8/5/24 5:40 PM, Gustavo Padovan via groups.io wrote:
> > >> > > Why does this have kernelci, syzbot and tuxsuite. Let's remove it for now.
> > >> >
> > >> > It lists all the CI systems the report has been aggregated with. Although
> > > only maestro and broonie reported failures, we included
> > >> > builds and test stats from all the 5 CI systems.
> > >>
> > >> I see. Although, we can't really guarantee the quality of data coming from
> > >> the other origins, hence my request to use only maestro and broonie for the
> > >> time being. We need a high quality report, not a report with as much results
> > >> as possible.
> > >
> > > This is alright for a PoC. However, I'd like you guys to mention to Greg that
> > > we have more data from other origins, when you engage him with this, and that
> > > they're available on the dashboards. Or perhaps better just put that into the
> > > notification message itself.
> > >
> > > Finally, as we've discussed before, if a maintainer has requirements for
> > > particular result quality, it's best to directly apply them to data, where
> > > possible (e.g. check that log_excerpt is there), instead of simply selecting
> > > origins. That could be the next step, once we define what "good quality"
> > > means exactly.
> >
> > This is what all the CI systems signed up for, after all: reaching out to
> > maintainers. And we would be breaking our promise, if we make these decisions
> > for them, or discriminate by origin.
>
> Not exactly. That would be the perfectionist take. We are investing a lot in the test
> and results data quality so we can be sure we can be sure we are showing good,
> high quality data to maintainers we are engaging actively.
>
> That's exactly the case of the stable-rc report. Greg has said to us already that just
> throwing data at him wouldn't help much. We have to make sure that data can actually
> help him. This has been our goal since we started the current stable-rc report process
> since LPC. The Collabora team has invested dozens and dozens of ours into evaluating
> and improving the quality of the data.
>
> In summary, we are not breaking any promises. We can really throw any data we have into
> maintainers and expect that would be helping them.
Sure, but our (KCIDB) job as the "middle man" is to advertise results of all
submitters *equally*, let the maintainers *themselves* decide what's good and
bad for them, implement their requirements, and *also* pass their feedback to
the submitters. *This* is what they expect of us.
We should not be deciding *ourselves* alone what's good and bad for
maintainers (without even specifying actual requirements we're applying), and
not telling the submitters we're ignoring their data, while sending our *own*
to maintainers.
> The Collabora team has invested dozens and dozens of ours into evaluating
> and improving the quality of the data
You can be sure that other CI systems worked hard on this too. Microsoft
results were top-notch when they decided to join. Syzbot results are almost
*perfect*, they're doing a *very* good job. CKI has an *army* of QE looking
after tests. And so on.
I understand the desire and the need to hit the mark first try, and sure, this
is fine for the start, as long as we're clear about what we're actually doing,
and what *other* data is available and being omitted from notifications.
Finally, once again, we should be judging data based on its merits, not on who
sent it. Let's specify what kind of quality we're looking for exactly, and
implement filtering according to these requirements as one of the next steps.
Nick
prev parent reply other threads:[~2024-08-06 16:32 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <66ae1e21.050a0220.1c5959.0148@mx.google.com>
[not found] ` <1911838cd2e.ddb71f5356531.3914011082698955929@collabora.com>
[not found] ` <19122d7a43d.dde9a53983019.6953377593724593632@collabora.com>
[not found] ` <19122fc3d66.11f5a856693778.1637642610124539898@collabora.com>
[not found] ` <17E91B8D1664855A.28314@groups.io>
2024-08-06 10:06 ` KernelCI report for stable-rc: v5.15.164 Nikolai Kondrashov
2024-08-06 16:03 ` Gustavo Padovan
2024-08-06 16:32 ` Nikolai Kondrashov [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=d0b5b62d-9f19-468f-810b-6533fb1c23e4@gmail.com \
--to=spbnick@gmail.com \
--cc=bot@kernelci.org \
--cc=gus@collabora.com \
--cc=jeny.sadadia@collabora.com \
--cc=kernelci-results-staging@groups.io \
--cc=kernelci@lists.linux.dev \
--cc=shreeya.patel@collabora.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).