* kdevops BoF at LSFMM
@ 2024-05-07 18:44 Luis Chamberlain
2024-05-07 18:56 ` Chuck Lever III
2024-05-08 7:45 ` Amir Goldstein
0 siblings, 2 replies; 9+ messages in thread
From: Luis Chamberlain @ 2024-05-07 18:44 UTC (permalink / raw
To: lsf-pc; +Cc: kdevops, Linux FS Devel, linux-mm, linux-cxl
Dear LPC session leads,
We'd like to gather together and talk about current ongoing
developments / changes on kdevops at LSFMM. Those interested in
automation on complex workflows with kdevops are also welcomed. This
is best addressed informally, but since I see an open slot for at
10:30am for Tuesday, figured I'd check to see if we can snatch it.
BoFs for filesystems are scheduled towards the end of the conference
on Wednesday it seems, so ideally this would just take place then, but
the last BoF for XFS at Linux Plumbers took... 4 hours, and if such
filesystem BoFs take place I suspect each FS developer would also want
to attend their own respective FS BoF... so perhaps best we get a
kdevops BoF out of the way before the respective filesystem BoFs.
Agenda items?
Guestfs migration progress - have we killed vagrant?
Automation on testing filesystem baselines
xarray / maple tree testing and userspace testing
OpenTofu
kdevops-results-archive split
Any others?
Luis
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: kdevops BoF at LSFMM
2024-05-07 18:44 kdevops BoF at LSFMM Luis Chamberlain
@ 2024-05-07 18:56 ` Chuck Lever III
2024-05-08 7:45 ` Amir Goldstein
1 sibling, 0 replies; 9+ messages in thread
From: Chuck Lever III @ 2024-05-07 18:56 UTC (permalink / raw
To: Luis Chamberlain
Cc: lsf-pc@lists.linux-foundation.org, kdevops@lists.linux.dev,
Linux FS Devel, linux-mm, linux-cxl@vger.kernel.org
> On May 7, 2024, at 2:44 PM, Luis Chamberlain <mcgrof@kernel.org> wrote:
>
> Dear LPC session leads,
>
> We'd like to gather together and talk about current ongoing
> developments / changes on kdevops at LSFMM. Those interested in
> automation on complex workflows with kdevops are also welcomed. This
> is best addressed informally, but since I see an open slot for at
> 10:30am for Tuesday, figured I'd check to see if we can snatch it.
> BoFs for filesystems are scheduled towards the end of the conference
> on Wednesday it seems, so ideally this would just take place then, but
> the last BoF for XFS at Linux Plumbers took... 4 hours, and if such
> filesystem BoFs take place I suspect each FS developer would also want
> to attend their own respective FS BoF... so perhaps best we get a
> kdevops BoF out of the way before the respective filesystem BoFs.
>
> Agenda items?
>
> Guestfs migration progress - have we killed vagrant?
> Automation on testing filesystem baselines
> xarray / maple tree testing and userspace testing
> OpenTofu
> kdevops-results-archive split
results archive - would be nice to have utilities for
workflows (not fstests) to manage their results.
We've added robust NFS, SMB, and tmpfs support over
the past year. What about 9p? bcachefs? Others?
I'm about to add support for creating iSCSI LUNs on
demand. What about NVMeoF as file system backing store?
--
Chuck Lever
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: kdevops BoF at LSFMM
2024-05-07 18:44 kdevops BoF at LSFMM Luis Chamberlain
2024-05-07 18:56 ` Chuck Lever III
@ 2024-05-08 7:45 ` Amir Goldstein
2024-05-08 15:57 ` Luis Chamberlain
2024-05-08 17:45 ` Steve French
1 sibling, 2 replies; 9+ messages in thread
From: Amir Goldstein @ 2024-05-08 7:45 UTC (permalink / raw
To: Luis Chamberlain
Cc: lsf-pc, kdevops, Linux FS Devel, linux-mm, linux-cxl, Jan Kara
On Tue, May 7, 2024 at 9:44 PM Luis Chamberlain <mcgrof@kernel.org> wrote:
>
> Dear LPC session leads,
>
> We'd like to gather together and talk about current ongoing
> developments / changes on kdevops at LSFMM. Those interested in
> automation on complex workflows with kdevops are also welcomed. This
> is best addressed informally, but since I see an open slot for at
> 10:30am for Tuesday, figured I'd check to see if we can snatch it.
The empty slot is there for flexibility of the schedule and also
wouldn't storage/MM people be interested in kdevops?
I've placed you session instead of the FS lightning talks on Tuesday
after Leah's FS testing session.
There are enough slots for FS lightning talks.
There are several empty slots throughout the agenda left for
flexibility, including the one you mentioned on Tue morning.
kdevops session is for a very specialized group of developers,
so if that group is assembled and decides to use an earlier slot
we can do that on the spot.
> BoFs for filesystems are scheduled towards the end of the conference
> on Wednesday it seems, so ideally this would just take place then, but
> the last BoF for XFS at Linux Plumbers took... 4 hours, and if such
> filesystem BoFs take place I suspect each FS developer would also want
> to attend their own respective FS BoF...
I certainly hope that FS maintainers will use this gathering to have
their own respective BoFs.
If anyone would like to lead an FS BoF in the FS track room,
please let me know and I will place it on the agenda.
Thanks,
Amir.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: kdevops BoF at LSFMM
2024-05-08 7:45 ` Amir Goldstein
@ 2024-05-08 15:57 ` Luis Chamberlain
2024-05-13 1:58 ` Matthew Wilcox
2024-05-08 17:45 ` Steve French
1 sibling, 1 reply; 9+ messages in thread
From: Luis Chamberlain @ 2024-05-08 15:57 UTC (permalink / raw
To: Amir Goldstein
Cc: lsf-pc, kdevops, Linux FS Devel, linux-mm, linux-cxl, Jan Kara
On Wed, May 08, 2024 at 10:45:33AM +0300, Amir Goldstein wrote:
> On Tue, May 7, 2024 at 9:44 PM Luis Chamberlain <mcgrof@kernel.org> wrote:
> >
> > Dear LPC session leads,
> >
> > We'd like to gather together and talk about current ongoing
> > developments / changes on kdevops at LSFMM. Those interested in
> > automation on complex workflows with kdevops are also welcomed. This
> > is best addressed informally
>
> wouldn't storage/MM people be interested in kdevops?
It is up to them, I mean, I've started to work on mmtests integration
so we can help test memory fragmentation, and plan is to integrate
automation of maple tree and xarray shortly, mm folks are more than
welcomed!
> I've placed you session instead of the FS lightning talks on Tuesday
> after Leah's FS testing session.
> There are enough slots for FS lightning talks.
That works great, thanks!
> kdevops session is for a very specialized group of developers,
> so if that group is assembled and decides to use an earlier slot
> we can do that on the spot.
I think the current timing is perfect, and does not even conflict with
mm folks, if they want to join.
Luis
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: kdevops BoF at LSFMM
2024-05-08 7:45 ` Amir Goldstein
2024-05-08 15:57 ` Luis Chamberlain
@ 2024-05-08 17:45 ` Steve French
2024-05-08 17:54 ` Chuck Lever III
1 sibling, 1 reply; 9+ messages in thread
From: Steve French @ 2024-05-08 17:45 UTC (permalink / raw
To: Amir Goldstein
Cc: Luis Chamberlain, lsf-pc, kdevops, Linux FS Devel, linux-mm,
linux-cxl, Jan Kara
On Wed, May 8, 2024 at 2:48 AM Amir Goldstein <amir73il@gmail.com> wrote:
>
> On Tue, May 7, 2024 at 9:44 PM Luis Chamberlain <mcgrof@kernel.org> wrote:
> >
> > Dear LPC session leads,
> >
> > We'd like to gather together and talk about current ongoing
> > developments / changes on kdevops at LSFMM. Those interested in
> > automation on complex workflows with kdevops are also welcomed. This
> > is best addressed informally, but since I see an open slot for at
> > 10:30am for Tuesday, figured I'd check to see if we can snatch it.
>
> The empty slot is there for flexibility of the schedule and also
> wouldn't storage/MM people be interested in kdevops?
>
> I've placed you session instead of the FS lightning talks on Tuesday
> after Leah's FS testing session.
> There are enough slots for FS lightning talks.
>
> There are several empty slots throughout the agenda left for
> flexibility, including the one you mentioned on Tue morning.
> kdevops session is for a very specialized group of developers,
> so if that group is assembled and decides to use an earlier slot
> we can do that on the spot.
kdevops could be *extrememly* useful to understand better (and
to share "best practices" and ideas on testing from various filesystems)
I would be very happy if there were an easy way to do three things
faster/easier:
1) make it easier to run a reasonably large set of fs tests automatically
on checkin of a commit or set of commits (e.g. to an externally visible
github branch) before it goes in linux-next, and a larger set
of test automation that is automatically run on P/Rs (I kick these tests
off semi-manually for cifs.ko and ksmbd.ko today)
2) make it easier as a maintainer to get reports of automated testing of
stable-rc (or automate running of tests against stable-rc by filesystem type
and send failures to the specific fs's mailing list). Make the tests run
for a particular fs more visible, so maintainers/contributors can note
where important tests are left out against a particular fs
3) make it easier to auto-bisect what commit regressed when a failing test
is spotted
4) make it easier to automatically enable certain fs specific debug
tooling (e.g. eBPF scripts or trace points or log capturing) when a
test fails, or when a test fails - enable tracing and restart tests
5) make it easier to collect log output at the end of each test to
catch "suspicious" things (like network reconnects/timeouts, dmesg
events logged, fs specific stats or debug data that show excessive
failures or slow responses)
6) an easy way to tell if a kdevops run is "suspiciously slow" (ie if a test
or set of tests is more than 20% slower than the baseline test run, it
could indicate a performance regression that needs to be bisected
and identified)
--
Thanks,
Steve
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: kdevops BoF at LSFMM
2024-05-08 17:45 ` Steve French
@ 2024-05-08 17:54 ` Chuck Lever III
2024-05-12 20:20 ` Luis Chamberlain
2024-05-13 13:17 ` Theodore Ts'o
0 siblings, 2 replies; 9+ messages in thread
From: Chuck Lever III @ 2024-05-08 17:54 UTC (permalink / raw
To: Steve French
Cc: Amir Goldstein, Luis Chamberlain,
lsf-pc@lists.linux-foundation.org, kdevops@lists.linux.dev,
Linux FS Devel, linux-mm, linux-cxl@vger.kernel.org, Jan Kara
> On May 8, 2024, at 1:45 PM, Steve French <smfrench@gmail.com> wrote:
>
> I would be very happy if there were an easy way to do three things
> faster/easier:
> 1) make it easier to run a reasonably large set of fs tests automatically
> on checkin of a commit or set of commits (e.g. to an externally visible
> github branch) before it goes in linux-next, and a larger set
> of test automation that is automatically run on P/Rs (I kick these tests
> off semi-manually for cifs.ko and ksmbd.ko today)
> 2) make it easier as a maintainer to get reports of automated testing of
> stable-rc (or automate running of tests against stable-rc by filesystem type
> and send failures to the specific fs's mailing list). Make the tests run
> for a particular fs more visible, so maintainers/contributors can note
> where important tests are left out against a particular fs
In my experience, these require the addition of a CI
apparatus like BuildBot or Jenkins -- they are not
directly part of kdevops' mission. Scott Mayhew and
I have been playing with BuildBot, and there are some
areas where integration between kdevops and BuildBot
could be improved (and could be discussed next week).
> 3) make it easier to auto-bisect what commit regressed when a failing test
> is spotted
Jeff Layton has mentioned this as well. I don't think
it would be impossible to get kdevops to orchestrate
a bisect, as long as it has an automatic way to decide
when to use "git bisect {good|bad}"
> 6) an easy way to tell if a kdevops run is "suspiciously slow" (ie if a test
> or set of tests is more than 20% slower than the baseline test run, it
> could indicate a performance regression that needs to be bisected
> and identified)
Well sometimes things are just slow because you've built
a test kernel with KASAN and lockdep, or because there are
other jobs running on your test system. Also, due to all
the virtualization involved, it might be difficult to get
consistent performance measurements.
This one seems like it would be hard to engineer, but maybe
there's something that could be done?
--
Chuck Lever
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: kdevops BoF at LSFMM
2024-05-08 17:54 ` Chuck Lever III
@ 2024-05-12 20:20 ` Luis Chamberlain
2024-05-13 13:17 ` Theodore Ts'o
1 sibling, 0 replies; 9+ messages in thread
From: Luis Chamberlain @ 2024-05-12 20:20 UTC (permalink / raw
To: Chuck Lever III, Kent Overstreet
Cc: Steve French, Amir Goldstein, lsf-pc@lists.linux-foundation.org,
kdevops@lists.linux.dev, Linux FS Devel, linux-mm,
linux-cxl@vger.kernel.org, Jan Kara
On Wed, May 08, 2024 at 05:54:50PM +0000, Chuck Lever III wrote:
>
>
> > On May 8, 2024, at 1:45 PM, Steve French <smfrench@gmail.com> wrote:
> >
> > I would be very happy if there were an easy way to do three things
> > faster/easier:
> > 1) make it easier to run a reasonably large set of fs tests automatically
> > on checkin of a commit or set of commits (e.g. to an externally visible
> > github branch) before it goes in linux-next, and a larger set
> > of test automation that is automatically run on P/Rs (I kick these tests
> > off semi-manually for cifs.ko and ksmbd.ko today)
> > 2) make it easier as a maintainer to get reports of automated testing of
> > stable-rc (or automate running of tests against stable-rc by filesystem type
> > and send failures to the specific fs's mailing list). Make the tests run
> > for a particular fs more visible, so maintainers/contributors can note
> > where important tests are left out against a particular fs
>
> In my experience, these require the addition of a CI
> apparatus like BuildBot or Jenkins -- they are not
> directly part of kdevops' mission.
Song Liu and Paul E Luse will have a talk on Wednesday about using a
CI framework for md/raid. The holy grail I think here is that they
have used their experience with eBPF patchwork CI integration, and
I think everyone likely wants something similar:
https://patchwork.kernel.org/project/netdevbpf/list/
The S / W / F is Success / warning/ fail.
I'd like to see how we can do that for kdevops. The work is already
put in place to ramp up complex workflows, now we just have to launch
them and collect information.
> Scott Mayhew and
> I have been playing with BuildBot, and there are some
> areas where integration between kdevops and BuildBot
> could be improved (and could be discussed next week).
Neat!
> > 3) make it easier to auto-bisect what commit regressed when a failing test
> > is spotted
>
> Jeff Layton has mentioned this as well. I don't think
> it would be impossible to get kdevops to orchestrate
> a bisect, as long as it has an automatic way to decide
> when to use "git bisect {good|bad}"
Kent alreeady seems to have this working too, we should try to see what
we can leverage.
Luis
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: kdevops BoF at LSFMM
2024-05-08 15:57 ` Luis Chamberlain
@ 2024-05-13 1:58 ` Matthew Wilcox
0 siblings, 0 replies; 9+ messages in thread
From: Matthew Wilcox @ 2024-05-13 1:58 UTC (permalink / raw
To: Luis Chamberlain
Cc: Amir Goldstein, lsf-pc, kdevops, Linux FS Devel, linux-mm,
linux-cxl, Jan Kara
On Wed, May 08, 2024 at 08:57:16AM -0700, Luis Chamberlain wrote:
> It is up to them, I mean, I've started to work on mmtests integration
> so we can help test memory fragmentation, and plan is to integrate
> automation of maple tree and xarray shortly, mm folks are more than
> welcomed!
I don't see how kdevops integration of the radix tree, xarray and maple
tree test suite is useful. It's a white-box test suite. kdevops is
more suited for black-box testing like xfstests and mmtests.
When we find a problem in the various data structures through the black
box testing of their users (like the page cache or the VMA), it's on us
to add tests that exercise that functionality. Like 2a0774c2886d.
The important thing is that the data structures by design don't depend
on much of the kernel. So if they work in-and-of themselves, there's
not much to gain by doing tests in-kernel.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: kdevops BoF at LSFMM
2024-05-08 17:54 ` Chuck Lever III
2024-05-12 20:20 ` Luis Chamberlain
@ 2024-05-13 13:17 ` Theodore Ts'o
1 sibling, 0 replies; 9+ messages in thread
From: Theodore Ts'o @ 2024-05-13 13:17 UTC (permalink / raw
To: Chuck Lever III
Cc: Steve French, Amir Goldstein, Luis Chamberlain,
lsf-pc@lists.linux-foundation.org, kdevops@lists.linux.dev,
Linux FS Devel, linux-mm, linux-cxl@vger.kernel.org, Jan Kara
On Wed, May 08, 2024 at 05:54:50PM +0000, Chuck Lever III wrote:
>
> In my experience, these require the addition of a CI
> apparatus like BuildBot or Jenkins -- they are not
> directly part of kdevops' mission. Scott Mayhew and
> I have been playing with BuildBot, and there are some
> areas where integration between kdevops and BuildBot
> could be improved (and could be discussed next week).
This is support which gce-xfstests has. You can either have it watch
a particular branch on a git repo, and whenever it changes it will
kick off a tests:
gce-xfstests ltm [-c <cfg>] [-g <group>]|[<tests>] ... [--repo <url>] \
--watch <branch>
As an example:
gce-xfstests ltm -c ext4/all -g auto --repo ext4 --watch dev
Or you can specify a specific commit that you want to run tests on:
gce-xfstests ltm -c ext4/all -g auto --repo ext4 --commit dev
In the two examples above ext4 is an abbrevation so you don't have to
type the full URL; you can define additional abbrevs in your config
file.
> > 3) make it easier to auto-bisect what commit regressed when a failing test
> > is spotted
>
> Jeff Layton has mentioned this as well. I don't think
> it would be impossible to get kdevops to orchestrate
> a bisect, as long as it has an automatic way to decide
> when to use "git bisect {good|bad}"
gce-xfstests has this as well:
gce-xfstests ltm [-c <cfg>] [-g <group>]|[<tests>] ... [--repo <url>] \
--bisect-bad <bad_rev> --bisect-good <good_rev> \
[--bisect-good <good_rev1> ...]
So yeah, not that hard --- we had intern learn how to program in Go
and to implement the base lightweight test manager, and a team of
undergraduates who implemented the build server, test spinner, and
auto-bisector as part of their software engineering classes's project. :-)
The code is up on github at github.com/tytso/xfstests-bld if anyone
wants to repurpose it.
- Ted
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2024-05-13 13:18 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-05-07 18:44 kdevops BoF at LSFMM Luis Chamberlain
2024-05-07 18:56 ` Chuck Lever III
2024-05-08 7:45 ` Amir Goldstein
2024-05-08 15:57 ` Luis Chamberlain
2024-05-13 1:58 ` Matthew Wilcox
2024-05-08 17:45 ` Steve French
2024-05-08 17:54 ` Chuck Lever III
2024-05-12 20:20 ` Luis Chamberlain
2024-05-13 13:17 ` Theodore Ts'o
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).