From: Richard Purdie <richard.purdie@linuxfoundation.org>
To: openembedded-architecture
<openembedded-architecture@lists.openembedded.org>
Cc: openembedded-core <openembedded-core@lists.openembedded.org>,
"<yocto@lists.yoctoproject.org>" <yocto@lists.yoctoproject.org>
Subject: Where are we at?
Date: Thu, 09 May 2024 10:37:10 +0100 [thread overview]
Message-ID: <737087f797ecd87620bc96abab014d4fac6bb1e5.camel@linuxfoundation.org> (raw)
I think I need to write down a bit of a summary status on where we (and
I) am at with things.
The good news is that scarthgap is released. On the downside, we messed
up the public hash server url in local.conf.sample so we have a point
release in progress to fix that.
The Sovereign Tech Fund (STF) work is drawing to a close now, some
tweaks are needed on some patch series to get them over the line and
able to merge but the basic pieces are there. We have the PoC for
bitbake-setup too. I have not had the time to look at it in detail yet
(see below for an idea of why). That work brings us many steps forward
in many key areas of the project and takes a number of weights off my
mind. It adds some new ones too but change is at least refreshing.
It isn't all rosy, in particular the autobuilder is struggling. The
quality of builds for scarthgap isn't what I'd have liked and Steve is
already abandoning rc builds due to intermittent failures. We're seeing
the same failures with master.
In particular:
A rust reproducibility issue lurking somewhere:
https://autobuilder.yoctoproject.org/typhoon/#/builders/97/builds/8318
https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/8906/steps/17/logs/stdio
https://bugzilla.yoctoproject.org/show_bug.cgi?id=15185
[both master and scarthgap]
Intermittent do_compile failure in libportal:
https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/4760/steps/12/logs/stdio
debuginfod keeps breaking:
https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/3299/steps/14/logs/stdio
https://bugzilla.yoctoproject.org/show_bug.cgi?id=14937
qemuppc with intermittent issues:
https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/8906/steps/17/logs/stdio
qemuarm eSDK intermittent failure:
https://autobuilder.yoctoproject.org/typhoon/#/builders/53/builds/8946/steps/15/logs/stdio
(network glitch or a problem with git/curl/certs in the tools?)
We did have an intermittent python3 test hang. We've taken the decision
just to disable that test for now as we don't have the time to debug
it.
The weird hang in the 6.6 kernel in early boot also remains. Paul
Gortmaker made some progress in debugging it but I haven't found any
time to help/support him.
There is some kind of issue where the longer the arm servers have
uptime, the more broken the builds get too. It is hard to be more
specific, someone needs to do some analysis of the failure and try and
find the pattern.
We then get to the non oe-core failures that seem to be piling up:
meta-oe source mirroring giving warnings:
https://autobuilder.yoctoproject.org/typhoon/#/builders/156/builds/353/steps/12/logs/warnings
meta-oe source mirroring fetch failures:
https://autobuilder.yoctoproject.org/typhoon/#/builders/156/builds/353/steps/16/logs/stdio
kirkstone meta-oe metrics failures:
https://autobuilder.yoctoproject.org/typhoon/#/builders/138/builds/1612/steps/16/logs/stdio
meta-ti nightly layer check failures:
https://autobuilder.yoctoproject.org/typhoon/#/builders/121/builds/2064/steps/19/logs/stdio
Yes, some of these need reporting to the right people, e.g. meta-ti.
*Someone* actually needs to do it. If someone doesn't do it, someone
needs to notice and find/remind someone to do it. We expanded testing
to meta-oe on the basiss there would be help in addressing the issues
but few people seem to be paying attention :(.
Meanwhile, we have a pile of gcc 14 fixes piling up and pressure to get
gcc 14 in so we can build a new uninative to fix things for people on
Fedora 40 and arch. There are simple things others could do to help
like replying to patches about missing patch information.
The current autobuilder hardware is failing. We have a plan for a new
setup however that new setup has issues that need debugging. The one
resource I can direct to help with that is me, everyone else just runs
away from intermittent issues or says it is too hard or they don't have
access/experience/whatever.
We have challenges in the CVE space, I'm hoping Marta has some patches
coming which can help there soon. The open letter idea is all well and
good but it needs socialising with people outside the project.
There are also some bigger changes I think the project needs to make,
WORKDIR changes are one example, we also need to revisit the layer
dependency situation and "Yocto Project Compatible" needs work too,
amongst many other things.
Personally, I'm now really struggling. The intensity of the last few
months has taken a toll on me and I simply can't switch into handling
all of the above, I'm running on empty. This means I need a break,
everyone knows that and agrees. The twist is that yes, everyone
encourages me to do that, as long as I fix the issues on the current
autobuilder, the issues on the new one, review and merge patches, make
architecture/development changes at the right point in the release
cycle, ensure proposals made for key changes are well thought out and
don't upset anyone and so on.
I'm sharing this not as a complaint, but to try and show people that if
their patches aren't merging, it isn't that I/we don't care, it is just
that there simply aren't enough hours in the day and I'm at breaking
point if I don't let some things slow down.
One of the key issues is that I act like a traffic director in the
middle of all this. Nobody else is willing to step into all those
different streams and help try and tame/direct them. The downside to
that is that if I stop or slow down, things do break.
Obviously, help in any of the most regular autobuilder failures would
be much appreciated.
Cheers,
Richard
next reply other threads:[~2024-05-09 9:37 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-05-09 9:37 Richard Purdie [this message]
[not found] ` <IA0PR14MB6188558226DBC7D15C1B36EF85E62@IA0PR14MB6188.namprd14.prod.outlook.com>
2024-05-09 14:59 ` [yocto] Where are we at? Alexander Kanavin
2024-05-09 18:58 ` [OE-core] " Ross Burton
2024-05-10 16:03 ` [Openembedded-architecture] " Ross Burton
2024-05-10 17:58 ` [yocto] " Alexander Kanavin
[not found] ` <a325fd9d7b923e76e0cba8786c9a4dcda81fd52f.camel@linuxfoundation.org>
[not found] ` <a84b5ef9-2314-4ba7-99d0-b5292298f736@windriver.com>
[not found] ` <835b59367fd827253789514b77e042fc99b55593.camel@linuxfoundation.org>
2024-05-13 20:35 ` build host architecture leaking into target artefacts - new twist on reproducibility Alexander Kanavin
[not found] ` <17CF26B1EA08DEA4.31802@lists.openembedded.org>
2024-05-15 12:57 ` [OE-core] " Alexander Kanavin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=737087f797ecd87620bc96abab014d4fac6bb1e5.camel@linuxfoundation.org \
--to=richard.purdie@linuxfoundation.org \
--cc=openembedded-architecture@lists.openembedded.org \
--cc=openembedded-core@lists.openembedded.org \
--cc=yocto@lists.yoctoproject.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).