* When building OpenBMC . . . ?
@ 2020-08-30 22:02 Bruce Mitchell
2020-08-31 10:57 ` Brad Bishop
` (2 more replies)
0 siblings, 3 replies; 22+ messages in thread
From: Bruce Mitchell @ 2020-08-30 22:02 UTC (permalink / raw
To: openbmc@lists.ozlabs.org
When selecting Target hardware https://github.com/openbmc/openbmc#3-target-your-hardware
to build for the is a tiogapass, now if I add a meta-phoenix/meta-tiogapass/conf how does
source setup tiogapass build
know which tiogapass to build?
Or am I not supposed to choose a name (i.e. tiogapass in this example) that is already in the list
when I need to create a new meta-phoenix/meta-<machine>/conf?
Thanks!
--
Bruce
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: When building OpenBMC . . . ?
2020-08-30 22:02 When building OpenBMC . . . ? Bruce Mitchell
@ 2020-08-31 10:57 ` Brad Bishop
2020-08-31 14:19 ` Bruce Mitchell
2020-08-31 22:48 ` Vijay Khemka
2020-09-01 12:24 ` Patrick Williams
2 siblings, 1 reply; 22+ messages in thread
From: Brad Bishop @ 2020-08-31 10:57 UTC (permalink / raw
To: Bruce Mitchell; +Cc: openbmc@lists.ozlabs.org
On Sun, Aug 30, 2020 at 10:02:41PM +0000, Bruce Mitchell wrote:
>When selecting Target hardware https://github.com/openbmc/openbmc#3-target-your-hardware
>to build for the is a tiogapass, now if I add a meta-phoenix/meta-tiogapass/conf how does
> source setup tiogapass build
>know which tiogapass to build?
Are there two different systems called tiogapass? I hope we are not
creating two distinct sets of bitbake metadata for the same system?
-brad
^ permalink raw reply [flat|nested] 22+ messages in thread
* RE: When building OpenBMC . . . ?
2020-08-31 10:57 ` Brad Bishop
@ 2020-08-31 14:19 ` Bruce Mitchell
2020-08-31 14:29 ` Ed Tanous
0 siblings, 1 reply; 22+ messages in thread
From: Bruce Mitchell @ 2020-08-31 14:19 UTC (permalink / raw
To: Brad Bishop; +Cc: openbmc@lists.ozlabs.org
We are building a separate port for Tioga Pass, so the question is should be not call it tiogapass?
> -----Original Message-----
> From: openbmc [mailto:openbmc-
> bounces+bruce_mitchell=phoenix.com@lists.ozlabs.org] On Behalf Of
> Brad Bishop
> Sent: Monday, August 31, 2020 03:57
> To: Bruce Mitchell
> Cc: openbmc@lists.ozlabs.org
> Subject: Re: When building OpenBMC . . . ?
>
> On Sun, Aug 30, 2020 at 10:02:41PM +0000, Bruce Mitchell wrote:
> >When selecting Target hardware
> https://github.com/openbmc/openbmc#3-target-your-hardware
> >to build for the is a tiogapass, now if I add a meta-phoenix/meta-
> tiogapass/conf how does
> > source setup tiogapass build
> >know which tiogapass to build?
>
> Are there two different systems called tiogapass? I hope we are not
> creating two distinct sets of bitbake metadata for the same system?
>
> -brad
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: When building OpenBMC . . . ?
2020-08-31 14:19 ` Bruce Mitchell
@ 2020-08-31 14:29 ` Ed Tanous
2020-08-31 14:34 ` Bruce Mitchell
0 siblings, 1 reply; 22+ messages in thread
From: Ed Tanous @ 2020-08-31 14:29 UTC (permalink / raw
To: Bruce Mitchell; +Cc: Brad Bishop, openbmc@lists.ozlabs.org
On Mon, Aug 31, 2020 at 7:23 AM Bruce Mitchell
<Bruce_Mitchell@phoenix.com> wrote:
>
> We are building a separate port for Tioga Pass, so the question is should be not call it tiogapass?
>
Don't create a separate "port". Check your fixes into the Tioga pass
machine and get them reviewed. If there's conflicts with featuresets
with other tioga pass users (as I'm sure there will be) determine what
they are, then roll the required configurability options up to the
project level layers so users can select the features they want in a
build.
Please do not add meta-phoenix/meta-tioga. The per-machine meta
layers are complicated enough as-is.
^ permalink raw reply [flat|nested] 22+ messages in thread
* RE: When building OpenBMC . . . ?
2020-08-31 14:29 ` Ed Tanous
@ 2020-08-31 14:34 ` Bruce Mitchell
0 siblings, 0 replies; 22+ messages in thread
From: Bruce Mitchell @ 2020-08-31 14:34 UTC (permalink / raw
To: Ed Tanous; +Cc: Brad Bishop, openbmc@lists.ozlabs.org
Thank you Ed,
Again I will need to work with our build people and architects.
> -----Original Message-----
> From: Ed Tanous [mailto:ed@tanous.net]
> Sent: Monday, August 31, 2020 07:30
> To: Bruce Mitchell
> Cc: Brad Bishop; openbmc@lists.ozlabs.org
> Subject: Re: When building OpenBMC . . . ?
>
> On Mon, Aug 31, 2020 at 7:23 AM Bruce Mitchell
> <Bruce_Mitchell@phoenix.com> wrote:
> >
> > We are building a separate port for Tioga Pass, so the question is
> should be not call it tiogapass?
> >
>
> Don't create a separate "port". Check your fixes into the Tioga pass
> machine and get them reviewed. If there's conflicts with featuresets
> with other tioga pass users (as I'm sure there will be) determine what
> they are, then roll the required configurability options up to the
> project level layers so users can select the features they want in a
> build.
>
> Please do not add meta-phoenix/meta-tioga. The per-machine meta
> layers are complicated enough as-is.
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: When building OpenBMC . . . ?
2020-08-30 22:02 When building OpenBMC . . . ? Bruce Mitchell
2020-08-31 10:57 ` Brad Bishop
@ 2020-08-31 22:48 ` Vijay Khemka
2020-08-31 22:51 ` Bruce Mitchell
2020-09-01 12:24 ` Patrick Williams
2 siblings, 1 reply; 22+ messages in thread
From: Vijay Khemka @ 2020-08-31 22:48 UTC (permalink / raw
To: Bruce Mitchell, openbmc@lists.ozlabs.org
On 8/30/20, 3:04 PM, "openbmc on behalf of Bruce Mitchell" <openbmc-bounces+vijaykhemka=fb.com@lists.ozlabs.org on behalf of Bruce_Mitchell@phoenix.com> wrote:
When selecting Target hardware https://github.com/openbmc/openbmc#3-target-your-hardware
to build for the is a tiogapass, now if I add a meta-phoenix/meta-tiogapass/conf how does
source setup tiogapass build
know which tiogapass to build?
There is only one tiogapass in the system and it kbows where to build from.
Or am I not supposed to choose a name (i.e. tiogapass in this example) that is already in the list
when I need to create a new meta-phoenix/meta-<machine>/conf?
Thanks!
--
Bruce
^ permalink raw reply [flat|nested] 22+ messages in thread
* RE: When building OpenBMC . . . ?
2020-08-31 22:48 ` Vijay Khemka
@ 2020-08-31 22:51 ` Bruce Mitchell
2020-08-31 22:57 ` Vijay Khemka
0 siblings, 1 reply; 22+ messages in thread
From: Bruce Mitchell @ 2020-08-31 22:51 UTC (permalink / raw
To: Vijay Khemka, openbmc@lists.ozlabs.org
So you are implying to not add a second tiogapass, correct?
> -----Original Message-----
> From: Vijay Khemka [mailto:vijaykhemka@fb.com]
> Sent: Monday, August 31, 2020 15:49
> To: Bruce Mitchell; openbmc@lists.ozlabs.org
> Subject: Re: When building OpenBMC . . . ?
>
>
>
> On 8/30/20, 3:04 PM, "openbmc on behalf of Bruce Mitchell" <openbmc-
> bounces+vijaykhemka=fb.com@lists.ozlabs.org on behalf of
> Bruce_Mitchell@phoenix.com> wrote:
>
> When selecting Target hardware
> https://github.com/openbmc/openbmc#3-target-your-hardware
> to build for the is a tiogapass, now if I add a meta-phoenix/meta-
> tiogapass/conf how does
> source setup tiogapass build
> know which tiogapass to build?
> There is only one tiogapass in the system and it kbows where to build
> from.
>
> Or am I not supposed to choose a name (i.e. tiogapass in this example)
> that is already in the list
> when I need to create a new meta-phoenix/meta-<machine>/conf?
>
> Thanks!
>
> --
> Bruce
>
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: When building OpenBMC . . . ?
2020-08-31 22:51 ` Bruce Mitchell
@ 2020-08-31 22:57 ` Vijay Khemka
2020-09-01 16:25 ` Bruce Mitchell
0 siblings, 1 reply; 22+ messages in thread
From: Vijay Khemka @ 2020-08-31 22:57 UTC (permalink / raw
To: Bruce Mitchell, openbmc@lists.ozlabs.org
Yes, please add any patch in the same tiogapass platform layer
On 8/31/20, 3:51 PM, "Bruce Mitchell" <Bruce_Mitchell@phoenix.com> wrote:
So you are implying to not add a second tiogapass, correct?
> -----Original Message-----
> From: Vijay Khemka [mailto:vijaykhemka@fb.com]
> Sent: Monday, August 31, 2020 15:49
> To: Bruce Mitchell; openbmc@lists.ozlabs.org
> Subject: Re: When building OpenBMC . . . ?
>
>
>
> On 8/30/20, 3:04 PM, "openbmc on behalf of Bruce Mitchell" <openbmc-
> bounces+vijaykhemka=fb.com@lists.ozlabs.org on behalf of
> Bruce_Mitchell@phoenix.com> wrote:
>
> When selecting Target hardware
> https://github.com/openbmc/openbmc#3-target-your-hardware
> to build for the is a tiogapass, now if I add a meta-phoenix/meta-
> tiogapass/conf how does
> source setup tiogapass build
> know which tiogapass to build?
> There is only one tiogapass in the system and it kbows where to build
> from.
>
> Or am I not supposed to choose a name (i.e. tiogapass in this example)
> that is already in the list
> when I need to create a new meta-phoenix/meta-<machine>/conf?
>
> Thanks!
>
> --
> Bruce
>
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: When building OpenBMC . . . ?
2020-08-30 22:02 When building OpenBMC . . . ? Bruce Mitchell
2020-08-31 10:57 ` Brad Bishop
2020-08-31 22:48 ` Vijay Khemka
@ 2020-09-01 12:24 ` Patrick Williams
2020-09-01 16:09 ` Ed Tanous
2020-09-01 16:26 ` Bruce Mitchell
2 siblings, 2 replies; 22+ messages in thread
From: Patrick Williams @ 2020-09-01 12:24 UTC (permalink / raw
To: Bruce Mitchell; +Cc: openbmc@lists.ozlabs.org
[-- Attachment #1: Type: text/plain, Size: 2626 bytes --]
On Sun, Aug 30, 2020 at 10:02:41PM +0000, Bruce Mitchell wrote:
> When selecting Target hardware https://github.com/openbmc/openbmc#3-target-your-hardware
> to build for the is a tiogapass, now if I add a meta-phoenix/meta-tiogapass/conf how does
> source setup tiogapass build
> know which tiogapass to build?
https://github.com/openbmc/openbmc/blob/master/setup#L34
The setup script just does a wildcard, which means you'll get the
alphabetically ordered machine. In this case, you should get the
meta-facebook one selected before the meta-phoenix (supposing they both
exist).
> Or am I not supposed to choose a name (i.e. tiogapass in this example) that is already in the list
> when I need to create a new meta-phoenix/meta-<machine>/conf?
The overwhelming preference seems to be to not make another
configuration with the same machine, and as one of the maintainers of
meta-facebook, I agree in this case. But, this answer doesn't solve
your underlying question.
I suspect you're going to make two kinds of changes:
1. Features you want to enable on Tiogapass that Facebook isn't
interested in. (I would cover bmcweb 'branding' changes here
also).
2. Fixes and configuration due to features we haven't enabled yet or
fully vetted.
#2 should go into either meta-facebook (or the underlying code
repository where the fix is needed). These will be common for any
tiogapass hardware, so lets keep it in the common location.
#1 should go into meta-phoenix. You're likely the first one doing this,
so we may need some experimentation on the best option. I have two
ideas (there are probably others):
* Make an alternative tiogapass variant, like tiogapass-phoenix, which
ends up including all the common tiogapass code from meta-facebook.
* Create a new distro type for phoenix, which enhances the underlying
openbmc distribution with your own branding tweaks. You'd still
build meta-facebook/tiogapass but with a different distro flavor.
I believe IBM has experiemented with both of these approaches for
witherspoon (see witherspoon-tacoma and
meta-ibm/conf/distro/openbmc-witherspoon.conf) and might have some
insight into what has worked well for them.
I'm more than willing to work with you and your team to help refactor
meta-facebook/tiogapass in a way that makes it more condusive to what
your team is interested in doing. I suspect we'll need to create some
additional bitbake '.inc' files and move some of the content we have in
'.conf' to '.inc'. Catch me here or on IRC as needed.
--
Patrick Williams
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: When building OpenBMC . . . ?
2020-09-01 12:24 ` Patrick Williams
@ 2020-09-01 16:09 ` Ed Tanous
2020-09-01 16:20 ` Patrick Williams
` (2 more replies)
2020-09-01 16:26 ` Bruce Mitchell
1 sibling, 3 replies; 22+ messages in thread
From: Ed Tanous @ 2020-09-01 16:09 UTC (permalink / raw
To: Patrick Williams; +Cc: Bruce Mitchell, openbmc@lists.ozlabs.org
On Tue, Sep 1, 2020 at 5:26 AM Patrick Williams <patrick@stwcx.xyz> wrote:
>
> On Sun, Aug 30, 2020 at 10:02:41PM +0000, Bruce Mitchell wrote:
> > When selecting Target hardware https://github.com/openbmc/openbmc#3-target-your-hardware
> > to build for the is a tiogapass, now if I add a meta-phoenix/meta-tiogapass/conf how does
> > source setup tiogapass build
> > know which tiogapass to build?
>
> https://github.com/openbmc/openbmc/blob/master/setup#L34
>
> The setup script just does a wildcard, which means you'll get the
> alphabetically ordered machine. In this case, you should get the
> meta-facebook one selected before the meta-phoenix (supposing they both
> exist).
>
> > Or am I not supposed to choose a name (i.e. tiogapass in this example) that is already in the list
> > when I need to create a new meta-phoenix/meta-<machine>/conf?
>
> The overwhelming preference seems to be to not make another
> configuration with the same machine, and as one of the maintainers of
> meta-facebook, I agree in this case. But, this answer doesn't solve
> your underlying question.
>
> I suspect you're going to make two kinds of changes:
> 1. Features you want to enable on Tiogapass that Facebook isn't
> interested in. (I would cover bmcweb 'branding' changes here
> also).
bmcweb branding is a machine independent feature. At some point we
might have the webui feature enable/disable things again, and who
knows, maybe we need machine specific branding, but I want to avoid it
wherever possible.
> 2. Fixes and configuration due to features we haven't enabled yet or
> fully vetted.
>
> #2 should go into either meta-facebook (or the underlying code
> repository where the fix is needed). These will be common for any
+1
Could we also make the statement that as a project, we will enable
every platform feature we are able to for every platform by default,
and if a company wants to specifically disable some features for their
use because they haven't vetted them, they should do that in a
specific distro? Said another way, the "default" for every machine
should be every feature enabled, as that's what helps users and
developers the most.
>
> #1 should go into meta-phoenix. You're likely the first one doing this,
> so we may need some experimentation on the best option. I have two
> ideas (there are probably others):
>
> * Make an alternative tiogapass variant, like tiogapass-phoenix, which
> ends up including all the common tiogapass code from meta-facebook.
>
> * Create a new distro type for phoenix, which enhances the underlying
> openbmc distribution with your own branding tweaks. You'd still
> build meta-facebook/tiogapass but with a different distro flavor.
This one would be my vote between the two, and I think there's
precedent with other companies doing similar things. Isn't this the
way yocto recommends?
>
> I believe IBM has experiemented with both of these approaches for
> witherspoon (see witherspoon-tacoma and
> meta-ibm/conf/distro/openbmc-witherspoon.conf) and might have some
> insight into what has worked well for them.
>
> I'm more than willing to work with you and your team to help refactor
> meta-facebook/tiogapass in a way that makes it more condusive to what
> your team is interested in doing. I suspect we'll need to create some
> additional bitbake '.inc' files and move some of the content we have in
> '.conf' to '.inc'. Catch me here or on IRC as needed.
>
> --
> Patrick Williams
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: When building OpenBMC . . . ?
2020-09-01 16:09 ` Ed Tanous
@ 2020-09-01 16:20 ` Patrick Williams
2020-09-01 16:29 ` Bruce Mitchell
2020-09-01 16:56 ` Ed Tanous
2020-09-01 16:28 ` Bruce Mitchell
2020-09-02 18:50 ` Patrick Voelker
2 siblings, 2 replies; 22+ messages in thread
From: Patrick Williams @ 2020-09-01 16:20 UTC (permalink / raw
To: Ed Tanous; +Cc: Bruce Mitchell, openbmc@lists.ozlabs.org
[-- Attachment #1: Type: text/plain, Size: 1595 bytes --]
On Tue, Sep 01, 2020 at 09:09:33AM -0700, Ed Tanous wrote:
> On Tue, Sep 1, 2020 at 5:26 AM Patrick Williams <patrick@stwcx.xyz> wrote:
> > On Sun, Aug 30, 2020 at 10:02:41PM +0000, Bruce Mitchell wrote:
> >
> > #2 should go into either meta-facebook (or the underlying code
> > repository where the fix is needed). These will be common for any
>
> +1
>
> Could we also make the statement that as a project, we will enable
> every platform feature we are able to for every platform by default,
> and if a company wants to specifically disable some features for their
> use because they haven't vetted them, they should do that in a
> specific distro? Said another way, the "default" for every machine
> should be every feature enabled, as that's what helps users and
> developers the most.
I think this is where we get some conflict between, for lack of better
words, commercial and hyperscale philosphies. We may make a decision
that we don't want net-ipmi in our datacenter, for security reasons, so
we have it disabled in our meta-facebook layer. Yes, we could disable
it dynamically like a customer of a commercial vendor might do, but it
is simpler to not even have the code in the image.
Today we've combined machine definition and image definition into a
single meta-layer across the board. This is probably reasonable for
a single vendor who designs their own machine in-house, but is less
reasonable for cases like Facebook where we do our work within OCP and
others can order the servers we've designed from various ODMs.
--
Patrick Williams
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 22+ messages in thread
* RE: When building OpenBMC . . . ?
2020-08-31 22:57 ` Vijay Khemka
@ 2020-09-01 16:25 ` Bruce Mitchell
0 siblings, 0 replies; 22+ messages in thread
From: Bruce Mitchell @ 2020-09-01 16:25 UTC (permalink / raw
To: Vijay Khemka, openbmc@lists.ozlabs.org
Thank you Vijay.
> -----Original Message-----
> From: Vijay Khemka [mailto:vijaykhemka@fb.com]
> Sent: Monday, August 31, 2020 15:57
> To: Bruce Mitchell; openbmc@lists.ozlabs.org
> Subject: Re: When building OpenBMC . . . ?
>
> Yes, please add any patch in the same tiogapass platform layer
>
> On 8/31/20, 3:51 PM, "Bruce Mitchell" <Bruce_Mitchell@phoenix.com>
> wrote:
>
> So you are implying to not add a second tiogapass, correct?
>
> > -----Original Message-----
> > From: Vijay Khemka [mailto:vijaykhemka@fb.com]
> > Sent: Monday, August 31, 2020 15:49
> > To: Bruce Mitchell; openbmc@lists.ozlabs.org
> > Subject: Re: When building OpenBMC . . . ?
> >
> >
> >
> > On 8/30/20, 3:04 PM, "openbmc on behalf of Bruce Mitchell"
> <openbmc-
> > bounces+vijaykhemka=fb.com@lists.ozlabs.org on behalf of
> > Bruce_Mitchell@phoenix.com> wrote:
> >
> > When selecting Target hardware
> > https://github.com/openbmc/openbmc#3-target-your-hardware
> > to build for the is a tiogapass, now if I add a meta-phoenix/meta-
> > tiogapass/conf how does
> > source setup tiogapass build
> > know which tiogapass to build?
> > There is only one tiogapass in the system and it kbows where to build
> > from.
> >
> > Or am I not supposed to choose a name (i.e. tiogapass in this
> example)
> > that is already in the list
> > when I need to create a new meta-phoenix/meta-<machine>/conf?
> >
> > Thanks!
> >
> > --
> > Bruce
> >
>
^ permalink raw reply [flat|nested] 22+ messages in thread
* RE: When building OpenBMC . . . ?
2020-09-01 12:24 ` Patrick Williams
2020-09-01 16:09 ` Ed Tanous
@ 2020-09-01 16:26 ` Bruce Mitchell
1 sibling, 0 replies; 22+ messages in thread
From: Bruce Mitchell @ 2020-09-01 16:26 UTC (permalink / raw
To: Patrick Williams; +Cc: openbmc@lists.ozlabs.org
Thank you Patrick! Your #1 and #2 scenario are close to what we are attempt.
> -----Original Message-----
> From: Patrick Williams [mailto:patrick@stwcx.xyz]
> Sent: Tuesday, September 1, 2020 05:24
> To: Bruce Mitchell
> Cc: openbmc@lists.ozlabs.org
> Subject: Re: When building OpenBMC . . . ?
>
> On Sun, Aug 30, 2020 at 10:02:41PM +0000, Bruce Mitchell wrote:
> > When selecting Target hardware
> > https://github.com/openbmc/openbmc#3-target-your-hardware
> > to build for the is a tiogapass, now if I add a meta-phoenix/meta-
> tiogapass/conf how does
> > source setup tiogapass build
> > know which tiogapass to build?
>
> https://github.com/openbmc/openbmc/blob/master/setup#L34
>
> The setup script just does a wildcard, which means you'll get the
> alphabetically ordered machine. In this case, you should get the meta-
> facebook one selected before the meta-phoenix (supposing they both
> exist).
>
> > Or am I not supposed to choose a name (i.e. tiogapass in this example)
> > that is already in the list when I need to create a new meta-
> phoenix/meta-<machine>/conf?
>
> The overwhelming preference seems to be to not make another
> configuration with the same machine, and as one of the maintainers of
> meta-facebook, I agree in this case. But, this answer doesn't solve your
> underlying question.
>
> I suspect you're going to make two kinds of changes:
> 1. Features you want to enable on Tiogapass that Facebook isn't
> interested in. (I would cover bmcweb 'branding' changes here
> also).
> 2. Fixes and configuration due to features we haven't enabled yet or
> fully vetted.
>
> #2 should go into either meta-facebook (or the underlying code
> repository where the fix is needed). These will be common for any
> tiogapass hardware, so lets keep it in the common location.
>
> #1 should go into meta-phoenix. You're likely the first one doing this, so
> we may need some experimentation on the best option. I have two
> ideas (there are probably others):
>
> * Make an alternative tiogapass variant, like tiogapass-phoenix, which
> ends up including all the common tiogapass code from meta-facebook.
>
> * Create a new distro type for phoenix, which enhances the underlying
> openbmc distribution with your own branding tweaks. You'd still
> build meta-facebook/tiogapass but with a different distro flavor.
>
> I believe IBM has experiemented with both of these approaches for
> witherspoon (see witherspoon-tacoma and
> meta-ibm/conf/distro/openbmc-witherspoon.conf) and might have
> some insight into what has worked well for them.
>
> I'm more than willing to work with you and your team to help refactor
> meta-facebook/tiogapass in a way that makes it more condusive to what
> your team is interested in doing. I suspect we'll need to create some
> additional bitbake '.inc' files and move some of the content we have in
> '.conf' to '.inc'. Catch me here or on IRC as needed.
>
> --
> Patrick Williams
^ permalink raw reply [flat|nested] 22+ messages in thread
* RE: When building OpenBMC . . . ?
2020-09-01 16:09 ` Ed Tanous
2020-09-01 16:20 ` Patrick Williams
@ 2020-09-01 16:28 ` Bruce Mitchell
2020-09-02 18:50 ` Patrick Voelker
2 siblings, 0 replies; 22+ messages in thread
From: Bruce Mitchell @ 2020-09-01 16:28 UTC (permalink / raw
To: Ed Tanous, Patrick Williams; +Cc: openbmc@lists.ozlabs.org
Thank you Ed! Your suggest of what Yocto recommends is good and we will investigate further.
> -----Original Message-----
> From: Ed Tanous [mailto:ed@tanous.net]
> Sent: Tuesday, September 1, 2020 09:10
> To: Patrick Williams
> Cc: Bruce Mitchell; openbmc@lists.ozlabs.org
> Subject: Re: When building OpenBMC . . . ?
>
> On Tue, Sep 1, 2020 at 5:26 AM Patrick Williams <patrick@stwcx.xyz>
> wrote:
> >
> > On Sun, Aug 30, 2020 at 10:02:41PM +0000, Bruce Mitchell wrote:
> > > When selecting Target hardware
> https://github.com/openbmc/openbmc#3-target-your-hardware
> > > to build for the is a tiogapass, now if I add a meta-phoenix/meta-
> tiogapass/conf how does
> > > source setup tiogapass build
> > > know which tiogapass to build?
> >
> > https://github.com/openbmc/openbmc/blob/master/setup#L34
> >
> > The setup script just does a wildcard, which means you'll get the
> > alphabetically ordered machine. In this case, you should get the
> > meta-facebook one selected before the meta-phoenix (supposing they
> both
> > exist).
> >
> > > Or am I not supposed to choose a name (i.e. tiogapass in this example)
> that is already in the list
> > > when I need to create a new meta-phoenix/meta-<machine>/conf?
> >
> > The overwhelming preference seems to be to not make another
> > configuration with the same machine, and as one of the maintainers of
> > meta-facebook, I agree in this case. But, this answer doesn't solve
> > your underlying question.
> >
> > I suspect you're going to make two kinds of changes:
> > 1. Features you want to enable on Tiogapass that Facebook isn't
> > interested in. (I would cover bmcweb 'branding' changes here
> > also).
>
> bmcweb branding is a machine independent feature. At some point we
> might have the webui feature enable/disable things again, and who
> knows, maybe we need machine specific branding, but I want to avoid it
> wherever possible.
>
> > 2. Fixes and configuration due to features we haven't enabled yet or
> > fully vetted.
> >
> > #2 should go into either meta-facebook (or the underlying code
> > repository where the fix is needed). These will be common for any
>
> +1
>
> Could we also make the statement that as a project, we will enable
> every platform feature we are able to for every platform by default,
> and if a company wants to specifically disable some features for their
> use because they haven't vetted them, they should do that in a
> specific distro? Said another way, the "default" for every machine
> should be every feature enabled, as that's what helps users and
> developers the most.
>
> >
> > #1 should go into meta-phoenix. You're likely the first one doing this,
> > so we may need some experimentation on the best option. I have two
> > ideas (there are probably others):
> >
> > * Make an alternative tiogapass variant, like tiogapass-phoenix, which
> > ends up including all the common tiogapass code from meta-
> facebook.
> >
> > * Create a new distro type for phoenix, which enhances the
> underlying
> > openbmc distribution with your own branding tweaks. You'd still
> > build meta-facebook/tiogapass but with a different distro flavor.
>
> This one would be my vote between the two, and I think there's
> precedent with other companies doing similar things. Isn't this the
> way yocto recommends?
>
> >
> > I believe IBM has experiemented with both of these approaches for
> > witherspoon (see witherspoon-tacoma and
> > meta-ibm/conf/distro/openbmc-witherspoon.conf) and might have
> some
> > insight into what has worked well for them.
> >
> > I'm more than willing to work with you and your team to help refactor
> > meta-facebook/tiogapass in a way that makes it more condusive to
> what
> > your team is interested in doing. I suspect we'll need to create some
> > additional bitbake '.inc' files and move some of the content we have in
> > '.conf' to '.inc'. Catch me here or on IRC as needed.
> >
> > --
> > Patrick Williams
^ permalink raw reply [flat|nested] 22+ messages in thread
* RE: When building OpenBMC . . . ?
2020-09-01 16:20 ` Patrick Williams
@ 2020-09-01 16:29 ` Bruce Mitchell
2020-09-01 16:56 ` Ed Tanous
1 sibling, 0 replies; 22+ messages in thread
From: Bruce Mitchell @ 2020-09-01 16:29 UTC (permalink / raw
To: Patrick Williams, Ed Tanous; +Cc: openbmc@lists.ozlabs.org
Thanks again Patrick, I see your point and will that up with our team as well.
> -----Original Message-----
> From: Patrick Williams [mailto:patrick@stwcx.xyz]
> Sent: Tuesday, September 1, 2020 09:20
> To: Ed Tanous
> Cc: Bruce Mitchell; openbmc@lists.ozlabs.org
> Subject: Re: When building OpenBMC . . . ?
>
> On Tue, Sep 01, 2020 at 09:09:33AM -0700, Ed Tanous wrote:
> > On Tue, Sep 1, 2020 at 5:26 AM Patrick Williams <patrick@stwcx.xyz>
> wrote:
> > > On Sun, Aug 30, 2020 at 10:02:41PM +0000, Bruce Mitchell wrote:
> > >
> > > #2 should go into either meta-facebook (or the underlying code
> > > repository where the fix is needed). These will be common for any
> >
> > +1
> >
> > Could we also make the statement that as a project, we will enable
> > every platform feature we are able to for every platform by default,
> > and if a company wants to specifically disable some features for their
> > use because they haven't vetted them, they should do that in a
> > specific distro? Said another way, the "default" for every machine
> > should be every feature enabled, as that's what helps users and
> > developers the most.
>
> I think this is where we get some conflict between, for lack of better
> words, commercial and hyperscale philosphies. We may make a decision
> that we don't want net-ipmi in our datacenter, for security reasons, so
> we have it disabled in our meta-facebook layer. Yes, we could disable it
> dynamically like a customer of a commercial vendor might do, but it is
> simpler to not even have the code in the image.
>
> Today we've combined machine definition and image definition into a
> single meta-layer across the board. This is probably reasonable for a
> single vendor who designs their own machine in-house, but is less
> reasonable for cases like Facebook where we do our work within OCP
> and others can order the servers we've designed from various ODMs.
>
> --
> Patrick Williams
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: When building OpenBMC . . . ?
2020-09-01 16:20 ` Patrick Williams
2020-09-01 16:29 ` Bruce Mitchell
@ 2020-09-01 16:56 ` Ed Tanous
1 sibling, 0 replies; 22+ messages in thread
From: Ed Tanous @ 2020-09-01 16:56 UTC (permalink / raw
To: Patrick Williams; +Cc: Bruce Mitchell, openbmc@lists.ozlabs.org
On Tue, Sep 1, 2020 at 9:20 AM Patrick Williams <patrick@stwcx.xyz> wrote:
>
> On Tue, Sep 01, 2020 at 09:09:33AM -0700, Ed Tanous wrote:
> > On Tue, Sep 1, 2020 at 5:26 AM Patrick Williams <patrick@stwcx.xyz> wrote:
> > > On Sun, Aug 30, 2020 at 10:02:41PM +0000, Bruce Mitchell wrote:
> > >
> > > #2 should go into either meta-facebook (or the underlying code
> > > repository where the fix is needed). These will be common for any
> >
> > +1
> >
> > Could we also make the statement that as a project, we will enable
> > every platform feature we are able to for every platform by default,
> > and if a company wants to specifically disable some features for their
> > use because they haven't vetted them, they should do that in a
> > specific distro? Said another way, the "default" for every machine
> > should be every feature enabled, as that's what helps users and
> > developers the most.
>
> I think this is where we get some conflict between, for lack of better
> words, commercial and hyperscale philosphies. We may make a decision
> that we don't want net-ipmi in our datacenter, for security reasons, so
> we have it disabled in our meta-facebook layer. Yes, we could disable
> it dynamically like a customer of a commercial vendor might do, but it
> is simpler to not even have the code in the image.
>
> Today we've combined machine definition and image definition into a
> single meta-layer across the board. This is probably reasonable for
> a single vendor who designs their own machine in-house
Single vendors have the same problem. Customers of said vendor want a
build that "just works the way they want", and don't want to mess
around with changing configurations per-machine, uploading public
certificates, uploading webui customization, or having the possibility
that an insecure protocol accidentally gets enabled in prod.
>, but is less
> reasonable for cases like Facebook where we do our work within OCP and
> others can order the servers we've designed from various ODMs.
There are companies using hyperscaler platforms that have net-ipmi
enabled. There are companies that have transition plans to replace
one protocol with another, and at some point, will "flip the switch"
moving from one protocol to another. I think explicitly separating an
OpenBMC featureset (for lack of a better word) from an OpenBMC
supported machine will lead to a better result overall. I also think
it has some nice scaling properties as hyperscalers ratchet up the
number of system types they support, they can have more reuse of
featuresets between machines. Also, when debugging said machines,
it's really nice to have a folder (meta layer) of "what's different
between machines" that's separate from "what's different between our
build". I've used that many....many times to do A-B compares to find
elusive bugs.
Anywho, I think we're mostly agreeing.
>
> --
> Patrick Williams
^ permalink raw reply [flat|nested] 22+ messages in thread
* RE: When building OpenBMC . . . ?
2020-09-01 16:09 ` Ed Tanous
2020-09-01 16:20 ` Patrick Williams
2020-09-01 16:28 ` Bruce Mitchell
@ 2020-09-02 18:50 ` Patrick Voelker
2020-09-02 19:10 ` Patrick Williams
2 siblings, 1 reply; 22+ messages in thread
From: Patrick Voelker @ 2020-09-02 18:50 UTC (permalink / raw
To: Ed Tanous, Patrick Williams; +Cc: Bruce Mitchell, openbmc@lists.ozlabs.org
I'm giving the first option below a try. I've defined an alternative variant and have included the meta-facebook/meta-tiogapass layer in my build.
One problem I'm running into is that meta-tiogapass includes a rsyslog*.bbappend and one of the other layers I'm using also has a similar rsyslog*.bbappend.
Each do an append to do_install() and each one tries to remove ${D}${sysconfdir}/rsyslog.d/imjournal.conf. Of course that file can only be removed once so the build fails.
My question now, is what's the best way to work around this? I don't need rsyslog from meta-tiogapass, just the machine specifics.
> -----Original Message-----
> From: openbmc [mailto:openbmc-
> bounces+patrick_voelker=phoenix.com@lists.ozlabs.org] On Behalf Of Ed
> Tanous
> Sent: Tuesday, September 1, 2020 9:10 AM
> To: Patrick Williams
> Cc: Bruce Mitchell; openbmc@lists.ozlabs.org
> Subject: Re: When building OpenBMC . . . ?
>
<snip>
> > #1 should go into meta-phoenix. You're likely the first one doing this,
> > so we may need some experimentation on the best option. I have two
> > ideas (there are probably others):
> >
> > * Make an alternative tiogapass variant, like tiogapass-phoenix, which
> > ends up including all the common tiogapass code from meta-facebook.
> >
> > * Create a new distro type for phoenix, which enhances the underlying
> > openbmc distribution with your own branding tweaks. You'd still
> > build meta-facebook/tiogapass but with a different distro flavor.
>
> This one would be my vote between the two, and I think there's
> precedent with other companies doing similar things. Isn't this the
> way yocto recommends?
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: When building OpenBMC . . . ?
2020-09-02 18:50 ` Patrick Voelker
@ 2020-09-02 19:10 ` Patrick Williams
2020-09-02 19:50 ` Patrick Voelker
0 siblings, 1 reply; 22+ messages in thread
From: Patrick Williams @ 2020-09-02 19:10 UTC (permalink / raw
To: Patrick Voelker; +Cc: Ed Tanous, Bruce Mitchell, openbmc@lists.ozlabs.org
[-- Attachment #1: Type: text/plain, Size: 1162 bytes --]
On Wed, Sep 02, 2020 at 06:50:01PM +0000, Patrick Voelker wrote:
> I'm giving the first option below a try. I've defined an alternative variant and have included the meta-facebook/meta-tiogapass layer in my build.
>
> One problem I'm running into is that meta-tiogapass includes a rsyslog*.bbappend and one of the other layers I'm using also has a similar rsyslog*.bbappend.
>
> Each do an append to do_install() and each one tries to remove ${D}${sysconfdir}/rsyslog.d/imjournal.conf. Of course that file can only be removed once so the build fails.
>
> My question now, is what's the best way to work around this? I don't need rsyslog from meta-tiogapass, just the machine specifics.
If this is a common pattern, we should try to contribute it upstream to
Yocto as a PACKAGECONFIG option. Then we can add to the PACKAGECONFIG
in the bbappend (you can do that as many times as you want).
If we don't think Yocto would accept it, or they reject it, but it is
still something we're seeing often in our systems we can similarly add
it as a common bbappend in meta-phosphor (ideally triggered by a
PACKAGECONFIG).
--
Patrick Williams
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 22+ messages in thread
* RE: When building OpenBMC . . . ?
2020-09-02 19:10 ` Patrick Williams
@ 2020-09-02 19:50 ` Patrick Voelker
2020-09-02 21:39 ` Bills, Jason M
2020-09-03 15:33 ` Patrick Williams
0 siblings, 2 replies; 22+ messages in thread
From: Patrick Voelker @ 2020-09-02 19:50 UTC (permalink / raw
To: Patrick Williams; +Cc: Ed Tanous, Bruce Mitchell, openbmc@lists.ozlabs.org
> On Wed, Sep 02, 2020 at 06:50:01PM +0000, Patrick Voelker wrote:
> > I'm giving the first option below a try. I've defined an alternative variant
> and have included the meta-facebook/meta-tiogapass layer in my build.
> >
> > One problem I'm running into is that meta-tiogapass includes a
> rsyslog*.bbappend and one of the other layers I'm using also has a similar
> rsyslog*.bbappend.
> >
> > Each do an append to do_install() and each one tries to remove
> ${D}${sysconfdir}/rsyslog.d/imjournal.conf. Of course that file can only be
> removed once so the build fails.
> >
> > My question now, is what's the best way to work around this? I don't need
> rsyslog from meta-tiogapass, just the machine specifics.
>
> If this is a common pattern, we should try to contribute it upstream to Yocto
> as a PACKAGECONFIG option. Then we can add to the PACKAGECONFIG in
> the bbappend (you can do that as many times as you want).
>
> If we don't think Yocto would accept it, or they reject it, but it is still
> something we're seeing often in our systems we can similarly add it as a
> common bbappend in meta-phosphor (ideally triggered by a
> PACKAGECONFIG).
Thanks for the response but I'm having a hard time connecting the dots.
My understanding of PACKAGECONFIG is that it's a way to provide build options for individual packages. In this case, what PACKAGECONFIG option would we be contributing?
Is there a way now for me to force bitbake to ignore (or not use) rsyslog*.bbappend in meta-tiogapass from another layer?
The two appends that are conflicting are:
meta-facebook/meta-tiogapass/recipes-extended/rsyslog/rsyslog_%.bbappend
meta-intel/meta-common/recipes-extended/rsyslog/rsyslog_%.bbappend
Can I choose one over the other instead of having them build upon eachother?
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: When building OpenBMC . . . ?
2020-09-02 19:50 ` Patrick Voelker
@ 2020-09-02 21:39 ` Bills, Jason M
2020-09-02 23:35 ` Patrick Voelker
2020-09-03 15:33 ` Patrick Williams
1 sibling, 1 reply; 22+ messages in thread
From: Bills, Jason M @ 2020-09-02 21:39 UTC (permalink / raw
To: openbmc
On 9/2/2020 12:50 PM, Patrick Voelker wrote:
>
>> On Wed, Sep 02, 2020 at 06:50:01PM +0000, Patrick Voelker wrote:
>>> I'm giving the first option below a try. I've defined an alternative variant
>> and have included the meta-facebook/meta-tiogapass layer in my build.
>>>
>>> One problem I'm running into is that meta-tiogapass includes a
>> rsyslog*.bbappend and one of the other layers I'm using also has a similar
>> rsyslog*.bbappend.
>>>
>>> Each do an append to do_install() and each one tries to remove
>> ${D}${sysconfdir}/rsyslog.d/imjournal.conf. Of course that file can only be
>> removed once so the build fails.
>>>
>>> My question now, is what's the best way to work around this? I don't need
>> rsyslog from meta-tiogapass, just the machine specifics.
>>
>> If this is a common pattern, we should try to contribute it upstream to Yocto
>> as a PACKAGECONFIG option. Then we can add to the PACKAGECONFIG in
>> the bbappend (you can do that as many times as you want).
>>
>> If we don't think Yocto would accept it, or they reject it, but it is still
>> something we're seeing often in our systems we can similarly add it as a
>> common bbappend in meta-phosphor (ideally triggered by a
>> PACKAGECONFIG).
>
> Thanks for the response but I'm having a hard time connecting the dots.
>
> My understanding of PACKAGECONFIG is that it's a way to provide build options for individual packages. In this case, what PACKAGECONFIG option would we be contributing?
>
> Is there a way now for me to force bitbake to ignore (or not use) rsyslog*.bbappend in meta-tiogapass from another layer?
>
> The two appends that are conflicting are:
> meta-facebook/meta-tiogapass/recipes-extended/rsyslog/rsyslog_%.bbappend
> meta-intel/meta-common/recipes-extended/rsyslog/rsyslog_%.bbappend
I hit a similar issue when moving this recipe out of my downstream
build. I was able to resolve it by putting this change in my downstream
version:
diff --git a/meta-common/recipes-extended/rsyslog/rsyslog_%.bbappend
b/meta-common/recipes-extended/rsyslog/rsyslog_%.bbappend
index 7e282804..ef670451 100644
--- a/meta-common/recipes-extended/rsyslog/rsyslog_%.bbappend
+++ b/meta-common/recipes-extended/rsyslog/rsyslog_%.bbappend
@@ -17,7 +17,7 @@ do_install_append() {
install -d ${D}${systemd_system_unitdir}/rsyslog.service.d
install -m 0644 ${WORKDIR}/rsyslog-override.conf \
${D}${systemd_system_unitdir}/rsyslog.service.d/rsyslog-override.conf
- rm ${D}${sysconfdir}/rsyslog.d/imjournal.conf
+ rm -f ${D}${sysconfdir}/rsyslog.d/imjournal.conf
}
SYSTEMD_SERVICE_${PN} += " rotate-event-logs.service
rotate-event-logs.timer"
We can apply a similar change to one or both of these upstream recipes.
Or, is this a candidate for meta-phosphor?
>
> Can I choose one over the other instead of having them build upon eachother?
>
^ permalink raw reply related [flat|nested] 22+ messages in thread
* RE: When building OpenBMC . . . ?
2020-09-02 21:39 ` Bills, Jason M
@ 2020-09-02 23:35 ` Patrick Voelker
0 siblings, 0 replies; 22+ messages in thread
From: Patrick Voelker @ 2020-09-02 23:35 UTC (permalink / raw
To: Bills, Jason M, openbmc@lists.ozlabs.org
> -----Original Message-----
> From: openbmc [mailto:openbmc-
> bounces+patrick_voelker=phoenix.com@lists.ozlabs.org] On Behalf Of Bills,
> Jason M
> Sent: Wednesday, September 2, 2020 2:40 PM
> To: openbmc@lists.ozlabs.org
> Subject: Re: When building OpenBMC . . . ?
<snip>
> > Is there a way now for me to force bitbake to ignore (or not use)
> rsyslog*.bbappend in meta-tiogapass from another layer?
> >
> > The two appends that are conflicting are:
> > meta-facebook/meta-tiogapass/recipes-
> extended/rsyslog/rsyslog_%.bbappend
> > meta-intel/meta-common/recipes-extended/rsyslog/rsyslog_%.bbappend
> I hit a similar issue when moving this recipe out of my downstream
> build. I was able to resolve it by putting this change in my downstream
> version:
> diff --git a/meta-common/recipes-extended/rsyslog/rsyslog_%.bbappend
> b/meta-common/recipes-extended/rsyslog/rsyslog_%.bbappend
> index 7e282804..ef670451 100644
> --- a/meta-common/recipes-extended/rsyslog/rsyslog_%.bbappend
> +++ b/meta-common/recipes-extended/rsyslog/rsyslog_%.bbappend
> @@ -17,7 +17,7 @@ do_install_append() {
> install -d ${D}${systemd_system_unitdir}/rsyslog.service.d
> install -m 0644 ${WORKDIR}/rsyslog-override.conf \
>
> ${D}${systemd_system_unitdir}/rsyslog.service.d/rsyslog-override.conf
> - rm ${D}${sysconfdir}/rsyslog.d/imjournal.conf
> + rm -f ${D}${sysconfdir}/rsyslog.d/imjournal.conf
> }
>
> SYSTEMD_SERVICE_${PN} += " rotate-event-logs.service
> rotate-event-logs.timer"
>
> We can apply a similar change to one or both of these upstream recipes.
> Or, is this a candidate for meta-phosphor?
Yes, I think the '-f' argument would be a good option to upstream to both .bbappend files. That would prevent conflict in case they both get used.
Conceptually though it doesn't make sense to keep layering the do_install() instructions in this way. Is there a way around that? Or can the meta-facebook/meta-tiogapass version be moved elsewhere?
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: When building OpenBMC . . . ?
2020-09-02 19:50 ` Patrick Voelker
2020-09-02 21:39 ` Bills, Jason M
@ 2020-09-03 15:33 ` Patrick Williams
1 sibling, 0 replies; 22+ messages in thread
From: Patrick Williams @ 2020-09-03 15:33 UTC (permalink / raw
To: Patrick Voelker; +Cc: Ed Tanous, Bruce Mitchell, openbmc@lists.ozlabs.org
[-- Attachment #1: Type: text/plain, Size: 1873 bytes --]
On Wed, Sep 02, 2020 at 07:50:07PM +0000, Patrick Voelker wrote:
> My understanding of PACKAGECONFIG is that it's a way to provide build options for individual packages. In this case, what PACKAGECONFIG option would we be contributing?
Yes, PACKAGECONFIG is for build options for indivdiual packages.
Similar to FEATURES which affect multiple packages. In this case we'd
be defining a new package-feature to rsyslog "don't install this file".
Sounds like the `-f` proposed elsewhere in the stream is reasonable.
> Is there a way now for me to force bitbake to ignore (or not use) rsyslog*.bbappend in meta-tiogapass from another layer?
I don't think so. Layers have priority (order with which they are
applied) but I don't know of any way to mask a bbappend from one layer.
> The two appends that are conflicting are:
> meta-facebook/meta-tiogapass/recipes-extended/rsyslog/rsyslog_%.bbappend
> meta-intel/meta-common/recipes-extended/rsyslog/rsyslog_%.bbappend
>
> Can I choose one over the other instead of having them build upon eachother?
I strongly suggest you don't do this. meta-facebook and meta-intel
are two different machine metas. There are going to be conflicts
between them because we've done a poor job of making sure that all the
machine metas uses VARIABLE_machine append syntax everywhere (which
ensures the variable is only applied for a particular machine). That
isn't to say we shouldn't fix those things, but in the short-term you're
likely to run into a lot of issues.
To step back, what is it you're trying to accomplish by including both
machine metas into a single build? If there is content in meta-intel
that is not machine specific, the direction seems to be that we should
move this to meta-phosphor. (Brad started a discussion about this on
the ML about 2 weeks ago.)
--
Patrick Williams
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 22+ messages in thread
end of thread, other threads:[~2020-09-03 15:33 UTC | newest]
Thread overview: 22+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2020-08-30 22:02 When building OpenBMC . . . ? Bruce Mitchell
2020-08-31 10:57 ` Brad Bishop
2020-08-31 14:19 ` Bruce Mitchell
2020-08-31 14:29 ` Ed Tanous
2020-08-31 14:34 ` Bruce Mitchell
2020-08-31 22:48 ` Vijay Khemka
2020-08-31 22:51 ` Bruce Mitchell
2020-08-31 22:57 ` Vijay Khemka
2020-09-01 16:25 ` Bruce Mitchell
2020-09-01 12:24 ` Patrick Williams
2020-09-01 16:09 ` Ed Tanous
2020-09-01 16:20 ` Patrick Williams
2020-09-01 16:29 ` Bruce Mitchell
2020-09-01 16:56 ` Ed Tanous
2020-09-01 16:28 ` Bruce Mitchell
2020-09-02 18:50 ` Patrick Voelker
2020-09-02 19:10 ` Patrick Williams
2020-09-02 19:50 ` Patrick Voelker
2020-09-02 21:39 ` Bills, Jason M
2020-09-02 23:35 ` Patrick Voelker
2020-09-03 15:33 ` Patrick Williams
2020-09-01 16:26 ` Bruce Mitchell
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.