* /var/log as persistent
@ 2024-05-28 17:31 Patrick Williams
2024-05-29 3:49 ` [External] " Lei Yu
0 siblings, 1 reply; 4+ messages in thread
From: Patrick Williams @ 2024-05-28 17:31 UTC (permalink / raw
To: OpenBMC List
[-- Attachment #1: Type: text/plain, Size: 1213 bytes --]
Hello,
It was pointed out that I did not do a good job of broadcasting a change
I made back in March, so I am sending this out for awareness now.
https://gerrit.openbmc.org/c/openbmc/openbmc/+/69959
A default setting from bitbake (to VOLATILE_LOG_DIR) was causing some
platforms to not be able to persist `/var/log` and instead it was
mounted as a temporary directory. This meant that even if you
explicitly configured journald to use `/var/log` (instead of the
`/run/log` it uses by default) you would not get persistent journalling.
It also meant that applications like `obmc-console` log files were not
persistent and would be lost in a BMC reboot.
I had asked a few machine owners and most of them either had it set to
explicitly unset `VOLATILE_LOG_DIR` in their meta-layer or through some
downstream changes had overwritten it. So, I made this the default.
I thought this only affected:
- machines that explicitly set `Storage=persistent` in the journald
config.
- everyone's obmc-console logs.
Based on the report from a downstream user, it seems like there might be
more effects? I'm not sure at this point, but advertising it wider.
--
Patrick Williams
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [External] /var/log as persistent
2024-05-28 17:31 /var/log as persistent Patrick Williams
@ 2024-05-29 3:49 ` Lei Yu
2024-05-29 12:29 ` Patrick Williams
0 siblings, 1 reply; 4+ messages in thread
From: Lei Yu @ 2024-05-29 3:49 UTC (permalink / raw
To: Patrick Williams; +Cc: OpenBMC List
On Wed, May 29, 2024 at 1:41 AM Patrick Williams <patrick@stwcx.xyz> wrote:
>
> Hello,
>
> It was pointed out that I did not do a good job of broadcasting a change
> I made back in March, so I am sending this out for awareness now.
>
> https://gerrit.openbmc.org/c/openbmc/openbmc/+/69959
>
> A default setting from bitbake (to VOLATILE_LOG_DIR) was causing some
> platforms to not be able to persist `/var/log` and instead it was
> mounted as a temporary directory. This meant that even if you
> explicitly configured journald to use `/var/log` (instead of the
> `/run/log` it uses by default) you would not get persistent journalling.
> It also meant that applications like `obmc-console` log files were not
> persistent and would be lost in a BMC reboot.
>
1. VOLATILE_LOG_DIR is defaulted to `yes` in poky/meta/conf/bitbake.conf
2. With static layout, the above config makes the `/var/log` a
volatile dir linked to `/var/volatile/log`, where `/var/volatile` is a
tmpfs.
Be noted that it's the default for OpenBMC machines with static flash layouts.
So the journal log and obmc-console were volatile **by default**.
Users should not expect the above logs to be persistent, and if they
do, they could config `VOLATILE_LOG_DIR` to `no`, which is done in
`mtjade` and `mtmitchell` layer.
The change `https://gerrit.openbmc.org/c/openbmc/openbmc/+/69959`
makes `VOLATILE_LOG_DIR` to `yes` by default in `meta-phosphor` layer,
which effectively affects all OpenBMC builds.
> I had asked a few machine owners and most of them either had it set to
> explicitly unset `VOLATILE_LOG_DIR` in their meta-layer or through some
> downstream changes had overwritten it. So, I made this the default.
>
> I thought this only affected:
> - machines that explicitly set `Storage=persistent` in the journald
> config.
> - everyone's obmc-console logs.
>
> Based on the report from a downstream user, it seems like there might be
> more effects? I'm not sure at this point, but advertising it wider.
As above information, OpenBMC users were expecting "volatile" logs
before, and we should keep it default.
So I would suggest we revert the change to keep the consistency about
the default volatile log dir.
And for the reasons why I prefer the volatile log directory:
* In most OpenBMC machine builds we see 32/64/128 SPI flashes are
used, so the `rwfs` is limited and the frequent "writes" to the SPI
flash costs the lifespan.
* To collect the logs, we could either use rsyslog or the host-logger
(which were "already there") to send the logs to remote servers. So
there is no strong requirement to make the logs persistent.
--
BRs,
Lei YU
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [External] /var/log as persistent
2024-05-29 3:49 ` [External] " Lei Yu
@ 2024-05-29 12:29 ` Patrick Williams
2024-05-30 6:44 ` Lei Yu
0 siblings, 1 reply; 4+ messages in thread
From: Patrick Williams @ 2024-05-29 12:29 UTC (permalink / raw
To: Lei Yu; +Cc: OpenBMC List
[-- Attachment #1: Type: text/plain, Size: 5244 bytes --]
On Wed, May 29, 2024 at 11:49:15AM +0800, Lei Yu wrote:
> On Wed, May 29, 2024 at 1:41 AM Patrick Williams <patrick@stwcx.xyz> wrote:
> >
> > Hello,
> >
> > It was pointed out that I did not do a good job of broadcasting a change
> > I made back in March, so I am sending this out for awareness now.
> >
> > https://gerrit.openbmc.org/c/openbmc/openbmc/+/69959
> >
> > A default setting from bitbake (to VOLATILE_LOG_DIR) was causing some
> > platforms to not be able to persist `/var/log` and instead it was
> > mounted as a temporary directory. This meant that even if you
> > explicitly configured journald to use `/var/log` (instead of the
> > `/run/log` it uses by default) you would not get persistent journalling.
> > It also meant that applications like `obmc-console` log files were not
> > persistent and would be lost in a BMC reboot.
> >
>
> 1. VOLATILE_LOG_DIR is defaulted to `yes` in poky/meta/conf/bitbake.conf
> 2. With static layout, the above config makes the `/var/log` a
> volatile dir linked to `/var/volatile/log`, where `/var/volatile` is a
> tmpfs.
>
> Be noted that it's the default for OpenBMC machines with static flash layouts.
> So the journal log and obmc-console were volatile **by default**.
I was still surprised that the journal was being persisted for you with
this change. I'm looking now in more detail at the journald.conf because
there are only a few platforms that explicitly set `Storage=persistent`.
The primary reason I made this change was because setting the journal
config had **no effect**, which is, I think very unintuitive.
It seems that the default journal config is `Storage=auto` which means
that it will be volatile only if `/var/log/journal` exists. This might
be why you are now seeing it persist.
> Users should not expect the above logs to be persistent, and if they
> do, they could config `VOLATILE_LOG_DIR` to `no`, which is done in
> `mtjade` and `mtmitchell` layer.
Conversely we could have machines that want it to be non-persistent to
set VOLATILE_LOG_DIR, right? It isn't obvious why one default is
"better" than another.
My impression is that VOLATILE_LOG_DIR is default partially because
syslog is also the default. As I said, with VOLATILE_LOG_DIR,
`Storage=persistent` has no affect.
If the only discussion here is really about the systemd-journal, we can
add a PACKAGECONFIG that chooses between `Storage=volatile` and
`Storage=persistent`. It seems we already have 3 different meta-layers
with a custom journald-config to trigger this, so we might as well
consolidate those.
> The change `https://gerrit.openbmc.org/c/openbmc/openbmc/+/69959`
> makes `VOLATILE_LOG_DIR` to `yes` by default in `meta-phosphor` layer,
> which effectively affects all OpenBMC builds.
>
> > I had asked a few machine owners and most of them either had it set to
> > explicitly unset `VOLATILE_LOG_DIR` in their meta-layer or through some
> > downstream changes had overwritten it. So, I made this the default.
> >
> > I thought this only affected:
> > - machines that explicitly set `Storage=persistent` in the journald
> > config.
> > - everyone's obmc-console logs.
> >
> > Based on the report from a downstream user, it seems like there might be
> > more effects? I'm not sure at this point, but advertising it wider.
>
> As above information, OpenBMC users were expecting "volatile" logs
> before, and we should keep it default.
> So I would suggest we revert the change to keep the consistency about
> the default volatile log dir.
Were they "expecting"? This is where I did ask a few of the machine
owners who have upstream systems in production. All of them told me
they overwrite this already.
I highly suspect that any commercial system is going to want
persistence. I can understand what you have referred to with rsyslog
for cloud systems. I think this is one of those cases where whatever
the default is, it won't satisfy everyone. (We would be considered a
"cloud" style system but don't want to rely exclusively on rsyslog
because you then don't get visibility to debug network issues.)
> And for the reasons why I prefer the volatile log directory:
> * In most OpenBMC machine builds we see 32/64/128 SPI flashes are
> used, so the `rwfs` is limited and the frequent "writes" to the SPI
> flash costs the lifespan.
Are you seeing lifetime issues with SPI-NOR? Aren't they rated for at
least 1 million erase cycles? Even with the static layout, where all
writes are going to the same location, that should give you 32 years of
use at 1 erase per second (assuming only 4MiB of rwfs space).
> * To collect the logs, we could either use rsyslog or the host-logger
> (which were "already there") to send the logs to remote servers. So
> there is no strong requirement to make the logs persistent.
Are you really wanting obmc-console to *not* persist? Do you think most
people want this? This is a bit surprising to me because obmc-console
already has configuration limiting the size of the log files and it has
a rotate functionality; the amount of space (and erase cycles) used by it
is probably low.
--
Patrick Williams
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: /var/log as persistent
2024-05-29 12:29 ` Patrick Williams
@ 2024-05-30 6:44 ` Lei Yu
0 siblings, 0 replies; 4+ messages in thread
From: Lei Yu @ 2024-05-30 6:44 UTC (permalink / raw
To: Patrick Williams, OpenBMC List
On Wed, May 29, 2024 at 8:29 PM Patrick Williams <patrick@stwcx.xyz> wrote:
>
> On Wed, May 29, 2024 at 11:49:15AM +0800, Lei Yu wrote:
> > On Wed, May 29, 2024 at 1:41 AM Patrick Williams <patrick@stwcx.xyz> wrote:
> > >
> > > Hello,
> > >
> > > It was pointed out that I did not do a good job of broadcasting a change
> > > I made back in March, so I am sending this out for awareness now.
> > >
> > > https://gerrit.openbmc.org/c/openbmc/openbmc/+/69959
> > >
> > > A default setting from bitbake (to VOLATILE_LOG_DIR) was causing some
> > > platforms to not be able to persist `/var/log` and instead it was
> > > mounted as a temporary directory. This meant that even if you
> > > explicitly configured journald to use `/var/log` (instead of the
> > > `/run/log` it uses by default) you would not get persistent journalling.
> > > It also meant that applications like `obmc-console` log files were not
> > > persistent and would be lost in a BMC reboot.
> > >
> >
> > 1. VOLATILE_LOG_DIR is defaulted to `yes` in poky/meta/conf/bitbake.conf
> > 2. With static layout, the above config makes the `/var/log` a
> > volatile dir linked to `/var/volatile/log`, where `/var/volatile` is a
> > tmpfs.
> >
> > Be noted that it's the default for OpenBMC machines with static flash layouts.
> > So the journal log and obmc-console were volatile **by default**.
>
> I was still surprised that the journal was being persisted for you with
> this change. I'm looking now in more detail at the journald.conf because
> there are only a few platforms that explicitly set `Storage=persistent`.
>
> The primary reason I made this change was because setting the journal
> config had **no effect**, which is, I think very unintuitive.
Agreed that it's "unintuitive" for `Storage=persistent` had **no effect**.
But as I mentioned before, it's the "default" behavior in OpenBMC
builds with static layout.
>
> It seems that the default journal config is `Storage=auto` which means
> that it will be volatile only if `/var/log/journal` exists. This might
> be why you are now seeing it persist.
>
> > Users should not expect the above logs to be persistent, and if they
> > do, they could config `VOLATILE_LOG_DIR` to `no`, which is done in
> > `mtjade` and `mtmitchell` layer.
>
> Conversely we could have machines that want it to be non-persistent to
> set VOLATILE_LOG_DIR, right? It isn't obvious why one default is
> "better" than another.
>
> My impression is that VOLATILE_LOG_DIR is default partially because
> syslog is also the default. As I said, with VOLATILE_LOG_DIR,
> `Storage=persistent` has no affect.
>
> If the only discussion here is really about the systemd-journal, we can
> add a PACKAGECONFIG that chooses between `Storage=volatile` and
> `Storage=persistent`. It seems we already have 3 different meta-layers
> with a custom journald-config to trigger this, so we might as well
> consolidate those.
>
> > The change `https://gerrit.openbmc.org/c/openbmc/openbmc/+/69959`
> > makes `VOLATILE_LOG_DIR` to `yes` by default in `meta-phosphor` layer,
> > which effectively affects all OpenBMC builds.
> >
> > > I had asked a few machine owners and most of them either had it set to
> > > explicitly unset `VOLATILE_LOG_DIR` in their meta-layer or through some
> > > downstream changes had overwritten it. So, I made this the default.
> > >
> > > I thought this only affected:
> > > - machines that explicitly set `Storage=persistent` in the journald
> > > config.
> > > - everyone's obmc-console logs.
> > >
> > > Based on the report from a downstream user, it seems like there might be
> > > more effects? I'm not sure at this point, but advertising it wider.
> >
> > As above information, OpenBMC users were expecting "volatile" logs
> > before, and we should keep it default.
> > So I would suggest we revert the change to keep the consistency about
> > the default volatile log dir.
>
> Were they "expecting"? This is where I did ask a few of the machine
> owners who have upstream systems in production. All of them told me
> they overwrite this already.
What do you mean by "overwrite" here? In upstream systems, we only see
`VOLATILE_LOG_DIR = "no"` in `meta-ampere` layer. (Check out to
`828c130f59^` and `git grep VOLATILE_LOG_DIR` shows the result.)
All other layers were using the default `VOLATILE_LOG_DIR = "yes"`
>
> I highly suspect that any commercial system is going to want
> persistence. I can understand what you have referred to with rsyslog
> for cloud systems. I think this is one of those cases where whatever
> the default is, it won't satisfy everyone. (We would be considered a
> "cloud" style system but don't want to rely exclusively on rsyslog
> because you then don't get visibility to debug network issues.)
>
> > And for the reasons why I prefer the volatile log directory:
> > * In most OpenBMC machine builds we see 32/64/128 SPI flashes are
> > used, so the `rwfs` is limited and the frequent "writes" to the SPI
> > flash costs the lifespan.
>
> Are you seeing lifetime issues with SPI-NOR? Aren't they rated for at
> least 1 million erase cycles? Even with the static layout, where all
> writes are going to the same location, that should give you 32 years of
> use at 1 erase per second (assuming only 4MiB of rwfs space).
In case you have many logs in obmc-console, there could be much more
erases per second.
>
> > * To collect the logs, we could either use rsyslog or the host-logger
> > (which were "already there") to send the logs to remote servers. So
> > there is no strong requirement to make the logs persistent.
>
> Are you really wanting obmc-console to *not* persist? Do you think most
> people want this? This is a bit surprising to me because obmc-console
> already has configuration limiting the size of the log files and it has
> a rotate functionality; the amount of space (and erase cycles) used by it
> is probably low.
I can not speak for others, but what is true is that before the commit
`828c130f59` all OpenBMC upstream builds (except meta-ampere) were
using `VOLATILE_LOG_DIR = "yes"`.
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2024-05-30 6:45 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-05-28 17:31 /var/log as persistent Patrick Williams
2024-05-29 3:49 ` [External] " Lei Yu
2024-05-29 12:29 ` Patrick Williams
2024-05-30 6:44 ` Lei Yu
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).