* Re: Security Working Group - Wednesday May 12 - results
2021-05-12 18:18 ` Security Working Group - Wednesday May 12 - results Joseph Reynolds
@ 2021-05-12 20:40 ` Patrick Williams
2021-05-14 18:26 ` Joseph Reynolds
2021-05-12 21:35 ` Michael Richardson
2021-05-13 0:25 ` Andrew Jeffery
2 siblings, 1 reply; 9+ messages in thread
From: Patrick Williams @ 2021-05-12 20:40 UTC (permalink / raw)
To: Joseph Reynolds; +Cc: openbmc
[-- Attachment #1: Type: text/plain, Size: 1546 bytes --]
On Wed, May 12, 2021 at 01:18:48PM -0500, Joseph Reynolds wrote:
> On 5/11/21 8:59 PM, Joseph Reynolds wrote:
[ Copying some stuff from the Google Doc. ]
> Security impacts:
> - Can be used to defeat secureboot.
> - Can this function be disabled? Via kernel config. Default?
> - Can restrict which images kexec can load?
> - Recommend? Validate the kernel signature before kexec’ing it. But that doesn't stop an attacker who uses wget to get a malicious image which they pass to kexec.
> - Why would an attacker want to use kexec? Opportunity to modify BMC code, load device drivers, create trojan horse(?) or back doors.
> - How can we force kexec to perform the same signature validation as uboot? (each part and the whole: kernel, device tree, file system, …)
Some of this is a concern only because the arm32 support for kexec is
far behind the other architectures. Can someone investigate what, if
anything, is going on upstream to get arm caught up?
On most other architectures there is a new set of system calls, enabled
by KEXEC_FILE, which moves some of the handling done by the kexec
executable into the kernel and has signature verification done there.
arm32 is one of the few architectures that do not currently support this
KConfig.
See:
- https://man7.org/linux/man-pages/man8/kexec.8.html (kexec-syscall-auto).
- http://people.redhat.com/vgoyal/papers-presentations/linux-plumbers-2013/vivek-linux-plumbers-conference-2013-presentation-kexec-secureboot.pdf
--
Patrick Williams
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Security Working Group - Wednesday May 12 - results
2021-05-12 20:40 ` Patrick Williams
@ 2021-05-14 18:26 ` Joseph Reynolds
0 siblings, 0 replies; 9+ messages in thread
From: Joseph Reynolds @ 2021-05-14 18:26 UTC (permalink / raw)
To: Patrick Williams; +Cc: openbmc
On 5/12/21 3:40 PM, Patrick Williams wrote:
> On Wed, May 12, 2021 at 01:18:48PM -0500, Joseph Reynolds wrote:
>> On 5/11/21 8:59 PM, Joseph Reynolds wrote:
> [ Copying some stuff from the Google Doc. ]
>
>> Security impacts:
>> - Can be used to defeat secureboot.
>> - Can this function be disabled? Via kernel config. Default?
>> - Can restrict which images kexec can load?
>> - Recommend? Validate the kernel signature before kexec’ing it. But that doesn't stop an attacker who uses wget to get a malicious image which they pass to kexec.
>> - Why would an attacker want to use kexec? Opportunity to modify BMC code, load device drivers, create trojan horse(?) or back doors.
>> - How can we force kexec to perform the same signature validation as uboot? (each part and the whole: kernel, device tree, file system, …)
> Some of this is a concern only because the arm32 support for kexec is
> far behind the other architectures. Can someone investigate what, if
> anything, is going on upstream to get arm caught up?
>
> On most other architectures there is a new set of system calls, enabled
> by KEXEC_FILE, which moves some of the handling done by the kexec
> executable into the kernel and has signature verification done there.
> arm32 is one of the few architectures that do not currently support this
> KConfig.
>
> See:
> - https://man7.org/linux/man-pages/man8/kexec.8.html (kexec-syscall-auto).
> - http://people.redhat.com/vgoyal/papers-presentations/linux-plumbers-2013/vivek-linux-plumbers-conference-2013-presentation-kexec-secureboot.pdf
>
Patrick,
Thanks for the info! (I'm learning as I go.) I'll ask my people to
look into this (with no promises).
- Joseph
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Security Working Group - Wednesday May 12 - results
2021-05-12 18:18 ` Security Working Group - Wednesday May 12 - results Joseph Reynolds
2021-05-12 20:40 ` Patrick Williams
@ 2021-05-12 21:35 ` Michael Richardson
2021-05-14 18:50 ` Joseph Reynolds
2021-05-13 0:25 ` Andrew Jeffery
2 siblings, 1 reply; 9+ messages in thread
From: Michael Richardson @ 2021-05-12 21:35 UTC (permalink / raw)
To: Joseph Reynolds, openbmc
[-- Attachment #1: Type: text/plain, Size: 2204 bytes --]
Joseph Reynolds <jrey@linux.ibm.com> wrote:
> 1. Security impacts of enabling kexec (load and optionally execute new
> kernel) in the BMC's production kernel. How does this work and play
> with secure boot and with IMA?
> 2. What are the security impacts of having the proc file system file
> /proc/sysrq-triggerwhich can cause kernel panics which can cause the
> BMC to terminate processing?
> 3. In general, how can you (an operator or the BMC's host system)
> recover a BMC which has become unresponsive, for example, because its
> kernel processing has failed. A design introduces using
> /proc/sysrq-triggertogether with a recovery kernel installed by kexec.
This tussle between locking down the system against all intrusions, vs being
able to fix stuff when in trouble is a serious debate.
(Based upon how easily random alien technology takes over the Enterprise, we
know which way Starfleet engineers went.)
So I suggest that in most cases, the secure boot process should disable
kexec (and sysrq-trigger!), but that this should be an tunable attribute
under control of the secure boot process.
For the majority of data center, business and home users of systems, the risk
of malware in the bootpath of the BMC exceeds the risk of BMC failures, and
the cost remediation (taking a machine out of commission when there is a BMC problem).
Having said that, there is a Right-to-Repair concern, and I really hope that
manufacturers will provide for a hardware jumper, and for installation of new
trust anchors.
But, there is a variety of ways to do that from kernel cmdlines, to being able to
boot alternate kernels, and perhaps this could be punted down the road for
the operator that needs (#3). Perhaps, coming back to my (humour) above, it
will in fact be Mars Rover missions or Starlink satellites that need it, and
probably, they can afford to do that work.
--
] Never tell me the odds! | ipv6 mesh networks [
] Michael Richardson, Sandelman Software Works | IoT architect [
] mcr@sandelman.ca http://www.sandelman.ca/ | ruby on rails [
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: Security Working Group - Wednesday May 12 - results
2021-05-12 21:35 ` Michael Richardson
@ 2021-05-14 18:50 ` Joseph Reynolds
0 siblings, 0 replies; 9+ messages in thread
From: Joseph Reynolds @ 2021-05-14 18:50 UTC (permalink / raw)
To: Michael Richardson, openbmc
On 5/12/21 4:35 PM, Michael Richardson wrote:
> Joseph Reynolds <jrey@linux.ibm.com> wrote:
> > 1. Security impacts of enabling kexec (load and optionally execute new
> > kernel) in the BMC's production kernel. How does this work and play
> > with secure boot and with IMA?
>
> > 2. What are the security impacts of having the proc file system file
> > /proc/sysrq-triggerwhich can cause kernel panics which can cause the
> > BMC to terminate processing?
>
> > 3. In general, how can you (an operator or the BMC's host system)
> > recover a BMC which has become unresponsive, for example, because its
> > kernel processing has failed. A design introduces using
> > /proc/sysrq-triggertogether with a recovery kernel installed by kexec.
>
> This tussle between locking down the system against all intrusions, vs being
> able to fix stuff when in trouble is a serious debate.
>
> (Based upon how easily random alien technology takes over the Enterprise, we
> know which way Starfleet engineers went.)
>
> So I suggest that in most cases, the secure boot process should disable
> kexec (and sysrq-trigger!), but that this should be an tunable attribute
> under control of the secure boot process.
>
> For the majority of data center, business and home users of systems, the risk
> of malware in the bootpath of the BMC exceeds the risk of BMC failures, and
> the cost remediation (taking a machine out of commission when there is a BMC problem).
> Having said that, there is a Right-to-Repair concern, and I really hope that
> manufacturers will provide for a hardware jumper, and for installation of new
> trust anchors.
>
> But, there is a variety of ways to do that from kernel cmdlines, to being able to
> boot alternate kernels, and perhaps this could be punted down the road for
> the operator that needs (#3). Perhaps, coming back to my (humour) above, it
> will in fact be Mars Rover missions or Starlink satellites that need it, and
> probably, they can afford to do that work.
Michael,
Thanks for the NASA, Elon Musk, and Star Trek references. (I loved the
Daleks in Star Wars!)
I believe kexec and sysrq-trigger should remain disabled in the OpenBMC
project defaults.
And the IBM design cited attempts to balance security and usability.
Although I understand there is work in the OCP security project and
other places to recover a trust anchor, I don't see anything practical
for OpenBMC to use.
- Joseph
IBM design: https://gerrit.openbmc-project.xyz/c/openbmc/docs/+/42948
>
> --
> ] Never tell me the odds! | ipv6 mesh networks [
> ] Michael Richardson, Sandelman Software Works | IoT architect [
> ] mcr@sandelman.ca http://www.sandelman.ca/ | ruby on rails [
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Security Working Group - Wednesday May 12 - results
2021-05-12 18:18 ` Security Working Group - Wednesday May 12 - results Joseph Reynolds
2021-05-12 20:40 ` Patrick Williams
2021-05-12 21:35 ` Michael Richardson
@ 2021-05-13 0:25 ` Andrew Jeffery
2021-05-14 19:02 ` Joseph Reynolds
2 siblings, 1 reply; 9+ messages in thread
From: Andrew Jeffery @ 2021-05-13 0:25 UTC (permalink / raw)
To: Joseph Reynolds, openbmc
Hi Joseph,
It tends to be useful to Cc the developers doing the work. Posting to
the list without Cc'ing relevant people leaves discovery of your
discussion to chance, where as adding them on To: or Cc: does two
things:
1. Raises the chance that they'll pay attention to your discussion
2. Removes the impression that you're intentionally talking past them
Please try to engage the relevant people directly in the discussion by
adding them in To: or Cc.
On Thu, 13 May 2021, at 03:48, Joseph Reynolds wrote:
> On 5/11/21 8:59 PM, Joseph Reynolds wrote:
> > This is a reminder of the OpenBMC Security Working Group meeting
> > scheduled for this Wednesday May 12 at 10:00am PDT.
> >
> > We'll discuss the following items on the agenda
> > <https://docs.google.com/document/d/1b7x9BaxsfcukQDqbvZsU2ehMq4xoJRQvLxxsDUWmAOI/edit>,
> > and anything else that comes up:
> >
>
> Three items were discussed. You might want to start with item 3 first
> to introduce the first two. Summary:
>
> 1. Security impacts of enabling kexec (load and optionally execute new
> kernel) in the BMC's production kernel. How does this work and play
> with secure boot and with IMA?
Have you engaged with OpenBMC's kernel developers? They might be are
interested in this problem. I'm vaguely aware of some work-in-progress
patches that allows kexec to load FIT images, which can be signed and
validated. This would mitigate execution of arbitrary kernels and also
helps avoid the problem of shipping multiple kernel binaries or
extracting artefacts from a FIT to pass to kexec.
>
> 2. What are the security impacts of having the proc file system file
> /proc/sysrq-triggerwhich can cause kernel panics which can cause the BMC
> to terminate processing?
>
> 3. In general, how can you (an operator or the BMC's host system)
> recover a BMC which has become unresponsive, for example, because its
> kernel processing has failed. A design introduces using
> /proc/sysrq-triggertogether with a recovery kernel installed by kexec.
To be clear, the use of /proc/sysrq-trigger is a temporary hack to
reboot the BMC in the absence of kexec/kdump. Once those features are
merged the application implementing this behaviour can invoke kexec
directly. The slight advantage of /proc/sysrq-trigger is that with or
without kexec/kdump enabled the BMC will reboot, and if kexec/kdump are
enabled then it will automatically take advantage of them.
In the specific case p10bmc platforms the host has access to a GPIO
tied to the BMC's EXTRST line, so with or without this software feature
the host can mount denial of service attacks of arbitrary length. This
hardware design places the BMC and host firmware in the same trust
domain.
Andrew
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Security Working Group - Wednesday May 12 - results
2021-05-13 0:25 ` Andrew Jeffery
@ 2021-05-14 19:02 ` Joseph Reynolds
2021-05-16 23:15 ` Andrew Jeffery
0 siblings, 1 reply; 9+ messages in thread
From: Joseph Reynolds @ 2021-05-14 19:02 UTC (permalink / raw)
To: Andrew Jeffery, openbmc
On 5/12/21 7:25 PM, Andrew Jeffery wrote:
> Hi Joseph,
>
> It tends to be useful to Cc the developers doing the work. Posting to
> the list without Cc'ing relevant people leaves discovery of your
> discussion to chance, where as adding them on To: or Cc: does two
> things:
>
> 1. Raises the chance that they'll pay attention to your discussion
> 2. Removes the impression that you're intentionally talking past them
>
> Please try to engage the relevant people directly in the discussion by
> adding them in To: or Cc.
Andrew,
Good advice, thanks! This was not my topic. I was simply recording the
conversation and did not have a chance to follow up. I am glad it got
your attention. In general, it is hard to know who to contact. Note
that I am following up on this item privately through other channels.
Finally, during the meeting, I encouraged attendees to make comments in
the relevant gerrit review process.
- Joseph
>
> On Thu, 13 May 2021, at 03:48, Joseph Reynolds wrote:
>> On 5/11/21 8:59 PM, Joseph Reynolds wrote:
>>> This is a reminder of the OpenBMC Security Working Group meeting
>>> scheduled for this Wednesday May 12 at 10:00am PDT.
>>>
>>> We'll discuss the following items on the agenda
>>> <https://docs.google.com/document/d/1b7x9BaxsfcukQDqbvZsU2ehMq4xoJRQvLxxsDUWmAOI/edit>,
>>> and anything else that comes up:
>>>
>> Three items were discussed. You might want to start with item 3 first
>> to introduce the first two. Summary:
>>
>> 1. Security impacts of enabling kexec (load and optionally execute new
>> kernel) in the BMC's production kernel. How does this work and play
>> with secure boot and with IMA?
> Have you engaged with OpenBMC's kernel developers? They might be are
> interested in this problem. I'm vaguely aware of some work-in-progress
> patches that allows kexec to load FIT images, which can be signed and
> validated. This would mitigate execution of arbitrary kernels and also
> helps avoid the problem of shipping multiple kernel binaries or
> extracting artefacts from a FIT to pass to kexec.
>
>> 2. What are the security impacts of having the proc file system file
>> /proc/sysrq-triggerwhich can cause kernel panics which can cause the BMC
>> to terminate processing?
>>
>> 3. In general, how can you (an operator or the BMC's host system)
>> recover a BMC which has become unresponsive, for example, because its
>> kernel processing has failed. A design introduces using
>> /proc/sysrq-triggertogether with a recovery kernel installed by kexec.
> To be clear, the use of /proc/sysrq-trigger is a temporary hack to
> reboot the BMC in the absence of kexec/kdump. Once those features are
> merged the application implementing this behaviour can invoke kexec
> directly. The slight advantage of /proc/sysrq-trigger is that with or
> without kexec/kdump enabled the BMC will reboot, and if kexec/kdump are
> enabled then it will automatically take advantage of them.
>
> In the specific case p10bmc platforms the host has access to a GPIO
> tied to the BMC's EXTRST line, so with or without this software feature
> the host can mount denial of service attacks of arbitrary length. This
> hardware design places the BMC and host firmware in the same trust
> domain.
>
> Andrew
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Security Working Group - Wednesday May 12 - results
2021-05-14 19:02 ` Joseph Reynolds
@ 2021-05-16 23:15 ` Andrew Jeffery
0 siblings, 0 replies; 9+ messages in thread
From: Andrew Jeffery @ 2021-05-16 23:15 UTC (permalink / raw)
To: Joseph Reynolds, openbmc
On Sat, 15 May 2021, at 04:32, Joseph Reynolds wrote:
> In general, it is hard to know who to contact.
I think it deserves some effort, no? Talking in abstractions doesn't
help as we're not discussing the abstract but specific patches, some of
which you've left a comment against.
Equivalently, saying "In general, it is hard to build secure systems"
and then not putting in any further effort as a consequence isn't
acceptable - we need to do the work; narrow the statement from the
abstract to the specific do our best to mitigate risks. That same
strategy of narrowing the abstract to the specific applies here.
Given you've already commented on one of the patches I don't think it's
a big leap to look at who the author is and include them on related
discussions in other mediums.
So anyway, I think this open source process works best if we recognise
that resolving issues requires bringing people together, and not
treating the work as some kind of abstract process. I feel like
broadcasting (1-to-many) the minutes here without including the people
impacted by the discussion creates a separation. Let's put the effort
in to bring the right people into discussions from the outset.
> Note
> that I am following up on this item privately through other channels.
Okay, hopefully I'm included on those discussions too.
> Finally, during the meeting, I encouraged attendees to make comments in
> the relevant gerrit review process.
Great! I hope we can capture the concrete concerns in the patch
comments and work to resolve them.
Andrew
^ permalink raw reply [flat|nested] 9+ messages in thread