* Re: Interested in ceph OSD encryption and key management
2015-05-28 20:03 ` Sage Weil
@ 2015-05-31 14:01 ` Wyllys Ingersoll
2015-06-03 0:07 ` Andrew Bartlett
2015-06-03 0:12 ` Andrew Bartlett
2015-06-17 2:37 ` Andrew Bartlett
2 siblings, 1 reply; 10+ messages in thread
From: Wyllys Ingersoll @ 2015-05-31 14:01 UTC (permalink / raw)
To: Sage Weil; +Cc: Andrew Bartlett, mbroz, ceph-devel
Also consider key management services like Barbican or using an external
Key Mgmt server for managing the master keys (wrapping keys,
key-encrypting-keys, etc etc).
One can store the wrapping key externally, and fetch it via KMIP or other
standard/secure protocol when the disks come online. The keys in
/etc/ceph/keys can be wrapped on disk and then unwrapped with the external
key to a ram disk when the disks come online. That way the raw dm-crypt
keys don't persist on the disk and are only available as long as the drive
is active and the system is up.
Just one possibility. Switching to LUKS or something more robust is
probably better long term solution.
Orthogonal to FDE, but possibly nice to have would be the ability to
encrypt individual pools using different keys (or none at all), but thats
probably a different conversation...
On Thu, May 28, 2015 at 4:03 PM, Sage Weil <sage@newdream.net> wrote:
> Hi Andrew,
>
> I'm copying Milan Broz, who has looked at this ome. There was some
> subsequent off-list discussion in Red Hat about using Petera[1] for the
> key management, but this'll require a bit more effort than what was
> described in that blueprint.
>
> On Thu, 28 May 2015, Andrew Bartlett wrote:
>> David Disseldorp was good enough to point me at this proposal for ceph
>> OSD key management:
>> https://wiki.ceph.com/Planning/Blueprints/Infernalis/osd%3A_simple_ceph-mon_dm-crypt_key_management
>>
>> I'm really interested in improving ceph on-disk encryption, and am
>> really glad folks are taking this beyond the local key storage we have
>> managed so far.
>>
>> So I can be part of the discussion, how do I get a login to the wiki? I
>> would like to indicate my interest there.
>
> The wiki logins are broken, but ignore that.. we're moving to
> tracker.ceph.com's wiki shortly anyway. Email is best in the meantime!
>
>> Regarding the proposal:
>>
>> In the default mode suggested in the wiki, my primary concern is that
>> I'm told, in a number of deployments, the monitor node is the same
>> server that also holds the OSDs, so we don't gain anything for those
>> cases over the /etc storage.
>>
>> In those cases, the hooks suggested in the wiki will be key, as will be
>> having those configurable in ceph.conf, so ceph-deploy can just pass it
>> down to all the nodes as they are built, just as the other dmcrypt
>> options are.
>>
>> I would like to see three things hookable:
>> - the command to obtain the key (on stdout)
>
> - a command to publish/store a new key (from ceph-disk create)
>
>> - to encrypt the key (so we can additionally pass it
>> via gpg, a HSM or remote encrypt/decrypt service)
>> - to decrypt the key
>
> Not quite sure what you have in mind here?
>
> I think the key things I'd like to see in any effort here is
>
> 1- keep the actual decryption key in LUKS and store the LUKS key in the
> key escrow location. The original dm-crypt stuff kept the actual key
> in /etc/ceph/keys, but that makes it hard to track key provenance, and
> also fixes the encryption algorithm... LUKS is more robust.
>
> 2- make the key management pluggable, so that we can use petera,
> /etc/ceph/keys, the ceph mons, or something else.
>
> Thanks!
> sage
>
>
> [1] https://github.com/npmccallum/petera
>
>>
>>
>> Thanks,
>>
>> Andrew Bartlett
>>
>>
>> --
>> Andrew Bartlett
>> http://samba.org/~abartlet/
>> Authentication Developer, Samba Team http://samba.org
>> Samba Developer, Catalyst IT http://catalyst.net.nz/services/samba
>>
>>
>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Interested in ceph OSD encryption and key management
2015-05-31 14:01 ` Wyllys Ingersoll
@ 2015-06-03 0:07 ` Andrew Bartlett
0 siblings, 0 replies; 10+ messages in thread
From: Andrew Bartlett @ 2015-06-03 0:07 UTC (permalink / raw)
To: Wyllys Ingersoll; +Cc: Sage Weil, mbroz, ceph-devel
On Sun, 2015-05-31 at 10:01 -0400, Wyllys Ingersoll wrote:
> Also consider key management services like Barbican or using an external
> Key Mgmt server for managing the master keys (wrapping keys,
> key-encrypting-keys, etc etc).
>
> One can store the wrapping key externally, and fetch it via KMIP or other
> standard/secure protocol when the disks come online. The keys in
> /etc/ceph/keys can be wrapped on disk and then unwrapped with the external
> key to a ram disk when the disks come online. That way the raw dm-crypt
> keys don't persist on the disk and are only available as long as the drive
> is active and the system is up.
>
> Just one possibility. Switching to LUKS or something more robust is
> probably better long term solution.
Indeed, and that is why we have done that :-)
The proposal as written is confusing, because what it calls a 'wrapper
key' is really a key access key, it doesn't even encrypt it.
Andrew Bartlett
--
Andrew Bartlett
http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org
Samba Developer, Catalyst IT http://catalyst.net.nz/services/samba
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Interested in ceph OSD encryption and key management
2015-05-28 20:03 ` Sage Weil
2015-05-31 14:01 ` Wyllys Ingersoll
@ 2015-06-03 0:12 ` Andrew Bartlett
2015-06-03 5:39 ` Milan Broz
2015-06-17 2:37 ` Andrew Bartlett
2 siblings, 1 reply; 10+ messages in thread
From: Andrew Bartlett @ 2015-06-03 0:12 UTC (permalink / raw)
To: Sage Weil; +Cc: mbroz, ceph-devel
On Thu, 2015-05-28 at 13:03 -0700, Sage Weil wrote:
> Hi Andrew,
>
> I'm copying Milan Broz, who has looked at this ome. There was some
> subsequent off-list discussion in Red Hat about using Petera[1] for the
> key management, but this'll require a bit more effort than what was
> described in that blueprint.
>
> On Thu, 28 May 2015, Andrew Bartlett wrote:
> > David Disseldorp was good enough to point me at this proposal for ceph
> > OSD key management:
> > https://wiki.ceph.com/Planning/Blueprints/Infernalis/osd%3A_simple_ceph-mon_dm-crypt_key_management
> >
> > I'm really interested in improving ceph on-disk encryption, and am
> > really glad folks are taking this beyond the local key storage we have
> > managed so far.
> >
> > So I can be part of the discussion, how do I get a login to the wiki? I
> > would like to indicate my interest there.
>
> The wiki logins are broken, but ignore that.. we're moving to
> tracker.ceph.com's wiki shortly anyway. Email is best in the meantime!
>
> > Regarding the proposal:
> >
> > In the default mode suggested in the wiki, my primary concern is that
> > I'm told, in a number of deployments, the monitor node is the same
> > server that also holds the OSDs, so we don't gain anything for those
> > cases over the /etc storage.
> >
> > In those cases, the hooks suggested in the wiki will be key, as will be
> > having those configurable in ceph.conf, so ceph-deploy can just pass it
> > down to all the nodes as they are built, just as the other dmcrypt
> > options are.
> >
> > I would like to see three things hookable:
> > - the command to obtain the key (on stdout)
>
> - a command to publish/store a new key (from ceph-disk create)
>
> > - to encrypt the key (so we can additionally pass it
> > via gpg, a HSM or remote encrypt/decrypt service)
> > - to decrypt the key
>
> Not quite sure what you have in mind here?
>
> I think the key things I'd like to see in any effort here is
>
> 1- keep the actual decryption key in LUKS and store the LUKS key in the
> key escrow location. The original dm-crypt stuff kept the actual key
> in /etc/ceph/keys, but that makes it hard to track key provenance, and
> also fixes the encryption algorithm... LUKS is more robust.
Yes.
> 2- make the key management pluggable, so that we can use petera,
> /etc/ceph/keys, the ceph mons, or something else.
Petera looks very much like what I was trying to home-grow, and I would
prefer that over using the monitor nodes, as those are often shared with
our storage nodes.
Thanks for the link, I'm glad we are all thinking in the same direction!
Andrew Bartlett
--
Andrew Bartlett
http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org
Samba Developer, Catalyst IT http://catalyst.net.nz/services/samba
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Interested in ceph OSD encryption and key management
2015-06-03 0:12 ` Andrew Bartlett
@ 2015-06-03 5:39 ` Milan Broz
0 siblings, 0 replies; 10+ messages in thread
From: Milan Broz @ 2015-06-03 5:39 UTC (permalink / raw)
To: Andrew Bartlett, Sage Weil; +Cc: ceph-devel
On 06/03/2015 02:12 AM, Andrew Bartlett wrote:
> On Thu, 2015-05-28 at 13:03 -0700, Sage Weil wrote:
>> 2- make the key management pluggable, so that we can use petera,
>> /etc/ceph/keys, the ceph mons, or something else.
>
> Petera looks very much like what I was trying to home-grow, and I would
> prefer that over using the monitor nodes, as those are often shared with
> our storage nodes.
Hi,
the preferred way to go now is using LUKS and Petera (https://github.com/npmccallum/petera),
I should probably edit the blueprint that is now kind of obsolete.
In general, the plain dm-crypt use is problematic because you have to handle
not only the key but also cipher metadata (cipher, key mode, etc).
Ceph currently uses cryptsetup default which can be different in future
or it can be changed per distro or during compilation time.
LUKS stores this information to the header ensuring you will be able to open it
with proper attributes everywhere.
(And LUKS header can be placed outside of disk itself, but I do not think this
should be done here.)
Milan
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Interested in ceph OSD encryption and key management
2015-05-28 20:03 ` Sage Weil
2015-05-31 14:01 ` Wyllys Ingersoll
2015-06-03 0:12 ` Andrew Bartlett
@ 2015-06-17 2:37 ` Andrew Bartlett
2015-06-17 4:16 ` Sage Weil
2 siblings, 1 reply; 10+ messages in thread
From: Andrew Bartlett @ 2015-06-17 2:37 UTC (permalink / raw)
To: Sage Weil; +Cc: mbroz, ceph-devel
On Thu, 2015-05-28 at 13:03 -0700, Sage Weil wrote:
> Hi Andrew,
>
> I'm copying Milan Broz, who has looked at this ome. There was some
> subsequent off-list discussion in Red Hat about using Petera[1] for the
> key management, but this'll require a bit more effort than what was
> described in that blueprint.
>
> On Thu, 28 May 2015, Andrew Bartlett wrote:
> > David Disseldorp was good enough to point me at this proposal for ceph
> > OSD key management:
> > https://wiki.ceph.com/Planning/Blueprints/Infernalis/osd%3A_simple_ceph-mon_dm-crypt_key_management
> >
> > I'm really interested in improving ceph on-disk encryption, and am
> > really glad folks are taking this beyond the local key storage we have
> > managed so far.
> >
> > So I can be part of the discussion, how do I get a login to the wiki? I
> > would like to indicate my interest there.
>
> The wiki logins are broken, but ignore that.. we're moving to
> tracker.ceph.com's wiki shortly anyway. Email is best in the meantime!
This proposal seems not have to have it to the new wiki. Is it still
alive? What do we need to do to keep this moving?
Thanks,
Andrew Bartlett
--
Andrew Bartlett
http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org
Samba Development and Support, Catalyst IT http://catalyst.net.nz/services/samba
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Interested in ceph OSD encryption and key management
2015-06-17 2:37 ` Andrew Bartlett
@ 2015-06-17 4:16 ` Sage Weil
2015-06-18 13:34 ` Milan Broz
0 siblings, 1 reply; 10+ messages in thread
From: Sage Weil @ 2015-06-17 4:16 UTC (permalink / raw)
To: Andrew Bartlett; +Cc: mbroz, ceph-devel
On Wed, 17 Jun 2015, Andrew Bartlett wrote:
> On Thu, 2015-05-28 at 13:03 -0700, Sage Weil wrote:
> > Hi Andrew,
> >
> > I'm copying Milan Broz, who has looked at this ome. There was some
> > subsequent off-list discussion in Red Hat about using Petera[1] for the
> > key management, but this'll require a bit more effort than what was
> > described in that blueprint.
> >
> > On Thu, 28 May 2015, Andrew Bartlett wrote:
> > > David Disseldorp was good enough to point me at this proposal for ceph
> > > OSD key management:
> > > https://wiki.ceph.com/Planning/Blueprints/Infernalis/osd%3A_simple_ceph-mon_dm-crypt_key_management
> > >
> > > I'm really interested in improving ceph on-disk encryption, and am
> > > really glad folks are taking this beyond the local key storage we have
> > > managed so far.
> > >
> > > So I can be part of the discussion, how do I get a login to the wiki? I
> > > would like to indicate my interest there.
> >
> > The wiki logins are broken, but ignore that.. we're moving to
> > tracker.ceph.com's wiki shortly anyway. Email is best in the meantime!
>
> This proposal seems not have to have it to the new wiki. Is it still
> alive? What do we need to do to keep this moving?
I can create a placeholder session. Can you two hash out a proposal over
the next week or so to discuss? I think there are some tricky questions
if petera is used if we want it to integrate with the monitors as well
(e.g., leverage the monitors for updating/distributing the petera
certs/keys).
sage
>
> Thanks,
>
> Andrew Bartlett
>
> --
> Andrew Bartlett
> http://samba.org/~abartlet/
> Authentication Developer, Samba Team http://samba.org
> Samba Development and Support, Catalyst IT http://catalyst.net.nz/services/samba
>
>
>
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Interested in ceph OSD encryption and key management
2015-06-17 4:16 ` Sage Weil
@ 2015-06-18 13:34 ` Milan Broz
2015-06-18 14:31 ` Sage Weil
0 siblings, 1 reply; 10+ messages in thread
From: Milan Broz @ 2015-06-18 13:34 UTC (permalink / raw)
To: Sage Weil, Andrew Bartlett; +Cc: ceph-devel
On 06/17/2015 06:16 AM, Sage Weil wrote:
>>> The wiki logins are broken, but ignore that.. we're moving to
>>> tracker.ceph.com's wiki shortly anyway. Email is best in the meantime!
>>
>> This proposal seems not have to have it to the new wiki. Is it still
>> alive? What do we need to do to keep this moving?
I am not able to even login to wiki, seems logins are still completely broken.
Probably nobody can edit it now?
> I can create a placeholder session. Can you two hash out a proposal over
> the next week or so to discuss? I think there are some tricky questions
> if petera is used if we want it to integrate with the monitors as well
> (e.g., leverage the monitors for updating/distributing the petera
> certs/keys).
Yes, but later please. We have a lot of fun with selinux and other things,
this feature could follow later.
BTW Petera is now renamed to Deo (and should be in Fedora packaged already).
The idea was:
- use LUKS as the encryption format
- use Deo to map OSD, for more info see
https://github.com/npmccallum/deo
In short, Deo will connect to MON node and negotiate key and directly
unlock LUKS device.
(New version now uses libcryptsetup direcly, so it is really one client binary.)
There must be a service listening on MONs nodes (Deo server).
I have no idea yet how problematic will be handling certificates here, but
because it ensures authentication and encrypts communication, it seems like
a nice design.
Anyway, delegating this task to one simple tool (maintained independently
of Ceph) is IMHO a good idea and the whole integration should not be complicated.
(Moreover it will allow easy integration with FreeIPA and similar tools later.)
Milan
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Interested in ceph OSD encryption and key management
2015-06-18 13:34 ` Milan Broz
@ 2015-06-18 14:31 ` Sage Weil
0 siblings, 0 replies; 10+ messages in thread
From: Sage Weil @ 2015-06-18 14:31 UTC (permalink / raw)
To: Milan Broz; +Cc: Andrew Bartlett, ceph-devel
Hi Milan-
On Thu, 18 Jun 2015, Milan Broz wrote:
> On 06/17/2015 06:16 AM, Sage Weil wrote:
> >>> The wiki logins are broken, but ignore that.. we're moving to
> >>> tracker.ceph.com's wiki shortly anyway. Email is best in the meantime!
> >>
> >> This proposal seems not have to have it to the new wiki. Is it still
> >> alive? What do we need to do to keep this moving?
>
> I am not able to even login to wiki, seems logins are still completely broken.
> Probably nobody can edit it now?
The old wiki is frozen.. see the new wiki at
http://tracker.ceph.com/projects/ceph/wiki (not much there yet). Jewel
CDS blueprints are at
http://tracker.ceph.com/projects/ceph/wiki/Jewel
> > I can create a placeholder session. Can you two hash out a proposal over
> > the next week or so to discuss? I think there are some tricky questions
> > if petera is used if we want it to integrate with the monitors as well
> > (e.g., leverage the monitors for updating/distributing the petera
> > certs/keys).
>
> Yes, but later please. We have a lot of fun with selinux and other things,
> this feature could follow later.
>
> BTW Petera is now renamed to Deo (and should be in Fedora packaged already).
(Good to hear--petera had bad google mojo.)
> The idea was:
>
> - use LUKS as the encryption format
>
> - use Deo to map OSD, for more info see
> https://github.com/npmccallum/deo
>
> In short, Deo will connect to MON node and negotiate key and directly
> unlock LUKS device.
> (New version now uses libcryptsetup direcly, so it is really one client binary.)
>
> There must be a service listening on MONs nodes (Deo server).
>
> I have no idea yet how problematic will be handling certificates here, but
> because it ensures authentication and encrypts communication, it seems like
> a nice design.
I think the ceph-mon piece is that we would need a monitor service that is
responsible for storing/distributing the certs/keys. In order to make
them visible to Deo, we'd need them to be stored in plaintext in a simple
directory structure in /var/lib/ceph/mon/$cluster-$id/ somewhere. If
we simply co-opt config-key to do this, or do this for a subset of the
config-key namespace, this should be pretty simple.
Do you know what the server-side file/directory should look like for Deo?
> Anyway, delegating this task to one simple tool (maintained independently
> of Ceph) is IMHO a good idea and the whole integration should not be complicated.
> (Moreover it will allow easy integration with FreeIPA and similar tools later.)
Yeah. My assumption is that this is essentially swapping out the bits
that use the ceph mons to install keys?
sage
^ permalink raw reply [flat|nested] 10+ messages in thread