From: Jarkko Sakkinen <jarkko@kernel.org>
To: Kai Huang <kai.huang@intel.com>
Cc: kvm@vger.kernel.org, x86@kernel.org, linux-sgx@vger.kernel.org,
linux-kernel@vger.kernel.org, seanjc@google.com, luto@kernel.org,
dave.hansen@intel.com, rick.p.edgecombe@intel.com,
haitao.huang@intel.com, pbonzini@redhat.com, bp@alien8.de,
tglx@linutronix.de, mingo@redhat.com, hpa@zytor.com,
jethro@fortanix.com, b.thiel@posteo.de, jmattson@google.com,
joro@8bytes.org, vkuznets@redhat.com, wanpengli@tencent.com,
corbet@lwn.net
Subject: Re: [PATCH v3 00/25] KVM SGX virtualization support
Date: Fri, 19 Mar 2021 16:52:01 +0200 [thread overview]
Message-ID: <YFS6kTe1SuAjiMFN@kernel.org> (raw)
In-Reply-To: <cover.1616136307.git.kai.huang@intel.com>
On Fri, Mar 19, 2021 at 08:29:27PM +1300, Kai Huang wrote:
> This series adds KVM SGX virtualization support. The first 14 patches starting
> with x86/sgx or x86/cpu.. are necessary changes to x86 and SGX core/driver to
> support KVM SGX virtualization, while the rest are patches to KVM subsystem.
>
> This series is based against latest tip/x86/sgx, which has Jarkko's NUMA
> allocation support.
>
> You can also get the code from upstream branch of kvm-sgx repo on github:
>
> https://github.com/intel/kvm-sgx.git upstream
>
> It also requires Qemu changes to create VM with SGX support. You can find Qemu
> repo here:
>
> https://github.com/intel/qemu-sgx.git upstream
>
> Please refer to README.md of above qemu-sgx repo for detail on how to create
> guest with SGX support. At meantime, for your quick reference you can use below
> command to create SGX guest:
>
> #qemu-system-x86_64 -smp 4 -m 2G -drive file=<your_vm_image>,if=virtio \
> -cpu host,+sgx_provisionkey \
> -sgx-epc id=epc1,memdev=mem1 \
> -object memory-backend-epc,id=mem1,size=64M,prealloc
>
> Please note that the SGX relevant part is:
>
> -cpu host,+sgx_provisionkey \
> -sgx-epc id=epc1,memdev=mem1 \
> -object memory-backend-epc,id=mem1,size=64M,prealloc
>
> And you can change other parameters of your qemu command based on your needs.
>
> =========
> Changelog:
>
> (Changelog here is for global changes. Please see each patch's changelog for
> changes made to specific patch.)
>
> v2->v3:
>
> - No big change in design, structure of patch series, etc.
> - Rebased to lastest tip/x86/sgx, to resolve merge conflict of patch 3
> (x86/sgx: Wipe out EREMOVE from sgx_free_epc_page()).
> - Addressed some Nit issues found by Sean in v2.
> - Also addressed some Nit issues reported by checkpatch.pl. Now there's no
> checkpatch issues.
> - Updated patch 3 (x86/sgx: Wipe out EREMOVE from sgx_free_epc_page()):
> - Removed Jarkko from author, per request.
> - Changed to replace all call sites of sgx_free_epc_page() with new
> sgx_encl_free_epc_page(), to make this patch doesn't have functional
> changes (except a WARN upon EREMOVE failure requestd by Dave).
> - Rebased to tip/x86/sgx, which has Jarkko's NUMA allocation.
> - Added Jarkko's Acked-by.
> - Updated patch 8 (x86/sgx: Expose SGX architectural definitions to the
> kernel) to add MAINTAINER file update to include new introduced asm/sgx.h.
> - Updated patch 13 (x86/sgx: Add helpers to expose ECREATE and EINIT to KVM)
> to use addr and size directly in access_ok()s (which won't be triggered
> anyway).
>
> v1->v2:
>
> - No big change in design, structural of patch series, etc.
> - Addressed Boris's comments regarding to suppressing both SGX1 and SGX2 in
> /proc/cpuinfo, and improvement in feat_ctl.c when enabling SGX (patch 2
> and 6).
> - Addressed Sean's comments for both x86 part patches and KVM patches (patch 3,
> 5, 9, 12, 19, 21).
> - Addressed Dave's comments in RFC v6 series (patch 13).
>
> RFC->v1:
>
> - Refined patch (x86/sgx: Wipe out EREMOVE from sgx_free_epc_page()) to print
> error msg that EPC page is leaked when EREMOVE failed, requested by Dave.
> - Changelog history of all RFC series is removed in both this cover letter
> and each individual patch, since majority of x86 part patches already got
> Acked-by from Dave and Jarkko. And the changelogs are not quite useful from
> my perspective.
>
> =========
> KVM SGX virtualization Overview
>
> - Virtual EPC
>
> SGX enclave memory is special and is reserved specifically for enclave use.
> In bare-metal SGX enclaves, the kernel allocates enclave pages, copies data
> into the pages with privileged instructions, then allows the enclave to start.
> In this scenario, only initialized pages already assigned to an enclave are
> mapped to userspace.
>
> In virtualized environments, the hypervisor still needs to do the physical
> enclave page allocation. The guest kernel is responsible for the data copying
> (among other things). This means that the job of starting an enclave is now
> split between hypervisor and guest.
>
> This series introduces a new misc device: /dev/sgx_vepc. This device allows
> the host to map *uninitialized* enclave memory into userspace, which can then
> be passed into a guest.
>
> While it might be *possible* to start a host-side enclave with /dev/sgx_enclave
> and pass its memory into a guest, it would be wasteful and convoluted.
>
> Implement the *raw* EPC allocation in the x86 core-SGX subsystem via
> /dev/sgx_vepc rather than in KVM. Doing so has two major advantages:
>
> - Does not require changes to KVM's uAPI, e.g. EPC gets handled as
> just another memory backend for guests.
>
> - EPC management is wholly contained in the SGX subsystem, e.g. SGX
> does not have to export any symbols, changes to reclaim flows don't
> need to be routed through KVM, SGX's dirty laundry doesn't have to
> get aired out for the world to see, and so on and so forth.
>
> The virtual EPC pages allocated to guests are currently not reclaimable.
> Reclaiming EPC page used by enclave requires a special reclaim mechanism
> separate from normal page reclaim, and that mechanism is not supported
> for virutal EPC pages. Due to the complications of handling reclaim
> conflicts between guest and host, reclaiming virtual EPC pages is
> significantly more complex than basic support for SGX virtualization.
>
> - Support SGX virtualization without SGX Flexible Launch Control
>
> SGX hardware supports two "launch control" modes to limit which enclaves can
> run. In the "locked" mode, the hardware prevents enclaves from running unless
> they are blessed by a third party. In the unlocked mode, the kernel is in
> full control of which enclaves can run. The bare-metal SGX code refuses to
> launch enclaves unless it is in the unlocked mode.
>
> This sgx_virt_epc driver does not have such a restriction. This allows guests
> which are OK with the locked mode to use SGX, even if the host kernel refuses
> to.
>
> - Support exposing SGX2
>
> Due to the same reason above, SGX2 feature detection is added to core SGX code
> to allow KVM to expose SGX2 to guest, even currently SGX driver doesn't support
> SGX2, because SGX2 can work just fine in guest w/o any interaction to host SGX
> driver.
>
> - Restricit SGX guest access to provisioning key
>
> To grant guest being able to fully use SGX, guest needs to be able to access
> provisioning key. The provisioning key is sensitive, and accessing to it should
> be restricted. In bare-metal driver, allowing enclave to access provisioning key
> is restricted by being able to open /dev/sgx_provision.
>
> Add a new KVM_CAP_SGX_ATTRIBUTE to KVM uAPI to extend above mechanism to KVM
> guests as well. When userspace hypervisor creates a new VM, the new cap is only
> added to VM when userspace hypervisior is able to open /dev/sgx_provision,
> following the same role as in bare-metal driver. KVM then traps ECREATE from
> guest, and only allows ECREATE with provisioning key bit to run when guest
> supports KVM_CAP_SGX_ATTRIBUTE.
>
>
> Kai Huang (4):
> x86/cpufeatures: Make SGX_LC feature bit depend on SGX bit
> x86/sgx: Wipe out EREMOVE from sgx_free_epc_page()
> x86/sgx: Initialize virtual EPC driver even when SGX driver is
> disabled
> x86/sgx: Add helper to update SGX_LEPUBKEYHASHn MSRs
>
> Sean Christopherson (21):
> x86/cpufeatures: Add SGX1 and SGX2 sub-features
> x86/sgx: Add SGX_CHILD_PRESENT hardware error code
> x86/sgx: Introduce virtual EPC for use by KVM guests
> x86/cpu/intel: Allow SGX virtualization without Launch Control support
> x86/sgx: Expose SGX architectural definitions to the kernel
> x86/sgx: Move ENCLS leaf definitions to sgx.h
> x86/sgx: Add SGX2 ENCLS leaf definitions (EAUG, EMODPR and EMODT)
> x86/sgx: Add encls_faulted() helper
> x86/sgx: Add helpers to expose ECREATE and EINIT to KVM
> x86/sgx: Move provisioning device creation out of SGX driver
> KVM: x86: Export kvm_mmu_gva_to_gpa_{read,write}() for SGX (VMX)
> KVM: x86: Define new #PF SGX error code bit
> KVM: x86: Add support for reverse CPUID lookup of scattered features
> KVM: x86: Add reverse-CPUID lookup support for scattered SGX features
> KVM: VMX: Add basic handling of VM-Exit from SGX enclave
> KVM: VMX: Frame in ENCLS handler for SGX virtualization
> KVM: VMX: Add SGX ENCLS[ECREATE] handler to enforce CPUID restrictions
> KVM: VMX: Add emulation of SGX Launch Control LE hash MSRs
> KVM: VMX: Add ENCLS[EINIT] handler to support SGX Launch Control (LC)
> KVM: VMX: Enable SGX virtualization for SGX1, SGX2 and LC
> KVM: x86: Add capability to grant VM access to privileged SGX
> attribute
>
> Documentation/virt/kvm/api.rst | 23 +
> MAINTAINERS | 1 +
> arch/x86/Kconfig | 12 +
> arch/x86/include/asm/cpufeatures.h | 2 +
> arch/x86/include/asm/kvm_host.h | 5 +
> .../cpu/sgx/arch.h => include/asm/sgx.h} | 50 +-
> arch/x86/include/asm/vmx.h | 1 +
> arch/x86/include/uapi/asm/vmx.h | 1 +
> arch/x86/kernel/cpu/cpuid-deps.c | 3 +
> arch/x86/kernel/cpu/feat_ctl.c | 71 ++-
> arch/x86/kernel/cpu/scattered.c | 2 +
> arch/x86/kernel/cpu/sgx/Makefile | 1 +
> arch/x86/kernel/cpu/sgx/driver.c | 17 -
> arch/x86/kernel/cpu/sgx/encl.c | 42 +-
> arch/x86/kernel/cpu/sgx/encl.h | 1 +
> arch/x86/kernel/cpu/sgx/encls.h | 30 +-
> arch/x86/kernel/cpu/sgx/ioctl.c | 29 +-
> arch/x86/kernel/cpu/sgx/main.c | 96 +++-
> arch/x86/kernel/cpu/sgx/sgx.h | 13 +-
> arch/x86/kernel/cpu/sgx/virt.c | 369 ++++++++++++++
> arch/x86/kvm/Makefile | 2 +
> arch/x86/kvm/cpuid.c | 89 +++-
> arch/x86/kvm/cpuid.h | 50 +-
> arch/x86/kvm/vmx/nested.c | 28 +-
> arch/x86/kvm/vmx/nested.h | 5 +
> arch/x86/kvm/vmx/sgx.c | 481 ++++++++++++++++++
> arch/x86/kvm/vmx/sgx.h | 34 ++
> arch/x86/kvm/vmx/vmcs12.c | 1 +
> arch/x86/kvm/vmx/vmcs12.h | 4 +-
> arch/x86/kvm/vmx/vmx.c | 109 +++-
> arch/x86/kvm/vmx/vmx.h | 2 +
> arch/x86/kvm/x86.c | 23 +
> include/uapi/linux/kvm.h | 1 +
> tools/testing/selftests/sgx/defines.h | 2 +-
> 34 files changed, 1476 insertions(+), 124 deletions(-)
> rename arch/x86/{kernel/cpu/sgx/arch.h => include/asm/sgx.h} (89%)
> create mode 100644 arch/x86/kernel/cpu/sgx/virt.c
> create mode 100644 arch/x86/kvm/vmx/sgx.c
> create mode 100644 arch/x86/kvm/vmx/sgx.h
>
> --
> 2.30.2
>
>
I just say add my ack to SGX specific patches where it is missing.
Good enough.
/Jarkko
next prev parent reply other threads:[~2021-03-19 14:53 UTC|newest]
Thread overview: 124+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-03-19 7:29 [PATCH v3 00/25] KVM SGX virtualization support Kai Huang
2021-03-19 7:22 ` [PATCH v3 01/25] x86/cpufeatures: Make SGX_LC feature bit depend on SGX bit Kai Huang
2021-04-07 10:03 ` [tip: x86/sgx] " tip-bot2 for Kai Huang
2021-03-19 7:22 ` [PATCH v3 02/25] x86/cpufeatures: Add SGX1 and SGX2 sub-features Kai Huang
2021-04-07 10:03 ` [tip: x86/sgx] " tip-bot2 for Sean Christopherson
2021-03-19 7:22 ` [PATCH v3 03/25] x86/sgx: Wipe out EREMOVE from sgx_free_epc_page() Kai Huang
2021-03-22 18:16 ` Borislav Petkov
2021-03-22 18:56 ` Sean Christopherson
2021-03-22 19:11 ` Paolo Bonzini
2021-03-22 20:43 ` Kai Huang
2021-03-23 16:40 ` Paolo Bonzini
2021-03-22 19:15 ` Borislav Petkov
2021-03-22 19:37 ` Sean Christopherson
2021-03-22 20:36 ` Kai Huang
2021-03-22 21:06 ` Borislav Petkov
2021-03-22 22:06 ` Kai Huang
2021-03-22 22:37 ` Borislav Petkov
2021-03-22 23:16 ` Kai Huang
2021-03-23 15:45 ` Sean Christopherson
2021-03-23 16:06 ` Borislav Petkov
2021-03-23 16:21 ` Sean Christopherson
2021-03-23 16:32 ` Borislav Petkov
2021-03-23 16:51 ` Sean Christopherson
2021-03-24 9:38 ` Kai Huang
2021-03-24 10:09 ` Paolo Bonzini
2021-03-24 10:48 ` Kai Huang
2021-03-24 11:24 ` Paolo Bonzini
2021-03-24 23:23 ` Kai Huang
2021-03-24 23:39 ` Paolo Bonzini
2021-03-24 23:46 ` Kai Huang
2021-03-25 8:42 ` Borislav Petkov
2021-03-25 9:38 ` Kai Huang
2021-03-25 16:52 ` Borislav Petkov
2021-03-24 9:28 ` Jarkko Sakkinen
2021-03-23 16:38 ` Paolo Bonzini
2021-03-23 17:02 ` Sean Christopherson
2021-03-23 17:06 ` Paolo Bonzini
2021-03-23 17:16 ` Sean Christopherson
2021-03-23 18:16 ` Borislav Petkov
2021-03-24 9:26 ` Jarkko Sakkinen
2021-03-22 22:23 ` Kai Huang
2021-03-25 9:30 ` [PATCH v4 " Kai Huang
2021-03-26 19:48 ` Jarkko Sakkinen
2021-03-26 20:38 ` Kai Huang
2021-03-26 21:39 ` Jarkko Sakkinen
2021-04-07 10:03 ` [tip: x86/sgx] " tip-bot2 for Kai Huang
2021-03-19 7:22 ` [PATCH v3 04/25] x86/sgx: Add SGX_CHILD_PRESENT hardware error code Kai Huang
2021-04-07 10:03 ` [tip: x86/sgx] " tip-bot2 for Sean Christopherson
2021-03-19 7:22 ` [PATCH v3 05/25] x86/sgx: Introduce virtual EPC for use by KVM guests Kai Huang
2021-03-25 9:36 ` Kai Huang
2021-03-26 15:03 ` Borislav Petkov
2021-03-26 15:17 ` Dave Hansen
2021-03-26 15:29 ` Borislav Petkov
2021-03-26 15:35 ` Dave Hansen
2021-03-26 17:02 ` Borislav Petkov
2021-03-31 1:10 ` Kai Huang
2021-03-31 6:44 ` Boris Petkov
2021-03-31 6:51 ` Kai Huang
2021-03-31 7:44 ` Boris Petkov
2021-03-31 8:53 ` Kai Huang
2021-03-31 12:20 ` Kai Huang
2021-04-01 18:31 ` Borislav Petkov
2021-04-01 23:38 ` Kai Huang
2021-04-01 9:45 ` Kai Huang
2021-04-01 9:42 ` [PATCH v4 " Kai Huang
2021-04-05 9:01 ` [PATCH v3 " Borislav Petkov
2021-04-05 21:46 ` Kai Huang
2021-04-06 8:28 ` Borislav Petkov
2021-04-06 9:04 ` Kai Huang
2021-04-07 10:03 ` [tip: x86/sgx] " tip-bot2 for Sean Christopherson
2021-03-19 7:22 ` [PATCH v3 06/25] x86/cpu/intel: Allow SGX virtualization without Launch Control support Kai Huang
2021-04-07 10:03 ` [tip: x86/sgx] " tip-bot2 for Sean Christopherson
2021-03-19 7:23 ` [PATCH v3 07/25] x86/sgx: Initialize virtual EPC driver even when SGX driver is disabled Kai Huang
2021-04-02 9:48 ` Borislav Petkov
2021-04-02 11:08 ` Kai Huang
2021-04-02 11:22 ` Borislav Petkov
2021-04-02 11:38 ` Kai Huang
2021-04-02 15:42 ` Sean Christopherson
2021-04-02 19:08 ` Kai Huang
2021-04-02 19:19 ` Borislav Petkov
2021-04-02 19:30 ` Sean Christopherson
2021-04-02 19:46 ` Borislav Petkov
2021-04-07 10:03 ` [tip: x86/sgx] " tip-bot2 for Kai Huang
2021-03-19 7:23 ` [PATCH v3 08/25] x86/sgx: Expose SGX architectural definitions to the kernel Kai Huang
2021-04-07 10:03 ` [tip: x86/sgx] " tip-bot2 for Sean Christopherson
2021-03-19 7:23 ` [PATCH v3 09/25] x86/sgx: Move ENCLS leaf definitions to sgx.h Kai Huang
2021-04-07 10:03 ` [tip: x86/sgx] " tip-bot2 for Sean Christopherson
2021-03-19 7:23 ` [PATCH v3 10/25] x86/sgx: Add SGX2 ENCLS leaf definitions (EAUG, EMODPR and EMODT) Kai Huang
2021-04-07 10:03 ` [tip: x86/sgx] " tip-bot2 for Sean Christopherson
2021-03-19 7:23 ` [PATCH v3 11/25] x86/sgx: Add encls_faulted() helper Kai Huang
2021-04-07 10:03 ` [tip: x86/sgx] " tip-bot2 for Sean Christopherson
2021-03-19 7:23 ` [PATCH v3 12/25] x86/sgx: Add helper to update SGX_LEPUBKEYHASHn MSRs Kai Huang
2021-04-07 10:03 ` [tip: x86/sgx] " tip-bot2 for Kai Huang
2021-03-19 7:23 ` [PATCH v3 13/25] x86/sgx: Add helpers to expose ECREATE and EINIT to KVM Kai Huang
2021-04-05 9:07 ` Borislav Petkov
2021-04-05 21:44 ` Kai Huang
2021-04-06 7:40 ` Borislav Petkov
2021-04-06 8:59 ` Kai Huang
2021-04-06 9:09 ` Borislav Petkov
2021-04-06 9:24 ` Kai Huang
2021-04-06 9:32 ` Borislav Petkov
2021-04-06 9:41 ` Kai Huang
2021-04-06 17:08 ` Borislav Petkov
2021-04-06 20:33 ` Kai Huang
2021-04-07 10:03 ` [tip: x86/sgx] " tip-bot2 for Sean Christopherson
2021-03-19 7:23 ` [PATCH v3 14/25] x86/sgx: Move provisioning device creation out of SGX driver Kai Huang
2021-04-07 10:03 ` [tip: x86/sgx] " tip-bot2 for Sean Christopherson
2021-03-19 7:23 ` [PATCH v3 15/25] KVM: x86: Export kvm_mmu_gva_to_gpa_{read,write}() for SGX (VMX) Kai Huang
2021-03-19 7:23 ` [PATCH v3 16/25] KVM: x86: Define new #PF SGX error code bit Kai Huang
2021-03-19 7:23 ` [PATCH v3 17/25] KVM: x86: Add support for reverse CPUID lookup of scattered features Kai Huang
2021-03-19 7:23 ` [PATCH v3 18/25] KVM: x86: Add reverse-CPUID lookup support for scattered SGX features Kai Huang
2021-03-19 7:23 ` [PATCH v3 19/25] KVM: VMX: Add basic handling of VM-Exit from SGX enclave Kai Huang
2021-03-19 7:23 ` [PATCH v3 20/25] KVM: VMX: Frame in ENCLS handler for SGX virtualization Kai Huang
2021-03-19 7:23 ` [PATCH v3 21/25] KVM: VMX: Add SGX ENCLS[ECREATE] handler to enforce CPUID restrictions Kai Huang
2021-03-19 7:23 ` [PATCH v3 22/25] KVM: VMX: Add emulation of SGX Launch Control LE hash MSRs Kai Huang
2021-03-19 7:23 ` [PATCH v3 23/25] KVM: VMX: Add ENCLS[EINIT] handler to support SGX Launch Control (LC) Kai Huang
2021-03-19 7:23 ` [PATCH v3 24/25] KVM: VMX: Enable SGX virtualization for SGX1, SGX2 and LC Kai Huang
2021-03-19 7:24 ` [PATCH v3 25/25] KVM: x86: Add capability to grant VM access to privileged SGX attribute Kai Huang
2021-03-19 14:52 ` Jarkko Sakkinen [this message]
2021-03-22 10:03 ` [PATCH v3 00/25] KVM SGX virtualization support Kai Huang
2021-03-22 10:31 ` Borislav Petkov
2021-03-26 22:46 ` Jarkko Sakkinen
2021-03-28 21:01 ` Huang, Kai
2021-03-31 23:23 ` Jarkko Sakkinen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=YFS6kTe1SuAjiMFN@kernel.org \
--to=jarkko@kernel.org \
--cc=b.thiel@posteo.de \
--cc=bp@alien8.de \
--cc=corbet@lwn.net \
--cc=dave.hansen@intel.com \
--cc=haitao.huang@intel.com \
--cc=hpa@zytor.com \
--cc=jethro@fortanix.com \
--cc=jmattson@google.com \
--cc=joro@8bytes.org \
--cc=kai.huang@intel.com \
--cc=kvm@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-sgx@vger.kernel.org \
--cc=luto@kernel.org \
--cc=mingo@redhat.com \
--cc=pbonzini@redhat.com \
--cc=rick.p.edgecombe@intel.com \
--cc=seanjc@google.com \
--cc=tglx@linutronix.de \
--cc=vkuznets@redhat.com \
--cc=wanpengli@tencent.com \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).