From patchwork Wed Jan 6 01:55:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Kai" X-Patchwork-Id: 12000757 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F0BF3C433DB for ; Wed, 6 Jan 2021 01:56:02 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id AD4AA22CE3 for ; Wed, 6 Jan 2021 01:56:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1725919AbhAFB4C (ORCPT ); Tue, 5 Jan 2021 20:56:02 -0500 Received: from mga04.intel.com ([192.55.52.120]:11738 "EHLO mga04.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725862AbhAFB4B (ORCPT ); Tue, 5 Jan 2021 20:56:01 -0500 IronPort-SDR: tSWTvbMIEd7IXNtfbIg5mUP+LLoxk7SqqmODszkEzc75EYdNSkeAwiTFvGLJz1nYFFGkSYz/tR GZUn/VGnrRQg== X-IronPort-AV: E=McAfee;i="6000,8403,9855"; a="174636823" X-IronPort-AV: E=Sophos;i="5.78,478,1599548400"; d="scan'208";a="174636823" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Jan 2021 17:55:21 -0800 IronPort-SDR: oVDvdbwTVC8/bptezjwLna1QIU5aVJi/ApOV4BGJBGeMlpSxcik3nc+xCIfx6fl7hvoN2RqgL5 dbLk01ITvPJQ== X-IronPort-AV: E=Sophos;i="5.78,478,1599548400"; d="scan'208";a="421993073" Received: from zhuoxuan-mobl.amr.corp.intel.com (HELO khuang2-desk.gar.corp.intel.com) ([10.251.29.237]) by orsmga001-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Jan 2021 17:55:17 -0800 From: Kai Huang To: linux-sgx@vger.kernel.org, kvm@vger.kernel.org, x86@kernel.org Cc: seanjc@google.com, jarkko@kernel.org, luto@kernel.org, dave.hansen@intel.com, haitao.huang@intel.com, pbonzini@redhat.com, bp@alien8.de, tglx@linutronix.de, mingo@redhat.com, hpa@zytor.com, jethro@fortanix.com, b.thiel@posteo.de, mattson@google.com, joro@8bytes.org, vkuznets@redhat.com, wanpengli@tencent.com, corbet@lwn.net Subject: [RFC PATCH 00/23] KVM SGX virtualization support Date: Wed, 6 Jan 2021 14:55:06 +1300 Message-Id: X-Mailer: git-send-email 2.29.2 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org --- Disclaimer --- These patches were originally written by Sean Christopherson while at Intel. Now that Sean has left Intel, I (Kai) have taken over getting them upstream. This series needs more review before it can be merged. It is being posted publicly and under RFC so Sean and others can review it. Maintainers are safe ignoring it for now. ------------------ Hi all, This series adds KVM SGX virtualization support. The first 12 patches starting with x86/sgx or x86/cpu.. are necessary changes to x86 and SGX core/driver to support KVM SGX virtualization, while the rest are patches to KVM subsystem. Please help to review this series. Also I'd like to hear what is the proper way to merge this series, since it contains change to both x86/SGX and KVM subsystem. Any feedback is highly appreciated. And please let me know if I forgot to CC anyone, or anyone wants to be removed from CC. Thanks in advance! This series is based against latest tip tree's x86/sgx branch. You can also get the code from tip branch of kvm-sgx repo on github: https://github.com/intel/kvm-sgx.git tip It also requires Qemu changes to create VM with SGX support. You can find Qemu repo here: https://github.com/intel/qemu-sgx.git next Please refer to README.md of above qemu-sgx repo for detail on how to create guest with SGX support. At meantime, for your quick reference you can use below command to create SGX guest: #qemu-system-x86_64 -smp 4 -m 2G -drive file=,if=virtio \ -cpu host,+sgx_provisionkey \ -sgx-epc id=epc1,memdev=mem1 \ -object memory-backend-epc,id=mem1,size=64M,prealloc Please note that the SGX relevant part is: -cpu host,+sgx_provisionkey \ -sgx-epc id=epc1,memdev=mem1 \ -object memory-backend-epc,id=mem1,size=64M,prealloc And you can change other parameters of your qemu command based on your needs. ========= KVM SGX virtualization Overview - Virtual EPC "Virtual EPC" is the EPC section exposed by KVM to guest so SGX software in guest can discover it and use it to create SGX enclaves. KVM exposes SGX to guest via CPUID, and exposes one or more "virtual EPC" sections for guest. The size of "virtual EPC" is passed as Qemu parameter when creating the guest, and the base address is calcualted internally according to guest's configuration. To support virtual EPC, add a new misc device /dev/sgx_virt_epc to SGX core/driver to allow userspace (Qemu) to allocate "raw" EPC, and use it as "virtual EPC" for guest. Obviously, unlike EPC allocated for host SGX driver, virtual EPC allocated via /dev/sgx_virt_epc doesn't have enclave associated, and how virtual EPC is used by guest is compeletely controlled by guest's SGX software. Implement the "raw" EPC allocation in the x86 core-SGX subsystem via /dev/sgx_virt_epc rather than in KVM. Doing so has two major advantages: - Does not require changes to KVM's uAPI, e.g. EPC gets handled as just another memory backend for guests. - EPC management is wholly contained in the SGX subsystem, e.g. SGX does not have to export any symbols, changes to reclaim flows don't need to be routed through KVM, SGX's dirty laundry doesn't have to get aired out for the world to see, and so on and so forth. The virtual EPC allocated to guests is currently not reclaimable, due to reclaiming EPC from KVM guests is not currently supported. Due to the complications of handling reclaim conflicts between guest and host, KVM EPC oversubscription, which allows total virtual EPC size greater than physical EPC by being able to reclaiming guests' EPC, is significantly more complex than basic support for SGX virtualization. - Support SGX virtualization without SGX Launch Control unlocked mode Although SGX driver requires SGX Launch Control unlocked mode to work, SGX virtualization doesn't, since how enclave is created is completely controlled by guest SGX software, which is not necessarily linux. Therefore, this series allows KVM to expose SGX to guest even SGX Launch Control is in locked mode, or is not present at all. The reason is the goal of SGX virtualization, or virtualization in general, is to expose hardware feature to guest, but not to make assumption how guest will use it. Therefore, KVM should support SGX guest as long as hardware is able to, to have chance to support more potential use cases in cloud environment. - Support exposing SGX2 Due to the same reason above, SGX2 feature detection is added to core SGX code to allow KVM to expose SGX2 to guest, even currently SGX driver doesn't support SGX2, because SGX2 can work just fine in guest w/o any interaction to host SGX driver. - Restricit SGX guest access to provisioning key To grant guest being able to fully use SGX, guest needs to be able to create provisioning enclave. However provisioning key is sensitive and is restricted by /dev/sgx_provision in host SGX driver, therefore KVM SGX virtualization follows the same role: a new KVM_CAP_SGX_ATTRIBUTE is added to KVM uAPI, and only file descriptor of /dev/sgx_provision is passed to that CAP by usersppace hypervisor (Qemu) when creating the guest, it can access provisioning bit. This is done by making KVM trape ECREATE instruction from guest, and check the provisioning bit in ECREATE's attribute. Kai Huang (1): x86/sgx: Add helper to update SGX_LEPUBKEYHASHn MSRs Sean Christopherson (22): x86/sgx: Split out adding EPC page to free list to separate helper x86/sgx: Add enum for SGX_CHILD_PRESENT error code x86/sgx: Introduce virtual EPC for use by KVM guests x86/cpufeatures: Add SGX1 and SGX2 sub-features x86/cpu/intel: Allow SGX virtualization without Launch Control support x86/sgx: Expose SGX architectural definitions to the kernel x86/sgx: Move ENCLS leaf definitions to sgx_arch.h x86/sgx: Add SGX2 ENCLS leaf definitions (EAUG, EMODPR and EMODT) x86/sgx: Add encls_faulted() helper x86/sgx: Add helpers to expose ECREATE and EINIT to KVM x86/sgx: Move provisioning device creation out of SGX driver KVM: VMX: Convert vcpu_vmx.exit_reason to a union KVM: x86: Export kvm_mmu_gva_to_gpa_{read,write}() for SGX (VMX) KVM: x86: Define new #PF SGX error code bit KVM: x86: Add SGX feature leaf to reverse CPUID lookup KVM: VMX: Add basic handling of VM-Exit from SGX enclave KVM: VMX: Frame in ENCLS handler for SGX virtualization KVM: VMX: Add SGX ENCLS[ECREATE] handler to enforce CPUID restrictions KVM: VMX: Add emulation of SGX Launch Control LE hash MSRs KVM: VMX: Add ENCLS[EINIT] handler to support SGX Launch Control (LC) KVM: VMX: Enable SGX virtualization for SGX1, SGX2 and LC KVM: x86: Add capability to grant VM access to privileged SGX attribute Documentation/virt/kvm/api.rst | 23 + arch/x86/Kconfig | 12 + arch/x86/include/asm/cpufeature.h | 5 +- arch/x86/include/asm/cpufeatures.h | 6 +- arch/x86/include/asm/disabled-features.h | 7 +- arch/x86/include/asm/kvm_host.h | 5 + arch/x86/include/asm/required-features.h | 2 +- arch/x86/include/asm/sgx.h | 19 + .../cpu/sgx/arch.h => include/asm/sgx_arch.h} | 20 + arch/x86/include/asm/vmx.h | 1 + arch/x86/include/uapi/asm/vmx.h | 1 + arch/x86/kernel/cpu/common.c | 4 + arch/x86/kernel/cpu/feat_ctl.c | 50 +- arch/x86/kernel/cpu/sgx/Makefile | 1 + arch/x86/kernel/cpu/sgx/driver.c | 17 - arch/x86/kernel/cpu/sgx/encl.c | 2 +- arch/x86/kernel/cpu/sgx/encls.h | 29 +- arch/x86/kernel/cpu/sgx/ioctl.c | 23 +- arch/x86/kernel/cpu/sgx/main.c | 79 ++- arch/x86/kernel/cpu/sgx/sgx.h | 5 +- arch/x86/kernel/cpu/sgx/virt.c | 318 ++++++++++++ arch/x86/kernel/cpu/sgx/virt.h | 14 + arch/x86/kvm/Makefile | 2 + arch/x86/kvm/cpuid.c | 58 ++- arch/x86/kvm/cpuid.h | 1 + arch/x86/kvm/vmx/nested.c | 70 ++- arch/x86/kvm/vmx/nested.h | 5 + arch/x86/kvm/vmx/sgx.c | 462 ++++++++++++++++++ arch/x86/kvm/vmx/sgx.h | 34 ++ arch/x86/kvm/vmx/vmcs12.c | 1 + arch/x86/kvm/vmx/vmcs12.h | 4 +- arch/x86/kvm/vmx/vmx.c | 171 +++++-- arch/x86/kvm/vmx/vmx.h | 27 +- arch/x86/kvm/x86.c | 24 + include/uapi/linux/kvm.h | 1 + tools/testing/selftests/sgx/defines.h | 2 +- 36 files changed, 1366 insertions(+), 139 deletions(-) create mode 100644 arch/x86/include/asm/sgx.h rename arch/x86/{kernel/cpu/sgx/arch.h => include/asm/sgx_arch.h} (96%) create mode 100644 arch/x86/kernel/cpu/sgx/virt.c create mode 100644 arch/x86/kernel/cpu/sgx/virt.h create mode 100644 arch/x86/kvm/vmx/sgx.c create mode 100644 arch/x86/kvm/vmx/sgx.h