From patchwork Tue Nov 19 08:49:41 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yang, Weijiang" X-Patchwork-Id: 11251547 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D568514E5 for ; Tue, 19 Nov 2019 08:47:55 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id AC691222DE for ; Tue, 19 Nov 2019 08:47:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727316AbfKSIry (ORCPT ); Tue, 19 Nov 2019 03:47:54 -0500 Received: from mga17.intel.com ([192.55.52.151]:26328 "EHLO mga17.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725798AbfKSIrx (ORCPT ); Tue, 19 Nov 2019 03:47:53 -0500 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga107.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 19 Nov 2019 00:47:53 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.68,322,1569308400"; d="scan'208";a="196426982" Received: from unknown (HELO local-michael-cet-test.sh.intel.com) ([10.239.159.128]) by orsmga007.jf.intel.com with ESMTP; 19 Nov 2019 00:47:50 -0800 From: Yang Weijiang To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, pbonzini@redhat.com, jmattson@google.com, sean.j.christopherson@intel.com Cc: yu.c.zhang@linux.intel.com, alazar@bitdefender.com, edwin.zhai@intel.com, Yang Weijiang Subject: [PATCH v7 1/9] Documentation: Introduce EPT based Subpage Protection and related ioctls Date: Tue, 19 Nov 2019 16:49:41 +0800 Message-Id: <20191119084949.15471-2-weijiang.yang@intel.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20191119084949.15471-1-weijiang.yang@intel.com> References: <20191119084949.15471-1-weijiang.yang@intel.com> Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Co-developed-by: yi.z.zhang@linux.intel.com Signed-off-by: yi.z.zhang@linux.intel.com Signed-off-by: Yang Weijiang --- Documentation/virt/kvm/api.txt | 46 +++++++ Documentation/virtual/kvm/spp_kvm.txt | 180 ++++++++++++++++++++++++++ 2 files changed, 226 insertions(+) create mode 100644 Documentation/virtual/kvm/spp_kvm.txt diff --git a/Documentation/virt/kvm/api.txt b/Documentation/virt/kvm/api.txt index 2d067767b617..85a764812f3d 100644 --- a/Documentation/virt/kvm/api.txt +++ b/Documentation/virt/kvm/api.txt @@ -4111,6 +4111,52 @@ Valid values for 'action': #define KVM_PMU_EVENT_ALLOW 0 #define KVM_PMU_EVENT_DENY 1 +4.121 KVM_INIT_SPP + +Architectures: x86 +Type: vm ioctl +Parameters: none +Returns: 0 on success, < 0 on error + +This ioctl initializes storage for sub-page permission vectors and +SPPT root page. + +4.121 KVM_SUBPAGES_GET_ACCESS + +Architectures: x86 +Type: vm ioctl +Parameters: struct kvm_subpage_info (in/out) +Returns: 0 on success, < 0 on error + +#define SUBPAGE_MAX_BITMAP 64 +struct kvm_subpage_info { + __u64 gfn; /* the first page gfn of the contiguous pages */ + __u64 npages; /* number of 4K pages */ + __u32 access_map[SUBPAGE_MAX_BITMAP]; /* sub-page write-access bitmap array */ +}; + +This ioctl fetches subpage permission from contiguous pages starting with +gfn. npages is the number of contiguous pages to fetch. access_map contains permission +vectors fetched for all the pages. + +4.122 KVM_SUBPAGES_SET_ACCESS + +Architectures: x86 +Type: vm ioctl +Parameters: struct kvm_subpage_info (in/out) +Returns: 0 on success, < 0 on error + +#define SUBPAGE_MAX_BITMAP 64 +struct kvm_subpage_info { + __u64 gfn; /* the first page gfn of the contiguous pages */ + __u64 npages; /* number of 4K pages */ + __u32 access_map[SUBPAGE_MAX_BITMAP]; /* sub-page write-access bitmap array */ +}; + +This ioctl sets subpage permission for contiguous pages starting with gfn. npages is +the number of contiguous pages to set. access_map contains permission vectors for all the +pages. Since during execution of the ioctl, it holds mmu_lock, so limits the MAX pages +to 64 to reduce the impact to EPT. 5. The kvm_run structure ------------------------ diff --git a/Documentation/virtual/kvm/spp_kvm.txt b/Documentation/virtual/kvm/spp_kvm.txt new file mode 100644 index 000000000000..2b49846c0722 --- /dev/null +++ b/Documentation/virtual/kvm/spp_kvm.txt @@ -0,0 +1,180 @@ +EPT-Based Sub-Page Protection (SPP) for KVM +==================================================== + +1.Overview + EPT-based Sub-Page Protection(SPP) allows VMM to specify + fine-grained(128byte per sub-page) write-protection for guest physical + memory. When it's enabled, the CPU enforces write-access permission + for the sub-pages within a 4KB page, if corresponding bit is set in + permission vector, write to sub-page region is allowed, otherwise, + it's prevented with a EPT violation. + + *Note*: In current implementation, SPP is exclusive with nested flag, + if it's on, SPP feature won't work. + +2.SPP Operation + Sub-Page Protection Table (SPPT) is introduced to manage sub-page + write-access permission. + + It is active when: + a) nested flag is turned off. + b) "sub-page write protection" VM-execution control is 1. + c) SPP is initialized with KVM_INIT_SPP ioctl. + d) Sub-page permissions are set with KVM_SUBPAGES_SET_ACCESS ioctl. + see below sections for details. + + __________________________________________________________________________ + + How SPP hardware works: + __________________________________________________________________________ + + Guest write access --> GPA --> Walk EPT --> EPT leaf entry -----| + |---------------------------------------------------------------| + |-> if VMexec_control.spp && ept_leaf_entry.spp_bit (bit 61) + | + |-> --> EPT legacy behavior + | + | + |-> --> if ept_leaf_entry.writable + | + |-> --> Ignore SPP + | + |-> --> GPA --> Walk SPP 4-level table--| + | + |------------<----------get-the-SPPT-point-from-VMCS-field-----<------| + | + Walk SPP L4E table + | + |---> if-entry-misconfiguration ------------>-------|-------<---------| + | | | + else | | + | | | + | |------------------SPP VMexit<-----------------| | + | | | + | |-> exit_qualification & sppt_misconfig --> sppt misconfig | + | | | + | |-> exit_qualification & sppt_miss --> sppt miss | + |---| | + | | + walk SPPT L3E--|--> if-entry-misconfiguration------------>------------| + | | + else | + | | + | | + walk SPPT L2E --|--> if-entry-misconfiguration-------->-------| + | | + else | + | | + | | + walk SPPT L1E --|-> if-entry-misconfiguration--->----| + | + else + | + |-> if sub-page writable + |-> allow, write access + |-> disallow, EPT violation + ______________________________________________________________________________ + +3.IOCTL Interfaces + + KVM_INIT_SPP: + Allocate storage for sub-page permission vectors and SPPT root page. + + KVM_SUBPAGES_GET_ACCESS: + Get sub-page write permission vectors for given contiguous guest pages. + + KVM_SUBPAGES_SET_ACCESS + Set SPP bit in EPT leaf entries for given contiguous guest pages. The + actual SPPT setup is triggered when SPP miss vm-exit is handled. + + #define SUBPAGE_MAX_BITMAP 64 + /* for KVM_SUBPAGES_GET_ACCESS and KVM_SUBPAGES_SET_ACCESS */ + struct kvm_subpage_info { + __u64 gfn; /* the first page gfn of the contiguous pages */ + __u64 npages; /* number of 4K pages */ + __u32 access_map[SUBPAGE_MAX_BITMAP]; /* sub-page write-access bitmap array */ + }; + + #define KVM_SUBPAGES_GET_ACCESS _IOR(KVMIO, 0x49, __u64) + #define KVM_SUBPAGES_SET_ACCESS _IOW(KVMIO, 0x4a, __u64) + #define KVM_INIT_SPP _IOW(KVMIO, 0x4b, __u64) + +4.Set Sub-Page Permission + + * To enable SPP protection, KVM user-space application sets sub-page permission + via KVM_SUBPAGES_SET_ACCESS ioctl: + (1) It first stores the access permissions in bitmap array. + + (2) Then, if the target 4KB pages are mapped as PT_PAGE_TABLE_LEVEL entry in EPT, + it sets SPP bit of the corresponding entry to mark sub-page protection. + If the 4KB pages are mapped within PT_DIRECTORY_LEVEL or PT_PDPE_LEVEL entry, + it first zaps the hugepage entries so as to let following memory access to trigger + EPT violation, there the gfn is check against SPP permission bitmap and + proper level is selected to set up EPT entry. + + + The SPPT paging structure format is as below: + + Format of the SPPT L4E, L3E, L2E: + | Bit | Contents | + | :----- | :------------------------------------------------------------------------| + | 0 | Valid entry when set; indicates whether the entry is present | + | 11:1 | Reserved (0) | + | N-1:12 | Physical address of 4KB aligned SPPT LX-1 Table referenced by this entry | + | 51:N | Reserved (0) | + | 63:52 | Reserved (0) | + Note: N is the physical address width supported by the processor. X is the page level + + Format of the SPPT L1E: + | Bit | Contents | + | :---- | :---------------------------------------------------------------- | + | 0+2i | Write permission for i-th 128 byte sub-page region. | + | 1+2i | Reserved (0). | + Note: 0<=i<=31 + +5.SPPT-induced VM exit + + * SPPT miss and misconfiguration induced VM exit + + A SPPT missing VM exit occurs when walk the SPPT, there is no SPPT + misconfiguration but a paging-structure entry is not + present in any of L4E/L3E/L2E entries. + + A SPPT misconfiguration VM exit occurs when reserved bits or unsupported values + are set in SPPT entry. + + *NOTE* SPPT miss and SPPT misconfigurations can occur only due to + "eligible" memory write, this excludes, e.g., guest paging structure, + please refer to SDM 28.2 for details of "non-eligible" cases. + + * SPP permission induced VM exit + SPP sub-page permission induced violation is reported as EPT violation + therefore causes VM exit. + +6.SPPT-induced VM exit handling + + #define EXIT_REASON_SPP 66 + + static int (*const kvm_vmx_exit_handlers[])(struct kvm_vcpu *vcpu) = { + ... + [EXIT_REASON_SPP] = handle_spp, + ... + }; + + New exit qualification for SPPT-induced vmexits. + + | Bit | Contents | + | :---- | :---------------------------------------------------------------- | + | 10:0 | Reserved (0). | + | 11 | SPPT VM exit type. Set for SPPT Miss, cleared for SPPT Misconfig. | + | 12 | NMI unblocking due to IRET | + | 63:13 | Reserved (0) | + + In addition to the exit qualification, guest linear address and guest + physical address fields will be reported. + + * SPPT miss and misconfiguration induced VM exit + Set up SPPT entries correctly. + + * SPP permission induced VM exit + This kind of VM exit is left to VMI tool to handle. From patchwork Tue Nov 19 08:49:42 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yang, Weijiang" X-Patchwork-Id: 11251565 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0097014E5 for ; Tue, 19 Nov 2019 08:48:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id DD669222DF for ; Tue, 19 Nov 2019 08:48:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727382AbfKSIr7 (ORCPT ); Tue, 19 Nov 2019 03:47:59 -0500 Received: from mga17.intel.com ([192.55.52.151]:26328 "EHLO mga17.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727351AbfKSIr6 (ORCPT ); Tue, 19 Nov 2019 03:47:58 -0500 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga107.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 19 Nov 2019 00:47:58 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.68,322,1569308400"; d="scan'208";a="196427002" Received: from unknown (HELO local-michael-cet-test.sh.intel.com) ([10.239.159.128]) by orsmga007.jf.intel.com with ESMTP; 19 Nov 2019 00:47:53 -0800 From: Yang Weijiang To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, pbonzini@redhat.com, jmattson@google.com, sean.j.christopherson@intel.com Cc: yu.c.zhang@linux.intel.com, alazar@bitdefender.com, edwin.zhai@intel.com, Yang Weijiang Subject: [PATCH v7 2/9] vmx: spp: Add control flags for Sub-Page Protection(SPP) Date: Tue, 19 Nov 2019 16:49:42 +0800 Message-Id: <20191119084949.15471-3-weijiang.yang@intel.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20191119084949.15471-1-weijiang.yang@intel.com> References: <20191119084949.15471-1-weijiang.yang@intel.com> Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Check SPP capability in MSR_IA32_VMX_PROCBASED_CTLS2, its 23-bit indicates SPP capability. Enable SPP feature bit in CPU capabilities bitmap if it's supported. Co-developed-by: He Chen Signed-off-by: He Chen Co-developed-by: Zhang Yi Signed-off-by: Zhang Yi Signed-off-by: Yang Weijiang --- arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/vmx.h | 1 + arch/x86/kernel/cpu/intel.c | 4 ++++ arch/x86/kvm/mmu.h | 2 ++ arch/x86/kvm/vmx/capabilities.h | 5 +++++ arch/x86/kvm/vmx/vmx.c | 9 +++++++++ 6 files changed, 22 insertions(+) diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h index e880f2408e29..ee2c76fdadf6 100644 --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -228,6 +228,7 @@ #define X86_FEATURE_FLEXPRIORITY ( 8*32+ 2) /* Intel FlexPriority */ #define X86_FEATURE_EPT ( 8*32+ 3) /* Intel Extended Page Table */ #define X86_FEATURE_VPID ( 8*32+ 4) /* Intel Virtual Processor ID */ +#define X86_FEATURE_SPP ( 8*32+ 5) /* Intel EPT-based Sub-Page Write Protection */ #define X86_FEATURE_VMMCALL ( 8*32+15) /* Prefer VMMCALL to VMCALL */ #define X86_FEATURE_XENPV ( 8*32+16) /* "" Xen paravirtual guest */ diff --git a/arch/x86/include/asm/vmx.h b/arch/x86/include/asm/vmx.h index a39136b0d509..e1137807affc 100644 --- a/arch/x86/include/asm/vmx.h +++ b/arch/x86/include/asm/vmx.h @@ -68,6 +68,7 @@ #define SECONDARY_EXEC_XSAVES 0x00100000 #define SECONDARY_EXEC_PT_USE_GPA 0x01000000 #define SECONDARY_EXEC_MODE_BASED_EPT_EXEC 0x00400000 +#define SECONDARY_EXEC_ENABLE_SPP 0x00800000 #define SECONDARY_EXEC_TSC_SCALING 0x02000000 #define PIN_BASED_EXT_INTR_MASK 0x00000001 diff --git a/arch/x86/kernel/cpu/intel.c b/arch/x86/kernel/cpu/intel.c index 8d6d92ebeb54..27617e522f01 100644 --- a/arch/x86/kernel/cpu/intel.c +++ b/arch/x86/kernel/cpu/intel.c @@ -503,6 +503,7 @@ static void detect_vmx_virtcap(struct cpuinfo_x86 *c) #define X86_VMX_FEATURE_PROC_CTLS2_EPT 0x00000002 #define X86_VMX_FEATURE_PROC_CTLS2_VPID 0x00000020 #define x86_VMX_FEATURE_EPT_CAP_AD 0x00200000 +#define X86_VMX_FEATURE_PROC_CTLS2_SPP 0x00800000 u32 vmx_msr_low, vmx_msr_high, msr_ctl, msr_ctl2; u32 msr_vpid_cap, msr_ept_cap; @@ -513,6 +514,7 @@ static void detect_vmx_virtcap(struct cpuinfo_x86 *c) clear_cpu_cap(c, X86_FEATURE_EPT); clear_cpu_cap(c, X86_FEATURE_VPID); clear_cpu_cap(c, X86_FEATURE_EPT_AD); + clear_cpu_cap(c, X86_FEATURE_SPP); rdmsr(MSR_IA32_VMX_PROCBASED_CTLS, vmx_msr_low, vmx_msr_high); msr_ctl = vmx_msr_high | vmx_msr_low; @@ -536,6 +538,8 @@ static void detect_vmx_virtcap(struct cpuinfo_x86 *c) } if (msr_ctl2 & X86_VMX_FEATURE_PROC_CTLS2_VPID) set_cpu_cap(c, X86_FEATURE_VPID); + if (msr_ctl2 & X86_VMX_FEATURE_PROC_CTLS2_SPP) + set_cpu_cap(c, X86_FEATURE_SPP); } } diff --git a/arch/x86/kvm/mmu.h b/arch/x86/kvm/mmu.h index 54c2a377795b..3c1423526a98 100644 --- a/arch/x86/kvm/mmu.h +++ b/arch/x86/kvm/mmu.h @@ -26,6 +26,8 @@ #define PT_PAGE_SIZE_MASK (1ULL << PT_PAGE_SIZE_SHIFT) #define PT_PAT_MASK (1ULL << 7) #define PT_GLOBAL_MASK (1ULL << 8) +#define PT_SPP_SHIFT 61 +#define PT_SPP_MASK (1ULL << PT_SPP_SHIFT) #define PT64_NX_SHIFT 63 #define PT64_NX_MASK (1ULL << PT64_NX_SHIFT) diff --git a/arch/x86/kvm/vmx/capabilities.h b/arch/x86/kvm/vmx/capabilities.h index d6664ee3d127..e3bde7a32123 100644 --- a/arch/x86/kvm/vmx/capabilities.h +++ b/arch/x86/kvm/vmx/capabilities.h @@ -241,6 +241,11 @@ static inline bool cpu_has_vmx_pml(void) return vmcs_config.cpu_based_2nd_exec_ctrl & SECONDARY_EXEC_ENABLE_PML; } +static inline bool cpu_has_vmx_ept_spp(void) +{ + return vmcs_config.cpu_based_2nd_exec_ctrl & SECONDARY_EXEC_ENABLE_SPP; +} + static inline bool vmx_xsaves_supported(void) { return vmcs_config.cpu_based_2nd_exec_ctrl & diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index c030c96fc81a..0fcd02b531ef 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -60,6 +60,7 @@ #include "vmcs12.h" #include "vmx.h" #include "x86.h" +#include "spp.h" MODULE_AUTHOR("Qumranet"); MODULE_LICENSE("GPL"); @@ -113,6 +114,7 @@ module_param_named(pml, enable_pml, bool, S_IRUGO); static bool __read_mostly dump_invalid_vmcs = 0; module_param(dump_invalid_vmcs, bool, 0644); +static bool __read_mostly spp_supported = 0; #define MSR_BITMAP_MODE_X2APIC 1 #define MSR_BITMAP_MODE_X2APIC_APICV 2 @@ -2279,6 +2281,7 @@ static __init int setup_vmcs_config(struct vmcs_config *vmcs_conf, SECONDARY_EXEC_RDSEED_EXITING | SECONDARY_EXEC_RDRAND_EXITING | SECONDARY_EXEC_ENABLE_PML | + SECONDARY_EXEC_ENABLE_SPP | SECONDARY_EXEC_TSC_SCALING | SECONDARY_EXEC_PT_USE_GPA | SECONDARY_EXEC_PT_CONCEAL_VMX | @@ -3931,6 +3934,9 @@ static void vmx_compute_secondary_exec_control(struct vcpu_vmx *vmx) if (!enable_pml) exec_control &= ~SECONDARY_EXEC_ENABLE_PML; + if (!spp_supported) + exec_control &= ~SECONDARY_EXEC_ENABLE_SPP; + if (vmx_xsaves_supported()) { /* Exposing XSAVES only when XSAVE is exposed */ bool xsaves_enabled = @@ -7521,6 +7527,9 @@ static __init int hardware_setup(void) if (!cpu_has_vmx_flexpriority()) flexpriority_enabled = 0; + if (cpu_has_vmx_ept_spp() && enable_ept) + spp_supported = 1; + if (!cpu_has_virtual_nmis()) enable_vnmi = 0; From patchwork Tue Nov 19 08:49:43 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yang, Weijiang" X-Patchwork-Id: 11251563 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7678B14C0 for ; Tue, 19 Nov 2019 08:48:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 53468222EF for ; Tue, 19 Nov 2019 08:48:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727432AbfKSIsB (ORCPT ); Tue, 19 Nov 2019 03:48:01 -0500 Received: from mga17.intel.com ([192.55.52.151]:26328 "EHLO mga17.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727409AbfKSIsA (ORCPT ); Tue, 19 Nov 2019 03:48:00 -0500 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga107.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 19 Nov 2019 00:48:00 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.68,322,1569308400"; d="scan'208";a="196427006" Received: from unknown (HELO local-michael-cet-test.sh.intel.com) ([10.239.159.128]) by orsmga007.jf.intel.com with ESMTP; 19 Nov 2019 00:47:58 -0800 From: Yang Weijiang To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, pbonzini@redhat.com, jmattson@google.com, sean.j.christopherson@intel.com Cc: yu.c.zhang@linux.intel.com, alazar@bitdefender.com, edwin.zhai@intel.com, Yang Weijiang Subject: [PATCH v7 3/9] mmu: spp: Add SPP Table setup functions Date: Tue, 19 Nov 2019 16:49:43 +0800 Message-Id: <20191119084949.15471-4-weijiang.yang@intel.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20191119084949.15471-1-weijiang.yang@intel.com> References: <20191119084949.15471-1-weijiang.yang@intel.com> Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org SPPT is a 4-level paging structure similar to EPT, when SPP is armed for target physical page, bit 61 of the corresponding EPT entry is flaged, then SPPT is traversed with the gfn, the leaf entry of SPPT contains the access bitmap of subpages inside the target 4KB physical page, one bit per 128-byte subpage. Co-developed-by: He Chen Signed-off-by: He Chen Co-developed-by: Zhang Yi Signed-off-by: Zhang Yi Signed-off-by: Yang Weijiang --- arch/x86/include/asm/kvm_host.h | 4 +- arch/x86/kvm/vmx/spp.c | 239 ++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/spp.h | 10 ++ 3 files changed, 252 insertions(+), 1 deletion(-) create mode 100644 arch/x86/kvm/vmx/spp.c create mode 100644 arch/x86/kvm/vmx/spp.h diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index bdc16b0aa7c6..eb18f4dd993d 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -270,7 +270,8 @@ union kvm_mmu_page_role { unsigned smap_andnot_wp:1; unsigned ad_disabled:1; unsigned guest_mode:1; - unsigned :6; + unsigned spp:1; + unsigned reserved:5; /* * This is left at the top of the word so that @@ -399,6 +400,7 @@ struct kvm_mmu { u64 *spte, const void *pte); hpa_t root_hpa; gpa_t root_cr3; + hpa_t sppt_root; union kvm_mmu_role mmu_role; u8 root_level; u8 shadow_root_level; diff --git a/arch/x86/kvm/vmx/spp.c b/arch/x86/kvm/vmx/spp.c new file mode 100644 index 000000000000..27fef18fcc4e --- /dev/null +++ b/arch/x86/kvm/vmx/spp.c @@ -0,0 +1,239 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include "spp.h" + +#define for_each_shadow_spp_entry(_vcpu, _addr, _walker) \ + for (shadow_spp_walk_init(&(_walker), _vcpu, _addr); \ + shadow_walk_okay(&(_walker)); \ + shadow_walk_next(&(_walker))) + +static void shadow_spp_walk_init(struct kvm_shadow_walk_iterator *iterator, + struct kvm_vcpu *vcpu, u64 addr) +{ + iterator->addr = addr; + iterator->shadow_addr = vcpu->arch.mmu->sppt_root; + + /* SPP Table is a 4-level paging structure */ + iterator->level = PT64_ROOT_4LEVEL; +} + +static int is_spp_shadow_present(u64 pte) +{ + return pte & PT_PRESENT_MASK; +} + +static bool __rmap_open_subpage_bit(struct kvm *kvm, + struct kvm_rmap_head *rmap_head) +{ + struct rmap_iterator iter; + bool flush = false; + u64 *sptep; + u64 spte; + + for_each_rmap_spte(rmap_head, &iter, sptep) { + /* + * SPP works only when the page is write-protected + * and SPP bit is set in EPT leaf entry. + */ + flush |= spte_write_protect(sptep, false); + spte = *sptep | PT_SPP_MASK; + flush |= mmu_spte_update(sptep, spte); + } + + return flush; +} + +static int kvm_spp_open_write_protect(struct kvm *kvm, + struct kvm_memory_slot *slot, + gfn_t gfn) +{ + struct kvm_rmap_head *rmap_head; + bool flush = false; + + /* + * SPP is only supported with 4KB level1 memory page, check + * if the page is mapped in EPT leaf entry. + */ + rmap_head = __gfn_to_rmap(gfn, PT_PAGE_TABLE_LEVEL, slot); + + if (!rmap_head->val) + return -EFAULT; + + flush |= __rmap_open_subpage_bit(kvm, rmap_head); + + if (flush) + kvm_flush_remote_tlbs(kvm); + + return 0; +} + +static bool __rmap_clear_subpage_bit(struct kvm *kvm, + struct kvm_rmap_head *rmap_head) +{ + struct rmap_iterator iter; + bool flush = false; + u64 *sptep; + u64 spte; + + for_each_rmap_spte(rmap_head, &iter, sptep) { + spte = (*sptep & ~PT_SPP_MASK); + flush |= mmu_spte_update(sptep, spte); + } + + return flush; +} + +static int kvm_spp_clear_write_protect(struct kvm *kvm, + struct kvm_memory_slot *slot, + gfn_t gfn) +{ + struct kvm_rmap_head *rmap_head; + bool flush = false; + + rmap_head = __gfn_to_rmap(gfn, PT_PAGE_TABLE_LEVEL, slot); + + if (!rmap_head->val) + return -EFAULT; + + flush |= __rmap_clear_subpage_bit(kvm, rmap_head); + + if (flush) + kvm_flush_remote_tlbs(kvm); + + return 0; +} + +struct kvm_mmu_page *kvm_spp_get_page(struct kvm_vcpu *vcpu, + gfn_t gfn, + unsigned int level) +{ + struct kvm_mmu_page *sp; + union kvm_mmu_page_role role; + + role = vcpu->arch.mmu->mmu_role.base; + role.level = level; + role.direct = true; + role.spp = true; + + for_each_valid_sp(vcpu->kvm, sp, gfn) { + if (sp->gfn != gfn) + continue; + if (sp->role.word != role.word) + continue; + if (sp->role.spp && sp->role.level == level) + goto out; + } + + sp = kvm_mmu_alloc_page(vcpu, true); + sp->gfn = gfn; + sp->role = role; + hlist_add_head(&sp->hash_link, + &vcpu->kvm->arch.mmu_page_hash + [kvm_page_table_hashfn(gfn)]); + sp->mmu_valid_gen = vcpu->kvm->arch.mmu_valid_gen; + clear_page(sp->spt); +out: + return sp; +} +EXPORT_SYMBOL_GPL(kvm_spp_get_page); + +static void link_spp_shadow_page(struct kvm_vcpu *vcpu, u64 *sptep, + struct kvm_mmu_page *sp) +{ + u64 spte; + + spte = __pa(sp->spt) | PT_PRESENT_MASK; + + mmu_spte_set(sptep, spte); + + mmu_page_add_parent_pte(vcpu, sp, sptep); +} + +static u64 format_spp_spte(u32 spp_wp_bitmap) +{ + u64 new_spte = 0; + int i = 0; + + /* + * One 4K page contains 32 sub-pages, in SPP table L4E, old bits + * are reserved, so we need to transfer u32 subpage write + * protect bitmap to u64 SPP L4E format. + */ + while (i < 32) { + if (spp_wp_bitmap & (1ULL << i)) + new_spte |= 1ULL << (i * 2); + + i++; + } + + return new_spte; +} + +static void spp_spte_set(u64 *sptep, u64 new_spte) +{ + __set_spte(sptep, new_spte); +} + +bool is_spp_spte(struct kvm_mmu_page *sp) +{ + return sp->role.spp; +} + +#define SPPT_ENTRY_PHA_MASK (0xFFFFFFFFFF << 12) + +int kvm_spp_setup_structure(struct kvm_vcpu *vcpu, + u32 access_map, gfn_t gfn) +{ + struct kvm_shadow_walk_iterator iter; + struct kvm_mmu_page *sp; + gfn_t pseudo_gfn; + u64 old_spte, spp_spte; + int ret = -EFAULT; + + /* direct_map spp start */ + if (!VALID_PAGE(vcpu->arch.mmu->sppt_root)) + return -EFAULT; + + for_each_shadow_spp_entry(vcpu, (u64)gfn << PAGE_SHIFT, iter) { + if (iter.level == PT_PAGE_TABLE_LEVEL) { + spp_spte = format_spp_spte(access_map); + old_spte = mmu_spte_get_lockless(iter.sptep); + if (old_spte != spp_spte) { + spp_spte_set(iter.sptep, spp_spte); + //kvm_make_request(KVM_REQ_TLB_FLUSH, vcpu); + kvm_flush_remote_tlbs(vcpu->kvm); + } + ret = 0; + break; + } + + if (!is_spp_shadow_present(*iter.sptep)) { + u64 base_addr = iter.addr; + + base_addr &= PT64_LVL_ADDR_MASK(iter.level); + pseudo_gfn = base_addr >> PAGE_SHIFT; + + spp_spte = *iter.sptep; + if (iter.level == PT_DIRECTORY_LEVEL && + !(spp_spte & PT_PRESENT_MASK) && + (spp_spte & SPPT_ENTRY_PHA_MASK)) { + spp_spte |= PT_PRESENT_MASK; + spp_spte_set(iter.sptep, spp_spte); + continue; + } + sp = kvm_spp_get_page(vcpu, pseudo_gfn, + iter.level - 1); + link_spp_shadow_page(vcpu, iter.sptep, sp); + } + } + + return ret; +} +EXPORT_SYMBOL_GPL(kvm_spp_setup_structure); + +inline u64 construct_spptp(unsigned long root_hpa) +{ + return root_hpa & PAGE_MASK; +} +EXPORT_SYMBOL_GPL(construct_spptp); + diff --git a/arch/x86/kvm/vmx/spp.h b/arch/x86/kvm/vmx/spp.h new file mode 100644 index 000000000000..25a23a4277eb --- /dev/null +++ b/arch/x86/kvm/vmx/spp.h @@ -0,0 +1,10 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __KVM_X86_VMX_SPP_H +#define __KVM_X86_VMX_SPP_H + +bool is_spp_spte(struct kvm_mmu_page *sp); +inline u64 construct_spptp(unsigned long root_hpa); +int kvm_spp_setup_structure(struct kvm_vcpu *vcpu, + u32 access_map, gfn_t gfn); + +#endif /* __KVM_X86_VMX_SPP_H */ From patchwork Tue Nov 19 08:49:44 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yang, Weijiang" X-Patchwork-Id: 11251549 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 270FA14C0 for ; Tue, 19 Nov 2019 08:48:06 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id F27AD21D7B for ; Tue, 19 Nov 2019 08:48:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727474AbfKSIsE (ORCPT ); Tue, 19 Nov 2019 03:48:04 -0500 Received: from mga17.intel.com ([192.55.52.151]:26340 "EHLO mga17.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727455AbfKSIsD (ORCPT ); Tue, 19 Nov 2019 03:48:03 -0500 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga107.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 19 Nov 2019 00:48:03 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.68,322,1569308400"; d="scan'208";a="196427030" Received: from unknown (HELO local-michael-cet-test.sh.intel.com) ([10.239.159.128]) by orsmga007.jf.intel.com with ESMTP; 19 Nov 2019 00:48:00 -0800 From: Yang Weijiang To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, pbonzini@redhat.com, jmattson@google.com, sean.j.christopherson@intel.com Cc: yu.c.zhang@linux.intel.com, alazar@bitdefender.com, edwin.zhai@intel.com, Yang Weijiang Subject: [PATCH v7 4/9] mmu: spp: Add functions to create/destroy SPP bitmap block Date: Tue, 19 Nov 2019 16:49:44 +0800 Message-Id: <20191119084949.15471-5-weijiang.yang@intel.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20191119084949.15471-1-weijiang.yang@intel.com> References: <20191119084949.15471-1-weijiang.yang@intel.com> Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Create access bitmap for SPP subpages, the bitmap can be accessed with a gfn. The initial access bitmap for each physical page is 0xFFFFFFFF, meaning SPP is not enabled for the subpages. Co-developed-by: He Chen Signed-off-by: He Chen Co-developed-by: Zhang Yi Signed-off-by: Zhang Yi Signed-off-by: Yang Weijiang --- arch/x86/include/asm/kvm_host.h | 3 + arch/x86/kvm/vmx/spp.c | 311 +++++++++++++++++++++++++++++++- arch/x86/kvm/vmx/spp.h | 7 + include/uapi/linux/kvm.h | 9 + 4 files changed, 325 insertions(+), 5 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index eb18f4dd993d..cc38670a0c45 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -799,6 +799,7 @@ struct kvm_lpage_info { struct kvm_arch_memory_slot { struct kvm_rmap_head *rmap[KVM_NR_PAGE_SIZES]; + u32 *subpage_wp_info; struct kvm_lpage_info *lpage_info[KVM_NR_PAGE_SIZES - 1]; unsigned short *gfn_track[KVM_PAGE_TRACK_MAX]; }; @@ -939,6 +940,8 @@ struct kvm_arch { bool exception_payload_enabled; struct kvm_pmu_event_filter *pmu_event_filter; + bool spp_active; + }; struct kvm_vm_stat { diff --git a/arch/x86/kvm/vmx/spp.c b/arch/x86/kvm/vmx/spp.c index 27fef18fcc4e..964d31fe9426 100644 --- a/arch/x86/kvm/vmx/spp.c +++ b/arch/x86/kvm/vmx/spp.c @@ -22,6 +22,15 @@ static int is_spp_shadow_present(u64 pte) return pte & PT_PRESENT_MASK; } +u32 *gfn_to_subpage_wp_info(struct kvm_memory_slot *slot, gfn_t gfn) +{ + unsigned long idx; + + idx = gfn_to_index(gfn, slot->base_gfn, PT_PAGE_TABLE_LEVEL); + return &slot->arch.subpage_wp_info[idx]; +} +EXPORT_SYMBOL_GPL(gfn_to_subpage_wp_info); + static bool __rmap_open_subpage_bit(struct kvm *kvm, struct kvm_rmap_head *rmap_head) { @@ -179,6 +188,24 @@ bool is_spp_spte(struct kvm_mmu_page *sp) return sp->role.spp; } +/* + * all vcpus share the same SPPT, vcpu->arch.mmu->sppt_root points to same + * SPPT root page, so any vcpu will do. + */ +static struct kvm_vcpu *kvm_spp_get_vcpu(struct kvm *kvm) +{ + struct kvm_vcpu *vcpu = NULL; + int idx; + + for (idx = 0; idx < atomic_read(&kvm->online_vcpus); idx++) { + vcpu = kvm_get_vcpu(kvm, idx); + if (vcpu) + break; + } + + return vcpu; +} + #define SPPT_ENTRY_PHA_MASK (0xFFFFFFFFFF << 12) int kvm_spp_setup_structure(struct kvm_vcpu *vcpu, @@ -198,11 +225,8 @@ int kvm_spp_setup_structure(struct kvm_vcpu *vcpu, if (iter.level == PT_PAGE_TABLE_LEVEL) { spp_spte = format_spp_spte(access_map); old_spte = mmu_spte_get_lockless(iter.sptep); - if (old_spte != spp_spte) { + if (old_spte != spp_spte) spp_spte_set(iter.sptep, spp_spte); - //kvm_make_request(KVM_REQ_TLB_FLUSH, vcpu); - kvm_flush_remote_tlbs(vcpu->kvm); - } ret = 0; break; } @@ -226,11 +250,288 @@ int kvm_spp_setup_structure(struct kvm_vcpu *vcpu, link_spp_shadow_page(vcpu, iter.sptep, sp); } } - + kvm_flush_remote_tlbs(vcpu->kvm); return ret; } EXPORT_SYMBOL_GPL(kvm_spp_setup_structure); +int vmx_spp_flush_sppt(struct kvm *kvm, struct kvm_subpage *spp_info) +{ + struct kvm_shadow_walk_iterator iter; + struct kvm_vcpu *vcpu; + gfn_t gfn = spp_info->base_gfn; + int npages = spp_info->npages; + u64 spde; + int i; + + vcpu = kvm_spp_get_vcpu(kvm); + /* direct_map spp start */ + if (!VALID_PAGE(vcpu->arch.mmu->sppt_root)) + return -EFAULT; + + for (i = 0; i < npages; ++i) { + for_each_shadow_spp_entry(vcpu, (u64)gfn << PAGE_SHIFT, iter) { + if (!is_spp_shadow_present(*iter.sptep)) + break; + + if (iter.level == PT_DIRECTORY_LEVEL) { + spde = *iter.sptep; + spde &= ~PT_PRESENT_MASK; + spp_spte_set(iter.sptep, spde); + break; + } + } + gfn++; + } + kvm_flush_remote_tlbs(kvm); + return 0; +} +EXPORT_SYMBOL_GPL(vmx_spp_flush_sppt); + +static int kvm_spp_create_bitmaps(struct kvm *kvm) +{ + struct kvm_memslots *slots; + struct kvm_memory_slot *memslot; + int i, j, ret; + u32 *buff; + + for (i = 0; i < KVM_ADDRESS_SPACE_NUM; i++) { + slots = __kvm_memslots(kvm, i); + kvm_for_each_memslot(memslot, slots) { + buff = kvzalloc(memslot->npages * + sizeof(*memslot->arch.subpage_wp_info), + GFP_KERNEL); + + if (!buff) { + ret = -ENOMEM; + goto out_free; + } + memslot->arch.subpage_wp_info = buff; + + for (j = 0; j < memslot->npages; j++) + buff[j] = FULL_SPP_ACCESS; + } + } + + return 0; +out_free: + for (i = 0; i < KVM_ADDRESS_SPACE_NUM; i++) { + slots = __kvm_memslots(kvm, i); + kvm_for_each_memslot(memslot, slots) { + if (memslot->arch.subpage_wp_info) { + kvfree(memslot->arch.subpage_wp_info); + memslot->arch.subpage_wp_info = NULL; + } + } + } + + return ret; +} + +int vmx_spp_init(struct kvm *kvm) +{ + int i, ret; + struct kvm_vcpu *vcpu; + int root_level; + struct kvm_mmu_page *ssp_sp; + bool first_root = true; + + /* SPP feature is exclusive with nested VM.*/ + if (kvm_x86_ops->get_nested_state) + return -EPERM; + + if (kvm->arch.spp_active) + return 0; + + ret = kvm_spp_create_bitmaps(kvm); + + if (ret) + return ret; + + kvm_for_each_vcpu(i, vcpu, kvm) { + if (first_root) { + /* prepare caches for SPP setup.*/ + mmu_topup_memory_caches(vcpu); + root_level = vcpu->arch.mmu->shadow_root_level; + ssp_sp = kvm_spp_get_page(vcpu, 0, root_level); + first_root = false; + } + ++ssp_sp->root_count; + vcpu->arch.mmu->sppt_root = __pa(ssp_sp->spt); + kvm_make_request(KVM_REQ_LOAD_CR3, vcpu); + } + + kvm->arch.spp_active = true; + return 0; +} +EXPORT_SYMBOL_GPL(vmx_spp_init); + +int kvm_spp_get_permission(struct kvm *kvm, struct kvm_subpage *spp_info) +{ + u32 *access = spp_info->access_map; + gfn_t gfn = spp_info->base_gfn; + int npages = spp_info->npages; + struct kvm_memory_slot *slot; + int i; + + if (!kvm->arch.spp_active) + return -ENODEV; + + for (i = 0; i < npages; i++, gfn++) { + slot = gfn_to_memslot(kvm, gfn); + if (!slot) + return -EFAULT; + access[i] = *gfn_to_subpage_wp_info(slot, gfn); + } + + return i; +} +EXPORT_SYMBOL_GPL(kvm_spp_get_permission); + +static void kvm_spp_zap_pte(struct kvm *kvm, u64 *spte, int level) +{ + u64 pte; + + pte = *spte; + if (is_shadow_present_pte(pte) && is_last_spte(pte, level)) { + drop_spte(kvm, spte); + if (is_large_pte(pte)) + --kvm->stat.lpages; + } +} + +int kvm_spp_zap_entry(struct kvm *kvm, gfn_t gfn_lower, gfn_t gfn_upper, + u64 *sptep, int level) +{ + int page_num = KVM_PAGES_PER_HPAGE(level); + gfn_t gfn_max = (gfn_lower & ~(page_num - 1)) + page_num - 1; + int ret; + + if (gfn_upper <= gfn_max) + ret = gfn_upper - gfn_lower + 1; + else + ret = gfn_max - gfn_lower + 1; + + kvm_spp_zap_pte(kvm, sptep, level); + kvm_flush_remote_tlbs(kvm); + + return ret; +} + +int kvm_spp_set_permission(struct kvm *kvm, struct kvm_subpage *spp_info) +{ + u32 *access = spp_info->access_map; + gfn_t gfn = spp_info->base_gfn; + int npages = spp_info->npages; + struct kvm_memory_slot *slot; + struct kvm_subpage sbp = {0}; + struct kvm_shadow_walk_iterator iterator; + struct kvm_vcpu *vcpu; + gfn_t max_gfn; + gfn_t old_gfn = gfn; + u32 *wp_map; + int i, count; + + if (!kvm->arch.spp_active) + return -ENODEV; + + if (npages > SUBPAGE_MAX_BITMAP) + return -EFAULT; + + for (i = 0; i < npages; i++, gfn++) { + slot = gfn_to_memslot(kvm, gfn); + if (!slot) + return -EFAULT; + + wp_map = gfn_to_subpage_wp_info(slot, gfn); + *wp_map = access[i]; + } + + gfn = old_gfn; + max_gfn = gfn + npages - 1; + vcpu = kvm_spp_get_vcpu(kvm); + + if (!vcpu || (vcpu && !VALID_PAGE(vcpu->arch.mmu->root_hpa))) + goto out; + + for (i = 0; gfn <= max_gfn; i++, gfn++) { + for_each_shadow_entry(vcpu, (u64)gfn << PAGE_SHIFT, iterator) { + if (!is_shadow_present_pte(*iterator.sptep)) + break; + + if (iterator.level == PT_PAGE_TABLE_LEVEL) { + sbp.base_gfn = gfn; + sbp.access_map[0] = access[i]; + sbp.npages = 1; + if (kvm_spp_mark_protection(kvm, &sbp) < 0) + return -EFAULT; + break; + } else if (is_large_pte(*iterator.sptep)) { + count = kvm_spp_zap_entry(kvm, gfn, max_gfn, + iterator.sptep, + iterator.level); + if (count >= npages) + goto out; + gfn += count - 1; + } + } + } +out: + return npages; +} + +int kvm_spp_mark_protection(struct kvm *kvm, struct kvm_subpage *spp_info) +{ + u32 *access = spp_info->access_map; + gfn_t gfn = spp_info->base_gfn; + struct kvm_memory_slot *slot; + struct kvm_rmap_head *rmap_head; + int ret; + + if (!kvm->arch.spp_active) + return -ENODEV; + + slot = gfn_to_memslot(kvm, gfn); + if (!slot) + return -EFAULT; + + /* + * check whether the target 4KB page exists in EPT leaf + * entry.If it's there, just flag SPP bit of the entry, + * defer the setup to SPPT miss induced vm-exit handler. + */ + rmap_head = __gfn_to_rmap(gfn, PT_PAGE_TABLE_LEVEL, slot); + + if (rmap_head->val) { + /* + * if all subpages are not writable, open SPP bit in + * EPT leaf entry to enable SPP protection for + * corresponding page. + */ + if (access[0] != FULL_SPP_ACCESS) { + ret = kvm_spp_open_write_protect(kvm, slot, gfn); + if (ret) + return ret; + } else { + ret = kvm_spp_clear_write_protect(kvm, slot, gfn); + if (ret) + return ret; + } + } + + return 0; +} + +void kvm_spp_free_memslot(struct kvm_memory_slot *free, + struct kvm_memory_slot *dont) +{ + if (!dont || free->arch.subpage_wp_info != + dont->arch.subpage_wp_info) { + kvfree(free->arch.subpage_wp_info); + free->arch.subpage_wp_info = NULL; + } +} + inline u64 construct_spptp(unsigned long root_hpa) { return root_hpa & PAGE_MASK; diff --git a/arch/x86/kvm/vmx/spp.h b/arch/x86/kvm/vmx/spp.h index 25a23a4277eb..56b287ec15fd 100644 --- a/arch/x86/kvm/vmx/spp.h +++ b/arch/x86/kvm/vmx/spp.h @@ -2,9 +2,16 @@ #ifndef __KVM_X86_VMX_SPP_H #define __KVM_X86_VMX_SPP_H +#define FULL_SPP_ACCESS ((u32)((1ULL << 32) - 1)) + bool is_spp_spte(struct kvm_mmu_page *sp); inline u64 construct_spptp(unsigned long root_hpa); int kvm_spp_setup_structure(struct kvm_vcpu *vcpu, u32 access_map, gfn_t gfn); +int vmx_spp_flush_sppt(struct kvm *kvm, struct kvm_subpage *spp_info); +void kvm_spp_free_memslot(struct kvm_memory_slot *free, + struct kvm_memory_slot *dont); +int vmx_spp_init(struct kvm *kvm); +u32 *gfn_to_subpage_wp_info(struct kvm_memory_slot *slot, gfn_t gfn); #endif /* __KVM_X86_VMX_SPP_H */ diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h index 5e3f12d5359e..9460830de536 100644 --- a/include/uapi/linux/kvm.h +++ b/include/uapi/linux/kvm.h @@ -102,6 +102,15 @@ struct kvm_userspace_memory_region { __u64 userspace_addr; /* start of the userspace allocated memory */ }; +/* for KVM_SUBPAGES_GET_ACCESS and KVM_SUBPAGES_SET_ACCESS */ +#define SUBPAGE_MAX_BITMAP 64 +struct kvm_subpage { + __u64 base_gfn; + __u64 npages; + /* sub-page write-access bitmap array */ + __u32 access_map[SUBPAGE_MAX_BITMAP]; +}; + /* * The bit 0 ~ bit 15 of kvm_memory_region::flags are visible for userspace, * other bits are reserved for kvm internal use which are defined in From patchwork Tue Nov 19 08:49:45 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yang, Weijiang" X-Patchwork-Id: 11251551 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3CBD514E5 for ; Tue, 19 Nov 2019 08:48:11 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 1BFCF222DF for ; Tue, 19 Nov 2019 08:48:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727509AbfKSIsG (ORCPT ); Tue, 19 Nov 2019 03:48:06 -0500 Received: from mga17.intel.com ([192.55.52.151]:26340 "EHLO mga17.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727495AbfKSIsF (ORCPT ); Tue, 19 Nov 2019 03:48:05 -0500 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga107.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 19 Nov 2019 00:48:05 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.68,322,1569308400"; d="scan'208";a="196427072" Received: from unknown (HELO local-michael-cet-test.sh.intel.com) ([10.239.159.128]) by orsmga007.jf.intel.com with ESMTP; 19 Nov 2019 00:48:03 -0800 From: Yang Weijiang To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, pbonzini@redhat.com, jmattson@google.com, sean.j.christopherson@intel.com Cc: yu.c.zhang@linux.intel.com, alazar@bitdefender.com, edwin.zhai@intel.com, Yang Weijiang Subject: [PATCH v7 5/9] x86: spp: Introduce user-space SPP IOCTLs Date: Tue, 19 Nov 2019 16:49:45 +0800 Message-Id: <20191119084949.15471-6-weijiang.yang@intel.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20191119084949.15471-1-weijiang.yang@intel.com> References: <20191119084949.15471-1-weijiang.yang@intel.com> Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org User application, e.g., QEMU or VMI, must initialize SPP before gets/sets SPP subpages, the dynamic initialization is to reduce the extra storage cost if the SPP feature is not not used. Co-developed-by: He Chen Signed-off-by: He Chen Co-developed-by: Zhang Yi Signed-off-by: Zhang Yi Signed-off-by: Yang Weijiang --- arch/x86/include/asm/kvm_host.h | 3 ++ arch/x86/kvm/vmx/spp.c | 54 +++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/spp.h | 5 +++ arch/x86/kvm/vmx/vmx.c | 8 +++++ arch/x86/kvm/x86.c | 49 ++++++++++++++++++++++++++++++ include/uapi/linux/kvm.h | 3 ++ 6 files changed, 122 insertions(+) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index cc38670a0c45..3863eb3c0e6a 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1216,6 +1216,9 @@ struct kvm_x86_ops { uint16_t (*nested_get_evmcs_version)(struct kvm_vcpu *vcpu); bool (*need_emulation_on_page_fault)(struct kvm_vcpu *vcpu); + + int (*init_spp)(struct kvm *kvm); + int (*flush_subpages)(struct kvm *kvm, struct kvm_subpage *spp_info); }; struct kvm_arch_async_pf { diff --git a/arch/x86/kvm/vmx/spp.c b/arch/x86/kvm/vmx/spp.c index 964d31fe9426..b5327faf512e 100644 --- a/arch/x86/kvm/vmx/spp.c +++ b/arch/x86/kvm/vmx/spp.c @@ -538,3 +538,57 @@ inline u64 construct_spptp(unsigned long root_hpa) } EXPORT_SYMBOL_GPL(construct_spptp); +int kvm_vm_ioctl_get_subpages(struct kvm *kvm, + struct kvm_subpage *spp_info) +{ + int ret; + + mutex_lock(&kvm->slots_lock); + ret = kvm_spp_get_permission(kvm, spp_info); + mutex_unlock(&kvm->slots_lock); + + return ret; +} +EXPORT_SYMBOL_GPL(kvm_vm_ioctl_get_subpages); + +int kvm_vm_ioctl_set_subpages(struct kvm *kvm, + struct kvm_subpage *spp_info) +{ + int ret; + + if (!kvm_x86_ops->flush_subpages) + return -EINVAL; + + spin_lock(&kvm->mmu_lock); + ret = kvm_x86_ops->flush_subpages(kvm, spp_info); + spin_unlock(&kvm->mmu_lock); + + if (ret < 0) + return ret; + + mutex_lock(&kvm->slots_lock); + spin_lock(&kvm->mmu_lock); + + ret = kvm_spp_set_permission(kvm, spp_info); + + spin_unlock(&kvm->mmu_lock); + mutex_unlock(&kvm->slots_lock); + + return ret; +} +EXPORT_SYMBOL_GPL(kvm_vm_ioctl_set_subpages); + +int kvm_vm_ioctl_init_spp(struct kvm *kvm) +{ + int ret; + + if (!kvm_x86_ops->init_spp) + return -ENODEV; + + mutex_lock(&kvm->slots_lock); + ret = kvm_x86_ops->init_spp(kvm); + mutex_unlock(&kvm->slots_lock); + + return ret; +} +EXPORT_SYMBOL_GPL(kvm_vm_ioctl_init_spp); diff --git a/arch/x86/kvm/vmx/spp.h b/arch/x86/kvm/vmx/spp.h index 56b287ec15fd..ad13a35bfca2 100644 --- a/arch/x86/kvm/vmx/spp.h +++ b/arch/x86/kvm/vmx/spp.h @@ -6,6 +6,11 @@ bool is_spp_spte(struct kvm_mmu_page *sp); inline u64 construct_spptp(unsigned long root_hpa); +int kvm_vm_ioctl_get_subpages(struct kvm *kvm, + struct kvm_subpage *spp_info); +int kvm_vm_ioctl_set_subpages(struct kvm *kvm, + struct kvm_subpage *spp_info); +int kvm_vm_ioctl_init_spp(struct kvm *kvm); int kvm_spp_setup_structure(struct kvm_vcpu *vcpu, u32 access_map, gfn_t gfn); int vmx_spp_flush_sppt(struct kvm *kvm, struct kvm_subpage *spp_info); diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 0fcd02b531ef..af296a290173 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -7596,6 +7596,11 @@ static __init int hardware_setup(void) kvm_x86_ops->enable_log_dirty_pt_masked = NULL; } + if (!spp_supported) { + kvm_x86_ops->flush_subpages = NULL; + kvm_x86_ops->init_spp = NULL; + } + if (!cpu_has_vmx_preemption_timer()) enable_preemption_timer = false; @@ -7808,6 +7813,9 @@ static struct kvm_x86_ops vmx_x86_ops __ro_after_init = { .nested_enable_evmcs = NULL, .nested_get_evmcs_version = NULL, .need_emulation_on_page_fault = vmx_need_emulation_on_page_fault, + + .flush_subpages = vmx_spp_flush_sppt, + .init_spp = vmx_spp_init, }; static void vmx_cleanup_l1d_flush(void) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 91602d310a3f..78605d75690e 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -26,6 +26,7 @@ #include "cpuid.h" #include "pmu.h" #include "hyperv.h" +#include "vmx/spp.h" #include #include @@ -4977,6 +4978,54 @@ long kvm_arch_vm_ioctl(struct file *filp, case KVM_SET_PMU_EVENT_FILTER: r = kvm_vm_ioctl_set_pmu_event_filter(kvm, argp); break; + case KVM_SUBPAGES_GET_ACCESS: { + struct kvm_subpage spp_info; + + if (!kvm->arch.spp_active) { + r = -ENODEV; + goto out; + } + + r = -EFAULT; + if (copy_from_user(&spp_info, argp, sizeof(spp_info))) + goto out; + + r = -EINVAL; + if (spp_info.npages == 0 || + spp_info.npages > SUBPAGE_MAX_BITMAP) + goto out; + + r = kvm_vm_ioctl_get_subpages(kvm, &spp_info); + if (copy_to_user(argp, &spp_info, sizeof(spp_info))) { + r = -EFAULT; + goto out; + } + break; + } + case KVM_SUBPAGES_SET_ACCESS: { + struct kvm_subpage spp_info; + + if (!kvm->arch.spp_active) { + r = -ENODEV; + goto out; + } + + r = -EFAULT; + if (copy_from_user(&spp_info, argp, sizeof(spp_info))) + goto out; + + r = -EINVAL; + if (spp_info.npages == 0 || + spp_info.npages > SUBPAGE_MAX_BITMAP) + goto out; + + r = kvm_vm_ioctl_set_subpages(kvm, &spp_info); + break; + } + case KVM_INIT_SPP: { + r = kvm_vm_ioctl_init_spp(kvm); + break; + } default: r = -ENOTTY; } diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h index 9460830de536..700f0825336d 100644 --- a/include/uapi/linux/kvm.h +++ b/include/uapi/linux/kvm.h @@ -1257,6 +1257,9 @@ struct kvm_vfio_spapr_tce { struct kvm_userspace_memory_region) #define KVM_SET_TSS_ADDR _IO(KVMIO, 0x47) #define KVM_SET_IDENTITY_MAP_ADDR _IOW(KVMIO, 0x48, __u64) +#define KVM_SUBPAGES_GET_ACCESS _IOR(KVMIO, 0x49, __u64) +#define KVM_SUBPAGES_SET_ACCESS _IOW(KVMIO, 0x4a, __u64) +#define KVM_INIT_SPP _IOW(KVMIO, 0x4b, __u64) /* enable ucontrol for s390 */ struct kvm_s390_ucas_mapping { From patchwork Tue Nov 19 08:49:46 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yang, Weijiang" X-Patchwork-Id: 11251553 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 736CB14C0 for ; Tue, 19 Nov 2019 08:48:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 4C035222DF for ; Tue, 19 Nov 2019 08:48:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727495AbfKSIsM (ORCPT ); Tue, 19 Nov 2019 03:48:12 -0500 Received: from mga17.intel.com ([192.55.52.151]:26340 "EHLO mga17.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727407AbfKSIsL (ORCPT ); Tue, 19 Nov 2019 03:48:11 -0500 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga107.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 19 Nov 2019 00:48:09 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.68,322,1569308400"; d="scan'208";a="196427099" Received: from unknown (HELO local-michael-cet-test.sh.intel.com) ([10.239.159.128]) by orsmga007.jf.intel.com with ESMTP; 19 Nov 2019 00:48:05 -0800 From: Yang Weijiang To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, pbonzini@redhat.com, jmattson@google.com, sean.j.christopherson@intel.com Cc: yu.c.zhang@linux.intel.com, alazar@bitdefender.com, edwin.zhai@intel.com, Yang Weijiang Subject: [PATCH v7 6/9] vmx: spp: Set up SPP paging table at vmentry/vmexit Date: Tue, 19 Nov 2019 16:49:46 +0800 Message-Id: <20191119084949.15471-7-weijiang.yang@intel.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20191119084949.15471-1-weijiang.yang@intel.com> References: <20191119084949.15471-1-weijiang.yang@intel.com> Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org If write to subpage is not allowed, EPT violation generates and it's handled in fast_page_fault(). In current implementation, SPPT setup is only handled in handle_spp() vmexit handler, it's triggered when SPP bit is set in EPT leaf entry while SPPT entries are not ready. A SPP specific bit(11) is added to exit_qualification and a new exit reason(66) is introduced for SPP. Suggested-by: Paolo Bonzini Co-developed-by: He Chen Signed-off-by: He Chen Co-developed-by: Zhang Yi Signed-off-by: Zhang Yi Signed-off-by: Yang Weijiang --- arch/x86/include/asm/vmx.h | 9 ++++ arch/x86/include/uapi/asm/vmx.h | 2 + arch/x86/kvm/mmu.c | 42 ++++++++++++++- arch/x86/kvm/vmx/spp.c | 12 +++++ arch/x86/kvm/vmx/spp.h | 2 + arch/x86/kvm/vmx/vmx.c | 94 +++++++++++++++++++++++++++++++++ arch/x86/kvm/x86.c | 2 + include/uapi/linux/kvm.h | 5 ++ 8 files changed, 167 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/vmx.h b/arch/x86/include/asm/vmx.h index e1137807affc..f41989eae5e2 100644 --- a/arch/x86/include/asm/vmx.h +++ b/arch/x86/include/asm/vmx.h @@ -211,6 +211,8 @@ enum vmcs_field { XSS_EXIT_BITMAP_HIGH = 0x0000202D, ENCLS_EXITING_BITMAP = 0x0000202E, ENCLS_EXITING_BITMAP_HIGH = 0x0000202F, + SPPT_POINTER = 0x00002030, + SPPT_POINTER_HIGH = 0x00002031, TSC_MULTIPLIER = 0x00002032, TSC_MULTIPLIER_HIGH = 0x00002033, GUEST_PHYSICAL_ADDRESS = 0x00002400, @@ -532,6 +534,13 @@ struct vmx_msr_entry { #define EPT_VIOLATION_EXECUTABLE (1 << EPT_VIOLATION_EXECUTABLE_BIT) #define EPT_VIOLATION_GVA_TRANSLATED (1 << EPT_VIOLATION_GVA_TRANSLATED_BIT) +/* + * Exit Qualifications for SPPT-Induced vmexits + */ +#define SPPT_INDUCED_EXIT_TYPE_BIT 11 +#define SPPT_INDUCED_EXIT_TYPE (1 << SPPT_INDUCED_EXIT_TYPE_BIT) +#define SPPT_INTR_INFO_UNBLOCK_NMI INTR_INFO_UNBLOCK_NMI + /* * VM-instruction error numbers */ diff --git a/arch/x86/include/uapi/asm/vmx.h b/arch/x86/include/uapi/asm/vmx.h index f0b0c90dd398..ac67622bac5a 100644 --- a/arch/x86/include/uapi/asm/vmx.h +++ b/arch/x86/include/uapi/asm/vmx.h @@ -85,6 +85,7 @@ #define EXIT_REASON_PML_FULL 62 #define EXIT_REASON_XSAVES 63 #define EXIT_REASON_XRSTORS 64 +#define EXIT_REASON_SPP 66 #define VMX_EXIT_REASONS \ { EXIT_REASON_EXCEPTION_NMI, "EXCEPTION_NMI" }, \ @@ -141,6 +142,7 @@ { EXIT_REASON_ENCLS, "ENCLS" }, \ { EXIT_REASON_RDSEED, "RDSEED" }, \ { EXIT_REASON_PML_FULL, "PML_FULL" }, \ + { EXIT_REASON_SPP, "SPP" }, \ { EXIT_REASON_XSAVES, "XSAVES" }, \ { EXIT_REASON_XRSTORS, "XRSTORS" } diff --git a/arch/x86/kvm/mmu.c b/arch/x86/kvm/mmu.c index a63964e7cec7..a632c6b3c326 100644 --- a/arch/x86/kvm/mmu.c +++ b/arch/x86/kvm/mmu.c @@ -20,6 +20,7 @@ #include "x86.h" #include "kvm_cache_regs.h" #include "cpuid.h" +#include "vmx/spp.h" #include #include @@ -137,6 +138,7 @@ module_param(dbg, bool, 0644); /* The mask for the R/X bits in EPT PTEs */ #define PT64_EPT_READABLE_MASK 0x1ull #define PT64_EPT_EXECUTABLE_MASK 0x4ull +#define PT64_SPP_SAVED_BIT (1ULL << (PT64_SECOND_AVAIL_BITS_SHIFT + 1)) #include @@ -918,6 +920,11 @@ static u64 mark_spte_for_access_track(u64 spte) shadow_acc_track_saved_bits_shift; spte &= ~shadow_acc_track_mask; + if (spte & PT_SPP_MASK) { + spte &= ~PT_SPP_MASK; + spte |= PT64_SPP_SAVED_BIT; + } + return spte; } @@ -1598,9 +1605,15 @@ static bool wrprot_ad_disabled_spte(u64 *sptep) { bool was_writable = test_and_clear_bit(PT_WRITABLE_SHIFT, (unsigned long *)sptep); + bool was_spp_armed = test_and_clear_bit(PT_SPP_SHIFT, + (unsigned long *)sptep); + if (was_writable) kvm_set_pfn_dirty(spte_to_pfn(*sptep)); + if (was_spp_armed) + *sptep |= PT64_SPP_SAVED_BIT; + return was_writable; } @@ -3453,7 +3466,23 @@ static bool fast_page_fault(struct kvm_vcpu *vcpu, gva_t gva, int level, if ((error_code & PFERR_WRITE_MASK) && spte_can_locklessly_be_made_writable(spte)) { - new_spte |= PT_WRITABLE_MASK; + /* + * Record write protect fault caused by + * Sub-page Protection, let VMI decide + * the next step. + */ + if (spte & PT_SPP_MASK) { + fault_handled = true; + vcpu->run->exit_reason = KVM_EXIT_SPP; + vcpu->run->spp.addr = gva; + kvm_skip_emulated_instruction(vcpu); + break; + } + + if (was_spp_armed(new_spte)) + restore_spp_bit(&new_spte); + else + new_spte |= PT_WRITABLE_MASK; /* * Do not fix write-permission on the large spte. Since @@ -3604,6 +3633,9 @@ void kvm_mmu_free_roots(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, (mmu->root_level >= PT64_ROOT_4LEVEL || mmu->direct_map)) { mmu_free_root_page(vcpu->kvm, &mmu->root_hpa, &invalid_list); + if (vcpu->kvm->arch.spp_active) + mmu_free_root_page(vcpu->kvm, &mmu->sppt_root, + &invalid_list); } else { for (i = 0; i < 4; ++i) if (mmu->pae_root[i] != 0) @@ -5083,6 +5115,8 @@ void kvm_init_mmu(struct kvm_vcpu *vcpu, bool reset_roots) uint i; vcpu->arch.mmu->root_hpa = INVALID_PAGE; + if (!vcpu->kvm->arch.spp_active) + vcpu->arch.mmu->sppt_root = INVALID_PAGE; for (i = 0; i < KVM_MMU_NUM_PREV_ROOTS; i++) vcpu->arch.mmu->prev_roots[i] = KVM_MMU_ROOT_INFO_INVALID; @@ -5400,6 +5434,10 @@ int kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gva_t cr2, u64 error_code, r = vcpu->arch.mmu->page_fault(vcpu, cr2, lower_32_bits(error_code), false); + + if (vcpu->run->exit_reason == KVM_EXIT_SPP) + return 0; + WARN_ON(r == RET_PF_INVALID); } @@ -6165,6 +6203,8 @@ unsigned long kvm_mmu_calculate_default_mmu_pages(struct kvm *kvm) return nr_mmu_pages; } +#include "vmx/spp.c" + void kvm_mmu_destroy(struct kvm_vcpu *vcpu) { kvm_mmu_unload(vcpu); diff --git a/arch/x86/kvm/vmx/spp.c b/arch/x86/kvm/vmx/spp.c index b5327faf512e..0ff23b97970a 100644 --- a/arch/x86/kvm/vmx/spp.c +++ b/arch/x86/kvm/vmx/spp.c @@ -17,6 +17,18 @@ static void shadow_spp_walk_init(struct kvm_shadow_walk_iterator *iterator, iterator->level = PT64_ROOT_4LEVEL; } +/* Restore an spp armed PTE */ +void restore_spp_bit(u64 *spte) +{ + *spte &= ~PT64_SPP_SAVED_BIT; + *spte |= PT_SPP_MASK; +} + +bool was_spp_armed(u64 spte) +{ + return !!(spte & PT64_SPP_SAVED_BIT); +} + static int is_spp_shadow_present(u64 pte) { return pte & PT_PRESENT_MASK; diff --git a/arch/x86/kvm/vmx/spp.h b/arch/x86/kvm/vmx/spp.h index ad13a35bfca2..208b557cac7d 100644 --- a/arch/x86/kvm/vmx/spp.h +++ b/arch/x86/kvm/vmx/spp.h @@ -5,6 +5,8 @@ #define FULL_SPP_ACCESS ((u32)((1ULL << 32) - 1)) bool is_spp_spte(struct kvm_mmu_page *sp); +void restore_spp_bit(u64 *spte); +bool was_spp_armed(u64 spte); inline u64 construct_spptp(unsigned long root_hpa); int kvm_vm_ioctl_get_subpages(struct kvm *kvm, struct kvm_subpage *spp_info); diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index af296a290173..5eb8830c459a 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -2891,6 +2891,7 @@ void vmx_set_cr3(struct kvm_vcpu *vcpu, unsigned long cr3) struct kvm *kvm = vcpu->kvm; unsigned long guest_cr3; u64 eptp; + u64 spptp; guest_cr3 = cr3; if (enable_ept) { @@ -2913,6 +2914,12 @@ void vmx_set_cr3(struct kvm_vcpu *vcpu, unsigned long cr3) ept_load_pdptrs(vcpu); } + if (kvm->arch.spp_active && VALID_PAGE(vcpu->arch.mmu->sppt_root)) { + spptp = construct_spptp(vcpu->arch.mmu->sppt_root); + vmcs_write64(SPPT_POINTER, spptp); + vmx_flush_tlb(vcpu, true); + } + vmcs_writel(GUEST_CR3, guest_cr3); } @@ -5333,6 +5340,89 @@ static int handle_monitor_trap(struct kvm_vcpu *vcpu) return 1; } +int handle_spp(struct kvm_vcpu *vcpu) +{ + unsigned long exit_qualification; + struct kvm_memory_slot *slot; + gpa_t gpa; + gfn_t gfn; + + exit_qualification = vmcs_readl(EXIT_QUALIFICATION); + + /* + * SPP VM exit happened while executing iret from NMI, + * "blocked by NMI" bit has to be set before next VM entry. + * There are errata that may cause this bit to not be set: + * AAK134, BY25. + */ + if (!(to_vmx(vcpu)->idt_vectoring_info & VECTORING_INFO_VALID_MASK) && + (exit_qualification & SPPT_INTR_INFO_UNBLOCK_NMI)) + vmcs_set_bits(GUEST_INTERRUPTIBILITY_INFO, + GUEST_INTR_STATE_NMI); + + vcpu->arch.exit_qualification = exit_qualification; + if (exit_qualification & SPPT_INDUCED_EXIT_TYPE) { + struct kvm_subpage spp_info = {0}; + int ret; + + /* + * SPPT missing + * We don't set SPP write access for the corresponding + * GPA, if we haven't setup, we need to construct + * SPP table here. + */ + gpa = vmcs_read64(GUEST_PHYSICAL_ADDRESS); + gfn = gpa >> PAGE_SHIFT; + pr_info("SPP - SPPT entry missing! gfn = 0x%llx\n", gfn); + slot = gfn_to_memslot(vcpu->kvm, gfn); + if (!slot) + return -EFAULT; + + /* + * In level 1 of SPPT, there's no PRESENT bit, all data is + * regarded as permission vector, so need to check from + * level 2 to set up the vector if target page is protected. + */ + spp_info.base_gfn = gfn; + spp_info.npages = 1; + + spin_lock(&vcpu->kvm->mmu_lock); + ret = kvm_spp_get_permission(vcpu->kvm, &spp_info); + if (ret == 1) { + int page_num = KVM_PAGES_PER_HPAGE(PT_DIRECTORY_LEVEL); + u32 access; + gfn_t gfn_max; + + gfn &= ~(page_num - 1); + gfn_max = gfn + page_num - 1; + for (; gfn <= gfn_max; gfn++) { + if (!gfn_to_memslot(vcpu->kvm, gfn)) + continue; + access = *gfn_to_subpage_wp_info(slot, gfn); + if (access != FULL_SPP_ACCESS) + kvm_spp_setup_structure(vcpu, + access, + gfn); + } + } + spin_unlock(&vcpu->kvm->mmu_lock); + + return 1; + } + + /* + * SPPT Misconfig + * This is probably caused by some mis-configuration in SPPT + * entries, cannot handle it here, escalate the fault to + * emulator. + */ + WARN_ON(1); + vcpu->run->exit_reason = KVM_EXIT_UNKNOWN; + vcpu->run->hw.hardware_exit_reason = EXIT_REASON_SPP; + pr_alert("SPP - SPPT Misconfiguration!\n"); + return 0; +} + static int handle_monitor(struct kvm_vcpu *vcpu) { printk_once(KERN_WARNING "kvm: MONITOR instruction emulated as NOP!\n"); @@ -5547,6 +5637,7 @@ static int (*kvm_vmx_exit_handlers[])(struct kvm_vcpu *vcpu) = { [EXIT_REASON_INVVPID] = handle_vmx_instruction, [EXIT_REASON_RDRAND] = handle_invalid_op, [EXIT_REASON_RDSEED] = handle_invalid_op, + [EXIT_REASON_SPP] = handle_spp, [EXIT_REASON_XSAVES] = handle_xsaves, [EXIT_REASON_XRSTORS] = handle_xrstors, [EXIT_REASON_PML_FULL] = handle_pml_full, @@ -5781,6 +5872,9 @@ void dump_vmcs(void) pr_err("PostedIntrVec = 0x%02x\n", vmcs_read16(POSTED_INTR_NV)); if ((secondary_exec_control & SECONDARY_EXEC_ENABLE_EPT)) pr_err("EPT pointer = 0x%016llx\n", vmcs_read64(EPT_POINTER)); + if ((secondary_exec_control & SECONDARY_EXEC_ENABLE_SPP)) + pr_err("SPPT pointer = 0x%016llx\n", vmcs_read64(SPPT_POINTER)); + n = vmcs_read32(CR3_TARGET_COUNT); for (i = 0; i + 1 < n; i += 4) pr_err("CR3 target%u=%016lx target%u=%016lx\n", diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 78605d75690e..fa114b5db672 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -9523,6 +9523,8 @@ void kvm_arch_free_memslot(struct kvm *kvm, struct kvm_memory_slot *free, } kvm_page_track_free_memslot(free, dont); + if (kvm->arch.spp_active) + kvm_spp_free_memslot(free, dont); } int kvm_arch_create_memslot(struct kvm *kvm, struct kvm_memory_slot *slot, diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h index 700f0825336d..3816f0ac5dad 100644 --- a/include/uapi/linux/kvm.h +++ b/include/uapi/linux/kvm.h @@ -244,6 +244,7 @@ struct kvm_hyperv_exit { #define KVM_EXIT_S390_STSI 25 #define KVM_EXIT_IOAPIC_EOI 26 #define KVM_EXIT_HYPERV 27 +#define KVM_EXIT_SPP 28 /* For KVM_EXIT_INTERNAL_ERROR */ /* Emulate instruction failed. */ @@ -399,6 +400,10 @@ struct kvm_run { struct { __u8 vector; } eoi; + /* KVM_EXIT_SPP */ + struct { + __u64 addr; + } spp; /* KVM_EXIT_HYPERV */ struct kvm_hyperv_exit hyperv; /* Fix the size of the union. */ From patchwork Tue Nov 19 08:49:47 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yang, Weijiang" X-Patchwork-Id: 11251555 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9DB0014E5 for ; Tue, 19 Nov 2019 08:48:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 884FF21D7B for ; Tue, 19 Nov 2019 08:48:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727579AbfKSIsQ (ORCPT ); Tue, 19 Nov 2019 03:48:16 -0500 Received: from mga17.intel.com ([192.55.52.151]:26340 "EHLO mga17.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727407AbfKSIsO (ORCPT ); Tue, 19 Nov 2019 03:48:14 -0500 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga107.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 19 Nov 2019 00:48:13 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.68,322,1569308400"; d="scan'208";a="196427121" Received: from unknown (HELO local-michael-cet-test.sh.intel.com) ([10.239.159.128]) by orsmga007.jf.intel.com with ESMTP; 19 Nov 2019 00:48:09 -0800 From: Yang Weijiang To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, pbonzini@redhat.com, jmattson@google.com, sean.j.christopherson@intel.com Cc: yu.c.zhang@linux.intel.com, alazar@bitdefender.com, edwin.zhai@intel.com, Yang Weijiang Subject: [PATCH v7 7/9] mmu: spp: Enable Lazy mode SPP protection Date: Tue, 19 Nov 2019 16:49:47 +0800 Message-Id: <20191119084949.15471-8-weijiang.yang@intel.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20191119084949.15471-1-weijiang.yang@intel.com> References: <20191119084949.15471-1-weijiang.yang@intel.com> Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org To deal with SPP protected 4KB pages within hugepage(2MB,1GB etc), the hugepage entry is first zapped when set subpage permission, then in tdp_page_fault(), it checks whether the gfn should be mapped to PT_PAGE_TABLE_LEVEL or PT_DIRECTORY_LEVEL level depending on gfn inclusion of SPP protected page range. Suggested-by: Paolo Bonzini Signed-off-by: Yang Weijiang --- arch/x86/kvm/mmu.c | 14 ++++++++++++++ arch/x86/kvm/vmx/spp.c | 39 +++++++++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/spp.h | 4 ++++ 3 files changed, 57 insertions(+) diff --git a/arch/x86/kvm/mmu.c b/arch/x86/kvm/mmu.c index a632c6b3c326..9c5be402a0b2 100644 --- a/arch/x86/kvm/mmu.c +++ b/arch/x86/kvm/mmu.c @@ -3240,6 +3240,17 @@ static int __direct_map(struct kvm_vcpu *vcpu, gpa_t gpa, int write, map_writable); direct_pte_prefetch(vcpu, it.sptep); ++vcpu->stat.pf_fixed; + if (level == PT_PAGE_TABLE_LEVEL) { + struct kvm_subpage sbp = {0}; + int pages; + + sbp.base_gfn = gfn; + sbp.npages = 1; + pages = kvm_spp_get_permission(vcpu->kvm, &sbp); + if (pages == 1 && sbp.access_map[0] != FULL_SPP_ACCESS) + kvm_spp_mark_protection(vcpu->kvm, &sbp); + } + return ret; } @@ -4183,6 +4194,9 @@ static int tdp_page_fault(struct kvm_vcpu *vcpu, gva_t gpa, u32 error_code, if (level > PT_DIRECTORY_LEVEL && !check_hugepage_cache_consistency(vcpu, gfn, level)) level = PT_DIRECTORY_LEVEL; + + check_spp_protection(vcpu, gfn, &force_pt_level, &level); + gfn &= ~(KVM_PAGES_PER_HPAGE(level) - 1); } diff --git a/arch/x86/kvm/vmx/spp.c b/arch/x86/kvm/vmx/spp.c index 0ff23b97970a..111e11bb2598 100644 --- a/arch/x86/kvm/vmx/spp.c +++ b/arch/x86/kvm/vmx/spp.c @@ -550,6 +550,45 @@ inline u64 construct_spptp(unsigned long root_hpa) } EXPORT_SYMBOL_GPL(construct_spptp); +bool is_spp_protected(struct kvm_memory_slot *slot, gfn_t gfn, int level) +{ + int page_num = KVM_PAGES_PER_HPAGE(level); + int i; + + gfn &= ~(page_num - 1); + for (i = 0; i < page_num; ++i) { + if (*gfn_to_subpage_wp_info(slot, gfn + i) != FULL_SPP_ACCESS) + return true; + } + return false; +} + +bool check_spp_protection(struct kvm_vcpu *vcpu, gfn_t gfn, + bool *force_pt_level, int *level) +{ + struct kvm *kvm = vcpu->kvm; + struct kvm_memory_slot *slot; + bool protected; + int old_level = *level; + + if (!kvm->arch.spp_active) + return false; + + slot = gfn_to_memslot(kvm, gfn); + + if (!slot) + return false; + protected = is_spp_protected(slot, gfn, PT_DIRECTORY_LEVEL); + + if (protected) { + *level = PT_PAGE_TABLE_LEVEL; + *force_pt_level = true; + } else if (is_spp_protected(slot, gfn, PT_PDPE_LEVEL)) + *level = PT_DIRECTORY_LEVEL; + + return (old_level != *level); +} + int kvm_vm_ioctl_get_subpages(struct kvm *kvm, struct kvm_subpage *spp_info) { diff --git a/arch/x86/kvm/vmx/spp.h b/arch/x86/kvm/vmx/spp.h index 208b557cac7d..1ad526866977 100644 --- a/arch/x86/kvm/vmx/spp.h +++ b/arch/x86/kvm/vmx/spp.h @@ -4,9 +4,13 @@ #define FULL_SPP_ACCESS ((u32)((1ULL << 32) - 1)) +int kvm_spp_get_permission(struct kvm *kvm, struct kvm_subpage *spp_info); +int kvm_spp_mark_protection(struct kvm *kvm, struct kvm_subpage *spp_info); bool is_spp_spte(struct kvm_mmu_page *sp); void restore_spp_bit(u64 *spte); bool was_spp_armed(u64 spte); +bool check_spp_protection(struct kvm_vcpu *vcpu, gfn_t gfn, + bool *force_pt_level, int *level); inline u64 construct_spptp(unsigned long root_hpa); int kvm_vm_ioctl_get_subpages(struct kvm *kvm, struct kvm_subpage *spp_info); From patchwork Tue Nov 19 08:49:48 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yang, Weijiang" X-Patchwork-Id: 11251557 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B0A0B14C0 for ; Tue, 19 Nov 2019 08:48:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 9A0B6222DE for ; Tue, 19 Nov 2019 08:48:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727626AbfKSIsU (ORCPT ); Tue, 19 Nov 2019 03:48:20 -0500 Received: from mga17.intel.com ([192.55.52.151]:26340 "EHLO mga17.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727574AbfKSIsQ (ORCPT ); Tue, 19 Nov 2019 03:48:16 -0500 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga107.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 19 Nov 2019 00:48:15 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.68,322,1569308400"; d="scan'208";a="196427136" Received: from unknown (HELO local-michael-cet-test.sh.intel.com) ([10.239.159.128]) by orsmga007.jf.intel.com with ESMTP; 19 Nov 2019 00:48:13 -0800 From: Yang Weijiang To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, pbonzini@redhat.com, jmattson@google.com, sean.j.christopherson@intel.com Cc: yu.c.zhang@linux.intel.com, alazar@bitdefender.com, edwin.zhai@intel.com, Yang Weijiang Subject: [PATCH v7 8/9] mmu: spp: Handle SPP protected pages when VM memory changes Date: Tue, 19 Nov 2019 16:49:48 +0800 Message-Id: <20191119084949.15471-9-weijiang.yang@intel.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20191119084949.15471-1-weijiang.yang@intel.com> References: <20191119084949.15471-1-weijiang.yang@intel.com> Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Host page swapping/migration may change the translation in EPT leaf entry, if the target page is SPP protected, re-enable SPP protection in MMU notifier. If SPPT shadow page is reclaimed, the level1 pages don't have rmap to clear. Signed-off-by: Yang Weijiang --- arch/x86/kvm/mmu.c | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/arch/x86/kvm/mmu.c b/arch/x86/kvm/mmu.c index 9c5be402a0b2..7e9959a4a12b 100644 --- a/arch/x86/kvm/mmu.c +++ b/arch/x86/kvm/mmu.c @@ -1828,6 +1828,24 @@ static int kvm_set_pte_rmapp(struct kvm *kvm, struct kvm_rmap_head *rmap_head, new_spte &= ~PT_WRITABLE_MASK; new_spte &= ~SPTE_HOST_WRITEABLE; + /* + * if it's EPT leaf entry and the physical page is + * SPP protected, then re-enable SPP protection for + * the page. + */ + if (kvm->arch.spp_active && + level == PT_PAGE_TABLE_LEVEL) { + struct kvm_subpage spp_info = {0}; + int i; + + spp_info.base_gfn = gfn; + spp_info.npages = 1; + i = kvm_spp_get_permission(kvm, &spp_info); + if (i == 1 && + spp_info.access_map[0] != FULL_SPP_ACCESS) + new_spte |= PT_SPP_MASK; + } + new_spte = mark_spte_for_access_track(new_spte); mmu_spte_clear_track_bits(sptep); @@ -2677,6 +2695,10 @@ static bool mmu_page_zap_pte(struct kvm *kvm, struct kvm_mmu_page *sp, pte = *spte; if (is_shadow_present_pte(pte)) { if (is_last_spte(pte, sp->role.level)) { + /* SPPT leaf entries don't have rmaps*/ + if (sp->role.level == PT_PAGE_TABLE_LEVEL && + is_spp_spte(sp)) + return true; drop_spte(kvm, spte); if (is_large_pte(pte)) --kvm->stat.lpages; From patchwork Tue Nov 19 08:49:49 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yang, Weijiang" X-Patchwork-Id: 11251559 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id F2B0314C0 for ; Tue, 19 Nov 2019 08:48:27 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id D0A93222AB for ; Tue, 19 Nov 2019 08:48:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727557AbfKSIs1 (ORCPT ); Tue, 19 Nov 2019 03:48:27 -0500 Received: from mga17.intel.com ([192.55.52.151]:26340 "EHLO mga17.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727603AbfKSIsT (ORCPT ); Tue, 19 Nov 2019 03:48:19 -0500 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga107.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 19 Nov 2019 00:48:19 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.68,322,1569308400"; d="scan'208";a="196427146" Received: from unknown (HELO local-michael-cet-test.sh.intel.com) ([10.239.159.128]) by orsmga007.jf.intel.com with ESMTP; 19 Nov 2019 00:48:16 -0800 From: Yang Weijiang To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, pbonzini@redhat.com, jmattson@google.com, sean.j.christopherson@intel.com Cc: yu.c.zhang@linux.intel.com, alazar@bitdefender.com, edwin.zhai@intel.com, Yang Weijiang Subject: [PATCH v7 9/9] x86: spp: Add SPP protection check in emulation. Date: Tue, 19 Nov 2019 16:49:49 +0800 Message-Id: <20191119084949.15471-10-weijiang.yang@intel.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20191119084949.15471-1-weijiang.yang@intel.com> References: <20191119084949.15471-1-weijiang.yang@intel.com> Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org In instruction/mmio emulation cases, if the target write memroy is SPP protected, exit to user-space to handle it as if it's caused by SPP induced EPT violation due to guest write. Signed-off-by: Yang Weijiang --- arch/x86/kvm/x86.c | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index fa114b5db672..71f5a8ae76cf 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -5525,6 +5525,36 @@ static const struct read_write_emulator_ops write_emultor = { .write = true, }; +static bool is_emulator_spp_protected(struct kvm_vcpu *vcpu, + gpa_t gpa, + unsigned int bytes) +{ + gfn_t gfn, start_gfn, end_gfn; + struct kvm *kvm = vcpu->kvm; + struct kvm_memory_slot *slot; + u32 access; + + if (!kvm->arch.spp_active) + return false; + + start_gfn = gpa >> PAGE_SHIFT; + end_gfn = (gpa + bytes) >> PAGE_SHIFT; + for (gfn = start_gfn; gfn <= end_gfn; gfn++) { + slot = gfn_to_memslot(kvm, gfn); + if (slot) { + access = *gfn_to_subpage_wp_info(slot, gfn); + if (access != FULL_SPP_ACCESS) { + vcpu->run->exit_reason = KVM_EXIT_SPP; + vcpu->run->spp.addr = gfn; + kvm_skip_emulated_instruction(vcpu); + return true; + } + } + } + + return false; +} + static int emulator_read_write_onepage(unsigned long addr, void *val, unsigned int bytes, struct x86_exception *exception, @@ -5555,6 +5585,9 @@ static int emulator_read_write_onepage(unsigned long addr, void *val, return X86EMUL_PROPAGATE_FAULT; } + if (write && is_emulator_spp_protected(vcpu, gpa, bytes)) + return X86EMUL_UNHANDLEABLE; + if (!ret && ops->read_write_emulate(vcpu, gpa, val, bytes)) return X86EMUL_CONTINUE; @@ -6616,6 +6649,9 @@ int x86_emulate_instruction(struct kvm_vcpu *vcpu, return EMULATE_DONE; if (r == EMULATION_FAILED) { + if (vcpu->run->exit_reason == KVM_EXIT_SPP) + return EMULATE_USER_EXIT; + if (reexecute_instruction(vcpu, cr2, write_fault_to_spt, emulation_type)) return EMULATE_DONE;