From patchwork Tue Oct 27 21:23:36 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 11861921 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 08B00C4363A for ; Tue, 27 Oct 2020 21:25:33 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6F963218AC for ; Tue, 27 Oct 2020 21:25:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S374004AbgJ0VZ3 (ORCPT ); Tue, 27 Oct 2020 17:25:29 -0400 Received: from mga02.intel.com ([134.134.136.20]:56181 "EHLO mga02.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S373767AbgJ0VXx (ORCPT ); Tue, 27 Oct 2020 17:23:53 -0400 IronPort-SDR: v7dvIGVwOkSq59LDxxMSfjrtdKgfvpeAOojNrSaM/O1jCD0+d6UGvAazJeVjnCO6EfZqQV6Ljr VUlLVjlqT+Gw== X-IronPort-AV: E=McAfee;i="6000,8403,9787"; a="155133698" X-IronPort-AV: E=Sophos;i="5.77,424,1596524400"; d="scan'208";a="155133698" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Oct 2020 14:23:50 -0700 IronPort-SDR: RfwQ6NaYvaDPLuW8O5i3P4kMqOe3xLBXkmptCtlN7HyRuXl5fw6nq+oPiqKpL0zYB4Ta7Z/i10 nJDsr6nhz5kg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.77,424,1596524400"; d="scan'208";a="524886376" Received: from sjchrist-coffee.jf.intel.com ([10.54.74.160]) by fmsmga006.fm.intel.com with ESMTP; 27 Oct 2020 14:23:50 -0700 From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v3 01/11] KVM: x86: Get active PCID only when writing a CR3 value Date: Tue, 27 Oct 2020 14:23:36 -0700 Message-Id: <20201027212346.23409-2-sean.j.christopherson@intel.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20201027212346.23409-1-sean.j.christopherson@intel.com> References: <20201027212346.23409-1-sean.j.christopherson@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Retrieve the active PCID only when writing a guest CR3 value, i.e. don't get the PCID when using EPT. The PCID is not used when EPT is enabled, and must be manually stripped, which is annoying and unnecessary. And on VMX, getting the active PCID also involves reading the guest's CR3 and CR4.PCIDE, i.e. may add pointless VMREADs. Opportunistically rename the pgd/pgd_level params to root_hpa and root_level to better reflect their new roles. Keep the function names, as "load the guest PGD" is still accurate/correct. Last, and probably least, pass root_hpa as a hpa_t/u64 instead of an unsigned long. The EPTP holds a 64-bit value, even in 32-bit mode, so in theory EPT could support HIGHMEM for 32-bit KVM. Never mind that doing so would require changing the MMU page allocators and reworking the MMU to use kmap(). Signed-off-by: Sean Christopherson Reviewed-by: Vitaly Kuznetsov --- arch/x86/include/asm/kvm_host.h | 4 ++-- arch/x86/kvm/mmu.h | 2 +- arch/x86/kvm/svm/svm.c | 4 ++-- arch/x86/kvm/vmx/vmx.c | 13 ++++++------- arch/x86/kvm/vmx/vmx.h | 3 +-- 5 files changed, 12 insertions(+), 14 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index d44858b69353..33b2acfd7869 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1181,8 +1181,8 @@ struct kvm_x86_ops { int (*set_identity_map_addr)(struct kvm *kvm, u64 ident_addr); u64 (*get_mt_mask)(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio); - void (*load_mmu_pgd)(struct kvm_vcpu *vcpu, unsigned long pgd, - int pgd_level); + void (*load_mmu_pgd)(struct kvm_vcpu *vcpu, hpa_t root_hpa, + int root_level); bool (*has_wbinvd_exit)(void); diff --git a/arch/x86/kvm/mmu.h b/arch/x86/kvm/mmu.h index 9c4a9c8e43d9..add537a39177 100644 --- a/arch/x86/kvm/mmu.h +++ b/arch/x86/kvm/mmu.h @@ -95,7 +95,7 @@ static inline void kvm_mmu_load_pgd(struct kvm_vcpu *vcpu) if (!VALID_PAGE(root_hpa)) return; - kvm_x86_ops.load_mmu_pgd(vcpu, root_hpa | kvm_get_active_pcid(vcpu), + kvm_x86_ops.load_mmu_pgd(vcpu, root_hpa, vcpu->arch.mmu->shadow_root_level); } diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index cf951e588dd1..4a6a5a3dc963 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -3667,13 +3667,13 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu) return svm_exit_handlers_fastpath(vcpu); } -static void svm_load_mmu_pgd(struct kvm_vcpu *vcpu, unsigned long root, +static void svm_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int root_level) { struct vcpu_svm *svm = to_svm(vcpu); unsigned long cr3; - cr3 = __sme_set(root); + cr3 = __sme_set(root_hpa) | kvm_get_active_pcid(vcpu); if (npt_enabled) { svm->vmcb->control.nested_cr3 = cr3; vmcb_mark_dirty(svm->vmcb, VMCB_NPT); diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 281c405c7ea3..273a3206cef7 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -3043,8 +3043,7 @@ static int vmx_get_max_tdp_level(void) return 4; } -u64 construct_eptp(struct kvm_vcpu *vcpu, unsigned long root_hpa, - int root_level) +u64 construct_eptp(struct kvm_vcpu *vcpu, hpa_t root_hpa, int root_level) { u64 eptp = VMX_EPTP_MT_WB; @@ -3053,13 +3052,13 @@ u64 construct_eptp(struct kvm_vcpu *vcpu, unsigned long root_hpa, if (enable_ept_ad_bits && (!is_guest_mode(vcpu) || nested_ept_ad_enabled(vcpu))) eptp |= VMX_EPTP_AD_ENABLE_BIT; - eptp |= (root_hpa & PAGE_MASK); + eptp |= root_hpa; return eptp; } -static void vmx_load_mmu_pgd(struct kvm_vcpu *vcpu, unsigned long pgd, - int pgd_level) +static void vmx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, + int root_level) { struct kvm *kvm = vcpu->kvm; bool update_guest_cr3 = true; @@ -3067,7 +3066,7 @@ static void vmx_load_mmu_pgd(struct kvm_vcpu *vcpu, unsigned long pgd, u64 eptp; if (enable_ept) { - eptp = construct_eptp(vcpu, pgd, pgd_level); + eptp = construct_eptp(vcpu, root_hpa, root_level); vmcs_write64(EPT_POINTER, eptp); if (kvm_x86_ops.tlb_remote_flush) { @@ -3086,7 +3085,7 @@ static void vmx_load_mmu_pgd(struct kvm_vcpu *vcpu, unsigned long pgd, update_guest_cr3 = false; vmx_ept_load_pdptrs(vcpu); } else { - guest_cr3 = pgd; + guest_cr3 = root_hpa | kvm_get_active_pcid(vcpu); } if (update_guest_cr3) diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h index f6f66e5c6510..a2d143276603 100644 --- a/arch/x86/kvm/vmx/vmx.h +++ b/arch/x86/kvm/vmx/vmx.h @@ -326,8 +326,7 @@ void set_cr4_guest_host_mask(struct vcpu_vmx *vmx); void ept_save_pdptrs(struct kvm_vcpu *vcpu); void vmx_get_segment(struct kvm_vcpu *vcpu, struct kvm_segment *var, int seg); void vmx_set_segment(struct kvm_vcpu *vcpu, struct kvm_segment *var, int seg); -u64 construct_eptp(struct kvm_vcpu *vcpu, unsigned long root_hpa, - int root_level); +u64 construct_eptp(struct kvm_vcpu *vcpu, hpa_t root_hpa, int root_level); void update_exception_bitmap(struct kvm_vcpu *vcpu); void vmx_update_msr_bitmap(struct kvm_vcpu *vcpu); From patchwork Tue Oct 27 21:23:37 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 11861905 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 12F66C55178 for ; Tue, 27 Oct 2020 21:23:55 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id BC4D622283 for ; Tue, 27 Oct 2020 21:23:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S373787AbgJ0VXx (ORCPT ); Tue, 27 Oct 2020 17:23:53 -0400 Received: from mga02.intel.com ([134.134.136.20]:56183 "EHLO mga02.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S373768AbgJ0VXw (ORCPT ); Tue, 27 Oct 2020 17:23:52 -0400 IronPort-SDR: MjnUB3EqW/biV6+XQZAVlgUgfzJCSMlWuMBiYh+7SKOsubFq4iD8kywFH96SoJNFNAtzixva6O BKGmfpSYjrYw== X-IronPort-AV: E=McAfee;i="6000,8403,9787"; a="155133700" X-IronPort-AV: E=Sophos;i="5.77,424,1596524400"; d="scan'208";a="155133700" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Oct 2020 14:23:50 -0700 IronPort-SDR: 3UozLD4kj4WnCWtLJ7DqPUL0Cpp8zQELJIy/Gr/IV75Ox6lYLnHHhqYtpMhw50iNRpXhqWMM6l 8UsKHQmoMRiA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.77,424,1596524400"; d="scan'208";a="524886379" Received: from sjchrist-coffee.jf.intel.com ([10.54.74.160]) by fmsmga006.fm.intel.com with ESMTP; 27 Oct 2020 14:23:50 -0700 From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v3 02/11] KVM: VMX: Track common EPTP for Hyper-V's paravirt TLB flush Date: Tue, 27 Oct 2020 14:23:37 -0700 Message-Id: <20201027212346.23409-3-sean.j.christopherson@intel.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20201027212346.23409-1-sean.j.christopherson@intel.com> References: <20201027212346.23409-1-sean.j.christopherson@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Explicitly track the EPTP that is common to all vCPUs instead of grabbing vCPU0's EPTP when invoking Hyper-V's paravirt TLB flush. Tracking the EPTP will allow optimizing the checks when loading a new EPTP and will also allow dropping ept_pointer_match, e.g. by marking the common EPTP as invalid. This also technically fixes a bug where KVM could theoretically flush an invalid GPA if all vCPUs have an invalid root. In practice, it's likely impossible to trigger a remote TLB flush in such a scenario. In any case, the superfluous flush is completely benign. Signed-off-by: Sean Christopherson Reviewed-by: Vitaly Kuznetsov --- arch/x86/kvm/vmx/vmx.c | 20 +++++++++----------- arch/x86/kvm/vmx/vmx.h | 1 + 2 files changed, 10 insertions(+), 11 deletions(-) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 273a3206cef7..ebc87df4da4d 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -483,12 +483,14 @@ static void check_ept_pointer_match(struct kvm *kvm) if (!VALID_PAGE(tmp_eptp)) { tmp_eptp = to_vmx(vcpu)->ept_pointer; } else if (tmp_eptp != to_vmx(vcpu)->ept_pointer) { + to_kvm_vmx(kvm)->hv_tlb_eptp = INVALID_PAGE; to_kvm_vmx(kvm)->ept_pointers_match = EPT_POINTERS_MISMATCH; return; } } + to_kvm_vmx(kvm)->hv_tlb_eptp = tmp_eptp; to_kvm_vmx(kvm)->ept_pointers_match = EPT_POINTERS_MATCH; } @@ -501,21 +503,18 @@ static int kvm_fill_hv_flush_list_func(struct hv_guest_mapping_flush_list *flush range->pages); } -static inline int __hv_remote_flush_tlb_with_range(struct kvm *kvm, - struct kvm_vcpu *vcpu, struct kvm_tlb_range *range) +static inline int hv_remote_flush_eptp(u64 eptp, struct kvm_tlb_range *range) { - u64 ept_pointer = to_vmx(vcpu)->ept_pointer; - /* * FLUSH_GUEST_PHYSICAL_ADDRESS_SPACE hypercall needs address * of the base of EPT PML4 table, strip off EPT configuration * information. */ if (range) - return hyperv_flush_guest_mapping_range(ept_pointer & PAGE_MASK, + return hyperv_flush_guest_mapping_range(eptp & PAGE_MASK, kvm_fill_hv_flush_list_func, (void *)range); else - return hyperv_flush_guest_mapping(ept_pointer & PAGE_MASK); + return hyperv_flush_guest_mapping(eptp & PAGE_MASK); } static int hv_remote_flush_tlb_with_range(struct kvm *kvm, @@ -533,12 +532,11 @@ static int hv_remote_flush_tlb_with_range(struct kvm *kvm, kvm_for_each_vcpu(i, vcpu, kvm) { /* If ept_pointer is invalid pointer, bypass flush request. */ if (VALID_PAGE(to_vmx(vcpu)->ept_pointer)) - ret |= __hv_remote_flush_tlb_with_range( - kvm, vcpu, range); + ret |= hv_remote_flush_eptp(to_vmx(vcpu)->ept_pointer, + range); } - } else { - ret = __hv_remote_flush_tlb_with_range(kvm, - kvm_get_vcpu(kvm, 0), range); + } else if (VALID_PAGE(to_kvm_vmx(kvm)->hv_tlb_eptp)) { + ret = hv_remote_flush_eptp(to_kvm_vmx(kvm)->hv_tlb_eptp, range); } spin_unlock(&to_kvm_vmx(kvm)->ept_pointer_lock); diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h index a2d143276603..9a25e83f8b96 100644 --- a/arch/x86/kvm/vmx/vmx.h +++ b/arch/x86/kvm/vmx/vmx.h @@ -301,6 +301,7 @@ struct kvm_vmx { bool ept_identity_pagetable_done; gpa_t ept_identity_map_addr; + hpa_t hv_tlb_eptp; enum ept_pointers_status ept_pointers_match; spinlock_t ept_pointer_lock; }; From patchwork Tue Oct 27 21:23:38 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 11861993 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7023EC4363A for ; Tue, 27 Oct 2020 22:04:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0DFCC20878 for ; Tue, 27 Oct 2020 22:04:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2505734AbgJ0WEz (ORCPT ); Tue, 27 Oct 2020 18:04:55 -0400 Received: from mga02.intel.com ([134.134.136.20]:56183 "EHLO mga02.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S373764AbgJ0VXx (ORCPT ); Tue, 27 Oct 2020 17:23:53 -0400 IronPort-SDR: qlJE5TLc89lcrV2zCklCPu9OFDfQ0gKazQlVOG9QOzkJ3TFtVsl2TnrIRohWyGZAXs+nIrsp2Q egF7MxNKr6NA== X-IronPort-AV: E=McAfee;i="6000,8403,9787"; a="155133701" X-IronPort-AV: E=Sophos;i="5.77,424,1596524400"; d="scan'208";a="155133701" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Oct 2020 14:23:50 -0700 IronPort-SDR: 4JNPBWdgSdsf8ADQJ6GylQiZmOh5AeqfcZ/GD2hdtne5kLROeee31lwRVblU0yYgTzxuhEugf2 WXNYeknAejmQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.77,424,1596524400"; d="scan'208";a="524886383" Received: from sjchrist-coffee.jf.intel.com ([10.54.74.160]) by fmsmga006.fm.intel.com with ESMTP; 27 Oct 2020 14:23:50 -0700 From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v3 03/11] KVM: VMX: Stash kvm_vmx in a local variable for Hyper-V paravirt TLB flush Date: Tue, 27 Oct 2020 14:23:38 -0700 Message-Id: <20201027212346.23409-4-sean.j.christopherson@intel.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20201027212346.23409-1-sean.j.christopherson@intel.com> References: <20201027212346.23409-1-sean.j.christopherson@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Capture kvm_vmx in a local variable instead of polluting hv_remote_flush_tlb_with_range() with to_kvm_vmx(kvm). No functional change intended. Reviewed-by: Vitaly Kuznetsov Signed-off-by: Sean Christopherson --- arch/x86/kvm/vmx/vmx.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index ebc87df4da4d..a6442a861ffc 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -520,26 +520,27 @@ static inline int hv_remote_flush_eptp(u64 eptp, struct kvm_tlb_range *range) static int hv_remote_flush_tlb_with_range(struct kvm *kvm, struct kvm_tlb_range *range) { + struct kvm_vmx *kvm_vmx = to_kvm_vmx(kvm); struct kvm_vcpu *vcpu; int ret = 0, i; - spin_lock(&to_kvm_vmx(kvm)->ept_pointer_lock); + spin_lock(&kvm_vmx->ept_pointer_lock); - if (to_kvm_vmx(kvm)->ept_pointers_match == EPT_POINTERS_CHECK) + if (kvm_vmx->ept_pointers_match == EPT_POINTERS_CHECK) check_ept_pointer_match(kvm); - if (to_kvm_vmx(kvm)->ept_pointers_match != EPT_POINTERS_MATCH) { + if (kvm_vmx->ept_pointers_match != EPT_POINTERS_MATCH) { kvm_for_each_vcpu(i, vcpu, kvm) { /* If ept_pointer is invalid pointer, bypass flush request. */ if (VALID_PAGE(to_vmx(vcpu)->ept_pointer)) ret |= hv_remote_flush_eptp(to_vmx(vcpu)->ept_pointer, range); } - } else if (VALID_PAGE(to_kvm_vmx(kvm)->hv_tlb_eptp)) { - ret = hv_remote_flush_eptp(to_kvm_vmx(kvm)->hv_tlb_eptp, range); + } else if (VALID_PAGE(kvm_vmx->hv_tlb_eptp)) { + ret = hv_remote_flush_eptp(kvm_vmx->hv_tlb_eptp, range); } - spin_unlock(&to_kvm_vmx(kvm)->ept_pointer_lock); + spin_unlock(&kvm_vmx->ept_pointer_lock); return ret; } static int hv_remote_flush_tlb(struct kvm *kvm) From patchwork Tue Oct 27 21:23:39 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 11861907 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9DFCBC4363A for ; Tue, 27 Oct 2020 21:23:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5A67B218AC for ; Tue, 27 Oct 2020 21:23:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S373810AbgJ0VXz (ORCPT ); Tue, 27 Oct 2020 17:23:55 -0400 Received: from mga02.intel.com ([134.134.136.20]:56181 "EHLO mga02.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S373785AbgJ0VXx (ORCPT ); Tue, 27 Oct 2020 17:23:53 -0400 IronPort-SDR: RZibVdc4dnmYffmCUifUdVynpU9G8w2Vry/M3x+RjFsBYyqRci39UU1TKWcskdUWxyPPI2DOEk sNOzQWXiLqzw== X-IronPort-AV: E=McAfee;i="6000,8403,9787"; a="155133702" X-IronPort-AV: E=Sophos;i="5.77,424,1596524400"; d="scan'208";a="155133702" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Oct 2020 14:23:51 -0700 IronPort-SDR: HuWIfKBYzUUNIzP4fzVRUsi67CoH9FVWx458ywP6L2Tx4VKKlA78Llt08h2KTbSqwbMfDUFCy1 uJAcxh3LDWMA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.77,424,1596524400"; d="scan'208";a="524886386" Received: from sjchrist-coffee.jf.intel.com ([10.54.74.160]) by fmsmga006.fm.intel.com with ESMTP; 27 Oct 2020 14:23:50 -0700 From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v3 04/11] KVM: VMX: Fold Hyper-V EPTP checking into it's only caller Date: Tue, 27 Oct 2020 14:23:39 -0700 Message-Id: <20201027212346.23409-5-sean.j.christopherson@intel.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20201027212346.23409-1-sean.j.christopherson@intel.com> References: <20201027212346.23409-1-sean.j.christopherson@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Fold check_ept_pointer_match() into hv_remote_flush_tlb_with_range() in preparation for combining the kvm_for_each_vcpu loops of the ==CHECK and !=MATCH statements. No functional change intended. Signed-off-by: Sean Christopherson Reviewed-by: Vitaly Kuznetsov --- arch/x86/kvm/vmx/vmx.c | 44 +++++++++++++++++++----------------------- 1 file changed, 20 insertions(+), 24 deletions(-) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index a6442a861ffc..f5e9e2f61e10 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -472,28 +472,6 @@ static const u32 vmx_uret_msrs_list[] = { static bool __read_mostly enlightened_vmcs = true; module_param(enlightened_vmcs, bool, 0444); -/* check_ept_pointer() should be under protection of ept_pointer_lock. */ -static void check_ept_pointer_match(struct kvm *kvm) -{ - struct kvm_vcpu *vcpu; - u64 tmp_eptp = INVALID_PAGE; - int i; - - kvm_for_each_vcpu(i, vcpu, kvm) { - if (!VALID_PAGE(tmp_eptp)) { - tmp_eptp = to_vmx(vcpu)->ept_pointer; - } else if (tmp_eptp != to_vmx(vcpu)->ept_pointer) { - to_kvm_vmx(kvm)->hv_tlb_eptp = INVALID_PAGE; - to_kvm_vmx(kvm)->ept_pointers_match - = EPT_POINTERS_MISMATCH; - return; - } - } - - to_kvm_vmx(kvm)->hv_tlb_eptp = tmp_eptp; - to_kvm_vmx(kvm)->ept_pointers_match = EPT_POINTERS_MATCH; -} - static int kvm_fill_hv_flush_list_func(struct hv_guest_mapping_flush_list *flush, void *data) { @@ -523,11 +501,29 @@ static int hv_remote_flush_tlb_with_range(struct kvm *kvm, struct kvm_vmx *kvm_vmx = to_kvm_vmx(kvm); struct kvm_vcpu *vcpu; int ret = 0, i; + u64 tmp_eptp; spin_lock(&kvm_vmx->ept_pointer_lock); - if (kvm_vmx->ept_pointers_match == EPT_POINTERS_CHECK) - check_ept_pointer_match(kvm); + if (kvm_vmx->ept_pointers_match == EPT_POINTERS_CHECK) { + kvm_vmx->ept_pointers_match = EPT_POINTERS_MATCH; + kvm_vmx->hv_tlb_eptp = INVALID_PAGE; + + kvm_for_each_vcpu(i, vcpu, kvm) { + tmp_eptp = to_vmx(vcpu)->ept_pointer; + if (!VALID_PAGE(tmp_eptp)) + continue; + + if (!VALID_PAGE(kvm_vmx->hv_tlb_eptp)) { + kvm_vmx->hv_tlb_eptp = tmp_eptp; + } else if (kvm_vmx->hv_tlb_eptp != tmp_eptp) { + kvm_vmx->hv_tlb_eptp = INVALID_PAGE; + kvm_vmx->ept_pointers_match + = EPT_POINTERS_MISMATCH; + break; + } + } + } if (kvm_vmx->ept_pointers_match != EPT_POINTERS_MATCH) { kvm_for_each_vcpu(i, vcpu, kvm) { From patchwork Tue Oct 27 21:23:40 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 11861925 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 17134C388F9 for ; Tue, 27 Oct 2020 21:34:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D319522283 for ; Tue, 27 Oct 2020 21:34:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S374013AbgJ0VZY (ORCPT ); Tue, 27 Oct 2020 17:25:24 -0400 Received: from mga02.intel.com ([134.134.136.20]:56183 "EHLO mga02.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S373788AbgJ0VXx (ORCPT ); Tue, 27 Oct 2020 17:23:53 -0400 IronPort-SDR: MjaxPnqVvhOpaDtIW0aApqqjEuV0epFKE2PYvksI6ZBKZCiChXCaXYwALV+z5Q5bNJXil0QdQS 1xMSRS5/wwKw== X-IronPort-AV: E=McAfee;i="6000,8403,9787"; a="155133705" X-IronPort-AV: E=Sophos;i="5.77,424,1596524400"; d="scan'208";a="155133705" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Oct 2020 14:23:51 -0700 IronPort-SDR: vzsC4nexef7kWrWq/Cq/SPRg1mTlt51mgSCCsWjdQwg7b9VzKuCA1CkcDGQtvCW03sUxJ8nJQ6 X7YdROI+iKpA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.77,424,1596524400"; d="scan'208";a="524886389" Received: from sjchrist-coffee.jf.intel.com ([10.54.74.160]) by fmsmga006.fm.intel.com with ESMTP; 27 Oct 2020 14:23:51 -0700 From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v3 05/11] KVM: VMX: Do Hyper-V TLB flush iff vCPU's EPTP hasn't been flushed Date: Tue, 27 Oct 2020 14:23:40 -0700 Message-Id: <20201027212346.23409-6-sean.j.christopherson@intel.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20201027212346.23409-1-sean.j.christopherson@intel.com> References: <20201027212346.23409-1-sean.j.christopherson@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Combine the for-loops for Hyper-V TLB EPTP checking and flushing, and in doing so skip flushes for vCPUs whose EPTP matches the target EPTP. Signed-off-by: Sean Christopherson Reviewed-by: Vitaly Kuznetsov --- arch/x86/kvm/vmx/vmx.c | 23 ++++++++--------------- 1 file changed, 8 insertions(+), 15 deletions(-) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index f5e9e2f61e10..17b228c4ba19 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -505,33 +505,26 @@ static int hv_remote_flush_tlb_with_range(struct kvm *kvm, spin_lock(&kvm_vmx->ept_pointer_lock); - if (kvm_vmx->ept_pointers_match == EPT_POINTERS_CHECK) { + if (kvm_vmx->ept_pointers_match != EPT_POINTERS_MATCH) { kvm_vmx->ept_pointers_match = EPT_POINTERS_MATCH; kvm_vmx->hv_tlb_eptp = INVALID_PAGE; kvm_for_each_vcpu(i, vcpu, kvm) { tmp_eptp = to_vmx(vcpu)->ept_pointer; - if (!VALID_PAGE(tmp_eptp)) + if (!VALID_PAGE(tmp_eptp) || + tmp_eptp == kvm_vmx->hv_tlb_eptp) continue; - if (!VALID_PAGE(kvm_vmx->hv_tlb_eptp)) { + if (!VALID_PAGE(kvm_vmx->hv_tlb_eptp)) kvm_vmx->hv_tlb_eptp = tmp_eptp; - } else if (kvm_vmx->hv_tlb_eptp != tmp_eptp) { - kvm_vmx->hv_tlb_eptp = INVALID_PAGE; + else kvm_vmx->ept_pointers_match = EPT_POINTERS_MISMATCH; - break; - } - } - } - if (kvm_vmx->ept_pointers_match != EPT_POINTERS_MATCH) { - kvm_for_each_vcpu(i, vcpu, kvm) { - /* If ept_pointer is invalid pointer, bypass flush request. */ - if (VALID_PAGE(to_vmx(vcpu)->ept_pointer)) - ret |= hv_remote_flush_eptp(to_vmx(vcpu)->ept_pointer, - range); + ret |= hv_remote_flush_eptp(tmp_eptp, range); } + if (kvm_vmx->ept_pointers_match == EPT_POINTERS_MISMATCH) + kvm_vmx->hv_tlb_eptp = INVALID_PAGE; } else if (VALID_PAGE(kvm_vmx->hv_tlb_eptp)) { ret = hv_remote_flush_eptp(kvm_vmx->hv_tlb_eptp, range); } From patchwork Tue Oct 27 21:23:41 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 11861909 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 118A5C388F9 for ; Tue, 27 Oct 2020 21:24:06 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A4596207E8 for ; Tue, 27 Oct 2020 21:24:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S373831AbgJ0VX4 (ORCPT ); Tue, 27 Oct 2020 17:23:56 -0400 Received: from mga02.intel.com ([134.134.136.20]:56185 "EHLO mga02.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S373792AbgJ0VXx (ORCPT ); Tue, 27 Oct 2020 17:23:53 -0400 IronPort-SDR: vnGtHuTiKOTaRfF1oby53cCR6VXxmRFkT8DT/aNx2IsmM4cDtZq+pUWmedacisMQIq1sRPYtPo 6G34VpxmAk8g== X-IronPort-AV: E=McAfee;i="6000,8403,9787"; a="155133707" X-IronPort-AV: E=Sophos;i="5.77,424,1596524400"; d="scan'208";a="155133707" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Oct 2020 14:23:51 -0700 IronPort-SDR: sXAJxwM1RQOb5fQjqzpMnWsLfJwVufp+PmFoQC3eCUClLlnNU9cXiTe/7HFdiBL3j4mngwS0BX lfzdk1RKVxvg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.77,424,1596524400"; d="scan'208";a="524886393" Received: from sjchrist-coffee.jf.intel.com ([10.54.74.160]) by fmsmga006.fm.intel.com with ESMTP; 27 Oct 2020 14:23:51 -0700 From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v3 06/11] KVM: VMX: Invalidate hv_tlb_eptp to denote an EPTP mismatch Date: Tue, 27 Oct 2020 14:23:41 -0700 Message-Id: <20201027212346.23409-7-sean.j.christopherson@intel.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20201027212346.23409-1-sean.j.christopherson@intel.com> References: <20201027212346.23409-1-sean.j.christopherson@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Drop the dedicated 'ept_pointers_match' field in favor of stuffing 'hv_tlb_eptp' with INVALID_PAGE to mark it as invalid, i.e. to denote that there is at least one EPTP mismatch. Use a local variable to track whether or not a mismatch is detected so that hv_tlb_eptp can be used to skip redundant flushes. No functional change intended. Signed-off-by: Sean Christopherson Reviewed-by: Vitaly Kuznetsov --- arch/x86/kvm/vmx/vmx.c | 35 +++++++++++++++++++++++------------ arch/x86/kvm/vmx/vmx.h | 7 ------- 2 files changed, 23 insertions(+), 19 deletions(-) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 17b228c4ba19..25a714cda662 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -500,32 +500,44 @@ static int hv_remote_flush_tlb_with_range(struct kvm *kvm, { struct kvm_vmx *kvm_vmx = to_kvm_vmx(kvm); struct kvm_vcpu *vcpu; - int ret = 0, i; + int ret = 0, i, nr_unique_valid_eptps; u64 tmp_eptp; spin_lock(&kvm_vmx->ept_pointer_lock); - if (kvm_vmx->ept_pointers_match != EPT_POINTERS_MATCH) { - kvm_vmx->ept_pointers_match = EPT_POINTERS_MATCH; - kvm_vmx->hv_tlb_eptp = INVALID_PAGE; + if (!VALID_PAGE(kvm_vmx->hv_tlb_eptp)) { + nr_unique_valid_eptps = 0; + /* + * Flush all valid EPTPs, and see if all vCPUs have converged + * on a common EPTP, in which case future flushes can skip the + * loop and flush the common EPTP. + */ kvm_for_each_vcpu(i, vcpu, kvm) { tmp_eptp = to_vmx(vcpu)->ept_pointer; if (!VALID_PAGE(tmp_eptp) || tmp_eptp == kvm_vmx->hv_tlb_eptp) continue; - if (!VALID_PAGE(kvm_vmx->hv_tlb_eptp)) + /* + * Set the tracked EPTP to the first valid EPTP. Keep + * this EPTP for the entirety of the loop even if more + * EPTPs are encountered as a low effort optimization + * to avoid flushing the same (first) EPTP again. + */ + if (++nr_unique_valid_eptps == 1) kvm_vmx->hv_tlb_eptp = tmp_eptp; - else - kvm_vmx->ept_pointers_match - = EPT_POINTERS_MISMATCH; ret |= hv_remote_flush_eptp(tmp_eptp, range); } - if (kvm_vmx->ept_pointers_match == EPT_POINTERS_MISMATCH) + + /* + * The optimized flush of a single EPTP can't be used if there + * are multiple valid EPTPs (obviously). + */ + if (nr_unique_valid_eptps > 1) kvm_vmx->hv_tlb_eptp = INVALID_PAGE; - } else if (VALID_PAGE(kvm_vmx->hv_tlb_eptp)) { + } else { ret = hv_remote_flush_eptp(kvm_vmx->hv_tlb_eptp, range); } @@ -3060,8 +3072,7 @@ static void vmx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, if (kvm_x86_ops.tlb_remote_flush) { spin_lock(&to_kvm_vmx(kvm)->ept_pointer_lock); to_vmx(vcpu)->ept_pointer = eptp; - to_kvm_vmx(kvm)->ept_pointers_match - = EPT_POINTERS_CHECK; + to_kvm_vmx(kvm)->hv_tlb_eptp = INVALID_PAGE; spin_unlock(&to_kvm_vmx(kvm)->ept_pointer_lock); } diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h index 9a25e83f8b96..cecc2a641e19 100644 --- a/arch/x86/kvm/vmx/vmx.h +++ b/arch/x86/kvm/vmx/vmx.h @@ -288,12 +288,6 @@ struct vcpu_vmx { } shadow_msr_intercept; }; -enum ept_pointers_status { - EPT_POINTERS_CHECK = 0, - EPT_POINTERS_MATCH = 1, - EPT_POINTERS_MISMATCH = 2 -}; - struct kvm_vmx { struct kvm kvm; @@ -302,7 +296,6 @@ struct kvm_vmx { gpa_t ept_identity_map_addr; hpa_t hv_tlb_eptp; - enum ept_pointers_status ept_pointers_match; spinlock_t ept_pointer_lock; }; From patchwork Tue Oct 27 21:23:42 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 11861913 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 76E9FC388F9 for ; Tue, 27 Oct 2020 21:24:32 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1B6C6218AC for ; Tue, 27 Oct 2020 21:24:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S373819AbgJ0VXz (ORCPT ); Tue, 27 Oct 2020 17:23:55 -0400 Received: from mga02.intel.com ([134.134.136.20]:56183 "EHLO mga02.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S373794AbgJ0VXy (ORCPT ); Tue, 27 Oct 2020 17:23:54 -0400 IronPort-SDR: 38zIJuFf5pTgrVTR2EPU6chkpMy8QLKQQuoOeOYeFoIZ1aEMNRhL2s5jhaqwMyqhf3RAJFVmSg 7tK5wwgaIEMw== X-IronPort-AV: E=McAfee;i="6000,8403,9787"; a="155133710" X-IronPort-AV: E=Sophos;i="5.77,424,1596524400"; d="scan'208";a="155133710" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Oct 2020 14:23:51 -0700 IronPort-SDR: KQ69Sypkd5wBlU5bE+/28EgkHc5UVWi86kWdtenG4MLI1yuuWEaGp/mb+X0QyUNmEuNvgBXt6K WhnZe8cJ2zxA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.77,424,1596524400"; d="scan'208";a="524886396" Received: from sjchrist-coffee.jf.intel.com ([10.54.74.160]) by fmsmga006.fm.intel.com with ESMTP; 27 Oct 2020 14:23:51 -0700 From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v3 07/11] KVM: VMX: Don't invalidate hv_tlb_eptp if the new EPTP matches Date: Tue, 27 Oct 2020 14:23:42 -0700 Message-Id: <20201027212346.23409-8-sean.j.christopherson@intel.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20201027212346.23409-1-sean.j.christopherson@intel.com> References: <20201027212346.23409-1-sean.j.christopherson@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Don't invalidate the common EPTP, and thus trigger rechecking of EPTPs across all vCPUs, if the new EPTP matches the old/common EPTP. In all likelihood this is a meaningless optimization, but there are (uncommon) scenarios where KVM can reload the same EPTP. Reviewed-by: Vitaly Kuznetsov Signed-off-by: Sean Christopherson --- arch/x86/kvm/vmx/vmx.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 25a714cda662..4d9bc0d3a929 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -3072,7 +3072,8 @@ static void vmx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, if (kvm_x86_ops.tlb_remote_flush) { spin_lock(&to_kvm_vmx(kvm)->ept_pointer_lock); to_vmx(vcpu)->ept_pointer = eptp; - to_kvm_vmx(kvm)->hv_tlb_eptp = INVALID_PAGE; + if (eptp != to_kvm_vmx(kvm)->hv_tlb_eptp) + to_kvm_vmx(kvm)->hv_tlb_eptp = INVALID_PAGE; spin_unlock(&to_kvm_vmx(kvm)->ept_pointer_lock); } From patchwork Tue Oct 27 21:23:43 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 11861923 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 83FE6C55179 for ; Tue, 27 Oct 2020 21:34:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3AED22224E for ; Tue, 27 Oct 2020 21:34:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S373890AbgJ0VZS (ORCPT ); Tue, 27 Oct 2020 17:25:18 -0400 Received: from mga02.intel.com ([134.134.136.20]:56183 "EHLO mga02.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S373801AbgJ0VXy (ORCPT ); Tue, 27 Oct 2020 17:23:54 -0400 IronPort-SDR: YX31yzWvscM+zfv2stnSstE6ELyuXIZLCra4Oa4LILjEtI6rlnOY6xFFV8oWAbzf55WZ6qdkXG 17Nlg5o9oSQA== X-IronPort-AV: E=McAfee;i="6000,8403,9787"; a="155133711" X-IronPort-AV: E=Sophos;i="5.77,424,1596524400"; d="scan'208";a="155133711" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Oct 2020 14:23:51 -0700 IronPort-SDR: s0znNbddgf/mtuuNyzmf4ASexGiHxXWFgmyLO1I/TAAz2/tq5x7O+1mWfz2bAYSM4xFy09wut3 fCQv/9SqPsug== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.77,424,1596524400"; d="scan'208";a="524886399" Received: from sjchrist-coffee.jf.intel.com ([10.54.74.160]) by fmsmga006.fm.intel.com with ESMTP; 27 Oct 2020 14:23:51 -0700 From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v3 08/11] KVM: VMX: Explicitly check for hv_remote_flush_tlb when loading pgd Date: Tue, 27 Oct 2020 14:23:43 -0700 Message-Id: <20201027212346.23409-9-sean.j.christopherson@intel.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20201027212346.23409-1-sean.j.christopherson@intel.com> References: <20201027212346.23409-1-sean.j.christopherson@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Explicitly check that kvm_x86_ops.tlb_remote_flush() points at Hyper-V's implementation for PV flushing instead of assuming that a non-NULL implementation means running on Hyper-V. Wrap the related logic in ifdeffery as hv_remote_flush_tlb() is defined iff CONFIG_HYPERV!=n. Short term, the explicit check makes it more obvious why a non-NULL tlb_remote_flush() triggers EPTP shenanigans. Long term, this will allow TDX to define its own implementation of tlb_remote_flush() without running afoul of Hyper-V. Reviewed-by: Vitaly Kuznetsov Signed-off-by: Sean Christopherson --- arch/x86/kvm/vmx/vmx.c | 23 ++++++++++++++++------- 1 file changed, 16 insertions(+), 7 deletions(-) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 4d9bc0d3a929..b684f45d6a78 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -576,6 +576,21 @@ static int hv_enable_direct_tlbflush(struct kvm_vcpu *vcpu) #endif /* IS_ENABLED(CONFIG_HYPERV) */ +static void hv_load_mmu_eptp(struct kvm_vcpu *vcpu, u64 eptp) +{ +#if IS_ENABLED(CONFIG_HYPERV) + struct kvm_vmx *kvm_vmx = to_kvm_vmx(vcpu->kvm); + + if (kvm_x86_ops.tlb_remote_flush == hv_remote_flush_tlb) { + spin_lock(&kvm_vmx->ept_pointer_lock); + to_vmx(vcpu)->ept_pointer = eptp; + if (eptp != kvm_vmx->hv_tlb_eptp) + kvm_vmx->hv_tlb_eptp = INVALID_PAGE; + spin_unlock(&kvm_vmx->ept_pointer_lock); + } +#endif +} + /* * Comment's format: document - errata name - stepping - processor name. * Refer from @@ -3069,13 +3084,7 @@ static void vmx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, eptp = construct_eptp(vcpu, root_hpa, root_level); vmcs_write64(EPT_POINTER, eptp); - if (kvm_x86_ops.tlb_remote_flush) { - spin_lock(&to_kvm_vmx(kvm)->ept_pointer_lock); - to_vmx(vcpu)->ept_pointer = eptp; - if (eptp != to_kvm_vmx(kvm)->hv_tlb_eptp) - to_kvm_vmx(kvm)->hv_tlb_eptp = INVALID_PAGE; - spin_unlock(&to_kvm_vmx(kvm)->ept_pointer_lock); - } + hv_load_mmu_eptp(vcpu, eptp); if (!enable_unrestricted_guest && !is_paging(vcpu)) guest_cr3 = to_kvm_vmx(kvm)->ept_identity_map_addr; From patchwork Tue Oct 27 21:23:44 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 11861915 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 91685C388F9 for ; Tue, 27 Oct 2020 21:24:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 50C79218AC for ; Tue, 27 Oct 2020 21:24:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S373866AbgJ0VYO (ORCPT ); Tue, 27 Oct 2020 17:24:14 -0400 Received: from mga02.intel.com ([134.134.136.20]:56185 "EHLO mga02.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S373803AbgJ0VXz (ORCPT ); Tue, 27 Oct 2020 17:23:55 -0400 IronPort-SDR: xym8sBF/GXmtfisPENrn1CK8QwrHfQwrZZt71N3mHutqWWU2++1lCpODNXHFSE+scDYCpgOcet dmdMhXiAtBYw== X-IronPort-AV: E=McAfee;i="6000,8403,9787"; a="155133712" X-IronPort-AV: E=Sophos;i="5.77,424,1596524400"; d="scan'208";a="155133712" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Oct 2020 14:23:51 -0700 IronPort-SDR: wuwGtIlXxI9tKeWTdszwyjGPEbWXNONNeIX6qxWIC5MhE4S5EPqNSGWBQ3lFeoeSMz2yVyny12 f5J1NyTWku9w== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.77,424,1596524400"; d="scan'208";a="524886402" Received: from sjchrist-coffee.jf.intel.com ([10.54.74.160]) by fmsmga006.fm.intel.com with ESMTP; 27 Oct 2020 14:23:51 -0700 From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v3 09/11] KVM: VMX: Define Hyper-V paravirt TLB flush fields iff Hyper-V is enabled Date: Tue, 27 Oct 2020 14:23:44 -0700 Message-Id: <20201027212346.23409-10-sean.j.christopherson@intel.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20201027212346.23409-1-sean.j.christopherson@intel.com> References: <20201027212346.23409-1-sean.j.christopherson@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Ifdef away the Hyper-V specific fields in structs kvm_vmx and vcpu_vmx as each field has only a single reference outside of the struct itself that isn't already wrapped in ifdeffery (and both are initialization). vcpu_vmx.ept_pointer in particular should be wrapped as it is valid if and only if Hyper-v is active, i.e. non-Hyper-V code cannot rely on it to actually track the current EPTP (without additional code changes). Signed-off-by: Sean Christopherson Reviewed-by: Vitaly Kuznetsov --- arch/x86/kvm/vmx/vmx.c | 5 ++++- arch/x86/kvm/vmx/vmx.h | 4 ++++ 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index b684f45d6a78..5b7c5b2fd2c7 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -6955,8 +6955,9 @@ static int vmx_create_vcpu(struct kvm_vcpu *vcpu) vmx->pi_desc.nv = POSTED_INTR_VECTOR; vmx->pi_desc.sn = 1; +#if IS_ENABLED(CONFIG_HYPERV) vmx->ept_pointer = INVALID_PAGE; - +#endif return 0; free_vmcs: @@ -6973,7 +6974,9 @@ static int vmx_create_vcpu(struct kvm_vcpu *vcpu) static int vmx_vm_init(struct kvm *kvm) { +#if IS_ENABLED(CONFIG_HYPERV) spin_lock_init(&to_kvm_vmx(kvm)->ept_pointer_lock); +#endif if (!ple_gap) kvm->arch.pause_in_guest = true; diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h index cecc2a641e19..2bd86d8b2f4b 100644 --- a/arch/x86/kvm/vmx/vmx.h +++ b/arch/x86/kvm/vmx/vmx.h @@ -276,7 +276,9 @@ struct vcpu_vmx { */ u64 msr_ia32_feature_control; u64 msr_ia32_feature_control_valid_bits; +#if IS_ENABLED(CONFIG_HYPERV) u64 ept_pointer; +#endif struct pt_desc pt_desc; @@ -295,8 +297,10 @@ struct kvm_vmx { bool ept_identity_pagetable_done; gpa_t ept_identity_map_addr; +#if IS_ENABLED(CONFIG_HYPERV) hpa_t hv_tlb_eptp; spinlock_t ept_pointer_lock; +#endif }; bool nested_vmx_allowed(struct kvm_vcpu *vcpu); From patchwork Tue Oct 27 21:23:45 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 11861919 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1F756C4363A for ; Tue, 27 Oct 2020 21:25:23 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C28B5218AC for ; Tue, 27 Oct 2020 21:25:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S373939AbgJ0VZU (ORCPT ); Tue, 27 Oct 2020 17:25:20 -0400 Received: from mga02.intel.com ([134.134.136.20]:56181 "EHLO mga02.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S373799AbgJ0VXy (ORCPT ); Tue, 27 Oct 2020 17:23:54 -0400 IronPort-SDR: CS7krmnNGQSxPozfZw5e2p28Fouh+ddThmW/2Z97RCztoZE/027+AvIgf38NcR7SKfi+z2iFsD P+XDwbHzZlKg== X-IronPort-AV: E=McAfee;i="6000,8403,9787"; a="155133713" X-IronPort-AV: E=Sophos;i="5.77,424,1596524400"; d="scan'208";a="155133713" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Oct 2020 14:23:52 -0700 IronPort-SDR: ywuLMY040oWwJro92QHR0lHKimxFI4iRXa/MiR2z6+0/95VIvr99RI4jC812JLXrm+GLEDHTPH hCpADSMoVYlA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.77,424,1596524400"; d="scan'208";a="524886407" Received: from sjchrist-coffee.jf.intel.com ([10.54.74.160]) by fmsmga006.fm.intel.com with ESMTP; 27 Oct 2020 14:23:51 -0700 From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v3 10/11] KVM: VMX: Skip additional Hyper-V TLB EPTP flushes if one fails Date: Tue, 27 Oct 2020 14:23:45 -0700 Message-Id: <20201027212346.23409-11-sean.j.christopherson@intel.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20201027212346.23409-1-sean.j.christopherson@intel.com> References: <20201027212346.23409-1-sean.j.christopherson@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Skip additional EPTP flushes if one fails when processing EPTPs for Hyper-V's paravirt TLB flushing. If _any_ flush fails, KVM falls back to a full global flush, i.e. additional flushes are unnecessary (and will likely fail anyways). Continue processing the loop unless a mismatch was already detected, e.g. to handle the case where the first flush fails and there is a yet-to-be-detected mismatch. Signed-off-by: Sean Christopherson Reviewed-by: Vitaly Kuznetsov --- arch/x86/kvm/vmx/vmx.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 5b7c5b2fd2c7..40a67dd45c8c 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -528,7 +528,15 @@ static int hv_remote_flush_tlb_with_range(struct kvm *kvm, if (++nr_unique_valid_eptps == 1) kvm_vmx->hv_tlb_eptp = tmp_eptp; - ret |= hv_remote_flush_eptp(tmp_eptp, range); + if (!ret) + ret = hv_remote_flush_eptp(tmp_eptp, range); + + /* + * Stop processing EPTPs if a failure occurred and + * there is already a detected EPTP mismatch. + */ + if (ret && nr_unique_valid_eptps > 1) + break; } /* From patchwork Tue Oct 27 21:23:46 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 11861917 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E3407C4363A for ; Tue, 27 Oct 2020 21:24:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 944B2207E8 for ; Tue, 27 Oct 2020 21:24:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S373878AbgJ0VYO (ORCPT ); Tue, 27 Oct 2020 17:24:14 -0400 Received: from mga02.intel.com ([134.134.136.20]:56183 "EHLO mga02.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S373808AbgJ0VXz (ORCPT ); Tue, 27 Oct 2020 17:23:55 -0400 IronPort-SDR: DvEckXsFhCbfikHG4B4pqZnRUQxS4imJXbgIvE8kYUMt5CCC1HgC0zR8QXOfSqd1CO88pyiCzq 4DY2BkCTnlOw== X-IronPort-AV: E=McAfee;i="6000,8403,9787"; a="155133714" X-IronPort-AV: E=Sophos;i="5.77,424,1596524400"; d="scan'208";a="155133714" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Oct 2020 14:23:52 -0700 IronPort-SDR: iVnCIQz6QolkkiKUCdSmfio/AcZTyn9hi6zYsZZ3zl8vLa9YBJx9BeBXhDOiOl49vfLTXJqCMW 7P9Zq2TKiuQw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.77,424,1596524400"; d="scan'208";a="524886411" Received: from sjchrist-coffee.jf.intel.com ([10.54.74.160]) by fmsmga006.fm.intel.com with ESMTP; 27 Oct 2020 14:23:52 -0700 From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v3 11/11] KVM: VMX: Track root HPA instead of EPTP for paravirt Hyper-V TLB flush Date: Tue, 27 Oct 2020 14:23:46 -0700 Message-Id: <20201027212346.23409-12-sean.j.christopherson@intel.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20201027212346.23409-1-sean.j.christopherson@intel.com> References: <20201027212346.23409-1-sean.j.christopherson@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Track the address of the top-level EPT struct, a.k.a. the root HPA, instead of the EPTP itself for Hyper-V's paravirt TLB flush. The paravirt API takes only the address, not the full EPTP, and in theory tracking the EPTP could lead to false negatives, e.g. if the HPA matched but the attributes in the EPTP do not. In practice, such a mismatch is extremely unlikely, if not flat out impossible, given how KVM generates the EPTP. Opportunsitically rename the related fields to use the 'root' nomenclature, and to prefix them with 'hv_' to connect them to Hyper-V's paravirt TLB flushing. Signed-off-by: Sean Christopherson Reviewed-by: Vitaly Kuznetsov --- arch/x86/kvm/vmx/vmx.c | 83 ++++++++++++++++++++---------------------- arch/x86/kvm/vmx/vmx.h | 6 +-- 2 files changed, 42 insertions(+), 47 deletions(-) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 40a67dd45c8c..330d42ac5e02 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -481,18 +481,14 @@ static int kvm_fill_hv_flush_list_func(struct hv_guest_mapping_flush_list *flush range->pages); } -static inline int hv_remote_flush_eptp(u64 eptp, struct kvm_tlb_range *range) +static inline int hv_remote_flush_root_ept(hpa_t root_ept, + struct kvm_tlb_range *range) { - /* - * FLUSH_GUEST_PHYSICAL_ADDRESS_SPACE hypercall needs address - * of the base of EPT PML4 table, strip off EPT configuration - * information. - */ if (range) - return hyperv_flush_guest_mapping_range(eptp & PAGE_MASK, + return hyperv_flush_guest_mapping_range(root_ept, kvm_fill_hv_flush_list_func, (void *)range); else - return hyperv_flush_guest_mapping(eptp & PAGE_MASK); + return hyperv_flush_guest_mapping(root_ept); } static int hv_remote_flush_tlb_with_range(struct kvm *kvm, @@ -500,56 +496,55 @@ static int hv_remote_flush_tlb_with_range(struct kvm *kvm, { struct kvm_vmx *kvm_vmx = to_kvm_vmx(kvm); struct kvm_vcpu *vcpu; - int ret = 0, i, nr_unique_valid_eptps; - u64 tmp_eptp; + int ret = 0, i, nr_unique_valid_roots; + hpa_t root; - spin_lock(&kvm_vmx->ept_pointer_lock); + spin_lock(&kvm_vmx->hv_root_ept_lock); - if (!VALID_PAGE(kvm_vmx->hv_tlb_eptp)) { - nr_unique_valid_eptps = 0; + if (!VALID_PAGE(kvm_vmx->hv_root_ept)) { + nr_unique_valid_roots = 0; /* - * Flush all valid EPTPs, and see if all vCPUs have converged - * on a common EPTP, in which case future flushes can skip the - * loop and flush the common EPTP. + * Flush all valid roots, and see if all vCPUs have converged + * on a common root, in which case future flushes can skip the + * loop and flush the common root. */ kvm_for_each_vcpu(i, vcpu, kvm) { - tmp_eptp = to_vmx(vcpu)->ept_pointer; - if (!VALID_PAGE(tmp_eptp) || - tmp_eptp == kvm_vmx->hv_tlb_eptp) + root = to_vmx(vcpu)->hv_root_ept; + if (!VALID_PAGE(root) || root == kvm_vmx->hv_root_ept) continue; /* - * Set the tracked EPTP to the first valid EPTP. Keep - * this EPTP for the entirety of the loop even if more - * EPTPs are encountered as a low effort optimization - * to avoid flushing the same (first) EPTP again. + * Set the tracked root to the first valid root. Keep + * this root for the entirety of the loop even if more + * roots are encountered as a low effort optimization + * to avoid flushing the same (first) root again. */ - if (++nr_unique_valid_eptps == 1) - kvm_vmx->hv_tlb_eptp = tmp_eptp; + if (++nr_unique_valid_roots == 1) + kvm_vmx->hv_root_ept = root; if (!ret) - ret = hv_remote_flush_eptp(tmp_eptp, range); + ret = hv_remote_flush_root_ept(root, range); /* - * Stop processing EPTPs if a failure occurred and - * there is already a detected EPTP mismatch. + * Stop processing roots if a failure occurred and + * multiple valid roots have already been detected. */ - if (ret && nr_unique_valid_eptps > 1) + if (ret && nr_unique_valid_roots > 1) break; } /* - * The optimized flush of a single EPTP can't be used if there - * are multiple valid EPTPs (obviously). + * The optimized flush of a single root can't be used if there + * are multiple valid roots (obviously). */ - if (nr_unique_valid_eptps > 1) - kvm_vmx->hv_tlb_eptp = INVALID_PAGE; + if (nr_unique_valid_roots > 1) + kvm_vmx->hv_root_ept = INVALID_PAGE; } else { - ret = hv_remote_flush_eptp(kvm_vmx->hv_tlb_eptp, range); + ret = hv_remote_flush_root_ept(kvm_vmx->hv_root_ept, range); } - spin_unlock(&kvm_vmx->ept_pointer_lock); + spin_unlock(&kvm_vmx->hv_root_ept_lock); return ret; } static int hv_remote_flush_tlb(struct kvm *kvm) @@ -584,17 +579,17 @@ static int hv_enable_direct_tlbflush(struct kvm_vcpu *vcpu) #endif /* IS_ENABLED(CONFIG_HYPERV) */ -static void hv_load_mmu_eptp(struct kvm_vcpu *vcpu, u64 eptp) +static void hv_track_root_ept(struct kvm_vcpu *vcpu, hpa_t root_ept) { #if IS_ENABLED(CONFIG_HYPERV) struct kvm_vmx *kvm_vmx = to_kvm_vmx(vcpu->kvm); if (kvm_x86_ops.tlb_remote_flush == hv_remote_flush_tlb) { - spin_lock(&kvm_vmx->ept_pointer_lock); - to_vmx(vcpu)->ept_pointer = eptp; - if (eptp != kvm_vmx->hv_tlb_eptp) - kvm_vmx->hv_tlb_eptp = INVALID_PAGE; - spin_unlock(&kvm_vmx->ept_pointer_lock); + spin_lock(&kvm_vmx->hv_root_ept_lock); + to_vmx(vcpu)->hv_root_ept = root_ept; + if (root_ept != kvm_vmx->hv_root_ept) + kvm_vmx->hv_root_ept = INVALID_PAGE; + spin_unlock(&kvm_vmx->hv_root_ept_lock); } #endif } @@ -3092,7 +3087,7 @@ static void vmx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, eptp = construct_eptp(vcpu, root_hpa, root_level); vmcs_write64(EPT_POINTER, eptp); - hv_load_mmu_eptp(vcpu, eptp); + hv_track_root_ept(vcpu, root_hpa); if (!enable_unrestricted_guest && !is_paging(vcpu)) guest_cr3 = to_kvm_vmx(kvm)->ept_identity_map_addr; @@ -6964,7 +6959,7 @@ static int vmx_create_vcpu(struct kvm_vcpu *vcpu) vmx->pi_desc.sn = 1; #if IS_ENABLED(CONFIG_HYPERV) - vmx->ept_pointer = INVALID_PAGE; + vmx->hv_root_ept = INVALID_PAGE; #endif return 0; @@ -6983,7 +6978,7 @@ static int vmx_create_vcpu(struct kvm_vcpu *vcpu) static int vmx_vm_init(struct kvm *kvm) { #if IS_ENABLED(CONFIG_HYPERV) - spin_lock_init(&to_kvm_vmx(kvm)->ept_pointer_lock); + spin_lock_init(&to_kvm_vmx(kvm)->hv_root_ept_lock); #endif if (!ple_gap) diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h index 2bd86d8b2f4b..5c98d16a0c6f 100644 --- a/arch/x86/kvm/vmx/vmx.h +++ b/arch/x86/kvm/vmx/vmx.h @@ -277,7 +277,7 @@ struct vcpu_vmx { u64 msr_ia32_feature_control; u64 msr_ia32_feature_control_valid_bits; #if IS_ENABLED(CONFIG_HYPERV) - u64 ept_pointer; + u64 hv_root_ept; #endif struct pt_desc pt_desc; @@ -298,8 +298,8 @@ struct kvm_vmx { gpa_t ept_identity_map_addr; #if IS_ENABLED(CONFIG_HYPERV) - hpa_t hv_tlb_eptp; - spinlock_t ept_pointer_lock; + hpa_t hv_root_ept; + spinlock_t hv_root_ept_lock; #endif };