From patchwork Fri Mar 5 18:31:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12119179 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 869C3C433E9 for ; Fri, 5 Mar 2021 18:32:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4903D6509F for ; Fri, 5 Mar 2021 18:32:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230002AbhCESbo (ORCPT ); Fri, 5 Mar 2021 13:31:44 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58274 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229446AbhCESbb (ORCPT ); Fri, 5 Mar 2021 13:31:31 -0500 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B4603C061756 for ; Fri, 5 Mar 2021 10:31:31 -0800 (PST) Received: by mail-yb1-xb4a.google.com with SMTP id p136so3366088ybc.21 for ; Fri, 05 Mar 2021 10:31:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=Ez6kaNpvX5WDqy2a2Fkcncr/macPJ8v8cBzt3I2dkQE=; b=dfkixw7Hg7+i/kt8hB8quDhlBcEbqlneYKJ6B03h2Wx0CvZP4+Snl2WQetxtJNd0jX 5gOhsLYc1pu+ulZlVbm3NLeNDAmJmqsQqYKmrOyTPy54SrI1n9vLdlbr6GOkpm0UM9gB nBwfuDJtq2pBBFr4SD4Q4l7PJdUQ5ftD3eUTNmfqsIGWwei32i62HMkMmibETTOzQzYf ZbeKu+JARUbN6rW0Eh4+3VVsb9Qfs4DDnlrXsU8e2MJQHB+D1YEZ0z8+ZeDH4PJY2xH7 UwuOrejd7OFzcDiuDoBYV5AVbg+ximPGfQ5Quj6QKoq286j8ZCaNbDpyh4jWC0WAj/Y9 U6bw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=Ez6kaNpvX5WDqy2a2Fkcncr/macPJ8v8cBzt3I2dkQE=; b=h7uNsEKkxZ4/gh5deHzUuIbm8dU1DjaG34LHnbccZsW2lwirPBA8zHuOzyxeG/QXaC TCs9fq3NwadCgGwvI5eK34LF2DnkbXJ2ymD8MJ3MFKHhdLvGRM+1NefGTaNRKSmtAPqJ nHb93XbtcFgiv3oYyFy2gdD2BmRw3ig30lto3/srWsvr46Vl73nxICUSKIAPNJDXrJJ4 jp5XZIkOLZc2ydiMQzCV1uMbPsY9mrVgPIg7wq5vGv/lWa7DcpYyJuqB8ORHWmoETcPU q0H9sSrCiB19ECdT2XDVXaOwblu0P3+4KwlDtrDDe0KI7v2VZyC42QZx+KPrzK5Pn4hs RE3A== X-Gm-Message-State: AOAM530xkeCPxsrCQtO9uuz+JZP6bwx42R5a2HrX0CpbQWJ8wmlNMzCY fMxi0la4WJ2XcxjmX2GtCdN452Pnsis= X-Google-Smtp-Source: ABdhPJwkLqyDK3iGSyjPchE/cUxSowy6gHbbUTDSoVntlUxbds3j6SZNNAXArYdWVUJrLeKFz/AcmKhqb6w= Sender: "seanjc via sendgmr" X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:9857:be95:97a2:e91c]) (user=seanjc job=sendgmr) by 2002:a25:7613:: with SMTP id r19mr15575268ybc.212.1614969090998; Fri, 05 Mar 2021 10:31:30 -0800 (PST) Reply-To: Sean Christopherson Date: Fri, 5 Mar 2021 10:31:13 -0800 In-Reply-To: <20210305183123.3978098-1-seanjc@google.com> Message-Id: <20210305183123.3978098-2-seanjc@google.com> Mime-Version: 1.0 References: <20210305183123.3978098-1-seanjc@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v4 01/11] KVM: x86: Get active PCID only when writing a CR3 value From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Sean Christopherson Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Sean Christopherson Retrieve the active PCID only when writing a guest CR3 value, i.e. don't get the PCID when using EPT or NPT. The PCID is especially problematic for EPT as the bits have different meaning, and so the PCID and must be manually stripped, which is annoying and unnecessary. And on VMX, getting the active PCID also involves reading the guest's CR3 and CR4.PCIDE, i.e. may add pointless VMREADs. Opportunistically rename the pgd/pgd_level params to root_hpa and root_level to better reflect their new roles. Keep the function names, as "load the guest PGD" is still accurate/correct. Last, and probably least, pass root_hpa as a hpa_t/u64 instead of an unsigned long. The EPTP holds a 64-bit value, even in 32-bit mode, so in theory EPT could support HIGHMEM for 32-bit KVM. Never mind that doing so would require changing the MMU page allocators and reworking the MMU to use kmap(). Signed-off-by: Sean Christopherson Signed-off-by: Sean Christopherson Reviewed-by: Vitaly Kuznetsov --- arch/x86/include/asm/kvm_host.h | 4 ++-- arch/x86/kvm/mmu.h | 4 ++-- arch/x86/kvm/svm/svm.c | 10 ++++++---- arch/x86/kvm/vmx/vmx.c | 13 ++++++------- arch/x86/kvm/vmx/vmx.h | 3 +-- 5 files changed, 17 insertions(+), 17 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index 2da6c9f5935a..51725e994451 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1245,8 +1245,8 @@ struct kvm_x86_ops { int (*set_identity_map_addr)(struct kvm *kvm, u64 ident_addr); u64 (*get_mt_mask)(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio); - void (*load_mmu_pgd)(struct kvm_vcpu *vcpu, unsigned long pgd, - int pgd_level); + void (*load_mmu_pgd)(struct kvm_vcpu *vcpu, hpa_t root_hpa, + int root_level); bool (*has_wbinvd_exit)(void); diff --git a/arch/x86/kvm/mmu.h b/arch/x86/kvm/mmu.h index 67e8c7c7a6ce..88d0ed5225a4 100644 --- a/arch/x86/kvm/mmu.h +++ b/arch/x86/kvm/mmu.h @@ -107,8 +107,8 @@ static inline void kvm_mmu_load_pgd(struct kvm_vcpu *vcpu) if (!VALID_PAGE(root_hpa)) return; - static_call(kvm_x86_load_mmu_pgd)(vcpu, root_hpa | kvm_get_active_pcid(vcpu), - vcpu->arch.mmu->shadow_root_level); + static_call(kvm_x86_load_mmu_pgd)(vcpu, root_hpa, + vcpu->arch.mmu->shadow_root_level); } int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code, diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index dfc8fe231e8b..c7a685bf6862 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -3902,14 +3902,14 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu) return svm_exit_handlers_fastpath(vcpu); } -static void svm_load_mmu_pgd(struct kvm_vcpu *vcpu, unsigned long root, +static void svm_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int root_level) { struct vcpu_svm *svm = to_svm(vcpu); unsigned long cr3; if (npt_enabled) { - svm->vmcb->control.nested_cr3 = __sme_set(root); + svm->vmcb->control.nested_cr3 = __sme_set(root_hpa); vmcb_mark_dirty(svm->vmcb, VMCB_NPT); /* Loading L2's CR3 is handled by enter_svm_guest_mode. */ @@ -3917,9 +3917,11 @@ static void svm_load_mmu_pgd(struct kvm_vcpu *vcpu, unsigned long root, return; cr3 = vcpu->arch.cr3; } else if (vcpu->arch.mmu->shadow_root_level >= PT64_ROOT_4LEVEL) { - cr3 = __sme_set(root); + cr3 = __sme_set(root_hpa) | kvm_get_active_pcid(vcpu); } else { - cr3 = root; + /* PCID in the guest should be impossible with a 32-bit MMU. */ + WARN_ON_ONCE(kvm_get_active_pcid(vcpu)); + cr3 = root_hpa; } svm->vmcb->save.cr3 = cr3; diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 6d7e760fdfa0..dde2ebc7cf3a 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -3088,8 +3088,7 @@ static int vmx_get_max_tdp_level(void) return 4; } -u64 construct_eptp(struct kvm_vcpu *vcpu, unsigned long root_hpa, - int root_level) +u64 construct_eptp(struct kvm_vcpu *vcpu, hpa_t root_hpa, int root_level) { u64 eptp = VMX_EPTP_MT_WB; @@ -3098,13 +3097,13 @@ u64 construct_eptp(struct kvm_vcpu *vcpu, unsigned long root_hpa, if (enable_ept_ad_bits && (!is_guest_mode(vcpu) || nested_ept_ad_enabled(vcpu))) eptp |= VMX_EPTP_AD_ENABLE_BIT; - eptp |= (root_hpa & PAGE_MASK); + eptp |= root_hpa; return eptp; } -static void vmx_load_mmu_pgd(struct kvm_vcpu *vcpu, unsigned long pgd, - int pgd_level) +static void vmx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, + int root_level) { struct kvm *kvm = vcpu->kvm; bool update_guest_cr3 = true; @@ -3112,7 +3111,7 @@ static void vmx_load_mmu_pgd(struct kvm_vcpu *vcpu, unsigned long pgd, u64 eptp; if (enable_ept) { - eptp = construct_eptp(vcpu, pgd, pgd_level); + eptp = construct_eptp(vcpu, root_hpa, root_level); vmcs_write64(EPT_POINTER, eptp); if (kvm_x86_ops.tlb_remote_flush) { @@ -3131,7 +3130,7 @@ static void vmx_load_mmu_pgd(struct kvm_vcpu *vcpu, unsigned long pgd, update_guest_cr3 = false; vmx_ept_load_pdptrs(vcpu); } else { - guest_cr3 = pgd; + guest_cr3 = root_hpa | kvm_get_active_pcid(vcpu); } if (update_guest_cr3) diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h index 89da5e1251f1..4795955490cb 100644 --- a/arch/x86/kvm/vmx/vmx.h +++ b/arch/x86/kvm/vmx/vmx.h @@ -376,8 +376,7 @@ void set_cr4_guest_host_mask(struct vcpu_vmx *vmx); void ept_save_pdptrs(struct kvm_vcpu *vcpu); void vmx_get_segment(struct kvm_vcpu *vcpu, struct kvm_segment *var, int seg); void vmx_set_segment(struct kvm_vcpu *vcpu, struct kvm_segment *var, int seg); -u64 construct_eptp(struct kvm_vcpu *vcpu, unsigned long root_hpa, - int root_level); +u64 construct_eptp(struct kvm_vcpu *vcpu, hpa_t root_hpa, int root_level); void vmx_update_exception_bitmap(struct kvm_vcpu *vcpu); void vmx_update_msr_bitmap(struct kvm_vcpu *vcpu); From patchwork Fri Mar 5 18:31:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12119175 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2364AC433DB for ; Fri, 5 Mar 2021 18:32:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DED29650A1 for ; Fri, 5 Mar 2021 18:32:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230050AbhCESbp (ORCPT ); Fri, 5 Mar 2021 13:31:45 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58288 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229526AbhCESbe (ORCPT ); Fri, 5 Mar 2021 13:31:34 -0500 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 058A6C061574 for ; Fri, 5 Mar 2021 10:31:34 -0800 (PST) Received: by mail-yb1-xb4a.google.com with SMTP id l3so3358822ybf.17 for ; Fri, 05 Mar 2021 10:31:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=zBZf9XbdDoEer6H8DADVbIaEOxSfCjbFi3iuHxyt0ms=; b=KTBDgfSe5LeIa7xBzM3hFQj0aQ3RG9tRlpRCR+pTgtUywvTNIB3ryG/8bSK50XMw9m H5PgIlYyQHOjzDcBW91MWbWh73qcIxs0ABT/4WYclZ8LzKk7lfe+eCCxc2USWdAJyNzx cMajkXr813oxu/VepDXvf0lmelSMr7/yGorz+YtQkXMYwcjfwUP13kEWZg1sppqDW5aq fWgD45BVM4tPpZIYU/KiVsRmc37YJtTMjykfV2GJvzCGZ99m0/OntnsBFCrWZ5/i2Rmq ZO+b2CRC5AfqrjPWExPVULtpwDmMIuXSqMuWLtzqlIt01G1P0tMXHr291xfduRI4pXwF yRBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=zBZf9XbdDoEer6H8DADVbIaEOxSfCjbFi3iuHxyt0ms=; b=cX85bOrW3Hax5lc/cEPyAAD3rUIHJolTZboBC71fIsbUJpuwlmo+rnnpU8xKclHTKC ZDBNPrrX0yAECWLcWzYnPp4BkLigE1rzKzh1MtjPnTdUQj+ZMOVlYdS4c/1oGxuD4gI4 C9D88zcnKHf1M5jdjwKQkuE6b4uqfC2SntTXlRPWQolzsXLY2Cuc20iWCa0LRytR8+wm Ui5w65xp0EszuRBBK/Wz+I6+FBBnuFpYiBMmTo0mMmE65e4I4tHxaSzv+fm+tvOG4jOL v7VMBNcb9+fwZWk6CE4TkffFaRSroOtbdn3l/wDLjoaOnKx78vZ7bOzyhmbKfZYzuqj/ 8Diw== X-Gm-Message-State: AOAM530Ech0hXc5Qy3P9DGTemfMWg7GmibvOeahJh7hrlafaSqu2t/f+ F3IGPVcU05IWfJ+e1hKNm/n/ZVyRJI4= X-Google-Smtp-Source: ABdhPJwsYNYgV7c0HUnqJhultwv9JP3/XlFG52G5c1klvEbzi232XiE57OGPzBaI7ZfoTxRLwMySpuPMb/4= Sender: "seanjc via sendgmr" X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:9857:be95:97a2:e91c]) (user=seanjc job=sendgmr) by 2002:a25:38d6:: with SMTP id f205mr16093928yba.448.1614969093316; Fri, 05 Mar 2021 10:31:33 -0800 (PST) Reply-To: Sean Christopherson Date: Fri, 5 Mar 2021 10:31:14 -0800 In-Reply-To: <20210305183123.3978098-1-seanjc@google.com> Message-Id: <20210305183123.3978098-3-seanjc@google.com> Mime-Version: 1.0 References: <20210305183123.3978098-1-seanjc@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v4 02/11] KVM: VMX: Track common EPTP for Hyper-V's paravirt TLB flush From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Sean Christopherson Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Sean Christopherson Explicitly track the EPTP that is common to all vCPUs instead of grabbing vCPU0's EPTP when invoking Hyper-V's paravirt TLB flush. Tracking the EPTP will allow optimizing the checks when loading a new EPTP and will also allow dropping ept_pointer_match, e.g. by marking the common EPTP as invalid. This also technically fixes a bug where KVM could theoretically flush an invalid GPA if all vCPUs have an invalid root. In practice, it's likely impossible to trigger a remote TLB flush in such a scenario. In any case, the superfluous flush is completely benign. Reviewed-by: Vitaly Kuznetsov Signed-off-by: Sean Christopherson Signed-off-by: Sean Christopherson --- arch/x86/kvm/vmx/vmx.c | 20 +++++++++----------- arch/x86/kvm/vmx/vmx.h | 1 + 2 files changed, 10 insertions(+), 11 deletions(-) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index dde2ebc7cf3a..4082c7a26612 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -483,12 +483,14 @@ static void check_ept_pointer_match(struct kvm *kvm) if (!VALID_PAGE(tmp_eptp)) { tmp_eptp = to_vmx(vcpu)->ept_pointer; } else if (tmp_eptp != to_vmx(vcpu)->ept_pointer) { + to_kvm_vmx(kvm)->hv_tlb_eptp = INVALID_PAGE; to_kvm_vmx(kvm)->ept_pointers_match = EPT_POINTERS_MISMATCH; return; } } + to_kvm_vmx(kvm)->hv_tlb_eptp = tmp_eptp; to_kvm_vmx(kvm)->ept_pointers_match = EPT_POINTERS_MATCH; } @@ -501,21 +503,18 @@ static int kvm_fill_hv_flush_list_func(struct hv_guest_mapping_flush_list *flush range->pages); } -static inline int __hv_remote_flush_tlb_with_range(struct kvm *kvm, - struct kvm_vcpu *vcpu, struct kvm_tlb_range *range) +static inline int hv_remote_flush_eptp(u64 eptp, struct kvm_tlb_range *range) { - u64 ept_pointer = to_vmx(vcpu)->ept_pointer; - /* * FLUSH_GUEST_PHYSICAL_ADDRESS_SPACE hypercall needs address * of the base of EPT PML4 table, strip off EPT configuration * information. */ if (range) - return hyperv_flush_guest_mapping_range(ept_pointer & PAGE_MASK, + return hyperv_flush_guest_mapping_range(eptp & PAGE_MASK, kvm_fill_hv_flush_list_func, (void *)range); else - return hyperv_flush_guest_mapping(ept_pointer & PAGE_MASK); + return hyperv_flush_guest_mapping(eptp & PAGE_MASK); } static int hv_remote_flush_tlb_with_range(struct kvm *kvm, @@ -533,12 +532,11 @@ static int hv_remote_flush_tlb_with_range(struct kvm *kvm, kvm_for_each_vcpu(i, vcpu, kvm) { /* If ept_pointer is invalid pointer, bypass flush request. */ if (VALID_PAGE(to_vmx(vcpu)->ept_pointer)) - ret |= __hv_remote_flush_tlb_with_range( - kvm, vcpu, range); + ret |= hv_remote_flush_eptp(to_vmx(vcpu)->ept_pointer, + range); } - } else { - ret = __hv_remote_flush_tlb_with_range(kvm, - kvm_get_vcpu(kvm, 0), range); + } else if (VALID_PAGE(to_kvm_vmx(kvm)->hv_tlb_eptp)) { + ret = hv_remote_flush_eptp(to_kvm_vmx(kvm)->hv_tlb_eptp, range); } spin_unlock(&to_kvm_vmx(kvm)->ept_pointer_lock); diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h index 4795955490cb..f846cf3a5d25 100644 --- a/arch/x86/kvm/vmx/vmx.h +++ b/arch/x86/kvm/vmx/vmx.h @@ -351,6 +351,7 @@ struct kvm_vmx { bool ept_identity_pagetable_done; gpa_t ept_identity_map_addr; + hpa_t hv_tlb_eptp; enum ept_pointers_status ept_pointers_match; spinlock_t ept_pointer_lock; }; From patchwork Fri Mar 5 18:31:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12119177 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8F40BC4332D for ; Fri, 5 Mar 2021 18:32:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 69A5865085 for ; Fri, 5 Mar 2021 18:32:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229837AbhCESbq (ORCPT ); Fri, 5 Mar 2021 13:31:46 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58300 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229629AbhCESbg (ORCPT ); Fri, 5 Mar 2021 13:31:36 -0500 Received: from mail-qk1-x74a.google.com (mail-qk1-x74a.google.com [IPv6:2607:f8b0:4864:20::74a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 59B23C061574 for ; Fri, 5 Mar 2021 10:31:36 -0800 (PST) Received: by mail-qk1-x74a.google.com with SMTP id i188so2459774qkd.7 for ; Fri, 05 Mar 2021 10:31:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=2I8FvDRJX4diQtgWxReNsunkTG6GVrDEzb4ROeqCpCA=; b=gUWb+oGx2kmyiKIq7Ji4QKdmYRwNg3ql6n8PT1ctaw7mpo68o5gZ3QR/clao5244n7 RLSZg54MdHDyYUUZbZSg7pSSSDxqTFK35WRCh9TLJDMon7yETm0ZDslaiTKePL+2J+ci KapX3/QSdD6EJK0914fBO9wFclKZvDiGvRY9SeNxVAXKxKO5ISW98V84d+dkOVcgNavV aGQMy19sB5cdvnJC3+OS/y7En2p1w/rLTDT0pmgPjUk09mqZzQ7PLxNqeFY5615PpSS2 Q7nD44i6rh5uxgk9F8aS0M4az6PBuhxw4INFgT32HnMimcYnPjj/m8wKi5UCr5gB9lW1 6MnQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=2I8FvDRJX4diQtgWxReNsunkTG6GVrDEzb4ROeqCpCA=; b=C9gecCPWjQzn7t7xtHqyFkY4RJtKocquypiu81K2pmhk+yVG45ogFuU6Rf//0JK9ra Ew2ujU2Q64iI3ltWjetOW47/e9rjMojBvvUD1Aeya7I8XNj8SScy+4rJIau8+4KmUEzo sAVsfPDTVcqLMT3bpGnf4v4QXrGKmI81sK9JNpjcfxsN3qNnvPINt/THjmszdqV1KeVo 4vfQRDDcaezbHISL6ecEwWFy2oSlV8tu5kj85tFHquZ5UTxUlPx1c7kT2n7tv4zmF7wE dBE0wS+UVYmpN10npLu2hOfuoaMfiFeJTxatQTC4LwMl/naNb1CSiZFyxt/gKRdXnefA 8h/Q== X-Gm-Message-State: AOAM530yqMePIzTU0o/q2WiWdCkgpnnJTicuiQ+HWV7+P/B7nKRLI37v ChZWX2DcoPN3hEmlvJjjiQ2exEd1tkw= X-Google-Smtp-Source: ABdhPJzVgmTYSKJD6HI2e4kOKYb2BPoDBr6v0KNAO2kWQJUtrVakt+9VMJN58RNlGc+GxEZZgv3mjEximF0= Sender: "seanjc via sendgmr" X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:9857:be95:97a2:e91c]) (user=seanjc job=sendgmr) by 2002:a0c:d6c8:: with SMTP id l8mr10149663qvi.1.1614969095547; Fri, 05 Mar 2021 10:31:35 -0800 (PST) Reply-To: Sean Christopherson Date: Fri, 5 Mar 2021 10:31:15 -0800 In-Reply-To: <20210305183123.3978098-1-seanjc@google.com> Message-Id: <20210305183123.3978098-4-seanjc@google.com> Mime-Version: 1.0 References: <20210305183123.3978098-1-seanjc@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v4 03/11] KVM: VMX: Stash kvm_vmx in a local variable for Hyper-V paravirt TLB flush From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Sean Christopherson Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Sean Christopherson Capture kvm_vmx in a local variable instead of polluting hv_remote_flush_tlb_with_range() with to_kvm_vmx(kvm). No functional change intended. Reviewed-by: Vitaly Kuznetsov Signed-off-by: Sean Christopherson Signed-off-by: Sean Christopherson --- arch/x86/kvm/vmx/vmx.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 4082c7a26612..cf79fc6c01bb 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -520,26 +520,27 @@ static inline int hv_remote_flush_eptp(u64 eptp, struct kvm_tlb_range *range) static int hv_remote_flush_tlb_with_range(struct kvm *kvm, struct kvm_tlb_range *range) { + struct kvm_vmx *kvm_vmx = to_kvm_vmx(kvm); struct kvm_vcpu *vcpu; int ret = 0, i; - spin_lock(&to_kvm_vmx(kvm)->ept_pointer_lock); + spin_lock(&kvm_vmx->ept_pointer_lock); - if (to_kvm_vmx(kvm)->ept_pointers_match == EPT_POINTERS_CHECK) + if (kvm_vmx->ept_pointers_match == EPT_POINTERS_CHECK) check_ept_pointer_match(kvm); - if (to_kvm_vmx(kvm)->ept_pointers_match != EPT_POINTERS_MATCH) { + if (kvm_vmx->ept_pointers_match != EPT_POINTERS_MATCH) { kvm_for_each_vcpu(i, vcpu, kvm) { /* If ept_pointer is invalid pointer, bypass flush request. */ if (VALID_PAGE(to_vmx(vcpu)->ept_pointer)) ret |= hv_remote_flush_eptp(to_vmx(vcpu)->ept_pointer, range); } - } else if (VALID_PAGE(to_kvm_vmx(kvm)->hv_tlb_eptp)) { - ret = hv_remote_flush_eptp(to_kvm_vmx(kvm)->hv_tlb_eptp, range); + } else if (VALID_PAGE(kvm_vmx->hv_tlb_eptp)) { + ret = hv_remote_flush_eptp(kvm_vmx->hv_tlb_eptp, range); } - spin_unlock(&to_kvm_vmx(kvm)->ept_pointer_lock); + spin_unlock(&kvm_vmx->ept_pointer_lock); return ret; } static int hv_remote_flush_tlb(struct kvm *kvm) From patchwork Fri Mar 5 18:31:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12119183 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BD22DC43332 for ; Fri, 5 Mar 2021 18:32:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8BD32650BF for ; Fri, 5 Mar 2021 18:32:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230093AbhCESbr (ORCPT ); Fri, 5 Mar 2021 13:31:47 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58312 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229906AbhCESbi (ORCPT ); Fri, 5 Mar 2021 13:31:38 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9799BC061756 for ; Fri, 5 Mar 2021 10:31:38 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id b127so3353499ybc.13 for ; Fri, 05 Mar 2021 10:31:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=4YINoE2f5i/FWyXIMOd2/gI6pVHQ38yw3mdEyw0YPEA=; b=lMu7b4dH22Ybw3HJnHv0sZaIAwHMSoQbg6RZEHzU7v9m9bxL18jZXP4/BmAs3p2909 aNLzfgExkDPjm1/lPx3xqhF8qREngm9n23xqLulPbctAc6Z4uVd/9mgUy+GbPrDX315P vU9sba68QLHhUxxM13nuaBkeo1qz5aH1RpTKZp6bs6JKewiGyfH2cR3tI5oYdiIu0dNi C3Y7jTvRRIoCnMb/XEmtyGvx/3jqUvrEfOFjycfgyVGGj9tBQormUSpiSRpiyjE8QN7D TyJMS4jK/Fb07J9mx1a4BoNHJyTR/hnjUew5zPAiXKVnxer4q0y6AC6OsFd7MFmOr5BJ I/Qg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=4YINoE2f5i/FWyXIMOd2/gI6pVHQ38yw3mdEyw0YPEA=; b=p9RTD0BgmNz5fIg5yMOQrLloMQbzTqlfF2D2vNV4B++BAKzFgTw7rihAA5KMeH+jMZ 36vHT+sPzlgY9boO0/HOQBprslQrca+gkxzbljiLU5qpCkxnMT6W8ZLgo5HI64splxyT GC6z/qrsT4kWL6chDJ0TVbzvSx6XpT3rUSYbpV6ZBLEVUjqEtQ7VY814hFbBcjuzPeoa Sbtq0vFb2cYvvKcEJFaPpV4jh6qRhtKnLGO1+FuMaIQy/RvVrR1rzEvHhHBv62VyPK8N QOGn2ryLx+UIle8kpPzYkrvD4fYsujAcs/tUieEetwM43F0grq0aSRLRXjQcwIIF84m/ 0nyg== X-Gm-Message-State: AOAM533x3wXf+Ir/Pv2b1bxH/efGRRe79CNgP8CYhiiP/qQM+ePAFAr3 nEBBNiok7mY3PlNv/DEeQeY2vD03KNY= X-Google-Smtp-Source: ABdhPJxuOqghynkRdvEOEwuikdEEE9eOWiqKQQRqyr7JZ7A0yy3dHoyMDl+yh5i4HXQaLgk3EiMPb2iqgTE= Sender: "seanjc via sendgmr" X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:9857:be95:97a2:e91c]) (user=seanjc job=sendgmr) by 2002:a25:dc85:: with SMTP id y127mr16205853ybe.198.1614969097895; Fri, 05 Mar 2021 10:31:37 -0800 (PST) Reply-To: Sean Christopherson Date: Fri, 5 Mar 2021 10:31:16 -0800 In-Reply-To: <20210305183123.3978098-1-seanjc@google.com> Message-Id: <20210305183123.3978098-5-seanjc@google.com> Mime-Version: 1.0 References: <20210305183123.3978098-1-seanjc@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v4 04/11] KVM: VMX: Fold Hyper-V EPTP checking into it's only caller From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Sean Christopherson Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Sean Christopherson Fold check_ept_pointer_match() into hv_remote_flush_tlb_with_range() in preparation for combining the kvm_for_each_vcpu loops of the ==CHECK and !=MATCH statements. No functional change intended. Reviewed-by: Vitaly Kuznetsov Signed-off-by: Sean Christopherson Signed-off-by: Sean Christopherson --- arch/x86/kvm/vmx/vmx.c | 44 +++++++++++++++++++----------------------- 1 file changed, 20 insertions(+), 24 deletions(-) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index cf79fc6c01bb..a1c7ba0918e7 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -472,28 +472,6 @@ static const u32 vmx_uret_msrs_list[] = { static bool __read_mostly enlightened_vmcs = true; module_param(enlightened_vmcs, bool, 0444); -/* check_ept_pointer() should be under protection of ept_pointer_lock. */ -static void check_ept_pointer_match(struct kvm *kvm) -{ - struct kvm_vcpu *vcpu; - u64 tmp_eptp = INVALID_PAGE; - int i; - - kvm_for_each_vcpu(i, vcpu, kvm) { - if (!VALID_PAGE(tmp_eptp)) { - tmp_eptp = to_vmx(vcpu)->ept_pointer; - } else if (tmp_eptp != to_vmx(vcpu)->ept_pointer) { - to_kvm_vmx(kvm)->hv_tlb_eptp = INVALID_PAGE; - to_kvm_vmx(kvm)->ept_pointers_match - = EPT_POINTERS_MISMATCH; - return; - } - } - - to_kvm_vmx(kvm)->hv_tlb_eptp = tmp_eptp; - to_kvm_vmx(kvm)->ept_pointers_match = EPT_POINTERS_MATCH; -} - static int kvm_fill_hv_flush_list_func(struct hv_guest_mapping_flush_list *flush, void *data) { @@ -523,11 +501,29 @@ static int hv_remote_flush_tlb_with_range(struct kvm *kvm, struct kvm_vmx *kvm_vmx = to_kvm_vmx(kvm); struct kvm_vcpu *vcpu; int ret = 0, i; + u64 tmp_eptp; spin_lock(&kvm_vmx->ept_pointer_lock); - if (kvm_vmx->ept_pointers_match == EPT_POINTERS_CHECK) - check_ept_pointer_match(kvm); + if (kvm_vmx->ept_pointers_match == EPT_POINTERS_CHECK) { + kvm_vmx->ept_pointers_match = EPT_POINTERS_MATCH; + kvm_vmx->hv_tlb_eptp = INVALID_PAGE; + + kvm_for_each_vcpu(i, vcpu, kvm) { + tmp_eptp = to_vmx(vcpu)->ept_pointer; + if (!VALID_PAGE(tmp_eptp)) + continue; + + if (!VALID_PAGE(kvm_vmx->hv_tlb_eptp)) { + kvm_vmx->hv_tlb_eptp = tmp_eptp; + } else if (kvm_vmx->hv_tlb_eptp != tmp_eptp) { + kvm_vmx->hv_tlb_eptp = INVALID_PAGE; + kvm_vmx->ept_pointers_match + = EPT_POINTERS_MISMATCH; + break; + } + } + } if (kvm_vmx->ept_pointers_match != EPT_POINTERS_MATCH) { kvm_for_each_vcpu(i, vcpu, kvm) { From patchwork Fri Mar 5 18:31:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12119181 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 98D1CC4332B for ; Fri, 5 Mar 2021 18:32:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 79BF3650A3 for ; Fri, 5 Mar 2021 18:32:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229899AbhCESbr (ORCPT ); Fri, 5 Mar 2021 13:31:47 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58324 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229951AbhCESbl (ORCPT ); Fri, 5 Mar 2021 13:31:41 -0500 Received: from mail-qt1-x849.google.com (mail-qt1-x849.google.com [IPv6:2607:f8b0:4864:20::849]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EC68EC061756 for ; Fri, 5 Mar 2021 10:31:40 -0800 (PST) Received: by mail-qt1-x849.google.com with SMTP id k4so2341561qtd.20 for ; Fri, 05 Mar 2021 10:31:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=QDgGIBKwEx0IcrZDmaovjYwei3TYW6SVG8g5soBde6c=; b=Y9E9rvso32rvxg65Z/hGLo9l2wp4yjO0ntQNV0P9/UGS6t6R3CyPnr85JRrrAgE9Eh PVMnt6GqL4jnmhArsOaxazCprp8wVxqf7e7yJX8CDRs29EGlQ/Zb1q5qs+owWMvCCM8n LVHK7W/DazTg3RpQ/ABsq5BOa+OJXIh6VCRO5+4PAiERZLjYx34nY58s62oNTH/aF8Bt UDivVUnKgy4PNaLzv+3VB8qjocaS7aVeKqwwEA00l89QD0pV/kbW2cWQ0UIw6AahSlMj gylyPaYlWxEkqAS5RHKkHcjv3sL0KQ8Zo+lSpw4uiK0rQXnObI6vs+m5kc0ZX+AqCIHB 4WFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=QDgGIBKwEx0IcrZDmaovjYwei3TYW6SVG8g5soBde6c=; b=KL4mP0ESqBChJtZS63QT6fIfC6ynnCaYzR87tyS7DncjGxu9pc4HXiLAYT6gShqPZ6 J/be0Vh9HibbZMpMnhATBtASRlPcM4fIEimsaqCRQVRMNwB/vyHpRe6WyG0F2zjGZPhU I0bOzTh0HZRpWzfz5UcqMv05z3wTy7AEj2EMyKcZVl9jFwDdg/L2ujFRqXdpAum8Djeq hUUb/xvI3u+e5utBPPKUs4rj5sz2lEzNPwl75mBM+cyfxicstxyc4KbpIrt4h3K/M0xc muaUOl5DdLcKB+JaGn+ASDs309My0aLk5SPRjPUpOwx5WjfFTmFp208yMmQcOPM72SbH dE+g== X-Gm-Message-State: AOAM532nxfiGtjgxpvhHjoZMb2U3kbJC/BREm+DsVRkMhIh2w9z231EL +7oNZDyeDCD62TjKyCeMpb6tAoah5Uk= X-Google-Smtp-Source: ABdhPJzIJ+98kwAqBiAVXJfxBB5p9HFDc8EdbKKTCNa1dh86kF2JNkLJNmkA+2iJJGRxAPZa7/e0wdXtCrI= Sender: "seanjc via sendgmr" X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:9857:be95:97a2:e91c]) (user=seanjc job=sendgmr) by 2002:a0c:a819:: with SMTP id w25mr10035806qva.6.1614969100131; Fri, 05 Mar 2021 10:31:40 -0800 (PST) Reply-To: Sean Christopherson Date: Fri, 5 Mar 2021 10:31:17 -0800 In-Reply-To: <20210305183123.3978098-1-seanjc@google.com> Message-Id: <20210305183123.3978098-6-seanjc@google.com> Mime-Version: 1.0 References: <20210305183123.3978098-1-seanjc@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v4 05/11] KVM: VMX: Do Hyper-V TLB flush iff vCPU's EPTP hasn't been flushed From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Sean Christopherson Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Sean Christopherson Combine the for-loops for Hyper-V TLB EPTP checking and flushing, and in doing so skip flushes for vCPUs whose EPTP matches the target EPTP. Reviewed-by: Vitaly Kuznetsov Signed-off-by: Sean Christopherson Signed-off-by: Sean Christopherson --- arch/x86/kvm/vmx/vmx.c | 23 ++++++++--------------- 1 file changed, 8 insertions(+), 15 deletions(-) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index a1c7ba0918e7..a1a5b411c903 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -505,33 +505,26 @@ static int hv_remote_flush_tlb_with_range(struct kvm *kvm, spin_lock(&kvm_vmx->ept_pointer_lock); - if (kvm_vmx->ept_pointers_match == EPT_POINTERS_CHECK) { + if (kvm_vmx->ept_pointers_match != EPT_POINTERS_MATCH) { kvm_vmx->ept_pointers_match = EPT_POINTERS_MATCH; kvm_vmx->hv_tlb_eptp = INVALID_PAGE; kvm_for_each_vcpu(i, vcpu, kvm) { tmp_eptp = to_vmx(vcpu)->ept_pointer; - if (!VALID_PAGE(tmp_eptp)) + if (!VALID_PAGE(tmp_eptp) || + tmp_eptp == kvm_vmx->hv_tlb_eptp) continue; - if (!VALID_PAGE(kvm_vmx->hv_tlb_eptp)) { + if (!VALID_PAGE(kvm_vmx->hv_tlb_eptp)) kvm_vmx->hv_tlb_eptp = tmp_eptp; - } else if (kvm_vmx->hv_tlb_eptp != tmp_eptp) { - kvm_vmx->hv_tlb_eptp = INVALID_PAGE; + else kvm_vmx->ept_pointers_match = EPT_POINTERS_MISMATCH; - break; - } - } - } - if (kvm_vmx->ept_pointers_match != EPT_POINTERS_MATCH) { - kvm_for_each_vcpu(i, vcpu, kvm) { - /* If ept_pointer is invalid pointer, bypass flush request. */ - if (VALID_PAGE(to_vmx(vcpu)->ept_pointer)) - ret |= hv_remote_flush_eptp(to_vmx(vcpu)->ept_pointer, - range); + ret |= hv_remote_flush_eptp(tmp_eptp, range); } + if (kvm_vmx->ept_pointers_match == EPT_POINTERS_MISMATCH) + kvm_vmx->hv_tlb_eptp = INVALID_PAGE; } else if (VALID_PAGE(kvm_vmx->hv_tlb_eptp)) { ret = hv_remote_flush_eptp(kvm_vmx->hv_tlb_eptp, range); } From patchwork Fri Mar 5 18:31:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12119185 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A9A62C433DB for ; Fri, 5 Mar 2021 18:35:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7C0F16509A for ; Fri, 5 Mar 2021 18:35:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229512AbhCESfP (ORCPT ); Fri, 5 Mar 2021 13:35:15 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58336 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229741AbhCESbn (ORCPT ); Fri, 5 Mar 2021 13:31:43 -0500 Received: from mail-qk1-x74a.google.com (mail-qk1-x74a.google.com [IPv6:2607:f8b0:4864:20::74a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 54B4BC061756 for ; Fri, 5 Mar 2021 10:31:43 -0800 (PST) Received: by mail-qk1-x74a.google.com with SMTP id i188so2460077qkd.7 for ; Fri, 05 Mar 2021 10:31:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=nktY5YypTxDE2PiApdZomAuTOvGlFQpDvQWISfSElkE=; b=GC/itasiXPuxLa1SrfizgHqvBQTVq6rY/d1sMzoZu0HxJrSwxfE1bK2gXS+BqEQxOt f5svi/6XBf9Ds7rQvhyBss8A/kiP7XIsD3N5drIlL7zy0ifNk2RazXmkAqKYDY2e2z+7 zju/INmjUxdyaWzKtMZvZvBJDxwrHWhVX9orFUsX1Uuv3fge0mmMYFGdm+bOt1aNnZMv NTOJ4hUyfkKb57P24oRiwyUn2yIFOssOEcuhEIVlzZ5zeLAq0+Me1L8m2SpetfIDhsDw dlhIndOpoHBpiZECkRiVW2WaRxR1lgRqDPVkTzwpXzOkhGyjJ8yLs8peeWCDBIs7pG+G Kcww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=nktY5YypTxDE2PiApdZomAuTOvGlFQpDvQWISfSElkE=; b=qMQg2HtsUt71iDgnXAKcRcPxKQZ4YrIlR36LBWUmEppdVV7oEDekZ07pc/y2vscZ7o a6BTFRBMMzxVIZSkeh1ErNwgiyz4KbOkR4rP+/T7JB27PRt7xRkNDA6RI4Ewf5XHErBB LTRPonUrjecAUAOSISuERKOdUFqWLOIuBxJ+pwT+Hnq/xksMEu0Fx56ZI0YBbfCwb4IG 4bcBF4cGBF/L6m52HoNd/nIwWxhQu8Jy+ix+V/obu6moiNFEobm6MgqCxFTzqiGjqBen pZ9Qc8GAcNRAhu6vQvS95kZJu5V6cKAXrSWtraSiMs+Fez+ZojZiJpk1kkZsFnCKSrsh 1AbA== X-Gm-Message-State: AOAM5330l+ggcX6CvRbbq1yjsNg0nUFqXv4i9UoG2JJQA5iohJMVFOc3 FW2O1ayuSQ1T5MljlYWzlPgReQcyPN8= X-Google-Smtp-Source: ABdhPJwZ+2c//1yanDKNKhMsNxKtczbRMMwVEh90as4xQI484nk2bKXYLo7uENk/cvEKqyPDFw3H/zHsS18= Sender: "seanjc via sendgmr" X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:9857:be95:97a2:e91c]) (user=seanjc job=sendgmr) by 2002:a05:6214:1424:: with SMTP id o4mr10183490qvx.34.1614969102470; Fri, 05 Mar 2021 10:31:42 -0800 (PST) Reply-To: Sean Christopherson Date: Fri, 5 Mar 2021 10:31:18 -0800 In-Reply-To: <20210305183123.3978098-1-seanjc@google.com> Message-Id: <20210305183123.3978098-7-seanjc@google.com> Mime-Version: 1.0 References: <20210305183123.3978098-1-seanjc@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v4 06/11] KVM: VMX: Invalidate hv_tlb_eptp to denote an EPTP mismatch From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Sean Christopherson Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Sean Christopherson Drop the dedicated 'ept_pointers_match' field in favor of stuffing 'hv_tlb_eptp' with INVALID_PAGE to mark it as invalid, i.e. to denote that there is at least one EPTP mismatch. Use a local variable to track whether or not a mismatch is detected so that hv_tlb_eptp can be used to skip redundant flushes. No functional change intended. Reviewed-by: Vitaly Kuznetsov Signed-off-by: Sean Christopherson Signed-off-by: Sean Christopherson --- arch/x86/kvm/vmx/vmx.c | 35 +++++++++++++++++++++++------------ arch/x86/kvm/vmx/vmx.h | 7 ------- 2 files changed, 23 insertions(+), 19 deletions(-) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index a1a5b411c903..2582df5ae64d 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -500,32 +500,44 @@ static int hv_remote_flush_tlb_with_range(struct kvm *kvm, { struct kvm_vmx *kvm_vmx = to_kvm_vmx(kvm); struct kvm_vcpu *vcpu; - int ret = 0, i; + int ret = 0, i, nr_unique_valid_eptps; u64 tmp_eptp; spin_lock(&kvm_vmx->ept_pointer_lock); - if (kvm_vmx->ept_pointers_match != EPT_POINTERS_MATCH) { - kvm_vmx->ept_pointers_match = EPT_POINTERS_MATCH; - kvm_vmx->hv_tlb_eptp = INVALID_PAGE; + if (!VALID_PAGE(kvm_vmx->hv_tlb_eptp)) { + nr_unique_valid_eptps = 0; + /* + * Flush all valid EPTPs, and see if all vCPUs have converged + * on a common EPTP, in which case future flushes can skip the + * loop and flush the common EPTP. + */ kvm_for_each_vcpu(i, vcpu, kvm) { tmp_eptp = to_vmx(vcpu)->ept_pointer; if (!VALID_PAGE(tmp_eptp) || tmp_eptp == kvm_vmx->hv_tlb_eptp) continue; - if (!VALID_PAGE(kvm_vmx->hv_tlb_eptp)) + /* + * Set the tracked EPTP to the first valid EPTP. Keep + * this EPTP for the entirety of the loop even if more + * EPTPs are encountered as a low effort optimization + * to avoid flushing the same (first) EPTP again. + */ + if (++nr_unique_valid_eptps == 1) kvm_vmx->hv_tlb_eptp = tmp_eptp; - else - kvm_vmx->ept_pointers_match - = EPT_POINTERS_MISMATCH; ret |= hv_remote_flush_eptp(tmp_eptp, range); } - if (kvm_vmx->ept_pointers_match == EPT_POINTERS_MISMATCH) + + /* + * The optimized flush of a single EPTP can't be used if there + * are multiple valid EPTPs (obviously). + */ + if (nr_unique_valid_eptps > 1) kvm_vmx->hv_tlb_eptp = INVALID_PAGE; - } else if (VALID_PAGE(kvm_vmx->hv_tlb_eptp)) { + } else { ret = hv_remote_flush_eptp(kvm_vmx->hv_tlb_eptp, range); } @@ -3105,8 +3117,7 @@ static void vmx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, if (kvm_x86_ops.tlb_remote_flush) { spin_lock(&to_kvm_vmx(kvm)->ept_pointer_lock); to_vmx(vcpu)->ept_pointer = eptp; - to_kvm_vmx(kvm)->ept_pointers_match - = EPT_POINTERS_CHECK; + to_kvm_vmx(kvm)->hv_tlb_eptp = INVALID_PAGE; spin_unlock(&to_kvm_vmx(kvm)->ept_pointer_lock); } diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h index f846cf3a5d25..fb7b2000bd0e 100644 --- a/arch/x86/kvm/vmx/vmx.h +++ b/arch/x86/kvm/vmx/vmx.h @@ -338,12 +338,6 @@ struct vcpu_vmx { } shadow_msr_intercept; }; -enum ept_pointers_status { - EPT_POINTERS_CHECK = 0, - EPT_POINTERS_MATCH = 1, - EPT_POINTERS_MISMATCH = 2 -}; - struct kvm_vmx { struct kvm kvm; @@ -352,7 +346,6 @@ struct kvm_vmx { gpa_t ept_identity_map_addr; hpa_t hv_tlb_eptp; - enum ept_pointers_status ept_pointers_match; spinlock_t ept_pointer_lock; }; From patchwork Fri Mar 5 18:31:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12119187 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 293EBC433E0 for ; Fri, 5 Mar 2021 18:35:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 069296508B for ; Fri, 5 Mar 2021 18:35:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230445AbhCESfU (ORCPT ); Fri, 5 Mar 2021 13:35:20 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58348 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230070AbhCESbq (ORCPT ); Fri, 5 Mar 2021 13:31:46 -0500 Received: from mail-qv1-xf49.google.com (mail-qv1-xf49.google.com [IPv6:2607:f8b0:4864:20::f49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A9C97C06175F for ; Fri, 5 Mar 2021 10:31:45 -0800 (PST) Received: by mail-qv1-xf49.google.com with SMTP id u8so2199550qvm.5 for ; Fri, 05 Mar 2021 10:31:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=Rsbn/HNTRnyhwzeN24lyp/WE9lyzNH62hzhEpuOPsV4=; b=dAtMvBa14Eut/SotL7TsMQxj7t6qm67K1MBE0aFLgD4dvMy1D6xhg+EMaRCiPdTU2+ sv5+o+L0DiSjKn2qIUuT+bkkzHfGr3lZDVdLdQtEg30sXsI70ymiur5/GiOAMamtenFy w/zavTYp1m/sPOeZyRv4P70b9Jywhhvaiky+9NVEK3OpDd4DcCxwz2tat+HZ29yLjjFr ItWjxfUeEj1GCpiVJ94JOxG7dAXi/vDboav5uQbBR5/8jZkRqMaQ3bFwmdt98D2PVEc5 9yatRPsnJgyWKz4nABzoL9CHAuEkHVkMFvV9QJIeURc6GlddLtyrn11p/JiDdI9J0Ut0 2yBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=Rsbn/HNTRnyhwzeN24lyp/WE9lyzNH62hzhEpuOPsV4=; b=Q0nLnYdvUbEXL8VOg71P2QI5FHiA+K90I9ByD2ycaQ2QutOFR0mfbUdXpcMCfnLu+W Vlyp1EwD+m86D/oQM1DwKxXiaC17FaJqtTNF36kK2ZUbzMuJTHSLnr3p29wN/brskvVK WVpeIvytk/0zZdqSgZuLiO0tYZ092yGU9iNtYRGEAxo/N9FF3sRtog4hReTS/7MuJv7K uvxsuBWrlCAyDUoL+tNNple+nLGnpslrfNJmdEKsAbGcl4kgojtiMjt0cq7idLkQMEv0 osq2uv5VX8/jYq3xJDk2V5ICLYUTgD1fWTwSg5NzuCXZJD0Fx7lrNTm+Ult/ZBK3lBcr E/QA== X-Gm-Message-State: AOAM530P05b93+GEdH4Co3EElWcR8E8nM/F9UNR4p0p1651ADAEaMKD2 e1oTmAcBPq4XG66C9S2ErCegXc45j7o= X-Google-Smtp-Source: ABdhPJyjcCNm7PSQal+eam3lizqLiyYhUUXvYHe2LS3onAtipn2RPNvZoZaLI9usNDeoKN9oVK6KYzpeyZI= Sender: "seanjc via sendgmr" X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:9857:be95:97a2:e91c]) (user=seanjc job=sendgmr) by 2002:a05:6214:4ae:: with SMTP id w14mr10001102qvz.45.1614969104803; Fri, 05 Mar 2021 10:31:44 -0800 (PST) Reply-To: Sean Christopherson Date: Fri, 5 Mar 2021 10:31:19 -0800 In-Reply-To: <20210305183123.3978098-1-seanjc@google.com> Message-Id: <20210305183123.3978098-8-seanjc@google.com> Mime-Version: 1.0 References: <20210305183123.3978098-1-seanjc@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v4 07/11] KVM: VMX: Don't invalidate hv_tlb_eptp if the new EPTP matches From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Sean Christopherson Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Sean Christopherson Don't invalidate the common EPTP, and thus trigger rechecking of EPTPs across all vCPUs, if the new EPTP matches the old/common EPTP. In all likelihood this is a meaningless optimization, but there are (uncommon) scenarios where KVM can reload the same EPTP. Reviewed-by: Vitaly Kuznetsov Signed-off-by: Sean Christopherson Signed-off-by: Sean Christopherson --- arch/x86/kvm/vmx/vmx.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 2582df5ae64d..cd0d787b0a85 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -3117,7 +3117,8 @@ static void vmx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, if (kvm_x86_ops.tlb_remote_flush) { spin_lock(&to_kvm_vmx(kvm)->ept_pointer_lock); to_vmx(vcpu)->ept_pointer = eptp; - to_kvm_vmx(kvm)->hv_tlb_eptp = INVALID_PAGE; + if (eptp != to_kvm_vmx(kvm)->hv_tlb_eptp) + to_kvm_vmx(kvm)->hv_tlb_eptp = INVALID_PAGE; spin_unlock(&to_kvm_vmx(kvm)->ept_pointer_lock); } From patchwork Fri Mar 5 18:31:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12119191 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 539CFC43381 for ; Fri, 5 Mar 2021 18:35:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2D2CA6509A for ; Fri, 5 Mar 2021 18:35:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229601AbhCESfV (ORCPT ); Fri, 5 Mar 2021 13:35:21 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58364 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230124AbhCESbs (ORCPT ); Fri, 5 Mar 2021 13:31:48 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E5E20C061574 for ; Fri, 5 Mar 2021 10:31:47 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id p136so3366715ybc.21 for ; Fri, 05 Mar 2021 10:31:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=3vB4Y/h6QDVL+3j4AiUrMRB5AA+2aPlfzfXf1TsAcOY=; b=os6DcV8JeNVxhBoTQ0eq0PFgNN0naOyIneLuR0FftGMvBQYnR0HxIUpns3/gN3dAhA gXMx9aMHvSH6tNVqLL6EqF6ebpnvPb0u5RB1/aLKxOmkG6ZFKleGVdjloh8jXy1Lsp6a pPQI+h5Xp+a89H90mHiG1kbWbJT8JQhIjnpvrS7YvW1fQIyrykCQWIGEO92vvmxxCPKh hENHbrbtQ96/MfnKhz6mRnLFsh5mabxcrIVfyQViWeBKUqQ/lWwWWjj4OCNajS1IwgNX KM11CVMMLNdgUhopA4kr0qnphyzP0tiBk3ZHPiNYT959AC6ueUPpi+SPPA/7rI06A5Zr HKMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=3vB4Y/h6QDVL+3j4AiUrMRB5AA+2aPlfzfXf1TsAcOY=; b=dVmodbCEzWfvPZwMqllr2tPQ3vdCnl9gOBznYmlIUIjTbB8STSZdw2kpSVGXJTqR/b 7yhcSFp62CqUkBt8sjbsUs9FFyFymVa5TDy8mS0KwbsTX9X/GI/2DSqVf+nph35y/fcy 6K6nJVpT5RVoCDTDe0F7ID8JHsayCt2M6uzaCzyEs3622UjfKNZsVzntkpu0omULf1b5 nbPSM5EGpOP9L5b3XkzJrZP/Zua2zgBXYDcxiMqxFs1mTIawLTvUM9PmM9fuAhsrI1o9 0vlx/DcVHnGyLuSb0U6F0hQ9Vro5K/8RX65CVIeMEhNSv6TOsk1oxwSGfMBeWkNPgcC2 fZVw== X-Gm-Message-State: AOAM531AxMyD+a38Kq3BT6sgap2lmZchSOqRvlP54SE3qLp8lFtkLDdn 9gtWUs6gcqSabGEeyaKegBbNd73Be0Y= X-Google-Smtp-Source: ABdhPJyEL75Wk6hzvC7QA1WtSNY6jXvBvzJrkAD2foYDx8aOEcCNw123tSMzGMDg5zeo43zgHLAmOYSm8e8= Sender: "seanjc via sendgmr" X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:9857:be95:97a2:e91c]) (user=seanjc job=sendgmr) by 2002:a25:ab6a:: with SMTP id u97mr15053279ybi.288.1614969107235; Fri, 05 Mar 2021 10:31:47 -0800 (PST) Reply-To: Sean Christopherson Date: Fri, 5 Mar 2021 10:31:20 -0800 In-Reply-To: <20210305183123.3978098-1-seanjc@google.com> Message-Id: <20210305183123.3978098-9-seanjc@google.com> Mime-Version: 1.0 References: <20210305183123.3978098-1-seanjc@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v4 08/11] KVM: VMX: Explicitly check for hv_remote_flush_tlb when loading pgd From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Sean Christopherson Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Sean Christopherson Explicitly check that kvm_x86_ops.tlb_remote_flush() points at Hyper-V's implementation for PV flushing instead of assuming that a non-NULL implementation means running on Hyper-V. Wrap the related logic in ifdeffery as hv_remote_flush_tlb() is defined iff CONFIG_HYPERV!=n. Short term, the explicit check makes it more obvious why a non-NULL tlb_remote_flush() triggers EPTP shenanigans. Long term, this will allow TDX to define its own implementation of tlb_remote_flush() without running afoul of Hyper-V. Reviewed-by: Vitaly Kuznetsov Signed-off-by: Sean Christopherson Signed-off-by: Sean Christopherson --- arch/x86/kvm/vmx/vmx.c | 23 ++++++++++++++++------- 1 file changed, 16 insertions(+), 7 deletions(-) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index cd0d787b0a85..aca6849c409a 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -576,6 +576,21 @@ static int hv_enable_direct_tlbflush(struct kvm_vcpu *vcpu) #endif /* IS_ENABLED(CONFIG_HYPERV) */ +static void hv_load_mmu_eptp(struct kvm_vcpu *vcpu, u64 eptp) +{ +#if IS_ENABLED(CONFIG_HYPERV) + struct kvm_vmx *kvm_vmx = to_kvm_vmx(vcpu->kvm); + + if (kvm_x86_ops.tlb_remote_flush == hv_remote_flush_tlb) { + spin_lock(&kvm_vmx->ept_pointer_lock); + to_vmx(vcpu)->ept_pointer = eptp; + if (eptp != kvm_vmx->hv_tlb_eptp) + kvm_vmx->hv_tlb_eptp = INVALID_PAGE; + spin_unlock(&kvm_vmx->ept_pointer_lock); + } +#endif +} + /* * Comment's format: document - errata name - stepping - processor name. * Refer from @@ -3114,13 +3129,7 @@ static void vmx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, eptp = construct_eptp(vcpu, root_hpa, root_level); vmcs_write64(EPT_POINTER, eptp); - if (kvm_x86_ops.tlb_remote_flush) { - spin_lock(&to_kvm_vmx(kvm)->ept_pointer_lock); - to_vmx(vcpu)->ept_pointer = eptp; - if (eptp != to_kvm_vmx(kvm)->hv_tlb_eptp) - to_kvm_vmx(kvm)->hv_tlb_eptp = INVALID_PAGE; - spin_unlock(&to_kvm_vmx(kvm)->ept_pointer_lock); - } + hv_load_mmu_eptp(vcpu, eptp); if (!enable_unrestricted_guest && !is_paging(vcpu)) guest_cr3 = to_kvm_vmx(kvm)->ept_identity_map_addr; From patchwork Fri Mar 5 18:31:21 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12119189 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6869DC433E9 for ; Fri, 5 Mar 2021 18:35:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3E913650A5 for ; Fri, 5 Mar 2021 18:35:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230458AbhCESfW (ORCPT ); Fri, 5 Mar 2021 13:35:22 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58376 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230155AbhCESbu (ORCPT ); Fri, 5 Mar 2021 13:31:50 -0500 Received: from mail-qk1-x749.google.com (mail-qk1-x749.google.com [IPv6:2607:f8b0:4864:20::749]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 681EFC061756 for ; Fri, 5 Mar 2021 10:31:50 -0800 (PST) Received: by mail-qk1-x749.google.com with SMTP id a1so2455467qkn.11 for ; Fri, 05 Mar 2021 10:31:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=DQ7Tmt2gW4abXYiFpma/L2uFHht8uCDneBUGK83ED/s=; b=JFgVTA12GZBrLquq22LzialuSwFqoDDQbHjexYDuqEkeP8zWA1xwrprg77bw1FiwZA E0zSW/dGVYPRpD5hhtW+qkZLCLCUaP6kIJpyx6nX/5jdB4aQ8KPt3YUv1GqbT5fMZBjg 4SO4ojwLrqrEMol0w+elnM5kjBDfD2sfBN7TzQZWWi7sIv6sxUZpq3jp4A9O4c1zuesc CwBbZp+CJq4IcuZTM35esAY22qMiNhstivQ9gY2lNiONxxbngn75zQusKFqAnp7UM9RJ +hZNjle+2cREfm+GTBdb4f9Zpwm0ts5sdNVZSgOJxzg1/4j3gGIb03SFs461r9QueXfl +QWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=DQ7Tmt2gW4abXYiFpma/L2uFHht8uCDneBUGK83ED/s=; b=rKlEr8TBYA520+KzaG5upnNRxEiCSOexqHrk7yoFjLgCwF4/T1oMIjMqR4DT5hxXRT SAUaokwrAF7H53jhMysqV62pBoJ3BBuzbZdZdQTvbRvBJPDXddrlfX/50arYEUCZhmgL eopx0PaLQs2fF/P1dMzlYg7Z+ITZ1J41UeWY9YsZoX49D9jyFFn8wtMkxeoN/Po3sRHN +pmEQ5Ob/c+OuhpaZbsMVFSHRYO6tSCZ6tq9WUrX9UZ/zyx1tmBlqNeT8WLdm/N7/Dbe G8eXi9YEXj3tYjuZZ4Q//kV5hh/DWx0a8yhZSthQt+j3Bb7eI+9pIRb6dj0A6nuieMjj W81Q== X-Gm-Message-State: AOAM533CeEfp/Z25wqQykrCURmuQs+qUXfhLzinfYs4veEXBMTK+rRfQ ZXgp9Y4n5Qt3R9VZtSRX07/PizF26tI= X-Google-Smtp-Source: ABdhPJzeAZr63UQ+tLroOoFR4kgoHB5AA3OzwqeYoBZjOIKebTedIJ3Bfhed6XfDiGAHRCSyCFrIGTxV7ZA= Sender: "seanjc via sendgmr" X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:9857:be95:97a2:e91c]) (user=seanjc job=sendgmr) by 2002:ad4:5614:: with SMTP id ca20mr10084795qvb.37.1614969109574; Fri, 05 Mar 2021 10:31:49 -0800 (PST) Reply-To: Sean Christopherson Date: Fri, 5 Mar 2021 10:31:21 -0800 In-Reply-To: <20210305183123.3978098-1-seanjc@google.com> Message-Id: <20210305183123.3978098-10-seanjc@google.com> Mime-Version: 1.0 References: <20210305183123.3978098-1-seanjc@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v4 09/11] KVM: VMX: Define Hyper-V paravirt TLB flush fields iff Hyper-V is enabled From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Sean Christopherson Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Sean Christopherson Ifdef away the Hyper-V specific fields in structs kvm_vmx and vcpu_vmx as each field has only a single reference outside of the struct itself that isn't already wrapped in ifdeffery (and both are initialization). vcpu_vmx.ept_pointer in particular should be wrapped as it is valid if and only if Hyper-v is active, i.e. non-Hyper-V code cannot rely on it to actually track the current EPTP (without additional code changes). Reviewed-by: Vitaly Kuznetsov Signed-off-by: Sean Christopherson Signed-off-by: Sean Christopherson --- arch/x86/kvm/vmx/vmx.c | 5 ++++- arch/x86/kvm/vmx/vmx.h | 4 ++++ 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index aca6849c409a..78bda73173d2 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -6920,8 +6920,9 @@ static int vmx_create_vcpu(struct kvm_vcpu *vcpu) vmx->pi_desc.nv = POSTED_INTR_VECTOR; vmx->pi_desc.sn = 1; +#if IS_ENABLED(CONFIG_HYPERV) vmx->ept_pointer = INVALID_PAGE; - +#endif return 0; free_vmcs: @@ -6938,7 +6939,9 @@ static int vmx_create_vcpu(struct kvm_vcpu *vcpu) static int vmx_vm_init(struct kvm *kvm) { +#if IS_ENABLED(CONFIG_HYPERV) spin_lock_init(&to_kvm_vmx(kvm)->ept_pointer_lock); +#endif if (!ple_gap) kvm->arch.pause_in_guest = true; diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h index fb7b2000bd0e..6d97b5a64b62 100644 --- a/arch/x86/kvm/vmx/vmx.h +++ b/arch/x86/kvm/vmx/vmx.h @@ -325,7 +325,9 @@ struct vcpu_vmx { */ u64 msr_ia32_feature_control; u64 msr_ia32_feature_control_valid_bits; +#if IS_ENABLED(CONFIG_HYPERV) u64 ept_pointer; +#endif struct pt_desc pt_desc; struct lbr_desc lbr_desc; @@ -345,8 +347,10 @@ struct kvm_vmx { bool ept_identity_pagetable_done; gpa_t ept_identity_map_addr; +#if IS_ENABLED(CONFIG_HYPERV) hpa_t hv_tlb_eptp; spinlock_t ept_pointer_lock; +#endif }; bool nested_vmx_allowed(struct kvm_vcpu *vcpu); From patchwork Fri Mar 5 18:31:22 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12119193 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 69A36C4332D for ; Fri, 5 Mar 2021 18:35:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 563A6650B7 for ; Fri, 5 Mar 2021 18:35:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230471AbhCESfY (ORCPT ); Fri, 5 Mar 2021 13:35:24 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58386 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229688AbhCESbw (ORCPT ); Fri, 5 Mar 2021 13:31:52 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C357CC061574 for ; Fri, 5 Mar 2021 10:31:52 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id y7so3333238ybh.20 for ; Fri, 05 Mar 2021 10:31:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=VJQbdhbCWyNk4Q85Oh9qgYOeUZXmpotWEbehzwgzXvA=; b=tpcTqy0NmGqO15Oj70zENzNSj2GWhEUMFj4hFwqOjkZwT+4+qtw4ahDONdmhqYAyCZ mOiIVvss7585J72kdpTDQSCCUFxa94tFXNEnuuk8K5x6ToOwvX4fhaBt/+C03Pqzcjx5 zUHUz7S7UDzMisaw5izb/WkZifGVDlVfb+msPrM32ydHOLCAgcW/u++kFxQnM+J60OT4 aS/nnZp9uN/uIUY6WqpVrPh7/I6vUb8+/dblyuaAM5iguQCvJU+qDz2RU4hGqQt4N1DU y7pBpWz3AwtUa5w7Po4cOjywIEx+za/Id29wZ4zrt2u31V4zi83jik3LkTpAdXEuM49m 1FRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=VJQbdhbCWyNk4Q85Oh9qgYOeUZXmpotWEbehzwgzXvA=; b=kYbFiWG+wuN/VjevVCbF3+Tc8Nb17rY9BVtI/Y83TzCqu5dpHwe/kwoMbgxraNZskn Tlb80bNK36E+JlkJXob2Os+NIe7efv3fIT6n7Tkm0kw3ixrxnJddzhT0LQkqnXyTONSE rb5NTKQUpJKYSdGmgKHd1GBfsM2zSeW04fbPSDHSgYdsS+NcIux7SK94fQWgEfAhaeKS YdT/NQFPghRGtbs3vGHpX+nLL6aMjHhNzwq2EPIl+A0XQzVQiwfL72RsFSKoTliQJ5i6 Y4QNFJbmC8oEmcyd/HuNf6oV0MQ7kEfMYciz+fYit8yUTjXE0MLz2uCTFDW8HWY92Imc 6oyw== X-Gm-Message-State: AOAM531RY+ogZbSP5jsVVyjNXX2peggwaiidfa3AGpmIpxEWOmegurOL pyGv5j6ilOSHsxD9bRIirv2wOwc5cCg= X-Google-Smtp-Source: ABdhPJy2oVGL1F/enUJcmeJKH8KtY1MgerK3ptc1eh7T8ht+g7Fgpuv3AMeMcmd3+9aJmwQOAIx825UbtRI= Sender: "seanjc via sendgmr" X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:9857:be95:97a2:e91c]) (user=seanjc job=sendgmr) by 2002:a25:2cce:: with SMTP id s197mr15752377ybs.88.1614969112047; Fri, 05 Mar 2021 10:31:52 -0800 (PST) Reply-To: Sean Christopherson Date: Fri, 5 Mar 2021 10:31:22 -0800 In-Reply-To: <20210305183123.3978098-1-seanjc@google.com> Message-Id: <20210305183123.3978098-11-seanjc@google.com> Mime-Version: 1.0 References: <20210305183123.3978098-1-seanjc@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v4 10/11] KVM: VMX: Skip additional Hyper-V TLB EPTP flushes if one fails From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Sean Christopherson Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Sean Christopherson Skip additional EPTP flushes if one fails when processing EPTPs for Hyper-V's paravirt TLB flushing. If _any_ flush fails, KVM falls back to a full global flush, i.e. additional flushes are unnecessary (and will likely fail anyways). Continue processing the loop unless a mismatch was already detected, e.g. to handle the case where the first flush fails and there is a yet-to-be-detected mismatch. Reviewed-by: Vitaly Kuznetsov Signed-off-by: Sean Christopherson Signed-off-by: Sean Christopherson --- arch/x86/kvm/vmx/vmx.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 78bda73173d2..720dcfe2a57d 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -528,7 +528,15 @@ static int hv_remote_flush_tlb_with_range(struct kvm *kvm, if (++nr_unique_valid_eptps == 1) kvm_vmx->hv_tlb_eptp = tmp_eptp; - ret |= hv_remote_flush_eptp(tmp_eptp, range); + if (!ret) + ret = hv_remote_flush_eptp(tmp_eptp, range); + + /* + * Stop processing EPTPs if a failure occurred and + * there is already a detected EPTP mismatch. + */ + if (ret && nr_unique_valid_eptps > 1) + break; } /* From patchwork Fri Mar 5 18:31:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12119195 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EB9F4C43331 for ; Fri, 5 Mar 2021 18:35:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B63C965085 for ; Fri, 5 Mar 2021 18:35:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230476AbhCESfY (ORCPT ); Fri, 5 Mar 2021 13:35:24 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58400 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230194AbhCESbz (ORCPT ); Fri, 5 Mar 2021 13:31:55 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 098ECC061756 for ; Fri, 5 Mar 2021 10:31:55 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id v6so3376314ybk.9 for ; Fri, 05 Mar 2021 10:31:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=oPnB+15Iks7tPihY57VTLY3xzIc6/gS7fJ46t5oskwQ=; b=cXraSC8Gn8J0EJA/KmySSEC0IcGtyz9uOZmMQR0FryXSvIJvLVvuRrmZP1Q7ENv2vM zKHtE4wLxIAHMtUTTp3rPg+/rc7g0B5e56tIKi/oiDty5qodIEM6blAhaJ4KpMaTPVQb 0fP/uA1TXyY7QJ+NhfGT4uuvITF5/qiU38BMjweRzIVbIkIP03KSRN0V+xW6/YW8ocpw 2+2vFinOxxg8q79YXOfUlYUH6AA2KbZriu/KP8eyC7vWOXVOnWxPnGBRPBq8xqH7otvX XHn3rw/ddddY5X9ppDQSN1YbYffMrb+dzy4mv8Z3wytM+h6orKWP78DEAO+x589jXSyI 4Iww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=oPnB+15Iks7tPihY57VTLY3xzIc6/gS7fJ46t5oskwQ=; b=KH+lcRt3zCFwHaQM5V6+Xb9doH6GnstiZ8z2Ibrwzs9seTMUJ8zPoUQdorq6rzdrik t9dqsHV/KKonT1HopQjcDEsZ+d0vqmgvuKoGUepE9W0BVYgmGSZBxtVE7gz4PJHGd1OA JRD7+fUdvbFJM4TSF7uY299yO6nrZm50BlUuKu8QMAM5d+xKB0B5MbTwWBNrzvSMhJcS 7rc5wkLUBq35KKNu7gW7RzMKr3YKi+y+WfufZ5d6/WtVSLjM9FzmS7ex6oKni+cBZb9A /IxoXZpGG7kLmGdmCatNyLtQNUHLUlIVQJsTvXf76PcPyBYrOP2LXgeY1hugA+cji7Zo cqOQ== X-Gm-Message-State: AOAM531Ls8hoaBc4Xw0/s5ln9iccS85A7a3xrzomqQrCY9hB4dCozQ5U KLLVepGsFRnW/EKxtVi97ZfO4my8RLs= X-Google-Smtp-Source: ABdhPJxvzzCdnXSgD6gNL8R+EsmkLrp9RaC/yeM4CegXhGkzvzA71cjvMwng2d1RCujiYhEozU/+oiTk3ao= Sender: "seanjc via sendgmr" X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:9857:be95:97a2:e91c]) (user=seanjc job=sendgmr) by 2002:a25:c090:: with SMTP id c138mr15081978ybf.314.1614969114273; Fri, 05 Mar 2021 10:31:54 -0800 (PST) Reply-To: Sean Christopherson Date: Fri, 5 Mar 2021 10:31:23 -0800 In-Reply-To: <20210305183123.3978098-1-seanjc@google.com> Message-Id: <20210305183123.3978098-12-seanjc@google.com> Mime-Version: 1.0 References: <20210305183123.3978098-1-seanjc@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v4 11/11] KVM: VMX: Track root HPA instead of EPTP for paravirt Hyper-V TLB flush From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Sean Christopherson Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Sean Christopherson Track the address of the top-level EPT struct, a.k.a. the root HPA, instead of the EPTP itself for Hyper-V's paravirt TLB flush. The paravirt API takes only the address, not the full EPTP, and in theory tracking the EPTP could lead to false negatives, e.g. if the HPA matched but the attributes in the EPTP do not. In practice, such a mismatch is extremely unlikely, if not flat out impossible, given how KVM generates the EPTP. Opportunsitically rename the related fields to use the 'root' nomenclature, and to prefix them with 'hv_' to connect them to Hyper-V's paravirt TLB flushing. Reviewed-by: Vitaly Kuznetsov Signed-off-by: Sean Christopherson Signed-off-by: Sean Christopherson --- arch/x86/kvm/vmx/vmx.c | 83 ++++++++++++++++++++---------------------- arch/x86/kvm/vmx/vmx.h | 6 +-- 2 files changed, 42 insertions(+), 47 deletions(-) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 720dcfe2a57d..ef826594365f 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -481,18 +481,14 @@ static int kvm_fill_hv_flush_list_func(struct hv_guest_mapping_flush_list *flush range->pages); } -static inline int hv_remote_flush_eptp(u64 eptp, struct kvm_tlb_range *range) +static inline int hv_remote_flush_root_ept(hpa_t root_ept, + struct kvm_tlb_range *range) { - /* - * FLUSH_GUEST_PHYSICAL_ADDRESS_SPACE hypercall needs address - * of the base of EPT PML4 table, strip off EPT configuration - * information. - */ if (range) - return hyperv_flush_guest_mapping_range(eptp & PAGE_MASK, + return hyperv_flush_guest_mapping_range(root_ept, kvm_fill_hv_flush_list_func, (void *)range); else - return hyperv_flush_guest_mapping(eptp & PAGE_MASK); + return hyperv_flush_guest_mapping(root_ept); } static int hv_remote_flush_tlb_with_range(struct kvm *kvm, @@ -500,56 +496,55 @@ static int hv_remote_flush_tlb_with_range(struct kvm *kvm, { struct kvm_vmx *kvm_vmx = to_kvm_vmx(kvm); struct kvm_vcpu *vcpu; - int ret = 0, i, nr_unique_valid_eptps; - u64 tmp_eptp; + int ret = 0, i, nr_unique_valid_roots; + hpa_t root; - spin_lock(&kvm_vmx->ept_pointer_lock); + spin_lock(&kvm_vmx->hv_root_ept_lock); - if (!VALID_PAGE(kvm_vmx->hv_tlb_eptp)) { - nr_unique_valid_eptps = 0; + if (!VALID_PAGE(kvm_vmx->hv_root_ept)) { + nr_unique_valid_roots = 0; /* - * Flush all valid EPTPs, and see if all vCPUs have converged - * on a common EPTP, in which case future flushes can skip the - * loop and flush the common EPTP. + * Flush all valid roots, and see if all vCPUs have converged + * on a common root, in which case future flushes can skip the + * loop and flush the common root. */ kvm_for_each_vcpu(i, vcpu, kvm) { - tmp_eptp = to_vmx(vcpu)->ept_pointer; - if (!VALID_PAGE(tmp_eptp) || - tmp_eptp == kvm_vmx->hv_tlb_eptp) + root = to_vmx(vcpu)->hv_root_ept; + if (!VALID_PAGE(root) || root == kvm_vmx->hv_root_ept) continue; /* - * Set the tracked EPTP to the first valid EPTP. Keep - * this EPTP for the entirety of the loop even if more - * EPTPs are encountered as a low effort optimization - * to avoid flushing the same (first) EPTP again. + * Set the tracked root to the first valid root. Keep + * this root for the entirety of the loop even if more + * roots are encountered as a low effort optimization + * to avoid flushing the same (first) root again. */ - if (++nr_unique_valid_eptps == 1) - kvm_vmx->hv_tlb_eptp = tmp_eptp; + if (++nr_unique_valid_roots == 1) + kvm_vmx->hv_root_ept = root; if (!ret) - ret = hv_remote_flush_eptp(tmp_eptp, range); + ret = hv_remote_flush_root_ept(root, range); /* - * Stop processing EPTPs if a failure occurred and - * there is already a detected EPTP mismatch. + * Stop processing roots if a failure occurred and + * multiple valid roots have already been detected. */ - if (ret && nr_unique_valid_eptps > 1) + if (ret && nr_unique_valid_roots > 1) break; } /* - * The optimized flush of a single EPTP can't be used if there - * are multiple valid EPTPs (obviously). + * The optimized flush of a single root can't be used if there + * are multiple valid roots (obviously). */ - if (nr_unique_valid_eptps > 1) - kvm_vmx->hv_tlb_eptp = INVALID_PAGE; + if (nr_unique_valid_roots > 1) + kvm_vmx->hv_root_ept = INVALID_PAGE; } else { - ret = hv_remote_flush_eptp(kvm_vmx->hv_tlb_eptp, range); + ret = hv_remote_flush_root_ept(kvm_vmx->hv_root_ept, range); } - spin_unlock(&kvm_vmx->ept_pointer_lock); + spin_unlock(&kvm_vmx->hv_root_ept_lock); return ret; } static int hv_remote_flush_tlb(struct kvm *kvm) @@ -584,17 +579,17 @@ static int hv_enable_direct_tlbflush(struct kvm_vcpu *vcpu) #endif /* IS_ENABLED(CONFIG_HYPERV) */ -static void hv_load_mmu_eptp(struct kvm_vcpu *vcpu, u64 eptp) +static void hv_track_root_ept(struct kvm_vcpu *vcpu, hpa_t root_ept) { #if IS_ENABLED(CONFIG_HYPERV) struct kvm_vmx *kvm_vmx = to_kvm_vmx(vcpu->kvm); if (kvm_x86_ops.tlb_remote_flush == hv_remote_flush_tlb) { - spin_lock(&kvm_vmx->ept_pointer_lock); - to_vmx(vcpu)->ept_pointer = eptp; - if (eptp != kvm_vmx->hv_tlb_eptp) - kvm_vmx->hv_tlb_eptp = INVALID_PAGE; - spin_unlock(&kvm_vmx->ept_pointer_lock); + spin_lock(&kvm_vmx->hv_root_ept_lock); + to_vmx(vcpu)->hv_root_ept = root_ept; + if (root_ept != kvm_vmx->hv_root_ept) + kvm_vmx->hv_root_ept = INVALID_PAGE; + spin_unlock(&kvm_vmx->hv_root_ept_lock); } #endif } @@ -3137,7 +3132,7 @@ static void vmx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, eptp = construct_eptp(vcpu, root_hpa, root_level); vmcs_write64(EPT_POINTER, eptp); - hv_load_mmu_eptp(vcpu, eptp); + hv_track_root_ept(vcpu, root_hpa); if (!enable_unrestricted_guest && !is_paging(vcpu)) guest_cr3 = to_kvm_vmx(kvm)->ept_identity_map_addr; @@ -6929,7 +6924,7 @@ static int vmx_create_vcpu(struct kvm_vcpu *vcpu) vmx->pi_desc.sn = 1; #if IS_ENABLED(CONFIG_HYPERV) - vmx->ept_pointer = INVALID_PAGE; + vmx->hv_root_ept = INVALID_PAGE; #endif return 0; @@ -6948,7 +6943,7 @@ static int vmx_create_vcpu(struct kvm_vcpu *vcpu) static int vmx_vm_init(struct kvm *kvm) { #if IS_ENABLED(CONFIG_HYPERV) - spin_lock_init(&to_kvm_vmx(kvm)->ept_pointer_lock); + spin_lock_init(&to_kvm_vmx(kvm)->hv_root_ept_lock); #endif if (!ple_gap) diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h index 6d97b5a64b62..0fb3236b0283 100644 --- a/arch/x86/kvm/vmx/vmx.h +++ b/arch/x86/kvm/vmx/vmx.h @@ -326,7 +326,7 @@ struct vcpu_vmx { u64 msr_ia32_feature_control; u64 msr_ia32_feature_control_valid_bits; #if IS_ENABLED(CONFIG_HYPERV) - u64 ept_pointer; + u64 hv_root_ept; #endif struct pt_desc pt_desc; @@ -348,8 +348,8 @@ struct kvm_vmx { gpa_t ept_identity_map_addr; #if IS_ENABLED(CONFIG_HYPERV) - hpa_t hv_tlb_eptp; - spinlock_t ept_pointer_lock; + hpa_t hv_root_ept; + spinlock_t hv_root_ept_lock; #endif };