From patchwork Mon Feb 27 08:45:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robert Hoo X-Patchwork-Id: 13153098 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 60450C64ED8 for ; Mon, 27 Feb 2023 08:55:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231265AbjB0Izv (ORCPT ); Mon, 27 Feb 2023 03:55:51 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43652 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231673AbjB0IzF (ORCPT ); Mon, 27 Feb 2023 03:55:05 -0500 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AD0B524CA5 for ; Mon, 27 Feb 2023 00:47:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677487644; x=1709023644; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=NMP70Ag3PPZSxGE2TJyEUxCricXuuRSM5JR0FCqODZM=; b=Uv803NNg8QQJZEraqzgLjdw7Uuhk5aE4imPHOtysv2nwKgVEfCTvG/SQ IViK29kudb65pX3R0ntDGcRApvDY4lgoyJ8q9yapSZQNnvVjrOInsPCpa buX/9kT7didc25MtxbgCoezc6lxjvTP4DtESDUq7BoWXEyunu/pYDltbA 6mhP7ggDmrJ344lvVHzx0OLvhUINJSXAvDop7/fcX7Tp1BCDTdtEEHGBM MKrynNWMBp0C8D7KRwXsPHHD779fyHBV6l/4smScum6XV22uhnAq9dMqQ eu5g6p7RfMGNgQLLH57zWCxPxKB/4ypJzpAkXLVDAdZUOPZL3iBW0yRqW Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10633"; a="322057679" X-IronPort-AV: E=Sophos;i="5.97,331,1669104000"; d="scan'208";a="322057679" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 00:46:12 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10633"; a="651127072" X-IronPort-AV: E=Sophos;i="5.97,331,1669104000"; d="scan'208";a="651127072" Received: from sqa-gate.sh.intel.com (HELO robert-clx2.tsp.org) ([10.239.48.212]) by orsmga006.jf.intel.com with ESMTP; 27 Feb 2023 00:46:11 -0800 From: Robert Hoo To: seanjc@google.com, pbonzini@redhat.com, chao.gao@intel.com, binbin.wu@linux.intel.com Cc: kvm@vger.kernel.org, Robert Hoo Subject: [PATCH v5 1/5] KVM: x86: Virtualize CR4.LAM_SUP Date: Mon, 27 Feb 2023 16:45:43 +0800 Message-Id: <20230227084547.404871-2-robert.hu@linux.intel.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230227084547.404871-1-robert.hu@linux.intel.com> References: <20230227084547.404871-1-robert.hu@linux.intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org LAM feature uses CR4 bit[28] (LAM_SUP) to enable/config LAM masking on supervisor mode address. To virtualize that, move CR4.LAM_SUP out of unconditional CR4_RESERVED_BITS; its reservation now depends on vCPU has LAM feature or not. Not passing through to guest but intercept it, is to avoid read VMCS field every time when KVM fetch its value, with expectation that guest won't toggle this bit frequently. There's no other features/vmx_exec_controls connections, therefore no code need to be complemented in kvm/vmx_set_cr4(). Signed-off-by: Robert Hoo Reviewed-by: Chao Gao --- arch/x86/include/asm/kvm_host.h | 3 ++- arch/x86/kvm/x86.h | 2 ++ 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index f35f1ff4427b..4684896698f4 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -125,7 +125,8 @@ | X86_CR4_PGE | X86_CR4_PCE | X86_CR4_OSFXSR | X86_CR4_PCIDE \ | X86_CR4_OSXSAVE | X86_CR4_SMEP | X86_CR4_FSGSBASE \ | X86_CR4_OSXMMEXCPT | X86_CR4_LA57 | X86_CR4_VMXE \ - | X86_CR4_SMAP | X86_CR4_PKE | X86_CR4_UMIP)) + | X86_CR4_SMAP | X86_CR4_PKE | X86_CR4_UMIP \ + | X86_CR4_LAM_SUP)) #define CR8_RESERVED_BITS (~(unsigned long)X86_CR8_TPR) diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h index 9de72586f406..8ec5cc983062 100644 --- a/arch/x86/kvm/x86.h +++ b/arch/x86/kvm/x86.h @@ -475,6 +475,8 @@ bool kvm_msr_allowed(struct kvm_vcpu *vcpu, u32 index, u32 type); __reserved_bits |= X86_CR4_VMXE; \ if (!__cpu_has(__c, X86_FEATURE_PCID)) \ __reserved_bits |= X86_CR4_PCIDE; \ + if (!__cpu_has(__c, X86_FEATURE_LAM)) \ + __reserved_bits |= X86_CR4_LAM_SUP; \ __reserved_bits; \ }) From patchwork Mon Feb 27 08:45:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robert Hoo X-Patchwork-Id: 13153101 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DD1CCC64ED6 for ; Mon, 27 Feb 2023 08:56:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231285AbjB0I4V (ORCPT ); Mon, 27 Feb 2023 03:56:21 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44852 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231320AbjB0Izx (ORCPT ); Mon, 27 Feb 2023 03:55:53 -0500 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6797B1713 for ; Mon, 27 Feb 2023 00:47:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677487671; x=1709023671; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=KDJQsv8v8pA0Pa5ghjuA3PzGLpKYZKTlr2UU2oYWW58=; b=CsDN2bTG3vw3jPnLhGCrFFYf5rJv3JPB1qAiM2V3H2qC8JuEJgRBEAMx gmZDhQWVjYIKEoum+G4rXIvYPatGkv5dnmYJIrK2KiXMyBtyp2wUexRQ2 SPXIiWtF7kLv9t6h6gJ1ttQo8dvMH+bD4lSLwLU7wLVX+FpcVBluEg9zb ob2sWshvpgmmYReNH+vt1zgWB+opMf2elfMgzkzgUGeYHWJFoJ9B4H9Lo gQrIDrn7OEhycKYlFZN99udKAepQDoQ0rvIbPrxl7nWwe2+ORTsJb+HNO h43qzLVLrA+AUZkORbnFrGBsqNlsvBthv6VrRM7ZtkFojMRIssEuMQUql Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10633"; a="322057690" X-IronPort-AV: E=Sophos;i="5.97,331,1669104000"; d="scan'208";a="322057690" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 00:46:14 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10633"; a="651127080" X-IronPort-AV: E=Sophos;i="5.97,331,1669104000"; d="scan'208";a="651127080" Received: from sqa-gate.sh.intel.com (HELO robert-clx2.tsp.org) ([10.239.48.212]) by orsmga006.jf.intel.com with ESMTP; 27 Feb 2023 00:46:13 -0800 From: Robert Hoo To: seanjc@google.com, pbonzini@redhat.com, chao.gao@intel.com, binbin.wu@linux.intel.com Cc: kvm@vger.kernel.org, Robert Hoo Subject: [PATCH v5 2/5] [Trivial]KVM: x86: Explicitly cast ulong to bool in kvm_set_cr3() Date: Mon, 27 Feb 2023 16:45:44 +0800 Message-Id: <20230227084547.404871-3-robert.hu@linux.intel.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230227084547.404871-1-robert.hu@linux.intel.com> References: <20230227084547.404871-1-robert.hu@linux.intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org kvm_read_cr4_bits() returns ulong, explicitly cast it bool when assign to a bool variable. Signed-off-by: Robert Hoo --- arch/x86/kvm/x86.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 312aea1854ae..b9611690561d 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -1236,7 +1236,7 @@ int kvm_set_cr3(struct kvm_vcpu *vcpu, unsigned long cr3) bool skip_tlb_flush = false; unsigned long pcid = 0; #ifdef CONFIG_X86_64 - bool pcid_enabled = kvm_read_cr4_bits(vcpu, X86_CR4_PCIDE); + bool pcid_enabled = !!kvm_read_cr4_bits(vcpu, X86_CR4_PCIDE); if (pcid_enabled) { skip_tlb_flush = cr3 & X86_CR3_PCID_NOFLUSH; From patchwork Mon Feb 27 08:45:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robert Hoo X-Patchwork-Id: 13153099 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0A7D4C64ED6 for ; Mon, 27 Feb 2023 08:56:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231418AbjB0I4F (ORCPT ); Mon, 27 Feb 2023 03:56:05 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51614 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229753AbjB0Izk (ORCPT ); Mon, 27 Feb 2023 03:55:40 -0500 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4697822782 for ; Mon, 27 Feb 2023 00:47:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677487660; x=1709023660; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=THmjUcfk/6GSHNNmMU6Ebzb5sJJPJioIWDLVrFSanP8=; b=Bagqss00CL+oNd2QVX/Du0BuAEY86agMuhi/xqjrkZHIVoZgoDxJd1YI Nm3WqKR3a3WGld1ai+f14lpoI6hH7g96vPwq8qBR9VAmLFoGyHkPYxH5X quJ3KAakqEmHB3ntHXf7V/NvKpMSiq4GUkHh/vydikT1mPeOqo7D+1aZR 7F1xh/QLKVNCoRT/LjGP4kLEef+kSlejSr8CzZpSs0AALZtwbIIjDhDJV nT1D+sf6j+C52g9t6vl1i7vm6uyhffTv0xeyuPrC0bWE71eBIfgQijuBD LFdTA1L5cTrpuP7dnKakj2qejRLviCJXB8WD0iftRaxQ7jJ+m8YrqGnnM w==; X-IronPort-AV: E=McAfee;i="6500,9779,10633"; a="322057696" X-IronPort-AV: E=Sophos;i="5.97,331,1669104000"; d="scan'208";a="322057696" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 00:46:17 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10633"; a="651127087" X-IronPort-AV: E=Sophos;i="5.97,331,1669104000"; d="scan'208";a="651127087" Received: from sqa-gate.sh.intel.com (HELO robert-clx2.tsp.org) ([10.239.48.212]) by orsmga006.jf.intel.com with ESMTP; 27 Feb 2023 00:46:15 -0800 From: Robert Hoo To: seanjc@google.com, pbonzini@redhat.com, chao.gao@intel.com, binbin.wu@linux.intel.com Cc: kvm@vger.kernel.org, Robert Hoo Subject: [PATCH v5 3/5] KVM: x86: Virtualize CR3.LAM_{U48,U57} Date: Mon, 27 Feb 2023 16:45:45 +0800 Message-Id: <20230227084547.404871-4-robert.hu@linux.intel.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230227084547.404871-1-robert.hu@linux.intel.com> References: <20230227084547.404871-1-robert.hu@linux.intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org LAM feature uses 2 high bits in CR3 (bit 62 for LAM_U48 and bit 61 for LAM_U57) to enable/config LAM feature for user mode addresses. The LAM masking is done before legacy canonical checks. To virtualize LAM CR3 bits usage, this patch: 1. Don't reserve these 2 bits when LAM is enable on the vCPU. Previously when validate CR3, kvm uses kvm_vcpu_is_legal_gpa(), now define kvm_vcpu_is_valid_cr3() which is actually kvm_vcpu_is_legal_gpa() + CR3.LAM bits validation. Substitutes kvm_vcpu_is_legal/illegal_gpa() with kvm_vcpu_is_valid_cr3() in call sites where is validating CR3 rather than pure GPA. 2. mmu::get_guest_pgd(), its implementation is get_cr3() which returns whole guest CR3 value. Strip LAM bits in those call sites that need pure PGD value, e.g. mmu_alloc_shadow_roots(), FNAME(walk_addr_generic)(). 3. When form a new guest CR3 (vmx_load_mmu_pgd()), melt in LAM bit (kvm_get_active_lam()). 4. When guest sets CR3, identify ONLY-LAM-bits-toggling cases, where it is unnecessary to make new pgd, but just make request of load pgd, then new CR3.LAM bits configuration will be melt in (above point 3). To be conservative, this case still do TLB flush. 5. For nested VM entry, allow the 2 CR3 bits set in corresponding VMCS host/guest fields. Signed-off-by: Robert Hoo --- arch/x86/kvm/mmu.h | 5 +++++ arch/x86/kvm/mmu/mmu.c | 9 ++++++++- arch/x86/kvm/mmu/paging_tmpl.h | 2 +- arch/x86/kvm/vmx/nested.c | 4 ++-- arch/x86/kvm/vmx/vmx.c | 3 ++- arch/x86/kvm/x86.c | 33 ++++++++++++++++++++++++++++----- arch/x86/kvm/x86.h | 1 + 7 files changed, 47 insertions(+), 10 deletions(-) diff --git a/arch/x86/kvm/mmu.h b/arch/x86/kvm/mmu.h index 6bdaacb6faa0..866f2b7cb509 100644 --- a/arch/x86/kvm/mmu.h +++ b/arch/x86/kvm/mmu.h @@ -142,6 +142,11 @@ static inline unsigned long kvm_get_active_pcid(struct kvm_vcpu *vcpu) return kvm_get_pcid(vcpu, kvm_read_cr3(vcpu)); } +static inline u64 kvm_get_active_lam(struct kvm_vcpu *vcpu) +{ + return kvm_read_cr3(vcpu) & (X86_CR3_LAM_U48 | X86_CR3_LAM_U57); +} + static inline void kvm_mmu_load_pgd(struct kvm_vcpu *vcpu) { u64 root_hpa = vcpu->arch.mmu->root.hpa; diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 835426254e76..3efec7f8d8c6 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -3699,7 +3699,14 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) int quadrant, i, r; hpa_t root; - root_pgd = mmu->get_guest_pgd(vcpu); + /* + * Omit guest_cpuid_has(X86_FEATURE_LAM) check but can unconditionally + * strip CR3 LAM bits because they resides in high reserved bits, + * with LAM or not, those high bits should be striped anyway when + * interpreted to pgd. + */ + root_pgd = mmu->get_guest_pgd(vcpu) & + ~(X86_CR3_LAM_U48 | X86_CR3_LAM_U57); root_gfn = root_pgd >> PAGE_SHIFT; if (mmu_check_root(vcpu, root_gfn)) diff --git a/arch/x86/kvm/mmu/paging_tmpl.h b/arch/x86/kvm/mmu/paging_tmpl.h index 0f6455072055..57f39c7492ed 100644 --- a/arch/x86/kvm/mmu/paging_tmpl.h +++ b/arch/x86/kvm/mmu/paging_tmpl.h @@ -324,7 +324,7 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker, trace_kvm_mmu_pagetable_walk(addr, access); retry_walk: walker->level = mmu->cpu_role.base.level; - pte = mmu->get_guest_pgd(vcpu); + pte = mmu->get_guest_pgd(vcpu) & ~(X86_CR3_LAM_U48 | X86_CR3_LAM_U57); have_ad = PT_HAVE_ACCESSED_DIRTY(mmu); #if PTTYPE == 64 diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c index b6f4411b613e..301912155dd1 100644 --- a/arch/x86/kvm/vmx/nested.c +++ b/arch/x86/kvm/vmx/nested.c @@ -1078,7 +1078,7 @@ static int nested_vmx_load_cr3(struct kvm_vcpu *vcpu, unsigned long cr3, bool nested_ept, bool reload_pdptrs, enum vm_entry_failure_code *entry_failure_code) { - if (CC(kvm_vcpu_is_illegal_gpa(vcpu, cr3))) { + if (CC(!kvm_vcpu_is_valid_cr3(vcpu, cr3))) { *entry_failure_code = ENTRY_FAIL_DEFAULT; return -EINVAL; } @@ -2906,7 +2906,7 @@ static int nested_vmx_check_host_state(struct kvm_vcpu *vcpu, if (CC(!nested_host_cr0_valid(vcpu, vmcs12->host_cr0)) || CC(!nested_host_cr4_valid(vcpu, vmcs12->host_cr4)) || - CC(kvm_vcpu_is_illegal_gpa(vcpu, vmcs12->host_cr3))) + CC(!kvm_vcpu_is_valid_cr3(vcpu, vmcs12->host_cr3))) return -EINVAL; if (CC(is_noncanonical_address(vmcs12->host_ia32_sysenter_esp, vcpu)) || diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index fe5615fd8295..66edd091f145 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -3289,7 +3289,8 @@ static void vmx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, update_guest_cr3 = false; vmx_ept_load_pdptrs(vcpu); } else { - guest_cr3 = root_hpa | kvm_get_active_pcid(vcpu); + guest_cr3 = root_hpa | kvm_get_active_pcid(vcpu) | + kvm_get_active_lam(vcpu); } if (update_guest_cr3) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index b9611690561d..be6c7c986f0f 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -1231,10 +1231,21 @@ static void kvm_invalidate_pcid(struct kvm_vcpu *vcpu, unsigned long pcid) kvm_mmu_free_roots(vcpu->kvm, mmu, roots_to_free); } +bool kvm_vcpu_is_valid_cr3(struct kvm_vcpu *vcpu, unsigned long cr3) +{ + if (guest_cpuid_has(vcpu, X86_FEATURE_LAM)) + cr3 &= ~(X86_CR3_LAM_U48 | X86_CR3_LAM_U57); + else if (cr3 & (X86_CR3_LAM_U48 | X86_CR3_LAM_U57)) + return false; + + return kvm_vcpu_is_legal_gpa(vcpu, cr3); +} +EXPORT_SYMBOL_GPL(kvm_vcpu_is_valid_cr3); + int kvm_set_cr3(struct kvm_vcpu *vcpu, unsigned long cr3) { bool skip_tlb_flush = false; - unsigned long pcid = 0; + unsigned long pcid = 0, old_cr3; #ifdef CONFIG_X86_64 bool pcid_enabled = !!kvm_read_cr4_bits(vcpu, X86_CR4_PCIDE); @@ -1254,14 +1265,26 @@ int kvm_set_cr3(struct kvm_vcpu *vcpu, unsigned long cr3) * stuff CR3, e.g. for RSM emulation, and there is no guarantee that * the current vCPU mode is accurate. */ - if (kvm_vcpu_is_illegal_gpa(vcpu, cr3)) + if (!kvm_vcpu_is_valid_cr3(vcpu, cr3)) return 1; if (is_pae_paging(vcpu) && !load_pdptrs(vcpu, cr3)) return 1; - if (cr3 != kvm_read_cr3(vcpu)) - kvm_mmu_new_pgd(vcpu, cr3); + old_cr3 = kvm_read_cr3(vcpu); + if (cr3 != old_cr3) { + if ((cr3 ^ old_cr3) & ~(X86_CR3_LAM_U48 | X86_CR3_LAM_U57)) { + kvm_mmu_new_pgd(vcpu, cr3 & ~(X86_CR3_LAM_U48 | + X86_CR3_LAM_U57)); + } else { + /* + * Though only LAM bits change, mark the + * request in order to force an update on guest CR3 + * because the LAM bits of old CR3 is stale + */ + kvm_make_request(KVM_REQ_LOAD_MMU_PGD, vcpu); + } + } vcpu->arch.cr3 = cr3; kvm_register_mark_dirty(vcpu, VCPU_EXREG_CR3); @@ -11185,7 +11208,7 @@ static bool kvm_is_valid_sregs(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs) */ if (!(sregs->cr4 & X86_CR4_PAE) || !(sregs->efer & EFER_LMA)) return false; - if (kvm_vcpu_is_illegal_gpa(vcpu, sregs->cr3)) + if (!kvm_vcpu_is_valid_cr3(vcpu, sregs->cr3)) return false; } else { /* diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h index 8ec5cc983062..6b6bfddc84e0 100644 --- a/arch/x86/kvm/x86.h +++ b/arch/x86/kvm/x86.h @@ -440,6 +440,7 @@ void kvm_load_guest_xsave_state(struct kvm_vcpu *vcpu); void kvm_load_host_xsave_state(struct kvm_vcpu *vcpu); int kvm_spec_ctrl_test_value(u64 value); bool __kvm_is_valid_cr4(struct kvm_vcpu *vcpu, unsigned long cr4); +bool kvm_vcpu_is_valid_cr3(struct kvm_vcpu *vcpu, unsigned long cr3); int kvm_handle_memory_failure(struct kvm_vcpu *vcpu, int r, struct x86_exception *e); int kvm_handle_invpcid(struct kvm_vcpu *vcpu, unsigned long type, gva_t gva); From patchwork Mon Feb 27 08:45:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robert Hoo X-Patchwork-Id: 13153100 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 831B9C64ED8 for ; Mon, 27 Feb 2023 08:56:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231567AbjB0I4L (ORCPT ); Mon, 27 Feb 2023 03:56:11 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51822 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229874AbjB0Izr (ORCPT ); Mon, 27 Feb 2023 03:55:47 -0500 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8E62924CAB for ; Mon, 27 Feb 2023 00:47:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677487666; x=1709023666; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=J5qjLMWnAEH++IzEHXNi7VwHmVyDNemr2qezxGGrK/w=; b=oIJPDwQB+hj8AEXSZTeDcqMJKvlsgp64J717xhFQcRMfiRMzrwXnmBuv qsYYaRKP3vnqJJ533raayoImvBzBkVQPdmScpB3eX9Odr+ExrLqfQI2l3 5XbTXFkNtiOt+MDOyM8Cp4JNeXTs91OBlf9sIIoy1SKxjE55AFosSTsCQ GVtjWH1xqxViMQDF+nSRR+j1lhsqe2GB1YnQ71Br8GQFu6eVEjnonxezd E2RaTsKeTWMcUedH6bjflG9AXbHl4Eio95Am7i8nDBliufQ2Xlt3ZN5kG wlDxoqt3xBTLzg3VR4HpNWpe+9YD/TQgwKXQ30bXNJgKRIG/A9RpVao8G Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10633"; a="322057701" X-IronPort-AV: E=Sophos;i="5.97,331,1669104000"; d="scan'208";a="322057701" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 00:46:19 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10633"; a="651127091" X-IronPort-AV: E=Sophos;i="5.97,331,1669104000"; d="scan'208";a="651127091" Received: from sqa-gate.sh.intel.com (HELO robert-clx2.tsp.org) ([10.239.48.212]) by orsmga006.jf.intel.com with ESMTP; 27 Feb 2023 00:46:17 -0800 From: Robert Hoo To: seanjc@google.com, pbonzini@redhat.com, chao.gao@intel.com, binbin.wu@linux.intel.com Cc: kvm@vger.kernel.org, Robert Hoo Subject: [PATCH v5 4/5] KVM: x86: emulation: Apply LAM mask when emulating data access in 64-bit mode Date: Mon, 27 Feb 2023 16:45:46 +0800 Message-Id: <20230227084547.404871-5-robert.hu@linux.intel.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230227084547.404871-1-robert.hu@linux.intel.com> References: <20230227084547.404871-1-robert.hu@linux.intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Emulate HW LAM masking when doing data access under 64-bit mode. kvm_lam_untag_addr() implements this: per CR4/CR3 LAM bits configuration, firstly check the linear addr conforms LAM canonical, i.e. the highest address bit matches bit 63. Then mask out meta data per LAM configuration. If failed in above process, emulate #GP to guest. Signed-off-by: Robert Hoo --- arch/x86/kvm/emulate.c | 13 ++++++++ arch/x86/kvm/x86.h | 70 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 83 insertions(+) diff --git a/arch/x86/kvm/emulate.c b/arch/x86/kvm/emulate.c index 5cc3efa0e21c..77bd13f40711 100644 --- a/arch/x86/kvm/emulate.c +++ b/arch/x86/kvm/emulate.c @@ -700,6 +700,19 @@ static __always_inline int __linearize(struct x86_emulate_ctxt *ctxt, *max_size = 0; switch (mode) { case X86EMUL_MODE_PROT64: + /* LAM applies only on data access */ + if (!fetch && guest_cpuid_has(ctxt->vcpu, X86_FEATURE_LAM)) { + enum lam_type type; + + type = kvm_vcpu_lam_type(la, ctxt->vcpu); + if (type == LAM_ILLEGAL) { + *linear = la; + goto bad; + } else { + la = kvm_lam_untag_addr(la, type); + } + } + *linear = la; va_bits = ctxt_virt_addr_bits(ctxt); if (!__is_canonical_address(la, va_bits)) diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h index 6b6bfddc84e0..d992e5220602 100644 --- a/arch/x86/kvm/x86.h +++ b/arch/x86/kvm/x86.h @@ -201,6 +201,76 @@ static inline bool is_noncanonical_address(u64 la, struct kvm_vcpu *vcpu) return !__is_canonical_address(la, vcpu_virt_addr_bits(vcpu)); } +enum lam_type { + LAM_ILLEGAL = -1, + LAM_U57, + LAM_U48, + LAM_S57, + LAM_S48, + LAM_NONE +}; + +#ifdef CONFIG_X86_64 +/* + * LAM Canonical Rule: + * LAM_U/S48 -- bit 63 == bit 47 + * LAM_U/S57 -- bit 63 == bit 56 + */ +static inline bool lam_canonical(u64 addr, int effect_width) +{ + return (addr >> 63) == ((addr >> effect_width) & BIT(0)); +} + +static inline enum lam_type kvm_vcpu_lam_type(u64 addr, struct kvm_vcpu *vcpu) +{ + WARN_ON_ONCE(!is_64_bit_mode(vcpu)); + + if (addr >> 63 == 0) { + if (kvm_read_cr3(vcpu) & X86_CR3_LAM_U57) + return lam_canonical(addr, 56) ? LAM_U57 : LAM_ILLEGAL; + else if (kvm_read_cr3(vcpu) & X86_CR3_LAM_U48) + return lam_canonical(addr, 47) ? LAM_U48 : LAM_ILLEGAL; + } else if (kvm_read_cr4_bits(vcpu, X86_CR4_LAM_SUP)) { + if (kvm_read_cr4_bits(vcpu, X86_CR4_LA57)) + return lam_canonical(addr, 56) ? LAM_S57 : LAM_ILLEGAL; + else + return lam_canonical(addr, 47) ? LAM_S48 : LAM_ILLEGAL; + } + + return LAM_NONE; +} + +/* untag addr for guest, according to vCPU's LAM config */ +static inline u64 kvm_lam_untag_addr(u64 addr, enum lam_type type) +{ + switch (type) { + case LAM_U57: + case LAM_S57: + addr = __canonical_address(addr, 57); + break; + case LAM_U48: + case LAM_S48: + addr = __canonical_address(addr, 48); + break; + case LAM_NONE: + default: + break; + } + + return addr; +} +#else +static inline enum lam_type kvm_vcpu_lam_type(u64 addr, struct kvm_vcpu *vcpu) +{ + return LAM_NONE; +} + +static inline u64 kvm_lam_untag_addr(u64 addr, enum lam_type type) +{ + return addr; +} +#endif + static inline void vcpu_cache_mmio_info(struct kvm_vcpu *vcpu, gva_t gva, gfn_t gfn, unsigned access) { From patchwork Mon Feb 27 08:45:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robert Hoo X-Patchwork-Id: 13153103 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7BF39C64ED8 for ; Mon, 27 Feb 2023 08:56:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231710AbjB0I4j (ORCPT ); Mon, 27 Feb 2023 03:56:39 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51616 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229693AbjB0I4E (ORCPT ); Mon, 27 Feb 2023 03:56:04 -0500 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E272D1114B for ; Mon, 27 Feb 2023 00:48:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677487682; x=1709023682; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=e8fJJStm95QS8qPwDBv072byIRMXgz9xMMo6V5XCCvA=; b=AK/fmqRzYDCPosFzu5nSYJy8Zw6G/wVaNtN+2/Oy2BwBXhTOP3ilai7T KOPKfCpqDm71ikfrrp8XAql9FhcH2GFGFQ5SlcluINr5BLXl2DhjUFN+c gP5A7RNVWTF6pGP2XG2cpCWobomQKTKSg6LeWF5pbd4G0mVtdZeL13OHN rmxwTed0IIb3bo1bE/FcH6JcSG/mwf4kf7eV7k9vM92qAckJD63pXKSoG 0pSZGTWc9f/gyW2/PV9LdiPXed2D20xUrJ4bTGJn4/IEKPhfxCRXa/r8n Zvmk07GkCr7bjijwz0CZ8fCyjjDZg5pUInO+zcceN6cSB56ibHUbRavR3 A==; X-IronPort-AV: E=McAfee;i="6500,9779,10633"; a="322057707" X-IronPort-AV: E=Sophos;i="5.97,331,1669104000"; d="scan'208";a="322057707" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Feb 2023 00:46:21 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10633"; a="651127095" X-IronPort-AV: E=Sophos;i="5.97,331,1669104000"; d="scan'208";a="651127095" Received: from sqa-gate.sh.intel.com (HELO robert-clx2.tsp.org) ([10.239.48.212]) by orsmga006.jf.intel.com with ESMTP; 27 Feb 2023 00:46:19 -0800 From: Robert Hoo To: seanjc@google.com, pbonzini@redhat.com, chao.gao@intel.com, binbin.wu@linux.intel.com Cc: kvm@vger.kernel.org, Robert Hoo , Jingqi Liu Subject: [PATCH v5 5/5] KVM: x86: LAM: Expose LAM CPUID to user space VMM Date: Mon, 27 Feb 2023 16:45:47 +0800 Message-Id: <20230227084547.404871-6-robert.hu@linux.intel.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230227084547.404871-1-robert.hu@linux.intel.com> References: <20230227084547.404871-1-robert.hu@linux.intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org LAM feature is enumerated by (EAX=07H, ECX=01H):EAX.LAM[bit26]. Signed-off-by: Robert Hoo Reviewed-by: Jingqi Liu --- arch/x86/kvm/cpuid.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c index b14653b61470..79f45cbe587e 100644 --- a/arch/x86/kvm/cpuid.c +++ b/arch/x86/kvm/cpuid.c @@ -664,7 +664,7 @@ void kvm_set_cpu_caps(void) kvm_cpu_cap_mask(CPUID_7_1_EAX, F(AVX_VNNI) | F(AVX512_BF16) | F(CMPCCXADD) | F(AMX_FP16) | - F(AVX_IFMA) + F(AVX_IFMA) | F(LAM) ); kvm_cpu_cap_init_kvm_defined(CPUID_7_1_EDX,