From patchwork Thu Aug 1 18:34:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 13750846 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4E980144D00 for ; Thu, 1 Aug 2024 18:34:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722537300; cv=none; b=jEG+LY2AVRY1197kYX29suJNUWQaRXeApoEp/cxp9SVmrOcjz5y4moFPmuRy/VCJaaMTLv9O3/nu/86/GQRFpdMA8iA+lgOKP3Vnv/bIdGf2W4SgEFABlZuzFL32Vl4sJTMHGkO2vOiao8CzRKcxu2BwHbWQeBFlM+ZgmW/J5F8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722537300; c=relaxed/simple; bh=B4M5zM2LxlbX4e+OKo938Pgq8tauyjOo2Mz9UK1WRwE=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=X9plHNFGXjcFFaoL1YaUyffLyU3qzuQdZHvTEeujguU+jmZxWxbP8r0p2+MCNs2FZKPOZR+w0VpieodJYzfkmvP08CvrEbX1sTy38hufzt0p72Q6XAa2PxxIaMYI5sR+JSnMbjgTIolFs8i4qRrx7d4viwTVYAvRWPjwRTuCfIk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=xqnkioJg; arc=none smtp.client-ip=209.85.219.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="xqnkioJg" Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-e0ba463c970so5195654276.0 for ; Thu, 01 Aug 2024 11:34:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1722537298; x=1723142098; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=pYuydgqnu7KvS+vD/kJx/xLlR2Fg91BOlW6zTjcno7Q=; b=xqnkioJgRWzyMTa9C3AyqvcLKXN1NXw1GUktnC8nbyN5UNZOJf49BsVPbZikkX1xXR x+4xK5Q2URGVSmX0ZbAQ70ZwN9eFbPW6jrUJuqA2rhMI2vzLwHxVK/CU8VHU26QyyPmv sB5LDDsFOym5GSofCSESv1Iop7cZzQ1TOMBN38YOTaHqQEClvXoed/8ZO0DEbpk674Xb EYzGWJbuNXZJrepSbzsqm/WUThX5yu3M8+5kAUN93pvcVDOl8Ki8vMwdJnKUc46M73TZ VP5TFcLtBjfYB1JCMysdb/tUVWiJCKM7sAKuL7c9p2o+GDrq/Fgc5vtCKYUnwadbuB7b 0AFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722537298; x=1723142098; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=pYuydgqnu7KvS+vD/kJx/xLlR2Fg91BOlW6zTjcno7Q=; b=EH1DM0nFmUP2D1OXgvvNaXxUmB5xLewiYP2KkeJA7AxdHXRVkI0v00c8qmnsfBb1Od iuOsb1Xg71Ltxc11WI7vOHyOwL6MkI1sbNYnfxvsYyU9/rgUjfdlTzcS8xjXUWk0Qvku SG5SUjq6I3fTL5oh5m5s2nwatu2POOR7k+nvv1ucTW8KJ7a8bnrexB6gG5hdtXZ4/85O GhB18TcyreFILuiaaZgVY1GWyLjfsta+8BgZKI0mCkr9vy3SRTDZpef9EnJLdmwi54wa aYm/8dkdfQQFCJ+rFM+gmTUTteufybG8M0g7eBOiBmbq3JLv/FGxcvpdmaF1FoUOyVYS +2Uw== X-Gm-Message-State: AOJu0YxCep+ZCPYOw+pzSektJvD/W6YojZxm8CHVvRTpAzfb3/MlM1/r k0Z6Im6/U045/0Sh2wBveDGsRO10MD0AI15QQonodfC5eqZPS8yMj96tYBLHu3CTF/D3Hu1Yu97 fbg== X-Google-Smtp-Source: AGHT+IFfRgjDzueDHA1wB4xIJlPq2Sh0SGDxzPYZTvnWrBT2GBLIr7FzlALzApVC+WiYgVOffmnlJ9uzeLA= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a05:6902:2b02:b0:e03:31ec:8a24 with SMTP id 3f1490d57ef6-e0bde422b24mr29349276.8.1722537298373; Thu, 01 Aug 2024 11:34:58 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 1 Aug 2024 11:34:45 -0700 In-Reply-To: <20240801183453.57199-1-seanjc@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240801183453.57199-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.rc2.264.g509ed76dc8-goog Message-ID: <20240801183453.57199-2-seanjc@google.com> Subject: [RFC PATCH 1/9] KVM: x86/mmu: Add a dedicated flag to track if A/D bits are globally enabled From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Add a dedicated flag to track if KVM has enabled A/D bits at the module level, instead of inferring the state based on whether or not the MMU's shadow_accessed_mask is non-zero. This will allow defining and using shadow_accessed_mask even when A/D bits aren't used by hardware. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 6 +++--- arch/x86/kvm/mmu/spte.c | 6 ++++++ arch/x86/kvm/mmu/spte.h | 20 +++++++++----------- arch/x86/kvm/mmu/tdp_mmu.c | 4 ++-- 4 files changed, 20 insertions(+), 16 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 5979eeb916cd..1e24bc4a06db 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -3319,7 +3319,7 @@ static bool page_fault_can_be_fast(struct kvm *kvm, struct kvm_page_fault *fault * by setting the Writable bit, which can be done out of mmu_lock. */ if (!fault->present) - return !kvm_ad_enabled(); + return !kvm_ad_enabled; /* * Note, instruction fetches and writes are mutually exclusive, ignore @@ -3454,7 +3454,7 @@ static int fast_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) * uses A/D bits for non-nested MMUs. Thus, if A/D bits are * enabled, the SPTE can't be an access-tracked SPTE. */ - if (unlikely(!kvm_ad_enabled()) && is_access_track_spte(spte)) + if (unlikely(!kvm_ad_enabled) && is_access_track_spte(spte)) new_spte = restore_acc_track_spte(new_spte); /* @@ -5429,7 +5429,7 @@ kvm_calc_tdp_mmu_root_page_role(struct kvm_vcpu *vcpu, role.efer_nx = true; role.smm = cpu_role.base.smm; role.guest_mode = cpu_role.base.guest_mode; - role.ad_disabled = !kvm_ad_enabled(); + role.ad_disabled = !kvm_ad_enabled; role.level = kvm_mmu_get_tdp_level(vcpu); role.direct = true; role.has_4_byte_gpte = false; diff --git a/arch/x86/kvm/mmu/spte.c b/arch/x86/kvm/mmu/spte.c index 2c5650390d3b..b713a6542eeb 100644 --- a/arch/x86/kvm/mmu/spte.c +++ b/arch/x86/kvm/mmu/spte.c @@ -24,6 +24,8 @@ static bool __ro_after_init allow_mmio_caching; module_param_named(mmio_caching, enable_mmio_caching, bool, 0444); EXPORT_SYMBOL_GPL(enable_mmio_caching); +bool __read_mostly kvm_ad_enabled; + u64 __read_mostly shadow_host_writable_mask; u64 __read_mostly shadow_mmu_writable_mask; u64 __read_mostly shadow_nx_mask; @@ -435,6 +437,8 @@ EXPORT_SYMBOL_GPL(kvm_mmu_set_me_spte_mask); void kvm_mmu_set_ept_masks(bool has_ad_bits, bool has_exec_only) { + kvm_ad_enabled = has_ad_bits; + shadow_user_mask = VMX_EPT_READABLE_MASK; shadow_accessed_mask = has_ad_bits ? VMX_EPT_ACCESS_BIT : 0ull; shadow_dirty_mask = has_ad_bits ? VMX_EPT_DIRTY_BIT : 0ull; @@ -468,6 +472,8 @@ void kvm_mmu_reset_all_pte_masks(void) u8 low_phys_bits; u64 mask; + kvm_ad_enabled = true; + /* * If the CPU has 46 or less physical address bits, then set an * appropriate mask to guard against L1TF attacks. Otherwise, it is diff --git a/arch/x86/kvm/mmu/spte.h b/arch/x86/kvm/mmu/spte.h index ef793c459b05..d722b37b7434 100644 --- a/arch/x86/kvm/mmu/spte.h +++ b/arch/x86/kvm/mmu/spte.h @@ -167,6 +167,15 @@ static_assert(!(SHADOW_NONPRESENT_VALUE & SPTE_MMU_PRESENT_MASK)); #define SHADOW_NONPRESENT_VALUE 0ULL #endif + +/* + * True if A/D bits are supported in hardware and are enabled by KVM. When + * enabled, KVM uses A/D bits for all non-nested MMUs. Because L1 can disable + * A/D bits in EPTP12, SP and SPTE variants are needed to handle the scenario + * where KVM is using A/D bits for L1, but not L2. + */ +extern bool __read_mostly kvm_ad_enabled; + extern u64 __read_mostly shadow_host_writable_mask; extern u64 __read_mostly shadow_mmu_writable_mask; extern u64 __read_mostly shadow_nx_mask; @@ -285,17 +294,6 @@ static inline bool is_ept_ve_possible(u64 spte) (spte & VMX_EPT_RWX_MASK) != VMX_EPT_MISCONFIG_WX_VALUE; } -/* - * Returns true if A/D bits are supported in hardware and are enabled by KVM. - * When enabled, KVM uses A/D bits for all non-nested MMUs. Because L1 can - * disable A/D bits in EPTP12, SP and SPTE variants are needed to handle the - * scenario where KVM is using A/D bits for L1, but not L2. - */ -static inline bool kvm_ad_enabled(void) -{ - return !!shadow_accessed_mask; -} - static inline bool sp_ad_disabled(struct kvm_mmu_page *sp) { return sp->role.ad_disabled; diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index dc153cf92a40..2b0fc601d2ce 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -1072,7 +1072,7 @@ static int tdp_mmu_map_handle_target_level(struct kvm_vcpu *vcpu, static int tdp_mmu_link_sp(struct kvm *kvm, struct tdp_iter *iter, struct kvm_mmu_page *sp, bool shared) { - u64 spte = make_nonleaf_spte(sp->spt, !kvm_ad_enabled()); + u64 spte = make_nonleaf_spte(sp->spt, !kvm_ad_enabled); int ret = 0; if (shared) { @@ -1488,7 +1488,7 @@ static bool tdp_mmu_need_write_protect(struct kvm_mmu_page *sp) * from level, so it is valid to key off any shadow page to determine if * write protection is needed for an entire tree. */ - return kvm_mmu_page_ad_need_write_protect(sp) || !kvm_ad_enabled(); + return kvm_mmu_page_ad_need_write_protect(sp) || !kvm_ad_enabled; } static bool clear_dirty_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root, From patchwork Thu Aug 1 18:34:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 13750847 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3FFDF1494DD for ; Thu, 1 Aug 2024 18:35:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722537302; cv=none; b=OFAdOUQFWCAXUOPMGI7Q0oHFnrixtVGsl0ifaI13V/PZZTar1174/bCIao2AlYAgO7T+vbxEme21SQslzLSRhUYgZTeafZV2jOAxywDprpEztnHk1pZ0Gk0te1AA9lmcEbCsUyJn2ksGw0y/eiRxnQ0Xw15o83gx7K7+ENWhM08= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722537302; c=relaxed/simple; bh=sQ3534Fng9t06Zq2te+F41BHRISWvOlzZzrXklcUmXg=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=il5ZAnggOIVgO2lSLMAPNJzXECmKyy6SSqR5HGwkMwMR3xHFY7/6JLvkaiNNazNKvx4TIHGj6tsCwVyhnt+Nvo92mU5Kv+okB4QYUwTWfagQgWiXu6VuenzpDt32+bnoAEIttWCmROzrmwcejjthFzBF1De5TTQIZIzhEErtXIg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=tvaSzOsE; arc=none smtp.client-ip=209.85.214.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="tvaSzOsE" Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-1fd8a1a75e7so59656485ad.3 for ; Thu, 01 Aug 2024 11:35:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1722537300; x=1723142100; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=Ilp3BRqf5qrS1kR3cVkw+ddXUmTszI9WBxmraRZDLeQ=; b=tvaSzOsEsshFYilc3fDBpqCUBHqDrRvBz6odHOx50TtaGq/Za95Wbv5wL4QIyiQtOX iG7AKKyG3M1VhD/2Sa/xeqa2y+V3h6B/mnxuWOuyaRBcNO3Y1k3bE8BAYEg4KoQtnxL8 PMzlp/v0XkprRy3667xUllopOZlvAm4TFjEHxFINhUqu1kVA5ZGeaRv6pGF/SbcBEwXR aeld0XGNIYow9oTDQHsh2RO7G4Cw9TfGgjPIDhotcPQRb9h9UIJ4qRpqBNK2gbZJeqCI AVuMZ27rkbD+oNNf1BYWJUbm38weJaCF+P3S+S0ewYlY5t+VZlgoyGva6L1Fwi1DXVS/ IJ8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722537300; x=1723142100; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Ilp3BRqf5qrS1kR3cVkw+ddXUmTszI9WBxmraRZDLeQ=; b=WC9W6X/fbwJpb7Y0AjTRtuKMVbCv1ESobcI1NwvN7t811SoqgGajO/dDL2M22ghC2N k2H7Mw03ixE8wa9xSChmDg42dPhfAIS5ITaFRNSD1zu5nIs53qqzzUMao2nnM431OF1k oDjrnbYONfiDuJt/9/AzauQ+sJEg9iUVhOtREqrX+mBYBXAnaFFffrjRBLZaFeFBLvA5 saTHwDk8zjlOdb8dbZoganDj+v18DHJy+walnoTRnsnFcJzN49GcWdy3mLKgESKfKnjN rM+pZd7sttSpzoVmFrzKPqGG5Y/tzrtcFhri1rxxxOcv++hAiFVlFMlybNAfYEiMe38w T5tQ== X-Gm-Message-State: AOJu0YysqtfAF9PuEuyyvzsASa+vm3TC1Kgy/tYX0Sphc8UlP/ZRA6Mr N8qQ6ZbeeaPXFthZgLAUeFR3/UKZ3MDMcw0flhuJ8lUhFZ58Aj9Xt9sRj5x8HwL9Z9newbLHEVG Wmg== X-Google-Smtp-Source: AGHT+IFVeWW3z3HLbr3aiIANRVn0Pzn9JWDp1pBfkw1NJGvfolLz0JIjag5F+BCKoOgA8ow9J6dxgy2bag4= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a17:903:1d2:b0:1fc:6faf:671f with SMTP id d9443c01a7336-1ff57262fc9mr25895ad.6.1722537300275; Thu, 01 Aug 2024 11:35:00 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 1 Aug 2024 11:34:46 -0700 In-Reply-To: <20240801183453.57199-1-seanjc@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240801183453.57199-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.rc2.264.g509ed76dc8-goog Message-ID: <20240801183453.57199-3-seanjc@google.com> Subject: [RFC PATCH 2/9] KVM: x86/mmu: Set shadow_accessed_mask for EPT even if A/D bits disabled From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Now that KVM doesn't use shadow_accessed_mask to detect if hardware A/D bits are enabled, set shadow_accessed_mask for EPT even when A/D bits are disabled in hardware. This will allow using shadow_accessed_mask for software purposes, e.g. to preserve accessed status in a non-present SPTE. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/spte.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/kvm/mmu/spte.c b/arch/x86/kvm/mmu/spte.c index b713a6542eeb..cae45825617c 100644 --- a/arch/x86/kvm/mmu/spte.c +++ b/arch/x86/kvm/mmu/spte.c @@ -440,7 +440,7 @@ void kvm_mmu_set_ept_masks(bool has_ad_bits, bool has_exec_only) kvm_ad_enabled = has_ad_bits; shadow_user_mask = VMX_EPT_READABLE_MASK; - shadow_accessed_mask = has_ad_bits ? VMX_EPT_ACCESS_BIT : 0ull; + shadow_accessed_mask = VMX_EPT_ACCESS_BIT; shadow_dirty_mask = has_ad_bits ? VMX_EPT_DIRTY_BIT : 0ull; shadow_nx_mask = 0ull; shadow_x_mask = VMX_EPT_EXECUTABLE_MASK; From patchwork Thu Aug 1 18:34:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 13750848 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 20BAA14A4F0 for ; Thu, 1 Aug 2024 18:35:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722537304; cv=none; b=ebF3lTCsFbJt8xOJAYZEwlbyVNfSlEIrQfh/uba1upHtjaoKlgJSZWKIIsSiRxuWQXCAQRDQeOFEK6BDB55+22nUBFZ0kAMhu7W6dqlo2HF4llr3zJHB1WhhEA4cXVsRGyHan54rFeD9G8uIMY+b2yiwbnrzg2mGL3FAqQyCGXo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722537304; c=relaxed/simple; bh=Vh7Yjg37p23UJZ6ej/K5Cd4ShyYhS3DKAU1YW8l2Bu8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=StNwAbVcUTqTmsp+/fApkwyTtxHPoDaCMrGwXOQRBuk99l3y+LlkNgQr+j5+7C7Fs5cbj86G2M6tCFkK+ocLQFZY/6e2GLb/wf1I3eBZsJwdF/skJ5ZIjpLllBGEfxJQfcB8GPJOyRR/We686xxT7O+SX+laKFhRHK/hBKYiPjQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Z2gB1DkB; arc=none smtp.client-ip=209.85.128.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Z2gB1DkB" Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-65b985bb059so135676247b3.2 for ; Thu, 01 Aug 2024 11:35:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1722537302; x=1723142102; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=1YHjewNqSuQw+aKISRkd4hlaIK++6vC5e9hG8f7rkrc=; b=Z2gB1DkBTKg5tFcjlUi+t1manENVuYnoPqTdf6wIhk/QXJLXfQMlQdcPBiCPfxWTRw JR/f1k7qLLxyAmValomylwHEkd7dVs/kYm6g87bObW4+LsM92kL3aYEDTpvfuEMmabq8 c8Jb4zVyjtL1FDTTvQ8uG2198/5pfyfWykaW0MPsmvApdt/AJqUW74j2bSm1fV9UyTzV ztUt52sm5f8womKcA1fOKwXufZm3m7hCcT4Wze8VQf0/KaaWDdWwMUvIR9N5ovW7P7lu zG8NmSmqf33ZnsbkfdQve+aMnTFofH8OuYP9hdde0EKfzeQE+9j31WzI94py3SDlyFuM A8Sw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722537302; x=1723142102; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=1YHjewNqSuQw+aKISRkd4hlaIK++6vC5e9hG8f7rkrc=; b=hDC3g9mTxCQjJ0r+jyS2OXnx9APn0roH4ahuOOKTreJFruwghKDrcr+mHxQoKpYzXK sCmYBlt8BkQ66g9j0iQLEY1hs9mskg7oFNY1as0bURAD9FDvPPoozyRz/ldi5Jn00YfF yKEb1ksGISLzUTRiaiyMjVWRKtGfElCOBfuORJ1AKW6ssCgCSrZtYdQzibjR8H4sZFwu rvJRRtt8oQPxXkDkYqGh5pO+kb/rwgigTBuJEI0K0Bx0Y39yqLoVArzjux0Swgtw3iSk 8kxcv194ic7eokBiORCMgWIDlErfYVBj3jy/IBaHdxFCq6cL3gVAAL/Tx9QA7vyNrkRI h1Tg== X-Gm-Message-State: AOJu0YzgLfQ/soNlGZhOhdYyjUMRt0xmGmEg1PT4buTDIUD5xohr4XY8 OAU7MKyb4JjIRsy7h8lC7iJPpwz1n5rSbMp3/OS/VeaKScovaLCn9MYr6FmMecVWMjyP6gLUvEB YdA== X-Google-Smtp-Source: AGHT+IEAyNBNNAjkoq0+TR2DU9aO3yJBNbaHwXM7Z2FBl7YPBn99Zn4C9dL3lWrRoy2Zle815Z1OJBKpL3o= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a05:690c:660c:b0:61b:e73d:bea2 with SMTP id 00721157ae682-68963706dc0mr17197b3.5.1722537302250; Thu, 01 Aug 2024 11:35:02 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 1 Aug 2024 11:34:47 -0700 In-Reply-To: <20240801183453.57199-1-seanjc@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240801183453.57199-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.rc2.264.g509ed76dc8-goog Message-ID: <20240801183453.57199-4-seanjc@google.com> Subject: [RFC PATCH 3/9] KVM: x86/mmu: Set shadow_dirty_mask for EPT even if A/D bits disabled From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Set shadow_dirty_mask to the architectural EPT Dirty bit value even if A/D bits are disabled at the module level, i.e. even if KVM will never enable A/D bits in hardware. Doing so provides consistent behavior for Accessed and Dirty bits, i.e. doesn't leave KVM in a state where it sets shadow_accessed_mask but not shadow_dirty_mask. Functionally, this should be one big nop, as consumption of shadow_dirty_mask is always guarded by a check that hardware A/D bits are enabled. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/spte.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/kvm/mmu/spte.c b/arch/x86/kvm/mmu/spte.c index cae45825617c..a0ff504f1e7e 100644 --- a/arch/x86/kvm/mmu/spte.c +++ b/arch/x86/kvm/mmu/spte.c @@ -441,7 +441,7 @@ void kvm_mmu_set_ept_masks(bool has_ad_bits, bool has_exec_only) shadow_user_mask = VMX_EPT_READABLE_MASK; shadow_accessed_mask = VMX_EPT_ACCESS_BIT; - shadow_dirty_mask = has_ad_bits ? VMX_EPT_DIRTY_BIT : 0ull; + shadow_dirty_mask = VMX_EPT_DIRTY_BIT; shadow_nx_mask = 0ull; shadow_x_mask = VMX_EPT_EXECUTABLE_MASK; /* VMX_EPT_SUPPRESS_VE_BIT is needed for W or X violation. */ From patchwork Thu Aug 1 18:34:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 13750849 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E15DF14AD32 for ; Thu, 1 Aug 2024 18:35:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722537306; cv=none; b=T5ZxW+AAhGI8/f9Jtz3Gy3dTsQks3qsLdmEOZU9n1l+mWGpIUOvuOb8niuxfHubcP3SYeSJ2F5DrCWicej/BQh268QM1Rd7pp8wPwwiABQCnyk8ALQVHJ1Kg7d1ZFC+rao8k99k4Kw+D/ZwDG8KXPyEYvjvwy789ZigXX1mgWMs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722537306; c=relaxed/simple; bh=AFPTyvXyadKJHf2QzotKTohG8D5ttOVVpldlFI2PF/8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=SggkFADCQHio5qD9QkTmXt+NLkedP+DFIdw7ff61reZFAbsixYRADt1XeyiBgIuSlYlsrXOopRKfMmM8fDp1TKNcq3P4osIA9y+1zXHWjd+JtPKr8gHp/8W9zH9ha6HkW/xc9ACo6pO+UTICLwZloEGCo/JeCIpmFpzNg+srdZg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=C7tdv47U; arc=none smtp.client-ip=209.85.219.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="C7tdv47U" Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-e03623b24ddso10619397276.1 for ; Thu, 01 Aug 2024 11:35:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1722537304; x=1723142104; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=3iPJtAYuasIIRGf+GbxPHB6qYRb6caAhSB2/PDr0qIc=; b=C7tdv47URhonZ+WxIGRenpMDCXavAWGZS2cD2LJ1vVBzYKxwrDTJeULd6p4pt26iGD jEnLYrUgL+4qKG7R+ssYR6RwTVbXN9WAHW8n5IFqLrQpeLotqioICzTPrHOzBdXCkMgF 0s45KB1ZjtEezrkU98nMIJ17jXGjGODUQXPX2cYQiN/xI3yCgAOQTIqyUZ/yyJssJOp4 2pUvNcMqjQittJ44Z26466+qHpvtLxrkWTMH27jKiA4E7xswA7XXj0CBNonKLbKjqDZx Id4obGNAHSM73X7GbPTbkZiFulvePz9lfdPc1Qy7VZ2iVhXslYGZOkz82b6wbuuTepwr 9/Lw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722537304; x=1723142104; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=3iPJtAYuasIIRGf+GbxPHB6qYRb6caAhSB2/PDr0qIc=; b=qdVkGwGhQm10Inb3v7YDaZoKS6Dvubu6QiGkwPorPILPahCeSPEQj/fBbWVI74x/aN BSbRZdRe9t0IEeUL9FeENRJTrCiGgU03fpAsa7J+Ovf1QBjIyHFl5EO4iwb1mYIC2evn 0F54U7EqLXytIOyi2pyNmfvOVuK+xcFdDoDQVVTZo0u9uOmPYZHPBWmqs6NTtMinVxZt vMo3ARXVBsaQgTMtpIX6aifVj+rRZV5iu1UkOzzUmIMlrtGxim2f73UMmqr/uvlGp7eA lmzZPg2mGGR62JNKVMSAmmV8DsWjhpE/Gg75kbRGOZ0SA9BwecPsEtm/Aky/1gVGbZhR ovXQ== X-Gm-Message-State: AOJu0Yx37CbMM9SH7D9wEirdkbwTF8O4v9kakXMdNO+XjX22K70Ebcka MhSrqvGGqqsUjVXEXxbWbUlF0Tp+2H5ctgkBMxZLyOBUJ/TV/xmE4ktFbNac4VaNk1+Y26U2BY4 C7Q== X-Google-Smtp-Source: AGHT+IEPP91NM9n4/Te5UnC7Xo0l/Vq7qcp/6WQ3dD16sls4NkylR/VCD/51Fsi8RHqYZr1QyafatST0IEY= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a05:6902:2605:b0:e0b:bd79:307b with SMTP id 3f1490d57ef6-e0bde439c87mr43504276.9.1722537303997; Thu, 01 Aug 2024 11:35:03 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 1 Aug 2024 11:34:48 -0700 In-Reply-To: <20240801183453.57199-1-seanjc@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240801183453.57199-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.rc2.264.g509ed76dc8-goog Message-ID: <20240801183453.57199-5-seanjc@google.com> Subject: [RFC PATCH 4/9] KVM: x86/mmu: Use Accessed bit even when _hardware_ A/D bits are disabled From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Use the Accessed bit in SPTEs even when A/D bits are disabled in hardware, i.e. propagate accessed information to SPTE.Accessed even when KVM is doing manual tracking by making SPTEs not-present. In addition to eliminating a small amount of code in is_accessed_spte(), this also paves the way for preserving Accessed information when a SPTE is zapped in response to a mmu_notifier PROTECTION event, e.g. if a SPTE is zapped because NUMA balancing kicks in. Note, EPT is the only flavor of paging in which A/D bits are conditionally enabled, and the Accessed (and Dirty) bit is software-available when A/D bits are disabled. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 6 ++++-- arch/x86/kvm/mmu/spte.c | 6 +++--- arch/x86/kvm/mmu/spte.h | 11 +---------- 3 files changed, 8 insertions(+), 15 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 1e24bc4a06db..c8fc59fcc8e0 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -3454,8 +3454,10 @@ static int fast_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) * uses A/D bits for non-nested MMUs. Thus, if A/D bits are * enabled, the SPTE can't be an access-tracked SPTE. */ - if (unlikely(!kvm_ad_enabled) && is_access_track_spte(spte)) - new_spte = restore_acc_track_spte(new_spte); + if (unlikely(!kvm_ad_enabled) && is_access_track_spte(spte)) { + new_spte = restore_acc_track_spte(new_spte) | + shadow_accessed_mask; + } /* * To keep things simple, only SPTEs that are MMU-writable can diff --git a/arch/x86/kvm/mmu/spte.c b/arch/x86/kvm/mmu/spte.c index a0ff504f1e7e..ca1a8116de34 100644 --- a/arch/x86/kvm/mmu/spte.c +++ b/arch/x86/kvm/mmu/spte.c @@ -181,7 +181,7 @@ bool make_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp, spte |= shadow_present_mask; if (!prefetch) - spte |= spte_shadow_accessed_mask(spte); + spte |= shadow_accessed_mask; /* * For simplicity, enforce the NX huge page mitigation even if not @@ -258,7 +258,7 @@ bool make_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp, } if (pte_access & ACC_WRITE_MASK) - spte |= spte_shadow_dirty_mask(spte); + spte |= shadow_accessed_mask; out: if (prefetch) @@ -367,7 +367,7 @@ u64 mark_spte_for_access_track(u64 spte) spte |= (spte & SHADOW_ACC_TRACK_SAVED_BITS_MASK) << SHADOW_ACC_TRACK_SAVED_BITS_SHIFT; - spte &= ~shadow_acc_track_mask; + spte &= ~(shadow_acc_track_mask | shadow_accessed_mask); return spte; } diff --git a/arch/x86/kvm/mmu/spte.h b/arch/x86/kvm/mmu/spte.h index d722b37b7434..ba7ff1dfbeb2 100644 --- a/arch/x86/kvm/mmu/spte.h +++ b/arch/x86/kvm/mmu/spte.h @@ -316,12 +316,6 @@ static inline bool spte_ad_need_write_protect(u64 spte) return (spte & SPTE_TDP_AD_MASK) != SPTE_TDP_AD_ENABLED; } -static inline u64 spte_shadow_accessed_mask(u64 spte) -{ - KVM_MMU_WARN_ON(!is_shadow_present_pte(spte)); - return spte_ad_enabled(spte) ? shadow_accessed_mask : 0; -} - static inline u64 spte_shadow_dirty_mask(u64 spte) { KVM_MMU_WARN_ON(!is_shadow_present_pte(spte)); @@ -355,10 +349,7 @@ static inline kvm_pfn_t spte_to_pfn(u64 pte) static inline bool is_accessed_spte(u64 spte) { - u64 accessed_mask = spte_shadow_accessed_mask(spte); - - return accessed_mask ? spte & accessed_mask - : !is_access_track_spte(spte); + return spte & shadow_accessed_mask; } static inline bool is_dirty_spte(u64 spte) From patchwork Thu Aug 1 18:34:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 13750850 Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5A3CF14B94B for ; Thu, 1 Aug 2024 18:35:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722537307; cv=none; b=h9YFcZsSKSzol0suFTWaw6rN83moOBm3riW6TmKFs0OMuFjn/j8svHAXa9bF+9cPTkAUhtuZtWMyW+iwARa/7nC4eIV0h5+IaS+JQNfE6l20/827N7oSEbdheaTuWmeTw8ep5sPuIYCcHar7NW8DZfBSVlLfSVEpNYaBeKFAxDY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722537307; c=relaxed/simple; bh=+209Dl5mjZfEzPCjJbS6cYtHb+T8VqI2r8B2gBeStTo=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=mrWSaFA/OKBeObYVYXbTmR8QFj6twgSfqT0cVgzVPyIc4Zud2X7FUyozZqrhlJUbpGuaDIUj3vBhk3O34VLvHkNYGEPaFE/AfP6ALfqFGeDkV8oreKOjKrvn/QFnuLEKkaqOxiF+bSxw94Gtm68bsyERcPDbWwcLOaS73ODf2sg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=yg+DOyGW; arc=none smtp.client-ip=209.85.215.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="yg+DOyGW" Received: by mail-pg1-f201.google.com with SMTP id 41be03b00d2f7-78e323b3752so4784002a12.0 for ; Thu, 01 Aug 2024 11:35:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1722537306; x=1723142106; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=nLllgIeEL1wMurU3ykZwAR8levnh4hSK0+7xKRqL5Is=; b=yg+DOyGW4l4CuPriZwCOwA1CXdeCsL5OHnUHQq4LbhxF81Vjc7HCcYYkvOKz+kvbt2 ABmdS3dqKolKdl9fGuiZ5n/vTP18iMb3WEWlr6jbjMMEMihkBvMPkV1IZNyV0Yk/3qVm DWCXtK1YziHMIcnvFP8g0EgdfDL7jz1GbnNll3EAmTMcACLhmnvSKXZXZmKUHsJqamLZ SfFCqFkhiiyFAFAzLAUYQpwBZnC7l3+k0w3LyYltZTlRehAPBMffsQykie7Nr7azDIG/ WgyN2cMvPQS9V/cV5jhzxCrjtRZ5VvxbD252RIKZDWlqjhMP5pBnPWD8CiipV6q61JMu K5tA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722537306; x=1723142106; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=nLllgIeEL1wMurU3ykZwAR8levnh4hSK0+7xKRqL5Is=; b=Khh17TE2ZzglrP3xTRSJ/BFWX2zPOYX5STh5+cKQ2pjTm5KSwSVsY4jTMQawE4IJEt BtIgs685DJ9vads7cLZOcb1Ee+XGXteYuOSoN1gxnQICOyBlSC+/wtJVlg1GjKgNsOnV DWwrd5Tn0/1jUzU+in5O8yXjakidCz0aQgOqHi4WGqY669cWguJbvU+VR6JFVE93bJxy a2f0STtfsWPi1oYIGa+Ezzh5Su2qtcEXM9ERGgfG+RigD7Yx0WO2igeaBmeRZD4NcM3z Z/hElpofg/NVos8ac2rCu0xK080AtWpRGIS3beqx2YJvCDt/BHKICPWfWsLJ5dKKVSmg M91A== X-Gm-Message-State: AOJu0YxCxRonm37mbRXTQZMlXMusk3rfOtM+2HK5dkMTVjscgdT1OSli n0fJf5Hy824eB+nJ7CDXREGEZXX41DzTUVhyoKb+SPZfvEPcVvQ/Rqec78Ykd74FbgiuLR3bxlN 2tw== X-Google-Smtp-Source: AGHT+IGJTez0OFwM+Kmo7Ln7XZmkdAZ4Q1iTleNtDtm577nIAAAdrxOHonp4x5mucOt9Mkl1xYEJT8P1AFM= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a63:3f41:0:b0:7a1:2fb5:3ff7 with SMTP id 41be03b00d2f7-7b7438b1dc8mr1940a12.0.1722537305573; Thu, 01 Aug 2024 11:35:05 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 1 Aug 2024 11:34:49 -0700 In-Reply-To: <20240801183453.57199-1-seanjc@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240801183453.57199-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.rc2.264.g509ed76dc8-goog Message-ID: <20240801183453.57199-6-seanjc@google.com> Subject: [RFC PATCH 5/9] KVM: x86/mmu: Free up A/D bits in FROZEN_SPTE From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Remove all flavors of A/D bits from FROZEN_SPTE so that KVM can keep A/D bits set in SPTEs that are frozen, without getting false positives. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/spte.h | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/arch/x86/kvm/mmu/spte.h b/arch/x86/kvm/mmu/spte.h index ba7ff1dfbeb2..d403ecdfcb8e 100644 --- a/arch/x86/kvm/mmu/spte.h +++ b/arch/x86/kvm/mmu/spte.h @@ -216,15 +216,17 @@ extern u64 __read_mostly shadow_nonpresent_or_rsvd_mask; * should not modify the SPTE. * * Use a semi-arbitrary value that doesn't set RWX bits, i.e. is not-present on - * both AMD and Intel CPUs, and doesn't set PFN bits, i.e. doesn't create a L1TF - * vulnerability. + * both AMD and Intel CPUs, doesn't set any A/D bits, and doesn't set PFN bits, + * i.e. doesn't create a L1TF vulnerability. * * Only used by the TDP MMU. */ -#define FROZEN_SPTE (SHADOW_NONPRESENT_VALUE | 0x5a0ULL) +#define FROZEN_SPTE (SHADOW_NONPRESENT_VALUE | 0x498ULL) /* Removed SPTEs must not be misconstrued as shadow present PTEs. */ static_assert(!(FROZEN_SPTE & SPTE_MMU_PRESENT_MASK)); +static_assert(!(FROZEN_SPTE & (PT_ACCESSED_MASK | VMX_EPT_ACCESS_BIT))); +static_assert(!(FROZEN_SPTE & (PT_DIRTY_MASK | VMX_EPT_DIRTY_BIT))); static inline bool is_frozen_spte(u64 spte) { From patchwork Thu Aug 1 18:34:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 13750851 Received: from mail-pf1-f201.google.com (mail-pf1-f201.google.com [209.85.210.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 81A2E14C5A7 for ; Thu, 1 Aug 2024 18:35:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722537309; cv=none; b=GL0J2+NYs08aBOkJBSibhszTNfUyGZ2ff4K577O59BpQzPKrDGCOtZhflRu1pzTsDRFG/dqCdcK+z24jtRDndLV5LxjdLMzvy3KRgxn5ajdmUVr03hrst9OW0f5sFkBgo31iZ6grau3Qc2/bVEXnCtJ+PtKVHnxEg0xXKAIXOGQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722537309; c=relaxed/simple; bh=/XSoRdfaJzNo/ZGf04C230tSmY5GFwai1Y6z9V+ll3s=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=TbOroBtNPsgZ8kSwXADjqkLE6M6mIUW3icrru3skLkkiXqNX+ukdEkdWy6hIdJPSlZ/h4x9hQFXgLhCmCX1nwCmPwfHgghUQRel4mySgfiIUEOI4p6Fwwt9PrsZUbVVMBLF9cRd7Z7SkkmAChcutOudIJgO3TJhlZ3upQP39ggk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=T6f4Ughv; arc=none smtp.client-ip=209.85.210.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="T6f4Ughv" Received: by mail-pf1-f201.google.com with SMTP id d2e1a72fcca58-7104d2cac39so2257873b3a.1 for ; Thu, 01 Aug 2024 11:35:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1722537308; x=1723142108; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=YgfeHTUFXLVGjVFJpfNzwAaN7GsDEU7AXhuO1w5LlJU=; b=T6f4UghvEEptu57168f+L/zM2OSsfTUEmq1aYaownOtHbbaFnbOQ8PWU0JuKJgce0H 9fQZZawMVXL1956CtTaup8jRe66DfJ2jxln5sAhr19titUwv3w2uBWqXPqqVv6dd5WTu cGEgRY1qU5xXsf66qMwFiRjqe1LhSabKEpXXdGiP8fkoZFZ/AXnxagDFTf506ChfutbB BerF3q22V5B07c89wofMVqoMFo/AZJvDayCRHBYir2P56J1tZ7SoqPkDBHGQY5kJMLAd LCfJVXCXpldMdtKUwjNRwX2jOX+xLfvMyHOearGAJAIiUZPadY5Q6Hq2DGw2t2lH8qkY Il3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722537308; x=1723142108; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=YgfeHTUFXLVGjVFJpfNzwAaN7GsDEU7AXhuO1w5LlJU=; b=ZvRRLKxtW8a3hLjznc3wd/Np5uOrsiV/BfCnbP2fGQEy3EQ4bvEWvbVinM/+ATBQed f7VudWlQdP0TtdI4HtnwX5qNYR6eYWeoFjVThVrAQ1JU6wNadVUFIca1OPJEkT8a3NNT TdYven33xQ1Uvcsevb6Q6QT6hRiyCf2QoyBrbYj4iAETYnd11skriNaUIZcmLCopjE2d NtvPhPjOeM1O+I2tRqLng50gbOLgFVsEU8pGEKmF+P0u2AVu7aQQj+gd0eSbnRfGEsGI fWLk5yb9/H/a2QrG+FZbQ/peh8ptg8TfCB1XM9zdbjn0BZCkGV6F4x1VbXw1qTsAv7Ma dRAQ== X-Gm-Message-State: AOJu0Yz9jQR9pgdjqC7zX1QaENo6erndNTlT8wOUCi9FqPcEhGyds5L0 MMCP4CIcvOpCpFYytyY/fFK9slKqk7f0MsZPjfaqFbcg6OHQVLeBXFs/zf2m7kJqs/XK1B3aooe Q3A== X-Google-Smtp-Source: AGHT+IHTWEhjwHtp09vWEbUgtVyznzcr5+x5rCkfOWJrq49bIvfwRkVTlE43qqkCOx+dDWfaukt6/MOxYM0= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a05:6a00:949c:b0:705:ca19:2d08 with SMTP id d2e1a72fcca58-7106d0ca7camr8792b3a.6.1722537307533; Thu, 01 Aug 2024 11:35:07 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 1 Aug 2024 11:34:50 -0700 In-Reply-To: <20240801183453.57199-1-seanjc@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240801183453.57199-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.rc2.264.g509ed76dc8-goog Message-ID: <20240801183453.57199-7-seanjc@google.com> Subject: [RFC PATCH 6/9] KVM: x86/mmu: Process only valid TDP MMU roots when aging a gfn range From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Skip invalid TDP MMU roots when aging a gfn range. There is zero reason to process invalid roots, as they by definition hold stale information. E.g. if a root is invalid because its from a previous memslot generation, in the unlikely event the root has a SPTE for the gfn, then odds are good that the gfn=>hva mapping is different, i.e. doesn't map to the hva that is being aged by the primary MMU. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/tdp_mmu.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 2b0fc601d2ce..b358642890e1 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -1202,9 +1202,11 @@ static __always_inline bool kvm_tdp_mmu_handle_gfn(struct kvm *kvm, /* * Don't support rescheduling, none of the MMU notifiers that funnel - * into this helper allow blocking; it'd be dead, wasteful code. + * into this helper allow blocking; it'd be dead, wasteful code. Note, + * this helper must NOT be used to unmap GFNs, as it processes only + * valid roots! */ - for_each_tdp_mmu_root(kvm, root, range->slot->as_id) { + for_each_valid_tdp_mmu_root(kvm, root, range->slot->as_id) { rcu_read_lock(); tdp_root_for_each_leaf_pte(iter, root, range->start, range->end) From patchwork Thu Aug 1 18:34:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 13750852 Received: from mail-pf1-f201.google.com (mail-pf1-f201.google.com [209.85.210.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EBBDA14D2B5 for ; Thu, 1 Aug 2024 18:35:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722537312; cv=none; b=QuH9YU5Ub/Ad5AhvmragkrDj7cvbk/4TzZdRDL+bPGyf5Wvt9Rldblh1Ab6S3rI5WtUHrp09J8jvO82WequJ81KnkK9vCyoQNvKofE8gy2pnMk07HGJe2UyOG/xf8xCDB+QWd7xyxiPMwpQ9UQUNU0slvpJ09aAPVmWxihYHxpA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722537312; c=relaxed/simple; bh=kcyZHtFtSWTsZyQHZ3oFwxa1IXjn9+p3AjCTE5ktFqM=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=LKDvQabbkj6tXbYso49c+xtUOMrA0e+4cebXlVQZeTBTkf2S/RXh2yp4EJEWikf+6+bK+HhRMnrTC9w9V6fo/GQNAqFAB5livCnGinF4D363QE9bKBWxHF/NWkYAi/iArYEpGoN5rAHu3GBtUlKSQhA4RZfB0OwswvxV74kHCQU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=R/AJ2u0i; arc=none smtp.client-ip=209.85.210.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="R/AJ2u0i" Received: by mail-pf1-f201.google.com with SMTP id d2e1a72fcca58-71050384c9aso2329680b3a.1 for ; Thu, 01 Aug 2024 11:35:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1722537309; x=1723142109; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=MzTNDvjeVqrs7z6ZdAikXXC/sttVLmYnLoCNz/Cy7ro=; b=R/AJ2u0in/q7N5U13dxxbPnEoXpNcBME4V/loLgl8eYiNLb455O8l9sqEB4GPJa+xF 5LOt4K98IAOjlX4JOv8O7uyVVB4mAliIgWKcMFqdjqS5m6rePZd8ZuI4dbTpB/Oyu2iM MXjtLD+C6DZBoCXTMByNLtWV3l+lFUD8tDIQlh0wsU7hNIoWG6yM+590HHzII+QDfkar VR12eVFGBcdFCvtzbiKME+HqEHawIOcyXzTcoFCS+gs0C5GinTiEYUFfjttPUyuduJzI U0vJIVJJXKcoRZbOzayOgcyXS2eVHowuJD0YpfX5jNjvq1LJke0KEFcCn6u5cVlQhr1z sFnQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722537309; x=1723142109; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=MzTNDvjeVqrs7z6ZdAikXXC/sttVLmYnLoCNz/Cy7ro=; b=Z0PLyA7f2GzOqbg7FqHc+62Q2M/x26e+t6RmfXpVZ2BIfEPV+7U5qGdc+RRfmt0Qv0 JvtwyymMnHciOP7ENI/QXslTJj9h+AFAJjr8a85fPQ6EMJxHCcOUnmiTEx8t45vrEO1/ aqSrfAfq2YLkQMW1MROo72iUBmPCD40qjcnr7j72Ru54hZa2S9j9lpTvP0Sw2L8hwLfw 0HEEED3h5PbeUxj7aG8Y+Ijc6A0rIc48ctYvhQ+dF0TBR1gGgQMM/L26HC1pIOdhAZ1X /AtyMz17bMlMRA/0WF3YiE3iS83KYIpaZX219ZYuiNBFalyEmUJFol6FXM2vmAatS4pc REBA== X-Gm-Message-State: AOJu0YzJiNwUdylk5uvFKBG9CanQoI9YtCBC/o1KGHyNZ9j07NQN7Az9 HXHWMsVrAn5Xe/zuvN1+PcW2RlrskM2/QGkziHeMORmfk5KOLxb3HyundGTD4R3Dd5NjcZPze1Z /yw== X-Google-Smtp-Source: AGHT+IHBKWw5BAUfyMToUbh6nUKAFn26IAUUz572bHA0Rz/4Cn41vP3IxvpndgvHdAhxE1s8QOG25zRXuL0= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a05:6a00:6f12:b0:710:4e4c:a4ad with SMTP id d2e1a72fcca58-71065b6f385mr40762b3a.0.1722537309268; Thu, 01 Aug 2024 11:35:09 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 1 Aug 2024 11:34:51 -0700 In-Reply-To: <20240801183453.57199-1-seanjc@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240801183453.57199-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.rc2.264.g509ed76dc8-goog Message-ID: <20240801183453.57199-8-seanjc@google.com> Subject: [RFC PATCH 7/9] KVM: x86/mmu: Stop processing TDP MMU roots for test_age if young SPTE found From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Return immediately if a young SPTE is found when testing, but not updating, SPTEs. The return value is a boolean, i.e. whether there is one young SPTE or fifty is irrelevant (ignoring the fact that it's impossible for there to be fifty SPTEs, as KVM has a hard limit on the number of valid TDP MMU roots). Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/tdp_mmu.c | 84 ++++++++++++++++++-------------------- 1 file changed, 40 insertions(+), 44 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index b358642890e1..ac3200ce00f9 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -1189,35 +1189,6 @@ bool kvm_tdp_mmu_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range, return flush; } -typedef bool (*tdp_handler_t)(struct kvm *kvm, struct tdp_iter *iter, - struct kvm_gfn_range *range); - -static __always_inline bool kvm_tdp_mmu_handle_gfn(struct kvm *kvm, - struct kvm_gfn_range *range, - tdp_handler_t handler) -{ - struct kvm_mmu_page *root; - struct tdp_iter iter; - bool ret = false; - - /* - * Don't support rescheduling, none of the MMU notifiers that funnel - * into this helper allow blocking; it'd be dead, wasteful code. Note, - * this helper must NOT be used to unmap GFNs, as it processes only - * valid roots! - */ - for_each_valid_tdp_mmu_root(kvm, root, range->slot->as_id) { - rcu_read_lock(); - - tdp_root_for_each_leaf_pte(iter, root, range->start, range->end) - ret |= handler(kvm, &iter, range); - - rcu_read_unlock(); - } - - return ret; -} - /* * Mark the SPTEs range of GFNs [start, end) unaccessed and return non-zero * if any of the GFNs in the range have been accessed. @@ -1226,15 +1197,10 @@ static __always_inline bool kvm_tdp_mmu_handle_gfn(struct kvm *kvm, * from the clear_young() or clear_flush_young() notifier, which uses the * return value to determine if the page has been accessed. */ -static bool age_gfn_range(struct kvm *kvm, struct tdp_iter *iter, - struct kvm_gfn_range *range) +static void kvm_tdp_mmu_age_spte(struct tdp_iter *iter) { u64 new_spte; - /* If we have a non-accessed entry we don't need to change the pte. */ - if (!is_accessed_spte(iter->old_spte)) - return false; - if (spte_ad_enabled(iter->old_spte)) { iter->old_spte = tdp_mmu_clear_spte_bits(iter->sptep, iter->old_spte, @@ -1250,23 +1216,53 @@ static bool age_gfn_range(struct kvm *kvm, struct tdp_iter *iter, trace_kvm_tdp_mmu_spte_changed(iter->as_id, iter->gfn, iter->level, iter->old_spte, new_spte); - return true; +} + +static bool __kvm_tdp_mmu_age_gfn_range(struct kvm *kvm, + struct kvm_gfn_range *range, + bool test_only) +{ + struct kvm_mmu_page *root; + struct tdp_iter iter; + bool ret = false; + + /* + * Don't support rescheduling, none of the MMU notifiers that funnel + * into this helper allow blocking; it'd be dead, wasteful code. Note, + * this helper must NOT be used to unmap GFNs, as it processes only + * valid roots! + */ + for_each_valid_tdp_mmu_root(kvm, root, range->slot->as_id) { + rcu_read_lock(); + + tdp_root_for_each_leaf_pte(iter, root, range->start, range->end) { + if (!is_accessed_spte(iter.old_spte)) + continue; + + ret = true; + if (test_only) + break; + + kvm_tdp_mmu_age_spte(&iter); + } + + rcu_read_unlock(); + + if (ret && test_only) + break; + } + + return ret; } bool kvm_tdp_mmu_age_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range) { - return kvm_tdp_mmu_handle_gfn(kvm, range, age_gfn_range); -} - -static bool test_age_gfn(struct kvm *kvm, struct tdp_iter *iter, - struct kvm_gfn_range *range) -{ - return is_accessed_spte(iter->old_spte); + return __kvm_tdp_mmu_age_gfn_range(kvm, range, false); } bool kvm_tdp_mmu_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) { - return kvm_tdp_mmu_handle_gfn(kvm, range, test_age_gfn); + return __kvm_tdp_mmu_age_gfn_range(kvm, range, true); } /* From patchwork Thu Aug 1 18:34:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 13750853 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E58F214D6FE for ; Thu, 1 Aug 2024 18:35:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722537313; cv=none; b=ZkF+OD2iSN2AAi3CyLyLSy8vSL/GP2uyCOowZxcy5btycd7f9ILgEDNgx7Soe3mDWJEBsd15y6FL7r4ThlYoQ+8lOvdjD6bebuWBFSK1SZFTCAs5L+B6XT/mbmPPFfq61IfkXlXzHgl2TDZaGW0IrTUnw6OdstyDHHbL7/f7zDE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722537313; c=relaxed/simple; bh=RbIH3IP3u28tDH3NR3/offoYXurj3HvTAqkxRIuQJEQ=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=nUA2jTqUAd4tkNrWzpyP0u/vwvrYZglmqa+vXbvls1UUHXOTcjX9WxHORvpePZfCjQ1judPpwGuZmejQJuF4STQr3MaQVjhT5RYtWZXqc/QQPOBcqqvhq4jPjskmK2zGbOhyiIqOAG+BTsek2iZfkLtQTZuIBQ8c/HEe+WOw24c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=38WhoNgc; arc=none smtp.client-ip=209.85.219.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="38WhoNgc" Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-e0babce6718so5540997276.3 for ; Thu, 01 Aug 2024 11:35:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1722537311; x=1723142111; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=reMwPkVdpNL6CI3+i6mf1k9M5Xmr8NrhYIDXv6qn8Wo=; b=38WhoNgcqwS1ix5Duj6UAQjG74zQZN6kXOnA8QAjZxDHPynKrn8mwHwbqhFw0Vgu7s uKCv/OEnLAcKY7on2oUUyUNZMXRVyjCnDnXrxGVKxq5fuKUPLvWc4f/NfvmzcvNVj8O5 UDuuxE33Hzwgj9k9x2D3ma5rdrwRqnakk9OjzA8TchN4u6KC7qVXVA6N0EUxN03LFP/K MK65/QCl5tdOCxJsE7M4AshoS/qgDBOlfqRbV3lyO7YoYn9nAL2aL/cvtTUf7sMvdv3x OA75tTlfPTmC6uLLIFlBPpjWDOZtaK3ShrZIQhGhxDu9mPOrfVmFxe9LSPk+pqF/t+sA OXiA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722537311; x=1723142111; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=reMwPkVdpNL6CI3+i6mf1k9M5Xmr8NrhYIDXv6qn8Wo=; b=Jm7ZqfY39US16fO+oqhzFYG0cj7uPFuMnOXZvgHjU5P1yffnUeuH7YSrASzRwv8WAe nO5cLcTPX6yzNxfImhHgYfc8jJEMA8E/fyfCrvc8J1jBxPOTLV//OvrkWHgfvozzUe4x hes5pD59+iThGUQuncEi1NPpMuEa1j+pkLfRbIJafAZaHprQCJu6l1uXqYV5A6IBG6eS 0GaybUQG0PCLLjgB+v8sDnvZzf6jK4PXfa1tWBrijW1aNL8V8AkdaudnD7QDVdHFKvuX vbBSPoTIchWMQyE/9eAEcjS3sXysMmbqUgh375JafcgZjUqqRnbj++sLNamn8xjxoJc2 lq7A== X-Gm-Message-State: AOJu0YwApVWodgCy6XkDBh9JnyyOkE1zut/dFnyLLo7cK10oQc5aZI+q e2DDrPqFlyoV1Edfcnkw2MGFipkXKZ0SyEuQXe5bOZaU0jeokmby/VsC6LnmawSQi1fVWgKSdcI B7g== X-Google-Smtp-Source: AGHT+IF7cOBWTMyYm7b893kA5i2raHJ45P5vW+o+z7f1YVRpVI+wpdWY6KX7vJ+SzwpepoEVO382RoCb96g= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a05:6902:2483:b0:e03:a0b2:f73 with SMTP id 3f1490d57ef6-e0bde2f3cdbmr14329276.6.1722537310980; Thu, 01 Aug 2024 11:35:10 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 1 Aug 2024 11:34:52 -0700 In-Reply-To: <20240801183453.57199-1-seanjc@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240801183453.57199-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.rc2.264.g509ed76dc8-goog Message-ID: <20240801183453.57199-9-seanjc@google.com> Subject: [RFC PATCH 8/9] KVM: Plumb mmu_notifier invalidation event type into arch code From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Forward the mmu_notifier invalidation event information into the arch handler so that arch code can take different actions based on the invalidation type. E.g. x86 will use the information to preserve Accessed information when zapping SPTEs because of a protection change. Signed-off-by: Sean Christopherson --- include/linux/kvm_host.h | 1 + virt/kvm/kvm_main.c | 1 + 2 files changed, 2 insertions(+) diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 006668db9911..1fce5cf73b8e 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -252,6 +252,7 @@ int kvm_async_pf_wakeup_all(struct kvm_vcpu *vcpu); #ifdef CONFIG_KVM_GENERIC_MMU_NOTIFIER union kvm_mmu_notifier_arg { unsigned long attributes; + enum mmu_notifier_event event; }; struct kvm_gfn_range { diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index e279140f2425..3aa04e785d32 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -719,6 +719,7 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn, .start = range->start, .end = range->end, .handler = kvm_mmu_unmap_gfn_range, + .arg.event = range->event, .on_lock = kvm_mmu_invalidate_begin, .flush_on_ret = true, .may_block = mmu_notifier_range_blockable(range), From patchwork Thu Aug 1 18:34:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 13750854 Received: from mail-pf1-f202.google.com (mail-pf1-f202.google.com [209.85.210.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 855FE14E2D9 for ; Thu, 1 Aug 2024 18:35:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722537314; cv=none; b=gRyxwaA2jNOmdYB4v6iBAKc7uVllWppWtb7PkFpjHRjkD6qvpBpUgshVJhxEwLTypWIoDlOhu8vjXik4WlZCo3U4gBf5BY62zXyR13DmnGvbFyPW/Hv1WYj3zFXScR3GyuPYi8b56ktg6EmnxDi22IHGmt2wb5PXol9CWts9Coc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722537314; c=relaxed/simple; bh=QjtPf6OwO3F3v2qbeeMqBdAhvi3//8swMm8d6nCHE5E=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=ogIraGDKahWmwlLPRlcfyK8kC7C/msJVzHvg0QSMGyC5Ab+m1KUMiZn2UKVwq1l8zgV70jtBRXsP1XFahcZoVJm4J8NVE7la2ZurFNip5WQYvZrN02u2j2PxYRbQhNnO0j4uiqAWMDwn8qdi0zkOzuFGmWD/Xr7i50Alf7PBiWk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=0vlCK4Du; arc=none smtp.client-ip=209.85.210.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="0vlCK4Du" Received: by mail-pf1-f202.google.com with SMTP id d2e1a72fcca58-70d1cc32058so6101922b3a.2 for ; Thu, 01 Aug 2024 11:35:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1722537313; x=1723142113; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=pjpYJOfuFfBZxoqHyzYZOHO/aDFDcIHjHTQpaKTRfzU=; b=0vlCK4DuMb/cGY3MjcBI30JIteqLSMLxIDLShEgeDyBjmBo32FiQfYT0z2cRE01Ent LJFmalieCjcyZV5xjEFgd1LvEjVKr1/EMFw6WZLb/H/1Tdcbuk2WIXjP22m+fLM2wtGo /vnsYno5ZhfdjGu+zESkheRexa1pTHDrkWZsrZzUJ+GlEKO82GldDIDd6CLdbShu7BOZ 6CFRZcOEzQAZjQodiJUI0iNuHyY69vs70AWWmP1l1IM7O7lBur/1OPyFQ93SKdDC+UDu RFCvQEtCoRHBbGSGinbDRLVPgWtHytidmdSAmNXGuvfVbqzIzFujWkErrN5d2GpMkK7r CE4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722537313; x=1723142113; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=pjpYJOfuFfBZxoqHyzYZOHO/aDFDcIHjHTQpaKTRfzU=; b=F+56O7KSX5k6Md3NBWulM46ZN9ywotwdpda2Ft+9hOQhO8RyDgAXhmtDP6/nPG3zfg r2yP8hpbqiGEwQ8Zi3vRRz0qk27zuvHyEcDGPxIVXh3m3lHAKLWxs086UWLPKfeYFFwy 4TIlUhkwCqO/0mG2F5yQ5RYHe3wqjix0zVaPHIem7kNyoThqWGtENc4/nEH3cCPKnaDH l5tQRVe2Kw5/gZfYvDb18p5DW8Txa2vrF//8pZbSR6AvxUwsCGM1G6b6Fn/hFfLvQDye VqFIJPqdqSUpiiFDRO8z/0XTA9SRRMp+Gs6/xqOe6VLrvOA0sZbZgiQhYZu8uPxvFNXp 9POg== X-Gm-Message-State: AOJu0Yzx+K6cbdESQEWlcn1j9LqgkbZDpmFnR+E+1AE3U9fBHft3Ro3x 9XdOFDM5ShGlTKIZ3FQk1ktheZs+zLaCKJp+YNUvq2YPGWKMUvCxQRtWducqJ0Bo9EtKW/izeE3 L6Q== X-Google-Smtp-Source: AGHT+IFcO+Vr6uQpb300nP1wpu6DqsAAMIX15/3LQD4+pSQyjhlaJV4bhz777we1jEIWnrv4DvkpVknRK04= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a05:6a00:6f13:b0:70e:9e1b:1a83 with SMTP id d2e1a72fcca58-7106d01e048mr17749b3a.2.1722537312706; Thu, 01 Aug 2024 11:35:12 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 1 Aug 2024 11:34:53 -0700 In-Reply-To: <20240801183453.57199-1-seanjc@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240801183453.57199-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.rc2.264.g509ed76dc8-goog Message-ID: <20240801183453.57199-10-seanjc@google.com> Subject: [RFC PATCH 9/9] KVM: x86/mmu: Track SPTE accessed info across mmu_notifier PROT changes From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Preserve Accessed information when zapping SPTEs in response to an mmu_notifier protection change, e.g. if KVM is zapping SPTEs because NUMA balancing kicked in. KVM is not required to fully unmap the SPTE, and the core VMA information isn't changing, i.e. the information is still fresh and useful. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/tdp_mmu.c | 31 +++++++++++++++++++++++++------ 1 file changed, 25 insertions(+), 6 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index ac3200ce00f9..780f35a22c05 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -838,7 +838,8 @@ bool kvm_tdp_mmu_zap_sp(struct kvm *kvm, struct kvm_mmu_page *sp) * operation can cause a soft lockup. */ static bool tdp_mmu_zap_leafs(struct kvm *kvm, struct kvm_mmu_page *root, - gfn_t start, gfn_t end, bool can_yield, bool flush) + gfn_t start, gfn_t end, bool can_yield, + bool keep_accessed_bit, bool flush) { struct tdp_iter iter; @@ -849,17 +850,29 @@ static bool tdp_mmu_zap_leafs(struct kvm *kvm, struct kvm_mmu_page *root, rcu_read_lock(); for_each_tdp_pte_min_level(iter, root, PG_LEVEL_4K, start, end) { + u64 new_spte = SHADOW_NONPRESENT_VALUE; + if (can_yield && tdp_mmu_iter_cond_resched(kvm, &iter, flush, false)) { flush = false; continue; } + /* + * Note, this will fail to clear non-present, accessed SPTEs, + * but that isn't a functional problem, it can only result in + * a _potential_ false positive in the unlikely scenario that + * the primary MMU zaps an hva, reinstalls a new hva, and ages + * the new hva, all before KVM accesses the hva. + */ if (!is_shadow_present_pte(iter.old_spte) || !is_last_spte(iter.old_spte, iter.level)) continue; - tdp_mmu_iter_set_spte(kvm, &iter, SHADOW_NONPRESENT_VALUE); + if (keep_accessed_bit) + new_spte |= iter.old_spte & shadow_accessed_mask; + + tdp_mmu_iter_set_spte(kvm, &iter, new_spte); /* * Zappings SPTEs in invalid roots doesn't require a TLB flush, @@ -889,7 +902,7 @@ bool kvm_tdp_mmu_zap_leafs(struct kvm *kvm, gfn_t start, gfn_t end, bool flush) lockdep_assert_held_write(&kvm->mmu_lock); for_each_valid_tdp_mmu_root_yield_safe(kvm, root, -1) - flush = tdp_mmu_zap_leafs(kvm, root, start, end, true, flush); + flush = tdp_mmu_zap_leafs(kvm, root, start, end, true, false, flush); return flush; } @@ -1180,11 +1193,13 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) bool kvm_tdp_mmu_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range, bool flush) { + bool keep_a_bit = range->arg.event == MMU_NOTIFY_PROTECTION_VMA || + range->arg.event == MMU_NOTIFY_PROTECTION_PAGE; struct kvm_mmu_page *root; __for_each_tdp_mmu_root_yield_safe(kvm, root, range->slot->as_id, false) flush = tdp_mmu_zap_leafs(kvm, root, range->start, range->end, - range->may_block, flush); + range->may_block, keep_a_bit, flush); return flush; } @@ -1201,7 +1216,11 @@ static void kvm_tdp_mmu_age_spte(struct tdp_iter *iter) { u64 new_spte; - if (spte_ad_enabled(iter->old_spte)) { + if (spte_ad_enabled(iter->old_spte) || + !is_shadow_present_pte(iter->old_spte)) { + KVM_MMU_WARN_ON(!is_shadow_present_pte(iter->old_spte) && + iter->old_spte != (SHADOW_NONPRESENT_VALUE | shadow_accessed_mask)); + iter->old_spte = tdp_mmu_clear_spte_bits(iter->sptep, iter->old_spte, shadow_accessed_mask, @@ -1235,7 +1254,7 @@ static bool __kvm_tdp_mmu_age_gfn_range(struct kvm *kvm, for_each_valid_tdp_mmu_root(kvm, root, range->slot->as_id) { rcu_read_lock(); - tdp_root_for_each_leaf_pte(iter, root, range->start, range->end) { + tdp_root_for_each_pte(iter, root, range->start, range->end) { if (!is_accessed_spte(iter.old_spte)) continue;