From patchwork Sat Feb 11 01:46:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vipin Sharma X-Patchwork-Id: 13136692 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7C9CCC05027 for ; Sat, 11 Feb 2023 01:46:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229954AbjBKBqh (ORCPT ); Fri, 10 Feb 2023 20:46:37 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57514 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229921AbjBKBqf (ORCPT ); Fri, 10 Feb 2023 20:46:35 -0500 Received: from mail-pf1-x449.google.com (mail-pf1-x449.google.com [IPv6:2607:f8b0:4864:20::449]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 628E97E8F1 for ; Fri, 10 Feb 2023 17:46:34 -0800 (PST) Received: by mail-pf1-x449.google.com with SMTP id ds10-20020a056a004aca00b0059c8629c220so3471514pfb.23 for ; Fri, 10 Feb 2023 17:46:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=6MGvE068epLiCiKO1TAoNVUmSZBBRkXrwHXhAiWz0E8=; b=MGK8U9JKxXLEY3zJNoNM7+6IniYornmEEv05U2sGpm2O9eFj4auGyNcX24Eo43wk5O fHZiLPKVSIf3MFAZRo8T7W98SKx92zL+WmqcssJm9yZxLK51iIVdWbS5asBU401uiVds 05Ktad81SM43kWtp3PBg8EiF0ld/oYIFeqIdI/znw/MCZ/a8vG+K0D23LMJpovSyNHZc yRP7ru9d/M1lHVBsdfT8G1bbcfrJnZjIzpNyTwgrQCwPM4oLUOBvaYjLPnpSIx+ki7qj 4tOoKaz7LslUVY+It/VYOrY5IHw2lpL5kjqnm8OCBNXsvcFQ5B6AUOBUQd63HPuB0YNP LIdg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6MGvE068epLiCiKO1TAoNVUmSZBBRkXrwHXhAiWz0E8=; b=Wj+J5VWKM5PUKpKWTrjgrsT26/dChEtp1wRkYt4kTxjGz8MHsNDNayVPEK0VTXt+7R t0jkhPARdRw4beShScGi61yf38q+lK/SFgoPQbLzHlO81CU30bKGvXmlPaARrLinbO7u 8R6NZaKNNFwxj+Ih+PmdPShfI9R9Dk/CAgSjE4KOnolY6+sJDvPhBqTp0R4HcejjJJ+5 Jaiiz463/MvzA0E4llGEyoabA7w5NQ/y0bPJHKvW98Wq6u3Tt+I0DQqC5atMxXCIO1SV JKc4sKPi5f4Rsuz+5EcSqx1JEP4YBCvC9vCJocIprEuZ0GJtucNX+krUnIfTpGnGqWtP t/cA== X-Gm-Message-State: AO0yUKXiOaC63Y2zMiyhQNJ36J9xWtcMrJL7pwhjIZoqTU2ivisyqiTv 27ePUSSNUEO4tJyQnE34vOZZ/UtvZNo1 X-Google-Smtp-Source: AK7set/JFBGz37t345sdNfBqXNYa7A4iPZMNWpg/ZmlKXGa/6FL7sIXl00Sw+uNwrB4SiFY6PM+UfWliQ+VL X-Received: from vipin.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:479f]) (user=vipinsh job=sendgmr) by 2002:a63:9602:0:b0:4da:85b1:e9c with SMTP id c2-20020a639602000000b004da85b10e9cmr3134854pge.100.1676079993740; Fri, 10 Feb 2023 17:46:33 -0800 (PST) Date: Fri, 10 Feb 2023 17:46:20 -0800 In-Reply-To: <20230211014626.3659152-1-vipinsh@google.com> Mime-Version: 1.0 References: <20230211014626.3659152-1-vipinsh@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230211014626.3659152-2-vipinsh@google.com> Subject: [Patch v3 1/7] KVM: x86/mmu: Add a helper function to check if an SPTE needs atomic write From: Vipin Sharma To: seanjc@google.com, pbonzini@redhat.com, bgardon@google.com, dmatlack@google.com Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Vipin Sharma Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Move conditions in kvm_tdp_mmu_write_spte() to check if an SPTE should be written atomically or not to a separate function. This new function, kvm_tdp_mmu_spte_need_atomic_write(), will be used in future commits to optimize clearing bits in SPTEs. Signed-off-by: Vipin Sharma Reviewed-by: David Matlack Reviewed-by: Ben Gardon --- arch/x86/kvm/mmu/tdp_iter.h | 34 ++++++++++++++++++++-------------- 1 file changed, 20 insertions(+), 14 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_iter.h b/arch/x86/kvm/mmu/tdp_iter.h index f0af385c56e0..c11c5d00b2c1 100644 --- a/arch/x86/kvm/mmu/tdp_iter.h +++ b/arch/x86/kvm/mmu/tdp_iter.h @@ -29,23 +29,29 @@ static inline void __kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 new_spte) WRITE_ONCE(*rcu_dereference(sptep), new_spte); } +/* + * SPTEs must be modified atomically if they are shadow-present, leaf + * SPTEs, and have volatile bits, i.e. has bits that can be set outside + * of mmu_lock. The Writable bit can be set by KVM's fast page fault + * handler, and Accessed and Dirty bits can be set by the CPU. + * + * Note, non-leaf SPTEs do have Accessed bits and those bits are + * technically volatile, but KVM doesn't consume the Accessed bit of + * non-leaf SPTEs, i.e. KVM doesn't care if it clobbers the bit. This + * logic needs to be reassessed if KVM were to use non-leaf Accessed + * bits, e.g. to skip stepping down into child SPTEs when aging SPTEs. + */ +static inline bool kvm_tdp_mmu_spte_need_atomic_write(u64 old_spte, int level) +{ + return is_shadow_present_pte(old_spte) && + is_last_spte(old_spte, level) && + spte_has_volatile_bits(old_spte); +} + static inline u64 kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 old_spte, u64 new_spte, int level) { - /* - * Atomically write the SPTE if it is a shadow-present, leaf SPTE with - * volatile bits, i.e. has bits that can be set outside of mmu_lock. - * The Writable bit can be set by KVM's fast page fault handler, and - * Accessed and Dirty bits can be set by the CPU. - * - * Note, non-leaf SPTEs do have Accessed bits and those bits are - * technically volatile, but KVM doesn't consume the Accessed bit of - * non-leaf SPTEs, i.e. KVM doesn't care if it clobbers the bit. This - * logic needs to be reassessed if KVM were to use non-leaf Accessed - * bits, e.g. to skip stepping down into child SPTEs when aging SPTEs. - */ - if (is_shadow_present_pte(old_spte) && is_last_spte(old_spte, level) && - spte_has_volatile_bits(old_spte)) + if (kvm_tdp_mmu_spte_need_atomic_write(old_spte, level)) return kvm_tdp_mmu_write_spte_atomic(sptep, new_spte); __kvm_tdp_mmu_write_spte(sptep, new_spte); From patchwork Sat Feb 11 01:46:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vipin Sharma X-Patchwork-Id: 13136693 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 659BBC636D4 for ; Sat, 11 Feb 2023 01:46:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229968AbjBKBqi (ORCPT ); Fri, 10 Feb 2023 20:46:38 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57534 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229951AbjBKBqg (ORCPT ); Fri, 10 Feb 2023 20:46:36 -0500 Received: from mail-pl1-x649.google.com (mail-pl1-x649.google.com [IPv6:2607:f8b0:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F1DD77E8F1 for ; Fri, 10 Feb 2023 17:46:35 -0800 (PST) Received: by mail-pl1-x649.google.com with SMTP id u6-20020a170903124600b00188cd4769bcso3796851plh.0 for ; Fri, 10 Feb 2023 17:46:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ylQ3Rvj2lB+88rdqpws35hu/21Y36j4VWX2N801NtKE=; b=C5Kk4fsWBf9kv5U1Qpa+U0cRxkXaRvZkMCQcwSbzIWmaJ+hHmiprlZeqdKj2u+eYxA fAGqWnQXXpPqhjIRRFJ0011pgAMHb+sMwNM5eAm8C4adyGpalV5McmX4s75Iqs+PHmAj 5Y/c3WfgxzXMriP4aoE3FO4LAtuVVpqEAy78xXV9q6uAsV+0WslQoyXOpQIaINDz6BYC /47DxtT2suW/zMdIuyrcGsvKYv5iQVj2161i8iCLYnoqm3KwMzkw+Qwyt7wLZDnH3M0B t16nDYRqLoiQ1N3KLASolNpwFD18YbaKeu+NF0S3NWPpbJ9jboy/7yHV3LtNoFSAqt9I PwNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ylQ3Rvj2lB+88rdqpws35hu/21Y36j4VWX2N801NtKE=; b=V+DeHgxtmoPnvzh/E2o+s6u+NRKOpyotN+VCNUrlGyoFiH/U5oCQV0hWyv+DiRgxpW yG4lRI0f2wun27iokvzIU8/kfgekaE+UJP0eweGKfuOCyU9j3/pBoU5sY6wuJd0cA91+ 0pZvj67oxyjXEBJPwvtQzSKvpk6RJssy8YJBhxIG1JnsbT2dBDR0HYSbOr7gvRedlTkQ TPARLWWaly/isxFUG6Jq4Bpqf3fstALff/HEkpteaWfCsThiTDjxPOYiPUfZMBeq32Hz 4OoBIBLriMxYsri6Ymc7eFuykyTGu494OukFUm4zqU3DIkZrVnWkpYxvyGTkx2d6wARK /KoA== X-Gm-Message-State: AO0yUKUYKXIPwL7qAzCYGK7iwdV1VrGEFKiwAwcX9MMI1lgjhMx/oyKN PIraRObedDkr+oZBZ0C94v2lCh4Qp3x7 X-Google-Smtp-Source: AK7set/E73OchTTBFaySUO9KPeYHWBzgKB9X+c//+xmmRYkSgD0kBZIqT1GdHv9Z4Yoc6TvtA9nWmbJaNcyP X-Received: from vipin.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:479f]) (user=vipinsh job=sendgmr) by 2002:a17:90b:312:b0:233:bf8f:82a4 with SMTP id ay18-20020a17090b031200b00233bf8f82a4mr329176pjb.72.1676079995322; Fri, 10 Feb 2023 17:46:35 -0800 (PST) Date: Fri, 10 Feb 2023 17:46:21 -0800 In-Reply-To: <20230211014626.3659152-1-vipinsh@google.com> Mime-Version: 1.0 References: <20230211014626.3659152-1-vipinsh@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230211014626.3659152-3-vipinsh@google.com> Subject: [Patch v3 2/7] KVM: x86/mmu: Atomically clear SPTE dirty state in the clear-dirty-log flow From: Vipin Sharma To: seanjc@google.com, pbonzini@redhat.com, bgardon@google.com, dmatlack@google.com Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Vipin Sharma Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Do atomic-AND to clear the dirty state of SPTEs. Optimize clear-dirty-log flow by avoiding to go through __handle_changed_spte() and directly call kvm_set_pfn_dirty() instead. Atomic-AND allows to fetch the latest value in SPTE, clear only its dirty state and set the new SPTE value. This optimization avoids executing unnecessary checks by not calling __handle_changed_spte(). With the removal of tdp_mmu_set_spte_no_dirty_log(), "record_dirty_log" parameter in __tdp_mmu_set_spte() is now obsolete. It will always be set to true by its caller. This dead code will be cleaned up in future commits. Tested on a VM (160 vCPUs, 160 GB memory) and found that performance of clear dirty log stage improved by ~40% in dirty_log_perf_test Before optimization: -------------------- Iteration 1 clear dirty log time: 3.638543593s Iteration 2 clear dirty log time: 3.145032742s Iteration 3 clear dirty log time: 3.142340358s Clear dirty log over 3 iterations took 9.925916693s. (Avg 3.308638897s/iteration) After optimization: ------------------- Iteration 1 clear dirty log time: 2.318988110s Iteration 2 clear dirty log time: 1.794470164s Iteration 3 clear dirty log time: 1.791668628s Clear dirty log over 3 iterations took 5.905126902s. (Avg 1.968375634s/iteration) Signed-off-by: Vipin Sharma Reviewed-by: David Matlack --- arch/x86/kvm/mmu/tdp_iter.h | 14 ++++++++++++++ arch/x86/kvm/mmu/tdp_mmu.c | 35 +++++++++++++++-------------------- 2 files changed, 29 insertions(+), 20 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_iter.h b/arch/x86/kvm/mmu/tdp_iter.h index c11c5d00b2c1..fae559559a80 100644 --- a/arch/x86/kvm/mmu/tdp_iter.h +++ b/arch/x86/kvm/mmu/tdp_iter.h @@ -58,6 +58,20 @@ static inline u64 kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 old_spte, return old_spte; } +static inline u64 tdp_mmu_clear_spte_bits(tdp_ptep_t sptep, u64 old_spte, + u64 mask, int level) +{ + atomic64_t *sptep_atomic; + + if (kvm_tdp_mmu_spte_need_atomic_write(old_spte, level)) { + sptep_atomic = (atomic64_t *)rcu_dereference(sptep); + return (u64)atomic64_fetch_and(~mask, sptep_atomic); + } + + __kvm_tdp_mmu_write_spte(sptep, old_spte & ~mask); + return old_spte; +} + /* * A TDP iterator performs a pre-order walk over a TDP paging structure. */ diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index bba33aea0fb0..66ccbeb9d845 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -771,13 +771,6 @@ static inline void tdp_mmu_set_spte_no_acc_track(struct kvm *kvm, _tdp_mmu_set_spte(kvm, iter, new_spte, false, true); } -static inline void tdp_mmu_set_spte_no_dirty_log(struct kvm *kvm, - struct tdp_iter *iter, - u64 new_spte) -{ - _tdp_mmu_set_spte(kvm, iter, new_spte, true, false); -} - #define tdp_root_for_each_pte(_iter, _root, _start, _end) \ for_each_tdp_pte(_iter, _root, _start, _end) @@ -1677,8 +1670,13 @@ bool kvm_tdp_mmu_clear_dirty_slot(struct kvm *kvm, static void clear_dirty_pt_masked(struct kvm *kvm, struct kvm_mmu_page *root, gfn_t gfn, unsigned long mask, bool wrprot) { + /* + * Either all SPTEs in TDP MMU will need write protection or none. This + * contract will not be modified for TDP MMU pages. + */ + u64 clear_bit = (wrprot || !kvm_ad_enabled()) ? PT_WRITABLE_MASK : + shadow_dirty_mask; struct tdp_iter iter; - u64 new_spte; rcu_read_lock(); @@ -1693,19 +1691,16 @@ static void clear_dirty_pt_masked(struct kvm *kvm, struct kvm_mmu_page *root, mask &= ~(1UL << (iter.gfn - gfn)); - if (wrprot || spte_ad_need_write_protect(iter.old_spte)) { - if (is_writable_pte(iter.old_spte)) - new_spte = iter.old_spte & ~PT_WRITABLE_MASK; - else - continue; - } else { - if (iter.old_spte & shadow_dirty_mask) - new_spte = iter.old_spte & ~shadow_dirty_mask; - else - continue; - } + if (!(iter.old_spte & clear_bit)) + continue; - tdp_mmu_set_spte_no_dirty_log(kvm, &iter, new_spte); + iter.old_spte = tdp_mmu_clear_spte_bits(iter.sptep, + iter.old_spte, + clear_bit, iter.level); + trace_kvm_tdp_mmu_spte_changed(iter.as_id, iter.gfn, iter.level, + iter.old_spte, + iter.old_spte & ~clear_bit); + kvm_set_pfn_dirty(spte_to_pfn(iter.old_spte)); } rcu_read_unlock(); From patchwork Sat Feb 11 01:46:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vipin Sharma X-Patchwork-Id: 13136694 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F0F85C64EC7 for ; Sat, 11 Feb 2023 01:46:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229984AbjBKBql (ORCPT ); Fri, 10 Feb 2023 20:46:41 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57608 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229969AbjBKBqj (ORCPT ); Fri, 10 Feb 2023 20:46:39 -0500 Received: from mail-pg1-x54a.google.com (mail-pg1-x54a.google.com [IPv6:2607:f8b0:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4B05B84B93 for ; Fri, 10 Feb 2023 17:46:38 -0800 (PST) Received: by mail-pg1-x54a.google.com with SMTP id bs128-20020a632886000000b004fb3816e89eso3223885pgb.13 for ; Fri, 10 Feb 2023 17:46:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=qvyXszpTjt01Q056Izvp9tkChdBBaJSX/3iFEV2GgtY=; b=pXbFYzGjBuwJfNS6gYgtZkM+mMuYItL/rPu/0kjg9o1lSwd6WeiH3PKpTXldxh71c6 RThUEh1i98Z9k2lbyktBMh4JMSqtNTpqq9xRTAVGI+c7urYxueHJva9EwOqDS/PcJuh4 yLqteVd2RHO54sWBM1gpvQQV9oNYp6gcv9EupI+qdSMLaivRobnO+c05PKqDPWPImyjq On6g2elHTKbcTVgvPnv01ouk5fStSdgun/QMR8uVb4RTQ6EWCaASFeMVu/mkxpow4ac3 leyLD8IJ9xI2tGRZ/jSI7/vyhL04+iGYD0l1qFD9CKfwbjarb6laL+4AT08Q9b/GT/ay PGmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=qvyXszpTjt01Q056Izvp9tkChdBBaJSX/3iFEV2GgtY=; b=ZQVtsStLxIPY9JEYAcRwfgYCfcM2w41Oj2kTzVAAcz2vn4yFIIGTYHtdXPch8XRbji 3meo7vjQJ+JgowfX8wRvP1CHKR+8Cwx5bKDJUWWUTu1/y3O0RZV4ChWi2Y79P4AAvo5e /zlZcNc5B66TXBcjYIrbqwt8HXdYwrVdeL2+2B+hG4aq6/FtyopzuIx3YRTRwoUNbeEu cJ+BUhswTyYoTvdg4EyKNj5gf/+aJFXNz/LMf5WXKlLNrZI5WqoOyeU1ePtI+X2xKFAS vEFpqwgNSsPdFci80zHRn5oTp+894vkoWYvKHxeHaK0qRYSafHhVgk+p2ODqgvavAl6Z MGAA== X-Gm-Message-State: AO0yUKXp9c6Jw3OHDJEMcV+qONr6YjFKCbjq94jH4xc9VQ1CenuNsy7M RHkwXTq8vXsvnR0ZM59VxlhL53BNhFTQ X-Google-Smtp-Source: AK7set88Pn+HoZLc3TBke6+sHNxlhxbdjSn/64zcLE8iYOBdZHck60mTQXFtdxLlJC3hr723/QgnYUHhyoB1 X-Received: from vipin.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:479f]) (user=vipinsh job=sendgmr) by 2002:aa7:9a0e:0:b0:56b:b520:3751 with SMTP id w14-20020aa79a0e000000b0056bb5203751mr3710282pfj.29.1676079997607; Fri, 10 Feb 2023 17:46:37 -0800 (PST) Date: Fri, 10 Feb 2023 17:46:22 -0800 In-Reply-To: <20230211014626.3659152-1-vipinsh@google.com> Mime-Version: 1.0 References: <20230211014626.3659152-1-vipinsh@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230211014626.3659152-4-vipinsh@google.com> Subject: [Patch v3 3/7] KVM: x86/mmu: Remove "record_dirty_log" in __tdp_mmu_set_spte() From: Vipin Sharma To: seanjc@google.com, pbonzini@redhat.com, bgardon@google.com, dmatlack@google.com Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Vipin Sharma Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Remove bool parameter "record_dirty_log" from __tdp_mmu_set_spte() and refactor the code as this variable is always set to true by its caller. Signed-off-by: Vipin Sharma Reviewed-by: David Matlack --- arch/x86/kvm/mmu/tdp_mmu.c | 24 +++++++++--------------- 1 file changed, 9 insertions(+), 15 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 66ccbeb9d845..c895560244de 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -710,18 +710,13 @@ static inline int tdp_mmu_zap_spte_atomic(struct kvm *kvm, * notifier for access tracking. Leaving record_acc_track * unset in that case prevents page accesses from being * double counted. - * @record_dirty_log: Record the page as dirty in the dirty bitmap if - * appropriate for the change being made. Should be set - * unless performing certain dirty logging operations. - * Leaving record_dirty_log unset in that case prevents page - * writes from being double counted. * * Returns the old SPTE value, which _may_ be different than @old_spte if the * SPTE had voldatile bits. */ static u64 __tdp_mmu_set_spte(struct kvm *kvm, int as_id, tdp_ptep_t sptep, u64 old_spte, u64 new_spte, gfn_t gfn, int level, - bool record_acc_track, bool record_dirty_log) + bool record_acc_track) { lockdep_assert_held_write(&kvm->mmu_lock); @@ -740,35 +735,34 @@ static u64 __tdp_mmu_set_spte(struct kvm *kvm, int as_id, tdp_ptep_t sptep, if (record_acc_track) handle_changed_spte_acc_track(old_spte, new_spte, level); - if (record_dirty_log) - handle_changed_spte_dirty_log(kvm, as_id, gfn, old_spte, - new_spte, level); + + handle_changed_spte_dirty_log(kvm, as_id, gfn, old_spte, new_spte, + level); return old_spte; } static inline void _tdp_mmu_set_spte(struct kvm *kvm, struct tdp_iter *iter, - u64 new_spte, bool record_acc_track, - bool record_dirty_log) + u64 new_spte, bool record_acc_track) { WARN_ON_ONCE(iter->yielded); iter->old_spte = __tdp_mmu_set_spte(kvm, iter->as_id, iter->sptep, iter->old_spte, new_spte, iter->gfn, iter->level, - record_acc_track, record_dirty_log); + record_acc_track); } static inline void tdp_mmu_set_spte(struct kvm *kvm, struct tdp_iter *iter, u64 new_spte) { - _tdp_mmu_set_spte(kvm, iter, new_spte, true, true); + _tdp_mmu_set_spte(kvm, iter, new_spte, true); } static inline void tdp_mmu_set_spte_no_acc_track(struct kvm *kvm, struct tdp_iter *iter, u64 new_spte) { - _tdp_mmu_set_spte(kvm, iter, new_spte, false, true); + _tdp_mmu_set_spte(kvm, iter, new_spte, false); } #define tdp_root_for_each_pte(_iter, _root, _start, _end) \ @@ -918,7 +912,7 @@ bool kvm_tdp_mmu_zap_sp(struct kvm *kvm, struct kvm_mmu_page *sp) return false; __tdp_mmu_set_spte(kvm, kvm_mmu_page_as_id(sp), sp->ptep, old_spte, 0, - sp->gfn, sp->role.level + 1, true, true); + sp->gfn, sp->role.level + 1, true); return true; } From patchwork Sat Feb 11 01:46:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vipin Sharma X-Patchwork-Id: 13136695 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0EC32C05027 for ; Sat, 11 Feb 2023 01:46:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230016AbjBKBqv (ORCPT ); Fri, 10 Feb 2023 20:46:51 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57678 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229977AbjBKBql (ORCPT ); Fri, 10 Feb 2023 20:46:41 -0500 Received: from mail-pf1-x449.google.com (mail-pf1-x449.google.com [IPv6:2607:f8b0:4864:20::449]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DF84B84B91 for ; Fri, 10 Feb 2023 17:46:39 -0800 (PST) Received: by mail-pf1-x449.google.com with SMTP id s4-20020a056a00194400b0058d9b9fecb6so3436637pfk.1 for ; Fri, 10 Feb 2023 17:46:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=+gGMkWLRs9fWzgoKMZp1xjTHNEtbWSt2tA+J164dR+I=; b=n4w2JvSZxbvAx7VDuTa8a6gnidk7+fM4SN3MoyITzRdo8Zusatugfb0N1mp7zDdAtm nkhT4ceurl+KSeBYu9KzUTa8oFiachS16q0TOsJhkSXmoY5VcCxLGE77JXnMLtd4AlJm nYaND+RHtk6DaGb8UbXmjF67jjmu7+tbRomwRl8q4whlT68/prZH4lf6oAKIuMqg3Oog 7jRUqsAdomH6eGOi5pQpQdIXfWvKdc17woQPwGK56J49NXm1kQFEkeER/UbFB7IRBE4E ds6/M+/hv1rxXm244otjozs+JI1/o3dSAAc9NOQzc/cN4c27HBZFHrDd9YmeV9F+D+Ts TwYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=+gGMkWLRs9fWzgoKMZp1xjTHNEtbWSt2tA+J164dR+I=; b=Fo2dcUIIxBt2y6eqt4uW3vkVXd9oMfCbiH1kfFf8DKMFKyGrgzctRu5wTEZRY5r2Hn xz821AqPLImV4BHQ/d4V8TWMrcR+GcEKvd51AlxA8BCADAoZhRAeU/PYcVOwKo/gcUJN OpMM+9EKqwMuy4Z8B/dgd+JU5ivrW+6hE8LeWjErIXrKcwzyXhIeJsDFp9V2d8tFsJbm ZriSMHkjtHiR5aKWxRGpo0Xhvn0X6kQ10f5Y970NZPtEjuH5MjEOsLBLVBl6qZBxdVEY HfVeS9a6ErRBFlTakWH13CQsyVHwTysQEUknYmwwRhIuB5Zm+TptldCqToDFjHxqHsh0 Z+yg== X-Gm-Message-State: AO0yUKX2oOJkHhe3b4yD59heEjjvID2RXKn5t2U8Pxu06CGngJJqYGW2 VIirNQvFNxMJ7nfYp/4VrmZgc4/MyDnN X-Google-Smtp-Source: AK7set8K9Pxm27vSO52D5ATEFyE/B/V3FGyDTFtREQw5bTFuoc9urcOJfrVPxABPSX++bn6TZOR+0gt2vq91 X-Received: from vipin.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:479f]) (user=vipinsh job=sendgmr) by 2002:a17:902:e84f:b0:189:6df9:4f85 with SMTP id t15-20020a170902e84f00b001896df94f85mr4137680plg.27.1676079999320; Fri, 10 Feb 2023 17:46:39 -0800 (PST) Date: Fri, 10 Feb 2023 17:46:23 -0800 In-Reply-To: <20230211014626.3659152-1-vipinsh@google.com> Mime-Version: 1.0 References: <20230211014626.3659152-1-vipinsh@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230211014626.3659152-5-vipinsh@google.com> Subject: [Patch v3 4/7] KVM: x86/mmu: Optimize SPTE change for aging gfn range From: Vipin Sharma To: seanjc@google.com, pbonzini@redhat.com, bgardon@google.com, dmatlack@google.com Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Vipin Sharma Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org No need to check all of the conditions in __handle_changed_spte(). Aging a gfn range implies resetting access bit or marking spte for access tracking. Use atomic operation to only reset those bits. This avoids checking many conditions in __handle_changed_spte() API. Also, clean up code by removing dead code and API parameters. Signed-off-by: Vipin Sharma Reviewed-by: David Matlack --- arch/x86/kvm/mmu/tdp_mmu.c | 36 +++++++++++++++++++----------------- 1 file changed, 19 insertions(+), 17 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index c895560244de..5d6e77554797 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -758,13 +758,6 @@ static inline void tdp_mmu_set_spte(struct kvm *kvm, struct tdp_iter *iter, _tdp_mmu_set_spte(kvm, iter, new_spte, true); } -static inline void tdp_mmu_set_spte_no_acc_track(struct kvm *kvm, - struct tdp_iter *iter, - u64 new_spte) -{ - _tdp_mmu_set_spte(kvm, iter, new_spte, false); -} - #define tdp_root_for_each_pte(_iter, _root, _start, _end) \ for_each_tdp_pte(_iter, _root, _start, _end) @@ -1251,32 +1244,41 @@ static __always_inline bool kvm_tdp_mmu_handle_gfn(struct kvm *kvm, /* * Mark the SPTEs range of GFNs [start, end) unaccessed and return non-zero * if any of the GFNs in the range have been accessed. + * + * No need to mark corresponding PFN as accessed as this call is coming from + * the clear_young() or clear_flush_young() notifier, which uses the return + * value to determine if the page has been accessed. */ static bool age_gfn_range(struct kvm *kvm, struct tdp_iter *iter, struct kvm_gfn_range *range) { - u64 new_spte = 0; + u64 new_spte; /* If we have a non-accessed entry we don't need to change the pte. */ if (!is_accessed_spte(iter->old_spte)) return false; - new_spte = iter->old_spte; - - if (spte_ad_enabled(new_spte)) { - new_spte &= ~shadow_accessed_mask; + if (spte_ad_enabled(iter->old_spte)) { + iter->old_spte = tdp_mmu_clear_spte_bits(iter->sptep, + iter->old_spte, + shadow_accessed_mask, + iter->level); + new_spte = iter->old_spte & ~shadow_accessed_mask; } else { + new_spte = mark_spte_for_access_track(iter->old_spte); + iter->old_spte = kvm_tdp_mmu_write_spte(iter->sptep, + iter->old_spte, new_spte, + iter->level); /* * Capture the dirty status of the page, so that it doesn't get * lost when the SPTE is marked for access tracking. */ - if (is_writable_pte(new_spte)) - kvm_set_pfn_dirty(spte_to_pfn(new_spte)); - - new_spte = mark_spte_for_access_track(new_spte); + if (is_writable_pte(iter->old_spte)) + kvm_set_pfn_dirty(spte_to_pfn(iter->old_spte)); } - tdp_mmu_set_spte_no_acc_track(kvm, iter, new_spte); + trace_kvm_tdp_mmu_spte_changed(iter->as_id, iter->gfn, iter->level, + iter->old_spte, new_spte); return true; } From patchwork Sat Feb 11 01:46:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vipin Sharma X-Patchwork-Id: 13136696 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1ACD3C05027 for ; Sat, 11 Feb 2023 01:47:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230045AbjBKBq6 (ORCPT ); Fri, 10 Feb 2023 20:46:58 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58628 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230020AbjBKBqv (ORCPT ); Fri, 10 Feb 2023 20:46:51 -0500 Received: from mail-pf1-x449.google.com (mail-pf1-x449.google.com [IPv6:2607:f8b0:4864:20::449]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8606E84F72 for ; Fri, 10 Feb 2023 17:46:42 -0800 (PST) Received: by mail-pf1-x449.google.com with SMTP id u18-20020a62ed12000000b00593cc641da4so3431042pfh.0 for ; Fri, 10 Feb 2023 17:46:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ilQD1ZwfOZ+d2+O+AXt7dFLjcZcLaA1r/kAp0uBR7mU=; b=PdwRuNZr5qWt72w20yZfAeNVSTn+Uwkm5WaseLyWoj7zvwCv5tBjeMIQQwybX4veDF 006bGVjCu6QI8eRjrg32Xec+RnDfgzWKAiq6UcM6h8DUFq5IEkcMpD++bzRF4nC56zP+ LO7TiApbuF2VCN52PILf2eZ9f2dAanYCU+9Ki7zwdjTUUIF1cdpPCCZ1H+Q9riLM3c/K AknD+8fIbtUCg10CXhbQYiPpxBErkWPZDfpuCGjwX8y1NB7XClNWsirwkeCsiUfnUhqu 0Koth+qR6OIxRElg/2asJa+iK7/rpmoDuuS6qX9m9hA7rNk3qJ9C6mIQqrnKckRs8E8U 6LUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ilQD1ZwfOZ+d2+O+AXt7dFLjcZcLaA1r/kAp0uBR7mU=; b=fSt21F2t1VVuWf0IsUgFAMnx35+d+LfiFfqfYkAJZG67qaC5LFsl+IqNWXvqsIpfGm fSaTCeLzZratRa9W3JbcRaMLzVb8pzDGdxbfu0d9p0FU5jI8Fo9tQwWjPB7zosdy+iS2 rIUWDt71/3zAAVUlQWXV/OT9B+qoYOA2cdUJS3D8v0zMKcykJ6+jjJlhEBDJBsgd8i5Q lkQgYDX9sf5GBi9hIM1su28bqDfm8pKn5Ul6u2VLOlIJhqRA7zvkuTqQlfMbrPjUnDjT 0TTqrZSnx2GivNXau9ZAm4Uvki1TRb3uqBEtrhDvglmrXOFGy5Z/OY6907wbXLy6vmLh Wd7g== X-Gm-Message-State: AO0yUKUEDznXAxGUQxZpHG5W6d9AMgYf21xJvJocZaBITBwDQtttcQyN IQWfwNyVXuTC/yQPB6Wz94VeeCublsev X-Google-Smtp-Source: AK7set9U4Yaa0zpTFVx5Wxa0jeIjgqiyiZa13NAhHdcyUs62NVEkjx/lJfvzmjwiNtL4FFIrziz6y4YeF4qv X-Received: from vipin.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:479f]) (user=vipinsh job=sendgmr) by 2002:a17:902:e806:b0:198:eeb5:637b with SMTP id u6-20020a170902e80600b00198eeb5637bmr4391689plg.23.1676080000973; Fri, 10 Feb 2023 17:46:40 -0800 (PST) Date: Fri, 10 Feb 2023 17:46:24 -0800 In-Reply-To: <20230211014626.3659152-1-vipinsh@google.com> Mime-Version: 1.0 References: <20230211014626.3659152-1-vipinsh@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230211014626.3659152-6-vipinsh@google.com> Subject: [Patch v3 5/7] KVM: x86/mmu: Remove "record_acc_track" in __tdp_mmu_set_spte() From: Vipin Sharma To: seanjc@google.com, pbonzini@redhat.com, bgardon@google.com, dmatlack@google.com Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Vipin Sharma Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Remove bool parameter "record_acc_track" from __tdp_mmu_set_spte() and refactor the code. This variable is always set to true by its caller. Remove single and double underscore prefix from tdp_mmu_set_spte() related APIs: 1. Change __tdp_mmu_set_spte() to tdp_mmu_set_spte() 2. Change _tdp_mmu_set_spte() to tdp_mmu_iter_set_spte() Signed-off-by: Vipin Sharma Reviewed-by: David Matlack --- arch/x86/kvm/mmu/tdp_mmu.c | 51 +++++++++++++------------------------- 1 file changed, 17 insertions(+), 34 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 5d6e77554797..e50e869bf879 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -697,7 +697,7 @@ static inline int tdp_mmu_zap_spte_atomic(struct kvm *kvm, /* - * __tdp_mmu_set_spte - Set a TDP MMU SPTE and handle the associated bookkeeping + * tdp_mmu_set_spte - Set a TDP MMU SPTE and handle the associated bookkeeping * @kvm: KVM instance * @as_id: Address space ID, i.e. regular vs. SMM * @sptep: Pointer to the SPTE @@ -705,18 +705,12 @@ static inline int tdp_mmu_zap_spte_atomic(struct kvm *kvm, * @new_spte: The new value that will be set for the SPTE * @gfn: The base GFN that was (or will be) mapped by the SPTE * @level: The level _containing_ the SPTE (its parent PT's level) - * @record_acc_track: Notify the MM subsystem of changes to the accessed state - * of the page. Should be set unless handling an MMU - * notifier for access tracking. Leaving record_acc_track - * unset in that case prevents page accesses from being - * double counted. * * Returns the old SPTE value, which _may_ be different than @old_spte if the * SPTE had voldatile bits. */ -static u64 __tdp_mmu_set_spte(struct kvm *kvm, int as_id, tdp_ptep_t sptep, - u64 old_spte, u64 new_spte, gfn_t gfn, int level, - bool record_acc_track) +static u64 tdp_mmu_set_spte(struct kvm *kvm, int as_id, tdp_ptep_t sptep, + u64 old_spte, u64 new_spte, gfn_t gfn, int level) { lockdep_assert_held_write(&kvm->mmu_lock); @@ -732,30 +726,19 @@ static u64 __tdp_mmu_set_spte(struct kvm *kvm, int as_id, tdp_ptep_t sptep, old_spte = kvm_tdp_mmu_write_spte(sptep, old_spte, new_spte, level); __handle_changed_spte(kvm, as_id, gfn, old_spte, new_spte, level, false); - - if (record_acc_track) - handle_changed_spte_acc_track(old_spte, new_spte, level); - + handle_changed_spte_acc_track(old_spte, new_spte, level); handle_changed_spte_dirty_log(kvm, as_id, gfn, old_spte, new_spte, level); return old_spte; } -static inline void _tdp_mmu_set_spte(struct kvm *kvm, struct tdp_iter *iter, - u64 new_spte, bool record_acc_track) +static inline void tdp_mmu_iter_set_spte(struct kvm *kvm, struct tdp_iter *iter, + u64 new_spte) { WARN_ON_ONCE(iter->yielded); - - iter->old_spte = __tdp_mmu_set_spte(kvm, iter->as_id, iter->sptep, - iter->old_spte, new_spte, - iter->gfn, iter->level, - record_acc_track); -} - -static inline void tdp_mmu_set_spte(struct kvm *kvm, struct tdp_iter *iter, - u64 new_spte) -{ - _tdp_mmu_set_spte(kvm, iter, new_spte, true); + iter->old_spte = tdp_mmu_set_spte(kvm, iter->as_id, iter->sptep, + iter->old_spte, new_spte, + iter->gfn, iter->level); } #define tdp_root_for_each_pte(_iter, _root, _start, _end) \ @@ -847,7 +830,7 @@ static void __tdp_mmu_zap_root(struct kvm *kvm, struct kvm_mmu_page *root, continue; if (!shared) - tdp_mmu_set_spte(kvm, &iter, 0); + tdp_mmu_iter_set_spte(kvm, &iter, 0); else if (tdp_mmu_set_spte_atomic(kvm, &iter, 0)) goto retry; } @@ -904,8 +887,8 @@ bool kvm_tdp_mmu_zap_sp(struct kvm *kvm, struct kvm_mmu_page *sp) if (WARN_ON_ONCE(!is_shadow_present_pte(old_spte))) return false; - __tdp_mmu_set_spte(kvm, kvm_mmu_page_as_id(sp), sp->ptep, old_spte, 0, - sp->gfn, sp->role.level + 1, true); + tdp_mmu_set_spte(kvm, kvm_mmu_page_as_id(sp), sp->ptep, old_spte, 0, + sp->gfn, sp->role.level + 1); return true; } @@ -939,7 +922,7 @@ static bool tdp_mmu_zap_leafs(struct kvm *kvm, struct kvm_mmu_page *root, !is_last_spte(iter.old_spte, iter.level)) continue; - tdp_mmu_set_spte(kvm, &iter, 0); + tdp_mmu_iter_set_spte(kvm, &iter, 0); flush = true; } @@ -1110,7 +1093,7 @@ static int tdp_mmu_link_sp(struct kvm *kvm, struct tdp_iter *iter, if (ret) return ret; } else { - tdp_mmu_set_spte(kvm, iter, spte); + tdp_mmu_iter_set_spte(kvm, iter, spte); } tdp_account_mmu_page(kvm, sp); @@ -1317,13 +1300,13 @@ static bool set_spte_gfn(struct kvm *kvm, struct tdp_iter *iter, * invariant that the PFN of a present * leaf SPTE can never change. * See __handle_changed_spte(). */ - tdp_mmu_set_spte(kvm, iter, 0); + tdp_mmu_iter_set_spte(kvm, iter, 0); if (!pte_write(range->pte)) { new_spte = kvm_mmu_changed_pte_notifier_make_spte(iter->old_spte, pte_pfn(range->pte)); - tdp_mmu_set_spte(kvm, iter, new_spte); + tdp_mmu_iter_set_spte(kvm, iter, new_spte); } return true; @@ -1814,7 +1797,7 @@ static bool write_protect_gfn(struct kvm *kvm, struct kvm_mmu_page *root, if (new_spte == iter.old_spte) break; - tdp_mmu_set_spte(kvm, &iter, new_spte); + tdp_mmu_iter_set_spte(kvm, &iter, new_spte); spte_set = true; } From patchwork Sat Feb 11 01:46:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vipin Sharma X-Patchwork-Id: 13136697 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E7E9DC05027 for ; Sat, 11 Feb 2023 01:47:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230079AbjBKBrC (ORCPT ); Fri, 10 Feb 2023 20:47:02 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58684 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230025AbjBKBqw (ORCPT ); Fri, 10 Feb 2023 20:46:52 -0500 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0D50884BA2 for ; Fri, 10 Feb 2023 17:46:43 -0800 (PST) Received: by mail-yb1-xb4a.google.com with SMTP id x15-20020a25accf000000b008efe9505b2eso3242745ybd.22 for ; Fri, 10 Feb 2023 17:46:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=9cV6anjCgKUaY0e+B+aFQ5NbnTuwKip5tCWUOTAoaYc=; b=HZV5xjcaT/zj7344N4oejB5i1K5DQKr/S3o+xMZz7NM4yxEvq5V75WAGRuVSQ7qaRm ItCLApgwaCA+CtdXN9GgtI/046ysapHYNDO77M6uqY/hu1RY1xjhUnxfNFGA2YRnNVDB krpQ7jimSaCg2t1vnRdk5sXw9CUfJgtg2OGvMKnoe7sl0Vne/xQjDl6RoDhuwh+6c8Sd W7Uf9eUm6wE2Q4d/4vCLjUbf9PbV5fB1I/yADR8b0rWHfZDv61eRsBmAapmijMIWWef1 +1kTpL2ruNBjIjGCY22pwhL7DMqUgYJfP9Z98dboKDd430bC4cH3R71VZaCsl/B2hNmm AiMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=9cV6anjCgKUaY0e+B+aFQ5NbnTuwKip5tCWUOTAoaYc=; b=PJvEQFT6oRbDboSsTPE+SGJFA4lvYVAts4/X8EGVoDVyxTxmUGeOgDukTnbEb8ULB2 tXwzQC+tSLd2CEUMM23+vPbZK9yeVFiYVTRrg/FVeVLIvh7p1QVRCjBRWDGq7lyTLq5V DBambSvQVRZ/oDPE/rgPsDuyuxnX/doCp0JQTX11H++u2gy3Caf9D7SeD+f/mjOH7EUY z0U0GNL1oHD3eD5uk/xeLmQRv9sin9u3+4XQPBHfobARSfX2JdNCO5lDlXiFCVhowLY2 U/mkP16144vAojI/VFP0P0OL7qENwVo1B2ERMf6Pal5jLjUpPMVamy9vsUQQ6AmnVj35 iGrQ== X-Gm-Message-State: AO0yUKWUu2NH8VvDNE32Ml1cJJxUs73uONPphJ+H+hGtoK3p8sKFhCy+ dLXWXruJvYA7Bugs1qnjRZzLArdPd6jV X-Google-Smtp-Source: AK7set+8rZs7v/xCMonTIpZIgut8OUh9JbDVoCmWOxbweb0/DRK5Qdc6F3UTroIqg/uAt5W2yJMbGe9Qzs4G X-Received: from vipin.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:479f]) (user=vipinsh job=sendgmr) by 2002:a0d:d813:0:b0:52e:ec03:9b2f with SMTP id a19-20020a0dd813000000b0052eec039b2fmr0ywe.8.1676080002679; Fri, 10 Feb 2023 17:46:42 -0800 (PST) Date: Fri, 10 Feb 2023 17:46:25 -0800 In-Reply-To: <20230211014626.3659152-1-vipinsh@google.com> Mime-Version: 1.0 References: <20230211014626.3659152-1-vipinsh@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230211014626.3659152-7-vipinsh@google.com> Subject: [Patch v3 6/7] KVM: x86/mmu: Remove handle_changed_spte_dirty_log() From: Vipin Sharma To: seanjc@google.com, pbonzini@redhat.com, bgardon@google.com, dmatlack@google.com Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Vipin Sharma Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Remove handle_changed_spte_dirty_log() as there is no code flow which sets 4KiB SPTE writable and hit this path. This function marks the page dirty in a memslot only if new SPTE is 4KiB in size and writable. Current users of handle_changed_spte_dirty_log() are: 1. set_spte_gfn() - Create only non writable SPTEs. 2. write_protect_gfn() - Change an SPTE to non writable. 3. zap leaf and roots APIs - Everything is 0. 4. handle_removed_pt() - Sets SPTEs to REMOVED_SPTE 5. tdp_mmu_link_sp() - Makes non leaf SPTEs. There is also no path which creates a writable 4KiB without going through make_spte() and this functions takes care of marking SPTE dirty in the memslot if it is PT_WRITABLE. Signed-off-by: Vipin Sharma Reviewed-by: David Matlack --- arch/x86/kvm/mmu/tdp_mmu.c | 22 ---------------------- 1 file changed, 22 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index e50e869bf879..0c031319e901 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -345,24 +345,6 @@ static void handle_changed_spte_acc_track(u64 old_spte, u64 new_spte, int level) kvm_set_pfn_accessed(spte_to_pfn(old_spte)); } -static void handle_changed_spte_dirty_log(struct kvm *kvm, int as_id, gfn_t gfn, - u64 old_spte, u64 new_spte, int level) -{ - bool pfn_changed; - struct kvm_memory_slot *slot; - - if (level > PG_LEVEL_4K) - return; - - pfn_changed = spte_to_pfn(old_spte) != spte_to_pfn(new_spte); - - if ((!is_writable_pte(old_spte) || pfn_changed) && - is_writable_pte(new_spte)) { - slot = __gfn_to_memslot(__kvm_memslots(kvm, as_id), gfn); - mark_page_dirty_in_slot(kvm, slot, gfn); - } -} - static void tdp_account_mmu_page(struct kvm *kvm, struct kvm_mmu_page *sp) { kvm_account_pgtable_pages((void *)sp->spt, +1); @@ -614,8 +596,6 @@ static void handle_changed_spte(struct kvm *kvm, int as_id, gfn_t gfn, __handle_changed_spte(kvm, as_id, gfn, old_spte, new_spte, level, shared); handle_changed_spte_acc_track(old_spte, new_spte, level); - handle_changed_spte_dirty_log(kvm, as_id, gfn, old_spte, - new_spte, level); } /* @@ -727,8 +707,6 @@ static u64 tdp_mmu_set_spte(struct kvm *kvm, int as_id, tdp_ptep_t sptep, __handle_changed_spte(kvm, as_id, gfn, old_spte, new_spte, level, false); handle_changed_spte_acc_track(old_spte, new_spte, level); - handle_changed_spte_dirty_log(kvm, as_id, gfn, old_spte, new_spte, - level); return old_spte; } From patchwork Sat Feb 11 01:46:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vipin Sharma X-Patchwork-Id: 13136698 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2FBA0C636D7 for ; Sat, 11 Feb 2023 01:47:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229643AbjBKBrG (ORCPT ); Fri, 10 Feb 2023 20:47:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58626 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229999AbjBKBqx (ORCPT ); Fri, 10 Feb 2023 20:46:53 -0500 Received: from mail-pl1-x649.google.com (mail-pl1-x649.google.com [IPv6:2607:f8b0:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3A2AC84F58 for ; Fri, 10 Feb 2023 17:46:45 -0800 (PST) Received: by mail-pl1-x649.google.com with SMTP id t3-20020a170902e1c300b0019a84b88f70so150098pla.19 for ; Fri, 10 Feb 2023 17:46:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=LQavnvPA7sTuc9JB1orqdV5Xojq0c/KlFgV3VwSeK0k=; b=icmDPkGL44J2kqQwV1RYnsD2fVzMq1tDWbJdeuolq0tIfvmvcyAgR3mvlsuZ0H6PSp XFxgo7b+BhE+x73Qz71vIt8UnQba+Y1xvdSJUh2T8z3+iRarDp2n6ZlFkO13JS8UC+QX tmh9QJdHEh5YTxZVvQbw1/eV24cBMlusEhDV+xi7JyIXvLZGPr9gYWsDUDxZw/b2ELGH EwPotRyQi+Tc7RwkasEKUeF3Jul+fL+otCjmhdZXsZs880/yCSmGAsMhcdwXxxlTsG6V M2da1299K16+NoVxLp0adPTY0g0lYM2etXmhW+xSz8GeSF/ZSxqSFRWkWIa7qJnJ5bS3 dRxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=LQavnvPA7sTuc9JB1orqdV5Xojq0c/KlFgV3VwSeK0k=; b=pOxAjMMuVLUcDroJdtumieM8SCmvLtTKY9907kPKUso++5NIPTV+V3ourkch+TKhiN lvcGS87TovfgFXo+FmzKHZolnzMHgNnPs1/cphWupWeeAfDNFEKn9dzIgTEfNpWZWytj DM6HtLDQoKx77qet7MNnLox47dgmnzF9k1exUEcKzHThC4d9yjMQOkopqo8eZQhji5ps mh3ldSBAJ51NbKfMQzYOm+xFzh75OTiKzKhoZ+1fZC1CWEUfQnxD6WucG6yHdduV6WNg 2IvWE4fkMMr8KFbTGf2bxFHsPq+/mvfgX7ymJN+aZDtInoKhHxMyNeUIfOSCcV6QXzYh yiVA== X-Gm-Message-State: AO0yUKWCkbA1dqzy3ht0fpy/hQnJeaIG3/flxVY+3IkFfxkGREJTDTh7 BHzJ4D9v7HeEemuGn4K541LMsRmwDvUG X-Google-Smtp-Source: AK7set/eG1ZlNuQSoJZEd5WvqmwTqycjbs629sGD/uywCliUkSpNTY+oyNqP12ndtyXlzxB00C9kiPCFkQjl X-Received: from vipin.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:479f]) (user=vipinsh job=sendgmr) by 2002:a05:6a00:dd:b0:5a8:189d:b53f with SMTP id e29-20020a056a0000dd00b005a8189db53fmr2515330pfj.6.1676080004721; Fri, 10 Feb 2023 17:46:44 -0800 (PST) Date: Fri, 10 Feb 2023 17:46:26 -0800 In-Reply-To: <20230211014626.3659152-1-vipinsh@google.com> Mime-Version: 1.0 References: <20230211014626.3659152-1-vipinsh@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230211014626.3659152-8-vipinsh@google.com> Subject: [Patch v3 7/7] KVM: x86/mmu: Merge all handle_changed_pte* functions. From: Vipin Sharma To: seanjc@google.com, pbonzini@redhat.com, bgardon@google.com, dmatlack@google.com Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Vipin Sharma Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org __handle_changed_pte() and handle_changed_spte_acc_track() are always used together. Merge these two functions and name the new function handle_changed_pte(). Remove the existing handle_changed_pte() function which just calls __handle_changed_pte and handle_changed_spte_acc_track(). This converges SPTEs change handling code to a single place. Signed-off-by: Vipin Sharma Reviewed-by: Ben Gardon Reviewed-by: David Matlack --- arch/x86/kvm/mmu/tdp_mmu.c | 42 +++++++++++--------------------------- 1 file changed, 12 insertions(+), 30 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 0c031319e901..67538ec48ce5 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -334,17 +334,6 @@ static void handle_changed_spte(struct kvm *kvm, int as_id, gfn_t gfn, u64 old_spte, u64 new_spte, int level, bool shared); -static void handle_changed_spte_acc_track(u64 old_spte, u64 new_spte, int level) -{ - if (!is_shadow_present_pte(old_spte) || !is_last_spte(old_spte, level)) - return; - - if (is_accessed_spte(old_spte) && - (!is_shadow_present_pte(new_spte) || !is_accessed_spte(new_spte) || - spte_to_pfn(old_spte) != spte_to_pfn(new_spte))) - kvm_set_pfn_accessed(spte_to_pfn(old_spte)); -} - static void tdp_account_mmu_page(struct kvm *kvm, struct kvm_mmu_page *sp) { kvm_account_pgtable_pages((void *)sp->spt, +1); @@ -487,7 +476,7 @@ static void handle_removed_pt(struct kvm *kvm, tdp_ptep_t pt, bool shared) } /** - * __handle_changed_spte - handle bookkeeping associated with an SPTE change + * handle_changed_spte - handle bookkeeping associated with an SPTE change * @kvm: kvm instance * @as_id: the address space of the paging structure the SPTE was a part of * @gfn: the base GFN that was mapped by the SPTE @@ -501,9 +490,9 @@ static void handle_removed_pt(struct kvm *kvm, tdp_ptep_t pt, bool shared) * Handle bookkeeping that might result from the modification of a SPTE. * This function must be called for all TDP SPTE modifications. */ -static void __handle_changed_spte(struct kvm *kvm, int as_id, gfn_t gfn, - u64 old_spte, u64 new_spte, int level, - bool shared) +static void handle_changed_spte(struct kvm *kvm, int as_id, gfn_t gfn, + u64 old_spte, u64 new_spte, int level, + bool shared) { bool was_present = is_shadow_present_pte(old_spte); bool is_present = is_shadow_present_pte(new_spte); @@ -587,15 +576,10 @@ static void __handle_changed_spte(struct kvm *kvm, int as_id, gfn_t gfn, if (was_present && !was_leaf && (is_leaf || !is_present || WARN_ON_ONCE(pfn_changed))) handle_removed_pt(kvm, spte_to_child_pt(old_spte, level), shared); -} -static void handle_changed_spte(struct kvm *kvm, int as_id, gfn_t gfn, - u64 old_spte, u64 new_spte, int level, - bool shared) -{ - __handle_changed_spte(kvm, as_id, gfn, old_spte, new_spte, level, - shared); - handle_changed_spte_acc_track(old_spte, new_spte, level); + if (was_leaf && is_accessed_spte(old_spte) && + (!is_present || !is_accessed_spte(new_spte) || pfn_changed)) + kvm_set_pfn_accessed(spte_to_pfn(old_spte)); } /* @@ -638,9 +622,8 @@ static inline int tdp_mmu_set_spte_atomic(struct kvm *kvm, if (!try_cmpxchg64(sptep, &iter->old_spte, new_spte)) return -EBUSY; - __handle_changed_spte(kvm, iter->as_id, iter->gfn, iter->old_spte, - new_spte, iter->level, true); - handle_changed_spte_acc_track(iter->old_spte, new_spte, iter->level); + handle_changed_spte(kvm, iter->as_id, iter->gfn, iter->old_spte, + new_spte, iter->level, true); return 0; } @@ -705,8 +688,7 @@ static u64 tdp_mmu_set_spte(struct kvm *kvm, int as_id, tdp_ptep_t sptep, old_spte = kvm_tdp_mmu_write_spte(sptep, old_spte, new_spte, level); - __handle_changed_spte(kvm, as_id, gfn, old_spte, new_spte, level, false); - handle_changed_spte_acc_track(old_spte, new_spte, level); + handle_changed_spte(kvm, as_id, gfn, old_spte, new_spte, level, false); return old_spte; } @@ -1276,7 +1258,7 @@ static bool set_spte_gfn(struct kvm *kvm, struct tdp_iter *iter, * Note, when changing a read-only SPTE, it's not strictly necessary to * zero the SPTE before setting the new PFN, but doing so preserves the * invariant that the PFN of a present * leaf SPTE can never change. - * See __handle_changed_spte(). + * See handle_changed_spte(). */ tdp_mmu_iter_set_spte(kvm, iter, 0); @@ -1301,7 +1283,7 @@ bool kvm_tdp_mmu_set_spte_gfn(struct kvm *kvm, struct kvm_gfn_range *range) /* * No need to handle the remote TLB flush under RCU protection, the * target SPTE _must_ be a leaf SPTE, i.e. cannot result in freeing a - * shadow page. See the WARN on pfn_changed in __handle_changed_spte(). + * shadow page. See the WARN on pfn_changed in handle_changed_spte(). */ return kvm_tdp_mmu_handle_gfn(kvm, range, set_spte_gfn); }