From patchwork Tue Aug 3 04:46:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mingwei Zhang X-Patchwork-Id: 12415289 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 956B6C4338F for ; Tue, 3 Aug 2021 04:46:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7501060EE9 for ; Tue, 3 Aug 2021 04:46:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233734AbhHCEqm (ORCPT ); Tue, 3 Aug 2021 00:46:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57726 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229611AbhHCEqj (ORCPT ); Tue, 3 Aug 2021 00:46:39 -0400 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 83F33C061796 for ; Mon, 2 Aug 2021 21:46:28 -0700 (PDT) Received: by mail-pj1-x1049.google.com with SMTP id n4-20020a17090ac684b0290177656cfbc7so2272437pjt.7 for ; Mon, 02 Aug 2021 21:46:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=uvGN8o0svNEgNpDAlc6hd0uZYNG8aDJpGIKyIlckztA=; b=EstkdbR37/abuK+nZocc83eoDyxbAueC4xW7VBFBL+11H/s8A7Nj3uKtpONLq5G/80 3A7l15GjFqzxPajJGMmk70pYdv1YC1UE8P0m9zC/39TbfL61vU0RPUIOK5R5SwNsNabe IzkvkvPu9WCs2VagDvbLQNJpDjTpgnUs2FfQCESqF+GoSNgq/+wM/IYW7yNtaYy8lPBN ggEN1Q6XqsdfJPdBJGzFtVTl1EKWycwQ7oJI+LlHeeiJcCG6f5XS3KkRiTyk99EbYbWI MAP7GBEINtiboJmngr63BAc2Hj7CRxN2bqDyDY9RZRKzUvpYTR80oNAHdOu2Nb17IFWR e30w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=uvGN8o0svNEgNpDAlc6hd0uZYNG8aDJpGIKyIlckztA=; b=TSgVOGrTIKflXDskea82vfE6PEfAya8wepR5LycDrX2PAAmqJ+6ZZ5wgTxvitmmvQj 2YOw5rUxZknwjBF7pvyMUAZoYQPFKBzQeSEovr033AtuJ5Lk1/X72ZjCdL1a8GE1W+Ka cJxULAYyEIPsXHNP9rSia4VIAFxiCdvgaKWiUArBvIIFVLTZ3LpfBQite1uLIhLRXV/V 4PXMYIS2Vr8/BIUe0N4mFJC5PBunr6Q7W4YD8xaRUxLR0rCvdE0qc99E1m3U+KXlrMDq ol3UjpWhRAZ/GetEzct7N7/ULKVwxvf9HuTnCw5F3mqatZmNKDwDxSRQeZNFUTyr40U7 1OFQ== X-Gm-Message-State: AOAM533jTPlVblmJ87smZl6OCQFiD2zD2a1vNM+ESzQYnBrxyZKJtpLS nHqi/hURDRATXwV46hQhOxPI4aMLeaa/ X-Google-Smtp-Source: ABdhPJyaw9WJ7iFBQaTLk0jwa6AM5CsSnxVzPD4B9jsp6ckJsVDFg8I3HQCXj6lroq4+P5FfD6XYc6lJFXoZ X-Received: from mihenry-linux-desktop.kir.corp.google.com ([2620:15c:29:204:4304:2e3e:d2f5:48c8]) (user=mizhang job=sendgmr) by 2002:a17:902:e88f:b029:12c:c949:36dd with SMTP id w15-20020a170902e88fb029012cc94936ddmr1292433plg.84.1627965987962; Mon, 02 Aug 2021 21:46:27 -0700 (PDT) Reply-To: Mingwei Zhang Date: Mon, 2 Aug 2021 21:46:05 -0700 In-Reply-To: <20210803044607.599629-1-mizhang@google.com> Message-Id: <20210803044607.599629-2-mizhang@google.com> Mime-Version: 1.0 References: <20210803044607.599629-1-mizhang@google.com> X-Mailer: git-send-email 2.32.0.554.ge1b32706d8-goog Subject: [PATCH v4 1/3] KVM: x86/mmu: Remove redundant spte present check in mmu_set_spte From: Mingwei Zhang To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Ben Gardon , David Matlack , Mingwei Zhang , Jing Zhang Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Drop an unnecessary is_shadow_present_pte() check when updating the rmaps after installing a non-MMIO SPTE. set_spte() is used only to create shadow-present SPTEs, e.g. MMIO SPTEs are handled early on, mmu_set_spte() runs with mmu_lock held for write, i.e. the SPTE can't be zapped between writing the SPTE and updating the rmaps. Opportunistically combine the "new SPTE" logic for large pages and rmaps. No functional change intended. Suggested-by: Ben Gardon Reviewed-by: David Matlack Reviewed-by: Ben Gardon Reviewed-by: Sean Christopherson Signed-off-by: Mingwei Zhang --- arch/x86/kvm/mmu/mmu.c | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index a8cdfd8d45c4..f614e9df3c3b 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -2734,17 +2734,13 @@ static int mmu_set_spte(struct kvm_vcpu *vcpu, u64 *sptep, pgprintk("%s: setting spte %llx\n", __func__, *sptep); trace_kvm_mmu_set_spte(level, gfn, sptep); - if (!was_rmapped && is_large_pte(*sptep)) - ++vcpu->kvm->stat.lpages; - if (is_shadow_present_pte(*sptep)) { - if (!was_rmapped) { - rmap_count = rmap_add(vcpu, sptep, gfn); - if (rmap_count > vcpu->kvm->stat.max_mmu_rmap_size) - vcpu->kvm->stat.max_mmu_rmap_size = rmap_count; - if (rmap_count > RMAP_RECYCLE_THRESHOLD) - rmap_recycle(vcpu, sptep, gfn); - } + if (!was_rmapped) { + if (is_large_pte(*sptep)) + ++vcpu->kvm->stat.lpages; + rmap_count = rmap_add(vcpu, sptep, gfn); + if (rmap_count > RMAP_RECYCLE_THRESHOLD) + rmap_recycle(vcpu, sptep, gfn); } return ret; From patchwork Tue Aug 3 04:46:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mingwei Zhang X-Patchwork-Id: 12415293 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E2E7DC4320A for ; Tue, 3 Aug 2021 04:46:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C65FD6052B for ; Tue, 3 Aug 2021 04:46:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233853AbhHCEqq (ORCPT ); Tue, 3 Aug 2021 00:46:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57736 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233725AbhHCEql (ORCPT ); Tue, 3 Aug 2021 00:46:41 -0400 Received: from mail-pl1-x64a.google.com (mail-pl1-x64a.google.com [IPv6:2607:f8b0:4864:20::64a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E1A92C061799 for ; Mon, 2 Aug 2021 21:46:29 -0700 (PDT) Received: by mail-pl1-x64a.google.com with SMTP id l16-20020a170902f690b029012cb82f15afso4476891plg.10 for ; Mon, 02 Aug 2021 21:46:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=/kt3GsReI0qxlgk6gWKn2NJ23HEdkW7Aud8Gxnwk7qw=; b=Xh81q1RmonSbkzq73aaMKLEqC/x2Cco72xRZ8gNWiIBwOPcwOaB5/GGrvb9rz47gBS XXg2t0f2JCAihWnYPpJvZoalPTYvi4DJ6RopV8A0fsfodMASneaUYJ4FU83DeTRGCEEI +ePoGauvyPKOeP+/kewT9XOp+C4TiJ/sx3qD202pbHcE0BkpoWjwjHlPkhEirp09x6EV ySIzlpWuN239XjN2P7ke1ftWkVWAiqPeiTPZGGgEg/fUMjjl5S+hVikebIXw2ZchHdGO vCJ3/k8Z7iTWrT6YGaSsGH6pJkJWNhgsORa+N1ljG3poKuIoGshhO0P/TLwzMZsapNVO aBow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=/kt3GsReI0qxlgk6gWKn2NJ23HEdkW7Aud8Gxnwk7qw=; b=sqoQqmaHsgiwyM6TAlaUvtTVALMU4na+FIUNhhjf7Ue8qpAD/4keCPCZYDHqzFkArU F7uMtSnAkTWBh3Q727nCP0SDgmcDvre9+guwgQrJT9n+SN0F3gQ5LcsXwrVgDwyr9cRZ gtlyJdJ6wcdBY8Yaxohs3dW5EmSxaE4EzNYM/K5vIkfVAsVuCkJF5xhiZThVkhoNMDaP lfRB6yhDW+qIozdRdwCQc+jQCmH6vSnUXD3FR+OHRtAEajK/fcrz9VY3WBuodxkhBKw5 cll/d3C35rFjcLN3/pVyXPP0ewV8hjCpIMIpDbaoic2zzGtVnHDQWXohgilddPjo6wDt jwfA== X-Gm-Message-State: AOAM531E2towg+6kbBfMXspJwjfQ7mtL+Fce71XEMLjXSqtWW1dPq6US idGWV1i5F/ZGwo9naaIb0I7iRmvT+Het X-Google-Smtp-Source: ABdhPJxps/Vl+UCc7rA/SBKbGz0jwmpKAc00VAX8dRqRa4N0RWHQf3ZnxU+ioFSRa6KEOCh6it2TAaN1BkTM X-Received: from mihenry-linux-desktop.kir.corp.google.com ([2620:15c:29:204:4304:2e3e:d2f5:48c8]) (user=mizhang job=sendgmr) by 2002:a17:902:a5c5:b029:12c:9108:d83 with SMTP id t5-20020a170902a5c5b029012c91080d83mr17010060plq.33.1627965989421; Mon, 02 Aug 2021 21:46:29 -0700 (PDT) Reply-To: Mingwei Zhang Date: Mon, 2 Aug 2021 21:46:06 -0700 In-Reply-To: <20210803044607.599629-1-mizhang@google.com> Message-Id: <20210803044607.599629-3-mizhang@google.com> Mime-Version: 1.0 References: <20210803044607.599629-1-mizhang@google.com> X-Mailer: git-send-email 2.32.0.554.ge1b32706d8-goog Subject: [PATCH v4 2/3] KVM: x86/mmu: Avoid collision with !PRESENT SPTEs in TDP MMU lpage stats From: Mingwei Zhang To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Ben Gardon , David Matlack , Mingwei Zhang , Jing Zhang Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Sean Christopherson Factor in whether or not the old/new SPTEs are shadow-present when adjusting the large page stats in the TDP MMU. A modified MMIO SPTE can toggle the page size bit, as bit 7 is used to store the MMIO generation, i.e. is_large_pte() can get a false positive when called on a MMIO SPTE. Ditto for nuking SPTEs with REMOVED_SPTE, which sets bit 7 in its magic value. Opportunistically move the logic below the check to verify at least one of the old/new SPTEs is shadow present. Use is/was_leaf even though is/was_present would suffice. The code generation is roughly equivalent since all flags need to be computed prior to the code in question, and using the *_leaf flags will minimize the diff in a future enhancement to account all pages, i.e. will change the check to "is_leaf != was_leaf". Reviewed-by: David Matlack Reviewed-by: Ben Gardon Fixes: 1699f65c8b65 ("kvm/x86: Fix 'lpages' kvm stat for TDM MMU") Cc: stable@vger.kernel.org Signed-off-by: Sean Christopherson Signed-off-by: Mingwei Zhang --- arch/x86/kvm/mmu/tdp_mmu.c | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 43f12f5d12c0..4b0953fed12e 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -413,6 +413,7 @@ static void __handle_changed_spte(struct kvm *kvm, int as_id, gfn_t gfn, bool was_leaf = was_present && is_last_spte(old_spte, level); bool is_leaf = is_present && is_last_spte(new_spte, level); bool pfn_changed = spte_to_pfn(old_spte) != spte_to_pfn(new_spte); + bool was_large, is_large; WARN_ON(level > PT64_ROOT_MAX_LEVEL); WARN_ON(level < PG_LEVEL_4K); @@ -446,13 +447,6 @@ static void __handle_changed_spte(struct kvm *kvm, int as_id, gfn_t gfn, trace_kvm_tdp_mmu_spte_changed(as_id, gfn, level, old_spte, new_spte); - if (is_large_pte(old_spte) != is_large_pte(new_spte)) { - if (is_large_pte(old_spte)) - atomic64_sub(1, (atomic64_t*)&kvm->stat.lpages); - else - atomic64_add(1, (atomic64_t*)&kvm->stat.lpages); - } - /* * The only times a SPTE should be changed from a non-present to * non-present state is when an MMIO entry is installed/modified/ @@ -478,6 +472,18 @@ static void __handle_changed_spte(struct kvm *kvm, int as_id, gfn_t gfn, return; } + /* + * Update large page stats if a large page is being zapped, created, or + * is replacing an existing shadow page. + */ + was_large = was_leaf && is_large_pte(old_spte); + is_large = is_leaf && is_large_pte(new_spte); + if (was_large != is_large) { + if (was_large) + atomic64_sub(1, (atomic64_t *)&kvm->stat.lpages); + else + atomic64_add(1, (atomic64_t *)&kvm->stat.lpages); + } if (was_leaf && is_dirty_spte(old_spte) && (!is_present || !is_dirty_spte(new_spte) || pfn_changed)) From patchwork Tue Aug 3 04:46:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mingwei Zhang X-Patchwork-Id: 12415295 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EF63DC432BE for ; Tue, 3 Aug 2021 04:46:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DCEAA60EE9 for ; Tue, 3 Aug 2021 04:46:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233709AbhHCEqs (ORCPT ); Tue, 3 Aug 2021 00:46:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57740 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233737AbhHCEqn (ORCPT ); Tue, 3 Aug 2021 00:46:43 -0400 Received: from mail-pl1-x649.google.com (mail-pl1-x649.google.com [IPv6:2607:f8b0:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D19FFC06175F for ; Mon, 2 Aug 2021 21:46:31 -0700 (PDT) Received: by mail-pl1-x649.google.com with SMTP id l16-20020a170902f690b029012cb82f15afso4476954plg.10 for ; Mon, 02 Aug 2021 21:46:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=5UjpISDzrwGZMuhZ3ho5okX6MWp1EpQ+2CzL/sQfoi8=; b=KAelLFMnY3ptM3Vh51kl5Kr8w9a7XsV9tRFIdwinbxUio60eYEp5hINBI3by/dElW1 Q9e7uScEWVbgKie9JhELJW0UZzob3a4ZeaLEx2H3CNA9tWMPdi+UokLIG7M7LdGrvLeB KY6d6zg/zb/TgOu6slVeR4P/21AtA1y5mUzlipkKjfYy+JuCrmtrVpy57WsnJc3UPQ76 CeYkHbXOPu7u/7ulAF9mepbe+xvXZC/UmaegcjtiFa4LM6KjpnTjm56vD/qH0+jVAZgM yExJfe9SpCeCstE+VyjVPt4l2CnUqFdyhFqN8UPCkrSl7460UmwC02y2IOMpdW2V+qUy cebw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=5UjpISDzrwGZMuhZ3ho5okX6MWp1EpQ+2CzL/sQfoi8=; b=MIE1wruzGTbPBgsyzFnk+KvFbiJqMCbQczIU2IxkWDO0QNpr4/X7quOnqgkUV99slM kRsxm3H5h4sBxFuPtl9pHrxqUiD0RstjV/gxp9xLqOCSqx5We1q+yMyAOLeV2VGEaTm5 jENXmURiZASJ5DItzEaCLMp2MkTcRu1Q4IRi6TowsoePf2HGiKRvoEr3Awkm2I3jrzFb BIxqh5WN+Z3D3mJ/W2/SRgBeDPBP0dtJjhRwblUqTfb64SBhDz5Xfs824np+iJ0JKftM n17TRb5igjrTOLtYQs+xzUhETMyslgkCWCVgytAbzgzLxsSbuBPO1KbKYpLIbHizHAPy XrEg== X-Gm-Message-State: AOAM5338c+OuVGw7+8MEGStlP1J5m08DJ8z6xn1KiHlxnMG7yeb6V/ts CFVq7AKabE58xF4HzWFy/vUQPTZ9ssc4 X-Google-Smtp-Source: ABdhPJy6wPdqBkcDiUx8rIlq1qz8BE0dfJT1TMDKJF8oJTGoGFqpFh3fZMBXDya6ruyTJNb2jexEaMbuGool X-Received: from mihenry-linux-desktop.kir.corp.google.com ([2620:15c:29:204:4304:2e3e:d2f5:48c8]) (user=mizhang job=sendgmr) by 2002:a05:6a00:a88:b029:31a:c2ef:d347 with SMTP id b8-20020a056a000a88b029031ac2efd347mr20773323pfl.20.1627965991322; Mon, 02 Aug 2021 21:46:31 -0700 (PDT) Reply-To: Mingwei Zhang Date: Mon, 2 Aug 2021 21:46:07 -0700 In-Reply-To: <20210803044607.599629-1-mizhang@google.com> Message-Id: <20210803044607.599629-4-mizhang@google.com> Mime-Version: 1.0 References: <20210803044607.599629-1-mizhang@google.com> X-Mailer: git-send-email 2.32.0.554.ge1b32706d8-goog Subject: [PATCH v4 3/3] KVM: x86/mmu: Add detailed page size stats From: Mingwei Zhang To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Ben Gardon , David Matlack , Mingwei Zhang , Jing Zhang Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Existing KVM code tracks the number of large pages regardless of their sizes. Therefore, when large page of 1GB (or larger) is adopted, the information becomes less useful because lpages counts a mix of 1G and 2M pages. So remove the lpages since it is easy for user space to aggregate the info. Instead, provide a comprehensive page stats of all sizes from 4K to 512G. Suggested-by: Ben Gardon Reviewed-by: David Matlack Reviewed-by: Ben Gardon Signed-off-by: Mingwei Zhang Cc: Jing Zhang Cc: David Matlack Cc: Sean Christopherson --- arch/x86/include/asm/kvm_host.h | 10 ++++++++- arch/x86/kvm/mmu.h | 4 ++++ arch/x86/kvm/mmu/mmu.c | 38 ++++++++++++++++----------------- arch/x86/kvm/mmu/tdp_mmu.c | 15 ++----------- arch/x86/kvm/x86.c | 5 ++++- 5 files changed, 38 insertions(+), 34 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index 99f37781a6fc..c95bf4bbd2ff 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1206,7 +1206,15 @@ struct kvm_vm_stat { u64 mmu_recycled; u64 mmu_cache_miss; u64 mmu_unsync; - u64 lpages; + union { + struct { + atomic64_t pages_4k; + atomic64_t pages_2m; + atomic64_t pages_1g; + atomic64_t pages_512g; + }; + atomic64_t pages[4]; + }; u64 nx_lpage_splits; u64 max_mmu_page_hash_collisions; u64 max_mmu_rmap_size; diff --git a/arch/x86/kvm/mmu.h b/arch/x86/kvm/mmu.h index 83e6c6965f1e..2883789fb5fb 100644 --- a/arch/x86/kvm/mmu.h +++ b/arch/x86/kvm/mmu.h @@ -240,4 +240,8 @@ static inline bool kvm_memslots_have_rmaps(struct kvm *kvm) return smp_load_acquire(&kvm->arch.memslots_have_rmaps); } +static inline void kvm_update_page_stats(struct kvm *kvm, int level, int count) +{ + atomic64_add(count, &kvm->stat.pages[level - 1]); +} #endif diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index f614e9df3c3b..8f46c1164f3f 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -599,10 +599,11 @@ static bool mmu_spte_update(u64 *sptep, u64 new_spte) * state bits, it is used to clear the last level sptep. * Returns the old PTE. */ -static u64 mmu_spte_clear_track_bits(u64 *sptep) +static int mmu_spte_clear_track_bits(struct kvm *kvm, u64 *sptep) { kvm_pfn_t pfn; u64 old_spte = *sptep; + int level = sptep_to_sp(sptep)->role.level; if (!spte_has_volatile_bits(old_spte)) __update_clear_spte_fast(sptep, 0ull); @@ -612,6 +613,8 @@ static u64 mmu_spte_clear_track_bits(u64 *sptep) if (!is_shadow_present_pte(old_spte)) return old_spte; + kvm_update_page_stats(kvm, level, -1); + pfn = spte_to_pfn(old_spte); /* @@ -996,15 +999,16 @@ static void __pte_list_remove(u64 *spte, struct kvm_rmap_head *rmap_head) } } -static void pte_list_remove(struct kvm_rmap_head *rmap_head, u64 *sptep) +static void pte_list_remove(struct kvm *kvm, struct kvm_rmap_head *rmap_head, + u64 *sptep) { - mmu_spte_clear_track_bits(sptep); + mmu_spte_clear_track_bits(kvm, sptep); __pte_list_remove(sptep, rmap_head); } /* Return true if rmap existed and callback called, false otherwise */ -static bool pte_list_destroy(struct kvm_rmap_head *rmap_head, - void (*callback)(u64 *sptep)) +static bool pte_list_destroy(struct kvm *kvm, struct kvm_rmap_head *rmap_head, + void (*callback)(struct kvm *kvm, u64 *sptep)) { struct pte_list_desc *desc, *next; int i; @@ -1014,7 +1018,7 @@ static bool pte_list_destroy(struct kvm_rmap_head *rmap_head, if (!(rmap_head->val & 1)) { if (callback) - callback((u64 *)rmap_head->val); + callback(kvm, (u64 *)rmap_head->val); goto out; } @@ -1023,7 +1027,7 @@ static bool pte_list_destroy(struct kvm_rmap_head *rmap_head, while (desc) { if (callback) for (i = 0; i < desc->spte_count; i++) - callback(desc->sptes[i]); + callback(kvm, desc->sptes[i]); next = desc->more; mmu_free_pte_list_desc(desc); desc = next; @@ -1163,7 +1167,7 @@ static u64 *rmap_get_next(struct rmap_iterator *iter) static void drop_spte(struct kvm *kvm, u64 *sptep) { - u64 old_spte = mmu_spte_clear_track_bits(sptep); + u64 old_spte = mmu_spte_clear_track_bits(kvm, sptep); if (is_shadow_present_pte(old_spte)) rmap_remove(kvm, sptep); @@ -1175,7 +1179,6 @@ static bool __drop_large_spte(struct kvm *kvm, u64 *sptep) if (is_large_pte(*sptep)) { WARN_ON(sptep_to_sp(sptep)->role.level == PG_LEVEL_4K); drop_spte(kvm, sptep); - --kvm->stat.lpages; return true; } @@ -1422,15 +1425,15 @@ static bool rmap_write_protect(struct kvm_vcpu *vcpu, u64 gfn) return kvm_mmu_slot_gfn_write_protect(vcpu->kvm, slot, gfn, PG_LEVEL_4K); } -static void mmu_spte_clear_track_bits_cb(u64 *sptep) +static void mmu_spte_clear_track_bits_cb(struct kvm *kvm, u64 *sptep) { - mmu_spte_clear_track_bits(sptep); + mmu_spte_clear_track_bits(kvm, sptep); } static bool kvm_zap_rmapp(struct kvm *kvm, struct kvm_rmap_head *rmap_head, const struct kvm_memory_slot *slot) { - return pte_list_destroy(rmap_head, mmu_spte_clear_track_bits_cb); + return pte_list_destroy(kvm, rmap_head, mmu_spte_clear_track_bits_cb); } static bool kvm_unmap_rmapp(struct kvm *kvm, struct kvm_rmap_head *rmap_head, @@ -1461,13 +1464,13 @@ static bool kvm_set_pte_rmapp(struct kvm *kvm, struct kvm_rmap_head *rmap_head, need_flush = 1; if (pte_write(pte)) { - pte_list_remove(rmap_head, sptep); + pte_list_remove(kvm, rmap_head, sptep); goto restart; } else { new_spte = kvm_mmu_changed_pte_notifier_make_spte( *sptep, new_pfn); - mmu_spte_clear_track_bits(sptep); + mmu_spte_clear_track_bits(kvm, sptep); mmu_spte_set(sptep, new_spte); } } @@ -2276,8 +2279,6 @@ static int mmu_page_zap_pte(struct kvm *kvm, struct kvm_mmu_page *sp, if (is_shadow_present_pte(pte)) { if (is_last_spte(pte, sp->role.level)) { drop_spte(kvm, spte); - if (is_large_pte(pte)) - --kvm->stat.lpages; } else { child = to_shadow_page(pte & PT64_BASE_ADDR_MASK); drop_parent_pte(child, spte); @@ -2736,8 +2737,7 @@ static int mmu_set_spte(struct kvm_vcpu *vcpu, u64 *sptep, trace_kvm_mmu_set_spte(level, gfn, sptep); if (!was_rmapped) { - if (is_large_pte(*sptep)) - ++vcpu->kvm->stat.lpages; + kvm_update_page_stats(vcpu->kvm, level, 1); rmap_count = rmap_add(vcpu, sptep, gfn); if (rmap_count > RMAP_RECYCLE_THRESHOLD) rmap_recycle(vcpu, sptep, gfn); @@ -5740,7 +5740,7 @@ static bool kvm_mmu_zap_collapsible_spte(struct kvm *kvm, if (sp->role.direct && !kvm_is_reserved_pfn(pfn) && sp->role.level < kvm_mmu_max_mapping_level(kvm, slot, sp->gfn, pfn, PG_LEVEL_NUM)) { - pte_list_remove(rmap_head, sptep); + pte_list_remove(kvm, rmap_head, sptep); if (kvm_available_flush_tlb_with_range()) kvm_flush_remote_tlbs_with_address(kvm, sp->gfn, diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 4b0953fed12e..45a5c1f43433 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -413,7 +413,6 @@ static void __handle_changed_spte(struct kvm *kvm, int as_id, gfn_t gfn, bool was_leaf = was_present && is_last_spte(old_spte, level); bool is_leaf = is_present && is_last_spte(new_spte, level); bool pfn_changed = spte_to_pfn(old_spte) != spte_to_pfn(new_spte); - bool was_large, is_large; WARN_ON(level > PT64_ROOT_MAX_LEVEL); WARN_ON(level < PG_LEVEL_4K); @@ -472,18 +471,8 @@ static void __handle_changed_spte(struct kvm *kvm, int as_id, gfn_t gfn, return; } - /* - * Update large page stats if a large page is being zapped, created, or - * is replacing an existing shadow page. - */ - was_large = was_leaf && is_large_pte(old_spte); - is_large = is_leaf && is_large_pte(new_spte); - if (was_large != is_large) { - if (was_large) - atomic64_sub(1, (atomic64_t *)&kvm->stat.lpages); - else - atomic64_add(1, (atomic64_t *)&kvm->stat.lpages); - } + if (is_leaf != was_leaf) + kvm_update_page_stats(kvm, level, is_leaf ? 1 : -1); if (was_leaf && is_dirty_spte(old_spte) && (!is_present || !is_dirty_spte(new_spte) || pfn_changed)) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 916c976e99ab..a0a1d70981a8 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -233,7 +233,10 @@ const struct _kvm_stats_desc kvm_vm_stats_desc[] = { STATS_DESC_COUNTER(VM, mmu_recycled), STATS_DESC_COUNTER(VM, mmu_cache_miss), STATS_DESC_ICOUNTER(VM, mmu_unsync), - STATS_DESC_ICOUNTER(VM, lpages), + STATS_DESC_ICOUNTER(VM, pages_4k), + STATS_DESC_ICOUNTER(VM, pages_2m), + STATS_DESC_ICOUNTER(VM, pages_1g), + STATS_DESC_ICOUNTER(VM, pages_512g), STATS_DESC_ICOUNTER(VM, nx_lpage_splits), STATS_DESC_PCOUNTER(VM, max_mmu_rmap_size), STATS_DESC_PCOUNTER(VM, max_mmu_page_hash_collisions)