From patchwork Fri Aug 5 23:05:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12937889 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4F1AAC00140 for ; Fri, 5 Aug 2022 23:05:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241642AbiHEXFl (ORCPT ); Fri, 5 Aug 2022 19:05:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59954 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241573AbiHEXFd (ORCPT ); Fri, 5 Aug 2022 19:05:33 -0400 Received: from mail-pl1-x64a.google.com (mail-pl1-x64a.google.com [IPv6:2607:f8b0:4864:20::64a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D7C44753B5 for ; Fri, 5 Aug 2022 16:05:31 -0700 (PDT) Received: by mail-pl1-x64a.google.com with SMTP id a15-20020a170902eccf00b0016f92ee2d54so1705100plh.15 for ; Fri, 05 Aug 2022 16:05:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:reply-to:from:to:cc; bh=7iv2nkYs2Vt5ti4O0QvNxO0OSZhrVlUXfvDgfst3my8=; b=Ju2r70VXaQaBCfA4/9Ehm4gB0L5NiBP1hooL4HnyKWIcEYGbQOZWdNUOPU1XmaSkcq /PuKMks4HOOCC3C9X/jiH9r3zNN7Qjeful+CPThn0pd4iiaAkCPXOMzi2tXWo+WmWCgK E2Kk25DioybC9HTR+89W1FGK/A1wQ2tA9hCs3vH3sQztLf0cJm6AWKmkSreEBBN9YdBn TBzhe98kIwtY05XH3SzepbCXIXeU6qRbH7b3M+QLMp4rZEhh1g5Ye4LmLb5vOBsGo3OU kBAVT8NcLgSipzYSEN90in0Uh3DftRjtewjeunalmlUnV0s3KbWIKEpPB4bjvIask8aT PuXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc; bh=7iv2nkYs2Vt5ti4O0QvNxO0OSZhrVlUXfvDgfst3my8=; b=N2wP4JHoxOylWTn8JwjBUyY8I3M1fN9QptwXdnoHwbrb3qlh/zudTTNb3KfdaH9GSM JwtvPrs8yNDlsxxpIyfAWw2cFdkW23zO0laA4n8dTA0Xq/jIe/ZjJy5cZIb7rvIfv3GD qNzsYbJAFDIc8LBrML+L18/a+/SK42El2zcJVLVE39xAvjHjDI9xqIi78vH8onOCNBKh 3QhUTqBdUDmYdepXo/chvDGscnkomoqld8KNCBGYclzuoPER0mYlSPG4h12zuYH0DL+A RaoC4P02R8LT6asHd42kXrCB+hS56Z3a1PaEShnSpOO8J6WzJs3fAK7kZyNkSM3CtxRR Zoeg== X-Gm-Message-State: ACgBeo0J4LCjSt0ZLf7PP1wlhokzDEOpuqr7oKivzGlFoMOKGU+cqf25 nXR1s+4HT3l4iM0c6d94jQHbTCZXvpc= X-Google-Smtp-Source: AA6agR6Wuy0rugKFS69l72MhpsgCDGkbaHDDzKGZo1mdiNyxouwP60a7Vc0b0twmCliZeFK+daVqExOnEYE= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a17:90b:3807:b0:1f4:ecf7:5987 with SMTP id mq7-20020a17090b380700b001f4ecf75987mr9545835pjb.13.1659740730404; Fri, 05 Aug 2022 16:05:30 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 5 Aug 2022 23:05:11 +0000 In-Reply-To: <20220805230513.148869-1-seanjc@google.com> Message-Id: <20220805230513.148869-7-seanjc@google.com> Mime-Version: 1.0 References: <20220805230513.148869-1-seanjc@google.com> X-Mailer: git-send-email 2.37.1.559.g78731f0fdb-goog Subject: [PATCH v3 6/8] KVM: x86/mmu: Track the number of TDP MMU pages, but not the actual pages From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, David Matlack , Yan Zhao , Mingwei Zhang , Ben Gardon Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Track the number of TDP MMU "shadow" pages instead of tracking the pages themselves. With the NX huge page list manipulation moved out of the common linking flow, elminating the list-based tracking means the happy path of adding a shadow page doesn't need to acquire a spinlock and can instead inc/dec an atomic. Keep the tracking as the WARN during TDP MMU teardown on leaked shadow pages is very, very useful for detecting KVM bugs. Tracking the number of pages will also make it trivial to expose the counter to userspace as a stat in the future, which may or may not be desirable. Note, the TDP MMU needs to use a separate counter (and stat if that ever comes to be) from the existing n_used_mmu_pages. The TDP MMU doesn't bother supporting the shrinker nor does it honor KVM_SET_NR_MMU_PAGES (because the TDP MMU consumes so few pages relative to shadow paging), and including TDP MMU pages in that counter would break both the shrinker and shadow MMUs, e.g. if a VM is using nested TDP. Cc: Yan Zhao Reviewed-by: Mingwei Zhang Reviewed-by: David Matlack Signed-off-by: Sean Christopherson --- arch/x86/include/asm/kvm_host.h | 11 +++-------- arch/x86/kvm/mmu/tdp_mmu.c | 28 +++++++++++++--------------- 2 files changed, 16 insertions(+), 23 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index 5634347e5d05..7ac0c5612319 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1271,6 +1271,9 @@ struct kvm_arch { */ bool tdp_mmu_enabled; + /* The number of TDP MMU pages across all roots. */ + atomic64_t tdp_mmu_pages; + /* * List of struct kvm_mmu_pages being used as roots. * All struct kvm_mmu_pages in the list should have @@ -1291,18 +1294,10 @@ struct kvm_arch { */ struct list_head tdp_mmu_roots; - /* - * List of struct kvmp_mmu_pages not being used as roots. - * All struct kvm_mmu_pages in the list should have - * tdp_mmu_page set and a tdp_mmu_root_count of 0. - */ - struct list_head tdp_mmu_pages; - /* * Protects accesses to the following fields when the MMU lock * is held in read mode: * - tdp_mmu_roots (above) - * - tdp_mmu_pages (above) * - the link field of struct kvm_mmu_pages used by the TDP MMU * - possible_nx_huge_pages; * - the possible_nx_huge_page_link field of struct kvm_mmu_pages used diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 34994ca3d45b..526d38704e5c 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -29,7 +29,6 @@ int kvm_mmu_init_tdp_mmu(struct kvm *kvm) kvm->arch.tdp_mmu_enabled = true; INIT_LIST_HEAD(&kvm->arch.tdp_mmu_roots); spin_lock_init(&kvm->arch.tdp_mmu_pages_lock); - INIT_LIST_HEAD(&kvm->arch.tdp_mmu_pages); kvm->arch.tdp_mmu_zap_wq = wq; return 1; } @@ -54,7 +53,7 @@ void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm) /* Also waits for any queued work items. */ destroy_workqueue(kvm->arch.tdp_mmu_zap_wq); - WARN_ON(!list_empty(&kvm->arch.tdp_mmu_pages)); + WARN_ON(atomic64_read(&kvm->arch.tdp_mmu_pages)); WARN_ON(!list_empty(&kvm->arch.tdp_mmu_roots)); /* @@ -386,12 +385,7 @@ static void handle_changed_spte_dirty_log(struct kvm *kvm, int as_id, gfn_t gfn, static void tdp_mmu_unlink_sp(struct kvm *kvm, struct kvm_mmu_page *sp, bool shared) { - if (shared) - spin_lock(&kvm->arch.tdp_mmu_pages_lock); - else - lockdep_assert_held_write(&kvm->mmu_lock); - - list_del(&sp->link); + atomic64_dec(&kvm->arch.tdp_mmu_pages); /* * Ensure nx_huge_page_disallowed is read after observing the present @@ -401,10 +395,16 @@ static void tdp_mmu_unlink_sp(struct kvm *kvm, struct kvm_mmu_page *sp, */ smp_rmb(); - if (sp->nx_huge_page_disallowed) { - sp->nx_huge_page_disallowed = false; - untrack_possible_nx_huge_page(kvm, sp); - } + if (!sp->nx_huge_page_disallowed) + return; + + if (shared) + spin_lock(&kvm->arch.tdp_mmu_pages_lock); + else + lockdep_assert_held_write(&kvm->mmu_lock); + + sp->nx_huge_page_disallowed = false; + untrack_possible_nx_huge_page(kvm, sp); if (shared) spin_unlock(&kvm->arch.tdp_mmu_pages_lock); @@ -1137,9 +1137,7 @@ static int tdp_mmu_link_sp(struct kvm *kvm, struct tdp_iter *iter, tdp_mmu_set_spte(kvm, iter, spte); } - spin_lock(&kvm->arch.tdp_mmu_pages_lock); - list_add(&sp->link, &kvm->arch.tdp_mmu_pages); - spin_unlock(&kvm->arch.tdp_mmu_pages_lock); + atomic64_inc(&kvm->arch.tdp_mmu_pages); return 0; }