From patchwork Thu Sep 26 01:34:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13812676 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7820BCCFA12 for ; Thu, 26 Sep 2024 01:35:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 996DA6B00AD; Wed, 25 Sep 2024 21:35:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 944B66B00AE; Wed, 25 Sep 2024 21:35:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7E4D86B00B4; Wed, 25 Sep 2024 21:35:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 61DE86B00AD for ; Wed, 25 Sep 2024 21:35:19 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id E6EF81A1211 for ; Thu, 26 Sep 2024 01:35:18 +0000 (UTC) X-FDA: 82605171516.13.F28C275 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf09.hostedemail.com (Postfix) with ESMTP id 2C83B14000A for ; Thu, 26 Sep 2024 01:35:17 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=bvlj7Ksl; spf=pass (imf09.hostedemail.com: domain of 3VLr0ZgoKCNkEOCJPBCOJIBJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--jthoughton.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3VLr0ZgoKCNkEOCJPBCOJIBJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1727314357; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xCPXKlvg3fvHTzjtwRf5eteIHvTFuMld4Zvbq5X/Dxs=; b=jVaxCZZ2Bmd/s3SH+bF/GNa8orQ5SAo8E4BLs0DPY4KodD1kMr2Z8Jd4VhmGz+Pd9eOKgX cwGK9SlroZY76MlYgVjpvYg47Bk116BHMakAUe0YapsY+WBisIgI1Z22MhYjK4Ont+l/vi 4LnToq/LvOF0Sot7CvXGJcKAak7uOac= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=bvlj7Ksl; spf=pass (imf09.hostedemail.com: domain of 3VLr0ZgoKCNkEOCJPBCOJIBJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--jthoughton.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3VLr0ZgoKCNkEOCJPBCOJIBJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1727314357; a=rsa-sha256; cv=none; b=t5MOQROMYdB/BqxgLeayAk1Djdta63ioSBoc/t+m17+kMkCWfWY3UMZyk+jziEMHUnKy+V 8bRQ5mFopRr6V0B788Uczbl08py1gH6o1lmzjHjC9cwaCGYoLddfYkUcOqzYCCbmXykCBL jEeqyrTMmP/9wxmOYbZEeJDxfQXzqls= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-e25cfee6581so84922276.2 for ; Wed, 25 Sep 2024 18:35:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1727314516; x=1727919316; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=xCPXKlvg3fvHTzjtwRf5eteIHvTFuMld4Zvbq5X/Dxs=; b=bvlj7Ksl6cxdh56fj9d6mO4ikFd5vSmjoS+UhwuW5l0iUAziBjiRU1X8Sw6SshFocr RnrD5ykx0pgv+POaKdBpV4lN9XNDhKAA+G6G+0lY/G5jBTmeyjjrYwXyonCtAauJRkXB 8/zqJ0z/hUTl1QY7ojCeVJWH6BIC3hTlujMYB70gc938iz6b1DSosUePQQQ1Bstq1qi9 lTyzDF8s+g/6zAZYcqP5BNPjjtVenHv8o1oxgg3+mLxQEvvaQD7NlLY2abZapgwEyMVp YrPdVlIEliFWnJW6VzeB9N7y7xFZG3xkiHB3Xfv1L5e4P+fv87IRH6pf6E3hKrWwrkbh JBwA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727314516; x=1727919316; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=xCPXKlvg3fvHTzjtwRf5eteIHvTFuMld4Zvbq5X/Dxs=; b=KjefD42k9gC+OI1WfXgjimWs7h34LmobxfowVBG/3tC4uwWlb47D6VuZJ2phymuAh1 WeLZAYa3g8ov+g3fncBc8tlndkvzGbrh8Pk4GhfvH2Pd61RgVrY+mOkwgSYNR5Ef1yXx b9nsrTwm6X0yiMDiKd2XvNA7Dc6OLrbzPkq3eqaOMzFt7elZdeygRH9Ewjs9yMZgC95u 86D76exVHp3gJrESoXWoE/9YXRrKCcYdImphwCG9IgXmNra/D1aK5DBk9/eO9PM7gax5 j2rhyRwEslQt6eEq/SLUc0llqFbuhZpclAsHgtVFgpuz/ayjgNHUdp5jiisDxsFxHo2j l3Fw== X-Forwarded-Encrypted: i=1; AJvYcCUSQ5WeltfPJyAWDJHrv17DFDuXURjkerGu8jqDq1sui2K6zKTTo88g1Z3RW9T9DLlehxWDC6shbw==@kvack.org X-Gm-Message-State: AOJu0YxWE/MzgyO0IdjIIr8S/4femVXgQd4NWA6AHDIifqyHbZ25hmKE 16SIxTJRa7cnLi0SNo3SlYlpdr2I1BDjUGiZG5P1viJuUuCgpKollfjizIPHtL3WpmquoGgt6xE LJDi8yJF409Q2gXDL8g== X-Google-Smtp-Source: AGHT+IHCtBDgdi+XBFMofkXIkzVJkl+teLSBao9sOlv55Q8B7jZN9hLVgp83Pu834/Q28A3MUQoEEh6YXmiO2wqz X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:13d:fb22:ac12:a84b]) (user=jthoughton job=sendgmr) by 2002:a25:97c7:0:b0:e25:c8fc:b78c with SMTP id 3f1490d57ef6-e25c8fcbb92mr1586276.9.1727314516206; Wed, 25 Sep 2024 18:35:16 -0700 (PDT) Date: Thu, 26 Sep 2024 01:34:49 +0000 In-Reply-To: <20240926013506.860253-1-jthoughton@google.com> Mime-Version: 1.0 References: <20240926013506.860253-1-jthoughton@google.com> X-Mailer: git-send-email 2.46.0.792.g87dc391469-goog Message-ID: <20240926013506.860253-2-jthoughton@google.com> Subject: [PATCH v7 01/18] KVM: Remove kvm_handle_hva_range helper functions From: James Houghton To: Sean Christopherson , Paolo Bonzini Cc: Andrew Morton , David Matlack , David Rientjes , James Houghton , Jason Gunthorpe , Jonathan Corbet , Marc Zyngier , Oliver Upton , Wei Xu , Yu Zhao , Axel Rasmussen , kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 2C83B14000A X-Stat-Signature: hcpkdp7deqdnauxxh6iyhfnjsdweu8x4 X-Rspam-User: X-HE-Tag: 1727314517-165223 X-HE-Meta: U2FsdGVkX1+L5vjuIaETZU7gtCoZX2/YiEldwGU4o+6Ka8I1361sENuD+b5T5HF/R6KULIlB+CAm2jY3XHyAN/1J1SJnRNR+vukNJPan2sgysB9nifFE0LLKpy/3tnTmNKZXlDYwOBQfGoY+yGFPU0234N6kQ/ygA38TTjqqaEeFLgK1wlPUff9H8SdalYVLJAVBaZD+FbIp/WSx5oyjXQDDEIC6C3ZXbk+eHAGKwSTMAySg+GDZVdxlWSaM/Jm+ZrK4+61zUG8OdtUyQULBwjSMKCTerRa97p7B1OlOULu+nrsoFF0G1SP8IGM6RC3U0I6VCj7M3dB/LYdHnavQh+7Z7kktkvvEeLI8wc6p6XB+Q53QXCXGJ/0bVgryz6mjY7fCwjeL3KvnuPUvVle0TPeGho6JXue3SCbSz5q3bLhevbrExmj8GlfSNzEVXE+ISo6ZpD+OVxUqRTNt0x+Nry0l6VTw5WCjcwV/pMiXj+VPxJJ39uQ30p5SVnyYjZq4VlhqM2EoJb91y0Ujev/g1Zo34xs5R66FMBvr5B4k0nnbbum/7LISEQrV12r1rjmzrUmEB65HB4+eJXJA1u9pXXhz+rwM+sgjJKlp4GpBx97h0NGYIY2Fjm5J0598/4upgVn3kILg5slpaEiDEGwjXVU1GPkJWTpZrb3hHGPyWzcD6lWs52AYhF6xYzd88lUBI5+MKIEenUz4sz97Q2nosGrxoqYqpdPyFgp336Zb7oE7PRJugI+m1DAQ2PIaTR61WOnFAgEoVSekWIVRPt7rK/iEHysgZwpeVCEUk0vRD/Vwy2UlsOqz1LrPMpkwFNAR1y1ow+9iridQw44eb2qh82bUyUwkudycAx7jDBof8jjKLRgAS20JCpiibkIsKLL0rFANTaf462PVSnVy2SU2Hg2OljY5iQwxkIwNqFry7VV7hxkIXBWdMio3fgxk/TxO6HaHdxgp3pFDPX+gTX6 tzeAI/m6 hiWjdP8E9vvlTIo8zh/iiLtqSkLc4Sks/I0RQig65qp3don9uF38Ea5/kb69QcXhPP1fiHyoBsX6iNT5AqMzkYTfQxIXuTFsPxKj+l3AC4OGOakidAHJkkalHTkZ16QlsFYPbF62wRQaKPJhiJqAGXKgv1s1h5z/fRg/Vzi3UEFR1C7+VbHP6Aa1mYX/uM1SBbuAhQZjAqdOzq9ghpU5wN9ZF0ULQ2UWxVgZmBkQVBpjhar5bXqmJJLr5cDfA843JTMO435CjbWV7n5sDxt4xT8svgCMoMCbbCy2GKGXPBH6kA91AcXuMjhGXJGyS1FigMxtSSrYYR4ObOQGRkI9ReE6teWUOptiT0pwLzXa5xyQb5AUpN826zWPVJm8srW4ReiaHrr0oUkV0tl1dsY6u32oNZ5irkHWZqbG5sniFJl22sPwdk6SC6CUQTfy5GuWQ3ZGJvVpT0N1jKCV7E9sQlJtgvoVgQwLXt91yDPavC374kKUkLhIWBKcWdTiOO10rrzG+piHyb5GREn/m8fqkWN+fn2NSAn2etVpHPmEmy/lDJTHgDO06TJmbk4uGR0GixqZJAf0YyLgS49d7pBdIv8NZhJtKvgf8EjAmimQ1s2WgRsg923y1+UxGmQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: kvm_handle_hva_range is only used by the young notifiers. In a later patch, it will be even further tied to the young notifiers. Instead of renaming kvm_handle_hva_range to something like kvm_handle_hva_range_young, simply remove kvm_handle_hva_range. This seems slightly more readable, though there is slightly more code duplication. Finally, rename __kvm_handle_hva_range to kvm_handle_hva_range, now that the name is available. Suggested-by: David Matlack Signed-off-by: James Houghton --- virt/kvm/kvm_main.c | 81 +++++++++++++++++++++------------------------ 1 file changed, 37 insertions(+), 44 deletions(-) diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index d51357fd28d7..090e79e4304f 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -589,8 +589,8 @@ static void kvm_null_fn(void) node; \ node = interval_tree_iter_next(node, start, last)) \ -static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm, - const struct kvm_mmu_notifier_range *range) +static __always_inline kvm_mn_ret_t kvm_handle_hva_range(struct kvm *kvm, + const struct kvm_mmu_notifier_range *range) { struct kvm_mmu_notifier_return r = { .ret = false, @@ -666,42 +666,6 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm, return r; } -static __always_inline int kvm_handle_hva_range(struct mmu_notifier *mn, - unsigned long start, - unsigned long end, - gfn_handler_t handler) -{ - struct kvm *kvm = mmu_notifier_to_kvm(mn); - const struct kvm_mmu_notifier_range range = { - .start = start, - .end = end, - .handler = handler, - .on_lock = (void *)kvm_null_fn, - .flush_on_ret = true, - .may_block = false, - }; - - return __kvm_handle_hva_range(kvm, &range).ret; -} - -static __always_inline int kvm_handle_hva_range_no_flush(struct mmu_notifier *mn, - unsigned long start, - unsigned long end, - gfn_handler_t handler) -{ - struct kvm *kvm = mmu_notifier_to_kvm(mn); - const struct kvm_mmu_notifier_range range = { - .start = start, - .end = end, - .handler = handler, - .on_lock = (void *)kvm_null_fn, - .flush_on_ret = false, - .may_block = false, - }; - - return __kvm_handle_hva_range(kvm, &range).ret; -} - void kvm_mmu_invalidate_begin(struct kvm *kvm) { lockdep_assert_held_write(&kvm->mmu_lock); @@ -794,7 +758,7 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn, * that guest memory has been reclaimed. This needs to be done *after* * dropping mmu_lock, as x86's reclaim path is slooooow. */ - if (__kvm_handle_hva_range(kvm, &hva_range).found_memslot) + if (kvm_handle_hva_range(kvm, &hva_range).found_memslot) kvm_arch_guest_memory_reclaimed(kvm); return 0; @@ -840,7 +804,7 @@ static void kvm_mmu_notifier_invalidate_range_end(struct mmu_notifier *mn, }; bool wake; - __kvm_handle_hva_range(kvm, &hva_range); + kvm_handle_hva_range(kvm, &hva_range); /* Pairs with the increment in range_start(). */ spin_lock(&kvm->mn_invalidate_lock); @@ -862,9 +826,19 @@ static int kvm_mmu_notifier_clear_flush_young(struct mmu_notifier *mn, unsigned long start, unsigned long end) { + struct kvm *kvm = mmu_notifier_to_kvm(mn); + const struct kvm_mmu_notifier_range range = { + .start = start, + .end = end, + .handler = kvm_age_gfn, + .on_lock = (void *)kvm_null_fn, + .flush_on_ret = true, + .may_block = false, + }; + trace_kvm_age_hva(start, end); - return kvm_handle_hva_range(mn, start, end, kvm_age_gfn); + return kvm_handle_hva_range(kvm, &range).ret; } static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn, @@ -872,6 +846,16 @@ static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn, unsigned long start, unsigned long end) { + struct kvm *kvm = mmu_notifier_to_kvm(mn); + const struct kvm_mmu_notifier_range range = { + .start = start, + .end = end, + .handler = kvm_age_gfn, + .on_lock = (void *)kvm_null_fn, + .flush_on_ret = false, + .may_block = false, + }; + trace_kvm_age_hva(start, end); /* @@ -887,17 +871,26 @@ static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn, * cadence. If we find this inaccurate, we might come up with a * more sophisticated heuristic later. */ - return kvm_handle_hva_range_no_flush(mn, start, end, kvm_age_gfn); + return kvm_handle_hva_range(kvm, &range).ret; } static int kvm_mmu_notifier_test_young(struct mmu_notifier *mn, struct mm_struct *mm, unsigned long address) { + struct kvm *kvm = mmu_notifier_to_kvm(mn); + const struct kvm_mmu_notifier_range range = { + .start = address, + .end = address + 1, + .handler = kvm_test_age_gfn, + .on_lock = (void *)kvm_null_fn, + .flush_on_ret = false, + .may_block = false, + }; + trace_kvm_test_age_hva(address); - return kvm_handle_hva_range_no_flush(mn, address, address + 1, - kvm_test_age_gfn); + return kvm_handle_hva_range(kvm, &range).ret; } static void kvm_mmu_notifier_release(struct mmu_notifier *mn, From patchwork Thu Sep 26 01:34:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13812677 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 48F96CCFA10 for ; Thu, 26 Sep 2024 01:35:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 74C4A6B00AE; Wed, 25 Sep 2024 21:35:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6FB5F6B00B4; Wed, 25 Sep 2024 21:35:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 574986B00B5; Wed, 25 Sep 2024 21:35:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 373166B00AE for ; Wed, 25 Sep 2024 21:35:20 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id DB5188123D for ; Thu, 26 Sep 2024 01:35:19 +0000 (UTC) X-FDA: 82605171558.11.2D10521 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf14.hostedemail.com (Postfix) with ESMTP id 1428810000F for ; Thu, 26 Sep 2024 01:35:17 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ZaDTARha; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf14.hostedemail.com: domain of 3Vbr0ZgoKCNoFPDKQCDPKJCKKCHA.8KIHEJQT-IIGR68G.KNC@flex--jthoughton.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3Vbr0ZgoKCNoFPDKQCDPKJCKKCHA.8KIHEJQT-IIGR68G.KNC@flex--jthoughton.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1727314420; a=rsa-sha256; cv=none; b=SAFzrMS0sfT5NAOnVdv4zM5+zRk8rHkLHd3hxTrR54UpluDKcDUeow0snLf2Jtcz+OFQTN 8nSRDHOwgZFOsCAVwBn2jTs6HvvxMQyBZ7EEgMT/dfI+qfidyuAwYsNjZ0Rxfj7jHcDO+D QVNXIlD8wZ/RchKIvBD2IoMJ4SbN2ro= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ZaDTARha; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf14.hostedemail.com: domain of 3Vbr0ZgoKCNoFPDKQCDPKJCKKCHA.8KIHEJQT-IIGR68G.KNC@flex--jthoughton.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3Vbr0ZgoKCNoFPDKQCDPKJCKKCHA.8KIHEJQT-IIGR68G.KNC@flex--jthoughton.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1727314420; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nrPGFdAf3DB4vj7nij04fMkTLi5+MW1J/YyJNP+NgJQ=; b=W1ia44GI9QcQRZRe9pjmS/gSg7CKPYEkO9nuL0mgTYMzMEAHa7WqvWYbELsKvY8g8b3G8w IzNklfOrp7erSIqXYFMcTQhgaNgFHM0L7zt4Cxv1kmgoGzBL1ylojJC52VoB9198uLcHib VLyQMqK7odUwMs1ff3zO+qZETSIY78M= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-e25cae769abso816232276.0 for ; Wed, 25 Sep 2024 18:35:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1727314517; x=1727919317; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=nrPGFdAf3DB4vj7nij04fMkTLi5+MW1J/YyJNP+NgJQ=; b=ZaDTARhaWJUiJxLTJW4Di3DOGr9TDlMXL12ZxCzIq+l899I6V6SaImQ0//lxSjkbAU JDSKwSh+2odQ32LPSGHQNw8aOnOYxcHon01ek8PZFTZLfuRAbaWwIah1YOk5dS1vA1HF AphYty54RPYcacvO6L+FH33MaGdTY6H7m5yIosc0ifL1MrDHa1PxIll8BfsjmY9AlgTy lvOq/ZcRxREOUzRHK2mZMPVpCy5s9D6WLZz/jn2fmzoNA5jD+FixwVoIWRDTwM1Qlfr4 zfhQFQJnN4X/yvnLjScH5RfbqwUyB/vrumXoyZ80N2LcSIeGpejGV6nuEBkwaL+pPmEv ++EQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727314517; x=1727919317; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=nrPGFdAf3DB4vj7nij04fMkTLi5+MW1J/YyJNP+NgJQ=; b=QTOjeA5pj1HzPod3HspG0NF40ohzWpTL8CdkOSf8j0iEV5GtKysrH6cqH8Z5rJ7/Pc bdUf85T8lziP8eDsbMqsM0mJzXMjkQFcZzr5q/XGqzMcQEGagaS3uxa5v/XuuxqFbv8S /tXxKTKXRPzYN9qpGeOvUO7GgUjtiH0OlP+0Etehx6wsBNDacO2WZeACcJcnI6GsHzDy j0GG07/ruN15lycFWCzm203zzJgF7UOmrNAAHF2ZWbjz0N0ghslPX7II9ks+4qgp0umh 3jB5u9DRomURifbo+sY55N/2OzLcCpg31O8MV80SBCsPmW2UEk3Z6jHfC590Dn+0Ww5k GClQ== X-Forwarded-Encrypted: i=1; AJvYcCU6VetQ46ZtXmsdeJH2Bo9ztmK2JqQZnrHzcO3JEGUftT2c8mufO/CbCv2cSbkpuwOwaSQtdoAIiQ==@kvack.org X-Gm-Message-State: AOJu0YxKkXSpD2WbiYuftilqhofYW7o9vNFhRKq53tYT0e4DJTuXm7DW 0MdiiJvGCcT/fqEsD5Hxdtb7RvzJPogrRA2ALo5dZQbgM4kVbHubq+fTLWORpfwo9SHl6l/hUb6 cYmZ0ok13F5bFe1qbSg== X-Google-Smtp-Source: AGHT+IHBkpgMYqE9bFhZXKzj+UxHtQzg97Wx4pqBE3O5Wvje89pjGl0kpLMaEu3zaYYGi2UruxZxZDhUO7JaGYM6 X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:13d:fb22:ac12:a84b]) (user=jthoughton job=sendgmr) by 2002:a25:3546:0:b0:e1b:10df:78ea with SMTP id 3f1490d57ef6-e24d7df1a54mr3612276.4.1727314517106; Wed, 25 Sep 2024 18:35:17 -0700 (PDT) Date: Thu, 26 Sep 2024 01:34:50 +0000 In-Reply-To: <20240926013506.860253-1-jthoughton@google.com> Mime-Version: 1.0 References: <20240926013506.860253-1-jthoughton@google.com> X-Mailer: git-send-email 2.46.0.792.g87dc391469-goog Message-ID: <20240926013506.860253-3-jthoughton@google.com> Subject: [PATCH v7 02/18] KVM: Add lockless memslot walk to KVM From: James Houghton To: Sean Christopherson , Paolo Bonzini Cc: Andrew Morton , David Matlack , David Rientjes , James Houghton , Jason Gunthorpe , Jonathan Corbet , Marc Zyngier , Oliver Upton , Wei Xu , Yu Zhao , Axel Rasmussen , kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 1428810000F X-Stat-Signature: jkjmpyz13rjmsnnajeugsfbujxtm9d9z X-Rspam-User: X-HE-Tag: 1727314517-347596 X-HE-Meta: U2FsdGVkX19m+7rFO8rAudDJ3N9wG4VzlnYKJ7f2pN8ZmAL0tyDgBrWzT4NDrbT2C6k9/wEF/MKOuq4TnT0AJCd0ICsBoG+H6LaEhbZNMKNwhBB9PE+rovklzaL/Acrvwm9zEuLJjnHo1BlngO7matOxf2GnGBGv2OcYgWUgGrY8aXpS8SsTzwyzLroFJH3aHGpXEBrXPkRLazWf8JHujM2vJigIObBdW5doqFj4oM6D50YXnXfMo4y0egPJtJPrHTpf4SEbnbrCQ/CIitE7GZ3zPiHUVaHRtRp1WbZ32aRHqeX6hqq+LESI/WSWeoinSrV/ItB4qi4QuZCEsoddECnd/Yue3AqjqvTOKLadoiU9Ocj9tnuogBF+O9PsYNqiYstON50IisnZz73cP/9xw0LWOsFa1y5qFHQo1fibRf1bll6J6yynke0eEujBm3z2tKwZqYfbGLJ1RcdUje8UTgjjUUuW8QSpsT2qlzGb94BgvY0pO0UNI9V4/U6my/zOkrpTKWfA12RVqA42CUoVv3USCN7Geb2rEoUi4cIklaABRII5ngFAQujo9CwNfksCRyPE/XL6UUAVJm2csoGf9XXRF4qZYn5Y7nWkt1uqIe6bYBjrA5wxF9n2o3xdtrRCyAbnDksziyZOBsYuhrHt0+HR/ULN41iNwDlR2yZtU2OabzZZYRs3E2ROFQxEtuqA027U5KCnyvwmkWwacQXK8RtPxTAn8SkuGJw1BBDKByTcvgVmzwBSd9GsQn9bPN0am8cYlYox1j9sozFI3DJ76ahIekoq2QroZ0BmWGI8mKJPaYDdovS2fH7gtIFjSRx092zu+4OZ4cUBm1jMv6kn4Zk2yD6NKemVwEAxcJzy0tvYBnTnxT9ElDhgHAEF92dvh4Q/6ZFysQK3ewEYRR7slsi9xLhGfzp0JHs9bogU6XkPh+IfuZwp6j/yyWtM1WdOeXPhohwLZ+DVF0gbJSs /9D2eiKa XmSu4b3kPBwBIHx5iYLvxMkPLeRv/d0o7WJHuPwimInMQCgy1lL5yVWlP/S3U4I7qS3BFpIJTL6igubUSAFHljAjYH8RGqhlszYJTUpvt5WCaHCjx1WnZT+CneM8DEhRKs8YH2DAhQPtJfGadwW1ag+n0ngdFHqmYHFPLs7f3WDvHNEG/Sl1Ji4jNmuVevmdnSpxFtx7P5Wxqi5+2b5XSh8i0IIkCTKs8R0Pd0F11E3dOraKFJOthEt5raAAHK4Nu133p+gtOVNLfroO4GqLa5oikFpzLEYBpbZU2g1VDe3LCvmKaCH0PqsMTDN3KNpveiR/BtwIfQxrtfERfqRZy6oMFQOPx5ha5LWJb447a4Yx++AJRlpaQ2g/8C4Qjay1rpr22cby3cCI/KbloivCLCuDwLTv3HFJgOh02JEFnL/p1u5de2LpZT3HSwiggnD4FkIR8UFNAlduWl+f+XIdlrhyFbYnBt8/Du8y3D0ygCzXYqlZdPo/yuwxkMG/vXpmQJ/e9AUFN7Jfily0IVdW9+R4jT5IITO4I1Ssr9M2umZ/OTALGGp9GKe27weiyJMVLkhKB X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Provide flexibility to the architecture to synchronize as optimally as they can instead of always taking the MMU lock for writing. Architectures that do their own locking must select CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS. The immediate application is to allow architectures to implement the test/clear_young MMU notifiers more cheaply. Suggested-by: Yu Zhao Signed-off-by: James Houghton --- include/linux/kvm_host.h | 1 + virt/kvm/Kconfig | 3 +++ virt/kvm/kvm_main.c | 28 +++++++++++++++++++++------- 3 files changed, 25 insertions(+), 7 deletions(-) diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index b23c6d48392f..98a987e88578 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -266,6 +266,7 @@ struct kvm_gfn_range { gfn_t end; union kvm_mmu_notifier_arg arg; bool may_block; + bool lockless; }; bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range); bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range); diff --git a/virt/kvm/Kconfig b/virt/kvm/Kconfig index fd6a3010afa8..58d896b2f4ed 100644 --- a/virt/kvm/Kconfig +++ b/virt/kvm/Kconfig @@ -100,6 +100,9 @@ config KVM_GENERIC_MMU_NOTIFIER select MMU_NOTIFIER bool +config KVM_MMU_NOTIFIER_YOUNG_LOCKLESS + bool + config KVM_GENERIC_MEMORY_ATTRIBUTES depends on KVM_GENERIC_MMU_NOTIFIER bool diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 090e79e4304f..7d5b35cfc1ed 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -555,6 +555,7 @@ struct kvm_mmu_notifier_range { on_lock_fn_t on_lock; bool flush_on_ret; bool may_block; + bool lockless; }; /* @@ -609,6 +610,10 @@ static __always_inline kvm_mn_ret_t kvm_handle_hva_range(struct kvm *kvm, IS_KVM_NULL_FN(range->handler))) return r; + /* on_lock will never be called for lockless walks */ + if (WARN_ON_ONCE(range->lockless && !IS_KVM_NULL_FN(range->on_lock))) + return r; + idx = srcu_read_lock(&kvm->srcu); for (i = 0; i < kvm_arch_nr_memslot_as_ids(kvm); i++) { @@ -640,15 +645,18 @@ static __always_inline kvm_mn_ret_t kvm_handle_hva_range(struct kvm *kvm, gfn_range.start = hva_to_gfn_memslot(hva_start, slot); gfn_range.end = hva_to_gfn_memslot(hva_end + PAGE_SIZE - 1, slot); gfn_range.slot = slot; + gfn_range.lockless = range->lockless; if (!r.found_memslot) { r.found_memslot = true; - KVM_MMU_LOCK(kvm); - if (!IS_KVM_NULL_FN(range->on_lock)) - range->on_lock(kvm); - - if (IS_KVM_NULL_FN(range->handler)) - goto mmu_unlock; + if (!range->lockless) { + KVM_MMU_LOCK(kvm); + if (!IS_KVM_NULL_FN(range->on_lock)) + range->on_lock(kvm); + + if (IS_KVM_NULL_FN(range->handler)) + goto mmu_unlock; + } } r.ret |= range->handler(kvm, &gfn_range); } @@ -658,7 +666,7 @@ static __always_inline kvm_mn_ret_t kvm_handle_hva_range(struct kvm *kvm, kvm_flush_remote_tlbs(kvm); mmu_unlock: - if (r.found_memslot) + if (r.found_memslot && !range->lockless) KVM_MMU_UNLOCK(kvm); srcu_read_unlock(&kvm->srcu, idx); @@ -834,6 +842,8 @@ static int kvm_mmu_notifier_clear_flush_young(struct mmu_notifier *mn, .on_lock = (void *)kvm_null_fn, .flush_on_ret = true, .may_block = false, + .lockless = + IS_ENABLED(CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS), }; trace_kvm_age_hva(start, end); @@ -854,6 +864,8 @@ static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn, .on_lock = (void *)kvm_null_fn, .flush_on_ret = false, .may_block = false, + .lockless = + IS_ENABLED(CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS), }; trace_kvm_age_hva(start, end); @@ -886,6 +898,8 @@ static int kvm_mmu_notifier_test_young(struct mmu_notifier *mn, .on_lock = (void *)kvm_null_fn, .flush_on_ret = false, .may_block = false, + .lockless = + IS_ENABLED(CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS), }; trace_kvm_test_age_hva(address); From patchwork Thu Sep 26 01:34:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13812678 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E2938CCF9EB for ; Thu, 26 Sep 2024 01:35:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 050806B0083; Wed, 25 Sep 2024 21:35:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EEBA06B00B6; Wed, 25 Sep 2024 21:35:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D67DA6B00B8; Wed, 25 Sep 2024 21:35:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id B29396B00B5 for ; Wed, 25 Sep 2024 21:35:21 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 2AA661411BB for ; Thu, 26 Sep 2024 01:35:21 +0000 (UTC) X-FDA: 82605171642.27.AB45ECC Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf13.hostedemail.com (Postfix) with ESMTP id 55F8E20003 for ; Thu, 26 Sep 2024 01:35:19 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=O+4W2xzc; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf13.hostedemail.com: domain of 3Vrr0ZgoKCNsGQELRDEQLKDLLDIB.9LJIFKRU-JJHS79H.LOD@flex--jthoughton.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3Vrr0ZgoKCNsGQELRDEQLKDLLDIB.9LJIFKRU-JJHS79H.LOD@flex--jthoughton.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1727314503; a=rsa-sha256; cv=none; b=x7E02KM8FTj1y4NW3g6wFQE3eqszTQ323WMTvFI6FeMqNKzOkUexMZRXg585OX6C3WZjcH 0UAouQ/MEbzTJV5F315U8AT7/D4kMho3TqI6zte1xVJy+yG0JgINswadsjy+FajLedeRWD eBStCGeAcekd9pMVh9N44glqYzz3NGM= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=O+4W2xzc; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf13.hostedemail.com: domain of 3Vrr0ZgoKCNsGQELRDEQLKDLLDIB.9LJIFKRU-JJHS79H.LOD@flex--jthoughton.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3Vrr0ZgoKCNsGQELRDEQLKDLLDIB.9LJIFKRU-JJHS79H.LOD@flex--jthoughton.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1727314503; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=CIRgMt/tPt8dDHjlNVeDudvk1PSxwGvmBpyKYz5f4us=; b=GJS7pmTjynfGT/5LY23Z7YhFIsKOK4Ylk05XXkd5pnvJJBfQLMoRrb10El7fAlduZx4wdf 0srjyX7sA+YsxFk7ijLzsQ6fMxX38KphkIq9HZIyCzHN3IFBpTtBbc0mbn8KEYfPTtsjv1 w0QmSkgfdojnW4wu96L6f6I456kZpNk= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-6d5235d1bcaso9766907b3.2 for ; Wed, 25 Sep 2024 18:35:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1727314518; x=1727919318; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=CIRgMt/tPt8dDHjlNVeDudvk1PSxwGvmBpyKYz5f4us=; b=O+4W2xzcXxIAJtkp9LBRIVdLqSfSbu2Q6B4OMddwcmWQ4eQcP6b692wXHOlKOT4cZD WHLIdwQAcyGYzDQLVflCsDeEBsfwDd+KRFu9MQ3mBkaMZIL5q3yVWzJKa1P33olGmbfN sllNAmYEmMGXfF3a92lWhr3rPPphkV2QBHK5wsqDB8mg4jQIQ73CUAyb6hbKq9BKoeST Jzu1taCVnK18xbf6Oxh3nNQV+dljDfxLmr28XoFr2E7GOcrVSKfVE2Cj3nvU35KmN6EF RUvaSVgx6A0fMCxqh4YuCtNAmVNfTSg76WbsL3CG9ChMuFlcQ0vxHwgTIa5jrXb0KZJX pT2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727314518; x=1727919318; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=CIRgMt/tPt8dDHjlNVeDudvk1PSxwGvmBpyKYz5f4us=; b=I7Fx/9rrBIftVSTBLef2VH9EwMWyicJZFSZ+KYHGyp6SiKuWhMwDDGHmmSr94DCSBr izsvzMwQ/3Y2CZMp9syqBm7qHyKsWq3/YgwP2gwqj0sn4npLT4kEbQ+6/cRq9S0bvoxr IUBRzKSqqMked8vikwuo+7xW++A3XhnWFeFw4fp2iMqPQnQz7pCbU6L/6aNorYvPbPpf EczbWsR27uUC0EoGqROqg2/COqhRdiJr977MEiASY2d3Kf7duYuxFXGQxUiVeZbUqBON 01MjgZS1ODTgygljqFfvtCltpH0APPsK5vFhFwfh4IpVOz4tN6Zvf84re4W4fpJSO1Uc VPZQ== X-Forwarded-Encrypted: i=1; AJvYcCVk6GjS/M5Pcuv4cds/v8Zu3iPxB2d0ldFTXKZFmUhHcvqeH1thFOWkGIQGLt0RRITi0sJDJ1HTyA==@kvack.org X-Gm-Message-State: AOJu0YwWZpoAV7VYBJyDlFzWHg9hM4o3GCZTeTSMjOQp/XvSPZVFDIvz z3p04NX9Jc9c6s/79W6yqCu8jHMs5VOsAQMtch5jz0GlC6c307Ni57N90L3rF2ZWqfbtq1T7jOz qE51mEAiAkJBD4G7liA== X-Google-Smtp-Source: AGHT+IHo9npkiN5cwqAiCiI3e58k0AahBEai2pKIEYXOCcGxS2Z6C3puLSeMWFdZt66z08Hill7horZ6pvVyPTNq X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:13d:fb22:ac12:a84b]) (user=jthoughton job=sendgmr) by 2002:a05:690c:4a08:b0:6e2:371f:4afe with SMTP id 00721157ae682-6e2371f4d7dmr1417b3.4.1727314518344; Wed, 25 Sep 2024 18:35:18 -0700 (PDT) Date: Thu, 26 Sep 2024 01:34:51 +0000 In-Reply-To: <20240926013506.860253-1-jthoughton@google.com> Mime-Version: 1.0 References: <20240926013506.860253-1-jthoughton@google.com> X-Mailer: git-send-email 2.46.0.792.g87dc391469-goog Message-ID: <20240926013506.860253-4-jthoughton@google.com> Subject: [PATCH v7 03/18] KVM: x86/mmu: Factor out spte atomic bit clearing routine From: James Houghton To: Sean Christopherson , Paolo Bonzini Cc: Andrew Morton , David Matlack , David Rientjes , James Houghton , Jason Gunthorpe , Jonathan Corbet , Marc Zyngier , Oliver Upton , Wei Xu , Yu Zhao , Axel Rasmussen , kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org X-Rspam-User: X-Stat-Signature: dw66adrjkt9we85i8ozka3rjbpt3zzez X-Rspamd-Queue-Id: 55F8E20003 X-Rspamd-Server: rspam02 X-HE-Tag: 1727314519-213099 X-HE-Meta: U2FsdGVkX1+z7LGH4zoaE2PgAcjLDvG12+0JcoaxLgjQyR6kG4WeU2PEVHmktNHbWc3ts7DgVSUqi7wP7tsNceSH9DV1b1x7/cSKddJHzWR2KDWDlmvXywb4l+HTxvrsbY28XBvo8ggPD5Yos1997UyxcrQ5r44AmL5lVzYmZagYK4Ff2MkKGSEsoo65prNi7QOwURLgqqycbAn5mH2TWQluMzzJG2PAaoFo3eCicplwZB/3dySQ7A6LUYWJGvZjS1pw/mT5KUDdmtWBq71X3hok0gDuBKCqA59WT/MU9hiqpcV3gZHPgs6L142hXUUOzJ2yrRoyPDkQSUiWxXJzLirOgbeOkJioWDnWJ0Jl+dGYvF9WpmEhYbRbxyqSEjjztQZmWgUxad4o5d4u8GfHzublgJMzH91CrHyauQTS1vZ1AV5UxE3S7HF8I/D0dpH0k1QIcmnQvIHV8f0ac+ghFoDEVVauDtQPJEJ5C2Koi24MvaWApYgNLCKJ1mhMlCtjo2yvRGsnMsTFTIJUsQnB/9WmqrbMWSdjUkvoG7aZn4mG9D3PSiBkjuhUKbyzR1XeRUGZJlOfiOrGPnUmbWM2scY91M3WQXR+fo0h+gnHb5648d7W93FWJtZdDI5rKm7dPrbwqezwhvoQGuTGHrpR6n5Wf0TjMuFrpYImf6gcLj6mAZRxkTWbwW1jRLKzYyv7gv4yOT+/xHusnu1CIWJ8WJo9OcY41smvXMxmfhQ33J2rMinei5TpCPSkftnFU2dFKLc2/08wYX7bthgjWb6QiRngF1+fofPBepzuwK3t7Q7q3ZiZmsoULtTfN+rBDkoea31LfkhzV57RwqDW00ceRnnkvdwv+rvYTL7XW2wju6zgmxonvZlBgrHFyAW1+OneRouzaFkmX83fWGLXoQIxQHi0bM8d1rWkLmXuOn74qpBvxaepH5SPaDUhGi5aHL32MNy4an1qnQsVCU5l3Fa l0llHbL4 k5mgdBv80RgOAYhN5d9H6UvPn1sfAmvemmS9XLFJPM1qmV1d6OIwXH9n7ziNXeWwIprrUyfCTg5JnhXNNEiUSj4jeamweuyFN56TQdtAjGGC8S4wKpwPwtHZRGHOK3Y9EuLPoMTWv/gzu+ZE0OMp6BJWZFZ9IPM1M3+6dwfcF7QOOYAnDDrXOGU/VlGERFoF+vdpjeSysYk5T4qM3OzVuLU2aq2I99UK9UfLASZA66O1ynOcqy1mO+l1COX1xnO/dEBJ4boeq+rpJJeu93hhFds2eLHnIa8EddE2JTK2LxP3g+2xUTyHa5hdvzrI6bOvPSN+1yaPwTNCpF5BnLrYESBxyTFPoSE5gWKawc1wzJ3zvooFO4KwLRpZ3ssUh2xFkaJuTOk7Yq19uo1I8v4DmjJvxsOYCCCeYy683VyahhejWg8O38kEpqbw6SIS57pSmDZM57vUSyNcTeIja6EL0sJlV67O5+BqLhQTm7qTo3bQbcYBaf0HBdeKY3NyUmF39sNYeKa1ngsurQm+4Q5CyASa+Ne6UV89PgSSuj09iIU7p7GnBpZzTOC1A81/qWFM/iHs7cHALT+Apk24= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000211, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This new function, tdp_mmu_clear_spte_bits_atomic(), will be used in a follow-up patch to enable lockless Accessed and R/W/X bit clearing. Signed-off-by: James Houghton --- arch/x86/kvm/mmu/tdp_iter.h | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_iter.h b/arch/x86/kvm/mmu/tdp_iter.h index 2880fd392e0c..ec171568487c 100644 --- a/arch/x86/kvm/mmu/tdp_iter.h +++ b/arch/x86/kvm/mmu/tdp_iter.h @@ -25,6 +25,13 @@ static inline u64 kvm_tdp_mmu_write_spte_atomic(tdp_ptep_t sptep, u64 new_spte) return xchg(rcu_dereference(sptep), new_spte); } +static inline u64 tdp_mmu_clear_spte_bits_atomic(tdp_ptep_t sptep, u64 mask) +{ + atomic64_t *sptep_atomic = (atomic64_t *)rcu_dereference(sptep); + + return (u64)atomic64_fetch_and(~mask, sptep_atomic); +} + static inline void __kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 new_spte) { KVM_MMU_WARN_ON(is_ept_ve_possible(new_spte)); @@ -65,10 +72,8 @@ static inline u64 tdp_mmu_clear_spte_bits(tdp_ptep_t sptep, u64 old_spte, { atomic64_t *sptep_atomic; - if (kvm_tdp_mmu_spte_need_atomic_write(old_spte, level)) { - sptep_atomic = (atomic64_t *)rcu_dereference(sptep); - return (u64)atomic64_fetch_and(~mask, sptep_atomic); - } + if (kvm_tdp_mmu_spte_need_atomic_write(old_spte, level)) + return tdp_mmu_clear_spte_bits_atomic(sptep, mask); __kvm_tdp_mmu_write_spte(sptep, old_spte & ~mask); return old_spte; From patchwork Thu Sep 26 01:34:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13812679 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E0792CCF9EB for ; Thu, 26 Sep 2024 01:35:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 23F8C6B0093; Wed, 25 Sep 2024 21:35:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 108906B00B5; Wed, 25 Sep 2024 21:35:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E4EF36B00B6; Wed, 25 Sep 2024 21:35:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id C32EB6B0093 for ; Wed, 25 Sep 2024 21:35:22 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 795AD141163 for ; Thu, 26 Sep 2024 01:35:22 +0000 (UTC) X-FDA: 82605171684.20.3E5FF04 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf25.hostedemail.com (Postfix) with ESMTP id BD78FA0005 for ; Thu, 26 Sep 2024 01:35:20 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=wKAyLZqf; spf=pass (imf25.hostedemail.com: domain of 3V7r0ZgoKCNwHRFMSEFRMLEMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--jthoughton.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3V7r0ZgoKCNwHRFMSEFRMLEMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1727314484; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=E9ikx7qvh9LLa0/zOZpb6K1KXC0ViYbLtiviLVHMlqE=; b=k9RUb9zMOSdabk/cBSQbEIFlF6hdn+kkiWySGDXxyVTwifk/aMYSZOo+M8qOJYG3V1PZbg 3MopLGXNmACb0kKL5+jnkno6+r5c5yzeU3aIBUshNEAAKdFqvAXyn9u3BGmelFIwTlTIs5 UPcLzuvQoUIzbPEQGC48b1VdhrCr4Vg= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=wKAyLZqf; spf=pass (imf25.hostedemail.com: domain of 3V7r0ZgoKCNwHRFMSEFRMLEMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--jthoughton.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3V7r0ZgoKCNwHRFMSEFRMLEMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1727314484; a=rsa-sha256; cv=none; b=jD5JnLVeKg61GW4PharQKe7k0ikqCaOCi6Ii+MTUJZEJOf2ju/BONxk8ZNOnlxq/q1hKf3 YEII6KTJlBNYgr7eQ85iqw2C6/nmjQs0Fie2w3lToZxD4X4E5b9IxCQUsjYAJMAAknzdmv sCcTQwO9Cgc7DR6OKBtY1Kl5x9cFDMQ= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-6dbbeee08f0so23490077b3.0 for ; Wed, 25 Sep 2024 18:35:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1727314520; x=1727919320; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=E9ikx7qvh9LLa0/zOZpb6K1KXC0ViYbLtiviLVHMlqE=; b=wKAyLZqfk7rJySIS4NzbnByeI6MPYk5MXEcR9DryaRq49eDduhVtu25v013IDgDua6 P8QuOyixCuYVHK0FVHQ9q4U1lDg+s3ST1x6H7x3+q9tGosXfFDt/ZJ2lfyr74tchK51a y6BwnUc8sCjNtMPwk7yBIcab4Y4JokK23nFjG/EWaVlPfMyrG/jX9iPauh5g/QtYEz+J SNY/IX4wVOjKDlqYWbX3nHxvexXUcV1P4qH4PJcBQ3wxpUMNqONgD6GQ9+2ow8TjM0+F Q7M+wK0RqV4L0sAH1g6pEJEi86l5/9Lu6yeQYwFIujBahbruOgANY3jNmDp6+kWikTxx sP+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727314520; x=1727919320; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=E9ikx7qvh9LLa0/zOZpb6K1KXC0ViYbLtiviLVHMlqE=; b=hUlQ0P1jHWapQuatOVPwGFJ5QpP4u0EcTIKSHf+n+vEbPY1QiwUASzHxvKBvGMTJQ3 c64zaJu3G5dmcBm1L8/xD4k9X6L/NW3DQs4oyiM4UtphXpkFoS3rjLk5q4jmoKlJ9IIn 1ZesRxg8NDGLWusX23/Y1ZYpHk326feUAOHzqVf5DKtNjejjFtRYosE8NeFjBazYQ9A7 lcu84YoWhrO6t7dP5vnG3xp/ERiLNicpLw66SNyHKoSjybVv/dXuVv8S0ugoWK4U+Chb KP7Rrta4sEoscFrh67KBNFI7wNi0q4Az0H9BC9h/Ix55bqciH+z3S0x4Uurw+48vD5SZ 4jpg== X-Forwarded-Encrypted: i=1; AJvYcCU20Q+9msBZLoKlNFGbBERIWpLI+5lnkRNhEDuoSP6lHEFpOcYFIB9U/LpKf3xY2mMRUhCyq+hgIg==@kvack.org X-Gm-Message-State: AOJu0YxnH3B4vBsuWcehP6/bzCzJPjCiSUqojCp89nxz8l1ycYo87Q0F 2yMeBaYFUQiOvH/i8eKKAjRIkJwfZ08H1PxM018uTqied4NyfLEquNIZwjI75JgqnUSG4hj/mzY K34S1Mm4EQUwHNwvW8A== X-Google-Smtp-Source: AGHT+IH0FiQjxyLo+JfGazQOCkHrPRgPRywE06mMVfat9iBHnlYATt5rsXjVLhFjPfx7L1BClqPZPXX/no7MzpGQ X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:13d:fb22:ac12:a84b]) (user=jthoughton job=sendgmr) by 2002:a05:690c:2b92:b0:6dd:bb6e:ec89 with SMTP id 00721157ae682-6e22efd31b8mr137077b3.2.1727314519545; Wed, 25 Sep 2024 18:35:19 -0700 (PDT) Date: Thu, 26 Sep 2024 01:34:52 +0000 In-Reply-To: <20240926013506.860253-1-jthoughton@google.com> Mime-Version: 1.0 References: <20240926013506.860253-1-jthoughton@google.com> X-Mailer: git-send-email 2.46.0.792.g87dc391469-goog Message-ID: <20240926013506.860253-5-jthoughton@google.com> Subject: [PATCH v7 04/18] KVM: x86/mmu: Relax locking for kvm_test_age_gfn and kvm_age_gfn From: James Houghton To: Sean Christopherson , Paolo Bonzini Cc: Andrew Morton , David Matlack , David Rientjes , James Houghton , Jason Gunthorpe , Jonathan Corbet , Marc Zyngier , Oliver Upton , Wei Xu , Yu Zhao , Axel Rasmussen , kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org X-Rspam-User: X-Stat-Signature: h4pueh3b3me6zrfzaebqufsxmynestwq X-Rspamd-Queue-Id: BD78FA0005 X-Rspamd-Server: rspam11 X-HE-Tag: 1727314520-662241 X-HE-Meta: U2FsdGVkX1+j1bBFts7dU31T1TCaARVNxLikVK7/CYCwI+2it9xk84x+OtzErkHxwdiOG+n/bHeNNCUx71KDfgLvrDN68SnfUZkeySWX3HMS0heefZsd5RnLhdpw91LhJ3T6GiVI8ki6NrhXA6ocerlG4Rfqn8QxJkCA7j7PAJH59y2m7lpc4BWStzetDfHu2D2dcIKUxQwLjvc1E2E9TnCIZqHwPzl/SDG9A/oUBh+BmvFQZe0VkyIU3CkqYouNMBkV/JWbE51WrDGjTHoLYBD0bvaJjlPhVQW5uJCqljwzoEwa99pV9itHqHttqiIiU4P4dPlDlqzqkNLZQ+55UZIviJ17D1CNWN72yZFDbjqeyugHH1j+xkpVoGcyivU9LUPuQ0xUnm8pBFBx8LmFmbaP1rMVeqUqZC2/KII7Fc5oM3tLqThnOK4f5d1RUwlFRh9RB8EBlm+rekSMQXyxrUEnSzFc/vqsiNvnJdtrIJaStSq/A5sSX3CSSjL1WwehzJHcY/q/kRqMjs/i+L9VmY0T01Qh+njMvjWH8xCcFPNF1CpqAQPFefykGnqrj/YjI7747o+AIMbVntUU9lc9wCRlKabHLLgWEqPni2llLZuc95kTpFHg5v6yrdXwukeCw0TtG+E7Cdi29CcpOS7g1xlEIumoRhjxGMJh9sqdRcvL04JE5WllHXr5NEaYxZkMhA4BSmb4SfjmFHj8zZ1B+rUyzgiK5vsQ4yh8Fcf5Fv10OmN275FWF44Bh2aLImyGCI2ZXnE/k5LprkjT72Y9M9NnGbI7W+fJkYnnmSHIgmg2I6u7g66FhHntEZjEk7UxZKYgo/CQ+vKAqchw86oCcUkUcsjyIdqRx9x1+MvjdpRsLm7E+qX1X2/1g8nxBaX2Dj8gAQ4ZWb8oFe/Fbh1AM6wGWnpuRX/ZhcMTkCmkzNwAEPan40vKu0dq9T1+tpqDKrW6ohg2aLJFdOnmIGV DkpBxxAs xLQxjel/3/z7bVzLt/Fk3Y1kDrbj4/rHWByRtg9wR9U5nAUOIJ3x1RXSuUgJuwX2GgocfibiNP0H0Q2cKPq9AhTywMD7SYpFFDQJLTIvJEdkrQ+x/caVzJzGWKdVkuF+LkPrbsBxY86fnHV/n0CjX02Md1TCDpJa3EG1vfAV4znbsKEDWE6zXnpIXpdn8pisiwcoAsiP03XI0MF0S4ronntX6Yd8ufdq75qBHN4zlRkGOQZcYltrLw6A8jcJT/UJH2qYIw+gLD90Mc+6GrhreKX6RWvV8xHy2epsQLSGKQj0zTLd9IGNNRp79UyfuPtsroZtGCNmvaQxlIzQlJkE6v8/8yIEBHO94e9n4RVJtB06nAktvX747frH6l8SYDsWWOxWNlMdbpytrsUmEJ3NOOI1rTLpvXCtZk2M/iYqugqhclhFw7ivmQZQpxcaxCB1oQ86u1A2NLgPVaYz2Z88bjbuGGdQM4kErPRFh6IwcAzhmJ++qSG2CIh5TF2vrKLB0fv9w6NZMkKOi2ehUbZLBglfI/hmmBqluh3Ryc17Bkf025Ao5d+xEnFHoFNEvnOfaLXCzt+KeiNxu7EeDXmA2hy8ExoYDSBXdON5ooWXs4E0FBjA2SB4gN/0lDg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Walk the TDP MMU in an RCU read-side critical section without holding mmu_lock when harvesting and potentially updating age information on sptes. This requires a way to do RCU-safe walking of the tdp_mmu_roots; do this with a new macro. The PTE modifications are now done atomically, and kvm_tdp_mmu_spte_need_atomic_write() has been updated to account for the fact that kvm_age_gfn can now lockless update the accessed bit and the W/R/X bits). If the cmpxchg for marking the spte for access tracking fails, leave it as is and treat it as if it were young, as if the spte is being actively modified, it is most likely young. Harvesting age information from the shadow MMU is still done while holding the MMU write lock. Suggested-by: Yu Zhao Signed-off-by: James Houghton --- arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/Kconfig | 1 + arch/x86/kvm/mmu/mmu.c | 10 ++++-- arch/x86/kvm/mmu/tdp_iter.h | 14 ++++---- arch/x86/kvm/mmu/tdp_mmu.c | 57 ++++++++++++++++++++++++--------- 5 files changed, 58 insertions(+), 25 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index 46e0a466d7fb..adc814bad4bb 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1454,6 +1454,7 @@ struct kvm_arch { * tdp_mmu_page set. * * For reads, this list is protected by: + * RCU alone or * the MMU lock in read mode + RCU or * the MMU lock in write mode * diff --git a/arch/x86/kvm/Kconfig b/arch/x86/kvm/Kconfig index faed96e33e38..3928e9b2d84a 100644 --- a/arch/x86/kvm/Kconfig +++ b/arch/x86/kvm/Kconfig @@ -23,6 +23,7 @@ config KVM depends on X86_LOCAL_APIC select KVM_COMMON select KVM_GENERIC_MMU_NOTIFIER + select KVM_MMU_NOTIFIER_YOUNG_LOCKLESS select HAVE_KVM_IRQCHIP select HAVE_KVM_PFNCACHE select HAVE_KVM_DIRTY_RING_TSO diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 0d94354bb2f8..355a66c26517 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -1649,8 +1649,11 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) { bool young = false; - if (kvm_memslots_have_rmaps(kvm)) + if (kvm_memslots_have_rmaps(kvm)) { + write_lock(&kvm->mmu_lock); young = kvm_rmap_age_gfn_range(kvm, range, false); + write_unlock(&kvm->mmu_lock); + } if (tdp_mmu_enabled) young |= kvm_tdp_mmu_age_gfn_range(kvm, range); @@ -1662,8 +1665,11 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) { bool young = false; - if (kvm_memslots_have_rmaps(kvm)) + if (kvm_memslots_have_rmaps(kvm)) { + write_lock(&kvm->mmu_lock); young = kvm_rmap_age_gfn_range(kvm, range, true); + write_unlock(&kvm->mmu_lock); + } if (tdp_mmu_enabled) young |= kvm_tdp_mmu_test_age_gfn(kvm, range); diff --git a/arch/x86/kvm/mmu/tdp_iter.h b/arch/x86/kvm/mmu/tdp_iter.h index ec171568487c..510936a8455a 100644 --- a/arch/x86/kvm/mmu/tdp_iter.h +++ b/arch/x86/kvm/mmu/tdp_iter.h @@ -39,10 +39,11 @@ static inline void __kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 new_spte) } /* - * SPTEs must be modified atomically if they are shadow-present, leaf - * SPTEs, and have volatile bits, i.e. has bits that can be set outside - * of mmu_lock. The Writable bit can be set by KVM's fast page fault - * handler, and Accessed and Dirty bits can be set by the CPU. + * SPTEs must be modified atomically if they have bits that can be set outside + * of the mmu_lock. This can happen for any shadow-present leaf SPTEs, as the + * Writable bit can be set by KVM's fast page fault handler, the Accessed and + * Dirty bits can be set by the CPU, and the Accessed and R/X bits can be + * cleared by age_gfn_range. * * Note, non-leaf SPTEs do have Accessed bits and those bits are * technically volatile, but KVM doesn't consume the Accessed bit of @@ -53,8 +54,7 @@ static inline void __kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 new_spte) static inline bool kvm_tdp_mmu_spte_need_atomic_write(u64 old_spte, int level) { return is_shadow_present_pte(old_spte) && - is_last_spte(old_spte, level) && - spte_has_volatile_bits(old_spte); + is_last_spte(old_spte, level); } static inline u64 kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 old_spte, @@ -70,8 +70,6 @@ static inline u64 kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 old_spte, static inline u64 tdp_mmu_clear_spte_bits(tdp_ptep_t sptep, u64 old_spte, u64 mask, int level) { - atomic64_t *sptep_atomic; - if (kvm_tdp_mmu_spte_need_atomic_write(old_spte, level)) return tdp_mmu_clear_spte_bits_atomic(sptep, mask); diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 3b996c1fdaab..4477201c2d53 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -178,6 +178,15 @@ static struct kvm_mmu_page *tdp_mmu_next_root(struct kvm *kvm, ((_only_valid) && (_root)->role.invalid))) { \ } else +/* + * Iterate over all TDP MMU roots in an RCU read-side critical section. + */ +#define for_each_valid_tdp_mmu_root_rcu(_kvm, _root, _as_id) \ + list_for_each_entry_rcu(_root, &_kvm->arch.tdp_mmu_roots, link) \ + if ((_as_id >= 0 && kvm_mmu_page_as_id(_root) != _as_id) || \ + (_root)->role.invalid) { \ + } else + #define for_each_tdp_mmu_root(_kvm, _root, _as_id) \ __for_each_tdp_mmu_root(_kvm, _root, _as_id, false) @@ -1222,6 +1231,26 @@ static __always_inline bool kvm_tdp_mmu_handle_gfn(struct kvm *kvm, return ret; } +static __always_inline bool kvm_tdp_mmu_handle_gfn_lockless(struct kvm *kvm, + struct kvm_gfn_range *range, + tdp_handler_t handler) +{ + struct kvm_mmu_page *root; + struct tdp_iter iter; + bool ret = false; + + rcu_read_lock(); + + for_each_valid_tdp_mmu_root_rcu(kvm, root, range->slot->as_id) { + tdp_root_for_each_leaf_pte(iter, root, range->start, range->end) + ret |= handler(kvm, &iter, range); + } + + rcu_read_unlock(); + + return ret; +} + /* * Mark the SPTEs range of GFNs [start, end) unaccessed and return non-zero * if any of the GFNs in the range have been accessed. @@ -1240,23 +1269,21 @@ static bool age_gfn_range(struct kvm *kvm, struct tdp_iter *iter, return false; if (spte_ad_enabled(iter->old_spte)) { - iter->old_spte = tdp_mmu_clear_spte_bits(iter->sptep, - iter->old_spte, - shadow_accessed_mask, - iter->level); + iter->old_spte = tdp_mmu_clear_spte_bits_atomic(iter->sptep, + shadow_accessed_mask); new_spte = iter->old_spte & ~shadow_accessed_mask; } else { - /* - * Capture the dirty status of the page, so that it doesn't get - * lost when the SPTE is marked for access tracking. - */ + new_spte = mark_spte_for_access_track(iter->old_spte); + if (__tdp_mmu_set_spte_atomic(iter, new_spte)) + /* + * The cmpxchg failed. Even if we had cleared the + * Accessed bit, it likely would have been set again, + * so this spte is probably young. + */ + return true; + if (is_writable_pte(iter->old_spte)) kvm_set_pfn_dirty(spte_to_pfn(iter->old_spte)); - - new_spte = mark_spte_for_access_track(iter->old_spte); - iter->old_spte = kvm_tdp_mmu_write_spte(iter->sptep, - iter->old_spte, new_spte, - iter->level); } trace_kvm_tdp_mmu_spte_changed(iter->as_id, iter->gfn, iter->level, @@ -1266,7 +1293,7 @@ static bool age_gfn_range(struct kvm *kvm, struct tdp_iter *iter, bool kvm_tdp_mmu_age_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range) { - return kvm_tdp_mmu_handle_gfn(kvm, range, age_gfn_range); + return kvm_tdp_mmu_handle_gfn_lockless(kvm, range, age_gfn_range); } static bool test_age_gfn(struct kvm *kvm, struct tdp_iter *iter, @@ -1277,7 +1304,7 @@ static bool test_age_gfn(struct kvm *kvm, struct tdp_iter *iter, bool kvm_tdp_mmu_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) { - return kvm_tdp_mmu_handle_gfn(kvm, range, test_age_gfn); + return kvm_tdp_mmu_handle_gfn_lockless(kvm, range, test_age_gfn); } /* From patchwork Thu Sep 26 01:34:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13812680 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3C79CCCFA10 for ; Thu, 26 Sep 2024 01:35:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 24D886B00B8; Wed, 25 Sep 2024 21:35:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1A9F16B00BA; Wed, 25 Sep 2024 21:35:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F143B6B00BC; Wed, 25 Sep 2024 21:35:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id CDD7F6B00B8 for ; Wed, 25 Sep 2024 21:35:23 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 80E60160A18 for ; Thu, 26 Sep 2024 01:35:23 +0000 (UTC) X-FDA: 82605171726.23.7AC086C Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf25.hostedemail.com (Postfix) with ESMTP id C366AA000E for ; Thu, 26 Sep 2024 01:35:21 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="sW/2tXo2"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf25.hostedemail.com: domain of 3WLr0ZgoKCN0ISGNTFGSNMFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--jthoughton.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3WLr0ZgoKCN0ISGNTFGSNMFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--jthoughton.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1727314432; a=rsa-sha256; cv=none; b=jI3D+ouqwN7QXJBkueCsZLKtSJAffahPG0ghy++Pft2cL0PJo1ybBjHm5ejRYO811oFCMd X7rttukaIomGF1dNU4tLHJ+3Ri+jm5KSnz733JirlvaLyPrxogmKHQSEhsjckzcVTYuLBg mQOIfQHqcTerhb6fJwRJh3vokkqQZBo= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="sW/2tXo2"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf25.hostedemail.com: domain of 3WLr0ZgoKCN0ISGNTFGSNMFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--jthoughton.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3WLr0ZgoKCN0ISGNTFGSNMFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--jthoughton.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1727314432; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=dAogBQT7i8/Ut4GrJolXC+/DUS73zZR+MbirDcTLTRg=; b=QtwP+v9rqOSKTKgdR6rlKdeYexHrvcM6wCPDkt7OnFYtkKJDgvTMf0KtXXR6cHBYbqnLH7 qUzuTfkHKYb7SEKxWVW1lZNsbjpFA9khKtsoW4kbtVU2Iug8SH+OSQoWNzhvfp0lcavjR3 2z30xKvIoG7qxZwr4Ir3RSlAvgUYZ+g= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-e1159fb161fso727384276.1 for ; Wed, 25 Sep 2024 18:35:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1727314521; x=1727919321; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=dAogBQT7i8/Ut4GrJolXC+/DUS73zZR+MbirDcTLTRg=; b=sW/2tXo2ZEbnGg9NM85be4p1EdqkqaHW3A8AF8T0LAHJp8epLO6reEJffeRecNkBaY ODRizHz48oKp98+v39Sverf/Lw1gjcZTV8otRbaezAXjzGmg02b995NXBMpy+nXOjhRr +Vwsmo8TP7qXnzSVzmAn4FOAubO+aXjGxSTGaScidTdjm9TqBTjPK3vjnEumHpSAVu58 vbbZFwBOr2xEoi9QrNA/mgqYuc9l74AOotHlLILff7MKWwzvB2vgLLTnw3O9QXINHbrQ RTP+/0fxbSX/OER5o6cKrCyWfB0PnPEN5vRiEOSzHQRK0scfiVfAxPvMDGXF9Hu7Kz7t 9IUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727314521; x=1727919321; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=dAogBQT7i8/Ut4GrJolXC+/DUS73zZR+MbirDcTLTRg=; b=XSYOZLdAfmbHI0c7seQ86VM5vY7R7q3cVixVjot87dqptw+H5xxmnMPUZaAV4xtskE 4x4uJbmAQyHM7gyAOiOyPGUMp0nFWNFvkGqkmWcqWvTTbHBsMKSQUNcWvkW180tRtqO4 /TuktZ2WxURuvLQUTNXTaU2t1/OyWKEpzT+IPfizWsn3/jqTcM3g/nGzh0Tx0NBo2hK7 O+FuMFIyKyOKdcl1/mhr2/2OKE32dD+UKov50Dny8bQHQyNtDZ0sewHBAY08AwMmgPhj b786YCtXsfbvfLQPtA840ZKXvdWgo6B5fitonLZkJHQghnD9LzJEES8D/bCJMWvXEIk8 Quvw== X-Forwarded-Encrypted: i=1; AJvYcCUldE6F2fXLXuMzRKK78HzU9HFJkkf8Zv4LbRNFjX6U1rnslosUyIuQoeVSdVYThbB0UayFQL+TNw==@kvack.org X-Gm-Message-State: AOJu0YxEt4nzLVwOwLj4i/WGj7I2veHo5MvazmMWKoYcr4Ij+HM9kxyv hAzbeObjdCqPMEMYRiVjDfUOTmFK8KsfXeJmO+1Wwq/5L+xKezIn7isnqzSAVaYwdVT22NlTjHK 8d9op7gnx7Md3yQWEJg== X-Google-Smtp-Source: AGHT+IFQs3DNBWoE9LT9ciS2x6yC/841ISG2URP+wJo+4FsamoM+oSoqTqpMkGycanRDnHNwxgVEwB1O11ztgMfN X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:13d:fb22:ac12:a84b]) (user=jthoughton job=sendgmr) by 2002:a05:6902:1782:b0:e11:7a38:8883 with SMTP id 3f1490d57ef6-e24d9613a90mr3400276.7.1727314520739; Wed, 25 Sep 2024 18:35:20 -0700 (PDT) Date: Thu, 26 Sep 2024 01:34:53 +0000 In-Reply-To: <20240926013506.860253-1-jthoughton@google.com> Mime-Version: 1.0 References: <20240926013506.860253-1-jthoughton@google.com> X-Mailer: git-send-email 2.46.0.792.g87dc391469-goog Message-ID: <20240926013506.860253-6-jthoughton@google.com> Subject: [PATCH v7 05/18] KVM: x86/mmu: Rearrange kvm_{test_,}age_gfn From: James Houghton To: Sean Christopherson , Paolo Bonzini Cc: Andrew Morton , David Matlack , David Rientjes , James Houghton , Jason Gunthorpe , Jonathan Corbet , Marc Zyngier , Oliver Upton , Wei Xu , Yu Zhao , Axel Rasmussen , kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org X-Rspamd-Queue-Id: C366AA000E X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: e81tif9p646bwrcc8d4bc1aypee9f6k1 X-HE-Tag: 1727314521-168189 X-HE-Meta: U2FsdGVkX19EXkYdDEsVvOLvSruOCOdgMCf7T7ZYYIzatKMB4sbW4iJ1KiXqnYARnbs1Sba9a8WL38GAoseaN6n3jG4DGc7VUU6oksaLq2L3bNZy+zsuNOCp5h8hBgXBZPt5+n8+s4dRppkGhOgDWqfSxadva+hZA6GWV5bkuVOI68yorCVDwuge/aL1e4lfCFVOMO8y+e36U4u3oK6bW6nyj5f8ZSq/II6manx1AUbHy9RaKzt1me3feQkHEFrQs6zM0LREOScglye6so/gQbgdSyWK7LZ8OlD80XV+UKy0rycJ9bG235gPkLyaatstFof5ARaxRilbbiXQzhw1s+W3o/dh7a9e0EM5Md6TN14jXGQgYdLbyVfnZAHjQ9IIpjx6fmpJLeCOcxHeHLhV4Y5k+yEkKftzRh5a1DGvfTt6wzVpdN5jx8oDNr6BT20mSxNj3RvuMuXSflm1ejEITLoaL25pGfuAvxinnkaue910+ZXqqxzkzJECFoXiIzfEQB5nZVJEvXFRW/IvdISeKYPFbAMUdAAx/xbaIZGHipNDSG65hNFkXWVRkuM2MHblLAq85uKvJUHmYpC9dB8dLMB1JNiYRhFEicwMz8M+TukC/NHaLhLLn5mfxy4Pf3o3WrJi/BEarIMsGIIOTOiKkR7yUOPOLy/PApbaePO75HTrO8TN1MatEEUX5Kl3gKYkqFvGpC6lnq2oSX6Vam+nSElm167Wn9hRKRmNyEX2PHOGsY2Bomce5ihuL8s2ya2yEqT3+qHzctkSJvl8Z5VHhwZ181D4h+hZOmAEMrrViN3STYadXxBV77He+ED7Ej8LqlXPUt+MPbB/rzXb7LaIp04p55+AWM9mdUneFuD1b92AdJL5sg7LMAcz88ZMnfRwfsTBmvSBDBD5lzwdZTtcTLqEIn+J2qtORNw/q5OcWmfJRVskceiCn5AA7+1qa8ubF0/SzO5NU3oLl4gr12r iWd5hYxw MUiaL000aTbz3k5LKRXG72Uab7U3W40c0JqDc9qrjPN5pwtL8bEL50mrhPGjrLsecMY1RhxBmjuR9QN1wQRlxYDpHyW/ubK/9AErZzqVRF1CbWbTVjis2h1L6i0BY9tG1f3jPZr9nY0HXHFzMPBPsnuUOEJC0Eao25nWm0fxlPmwTewrOG2tXhhXjhbJeEOnrDjoTcKnu294SDuh9dyGVIkW2N52j7bATYN4REdVWqCB1G+vOmwW+mM7BDRjw6uBJRkSRVLgEnUsAMF/kgJGZ5I7AU4nbDd+EQUf2tFE0oXz+ZIYzodROYiGZvSK1lsOfEY/xDdo35KdA0p1kIlOyCDjBXijUa8kbIR7oBIxsD38wezgYDJDsFUl9e3fxcSmmQp39FnC4JNYESdp6JwE7gHOS1X799aCp4330MWQcAIJlH9/f5DfaHCzEimfCKA9VRZXaWAviKCeURL917uSyFdDhGnGzCNBqSLzH0hQj7bEpgwtIY0YlU5iW7w2wWClKIZoWdU/H8yV2de1dPXnmiSepOEzHt33V30rZ46GB60GNGwGU6OYeW+jCNJW89YbUw0y3eO98qK2KDqkjoIBBMQ3tDumtuigv9OEkWiEn0w3dwKRR46EVsz3ulw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000042, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Reorder the TDP MMU check to be first for both kvm_test_age_gfn and kvm_age_gfn. For kvm_test_age_gfn, this allows us to completely avoid needing to grab the MMU lock when the TDP MMU reports that the page is young. Do the same for kvm_age_gfn merely for consistency. Signed-off-by: James Houghton --- arch/x86/kvm/mmu/mmu.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 355a66c26517..03df592284ac 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -1649,15 +1649,15 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) { bool young = false; + if (tdp_mmu_enabled) + young = kvm_tdp_mmu_age_gfn_range(kvm, range); + if (kvm_memslots_have_rmaps(kvm)) { write_lock(&kvm->mmu_lock); - young = kvm_rmap_age_gfn_range(kvm, range, false); + young |= kvm_rmap_age_gfn_range(kvm, range, false); write_unlock(&kvm->mmu_lock); } - if (tdp_mmu_enabled) - young |= kvm_tdp_mmu_age_gfn_range(kvm, range); - return young; } @@ -1665,15 +1665,15 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) { bool young = false; - if (kvm_memslots_have_rmaps(kvm)) { + if (tdp_mmu_enabled) + young = kvm_tdp_mmu_test_age_gfn(kvm, range); + + if (!young && kvm_memslots_have_rmaps(kvm)) { write_lock(&kvm->mmu_lock); - young = kvm_rmap_age_gfn_range(kvm, range, true); + young |= kvm_rmap_age_gfn_range(kvm, range, true); write_unlock(&kvm->mmu_lock); } - if (tdp_mmu_enabled) - young |= kvm_tdp_mmu_test_age_gfn(kvm, range); - return young; } From patchwork Thu Sep 26 01:34:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13812681 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2B661CCFA10 for ; Thu, 26 Sep 2024 01:35:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5543B6B00BA; Wed, 25 Sep 2024 21:35:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4DA776B00BC; Wed, 25 Sep 2024 21:35:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 32C9C6B00BD; Wed, 25 Sep 2024 21:35:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 072D46B00BA for ; Wed, 25 Sep 2024 21:35:26 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 96CAEA06FE for ; Thu, 26 Sep 2024 01:35:25 +0000 (UTC) X-FDA: 82605171810.04.0B078B4 Received: from mail-qt1-f201.google.com (mail-qt1-f201.google.com [209.85.160.201]) by imf12.hostedemail.com (Postfix) with ESMTP id DF68B4000C for ; Thu, 26 Sep 2024 01:35:22 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=bR+2CdIP; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf12.hostedemail.com: domain of 3Wbr0ZgoKCN4JTHOUGHTONGOOGLE.COMLINUX-MMKVACK.ORG@flex--jthoughton.bounces.google.com designates 209.85.160.201 as permitted sender) smtp.mailfrom=3Wbr0ZgoKCN4JTHOUGHTONGOOGLE.COMLINUX-MMKVACK.ORG@flex--jthoughton.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1727314486; a=rsa-sha256; cv=none; b=yTBjrtj5Wlnz3UM8dVmem6nZPG7jgwnQ39S8FB2RB9eI6iDyRKOGJXXQFc2qGrqq8OVoph OO4cEI0iIMOIEEwV3r3FfUSwWRYT7g7W3nJH+MycBsbS6doF9A6+cWs5S64C54uR5h4gO3 4/TkhJ7xgL8MtxVtRh3GqHcf9Jxrn2U= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=bR+2CdIP; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf12.hostedemail.com: domain of 3Wbr0ZgoKCN4JTHOUGHTONGOOGLE.COMLINUX-MMKVACK.ORG@flex--jthoughton.bounces.google.com designates 209.85.160.201 as permitted sender) smtp.mailfrom=3Wbr0ZgoKCN4JTHOUGHTONGOOGLE.COMLINUX-MMKVACK.ORG@flex--jthoughton.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1727314486; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ZiB+e++yi01uCPO64xcLvipIH38DTgOpYlfSBrXkWpM=; b=nZfMz2rHpwyDAdcST4t557Z0kMXj/i78JxMewF1atP1Im1T6TNmteYgbssl9D6wpCzncRf IvpEpyp+seCI5fDc8qVnTv231GglxxLP9dZiZYeDnQNsjLJyKtzkghexvn+hBvd9F4kYqj aQpefxS7BLsNCV26ghJUUSP2zsrFA1I= Received: by mail-qt1-f201.google.com with SMTP id d75a77b69052e-458353d3635so9499241cf.0 for ; Wed, 25 Sep 2024 18:35:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1727314522; x=1727919322; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ZiB+e++yi01uCPO64xcLvipIH38DTgOpYlfSBrXkWpM=; b=bR+2CdIPcvpJHh0a1zvXvi7svmoT2fEApXuCiS0k1UPrd13Z5PEP+wglieIsSk2RZa YdX/x4RsDs0I4reakX0ULEVYPpfhXEFTIcFIkw7vrm+gAqf8VKTu6p0CQfCQ2i99qFPK XQFB7BEJxLd5HGlqUvALG1kvCdRp+bYUrYpMAVUSoG2GnxhTvrSWYDDe0LEhzZkOg9X7 iClUaxXnkwwXEkjzLl2aqp5y0Cz6gBfLNVLhsJiyMRBHgso6rAS8uOWAeVpsPJ78g0z6 4FZs+/4cTfqMPm+qsdq3ZKUAAyrhjanMTvL7mVp6Sr971lxiphqjjuBQkYOKcepZ4zhI 7oAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727314522; x=1727919322; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ZiB+e++yi01uCPO64xcLvipIH38DTgOpYlfSBrXkWpM=; b=jvYOJptH/wNF7n601V3DDOrjZRTvteGys3TSC8iW92Yg4Dt3AAszm7x0ghVL1ZoFcS Axhs/hXL5Y9HDHTTA513+KaAy3XS/dWSmgO2tr3/Jd8yEUXvPzCEhiE1jT/g1PqRQ7Xn SRBSdyQ2Y4Nqs+6JvDNLkSlhsReTDyGVWLhCqKzXErsBHbpIn7UFqD052EcQ7coEruhG JzqPxyjAFy0qiLk/haoEDGcj4XtLj8wX94Zxb+6UaQ9YNog+f2g2nhJvzk5Paq2lVDYd f7TjJsQLAidXPAwFEU3INPHVOVGALwf0/LjQ5g0qm2bdFiR5X8lZ1JUSdiglG5WoZAlR bpbw== X-Forwarded-Encrypted: i=1; AJvYcCVQAWuzz7ntEZm9a5Lp0UsWj7O9WsvhlqaySpHcwYWahPaWzkDp4R77oKl8lwR5M69F/1JD7O6QMg==@kvack.org X-Gm-Message-State: AOJu0YxLzFGXUP7ZoZ1m+CSdF9DReLqgzxg5Ehv3XNWk2XYLEenQARTQ 46bcEV6sFrNMqIdITyzYuGurwr2g7U+Q6EY8TD6f6e73KnQ8MfSjgXAP84qimANUNGcEW0iFR4p zm1e2cYgr78m2baRs1A== X-Google-Smtp-Source: AGHT+IEij2jZTZ7Qfz4w3meuEhq8eeslauQ2yZtvTlXKLk5ZsUhbavG3fPazfw188lkIpQjxnlBFPCvmtl1c33sQ X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:13d:fb22:ac12:a84b]) (user=jthoughton job=sendgmr) by 2002:ac8:43cf:0:b0:458:4a61:2020 with SMTP id d75a77b69052e-45c9493ec3dmr13541cf.2.1727314521861; Wed, 25 Sep 2024 18:35:21 -0700 (PDT) Date: Thu, 26 Sep 2024 01:34:54 +0000 In-Reply-To: <20240926013506.860253-1-jthoughton@google.com> Mime-Version: 1.0 References: <20240926013506.860253-1-jthoughton@google.com> X-Mailer: git-send-email 2.46.0.792.g87dc391469-goog Message-ID: <20240926013506.860253-7-jthoughton@google.com> Subject: [PATCH v7 06/18] KVM: x86/mmu: Only check gfn age in shadow MMU if indirect_shadow_pages > 0 From: James Houghton To: Sean Christopherson , Paolo Bonzini Cc: Andrew Morton , David Matlack , David Rientjes , James Houghton , Jason Gunthorpe , Jonathan Corbet , Marc Zyngier , Oliver Upton , Wei Xu , Yu Zhao , Axel Rasmussen , kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org X-Rspam-User: X-Rspamd-Queue-Id: DF68B4000C X-Rspamd-Server: rspam01 X-Stat-Signature: yiy551eb83i7616on7w3xjuhk6c8h3a4 X-HE-Tag: 1727314522-877493 X-HE-Meta: U2FsdGVkX1+NAFUuonPKw05nW5ddeJcJshE7pDBBxCPn583e2iiwCB/2FdI6gqtFc8qclq0WbYoTvSNb5lLssPEePPnciSAkP6jfGHbl90xx8HoW/0/fqWS6r7cfAUPT7w1Ttbxio39TcyekdVB+iwCNo5GvtYqdY0vGX2q/v4uN0NjUi/nl0AhTm7WTX6FRtU4HzHcRwITXwvStHvOVt3itszNMvmFpIfduZDw5PETF1m9e7s34lbkyDQ9Yp2vwm+4mQgimrPYNIZLIC5DxHrpmgyHcYHC+kSFxgYbaWKEK+IVPiWyFMT/ViWuGf9iSC6OFXnX2wAkSzi7PtM/GzhjbX7LBZPprzE3xLJC06TC6cduq5hSgYZi+bM44Pm8DK9UgAkGse5nC0SqkTsCKIlq3/tttQ6Sj9a1nF8vq/wiaz8iuY52bJZkrYTYwx3iryb9mV3wgN4RHP9RR8mfi7pitSgJe4uDbKkhm9GA8FSxtm+AlTSjCfUUDO2XgmNwLrNNtKaL0PyQzS5cQjVmRo3jQn+UXlNALHqeE3gOvVVf8v9UpCdVeLcak35FvY9FyfPfv8SzXCO03dLm1YgJ+U577n1B9C0nEdunRLwlXeCimwDldCl8YOtAW5oi+1HIM8SuH4LQRAkdjytTeW/DPNMFrxM8CbpW+uyvTVfEfsPPJNN6pGxH6L4Yg8G0HUIzyVNl+kQqdicwnET0XOYY0rEq/+JX89SwcBwpy/JCG5xoot5OMqTGj/jwHCD8JwAlD8dvu7oLZ7YM6uC3dcDQzQEBw6mqDR+HEUPsaG56Adum7IIDTcQlztuMu7e8R/LWqWrrxzD9wnc32Z+WSGP8AJhSAFvPIT6YZLVYv8KZkUahsO+dICBdCQB2v9NU2THYdoiFeNOLkVg2VOzXgtnXki9bUOa0MoV9+mnZ4TTm4uUqcZC5DueG/ExZjNk6Zxpm862K8x8DIykQXzfvGUfD 5NKlyCF+ iPv+YdZeEw57d/6u/Kfu6kTfH6tWTpGIQ2+Eyp7uoFiWJhS6c8emf3ij+rS0DEetAkkN35k7+PRkaJN3EGmhqPt7LeKsgkrp6vGoCcJU1fvDcsRD5J6gzxyXRRK+2udRDfuLciRglin4plP/OTklCAF15iGVWU9Te37WTgERNhWnWMqajXdir5Ra/JjkaZi88vw0XfJLlrsfzm1b9xj3oPPLR956IlrPD5zTi6ZX9ixNSewNqi3ErHEcJ5hbswtC0h09VJ/VCBmnGr8hJ0JFKAs0N+mcPfqN8J7y5J+5G1gRPcsCGQQIZEPPQ0G2VNclPpC47ZTVFTP7sM0oXE+RQZOksMw2GJtPDz00AddCqny6BshJTXRcJ+7tiZ5tSg5/cYw6GS7R6svyi+wbNy0JmB2xNhTv2UPJxND0vQALJ9/u/TiuQHV3QmVnp/DRdhsDdSBYwaQuGC8gkYZJsbD+hwcdtNnFEw+wkoecNxI8PqFs2rKo45FQEBFFl6my3heTxzK69+hx2U/44tdrEswzKf98BtSLvLKJADNiO/j8kgRP+IJcIJGEhchL7UXUAzLBMmjNffGI1zEAcv2s5AXhcexqG8c1i17c+2qtKAVmAVGmMadzSGjH65Df5qA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000026, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Optimize both kvm_age_gfn and kvm_test_age_gfn's interaction with the shadow MMU by, rather than checking if our memslot has rmaps, check if there are any indirect_shadow_pages at all. Signed-off-by: James Houghton --- arch/x86/kvm/mmu/mmu.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 03df592284ac..b4e543bdf3f0 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -1645,6 +1645,11 @@ static bool kvm_rmap_age_gfn_range(struct kvm *kvm, return young; } +static bool kvm_has_shadow_mmu_sptes(struct kvm *kvm) +{ + return !tdp_mmu_enabled || READ_ONCE(kvm->arch.indirect_shadow_pages); +} + bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) { bool young = false; @@ -1652,7 +1657,7 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) if (tdp_mmu_enabled) young = kvm_tdp_mmu_age_gfn_range(kvm, range); - if (kvm_memslots_have_rmaps(kvm)) { + if (kvm_has_shadow_mmu_sptes(kvm)) { write_lock(&kvm->mmu_lock); young |= kvm_rmap_age_gfn_range(kvm, range, false); write_unlock(&kvm->mmu_lock); @@ -1668,7 +1673,7 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) if (tdp_mmu_enabled) young = kvm_tdp_mmu_test_age_gfn(kvm, range); - if (!young && kvm_memslots_have_rmaps(kvm)) { + if (!young && kvm_has_shadow_mmu_sptes(kvm)) { write_lock(&kvm->mmu_lock); young |= kvm_rmap_age_gfn_range(kvm, range, true); write_unlock(&kvm->mmu_lock); From patchwork Thu Sep 26 01:34:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13812682 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 30303CCF9EB for ; Thu, 26 Sep 2024 01:35:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0BE896B00BC; Wed, 25 Sep 2024 21:35:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0461D6B00BF; Wed, 25 Sep 2024 21:35:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E00FA6B00C0; Wed, 25 Sep 2024 21:35:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id C19186B00BC for ; Wed, 25 Sep 2024 21:35:26 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 725D1120741 for ; Thu, 26 Sep 2024 01:35:26 +0000 (UTC) X-FDA: 82605171852.25.0ABADD6 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf24.hostedemail.com (Postfix) with ESMTP id AE15E180007 for ; Thu, 26 Sep 2024 01:35:24 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=1zCzpgur; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf24.hostedemail.com: domain of 3W7r0ZgoKCOALVJQWIJVQPIQQING.EQONKPWZ-OOMXCEM.QTI@flex--jthoughton.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3W7r0ZgoKCOALVJQWIJVQPIQQING.EQONKPWZ-OOMXCEM.QTI@flex--jthoughton.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1727314426; a=rsa-sha256; cv=none; b=sBoaZnAGL9GXSup+FF9G+9+GQ5VVoO7sYoii5sJxYtzz9xkLNUb2hUGfc/Nd+ms7JMDVil 2lyQd3SwvLlr4UUQspj+/jVcGthwsZkQPIK8pN3WQM1+85yEGdP8tj1bgB1pbMUIlxG2sZ pjRUGdGScAMHQZDvgaSuCBmkJ4UVMn0= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=1zCzpgur; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf24.hostedemail.com: domain of 3W7r0ZgoKCOALVJQWIJVQPIQQING.EQONKPWZ-OOMXCEM.QTI@flex--jthoughton.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3W7r0ZgoKCOALVJQWIJVQPIQQING.EQONKPWZ-OOMXCEM.QTI@flex--jthoughton.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1727314426; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MAX0OZs2fthT88FLek8b9DqcvP6EmNxdVeFSdhP6yzs=; b=m/SnhjHLAwfc4YbjBuKWNYRDVo3sE6HnVpd7LXZh3h+Bwe3SobsMnCHjFHQXhJVjw4fQFe LraZXIfk5+28kefX7NFGxjDXBxNd33LXWyTHk2EUkdu6c2QKBfkN+6JFR+f6d9mdtQD8Ea mG1CoH6HN3CtLuPlAF/HeiEP2M1ndTs= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-e25cae769abso816330276.0 for ; Wed, 25 Sep 2024 18:35:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1727314524; x=1727919324; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=MAX0OZs2fthT88FLek8b9DqcvP6EmNxdVeFSdhP6yzs=; b=1zCzpguraRpVW6/XfG2e2GkCgmdFFXYr7UK+h+XhSF+u0hQbrCsO/AvT3QGFXiyGKl HDgwzan5roRGSEqxK70mOuL5sLKYHKIWpIf/x9Gre2y+t+a5H3tZkk8GKOIkVx0kjN6K iOzgVMnMuC0+y1+b8biSKVvZ0mK5cAp8aQ7QyISI/zhTd0D8m4YrSNvvLMyXcYKnwpZZ M9gMAIh0252qwSpLVCWzfeRKGZeVQ6bpYAmfxgQ/zndezS8G/PPxpBnEX9Z0vbJ89AQF 0elHyYcJbqQTymrDp83HodNOhVAI/uZRaTTF6kxv6ZmaU/L2/7Y//l9VIF1oaLjHt6XG /t3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727314524; x=1727919324; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=MAX0OZs2fthT88FLek8b9DqcvP6EmNxdVeFSdhP6yzs=; b=hOuDn2ndfsAxmWWz9972KO3V0RoFUImNtESBBs+mR6SLBIaLwVOahpcABc79ABq+uP 2T9FCf7cvyARNmuW8zDNPy5dwP+CrMvMNS+KMcsxAknXdb+UD3gVdThUC5iIH7BCwXfz Dp29/gunpa2JF1XmrsMTpt7tmulEvI0/plgNmklzMywBsgj5z/C2LNW4S2IT7Ih9QMYQ OgH1VZ6hHFs/7byDNo2dbmu0bQnVrAfb3VrqcL5vdSUTSwu3cPubBWEQDhjk9GA7Nr0+ JsmZdW3kheqN3PPMSsaVDVCJM6k5q9uhw58lUcQu7/maO40v6X2mTKKAiWRQ69vH5F02 RRbQ== X-Forwarded-Encrypted: i=1; AJvYcCUDvES5+ryJHH1equ9sXT55ZfC8qaxHbTlfznBwC/LC9bnbf9zqgIGQg46rF81VJOp8/6HDdhfZNg==@kvack.org X-Gm-Message-State: AOJu0YwIjL9IS1UXh4xfYz6M/ZO2aJwa6cVxyvSsm+oG/Qk3KBAndrZM ImJ3tkmhAlb85cF/5I/jvtwB/8fUxDQNWi6V3/sfyV+awUMVjZLAENb4xCj7/teepmm2T7ODHk5 b3RIeBGOrskcmV4Vh1Q== X-Google-Smtp-Source: AGHT+IGaRNcJ0pUCHmWao5UQiFUuGRcX2zXYa1voIITtc2canA4FkWeSC7N08ZTT/PlEa6ZAuN210z4HqmS+Jzo6 X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:13d:fb22:ac12:a84b]) (user=jthoughton job=sendgmr) by 2002:a25:d608:0:b0:e11:6a73:b0d with SMTP id 3f1490d57ef6-e24d9dc640bmr3172276.6.1727314523065; Wed, 25 Sep 2024 18:35:23 -0700 (PDT) Date: Thu, 26 Sep 2024 01:34:55 +0000 In-Reply-To: <20240926013506.860253-1-jthoughton@google.com> Mime-Version: 1.0 References: <20240926013506.860253-1-jthoughton@google.com> X-Mailer: git-send-email 2.46.0.792.g87dc391469-goog Message-ID: <20240926013506.860253-8-jthoughton@google.com> Subject: [PATCH v7 07/18] KVM: x86/mmu: Refactor low level rmap helpers to prep for walking w/o mmu_lock From: James Houghton To: Sean Christopherson , Paolo Bonzini Cc: Andrew Morton , David Matlack , David Rientjes , James Houghton , Jason Gunthorpe , Jonathan Corbet , Marc Zyngier , Oliver Upton , Wei Xu , Yu Zhao , Axel Rasmussen , kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: AE15E180007 X-Stat-Signature: 43361yjgupkxixgmtc1596ah1tu98u8z X-Rspam-User: X-HE-Tag: 1727314524-374814 X-HE-Meta: U2FsdGVkX1+UTaztGV6IxHNNROisbDejRZzsO1yWc3cSiQQvLxtXAXa539djnTCiIABcgENRoEjWCgXor/hr88YwiX4sfVyfBaHNGcNsWSmIFUr4d/5J26EMaZi38+BE6XTZ4NVe4vVUbKfLZRmZ4F11A/28XkUIC80Ss5N1pfaOc41hAXLXq9HnuPCJeJQuKfVawOQtv1WsWvcOTz9YHLAR8ixDiWGcFgKWH4RCgLQb3cHoaIEc2ML4d3ELKqCd+REhTNQ3Wp1h8Nsezdwii0Dg57vLPaiHIzMhLHah7pt4OXp+ktN/HHTrvEgfL+0vGXjffsHEoMTdLemEn2HBdToIvULVBs6MfmxARpQhD4+rW1eB/FhMpx/SabEq322aL/pezN8J8hH0Zpiws4hCQv//V0WNF6LeGfQsQwzPf7RJZI3dL/2mSDeOHRGGeGKZbrK05cg11QBUvSyZp/27MX8MNWwnZNg3ReekSuXk5f3OSHosX73GY9jZoog5KduWU/E0j5LYrCtIG5OMB3sQyOZ4sfk6vdGgzs419gf6H2LG3CtxseIVN/jYMXdpAiT1cpw6qNLcSoAThEJfJIumxne9ly/tu+1AORl5cz2Zn/TOMFSOSqgm4iSaOjwfdjjY9VK9DgiFKLineqmV70bJapcKqdrhwWm2v8Wd5x8RreIFFn7PwtsUXX5UQiDs5RUH5ZhfSvUK5eq8gKr8GtTI3QFY+Zp5sloNpW0x8aCwMMUaZWku/7VHEQfE3NYw4iYcPD4Wdvk2wTHuJdghRIdH9wYbDRFxlu9yjtHK63aMmGNtbQjqYsuJSS6jmQehX0/plBsIhXn5FCKYAC6OQogew4hccDamE4VIGoAE0JDRSxlr39Giv8RwRSFQ/V0plgUDFLht57/f8qDLFX8g61Jco3gtN8BakkIRyNNAKUSswGfU+SOhfU32qod5B8LWHMdK3cm8NAZGypP30h7Z2or l8G50bMI 17/jeSjHFKSMM9Gw51t0K7eepWh2YucqYjEktdY4t8ZGegZMEWrxs810EByO/QFc+Vwg93pRZmbeV8gNR0ZfV8c5xlmU+bb2gsM/NM7e2bxp7vdVXdCt9ZMcF5drndlfVCoI4vEVbgWHnWf8vGWfVjkpxuMDw4Eof0dVVrXzd46RHulqfE8orBSJEFogg2v6fUIXSGOJ5bzr7vQKke57zD6ubjDmzpRxSMcyOFSUhY11BmzcLBZc76DR33L9BsG2XFafIeaALx5QYqHq2L4pbhqt8J/MTJuN+BXsHpiE962D+hun7dLURXGzg2B7jr+vEnAirUPngE80IALnWCeX4tv2gq3CokecdjVXQcfvzuIPf72yXQf6GMzvCk+RioO6LGcGY6+zPJBmlI1kkzFKq457oLiwrk79izHPac7LYiEKodU3gcgf8dr91FB0PgPASWKeIpJC55yoooDQwxy3q1qOY4y80aX13hPwTaWed1ocIrPO41Xf05AxRkewXsAeCjIrXp5tO1LoNllAVGwGvIuJhCYYP3VYEhPi5qFCpr6a+Y0Qz3pUlLO+s4ynLYrA+66qtnugql9bbjMFWiH6oHsyIPGPU0yoQJdWDqFzHiICsrsVyQkqb1zW8bjLCsiUhQ+2UCBvGp0/CzaEB60/iMJ03Mw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000002, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Sean Christopherson Refactor the pte_list and rmap code to always read and write rmap_head->val exactly once, e.g. by collecting changes in a local variable and then propagating those changes back to rmap_head->val as appropriate. This will allow implementing a per-rmap rwlock (of sorts) by adding a LOCKED bit into the rmap value alongside the MANY bit. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 83 +++++++++++++++++++++++++----------------- 1 file changed, 50 insertions(+), 33 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index b4e543bdf3f0..17de470f542c 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -920,21 +920,24 @@ static struct kvm_memory_slot *gfn_to_memslot_dirty_bitmap(struct kvm_vcpu *vcpu static int pte_list_add(struct kvm_mmu_memory_cache *cache, u64 *spte, struct kvm_rmap_head *rmap_head) { + unsigned long old_val, new_val; struct pte_list_desc *desc; int count = 0; - if (!rmap_head->val) { - rmap_head->val = (unsigned long)spte; - } else if (!(rmap_head->val & KVM_RMAP_MANY)) { + old_val = rmap_head->val; + + if (!old_val) { + new_val = (unsigned long)spte; + } else if (!(old_val & KVM_RMAP_MANY)) { desc = kvm_mmu_memory_cache_alloc(cache); - desc->sptes[0] = (u64 *)rmap_head->val; + desc->sptes[0] = (u64 *)old_val; desc->sptes[1] = spte; desc->spte_count = 2; desc->tail_count = 0; - rmap_head->val = (unsigned long)desc | KVM_RMAP_MANY; + new_val = (unsigned long)desc | KVM_RMAP_MANY; ++count; } else { - desc = (struct pte_list_desc *)(rmap_head->val & ~KVM_RMAP_MANY); + desc = (struct pte_list_desc *)(old_val & ~KVM_RMAP_MANY); count = desc->tail_count + desc->spte_count; /* @@ -943,21 +946,25 @@ static int pte_list_add(struct kvm_mmu_memory_cache *cache, u64 *spte, */ if (desc->spte_count == PTE_LIST_EXT) { desc = kvm_mmu_memory_cache_alloc(cache); - desc->more = (struct pte_list_desc *)(rmap_head->val & ~KVM_RMAP_MANY); + desc->more = (struct pte_list_desc *)(old_val & ~KVM_RMAP_MANY); desc->spte_count = 0; desc->tail_count = count; - rmap_head->val = (unsigned long)desc | KVM_RMAP_MANY; + new_val = (unsigned long)desc | KVM_RMAP_MANY; + } else { + new_val = old_val; } desc->sptes[desc->spte_count++] = spte; } + + rmap_head->val = new_val; + return count; } -static void pte_list_desc_remove_entry(struct kvm *kvm, - struct kvm_rmap_head *rmap_head, +static void pte_list_desc_remove_entry(struct kvm *kvm, unsigned long *rmap_val, struct pte_list_desc *desc, int i) { - struct pte_list_desc *head_desc = (struct pte_list_desc *)(rmap_head->val & ~KVM_RMAP_MANY); + struct pte_list_desc *head_desc = (struct pte_list_desc *)(*rmap_val & ~KVM_RMAP_MANY); int j = head_desc->spte_count - 1; /* @@ -984,9 +991,9 @@ static void pte_list_desc_remove_entry(struct kvm *kvm, * head at the next descriptor, i.e. the new head. */ if (!head_desc->more) - rmap_head->val = 0; + *rmap_val = 0; else - rmap_head->val = (unsigned long)head_desc->more | KVM_RMAP_MANY; + *rmap_val = (unsigned long)head_desc->more | KVM_RMAP_MANY; mmu_free_pte_list_desc(head_desc); } @@ -994,24 +1001,26 @@ static void pte_list_remove(struct kvm *kvm, u64 *spte, struct kvm_rmap_head *rmap_head) { struct pte_list_desc *desc; + unsigned long rmap_val; int i; - if (KVM_BUG_ON_DATA_CORRUPTION(!rmap_head->val, kvm)) - return; + rmap_val = rmap_head->val; + if (KVM_BUG_ON_DATA_CORRUPTION(!rmap_val, kvm)) + goto out; - if (!(rmap_head->val & KVM_RMAP_MANY)) { - if (KVM_BUG_ON_DATA_CORRUPTION((u64 *)rmap_head->val != spte, kvm)) - return; + if (!(rmap_val & KVM_RMAP_MANY)) { + if (KVM_BUG_ON_DATA_CORRUPTION((u64 *)rmap_val != spte, kvm)) + goto out; - rmap_head->val = 0; + rmap_val = 0; } else { - desc = (struct pte_list_desc *)(rmap_head->val & ~KVM_RMAP_MANY); + desc = (struct pte_list_desc *)(rmap_val & ~KVM_RMAP_MANY); while (desc) { for (i = 0; i < desc->spte_count; ++i) { if (desc->sptes[i] == spte) { - pte_list_desc_remove_entry(kvm, rmap_head, + pte_list_desc_remove_entry(kvm, &rmap_val, desc, i); - return; + goto out; } } desc = desc->more; @@ -1019,6 +1028,9 @@ static void pte_list_remove(struct kvm *kvm, u64 *spte, KVM_BUG_ON_DATA_CORRUPTION(true, kvm); } + +out: + rmap_head->val = rmap_val; } static void kvm_zap_one_rmap_spte(struct kvm *kvm, @@ -1033,17 +1045,19 @@ static bool kvm_zap_all_rmap_sptes(struct kvm *kvm, struct kvm_rmap_head *rmap_head) { struct pte_list_desc *desc, *next; + unsigned long rmap_val; int i; - if (!rmap_head->val) + rmap_val = rmap_head->val; + if (!rmap_val) return false; - if (!(rmap_head->val & KVM_RMAP_MANY)) { - mmu_spte_clear_track_bits(kvm, (u64 *)rmap_head->val); + if (!(rmap_val & KVM_RMAP_MANY)) { + mmu_spte_clear_track_bits(kvm, (u64 *)rmap_val); goto out; } - desc = (struct pte_list_desc *)(rmap_head->val & ~KVM_RMAP_MANY); + desc = (struct pte_list_desc *)(rmap_val & ~KVM_RMAP_MANY); for (; desc; desc = next) { for (i = 0; i < desc->spte_count; i++) @@ -1059,14 +1073,15 @@ static bool kvm_zap_all_rmap_sptes(struct kvm *kvm, unsigned int pte_list_count(struct kvm_rmap_head *rmap_head) { + unsigned long rmap_val = rmap_head->val; struct pte_list_desc *desc; - if (!rmap_head->val) + if (!rmap_val) return 0; - else if (!(rmap_head->val & KVM_RMAP_MANY)) + else if (!(rmap_val & KVM_RMAP_MANY)) return 1; - desc = (struct pte_list_desc *)(rmap_head->val & ~KVM_RMAP_MANY); + desc = (struct pte_list_desc *)(rmap_val & ~KVM_RMAP_MANY); return desc->tail_count + desc->spte_count; } @@ -1109,6 +1124,7 @@ static void rmap_remove(struct kvm *kvm, u64 *spte) */ struct rmap_iterator { /* private fields */ + struct rmap_head *head; struct pte_list_desc *desc; /* holds the sptep if not NULL */ int pos; /* index of the sptep */ }; @@ -1123,18 +1139,19 @@ struct rmap_iterator { static u64 *rmap_get_first(struct kvm_rmap_head *rmap_head, struct rmap_iterator *iter) { + unsigned long rmap_val = rmap_head->val; u64 *sptep; - if (!rmap_head->val) + if (!rmap_val) return NULL; - if (!(rmap_head->val & KVM_RMAP_MANY)) { + if (!(rmap_val & KVM_RMAP_MANY)) { iter->desc = NULL; - sptep = (u64 *)rmap_head->val; + sptep = (u64 *)rmap_val; goto out; } - iter->desc = (struct pte_list_desc *)(rmap_head->val & ~KVM_RMAP_MANY); + iter->desc = (struct pte_list_desc *)(rmap_val & ~KVM_RMAP_MANY); iter->pos = 0; sptep = iter->desc->sptes[iter->pos]; out: From patchwork Thu Sep 26 01:34:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13812683 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4A13DCCF9EB for ; Thu, 26 Sep 2024 01:35:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 727FE6B00C0; Wed, 25 Sep 2024 21:35:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6D4DD6B00C1; Wed, 25 Sep 2024 21:35:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5287C6B00C2; Wed, 25 Sep 2024 21:35:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 30EF26B00C0 for ; Wed, 25 Sep 2024 21:35:28 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id A11091C4E7D for ; Thu, 26 Sep 2024 01:35:27 +0000 (UTC) X-FDA: 82605171894.03.1D852B7 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf13.hostedemail.com (Postfix) with ESMTP id CE2722000B for ; Thu, 26 Sep 2024 01:35:25 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=kH+exXd0; spf=pass (imf13.hostedemail.com: domain of 3XLr0ZgoKCOEMWKRXJKWRQJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--jthoughton.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3XLr0ZgoKCOEMWKRXJKWRQJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1727314464; a=rsa-sha256; cv=none; b=jalna1V7joubQ8RfsVgA9ZS8/d2/oA/Kgt401hiSMPzMzFDFsbL95r+TmRYpY9Y44najC9 +Hiax1Mt+e5+CpnLSZK8En+F9aYJXA4v38AJKWYQRKDR9OmYaQrFv480/9EDA5rTJ6sbaI AUh3Z5ojLj1gLUsrMBEgE8/lCGDAsmQ= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=kH+exXd0; spf=pass (imf13.hostedemail.com: domain of 3XLr0ZgoKCOEMWKRXJKWRQJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--jthoughton.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3XLr0ZgoKCOEMWKRXJKWRQJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1727314464; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PEKVE3jXTsgIQ3oSBpZASsaX7oLzK9gVxgGb+Q0WSSI=; b=bjZQ+3xz/WyooD5BkAkIL46vzoDSPTsGZyQiwdDW3Wx4Zd2MEMBp64Igfwmw5EJxDqJyL/ tz0CAcgmiZeAQJQID4gPxXUaGOl65dKnyyFON+tpMR1WKeqjgOzkuW4lhAAVskuggHIALc l4GQBRosfyjOQxfbW0AatXuMrYqAlSU= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-e1159159528so2256784276.1 for ; Wed, 25 Sep 2024 18:35:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1727314525; x=1727919325; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=PEKVE3jXTsgIQ3oSBpZASsaX7oLzK9gVxgGb+Q0WSSI=; b=kH+exXd0hqAmT9LwnyqvOsdAEOt+ugiirRvFpIvfpfgFpCLGFUGJu4g+1Tw95R8ge0 ycIP69U1BzeSXJeW+GxcQ9gtX67PApNiHY8te52ZPneDNxUl6bR24GZnTJpBra9gmX8C AdNP4crlzRpWMM3Lomp4ldFd+SaBnjaTBtz5Q5WFcWeSURCJzWSFLg0JyS/GybkayMfl ECCw77XvqQ779Ot07qvhA+1iSFMu1TBqkyE3Zl4xRUyFrZMv1YjAyOSeoxqrWANJ7r0b 27yHjWvhV9/+kygXrztdVzZIno3mARCWUJ9KLPz4Artb+mLKlIqa9nlIHo+Imqm/kg0f yTBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727314525; x=1727919325; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=PEKVE3jXTsgIQ3oSBpZASsaX7oLzK9gVxgGb+Q0WSSI=; b=Ob0DdVx20LpduIbvHXFD5NKgOC8kz2cI2ZbsGYaPYOmku8pt+l0bm/gYeHxNiSKpyT H9m5ELAUOPB5I3zKdVdatZalhFjakFrERqaBlbl4JrpsXQaX09A1SDcjmYcBrtYDAtr6 6ZwJCYYVYQ/tH+ZMhM7oG1x7WJ1C0nwaYk6UoShbIQKF1LpS0taH6tyCXxuhJ8jWMkRB xhYgzfCNfCzvtYXkTkmSc45t/b1zCvokXrMAaITIpatFN2l+4weFQEkdfwJ2rQWtWHZx OEQjsUf52p1wyI1xVDy4hFEzOsDp7Qt7i78EMXSPbcPyM2XNcweFZxClNHOxuswvbReh 92YQ== X-Forwarded-Encrypted: i=1; AJvYcCWaszRiyb20RmclyFt/o4fhP1C413c7SQBWnm2vO1S/wxtujItLtimau2g9+oMwgse9r0k+PwoSew==@kvack.org X-Gm-Message-State: AOJu0Ywjqtthxc1xMcnNX8KtHUfzkb3rctiuOjY1gSiUIBt/x1fw62/9 acQ4N8QMleFc+72N4HH4+SrCC4+i7Z/IiVWNX/zPCjOkHmk4TFkidyHroklYkDzVngvZP0zitPZ 9eh0l3u+AKP5ajUvN9g== X-Google-Smtp-Source: AGHT+IF3/FYwkPT4bz8o9cvma+KKRWpatKrEO97ngoYUR5+iC7Jt4hs+IFCE9RrvcEVoXK4mDoAHACDMu6yNCXzE X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:13d:fb22:ac12:a84b]) (user=jthoughton job=sendgmr) by 2002:a5b:704:0:b0:e20:2da6:ed77 with SMTP id 3f1490d57ef6-e25ca95c803mr24214276.5.1727314524854; Wed, 25 Sep 2024 18:35:24 -0700 (PDT) Date: Thu, 26 Sep 2024 01:34:56 +0000 In-Reply-To: <20240926013506.860253-1-jthoughton@google.com> Mime-Version: 1.0 References: <20240926013506.860253-1-jthoughton@google.com> X-Mailer: git-send-email 2.46.0.792.g87dc391469-goog Message-ID: <20240926013506.860253-9-jthoughton@google.com> Subject: [PATCH v7 08/18] KVM: x86/mmu: Add infrastructure to allow walking rmaps outside of mmu_lock From: James Houghton To: Sean Christopherson , Paolo Bonzini Cc: Andrew Morton , David Matlack , David Rientjes , James Houghton , Jason Gunthorpe , Jonathan Corbet , Marc Zyngier , Oliver Upton , Wei Xu , Yu Zhao , Axel Rasmussen , kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org X-Stat-Signature: 7wb85f5ubr1yirjmgkqmi7whtssft1ma X-Rspamd-Queue-Id: CE2722000B X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1727314525-719377 X-HE-Meta: U2FsdGVkX18Sx0wusWGQnyhKqF6WQEO3m7ymqBc0/nV9diKJ/VoJ2FQ8uDS1wIa/olMjM/Qyfr5OVMF0pbUqfg1avcpX2DkDg5XmOA0wtMenj3eCRY2Vyn82dp7h93/K3kI+T3z/qoMvbEB6L04M4L2a0dxPw0AY3O0L8TpTqvTyXZLSfRrMuudCGsBerFL1esvLf3bMgvppJf0Le/97ro/NFwZ89aXSz3O3fZOjphoIGcrdq18/kT0DtFX0LxaItmE9Qg6k4h6ap7Q9VDsPFKcLHMEPpJTrakVbdlqEKE/ORBho+/vmNOyECXx2eODlQHuHoR91qxq/BKuDmJAvXRXxBBjALX+wklSw5mUy1Jrltw72ayndvZY/twCGSCHuDkC5lZtxpUUdR7tYga6VTG/sxMn7REHpwrl2PXcrYYNVMb7JzV4S3MgTywmfjEP4eZQZ7ARMmqvQxTiFXx2NXtnJxnDMbiRZlGKz+77aIITPR5a3ApOfG1UE14WP09/yYupk6vUSxb1mj0M6BBruPES+i5forqXYrWofB0900ORSs9b0I1BElzzQptxdDvkkO5/eOgN2xJG95mVEQyp5cuR/mG4fgNDTK48GtlXr4/6Oqs04ex19pKYSRW+zYwsc+fYolpfYPCRRdsoFot937H134vy1IbikroFS83ItcCqE0zqW13ihMct2X4LP/KBRwp84PbemQi2ZmiHP4L9w0iowewspvPLu4vnW/B49lRp5jPeCyy2aM42YIWoiNTFM2BxwQmF9k1rZxHIS+oYz51OEPr3Ha1Y+L91ChgYf0bRYEXf7QzviDRmADWS+EkhbMMHK6MRF2DqT3NX/uaG+9WlwDsaNt/LsS1mJwez+oG+wvDDGMVc+DsM6nNUtFmAkEcTWRqLIUnI2yN7V+Sl6fuLfdPqG2HSmdMOx/QIwPT6dLQAk7WSB6/nbLEBdIaLaHU3DQJD/qI8eWKPVD0H SjaKFFRm LceAydRI1EoJh3ecSrwttvgX3nMtKi3e+Zq55iVagmG70E4P3kPXbsxldtTeKIdgjgGBB/A+vukLGib30ASV4mWLNg9Nfm1VKrd+O3IzzrI73QjnGdpnfqgRbBNWVMm5tW9iuL/AJhOD8DO2dZ0njQ6Jt/UY/CwqKPtevwZRPW3bJC+pNxMHZbJUl7pqFxczsFs+G6c8Ac42aZXb54o79PBK6xTecIf0Utvp/CaIm9V7Hj21eciHf0APIKGT0CJhu1ITQZGxlN5qvdevozOl/CD+BC4hSlkaz3JeEL3DUOKgXxYCtzGY1jx44QX6Q6cKKRMpWbeKV04C2eVQOYTBYbmhIQGQylr4jvYr5ZI/0JAIhd+y0HtP+gLwdKgLsB01K/ROxMoKB9dLnNNJTl+QCvwC7eok1phZbPZg8vL05agVxiNUHF4fcTC295sGSjBzHwq3YUsogfwBQDtp/3NGtczy6J63jrnU0vetJvCKunikllEcXV3D5A2Epgt++xajEdVYxvQMAkoR7qtiWTOTGaJFER3LrZAHDnnn165w9GAH94UtdeF3Mm7Ub+AFcGiKJmUhNoMQdmiBhWyhfB2ilmtPn5rEEyhnN2LsmWHHVBth5Jzs/hgX8CNOOTg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Sean Christopherson Steal another bit from rmap entries (which are word aligned pointers, i.e. have 2 free bits on 32-bit KVM, and 3 free bits on 64-bit KVM), and use the bit to implement a *very* rudimentary per-rmap spinlock. The only anticipated usage of the lock outside of mmu_lock is for aging gfns, and collisions between aging and other MMU rmap operations are quite rare, e.g. unless userspace is being silly and aging a tiny range over and over in a tight loop, time between contention when aging an actively running VM is O(seconds). In short, a more sophisticated locking scheme shouldn't be necessary. Note, the lock only protects the rmap structure itself, SPTEs that are pointed at by a locked rmap can still be modified and zapped by another task (KVM drops/zaps SPTEs before deleting the rmap entries) Signed-off-by: Sean Christopherson Co-developed-by: James Houghton Signed-off-by: James Houghton --- arch/x86/include/asm/kvm_host.h | 3 +- arch/x86/kvm/mmu/mmu.c | 129 +++++++++++++++++++++++++++++--- 2 files changed, 120 insertions(+), 12 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index adc814bad4bb..d1164ca3e840 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -26,6 +26,7 @@ #include #include #include +#include #include #include @@ -401,7 +402,7 @@ union kvm_cpu_role { }; struct kvm_rmap_head { - unsigned long val; + atomic_long_t val; }; struct kvm_pio_request { diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 17de470f542c..79676798ba77 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -909,11 +909,117 @@ static struct kvm_memory_slot *gfn_to_memslot_dirty_bitmap(struct kvm_vcpu *vcpu * About rmap_head encoding: * * If the bit zero of rmap_head->val is clear, then it points to the only spte - * in this rmap chain. Otherwise, (rmap_head->val & ~1) points to a struct + * in this rmap chain. Otherwise, (rmap_head->val & ~3) points to a struct * pte_list_desc containing more mappings. */ #define KVM_RMAP_MANY BIT(0) +/* + * rmaps and PTE lists are mostly protected by mmu_lock (the shadow MMU always + * operates with mmu_lock held for write), but rmaps can be walked without + * holding mmu_lock so long as the caller can tolerate SPTEs in the rmap chain + * being zapped/dropped _while the rmap is locked_. + * + * Other than the KVM_RMAP_LOCKED flag, modifications to rmap entries must be + * done while holding mmu_lock for write. This allows a task walking rmaps + * without holding mmu_lock to concurrently walk the same entries as a task + * that is holding mmu_lock but _not_ the rmap lock. Neither task will modify + * the rmaps, thus the walks are stable. + * + * As alluded to above, SPTEs in rmaps are _not_ protected by KVM_RMAP_LOCKED, + * only the rmap chains themselves are protected. E.g. holding an rmap's lock + * ensures all "struct pte_list_desc" fields are stable. + */ +#define KVM_RMAP_LOCKED BIT(1) + +static unsigned long kvm_rmap_lock(struct kvm_rmap_head *rmap_head) +{ + unsigned long old_val, new_val; + + /* + * Elide the lock if the rmap is empty, as lockless walkers (read-only + * mode) don't need to (and can't) walk an empty rmap, nor can they add + * entries to the rmap. I.e. the only paths that process empty rmaps + * do so while holding mmu_lock for write, and are mutually exclusive. + */ + old_val = atomic_long_read(&rmap_head->val); + if (!old_val) + return 0; + + do { + /* + * If the rmap is locked, wait for it to be unlocked before + * trying acquire the lock, e.g. to bounce the cache line. + */ + while (old_val & KVM_RMAP_LOCKED) { + old_val = atomic_long_read(&rmap_head->val); + cpu_relax(); + } + + /* + * Recheck for an empty rmap, it may have been purged by the + * task that held the lock. + */ + if (!old_val) + return 0; + + new_val = old_val | KVM_RMAP_LOCKED; + /* + * Use try_cmpxchg_acquire to prevent reads and writes to the rmap + * from being reordered outside of the critical section created by + * __kvm_rmap_lock. + * + * Pairs with smp_store_release in kvm_rmap_unlock. + * + * For the !old_val case, no ordering is needed, as there is no rmap + * to walk. + */ + } while (!atomic_long_try_cmpxchg_acquire(&rmap_head->val, &old_val, new_val)); + + /* Return the old value, i.e. _without_ the LOCKED bit set. */ + return old_val; +} + +static void kvm_rmap_unlock(struct kvm_rmap_head *rmap_head, + unsigned long new_val) +{ + WARN_ON_ONCE(new_val & KVM_RMAP_LOCKED); + /* + * Ensure that all accesses to the rmap have completed + * before we actually unlock the rmap. + * + * Pairs with the atomic_long_try_cmpxchg_acquire in __kvm_rmap_lock. + */ + atomic_long_set_release(&rmap_head->val, new_val); +} + +static unsigned long kvm_rmap_get(struct kvm_rmap_head *rmap_head) +{ + return atomic_long_read(&rmap_head->val) & ~KVM_RMAP_LOCKED; +} + +/* + * If mmu_lock isn't held, rmaps can only locked in read-only mode. The actual + * locking is the same, but the caller is disallowed from modifying the rmap, + * and so the unlock flow is a nop if the rmap is/was empty. + */ +__maybe_unused +static unsigned long kvm_rmap_lock_readonly(struct kvm_rmap_head *rmap_head) +{ + return __kvm_rmap_lock(rmap_head); +} + +__maybe_unused +static void kvm_rmap_unlock_readonly(struct kvm_rmap_head *rmap_head, + unsigned long old_val) +{ + if (!old_val) + return; + + KVM_MMU_WARN_ON(old_val != kvm_rmap_get(rmap_head)); + atomic_long_set(&rmap_head->val, old_val); +} + /* * Returns the number of pointers in the rmap chain, not counting the new one. */ @@ -924,7 +1030,7 @@ static int pte_list_add(struct kvm_mmu_memory_cache *cache, u64 *spte, struct pte_list_desc *desc; int count = 0; - old_val = rmap_head->val; + old_val = kvm_rmap_lock(rmap_head); if (!old_val) { new_val = (unsigned long)spte; @@ -956,7 +1062,7 @@ static int pte_list_add(struct kvm_mmu_memory_cache *cache, u64 *spte, desc->sptes[desc->spte_count++] = spte; } - rmap_head->val = new_val; + kvm_rmap_unlock(rmap_head, new_val); return count; } @@ -1004,7 +1110,7 @@ static void pte_list_remove(struct kvm *kvm, u64 *spte, unsigned long rmap_val; int i; - rmap_val = rmap_head->val; + rmap_val = kvm_rmap_lock(rmap_head); if (KVM_BUG_ON_DATA_CORRUPTION(!rmap_val, kvm)) goto out; @@ -1030,7 +1136,7 @@ static void pte_list_remove(struct kvm *kvm, u64 *spte, } out: - rmap_head->val = rmap_val; + kvm_rmap_unlock(rmap_head, rmap_val); } static void kvm_zap_one_rmap_spte(struct kvm *kvm, @@ -1048,7 +1154,7 @@ static bool kvm_zap_all_rmap_sptes(struct kvm *kvm, unsigned long rmap_val; int i; - rmap_val = rmap_head->val; + rmap_val = kvm_rmap_lock(rmap_head); if (!rmap_val) return false; @@ -1067,13 +1173,13 @@ static bool kvm_zap_all_rmap_sptes(struct kvm *kvm, } out: /* rmap_head is meaningless now, remember to reset it */ - rmap_head->val = 0; + kvm_rmap_unlock(rmap_head, 0); return true; } unsigned int pte_list_count(struct kvm_rmap_head *rmap_head) { - unsigned long rmap_val = rmap_head->val; + unsigned long rmap_val = kvm_rmap_get(rmap_head); struct pte_list_desc *desc; if (!rmap_val) @@ -1139,7 +1245,7 @@ struct rmap_iterator { static u64 *rmap_get_first(struct kvm_rmap_head *rmap_head, struct rmap_iterator *iter) { - unsigned long rmap_val = rmap_head->val; + unsigned long rmap_val = kvm_rmap_get(rmap_head); u64 *sptep; if (!rmap_val) @@ -1483,7 +1589,7 @@ static void slot_rmap_walk_next(struct slot_rmap_walk_iterator *iterator) while (++iterator->rmap <= iterator->end_rmap) { iterator->gfn += KVM_PAGES_PER_HPAGE(iterator->level); - if (iterator->rmap->val) + if (atomic_long_read(&iterator->rmap->val)) return; } @@ -2513,7 +2619,8 @@ static int mmu_page_zap_pte(struct kvm *kvm, struct kvm_mmu_page *sp, * avoids retaining a large number of stale nested SPs. */ if (tdp_enabled && invalid_list && - child->role.guest_mode && !child->parent_ptes.val) + child->role.guest_mode && + !atomic_long_read(&child->parent_ptes.val)) return kvm_mmu_prepare_zap_page(kvm, child, invalid_list); } From patchwork Thu Sep 26 01:34:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13812684 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D554CCCF9EB for ; Thu, 26 Sep 2024 01:35:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6B3726B0095; Wed, 25 Sep 2024 21:35:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 639D46B009C; Wed, 25 Sep 2024 21:35:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 443366B00C1; Wed, 25 Sep 2024 21:35:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 132DE6B0095 for ; Wed, 25 Sep 2024 21:35:29 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id B7D48160720 for ; Thu, 26 Sep 2024 01:35:28 +0000 (UTC) X-FDA: 82605171936.10.765BB3B Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf01.hostedemail.com (Postfix) with ESMTP id E900C4000B for ; Thu, 26 Sep 2024 01:35:26 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=V9zIoD3q; spf=pass (imf01.hostedemail.com: domain of 3Xbr0ZgoKCOINXLSYKLXSRKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--jthoughton.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3Xbr0ZgoKCOINXLSYKLXSRKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1727314490; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MfYThc4u3zoEwhsYqTZzHZoRr9LypPNnm95GFpy1spA=; b=t4+BArPj7jURF9UinzPJMZ2K1hGNCqG8jV1amuBY65kJbDSpolmZJnOt/MuSkwu6xHUNmR H1kjLf3niFoHGNaNUSbVMrUGhN/tMORgWWI2xiNXahCPuY/pZUT/uZqL07VWmJCl1E5j9D xbNHHzSUVIQDAIKodRiIXPdDiFJIDpI= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=V9zIoD3q; spf=pass (imf01.hostedemail.com: domain of 3Xbr0ZgoKCOINXLSYKLXSRKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--jthoughton.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3Xbr0ZgoKCOINXLSYKLXSRKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1727314490; a=rsa-sha256; cv=none; b=rR4d17JVo+H9OSSGXQtAAP8Zd9ghgwrI8QjAa6Pet15Bkg4TXf1imy7ZB5yxU6oYoMNZZS BCt5gS/hgXlpFytoYt9wZEhiz1SwH+QHPgbv9UGPT9KaUOOByV+90k8n0bEKkUyq44eVMx 7laSPWXjuw9J2T4sw0ecB39bAhOce+0= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-6e230808388so6923337b3.0 for ; Wed, 25 Sep 2024 18:35:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1727314526; x=1727919326; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=MfYThc4u3zoEwhsYqTZzHZoRr9LypPNnm95GFpy1spA=; b=V9zIoD3qVfJ2VypVVbqwyLriwuz2JItn1BApSC2FV5YQeTqvQLI60oaiID3/6bkX3W R/c/Im2FOyEXJ1lfw+LvLKi2RX6bVlOOVltXO0QvrFA9+wUSu0V5hcYtAaDiwfkIQpkE zYtog2fGAcnB1Gx+28c7CRLO6xODt7ds3ojneTiyS/DxeFuxaWRr7ln6q9PVPOOgLpi+ 2Zs/NMzrTVK73wzP5dJCZOOPyhMm0SCoo7Q22+js3+ExBQWnAi8hOXH8LrLS5/0ie+lj 1QjjQcgST6vWDQ2bCjbL2fPuGRw3/r1nmZxfhIHcDWgTxJ+RP4setF5nXcV95rAQ2WPs g9Xw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727314526; x=1727919326; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=MfYThc4u3zoEwhsYqTZzHZoRr9LypPNnm95GFpy1spA=; b=n0cX3YTikvV8ngU77vWVMAlHIdty12nuvUYwNuwOMgyUdrxiTCm3b/97KKYj9hmA4N 28KTmLxw++XWmzNibQczxkki2iWuZ7xCqvRnHPa6nUu0PuuwS9NVLadKQvgRho9PEk1j xVxM1qG5iF1j9Z1Sv28GtHQBl+QC4HxjAvzGI4Cua2ZfACUAp3RY1FVRC9RGS1zvhqrQ /qkF6Q2LAgBRRCwFjCj51GyE3L+5s47itRbJN3oZkyAOlOjyZf0nk0eHtDDMubS1HUOg WIhYsYGdzgRX225Siu12+sK6vztLDBPhm/0xo7qfZ/OKqhYSVY+YWQm4Y9J0+KlWARG6 eLEw== X-Forwarded-Encrypted: i=1; AJvYcCWMIFKgk5Vssj3W48VAgpDrc9id4j/hd4L83zoTdW3+UbuzIq3h9vI8th6fSDksf7QMfWWRr6mwbA==@kvack.org X-Gm-Message-State: AOJu0YyA8Ub/+BJ2oWkJ6PHiaYvhNX9d60BA3P4mrTMGH5YO4jSRTIRC G2/6/BksIJs54QDkcLe/dJRolc8IUos9lm0nvpJ6wsnqSYzB4ayu2Afeb8kq6lf0sbMOH90804r CU9qgTGhuw3i0Kno01w== X-Google-Smtp-Source: AGHT+IFc2BTb/n/6/Pa4YiRnYJohGJI1GOQLPmGDGx2avUwnGUEvHFpkm5PUE1m++wXUG9sRJDYpg95gpRMXrHPb X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:13d:fb22:ac12:a84b]) (user=jthoughton job=sendgmr) by 2002:a05:690c:5086:b0:6e2:1b8c:39bf with SMTP id 00721157ae682-6e21d835b06mr289447b3.2.1727314525948; Wed, 25 Sep 2024 18:35:25 -0700 (PDT) Date: Thu, 26 Sep 2024 01:34:57 +0000 In-Reply-To: <20240926013506.860253-1-jthoughton@google.com> Mime-Version: 1.0 References: <20240926013506.860253-1-jthoughton@google.com> X-Mailer: git-send-email 2.46.0.792.g87dc391469-goog Message-ID: <20240926013506.860253-10-jthoughton@google.com> Subject: [PATCH v7 09/18] KVM: x86/mmu: Add support for lockless walks of rmap SPTEs From: James Houghton To: Sean Christopherson , Paolo Bonzini Cc: Andrew Morton , David Matlack , David Rientjes , James Houghton , Jason Gunthorpe , Jonathan Corbet , Marc Zyngier , Oliver Upton , Wei Xu , Yu Zhao , Axel Rasmussen , kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org X-Rspam-User: X-Stat-Signature: pnxunjxwm6hdan8ojbdm8i5wq4xrzhqj X-Rspamd-Queue-Id: E900C4000B X-Rspamd-Server: rspam11 X-HE-Tag: 1727314526-454407 X-HE-Meta: U2FsdGVkX18+YhkBwKvQJU5cVMbL97XFsTh6ieWjLJWC43BJNVrSrWwj8/6A6sEIvUdyWyvsJFtnO8aB93ocRaKUBR8MrkqTyfuTKn0pPuyRjcISuq+AI4SlADvD8TATpHF0UwveM5SCCxt4hEjhTJSxAxR2vzPWRm41BRRD35ZyixVbOuUqzxnR/BubzouupTwAhJS2fUVXeZuCwepQIEvwbhJI6RjWO7/6/2NA5rqWw0KLEzrMb2coOcIlY1v6t2UdqMkGP+AOTMvFzSExakHYk0QqBptf+DcXBtw8lad9i4IHUbY+lXh2VsFmWsHhHLCtCd527OrKbgziCPI/Sy6mZeDLyIoG8qudlAKso3+Zj3bj0FqR6J5gPcxQ5Lv8XICOa1WctUb+spaWTlZlkMXxBBH4nZ06iTU9KCY8uwIja0zaFHtrkcy11w3XcpwfASaDNtu4riFHffenDwpED9NUBp1Yv5yM5nzxM8bdBKkF0YCr41+XFFmkL7IioWWWrkML2Uk6ule4DRpeWEVGyEn797fijoBFy576+uLpL6KfruJhIfvPDft3Oq4tvQa5JTG2D8BHxRlExbm7lwXLIXGNhbL8I7kbt8XuCJ9A9B9a3Qnyrw71GQJGPqW5NONOBBlXWST8BPOhqSJmhRSNp4sv4BAO7Ddm3Z3hCloNGWev3qRqECzzB/oUUu9GKdyiZ1Ttyt7JSnEvNDDPMWhRMqR3XkFMRhJ6IcZTuqQ4BvoZmhqqlbTesAX9B5su2IoDi2d5xETkiZUosYqBBkjBPfW4c1AIJDM045CYXrTvD8X/uJBA/Yo9J5T9/NoEsflxNAnyOT/nAw6xLuUFaSXeUf5CU7vQ6mU1Ai13RDpbDB2/l0qqy3odsKLb/5234bHRktE71HyOIKTX6JGvev0yS23yAmyFNz2gyqrQMIimEen89AfSlRnDyp0hQ9syfCjxaI+mEsTOkBHa/C0EorS lkl9D6Dv caQQe5E3de+wWmrhu8SUNqQ/AxK3pNMKpUtc4LNknTh473+IzjtmOtlIYbl/ZzHtdJxaGvi30K56G/3Nift6aETH9Wno3mkimSrmWdn9zrVUGMJwTIP55DryGDz9lpPefOp8Sqq46MfHq8QEU2Z6tGrEGdMIKVTsMG7b4r3GBjBnHOJLXAKnxHnZG+s3K4+37aWDZrBmPHzbq7C34G7jzww7pecBwfTHvAPa3hOPTr2v6N6AXc1PXUV2PiKcfZ4ZM5SWZWAycHssR2qfEHreGtOKO0J+vq46hfJtSUo1kIJzJfdwtgQVN+ov4f6J0rHkr05YkXNRpM9h/Vcu9/toPQ7aUcW+VHj8o1675UqYOZ5VBavz0W50TwxePl75Nuf+7FRwGf2rb2+On5m2MD66Vk6DddkAgKFhF6CHXq8EM24vCnZMP6hb/QZEIqBSrWhEYRqszbtnV4LmjxEoAm0hMuEZaK0i6lsXOFctaGkfqZQBEA6PlkA4LX3QMMJlCwet59yOUrYDCH2ZgNQuGaCvmo+PEKQ7dewh5eUEQ/L1JdFGySOuU1b46K5f3PBdayHBDtP2aVVod+jxdymk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Sean Christopherson Add a lockless version of for_each_rmap_spte(), which is pretty much the same as the normal version, except that it doesn't BUG() the host if a non-present SPTE is encountered. When mmu_lock is held, it should be impossible for a different task to zap a SPTE, _and_ zapped SPTEs must be removed from their rmap chain prior to dropping mmu_lock. Thus, the normal walker BUG()s if a non-present SPTE is encountered as something is wildly broken. When walking rmaps without holding mmu_lock, the SPTEs pointed at by the rmap chain can be zapped/dropped, and so a lockless walk can observe a non-present SPTE if it runs concurrently with a different operation that is zapping SPTEs. Signed-off-by: Sean Christopherson [jthoughton: Added lockdep assertion for kvm_rmap_lock, synchronization fixup] Signed-off-by: James Houghton --- arch/x86/kvm/mmu/mmu.c | 75 +++++++++++++++++++++++------------------- 1 file changed, 42 insertions(+), 33 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 79676798ba77..72c682fa207a 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -932,7 +932,7 @@ static struct kvm_memory_slot *gfn_to_memslot_dirty_bitmap(struct kvm_vcpu *vcpu */ #define KVM_RMAP_LOCKED BIT(1) -static unsigned long kvm_rmap_lock(struct kvm_rmap_head *rmap_head) +static unsigned long __kvm_rmap_lock(struct kvm_rmap_head *rmap_head) { unsigned long old_val, new_val; @@ -976,14 +976,25 @@ static unsigned long kvm_rmap_lock(struct kvm_rmap_head *rmap_head) */ } while (!atomic_long_try_cmpxchg_acquire(&rmap_head->val, &old_val, new_val)); - /* Return the old value, i.e. _without_ the LOCKED bit set. */ + /* + * Return the old value, i.e. _without_ the LOCKED bit set. It's + * impossible for the return value to be 0 (see above), i.e. the read- + * only unlock flow can't get a false positive and fail to unlock. + */ return old_val; } +static unsigned long kvm_rmap_lock(struct kvm *kvm, + struct kvm_rmap_head *rmap_head) +{ + lockdep_assert_held_write(&kvm->mmu_lock); + return __kvm_rmap_lock(rmap_head); +} + static void kvm_rmap_unlock(struct kvm_rmap_head *rmap_head, unsigned long new_val) { - WARN_ON_ONCE(new_val & KVM_RMAP_LOCKED); + KVM_MMU_WARN_ON(new_val & KVM_RMAP_LOCKED); /* * Ensure that all accesses to the rmap have completed * before we actually unlock the rmap. @@ -1023,14 +1034,14 @@ static void kvm_rmap_unlock_readonly(struct kvm_rmap_head *rmap_head, /* * Returns the number of pointers in the rmap chain, not counting the new one. */ -static int pte_list_add(struct kvm_mmu_memory_cache *cache, u64 *spte, - struct kvm_rmap_head *rmap_head) +static int pte_list_add(struct kvm *kvm, struct kvm_mmu_memory_cache *cache, + u64 *spte, struct kvm_rmap_head *rmap_head) { unsigned long old_val, new_val; struct pte_list_desc *desc; int count = 0; - old_val = kvm_rmap_lock(rmap_head); + old_val = kvm_rmap_lock(kvm, rmap_head); if (!old_val) { new_val = (unsigned long)spte; @@ -1110,7 +1121,7 @@ static void pte_list_remove(struct kvm *kvm, u64 *spte, unsigned long rmap_val; int i; - rmap_val = kvm_rmap_lock(rmap_head); + rmap_val = kvm_rmap_lock(kvm, rmap_head); if (KVM_BUG_ON_DATA_CORRUPTION(!rmap_val, kvm)) goto out; @@ -1154,7 +1165,7 @@ static bool kvm_zap_all_rmap_sptes(struct kvm *kvm, unsigned long rmap_val; int i; - rmap_val = kvm_rmap_lock(rmap_head); + rmap_val = kvm_rmap_lock(kvm, rmap_head); if (!rmap_val) return false; @@ -1246,23 +1257,18 @@ static u64 *rmap_get_first(struct kvm_rmap_head *rmap_head, struct rmap_iterator *iter) { unsigned long rmap_val = kvm_rmap_get(rmap_head); - u64 *sptep; if (!rmap_val) return NULL; if (!(rmap_val & KVM_RMAP_MANY)) { iter->desc = NULL; - sptep = (u64 *)rmap_val; - goto out; + return (u64 *)rmap_val; } iter->desc = (struct pte_list_desc *)(rmap_val & ~KVM_RMAP_MANY); iter->pos = 0; - sptep = iter->desc->sptes[iter->pos]; -out: - BUG_ON(!is_shadow_present_pte(*sptep)); - return sptep; + return iter->desc->sptes[iter->pos]; } /* @@ -1272,14 +1278,11 @@ static u64 *rmap_get_first(struct kvm_rmap_head *rmap_head, */ static u64 *rmap_get_next(struct rmap_iterator *iter) { - u64 *sptep; - if (iter->desc) { if (iter->pos < PTE_LIST_EXT - 1) { ++iter->pos; - sptep = iter->desc->sptes[iter->pos]; - if (sptep) - goto out; + if (iter->desc->sptes[iter->pos]) + return iter->desc->sptes[iter->pos]; } iter->desc = iter->desc->more; @@ -1287,20 +1290,24 @@ static u64 *rmap_get_next(struct rmap_iterator *iter) if (iter->desc) { iter->pos = 0; /* desc->sptes[0] cannot be NULL */ - sptep = iter->desc->sptes[iter->pos]; - goto out; + return iter->desc->sptes[iter->pos]; } } return NULL; -out: - BUG_ON(!is_shadow_present_pte(*sptep)); - return sptep; } -#define for_each_rmap_spte(_rmap_head_, _iter_, _spte_) \ - for (_spte_ = rmap_get_first(_rmap_head_, _iter_); \ - _spte_; _spte_ = rmap_get_next(_iter_)) +#define __for_each_rmap_spte(_rmap_head_, _iter_, _sptep_) \ + for (_sptep_ = rmap_get_first(_rmap_head_, _iter_); \ + _sptep_; _sptep_ = rmap_get_next(_iter_)) + +#define for_each_rmap_spte(_rmap_head_, _iter_, _sptep_) \ + __for_each_rmap_spte(_rmap_head_, _iter_, _sptep_) \ + if (!WARN_ON_ONCE(!is_shadow_present_pte(*(_sptep_)))) \ + +#define for_each_rmap_spte_lockless(_rmap_head_, _iter_, _sptep_, _spte_) \ + __for_each_rmap_spte(_rmap_head_, _iter_, _sptep_) \ + if (is_shadow_present_pte(_spte_ = mmu_spte_get_lockless(sptep))) static void drop_spte(struct kvm *kvm, u64 *sptep) { @@ -1396,11 +1403,12 @@ static bool __rmap_clear_dirty(struct kvm *kvm, struct kvm_rmap_head *rmap_head, struct rmap_iterator iter; bool flush = false; - for_each_rmap_spte(rmap_head, &iter, sptep) + for_each_rmap_spte(rmap_head, &iter, sptep) { if (spte_ad_need_write_protect(*sptep)) flush |= spte_wrprot_for_clear_dirty(sptep); else flush |= spte_clear_dirty(sptep); + } return flush; } @@ -1710,7 +1718,7 @@ static void __rmap_add(struct kvm *kvm, kvm_update_page_stats(kvm, sp->role.level, 1); rmap_head = gfn_to_rmap(gfn, sp->role.level, slot); - rmap_count = pte_list_add(cache, spte, rmap_head); + rmap_count = pte_list_add(kvm, cache, spte, rmap_head); if (rmap_count > kvm->stat.max_mmu_rmap_size) kvm->stat.max_mmu_rmap_size = rmap_count; @@ -1859,13 +1867,14 @@ static unsigned kvm_page_table_hashfn(gfn_t gfn) return hash_64(gfn, KVM_MMU_HASH_SHIFT); } -static void mmu_page_add_parent_pte(struct kvm_mmu_memory_cache *cache, +static void mmu_page_add_parent_pte(struct kvm *kvm, + struct kvm_mmu_memory_cache *cache, struct kvm_mmu_page *sp, u64 *parent_pte) { if (!parent_pte) return; - pte_list_add(cache, parent_pte, &sp->parent_ptes); + pte_list_add(kvm, cache, parent_pte, &sp->parent_ptes); } static void mmu_page_remove_parent_pte(struct kvm *kvm, struct kvm_mmu_page *sp, @@ -2555,7 +2564,7 @@ static void __link_shadow_page(struct kvm *kvm, mmu_spte_set(sptep, spte); - mmu_page_add_parent_pte(cache, sp, sptep); + mmu_page_add_parent_pte(kvm, cache, sp, sptep); /* * The non-direct sub-pagetable must be updated before linking. For From patchwork Thu Sep 26 01:34:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13812685 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 62554CCF9EB for ; Thu, 26 Sep 2024 01:35:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 380076B009C; Wed, 25 Sep 2024 21:35:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 32A426B00C1; Wed, 25 Sep 2024 21:35:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1316D6B00C2; Wed, 25 Sep 2024 21:35:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id E27236B009C for ; Wed, 25 Sep 2024 21:35:29 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id A3D6740716 for ; Thu, 26 Sep 2024 01:35:29 +0000 (UTC) X-FDA: 82605171978.07.9E1214D Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf08.hostedemail.com (Postfix) with ESMTP id E96C416000F for ; Thu, 26 Sep 2024 01:35:27 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=cVk0OA25; spf=pass (imf08.hostedemail.com: domain of 3X7r0ZgoKCOQPZNUaMNZUTMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--jthoughton.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3X7r0ZgoKCOQPZNUaMNZUTMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1727314408; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/OnVwOE5c+4OTD3Iw61tcspb9ex9OyShqju36BWGlvg=; b=pqW9272RgntcOqKF2Z+M2hz/eZlc+YaUZvpQXY9OgPO6mm+/9li7zDTV+nMI1e0/It5GZD EC7K+U/qu9jQlZcYxmXRB06rFS4pdNhbO/NIAGc2R4oU7T4Z5Z4rNir3QOJAhSdZxSkAn0 A0KLeov+8mEbfADNCfih1W6q7fDsUH0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1727314408; a=rsa-sha256; cv=none; b=eIHlulFrIsFPFzIubKm+tZOhR/91O0jLo5lGyHcP4xQDR72FXJ7mKYuhJIakH/cRgA9gOc qUIerQ8nQVGElPgsB09Ic9Kq5nBOzxVx0ZLlYdeY1nr0DB1WACCukMOFCeMQtNP+D2nzm5 QC3xSTe4Vdds9xuj5VxQOZmJf+1273Q= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=cVk0OA25; spf=pass (imf08.hostedemail.com: domain of 3X7r0ZgoKCOQPZNUaMNZUTMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--jthoughton.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3X7r0ZgoKCOQPZNUaMNZUTMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-e25cfc1c8e3so103101276.1 for ; Wed, 25 Sep 2024 18:35:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1727314527; x=1727919327; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=/OnVwOE5c+4OTD3Iw61tcspb9ex9OyShqju36BWGlvg=; b=cVk0OA25UoyVMWClZdLqpHnp4Tw2wwD36Ha9qPXOsmt8yu17CTRtm8H6qvlOBfJIqp Oo9C2pnBiIiOS28uxUVJiigGrZqpqSV+nYVO+7V7VKb6n0wUWOSixEehc/QbFBrLCm0R uePGJ1455Yk/xhaEbbGIgrERRIN7RCRnFUa0Afx1IFj/NBMcY2iRGJCXUOhKD1Ee1otR FzAAE/6Xka//HwbpDxTLLIUsoN0bsOdh+4+KZ6QY/fiTCR0v6K9OjxtEY4Mx/PaJ+cY0 hqQ83+cUseQfGT/fwXyG/Fr0dajoY7sNTiw3o/ejPFUOx071D0/jX1ucll90GZCF9RMO wiBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727314527; x=1727919327; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/OnVwOE5c+4OTD3Iw61tcspb9ex9OyShqju36BWGlvg=; b=Q8QHPcKjj27wrhhYexcxBd/NIsrBsNW8ujE8VJSGI7+Lgqpu7mfg5rHNMNKG9vAlR9 0bzJEGBVMQKCsP7a47WtkLawLW1Jl6Fg3/n5kbpVrmAgheoyrv52sy4bMKYvqo2e7b4q +aKTRzQ/rIPZPj+j3+wxw5KyddadiX/HZN5TbJ38+Wwf4Vm8C2Uq76quJqHCErW8+ILk nu+m4yDeA2tOpSet2nXGXRceL7wUVUPdT7VVN/ib254AYL30CTVPYU2GIuOgCMyFsAC7 H9/1dfsBmlIVUXkULylu6NzudDrFGwLZuDVjciYVXN8cDZ/x56iGaorf05Qj+SIP7CK1 gqTQ== X-Forwarded-Encrypted: i=1; AJvYcCUKSpe3PLJ/gUVY5Au1npOnCHjjnHivwm02Dzd/knB34LGv0W4U3hnrHi1tek0RsBy1EfwQFIHrPQ==@kvack.org X-Gm-Message-State: AOJu0YxRiLSmSEw1UuLI9ffi3k9S8i1LLZDXQP8PXwfMYzapw244bdHL qRR7fowlatwPTzLjC9SCJN6sSCNepk4qfabrXu15YjfrXwYb28LsJlVhGS22rSmXJYgKX81Un5e Lu0YcSnLDpT7bGhwzOg== X-Google-Smtp-Source: AGHT+IFP6wbu5d5xFxibAPH1VEJb74N56TVg2vvH80n5yLYaLmdn9VnaHIq1SmzVQiXoW0zSRxLWmsPWaG4W/Pg1 X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:13d:fb22:ac12:a84b]) (user=jthoughton job=sendgmr) by 2002:a25:74cc:0:b0:e25:cced:3e3f with SMTP id 3f1490d57ef6-e25cced3f41mr11837276.4.1727314527042; Wed, 25 Sep 2024 18:35:27 -0700 (PDT) Date: Thu, 26 Sep 2024 01:34:58 +0000 In-Reply-To: <20240926013506.860253-1-jthoughton@google.com> Mime-Version: 1.0 References: <20240926013506.860253-1-jthoughton@google.com> X-Mailer: git-send-email 2.46.0.792.g87dc391469-goog Message-ID: <20240926013506.860253-11-jthoughton@google.com> Subject: [PATCH v7 10/18] KVM: x86/mmu: Support rmap walks without holding mmu_lock when aging gfns From: James Houghton To: Sean Christopherson , Paolo Bonzini Cc: Andrew Morton , David Matlack , David Rientjes , James Houghton , Jason Gunthorpe , Jonathan Corbet , Marc Zyngier , Oliver Upton , Wei Xu , Yu Zhao , Axel Rasmussen , kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org X-Rspamd-Queue-Id: E96C416000F X-Stat-Signature: 6jrdbs3idthp6ttf31iqrgdn4zxfi78x X-Rspamd-Server: rspam09 X-Rspam-User: X-HE-Tag: 1727314527-1532 X-HE-Meta: U2FsdGVkX1/w25qc44ASM5cgckTt4qDTHp6ukImIBhkF7VygPb5LKLvkXn5ZVSk32sjxVKP+O+bN68Z3aHwxV+8ssoT3G2rkzEYd3xsa2cJ+ot8/LwYAadMK2VKcZzcU8HMlksBZepKjR40qh2HiMG3aTbTVaSkd0lMTUkqVJhYHesKEw+6IUjtIqEJ9L24ezPNzwXTDsqDdvGChCVTJ11OH+rLF2IJ2zQCmg0CaDlPmS7MkWnObzKSOoOUV9yYzDxrQX+t3RpIoRp9cBqsbrY+0e4trxjPYrm4ywK9NVJ4OtBFF56S6zfqL/7oCAo++x/jBWRIgvCB9xAuyrerpjSBw+JOYGjaXjmvGNHmLFAZc9Ig0VexYO2yk1AUGxLf8rR/vOpABG8bwK04n1466cLE09Hf4fWP482ead1FkFN5KSuUeWCg+F88pP6sPtyQsb/VIPNKFK0k+1OSo2ZysqHykYJ+6RBVTrklyvvzbF6ujWaJjoln3/qtoat7sJ1oxlXEhtf9y2RtOamDlYiXEw1qmjDn9UZq5iJGr+b9YBSXqPhWjtBVASgltlaS+AhqBmMwFpWd4Ki2qcMP4XP8sPAV8aRzo21rlglL8XwhHwdG1Jnab+MfwPc5Ycc2avAo30U1o8q8qJmvxMmYcdFqfunI0/489vIUzU48NWa0Ay9Rk9c4p+fcUYfU8zoRQcGkzggZF7E7QzVACE04bNA7NNus8s48qSoWn5FcSvyoz5UQjcxGDq2AusDbdxssF8lHf1pmyyizqlODqzFnCgynuctpUXQ7FGJmOvYGDUPTZURtTn6QZUqgPrVT6eGmnQhUJYtVgtjdmr5pFEGMwdT5wR0K2kEvFf15Gp+JPV3k07BcEfSPkLen4ZqBIWzQVc5hGAFJVnlWmpHoetUueBWe05LeHhofjwO3QhfgE4r4sjQ6Ewb7JlJNVbiXK6PLd2ip86TjsYQf+g9rSdMdeCK3 6jJbejFn IeuB5dWzJTo8rVYNj7JLr8whlpCyV0skPeEWQpiTkDFHvmg+VCuf7m7+UaQT2sSvn7xeH2yQkDPfYHDLci5NUSVdAer41S6pcEw301gyM1NyGFrvD8qeUAVfihIBxaF7k7Pd//DJJHit7X9bI5+jglGiUwu7eJXfgtX2p+Hc0bvEK17MWhjyWDETfQN4VXFC39zsWdZPaJ3K8IPmXsAR8csHkTZ3WIlwURf0u4NCPMoaJFPVcQIsrLqUP107vBJ9bBZTGsWR6IhK/eV99GvUEofT03/hxEIgdbVKyvqgZToUkwM2ks6yUAiirOyeyql9Z5RtEbo5tgN45R/s/YOfJVwWoQpWbPrcQcbDasz9SQNCo7X2rH18OKn6gGCIF89bezdxUIYFSecM3kH8CejWTpeJp0pIhx2dreDrYhDKafjp1gr/yJcRnxDYF9ydOa7JHtranaBh7CQfBoxQrda4n3hcedk1kL/PtjOw11Hk/sN4N9J1J1uOMgrYLWpCBlMPxIg1WaFnZ65HkQQ8GDkhS9k3l41V2N3A3FjOb4X4Yneef5jPr/6faqTxLmeOWxqjKVJJzsfeKbgVim1WsafxEi7HSZvrECmLcQyPQgokqymxmVh6fh4WYfmQd2zj9HDWm/aHKky78hhMfEu9HunBnLrD90g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Sean Christopherson Because an L1 KVM can disable A/D bits for its L2, even if kvm_ad_enabled() in L0, we cannot always locklessly age, as aging requires marking non-A/D sptes for access tracking, which is not supported locklessly yet. We can always gather age information locklessly though. Signed-off-by: Sean Christopherson [jthoughton: Added changelog, adjusted conditional] Signed-off-by: James Houghton --- arch/x86/kvm/mmu/mmu.c | 66 +++++++++++++++++++++++++++++++++++++++--- 1 file changed, 62 insertions(+), 4 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 72c682fa207a..a63497bbcc61 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -1014,13 +1014,11 @@ static unsigned long kvm_rmap_get(struct kvm_rmap_head *rmap_head) * locking is the same, but the caller is disallowed from modifying the rmap, * and so the unlock flow is a nop if the rmap is/was empty. */ -__maybe_unused static unsigned long kvm_rmap_lock_readonly(struct kvm_rmap_head *rmap_head) { return __kvm_rmap_lock(rmap_head); } -__maybe_unused static void kvm_rmap_unlock_readonly(struct kvm_rmap_head *rmap_head, unsigned long old_val) { @@ -1736,8 +1734,53 @@ static void rmap_add(struct kvm_vcpu *vcpu, const struct kvm_memory_slot *slot, __rmap_add(vcpu->kvm, cache, slot, spte, gfn, access); } -static bool kvm_rmap_age_gfn_range(struct kvm *kvm, - struct kvm_gfn_range *range, bool test_only) +static bool kvm_rmap_age_gfn_range_lockless(struct kvm *kvm, + struct kvm_gfn_range *range, + bool test_only) +{ + struct kvm_rmap_head *rmap_head; + struct rmap_iterator iter; + unsigned long rmap_val; + bool young = false; + u64 *sptep; + gfn_t gfn; + int level; + u64 spte; + + for (level = PG_LEVEL_4K; level <= KVM_MAX_HUGEPAGE_LEVEL; level++) { + for (gfn = range->start; gfn < range->end; + gfn += KVM_PAGES_PER_HPAGE(level)) { + rmap_head = gfn_to_rmap(gfn, level, range->slot); + rmap_val = kvm_rmap_lock_readonly(rmap_head); + + for_each_rmap_spte_lockless(rmap_head, &iter, sptep, spte) { + if (!is_accessed_spte(spte)) + continue; + + if (test_only) { + kvm_rmap_unlock_readonly(rmap_head, rmap_val); + return true; + } + + /* + * Marking SPTEs for access tracking outside of + * mmu_lock is unsupported. Report the page as + * young, but otherwise leave it as-is. + */ + if (spte_ad_enabled(spte)) + clear_bit((ffs(shadow_accessed_mask) - 1), + (unsigned long *)sptep); + young = true; + } + + kvm_rmap_unlock_readonly(rmap_head, rmap_val); + } + } + return young; +} + +static bool __kvm_rmap_age_gfn_range(struct kvm *kvm, + struct kvm_gfn_range *range, bool test_only) { struct slot_rmap_walk_iterator iterator; struct rmap_iterator iter; @@ -1776,6 +1819,21 @@ static bool kvm_rmap_age_gfn_range(struct kvm *kvm, return young; } +static bool kvm_rmap_age_gfn_range(struct kvm *kvm, + struct kvm_gfn_range *range, bool test_only) +{ + /* + * We can always locklessly test if an spte is young. Because marking + * non-A/D sptes for access tracking without holding the mmu_lock is + * not currently supported, we cannot always locklessly clear. + */ + if (test_only) + return kvm_rmap_age_gfn_range_lockless(kvm, range, test_only); + + lockdep_assert_held_write(&kvm->mmu_lock); + return __kvm_rmap_age_gfn_range(kvm, range, test_only); +} + static bool kvm_has_shadow_mmu_sptes(struct kvm *kvm) { return !tdp_mmu_enabled || READ_ONCE(kvm->arch.indirect_shadow_pages); From patchwork Thu Sep 26 01:34:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13812686 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 718BCCCF9EB for ; Thu, 26 Sep 2024 01:35:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EED4B6B00C3; Wed, 25 Sep 2024 21:35:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E9DA56B00C4; Wed, 25 Sep 2024 21:35:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CF03C6B00C5; Wed, 25 Sep 2024 21:35:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id ACFE26B00C3 for ; Wed, 25 Sep 2024 21:35:31 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 5BD7A16071F for ; Thu, 26 Sep 2024 01:35:31 +0000 (UTC) X-FDA: 82605172062.15.4DBB289 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf09.hostedemail.com (Postfix) with ESMTP id 7E2FA140018 for ; Thu, 26 Sep 2024 01:35:29 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=nzrkjRsR; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf09.hostedemail.com: domain of 3YLr0ZgoKCOUQaOVbNOaVUNVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--jthoughton.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3YLr0ZgoKCOUQaOVbNOaVUNVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--jthoughton.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1727314493; a=rsa-sha256; cv=none; b=fNi17HC2k1MEylOUlVqLLD5Mw5b4xu/6RzQhPTxFtlJVFeGkQT+uuQs0tI7UXH0XXY9rhE BAlwLzqKZ97fd1/lvNiCmVeieyOGqHtheb36q1DIbRtJLSYpydaYAfxcwVsrERyE2YCUcq kfLCMEeIlqgp1HZIkpyil0z1xQetn0Q= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=nzrkjRsR; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf09.hostedemail.com: domain of 3YLr0ZgoKCOUQaOVbNOaVUNVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--jthoughton.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3YLr0ZgoKCOUQaOVbNOaVUNVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--jthoughton.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1727314493; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=V1Z3PBXE4Aegel3/riahgcLFAyLXvOzS5uEE7fXt5qw=; b=CuCPY/2fd+nK9t6pTTNpfguZSAVPNY39XlUCvCF2UHSRTS2IR+tOtTqxrVesnIqI2h1RgL UcT9OgxtaH1lKFc0xf9UbR4DLQ6MI07wL3oGnhUbvWYdnEgZw/F9P0new6na5O1pBzCTJH INuvWdcyloRP/fP8A97AKKuh2Ed/ikY= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-6e2146737ccso12364017b3.2 for ; Wed, 25 Sep 2024 18:35:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1727314528; x=1727919328; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=V1Z3PBXE4Aegel3/riahgcLFAyLXvOzS5uEE7fXt5qw=; b=nzrkjRsR5w7kb/dy2dqchO8tsWBXD4eLxRQpRv5bE9rVyQKWiRR3N9YtvyEaKsIKlj kodQ6G0aRLWIIdbfnpruTHGI4xzvuxq3GCTmXo9mWJhUB3NyZdhr429ebWZsmzT5QQLf WZaLeyaF9dxkee5bVUBLZQQwSbQQ3ypsN6CpvxV7hZ1b9dfN7mADbyeNG/GImYjhlYPk mrg1zPeCd7+XH4OpNE8WAhO3DoEyGt539ZmbTD5QMhGbePPVO9yBeaqp6DHKdT/0Lmq5 OLFdP8BIIDgpKEKVSIOjNtOsXfIlu+HAJmXkJtUrzAWWn/do99d+z3Oazh7F1EK/B/jh AiPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727314528; x=1727919328; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=V1Z3PBXE4Aegel3/riahgcLFAyLXvOzS5uEE7fXt5qw=; b=Y+o5xymL9m+JuNrK1C1CYQiYmsV4/ZantEKB9poEvbAqZl2VVyr1HdStuhLp9EUzF4 JaH0oo/OvqTTqQGveHl3OWsQA3wTuj67sIViIHtOYp9bfjGoov20B94iewlWDe7O3o5Q mlzQOyROwSGx9FJaB+z0rVmkPg1RYFZ/NUnIEYny8bO2/wuAwlGw4hz5uq/8Hvy6D0Fg HFGeGUrc/5b1WoDxCoU9LPrcOw7NtUxVZ0FmiXq+0Tw7l1ClSnglt/sa8J6ZI7ccT12o hwjsJxNPCiWIwBAkfWTV+gZ8OBg61f91a/EcKyG4R7edfdmLjmuwDbhzJueIdA6JzdEY Wuww== X-Forwarded-Encrypted: i=1; AJvYcCUjlJmLaT26d5ZvUzKNbMmh6eIQqxzwMVRZTqidxqnDoE0xlAVBBX6dPQR+S/ezSHCVAplYhy2qbg==@kvack.org X-Gm-Message-State: AOJu0Ywk3a+gPqV6qQ15bbxDu8Ll0AUyQPMKEtq70QoVlSDtK79CS4Lx 7CoxpG/tXAB7RU/TmPx/gF/LJDhc0VQL+wck2TtU23F1adZ3bW+X9JD4vcNS1eUd5aZMMgIrUIn TXZ60IIhL/hi8hvlzCw== X-Google-Smtp-Source: AGHT+IEpNqlcby3XCzPxmMXEa3yAsp88MrqHgIHgIiYwCpactoczoPP9Ca8XqrYS/AN0Db4KS1YJcToucGPf0cLH X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:13d:fb22:ac12:a84b]) (user=jthoughton job=sendgmr) by 2002:a05:690c:2892:b0:648:fc8a:cd23 with SMTP id 00721157ae682-6e21d6e1f34mr309987b3.2.1727314528595; Wed, 25 Sep 2024 18:35:28 -0700 (PDT) Date: Thu, 26 Sep 2024 01:34:59 +0000 In-Reply-To: <20240926013506.860253-1-jthoughton@google.com> Mime-Version: 1.0 References: <20240926013506.860253-1-jthoughton@google.com> X-Mailer: git-send-email 2.46.0.792.g87dc391469-goog Message-ID: <20240926013506.860253-12-jthoughton@google.com> Subject: [PATCH v7 11/18] mm: Add missing mmu_notifier_clear_young for !MMU_NOTIFIER From: James Houghton To: Sean Christopherson , Paolo Bonzini Cc: Andrew Morton , David Matlack , David Rientjes , James Houghton , Jason Gunthorpe , Jonathan Corbet , Marc Zyngier , Oliver Upton , Wei Xu , Yu Zhao , Axel Rasmussen , kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Jason Gunthorpe , David Hildenbrand X-Rspam-User: X-Rspamd-Queue-Id: 7E2FA140018 X-Rspamd-Server: rspam01 X-Stat-Signature: t615xxst5cjgpa77ju6huphoy31g18sy X-HE-Tag: 1727314529-778611 X-HE-Meta: U2FsdGVkX18nIEMruDUM6J3k7ZA9oFoDqM20AIdQFCwEQBOT4cr1woIPfFBxc6swgGy747a+22k2cScOMFNBSQYIE1+4hJ9Nh05HY+Rk2S4VIZaocPg0xX6GOljZwUBypwf2B7ow4g16EOqmmyuZwgwXBlt3ZtA+5zyf+YiH3ZvY9p1N7KqjeSaMG+rejyHoqb8DhVszNsSh2na8P0XW0xIqClFVRDDU5vUXSXtuIqVDnARICO6GUM81QuImRzI0Ms6EOprtGSSd4LyvozhqsRMNK0bs7t+MWkNQBixDrokPYbO/AV8wPBL562xDI6XbAWEGWLuFQPygTmQIZ8UP5EBT93wT1WREyrdhmDYQSThrgiXWeLYvpghQmGLRDvCCnztxQWAEUdDy+fi7T6+laJZYEIEGX/tQd1GZJeTAn0cow8QdDwBeCtnV2R7LstrgIFWwp/C3P8CBBwNxkYYdPp2dyAYkzCKHYlMzVGnBJCWiXACgEuKwKnr4udUCRTbgI65DIVFECfKLITDb+ZfrwVIiCRA4Fp9zH5nW4oU4ZA8BoEYKl2SK0f8wc9qfs4UaQXBhyK+GvIn6ZuRTPk46A+ufutoloXCuD/JPqo3/nNOWF/kmHvYLF0SZATeQdZIaY1E8GOfCvibhlrpWwMAb61WJaJIRoDGTt9siqa20nVKsRDSJVOByRLwq21QeQUMHFD3jMBGmklNPjPt+0Oqw/FUoXHrA/9PotK1pMx4ovh0Ii0IMG8ulikQrfFGJ+/S1/nm29KpFyy1DaSOS6f3IBK6rz5T+YpeZZyVWMImL70C9fZbTEiokWyeRuhzWhSvBLtd3qAt4DIGRLRRkSgO+Ps7d+jV0ILGZHoq5nfSnH8SeqiA/KnHZiqW/p++8H7RgIyrr7mf6JGnZZi2ZpDfCtPiAhqiXr0hI2hsQWJd929knQc5M1B9CzPC3qK8Kf6StJTyN+MH9RKiY19I6R1q EXFTJk/7 6bN/BJu80MIus8rhysS7mEsa/TAO4BgfGlZesWcrgVgFBVyqG7/nQSAdwkZBZRbORHo+gz6Qv8+rLw1MwlNOpC4Yc3dwI2TVV3ve+6hcI2nBzrdjc0BMpcTvHVahEs+wQD3+MvhuRIDYw5Ic1Bd7i2AwcvChYfbdLjD4hJ28w5gYpT0F85JnrXnpf9dtkyihtPXkKG7HLhEF8Xd3+4Fm3jsM2eM6xmYZYY9k2H9QPGf/Z09uVaanpbf9H6Qc6uH7WiT8pKsJNW06ihMg2X6jwcvedpUkbA8/0XhLeNNw2rJHfzT8k4HkxtRAKOmh3wsF/RulCjaxIopg+D8o49Q/Aompv3qexKUeO786SQjZRhXjawq6jANTJ2nJqsnfXNWqyhdYrZh2UaUmofsmjxz/HQVrA6Y15YEHXwnooUMwM5CRQz3qOTPVBIUKgMpuk2ZevGFRvz1weDEhlOtGuot/ypnEPdN64jq+1CfP3tM/msSiL0aQKxduoRnERuL9xaRLaQMb1aBr27U3Zg2inCrYgXW7k2x12NfBB5bd+Vq0W7aYPGlBnRIyJTIVAgw7D/6+pN4weRKxUTbSMWn9oDB5A0OC5cYNgpr9szYURR7620BaaTMiwCwEGQqa+KVa6553oX35vFEduCakNDRk0mJXFNIOw+8oQsJFFOUyqRKIdzRIDZZgGHIJT8Rtz8a0DHzPxngui7dEHXf4JerT7no3q/daukg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000002, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Remove the now unnecessary ifdef in mm/damon/vaddr.c as well. Signed-off-by: James Houghton Reviewed-by: Jason Gunthorpe Acked-by: David Hildenbrand --- include/linux/mmu_notifier.h | 7 +++++++ mm/damon/vaddr.c | 2 -- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/include/linux/mmu_notifier.h b/include/linux/mmu_notifier.h index d39ebb10caeb..e2dd57ca368b 100644 --- a/include/linux/mmu_notifier.h +++ b/include/linux/mmu_notifier.h @@ -606,6 +606,13 @@ static inline int mmu_notifier_clear_flush_young(struct mm_struct *mm, return 0; } +static inline int mmu_notifier_clear_young(struct mm_struct *mm, + unsigned long start, + unsigned long end) +{ + return 0; +} + static inline int mmu_notifier_test_young(struct mm_struct *mm, unsigned long address) { diff --git a/mm/damon/vaddr.c b/mm/damon/vaddr.c index 58829baf8b5d..2d5b53253bc2 100644 --- a/mm/damon/vaddr.c +++ b/mm/damon/vaddr.c @@ -351,11 +351,9 @@ static void damon_hugetlb_mkold(pte_t *pte, struct mm_struct *mm, set_huge_pte_at(mm, addr, pte, entry, psize); } -#ifdef CONFIG_MMU_NOTIFIER if (mmu_notifier_clear_young(mm, addr, addr + huge_page_size(hstate_vma(vma)))) referenced = true; -#endif /* CONFIG_MMU_NOTIFIER */ if (referenced) folio_set_young(folio); From patchwork Thu Sep 26 01:35:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13812688 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C5812CCFA10 for ; Thu, 26 Sep 2024 01:36:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E293E6B00C6; Wed, 25 Sep 2024 21:35:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DDABD6B00C7; Wed, 25 Sep 2024 21:35:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C0F766B00C8; Wed, 25 Sep 2024 21:35:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 9F4896B00C6 for ; Wed, 25 Sep 2024 21:35:34 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 574B5141193 for ; Thu, 26 Sep 2024 01:35:34 +0000 (UTC) X-FDA: 82605172188.21.7DED5DE Received: from mail-ua1-f73.google.com (mail-ua1-f73.google.com [209.85.222.73]) by imf04.hostedemail.com (Postfix) with ESMTP id 9288B40007 for ; Thu, 26 Sep 2024 01:35:32 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=MHZomqoV; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf04.hostedemail.com: domain of 3Ybr0ZgoKCOYRbPWcOPbWVOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--jthoughton.bounces.google.com designates 209.85.222.73 as permitted sender) smtp.mailfrom=3Ybr0ZgoKCOYRbPWcOPbWVOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--jthoughton.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1727314443; a=rsa-sha256; cv=none; b=eLUHZIrprDuW1rfFPEa7fMtAnpWVZq468w4fnzQKj/95Ctu98MniRT3oLfsG2nh8o6sktm jEHmK7DuG0XBIbTAygJOhGNY3MyaWq9EBuSzRihZD2nSPKdeEx+upjsJJn35+MUiMeLNha 0YoxL2Q4iD/xfYEBUs4Sarl+XR/iaLc= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=MHZomqoV; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf04.hostedemail.com: domain of 3Ybr0ZgoKCOYRbPWcOPbWVOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--jthoughton.bounces.google.com designates 209.85.222.73 as permitted sender) smtp.mailfrom=3Ybr0ZgoKCOYRbPWcOPbWVOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--jthoughton.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1727314443; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jwY5XxMaAH9vrNcOxKBlxaXWmYulPCq+P3CKqgHldo8=; b=sLuHDWBz4n494+yjXWoo+W3EMAyC6uWQF2pFi5joLvPAwv+P+4E2M2wKF3NUUbKNYK9prz jyENqpzAfZ8m5/dH5m6AjypXvlDPaBl9VKmJJVEwlOmXZzHyd0Crz+orIm+YA6hhF1ctN2 jk/HDL7XhaoN171PMEV4t+EszWfC0GU= Received: by mail-ua1-f73.google.com with SMTP id a1e0cc1a2514c-846df072591so165038241.1 for ; Wed, 25 Sep 2024 18:35:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1727314531; x=1727919331; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=jwY5XxMaAH9vrNcOxKBlxaXWmYulPCq+P3CKqgHldo8=; b=MHZomqoVNtdoenfGdp34L1jF7GSFrnX8hAqAr1VldwXfzvtk9MOdev+N3lpF++E9IB FcwmTLHhP3CjdR1anUh/Phy+SWx697Q1CHE3nxH2tTDpHYxaoy3p2gjOaCMyfVsTZZzx 1M+JbOs7OI0ZvIvVdx+TjW78P1aoh5djg5Qy+aLE2qrI/rZciSSVPMmCRhB7KGWCKoZD Wsm5AXeJK/UxPMCz3zSKM3Yyhgf7heeuAlQuXP1F4v5mekERFju1el09QOgfgH2zZabZ 4zSGELOaJn6tW+vEWOREADV+OCckWC2w4PxO7EnzlTa5v//YrGVmh/idicEDrJaJlxNy Mtaw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727314531; x=1727919331; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=jwY5XxMaAH9vrNcOxKBlxaXWmYulPCq+P3CKqgHldo8=; b=fHbs46pYept6Bxb2JzNNSNsTt7i0+mdB2ZLkQiZHuZN3DWraCUlnK9eQd1JP/JXsSS auNp3B4P2+zmFJCVNS1fIPp9Xj9j1a6OOIlkzs0o4SEZ/BhqUlr5nq5wms4HAOa1p0Kf U19s16QKsA3irRpgXPo3WjQXb545syiPNTIA8agY6OtV12/NNcfh+rgR9/oFmwuH9Kn6 J+qyS4dZxfIIXHXTnTcA0HK4cgvLuz9/BqOBx/0VD2DuGg+bU4Ck5IYyqbchXdTpu3hL +kAjq+KTrXpiRXDkdbLLYtLufwwwkFy/nKQneGDDIaSBW1gpYBjCnzSeKH122LCycNP5 bKLg== X-Forwarded-Encrypted: i=1; AJvYcCUm9d3Pt4VVpjdWRqSBy6OpnMJp27sPaEMN6ARYwnk8sVcqhP0sJJugjCV5ezkmBtfOrN7yrvtFGQ==@kvack.org X-Gm-Message-State: AOJu0YyBK2cKLYfm6AoG5yX5Q4TfH9CK/AbpA5RLMwSOql5LtbuSXuT0 BBRrB5hisbTrwa+PYHqy4qy7+rRKja6fgAzI2D2niMSbHM5VpeAUCt6Jw9LWkyPbsLTow+7wH/q kHbsVhirZ8Qpeu9tvCw== X-Google-Smtp-Source: AGHT+IFR+2qF3anIZYWl2APj1JzmMbbMXKjDESaaFvEMKsw92aYBHlJKDywpCEXIkyk1bOavtsxa/wF5vSbBnBSJ X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:13d:fb22:ac12:a84b]) (user=jthoughton job=sendgmr) by 2002:ab0:380d:0:b0:846:d5f9:2186 with SMTP id a1e0cc1a2514c-84e83c0a7fdmr10563241.2.1727314529698; Wed, 25 Sep 2024 18:35:29 -0700 (PDT) Date: Thu, 26 Sep 2024 01:35:00 +0000 In-Reply-To: <20240926013506.860253-1-jthoughton@google.com> Mime-Version: 1.0 References: <20240926013506.860253-1-jthoughton@google.com> X-Mailer: git-send-email 2.46.0.792.g87dc391469-goog Message-ID: <20240926013506.860253-13-jthoughton@google.com> Subject: [PATCH v7 12/18] mm: Add has_fast_aging to struct mmu_notifier From: James Houghton To: Sean Christopherson , Paolo Bonzini Cc: Andrew Morton , David Matlack , David Rientjes , James Houghton , Jason Gunthorpe , Jonathan Corbet , Marc Zyngier , Oliver Upton , Wei Xu , Yu Zhao , Axel Rasmussen , kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org X-Rspamd-Queue-Id: 9288B40007 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: hsui1z1386bcec9f6e3pggaiiy39bagh X-HE-Tag: 1727314532-697958 X-HE-Meta: U2FsdGVkX19lAuPwsAzT+FQ1hc2Q5wFTxxWeaJjTSTsnDHHNpbOVIH9ZuupXgS/TETUYAL08tpHjx6sMOKAyp1S78URPXLVnZopATohutBhQWMC02VDy7IrudkhUnuWim8b+gV9eE4MkGAxJH5i20ILbEimG1YlrAgD1rwhsWTsoEakIgZCrJ2tsWLsDMM8evWjSc7nwIWCuhMFpPMRpOS2+/OniXowfBXnPb8IMak0QwdN2BKPE/SoAYDLmjrGv5eJ9CvTHbZQy9RZseCiAg3vRtG8ysZXJl8TZTMb8ggLRrUS9iXh3CrvGAIzi6fDm5otNaPeHPO57lTUqWjz+44ZoXLCJq8Qcsom9D6gonPTf7DxGG5dOob5Xk+LVpCvjTXj0nsbIeEyqrFfbhVtO1XURBWAXUW9aZGTmiuZzv8eC4/nDLxFWO/0FXw+9ghygBZ24nsmxn6PyJvzVr0v29Va/S4uzuNva4+rKIUHS7T77XGVnlEM7fdCTjo50uVPiObLX3O2tUbpnibH1F+/R/nhpbiZf+D251HN5FaBBPQy7hrav4sDeLztxFOzbSdjNkQrdvWdCXQHPL3QqDwPfikJCOiUzpksdslzwBiiXx1/mfgIoytbPOzi4BV2MGFBQdD8j8wR8tHWE1/UshH09M2/2QV2VfqtM+jFbWjA7HG4pMyHhBb8s9xERy9+1YVwZEvCRl945I0j2IyFxhsTo59QQy804z5kWzZDja3+icPTODBef+hUYg8QRqUvIAIQBaKTxNGBxIzaliMNU/ggOpzU/jjVA4CRM/njFOI3OqqIDfFPCk1KJnkIhCj8IjdYkTt5qi8GV1lVtioF/2NvOKpsaV19QifgrEPFbD63tluK9xBbuv1W97GyFbxK3miUCHkpC8kTqLt9mv/qQ9Kw9w9tG8T+vk0HaoIYaEw9p5DV+ARzhLGm701ynlZnGOZ36Ics0o6SP2m4x4KLrof4 fh3MeGs4 1X6QOcAT8HmeWTFQUjmdtf5oqpuznTFxyhkcGA4zISMFg32O6Utc1D+C6hCb95WNViAsp07f53bdZvzEKzMwwVZynJaHZtpqXDuZoNrD0zsjqWdPhctvdZAQH78qbml1mtF6rdJkmZ92SdfgnAYORffOBkdVYGSJtvBxdWNOEa4fqxMia4UO7zM+6Ko9sZqbNuG7U0nSSVNqlskimtZA6psriZehHbJfDX2R9Qctkw8qBY20td789mZ1TIYH0gGzuDYKjvRHBvZc+zuWdCztS57E+JQjyLkBUIFXSGrMaBq5MBnwvzpErHTRZNbOgLdB7KWdl/yrE6tCHrGvvPQDg/4MzWujSpO2SZOlu28/seHNZPYoDz7qEmXawLrvRl1VYAEJFndGoydDrvkRu8/Bb0nWpAhIMI4BpzjkdhyUEpO5nFbU08YU0iSZnau6IUGXlOq+sX5ZGaAEDmeK5rlouR6laqUgh7YkcM6IRadooca9mgDyDGqbFIYRaby7TEouvXrY1Jv2Bg6WwT8H7WXJfO/OMjM3u29cD+tYCo3SazuEaOzHmhavy7BQqHHBw7vSTp2liMXTfXFpEEMalpwY6Vxfeuwx2XppNJCPe/pTcoVxD/eaBt0afOmmGYA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: has_fast_aging should be set by subscribers that non-trivially implement fast-only versions of both test_young() and clear_young(). Fast aging must be opt-in. For a subscriber that has not been enlightened with "fast aging", the test/clear_young() will behave identically whether or not fast_only is given. Have KVM always opt out for now; specific architectures can opt-in later. Given that KVM is the only test/clear_young() implementer, we could instead add an equivalent check in KVM, but doing so would incur an indirect function call every time, even if the notifier ends up being a no-op. Add mm_has_fast_young_notifiers() in case a caller wants to know if it should skip many calls to the mmu notifiers that may not be necessary (like MGLRU look-around). Signed-off-by: James Houghton --- include/linux/mmu_notifier.h | 14 ++++++++++++++ mm/mmu_notifier.c | 20 ++++++++++++++++++++ virt/kvm/kvm_main.c | 1 + 3 files changed, 35 insertions(+) diff --git a/include/linux/mmu_notifier.h b/include/linux/mmu_notifier.h index e2dd57ca368b..37643fa43687 100644 --- a/include/linux/mmu_notifier.h +++ b/include/linux/mmu_notifier.h @@ -231,6 +231,7 @@ struct mmu_notifier { struct mm_struct *mm; struct rcu_head rcu; unsigned int users; + bool has_fast_aging; }; /** @@ -383,6 +384,7 @@ extern int __mmu_notifier_clear_young(struct mm_struct *mm, unsigned long end); extern int __mmu_notifier_test_young(struct mm_struct *mm, unsigned long address); +extern bool __mm_has_fast_young_notifiers(struct mm_struct *mm); extern int __mmu_notifier_invalidate_range_start(struct mmu_notifier_range *r); extern void __mmu_notifier_invalidate_range_end(struct mmu_notifier_range *r); extern void __mmu_notifier_arch_invalidate_secondary_tlbs(struct mm_struct *mm, @@ -428,6 +430,13 @@ static inline int mmu_notifier_test_young(struct mm_struct *mm, return 0; } +static inline bool mm_has_fast_young_notifiers(struct mm_struct *mm) +{ + if (mm_has_notifiers(mm)) + return __mm_has_fast_young_notifiers(mm); + return 0; +} + static inline void mmu_notifier_invalidate_range_start(struct mmu_notifier_range *range) { @@ -619,6 +628,11 @@ static inline int mmu_notifier_test_young(struct mm_struct *mm, return 0; } +static inline bool mm_has_fast_young_notifiers(struct mm_struct *mm) +{ + return 0; +} + static inline void mmu_notifier_invalidate_range_start(struct mmu_notifier_range *range) { diff --git a/mm/mmu_notifier.c b/mm/mmu_notifier.c index 8982e6139d07..c405e5b072cf 100644 --- a/mm/mmu_notifier.c +++ b/mm/mmu_notifier.c @@ -382,6 +382,26 @@ int __mmu_notifier_clear_flush_young(struct mm_struct *mm, return young; } +bool __mm_has_fast_young_notifiers(struct mm_struct *mm) +{ + struct mmu_notifier *subscription; + bool has_fast_aging = false; + int id; + + id = srcu_read_lock(&srcu); + hlist_for_each_entry_rcu(subscription, + &mm->notifier_subscriptions->list, hlist, + srcu_read_lock_held(&srcu)) { + if (subscription->has_fast_aging) { + has_fast_aging = true; + break; + } + } + srcu_read_unlock(&srcu, id); + + return has_fast_aging; +} + int __mmu_notifier_clear_young(struct mm_struct *mm, unsigned long start, unsigned long end) diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 7d5b35cfc1ed..f6c369eccd2a 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -930,6 +930,7 @@ static const struct mmu_notifier_ops kvm_mmu_notifier_ops = { static int kvm_init_mmu_notifier(struct kvm *kvm) { kvm->mmu_notifier.ops = &kvm_mmu_notifier_ops; + kvm->mmu_notifier.has_fast_aging = false; return mmu_notifier_register(&kvm->mmu_notifier, current->mm); } From patchwork Thu Sep 26 01:35:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13812687 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1DC7DCCFA10 for ; Thu, 26 Sep 2024 01:35:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E301B6B00C4; Wed, 25 Sep 2024 21:35:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DB3F26B00C6; Wed, 25 Sep 2024 21:35:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C076A6B00C7; Wed, 25 Sep 2024 21:35:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 997486B00C4 for ; Wed, 25 Sep 2024 21:35:33 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 56AEA120741 for ; Thu, 26 Sep 2024 01:35:33 +0000 (UTC) X-FDA: 82605172146.12.762156F Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf12.hostedemail.com (Postfix) with ESMTP id 8E14140007 for ; Thu, 26 Sep 2024 01:35:31 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=TLY+tJJX; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf12.hostedemail.com: domain of 3Yrr0ZgoKCOcScQXdPQcXWPXXPUN.LXVURWdg-VVTeJLT.XaP@flex--jthoughton.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3Yrr0ZgoKCOcScQXdPQcXWPXXPUN.LXVURWdg-VVTeJLT.XaP@flex--jthoughton.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1727314442; a=rsa-sha256; cv=none; b=CwizTBKyZrk/uMB250kuI5XrbsLVbsYzzdcxaVQkp3SlLkge6OcapJGm38592dxXfVc/Y+ 5hfsI77MoS0r+mtXI6fqAQIKNM2/kJVI1LonBTBjwEbrulZkFAP+X1Q5jFhVYtti10bIUG OsdkQBu56Mfy/Z+maCkjOiKiQUAB8g0= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=TLY+tJJX; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf12.hostedemail.com: domain of 3Yrr0ZgoKCOcScQXdPQcXWPXXPUN.LXVURWdg-VVTeJLT.XaP@flex--jthoughton.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3Yrr0ZgoKCOcScQXdPQcXWPXXPUN.LXVURWdg-VVTeJLT.XaP@flex--jthoughton.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1727314442; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ogisB+p0t7E+MEmwUL2EdeyMfss1r+UivrCLQXciZ0o=; b=XApiFHBFSdNUJWGjWmEoYPzz9tMjffNkKdanikkyi0FhVOJHRcBiUJHohASp4hsDOlmJPT PLVC7hLhE3H3U3dTJp1NvnFhdrnqjdYTYglvg05QI2YX690EYfdmwFyba2oppaAtRs8mue mLgt4Y7DJbrlnzNvtFPy1NRGfrJFW9U= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-6e2261adfdeso10933357b3.2 for ; Wed, 25 Sep 2024 18:35:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1727314531; x=1727919331; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ogisB+p0t7E+MEmwUL2EdeyMfss1r+UivrCLQXciZ0o=; b=TLY+tJJXdCjLfPM1nVWPL38hS+3ei/EuCl//2dG2IWRpeLdcB/TB56eKygsUKVB6/4 e5g6jLKo7GtWnAPpZ5gUNYuZvy73ID5U9pxz3jyFWHiSYNZRzQ88ApGOXSKy0nM1JZ5b VsXJqKswDnpEJTMSDcwcYDUN7ah322WrchZQJT8layaWX5bpqUBDqbGVSV6T/4yiGc3U JX/hT3WKYHESNG7lZdCQaQHGyyfTkQ0vDToqM7vUbgn4Mh+pZ2KcuinRpebWMGkgWlB0 NRokJzbNufl1NyetYBy2Rp82hPlotJ/bzJunrF3cd5nIS5wximSAeBygbsUYUr8q4oBA KPjg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727314531; x=1727919331; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ogisB+p0t7E+MEmwUL2EdeyMfss1r+UivrCLQXciZ0o=; b=MxK0IN1Aao6J+Hp7TkLlLpInkFeHMygBv+f1ByPrNeEmmguw2cU1opiGlqVEV60Hz3 2C7DupsaYmka+e9rduuj/KwCsFuhq7KOQH0q22WaAXb3EvG/TaErVd+eiqufugShR+8P +h43HMrpMkjEO+A5Fe6iTrM0tktIO629tObLNl9yaV1gDGz2X33xU0KyPN3uzO4U2l6C /xQ4kx63iL7cEeqBcFu/15wfiOi3voTj6sENrTVmo4Aamb6wDx4eT/1cgIhalOfw+Uph Baqa4XJgX7vXY/SV9sAlD6SBfCxr0yogexVLoy9cacvR6MZyqam4oh4+PNP22D9aG2/x FUew== X-Forwarded-Encrypted: i=1; AJvYcCVTcafgzcgkH19+STiKwMemE++cc/olLWWVFfQR2RXlX9QiCVOcXUOkBlLcpltE6VP/nKXEhwQJIw==@kvack.org X-Gm-Message-State: AOJu0YxnOrSLVYs9c5Vb7j0SjxijMR1d3pezVHVNPcAN3ynLHaipRqSE LguojKZ9dN1V4qRI3NpzEY/b43HauBxk9x6xvaPUqwahJlqNa5eqXAz/6+nkxsxd+NkkZXECMqn K7GjZXMMCWd+axGtfeg== X-Google-Smtp-Source: AGHT+IEFdzw0O48kZ1pdx1YoauWRvV0zu2R4nMdxsgoP5v/WIUZXOoAYIDJzZnEXzkuO0wTdBSaqUczWpwalNBYg X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:13d:fb22:ac12:a84b]) (user=jthoughton job=sendgmr) by 2002:a05:690c:5292:b0:62c:f976:a763 with SMTP id 00721157ae682-6e21d6c27c9mr136077b3.1.1727314530709; Wed, 25 Sep 2024 18:35:30 -0700 (PDT) Date: Thu, 26 Sep 2024 01:35:01 +0000 In-Reply-To: <20240926013506.860253-1-jthoughton@google.com> Mime-Version: 1.0 References: <20240926013506.860253-1-jthoughton@google.com> X-Mailer: git-send-email 2.46.0.792.g87dc391469-goog Message-ID: <20240926013506.860253-14-jthoughton@google.com> Subject: [PATCH v7 13/18] mm: Add fast_only bool to test_young and clear_young MMU notifiers From: James Houghton To: Sean Christopherson , Paolo Bonzini Cc: Andrew Morton , David Matlack , David Rientjes , James Houghton , Jason Gunthorpe , Jonathan Corbet , Marc Zyngier , Oliver Upton , Wei Xu , Yu Zhao , Axel Rasmussen , kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org X-Rspamd-Queue-Id: 8E14140007 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: pgw9eabmwwwpeui46wjgeqfzzdm3odw3 X-HE-Tag: 1727314531-233135 X-HE-Meta: U2FsdGVkX18m74Jc5/jujk/tBWOStq7F11/iOE2pvXyjOXfORUpIPXBaxyqNELNoVDILWVfOklEKQuoqQv1JjC4IiVp2I1GXkYq11cRgv18PxOjzgyGTE2FwqvudJDqjHzOcrwK+MeIdLEdss+vrbLst9wMoHZ1ShpQXqqAOTebCinoXBU2fFqbhvMoGFhqZCONse1lHqr9b9apbaj1TQlRWwG9u6KU7FhQlx9z1uWlL98YsKn6h+nw9JCQTN9MnaRIOMYEdRXt5JXF+iGNQrCWifhkQIygOuHl8I96s6VWdolTySHvOLtblDigK846YjYn0JixbgtLiVa3lX8tyCESVPVawEpMyZHaRi8qzXn+tfF5tO/CSzKNnmhWQuqoMBfTh/eb0cLn0b2g88hq+VFxnMa3rZ9+eENZ9g5pfcwDqIsmnq/tnLgXFwxyvJZU6WTVNjQ46VQu2vsZTEcg2rGG2lklQg90TDlTdYjC9mbS7izDEgwzFzst9b8/Er/ERAONJCxISX92hFvEeBM5g06sNJmjkyTlXosLy/TWhgax7ZvszOeE9gegDxyC+Ih39+Znt4STDTkgHOfE6EqlTsMM42oZ83HgLcTfX8IDvv5p3Zv4f8wEqmp+neayXlh794D1s5IddikZS5XgT4z7lJIKvxoafpH0LGQ3HQtZY2zv+raqUlem61vPufAOUG/7cpsiVq9a5o6PkaA8edpL9wSwhnEDRjG8mLJuT3JMWRA4E0UAYUb0fePVdDBI0rCpN5rDi4GaLkiun4y8T9n97nr7mGnV5Euz9FVtfG5m0N4kOjOlKafkhgHL/wQ5yvFBb0PPJh3U2qW0454eLWVMmE86emuceg5IbRpkKivKjJxzbgng/yQuZaMyqef0tKDC5E7wwHHK1lUZ4HcXUHXRhgeQUyPbkFBRlrIde1bFA7q3ffj2t54Gb1qvBl1IYzmWXCQ68LUv3IoZ9gKLJzzl RB47M8x/ FTmXYi6bAtN+IcICZJ/d2c0+YsAydIpvZJ1aOGlxFtir8MKg68/Gn5ILWozXjbWAmVYvkYmtP2jCZ1tExTlWJS7OhBZQu2UXrL5gAl5BNgnbqR8e34kXAFGGx6W84NDuZkZxmZYkYb7hXiD7p+u+trO8/tfycq/s5pjRcb5UBfVPBYvXHs+299GbJufpx4qx+bpuG7qbmSbw2sQRgXEa19RnJCpOHhxGn2uJtL5vPsss6u3WcOBq3P/N84HuMXDOAKn2Z137QwpXW/kv7sBygO3yAS28amx8+a/UvBR14WxkMDM6YvkjldFGSH2gl0wo4xv7oXywONa9WymKIquVlXUrocWkC9WVXhDC0iCd+lovnssIi5nWvCqlYVYapJ99JAkV1mLn5094frKG8zeES3dFMsxQ52KUYeSWpKf8M2lBXmPlfxbyvu/1VXnfvYa/o4+Mkq8fHtVEQFDrUlX52rIoEo6DBqaAkek6yCDGDzeod+IOf+DTWWZCjYuGjNLe4U5qJI0QYVKCkJcsEn5YeUch6NlDMqXbV2f+TrSmvBIAA344OQMkp3fIDx9AqP3wbQKVsS2WLtv2MGwdfi0H/BlJI+EGCjumNhTmPCr/10oIalT6S+D+KYFTn4w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: For implementers, the fast_only bool indicates that the age information needs to be harvested such that we do not slow down other MMU operations, and ideally that we are not ourselves slowed down by other MMU operations. Usually this means that the implementation should be lockless. Also add mmu_notifier_test_young_fast_only() and mmu_notifier_clear_young_fast_only() helpers to set fast_only for these notifiers. Signed-off-by: James Houghton --- include/linux/mmu_notifier.h | 61 ++++++++++++++++++++++++++++++++---- include/trace/events/kvm.h | 19 ++++++----- mm/mmu_notifier.c | 18 ++++++++--- virt/kvm/kvm_main.c | 12 ++++--- 4 files changed, 88 insertions(+), 22 deletions(-) diff --git a/include/linux/mmu_notifier.h b/include/linux/mmu_notifier.h index 37643fa43687..7c17e2871c66 100644 --- a/include/linux/mmu_notifier.h +++ b/include/linux/mmu_notifier.h @@ -106,21 +106,38 @@ struct mmu_notifier_ops { * clear_young is a lightweight version of clear_flush_young. Like the * latter, it is supposed to test-and-clear the young/accessed bitflag * in the secondary pte, but it may omit flushing the secondary tlb. + * + * The fast_only parameter indicates that this call should not block, + * and this function should not cause other MMU notifier calls to + * block. Usually this means that the implementation should be + * lockless. + * + * When called with fast_only, this notifier will be a no-op (and + * return that the range is NOT young), unless has_fast_aging is set + * on the struct mmu_notifier. + * + * When fast_only is true, if the implementer cannot determine that a + * range is young without blocking, it should return 0 (i.e., that + * the range is NOT young). */ int (*clear_young)(struct mmu_notifier *subscription, struct mm_struct *mm, unsigned long start, - unsigned long end); + unsigned long end, + bool fast_only); /* * test_young is called to check the young/accessed bitflag in * the secondary pte. This is used to know if the page is * frequently used without actually clearing the flag or tearing * down the secondary mapping on the page. + * + * The fast_only parameter has the same meaning as with clear_young. */ int (*test_young)(struct mmu_notifier *subscription, struct mm_struct *mm, - unsigned long address); + unsigned long address, + bool fast_only); /* * invalidate_range_start() and invalidate_range_end() must be @@ -381,9 +398,11 @@ extern int __mmu_notifier_clear_flush_young(struct mm_struct *mm, unsigned long end); extern int __mmu_notifier_clear_young(struct mm_struct *mm, unsigned long start, - unsigned long end); + unsigned long end, + bool fast_only); extern int __mmu_notifier_test_young(struct mm_struct *mm, - unsigned long address); + unsigned long address, + bool fast_only); extern bool __mm_has_fast_young_notifiers(struct mm_struct *mm); extern int __mmu_notifier_invalidate_range_start(struct mmu_notifier_range *r); extern void __mmu_notifier_invalidate_range_end(struct mmu_notifier_range *r); @@ -418,7 +437,16 @@ static inline int mmu_notifier_clear_young(struct mm_struct *mm, unsigned long end) { if (mm_has_notifiers(mm)) - return __mmu_notifier_clear_young(mm, start, end); + return __mmu_notifier_clear_young(mm, start, end, false); + return 0; +} + +static inline int mmu_notifier_clear_young_fast_only(struct mm_struct *mm, + unsigned long start, + unsigned long end) +{ + if (mm_has_notifiers(mm)) + return __mmu_notifier_clear_young(mm, start, end, true); return 0; } @@ -426,7 +454,15 @@ static inline int mmu_notifier_test_young(struct mm_struct *mm, unsigned long address) { if (mm_has_notifiers(mm)) - return __mmu_notifier_test_young(mm, address); + return __mmu_notifier_test_young(mm, address, false); + return 0; +} + +static inline int mmu_notifier_test_young_fast_only(struct mm_struct *mm, + unsigned long address) +{ + if (mm_has_notifiers(mm)) + return __mmu_notifier_test_young(mm, address, true); return 0; } @@ -622,12 +658,25 @@ static inline int mmu_notifier_clear_young(struct mm_struct *mm, return 0; } +static inline int mmu_notifier_clear_young_fast_only(struct mm_struct *mm, + unsigned long start, + unsigned long end) +{ + return 0; +} + static inline int mmu_notifier_test_young(struct mm_struct *mm, unsigned long address) { return 0; } +static inline int mmu_notifier_test_young_fast_only(struct mm_struct *mm, + unsigned long address) +{ + return 0; +} + static inline bool mm_has_fast_young_notifiers(struct mm_struct *mm) { return 0; diff --git a/include/trace/events/kvm.h b/include/trace/events/kvm.h index 74e40d5d4af4..6d9485cf3e51 100644 --- a/include/trace/events/kvm.h +++ b/include/trace/events/kvm.h @@ -457,36 +457,41 @@ TRACE_EVENT(kvm_unmap_hva_range, ); TRACE_EVENT(kvm_age_hva, - TP_PROTO(unsigned long start, unsigned long end), - TP_ARGS(start, end), + TP_PROTO(unsigned long start, unsigned long end, bool fast_only), + TP_ARGS(start, end, fast_only), TP_STRUCT__entry( __field( unsigned long, start ) __field( unsigned long, end ) + __field( bool, fast_only ) ), TP_fast_assign( __entry->start = start; __entry->end = end; + __entry->fast_only = fast_only; ), - TP_printk("mmu notifier age hva: %#016lx -- %#016lx", - __entry->start, __entry->end) + TP_printk("mmu notifier age hva: %#016lx -- %#016lx fast_only: %d", + __entry->start, __entry->end, __entry->fast_only) ); TRACE_EVENT(kvm_test_age_hva, - TP_PROTO(unsigned long hva), - TP_ARGS(hva), + TP_PROTO(unsigned long hva, bool fast_only), + TP_ARGS(hva, fast_only), TP_STRUCT__entry( __field( unsigned long, hva ) + __field( bool, fast_only ) ), TP_fast_assign( __entry->hva = hva; + __entry->fast_only = fast_only; ), - TP_printk("mmu notifier test age hva: %#016lx", __entry->hva) + TP_printk("mmu notifier test age hva: %#016lx fast_only: %d", + __entry->hva, __entry->fast_only) ); #endif /* _TRACE_KVM_MAIN_H */ diff --git a/mm/mmu_notifier.c b/mm/mmu_notifier.c index c405e5b072cf..f9ec810c8a1b 100644 --- a/mm/mmu_notifier.c +++ b/mm/mmu_notifier.c @@ -404,7 +404,8 @@ bool __mm_has_fast_young_notifiers(struct mm_struct *mm) int __mmu_notifier_clear_young(struct mm_struct *mm, unsigned long start, - unsigned long end) + unsigned long end, + bool fast_only) { struct mmu_notifier *subscription; int young = 0, id; @@ -413,9 +414,13 @@ int __mmu_notifier_clear_young(struct mm_struct *mm, hlist_for_each_entry_rcu(subscription, &mm->notifier_subscriptions->list, hlist, srcu_read_lock_held(&srcu)) { + if (fast_only && !subscription->has_fast_aging) + continue; + if (subscription->ops->clear_young) young |= subscription->ops->clear_young(subscription, - mm, start, end); + mm, start, end, + fast_only); } srcu_read_unlock(&srcu, id); @@ -423,7 +428,8 @@ int __mmu_notifier_clear_young(struct mm_struct *mm, } int __mmu_notifier_test_young(struct mm_struct *mm, - unsigned long address) + unsigned long address, + bool fast_only) { struct mmu_notifier *subscription; int young = 0, id; @@ -432,9 +438,13 @@ int __mmu_notifier_test_young(struct mm_struct *mm, hlist_for_each_entry_rcu(subscription, &mm->notifier_subscriptions->list, hlist, srcu_read_lock_held(&srcu)) { + if (fast_only && !subscription->has_fast_aging) + continue; + if (subscription->ops->test_young) { young = subscription->ops->test_young(subscription, mm, - address); + address, + fast_only); if (young) break; } diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index f6c369eccd2a..ec07caaed6b6 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -846,7 +846,7 @@ static int kvm_mmu_notifier_clear_flush_young(struct mmu_notifier *mn, IS_ENABLED(CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS), }; - trace_kvm_age_hva(start, end); + trace_kvm_age_hva(start, end, false); return kvm_handle_hva_range(kvm, &range).ret; } @@ -854,7 +854,8 @@ static int kvm_mmu_notifier_clear_flush_young(struct mmu_notifier *mn, static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn, struct mm_struct *mm, unsigned long start, - unsigned long end) + unsigned long end, + bool fast_only) { struct kvm *kvm = mmu_notifier_to_kvm(mn); const struct kvm_mmu_notifier_range range = { @@ -868,7 +869,7 @@ static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn, IS_ENABLED(CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS), }; - trace_kvm_age_hva(start, end); + trace_kvm_age_hva(start, end, fast_only); /* * Even though we do not flush TLB, this will still adversely @@ -888,7 +889,8 @@ static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn, static int kvm_mmu_notifier_test_young(struct mmu_notifier *mn, struct mm_struct *mm, - unsigned long address) + unsigned long address, + bool fast_only) { struct kvm *kvm = mmu_notifier_to_kvm(mn); const struct kvm_mmu_notifier_range range = { @@ -902,7 +904,7 @@ static int kvm_mmu_notifier_test_young(struct mmu_notifier *mn, IS_ENABLED(CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS), }; - trace_kvm_test_age_hva(address); + trace_kvm_test_age_hva(address, fast_only); return kvm_handle_hva_range(kvm, &range).ret; } From patchwork Thu Sep 26 01:35:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13812689 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E7268CCF9EB for ; Thu, 26 Sep 2024 01:36:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 79FC56B00C7; Wed, 25 Sep 2024 21:35:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 74CBE6B00C8; Wed, 25 Sep 2024 21:35:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5C3E06B00C9; Wed, 25 Sep 2024 21:35:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 376FE6B00C7 for ; Wed, 25 Sep 2024 21:35:35 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id E77041A10DD for ; Thu, 26 Sep 2024 01:35:34 +0000 (UTC) X-FDA: 82605172188.11.0D371AF Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf13.hostedemail.com (Postfix) with ESMTP id 2D05C20003 for ; Thu, 26 Sep 2024 01:35:33 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ldUT63r+; spf=pass (imf13.hostedemail.com: domain of 3ZLr0ZgoKCOkUeSZfRSeZYRZZRWP.NZXWTYfi-XXVgLNV.ZcR@flex--jthoughton.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3ZLr0ZgoKCOkUeSZfRSeZYRZZRWP.NZXWTYfi-XXVgLNV.ZcR@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1727314373; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=W5MO1wGLHyIriiPw0D1mbZgOsdCV5vF/ShV42LiObvw=; b=YSZpEWYd5HGcdZwerIUCa0/+VfWu2Ski1z55ppDP25g44kIQGY9GMC502Nh7B9qmqdi6Np drREPbWCfl9F5NTqVh0Cht7wsnb3egwDinfEuFAfqhb/BYnkFNCzS+coBn5awdP/jln7Fb mc4a/Weq1KEy9NJKtt140pjUzu63zUY= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ldUT63r+; spf=pass (imf13.hostedemail.com: domain of 3ZLr0ZgoKCOkUeSZfRSeZYRZZRWP.NZXWTYfi-XXVgLNV.ZcR@flex--jthoughton.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3ZLr0ZgoKCOkUeSZfRSeZYRZZRWP.NZXWTYfi-XXVgLNV.ZcR@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1727314373; a=rsa-sha256; cv=none; b=gcBA32vQ8lIhsYDRdd+oP2CFYO8DSbwWbt2msd3HbkWFhpRWQ2tkjxQln1yTKQBF4DxUNU 53VoYudfhmoun8qz/nNJUo1DIM273+9eAa2yrwK4gS7w2iTrd67Kms4EA3JbL/AgrXolOG Yo2TVP0muZ2XfbtHU9JYLs48MMltDSs= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-e24b43799e9so598141276.2 for ; Wed, 25 Sep 2024 18:35:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1727314532; x=1727919332; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=W5MO1wGLHyIriiPw0D1mbZgOsdCV5vF/ShV42LiObvw=; b=ldUT63r+UxBJpp73W0QiJWavW+BqHvAj3vizQ9RoeOHL8+H7W1USViOisgwdI2Pb6f QZhTXKSIVoSJ8dRLRZcB/uLSKUkAmVWZ5B1zAcWmiBbgqYAAGyKlxUR6+jyy+CS9Kv1q QrXa6QisAGWMH8rZLYaRsBMXaclt0+ZdpeMRRX3U4fDObQLt5C5NGkfLQR2eFREsZzcD KojaXg5dkHJ5qpOfjgRiJPQ0ysSIxd/eCLACx82JeEjR0X8irWMiF0K0/FVrY0XpeUdA C3AsrsRyV9q+Abh0Hi3EzLhDosuQ8ihASh6RNtnXIGdxExuzRr1edwLhxpb17EdgSo6P EE7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727314532; x=1727919332; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=W5MO1wGLHyIriiPw0D1mbZgOsdCV5vF/ShV42LiObvw=; b=iydJGj/BJwkh2ewWwKbz+FPmW2ZLCaP7yglpH6ahxQE1SoYaQlgQjulxnntOhuHCiq MxRyNF3p3t5gXEugtGo3X9ruABuXJmd8BF93DtCr7DZc0R9vBe7zvNhSiX5KBdCA+aFX D9VvTgMHA9ORyoQeosR0g+tPYgmeg86OhkzN1HBCLTLy73oLkmWSCmIwSMKFkYvOSSk1 Fq+Co6hyOcxTlgI5bQ9mYJALkpdBZnufTgNcojd6njsMDB3hSKvzTkdMpjo4WPFVK+qe FQWpav4QflVc2Q2zo97lWXMw4yHKKszISdSIrGyLkiL3Bl5kzZDgmoSPdeTBSdhKIE3Z 1oVQ== X-Forwarded-Encrypted: i=1; AJvYcCXKTJX1+d7c134lBfLpNNgWEk4IIjza8PA+ASGiRu3jS71/ZDa/GR1q+5Ajzxz4glnAUeizDH9zSw==@kvack.org X-Gm-Message-State: AOJu0YxBi6VJjrbrW6QlLkR1OIkhAuW0hxZ/HEBE1JMHfnp4i+jnvJFV 8MdAyvD3g8Vx6QOEZvXEobbhl0al5mIDS0YRfOyiSOwIA/WLvKTKljZSSCdsrGfTcXXK6rQoXGO q8UJdwytCPLWyWk8gxg== X-Google-Smtp-Source: AGHT+IHu6+UOTvIBD0ygLiAF0PUJIN5o89y1zOQ3DCuJr2JFhL/uFLj8FE3ghAKMMweGgziAQ1oo3bEK7MWZtMkm X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:13d:fb22:ac12:a84b]) (user=jthoughton job=sendgmr) by 2002:a25:bb48:0:b0:e03:53a4:1a7 with SMTP id 3f1490d57ef6-e24da1a380bmr81561276.10.1727314532143; Wed, 25 Sep 2024 18:35:32 -0700 (PDT) Date: Thu, 26 Sep 2024 01:35:02 +0000 In-Reply-To: <20240926013506.860253-1-jthoughton@google.com> Mime-Version: 1.0 References: <20240926013506.860253-1-jthoughton@google.com> X-Mailer: git-send-email 2.46.0.792.g87dc391469-goog Message-ID: <20240926013506.860253-15-jthoughton@google.com> Subject: [PATCH v7 14/18] KVM: Pass fast_only to kvm_{test_,}age_gfn From: James Houghton To: Sean Christopherson , Paolo Bonzini Cc: Andrew Morton , David Matlack , David Rientjes , James Houghton , Jason Gunthorpe , Jonathan Corbet , Marc Zyngier , Oliver Upton , Wei Xu , Yu Zhao , Axel Rasmussen , kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 2D05C20003 X-Stat-Signature: w7ktpw3fysden43ehybdb3y6wsi38pje X-Rspam-User: X-HE-Tag: 1727314533-936139 X-HE-Meta: U2FsdGVkX1/re63ugVKs5+lIUFTzK7b1XJZNLriN28llQcH0rAWyAk5IFKvzwchJ9CyyuffvJM5FvJlrDLU3hcIk7RFoobaOQm8RmxdgX5utOpbOBA3Qt+EEEupPWVyTk5fyu4sN7wOxHCmnRhqc0j4rfxe4+Y922B4GsI5AOzy/MHwXkxhZK3Cs8fYOPELBr8vtpaDIBeB6l4lc46tSC54/ESxyzwaqsrtbVATR46L1Kg5XUu8XsQOnhZrVn548eMtz2X/js0a6a/FuB7SlXd7hBTiReo6zqzL2z9U4DtP0SWKTB4ZkzocxcT6AKCLzuD9U2gPBu6y4xFBElgC3o2PKRGl9yxiWsGIISBL2IBTxZi/KMAbAhYFYkyGRMGiH95F5j/ElvheQJOGPwnVVHG6iW9Mv8mlFWArrwR4k/TrULZRPwhK736EKmx7GPMFrf66ZUHA6bDjcGfClkc3m6LecWfzePyPYtAUjMo5O1akoV5OySCajW2d0Hr6kIR6Klz39iQkq137bQge0J5DNHbfTLkQ2Xs1DRzewISK4SidI0GkMn/0ExFKPzpRCr3UbVVSfEMXHwm0t7YYAw21HqsREv6cTvZIEVaOjSEWenbnaNwlfe1jDql4q9mZhC0VVL0EwFRSzB5I3J2+xxu5fc1aLNnxyHYHfED79MT/HSF4naLmRq5zxOZ6zbKZlJq1M/uixo09qrgWlOsoC1MkY6hukfwj+rPCPr9apmcfVDxYDbOfIckK0JeK7GxBowLXsEs8Oshd25NpDzSYuj7skxBHV+ISsDX0fo+GMbuidRGwwEzrviczgnpn3ha99MkiBE7nP6RddkducN15xq2YdL1sxJrb/3tjn3BNxix62OpyVNK8ARoEEsAwgQNYV6M5t9ffJrrzmqaFli1DXuU/lmM+kFTWlux42P/5r/yrvL0DR8+FqT471x5wbQ+vFMQq9NbPq4rePhpnrxYl2bW9 sP82kLkM mIOfPSW0pSL3MmfpJjm1k1r1YaXDYSAYrvHBfGrJ9i71Yf61oji62yTPLgTTlnhK2GA9s3JWqtr7iLmilmpN/xWY2wtlhrllTV8kW3bcWaNxvBA16SmXexP3eCLx+Zxs+vxrrWQ4vCqml/nqcVwg4Vbm4wPVTfU1NZKKKqAlJ8WKmMVCUZawe9up38zBh89bgMZdDIqFqbwQTjfh89NqoF0HonwnJfAZ9tNzUxSfzzYpcPI2tbPNZvnM7tf8x+RRr83ZC5cm1hw80QHxvKHFAaEht3ufT9qHf7hIeU4Ji2ewtQ6XDXsGG4bcwp61PklpHPH5s4sN/otbU5raggKbasva/ssQPFpjWdTDqbhh2Fr090x/0YvR9PYOQcN8yguY6/9eK1Ej+nzXYn7a6PU3NVab9EAEkjvtVu/fjoBw4DA7GsVB/d/noijjzgcL4SiUsC++nMkOalaQjrtj4xG8rK4iwePqnQsc7eccC+HVv2rIVakmhUccw6obmPz2CDfVoR2egPrZKYqC64qy1mw1PrxbPAGaz/P3g/rn0nebsPURXRfuysE63YXctHqivwVJFnjMQRffONQXOu9gNqn+asKBFquYhvd/ubtzR5FgX4HQJFKsl6AWl89uKCw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Provide the basics for architectures to implement a fast-only version of kvm_{test_,}age_gfn. Signed-off-by: James Houghton --- include/linux/kvm_host.h | 1 + virt/kvm/kvm_main.c | 2 ++ 2 files changed, 3 insertions(+) diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 98a987e88578..55861db556e2 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -258,6 +258,7 @@ int kvm_async_pf_wakeup_all(struct kvm_vcpu *vcpu); #ifdef CONFIG_KVM_GENERIC_MMU_NOTIFIER union kvm_mmu_notifier_arg { unsigned long attributes; + bool fast_only; }; struct kvm_gfn_range { diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index ec07caaed6b6..8630dfc82d61 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -867,6 +867,7 @@ static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn, .may_block = false, .lockless = IS_ENABLED(CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS), + .arg.fast_only = fast_only, }; trace_kvm_age_hva(start, end, fast_only); @@ -902,6 +903,7 @@ static int kvm_mmu_notifier_test_young(struct mmu_notifier *mn, .may_block = false, .lockless = IS_ENABLED(CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS), + .arg.fast_only = fast_only, }; trace_kvm_test_age_hva(address, fast_only); From patchwork Thu Sep 26 01:35:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13812690 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DEDCBCCF9EB for ; Thu, 26 Sep 2024 01:36:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8AE616B00A5; Wed, 25 Sep 2024 21:35:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 85D5E6B00A8; Wed, 25 Sep 2024 21:35:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 661D96B00C8; Wed, 25 Sep 2024 21:35:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 4734B6B00A5 for ; Wed, 25 Sep 2024 21:35:36 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 09547120717 for ; Thu, 26 Sep 2024 01:35:36 +0000 (UTC) X-FDA: 82605172272.10.258289E Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf13.hostedemail.com (Postfix) with ESMTP id 4581D20006 for ; Thu, 26 Sep 2024 01:35:34 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=bA+pQUzg; spf=pass (imf13.hostedemail.com: domain of 3Zbr0ZgoKCOoVfTagSTfaZSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--jthoughton.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3Zbr0ZgoKCOoVfTagSTfaZSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1727314473; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/c4iHe6I2dqFJ8L4xEL/wcuXYs+QQdS3mQ0rZKnPlRM=; b=ZNiF4SnEr4hCiBbNctiu2Zq21UZAy9KyBbA08ErHFfXN39QXg3n6xGrc3ybv0WkdMSE3QU v+sj+jebdxPbmC9dTKSMGOmzMag3SBpqBKVwQ1H9F2w0+4pVIwJJ548n1geGMFPXX55ZQa 0q3NIMFkBD8XWDfdTxiv9NpEaBhoeIU= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=bA+pQUzg; spf=pass (imf13.hostedemail.com: domain of 3Zbr0ZgoKCOoVfTagSTfaZSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--jthoughton.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3Zbr0ZgoKCOoVfTagSTfaZSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1727314473; a=rsa-sha256; cv=none; b=qz9I5u1pZE5P1HdzMWtkpxgXWcUGXJAMlHXmOU5Tvg8RZGPSB7u9zqpvwCQjOcfRdIit/I QrgeyvAigeXKey9Qc5/I9iHvfdlrKJIHWzzjDFsLd3vVn4lhudWmXA71E8x2ZOIcMTKqOP I0eISskcwUBBOiBkiMfsrlgXf9e/5Ak= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-e02b5792baaso797092276.2 for ; Wed, 25 Sep 2024 18:35:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1727314533; x=1727919333; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=/c4iHe6I2dqFJ8L4xEL/wcuXYs+QQdS3mQ0rZKnPlRM=; b=bA+pQUzgNSm2/Qa7NzLpKDnuviGD14U/8XvC8CsL2yHxL0mgenK2SNYFVemU6DyuhR E0P8YIAKuszhYK3uAXUD/IikJ0UvlxQP3+g51pffoKdp6EFWbZ885vcRvJg2FQHsCNQN CQtiib+/BLdTLclIk+Tuut91ycSCr1bxP0xDN68vkWFfiAx7HsSNFHn0iYnClHMbcuPr GnAX6oVPbDIC7fBpb4wfxT2pmHF1LNm2QT7tsuCuNJglzzVk7KGImZ0rDn4eBFMsBV/D idodOL0fzyBlg6JVGIgU89ckumr7JSn+sY60INUkYjenaLbQoerFGlOHVlt/3eSsiAxU J64w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727314533; x=1727919333; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/c4iHe6I2dqFJ8L4xEL/wcuXYs+QQdS3mQ0rZKnPlRM=; b=BOyBBarGkH8ky4NRRdnbjGMojxZfIJft5r64tO58eNNKNZRXpkMmcou3XWA5mOn9sw p4Emw/hODKwPF+DPX1LXj7obbx8N5Ut6iji//SsK2r2vscC5ZUrSvgo3DBm4pfuvCn7l qOA6XXHEUig2E/Eif/t89+WmzcdUanO+HXU3gJUiU8+Q1yB4iX+t4iPGFaaf+KRFCMMA /P9ci3eDj9ogUwkYVcibL/gX1/AbwYOyfhAHJLCM0aVk5clKCyfBVhKI5Y3vLjvCItoh CrMeTeDUn/mS0BX5TPdJ/cN1Gz16fRgQbNCxj6fhLJTKt2cgsnzAtollr52exattR7MH 1PFQ== X-Forwarded-Encrypted: i=1; AJvYcCXwiD6pdHQVwJgLufTgZqGZbareqF2+xhmhp2iOhz8/cPMXim1sdyp+tXoyDoTxAkD/9nKTjS3bRg==@kvack.org X-Gm-Message-State: AOJu0YxRqjkqZWKZnDxpfkC5I6LTISnV4bvg56LknpOGjmjv1+Rhy3nT Wq+UKQiHGgYL7yiCEFPFTbH3blEXvoMcyypIAEzoGNgYM4d/roxHHtED7PHAo/g1FlpIm5EEbSu kCiHsZ39ggcS/l8+RWQ== X-Google-Smtp-Source: AGHT+IGK4m8FSya1x5h1yKPn0SlYZqL+0YnNSUJKqrhSqJhacVQdW7OQ9a4614HCOKTuoSQjtwwz7rbSxAQZQC1c X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:13d:fb22:ac12:a84b]) (user=jthoughton job=sendgmr) by 2002:a5b:a:0:b0:e25:cf7f:a065 with SMTP id 3f1490d57ef6-e25cf7fa247mr2793276.8.1727314533354; Wed, 25 Sep 2024 18:35:33 -0700 (PDT) Date: Thu, 26 Sep 2024 01:35:03 +0000 In-Reply-To: <20240926013506.860253-1-jthoughton@google.com> Mime-Version: 1.0 References: <20240926013506.860253-1-jthoughton@google.com> X-Mailer: git-send-email 2.46.0.792.g87dc391469-goog Message-ID: <20240926013506.860253-16-jthoughton@google.com> Subject: [PATCH v7 15/18] KVM: x86/mmu: Locklessly harvest access information from shadow MMU From: James Houghton To: Sean Christopherson , Paolo Bonzini Cc: Andrew Morton , David Matlack , David Rientjes , James Houghton , Jason Gunthorpe , Jonathan Corbet , Marc Zyngier , Oliver Upton , Wei Xu , Yu Zhao , Axel Rasmussen , kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: 4581D20006 X-Stat-Signature: ya4qcitj8gzazj6ibrryhpw8s88858be X-HE-Tag: 1727314534-324583 X-HE-Meta: U2FsdGVkX1+4QLOXFPl6No9IwJbIETj7and4w78vom8jt0TtR5x4geZbR8iVjR6A5ZfufQy1rYJLJ5wFdRoxd0xhfyQ9vftV7yqyNTZ/qhciZutNNePjNOUHdpwy5nhpUXGUKh7TvANhmRYLEeFHNN5m84TEdXZsK8LerJSXYsM9iWIQbHmL7rx1V4v4GP6TKD36dq+qB9QNN4aArNrjtwF539IX8vub6u0SqktiM6dnv8jSgdAyvqImoEaU+s/ZZyp6wZmoAzOwFeTXVvbyHVd9JcKh6dc1ajshXdQNjct2rZX3r7cJlLjFKUj75APEhdoVm+VW9/FhRmOHv2JBp3qeyZVUrz3H9x04qpdi3YQ5eMKrhpAG93OG4cYKcGggewZ20CO4aSfUWlc4wGlsA3oSYZ9lD2DUnMd5dlM8EtrlsylHqEnyBRMYqL4e9U1/y32P633pDMyvn5rpFZG3PkDqqrYd8BWorKFEypt0/KIyoOWEKtUJYLdkFPBgP63JSmaEFhBruj1hHs7ya2k410l3VczOJ9YnWXmLJTH2KrynXlg76u0WPM9n4xJjKG560hCkWcohkZekKGPMCmu06aYuwyGVdEbgL6pRNrB4X5fxv+5Dlf7d/mP2RPMB48TsfytJJ3KoOV/afzo/v3Agza1Ffwk8fl/UiSF1kNV764HqpTCj1DsJNL5QF+DBu5NFdafx3Wn10pBmwR0z43FhR/0zNEd4mPp9QCstnkylkYCORmcVnTm+qMCb2ofbTfTG/AvNz3cu6z4qZpynmsqRYoVzI+4HT2B6aBOLlHHsgH/28Hsvp/vNFoCt6xxuLr9VD9TEJtnFNhummkTapvFnYGuII7333uRhhV8QfXIMnJRtMHZYttqU61cWZ362mcvnaObKG9N7RJLvJzuL0TZmZgE9b/ZtnlKiHFg4i3nvcC+Wi6TsBzkpPFChoZPN/PD6C7Zoj7ZARhEFugU8Ly5 MJu0N/jc DboyYYL3RfiE5inYg7+6zFt2itZKCZHjocSVYGBwM2akdMKWHTO1TWgpzVmxrSyulJv9URO8IGhdV4reSSsNzx2wk9WXA7Rskq3YgZYxdk0fkTUecOjbRSQD1fHKcXdcQOhDdMH3vFbFnDO2UfKtdjECnnm/DMqoAiT7RSz+B3ZntpXyfCLOQE2uW47R1Hvaf0/X8mikAX7PSHqFaGqNVMAojsELBPMavrGYgdXo1FKdQSrr5v5FwXPvEKamTthT510z2v6ddGk8ppwbehAfkEmhJZRkDOZNwMpc2AtnfdpvlTrECxiVIFYwjWgqJq4vzwRjxCxKawgaybpTq4/KOhjPHxKzMJCh7vj4BwLsxQdwUCjx812sYbMlztRq0OSMJlVXmv0/1f4iLDaDD888YH02rTgRl3LK3eG+qptnEyDpzlyvrqEw7TMbG6PNDmhQqxH3T6j6hii+uvDPrtAF5BXhUVkr80aTcN7JSnhgZXd19oLhwa1cLFDWyvhxVAQPjUtAHMEWss4HbV7u86NJjFP9sTffqBbCU3+6PCPu38DbQfq7Nba44qy67CjBcP+9qS73R/nei3sXwQdU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Move where the lock is taken for the shadow MMU case to only take the lock when !range->arg.fast_only (i.e., for the non-fast_only aging MMU notifiers). Signed-off-by: James Houghton --- arch/x86/kvm/mmu/mmu.c | 27 ++++++++++++++++----------- 1 file changed, 16 insertions(+), 11 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index a63497bbcc61..f47bd88b55e3 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -1822,16 +1822,24 @@ static bool __kvm_rmap_age_gfn_range(struct kvm *kvm, static bool kvm_rmap_age_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range, bool test_only) { + bool young; + /* * We can always locklessly test if an spte is young. Because marking * non-A/D sptes for access tracking without holding the mmu_lock is * not currently supported, we cannot always locklessly clear. + * + * For fast_only, we must not take the mmu_lock, so locklessly age in + * that case even though we will not be able to clear the age for + * non-A/D sptes. */ - if (test_only) + if (test_only || range->arg.fast_only) return kvm_rmap_age_gfn_range_lockless(kvm, range, test_only); - lockdep_assert_held_write(&kvm->mmu_lock); - return __kvm_rmap_age_gfn_range(kvm, range, test_only); + write_lock(&kvm->mmu_lock); + young = __kvm_rmap_age_gfn_range(kvm, range, test_only); + write_unlock(&kvm->mmu_lock); + return young; } static bool kvm_has_shadow_mmu_sptes(struct kvm *kvm) @@ -1846,11 +1854,8 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) if (tdp_mmu_enabled) young = kvm_tdp_mmu_age_gfn_range(kvm, range); - if (kvm_has_shadow_mmu_sptes(kvm)) { - write_lock(&kvm->mmu_lock); + if (kvm_has_shadow_mmu_sptes(kvm)) young |= kvm_rmap_age_gfn_range(kvm, range, false); - write_unlock(&kvm->mmu_lock); - } return young; } @@ -1862,11 +1867,11 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) if (tdp_mmu_enabled) young = kvm_tdp_mmu_test_age_gfn(kvm, range); - if (!young && kvm_has_shadow_mmu_sptes(kvm)) { - write_lock(&kvm->mmu_lock); + if (young) + return young; + + if (kvm_has_shadow_mmu_sptes(kvm)) young |= kvm_rmap_age_gfn_range(kvm, range, true); - write_unlock(&kvm->mmu_lock); - } return young; } From patchwork Thu Sep 26 01:35:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13812691 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AB199CCF9EB for ; Thu, 26 Sep 2024 01:36:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1598D6B00C8; Wed, 25 Sep 2024 21:35:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 108A66B00C9; Wed, 25 Sep 2024 21:35:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EC62E6B00CA; Wed, 25 Sep 2024 21:35:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id CCD516B00C8 for ; Wed, 25 Sep 2024 21:35:37 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 85188160755 for ; Thu, 26 Sep 2024 01:35:37 +0000 (UTC) X-FDA: 82605172314.02.C8FB53B Received: from mail-ua1-f74.google.com (mail-ua1-f74.google.com [209.85.222.74]) by imf22.hostedemail.com (Postfix) with ESMTP id C7CDEC000B for ; Thu, 26 Sep 2024 01:35:35 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=HwLo5Sf1; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf22.hostedemail.com: domain of 3Zrr0ZgoKCOsWgUbhTUgbaTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--jthoughton.bounces.google.com designates 209.85.222.74 as permitted sender) smtp.mailfrom=3Zrr0ZgoKCOsWgUbhTUgbaTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--jthoughton.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1727314437; a=rsa-sha256; cv=none; b=azZcDCRFhsK3IagSaC7QlJduYi0QO5TNskfKmGISid7laIrNpVfJEpbPGX9NGUisU3P1a1 Nxe2oq4w0Mjy3gJHWbJQWqgThdROBLurtAltUCUsS73K5ouQJrcTal44Br93is9vw/1zJE j0lpxZxIJPXX4V78aANwR4DdFuhO5Do= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=HwLo5Sf1; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf22.hostedemail.com: domain of 3Zrr0ZgoKCOsWgUbhTUgbaTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--jthoughton.bounces.google.com designates 209.85.222.74 as permitted sender) smtp.mailfrom=3Zrr0ZgoKCOsWgUbhTUgbaTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--jthoughton.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1727314437; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mEOlPh0OL8PB5uPzly3STECymuKs02gkKhKmsw1ZgQM=; b=mizhozegVlsrMXeiRfTRHdqCAGd0fB5Y3ImqSA80Rim1V6OWnQqAwf2EmHYMDoeQIw4FQl SviuapBUlBh986+GTP724LJOszUNur5jz61UHjy9b60C8dpyziPAC7Pj6FW8+Pn7cVnelC wklE1mpr5zDZ1wPwTn/sYWVw2RqsTP0= Received: by mail-ua1-f74.google.com with SMTP id a1e0cc1a2514c-848ab182357so373678241.0 for ; Wed, 25 Sep 2024 18:35:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1727314535; x=1727919335; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=mEOlPh0OL8PB5uPzly3STECymuKs02gkKhKmsw1ZgQM=; b=HwLo5Sf12e+W6Ac0mo36vjz46oNQzz/Usqw6Ru/7CUGbFDQEnKkoFi0bIp+flZYVfU KXVFm6qHiDn+1NIV06RIm3wX+m9FZuRCEMnxm2H8d23KqmVID8Dq8ajJOWd1xZg2kTIK 8LfgvixrjgmR/eB4KA8us6/H058XC5E5Vym3GOzJjOgQBxm5Z4EqfMXyGzgWNqIOJExG q9CHszToqZy7sd/tMioOM9jkCeLGTCwON7SNkUvAiycIPQ9i8nDDjGeibMTXd2l9O52u FXubw+l8FKxnodjD79tVR7KE6PEx3lfA1qS9HU4syOkG3hGOkOBOMUdFfbreT+11ZF8x go/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727314535; x=1727919335; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=mEOlPh0OL8PB5uPzly3STECymuKs02gkKhKmsw1ZgQM=; b=ZDWshzARPpRxc2+EcN6v2iM8m73hMcbvY6+iuP0VH+VON9OwyQkxZMQsWnRUkH9p7D gCmFTsQaDLn5+bKyIhavQF7GEQ591C6BJ67WJgn6t/TabLnxGxl+GLxxZFp6WwF71eY8 gKDpKE2rHru4E4HjxOyGSZNRjBFuwrjYxzRSNGajP2xVrZQtsfSJ9Tz0V/yeZl/DGI5K Xt+d7SZTrcMyc9wUQpQNV/wlzEuq+vhzNf5el1q9DXVxoNxHwlCPCrezMoAbOWBI0z5o 9WeTIIoLj+rZURJAfczHT/CsakyYxmxrFTnc0Z8RgUx0kcir+2dbujz/87/FUtkj1tZ8 idzg== X-Forwarded-Encrypted: i=1; AJvYcCXGD9IIB41waVpMhGAcZ+yQBt0CZuDAdOgF5t6Th+2mnWHxRey+Iyrt8o4vS+ltTdHw9a2yhn60tA==@kvack.org X-Gm-Message-State: AOJu0YyvhJpNs8f7PaNVw9M55UKe5zZxt1FM8uyz6zl+jo7xi7Wp6Lyj 3W+RHgJWsdSYNeYnzReURFfyn0dqtkRtoit5kEYgfUV024ZceNTtJwdqnQsIy96zfMn1XRtl6/v tmq/BRx8K1BmjWsWqDw== X-Google-Smtp-Source: AGHT+IEMx7yb7oZJxjUcxiV+ubu09F3MgF2iO9wA9qRkGH/snKIzoGYYivc2BPhRyOg04he4NRTTUUTncXdmo6QX X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:13d:fb22:ac12:a84b]) (user=jthoughton job=sendgmr) by 2002:ab0:71d3:0:b0:84e:89c5:3bdf with SMTP id a1e0cc1a2514c-84e9940fb1bmr4272241.0.1727314534796; Wed, 25 Sep 2024 18:35:34 -0700 (PDT) Date: Thu, 26 Sep 2024 01:35:04 +0000 In-Reply-To: <20240926013506.860253-1-jthoughton@google.com> Mime-Version: 1.0 References: <20240926013506.860253-1-jthoughton@google.com> X-Mailer: git-send-email 2.46.0.792.g87dc391469-goog Message-ID: <20240926013506.860253-17-jthoughton@google.com> Subject: [PATCH v7 16/18] KVM: x86/mmu: Enable has_fast_aging From: James Houghton To: Sean Christopherson , Paolo Bonzini Cc: Andrew Morton , David Matlack , David Rientjes , James Houghton , Jason Gunthorpe , Jonathan Corbet , Marc Zyngier , Oliver Upton , Wei Xu , Yu Zhao , Axel Rasmussen , kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: C7CDEC000B X-Stat-Signature: 13h5k46xxcsp5nccfanjmkb1mh3sqib4 X-Rspam-User: X-HE-Tag: 1727314535-303356 X-HE-Meta: U2FsdGVkX1/kcIFQHA2rh08kChgDYUAd1r8PcbAAgQ/pUXY/7UCF6Dra9BqQjlVVZnlBJoVBfQEqsxDbkbOdUssszfaqgVX/mgYciBkOdGC7YCNKyrree+fLCkCAP5PiZD//pAdhewS+X0s92ydZgfYUuetC+dYJ1MIfyJui49d8I/MO2pKGsAbpEPvL+g7K32FddSeuIpTcHlbxwWGZtT72NuCu0tIRsiVhJxHoECiNu4+PoklmX7ryGm+0MG2YyRcqCkYDZNb7gR0BVpGMfxmw6FLeycmuBLHsSi3IoKwWwDKwGNx9hT8f+vPDiILTbheVlpMx1GjnTurewR5tewWG9tDSiydWdIihfhRxHBUt/x/qJeIrUSo3Wq8NtOjLEXSIxPYiQZq4qyUg8Jr1dDft94RV65iZuVt8hgVjnwPAIfeS52/WepaAq14ZeOi9RZ0L6rh6C/EEFpFIWp83/3YFjZnpFvLhd9MclpVB9KmNuOnp4l2ifdEdeeC/nb3IAA+Qp2RnBlmKp0uGSbPW7DnjGo/qNXEStqvT+1vAzSYoV0KovTzZYOE0DXp6/5DMVvf8Spg728AvYxkwfS8yvcumtXChL1SudeovD5TEjnungQskfVOz9T1Gca7EQv/yxsMzWgjAwxQoLNnlQMREgmV99895MBahwmohOOfdV36aIvR48sr+HwxcusfzPxUNosWnUWW09MwqExandRB73m1mU3D+R/mohZdhdwBzgtKL3mmHIIajbo4kH2GHw6TL7EAbUH+X8b42XpbZt/0NrxrB5IerVDkDLQ6nf/f7DFolX7gpogVUfXhOw1TrnsPYvzbd5HMPwoPKbxeR9czbcP7ZBF88fERbCRJCIQwgUxede7Ls/Q0QmLIV5bnnEleVWDKf8mf1ISk48XBDdXeZ/c5wYRNOnC8tuk4fjrH67mVSgPmpOpIjZGxeSZDksLj4U/iZUCaLZ5LCQr8gxq2 St8MvQpx jgesokpjyFpbTOYfYMiJOe60Lq/FPiFNo5FFlT5WfMzBKj/mkTkZdUoWkh/SFHIEUfjbZ2TnTcFT/bH7PF/XTKRtgYU/akvcBoBnRbrFIQYOM8pgPA2oVM9KA40g7o6HT7ljbVxfbw6Ho6jarCGq+RstIt+z9ma4nvsUxqXTTp3yW4KmVaOggE41B7Nr4utQE+8OINGoJtRwcFuKasQc4wh3kl8ytzez/RgavawqQ2DV0N5EhxMo3HptNVpPMaovu5FrmO6A1efenqBgRYNVcXxLy6cl2m1fJpHsi+D0ht6v+YvORKPjX9PsQj49WTxDqjQzlj7heJpB7P72Otsg/vp4Vy+Xhi3+0PhiIGbf1ps8aAEz0OVHDMJpj3YEkbqotn8eMDxvs4iUbUI6+M9Gw+q15Cek0s+65YWdla4N4Q1m6VpVghayZEtdAP5RQtmPJBgm9xKvhITGTI295B6GoGMVBiOTzwagnj3uXSGgtuw1PFehzYcBbPBKDK7teJbHQS2xxbVS/e0n4iiRU2b09wPFIaScy5EQmwnrfhoY3NMmtMyzpL36coknPF/EpugCr09iYBBida/VEPpjkqL0wWeCJR/MBRkSh5DXPH7G4vcNR17zw/O1pAYGt1A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000002, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Because the x86 MMU locklessly implements fast_only versions of kvm_test_age_gfn and kvm_age_gfn, we can advertise support for has_fast_aging to allow MGLRU to quickly find better eviction candidates. There is one case where the MMU is not 100% accurate: for the shadow MMU, when A/D bits are not in use, young sptes will never be aged with the fast_only kvm_age_gfn. In this case, such pages will consistently appear young, so they will be the least likely eviction candidates. Signed-off-by: James Houghton --- arch/x86/kvm/mmu/mmu.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index f47bd88b55e3..1798e3853d27 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -7708,6 +7708,8 @@ int kvm_mmu_post_init_vm(struct kvm *kvm) { int err; + kvm->mmu_notifier.has_fast_aging = true; + if (nx_hugepage_mitigation_hard_disabled) return 0; From patchwork Thu Sep 26 01:35:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13812692 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B0270CCFA10 for ; Thu, 26 Sep 2024 01:36:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D1D876B00CA; Wed, 25 Sep 2024 21:35:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CCC8D6B00CC; Wed, 25 Sep 2024 21:35:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A83EB6B00CD; Wed, 25 Sep 2024 21:35:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 826F56B00CA for ; Wed, 25 Sep 2024 21:35:40 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 36B5216073C for ; Thu, 26 Sep 2024 01:35:40 +0000 (UTC) X-FDA: 82605172440.16.008BE9D Received: from mail-vs1-f73.google.com (mail-vs1-f73.google.com [209.85.217.73]) by imf10.hostedemail.com (Postfix) with ESMTP id 6A0FBC000D for ; Thu, 26 Sep 2024 01:35:37 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=HWS0dRqN; spf=pass (imf10.hostedemail.com: domain of 3aLr0ZgoKCO0YiWdjVWidcVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--jthoughton.bounces.google.com designates 209.85.217.73 as permitted sender) smtp.mailfrom=3aLr0ZgoKCO0YiWdjVWidcVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1727314501; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MWj+EtmYN6BoWav4C6XYFCP/YJSRpIEJzj9cj+E6cQI=; b=4ZSmsGCB3yhXO8lN6QThvzj7wmeid6A0tPzhRBsUH44Qx5uq9uK1CT5qoZEyxRG9gh6wZM gvmga/KdX5xBDAVlcH8eZ1YsOMfxy8KA6c0frhPuqTjmtHNfk2V0Ft6Aj0M5wjVzO2vsmS wlhOD5RixYD4M6GPO0g5kOeD4DXWiIo= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=HWS0dRqN; spf=pass (imf10.hostedemail.com: domain of 3aLr0ZgoKCO0YiWdjVWidcVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--jthoughton.bounces.google.com designates 209.85.217.73 as permitted sender) smtp.mailfrom=3aLr0ZgoKCO0YiWdjVWidcVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1727314501; a=rsa-sha256; cv=none; b=Eerj0Y7yxbs2hJy/ANAzuDKc1alX0OhwJbajt9Bd+YZR6koewU0zcJfKewbohpmIZbXprM mpg5165xNso4X9qqnufsixOQkTwWbOwn5p2kj2IIttHg66aFzA7yGwq2rRKa2CAwDflsCE WixS5yumkhxAS5oHH3rHdpp/QF7ShM0= Received: by mail-vs1-f73.google.com with SMTP id ada2fe7eead31-49bc7d7b31fso191943137.1 for ; Wed, 25 Sep 2024 18:35:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1727314536; x=1727919336; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=MWj+EtmYN6BoWav4C6XYFCP/YJSRpIEJzj9cj+E6cQI=; b=HWS0dRqNUbZ3nOmld805JSKeJLfhKmkUV2v7f8f41IJIbOWWdTF7hXKu5TQKAPnwZP kyXjmheR495dYmYwbQkcrJUlCmKr4lvcIj6hXZPeGtSOJl6mNHhTojjW9izG4mfrimoG F05vDGWO1XI76dZXOHk7UTK3481QNdyhRJAvKYiCPISw4h5Yj/KKck66ItlfzpgKiBTg /qOGpDVeY9fd290v06hu0uahovgUkLx6gEE282MncebqSR0dui23MD0+wKOMl10Uyawc FPmD5GHymClXoe1FEfOCbbEhx/YM0xtOntbq2PFT+eOSClV/rEG8uRrIk/tKI6RlIRRg 4UDA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727314536; x=1727919336; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=MWj+EtmYN6BoWav4C6XYFCP/YJSRpIEJzj9cj+E6cQI=; b=uyrXr5fO549FYrsjUsjgiCiqr4AMw711NdjkrbqtujVw3r+CEl5p3pikJ0QjWK2pyG pxD6v+E584eEQ+PwDeSAmV9wylX02Owe0jo9rZl+flXKC1Zfc1xDeXBwkb4Z1cS4T/fT fS0Jyj+pgKt+Npw28BHjwQ1JspplseurrAii4uNRSgw/staCUcBl5kBS8/iKM4kDt2U1 DxE0JloIdtSz4qkZIvRA87Y1w8BfbwJtaHhhxiBWCtN2fxoU1CrOCBjlrOFGJdsVDRT6 /aCu+4V2ui1UJU4GKM8EQf9qiITZqO3zXdqjUw0hL+cF/39vxoAlH723ZTC7/NNZUUMB YCyg== X-Forwarded-Encrypted: i=1; AJvYcCVjXttT/kWglO2jo0iKota1Vocp8xPTzZ7cTbuRSG0HaqZeK4yBFAOQiDs1VpF69j38ADgHVEJInQ==@kvack.org X-Gm-Message-State: AOJu0Yx2xs8Tp9svmYtRgoAQbqW97qKA/EZh/7tCa0RUm6P4mWMBdnAJ 3xmZrkRh0mPSxqf9WgGeICEXkAmGNVMk+TMY8PDkQFAktTHkEJtiOL18XR1qBhVikifZPwQuNMs kH/CUsCIStLAaUEZSMQ== X-Google-Smtp-Source: AGHT+IEWbFq5+mKO2q49QysNNPvZDCXSWbklfJ2UiM9KvwQART8aUu+/JLlyrv3E0SSfh27Nwb4MbN5v9Rl58JjT X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:13d:fb22:ac12:a84b]) (user=jthoughton job=sendgmr) by 2002:a05:6102:5e97:b0:48d:9c9b:f391 with SMTP id ada2fe7eead31-4a15dd4beb2mr104946137.5.1727314536465; Wed, 25 Sep 2024 18:35:36 -0700 (PDT) Date: Thu, 26 Sep 2024 01:35:05 +0000 In-Reply-To: <20240926013506.860253-1-jthoughton@google.com> Mime-Version: 1.0 References: <20240926013506.860253-1-jthoughton@google.com> X-Mailer: git-send-email 2.46.0.792.g87dc391469-goog Message-ID: <20240926013506.860253-18-jthoughton@google.com> Subject: [PATCH v7 17/18] mm: multi-gen LRU: Have secondary MMUs participate in aging From: James Houghton To: Sean Christopherson , Paolo Bonzini Cc: Andrew Morton , David Matlack , David Rientjes , James Houghton , Jason Gunthorpe , Jonathan Corbet , Marc Zyngier , Oliver Upton , Wei Xu , Yu Zhao , Axel Rasmussen , kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org X-Rspam-User: X-Stat-Signature: j4yiy39bbgxwdbgtsibf88epwba9gsyr X-Rspamd-Queue-Id: 6A0FBC000D X-Rspamd-Server: rspam11 X-HE-Tag: 1727314537-590744 X-HE-Meta: U2FsdGVkX1/pGIidV7LVOGRrEyBXHZ6CDE/D24OTEINcbWYX3ScNy6hi4L+vPMJLfIdcDvum9hugCpDyOezmNo/7r4DXbHucr1T0RylMCYT0hrSeW+AIz9O2wSXl2YbKKkTKx+jOvmgZlPpa7jkSrF4DBncyiOeRlRSN98xnW23Wokl2ACpfUSO2zXPAsAhuI4vQm9awCJCZt5dZ5Pvv7DWuxUNV/CPmKhH6cIR+MkUDbapBudeL/0Ri6noxg+X+QiJWq263V+Ddg3hjfjWA1wZN3aOPQ8XuyR+HHHIcmRt8wMKM6trGV4097hzUDVLBPbzklmU6LJOvSVSQOdvv6P7mP3AlFrx6JdgeHPHBsaLsmHrO6kHFF3qNoqq6BPCfqD/L9o2f15zhkqmYDmnCeXCJADd6uaBbvNBgUzgkh0qLUZfTibXoNmHAsE7B15c7ddouVEFjQ1xkMjgQJawhn8meqqhsBB/m6OauLHB5czHKFOxpa99TyjsIb9+km1E3awkKFTejnOW0yna5TwbEH2YanNDiRVT/D9ogOzhs/N5eg0AJt66fNVG7kuDiV4v6DwwRer792uMPfFD6k825Nc6OUSHwvXRIaMGohm8ZH/1ullF0W71eHCyih1DBTBikVbHeFHSXpO/xMOXidPsYw3dntaBsw+V44vUJkfQLLF/EwUQUKuZIJ9B4c86XYPkQKUspvEtDqH77UyTM669a023XZyKdBJ7Wuz2Fkf36It4/d8I232nvWS6Xo/WMEPr28rcahMF847JEluIE8s6+Xo1XR9vk3Cv/7d8Fa1qhkklHwOa0RmSg7uk2vvHLwaF0EQkNE38DbE5QRa2BeKBwYrT+A2+uq3EYeB/WIB7oXqWoo4ayyglIWAuxgCjsU/vUUoKviduYoBX7gRZNPA5cjNbapOew8YZav6oYpeUz5WZOYAmmI/p9ud+XNp8oVy6pxOAbxGv2TbNDj2BaICS 30R8HG6P VOV04J+nP61VDEyE4/AZ54jkTHAP2zG7Z8aEK7D8HywJ27Au5ETI9y05qG1Kya/mCpLVRnDCRm4I4kOFbBSvgXqpiZHxlXUsLmrb2C4KAJxvEpkIVbWZY9GpeW87v2u4ZWSlzEOHOAiPMiTzJiS5GgObCgyOKDYJ+14KEHC96zTjuS9eP/3m58dKy+C2MmIg3qDDjB4m/Aj0xuKqVx07V+lOp7EgZ4oQPmfRHAfHpcXFjvw6gBDeFxZD/7rP20+nQCkbHg1IXYdY0znqinJVHCntoVbSaFgreKdRxWq2ntLyEiPRrNVjR18sNh09/mjmkF+/hljpDXwvuQuqUwNQFI8OIVWuMHx/HKTfZbfotchSkg9WfN7675iZpB1N8KoJAkSpqBgCfG8rvCyZnBgqoglkRUKJ08rrUKmqEtiEtE1tPgBF2GmwcV19+DFs9ZwOFEYQkWhHWg3q53868k24+tgu1ckN8TS9URUJIPY2158ZwaJCkPpDRC3Tyb9p4M5wKTPC3qCL5e72ncbNBAAblu09teZtn4vKAKfb1p/AeWYSFKqYm/rDyW30XRZX8fB3zsdNwr8EKJOh7XNM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Secondary MMUs are currently consulted for access/age information at eviction time, but before then, we don't get accurate age information. That is, pages that are mostly accessed through a secondary MMU (like guest memory, used by KVM) will always just proceed down to the oldest generation, and then at eviction time, if KVM reports the page to be young, the page will be activated/promoted back to the youngest generation. The added feature bit (0x8), if disabled, will make MGLRU behave as if there are no secondary MMUs subscribed to MMU notifiers except at eviction time. Implement aging with the new mmu_notifier_clear_young_fast_only() notifier. For architectures that do not support this notifier, this becomes a no-op. For architectures that do implement it, it should be fast enough to make aging worth it (usually the case if the notifier is implemented locklessly). Suggested-by: Yu Zhao Signed-off-by: James Houghton --- Documentation/admin-guide/mm/multigen_lru.rst | 6 +- include/linux/mmzone.h | 6 +- mm/rmap.c | 9 +- mm/vmscan.c | 148 ++++++++++++++---- 4 files changed, 127 insertions(+), 42 deletions(-) diff --git a/Documentation/admin-guide/mm/multigen_lru.rst b/Documentation/admin-guide/mm/multigen_lru.rst index 33e068830497..e1862407652c 100644 --- a/Documentation/admin-guide/mm/multigen_lru.rst +++ b/Documentation/admin-guide/mm/multigen_lru.rst @@ -48,6 +48,10 @@ Values Components verified on x86 varieties other than Intel and AMD. If it is disabled, the multi-gen LRU will suffer a negligible performance degradation. +0x0008 Clear the accessed bit in secondary MMU page tables when aging + instead of waiting until eviction time. This results in accurate + page age information for pages that are mainly used by a + secondary MMU. [yYnN] Apply to all the components above. ====== =============================================================== @@ -56,7 +60,7 @@ E.g., echo y >/sys/kernel/mm/lru_gen/enabled cat /sys/kernel/mm/lru_gen/enabled - 0x0007 + 0x000f echo 5 >/sys/kernel/mm/lru_gen/enabled cat /sys/kernel/mm/lru_gen/enabled 0x0005 diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 1dc6248feb83..dbfb868c3708 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -400,6 +400,7 @@ enum { LRU_GEN_CORE, LRU_GEN_MM_WALK, LRU_GEN_NONLEAF_YOUNG, + LRU_GEN_SECONDARY_MMU_WALK, NR_LRU_GEN_CAPS }; @@ -557,7 +558,7 @@ struct lru_gen_memcg { void lru_gen_init_pgdat(struct pglist_data *pgdat); void lru_gen_init_lruvec(struct lruvec *lruvec); -void lru_gen_look_around(struct page_vma_mapped_walk *pvmw); +bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw); void lru_gen_init_memcg(struct mem_cgroup *memcg); void lru_gen_exit_memcg(struct mem_cgroup *memcg); @@ -576,8 +577,9 @@ static inline void lru_gen_init_lruvec(struct lruvec *lruvec) { } -static inline void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) +static inline bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw) { + return false; } static inline void lru_gen_init_memcg(struct mem_cgroup *memcg) diff --git a/mm/rmap.c b/mm/rmap.c index 2490e727e2dc..51bbda3bae60 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -870,13 +870,10 @@ static bool folio_referenced_one(struct folio *folio, continue; } - if (pvmw.pte) { - if (lru_gen_enabled() && - pte_young(ptep_get(pvmw.pte))) { - lru_gen_look_around(&pvmw); + if (lru_gen_enabled() && pvmw.pte) { + if (lru_gen_look_around(&pvmw)) referenced++; - } - + } else if (pvmw.pte) { if (ptep_clear_flush_young_notify(vma, address, pvmw.pte)) referenced++; diff --git a/mm/vmscan.c b/mm/vmscan.c index cfa839284b92..6ab87dd1c6d9 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -56,6 +56,7 @@ #include #include #include +#include #include #include @@ -2594,6 +2595,11 @@ static bool should_clear_pmd_young(void) return arch_has_hw_nonleaf_pmd_young() && get_cap(LRU_GEN_NONLEAF_YOUNG); } +static bool should_walk_secondary_mmu(void) +{ + return get_cap(LRU_GEN_SECONDARY_MMU_WALK); +} + /****************************************************************************** * shorthand helpers ******************************************************************************/ @@ -3291,7 +3297,8 @@ static bool get_next_vma(unsigned long mask, unsigned long size, struct mm_walk return false; } -static unsigned long get_pte_pfn(pte_t pte, struct vm_area_struct *vma, unsigned long addr) +static unsigned long get_pte_pfn(pte_t pte, struct vm_area_struct *vma, unsigned long addr, + struct pglist_data *pgdat) { unsigned long pfn = pte_pfn(pte); @@ -3306,10 +3313,15 @@ static unsigned long get_pte_pfn(pte_t pte, struct vm_area_struct *vma, unsigned if (WARN_ON_ONCE(!pfn_valid(pfn))) return -1; + /* try to avoid unnecessary memory loads */ + if (pfn < pgdat->node_start_pfn || pfn >= pgdat_end_pfn(pgdat)) + return -1; + return pfn; } -static unsigned long get_pmd_pfn(pmd_t pmd, struct vm_area_struct *vma, unsigned long addr) +static unsigned long get_pmd_pfn(pmd_t pmd, struct vm_area_struct *vma, unsigned long addr, + struct pglist_data *pgdat) { unsigned long pfn = pmd_pfn(pmd); @@ -3324,6 +3336,10 @@ static unsigned long get_pmd_pfn(pmd_t pmd, struct vm_area_struct *vma, unsigned if (WARN_ON_ONCE(!pfn_valid(pfn))) return -1; + /* try to avoid unnecessary memory loads */ + if (pfn < pgdat->node_start_pfn || pfn >= pgdat_end_pfn(pgdat)) + return -1; + return pfn; } @@ -3332,10 +3348,6 @@ static struct folio *get_pfn_folio(unsigned long pfn, struct mem_cgroup *memcg, { struct folio *folio; - /* try to avoid unnecessary memory loads */ - if (pfn < pgdat->node_start_pfn || pfn >= pgdat_end_pfn(pgdat)) - return NULL; - folio = pfn_folio(pfn); if (folio_nid(folio) != pgdat->node_id) return NULL; @@ -3358,6 +3370,26 @@ static bool suitable_to_scan(int total, int young) return young * n >= total; } +static bool lru_gen_notifier_clear_young(struct mm_struct *mm, + unsigned long start, + unsigned long end) +{ + return should_walk_secondary_mmu() && + mmu_notifier_clear_young_fast_only(mm, start, end); +} + +static bool lru_gen_pmdp_test_and_clear_young(struct vm_area_struct *vma, + unsigned long addr, + pmd_t *pmd) +{ + bool young = pmdp_test_and_clear_young(vma, addr, pmd); + + if (lru_gen_notifier_clear_young(vma->vm_mm, addr, addr + PMD_SIZE)) + young = true; + + return young; +} + static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, struct mm_walk *args) { @@ -3372,8 +3404,9 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, struct pglist_data *pgdat = lruvec_pgdat(walk->lruvec); DEFINE_MAX_SEQ(walk->lruvec); int old_gen, new_gen = lru_gen_from_seq(max_seq); + struct mm_struct *mm = args->mm; - pte = pte_offset_map_nolock(args->mm, pmd, start & PMD_MASK, &ptl); + pte = pte_offset_map_nolock(mm, pmd, start & PMD_MASK, &ptl); if (!pte) return false; if (!spin_trylock(ptl)) { @@ -3391,11 +3424,11 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, total++; walk->mm_stats[MM_LEAF_TOTAL]++; - pfn = get_pte_pfn(ptent, args->vma, addr); + pfn = get_pte_pfn(ptent, args->vma, addr, pgdat); if (pfn == -1) continue; - if (!pte_young(ptent)) { + if (!pte_young(ptent) && !mm_has_notifiers(mm)) { walk->mm_stats[MM_LEAF_OLD]++; continue; } @@ -3404,8 +3437,14 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, if (!folio) continue; - if (!ptep_test_and_clear_young(args->vma, addr, pte + i)) - VM_WARN_ON_ONCE(true); + if (!lru_gen_notifier_clear_young(mm, addr, addr + PAGE_SIZE) && + !pte_young(ptent)) { + walk->mm_stats[MM_LEAF_OLD]++; + continue; + } + + if (pte_young(ptent)) + ptep_test_and_clear_young(args->vma, addr, pte + i); young++; walk->mm_stats[MM_LEAF_YOUNG]++; @@ -3471,22 +3510,25 @@ static void walk_pmd_range_locked(pud_t *pud, unsigned long addr, struct vm_area /* don't round down the first address */ addr = i ? (*first & PMD_MASK) + i * PMD_SIZE : *first; - pfn = get_pmd_pfn(pmd[i], vma, addr); - if (pfn == -1) - goto next; - - if (!pmd_trans_huge(pmd[i])) { - if (should_clear_pmd_young()) + if (pmd_present(pmd[i]) && !pmd_trans_huge(pmd[i])) { + if (should_clear_pmd_young() && + !should_walk_secondary_mmu()) pmdp_test_and_clear_young(vma, addr, pmd + i); goto next; } + pfn = get_pmd_pfn(pmd[i], vma, addr, pgdat); + if (pfn == -1) + goto next; + folio = get_pfn_folio(pfn, memcg, pgdat, walk->can_swap); if (!folio) goto next; - if (!pmdp_test_and_clear_young(vma, addr, pmd + i)) + if (!lru_gen_pmdp_test_and_clear_young(vma, addr, pmd + i)) { + walk->mm_stats[MM_LEAF_OLD]++; goto next; + } walk->mm_stats[MM_LEAF_YOUNG]++; @@ -3543,19 +3585,18 @@ static void walk_pmd_range(pud_t *pud, unsigned long start, unsigned long end, } if (pmd_trans_huge(val)) { - unsigned long pfn = pmd_pfn(val); struct pglist_data *pgdat = lruvec_pgdat(walk->lruvec); + unsigned long pfn = get_pmd_pfn(val, vma, addr, pgdat); walk->mm_stats[MM_LEAF_TOTAL]++; - if (!pmd_young(val)) { - walk->mm_stats[MM_LEAF_OLD]++; + if (pfn == -1) continue; - } - /* try to avoid unnecessary memory loads */ - if (pfn < pgdat->node_start_pfn || pfn >= pgdat_end_pfn(pgdat)) + if (!pmd_young(val) && !mm_has_notifiers(args->mm)) { + walk->mm_stats[MM_LEAF_OLD]++; continue; + } walk_pmd_range_locked(pud, addr, vma, args, bitmap, &first); continue; @@ -3563,7 +3604,7 @@ static void walk_pmd_range(pud_t *pud, unsigned long start, unsigned long end, walk->mm_stats[MM_NONLEAF_TOTAL]++; - if (should_clear_pmd_young()) { + if (should_clear_pmd_young() && !should_walk_secondary_mmu()) { if (!pmd_young(val)) continue; @@ -4030,6 +4071,31 @@ static void lru_gen_age_node(struct pglist_data *pgdat, struct scan_control *sc) * rmap/PT walk feedback ******************************************************************************/ +static bool should_look_around(struct vm_area_struct *vma, unsigned long addr, + pte_t *pte, int *young) +{ + int secondary_young = mmu_notifier_clear_young( + vma->vm_mm, addr, addr + PAGE_SIZE); + + /* + * Look around if (1) the PTE is young or (2) the secondary PTE was + * young and one of the "fast" MMUs of one of the secondary MMUs + * reported that the page was young. + */ + if (pte_young(ptep_get(pte))) { + ptep_test_and_clear_young(vma, addr, pte); + *young = true; + return true; + } + + if (secondary_young) { + *young = true; + return mm_has_fast_young_notifiers(vma->vm_mm); + } + + return false; +} + /* * This function exploits spatial locality when shrink_folio_list() walks the * rmap. It scans the adjacent PTEs of a young PTE and promotes hot pages. If @@ -4037,7 +4103,7 @@ static void lru_gen_age_node(struct pglist_data *pgdat, struct scan_control *sc) * the PTE table to the Bloom filter. This forms a feedback loop between the * eviction and the aging. */ -void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) +bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw) { int i; unsigned long start; @@ -4055,16 +4121,20 @@ void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) struct lru_gen_mm_state *mm_state = get_mm_state(lruvec); DEFINE_MAX_SEQ(lruvec); int old_gen, new_gen = lru_gen_from_seq(max_seq); + struct mm_struct *mm = pvmw->vma->vm_mm; lockdep_assert_held(pvmw->ptl); VM_WARN_ON_ONCE_FOLIO(folio_test_lru(folio), folio); + if (!should_look_around(vma, addr, pte, &young)) + return young; + if (spin_is_contended(pvmw->ptl)) - return; + return young; /* exclude special VMAs containing anon pages from COW */ if (vma->vm_flags & VM_SPECIAL) - return; + return young; /* avoid taking the LRU lock under the PTL when possible */ walk = current->reclaim_state ? current->reclaim_state->mm_walk : NULL; @@ -4072,6 +4142,9 @@ void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) start = max(addr & PMD_MASK, vma->vm_start); end = min(addr | ~PMD_MASK, vma->vm_end - 1) + 1; + if (end - start == PAGE_SIZE) + return young; + if (end - start > MIN_LRU_BATCH * PAGE_SIZE) { if (addr - start < MIN_LRU_BATCH * PAGE_SIZE / 2) end = start + MIN_LRU_BATCH * PAGE_SIZE; @@ -4085,7 +4158,7 @@ void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) /* folio_update_gen() requires stable folio_memcg() */ if (!mem_cgroup_trylock_pages(memcg)) - return; + return young; arch_enter_lazy_mmu_mode(); @@ -4095,19 +4168,23 @@ void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) unsigned long pfn; pte_t ptent = ptep_get(pte + i); - pfn = get_pte_pfn(ptent, vma, addr); + pfn = get_pte_pfn(ptent, vma, addr, pgdat); if (pfn == -1) continue; - if (!pte_young(ptent)) + if (!pte_young(ptent) && !mm_has_notifiers(mm)) continue; folio = get_pfn_folio(pfn, memcg, pgdat, can_swap); if (!folio) continue; - if (!ptep_test_and_clear_young(vma, addr, pte + i)) - VM_WARN_ON_ONCE(true); + if (!lru_gen_notifier_clear_young(mm, addr, addr + PAGE_SIZE) && + !pte_young(ptent)) + continue; + + if (pte_young(ptent)) + ptep_test_and_clear_young(vma, addr, pte + i); young++; @@ -4137,6 +4214,8 @@ void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) /* feedback from rmap walkers to page table walkers */ if (mm_state && suitable_to_scan(i, young)) update_bloom_filter(mm_state, max_seq, pvmw->pmd); + + return young; } /****************************************************************************** @@ -5140,6 +5219,9 @@ static ssize_t enabled_show(struct kobject *kobj, struct kobj_attribute *attr, c if (should_clear_pmd_young()) caps |= BIT(LRU_GEN_NONLEAF_YOUNG); + if (should_walk_secondary_mmu()) + caps |= BIT(LRU_GEN_SECONDARY_MMU_WALK); + return sysfs_emit(buf, "0x%04x\n", caps); } From patchwork Thu Sep 26 01:35:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13812693 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2E116CCFA10 for ; Thu, 26 Sep 2024 01:36:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CCC0F6B00CC; Wed, 25 Sep 2024 21:35:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C7A876B00CD; Wed, 25 Sep 2024 21:35:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AA67C6B00CE; Wed, 25 Sep 2024 21:35:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 83B4D6B00CC for ; Wed, 25 Sep 2024 21:35:41 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 02126ACA5D for ; Thu, 26 Sep 2024 01:35:40 +0000 (UTC) X-FDA: 82605172482.21.9B07C04 Received: from mail-vk1-f201.google.com (mail-vk1-f201.google.com [209.85.221.201]) by imf20.hostedemail.com (Postfix) with ESMTP id 235141C0002 for ; Thu, 26 Sep 2024 01:35:39 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=nGMVZPL3; spf=pass (imf20.hostedemail.com: domain of 3arr0ZgoKCO8akYflXYkfeXffXcV.TfdcZelo-ddbmRTb.fiX@flex--jthoughton.bounces.google.com designates 209.85.221.201 as permitted sender) smtp.mailfrom=3arr0ZgoKCO8akYflXYkfeXffXcV.TfdcZelo-ddbmRTb.fiX@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1727314478; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=1HitpDOpIhCd6Znj3amk60TrYFnYLRb+txcGZehcXFE=; b=4KJfb97pnBog6PlWebyz6DtcZxT2a7eDobzg8lyZEs6Xj0p1oGGqnuhbcY+p6o4+0Dl/im 4fbYoQOSHQEkwoaP9rwsOrF8ZTKe5+2YvzQS9C52h1+jeuYeg5xFcos7evqvgVfgUIWxVI AP1gTMlBm/Eg7XtkopGOOYmFJOdeqWc= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=nGMVZPL3; spf=pass (imf20.hostedemail.com: domain of 3arr0ZgoKCO8akYflXYkfeXffXcV.TfdcZelo-ddbmRTb.fiX@flex--jthoughton.bounces.google.com designates 209.85.221.201 as permitted sender) smtp.mailfrom=3arr0ZgoKCO8akYflXYkfeXffXcV.TfdcZelo-ddbmRTb.fiX@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1727314478; a=rsa-sha256; cv=none; b=bOGxvQQRsh2CWkfp9pp1Xoyv8v3ArxKYLnt3EcND/1irSadYgHTjQkWfY36G3tm3WwYsdK YEMp5UELDtFkmh+nLg1DI+F/dLfpU4uDRTStPMl9nfEX5WYjLtW0IrY5gwTHx/hu2yNmPw 45iMAowx8jvDoYxSTRyjZxr5CbzyGVQ= Received: by mail-vk1-f201.google.com with SMTP id 71dfb90a1353d-50113b70fbfso133032e0c.3 for ; Wed, 25 Sep 2024 18:35:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1727314538; x=1727919338; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=1HitpDOpIhCd6Znj3amk60TrYFnYLRb+txcGZehcXFE=; b=nGMVZPL3j30DADq33OkE7iiiVvvmaf5AXIT/b448vmJlCf2sipA8sDZ/dIRU7qd2JU 49kuRCGUk6+s3CR8cH7fvlUJ9IHQHgQoxjlENtDN/o0+F+fvcGM3tuRKsb6umV6ajzf6 PhRI8GgXHhRp2gfALO+8qR1Ek9h7/1E6RxMiV2AQCqHuqhhQiy6S7tyOAnICiS/nsNqZ 1/EQLPmJFgnjUfQw79AJDhjTLsqu+GUg+294iPLEYvl15vjcU06x/cugR1y7JKEZvUMV Er7bkdUltDkQyEgmLeMyyggO/lxGKcWwpAz56DhrDxisR02y9wGucfEonAjng5ZoAc3T nE+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727314538; x=1727919338; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=1HitpDOpIhCd6Znj3amk60TrYFnYLRb+txcGZehcXFE=; b=uVE7TFnRQnFzR0oieJoZEHrwFB6GKyygSoPY20u40fl82YvInKCZXEH6m39fx7ZlB4 2P8sPjUfYb06ke69wBtltiNuJEe1mgdx8bzTVkfvVFMno3gw+Fr0FENU9Q/ch4kOG+8c jvgbm4dTzMjHTFNTkKOrkvsxm7cadc865bgGIRxUe6KpIqgPhay1xxmTt3JNeQTYUrG1 xcwGmnGO1EHRJBSZtIcdOy4OeTNEluXo28xPu51qKyFHuB3BEF/qM2czYm1/TgJ/dLl0 DBI9S1cQKhZjSmsel108QmC7PRQSnIyin/3/6trsm2WVfuB0MnhAA1yUjouqnnw/WiBw JoAQ== X-Forwarded-Encrypted: i=1; AJvYcCXWjBki2yysmZaQjMZ8UmAySfdW3M8Eb//TWibFt5Sm/9AgcVzUOk3RQdAQovtvczF9ft6udaHK0Q==@kvack.org X-Gm-Message-State: AOJu0YzrVSjBevbmhgmRb+BD+PceX5QENoGvFZKiHjQDR3mleTnc+FIg bi+k7MsSGDMM9SZM/Vamdwc6ZLnD4nUjK5zykPCz9lrHkwpZg0lv29ManVRvqnzoaYaYQqwdcF7 bX2vH+nTBmcZsVDP0rA== X-Google-Smtp-Source: AGHT+IFV8iyT3LpbXR6TuEpCRITD2JCMqJuJWTypvZWR30Nc3wsvgcazQHmES7rpaskbP+3JFCF4Mj/JcwI7L1Ez X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:13d:fb22:ac12:a84b]) (user=jthoughton job=sendgmr) by 2002:a05:6122:30aa:b0:503:dbaa:4402 with SMTP id 71dfb90a1353d-505c206d347mr80904e0c.2.1727314538169; Wed, 25 Sep 2024 18:35:38 -0700 (PDT) Date: Thu, 26 Sep 2024 01:35:06 +0000 In-Reply-To: <20240926013506.860253-1-jthoughton@google.com> Mime-Version: 1.0 References: <20240926013506.860253-1-jthoughton@google.com> X-Mailer: git-send-email 2.46.0.792.g87dc391469-goog Message-ID: <20240926013506.860253-19-jthoughton@google.com> Subject: [PATCH v7 18/18] KVM: selftests: Add multi-gen LRU aging to access_tracking_perf_test From: James Houghton To: Sean Christopherson , Paolo Bonzini Cc: Andrew Morton , David Matlack , David Rientjes , James Houghton , Jason Gunthorpe , Jonathan Corbet , Marc Zyngier , Oliver Upton , Wei Xu , Yu Zhao , Axel Rasmussen , kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: 235141C0002 X-Stat-Signature: iaiu3no9cz8q1wh6rmmoxaxq9gbmmxbc X-HE-Tag: 1727314538-907697 X-HE-Meta: U2FsdGVkX184kA2QArC8TlBWA6CQTrU1t6OowK/OTIgItQe6+JOK7Xnd+9wlJ7jvmO1svPoEpuWSq1ZWhBs4wsTFj+2n7p5aTMgw24MumUnGgyi3HsEdSrFyBOPGx3keiEhJ/1D3rsXGN53mh+dWjPvw7AbIrbXoAgcdyFjUj4k8XZ0G7MmrqUg298rY3suHcX/8hfHK1ZuFtAuZ/1Ef/VS6YOcVAf7h/T0HCwp6kkY1Io7g9O5/4Jel+6BaOHhNQKUKShC9SkSH8lQ3m24szVm0hRG31UAvQQeP1A7iRom4dMY3JcMJ4H1B9qnQumnj3w6tbqdg0tDVYEOV/Eori0spA3E0lWK+jvVj7Z4WH+WCU6qeN2nHaquBYWoZbF2aO8kYsw0X1c63S+DGb3EfLcMshHRD//ZjGOTepXUytMudgqtSpe3LY3BHSxuHdqW+Ttyct53XoUESUX+uEE4LB9WB6h3Etpz6HZH2i9MDZogLFFLSJdygzBaZ/GBXx3rWgjzJibkpUlStdyVEyRxqrp93CELNeve74qSgAdPdz6/KPsXdksKYwlmIvlWsIcDTnzqtHPmeNSKXWrS/Ed+TghL9KiGPG6B/QM9REFH+j6jf8FeIvrJI1r2qlb57AgjLM9jXuYwpO9BJCG9nwRhhF3+8CVPfRQ9ZF5RWU4IlBYVW2S145AjIBIVJYnrxQ1bNedkkd7JHyYDXXFUfplbFrJPdbdFkoZYpz1Fw1vv6L1xM8F+1LeSll1rxyknMklYMiTshvIJ7+5wZp9rOjbfVWNg4jvRcNWVG1kYwhAwTw6rDy/Q3U3aQPvepAWUfqHGI6cx9qOC5Lr6IQ8ZlbgWP4sJtmNJoC4E1vJ49ErhHl2/cRZqYOwRtZrfNlwM115MyIdDDSLEsD35JkUiYRfv0DxY3nx2HQKir1YHt70BV68WLyDz8eRifwalWpSVI0h210u+s4tOzkmHxTGhgx8R w3GejwY0 GSfHKv/sKeu2vpWcsGYBJATucxexHQhZppc+Fb+Sd8fz4GxAzV3vvuDa7tswcqI3xqxWTxxJQ2ERjuz3c2Ht3nhdGSmGX8a/8VDoVM4RFrCKtngXJloo0apdUUqm8DnWWmYn4ZjXmNYVOQ0Zd3mV+iFSz4Q4RD9R3CtNOKGAJLVpqF2m4hKKoUSw1gvq2nOLdRxs35DPGhDzsqmiiX+qgIwlUWASIJAHjrAP6ROHZ26MaJx+fYI3CImyik9REVqsc1uZx1uKwtCrnjGfpBqdP5aNTEIg+qhlOjTZDQwyrbqTqxMchLwEAJ+zCrDCvfVBw7TSMqv6j2qJ+dknATsjtT8ZgRVpdJLqKVrmXvtKN9+xeiJr5faPp50QnhJuIn0ySDFZaabfXhq72em5Ssv/SDRAGAM1f4/iArx7adTZPA03K1dzWS9WxsokFHWtmhtIUlJTHiMAda7j1IJbr+scxFE4c7SsM67V5Jfppcy0KgwwDTe8pIIFidrWtVJnPjKRBeEp0idQwVKpmutCmRAMUJWSCgj7hbfVO06F0IGYmWnQ1ZovAd28k5lTn8x5Ht/60ItkUAGWgbAcAMSfbFrU3jRJwg1cmlPF62cKZFbcsZRziGWWq39DL9z0pYw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This test now has two modes of operation: 1. (default) To check how much vCPU performance was affected by access tracking (previously existed, now supports MGLRU aging). 2. (-p) To also benchmark how fast MGLRU can do aging while vCPUs are faulting in memory. Mode (1) also serves as a way to verify that aging is working properly for pages only accessed by KVM. It will fail if one does not have the 0x8 lru_gen feature bit. To support MGLRU, the test creates a memory cgroup, moves itself into it, then uses the lru_gen debugfs output to track memory in that cgroup. The logic to parse the lru_gen debugfs output has been put into selftests/kvm/lib/lru_gen_util.c. Co-developed-by: Axel Rasmussen Signed-off-by: Axel Rasmussen Signed-off-by: James Houghton --- tools/testing/selftests/kvm/Makefile | 1 + .../selftests/kvm/access_tracking_perf_test.c | 369 +++++++++++++++-- .../selftests/kvm/include/lru_gen_util.h | 55 +++ .../testing/selftests/kvm/lib/lru_gen_util.c | 391 ++++++++++++++++++ 4 files changed, 786 insertions(+), 30 deletions(-) create mode 100644 tools/testing/selftests/kvm/include/lru_gen_util.h create mode 100644 tools/testing/selftests/kvm/lib/lru_gen_util.c diff --git a/tools/testing/selftests/kvm/Makefile b/tools/testing/selftests/kvm/Makefile index 4b3a58d3d473..4b89ab5aff43 100644 --- a/tools/testing/selftests/kvm/Makefile +++ b/tools/testing/selftests/kvm/Makefile @@ -22,6 +22,7 @@ LIBKVM += lib/elf.c LIBKVM += lib/guest_modes.c LIBKVM += lib/io.c LIBKVM += lib/kvm_util.c +LIBKVM += lib/lru_gen_util.c LIBKVM += lib/memstress.c LIBKVM += lib/guest_sprintf.c LIBKVM += lib/rbtree.c diff --git a/tools/testing/selftests/kvm/access_tracking_perf_test.c b/tools/testing/selftests/kvm/access_tracking_perf_test.c index 3c7defd34f56..6ff64ac349a9 100644 --- a/tools/testing/selftests/kvm/access_tracking_perf_test.c +++ b/tools/testing/selftests/kvm/access_tracking_perf_test.c @@ -38,6 +38,7 @@ #include #include #include +#include #include #include #include @@ -47,6 +48,20 @@ #include "memstress.h" #include "guest_modes.h" #include "processor.h" +#include "lru_gen_util.h" + +static const char *TEST_MEMCG_NAME = "access_tracking_perf_test"; +static const int LRU_GEN_ENABLED = 0x1; +static const int LRU_GEN_MM_WALK = 0x2; +static const int LRU_GEN_SECONDARY_MMU_WALK = 0x8; +static const char *CGROUP_PROCS = "cgroup.procs"; +/* + * If using MGLRU, this test assumes a cgroup v2 or cgroup v1 memory hierarchy + * is mounted at cgroup_root. + * + * Can be changed with -r. + */ +static const char *cgroup_root = "/sys/fs/cgroup"; /* Global variable used to synchronize all of the vCPU threads. */ static int iteration; @@ -62,6 +77,9 @@ static enum { /* The iteration that was last completed by each vCPU. */ static int vcpu_last_completed_iteration[KVM_MAX_VCPUS]; +/* The time at which the last iteration was completed */ +static struct timespec vcpu_last_completed_time[KVM_MAX_VCPUS]; + /* Whether to overlap the regions of memory vCPUs access. */ static bool overlap_memory_access; @@ -74,6 +92,12 @@ struct test_params { /* The number of vCPUs to create in the VM. */ int nr_vcpus; + + /* Whether to use lru_gen aging instead of idle page tracking. */ + bool lru_gen; + + /* Whether to test the performance of aging itself. */ + bool benchmark_lru_gen; }; static uint64_t pread_uint64(int fd, const char *filename, uint64_t index) @@ -89,6 +113,50 @@ static uint64_t pread_uint64(int fd, const char *filename, uint64_t index) } +static void write_file_long(const char *path, long v) +{ + FILE *f; + + f = fopen(path, "w"); + TEST_ASSERT(f, "fopen(%s) failed", path); + TEST_ASSERT(fprintf(f, "%ld\n", v) > 0, + "fprintf to %s failed", path); + TEST_ASSERT(!fclose(f), "fclose(%s) failed", path); +} + +static char *path_join(const char *parent, const char *child) +{ + char *out = NULL; + + return asprintf(&out, "%s/%s", parent, child) >= 0 ? out : NULL; +} + +static char *memcg_path(const char *memcg) +{ + return path_join(cgroup_root, memcg); +} + +static char *memcg_file_path(const char *memcg, const char *file) +{ + char *mp = memcg_path(memcg); + char *fp; + + if (!mp) + return NULL; + fp = path_join(mp, file); + free(mp); + return fp; +} + +static void move_to_memcg(const char *memcg, pid_t pid) +{ + char *procs = memcg_file_path(memcg, CGROUP_PROCS); + + TEST_ASSERT(procs, "Failed to construct cgroup.procs path"); + write_file_long(procs, pid); + free(procs); +} + #define PAGEMAP_PRESENT (1ULL << 63) #define PAGEMAP_PFN_MASK ((1ULL << 55) - 1) @@ -242,6 +310,8 @@ static void vcpu_thread_main(struct memstress_vcpu_args *vcpu_args) }; vcpu_last_completed_iteration[vcpu_idx] = current_iteration; + clock_gettime(CLOCK_MONOTONIC, + &vcpu_last_completed_time[vcpu_idx]); } } @@ -253,38 +323,68 @@ static void spin_wait_for_vcpu(int vcpu_idx, int target_iteration) } } +static bool all_vcpus_done(int target_iteration, int nr_vcpus) +{ + for (int i = 0; i < nr_vcpus; ++i) + if (READ_ONCE(vcpu_last_completed_iteration[i]) != + target_iteration) + return false; + + return true; +} + /* The type of memory accesses to perform in the VM. */ enum access_type { ACCESS_READ, ACCESS_WRITE, }; -static void run_iteration(struct kvm_vm *vm, int nr_vcpus, const char *description) +static void run_iteration(struct kvm_vm *vm, int nr_vcpus, const char *description, + bool wait) { - struct timespec ts_start; - struct timespec ts_elapsed; int next_iteration, i; /* Kick off the vCPUs by incrementing iteration. */ next_iteration = ++iteration; - clock_gettime(CLOCK_MONOTONIC, &ts_start); - /* Wait for all vCPUs to finish the iteration. */ - for (i = 0; i < nr_vcpus; i++) - spin_wait_for_vcpu(i, next_iteration); + if (wait) { + struct timespec ts_start; + struct timespec ts_elapsed; + + clock_gettime(CLOCK_MONOTONIC, &ts_start); - ts_elapsed = timespec_elapsed(ts_start); - pr_info("%-30s: %ld.%09lds\n", - description, ts_elapsed.tv_sec, ts_elapsed.tv_nsec); + for (i = 0; i < nr_vcpus; i++) + spin_wait_for_vcpu(i, next_iteration); + + ts_elapsed = timespec_elapsed(ts_start); + + pr_info("%-30s: %ld.%09lds\n", + description, ts_elapsed.tv_sec, ts_elapsed.tv_nsec); + } else + pr_info("%-30s\n", description); } -static void access_memory(struct kvm_vm *vm, int nr_vcpus, - enum access_type access, const char *description) +static void _access_memory(struct kvm_vm *vm, int nr_vcpus, + enum access_type access, const char *description, + bool wait) { memstress_set_write_percent(vm, (access == ACCESS_READ) ? 0 : 100); iteration_work = ITERATION_ACCESS_MEMORY; - run_iteration(vm, nr_vcpus, description); + run_iteration(vm, nr_vcpus, description, wait); +} + +static void access_memory(struct kvm_vm *vm, int nr_vcpus, + enum access_type access, const char *description) +{ + return _access_memory(vm, nr_vcpus, access, description, true); +} + +static void access_memory_async(struct kvm_vm *vm, int nr_vcpus, + enum access_type access, + const char *description) +{ + return _access_memory(vm, nr_vcpus, access, description, false); } static void mark_memory_idle(struct kvm_vm *vm, int nr_vcpus) @@ -297,19 +397,115 @@ static void mark_memory_idle(struct kvm_vm *vm, int nr_vcpus) */ pr_debug("Marking VM memory idle (slow)...\n"); iteration_work = ITERATION_MARK_IDLE; - run_iteration(vm, nr_vcpus, "Mark memory idle"); + run_iteration(vm, nr_vcpus, "Mark memory idle", true); } -static void run_test(enum vm_guest_mode mode, void *arg) +static void create_memcg(const char *memcg) +{ + const char *full_memcg_path = memcg_path(memcg); + int ret; + + TEST_ASSERT(full_memcg_path, "Failed to construct full memcg path"); +retry: + ret = mkdir(full_memcg_path, 0755); + if (ret && errno == EEXIST) { + TEST_ASSERT(!rmdir(full_memcg_path), + "Found existing memcg at %s, but rmdir failed", + full_memcg_path); + goto retry; + } + TEST_ASSERT(!ret, "Creating the memcg failed: mkdir(%s) failed", + full_memcg_path); + + pr_info("Created memcg at %s\n", full_memcg_path); +} + +/* + * Test lru_gen aging speed while vCPUs are faulting memory in. + * + * This test will run lru_gen aging until the vCPUs have finished all of + * the faulting work, reporting: + * - vcpu wall time (wall time for slowest vCPU) + * - average aging pass duration + * - total number of aging passes + * - total time spent aging + * + * This test produces the most useful results when the vcpu wall time and the + * total time spent aging are similar (i.e., we want to avoid timing aging + * while the vCPUs aren't doing any work). + */ +static void run_benchmark(enum vm_guest_mode mode, struct kvm_vm *vm, + struct test_params *params) { - struct test_params *params = arg; - struct kvm_vm *vm; int nr_vcpus = params->nr_vcpus; + struct memcg_stats stats; + struct timespec ts_start, ts_max, ts_vcpus_elapsed, + ts_aging_elapsed, ts_aging_elapsed_avg; + int num_passes = 0; - vm = memstress_create_vm(mode, nr_vcpus, params->vcpu_memory_bytes, 1, - params->backing_src, !overlap_memory_access); + printf("Running lru_gen benchmark...\n"); - memstress_start_vcpu_threads(nr_vcpus, vcpu_thread_main); + clock_gettime(CLOCK_MONOTONIC, &ts_start); + access_memory_async(vm, nr_vcpus, ACCESS_WRITE, + "Populating memory (async)"); + while (!all_vcpus_done(iteration, nr_vcpus)) { + lru_gen_do_aging_quiet(&stats, TEST_MEMCG_NAME); + ++num_passes; + } + + ts_aging_elapsed = timespec_elapsed(ts_start); + ts_aging_elapsed_avg = timespec_div(ts_aging_elapsed, num_passes); + + /* Find out when the slowest vCPU finished. */ + ts_max = ts_start; + for (int i = 0; i < nr_vcpus; ++i) { + struct timespec *vcpu_ts = &vcpu_last_completed_time[i]; + + if (ts_max.tv_sec < vcpu_ts->tv_sec || + (ts_max.tv_sec == vcpu_ts->tv_sec && + ts_max.tv_nsec < vcpu_ts->tv_nsec)) + ts_max = *vcpu_ts; + } + + ts_vcpus_elapsed = timespec_sub(ts_max, ts_start); + + pr_info("%-30s: %ld.%09lds\n", "vcpu wall time", + ts_vcpus_elapsed.tv_sec, ts_vcpus_elapsed.tv_nsec); + + pr_info("%-30s: %ld.%09lds, (passes:%d, total:%ld.%09lds)\n", + "lru_gen avg pass duration", + ts_aging_elapsed_avg.tv_sec, + ts_aging_elapsed_avg.tv_nsec, + num_passes, + ts_aging_elapsed.tv_sec, + ts_aging_elapsed.tv_nsec); +} + +/* + * Test how much access tracking affects vCPU performance. + * + * Supports two modes of access tracking: + * - idle page tracking + * - lru_gen aging + * + * When using lru_gen, this test additionally verifies that the pages are in + * fact getting younger and older, otherwise the performance data would be + * invalid. + * + * The forced lru_gen aging can race with aging that occurs naturally. + */ +static void run_test(enum vm_guest_mode mode, struct kvm_vm *vm, + struct test_params *params) +{ + int nr_vcpus = params->nr_vcpus; + bool lru_gen = params->lru_gen; + struct memcg_stats stats; + // If guest_page_size is larger than the host's page size, the + // guest (memstress) will only fault in a subset of the host's pages. + long total_pages = nr_vcpus * params->vcpu_memory_bytes / + max(memstress_args.guest_page_size, + (uint64_t)getpagesize()); + int found_gens[5]; pr_info("\n"); access_memory(vm, nr_vcpus, ACCESS_WRITE, "Populating memory"); @@ -319,11 +515,78 @@ static void run_test(enum vm_guest_mode mode, void *arg) access_memory(vm, nr_vcpus, ACCESS_READ, "Reading from populated memory"); /* Repeat on memory that has been marked as idle. */ - mark_memory_idle(vm, nr_vcpus); + if (lru_gen) { + /* Do an initial page table scan */ + lru_gen_do_aging(&stats, TEST_MEMCG_NAME); + TEST_ASSERT(sum_memcg_stats(&stats) >= total_pages, + "Not all pages tracked in lru_gen stats.\n" + "Is lru_gen enabled? Did the memcg get created properly?"); + + /* Find the generation we're currently in (probably youngest) */ + found_gens[0] = lru_gen_find_generation(&stats, total_pages); + + /* Do an aging pass now */ + lru_gen_do_aging(&stats, TEST_MEMCG_NAME); + + /* Same generation, but a newer generation has been made */ + found_gens[1] = lru_gen_find_generation(&stats, total_pages); + TEST_ASSERT(found_gens[1] == found_gens[0], + "unexpected gen change: %d vs. %d", + found_gens[1], found_gens[0]); + } else + mark_memory_idle(vm, nr_vcpus); + access_memory(vm, nr_vcpus, ACCESS_WRITE, "Writing to idle memory"); - mark_memory_idle(vm, nr_vcpus); + + if (lru_gen) { + /* Scan the page tables again */ + lru_gen_do_aging(&stats, TEST_MEMCG_NAME); + + /* The pages should now be young again, so in a newer generation */ + found_gens[2] = lru_gen_find_generation(&stats, total_pages); + TEST_ASSERT(found_gens[2] > found_gens[1], + "pages did not get younger"); + + /* Do another aging pass */ + lru_gen_do_aging(&stats, TEST_MEMCG_NAME); + + /* Same generation; new generation has been made */ + found_gens[3] = lru_gen_find_generation(&stats, total_pages); + TEST_ASSERT(found_gens[3] == found_gens[2], + "unexpected gen change: %d vs. %d", + found_gens[3], found_gens[2]); + } else + mark_memory_idle(vm, nr_vcpus); + access_memory(vm, nr_vcpus, ACCESS_READ, "Reading from idle memory"); + if (lru_gen) { + /* Scan the pages tables again */ + lru_gen_do_aging(&stats, TEST_MEMCG_NAME); + + /* The pages should now be young again, so in a newer generation */ + found_gens[4] = lru_gen_find_generation(&stats, total_pages); + TEST_ASSERT(found_gens[4] > found_gens[3], + "pages did not get younger"); + } +} + +static void setup_vm_and_run(enum vm_guest_mode mode, void *arg) +{ + struct test_params *params = arg; + int nr_vcpus = params->nr_vcpus; + struct kvm_vm *vm; + + vm = memstress_create_vm(mode, nr_vcpus, params->vcpu_memory_bytes, 1, + params->backing_src, !overlap_memory_access); + + memstress_start_vcpu_threads(nr_vcpus, vcpu_thread_main); + + if (params->benchmark_lru_gen) + run_benchmark(mode, vm, params); + else + run_test(mode, vm, params); + memstress_join_vcpu_threads(nr_vcpus); memstress_destroy_vm(vm); } @@ -331,8 +594,8 @@ static void run_test(enum vm_guest_mode mode, void *arg) static void help(char *name) { puts(""); - printf("usage: %s [-h] [-m mode] [-b vcpu_bytes] [-v vcpus] [-o] [-s mem_type]\n", - name); + printf("usage: %s [-h] [-m mode] [-b vcpu_bytes] [-v vcpus] [-o]" + " [-s mem_type] [-l] [-r memcg_root]\n", name); puts(""); printf(" -h: Display this help message."); guest_modes_help(); @@ -342,6 +605,9 @@ static void help(char *name) printf(" -v: specify the number of vCPUs to run.\n"); printf(" -o: Overlap guest memory accesses instead of partitioning\n" " them into a separate region of memory for each vCPU.\n"); + printf(" -l: Use MGLRU aging instead of idle page tracking\n"); + printf(" -p: Benchmark MGLRU aging while faulting memory in\n"); + printf(" -r: The memory cgroup hierarchy root to use (when -l is given)\n"); backing_src_help("-s"); puts(""); exit(0); @@ -353,13 +619,15 @@ int main(int argc, char *argv[]) .backing_src = DEFAULT_VM_MEM_SRC, .vcpu_memory_bytes = DEFAULT_PER_VCPU_MEM_SIZE, .nr_vcpus = 1, + .lru_gen = false, + .benchmark_lru_gen = false, }; int page_idle_fd; int opt; guest_modes_append_default(); - while ((opt = getopt(argc, argv, "hm:b:v:os:")) != -1) { + while ((opt = getopt(argc, argv, "hm:b:v:os:lr:p")) != -1) { switch (opt) { case 'm': guest_modes_cmdline(optarg); @@ -376,6 +644,15 @@ int main(int argc, char *argv[]) case 's': params.backing_src = parse_backing_src_type(optarg); break; + case 'l': + params.lru_gen = true; + break; + case 'p': + params.benchmark_lru_gen = true; + break; + case 'r': + cgroup_root = strdup(optarg); + break; case 'h': default: help(argv[0]); @@ -383,12 +660,44 @@ int main(int argc, char *argv[]) } } - page_idle_fd = open("/sys/kernel/mm/page_idle/bitmap", O_RDWR); - __TEST_REQUIRE(page_idle_fd >= 0, - "CONFIG_IDLE_PAGE_TRACKING is not enabled"); - close(page_idle_fd); + if (!params.lru_gen) { + page_idle_fd = open("/sys/kernel/mm/page_idle/bitmap", O_RDWR); + __TEST_REQUIRE(page_idle_fd >= 0, + "CONFIG_IDLE_PAGE_TRACKING is not enabled"); + close(page_idle_fd); + } else { + int lru_gen_fd, lru_gen_debug_fd; + long mglru_features; + char mglru_feature_str[8] = {}; + + lru_gen_fd = open("/sys/kernel/mm/lru_gen/enabled", O_RDONLY); + __TEST_REQUIRE(lru_gen_fd >= 0, + "CONFIG_LRU_GEN is not enabled"); + TEST_ASSERT(read(lru_gen_fd, &mglru_feature_str, 7) > 0, + "couldn't read lru_gen features"); + mglru_features = strtol(mglru_feature_str, NULL, 16); + __TEST_REQUIRE(mglru_features & LRU_GEN_ENABLED, + "lru_gen is not enabled"); + __TEST_REQUIRE(mglru_features & LRU_GEN_MM_WALK, + "lru_gen does not support MM_WALK"); + __TEST_REQUIRE(mglru_features & LRU_GEN_SECONDARY_MMU_WALK, + "lru_gen does not support SECONDARY_MMU_WALK"); + + lru_gen_debug_fd = open(DEBUGFS_LRU_GEN, O_RDWR); + __TEST_REQUIRE(lru_gen_debug_fd >= 0, + "Cannot access %s", DEBUGFS_LRU_GEN); + close(lru_gen_debug_fd); + } + + TEST_ASSERT(!params.benchmark_lru_gen || params.lru_gen, + "-p specified without -l"); + + if (params.lru_gen) { + create_memcg(TEST_MEMCG_NAME); + move_to_memcg(TEST_MEMCG_NAME, getpid()); + } - for_each_guest_mode(run_test, ¶ms); + for_each_guest_mode(setup_vm_and_run, ¶ms); return 0; } diff --git a/tools/testing/selftests/kvm/include/lru_gen_util.h b/tools/testing/selftests/kvm/include/lru_gen_util.h new file mode 100644 index 000000000000..4eef8085a3cb --- /dev/null +++ b/tools/testing/selftests/kvm/include/lru_gen_util.h @@ -0,0 +1,55 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Tools for integrating with lru_gen, like parsing the lru_gen debugfs output. + * + * Copyright (C) 2024, Google LLC. + */ +#ifndef SELFTEST_KVM_LRU_GEN_UTIL_H +#define SELFTEST_KVM_LRU_GEN_UTIL_H + +#include +#include +#include + +#include "test_util.h" + +#define MAX_NR_GENS 16 /* MAX_NR_GENS in include/linux/mmzone.h */ +#define MAX_NR_NODES 4 /* Maximum number of nodes we support */ + +static const char *DEBUGFS_LRU_GEN = "/sys/kernel/debug/lru_gen"; + +struct generation_stats { + int gen; + long age_ms; + long nr_anon; + long nr_file; +}; + +struct node_stats { + int node; + int nr_gens; /* Number of populated gens entries. */ + struct generation_stats gens[MAX_NR_GENS]; +}; + +struct memcg_stats { + unsigned long memcg_id; + int nr_nodes; /* Number of populated nodes entries. */ + struct node_stats nodes[MAX_NR_NODES]; +}; + +void print_memcg_stats(const struct memcg_stats *stats, const char *name); + +void read_memcg_stats(struct memcg_stats *stats, const char *memcg); + +void read_print_memcg_stats(struct memcg_stats *stats, const char *memcg); + +long sum_memcg_stats(const struct memcg_stats *stats); + +void lru_gen_do_aging(struct memcg_stats *stats, const char *memcg); + +void lru_gen_do_aging_quiet(struct memcg_stats *stats, const char *memcg); + +int lru_gen_find_generation(const struct memcg_stats *stats, + unsigned long total_pages); + +#endif /* SELFTEST_KVM_LRU_GEN_UTIL_H */ diff --git a/tools/testing/selftests/kvm/lib/lru_gen_util.c b/tools/testing/selftests/kvm/lib/lru_gen_util.c new file mode 100644 index 000000000000..3c02a635a9f7 --- /dev/null +++ b/tools/testing/selftests/kvm/lib/lru_gen_util.c @@ -0,0 +1,391 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2024, Google LLC. + */ + +#include + +#include "lru_gen_util.h" + +/* + * Tracks state while we parse memcg lru_gen stats. The file we're parsing is + * structured like this (some extra whitespace elided): + * + * memcg (id) (path) + * node (id) + * (gen_nr) (age_in_ms) (nr_anon_pages) (nr_file_pages) + */ +struct memcg_stats_parse_context { + bool consumed; /* Whether or not this line was consumed */ + /* Next parse handler to invoke */ + void (*next_handler)(struct memcg_stats *, + struct memcg_stats_parse_context *, char *); + int current_node_idx; /* Current index in nodes array */ + const char *name; /* The name of the memcg we're looking for */ +}; + +static void memcg_stats_handle_searching(struct memcg_stats *stats, + struct memcg_stats_parse_context *ctx, + char *line); +static void memcg_stats_handle_in_memcg(struct memcg_stats *stats, + struct memcg_stats_parse_context *ctx, + char *line); +static void memcg_stats_handle_in_node(struct memcg_stats *stats, + struct memcg_stats_parse_context *ctx, + char *line); + +struct split_iterator { + char *str; + char *save; +}; + +static char *split_next(struct split_iterator *it) +{ + char *ret = strtok_r(it->str, " \t\n\r", &it->save); + + it->str = NULL; + return ret; +} + +static void memcg_stats_handle_searching(struct memcg_stats *stats, + struct memcg_stats_parse_context *ctx, + char *line) +{ + struct split_iterator it = { .str = line }; + char *prefix = split_next(&it); + char *memcg_id = split_next(&it); + char *memcg_name = split_next(&it); + char *end; + + ctx->consumed = true; + + if (!prefix || strcmp("memcg", prefix)) + return; /* Not a memcg line (maybe empty), skip */ + + TEST_ASSERT(memcg_id && memcg_name, + "malformed memcg line; no memcg id or memcg_name"); + + if (strcmp(memcg_name + 1, ctx->name)) + return; /* Wrong memcg, skip */ + + /* Found it! */ + + stats->memcg_id = strtoul(memcg_id, &end, 10); + TEST_ASSERT(*end == '\0', "malformed memcg id '%s'", memcg_id); + if (!stats->memcg_id) + return; /* Removed memcg? */ + + ctx->next_handler = memcg_stats_handle_in_memcg; +} + +static void memcg_stats_handle_in_memcg(struct memcg_stats *stats, + struct memcg_stats_parse_context *ctx, + char *line) +{ + struct split_iterator it = { .str = line }; + char *prefix = split_next(&it); + char *id = split_next(&it); + long found_node_id; + char *end; + + ctx->consumed = true; + ctx->current_node_idx = -1; + + if (!prefix) + return; /* Skip empty lines */ + + if (!strcmp("memcg", prefix)) { + /* Memcg done, found next one; stop. */ + ctx->next_handler = NULL; + return; + } else if (strcmp("node", prefix)) + TEST_ASSERT(false, "found malformed line after 'memcg ...'," + "token: '%s'", prefix); + + /* At this point we know we have a node line. Parse the ID. */ + + TEST_ASSERT(id, "malformed node line; no node id"); + + found_node_id = strtol(id, &end, 10); + TEST_ASSERT(*end == '\0', "malformed node id '%s'", id); + + ctx->current_node_idx = stats->nr_nodes++; + TEST_ASSERT(ctx->current_node_idx < MAX_NR_NODES, + "memcg has stats for too many nodes, max is %d", + MAX_NR_NODES); + stats->nodes[ctx->current_node_idx].node = found_node_id; + + ctx->next_handler = memcg_stats_handle_in_node; +} + +static void memcg_stats_handle_in_node(struct memcg_stats *stats, + struct memcg_stats_parse_context *ctx, + char *line) +{ + /* Have to copy since we might not consume */ + char *my_line = strdup(line); + struct split_iterator it = { .str = my_line }; + char *gen, *age, *nr_anon, *nr_file; + struct node_stats *node_stats; + struct generation_stats *gen_stats; + char *end; + + TEST_ASSERT(it.str, "failed to copy input line"); + + gen = split_next(&it); + + /* Skip empty lines */ + if (!gen) + goto out_consume; /* Skip empty lines */ + + if (!strcmp("memcg", gen) || !strcmp("node", gen)) { + /* + * Reached next memcg or node section. Don't consume, let the + * other handler deal with this. + */ + ctx->next_handler = memcg_stats_handle_in_memcg; + goto out; + } + + node_stats = &stats->nodes[ctx->current_node_idx]; + TEST_ASSERT(node_stats->nr_gens < MAX_NR_GENS, + "found too many generation lines; max is %d", + MAX_NR_GENS); + gen_stats = &node_stats->gens[node_stats->nr_gens++]; + + age = split_next(&it); + nr_anon = split_next(&it); + nr_file = split_next(&it); + + TEST_ASSERT(age && nr_anon && nr_file, + "malformed generation line; not enough tokens"); + + gen_stats->gen = (int)strtol(gen, &end, 10); + TEST_ASSERT(*end == '\0', "malformed generation number '%s'", gen); + + gen_stats->age_ms = strtol(age, &end, 10); + TEST_ASSERT(*end == '\0', "malformed generation age '%s'", age); + + gen_stats->nr_anon = strtol(nr_anon, &end, 10); + TEST_ASSERT(*end == '\0', "malformed anonymous page count '%s'", + nr_anon); + + gen_stats->nr_file = strtol(nr_file, &end, 10); + TEST_ASSERT(*end == '\0', "malformed file page count '%s'", nr_file); + +out_consume: + ctx->consumed = true; +out: + free(my_line); +} + +/* Pretty-print lru_gen @stats. */ +void print_memcg_stats(const struct memcg_stats *stats, const char *name) +{ + int node, gen; + + fprintf(stderr, "stats for memcg %s (id %lu):\n", + name, stats->memcg_id); + for (node = 0; node < stats->nr_nodes; ++node) { + fprintf(stderr, "\tnode %d\n", stats->nodes[node].node); + for (gen = 0; gen < stats->nodes[node].nr_gens; ++gen) { + const struct generation_stats *gstats = + &stats->nodes[node].gens[gen]; + + fprintf(stderr, + "\t\tgen %d\tage_ms %ld" + "\tnr_anon %ld\tnr_file %ld\n", + gstats->gen, gstats->age_ms, gstats->nr_anon, + gstats->nr_file); + } + } +} + +/* Re-read lru_gen debugfs information for @memcg into @stats. */ +void read_memcg_stats(struct memcg_stats *stats, const char *memcg) +{ + FILE *f; + ssize_t read = 0; + char *line = NULL; + size_t bufsz; + struct memcg_stats_parse_context ctx = { + .next_handler = memcg_stats_handle_searching, + .name = memcg, + }; + + memset(stats, 0, sizeof(struct memcg_stats)); + + f = fopen(DEBUGFS_LRU_GEN, "r"); + TEST_ASSERT(f, "fopen(%s) failed", DEBUGFS_LRU_GEN); + + while (ctx.next_handler && (read = getline(&line, &bufsz, f)) > 0) { + ctx.consumed = false; + + do { + ctx.next_handler(stats, &ctx, line); + if (!ctx.next_handler) + break; + } while (!ctx.consumed); + } + + if (read < 0 && !feof(f)) + TEST_ASSERT(false, "getline(%s) failed", DEBUGFS_LRU_GEN); + + TEST_ASSERT(stats->memcg_id > 0, "Couldn't find memcg: %s\n" + "Did the memcg get created in the proper mount?", + memcg); + if (line) + free(line); + TEST_ASSERT(!fclose(f), "fclose(%s) failed", DEBUGFS_LRU_GEN); +} + +/* + * Find all pages tracked by lru_gen for this memcg in generation @target_gen. + * + * If @target_gen is negative, look for all generations. + */ +static long sum_memcg_stats_for_gen(int target_gen, + const struct memcg_stats *stats) +{ + int node, gen; + long total_nr = 0; + + for (node = 0; node < stats->nr_nodes; ++node) { + const struct node_stats *node_stats = &stats->nodes[node]; + + for (gen = 0; gen < node_stats->nr_gens; ++gen) { + const struct generation_stats *gen_stats = + &node_stats->gens[gen]; + + if (target_gen >= 0 && gen_stats->gen != target_gen) + continue; + + total_nr += gen_stats->nr_anon + gen_stats->nr_file; + } + } + + return total_nr; +} + +/* Find all pages tracked by lru_gen for this memcg. */ +long sum_memcg_stats(const struct memcg_stats *stats) +{ + return sum_memcg_stats_for_gen(-1, stats); +} + +/* Read the memcg stats and optionally print if this is a debug build. */ +void read_print_memcg_stats(struct memcg_stats *stats, const char *memcg) +{ + read_memcg_stats(stats, memcg); +#ifdef DEBUG + print_memcg_stats(stats, memcg); +#endif +} + +/* + * If lru_gen aging should force page table scanning. + * + * If you want to set this to false, you will need to do eviction + * before doing extra aging passes. + */ +static const bool force_scan = true; + +static void run_aging_impl(unsigned long memcg_id, int node_id, int max_gen) +{ + FILE *f = fopen(DEBUGFS_LRU_GEN, "w"); + char *command; + size_t sz; + + TEST_ASSERT(f, "fopen(%s) failed", DEBUGFS_LRU_GEN); + sz = asprintf(&command, "+ %lu %d %d 1 %d\n", + memcg_id, node_id, max_gen, force_scan); + TEST_ASSERT(sz > 0, "creating aging command failed"); + + pr_debug("Running aging command: %s", command); + if (fwrite(command, sizeof(char), sz, f) < sz) { + TEST_ASSERT(false, "writing aging command %s to %s failed", + command, DEBUGFS_LRU_GEN); + } + + TEST_ASSERT(!fclose(f), "fclose(%s) failed", DEBUGFS_LRU_GEN); +} + +static void _lru_gen_do_aging(struct memcg_stats *stats, const char *memcg, + bool verbose) +{ + int node, gen; + struct timespec ts_start; + struct timespec ts_elapsed; + + pr_debug("lru_gen: invoking aging...\n"); + + /* Must read memcg stats to construct the proper aging command. */ + read_print_memcg_stats(stats, memcg); + + if (verbose) + clock_gettime(CLOCK_MONOTONIC, &ts_start); + + for (node = 0; node < stats->nr_nodes; ++node) { + int max_gen = 0; + + for (gen = 0; gen < stats->nodes[node].nr_gens; ++gen) { + int this_gen = stats->nodes[node].gens[gen].gen; + + max_gen = max_gen > this_gen ? max_gen : this_gen; + } + + run_aging_impl(stats->memcg_id, stats->nodes[node].node, + max_gen); + } + + if (verbose) { + ts_elapsed = timespec_elapsed(ts_start); + pr_info("%-30s: %ld.%09lds\n", "lru_gen: Aging", + ts_elapsed.tv_sec, ts_elapsed.tv_nsec); + } + + /* Re-read so callers get updated information */ + read_print_memcg_stats(stats, memcg); +} + +/* Do aging, and print how long it took. */ +void lru_gen_do_aging(struct memcg_stats *stats, const char *memcg) +{ + return _lru_gen_do_aging(stats, memcg, true); +} + +/* Do aging, don't print anything. */ +void lru_gen_do_aging_quiet(struct memcg_stats *stats, const char *memcg) +{ + return _lru_gen_do_aging(stats, memcg, false); +} + +/* + * Find which generation contains more than half of @total_pages, assuming that + * such a generation exists. + */ +int lru_gen_find_generation(const struct memcg_stats *stats, + unsigned long total_pages) +{ + int node, gen, gen_idx, min_gen = INT_MAX, max_gen = -1; + + for (node = 0; node < stats->nr_nodes; ++node) + for (gen_idx = 0; gen_idx < stats->nodes[node].nr_gens; + ++gen_idx) { + gen = stats->nodes[node].gens[gen_idx].gen; + max_gen = gen > max_gen ? gen : max_gen; + min_gen = gen < min_gen ? gen : min_gen; + } + + for (gen = min_gen; gen < max_gen; ++gen) + /* See if the most pages are in this generation. */ + if (sum_memcg_stats_for_gen(gen, stats) > + total_pages / 2) + return gen; + + TEST_ASSERT(false, "No generation includes majority of %lu pages.", + total_pages); + + /* unreachable, but make the compiler happy */ + return -1; +}