From patchwork Thu Sep 26 01:35:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13812687 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1DC7DCCFA10 for ; Thu, 26 Sep 2024 01:35:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E301B6B00C4; Wed, 25 Sep 2024 21:35:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DB3F26B00C6; Wed, 25 Sep 2024 21:35:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C076A6B00C7; Wed, 25 Sep 2024 21:35:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 997486B00C4 for ; Wed, 25 Sep 2024 21:35:33 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 56AEA120741 for ; Thu, 26 Sep 2024 01:35:33 +0000 (UTC) X-FDA: 82605172146.12.762156F Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf12.hostedemail.com (Postfix) with ESMTP id 8E14140007 for ; Thu, 26 Sep 2024 01:35:31 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=TLY+tJJX; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf12.hostedemail.com: domain of 3Yrr0ZgoKCOcScQXdPQcXWPXXPUN.LXVURWdg-VVTeJLT.XaP@flex--jthoughton.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3Yrr0ZgoKCOcScQXdPQcXWPXXPUN.LXVURWdg-VVTeJLT.XaP@flex--jthoughton.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1727314442; a=rsa-sha256; cv=none; b=CwizTBKyZrk/uMB250kuI5XrbsLVbsYzzdcxaVQkp3SlLkge6OcapJGm38592dxXfVc/Y+ 5hfsI77MoS0r+mtXI6fqAQIKNM2/kJVI1LonBTBjwEbrulZkFAP+X1Q5jFhVYtti10bIUG OsdkQBu56Mfy/Z+maCkjOiKiQUAB8g0= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=TLY+tJJX; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf12.hostedemail.com: domain of 3Yrr0ZgoKCOcScQXdPQcXWPXXPUN.LXVURWdg-VVTeJLT.XaP@flex--jthoughton.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3Yrr0ZgoKCOcScQXdPQcXWPXXPUN.LXVURWdg-VVTeJLT.XaP@flex--jthoughton.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1727314442; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ogisB+p0t7E+MEmwUL2EdeyMfss1r+UivrCLQXciZ0o=; b=XApiFHBFSdNUJWGjWmEoYPzz9tMjffNkKdanikkyi0FhVOJHRcBiUJHohASp4hsDOlmJPT PLVC7hLhE3H3U3dTJp1NvnFhdrnqjdYTYglvg05QI2YX690EYfdmwFyba2oppaAtRs8mue mLgt4Y7DJbrlnzNvtFPy1NRGfrJFW9U= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-6e2261adfdeso10933357b3.2 for ; Wed, 25 Sep 2024 18:35:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1727314531; x=1727919331; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ogisB+p0t7E+MEmwUL2EdeyMfss1r+UivrCLQXciZ0o=; b=TLY+tJJXdCjLfPM1nVWPL38hS+3ei/EuCl//2dG2IWRpeLdcB/TB56eKygsUKVB6/4 e5g6jLKo7GtWnAPpZ5gUNYuZvy73ID5U9pxz3jyFWHiSYNZRzQ88ApGOXSKy0nM1JZ5b VsXJqKswDnpEJTMSDcwcYDUN7ah322WrchZQJT8layaWX5bpqUBDqbGVSV6T/4yiGc3U JX/hT3WKYHESNG7lZdCQaQHGyyfTkQ0vDToqM7vUbgn4Mh+pZ2KcuinRpebWMGkgWlB0 NRokJzbNufl1NyetYBy2Rp82hPlotJ/bzJunrF3cd5nIS5wximSAeBygbsUYUr8q4oBA KPjg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727314531; x=1727919331; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ogisB+p0t7E+MEmwUL2EdeyMfss1r+UivrCLQXciZ0o=; b=MxK0IN1Aao6J+Hp7TkLlLpInkFeHMygBv+f1ByPrNeEmmguw2cU1opiGlqVEV60Hz3 2C7DupsaYmka+e9rduuj/KwCsFuhq7KOQH0q22WaAXb3EvG/TaErVd+eiqufugShR+8P +h43HMrpMkjEO+A5Fe6iTrM0tktIO629tObLNl9yaV1gDGz2X33xU0KyPN3uzO4U2l6C /xQ4kx63iL7cEeqBcFu/15wfiOi3voTj6sENrTVmo4Aamb6wDx4eT/1cgIhalOfw+Uph Baqa4XJgX7vXY/SV9sAlD6SBfCxr0yogexVLoy9cacvR6MZyqam4oh4+PNP22D9aG2/x FUew== X-Forwarded-Encrypted: i=1; AJvYcCVTcafgzcgkH19+STiKwMemE++cc/olLWWVFfQR2RXlX9QiCVOcXUOkBlLcpltE6VP/nKXEhwQJIw==@kvack.org X-Gm-Message-State: AOJu0YxnOrSLVYs9c5Vb7j0SjxijMR1d3pezVHVNPcAN3ynLHaipRqSE LguojKZ9dN1V4qRI3NpzEY/b43HauBxk9x6xvaPUqwahJlqNa5eqXAz/6+nkxsxd+NkkZXECMqn K7GjZXMMCWd+axGtfeg== X-Google-Smtp-Source: AGHT+IEFdzw0O48kZ1pdx1YoauWRvV0zu2R4nMdxsgoP5v/WIUZXOoAYIDJzZnEXzkuO0wTdBSaqUczWpwalNBYg X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:13d:fb22:ac12:a84b]) (user=jthoughton job=sendgmr) by 2002:a05:690c:5292:b0:62c:f976:a763 with SMTP id 00721157ae682-6e21d6c27c9mr136077b3.1.1727314530709; Wed, 25 Sep 2024 18:35:30 -0700 (PDT) Date: Thu, 26 Sep 2024 01:35:01 +0000 In-Reply-To: <20240926013506.860253-1-jthoughton@google.com> Mime-Version: 1.0 References: <20240926013506.860253-1-jthoughton@google.com> X-Mailer: git-send-email 2.46.0.792.g87dc391469-goog Message-ID: <20240926013506.860253-14-jthoughton@google.com> Subject: [PATCH v7 13/18] mm: Add fast_only bool to test_young and clear_young MMU notifiers From: James Houghton To: Sean Christopherson , Paolo Bonzini Cc: Andrew Morton , David Matlack , David Rientjes , James Houghton , Jason Gunthorpe , Jonathan Corbet , Marc Zyngier , Oliver Upton , Wei Xu , Yu Zhao , Axel Rasmussen , kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org X-Rspamd-Queue-Id: 8E14140007 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: pgw9eabmwwwpeui46wjgeqfzzdm3odw3 X-HE-Tag: 1727314531-233135 X-HE-Meta: U2FsdGVkX18m74Jc5/jujk/tBWOStq7F11/iOE2pvXyjOXfORUpIPXBaxyqNELNoVDILWVfOklEKQuoqQv1JjC4IiVp2I1GXkYq11cRgv18PxOjzgyGTE2FwqvudJDqjHzOcrwK+MeIdLEdss+vrbLst9wMoHZ1ShpQXqqAOTebCinoXBU2fFqbhvMoGFhqZCONse1lHqr9b9apbaj1TQlRWwG9u6KU7FhQlx9z1uWlL98YsKn6h+nw9JCQTN9MnaRIOMYEdRXt5JXF+iGNQrCWifhkQIygOuHl8I96s6VWdolTySHvOLtblDigK846YjYn0JixbgtLiVa3lX8tyCESVPVawEpMyZHaRi8qzXn+tfF5tO/CSzKNnmhWQuqoMBfTh/eb0cLn0b2g88hq+VFxnMa3rZ9+eENZ9g5pfcwDqIsmnq/tnLgXFwxyvJZU6WTVNjQ46VQu2vsZTEcg2rGG2lklQg90TDlTdYjC9mbS7izDEgwzFzst9b8/Er/ERAONJCxISX92hFvEeBM5g06sNJmjkyTlXosLy/TWhgax7ZvszOeE9gegDxyC+Ih39+Znt4STDTkgHOfE6EqlTsMM42oZ83HgLcTfX8IDvv5p3Zv4f8wEqmp+neayXlh794D1s5IddikZS5XgT4z7lJIKvxoafpH0LGQ3HQtZY2zv+raqUlem61vPufAOUG/7cpsiVq9a5o6PkaA8edpL9wSwhnEDRjG8mLJuT3JMWRA4E0UAYUb0fePVdDBI0rCpN5rDi4GaLkiun4y8T9n97nr7mGnV5Euz9FVtfG5m0N4kOjOlKafkhgHL/wQ5yvFBb0PPJh3U2qW0454eLWVMmE86emuceg5IbRpkKivKjJxzbgng/yQuZaMyqef0tKDC5E7wwHHK1lUZ4HcXUHXRhgeQUyPbkFBRlrIde1bFA7q3ffj2t54Gb1qvBl1IYzmWXCQ68LUv3IoZ9gKLJzzl RB47M8x/ FTmXYi6bAtN+IcICZJ/d2c0+YsAydIpvZJ1aOGlxFtir8MKg68/Gn5ILWozXjbWAmVYvkYmtP2jCZ1tExTlWJS7OhBZQu2UXrL5gAl5BNgnbqR8e34kXAFGGx6W84NDuZkZxmZYkYb7hXiD7p+u+trO8/tfycq/s5pjRcb5UBfVPBYvXHs+299GbJufpx4qx+bpuG7qbmSbw2sQRgXEa19RnJCpOHhxGn2uJtL5vPsss6u3WcOBq3P/N84HuMXDOAKn2Z137QwpXW/kv7sBygO3yAS28amx8+a/UvBR14WxkMDM6YvkjldFGSH2gl0wo4xv7oXywONa9WymKIquVlXUrocWkC9WVXhDC0iCd+lovnssIi5nWvCqlYVYapJ99JAkV1mLn5094frKG8zeES3dFMsxQ52KUYeSWpKf8M2lBXmPlfxbyvu/1VXnfvYa/o4+Mkq8fHtVEQFDrUlX52rIoEo6DBqaAkek6yCDGDzeod+IOf+DTWWZCjYuGjNLe4U5qJI0QYVKCkJcsEn5YeUch6NlDMqXbV2f+TrSmvBIAA344OQMkp3fIDx9AqP3wbQKVsS2WLtv2MGwdfi0H/BlJI+EGCjumNhTmPCr/10oIalT6S+D+KYFTn4w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: For implementers, the fast_only bool indicates that the age information needs to be harvested such that we do not slow down other MMU operations, and ideally that we are not ourselves slowed down by other MMU operations. Usually this means that the implementation should be lockless. Also add mmu_notifier_test_young_fast_only() and mmu_notifier_clear_young_fast_only() helpers to set fast_only for these notifiers. Signed-off-by: James Houghton --- include/linux/mmu_notifier.h | 61 ++++++++++++++++++++++++++++++++---- include/trace/events/kvm.h | 19 ++++++----- mm/mmu_notifier.c | 18 ++++++++--- virt/kvm/kvm_main.c | 12 ++++--- 4 files changed, 88 insertions(+), 22 deletions(-) diff --git a/include/linux/mmu_notifier.h b/include/linux/mmu_notifier.h index 37643fa43687..7c17e2871c66 100644 --- a/include/linux/mmu_notifier.h +++ b/include/linux/mmu_notifier.h @@ -106,21 +106,38 @@ struct mmu_notifier_ops { * clear_young is a lightweight version of clear_flush_young. Like the * latter, it is supposed to test-and-clear the young/accessed bitflag * in the secondary pte, but it may omit flushing the secondary tlb. + * + * The fast_only parameter indicates that this call should not block, + * and this function should not cause other MMU notifier calls to + * block. Usually this means that the implementation should be + * lockless. + * + * When called with fast_only, this notifier will be a no-op (and + * return that the range is NOT young), unless has_fast_aging is set + * on the struct mmu_notifier. + * + * When fast_only is true, if the implementer cannot determine that a + * range is young without blocking, it should return 0 (i.e., that + * the range is NOT young). */ int (*clear_young)(struct mmu_notifier *subscription, struct mm_struct *mm, unsigned long start, - unsigned long end); + unsigned long end, + bool fast_only); /* * test_young is called to check the young/accessed bitflag in * the secondary pte. This is used to know if the page is * frequently used without actually clearing the flag or tearing * down the secondary mapping on the page. + * + * The fast_only parameter has the same meaning as with clear_young. */ int (*test_young)(struct mmu_notifier *subscription, struct mm_struct *mm, - unsigned long address); + unsigned long address, + bool fast_only); /* * invalidate_range_start() and invalidate_range_end() must be @@ -381,9 +398,11 @@ extern int __mmu_notifier_clear_flush_young(struct mm_struct *mm, unsigned long end); extern int __mmu_notifier_clear_young(struct mm_struct *mm, unsigned long start, - unsigned long end); + unsigned long end, + bool fast_only); extern int __mmu_notifier_test_young(struct mm_struct *mm, - unsigned long address); + unsigned long address, + bool fast_only); extern bool __mm_has_fast_young_notifiers(struct mm_struct *mm); extern int __mmu_notifier_invalidate_range_start(struct mmu_notifier_range *r); extern void __mmu_notifier_invalidate_range_end(struct mmu_notifier_range *r); @@ -418,7 +437,16 @@ static inline int mmu_notifier_clear_young(struct mm_struct *mm, unsigned long end) { if (mm_has_notifiers(mm)) - return __mmu_notifier_clear_young(mm, start, end); + return __mmu_notifier_clear_young(mm, start, end, false); + return 0; +} + +static inline int mmu_notifier_clear_young_fast_only(struct mm_struct *mm, + unsigned long start, + unsigned long end) +{ + if (mm_has_notifiers(mm)) + return __mmu_notifier_clear_young(mm, start, end, true); return 0; } @@ -426,7 +454,15 @@ static inline int mmu_notifier_test_young(struct mm_struct *mm, unsigned long address) { if (mm_has_notifiers(mm)) - return __mmu_notifier_test_young(mm, address); + return __mmu_notifier_test_young(mm, address, false); + return 0; +} + +static inline int mmu_notifier_test_young_fast_only(struct mm_struct *mm, + unsigned long address) +{ + if (mm_has_notifiers(mm)) + return __mmu_notifier_test_young(mm, address, true); return 0; } @@ -622,12 +658,25 @@ static inline int mmu_notifier_clear_young(struct mm_struct *mm, return 0; } +static inline int mmu_notifier_clear_young_fast_only(struct mm_struct *mm, + unsigned long start, + unsigned long end) +{ + return 0; +} + static inline int mmu_notifier_test_young(struct mm_struct *mm, unsigned long address) { return 0; } +static inline int mmu_notifier_test_young_fast_only(struct mm_struct *mm, + unsigned long address) +{ + return 0; +} + static inline bool mm_has_fast_young_notifiers(struct mm_struct *mm) { return 0; diff --git a/include/trace/events/kvm.h b/include/trace/events/kvm.h index 74e40d5d4af4..6d9485cf3e51 100644 --- a/include/trace/events/kvm.h +++ b/include/trace/events/kvm.h @@ -457,36 +457,41 @@ TRACE_EVENT(kvm_unmap_hva_range, ); TRACE_EVENT(kvm_age_hva, - TP_PROTO(unsigned long start, unsigned long end), - TP_ARGS(start, end), + TP_PROTO(unsigned long start, unsigned long end, bool fast_only), + TP_ARGS(start, end, fast_only), TP_STRUCT__entry( __field( unsigned long, start ) __field( unsigned long, end ) + __field( bool, fast_only ) ), TP_fast_assign( __entry->start = start; __entry->end = end; + __entry->fast_only = fast_only; ), - TP_printk("mmu notifier age hva: %#016lx -- %#016lx", - __entry->start, __entry->end) + TP_printk("mmu notifier age hva: %#016lx -- %#016lx fast_only: %d", + __entry->start, __entry->end, __entry->fast_only) ); TRACE_EVENT(kvm_test_age_hva, - TP_PROTO(unsigned long hva), - TP_ARGS(hva), + TP_PROTO(unsigned long hva, bool fast_only), + TP_ARGS(hva, fast_only), TP_STRUCT__entry( __field( unsigned long, hva ) + __field( bool, fast_only ) ), TP_fast_assign( __entry->hva = hva; + __entry->fast_only = fast_only; ), - TP_printk("mmu notifier test age hva: %#016lx", __entry->hva) + TP_printk("mmu notifier test age hva: %#016lx fast_only: %d", + __entry->hva, __entry->fast_only) ); #endif /* _TRACE_KVM_MAIN_H */ diff --git a/mm/mmu_notifier.c b/mm/mmu_notifier.c index c405e5b072cf..f9ec810c8a1b 100644 --- a/mm/mmu_notifier.c +++ b/mm/mmu_notifier.c @@ -404,7 +404,8 @@ bool __mm_has_fast_young_notifiers(struct mm_struct *mm) int __mmu_notifier_clear_young(struct mm_struct *mm, unsigned long start, - unsigned long end) + unsigned long end, + bool fast_only) { struct mmu_notifier *subscription; int young = 0, id; @@ -413,9 +414,13 @@ int __mmu_notifier_clear_young(struct mm_struct *mm, hlist_for_each_entry_rcu(subscription, &mm->notifier_subscriptions->list, hlist, srcu_read_lock_held(&srcu)) { + if (fast_only && !subscription->has_fast_aging) + continue; + if (subscription->ops->clear_young) young |= subscription->ops->clear_young(subscription, - mm, start, end); + mm, start, end, + fast_only); } srcu_read_unlock(&srcu, id); @@ -423,7 +428,8 @@ int __mmu_notifier_clear_young(struct mm_struct *mm, } int __mmu_notifier_test_young(struct mm_struct *mm, - unsigned long address) + unsigned long address, + bool fast_only) { struct mmu_notifier *subscription; int young = 0, id; @@ -432,9 +438,13 @@ int __mmu_notifier_test_young(struct mm_struct *mm, hlist_for_each_entry_rcu(subscription, &mm->notifier_subscriptions->list, hlist, srcu_read_lock_held(&srcu)) { + if (fast_only && !subscription->has_fast_aging) + continue; + if (subscription->ops->test_young) { young = subscription->ops->test_young(subscription, mm, - address); + address, + fast_only); if (young) break; } diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index f6c369eccd2a..ec07caaed6b6 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -846,7 +846,7 @@ static int kvm_mmu_notifier_clear_flush_young(struct mmu_notifier *mn, IS_ENABLED(CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS), }; - trace_kvm_age_hva(start, end); + trace_kvm_age_hva(start, end, false); return kvm_handle_hva_range(kvm, &range).ret; } @@ -854,7 +854,8 @@ static int kvm_mmu_notifier_clear_flush_young(struct mmu_notifier *mn, static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn, struct mm_struct *mm, unsigned long start, - unsigned long end) + unsigned long end, + bool fast_only) { struct kvm *kvm = mmu_notifier_to_kvm(mn); const struct kvm_mmu_notifier_range range = { @@ -868,7 +869,7 @@ static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn, IS_ENABLED(CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS), }; - trace_kvm_age_hva(start, end); + trace_kvm_age_hva(start, end, fast_only); /* * Even though we do not flush TLB, this will still adversely @@ -888,7 +889,8 @@ static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn, static int kvm_mmu_notifier_test_young(struct mmu_notifier *mn, struct mm_struct *mm, - unsigned long address) + unsigned long address, + bool fast_only) { struct kvm *kvm = mmu_notifier_to_kvm(mn); const struct kvm_mmu_notifier_range range = { @@ -902,7 +904,7 @@ static int kvm_mmu_notifier_test_young(struct mmu_notifier *mn, IS_ENABLED(CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS), }; - trace_kvm_test_age_hva(address); + trace_kvm_test_age_hva(address, fast_only); return kvm_handle_hva_range(kvm, &range).ret; }