From patchwork Tue Jun 11 00:21:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13692709 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A91C1C27C4F for ; Tue, 11 Jun 2024 00:22:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=tBvPrH2b2rJBKstC2Tylxy4mLYS7NvJPIg6QIbRKK7A=; b=Zv7TcDd27/9QLfXB+hy8rcDyal pA6Zn8JyJi1F5sCYk3ri5RiYqqa7PmAweMVttAtPEPG0Jr92MeR4Rg2HR4Eo6igRbRnBLuBbS+jGU KTytQWqFeV75HRTdhrx0IR/t/LlEbTgiWnAgJvCsCuEekLQG/D5hnM57hiDPYVmB/lLJkxow13W7/ lwFqWFvQqAjYe8ZQtHj58OsZJRNqXq0q12qIh6mrkhupmpjam2BpBkClyn4NuUXccvILCaoxrz9HO JKW4QqmJwr7C7NTpk38EjDFw1/H4FWnOzNVDwsPhBuV9phSZm/feaMHCr9678JQjzWoA9ngEbltcW IkqWR73Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sGpHU-00000006w4W-0kJR; Tue, 11 Jun 2024 00:22:32 +0000 Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sGpH3-00000006vkZ-3yFk for linux-arm-kernel@lists.infradead.org; Tue, 11 Jun 2024 00:22:07 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-627f20cff42so9381007b3.0 for ; Mon, 10 Jun 2024 17:22:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1718065322; x=1718670122; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=6j3rWVw+E6yiyAll83pvyiHc59e6Q6WaEN3pW8ayg5o=; b=stYgXj+T8onD1mqI3+9W/GF4R71WpB0RJZA36nYoXFVNbroGgJMYk4xOkkOJ8D6dPp BLjDk9bl3r/g0p6MDnhfDzw2b6UbNSKCJzvuCbfXEuo1qapzILIC0GYIyEBdmb2iwmVf L6KUxQTLtCMUpmWbbnZKiiHbl9Aq9JT2PKWN/oGXbj71KIsjDOizLuf01BRQx5lm5QJp FsqvlPJAyekC4+6ki0l9bZMTIjaRg/HicfAZF3psJPqFtb5piY9WwF5UWKUnQAo8FBSP r3IbrzIRaZPibdic+dTXAnhgShLuM8yv+3ohAfEw91c1X4jo32I3KyRMd+z6a/ZJUWMU BBew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718065322; x=1718670122; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6j3rWVw+E6yiyAll83pvyiHc59e6Q6WaEN3pW8ayg5o=; b=p8U/BDLyOIfjlPZXgCMVdeXbXtHYOxhYHSke/k1t/YBAgJ+czs1pcHjyLHt5Tjn6Ri v423zNZhcAGqT/J3cJjH3fFbUHDHBMq9wdDOlCYm0hgFFJQJ0KBUGmbtn4BCLco/VLwq 7RBVlsqjD/jMAdDM0K/L1gmWh06MOqGthqCx8kSm9dI4kHA5pV76SCdlNoUvtGLqimq2 OvQbVI/KXrXzmFisLDvkVfZhwlYoXntzCmACIHmzf+mLViwcuJ3WymzvMPG24rA0xIoe sCgbRlOuh23M1R2Ry+UtnvfeMyhiSfbGPdMcNOoI/yKRP1hA8uelsDaQO/zpRT0oNWd9 darQ== X-Forwarded-Encrypted: i=1; AJvYcCV2eUGrSJbn97jaRr9xqNSxuw12+uPMxzLhCCBGSaDMafDJh2JtAgQcGEsIjoaCpoa4+8E/C3CASMJRZHjvAsxfwWohoWJcPUJkdaCf/hKokBtSqxg= X-Gm-Message-State: AOJu0YyOb5BBFmZpc2vV/vrnkoFYVQmIVLU8ftUqfNtETaHfhGan/SZu 0bLyyeu8hpiO7oTH5W8+/u3D5FmALSKTB0GkL25fBbHvx6ctiBAguoZ5HR0oWMb+ojATZsPZfWK fW1lR/4FxD0q88nb8tQ== X-Google-Smtp-Source: AGHT+IEvrN0k+l/fqR0GYBVcFmKI7zO7AJD796Zxk8tVrT0hIaj/Sm704UQ/Nq6b1QsHmo+//ovsajrvliWZ5C+2 X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a05:690c:4485:b0:62d:fbf:920a with SMTP id 00721157ae682-62d0fbf9666mr11256657b3.10.1718065322252; Mon, 10 Jun 2024 17:22:02 -0700 (PDT) Date: Tue, 11 Jun 2024 00:21:40 +0000 In-Reply-To: <20240611002145.2078921-1-jthoughton@google.com> Mime-Version: 1.0 References: <20240611002145.2078921-1-jthoughton@google.com> X-Mailer: git-send-email 2.45.2.505.gda0bf45e8d-goog Message-ID: <20240611002145.2078921-5-jthoughton@google.com> Subject: [PATCH v5 4/9] mm: Add test_clear_young_fast_only MMU notifier From: James Houghton To: Andrew Morton , Paolo Bonzini Cc: Ankit Agrawal , Axel Rasmussen , Catalin Marinas , David Matlack , David Rientjes , James Houghton , James Morse , Jonathan Corbet , Marc Zyngier , Oliver Upton , Raghavendra Rao Ananta , Ryan Roberts , Sean Christopherson , Shaoqin Huang , Suzuki K Poulose , Wei Xu , Will Deacon , Yu Zhao , Zenghui Yu , kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240610_172206_023423_FE10DC1C X-CRM114-Status: GOOD ( 18.08 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org This new notifier is for multi-gen LRU specifically, as it wants to be able to get and clear age information from secondary MMUs only if it can be done "fast". By having this notifier specifically created for MGLRU, what "fast" means comes down to what is "fast" enough to improve MGLRU's ability to reclaim most of the time. Signed-off-by: James Houghton --- include/linux/mmu_notifier.h | 50 ++++++++++++++++++++++++++++++++++++ mm/mmu_notifier.c | 26 +++++++++++++++++++ 2 files changed, 76 insertions(+) diff --git a/include/linux/mmu_notifier.h b/include/linux/mmu_notifier.h index d39ebb10caeb..2655d841a409 100644 --- a/include/linux/mmu_notifier.h +++ b/include/linux/mmu_notifier.h @@ -61,6 +61,15 @@ enum mmu_notifier_event { #define MMU_NOTIFIER_RANGE_BLOCKABLE (1 << 0) +/* + * Bits in the return value for test_clear_young_fast_only. + * + * MMU_NOTIFIER_FAST_YOUNG: notifier succeeded, secondary MMU reports young. + * MMU_NOTIFIER_FAST_FAILED: notifier failed. + */ +#define MMU_NOTIFIER_FAST_YOUNG (1 << 0) +#define MMU_NOTIFIER_FAST_FAILED (1 << 1) + struct mmu_notifier_ops { /* * Called either by mmu_notifier_unregister or when the mm is @@ -122,6 +131,24 @@ struct mmu_notifier_ops { struct mm_struct *mm, unsigned long address); + /* + * test_clear_young_fast_only is called to check (and optionally clear) + * the young/accessed bitflag in the secondary pte such that the + * secondary MMU must implement it in a way that will not significantly + * disrupt other MMU operations. In other words, speed is more + * important than accuracy. + * + * Returns MMU_NOTIFIER_FAST_YOUNG if the secondary pte(s) were young. + * Returns MMU_NOTIFIER_FAST_FAILED if the secondary MMU could not do + * an accurate fast-only test and/or clear of the young/accessed + * flag. + */ + int (*test_clear_young_fast_only)(struct mmu_notifier *subscription, + struct mm_struct *mm, + unsigned long start, + unsigned long end, + bool clear); + /* * invalidate_range_start() and invalidate_range_end() must be * paired and are called only when the mmap_lock and/or the @@ -383,6 +410,10 @@ extern int __mmu_notifier_clear_young(struct mm_struct *mm, unsigned long end); extern int __mmu_notifier_test_young(struct mm_struct *mm, unsigned long address); +extern int __mmu_notifier_test_clear_young_fast_only(struct mm_struct *mm, + unsigned long start, + unsigned long end, + bool clear); extern int __mmu_notifier_invalidate_range_start(struct mmu_notifier_range *r); extern void __mmu_notifier_invalidate_range_end(struct mmu_notifier_range *r); extern void __mmu_notifier_arch_invalidate_secondary_tlbs(struct mm_struct *mm, @@ -428,6 +459,17 @@ static inline int mmu_notifier_test_young(struct mm_struct *mm, return 0; } +static inline int mmu_notifier_test_clear_young_fast_only(struct mm_struct *mm, + unsigned long start, + unsigned long end, + bool clear) +{ + if (mm_has_notifiers(mm)) + return __mmu_notifier_test_clear_young_fast_only(mm, start, end, + clear); + return 0; +} + static inline void mmu_notifier_invalidate_range_start(struct mmu_notifier_range *range) { @@ -612,6 +654,14 @@ static inline int mmu_notifier_test_young(struct mm_struct *mm, return 0; } +static inline int mmu_notifier_test_clear_young_fast_only(struct mm_struct *mm, + unsigned long start, + unsigned long end, + bool clear) +{ + return 0; +} + static inline void mmu_notifier_invalidate_range_start(struct mmu_notifier_range *range) { diff --git a/mm/mmu_notifier.c b/mm/mmu_notifier.c index 8982e6139d07..7b77ad6cf833 100644 --- a/mm/mmu_notifier.c +++ b/mm/mmu_notifier.c @@ -424,6 +424,32 @@ int __mmu_notifier_test_young(struct mm_struct *mm, return young; } +int __mmu_notifier_test_clear_young_fast_only(struct mm_struct *mm, + unsigned long start, + unsigned long end, + bool clear) +{ + struct mmu_notifier *subscription; + int ret = 0, id; + + id = srcu_read_lock(&srcu); + hlist_for_each_entry_rcu(subscription, + &mm->notifier_subscriptions->list, hlist, + srcu_read_lock_held(&srcu)) { + if (subscription->ops->test_clear_young_fast_only) { + ret = subscription->ops->test_clear_young_fast_only( + subscription, mm, start, end, clear); + if (ret & MMU_NOTIFIER_FAST_FAILED) + break; + if (!clear && (ret & MMU_NOTIFIER_FAST_YOUNG)) + break; + } + } + srcu_read_unlock(&srcu, id); + + return ret; +} + static int mn_itree_invalidate(struct mmu_notifier_subscriptions *subscriptions, const struct mmu_notifier_range *range) {