From patchwork Wed Oct 2 01:02:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mathieu Desnoyers X-Patchwork-Id: 13819084 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ECCD0CF3189 for ; Wed, 2 Oct 2024 01:04:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6A7B468002B; Tue, 1 Oct 2024 21:04:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2F41A680048; Tue, 1 Oct 2024 21:04:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 148F5680045; Tue, 1 Oct 2024 21:04:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id C9077680048 for ; Tue, 1 Oct 2024 21:04:43 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 71F411C6D1E for ; Wed, 2 Oct 2024 01:04:43 +0000 (UTC) X-FDA: 82626867246.26.91DFA62 Received: from smtpout.efficios.com (smtpout.efficios.com [167.114.26.122]) by imf07.hostedemail.com (Postfix) with ESMTP id E6A544000B for ; Wed, 2 Oct 2024 01:04:41 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=efficios.com header.s=smtpout1 header.b="HkrAj/ua"; spf=pass (imf07.hostedemail.com: domain of mathieu.desnoyers@efficios.com designates 167.114.26.122 as permitted sender) smtp.mailfrom=mathieu.desnoyers@efficios.com; dmarc=pass (policy=none) header.from=efficios.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1727830954; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6f1pQa0MZKxKcjeKUtMHjt0Q4k4o+ayrZgYrd0KgBGc=; b=IbI0/LfFuUqumbdVBHX7idyhD1q96f5dCDJNp20SzEe4z4uRRrtasjTxonnP1g79vBWYur Nlf/7iMXuVjOb0ZPuzAseOppguF0882e0S0HeAxP9+T21Ms9K9QUSsM2PLMnCCeSa+E4Dd bLj1waNwH7+X/0/rZ++c6hG2EuB2i0E= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1727830954; a=rsa-sha256; cv=none; b=6FsOrRC2viSt9lSozPDR9OVcNfk99xew+MwbQ6PUP/POrrIkjhy05JHBmtYgaJOPJ6uLTo au9XoLggORzhA6ib2HeOT95/mQuyyX+ihFyOAo4wNIu5rds4HWvIu0FWBFtQIcbavm93E1 ENZrsZUpKIogAIt0ttMpvBehRaXZuC0= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=efficios.com header.s=smtpout1 header.b="HkrAj/ua"; spf=pass (imf07.hostedemail.com: domain of mathieu.desnoyers@efficios.com designates 167.114.26.122 as permitted sender) smtp.mailfrom=mathieu.desnoyers@efficios.com; dmarc=pass (policy=none) header.from=efficios.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=efficios.com; s=smtpout1; t=1727831081; bh=KhCgq8/VW/73gqkVz00pGJVD+ke1nSXtEyCBARYvylk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=HkrAj/uaDf9kJ0Wx95CfYdHab3m/D6/qZeMTmPxsJAPWkP9HrS5GyF7WU8mEQnr0C wkgOwNFa6gGPpnkhFoKH4lc5p7qQ4UWB8d84TOb6NocY1xbcEvfSNf5kHnwXnVbybq 5o8VS18IhqJFexDKc9urmTu0OHHSmsfMdPJ7XpcBzr22uwz0FAGSJur3cAg0lb0EkC u197UJunNMnHIFIQ7qKskh3Y+fd9+rDuh9adRab7e6Ti8itqFyuvY68pyXAcONGTRI n+Omd8oSuTGhKVKEDPOAa4LXEE9OAxse5ohUnwJXfRADrOAjqZXpry4GqyzeLBZ6ih SKY00Omhfzgjw== Received: from thinkos.internal.efficios.com (unknown [IPv6:2606:6d00:100:4000:cacb:9855:de1f:ded2]) by smtpout.efficios.com (Postfix) with ESMTPSA id 4XJGmm6WCqzj09; Tue, 1 Oct 2024 21:04:40 -0400 (EDT) From: Mathieu Desnoyers To: Linus Torvalds , Andrew Morton , Peter Zijlstra Cc: linux-kernel@vger.kernel.org, Mathieu Desnoyers , Nicholas Piggin , Michael Ellerman , Greg Kroah-Hartman , Sebastian Andrzej Siewior , "Paul E. McKenney" , Will Deacon , Boqun Feng , Alan Stern , John Stultz , Neeraj Upadhyay , Frederic Weisbecker , Joel Fernandes , Josh Triplett , Uladzislau Rezki , Steven Rostedt , Lai Jiangshan , Zqiang , Ingo Molnar , Waiman Long , Mark Rutland , Thomas Gleixner , Vlastimil Babka , maged.michael@gmail.com, Mateusz Guzik , Jonas Oberhauser , rcu@vger.kernel.org, linux-mm@kvack.org, lkmm@lists.linux.dev Subject: [RFC PATCH 4/4] sched+mm: Use hazard pointers to track lazy active mm existence Date: Tue, 1 Oct 2024 21:02:05 -0400 Message-Id: <20241002010205.1341915-5-mathieu.desnoyers@efficios.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20241002010205.1341915-1-mathieu.desnoyers@efficios.com> References: <20241002010205.1341915-1-mathieu.desnoyers@efficios.com> MIME-Version: 1.0 X-Stat-Signature: d4yniamzym3wc3ftes5xp8enwbomxaua X-Rspamd-Queue-Id: E6A544000B X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1727831081-35100 X-HE-Meta: U2FsdGVkX1+W+TUF5i/N9phvXJMvv5iC3x9aZNbazHb7Kq+H9AgIsYcloIQV2qVH2PfuI8zgAY+CsHRBd1HTVJ0kVktMJR1LHAXNS39y/ntlkn9fAePMDBUzi9aQuMV9KYEqARQnXuTJ6NPuICMyJXt8WMLV0MUa/kQyOB9WbeXkRF0zy/31N7UCox7FaYKGRT/RGOUZHjje1VdsAmZaNlNCVykypW9L3IG0+8oWXKwm2CvGPYDGHoRU83uPVFWYnmcHMSytl6fFvRO66MKBmU1QGzh6jF+q47UYhWh28+86Ly2dJW2NhIalghR8N7yflemg6DLgjkeX7U3UruCKHcsGaEqGj1a/8AIPqn+50EclZTXDcb/ToIfzIOapPTo/PUnst8PIS3NKJxZAcT+A6C/68RAOeEpVhW5cfliIXC6V0OQ5nYqm0AnKbp0peuN3F1qmfP1GUOZQO0EZnqeLn6cg1bpktXJvz3FetH28HuZVGRAcFq04BKMjNOEEpmWuhCQHIbXUErMHb6SCKuEko+KBhUFWqUaQBrQCOHWZBxWkaD3Nss6s3D9/RjphG+ClEIg6Bo4F8nDebYsyUOhqQCYocOlx3t5bU+ALsZGmePjokpTlWfbDHe+k5JUC0AfTwIbwJfGqckGwIm7tf48Fhp2P7X0lfg2auFYVLclsHuUnrui5JfWMfCZ5I2ed7QoJegF6W4X/M7Lub9DP1xi7YP7OZkL+9VthYAzirRI+Mf53mHthq/ZnRsL+QK3cYk1nIC4jr/NyIrfAhEHDi4hBvhCMK/XCCbFI/fn2Uy+fBlQ0gCKP0u0Gy0FKeSsrQBwKqIAu+uHHqRGYyUTt5WNjxaAGHB46Cdoj2skiqNi7vxysS4m0xmqRAv37dU/6SvyeNFGC00NUs1L8POM41+lq2KRkN09rgwhkj3BKjJ8v8M81VkL7X6WATwxFtClJwAfNbDRYz5Upxq9RQQBor7k Jb3trE5W Djld/0/5tP1nTJ1REFY4jkCyF3hQxICL4vwmkE++AWI2AkKiu/3XDafoi1VFGFSqBG3gL77cKVj2Yu68xDCN/l+UFZClQxw2vliMgW4mYxXDU1E9yCAfeD87YgCfN1xlBtOWJQKjJH4GY0SzmC/C7Qh/no07FQdfzxA4ZFO9nds/HQTk0/BkngO5GqB+B6RB4LjThesT+T6+EK3ZtyeAvwfSabWVtZoCconC0FsALPyIqV9vUNP1Z9xFvtreHyx/sjpXZKNZc3+OK1pdQVvPC5ZoK3/HCRuZLXhnhMJTVsU2ueeSNLKoFWwCFH4N7Y/vnMo795Z0GqgLQ04Gxb7KzG1TGMoZ5Kr2DC9TMI/xPGAi2OhcaTG63eKFIDccByGPvSEUF1fZfXeFjA1gStQeJ60T+Wc0JmsuXFkYigfS0P78Lo61H01jVFQph3HpGQQPCrb56eFdBaqviQhscvvPZ2elrK4+T7PMR5gINvR9stBhKlcjtamUP6fljy5sBe0LdPcLOisN4GSrxX/V3Egj4CPvGuMTHfFSlI9FeoM1blzs4UXR9di8ht66pRACSc0diSBW3TkI7xnBBDD4i5yoS4BLAEMis4bMBMHZHnM5BnkFCs4EnHuSgc6cXwOj2/AtL14UMFx2UkyDXAgNLWeZnJMei7RpYtsehK8YiR3nrYt+ZeHEMDyUcu7R1NGaTSkH/S7RjkwWutC67ENHg/kDnIZ1jGAjPNEHzOvPYIlBYQBevmxlGnckz6ULCIWZmznnBmQE4RGJSPTGQi2HmQ4lMZpXML5NmTpKjDBRdONyh/0tP6ArA4PWSt8IImd3grHVkxUx2eA81qZER7vqke0zhrTuLJN9IlihQXb2KELac9L/W/RyMUdipjEZtH89ytm58mGIDEWZ8F6yH6YxwMRONpx1auD60RPQnH0azz3Ax2LjzHbNeMnqqj/TTWI7GPHvqiuluzBLu9u/v5FhlVH3m0PWShB6c o2aP1x7v A5TTkmjvVcgzke/iFA3W2wBoeLs73Ed4aWW7yyV2cco= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Replace lazy active mm existence tracking with hazard pointers. This removes the following implementations and their associated config options: - MMU_LAZY_TLB_REFCOUNT - MMU_LAZY_TLB_SHOOTDOWN - This removes the call_rcu delayed mm drop for RT. It leverages the fact that each CPU only ever have at most one single lazy active mm. This makes it a very good fit for a hazard pointer domain implemented with one hazard pointer slot per CPU. * Benchmarks: will-it-scale context_switch1_threads nr threads (-t) speedup 24 +3% 48 +12% 96 +21% 192 +28% Methodology: Each test is the average of 20 iterations. Use median result of 3 test runs. Test hardware: CPU(s): 384 On-line CPU(s) list: 0-383 Vendor ID: AuthenticAMD Model name: AMD EPYC 9654 96-Core Processor CPU family: 25 Model: 17 Thread(s) per core: 2 Core(s) per socket: 96 Socket(s): 2 Stepping: 1 Frequency boost: enabled CPU(s) scaling MHz: 100% CPU max MHz: 3709.0000 CPU min MHz: 400.0000 BogoMIPS: 4799.75 Memory: 768 GB ram. Signed-off-by: Mathieu Desnoyers Cc: Nicholas Piggin Cc: Michael Ellerman Cc: Greg Kroah-Hartman Cc: Sebastian Andrzej Siewior Cc: "Paul E. McKenney" Cc: Will Deacon Cc: Peter Zijlstra Cc: Boqun Feng Cc: Alan Stern Cc: John Stultz Cc: Neeraj Upadhyay Cc: Linus Torvalds Cc: Andrew Morton Cc: Boqun Feng Cc: Frederic Weisbecker Cc: Joel Fernandes Cc: Josh Triplett Cc: Uladzislau Rezki Cc: Steven Rostedt Cc: Lai Jiangshan Cc: Zqiang Cc: Ingo Molnar Cc: Waiman Long Cc: Mark Rutland Cc: Thomas Gleixner Cc: Vlastimil Babka Cc: maged.michael@gmail.com Cc: Mateusz Guzik Cc: Jonas Oberhauser Cc: rcu@vger.kernel.org Cc: linux-mm@kvack.org Cc: lkmm@lists.linux.dev --- Documentation/mm/active_mm.rst | 9 ++-- arch/Kconfig | 32 ------------- arch/powerpc/Kconfig | 1 - arch/powerpc/mm/book3s64/radix_tlb.c | 23 +-------- include/linux/mm_types.h | 3 -- include/linux/sched/mm.h | 71 +++++++++++----------------- kernel/exit.c | 4 +- kernel/fork.c | 47 +++++------------- kernel/sched/sched.h | 8 +--- lib/Kconfig.debug | 10 ---- 10 files changed, 49 insertions(+), 159 deletions(-) diff --git a/Documentation/mm/active_mm.rst b/Documentation/mm/active_mm.rst index d096fc091e23..c225cac49c30 100644 --- a/Documentation/mm/active_mm.rst +++ b/Documentation/mm/active_mm.rst @@ -2,11 +2,10 @@ Active MM ========= -Note, the mm_count refcount may no longer include the "lazy" users -(running tasks with ->active_mm == mm && ->mm == NULL) on kernels -with CONFIG_MMU_LAZY_TLB_REFCOUNT=n. Taking and releasing these lazy -references must be done with mmgrab_lazy_tlb() and mmdrop_lazy_tlb() -helpers, which abstract this config option. +Note, the mm_count refcount no longer include the "lazy" users (running +tasks with ->active_mm == mm && ->mm == NULL) Taking and releasing these +lazy references must be done with mmgrab_lazy_tlb() and mmdrop_lazy_tlb() +helpers, which are implemented with hazard pointers. :: diff --git a/arch/Kconfig b/arch/Kconfig index 975dd22a2dbd..d4261935f8dc 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -475,38 +475,6 @@ config ARCH_WANT_IRQS_OFF_ACTIVATE_MM irqs disabled over activate_mm. Architectures that do IPI based TLB shootdowns should enable this. -# Use normal mm refcounting for MMU_LAZY_TLB kernel thread references. -# MMU_LAZY_TLB_REFCOUNT=n can improve the scalability of context switching -# to/from kernel threads when the same mm is running on a lot of CPUs (a large -# multi-threaded application), by reducing contention on the mm refcount. -# -# This can be disabled if the architecture ensures no CPUs are using an mm as a -# "lazy tlb" beyond its final refcount (i.e., by the time __mmdrop frees the mm -# or its kernel page tables). This could be arranged by arch_exit_mmap(), or -# final exit(2) TLB flush, for example. -# -# To implement this, an arch *must*: -# Ensure the _lazy_tlb variants of mmgrab/mmdrop are used when manipulating -# the lazy tlb reference of a kthread's ->active_mm (non-arch code has been -# converted already). -config MMU_LAZY_TLB_REFCOUNT - def_bool y - depends on !MMU_LAZY_TLB_SHOOTDOWN - -# This option allows MMU_LAZY_TLB_REFCOUNT=n. It ensures no CPUs are using an -# mm as a lazy tlb beyond its last reference count, by shooting down these -# users before the mm is deallocated. __mmdrop() first IPIs all CPUs that may -# be using the mm as a lazy tlb, so that they may switch themselves to using -# init_mm for their active mm. mm_cpumask(mm) is used to determine which CPUs -# may be using mm as a lazy tlb mm. -# -# To implement this, an arch *must*: -# - At the time of the final mmdrop of the mm, ensure mm_cpumask(mm) contains -# at least all possible CPUs in which the mm is lazy. -# - It must meet the requirements for MMU_LAZY_TLB_REFCOUNT=n (see above). -config MMU_LAZY_TLB_SHOOTDOWN - bool - config ARCH_HAVE_NMI_SAFE_CMPXCHG bool diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index d7b09b064a8a..b1e25e75baab 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -291,7 +291,6 @@ config PPC select MMU_GATHER_PAGE_SIZE select MMU_GATHER_RCU_TABLE_FREE select MMU_GATHER_MERGE_VMAS - select MMU_LAZY_TLB_SHOOTDOWN if PPC_BOOK3S_64 select MODULES_USE_ELF_RELA select NEED_DMA_MAP_STATE if PPC64 || NOT_COHERENT_CACHE select NEED_PER_CPU_EMBED_FIRST_CHUNK if PPC64 diff --git a/arch/powerpc/mm/book3s64/radix_tlb.c b/arch/powerpc/mm/book3s64/radix_tlb.c index 9e1f6558d026..ff0d4f28cf52 100644 --- a/arch/powerpc/mm/book3s64/radix_tlb.c +++ b/arch/powerpc/mm/book3s64/radix_tlb.c @@ -1197,28 +1197,7 @@ void radix__tlb_flush(struct mmu_gather *tlb) * See the comment for radix in arch_exit_mmap(). */ if (tlb->fullmm) { - if (IS_ENABLED(CONFIG_MMU_LAZY_TLB_SHOOTDOWN)) { - /* - * Shootdown based lazy tlb mm refcounting means we - * have to IPI everyone in the mm_cpumask anyway soon - * when the mm goes away, so might as well do it as - * part of the final flush now. - * - * If lazy shootdown was improved to reduce IPIs (e.g., - * by batching), then it may end up being better to use - * tlbies here instead. - */ - preempt_disable(); - - smp_mb(); /* see radix__flush_tlb_mm */ - exit_flush_lazy_tlbs(mm); - __flush_all_mm(mm, true); - - preempt_enable(); - } else { - __flush_all_mm(mm, true); - } - + __flush_all_mm(mm, true); } else if ( (psize = radix_get_mmu_psize(page_size)) == -1) { if (!tlb->freed_tables) radix__flush_tlb_mm(mm); diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 485424979254..db5f13554485 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -975,9 +975,6 @@ struct mm_struct { atomic_t tlb_flush_batched; #endif struct uprobes_state uprobes_state; -#ifdef CONFIG_PREEMPT_RT - struct rcu_head delayed_drop; -#endif #ifdef CONFIG_HUGETLB_PAGE atomic_long_t hugetlb_usage; #endif diff --git a/include/linux/sched/mm.h b/include/linux/sched/mm.h index 91546493c43d..0fecd1a3311d 100644 --- a/include/linux/sched/mm.h +++ b/include/linux/sched/mm.h @@ -9,6 +9,10 @@ #include #include #include +#include + +/* Sched lazy mm hazard pointer domain. */ +DECLARE_PER_CPU(struct hp_slot, hp_domain_sched_lazy_mm); /* * Routines for handling mm_structs @@ -55,61 +59,42 @@ static inline void mmdrop(struct mm_struct *mm) __mmdrop(mm); } -#ifdef CONFIG_PREEMPT_RT -/* - * RCU callback for delayed mm drop. Not strictly RCU, but call_rcu() is - * by far the least expensive way to do that. - */ -static inline void __mmdrop_delayed(struct rcu_head *rhp) +/* Helpers for lazy TLB mm refcounting */ +static inline void mmgrab_lazy_tlb(struct mm_struct *mm) { - struct mm_struct *mm = container_of(rhp, struct mm_struct, delayed_drop); + struct hp_ctx ctx; - __mmdrop(mm); -} + /* + * mmgrab_lazy_tlb must provide a full memory barrier, see the + * membarrier comment finish_task_switch which relies on this. + */ + smp_mb(); -/* - * Invoked from finish_task_switch(). Delegates the heavy lifting on RT - * kernels via RCU. - */ -static inline void mmdrop_sched(struct mm_struct *mm) -{ - /* Provides a full memory barrier. See mmdrop() */ - if (atomic_dec_and_test(&mm->mm_count)) - call_rcu(&mm->delayed_drop, __mmdrop_delayed); -} -#else -static inline void mmdrop_sched(struct mm_struct *mm) -{ - mmdrop(mm); -} -#endif + /* + * The caller guarantees existence of mm. Allocate a hazard + * pointer to chain this existence guarantee to a hazard + * pointer. + */ + ctx = hp_allocate(&hp_domain_sched_lazy_mm, mm); -/* Helpers for lazy TLB mm refcounting */ -static inline void mmgrab_lazy_tlb(struct mm_struct *mm) -{ - if (IS_ENABLED(CONFIG_MMU_LAZY_TLB_REFCOUNT)) - mmgrab(mm); + /* There is only a single lazy mm per CPU at any time. */ + WARN_ON_ONCE(!hp_ctx_addr(ctx)); } static inline void mmdrop_lazy_tlb(struct mm_struct *mm) { - if (IS_ENABLED(CONFIG_MMU_LAZY_TLB_REFCOUNT)) { - mmdrop(mm); - } else { - /* - * mmdrop_lazy_tlb must provide a full memory barrier, see the - * membarrier comment finish_task_switch which relies on this. - */ - smp_mb(); - } + /* + * mmdrop_lazy_tlb must provide a full memory barrier, see the + * membarrier comment finish_task_switch which relies on this. + */ + smp_mb(); + WRITE_ONCE(this_cpu_ptr(&hp_domain_sched_lazy_mm)->addr, NULL); } static inline void mmdrop_lazy_tlb_sched(struct mm_struct *mm) { - if (IS_ENABLED(CONFIG_MMU_LAZY_TLB_REFCOUNT)) - mmdrop_sched(mm); - else - smp_mb(); /* see mmdrop_lazy_tlb() above */ + smp_mb(); /* see mmdrop_lazy_tlb() above */ + WRITE_ONCE(this_cpu_ptr(&hp_domain_sched_lazy_mm)->addr, NULL); } /** diff --git a/kernel/exit.c b/kernel/exit.c index 7430852a8571..cb4ace06c0f0 100644 --- a/kernel/exit.c +++ b/kernel/exit.c @@ -545,8 +545,6 @@ static void exit_mm(void) if (!mm) return; mmap_read_lock(mm); - mmgrab_lazy_tlb(mm); - BUG_ON(mm != current->active_mm); /* more a memory barrier than a real lock */ task_lock(current); /* @@ -561,6 +559,8 @@ static void exit_mm(void) */ smp_mb__after_spinlock(); local_irq_disable(); + mmgrab_lazy_tlb(mm); + BUG_ON(mm != current->active_mm); current->mm = NULL; membarrier_update_current_mm(NULL); enter_lazy_tlb(mm, current); diff --git a/kernel/fork.c b/kernel/fork.c index cc760491f201..42c652ec39b5 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -149,6 +149,9 @@ DEFINE_PER_CPU(unsigned long, process_counts) = 0; __cacheline_aligned DEFINE_RWLOCK(tasklist_lock); /* outer */ +/* Sched lazy mm hazard pointer domain. */ +DEFINE_PER_CPU(struct hp_slot, hp_domain_sched_lazy_mm); + #ifdef CONFIG_PROVE_RCU int lockdep_tasklist_lock_is_held(void) { @@ -855,50 +858,24 @@ static void do_shoot_lazy_tlb(void *arg) WARN_ON_ONCE(current->mm); current->active_mm = &init_mm; switch_mm(mm, &init_mm, current); + WRITE_ONCE(this_cpu_ptr(&hp_domain_sched_lazy_mm)->addr, NULL); } } -static void cleanup_lazy_tlbs(struct mm_struct *mm) +static void retire_lazy_mm_hp(int cpu, struct hp_slot *slot, void *addr) { - if (!IS_ENABLED(CONFIG_MMU_LAZY_TLB_SHOOTDOWN)) { - /* - * In this case, lazy tlb mms are refounted and would not reach - * __mmdrop until all CPUs have switched away and mmdrop()ed. - */ - return; - } + smp_call_function_single(cpu, do_shoot_lazy_tlb, addr, 1); + smp_call_function_single(cpu, do_check_lazy_tlb, addr, 1); +} +static void cleanup_lazy_tlbs(struct mm_struct *mm) +{ /* - * Lazy mm shootdown does not refcount "lazy tlb mm" usage, rather it - * requires lazy mm users to switch to another mm when the refcount + * Require lazy mm users to switch to another mm when the refcount * drops to zero, before the mm is freed. This requires IPIs here to * switch kernel threads to init_mm. - * - * archs that use IPIs to flush TLBs can piggy-back that lazy tlb mm - * switch with the final userspace teardown TLB flush which leaves the - * mm lazy on this CPU but no others, reducing the need for additional - * IPIs here. There are cases where a final IPI is still required here, - * such as the final mmdrop being performed on a different CPU than the - * one exiting, or kernel threads using the mm when userspace exits. - * - * IPI overheads have not found to be expensive, but they could be - * reduced in a number of possible ways, for example (roughly - * increasing order of complexity): - * - The last lazy reference created by exit_mm() could instead switch - * to init_mm, however it's probable this will run on the same CPU - * immediately afterwards, so this may not reduce IPIs much. - * - A batch of mms requiring IPIs could be gathered and freed at once. - * - CPUs store active_mm where it can be remotely checked without a - * lock, to filter out false-positives in the cpumask. - * - After mm_users or mm_count reaches zero, switching away from the - * mm could clear mm_cpumask to reduce some IPIs, perhaps together - * with some batching or delaying of the final IPIs. - * - A delayed freeing and RCU-like quiescing sequence based on mm - * switching to avoid IPIs completely. */ - on_each_cpu_mask(mm_cpumask(mm), do_shoot_lazy_tlb, (void *)mm, 1); - if (IS_ENABLED(CONFIG_DEBUG_VM_SHOOT_LAZIES)) - on_each_cpu(do_check_lazy_tlb, (void *)mm, 1); + hp_scan(&hp_domain_sched_lazy_mm, mm, retire_lazy_mm_hp); } /* diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 4c36cc680361..d883c2aa3518 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -3527,12 +3527,8 @@ static inline void switch_mm_cid(struct rq *rq, if (!next->mm) { // to kernel /* * user -> kernel transition does not guarantee a barrier, but - * we can use the fact that it performs an atomic operation in - * mmgrab(). - */ - if (prev->mm) // from user - smp_mb__after_mmgrab(); - /* + * we can use the fact that mmgrab() has a full barrier. + * * kernel -> kernel transition does not change rq->curr->mm * state. It stays NULL. */ diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index a30c03a66172..1cb9dab361c9 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -803,16 +803,6 @@ config DEBUG_VM If unsure, say N. -config DEBUG_VM_SHOOT_LAZIES - bool "Debug MMU_LAZY_TLB_SHOOTDOWN implementation" - depends on DEBUG_VM - depends on MMU_LAZY_TLB_SHOOTDOWN - help - Enable additional IPIs that ensure lazy tlb mm references are removed - before the mm is freed. - - If unsure, say N. - config DEBUG_VM_MAPLE_TREE bool "Debug VM maple trees" depends on DEBUG_VM