From patchwork Thu Jan 23 10:37:18 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13948127 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A9E64C0218C for ; Thu, 23 Jan 2025 10:38:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EC809280001; Thu, 23 Jan 2025 05:38:00 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DEC4F6B0089; Thu, 23 Jan 2025 05:38:00 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B4E18280002; Thu, 23 Jan 2025 05:38:00 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 8F31A6B0082 for ; Thu, 23 Jan 2025 05:38:00 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 19484480AD for ; Thu, 23 Jan 2025 10:38:00 +0000 (UTC) X-FDA: 83038366320.23.2E211BD Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) by imf28.hostedemail.com (Postfix) with ESMTP id B7248C000F for ; Thu, 23 Jan 2025 10:37:57 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=Hz8qowaP; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=yOpvhHOx; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=Hz8qowaP; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=yOpvhHOx; spf=pass (imf28.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.130 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1737628678; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=t3re1HhaNJ7ult5EVdCJjMXAxd3DND62tEqbEIKH98g=; b=SzHMTfEfiU5t/7u+97h0P6tTmb6kg86D4UmLre1+PBn7XO9jTM44ArA235jf1YE0dHoC4V VdFLQXvjHib9vI8jklEdFaqoXG4ZSPx5Yt/7Me/MLR/BIYtRpWzA4nHTANt6dERbqLzyiz CFxDDcHfGPwT2CxaqQsNFB+xY8/kU58= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=Hz8qowaP; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=yOpvhHOx; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=Hz8qowaP; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=yOpvhHOx; spf=pass (imf28.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.130 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1737628678; a=rsa-sha256; cv=none; b=D8pXpNt7ivY707gUtFJkh69XYcFhvTbHb5WLx1d13df5vrqhsTg/FmxGNIDHNYGnlqYeP9 hdtZNhwHSYtWSCMQajEP1gcVhpAPvpiSOyyfFYL6dAVPtAaiJdWosuf/17hZ37d96vobwT DGSwzQUq3aRs7zFpKKue6iXUXZD0d0s= Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 391CD2117D; Thu, 23 Jan 2025 10:37:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1737628676; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=t3re1HhaNJ7ult5EVdCJjMXAxd3DND62tEqbEIKH98g=; b=Hz8qowaPTUkggcfEOrGEszBiecjRHVJlrmCZNgrSyPprj+Zuiha7KX00JQvdMpeqxW+RMZ LV8NoOAG3sDGDylHf00TH1AQzw5Tcnqeqeh4ckhzI+/R+Yu85QlKye62CeJyqaw9W470Q+ d+eOvrweI1PAERjSOZIrvxmsxaJl78w= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1737628676; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=t3re1HhaNJ7ult5EVdCJjMXAxd3DND62tEqbEIKH98g=; b=yOpvhHOxTivTJkIIsCrFTvfOSaFb9/7Xab6dbAq6LINQzThBwm2xq9ORa34U1t/90zu3z1 ULPHOVu9wJ8DoTCA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1737628676; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=t3re1HhaNJ7ult5EVdCJjMXAxd3DND62tEqbEIKH98g=; b=Hz8qowaPTUkggcfEOrGEszBiecjRHVJlrmCZNgrSyPprj+Zuiha7KX00JQvdMpeqxW+RMZ LV8NoOAG3sDGDylHf00TH1AQzw5Tcnqeqeh4ckhzI+/R+Yu85QlKye62CeJyqaw9W470Q+ d+eOvrweI1PAERjSOZIrvxmsxaJl78w= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1737628676; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=t3re1HhaNJ7ult5EVdCJjMXAxd3DND62tEqbEIKH98g=; b=yOpvhHOxTivTJkIIsCrFTvfOSaFb9/7Xab6dbAq6LINQzThBwm2xq9ORa34U1t/90zu3z1 ULPHOVu9wJ8DoTCA== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 19F5B13A51; Thu, 23 Jan 2025 10:37:56 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id sHsPBgQckmdrAQAAD6G6ig (envelope-from ); Thu, 23 Jan 2025 10:37:56 +0000 From: Vlastimil Babka Date: Thu, 23 Jan 2025 11:37:18 +0100 Subject: [PATCH RFC 1/4] slab, rcu: move TINY_RCU variant of kvfree_rcu() to SLAB MIME-Version: 1.0 Message-Id: <20250123-slub-tiny-kfree_rcu-v1-1-0e386ef1541a@suse.cz> References: <20250123-slub-tiny-kfree_rcu-v1-0-0e386ef1541a@suse.cz> In-Reply-To: <20250123-slub-tiny-kfree_rcu-v1-0-0e386ef1541a@suse.cz> To: Christoph Lameter , David Rientjes , "Paul E. McKenney" , Joel Fernandes , Josh Triplett , Boqun Feng , Uladzislau Rezki Cc: Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang , rcu@vger.kernel.org, Vlastimil Babka X-Mailer: b4 0.14.2 X-Developer-Signature: v=1; a=openpgp-sha256; l=6322; i=vbabka@suse.cz; h=from:subject:message-id; bh=V3CDNBQ65RoS8BNy/YQECuCfUSQlpxlHN6MkeKNjSZU=; b=owEBbQGS/pANAwAIAbvgsHXSRYiaAcsmYgBnkhv5HlHabD5/rYKLN4U4Y8eOLgNzVkyqBNRv/ UW7L9RMLCWJATMEAAEIAB0WIQR7u8hBFZkjSJZITfG74LB10kWImgUCZ5Ib+QAKCRC74LB10kWI mmYXB/9NvbzObEFhfbeTeuiQh79OGIklwhK5mw1vivzIcgBKy/VoEuNPD2Q4EOY/2UDY8h8SOGb pk3ZUdjfO4GihAALiwwjpLYGwIDshktM3PYExu0souNPXsg6yi2q+bxJWA0um9+wL8K+ap1EYNJ oxEY74VfDKXszUz8vloXNiDa/N2KVxtMnOQQVEm5mie27HXlLtxhmy3RR3z+Rp6HYnKTtGNklX4 tjn5h34+NQKbNv2NvkKTI/0Nvmhle3j51IjmFwYELB/fQCoy0O9CgofE6WHjLdBAOtaH14N2rJY +Cdm3tVWyQJrn6JStD8UT7XKepFvp/mV/e4dfJcoJGPewXJh X-Developer-Key: i=vbabka@suse.cz; a=openpgp; fpr=A940D434992C2E8E99103D50224FA7E7CC82A664 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: B7248C000F X-Stat-Signature: zyqwgeb9pkgx4nemms1fr4cprhfe9bh8 X-Rspam-User: X-HE-Tag: 1737628677-760641 X-HE-Meta: U2FsdGVkX1/l+O7Bxix09UFOxk8yG8WJBkpYOMJfJJ+aYEk1zOcvaGmhXpTdyW55vdbp1BudxK6rs135Fr6/+YMhA94OrmafJTqkkuaX5vNE9cPpFqWUs+LDXe7nlnT8IdV4SnVjsL84gz9pq3y2oFs64jn/lhHhlfLGv0PcvURT2hMVwO7LB15cOwut/mckgkVqwASS1mdkOVOHZWPODv7If8ExKOqnLNJYOrJU8ZUDtCNojiRPcd9QXsb5hVPv6+LDZVhcGWkpTwR9nexhbynjy0qil3bkHikGkKO+5xdH/ZTOs7Gy9dom7n+z5Hnu+WMF9Grc5JbNV8fiGFxe/uAgXzvkadU6GK2RU7MthptVpkqKLg5Vd3XJbQgAS81SF9enLRIemApW4Axl+RZTlnCockwA0FTtwZ3HwZpWiFtIwd5zuXeYl8dFR7jeTOBZVbvmnAdHgUFyR930sdrhW5HcBUKRYGbJzitGjv02iLOkC3u+LKYhR+EG9eZ7dYAlgIsICcnio8iQGtmVDtQWLLX3JOWw+d7aD3WLEIz1NAMda0lDW/1FqaTe3c8RkU8ZHntfbmvtjJrP32LwAJv4KdJHYtu9JHgVzDlrzx7/fdDh5tO3Nac3lb2AgqgNNaPk8dSysXUUjox82VxjJnHTW4A5/+Ljei9NUYSILeJpu8cy5+slgJYLjA0U7uTxxcwWsHETsWXcdNSpfwS/Fp8QfSx4FsN7nEhoEdCNsbrIvai3szQwQmPrUAqMQYHTgUtkqa5VxBfRPO9yfANU67GX3wFVrZ1gYhW23O+M9ggfjE82QrGp9PZc6FWjLarG/t+OHREvCR+cml+wHOd15/XJ/NYYp1TYdbRJlqVSTj4lD1izxQjX91V6TwA1uDTA66ST04PEZfAhriJ8dosI0/UrSLYinwSHjErfxC0m/Gp6uEB1qpDiTM+h1+XQYCelVAAJyLYfHYlGn7uGC/2oeRr 32C301Zg g4V5hIKFJP1KmqkI5n1Zfeq5cM69lwAdEqmhuhCKg4NAe9d7rghGMshERfRhtyIlb57EFa+1peLgJi5P9VpQSv/OrhauAi6j1RqbH7B9vmbB0ygfyw97hNxj4pBwpx1PCjVhY09oUUNYSP4ukVXpCNYkmaOOBpTp/k1+h1yES588fTAIM5ZECUGYCoFqC74KpYpqtd7GAk/bSkQxDwsibZrDXSNgSy6ja9Ues1VO20IK2x6NhLGzJq1O46UKpQLPMQUfW7bk/8d1zO+8VoMBR95bp4LEAVEmLKOxWUOWwa/iWwlgBeh1ActuJt4asNw+VsFhvZolKB4wa1itnIfKLP8XcSWFV5tc7ZtRImIBCrGgpvG4D7xLA7qW4GQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Following the move of TREE_RCU implementation, let's move also the TINY_RCU one for consistency and subsequent refactoring. For simplicity, remove the separate inline __kvfree_call_rcu() as TINY_RCU is not meant for high-performance hardware anyway. Declare kvfree_call_rcu() in rcupdate.h to avoid header dependency issues. Also move the kvfree_rcu_barrier() declaration to slab.h Signed-off-by: Vlastimil Babka --- include/linux/rcupdate.h | 5 +++++ include/linux/rcutiny.h | 36 ------------------------------------ include/linux/rcutree.h | 3 --- include/linux/slab.h | 14 ++++++++++++++ kernel/rcu/tiny.c | 11 ----------- mm/slab_common.c | 20 ++++++++++++++++++-- 6 files changed, 37 insertions(+), 52 deletions(-) diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index 48e5c03df1dd83c246a61d0fcc8aa638adcd7654..3f70d1c8144426f40553c8c589f07097ece8a706 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -1082,6 +1082,11 @@ static inline notrace void rcu_read_unlock_sched_notrace(void) #define kfree_rcu_mightsleep(ptr) kvfree_rcu_arg_1(ptr) #define kvfree_rcu_mightsleep(ptr) kvfree_rcu_arg_1(ptr) +/* + * In mm/slab_common.c, no suitable header to include here. + */ +void kvfree_call_rcu(struct rcu_head *head, void *ptr); + #define kvfree_rcu_arg_2(ptr, rhf) \ do { \ typeof (ptr) ___p = (ptr); \ diff --git a/include/linux/rcutiny.h b/include/linux/rcutiny.h index fe42315f667fc5be7f2ed8eae6ea0c7193030846..f519cd6802286710bdd56588b5ff3d07bcd30b92 100644 --- a/include/linux/rcutiny.h +++ b/include/linux/rcutiny.h @@ -90,41 +90,6 @@ static inline void synchronize_rcu_expedited(void) synchronize_rcu(); } -/* - * Add one more declaration of kvfree() here. It is - * not so straight forward to just include - * where it is defined due to getting many compile - * errors caused by that include. - */ -extern void kvfree(const void *addr); - -static inline void __kvfree_call_rcu(struct rcu_head *head, void *ptr) -{ - if (head) { - call_rcu(head, (rcu_callback_t) ((void *) head - ptr)); - return; - } - - // kvfree_rcu(one_arg) call. - might_sleep(); - synchronize_rcu(); - kvfree(ptr); -} - -static inline void kvfree_rcu_barrier(void) -{ - rcu_barrier(); -} - -#ifdef CONFIG_KASAN_GENERIC -void kvfree_call_rcu(struct rcu_head *head, void *ptr); -#else -static inline void kvfree_call_rcu(struct rcu_head *head, void *ptr) -{ - __kvfree_call_rcu(head, ptr); -} -#endif - void rcu_qs(void); static inline void rcu_softirq_qs(void) @@ -164,7 +129,6 @@ static inline void rcu_end_inkernel_boot(void) { } static inline bool rcu_inkernel_boot_has_ended(void) { return true; } static inline bool rcu_is_watching(void) { return true; } static inline void rcu_momentary_eqs(void) { } -static inline void kfree_rcu_scheduler_running(void) { } /* Avoid RCU read-side critical sections leaking across. */ static inline void rcu_all_qs(void) { barrier(); } diff --git a/include/linux/rcutree.h b/include/linux/rcutree.h index 27d86d9127817e50f8d4dd79e1990d70a02435bb..dbe77b5fe06ec89a393b5444d6c479ced346a37b 100644 --- a/include/linux/rcutree.h +++ b/include/linux/rcutree.h @@ -34,12 +34,9 @@ static inline void rcu_virt_note_context_switch(void) } void synchronize_rcu_expedited(void); -void kvfree_call_rcu(struct rcu_head *head, void *ptr); -void kvfree_rcu_barrier(void); void rcu_barrier(void); void rcu_momentary_eqs(void); -void kfree_rcu_scheduler_running(void); struct rcu_gp_oldstate { unsigned long rgos_norm; diff --git a/include/linux/slab.h b/include/linux/slab.h index 09eedaecf1205672bb2e7c8cd57ae8fccebc2737..bcc62e5656c35c6a3f4caf26fb33d7447dead39a 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -16,6 +16,7 @@ #include #include #include +#include #include #include #include @@ -1082,6 +1083,19 @@ extern void kvfree_sensitive(const void *addr, size_t len); unsigned int kmem_cache_size(struct kmem_cache *s); +#ifdef CONFIG_TINY_RCU +static inline void kvfree_rcu_barrier(void) +{ + rcu_barrier(); +} + +static inline void kfree_rcu_scheduler_running(void) { } +#else +void kvfree_rcu_barrier(void); + +void kfree_rcu_scheduler_running(void); +#endif + /** * kmalloc_size_roundup - Report allocation bucket size for the given size * diff --git a/kernel/rcu/tiny.c b/kernel/rcu/tiny.c index b3b3ce34df6310f7bddba40b2be1bdf6c9f00232..0ec27093d0e14a4b1060ea08932c4ac13f9b0f26 100644 --- a/kernel/rcu/tiny.c +++ b/kernel/rcu/tiny.c @@ -246,17 +246,6 @@ bool poll_state_synchronize_rcu(unsigned long oldstate) } EXPORT_SYMBOL_GPL(poll_state_synchronize_rcu); -#ifdef CONFIG_KASAN_GENERIC -void kvfree_call_rcu(struct rcu_head *head, void *ptr) -{ - if (head) - kasan_record_aux_stack_noalloc(ptr); - - __kvfree_call_rcu(head, ptr); -} -EXPORT_SYMBOL_GPL(kvfree_call_rcu); -#endif - void __init rcu_init(void) { open_softirq(RCU_SOFTIRQ, rcu_process_callbacks); diff --git a/mm/slab_common.c b/mm/slab_common.c index 69f2d19010dedaa3e5b303ab9803c8cdd40152fa..330cdd8ebc5380090ee784c58e8ca1d1a52b3758 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1861,7 +1861,23 @@ add_ptr_to_bulk_krc_lock(struct kfree_rcu_cpu **krcp, return true; } -#if !defined(CONFIG_TINY_RCU) +#ifdef CONFIG_TINY_RCU + +void kvfree_call_rcu(struct rcu_head *head, void *ptr) +{ + if (head) { + kasan_record_aux_stack_noalloc(ptr); + call_rcu(head, (rcu_callback_t) ((void *) head - ptr)); + return; + } + + // kvfree_rcu(one_arg) call. + might_sleep(); + synchronize_rcu(); + kvfree(ptr); +} + +#else /* !CONFIG_TINY_RCU */ static enum hrtimer_restart schedule_page_work_fn(struct hrtimer *t) @@ -2071,7 +2087,7 @@ void kvfree_rcu_barrier(void) } EXPORT_SYMBOL_GPL(kvfree_rcu_barrier); -#endif /* #if !defined(CONFIG_TINY_RCU) */ +#endif /* !CONFIG_TINY_RCU */ static unsigned long kfree_rcu_shrink_count(struct shrinker *shrink, struct shrink_control *sc)