From patchwork Thu Jan 23 10:37:18 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13948127 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A9E64C0218C for ; Thu, 23 Jan 2025 10:38:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EC809280001; Thu, 23 Jan 2025 05:38:00 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DEC4F6B0089; Thu, 23 Jan 2025 05:38:00 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B4E18280002; Thu, 23 Jan 2025 05:38:00 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 8F31A6B0082 for ; Thu, 23 Jan 2025 05:38:00 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 19484480AD for ; Thu, 23 Jan 2025 10:38:00 +0000 (UTC) X-FDA: 83038366320.23.2E211BD Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) by imf28.hostedemail.com (Postfix) with ESMTP id B7248C000F for ; Thu, 23 Jan 2025 10:37:57 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=Hz8qowaP; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=yOpvhHOx; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=Hz8qowaP; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=yOpvhHOx; spf=pass (imf28.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.130 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1737628678; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=t3re1HhaNJ7ult5EVdCJjMXAxd3DND62tEqbEIKH98g=; b=SzHMTfEfiU5t/7u+97h0P6tTmb6kg86D4UmLre1+PBn7XO9jTM44ArA235jf1YE0dHoC4V VdFLQXvjHib9vI8jklEdFaqoXG4ZSPx5Yt/7Me/MLR/BIYtRpWzA4nHTANt6dERbqLzyiz CFxDDcHfGPwT2CxaqQsNFB+xY8/kU58= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=Hz8qowaP; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=yOpvhHOx; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=Hz8qowaP; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=yOpvhHOx; spf=pass (imf28.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.130 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1737628678; a=rsa-sha256; cv=none; b=D8pXpNt7ivY707gUtFJkh69XYcFhvTbHb5WLx1d13df5vrqhsTg/FmxGNIDHNYGnlqYeP9 hdtZNhwHSYtWSCMQajEP1gcVhpAPvpiSOyyfFYL6dAVPtAaiJdWosuf/17hZ37d96vobwT DGSwzQUq3aRs7zFpKKue6iXUXZD0d0s= Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 391CD2117D; Thu, 23 Jan 2025 10:37:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1737628676; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=t3re1HhaNJ7ult5EVdCJjMXAxd3DND62tEqbEIKH98g=; b=Hz8qowaPTUkggcfEOrGEszBiecjRHVJlrmCZNgrSyPprj+Zuiha7KX00JQvdMpeqxW+RMZ LV8NoOAG3sDGDylHf00TH1AQzw5Tcnqeqeh4ckhzI+/R+Yu85QlKye62CeJyqaw9W470Q+ d+eOvrweI1PAERjSOZIrvxmsxaJl78w= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1737628676; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=t3re1HhaNJ7ult5EVdCJjMXAxd3DND62tEqbEIKH98g=; b=yOpvhHOxTivTJkIIsCrFTvfOSaFb9/7Xab6dbAq6LINQzThBwm2xq9ORa34U1t/90zu3z1 ULPHOVu9wJ8DoTCA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1737628676; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=t3re1HhaNJ7ult5EVdCJjMXAxd3DND62tEqbEIKH98g=; b=Hz8qowaPTUkggcfEOrGEszBiecjRHVJlrmCZNgrSyPprj+Zuiha7KX00JQvdMpeqxW+RMZ LV8NoOAG3sDGDylHf00TH1AQzw5Tcnqeqeh4ckhzI+/R+Yu85QlKye62CeJyqaw9W470Q+ d+eOvrweI1PAERjSOZIrvxmsxaJl78w= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1737628676; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=t3re1HhaNJ7ult5EVdCJjMXAxd3DND62tEqbEIKH98g=; b=yOpvhHOxTivTJkIIsCrFTvfOSaFb9/7Xab6dbAq6LINQzThBwm2xq9ORa34U1t/90zu3z1 ULPHOVu9wJ8DoTCA== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 19F5B13A51; Thu, 23 Jan 2025 10:37:56 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id sHsPBgQckmdrAQAAD6G6ig (envelope-from ); Thu, 23 Jan 2025 10:37:56 +0000 From: Vlastimil Babka Date: Thu, 23 Jan 2025 11:37:18 +0100 Subject: [PATCH RFC 1/4] slab, rcu: move TINY_RCU variant of kvfree_rcu() to SLAB MIME-Version: 1.0 Message-Id: <20250123-slub-tiny-kfree_rcu-v1-1-0e386ef1541a@suse.cz> References: <20250123-slub-tiny-kfree_rcu-v1-0-0e386ef1541a@suse.cz> In-Reply-To: <20250123-slub-tiny-kfree_rcu-v1-0-0e386ef1541a@suse.cz> To: Christoph Lameter , David Rientjes , "Paul E. McKenney" , Joel Fernandes , Josh Triplett , Boqun Feng , Uladzislau Rezki Cc: Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang , rcu@vger.kernel.org, Vlastimil Babka X-Mailer: b4 0.14.2 X-Developer-Signature: v=1; a=openpgp-sha256; l=6322; i=vbabka@suse.cz; h=from:subject:message-id; bh=V3CDNBQ65RoS8BNy/YQECuCfUSQlpxlHN6MkeKNjSZU=; b=owEBbQGS/pANAwAIAbvgsHXSRYiaAcsmYgBnkhv5HlHabD5/rYKLN4U4Y8eOLgNzVkyqBNRv/ UW7L9RMLCWJATMEAAEIAB0WIQR7u8hBFZkjSJZITfG74LB10kWImgUCZ5Ib+QAKCRC74LB10kWI mmYXB/9NvbzObEFhfbeTeuiQh79OGIklwhK5mw1vivzIcgBKy/VoEuNPD2Q4EOY/2UDY8h8SOGb pk3ZUdjfO4GihAALiwwjpLYGwIDshktM3PYExu0souNPXsg6yi2q+bxJWA0um9+wL8K+ap1EYNJ oxEY74VfDKXszUz8vloXNiDa/N2KVxtMnOQQVEm5mie27HXlLtxhmy3RR3z+Rp6HYnKTtGNklX4 tjn5h34+NQKbNv2NvkKTI/0Nvmhle3j51IjmFwYELB/fQCoy0O9CgofE6WHjLdBAOtaH14N2rJY +Cdm3tVWyQJrn6JStD8UT7XKepFvp/mV/e4dfJcoJGPewXJh X-Developer-Key: i=vbabka@suse.cz; a=openpgp; fpr=A940D434992C2E8E99103D50224FA7E7CC82A664 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: B7248C000F X-Stat-Signature: zyqwgeb9pkgx4nemms1fr4cprhfe9bh8 X-Rspam-User: X-HE-Tag: 1737628677-760641 X-HE-Meta: U2FsdGVkX1/l+O7Bxix09UFOxk8yG8WJBkpYOMJfJJ+aYEk1zOcvaGmhXpTdyW55vdbp1BudxK6rs135Fr6/+YMhA94OrmafJTqkkuaX5vNE9cPpFqWUs+LDXe7nlnT8IdV4SnVjsL84gz9pq3y2oFs64jn/lhHhlfLGv0PcvURT2hMVwO7LB15cOwut/mckgkVqwASS1mdkOVOHZWPODv7If8ExKOqnLNJYOrJU8ZUDtCNojiRPcd9QXsb5hVPv6+LDZVhcGWkpTwR9nexhbynjy0qil3bkHikGkKO+5xdH/ZTOs7Gy9dom7n+z5Hnu+WMF9Grc5JbNV8fiGFxe/uAgXzvkadU6GK2RU7MthptVpkqKLg5Vd3XJbQgAS81SF9enLRIemApW4Axl+RZTlnCockwA0FTtwZ3HwZpWiFtIwd5zuXeYl8dFR7jeTOBZVbvmnAdHgUFyR930sdrhW5HcBUKRYGbJzitGjv02iLOkC3u+LKYhR+EG9eZ7dYAlgIsICcnio8iQGtmVDtQWLLX3JOWw+d7aD3WLEIz1NAMda0lDW/1FqaTe3c8RkU8ZHntfbmvtjJrP32LwAJv4KdJHYtu9JHgVzDlrzx7/fdDh5tO3Nac3lb2AgqgNNaPk8dSysXUUjox82VxjJnHTW4A5/+Ljei9NUYSILeJpu8cy5+slgJYLjA0U7uTxxcwWsHETsWXcdNSpfwS/Fp8QfSx4FsN7nEhoEdCNsbrIvai3szQwQmPrUAqMQYHTgUtkqa5VxBfRPO9yfANU67GX3wFVrZ1gYhW23O+M9ggfjE82QrGp9PZc6FWjLarG/t+OHREvCR+cml+wHOd15/XJ/NYYp1TYdbRJlqVSTj4lD1izxQjX91V6TwA1uDTA66ST04PEZfAhriJ8dosI0/UrSLYinwSHjErfxC0m/Gp6uEB1qpDiTM+h1+XQYCelVAAJyLYfHYlGn7uGC/2oeRr 32C301Zg g4V5hIKFJP1KmqkI5n1Zfeq5cM69lwAdEqmhuhCKg4NAe9d7rghGMshERfRhtyIlb57EFa+1peLgJi5P9VpQSv/OrhauAi6j1RqbH7B9vmbB0ygfyw97hNxj4pBwpx1PCjVhY09oUUNYSP4ukVXpCNYkmaOOBpTp/k1+h1yES588fTAIM5ZECUGYCoFqC74KpYpqtd7GAk/bSkQxDwsibZrDXSNgSy6ja9Ues1VO20IK2x6NhLGzJq1O46UKpQLPMQUfW7bk/8d1zO+8VoMBR95bp4LEAVEmLKOxWUOWwa/iWwlgBeh1ActuJt4asNw+VsFhvZolKB4wa1itnIfKLP8XcSWFV5tc7ZtRImIBCrGgpvG4D7xLA7qW4GQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Following the move of TREE_RCU implementation, let's move also the TINY_RCU one for consistency and subsequent refactoring. For simplicity, remove the separate inline __kvfree_call_rcu() as TINY_RCU is not meant for high-performance hardware anyway. Declare kvfree_call_rcu() in rcupdate.h to avoid header dependency issues. Also move the kvfree_rcu_barrier() declaration to slab.h Signed-off-by: Vlastimil Babka --- include/linux/rcupdate.h | 5 +++++ include/linux/rcutiny.h | 36 ------------------------------------ include/linux/rcutree.h | 3 --- include/linux/slab.h | 14 ++++++++++++++ kernel/rcu/tiny.c | 11 ----------- mm/slab_common.c | 20 ++++++++++++++++++-- 6 files changed, 37 insertions(+), 52 deletions(-) diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index 48e5c03df1dd83c246a61d0fcc8aa638adcd7654..3f70d1c8144426f40553c8c589f07097ece8a706 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -1082,6 +1082,11 @@ static inline notrace void rcu_read_unlock_sched_notrace(void) #define kfree_rcu_mightsleep(ptr) kvfree_rcu_arg_1(ptr) #define kvfree_rcu_mightsleep(ptr) kvfree_rcu_arg_1(ptr) +/* + * In mm/slab_common.c, no suitable header to include here. + */ +void kvfree_call_rcu(struct rcu_head *head, void *ptr); + #define kvfree_rcu_arg_2(ptr, rhf) \ do { \ typeof (ptr) ___p = (ptr); \ diff --git a/include/linux/rcutiny.h b/include/linux/rcutiny.h index fe42315f667fc5be7f2ed8eae6ea0c7193030846..f519cd6802286710bdd56588b5ff3d07bcd30b92 100644 --- a/include/linux/rcutiny.h +++ b/include/linux/rcutiny.h @@ -90,41 +90,6 @@ static inline void synchronize_rcu_expedited(void) synchronize_rcu(); } -/* - * Add one more declaration of kvfree() here. It is - * not so straight forward to just include - * where it is defined due to getting many compile - * errors caused by that include. - */ -extern void kvfree(const void *addr); - -static inline void __kvfree_call_rcu(struct rcu_head *head, void *ptr) -{ - if (head) { - call_rcu(head, (rcu_callback_t) ((void *) head - ptr)); - return; - } - - // kvfree_rcu(one_arg) call. - might_sleep(); - synchronize_rcu(); - kvfree(ptr); -} - -static inline void kvfree_rcu_barrier(void) -{ - rcu_barrier(); -} - -#ifdef CONFIG_KASAN_GENERIC -void kvfree_call_rcu(struct rcu_head *head, void *ptr); -#else -static inline void kvfree_call_rcu(struct rcu_head *head, void *ptr) -{ - __kvfree_call_rcu(head, ptr); -} -#endif - void rcu_qs(void); static inline void rcu_softirq_qs(void) @@ -164,7 +129,6 @@ static inline void rcu_end_inkernel_boot(void) { } static inline bool rcu_inkernel_boot_has_ended(void) { return true; } static inline bool rcu_is_watching(void) { return true; } static inline void rcu_momentary_eqs(void) { } -static inline void kfree_rcu_scheduler_running(void) { } /* Avoid RCU read-side critical sections leaking across. */ static inline void rcu_all_qs(void) { barrier(); } diff --git a/include/linux/rcutree.h b/include/linux/rcutree.h index 27d86d9127817e50f8d4dd79e1990d70a02435bb..dbe77b5fe06ec89a393b5444d6c479ced346a37b 100644 --- a/include/linux/rcutree.h +++ b/include/linux/rcutree.h @@ -34,12 +34,9 @@ static inline void rcu_virt_note_context_switch(void) } void synchronize_rcu_expedited(void); -void kvfree_call_rcu(struct rcu_head *head, void *ptr); -void kvfree_rcu_barrier(void); void rcu_barrier(void); void rcu_momentary_eqs(void); -void kfree_rcu_scheduler_running(void); struct rcu_gp_oldstate { unsigned long rgos_norm; diff --git a/include/linux/slab.h b/include/linux/slab.h index 09eedaecf1205672bb2e7c8cd57ae8fccebc2737..bcc62e5656c35c6a3f4caf26fb33d7447dead39a 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -16,6 +16,7 @@ #include #include #include +#include #include #include #include @@ -1082,6 +1083,19 @@ extern void kvfree_sensitive(const void *addr, size_t len); unsigned int kmem_cache_size(struct kmem_cache *s); +#ifdef CONFIG_TINY_RCU +static inline void kvfree_rcu_barrier(void) +{ + rcu_barrier(); +} + +static inline void kfree_rcu_scheduler_running(void) { } +#else +void kvfree_rcu_barrier(void); + +void kfree_rcu_scheduler_running(void); +#endif + /** * kmalloc_size_roundup - Report allocation bucket size for the given size * diff --git a/kernel/rcu/tiny.c b/kernel/rcu/tiny.c index b3b3ce34df6310f7bddba40b2be1bdf6c9f00232..0ec27093d0e14a4b1060ea08932c4ac13f9b0f26 100644 --- a/kernel/rcu/tiny.c +++ b/kernel/rcu/tiny.c @@ -246,17 +246,6 @@ bool poll_state_synchronize_rcu(unsigned long oldstate) } EXPORT_SYMBOL_GPL(poll_state_synchronize_rcu); -#ifdef CONFIG_KASAN_GENERIC -void kvfree_call_rcu(struct rcu_head *head, void *ptr) -{ - if (head) - kasan_record_aux_stack_noalloc(ptr); - - __kvfree_call_rcu(head, ptr); -} -EXPORT_SYMBOL_GPL(kvfree_call_rcu); -#endif - void __init rcu_init(void) { open_softirq(RCU_SOFTIRQ, rcu_process_callbacks); diff --git a/mm/slab_common.c b/mm/slab_common.c index 69f2d19010dedaa3e5b303ab9803c8cdd40152fa..330cdd8ebc5380090ee784c58e8ca1d1a52b3758 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1861,7 +1861,23 @@ add_ptr_to_bulk_krc_lock(struct kfree_rcu_cpu **krcp, return true; } -#if !defined(CONFIG_TINY_RCU) +#ifdef CONFIG_TINY_RCU + +void kvfree_call_rcu(struct rcu_head *head, void *ptr) +{ + if (head) { + kasan_record_aux_stack_noalloc(ptr); + call_rcu(head, (rcu_callback_t) ((void *) head - ptr)); + return; + } + + // kvfree_rcu(one_arg) call. + might_sleep(); + synchronize_rcu(); + kvfree(ptr); +} + +#else /* !CONFIG_TINY_RCU */ static enum hrtimer_restart schedule_page_work_fn(struct hrtimer *t) @@ -2071,7 +2087,7 @@ void kvfree_rcu_barrier(void) } EXPORT_SYMBOL_GPL(kvfree_rcu_barrier); -#endif /* #if !defined(CONFIG_TINY_RCU) */ +#endif /* !CONFIG_TINY_RCU */ static unsigned long kfree_rcu_shrink_count(struct shrinker *shrink, struct shrink_control *sc) From patchwork Thu Jan 23 10:37:19 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13948128 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3CAF6C02182 for ; Thu, 23 Jan 2025 10:38:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 498626B007B; Thu, 23 Jan 2025 05:38:01 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 38C496B0085; Thu, 23 Jan 2025 05:38:01 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CAC786B008A; Thu, 23 Jan 2025 05:38:00 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 91D3C6B0085 for ; Thu, 23 Jan 2025 05:38:00 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 576F2C0ED6 for ; Thu, 23 Jan 2025 10:38:00 +0000 (UTC) X-FDA: 83038366320.08.D50D870 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) by imf05.hostedemail.com (Postfix) with ESMTP id 1C97C10000B for ; Thu, 23 Jan 2025 10:37:57 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=j5OLBnpr; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=NUnVtHAs; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=j5OLBnpr; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=NUnVtHAs; spf=pass (imf05.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.130 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1737628678; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zwElIuMESkHawfK25L5K4feRdFtcRaanmIbm0q1JuoM=; b=ZmW6YkJeFCnpNkhoc88lzVixcuItIgdUbH89OTuLmxfwy8kHqTrfXtu3zN/iqgUR2X7VcL Vs3gTq7BWSg4LAH0yVcAC7Ra+yDq+RVvZD1fqDD6GLtdTFWyKEkJAXdn380JggEgigaYry TQ910vgZ+8q5+35wMZ0xRnDtWyzs0VA= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=j5OLBnpr; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=NUnVtHAs; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=j5OLBnpr; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=NUnVtHAs; spf=pass (imf05.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.130 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1737628678; a=rsa-sha256; cv=none; b=Hdrv1UkFJKCk+QEgPXVuOXffbgB4hwKvvi2CASW8lrPabcL78p1pFuNEPog2VjPlrhFEz7 B17xbK5CerZl1LIVW3riCqess8Tzdv3SYEgLmlDzZdlV1yhOwshF1y0FF7p+vS2e6iSQp8 7xVATOQKZzNJFYM0vT4eNMDIIWHAWLA= Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 531D72117E; Thu, 23 Jan 2025 10:37:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1737628676; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zwElIuMESkHawfK25L5K4feRdFtcRaanmIbm0q1JuoM=; b=j5OLBnprbJ+Q9D1+Whj6Ma7EQzwCeCix17crcPsfU5C71LVEK7enHOn127kJlDVtAzEJvB wAwN7Iyls5Ek8x2EGrPg2d2QrqU0Fph3BQRjUcP8jUTvsOsTKK/D9an7ouKsIQIDInY8X7 iPVdjyKSUvbc/XixUwXGvJG6o7zRJjY= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1737628676; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zwElIuMESkHawfK25L5K4feRdFtcRaanmIbm0q1JuoM=; b=NUnVtHAs2IGVZeowl1Nhf5Fv+i2x3tGq2tq8SYA/Cb6vj1Payt8CtFwER0n7cf30oDxhU5 fOZLc968+jXNO6Cg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1737628676; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zwElIuMESkHawfK25L5K4feRdFtcRaanmIbm0q1JuoM=; b=j5OLBnprbJ+Q9D1+Whj6Ma7EQzwCeCix17crcPsfU5C71LVEK7enHOn127kJlDVtAzEJvB wAwN7Iyls5Ek8x2EGrPg2d2QrqU0Fph3BQRjUcP8jUTvsOsTKK/D9an7ouKsIQIDInY8X7 iPVdjyKSUvbc/XixUwXGvJG6o7zRJjY= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1737628676; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zwElIuMESkHawfK25L5K4feRdFtcRaanmIbm0q1JuoM=; b=NUnVtHAs2IGVZeowl1Nhf5Fv+i2x3tGq2tq8SYA/Cb6vj1Payt8CtFwER0n7cf30oDxhU5 fOZLc968+jXNO6Cg== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 3444313A5B; Thu, 23 Jan 2025 10:37:56 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id gDt0DAQckmdrAQAAD6G6ig (envelope-from ); Thu, 23 Jan 2025 10:37:56 +0000 From: Vlastimil Babka Date: Thu, 23 Jan 2025 11:37:19 +0100 Subject: [PATCH RFC 2/4] rcu: remove trace_rcu_kvfree_callback MIME-Version: 1.0 Message-Id: <20250123-slub-tiny-kfree_rcu-v1-2-0e386ef1541a@suse.cz> References: <20250123-slub-tiny-kfree_rcu-v1-0-0e386ef1541a@suse.cz> In-Reply-To: <20250123-slub-tiny-kfree_rcu-v1-0-0e386ef1541a@suse.cz> To: Christoph Lameter , David Rientjes , "Paul E. McKenney" , Joel Fernandes , Josh Triplett , Boqun Feng , Uladzislau Rezki Cc: Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang , rcu@vger.kernel.org, Vlastimil Babka X-Mailer: b4 0.14.2 X-Developer-Signature: v=1; a=openpgp-sha256; l=2762; i=vbabka@suse.cz; h=from:subject:message-id; bh=GC9uYdt4KS07fymuM3NSYxhL9MvirmkNR5XsSlOEbeI=; b=owEBbQGS/pANAwAIAbvgsHXSRYiaAcsmYgBnkhv8v39EUjnAdbXssma/HuIwh410GUk5XJsV2 aWJ2mS6tbuJATMEAAEIAB0WIQR7u8hBFZkjSJZITfG74LB10kWImgUCZ5Ib/AAKCRC74LB10kWI mlFrB/9XB0cSa+GSbLHOboEHSo34AX4MdVIdnULTltV8UqQGI2wop2o8P5LtVC88DvpKSnuwiDx aFVn9UiybtK/Xh/ObijyalPTc5mFJbKGX0lOzXmSICVUv+y1BLZU0lUaUsaQB0zl29+hX5QQx7A 0tZmYSmyW9KN8Ne16OyMmIRGv2xiUCSGw5U6G4Vgoz4OY5R9Kfyw53YQ3Is90JtC8YLI4pAR3Ph q/EoLopr0cGGRLPPU6lOc1ETjxum6oZBij7P2UhRdXXSJY7GnvQ2X76OUp78XdeszbMRiZcfDX6 qgr4Buktl+rhHR/4MLU0q4J5gnypiqHnXhyhwPJHH45PhmUL X-Developer-Key: i=vbabka@suse.cz; a=openpgp; fpr=A940D434992C2E8E99103D50224FA7E7CC82A664 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 1C97C10000B X-Stat-Signature: g6ym7y516yhsma5u9fih13ninjnhy8rs X-Rspam-User: X-HE-Tag: 1737628677-634417 X-HE-Meta: U2FsdGVkX19hhy4B8L2MC0ARpkuOLrt1BcO8XzcR6Z25t0SaRhhJTzDGKJhtsJhCDHHB80Gw/282QRAw4nNDlbXh6MJHTe6V1Gq9OncYdxwOU/88f0aX8Abb6MI4Nm836oG9Lc18pFtymCGl9NpUUnSurhzaXQH2lnbTV3hBSq6DFccV2CN6+JElcO35cGDm8ACnTw7qS78/GuT1bKpw6Fsu6uh+a54St56MNhrzb991XFKheXKJmdFGo/481XsUYPvaxeFbkjRvIhAxgvHecADE4E3U84zPfO1FAP1ME5UJ7Dj69wJd9iiv+A8TmUMapxk4l7WBLJaBga2U6pb5e4O1F0Gy/czd/d8ZF/j+Xr/LXC8ThJhPq3kn6cSMJeIRPq4vWaaQaFCvpQyoYWxsLRlmd4umM4wGgKujwuDbDxEYmdj5r/G3hxck9SaDC6XP7DQtB2oDGX13vQjNXCwdPzRHi1Uci57aF0xIJKVK7T2d4aoSTPhEnD0+Tg2KmP2dADdX/e+HA5gEog8qsSbEcCsdbvVfn8vgGJRg+OBFtW10tJJEr9BU5zABJrB69LBUnvQzeoAm3EGoj8N/OH5/9THzwT2fPO7hMD5S5a+vpqk8RpqVFzUknI4JIitCePb2Tq5HH4H7EQGu5Z9Lg/k2hnhKQQ49XQ6s4ZhHXLBaKfGwCS95XcubC/J+vwufbYv1Yds6px5G0oHZ9AnF5PFAMb6q3D2cwSaZo2Hb7R1ELSEU7A8RZ82wROHmWo6c0l//aqY2MeTLfOAdfdNFD6iOGSbATTD+VAa2EiJdj7oW6U0lNo02b8yqlP4HdtbuvpcpUD6sjZ9ZstsfGtXDdfYxj5Ra4fYFTBbiGp8gucnfjE1a5BW3yAfU1c+5mHLeRaeMLP/XR0wTIL8PdsoxCyi5b62MN/G8xPebrz4ubzkJsC27STB0kX3lqNICAIBA0q4CJeJZGiUPT/1fcoHfqcK rtkyVa8R GERGKXt1aiH5RN9y4JQmI8kAsX+uQeLbXUMdmPmPT43A39Em4V4Tz5oBAnJJVZYQEz1/EK/l8sQS4XrtTxLV+EvX4vkkEGfeuFuqdfbHdc6F6BqGMVJgRqgcYsxVuSDBrRh/ALh6JK1UN37y0YW3r2R2fYcsEixSiZxdrsvu4rUdJZHmRzNT5IYnaVeT1vWCvK+u/1Rixd4CtBClntIVDBiLd19xOyJTyyA5weRZw44hyIJXDdUhPrZsCCK3i3Be76qWcaPeR5gtrwpLn7PlrEkkFGMh9hGL1yVVj5W6zxHDHNPBGnkXWItrf83sUs3z0N9tdSJIMGmx/ZXPaQSa7/hFsh5/jhkRjEHWNltUxlSBw5wIAxJwyWz2SNw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Tree RCU does not handle kvfree_rcu() by queueing individual objects by call_rcu() anymore, thus the tracepoint and associated __is_kvfree_rcu_offset() check is dead code now. Remove it. Signed-off-by: Vlastimil Babka --- include/trace/events/rcu.h | 34 ---------------------------------- kernel/rcu/tree.c | 9 ++------- 2 files changed, 2 insertions(+), 41 deletions(-) diff --git a/include/trace/events/rcu.h b/include/trace/events/rcu.h index e81431deaa50b21725b08ad7c1a2992a0074065a..ac3b28b8939b78843aecb44bed8cb76c451ab060 100644 --- a/include/trace/events/rcu.h +++ b/include/trace/events/rcu.h @@ -560,40 +560,6 @@ TRACE_EVENT_RCU(rcu_segcb_stats, ); -/* - * Tracepoint for the registration of a single RCU callback of the special - * kvfree() form. The first argument is the RCU type, the second argument - * is a pointer to the RCU callback, the third argument is the offset - * of the callback within the enclosing RCU-protected data structure, - * the fourth argument is the number of lazy callbacks queued, and the - * fifth argument is the total number of callbacks queued. - */ -TRACE_EVENT_RCU(rcu_kvfree_callback, - - TP_PROTO(const char *rcuname, struct rcu_head *rhp, unsigned long offset, - long qlen), - - TP_ARGS(rcuname, rhp, offset, qlen), - - TP_STRUCT__entry( - __field(const char *, rcuname) - __field(void *, rhp) - __field(unsigned long, offset) - __field(long, qlen) - ), - - TP_fast_assign( - __entry->rcuname = rcuname; - __entry->rhp = rhp; - __entry->offset = offset; - __entry->qlen = qlen; - ), - - TP_printk("%s rhp=%p func=%ld %ld", - __entry->rcuname, __entry->rhp, __entry->offset, - __entry->qlen) -); - /* * Tracepoint for marking the beginning rcu_do_batch, performed to start * RCU callback invocation. The first argument is the RCU flavor, diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 6af042cde9727c8d7f7e3d88f26ba222d0d9c535..f3d5a9caebd9d0772931725bee93c1e172f8b6da 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2932,13 +2932,8 @@ static int __init rcu_spawn_core_kthreads(void) static void rcutree_enqueue(struct rcu_data *rdp, struct rcu_head *head, rcu_callback_t func) { rcu_segcblist_enqueue(&rdp->cblist, head); - if (__is_kvfree_rcu_offset((unsigned long)func)) - trace_rcu_kvfree_callback(rcu_state.name, head, - (unsigned long)func, - rcu_segcblist_n_cbs(&rdp->cblist)); - else - trace_rcu_callback(rcu_state.name, head, - rcu_segcblist_n_cbs(&rdp->cblist)); + trace_rcu_callback(rcu_state.name, head, + rcu_segcblist_n_cbs(&rdp->cblist)); trace_rcu_segcb_stats(&rdp->cblist, TPS("SegCBQueued")); } From patchwork Thu Jan 23 10:37:20 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13948126 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 53087C02182 for ; Thu, 23 Jan 2025 10:38:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C13EB6B0082; Thu, 23 Jan 2025 05:38:00 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BC5976B007B; Thu, 23 Jan 2025 05:38:00 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A65E9280001; Thu, 23 Jan 2025 05:38:00 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 7F5F26B007B for ; Thu, 23 Jan 2025 05:38:00 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 29985120E9E for ; Thu, 23 Jan 2025 10:38:00 +0000 (UTC) X-FDA: 83038366320.19.690D5FC Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) by imf29.hostedemail.com (Postfix) with ESMTP id CF281120008 for ; Thu, 23 Jan 2025 10:37:57 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf29.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.131 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1737628678; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hYjobfsRV6Bv9yMyF621UzjH0Jk51dGWfTZXNQOPQhA=; b=HDx4s06naYtcbne0uWGP+GmCJAVawJDwQyXynAHCof+1/kW26O39AbKtTCSEU62uev+nY/ UwK5WUq3rS7zjawfN3BOEQUaLxowDI+tFd5xX3rojF1gIzfUUaUjUXIvR7GSSwlHYe8RPt ftc25dA97ns2TYAqJ6zQxoOgI+wpX94= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1737628678; a=rsa-sha256; cv=none; b=X17FewLUL1lYwdrkV9hr+FdZCcuPi1lhtrf1ddimlzMfvajmW48OJ9PPmHwZp7blGI7c6E Bvdr5Cgci6ADs3/gkdTzyWqYPxEVW5wGOyMm0hW0w3zqJXRHz1YVpjQrZwOKxMGiaR49Gu FqTiey+6+hF8EDom7erveaW6FNxvhIw= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf29.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.131 as permitted sender) smtp.mailfrom=vbabka@suse.cz Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 691061F38E; Thu, 23 Jan 2025 10:37:56 +0000 (UTC) Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 4E17913AEB; Thu, 23 Jan 2025 10:37:56 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id 0J/IEgQckmdrAQAAD6G6ig (envelope-from ); Thu, 23 Jan 2025 10:37:56 +0000 From: Vlastimil Babka Date: Thu, 23 Jan 2025 11:37:20 +0100 Subject: [PATCH RFC 3/4] rcu, slab: use a regular callback function for kvfree_rcu MIME-Version: 1.0 Message-Id: <20250123-slub-tiny-kfree_rcu-v1-3-0e386ef1541a@suse.cz> References: <20250123-slub-tiny-kfree_rcu-v1-0-0e386ef1541a@suse.cz> In-Reply-To: <20250123-slub-tiny-kfree_rcu-v1-0-0e386ef1541a@suse.cz> To: Christoph Lameter , David Rientjes , "Paul E. McKenney" , Joel Fernandes , Josh Triplett , Boqun Feng , Uladzislau Rezki Cc: Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang , rcu@vger.kernel.org, Vlastimil Babka X-Mailer: b4 0.14.2 X-Developer-Signature: v=1; a=openpgp-sha256; l=7919; i=vbabka@suse.cz; h=from:subject:message-id; bh=WKAvcWMavpHcTSt989mw1tvjqHP9LL5fcBho8SiSQTA=; b=owEBbQGS/pANAwAIAbvgsHXSRYiaAcsmYgBnkhv+Bcof+4VLA8ZCmcNF8uGQ+HPFDOAc3ycvb OlK4KbXIoGJATMEAAEIAB0WIQR7u8hBFZkjSJZITfG74LB10kWImgUCZ5Ib/gAKCRC74LB10kWI mjj9B/wLJAtKN4UJIhpxj+DobUk7Ek5tOmZzstc+dAZsuRqyn198EWVLIS8Uhc0dWq63TqWdQWn /UVNlKV69e0eP38w+/gU5B+BTgFzQ1kkJmxQtUpfao/K0COlP5oIriK+j2W645dlUaK/ztsuTWi Obv0B/MI2gi72hTbQx6gvkzvsRG2utl7kcHKbJqkowShnxyuHhamYklwDT7hk6OLm3agO2tIyRi 3z+cx3ya23EMKYEKGJjHMvhcAGq5pMRUM8jwTvs5ZozAQnw57YEqC3F6FoyxpONhF4yKFNHriGo zanwGQGLGDP/OTdWk4WhZ34dM18Hgr19k/vrwKa6VvcpRYLk X-Developer-Key: i=vbabka@suse.cz; a=openpgp; fpr=A940D434992C2E8E99103D50224FA7E7CC82A664 X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Action: no action X-Rspamd-Queue-Id: CF281120008 X-Stat-Signature: x4ddnn8xd6h3z47iz6czbocu5kc53cuu X-Rspam-User: X-Rspamd-Server: rspam12 X-HE-Tag: 1737628677-450439 X-HE-Meta: U2FsdGVkX19pmER7hULD6+WjCozlGa8p8G/o/36vESW+RxS8I55/84iRcCkfiAZtkNz26KKPXpyCMfZqyTEvqjg05a71SuuAd8bPAd8xvAuuxwGfuEsfUWfeVypiFZ1e3E2ckWEWRht6ov5W1IuRy+DjGm0JqaJyEIQKIXnPj2bd092DrcluVSBrsf1SHS8mZDuW1J2uq8W1QjR6yM6OvwrFX0kUlUMOqO+TIa2MDhrFmTgfSU+VtAl7ed8QSAymlyP0tMNgwVZ+74J4/7tMqi6Q/XvOZtV4Bkh9IQW5+q2hjbOV2LzCYU3ahALmCR84P+dyMkgjX96ZATlHRIc3iZVc1tJ8t54Mvu9T0zjf1loDvUbKaiOpaPEqq6Doch0REelBtV3mdXzV8Z8zgyps93OUtcl4y/iDJP6/PVDZ+s1oQB2G3C6XRswo7Onn2YvFA4ZAW+cam0AnnYSWNCZxBNWzhNo+aWBjVXyZMRskSzFtnoWGUJ4upyfh6TDJJNEK0tOICTelTkwwJqRYuP7g26NymtSku1aHnQD5ksScqqewPB3dps8VGX7Nb0W7BkpsQmVfZtBp92o2o6wmBwzmPxj6nQyLeFFHpAG/f7cfEN/C8fM57oW6I9p+uvFEiPKv445WshAAtIoU4EyMQ3up7gBrSNd9o5xuvsjOAQca0cG0FSan7e3FXJDdl8nbAUchuF3r7bNgsjejPj3R922BgUNjQ262xAvckBMpaUGK8GCTnpWFpa0IwDP+dneW+uNImYhd1H9hX0joJTIi91vI3/aQguDnlwknvASTi/ty1gbwj5MweY1jkDmCTxTKUhfA96zLUihZ7AL8M9nVV2S5MKUxCAhZR/c89rsum72EFs8qf+8eswwcM7a/jSP9IynXtv8UD/GHN2OUTdMgE17inC9yjegxuCvx9/D/Sd6aYeeDZeCy2JyUDvFC4G6MUKjZLBy2rkxRib4jHubmWAQ /pWnZdvF gi/mhcT8SyKgZs+3/m+IYy2DdNUIfY0wNSGmmLP9212mpVYygXeElo6tLFzgOf2ltddhTfdNWC+zcup7EEQZ521tXYf22MocsMB+erf26t2c4UXS+roq0b6jEq/pMRXT2B/0TT7V3/dXtZRH5c1Ep32GAvI4ceL0U2NtXjD2u2HNxoZxYFnOVchbrgN9mkNyCFb8FLewPkFU+YHZc/UKZwktx5aLoOFY8JCi14gCGzQTLVA4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: RCU has been special-casing callback function pointers that are integers lower than 4096 as offsets of rcu_head for kvfree() instead. The tree RCU implementation no longer does that as the batched kvfree_rcu() is not a simple call_rcu(). The tiny RCU still does, and the plan is also to make tree RCU use call_rcu() for SLUB_TINY configurations. Instead of teaching tree RCU again to special case the offsets, let's remove the special casing completely. Since there's no SLOB anymore, it is possible to create a callback function that can take a pointer to a middle of slab object with unknown offset and determine the object's pointer before freeing it, so implement that as kvfree_rcu_cb(). Large kmalloc and vmalloc allocations are handled simply by aligning down to page size. For that we retain the requirement that the offset is smaller than 4096. But we can remove __is_kvfree_rcu_offset() completely and instead just opencode the condition in the BUILD_BUG_ON() check. Signed-off-by: Vlastimil Babka --- include/linux/rcupdate.h | 24 +++++++++--------------- kernel/rcu/tiny.c | 13 ------------- mm/slab.h | 2 ++ mm/slab_common.c | 5 +---- mm/slub.c | 42 ++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 54 insertions(+), 32 deletions(-) diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index 3f70d1c8144426f40553c8c589f07097ece8a706..7ff16a70ca1c0fb1012c4118388f60687c5e5b3f 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -1025,12 +1025,6 @@ static inline notrace void rcu_read_unlock_sched_notrace(void) #define RCU_POINTER_INITIALIZER(p, v) \ .p = RCU_INITIALIZER(v) -/* - * Does the specified offset indicate that the corresponding rcu_head - * structure can be handled by kvfree_rcu()? - */ -#define __is_kvfree_rcu_offset(offset) ((offset) < 4096) - /** * kfree_rcu() - kfree an object after a grace period. * @ptr: pointer to kfree for double-argument invocations. @@ -1041,11 +1035,11 @@ static inline notrace void rcu_read_unlock_sched_notrace(void) * when they are used in a kernel module, that module must invoke the * high-latency rcu_barrier() function at module-unload time. * - * The kfree_rcu() function handles this issue. Rather than encoding a - * function address in the embedded rcu_head structure, kfree_rcu() instead - * encodes the offset of the rcu_head structure within the base structure. - * Because the functions are not allowed in the low-order 4096 bytes of - * kernel virtual memory, offsets up to 4095 bytes can be accommodated. + * The kfree_rcu() function handles this issue. In order to have a universal + * callback function handling different offsets of rcu_head, the callback needs + * to determine the starting address of the freed object, which can be a large + * kmalloc of vmalloc allocation. To allow simply aligning the pointer down to + * page boundary for those, only offsets up to 4095 bytes can be accommodated. * If the offset is larger than 4095 bytes, a compile-time error will * be generated in kvfree_rcu_arg_2(). If this error is triggered, you can * either fall back to use of call_rcu() or rearrange the structure to @@ -1091,10 +1085,10 @@ void kvfree_call_rcu(struct rcu_head *head, void *ptr); do { \ typeof (ptr) ___p = (ptr); \ \ - if (___p) { \ - BUILD_BUG_ON(!__is_kvfree_rcu_offset(offsetof(typeof(*(ptr)), rhf))); \ - kvfree_call_rcu(&((___p)->rhf), (void *) (___p)); \ - } \ + if (___p) { \ + BUILD_BUG_ON(offsetof(typeof(*(ptr)), rhf) >= 4096); \ + kvfree_call_rcu(&((___p)->rhf), (void *) (___p)); \ + } \ } while (0) #define kvfree_rcu_arg_1(ptr) \ diff --git a/kernel/rcu/tiny.c b/kernel/rcu/tiny.c index 0ec27093d0e14a4b1060ea08932c4ac13f9b0f26..77e0db0221364376a99ebeb17485650879385a6e 100644 --- a/kernel/rcu/tiny.c +++ b/kernel/rcu/tiny.c @@ -88,12 +88,6 @@ static inline bool rcu_reclaim_tiny(struct rcu_head *head) unsigned long offset = (unsigned long)head->func; rcu_lock_acquire(&rcu_callback_map); - if (__is_kvfree_rcu_offset(offset)) { - trace_rcu_invoke_kvfree_callback("", head, offset); - kvfree((void *)head - offset); - rcu_lock_release(&rcu_callback_map); - return true; - } trace_rcu_invoke_callback("", head); f = head->func; @@ -159,10 +153,6 @@ void synchronize_rcu(void) } EXPORT_SYMBOL_GPL(synchronize_rcu); -static void tiny_rcu_leak_callback(struct rcu_head *rhp) -{ -} - /* * Post an RCU callback to be invoked after the end of an RCU grace * period. But since we have but one CPU, that would be after any @@ -178,9 +168,6 @@ void call_rcu(struct rcu_head *head, rcu_callback_t func) pr_err("%s(): Double-freed CB %p->%pS()!!! ", __func__, head, head->func); mem_dump_obj(head); } - - if (!__is_kvfree_rcu_offset((unsigned long)head->func)) - WRITE_ONCE(head->func, tiny_rcu_leak_callback); return; } diff --git a/mm/slab.h b/mm/slab.h index e9fd9bf0bfa65b343a4ae0ecd5b4c2a325b04883..2f01c7317988ce036f0b22807403226a59f0f708 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -604,6 +604,8 @@ void __memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p, int objects, struct slabobj_ext *obj_exts); #endif +void kvfree_rcu_cb(struct rcu_head *head); + size_t __ksize(const void *objp); static inline size_t slab_ksize(const struct kmem_cache *s) diff --git a/mm/slab_common.c b/mm/slab_common.c index 330cdd8ebc5380090ee784c58e8ca1d1a52b3758..f13d2c901daf1419993620459fbd5845eecb85f1 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1532,9 +1532,6 @@ kvfree_rcu_list(struct rcu_head *head) rcu_lock_acquire(&rcu_callback_map); trace_rcu_invoke_kvfree_callback("slab", head, offset); - if (!WARN_ON_ONCE(!__is_kvfree_rcu_offset(offset))) - kvfree(ptr); - rcu_lock_release(&rcu_callback_map); cond_resched_tasks_rcu_qs(); } @@ -1867,7 +1864,7 @@ void kvfree_call_rcu(struct rcu_head *head, void *ptr) { if (head) { kasan_record_aux_stack_noalloc(ptr); - call_rcu(head, (rcu_callback_t) ((void *) head - ptr)); + call_rcu(head, kvfree_rcu_cb); return; } diff --git a/mm/slub.c b/mm/slub.c index c2151c9fee228d121a9cbcc220c3ae054769dacf..651381bf05566e88de8493e0550f121d23b757a1 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -19,6 +19,7 @@ #include #include #include "slab.h" +#include #include #include #include @@ -4732,6 +4733,47 @@ static void free_large_kmalloc(struct folio *folio, void *object) folio_put(folio); } +void kvfree_rcu_cb(struct rcu_head *head) +{ + void *obj = head; + struct folio *folio; + struct slab *slab; + struct kmem_cache *s; + void *slab_addr; + + if (unlikely(is_vmalloc_addr(obj))) { + obj = (void *) PAGE_ALIGN_DOWN((unsigned long)obj); + vfree(obj); + return; + } + + folio = virt_to_folio(obj); + if (unlikely(!folio_test_slab(folio))) { + /* + * rcu_head offset can be only less than page size so no need to + * consider folio order + */ + obj = (void *) PAGE_ALIGN_DOWN((unsigned long)obj); + free_large_kmalloc(folio, obj); + return; + } + + slab = folio_slab(folio); + s = slab->slab_cache; + slab_addr = folio_address(folio); + + if (is_kfence_address(obj)) { + obj = kfence_object_start(obj); + } else { + unsigned int idx = __obj_to_index(s, slab_addr, obj); + + obj = slab_addr + s->size * idx; + obj = fixup_red_left(s, obj); + } + + slab_free(s, slab, obj, _RET_IP_); +} + /** * kfree - free previously allocated memory * @object: pointer returned by kmalloc() or kmem_cache_alloc() From patchwork Thu Jan 23 10:37:21 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13948130 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EC29FC0218B for ; Thu, 23 Jan 2025 10:38:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 967276B0083; Thu, 23 Jan 2025 05:38:01 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 61E5A6B0089; Thu, 23 Jan 2025 05:38:01 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 24F696B008C; Thu, 23 Jan 2025 05:38:01 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id C1D396B0088 for ; Thu, 23 Jan 2025 05:38:00 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 6E4421A0E65 for ; Thu, 23 Jan 2025 10:38:00 +0000 (UTC) X-FDA: 83038366320.25.BAE90CA Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) by imf26.hostedemail.com (Postfix) with ESMTP id 29A05140007 for ; Thu, 23 Jan 2025 10:37:57 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf26.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.131 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1737628678; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bJU9oR0arnP8n1uRZSYVsRUHicbwL4JDggXTdjwbv4M=; b=n2N3AhfklRw+NV/UG0T7tzOzGlMCojt0l0NsMqZ1jJhPgW2XPionDMzdSB9Tj7XgCSu22W 6Kx3WmlRAjC0jxUQUZ7Q6Y/Cvt2EOVglRTKMZ+bN4kyozv6fNwmoBib7kH+8f0RQdxTFyl Vjnsjr05yT3W6kmtGtNrh15eMqaXMig= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf26.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.131 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1737628678; a=rsa-sha256; cv=none; b=dA8NyKMRgoIavnWULFMfwDkOJ7IHzvFin4N2s7DlWTWW16dUEguPby8MHKEtb0zrh1itQS wrU8N/hNMV8vuFTRdJulaEp92o61geTUmMYjJGa2ZW1Q/DNa9LmmhU40Z4isU9dvbEmVuw SrhLKjVHz7WKEsug6HTH4amTNXiJtA0= Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 8E1421F38C; Thu, 23 Jan 2025 10:37:56 +0000 (UTC) Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 66BEE136A1; Thu, 23 Jan 2025 10:37:56 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id cBjSGAQckmdrAQAAD6G6ig (envelope-from ); Thu, 23 Jan 2025 10:37:56 +0000 From: Vlastimil Babka Date: Thu, 23 Jan 2025 11:37:21 +0100 Subject: [PATCH RFC 4/4] slab: don't batch kvfree_rcu() with SLUB_TINY MIME-Version: 1.0 Message-Id: <20250123-slub-tiny-kfree_rcu-v1-4-0e386ef1541a@suse.cz> References: <20250123-slub-tiny-kfree_rcu-v1-0-0e386ef1541a@suse.cz> In-Reply-To: <20250123-slub-tiny-kfree_rcu-v1-0-0e386ef1541a@suse.cz> To: Christoph Lameter , David Rientjes , "Paul E. McKenney" , Joel Fernandes , Josh Triplett , Boqun Feng , Uladzislau Rezki Cc: Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang , rcu@vger.kernel.org, Vlastimil Babka X-Mailer: b4 0.14.2 X-Developer-Signature: v=1; a=openpgp-sha256; l=3787; i=vbabka@suse.cz; h=from:subject:message-id; bh=2rhWfguSgm3rCjGx5bIjjhtFSjU4v/3Sg/+BOOfL7Eo=; b=owEBbQGS/pANAwAIAbvgsHXSRYiaAcsmYgBnkhwBZZrvN3xrjWEE5R0q5c5uyHyaO6141n+Pt VpRWGoQd/qJATMEAAEIAB0WIQR7u8hBFZkjSJZITfG74LB10kWImgUCZ5IcAQAKCRC74LB10kWI mimFB/9iIoogv+10l0+P5335y9cW2V1fDBnT4BAFsftyWIlYMy6+BAK2pDqBePJdg5UOgEmSC3u dpsuw6cicMFvMLdmuNNgIB7jD0rX2vjQI9Q6TFpwDJW8MFgr85+c3/jesi40ihe8suwsv0uedHQ GOmLBikFf/xFxleq5oXTj6GhmdV82aobIrz+gMXc5aDP7+HjmIzjDJCUtJWb4MVVLrCNB8lfrNr 3bU8tPhGbyLjYuQv2jzKjdnM26rGPFlLPoFinwjayQ1IJSVA+wePpQGlnkrNZSOEf1lQ/JzYM1d /x27eZhMmwYu4rbKBHVaU+QfAz+KUiKGsRok7XaPXclI1R4O X-Developer-Key: i=vbabka@suse.cz; a=openpgp; fpr=A940D434992C2E8E99103D50224FA7E7CC82A664 X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Action: no action X-Rspamd-Queue-Id: 29A05140007 X-Stat-Signature: fxpm78mzxdypribnkwordryoixo6s76d X-Rspamd-Server: rspam08 X-Rspam-User: X-HE-Tag: 1737628677-182418 X-HE-Meta: U2FsdGVkX18dMrMYevTaKH2JHhJDIu3WRfCigrnHXAzAmVV3ETRDev14GTn9vqbdxpGClZOdDFpavAdhWD337LDwR+xYyTEdkcSsn9LtsTUa5KBzVT8LcLCYWLnxfppi8k8LkUXIUeQsaCy1B2wR0FGYehFYwf0brG2ptrLuXKV6yrRBDYj79jAT0n198UqCJSwJwCXr++lpCP7kQL5KMQAbMpzrfvyEvQlHhaVlIgTOdSWS4KqsR0ewpOkmkqHTdo0o/iew9NMKTHMDXBhIma/SRCnQ/3AobfKjLPBn79LJIrDacnxVwdtgWO+0s2yC+dMNjOEf4NHFVnqkwsT+L4XxvzeVrQDTiuvTU99rfHe+D/Riw+ALdHS7/OP3PSmDb4MjwaG3PaHVttxonZsdO2rIp3xdEvYUfBLycANQvpz7akf+J+MAFO2Y8CQ2EIAtIPWDoLVopERbAS1LjTqpzHJq1B770Yim/91CE/MxmTLpkjV+pfHOKOAgLVgVYN2BZgaaQjwyh6nljHIgJObjO1G2veWAeNOoU0BD9ltMJGjpAjNkI51Ef3fwNnOTsFgi2ph5EXEC15PIJfnRtzN5ect6YFhaSoVy5sjk+b69G4ldF2yOA0t7bJhwI8oTu6Ndw7wUt2sL3bztEQ5ojHvki6epLEzk2LivSkV1f/Cf2s7G/uR2cNUlz9+Z5ows+ffWN1NJNIHpym9w/xaUqvA/Egke6TLXZ7wMUL8llGylQD+wGLSvALgzALeH8iLfxP7JjycYEsHciq/wetJT2yGjmibifCR2Dn6cCcBqBod3xBuA7JvLK1Z7oNEfI51Ox8rLNE5kbBihWD6oml3WVxsU2QAiigjS5Bnn5qMKysfI2ps32X10H2fSYuZIm+l5AtKsjp0mXnVhhExpC/Vgj1kzI5nvBhEkjEGHoNUxKzL4FKILNNFdOqcutjORAaPkGsd0YRXXeYP2e9thM/nfIYm W01fn2gV zcQH8HTrovLpmXSYYFHzIXONQZP0hrkNxVl6aw/sli1lDOE8hB1w4Zw2MVHEiDSE+iFdjKCtZ0Th3fM/fKs/GaQV7gl6lBfuPk31xKA5O1pt5zS72nwibPbI8dprkFIE8U8hA6D4HIjX4dyFACqFuHsxeLLPX53iKkNbjg6ik6nQPDKytxcmCBOYjepZkTxHesiS0R0RwFtyDbco5eqONZK9wvr0O8ZCugMKssERGT9Glud8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: kvfree_rcu() is batched for better performance except on TINY_RCU, which is a simple implementation for small UP systems. Similarly SLUB_TINY is an option intended for small systems, whether or not used together with TINY_RCU. In case SLUB_TINY is used with !TINY_RCU, it makes arguably sense to not do the batching and limit the memory footprint. It's also suboptimal to have RCU-specific #ifdefs in slab code. With that, add CONFIG_KFREE_RCU_BATCHED to determine whether batching kvfree_rcu() implementation is used. It is not set by a user prompt, but enabled by default and disabled in case TINY_RCU or SLUB_TINY are enabled. Use the new config for #ifdef's in slab code and extend their scope to cover all code used by the batched kvfree_rcu(). For example there's no need to perform kvfree_rcu_init() if the batching is disabled. Signed-off-by: Vlastimil Babka --- include/linux/slab.h | 2 +- mm/Kconfig | 4 ++++ mm/slab_common.c | 45 +++++++++++++++++++++++++-------------------- 3 files changed, 30 insertions(+), 21 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index bcc62e5656c35c6a3f4caf26fb33d7447dead39a..9faf33734a8eee2425b90e679c0457ab459422a3 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -1083,7 +1083,7 @@ extern void kvfree_sensitive(const void *addr, size_t len); unsigned int kmem_cache_size(struct kmem_cache *s); -#ifdef CONFIG_TINY_RCU +#ifndef CONFIG_KFREE_RCU_BATCHED static inline void kvfree_rcu_barrier(void) { rcu_barrier(); diff --git a/mm/Kconfig b/mm/Kconfig index 84000b01680869801a10f56f06d0c43d6521a8d2..e513308a4aed640ee556ecb5793c7f3f195bbcae 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -242,6 +242,10 @@ menu "Slab allocator options" config SLUB def_bool y +config KFREE_RCU_BATCHED + def_bool y + depends on !SLUB_TINY && !TINY_RCU + config SLUB_TINY bool "Configure for minimal memory footprint" depends on EXPERT diff --git a/mm/slab_common.c b/mm/slab_common.c index f13d2c901daf1419993620459fbd5845eecb85f1..9f6d66313afc6684bdc0f32908fe01c83c60f283 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1284,6 +1284,28 @@ EXPORT_TRACEPOINT_SYMBOL(kmem_cache_alloc); EXPORT_TRACEPOINT_SYMBOL(kfree); EXPORT_TRACEPOINT_SYMBOL(kmem_cache_free); +#ifndef CONFIG_KFREE_RCU_BATCHED + +void kvfree_call_rcu(struct rcu_head *head, void *ptr) +{ + if (head) { + kasan_record_aux_stack_noalloc(ptr); + call_rcu(head, kvfree_rcu_cb); + return; + } + + // kvfree_rcu(one_arg) call. + might_sleep(); + synchronize_rcu(); + kvfree(ptr); +} + +void __init kvfree_rcu_init(void) +{ +} + +#else /* CONFIG_KFREE_RCU_BATCHED */ + /* * This rcu parameter is runtime-read-only. It reflects * a minimum allowed number of objects which can be cached @@ -1858,24 +1880,6 @@ add_ptr_to_bulk_krc_lock(struct kfree_rcu_cpu **krcp, return true; } -#ifdef CONFIG_TINY_RCU - -void kvfree_call_rcu(struct rcu_head *head, void *ptr) -{ - if (head) { - kasan_record_aux_stack_noalloc(ptr); - call_rcu(head, kvfree_rcu_cb); - return; - } - - // kvfree_rcu(one_arg) call. - might_sleep(); - synchronize_rcu(); - kvfree(ptr); -} - -#else /* !CONFIG_TINY_RCU */ - static enum hrtimer_restart schedule_page_work_fn(struct hrtimer *t) { @@ -2084,8 +2088,6 @@ void kvfree_rcu_barrier(void) } EXPORT_SYMBOL_GPL(kvfree_rcu_barrier); -#endif /* !CONFIG_TINY_RCU */ - static unsigned long kfree_rcu_shrink_count(struct shrinker *shrink, struct shrink_control *sc) { @@ -2175,3 +2177,6 @@ void __init kvfree_rcu_init(void) shrinker_register(kfree_rcu_shrinker); } + +#endif /* CONFIG_KFREE_RCU_BATCHED */ +