From patchwork Mon Feb 3 09:28:47 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13957133 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D792EC02192 for ; Mon, 3 Feb 2025 09:30:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6D7E7280012; Mon, 3 Feb 2025 04:30:39 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 66133280002; Mon, 3 Feb 2025 04:30:39 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4B344280012; Mon, 3 Feb 2025 04:30:39 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 294E6280002 for ; Mon, 3 Feb 2025 04:30:39 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 66C76B4F8B for ; Mon, 3 Feb 2025 09:29:32 +0000 (UTC) X-FDA: 83078110626.28.BFDEA68 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) by imf12.hostedemail.com (Postfix) with ESMTP id 1C05A4000C for ; Mon, 3 Feb 2025 09:29:29 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=MzMfL4uI; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=pHThM6s8; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=MzMfL4uI; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=pHThM6s8; spf=pass (imf12.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.131 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738574970; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XXDMn7CNaiwzg65IjzvDMWn1fcydgyKYjKgaCniNb+k=; b=gIrN5iPVOMcCCTexHJ2Yps3CrUF7FEZWGnYJKCeR7oC8HVNA+bXI4c2F7O1vYyUF/gAwAt D+anm6tnePbLLN0MD3C09EqQi3VN0Y0xjCOiX/ARmh2E3+aKwYIAc3hR1kRrf3g1Hr/CI3 rxK/iQEZBNT8IbQxypYwYzuQhrYoLfs= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=MzMfL4uI; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=pHThM6s8; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=MzMfL4uI; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=pHThM6s8; spf=pass (imf12.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.131 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738574970; a=rsa-sha256; cv=none; b=0tig3o1625aqoJpd4jEufne9XmTKHLx/Wpn0E7PIWGkNjBjjP2AZKNEw0OK5L3zRos/l4Y +JOdqUDGT4Q48257RQlCWl4V4XFsY8U8mb9Fbpg7iPsC6NniXyDleMUSfh4VW3opAO587c IaWHAR1F6FMNJjT/dsV014lG1C9c9VE= Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 7F68A1F37C; Mon, 3 Feb 2025 09:29:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1738574968; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XXDMn7CNaiwzg65IjzvDMWn1fcydgyKYjKgaCniNb+k=; b=MzMfL4uIU/3QqymoYS4v8PqmFnz21cfP5NEngfEhatZvlZy29ntpMjAXQPS6OQJ6vX8hyv O0j4fvLPgufOeXnqmK3USltq91hjeYfjnZtWfIbxuNdn4CQiYWQcIWqea3EtYJnE1bakWh dO3fRmW5KHEEIRFJyaylIU20ealDzvw= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1738574968; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XXDMn7CNaiwzg65IjzvDMWn1fcydgyKYjKgaCniNb+k=; b=pHThM6s80q1IJrVjq+VNKki1sk+L4MdY/g+oiJkl/2/bmeSm9OVi81mXJdio0H3xjcHyJK JCcFBEhg7qHi7PCg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1738574968; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XXDMn7CNaiwzg65IjzvDMWn1fcydgyKYjKgaCniNb+k=; b=MzMfL4uIU/3QqymoYS4v8PqmFnz21cfP5NEngfEhatZvlZy29ntpMjAXQPS6OQJ6vX8hyv O0j4fvLPgufOeXnqmK3USltq91hjeYfjnZtWfIbxuNdn4CQiYWQcIWqea3EtYJnE1bakWh dO3fRmW5KHEEIRFJyaylIU20ealDzvw= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1738574968; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XXDMn7CNaiwzg65IjzvDMWn1fcydgyKYjKgaCniNb+k=; b=pHThM6s80q1IJrVjq+VNKki1sk+L4MdY/g+oiJkl/2/bmeSm9OVi81mXJdio0H3xjcHyJK JCcFBEhg7qHi7PCg== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 5EF1313A7F; Mon, 3 Feb 2025 09:29:28 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id KJXgFniMoGfwYAAAD6G6ig (envelope-from ); Mon, 03 Feb 2025 09:29:28 +0000 From: Vlastimil Babka Date: Mon, 03 Feb 2025 10:28:47 +0100 Subject: [PATCH 1/4] slab, rcu: move TINY_RCU variant of kvfree_rcu() to SLAB MIME-Version: 1.0 Message-Id: <20250203-slub-tiny-kfree_rcu-v1-1-d4428bf9a8a1@suse.cz> References: <20250203-slub-tiny-kfree_rcu-v1-0-d4428bf9a8a1@suse.cz> In-Reply-To: <20250203-slub-tiny-kfree_rcu-v1-0-d4428bf9a8a1@suse.cz> To: Christoph Lameter , David Rientjes , "Paul E. McKenney" , Joel Fernandes , Josh Triplett , Boqun Feng , Uladzislau Rezki Cc: Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang , rcu@vger.kernel.org, Vlastimil Babka X-Mailer: b4 0.14.2 X-Rspamd-Action: no action X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 1C05A4000C X-Stat-Signature: nwk5qbkgqo8sz6hcxbzjqj7bmbdb3a7u X-Rspam-User: X-HE-Tag: 1738574969-790393 X-HE-Meta: U2FsdGVkX1/Ln400juHWlG2zpMYnWYYrvujBUofeTnHAwZz6sB80Y8GKxxKhm9TwOGg9rhaO+z35u+kPbo4sKYae0myxKo6zeJFOE5YDuK7B4ONjjFaSDY1UONuNQ6NXD+6Mt3LIrG5nNZ5HwmSDehBzfr+188IUlvUublnsoyLR0ZKYjo0K7R6HE2p+ylhksePtGeX/bjQiED1LD9J1yZyPJ3EQVyzBjOSvM52wbIYLGibI9KQb75YvBD7GKgK24x+RYz2LHFRBVg8BvdAOgag4eI1GXVowvcspAWi/QzU+0XnMMFs4BrAGj7ErdLSxUV57ERDwUg3bUgX/tFzYXZaULxZMtm4HDqIigDVH24FnV3pfSsca4VObvhWAUQ2YryIBGiGRVzv/fogyV9mqIWGLqZtrvtZW9xJgZahYjl68sED1qNcitfsMYzSjXaKLxS8msBHgL5bWtrMxJDJqJ+wn7xBtOLKZPF7aRrHNf4XN/xhU73EoqKDPkew5bTINPJrj0vG5xYnOZXszE7V/FW52Q9f0RKmufCm0rV47aTNCcHWqKAJ3Cqg8DPPYuZd4s6vR6i/yOSzq2XXJyU3QoFPBdEMfd4JuG7/9vVCL+B/7+1+p8+86vnX6docxLEy8Cvc+TR9JBw9NZcVCnK5VxJIuoUw5O0RRr4AkpA+9C43pxfEepMl/xoawRONzMwtnTVYQ9Z7ul6jhTpJZfdAfaXnKfIQv7DBcaBYKg3NHHjbCLewFwKRdjl3mHQp46cXcEsIW8X8GtrEkKc61yS12A7N7amMzNM+YdCe/wWd89LSO2LWaMcnt7SXZkYVP9jErVOrvwoNdes3pZCRgzioAydOP54Q4vr1nRScaAAw2FdbRU6aB5EISV0Y6m3R2SBIfi95gh+ksqx75n+xAH4ODz38fLxl9ZA0ik4wlk/1TzX4yjZ7lLt5vCViSWT74G/MmD5xH7NguNBkee3Z/K7a 1xv+uCnw +ggwpSZ8Ch4wLhaxFs6s0RQPbbNVTSKOVhEJO2p3zq+ludjnupNN3isQOgC/IVDASskv35KwQMmd+qvWm5rzRa/sa2FSPJSie+AAkb/Qs+IXEt2Uchjg6K7uUil0gu1EwTBTCt0R9/vI4b1Hoq/tE2KVE2wJg8ubTJHvC74gAvBUQ3fT79A05lE9UZbaPZxz5mc8wNv5xPwlKf/D8ZA5XDFyPA4GQb+799ef0NBS/rgsWRI/dwqXBR9+hct/w2vsa6o1PSEdVlbAUiOSwBsvDXqZYU+eyYnTD9E+dCRfii9jDo1WooV7fUUYEUAoenabws0VY+ypgI1K85E4Zh1FAEOipnUi+AgMNw3qcTTwbcbzBP4Enp2QtM55SXaGXpvChv9vd2KJkQ9nQWKYY3d5TQtxG4ilMgqRx2L3oB0GSqP95wwnA7Y7BBFToWhcBMznWJHWQIE6VDbosEew= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Following the move of TREE_RCU implementation, let's move also the TINY_RCU one for consistency and subsequent refactoring. For simplicity, remove the separate inline __kvfree_call_rcu() as TINY_RCU is not meant for high-performance hardware anyway. Declare kvfree_call_rcu() in rcupdate.h to avoid header dependency issues. Also move the kvfree_rcu_barrier() declaration to slab.h Reviewed-by: Uladzislau Rezki (Sony) Reviewed-by: Joel Fernandes (Google) Signed-off-by: Vlastimil Babka Reviewed-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- include/linux/rcupdate.h | 5 +++++ include/linux/rcutiny.h | 36 ------------------------------------ include/linux/rcutree.h | 3 --- include/linux/slab.h | 14 ++++++++++++++ kernel/rcu/tiny.c | 11 ----------- mm/slab_common.c | 19 +++++++++++++++++++ 6 files changed, 38 insertions(+), 50 deletions(-) diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index 48e5c03df1dd83c246a61d0fcc8aa638adcd7654..3f70d1c8144426f40553c8c589f07097ece8a706 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -1082,6 +1082,11 @@ static inline notrace void rcu_read_unlock_sched_notrace(void) #define kfree_rcu_mightsleep(ptr) kvfree_rcu_arg_1(ptr) #define kvfree_rcu_mightsleep(ptr) kvfree_rcu_arg_1(ptr) +/* + * In mm/slab_common.c, no suitable header to include here. + */ +void kvfree_call_rcu(struct rcu_head *head, void *ptr); + #define kvfree_rcu_arg_2(ptr, rhf) \ do { \ typeof (ptr) ___p = (ptr); \ diff --git a/include/linux/rcutiny.h b/include/linux/rcutiny.h index fe42315f667fc5be7f2ed8eae6ea0c7193030846..f519cd6802286710bdd56588b5ff3d07bcd30b92 100644 --- a/include/linux/rcutiny.h +++ b/include/linux/rcutiny.h @@ -90,41 +90,6 @@ static inline void synchronize_rcu_expedited(void) synchronize_rcu(); } -/* - * Add one more declaration of kvfree() here. It is - * not so straight forward to just include - * where it is defined due to getting many compile - * errors caused by that include. - */ -extern void kvfree(const void *addr); - -static inline void __kvfree_call_rcu(struct rcu_head *head, void *ptr) -{ - if (head) { - call_rcu(head, (rcu_callback_t) ((void *) head - ptr)); - return; - } - - // kvfree_rcu(one_arg) call. - might_sleep(); - synchronize_rcu(); - kvfree(ptr); -} - -static inline void kvfree_rcu_barrier(void) -{ - rcu_barrier(); -} - -#ifdef CONFIG_KASAN_GENERIC -void kvfree_call_rcu(struct rcu_head *head, void *ptr); -#else -static inline void kvfree_call_rcu(struct rcu_head *head, void *ptr) -{ - __kvfree_call_rcu(head, ptr); -} -#endif - void rcu_qs(void); static inline void rcu_softirq_qs(void) @@ -164,7 +129,6 @@ static inline void rcu_end_inkernel_boot(void) { } static inline bool rcu_inkernel_boot_has_ended(void) { return true; } static inline bool rcu_is_watching(void) { return true; } static inline void rcu_momentary_eqs(void) { } -static inline void kfree_rcu_scheduler_running(void) { } /* Avoid RCU read-side critical sections leaking across. */ static inline void rcu_all_qs(void) { barrier(); } diff --git a/include/linux/rcutree.h b/include/linux/rcutree.h index 27d86d9127817e50f8d4dd79e1990d70a02435bb..dbe77b5fe06ec89a393b5444d6c479ced346a37b 100644 --- a/include/linux/rcutree.h +++ b/include/linux/rcutree.h @@ -34,12 +34,9 @@ static inline void rcu_virt_note_context_switch(void) } void synchronize_rcu_expedited(void); -void kvfree_call_rcu(struct rcu_head *head, void *ptr); -void kvfree_rcu_barrier(void); void rcu_barrier(void); void rcu_momentary_eqs(void); -void kfree_rcu_scheduler_running(void); struct rcu_gp_oldstate { unsigned long rgos_norm; diff --git a/include/linux/slab.h b/include/linux/slab.h index 09eedaecf1205672bb2e7c8cd57ae8fccebc2737..bcc62e5656c35c6a3f4caf26fb33d7447dead39a 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -16,6 +16,7 @@ #include #include #include +#include #include #include #include @@ -1082,6 +1083,19 @@ extern void kvfree_sensitive(const void *addr, size_t len); unsigned int kmem_cache_size(struct kmem_cache *s); +#ifdef CONFIG_TINY_RCU +static inline void kvfree_rcu_barrier(void) +{ + rcu_barrier(); +} + +static inline void kfree_rcu_scheduler_running(void) { } +#else +void kvfree_rcu_barrier(void); + +void kfree_rcu_scheduler_running(void); +#endif + /** * kmalloc_size_roundup - Report allocation bucket size for the given size * diff --git a/kernel/rcu/tiny.c b/kernel/rcu/tiny.c index 4b3f319114650d372d490919669a2a9e248e731d..0ec27093d0e14a4b1060ea08932c4ac13f9b0f26 100644 --- a/kernel/rcu/tiny.c +++ b/kernel/rcu/tiny.c @@ -246,17 +246,6 @@ bool poll_state_synchronize_rcu(unsigned long oldstate) } EXPORT_SYMBOL_GPL(poll_state_synchronize_rcu); -#ifdef CONFIG_KASAN_GENERIC -void kvfree_call_rcu(struct rcu_head *head, void *ptr) -{ - if (head) - kasan_record_aux_stack(ptr); - - __kvfree_call_rcu(head, ptr); -} -EXPORT_SYMBOL_GPL(kvfree_call_rcu); -#endif - void __init rcu_init(void) { open_softirq(RCU_SOFTIRQ, rcu_process_callbacks); diff --git a/mm/slab_common.c b/mm/slab_common.c index 4030907b6b7d89adcf62c70e82e95c8deb41adba..81a0ce77b11c2ef8db9164cdca5853069402f161 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1284,6 +1284,25 @@ EXPORT_TRACEPOINT_SYMBOL(kmem_cache_alloc); EXPORT_TRACEPOINT_SYMBOL(kfree); EXPORT_TRACEPOINT_SYMBOL(kmem_cache_free); +#ifdef CONFIG_TINY_RCU + +void kvfree_call_rcu(struct rcu_head *head, void *ptr) +{ + if (head) { + kasan_record_aux_stack(ptr); + call_rcu(head, (rcu_callback_t) ((void *) head - ptr)); + return; + } + + // kvfree_rcu(one_arg) call. + might_sleep(); + synchronize_rcu(); + kvfree(ptr); +} +EXPORT_SYMBOL_GPL(kvfree_call_rcu); + +#endif + /* * This rcu parameter is runtime-read-only. It reflects * a minimum allowed number of objects which can be cached From patchwork Mon Feb 3 09:28:48 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13957129 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7D73DC02192 for ; Mon, 3 Feb 2025 09:29:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E54D028000F; Mon, 3 Feb 2025 04:29:32 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DE806280010; Mon, 3 Feb 2025 04:29:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C2CE828000F; Mon, 3 Feb 2025 04:29:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id A06A9280002 for ; Mon, 3 Feb 2025 04:29:32 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 4A621142688 for ; Mon, 3 Feb 2025 09:29:32 +0000 (UTC) X-FDA: 83078110584.05.61E744C Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) by imf14.hostedemail.com (Postfix) with ESMTP id 26391100003 for ; Mon, 3 Feb 2025 09:29:29 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf14.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.130 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738574970; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NLVxjPmr18uIiWai3L/1VQFpWIRfTic1Kusz0acFGBc=; b=iUO/hbKnpTN7r1QuXLKIY98L11Tlq1eJBXZSoUY2suXfzmDG7hQuPZb03SeGvBR6Ht32lE ZvON9J5sIVxcG4lB2dpXpnmxTU4eUqm2/pB3XWSR8+zuc5BaArwrIIB8vEL/UYTjJGv1kp QW1gzCzMDoJGJl+mie8wYmWs8Feifaw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738574970; a=rsa-sha256; cv=none; b=FXCWouEIOyYV7GfEOnfmWtRDvc1xxz8HfUu7fIVKv0cL5gn851WmGipOoc6zwxmxvjSLww 2CUCddAJ54N9cbjch5fsyjsbUqNyI6BPtOvhXYS784GI8iboIyudZgfzjB6hVmhDxJ2JHN a8R3gU+lejpJBitSQuCl5Nf+6/i9Qiw= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf14.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.130 as permitted sender) smtp.mailfrom=vbabka@suse.cz Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 98FFC21137; Mon, 3 Feb 2025 09:29:28 +0000 (UTC) Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 78FEC13A96; Mon, 3 Feb 2025 09:29:28 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id ALNDHXiMoGfwYAAAD6G6ig (envelope-from ); Mon, 03 Feb 2025 09:29:28 +0000 From: Vlastimil Babka Date: Mon, 03 Feb 2025 10:28:48 +0100 Subject: [PATCH 2/4] rcu: remove trace_rcu_kvfree_callback MIME-Version: 1.0 Message-Id: <20250203-slub-tiny-kfree_rcu-v1-2-d4428bf9a8a1@suse.cz> References: <20250203-slub-tiny-kfree_rcu-v1-0-d4428bf9a8a1@suse.cz> In-Reply-To: <20250203-slub-tiny-kfree_rcu-v1-0-d4428bf9a8a1@suse.cz> To: Christoph Lameter , David Rientjes , "Paul E. McKenney" , Joel Fernandes , Josh Triplett , Boqun Feng , Uladzislau Rezki Cc: Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang , rcu@vger.kernel.org, Vlastimil Babka X-Mailer: b4 0.14.2 X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Action: no action X-Stat-Signature: gisoo4uusg4h7nifbshqym98eopw5ky6 X-Rspam-User: X-Rspamd-Queue-Id: 26391100003 X-Rspamd-Server: rspam03 X-HE-Tag: 1738574969-541509 X-HE-Meta: U2FsdGVkX18jvao2yMeZpRD5JjgQrx4eSH+/8GYNjyYZiHvubudfcyY5RRvAlfy6gRZr26YsXXMgz7TH9H9TUrHgQlcetfOid33XbYlSO3rau5AMXA5LLA5pWa6ELLBDWlnQA9K37oeXl4/+7tuyz6CyP8a3SG6cofwLnQeTSiCqcBy6gox/tDMqVck19zao0gvR7MMraumUigG5e/+Ycs4qj6JQRP1PqtpUWsj6gNqb+1UADkArR7eXnSMjSDzXnRdcWKzrZSWMLwd5Ol2BTH1kr7umTUTYQQZCTTUNx9UGHtRrfXesgHWIpX0TQ9TzwAI5nDpn+VPEUrNBtT2Q9RCnUmxI6aVDZLHUqFsU+4/O7M1+Ru8k4YSAGKQTxxGHuw2HJvZfggUu7S4PPSjHaxEQMZWrMc4liNf89a6LhG6RokIJP2jukXI6inrNeLAJ9jR+ZdZDMZImStuihNnLak2jxvXTLkYEsAmIazl0/SnstoMCpNpwlio6poQChmrcZFb4vLdgBb4MoHlXiNAYKREHbQtF8icUrfmW9TWFHbm5Jbf4pnwryrP2iE9hGHVtApY/dsjDeALRkksNGV1Plbx3SG9uccjKrfhM/jUH0gahJ2xqDKQeazANYhmFfK7XIwDkSQypk/nE9NIeUePPHHIcUPXjMUBxZcc42GAmfxJciGdiONYsns0EueZbV1YqXk42x2/dqjk13xtFUJcf28m5E7TV+atOMynywHA87BySNvsTnoHSYLxQGFEF5buvqaSjJPrFJ6z5iTc+cZtWhpl/wWnyj6aMspD0rxzfE0VzmXhfNN+0clVDNL3xro8j93y4GcIQWchL+6y1yyvmmAlvgmDNC50uBAoGc1gM23jUdn3jHYy5dwqr6KiUygFMYqFLizUL7ZxJ2BFqGzAls0x6qFDFbZe3Qd/GXaoG51Yp94cbX7isOoW+Lyb3BO+7vulSIdqXQPlmly4fYud mnYlqO4D VTPQ/YVkMLVHdoYvd8v8akHksLpo69lONUtpkkeG++pyVh9QQZT3lwTIggKxCbEiazVZU8rN0ZwzTmKSD5mXmZkJz4/mT435Uce0H7pSgucO/w+/fFfa+y3u+Z5Ysdc8qZ18QjYv0DGlC/e7ULaUtkGUdBO7iTfK0O4e9C50h5j89D/OCLmIg2IbhwBXuXgtMucufmnaptektIzvanvZCEJOgcpE5viA+ni1NXX1Wt6LKm7kpVNjeWLBYsiwxTf10sBGmFB13J1KapkzivdDTo2l23ZpjMyZl8AMd X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Tree RCU does not handle kvfree_rcu() by queueing individual objects by call_rcu() anymore, thus the tracepoint and associated __is_kvfree_rcu_offset() check is dead code now. Remove it. Reviewed-by: Joel Fernandes (Google) Signed-off-by: Vlastimil Babka Reviewed-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- include/trace/events/rcu.h | 34 ---------------------------------- kernel/rcu/tree.c | 9 ++------- 2 files changed, 2 insertions(+), 41 deletions(-) diff --git a/include/trace/events/rcu.h b/include/trace/events/rcu.h index e81431deaa50b21725b08ad7c1a2992a0074065a..ac3b28b8939b78843aecb44bed8cb76c451ab060 100644 --- a/include/trace/events/rcu.h +++ b/include/trace/events/rcu.h @@ -560,40 +560,6 @@ TRACE_EVENT_RCU(rcu_segcb_stats, ); -/* - * Tracepoint for the registration of a single RCU callback of the special - * kvfree() form. The first argument is the RCU type, the second argument - * is a pointer to the RCU callback, the third argument is the offset - * of the callback within the enclosing RCU-protected data structure, - * the fourth argument is the number of lazy callbacks queued, and the - * fifth argument is the total number of callbacks queued. - */ -TRACE_EVENT_RCU(rcu_kvfree_callback, - - TP_PROTO(const char *rcuname, struct rcu_head *rhp, unsigned long offset, - long qlen), - - TP_ARGS(rcuname, rhp, offset, qlen), - - TP_STRUCT__entry( - __field(const char *, rcuname) - __field(void *, rhp) - __field(unsigned long, offset) - __field(long, qlen) - ), - - TP_fast_assign( - __entry->rcuname = rcuname; - __entry->rhp = rhp; - __entry->offset = offset; - __entry->qlen = qlen; - ), - - TP_printk("%s rhp=%p func=%ld %ld", - __entry->rcuname, __entry->rhp, __entry->offset, - __entry->qlen) -); - /* * Tracepoint for marking the beginning rcu_do_batch, performed to start * RCU callback invocation. The first argument is the RCU flavor, diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 475f31deed1418d73d097bb4d39fe714814fefa3..5dbc4189037c501b8c671598131ed5ea6d4afd71 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2931,13 +2931,8 @@ static int __init rcu_spawn_core_kthreads(void) static void rcutree_enqueue(struct rcu_data *rdp, struct rcu_head *head, rcu_callback_t func) { rcu_segcblist_enqueue(&rdp->cblist, head); - if (__is_kvfree_rcu_offset((unsigned long)func)) - trace_rcu_kvfree_callback(rcu_state.name, head, - (unsigned long)func, - rcu_segcblist_n_cbs(&rdp->cblist)); - else - trace_rcu_callback(rcu_state.name, head, - rcu_segcblist_n_cbs(&rdp->cblist)); + trace_rcu_callback(rcu_state.name, head, + rcu_segcblist_n_cbs(&rdp->cblist)); trace_rcu_segcb_stats(&rdp->cblist, TPS("SegCBQueued")); } From patchwork Mon Feb 3 09:28:49 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13957131 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B9E01C02192 for ; Mon, 3 Feb 2025 09:29:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5A57E280011; Mon, 3 Feb 2025 04:29:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 52826280002; Mon, 3 Feb 2025 04:29:34 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2DE38280011; Mon, 3 Feb 2025 04:29:34 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 041AC280002 for ; Mon, 3 Feb 2025 04:29:33 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 7BC6C1CAA81 for ; Mon, 3 Feb 2025 09:29:32 +0000 (UTC) X-FDA: 83078110584.21.FA33236 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) by imf01.hostedemail.com (Postfix) with ESMTP id 3795A40008 for ; Mon, 3 Feb 2025 09:29:29 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=0Qxyruqu; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=mpHr8Utj; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=0Qxyruqu; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=mpHr8Utj; spf=pass (imf01.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.131 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738574970; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0EAiV0EQ0UZEjL4mUXNSvL6nqq3icXpaRdqmV/qKRdM=; b=ZhqdRzqU3F8RWGrPSCpKkNlRHs4Z+jPMouuWgjwiotW/duIFc5eC4hMlnCl+vv9AlJp0Ri zhF+v6rFM8rNSXbDzxJuzgu2gsE5a83yt4ePj6K8vMpq3ftGGZeXOnhiSQGE8TykuyhHhC A+ojVHGjAQ8N2vnkkMwHjsYAcv9bBlw= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=0Qxyruqu; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=mpHr8Utj; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=0Qxyruqu; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=mpHr8Utj; spf=pass (imf01.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.131 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738574970; a=rsa-sha256; cv=none; b=RlFaN7jiwQ2E3b/nkhnGEPjC2jGWM0HPjuEURtqt9R7cK7Lopno9fNpBBQ3yuyOH9NGl5V iwFiGM4xYrNxPH4Ay5zN4gUKq1H+B6FialUnY1JZ8OC6zxcregWTfNBxUyBX5EypRTbMvi Vdq6zbawDVqvNxrzpr3dz9BQMcdWacQ= Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id AEF321F381; Mon, 3 Feb 2025 09:29:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1738574968; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0EAiV0EQ0UZEjL4mUXNSvL6nqq3icXpaRdqmV/qKRdM=; b=0Qxyruqu06l41TSkvtmP7rXzERFZbhb89hIZTGSuoVH94HG5UYmQ8ntLjtkYKDxAvW/NJ+ xD6OHEa9E9TNVLWm+f4DqxI707Kd9xN4fgpMcYdmT0/hkMqC/RYAQzW1whoIUlr5BOs+vK wNaDLpf4+uAnoUAp1pRhXx1Wq1NhcwM= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1738574968; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0EAiV0EQ0UZEjL4mUXNSvL6nqq3icXpaRdqmV/qKRdM=; b=mpHr8Utj3tzbk8Isy/GcXAvB4aFtM3pfE2N1pgSKK+FihsiFNUYJgfRgbY46QhTV4LspCb ZoxxFIqBM4d4LYBg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1738574968; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0EAiV0EQ0UZEjL4mUXNSvL6nqq3icXpaRdqmV/qKRdM=; b=0Qxyruqu06l41TSkvtmP7rXzERFZbhb89hIZTGSuoVH94HG5UYmQ8ntLjtkYKDxAvW/NJ+ xD6OHEa9E9TNVLWm+f4DqxI707Kd9xN4fgpMcYdmT0/hkMqC/RYAQzW1whoIUlr5BOs+vK wNaDLpf4+uAnoUAp1pRhXx1Wq1NhcwM= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1738574968; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0EAiV0EQ0UZEjL4mUXNSvL6nqq3icXpaRdqmV/qKRdM=; b=mpHr8Utj3tzbk8Isy/GcXAvB4aFtM3pfE2N1pgSKK+FihsiFNUYJgfRgbY46QhTV4LspCb ZoxxFIqBM4d4LYBg== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 916EA13AE2; Mon, 3 Feb 2025 09:29:28 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id UEs8I3iMoGfwYAAAD6G6ig (envelope-from ); Mon, 03 Feb 2025 09:29:28 +0000 From: Vlastimil Babka Date: Mon, 03 Feb 2025 10:28:49 +0100 Subject: [PATCH 3/4] rcu, slab: use a regular callback function for kvfree_rcu MIME-Version: 1.0 Message-Id: <20250203-slub-tiny-kfree_rcu-v1-3-d4428bf9a8a1@suse.cz> References: <20250203-slub-tiny-kfree_rcu-v1-0-d4428bf9a8a1@suse.cz> In-Reply-To: <20250203-slub-tiny-kfree_rcu-v1-0-d4428bf9a8a1@suse.cz> To: Christoph Lameter , David Rientjes , "Paul E. McKenney" , Joel Fernandes , Josh Triplett , Boqun Feng , Uladzislau Rezki Cc: Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang , rcu@vger.kernel.org, Vlastimil Babka X-Mailer: b4 0.14.2 X-Rspamd-Queue-Id: 3795A40008 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 8pnkreyky19gkgjzomu9n47qtihczefp X-HE-Tag: 1738574969-146156 X-HE-Meta: U2FsdGVkX1/K8Pi5su3uqmMkg8rN8Pjmo/GoKgghGYQFhOMtiAMjM+e7qOj9S4KjB6CgmhmgwrdlC5wTXPv9IJ/3WMo8qvKbcV62hBuBZfX6NgntchKB6hV72b9PCl6upzv3hZV1dmNvfxnTOYm774yHpZqzH+3fWo0xRBM2rQShpEwiYodMgFc44OLg+DpuQBAyI6o24DcljZD+9Qzk7PWu1ELshDXPVRZp6J7rPtkxTz15fkuoFbdHEp4ZpcH62AG0ro5LQ1GSFPTSjYG3YXaVsDzuCjWbQWpEPZWF5hFnHf9Ykuq1mKCsO+bxi2ENR1IoK2a0hFUhNq7S9xSZMk0lHtGpxyXyKSXQBAjvfOc6p/EVHCPGqKYZEGjBB/5FVHIliKdCx5v2ZKcAnOq6lnsRXBkmprpQE5ys8CUVZv6l475YnJZtcmTuHNh3KX5YZUmfO86km+7L8h4zDUtIuJKUj5+nqL7HEanUbhHLgtNpElaQTeUYeGi0Hjvd1rG1XFYWYco3/yMZ9HyKz03QZTnPRLYd5Fzz49WBR62KzeIGbfDU9z27WJQumQYbe5fPnA1g4YSM7V5PDVoN3pQJ+qCUBjRtg2X4v9+yU/BfVNZCXfho6rA8/1114FKrSTXhvE21IG5Eq/WDZFJ21tTI3iSsZ5Cv8TKkkMItrVu9pJhvbZnrNMxxD/Q45bb6dymXhThoA/x+YYBjWxx9XX6WYIljlD+fK2QywrV6aMBzAnChTGyTnaseRNpvagVVkniTKBhbuwQxOXvyb5W2TdeBvQNwIa7dcE8Xr/JWI5Ze5+6ej1/NHO7kB2A9fLPkt80+lpcyjy0gdIcaSuGxL9IY51DHBkEwegpuB1B42vAamtueqLyZQPu8eVmW6WrcnKKqW4z+kgaLrDwEbbY0ruYZCRz5oCli/JAZDz67ln5q+Xp5hby2+gIBVuFDQC+Zbb8vpEuV9Od4ZkTQtPuOvEz 0/Yyg29H Uq+PKwB0fS+vTWtSnTPnmQeOkf0LKWWzfW8MjVMpI/s+OjYziyPDkVTSzG9TsmQBqiLrMbEgog3Cmk/Axay+06UZZVQb/etMt7KE4o8sjaFJhKW3ON7XBhNfHC9dONDcLsoWWbO2pCe6EOXHtN45VW/UdL+PvpVBOxQWCZgyAOCW0mpfhxYRQXmgma76T8Wi9em5TLDq7yLfCCLAuA9P74eSRKJAUaXxktv3qZZmSj8juDvvjgwjJafmVo602qLliQ4aoT2ZNlGmfwaL3a84k9hC82H5punA8NS+NjJnjt/8UKvhgup038DymcvpiwSYLbjxOKhcq0ona8h+4EAuh0KOso+v/x1bX+mex/86gbK6Ma85KWHFo1zWtVlDvsqQG4I6gk77EGbSXd+jHjP5vbJvPUSlNpdk5zSlR73Z5NV1PJbo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: RCU has been special-casing callback function pointers that are integers lower than 4096 as offsets of rcu_head for kvfree() instead. The tree RCU implementation no longer does that as the batched kvfree_rcu() is not a simple call_rcu(). The tiny RCU still does, and the plan is also to make tree RCU use call_rcu() for SLUB_TINY configurations. Instead of teaching tree RCU again to special case the offsets, let's remove the special casing completely. Since there's no SLOB anymore, it is possible to create a callback function that can take a pointer to a middle of slab object with unknown offset and determine the object's pointer before freeing it, so implement that as kvfree_rcu_cb(). Large kmalloc and vmalloc allocations are handled simply by aligning down to page size. For that we retain the requirement that the offset is smaller than 4096. But we can remove __is_kvfree_rcu_offset() completely and instead just opencode the condition in the BUILD_BUG_ON() check. Reviewed-by: Joel Fernandes (Google) Signed-off-by: Vlastimil Babka Reviewed-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- include/linux/rcupdate.h | 28 +++++++++++++--------------- kernel/rcu/tiny.c | 14 -------------- mm/slab.h | 2 ++ mm/slab_common.c | 5 ++--- mm/slub.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 63 insertions(+), 32 deletions(-) diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index 3f70d1c8144426f40553c8c589f07097ece8a706..23bcf71ffb06ecf60d42690803ffc5adb7d1aedd 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -1025,12 +1025,6 @@ static inline notrace void rcu_read_unlock_sched_notrace(void) #define RCU_POINTER_INITIALIZER(p, v) \ .p = RCU_INITIALIZER(v) -/* - * Does the specified offset indicate that the corresponding rcu_head - * structure can be handled by kvfree_rcu()? - */ -#define __is_kvfree_rcu_offset(offset) ((offset) < 4096) - /** * kfree_rcu() - kfree an object after a grace period. * @ptr: pointer to kfree for double-argument invocations. @@ -1041,11 +1035,11 @@ static inline notrace void rcu_read_unlock_sched_notrace(void) * when they are used in a kernel module, that module must invoke the * high-latency rcu_barrier() function at module-unload time. * - * The kfree_rcu() function handles this issue. Rather than encoding a - * function address in the embedded rcu_head structure, kfree_rcu() instead - * encodes the offset of the rcu_head structure within the base structure. - * Because the functions are not allowed in the low-order 4096 bytes of - * kernel virtual memory, offsets up to 4095 bytes can be accommodated. + * The kfree_rcu() function handles this issue. In order to have a universal + * callback function handling different offsets of rcu_head, the callback needs + * to determine the starting address of the freed object, which can be a large + * kmalloc or vmalloc allocation. To allow simply aligning the pointer down to + * page boundary for those, only offsets up to 4095 bytes can be accommodated. * If the offset is larger than 4095 bytes, a compile-time error will * be generated in kvfree_rcu_arg_2(). If this error is triggered, you can * either fall back to use of call_rcu() or rearrange the structure to @@ -1087,14 +1081,18 @@ static inline notrace void rcu_read_unlock_sched_notrace(void) */ void kvfree_call_rcu(struct rcu_head *head, void *ptr); +/* + * The BUILD_BUG_ON() makes sure the rcu_head offset can be handled. See the + * comment of kfree_rcu() for details. + */ #define kvfree_rcu_arg_2(ptr, rhf) \ do { \ typeof (ptr) ___p = (ptr); \ \ - if (___p) { \ - BUILD_BUG_ON(!__is_kvfree_rcu_offset(offsetof(typeof(*(ptr)), rhf))); \ - kvfree_call_rcu(&((___p)->rhf), (void *) (___p)); \ - } \ + if (___p) { \ + BUILD_BUG_ON(offsetof(typeof(*(ptr)), rhf) >= 4096); \ + kvfree_call_rcu(&((___p)->rhf), (void *) (___p)); \ + } \ } while (0) #define kvfree_rcu_arg_1(ptr) \ diff --git a/kernel/rcu/tiny.c b/kernel/rcu/tiny.c index 0ec27093d0e14a4b1060ea08932c4ac13f9b0f26..7a34a99d4664f3fc050ec22378782fe26e8d1e95 100644 --- a/kernel/rcu/tiny.c +++ b/kernel/rcu/tiny.c @@ -85,15 +85,8 @@ void rcu_sched_clock_irq(int user) static inline bool rcu_reclaim_tiny(struct rcu_head *head) { rcu_callback_t f; - unsigned long offset = (unsigned long)head->func; rcu_lock_acquire(&rcu_callback_map); - if (__is_kvfree_rcu_offset(offset)) { - trace_rcu_invoke_kvfree_callback("", head, offset); - kvfree((void *)head - offset); - rcu_lock_release(&rcu_callback_map); - return true; - } trace_rcu_invoke_callback("", head); f = head->func; @@ -159,10 +152,6 @@ void synchronize_rcu(void) } EXPORT_SYMBOL_GPL(synchronize_rcu); -static void tiny_rcu_leak_callback(struct rcu_head *rhp) -{ -} - /* * Post an RCU callback to be invoked after the end of an RCU grace * period. But since we have but one CPU, that would be after any @@ -178,9 +167,6 @@ void call_rcu(struct rcu_head *head, rcu_callback_t func) pr_err("%s(): Double-freed CB %p->%pS()!!! ", __func__, head, head->func); mem_dump_obj(head); } - - if (!__is_kvfree_rcu_offset((unsigned long)head->func)) - WRITE_ONCE(head->func, tiny_rcu_leak_callback); return; } diff --git a/mm/slab.h b/mm/slab.h index e9fd9bf0bfa65b343a4ae0ecd5b4c2a325b04883..2f01c7317988ce036f0b22807403226a59f0f708 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -604,6 +604,8 @@ void __memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p, int objects, struct slabobj_ext *obj_exts); #endif +void kvfree_rcu_cb(struct rcu_head *head); + size_t __ksize(const void *objp); static inline size_t slab_ksize(const struct kmem_cache *s) diff --git a/mm/slab_common.c b/mm/slab_common.c index 81a0ce77b11c2ef8db9164cdca5853069402f161..6438a38aa5dc2ede0b5afa04bc3fbff5a4697d87 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1290,7 +1290,7 @@ void kvfree_call_rcu(struct rcu_head *head, void *ptr) { if (head) { kasan_record_aux_stack(ptr); - call_rcu(head, (rcu_callback_t) ((void *) head - ptr)); + call_rcu(head, kvfree_rcu_cb); return; } @@ -1551,8 +1551,7 @@ kvfree_rcu_list(struct rcu_head *head) rcu_lock_acquire(&rcu_callback_map); trace_rcu_invoke_kvfree_callback("slab", head, offset); - if (!WARN_ON_ONCE(!__is_kvfree_rcu_offset(offset))) - kvfree(ptr); + kvfree(ptr); rcu_lock_release(&rcu_callback_map); cond_resched_tasks_rcu_qs(); diff --git a/mm/slub.c b/mm/slub.c index 1f50129dcfb3cd1fc76ac9398fa7718cedb42385..e8273f28656936c05d015c53923f8fe69cd161b2 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -19,6 +19,7 @@ #include #include #include "slab.h" +#include #include #include #include @@ -4728,6 +4729,51 @@ static void free_large_kmalloc(struct folio *folio, void *object) folio_put(folio); } +/* + * Given an rcu_head embedded within an object obtained from kvmalloc at an + * offset < 4k, free the object in question. + */ +void kvfree_rcu_cb(struct rcu_head *head) +{ + void *obj = head; + struct folio *folio; + struct slab *slab; + struct kmem_cache *s; + void *slab_addr; + + if (is_vmalloc_addr(obj)) { + obj = (void *) PAGE_ALIGN_DOWN((unsigned long)obj); + vfree(obj); + return; + } + + folio = virt_to_folio(obj); + if (!folio_test_slab(folio)) { + /* + * rcu_head offset can be only less than page size so no need to + * consider folio order + */ + obj = (void *) PAGE_ALIGN_DOWN((unsigned long)obj); + free_large_kmalloc(folio, obj); + return; + } + + slab = folio_slab(folio); + s = slab->slab_cache; + slab_addr = folio_address(folio); + + if (is_kfence_address(obj)) { + obj = kfence_object_start(obj); + } else { + unsigned int idx = __obj_to_index(s, slab_addr, obj); + + obj = slab_addr + s->size * idx; + obj = fixup_red_left(s, obj); + } + + slab_free(s, slab, obj, _RET_IP_); +} + /** * kfree - free previously allocated memory * @object: pointer returned by kmalloc() or kmem_cache_alloc() From patchwork Mon Feb 3 09:28:50 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13957130 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AAD53C02193 for ; Mon, 3 Feb 2025 09:29:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 37B07280010; Mon, 3 Feb 2025 04:29:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3068D280002; Mon, 3 Feb 2025 04:29:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0CD5E280011; Mon, 3 Feb 2025 04:29:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id CC130280002 for ; Mon, 3 Feb 2025 04:29:32 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 7F544C2733 for ; Mon, 3 Feb 2025 09:29:32 +0000 (UTC) X-FDA: 83078110584.22.16A56CC Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) by imf28.hostedemail.com (Postfix) with ESMTP id 4E45EC0005 for ; Mon, 3 Feb 2025 09:29:30 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=Q8duSatx; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=75aIMP8z; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=Q8duSatx; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=75aIMP8z; spf=pass (imf28.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.131 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738574970; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Yfyz0ZAEn2QjhNmCRRI5FUDzK79CGjs9vV8RXmH1IWk=; b=ZKX9wL04olIp/LrdhHc2OsbnwihCkw7lUZmkBO4takDEiTPMukm+lTQpEiFwqIqIjUhWB1 o0wA9WDdkpQJ9QN060YR/bhM54fH9zLIlwIaPrTFeK86b9PW697NhVDthtx6gtoxLxO/ku FAGi+mW5EcCSaKM1WxGN/xdO5FGj8Mc= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=Q8duSatx; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=75aIMP8z; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=Q8duSatx; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=75aIMP8z; spf=pass (imf28.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.131 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738574970; a=rsa-sha256; cv=none; b=t9e2iHYfIvNLN2ZBcsPvtvxceIBYDV5PSo+Ck4C+9ZB3sv0BFsmZvzTf9niFROpbduiaXW RllvV//+BwYiYGpGN6+lN8cFJ7uc8c6mKR8xuTeD2cujfX3pvN6dNgyEgF2uyxRdTTsw08 RH6P95yb5XLgmTRt4Ao1RjwBKYP5eIw= Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id C63E41F390; Mon, 3 Feb 2025 09:29:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1738574968; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Yfyz0ZAEn2QjhNmCRRI5FUDzK79CGjs9vV8RXmH1IWk=; b=Q8duSatxg/eMFbaw3usssjoF/R3s1LZQB2gmoszRjhIDJ0xARMDziQM6EfuTNx/z2V2F2a yux0ImQ7l9HmxiklfR+DfbDZYWpr682JUP0AYMmK5CFR3AV7hPA6F/J1auZ885y3NPLW84 aNKHwczb5Ljd6rU0+x+XaTTKZB+R/Co= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1738574968; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Yfyz0ZAEn2QjhNmCRRI5FUDzK79CGjs9vV8RXmH1IWk=; b=75aIMP8zWhDXAHwXMvr1+OC5RtRrMyfuu4vczuQ6Hpy6WEe1NPPNkqSyyZDYVJSJ6LlFOH QWQmkFsb7Aky08Dg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1738574968; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Yfyz0ZAEn2QjhNmCRRI5FUDzK79CGjs9vV8RXmH1IWk=; b=Q8duSatxg/eMFbaw3usssjoF/R3s1LZQB2gmoszRjhIDJ0xARMDziQM6EfuTNx/z2V2F2a yux0ImQ7l9HmxiklfR+DfbDZYWpr682JUP0AYMmK5CFR3AV7hPA6F/J1auZ885y3NPLW84 aNKHwczb5Ljd6rU0+x+XaTTKZB+R/Co= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1738574968; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Yfyz0ZAEn2QjhNmCRRI5FUDzK79CGjs9vV8RXmH1IWk=; b=75aIMP8zWhDXAHwXMvr1+OC5RtRrMyfuu4vczuQ6Hpy6WEe1NPPNkqSyyZDYVJSJ6LlFOH QWQmkFsb7Aky08Dg== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id AC0B513A78; Mon, 3 Feb 2025 09:29:28 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id IOy6KXiMoGfwYAAAD6G6ig (envelope-from ); Mon, 03 Feb 2025 09:29:28 +0000 From: Vlastimil Babka Date: Mon, 03 Feb 2025 10:28:50 +0100 Subject: [PATCH 4/4] slab: don't batch kvfree_rcu() with SLUB_TINY MIME-Version: 1.0 Message-Id: <20250203-slub-tiny-kfree_rcu-v1-4-d4428bf9a8a1@suse.cz> References: <20250203-slub-tiny-kfree_rcu-v1-0-d4428bf9a8a1@suse.cz> In-Reply-To: <20250203-slub-tiny-kfree_rcu-v1-0-d4428bf9a8a1@suse.cz> To: Christoph Lameter , David Rientjes , "Paul E. McKenney" , Joel Fernandes , Josh Triplett , Boqun Feng , Uladzislau Rezki Cc: Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang , rcu@vger.kernel.org, Vlastimil Babka X-Mailer: b4 0.14.2 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 4E45EC0005 X-Stat-Signature: t373crz74tenima6z1n5f4r8g3iz5cjd X-Rspam-User: X-HE-Tag: 1738574970-528366 X-HE-Meta: U2FsdGVkX1/0hBn+SKKpgvAb3bwHHEe91o/VB43oeflyUjaa9dmRN3VxeWblT2BmzQNlMc3OwVjpWRn27LG9dZ+94MKKsklGP7O/aV9bGB0Kw2XdHJCLgWmhL7j38XONw6EqxJNRZusqOforrzyDApIyxAyhARjZOBnK7FU2rG7u1k1GSF8DNl++UxSu1sx/yIo5yVRjKx4Ogwj89rxsk8mGxlLClKoSAFbd18Tpbu3UVHuxl0HTTf6sRqy+RSqV8draAYiccI/bEdsQnYx34Ab+kchdcTMps1Xm3+2cEU/1Xc4LuK8BT4VX/p+fRrQtWlaEXQWE19bGDWvwHoTo/FUBwPueC7VbuuwtHlxf8XgNEJZoSzyrGPgMNI75cAA6ALkDt0LZ7Ag7jRhnv2yV1gmErtOZYso/z1lG/E9OoPx/bbPrHJ+22ZSuLA7NYbFJ/+VUAzH/J48GgTNqd3EF0ot3Xe+XPxsY8xegtyw2kZ8rhgYeBfT0OiBHu4z7nHp64Luti5UzcZ67vGf/78chSQnSRjxjlaCDqQllC6uf5CH0VhCDrU8zJLzS8BtppMU+WGUEHAxQDWBV5cp4xtwZ9je50447RLmbTuIyC0ol4lYbzrj9zieNCqI6zVNHSybus4qLVKaCNH7V0DctBJMI3G9JIC5/Zsx8iQJzpZuU2TfavLJX+WrJc6KlfZBAgoO1rTKK7gvWJwX9w3VZsfdNjlhOqCdDtnyYY3RgQarx5RgKHj1yNJOKio7xozziB4RxLDCckd//1Og1hUolUICwR7dqKddp4kFzvKQPXRKN1rs9/Ae8qbcLRMr1MXS5DmZp8zlOmueydsSSenawKZmcBhN0eZX6iCXAUjTOP5oQUKbveHz1p0FFtFnuBQ6R8wxqncvMQW2KNxQzJDL8qkxMo3QZ+nu3e5GxCEZpNROTnGdyW0kM8bHFHjGIh2Mzg2EqX3jPyxHyiJndC3ZCyAU A9PtXMbZ j07zmv/4vkKF+zUpDOWzULnjf/Vgn0J4jaTZdlfiV+E/d/BT4HckFViXLF8wflR4gLISg6ph2Pbt//IAYZfiVFQmwJcMRjT7gAsorcigTjCVrqJCZlXGRGTE8PB7Djc1VCV0Uuxp6zDwjKVUHtCsjYEmAn12rbT9VBfSPYApaXsH9Ui1++Cquumb63jQmgO7v/RqMp5kGWWggm5DhM9VtqWDMRlXfvAvGyiezVJFCS1Cvr86a4hu3HyC7Zln1TFBeE/g89sGYeIqFpBOYkmR1C3/Ix19py6JkiBpS5eBx7nViRaOGSiciDhErHKGFuNo3FuxC6Wp/aT1LVd5zPadrxn+WYuWu0zSgn/XzWPSjtNctAmDoB60gVCSUyKVR/KIenD//0YMAYdVUPXQB1jppdLQZkL3C5vAbHPr5KhX3JLiuBuU3UZ0SNJESRnauaXYnV1JRKkeQQ+HVHYo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: kvfree_rcu() is batched for better performance except on TINY_RCU, which is a simple implementation for small UP systems. Similarly SLUB_TINY is an option intended for small systems, whether or not used together with TINY_RCU. In case SLUB_TINY is used with !TINY_RCU, it makes arguably sense to not do the batching and limit the memory footprint. It's also suboptimal to have RCU-specific #ifdefs in slab code. With that, add CONFIG_KVFREE_RCU_BATCHED to determine whether batching kvfree_rcu() implementation is used. It is not set by a user prompt, but enabled by default and disabled in case TINY_RCU or SLUB_TINY are enabled. Use the new config for #ifdef's in slab code and extend their scope to cover all code used by the batched kvfree_rcu(). For example there's no need to perform kvfree_rcu_init() if the batching is disabled. Reviewed-by: Uladzislau Rezki (Sony) Reviewed-by: Joel Fernandes (Google) Signed-off-by: Vlastimil Babka Reviewed-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- include/linux/slab.h | 2 +- mm/Kconfig | 4 ++++ mm/slab_common.c | 15 +++++++++------ 3 files changed, 14 insertions(+), 7 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index bcc62e5656c35c6a3f4caf26fb33d7447dead39a..7686054dd494cc65def7f58748718e03eb78e481 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -1083,7 +1083,7 @@ extern void kvfree_sensitive(const void *addr, size_t len); unsigned int kmem_cache_size(struct kmem_cache *s); -#ifdef CONFIG_TINY_RCU +#ifndef CONFIG_KVFREE_RCU_BATCHED static inline void kvfree_rcu_barrier(void) { rcu_barrier(); diff --git a/mm/Kconfig b/mm/Kconfig index 1b501db064172cf54f1b1259893dfba676473c56..0b7f4bb5cb801c110aaa484ceae3063aa7204d29 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -242,6 +242,10 @@ menu "Slab allocator options" config SLUB def_bool y +config KVFREE_RCU_BATCHED + def_bool y + depends on !SLUB_TINY && !TINY_RCU + config SLUB_TINY bool "Configure for minimal memory footprint" depends on EXPERT diff --git a/mm/slab_common.c b/mm/slab_common.c index 6438a38aa5dc2ede0b5afa04bc3fbff5a4697d87..46d0a4cd33b5982fd79c307d572f231fdea9514a 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1284,7 +1284,7 @@ EXPORT_TRACEPOINT_SYMBOL(kmem_cache_alloc); EXPORT_TRACEPOINT_SYMBOL(kfree); EXPORT_TRACEPOINT_SYMBOL(kmem_cache_free); -#ifdef CONFIG_TINY_RCU +#ifndef CONFIG_KVFREE_RCU_BATCHED void kvfree_call_rcu(struct rcu_head *head, void *ptr) { @@ -1301,7 +1301,11 @@ void kvfree_call_rcu(struct rcu_head *head, void *ptr) } EXPORT_SYMBOL_GPL(kvfree_call_rcu); -#endif +void __init kvfree_rcu_init(void) +{ +} + +#else /* CONFIG_KVFREE_RCU_BATCHED */ /* * This rcu parameter is runtime-read-only. It reflects @@ -1879,8 +1883,6 @@ add_ptr_to_bulk_krc_lock(struct kfree_rcu_cpu **krcp, return true; } -#if !defined(CONFIG_TINY_RCU) - static enum hrtimer_restart schedule_page_work_fn(struct hrtimer *t) { @@ -2089,8 +2091,6 @@ void kvfree_rcu_barrier(void) } EXPORT_SYMBOL_GPL(kvfree_rcu_barrier); -#endif /* #if !defined(CONFIG_TINY_RCU) */ - static unsigned long kfree_rcu_shrink_count(struct shrinker *shrink, struct shrink_control *sc) { @@ -2180,3 +2180,6 @@ void __init kvfree_rcu_init(void) shrinker_register(kfree_rcu_shrinker); } + +#endif /* CONFIG_KVFREE_RCU_BATCHED */ +