From patchwork Mon Feb 3 09:28:47 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13957133 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D792EC02192 for ; Mon, 3 Feb 2025 09:30:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6D7E7280012; Mon, 3 Feb 2025 04:30:39 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 66133280002; Mon, 3 Feb 2025 04:30:39 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4B344280012; Mon, 3 Feb 2025 04:30:39 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 294E6280002 for ; Mon, 3 Feb 2025 04:30:39 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 66C76B4F8B for ; Mon, 3 Feb 2025 09:29:32 +0000 (UTC) X-FDA: 83078110626.28.BFDEA68 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) by imf12.hostedemail.com (Postfix) with ESMTP id 1C05A4000C for ; Mon, 3 Feb 2025 09:29:29 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=MzMfL4uI; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=pHThM6s8; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=MzMfL4uI; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=pHThM6s8; spf=pass (imf12.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.131 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738574970; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XXDMn7CNaiwzg65IjzvDMWn1fcydgyKYjKgaCniNb+k=; b=gIrN5iPVOMcCCTexHJ2Yps3CrUF7FEZWGnYJKCeR7oC8HVNA+bXI4c2F7O1vYyUF/gAwAt D+anm6tnePbLLN0MD3C09EqQi3VN0Y0xjCOiX/ARmh2E3+aKwYIAc3hR1kRrf3g1Hr/CI3 rxK/iQEZBNT8IbQxypYwYzuQhrYoLfs= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=MzMfL4uI; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=pHThM6s8; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=MzMfL4uI; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=pHThM6s8; spf=pass (imf12.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.131 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738574970; a=rsa-sha256; cv=none; b=0tig3o1625aqoJpd4jEufne9XmTKHLx/Wpn0E7PIWGkNjBjjP2AZKNEw0OK5L3zRos/l4Y +JOdqUDGT4Q48257RQlCWl4V4XFsY8U8mb9Fbpg7iPsC6NniXyDleMUSfh4VW3opAO587c IaWHAR1F6FMNJjT/dsV014lG1C9c9VE= Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 7F68A1F37C; Mon, 3 Feb 2025 09:29:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1738574968; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XXDMn7CNaiwzg65IjzvDMWn1fcydgyKYjKgaCniNb+k=; b=MzMfL4uIU/3QqymoYS4v8PqmFnz21cfP5NEngfEhatZvlZy29ntpMjAXQPS6OQJ6vX8hyv O0j4fvLPgufOeXnqmK3USltq91hjeYfjnZtWfIbxuNdn4CQiYWQcIWqea3EtYJnE1bakWh dO3fRmW5KHEEIRFJyaylIU20ealDzvw= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1738574968; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XXDMn7CNaiwzg65IjzvDMWn1fcydgyKYjKgaCniNb+k=; b=pHThM6s80q1IJrVjq+VNKki1sk+L4MdY/g+oiJkl/2/bmeSm9OVi81mXJdio0H3xjcHyJK JCcFBEhg7qHi7PCg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1738574968; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XXDMn7CNaiwzg65IjzvDMWn1fcydgyKYjKgaCniNb+k=; b=MzMfL4uIU/3QqymoYS4v8PqmFnz21cfP5NEngfEhatZvlZy29ntpMjAXQPS6OQJ6vX8hyv O0j4fvLPgufOeXnqmK3USltq91hjeYfjnZtWfIbxuNdn4CQiYWQcIWqea3EtYJnE1bakWh dO3fRmW5KHEEIRFJyaylIU20ealDzvw= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1738574968; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XXDMn7CNaiwzg65IjzvDMWn1fcydgyKYjKgaCniNb+k=; b=pHThM6s80q1IJrVjq+VNKki1sk+L4MdY/g+oiJkl/2/bmeSm9OVi81mXJdio0H3xjcHyJK JCcFBEhg7qHi7PCg== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 5EF1313A7F; Mon, 3 Feb 2025 09:29:28 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id KJXgFniMoGfwYAAAD6G6ig (envelope-from ); Mon, 03 Feb 2025 09:29:28 +0000 From: Vlastimil Babka Date: Mon, 03 Feb 2025 10:28:47 +0100 Subject: [PATCH 1/4] slab, rcu: move TINY_RCU variant of kvfree_rcu() to SLAB MIME-Version: 1.0 Message-Id: <20250203-slub-tiny-kfree_rcu-v1-1-d4428bf9a8a1@suse.cz> References: <20250203-slub-tiny-kfree_rcu-v1-0-d4428bf9a8a1@suse.cz> In-Reply-To: <20250203-slub-tiny-kfree_rcu-v1-0-d4428bf9a8a1@suse.cz> To: Christoph Lameter , David Rientjes , "Paul E. McKenney" , Joel Fernandes , Josh Triplett , Boqun Feng , Uladzislau Rezki Cc: Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang , rcu@vger.kernel.org, Vlastimil Babka X-Mailer: b4 0.14.2 X-Rspamd-Action: no action X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 1C05A4000C X-Stat-Signature: nwk5qbkgqo8sz6hcxbzjqj7bmbdb3a7u X-Rspam-User: X-HE-Tag: 1738574969-790393 X-HE-Meta: U2FsdGVkX1/Ln400juHWlG2zpMYnWYYrvujBUofeTnHAwZz6sB80Y8GKxxKhm9TwOGg9rhaO+z35u+kPbo4sKYae0myxKo6zeJFOE5YDuK7B4ONjjFaSDY1UONuNQ6NXD+6Mt3LIrG5nNZ5HwmSDehBzfr+188IUlvUublnsoyLR0ZKYjo0K7R6HE2p+ylhksePtGeX/bjQiED1LD9J1yZyPJ3EQVyzBjOSvM52wbIYLGibI9KQb75YvBD7GKgK24x+RYz2LHFRBVg8BvdAOgag4eI1GXVowvcspAWi/QzU+0XnMMFs4BrAGj7ErdLSxUV57ERDwUg3bUgX/tFzYXZaULxZMtm4HDqIigDVH24FnV3pfSsca4VObvhWAUQ2YryIBGiGRVzv/fogyV9mqIWGLqZtrvtZW9xJgZahYjl68sED1qNcitfsMYzSjXaKLxS8msBHgL5bWtrMxJDJqJ+wn7xBtOLKZPF7aRrHNf4XN/xhU73EoqKDPkew5bTINPJrj0vG5xYnOZXszE7V/FW52Q9f0RKmufCm0rV47aTNCcHWqKAJ3Cqg8DPPYuZd4s6vR6i/yOSzq2XXJyU3QoFPBdEMfd4JuG7/9vVCL+B/7+1+p8+86vnX6docxLEy8Cvc+TR9JBw9NZcVCnK5VxJIuoUw5O0RRr4AkpA+9C43pxfEepMl/xoawRONzMwtnTVYQ9Z7ul6jhTpJZfdAfaXnKfIQv7DBcaBYKg3NHHjbCLewFwKRdjl3mHQp46cXcEsIW8X8GtrEkKc61yS12A7N7amMzNM+YdCe/wWd89LSO2LWaMcnt7SXZkYVP9jErVOrvwoNdes3pZCRgzioAydOP54Q4vr1nRScaAAw2FdbRU6aB5EISV0Y6m3R2SBIfi95gh+ksqx75n+xAH4ODz38fLxl9ZA0ik4wlk/1TzX4yjZ7lLt5vCViSWT74G/MmD5xH7NguNBkee3Z/K7a 1xv+uCnw +ggwpSZ8Ch4wLhaxFs6s0RQPbbNVTSKOVhEJO2p3zq+ludjnupNN3isQOgC/IVDASskv35KwQMmd+qvWm5rzRa/sa2FSPJSie+AAkb/Qs+IXEt2Uchjg6K7uUil0gu1EwTBTCt0R9/vI4b1Hoq/tE2KVE2wJg8ubTJHvC74gAvBUQ3fT79A05lE9UZbaPZxz5mc8wNv5xPwlKf/D8ZA5XDFyPA4GQb+799ef0NBS/rgsWRI/dwqXBR9+hct/w2vsa6o1PSEdVlbAUiOSwBsvDXqZYU+eyYnTD9E+dCRfii9jDo1WooV7fUUYEUAoenabws0VY+ypgI1K85E4Zh1FAEOipnUi+AgMNw3qcTTwbcbzBP4Enp2QtM55SXaGXpvChv9vd2KJkQ9nQWKYY3d5TQtxG4ilMgqRx2L3oB0GSqP95wwnA7Y7BBFToWhcBMznWJHWQIE6VDbosEew= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Following the move of TREE_RCU implementation, let's move also the TINY_RCU one for consistency and subsequent refactoring. For simplicity, remove the separate inline __kvfree_call_rcu() as TINY_RCU is not meant for high-performance hardware anyway. Declare kvfree_call_rcu() in rcupdate.h to avoid header dependency issues. Also move the kvfree_rcu_barrier() declaration to slab.h Reviewed-by: Uladzislau Rezki (Sony) Reviewed-by: Joel Fernandes (Google) Signed-off-by: Vlastimil Babka Reviewed-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- include/linux/rcupdate.h | 5 +++++ include/linux/rcutiny.h | 36 ------------------------------------ include/linux/rcutree.h | 3 --- include/linux/slab.h | 14 ++++++++++++++ kernel/rcu/tiny.c | 11 ----------- mm/slab_common.c | 19 +++++++++++++++++++ 6 files changed, 38 insertions(+), 50 deletions(-) diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index 48e5c03df1dd83c246a61d0fcc8aa638adcd7654..3f70d1c8144426f40553c8c589f07097ece8a706 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -1082,6 +1082,11 @@ static inline notrace void rcu_read_unlock_sched_notrace(void) #define kfree_rcu_mightsleep(ptr) kvfree_rcu_arg_1(ptr) #define kvfree_rcu_mightsleep(ptr) kvfree_rcu_arg_1(ptr) +/* + * In mm/slab_common.c, no suitable header to include here. + */ +void kvfree_call_rcu(struct rcu_head *head, void *ptr); + #define kvfree_rcu_arg_2(ptr, rhf) \ do { \ typeof (ptr) ___p = (ptr); \ diff --git a/include/linux/rcutiny.h b/include/linux/rcutiny.h index fe42315f667fc5be7f2ed8eae6ea0c7193030846..f519cd6802286710bdd56588b5ff3d07bcd30b92 100644 --- a/include/linux/rcutiny.h +++ b/include/linux/rcutiny.h @@ -90,41 +90,6 @@ static inline void synchronize_rcu_expedited(void) synchronize_rcu(); } -/* - * Add one more declaration of kvfree() here. It is - * not so straight forward to just include - * where it is defined due to getting many compile - * errors caused by that include. - */ -extern void kvfree(const void *addr); - -static inline void __kvfree_call_rcu(struct rcu_head *head, void *ptr) -{ - if (head) { - call_rcu(head, (rcu_callback_t) ((void *) head - ptr)); - return; - } - - // kvfree_rcu(one_arg) call. - might_sleep(); - synchronize_rcu(); - kvfree(ptr); -} - -static inline void kvfree_rcu_barrier(void) -{ - rcu_barrier(); -} - -#ifdef CONFIG_KASAN_GENERIC -void kvfree_call_rcu(struct rcu_head *head, void *ptr); -#else -static inline void kvfree_call_rcu(struct rcu_head *head, void *ptr) -{ - __kvfree_call_rcu(head, ptr); -} -#endif - void rcu_qs(void); static inline void rcu_softirq_qs(void) @@ -164,7 +129,6 @@ static inline void rcu_end_inkernel_boot(void) { } static inline bool rcu_inkernel_boot_has_ended(void) { return true; } static inline bool rcu_is_watching(void) { return true; } static inline void rcu_momentary_eqs(void) { } -static inline void kfree_rcu_scheduler_running(void) { } /* Avoid RCU read-side critical sections leaking across. */ static inline void rcu_all_qs(void) { barrier(); } diff --git a/include/linux/rcutree.h b/include/linux/rcutree.h index 27d86d9127817e50f8d4dd79e1990d70a02435bb..dbe77b5fe06ec89a393b5444d6c479ced346a37b 100644 --- a/include/linux/rcutree.h +++ b/include/linux/rcutree.h @@ -34,12 +34,9 @@ static inline void rcu_virt_note_context_switch(void) } void synchronize_rcu_expedited(void); -void kvfree_call_rcu(struct rcu_head *head, void *ptr); -void kvfree_rcu_barrier(void); void rcu_barrier(void); void rcu_momentary_eqs(void); -void kfree_rcu_scheduler_running(void); struct rcu_gp_oldstate { unsigned long rgos_norm; diff --git a/include/linux/slab.h b/include/linux/slab.h index 09eedaecf1205672bb2e7c8cd57ae8fccebc2737..bcc62e5656c35c6a3f4caf26fb33d7447dead39a 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -16,6 +16,7 @@ #include #include #include +#include #include #include #include @@ -1082,6 +1083,19 @@ extern void kvfree_sensitive(const void *addr, size_t len); unsigned int kmem_cache_size(struct kmem_cache *s); +#ifdef CONFIG_TINY_RCU +static inline void kvfree_rcu_barrier(void) +{ + rcu_barrier(); +} + +static inline void kfree_rcu_scheduler_running(void) { } +#else +void kvfree_rcu_barrier(void); + +void kfree_rcu_scheduler_running(void); +#endif + /** * kmalloc_size_roundup - Report allocation bucket size for the given size * diff --git a/kernel/rcu/tiny.c b/kernel/rcu/tiny.c index 4b3f319114650d372d490919669a2a9e248e731d..0ec27093d0e14a4b1060ea08932c4ac13f9b0f26 100644 --- a/kernel/rcu/tiny.c +++ b/kernel/rcu/tiny.c @@ -246,17 +246,6 @@ bool poll_state_synchronize_rcu(unsigned long oldstate) } EXPORT_SYMBOL_GPL(poll_state_synchronize_rcu); -#ifdef CONFIG_KASAN_GENERIC -void kvfree_call_rcu(struct rcu_head *head, void *ptr) -{ - if (head) - kasan_record_aux_stack(ptr); - - __kvfree_call_rcu(head, ptr); -} -EXPORT_SYMBOL_GPL(kvfree_call_rcu); -#endif - void __init rcu_init(void) { open_softirq(RCU_SOFTIRQ, rcu_process_callbacks); diff --git a/mm/slab_common.c b/mm/slab_common.c index 4030907b6b7d89adcf62c70e82e95c8deb41adba..81a0ce77b11c2ef8db9164cdca5853069402f161 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1284,6 +1284,25 @@ EXPORT_TRACEPOINT_SYMBOL(kmem_cache_alloc); EXPORT_TRACEPOINT_SYMBOL(kfree); EXPORT_TRACEPOINT_SYMBOL(kmem_cache_free); +#ifdef CONFIG_TINY_RCU + +void kvfree_call_rcu(struct rcu_head *head, void *ptr) +{ + if (head) { + kasan_record_aux_stack(ptr); + call_rcu(head, (rcu_callback_t) ((void *) head - ptr)); + return; + } + + // kvfree_rcu(one_arg) call. + might_sleep(); + synchronize_rcu(); + kvfree(ptr); +} +EXPORT_SYMBOL_GPL(kvfree_call_rcu); + +#endif + /* * This rcu parameter is runtime-read-only. It reflects * a minimum allowed number of objects which can be cached