From patchwork Thu Jan 23 10:37:21 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13948130 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EC29FC0218B for ; Thu, 23 Jan 2025 10:38:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 967276B0083; Thu, 23 Jan 2025 05:38:01 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 61E5A6B0089; Thu, 23 Jan 2025 05:38:01 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 24F696B008C; Thu, 23 Jan 2025 05:38:01 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id C1D396B0088 for ; Thu, 23 Jan 2025 05:38:00 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 6E4421A0E65 for ; Thu, 23 Jan 2025 10:38:00 +0000 (UTC) X-FDA: 83038366320.25.BAE90CA Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) by imf26.hostedemail.com (Postfix) with ESMTP id 29A05140007 for ; Thu, 23 Jan 2025 10:37:57 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf26.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.131 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1737628678; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bJU9oR0arnP8n1uRZSYVsRUHicbwL4JDggXTdjwbv4M=; b=n2N3AhfklRw+NV/UG0T7tzOzGlMCojt0l0NsMqZ1jJhPgW2XPionDMzdSB9Tj7XgCSu22W 6Kx3WmlRAjC0jxUQUZ7Q6Y/Cvt2EOVglRTKMZ+bN4kyozv6fNwmoBib7kH+8f0RQdxTFyl Vjnsjr05yT3W6kmtGtNrh15eMqaXMig= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf26.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.131 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1737628678; a=rsa-sha256; cv=none; b=dA8NyKMRgoIavnWULFMfwDkOJ7IHzvFin4N2s7DlWTWW16dUEguPby8MHKEtb0zrh1itQS wrU8N/hNMV8vuFTRdJulaEp92o61geTUmMYjJGa2ZW1Q/DNa9LmmhU40Z4isU9dvbEmVuw SrhLKjVHz7WKEsug6HTH4amTNXiJtA0= Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 8E1421F38C; Thu, 23 Jan 2025 10:37:56 +0000 (UTC) Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 66BEE136A1; Thu, 23 Jan 2025 10:37:56 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id cBjSGAQckmdrAQAAD6G6ig (envelope-from ); Thu, 23 Jan 2025 10:37:56 +0000 From: Vlastimil Babka Date: Thu, 23 Jan 2025 11:37:21 +0100 Subject: [PATCH RFC 4/4] slab: don't batch kvfree_rcu() with SLUB_TINY MIME-Version: 1.0 Message-Id: <20250123-slub-tiny-kfree_rcu-v1-4-0e386ef1541a@suse.cz> References: <20250123-slub-tiny-kfree_rcu-v1-0-0e386ef1541a@suse.cz> In-Reply-To: <20250123-slub-tiny-kfree_rcu-v1-0-0e386ef1541a@suse.cz> To: Christoph Lameter , David Rientjes , "Paul E. McKenney" , Joel Fernandes , Josh Triplett , Boqun Feng , Uladzislau Rezki Cc: Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang , rcu@vger.kernel.org, Vlastimil Babka X-Mailer: b4 0.14.2 X-Developer-Signature: v=1; a=openpgp-sha256; l=3787; i=vbabka@suse.cz; h=from:subject:message-id; bh=2rhWfguSgm3rCjGx5bIjjhtFSjU4v/3Sg/+BOOfL7Eo=; b=owEBbQGS/pANAwAIAbvgsHXSRYiaAcsmYgBnkhwBZZrvN3xrjWEE5R0q5c5uyHyaO6141n+Pt VpRWGoQd/qJATMEAAEIAB0WIQR7u8hBFZkjSJZITfG74LB10kWImgUCZ5IcAQAKCRC74LB10kWI mimFB/9iIoogv+10l0+P5335y9cW2V1fDBnT4BAFsftyWIlYMy6+BAK2pDqBePJdg5UOgEmSC3u dpsuw6cicMFvMLdmuNNgIB7jD0rX2vjQI9Q6TFpwDJW8MFgr85+c3/jesi40ihe8suwsv0uedHQ GOmLBikFf/xFxleq5oXTj6GhmdV82aobIrz+gMXc5aDP7+HjmIzjDJCUtJWb4MVVLrCNB8lfrNr 3bU8tPhGbyLjYuQv2jzKjdnM26rGPFlLPoFinwjayQ1IJSVA+wePpQGlnkrNZSOEf1lQ/JzYM1d /x27eZhMmwYu4rbKBHVaU+QfAz+KUiKGsRok7XaPXclI1R4O X-Developer-Key: i=vbabka@suse.cz; a=openpgp; fpr=A940D434992C2E8E99103D50224FA7E7CC82A664 X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Action: no action X-Rspamd-Queue-Id: 29A05140007 X-Stat-Signature: fxpm78mzxdypribnkwordryoixo6s76d X-Rspamd-Server: rspam08 X-Rspam-User: X-HE-Tag: 1737628677-182418 X-HE-Meta: U2FsdGVkX18dMrMYevTaKH2JHhJDIu3WRfCigrnHXAzAmVV3ETRDev14GTn9vqbdxpGClZOdDFpavAdhWD337LDwR+xYyTEdkcSsn9LtsTUa5KBzVT8LcLCYWLnxfppi8k8LkUXIUeQsaCy1B2wR0FGYehFYwf0brG2ptrLuXKV6yrRBDYj79jAT0n198UqCJSwJwCXr++lpCP7kQL5KMQAbMpzrfvyEvQlHhaVlIgTOdSWS4KqsR0ewpOkmkqHTdo0o/iew9NMKTHMDXBhIma/SRCnQ/3AobfKjLPBn79LJIrDacnxVwdtgWO+0s2yC+dMNjOEf4NHFVnqkwsT+L4XxvzeVrQDTiuvTU99rfHe+D/Riw+ALdHS7/OP3PSmDb4MjwaG3PaHVttxonZsdO2rIp3xdEvYUfBLycANQvpz7akf+J+MAFO2Y8CQ2EIAtIPWDoLVopERbAS1LjTqpzHJq1B770Yim/91CE/MxmTLpkjV+pfHOKOAgLVgVYN2BZgaaQjwyh6nljHIgJObjO1G2veWAeNOoU0BD9ltMJGjpAjNkI51Ef3fwNnOTsFgi2ph5EXEC15PIJfnRtzN5ect6YFhaSoVy5sjk+b69G4ldF2yOA0t7bJhwI8oTu6Ndw7wUt2sL3bztEQ5ojHvki6epLEzk2LivSkV1f/Cf2s7G/uR2cNUlz9+Z5ows+ffWN1NJNIHpym9w/xaUqvA/Egke6TLXZ7wMUL8llGylQD+wGLSvALgzALeH8iLfxP7JjycYEsHciq/wetJT2yGjmibifCR2Dn6cCcBqBod3xBuA7JvLK1Z7oNEfI51Ox8rLNE5kbBihWD6oml3WVxsU2QAiigjS5Bnn5qMKysfI2ps32X10H2fSYuZIm+l5AtKsjp0mXnVhhExpC/Vgj1kzI5nvBhEkjEGHoNUxKzL4FKILNNFdOqcutjORAaPkGsd0YRXXeYP2e9thM/nfIYm W01fn2gV zcQH8HTrovLpmXSYYFHzIXONQZP0hrkNxVl6aw/sli1lDOE8hB1w4Zw2MVHEiDSE+iFdjKCtZ0Th3fM/fKs/GaQV7gl6lBfuPk31xKA5O1pt5zS72nwibPbI8dprkFIE8U8hA6D4HIjX4dyFACqFuHsxeLLPX53iKkNbjg6ik6nQPDKytxcmCBOYjepZkTxHesiS0R0RwFtyDbco5eqONZK9wvr0O8ZCugMKssERGT9Glud8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: kvfree_rcu() is batched for better performance except on TINY_RCU, which is a simple implementation for small UP systems. Similarly SLUB_TINY is an option intended for small systems, whether or not used together with TINY_RCU. In case SLUB_TINY is used with !TINY_RCU, it makes arguably sense to not do the batching and limit the memory footprint. It's also suboptimal to have RCU-specific #ifdefs in slab code. With that, add CONFIG_KFREE_RCU_BATCHED to determine whether batching kvfree_rcu() implementation is used. It is not set by a user prompt, but enabled by default and disabled in case TINY_RCU or SLUB_TINY are enabled. Use the new config for #ifdef's in slab code and extend their scope to cover all code used by the batched kvfree_rcu(). For example there's no need to perform kvfree_rcu_init() if the batching is disabled. Signed-off-by: Vlastimil Babka --- include/linux/slab.h | 2 +- mm/Kconfig | 4 ++++ mm/slab_common.c | 45 +++++++++++++++++++++++++-------------------- 3 files changed, 30 insertions(+), 21 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index bcc62e5656c35c6a3f4caf26fb33d7447dead39a..9faf33734a8eee2425b90e679c0457ab459422a3 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -1083,7 +1083,7 @@ extern void kvfree_sensitive(const void *addr, size_t len); unsigned int kmem_cache_size(struct kmem_cache *s); -#ifdef CONFIG_TINY_RCU +#ifndef CONFIG_KFREE_RCU_BATCHED static inline void kvfree_rcu_barrier(void) { rcu_barrier(); diff --git a/mm/Kconfig b/mm/Kconfig index 84000b01680869801a10f56f06d0c43d6521a8d2..e513308a4aed640ee556ecb5793c7f3f195bbcae 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -242,6 +242,10 @@ menu "Slab allocator options" config SLUB def_bool y +config KFREE_RCU_BATCHED + def_bool y + depends on !SLUB_TINY && !TINY_RCU + config SLUB_TINY bool "Configure for minimal memory footprint" depends on EXPERT diff --git a/mm/slab_common.c b/mm/slab_common.c index f13d2c901daf1419993620459fbd5845eecb85f1..9f6d66313afc6684bdc0f32908fe01c83c60f283 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1284,6 +1284,28 @@ EXPORT_TRACEPOINT_SYMBOL(kmem_cache_alloc); EXPORT_TRACEPOINT_SYMBOL(kfree); EXPORT_TRACEPOINT_SYMBOL(kmem_cache_free); +#ifndef CONFIG_KFREE_RCU_BATCHED + +void kvfree_call_rcu(struct rcu_head *head, void *ptr) +{ + if (head) { + kasan_record_aux_stack_noalloc(ptr); + call_rcu(head, kvfree_rcu_cb); + return; + } + + // kvfree_rcu(one_arg) call. + might_sleep(); + synchronize_rcu(); + kvfree(ptr); +} + +void __init kvfree_rcu_init(void) +{ +} + +#else /* CONFIG_KFREE_RCU_BATCHED */ + /* * This rcu parameter is runtime-read-only. It reflects * a minimum allowed number of objects which can be cached @@ -1858,24 +1880,6 @@ add_ptr_to_bulk_krc_lock(struct kfree_rcu_cpu **krcp, return true; } -#ifdef CONFIG_TINY_RCU - -void kvfree_call_rcu(struct rcu_head *head, void *ptr) -{ - if (head) { - kasan_record_aux_stack_noalloc(ptr); - call_rcu(head, kvfree_rcu_cb); - return; - } - - // kvfree_rcu(one_arg) call. - might_sleep(); - synchronize_rcu(); - kvfree(ptr); -} - -#else /* !CONFIG_TINY_RCU */ - static enum hrtimer_restart schedule_page_work_fn(struct hrtimer *t) { @@ -2084,8 +2088,6 @@ void kvfree_rcu_barrier(void) } EXPORT_SYMBOL_GPL(kvfree_rcu_barrier); -#endif /* !CONFIG_TINY_RCU */ - static unsigned long kfree_rcu_shrink_count(struct shrinker *shrink, struct shrink_control *sc) { @@ -2175,3 +2177,6 @@ void __init kvfree_rcu_init(void) shrinker_register(kfree_rcu_shrinker); } + +#endif /* CONFIG_KFREE_RCU_BATCHED */ +