From patchwork Mon Nov 20 18:34:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13461885 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21B7CC197A0 for ; Mon, 20 Nov 2023 18:35:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5C5436B03F7; Mon, 20 Nov 2023 13:34:48 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 26F9D6B03AE; Mon, 20 Nov 2023 13:34:48 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CA3186B03C4; Mon, 20 Nov 2023 13:34:47 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id B34696B03EC for ; Mon, 20 Nov 2023 13:34:47 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 954E91CB2EE for ; Mon, 20 Nov 2023 18:34:47 +0000 (UTC) X-FDA: 81479183814.09.9127D94 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by imf22.hostedemail.com (Postfix) with ESMTP id 6974CC0011 for ; Mon, 20 Nov 2023 18:34:45 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=qNkpX3Or; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=1F6OXLLw; dmarc=none; spf=pass (imf22.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1700505285; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=RPV7eFvOYqND65nC1EpsfH0qSohk6qGYOFkaS1IRDew=; b=HKI7iAwX83yxDO7p3A0YhXlmmOtShmaStcnyY4SOlM8Svwr6/ugIcImvwu9n2Rf9phuuAG hmwwqZZak8NQmdmG892Q0q5XurRZf7F96lwPwzaOjQusieXby8v3qbl8xMKtE9HG5PhTVC y+ZzVQdcfZ2cs8TD3ILyEK+Hu35ZGy4= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=qNkpX3Or; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=1F6OXLLw; dmarc=none; spf=pass (imf22.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1700505285; a=rsa-sha256; cv=none; b=IDdE2AYxfEadfaXrJdVPjsNLPvKw+LA/Uc+iR2Rw09fONNdnOdWWhBqJeciQ2G+15qGlkd xmxXNJZt6SANDRfZ82Y/pVuvdkVRFI1xE0IHP1WkbDhIqvz/7dvqXp/vzSkORQBX7cFdmD x3pGKxNC+4zkHbBNmNdVXCjiM+dQOLo= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 8E3EE21979; Mon, 20 Nov 2023 18:34:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1700505283; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RPV7eFvOYqND65nC1EpsfH0qSohk6qGYOFkaS1IRDew=; b=qNkpX3OrehFKN3IMjZB0hdxh2cOBxVTKyn/ennhzNMeRW+22upArxzVVUw9FireBsI2B+K 6o0AyP5tvtv0kA7cWeKrsQR8fW0MfWq4zne7jcB1kTLUt/2BKQeAVnEuNTYh2hYjrMch5Q 6ZhpqEnFb48mxwnLfpoRKgiL2/Kq5Ss= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1700505283; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RPV7eFvOYqND65nC1EpsfH0qSohk6qGYOFkaS1IRDew=; b=1F6OXLLwIavRartk+gk2uoy+BRwBlFG/rLZwTQtoS1o3P+Y7tzSJD4dvBrjFtpADghlRlM YIShfD2DKmbSu8AQ== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 3AD9213912; Mon, 20 Nov 2023 18:34:43 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id QFS4DcOmW2UUMgAAMHmgww (envelope-from ); Mon, 20 Nov 2023 18:34:43 +0000 From: Vlastimil Babka Date: Mon, 20 Nov 2023 19:34:28 +0100 Subject: [PATCH v2 17/21] mm/slab: move kmalloc_slab() to mm/slab.h MIME-Version: 1.0 Message-Id: <20231120-slab-remove-slab-v2-17-9c9c70177183@suse.cz> References: <20231120-slab-remove-slab-v2-0-9c9c70177183@suse.cz> In-Reply-To: <20231120-slab-remove-slab-v2-0-9c9c70177183@suse.cz> To: David Rientjes , Christoph Lameter , Pekka Enberg , Joonsoo Kim Cc: Andrew Morton , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Roman Gushchin , Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Marco Elver , Johannes Weiner , Michal Hocko , Shakeel Butt , Muchun Song , Kees Cook , linux-mm@kvack.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org, linux-hardening@vger.kernel.org, Vlastimil Babka X-Mailer: b4 0.12.4 X-Rspamd-Queue-Id: 6974CC0011 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: ym3jxh15u66bhrqeirpfine5r7e9c81k X-HE-Tag: 1700505285-749683 X-HE-Meta: U2FsdGVkX1+uiM/6JaA+U0zcvzpMopt0gucvDyN+1SUPRX1s2Wyo8Q4ouLF9cw2EpSHsny/zwkOg/RobYxRXEulo+TbtZPtzJEHsYOIIR1qBtU/2uVCAnzHY5w3skjKWSwBvQOR6Mblu14A1BoVX6YgEdBgu2oQ6jAqKQREJp/3yEPoXzkJI+X3O5NBZ+ByvJwxmJgbUlTUB6nVQ52idYjs55FdJJnohwwQ4p6ZzZaxiwq0U8UNN9iMEFiVkLNTSFyx041WHfWz1SN+pUbapnthfyrpVWXBIkoz3/QnZa5YN8hwGLEw7FKJb6HczMXjv83w3t4gyyNdIvCQB0tkNGJG8HsMbMtEp37iXA31564E3TSDII1cDH5Sp8L+U/c7m2kglbZqj9Lc1tFkVi70LeXR9Q11N/l6bCv6tBIXHC5qPxGtsd/KeAQEz3vgQvNNDclhBEu3RuuWzMXJ6NiHHNCbAm96AevsJEFXnkWrK0FasBy1MYGZgrLFqug50o1uejS2VhP9nR09LJaltWF47I5ClKDFMVKzrAYjmFWnIIyBPKxQRGPK1+KzmBVVymo0s0M1f+0i/hk+UY5H2mRm4D0DufAb3dEa6ueFvyNDDzgllyhJB02sJtNnYFj4s1Mlg3uD24gr+wR+iIBtauXq4o1BPQRRR9dEBOiQpKo/PMnVs2vruNZ59zVKqOqW3ew2fJnIQArqgDmdQ785vm6Yc3RjBikOvRtZYyXrbWQRrKWV7bQdwkSGDX2ZzDDMGJOaVIW36SjyH45/YYqTRH+y5BZpIPGinQT5ZcLcjpVlvaOFd9G5mtBFksCdHSd6U5ft6YKX/lXTZDSDiM8UKP9e5XHEIzINOlNX8ezpp8E0ryqFk4Y0KY38JW0qndpEYYiLRX57txvtdzcQBWyJeC1/nbf/W92INobkIbMAF1nw4zU0KlpKe61gL6cUAsX1gSAOFlmzttvhds8oOqG4HjHk mfhvqHfg OXW/BiMtJOzr5j3zbFw2Afw2kyNZcDEW7CueCrzKk6BYS7qWHTWv/oxbg4rgTQVbhjQbiIbBG5f0k9oAhlPIFybXjp40TACX1Q6hIOHioCd3hNHuH/p25f9oCjYtXR5s7IiNRGX3LZjWzRWJ7C3vn1LEmwq4YyGPXvRHkRHrpInL4JP74GXlaQ3LPrbcuQnZKMSU85AQU6qWkqhac8VedTHsjW+SFBpYAZtDSY9UAF2f2fZdiWtwLvFva1B19yPkizMtLqSgtc5s/AnC6DQGMYxce3SJ3poFNm9lVASUjE86hufkDiUozkLBGtnyXHC2dhKGYr5HJvu8UHzFTQrE+AFV6x8RQSqJgZVraghOspIKbXDMZWbnm4BUddQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In preparation for the next patch, move the kmalloc_slab() function to the header, as it will have callers from two files, and make it inline. To avoid unnecessary bloat, remove all size checks/warnings from kmalloc_slab() as they just duplicate those in callers, especially after recent changes to kmalloc_size_roundup(). We just need to adjust handling of zero size in __do_kmalloc_node(). Also we can stop handling NULL result from kmalloc_slab() there as that now cannot happen (unless called too early during boot). The size_index array becomes visible so rename it to a more specific kmalloc_size_index. Reviewed-by: Kees Cook Signed-off-by: Vlastimil Babka Reviewed-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- mm/slab.h | 28 ++++++++++++++++++++++++++-- mm/slab_common.c | 43 ++++++++----------------------------------- 2 files changed, 34 insertions(+), 37 deletions(-) diff --git a/mm/slab.h b/mm/slab.h index 35a55c4a407d..7d7cc7af614e 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -389,8 +389,32 @@ extern const struct kmalloc_info_struct { void setup_kmalloc_cache_index_table(void); void create_kmalloc_caches(slab_flags_t); -/* Find the kmalloc slab corresponding for a certain size */ -struct kmem_cache *kmalloc_slab(size_t size, gfp_t flags, unsigned long caller); +extern u8 kmalloc_size_index[24]; + +static inline unsigned int size_index_elem(unsigned int bytes) +{ + return (bytes - 1) / 8; +} + +/* + * Find the kmem_cache structure that serves a given size of + * allocation + * + * This assumes size is larger than zero and not larger than + * KMALLOC_MAX_CACHE_SIZE and the caller must check that. + */ +static inline struct kmem_cache * +kmalloc_slab(size_t size, gfp_t flags, unsigned long caller) +{ + unsigned int index; + + if (size <= 192) + index = kmalloc_size_index[size_index_elem(size)]; + else + index = fls(size - 1); + + return kmalloc_caches[kmalloc_type(flags, caller)][index]; +} void *__kmem_cache_alloc_node(struct kmem_cache *s, gfp_t gfpflags, int node, size_t orig_size, diff --git a/mm/slab_common.c b/mm/slab_common.c index f4f275613d2a..31ade17a7ad9 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -665,7 +665,7 @@ EXPORT_SYMBOL(random_kmalloc_seed); * of two cache sizes there. The size of larger slabs can be determined using * fls. */ -static u8 size_index[24] __ro_after_init = { +u8 kmalloc_size_index[24] __ro_after_init = { 3, /* 8 */ 4, /* 16 */ 5, /* 24 */ @@ -692,33 +692,6 @@ static u8 size_index[24] __ro_after_init = { 2 /* 192 */ }; -static inline unsigned int size_index_elem(unsigned int bytes) -{ - return (bytes - 1) / 8; -} - -/* - * Find the kmem_cache structure that serves a given size of - * allocation - */ -struct kmem_cache *kmalloc_slab(size_t size, gfp_t flags, unsigned long caller) -{ - unsigned int index; - - if (size <= 192) { - if (!size) - return ZERO_SIZE_PTR; - - index = size_index[size_index_elem(size)]; - } else { - if (WARN_ON_ONCE(size > KMALLOC_MAX_CACHE_SIZE)) - return NULL; - index = fls(size - 1); - } - - return kmalloc_caches[kmalloc_type(flags, caller)][index]; -} - size_t kmalloc_size_roundup(size_t size) { if (size && size <= KMALLOC_MAX_CACHE_SIZE) { @@ -843,9 +816,9 @@ void __init setup_kmalloc_cache_index_table(void) for (i = 8; i < KMALLOC_MIN_SIZE; i += 8) { unsigned int elem = size_index_elem(i); - if (elem >= ARRAY_SIZE(size_index)) + if (elem >= ARRAY_SIZE(kmalloc_size_index)) break; - size_index[elem] = KMALLOC_SHIFT_LOW; + kmalloc_size_index[elem] = KMALLOC_SHIFT_LOW; } if (KMALLOC_MIN_SIZE >= 64) { @@ -854,7 +827,7 @@ void __init setup_kmalloc_cache_index_table(void) * is 64 byte. */ for (i = 64 + 8; i <= 96; i += 8) - size_index[size_index_elem(i)] = 7; + kmalloc_size_index[size_index_elem(i)] = 7; } @@ -865,7 +838,7 @@ void __init setup_kmalloc_cache_index_table(void) * instead. */ for (i = 128 + 8; i <= 192; i += 8) - size_index[size_index_elem(i)] = 8; + kmalloc_size_index[size_index_elem(i)] = 8; } } @@ -977,10 +950,10 @@ void *__do_kmalloc_node(size_t size, gfp_t flags, int node, unsigned long caller return ret; } - s = kmalloc_slab(size, flags, caller); + if (unlikely(!size)) + return ZERO_SIZE_PTR; - if (unlikely(ZERO_OR_NULL_PTR(s))) - return s; + s = kmalloc_slab(size, flags, caller); ret = __kmem_cache_alloc_node(s, flags, node, size, caller); ret = kasan_kmalloc(s, ret, size, flags);