From patchwork Wed Feb 12 08:15:04 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: GONG Ruiqi X-Patchwork-Id: 13971344 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 486C6C0219E for ; Wed, 12 Feb 2025 08:05:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AD9486B0098; Wed, 12 Feb 2025 03:05:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A89CF6B0099; Wed, 12 Feb 2025 03:05:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8DCC26B009A; Wed, 12 Feb 2025 03:05:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 65F636B0098 for ; Wed, 12 Feb 2025 03:05:10 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 1F2ABB1FB6 for ; Wed, 12 Feb 2025 08:05:10 +0000 (UTC) X-FDA: 83110557180.24.04F42C1 Received: from szxga08-in.huawei.com (szxga08-in.huawei.com [45.249.212.255]) by imf12.hostedemail.com (Postfix) with ESMTP id 6290D40009 for ; Wed, 12 Feb 2025 08:05:05 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=none; spf=pass (imf12.hostedemail.com: domain of gongruiqi1@huawei.com designates 45.249.212.255 as permitted sender) smtp.mailfrom=gongruiqi1@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739347508; a=rsa-sha256; cv=none; b=sbzYshkm4JKI6E9qYF6BSGkQeaTOtcC7/Qv8ZU2CiuSeoRunKMTxXxW6yq1yEnFqRb8bQp Smyo7aPR8LsQMVBHc29549Z5NdRZALbFCZIyGopEii6GBiLMKOjOWsplmSxjP5cI/lHBfr QPTpbKO4ofAnD73WD6zv+MmUjn7eLVo= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=none; spf=pass (imf12.hostedemail.com: domain of gongruiqi1@huawei.com designates 45.249.212.255 as permitted sender) smtp.mailfrom=gongruiqi1@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739347508; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qg6JnXt984AvxRkxRoJvQUtUJFG0YGTlgVBwpQ0UBrg=; b=bxtV+CWoh+PEdH1nba1Fj3zCtERLtSDgmQvyJqE6HCd5dggGCBx8PwkapUlGou+W/QZNig Spc3m3EVUINCp1E517hidgcutVAfmhprMR8OH0wV+DJiTWHgN0lxDFy4ZABhTF1CnF6ezP uB4129s9bTbuQtoLY/KZtvPuaLEre20= Received: from mail.maildlp.com (unknown [172.19.163.48]) by szxga08-in.huawei.com (SkyGuard) with ESMTP id 4Yt9jF3gjhz16PjB; Wed, 12 Feb 2025 16:00:33 +0800 (CST) Received: from kwepemg100016.china.huawei.com (unknown [7.202.181.57]) by mail.maildlp.com (Postfix) with ESMTPS id 588A21802D0; Wed, 12 Feb 2025 16:05:00 +0800 (CST) Received: from huawei.com (10.67.174.33) by kwepemg100016.china.huawei.com (7.202.181.57) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.11; Wed, 12 Feb 2025 16:04:59 +0800 From: GONG Ruiqi To: Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Vlastimil Babka , Kees Cook CC: Tamas Koczka , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Xiu Jianfeng , , , , Subject: [PATCH v3 1/2] slab: Adjust placement of __kvmalloc_node_noprof Date: Wed, 12 Feb 2025 16:15:04 +0800 Message-ID: <20250212081505.2025320-2-gongruiqi1@huawei.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20250212081505.2025320-1-gongruiqi1@huawei.com> References: <20250212081505.2025320-1-gongruiqi1@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.67.174.33] X-ClientProxiedBy: dggems705-chm.china.huawei.com (10.3.19.182) To kwepemg100016.china.huawei.com (7.202.181.57) X-Stat-Signature: z9ruf5bmn5wrspay7qd7q56sepsr79be X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 6290D40009 X-Rspam-User: X-HE-Tag: 1739347505-872999 X-HE-Meta: U2FsdGVkX1+hS1x3KVZyKyEqlUWUYr/r4sRqHr1OtWZD7eE06nlCRp8/pIN3jv/j75xQuEFFd3rXudWq2C3cZzktc1QgjZ4609kUjhvMWDaBz2UUutu7r5DKT326+t9IQem1/J9TzcAlkyKD5tXvY0t4Yefd3+xSRYIbiEsNRfuk1c2np1p0EuxncV6XB3RR5BYBDL1e9MwqWApUZoxMSWxrJqfQcWNKzMC9srfGib2L9gfyp7h9C+tZI/hprfP19j60Lr63WDgUxWl6Le1aLwjNwiHL/TdX99W5LuzIRco6MKpxFUrKgTDH5626gF47BlLi8oTNotGmWqszT47andekg8GRsu3OC60MhLnQH3pfybKCyKoxT4deMi7X4fRp2w+4Cn9W7/Fk1wXWzQT+d5p9IRuYXbTGbOsHddz/T10L+bt4sEWZ2ezDHc6sVXHQIggvzf6A3yny0sOY91JW9C6PW7veXnAKh8bWL/mkO87jt48XF74weinWihlJbbdp06zU/PGUT7u4QFMzet0h9iM1ELUFTZgRLhBwbVFoZqWZ79gPFR2HmqwUxWv8MOfBnNqmC/w8Eiw0nYB7G7elWwpbRmMN5qtXdxp6KLLg63MOQdaZ96d3U6O+JXMANZ4wAgeKMosCJT/Zhvj/JJUxe6PRw+OQjs0JSodSdHRSAPIcMgDxcGx7Qak0G5rgZVG/5JVEbGBuclXt+McWo5lxbdzu3sChfE755VWRw+LSQPH8xs2uii434L0yQKsbOoX3ZT5OkjgAP+XftTDJfDNrQAZ2xMXmmoNM4Foz+hWQqvgn0i0TSg8EnokeP2gHwTvxJW6E4MOQoiv2amZfHZ7zvjldsn3jo6Y3vpJB0XnJZkEX2KXzPow4OU03WcTqo4XL5HIMHwJL8IUiNoPJe2YERZfcipUl7dBuSH8L+oDjwrJ1/Jhz13bsTbm5g+zhXG3KR4YvFQS6lBNCz/3JEaO 6/7N3Q2q WkYUwSiXv/L9aHfxgd0D9IQ9DeuPPvmbJtyoGoZwod3tQQchhkFVuWz+26Y/elGaXbXzILpb0wL++IYmVWhd/QM8LFXlDc5JhqeTgkJ01jFzGVWF6jZHvLqP69B4kLvdk10FLUDKHTYK+ELlaC4IHJvcbShbCXne3boN6w3IH8J1jVNrr1nqt4a90fShHpr1qK9qa7Kb6fXWqJpi7gJaZB/S9G69PSujmOcwS/MN2DeGJBHBoW41g3oLia2dYjR1lohyhdXqOfHlRW5U= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Move __kvmalloc_node_noprof (as well as kvfree*, kvrealloc_noprof and kmalloc_gfp_adjust for consistency) into mm/slub.c so that it can directly invoke __do_kmalloc_node, which is needed for the next patch. No functional changes intended. Signed-off-by: GONG Ruiqi --- mm/slub.c | 162 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ mm/util.c | 162 ------------------------------------------------------ 2 files changed, 162 insertions(+), 162 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 1f50129dcfb3..abc982d68feb 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4878,6 +4878,168 @@ void *krealloc_noprof(const void *p, size_t new_size, gfp_t flags) } EXPORT_SYMBOL(krealloc_noprof); +static gfp_t kmalloc_gfp_adjust(gfp_t flags, size_t size) +{ + /* + * We want to attempt a large physically contiguous block first because + * it is less likely to fragment multiple larger blocks and therefore + * contribute to a long term fragmentation less than vmalloc fallback. + * However make sure that larger requests are not too disruptive - no + * OOM killer and no allocation failure warnings as we have a fallback. + */ + if (size > PAGE_SIZE) { + flags |= __GFP_NOWARN; + + if (!(flags & __GFP_RETRY_MAYFAIL)) + flags |= __GFP_NORETRY; + + /* nofail semantic is implemented by the vmalloc fallback */ + flags &= ~__GFP_NOFAIL; + } + + return flags; +} + +/** + * __kvmalloc_node - attempt to allocate physically contiguous memory, but upon + * failure, fall back to non-contiguous (vmalloc) allocation. + * @size: size of the request. + * @b: which set of kmalloc buckets to allocate from. + * @flags: gfp mask for the allocation - must be compatible (superset) with GFP_KERNEL. + * @node: numa node to allocate from + * + * Uses kmalloc to get the memory but if the allocation fails then falls back + * to the vmalloc allocator. Use kvfree for freeing the memory. + * + * GFP_NOWAIT and GFP_ATOMIC are not supported, neither is the __GFP_NORETRY modifier. + * __GFP_RETRY_MAYFAIL is supported, and it should be used only if kmalloc is + * preferable to the vmalloc fallback, due to visible performance drawbacks. + * + * Return: pointer to the allocated memory of %NULL in case of failure + */ +void *__kvmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node) +{ + void *ret; + + /* + * It doesn't really make sense to fallback to vmalloc for sub page + * requests + */ + ret = __kmalloc_node_noprof(PASS_BUCKET_PARAMS(size, b), + kmalloc_gfp_adjust(flags, size), + node); + if (ret || size <= PAGE_SIZE) + return ret; + + /* non-sleeping allocations are not supported by vmalloc */ + if (!gfpflags_allow_blocking(flags)) + return NULL; + + /* Don't even allow crazy sizes */ + if (unlikely(size > INT_MAX)) { + WARN_ON_ONCE(!(flags & __GFP_NOWARN)); + return NULL; + } + + /* + * kvmalloc() can always use VM_ALLOW_HUGE_VMAP, + * since the callers already cannot assume anything + * about the resulting pointer, and cannot play + * protection games. + */ + return __vmalloc_node_range_noprof(size, 1, VMALLOC_START, VMALLOC_END, + flags, PAGE_KERNEL, VM_ALLOW_HUGE_VMAP, + node, __builtin_return_address(0)); +} +EXPORT_SYMBOL(__kvmalloc_node_noprof); + +/** + * kvfree() - Free memory. + * @addr: Pointer to allocated memory. + * + * kvfree frees memory allocated by any of vmalloc(), kmalloc() or kvmalloc(). + * It is slightly more efficient to use kfree() or vfree() if you are certain + * that you know which one to use. + * + * Context: Either preemptible task context or not-NMI interrupt. + */ +void kvfree(const void *addr) +{ + if (is_vmalloc_addr(addr)) + vfree(addr); + else + kfree(addr); +} +EXPORT_SYMBOL(kvfree); + +/** + * kvfree_sensitive - Free a data object containing sensitive information. + * @addr: address of the data object to be freed. + * @len: length of the data object. + * + * Use the special memzero_explicit() function to clear the content of a + * kvmalloc'ed object containing sensitive data to make sure that the + * compiler won't optimize out the data clearing. + */ +void kvfree_sensitive(const void *addr, size_t len) +{ + if (likely(!ZERO_OR_NULL_PTR(addr))) { + memzero_explicit((void *)addr, len); + kvfree(addr); + } +} +EXPORT_SYMBOL(kvfree_sensitive); + +/** + * kvrealloc - reallocate memory; contents remain unchanged + * @p: object to reallocate memory for + * @size: the size to reallocate + * @flags: the flags for the page level allocator + * + * If @p is %NULL, kvrealloc() behaves exactly like kvmalloc(). If @size is 0 + * and @p is not a %NULL pointer, the object pointed to is freed. + * + * If __GFP_ZERO logic is requested, callers must ensure that, starting with the + * initial memory allocation, every subsequent call to this API for the same + * memory allocation is flagged with __GFP_ZERO. Otherwise, it is possible that + * __GFP_ZERO is not fully honored by this API. + * + * In any case, the contents of the object pointed to are preserved up to the + * lesser of the new and old sizes. + * + * This function must not be called concurrently with itself or kvfree() for the + * same memory allocation. + * + * Return: pointer to the allocated memory or %NULL in case of error + */ +void *kvrealloc_noprof(const void *p, size_t size, gfp_t flags) +{ + void *n; + + if (is_vmalloc_addr(p)) + return vrealloc_noprof(p, size, flags); + + n = krealloc_noprof(p, size, kmalloc_gfp_adjust(flags, size)); + if (!n) { + /* We failed to krealloc(), fall back to kvmalloc(). */ + n = kvmalloc_noprof(size, flags); + if (!n) + return NULL; + + if (p) { + /* We already know that `p` is not a vmalloc address. */ + kasan_disable_current(); + memcpy(n, kasan_reset_tag(p), ksize(p)); + kasan_enable_current(); + + kfree(p); + } + } + + return n; +} +EXPORT_SYMBOL(kvrealloc_noprof); + struct detached_freelist { struct slab *slab; void *tail; diff --git a/mm/util.c b/mm/util.c index b6b9684a1438..c808668f0548 100644 --- a/mm/util.c +++ b/mm/util.c @@ -612,168 +612,6 @@ unsigned long vm_mmap(struct file *file, unsigned long addr, } EXPORT_SYMBOL(vm_mmap); -static gfp_t kmalloc_gfp_adjust(gfp_t flags, size_t size) -{ - /* - * We want to attempt a large physically contiguous block first because - * it is less likely to fragment multiple larger blocks and therefore - * contribute to a long term fragmentation less than vmalloc fallback. - * However make sure that larger requests are not too disruptive - no - * OOM killer and no allocation failure warnings as we have a fallback. - */ - if (size > PAGE_SIZE) { - flags |= __GFP_NOWARN; - - if (!(flags & __GFP_RETRY_MAYFAIL)) - flags |= __GFP_NORETRY; - - /* nofail semantic is implemented by the vmalloc fallback */ - flags &= ~__GFP_NOFAIL; - } - - return flags; -} - -/** - * __kvmalloc_node - attempt to allocate physically contiguous memory, but upon - * failure, fall back to non-contiguous (vmalloc) allocation. - * @size: size of the request. - * @b: which set of kmalloc buckets to allocate from. - * @flags: gfp mask for the allocation - must be compatible (superset) with GFP_KERNEL. - * @node: numa node to allocate from - * - * Uses kmalloc to get the memory but if the allocation fails then falls back - * to the vmalloc allocator. Use kvfree for freeing the memory. - * - * GFP_NOWAIT and GFP_ATOMIC are not supported, neither is the __GFP_NORETRY modifier. - * __GFP_RETRY_MAYFAIL is supported, and it should be used only if kmalloc is - * preferable to the vmalloc fallback, due to visible performance drawbacks. - * - * Return: pointer to the allocated memory of %NULL in case of failure - */ -void *__kvmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node) -{ - void *ret; - - /* - * It doesn't really make sense to fallback to vmalloc for sub page - * requests - */ - ret = __kmalloc_node_noprof(PASS_BUCKET_PARAMS(size, b), - kmalloc_gfp_adjust(flags, size), - node); - if (ret || size <= PAGE_SIZE) - return ret; - - /* non-sleeping allocations are not supported by vmalloc */ - if (!gfpflags_allow_blocking(flags)) - return NULL; - - /* Don't even allow crazy sizes */ - if (unlikely(size > INT_MAX)) { - WARN_ON_ONCE(!(flags & __GFP_NOWARN)); - return NULL; - } - - /* - * kvmalloc() can always use VM_ALLOW_HUGE_VMAP, - * since the callers already cannot assume anything - * about the resulting pointer, and cannot play - * protection games. - */ - return __vmalloc_node_range_noprof(size, 1, VMALLOC_START, VMALLOC_END, - flags, PAGE_KERNEL, VM_ALLOW_HUGE_VMAP, - node, __builtin_return_address(0)); -} -EXPORT_SYMBOL(__kvmalloc_node_noprof); - -/** - * kvfree() - Free memory. - * @addr: Pointer to allocated memory. - * - * kvfree frees memory allocated by any of vmalloc(), kmalloc() or kvmalloc(). - * It is slightly more efficient to use kfree() or vfree() if you are certain - * that you know which one to use. - * - * Context: Either preemptible task context or not-NMI interrupt. - */ -void kvfree(const void *addr) -{ - if (is_vmalloc_addr(addr)) - vfree(addr); - else - kfree(addr); -} -EXPORT_SYMBOL(kvfree); - -/** - * kvfree_sensitive - Free a data object containing sensitive information. - * @addr: address of the data object to be freed. - * @len: length of the data object. - * - * Use the special memzero_explicit() function to clear the content of a - * kvmalloc'ed object containing sensitive data to make sure that the - * compiler won't optimize out the data clearing. - */ -void kvfree_sensitive(const void *addr, size_t len) -{ - if (likely(!ZERO_OR_NULL_PTR(addr))) { - memzero_explicit((void *)addr, len); - kvfree(addr); - } -} -EXPORT_SYMBOL(kvfree_sensitive); - -/** - * kvrealloc - reallocate memory; contents remain unchanged - * @p: object to reallocate memory for - * @size: the size to reallocate - * @flags: the flags for the page level allocator - * - * If @p is %NULL, kvrealloc() behaves exactly like kvmalloc(). If @size is 0 - * and @p is not a %NULL pointer, the object pointed to is freed. - * - * If __GFP_ZERO logic is requested, callers must ensure that, starting with the - * initial memory allocation, every subsequent call to this API for the same - * memory allocation is flagged with __GFP_ZERO. Otherwise, it is possible that - * __GFP_ZERO is not fully honored by this API. - * - * In any case, the contents of the object pointed to are preserved up to the - * lesser of the new and old sizes. - * - * This function must not be called concurrently with itself or kvfree() for the - * same memory allocation. - * - * Return: pointer to the allocated memory or %NULL in case of error - */ -void *kvrealloc_noprof(const void *p, size_t size, gfp_t flags) -{ - void *n; - - if (is_vmalloc_addr(p)) - return vrealloc_noprof(p, size, flags); - - n = krealloc_noprof(p, size, kmalloc_gfp_adjust(flags, size)); - if (!n) { - /* We failed to krealloc(), fall back to kvmalloc(). */ - n = kvmalloc_noprof(size, flags); - if (!n) - return NULL; - - if (p) { - /* We already know that `p` is not a vmalloc address. */ - kasan_disable_current(); - memcpy(n, kasan_reset_tag(p), ksize(p)); - kasan_enable_current(); - - kfree(p); - } - } - - return n; -} -EXPORT_SYMBOL(kvrealloc_noprof); - /** * __vmalloc_array - allocate memory for a virtually contiguous array. * @n: number of elements.