From patchwork Wed Sep 11 06:45:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Feng Tang X-Patchwork-Id: 13799752 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E0F0FEE0211 for ; Wed, 11 Sep 2024 06:47:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 65C2194000A; Wed, 11 Sep 2024 02:47:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 60D986B0387; Wed, 11 Sep 2024 02:47:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4ACEC94000A; Wed, 11 Sep 2024 02:47:16 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 299A16B0177 for ; Wed, 11 Sep 2024 02:47:16 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id CDF27812D7 for ; Wed, 11 Sep 2024 06:47:15 +0000 (UTC) X-FDA: 82551525630.16.E327C40 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.12]) by imf28.hostedemail.com (Postfix) with ESMTP id B24ACC0005 for ; Wed, 11 Sep 2024 06:47:13 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="YQO2GKW/"; spf=pass (imf28.hostedemail.com: domain of feng.tang@intel.com designates 198.175.65.12 as permitted sender) smtp.mailfrom=feng.tang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1726037129; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/S8vF7imCQy4zA2SHeFyV8/wC75Gy306lr1UXEKj3dY=; b=XH6tpPV9tSirwyVQxmb0NYjWjmpUwEZCGknp99OnLKxNFl2tWBcTB8A4Yfem7D5yG5kBcF O5GBm8zOYsVXf55j2hFfFpOzQTCxsk9JkwERIs0iVCvgyVLkpksYTBpd+5ty6WTKobMra0 RAw2behijrXDU10+WD8uRjQF10nm2QI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1726037129; a=rsa-sha256; cv=none; b=x34FxcK9O1XIvijM2WgYW+3nrOTB/B5CjBKyi9AJjPl+Egw8WVvmMMS6OjWZB1lucqjSIT 0J4BwzUMt/tRHr1SDM7/1i7qjwIYJNDyzQ8fUqzCnLRsUFDjbMfxJ0f10Yud/xo6W4ut3t 1rfkQIHtHT9fE+l9tAdgp6eRGxiG5aI= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="YQO2GKW/"; spf=pass (imf28.hostedemail.com: domain of feng.tang@intel.com designates 198.175.65.12 as permitted sender) smtp.mailfrom=feng.tang@intel.com; dmarc=pass (policy=none) header.from=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1726037233; x=1757573233; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Ag0o94U2BMb2lFyFC32W4EFoRg3K+ZMunlGHmbIXB5s=; b=YQO2GKW/G5sciCsnPuQWVZtAo3uzFc9OL2TOCMyogI5GozM/KIENJW98 a0mf/ecRaXFDvU60whm1dv20bsx05ZqVaO0/JeW4/VaQjs5Z2CtsOCAjU SGEMnN5vjPmEVDBtY1yLfvIM432Vfh5Cm1jPGZEgy/TB0BWuYAAvvDWE+ y9gO8LtONaR1GhihLYgcyHifxdt/x9MvEETmYjUZU52SmBRdcY5Y2zBol P3A6qvEiOryMqlV7INiE38DOWXChJtztTmBqGqYAT6H/lBTsx7Dr8suEk gZR0+JY7YOujqbcBlfn50i9YbXDIPhF8OojboxuYKQ3RIYEmawyFhzFZO w==; X-CSE-ConnectionGUID: PkpNX5SrSNKhFZ6ABojImg== X-CSE-MsgGUID: xKjUeJfMTgK6Ch99iPF+NQ== X-IronPort-AV: E=McAfee;i="6700,10204,11191"; a="36173018" X-IronPort-AV: E=Sophos;i="6.10,219,1719903600"; d="scan'208";a="36173018" Received: from orviesa007.jf.intel.com ([10.64.159.147]) by orvoesa104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Sep 2024 23:46:11 -0700 X-CSE-ConnectionGUID: IgyzjmIvSCmBHDIIIAmEIw== X-CSE-MsgGUID: u/rfH00VRWC+1lv/88CKcQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,219,1719903600"; d="scan'208";a="67771497" Received: from feng-clx.sh.intel.com ([10.239.159.50]) by orviesa007.jf.intel.com with ESMTP; 10 Sep 2024 23:45:55 -0700 From: Feng Tang To: Vlastimil Babka , Andrew Morton , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Andrey Konovalov , Marco Elver , Shuah Khan , David Gow , Danilo Krummrich , Alexander Potapenko , Andrey Ryabinin , Dmitry Vyukov , Vincenzo Frascino Cc: linux-mm@kvack.org, kasan-dev@googlegroups.com, linux-kernel@vger.kernel.org, Feng Tang Subject: [PATCH v2 3/5] mm/slub: Move krealloc() and related code to slub.c Date: Wed, 11 Sep 2024 14:45:33 +0800 Message-Id: <20240911064535.557650-4-feng.tang@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240911064535.557650-1-feng.tang@intel.com> References: <20240911064535.557650-1-feng.tang@intel.com> MIME-Version: 1.0 X-Stat-Signature: 3a5mtsz5j9fogo3y7ugrto9o33tkhumh X-Rspamd-Queue-Id: B24ACC0005 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1726037233-477803 X-HE-Meta: U2FsdGVkX19F7PKwySDvIT93Rtl46ZM4AVOcBbLqR7Uf5SgFnNLnwVEXDNLq5k7roYQDiRmIHAJAcOjuSf122+e3IPgsV7XIH5S6/dbsa04wPEJxBUMpzVKpVFLxoCJUFQQ+fcTu1Fo1pRpnKA1/+iZUoAeUJYtau5GtqemxNBkh0s/43Jr0nn+jKTv0TfccTY0xHOTbzy5x9Jvzp/N6ResgkdkR1XJ13Pr5Nlfq9h2Hbg/t0+scCoDuNurUv4421RBog9sgC198qDy42RyhX2SAnSjDjo55AeEzsCI/blNMgTroEa/mVTOIR2JGVnVBWMuI8fEEvSxzZyfSRpapVS7kj3g5UBw2wbRcfvXb0ynzpOrUsaENGiuoT957qynGQ+WfjGsxny/rJaQ5N7iKN8UE/REydZnkDHqpVmtqvSlE6QOQUP99kcP1Sf790OPnqLfSfIAeKxAU8j6hWeSDBMz3ZqO7gbCyN6G7C0HtXyhV4HH+sGNWlkbXNOamDPju7NNZ2xAqY+XVCFG3XAZpYAP0kpkZU58+Xwrv1Sz63a9sfoYY11ApYEVLst6mc01lzN5SizNDboMw13tKBLePJbKyl3Bcm+mN9OVg3mYByGFyzzzqNss2oYPx/tNTvhEfG+wveC9DDnRD8HyD6iX7R1VFuCJ5KDOoagSqyCwbvgvY0tAj98cETKm39k/V0feOHOtEnN4DsKj8igvHIIjC+4klGHD/+XY8ifxiII+LyOP+UGAA506SCRREHluboIwG/9nXBFskt11Cu3dc09KRGeCb+tMbp1U3psUa933Ta44Cu48tv61gly7EPQwE1HKPsp+/iI0yz39gSCWBjum99XnPTiwd1oSSbbkHOgiq9WxHN1WfC8QWZqln+O2viVjwCXoUvZOOOfmWBQpFuu25GdE32LuwKCYlETQDrIi3iTIw5gpBcq66MNQoYE03dnT0VwyrkESOAXUPh9yqChq j3Aefv8+ gCuo1OfWnEJBqPpNx7YKj6lV/6xdnbygtX24wgC3vaeiw7DRQsR8AaRTAZ7u5V6gFJiEtmEnS7QCUzKvl07i2V6G0WRKc5qQdlskDfPYyuBotxtYVo3o59ENiiNpvfbouzHXk2x4IpD0bK0m7yKKHrjGAvsPdRgxoh3b9PWJUrNeh5ViikOgRRJiBH2G4S8UX4FbWf38qFfchW7UPtEM2ZgQRJ3cjfNUaUdfANSFbQIYe+FxiSgrV7EAvX9tqGTGLT8jQ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This is a preparation for the following refactoring of krealloc(), for more efficient function calling as it will call some internal functions defined in slub.c. Signed-off-by: Feng Tang --- mm/slab_common.c | 84 ------------------------------------------------ mm/slub.c | 84 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 84 insertions(+), 84 deletions(-) diff --git a/mm/slab_common.c b/mm/slab_common.c index af6b14769fbd..5734b61a106f 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1185,90 +1185,6 @@ module_init(slab_proc_init); #endif /* CONFIG_SLUB_DEBUG */ -static __always_inline __realloc_size(2) void * -__do_krealloc(const void *p, size_t new_size, gfp_t flags) -{ - void *ret; - size_t ks; - - /* Check for double-free before calling ksize. */ - if (likely(!ZERO_OR_NULL_PTR(p))) { - if (!kasan_check_byte(p)) - return NULL; - ks = ksize(p); - } else - ks = 0; - - /* If the object still fits, repoison it precisely. */ - if (ks >= new_size) { - /* Zero out spare memory. */ - if (want_init_on_alloc(flags)) { - kasan_disable_current(); - memset((void *)p + new_size, 0, ks - new_size); - kasan_enable_current(); - } - - p = kasan_krealloc((void *)p, new_size, flags); - return (void *)p; - } - - ret = kmalloc_node_track_caller_noprof(new_size, flags, NUMA_NO_NODE, _RET_IP_); - if (ret && p) { - /* Disable KASAN checks as the object's redzone is accessed. */ - kasan_disable_current(); - memcpy(ret, kasan_reset_tag(p), ks); - kasan_enable_current(); - } - - return ret; -} - -/** - * krealloc - reallocate memory. The contents will remain unchanged. - * @p: object to reallocate memory for. - * @new_size: how many bytes of memory are required. - * @flags: the type of memory to allocate. - * - * If @p is %NULL, krealloc() behaves exactly like kmalloc(). If @new_size - * is 0 and @p is not a %NULL pointer, the object pointed to is freed. - * - * If __GFP_ZERO logic is requested, callers must ensure that, starting with the - * initial memory allocation, every subsequent call to this API for the same - * memory allocation is flagged with __GFP_ZERO. Otherwise, it is possible that - * __GFP_ZERO is not fully honored by this API. - * - * This is the case, since krealloc() only knows about the bucket size of an - * allocation (but not the exact size it was allocated with) and hence - * implements the following semantics for shrinking and growing buffers with - * __GFP_ZERO. - * - * new bucket - * 0 size size - * |--------|----------------| - * | keep | zero | - * - * In any case, the contents of the object pointed to are preserved up to the - * lesser of the new and old sizes. - * - * Return: pointer to the allocated memory or %NULL in case of error - */ -void *krealloc_noprof(const void *p, size_t new_size, gfp_t flags) -{ - void *ret; - - if (unlikely(!new_size)) { - kfree(p); - return ZERO_SIZE_PTR; - } - - ret = __do_krealloc(p, new_size, flags); - if (ret && kasan_reset_tag(p) != kasan_reset_tag(ret)) - kfree(p); - - return ret; -} -EXPORT_SYMBOL(krealloc_noprof); - /** * kfree_sensitive - Clear sensitive information in memory before freeing * @p: object to free memory of diff --git a/mm/slub.c b/mm/slub.c index 021991e17287..c1796f9dd30f 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4712,6 +4712,90 @@ void kfree(const void *object) } EXPORT_SYMBOL(kfree); +static __always_inline __realloc_size(2) void * +__do_krealloc(const void *p, size_t new_size, gfp_t flags) +{ + void *ret; + size_t ks; + + /* Check for double-free before calling ksize. */ + if (likely(!ZERO_OR_NULL_PTR(p))) { + if (!kasan_check_byte(p)) + return NULL; + ks = ksize(p); + } else + ks = 0; + + /* If the object still fits, repoison it precisely. */ + if (ks >= new_size) { + /* Zero out spare memory. */ + if (want_init_on_alloc(flags)) { + kasan_disable_current(); + memset((void *)p + new_size, 0, ks - new_size); + kasan_enable_current(); + } + + p = kasan_krealloc((void *)p, new_size, flags); + return (void *)p; + } + + ret = kmalloc_node_track_caller_noprof(new_size, flags, NUMA_NO_NODE, _RET_IP_); + if (ret && p) { + /* Disable KASAN checks as the object's redzone is accessed. */ + kasan_disable_current(); + memcpy(ret, kasan_reset_tag(p), ks); + kasan_enable_current(); + } + + return ret; +} + +/** + * krealloc - reallocate memory. The contents will remain unchanged. + * @p: object to reallocate memory for. + * @new_size: how many bytes of memory are required. + * @flags: the type of memory to allocate. + * + * If @p is %NULL, krealloc() behaves exactly like kmalloc(). If @new_size + * is 0 and @p is not a %NULL pointer, the object pointed to is freed. + * + * If __GFP_ZERO logic is requested, callers must ensure that, starting with the + * initial memory allocation, every subsequent call to this API for the same + * memory allocation is flagged with __GFP_ZERO. Otherwise, it is possible that + * __GFP_ZERO is not fully honored by this API. + * + * This is the case, since krealloc() only knows about the bucket size of an + * allocation (but not the exact size it was allocated with) and hence + * implements the following semantics for shrinking and growing buffers with + * __GFP_ZERO. + * + * new bucket + * 0 size size + * |--------|----------------| + * | keep | zero | + * + * In any case, the contents of the object pointed to are preserved up to the + * lesser of the new and old sizes. + * + * Return: pointer to the allocated memory or %NULL in case of error + */ +void *krealloc_noprof(const void *p, size_t new_size, gfp_t flags) +{ + void *ret; + + if (unlikely(!new_size)) { + kfree(p); + return ZERO_SIZE_PTR; + } + + ret = __do_krealloc(p, new_size, flags); + if (ret && kasan_reset_tag(p) != kasan_reset_tag(ret)) + kfree(p); + + return ret; +} +EXPORT_SYMBOL(krealloc_noprof); + struct detached_freelist { struct slab *slab; void *tail;