From patchwork Mon Sep 9 01:29:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Feng Tang X-Patchwork-Id: 13795731 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4D78CCD4F4C for ; Mon, 9 Sep 2024 01:30:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D33246B00FE; Sun, 8 Sep 2024 21:30:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CE2496B0100; Sun, 8 Sep 2024 21:30:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B36666B0101; Sun, 8 Sep 2024 21:30:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 92D4B6B00FE for ; Sun, 8 Sep 2024 21:30:18 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 4702612095F for ; Mon, 9 Sep 2024 01:30:18 +0000 (UTC) X-FDA: 82543469316.25.8034BB0 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.15]) by imf30.hostedemail.com (Postfix) with ESMTP id 29EE880010 for ; Mon, 9 Sep 2024 01:30:16 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=UO0zbAn1; spf=pass (imf30.hostedemail.com: domain of feng.tang@intel.com designates 198.175.65.15 as permitted sender) smtp.mailfrom=feng.tang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1725845282; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5SIq2Al2KKMR6KlWBzxeKlCo6AeeNgLC3rgQdDAQ+Bw=; b=tnJGjbIqmoBK5/xt28nlrAfwarz8AXq/VV8TPtnXhlFxy6Qv1UUCTS82VpL/du5nEb8jSt U8dvFAMHEhFFgtGB4utEZqOz/ei3ooAij84JOWrDwEE32OGWhmZTjAbhL6AjlK4Vu98ivl 9WaGQmWfCW+FO1zcY8kzDFKRu0+R+3o= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=UO0zbAn1; spf=pass (imf30.hostedemail.com: domain of feng.tang@intel.com designates 198.175.65.15 as permitted sender) smtp.mailfrom=feng.tang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1725845282; a=rsa-sha256; cv=none; b=IEAnvDBVWv5bUe7d1sw8TqWsblVyJhRbqfZH1bJwxDA+GbGtdduHtFuy4GStE6xAZPzGOB VbjkynQvvs+Mf8IMTDU/NhazEJN1zqc/FJGaEq8WlfAvQbiYLe/7LHLj8L+hrIPr7vtZkS aZ+zqyc03D4UMmNMtH2MlRx70tGsZSM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1725845416; x=1757381416; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=B8mgHFZzGD1KfDB0WGMvxSgxbcgttavsuRFve7SsS4M=; b=UO0zbAn1j7vuzTnpmStJmkvfnOxNJ9Op+nqvh7P0xugVc96RZGWWASwb z/gE5KNpIX17z6sneuDNeuA3JaBXKEQIgzo7l8xqkQUFH4b/3mVC3dNkV OdJ5tM0lu7R912jZUzxPKYiiCB+6PQjPBSpJIsFeTDt5jzctY2JuSj9x5 Sda12nZMHLEHb0srYy2neudfWm2JQgmCOLuG/XRugpIIUJ7ZvdV0Oiaqp S+wza6LXdpf+iZ/2XOKkTzcetf1LbtoowfwjCIjCkA2mYR++ZOy0RfM2k hGc5mnrWQm2iuJqGNyLznY592COpxz8ONuNCH80Wmm8bIUKD7SPp6X9EA g==; X-CSE-ConnectionGUID: JEhuRQ/gQWCrtPk239EGVA== X-CSE-MsgGUID: KKxdpOhDS4qEup2PxUWykQ== X-IronPort-AV: E=McAfee;i="6700,10204,11189"; a="28258131" X-IronPort-AV: E=Sophos;i="6.10,213,1719903600"; d="scan'208";a="28258131" Received: from orviesa009.jf.intel.com ([10.64.159.149]) by orvoesa107.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Sep 2024 18:30:16 -0700 X-CSE-ConnectionGUID: cx70BTLlS3CxDDTgPHNKyA== X-CSE-MsgGUID: E/DBYOyRSA6Yo9pun0F7fg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,213,1719903600"; d="scan'208";a="66486467" Received: from feng-clx.sh.intel.com ([10.239.159.50]) by orviesa009.jf.intel.com with ESMTP; 08 Sep 2024 18:30:11 -0700 From: Feng Tang To: Vlastimil Babka , Andrew Morton , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Andrey Konovalov , Marco Elver , Shuah Khan , David Gow , Danilo Krummrich Cc: linux-mm@kvack.org, kasan-dev@googlegroups.com, linux-kernel@vger.kernel.org, Feng Tang Subject: [PATCH 3/5] mm/slub: Improve redzone check and zeroing for krealloc() Date: Mon, 9 Sep 2024 09:29:56 +0800 Message-Id: <20240909012958.913438-4-feng.tang@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240909012958.913438-1-feng.tang@intel.com> References: <20240909012958.913438-1-feng.tang@intel.com> MIME-Version: 1.0 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 29EE880010 X-Stat-Signature: e8wxn7yxebds16aczjyoof4fjha4h64f X-Rspam-User: X-HE-Tag: 1725845416-635326 X-HE-Meta: U2FsdGVkX1/pnJw8ARuRbW0SC835OW371LuGL4H0dbIHViKbSyiBdIoQfZsdRGSMu2Hg1LYUuLYr5737HramXIdYnhny2oEqOH2cVw4WjKlI0oZO35lYvKq/Rg/mBE4XxX7rNMYfpG+B8BSYUWhHt8hiFnsfuneoHwveVdNLm6p4r+8uHp/mhtwDnv91GMWkhuCvc4L93QunL/dk0IU9Ddg1xso9xZ22l8fumyoD6aSIBKpn2LMMEFObmqepBQm50E2fce/7V4Ca/+BacWh0CkG4nUFyk7163szdCZaaYghquxN7R0iXC6jUNpBZ5Vuy4jjSi722B3hd1rUAdXmTw93bFIV3MDvsCDmUQpBiyUnDOftZyHoqA5V2U/ZotZnVc9dfxNaTnhTDJnmwxwPTtC5/ggC/2kDphK6m57tId6RVM3AEN5lyJpZtCDjGlOJcC2exUe0b3xnAOsScRJe61Yq+ZXwVKSDWxSyJ+I+MooOE7guZ7X3lf5cgXqixMvo6YspwGqGIzrzOIu0pA8hA1YO1tGMZ7btqZUWrmm0a49jS6YrjDyuQdLQQk1Z3BCBFb+WaEv36nFtRzhPvBbxPeXM0ZEy+Pu2z5xHmyiwoetkwlHcCd8YMTkP6DLQu7EsLzkurBw5FVlGXEcTRiF6tMIk7NBBPNDMo7xH56jUnpLjepwB4KOHZNLNpAhlHuBRhe0loo0N/DWwRM9rwDSQ3sMnqBhSPRkCivbj4qCa5QPHaMHSrRSnawliiA05ugbZccPk/8HZI0MHNTD59R0/fiQ63HYa62xeVJtiu6KcMoAYdMDO1t4EUkW//KVFspCw+O65e1qiPc1az9TN8By/8VOpm9fCHl2/8EohyopAkqnhNcYW7rqyLfFvNnExdrJrsMlGIGvq2B46CgrcxMcAd9W29wIq6gOy1K6ohG8gBCwRPRLZ4SSITCR1aP+erpPcuwljBKb7MiWCSCyLTZpo ieIu25nh ldlJISNw58fJPDAYXS/GUg9OqQ0PD/8wMwdRX5LP1Wcuc/S8Bj+yEkt+qptxAL4lds/dlwsYhu4da1mf82zOY/WWQtnyQm2qX8IQWZUwZ1ra+ol4SPwqFOo2jgxYul/HHe75dSXKAZwtHsaCrf8fMqbCJGjyVRApk2Ppx1kpYH2LDh+m+ZwzhA8+hWSkrBJGBFJpDMhTiL5rQnWXBrOdp+8mxJNpHGH+jeCdGAo6tRa+uWQc2Okr2x/+rx5Zvh0iP0M1o X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: For current krealloc(), one problem is its caller doesn't know what's the actual request size, say the object is 64 bytes kmalloc one, but the original caller may only requested 48 bytes. And when krealloc() shrinks or grows in the same object, or allocate a new bigger object, it lacks this 'original size' information to do accurate data preserving or zeroing (when __GFP_ZERO is set). And when some slub debug option is enabled, kmalloc caches do have this 'orig_size' feature. So utilize it to do more accurate data handling, as well as enforce the kmalloc-redzone sanity check. The krealloc() related code is moved from slab_common.c to slub.c for more efficient function calling. Suggested-by: Vlastimil Babka Signed-off-by: Feng Tang --- mm/slab_common.c | 84 ------------------------------------- mm/slub.c | 106 +++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 106 insertions(+), 84 deletions(-) diff --git a/mm/slab_common.c b/mm/slab_common.c index ad438ba62485..e59942fb7970 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1297,90 +1297,6 @@ module_init(slab_proc_init); #endif /* CONFIG_SLUB_DEBUG */ -static __always_inline __realloc_size(2) void * -__do_krealloc(const void *p, size_t new_size, gfp_t flags) -{ - void *ret; - size_t ks; - - /* Check for double-free before calling ksize. */ - if (likely(!ZERO_OR_NULL_PTR(p))) { - if (!kasan_check_byte(p)) - return NULL; - ks = ksize(p); - } else - ks = 0; - - /* If the object still fits, repoison it precisely. */ - if (ks >= new_size) { - /* Zero out spare memory. */ - if (want_init_on_alloc(flags)) { - kasan_disable_current(); - memset((void *)p + new_size, 0, ks - new_size); - kasan_enable_current(); - } - - p = kasan_krealloc((void *)p, new_size, flags); - return (void *)p; - } - - ret = kmalloc_node_track_caller_noprof(new_size, flags, NUMA_NO_NODE, _RET_IP_); - if (ret && p) { - /* Disable KASAN checks as the object's redzone is accessed. */ - kasan_disable_current(); - memcpy(ret, kasan_reset_tag(p), ks); - kasan_enable_current(); - } - - return ret; -} - -/** - * krealloc - reallocate memory. The contents will remain unchanged. - * @p: object to reallocate memory for. - * @new_size: how many bytes of memory are required. - * @flags: the type of memory to allocate. - * - * If @p is %NULL, krealloc() behaves exactly like kmalloc(). If @new_size - * is 0 and @p is not a %NULL pointer, the object pointed to is freed. - * - * If __GFP_ZERO logic is requested, callers must ensure that, starting with the - * initial memory allocation, every subsequent call to this API for the same - * memory allocation is flagged with __GFP_ZERO. Otherwise, it is possible that - * __GFP_ZERO is not fully honored by this API. - * - * This is the case, since krealloc() only knows about the bucket size of an - * allocation (but not the exact size it was allocated with) and hence - * implements the following semantics for shrinking and growing buffers with - * __GFP_ZERO. - * - * new bucket - * 0 size size - * |--------|----------------| - * | keep | zero | - * - * In any case, the contents of the object pointed to are preserved up to the - * lesser of the new and old sizes. - * - * Return: pointer to the allocated memory or %NULL in case of error - */ -void *krealloc_noprof(const void *p, size_t new_size, gfp_t flags) -{ - void *ret; - - if (unlikely(!new_size)) { - kfree(p); - return ZERO_SIZE_PTR; - } - - ret = __do_krealloc(p, new_size, flags); - if (ret && kasan_reset_tag(p) != kasan_reset_tag(ret)) - kfree(p); - - return ret; -} -EXPORT_SYMBOL(krealloc_noprof); - /** * kfree_sensitive - Clear sensitive information in memory before freeing * @p: object to free memory of diff --git a/mm/slub.c b/mm/slub.c index 4cb3822dba08..d4c938dfb89e 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4709,6 +4709,112 @@ void kfree(const void *object) } EXPORT_SYMBOL(kfree); +static __always_inline __realloc_size(2) void * +__do_krealloc(const void *p, size_t new_size, gfp_t flags) +{ + void *ret; + size_t ks; + int orig_size = 0; + struct kmem_cache *s; + + /* Check for double-free before calling ksize. */ + if (likely(!ZERO_OR_NULL_PTR(p))) { + if (!kasan_check_byte(p)) + return NULL; + + s = virt_to_cache(p); + orig_size = get_orig_size(s, (void *)p); + ks = s->object_size; + } else + ks = 0; + + /* If the object doesn't fit, allocate a bigger one */ + if (new_size > ks) + goto alloc_new; + + /* Zero out spare memory. */ + if (want_init_on_alloc(flags)) { + kasan_disable_current(); + if (orig_size < new_size) + memset((void *)p + orig_size, 0, new_size - orig_size); + else + memset((void *)p + new_size, 0, ks - new_size); + kasan_enable_current(); + } + + if (slub_debug_orig_size(s) && !is_kfence_address(p)) { + set_orig_size(s, (void *)p, new_size); + if (s->flags & SLAB_RED_ZONE && new_size < ks) + memset_no_sanitize_memory((void *)p + new_size, + SLUB_RED_ACTIVE, ks - new_size); + } + + p = kasan_krealloc((void *)p, new_size, flags); + return (void *)p; + +alloc_new: + ret = kmalloc_node_track_caller_noprof(new_size, flags, NUMA_NO_NODE, _RET_IP_); + if (ret && p) { + /* Disable KASAN checks as the object's redzone is accessed. */ + kasan_disable_current(); + if (orig_size) + memcpy(ret, kasan_reset_tag(p), orig_size); + kasan_enable_current(); + } + + return ret; +} + +/** + * krealloc - reallocate memory. The contents will remain unchanged. + * @p: object to reallocate memory for. + * @new_size: how many bytes of memory are required. + * @flags: the type of memory to allocate. + * + * If @p is %NULL, krealloc() behaves exactly like kmalloc(). If @new_size + * is 0 and @p is not a %NULL pointer, the object pointed to is freed. + * + * If __GFP_ZERO logic is requested, callers must ensure that, starting with the + * initial memory allocation, every subsequent call to this API for the same + * memory allocation is flagged with __GFP_ZERO. Otherwise, it is possible that + * __GFP_ZERO is not fully honored by this API. + * + * When slub_debug_orig_size() is off, since krealloc() only knows about the + * bucket size of an allocation (but not the exact size it was allocated with) + * and hence implements the following semantics for shrinking and growing + * buffers with __GFP_ZERO. + * + * new bucket + * 0 size size + * |--------|----------------| + * | keep | zero | + * + * Otherwize, the original allocation size 'orig_size' could be used to + * precisely clear the requested size, and the new size will also be stored as + * the new 'orig_size'. + * + * In any case, the contents of the object pointed to are preserved up to the + * lesser of the new and old sizes. + * + * Return: pointer to the allocated memory or %NULL in case of error + */ +void *krealloc_noprof(const void *p, size_t new_size, gfp_t flags) +{ + void *ret; + + if (unlikely(!new_size)) { + kfree(p); + return ZERO_SIZE_PTR; + } + + ret = __do_krealloc(p, new_size, flags); + if (ret && kasan_reset_tag(p) != kasan_reset_tag(ret)) + kfree(p); + + return ret; +} +EXPORT_SYMBOL(krealloc_noprof); + struct detached_freelist { struct slab *slab; void *tail;