From patchwork Mon Jul 22 16:29:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Danilo Krummrich X-Patchwork-Id: 13738940 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6B626C3DA5D for ; Mon, 22 Jul 2024 16:32:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 010166B008C; Mon, 22 Jul 2024 12:32:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EB52E6B0092; Mon, 22 Jul 2024 12:31:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CDFFF6B0093; Mon, 22 Jul 2024 12:31:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id AC5D66B008C for ; Mon, 22 Jul 2024 12:31:59 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 28F69419D8 for ; Mon, 22 Jul 2024 16:31:59 +0000 (UTC) X-FDA: 82367930358.17.C60CA5A Received: from sin.source.kernel.org (sin.source.kernel.org [145.40.73.55]) by imf24.hostedemail.com (Postfix) with ESMTP id BD660180041 for ; Mon, 22 Jul 2024 16:31:56 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=VHf3266t; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf24.hostedemail.com: domain of dakr@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=dakr@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1721665883; a=rsa-sha256; cv=none; b=OmMkeLGqtGvGKp0eP5Ic2MiPUBSJUe0D91eImJ/OVk/JM74OhZXOO3W8rmyr4RAIYltLbC UiXKrOIoHrjc4B06knjet95stgD1uTCyH//A8SSG67HzZsO1uUVzU0Vx3wJGUVtzTFz31G x/NmD2801PxjmzbJZUwHrrrn8ySCe4I= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=VHf3266t; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf24.hostedemail.com: domain of dakr@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=dakr@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1721665883; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ZyLBWxgGE+VHmKuCq9HdRWltLszPSR1mi+zC2HfQJf0=; b=xJY9U61ylWfWO3596NQ2756UT9FduDUnLHt1oWxzTeIKsAkt0kvSylhu8g3k/X8IPijW+/ mBh97qQ21YM0fnmgFhUDdg+pq/PVtsd2kfr76/iC/m3pIk+rCQnoS01/O/HreHRfbulf3P ZZEMpxc6Ui4WN/rboXZLmJTy/25aGWU= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sin.source.kernel.org (Postfix) with ESMTP id B944BCE0A1B; Mon, 22 Jul 2024 16:31:53 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5BFDBC4AF11; Mon, 22 Jul 2024 16:31:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1721665913; bh=l9EjIfv6+sSlfSlf2706MZEMzHMLAVB6WTLtNdYV9ak=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=VHf3266tBLh5T0zuGEeX8TLkx25+9bXEtG0INx+1LdwxmnN8IjYrw7z+A4VC9EDcC tAOeKJrxempBnDc4SuzAjb6ReaTFHNv4WUJM/152sK4yR1S+k/g7vD1lz+3wn23BPS 6FJ9lymAWKXoBuhb02vspq3xVRMEm0kHUb/FCvRCICIfh1jM9etbv1Ln/tz2SrRQ1z KJZWeY6JHFC0bXtFqWXuwAbM85YkH9ZXiM5A8NORx9dWoRR62Bg2kgj6k2QXfUbP89 mHXRUQ6SK765EV688MbepBms1SrPfcQz8z32kBdreACuyzmbmkmOd3d4ebNlLNMMUI dZ1HQuJcQLxwg== From: Danilo Krummrich To: cl@linux.com, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, urezki@gmail.com, hch@infradead.org, kees@kernel.org, ojeda@kernel.org, wedsonaf@gmail.com, mhocko@kernel.org, mpe@ellerman.id.au, chandan.babu@oracle.com, christian.koenig@amd.com, maz@kernel.org, oliver.upton@linux.dev Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, rust-for-linux@vger.kernel.org, Danilo Krummrich Subject: [PATCH v2 2/2] mm: kvmalloc: align kvrealloc() with krealloc() Date: Mon, 22 Jul 2024 18:29:24 +0200 Message-ID: <20240722163111.4766-3-dakr@kernel.org> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240722163111.4766-1-dakr@kernel.org> References: <20240722163111.4766-1-dakr@kernel.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: BD660180041 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 9cg1oqameftjy7p3heo869di3zcdt93j X-HE-Tag: 1721665916-650797 X-HE-Meta: U2FsdGVkX1/qgHtBQheRlHZZrB695y15dJvchDGzftO/zrWWiakV1Ce04VP61fmoaR59U73VZCkAS+WI8UN+2uw3gxgmAwhNda5PYHk8Lu1S+Lw/Fb8l5LRowpvXwbpgqAs3DDW53YZ8WQ5/swn1/OzBp0Jsuv9gc8FIpuXSolO1VqjWnHTnoutxwrdSK3GHdprcs3j4xAzfEFb/tNXqnBOyddcJRndx3jLcPO6qpMpBm9l+Txm3cRtoulzyGiQhIhxZFspApGkq+m5ug5PTvEcUzCWEMhdwwRwle/ymJ3kGDM53ysdzjaPLNZf/FC61aLrOVKtCi0At/cMerG9QMy/8aUMJHDmUQbMT5IIxSZ9Iys3FHZrQPqhjnRpcwO+axoefC4iZsrpZj4A3t8kkzvtvbdh5F4yDU2yoTklxP/CFxz5d3q3N9k9RN8MV5Ny8Db0I2FjCQOT5qTH62GwPbU1Ci3eZgbscZiCJTSGwwqOr4aQ72n/nIObumxadK6a0XdEVxmRhBp4KimzJE2DtZog7/xzkwvj7fDhvTbxpXQk7AKCR8ECZVTTuscslEa4vMcJftSqBQVFhDXCn4+KMvyUMhANSv364rssTd4nIR7GoklYCIvv5lnaRA+B9XWUyoJ9WLusHm984XHMi6ptyfbDCvYFO9if/teFS24wfqSaC09qhHnC4lv7zdN8D6oo0UbFvmFhbsTn1fdb+DCeOl1RW7mHRXjDfkyQVP61u3Kp3ZHciYxvv03j0J2odenVsZpHj8uxN+sVPe8E7ntROPvl+UsIUflCTWhnjdJCm3iGSyfRec4EMHtGwM1JRRSwmLqXyADOWEKQrbozuQYKTOPEm63FdmcLsYHoRdjIgufUukSaPJ2F+GoxdmgQf8iUAvZyGNNZDwiqHWeY00hLr9wCIKJNWuirCYaWtMU2FsgZ5GODEHFKwkIbg69NftNf3n3H43360pgrHUiPdiEI bF7wBP9d Y7lC9vaDl9ovsd1LNUE5ovblWYXxgvlbhPN2xSVwQjKcBP2KD0a+/JKT0bXNFmVb1ZdFkbgW+Rml9Y0+jHfBIpPYeeBKGpcpNOP0yhuTeTAjKuFUb1dKHa58mSwYGJ6lAji5D8jfE3SiY+NlI5Sw1onTYh3MrTOjOU4iunu6l9XCd+pMQUYysKCW1bQoeSum5yUwL/9onrhm48DNSLVRnmiGnQawpYNLloLfCExTsB0ImRu/n6IPxQYYcfxf/sdjKXrB/1EBqyVt6RpiFx/X6azE6lO1zgXdpmd32BbGbs6gW0YTSGko8YPIXyw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Besides the obvious (and desired) difference between krealloc() and kvrealloc(), there is some inconsistency in their function signatures and behavior: - krealloc() frees the memory when the requested size is zero, whereas kvrealloc() simply returns a pointer to the existing allocation. - krealloc() behaves like kmalloc() if a NULL pointer is passed, whereas kvrealloc() does not accept a NULL pointer at all and, if passed, would fault instead. - krealloc() is self-contained, whereas kvrealloc() relies on the caller to provide the size of the previous allocation. Inconsistent behavior throughout allocation APIs is error prone, hence make kvrealloc() behave like krealloc(), which seems superior in all mentioned aspects. Besides that, implementing kvrealloc() by making use of krealloc() and vrealloc() provides oppertunities to grow (and shrink) allocations more efficiently. For instance, vrealloc() can be optimized to allocate and map additional pages to grow the allocation or unmap and free unused pages to shrink the allocation. Signed-off-by: Danilo Krummrich Acked-by: Vlastimil Babka --- arch/arm64/kvm/nested.c | 1 - arch/powerpc/platforms/pseries/papr-vpd.c | 5 +- drivers/gpu/drm/drm_exec.c | 3 +- fs/xfs/xfs_log_recover.c | 2 +- include/linux/slab.h | 4 +- kernel/resource.c | 3 +- lib/fortify_kunit.c | 3 +- mm/util.c | 89 +++++++++++++++-------- 8 files changed, 66 insertions(+), 44 deletions(-) diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c index de789e0f1ae9..1ff3079aabc9 100644 --- a/arch/arm64/kvm/nested.c +++ b/arch/arm64/kvm/nested.c @@ -62,7 +62,6 @@ int kvm_vcpu_init_nested(struct kvm_vcpu *vcpu) */ num_mmus = atomic_read(&kvm->online_vcpus) * S2_MMU_PER_VCPU; tmp = kvrealloc(kvm->arch.nested_mmus, - size_mul(sizeof(*kvm->arch.nested_mmus), kvm->arch.nested_mmus_size), size_mul(sizeof(*kvm->arch.nested_mmus), num_mmus), GFP_KERNEL_ACCOUNT | __GFP_ZERO); if (!tmp) diff --git a/arch/powerpc/platforms/pseries/papr-vpd.c b/arch/powerpc/platforms/pseries/papr-vpd.c index c29e85db5f35..1574176e3ffc 100644 --- a/arch/powerpc/platforms/pseries/papr-vpd.c +++ b/arch/powerpc/platforms/pseries/papr-vpd.c @@ -156,10 +156,7 @@ static int vpd_blob_extend(struct vpd_blob *blob, const char *data, size_t len) const char *old_ptr = blob->data; char *new_ptr; - new_ptr = old_ptr ? - kvrealloc(old_ptr, old_len, new_len, GFP_KERNEL_ACCOUNT) : - kvmalloc(len, GFP_KERNEL_ACCOUNT); - + new_ptr = kvrealloc(old_ptr, new_len, GFP_KERNEL_ACCOUNT); if (!new_ptr) return -ENOMEM; diff --git a/drivers/gpu/drm/drm_exec.c b/drivers/gpu/drm/drm_exec.c index 2da094bdf8a4..18e366cc4993 100644 --- a/drivers/gpu/drm/drm_exec.c +++ b/drivers/gpu/drm/drm_exec.c @@ -145,8 +145,7 @@ static int drm_exec_obj_locked(struct drm_exec *exec, size_t size = exec->max_objects * sizeof(void *); void *tmp; - tmp = kvrealloc(exec->objects, size, size + PAGE_SIZE, - GFP_KERNEL); + tmp = kvrealloc(exec->objects, size + PAGE_SIZE, GFP_KERNEL); if (!tmp) return -ENOMEM; diff --git a/fs/xfs/xfs_log_recover.c b/fs/xfs/xfs_log_recover.c index 4423dd344239..1997981827fb 100644 --- a/fs/xfs/xfs_log_recover.c +++ b/fs/xfs/xfs_log_recover.c @@ -2128,7 +2128,7 @@ xlog_recover_add_to_cont_trans( old_ptr = item->ri_buf[item->ri_cnt-1].i_addr; old_len = item->ri_buf[item->ri_cnt-1].i_len; - ptr = kvrealloc(old_ptr, old_len, len + old_len, GFP_KERNEL); + ptr = kvrealloc(old_ptr, len + old_len, GFP_KERNEL); if (!ptr) return -ENOMEM; memcpy(&ptr[old_len], dp, len); diff --git a/include/linux/slab.h b/include/linux/slab.h index eb2bf4629157..c9cb42203183 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -841,8 +841,8 @@ kvmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags, int node) #define kvcalloc_node(...) alloc_hooks(kvcalloc_node_noprof(__VA_ARGS__)) #define kvcalloc(...) alloc_hooks(kvcalloc_noprof(__VA_ARGS__)) -extern void *kvrealloc_noprof(const void *p, size_t oldsize, size_t newsize, gfp_t flags) - __realloc_size(3); +void *kvrealloc_noprof(const void *p, size_t size, gfp_t flags) + __realloc_size(2); #define kvrealloc(...) alloc_hooks(kvrealloc_noprof(__VA_ARGS__)) extern void kvfree(const void *addr); diff --git a/kernel/resource.c b/kernel/resource.c index 14777afb0a99..9f747bb7cd03 100644 --- a/kernel/resource.c +++ b/kernel/resource.c @@ -450,8 +450,7 @@ int walk_system_ram_res_rev(u64 start, u64 end, void *arg, /* re-alloc */ struct resource *rams_new; - rams_new = kvrealloc(rams, rams_size * sizeof(struct resource), - (rams_size + 16) * sizeof(struct resource), + rams_new = kvrealloc(rams, (rams_size + 16) * sizeof(struct resource), GFP_KERNEL); if (!rams_new) goto out; diff --git a/lib/fortify_kunit.c b/lib/fortify_kunit.c index f9ad60a9c7bd..ecb638d4cde1 100644 --- a/lib/fortify_kunit.c +++ b/lib/fortify_kunit.c @@ -306,8 +306,7 @@ DEFINE_ALLOC_SIZE_TEST_PAIR(vmalloc) orig = kvmalloc(prev_size, gfp); \ KUNIT_EXPECT_TRUE(test, orig != NULL); \ checker(((expected_pages) * PAGE_SIZE) * 2, \ - kvrealloc(orig, prev_size, \ - ((alloc_pages) * PAGE_SIZE) * 2, gfp), \ + kvrealloc(orig, ((alloc_pages) * PAGE_SIZE) * 2, gfp), \ kvfree(p)); \ } while (0) DEFINE_ALLOC_SIZE_TEST_PAIR(kvmalloc) diff --git a/mm/util.c b/mm/util.c index bc488f0121a7..0ff5898cc6de 100644 --- a/mm/util.c +++ b/mm/util.c @@ -608,6 +608,28 @@ unsigned long vm_mmap(struct file *file, unsigned long addr, } EXPORT_SYMBOL(vm_mmap); +static gfp_t kmalloc_gfp_adjust(gfp_t flags, size_t size) +{ + /* + * We want to attempt a large physically contiguous block first because + * it is less likely to fragment multiple larger blocks and therefore + * contribute to a long term fragmentation less than vmalloc fallback. + * However make sure that larger requests are not too disruptive - no + * OOM killer and no allocation failure warnings as we have a fallback. + */ + if (size > PAGE_SIZE) { + flags |= __GFP_NOWARN; + + if (!(flags & __GFP_RETRY_MAYFAIL)) + flags |= __GFP_NORETRY; + + /* nofail semantic is implemented by the vmalloc fallback */ + flags &= ~__GFP_NOFAIL; + } + + return flags; +} + /** * __kvmalloc_node - attempt to allocate physically contiguous memory, but upon * failure, fall back to non-contiguous (vmalloc) allocation. @@ -627,32 +649,15 @@ EXPORT_SYMBOL(vm_mmap); */ void *__kvmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node) { - gfp_t kmalloc_flags = flags; void *ret; - /* - * We want to attempt a large physically contiguous block first because - * it is less likely to fragment multiple larger blocks and therefore - * contribute to a long term fragmentation less than vmalloc fallback. - * However make sure that larger requests are not too disruptive - no - * OOM killer and no allocation failure warnings as we have a fallback. - */ - if (size > PAGE_SIZE) { - kmalloc_flags |= __GFP_NOWARN; - - if (!(kmalloc_flags & __GFP_RETRY_MAYFAIL)) - kmalloc_flags |= __GFP_NORETRY; - - /* nofail semantic is implemented by the vmalloc fallback */ - kmalloc_flags &= ~__GFP_NOFAIL; - } - - ret = __kmalloc_node_noprof(PASS_BUCKET_PARAMS(size, b), kmalloc_flags, node); - /* * It doesn't really make sense to fallback to vmalloc for sub page * requests */ + ret = __kmalloc_node_noprof(PASS_BUCKET_PARAMS(size, b), + kmalloc_gfp_adjust(flags, size), + node); if (ret || size <= PAGE_SIZE) return ret; @@ -715,18 +720,42 @@ void kvfree_sensitive(const void *addr, size_t len) } EXPORT_SYMBOL(kvfree_sensitive); -void *kvrealloc_noprof(const void *p, size_t oldsize, size_t newsize, gfp_t flags) +/** + * kvrealloc - reallocate memory; contents remain unchanged + * @p: object to reallocate memory for + * @size: the size to reallocate + * @flags: the flags for the page level allocator + * + * The contents of the object pointed to are preserved up to the lesser of the + * new and old size (__GFP_ZERO flag is effectively ignored). + * + * If @p is %NULL, kvrealloc() behaves exactly like kvmalloc(). If @size is 0 + * and @p is not a %NULL pointer, the object pointed to is freed. + * + * Return: pointer to the allocated memory or %NULL in case of error + */ +void *kvrealloc_noprof(const void *p, size_t size, gfp_t flags) { - void *newp; + void *n; + + if (is_vmalloc_addr(p)) + return vrealloc_noprof(p, size, flags); + + n = krealloc_noprof(p, size, kmalloc_gfp_adjust(flags, size)); + if (!n) { + /* We failed to krealloc(), fall back to kvmalloc(). */ + n = kvmalloc_noprof(size, flags); + if (!n) + return NULL; + + if (p) { + /* We already know that `p` is not a vmalloc address. */ + memcpy(n, p, ksize(p)); + kfree(p); + } + } - if (oldsize >= newsize) - return (void *)p; - newp = kvmalloc_noprof(newsize, flags); - if (!newp) - return NULL; - memcpy(newp, p, oldsize); - kvfree(p); - return newp; + return n; } EXPORT_SYMBOL(kvrealloc_noprof);