From patchwork Mon Jul 22 16:29:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Danilo Krummrich X-Patchwork-Id: 13738939 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AEB11C3DA5D for ; Mon, 22 Jul 2024 16:31:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 344D26B0089; Mon, 22 Jul 2024 12:31:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2CD3D6B008A; Mon, 22 Jul 2024 12:31:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 178156B008C; Mon, 22 Jul 2024 12:31:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id E95B76B0089 for ; Mon, 22 Jul 2024 12:31:55 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 2DA39C14F2 for ; Mon, 22 Jul 2024 16:31:55 +0000 (UTC) X-FDA: 82367930190.10.B3812A4 Received: from sin.source.kernel.org (sin.source.kernel.org [145.40.73.55]) by imf20.hostedemail.com (Postfix) with ESMTP id CFB291C002E for ; Mon, 22 Jul 2024 16:31:52 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=apt6UI3X; spf=pass (imf20.hostedemail.com: domain of dakr@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=dakr@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1721665852; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=lG3/KAkEApsl6UplgKKObpNoWCqOzKPnf5F+fwNa6uQ=; b=Ci56CS5cwIRj7JIfOIKAPUQ6ZtgH0CBkOtnLa5kdFqi4QhkdsSOB2sghJ+9WJOG0bNzaAZ 9GJh84OSMZFUbG43w5yldQCpvP+Zry+WRRVYmBC/nOfgrIbOpVo8iEys3Hv2NgTIjkcSMc 1q8RgcwQyy86NOII38WJkIupsLFLH1E= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=apt6UI3X; spf=pass (imf20.hostedemail.com: domain of dakr@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=dakr@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1721665852; a=rsa-sha256; cv=none; b=z4vrg/lwz1Qir5o5u/bniCz0eZqLi3MgjzCIGUY4dbdxyCFPK4pY3ldSHEAvZ9ncT5ZZaL t27402e9Bk+SHLjX5G9x/DhCIjsYOkE7dKC6jduVMNnkWOScbmsO6zXu32ZM/MK/HLgckJ XnUtb2q5SZK9eGqOeM1B1E21QmRBT60= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sin.source.kernel.org (Postfix) with ESMTP id ACB2CCE0B9A; Mon, 22 Jul 2024 16:31:48 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4D35FC116B1; Mon, 22 Jul 2024 16:31:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1721665907; bh=8R5zLx4UelCLfyVht0GJQcdBO0xBkZadv109mekdLLM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=apt6UI3XCpX9famE1RKBq9IxyCaLlbsuNOSSSCetsAo3Ccd6iOmtq2aw/OEgdOV9N yjQK039V0lwMuJluAOKh85GkhNES/hTRB0foU7Qehnd/Isq1dimaVRxVwrvW/PWWH+ g7kjlZ80Xul72QYxRUdKzhYUIht4L8LHGblAOAaA49j6k8iDQ0lE9BviAU8AWExbmp WpWO1NsmDthK2QhGzYIXLmYOJnRvTfIiqXEyNjkPEV/vyaW9nztTV4PoiXDtsAQYI6 BAlHaRVGQFEmdywmM7vomRsNKZWfD3HuRwIyfFlx/gqo+iwJVqIu0u8PfxpYGe6fxG RUdnzyypKuRog== From: Danilo Krummrich To: cl@linux.com, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, urezki@gmail.com, hch@infradead.org, kees@kernel.org, ojeda@kernel.org, wedsonaf@gmail.com, mhocko@kernel.org, mpe@ellerman.id.au, chandan.babu@oracle.com, christian.koenig@amd.com, maz@kernel.org, oliver.upton@linux.dev Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, rust-for-linux@vger.kernel.org, Danilo Krummrich Subject: [PATCH v2 1/2] mm: vmalloc: implement vrealloc() Date: Mon, 22 Jul 2024 18:29:23 +0200 Message-ID: <20240722163111.4766-2-dakr@kernel.org> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240722163111.4766-1-dakr@kernel.org> References: <20240722163111.4766-1-dakr@kernel.org> MIME-Version: 1.0 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: CFB291C002E X-Stat-Signature: y3iaobodmmqabho8rpght6aq9e8ktgf7 X-Rspam-User: X-HE-Tag: 1721665912-783444 X-HE-Meta: U2FsdGVkX18Io3wGjkgqMuGDcCMZhNQ4I1qh80qVnPxNX1B8ADQfGUJf6di092ZfuJuXun9U/08NA2mtNWw8WgD7ITSBARMMN0vZUKxG6826+mvqLmc4v847lyZXXp/hJ/hQ60j2YSSD1rwfY9EmZgDG9rbtQPc3+k5iGXT7sOr4z+uIhmqplUyJhgXGtJMDaEJkMR0ycldOmjLqNu1rctazmXNAHgi5+V+KEXH0R3d27qj2LJ4DxcrxCfeN47MoQs3h/RkBe2zwc+XBzRusdIVXN/YKxNYzn0JL/6rgBCnP7t2CRxPn3UPOiAPvVpalTEhMuW7RKRKDrkIIr+x1H27nwOp2KvpV3pQKAQLNkXt5iN856Dma2XGksE1i70MRY8S879zSN093TrTRUur4ToCpGjWCaCr2g6aVwlGjuOzXsyue1/hWGiKytXBRcVxoKghM55XRclhw4c5p1z7gzDPwoNkV4UrHpUnj6q7XurTfjE7FcjVN6YH9vy5mHgUYTVXuDv84J2AbVvPd0Ji1asOQUkjYyUWsWNkscnH6gy9MWfbp272FsC5IX+Fr6PZrWl1gu/eupZBqcn5lhldrHCUtepy2Zu73MhlNauwlz8fpBBBdVRsmRchbGAT09s01DFW0qQAVdmC1EJhDyI0YvJLaofrPW2gjhsBJjC1z2hGbz/yGpfHwxYL6mL2RysfMtk+8He3tN2A/uWZayH9J7KLZFV/64zJeeiA1LAzY4udFjAwkWMh4wziN5bJv4LM864eLLxmkAdbUxJit74GSjAIjJZ2LlCUVx1K9/rdCYUhsO6B0c+4+2wDDBC3yljSVLSgI/4iEg3aWUk2vl/7rHD02VCEvxwfSVxco9hlWlET2C1ZIPh8h8LdhXZJ3Cijc0lKVV3fBej89tLselCR701omTa5RjGMWXIY3FA4MDOgJH2ab5eVLwAnqP6iVbHsuhpzFrroKbzoH8zuLWo0 EcXgQVJi b17qHnWjLLPrsEwDnMvWvYT8W3d4yvbai5J7chr8aV/Mo65wzHtfcw6flT9WGzmnz7UuRldMrfg1SRl0veA0T2vhvVPoDDKz0ZSi/nbvupg/RFbj+VVgEQgpjW2nURQy73JYMD42R0SpFFng1GSCj286Q5iZvzsgeYFOaqEwFPjQaqEh+mdd6wCeQkfYA7YWSOcjAuiBJJqQizWDWo90TqBmUGFmqe5qhxTF4NfcXLJSIcHNJeMF4CqaD7g9JS9hQwk7AM3V9h0A6VZ6yoz7NM0CN5XST1mNeDLYkZPVc4pNt3+n5FxCLsuz18A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Implement vrealloc() analogous to krealloc(). Currently, krealloc() requires the caller to pass the size of the previous memory allocation, which, instead, should be self-contained. We attempt to fix this in a subsequent patch which, in order to do so, requires vrealloc(). Besides that, we need realloc() functions for kernel allocators in Rust too. With `Vec` or `KVec` respectively, potentially growing (and shrinking) data structures are rather common. Signed-off-by: Danilo Krummrich Acked-by: Vlastimil Babka --- include/linux/vmalloc.h | 4 +++ mm/vmalloc.c | 59 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 63 insertions(+) diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index e4a631ec430b..ad2ce7a6ab7a 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -189,6 +189,10 @@ extern void *__vcalloc_noprof(size_t n, size_t size, gfp_t flags) __alloc_size(1 extern void *vcalloc_noprof(size_t n, size_t size) __alloc_size(1, 2); #define vcalloc(...) alloc_hooks(vcalloc_noprof(__VA_ARGS__)) +void * __must_check vrealloc_noprof(const void *p, size_t size, gfp_t flags) + __realloc_size(2); +#define vrealloc(...) alloc_hooks(vrealloc_noprof(__VA_ARGS__)) + extern void vfree(const void *addr); extern void vfree_atomic(const void *addr); diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 6b783baf12a1..caf032f0bd69 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -4037,6 +4037,65 @@ void *vzalloc_node_noprof(unsigned long size, int node) } EXPORT_SYMBOL(vzalloc_node_noprof); +/** + * vrealloc - reallocate virtually contiguous memory; contents remain unchanged + * @p: object to reallocate memory for + * @size: the size to reallocate + * @flags: the flags for the page level allocator + * + * The contents of the object pointed to are preserved up to the lesser of the + * new and old size (__GFP_ZERO flag is effectively ignored). + * + * If @p is %NULL, vrealloc() behaves exactly like vmalloc(). If @size is 0 and + * @p is not a %NULL pointer, the object pointed to is freed. + * + * Return: pointer to the allocated memory; %NULL if @size is zero or in case of + * failure + */ +void *vrealloc_noprof(const void *p, size_t size, gfp_t flags) +{ + size_t old_size = 0; + void *n; + + if (!size) { + vfree(p); + return NULL; + } + + if (p) { + struct vm_struct *vm; + + vm = find_vm_area(p); + if (unlikely(!vm)) { + WARN(1, "Trying to vrealloc() nonexistent vm area (%p)\n", p); + return NULL; + } + + old_size = get_vm_area_size(vm); + } + + if (size <= old_size) { + /* + * TODO: Shrink the vm_area, i.e. unmap and free unused pages. + * What would be a good heuristic for when to shrink the + * vm_area? + */ + return (void *)p; + } + + /* TODO: Grow the vm_area, i.e. allocate and map additional pages. */ + n = __vmalloc_noprof(size, flags); + if (!n) + return NULL; + + if (p) { + memcpy(n, p, old_size); + vfree(p); + } + + return n; +} + #if defined(CONFIG_64BIT) && defined(CONFIG_ZONE_DMA32) #define GFP_VMALLOC32 (GFP_DMA32 | GFP_KERNEL) #elif defined(CONFIG_64BIT) && defined(CONFIG_ZONE_DMA) From patchwork Mon Jul 22 16:29:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Danilo Krummrich X-Patchwork-Id: 13738940 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6B626C3DA5D for ; Mon, 22 Jul 2024 16:32:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 010166B008C; Mon, 22 Jul 2024 12:32:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EB52E6B0092; Mon, 22 Jul 2024 12:31:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CDFFF6B0093; Mon, 22 Jul 2024 12:31:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id AC5D66B008C for ; Mon, 22 Jul 2024 12:31:59 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 28F69419D8 for ; Mon, 22 Jul 2024 16:31:59 +0000 (UTC) X-FDA: 82367930358.17.C60CA5A Received: from sin.source.kernel.org (sin.source.kernel.org [145.40.73.55]) by imf24.hostedemail.com (Postfix) with ESMTP id BD660180041 for ; Mon, 22 Jul 2024 16:31:56 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=VHf3266t; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf24.hostedemail.com: domain of dakr@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=dakr@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1721665883; a=rsa-sha256; cv=none; b=OmMkeLGqtGvGKp0eP5Ic2MiPUBSJUe0D91eImJ/OVk/JM74OhZXOO3W8rmyr4RAIYltLbC UiXKrOIoHrjc4B06knjet95stgD1uTCyH//A8SSG67HzZsO1uUVzU0Vx3wJGUVtzTFz31G x/NmD2801PxjmzbJZUwHrrrn8ySCe4I= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=VHf3266t; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf24.hostedemail.com: domain of dakr@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=dakr@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1721665883; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ZyLBWxgGE+VHmKuCq9HdRWltLszPSR1mi+zC2HfQJf0=; b=xJY9U61ylWfWO3596NQ2756UT9FduDUnLHt1oWxzTeIKsAkt0kvSylhu8g3k/X8IPijW+/ mBh97qQ21YM0fnmgFhUDdg+pq/PVtsd2kfr76/iC/m3pIk+rCQnoS01/O/HreHRfbulf3P ZZEMpxc6Ui4WN/rboXZLmJTy/25aGWU= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sin.source.kernel.org (Postfix) with ESMTP id B944BCE0A1B; Mon, 22 Jul 2024 16:31:53 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5BFDBC4AF11; Mon, 22 Jul 2024 16:31:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1721665913; bh=l9EjIfv6+sSlfSlf2706MZEMzHMLAVB6WTLtNdYV9ak=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=VHf3266tBLh5T0zuGEeX8TLkx25+9bXEtG0INx+1LdwxmnN8IjYrw7z+A4VC9EDcC tAOeKJrxempBnDc4SuzAjb6ReaTFHNv4WUJM/152sK4yR1S+k/g7vD1lz+3wn23BPS 6FJ9lymAWKXoBuhb02vspq3xVRMEm0kHUb/FCvRCICIfh1jM9etbv1Ln/tz2SrRQ1z KJZWeY6JHFC0bXtFqWXuwAbM85YkH9ZXiM5A8NORx9dWoRR62Bg2kgj6k2QXfUbP89 mHXRUQ6SK765EV688MbepBms1SrPfcQz8z32kBdreACuyzmbmkmOd3d4ebNlLNMMUI dZ1HQuJcQLxwg== From: Danilo Krummrich To: cl@linux.com, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, urezki@gmail.com, hch@infradead.org, kees@kernel.org, ojeda@kernel.org, wedsonaf@gmail.com, mhocko@kernel.org, mpe@ellerman.id.au, chandan.babu@oracle.com, christian.koenig@amd.com, maz@kernel.org, oliver.upton@linux.dev Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, rust-for-linux@vger.kernel.org, Danilo Krummrich Subject: [PATCH v2 2/2] mm: kvmalloc: align kvrealloc() with krealloc() Date: Mon, 22 Jul 2024 18:29:24 +0200 Message-ID: <20240722163111.4766-3-dakr@kernel.org> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240722163111.4766-1-dakr@kernel.org> References: <20240722163111.4766-1-dakr@kernel.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: BD660180041 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 9cg1oqameftjy7p3heo869di3zcdt93j X-HE-Tag: 1721665916-650797 X-HE-Meta: U2FsdGVkX1/qgHtBQheRlHZZrB695y15dJvchDGzftO/zrWWiakV1Ce04VP61fmoaR59U73VZCkAS+WI8UN+2uw3gxgmAwhNda5PYHk8Lu1S+Lw/Fb8l5LRowpvXwbpgqAs3DDW53YZ8WQ5/swn1/OzBp0Jsuv9gc8FIpuXSolO1VqjWnHTnoutxwrdSK3GHdprcs3j4xAzfEFb/tNXqnBOyddcJRndx3jLcPO6qpMpBm9l+Txm3cRtoulzyGiQhIhxZFspApGkq+m5ug5PTvEcUzCWEMhdwwRwle/ymJ3kGDM53ysdzjaPLNZf/FC61aLrOVKtCi0At/cMerG9QMy/8aUMJHDmUQbMT5IIxSZ9Iys3FHZrQPqhjnRpcwO+axoefC4iZsrpZj4A3t8kkzvtvbdh5F4yDU2yoTklxP/CFxz5d3q3N9k9RN8MV5Ny8Db0I2FjCQOT5qTH62GwPbU1Ci3eZgbscZiCJTSGwwqOr4aQ72n/nIObumxadK6a0XdEVxmRhBp4KimzJE2DtZog7/xzkwvj7fDhvTbxpXQk7AKCR8ECZVTTuscslEa4vMcJftSqBQVFhDXCn4+KMvyUMhANSv364rssTd4nIR7GoklYCIvv5lnaRA+B9XWUyoJ9WLusHm984XHMi6ptyfbDCvYFO9if/teFS24wfqSaC09qhHnC4lv7zdN8D6oo0UbFvmFhbsTn1fdb+DCeOl1RW7mHRXjDfkyQVP61u3Kp3ZHciYxvv03j0J2odenVsZpHj8uxN+sVPe8E7ntROPvl+UsIUflCTWhnjdJCm3iGSyfRec4EMHtGwM1JRRSwmLqXyADOWEKQrbozuQYKTOPEm63FdmcLsYHoRdjIgufUukSaPJ2F+GoxdmgQf8iUAvZyGNNZDwiqHWeY00hLr9wCIKJNWuirCYaWtMU2FsgZ5GODEHFKwkIbg69NftNf3n3H43360pgrHUiPdiEI bF7wBP9d Y7lC9vaDl9ovsd1LNUE5ovblWYXxgvlbhPN2xSVwQjKcBP2KD0a+/JKT0bXNFmVb1ZdFkbgW+Rml9Y0+jHfBIpPYeeBKGpcpNOP0yhuTeTAjKuFUb1dKHa58mSwYGJ6lAji5D8jfE3SiY+NlI5Sw1onTYh3MrTOjOU4iunu6l9XCd+pMQUYysKCW1bQoeSum5yUwL/9onrhm48DNSLVRnmiGnQawpYNLloLfCExTsB0ImRu/n6IPxQYYcfxf/sdjKXrB/1EBqyVt6RpiFx/X6azE6lO1zgXdpmd32BbGbs6gW0YTSGko8YPIXyw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Besides the obvious (and desired) difference between krealloc() and kvrealloc(), there is some inconsistency in their function signatures and behavior: - krealloc() frees the memory when the requested size is zero, whereas kvrealloc() simply returns a pointer to the existing allocation. - krealloc() behaves like kmalloc() if a NULL pointer is passed, whereas kvrealloc() does not accept a NULL pointer at all and, if passed, would fault instead. - krealloc() is self-contained, whereas kvrealloc() relies on the caller to provide the size of the previous allocation. Inconsistent behavior throughout allocation APIs is error prone, hence make kvrealloc() behave like krealloc(), which seems superior in all mentioned aspects. Besides that, implementing kvrealloc() by making use of krealloc() and vrealloc() provides oppertunities to grow (and shrink) allocations more efficiently. For instance, vrealloc() can be optimized to allocate and map additional pages to grow the allocation or unmap and free unused pages to shrink the allocation. Signed-off-by: Danilo Krummrich Acked-by: Vlastimil Babka --- arch/arm64/kvm/nested.c | 1 - arch/powerpc/platforms/pseries/papr-vpd.c | 5 +- drivers/gpu/drm/drm_exec.c | 3 +- fs/xfs/xfs_log_recover.c | 2 +- include/linux/slab.h | 4 +- kernel/resource.c | 3 +- lib/fortify_kunit.c | 3 +- mm/util.c | 89 +++++++++++++++-------- 8 files changed, 66 insertions(+), 44 deletions(-) diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c index de789e0f1ae9..1ff3079aabc9 100644 --- a/arch/arm64/kvm/nested.c +++ b/arch/arm64/kvm/nested.c @@ -62,7 +62,6 @@ int kvm_vcpu_init_nested(struct kvm_vcpu *vcpu) */ num_mmus = atomic_read(&kvm->online_vcpus) * S2_MMU_PER_VCPU; tmp = kvrealloc(kvm->arch.nested_mmus, - size_mul(sizeof(*kvm->arch.nested_mmus), kvm->arch.nested_mmus_size), size_mul(sizeof(*kvm->arch.nested_mmus), num_mmus), GFP_KERNEL_ACCOUNT | __GFP_ZERO); if (!tmp) diff --git a/arch/powerpc/platforms/pseries/papr-vpd.c b/arch/powerpc/platforms/pseries/papr-vpd.c index c29e85db5f35..1574176e3ffc 100644 --- a/arch/powerpc/platforms/pseries/papr-vpd.c +++ b/arch/powerpc/platforms/pseries/papr-vpd.c @@ -156,10 +156,7 @@ static int vpd_blob_extend(struct vpd_blob *blob, const char *data, size_t len) const char *old_ptr = blob->data; char *new_ptr; - new_ptr = old_ptr ? - kvrealloc(old_ptr, old_len, new_len, GFP_KERNEL_ACCOUNT) : - kvmalloc(len, GFP_KERNEL_ACCOUNT); - + new_ptr = kvrealloc(old_ptr, new_len, GFP_KERNEL_ACCOUNT); if (!new_ptr) return -ENOMEM; diff --git a/drivers/gpu/drm/drm_exec.c b/drivers/gpu/drm/drm_exec.c index 2da094bdf8a4..18e366cc4993 100644 --- a/drivers/gpu/drm/drm_exec.c +++ b/drivers/gpu/drm/drm_exec.c @@ -145,8 +145,7 @@ static int drm_exec_obj_locked(struct drm_exec *exec, size_t size = exec->max_objects * sizeof(void *); void *tmp; - tmp = kvrealloc(exec->objects, size, size + PAGE_SIZE, - GFP_KERNEL); + tmp = kvrealloc(exec->objects, size + PAGE_SIZE, GFP_KERNEL); if (!tmp) return -ENOMEM; diff --git a/fs/xfs/xfs_log_recover.c b/fs/xfs/xfs_log_recover.c index 4423dd344239..1997981827fb 100644 --- a/fs/xfs/xfs_log_recover.c +++ b/fs/xfs/xfs_log_recover.c @@ -2128,7 +2128,7 @@ xlog_recover_add_to_cont_trans( old_ptr = item->ri_buf[item->ri_cnt-1].i_addr; old_len = item->ri_buf[item->ri_cnt-1].i_len; - ptr = kvrealloc(old_ptr, old_len, len + old_len, GFP_KERNEL); + ptr = kvrealloc(old_ptr, len + old_len, GFP_KERNEL); if (!ptr) return -ENOMEM; memcpy(&ptr[old_len], dp, len); diff --git a/include/linux/slab.h b/include/linux/slab.h index eb2bf4629157..c9cb42203183 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -841,8 +841,8 @@ kvmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags, int node) #define kvcalloc_node(...) alloc_hooks(kvcalloc_node_noprof(__VA_ARGS__)) #define kvcalloc(...) alloc_hooks(kvcalloc_noprof(__VA_ARGS__)) -extern void *kvrealloc_noprof(const void *p, size_t oldsize, size_t newsize, gfp_t flags) - __realloc_size(3); +void *kvrealloc_noprof(const void *p, size_t size, gfp_t flags) + __realloc_size(2); #define kvrealloc(...) alloc_hooks(kvrealloc_noprof(__VA_ARGS__)) extern void kvfree(const void *addr); diff --git a/kernel/resource.c b/kernel/resource.c index 14777afb0a99..9f747bb7cd03 100644 --- a/kernel/resource.c +++ b/kernel/resource.c @@ -450,8 +450,7 @@ int walk_system_ram_res_rev(u64 start, u64 end, void *arg, /* re-alloc */ struct resource *rams_new; - rams_new = kvrealloc(rams, rams_size * sizeof(struct resource), - (rams_size + 16) * sizeof(struct resource), + rams_new = kvrealloc(rams, (rams_size + 16) * sizeof(struct resource), GFP_KERNEL); if (!rams_new) goto out; diff --git a/lib/fortify_kunit.c b/lib/fortify_kunit.c index f9ad60a9c7bd..ecb638d4cde1 100644 --- a/lib/fortify_kunit.c +++ b/lib/fortify_kunit.c @@ -306,8 +306,7 @@ DEFINE_ALLOC_SIZE_TEST_PAIR(vmalloc) orig = kvmalloc(prev_size, gfp); \ KUNIT_EXPECT_TRUE(test, orig != NULL); \ checker(((expected_pages) * PAGE_SIZE) * 2, \ - kvrealloc(orig, prev_size, \ - ((alloc_pages) * PAGE_SIZE) * 2, gfp), \ + kvrealloc(orig, ((alloc_pages) * PAGE_SIZE) * 2, gfp), \ kvfree(p)); \ } while (0) DEFINE_ALLOC_SIZE_TEST_PAIR(kvmalloc) diff --git a/mm/util.c b/mm/util.c index bc488f0121a7..0ff5898cc6de 100644 --- a/mm/util.c +++ b/mm/util.c @@ -608,6 +608,28 @@ unsigned long vm_mmap(struct file *file, unsigned long addr, } EXPORT_SYMBOL(vm_mmap); +static gfp_t kmalloc_gfp_adjust(gfp_t flags, size_t size) +{ + /* + * We want to attempt a large physically contiguous block first because + * it is less likely to fragment multiple larger blocks and therefore + * contribute to a long term fragmentation less than vmalloc fallback. + * However make sure that larger requests are not too disruptive - no + * OOM killer and no allocation failure warnings as we have a fallback. + */ + if (size > PAGE_SIZE) { + flags |= __GFP_NOWARN; + + if (!(flags & __GFP_RETRY_MAYFAIL)) + flags |= __GFP_NORETRY; + + /* nofail semantic is implemented by the vmalloc fallback */ + flags &= ~__GFP_NOFAIL; + } + + return flags; +} + /** * __kvmalloc_node - attempt to allocate physically contiguous memory, but upon * failure, fall back to non-contiguous (vmalloc) allocation. @@ -627,32 +649,15 @@ EXPORT_SYMBOL(vm_mmap); */ void *__kvmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node) { - gfp_t kmalloc_flags = flags; void *ret; - /* - * We want to attempt a large physically contiguous block first because - * it is less likely to fragment multiple larger blocks and therefore - * contribute to a long term fragmentation less than vmalloc fallback. - * However make sure that larger requests are not too disruptive - no - * OOM killer and no allocation failure warnings as we have a fallback. - */ - if (size > PAGE_SIZE) { - kmalloc_flags |= __GFP_NOWARN; - - if (!(kmalloc_flags & __GFP_RETRY_MAYFAIL)) - kmalloc_flags |= __GFP_NORETRY; - - /* nofail semantic is implemented by the vmalloc fallback */ - kmalloc_flags &= ~__GFP_NOFAIL; - } - - ret = __kmalloc_node_noprof(PASS_BUCKET_PARAMS(size, b), kmalloc_flags, node); - /* * It doesn't really make sense to fallback to vmalloc for sub page * requests */ + ret = __kmalloc_node_noprof(PASS_BUCKET_PARAMS(size, b), + kmalloc_gfp_adjust(flags, size), + node); if (ret || size <= PAGE_SIZE) return ret; @@ -715,18 +720,42 @@ void kvfree_sensitive(const void *addr, size_t len) } EXPORT_SYMBOL(kvfree_sensitive); -void *kvrealloc_noprof(const void *p, size_t oldsize, size_t newsize, gfp_t flags) +/** + * kvrealloc - reallocate memory; contents remain unchanged + * @p: object to reallocate memory for + * @size: the size to reallocate + * @flags: the flags for the page level allocator + * + * The contents of the object pointed to are preserved up to the lesser of the + * new and old size (__GFP_ZERO flag is effectively ignored). + * + * If @p is %NULL, kvrealloc() behaves exactly like kvmalloc(). If @size is 0 + * and @p is not a %NULL pointer, the object pointed to is freed. + * + * Return: pointer to the allocated memory or %NULL in case of error + */ +void *kvrealloc_noprof(const void *p, size_t size, gfp_t flags) { - void *newp; + void *n; + + if (is_vmalloc_addr(p)) + return vrealloc_noprof(p, size, flags); + + n = krealloc_noprof(p, size, kmalloc_gfp_adjust(flags, size)); + if (!n) { + /* We failed to krealloc(), fall back to kvmalloc(). */ + n = kvmalloc_noprof(size, flags); + if (!n) + return NULL; + + if (p) { + /* We already know that `p` is not a vmalloc address. */ + memcpy(n, p, ksize(p)); + kfree(p); + } + } - if (oldsize >= newsize) - return (void *)p; - newp = kvmalloc_noprof(newsize, flags); - if (!newp) - return NULL; - memcpy(newp, p, oldsize); - kvfree(p); - return newp; + return n; } EXPORT_SYMBOL(kvrealloc_noprof);