From patchwork Fri May 31 19:14:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 13682022 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ACB1C17B51D; Fri, 31 May 2024 19:14:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717182899; cv=none; b=ZXoeAaPapy9yYcl5qJh4LbyZu0XjSV9YViBsMcRH1IhrEJSyKIyNuekwGsJvKdBZBNjQPoKPgjtGrXUXcrvnwRX68tg1id3VC89t/K+s+qmiPH9QDxraAJZ+FIYYmugFy5k2f7LjwewkKw+y7SmXHpUZntSvwpgrT19qI5Y3FYs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717182899; c=relaxed/simple; bh=GCCmwO/29MDWHdLt+3VUIvxE5UJuxve2yweYGuL7qiA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=iVihNRSBl7TG+UIsHMyQFc6cw2bYgdueWH/lqK2J3hM33xgS3wgCjW2GwcYDgnO0HBt5pv5LabqeizMN2Y00zZ6cZ3vT/Aewt3oxCUTtHWW3tggkFbvzlhED9OtWzbtpj+ThEqwTwzEDCUFu6GlVbo6svVXIzg2d29UoZeP3mpM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=kcThVp9O; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="kcThVp9O" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5D812C4AF0A; Fri, 31 May 2024 19:14:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1717182899; bh=GCCmwO/29MDWHdLt+3VUIvxE5UJuxve2yweYGuL7qiA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=kcThVp9ODciDqR4t3BBbXTS6sQt/cbQ6XFTUxaxOcLk0CsG9Ctv6ST/2n88UxDg1i cTmfdZrqxOqWGGQr5jpF0d7AyiP3J1Aj5rgbLC8nfsT7EIw7Z4U7NqzNMW+xVh7N5p OsQImzr+tvg427WjiNLLdXg4mPwTJhgtCCTCiHWGC+r1TFZCq6JaSsR8s9KaEIHqWf OBHhQRCU/Cb5tOIWIP7oGBkHZ6k+sAB7d1yJg6JddK+brFv7eiTdaaVZ0RuPDOhYvU 6vztMnE/2hYdUKAqCUrWgEkWRpoGXwe5todNt35PgbRx7Al8iN1/YfFuqTl2oaHGq/ 1ii38BhumuqBw== From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , jvoisin , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, "GONG, Ruiqi" , Xiu Jianfeng , Suren Baghdasaryan , Kent Overstreet , Jann Horn , Matteo Rizzo , Thomas Graf , Herbert Xu , linux-kernel@vger.kernel.org, linux-hardening@vger.kernel.org, netdev@vger.kernel.org Subject: [PATCH v4 1/6] mm/slab: Introduce kmem_buckets typedef Date: Fri, 31 May 2024 12:14:53 -0700 Message-Id: <20240531191458.987345-1-kees@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240531191304.it.853-kees@kernel.org> References: <20240531191304.it.853-kees@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1809; i=kees@kernel.org; h=from:subject; bh=GCCmwO/29MDWHdLt+3VUIvxE5UJuxve2yweYGuL7qiA=; b=owEBbQKS/ZANAwAKAYly9N/cbcAmAcsmYgBmWiGxJyLrOsUa6xU6DBGw+2yp7OUHaMAtvGnjr vnReswagR2JAjMEAAEKAB0WIQSlw/aPIp3WD3I+bhOJcvTf3G3AJgUCZlohsQAKCRCJcvTf3G3A Jod5EACfz20WkxqWJaAiwfbSw0tisbkRIA7NQR5NKbhxr5QTTLMcM49PatlGCuMu/uVIDNgCnCE 632ua5oc0HCyQXf9IfE35ke9Jo/QNgyG9S++26daItquqVgKQW/8Ast7pQagboaHUxQW1NTdCI0 wUhT3zl16pYuO0Vc4sp/ECUxMl7ZzGTGEkNHtxcsq+5BM194IPqUFUeT/DoxgrR0GtpUCojtFyi 8As8UtO0GYv5rG76w2Q+67MOKcVsQNvA4pJ31EvnjPpKYhjFD//n+/gHC2XY0eGckm0hN43T88b y1LlH5ThWVmIKeP1JV7fgv0NCN4WMOZKqW1dlxJEVywCaL+0n9KmVUJ4Jm92aWqv7767SrgcLY3 h0/q5nAsQuoCgHFyi7u9WhVFxqr0j3InRibsVIOQRxImTET8qh6Wtmxv373Sf2atQP8a0xlcPMv VOVZg6X+un864Nh7tQ2+bMWltaqq9X6xIQN0Gun88AfPH1AHnujDjbKV1P4fEkfmvu5E8XgKd+E Av5G8lClHZxbCosNd7q33lZXSjwr3g8zRTFM9A3lGpWI6jPnTpDDsUInP37Ll9haL4zGGxaQZW7 +LIUy00LcQUCdv6Czd8IMansxZ7b7xBd+q6+zUMM/ktgono3dI7fSFrN4K80saSpy5mUEgjzqLE HK9LKzxl7bkm78A== X-Developer-Key: i=kees@kernel.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 Encapsulate the concept of a single set of kmem_caches that are used for the kmalloc size buckets. Redefine kmalloc_caches as an array of these buckets (for the different global cache buckets). Signed-off-by: Kees Cook --- Cc: Vlastimil Babka Cc: Christoph Lameter Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Cc: jvoisin Cc: Andrew Morton Cc: Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: linux-mm@kvack.org --- include/linux/slab.h | 5 +++-- mm/slab_common.c | 3 +-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 7247e217e21b..de2b7209cd05 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -426,8 +426,9 @@ enum kmalloc_cache_type { NR_KMALLOC_TYPES }; -extern struct kmem_cache * -kmalloc_caches[NR_KMALLOC_TYPES][KMALLOC_SHIFT_HIGH + 1]; +typedef struct kmem_cache * kmem_buckets[KMALLOC_SHIFT_HIGH + 1]; + +extern kmem_buckets kmalloc_caches[NR_KMALLOC_TYPES]; /* * Define gfp bits that should not be set for KMALLOC_NORMAL. diff --git a/mm/slab_common.c b/mm/slab_common.c index 1560a1546bb1..e0b1c109bed2 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -653,8 +653,7 @@ static struct kmem_cache *__init create_kmalloc_cache(const char *name, return s; } -struct kmem_cache * -kmalloc_caches[NR_KMALLOC_TYPES][KMALLOC_SHIFT_HIGH + 1] __ro_after_init = +kmem_buckets kmalloc_caches[NR_KMALLOC_TYPES] __ro_after_init = { /* initialization for https://llvm.org/pr42570 */ }; EXPORT_SYMBOL(kmalloc_caches); From patchwork Fri May 31 19:14:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 13682023 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ACA9220DCB; Fri, 31 May 2024 19:14:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717182899; cv=none; b=hfFTwPdeYX/POKx4umyz2DfDQdumEOSMDUjPDIC91EM+xDAx/Gh4kJiyRLTfNjPCHPRm05dNNnB0QEcbGkkBOnzAOqgpr3ieiJAMbwdwkdAsWeg7jvKpp/LejBOR3o04+Efk5Ravf7ZoIDK9QLnQBxRmvL69WMAa4IpX5G9iUGw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717182899; c=relaxed/simple; bh=dfqToh7vJ1YdlsbBNUJ40Ep+NVn/iVSzZcDLrQiUMXs=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=OupuLf9xe021ISE6ays99dThXhCVe6w/5wfn9VXCEPEqz6xjb5n55SYLvXP/coX3+c3elSvVsON6XU6OZEJzDJmfHSMUmIrizvHWlmG5Mgd5RKajMb4cEF8fVweGn//0lm3IwmQ8vVcFWbS/Pate7wMCelcY4S/GjFgGRyw/Uqg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=WHGi1HA+; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="WHGi1HA+" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 557C3C116B1; Fri, 31 May 2024 19:14:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1717182899; bh=dfqToh7vJ1YdlsbBNUJ40Ep+NVn/iVSzZcDLrQiUMXs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=WHGi1HA+mzpuDY95w6H9YHJnw7WoyCFefutajsvDEH22EdiD0jj78sajTrMgn9adg TXDxyJJM4IE3Qdcos5NJb1MV+ZM3Zmu/miiojHyruPwoIB8AcJSNAzFAB0YmyCpRnt +y+8LtkiuXr2iVUb2WcMRmk/cRL8xV4MCcE15GIIxrrkwaoIUTDsvwyQ7lCfb5dyrk TTUPH6rfBXfZmwFnGkpaIlmEm/vNmeA6sXvxDJ+ALSEpp1MyFEKoPLjDEcDAQDB1JX HJKQsARO9fdZ2oh/YYwgUsjLZIRqc/oMGDVREh4MNuK0Jd45N+l89FplmVwbjLA7O3 LA34z61hB+QTw== From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , jvoisin , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-hardening@vger.kernel.org, "GONG, Ruiqi" , Xiu Jianfeng , Suren Baghdasaryan , Kent Overstreet , Jann Horn , Matteo Rizzo , Thomas Graf , Herbert Xu , linux-kernel@vger.kernel.org, netdev@vger.kernel.org Subject: [PATCH v4 2/6] mm/slab: Plumb kmem_buckets into __do_kmalloc_node() Date: Fri, 31 May 2024 12:14:54 -0700 Message-Id: <20240531191458.987345-2-kees@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240531191304.it.853-kees@kernel.org> References: <20240531191304.it.853-kees@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=11282; i=kees@kernel.org; h=from:subject; bh=dfqToh7vJ1YdlsbBNUJ40Ep+NVn/iVSzZcDLrQiUMXs=; b=owEBbQKS/ZANAwAKAYly9N/cbcAmAcsmYgBmWiGxybA7iklkH7CPjAeV9H2xh/8ul0Vb4HEpp dLpSS9ljiSJAjMEAAEKAB0WIQSlw/aPIp3WD3I+bhOJcvTf3G3AJgUCZlohsQAKCRCJcvTf3G3A JtQAD/wPaCCfMzSt/HVD8r+nVIYMHyOAmxIoyIWlpmRaiu9RHSahTTRl+7hs0hlNJQKR0yR1ejv fyc94zIhyG6mTVmGD7qUUlF4Hbts6YianRXMHkq0vy2p1rNHoaRoU9Un/Zl0hrsyrH4d0eh91j7 LCU2gdpF4DztZdQ/79u6bKb5eyCFaGGJL8ouliSPwgXZq4SE+EJ4apzFLQBPa59UA0g30lvS5D2 wJlE+PXHL8j1iq/agqqyilTfX4DkfLEo/U3cGjgRUuxDbnRfA2HNwm1TTGI0lnR8iHsAm2v9KlO sc/ORVvVLcoqzSvBfmfBadt4fAXNZo4IKH/qMEDiPAD5SB4RsCbCjLqi0WkkXF9YkyeHiUpfvGI zF/g8bLCePUzlRMr5XXQyUHFb41aNdcVYaCvWjAFfB3u3C7qqyOpnkWKYnKfg+24/hjK+D3Vocl RbLziaXt2XLOQXp/k1oOsXI6YGugC38gqDa5HaI9c3+oKSmh8Sne4OexY1Qn26BzZyH6sOQ8x5U cfJhSNPWiiiNjDidIHGlZ+lFSioOsARR1/RYOjqcoXyIcH/sGEMymk6MXYa+2xFGdKxB+AkgM5C uRvt799u2/ClGdVRmZZ9qjHwDgFvrTzFI5t/h4xsBms7u8g3JenTGTdJ4PnsoHfQ2ym+j004zfX WRN8dbxsmM8kp2A== X-Developer-Key: i=kees@kernel.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 Introduce CONFIG_SLAB_BUCKETS which provides the infrastructure to support separated kmalloc buckets (in the follow kmem_buckets_create() patches and future codetag-based separation). Since this will provide a mitigation for a very common case of exploits, enable it by default. To be able to choose which buckets to allocate from, make the buckets available to the internal kmalloc interfaces by adding them as the first argument, rather than depending on the buckets being chosen from the fixed set of global buckets. Where the bucket is not available, pass NULL, which means "use the default system kmalloc bucket set" (the prior existing behavior), as implemented in kmalloc_slab(). To avoid adding the extra argument when !CONFIG_SLAB_BUCKETS, only the top-level macros and static inlines use the buckets argument (where they are stripped out and compiled out respectively). The actual extern functions can then been built without the argument, and the internals fall back to the global kmalloc buckets unconditionally. Signed-off-by: Kees Cook --- Cc: Vlastimil Babka Cc: Christoph Lameter Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Cc: jvoisin Cc: Andrew Morton Cc: Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: linux-mm@kvack.org Cc: linux-hardening@vger.kernel.org --- include/linux/slab.h | 34 ++++++++++++++++++++++++++-------- mm/Kconfig | 15 +++++++++++++++ mm/slab.h | 6 ++++-- mm/slab_common.c | 4 ++-- mm/slub.c | 34 ++++++++++++++++++++++++---------- mm/util.c | 2 +- 6 files changed, 72 insertions(+), 23 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index de2b7209cd05..b1165b22cc6f 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -569,8 +569,17 @@ static __always_inline void kfree_bulk(size_t size, void **p) kmem_cache_free_bulk(NULL, size, p); } -void *__kmalloc_node_noprof(size_t size, gfp_t flags, int node) __assume_kmalloc_alignment - __alloc_size(1); +#ifdef CONFIG_SLAB_BUCKETS +void *__kmalloc_buckets_node_noprof(kmem_buckets *b, size_t size, gfp_t flags, int node) + __assume_kmalloc_alignment __alloc_size(2); +# define __kmalloc_node_noprof(b, size, flags, node) \ + __kmalloc_buckets_node_noprof(b, size, flags, node) +#else +void *__kmalloc_buckets_node_noprof(size_t size, gfp_t flags, int node) + __assume_kmalloc_alignment __alloc_size(1); +# define __kmalloc_node_noprof(b, size, flags, node) \ + __kmalloc_buckets_node_noprof(size, flags, node) +#endif #define __kmalloc_node(...) alloc_hooks(__kmalloc_node_noprof(__VA_ARGS__)) void *kmem_cache_alloc_node_noprof(struct kmem_cache *s, gfp_t flags, @@ -679,7 +688,7 @@ static __always_inline __alloc_size(1) void *kmalloc_node_noprof(size_t size, gf kmalloc_caches[kmalloc_type(flags, _RET_IP_)][index], flags, node, size); } - return __kmalloc_node_noprof(size, flags, node); + return __kmalloc_node_noprof(NULL, size, flags, node); } #define kmalloc_node(...) alloc_hooks(kmalloc_node_noprof(__VA_ARGS__)) @@ -730,10 +739,19 @@ static inline __realloc_size(2, 3) void * __must_check krealloc_array_noprof(voi */ #define kcalloc(n, size, flags) kmalloc_array(n, size, (flags) | __GFP_ZERO) -void *kmalloc_node_track_caller_noprof(size_t size, gfp_t flags, int node, - unsigned long caller) __alloc_size(1); +#ifdef CONFIG_SLAB_BUCKETS +void *__kmalloc_node_track_caller_noprof(kmem_buckets *b, size_t size, gfp_t flags, int node, + unsigned long caller) __alloc_size(2); +# define kmalloc_node_track_caller_noprof(b, size, flags, node, caller) \ + __kmalloc_node_track_caller_noprof(b, size, flags, node, caller) +#else +void *__kmalloc_node_track_caller_noprof(size_t size, gfp_t flags, int node, + unsigned long caller) __alloc_size(1); +# define kmalloc_node_track_caller_noprof(b, size, flags, node, caller) \ + __kmalloc_node_track_caller_noprof(size, flags, node, caller) +#endif #define kmalloc_node_track_caller(...) \ - alloc_hooks(kmalloc_node_track_caller_noprof(__VA_ARGS__, _RET_IP_)) + alloc_hooks(kmalloc_node_track_caller_noprof(NULL, __VA_ARGS__, _RET_IP_)) /* * kmalloc_track_caller is a special version of kmalloc that records the @@ -746,7 +764,7 @@ void *kmalloc_node_track_caller_noprof(size_t size, gfp_t flags, int node, #define kmalloc_track_caller(...) kmalloc_node_track_caller(__VA_ARGS__, NUMA_NO_NODE) #define kmalloc_track_caller_noprof(...) \ - kmalloc_node_track_caller_noprof(__VA_ARGS__, NUMA_NO_NODE, _RET_IP_) + kmalloc_node_track_caller_noprof(NULL, __VA_ARGS__, NUMA_NO_NODE, _RET_IP_) static inline __alloc_size(1, 2) void *kmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags, int node) @@ -757,7 +775,7 @@ static inline __alloc_size(1, 2) void *kmalloc_array_node_noprof(size_t n, size_ return NULL; if (__builtin_constant_p(n) && __builtin_constant_p(size)) return kmalloc_node_noprof(bytes, flags, node); - return __kmalloc_node_noprof(bytes, flags, node); + return __kmalloc_node_noprof(NULL, bytes, flags, node); } #define kmalloc_array_node(...) alloc_hooks(kmalloc_array_node_noprof(__VA_ARGS__)) diff --git a/mm/Kconfig b/mm/Kconfig index b4cb45255a54..8c29af7835cc 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -273,6 +273,21 @@ config SLAB_FREELIST_HARDENED sacrifices to harden the kernel slab allocator against common freelist exploit methods. +config SLAB_BUCKETS + bool "Support allocation from separate kmalloc buckets" + default y + depends on !SLUB_TINY + help + Kernel heap attacks frequently depend on being able to create + specifically-sized allocations with user-controlled contents + that will be allocated into the same kmalloc bucket as a + target object. To avoid sharing these allocation buckets, + provide an explicitly separated set of buckets to be used for + user-controlled allocations. This may very slightly increase + memory fragmentation, though in practice it's only a handful + of extra pages since the bulk of user-controlled allocations + are relatively long-lived. + config SLUB_STATS default n bool "Enable performance statistics" diff --git a/mm/slab.h b/mm/slab.h index 5f8f47c5bee0..f459cd338852 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -403,16 +403,18 @@ static inline unsigned int size_index_elem(unsigned int bytes) * KMALLOC_MAX_CACHE_SIZE and the caller must check that. */ static inline struct kmem_cache * -kmalloc_slab(size_t size, gfp_t flags, unsigned long caller) +kmalloc_slab(kmem_buckets *b, size_t size, gfp_t flags, unsigned long caller) { unsigned int index; + if (!b) + b = &kmalloc_caches[kmalloc_type(flags, caller)]; if (size <= 192) index = kmalloc_size_index[size_index_elem(size)]; else index = fls(size - 1); - return kmalloc_caches[kmalloc_type(flags, caller)][index]; + return (*b)[index]; } gfp_t kmalloc_fix_flags(gfp_t flags); diff --git a/mm/slab_common.c b/mm/slab_common.c index e0b1c109bed2..b5c879fa66bc 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -702,7 +702,7 @@ size_t kmalloc_size_roundup(size_t size) * The flags don't matter since size_index is common to all. * Neither does the caller for just getting ->object_size. */ - return kmalloc_slab(size, GFP_KERNEL, 0)->object_size; + return kmalloc_slab(NULL, size, GFP_KERNEL, 0)->object_size; } /* Above the smaller buckets, size is a multiple of page size. */ @@ -1179,7 +1179,7 @@ __do_krealloc(const void *p, size_t new_size, gfp_t flags) return (void *)p; } - ret = kmalloc_node_track_caller_noprof(new_size, flags, NUMA_NO_NODE, _RET_IP_); + ret = kmalloc_node_track_caller_noprof(NULL, new_size, flags, NUMA_NO_NODE, _RET_IP_); if (ret && p) { /* Disable KASAN checks as the object's redzone is accessed. */ kasan_disable_current(); diff --git a/mm/slub.c b/mm/slub.c index 0809760cf789..ec682a325abe 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4099,7 +4099,7 @@ void *kmalloc_large_node_noprof(size_t size, gfp_t flags, int node) EXPORT_SYMBOL(kmalloc_large_node_noprof); static __always_inline -void *__do_kmalloc_node(size_t size, gfp_t flags, int node, +void *__do_kmalloc_node(kmem_buckets *b, size_t size, gfp_t flags, int node, unsigned long caller) { struct kmem_cache *s; @@ -4115,7 +4115,7 @@ void *__do_kmalloc_node(size_t size, gfp_t flags, int node, if (unlikely(!size)) return ZERO_SIZE_PTR; - s = kmalloc_slab(size, flags, caller); + s = kmalloc_slab(b, size, flags, caller); ret = slab_alloc_node(s, NULL, flags, node, caller, size); ret = kasan_kmalloc(s, ret, size, flags); @@ -4123,24 +4123,38 @@ void *__do_kmalloc_node(size_t size, gfp_t flags, int node, return ret; } -void *__kmalloc_node_noprof(size_t size, gfp_t flags, int node) +#ifdef CONFIG_SLAB_BUCKETS +# define __do_kmalloc_buckets_node(b, size, flags, node, caller) \ + __do_kmalloc_node(b, size, flags, node, caller) +void *__kmalloc_buckets_node_noprof(kmem_buckets *b, size_t size, gfp_t flags, int node) +#else +# define __do_kmalloc_buckets_node(b, size, flags, node, caller) \ + __do_kmalloc_node(NULL, size, flags, node, caller) +void *__kmalloc_buckets_node_noprof(size_t size, gfp_t flags, int node) +#endif { - return __do_kmalloc_node(size, flags, node, _RET_IP_); + return __do_kmalloc_buckets_node(b, size, flags, node, _RET_IP_); } -EXPORT_SYMBOL(__kmalloc_node_noprof); +EXPORT_SYMBOL(__kmalloc_buckets_node_noprof); void *__kmalloc_noprof(size_t size, gfp_t flags) { - return __do_kmalloc_node(size, flags, NUMA_NO_NODE, _RET_IP_); + return __do_kmalloc_buckets_node(NULL, size, flags, NUMA_NO_NODE, _RET_IP_); } EXPORT_SYMBOL(__kmalloc_noprof); -void *kmalloc_node_track_caller_noprof(size_t size, gfp_t flags, - int node, unsigned long caller) +#ifdef CONFIG_SLAB_BUCKETS +void *__kmalloc_node_track_caller_noprof(kmem_buckets *b, size_t size, gfp_t flags, + int node, unsigned long caller) +#else +void *__kmalloc_node_track_caller_noprof(size_t size, gfp_t flags, + int node, unsigned long caller) +#endif { - return __do_kmalloc_node(size, flags, node, caller); + return __do_kmalloc_buckets_node(b, size, flags, node, caller); + } -EXPORT_SYMBOL(kmalloc_node_track_caller_noprof); +EXPORT_SYMBOL(__kmalloc_node_track_caller_noprof); void *kmalloc_trace_noprof(struct kmem_cache *s, gfp_t gfpflags, size_t size) { diff --git a/mm/util.c b/mm/util.c index c9e519e6811f..80430e5ba981 100644 --- a/mm/util.c +++ b/mm/util.c @@ -128,7 +128,7 @@ void *kmemdup_noprof(const void *src, size_t len, gfp_t gfp) { void *p; - p = kmalloc_node_track_caller_noprof(len, gfp, NUMA_NO_NODE, _RET_IP_); + p = kmalloc_node_track_caller_noprof(NULL, len, gfp, NUMA_NO_NODE, _RET_IP_); if (p) memcpy(p, src, len); return p; From patchwork Fri May 31 19:14:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 13682024 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ACBA417C22E; Fri, 31 May 2024 19:14:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717182899; cv=none; b=f0Z5Uua1+fVcR0yk0ojcLxHmjWFZYzuOIHGZLGqBbvwF+XXu241Y5I5k2ZYTzzoyeA1apIfWLGLaWTb65Gry9CsTjMXqVsd4XluiAqdHy0uYRTqyGI+ukChx+JIUBoxramIGjGss0CObdJ3Hx0/vY+bnNGiC42UW8+4TnjSuH+M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717182899; c=relaxed/simple; bh=SEppnjIqapMPAdV3gOtlWmc+MgaX+XeR2G6jWLuYp7Y=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=lT88HfHxhhNGyseRbncISRqEOeZbtBPv95A5NQKXnh9wGOiW/Dmv2rFpT+VwsJ2EBz930R+K73s+t5WpSMVm19KkroZuZsKD3/To/+0q7yQnKKpLaX7UxhQKo46LPqcicqKM/Br7SeA0C8FpjI2956s6JZngw72qMn3miHFUeqQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=o2OPMeYn; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="o2OPMeYn" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5FDCAC4AF0B; Fri, 31 May 2024 19:14:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1717182899; bh=SEppnjIqapMPAdV3gOtlWmc+MgaX+XeR2G6jWLuYp7Y=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=o2OPMeYnvF/RFYKaf/zp7Z8RM2X/F+vCy9GQY8I0Xac+aLT6YtbqxYij5Ro2zWi+a MN9VGaIrehneZvdwFtmr4oRoqYDRm2KSiJtIsuhOObVx1I4PNbOB+Ouoa3tqEvWFDA HSO5HevkfkfN6er7AJp9RlHES2dgJa2zpAhykipseObxwNudHFe0HalIiGVTKVHnmi AqpKV4f8T8pSoz5kt0RKKbLDLOr9OupC6Q7Y7HM7+cbnWHXwXOBdD6b53spXB6kriD 6iSMpHTPLoSPE13x9RlwVt49lrgdHsnOwM4iDFAdAiH52QtrgFm7zUF5eHwg63Ugcm gwHZ+zd6FxRTw== From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , jvoisin , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, "GONG, Ruiqi" , Xiu Jianfeng , Suren Baghdasaryan , Kent Overstreet , Jann Horn , Matteo Rizzo , Thomas Graf , Herbert Xu , linux-kernel@vger.kernel.org, linux-hardening@vger.kernel.org, netdev@vger.kernel.org Subject: [PATCH v4 3/6] mm/slab: Introduce kvmalloc_buckets_node() that can take kmem_buckets argument Date: Fri, 31 May 2024 12:14:55 -0700 Message-Id: <20240531191458.987345-3-kees@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240531191304.it.853-kees@kernel.org> References: <20240531191304.it.853-kees@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=5214; i=kees@kernel.org; h=from:subject; bh=SEppnjIqapMPAdV3gOtlWmc+MgaX+XeR2G6jWLuYp7Y=; b=owEBbQKS/ZANAwAKAYly9N/cbcAmAcsmYgBmWiGxmv39xi7SHjeU5Qh9TUaZ7uKF3EFTYjEdf tCKJ2uSBP+JAjMEAAEKAB0WIQSlw/aPIp3WD3I+bhOJcvTf3G3AJgUCZlohsQAKCRCJcvTf3G3A Jr4cEACEhuBwLvPpMEMM6sggZmXcHry4TE03jPbQd9Mxmy5fFK55DUXqOjb7y21q5sv4rW+FYr+ cxS5RUtfUdTgJDLUlLXIFYuGfkjbVWyf0/UO1wjWEFajzvLZHhctDIxhDXOmz1eNdgiETeCuZy+ 0tK4LLZ7icrZDV64sHkxFdLp+QjgcG9dcn9ddPD+UMyzaehfifqCmbKPTSGQpVlWPDUz/lcu1L9 TpM1bUWcFosoEEoyPwiwzFFGSiB1K2FYqkZduo+6FElV+chVVesLAlL07MVW08zhnM5amAHBtX2 miRaeDPfZ7xIMoweuL4Vvg48VVwgnvfGJH9uu2AXACHy+sUPeUkzpqIeAUbr8pdxPHMbJ7R5p6N IOfEMRRT3t5Sja0RVGPoetNdjckFemnbuYPHydOPHxLEvx2yijFMqhzA21K1v6cp/zULxeNu6/a kAbSOq4OofTYQyu++5F1OmY4bOOeEcEWkLHmC/ZNSYxJI+PudtmIO8QYJSZcWQhhEFbGz2KivPz cdEOaIglVTuJPLq1Jwd6gg19ZqOdrKnyU3QWpKIbkR7YOHpHoc1iDMMIz+Rlg6pXcrnWBlrQeAX ggr1unJKL/WqIBTqG9zmyEyMw39wFI8PexEG8KeNryf3AdYLbv1kgoz2EMLNsURDKu2MjhCcplH x/xz3swVnXdautw== X-Developer-Key: i=kees@kernel.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 Plumb kmem_buckets arguments through kvmalloc_node_noprof() so it is possible to provide an API to perform kvmalloc-style allocations with a particular set of buckets. Introduce kvmalloc_buckets_node() that takes a kmem_buckets argument. Signed-off-by: Kees Cook --- Cc: Vlastimil Babka Cc: Christoph Lameter Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Cc: jvoisin Cc: Andrew Morton Cc: Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: linux-mm@kvack.org --- include/linux/slab.h | 19 +++++++++++++++---- lib/rhashtable.c | 2 +- mm/util.c | 13 +++++++++---- 3 files changed, 25 insertions(+), 9 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index b1165b22cc6f..8853c6eb20b4 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -799,11 +799,22 @@ static inline __alloc_size(1) void *kzalloc_noprof(size_t size, gfp_t flags) #define kzalloc(...) alloc_hooks(kzalloc_noprof(__VA_ARGS__)) #define kzalloc_node(_size, _flags, _node) kmalloc_node(_size, (_flags)|__GFP_ZERO, _node) -extern void *kvmalloc_node_noprof(size_t size, gfp_t flags, int node) __alloc_size(1); -#define kvmalloc_node(...) alloc_hooks(kvmalloc_node_noprof(__VA_ARGS__)) +#ifdef CONFIG_SLAB_BUCKETS +extern void *kvmalloc_buckets_node_noprof(kmem_buckets *b, size_t size, gfp_t flags, int node) + __alloc_size(2); +# define kvmalloc_node_noprof(b, size, flags, node) \ + kvmalloc_buckets_node_noprof(b, size, flags, node) +#else +extern void *kvmalloc_buckets_node_noprof(size_t size, gfp_t flags, int node) + __alloc_size(1); +# define kvmalloc_node_noprof(b, size, flags, node) \ + kvmalloc_buckets_node_noprof(size, flags, node) +#endif +#define kvmalloc_buckets_node(...) alloc_hooks(kvmalloc_node_noprof(__VA_ARGS__)) +#define kvmalloc_node(...) kvmalloc_buckets_node(NULL, __VA_ARGS__) #define kvmalloc(_size, _flags) kvmalloc_node(_size, _flags, NUMA_NO_NODE) -#define kvmalloc_noprof(_size, _flags) kvmalloc_node_noprof(_size, _flags, NUMA_NO_NODE) +#define kvmalloc_noprof(_size, _flags) kvmalloc_node_noprof(NULL, _size, _flags, NUMA_NO_NODE) #define kvzalloc(_size, _flags) kvmalloc(_size, (_flags)|__GFP_ZERO) #define kvzalloc_node(_size, _flags, _node) kvmalloc_node(_size, (_flags)|__GFP_ZERO, _node) @@ -816,7 +827,7 @@ kvmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags, int node) if (unlikely(check_mul_overflow(n, size, &bytes))) return NULL; - return kvmalloc_node_noprof(bytes, flags, node); + return kvmalloc_node_noprof(NULL, bytes, flags, node); } #define kvmalloc_array_noprof(...) kvmalloc_array_node_noprof(__VA_ARGS__, NUMA_NO_NODE) diff --git a/lib/rhashtable.c b/lib/rhashtable.c index dbbed19f8fff..ef0f496e4aed 100644 --- a/lib/rhashtable.c +++ b/lib/rhashtable.c @@ -184,7 +184,7 @@ static struct bucket_table *bucket_table_alloc(struct rhashtable *ht, static struct lock_class_key __key; tbl = alloc_hooks_tag(ht->alloc_tag, - kvmalloc_node_noprof(struct_size(tbl, buckets, nbuckets), + kvmalloc_node_noprof(NULL, struct_size(tbl, buckets, nbuckets), gfp|__GFP_ZERO, NUMA_NO_NODE)); size = nbuckets; diff --git a/mm/util.c b/mm/util.c index 80430e5ba981..53f7fc5912bd 100644 --- a/mm/util.c +++ b/mm/util.c @@ -593,9 +593,11 @@ unsigned long vm_mmap(struct file *file, unsigned long addr, } EXPORT_SYMBOL(vm_mmap); +#ifdef CONFIG_SLAB_BUCKETS /** - * kvmalloc_node - attempt to allocate physically contiguous memory, but upon + * kvmalloc_buckets_node_noprof - attempt to allocate physically contiguous memory, but upon * failure, fall back to non-contiguous (vmalloc) allocation. + * @b: which set of kmalloc buckets to allocate from. * @size: size of the request. * @flags: gfp mask for the allocation - must be compatible (superset) with GFP_KERNEL. * @node: numa node to allocate from @@ -609,7 +611,10 @@ EXPORT_SYMBOL(vm_mmap); * * Return: pointer to the allocated memory of %NULL in case of failure */ -void *kvmalloc_node_noprof(size_t size, gfp_t flags, int node) +void *kvmalloc_buckets_node_noprof(kmem_buckets *b, size_t size, gfp_t flags, int node) +#else +void *kvmalloc_buckets_node_noprof(size_t size, gfp_t flags, int node) +#endif { gfp_t kmalloc_flags = flags; void *ret; @@ -631,7 +636,7 @@ void *kvmalloc_node_noprof(size_t size, gfp_t flags, int node) kmalloc_flags &= ~__GFP_NOFAIL; } - ret = kmalloc_node_noprof(size, kmalloc_flags, node); + ret = __kmalloc_node_noprof(b, size, kmalloc_flags, node); /* * It doesn't really make sense to fallback to vmalloc for sub page @@ -660,7 +665,7 @@ void *kvmalloc_node_noprof(size_t size, gfp_t flags, int node) flags, PAGE_KERNEL, VM_ALLOW_HUGE_VMAP, node, __builtin_return_address(0)); } -EXPORT_SYMBOL(kvmalloc_node_noprof); +EXPORT_SYMBOL(kvmalloc_buckets_node_noprof); /** * kvfree() - Free memory. From patchwork Fri May 31 19:14:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 13682025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ACB6017C223; Fri, 31 May 2024 19:14:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717182899; cv=none; b=R58pE8utdr/kActYAhvpmSWthGiGjBdSzDZO9xLcJQF5+sqYmvI2CC6UDbU9vhD9+jjxwJjFOzk0pf/EP/ZjHfoZSa4Wmcywb6/LR3PZM8jHf0hBDxe866mqIFmU8a50FtcmYFk10NI6FQyOKT6ppXQ7GnlN9G3Yx/oAt8hSd+c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717182899; c=relaxed/simple; bh=XkviX316F0sKgldrjJOcJuBh0F8cjTJAd80cswOqbm0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=VQ0nQePeKPjZHCML/FEs+Dz8Ax5XCH6FV0Ef09uYEpuHU5s5v9q7Mq6HDQehpioMNaiGS4xpLuk6E+NRP2aAFXJ/441cv70yTGnXeMxeXQtP3V0upIpA3GCn7jNHhOyI5asd+Qn8BJUL/2TQJr1jjLxLGtGTMkXPo+9PtEX2e3g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=DHyvid1a; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="DHyvid1a" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 58D6DC2BD10; Fri, 31 May 2024 19:14:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1717182899; bh=XkviX316F0sKgldrjJOcJuBh0F8cjTJAd80cswOqbm0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DHyvid1au2bNZGaV6rOixhLrq7MOiYrCabUnzi41Dq7pBGjPqlrp75IbJCIY5pV1Z BEm+D3cxHrLXZ6YiK+11FkthPYzaZXvGMGzmn081Ctt/wSWCsyKbSmE6XUrZ7z87eb N6IBJnM/Xv6hLwqjIuh9H0aihfk+/hmR7CY65Qv1zbDJ9BtEqoTj6H1Z//O+JerERB JoRXwfHXZ0u9sWPlaPHEpPTrnoSduKydpPnSfUWAqz0zlSHm+I7LYMO0l7f7H6U/A8 V2OF+fc6wwDMCbv1IZKXPIBB5tKaSAMcncdfuRdhNH+FQDjrHVxfX+LB7pR9DvRDEU 7GbSSSOQzhbHg== From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , jvoisin , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, "GONG, Ruiqi" , Xiu Jianfeng , Suren Baghdasaryan , Kent Overstreet , Jann Horn , Matteo Rizzo , Thomas Graf , Herbert Xu , linux-kernel@vger.kernel.org, linux-hardening@vger.kernel.org, netdev@vger.kernel.org Subject: [PATCH v4 4/6] mm/slab: Introduce kmem_buckets_create() and family Date: Fri, 31 May 2024 12:14:56 -0700 Message-Id: <20240531191458.987345-4-kees@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240531191304.it.853-kees@kernel.org> References: <20240531191304.it.853-kees@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=7297; i=kees@kernel.org; h=from:subject; bh=XkviX316F0sKgldrjJOcJuBh0F8cjTJAd80cswOqbm0=; b=owEBbAKT/ZANAwAKAYly9N/cbcAmAcsmYgBmWiGxnRjACCy8SltAtPJOUgvzuMWkk/lDRtsU9 8AFAnMMLhKJAjIEAAEKAB0WIQSlw/aPIp3WD3I+bhOJcvTf3G3AJgUCZlohsQAKCRCJcvTf3G3A JmRFD/Yw7w25q4idmyi+jo6FRdhQJPYOuBOFidTy9RvWqBz1zWPQuzhkSUGAXjN/ByB2Y58w1JG sv0e4eT7HMslHpN1JuKhEYglPdnqig7jCvw7zR7tMHIMYuB58h6pSw5CdMrElEYWfHUnI1NXvfI bTijuUKytN0+0E/2fUVW//rL53sD7JuG3txA1pu9nUoyBS7r+ogR6m7HgXTsU/bHV6Fa+ZRNa9D AzGRLS8Z3SS1WuMVKTDu7oZuJICWhmUQxMd59lcHarzVLpgZEj9cFH/UHAaATCZfYWKV7FuzxdT CsQcqCFrSQZdC/LCPo2nybSbW7rbispbqmN4kfbseeL6nyL5EEQUr9pffGRjOzRb1QMD0nc77Fn 5WuLA8Yo5/s1fnxsFJhVva/+CsSg05hAxcY+v51ovPWU12jhGUZ6clUUCxfAN2R4YjGErfciFUs WUnvRKK2whpyvkSSaoPCjpd7ADLtoyjHvyS3+FItuX1VBYeYdQr/TjOQ+TewIdE84mfagZs+4pU tc5TQVK8DqFOlTgbn01Dg+IJXu2kMn8pGt2bPeoUJbkd/Z4BpBFOggRgzv9bqVIlRKL5yQPQIoz pCtUfQX+Xprqna3AI6wAJhX+GZqNwUz870Dh+ucmRCkyTg1dJAfpUAbpVesEAuqmhDRRb83mU0P qHABF5VsQA6PM X-Developer-Key: i=kees@kernel.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 Dedicated caches are available for fixed size allocations via kmem_cache_alloc(), but for dynamically sized allocations there is only the global kmalloc API's set of buckets available. This means it isn't possible to separate specific sets of dynamically sized allocations into a separate collection of caches. This leads to a use-after-free exploitation weakness in the Linux kernel since many heap memory spraying/grooming attacks depend on using userspace-controllable dynamically sized allocations to collide with fixed size allocations that end up in same cache. While CONFIG_RANDOM_KMALLOC_CACHES provides a probabilistic defense against these kinds of "type confusion" attacks, including for fixed same-size heap objects, we can create a complementary deterministic defense for dynamically sized allocations that are directly user controlled. Addressing these cases is limited in scope, so isolating these kinds of interfaces will not become an unbounded game of whack-a-mole. For example, many pass through memdup_user(), making isolation there very effective. In order to isolate user-controllable dynamically-sized allocations from the common system kmalloc allocations, introduce kmem_buckets_create(), which behaves like kmem_cache_create(). Introduce kmem_buckets_alloc(), which behaves like kmem_cache_alloc(). Introduce kmem_buckets_alloc_track_caller() for where caller tracking is needed. Introduce kmem_buckets_valloc() for cases where vmalloc fallback is needed. This can also be used in the future to extend allocation profiling's use of code tagging to implement per-caller allocation cache isolation[1] even for dynamic allocations. Memory allocation pinning[2] is still needed to plug the Use-After-Free cross-allocator weakness, but that is an existing and separate issue which is complementary to this improvement. Development continues for that feature via the SLAB_VIRTUAL[3] series (which could also provide guard pages -- another complementary improvement). Link: https://lore.kernel.org/lkml/202402211449.401382D2AF@keescook [1] Link: https://googleprojectzero.blogspot.com/2021/10/how-simple-linux-kernel-memory.html [2] Link: https://lore.kernel.org/lkml/20230915105933.495735-1-matteorizzo@google.com/ [3] Signed-off-by: Kees Cook --- Cc: Vlastimil Babka Cc: Christoph Lameter Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Cc: jvoisin Cc: Andrew Morton Cc: Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: linux-mm@kvack.org --- include/linux/slab.h | 12 +++++++ mm/slab_common.c | 80 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 92 insertions(+) diff --git a/include/linux/slab.h b/include/linux/slab.h index 8853c6eb20b4..b48c50d90aae 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -552,6 +552,11 @@ void *kmem_cache_alloc_lru_noprof(struct kmem_cache *s, struct list_lru *lru, void kmem_cache_free(struct kmem_cache *s, void *objp); +kmem_buckets *kmem_buckets_create(const char *name, unsigned int align, + slab_flags_t flags, + unsigned int useroffset, unsigned int usersize, + void (*ctor)(void *)); + /* * Bulk allocation and freeing operations. These are accelerated in an * allocator specific way to avoid taking locks repeatedly or building @@ -675,6 +680,12 @@ static __always_inline __alloc_size(1) void *kmalloc_noprof(size_t size, gfp_t f } #define kmalloc(...) alloc_hooks(kmalloc_noprof(__VA_ARGS__)) +#define kmem_buckets_alloc(_b, _size, _flags) \ + alloc_hooks(__kmalloc_node_noprof(_b, _size, _flags, NUMA_NO_NODE)) + +#define kmem_buckets_alloc_track_caller(_b, _size, _flags) \ + alloc_hooks(kmalloc_node_track_caller_noprof(_b, _size, _flags, NUMA_NO_NODE, _RET_IP_)) + static __always_inline __alloc_size(1) void *kmalloc_node_noprof(size_t size, gfp_t flags, int node) { if (__builtin_constant_p(size) && size) { @@ -818,6 +829,7 @@ extern void *kvmalloc_buckets_node_noprof(size_t size, gfp_t flags, int node) #define kvzalloc(_size, _flags) kvmalloc(_size, (_flags)|__GFP_ZERO) #define kvzalloc_node(_size, _flags, _node) kvmalloc_node(_size, (_flags)|__GFP_ZERO, _node) +#define kmem_buckets_valloc(_b, _size, _flags) kvmalloc_buckets_node(_b, _size, _flags, NUMA_NO_NODE) static inline __alloc_size(1, 2) void * kvmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags, int node) diff --git a/mm/slab_common.c b/mm/slab_common.c index b5c879fa66bc..f42a98d368a9 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -392,6 +392,82 @@ kmem_cache_create(const char *name, unsigned int size, unsigned int align, } EXPORT_SYMBOL(kmem_cache_create); +static struct kmem_cache *kmem_buckets_cache __ro_after_init; + +kmem_buckets *kmem_buckets_create(const char *name, unsigned int align, + slab_flags_t flags, + unsigned int useroffset, + unsigned int usersize, + void (*ctor)(void *)) +{ + kmem_buckets *b; + int idx; + + /* + * When the separate buckets API is not built in, just return + * a non-NULL value for the kmem_buckets pointer, which will be + * unused when performing allocations. + */ + if (!IS_ENABLED(CONFIG_SLAB_BUCKETS)) + return ZERO_SIZE_PTR; + + if (WARN_ON(!kmem_buckets_cache)) + return NULL; + + b = kmem_cache_alloc(kmem_buckets_cache, GFP_KERNEL|__GFP_ZERO); + if (WARN_ON(!b)) + return NULL; + + flags |= SLAB_NO_MERGE; + + for (idx = 0; idx < ARRAY_SIZE(kmalloc_caches[KMALLOC_NORMAL]); idx++) { + char *short_size, *cache_name; + unsigned int cache_useroffset, cache_usersize; + unsigned int size; + + if (!kmalloc_caches[KMALLOC_NORMAL][idx]) + continue; + + size = kmalloc_caches[KMALLOC_NORMAL][idx]->object_size; + if (!size) + continue; + + short_size = strchr(kmalloc_caches[KMALLOC_NORMAL][idx]->name, '-'); + if (WARN_ON(!short_size)) + goto fail; + + cache_name = kasprintf(GFP_KERNEL, "%s-%s", name, short_size + 1); + if (WARN_ON(!cache_name)) + goto fail; + + if (useroffset >= size) { + cache_useroffset = 0; + cache_usersize = 0; + } else { + cache_useroffset = useroffset; + cache_usersize = min(size - cache_useroffset, usersize); + } + (*b)[idx] = kmem_cache_create_usercopy(cache_name, size, + align, flags, cache_useroffset, + cache_usersize, ctor); + kfree(cache_name); + if (WARN_ON(!(*b)[idx])) + goto fail; + } + + return b; + +fail: + for (idx = 0; idx < ARRAY_SIZE(kmalloc_caches[KMALLOC_NORMAL]); idx++) { + if ((*b)[idx]) + kmem_cache_destroy((*b)[idx]); + } + kfree(b); + + return NULL; +} +EXPORT_SYMBOL(kmem_buckets_create); + #ifdef SLAB_SUPPORTS_SYSFS /* * For a given kmem_cache, kmem_cache_destroy() should only be called @@ -931,6 +1007,10 @@ void __init create_kmalloc_caches(void) /* Kmalloc array is now usable */ slab_state = UP; + + kmem_buckets_cache = kmem_cache_create("kmalloc_buckets", + sizeof(kmem_buckets), + 0, 0, NULL); } /** From patchwork Fri May 31 19:14:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 13682026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 233D917D8A5; Fri, 31 May 2024 19:14:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717182900; cv=none; b=qTTKNFx8WECDXJDL6G5OdKwXUezhLiI6E2J9oUfGdlPswYpqV3WrQi44TROpfBrjRHWIdQg6lM2CETgfbWq8NNlsYdTX/7z1ewNh667pHt505+Stj4Gs0Oh2JTTHc9VANHSdnm50z+52bfwrnU0lcwEk5jW3PD13Us6Kq53g3gg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717182900; c=relaxed/simple; bh=9A8yq+jsLnEY9XtsifO4pnJhUYQibC3G54xH7WF3Dj0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=gnNWWhONIQ8ZmiBt4Vy8RkcQ9XkMgZv1Z/+7o6C84OVMMxnpiTIEaDVSss9MrmTvjD00KAjpDPWxWiF2m0FlmVkhZ81WHfs536hJARkl/BOHnx8qDrVIe7FCLLg/Ronqm74ZmZYzbjs2Xj0I8mofBzcHw+Gpc86yve3Q3zoVMS0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=AJ94SEMU; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="AJ94SEMU" Received: by smtp.kernel.org (Postfix) with ESMTPSA id BABA8C4AF13; Fri, 31 May 2024 19:14:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1717182899; bh=9A8yq+jsLnEY9XtsifO4pnJhUYQibC3G54xH7WF3Dj0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=AJ94SEMUEAi0G+xM96x4XYoiJDcb0qfXFLRPz0/u++ZCuMBiMP4N+dQZ/5WzMuf0J 7nEYVNuzZRF7r3hwWa0Jsf5KheXxXfycWzjYaRsXudaEtjxhpVB0ikcDRnDFR8iaV9 OZGkA9+rXQcezVxLSrx1tqCyJtoLLdY68VZ0AzQSxYGw2Ztaevj4N+pOUt/iAaovAY rybbbuKLKSr/KXIqwHOIIK5bOhBm+2NSgCmZ6YzUwhS+3fvK/bIEXpEY7pyM6y3HDV Yiq2Mtmr3OSKj/fIskTPFM5frnLmXncLQcHQF+K+K2cMUBIP30k8CRAS7Zd7XWVbL3 Pig4fGkgA24nA== From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , "GONG, Ruiqi" , Xiu Jianfeng , Suren Baghdasaryan , Kent Overstreet , Jann Horn , Matteo Rizzo , jvoisin , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Thomas Graf , Herbert Xu , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-hardening@vger.kernel.org, netdev@vger.kernel.org Subject: [PATCH v4 5/6] ipc, msg: Use dedicated slab buckets for alloc_msg() Date: Fri, 31 May 2024 12:14:57 -0700 Message-Id: <20240531191458.987345-5-kees@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240531191304.it.853-kees@kernel.org> References: <20240531191304.it.853-kees@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2262; i=kees@kernel.org; h=from:subject; bh=9A8yq+jsLnEY9XtsifO4pnJhUYQibC3G54xH7WF3Dj0=; b=owEBbQKS/ZANAwAKAYly9N/cbcAmAcsmYgBmWiGxyTBp2lx8Omk4GJvqG8JlvO5V6TG04TJr5 OpKBK400V+JAjMEAAEKAB0WIQSlw/aPIp3WD3I+bhOJcvTf3G3AJgUCZlohsQAKCRCJcvTf3G3A JjytD/4tYkg8aTTXkEDJQTDS78uMcmGvgti9YYYONm70aRUsgZ2NOCsbfkTf5fxNyTqi9iRQhCG M8mNQcuLRnO0XfhtP8+WYinTIpI/rMHdm3dJQ5qpqud8iyNGvkD9T4vFMs3q+SjzaF6SXt1lefj dotmyKXABhbLGYH3rRKXR6vOAhDbktYauSb/gjvMkYFl/82yOenQTK0bP+ysn2sVOhqCdwaNRlV GVS+Z0DRHZHNzhCwS4iZAhR5BLKdy8ljeD13yC9JmDXJBLdkL6XO5GJFxrixfq92hk3Gvmf1npE CvgNLaa/Q1h0CylSCrHiM7fsji0R1rW2PFjR2I3Dwh8RC5BgPZFW7iKn7dPZKE/JlwNhAFB7P3d odTCg9G87cqbhIuzbfVW873JG4yeVERpzVHT/tZlGxSPH/Gi+qdoGFJouhc9vX0EzRvlybRzGRd yttjTi2+GwRmuToPZIjIeTq3UG0xjeoLSBZOf+yVKJ46jIQmp6kaR9R1fL/j1q4qHchC1JnQl4y ksVMniTRqY1mgqX35gEIb0TbqNneD6WspnMzFKNVSZzkQMsOmlwxbPEnIEC59OcYfkTsScAj6Bu tqc3tfKbSiPqujUwsxw6EIjtkVwoSkgPO+uyzj6qC4EaPBF756njG76W6fn7B38ITUDAU1DDmTG IGdaAR94n/zELiQ== X-Developer-Key: i=kees@kernel.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 The msg subsystem is a common target for exploiting[1][2][3][4][5][6][7] use-after-free type confusion flaws in the kernel for both read and write primitives. Avoid having a user-controlled dynamically-size allocation share the global kmalloc cache by using a separate set of kmalloc buckets via the kmem_buckets API. Link: https://blog.hacktivesecurity.com/index.php/2022/06/13/linux-kernel-exploit-development-1day-case-study/ [1] Link: https://hardenedvault.net/blog/2022-11-13-msg_msg-recon-mitigation-ved/ [2] Link: https://www.willsroot.io/2021/08/corctf-2021-fire-of-salvation-writeup.html [3] Link: https://a13xp0p0v.github.io/2021/02/09/CVE-2021-26708.html [4] Link: https://google.github.io/security-research/pocs/linux/cve-2021-22555/writeup.html [5] Link: https://zplin.me/papers/ELOISE.pdf [6] Link: https://syst3mfailure.io/wall-of-perdition/ [7] Signed-off-by: Kees Cook --- Cc: "GONG, Ruiqi" Cc: Xiu Jianfeng Cc: Suren Baghdasaryan Cc: Kent Overstreet Cc: Jann Horn Cc: Matteo Rizzo Cc: jvoisin --- ipc/msgutil.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/ipc/msgutil.c b/ipc/msgutil.c index d0a0e877cadd..f392f30a057a 100644 --- a/ipc/msgutil.c +++ b/ipc/msgutil.c @@ -42,6 +42,17 @@ struct msg_msgseg { #define DATALEN_MSG ((size_t)PAGE_SIZE-sizeof(struct msg_msg)) #define DATALEN_SEG ((size_t)PAGE_SIZE-sizeof(struct msg_msgseg)) +static kmem_buckets *msg_buckets __ro_after_init; + +static int __init init_msg_buckets(void) +{ + msg_buckets = kmem_buckets_create("msg_msg", 0, SLAB_ACCOUNT, + sizeof(struct msg_msg), + DATALEN_MSG, NULL); + + return 0; +} +subsys_initcall(init_msg_buckets); static struct msg_msg *alloc_msg(size_t len) { @@ -50,7 +61,7 @@ static struct msg_msg *alloc_msg(size_t len) size_t alen; alen = min(len, DATALEN_MSG); - msg = kmalloc(sizeof(*msg) + alen, GFP_KERNEL_ACCOUNT); + msg = kmem_buckets_alloc(msg_buckets, sizeof(*msg) + alen, GFP_KERNEL); if (msg == NULL) return NULL; From patchwork Fri May 31 19:14:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 13682027 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3D29D17DE1D; Fri, 31 May 2024 19:14:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717182900; cv=none; b=WaKzJlZWHVqp8DvmBAZedB/KNJF8Axe0Q4N06u/0FgyGgn+gKIA7FrZ/GW71+a8VR6X3DkkXGBjhkeM3mJK9M5GOESRd6L334GfVIqhEFbWnw3R0IWYpwGOINK3GOvONpY8gxD6o0e+1FHoQDMsNICGIWaJ8AN1G85Rdw2DHhM4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717182900; c=relaxed/simple; bh=Mz67Iw49GR9dHuxCXZEK5DP7jtaPkYbVmfaXomUh4Hg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=X2VBwwyT6chmqpgkNmbCJ2a8Nb0MdB/yIxzPymQ7boXn4mbsdASgy42hUbnBfXD9NQO37v9QjgYUaIIBg0H8MvjtStc/jgHUCFCuRUqaBFjIY+oysDGcz2aRjXsU9VDU45VcpdnTrihrKY0fAjuEU1QlGxfrVjVGdDHzNoKXS5s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=rfluF7TZ; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="rfluF7TZ" Received: by smtp.kernel.org (Postfix) with ESMTPSA id C83B3C4AF14; Fri, 31 May 2024 19:14:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1717182899; bh=Mz67Iw49GR9dHuxCXZEK5DP7jtaPkYbVmfaXomUh4Hg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=rfluF7TZUaUoGLyXwpABguT3qbLtGxJfRuvK1/sobxp2PtxVb1OvM+4f6W0OqHeBZ HVCzHLuNgSXNbiDvOWo9b7sp150CFJMu2u5Um36A0qhgQrBPy7TMnrLGv3uzbjkyag ETb5sbk1uj1K2LerZ6n2j87yVt5dO/HIggUL9fZR7Aul0NbBjm/n43QC1+Hv51nB1k mp09o+nVaHmN5D0x1KPcjGlapgB2ZG4WFgEWl5AlJB9c52+E8mPLywMYlQsclIHxnr 9KCd1BNYNPs9QuW+MPJzXFiKHsSxEdrofPeS5TkNIhK1n3PkfEQEpc0GwQ/T7EUEsr q8HVHOW9Ln5Yw== From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , "GONG, Ruiqi" , Xiu Jianfeng , Suren Baghdasaryan , Kent Overstreet , Jann Horn , Matteo Rizzo , jvoisin , linux-mm@kvack.org, Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Thomas Graf , Herbert Xu , linux-kernel@vger.kernel.org, linux-hardening@vger.kernel.org, netdev@vger.kernel.org Subject: [PATCH v4 6/6] mm/util: Use dedicated slab buckets for memdup_user() Date: Fri, 31 May 2024 12:14:58 -0700 Message-Id: <20240531191458.987345-6-kees@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240531191304.it.853-kees@kernel.org> References: <20240531191304.it.853-kees@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=3128; i=kees@kernel.org; h=from:subject; bh=Mz67Iw49GR9dHuxCXZEK5DP7jtaPkYbVmfaXomUh4Hg=; b=owEBbQKS/ZANAwAKAYly9N/cbcAmAcsmYgBmWiGxTogF+aXNFJapX/Rt8xhEoMFuFnyE/DJdq wqt+/Xyp5mJAjMEAAEKAB0WIQSlw/aPIp3WD3I+bhOJcvTf3G3AJgUCZlohsQAKCRCJcvTf3G3A JjfaD/9HhI1ZsSAtkAsPWK/ezBjLTmg9GWtl4cRE5dSCRoRpeonv9zJFzQSi/z6q8G0WsUdqG3O q3/OwvDkzuj5PVGBcnx18cxE+4jWy8wzjmDguTWCRaey3mtxit/8c4XHxisL60NntcKaMbwh1TN oi5hUUUDQgI71vHIiuoXRO7zD2LY7dCXiUhbTkKyehfUbQRNzSdc/2Pk73vDhtorWRkwDBJmAt0 uJkgJBPYxsUVHdzXRNgA49jdAloxUMLEOmlZE05GQz2kMjakueNT/8zW4hXJ7jhtvqRqOReYkWa CEnpQa2OzgwYHWFfEWJWI7MsyPy4M4h8Wy4r9jHIAPv1ZI/zkUTPZU9UmIpqDWi86FrEhYHDeqP 4CzdIm2QHlf9rFaFB3G8u+OmTO+U4UQPGxK10sZ/JNM5bmYPRvtF136Lz2cZZp6pDXpW5RIA31c pW3P212pGt3TZNiDXGRd6TG/m06XADFHeuA1H2z1/DCG0Pn0C2r1aG4jC/EWrlOH+81naGmWRHk YVXH7L3KauzmbodVlUZRHUYFlOFEtHjHjfnJWGZw1CIhpNZexJHeN8T4/Fl3NWTgqC6GdfKMkWI xvelqd86qCJCCvYtZVCbFr4MHOoTH1oyU+I8dvyKYQlqyqeJi07lpI6JnOevOTL03LyuLUsgq9e +5WiQK2W0zfBIpA== X-Developer-Key: i=kees@kernel.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 Both memdup_user() and vmemdup_user() handle allocations that are regularly used for exploiting use-after-free type confusion flaws in the kernel (e.g. prctl() PR_SET_VMA_ANON_NAME[1] and setxattr[2][3][4] respectively). Since both are designed for contents coming from userspace, it allows for userspace-controlled allocation sizes. Use a dedicated set of kmalloc buckets so these allocations do not share caches with the global kmalloc buckets. After a fresh boot under Ubuntu 23.10, we can see the caches are already in active use: # grep ^memdup /proc/slabinfo memdup_user-8k 4 4 8192 4 8 : ... memdup_user-4k 8 8 4096 8 8 : ... memdup_user-2k 16 16 2048 16 8 : ... memdup_user-1k 0 0 1024 16 4 : ... memdup_user-512 0 0 512 16 2 : ... memdup_user-256 0 0 256 16 1 : ... memdup_user-128 0 0 128 32 1 : ... memdup_user-64 256 256 64 64 1 : ... memdup_user-32 512 512 32 128 1 : ... memdup_user-16 1024 1024 16 256 1 : ... memdup_user-8 2048 2048 8 512 1 : ... memdup_user-192 0 0 192 21 1 : ... memdup_user-96 168 168 96 42 1 : ... Link: https://starlabs.sg/blog/2023/07-prctl-anon_vma_name-an-amusing-heap-spray/ [1] Link: https://duasynt.com/blog/linux-kernel-heap-spray [2] Link: https://etenal.me/archives/1336 [3] Link: https://github.com/a13xp0p0v/kernel-hack-drill/blob/master/drill_exploit_uaf.c [4] Signed-off-by: Kees Cook --- Cc: "GONG, Ruiqi" Cc: Xiu Jianfeng Cc: Suren Baghdasaryan Cc: Kent Overstreet Cc: Jann Horn Cc: Matteo Rizzo Cc: jvoisin Cc: linux-mm@kvack.org --- mm/util.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/mm/util.c b/mm/util.c index 53f7fc5912bd..f30460c82641 100644 --- a/mm/util.c +++ b/mm/util.c @@ -198,6 +198,16 @@ char *kmemdup_nul(const char *s, size_t len, gfp_t gfp) } EXPORT_SYMBOL(kmemdup_nul); +static kmem_buckets *user_buckets __ro_after_init; + +static int __init init_user_buckets(void) +{ + user_buckets = kmem_buckets_create("memdup_user", 0, 0, 0, INT_MAX, NULL); + + return 0; +} +subsys_initcall(init_user_buckets); + /** * memdup_user - duplicate memory region from user space * @@ -211,7 +221,7 @@ void *memdup_user(const void __user *src, size_t len) { void *p; - p = kmalloc_track_caller(len, GFP_USER | __GFP_NOWARN); + p = kmem_buckets_alloc_track_caller(user_buckets, len, GFP_USER | __GFP_NOWARN); if (!p) return ERR_PTR(-ENOMEM); @@ -237,7 +247,7 @@ void *vmemdup_user(const void __user *src, size_t len) { void *p; - p = kvmalloc(len, GFP_USER); + p = kmem_buckets_valloc(user_buckets, len, GFP_USER); if (!p) return ERR_PTR(-ENOMEM);