From patchwork Mon Jul 1 19:12:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 13718567 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0E30E15DBD6; Mon, 1 Jul 2024 19:13:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719861185; cv=none; b=M0+2G3PFtXhoRzEJOQiiIyHqitorrldo8AGuTzCZTBr7b/+950Y1MpgXOmGOk7g75W3kCFIHlwH8pmzIVp5P8uT5OiX4mOW7GFoauWEwGnSbrZoqUR9yGCzQ6UkLl3pF0EZs0UTrt8m1G4aNbT0HwRpA3lpUzbwVP0mnQn9dD1o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719861185; c=relaxed/simple; bh=jbXDHZFgMTEqVuIbGpjVHki9R28/opuEqbu8dA+PhS0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=LibInC9MuFu3CA+m+K7cRgO4B2F8K75T7AJ4hTUEkF/0P3l0O4RKy0RMeMkklvod1fJ3EQD1sznybQeQXZgygsKimKFrsV2nX/2V+Ha+1K9dsUPX5ugBbsBRGHGewXzLpCZAJMPxFKSzIqGN6mU6NuqlEYPZovWADArZts2sk5I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=ZWa0R0t0; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="ZWa0R0t0" Received: by smtp.kernel.org (Postfix) with ESMTPSA id A2F71C116B1; Mon, 1 Jul 2024 19:13:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1719861184; bh=jbXDHZFgMTEqVuIbGpjVHki9R28/opuEqbu8dA+PhS0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ZWa0R0t0CPMjfv3nmnvMRYyT+0Li3uUEKN+F63bIWHM5FgA0Zb/H43PrpIm3S1wN+ cSuMAUVGcDq0VPaulTBXEjeP2l0TB9xjG+2UEbX0qsjc7rkvAiBsK7+I89ex7LZgPn gSDpEciVyMFmJKwREVaeNILwLmK/hGuEJS/2ULgNgm5mGv1s6E5j7ocWwtsbwUki+K LpaAGUv8GppMD1wiOwENbbpsCWiN8oDvi+xBdQV6MjW9MpuGnMlORr8FgmERKVods9 T5l7dWXqZMYSVX7ZmrKdnfa13VayjeNzXw3WFOR4TiMnCppfx/DXKOHiSND+6cIfI8 fVtSgUXMY0gnA== From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , "GONG, Ruiqi" , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , jvoisin , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Xiu Jianfeng , Suren Baghdasaryan , Kent Overstreet , Jann Horn , Matteo Rizzo , Thomas Graf , Herbert Xu , "Gustavo A. R. Silva" , Jonathan Corbet , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-hardening@vger.kernel.org, netdev@vger.kernel.org Subject: [PATCH v6 1/6] mm/slab: Introduce kmem_buckets typedef Date: Mon, 1 Jul 2024 12:12:58 -0700 Message-Id: <20240701191304.1283894-1-kees@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240701190152.it.631-kees@kernel.org> References: <20240701190152.it.631-kees@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1406; i=kees@kernel.org; h=from:subject; bh=jbXDHZFgMTEqVuIbGpjVHki9R28/opuEqbu8dA+PhS0=; b=owEBbQKS/ZANAwAKAYly9N/cbcAmAcsmYgBmgv++6T8pqxaDs77JF9cx/6sa+j5FaKjCYQYlt 8Av1OrjnquJAjMEAAEKAB0WIQSlw/aPIp3WD3I+bhOJcvTf3G3AJgUCZoL/vgAKCRCJcvTf3G3A JrDmD/41/LShofUlbbo+pT14hOQzRPFxn19omAUJjuXG/RFbAAjGUsA1LVdU477HcV5iEqq9HwZ yownqKuZBSfRj5jSdbb8lWk+EvNPMJcq/bjuHRBHRVzCIF1dak6lCsCilziR3nDfx0QcIBdh9uM ZfgpyACf83OKtGsnoVrd9PtHkCxzzK4gCsg0cYfwsiMY7lZKRpJkm5Ovl/QeoBlcThOt4kLCnPh VK6h7O4Gi07HVvy/hSMdFhZ/Yg3usgb54keafxYrGlWCSoeYcfA7mccU3Fizf91lu9+yuMRRI+Q B6mnj29oSUPJAd63diYXlmeQHMKkR54f1Ugms2s/O2MuxJ5NN2PeSn3IyL3FbkxJlfCWK0XQ13/ FJoWJFVTtq1PfnOsq/2FR2LpfnPbNPfS/mBp2KVYU1xxcNrOTQ6PmfLN8NZ1dJZnWcctqhlAPvx qAyGmwfuj3emanIzCY/cVU6cLqr6IHmmy9Ejo46YzdG7GTbl1t+0VhPhVwMw+dnIFqJ2tHF6ffU 9S0/hRjxjVGzgpaGI3Rrfzzq/1Rg3PTRcuiXzLrK0K0yBR9YXwRUNuyztoKNbfzA9jitdfiuMBO Ro5XeQMITAZry0tFYeSs/l21CrQNLwzrU/+aIm+fSGZ9J74J2UiWplyhtTCe8DH74SORzWApt9Y atfme+bo8XXfHfg== X-Developer-Key: i=kees@kernel.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 Encapsulate the concept of a single set of kmem_caches that are used for the kmalloc size buckets. Redefine kmalloc_caches as an array of these buckets (for the different global cache buckets). Signed-off-by: Kees Cook --- include/linux/slab.h | 5 +++-- mm/slab_common.c | 3 +-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index ed6bee5ec2b6..8a006fac57c6 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -426,8 +426,9 @@ enum kmalloc_cache_type { NR_KMALLOC_TYPES }; -extern struct kmem_cache * -kmalloc_caches[NR_KMALLOC_TYPES][KMALLOC_SHIFT_HIGH + 1]; +typedef struct kmem_cache * kmem_buckets[KMALLOC_SHIFT_HIGH + 1]; + +extern kmem_buckets kmalloc_caches[NR_KMALLOC_TYPES]; /* * Define gfp bits that should not be set for KMALLOC_NORMAL. diff --git a/mm/slab_common.c b/mm/slab_common.c index 1560a1546bb1..e0b1c109bed2 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -653,8 +653,7 @@ static struct kmem_cache *__init create_kmalloc_cache(const char *name, return s; } -struct kmem_cache * -kmalloc_caches[NR_KMALLOC_TYPES][KMALLOC_SHIFT_HIGH + 1] __ro_after_init = +kmem_buckets kmalloc_caches[NR_KMALLOC_TYPES] __ro_after_init = { /* initialization for https://llvm.org/pr42570 */ }; EXPORT_SYMBOL(kmalloc_caches); From patchwork Mon Jul 1 19:12:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 13718565 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0E2C2C8C7; Mon, 1 Jul 2024 19:13:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719861185; cv=none; b=QHqFbq7tvgZ5ZVlzsY8ZFiJSBODQ1HixWGK6Q+uISnstTBAAVEeeq1pma1Y+WbN7LeWFo4l5bJYkglYJ+PfW09gIdhMnlaQZ4jNORXZ5fpu+2FYuiNBEjTNj0njDK+tv7Jsc1HoKMHHfAZoWM7z+zifIoYmRUXDXadWU7Vh+WVs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719861185; c=relaxed/simple; bh=Fh8hn82A6Gkidq5L2q9k9Jp9kuNH5GwUiFNzo6MraKY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=DLnBp8w+BK2kvFgS4Zb9kGR/SHh3crD139Tjmg2hVXF0Ao1aI95nVyW4KlTyLm0WNr02so1OZdXC7YWCEi9D98EW8nX0LaXbHI1wg5VUSZFqztTqBk+W547H0fKkMq2986hbQlId1I611e5r/CxBB9H+GP9fm/RQfHnXExCEH74= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=WwTCjn01; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="WwTCjn01" Received: by smtp.kernel.org (Postfix) with ESMTPSA id A62BBC32786; Mon, 1 Jul 2024 19:13:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1719861184; bh=Fh8hn82A6Gkidq5L2q9k9Jp9kuNH5GwUiFNzo6MraKY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=WwTCjn01rIrhnVzozUC8nSU9ygzSVOAWK2NwHfiIHz88ZxMfH78DfyBNZqPbgJhjt 1En0azbb+FzKarKw46ht5i47ODAV8FDJFJbLHdekbKlVvgy5q2vCiB5ZeWkJHdln/1 X/PkEyTnjr7iDbFCSDom1lDpgFvDRDsuB5rKrjSiBxCflX6oYP8LjJDXDPiaqt+j7M RBONoQcMp3d6DSSp4xZJsxVObs45dqtw4EetmwIMSdOBCvf/DloxqKQ6RMiW96hV6N /cF9uMmOesCr4A/8eGfvuj0uaKnoLQtFvne4BkLbRHuhb57/TY8sS7eCV1LW5/3Wjz tp4ark47oZ4pw== From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , "GONG, Ruiqi" , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , jvoisin , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Xiu Jianfeng , Suren Baghdasaryan , Kent Overstreet , Jann Horn , Matteo Rizzo , Thomas Graf , Herbert Xu , "Gustavo A. R. Silva" , Jonathan Corbet , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-hardening@vger.kernel.org, netdev@vger.kernel.org Subject: [PATCH v6 2/6] mm/slab: Plumb kmem_buckets into __do_kmalloc_node() Date: Mon, 1 Jul 2024 12:12:59 -0700 Message-Id: <20240701191304.1283894-2-kees@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240701190152.it.631-kees@kernel.org> References: <20240701190152.it.631-kees@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=10530; i=kees@kernel.org; h=from:subject; bh=Fh8hn82A6Gkidq5L2q9k9Jp9kuNH5GwUiFNzo6MraKY=; b=owEBbQKS/ZANAwAKAYly9N/cbcAmAcsmYgBmgv++vHvQ6T0Eu3tRk8QttwaaXPyJTfZbfA7bS LQp+ltrjfuJAjMEAAEKAB0WIQSlw/aPIp3WD3I+bhOJcvTf3G3AJgUCZoL/vgAKCRCJcvTf3G3A Jl9uD/0f7gAetPdbAgu8momSOZv6Cae+I/11HPjOsFzUo49cNUJlFFs/A+yXPo0sZUGd6BWGT8b 4z3V/y10bQjSk2Ce5NdLROSleTVt5Z0kKP4znE7LRvawxxIMcE6NRSE239WY8ClZprrMQFAFota z2qwPKdBLlvhsvt8prwxl4AtS3MSqFWNiQTOyFNXvqwzNzx7q0R3yGSHR11TyMPdDITDPCMfKFJ ql4xYz4/P/0hJ61sbBSCKo6HU3NmOnPYGDe/WkXIB3bTn++2ACQsbkcihNSU3jLY/Oc0Pfr53XA L3Rm69795J6HaMBcRy1JF5ff+JNuyhBe0fA2iH5NlrjNJpi1vh0qA3azfgQtcF3usL1kxLzwCy1 tkfMp8MyYUIzs/PPx+0AHGKFLqRk+brl+1t8vYB2nZ3eHv9BC/ow8/GbPG5A0+BJOHBFNzEQuha ACPwzCjwKOzUZTBeIjK+93446+7EpROWkRRNgNk1wsqQ8eMs0l5RVVAfMT1gH74tDWYlyIYro5z KgthZkHBkTsGnii0Twa5PwMkHLtwdf5khBhW6/geyBph5Lk2lbU9LozhwbbEQPenvXSj1hAEoaC LXc/OOvyAxp8ZKJIZZVEGbqvjTRyg32q6V36pjHIrB6OMsSgFlZMwaI63511/yR3zN3r8GxPbvU mWcGxNfTGQUNs+w== X-Developer-Key: i=kees@kernel.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 Introduce CONFIG_SLAB_BUCKETS which provides the infrastructure to support separated kmalloc buckets (in the following kmem_buckets_create() patches and future codetag-based separation). Since this will provide a mitigation for a very common case of exploits, it is recommended to enable this feature for general purpose distros. By default, the new Kconfig will be enabled if CONFIG_SLAB_FREELIST_HARDENED is enabled (and it is added to the hardening.config Kconfig fragment). To be able to choose which buckets to allocate from, make the buckets available to the internal kmalloc interfaces by adding them as the second argument, rather than depending on the buckets being chosen from the fixed set of global buckets. Where the bucket is not available, pass NULL, which means "use the default system kmalloc bucket set" (the prior existing behavior), as implemented in kmalloc_slab(). To avoid adding the extra argument when !CONFIG_SLAB_BUCKETS, only the top-level macros and static inlines use the buckets argument (where they are stripped out and compiled out respectively). The actual extern functions can then be built without the argument, and the internals fall back to the global kmalloc buckets unconditionally. Co-developed-by: Vlastimil Babka Signed-off-by: Vlastimil Babka Signed-off-by: Kees Cook --- include/linux/slab.h | 27 ++++++++++++++++++++++----- kernel/configs/hardening.config | 1 + mm/Kconfig | 17 +++++++++++++++++ mm/slab.h | 6 ++++-- mm/slab_common.c | 2 +- mm/slub.c | 20 ++++++++++---------- scripts/kernel-doc | 1 + 7 files changed, 56 insertions(+), 18 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 8a006fac57c6..708bde6039f0 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -570,6 +570,21 @@ void *kmem_cache_alloc_node_noprof(struct kmem_cache *s, gfp_t flags, int node) __assume_slab_alignment __malloc; #define kmem_cache_alloc_node(...) alloc_hooks(kmem_cache_alloc_node_noprof(__VA_ARGS__)) +/* + * These macros allow declaring a kmem_buckets * parameter alongside size, which + * can be compiled out with CONFIG_SLAB_BUCKETS=n so that a large number of call + * sites don't have to pass NULL. + */ +#ifdef CONFIG_SLAB_BUCKETS +#define DECL_BUCKET_PARAMS(_size, _b) size_t (_size), kmem_buckets *(_b) +#define PASS_BUCKET_PARAMS(_size, _b) (_size), (_b) +#define PASS_BUCKET_PARAM(_b) (_b) +#else +#define DECL_BUCKET_PARAMS(_size, _b) size_t (_size) +#define PASS_BUCKET_PARAMS(_size, _b) (_size) +#define PASS_BUCKET_PARAM(_b) NULL +#endif + /* * The following functions are not to be used directly and are intended only * for internal use from kmalloc() and kmalloc_node() @@ -579,7 +594,7 @@ void *kmem_cache_alloc_node_noprof(struct kmem_cache *s, gfp_t flags, void *__kmalloc_noprof(size_t size, gfp_t flags) __assume_kmalloc_alignment __alloc_size(1); -void *__kmalloc_node_noprof(size_t size, gfp_t flags, int node) +void *__kmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node) __assume_kmalloc_alignment __alloc_size(1); void *__kmalloc_cache_noprof(struct kmem_cache *s, gfp_t flags, size_t size) @@ -679,7 +694,7 @@ static __always_inline __alloc_size(1) void *kmalloc_node_noprof(size_t size, gf kmalloc_caches[kmalloc_type(flags, _RET_IP_)][index], flags, node, size); } - return __kmalloc_node_noprof(size, flags, node); + return __kmalloc_node_noprof(PASS_BUCKET_PARAMS(size, NULL), flags, node); } #define kmalloc_node(...) alloc_hooks(kmalloc_node_noprof(__VA_ARGS__)) @@ -730,8 +745,10 @@ static inline __realloc_size(2, 3) void * __must_check krealloc_array_noprof(voi */ #define kcalloc(n, size, flags) kmalloc_array(n, size, (flags) | __GFP_ZERO) -void *kmalloc_node_track_caller_noprof(size_t size, gfp_t flags, int node, - unsigned long caller) __alloc_size(1); +void *__kmalloc_node_track_caller_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node, + unsigned long caller) __alloc_size(1); +#define kmalloc_node_track_caller_noprof(size, flags, node, caller) \ + __kmalloc_node_track_caller_noprof(PASS_BUCKET_PARAMS(size, NULL), flags, node, caller) #define kmalloc_node_track_caller(...) \ alloc_hooks(kmalloc_node_track_caller_noprof(__VA_ARGS__, _RET_IP_)) @@ -757,7 +774,7 @@ static inline __alloc_size(1, 2) void *kmalloc_array_node_noprof(size_t n, size_ return NULL; if (__builtin_constant_p(n) && __builtin_constant_p(size)) return kmalloc_node_noprof(bytes, flags, node); - return __kmalloc_node_noprof(bytes, flags, node); + return __kmalloc_node_noprof(PASS_BUCKET_PARAMS(bytes, NULL), flags, node); } #define kmalloc_array_node(...) alloc_hooks(kmalloc_array_node_noprof(__VA_ARGS__)) diff --git a/kernel/configs/hardening.config b/kernel/configs/hardening.config index 8a7ce7a6b3ab..3fabb8f55ef6 100644 --- a/kernel/configs/hardening.config +++ b/kernel/configs/hardening.config @@ -20,6 +20,7 @@ CONFIG_RANDOMIZE_MEMORY=y # Randomize allocator freelists, harden metadata. CONFIG_SLAB_FREELIST_RANDOM=y CONFIG_SLAB_FREELIST_HARDENED=y +CONFIG_SLAB_BUCKETS=y CONFIG_SHUFFLE_PAGE_ALLOCATOR=y CONFIG_RANDOM_KMALLOC_CACHES=y diff --git a/mm/Kconfig b/mm/Kconfig index b4cb45255a54..e0dfb268717c 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -273,6 +273,23 @@ config SLAB_FREELIST_HARDENED sacrifices to harden the kernel slab allocator against common freelist exploit methods. +config SLAB_BUCKETS + bool "Support allocation from separate kmalloc buckets" + depends on !SLUB_TINY + default SLAB_FREELIST_HARDENED + help + Kernel heap attacks frequently depend on being able to create + specifically-sized allocations with user-controlled contents + that will be allocated into the same kmalloc bucket as a + target object. To avoid sharing these allocation buckets, + provide an explicitly separated set of buckets to be used for + user-controlled allocations. This may very slightly increase + memory fragmentation, though in practice it's only a handful + of extra pages since the bulk of user-controlled allocations + are relatively long-lived. + + If unsure, say Y. + config SLUB_STATS default n bool "Enable performance statistics" diff --git a/mm/slab.h b/mm/slab.h index b16e63191578..d5e8034af9d5 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -403,16 +403,18 @@ static inline unsigned int size_index_elem(unsigned int bytes) * KMALLOC_MAX_CACHE_SIZE and the caller must check that. */ static inline struct kmem_cache * -kmalloc_slab(size_t size, gfp_t flags, unsigned long caller) +kmalloc_slab(size_t size, kmem_buckets *b, gfp_t flags, unsigned long caller) { unsigned int index; + if (!b) + b = &kmalloc_caches[kmalloc_type(flags, caller)]; if (size <= 192) index = kmalloc_size_index[size_index_elem(size)]; else index = fls(size - 1); - return kmalloc_caches[kmalloc_type(flags, caller)][index]; + return (*b)[index]; } gfp_t kmalloc_fix_flags(gfp_t flags); diff --git a/mm/slab_common.c b/mm/slab_common.c index e0b1c109bed2..9b0f2ef951f1 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -702,7 +702,7 @@ size_t kmalloc_size_roundup(size_t size) * The flags don't matter since size_index is common to all. * Neither does the caller for just getting ->object_size. */ - return kmalloc_slab(size, GFP_KERNEL, 0)->object_size; + return kmalloc_slab(size, NULL, GFP_KERNEL, 0)->object_size; } /* Above the smaller buckets, size is a multiple of page size. */ diff --git a/mm/slub.c b/mm/slub.c index 3d19a0ee411f..80f0a51242d1 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4117,7 +4117,7 @@ void *__kmalloc_large_node_noprof(size_t size, gfp_t flags, int node) EXPORT_SYMBOL(__kmalloc_large_node_noprof); static __always_inline -void *__do_kmalloc_node(size_t size, gfp_t flags, int node, +void *__do_kmalloc_node(size_t size, kmem_buckets *b, gfp_t flags, int node, unsigned long caller) { struct kmem_cache *s; @@ -4133,32 +4133,32 @@ void *__do_kmalloc_node(size_t size, gfp_t flags, int node, if (unlikely(!size)) return ZERO_SIZE_PTR; - s = kmalloc_slab(size, flags, caller); + s = kmalloc_slab(size, b, flags, caller); ret = slab_alloc_node(s, NULL, flags, node, caller, size); ret = kasan_kmalloc(s, ret, size, flags); trace_kmalloc(caller, ret, size, s->size, flags, node); return ret; } - -void *__kmalloc_node_noprof(size_t size, gfp_t flags, int node) +void *__kmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node) { - return __do_kmalloc_node(size, flags, node, _RET_IP_); + return __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), flags, node, _RET_IP_); } EXPORT_SYMBOL(__kmalloc_node_noprof); void *__kmalloc_noprof(size_t size, gfp_t flags) { - return __do_kmalloc_node(size, flags, NUMA_NO_NODE, _RET_IP_); + return __do_kmalloc_node(size, NULL, flags, NUMA_NO_NODE, _RET_IP_); } EXPORT_SYMBOL(__kmalloc_noprof); -void *kmalloc_node_track_caller_noprof(size_t size, gfp_t flags, - int node, unsigned long caller) +void *__kmalloc_node_track_caller_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, + int node, unsigned long caller) { - return __do_kmalloc_node(size, flags, node, caller); + return __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), flags, node, caller); + } -EXPORT_SYMBOL(kmalloc_node_track_caller_noprof); +EXPORT_SYMBOL(__kmalloc_node_track_caller_noprof); void *__kmalloc_cache_noprof(struct kmem_cache *s, gfp_t gfpflags, size_t size) { diff --git a/scripts/kernel-doc b/scripts/kernel-doc index 95a59ac78f82..2791f8195203 100755 --- a/scripts/kernel-doc +++ b/scripts/kernel-doc @@ -1729,6 +1729,7 @@ sub dump_function($$) { $prototype =~ s/__printf\s*\(\s*\d*\s*,\s*\d*\s*\) +//; $prototype =~ s/__(?:re)?alloc_size\s*\(\s*\d+\s*(?:,\s*\d+\s*)?\) +//; $prototype =~ s/__diagnose_as\s*\(\s*\S+\s*(?:,\s*\d+\s*)*\) +//; + $prototype =~ s/DECL_BUCKET_PARAMS\s*\(\s*(\S+)\s*,\s*(\S+)\s*\)/$1, $2/; my $define = $prototype =~ s/^#\s*define\s+//; #ak added $prototype =~ s/__attribute_const__ +//; $prototype =~ s/__attribute__\s*\(\( From patchwork Mon Jul 1 19:13:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 13718569 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1F72816F278; Mon, 1 Jul 2024 19:13:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719861185; cv=none; b=h+KO/81aKlSSRE1Niy/rTn7vkWgjIgelYexo8pZHH+46MNK8lbHIxQoGX0VEMQOG+7KfgVM0/TxwRUrLZnWVvrXjgKsRDurDyLLO0k++Oz0oU4tXH9bsl8gwQ2wpJuTMQ9d0x5TEuY1zQdtd/W+9J1NaITsI6ARqeRHr03Swlc4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719861185; c=relaxed/simple; bh=xxet30UEfeVq/dWdKRNFzCWZboWOxErsseLhti/TXwg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=JBj5QrBmnfNOg2eP1ww6mVRppb+dQwJGnmT2uVJWLCoreJOwqaZmd5UHbFp/PiMqXw8t3EYh1hIBu6Bxb11QlwLZkhqlvd/7ycwbsahALWi9a/4VN0KHvCKAVmW+c58wGmyN7DLZTJI6T+nJ8HcuqQ5EK+10sf+A3qJCnUsz2PA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=mMao9yoX; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="mMao9yoX" Received: by smtp.kernel.org (Postfix) with ESMTPSA id B4989C4AF0A; Mon, 1 Jul 2024 19:13:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1719861184; bh=xxet30UEfeVq/dWdKRNFzCWZboWOxErsseLhti/TXwg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=mMao9yoX18zMVhMQY5kIolTbGVQ1jNtDDjmsTCB1JdDobR3dwbrFWHV4S6VLmWFQ6 8klmPbsi/OtNrbFwfYS/rBPG3KZGTj2fhyS7xNyw5asCH7SzSgCPO5zq4qlbgmJVC4 NlzKgmInTFyAWCzXWeqI/pP1k9mxSB9+5iA/G25w4RRI/8F7Fn4l/TaWWuIZgjbV0O 4PFTz+aPo3ftiGxkUDdGoJVCN+BSH5sVJriXl+rpVGOiq3qanJo2M802ddSfZPGdfg rZo1KXvtSOciCb+zMGA/Wx9Mm8L8/fQQBF2P5oOH8z0fF+CUrt0RzcxI+SL5n/fjyN gvLwRdyObFs6A== From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , "GONG, Ruiqi" , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , jvoisin , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Xiu Jianfeng , Suren Baghdasaryan , Kent Overstreet , Jann Horn , Matteo Rizzo , Thomas Graf , Herbert Xu , "Gustavo A. R. Silva" , Jonathan Corbet , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-hardening@vger.kernel.org, netdev@vger.kernel.org Subject: [PATCH v6 3/6] mm/slab: Introduce kvmalloc_buckets_node() that can take kmem_buckets argument Date: Mon, 1 Jul 2024 12:13:00 -0700 Message-Id: <20240701191304.1283894-3-kees@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240701190152.it.631-kees@kernel.org> References: <20240701190152.it.631-kees@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2960; i=kees@kernel.org; h=from:subject; bh=xxet30UEfeVq/dWdKRNFzCWZboWOxErsseLhti/TXwg=; b=owEBbQKS/ZANAwAKAYly9N/cbcAmAcsmYgBmgv++rAHw0U+CEnyXA/4XClLK6QAyqHm8wswfo OqdafG1Bj+JAjMEAAEKAB0WIQSlw/aPIp3WD3I+bhOJcvTf3G3AJgUCZoL/vgAKCRCJcvTf3G3A JnwwEACfJUwABmsGkmhqQnATf7Y0sAvd2R6x4c0rtnoHIuF4bVZHVNfrX+sm0soE98hTF6dJhSb 54g7YCknTpz7GmrfaJ5B+6Nt892AgOuCstHR6q3rMDkpbqRRk7a3VIOEqbNnPUNJpyf4Wj6oTB+ FpKGwepyFVPdvd7Prn6jPRJzZQmnRlJRqG73USLhSw5sSEQjd452tI2JG+gHL/qwx11dQeJHiEr 9+/VZ1zhVvgJ1rrfQ4zOcSEPDuuIKjF75SGADyD6H0y20WrMSuGAHFoJpbJYxDkAJE4Gv1Yq28b NHiH8cJzxNi85ddufgze9mVDh6j/09oxZUB6Q4kZYmqy9Xu+f7qrxqQbLbnKV1XYYP9ZGJN0v8H Ludofn6ojZhgH8KOYlpEMD77aG9nOAYKMFPg5wBqXoXlzfhUVhS1Q3UlodTk7WQR9dhez0JEBpW /jw6xO8QkzUt0yBRT3nry/LJsmZmaeY54ekml6BugsOLniwFjBaq/fs/7a6TIDRO8ESL24TtvrB 1GNHbIRz/GxaxHXegXWceNTTLerX3JSOVEnG3qv1zJsV9XPTMAhmlZAV2DrLasptG6tkwgPC9zo Mf0SCpzv9ElM+HZzsRuDAJL6CkBHzXvbgW0IH4traaTQNdAHVoBKHn0pxZfluvUuUt4eiE9VO3D 14HEVxu1s4wtbJg== X-Developer-Key: i=kees@kernel.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 Plumb kmem_buckets arguments through kvmalloc_node_noprof() so it is possible to provide an API to perform kvmalloc-style allocations with a particular set of buckets. Introduce kvmalloc_buckets_node() that takes a kmem_buckets argument. Signed-off-by: Kees Cook --- include/linux/slab.h | 4 +++- mm/util.c | 9 +++++---- 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 708bde6039f0..8d0800c7579a 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -798,7 +798,9 @@ static inline __alloc_size(1) void *kzalloc_noprof(size_t size, gfp_t flags) #define kzalloc(...) alloc_hooks(kzalloc_noprof(__VA_ARGS__)) #define kzalloc_node(_size, _flags, _node) kmalloc_node(_size, (_flags)|__GFP_ZERO, _node) -extern void *kvmalloc_node_noprof(size_t size, gfp_t flags, int node) __alloc_size(1); +void *__kvmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node) __alloc_size(1); +#define kvmalloc_node_noprof(size, flags, node) \ + __kvmalloc_node_noprof(PASS_BUCKET_PARAMS(size, NULL), flags, node) #define kvmalloc_node(...) alloc_hooks(kvmalloc_node_noprof(__VA_ARGS__)) #define kvmalloc(_size, _flags) kvmalloc_node(_size, _flags, NUMA_NO_NODE) diff --git a/mm/util.c b/mm/util.c index c9e519e6811f..28c5356b9f1c 100644 --- a/mm/util.c +++ b/mm/util.c @@ -594,9 +594,10 @@ unsigned long vm_mmap(struct file *file, unsigned long addr, EXPORT_SYMBOL(vm_mmap); /** - * kvmalloc_node - attempt to allocate physically contiguous memory, but upon + * __kvmalloc_node - attempt to allocate physically contiguous memory, but upon * failure, fall back to non-contiguous (vmalloc) allocation. * @size: size of the request. + * @b: which set of kmalloc buckets to allocate from. * @flags: gfp mask for the allocation - must be compatible (superset) with GFP_KERNEL. * @node: numa node to allocate from * @@ -609,7 +610,7 @@ EXPORT_SYMBOL(vm_mmap); * * Return: pointer to the allocated memory of %NULL in case of failure */ -void *kvmalloc_node_noprof(size_t size, gfp_t flags, int node) +void *__kvmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node) { gfp_t kmalloc_flags = flags; void *ret; @@ -631,7 +632,7 @@ void *kvmalloc_node_noprof(size_t size, gfp_t flags, int node) kmalloc_flags &= ~__GFP_NOFAIL; } - ret = kmalloc_node_noprof(size, kmalloc_flags, node); + ret = __kmalloc_node_noprof(PASS_BUCKET_PARAMS(size, b), kmalloc_flags, node); /* * It doesn't really make sense to fallback to vmalloc for sub page @@ -660,7 +661,7 @@ void *kvmalloc_node_noprof(size_t size, gfp_t flags, int node) flags, PAGE_KERNEL, VM_ALLOW_HUGE_VMAP, node, __builtin_return_address(0)); } -EXPORT_SYMBOL(kvmalloc_node_noprof); +EXPORT_SYMBOL(__kvmalloc_node_noprof); /** * kvfree() - Free memory. From patchwork Mon Jul 1 19:13:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 13718568 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1E0FF16F26E; Mon, 1 Jul 2024 19:13:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719861185; cv=none; b=ozGDK57V6BoKit1Ifsa8GGbNyd1XfJ0ACiLrr6w5akH42UOBTAxwqpTCYM5QA4eazT/FUaT1cia6Ioc3uwn/b5k05WKRmnz1VQoog4o1e5wYJXGXBimJm26eCVE+eGb/4GQ3lnCWqJLxXSUjQE21puInE2s9S2h22XAMUNC6QHI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719861185; c=relaxed/simple; bh=iLr1JO1gVFtydBJDlW/Z6SdDIb8sQtIWfFXCMaocDcg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=iWUZszIoXylh/EeCXSJBJimz2h86sPt6jvM6ZFOa/QOKgLDBrrQtl30Ccj1459+m2YJWiVkdPOy+4QyKSwf3cnbyqgMC7GtSuQK1OaIlJiT6qxRTobohav95VnnbmbpIDTjhhYEchT/wa/QYQDZEnSvmYUIol7GeW2mnFAQPfAo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=RpSfmTgx; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="RpSfmTgx" Received: by smtp.kernel.org (Postfix) with ESMTPSA id BEF06C4AF0C; Mon, 1 Jul 2024 19:13:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1719861184; bh=iLr1JO1gVFtydBJDlW/Z6SdDIb8sQtIWfFXCMaocDcg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=RpSfmTgxm2O1XtuIFl+3ZxeSoxYypsrEU0+nDkunMVjz9fqvzNZXImfKfOgbhxqvZ CWJEEidhNGTyifG0nR+rXVJIx/A2qesIuRGrcbMilA4qSyFRmRnfeGtpcB6RwcdXAO 8FlpEO+3qZiqtJZNOnTdQuAwR7KU7OgZ5Fyv++2LC0iKVbRQ6H3cWeLFzRXFvoqnM0 eSrGAf0KiZ/UCgZ8S2+lv2V61tOvs+nM9c+XIVf2D9KThxnpza0obf9zEimu91+0nu 5mMnT0ywl5hdZUH+bX9G3igqs+pfRNBBTHgMRtShjduDuQlJt4j1RjRb3SuLNeQ4xo mLYYJCU2yV6Ew== From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , "GONG, Ruiqi" , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , jvoisin , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Xiu Jianfeng , Suren Baghdasaryan , Kent Overstreet , Jann Horn , Matteo Rizzo , Thomas Graf , Herbert Xu , "Gustavo A. R. Silva" , Jonathan Corbet , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-hardening@vger.kernel.org, netdev@vger.kernel.org Subject: [PATCH v6 4/6] mm/slab: Introduce kmem_buckets_create() and family Date: Mon, 1 Jul 2024 12:13:01 -0700 Message-Id: <20240701191304.1283894-4-kees@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240701190152.it.631-kees@kernel.org> References: <20240701190152.it.631-kees@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=8116; i=kees@kernel.org; h=from:subject; bh=iLr1JO1gVFtydBJDlW/Z6SdDIb8sQtIWfFXCMaocDcg=; b=owEBbQKS/ZANAwAKAYly9N/cbcAmAcsmYgBmgv++kshPnyXBQpYmXkhyaUZvkPmyCCbIBrXnN 5/UgqpveV2JAjMEAAEKAB0WIQSlw/aPIp3WD3I+bhOJcvTf3G3AJgUCZoL/vgAKCRCJcvTf3G3A Ji6UEACxYLIBboDvKj5qoLIIzqp3hMa09xT5pXc8GbKmY2/GyXBftV83vive0p+5epvrpOMcCJj UdH1IPLxIdnSKJP/L/8+Rv9jj1BGjR6WTWWOv7GcNtMFdeXzAuh6c/mxpLYV8FRSYvFkXuhbab4 ERTeCuecezcFkNxqUYpcRO75k/eyY5twXVS4i42r/VGX1MOQOC00xBBZI4QqwAirHWU7TzmshBX t9dPiM/LqzE54MRSghuxmUFpVavRcaIGqE9gjNIEeV7ELNEBvT3UpdRuImMkWoxNKQE7he5T6h/ NtO+zOa14B1oBkp4YFjZHLuorD9gkhEsxIVkaPxe7eniA1a4r8s8/IY2/MbMeznYldyYmrNsxwY jbAGyovGKjvR/UWHJSNqT7X2ImJ2UdEt8dkJ8Zck+crZ4wBGPUnPgbANbT/OaiY6MSV5H5pwvG0 K7JQntezgoxoQ29Y1GTD0QpiESxigH3wAERxkJaXnMEL59PyEBWblaDhJz76ttKLqon7S79FZpp jAHE5SZWTFjOBBAwCDhsCp0NlD5jAQ70cJkGWgLYmfK8KWb0geMS3TKZs8yGCytCBA8YpTkM1Xc V2jkNV/A54x/ujur4vKgrirvvVJ+uK2WPCIB3AOYVdfxHAL6EknMufnamvya3upXuHwTcD3TU8u RuEkWXQJaZEN9eQ== X-Developer-Key: i=kees@kernel.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 Dedicated caches are available for fixed size allocations via kmem_cache_alloc(), but for dynamically sized allocations there is only the global kmalloc API's set of buckets available. This means it isn't possible to separate specific sets of dynamically sized allocations into a separate collection of caches. This leads to a use-after-free exploitation weakness in the Linux kernel since many heap memory spraying/grooming attacks depend on using userspace-controllable dynamically sized allocations to collide with fixed size allocations that end up in same cache. While CONFIG_RANDOM_KMALLOC_CACHES provides a probabilistic defense against these kinds of "type confusion" attacks, including for fixed same-size heap objects, we can create a complementary deterministic defense for dynamically sized allocations that are directly user controlled. Addressing these cases is limited in scope, so isolating these kinds of interfaces will not become an unbounded game of whack-a-mole. For example, many pass through memdup_user(), making isolation there very effective. In order to isolate user-controllable dynamically-sized allocations from the common system kmalloc allocations, introduce kmem_buckets_create(), which behaves like kmem_cache_create(). Introduce kmem_buckets_alloc(), which behaves like kmem_cache_alloc(). Introduce kmem_buckets_alloc_track_caller() for where caller tracking is needed. Introduce kmem_buckets_valloc() for cases where vmalloc fallback is needed. Note that these caches are specifically flagged with SLAB_NO_MERGE, since merging would defeat the entire purpose of the mitigation. This can also be used in the future to extend allocation profiling's use of code tagging to implement per-caller allocation cache isolation[1] even for dynamic allocations. Memory allocation pinning[2] is still needed to plug the Use-After-Free cross-allocator weakness (where attackers can arrange to free an entire slab page and have it reallocated to a different cache), but that is an existing and separate issue which is complementary to this improvement. Development continues for that feature via the SLAB_VIRTUAL[3] series (which could also provide guard pages -- another complementary improvement). Link: https://lore.kernel.org/lkml/202402211449.401382D2AF@keescook [1] Link: https://googleprojectzero.blogspot.com/2021/10/how-simple-linux-kernel-memory.html [2] Link: https://lore.kernel.org/lkml/20230915105933.495735-1-matteorizzo@google.com/ [3] Signed-off-by: Kees Cook --- include/linux/slab.h | 12 ++++++ mm/slab_common.c | 96 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 108 insertions(+) diff --git a/include/linux/slab.h b/include/linux/slab.h index 8d0800c7579a..4c083f3196f4 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -549,6 +549,10 @@ void *kmem_cache_alloc_lru_noprof(struct kmem_cache *s, struct list_lru *lru, void kmem_cache_free(struct kmem_cache *s, void *objp); +kmem_buckets *kmem_buckets_create(const char *name, slab_flags_t flags, + unsigned int useroffset, unsigned int usersize, + void (*ctor)(void *)); + /* * Bulk allocation and freeing operations. These are accelerated in an * allocator specific way to avoid taking locks repeatedly or building @@ -681,6 +685,12 @@ static __always_inline __alloc_size(1) void *kmalloc_noprof(size_t size, gfp_t f } #define kmalloc(...) alloc_hooks(kmalloc_noprof(__VA_ARGS__)) +#define kmem_buckets_alloc(_b, _size, _flags) \ + alloc_hooks(__kmalloc_node_noprof(PASS_BUCKET_PARAMS(_size, _b), _flags, NUMA_NO_NODE)) + +#define kmem_buckets_alloc_track_caller(_b, _size, _flags) \ + alloc_hooks(__kmalloc_node_track_caller_noprof(PASS_BUCKET_PARAMS(_size, _b), _flags, NUMA_NO_NODE, _RET_IP_)) + static __always_inline __alloc_size(1) void *kmalloc_node_noprof(size_t size, gfp_t flags, int node) { if (__builtin_constant_p(size) && size) { @@ -808,6 +818,8 @@ void *__kvmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node) #define kvzalloc(_size, _flags) kvmalloc(_size, (_flags)|__GFP_ZERO) #define kvzalloc_node(_size, _flags, _node) kvmalloc_node(_size, (_flags)|__GFP_ZERO, _node) +#define kmem_buckets_valloc(_b, _size, _flags) \ + alloc_hooks(__kvmalloc_node_noprof(PASS_BUCKET_PARAMS(_size, _b), _flags, NUMA_NO_NODE)) static inline __alloc_size(1, 2) void * kvmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags, int node) diff --git a/mm/slab_common.c b/mm/slab_common.c index 9b0f2ef951f1..641cf513d2d1 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -392,6 +392,98 @@ kmem_cache_create(const char *name, unsigned int size, unsigned int align, } EXPORT_SYMBOL(kmem_cache_create); +static struct kmem_cache *kmem_buckets_cache __ro_after_init; + +/** + * kmem_buckets_create - Create a set of caches that handle dynamic sized + * allocations via kmem_buckets_alloc() + * @name: A prefix string which is used in /proc/slabinfo to identify this + * cache. The individual caches with have their sizes as the suffix. + * @flags: SLAB flags (see kmem_cache_create() for details). + * @useroffset: Starting offset within an allocation that may be copied + * to/from userspace. + * @usersize: How many bytes, starting at @useroffset, may be copied + * to/from userspace. + * @ctor: A constructor for the objects, run when new allocations are made. + * + * Cannot be called within an interrupt, but can be interrupted. + * + * Return: a pointer to the cache on success, NULL on failure. When + * CONFIG_SLAB_BUCKETS is not enabled, ZERO_SIZE_PTR is returned, and + * subsequent calls to kmem_buckets_alloc() will fall back to kmalloc(). + * (i.e. callers only need to check for NULL on failure.) + */ +kmem_buckets *kmem_buckets_create(const char *name, slab_flags_t flags, + unsigned int useroffset, + unsigned int usersize, + void (*ctor)(void *)) +{ + kmem_buckets *b; + int idx; + + /* + * When the separate buckets API is not built in, just return + * a non-NULL value for the kmem_buckets pointer, which will be + * unused when performing allocations. + */ + if (!IS_ENABLED(CONFIG_SLAB_BUCKETS)) + return ZERO_SIZE_PTR; + + if (WARN_ON(!kmem_buckets_cache)) + return NULL; + + b = kmem_cache_alloc(kmem_buckets_cache, GFP_KERNEL|__GFP_ZERO); + if (WARN_ON(!b)) + return NULL; + + flags |= SLAB_NO_MERGE; + + for (idx = 0; idx < ARRAY_SIZE(kmalloc_caches[KMALLOC_NORMAL]); idx++) { + char *short_size, *cache_name; + unsigned int cache_useroffset, cache_usersize; + unsigned int size; + + if (!kmalloc_caches[KMALLOC_NORMAL][idx]) + continue; + + size = kmalloc_caches[KMALLOC_NORMAL][idx]->object_size; + if (!size) + continue; + + short_size = strchr(kmalloc_caches[KMALLOC_NORMAL][idx]->name, '-'); + if (WARN_ON(!short_size)) + goto fail; + + cache_name = kasprintf(GFP_KERNEL, "%s-%s", name, short_size + 1); + if (WARN_ON(!cache_name)) + goto fail; + + if (useroffset >= size) { + cache_useroffset = 0; + cache_usersize = 0; + } else { + cache_useroffset = useroffset; + cache_usersize = min(size - cache_useroffset, usersize); + } + (*b)[idx] = kmem_cache_create_usercopy(cache_name, size, + 0, flags, cache_useroffset, + cache_usersize, ctor); + kfree(cache_name); + if (WARN_ON(!(*b)[idx])) + goto fail; + } + + return b; + +fail: + for (idx = 0; idx < ARRAY_SIZE(kmalloc_caches[KMALLOC_NORMAL]); idx++) + kmem_cache_destroy((*b)[idx]); + kfree(b); + + return NULL; +} +EXPORT_SYMBOL(kmem_buckets_create); + #ifdef SLAB_SUPPORTS_SYSFS /* * For a given kmem_cache, kmem_cache_destroy() should only be called @@ -931,6 +1023,10 @@ void __init create_kmalloc_caches(void) /* Kmalloc array is now usable */ slab_state = UP; + + kmem_buckets_cache = kmem_cache_create("kmalloc_buckets", + sizeof(kmem_buckets), + 0, SLAB_NO_MERGE, NULL); } /** From patchwork Mon Jul 1 19:13:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 13718570 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6D2C816F85E; Mon, 1 Jul 2024 19:13:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719861185; cv=none; b=FVOG2zRpSacA7q83gZ8/5p9N3gWndarj6ULyfGZwA87jWDuoKWd8er8ztJ8XlFiNdR4qmlT6QXqEnLl5Q5IgWTQlqRj3TmHldd83W1AfQOQLxAaL63imSIKDDgXXFRrOEk+lJuomlHAybh2H4Row2b4UT08zeuDJc6hZ0+H1p/Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719861185; c=relaxed/simple; bh=ZC+4xLsqMS25+wle+eOPiqDyMZP//OUcAmXX2J5RAVo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=fJ/4pguvMJs/zZVU63W7lKwyfL5M8rt+5j3FITfTOjug7PdZnwGou0HrN1sobQA4VoR0e5/sExK4rE9CquvN1KlKyMej10f25p6YsX9uGC2U55pZk0ENksohjZGvmxd3W1UXVBX7W2kA5VHf9xUqZmNaaV18RucMvLXMipwtPs0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=gghdBhp7; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="gghdBhp7" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2019EC4AF0F; Mon, 1 Jul 2024 19:13:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1719861185; bh=ZC+4xLsqMS25+wle+eOPiqDyMZP//OUcAmXX2J5RAVo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=gghdBhp7ZB7XhRBrUQVzahbv03mmZLCVplJYE8pWnFaiQBh2snWO0KElN7hj5L0Ny TCJLGI1T2/AYbR2OOeDbeBWZfI7c7oL4M/JJ+RlL0S6mrIsnfyUr2asG/Co/Y4NEib 2S8SMe6fCT9MJr1eFjg0+faPtFSQNhPRNUFqlzP/oVpb1NFgNCMw9FHbeSyp8aQwd8 Rpp3mces1RFAQJXcr+XIjwJD+CSoa+o3l+nguqgsSPs0GKDCrrDHUS+spXwD2uqiPF isAeixm5YYh/xbBj+DX8ZhRg76tY6LlFeuQrEUT5H9ThGb16jkfh9m3ifBk/g38bmt dF3WKO2zZc9Gg== From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , "GONG, Ruiqi" , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , jvoisin , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Xiu Jianfeng , Suren Baghdasaryan , Kent Overstreet , Jann Horn , Matteo Rizzo , Thomas Graf , Herbert Xu , "Gustavo A. R. Silva" , Jonathan Corbet , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-hardening@vger.kernel.org, netdev@vger.kernel.org Subject: [PATCH v6 5/6] ipc, msg: Use dedicated slab buckets for alloc_msg() Date: Mon, 1 Jul 2024 12:13:02 -0700 Message-Id: <20240701191304.1283894-5-kees@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240701190152.it.631-kees@kernel.org> References: <20240701190152.it.631-kees@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1954; i=kees@kernel.org; h=from:subject; bh=ZC+4xLsqMS25+wle+eOPiqDyMZP//OUcAmXX2J5RAVo=; b=owEBbQKS/ZANAwAKAYly9N/cbcAmAcsmYgBmgv++179k0fHAvGjqEU5ltN8MHRwh6Dca9ahz8 0TL+TDEUBSJAjMEAAEKAB0WIQSlw/aPIp3WD3I+bhOJcvTf3G3AJgUCZoL/vgAKCRCJcvTf3G3A JmaMEAChdkbMO+SdicBA6J36xZMom97yPCUJe+AQcsVFpIdELNyttcTScgDv/IxVDdTTaDyBHTr YayvHGh0Bi1wtq7wdAnfUZIju5ZUa/WCgNCX2fLmkOeyKifYhjk2i9+VtTvLMF6K9caBNiPSieA zYCUvURzBk7S44/7T4ThX23ycIiwE+oU9Ud4uP15daHyrsTeeX+aLrWunFHWf4kTgAr71yIitjr 3mqyNbU+g8ZieFgH4PDLYynu+TtPt4iO2wZWiZec5JKUOCxFXM8gah47jLrZhvfdS4y8JwaEt9m Vjla4RVs4n8rqtja/7dPtTBSB1GkhywqySFJoH1BYAt2Vn2vFpWf5I8u5bLmvJM87SdhhRDZyf6 WLCvokYur6sn8yBFlXEw4LO99JBoilqOqAWn+ErUmZLEJeY1HXMJhqTpqU7H2HHlBpleEymLIiN nkdQyxDF92oetfAMKw6rcHGic8gH4aqBBSAELkoPYc3RQg7CKUb8oQDwtNSCtdEgmYI0+rJ2J13 XH821MfWZoLrYcu86MJQ8X3ue/s6Qz+Fj3yp/11eMIW48V1qmlShzTv+ZoL365+1pAl5J0Wbilv XRniyDfgf6MQmT5FSDohh8ztx9NNEwlKNAGEaogx5tGQ8x9sQVSpCXj3bPqAzpYTbjOlm/kTmqh rmeH/xNCP1FCX6Q== X-Developer-Key: i=kees@kernel.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 The msg subsystem is a common target for exploiting[1][2][3][4][5][6][7] use-after-free type confusion flaws in the kernel for both read and write primitives. Avoid having a user-controlled dynamically-size allocation share the global kmalloc cache by using a separate set of kmalloc buckets via the kmem_buckets API. Link: https://blog.hacktivesecurity.com/index.php/2022/06/13/linux-kernel-exploit-development-1day-case-study/ [1] Link: https://hardenedvault.net/blog/2022-11-13-msg_msg-recon-mitigation-ved/ [2] Link: https://www.willsroot.io/2021/08/corctf-2021-fire-of-salvation-writeup.html [3] Link: https://a13xp0p0v.github.io/2021/02/09/CVE-2021-26708.html [4] Link: https://google.github.io/security-research/pocs/linux/cve-2021-22555/writeup.html [5] Link: https://zplin.me/papers/ELOISE.pdf [6] Link: https://syst3mfailure.io/wall-of-perdition/ [7] Signed-off-by: Kees Cook --- ipc/msgutil.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/ipc/msgutil.c b/ipc/msgutil.c index d0a0e877cadd..c7be0c792647 100644 --- a/ipc/msgutil.c +++ b/ipc/msgutil.c @@ -42,6 +42,17 @@ struct msg_msgseg { #define DATALEN_MSG ((size_t)PAGE_SIZE-sizeof(struct msg_msg)) #define DATALEN_SEG ((size_t)PAGE_SIZE-sizeof(struct msg_msgseg)) +static kmem_buckets *msg_buckets __ro_after_init; + +static int __init init_msg_buckets(void) +{ + msg_buckets = kmem_buckets_create("msg_msg", SLAB_ACCOUNT, + sizeof(struct msg_msg), + DATALEN_MSG, NULL); + + return 0; +} +subsys_initcall(init_msg_buckets); static struct msg_msg *alloc_msg(size_t len) { @@ -50,7 +61,7 @@ static struct msg_msg *alloc_msg(size_t len) size_t alen; alen = min(len, DATALEN_MSG); - msg = kmalloc(sizeof(*msg) + alen, GFP_KERNEL_ACCOUNT); + msg = kmem_buckets_alloc(msg_buckets, sizeof(*msg) + alen, GFP_KERNEL); if (msg == NULL) return NULL; From patchwork Mon Jul 1 19:13:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 13718571 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6E3D416F8E0; Mon, 1 Jul 2024 19:13:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719861185; cv=none; b=NMgIwQ8MXZqObKVJMeFzB42UPITBQt94pr+qS8ofjudzt3JluCE6jN0Kr3Bk4vW+an+KLOMBlp4pu2GTNNejCmqLqlEbaVv+CPfPjKynGQnzaPaE7ib88dxNJLP1AuMybCYbZ0PnbHZaLZhT2tkuviQUNy0TeP1mEU2nHcRN3WE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719861185; c=relaxed/simple; bh=BZ80Ejo56zseoiz3weNc0AtXfj9+HOXw5S5R0du1E70=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=RxGPtB5ST5ZzZFX9JJwQPeFDALcRgpjkAJzdxshp3521tCHkUJUjTZCKuQm1ExgODg7FMGGLzmtLTphIr45LB6KlfEWJjSaf9hQw1pxJ2biLVvkP16o1FTAescI2+tir6EURJ1hH4hREXEBJJ/VlbKuctGGO0pmoP1kYS2ZATJM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=bFl59nSl; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="bFl59nSl" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1EA58C4AF0D; Mon, 1 Jul 2024 19:13:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1719861185; bh=BZ80Ejo56zseoiz3weNc0AtXfj9+HOXw5S5R0du1E70=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=bFl59nSlnVaSRU4+0HTFQ/YeTaJahbBTt5u256YqqA/IZQs7WUOzBvYKYRUR7Jmb+ wChXJYbVxmTLzNRLpAAdNpOUmpNsdgtdPYaJohsJXrbZU6BCCf7ByLGnDH/BQ216L3 560Mu5RiCeTyUZ/1umQVt9bDpooP1+0DEUxtHU2rZe8fjdKsqR/lSu/h5TyYmL36gl B+EZzQJtEw/HTSqmuDqIJgLCnr4S7c9/dCzAmRxEeHSGjY5ypxgKwo/Qu6ELpXbvlh V1yE6QbOqM4goWOm1jibPASL3Mm/dft5RrFbUr9E7X9HNaKn/bgWcHd4IL6jWnK2Nr qPcQDrAj7jZcw== From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , "GONG, Ruiqi" , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , jvoisin , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Xiu Jianfeng , Suren Baghdasaryan , Kent Overstreet , Jann Horn , Matteo Rizzo , Thomas Graf , Herbert Xu , "Gustavo A. R. Silva" , Jonathan Corbet , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-hardening@vger.kernel.org, netdev@vger.kernel.org Subject: [PATCH v6 6/6] mm/util: Use dedicated slab buckets for memdup_user() Date: Mon, 1 Jul 2024 12:13:03 -0700 Message-Id: <20240701191304.1283894-6-kees@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240701190152.it.631-kees@kernel.org> References: <20240701190152.it.631-kees@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2796; i=kees@kernel.org; h=from:subject; bh=BZ80Ejo56zseoiz3weNc0AtXfj9+HOXw5S5R0du1E70=; b=owEBbQKS/ZANAwAKAYly9N/cbcAmAcsmYgBmgv++lHHVc7IaWVXSWcs6s/+88cmgFNyMQ+/tk NfYGR7ScZKJAjMEAAEKAB0WIQSlw/aPIp3WD3I+bhOJcvTf3G3AJgUCZoL/vgAKCRCJcvTf3G3A JqpJD/9rs0Nsc8oYdGcPhljjR3Oz9ava6B9ExnvhV2nq0WD+FJVM5EdnZQAwoNyWeBpcuY2PjdX lGeJh61ZCUx/6o593wAWAgLwf2e/Mhg8nSPBCedp2wVFCHCVtzZhrNBAaQlXjO3gqzsR3DB03kj bchxInUC1Jscc86bOfbjVaIQ5IEy5V7YzHU6nQndBY2UIg+uZZmOkFzlZJpnMapIejOc6SpanQA hV6NVDpX2xdgr7MF4aOqxOwTrZz75rZo/5GXBqomoCrJB/fD4y99gSAraNxMJGvzAIKI8sJVGKr nsL6Q/dH6LCFuZlplSg3SIYt9Gos+RIxGQkllf9g7DUjeUG95w+fsoV1avbkZXjgQnI2mfAyjMF LgFIqrfiAFybNMFQqTOyMz5qVVeFFFi/KS+LDgZLc8yQlhZP519x+tJ6F4oVUtqM4dW5u4yP6Vp DG21pqkK6pZGhWARTM64DaxCBk/pN0rgw4Kuu/yLd29xhS7+fvJkksPam8vLip8OXe83i4Euu0m JTpZvCyJaLkwRmeqjgLa8fTykUMXm3m1HS9xyfPOrqmY9zOiJg6hGNR8aS7wdw+z9wjeph7E6pc z2OqisDHlJ0Pad2aMEfBofKXW2BKrBAxRPKjOnkC7biQAMODdjv2mPwU4BWxkHlSZwtHdrCs+GE Lj0ThhUc9z01RFQ== X-Developer-Key: i=kees@kernel.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 Both memdup_user() and vmemdup_user() handle allocations that are regularly used for exploiting use-after-free type confusion flaws in the kernel (e.g. prctl() PR_SET_VMA_ANON_NAME[1] and setxattr[2][3][4] respectively). Since both are designed for contents coming from userspace, it allows for userspace-controlled allocation sizes. Use a dedicated set of kmalloc buckets so these allocations do not share caches with the global kmalloc buckets. After a fresh boot under Ubuntu 23.10, we can see the caches are already in active use: # grep ^memdup /proc/slabinfo memdup_user-8k 4 4 8192 4 8 : ... memdup_user-4k 8 8 4096 8 8 : ... memdup_user-2k 16 16 2048 16 8 : ... memdup_user-1k 0 0 1024 16 4 : ... memdup_user-512 0 0 512 16 2 : ... memdup_user-256 0 0 256 16 1 : ... memdup_user-128 0 0 128 32 1 : ... memdup_user-64 256 256 64 64 1 : ... memdup_user-32 512 512 32 128 1 : ... memdup_user-16 1024 1024 16 256 1 : ... memdup_user-8 2048 2048 8 512 1 : ... memdup_user-192 0 0 192 21 1 : ... memdup_user-96 168 168 96 42 1 : ... Link: https://starlabs.sg/blog/2023/07-prctl-anon_vma_name-an-amusing-heap-spray/ [1] Link: https://duasynt.com/blog/linux-kernel-heap-spray [2] Link: https://etenal.me/archives/1336 [3] Link: https://github.com/a13xp0p0v/kernel-hack-drill/blob/master/drill_exploit_uaf.c [4] Signed-off-by: Kees Cook --- mm/util.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/mm/util.c b/mm/util.c index 28c5356b9f1c..29189f48ee04 100644 --- a/mm/util.c +++ b/mm/util.c @@ -198,6 +198,16 @@ char *kmemdup_nul(const char *s, size_t len, gfp_t gfp) } EXPORT_SYMBOL(kmemdup_nul); +static kmem_buckets *user_buckets __ro_after_init; + +static int __init init_user_buckets(void) +{ + user_buckets = kmem_buckets_create("memdup_user", 0, 0, INT_MAX, NULL); + + return 0; +} +subsys_initcall(init_user_buckets); + /** * memdup_user - duplicate memory region from user space * @@ -211,7 +221,7 @@ void *memdup_user(const void __user *src, size_t len) { void *p; - p = kmalloc_track_caller(len, GFP_USER | __GFP_NOWARN); + p = kmem_buckets_alloc_track_caller(user_buckets, len, GFP_USER | __GFP_NOWARN); if (!p) return ERR_PTR(-ENOMEM); @@ -237,7 +247,7 @@ void *vmemdup_user(const void __user *src, size_t len) { void *p; - p = kvmalloc(len, GFP_USER); + p = kmem_buckets_valloc(user_buckets, len, GFP_USER); if (!p) return ERR_PTR(-ENOMEM);