From patchwork Mon Jul 1 19:13:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 13718562 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ADD76C2BD09 for ; Mon, 1 Jul 2024 19:13:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 88D186B009A; Mon, 1 Jul 2024 15:13:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 751796B00A0; Mon, 1 Jul 2024 15:13:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 334106B009A; Mon, 1 Jul 2024 15:13:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id E17E36B009E for ; Mon, 1 Jul 2024 15:13:08 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 99A351C3641 for ; Mon, 1 Jul 2024 19:13:08 +0000 (UTC) X-FDA: 82292131656.26.4636ED3 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf24.hostedemail.com (Postfix) with ESMTP id C1585180011 for ; Mon, 1 Jul 2024 19:13:06 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=RpSfmTgx; spf=pass (imf24.hostedemail.com: domain of kees@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=kees@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1719861175; a=rsa-sha256; cv=none; b=ElCMlZh2EEbHVQNExGE17OF5Z28KKuCjxnugIkElZMiFtNiNKh9py/fW7JgekDfOcpOv62 V08f2skEBYAI2iOSp8uWOucVlZj7f7gcAYWfxiKl3UqFJUqPmGkAbrlo3mmEyf3vP/HKnw 6dxxSK9aRevtJEbxv8a5zPN5OM2Qc98= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=RpSfmTgx; spf=pass (imf24.hostedemail.com: domain of kees@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=kees@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1719861175; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IL8BUaVY4e889rtWHb5bVtbIQfIcAxl0t0VbfdZZ2UE=; b=6W3qmOPTqT7BmObXz94BQwYek8r+WSup7VlbmqpVs0LKKbESFJY2YN3AlBmwXJ0WuWuMqD 9/wnu5Z+X7KCMj8XzNS6m7Z9RF6xfH/sjqXk69pCSudSADaHb1dX1DFwwactbBg5PsNqCE AxN9AWziMdBtglOsJxbwD4DzJ65iass= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 8698F61784; Mon, 1 Jul 2024 19:13:05 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id BEF06C4AF0C; Mon, 1 Jul 2024 19:13:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1719861184; bh=iLr1JO1gVFtydBJDlW/Z6SdDIb8sQtIWfFXCMaocDcg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=RpSfmTgxm2O1XtuIFl+3ZxeSoxYypsrEU0+nDkunMVjz9fqvzNZXImfKfOgbhxqvZ CWJEEidhNGTyifG0nR+rXVJIx/A2qesIuRGrcbMilA4qSyFRmRnfeGtpcB6RwcdXAO 8FlpEO+3qZiqtJZNOnTdQuAwR7KU7OgZ5Fyv++2LC0iKVbRQ6H3cWeLFzRXFvoqnM0 eSrGAf0KiZ/UCgZ8S2+lv2V61tOvs+nM9c+XIVf2D9KThxnpza0obf9zEimu91+0nu 5mMnT0ywl5hdZUH+bX9G3igqs+pfRNBBTHgMRtShjduDuQlJt4j1RjRb3SuLNeQ4xo mLYYJCU2yV6Ew== From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , "GONG, Ruiqi" , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , jvoisin , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Xiu Jianfeng , Suren Baghdasaryan , Kent Overstreet , Jann Horn , Matteo Rizzo , Thomas Graf , Herbert Xu , "Gustavo A. R. Silva" , Jonathan Corbet , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-hardening@vger.kernel.org, netdev@vger.kernel.org Subject: [PATCH v6 4/6] mm/slab: Introduce kmem_buckets_create() and family Date: Mon, 1 Jul 2024 12:13:01 -0700 Message-Id: <20240701191304.1283894-4-kees@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240701190152.it.631-kees@kernel.org> References: <20240701190152.it.631-kees@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=8116; i=kees@kernel.org; h=from:subject; bh=iLr1JO1gVFtydBJDlW/Z6SdDIb8sQtIWfFXCMaocDcg=; b=owEBbQKS/ZANAwAKAYly9N/cbcAmAcsmYgBmgv++kshPnyXBQpYmXkhyaUZvkPmyCCbIBrXnN 5/UgqpveV2JAjMEAAEKAB0WIQSlw/aPIp3WD3I+bhOJcvTf3G3AJgUCZoL/vgAKCRCJcvTf3G3A Ji6UEACxYLIBboDvKj5qoLIIzqp3hMa09xT5pXc8GbKmY2/GyXBftV83vive0p+5epvrpOMcCJj UdH1IPLxIdnSKJP/L/8+Rv9jj1BGjR6WTWWOv7GcNtMFdeXzAuh6c/mxpLYV8FRSYvFkXuhbab4 ERTeCuecezcFkNxqUYpcRO75k/eyY5twXVS4i42r/VGX1MOQOC00xBBZI4QqwAirHWU7TzmshBX t9dPiM/LqzE54MRSghuxmUFpVavRcaIGqE9gjNIEeV7ELNEBvT3UpdRuImMkWoxNKQE7he5T6h/ NtO+zOa14B1oBkp4YFjZHLuorD9gkhEsxIVkaPxe7eniA1a4r8s8/IY2/MbMeznYldyYmrNsxwY jbAGyovGKjvR/UWHJSNqT7X2ImJ2UdEt8dkJ8Zck+crZ4wBGPUnPgbANbT/OaiY6MSV5H5pwvG0 K7JQntezgoxoQ29Y1GTD0QpiESxigH3wAERxkJaXnMEL59PyEBWblaDhJz76ttKLqon7S79FZpp jAHE5SZWTFjOBBAwCDhsCp0NlD5jAQ70cJkGWgLYmfK8KWb0geMS3TKZs8yGCytCBA8YpTkM1Xc V2jkNV/A54x/ujur4vKgrirvvVJ+uK2WPCIB3AOYVdfxHAL6EknMufnamvya3upXuHwTcD3TU8u RuEkWXQJaZEN9eQ= = X-Developer-Key: i=kees@kernel.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 X-Stat-Signature: my9kcg69tuhhaxgaywbb8undkrntok7c X-Rspamd-Queue-Id: C1585180011 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1719861186-398408 X-HE-Meta: U2FsdGVkX18e2Ea87fw4QUPJ0sFjOXfEYiVm6hB1ha3SDrvdi7IHqh0yyCTHszLB+CfybwRGX8GADpIGf11I7eM0YpvyK4zS0rRS8iEZoYEUztu4Ygpur4DeGUUS3Y6YztZzjVI0ePcZ3PCo9d2W50GImQMdJ9kEgOEpU4cQZQn1vKSYZsN5lRhL00w8/VIomgbWq64v6TGhDe1SI+zYm+3k4nCTvJHA6Y6Qd+ZJjTh5HtkgwORw0k/ELbTBn0tSPQE8u4HR4PCjVXyY1sCp/x7R5WGEGQNQWKZ2j/3UmOm6kBixjYlB32Ucvmyz8uoHFn50/BukwbANjgXFt3AhE1cSnG0I9xOIAWaoqzYW4U8SRlY10FkXWxPAsFVWedH6WnMBnQglpMs4u+98PSfSguGWOX47Ct63RZkiQFGGudxDsbLKipza+M8aE7YT+m7v+JwoGhCRa/ncLR++q3kpL9k3vtm5gqoD2sE251qpM4NUz9BGo4TtadqkgssW//euL5Y0Hilm1k9Cj1CuROT+KDCJFmaeDAYuqfIFTmfxtsZvIP5OmtHv1nk0Rn4HhyHRxHxNpk5bZJTi76b30odGZUXxxvgQUY4O4MGpaBdbXZMUXboQbY4TuAZpj71OdWHpDw/Z+i65tQ0dJ/4H3VpwBdwl4gosJR09A3Go3C8qAYKwWbGqpwL5zdiKivr/D4wqIwS9djmx0xvJCxESlwyQ0RAiRkJ/ABvutg+hjdjMm/aSdwKYDHZWNnIs0F9dtApd7dNuMTvBoHeQWJlGPAvJforF7SQ1eHSCx4F82Z26ehon6eMpte+wJlxcHLDGTNdP6CT5wqFobAVimlaIXrPluEBgqU8E/wgrFbmRO+8lLDBOcXZx3gRpwTAXpE9S8JYA3f/6k1ZoIVCPaGjIqfp+KIehyBmthkzDouHgWvwb76PpFd+PVLe6AvAn9enEq2Bos2kFMy/mwiRY4yvvJHi nZhdrC6Z 17Aj61mBVAxyJWQ1cuOdTx9Tc8gxALwaC/+c/G9J1nfzWROB0TBqEVPP7C3SJQ6MSf5E3cSMvY52e1hZ4XfBv2sYEoGEVMZZn7qkIAzP+psVxQvcfPVyNzXdlztrWppzMFx4K56P8C/43BmJ1+YHj7lqy440/a8j/0STwwWzGVSSqL7UhGybgSq83vGsqb8qunjFvx4n82aRTg0VZ2IbvCar9gq6D/DMyGuU7o7tKqTUpqQ39aocjce1pyCvQo/CxrYIVo6QQOCPQnzc9964vvHLIshi6w9ZYC1zFPDBXg9MfXoFB077mOQ+ffg96QkBEWay/jWoJ02mOrn0lXXj+XZHNpN2UGeYOqRbg9WmmRo/2C7Gt9QO1pVAwdcyr3jLRGP1Y9vaGdtK/c92aHoobDQf+236xw1ReCsIKQ8o/nYoIXTcashY+vEWf0uE9PzLenJqQJV+TIH4wslul0Do8cGOmNw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Dedicated caches are available for fixed size allocations via kmem_cache_alloc(), but for dynamically sized allocations there is only the global kmalloc API's set of buckets available. This means it isn't possible to separate specific sets of dynamically sized allocations into a separate collection of caches. This leads to a use-after-free exploitation weakness in the Linux kernel since many heap memory spraying/grooming attacks depend on using userspace-controllable dynamically sized allocations to collide with fixed size allocations that end up in same cache. While CONFIG_RANDOM_KMALLOC_CACHES provides a probabilistic defense against these kinds of "type confusion" attacks, including for fixed same-size heap objects, we can create a complementary deterministic defense for dynamically sized allocations that are directly user controlled. Addressing these cases is limited in scope, so isolating these kinds of interfaces will not become an unbounded game of whack-a-mole. For example, many pass through memdup_user(), making isolation there very effective. In order to isolate user-controllable dynamically-sized allocations from the common system kmalloc allocations, introduce kmem_buckets_create(), which behaves like kmem_cache_create(). Introduce kmem_buckets_alloc(), which behaves like kmem_cache_alloc(). Introduce kmem_buckets_alloc_track_caller() for where caller tracking is needed. Introduce kmem_buckets_valloc() for cases where vmalloc fallback is needed. Note that these caches are specifically flagged with SLAB_NO_MERGE, since merging would defeat the entire purpose of the mitigation. This can also be used in the future to extend allocation profiling's use of code tagging to implement per-caller allocation cache isolation[1] even for dynamic allocations. Memory allocation pinning[2] is still needed to plug the Use-After-Free cross-allocator weakness (where attackers can arrange to free an entire slab page and have it reallocated to a different cache), but that is an existing and separate issue which is complementary to this improvement. Development continues for that feature via the SLAB_VIRTUAL[3] series (which could also provide guard pages -- another complementary improvement). Link: https://lore.kernel.org/lkml/202402211449.401382D2AF@keescook [1] Link: https://googleprojectzero.blogspot.com/2021/10/how-simple-linux-kernel-memory.html [2] Link: https://lore.kernel.org/lkml/20230915105933.495735-1-matteorizzo@google.com/ [3] Signed-off-by: Kees Cook --- include/linux/slab.h | 12 ++++++ mm/slab_common.c | 96 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 108 insertions(+) diff --git a/include/linux/slab.h b/include/linux/slab.h index 8d0800c7579a..4c083f3196f4 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -549,6 +549,10 @@ void *kmem_cache_alloc_lru_noprof(struct kmem_cache *s, struct list_lru *lru, void kmem_cache_free(struct kmem_cache *s, void *objp); +kmem_buckets *kmem_buckets_create(const char *name, slab_flags_t flags, + unsigned int useroffset, unsigned int usersize, + void (*ctor)(void *)); + /* * Bulk allocation and freeing operations. These are accelerated in an * allocator specific way to avoid taking locks repeatedly or building @@ -681,6 +685,12 @@ static __always_inline __alloc_size(1) void *kmalloc_noprof(size_t size, gfp_t f } #define kmalloc(...) alloc_hooks(kmalloc_noprof(__VA_ARGS__)) +#define kmem_buckets_alloc(_b, _size, _flags) \ + alloc_hooks(__kmalloc_node_noprof(PASS_BUCKET_PARAMS(_size, _b), _flags, NUMA_NO_NODE)) + +#define kmem_buckets_alloc_track_caller(_b, _size, _flags) \ + alloc_hooks(__kmalloc_node_track_caller_noprof(PASS_BUCKET_PARAMS(_size, _b), _flags, NUMA_NO_NODE, _RET_IP_)) + static __always_inline __alloc_size(1) void *kmalloc_node_noprof(size_t size, gfp_t flags, int node) { if (__builtin_constant_p(size) && size) { @@ -808,6 +818,8 @@ void *__kvmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node) #define kvzalloc(_size, _flags) kvmalloc(_size, (_flags)|__GFP_ZERO) #define kvzalloc_node(_size, _flags, _node) kvmalloc_node(_size, (_flags)|__GFP_ZERO, _node) +#define kmem_buckets_valloc(_b, _size, _flags) \ + alloc_hooks(__kvmalloc_node_noprof(PASS_BUCKET_PARAMS(_size, _b), _flags, NUMA_NO_NODE)) static inline __alloc_size(1, 2) void * kvmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags, int node) diff --git a/mm/slab_common.c b/mm/slab_common.c index 9b0f2ef951f1..641cf513d2d1 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -392,6 +392,98 @@ kmem_cache_create(const char *name, unsigned int size, unsigned int align, } EXPORT_SYMBOL(kmem_cache_create); +static struct kmem_cache *kmem_buckets_cache __ro_after_init; + +/** + * kmem_buckets_create - Create a set of caches that handle dynamic sized + * allocations via kmem_buckets_alloc() + * @name: A prefix string which is used in /proc/slabinfo to identify this + * cache. The individual caches with have their sizes as the suffix. + * @flags: SLAB flags (see kmem_cache_create() for details). + * @useroffset: Starting offset within an allocation that may be copied + * to/from userspace. + * @usersize: How many bytes, starting at @useroffset, may be copied + * to/from userspace. + * @ctor: A constructor for the objects, run when new allocations are made. + * + * Cannot be called within an interrupt, but can be interrupted. + * + * Return: a pointer to the cache on success, NULL on failure. When + * CONFIG_SLAB_BUCKETS is not enabled, ZERO_SIZE_PTR is returned, and + * subsequent calls to kmem_buckets_alloc() will fall back to kmalloc(). + * (i.e. callers only need to check for NULL on failure.) + */ +kmem_buckets *kmem_buckets_create(const char *name, slab_flags_t flags, + unsigned int useroffset, + unsigned int usersize, + void (*ctor)(void *)) +{ + kmem_buckets *b; + int idx; + + /* + * When the separate buckets API is not built in, just return + * a non-NULL value for the kmem_buckets pointer, which will be + * unused when performing allocations. + */ + if (!IS_ENABLED(CONFIG_SLAB_BUCKETS)) + return ZERO_SIZE_PTR; + + if (WARN_ON(!kmem_buckets_cache)) + return NULL; + + b = kmem_cache_alloc(kmem_buckets_cache, GFP_KERNEL|__GFP_ZERO); + if (WARN_ON(!b)) + return NULL; + + flags |= SLAB_NO_MERGE; + + for (idx = 0; idx < ARRAY_SIZE(kmalloc_caches[KMALLOC_NORMAL]); idx++) { + char *short_size, *cache_name; + unsigned int cache_useroffset, cache_usersize; + unsigned int size; + + if (!kmalloc_caches[KMALLOC_NORMAL][idx]) + continue; + + size = kmalloc_caches[KMALLOC_NORMAL][idx]->object_size; + if (!size) + continue; + + short_size = strchr(kmalloc_caches[KMALLOC_NORMAL][idx]->name, '-'); + if (WARN_ON(!short_size)) + goto fail; + + cache_name = kasprintf(GFP_KERNEL, "%s-%s", name, short_size + 1); + if (WARN_ON(!cache_name)) + goto fail; + + if (useroffset >= size) { + cache_useroffset = 0; + cache_usersize = 0; + } else { + cache_useroffset = useroffset; + cache_usersize = min(size - cache_useroffset, usersize); + } + (*b)[idx] = kmem_cache_create_usercopy(cache_name, size, + 0, flags, cache_useroffset, + cache_usersize, ctor); + kfree(cache_name); + if (WARN_ON(!(*b)[idx])) + goto fail; + } + + return b; + +fail: + for (idx = 0; idx < ARRAY_SIZE(kmalloc_caches[KMALLOC_NORMAL]); idx++) + kmem_cache_destroy((*b)[idx]); + kfree(b); + + return NULL; +} +EXPORT_SYMBOL(kmem_buckets_create); + #ifdef SLAB_SUPPORTS_SYSFS /* * For a given kmem_cache, kmem_cache_destroy() should only be called @@ -931,6 +1023,10 @@ void __init create_kmalloc_caches(void) /* Kmalloc array is now usable */ slab_state = UP; + + kmem_buckets_cache = kmem_cache_create("kmalloc_buckets", + sizeof(kmem_buckets), + 0, SLAB_NO_MERGE, NULL); } /**