From patchwork Mon Mar 4 18:49:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 13581008 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3A1F6C48BF6 for ; Mon, 4 Mar 2024 18:49:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AD33C6B0085; Mon, 4 Mar 2024 13:49:40 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A3FCA6B0092; Mon, 4 Mar 2024 13:49:40 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7C7586B008C; Mon, 4 Mar 2024 13:49:40 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 658426B0085 for ; Mon, 4 Mar 2024 13:49:40 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 3AEA31C0D5F for ; Mon, 4 Mar 2024 18:49:40 +0000 (UTC) X-FDA: 81860245320.04.907ACA5 Received: from mail-pg1-f182.google.com (mail-pg1-f182.google.com [209.85.215.182]) by imf28.hostedemail.com (Postfix) with ESMTP id 69D71C001A for ; Mon, 4 Mar 2024 18:49:37 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=JEJVApmK; spf=pass (imf28.hostedemail.com: domain of keescook@chromium.org designates 209.85.215.182 as permitted sender) smtp.mailfrom=keescook@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709578177; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=cMb+PmcOht+K9M0r7eBe/cOsz5NTpNrdAZ0O4xFPS4c=; b=6/65nHIoFy55DOI3Y8VX9gNfMMXd7qyDZww+VH6Fr1MeONyw7yRdCU6ILZ9TwtLcbAk9R5 bckhjyS1RrRhMMsyh3iwgF6Bs2z7+XoJTYUgYrMXItYF9jhVL7FyBJRlIoe+IXJDwiU3Nx /dHI+B1ObwT0vhQW+E926g9kw9tupiU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709578177; a=rsa-sha256; cv=none; b=KMD1Ix9AOQgbshAzLb1aeUEqxJcYwUhC2V91z5IGg/52vBKTwFF+KDDyV3lRjzFbG8jn2N cVEaEcUubFEfey+1lRn+pNQ3hcinHt8kmWXryYRr2NsoEhCUIVMtEwD0Xe5XEQlrlzJuSx IS0XQMRjf+VoQq5iPcRAbQZs9RDzrXk= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=JEJVApmK; spf=pass (imf28.hostedemail.com: domain of keescook@chromium.org designates 209.85.215.182 as permitted sender) smtp.mailfrom=keescook@chromium.org; dmarc=pass (policy=none) header.from=chromium.org Received: by mail-pg1-f182.google.com with SMTP id 41be03b00d2f7-5dc949f998fso3766145a12.3 for ; Mon, 04 Mar 2024 10:49:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1709578176; x=1710182976; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=cMb+PmcOht+K9M0r7eBe/cOsz5NTpNrdAZ0O4xFPS4c=; b=JEJVApmKZPGvpp1TPsV2rvRkatsJazqcfBr8s8Kp6JJhCQZEZzWYF1ZkKNZzLCzTyq wNmLcih+Bw6sk4b90b9CgjXW++OSH3p3AhdwGqovC2u75953QI+s2jJIGRhp0tL28V9T AKbKx3KTmrxgnYnvx/bK59f8TZOwYkYdZfXzU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709578176; x=1710182976; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cMb+PmcOht+K9M0r7eBe/cOsz5NTpNrdAZ0O4xFPS4c=; b=VYfrMleYm6e/EYuARTUEVu+1BrVz2h14qd6Rg7IuFy6ygmank9qhpb2ohqkllfnoN3 1PKutfWGdigKqTJY4hiCKpafLG1aNGF1Dr165EY/EC9YGegx8SZK9aBmHoAYdOsQNGwE Z9q7L7cUG4L4UiC437K75S4nT5uF9kKnZxGTqK8sIQfezzIi8nUXTw8b4JpvS47QAOhC iaz7ihPAnrtaffzoGOnDcyXZOSxgLqFx21YAyN5BstXstu8W4/5Pb8WW2U0fwvJoZN8V gxQL2n1BsGfdLN3hZA40+JESbQKG2sk7p+mlmQrEQ7dKul+yz1hjpQbCZ19vD6hn81cS 6UVQ== X-Forwarded-Encrypted: i=1; AJvYcCWczyWiy9QIFibO8A8aiqQX5ZjGu2CofanW0YtSal0wX2jsl7ySEVTxoYmCN2GR3810vYINLLbKdnewomBtBmzr6a0= X-Gm-Message-State: AOJu0YxRxHBMhsolqg2p7k+5KVswPbZ+ssMq/kd3NPsSY8DKsTf4JUf8 9lNQbpA9bjb2dacDyBfDX/k2NqvXXIaWOZC5Pov71IICHJJPnjYoexWY5tsVoQ== X-Google-Smtp-Source: AGHT+IHXfzDDi+Jk/MGWJiolbgqaEdJ79sjoSCtqB+82MCn/uMPrW4nNSDO9Xzsngfqy/Yg749crOw== X-Received: by 2002:a05:6a20:a11a:b0:1a1:4a21:1fe3 with SMTP id q26-20020a056a20a11a00b001a14a211fe3mr1759584pzk.0.1709578176143; Mon, 04 Mar 2024 10:49:36 -0800 (PST) Received: from www.outflux.net ([198.0.35.241]) by smtp.gmail.com with ESMTPSA id z12-20020aa785cc000000b006e56e5c09absm7810419pfn.14.2024.03.04.10.49.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Mar 2024 10:49:34 -0800 (PST) From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, "GONG, Ruiqi" , Xiu Jianfeng , Suren Baghdasaryan , Kent Overstreet , Christian Brauner , Al Viro , Jan Kara , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-hardening@vger.kernel.org Subject: [PATCH 1/4] slab: Introduce dedicated bucket allocator Date: Mon, 4 Mar 2024 10:49:29 -0800 Message-Id: <20240304184933.3672759-1-keescook@chromium.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240304184252.work.496-kees@kernel.org> References: <20240304184252.work.496-kees@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=5798; i=keescook@chromium.org; h=from:subject; bh=h85WkZi4o3uylzu/3xQb8d5MHb8DhVgIptaT/36yfQM=; b=owEBbQKS/ZANAwAKAYly9N/cbcAmAcsmYgBl5he8a2IWxCB3p2pd4ASVvCAfbFqAl2DgpfxWQ QjMAXgEaQyJAjMEAAEKAB0WIQSlw/aPIp3WD3I+bhOJcvTf3G3AJgUCZeYXvAAKCRCJcvTf3G3A Ju0qD/9wLY1KtAVlKpiIcbRaHdOPbTRV4/D+hFxL5B9cNYkqbIS1fUdUJI71wBUD6kJnhCd9kCz zY7AbWXcvcgkl2cA36q3sg5ftbAUAugShF8SRrO7cuJBDBjz61Sd3q8a2ajePTuRLXO+IkiOZSW fEs0uw7D3bTOvVIQAaqxYIRhm5g0wDeFJcikNAE06pyX++iA/ePzCLUTui8hhrErnor+Eok1Gio km/LhI+jGhbmoV8EFNFn5yCij/hBhmf/m80ThCIvRs4Lrfs+Jc6NOeSHI0vjyiNxh2dY+XB5kpD yrScvBaBU8MD0PtoCFyo3sqIlhB8epaO9WymgCqRuF+mYuac6RjWgjZh2wBApo+SwpQFUb8g2fH as59RRHY0OQj+s75GtR8Ra9PHvO03ZpOGrKZj/tZqCkbIt92ysyof2t2O2n5yBLKAnTWrUAbw5g SUF7DF0cRG3BfxkZknrhUe6vrhx7uHHB3EHgs3aqEPbc15a38JptIoGR6vga6dtfq38DvPQguDC 2FQBjQRAQbIhYnATKR31hLJTYiwQ2n+HwHoOW0NawX/LV5lAFUf/Z6+hHodr5UK8PFA6Nd+44eM BnNhe+/usfeJyldh8jlzjBfT3UJIcUYGR+sfEjSE48hTeze4iwwNvv/OMOHmv417pCMNY8ohvDd m7Otqn2ryvvKzHg== X-Developer-Key: i=keescook@chromium.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 X-Rspamd-Queue-Id: 69D71C001A X-Rspam-User: X-Stat-Signature: ifguo5ptxh6fz4mcegqtekdikmxg7kb8 X-Rspamd-Server: rspam03 X-HE-Tag: 1709578177-719155 X-HE-Meta: U2FsdGVkX19qG88lSPLYUtGQ++RWpVw1bfjX5cgWrCfGdi0e8NTIn7CFGK3WEpQf27ufY1czNVyXbh2mbTMIoKhkgucwbqH3lPpO+ZbKcmllme3gsqWDfBb5AcyAI/tJ84E+KUuhwxKrfyz+dzDaGjP4LVilUzgeOki/My6aI6Y9YPnw/gERUSVjSOzdvD2CLXHeVPt5NPjTmi3gB3hKnaCAyvDTPQJQFBYLayxrb5d6bXGFILn0tAFsX8sIw6Rxe5VcLv+Z8uBOJBiDNh674Nos84Ct/rd9X8p1BYGeAcWcMwBSzBZ+EwqdoeQ54Ue+QBuie6GvNnkKYh5H6H05YzYwVR6vh7IzQ8bsbgy1czaGUSaT4eLhjPZ9nOw/YQhd2msXk9iR4W0+LYC1RCfEIMUgFE9owYLjh6mS9u9c6JClz9dV301mdPSz833Xd7DGHLJrsqNaLle/cYxx8ldjdCarRpy0M+59tEZd1ulNEKADkp1XzYQS2wqnJlHJ/Bkss3x9pzUFqBq29qcWC/GsD0QfXinZr+Xp9pkcQ5vcCALSwrfBr2hwKvoIbQs1RC0Trqe4ODthbo13OXmoOoOV/9bzRsTatp1eP7xUtwFY4VeD6RI8o+i/tgTc74MHfYJOFSi3d0Tnvp6wQKC10VFQ3qh20bJyXIci6/AiaLPa+xIkLWW/0pyvITLr/L/xiFEganTykei65c3eXrECnJ+ucTXyy1QpC1a4+rRvZXeZ3o8wHcy2yQmZR2tAJOFlkbF0vB5T770rtsDhJXEGal+nKeD6DKwVUzVZIkIs4095tybCtx0SFf5F3Tbvrwl78NwOzmEmexBvf22lSdSMynUP3MUeUJpZvgXa1E6BgbwvvOpRCdWDTv8fb6hty2RjlozZpAW+KBoUOsSbhYPQmFkdrrCIGJZDNq8BxHX/U/dnSw9A5fGviVHOoK4M8Gqto1o2xy9AcFddLXZ+5sToWbG XDSLCnvJ R/RaCURJAfuYnWINMCEk/As5DeU3XC3MjQAgeK+cMKgwPcFYT4ZPqO1JMYfTbu+Nd8flD5z+lU6Eb77YyQJQ3YGeFsp0wv0/3O93bes/R3StbmI8PlPduvgFCb01b/EqU+W7HbnXsjMn84/TwUIcYX6rH0u0KY42kFrOH8KiGMTFnwQ7ikEiWcjfa+VsrnbEK2OJL X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Dedicated caches are available For fixed size allocations via kmem_cache_alloc(), but for dynamically sized allocations there is only the global kmalloc API's set of buckets available. This means it isn't possible to separate specific sets of dynamically sized allocations into a separate collection of caches. This leads to a use-after-free exploitation weakness in the Linux kernel since many heap memory spraying/grooming attacks depend on using userspace-controllable dynamically sized allocations to collide with fixed size allocations that end up in same cache. While CONFIG_RANDOM_KMALLOC_CACHES provides a probabilistic defense against these kinds of "type confusion" attacks, including for fixed same-size heap objects, we can create a complementary deterministic defense for dynamically sized allocations. In order to isolate user-controllable sized allocations from system allocations, introduce kmem_buckets_create() and kmem_buckets_alloc(), which behave like kmem_cache_create() and like kmem_cache_alloc() for confining allocations to a dedicated set of sized caches (which have the same layout as the kmalloc caches). This can also be used in the future once codetag allocation annotations exist to implement per-caller allocation cache isolation[1] even for dynamic allocations. Link: https://lore.kernel.org/lkml/202402211449.401382D2AF@keescook [1] Signed-off-by: Kees Cook --- Cc: Vlastimil Babka Cc: Christoph Lameter Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Cc: Andrew Morton Cc: Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: linux-mm@kvack.org --- include/linux/slab.h | 26 ++++++++++++++++++ mm/slab_common.c | 64 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 90 insertions(+) diff --git a/include/linux/slab.h b/include/linux/slab.h index b5f5ee8308d0..4a4ff84534be 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -492,6 +492,16 @@ void *kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru, gfp_t gfpflags) __assume_slab_alignment __malloc; void kmem_cache_free(struct kmem_cache *s, void *objp); +struct kmem_buckets { + struct kmem_cache *caches[ARRAY_SIZE(kmalloc_caches[KMALLOC_NORMAL])]; +}; + +struct kmem_buckets * +kmem_buckets_create(const char *name, unsigned int align, slab_flags_t flags, + unsigned int useroffset, unsigned int usersize, + void (*ctor)(void *)); + + /* * Bulk allocation and freeing operations. These are accelerated in an * allocator specific way to avoid taking locks repeatedly or building @@ -594,6 +604,22 @@ static __always_inline __alloc_size(1) void *kmalloc(size_t size, gfp_t flags) return __kmalloc(size, flags); } +static __always_inline __alloc_size(2) +void *kmem_buckets_alloc(struct kmem_buckets *b, size_t size, gfp_t flags) +{ + unsigned int index; + + if (size > KMALLOC_MAX_CACHE_SIZE) + return kmalloc_large(size, flags); + if (WARN_ON_ONCE(!b)) + return NULL; + index = kmalloc_index(size); + if (WARN_ONCE(!b->caches[index], + "missing cache for size %zu (index %d)\n", size, index)) + return kmalloc(size, flags); + return kmalloc_trace(b->caches[index], flags, size); +} + static __always_inline __alloc_size(1) void *kmalloc_node(size_t size, gfp_t flags, int node) { if (__builtin_constant_p(size) && size) { diff --git a/mm/slab_common.c b/mm/slab_common.c index 238293b1dbe1..6002a182d014 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -392,6 +392,66 @@ kmem_cache_create(const char *name, unsigned int size, unsigned int align, } EXPORT_SYMBOL(kmem_cache_create); +static struct kmem_cache *kmem_buckets_cache __ro_after_init; + +struct kmem_buckets * +kmem_buckets_create(const char *name, unsigned int align, + slab_flags_t flags, + unsigned int useroffset, unsigned int usersize, + void (*ctor)(void *)) +{ + struct kmem_buckets *b; + int idx; + + if (WARN_ON(!kmem_buckets_cache)) + return NULL; + + b = kmem_cache_alloc(kmem_buckets_cache, GFP_KERNEL|__GFP_ZERO); + if (WARN_ON(!b)) + return NULL; + + for (idx = 0; idx < ARRAY_SIZE(kmalloc_caches[KMALLOC_NORMAL]); idx++) { + char *short_size, *cache_name; + unsigned int size; + + if (!kmalloc_caches[KMALLOC_NORMAL][idx]) + continue; + + size = kmalloc_caches[KMALLOC_NORMAL][idx]->object_size; + if (!size) + continue; + + short_size = strchr(kmalloc_caches[KMALLOC_NORMAL][idx]->name, '-'); + if (WARN_ON(!short_size)) + goto fail; + + cache_name = kasprintf(GFP_KERNEL, "%s-%s", name, short_size + 1); + if (WARN_ON(!cache_name)) + goto fail; + + b->caches[idx] = kmem_cache_create_usercopy(cache_name, size, + align, flags, useroffset, + min(size - useroffset, usersize), ctor); + kfree(cache_name); + if (WARN_ON(!b->caches[idx])) + goto fail; + } + + return b; + +fail: + for (idx = 0; idx < ARRAY_SIZE(kmalloc_caches[KMALLOC_NORMAL]); idx++) { + if (b->caches[idx]) { + kfree(b->caches[idx]->name); + kmem_cache_destroy(b->caches[idx]); + } + } + kfree(b); + + return NULL; +} +EXPORT_SYMBOL(kmem_buckets_create); + #ifdef SLAB_SUPPORTS_SYSFS /* * For a given kmem_cache, kmem_cache_destroy() should only be called @@ -934,6 +994,10 @@ void __init create_kmalloc_caches(slab_flags_t flags) /* Kmalloc array is now usable */ slab_state = UP; + + kmem_buckets_cache = kmem_cache_create("kmalloc_buckets", + sizeof(struct kmem_buckets) * ARRAY_SIZE(kmalloc_info), + 0, 0, NULL); } /**