From patchwork Tue Mar 5 10:10:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 13581909 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BC8A5C54798 for ; Tue, 5 Mar 2024 10:10:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 48B746B008C; Tue, 5 Mar 2024 05:10:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3EDB56B0092; Tue, 5 Mar 2024 05:10:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 217946B0093; Tue, 5 Mar 2024 05:10:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 071A36B008C for ; Tue, 5 Mar 2024 05:10:33 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id C420680D70 for ; Tue, 5 Mar 2024 10:10:32 +0000 (UTC) X-FDA: 81862565904.26.982A44D Received: from mail-pf1-f181.google.com (mail-pf1-f181.google.com [209.85.210.181]) by imf28.hostedemail.com (Postfix) with ESMTP id 034E7C0015 for ; Tue, 5 Mar 2024 10:10:30 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=M9NwuOVq; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf28.hostedemail.com: domain of keescook@chromium.org designates 209.85.210.181 as permitted sender) smtp.mailfrom=keescook@chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709633431; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BeUYrlEBxVnkyCc/znLfwIQ49KwaP7QNJJ/emVdVuls=; b=Xo+io59PmeYd2dpnJhNjpCrAX7w3jDBAue5zrf87MxZV7K1RkFkB4rPxRBpl0BcAJ5xKGL bEW51lS3poDO/ikw2Hq5C4X0hg2ZO52TqfP2WXUjv9dPsPY/6AeBznWx7CWZmuz5Gb0yjL dWRHDKVdYviYTup6CMZMwpgwMARd+ZY= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=M9NwuOVq; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf28.hostedemail.com: domain of keescook@chromium.org designates 209.85.210.181 as permitted sender) smtp.mailfrom=keescook@chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709633431; a=rsa-sha256; cv=none; b=mXT6EdPXkKpt4zi66B4QgBdtOfM7t19SyYx10SlhGZyY79OYHygF0cNnae0C+LGoNztPrf qEucjLOk2KdKxPViEEkCRCkxOiF3MsfWHith1X9cz0c3UBs7rHCFFYKP9SCRWF0UShz8pO Jyz9Shym7mHDZox5hTKtMScbc7cafmM= Received: by mail-pf1-f181.google.com with SMTP id d2e1a72fcca58-6e5c81ccfb9so2708909b3a.3 for ; Tue, 05 Mar 2024 02:10:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1709633430; x=1710238230; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BeUYrlEBxVnkyCc/znLfwIQ49KwaP7QNJJ/emVdVuls=; b=M9NwuOVqEfXPzYqqZofuANr6XcC5ZNhd9Af3EsJn2Jdvg1ps2J80jIhqNmxBSRuWX3 sZbn5JSAIGodujz1WS7ZPRa6iSS4YYXTjuBw3cAMXkjmLAVDBGknYfLFCjVjJVP2tN11 tg1HWS0jPkJ56dI/ltLaJPbUWBfwX02dTj1iE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709633430; x=1710238230; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BeUYrlEBxVnkyCc/znLfwIQ49KwaP7QNJJ/emVdVuls=; b=R2gLlN6SrhO7braekYkA9JOhMIOvOQBkjZK8lFHFcnwav5T2915MHuVawH2i8An3Vf cx/KwnKssw+ZqUy9dF/wjywIGwJ/Aiiqt74Br15Y/QW6Xxyv86VPmh17MD8+eXVcB5Z4 VjZ319emaUc04VYZ6K1fiUYqfiOBmNDqsgjwn+HBVaAY4JAhyRPqPzOfXL+LlkZRAOxm SPKeUR2YnYpyjQeqMFqxqERFj9ir1UGTd3bYtgFHlQ5hK66X8xD8Nda7UpVh4ZNhZ287 CrW4KFYjC5MVyWlJ6a0IKF7hG+hVrYqZKi2j54VHIyeErtNzqvWJZAEzRLZsJe3TOaM5 c6lg== X-Forwarded-Encrypted: i=1; AJvYcCUZLV2hV3rXdkLu52yq5UNe8+3WoTinVHS6B/5E00GDiTuThsgsxAXJydjXMLJzvLyGXPVCiQ/seDzBxjp7MThDYeg= X-Gm-Message-State: AOJu0YxaNw50xOnVn++fng6cYXFSdmC5HlD67DQcBGFb5hvoTMqj8WGa RmXVTN3SpqrRBxr2YqrLXrBBhFmYqZT4ohvsC981JzH7nWA3YhshTBhtYeEcJQ== X-Google-Smtp-Source: AGHT+IEA1CSyGq/q0upjn8FhCc79X1Sz1fnRam7a7llwA6f0dlAvS0keaevB9x35pohBL6JFKtxYYg== X-Received: by 2002:a05:6a20:244c:b0:1a1:4cbe:bf74 with SMTP id t12-20020a056a20244c00b001a14cbebf74mr1486585pzc.15.1709633429921; Tue, 05 Mar 2024 02:10:29 -0800 (PST) Received: from www.outflux.net ([198.0.35.241]) by smtp.gmail.com with ESMTPSA id g5-20020a170902c38500b001dc78455383sm10146541plg.223.2024.03.05.02.10.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Mar 2024 02:10:27 -0800 (PST) From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, "GONG, Ruiqi" , Xiu Jianfeng , Suren Baghdasaryan , Kent Overstreet , Jann Horn , Matteo Rizzo , linux-kernel@vger.kernel.org, linux-hardening@vger.kernel.org Subject: [PATCH v2 4/9] slab: Introduce kmem_buckets_create() Date: Tue, 5 Mar 2024 02:10:20 -0800 Message-Id: <20240305101026.694758-4-keescook@chromium.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240305100933.it.923-kees@kernel.org> References: <20240305100933.it.923-kees@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=5125; i=keescook@chromium.org; h=from:subject; bh=y3vf/uqNKTNnOBLgqwE3tYFT2Luy4iHXIHzaxbjeLuY=; b=owEBbAKT/ZANAwAKAYly9N/cbcAmAcsmYgBl5u+QPWE1fKjC5eCeZH4vakqfDEWTI/IohhLqT GoUV/5ej76JAjIEAAEKAB0WIQSlw/aPIp3WD3I+bhOJcvTf3G3AJgUCZebvkAAKCRCJcvTf3G3A Jm/UD/jfSpO31HjXWApCYeolkFhb19QZEq8SrIOIqsqGoGZVAU3k6RWg8yQniQMy3yngBQfa2Ww h/QK0A6lEPzXyLI42kPBmua1RHFrT+Vqe3/1WofvSSsTESICw9TsnzitVinrUOM0nQo51nWISXI rTAc2Dqc1+eteEhuO2mBf61hDJZKYlE3dAE5Oy4yiPLQFOXU363T3VyPS1+jHmjCX9a7U/Fwp1L RE0Jo+IrwYWkV7JCru2Lf9QUOZcLZe1kaDT1S8KGCLBUj0qGfSQ0+6oYCVle7ZtWnAgM69VdfEM u0aSFr9DMPQLnL8DW+S2i4Y7E5d0OaG/m+vACFCAyEOjbMvqBotLX2oVqDLe3wNMWZN/GBxUDdu 9D/jUe03FvtktaMo2UTni3Qsx3d0boMxXGArJ5wo+vyGDDDBs8hxF/3ZhnEd58EUf4/WPWb7Hax 2EybOpRinpRqyal2KGNFPDIBOG4YgGqWYgGXRb3pw4pH6eFdr/PLQhggawSo/Wg96qL8WHZRvZ8 Ndwaph2SCDS3DM+r5nhF8mm7Tw5P9exoRn+a0tJNyv0i8CcN7W2IbJjkG9d/LV/uokfH1SLJWNJ VY+Op6bioLEyekWFsA4tTJyWA5wDB2UiYy2llni7suxYh5HYAzl4ULcPr/X6iHSyvXfscabaGjQ iPexMVWCtL cF/ X-Developer-Key: i=keescook@chromium.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 034E7C0015 X-Stat-Signature: cyc4m3uswxzdgncnxdmatn9h519p3u49 X-HE-Tag: 1709633430-892272 X-HE-Meta: U2FsdGVkX1/xnrxuS/dkWEYL3rk30GmGLvkFJiOqFqjfUueAEPL6bZLe6iBGi1gguiG6wVm0McKDADZFH1Ta3PPLf9H8DprW+amRRckOFGO5YLRbHsak9HcdTtgNMSeXASQOmk4BrfvId2HJSX0CQMvk9vNrF8zaZOypT8oyMQlZE1kHMzzuIAKxhnQYydv0HMpobmc5FsdExUjpUbf5C63ih2biiw920uptjo7pZoDAprd/114hs4dAIrXlIxcC4FFOt1b0vzmFyMGeLRWV12dYTHHrX2bl0Scra3iBgkJpAOVcpPYI8FE8tIBklA3Zymq21Kr4IqGOoiRzx/mYmf73nARqJqztXzrsxXTROOSWDmoVfd3bHpV1xhEuj8DLgZOViYeO0GoQSeZAtyJOHr/kFKA0Ug5xmXx7opWvccpus1AiX3z4CqqaQxkP4196Gn5EHy7W+NhggbC1kvYU6XW35mbTZIbs/OjMxuxWkSZdg+0suCJtJX/8Wol+IwrvHmDLwOeLF8lturx+CgXEaZZADGl93u9blMwHyj1d3x7vkwtDu5Tsx924Q45Y8l9s5UEygAWqb4cY9mzFS7+PBoVqTu0adLNX9Het/3ivjqh9dfXvOsht7edpO410eahJgcLl0n1WMkxlDBaA7iSUQqoDOXSyJNZLuEx1EX9SWVSOYNOd6KaUhPnLVuRz3Kgb+sALT70AjLY2I5wn9iCWkJdBxefECxh8PyO9v8J0TInhVpjnI5S42Y/t0AGtxVFl5gXh4N+QPrbLNIGzzZbM6xgNmK4oPXsPK1YoFjSHrW0xoJ+LpYKa99a1T/HSHxtr2XJ7d+Pf6WTidVlNyNTOvdA88vrZGdIu0ar/MvCS+4l/WdptI1KH04aI3CETabr8BH8GZ3PhGAoLDzpSM5nQzxV4jxnS34ZLghkVQWkLNxqw/IHIDYSK0ZWyyqI34ar5k/kXHCGOx9m+BETJ8Gl QqJHGR8r wnr76KB2O+5IZl7awYlg4NBTSFZdR0+b9yyNiSEU6TeIjLIVUFLEsRKi8Y8lA8leW8AAfoBl9ZD4FI5Sgj7ICfgIl5ZvvfBRqZrhbUK9QEfp/9BeuQ6ToBcXA0BR3kOeMIeUxwuSQB9mXkYgXlFPW+pr9EqKNB7q33gtwMbCB8MkH3vgho7YwrQ8GV9Zi5exruWxDfwnrdxQZamsEL9nhlSIudtf+6mJGx5GCO+hRl4NbvhRET1SJDLRfHA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Dedicated caches are available For fixed size allocations via kmem_cache_alloc(), but for dynamically sized allocations there is only the global kmalloc API's set of buckets available. This means it isn't possible to separate specific sets of dynamically sized allocations into a separate collection of caches. This leads to a use-after-free exploitation weakness in the Linux kernel since many heap memory spraying/grooming attacks depend on using userspace-controllable dynamically sized allocations to collide with fixed size allocations that end up in same cache. While CONFIG_RANDOM_KMALLOC_CACHES provides a probabilistic defense against these kinds of "type confusion" attacks, including for fixed same-size heap objects, we can create a complementary deterministic defense for dynamically sized allocations. In order to isolate user-controllable sized allocations from system allocations, introduce kmem_buckets_create(), which behaves like kmem_cache_create(). (The next patch will introduce kmem_buckets_alloc(), which behaves like kmem_cache_alloc().) Allows for confining allocations to a dedicated set of sized caches (which have the same layout as the kmalloc caches). This can also be used in the future once codetag allocation annotations exist to implement per-caller allocation cache isolation[1] even for dynamic allocations. Link: https://lore.kernel.org/lkml/202402211449.401382D2AF@keescook [1] Signed-off-by: Kees Cook --- Cc: Vlastimil Babka Cc: Christoph Lameter Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Cc: Andrew Morton Cc: Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: linux-mm@kvack.org --- include/linux/slab.h | 5 +++ mm/slab_common.c | 72 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 77 insertions(+) diff --git a/include/linux/slab.h b/include/linux/slab.h index f26ac9a6ef9f..058d0e3cd181 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -493,6 +493,11 @@ void *kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru, gfp_t gfpflags) __assume_slab_alignment __malloc; void kmem_cache_free(struct kmem_cache *s, void *objp); +kmem_buckets *kmem_buckets_create(const char *name, unsigned int align, + slab_flags_t flags, + unsigned int useroffset, unsigned int usersize, + void (*ctor)(void *)); + /* * Bulk allocation and freeing operations. These are accelerated in an * allocator specific way to avoid taking locks repeatedly or building diff --git a/mm/slab_common.c b/mm/slab_common.c index 1d0f25b6ae91..03ba9aac96b6 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -392,6 +392,74 @@ kmem_cache_create(const char *name, unsigned int size, unsigned int align, } EXPORT_SYMBOL(kmem_cache_create); +static struct kmem_cache *kmem_buckets_cache __ro_after_init; + +kmem_buckets *kmem_buckets_create(const char *name, unsigned int align, + slab_flags_t flags, + unsigned int useroffset, + unsigned int usersize, + void (*ctor)(void *)) +{ + kmem_buckets *b; + int idx; + + if (WARN_ON(!kmem_buckets_cache)) + return NULL; + + b = kmem_cache_alloc(kmem_buckets_cache, GFP_KERNEL|__GFP_ZERO); + if (WARN_ON(!b)) + return NULL; + + flags |= SLAB_NO_MERGE; + + for (idx = 0; idx < ARRAY_SIZE(kmalloc_caches[KMALLOC_NORMAL]); idx++) { + char *short_size, *cache_name; + unsigned int cache_useroffset, cache_usersize; + unsigned int size; + + if (!kmalloc_caches[KMALLOC_NORMAL][idx]) + continue; + + size = kmalloc_caches[KMALLOC_NORMAL][idx]->object_size; + if (!size) + continue; + + short_size = strchr(kmalloc_caches[KMALLOC_NORMAL][idx]->name, '-'); + if (WARN_ON(!short_size)) + goto fail; + + cache_name = kasprintf(GFP_KERNEL, "%s-%s", name, short_size + 1); + if (WARN_ON(!cache_name)) + goto fail; + + if (useroffset >= size) { + cache_useroffset = 0; + cache_usersize = 0; + } else { + cache_useroffset = useroffset; + cache_usersize = min(size - cache_useroffset, usersize); + } + (*b)[idx] = kmem_cache_create_usercopy(cache_name, size, + align, flags, cache_useroffset, + cache_usersize, ctor); + kfree(cache_name); + if (WARN_ON(!(*b)[idx])) + goto fail; + } + + return b; + +fail: + for (idx = 0; idx < ARRAY_SIZE(kmalloc_caches[KMALLOC_NORMAL]); idx++) { + if ((*b)[idx]) + kmem_cache_destroy((*b)[idx]); + } + kfree(b); + + return NULL; +} +EXPORT_SYMBOL(kmem_buckets_create); + #ifdef SLAB_SUPPORTS_SYSFS /* * For a given kmem_cache, kmem_cache_destroy() should only be called @@ -933,6 +1001,10 @@ void __init create_kmalloc_caches(slab_flags_t flags) /* Kmalloc array is now usable */ slab_state = UP; + + kmem_buckets_cache = kmem_cache_create("kmalloc_buckets", + sizeof(kmem_buckets), + 0, 0, NULL); } /**