From patchwork Fri Aug 9 07:33:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 13758478 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0098F42A80; Fri, 9 Aug 2024 07:33:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723188790; cv=none; b=EnTZJ/FSPXrbl6wnegowtYiJe/sRhac5ayCKNo3SuQEzVwqpfRc4VnAyg4/isTqBnl67xVUhAb0tpQUnjq2EcI3/pVdl5NDjuiUakHQxBpDoHm1Dfalc1Jj75wcuEfSMmyfUkpBXS/lBKF7HKnuy0nYQUcPmXoTYmMj9d0wrFZ0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723188790; c=relaxed/simple; bh=fH+32u2hbe+aYvteHR+ZPHJXyfY1tBWhDA7Vvn2Or58=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=e+gQmH+GTW/5BSS/Kml+P8c1gpwqkeGj6XTOJew18wkxRZN6di3Q2GvvLgAarX0OqSmBZ+2m9F0ZB60FreAjN2gjOzACXGLFv/tCNc+ZON0ks2D0XuwUCrCkzk4UxqmjsfoKvNjfBmiJLNMW3pNRiL0q2ksWDKPGzf+qgs78MMk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=U1GOg7df; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="U1GOg7df" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8F687C32782; Fri, 9 Aug 2024 07:33:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723188789; bh=fH+32u2hbe+aYvteHR+ZPHJXyfY1tBWhDA7Vvn2Or58=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=U1GOg7dfFb6vCTpdmAesrPycht4b3E2BP61/NHu5KkhADUw+F/qMwpsb+4EE0wwPr L9nvFDqSei8WJyngCv1Q76XygeWaSsjBlLrD6F/LC8/k4VknfL0og/dzm76kmgZgtR 95TPrS8GpHcyJsvMNXhSCnbwb/zN8o2cGRSd7yux2dwYNZdWYITUiYaoPN0QRrAz4J xWzfOO8GT/bjnPzIqghQGJWNHo7TvFmqxfVY9RsL2FSgfXLSt5noN6OfXW0U8mVhWW LIZVWRO8MeNF1Hpm2FoGdrC++nOc8Th3OtGsIxGUZ8FgbOdZ6MVMjlaGpBWyS3Y5Uq lUO1diWANHxyg== From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, Suren Baghdasaryan , Kent Overstreet , "GONG, Ruiqi" , Jann Horn , Matteo Rizzo , jvoisin , Xiu Jianfeng , linux-kernel@vger.kernel.org, linux-hardening@vger.kernel.org Subject: [PATCH 1/5] slab: Introduce kmem_buckets_destroy() Date: Fri, 9 Aug 2024 00:33:02 -0700 Message-Id: <20240809073309.2134488-1-kees@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240809072532.work.266-kees@kernel.org> References: <20240809072532.work.266-kees@kernel.org> Precedence: bulk X-Mailing-List: linux-hardening@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2139; i=kees@kernel.org; h=from:subject; bh=fH+32u2hbe+aYvteHR+ZPHJXyfY1tBWhDA7Vvn2Or58=; b=owGbwMvMwCVmps19z/KJym7G02pJDGlbjxkoHv8l8yD/Mrump/fE6Mr5/aFh0Z63wl+F6swJe 9mx9VNSRykLgxgXg6yYIkuQnXuci8fb9nD3uYowc1iZQIYwcHEKwER0XzEyzOnisNb8799w6KzH TqX4LWYLH5sfsG/JsPy17/be9sMdXxgZDioISWWoefx3MnrGUf7s0jzWyOPSmyWaxZQnrvt/sGs bNwA= X-Developer-Key: i=kees@kernel.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 Modular use of kmem_buckets_create() means that kmem_buckets will need to be removed as well. Introduce kmem_buckets_destroy(), matching kmem_cache_destroy(). Signed-off-by: Kees Cook --- Cc: Vlastimil Babka Cc: Christoph Lameter Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Cc: Andrew Morton Cc: Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: linux-mm@kvack.org --- include/linux/slab.h | 1 + mm/slab_common.c | 17 ++++++++++++++--- 2 files changed, 15 insertions(+), 3 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index eb2bf4629157..86cb61a0102c 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -552,6 +552,7 @@ void kmem_cache_free(struct kmem_cache *s, void *objp); kmem_buckets *kmem_buckets_create(const char *name, slab_flags_t flags, unsigned int useroffset, unsigned int usersize, void (*ctor)(void *)); +void kmem_buckets_destroy(kmem_buckets *b); /* * Bulk allocation and freeing operations. These are accelerated in an diff --git a/mm/slab_common.c b/mm/slab_common.c index 40b582a014b8..fc698cba0ebe 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -392,6 +392,19 @@ kmem_cache_create(const char *name, unsigned int size, unsigned int align, } EXPORT_SYMBOL(kmem_cache_create); +void kmem_buckets_destroy(kmem_buckets *b) +{ + int idx; + + if (!b) + return; + + for (idx = 0; idx < ARRAY_SIZE(kmalloc_caches[KMALLOC_NORMAL]); idx++) + kmem_cache_destroy((*b)[idx]); + kfree(b); +} +EXPORT_SYMBOL(kmem_buckets_destroy); + static struct kmem_cache *kmem_buckets_cache __ro_after_init; /** @@ -476,9 +489,7 @@ kmem_buckets *kmem_buckets_create(const char *name, slab_flags_t flags, return b; fail: - for (idx = 0; idx < ARRAY_SIZE(kmalloc_caches[KMALLOC_NORMAL]); idx++) - kmem_cache_destroy((*b)[idx]); - kfree(b); + kmem_buckets_destroy(b); return NULL; } From patchwork Fri Aug 9 07:33:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 13758480 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 16C66175D2F; Fri, 9 Aug 2024 07:33:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723188790; cv=none; b=SOZtAw7rVBqTKycKdAReD6J/WoNfRnIVsRZD9zP29CLKW3DgBDgABC503hZfiHT/N01jn3v01Aao0VdmeNo2Pe/HNfpDI2C3+WXGbLDujdBwLLGtzV++nrOqTKZJngWNuooo75ZFPDPWsAH1r2n6v3p8k4XTAvo8JaNgjFsCqfk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723188790; c=relaxed/simple; bh=Bm7wZvIpH1u9Fue5VhYGh2wN4XsW+HXCBPDIfO+4Wi0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=WKjo8ng11sVv8znPDT4PMdaoMTqb3r4379FA/1bLpj1Y4pdaeT/JoKu8Glu8g1HXOtGGMDka1q73frNJGGMrenABcd85mSBO5CoAtomt4tFcHG+wSmqrqYPLvR1V/ZDwS2YNjDum2Yd4HfXZE58PxtIaAvjX4T9oew61yA0xyG4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=QqpCoLXI; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="QqpCoLXI" Received: by smtp.kernel.org (Postfix) with ESMTPSA id A604FC4AF0D; Fri, 9 Aug 2024 07:33:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723188789; bh=Bm7wZvIpH1u9Fue5VhYGh2wN4XsW+HXCBPDIfO+4Wi0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=QqpCoLXIIBMnBfGGkEX09zukRI2+BmDPINyNOYod5uVhbZDarz3N5Gon+RuXzFMAb oPfcGVx2F89WvXdR38xVPnVsVlOUmNcyHLnIz/i0Chy93oj1L4sP0lHM9ULrx3F+Dy 3wtCujBr/ZAjZcrcrOYEQHYox5wX8h0ySRVdiy4kQYmXwl16b/dgOdo7qfIJdcuOze /My+X+Esi8xyVCPP+QA8hTU/aKTo2FyoEfdKL29fwNZw2fSN5v+VcPTVRG6MpbJbIh z6fXqwUOtqJjrwqRQmB4cR3qiejfPc+eHmuFlse3M/B3SuOYmoyKMkjQGsPZiZMAAm X3fYWCL8pp4Yg== From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , Suren Baghdasaryan , Kent Overstreet , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, "GONG, Ruiqi" , Jann Horn , Matteo Rizzo , jvoisin , Xiu Jianfeng , linux-kernel@vger.kernel.org, linux-hardening@vger.kernel.org Subject: [PATCH 2/5] codetag: Run module_load hooks for builtin codetags Date: Fri, 9 Aug 2024 00:33:03 -0700 Message-Id: <20240809073309.2134488-2-kees@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240809072532.work.266-kees@kernel.org> References: <20240809072532.work.266-kees@kernel.org> Precedence: bulk X-Mailing-List: linux-hardening@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1658; i=kees@kernel.org; h=from:subject; bh=Bm7wZvIpH1u9Fue5VhYGh2wN4XsW+HXCBPDIfO+4Wi0=; b=owGbwMvMwCVmps19z/KJym7G02pJDGlbjxke+n5p3ULn87cf9DTKdwmaspUUamrMS9iRlnGLf TIne3NeRykLgxgXg6yYIkuQnXuci8fb9nD3uYowc1iZQIYwcHEKwEQuXGZkeC42Q9N+bpD98SIX x6h7Lvs+C7ZwcS3zP3qIZfUbhd77AQz/IwUeVCk7XGpQ0dpqWr5Ru4/R+rV7RXv5Wob9sVHfDmx hBwA= X-Developer-Key: i=kees@kernel.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 The module_load callback should still run for builtin codetags that define it, even in a non-modular kernel. (i.e. for the cmod->mod == NULL case). Signed-off-by: Kees Cook --- Cc: Suren Baghdasaryan Cc: Kent Overstreet Cc: Vlastimil Babka Cc: Christoph Lameter Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Cc: Andrew Morton Cc: Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: linux-mm@kvack.org --- lib/codetag.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/lib/codetag.c b/lib/codetag.c index 5ace625f2328..ef7634c7ee18 100644 --- a/lib/codetag.c +++ b/lib/codetag.c @@ -125,7 +125,6 @@ static inline size_t range_size(const struct codetag_type *cttype, cttype->desc.tag_size; } -#ifdef CONFIG_MODULES static void *get_symbol(struct module *mod, const char *prefix, const char *name) { DECLARE_SEQ_BUF(sb, KSYM_NAME_LEN); @@ -199,6 +198,7 @@ static int codetag_module_init(struct codetag_type *cttype, struct module *mod) return 0; } +#ifdef CONFIG_MODULES void codetag_load_module(struct module *mod) { struct codetag_type *cttype; @@ -248,9 +248,6 @@ bool codetag_unload_module(struct module *mod) return unload_ok; } - -#else /* CONFIG_MODULES */ -static int codetag_module_init(struct codetag_type *cttype, struct module *mod) { return 0; } #endif /* CONFIG_MODULES */ struct codetag_type * From patchwork Fri Aug 9 07:33:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 13758482 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 16CB4175D36; Fri, 9 Aug 2024 07:33:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723188790; cv=none; b=e3PQ1TMgyyRTqUUjXO5dBZxdBnii/73Ce+e4bKVJP1c79mFl7cWzI3RZhnGdFHb38VhKWBTnyUAAeMwK3hMpVwx8OWkLz7/CBxwxVdOvWwR8n+IO47/WmXM7g1cXxz7mDUkirsu+M2XMVpcDc65W+c1DIk3oix4+3xeUXhNU2Vw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723188790; c=relaxed/simple; bh=Stx3EEnB/9LWtVMs8bPtcAXmif1O8Z/znzWwrIr1yQM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=hMY+M44pGGrhVGVkFaIf0T/tcTeYW4KHA2VKkZOpQwOxJxp1BH9py2i3ZF9VNb8yjwayI/Qatq8bYD4ip7LYuheSqi+5eEHEINkBCiIGLl40p0xOQsL4epAbUQlb+N9mwcwtqHWTLYGSCBa+pZw4ILpPUuMyJQ4bOQn13v1dF7w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=G96xFR9z; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="G96xFR9z" Received: by smtp.kernel.org (Postfix) with ESMTPSA id AB099C4AF10; Fri, 9 Aug 2024 07:33:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723188789; bh=Stx3EEnB/9LWtVMs8bPtcAXmif1O8Z/znzWwrIr1yQM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=G96xFR9zVeh09ObUFoqqZFktxg1j3FtzvLiXQZGmQZW/HVQIm7p5HIUbK5Ww99589 CNxj96mi3eBlCxegp72LfDbiC9rOvnrWCQzO0YvRFp6aR3VIj/SuoeDWAMXhIbxV8O cjeBU/r8ZOz+n88QSTZXJ9cFA1/i28S4PPfS/AKJ2uwN4tSkbb3JKof8G+eosTEybG 485xFvfsK3AuAe/sxtfTl47vL4CU3b2ybBPEAFXxACwNrNDgvU5G2oZNtuTk4AYoyV 9pYD8+LMbuhN6C9vBvYfaJZfhUZngVa/DpU5F+dlB9t9jc6WTqOaLW4mZPlzd8Hk+n ziag+GbKspAEQ== From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , Suren Baghdasaryan , Kent Overstreet , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, "GONG, Ruiqi" , Jann Horn , Matteo Rizzo , jvoisin , Xiu Jianfeng , linux-kernel@vger.kernel.org, linux-hardening@vger.kernel.org Subject: [PATCH 3/5] codetag: Introduce codetag_early_walk() Date: Fri, 9 Aug 2024 00:33:04 -0700 Message-Id: <20240809073309.2134488-3-kees@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240809072532.work.266-kees@kernel.org> References: <20240809072532.work.266-kees@kernel.org> Precedence: bulk X-Mailing-List: linux-hardening@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2359; i=kees@kernel.org; h=from:subject; bh=Stx3EEnB/9LWtVMs8bPtcAXmif1O8Z/znzWwrIr1yQM=; b=owGbwMvMwCVmps19z/KJym7G02pJDGlbjxnOseXU/pJl/43T0XOt5r2jXsvfZqg+sjDOii3I3 vMo2r2zo5SFQYyLQVZMkSXIzj3OxeNte7j7XEWYOaxMIEMYuDgFYCKPwhgZPkd03P/cfODJhK1S TXHcc1wl7ylP8ufytdEOmixz1c7Yg+F/pc/9q0Ub7q7Zv+boUV0X+f1xT5Q6ndp1tt0+78NeFv6 LAwA= X-Developer-Key: i=kees@kernel.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 In order to process builtin alloc_tags much earlier during boot (before register_codetag() is processed), provide codetag_early_walk() that perform a lockless walk with a specified callback function. This will be used to allocate required caches that cannot be allocated on demand. Signed-off-by: Kees Cook --- Cc: Suren Baghdasaryan Cc: Kent Overstreet Cc: Vlastimil Babka Cc: Christoph Lameter Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Cc: Andrew Morton Cc: Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: linux-mm@kvack.org --- include/linux/codetag.h | 2 ++ lib/codetag.c | 16 ++++++++++++++++ 2 files changed, 18 insertions(+) diff --git a/include/linux/codetag.h b/include/linux/codetag.h index c2a579ccd455..9eb1fcd90570 100644 --- a/include/linux/codetag.h +++ b/include/linux/codetag.h @@ -64,6 +64,8 @@ void codetag_lock_module_list(struct codetag_type *cttype, bool lock); bool codetag_trylock_module_list(struct codetag_type *cttype); struct codetag_iterator codetag_get_ct_iter(struct codetag_type *cttype); struct codetag *codetag_next_ct(struct codetag_iterator *iter); +void codetag_early_walk(const struct codetag_type_desc *desc, + void (*callback)(struct codetag *ct)); void codetag_to_text(struct seq_buf *out, struct codetag *ct); diff --git a/lib/codetag.c b/lib/codetag.c index ef7634c7ee18..9d563c8c088a 100644 --- a/lib/codetag.c +++ b/lib/codetag.c @@ -154,6 +154,22 @@ static struct codetag_range get_section_range(struct module *mod, }; } +void codetag_early_walk(const struct codetag_type_desc *desc, + void (*callback)(struct codetag *ct)) +{ + struct codetag_range range; + struct codetag *ct; + + range = get_section_range(NULL, desc->section); + if (!range.start || !range.stop || + range.start == range.stop || + range.start > range.stop) + return; + + for (ct = range.start; ct < range.stop; ct = ((void *)ct + desc->tag_size)) + callback(ct); +} + static int codetag_module_init(struct codetag_type *cttype, struct module *mod) { struct codetag_range range; From patchwork Fri Aug 9 07:33:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 13758481 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1919A176228; Fri, 9 Aug 2024 07:33:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723188790; cv=none; b=imsZ1EDm1kwZH21YMr7eCNzBDPrKz+uMeKpmYf08gUjA1fQJBcQ/A0mlopUPr1eN8t+N0XgZdf0eWvdYNupOGjd96yC0hkPdc7udOOpYaRini9SpiJI92lobYQIiwqXv4PAnBIM/3rDq8zNbX4yMYbaVW6+alg6nFGE8krcUfp0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723188790; c=relaxed/simple; bh=Iabo073v2g0BX5idKAlpNK0sS3Na+mbsF9YuoIGYIVE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Z2QqUaVOzniuxkF5S2sPol8m6xwmCRJ4pdmO9Z9uYU+rQ369/aIOhkWuPt/zVoaXmrcylNT1hW+FTBPEiArVjAaZLmP0wkSOol9+7Z3EUMoPKZNa8vQLqpE65JoIhqnvVkwfgzuUQUcUd3t+efKR+XfiaMUV56cgc0IiHiPE/Bs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=o4q8IAk/; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="o4q8IAk/" Received: by smtp.kernel.org (Postfix) with ESMTPSA id A8720C4AF0F; Fri, 9 Aug 2024 07:33:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723188789; bh=Iabo073v2g0BX5idKAlpNK0sS3Na+mbsF9YuoIGYIVE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=o4q8IAk/qlgqcTCea3BBJdIf8hDOC1XjjLxhcI8dkm1ljzbm+iszrHUacqp7lGQQq jGNBVMkdhPRB0YBHgDKEbwNi4qZIaA7IOQMRouJlraAAcY8ZANl/rh82rIf2q7a7Sk RGDqTlGc0UkZl79HBkzQlQPhNfE3UMMSY31PiSUNZY49KE4SM+jd9QH+P+ipVYPvlr PHnohauWkmIwt2xg5wEQ8yc8x/rD4IETjsdn1fVLzbTajnMZKrEL/FtYFbIOrNxBUu jCkle8QjxtskMzf4qXmks1dSVdfJo3x06Hf3Uia/frmvQwd2b0c1VatOnwxcL02pJN EqUeRe1J4h/QQ== From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , Suren Baghdasaryan , Kent Overstreet , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, "GONG, Ruiqi" , Jann Horn , Matteo Rizzo , jvoisin , Xiu Jianfeng , linux-kernel@vger.kernel.org, linux-hardening@vger.kernel.org Subject: [PATCH 4/5] alloc_tag: Track fixed vs dynamic sized kmalloc calls Date: Fri, 9 Aug 2024 00:33:05 -0700 Message-Id: <20240809073309.2134488-4-kees@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240809072532.work.266-kees@kernel.org> References: <20240809072532.work.266-kees@kernel.org> Precedence: bulk X-Mailing-List: linux-hardening@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=8513; i=kees@kernel.org; h=from:subject; bh=Iabo073v2g0BX5idKAlpNK0sS3Na+mbsF9YuoIGYIVE=; b=owGbwMvMwCVmps19z/KJym7G02pJDGlbjxla/OkOZpqjmTXXV1pxl+T8wjPTGhjbs13unfB2Y DSfGRnZUcrCIMbFICumyBJk5x7n4vG2Pdx9riLMHFYmkCEMXJwCMBFZZ0aG2zfLDrTphUWynG79 HvZ6UuCs3+Fnq13vX2MPMGJ4Pkd5ESPDrh3z73lW79405Y3IHS2OfPYiTYNZKYUfe4ubfrw2nb2 WGQA= X-Developer-Key: i=kees@kernel.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 For slab allocations, record whether the call site is using a fixed size (i.e. compile time constant) or a dynamic size. Report the results in /proc/allocinfo. Improvements needed: - examine realloc routines for needed coverage Signed-off-by: Kees Cook --- Cc: Suren Baghdasaryan Cc: Kent Overstreet Cc: Vlastimil Babka Cc: Christoph Lameter Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Cc: Andrew Morton Cc: Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: linux-mm@kvack.org --- include/linux/alloc_tag.h | 30 ++++++++++++++++++++++++++---- include/linux/slab.h | 16 ++++++++-------- lib/alloc_tag.c | 8 ++++++++ mm/Kconfig | 8 ++++++++ 4 files changed, 50 insertions(+), 12 deletions(-) diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h index 8c61ccd161ba..f5d8c5849b82 100644 --- a/include/linux/alloc_tag.h +++ b/include/linux/alloc_tag.h @@ -20,6 +20,19 @@ struct alloc_tag_counters { u64 calls; }; +#ifdef CONFIG_SLAB_PER_SITE +struct alloc_meta { + /* 0 means non-slab, SIZE_MAX means dynamic, and everything else is fixed-size. */ + size_t sized; +}; +#define ALLOC_META_INIT(_size) { \ + .sized = (__builtin_constant_p(_size) ? (_size) : SIZE_MAX), \ + } +#else +struct alloc_meta { }; +#define ALLOC_META_INIT(_size) { } +#endif + /* * An instance of this structure is created in a special ELF section at every * allocation callsite. At runtime, the special section is treated as @@ -27,6 +40,7 @@ struct alloc_tag_counters { */ struct alloc_tag { struct codetag ct; + struct alloc_meta meta; struct alloc_tag_counters __percpu *counters; } __aligned(8); @@ -74,19 +88,21 @@ static inline struct alloc_tag *ct_to_alloc_tag(struct codetag *ct) */ DECLARE_PER_CPU(struct alloc_tag_counters, _shared_alloc_tag); -#define DEFINE_ALLOC_TAG(_alloc_tag) \ +#define DEFINE_ALLOC_TAG(_alloc_tag, _meta_init) \ static struct alloc_tag _alloc_tag __used __aligned(8) \ __section("alloc_tags") = { \ .ct = CODE_TAG_INIT, \ + .meta = _meta_init, \ .counters = &_shared_alloc_tag }; #else /* ARCH_NEEDS_WEAK_PER_CPU */ -#define DEFINE_ALLOC_TAG(_alloc_tag) \ +#define DEFINE_ALLOC_TAG(_alloc_tag, _meta_init) \ static DEFINE_PER_CPU(struct alloc_tag_counters, _alloc_tag_cntr); \ static struct alloc_tag _alloc_tag __used __aligned(8) \ __section("alloc_tags") = { \ .ct = CODE_TAG_INIT, \ + .meta = _meta_init, \ .counters = &_alloc_tag_cntr }; #endif /* ARCH_NEEDS_WEAK_PER_CPU */ @@ -191,7 +207,7 @@ static inline void alloc_tag_sub(union codetag_ref *ref, size_t bytes) #else /* CONFIG_MEM_ALLOC_PROFILING */ -#define DEFINE_ALLOC_TAG(_alloc_tag) +#define DEFINE_ALLOC_TAG(_alloc_tag, _meta_init) static inline bool mem_alloc_profiling_enabled(void) { return false; } static inline void alloc_tag_add(union codetag_ref *ref, struct alloc_tag *tag, size_t bytes) {} @@ -210,8 +226,14 @@ static inline void alloc_tag_sub(union codetag_ref *ref, size_t bytes) {} #define alloc_hooks(_do_alloc) \ ({ \ - DEFINE_ALLOC_TAG(_alloc_tag); \ + DEFINE_ALLOC_TAG(_alloc_tag, { }); \ alloc_hooks_tag(&_alloc_tag, _do_alloc); \ }) +#define alloc_sized_hooks(_do_alloc, _size, ...) \ +({ \ + DEFINE_ALLOC_TAG(_alloc_tag, ALLOC_META_INIT(_size)); \ + alloc_hooks_tag(&_alloc_tag, _do_alloc(_size, __VA_ARGS__)); \ +}) + #endif /* _LINUX_ALLOC_TAG_H */ diff --git a/include/linux/slab.h b/include/linux/slab.h index 86cb61a0102c..314d24c79e05 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -541,7 +541,7 @@ static_assert(PAGE_SHIFT <= 20); */ void *kmem_cache_alloc_noprof(struct kmem_cache *cachep, gfp_t flags) __assume_slab_alignment __malloc; -#define kmem_cache_alloc(...) alloc_hooks(kmem_cache_alloc_noprof(__VA_ARGS__)) +#define kmem_cache_alloc(...) alloc_hooks(kmem_cache_alloc_noprof(__VA_ARGS__)) void *kmem_cache_alloc_lru_noprof(struct kmem_cache *s, struct list_lru *lru, gfp_t gfpflags) __assume_slab_alignment __malloc; @@ -685,7 +685,7 @@ static __always_inline __alloc_size(1) void *kmalloc_noprof(size_t size, gfp_t f } return __kmalloc_noprof(size, flags); } -#define kmalloc(...) alloc_hooks(kmalloc_noprof(__VA_ARGS__)) +#define kmalloc(size, ...) alloc_sized_hooks(kmalloc_noprof, size, __VA_ARGS__) #define kmem_buckets_alloc(_b, _size, _flags) \ alloc_hooks(__kmalloc_node_noprof(PASS_BUCKET_PARAMS(_size, _b), _flags, NUMA_NO_NODE)) @@ -708,7 +708,7 @@ static __always_inline __alloc_size(1) void *kmalloc_node_noprof(size_t size, gf } return __kmalloc_node_noprof(PASS_BUCKET_PARAMS(size, NULL), flags, node); } -#define kmalloc_node(...) alloc_hooks(kmalloc_node_noprof(__VA_ARGS__)) +#define kmalloc_node(size, ...) alloc_sized_hooks(kmalloc_node_noprof, size, __VA_ARGS__) /** * kmalloc_array - allocate memory for an array. @@ -726,7 +726,7 @@ static inline __alloc_size(1, 2) void *kmalloc_array_noprof(size_t n, size_t siz return kmalloc_noprof(bytes, flags); return kmalloc_noprof(bytes, flags); } -#define kmalloc_array(...) alloc_hooks(kmalloc_array_noprof(__VA_ARGS__)) +#define kmalloc_array(...) alloc_hooks(kmalloc_array_noprof(__VA_ARGS__)) /** * krealloc_array - reallocate memory for an array. @@ -761,8 +761,8 @@ void *__kmalloc_node_track_caller_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flag unsigned long caller) __alloc_size(1); #define kmalloc_node_track_caller_noprof(size, flags, node, caller) \ __kmalloc_node_track_caller_noprof(PASS_BUCKET_PARAMS(size, NULL), flags, node, caller) -#define kmalloc_node_track_caller(...) \ - alloc_hooks(kmalloc_node_track_caller_noprof(__VA_ARGS__, _RET_IP_)) +#define kmalloc_node_track_caller(size, ...) \ + alloc_sized_hooks(kmalloc_node_track_caller_noprof, size, __VA_ARGS__, _RET_IP_) /* * kmalloc_track_caller is a special version of kmalloc that records the @@ -807,13 +807,13 @@ static inline __alloc_size(1) void *kzalloc_noprof(size_t size, gfp_t flags) { return kmalloc_noprof(size, flags | __GFP_ZERO); } -#define kzalloc(...) alloc_hooks(kzalloc_noprof(__VA_ARGS__)) +#define kzalloc(size, ...) alloc_sized_hooks(kzalloc_noprof, size, __VA_ARGS__) #define kzalloc_node(_size, _flags, _node) kmalloc_node(_size, (_flags)|__GFP_ZERO, _node) void *__kvmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node) __alloc_size(1); #define kvmalloc_node_noprof(size, flags, node) \ __kvmalloc_node_noprof(PASS_BUCKET_PARAMS(size, NULL), flags, node) -#define kvmalloc_node(...) alloc_hooks(kvmalloc_node_noprof(__VA_ARGS__)) +#define kvmalloc_node(size, ...) alloc_sized_hooks(kvmalloc_node_noprof, size, __VA_ARGS__) #define kvmalloc(_size, _flags) kvmalloc_node(_size, _flags, NUMA_NO_NODE) #define kvmalloc_noprof(_size, _flags) kvmalloc_node_noprof(_size, _flags, NUMA_NO_NODE) diff --git a/lib/alloc_tag.c b/lib/alloc_tag.c index 81e5f9a70f22..6d2cb72bf269 100644 --- a/lib/alloc_tag.c +++ b/lib/alloc_tag.c @@ -78,6 +78,14 @@ static void alloc_tag_to_text(struct seq_buf *out, struct codetag *ct) seq_buf_printf(out, "%12lli %8llu ", bytes, counter.calls); codetag_to_text(out, ct); +#ifdef CONFIG_SLAB_PER_SITE + seq_buf_putc(out, ' '); + seq_buf_printf(out, "size:%s(%zu) slab:%s", + tag->meta.sized == 0 ? "non-slab" : + tag->meta.sized == SIZE_MAX ? "dynamic" : "fixed", + tag->meta.sized == SIZE_MAX ? 0 : tag->meta.sized, + tag->meta.cache ? "ready" : "unused"); +#endif seq_buf_putc(out, ' '); seq_buf_putc(out, '\n'); } diff --git a/mm/Kconfig b/mm/Kconfig index b72e7d040f78..855c63c3270d 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -296,6 +296,14 @@ config SLAB_BUCKETS If unsure, say Y. +config SLAB_PER_SITE + bool "Separate slab allocations by call size" + depends on !SLUB_TINY + default SLAB_FREELIST_HARDENED + select SLAB_BUCKETS + help + Track sizes of kmalloc() call sites. + config SLUB_STATS default n bool "Enable performance statistics" From patchwork Fri Aug 9 07:33:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 13758483 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6D5421741FB; Fri, 9 Aug 2024 07:33:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723188790; cv=none; b=eUho/FCwuZTUq3dOQz/Z9uEyDkLjzXnsN8Xi56QTRM9gZ4r7PU3oE5eAqX62ofu89e+H86Jlquk/1tzzLWjJzd8UBNnNhq9xIXQTIK7hE9tdMhJiTn4LDB2SaQoCB04CCaWu1jnKl90iOHAUd6sUhXROpzBQlYJgIoq3RycGOYk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723188790; c=relaxed/simple; bh=WwVEHZxXP/denNiV+dxfYy7hsfC8akgc4PDoJIzTgCM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=cZSjDUol97pCjlsXAncQykW4NErbXCZSpvS9G73S3o0SfHkUE5yFguRxJrJv5187OdX49fOjRHpWyL6PWs37MDMbQqW/cXZx71ahkuc3PqY6NK9z5bvKhLGfRn5vEtMowme6f4oJI39eleOPO3sKX5EDNDUKcW+H7wlKqQbxHCQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=KAgXWs1X; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="KAgXWs1X" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 16C99C4AF11; Fri, 9 Aug 2024 07:33:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723188790; bh=WwVEHZxXP/denNiV+dxfYy7hsfC8akgc4PDoJIzTgCM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=KAgXWs1XKo6iAYzTDWoioXqzTgVrNa4MoYZ4pl3diCJ3d8IMaeMQTl3LUdU4MmSNJ moeSkqQ3YonWd/mpWyVpQNXdVhvpFglG6LQ3r1ZlZxG4tJ1UPkeCEKI6TxVTxfjDTU lGRHfrPSDVCBwY5kVX34XejsM3+3wl0+XynFbgCjBw6fkywEUA4Q8Jz9oJcy6oIGbz oEVwUOwqBv22WLCGoFb22ZbuO2H75EAQP60xliAgxoPNmxUrZFdaPLm1a1owS/jL7b XcuYfJ0FqyUH38NLCXBsgiQ6pyQV7eCEWMSnDxxnOD2WX/Nb2OCPf1KWNK+sdyB09Y 6HKHxYMV8Vq9w== From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , Suren Baghdasaryan , Kent Overstreet , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, "GONG, Ruiqi" , Jann Horn , Matteo Rizzo , jvoisin , Xiu Jianfeng , linux-kernel@vger.kernel.org, linux-hardening@vger.kernel.org Subject: [PATCH 5/5] slab: Allocate and use per-call-site caches Date: Fri, 9 Aug 2024 00:33:06 -0700 Message-Id: <20240809073309.2134488-5-kees@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240809072532.work.266-kees@kernel.org> References: <20240809072532.work.266-kees@kernel.org> Precedence: bulk X-Mailing-List: linux-hardening@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=10722; i=kees@kernel.org; h=from:subject; bh=WwVEHZxXP/denNiV+dxfYy7hsfC8akgc4PDoJIzTgCM=; b=owGbwMvMwCVmps19z/KJym7G02pJDGlbjxn1Tdz6Xbdh/s6WL17vDgv/KesuvZIWvl70Z+FGn /x65TbrjlIWBjEuBlkxRZYgO/c4F4+37eHucxVh5rAygQxh4OIUgInc/8HIcLbu+bae0p1OfcrS 3ofrO24W5Xt+yPlS1MpqFhTQf9RCiuF/gKqyfHxc5OOXDx6wX+2LmJTo7tm64m3mIyZr87OiVWY 8AA== X-Developer-Key: i=kees@kernel.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 Use separate per-call-site kmem_cache or kmem_buckets. These are allocated on demand to avoid wasting memory for unused caches. A few caches need to be allocated very early to support allocating the caches themselves: kstrdup(), kvasprintf(), and pcpu_mem_zalloc(). Any GFP_ATOMIC allocations are currently left to be allocated from KMALLOC_NORMAL. With a distro config, /proc/slabinfo grows from ~400 entries to ~2200. Since this feature (CONFIG_SLAB_PER_SITE) is redundant to CONFIG_RANDOM_KMALLOC_CACHES, mark it a incompatible. Add Kconfig help text that compares the features. Improvements needed: - Retain call site gfp flags in alloc_tag meta field to: - pre-allocate all GFP_ATOMIC caches (since their caches cannot be allocated on demand unless we want them to be GFP_ATOMIC themselves...) - Separate MEMCG allocations as well - Allocate individual caches within kmem_buckets on demand to further reduce memory usage overhead. Signed-off-by: Kees Cook --- Cc: Suren Baghdasaryan Cc: Kent Overstreet Cc: Vlastimil Babka Cc: Christoph Lameter Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Cc: Andrew Morton Cc: Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: linux-mm@kvack.org --- include/linux/alloc_tag.h | 8 +++ lib/alloc_tag.c | 121 +++++++++++++++++++++++++++++++++++--- mm/Kconfig | 19 +++++- mm/slab_common.c | 1 + mm/slub.c | 31 +++++++++- 5 files changed, 170 insertions(+), 10 deletions(-) diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h index f5d8c5849b82..c95628f9b049 100644 --- a/include/linux/alloc_tag.h +++ b/include/linux/alloc_tag.h @@ -24,6 +24,7 @@ struct alloc_tag_counters { struct alloc_meta { /* 0 means non-slab, SIZE_MAX means dynamic, and everything else is fixed-size. */ size_t sized; + void *cache; }; #define ALLOC_META_INIT(_size) { \ .sized = (__builtin_constant_p(_size) ? (_size) : SIZE_MAX), \ @@ -216,6 +217,13 @@ static inline void alloc_tag_sub(union codetag_ref *ref, size_t bytes) {} #endif /* CONFIG_MEM_ALLOC_PROFILING */ +#ifdef CONFIG_SLAB_PER_SITE +void alloc_tag_early_walk(void); +void alloc_tag_site_init(struct codetag *ct, bool ondemand); +#else +static inline void alloc_tag_early_walk(void) {} +#endif + #define alloc_hooks_tag(_tag, _do_alloc) \ ({ \ struct alloc_tag * __maybe_unused _old = alloc_tag_save(_tag); \ diff --git a/lib/alloc_tag.c b/lib/alloc_tag.c index 6d2cb72bf269..e8a66a7c4a6b 100644 --- a/lib/alloc_tag.c +++ b/lib/alloc_tag.c @@ -157,6 +157,89 @@ static void __init procfs_init(void) proc_create_seq("allocinfo", 0400, NULL, &allocinfo_seq_op); } +#ifdef CONFIG_SLAB_PER_SITE +static bool ondemand_ready; + +void alloc_tag_site_init(struct codetag *ct, bool ondemand) +{ + struct alloc_tag *tag = ct_to_alloc_tag(ct); + char *name; + void *p, *old; + + /* Only handle kmalloc allocations. */ + if (!tag->meta.sized) + return; + + /* Must be ready for on-demand allocations. */ + if (ondemand && !ondemand_ready) + return; + + old = READ_ONCE(tag->meta.cache); + /* Already allocated? */ + if (old) + return; + + if (tag->meta.sized < SIZE_MAX) { + /* Fixed-size allocations. */ + name = kasprintf(GFP_KERNEL, "f:%zu:%s:%d", tag->meta.sized, ct->function, ct->lineno); + if (WARN_ON_ONCE(!name)) + return; + /* + * As with KMALLOC_NORMAL, the entire allocation needs to be + * open to usercopy access. :( + */ + p = kmem_cache_create_usercopy(name, tag->meta.sized, 0, + SLAB_NO_MERGE, 0, tag->meta.sized, + NULL); + } else { + /* Dynamically-size allocations. */ + name = kasprintf(GFP_KERNEL, "d:%s:%d", ct->function, ct->lineno); + if (WARN_ON_ONCE(!name)) + return; + p = kmem_buckets_create(name, SLAB_NO_MERGE, 0, UINT_MAX, NULL); + } + if (p) { + if (unlikely(!try_cmpxchg(&tag->meta.cache, &old, p))) { + /* We lost the allocation race; clean up. */ + if (tag->meta.sized < SIZE_MAX) + kmem_cache_destroy(p); + else + kmem_buckets_destroy(p); + } + } + kfree(name); +} + +static void alloc_tag_site_init_early(struct codetag *ct) +{ + /* Explicitly initialize the caches needed to initialize caches. */ + if (strcmp(ct->function, "kstrdup") == 0 || + strcmp(ct->function, "kvasprintf") == 0 || + strcmp(ct->function, "pcpu_mem_zalloc") == 0) + alloc_tag_site_init(ct, false); + + /* TODO: pre-allocate GFP_ATOMIC caches here. */ +} +#endif + +static void alloc_tag_module_load(struct codetag_type *cttype, + struct codetag_module *cmod) +{ +#ifdef CONFIG_SLAB_PER_SITE + struct codetag_iterator iter; + struct codetag *ct; + + iter = codetag_get_ct_iter(cttype); + for (ct = codetag_next_ct(&iter); ct; ct = codetag_next_ct(&iter)) { + if (iter.cmod != cmod) + continue; + + /* TODO: pre-allocate GFP_ATOMIC caches here. */ + //alloc_tag_site_init(ct, false); + } +#endif +} + static bool alloc_tag_module_unload(struct codetag_type *cttype, struct codetag_module *cmod) { @@ -175,8 +258,21 @@ static bool alloc_tag_module_unload(struct codetag_type *cttype, if (WARN(counter.bytes, "%s:%u module %s func:%s has %llu allocated at module unload", - ct->filename, ct->lineno, ct->modname, ct->function, counter.bytes)) + ct->filename, ct->lineno, ct->modname, ct->function, counter.bytes)) { module_unused = false; + } +#ifdef CONFIG_SLAB_PER_SITE + else if (tag->meta.sized) { + /* Remove the allocated caches, if possible. */ + void *p = READ_ONCE(tag->meta.cache); + + WRITE_ONCE(tag->meta.cache, NULL); + if (tag->meta.sized < SIZE_MAX) + kmem_cache_destroy(p); + else + kmem_buckets_destroy(p); + } +#endif } return module_unused; @@ -260,15 +356,16 @@ static void __init sysctl_init(void) static inline void sysctl_init(void) {} #endif /* CONFIG_SYSCTL */ +static const struct codetag_type_desc alloc_tag_desc = { + .section = "alloc_tags", + .tag_size = sizeof(struct alloc_tag), + .module_load = alloc_tag_module_load, + .module_unload = alloc_tag_module_unload, +}; + static int __init alloc_tag_init(void) { - const struct codetag_type_desc desc = { - .section = "alloc_tags", - .tag_size = sizeof(struct alloc_tag), - .module_unload = alloc_tag_module_unload, - }; - - alloc_tag_cttype = codetag_register_type(&desc); + alloc_tag_cttype = codetag_register_type(&alloc_tag_desc); if (IS_ERR(alloc_tag_cttype)) return PTR_ERR(alloc_tag_cttype); @@ -278,3 +375,11 @@ static int __init alloc_tag_init(void) return 0; } module_init(alloc_tag_init); + +#ifdef CONFIG_SLAB_PER_SITE +void alloc_tag_early_walk(void) +{ + codetag_early_walk(&alloc_tag_desc, alloc_tag_site_init_early); + ondemand_ready = true; +} +#endif diff --git a/mm/Kconfig b/mm/Kconfig index 855c63c3270d..4f01cb6dd32e 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -302,7 +302,20 @@ config SLAB_PER_SITE default SLAB_FREELIST_HARDENED select SLAB_BUCKETS help - Track sizes of kmalloc() call sites. + As a defense against shared-cache "type confusion" use-after-free + attacks, every kmalloc()-family call allocates from a separate + kmem_cache (or when dynamically sized, kmem_buckets). Attackers + will no longer be able to groom malicious objects via similarly + sized allocations that share the same cache as the target object. + + This increases the "at rest" kmalloc slab memory usage by + roughly 5x (around 7MiB), and adds the potential for greater + long-term memory fragmentation. However, some workloads + actually see performance improvements when single allocation + sites are hot. + + For a similar defense, see CONFIG_RANDOM_KMALLOC_CACHES, which + has less memory usage overhead, but is probabilistic. config SLUB_STATS default n @@ -331,6 +344,7 @@ config SLUB_CPU_PARTIAL config RANDOM_KMALLOC_CACHES default n depends on !SLUB_TINY + depends on !SLAB_PER_SITE bool "Randomize slab caches for normal kmalloc" help A hardening feature that creates multiple copies of slab caches for @@ -345,6 +359,9 @@ config RANDOM_KMALLOC_CACHES limited degree of memory and CPU overhead that relates to hardware and system workload. + For a similar defense, see CONFIG_SLAB_PER_SITE, which is + deterministic, but has greater memory usage overhead. + endmenu # Slab allocator options config SHUFFLE_PAGE_ALLOCATOR diff --git a/mm/slab_common.c b/mm/slab_common.c index fc698cba0ebe..09506bfa972c 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1040,6 +1040,7 @@ void __init create_kmalloc_caches(void) kmem_buckets_cache = kmem_cache_create("kmalloc_buckets", sizeof(kmem_buckets), 0, SLAB_NO_MERGE, NULL); + alloc_tag_early_walk(); } /** diff --git a/mm/slub.c b/mm/slub.c index 3520acaf9afa..d14102c4b4d7 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4135,6 +4135,35 @@ void *__kmalloc_large_node_noprof(size_t size, gfp_t flags, int node) } EXPORT_SYMBOL(__kmalloc_large_node_noprof); +static __always_inline +struct kmem_cache *choose_slab(size_t size, kmem_buckets *b, gfp_t flags, + unsigned long caller) +{ +#ifdef CONFIG_SLAB_PER_SITE + struct alloc_tag *tag = current->alloc_tag; + + if (!b && tag && tag->meta.sized && + kmalloc_type(flags, caller) == KMALLOC_NORMAL && + (flags & GFP_ATOMIC) != GFP_ATOMIC) { + void *p = READ_ONCE(tag->meta.cache); + + if (!p && slab_state >= UP) { + alloc_tag_site_init(&tag->ct, true); + p = READ_ONCE(tag->meta.cache); + } + + if (tag->meta.sized < SIZE_MAX) { + if (p) + return p; + /* Otherwise continue with default buckets. */ + } else { + b = p; + } + } +#endif + return kmalloc_slab(size, b, flags, caller); +} + static __always_inline void *__do_kmalloc_node(size_t size, kmem_buckets *b, gfp_t flags, int node, unsigned long caller) @@ -4152,7 +4181,7 @@ void *__do_kmalloc_node(size_t size, kmem_buckets *b, gfp_t flags, int node, if (unlikely(!size)) return ZERO_SIZE_PTR; - s = kmalloc_slab(size, b, flags, caller); + s = choose_slab(size, b, flags, caller); ret = slab_alloc_node(s, NULL, flags, node, caller, size); ret = kasan_kmalloc(s, ret, size, flags);