From patchwork Fri Aug 9 07:33:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 13758473 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 349D8C52D73 for ; Fri, 9 Aug 2024 07:33:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5550D6B009F; Fri, 9 Aug 2024 03:33:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4D3F56B00A0; Fri, 9 Aug 2024 03:33:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 336C96B009F; Fri, 9 Aug 2024 03:33:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id EBC806B009A for ; Fri, 9 Aug 2024 03:33:12 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 989701A0A1B for ; Fri, 9 Aug 2024 07:33:12 +0000 (UTC) X-FDA: 82431891024.08.557D86A Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf03.hostedemail.com (Postfix) with ESMTP id D1A7A20031 for ; Fri, 9 Aug 2024 07:33:10 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=U1GOg7df; spf=pass (imf03.hostedemail.com: domain of kees@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=kees@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723188757; a=rsa-sha256; cv=none; b=xg7I4ebMIANLaShDWXWJqRpe03ywpOepny5OSAHahO2BtYQG5oCfEZapGLvgJZZdYdBURM O0hpFu8hxwvJCt4hYgbMTYQlsug8y3QIxQq0kXL9Qq4L97lXvUcVNSdbObtnZ+I3DFojk3 q6YUgI46CBJg9+Wvgnm3KxmgZ3dcVP0= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=U1GOg7df; spf=pass (imf03.hostedemail.com: domain of kees@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=kees@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723188757; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=r2Pzva1Psfsb7VABsc6WR1oxtK6JxECeObZJPE4S0PM=; b=PRECEt/q0YZcAVw+s0592aVpVT2d74ycHT7vvh3hEAJGJZdUCn5lR1npp4zMkem41G9r8r emmv03u/fLrYmMi7BWzKaO7p8ncs1y6FsvLTx3CKeznyBe3iV2zcyW7oIrNKQ+qa52WyEW 9UY99FGGaDsBK+ukWULKkK0jeqrZ1Cw= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id C8ACE61645; Fri, 9 Aug 2024 07:33:09 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8F687C32782; Fri, 9 Aug 2024 07:33:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723188789; bh=fH+32u2hbe+aYvteHR+ZPHJXyfY1tBWhDA7Vvn2Or58=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=U1GOg7dfFb6vCTpdmAesrPycht4b3E2BP61/NHu5KkhADUw+F/qMwpsb+4EE0wwPr L9nvFDqSei8WJyngCv1Q76XygeWaSsjBlLrD6F/LC8/k4VknfL0og/dzm76kmgZgtR 95TPrS8GpHcyJsvMNXhSCnbwb/zN8o2cGRSd7yux2dwYNZdWYITUiYaoPN0QRrAz4J xWzfOO8GT/bjnPzIqghQGJWNHo7TvFmqxfVY9RsL2FSgfXLSt5noN6OfXW0U8mVhWW LIZVWRO8MeNF1Hpm2FoGdrC++nOc8Th3OtGsIxGUZ8FgbOdZ6MVMjlaGpBWyS3Y5Uq lUO1diWANHxyg== From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, Suren Baghdasaryan , Kent Overstreet , "GONG, Ruiqi" , Jann Horn , Matteo Rizzo , jvoisin , Xiu Jianfeng , linux-kernel@vger.kernel.org, linux-hardening@vger.kernel.org Subject: [PATCH 1/5] slab: Introduce kmem_buckets_destroy() Date: Fri, 9 Aug 2024 00:33:02 -0700 Message-Id: <20240809073309.2134488-1-kees@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240809072532.work.266-kees@kernel.org> References: <20240809072532.work.266-kees@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2139; i=kees@kernel.org; h=from:subject; bh=fH+32u2hbe+aYvteHR+ZPHJXyfY1tBWhDA7Vvn2Or58=; b=owGbwMvMwCVmps19z/KJym7G02pJDGlbjxkoHv8l8yD/Mrump/fE6Mr5/aFh0Z63wl+F6swJe 9mx9VNSRykLgxgXg6yYIkuQnXuci8fb9nD3uYowc1iZQIYwcHEKwER0XzEyzOnisNb8799w6KzH TqX4LWYLH5sfsG/JsPy17/be9sMdXxgZDioISWWoefx3MnrGUf7s0jzWyOPSmyWaxZQnrvt/sGs bNwA= X-Developer-Key: i=kees@kernel.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 X-Stat-Signature: 77ipjejseuzssgrmr3gp5s4abozg7qaf X-Rspamd-Queue-Id: D1A7A20031 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1723188790-378772 X-HE-Meta: U2FsdGVkX1/WWXCrmKMPaPGh84IlBnTIGQIrlnf7QmNzEuUDXeEfZjdqx0FiHe3CKA1SSxWfpuUI3R2Z1AcVjDNwGcuYd/fc5DkJ4+cdqgnrcAsqSU9GTd33Ut+P+W7/4I7OGIkerFt2TOZTRssolRr0sDi0EMOqpnXbnJYuWALkDI0Pz8DE2wWlmeGxm/R/d4NREuxd3aVlFJA0gXXXGL351x60oiVa4Wu+ZljEnsKwSqXoMcOMaOu3KRBA/rqvpwFYj7K18S78KQpQI78jnkBV6k0WKJk4uo8pcRR4dZqowcQxO36F9nXzl67cQmovDdlnzjPHGKqwUXhYy8WHDoB5nXFmkqGhunijBq7uvKU1NM0EDVHEoFGUbobGTjXFZYg36veRaypwMZZCVc5IUFNcCgOe7y5EnmjkIWvkCYY/JL6R1F6by0RMpo5DljuqQ6lcjxPKazQ423qjDcwFqhmMwujfNfTEBVf78+8f1jLrQF8pQ78lkInxeai2FlO2sUyK03K4//T+wo7jTRosU1UV8nN07HnifR9s3BdW4KDzW54iSwtylk+C/Qet7ydkkOLyt5ItVYW199A/HYKUJkPAPyYsRkVXzOw5wAUid1vpbHOCe3fQ9/WL/gsI69GOhaPa+CcEE9hTHoGL8y+SCYj0bvP5rHdB7ANYg1OMuG3mZ/SOPRGrmhbUNVD0S5i8S0mUs3sc/ECaSzI/faMfCLS1mIuPSd+d2+GLKCdbEGGJu6fKoRz9flnhWpR/FkJ8rGZWnQewdxzX1YIyvsFzl9IN1HPD+EVxhFBv2qwLK9L3nSw+C3F++YzcVi3lBZMTyInxplkee2SctLJYNvbw2ABQh+Vk220I3YNzbbcqYsyBu5i+G+plPcviILXbQjLEZz7bzIruA4+rv6oCAjzk6QTOhxqqKfq3L3IQTRqGEoANZUM5jRHardbfVYAzX29HZSxyTgnGArcJ+cYwyZD Jts5Th5k EuNgrXKCn7kQV1qot/OdEHFoFFbvAcQlRcULdmZpDXhNS0e9dfQ09TLJnj6nFLGnmGNAhvxG3XvttexYPrfsoql4WRcNrXSJ/7NKrIzXU0imBEYJY9DcXZlc1TzEq8mYov9Yk7P66+U8O3Q1HLXUncOLRMzI2cf74lKlMAhozcnXdJDXnvqhSEpxtuPRCrH0srsmXkRWn8MIbeP+/ajxWKTlcx+okq9YTIPBDquaaa0pLsNANGwFW4+gGViVwj0hoBnop1XYqCr6mlGnkAvOKJ99XK6RlJGM9HUSxfHankkkvVErWJdS1yu8hqXf3CJE9XyJY1kOfBc1qy7HabYAlMM3a0l99fVcGYlcvTPUcHbhP+PvWGli6DLndT/bHx4LCGkL7U/Poe1WXznwP9Ux7/AUog2KlpkreUMDyTh7OAb6Q9+50B+QmZ9gSm261VxFxwOYBDVvN1CGaTjsdudqwYkbBkVAMmjCyzgL2wkYJwhbJ4TU9veV3TijvkwhmXJ/I7Ptw1NXDMdC5GGr/1vj5G6PSbtl4mqpC5rpdzvZpIkCAdsNogSTEJD4lJFB5oKb/0bEnAjRzhwYBfnKIxwA4t4Wx1ZSK7JDkzIWmVe/i/1iZO5qQj3rjLeEd/jsfJGKElQhE X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Modular use of kmem_buckets_create() means that kmem_buckets will need to be removed as well. Introduce kmem_buckets_destroy(), matching kmem_cache_destroy(). Signed-off-by: Kees Cook --- Cc: Vlastimil Babka Cc: Christoph Lameter Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Cc: Andrew Morton Cc: Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: linux-mm@kvack.org --- include/linux/slab.h | 1 + mm/slab_common.c | 17 ++++++++++++++--- 2 files changed, 15 insertions(+), 3 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index eb2bf4629157..86cb61a0102c 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -552,6 +552,7 @@ void kmem_cache_free(struct kmem_cache *s, void *objp); kmem_buckets *kmem_buckets_create(const char *name, slab_flags_t flags, unsigned int useroffset, unsigned int usersize, void (*ctor)(void *)); +void kmem_buckets_destroy(kmem_buckets *b); /* * Bulk allocation and freeing operations. These are accelerated in an diff --git a/mm/slab_common.c b/mm/slab_common.c index 40b582a014b8..fc698cba0ebe 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -392,6 +392,19 @@ kmem_cache_create(const char *name, unsigned int size, unsigned int align, } EXPORT_SYMBOL(kmem_cache_create); +void kmem_buckets_destroy(kmem_buckets *b) +{ + int idx; + + if (!b) + return; + + for (idx = 0; idx < ARRAY_SIZE(kmalloc_caches[KMALLOC_NORMAL]); idx++) + kmem_cache_destroy((*b)[idx]); + kfree(b); +} +EXPORT_SYMBOL(kmem_buckets_destroy); + static struct kmem_cache *kmem_buckets_cache __ro_after_init; /** @@ -476,9 +489,7 @@ kmem_buckets *kmem_buckets_create(const char *name, slab_flags_t flags, return b; fail: - for (idx = 0; idx < ARRAY_SIZE(kmalloc_caches[KMALLOC_NORMAL]); idx++) - kmem_cache_destroy((*b)[idx]); - kfree(b); + kmem_buckets_destroy(b); return NULL; } From patchwork Fri Aug 9 07:33:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 13758476 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3EE23C52D71 for ; Fri, 9 Aug 2024 07:33:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 41ADF6B00A0; Fri, 9 Aug 2024 03:33:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 374686B00A3; Fri, 9 Aug 2024 03:33:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 17B516B00A2; Fri, 9 Aug 2024 03:33:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id D4E816B00A4 for ; Fri, 9 Aug 2024 03:33:13 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 9AE9D1A0A0A for ; Fri, 9 Aug 2024 07:33:13 +0000 (UTC) X-FDA: 82431891066.10.AF15A89 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf29.hostedemail.com (Postfix) with ESMTP id D6AF112002D for ; Fri, 9 Aug 2024 07:33:10 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=QqpCoLXI; spf=pass (imf29.hostedemail.com: domain of kees@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=kees@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723188758; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ZtVqoLL30FNVIrIzNRmlrz4pHlXBNgqPNuW4pj66gso=; b=uO3nXkaT81Z94w/PpI+4cBT5y35zYeBGb+FNsEODJAvOqOhmMSFFxhHMVXzU5b8JcEuaf0 X2YG6ZxibsTN1g7KQynQaQSwgeL44fuwRo8ptJHvPIweiXMdNaqHW6IWSXJw9wJ2Fe+MT0 B3zzcrqjf4vmSHNdBxiYhGN3ws1vChM= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=QqpCoLXI; spf=pass (imf29.hostedemail.com: domain of kees@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=kees@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723188758; a=rsa-sha256; cv=none; b=DevgkxADPbafnuPqL54qHNBu8zGTKAGEWLTPXwQtqYMzknaTWg6+EZq00hvBr6Kv4PgYfb Q7i6Kh6EE4US1rb/Mw6QeOee0MkEDLgvNf+gAfqdTxlvsxrC+6HAE2Wp7LayAInpkKGNkR i9bXZwzCRW2N5CBlAMrAkliLJg2SgHA= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 05FAC6165E; Fri, 9 Aug 2024 07:33:10 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A604FC4AF0D; Fri, 9 Aug 2024 07:33:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723188789; bh=Bm7wZvIpH1u9Fue5VhYGh2wN4XsW+HXCBPDIfO+4Wi0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=QqpCoLXIIBMnBfGGkEX09zukRI2+BmDPINyNOYod5uVhbZDarz3N5Gon+RuXzFMAb oPfcGVx2F89WvXdR38xVPnVsVlOUmNcyHLnIz/i0Chy93oj1L4sP0lHM9ULrx3F+Dy 3wtCujBr/ZAjZcrcrOYEQHYox5wX8h0ySRVdiy4kQYmXwl16b/dgOdo7qfIJdcuOze /My+X+Esi8xyVCPP+QA8hTU/aKTo2FyoEfdKL29fwNZw2fSN5v+VcPTVRG6MpbJbIh z6fXqwUOtqJjrwqRQmB4cR3qiejfPc+eHmuFlse3M/B3SuOYmoyKMkjQGsPZiZMAAm X3fYWCL8pp4Yg== From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , Suren Baghdasaryan , Kent Overstreet , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, "GONG, Ruiqi" , Jann Horn , Matteo Rizzo , jvoisin , Xiu Jianfeng , linux-kernel@vger.kernel.org, linux-hardening@vger.kernel.org Subject: [PATCH 2/5] codetag: Run module_load hooks for builtin codetags Date: Fri, 9 Aug 2024 00:33:03 -0700 Message-Id: <20240809073309.2134488-2-kees@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240809072532.work.266-kees@kernel.org> References: <20240809072532.work.266-kees@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1658; i=kees@kernel.org; h=from:subject; bh=Bm7wZvIpH1u9Fue5VhYGh2wN4XsW+HXCBPDIfO+4Wi0=; b=owGbwMvMwCVmps19z/KJym7G02pJDGlbjxke+n5p3ULn87cf9DTKdwmaspUUamrMS9iRlnGLf TIne3NeRykLgxgXg6yYIkuQnXuci8fb9nD3uYowc1iZQIYwcHEKwEQuXGZkeC42Q9N+bpD98SIX x6h7Lvs+C7ZwcS3zP3qIZfUbhd77AQz/IwUeVCk7XGpQ0dpqWr5Ru4/R+rV7RXv5Wob9sVHfDmx hBwA= X-Developer-Key: i=kees@kernel.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: D6AF112002D X-Stat-Signature: 4ykktbde95ygrhm54ohw83a8rej1skbb X-HE-Tag: 1723188790-964777 X-HE-Meta: U2FsdGVkX1/Djrx7TawDgBQsFvl81DbzayCmB/TWSp/M+dadwyr7Qtxv872Z3wrGQ0FzLrlvIzo3UXFqNmubrM7wIWSLcbM+eGW0ptwl1Enc2mm1c+zIGR2iZY+Rp/GlauLgtSwSmhmAFyeEPZYAIZ3CrHEqovD6UutV9G0Ojy3dxjv3lwTGcLOGt9z0P1xIDyHb7BrSExWjyv5GeDMt5mkdNQRbaL8UcCN7G8geM3kInvVEP6jLGgZC4xvjNu1rb4sm1/CQnVGShCmAL5uE34mLxQqY9Tdo4J0dKUNoppH9sXw9P+ui+zcsJw/lDQT+O0p3uaUyva2VWjCQshGekXYZmNVyAnNg8SnPgL51pMvPWVA5BbdIUstPWG1Kzn0gCw1Nv8ka8BKsN7v7wMITG/vhNvjALzFygTaS4OhilnNYAhYargUPthySdOoiETxZfAQf9nfBxBIqN19uNO+CS6h/Z75/hzYpIckL1ahSLDLQUEaiwznUdCk97uPI0RIe9faXb+iUEhdcf2b1fE5DYydtzWrdn1owxJRv1ZRLe/j/sOjutxzQxl2i9ubQvlu3Xnss8hFZSwqios1wgEP10w6b3y6ZBZRO/5WkumEsx2EYrLYQBV6Z/QtrifhPsRAa4ArAL8y39qCUg8eoCHoXZDyQugVyjhXvDwE3cBdbLussg3WE102A8tpgOk0Bgz2e87uQJICWwqvTGEcjXd/vPS895sG0bW88bmbGzz2VgMpv/eqfYVZDTbMelkQkBgy9SUAVuiKcp+S3gERmi6AMaNkkD86Enu2KU8pDR8ousUKykA0g6bXZB3iEVOcO6J7aZ/Zd5n3iXsL4Q1zvZubhdEPK/gQQ4XecJrU2F2s5sxPRepC19pMyhgcr6CqDEQEGISL3Tc64UCFQE3vUoyOFT3AyWAZAnUq1G8H+p1vnZbX7ScaA3lhY3BGYZzBV0lLSm2ECIkZKnfZpFBDLplz oWQjB9Hx XaSit79Ls2GNkToS1qTskuE7dIa5hHxom5SvM7SyihBWOE0S4MibzQFUrNaOBvgi32YrNcFEPRRMsTSiahQE1qa7iUbnfZry/IhC+bb9fGgWmuK+noymKDmSs12L31JTXG3aS9Y9rw1h4F98oZjTUd0qw69aP7HF2r3ts49HEKaODakDCWkzQiC6GYOSTF7iOZH4Jw7teNx/08DURgkEHupUSH3JKF6cTmVWXq80ddgmNe0+VSOwKfS1bKAyLRoPp4TZCvnLzWWlloht7A6n4Vy1dHNZSl8DdDJskX25v8xliTkcS7dz8HzQaBvFLt6PIjOYL0CJURRVnhoQv/AiG+93gc5m3ZvIDGfIXGG8MQ88ZvLYzskt717ZycmZugRvfz59wdwnYzt39mfmYp2qgRn6SzMFlpUHu3cyR3VIQF2ICJt08J/CCXTnSjAZiITKUrIBaWAWN1wBnMXyWkvfedCf9vL/zY94C12chquTUsJP2tjhrf+UxbK4ffb2I1SAYfBH8lkH1RrtPSGTcBn5GUqhb4kYmDdSztVWygPOY7K65yEjkcRhfZj0zBisDvFtAm9lLNeOlk8vHcpWbBzEtN/kKiawHDswS7ROeT2FjhSlbxOl1tm1UV6s8hlQXVWZP9Abv X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The module_load callback should still run for builtin codetags that define it, even in a non-modular kernel. (i.e. for the cmod->mod == NULL case). Signed-off-by: Kees Cook --- Cc: Suren Baghdasaryan Cc: Kent Overstreet Cc: Vlastimil Babka Cc: Christoph Lameter Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Cc: Andrew Morton Cc: Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: linux-mm@kvack.org --- lib/codetag.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/lib/codetag.c b/lib/codetag.c index 5ace625f2328..ef7634c7ee18 100644 --- a/lib/codetag.c +++ b/lib/codetag.c @@ -125,7 +125,6 @@ static inline size_t range_size(const struct codetag_type *cttype, cttype->desc.tag_size; } -#ifdef CONFIG_MODULES static void *get_symbol(struct module *mod, const char *prefix, const char *name) { DECLARE_SEQ_BUF(sb, KSYM_NAME_LEN); @@ -199,6 +198,7 @@ static int codetag_module_init(struct codetag_type *cttype, struct module *mod) return 0; } +#ifdef CONFIG_MODULES void codetag_load_module(struct module *mod) { struct codetag_type *cttype; @@ -248,9 +248,6 @@ bool codetag_unload_module(struct module *mod) return unload_ok; } - -#else /* CONFIG_MODULES */ -static int codetag_module_init(struct codetag_type *cttype, struct module *mod) { return 0; } #endif /* CONFIG_MODULES */ struct codetag_type * From patchwork Fri Aug 9 07:33:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 13758474 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7362AC52D71 for ; Fri, 9 Aug 2024 07:33:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AF1536B009E; Fri, 9 Aug 2024 03:33:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A51D46B009A; Fri, 9 Aug 2024 03:33:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8A3416B00A2; Fri, 9 Aug 2024 03:33:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 5B6246B009A for ; Fri, 9 Aug 2024 03:33:13 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 13D2E140A0C for ; Fri, 9 Aug 2024 07:33:13 +0000 (UTC) X-FDA: 82431891066.25.B9D27E7 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf28.hostedemail.com (Postfix) with ESMTP id 49131C0006 for ; Fri, 9 Aug 2024 07:33:11 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=G96xFR9z; spf=pass (imf28.hostedemail.com: domain of kees@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=kees@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723188739; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=q6aByz+AjLMXLRqYk6daYpayybsUPOOvuxDJo0eVHWw=; b=j6psHLU+MRncOMDVAFCg1JK1tXz+80+Hvc+VPX6uo3iKwGn1qy50jwBNqQn0i0UAy+O4sj SWojMthCOzCGcOWgoeXc2U1gjJIni7R7yUzrGVMmRVIdEqoea/znbJY4LWNzWLuvgQkJQA 3YrBYSld1ZRoWcoF0IxeMcn7EbZKWKc= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=G96xFR9z; spf=pass (imf28.hostedemail.com: domain of kees@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=kees@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723188739; a=rsa-sha256; cv=none; b=OtTwhvNlgGnHY0ykwfCldEsAXdB70/POvKiqN5g0KLMfsN9ti1EHXG6UiNbuikHGGdbeMW 2gkdGLG6O+Uuv2gsEdhvegOv6HojGJF7IFRVVZFDkrM8Fi13x/ZaByGLcYA5NPkIGMcLzw atVbJxXN6C5EOiT5Q1BtqAOdAoH53K8= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 1518A6158B; Fri, 9 Aug 2024 07:33:10 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id AB099C4AF10; Fri, 9 Aug 2024 07:33:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723188789; bh=Stx3EEnB/9LWtVMs8bPtcAXmif1O8Z/znzWwrIr1yQM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=G96xFR9zVeh09ObUFoqqZFktxg1j3FtzvLiXQZGmQZW/HVQIm7p5HIUbK5Ww99589 CNxj96mi3eBlCxegp72LfDbiC9rOvnrWCQzO0YvRFp6aR3VIj/SuoeDWAMXhIbxV8O cjeBU/r8ZOz+n88QSTZXJ9cFA1/i28S4PPfS/AKJ2uwN4tSkbb3JKof8G+eosTEybG 485xFvfsK3AuAe/sxtfTl47vL4CU3b2ybBPEAFXxACwNrNDgvU5G2oZNtuTk4AYoyV 9pYD8+LMbuhN6C9vBvYfaJZfhUZngVa/DpU5F+dlB9t9jc6WTqOaLW4mZPlzd8Hk+n ziag+GbKspAEQ== From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , Suren Baghdasaryan , Kent Overstreet , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, "GONG, Ruiqi" , Jann Horn , Matteo Rizzo , jvoisin , Xiu Jianfeng , linux-kernel@vger.kernel.org, linux-hardening@vger.kernel.org Subject: [PATCH 3/5] codetag: Introduce codetag_early_walk() Date: Fri, 9 Aug 2024 00:33:04 -0700 Message-Id: <20240809073309.2134488-3-kees@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240809072532.work.266-kees@kernel.org> References: <20240809072532.work.266-kees@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2359; i=kees@kernel.org; h=from:subject; bh=Stx3EEnB/9LWtVMs8bPtcAXmif1O8Z/znzWwrIr1yQM=; b=owGbwMvMwCVmps19z/KJym7G02pJDGlbjxnOseXU/pJl/43T0XOt5r2jXsvfZqg+sjDOii3I3 vMo2r2zo5SFQYyLQVZMkSXIzj3OxeNte7j7XEWYOaxMIEMYuDgFYCKPwhgZPkd03P/cfODJhK1S TXHcc1wl7ylP8ufytdEOmixz1c7Yg+F/pc/9q0Ub7q7Zv+boUV0X+f1xT5Q6ndp1tt0+78NeFv6 LAwA= X-Developer-Key: i=kees@kernel.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 X-Stat-Signature: sm1fni9q3gzch7ub5op7btj34wx1i9cx X-Rspam-User: X-Rspamd-Queue-Id: 49131C0006 X-Rspamd-Server: rspam02 X-HE-Tag: 1723188791-857760 X-HE-Meta: U2FsdGVkX19krTA8vLXdpio9wbw1ROw5r2f7LhAf2PJcAgmhrlLScR519UqxRPXDlTJQ8sNIpbPaCSltI+yCuyDhRdu8E05NX/Zr7SfRFLMogVICBIqv+OaMDvxhtVRPqQ8Cc2iv5YcPCMQengfOnUaYPYfTLFL+i7h9zh51fzoZUDIfFFeU3McgX+zu+xD42DryvJzpui/GlCskAW4WVa3/g+afh1BxudmjjzGVCZ19rQBvmbqN6JTXgpxBLfZgZiyc+gsdCE/2FwDHDna3vclx71FVSn80Fq5pRXMfu4b0J0ml5T3BmFo1OpuCGX0koihzf7o8qsm5R2rwcfnNzJyyiBcEhmdwuX3Y63tiCVkpzDDKCTDd3NCn9Xon8qz4ftz2D5UJFWcDhLF1zsfTmTNZ0/chvwpqSQ1a88QIRGC0BQ9sp17VHiTLh46lANg88PiF6Asq41rVLae33M2GzLJfKKIToV8n5L8iElA1aByx689vOFhGrKG/d9UaYoFq1YBOpoDwdmWg5cJJnNjG+GqfpeXsUHFfmBDjhgl+IjXBsocE1HPBbAtCEIwZM0sgYfCpdH2Nbdhfd0cAEbW2whRjVGj41jVBt7ZXZrRn7rQmHJgC8zPNL5aJsr1nXc7NEztx0v4gPhROowz0Es6ivH/b8/W5ackdz5pe5183XZOHK9Y5j+AU5srX/avGcvmGMOTk7foQ4VAv7wfwkLF733KBe8ZiISeYO1YBALUP/eCM5Nzbztk+Q6NO05lXt/oT/8hshvysnkdt7e6j9FxOskoKj1kV3uGUlHP0/A08bBRgFEDX2VL0kJTcUtmq177gYbNB5BKsfe+VHE3OtUeyWtqAnlr3jDbd+oTbOmgKqrkN9KiZKUr1q9eABaRk4QQxGSEeMzWA+OzyAnab7+B0AVaRt8BV0n8lXztGp9ip2iO/W2SagLT28tCMWV3Ty+wkEiT/WkmRY3SwSbyYZuO K7Z7T8iq GYsMkvCsnTNCkYO2HedRpXmb5PQWWUPOGSvW3epGyW42mMsJdqYRI8aydTOSTgXakrZz1svAp+Y8tCeiBT1mwk0qmB0H8Plx3oFYWbDuBEoLbGbriXL2uLf2v6NF37gYfjMJeGvMTcasuyZfXoFErtkOCgSwHq1S/D8vjhqNyb1p5d65YdTsasiF/aAVrF9P1Qt5C8fZPBl7Dk/IpXOWMCVbYoWosLcv82fHGGltFQPammc0ifDccWMPaejYjeAnX0zq8wOX/scKisIiskaWl3t9w5ZXZ+TXTUE5y8oSfTrNiyvTqz8hAQSilFVi2H5lE218ZpSzJdGP2NMSgWswr4GbWw/xCuO20ulxW+kFSUii/lCYJnZ+EsCd0MwLb4KVEkOLTJ7vZ9iYnk1Zly4T5NWmrZDT2RpFbqqNB/rmszS7f720OpBM6J/rNYrY82R0d8O1UIZe5MO30cIrVW90/sdNYpr8cQrLujRfwHeTt3jRJ1bPkSprhMQUMNRURBkUOe00FSK0z5a536qVzdQ3p0DWlJ/9bhVGz8Mv/O0svVk4ldVTa2JaSBf+mZjdSyG3nPcEQ2o2VYK8XJ6nsmFTBUrq20WMjLFEpWmlb3PP6byQ5n85ujq1eMLXkBwb7sKoypwBH X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In order to process builtin alloc_tags much earlier during boot (before register_codetag() is processed), provide codetag_early_walk() that perform a lockless walk with a specified callback function. This will be used to allocate required caches that cannot be allocated on demand. Signed-off-by: Kees Cook --- Cc: Suren Baghdasaryan Cc: Kent Overstreet Cc: Vlastimil Babka Cc: Christoph Lameter Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Cc: Andrew Morton Cc: Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: linux-mm@kvack.org --- include/linux/codetag.h | 2 ++ lib/codetag.c | 16 ++++++++++++++++ 2 files changed, 18 insertions(+) diff --git a/include/linux/codetag.h b/include/linux/codetag.h index c2a579ccd455..9eb1fcd90570 100644 --- a/include/linux/codetag.h +++ b/include/linux/codetag.h @@ -64,6 +64,8 @@ void codetag_lock_module_list(struct codetag_type *cttype, bool lock); bool codetag_trylock_module_list(struct codetag_type *cttype); struct codetag_iterator codetag_get_ct_iter(struct codetag_type *cttype); struct codetag *codetag_next_ct(struct codetag_iterator *iter); +void codetag_early_walk(const struct codetag_type_desc *desc, + void (*callback)(struct codetag *ct)); void codetag_to_text(struct seq_buf *out, struct codetag *ct); diff --git a/lib/codetag.c b/lib/codetag.c index ef7634c7ee18..9d563c8c088a 100644 --- a/lib/codetag.c +++ b/lib/codetag.c @@ -154,6 +154,22 @@ static struct codetag_range get_section_range(struct module *mod, }; } +void codetag_early_walk(const struct codetag_type_desc *desc, + void (*callback)(struct codetag *ct)) +{ + struct codetag_range range; + struct codetag *ct; + + range = get_section_range(NULL, desc->section); + if (!range.start || !range.stop || + range.start == range.stop || + range.start > range.stop) + return; + + for (ct = range.start; ct < range.stop; ct = ((void *)ct + desc->tag_size)) + callback(ct); +} + static int codetag_module_init(struct codetag_type *cttype, struct module *mod) { struct codetag_range range; From patchwork Fri Aug 9 07:33:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 13758475 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 27979C52D73 for ; Fri, 9 Aug 2024 07:33:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EA5E56B009A; Fri, 9 Aug 2024 03:33:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D0DCE6B00A3; Fri, 9 Aug 2024 03:33:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 96AA86B00A0; Fri, 9 Aug 2024 03:33:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 4921C6B009E for ; Fri, 9 Aug 2024 03:33:13 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id D7EFB140A32 for ; Fri, 9 Aug 2024 07:33:12 +0000 (UTC) X-FDA: 82431891024.09.8A07AAF Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf09.hostedemail.com (Postfix) with ESMTP id 1692614000A for ; Fri, 9 Aug 2024 07:33:10 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b="o4q8IAk/"; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf09.hostedemail.com: domain of kees@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=kees@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723188737; a=rsa-sha256; cv=none; b=ydKh7AaPO9Mos5paB+KUmrm7cYnYgZFDPi/1I0X7LBUVDQGXZ/XR5Of9c6dSYVlalm8v+5 FQ/YnMDhDFmZIk8Dy5zeJ+GT4/58OY/ig78sVILj7cN1lEQ2VPF8hpcKip4tIh8O7tqd+c bv1QJkEzqvGYb25fXE7sMSpUODOzdtg= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b="o4q8IAk/"; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf09.hostedemail.com: domain of kees@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=kees@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723188737; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Nh7dt7bQuU6LNtnXjDqc3kjt5MiWMUuLcZZcyTOBVhU=; b=onUp/XOpByfSkX3hityv23SLfikJ21FqLUWaV8MUD5QjbPZnXGpJHtI3BM7UKdqGLwgUgr aCAa19eK78ZGRVM0g0HE+9+0DacvelGDn6KlhSSExlfbym1kFtf3ZnPmu27/Cr1oKSYNZs oemA67AMc1Iqx2DMCtiygRj2UpYcMqE= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 062536165F; Fri, 9 Aug 2024 07:33:10 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A8720C4AF0F; Fri, 9 Aug 2024 07:33:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723188789; bh=Iabo073v2g0BX5idKAlpNK0sS3Na+mbsF9YuoIGYIVE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=o4q8IAk/qlgqcTCea3BBJdIf8hDOC1XjjLxhcI8dkm1ljzbm+iszrHUacqp7lGQQq jGNBVMkdhPRB0YBHgDKEbwNi4qZIaA7IOQMRouJlraAAcY8ZANl/rh82rIf2q7a7Sk RGDqTlGc0UkZl79HBkzQlQPhNfE3UMMSY31PiSUNZY49KE4SM+jd9QH+P+ipVYPvlr PHnohauWkmIwt2xg5wEQ8yc8x/rD4IETjsdn1fVLzbTajnMZKrEL/FtYFbIOrNxBUu jCkle8QjxtskMzf4qXmks1dSVdfJo3x06Hf3Uia/frmvQwd2b0c1VatOnwxcL02pJN EqUeRe1J4h/QQ== From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , Suren Baghdasaryan , Kent Overstreet , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, "GONG, Ruiqi" , Jann Horn , Matteo Rizzo , jvoisin , Xiu Jianfeng , linux-kernel@vger.kernel.org, linux-hardening@vger.kernel.org Subject: [PATCH 4/5] alloc_tag: Track fixed vs dynamic sized kmalloc calls Date: Fri, 9 Aug 2024 00:33:05 -0700 Message-Id: <20240809073309.2134488-4-kees@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240809072532.work.266-kees@kernel.org> References: <20240809072532.work.266-kees@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=8513; i=kees@kernel.org; h=from:subject; bh=Iabo073v2g0BX5idKAlpNK0sS3Na+mbsF9YuoIGYIVE=; b=owGbwMvMwCVmps19z/KJym7G02pJDGlbjxla/OkOZpqjmTXXV1pxl+T8wjPTGhjbs13unfB2Y DSfGRnZUcrCIMbFICumyBJk5x7n4vG2Pdx9riLMHFYmkCEMXJwCMBFZZ0aG2zfLDrTphUWynG79 HvZ6UuCs3+Fnq13vX2MPMGJ4Pkd5ESPDrh3z73lW79405Y3IHS2OfPYiTYNZKYUfe4ubfrw2nb2 WGQA= X-Developer-Key: i=kees@kernel.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 1692614000A X-Stat-Signature: bwjd7jjegkuof7gmnqx1j7cj3d46t8iw X-Rspam-User: X-HE-Tag: 1723188790-242937 X-HE-Meta: U2FsdGVkX18I3EmSkRSQbLngBk13iH1nJA3RF3vSGPTPJcIwWnvGo1fg1ENSqG29bC2mYQMmuLmrnYxhR41WhTBisz+vCN77pq5fCAivWsjutPFryXeAe/eX0r/C/UJBVnLseBxVywLm376ixEmjWJC/pWA3sJ1rwwVV3h087lVXHOgR+gCVV2fAg/LC1EBQdCOchSuOqeyAbDh4bGHyZrsziTYlxXTw+uZe5tq89pD98928SdybyVTMguIcnv9W9gMgvYVR3f0sTYJJ9o8FW96HSm09Bntyly1D1foBrhBPiKYZ7oWGTFjYxDixL4w7dkeZfDlkzMeHprK/IkZcP5pqet2vsZdJF03HS/KjvrHclzyN3KIEeOaB87YCUSyD8nUvxK1EDlADV2v5XD1wAvLh29zotruVvBgomSCz7Qa2oMKYznRk6HdsmORB2s2agJEEQdA7LjyJDrTaGaU7kID0H/jrBSd2fV+YXFUAw9nus4Mxq33w3XQntCOfy0zXjf4S9vYFs9UYrUT5+EUIYHGzwFLOyrY+QoTQBUZhh0opCp9hLUhmKRq9bfo7vdz3Acml6hVXnEwIipPkP/YYndNljm0+xNKKDHEoC+s3HcXc3fDa1K3WeAdujMQUUYM8U+U2Zd4+PNtw1lvccJaEkGZxAV/JqsbrVicETcoyayNL5qVlzqOMUNR2rsoUX4AVDK0vAP7NLmf3SNlcdP36ymH3v7Dw+hoWcqmm7LiDCtZwj0oibZThqaE3v+WdRl/Zl6twVbdjI0hXLBumE5/XY/xXXu2ZjOdpCkXlmKc1K+7epYPlg1omz72lEQDOeRaU7SLQHjh7wps+x+F5bDrlnv9w5UB5VODKdudXX6WR7T6eghCDDDvXOfSWN+r+hTdiwMPaSmt595RlDXnGQuDJFzflnl9Hq3amw6xwLz1J7YCWaZ0ljBFVAKSreUSLRgNf0Kkn5qFc/ieYu5uxBEN T2vcjOqO mVAnS+WqBIt0JsvoyYL2Zna9bXqjr/+6q17JErXNwYJko1cJ4/j7h4++Aqi3uW6E8WerPClXCkyQzQWWXIwhTlu2+G66XoGSfO2Dk92d3gOeVsofOwFY2PdB8cSftjXt0KIuW90feQiO0u9JhxplvjHKvFwDPDysN7HCph4BUBK4zyqxOEJr41voDb5hoZOQphggVBrP6+XqKa/dXKqyCXZJPfHcROoGuBQnkkjZUm1/gZSyrIKjDHvRg3jKj6qJoZwpwCj+Jh32eBCdelEnBtwFFDo+4cDbe/ZDEqtG+yTkYq1/smJ5f9Dm6YwFAgQz/QGYHD1R0ldEWiTfX/7wfDkKnFaS7SiClvOTMqnp2w4S1QXxD3fR11SQ6X5cVstGeP2EEtFfkq+DWus5hNFXx4NoYKdRnTBqbcHwCjGLZe9ODfBsbVw17ZKObJyvILIKM/sRf+yoh0RsWslH/PF5BKw5GjkbKw+91MfJKJ9ZDIl+vsqLt/e/NLue4h6PaI4Iv15RxRVXCrDZcjsfEBhIoOZ7AZ9dClUUVU5h8Y221n8axWTGzVbF8/l9Q45WtTAJrEzRxX3i/8Z0txXRO5h+QYL4+94KuOw3+/DXXQ9Hh0nShsPe1wyjWuVM1Ola27BvPU5MGv7aHez+bU9I= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: For slab allocations, record whether the call site is using a fixed size (i.e. compile time constant) or a dynamic size. Report the results in /proc/allocinfo. Improvements needed: - examine realloc routines for needed coverage Signed-off-by: Kees Cook --- Cc: Suren Baghdasaryan Cc: Kent Overstreet Cc: Vlastimil Babka Cc: Christoph Lameter Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Cc: Andrew Morton Cc: Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: linux-mm@kvack.org --- include/linux/alloc_tag.h | 30 ++++++++++++++++++++++++++---- include/linux/slab.h | 16 ++++++++-------- lib/alloc_tag.c | 8 ++++++++ mm/Kconfig | 8 ++++++++ 4 files changed, 50 insertions(+), 12 deletions(-) diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h index 8c61ccd161ba..f5d8c5849b82 100644 --- a/include/linux/alloc_tag.h +++ b/include/linux/alloc_tag.h @@ -20,6 +20,19 @@ struct alloc_tag_counters { u64 calls; }; +#ifdef CONFIG_SLAB_PER_SITE +struct alloc_meta { + /* 0 means non-slab, SIZE_MAX means dynamic, and everything else is fixed-size. */ + size_t sized; +}; +#define ALLOC_META_INIT(_size) { \ + .sized = (__builtin_constant_p(_size) ? (_size) : SIZE_MAX), \ + } +#else +struct alloc_meta { }; +#define ALLOC_META_INIT(_size) { } +#endif + /* * An instance of this structure is created in a special ELF section at every * allocation callsite. At runtime, the special section is treated as @@ -27,6 +40,7 @@ struct alloc_tag_counters { */ struct alloc_tag { struct codetag ct; + struct alloc_meta meta; struct alloc_tag_counters __percpu *counters; } __aligned(8); @@ -74,19 +88,21 @@ static inline struct alloc_tag *ct_to_alloc_tag(struct codetag *ct) */ DECLARE_PER_CPU(struct alloc_tag_counters, _shared_alloc_tag); -#define DEFINE_ALLOC_TAG(_alloc_tag) \ +#define DEFINE_ALLOC_TAG(_alloc_tag, _meta_init) \ static struct alloc_tag _alloc_tag __used __aligned(8) \ __section("alloc_tags") = { \ .ct = CODE_TAG_INIT, \ + .meta = _meta_init, \ .counters = &_shared_alloc_tag }; #else /* ARCH_NEEDS_WEAK_PER_CPU */ -#define DEFINE_ALLOC_TAG(_alloc_tag) \ +#define DEFINE_ALLOC_TAG(_alloc_tag, _meta_init) \ static DEFINE_PER_CPU(struct alloc_tag_counters, _alloc_tag_cntr); \ static struct alloc_tag _alloc_tag __used __aligned(8) \ __section("alloc_tags") = { \ .ct = CODE_TAG_INIT, \ + .meta = _meta_init, \ .counters = &_alloc_tag_cntr }; #endif /* ARCH_NEEDS_WEAK_PER_CPU */ @@ -191,7 +207,7 @@ static inline void alloc_tag_sub(union codetag_ref *ref, size_t bytes) #else /* CONFIG_MEM_ALLOC_PROFILING */ -#define DEFINE_ALLOC_TAG(_alloc_tag) +#define DEFINE_ALLOC_TAG(_alloc_tag, _meta_init) static inline bool mem_alloc_profiling_enabled(void) { return false; } static inline void alloc_tag_add(union codetag_ref *ref, struct alloc_tag *tag, size_t bytes) {} @@ -210,8 +226,14 @@ static inline void alloc_tag_sub(union codetag_ref *ref, size_t bytes) {} #define alloc_hooks(_do_alloc) \ ({ \ - DEFINE_ALLOC_TAG(_alloc_tag); \ + DEFINE_ALLOC_TAG(_alloc_tag, { }); \ alloc_hooks_tag(&_alloc_tag, _do_alloc); \ }) +#define alloc_sized_hooks(_do_alloc, _size, ...) \ +({ \ + DEFINE_ALLOC_TAG(_alloc_tag, ALLOC_META_INIT(_size)); \ + alloc_hooks_tag(&_alloc_tag, _do_alloc(_size, __VA_ARGS__)); \ +}) + #endif /* _LINUX_ALLOC_TAG_H */ diff --git a/include/linux/slab.h b/include/linux/slab.h index 86cb61a0102c..314d24c79e05 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -541,7 +541,7 @@ static_assert(PAGE_SHIFT <= 20); */ void *kmem_cache_alloc_noprof(struct kmem_cache *cachep, gfp_t flags) __assume_slab_alignment __malloc; -#define kmem_cache_alloc(...) alloc_hooks(kmem_cache_alloc_noprof(__VA_ARGS__)) +#define kmem_cache_alloc(...) alloc_hooks(kmem_cache_alloc_noprof(__VA_ARGS__)) void *kmem_cache_alloc_lru_noprof(struct kmem_cache *s, struct list_lru *lru, gfp_t gfpflags) __assume_slab_alignment __malloc; @@ -685,7 +685,7 @@ static __always_inline __alloc_size(1) void *kmalloc_noprof(size_t size, gfp_t f } return __kmalloc_noprof(size, flags); } -#define kmalloc(...) alloc_hooks(kmalloc_noprof(__VA_ARGS__)) +#define kmalloc(size, ...) alloc_sized_hooks(kmalloc_noprof, size, __VA_ARGS__) #define kmem_buckets_alloc(_b, _size, _flags) \ alloc_hooks(__kmalloc_node_noprof(PASS_BUCKET_PARAMS(_size, _b), _flags, NUMA_NO_NODE)) @@ -708,7 +708,7 @@ static __always_inline __alloc_size(1) void *kmalloc_node_noprof(size_t size, gf } return __kmalloc_node_noprof(PASS_BUCKET_PARAMS(size, NULL), flags, node); } -#define kmalloc_node(...) alloc_hooks(kmalloc_node_noprof(__VA_ARGS__)) +#define kmalloc_node(size, ...) alloc_sized_hooks(kmalloc_node_noprof, size, __VA_ARGS__) /** * kmalloc_array - allocate memory for an array. @@ -726,7 +726,7 @@ static inline __alloc_size(1, 2) void *kmalloc_array_noprof(size_t n, size_t siz return kmalloc_noprof(bytes, flags); return kmalloc_noprof(bytes, flags); } -#define kmalloc_array(...) alloc_hooks(kmalloc_array_noprof(__VA_ARGS__)) +#define kmalloc_array(...) alloc_hooks(kmalloc_array_noprof(__VA_ARGS__)) /** * krealloc_array - reallocate memory for an array. @@ -761,8 +761,8 @@ void *__kmalloc_node_track_caller_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flag unsigned long caller) __alloc_size(1); #define kmalloc_node_track_caller_noprof(size, flags, node, caller) \ __kmalloc_node_track_caller_noprof(PASS_BUCKET_PARAMS(size, NULL), flags, node, caller) -#define kmalloc_node_track_caller(...) \ - alloc_hooks(kmalloc_node_track_caller_noprof(__VA_ARGS__, _RET_IP_)) +#define kmalloc_node_track_caller(size, ...) \ + alloc_sized_hooks(kmalloc_node_track_caller_noprof, size, __VA_ARGS__, _RET_IP_) /* * kmalloc_track_caller is a special version of kmalloc that records the @@ -807,13 +807,13 @@ static inline __alloc_size(1) void *kzalloc_noprof(size_t size, gfp_t flags) { return kmalloc_noprof(size, flags | __GFP_ZERO); } -#define kzalloc(...) alloc_hooks(kzalloc_noprof(__VA_ARGS__)) +#define kzalloc(size, ...) alloc_sized_hooks(kzalloc_noprof, size, __VA_ARGS__) #define kzalloc_node(_size, _flags, _node) kmalloc_node(_size, (_flags)|__GFP_ZERO, _node) void *__kvmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node) __alloc_size(1); #define kvmalloc_node_noprof(size, flags, node) \ __kvmalloc_node_noprof(PASS_BUCKET_PARAMS(size, NULL), flags, node) -#define kvmalloc_node(...) alloc_hooks(kvmalloc_node_noprof(__VA_ARGS__)) +#define kvmalloc_node(size, ...) alloc_sized_hooks(kvmalloc_node_noprof, size, __VA_ARGS__) #define kvmalloc(_size, _flags) kvmalloc_node(_size, _flags, NUMA_NO_NODE) #define kvmalloc_noprof(_size, _flags) kvmalloc_node_noprof(_size, _flags, NUMA_NO_NODE) diff --git a/lib/alloc_tag.c b/lib/alloc_tag.c index 81e5f9a70f22..6d2cb72bf269 100644 --- a/lib/alloc_tag.c +++ b/lib/alloc_tag.c @@ -78,6 +78,14 @@ static void alloc_tag_to_text(struct seq_buf *out, struct codetag *ct) seq_buf_printf(out, "%12lli %8llu ", bytes, counter.calls); codetag_to_text(out, ct); +#ifdef CONFIG_SLAB_PER_SITE + seq_buf_putc(out, ' '); + seq_buf_printf(out, "size:%s(%zu) slab:%s", + tag->meta.sized == 0 ? "non-slab" : + tag->meta.sized == SIZE_MAX ? "dynamic" : "fixed", + tag->meta.sized == SIZE_MAX ? 0 : tag->meta.sized, + tag->meta.cache ? "ready" : "unused"); +#endif seq_buf_putc(out, ' '); seq_buf_putc(out, '\n'); } diff --git a/mm/Kconfig b/mm/Kconfig index b72e7d040f78..855c63c3270d 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -296,6 +296,14 @@ config SLAB_BUCKETS If unsure, say Y. +config SLAB_PER_SITE + bool "Separate slab allocations by call size" + depends on !SLUB_TINY + default SLAB_FREELIST_HARDENED + select SLAB_BUCKETS + help + Track sizes of kmalloc() call sites. + config SLUB_STATS default n bool "Enable performance statistics" From patchwork Fri Aug 9 07:33:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 13758477 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E5DFDC52D71 for ; Fri, 9 Aug 2024 07:33:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 859BC6B00A2; Fri, 9 Aug 2024 03:33:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 793846B00A3; Fri, 9 Aug 2024 03:33:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 570CC6B00A4; Fri, 9 Aug 2024 03:33:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 323296B00A2 for ; Fri, 9 Aug 2024 03:33:15 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id DF3B8160A4F for ; Fri, 9 Aug 2024 07:33:14 +0000 (UTC) X-FDA: 82431891108.25.FD837E6 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf13.hostedemail.com (Postfix) with ESMTP id 2A64A2001F for ; Fri, 9 Aug 2024 07:33:13 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=KAgXWs1X; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf13.hostedemail.com: domain of kees@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=kees@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723188783; a=rsa-sha256; cv=none; b=PkTG1gSqtkXO5/rZr44q6KQJCeMBlv6MQHzkAn/UcNrf2fdcGopFuVlpjKwa+Q0/gB7b2n EGtvjJRCpkPhw1dH8c7ZE0t9nYdOZii6cYggeQ8eMftMgfLvxVfXkt8ZRAoy/OcwANOyhH dDz5CmDL9xCtpoSeA5iL5HUqct5fULg= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=KAgXWs1X; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf13.hostedemail.com: domain of kees@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=kees@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723188783; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=1LHppLDOPKVGGYTOKfAnP782hJ0aLQibHbvq68MVdt8=; b=vNXAoCVq9XAu+OzOEn48jJRmjcSfUTvK01NTDBeDp8jFSPcz9G6VHobXDJXqhyxg9otOki bp671LEk6dx22z2kPJAo958B4Ktcxgce8kcsrIxgtSop3kgNgjCU8YODyj474pxlvT/uIo Keo2RL/j0awxGz5XwN1uvSulXXxvAfs= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 4841861668; Fri, 9 Aug 2024 07:33:10 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 16C99C4AF11; Fri, 9 Aug 2024 07:33:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723188790; bh=WwVEHZxXP/denNiV+dxfYy7hsfC8akgc4PDoJIzTgCM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=KAgXWs1XKo6iAYzTDWoioXqzTgVrNa4MoYZ4pl3diCJ3d8IMaeMQTl3LUdU4MmSNJ moeSkqQ3YonWd/mpWyVpQNXdVhvpFglG6LQ3r1ZlZxG4tJ1UPkeCEKI6TxVTxfjDTU lGRHfrPSDVCBwY5kVX34XejsM3+3wl0+XynFbgCjBw6fkywEUA4Q8Jz9oJcy6oIGbz oEVwUOwqBv22WLCGoFb22ZbuO2H75EAQP60xliAgxoPNmxUrZFdaPLm1a1owS/jL7b XcuYfJ0FqyUH38NLCXBsgiQ6pyQV7eCEWMSnDxxnOD2WX/Nb2OCPf1KWNK+sdyB09Y 6HKHxYMV8Vq9w== From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , Suren Baghdasaryan , Kent Overstreet , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, "GONG, Ruiqi" , Jann Horn , Matteo Rizzo , jvoisin , Xiu Jianfeng , linux-kernel@vger.kernel.org, linux-hardening@vger.kernel.org Subject: [PATCH 5/5] slab: Allocate and use per-call-site caches Date: Fri, 9 Aug 2024 00:33:06 -0700 Message-Id: <20240809073309.2134488-5-kees@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240809072532.work.266-kees@kernel.org> References: <20240809072532.work.266-kees@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=10722; i=kees@kernel.org; h=from:subject; bh=WwVEHZxXP/denNiV+dxfYy7hsfC8akgc4PDoJIzTgCM=; b=owGbwMvMwCVmps19z/KJym7G02pJDGlbjxn1Tdz6Xbdh/s6WL17vDgv/KesuvZIWvl70Z+FGn /x65TbrjlIWBjEuBlkxRZYgO/c4F4+37eHucxVh5rAygQxh4OIUgInc/8HIcLbu+bae0p1OfcrS 3ofrO24W5Xt+yPlS1MpqFhTQf9RCiuF/gKqyfHxc5OOXDx6wX+2LmJTo7tm64m3mIyZr87OiVWY 8AA== X-Developer-Key: i=kees@kernel.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 X-Rspam-User: X-Rspamd-Queue-Id: 2A64A2001F X-Rspamd-Server: rspam01 X-Stat-Signature: 3kb94zr7kusrarcjn78pnheumjf9pmhr X-HE-Tag: 1723188793-370049 X-HE-Meta: U2FsdGVkX1+wuNeesk3wpP2vfZ1Lu0h01EbfBgIUtu7H0pfq3U4DKdV1Zi2DFafchq2/J8qujzQu9Qo6/IDuI6z6ssgCBNF3Qe+v9LWDvb+Vy541PZ+oPB5DRejQDWFWEcXw+jP/QOmpksIdX9qJ4GsLPsDSOA2+2xq//BKQSfFC70+bSe9hdvCoS5Yc4Xzd/4suTuQGWcuiBUdyQFZT85u8vOkMz2yDMNjUJ17bdZbuteCGyR5Ni1MU94s3sFcntyw0mCFQLkS90Ye+GRk9d9YqIBxHt5s8kCzoxFhELQcThzzMU/Cbmkv710iWfsBTsh40ykrGsVsN2EWpnyWEWYwrFOm2XssYgq0drJvG+KWw1zS1kjadO+6mO4PKIVcZ2R5rU6wR67a+SHMuL7SBJIybuf3hJ/x2DYjnITsvoPKDF162uwFQhNA8FyZY0Tq5k2vPiN5eQLqL9xg2Z7K09egdogyHefjIa4PTFF+LgGUNtIIsSxNaMkI+KgOfYzIHyvzvZo7wTcqqWUEdZQsTSrkln7uHXp2gDjBJg0b2MM8m3TrxgQNBvegWON6VAVQRFtmnkPt8IHfpC9KBzb/km6Ja5uLyAzU1VfXpOpDahLc0x4VcSWeqEYUVKzAJa96IIkBdO5hPVdfSIxhNopk71/nzIcXTTj5Mh1Se9vkuQ4oOY9ERS44rq08Xu02YJWJlFg5hEsVl4eO3DML9PNqFVuFMkynmkMJInwRU0bjzrf8xUrt+Qx+FKZj9RcMvQrv3wAvXtiaYU3bmgyaolOaygwbN3XQ3mCKCoYHqyYMgy+BK+XWq8qStxQTeSNM0B0OEr7tdfShRVaPIFn/J/I50jfIjjh3br4x57lyAyacfRoRizxnNITfdPGIW9/mpkfMHjFC2h3dWnGXww3XfzyA5X2Wq2TNxTtq0rKq9nMhvLGVa3SIBtNEzMJUqOQxmS1cwJXqsznVkicgTFldXdIw 59NcRlwW lPBhToCbr4xeATeRKYyhDb6T0/lQOtJAMTY5IzYzFbIoXNvP5McQ0qAeWDNii0iQNnhE634BJlPKmor6anOT4cBRJ5kD37EgEe44EyaEn21vqcKbvtrOu7OKA02tkA0vyPF81kI8z4QP0dG1QY59OnaCampY3TuYxJziXctytNusjcAN02NcYI+gOcI6S4AWOcZ/xtp5k9YRZHHUZXr/QfvnJ7W8zWNVazXp5lAkd2/Ww5js1+SOPI086gh0g2StoGH/6nRzrv4gRjwovIaVha/T3HmjiJExN/hjrAr+PFK4ZR74XtUpZS2/AQhOhGHRnQ6q3eqbeyjaTgKtogALrNRc64K5gePIiPxt24qAlHwxgZ/wpp/ZEv7fKe2xC11KhbXeosxTN/I74MOLD2aK4pV2AODQocvZ00+MNTpqgaq3Y7hEtdcLcdfaTzuRdc1izwclcAO1ddJgRO6Sy1opRUUq6B1p7cqVhf1VkYot6SCVYnpfJDkE44EM2h/z0TTKdSp1dpk0p5PC82AlAkNLy1wg6Gg4ONrhKQJB6OgmwFLdbZd41dGrZgIKBSIVyu4ikfmRO8lJpXytFKwyAFoYX4aC2sQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Use separate per-call-site kmem_cache or kmem_buckets. These are allocated on demand to avoid wasting memory for unused caches. A few caches need to be allocated very early to support allocating the caches themselves: kstrdup(), kvasprintf(), and pcpu_mem_zalloc(). Any GFP_ATOMIC allocations are currently left to be allocated from KMALLOC_NORMAL. With a distro config, /proc/slabinfo grows from ~400 entries to ~2200. Since this feature (CONFIG_SLAB_PER_SITE) is redundant to CONFIG_RANDOM_KMALLOC_CACHES, mark it a incompatible. Add Kconfig help text that compares the features. Improvements needed: - Retain call site gfp flags in alloc_tag meta field to: - pre-allocate all GFP_ATOMIC caches (since their caches cannot be allocated on demand unless we want them to be GFP_ATOMIC themselves...) - Separate MEMCG allocations as well - Allocate individual caches within kmem_buckets on demand to further reduce memory usage overhead. Signed-off-by: Kees Cook --- Cc: Suren Baghdasaryan Cc: Kent Overstreet Cc: Vlastimil Babka Cc: Christoph Lameter Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Cc: Andrew Morton Cc: Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: linux-mm@kvack.org --- include/linux/alloc_tag.h | 8 +++ lib/alloc_tag.c | 121 +++++++++++++++++++++++++++++++++++--- mm/Kconfig | 19 +++++- mm/slab_common.c | 1 + mm/slub.c | 31 +++++++++- 5 files changed, 170 insertions(+), 10 deletions(-) diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h index f5d8c5849b82..c95628f9b049 100644 --- a/include/linux/alloc_tag.h +++ b/include/linux/alloc_tag.h @@ -24,6 +24,7 @@ struct alloc_tag_counters { struct alloc_meta { /* 0 means non-slab, SIZE_MAX means dynamic, and everything else is fixed-size. */ size_t sized; + void *cache; }; #define ALLOC_META_INIT(_size) { \ .sized = (__builtin_constant_p(_size) ? (_size) : SIZE_MAX), \ @@ -216,6 +217,13 @@ static inline void alloc_tag_sub(union codetag_ref *ref, size_t bytes) {} #endif /* CONFIG_MEM_ALLOC_PROFILING */ +#ifdef CONFIG_SLAB_PER_SITE +void alloc_tag_early_walk(void); +void alloc_tag_site_init(struct codetag *ct, bool ondemand); +#else +static inline void alloc_tag_early_walk(void) {} +#endif + #define alloc_hooks_tag(_tag, _do_alloc) \ ({ \ struct alloc_tag * __maybe_unused _old = alloc_tag_save(_tag); \ diff --git a/lib/alloc_tag.c b/lib/alloc_tag.c index 6d2cb72bf269..e8a66a7c4a6b 100644 --- a/lib/alloc_tag.c +++ b/lib/alloc_tag.c @@ -157,6 +157,89 @@ static void __init procfs_init(void) proc_create_seq("allocinfo", 0400, NULL, &allocinfo_seq_op); } +#ifdef CONFIG_SLAB_PER_SITE +static bool ondemand_ready; + +void alloc_tag_site_init(struct codetag *ct, bool ondemand) +{ + struct alloc_tag *tag = ct_to_alloc_tag(ct); + char *name; + void *p, *old; + + /* Only handle kmalloc allocations. */ + if (!tag->meta.sized) + return; + + /* Must be ready for on-demand allocations. */ + if (ondemand && !ondemand_ready) + return; + + old = READ_ONCE(tag->meta.cache); + /* Already allocated? */ + if (old) + return; + + if (tag->meta.sized < SIZE_MAX) { + /* Fixed-size allocations. */ + name = kasprintf(GFP_KERNEL, "f:%zu:%s:%d", tag->meta.sized, ct->function, ct->lineno); + if (WARN_ON_ONCE(!name)) + return; + /* + * As with KMALLOC_NORMAL, the entire allocation needs to be + * open to usercopy access. :( + */ + p = kmem_cache_create_usercopy(name, tag->meta.sized, 0, + SLAB_NO_MERGE, 0, tag->meta.sized, + NULL); + } else { + /* Dynamically-size allocations. */ + name = kasprintf(GFP_KERNEL, "d:%s:%d", ct->function, ct->lineno); + if (WARN_ON_ONCE(!name)) + return; + p = kmem_buckets_create(name, SLAB_NO_MERGE, 0, UINT_MAX, NULL); + } + if (p) { + if (unlikely(!try_cmpxchg(&tag->meta.cache, &old, p))) { + /* We lost the allocation race; clean up. */ + if (tag->meta.sized < SIZE_MAX) + kmem_cache_destroy(p); + else + kmem_buckets_destroy(p); + } + } + kfree(name); +} + +static void alloc_tag_site_init_early(struct codetag *ct) +{ + /* Explicitly initialize the caches needed to initialize caches. */ + if (strcmp(ct->function, "kstrdup") == 0 || + strcmp(ct->function, "kvasprintf") == 0 || + strcmp(ct->function, "pcpu_mem_zalloc") == 0) + alloc_tag_site_init(ct, false); + + /* TODO: pre-allocate GFP_ATOMIC caches here. */ +} +#endif + +static void alloc_tag_module_load(struct codetag_type *cttype, + struct codetag_module *cmod) +{ +#ifdef CONFIG_SLAB_PER_SITE + struct codetag_iterator iter; + struct codetag *ct; + + iter = codetag_get_ct_iter(cttype); + for (ct = codetag_next_ct(&iter); ct; ct = codetag_next_ct(&iter)) { + if (iter.cmod != cmod) + continue; + + /* TODO: pre-allocate GFP_ATOMIC caches here. */ + //alloc_tag_site_init(ct, false); + } +#endif +} + static bool alloc_tag_module_unload(struct codetag_type *cttype, struct codetag_module *cmod) { @@ -175,8 +258,21 @@ static bool alloc_tag_module_unload(struct codetag_type *cttype, if (WARN(counter.bytes, "%s:%u module %s func:%s has %llu allocated at module unload", - ct->filename, ct->lineno, ct->modname, ct->function, counter.bytes)) + ct->filename, ct->lineno, ct->modname, ct->function, counter.bytes)) { module_unused = false; + } +#ifdef CONFIG_SLAB_PER_SITE + else if (tag->meta.sized) { + /* Remove the allocated caches, if possible. */ + void *p = READ_ONCE(tag->meta.cache); + + WRITE_ONCE(tag->meta.cache, NULL); + if (tag->meta.sized < SIZE_MAX) + kmem_cache_destroy(p); + else + kmem_buckets_destroy(p); + } +#endif } return module_unused; @@ -260,15 +356,16 @@ static void __init sysctl_init(void) static inline void sysctl_init(void) {} #endif /* CONFIG_SYSCTL */ +static const struct codetag_type_desc alloc_tag_desc = { + .section = "alloc_tags", + .tag_size = sizeof(struct alloc_tag), + .module_load = alloc_tag_module_load, + .module_unload = alloc_tag_module_unload, +}; + static int __init alloc_tag_init(void) { - const struct codetag_type_desc desc = { - .section = "alloc_tags", - .tag_size = sizeof(struct alloc_tag), - .module_unload = alloc_tag_module_unload, - }; - - alloc_tag_cttype = codetag_register_type(&desc); + alloc_tag_cttype = codetag_register_type(&alloc_tag_desc); if (IS_ERR(alloc_tag_cttype)) return PTR_ERR(alloc_tag_cttype); @@ -278,3 +375,11 @@ static int __init alloc_tag_init(void) return 0; } module_init(alloc_tag_init); + +#ifdef CONFIG_SLAB_PER_SITE +void alloc_tag_early_walk(void) +{ + codetag_early_walk(&alloc_tag_desc, alloc_tag_site_init_early); + ondemand_ready = true; +} +#endif diff --git a/mm/Kconfig b/mm/Kconfig index 855c63c3270d..4f01cb6dd32e 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -302,7 +302,20 @@ config SLAB_PER_SITE default SLAB_FREELIST_HARDENED select SLAB_BUCKETS help - Track sizes of kmalloc() call sites. + As a defense against shared-cache "type confusion" use-after-free + attacks, every kmalloc()-family call allocates from a separate + kmem_cache (or when dynamically sized, kmem_buckets). Attackers + will no longer be able to groom malicious objects via similarly + sized allocations that share the same cache as the target object. + + This increases the "at rest" kmalloc slab memory usage by + roughly 5x (around 7MiB), and adds the potential for greater + long-term memory fragmentation. However, some workloads + actually see performance improvements when single allocation + sites are hot. + + For a similar defense, see CONFIG_RANDOM_KMALLOC_CACHES, which + has less memory usage overhead, but is probabilistic. config SLUB_STATS default n @@ -331,6 +344,7 @@ config SLUB_CPU_PARTIAL config RANDOM_KMALLOC_CACHES default n depends on !SLUB_TINY + depends on !SLAB_PER_SITE bool "Randomize slab caches for normal kmalloc" help A hardening feature that creates multiple copies of slab caches for @@ -345,6 +359,9 @@ config RANDOM_KMALLOC_CACHES limited degree of memory and CPU overhead that relates to hardware and system workload. + For a similar defense, see CONFIG_SLAB_PER_SITE, which is + deterministic, but has greater memory usage overhead. + endmenu # Slab allocator options config SHUFFLE_PAGE_ALLOCATOR diff --git a/mm/slab_common.c b/mm/slab_common.c index fc698cba0ebe..09506bfa972c 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1040,6 +1040,7 @@ void __init create_kmalloc_caches(void) kmem_buckets_cache = kmem_cache_create("kmalloc_buckets", sizeof(kmem_buckets), 0, SLAB_NO_MERGE, NULL); + alloc_tag_early_walk(); } /** diff --git a/mm/slub.c b/mm/slub.c index 3520acaf9afa..d14102c4b4d7 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4135,6 +4135,35 @@ void *__kmalloc_large_node_noprof(size_t size, gfp_t flags, int node) } EXPORT_SYMBOL(__kmalloc_large_node_noprof); +static __always_inline +struct kmem_cache *choose_slab(size_t size, kmem_buckets *b, gfp_t flags, + unsigned long caller) +{ +#ifdef CONFIG_SLAB_PER_SITE + struct alloc_tag *tag = current->alloc_tag; + + if (!b && tag && tag->meta.sized && + kmalloc_type(flags, caller) == KMALLOC_NORMAL && + (flags & GFP_ATOMIC) != GFP_ATOMIC) { + void *p = READ_ONCE(tag->meta.cache); + + if (!p && slab_state >= UP) { + alloc_tag_site_init(&tag->ct, true); + p = READ_ONCE(tag->meta.cache); + } + + if (tag->meta.sized < SIZE_MAX) { + if (p) + return p; + /* Otherwise continue with default buckets. */ + } else { + b = p; + } + } +#endif + return kmalloc_slab(size, b, flags, caller); +} + static __always_inline void *__do_kmalloc_node(size_t size, kmem_buckets *b, gfp_t flags, int node, unsigned long caller) @@ -4152,7 +4181,7 @@ void *__do_kmalloc_node(size_t size, kmem_buckets *b, gfp_t flags, int node, if (unlikely(!size)) return ZERO_SIZE_PTR; - s = kmalloc_slab(size, b, flags, caller); + s = choose_slab(size, b, flags, caller); ret = slab_alloc_node(s, NULL, flags, node, caller, size); ret = kasan_kmalloc(s, ret, size, flags);