From patchwork Tue Mar 5 10:10:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 13581906 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E9138C54E41 for ; Tue, 5 Mar 2024 10:10:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 66CC56B0087; Tue, 5 Mar 2024 05:10:31 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 598996B0089; Tue, 5 Mar 2024 05:10:31 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 413416B008A; Tue, 5 Mar 2024 05:10:31 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 2B8B36B0087 for ; Tue, 5 Mar 2024 05:10:31 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id D513D40DE7 for ; Tue, 5 Mar 2024 10:10:30 +0000 (UTC) X-FDA: 81862565820.13.F575FCB Received: from mail-oi1-f171.google.com (mail-oi1-f171.google.com [209.85.167.171]) by imf25.hostedemail.com (Postfix) with ESMTP id D93C5A000E for ; Tue, 5 Mar 2024 10:10:28 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b="LMtbRI/a"; spf=pass (imf25.hostedemail.com: domain of keescook@chromium.org designates 209.85.167.171 as permitted sender) smtp.mailfrom=keescook@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709633428; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=D8rMBFW0tg47V1vhR+hT7ZAeuwLlaibSycxaH+YicYQ=; b=U6MDQxZsBFp3tE9Txdb7SdY/FbtE2fyLAO+KX8bp0JACNTkChQ4WrxLiKH8tdu4VuSUKiS B1+3hSbBmrVwELUv5WDXLl+BKB57h9VSf+/B2SdXV6LP1A7hqyphbq0C2smZmt75sSyQcx FffImkzyxtjgJfmBmRZykJM2Zq4a3zs= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709633428; a=rsa-sha256; cv=none; b=3pYaOZZE9yoPgRYuS7MMSaI/miB9bQqAqwCvW5llKnxDtMUIj0aooPkcfcb7Ho045frJRO U29Zf8K6sQk8sgrqXuKVYX6IXtAEiBTQWr5pBBxaoXZbtEDZrSaTDrRIV814ZzZPg1JO7E 6lueiRrwgBNhP/7TFQ0VCCPDH5nkvyY= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b="LMtbRI/a"; spf=pass (imf25.hostedemail.com: domain of keescook@chromium.org designates 209.85.167.171 as permitted sender) smtp.mailfrom=keescook@chromium.org; dmarc=pass (policy=none) header.from=chromium.org Received: by mail-oi1-f171.google.com with SMTP id 5614622812f47-3c1ea59cf81so1177187b6e.1 for ; Tue, 05 Mar 2024 02:10:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1709633428; x=1710238228; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=D8rMBFW0tg47V1vhR+hT7ZAeuwLlaibSycxaH+YicYQ=; b=LMtbRI/aNeLQE1gJYbIVjvi4a5+6w72rzf2TotbmrdGuJ24BAX78QfOXRQwMuSZt35 rgqd1ch1HbAeibl5aBDLprxUo7nj005puIBDmbNhARVfntCj7U1woyGR5pR30MTNcexq 6AsOlxLbkrRqjQhf1khnzH4DOvalsbC+RufwU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709633428; x=1710238228; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=D8rMBFW0tg47V1vhR+hT7ZAeuwLlaibSycxaH+YicYQ=; b=omxNsvWR3LLCR1rF6XKD5jmbsbaXcPfAhWGfxkAar23g6qb6juidOOWCnyLtm5NoEv ++HZGx4t95UX6y1tX0BIcfUNKiz6xhvxQxxN+83wb2EvJk0B6MukGAelyfpOMhLtF/Hs 0oY0Gp5dEfLdvAyTjodapnv6NS/72WVGVhpgpvMioqdE98moIlM9hQRluxoYS+kZJ69/ ry5I972wHTfScP9kjU4qzDYl40pNMeaU+tp4fFCumXi9RzGBGM9fmTbSAVyGu2W2OCle eGT6/6Phx24lZuLT+YHv9ZnTRp+OpWTf1dvJuOD0xTtZmFnvWNa/uxOkJrqDKtlWW8k/ zZ2g== X-Forwarded-Encrypted: i=1; AJvYcCX5fKQyV5fuESmAqd/OUzI2A9+WJ9ojlWRr1/uJZKMyundm2ifynad3+dkKQXx7olSPTGrSdKRG7sUBpHcWLn49g/s= X-Gm-Message-State: AOJu0YyL+u6TCW3jMYo63Vy7UFyPdYbymnDyQlMKCpa3otKrdoHBviDT 9Zi8WkGAc3gFsmodZOASCPQKtpJPQtVIVrkEcHgVOnOaAkolHDwWy4q6cf1hkg== X-Google-Smtp-Source: AGHT+IGYfAbScDqt9gSvtMMgpvMOmJMAT2r+uZEpZv8yZZyNk7NeKQx1h1ByxW1TZ8pqZAe02uLOcw== X-Received: by 2002:a05:6808:607:b0:3c1:559b:4290 with SMTP id y7-20020a056808060700b003c1559b4290mr1272920oih.42.1709633427854; Tue, 05 Mar 2024 02:10:27 -0800 (PST) Received: from www.outflux.net ([198.0.35.241]) by smtp.gmail.com with ESMTPSA id y185-20020a638ac2000000b005df41b00ee9sm8692546pgd.68.2024.03.05.02.10.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Mar 2024 02:10:27 -0800 (PST) From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, "GONG, Ruiqi" , Xiu Jianfeng , Suren Baghdasaryan , Kent Overstreet , Jann Horn , Matteo Rizzo , linux-kernel@vger.kernel.org, linux-hardening@vger.kernel.org Subject: [PATCH v2 1/9] slab: Introduce kmem_buckets typedef Date: Tue, 5 Mar 2024 02:10:17 -0800 Message-Id: <20240305101026.694758-1-keescook@chromium.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240305100933.it.923-kees@kernel.org> References: <20240305100933.it.923-kees@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1794; i=keescook@chromium.org; h=from:subject; bh=Wyo8wzm4MTDz4JStiYyO080jO0R+E8V6S6bv9ZTR4iA=; b=owEBbQKS/ZANAwAKAYly9N/cbcAmAcsmYgBl5u+QdzwZRrBHJ6q6tZF3ljbnIROOFCv0Jfi5y dLhu0dVnMWJAjMEAAEKAB0WIQSlw/aPIp3WD3I+bhOJcvTf3G3AJgUCZebvkAAKCRCJcvTf3G3A JhRGEACXqOYvigduxdiLg5s3NKoZ5sAGZvlIOApR3AAy5NlVdVm3El7+ClqD7kAsaRASlCZzfCE SnrdJUJTmH8pIf9m0+2frgqPwuRmMlb09607iDkUQPLPN1YxJMR27tBwJWq9BgA3ubUWo+QsdgT YXE7VP7nPQdqREOzraeyi2MHH+U3DNUaP61lbMW6KxmdVKDTNvRohGXZ4t+fDkAx3qPc5WUijJu +T0ZdLwbHIqRgXmTleQTynxNAt+ias93uGlGLgEDjoQZXs3asvnUvukRlD4dRoL8Jw9iQijFq1Q Dt4OD7yUPU+PD4XJr3GZIJhcJtIYM4IVHGlAVfYnKDclM8zlcITotaYHXQpXMql4PslEEE4eao3 2lj3oguta9XJ6ZA8lMy1oJUDczWNwOLRtiE4ocEojO7GA2/sMkSDKFBWiQoXoqqozQ72/GOfjya z0P7wyB5UCUQt13UQzDHMpYOQ/D5+iqge17qSf3KrlYD3bDZTO0UxJ6zzyDqVGApKdVJXmzv1nS 5BCsV9JIyHRHMs1bxaWyexVgrO27nMe+VW7LxuZWBm53lHjxxXQXT0bPu26dEUkeQrRMBN6BrfR Hi8QkUDZmoqSL/wiOIGv/c+qZfMpokFWp+beQoz6lhK2cuoYx+8nFdtG2JAcDlfnXKJ5RU+DLD4 BUBdZTAp9dA4F9w== X-Developer-Key: i=keescook@chromium.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 X-Stat-Signature: zsw4hwoeeqxt1sfrg9p51paaat9bpwt4 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: D93C5A000E X-Rspam-User: X-HE-Tag: 1709633428-208932 X-HE-Meta: U2FsdGVkX19x6nnOg7jcZwEqgGZjaLOTV57sx3uauD+07qhls35Jozcpvtdjzqls50qGY4kUD5Ss/+dPQeboYLsGvJcnOKWGE3n2kqZvXN4BKeEfV8WpumoPEJ2pClnAFrWrIDHNR7WPHYg6lwNL5M3d+qaxv2I1jzS6GeuBDLmQsEXws2CtvEZSDhY29yXOv3St7myN0S1kqt4gPOXMP07Th81Bx3qjbJlvxXU9Kzxo9q7wB899vFwzDUcxKTpO3yIr/+ShXeqSa8jmHAkU5f2hnuKuGQR+4yP5VDscwFSB+CW/ar7zCudxccKXmOfm06HgsMNtlpQA4jJ55tREElrAXifrM0ri7eBdkI3SYsAPyNlPUxACN/l/fU2Vdi+S+mUGZwCCf1LQz149wPkwgFpZyQco5p7Pkn7xIYD3nQb6y1/lFdB3TEUqEyejxKdnsyKRjWbta19usS4wMzounXqJTDTwBN9kaRpnnvWLYrWxSiBcUAvaRaM8AqyuAWDoA7dRqkW8mUOlUelw2yiZHCDc5DG5BarXcDWIS9MXvrAs2lltVwiFamAYqDF+gGaW6w1yoIRNtUfzZh81TX0OUrdPjTCDUkdH0KsBcZ/Kx8gGFJsahIA6HHjw8oedV+lGaaR21IbSaiiLpa/wfHKjU1s6TU5ZafDMx3yRt9yJVC68EZATiko27HXMqBYX3ovC0NDJvu5uDaaO7TJ/gH65gAhr5EMejao/2DzDppff8UQWdLJnC7N1Igo0H2NOUVSKe5oA/LeYEBGzzbGHvrSwdvkHWAa7jYriahEjuCjDIugeW2dnhd8AJo9d4Erhlw4hOvb0IPZRSiW92pzdQROAmzo+udgP+FFEidSpfmy1XkPzPNeSB/J/uvPticIhY4PKnbHvteYVFmDNbgcDQdaITzZ0Z9krR2MasbtdiFTeF6BILcb8qskpf7odaledy8P+TJRHvoMMnEPo1j3CzJh a7sfXbw+ vU84ty1jhNGSHJQBau7n7lveP/aYOBYVxAz+jFHNtf6MeZCAi5Kw7l01z5MNtPIxC5bq1mncHr2olOeCE5AjtKQYKF0c+Tm3MJ1ZHQh9rBsE7VgkzjJTaUbEZs3rP8vJ+pJI/AlyHqSkKrWcOAGjxUGOkxPaUPU6YDAK8pMpUAZB8mgdae/140X/Ge81Rubk5ccmFRfYvQvcf8i+ln9xYMjC/qdfISKGItBUk8d3nrHp72+izVo4EbDuTi5FIoIECROOg5EjT9j5cBS7R1pndaB9YBEKrBoui2JjoNcnWE4r2809xqPy+PtmJtNxyJmI/Al9iDhjOjuAISRHgVXBZIoBOAEbO/3OsCH/doXs6LzQ+s7+shU2U30T0siI41ds1+wTjLl5ZA1QX9cFxjJc1Fj1r5cONDKmko9tvSRXQZc9rViEAdp1Mr+guUIMYm0D3AFtUJgly/KRmZazIos6CLdhHsxBaS3FYFZTmT4cnPhsH2sSVIjdW71EOfp43d3vZHrTI6faMAkQZVLbeXeEkpz7gGY8WtF6X9xKEmPyNO4P2FTBer3o62EMY8yDS4/pmbnRwOEYRDuaQBkNB7aY3m3urI5PJRGwmUYUEEL2G/rG7UCKezE6JcQqbiwajBIaHP+LEA208UdPqJHoxc8QRDnuuu4pcw7J9gIu6eOAwqQpfD8s21QIke8A3YX7dQjHU2ethucHK/5CHlH89dwgEtYHF3kG0YtquWnHN5ByDm11lPbIgNrXi6VHAZ+uOrk0Dwo0b1p9cNtcCK6XuoWrGI2SdJp0FeF+fKEDdvqj7J8O+YWq6SK5dhD8DyZL8GNIYNaL2paEXpoJaZpSttIQGtYXelXxNDqZG0hpdAX7h+P67PMbY42nVaMGRbjnJYMoC7nqXd7Vs8DLIPaq2uj7cNxswSo1pbFLEUbeRaerE9uEqos8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Encapsulate the concept of a single set of kmem_caches that are used for the kmalloc size buckets. Redefine kmalloc_caches as an array of these buckets (for the different global cache buckets). Signed-off-by: Kees Cook --- Cc: Vlastimil Babka Cc: Christoph Lameter Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Cc: Andrew Morton Cc: Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: linux-mm@kvack.org --- include/linux/slab.h | 5 +++-- mm/slab_common.c | 3 +-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index b5f5ee8308d0..55059faf166c 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -375,8 +375,9 @@ enum kmalloc_cache_type { NR_KMALLOC_TYPES }; -extern struct kmem_cache * -kmalloc_caches[NR_KMALLOC_TYPES][KMALLOC_SHIFT_HIGH + 1]; +typedef struct kmem_cache * kmem_buckets[KMALLOC_SHIFT_HIGH + 1]; + +extern kmem_buckets kmalloc_caches[NR_KMALLOC_TYPES]; /* * Define gfp bits that should not be set for KMALLOC_NORMAL. diff --git a/mm/slab_common.c b/mm/slab_common.c index 238293b1dbe1..8787cf17d6e4 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -649,8 +649,7 @@ static struct kmem_cache *__init create_kmalloc_cache(const char *name, return s; } -struct kmem_cache * -kmalloc_caches[NR_KMALLOC_TYPES][KMALLOC_SHIFT_HIGH + 1] __ro_after_init = +kmem_buckets kmalloc_caches[NR_KMALLOC_TYPES] __ro_after_init = { /* initialization for https://bugs.llvm.org/show_bug.cgi?id=42570 */ }; EXPORT_SYMBOL(kmalloc_caches); From patchwork Tue Mar 5 10:10:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 13581910 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3226DC54E41 for ; Tue, 5 Mar 2024 10:10:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 771B36B0098; Tue, 5 Mar 2024 05:10:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6FB9D6B0095; Tue, 5 Mar 2024 05:10:34 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4B1A46B0098; Tue, 5 Mar 2024 05:10:34 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 365CB6B0095 for ; Tue, 5 Mar 2024 05:10:34 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id D76F61C0221 for ; Tue, 5 Mar 2024 10:10:33 +0000 (UTC) X-FDA: 81862565946.21.69C3BD6 Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) by imf30.hostedemail.com (Postfix) with ESMTP id D01C58001D for ; Tue, 5 Mar 2024 10:10:31 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=BNoCoHdN; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf30.hostedemail.com: domain of keescook@chromium.org designates 209.85.214.174 as permitted sender) smtp.mailfrom=keescook@chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709633431; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=DCac96MRWhgh1/2EEmwcpolePHb0x1wbAjOzVa8BY+0=; b=ohyEEp9A4QTDSH3794fCxMwWHBvniQAgP6B1+nuoS0VLuyWdl+UHoNcaeiAJiB1IZHTgEa sCUBxuaGdWuSXFq/JdUV75OFFStgbsOW5csMyGvIaYzJ4GZv0jjgWd0kUhHt6OUTVlnaUt YepO5ZtFjixzbGeRrzJhz37piRXsXQo= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=BNoCoHdN; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf30.hostedemail.com: domain of keescook@chromium.org designates 209.85.214.174 as permitted sender) smtp.mailfrom=keescook@chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709633431; a=rsa-sha256; cv=none; b=0yA35rZ6NT9o98fd+PsCQ/HZKcAkQ/9EM01oju2rWLZbAMBFRdgvOYga8yAtAKO7OiLr64 vltY+ir6ihjzonmEaZRt6QQccMgiN5wC/yuP85T/V7hffGUIHxfYsK+qmaz+W6X4lTyIkL b3IVgu4ugSIOAWiRpcOPG58gqfOuXCE= Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-1dc1ff58fe4so47588845ad.1 for ; Tue, 05 Mar 2024 02:10:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1709633431; x=1710238231; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DCac96MRWhgh1/2EEmwcpolePHb0x1wbAjOzVa8BY+0=; b=BNoCoHdN7890973QOtoQErf4zHirwl10BqwfIMAybJrRuaSw2yUJUDxAnCyVPW2yrC RLAqwiJPKvO63S+MZATqlMzp1pGXO2JgJETZAmB7quU9FUE7+FcFlM3SOptJ+pYIy+hm 45q4FC+FlGZt2HDobVBLjUUA/RuE3viVenuYQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709633431; x=1710238231; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DCac96MRWhgh1/2EEmwcpolePHb0x1wbAjOzVa8BY+0=; b=UybFbl7jXAOXNOVqHJpYHQqGyJybLdvq8XZsn7whAhVKKcLoD/onr9CbthjGqB1O7b Vyof+ugl/YiB56vbdtT51VCpqdwb15uuFCRgZrh1D5i4XJJwgav6bSdwDuilT9royBD2 2C9sWCuSETB4irGr9+sbjDNRoZdkCfATCrB8IyuD2teRXIMbpJHelUdqri1BCqv47PHl rUugSfcVW76eFBeoW49OaHvNYo2FQF7RIuSY0LNx1dtnWlx3izW8KUrqOHcQS403IOLo iPh7eLHhQL3snlXQbAKFeMi2kZqQuXC8E/oE0SL4h6ZLLRxFyDtkSoWG4BLlsJlscn1w EpxA== X-Forwarded-Encrypted: i=1; AJvYcCVbX3dRMY9MdUqU5jIgG2uxt81tssBfO3sac9bgg/s3CFyuoA0iGfG+RVSLgsnUEqE6QiIQqH/QBy2AW23pAScOUgY= X-Gm-Message-State: AOJu0Yxx60KfBkS1sQPqPByagV7n/5Nw77PvUkL1r893k/0Vc1RqMGK6 IHNTUNa66MyVhj98uFC9VYKmlZnEX2puhu/e/oDPZBlwc2V9IhjvOY9H7w2rqcD8FpPNpgLlvdw = X-Google-Smtp-Source: AGHT+IGQD7egJY3N2jy+qTOLk98RlzGpfSgXtQ8qNR83/EYIZpLvDqeKxbLyOSOxTxFrRo9HzKuI6Q== X-Received: by 2002:a17:903:230a:b0:1dc:fb9d:402d with SMTP id d10-20020a170903230a00b001dcfb9d402dmr1584663plh.58.1709633430745; Tue, 05 Mar 2024 02:10:30 -0800 (PST) Received: from www.outflux.net ([198.0.35.241]) by smtp.gmail.com with ESMTPSA id n7-20020a170903110700b001dc9893b03bsm10063682plh.272.2024.03.05.02.10.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Mar 2024 02:10:27 -0800 (PST) From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-hardening@vger.kernel.org, "GONG, Ruiqi" , Xiu Jianfeng , Suren Baghdasaryan , Kent Overstreet , Jann Horn , Matteo Rizzo , linux-kernel@vger.kernel.org Subject: [PATCH v2 2/9] slub: Plumb kmem_buckets into __do_kmalloc_node() Date: Tue, 5 Mar 2024 02:10:18 -0800 Message-Id: <20240305101026.694758-2-keescook@chromium.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240305100933.it.923-kees@kernel.org> References: <20240305100933.it.923-kees@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=5763; i=keescook@chromium.org; h=from:subject; bh=dsYwePKLsCLMzShH9MEQL3PgSxerM2N+02XNBrgQB3U=; b=owEBbQKS/ZANAwAKAYly9N/cbcAmAcsmYgBl5u+Qh5OvUkS2WLPejBFffVYr3X7AdnuJ+g+ss wawW9pNc9iJAjMEAAEKAB0WIQSlw/aPIp3WD3I+bhOJcvTf3G3AJgUCZebvkAAKCRCJcvTf3G3A Jpz5D/90L2rsJJvl6+P+wDTk00KFpS0yvEOEnTFGCVFHdm1gLjjRgBEZOCCPWclrHgGTkc8hBia nOccJ3xAylGH42H9q91nYChjlw/IZUry4o+uK9hKbxXcdC57QMGm38XenTvoOd4Eeqry07ryFmS Vtt0qurq/+5ki/6NecKkeipCrrNoA5RCxIDY0d95UiJV9EO4VZiWoeJ6j+YW6sCHuYqojJUmth8 nASULi9PDpCRxUlhbDJGHsgd7vhSSVKhr+iAVeILIgKu0ABrT+JLWABfwImg0azZt5Ls1q6akOw gnS+llQjZPPJ3vLk3Yqy3vdLff1gpUvad4G0TTJ69rOlneNSxmYTtrZZcxO/7jqHYl3Y3+pZWcI Ag/9AGf2AlCALy0WdUlBXXNfJE4lwywJnBfsr7GkeTtnd8JhTwB2w92l+xuHv5JUmzB5cSk6EOK dLaYXgNTlKnMTIq8eiOLrYhtEZGOpGgC15pVSSGiUBGviTyv+Rdi59+L+NVkvcHyG9z+2qVDc9/ Lvm8+g1VDnXM8KSNOfUFbX48M+wlbPFG1fhjcwxttYIsH2AqUtWu8K48KlXf5ugIsLGlhRQmjEX 8bFw/IhfXee/9Bt9phiU1TThOqIbodEUWlAb14Y/A/0YMUIOJbkeUWTR5mjkPZDaIFUh0MEGkE/ lg0QYFHUdAVLRJg== X-Developer-Key: i=keescook@chromium.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 X-Rspamd-Queue-Id: D01C58001D X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: a5gyrprjg4i856u9hxfbdweatk1ao9qh X-HE-Tag: 1709633431-564793 X-HE-Meta: U2FsdGVkX1/qRF4pRmQUJCfPzE84onkNeniSuW8qfEVFk75sgNGeb53hI3UAZvJMBdnlaiGygH7pHtxFpE7wdMHPsQPPH1Fnx34VEpvpLLCyJ7XMa+6BV5+0zVcOsvzGqyPx3fIJrhSgr9TbnJSyAsYD8WDwfWzKxqXMrmek9EsY0AjWXT6BvcPmoK5aFkZeROOSwg5dSEd1yVK3KDzNJ/0qPynA7zUDuR6gDXJtJ31XKW448/lAeuLc4+xt+A2mIwY/hL5PI6tdbciU9VfUkkwQwXWxTLUu+jULM0fSsxoESs1Uj4W9pz4mr7UacBsWQlTTsLKJcC3vc1AcOKyoiUacZ9dHRBjpKLfEWwkqdQRP4McBQl1nRsZwlm502T++vsm393idmBZabp8T9GGFCLW7li+8l+VZcY06afDPtv8yCJql2Ae6LT9g7MEfPyhsLueAFvZl3MSv8lzSJIMCuZQmYXuVOdP5r4tEqJbXY0U2L4F9QsJcIEFcW1vKCzIo9W6hM62PS49j8tMiPeB3m4FZQsDWsEL70/8HGL/s9Av+Mx9StEytB/TtmlgGSed8POz9zwlxNu/Ec5wW8ncj6tib2c/vPHEXq3GQR0iEcsybOfJdvPRCfLpCl68VTqGOY3jGL8MXZH7zVNB3206k4EKamtvkjxOx3YZ6IJ/5SeSYzSt7A/MqspZfS+jW/bR7ouIi1MO+EvLudUaeNP1K/LexqwNwyx7mRj6WXgtD0eJi7lVk6wqdzu1SPlQOBcnJZKUKktm93Qr5jv8Vsris5goT3Y2KqDJz/c2RNzwD0EH1gRkrW5+59gbZpChEBNrVAXn2ihE5NNSp9hFsL6RJaOeR47g0IXV4j6aRlUokfAwNm0XUscPuE0l4NRzkqrthkfn1WLwhYyQuERi7K0/E2w4MuCnpXpDzZ0Uryq0iQCZrJxrA6h/uuLl8NsWs+oSwLmYH8kZTS8elHDGkjhQ saXbYYMB /SdSQON/RnslS9RkUvf6GI6vhg7DAhtUV/KTzGPpfl/4hO2EB8Ngljvh4BQwG+oscF7EY4Q4wVAaEg8KWajlf++dKPadbJHqzvYtYY+dM/I05U9bwyrVdGQRN3Rg09v5YqHCriXImeI+I3HLVsMBC5Q33GY0lwTKgi1f2uPeEaVf8Ck+4XSoEiI1Zc8qEtKVcu2EZas27lSekHTGKmy3PDVbBfQ4Tz0QKcDnNJWQrVlXF8TYC+/Z/nl30Xb6NKZT/iKxSeHDM93pD9saQCkQr7L9iU79kn0JrI0K0BvHYHpgWbt6PuLQKnQBAdk6yYZ0NDnmpWAeQLi6bHEuhpu4l/1pPRVaCb2xaSzeDPuX147jXbQyaQa/3Ln1OH0N0TzQgEbnnGAlyDsRTWl7jUk6IFx86dey+OBowdFLUWkYl2+4JzIB0t8ke6qq9A14xzv9Fo588OUQpfGxCKhf94A6lNlpRxD2IVLLHAvUkoUkf16N3YgcbQQPic+8h1Vk9WeCnZFBZzUOdS2yB/xJg7LBFD5fvGPcE/YIZD52bg46EZ1pCdVi8bbEOGud4M+mRwaPIo+NVQvu+bcqqpgbV5b2vVOUhuShqPFh7VNP5idvqojaQfrgSySCNGUI9np7OyxvN7ZBjjIsA2RSsx/bAIlQ5kcY4e0/nKkyNaux84lhe71mDimv2c06VF8uqQL2KIydcW4IeFG0dWRtZ7+EHSqZKmah4R5O3pRarqhWFSwtmx1EVgtNcdellQ3PeRRx6N5+8FnY201TDlHrJo5mVtkUCJBVopS5K16RAHSQV/GkIm116uqcw3JL5E6WfbSWEQWkWk+aNQFA26YtSkc11V10ZCwkwZ7eod4UUCTsfJWo9SkUnUIs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: To be able to choose which buckets to allocate from, make the buckets available to the lower level kmalloc interfaces. Signed-off-by: Kees Cook --- Cc: Vlastimil Babka Cc: Christoph Lameter Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Cc: Andrew Morton Cc: Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: linux-mm@kvack.org Cc: linux-hardening@vger.kernel.org --- include/linux/slab.h | 8 ++++---- lib/fortify_kunit.c | 2 +- mm/slab.h | 6 ++++-- mm/slab_common.c | 2 +- mm/slub.c | 12 ++++++------ 5 files changed, 16 insertions(+), 14 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 55059faf166c..1cc1a7637b56 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -508,8 +508,8 @@ static __always_inline void kfree_bulk(size_t size, void **p) kmem_cache_free_bulk(NULL, size, p); } -void *__kmalloc_node(size_t size, gfp_t flags, int node) __assume_kmalloc_alignment - __alloc_size(1); +void *__kmalloc_node(kmem_buckets *b, size_t size, gfp_t flags, int node) + __assume_kmalloc_alignment __alloc_size(2); void *kmem_cache_alloc_node(struct kmem_cache *s, gfp_t flags, int node) __assume_slab_alignment __malloc; @@ -608,7 +608,7 @@ static __always_inline __alloc_size(1) void *kmalloc_node(size_t size, gfp_t fla kmalloc_caches[kmalloc_type(flags, _RET_IP_)][index], flags, node, size); } - return __kmalloc_node(size, flags, node); + return __kmalloc_node(NULL, size, flags, node); } /** @@ -686,7 +686,7 @@ static inline __alloc_size(1, 2) void *kmalloc_array_node(size_t n, size_t size, return NULL; if (__builtin_constant_p(n) && __builtin_constant_p(size)) return kmalloc_node(bytes, flags, node); - return __kmalloc_node(bytes, flags, node); + return __kmalloc_node(NULL, bytes, flags, node); } static inline __alloc_size(1, 2) void *kcalloc_node(size_t n, size_t size, gfp_t flags, int node) diff --git a/lib/fortify_kunit.c b/lib/fortify_kunit.c index 2e4fedc81621..c44400b577f3 100644 --- a/lib/fortify_kunit.c +++ b/lib/fortify_kunit.c @@ -182,7 +182,7 @@ static void alloc_size_##allocator##_dynamic_test(struct kunit *test) \ checker(expected_size, __kmalloc(alloc_size, gfp), \ kfree(p)); \ checker(expected_size, \ - __kmalloc_node(alloc_size, gfp, NUMA_NO_NODE), \ + __kmalloc_node(NULL, alloc_size, gfp, NUMA_NO_NODE), \ kfree(p)); \ \ orig = kmalloc(alloc_size, gfp); \ diff --git a/mm/slab.h b/mm/slab.h index 54deeb0428c6..931f261bde48 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -404,16 +404,18 @@ static inline unsigned int size_index_elem(unsigned int bytes) * KMALLOC_MAX_CACHE_SIZE and the caller must check that. */ static inline struct kmem_cache * -kmalloc_slab(size_t size, gfp_t flags, unsigned long caller) +kmalloc_slab(kmem_buckets *b, size_t size, gfp_t flags, unsigned long caller) { unsigned int index; + if (!b) + b = &kmalloc_caches[kmalloc_type(flags, caller)]; if (size <= 192) index = kmalloc_size_index[size_index_elem(size)]; else index = fls(size - 1); - return kmalloc_caches[kmalloc_type(flags, caller)][index]; + return (*b)[index]; } gfp_t kmalloc_fix_flags(gfp_t flags); diff --git a/mm/slab_common.c b/mm/slab_common.c index 8787cf17d6e4..1d0f25b6ae91 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -698,7 +698,7 @@ size_t kmalloc_size_roundup(size_t size) * The flags don't matter since size_index is common to all. * Neither does the caller for just getting ->object_size. */ - return kmalloc_slab(size, GFP_KERNEL, 0)->object_size; + return kmalloc_slab(NULL, size, GFP_KERNEL, 0)->object_size; } /* Above the smaller buckets, size is a multiple of page size. */ diff --git a/mm/slub.c b/mm/slub.c index 2ef88bbf56a3..71220b4b1f79 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3959,7 +3959,7 @@ void *kmalloc_large_node(size_t size, gfp_t flags, int node) EXPORT_SYMBOL(kmalloc_large_node); static __always_inline -void *__do_kmalloc_node(size_t size, gfp_t flags, int node, +void *__do_kmalloc_node(kmem_buckets *b, size_t size, gfp_t flags, int node, unsigned long caller) { struct kmem_cache *s; @@ -3975,7 +3975,7 @@ void *__do_kmalloc_node(size_t size, gfp_t flags, int node, if (unlikely(!size)) return ZERO_SIZE_PTR; - s = kmalloc_slab(size, flags, caller); + s = kmalloc_slab(b, size, flags, caller); ret = slab_alloc_node(s, NULL, flags, node, caller, size); ret = kasan_kmalloc(s, ret, size, flags); @@ -3983,22 +3983,22 @@ void *__do_kmalloc_node(size_t size, gfp_t flags, int node, return ret; } -void *__kmalloc_node(size_t size, gfp_t flags, int node) +void *__kmalloc_node(kmem_buckets *b, size_t size, gfp_t flags, int node) { - return __do_kmalloc_node(size, flags, node, _RET_IP_); + return __do_kmalloc_node(b, size, flags, node, _RET_IP_); } EXPORT_SYMBOL(__kmalloc_node); void *__kmalloc(size_t size, gfp_t flags) { - return __do_kmalloc_node(size, flags, NUMA_NO_NODE, _RET_IP_); + return __do_kmalloc_node(NULL, size, flags, NUMA_NO_NODE, _RET_IP_); } EXPORT_SYMBOL(__kmalloc); void *__kmalloc_node_track_caller(size_t size, gfp_t flags, int node, unsigned long caller) { - return __do_kmalloc_node(size, flags, node, caller); + return __do_kmalloc_node(NULL, size, flags, node, caller); } EXPORT_SYMBOL(__kmalloc_node_track_caller); From patchwork Tue Mar 5 10:10:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 13581908 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A52BCC54E4A for ; Tue, 5 Mar 2024 10:10:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D17CC6B008A; Tue, 5 Mar 2024 05:10:32 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CD2DC6B008C; Tue, 5 Mar 2024 05:10:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A55396B0092; Tue, 5 Mar 2024 05:10:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 811856B008A for ; Tue, 5 Mar 2024 05:10:32 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 49167C0E05 for ; Tue, 5 Mar 2024 10:10:32 +0000 (UTC) X-FDA: 81862565904.03.4695BFA Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) by imf09.hostedemail.com (Postfix) with ESMTP id 635F514000A for ; Tue, 5 Mar 2024 10:10:30 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=FfYsyFIw; spf=pass (imf09.hostedemail.com: domain of keescook@chromium.org designates 209.85.214.175 as permitted sender) smtp.mailfrom=keescook@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709633430; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=99J+yh1I+zD4qoBsjvOLBvxyDWRgmU0/2arb8UvstmI=; b=CFCwB5IWHQfPeKBejzYN3OEXWUYbUKCe9C6TgTFH2NXaSDtfP4mTpjnQi9TOgDd2fL/ndS p4ycgpi1YfzGh8VCIcZkN402xpc6c1TysBA9cQ/tRKDM9ugHr3wHcvBiDq8QnrdNi+HP6G 9MlPoH5CczzOpaFFdyngxScuG7Nbpqw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709633430; a=rsa-sha256; cv=none; b=kkLk7Z1TTMTYollEar2wo+Ai9Jbn/E+NNA0krr+maZSOhGvPvKcnghAwZ+TmxI+NM/HL+8 4wrJohMYiskrfJUw1u7MWF2TtmXApitJXbG7Bhs6zDjdn7txOfza02FKjrwK+xr8MvO1nN fiiMr1vVK/rczJUZfk8o0WRUSAarUfQ= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=FfYsyFIw; spf=pass (imf09.hostedemail.com: domain of keescook@chromium.org designates 209.85.214.175 as permitted sender) smtp.mailfrom=keescook@chromium.org; dmarc=pass (policy=none) header.from=chromium.org Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-1dd10ae77d8so15610095ad.0 for ; Tue, 05 Mar 2024 02:10:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1709633429; x=1710238229; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=99J+yh1I+zD4qoBsjvOLBvxyDWRgmU0/2arb8UvstmI=; b=FfYsyFIwaGOeQ6KWQkIEuOC6mluRFOfKAcny9Bh87WSNtIHd2P7nvzUy0sly1+JHx8 jJOizy7FD71AgRqJVtg2oPyB53RVLchrzSD5+75DvCXoBsRQ9S5p/TUVhzpiQqafawI6 aMynzY7zE9xVYlqFUeXAOQIPq9mPnJtD8rNho= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709633429; x=1710238229; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=99J+yh1I+zD4qoBsjvOLBvxyDWRgmU0/2arb8UvstmI=; b=EOZNjhUI23Xm58pooTwYpOyAN8260kloV3GEufzMaY5nEPRcCASf4RnROC/LKXalri 6tUnpfFn3J3T19Dd1tQXNQKZvz4QVgM3qvk0+k44zzAEB/hbyGN6v7F/+9HAY/lFENPv MmnpEw2I34OUIUWJseGSYNsrtqu41VuDE2+Zj6Zt3nLQi9UtC4jDmp3KL1AoRBcNzlu+ 1cHJREaycofd8/YGF2WzYRg2mcQ8PyHcRlPctmr0cSWAGX0W1BNQAzbuhe9Ez+p5F2Pj SDFC9kwgfxzo0oLT0Rnxa3CfRjdShY1o/xOUPdR2mWNBgVQYf57OG6IMe8+wqMaltp/n YBGQ== X-Forwarded-Encrypted: i=1; AJvYcCVWtlNReJopDJQ5HdUNMYPvLTgL0QVrdetX/gJUfh7gEdWRIZRPSU4i5cFjZ/klQ67ObMXpikpnaIb01IUEz4f+HoQ= X-Gm-Message-State: AOJu0YyVfSg4XYbH7uBgx8Jih9SUcAVeIJIsSlrBEzECxOaXLeC/42Nq jS0nkoZxabzqKBsz6scsfJMkGdw4w6uIMA+8zfM/cpAZ+sythmlmq2Pp2I+N5A== X-Google-Smtp-Source: AGHT+IFqjHRULQpfLuOIfWcrg86BprEdjHxZ7A04s1os533fRHV67h+yavqSGXCERg/CRb9UyIa8KQ== X-Received: by 2002:a17:902:d4c1:b0:1d8:ab27:d76c with SMTP id o1-20020a170902d4c100b001d8ab27d76cmr1643127plg.51.1709633429324; Tue, 05 Mar 2024 02:10:29 -0800 (PST) Received: from www.outflux.net ([198.0.35.241]) by smtp.gmail.com with ESMTPSA id n15-20020a170903110f00b001dcc3a46a6bsm10081558plh.262.2024.03.05.02.10.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Mar 2024 02:10:27 -0800 (PST) From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, "GONG, Ruiqi" , Xiu Jianfeng , Suren Baghdasaryan , Kent Overstreet , Jann Horn , Matteo Rizzo , linux-kernel@vger.kernel.org, linux-hardening@vger.kernel.org Subject: [PATCH v2 3/9] util: Introduce __kvmalloc_node() that can take kmem_buckets argument Date: Tue, 5 Mar 2024 02:10:19 -0800 Message-Id: <20240305101026.694758-3-keescook@chromium.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240305100933.it.923-kees@kernel.org> References: <20240305100933.it.923-kees@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2996; i=keescook@chromium.org; h=from:subject; bh=JqAs2ONNE7kMO16unP+0GI5vQbPsq9Mj6KA7f6Xw/go=; b=owEBbQKS/ZANAwAKAYly9N/cbcAmAcsmYgBl5u+QU4fAZ6X0I2VVjtMMEATXAWJJ5dqn0j2T+ P8LqusMSw+JAjMEAAEKAB0WIQSlw/aPIp3WD3I+bhOJcvTf3G3AJgUCZebvkAAKCRCJcvTf3G3A JmecD/92uSdARNPOF/Ig41R9iblLk9UxCbO1fB2HKnaa/w+yLHjuEWP2fptZr1u4cl7oBi+V+7e XCMsvWK4pkOYAP7NuRX8GU6AxmvUqlZpmQXa283pIL7cWbyoBryIVD0uVp7YjUDWMT3mZYo8kTI WOxJrg2BS22j8FjJ0SmfK28wsBeG/6y4f11iNleBFW5KpXlGaRw2NoFGPv/oYzmVJdpOvno7tv1 SrJhdM++M5F4fg8dy8UZdPFSBIjoz4VC49dlXRzdsM9GoSdV+aakrKnzGaX8WPqr1QTfDcy5qEv RCskJbWRYLonWFp9KcpeVrWeXB43I5oPU6H/DHx0PGqLxhEKdHMseOCtWkAKNPVhtqOufAQLdct 2XPAIyB+CIavIHRn8oZvNB8dhF46U1bKOM9jFISSARfnGEZp8WC4c4OStJpN9nCnwYHPz32sXy5 IFPtLolXYwV9mqfABdpmFQb/QG6+XuvjFWZ+HpxdxuAD/41pG2HrHPS3XlG77pqtqM+/vtWm5LH hf8/tXhzxjo8B8vey28JSnydpkHd63SeElyfCf/JQ/Vbe5wzsRt/oHddPw1OThs95gN/vEjmHXH rP+vV3/i9F7vKu273jzC4MIKRmYWSs3/Lc000Nj3NCnpIp6uaAeGZFpJIzs1h/qLp3rQGBKT8yw TPX8rWd+pV6qhAA== X-Developer-Key: i=keescook@chromium.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 X-Rspamd-Queue-Id: 635F514000A X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: igtkfth6tdj7a5ze46uh4kg4qzjfnkua X-HE-Tag: 1709633430-911072 X-HE-Meta: U2FsdGVkX1/G26VbJ0mEs1o0W7fAfHgHZ0MjNFkm4iJdv1nrsb+M60giQMRisgE8EDnSYFrHsNL6o8kLP7f6BYjV4q2CpGinElklzR2HOmn7M0X8jHNg+jovh2aW4K6HvlRCOEqVqh9RjF0rZD+a3qmcTxqsqgnCYzOEJZEOTd81qT/wHHIfGx26DdwMBAf1Urp1vgtejJGLUvGH+fl96bBUuePOnrkfNjV1oDdX+yX54GF7uBq+WLx3JsAgfAs7Cdd2OJcuhdZIX7Z5qgw2UIDOY31cpnWVM6OZQtzJ+X2nRrHfCI6yEbbgXbJtqdZbFVRKaFfZbqnLj6ZoZFpepLqK6eEcboayacb5cE8F76xW7LKbRd+Xl+ryULx7Ns6iN0vAqvtCrG+DMryRX2wrZqP/iOysLKlV7gdn1F+bmAjbBa5HvOA4/ukVOKXa8Ro/N1t/6+ra7YA/j3/T7R2SIGH2MUBmH1GyQNR5WKhhRiiOb8GL9IKcEAmCNZCk2G76JZkMaRaQbITpUaTVfZ3SuVsoCykuz+4eRAtO2n41cEiBChLQERSXH+dQtiF6dNkYPWcwzR8o0cZtJEqtKgAOubX/K0sbKQKQskEWWr0dFAujyxxVEiRZC+zjsR0XACjsE+BFt0NB6AFza3BJ0GmAz8Q/cRr+pjK6ya9z1A1FhahpW/LW2qGLjgt6vwCCZaC7WXfkzhUNHE8oaZM5J6mpiC7qIC/G6iS/aqkznKcfhAc+kFSUdoHrh8pe4L8AXDO+VfFjdlfqZXQdp3BT2lbu00VTDN/Yz6PNgfM3luq72BhL5cIYLTIQlhioI8D4LFkBGnQBgImt35boHyKg3iOsegklkfUTMAkWNFR6RCJBiV1JDGuQywt0THpRWpaequQxXeQxAkct7IM6hFTSVKNiVbs0tPG1VABQSsfNRl/OK2HVVfFHipGWsqyxn/0K5PFEzjsM7Blz2hP7/fCLNT/ cc+md4J7 6cnMKyDYRUqwp0bIEwuykwODa9b381C3RkOp7RAw5NrzUeJIhK1071jGbpNhpBer7gwa+dTyIeRaHa4uk6CnRJHCD5Qk7e0xTmjz3MIT0/46sHTAgwaNbr0jOrlzIPU4jOJhLF286ec9KONlolY8RQXbx6ji12kKZ8tEMVo1t0+02hAYxejxNEDynUuK9zkjDFfcogwYfV+KGHu5SPaRKsR7/AAM6SOHIDPRxNsKwWkTFURg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Provide an API to perform kvmalloc-style allocations with a particular set of buckets. Signed-off-by: Kees Cook --- Cc: Vlastimil Babka Cc: Christoph Lameter Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Cc: Andrew Morton Cc: Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: linux-mm@kvack.org --- include/linux/slab.h | 9 ++++++++- mm/util.c | 9 +++++---- 2 files changed, 13 insertions(+), 5 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 1cc1a7637b56..f26ac9a6ef9f 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -723,7 +723,14 @@ static inline __alloc_size(1) void *kzalloc_node(size_t size, gfp_t flags, int n return kmalloc_node(size, flags | __GFP_ZERO, node); } -extern void *kvmalloc_node(size_t size, gfp_t flags, int node) __alloc_size(1); +void * __alloc_size(2) +__kvmalloc_node(kmem_buckets *b, size_t size, gfp_t flags, int node); + +static inline __alloc_size(1) void *kvmalloc_node(size_t size, gfp_t flags, int node) +{ + return __kvmalloc_node(NULL, size, flags, node); +} + static inline __alloc_size(1) void *kvmalloc(size_t size, gfp_t flags) { return kvmalloc_node(size, flags, NUMA_NO_NODE); diff --git a/mm/util.c b/mm/util.c index 5a6a9802583b..02c895b87a28 100644 --- a/mm/util.c +++ b/mm/util.c @@ -577,8 +577,9 @@ unsigned long vm_mmap(struct file *file, unsigned long addr, EXPORT_SYMBOL(vm_mmap); /** - * kvmalloc_node - attempt to allocate physically contiguous memory, but upon + * __kvmalloc_node - attempt to allocate physically contiguous memory, but upon * failure, fall back to non-contiguous (vmalloc) allocation. + * @b: which set of kmalloc buckets to allocate from. * @size: size of the request. * @flags: gfp mask for the allocation - must be compatible (superset) with GFP_KERNEL. * @node: numa node to allocate from @@ -592,7 +593,7 @@ EXPORT_SYMBOL(vm_mmap); * * Return: pointer to the allocated memory of %NULL in case of failure */ -void *kvmalloc_node(size_t size, gfp_t flags, int node) +void *__kvmalloc_node(kmem_buckets *b, size_t size, gfp_t flags, int node) { gfp_t kmalloc_flags = flags; void *ret; @@ -614,7 +615,7 @@ void *kvmalloc_node(size_t size, gfp_t flags, int node) kmalloc_flags &= ~__GFP_NOFAIL; } - ret = kmalloc_node(size, kmalloc_flags, node); + ret = __kmalloc_node(b, size, kmalloc_flags, node); /* * It doesn't really make sense to fallback to vmalloc for sub page @@ -643,7 +644,7 @@ void *kvmalloc_node(size_t size, gfp_t flags, int node) flags, PAGE_KERNEL, VM_ALLOW_HUGE_VMAP, node, __builtin_return_address(0)); } -EXPORT_SYMBOL(kvmalloc_node); +EXPORT_SYMBOL(__kvmalloc_node); /** * kvfree() - Free memory. From patchwork Tue Mar 5 10:10:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 13581909 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BC8A5C54798 for ; Tue, 5 Mar 2024 10:10:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 48B746B008C; Tue, 5 Mar 2024 05:10:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3EDB56B0092; Tue, 5 Mar 2024 05:10:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 217946B0093; Tue, 5 Mar 2024 05:10:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 071A36B008C for ; Tue, 5 Mar 2024 05:10:33 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id C420680D70 for ; Tue, 5 Mar 2024 10:10:32 +0000 (UTC) X-FDA: 81862565904.26.982A44D Received: from mail-pf1-f181.google.com (mail-pf1-f181.google.com [209.85.210.181]) by imf28.hostedemail.com (Postfix) with ESMTP id 034E7C0015 for ; Tue, 5 Mar 2024 10:10:30 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=M9NwuOVq; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf28.hostedemail.com: domain of keescook@chromium.org designates 209.85.210.181 as permitted sender) smtp.mailfrom=keescook@chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709633431; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BeUYrlEBxVnkyCc/znLfwIQ49KwaP7QNJJ/emVdVuls=; b=Xo+io59PmeYd2dpnJhNjpCrAX7w3jDBAue5zrf87MxZV7K1RkFkB4rPxRBpl0BcAJ5xKGL bEW51lS3poDO/ikw2Hq5C4X0hg2ZO52TqfP2WXUjv9dPsPY/6AeBznWx7CWZmuz5Gb0yjL dWRHDKVdYviYTup6CMZMwpgwMARd+ZY= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=M9NwuOVq; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf28.hostedemail.com: domain of keescook@chromium.org designates 209.85.210.181 as permitted sender) smtp.mailfrom=keescook@chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709633431; a=rsa-sha256; cv=none; b=mXT6EdPXkKpt4zi66B4QgBdtOfM7t19SyYx10SlhGZyY79OYHygF0cNnae0C+LGoNztPrf qEucjLOk2KdKxPViEEkCRCkxOiF3MsfWHith1X9cz0c3UBs7rHCFFYKP9SCRWF0UShz8pO Jyz9Shym7mHDZox5hTKtMScbc7cafmM= Received: by mail-pf1-f181.google.com with SMTP id d2e1a72fcca58-6e5c81ccfb9so2708909b3a.3 for ; Tue, 05 Mar 2024 02:10:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1709633430; x=1710238230; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BeUYrlEBxVnkyCc/znLfwIQ49KwaP7QNJJ/emVdVuls=; b=M9NwuOVqEfXPzYqqZofuANr6XcC5ZNhd9Af3EsJn2Jdvg1ps2J80jIhqNmxBSRuWX3 sZbn5JSAIGodujz1WS7ZPRa6iSS4YYXTjuBw3cAMXkjmLAVDBGknYfLFCjVjJVP2tN11 tg1HWS0jPkJ56dI/ltLaJPbUWBfwX02dTj1iE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709633430; x=1710238230; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BeUYrlEBxVnkyCc/znLfwIQ49KwaP7QNJJ/emVdVuls=; b=R2gLlN6SrhO7braekYkA9JOhMIOvOQBkjZK8lFHFcnwav5T2915MHuVawH2i8An3Vf cx/KwnKssw+ZqUy9dF/wjywIGwJ/Aiiqt74Br15Y/QW6Xxyv86VPmh17MD8+eXVcB5Z4 VjZ319emaUc04VYZ6K1fiUYqfiOBmNDqsgjwn+HBVaAY4JAhyRPqPzOfXL+LlkZRAOxm SPKeUR2YnYpyjQeqMFqxqERFj9ir1UGTd3bYtgFHlQ5hK66X8xD8Nda7UpVh4ZNhZ287 CrW4KFYjC5MVyWlJ6a0IKF7hG+hVrYqZKi2j54VHIyeErtNzqvWJZAEzRLZsJe3TOaM5 c6lg== X-Forwarded-Encrypted: i=1; AJvYcCUZLV2hV3rXdkLu52yq5UNe8+3WoTinVHS6B/5E00GDiTuThsgsxAXJydjXMLJzvLyGXPVCiQ/seDzBxjp7MThDYeg= X-Gm-Message-State: AOJu0YxaNw50xOnVn++fng6cYXFSdmC5HlD67DQcBGFb5hvoTMqj8WGa RmXVTN3SpqrRBxr2YqrLXrBBhFmYqZT4ohvsC981JzH7nWA3YhshTBhtYeEcJQ== X-Google-Smtp-Source: AGHT+IEA1CSyGq/q0upjn8FhCc79X1Sz1fnRam7a7llwA6f0dlAvS0keaevB9x35pohBL6JFKtxYYg== X-Received: by 2002:a05:6a20:244c:b0:1a1:4cbe:bf74 with SMTP id t12-20020a056a20244c00b001a14cbebf74mr1486585pzc.15.1709633429921; Tue, 05 Mar 2024 02:10:29 -0800 (PST) Received: from www.outflux.net ([198.0.35.241]) by smtp.gmail.com with ESMTPSA id g5-20020a170902c38500b001dc78455383sm10146541plg.223.2024.03.05.02.10.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Mar 2024 02:10:27 -0800 (PST) From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, "GONG, Ruiqi" , Xiu Jianfeng , Suren Baghdasaryan , Kent Overstreet , Jann Horn , Matteo Rizzo , linux-kernel@vger.kernel.org, linux-hardening@vger.kernel.org Subject: [PATCH v2 4/9] slab: Introduce kmem_buckets_create() Date: Tue, 5 Mar 2024 02:10:20 -0800 Message-Id: <20240305101026.694758-4-keescook@chromium.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240305100933.it.923-kees@kernel.org> References: <20240305100933.it.923-kees@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=5125; i=keescook@chromium.org; h=from:subject; bh=y3vf/uqNKTNnOBLgqwE3tYFT2Luy4iHXIHzaxbjeLuY=; b=owEBbAKT/ZANAwAKAYly9N/cbcAmAcsmYgBl5u+QPWE1fKjC5eCeZH4vakqfDEWTI/IohhLqT GoUV/5ej76JAjIEAAEKAB0WIQSlw/aPIp3WD3I+bhOJcvTf3G3AJgUCZebvkAAKCRCJcvTf3G3A Jm/UD/jfSpO31HjXWApCYeolkFhb19QZEq8SrIOIqsqGoGZVAU3k6RWg8yQniQMy3yngBQfa2Ww h/QK0A6lEPzXyLI42kPBmua1RHFrT+Vqe3/1WofvSSsTESICw9TsnzitVinrUOM0nQo51nWISXI rTAc2Dqc1+eteEhuO2mBf61hDJZKYlE3dAE5Oy4yiPLQFOXU363T3VyPS1+jHmjCX9a7U/Fwp1L RE0Jo+IrwYWkV7JCru2Lf9QUOZcLZe1kaDT1S8KGCLBUj0qGfSQ0+6oYCVle7ZtWnAgM69VdfEM u0aSFr9DMPQLnL8DW+S2i4Y7E5d0OaG/m+vACFCAyEOjbMvqBotLX2oVqDLe3wNMWZN/GBxUDdu 9D/jUe03FvtktaMo2UTni3Qsx3d0boMxXGArJ5wo+vyGDDDBs8hxF/3ZhnEd58EUf4/WPWb7Hax 2EybOpRinpRqyal2KGNFPDIBOG4YgGqWYgGXRb3pw4pH6eFdr/PLQhggawSo/Wg96qL8WHZRvZ8 Ndwaph2SCDS3DM+r5nhF8mm7Tw5P9exoRn+a0tJNyv0i8CcN7W2IbJjkG9d/LV/uokfH1SLJWNJ VY+Op6bioLEyekWFsA4tTJyWA5wDB2UiYy2llni7suxYh5HYAzl4ULcPr/X6iHSyvXfscabaGjQ iPexMVWCtL cF/ X-Developer-Key: i=keescook@chromium.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 034E7C0015 X-Stat-Signature: cyc4m3uswxzdgncnxdmatn9h519p3u49 X-HE-Tag: 1709633430-892272 X-HE-Meta: U2FsdGVkX1/xnrxuS/dkWEYL3rk30GmGLvkFJiOqFqjfUueAEPL6bZLe6iBGi1gguiG6wVm0McKDADZFH1Ta3PPLf9H8DprW+amRRckOFGO5YLRbHsak9HcdTtgNMSeXASQOmk4BrfvId2HJSX0CQMvk9vNrF8zaZOypT8oyMQlZE1kHMzzuIAKxhnQYydv0HMpobmc5FsdExUjpUbf5C63ih2biiw920uptjo7pZoDAprd/114hs4dAIrXlIxcC4FFOt1b0vzmFyMGeLRWV12dYTHHrX2bl0Scra3iBgkJpAOVcpPYI8FE8tIBklA3Zymq21Kr4IqGOoiRzx/mYmf73nARqJqztXzrsxXTROOSWDmoVfd3bHpV1xhEuj8DLgZOViYeO0GoQSeZAtyJOHr/kFKA0Ug5xmXx7opWvccpus1AiX3z4CqqaQxkP4196Gn5EHy7W+NhggbC1kvYU6XW35mbTZIbs/OjMxuxWkSZdg+0suCJtJX/8Wol+IwrvHmDLwOeLF8lturx+CgXEaZZADGl93u9blMwHyj1d3x7vkwtDu5Tsx924Q45Y8l9s5UEygAWqb4cY9mzFS7+PBoVqTu0adLNX9Het/3ivjqh9dfXvOsht7edpO410eahJgcLl0n1WMkxlDBaA7iSUQqoDOXSyJNZLuEx1EX9SWVSOYNOd6KaUhPnLVuRz3Kgb+sALT70AjLY2I5wn9iCWkJdBxefECxh8PyO9v8J0TInhVpjnI5S42Y/t0AGtxVFl5gXh4N+QPrbLNIGzzZbM6xgNmK4oPXsPK1YoFjSHrW0xoJ+LpYKa99a1T/HSHxtr2XJ7d+Pf6WTidVlNyNTOvdA88vrZGdIu0ar/MvCS+4l/WdptI1KH04aI3CETabr8BH8GZ3PhGAoLDzpSM5nQzxV4jxnS34ZLghkVQWkLNxqw/IHIDYSK0ZWyyqI34ar5k/kXHCGOx9m+BETJ8Gl QqJHGR8r wnr76KB2O+5IZl7awYlg4NBTSFZdR0+b9yyNiSEU6TeIjLIVUFLEsRKi8Y8lA8leW8AAfoBl9ZD4FI5Sgj7ICfgIl5ZvvfBRqZrhbUK9QEfp/9BeuQ6ToBcXA0BR3kOeMIeUxwuSQB9mXkYgXlFPW+pr9EqKNB7q33gtwMbCB8MkH3vgho7YwrQ8GV9Zi5exruWxDfwnrdxQZamsEL9nhlSIudtf+6mJGx5GCO+hRl4NbvhRET1SJDLRfHA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Dedicated caches are available For fixed size allocations via kmem_cache_alloc(), but for dynamically sized allocations there is only the global kmalloc API's set of buckets available. This means it isn't possible to separate specific sets of dynamically sized allocations into a separate collection of caches. This leads to a use-after-free exploitation weakness in the Linux kernel since many heap memory spraying/grooming attacks depend on using userspace-controllable dynamically sized allocations to collide with fixed size allocations that end up in same cache. While CONFIG_RANDOM_KMALLOC_CACHES provides a probabilistic defense against these kinds of "type confusion" attacks, including for fixed same-size heap objects, we can create a complementary deterministic defense for dynamically sized allocations. In order to isolate user-controllable sized allocations from system allocations, introduce kmem_buckets_create(), which behaves like kmem_cache_create(). (The next patch will introduce kmem_buckets_alloc(), which behaves like kmem_cache_alloc().) Allows for confining allocations to a dedicated set of sized caches (which have the same layout as the kmalloc caches). This can also be used in the future once codetag allocation annotations exist to implement per-caller allocation cache isolation[1] even for dynamic allocations. Link: https://lore.kernel.org/lkml/202402211449.401382D2AF@keescook [1] Signed-off-by: Kees Cook --- Cc: Vlastimil Babka Cc: Christoph Lameter Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Cc: Andrew Morton Cc: Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: linux-mm@kvack.org --- include/linux/slab.h | 5 +++ mm/slab_common.c | 72 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 77 insertions(+) diff --git a/include/linux/slab.h b/include/linux/slab.h index f26ac9a6ef9f..058d0e3cd181 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -493,6 +493,11 @@ void *kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru, gfp_t gfpflags) __assume_slab_alignment __malloc; void kmem_cache_free(struct kmem_cache *s, void *objp); +kmem_buckets *kmem_buckets_create(const char *name, unsigned int align, + slab_flags_t flags, + unsigned int useroffset, unsigned int usersize, + void (*ctor)(void *)); + /* * Bulk allocation and freeing operations. These are accelerated in an * allocator specific way to avoid taking locks repeatedly or building diff --git a/mm/slab_common.c b/mm/slab_common.c index 1d0f25b6ae91..03ba9aac96b6 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -392,6 +392,74 @@ kmem_cache_create(const char *name, unsigned int size, unsigned int align, } EXPORT_SYMBOL(kmem_cache_create); +static struct kmem_cache *kmem_buckets_cache __ro_after_init; + +kmem_buckets *kmem_buckets_create(const char *name, unsigned int align, + slab_flags_t flags, + unsigned int useroffset, + unsigned int usersize, + void (*ctor)(void *)) +{ + kmem_buckets *b; + int idx; + + if (WARN_ON(!kmem_buckets_cache)) + return NULL; + + b = kmem_cache_alloc(kmem_buckets_cache, GFP_KERNEL|__GFP_ZERO); + if (WARN_ON(!b)) + return NULL; + + flags |= SLAB_NO_MERGE; + + for (idx = 0; idx < ARRAY_SIZE(kmalloc_caches[KMALLOC_NORMAL]); idx++) { + char *short_size, *cache_name; + unsigned int cache_useroffset, cache_usersize; + unsigned int size; + + if (!kmalloc_caches[KMALLOC_NORMAL][idx]) + continue; + + size = kmalloc_caches[KMALLOC_NORMAL][idx]->object_size; + if (!size) + continue; + + short_size = strchr(kmalloc_caches[KMALLOC_NORMAL][idx]->name, '-'); + if (WARN_ON(!short_size)) + goto fail; + + cache_name = kasprintf(GFP_KERNEL, "%s-%s", name, short_size + 1); + if (WARN_ON(!cache_name)) + goto fail; + + if (useroffset >= size) { + cache_useroffset = 0; + cache_usersize = 0; + } else { + cache_useroffset = useroffset; + cache_usersize = min(size - cache_useroffset, usersize); + } + (*b)[idx] = kmem_cache_create_usercopy(cache_name, size, + align, flags, cache_useroffset, + cache_usersize, ctor); + kfree(cache_name); + if (WARN_ON(!(*b)[idx])) + goto fail; + } + + return b; + +fail: + for (idx = 0; idx < ARRAY_SIZE(kmalloc_caches[KMALLOC_NORMAL]); idx++) { + if ((*b)[idx]) + kmem_cache_destroy((*b)[idx]); + } + kfree(b); + + return NULL; +} +EXPORT_SYMBOL(kmem_buckets_create); + #ifdef SLAB_SUPPORTS_SYSFS /* * For a given kmem_cache, kmem_cache_destroy() should only be called @@ -933,6 +1001,10 @@ void __init create_kmalloc_caches(slab_flags_t flags) /* Kmalloc array is now usable */ slab_state = UP; + + kmem_buckets_cache = kmem_cache_create("kmalloc_buckets", + sizeof(kmem_buckets), + 0, 0, NULL); } /** From patchwork Tue Mar 5 10:10:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 13581912 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 24015C54798 for ; Tue, 5 Mar 2024 10:10:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C071D6B0099; Tue, 5 Mar 2024 05:10:35 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B67B26B009C; Tue, 5 Mar 2024 05:10:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9BB3F6B009A; Tue, 5 Mar 2024 05:10:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 822886B0095 for ; Tue, 5 Mar 2024 05:10:35 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 5383040DEA for ; Tue, 5 Mar 2024 10:10:35 +0000 (UTC) X-FDA: 81862566030.07.50A82B2 Received: from mail-pg1-f172.google.com (mail-pg1-f172.google.com [209.85.215.172]) by imf02.hostedemail.com (Postfix) with ESMTP id 56BFE80011 for ; Tue, 5 Mar 2024 10:10:33 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=Z7F4AqBL; spf=pass (imf02.hostedemail.com: domain of keescook@chromium.org designates 209.85.215.172 as permitted sender) smtp.mailfrom=keescook@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709633433; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YqypDeuefMJmnQ8sf5GSNnN+8VJFJPXjvaUOYfHLyEk=; b=p0E22ATK92k7j5G/Xa4wAFE66HqrhNKnfVSJZaZXdtkmT8rkGPni/Y6/gCeufu16RlN1ls NDAlBL8Z2kERDO4VdnDjCP4gn5O3yGHbFg5mke4itVOnKKjTFkV0Fp+SfGjFbAeGuRaGIE Dez3ZgqDrYuKYKF1ZTA/D97jvlY/Bys= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709633433; a=rsa-sha256; cv=none; b=1QOiKX0NrTM6dSdP31xmPS1d7eIsf0aYQJakhv+mVB1q4kIEUCQgOtR4nZ74Ly7UItY1KZ 6c49xRIj9u6enfvk1d7QyclhGQQ9SlFzRVpuu9Fn7hYt7enRkHd5NouZT0uL6jTTzfpHWA lBPKdEfwctfRiEd4gDil+j+mlgDEQS8= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=Z7F4AqBL; spf=pass (imf02.hostedemail.com: domain of keescook@chromium.org designates 209.85.215.172 as permitted sender) smtp.mailfrom=keescook@chromium.org; dmarc=pass (policy=none) header.from=chromium.org Received: by mail-pg1-f172.google.com with SMTP id 41be03b00d2f7-5c229dabbb6so374483a12.0 for ; Tue, 05 Mar 2024 02:10:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1709633432; x=1710238232; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=YqypDeuefMJmnQ8sf5GSNnN+8VJFJPXjvaUOYfHLyEk=; b=Z7F4AqBL+TqMLzt9CUsTOOGSjjgGQkUFpGbYl5ZQyp7cXTV/urTPQpVQ1SnNKA88t8 RayI9ozW+TTr/E1wTWk6vFRp+GXyY5DAXL6kZuNTfYWJiECVZcuMkTfgTcUUMIfsvGkj LPJancC5d94IfrIJJvInC7OeLLFnN+UBs0iAs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709633432; x=1710238232; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YqypDeuefMJmnQ8sf5GSNnN+8VJFJPXjvaUOYfHLyEk=; b=HLX5qLRS5RMy1Xyg1XFg9mPyHFVET8RFCOm0KqiCnYn4xgmMvec2RZTq31G5ERw6ox G1o+FgTlx/C39+KR4LahSu8JMUQRr9LM0irDXiHHQqSdhIBPdP/gJBkg2X2C1VKD0IGF gIYKLHEkTviJBiZdku/tp4AOyS2MOyuOWVh8EhvcFcnuWRgPAjQ6x+w29yKFotR15Jim S/LT9l4hqVHmchy+pPlSI7OuOaMwBTOb/H+boVtrdC4F+f26bx3ThOq6petzXDelltSA X1Lolt7MVTP8z+r3kDjOWokDfkf9bT0tf2IvwLavpzqYH3vyL1zQ3w7VWbkYOHtAOdd0 JzqQ== X-Forwarded-Encrypted: i=1; AJvYcCXXKl62MNWX5296aSyFcDPSCwwzhSc3NtrnPn4mmTLVsWi+CtWTcd1G655N6hnHtKjHReYwB6yGvvbFU7DweUejYKM= X-Gm-Message-State: AOJu0YxTRBFgMjzIxOcjfdA39fehMn9/l1DWY5NGYP+MGj76yydHcruM ggQFL7TIRriw41hJfdjmZqwsup+UOmpSHMCq8b8Mefise9HPGY0NPMdnzeVNug== X-Google-Smtp-Source: AGHT+IGeJZyVJ09gI9gdMeeZa2TNl9QqRwg39EBpS2gnYKvxtqTdu16IIH+Z1sVU6rZsLUOu3KxERg== X-Received: by 2002:a17:90a:2c8:b0:299:ecd0:483b with SMTP id d8-20020a17090a02c800b00299ecd0483bmr8668816pjd.10.1709633432154; Tue, 05 Mar 2024 02:10:32 -0800 (PST) Received: from www.outflux.net ([198.0.35.241]) by smtp.gmail.com with ESMTPSA id h8-20020a17090a3d0800b00299be0e00c1sm12409317pjc.33.2024.03.05.02.10.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Mar 2024 02:10:30 -0800 (PST) From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, "GONG, Ruiqi" , Xiu Jianfeng , Suren Baghdasaryan , Kent Overstreet , Jann Horn , Matteo Rizzo , linux-kernel@vger.kernel.org, linux-hardening@vger.kernel.org Subject: [PATCH v2 5/9] slab: Introduce kmem_buckets_alloc() Date: Tue, 5 Mar 2024 02:10:21 -0800 Message-Id: <20240305101026.694758-5-keescook@chromium.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240305100933.it.923-kees@kernel.org> References: <20240305100933.it.923-kees@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1274; i=keescook@chromium.org; h=from:subject; bh=qg6+zLyXOvclDNk40orVz0UwK8KHs3WN88KHdjuAoYc=; b=owEBbQKS/ZANAwAKAYly9N/cbcAmAcsmYgBl5u+QePW1cikxrAkyTkNPDsM7P6WLLogreKw78 DkUscrwcj2JAjMEAAEKAB0WIQSlw/aPIp3WD3I+bhOJcvTf3G3AJgUCZebvkAAKCRCJcvTf3G3A JkIeD/9SGhc8L8n+b0l4YrTF1D9VoErEfn1Ca1UNFW/W0gggb8pmBG/YuAzR/noRa8vP/Obacdv xX/kz3RVVUNa5CYWF20lMDU/Lp8oLF05rVXs0+FZ7Y+Hj4NcwGb4ZDEcTwKpevYm7+jo9WEDQ5d zMSamVXJp/wkyr1XfGZ7cxjHlDRKSjdplhbJgG0UpihMCnecqlqWCXwJbsKgeaEq4Sl26MCcrlM K0ULjcAvpm6F3d4s636IjEqcvuIG1obEke1QFfbVJC5/dJduyyEKm1Lgraw5dGP56FR+aC56qjS 65EMTIxf1kNAW+Fy8bPuL7WDntcwNqNENqpozW05vn4+JPT/wOhqoVmXjQZ3DQfbw1ikOaWsuGR +wrVI1e9aMbAxyN+NV6Dk09hIjGge2x1xZ4ikVrEqfYiJl/PlZG1rZ3YtAjKzVhhHSlziq1EXO7 eLtSu666WzevVuCoqb7mheOeAM4BFhlrgwfm+ZnWU0YM/Ycmrv70Q8134urLdvFSqmvc8qAL1CF xi+VJmmdh42aZ/SeMDt4C3lJogV3xKZGfmixjPA3HWDF8q7OPo6tVZ0GK5r+TW9BmbGeL5G/Mz5 ehqYwoqpMNk2S8s4t7WPvdOakD8dz8vt+onslIuOlwmQgDswizHCZFv3m2uwtvVSPNlj7ZIDrUi W3INC3N8UCVkL0g== X-Developer-Key: i=keescook@chromium.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 X-Rspamd-Queue-Id: 56BFE80011 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: gu8f8grcthsrify85wrdy8ink863w3nb X-HE-Tag: 1709633433-99752 X-HE-Meta: U2FsdGVkX188L0rLxYSakOHks2gMcV9oRKZ9/GTB+4zjJgL5BAnnDAVQhIfqh1BBfk64x2hy7WsVabZXllM4w7Mn6ZsejNjFA3I8mWhNo6dsqFwtWnWYfsEiFtZzKQeni9rjjv4/0BfG/EYuD/2JiQgNzTKBo7joGbj2jplwF0bpO3WlFovdaMh18wjTTjmcM9/Eyd4DQfxuXSQxTEcavNUc9w+3qHXsq8OWl/8HCYTIzYlbkNxbngI7vQ4pOzGqWtu39ZV8TYZrnVZwxRsxRaoNfR1mQuU4klhDLuOCRPhXTYXrT92EyRdG6w/D62zMPopBmcfQm5qq03ZOiA5Cw1Pwr05Hho7sc0FrnjiAmTfhEZ+0H+XEFh60vVKNdKm5nU4bH70lcMPzL2rOtGgwPfcqQsk/7uVJUVy06fzJaqtAaua+HnSzopunsPgRbszSYBMP/gikvSX8nyI/MxB2YdKLJq+xZXAnYSZcEgn2v3rJROOfdpweEFQVBeVKmw4vIY78MQ3/9tSf3wcMDlwLqekQsHw9Yuy5UCon+lVenRAFh1N4ks6xm7psAOfNkmzk9ij6UNXACB+RSfBYq0Vn5+s6b3aXnxRr58wJqkFwNKqsOeRnrZM8XtjyFy2jsaSp0cuPLFNREWoHZlkA4bf/sium+NDH/jhOFO1cobGRdWoTd3tuyvWZSfKOoHu8DInY3CqM9sD5m7/dHOQU1cGi9Eo2dtNvvrNmvOSuc+D5pIYa14ItGJKRHFwVLbBDRz/gMk8mS7tfl2SQUW79ybbiLI0H3P5mckkjEUjjaCM1Gp2Fy9yuvCLOjK+nglXYBe01Cb+uK34yPXNTadFr3S/PPNRwlgV+T1Tsp7qOWgy8LwyLwluvdnSwDb24NJXPwnUFbEqsG3j1Yaq0Lzrn++Stm4gjUpmqjCxHpPX/8DSypg0X869yLgmE9Ym5sWSXKPnPxhNDJZO+ETgK4NpQ6M2 enQ2ObM/ bTsioOPRsdx1srpFJqAIh1aS6A0QyRbXDmU+cvwKTf+S2Rwn+IJ08RWs2ZnjZnezbUJ+PKfJQot6Ea0TTyYCYPsDxv/9XMo4mUt+wImJSGRffiCHpW8pG0nkHZUprhZFdlXMQwMgBcDsmCYUnSUDlZ+P3Q+KaSCiZpxGs5Xeq8c8ebf5slP2miHxped+nmHHtxGSWqPPys4YMy1gyuwYuNSJzhvngdEV2AqXQYyWQkM75Sj9uXAr+kgMD3fQP5Dpfmre+hhZe/INDupCJjQlpQ+h6mJNQLGeebO58zxNiCG5V1eYJ1A+8qEo+b+gvEvxKXf6HsiJdHWAqX8capU9nGSZfvQYwrcUvn6p28M8eC4NiaDvpDU7FHwzGw1Tx7uI/MJ0R0uKU61VgYkLHioFABLUTC0qNfawl52D9vcFalkErRYT2wpvM46m7FVShCnzZGL53ySRMmXwdnHb2XgHlUrjOEJtZxK14hZ5naqGLPZwkDimDyiUAfYdniuwWOEs562R1iBbH+6ZTWO53JMcrgzrN8A1F7QX/4dJHTwVT2Fh1Cwm5n1faCMqHf8cC5vrc13KO7vLC3p5D4rs87TgfX0e67VxzFc1uKV6lIp4hym1OktFj/8IV4NUn8bk3mjUeG3n5uZRNPNEZubNHKdmVEyBfu7VDKBELMi/TjGLMQj5PcEQbr38EZ3pAMKhLrWogBTj9EsGDREnavwGdcAsFAvcwoWIvhUpeJ12Vm0yOoeJuLfdSbDsE7tv+LB0AvLQSE3lR4tfXjoukLDBIhpUE8DVrRv178wQ0yf+jsehIxCqMTS1N/zxIUVOJGZ1KYkMtPT25eQasIPa9iUijma2YyFGP40ttH2nJl5KvI+flrf8Iy80= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: To perform allocations with the buckets allocated with kmem_buckets_create(), introduce kmem_buckets_alloc() which behaves like kmem_cache_alloc(). Signed-off-by: Kees Cook --- Cc: Vlastimil Babka Cc: Christoph Lameter Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Cc: Andrew Morton Cc: Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: linux-mm@kvack.org --- include/linux/slab.h | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/include/linux/slab.h b/include/linux/slab.h index 058d0e3cd181..08d248f9a1ba 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -600,6 +600,12 @@ static __always_inline __alloc_size(1) void *kmalloc(size_t size, gfp_t flags) return __kmalloc(size, flags); } +static __always_inline __alloc_size(2) +void *kmem_buckets_alloc(kmem_buckets *b, size_t size, gfp_t flags) +{ + return __kmalloc_node(b, size, flags, NUMA_NO_NODE); +} + static __always_inline __alloc_size(1) void *kmalloc_node(size_t size, gfp_t flags, int node) { if (__builtin_constant_p(size) && size) { From patchwork Tue Mar 5 10:10:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 13581913 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AB72CC54798 for ; Tue, 5 Mar 2024 10:10:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 01EAD6B0095; Tue, 5 Mar 2024 05:10:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EC6146B009A; Tue, 5 Mar 2024 05:10:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C03356B0096; Tue, 5 Mar 2024 05:10:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 9B96E6B0099 for ; Tue, 5 Mar 2024 05:10:35 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 6471B160CA8 for ; Tue, 5 Mar 2024 10:10:35 +0000 (UTC) X-FDA: 81862566030.07.93AF800 Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) by imf09.hostedemail.com (Postfix) with ESMTP id 70477140008 for ; Tue, 5 Mar 2024 10:10:33 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=U7LDummB; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf09.hostedemail.com: domain of keescook@chromium.org designates 209.85.210.174 as permitted sender) smtp.mailfrom=keescook@chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709633433; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=W6tSmWQUz9mHEs7KKdtWRSqXpZBjknnUveJhwT+v2FU=; b=lFK59U2oYv4hGan1UcJWqMWKV/jW/2FlcjlmwWjPsDtX7W32UOdXCMj+IV60AyQ7QoEgOY JgiZaUDCpAgjaYvNVFZL9BT9cObDan/I1rOeI4zqDVi3yqAbXmu4yB/+N3aEuTgv8oVlss uTyp764RW9t1r6Z4Rj3gRIuzp4pMFo4= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=U7LDummB; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf09.hostedemail.com: domain of keescook@chromium.org designates 209.85.210.174 as permitted sender) smtp.mailfrom=keescook@chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709633433; a=rsa-sha256; cv=none; b=0rEquFoiBY1vGhnQwlE2WAoAag4lZ0+kkys/hOiYaR4SvbkwL0XaKvZ4eDUVtvFOcWZVIr L28/giOIOhkF+U0auU3Dss+uXtYyundbAPHz9LonkjA53MYzSS3tgwlwRWXodOxufBgZjb ssfq4c95uTq4F9ccjue4HthCcTpLLh4= Received: by mail-pf1-f174.google.com with SMTP id d2e1a72fcca58-6e5dddd3b95so2349453b3a.1 for ; Tue, 05 Mar 2024 02:10:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1709633432; x=1710238232; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=W6tSmWQUz9mHEs7KKdtWRSqXpZBjknnUveJhwT+v2FU=; b=U7LDummBLlnsabgDIUYq/Ksj7IOxE2Fo1ebHUuLZmryeeGpVzWgwtOey24kVqdP4HS UJ1thtbxUuRYzIb9GjrtSTvF8cFwDZRxQbUjm7umjKuRaLWpJMO8Y+kE90J6GDY2uZCr qdCNTT/Fsy+AZwR7D2fqAWEiTF7M246A/vd8c= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709633432; x=1710238232; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=W6tSmWQUz9mHEs7KKdtWRSqXpZBjknnUveJhwT+v2FU=; b=FP0YEh6I9W7wUQqzR4wE0X0IKJLDcMub9RCRm1TuqICVBtZEUXgh2B8bLghs7+ld0a O3gcJWX5yAL7DHQO7VQHb65Hr6dMjj/k0iRQ8apE/xn00iqzaSGY1ie7plwlrkPUleWa XHjbBS2ckcihhIoEZwm1BtBRGpEieFRnSlL0+dIoFkb8R/gdmEvdAqfJC79MGLxko9uf YzEz/2iT0uu5BDjcpp2TpqiQT5JJfaHFish9oEwMT89ASlAm8cecHoNpmiN6eiHEJr3v e7zrrI7MLqCajgzfJX1/96MNnmZvAY+dSCDPNWc7ah0mEAJmKcWmXresyrK7ZVW18rAI uniA== X-Forwarded-Encrypted: i=1; AJvYcCVruBJ4fH6Wl4HcShZvf5EMvnWLRsAAn/F5u2d8wkq8SsCP5Z5wva4Uuv2Pu6DPiHmEghM70nbhx2wqcwWl5XPtU5g= X-Gm-Message-State: AOJu0Yyh5gY1gJHllbxxKtJ2QBcqBv/9+iW8v9GyWWZ/pnlXCuC5NVQ+ Lj2GqexbTvXiEjZ7Pp3ZEfGoGjPYe1eU6LuQ2nSR7WIvWrvvH8KPRrvZAMHVOg== X-Google-Smtp-Source: AGHT+IGioGxnLf58krzooyXRSux/Zxx7hWwNEvHeCXchceFMByRkqxZ+PaMIxnsL+kY8mFgOqRxI3w== X-Received: by 2002:a05:6a20:729a:b0:1a1:3b27:4f3f with SMTP id o26-20020a056a20729a00b001a13b274f3fmr1585251pzk.2.1709633432471; Tue, 05 Mar 2024 02:10:32 -0800 (PST) Received: from www.outflux.net ([198.0.35.241]) by smtp.gmail.com with ESMTPSA id m7-20020a62f207000000b006e580678dfbsm8485054pfh.193.2024.03.05.02.10.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Mar 2024 02:10:30 -0800 (PST) From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, "GONG, Ruiqi" , Xiu Jianfeng , Suren Baghdasaryan , Kent Overstreet , Jann Horn , Matteo Rizzo , linux-kernel@vger.kernel.org, linux-hardening@vger.kernel.org Subject: [PATCH v2 6/9] slub: Introduce kmem_buckets_alloc_track_caller() Date: Tue, 5 Mar 2024 02:10:22 -0800 Message-Id: <20240305101026.694758-6-keescook@chromium.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240305100933.it.923-kees@kernel.org> References: <20240305100933.it.923-kees@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2914; i=keescook@chromium.org; h=from:subject; bh=5+bllf7wQXoQ1iiymcQxAaVJUnLpEaTtP4WhLyfCGPk=; b=owEBbQKS/ZANAwAKAYly9N/cbcAmAcsmYgBl5u+Q9Fez113NbGGVqjvVTslyLAH8TI+6Wb4a3 UHTy9ESMGeJAjMEAAEKAB0WIQSlw/aPIp3WD3I+bhOJcvTf3G3AJgUCZebvkAAKCRCJcvTf3G3A Jum0D/9tfCyWy09BzH1L9P4M66z0u2ctSfRJPeQ1x4bgaEnW38mkbjG41ME6e+R5Zj0gpmBgZ+L BVNaHnefEXko8smuTRzZ4FGGksACNwKf0+N86253pNgtssxWUfgfa3aQ1EA8XEBCMceHWGnr84H h34x9bAbwoUCp96M4lvRDs8cKxVlleF0jjCvDpvMT3T1ZAyvoPvNHYpvj6mth+oFIZTSkt3Wz+X 919IaLbl/IDuiJVd9rtpALExUWOh+KXL6FuuwYZHWN+SaSa712xlgG/5pLuJ+2l8Ub1IJiEQVWp r/4azSgELrhUJFvtIbkEWWKreqYhRzNs/rKZUmxW7K1gGKiGdDk4ZDqhqhclVbw3MMBNLZbLerx rSCyI/khhcdBGZYwjg6IFEY/vpFVkrL44xPf13VdK07PEsrsff7tXBBEX59HFI2KvQG5OsvWVhP aHHbbGxMkh8n+z9aHH4WpY49kUBjUPRrASB5fIz5FsMzXVgyemQHiUSYTVdkJ3aT34AdKNF6pks Immb98LUo/rCvKgXzxq/u49qS0zQkVpV4BWtD/SwK4MuhHIOU8Rm9ytXz8ZSxeZ0v1pYc89qObI o3aW/HpIQI7IY2DuYh0+o+TW5k0hDDZa00o0vGiCTiQIKa61gmW3pYCwmM5FPO2U0gagUkJ30W6 V9QFinQCp7WdukA== X-Developer-Key: i=keescook@chromium.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 70477140008 X-Stat-Signature: rdh87f9ogkbemeun1ycbez5wiz5r3o4e X-HE-Tag: 1709633433-48609 X-HE-Meta: U2FsdGVkX1/RwxIst/cEyXmtc6HlC5m8NfYNhCiXNdAredDmPNq9A4oJa7BJi5AiI7UcZbdQZX2G1hSq4xj39OVfhym2c6AfsQQjdOPdNdaN3dz+szZkfTNx1q97KYBdn5sy59Sk2RaioM35/CDrP+Xgzt1YbNpEx3gLSRG5CdjxUP6/yRo9IV6GXtU8BMDBa96+E7yAduL5zPtz87xDW4DO3eQD7xcyd/JYxwbtofj0Hh5pbMuhLAEBzG6pn57d7S4brQUoWqPtsqg/ffHOkzXzosP+chTSC67lZhe6z8w61cqRIJFFCImqsM2/nmX7R72l7iQ6tCXXKvLdCNkl/MRg1iuA65/MDF2D1XDAKg8kutxKum/DsJy8EYqArS+d+Fj319eog2liH6L1rPHF+wwjI4zDaCWaOa5w2yvpmAvIszYDVQRQbtXOR3Hf6Mvn+pVY0qpoQ05tumWKnLnuC/v6rsgahVXjPbSf2TFG8cUIQFl1oyiLIKeBRRsH9RI/jiMoBSG1IYef7n75h1ovtcu+bNgpvL/23IrkvLezpj/G8uTqhKW7lEt6b9P+BSFl2W/2TZbBGB9+5rc2M6GfjfvaIcqGcgVt6ZRUDJAqXEtsKRlhlqKQ7FWRqF8VGlck2aQll1Jgfggo/gIhvcZdpsnCIxYyo1TYZ87WlN4EdcqJnazkJe2roaEAmig/1yiqetjD/GMe0H1Pm7ebkQOKHe8cSciovt0W9y2VyFkge3XT6P/eExioL9sTwO3CC39CgC1+QNe1GkscaSRlKu6e8ROc5w1OVFjTB65j1VB63u7GBSQ03DybKrj+rNn5AVNPSwkDRxp29XFyZQgnWYf2mWypxcILoiYX76tymuaz16uZknRyk2Kxz8dBHNaBs7xuwd8DFFBDMYuGyLADcaaNkXmyXInNOcS+huJefuy/aJmqgvHz3cXh6z/09sR/XrCIYzF/mZwHrOCCXHmq5se KTgKM4a2 LGpsTkQRJOUOkZXbBBwMQPtT6GZcje+aONNRvtNXqvnDWkAuev4yODWmG4nv24ycVBpHwzU2wf5NM+ZDgvomK6sNghyQFsuUZuXh9YS6Kure40axqUviQp0s4/SC7ATNyndJmsbZ7A5wvhBq6t8p2p16IKox9RK+mDXmDOXWbQdu7nDJYEmQ3Uju2krgsP53hHLGqnbIUCtdVbaUggzD7RGPTerfoALhwZrn1xRsWH2nNlTK7yRCMZLhkJxmkBgLoIxZgUnEVWwlH2pPArBiQtYvP6CJNotYoW3trxpAAJd1WkqWCzaSm9XuwmuCcCuwBRCzM5B3+B68shSM6yLliZYbLYUYLuFHeZy+biNH1NHa+Xbi7OWndMkVBreYD3uNM9W2+zBCoS59FfKLgfmO2JibiKjwKRXjpza0HdIMPdT78rjEHcl0E41+QvSqUKNfbnp0FsI0dkPDsp+3cOA0ruOvxpxDRNSgmgAafTovg7rKDki360pgOQGwMqlUzrouWj64l/6yvZSngCUJr5Xs5VHu1KK28WpQYEsMjt33RIRExGXflH/93ZhjYFKmkHFpBYKJ37TaCbYW/ZnGcYRJgDweD0eVztoKqdn6XrMSO8Fg7GmLQqxOPv+6p9iF8MFdNIzIPOTmEK3Ez90sa2YFGU6fOWBO0Dz3xwixmJFJLP8Q2T3T3aSg46jtOqEJO++/rhDi25XHpe2ngIu/ST7JK7VOX6zTMFuNOufuJoXQx1JWEPzhD4PqlNZ4W/i7agogWv8C2wNk2I3j1vztUPYdPW8dOMR36w15jPWktmEAxl71/30MsTcV29N/0rYb64yHkHVtbHOgvtdB0MKEb/qQS/9wWIReLSdaUNzpflFyh+k+S60s= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: For better capturing the caller details for allocation wrappers, introduce kmem_buckets_alloc_track_caller() by plumbing the buckets into the existing *_track_caller() interfaces. Signed-off-by: Kees Cook --- Cc: Vlastimil Babka Cc: Christoph Lameter Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Cc: Andrew Morton Cc: Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: linux-mm@kvack.org --- include/linux/slab.h | 11 +++++++---- mm/slub.c | 4 ++-- 2 files changed, 9 insertions(+), 6 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 08d248f9a1ba..7d84f875dcf4 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -606,6 +606,9 @@ void *kmem_buckets_alloc(kmem_buckets *b, size_t size, gfp_t flags) return __kmalloc_node(b, size, flags, NUMA_NO_NODE); } +#define kmem_buckets_alloc_track_caller(b, size, flags) \ + __kmalloc_node_track_caller(b, size, flags, NUMA_NO_NODE, _RET_IP_) + static __always_inline __alloc_size(1) void *kmalloc_node(size_t size, gfp_t flags, int node) { if (__builtin_constant_p(size) && size) { @@ -670,10 +673,10 @@ static inline __alloc_size(1, 2) void *kcalloc(size_t n, size_t size, gfp_t flag return kmalloc_array(n, size, flags | __GFP_ZERO); } -void *__kmalloc_node_track_caller(size_t size, gfp_t flags, int node, - unsigned long caller) __alloc_size(1); +void *__kmalloc_node_track_caller(kmem_buckets *b, size_t size, gfp_t flags, int node, + unsigned long caller) __alloc_size(2); #define kmalloc_node_track_caller(size, flags, node) \ - __kmalloc_node_track_caller(size, flags, node, \ + __kmalloc_node_track_caller(NULL, size, flags, node, \ _RET_IP_) /* @@ -685,7 +688,7 @@ void *__kmalloc_node_track_caller(size_t size, gfp_t flags, int node, * request comes from. */ #define kmalloc_track_caller(size, flags) \ - __kmalloc_node_track_caller(size, flags, \ + __kmalloc_node_track_caller(NULL, size, flags, \ NUMA_NO_NODE, _RET_IP_) static inline __alloc_size(1, 2) void *kmalloc_array_node(size_t n, size_t size, gfp_t flags, diff --git a/mm/slub.c b/mm/slub.c index 71220b4b1f79..ae54ec452a11 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3995,10 +3995,10 @@ void *__kmalloc(size_t size, gfp_t flags) } EXPORT_SYMBOL(__kmalloc); -void *__kmalloc_node_track_caller(size_t size, gfp_t flags, +void *__kmalloc_node_track_caller(kmem_buckets *b, size_t size, gfp_t flags, int node, unsigned long caller) { - return __do_kmalloc_node(NULL, size, flags, node, caller); + return __do_kmalloc_node(b, size, flags, node, caller); } EXPORT_SYMBOL(__kmalloc_node_track_caller); From patchwork Tue Mar 5 10:10:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 13581911 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6ACA7C54798 for ; Tue, 5 Mar 2024 10:10:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A05EE6B0093; Tue, 5 Mar 2024 05:10:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 970C36B0099; Tue, 5 Mar 2024 05:10:34 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 598006B0096; Tue, 5 Mar 2024 05:10:34 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 2FB946B0093 for ; Tue, 5 Mar 2024 05:10:34 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 011D3A0CEC for ; Tue, 5 Mar 2024 10:10:33 +0000 (UTC) X-FDA: 81862565988.16.B92B0A2 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) by imf06.hostedemail.com (Postfix) with ESMTP id 3147618000B for ; Tue, 5 Mar 2024 10:10:31 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=Ud80XEOv; spf=pass (imf06.hostedemail.com: domain of keescook@chromium.org designates 209.85.214.169 as permitted sender) smtp.mailfrom=keescook@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709633432; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=J4oJMKLdbK31ulnxzAH51pNhqUof8xsZThnS45kbbx0=; b=wQipVyfT4WcKGTpfoxF0imB2Pw0NWsC3Mk/ltQdLhosUNohKJbfTHdZz7CrEUhyrea0okY 0bzmQaZpq/Sa1RNxmolz7HBEdj4a/2bfoxbssTVD8WHCeJUnaGz7B8IroOqH8hNo2c/huG WyxNXknTUanIgmrsOIppLuylI3amk1w= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=Ud80XEOv; spf=pass (imf06.hostedemail.com: domain of keescook@chromium.org designates 209.85.214.169 as permitted sender) smtp.mailfrom=keescook@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709633432; a=rsa-sha256; cv=none; b=4YnzYsgfDtLRpTRBGFhAjfXv6t1E3wT0Z08s9ljtV7Ua6/n1iPDOOisFJB/ZfFBKXnfPRe wlFoT1pCB07rVfs27rm6muTblS+HwUu0M87qUbdWnCcp7Tv2Tw+Lys87be3fXeKBrPQGz2 UNg0Ygvfuk0h4VANdPzEEFZkBXx676E= Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-1dba177c596so34016505ad.0 for ; Tue, 05 Mar 2024 02:10:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1709633431; x=1710238231; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=J4oJMKLdbK31ulnxzAH51pNhqUof8xsZThnS45kbbx0=; b=Ud80XEOvdwmmO4UL17RU8yCA+ngx4kZ/ye2JSjW+7nFLqlGNCJCyaHgf3IObD2sovT F0PYron9OzMGENlx3WpqlDHQYvMDMN/XqtF1rzdaTbirQPMeCEaU0k7IV8fO0Jfhecvn QsC5XdSsbVFg6wmJjQ5zjzYP6oQrwxBWWGgOw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709633431; x=1710238231; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=J4oJMKLdbK31ulnxzAH51pNhqUof8xsZThnS45kbbx0=; b=N4Rzy0oIBt4d7My2QjBvIT7BR3jUbankJbdlKhRTzlsy6D6wsN8n87wwLuh8uMiXMi 1JerGg0s5ME3ligCjCzyrLA1QXCqXcQhFl7S9oqBEGNR+Zt/CKSnk3hcgcwTd3w28k7O L6mj0TdSL3ZAeB1w08WmBIU8RjuzcUcNcvqESvtLXTZN+q3KhNPU/M+0BqFof/uubg/x DkYkz762TOkpW/MRZcNMyTjf7yGLYF76iQL6SQl3Sjhcx6v5PGZ3Ob3Hn/odZV5cdYpH 0WrzZpIv2u2c0CS/gcqbB5M/R10r8i7cfHRRMJbRsr44jy0ndjxOM8wd67kT42KolC9j D2JA== X-Forwarded-Encrypted: i=1; AJvYcCUwLFjA0j4c73lwDWjnRWxO2TDX3Y/MkemxA5d4yhB5rsWMcOGUA7M6ysFvh5QPoAEjnQKJrvDYEejiq/5K+r/atNM= X-Gm-Message-State: AOJu0Yw1+qypEQFrCcOHmEPPOZnzOd9wxbtw9eJP0xBewR0HWdDX8AB+ xuBXNcZsCRebT18SAKL5t/vas0gnX1xvoSJEshYoL9ZAKtbApNLHvzfa9f3fnA== X-Google-Smtp-Source: AGHT+IHrjjh8uwOGfKisJHpLEa0Atyc/w3DC95jENvS8m/9tWBeTRxSog9SPn/yX99K3zySiGpAi+g== X-Received: by 2002:a17:902:c946:b0:1dc:f989:3116 with SMTP id i6-20020a170902c94600b001dcf9893116mr1689963pla.18.1709633431171; Tue, 05 Mar 2024 02:10:31 -0800 (PST) Received: from www.outflux.net ([198.0.35.241]) by smtp.gmail.com with ESMTPSA id b4-20020a170902d50400b001dcf93e90a0sm6282358plg.20.2024.03.05.02.10.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Mar 2024 02:10:30 -0800 (PST) From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, "GONG, Ruiqi" , Xiu Jianfeng , Suren Baghdasaryan , Kent Overstreet , Jann Horn , Matteo Rizzo , linux-kernel@vger.kernel.org, linux-hardening@vger.kernel.org Subject: [PATCH v2 7/9] slab: Introduce kmem_buckets_valloc() Date: Tue, 5 Mar 2024 02:10:23 -0800 Message-Id: <20240305101026.694758-7-keescook@chromium.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240305100933.it.923-kees@kernel.org> References: <20240305100933.it.923-kees@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1268; i=keescook@chromium.org; h=from:subject; bh=k3ROePG0wD7icc8xiWOerLraaHU6zkW2VMYF75ammOw=; b=owEBbQKS/ZANAwAKAYly9N/cbcAmAcsmYgBl5u+Q3E2uwV8NIpzb5dXDsQuDADvQD8oEdZqqt GBc07SpeUKJAjMEAAEKAB0WIQSlw/aPIp3WD3I+bhOJcvTf3G3AJgUCZebvkAAKCRCJcvTf3G3A JkIREACdPvVBaWa/0ViOnT81LdqxheGCGDtUUmVO8IhLpzHQI2rh/b5CfK7msJHDhECQIcRrSdi 9nheGlO4Ww0GdAzyGQiuG+uKl+8tPoMsEDmLN9mYMJzOFVR9qtmtiAj8CWKbMecilAJj7/jLqds iCJqrBzYT0DBMPzR+RZ9sy4+mqmsXB8/VLVTfQt56mISQNPQ4HxWJF2ILoadyqB9vPcefR9U1EU OzMaLq1/npProPdec6qADdEauw08Z36eJCQphHgRvJ82WAFVrMNxpshJ6EMEpBpK8K5rISvVsar aqXEgU6wRqJfn9CHNyLBKbNEsewc+MTNzTH6kW30tWM/d0EuaYIykGXqwZFz6ALQ8Drk6s3yo+u KN1Nb+BF9a2zqYfrY9DNDIRvNra8AzrHVlKP3RHbTjzJDcvZDyjoEkPmDR2fCR2H+Z0kdexLb2B kk584A9xUsXzzD2J+uVGtk6XiiRF9RMheuM13Hn5v4gt4o/evttZNtO2TvSMEgiYlV7G77+lk27 THKPCHQ33k+oMkMYU93Eun2edSp14I5OInVVbt0+av7+/C3NRYD3Li2IJNOs4cX4OpXAliq48bk jKoGMk0FoWujDweGgEaojZtYDuVb8Uyxqw3TBzo2P0/2LUFvkVfm1Ovlo8NLjCSYQ6x2M+f71mk 5bgLk5y9jX9jonw== X-Developer-Key: i=keescook@chromium.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 X-Rspamd-Queue-Id: 3147618000B X-Rspam-User: X-Stat-Signature: b816febeures9c5oc11npkp1weckez9o X-Rspamd-Server: rspam01 X-HE-Tag: 1709633431-834803 X-HE-Meta: U2FsdGVkX19U3ySY425vhmKN60TnlHZkqsiKNItPwCGdxPulRv5O4pF+dMcCG4zRZ6BgR2oxqifkyHLfygEvBOzvuDyN3Nf2yeK8fRN1Z2vrG4G1XlxJSSyCKiUEjdPBbdGQgJpbsAzKHo4Jf5Pk9L7Zt4wC4MKgP0axlJEYGMWMGU3JrGcK7lZcCaAHP79SMif/4s6Ao/5yS63EfMfXpQYegDMfqceq9ca5ViPB4Mzf1k67DvIjTEkjPEEIff7Vm6gAyp2Z56XYgyUMWdVCt+86G1RaXRU/BxKoG0wCXCc5Kt7dhyRnZ3Tnyu6HS46KZ0bycfe33LUK+iiRSMNFVykUTQY/lwULfHQ0FkRqPuNeEU8YjAVF9l5TJyHcVwvlxw8iGNqkI6N24tq8HyA8O9nc6vjlWmS4qEfAnR1KVvVhwBfMl3q0chJVzYl6ed8EniEdyhvDkkA1fEjzwqpV+5WGIduv1/yii7uMX5V204IO9vG3GoOjXSeuYInOJ8JnqvOsnhARM4d2cnMeijkbSgRgfOZgZFAbN9B/e9qOurUbeMsiC96m0InaPA9m7+sRfYfWDRRG6aIbSINX0oPKT8m9smmnCCRGvH4wHPXA6mkfeurUcUaHSRlkBpybZTXy2J+SHje+ixVOSySWMcS4pkidbzXZ/JkKfJAOslT0KsONb0LRkhPdrSs+6B8CfgbAxffZdeoRQEImeKpOglmZwVnPnFB+jRvblcowv0yINCF1jpFYHtN1EJ4/3+6CpkkKahBHcBpLW4m2XLpltn0+mZmdLeujwHJoz44HhSXIDxWMw6R1nEMW2ykRFP9u35ezTKnEaHBp8hJTgjAx6K7dt5LUElP+TGNYirRfMmwOlnzrjmDjphjhXhQ6sy970Gxi7ccNSZOOGxSvd051FTvIyvVTsjRD1n0sK2x6Ir1P49Bzwxtk91S5gkgVe5h4oX/7B58Ys6APU/zs62FVHDv w14TKYzZ MwJb7GcHq7NK7d3vsF4sslurpeXwUjsRAhvUznX/kTnRo88VsBd8M3FaJkTFIOlijfC9yJT9Ny8KjR1A22Lwf44mWoI7yX4CEnfwzOvsxKVs1ZOL9SY33hUWX2oSrci0bWTxUBhaZrcG7IZvrDvvfhKMv7QqRUPDBbvlB1ZNIP+V0KoM58V3RmWxsEJxJ0gPzJtraibpYyCQHS9VZFeOQPfxu86l3andghkyJRhxq4uGc2/c= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: For allocations that may need to fallback to vmalloc, add kmem_buckets_valloc(). Signed-off-by: Kees Cook --- Cc: Vlastimil Babka Cc: Christoph Lameter Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Cc: Andrew Morton Cc: Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: linux-mm@kvack.org --- include/linux/slab.h | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/include/linux/slab.h b/include/linux/slab.h index 7d84f875dcf4..0cf72861d5fa 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -740,6 +740,12 @@ static inline __alloc_size(1) void *kzalloc_node(size_t size, gfp_t flags, int n void * __alloc_size(2) __kvmalloc_node(kmem_buckets *b, size_t size, gfp_t flags, int node); +static __always_inline __alloc_size(2) +void *kmem_buckets_valloc(kmem_buckets *b, size_t size, gfp_t flags) +{ + return __kvmalloc_node(b, size, flags, NUMA_NO_NODE); +} + static inline __alloc_size(1) void *kvmalloc_node(size_t size, gfp_t flags, int node) { return __kvmalloc_node(NULL, size, flags, node); From patchwork Tue Mar 5 10:10:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 13581914 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1995DC54798 for ; Tue, 5 Mar 2024 10:10:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 74A7F6B009C; Tue, 5 Mar 2024 05:10:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6D3656B009D; Tue, 5 Mar 2024 05:10:38 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4B2EE6B009E; Tue, 5 Mar 2024 05:10:38 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 2C6A96B009C for ; Tue, 5 Mar 2024 05:10:38 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id EE3E280D75 for ; Tue, 5 Mar 2024 10:10:37 +0000 (UTC) X-FDA: 81862566114.16.964A3BA Received: from mail-pf1-f178.google.com (mail-pf1-f178.google.com [209.85.210.178]) by imf04.hostedemail.com (Postfix) with ESMTP id 2D97640006 for ; Tue, 5 Mar 2024 10:10:35 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=Cqf3Pp9R; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf04.hostedemail.com: domain of keescook@chromium.org designates 209.85.210.178 as permitted sender) smtp.mailfrom=keescook@chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709633436; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pnsMPssoThTQkcXoWA9Ekbi5Q1zZTU79wL6KH4AEkgQ=; b=SRPko4OJOxYt7jaB91RWinFZg05ic7kTfHh+BfC3UXQ70AvvfutLM88Vmqa/NUQhR9Cd4+ rEBVqadDcs/XEH/flMwNgT02l4QfM+umJqounjSYfphs6p+IN6RmxAw9+7jPJaLax5Ww6u W6kZbr2CeUqnteNFXoFgm6JGJplOZVk= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=Cqf3Pp9R; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf04.hostedemail.com: domain of keescook@chromium.org designates 209.85.210.178 as permitted sender) smtp.mailfrom=keescook@chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709633436; a=rsa-sha256; cv=none; b=JofTA2Tl6I5pWXvyIK4w3wICWxdgcY/rUwSVgglBfhBNCPxCHmhaSlUuQh5C9hqXoxKhCZ SPoxW++O0Pbhvxmbnn6xBV5JgC080DVN48uljdO6tXH4S8UjXYizbTUZDq4RwyHQT8lTzg rnEQ9vaNetuvc+mYZwGX33IhEGo11SI= Received: by mail-pf1-f178.google.com with SMTP id d2e1a72fcca58-6e6082eab17so2010080b3a.1 for ; Tue, 05 Mar 2024 02:10:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1709633435; x=1710238235; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pnsMPssoThTQkcXoWA9Ekbi5Q1zZTU79wL6KH4AEkgQ=; b=Cqf3Pp9RQvIG0dWjIE0nfsAQ/gOOh46CkU8L5uBeJmrZXdkcRTIZx4VUZl5NJS9Ikt Si1gzbvtPJd87vzpdbLN+SN5MCCzg+CuKObWbyBTTIPcr5E/Fy3Wzhi+gTEbzVoVNn4H 7CUnKk00DpY+KNvd497vlp6bWqukYCx84tIjc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709633435; x=1710238235; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pnsMPssoThTQkcXoWA9Ekbi5Q1zZTU79wL6KH4AEkgQ=; b=SOyWvrslJGGc8vNwUWuJ/i9fjoRo7Snm04g10IQLp8shMaorrnXEUKA89dCSB8a6p9 t1mVi1pdHhaX3h23Gc9oazmLWpUcl0HeYZBihYCpkNCXpk51UHzit/sv6X6m0jfgK1+f c2/DXnAW4zpWjye7xAXQwRH65FAfEg++y5TiXBvmBsvE8VGSESnZbL9tTQEYDBufi1Q9 vv/SlEeDf3dLWYQY94dj47T6YQqF5kWhsPtVLH5n1EQ7rry2Qb0cfGMvmAnl+LiNnYkl czp7WgiyxQzUhIkeTJGJUBxH9kt7RtxrGxrp11HBuTNyJOD1ia2rYmXwcgQUFkN8+oSg qd0Q== X-Forwarded-Encrypted: i=1; AJvYcCV46BwW2dmjtqW5H34rAk9XHiDaJd+2XyDLO6PHsraH9FC+DGe97KT4S6Uu2aAN99fTe4z+rYufwFhNWnTL6Aa0PXg= X-Gm-Message-State: AOJu0YzhdPZqvZXDJeqxuGfc++YRIM+Mh4FyJXzv7PZudt77BPALioGi 08KIWtjC/JR52xuro3ewtr76fa1BNidlsbIE5fZbLf04ge+2LPGXDxwmF0JCYg== X-Google-Smtp-Source: AGHT+IEiAWTgdgJYXNyV1wO/0LoKLC3IKAkw+TtluHY1zuMxHwA3vOiKJTRV4nyNTDEQlhJfiH3Lwg== X-Received: by 2002:a05:6a00:2354:b0:6e5:80a4:2ff2 with SMTP id j20-20020a056a00235400b006e580a42ff2mr12396375pfj.30.1709633435073; Tue, 05 Mar 2024 02:10:35 -0800 (PST) Received: from www.outflux.net ([198.0.35.241]) by smtp.gmail.com with ESMTPSA id n9-20020aa79849000000b006e61b0a4b17sm3625323pfq.185.2024.03.05.02.10.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Mar 2024 02:10:30 -0800 (PST) From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , "GONG, Ruiqi" , Xiu Jianfeng , Suren Baghdasaryan , Kent Overstreet , Jann Horn , Matteo Rizzo , Andrew Morton , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-hardening@vger.kernel.org Subject: [PATCH v2 8/9] ipc, msg: Use dedicated slab buckets for alloc_msg() Date: Tue, 5 Mar 2024 02:10:24 -0800 Message-Id: <20240305101026.694758-8-keescook@chromium.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240305100933.it.923-kees@kernel.org> References: <20240305100933.it.923-kees@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2182; i=keescook@chromium.org; h=from:subject; bh=Ltfy4agIWW8GFQ2wB/SGVqMVh7NAol3Xo3zP80oOSxA=; b=owEBbQKS/ZANAwAKAYly9N/cbcAmAcsmYgBl5u+QWH/R4wFO4/C60XHEU8ysAaQUzkDa/3+3d QWUIDhM692JAjMEAAEKAB0WIQSlw/aPIp3WD3I+bhOJcvTf3G3AJgUCZebvkAAKCRCJcvTf3G3A JrvaD/wNahnrlFTqZJolsZvbWW0JKOIUHvIZ9tu5KXbYjElEyGOPqLCj2ubk3S/Q6iA9219qfa6 pGqd/gb14P9uZyXmRoI9+3bS4VDQCYU2pVYoTkpLeg4mS3bLPisEIDg13VQAB8yGYFFzgx570th kBc2psi8XalZRDb+22nGBoKA0hP91pQ00G0kJzgtIobHvLf08qFlMvNbmsE5Pme/RYM37z+uKJo 6Ht69bJWQXxP3brIzWYPe9S0kO8dNubvglHpUetBciWXQNmWqzDhkprvcsUARwd7gUvgA4Zl2bU BvI0Fxk7BBq4ju+5KUIEligRbrLMIOn2dxUIdSfr3k4opmEl5ETlonQ+wt0savTeYPsbW/lgGwn v86/q0pgtJ/5Oxcd8pnWLe6dDPg5ZGQ00IiZ9h1x5iSLqXFJ1MFi/B+uFSaLnK0guFBVXHIUCmV I6mhgVKYMZALTQfYuz9ZZDYkV9pTJCysElPoennpSXdeUJ7Pc/edHh0770uIOALNa6rXB1b+K6i Yd1XOLuB2X7tDGZOo7NgNDNoxOX71alUQybWMW+kUfUFYB5XoVlC7XOD3Ax3ixm0n6sjPCVCuY/ t2ge5/Gjz6GsJY+hlaCoZ/t+yhtmsbD3m+Gu2eA5hteP/AAmSNJGiBGNGgQ09SV6+yyLUeBYps5 sysZ56s4nY6VilQ== X-Developer-Key: i=keescook@chromium.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 2D97640006 X-Stat-Signature: g38bejryeehmooujj4n7uc1rtd8pt64h X-Rspam-User: X-HE-Tag: 1709633435-928886 X-HE-Meta: U2FsdGVkX1+3c4+4c9XkxV/wtuxQF/HZRWEtWh3D8pdt+M7YLO3iD7qwlidvsLimoF3+mwyT+HJwclbyIrdsFSpc5Udcd7u6tuBG9yjVYbPygAnPQvBYbrCbX9LY2HdhEDxMwVZxmCNlWbHaVX+pghoHMepeb8O/3wqW6Cjc9T8o6Ec/gQZFz0NogeG4lJfi7ns4Mrhf8fRy3YxQCNGRdJdlnwa/zYB2v132krCU2Xs+/ZkeAZ0PPplOEeQGdQejtCmHP6EPrCoqSC8USpNhMT/rTFwiShm3COcuIa9LimVmS2iruJgz3vZeeCVKaX6ZJICstnJ1KQKnhmJZFGHQeHMuAVcIfd0/37dOI2WomgxxiNcf1syWL9D05Kr1cW1XHRlSgJDPNirp3rh0sQYcojjJ5qjgpGXb7pp9F0ydV9qYS13dCnmv5RoG/NM9evoQ6BIu8fbWGIGRXJsEM5y1yNQkAixr/xtpnSZC4DnwwE355QJOGvMAMXNj54IW4ccIQik7dAGqOi/mORmxkzj3Pikd0VnX9jyTPFSvU76Hr7JQuMUHDoRePeaoXt1DaTzlxeRjnAFlUqzVw5OTA8cObkFUoNuELRjKKOqUGNW5Iruq27yKCZ4r3fXh2jRHwvxU3tCeNzA1g/dT9GT1LrjGYCp47gSk7DsypSUdRN+TZLEg4AQnkzl/ONFWMbw8CItnxeCIzSecIt5N57H4b1CnMzud0hG2v3wNNp3Dki8t8ptMD8COXKRur/LLn13829ZKvwML1Pt7kI/pjFXiC8mCw5tRsDoZIo+t5bG0sb+JvP2KlpjMygQrNZRBLEAJPXCbwLvQt2uXK0BwlJgx8SpNft/cxGAX/c/mpmNnAVWyz7bzN3vTSOQbTMgZXGZ3hosNZzmGIKizqhIRQzqYUrRHELMV5SkEjFVv1PV9HogfuhFmYqky/AWSshZ2Rs3yhbbyXbu7YIZtt23sKRso4d4 auZS2T+m De6B3S6Q9WrNDI/GnQukND/FA+PPhlRB6E7WuI2N4EbNOkEuIh6QVk2rcNCon47TcLamcJBcIajfjSna+c6yW+0gXYP3E1gd4e4g1U46N8JFFLgTyhfadlp4DnVpwi3jyNFr3Wsx2uLOcQ8ZSNOAMbbY/PnY5/NSwoa6k8DnCSJHacLhy30NBBCZtRHnGsIPuWc33kLE2f6ZvssEAaec71y5X2ic6soQT8ftZLJxvPSlUP1k= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The msg subsystem is a common target for exploiting[1][2][3][4][5][6] use-after-free type confusion flaws in the kernel for both read and write primitives. Avoid having a user-controlled size cache share the global kmalloc allocator by using a separate set of kmalloc buckets. Link: https://blog.hacktivesecurity.com/index.php/2022/06/13/linux-kernel-exploit-development-1day-case-study/ [1] Link: https://hardenedvault.net/blog/2022-11-13-msg_msg-recon-mitigation-ved/ [2] Link: https://www.willsroot.io/2021/08/corctf-2021-fire-of-salvation-writeup.html [3] Link: https://a13xp0p0v.github.io/2021/02/09/CVE-2021-26708.html [4] Link: https://google.github.io/security-research/pocs/linux/cve-2021-22555/writeup.html [5] Link: https://zplin.me/papers/ELOISE.pdf [6] Link: https://syst3mfailure.io/wall-of-perdition/ Signed-off-by: Kees Cook --- Cc: "GONG, Ruiqi" Cc: Xiu Jianfeng Cc: Suren Baghdasaryan Cc: Kent Overstreet Cc: Jann Horn Cc: Matteo Rizzo --- ipc/msgutil.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/ipc/msgutil.c b/ipc/msgutil.c index d0a0e877cadd..f392f30a057a 100644 --- a/ipc/msgutil.c +++ b/ipc/msgutil.c @@ -42,6 +42,17 @@ struct msg_msgseg { #define DATALEN_MSG ((size_t)PAGE_SIZE-sizeof(struct msg_msg)) #define DATALEN_SEG ((size_t)PAGE_SIZE-sizeof(struct msg_msgseg)) +static kmem_buckets *msg_buckets __ro_after_init; + +static int __init init_msg_buckets(void) +{ + msg_buckets = kmem_buckets_create("msg_msg", 0, SLAB_ACCOUNT, + sizeof(struct msg_msg), + DATALEN_MSG, NULL); + + return 0; +} +subsys_initcall(init_msg_buckets); static struct msg_msg *alloc_msg(size_t len) { @@ -50,7 +61,7 @@ static struct msg_msg *alloc_msg(size_t len) size_t alen; alen = min(len, DATALEN_MSG); - msg = kmalloc(sizeof(*msg) + alen, GFP_KERNEL_ACCOUNT); + msg = kmem_buckets_alloc(msg_buckets, sizeof(*msg) + alen, GFP_KERNEL); if (msg == NULL) return NULL; From patchwork Tue Mar 5 10:10:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 13581915 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E74F3C54E41 for ; Tue, 5 Mar 2024 10:10:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3CC0D6B009D; Tue, 5 Mar 2024 05:10:39 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3534B6B009E; Tue, 5 Mar 2024 05:10:39 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 12F566B009F; Tue, 5 Mar 2024 05:10:39 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id E9DFF6B009D for ; Tue, 5 Mar 2024 05:10:38 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id C1E0BC0DF1 for ; Tue, 5 Mar 2024 10:10:38 +0000 (UTC) X-FDA: 81862566156.22.C3914D7 Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) by imf27.hostedemail.com (Postfix) with ESMTP id CC01740006 for ; Tue, 5 Mar 2024 10:10:34 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=kh7rnY4y; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf27.hostedemail.com: domain of keescook@chromium.org designates 209.85.214.174 as permitted sender) smtp.mailfrom=keescook@chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709633434; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Jdg+vWQ/glPg3PLzwoChyf7fue9eFP8PVvp8lsS5Lok=; b=zGwmclDV/VNXOJIzLj56rhgUJ+Q52xfCdF5c5j8mWRsS1dL85N91XdbjFYEIk4SCdA5I/1 8eYPf+JMNmPsJLvOMO1cHqGX70ARkp/Kq4IZi4tDIhCD+Fh8asL3ysd7cdtuiyXaTiMZN0 vr008sSIWk6al0cSicv1bXRbV3wegUc= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=kh7rnY4y; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf27.hostedemail.com: domain of keescook@chromium.org designates 209.85.214.174 as permitted sender) smtp.mailfrom=keescook@chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709633434; a=rsa-sha256; cv=none; b=ejhurCdcms+6GwBqWhVrzbYVN0eaBuxW3RN5970QfXDWBDn+KJA2l3MazHKvkPPzckw93M GKD0BVuNhaoeTOZphS2FFS0HCDqtx27CiZy0Y/Z2lU1ZMJv98h99P+sY0H21GeX7u5vlyR mzHs5fzGDPVlien1fMCLURgBhay57MU= Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-1dd2dca2007so2733155ad.2 for ; Tue, 05 Mar 2024 02:10:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1709633434; x=1710238234; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Jdg+vWQ/glPg3PLzwoChyf7fue9eFP8PVvp8lsS5Lok=; b=kh7rnY4yXM4pOYviwG+mSijHRF8brFjjjZPj/7dh+0JUQ8HJAB2RZMx5d7jQp4BdIJ w24tadbwY8lQsi5b1X7Dzrzb71czTNv7/H6pBe6hwFHrjEcIo1J3H6cBgomjOZnMr0ZV deV9yvS0apUhxLOYQMlEAOQMUyxAllE5Htz6M= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709633434; x=1710238234; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Jdg+vWQ/glPg3PLzwoChyf7fue9eFP8PVvp8lsS5Lok=; b=VBcah4cg+xbi6VzYPTJvulL7+DXIRO55zOKmlbTFZ874pTq9aYcaSn0Va6Pt/bKiuC zxuQG3th9Cc1V2K/Zc3mhPm/t94TIRn+msBch7EW4/gnmH1Rl+fJEbtz/5NEZGGMkGX1 tcvcJfEmvwuDlJ/U36MGcwBcssciHY5QNJgzctDqLJlMzh6xJe+O5DyGbF42BiJopx53 0jbo0pMewRRo7GmpAoQWTA//tQP+x5TiyJcaEqjNyzCIvBNx/rUWg2FL9REEjs/ehpOr Dw5ERMuPTjxrQzi2cHkIEp94TY0Z0pCTxOIN7b9Hc+1yEG/Oo9/KPHQ0El5S/CW63RSx 3wOg== X-Forwarded-Encrypted: i=1; AJvYcCVBXTfr3RXlwfL1DkTJCEhA4j9DUZDkacn9tfz5E5mx3Js6gogMgUvlhvMWtow07b+Xeep2LFOYsu0z/g/teUEt+So= X-Gm-Message-State: AOJu0Yw3Dblc9ueFsJgwCK1WNnu9qKKrgGPTw0lYunKfdM0LCLzeQX9v dqzrSGE/05JwLJGPw3Gn3FNVa/kGeiesl9TTgvrV3SUfJDrsgeWYuztRd7ez7A== X-Google-Smtp-Source: AGHT+IHYCn9DBg9xzTvpQ8G0/N3dK13ZQWLK1qCcLYJXKOoBEAWGORs21YgWZoWg8YTIQaCZeo4VAA== X-Received: by 2002:a17:903:2287:b0:1dd:7da:e0a9 with SMTP id b7-20020a170903228700b001dd07dae0a9mr1791845plh.69.1709633433734; Tue, 05 Mar 2024 02:10:33 -0800 (PST) Received: from www.outflux.net ([198.0.35.241]) by smtp.gmail.com with ESMTPSA id q17-20020a170902e31100b001dbcf653017sm10134281plc.289.2024.03.05.02.10.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Mar 2024 02:10:30 -0800 (PST) From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , Andrew Morton , "GONG, Ruiqi" , Xiu Jianfeng , Suren Baghdasaryan , Kent Overstreet , Jann Horn , Matteo Rizzo , linux-mm@kvack.org, Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-kernel@vger.kernel.org, linux-hardening@vger.kernel.org Subject: [PATCH v2 9/9] mm/util: Use dedicated slab buckets for memdup_user() Date: Tue, 5 Mar 2024 02:10:25 -0800 Message-Id: <20240305101026.694758-9-keescook@chromium.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240305100933.it.923-kees@kernel.org> References: <20240305100933.it.923-kees@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=3141; i=keescook@chromium.org; h=from:subject; bh=6qDL+BboHaqcTpZSUCl+S9ocvneIfXXeMs3ajiQ2O/Q=; b=owEBbQKS/ZANAwAKAYly9N/cbcAmAcsmYgBl5u+Ruvag2LjJsl5BflMWP3F10S0bpaicfQjrP em4Flv551eJAjMEAAEKAB0WIQSlw/aPIp3WD3I+bhOJcvTf3G3AJgUCZebvkQAKCRCJcvTf3G3A Jh5rD/wJ5mDNRMwSTsAR06j+XsdZ1LXMSaSe8WeQSfww+7fSDo2bwPbB2YuP2imZfi2Z0ALaDNc QjrOdKcPLNAptwDgTvG8rgq8Ads/uaBkD6rP1GpQd9XF7ZP+IN5Tj/Ust9Zx+YZOh1j757Tk9f4 /2I7O+56tM7rqcJYTzDVwwN3lx5z9LnJAIv9cA2fc5WZw4zdqRbzRZ2x2/jTvOictVZIOz75vdw eHxukjUd1puppQSWDfMARwCUPIILs8IjoAaR8UZAfPVGp0pxZe2GBSsLG2wB+yvlBFR58gg+qkr PPA4dy4re0Tig2xdNDMKxFzPI2MGhDuckv7+bdXYa1y03Qk/hE9om+kdYryhKz3AJ7c2FU6lReH mTQ/EPfpM/DP3hVk1yOvB4YQVT+SsRzeBQyHjE7vMel9E0zPFPRlsHvTQYQfzF3xQMEWSjWW4GG 5VQ0LGp6u8ey7adQvju72Jq5NXyiMZbaTebl1RgLT0AX7iek73VAJKe9Vp1MeHbuW3KzheQjEhv V+GYAiXu51x1VsKLuN69+yg2ofManoTrfcNE9k/KiZywKTTRvzJ7GlsihkF1zScLCrOfnIJLfWn mSg61FpR2fsWqar318TaCYm5iTXskabguFfuho/W7iHJo+XHIJRToiRH7s34zAmcK3ETFBpzTUP t3ctwbz/OBT4vHA== X-Developer-Key: i=keescook@chromium.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 X-Rspamd-Queue-Id: CC01740006 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: 5d5iwb5x1sixcmmd6fux44ohkrao4byk X-HE-Tag: 1709633434-981152 X-HE-Meta: U2FsdGVkX1/RJ4A22jkc02RsLZeZXqf63ZzImDTDLVQ7cQMayCp+2zD42r+SpJBiJFt5u4lU7eAom6g6lcEVdVEh4ua+7Y2asgDDB6yDJoFKh+yHG1ry5qU+sshOmn8+bEoT5OofA765VoDjG5vg/cuDwDtO3lIqK8QK0eUyX3+7AyAJf+XixzVGW8EcNt5xZwjyI0uz0oYPLh/4RBWURbTpSkuzQIzPWj9L7zY+6o7/INTQD/2NvW1t5dw2sjpYeveCG5Dxii480xtwlrmLM9BcD5KYRITUV4NWa47YkmDWtRsJXhBhEi8a/Dftk8VK6YE8TtXn7Qu6dXX72nLP8Eqz5pXoxiHYTn0T7LE3V3PyCaRUrGp18dG3RscdKHs2anw77bjgJj8Xs1xkFBzrHelgwMl7xzcX3NZGXauUjzFMibzan/zs45feXPectlkBLt6QZ5c+PuboZy5axz7jiV8jwA6Av1r+CpZAlRZpMkSMEkXOC7HVtNXN1LNY85p1CbSPPF73fZkBYwiLvETzPLerSOAeU4zq2uPTNnv5VbBwsSTHyUES6B3l9rq5qW1vqfN/wKZWv4xUeQKLE9NAX9gA81DYy0eBNioMpWt/vu/2/McsMGfJY93v26jHtjgXsBDgmTIgt0SWuvCLsSLRgX+hxWBVA1IIIoVGrlo/f4Lcbw+b0H/ISaKeQd4jtPs4XWBmJ0Inxk6MscVGVOIZQLUGD8Igrp+GNJzSuznrT+D/jACXa5ENF7lkjlWqFmbiwI2HktuAf5YmXI9J6wfmnkLa12K8hHqnzGJckWVhfqmYfoxJy9nAW33sxUpMR3p2Y5KyTCNfKGlIxCiRDjnwptDgPNC9oXtrduhpzSCtcLWKghKQJhXcC9naVbVGbCKU73B4YRUusurVRStVrILie4Z+TfoMxUlSOR3CTETM1WWvuGo8PcQ0mP2OJZiKY8XrGdtahOWtxO09QlACf5B gSGaBntz xzRkV7XuwCcPsKZQlaRll5R6lU3ju2mIfv0LI3syOUdpTRlRcT2tYTEbHLYr2L3vCIvLfsGH1UmHJmUfK2J5O1480BFLRCW9BeHKnp7htQpmq7CkIMpVOu7KyMYQ+Z3Kpv6J0qdLZuQe8E03xRfhGwfGdQW+tBvNx4Auq9XNnozc6bZTWSjN1ZjJAE3leMnoFYmRVKLz0M7QCXeCdM4/9xFrj1O4+VXj/Tzf5lZIjFU2V6u8rD3WR9JqR7jUeo7yugX3akT44mRLx2f2JlXPuWYCtxxIgJGsU7xd1pJVoQGZhUsLQ6L3zXhj83uK2+OWKZd3O00R00SRWEOF4Xzl1s6btjElBOggJmBarI9RfB66+4AhjU4RvhDk3Vj8I6A/G4taf1ucBrdEbHqtqSfX/LGWU1GbudIwK28SETeKznKvAjz9B4GODWi8NG/QNKOpnm5nY1/54g+uWG8TBCGCv0J8cuP6giHXyCSXaxuruI7bXp4i5+0EpNZmB1+xUYUDnfpcjTMHbIKH7ZBqPDxLzWFcWlNLqSeopVZi4W52+K1cYI57d2oDUXmuuvMreL/RHSUbQ7CPTx3aBaDUEXbL343GWugaFQa3nJIO0VrlkXJhHr7200O7pkxrySVFynpYqpyeQ977KfdNLTSnEvqXWK9rr8SktP6kndkPh99uA2WxSW4XPEVzpkUYRfqVJI5HmWIoq3+FTrmmSU12FvSTz8Nk4jI9hjU6oaPzXx8BTnNveYgYAWw0OvJkPG8uU+4K9M4EmZqr92RXI3h+SxYtNwZiQAlHyDMJPi10uF8EuycUZSwinhLJvKN9J0clDr0x/VoZjESunmZRXBNtTDgT0WypAE3LGAMimnOf+B7WOoEO99B7YOaERNI/qvkPCBbDYVDuWdD3ZMu38O7IT2wu8u0njPSKlpHp8ygIHVEj/bKM6TvFpRhtNvqTFzvC2zjIQqqppbGWwoF2DGiak/aYFlYqdIowR bGxSGJmq KEYm7Y+/+vw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Both memdup_user() and vmemdup_user() handle allocations that are regularly used for exploiting use-after-free type confusion flaws in the kernel (e.g. prctl() PR_SET_VMA_ANON_NAME[1] and setxattr[2][3][4] respectively). Since both are designed for contents coming from userspace, it allows for userspace-controlled allocation sizes. Use a dedicated set of kmalloc buckets so these allocations do not share caches with the global kmalloc buckets. After a fresh boot under Ubuntu 23.10, we can see the caches are already in active use: # grep ^memdup /proc/slabinfo memdup_user-8k 4 4 8192 4 8 : ... memdup_user-4k 8 8 4096 8 8 : ... memdup_user-2k 16 16 2048 16 8 : ... memdup_user-1k 0 0 1024 16 4 : ... memdup_user-512 0 0 512 16 2 : ... memdup_user-256 0 0 256 16 1 : ... memdup_user-128 0 0 128 32 1 : ... memdup_user-64 256 256 64 64 1 : ... memdup_user-32 512 512 32 128 1 : ... memdup_user-16 1024 1024 16 256 1 : ... memdup_user-8 2048 2048 8 512 1 : ... memdup_user-192 0 0 192 21 1 : ... memdup_user-96 168 168 96 42 1 : ... Link: https://starlabs.sg/blog/2023/07-prctl-anon_vma_name-an-amusing-heap-spray/ [1] Link: https://duasynt.com/blog/linux-kernel-heap-spray [2] Link: https://etenal.me/archives/1336 [3] Link: https://github.com/a13xp0p0v/kernel-hack-drill/blob/master/drill_exploit_uaf.c [4] Signed-off-by: Kees Cook --- Cc: Andrew Morton Cc: "GONG, Ruiqi" Cc: Xiu Jianfeng Cc: Suren Baghdasaryan Cc: Kent Overstreet Cc: Jann Horn Cc: Matteo Rizzo Cc: linux-mm@kvack.org --- mm/util.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/mm/util.c b/mm/util.c index 02c895b87a28..25b9122022a7 100644 --- a/mm/util.c +++ b/mm/util.c @@ -181,6 +181,16 @@ char *kmemdup_nul(const char *s, size_t len, gfp_t gfp) } EXPORT_SYMBOL(kmemdup_nul); +static kmem_buckets *user_buckets __ro_after_init; + +static int __init init_user_buckets(void) +{ + user_buckets = kmem_buckets_create("memdup_user", 0, 0, 0, INT_MAX, NULL); + + return 0; +} +subsys_initcall(init_user_buckets); + /** * memdup_user - duplicate memory region from user space * @@ -194,7 +204,7 @@ void *memdup_user(const void __user *src, size_t len) { void *p; - p = kmalloc_track_caller(len, GFP_USER | __GFP_NOWARN); + p = kmem_buckets_alloc_track_caller(user_buckets, len, GFP_USER | __GFP_NOWARN); if (!p) return ERR_PTR(-ENOMEM); @@ -220,7 +230,7 @@ void *vmemdup_user(const void __user *src, size_t len) { void *p; - p = kvmalloc(len, GFP_USER); + p = kmem_buckets_valloc(user_buckets, len, GFP_USER); if (!p) return ERR_PTR(-ENOMEM);