From patchwork Wed Apr 24 21:40:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 13642526 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 081C8C4345F for ; Wed, 24 Apr 2024 21:41:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 89A7A6B0313; Wed, 24 Apr 2024 17:41:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 84AD18D0031; Wed, 24 Apr 2024 17:41:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6ED1A6B0315; Wed, 24 Apr 2024 17:41:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 527AE6B0313 for ; Wed, 24 Apr 2024 17:41:10 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id C69FF41101 for ; Wed, 24 Apr 2024 21:41:09 +0000 (UTC) X-FDA: 82045746258.05.7139797 Received: from mail-ot1-f50.google.com (mail-ot1-f50.google.com [209.85.210.50]) by imf28.hostedemail.com (Postfix) with ESMTP id CEC95C0024 for ; Wed, 24 Apr 2024 21:41:07 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=kbyKnWGp; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf28.hostedemail.com: domain of keescook@chromium.org designates 209.85.210.50 as permitted sender) smtp.mailfrom=keescook@chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713994867; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6DE0gOuM6fqjcPRbynz9LQpOkql4Ebi3pYsD8G1TJgY=; b=T4wZf3TBmM6r0WKLB3wo8cVxuUOd1Wtm+FpbEHLFAijLX0jvYRtbZnncLpbCTzT0J7zkdL D41PyEBM9BNX3VP2D2BwqRxELymAfZ9zh53vyny4qR1D6jKuFlJVE9MQWk5LNdC3UlC66G dltHQcboQ0NjGNaBvFtd2pvum6gJ5pM= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=kbyKnWGp; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf28.hostedemail.com: domain of keescook@chromium.org designates 209.85.210.50 as permitted sender) smtp.mailfrom=keescook@chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713994867; a=rsa-sha256; cv=none; b=d1Of+nqxoAD7mbZqjjZ18tbl3satJOaWc9QbYbyuYlA+14qC87eeM5zIZT596/gKeSBYxA ThT1CixF/Vtt59xu3kHXRhIzVwf+1UCIcNV0EUjaAtkVvBaVO6nOTGnlnYjBADVG+nuu4H rga8TDsQEgn7oaKjTYPp1FHfxgJvvw8= Received: by mail-ot1-f50.google.com with SMTP id 46e09a7af769-6eb7ee5a776so226608a34.3 for ; Wed, 24 Apr 2024 14:41:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1713994867; x=1714599667; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6DE0gOuM6fqjcPRbynz9LQpOkql4Ebi3pYsD8G1TJgY=; b=kbyKnWGp260lD0HQBaD6v3xPRXbTPeJZGtNHO2lx37i8m/GvhjYkSjXW/P+YmUe8/c onH4ltldAU0H058PxhKLhIp3p3uDJrKLXIFQqoIQZuQXX6NxYgbermHg+aG3NNzW5s6V Q2U7HqLMYkbGAVqXVqbudR06D3i3bVJQFKLeI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713994867; x=1714599667; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6DE0gOuM6fqjcPRbynz9LQpOkql4Ebi3pYsD8G1TJgY=; b=hMWmuA7pXZB9mlap6aJWz5miboFiWIXP0KICdeEoIkteIRc2fraaVY16VnVq5/dzIB hYWTc4Xd10vTgVRO5EXA+JRP4sNuaspKBOiJ19Pwa5IAHnzEfcJFNU0IjJpRMQ4+9mrS wBThWNFcp7bffvH3OeAi3YChoHgP67bCduf8TglYtdrIJijTJTAksuUoRaa2nl/dyiqu /fs0RKsgLUA7VIcmXr2ChMbWTtnDfhZef1vXaLfz4DTUtf6ISCOYxLaee7ItkewxbK4t ZAi02XjmpmpX039t91EcQnohwCVuq2EMVIDC9hKq76Qb6uQmixfGybsfYfbYFWihXvOE RzcQ== X-Forwarded-Encrypted: i=1; AJvYcCXSG29uelmRMLaW/W9sjqejl++qep+/JmV+nwCMqPgWst3nrNRuy/n3V8xsCSq1toydO/6CFE+bNnepyGwCGxMau+0= X-Gm-Message-State: AOJu0YwYVJPZRR8F3EL40sacIIRKQ2uA8G2M1e9lBde+8DDQl1SVck5d CNn6XP/EEOuoNRy7NH9q1sH1/CMDFuW5/WZk2IkUCF9jmjXXLQ21hRxjAFHGUg== X-Google-Smtp-Source: AGHT+IEqM/ZyN3r0s9cglQZ0+oH+dar1+xY2j0+uac8cA/8FmytsL1pSqxY54sZztW/PC55nIBocHg== X-Received: by 2002:a05:6830:1d61:b0:6e6:b125:b8d0 with SMTP id l1-20020a0568301d6100b006e6b125b8d0mr4471407oti.13.1713994866613; Wed, 24 Apr 2024 14:41:06 -0700 (PDT) Received: from www.outflux.net ([198.0.35.241]) by smtp.gmail.com with ESMTPSA id i193-20020a636dca000000b005f7ff083182sm9774009pgc.36.2024.04.24.14.41.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 24 Apr 2024 14:41:05 -0700 (PDT) From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, "GONG, Ruiqi" , Xiu Jianfeng , Suren Baghdasaryan , Kent Overstreet , Jann Horn , Matteo Rizzo , Thomas Graf , Herbert Xu , julien.voisin@dustri.org, linux-kernel@vger.kernel.org, linux-hardening@vger.kernel.org Subject: [PATCH v3 1/6] mm/slab: Introduce kmem_buckets typedef Date: Wed, 24 Apr 2024 14:40:58 -0700 Message-Id: <20240424214104.3248214-1-keescook@chromium.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240424213019.make.366-kees@kernel.org> References: <20240424213019.make.366-kees@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1775; i=keescook@chromium.org; h=from:subject; bh=dMZ82hcwMjPtLdK0cTsVIA5XzWAQXcfcJoTMZY0cUd4=; b=owEBbQKS/ZANAwAKAYly9N/cbcAmAcsmYgBmKXxu1YqNqwqk+P9UKDsX65CurgWZ4S/pDrKwg o4x/WCX9JeJAjMEAAEKAB0WIQSlw/aPIp3WD3I+bhOJcvTf3G3AJgUCZil8bgAKCRCJcvTf3G3A JgHJD/47Q2MOcnzIviSgiD9VouiFmZpwzwVRRYkpLKsHQDoOvqMkZvk/eKBcgJq7gI4b4L0nlze 3A4oC+r7H2g5vpgNw2p9z0EvkqMxob05GHM58weuoSuAoQlXoGaw8CGnTLUzMStUBs7/mjNJP7N jOKumU3iDrflgcJc8zmRfTB9E2KEATWvOZpmZXea50xRjV/3n4cNGXAFgjVoaJv0vle6qpuUpb6 4OjdpzYsgmcST6rSqXidR/GKMAB9VIysWDYtFpipd8ZaFlRYrPqxbk9P/tbMGhgzbo+vVyvufOf 9GZtcQ8ed6xSFYIBM/lH9glEEfR8YyZQxwBlds599HevRmXmoSWUPAj2emDPcIvv9Nf/tBXqHHH EWle9HYub+cnGXO/ZZ7LHLFRn/1xE9ZnjnDir4uCUoHG2fQSN2yQimLPC7cQJ+aTFR+m0dAh/c/ IuScgHhC15yj0hphSDAQKOWHZdYclS69YaXCB6e8Z3GigOOs7SyRG+YXT5td5GtFI7ySlTKt9ql FdT7fBLlt3yFZUbcU9V/WM/2KZIR5WW8tMM2o74bV5l26yXIbdLO/yM3/Xy7OqmuhR4PIKn84Hb XLjEi93Mic+97/CkVzIvgfpa1UrRqOJQZVeWYgnDsADwKhZyn5oSzRhJesWrnlZSD/WDntlCm6P 3QTXTa8PTS1ijDg== X-Developer-Key: i=keescook@chromium.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 X-Rspam-User: X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: CEC95C0024 X-Stat-Signature: d98zhmutu7ns7gtoy3q6j1pettfw3oz4 X-HE-Tag: 1713994867-763342 X-HE-Meta: U2FsdGVkX1/kg+oz44HlPcnfQN/rHCKIFcOrWW+pE0mxn2vzsRrTPU6ETuNKkxiERFgzuWu6ttVBA/xfeuGYrt1QFd9dtCkYKEV+dTh78nqG+kBEMMsdkzOXOHkHdUCPmBCMjWFZxEwRFm+v4E+0Z9z5Gc/4TxVV66n3HBRarAvH1o5MAKHlqUNAbtZK7S5yvYDqUKtiZm1SG9lCT1zQbMn3mnYYa3enM2195tn4Mn6upWMBErctXoR1NCYM8y70etsgjKwLSp52xwpPKOruKNEYRWLxiWJ2qf/nRQl8sUlCQGIRm/SOcLB/BhoZBs++PO6/7253FXgJ1KCn2+2OydLFAWPpXy15XNf8GBizBY5FbbeCQDa9/YNFvDwUisBt9gxfr3yEt06GpmfXOaq5xnBgw0dTAhKaGi1mgFo14vpfL8eLo1nnPm8cDglN6tcY5GXK9D8XWHQ19NE2wI2VLMthapMS3aOTliQBKkWF/bajJcqxt9uJM93wpZy5AE0P97LEJYwo79Pyd6tYbR2FpdjSvJa3+P7zOQViAJotyLLx7Meh4sT6jIwTZmgySDYveFER0zNI+WXwLVx6uz2UB2whQ5fctN0wP2CVK7EGuyDm/L8IZd0B8St7/Vu9DxK4Y0cgCOULDiEEanQPESxvDfwykt9diAX6HfT1wcQh+juyKoVPNiU3wCCE5rAgkCqicQVGWA396PO/EhqBIWLAuMHwLNsWmnX4h9/dF9va9lBztv7qKGD+xuryqqMe6F2BVPKMZ+hI38juMK0v2J1gI8kE+NW46vyLpO8xawmE5W0O0gIeBMuG6z32nCqjePU1tMCdpULfTVVs8v5zLzxrN28OOG0iYJWWYz6aCLMGx8zCDpk8qTkemFFghyn/uZVysDO3tmOSkKGbU2aofxt+P0jsMEoDPGRZ3pZkOp7HTx2mbDatBZMN/usFDHU8LeJHlXVs4q01nJDxq0LOivY 75V0XjHQ Ecd2dk7X1s3KmiNGSivYqT68uY4+gFP4GVVaCSmBc6XCdABloATSIh+8alClOTng5ZC3suthFUH/Mog2NZuJMR+OHdc+79dbYXLVfUX4zoZ4JJIvSLEGEptaJeixBQ+5Upp4Eeg4c039Zql7QkgBfMQUe7tRcyX3KbBGq+fimVXIx3BhZfvPckM6y2hciGStsCpLaiwECrTSoXi5Cfx/rQDp1oVeNzWWFV8FOAaItURZz8GfQXTwGq0Q69qPwjGAMNRObPczxqWnBM9NHBa52sorOo8ljom4AkGqicMejRKhtW0g/yUE7XtdO6QtQfZIr1LT5WfjIi9EyS9NtHpQ5JDMl4Zt1T6Vx0YTJSsk8MQm1qOHhff0pYuU0RfENlG6Hf0bEAxP5QXix8y0RK1ZHCvlyuxyKsJvmuWr4fQi7Ae3BRrllxKTrH12W908PeF0PVgK1fVMglz1oFZ108mi7EpLBzj2XZDUk7iB1Yp/T0cppZXesWfq4FplzSxwqbulRtVIqSKwf6mgLOOIR4pwIct65pZPd+2i5TNkSNRHcV8lwz71uwQjfa3zzK3Hpu06cMgfxxkwt2thVTqFgLZRGm5todEl3mHf41OYmUWNSAYeRfNRc1YrA7DotIW/omhFamNy5TOJ++uBSl/jrAcldJ0M66xnbCIEXcpmDu5IrQip/Xu+d6tV9x/+rNrpTCu/a+81cVOZW0o43manOwWi+1kRJ4pLzJjMZ46lWxOdjYhz+BF9/o2bkKDjgV2xES2yZthneM5P12FhOWUZ+Y85TWdI1XueLyz9t0736WBcTd491M0qJH+VSIvGwyf5tseFtqt0SmCYIxSdtN98AN3WXuzAhrqHYO1+Ua4M8xH4Qrn8nNfIuFGljzkqf990vgOgG5iVELd0Hq/zULDLBfRVfFEARPA5iDX+rey6IXQ16ffpSlZ4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Encapsulate the concept of a single set of kmem_caches that are used for the kmalloc size buckets. Redefine kmalloc_caches as an array of these buckets (for the different global cache buckets). Signed-off-by: Kees Cook --- Cc: Vlastimil Babka Cc: Christoph Lameter Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Cc: Andrew Morton Cc: Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: linux-mm@kvack.org --- include/linux/slab.h | 5 +++-- mm/slab_common.c | 3 +-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 4cc37ef22aae..c8164d5db420 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -426,8 +426,9 @@ enum kmalloc_cache_type { NR_KMALLOC_TYPES }; -extern struct kmem_cache * -kmalloc_caches[NR_KMALLOC_TYPES][KMALLOC_SHIFT_HIGH + 1]; +typedef struct kmem_cache * kmem_buckets[KMALLOC_SHIFT_HIGH + 1]; + +extern kmem_buckets kmalloc_caches[NR_KMALLOC_TYPES]; /* * Define gfp bits that should not be set for KMALLOC_NORMAL. diff --git a/mm/slab_common.c b/mm/slab_common.c index 8664da88e843..db9e1b15efd5 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -653,8 +653,7 @@ static struct kmem_cache *__init create_kmalloc_cache(const char *name, return s; } -struct kmem_cache * -kmalloc_caches[NR_KMALLOC_TYPES][KMALLOC_SHIFT_HIGH + 1] __ro_after_init = +kmem_buckets kmalloc_caches[NR_KMALLOC_TYPES] __ro_after_init = { /* initialization for https://llvm.org/pr42570 */ }; EXPORT_SYMBOL(kmalloc_caches); From patchwork Wed Apr 24 21:40:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 13642530 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C0A42C4345F for ; Wed, 24 Apr 2024 21:41:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9F89C6B031A; Wed, 24 Apr 2024 17:41:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 899AE6B031D; Wed, 24 Apr 2024 17:41:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 603186B031B; Wed, 24 Apr 2024 17:41:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 3B23E8D0031 for ; Wed, 24 Apr 2024 17:41:13 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id F0AD5A1223 for ; Wed, 24 Apr 2024 21:41:12 +0000 (UTC) X-FDA: 82045746384.16.3DCD44D Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) by imf30.hostedemail.com (Postfix) with ESMTP id 63C6180018 for ; Wed, 24 Apr 2024 21:41:10 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=GvCXYf1u; spf=pass (imf30.hostedemail.com: domain of keescook@chromium.org designates 209.85.214.181 as permitted sender) smtp.mailfrom=keescook@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713994870; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=X+QuwAWtOf2xR22BKuasFxurK2ECAnfs8jdrPoithuU=; b=rz2itMzAgHkgT73dHuG2D5w8l+pE181tYxFPiuWQpC3OJuZCpj3b9osajJlKnOiE/l6pJR 12csISYDbSIC+0HMaJvoVJsGGG2zhcITHTcwwwytiIFEHbepmypDxdAq5QTpN+xjJ5GpLz HIw6IkeiOhfQNjUWrw94f5FfF5YAdsk= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=GvCXYf1u; spf=pass (imf30.hostedemail.com: domain of keescook@chromium.org designates 209.85.214.181 as permitted sender) smtp.mailfrom=keescook@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713994870; a=rsa-sha256; cv=none; b=0br34Ehr5w76Z2C0KE0Bsts20HOpllm2DIhoWCmoT9//piWBF/r0UnNTJFtRJrykKODC+5 Yw87UyKKeycIZZt43N6kg1fjGk5Avlxg6igwMs++/ycxe+u8WSqFg6ru0l9HC28K/oKFYS JxCjHiUxH5fR2hidOCF831AxB6U5IIQ= Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-1e4f341330fso3134925ad.0 for ; Wed, 24 Apr 2024 14:41:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1713994869; x=1714599669; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=X+QuwAWtOf2xR22BKuasFxurK2ECAnfs8jdrPoithuU=; b=GvCXYf1uQNd0NVOyt/hORV5TT4KMWYehKYaOfhxPL4cHt1WAD/jp3z/ABqBu0zt51q 7x6xttzM8GYl8C9f3SnBBGoGcxS4V58mKSTygBwZK9+oFVNiS9AQx/GIHh03kfU4MGkW yEOMPP9mN6805q7gs1ALE2v4nNwOmbC1UgxI8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713994869; x=1714599669; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=X+QuwAWtOf2xR22BKuasFxurK2ECAnfs8jdrPoithuU=; b=AiTI+d+6ViR73fHXhQ7QBvI963EAmGmqujbStiM/4BdxcGkFGa1j0EvqqYMixj38kK mzIObQEH+U5BcdSn3FvVA2GAnCLy+URzdD7B0izYAepN7/vHfkpkcUETigVG/Pd1LESB Gn/JgAGBK9DmOcSSKdo0zacwJVjL6A9F0/RncV9PKOrhG4/Q3jEzyiCZLXe8rSysbYMp rY8+PZQgI4BOAWKr6ohuUGh93y3CDrBBKsAAFSOJRyNFV2exVETKq1mq9Z77IJYmR4yU wszG07CFu9/dFZl5zHFHOh+IRojN1skN5dd/9YPZW1itpo6sW00NLzyY7/D9l5PVQm6d G9RA== X-Forwarded-Encrypted: i=1; AJvYcCXqie1mjluq7Vm610Buu4fJSbdudF7X+L+Am8u8mzwMQHwmsQ6o/i8tsytAELRmdKIllekyCWcw0louQV907yNoDgc= X-Gm-Message-State: AOJu0Yzh3ybY5SMC8Ci4OKszHl6CxHIVR9pZEQ6pThvPFnbltSsoIM/p +XZHpzjMk9FgYEzNXmMp8ze+/seWXfKGvFfA8tk0AGT95Q5hcH5+elwI2yBQvg== X-Google-Smtp-Source: AGHT+IHIXGHtzh8lHI9UAe8GBSXBeuQxS5IrYdgWRav1tlMqSSJ4dcU1FmJlbd5lVXFZJW2TuOd4FQ== X-Received: by 2002:a17:902:cf0e:b0:1e7:b775:64bd with SMTP id i14-20020a170902cf0e00b001e7b77564bdmr4707051plg.53.1713994869316; Wed, 24 Apr 2024 14:41:09 -0700 (PDT) Received: from www.outflux.net ([198.0.35.241]) by smtp.gmail.com with ESMTPSA id x5-20020a170902a38500b001e2bfd40b86sm12604928pla.47.2024.04.24.14.41.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 24 Apr 2024 14:41:05 -0700 (PDT) From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-hardening@vger.kernel.org, "GONG, Ruiqi" , Xiu Jianfeng , Suren Baghdasaryan , Kent Overstreet , Jann Horn , Matteo Rizzo , Thomas Graf , Herbert Xu , julien.voisin@dustri.org, linux-kernel@vger.kernel.org Subject: [PATCH v3 2/6] mm/slab: Plumb kmem_buckets into __do_kmalloc_node() Date: Wed, 24 Apr 2024 14:40:59 -0700 Message-Id: <20240424214104.3248214-2-keescook@chromium.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240424213019.make.366-kees@kernel.org> References: <20240424213019.make.366-kees@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=8366; i=keescook@chromium.org; h=from:subject; bh=lWcazK2402lkD4Wrba9Q2qaF3Ixn3Ax0z7KNqGLKz60=; b=owEBbQKS/ZANAwAKAYly9N/cbcAmAcsmYgBmKXxuXSr8rdPRSyF2wBgMoZZNHIsBtwsrinlKe bq327jZWyuJAjMEAAEKAB0WIQSlw/aPIp3WD3I+bhOJcvTf3G3AJgUCZil8bgAKCRCJcvTf3G3A JjhaD/9qXDqOgVCP5PmY3bLEzzzpXHAburpbBFoBFgjOtLRbbCVNi0RXeJRDstJhZjQ1pSFczdM QuP1rYez2GCqsbpwqP/39GLi2CfoHjGr8Fzd0SM0gFnyMvWKgCo57WdcbvolAyq7Yh/YNK77dPA egxi2E4zufNQMrJHa8yj1lRv2SXvUtiwwhqpEHyEijmiiK1Xh7Y6P9x5lMiG/5ncfmQqjm6fsN9 GYRLjidguOaWEL37PqW6mA48nj9La6uF2qVApKk8allKd437NMwr6ChknmdQny65ggUP+tN2ARy 45Nb/YgZURzOlMvply7vlgbJprBaIzmAI+A6k2EYsJRkdUYkcrI2Alv1eFiIrEUcpxgLbycSI/m pGM9FBfYJhF8yOpuwlD/XAifJ/BQ2P9ugQj+D0zBfi5iX8Mas3UB4nC4AXCmU6MbhN/U2e9Yw84 Tqj35Urvfvfv8HCb5jf7dTQjZP+iu8iLVIJpHJKim2PUwciik3OQQDAskJrqN70pYdexnylVCF9 85XdAQYytNgqDmDbKXoQUcksz8LtATIg0a1kmk26vJjGxtc/jNs5yvxJtsIqpxRpezTN8xvoqTA 4GUn6gWe/LrYQFSgwpF4PVsgQ75hZzGbG4gf2uGo/lJ5YN9Sps2ciWQOM185FEQy/V3w3lg4n9m M6uPM7mb/9qJGVQ== X-Developer-Key: i=keescook@chromium.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 X-Stat-Signature: 8p6oe454wf6z8qi7n8jkd9ydr6om1j64 X-Rspamd-Queue-Id: 63C6180018 X-Rspamd-Server: rspam02 X-Rspam-User: X-HE-Tag: 1713994870-683186 X-HE-Meta: U2FsdGVkX1/bcQJtzNrbkxKdetYUQRdIrIrsv/PFYE+Uz37Vxj82bkvfHaMRRIWstEFs58DVydQ3J5usQfPfQo/y+D1TiokIf38vCBiwdLlOra1AOFLNq53wt92Wyw3WkABD66+61Hm64wn6fhnntwx6RnrlMDBaWbPfXpuPEnftr0WXDN80O3+BgmmHqU+EWZmKr1EiHdy4BHfVeHad951t8SF75W0+VlKB+qLbAJm+atjpapDKuDoLYNTBIjv7GF5ig8VUgA+e9YU7TFwfsfxNP7sk/gYOuQ+E5kywqDG/rPfmcZsWN5b27OD2NArSjv2/H8SBqIPQ3K6Z3v4BrHhwkQiaC+ElkWeUky0dtE2xYw+1s3tRUXR/Uiv0gi5omNosvXpzwr7eRHqHmuufDf5o1FsjnoizRpxtJKILvzsT6upM6zQEAxnoQSYz7zmy/rRtvZThNGbxaX7OxVNSbjd9+jeFRj8GC5JfO3ambPsyOakFkZD5JrxP91qwztz/VtAQ2yUdspBklYPJF33zueb64ee99IU1iZxZjKI8idYOoToDg1MaVn98pJgiL5G0vY/Y2CUuEnFd3w0jRbeeuVVq7+UKGCMa9M3W01jM/7/ohwnJhYlisrmz/viE0St5iTuNv97aHWL2JazaxjgINKGCGgJoVKMyPxt0sM57mgmccRO35/tdBA3baisozeiik5yluqm+U2k3pf5OZ5KxJeemQfUVR4+pGENFSiVhbVLTt1CcAJjFvT3igdaM6TWhJo/pxXIgBU04l+EglHnCWcv5L5Wxu8n4Ea7yzwdPLDcFwYOrQ++N6sdAOQCmp95mDb9oYLAjF4kppgvVajkM9fHl4qoiQNseFaZE79ra8RLQyrlfR3Q5h0yWWIPl97ArfIVKrzdmkmPIMcwmJnew1JbMij8pMgW3brse+6Y+HdTJbcvUpRsJTzw+Ai1Z0QMgcFjcLBwJLrBNTA5YpNA tZex3kMS znLA5GRTM2C+TLorHUr5TIFHaBvEZfsPGLh8eqk0vAxWxJ8MDcarIVf97Wb+2+0WQNbJ3QJnQ3uORrUWGmUWhj0quxNhHrnbxQdtlg6A5nZHr/nF7nHdTWIdPInYAE2qEvGrblTjYeQKMHuujYQ8j/jM4EFLdpGbwRD2Un1dt1IufdiFQpwz4BxD+9+TK2DS3NBNzvctuGyDW/XwAw9DMolBB/Ke98atlrITXA7tcSMWQUJ+ztfQ2Mp4HRnUpy5vu8qZWBEJzz22Fi3SKaTIGtwdYzFVBXDwdIEQ72Bpd/PPMe036+783JbWMh4ccNdqeX1PK++B8SXPftwniKpLD36qmpWHrnQ2fsObjgvYbaQIjh7D9u4adQLlLbdI/mLrQD3zVSN9Zzol4VZU8QJsEYFM4hvz7WcHZ5yDmzVfM38jTv7/JVNjAv7FeAMpv7Rx2hgRw5CNwpzDu3EpTJfEzrixmg+y3h5+SUvdrGzW2V19zGN23zMW6bPuwYiu9l6RGn7TSTHlup0dxf2E9Oj83FcUG2bEnZVubjthxYCHrlDDCGg0tkCrvJRlpxYz0wm/elEEpiHSZp6N8zT4TaCLKLq5m/hnRINb5XBhgYSfO/AH0kS/tV5mHGoG5Lq7SQtLij7WBsFb1NoNSunPWeL6De7AVc4W/zxWN8SzPJ9URi5NIxHtlRCj8GAZ4or+a2JJwH4AOv0UhP+eXhQyMUmis895L7BFujIZ9ErSl4AOjTDTkIfxgQFJFdFsvZYZBboZOSf5D+19QgGgPy/SgaZGRovpuNXcEgFXrBiohrE9sAARtVTK9dc25fCLltAiZizZHRcTSfmNQrVpsD8qCJ3VWr7kxiaEWz8Z4nTpMny0PMs2l9q0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: To be able to choose which buckets to allocate from, make the buckets available to the lower level kmalloc interfaces by adding them as the first argument. Where the bucket is not available, pass NULL, which means "use the default system kmalloc bucket set" (the prior existing behavior), as implemented in kmalloc_slab(). Signed-off-by: Kees Cook --- Cc: Vlastimil Babka Cc: Christoph Lameter Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Cc: Andrew Morton Cc: Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: linux-mm@kvack.org Cc: linux-hardening@vger.kernel.org --- include/linux/slab.h | 16 ++++++++-------- lib/fortify_kunit.c | 2 +- mm/slab.h | 6 ++++-- mm/slab_common.c | 4 ++-- mm/slub.c | 14 +++++++------- mm/util.c | 2 +- 6 files changed, 23 insertions(+), 21 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index c8164d5db420..07373b680894 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -569,8 +569,8 @@ static __always_inline void kfree_bulk(size_t size, void **p) kmem_cache_free_bulk(NULL, size, p); } -void *__kmalloc_node_noprof(size_t size, gfp_t flags, int node) __assume_kmalloc_alignment - __alloc_size(1); +void *__kmalloc_node_noprof(kmem_buckets *b, size_t size, gfp_t flags, int node) + __assume_kmalloc_alignment __alloc_size(2); #define __kmalloc_node(...) alloc_hooks(__kmalloc_node_noprof(__VA_ARGS__)) void *kmem_cache_alloc_node_noprof(struct kmem_cache *s, gfp_t flags, @@ -679,7 +679,7 @@ static __always_inline __alloc_size(1) void *kmalloc_node_noprof(size_t size, gf kmalloc_caches[kmalloc_type(flags, _RET_IP_)][index], flags, node, size); } - return __kmalloc_node_noprof(size, flags, node); + return __kmalloc_node_noprof(NULL, size, flags, node); } #define kmalloc_node(...) alloc_hooks(kmalloc_node_noprof(__VA_ARGS__)) @@ -730,10 +730,10 @@ static inline __realloc_size(2, 3) void * __must_check krealloc_array_noprof(voi */ #define kcalloc(n, size, flags) kmalloc_array(n, size, (flags) | __GFP_ZERO) -void *kmalloc_node_track_caller_noprof(size_t size, gfp_t flags, int node, - unsigned long caller) __alloc_size(1); +void *kmalloc_node_track_caller_noprof(kmem_buckets *b, size_t size, gfp_t flags, int node, + unsigned long caller) __alloc_size(2); #define kmalloc_node_track_caller(...) \ - alloc_hooks(kmalloc_node_track_caller_noprof(__VA_ARGS__, _RET_IP_)) + alloc_hooks(kmalloc_node_track_caller_noprof(NULL, __VA_ARGS__, _RET_IP_)) /* * kmalloc_track_caller is a special version of kmalloc that records the @@ -746,7 +746,7 @@ void *kmalloc_node_track_caller_noprof(size_t size, gfp_t flags, int node, #define kmalloc_track_caller(...) kmalloc_node_track_caller(__VA_ARGS__, NUMA_NO_NODE) #define kmalloc_track_caller_noprof(...) \ - kmalloc_node_track_caller_noprof(__VA_ARGS__, NUMA_NO_NODE, _RET_IP_) + kmalloc_node_track_caller_noprof(NULL, __VA_ARGS__, NUMA_NO_NODE, _RET_IP_) static inline __alloc_size(1, 2) void *kmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags, int node) @@ -757,7 +757,7 @@ static inline __alloc_size(1, 2) void *kmalloc_array_node_noprof(size_t n, size_ return NULL; if (__builtin_constant_p(n) && __builtin_constant_p(size)) return kmalloc_node_noprof(bytes, flags, node); - return __kmalloc_node_noprof(bytes, flags, node); + return __kmalloc_node_noprof(NULL, bytes, flags, node); } #define kmalloc_array_node(...) alloc_hooks(kmalloc_array_node_noprof(__VA_ARGS__)) diff --git a/lib/fortify_kunit.c b/lib/fortify_kunit.c index 493ec02dd5b3..ff059d88d455 100644 --- a/lib/fortify_kunit.c +++ b/lib/fortify_kunit.c @@ -220,7 +220,7 @@ static void alloc_size_##allocator##_dynamic_test(struct kunit *test) \ checker(expected_size, __kmalloc(alloc_size, gfp), \ kfree(p)); \ checker(expected_size, \ - __kmalloc_node(alloc_size, gfp, NUMA_NO_NODE), \ + __kmalloc_node(NULL, alloc_size, gfp, NUMA_NO_NODE), \ kfree(p)); \ \ orig = kmalloc(alloc_size, gfp); \ diff --git a/mm/slab.h b/mm/slab.h index 5f8f47c5bee0..f459cd338852 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -403,16 +403,18 @@ static inline unsigned int size_index_elem(unsigned int bytes) * KMALLOC_MAX_CACHE_SIZE and the caller must check that. */ static inline struct kmem_cache * -kmalloc_slab(size_t size, gfp_t flags, unsigned long caller) +kmalloc_slab(kmem_buckets *b, size_t size, gfp_t flags, unsigned long caller) { unsigned int index; + if (!b) + b = &kmalloc_caches[kmalloc_type(flags, caller)]; if (size <= 192) index = kmalloc_size_index[size_index_elem(size)]; else index = fls(size - 1); - return kmalloc_caches[kmalloc_type(flags, caller)][index]; + return (*b)[index]; } gfp_t kmalloc_fix_flags(gfp_t flags); diff --git a/mm/slab_common.c b/mm/slab_common.c index db9e1b15efd5..7cb4e8fd1275 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -702,7 +702,7 @@ size_t kmalloc_size_roundup(size_t size) * The flags don't matter since size_index is common to all. * Neither does the caller for just getting ->object_size. */ - return kmalloc_slab(size, GFP_KERNEL, 0)->object_size; + return kmalloc_slab(NULL, size, GFP_KERNEL, 0)->object_size; } /* Above the smaller buckets, size is a multiple of page size. */ @@ -1186,7 +1186,7 @@ __do_krealloc(const void *p, size_t new_size, gfp_t flags) return (void *)p; } - ret = kmalloc_node_track_caller_noprof(new_size, flags, NUMA_NO_NODE, _RET_IP_); + ret = kmalloc_node_track_caller_noprof(NULL, new_size, flags, NUMA_NO_NODE, _RET_IP_); if (ret && p) { /* Disable KASAN checks as the object's redzone is accessed. */ kasan_disable_current(); diff --git a/mm/slub.c b/mm/slub.c index 23bc0d236c26..a94a0507e19c 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4093,7 +4093,7 @@ void *kmalloc_large_node_noprof(size_t size, gfp_t flags, int node) EXPORT_SYMBOL(kmalloc_large_node_noprof); static __always_inline -void *__do_kmalloc_node(size_t size, gfp_t flags, int node, +void *__do_kmalloc_node(kmem_buckets *b, size_t size, gfp_t flags, int node, unsigned long caller) { struct kmem_cache *s; @@ -4109,7 +4109,7 @@ void *__do_kmalloc_node(size_t size, gfp_t flags, int node, if (unlikely(!size)) return ZERO_SIZE_PTR; - s = kmalloc_slab(size, flags, caller); + s = kmalloc_slab(b, size, flags, caller); ret = slab_alloc_node(s, NULL, flags, node, caller, size); ret = kasan_kmalloc(s, ret, size, flags); @@ -4117,22 +4117,22 @@ void *__do_kmalloc_node(size_t size, gfp_t flags, int node, return ret; } -void *__kmalloc_node_noprof(size_t size, gfp_t flags, int node) +void *__kmalloc_node_noprof(kmem_buckets *b, size_t size, gfp_t flags, int node) { - return __do_kmalloc_node(size, flags, node, _RET_IP_); + return __do_kmalloc_node(b, size, flags, node, _RET_IP_); } EXPORT_SYMBOL(__kmalloc_node_noprof); void *__kmalloc_noprof(size_t size, gfp_t flags) { - return __do_kmalloc_node(size, flags, NUMA_NO_NODE, _RET_IP_); + return __do_kmalloc_node(NULL, size, flags, NUMA_NO_NODE, _RET_IP_); } EXPORT_SYMBOL(__kmalloc_noprof); -void *kmalloc_node_track_caller_noprof(size_t size, gfp_t flags, +void *kmalloc_node_track_caller_noprof(kmem_buckets *b, size_t size, gfp_t flags, int node, unsigned long caller) { - return __do_kmalloc_node(size, flags, node, caller); + return __do_kmalloc_node(b, size, flags, node, caller); } EXPORT_SYMBOL(kmalloc_node_track_caller_noprof); diff --git a/mm/util.c b/mm/util.c index c9e519e6811f..80430e5ba981 100644 --- a/mm/util.c +++ b/mm/util.c @@ -128,7 +128,7 @@ void *kmemdup_noprof(const void *src, size_t len, gfp_t gfp) { void *p; - p = kmalloc_node_track_caller_noprof(len, gfp, NUMA_NO_NODE, _RET_IP_); + p = kmalloc_node_track_caller_noprof(NULL, len, gfp, NUMA_NO_NODE, _RET_IP_); if (p) memcpy(p, src, len); return p; From patchwork Wed Apr 24 21:41:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 13642528 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D19BFC19F4F for ; Wed, 24 Apr 2024 21:41:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B5BEC6B0316; Wed, 24 Apr 2024 17:41:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B32276B0318; Wed, 24 Apr 2024 17:41:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 983DB6B0319; Wed, 24 Apr 2024 17:41:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 7326A6B0316 for ; Wed, 24 Apr 2024 17:41:11 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 1E6D4140D90 for ; Wed, 24 Apr 2024 21:41:11 +0000 (UTC) X-FDA: 82045746342.12.BA5C43B Received: from mail-pf1-f177.google.com (mail-pf1-f177.google.com [209.85.210.177]) by imf14.hostedemail.com (Postfix) with ESMTP id 29226100002 for ; Wed, 24 Apr 2024 21:41:08 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=mdaTI1vT; spf=pass (imf14.hostedemail.com: domain of keescook@chromium.org designates 209.85.210.177 as permitted sender) smtp.mailfrom=keescook@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713994869; a=rsa-sha256; cv=none; b=rRIqO6k4uon3qzmiZ8btb/8EByU92GcWt+7am75wAwYnKcTPeKL1n4DAr4fWP+ulJhI4lA 2nzaum5OtHJZOTVPFSixgpqiEwTiMS9X40mqWYgUoSzAKDT39naQgUza31zje3Yezjn1Fg x1Wp556aGimPITpVp+HK8+3KS/Q10Wg= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=mdaTI1vT; spf=pass (imf14.hostedemail.com: domain of keescook@chromium.org designates 209.85.210.177 as permitted sender) smtp.mailfrom=keescook@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713994869; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=avaWrT0khuREU5h1MMjzzZvo4YgUdeePn0h1OukYT6Y=; b=5kitBQy+wj3nuCwnwgFsL4ZNV12CZVyKG+HTEovliXrP8yVXOJSlsDf0+scKyNM9qlV0Js GmUQt/TjOCSJR9j8g9ekafvqGIznNUkOqjGPlbp+PTf4IEA9RIQr9b25OJEW3DUXirJxJz rLhPb05RnwX+hIeYm20PfY1Rdt2UA88= Received: by mail-pf1-f177.google.com with SMTP id d2e1a72fcca58-6ecee1f325bso394217b3a.2 for ; Wed, 24 Apr 2024 14:41:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1713994868; x=1714599668; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=avaWrT0khuREU5h1MMjzzZvo4YgUdeePn0h1OukYT6Y=; b=mdaTI1vTAce/Euwhh3T1GfY8nqMVY9pA2gjxEzI1xuhehFm0x9BWAThoHXbkry44gE 4aQ0d+uiHGr+wCe0Y8SmkaiD/XL+OQbPnBujWOhHGeQ/T1d+3YLqRPjnWAm1BheU6ZwY D5iu7XzTedKsStBxk560P24uzSBoU9U+33LqU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713994868; x=1714599668; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=avaWrT0khuREU5h1MMjzzZvo4YgUdeePn0h1OukYT6Y=; b=uijusWBsgfSgmoGsH93l51Z28gHN7ne5sCooaB60QGo8EL/ARmPezfFq1dpCKGBT5N Cy7Rm36YyNIWZs0+NcZJ8Cn4FajSe7iddBAg2kgIfmMQTiFnQ+wgrumqB+A3QXgdwH4C JREGCGO8ws7y7YlfmRqkSXWvJAANnoeRGt2PG+8MNkNHvzvZE/fRqUC25u+c0qKp30hW S6F5LsVd6Ja6l4TBfo6g6fVIaYeEMEajiNsybxOpBLUlb4Hzo4Opw7R+xy4ar4YxXvov HZ0PDiflIFg9C4rDKobpZUytk4KqDmywrWQq2+qDZ4kzoRZOjXKoM0p8wusQCIEPLrfm Tmfw== X-Forwarded-Encrypted: i=1; AJvYcCU4NPnJGpv2bTpdjWyfqXpwZhAqa+S2/2lPE21W/D8prp2ojJl51N+GW9cKlJ4tmap19YplAPmiB2dvPB3ljJOi3/s= X-Gm-Message-State: AOJu0YwOU8q9F3FGZmBgGoFFBg1fQItpSaxYnpcKwiPylYLzCHkX/6KQ c/yiIBszyHxtvjXmWX8ZEckRK8AhG6H0JmSy4hqCEqJd/5/5QNhWDEjpxvT2Ug== X-Google-Smtp-Source: AGHT+IHXJzscAQxkhpdn10b8o9bm2jTf9Syzbn/UYNQIO/7g7UQSGqLaU9uDrKL3kC60AEISiyGgRQ== X-Received: by 2002:a05:6a20:5616:b0:1a7:a353:939e with SMTP id ir22-20020a056a20561600b001a7a353939emr3679161pzc.56.1713994868043; Wed, 24 Apr 2024 14:41:08 -0700 (PDT) Received: from www.outflux.net ([198.0.35.241]) by smtp.gmail.com with ESMTPSA id x1-20020a63b201000000b005e838b99c96sm11611467pge.80.2024.04.24.14.41.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 24 Apr 2024 14:41:05 -0700 (PDT) From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, "GONG, Ruiqi" , Xiu Jianfeng , Suren Baghdasaryan , Kent Overstreet , Jann Horn , Matteo Rizzo , Thomas Graf , Herbert Xu , julien.voisin@dustri.org, linux-kernel@vger.kernel.org, linux-hardening@vger.kernel.org Subject: [PATCH v3 3/6] mm/slab: Introduce __kvmalloc_node() that can take kmem_buckets argument Date: Wed, 24 Apr 2024 14:41:00 -0700 Message-Id: <20240424214104.3248214-3-keescook@chromium.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240424213019.make.366-kees@kernel.org> References: <20240424213019.make.366-kees@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=4187; i=keescook@chromium.org; h=from:subject; bh=n44EIwrVbtUC/HODjVXTzsGeRWoVsJemo0On/8Qrf5o=; b=owEBbQKS/ZANAwAKAYly9N/cbcAmAcsmYgBmKXxuIQwklSExmXc87p72DCfh76AnwjYnK9Py8 7Fv1B+pgpeJAjMEAAEKAB0WIQSlw/aPIp3WD3I+bhOJcvTf3G3AJgUCZil8bgAKCRCJcvTf3G3A JupMD/0VCva4CsvxyFhP/1VkRIHNT9neF+T4OIPegvk9amzhCJfRNJLUcIqZXAK9xEgkjQfbh2D vH/7wvvpfhoK+5xeXHZOkczwMl8+V2SiJieg5YMyTQs9ngeiUc00hjMfo/xIUlrurpZtf2QksCA Z5jRwHG/EgB7hCDS3nYtURFBj/AXsQTcrLiVJx+SqAmUIZ5yIDa/CocjagCWKN5gwxWe3CEpp56 PqhbCSxFQc5T/e5xN2S15FaupnNJX9s6PXe2Nx8Q/QUSQXDdwL/XvWV8F6kdM/xCOfYhMu0lrxj Wi29HbzMDnNAhqYpc8Ia7vtAX8v6zRiVS0S5i6HRIiRWHJnkNCwXLR92/Gsr/Ta8rt3yVjgjwC6 WY0VfE2J4xY9wz3TgKXuEYNzOqkTyCXyMGGUZe21ghzrocTHqP/8jxO5T7WvFt4PoBBEAcZ5fxO SXGz+iIscVt3qMrVkMfbkgVlY1xaLCz2N4du08JCuwtL9+vntjBOMH4YH9rTxx8D3MRkShjZ0y0 dBVtSr8odvEdUwdHX4H7FbvCEUJ5aIr3hfLxW9PKBb1H7sv3zw0jMEVpbxezXhcbUxqoiodcJDg YlL/45F9uG1CLOOK3MZLM1s1rOOVst3540xcKfxutWb2HIAhckktrVwaCmRKz8NAfuZVOEOIp8a wbiLrOqzM2i8B/g== X-Developer-Key: i=keescook@chromium.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 29226100002 X-Stat-Signature: w9y7r9nmcykqw4zg84gp1fa5im1giopw X-Rspam-User: X-HE-Tag: 1713994868-114192 X-HE-Meta: U2FsdGVkX1/3yAFa2UHpIPwfEdp/LHvScGBzHCDIP6oKpMMuOENzB/bkqw6ShGWSbiMNYv4udQ0rH/++A3jgsw/PUR2qaGALAVCEXtztaBRQfp95OXI7qG2eA5qatbm+8x7kmIZgbUFzsM1FQGJW8x77k9YNd0CWj5lGgduGvRJghR2hxKzkNm0XxI5JcC6p2SW8O7l3qaanS5d72FqlrAkJCisXK9ww3970MTVeh/IMDBYBHxwiAVZmJ7Jq6AOBjG6GUfLa0zg+gqbfaWu0cghblyBpeJ9L1p0gauFtBv6Gn1H2XtgDvYp1mOgboGifNCiutt2dEDgIbwcSh7P5Dat5mchALzlQ7CmTuNO00ITXjqh4Q7cQ6npNlQi+VbMCyjtC7Cz2VrgxZM0yYRmZdcrasTlFM0LEeMC40wRglBM7wkNk8DEKpQHeHuAcH4/Fn5lN3dePw248FJeFGX4BbcYDCQAJwQDsbm0K2fohLO3HSxJh2FrPoBMzyAZxvrPtnIE+1uWl33H3sC2PfDWlTCWt68vy/eqO/BbQpjzAXQ9t6iVb73zpT/LDm3tTH1DQumrVNJGLJJ1GkV/2qI+P1+VADG+Mo+EVtwjIAB/dptKxpTkZEAC1Cv6Z40+3MAYLv8+whazfcivqZyGtqkraG/BWEMaHDm1eetbwGrSXod4hNRqrYM+darz10ANFD6pXm2sDSdQxv3i+F984wL3+eIlktvwcOhzbZvcyIqDhkb2PDBl+BviQgaLcc1geFhmmmKinmo1lYGKLWxR4yPmbCmeo2nL3PUi5WjxyZz8kScQuj+UwQ0Ezak7i/ZqAghDYgfyvaVHVp57Qi7KvdVV0Nbq81jFbBlNr79827ordnzhpe1Ups33jK7TsqW4XtyqbPS/6p9aigqncAjBJ3X0kBmdU1lcV2BKdJTGXvivuvR967GTlsD7fKEJBtioME4Wstpz8X6QNgoK7tFi2SN8 OZdOXheF m1zfMNqTjX8Ku8FKdCKKQXg6kGBDK/5lshh03zBrAUOkwOB1hANSodVdYa06qirFCl6iGsaoLkD/MjPdAfjhlJYdG62iyZB2CmIUf5bmvK5jsbTicqaiOuWkJp257TrvOCCzUy1aYzjWqIXhyQq9Ihum0z0P5bdwU6GoN906GVFlmWE+GSFK9kLyL5k4WvOZzNJCtstETI6gJo63e6jaKBnwKhxvQ1A17u6KaoxVhoBEawfCRxmHNgXxpvanL0x+bibukH67NDSXBamNp7+/eygx+7NoXiHRQp7WCWNjnG+eb4p/ui+8Gv6Yr3mMflxpW8WIxbYrL+1bxul3PZUxqUhVSpQCshD+Z5A1gLD6OHoRO/QvdTPhJj9VMlQ+VtYtkHRlyP1pf3BfC3t3EEeXiQ9FurjN5WtJd+qhPRBWJGaey25RSnWryZaKRInQR2MyHUVu6c6SyfJgxAdoa+RtVuxr5b2b8nsaK9p12O4UITWsQ7MjJqZyv6RADNyKglb6kdPoc+Hx1fl98CCKNs5+0WKwwZP5Be0toZlbC+hERfuJVidmOC5vJObX0+DZ/HCuXRCBIy6wuhzOdG+osJCAuZ3SfC5nNEhRtxR8ikHLsMmA5C2kRnGtjEE2sY/8v/V8TLlte5i0MXacD9C4WfViF0yy5b8Vzh+8XO6sKNA4rRWKZKdy9BtJfyuwJAOD5IwM5f0/jT/S7CpUu3SanGEauaFSRByM7QQM7ToKTh9IXNGv1KGcad+/bxOdLdH7q9wkdUFUeaT3DDkpitCSDH1LyiNYevy8Xz6/PSejUdQH6FRYQYzHunsJkMqqEL7+77grAg/NHm5IE+B4Y3dmZhQUUarRDs8xVyOcoanUsHXIPbJ1j62I= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Plumb kmem_buckets arguments through kvmalloc_node_noprof() so it is possible to provide an API to perform kvmalloc-style allocations with a particular set of buckets. Introduce __kvmalloc_node() that takes a kmem_buckets argument. Signed-off-by: Kees Cook --- Cc: Vlastimil Babka Cc: Christoph Lameter Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Cc: Andrew Morton Cc: Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: linux-mm@kvack.org --- include/linux/slab.h | 10 ++++++---- lib/rhashtable.c | 2 +- mm/util.c | 5 +++-- 3 files changed, 10 insertions(+), 7 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 07373b680894..23b13be0ac95 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -781,11 +781,13 @@ static inline __alloc_size(1) void *kzalloc_noprof(size_t size, gfp_t flags) #define kzalloc(...) alloc_hooks(kzalloc_noprof(__VA_ARGS__)) #define kzalloc_node(_size, _flags, _node) kmalloc_node(_size, (_flags)|__GFP_ZERO, _node) -extern void *kvmalloc_node_noprof(size_t size, gfp_t flags, int node) __alloc_size(1); -#define kvmalloc_node(...) alloc_hooks(kvmalloc_node_noprof(__VA_ARGS__)) +extern void *kvmalloc_node_noprof(kmem_buckets *b, size_t size, gfp_t flags, int node) + __alloc_size(2); +#define __kvmalloc_node(...) alloc_hooks(kvmalloc_node_noprof(__VA_ARGS__)) +#define kvmalloc_node(...) __kvmalloc_node(NULL, __VA_ARGS__) #define kvmalloc(_size, _flags) kvmalloc_node(_size, _flags, NUMA_NO_NODE) -#define kvmalloc_noprof(_size, _flags) kvmalloc_node_noprof(_size, _flags, NUMA_NO_NODE) +#define kvmalloc_noprof(_size, _flags) kvmalloc_node_noprof(NULL, _size, _flags, NUMA_NO_NODE) #define kvzalloc(_size, _flags) kvmalloc(_size, _flags|__GFP_ZERO) #define kvzalloc_node(_size, _flags, _node) kvmalloc_node(_size, _flags|__GFP_ZERO, _node) @@ -797,7 +799,7 @@ static inline __alloc_size(1, 2) void *kvmalloc_array_noprof(size_t n, size_t si if (unlikely(check_mul_overflow(n, size, &bytes))) return NULL; - return kvmalloc_node_noprof(bytes, flags, NUMA_NO_NODE); + return kvmalloc_node_noprof(NULL, bytes, flags, NUMA_NO_NODE); } #define kvmalloc_array(...) alloc_hooks(kvmalloc_array_noprof(__VA_ARGS__)) diff --git a/lib/rhashtable.c b/lib/rhashtable.c index dbbed19f8fff..ef0f496e4aed 100644 --- a/lib/rhashtable.c +++ b/lib/rhashtable.c @@ -184,7 +184,7 @@ static struct bucket_table *bucket_table_alloc(struct rhashtable *ht, static struct lock_class_key __key; tbl = alloc_hooks_tag(ht->alloc_tag, - kvmalloc_node_noprof(struct_size(tbl, buckets, nbuckets), + kvmalloc_node_noprof(NULL, struct_size(tbl, buckets, nbuckets), gfp|__GFP_ZERO, NUMA_NO_NODE)); size = nbuckets; diff --git a/mm/util.c b/mm/util.c index 80430e5ba981..bdec4954680a 100644 --- a/mm/util.c +++ b/mm/util.c @@ -596,6 +596,7 @@ EXPORT_SYMBOL(vm_mmap); /** * kvmalloc_node - attempt to allocate physically contiguous memory, but upon * failure, fall back to non-contiguous (vmalloc) allocation. + * @b: which set of kmalloc buckets to allocate from. * @size: size of the request. * @flags: gfp mask for the allocation - must be compatible (superset) with GFP_KERNEL. * @node: numa node to allocate from @@ -609,7 +610,7 @@ EXPORT_SYMBOL(vm_mmap); * * Return: pointer to the allocated memory of %NULL in case of failure */ -void *kvmalloc_node_noprof(size_t size, gfp_t flags, int node) +void *kvmalloc_node_noprof(kmem_buckets *b, size_t size, gfp_t flags, int node) { gfp_t kmalloc_flags = flags; void *ret; @@ -631,7 +632,7 @@ void *kvmalloc_node_noprof(size_t size, gfp_t flags, int node) kmalloc_flags &= ~__GFP_NOFAIL; } - ret = kmalloc_node_noprof(size, kmalloc_flags, node); + ret = __kmalloc_node_noprof(b, size, kmalloc_flags, node); /* * It doesn't really make sense to fallback to vmalloc for sub page From patchwork Wed Apr 24 21:41:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 13642529 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EEC1BC10F15 for ; Wed, 24 Apr 2024 21:41:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6EDFA6B0318; Wed, 24 Apr 2024 17:41:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 67C756B031A; Wed, 24 Apr 2024 17:41:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3E0688D0034; Wed, 24 Apr 2024 17:41:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 1C7396B0318 for ; Wed, 24 Apr 2024 17:41:13 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id B41C7C0A67 for ; Wed, 24 Apr 2024 21:41:12 +0000 (UTC) X-FDA: 82045746384.17.2AF0EF0 Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) by imf20.hostedemail.com (Postfix) with ESMTP id B9FB91C0008 for ; Wed, 24 Apr 2024 21:41:10 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=C5alb6vS; spf=pass (imf20.hostedemail.com: domain of keescook@chromium.org designates 209.85.214.179 as permitted sender) smtp.mailfrom=keescook@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713994870; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xgSg7tRErA3GTFqfC6/09RNvabn2ysccNK/8UlB0lqg=; b=bGu576w2ucP8BTBwEI20GfVenwsGbW92QautJwGWJQ463BZy+RwpdbnXAe5GLgDHLg/Ejf Ieafx3Wzm5PSZ2rOeYQDrm35VChaiQB8xmMNjvzdk/6BC6nHroypPpUuFpiFjJp0L8uflG CM/6DiDP7XVepcUlu4fsDyHSg3YQctU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713994870; a=rsa-sha256; cv=none; b=VttoCNZ8kbpyIZ0q5i6TLAg4rbbKdE5f93u7kX4YRK1aeA5R87hsaZfeY8WUzsSegp/bVT JNdZ+Dac+9eqE6CobEEZBAzCrG4Dh1qcqIQuaxXoWeM1r83ZeYfAy5VsTK4Ie56szRlegG gbv7pms8LTIzAPKGFJjl8UWTLsEXaCM= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=C5alb6vS; spf=pass (imf20.hostedemail.com: domain of keescook@chromium.org designates 209.85.214.179 as permitted sender) smtp.mailfrom=keescook@chromium.org; dmarc=pass (policy=none) header.from=chromium.org Received: by mail-pl1-f179.google.com with SMTP id d9443c01a7336-1e4bf0b3e06so3038135ad.1 for ; Wed, 24 Apr 2024 14:41:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1713994870; x=1714599670; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xgSg7tRErA3GTFqfC6/09RNvabn2ysccNK/8UlB0lqg=; b=C5alb6vSTJXw9aUwLu9Zwi3F74d57+N24De90pRd/sunEPy5fYM2zSoXeXmSqmscCD UreqdwRfLFWMhX0RwwF6dXYaMo4i8c8tav8juktjWU5oOZSJJ9Rgr8NwwOGw+Ruf0Df/ O4pJ/8d5HJ4YXiGT/WHUobUZ7lgL5OC1A5rQE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713994870; x=1714599670; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xgSg7tRErA3GTFqfC6/09RNvabn2ysccNK/8UlB0lqg=; b=A/ONEcNEFfZjO+pNlBoqh6yuGpAbAQPhabxkuaCgPuizm+8RIr3djGkzSnyEt058HT quNWY+ya7xK2Zo8EqFfUKiPpV+Ry2G1b7QwUq3Fk4qhvyyPVGrAXofVdqFZj8jHJtqCU KadY461sV/5zU6YjDFuOJNMdXUsxQBqnqqF6O7BhnvWCtSMRGZ+JUaKJeGoS1y4dSo8w s8YnDmEibnVperapgeTPnjIJn4aaLb6DzYhjmjDisriYQpPTsnABiVzceRreLP3Sxmta kkK5yH/rpPhSQxYhZNQRt3wYe2g0qGCOHthDJyAZMTcDVwo1froUqFoBo9NaxgMleEqh J79w== X-Forwarded-Encrypted: i=1; AJvYcCUmcyTDGfSZ9EYwhHDk/Kya5/PWjZ9yI5mJsJPv/E5m8Olg1e2CYgrMNwTnZvsKrAPDEI2jNE4JQqPpH1roKyev0UM= X-Gm-Message-State: AOJu0Yy0OjP+YuWPh0QmAOM2h2BKqnIcMmJIz18huYtrXkrcUGRdDOUz RlVFJKmviVuzhLngpJ6VdPfQh2aoO6sT69m3ny/qlE1GRrwR/tOcINSMqPOhgQ== X-Google-Smtp-Source: AGHT+IHb55e4QA/5/nvYk8hjmJDauEXroyLEAzvYHGIUTin2g6qZf3SjNVhCZtuDOfjBzSSBH/ffBQ== X-Received: by 2002:a17:903:32c5:b0:1e3:e1ff:2e79 with SMTP id i5-20020a17090332c500b001e3e1ff2e79mr4664344plr.45.1713994869664; Wed, 24 Apr 2024 14:41:09 -0700 (PDT) Received: from www.outflux.net ([198.0.35.241]) by smtp.gmail.com with ESMTPSA id a8-20020a170902ecc800b001e944fc9248sm8170647plh.194.2024.04.24.14.41.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 24 Apr 2024 14:41:05 -0700 (PDT) From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, "GONG, Ruiqi" , Xiu Jianfeng , Suren Baghdasaryan , Kent Overstreet , Jann Horn , Matteo Rizzo , Thomas Graf , Herbert Xu , julien.voisin@dustri.org, linux-kernel@vger.kernel.org, linux-hardening@vger.kernel.org Subject: [PATCH v3 4/6] mm/slab: Introduce kmem_buckets_create() and family Date: Wed, 24 Apr 2024 14:41:01 -0700 Message-Id: <20240424214104.3248214-4-keescook@chromium.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240424213019.make.366-kees@kernel.org> References: <20240424213019.make.366-kees@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=7010; i=keescook@chromium.org; h=from:subject; bh=C0Prx/nK0APcgMVkHJYdvYJapdDY3C5e64QBsP8kI7M=; b=owEBbQKS/ZANAwAKAYly9N/cbcAmAcsmYgBmKXxuY6dgso5CobllhqEMKf+YP1OAj0tPWmvHV V/Awo4RmLeJAjMEAAEKAB0WIQSlw/aPIp3WD3I+bhOJcvTf3G3AJgUCZil8bgAKCRCJcvTf3G3A Jqa7D/4xKQ5cm8fo1KortUawuJbdD58RkZxTRydP+JlQbVi/tshUUQFxdEjS36UO7Ine1xNL6eL VQkwIqn4v+jkXbAlyXJynBw6USOvJPLMvkjKyTbnF+xRbqnLcZwtbnDDAXWmCSDAAYQng+WwgxG 8lJQJftOJBVtOIeQmjDgzC2L8KX0WFPKVnlWuV7fSsdyPvqloa2Q1JNk2bdHwFCEv35X8aQSnt/ 4fO8eHYzhOInTIe5set0qN1Ho70Dkalud5cv/0BJTeECVfsmvozJ2hlq//jhZNHnurhzGNt4w7A GH4n4wynqIdieggoJBLVJ6ErY4SIvop/3G3BXB++Q9XgHMtnM+XfYb1cYXZuIQWTJKmLjX5Iwk0 XsCGCXyWWj9ciiwT67Adl/cOnovL6SNORwRVR+MIr/UR5BZTveOWwQEEyXKZewp9ucq0Euvm+PP no6zUeieXsC/WHnGd/ODPO0A3PD0d87TLhjz7B2P1IiwNEmcNrRYp03SIy3LlWBcjkEEBTLRzmH S1Mb+AyR8PLvQS4Sb29XYgL9l9e7oOLmdJjD9rMP4M1xRFcnb7Ki+Q2RO8CmI21fyf6pFNusf9a LdyNNrHcTf3EEJi+USDAgOp/hHIK2VXgig5x64lTso76okPx8USqNBnY8uET6RSCjriSCYhYFM6 0zaJmVbJuJfE0cQ== X-Developer-Key: i=keescook@chromium.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 X-Stat-Signature: 3jptob14wu43r4qp1qcu998cz8zd8dxg X-Rspamd-Queue-Id: B9FB91C0008 X-Rspamd-Server: rspam06 X-Rspam-User: X-HE-Tag: 1713994870-543423 X-HE-Meta: U2FsdGVkX192EHliRmGqZ8zhEPrRDvYT+HDcxnwumS8SoYhV/1x4UCmN9oBfcyTx1+sUadewjsit9KDwR0dqL8u8DSJCLVY2ZCoajGfUs0c/QZ9MqF1MKlej5xV6kT8JS7z0egfbUIWSDjb1BKfrmLItgy/manB5XQ3/7PPNCzhCiXO4XagzSAZ1RaRN6t6EeEp10Gu8zssX15EjqE4ioKv1iqvWyLQdFhj+Kc+K7uF7fEFHmGXkZSWY4kA3sz5h00EiiIs/vawcqxnUmTLX8dfzkAtGOJJw69Q1fldqQP0ta1a/Hzacv/tREpNV3mTWIlvqNzljfKd3ACT4FDO6Swtbijen5ShFXhr2bww56rL6pAP109Yolz7H1L23WAgoL0y0eJl50sFd8dIjprU9A6LFyQ3u41pH0N7wj+9Zr0iIt5bBAOtSxeaFpYzwNRvYgNDBs1V7XUiv3CbzM27LILIOjBOBNvkR4SV/JqDFHQVrDlVut5qTyRUJhqLx3LhhxgYI8ukNDCjc01YArroETUcOa9YcVc3VQEEl/DOWaYoX9I94vhfdumBI9GB1vwmiJCWYFKVn89SJfCTqM+wTJI0xVMJWp7mPh6E1KPV3RMV3imijryRm7e641DnM366d8kToJS+4oPAmCDk3grKRwTDGKqNGefPXHr7ldEyZCEwblA8A4rEZa9VrAeeD/VyzghhAnVr0AD+pDTOXJ0FFldpJqPFG4yOZr/RFyw0NKuTy04DP0UVcEuWhjHw56uwlcRRWJ3fxBz1N5vIRvy5LdIv7L5nzqq9F4ioW8+Z8PWbGSbnNF53fFZrH6C4PhTgA3ZagJTfJS4FajRL76/sYSRn0g1QlaCmAu/LYAJ5kltcw1hzSFvXOcuJlLmEhGowcmXseuixYvZbVWjMpQhOWiIZ4dRdd3drflpU/PnHKkWkb0qPN+aIZ5w00lJ6JxPNYiP09PKLxuXivSYj3p7U KeSlZOtL 1VhbcBdniEAt3k1PLN9cdPchi4FquXq+m3azuD6NVZZviZ/9N+J5CVsvDNaDhGAdecvY51nKKgzkLqkcRdmK2ZeQ4hQey/7Zu+qmwq08ziNWDYDJUyDrnLKLi5B6dohbxdMgQXpPCGiBkjwxNYhnuO6bJv/sesnGxtfiI+7o5WnCJRdaLIOjOrm44AJPKf3Kx2lNgWXAYmvkMPgG/VM80wxkwDLibvK1rxlebuNL7s2SJZV2rYlvNMdtXEJMQA67Ck2pxp2J1ZRZwi9TCRVk32wDJCJSOfNVSbvGvjcIw2ZJKNHUFZAPqTbMoLd7vxrVKOZRa4Waatnmoa/MJnPkgsoRFQyQS/OmozMa8WbhePzynnmglrcJfzW+Gi4IKEOWCeJCHeyOa36FIpLjaFmkenfWDrEaE4fmD6hBx4EoZ9cgD73SrnHozRt0UmGh4bpEraIaTI2oJUTzdxJsmN7WckuLG6r6hFPuw2NvHANKUQHegnYFuL3e9BAGGKkRLVTIuZyK9erA2lsljCEAFs1lgNy64vOwS4CzyWke3QHV01d24JpJFKBW8ZNpG54w6MWqzHmCXFfNxcC62wcDMXW1OjhSaoHRSmai9quGXaFdmTbPq+zHBjD31h2KdLxg2ebFUH4S5oOVasU/1lD2+6QV3asg5FSeCyL5QHQl/nkefv22UZfTxZnr1UTcsz9ixYSb4j403cYB6+Q/KVXVfEx8DFS2McBCVCAeWvoRKJfxT36LqhBGDUaB1mGF9GTOH7Wh7wYhhL4zxDTJm1dLO2luI0jRK6ioeVqPmDaTSiLW0LvVDNUrv74jL6xMm8tlnSHp3UmqyhL5oVh8xpJNX61S1FHMLHKJM94zECIRBohWUvhch4GMe0OWr4Qxka1cra1yKHNuW9WtVzCu+C0DZxq3KsIkXFBYUFDI0lBaPixvrUiWRndeoZZXTtiL82nm5XtWJgJMV X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Dedicated caches are available for fixed size allocations via kmem_cache_alloc(), but for dynamically sized allocations there is only the global kmalloc API's set of buckets available. This means it isn't possible to separate specific sets of dynamically sized allocations into a separate collection of caches. This leads to a use-after-free exploitation weakness in the Linux kernel since many heap memory spraying/grooming attacks depend on using userspace-controllable dynamically sized allocations to collide with fixed size allocations that end up in same cache. While CONFIG_RANDOM_KMALLOC_CACHES provides a probabilistic defense against these kinds of "type confusion" attacks, including for fixed same-size heap objects, we can create a complementary deterministic defense for dynamically sized allocations that are directly user controlled. Addressing these cases is limited in scope, so isolation these kinds of interfaces will not become an unbounded game of whack-a-mole. For example, pass through memdup_user(), making isolation there very effective. In order to isolate user-controllable sized allocations from system allocations, introduce kmem_buckets_create(), which behaves like kmem_cache_create(). Introduce kmem_buckets_alloc(), which behaves like kmem_cache_alloc(). Introduce kmem_buckets_alloc_track_caller() for where caller tracking is needed. Introduce kmem_buckets_valloc() for cases where vmalloc callback is needed. Allows for confining allocations to a dedicated set of sized caches (which have the same layout as the kmalloc caches). This can also be used in the future to extend codetag allocation annotations to implement per-caller allocation cache isolation[1] even for dynamic allocations. Memory allocation pinning[2] is still needed to plug the Use-After-Free cross-allocator weakness, but that is an existing and separate issue which is complementary to this improvement. Development continues for that feature via the SLAB_VIRTUAL[3] series (which could also provide guard pages -- another complementary improvement). Link: https://lore.kernel.org/lkml/202402211449.401382D2AF@keescook [1] Link: https://googleprojectzero.blogspot.com/2021/10/how-simple-linux-kernel-memory.html [2] Link: https://lore.kernel.org/lkml/20230915105933.495735-1-matteorizzo@google.com/ [3] Signed-off-by: Kees Cook --- Cc: Vlastimil Babka Cc: Christoph Lameter Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Cc: Andrew Morton Cc: Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: linux-mm@kvack.org --- include/linux/slab.h | 13 ++++++++ mm/slab_common.c | 72 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 85 insertions(+) diff --git a/include/linux/slab.h b/include/linux/slab.h index 23b13be0ac95..1f14a65741a6 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -552,6 +552,11 @@ void *kmem_cache_alloc_lru_noprof(struct kmem_cache *s, struct list_lru *lru, void kmem_cache_free(struct kmem_cache *s, void *objp); +kmem_buckets *kmem_buckets_create(const char *name, unsigned int align, + slab_flags_t flags, + unsigned int useroffset, unsigned int usersize, + void (*ctor)(void *)); + /* * Bulk allocation and freeing operations. These are accelerated in an * allocator specific way to avoid taking locks repeatedly or building @@ -666,6 +671,12 @@ static __always_inline __alloc_size(1) void *kmalloc_noprof(size_t size, gfp_t f } #define kmalloc(...) alloc_hooks(kmalloc_noprof(__VA_ARGS__)) +#define kmem_buckets_alloc(_b, _size, _flags) \ + alloc_hooks(__kmalloc_node_noprof(_b, _size, _flags, NUMA_NO_NODE)) + +#define kmem_buckets_alloc_track_caller(_b, _size, _flags) \ + alloc_hooks(kmalloc_node_track_caller_noprof(_b, _size, _flags, NUMA_NO_NODE, _RET_IP_)) + static __always_inline __alloc_size(1) void *kmalloc_node_noprof(size_t size, gfp_t flags, int node) { if (__builtin_constant_p(size) && size) { @@ -792,6 +803,8 @@ extern void *kvmalloc_node_noprof(kmem_buckets *b, size_t size, gfp_t flags, int #define kvzalloc_node(_size, _flags, _node) kvmalloc_node(_size, _flags|__GFP_ZERO, _node) +#define kmem_buckets_valloc(_b, _size, _flags) __kvmalloc_node(_b, _size, _flags, NUMA_NO_NODE) + static inline __alloc_size(1, 2) void *kvmalloc_array_noprof(size_t n, size_t size, gfp_t flags) { size_t bytes; diff --git a/mm/slab_common.c b/mm/slab_common.c index 7cb4e8fd1275..e36360e63ebd 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -392,6 +392,74 @@ kmem_cache_create(const char *name, unsigned int size, unsigned int align, } EXPORT_SYMBOL(kmem_cache_create); +static struct kmem_cache *kmem_buckets_cache __ro_after_init; + +kmem_buckets *kmem_buckets_create(const char *name, unsigned int align, + slab_flags_t flags, + unsigned int useroffset, + unsigned int usersize, + void (*ctor)(void *)) +{ + kmem_buckets *b; + int idx; + + if (WARN_ON(!kmem_buckets_cache)) + return NULL; + + b = kmem_cache_alloc(kmem_buckets_cache, GFP_KERNEL|__GFP_ZERO); + if (WARN_ON(!b)) + return NULL; + + flags |= SLAB_NO_MERGE; + + for (idx = 0; idx < ARRAY_SIZE(kmalloc_caches[KMALLOC_NORMAL]); idx++) { + char *short_size, *cache_name; + unsigned int cache_useroffset, cache_usersize; + unsigned int size; + + if (!kmalloc_caches[KMALLOC_NORMAL][idx]) + continue; + + size = kmalloc_caches[KMALLOC_NORMAL][idx]->object_size; + if (!size) + continue; + + short_size = strchr(kmalloc_caches[KMALLOC_NORMAL][idx]->name, '-'); + if (WARN_ON(!short_size)) + goto fail; + + cache_name = kasprintf(GFP_KERNEL, "%s-%s", name, short_size + 1); + if (WARN_ON(!cache_name)) + goto fail; + + if (useroffset >= size) { + cache_useroffset = 0; + cache_usersize = 0; + } else { + cache_useroffset = useroffset; + cache_usersize = min(size - cache_useroffset, usersize); + } + (*b)[idx] = kmem_cache_create_usercopy(cache_name, size, + align, flags, cache_useroffset, + cache_usersize, ctor); + kfree(cache_name); + if (WARN_ON(!(*b)[idx])) + goto fail; + } + + return b; + +fail: + for (idx = 0; idx < ARRAY_SIZE(kmalloc_caches[KMALLOC_NORMAL]); idx++) { + if ((*b)[idx]) + kmem_cache_destroy((*b)[idx]); + } + kfree(b); + + return NULL; +} +EXPORT_SYMBOL(kmem_buckets_create); + #ifdef SLAB_SUPPORTS_SYSFS /* * For a given kmem_cache, kmem_cache_destroy() should only be called @@ -938,6 +1006,10 @@ void __init create_kmalloc_caches(void) /* Kmalloc array is now usable */ slab_state = UP; + + kmem_buckets_cache = kmem_cache_create("kmalloc_buckets", + sizeof(kmem_buckets), + 0, 0, NULL); } /** From patchwork Wed Apr 24 21:41:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 13642532 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 02219C4345F for ; Wed, 24 Apr 2024 21:41:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 95FD36B031E; Wed, 24 Apr 2024 17:41:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 84E7E6B0320; Wed, 24 Apr 2024 17:41:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5DAB98D0031; Wed, 24 Apr 2024 17:41:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 41FAF6B0320 for ; Wed, 24 Apr 2024 17:41:14 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id EBA0B1601EE for ; Wed, 24 Apr 2024 21:41:13 +0000 (UTC) X-FDA: 82045746426.09.E2691A3 Received: from mail-oi1-f176.google.com (mail-oi1-f176.google.com [209.85.167.176]) by imf09.hostedemail.com (Postfix) with ESMTP id 0BA38140009 for ; Wed, 24 Apr 2024 21:41:11 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=LTrvwCwq; spf=pass (imf09.hostedemail.com: domain of keescook@chromium.org designates 209.85.167.176 as permitted sender) smtp.mailfrom=keescook@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713994872; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0dfA8VHRufYC0/a/WgaXWD6BvTAtlULzh1yIW0W2iCw=; b=m9jMypKHqxG9f/mRrc4eOzQM1vE7qo+0R99ld0RiSllyU5yVS7Sl3D6rSg179ZIBJpfjdL 47ZOYB/10KK9Q+LqiXhFytlqmTqOQ631RzyLpyxlO4kvnEIBdZ1hJVTYI6lFl2Q48aBqzA uE1i+DEuAKw9gPmkv41KN6jgXWILjyk= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=LTrvwCwq; spf=pass (imf09.hostedemail.com: domain of keescook@chromium.org designates 209.85.167.176 as permitted sender) smtp.mailfrom=keescook@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713994872; a=rsa-sha256; cv=none; b=YZHo59M7XLTL65l7ijUzJPfkSvPndHKditM2IUYOcKafFQ5rLioAfG5jrazSL1Xr204mP+ yyY9eWNg4CcQgC2motHfttL8QlPYvNrD/w2Pn+tYO/gBEoTk70sjCSzkkoezRKj8CnFLDl gqC6CJ3o7G0/ohFRn+aIpP6kCAmRQ+I= Received: by mail-oi1-f176.google.com with SMTP id 5614622812f47-3c74b27179dso225570b6e.1 for ; Wed, 24 Apr 2024 14:41:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1713994871; x=1714599671; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0dfA8VHRufYC0/a/WgaXWD6BvTAtlULzh1yIW0W2iCw=; b=LTrvwCwqwM4PBp4rdedyPVGfEB2GqpuC4filpo5Y+v+7v1xz2NRGDkBT/BSIadHtvo 0+I6iMjmQMddB5YbCyjcnv8tJ2AwcyyWQubCkO0Z+chqoPClzXV86n2qc6sAS8bVAAMo sN354OesCyBcIHgTZNF0+VnGq0/EgxVYrMNTg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713994871; x=1714599671; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0dfA8VHRufYC0/a/WgaXWD6BvTAtlULzh1yIW0W2iCw=; b=sp17+irk3mB5oka90VIjs2hPE6GOQyegPvwsa84mfms1sKEP28DndXYDOY5HdcoAcs frdQHoIbZXrh8aa7dtDj+vJGc4VyM2H3VgZLIu1ij5IQ6yK6Y8leIAidaU+hRVRzUouu tUoeTj1iHz8SJYneuKiqKqvZ93HLTME0pf8GxYLaa5SzOa2aW1zXyyXNA/SxTPcpedu3 axDjAVRTYoVjEgWXJ0Z5tB2L98S/gGn6pzu7cnjqvhtBFTnCKR2V+D1aeNzx0bcRaj44 Lh+UXHuPq7cNFGvdxVsrn0hzwfbals0yaC6kUHoNVZkA01hvfZE0IvjkEUdFNTbkfgzQ uqXA== X-Forwarded-Encrypted: i=1; AJvYcCV0DAvwSWS82lXGss96Tumro3amY4RGBEpwAYyLIV16wWPGzqXp7lZCaqDZrozMkFesoxoEcu6VVXG94XbvOHnd/cs= X-Gm-Message-State: AOJu0Yx969D1XhwrS+moqJHoJmTI4Akmx4HhUz/kiWaxOBl9fhN3W6zS zlFPmLxa+U9SmjsdnesyJyjygtkzSoQI2P3AloFfoHibxytD2+gJDslPHG60lA== X-Google-Smtp-Source: AGHT+IGTuYDjFaVfAgXkNY0pfv1mQjL/tXjniKZW/5Pacc5OQoxVYohiXozhOD8xw9pwqeWMR48vVw== X-Received: by 2002:aca:2201:0:b0:3c7:3b4b:a0c7 with SMTP id b1-20020aca2201000000b003c73b4ba0c7mr3979160oic.11.1713994870911; Wed, 24 Apr 2024 14:41:10 -0700 (PDT) Received: from www.outflux.net ([198.0.35.241]) by smtp.gmail.com with ESMTPSA id a38-20020a631a26000000b005e4fa511505sm11445807pga.69.2024.04.24.14.41.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 24 Apr 2024 14:41:09 -0700 (PDT) From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , "GONG, Ruiqi" , Xiu Jianfeng , Suren Baghdasaryan , Kent Overstreet , Jann Horn , Matteo Rizzo , Andrew Morton , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Thomas Graf , Herbert Xu , julien.voisin@dustri.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-hardening@vger.kernel.org Subject: [PATCH v3 5/6] ipc, msg: Use dedicated slab buckets for alloc_msg() Date: Wed, 24 Apr 2024 14:41:02 -0700 Message-Id: <20240424214104.3248214-5-keescook@chromium.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240424213019.make.366-kees@kernel.org> References: <20240424213019.make.366-kees@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2189; i=keescook@chromium.org; h=from:subject; bh=KdH6ME94d/1vtX6KQzOA990YXFPl16sDmJgA9NILnME=; b=owEBbQKS/ZANAwAKAYly9N/cbcAmAcsmYgBmKXxufNt0iRDMEi0UPkW2GLKPPmjE/aN5Rd00e n0cfMEJ/H6JAjMEAAEKAB0WIQSlw/aPIp3WD3I+bhOJcvTf3G3AJgUCZil8bgAKCRCJcvTf3G3A JkdID/sHJNQORdFCM/cUq/ZcYZpyyvQIcnzDV0Z965jFp87JwowP6i4oI+FBo9uCykMh+11Rrlu NQNcUFmJmo6LQpfaqi0B05b1CRJ477vtFjlz2E7iovHNz3HYXXQS/vzGQ/VQH/Sv0F4EzoSOn1l UHfroyky4aryqJjcJX6ieL75C9QED2YiAGSVez2BD9J6JQGhA43blUULYTI434b7898CwOpbG46 patNilg7fBIku4FUOr2K59si/Pzdtj99qzMLg3XRQMJYclitl++HUzWC7Dxg32YdMfqNiX2aQQq svnMXyh8tO9LQ+fNnIm3P+frmSHbfPjtC4CDynsNU6cCbfvmxn0HIY1Wk+nYHcEB63BSmjNca+z lCmwiBUp8svDj9Xh6LO9/AWzHR8ag+sXmENKUaWLGO3Z41XoSWwFir9WU5GzjDKKl9t82MfijIy QevedZy6odm1fVxYoObyDu0WndH3xDKEKIPaz6I0Fun46Z0Itu+gz8v/gwFw5nYTDhFyTfXqucx UX0IAG/fElAHOxq2HNfmsTXeBXIFhtGjqJpBtFTojR1EeuQ6rsOhrJZl+x1KJAMztGf7xR/2lRm ZQ4a+ovdgDGs998x2Xl084sou5A9U0NffnvW9SXI5UKYquH4wK1z+qtZghjKn8ArTIPCB0gBzx+ fIU7zdwHzbc6QFw== X-Developer-Key: i=keescook@chromium.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 X-Stat-Signature: qyzmsp5hjug79m1wi37dn5zq6n7qj94p X-Rspamd-Queue-Id: 0BA38140009 X-Rspamd-Server: rspam10 X-Rspam-User: X-HE-Tag: 1713994871-710494 X-HE-Meta: U2FsdGVkX19J/Kl2MmTpv6fn6JhqOU1WazIHCfuF/8KoQsESh7YCKkoUwZjL5ajLRPgMdXq2715KtjErF+OTtOba3prFCJOSc1erfidKVGvYYT7S65x2VE2Y4/8fQKf95mndenlrlD3ZEiDJD96kqc7x6JXLIdET0A7Mo8UVLjJVu6gRstpYDHCxUv36KLg+CZF/BDwr6Lc3tOM22jNtVagmU5HNK7sxpetL46nH91m6mC2eboPUuRi5zzvDkq3GxSwyQbhhiNBBfA4Eu4Ga6M7TXvDiyRn0RhZikQyGQu2Nnnvrnu9Gmu640hBUavsJMC5+3tccCiVvGQa2swj3NW+LANxO3VjBeRxZGOHrucGafoaRDv5eEf29WSV1NRfINPrK4fwFfT9sKCkAGC7UDOT4LJHUaUekMXdCKP9AupQqP1rp6H5mwp/eyFpxw/bWMyz8QtDTPO2IHDVeagtHTZVNx1fslq79omKbqXYqovE6mjhX0FTUsCj3iNzWkY9ElPpQCAxntT0LkNsOyxS2H6/hifH2LLcGhlYWHW23K8ODnpWMbP/ov6NPSBLfywLbw8RdsXdFEPMYQC0EjBGRrAPR4j3u/JnjZUog/zKEUSnMxzP50dwhMiEjUU2F9sIcSk3uKT1AYNpo5OAzjxerEkTBKUlN3t/5DKmNGutxPMQaPDxI2AzbRrr4r/ml1M6Y8JUB7vUD8oKcVFd4GKV/KAYdVukakMvYFCw6WC5521+9iCswTFWSoJ6JXWtvR5kGRIof/s/Bj6aw+EnrVAtFUANmo0Bf0mnIk0vvK3QOKWOBwt+RRLxJ6KXgWgMAsn1cD6t9FkXpJtNjnpVF0tKCZFReDIi9yc+XOLjLmgrzYbglN3/y2xc8SkP5lthWhf8f94051psQBKmwSkD1dGnlGbyqOZwMbPDJkQVYuBeG/9pCZIBFvNqtyRzbvTXF8SIfua88NKGdz2RWW2421Vo +l7ryvU8 pJnSUkP2CGtPIx5E+3d0iponX2Uhn3J6dFNAJ6HI6occ7SQTTB7NGAxiVmVux4qm4KhEhK/MXTyP7QhRD/qwdlePtA+xu/pno8YpSA2PSE1HBMG77dP5zHb53uuzXBU8Pqn8XNLJocs4ZBVKnDN8Ri9QtaxM9OTA0L+1gmsV0SCRAj3CpSBKly8ghMuqDA+u8JPY8kMlQ/CL0Hbo1wx4ePUCiMNip6YhsX25j6Tid2HBKW08wDc4kVls7T9Pvt2NMZDwS1Qbzs5obBwoK2Van41+pX+eimxsT8t6oUXxaypo1SDSLIW2lhdG+BxlumLeB0QwX8VPu7NYpxsHXdZbdXB5AUFcOuc/bvINT7N7s/DOyfRVesbIt26n2NUvRgNjXrUen6XVBgATqsqudsVISQLhd0tfJVXxmHq2b7gw+Ga4Dni/qqTb7SKVSR6b6cLhdO0fDFCPL50PZV2kt0duP7+IH2ZBqTpBgu6KsLKYvqz/CaeZuVWcXr2EVMeklNZNSPiyKUOZOfocuZUO0FE2ifBWRcRdjp/cp3JMd/nUcQTI0iKIEDX5iB77Ovtrv+O2AjP6PzmLRJHFFelDZyI2MoCs4p2eW8HNrX9lCDYxeaR9D3u/mTQcsv0bUfvKR/VIKdLM9j8qmDCXvge5UX1AEutQwwYu0naLWBXr+Hzbm/zeNRic5U64bCRaJLFYsIhU+IGCztOwelkyvuSXBpyvsjEZNrVbLptyNxokz6mvFB40JPTBIHzhn0FCfTjp3TOZCQzVdgg4HGFmkXZscZ/hRkHnKgzezIQR7yfi1Q4mTVHbNwyFQqutPdNkWA6okEneyA6bwti/rs+tmhiC8gsAdjhInJ4ORfn1YYEgO0EQVHMxEYgK07kU/fwO71CM8oqVjH1EF4CK+BuCe6GGYda2aJ7YlKvDSBlG2YW+DiP4eJy54ceqDq205QfEK6wAH3HcTmaEziE829pkHH6Y62U7B7tjhFDTl WEsMvIjt bNdgH94Z0hCDvaSt9NVBGEBFrjp4NiIiQgbqc16dKONwCeDH6tjUd7v6vElEp66ugLrqZK3zV+DTTED57Uu2KkxmJsdHLbZqOsXi6yeBpM+3F8oWTbmM50wcv99gf9049xvKrvI4MHCl+p3zxW9QueRrGPNt+dTKm+fus3u7lehcIYwqKXr48rKWyRTULN54OElWSdTa89mW15ctyP9AXdstYeSUr94QOYLiH9iFi3Y= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The msg subsystem is a common target for exploiting[1][2][3][4][5][6][7] use-after-free type confusion flaws in the kernel for both read and write primitives. Avoid having a user-controlled size cache share the global kmalloc allocator by using a separate set of kmalloc buckets. Link: https://blog.hacktivesecurity.com/index.php/2022/06/13/linux-kernel-exploit-development-1day-case-study/ [1] Link: https://hardenedvault.net/blog/2022-11-13-msg_msg-recon-mitigation-ved/ [2] Link: https://www.willsroot.io/2021/08/corctf-2021-fire-of-salvation-writeup.html [3] Link: https://a13xp0p0v.github.io/2021/02/09/CVE-2021-26708.html [4] Link: https://google.github.io/security-research/pocs/linux/cve-2021-22555/writeup.html [5] Link: https://zplin.me/papers/ELOISE.pdf [6] Link: https://syst3mfailure.io/wall-of-perdition/ [7] Signed-off-by: Kees Cook --- Cc: "GONG, Ruiqi" Cc: Xiu Jianfeng Cc: Suren Baghdasaryan Cc: Kent Overstreet Cc: Jann Horn Cc: Matteo Rizzo --- ipc/msgutil.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/ipc/msgutil.c b/ipc/msgutil.c index d0a0e877cadd..f392f30a057a 100644 --- a/ipc/msgutil.c +++ b/ipc/msgutil.c @@ -42,6 +42,17 @@ struct msg_msgseg { #define DATALEN_MSG ((size_t)PAGE_SIZE-sizeof(struct msg_msg)) #define DATALEN_SEG ((size_t)PAGE_SIZE-sizeof(struct msg_msgseg)) +static kmem_buckets *msg_buckets __ro_after_init; + +static int __init init_msg_buckets(void) +{ + msg_buckets = kmem_buckets_create("msg_msg", 0, SLAB_ACCOUNT, + sizeof(struct msg_msg), + DATALEN_MSG, NULL); + + return 0; +} +subsys_initcall(init_msg_buckets); static struct msg_msg *alloc_msg(size_t len) { @@ -50,7 +61,7 @@ static struct msg_msg *alloc_msg(size_t len) size_t alen; alen = min(len, DATALEN_MSG); - msg = kmalloc(sizeof(*msg) + alen, GFP_KERNEL_ACCOUNT); + msg = kmem_buckets_alloc(msg_buckets, sizeof(*msg) + alen, GFP_KERNEL); if (msg == NULL) return NULL; From patchwork Wed Apr 24 21:41:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 13642531 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5115AC10F15 for ; Wed, 24 Apr 2024 21:41:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 14A446B031D; Wed, 24 Apr 2024 17:41:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0FB1B6B031E; Wed, 24 Apr 2024 17:41:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E8C176B031F; Wed, 24 Apr 2024 17:41:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id CAD706B031E for ; Wed, 24 Apr 2024 17:41:13 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 8288F121197 for ; Wed, 24 Apr 2024 21:41:13 +0000 (UTC) X-FDA: 82045746426.15.CC7265A Received: from mail-pf1-f169.google.com (mail-pf1-f169.google.com [209.85.210.169]) by imf05.hostedemail.com (Postfix) with ESMTP id 9A762100017 for ; Wed, 24 Apr 2024 21:41:11 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=Svow4mpd; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf05.hostedemail.com: domain of keescook@chromium.org designates 209.85.210.169 as permitted sender) smtp.mailfrom=keescook@chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713994871; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=AHwyZxYzmrEMajI8SxktaczkjAAhwGi1bdDM+8q+7+o=; b=wp+ZBY4x3cJA1yiSk1VIcnVf+OTuQNjC985mmnFJQB8uQwt9pjWK0bT/Zy3cQMIvkf2XKN ZvFSgqL9nrH25dfYqS9ZAq0RdsvbtbmMBfjt9zLcxUMtkg6tyGqZm7uk14rSDd6zt5NRbL IsqTP+GaufCoCsn1az9OOR9CkkmRs5E= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=Svow4mpd; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf05.hostedemail.com: domain of keescook@chromium.org designates 209.85.210.169 as permitted sender) smtp.mailfrom=keescook@chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713994871; a=rsa-sha256; cv=none; b=3jDkSj2l4Iye2igsxOg45l10R0ei5cmmq7d9cw9RJeXCAwFOUeymTJjPk4YSft5mYAVzmR mCWjFjRMqw12hxEJiTyiGTapeUn0XZT0yuPkGb4eTWQJCWjGP5kp5rN5B0Ig8Pa2ixrjZh 7O7FG38B6Kd8pYpDDYt7CfqgRtAsgV0= Received: by mail-pf1-f169.google.com with SMTP id d2e1a72fcca58-6ed627829e6so440115b3a.1 for ; Wed, 24 Apr 2024 14:41:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1713994870; x=1714599670; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=AHwyZxYzmrEMajI8SxktaczkjAAhwGi1bdDM+8q+7+o=; b=Svow4mpd1p32K3JnWurAYYlODdC2OERZ5jkUoqIx1D/iBbtGvX1SBPwpu2tiEdlyJL IVwxC7dHE9hE/LVu64wUWbVM4BPv0cFqa2Om/IJpj8dG8LbPD5F+6+F5WOUvErAB3E6M JCc7C0jG18+j4zv0S7uuQPQptVr0nO5VA+0gI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713994870; x=1714599670; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=AHwyZxYzmrEMajI8SxktaczkjAAhwGi1bdDM+8q+7+o=; b=DtZL3vcfNHCR6glQ/+eXWu2EhYYkzo6ikpYHJdQmqSCNLe5kQxmERvzjY71WtMAFP9 KmO91voZjJ0meTnsauKCjVdNgZRvSQa5m8A9Ub7H9A9O9jh9Z4E9v1MqEkVyxVw0zzsR IUDopVe8XfB1nTvNN7PhWcbCVYMh6uKn7mDIttXrwFSRt0Ig7A0XOXm3ve1JFdrZ++Dp D71cJEnTulHi3umLUZfxtVqFaDbT/0iMtlLyol5AE2Tu0j+C6soqzFZ1i5lRNeowIcNU a1NBna42Qc40uMaGdbudPxseks9OCz3zubGRRIDjqH8iPr6hUrtGiMtHcbw2SZQIFy+c 4aaw== X-Forwarded-Encrypted: i=1; AJvYcCWPtO1bjhFv+1sM9Z/YzNT1fqR7IRe2gvbUXXuaXh3QgRUD9Kp9W3U6Xh03ZXqbLDuq0xoYucCgYbMCd9+Y440Xl/k= X-Gm-Message-State: AOJu0YzBhy+vsWDfZh/l4Mww670l4dY184QQ5RJEIOGVGymPF/Rcxlea cRL05yC9zCGw2B9AvTxlOnDH7g3ldR4cK1tfdn4wOIJk8B0LT5/qcXHH2lZ6Lw== X-Google-Smtp-Source: AGHT+IF+6ZBq5uRyKg/lyeaV7Zfjb1C4ubPI1uf/1ezK3wBx+DQpcIBEs1vcTlhgEw/fCjnkutcopg== X-Received: by 2002:a05:6a00:3909:b0:6ea:c2c7:5d6a with SMTP id fh9-20020a056a00390900b006eac2c75d6amr4392756pfb.2.1713994870491; Wed, 24 Apr 2024 14:41:10 -0700 (PDT) Received: from www.outflux.net ([198.0.35.241]) by smtp.gmail.com with ESMTPSA id d11-20020a056a0010cb00b006e685994cdesm11863906pfu.63.2024.04.24.14.41.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 24 Apr 2024 14:41:09 -0700 (PDT) From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , Andrew Morton , "GONG, Ruiqi" , Xiu Jianfeng , Suren Baghdasaryan , Kent Overstreet , Jann Horn , Matteo Rizzo , linux-mm@kvack.org, Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Thomas Graf , Herbert Xu , julien.voisin@dustri.org, linux-kernel@vger.kernel.org, linux-hardening@vger.kernel.org Subject: [PATCH v3 6/6] mm/util: Use dedicated slab buckets for memdup_user() Date: Wed, 24 Apr 2024 14:41:03 -0700 Message-Id: <20240424214104.3248214-6-keescook@chromium.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240424213019.make.366-kees@kernel.org> References: <20240424213019.make.366-kees@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=3141; i=keescook@chromium.org; h=from:subject; bh=b6S8APzjeiOvxh7QXxGMfpkIi0N6Ek4SYDm0EuC0dW4=; b=owEBbQKS/ZANAwAKAYly9N/cbcAmAcsmYgBmKXxvnemB4TZWw6/SnmxaAlLoQIy43uGWllQev OnPiAH/9CKJAjMEAAEKAB0WIQSlw/aPIp3WD3I+bhOJcvTf3G3AJgUCZil8bwAKCRCJcvTf3G3A JgDRD/9furWdb2wMJZeJdepRap6uPYO2c5ds4IQAE7tziQMuuSmONplOC3rkp+YC2fB+nkaxPWE d5SxYD9e0M/84q90AW2Ti0nh0jvfTtNMZ0w2uIaVV00CyQNhiPlE3NnR373AMGa7YCotWcpcwyX NCGb/Um52ghKJrB7GUeqADeH51g6QIiaQ5A+61wh2r58IOUBXzWTViSlBUz7tDW/wxytYUO0xLx MBsozAddCGBLWZde0yKTX/DhJtVJLK9QN57x8/gGzPrEbm+SwyhxWIlVSOFZCXIb1GmE2dCKP04 WKTMHUL9wBBDdVJHEAezmq1VUNjqrMAltoUhsU68zHrBMOZ0xALfArySjrQgYOZ/s/LhSN1eixy YB2HnwwW1Ob7iRyos9jHaqZuUfPhr5HkR1Xt7W48XR/orPhpxKNcC56bEVdjiNDqF2nH+7T6Dw1 wdT983ZF/WKKXFcuG9HEuWc/berdLOmFwtWLZ2o9u/HgPXUPUYUZ3y6nO89h5f63BoqgV9Pezyf VBHjGDS635QY8qwXZDBkeh/KqpQ6PKclipOmkFCvAKAWYzxi7Cae9uKfTTfmYrBkCvyAQ6TmATh nsSnp2IGyAwwXBGuXHvO3DSd4XidX2nYJrNw/qYXoDpZvebUgj46ue7R7B/WpnH8rgDo7mlnUJJ T33evI365HcKYwg== X-Developer-Key: i=keescook@chromium.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 X-Rspam-User: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 9A762100017 X-Stat-Signature: 5uas3wjofhdmmib3ifwkm4u8wktejgnu X-HE-Tag: 1713994871-698226 X-HE-Meta: U2FsdGVkX18Yil2v7hvSdHXZD7/X40OIPJg/baKwtZwmviYWnCbxvJr5t4aDr+Roo5pcjz/98UZel6bTPzZm3qWjSLb1DMCStSD+g/j3Yj3UvA6jknFvdahuXqgQIpSRR16L3nwDs/USv/Fg3eYnh3M93N7JmtInYGeD29G97rrM6sOcOJYASVDi/nW7gnByNINoWjfyN8hBatYQWrcUYax5Zv/RlLrvVqxdMdtikD6NM8LSE6irez5ECJg6ftseiIs01wCicrCpPcd5mBC/HIypa+P/+EEG2XmJlTUjMUCdRLGTg3TkFgNC6RH4jY0ATA4d2U9GLeFach/pk4E/J2W9n9GJANWlYU3fN+MyvOSr44uJS6X1hQZGJSuOwctUJPZlanK+y4ysfMNBtB72nBxq80uDeipC6hlsSA3bmp/OL5e7ZHH1/jCWMgw+8FGRV+8R8gnI7SpZbmbcMmJhLLeRmI+qN52sAh2dLRkQfTQkzKZFVnp6iHzkozrUaT2sFr9H3A7mOzogTpImtMjDoY8VPkIUK9BpXHE4CV7vjPiI+8esBbl3s/HyUUMxFRQLjgUSvjRoK/TvkVCtYH9u5GsWFG/jNfdM5bE92fjdzGkicbvvFr0O1x97KLcAhg32aitAXRBYnwiNg+K8pnkOUS/rKfZxI8XjuLNWBvhW7Ce99DRSyRZBdeyrm/FWz438Pcjg+pbAZfZAnRnLFwO45itO8VLMhaIfmt1tl/mm90KK0s6iTgQEHwCQ6lDV9mlnVU7CgaoRQZgrHaxlPcwlR7C7XuLt//g5/ALcnyo7MFeojz+krOxtG0V4Nb5F8gFKUyOqntnk84Nw/fX0a6/x9CmOuTsw0mSOiZweJaSFLrRujBszKPsejRxa1R7evc+PBAPoidQhHQkDNk9xkF2tb2RYXVr9BHNjaWV2BO3CsyLOIUMBXISBA2slqfQrysw8MbdwkaXUOsXuRgOcrYM YZeqtTef uPjg/9gnjYXw7UvcGs5mjvsNbA5t0I+B6xgFGnKX8hNJOtVGpReracmDFLOTXvS2hzNSE7V6y3BDp+w58HFFE1bbcHfE28ct8s16KqD9CPnjH0Gc/KWdhGkI6ewm9WgNYS/QWlAmenPTsTTJAUuvcOnpfEmU2cZFog8dgkyFM+rXLYqQ4+0iQ4l+70+joPZF8/APCFA807v9HrAadw85QEgoowx8pjnwSkSP/SS2AKvcAF3pvUp/TCaivMwYZspz67Cr4zHelqtGuuMZZnjyaQ0O6x3b6LNil9cFgH2aE51FXa0cBGXr7roohMzwdAuDRucXqq0+2KVX0kdVYxMtz/QHVRv8Zoda7quGbdcBN5El7Y1PA03uiAxR2VGtrjWJLhbAGwYxhdXU3AIZCXjUk4mjKCukKxg2WeXHAdsSz0ESrYiz/q3cvuMBlo+KwzdbpgEpR9lOUMo9ASPEmQwnyzj838Qp3EqKprRLCeHUVVFON8ZL6KkF43au7JhHPfQjGuEaNUK3iQmHMoc5zYUNrf5+YplfvaTs6kuY36UGy7fmPLnExyG4rmEy+vdH7iidzhAw2uirKsyM8hrmijgFi66HvasRzoqempnorwYwBxV6blFijcu3uG2NLUW4PO7Tc+f2LKgdby1KyaIQx+RGa7MjeknneKurADzi6g8dEBjouCzUnqeswYWOYTODIEUCt0+KFLqouoQz22+kic5ReyiOmm0CgI0tWYQ4nBBQ2nwgjLQEb/8kW4myOdLecFP0uqPznHlDoNkGeZNY/sQzH55npXjHOXR+pmO7x2WyN/N4xQUHVkRW3BSOWrPPnlr2DdQlRvcbSXRzFSxrqKKyCLGIkdKLPXfwlRPXnen3NPenLLzL0fRdqtGDRmMhOvmghlkfhgXNA601jeugfjg8a4C0tj5LlJCXbLW6rPNsCCkeMjlqiu2rlPP0MzxZ9CuyVTcaDngn0fElRcPPd/PFBzLGBngsH EBsAxfPR tawgNoJZnP4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Both memdup_user() and vmemdup_user() handle allocations that are regularly used for exploiting use-after-free type confusion flaws in the kernel (e.g. prctl() PR_SET_VMA_ANON_NAME[1] and setxattr[2][3][4] respectively). Since both are designed for contents coming from userspace, it allows for userspace-controlled allocation sizes. Use a dedicated set of kmalloc buckets so these allocations do not share caches with the global kmalloc buckets. After a fresh boot under Ubuntu 23.10, we can see the caches are already in active use: # grep ^memdup /proc/slabinfo memdup_user-8k 4 4 8192 4 8 : ... memdup_user-4k 8 8 4096 8 8 : ... memdup_user-2k 16 16 2048 16 8 : ... memdup_user-1k 0 0 1024 16 4 : ... memdup_user-512 0 0 512 16 2 : ... memdup_user-256 0 0 256 16 1 : ... memdup_user-128 0 0 128 32 1 : ... memdup_user-64 256 256 64 64 1 : ... memdup_user-32 512 512 32 128 1 : ... memdup_user-16 1024 1024 16 256 1 : ... memdup_user-8 2048 2048 8 512 1 : ... memdup_user-192 0 0 192 21 1 : ... memdup_user-96 168 168 96 42 1 : ... Link: https://starlabs.sg/blog/2023/07-prctl-anon_vma_name-an-amusing-heap-spray/ [1] Link: https://duasynt.com/blog/linux-kernel-heap-spray [2] Link: https://etenal.me/archives/1336 [3] Link: https://github.com/a13xp0p0v/kernel-hack-drill/blob/master/drill_exploit_uaf.c [4] Signed-off-by: Kees Cook --- Cc: Andrew Morton Cc: "GONG, Ruiqi" Cc: Xiu Jianfeng Cc: Suren Baghdasaryan Cc: Kent Overstreet Cc: Jann Horn Cc: Matteo Rizzo Cc: linux-mm@kvack.org --- mm/util.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/mm/util.c b/mm/util.c index bdec4954680a..c448f80ed441 100644 --- a/mm/util.c +++ b/mm/util.c @@ -198,6 +198,16 @@ char *kmemdup_nul(const char *s, size_t len, gfp_t gfp) } EXPORT_SYMBOL(kmemdup_nul); +static kmem_buckets *user_buckets __ro_after_init; + +static int __init init_user_buckets(void) +{ + user_buckets = kmem_buckets_create("memdup_user", 0, 0, 0, INT_MAX, NULL); + + return 0; +} +subsys_initcall(init_user_buckets); + /** * memdup_user - duplicate memory region from user space * @@ -211,7 +221,7 @@ void *memdup_user(const void __user *src, size_t len) { void *p; - p = kmalloc_track_caller(len, GFP_USER | __GFP_NOWARN); + p = kmem_buckets_alloc_track_caller(user_buckets, len, GFP_USER | __GFP_NOWARN); if (!p) return ERR_PTR(-ENOMEM); @@ -237,7 +247,7 @@ void *vmemdup_user(const void __user *src, size_t len) { void *p; - p = kvmalloc(len, GFP_USER); + p = kmem_buckets_valloc(user_buckets, len, GFP_USER); if (!p) return ERR_PTR(-ENOMEM);