From patchwork Wed Aug 28 10:56:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Brauner X-Patchwork-Id: 13781133 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1CA4AC54EDA for ; Wed, 28 Aug 2024 11:12:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 621136B0085; Wed, 28 Aug 2024 07:12:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5D05D6B0088; Wed, 28 Aug 2024 07:12:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4BF976B0089; Wed, 28 Aug 2024 07:12:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 302946B0085 for ; Wed, 28 Aug 2024 07:12:27 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id CCEAD1C638B for ; Wed, 28 Aug 2024 11:12:26 +0000 (UTC) X-FDA: 82501390692.06.693BDB9 Received: from nyc.source.kernel.org (nyc.source.kernel.org [147.75.193.91]) by imf16.hostedemail.com (Postfix) with ESMTP id 136C1180022 for ; Wed, 28 Aug 2024 11:12:24 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=mHXYuuar; spf=pass (imf16.hostedemail.com: domain of brauner@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=brauner@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1724843500; a=rsa-sha256; cv=none; b=aSKtcVgxTJAjUui9CwOOjmrm+NVVFp68+wwwuAXY9+bCD9AvB0i2Bc5Ztt5CuSwMQ5EKuV 9h7XywApc2WFT7ojXvNMSEu+P1RNqLt0PWrY2syNtf2QRt6b44bb+CLR8naq/wcHGKJACK oWlQOrDDhcQv0KgyqgAL+yxQV2RTOvg= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=mHXYuuar; spf=pass (imf16.hostedemail.com: domain of brauner@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=brauner@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1724843500; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=uazMo0IM558jInsehz70etwPs2IJBI+HgyGqAKmMDI8=; b=t0RNbguqXPDM2lmQ4ffzPsAVfnr/mHvHDXCkg0NxiA/XkLQK2U5Z9QZEz5ngO8To6s4hrS dBeDDQLOlpY41OPdtI+Cp6SCpn6iX0cB9DlHxlKqvLKyvahIas+4Brlu5iQ20hjP7k3lqJ snhbFDJt/SFJu7dLRVhUcJYRRPKLBB4= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by nyc.source.kernel.org (Postfix) with ESMTP id 21FE2A43517; Wed, 28 Aug 2024 11:12:17 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 330ABC4CAC3; Wed, 28 Aug 2024 10:57:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1724842626; bh=inSYKr8Tp4YUwStuHZTh2AxEw7grjX6tn9NgCfW/hs4=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=mHXYuuarQlAW7U9rrFr5eCA110uyKzTvABuqujGMCI8x8yGsinQLw40LlPWMiBwWT twonLhTSp5AybpY0QTkiMOkMefzwRzpN5AKh6kVgB8Xp1EAgmoSEmvOroL2g7oZ5/u bJvL1hi00LZ6gbkIMMgeWgvILzDyFsval9GVKssNlWZsIs/QZ1fwtQ7AZ3Fig9dNkP TB4AsIFcF66lIo5I8XdrRDfDuAShQJutYznIeFzHeCxso+8LxeKYVg45gTwHiIw4/W W1OtJmHigfrW1tf95ru0+eAXyowjppXP32idXQB7hF327uJfteNV/1V1tlWH+Iy7+4 +wTbnjGWZOrfg== From: Christian Brauner Date: Wed, 28 Aug 2024 12:56:23 +0200 Subject: [PATCH v3 1/3] mm: remove unused root_cache argument MIME-Version: 1.0 Message-Id: <20240828-work-kmem_cache-rcu-v3-1-5460bc1f09f6@kernel.org> References: <20240828-work-kmem_cache-rcu-v3-0-5460bc1f09f6@kernel.org> In-Reply-To: <20240828-work-kmem_cache-rcu-v3-0-5460bc1f09f6@kernel.org> To: Vlastimil Babka , Jens Axboe , "Paul E. McKenney" , Roman Gushchin , Jann Horn , Linus Torvalds , Mike Rapoport , linux-mm@kvack.org Cc: Christian Brauner , linux-fsdevel@vger.kernel.org X-Mailer: b4 0.15-dev-37811 X-Developer-Signature: v=1; a=openpgp-sha256; l=1077; i=brauner@kernel.org; h=from:subject:message-id; bh=inSYKr8Tp4YUwStuHZTh2AxEw7grjX6tn9NgCfW/hs4=; b=owGbwMvMwCU28Zj0gdSKO4sYT6slMaSdZ6q50aTDtP2ozaXkW19l83cwCshturj1es4EEVn5O z+yPof1dpSyMIhxMciKKbI4tJuEyy3nqdhslKkBM4eVCWQIAxenAEwk+ynDH564nRNXnDbjOJ3w odOdY9vlgrJ9Qiff8Pys+PdB+8sD0+eMDLMefWv2/srP8eDUldm3ApMDzggEMG7nqP/wZQXX80m pLowA X-Developer-Key: i=brauner@kernel.org; a=openpgp; fpr=4880B8C9BD0E5106FC070F4F7B3C391EFEA93624 X-Stat-Signature: og86kxo9wfht6xeeo98eqwxzro769psn X-Rspamd-Queue-Id: 136C1180022 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1724843544-54829 X-HE-Meta: U2FsdGVkX18SHZImsx3JYpB0hm1uU34TtbhOnXEk2YhVbrPnpxaJMeJc/nkIaVa6iFIUwN8JsatPQFAe7BSjMABlovVU/LS/srjlsc8pbCKSs/O1pb5nGDZEqv6SW8zC0hlQFRrIHKR+4ZZTkE6OVubEsILPSLoLIIUv29ozV6rH4RGZ1z/WHiD6Hfp0A0Py51BWvEA/aPvk2mjiHxXWFsZEaWta3kCxmiqEmcPwXlkP45wV9rLKqaD1v+n5490E4G431OmUj0zyjQ9OEwjwxqKaVd+lCn1CKEVzYqYPHH2U62nXVl+KZs5b10O0y2SsyTsjUSOIBvDer/8yhMmrfku0Jd072wuEqC+rSEAwrfu9OKvo2Xoe11ymKmifBGZGy/AwNB1/z7FaQDv5K9usAT3FP6uZDF2dwQ/aZKx6KvDUCyLoQtJydYYCa4OBBRg4rKp71sgs2Ygt6LjxyPqzdY8goZzFovm64jxXXQTq9Tbcxxwt7d0evFW1q0jHFTvQ/P+yY+XjpG6qcr+VNwioNngVJoIlaUpAzLoly3EzYNNsUZHqGqgg5QoJdsErO/sHzENDmf9WmW1ElkaLHvovJQSgacj+jrMCEJ2AAnJ+70yCVR9lAaz7p1zwsr0CFDS1thd05e95NC4I82Os2QPj6xVrXad3DeJTZb5gNoOinj0czqaRSzSAPcW/kE+ikAY3fsCOx5k9n6YAqTmgnp+GH8KOOwqa3Qte4towThP6ctXn+0IuqV0lE4HvQco6bhsMoUd72DLZev5YXWYMVcCw1SU0rc9UDyb8O0eIYOPb6xqCJJY7de/z9STfMSl9dV5wIcjzqNQsJSiMxSelHldRi8P+5CxWbuBYYi7qukrCssaismjt30T5Ycl45atuW10+XclEt04JStf2CCAt7anNzo9dRLocgTXDynax9HP7ibkSzuWV1NzdhYoDO6szx1fAL8x8Nz9lu5RmnR1Sr87 SrK/HPbI mBho2m3+nQOJALJCEl+iYP2fXBs8XjFsL7YUxNgCNd+kpeMzggptw7TgeNdFjkq0EtokPC2Igl+sToxXU9arvBqTnklZgjrFRRytJoR+Np30Vc3/e5n+whuZfBL2s0NULSLaaCZZ5GAeQzZNTLqwcqwXKnW7m90NWZCy6WoVjPE3uOOWtuMKtlQB07KdecuK0rUO+QrH9HtftZ9ktltDuzczUnB+55OuG6as2w/HUg1CvESM7PKAeGclWfEXhV895tsyTVL7tUIBrc0+hivYskHdyflTFIIf4IG/Ryglg8jBJwv7N5y5czZR/l+/QYXBfugIG X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: That argument is unused so remove it. Signed-off-by: Christian Brauner Reviewed-by: Vlastimil Babka --- mm/slab_common.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/mm/slab_common.c b/mm/slab_common.c index 40b582a014b8..c8dd7e08c5f6 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -204,8 +204,7 @@ struct kmem_cache *find_mergeable(unsigned int size, unsigned int align, static struct kmem_cache *create_cache(const char *name, unsigned int object_size, unsigned int align, slab_flags_t flags, unsigned int useroffset, - unsigned int usersize, void (*ctor)(void *), - struct kmem_cache *root_cache) + unsigned int usersize, void (*ctor)(void *)) { struct kmem_cache *s; int err; @@ -334,7 +333,7 @@ kmem_cache_create_usercopy(const char *name, s = create_cache(cache_name, size, calculate_alignment(flags, align, size), - flags, useroffset, usersize, ctor, NULL); + flags, useroffset, usersize, ctor); if (IS_ERR(s)) { err = PTR_ERR(s); kfree_const(cache_name); From patchwork Wed Aug 28 10:56:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Brauner X-Patchwork-Id: 13781135 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B6337C54EDA for ; Wed, 28 Aug 2024 11:12:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B2F806B008C; Wed, 28 Aug 2024 07:12:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id ADA9E6B0092; Wed, 28 Aug 2024 07:12:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 953DC6B0093; Wed, 28 Aug 2024 07:12:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 771FC6B008C for ; Wed, 28 Aug 2024 07:12:30 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 11096161E37 for ; Wed, 28 Aug 2024 11:12:30 +0000 (UTC) X-FDA: 82501390860.10.0E333E6 Received: from nyc.source.kernel.org (nyc.source.kernel.org [147.75.193.91]) by imf19.hostedemail.com (Postfix) with ESMTP id 574C31A0019 for ; Wed, 28 Aug 2024 11:12:28 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=KNOjJ3rS; spf=pass (imf19.hostedemail.com: domain of brauner@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=brauner@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1724843504; a=rsa-sha256; cv=none; b=wfp976DSO84xliY3ZwEWFy1JwSjjDkLD6ck2aEwdoMFgjdTK7rVdxl5LvQ892RswA0eKnU W2Z1YR+6HFdRSr8yif0Le6I2zoeYujw31U+VnpNHNHU/BKo5jWWMu2UDfiskgnmdVc0gzv HCoHWvter66yIzcc1+DfXBdIhtXdr+A= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=KNOjJ3rS; spf=pass (imf19.hostedemail.com: domain of brauner@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=brauner@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1724843504; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=iVm82165TaPtf+tyRU4vMdnmHmQNZpSCQ9Amk/uhcqY=; b=DBHY02O7jrDQarR6PIhov/G7qaWv+dMJBVBz2KGyv8MfarXGMWxWAxZ4yaSY0xlSFR5FFK HklqRdaIIxjlqTUgXWqsfQ/TKQXEXY2nHlIfSfARiU+LCiiXzC4fGFF6UYEmD0WRcvbPGT 9Ox86od0kNiESLwK7Cvd2WlwJ40xOhM= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by nyc.source.kernel.org (Postfix) with ESMTP id C9530A43565; Wed, 28 Aug 2024 11:12:19 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 78779C4CACD; Wed, 28 Aug 2024 10:57:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1724842630; bh=wNVAvroRRvpcORJPuYkDuLT0tHqBRq4ogN+7NtjeOC8=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=KNOjJ3rSY9QCIDCROpFWlqfF5Efb1VmHKi6WleD7nLsr/IuCwGZMkd/+g/VsE1o4m UvhbZkb+8IVnySVeAzEe4UHsrnvr2bSoY3qA5EgHVZgFLmBgrPxxZJHOkH5ruE3pBv T3FJT3o4fXGL7kY23O/yK0GAlaDOceXKTJLafQczX6KC2Y0nhbPK4ANkZf9Xxq6pi7 VQ70Qww2XfzW1RdNeOVfEjUL4p8HcYBmnLsyra0X6H2ongcCdNuDBZ6c65s6+pKhuL B/ueXJs08jKi7Tl6ByIQUu2JNJbPUxo9dwEE3r3sJ63GlOEqLf8v096d2H+tQTEMV/ m3eS7x6p0Nvww== From: Christian Brauner Date: Wed, 28 Aug 2024 12:56:24 +0200 Subject: [PATCH v3 2/3] mm: add kmem_cache_create_rcu() MIME-Version: 1.0 Message-Id: <20240828-work-kmem_cache-rcu-v3-2-5460bc1f09f6@kernel.org> References: <20240828-work-kmem_cache-rcu-v3-0-5460bc1f09f6@kernel.org> In-Reply-To: <20240828-work-kmem_cache-rcu-v3-0-5460bc1f09f6@kernel.org> To: Vlastimil Babka , Jens Axboe , "Paul E. McKenney" , Roman Gushchin , Jann Horn , Linus Torvalds , Mike Rapoport , linux-mm@kvack.org Cc: Christian Brauner , linux-fsdevel@vger.kernel.org X-Mailer: b4 0.15-dev-37811 X-Developer-Signature: v=1; a=openpgp-sha256; l=11892; i=brauner@kernel.org; h=from:subject:message-id; bh=wNVAvroRRvpcORJPuYkDuLT0tHqBRq4ogN+7NtjeOC8=; b=owGbwMvMwCU28Zj0gdSKO4sYT6slMaSdZ6rRYrBUadYONvdc0rbE+8DaIv0z3kdS+Xn3T/2RE GV6bMmJjlIWBjEuBlkxRRaHdpNwueU8FZuNMjVg5rAygQxh4OIUgIksOsHIMH3bAfer+4W2Ct9i XsjmHv1UdGZ3Ks/0X+dlYv3mzmPe5svw33u3cUA685GXxitWRH9dcF/2y8up7Lte5x1glOWueJ/ Nzw0A X-Developer-Key: i=brauner@kernel.org; a=openpgp; fpr=4880B8C9BD0E5106FC070F4F7B3C391EFEA93624 X-Stat-Signature: ee5h1p9bbg36f965jkg6imkq6bgmn5q8 X-Rspamd-Queue-Id: 574C31A0019 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1724843548-276894 X-HE-Meta: U2FsdGVkX1/dV3Cumf5gAPmEEIZQ2viSVYvA1MyWfd+mwgghvIcJEOqAPwBf/MMj1zV0hFXPxrD7u8gU0/C7bD2kYe37zTy1PztIk6bE8/ml3LCfQnFxeZOMNVxQgoTpwMRRmtaZ+GfiZSaMFabZxeoenmJVwlrTnnIWv0QAFw+NwISM90yUZNcImbDe2zFL653YImQU2HZ5Q/zPD0joNamPFEBeK3KL9qa0O7dXs4rETu3cWMsK2Rxhot6jcvgneApDhxRuKX+qC7PZS8OHGWhqBN+TMOA8EHnEbU/1DBDnI9SmMd+azOomWQykeKQAWNNrngLOT4t+q1Y38XB0Awp06cMIVxaoSrQlC4yo6qSsM42qWJEdsnb9wABH9DZfM35ltDEayAghF1IIq9q6+2F4J+Dcwp7tKLuTmvynbg4DIo8gavaIkOBLs3aIuyTYU1b2gM7DpvAKIUTOkA3f8stJghLaCAY6r4+0pRHy4ZI8dD6HMxHDlpHRpKIp4U3FnI3SUHZQtJ+ud1WY0wkiebRlOk9DtpjdCtkBzNupT5Tx9nPLUlJZAvheEXKV3+WykXFDO/sx/Y2I8G0bu2VhQw+y3Dw2WQVsPL756YR+RgKEVteZ2neiYx9+9TrioRm1IDMu1EsQ3Rva5q1HkCxNwxa+kwwaU/wJI/8FAUnZLfK0VicLOhif3ujgu1VeL0v8r3T0YMI4LDbit9Q/VQrVP7zhg1zcdGB5JbkPotx1sUdo5qU3NY0RCaI2jJu32ZhGfjVuNTWxgOY0JAlIB4ui+IsA82m81ljS876SFkxhMiMzkt6JFI9PngU0TJ3ooUggauucX4eSzTy/DR6UmvPXyu1ZVs/gpTwS8f3uNnq4L0JcufK2hgNLoGZB8euyVRHDTJyHxD/0mt8zhbAdH5dwEE5bKFq4SmpTagyfAtzFQWmwVKbuZcD4tDP2U96sH2iE1kxi5qZ7fMkv8VYIoxV j2ZnKHyU bcIsfLPmw0NeDvOw7tVpam9Ify7MGf3vlkHwBgr6fLD1Lfp1J/SlB7KiVAQxilGPMhrtyichJwtpBaZ1E9IIl1BRH+v7tstWy1BHmQteXOUHpiKmZSYXOb4m3VMv72C3g52m8+ADC7CjWeH80bXw+VM8bYNBiZfKUyYpeWseKWMbG6Tf4Fnfbpm0GdiCWuPoDzPdecIZL2QRVeKDw4iLZu4Ox0B3GmGh8sj3WzLpF3a6dhoC2AhHfFad2Cb97nT0SGtjIk07vgD5wv3S1x1lrKp9m3MsyP/GDz/uVHzg4d6gTwNm05FbZabk2QCOV1uMDwgVwl16oc0qDSUo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When a kmem cache is created with SLAB_TYPESAFE_BY_RCU the free pointer must be located outside of the object because we don't know what part of the memory can safely be overwritten as it may be needed to prevent object recycling. That has the consequence that SLAB_TYPESAFE_BY_RCU may end up adding a new cacheline. This is the case for e.g., struct file. After having it shrunk down by 40 bytes and having it fit in three cachelines we still have SLAB_TYPESAFE_BY_RCU adding a fourth cacheline because it needs to accommodate the free pointer. Add a new kmem_cache_create_rcu() function that allows the caller to specify an offset where the free pointer is supposed to be placed. Signed-off-by: Christian Brauner Reviewed-by: Vlastimil Babka Acked-by: Mike Rapoport (Microsoft) --- include/linux/slab.h | 9 ++++ mm/slab.h | 2 + mm/slab_common.c | 136 ++++++++++++++++++++++++++++++++++++--------------- mm/slub.c | 20 +++++--- 4 files changed, 121 insertions(+), 46 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index eb2bf4629157..5b2da2cf31a8 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -212,6 +212,12 @@ enum _slab_flag_bits { #define SLAB_NO_OBJ_EXT __SLAB_FLAG_UNUSED #endif +/* + * freeptr_t represents a SLUB freelist pointer, which might be encoded + * and not dereferenceable if CONFIG_SLAB_FREELIST_HARDENED is enabled. + */ +typedef struct { unsigned long v; } freeptr_t; + /* * ZERO_SIZE_PTR will be returned for zero sized kmalloc requests. * @@ -242,6 +248,9 @@ struct kmem_cache *kmem_cache_create_usercopy(const char *name, slab_flags_t flags, unsigned int useroffset, unsigned int usersize, void (*ctor)(void *)); +struct kmem_cache *kmem_cache_create_rcu(const char *name, unsigned int size, + unsigned int freeptr_offset, + slab_flags_t flags); void kmem_cache_destroy(struct kmem_cache *s); int kmem_cache_shrink(struct kmem_cache *s); diff --git a/mm/slab.h b/mm/slab.h index dcdb56b8e7f5..a6051385186e 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -261,6 +261,8 @@ struct kmem_cache { unsigned int object_size; /* Object size without metadata */ struct reciprocal_value reciprocal_size; unsigned int offset; /* Free pointer offset */ + /* Specific free pointer requested (if not UINT_MAX) */ + unsigned int rcu_freeptr_offset; #ifdef CONFIG_SLUB_CPU_PARTIAL /* Number of per cpu partial objects to keep around */ unsigned int cpu_partial; diff --git a/mm/slab_common.c b/mm/slab_common.c index c8dd7e08c5f6..887f6b9855dd 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -202,9 +202,10 @@ struct kmem_cache *find_mergeable(unsigned int size, unsigned int align, } static struct kmem_cache *create_cache(const char *name, - unsigned int object_size, unsigned int align, - slab_flags_t flags, unsigned int useroffset, - unsigned int usersize, void (*ctor)(void *)) + unsigned int object_size, unsigned int freeptr_offset, + unsigned int align, slab_flags_t flags, + unsigned int useroffset, unsigned int usersize, + void (*ctor)(void *)) { struct kmem_cache *s; int err; @@ -212,6 +213,13 @@ static struct kmem_cache *create_cache(const char *name, if (WARN_ON(useroffset + usersize > object_size)) useroffset = usersize = 0; + /* If a custom freelist pointer is requested make sure it's sane. */ + err = -EINVAL; + if (freeptr_offset != UINT_MAX && + (freeptr_offset >= object_size || !(flags & SLAB_TYPESAFE_BY_RCU) || + !IS_ALIGNED(freeptr_offset, sizeof(freeptr_t)))) + goto out; + err = -ENOMEM; s = kmem_cache_zalloc(kmem_cache, GFP_KERNEL); if (!s) @@ -219,13 +227,13 @@ static struct kmem_cache *create_cache(const char *name, s->name = name; s->size = s->object_size = object_size; + s->rcu_freeptr_offset = freeptr_offset; s->align = align; s->ctor = ctor; #ifdef CONFIG_HARDENED_USERCOPY s->useroffset = useroffset; s->usersize = usersize; #endif - err = __kmem_cache_create(s, flags); if (err) goto out_free_cache; @@ -240,38 +248,10 @@ static struct kmem_cache *create_cache(const char *name, return ERR_PTR(err); } -/** - * kmem_cache_create_usercopy - Create a cache with a region suitable - * for copying to userspace - * @name: A string which is used in /proc/slabinfo to identify this cache. - * @size: The size of objects to be created in this cache. - * @align: The required alignment for the objects. - * @flags: SLAB flags - * @useroffset: Usercopy region offset - * @usersize: Usercopy region size - * @ctor: A constructor for the objects. - * - * Cannot be called within a interrupt, but can be interrupted. - * The @ctor is run when new pages are allocated by the cache. - * - * The flags are - * - * %SLAB_POISON - Poison the slab with a known test pattern (a5a5a5a5) - * to catch references to uninitialised memory. - * - * %SLAB_RED_ZONE - Insert `Red` zones around the allocated memory to check - * for buffer overruns. - * - * %SLAB_HWCACHE_ALIGN - Align the objects in this cache to a hardware - * cacheline. This can be beneficial if you're counting cycles as closely - * as davem. - * - * Return: a pointer to the cache on success, NULL on failure. - */ -struct kmem_cache * -kmem_cache_create_usercopy(const char *name, - unsigned int size, unsigned int align, - slab_flags_t flags, +static struct kmem_cache * +do_kmem_cache_create_usercopy(const char *name, + unsigned int size, unsigned int freeptr_offset, + unsigned int align, slab_flags_t flags, unsigned int useroffset, unsigned int usersize, void (*ctor)(void *)) { @@ -331,7 +311,7 @@ kmem_cache_create_usercopy(const char *name, goto out_unlock; } - s = create_cache(cache_name, size, + s = create_cache(cache_name, size, freeptr_offset, calculate_alignment(flags, align, size), flags, useroffset, usersize, ctor); if (IS_ERR(s)) { @@ -355,6 +335,45 @@ kmem_cache_create_usercopy(const char *name, } return s; } + +/** + * kmem_cache_create_usercopy - Create a cache with a region suitable + * for copying to userspace + * @name: A string which is used in /proc/slabinfo to identify this cache. + * @size: The size of objects to be created in this cache. + * @freeptr_offset: Custom offset for the free pointer in RCU caches + * @align: The required alignment for the objects. + * @flags: SLAB flags + * @useroffset: Usercopy region offset + * @usersize: Usercopy region size + * @ctor: A constructor for the objects. + * + * Cannot be called within a interrupt, but can be interrupted. + * The @ctor is run when new pages are allocated by the cache. + * + * The flags are + * + * %SLAB_POISON - Poison the slab with a known test pattern (a5a5a5a5) + * to catch references to uninitialised memory. + * + * %SLAB_RED_ZONE - Insert `Red` zones around the allocated memory to check + * for buffer overruns. + * + * %SLAB_HWCACHE_ALIGN - Align the objects in this cache to a hardware + * cacheline. This can be beneficial if you're counting cycles as closely + * as davem. + * + * Return: a pointer to the cache on success, NULL on failure. + */ +struct kmem_cache * +kmem_cache_create_usercopy(const char *name, unsigned int size, + unsigned int align, slab_flags_t flags, + unsigned int useroffset, unsigned int usersize, + void (*ctor)(void *)) +{ + return do_kmem_cache_create_usercopy(name, size, UINT_MAX, align, flags, + useroffset, usersize, ctor); +} EXPORT_SYMBOL(kmem_cache_create_usercopy); /** @@ -386,11 +405,50 @@ struct kmem_cache * kmem_cache_create(const char *name, unsigned int size, unsigned int align, slab_flags_t flags, void (*ctor)(void *)) { - return kmem_cache_create_usercopy(name, size, align, flags, 0, 0, - ctor); + return do_kmem_cache_create_usercopy(name, size, UINT_MAX, align, flags, + 0, 0, ctor); } EXPORT_SYMBOL(kmem_cache_create); +/** + * kmem_cache_create_rcu - Create a SLAB_TYPESAFE_BY_RCU cache. + * @name: A string which is used in /proc/slabinfo to identify this cache. + * @size: The size of objects to be created in this cache. + * @freeptr_offset: The offset into the memory to the free pointer + * @flags: SLAB flags + * + * Cannot be called within an interrupt, but can be interrupted. + * + * See kmem_cache_create() for an explanation of possible @flags. + * + * By default SLAB_TYPESAFE_BY_RCU caches place the free pointer outside + * of the object. This might cause the object to grow in size. Callers + * that have a reason to avoid this can specify a custom free pointer + * offset in their struct where the free pointer will be placed. + * + * Note that placing the free pointer inside the object requires the + * caller to ensure that no fields are invalidated that are required to + * guard against object recycling (See SLAB_TYPESAFE_BY_RCU for + * details.). + * + * Using zero as a value for @freeptr_offset is valid. To request no + * offset UINT_MAX must be specified. + * + * Note that @ctor isn't supported with custom free pointers as a @ctor + * requires an external free pointer. + * + * Return: a pointer to the cache on success, NULL on failure. + */ +struct kmem_cache *kmem_cache_create_rcu(const char *name, unsigned int size, + unsigned int freeptr_offset, + slab_flags_t flags) +{ + return do_kmem_cache_create_usercopy(name, size, freeptr_offset, 0, + flags | SLAB_TYPESAFE_BY_RCU, 0, 0, + NULL); +} +EXPORT_SYMBOL(kmem_cache_create_rcu); + static struct kmem_cache *kmem_buckets_cache __ro_after_init; /** diff --git a/mm/slub.c b/mm/slub.c index c9d8a2497fd6..9aa5da1e8e27 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -465,12 +465,6 @@ static struct workqueue_struct *flushwq; * Core slab cache functions *******************************************************************/ -/* - * freeptr_t represents a SLUB freelist pointer, which might be encoded - * and not dereferenceable if CONFIG_SLAB_FREELIST_HARDENED is enabled. - */ -typedef struct { unsigned long v; } freeptr_t; - /* * Returns freelist pointer (ptr). With hardening, this is obfuscated * with an XOR of the address where the pointer is held and a per-cache @@ -3921,6 +3915,9 @@ static void *__slab_alloc_node(struct kmem_cache *s, /* * If the object has been wiped upon free, make sure it's fully initialized by * zeroing out freelist pointer. + * + * Note that we also wipe custom freelist pointers specified via + * s->rcu_freeptr_offset. */ static __always_inline void maybe_wipe_obj_freeptr(struct kmem_cache *s, void *obj) @@ -5144,6 +5141,12 @@ static void set_cpu_partial(struct kmem_cache *s) #endif } +/* Was a valid freeptr offset requested? */ +static inline bool has_freeptr_offset(const struct kmem_cache *s) +{ + return s->rcu_freeptr_offset != UINT_MAX; +} + /* * calculate_sizes() determines the order and the distribution of data within * a slab object. @@ -5189,7 +5192,8 @@ static int calculate_sizes(struct kmem_cache *s) */ s->inuse = size; - if ((flags & (SLAB_TYPESAFE_BY_RCU | SLAB_POISON)) || s->ctor || + if (((flags & SLAB_TYPESAFE_BY_RCU) && !has_freeptr_offset(s)) || + (flags & SLAB_POISON) || s->ctor || ((flags & SLAB_RED_ZONE) && (s->object_size < sizeof(void *) || slub_debug_orig_size(s)))) { /* @@ -5210,6 +5214,8 @@ static int calculate_sizes(struct kmem_cache *s) */ s->offset = size; size += sizeof(void *); + } else if ((flags & SLAB_TYPESAFE_BY_RCU) && has_freeptr_offset(s)) { + s->offset = s->rcu_freeptr_offset; } else { /* * Store freelist pointer near middle of object to keep From patchwork Wed Aug 28 10:56:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Brauner X-Patchwork-Id: 13781134 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19773C54ED1 for ; Wed, 28 Aug 2024 11:12:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 92FD26B0088; Wed, 28 Aug 2024 07:12:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8902B6B0089; Wed, 28 Aug 2024 07:12:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 757DE6B008C; Wed, 28 Aug 2024 07:12:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 58F266B0088 for ; Wed, 28 Aug 2024 07:12:29 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id BF0DA41F1F for ; Wed, 28 Aug 2024 11:12:28 +0000 (UTC) X-FDA: 82501390776.27.C9BEB9A Received: from nyc.source.kernel.org (nyc.source.kernel.org [147.75.193.91]) by imf15.hostedemail.com (Postfix) with ESMTP id 098CBA001D for ; Wed, 28 Aug 2024 11:12:26 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=B0tkqFU0; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf15.hostedemail.com: domain of brauner@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=brauner@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1724843482; a=rsa-sha256; cv=none; b=BLJ/dZAUcmdJ2b1mIyn57BFa/O/cxILOT5mQnBh/MpnC8xjN3kid+gA47TtoyvUsvwlHk8 X+4fyDpJ4uOIAuP/pk2tFqOoTVupFjtpvNlpb/EkyJsPK2gTesaRKK3TQma/WbYNVZ3fW8 Z+lOkZNk+n7QRaXDgTUVmVNqlauhXKM= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=B0tkqFU0; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf15.hostedemail.com: domain of brauner@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=brauner@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1724843482; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Q/4ZuQAHivHS47Qh1bVpCkPwi5OXsHv/q8ivNrkzfTg=; b=PxhWLmjQ+DtJy0xrX0mZTbBnKAkZfKeg+rv27iaBUtWceZVozRHdY4RKdphWsIKjKzzU9d 6zzpWaFKDp+T8yop6aDlyvA8Q0FLnpJEEJXiZ1z9HSDAv+xY1Xw4Yy68udtNM/J8h70Z5D PIKHXZFf4dKcz+VRt2BK+K/+Cz6lZmg= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by nyc.source.kernel.org (Postfix) with ESMTP id 23C61A43515; Wed, 28 Aug 2024 11:12:19 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A4835C4CADC; Wed, 28 Aug 2024 10:57:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1724842633; bh=sHidI5jLSIX8UVIkMjGOIDyMLbkLDe+hp8XvmRLq1Os=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=B0tkqFU0sJKz1kSgkSPagbo7d+km0HFluA7VxuYR7+178B+NrzPVKuoIwjxPWVUsz dcJxIroODwog7d2FVspEVzk83T586CuOtp9L0hCCnHfKFf18fNmfUL27A8R024RQ4O 3/VusbKgIvFSJHDJskhfUwwK/sdmw/km9mN60NLr2WswhtqA2ATORIq8C922InicQU Zds8j07G4y06GhWe06R6ZCWDoqOXttIIIbklmBjGuFRBCj2fsdE+2hiFTm9gtG9cAt u1Bs7l+z3UmoSG3gZLLIgwMCTlKUeXl4ZvmtTCJ+ssUYt3B9ylPudtO8zL8xsXWrYs 9/R/IuBXINTyw== From: Christian Brauner Date: Wed, 28 Aug 2024 12:56:25 +0200 Subject: [PATCH v3 3/3] fs: use kmem_cache_create_rcu() MIME-Version: 1.0 Message-Id: <20240828-work-kmem_cache-rcu-v3-3-5460bc1f09f6@kernel.org> References: <20240828-work-kmem_cache-rcu-v3-0-5460bc1f09f6@kernel.org> In-Reply-To: <20240828-work-kmem_cache-rcu-v3-0-5460bc1f09f6@kernel.org> To: Vlastimil Babka , Jens Axboe , "Paul E. McKenney" , Roman Gushchin , Jann Horn , Linus Torvalds , Mike Rapoport , linux-mm@kvack.org Cc: Christian Brauner , linux-fsdevel@vger.kernel.org X-Mailer: b4 0.15-dev-37811 X-Developer-Signature: v=1; a=openpgp-sha256; l=1718; i=brauner@kernel.org; h=from:subject:message-id; bh=sHidI5jLSIX8UVIkMjGOIDyMLbkLDe+hp8XvmRLq1Os=; b=owGbwMvMwCU28Zj0gdSKO4sYT6slMaSdZ6q5v6hBTF6IM/HeIbPn2qWv6zQ+c8mU3Fr3+Nyv7 h/fOkK/dZSyMIhxMciKKbI4tJuEyy3nqdhslKkBM4eVCWQIAxenAExk6y6G/wmJtYGf3WQnBFvY dAa9CWXdZ+QdMM18h0Xbcp/c3TlaWowMPyq+bkycXNlYJG8hGpGceuWrTLR61vd67V+d27ZYfmv hBAA= X-Developer-Key: i=brauner@kernel.org; a=openpgp; fpr=4880B8C9BD0E5106FC070F4F7B3C391EFEA93624 X-Rspamd-Queue-Id: 098CBA001D X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: rjxh1kng4adnkpikozf9p9ap7ia4h468 X-HE-Tag: 1724843546-512068 X-HE-Meta: U2FsdGVkX18COgHKXCPbvy0jfCyATZgxHQkPF+IQ0IObJtjnifEcsDPJ4mpvSdPzFafKMoQwfNflLz2QuFjujr9dKbVDy8CGICl/RcjeAoQlsSWLOZDHVsI4n36RDmQxNBJIQPkruUZCm8Zm9V8Mg79cgfw2KLS2ipaCNWDe09vXDPwFnSZgNIMR13S3ZMIoFQWA2Uk+HTzN/on+4C5j2bpTQ4BX16yrX89wIXDLxp12+TgYPqN/2Khe6CVNmWXSh2fDdqquFJKKfB2UvmIrBqChBW983h8wRLDL84+PX69/bxeDjJJR6juMCbNUx/wnxs5ZmdMeBJUiKeFnID8UNtrv208jCbBCb3xeMnTuY8Pc4sZffIRGIm9sh4cBxhbs6K6dd9FJ3oroOtSOj3wU2ut5+xdhVzGCmorE7oZO0D4PWLtvNbeJ0O1wzanbT/DyrQWUgaO58Px4L18JNf6YTU7mnP6l54kvdXYKjPDnsGALFwe9KBt80mSTEcpRmW4glEgDOrdk9u0w6usjS5mr40TyM6nWEL5NuLgG9s45Q0X3L8NYOhGyCct6G23bgFOYbde5QrTNNd/eqr5g7YJ8+JB+v5Uh4R5Ezh8M/O2Cde4z7yAR+bqTMJMatVyhXbQDbrXuCvH8BPM+P9LLBVxahJKsgTmMrnCMy0L0ohSCTgzLIfY9J60lnseeWy8oFOmOdFjz1SOQQXmi9QS4s7C4gMiH7o/s/U04HjTsv+vq+5H1KEISKoLUrDY5S8c5Y4LLxxd/Aogaaa6e8CxSc7OtHwAjb+YZDN5TLvV9gDH6pmDLLXhfuSLYQWyCwO6lR9XIq4EZnN/xa6Tx4/mR+PDyloZtP/8i3NpygdGMY5izT3a9CFicBiIro2cqPoOjmGC6l3litRzOg7plfI2sgr5NI+d6WhKNd7HlRWLmHGv82+44GjRNMt/0H33KO3l5J6A00DYaQ7xWB2UKROS/T2F XoFLJTWD 1M6qHzEBbbWdokiXAFtEUIXXuZAcXr4axhcFqhq3ea90QMZcCKJkQ1rZsWVtydj1CdUmgTxDWp+LnNGGUt6+QtoyoxZ5uO71dgxSp+8qGyXvg4WpaNFjkVg/FRteY/WYnxn7QodS9r0kZSj9EMactrnhyndv/H2CR4VobLFUtKlJfTATfSiQTDq8Kc0J6Hu7pnlmDjQg3M3qQ36h2FPXlGkaFhGKuT1BJa//yJXkEsPDlrpG8BN3s9NDzOiGWsapHyWZKxUjr6a+AgKnhrKzVF6FbJrpcdJ40WztTGvE1+c3NMBBCWlpMr1i5bQ2/GLEKkCQR X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Switch to the new kmem_cache_create_rcu() helper which allows us to use a custom free pointer offset avoiding the need to have an external free pointer which would grow struct file behind our backs. Signed-off-by: Christian Brauner Reviewed-by: Vlastimil Babka --- fs/file_table.c | 6 +++--- include/linux/fs.h | 2 ++ 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/fs/file_table.c b/fs/file_table.c index 694199a1a966..83d5ac1fadc0 100644 --- a/fs/file_table.c +++ b/fs/file_table.c @@ -514,9 +514,9 @@ EXPORT_SYMBOL(__fput_sync); void __init files_init(void) { - filp_cachep = kmem_cache_create("filp", sizeof(struct file), 0, - SLAB_TYPESAFE_BY_RCU | SLAB_HWCACHE_ALIGN | - SLAB_PANIC | SLAB_ACCOUNT, NULL); + filp_cachep = kmem_cache_create_rcu("filp", sizeof(struct file), + offsetof(struct file, f_freeptr), + SLAB_HWCACHE_ALIGN | SLAB_PANIC | SLAB_ACCOUNT); percpu_counter_init(&nr_files, 0, GFP_KERNEL); } diff --git a/include/linux/fs.h b/include/linux/fs.h index 61097a9cf317..5ca83e87adef 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -1026,6 +1026,7 @@ static inline int ra_has_index(struct file_ra_state *ra, pgoff_t index) * @f_task_work: task work entry point * @f_llist: work queue entrypoint * @f_ra: file's readahead state + * @f_freeptr: Pointer used by SLAB_TYPESAFE_BY_RCU file cache (don't touch.) */ struct file { atomic_long_t f_count; @@ -1057,6 +1058,7 @@ struct file { struct callback_head f_task_work; struct llist_node f_llist; struct file_ra_state f_ra; + freeptr_t f_freeptr; }; /* --- cacheline 3 boundary (192 bytes) --- */ } __randomize_layout