From patchwork Tue Aug 27 15:59:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Brauner X-Patchwork-Id: 13779778 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DB890C5472F for ; Tue, 27 Aug 2024 16:07:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 645D96B0093; Tue, 27 Aug 2024 12:07:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5F5BF6B0095; Tue, 27 Aug 2024 12:07:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4E5B96B0096; Tue, 27 Aug 2024 12:07:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 2F38C6B0093 for ; Tue, 27 Aug 2024 12:07:55 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 4CD9181C5D for ; Tue, 27 Aug 2024 16:07:31 +0000 (UTC) X-FDA: 82498505502.02.41ECB59 Received: from nyc.source.kernel.org (nyc.source.kernel.org [147.75.193.91]) by imf21.hostedemail.com (Postfix) with ESMTP id 20C721C0031 for ; Tue, 27 Aug 2024 16:07:28 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=L7XpTDGg; spf=pass (imf21.hostedemail.com: domain of brauner@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=brauner@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1724774780; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=uazMo0IM558jInsehz70etwPs2IJBI+HgyGqAKmMDI8=; b=espg9xTl2j/Z5FPiyOpqiXuGYmMMYw7OrKSwS0X2SAI9VPMvXufccQ7CJqXWwLHQBfrroV E9t+rQUc9xhGmP0vv+x/TZMuLEPAitCh3w0gj+uxixpmFCbWxzhcWmGeFnqV25gxzqfLyl aRSvncQSRG+Y3tWKlKGNARS6LRMB4qk= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=L7XpTDGg; spf=pass (imf21.hostedemail.com: domain of brauner@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=brauner@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1724774780; a=rsa-sha256; cv=none; b=ZQOkPTV91N6nD83BOrpmBRvMP/W2IRY2GAVXptt6FKvLKSMSJOHCVaJNz6GSWEzRGdU4c+ 0U6nUJdE3jm4XA/nOKvbp4bfd4oUrhAEoU0QUUosYwMnG1A/Fd+FW08vdVW5fyk8f7hHm8 HKf17NdP6cW9QR6Zt/PcI75dlus4Cl8= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by nyc.source.kernel.org (Postfix) with ESMTP id 9EF65A41FDF; Tue, 27 Aug 2024 16:07:20 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id F2129C4FDE2; Tue, 27 Aug 2024 15:59:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1724774399; bh=inSYKr8Tp4YUwStuHZTh2AxEw7grjX6tn9NgCfW/hs4=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=L7XpTDGgBL8T6Ryq+Mm+zh/RFrNd6ISBnEgZbezEGDKfSw0Hu0waaQEyNmPmbL7JH gesM8omul5uT5JPNUKogfwtZGlmAl/+u6sBThJPJjUCah2dP0LcyyFJNf4GSq78WLU 1VF/vGp3nrPrtiNEMl3/BxBwZs6eEa9xr1FQhGu+2SkT4C0EfIJ+Aw1yF0B1s0/4lK hBTeqjjqdwkjMlIZtD3yJ50RvUlWCturr7xam3hrxpuwp4HEzrPQ85vzzAOBeae4Ik pBKX6OIis2zpR/qk7/ncNircBh+YUkl6AlrzeMyUWHY7fOyWpRdGXR8SPnkQSoUw8I qu8N+Q7kwER2A== From: Christian Brauner Date: Tue, 27 Aug 2024 17:59:42 +0200 Subject: [PATCH v2 1/3] mm: remove unused root_cache argument MIME-Version: 1.0 Message-Id: <20240827-work-kmem_cache-rcu-v2-1-7bc9c90d5eef@kernel.org> References: <20240827-work-kmem_cache-rcu-v2-0-7bc9c90d5eef@kernel.org> In-Reply-To: <20240827-work-kmem_cache-rcu-v2-0-7bc9c90d5eef@kernel.org> To: Vlastimil Babka , Jens Axboe , "Paul E. McKenney" , Roman Gushchin , Jann Horn , Linus Torvalds , linux-mm@kvack.org Cc: Christian Brauner , linux-fsdevel@vger.kernel.org X-Mailer: b4 0.15-dev-37811 X-Developer-Signature: v=1; a=openpgp-sha256; l=1077; i=brauner@kernel.org; h=from:subject:message-id; bh=inSYKr8Tp4YUwStuHZTh2AxEw7grjX6tn9NgCfW/hs4=; b=owGbwMvMwCU28Zj0gdSKO4sYT6slMaSd/f674rHcioZtp3PKJ+07Nvv38q9nVynHlR6+WNmzy DN4k8r6OR2lLAxiXAyyYoosDu0m4XLLeSo2G2VqwMxhZQIZwsDFKQATOd3D8D/zk+MfqUt9y/L4 /hRvOv3XWj7nOesJHY7aiE25+3gu/kphZDj/+IJAzK9lEbvnb+38OkUgZ+6aU1LKvVvc1rx4vWm f3XluAA== X-Developer-Key: i=brauner@kernel.org; a=openpgp; fpr=4880B8C9BD0E5106FC070F4F7B3C391EFEA93624 X-Stat-Signature: y3gp138fey67pshsdiqz3qogi5his16q X-Rspam-User: X-Rspamd-Queue-Id: 20C721C0031 X-Rspamd-Server: rspam02 X-HE-Tag: 1724774848-758959 X-HE-Meta: U2FsdGVkX1/PnfBL8xV2/+JgF5Beoxh0eThBwv3q1DlizxXVokg3vIxqAee3rUCbSVI7exgW0lufdvMj+sObhCaDXBIwcd3dyQHLdZOGagKJljD932uP7Gspyj3sM70pCGaan2lkGnMgRIgyP3ctq49nYaPH8r/MX29NFHg9RZvlrK8swh2eLkKxGXBxYgBqnHiftWYNzX6Xph7vJMfPhSmZKfsX1o1pFTgwwevktCo7ecXKtFaRk4pxTguhQi60MUHOPBIljyQlC7NBxOITKK5spoo/LXB267MWUXzLmR3ulvnirYV7BzqerO2wxto+ldbds4Q34Wo2pLVoCaM+DGC7v5E5L3U1pO731Du7cAZsRaWAW2BUy9+IsV9JmqafsYWylaRCnzaEEBhQi2hnwozH2bFirXmXjUh7FS5FxjvWPL6By4HMKksXxFtpHUhw3ZIdwhozkD3BexFpwGvUQb/GttNF9eR0cre9N/Rpd69eqdgOOvitXwZUMQFF9UA+6ilDSAqIhlx1m2ySJ8odwo9AA7OmH6VPNKK18nH47zgQYPsPWFH7/0DCW46Npzh+S6f/2pDUhEAh5TjYlrYew+YJ7AlFKBi3UxaKYBNIIa5qnDhSJoVzntes4KDMZjML2cmY5u6DGM6H9AkwzZXDpf0F0lcOfe349eulBAbu59HnuT0nOwfm6mnidt1JHHYGDmWyWZWTAkRdgIB+MPmq9b99AsR/aaq7cRdGdnjexx8/D7Bh4TeoUgIg8tQHU3RR530UZAnhDP7ZCB6e5g8ABV2eVfmj9CQHWa1nJ9PhAIqNUeKxqBykXHvOjixB+K+Xy+D5T/68SQ911IqVv4qY/DGn/dMdfOTdIVGx4/+9uBD1/CpN7hMuGEwqNChxMq1fVSdmO16sWv2ZXw30dKz/Taq7yGIqHWsgYMsRiYqKrT/fNEqSZCcwB3K4VpyHf11kAXlr2Rla3fIDW+0hUAi xeVzOsYY OJS+LL5ePN3ii+ve7b3P+DXzkmiBj+prPXqbsjXNKvX3Hz1dFLM5mMcnQkty0iks8NycJcmNTN8q3Ueidh7wl6u8W8pEA+xoF6YctNF2dPLoOl2d+NRAkzdcno5u1DAbxv+W/aknWKpggaTu5tgBbhQ7V0PJiWr+/hQuH2KtxCsDd6y0iDuwgzgMSbV7XX74adKDxrpPAEedJVPI6P3Dx6OwC1gTRgkAiKvdFrWO3ztquQdAkIgytLFSWkXnt/G0NyT880ffwc1YxmQxuXmhcpZcPP8qCW57N5eA5XGgONJ+1VNjYdDKds14MzlKo7+OmBY3g X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: That argument is unused so remove it. Signed-off-by: Christian Brauner --- mm/slab_common.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/mm/slab_common.c b/mm/slab_common.c index 40b582a014b8..c8dd7e08c5f6 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -204,8 +204,7 @@ struct kmem_cache *find_mergeable(unsigned int size, unsigned int align, static struct kmem_cache *create_cache(const char *name, unsigned int object_size, unsigned int align, slab_flags_t flags, unsigned int useroffset, - unsigned int usersize, void (*ctor)(void *), - struct kmem_cache *root_cache) + unsigned int usersize, void (*ctor)(void *)) { struct kmem_cache *s; int err; @@ -334,7 +333,7 @@ kmem_cache_create_usercopy(const char *name, s = create_cache(cache_name, size, calculate_alignment(flags, align, size), - flags, useroffset, usersize, ctor, NULL); + flags, useroffset, usersize, ctor); if (IS_ERR(s)) { err = PTR_ERR(s); kfree_const(cache_name); From patchwork Tue Aug 27 15:59:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Brauner X-Patchwork-Id: 13779772 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 63461C54731 for ; Tue, 27 Aug 2024 16:07:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ED2116B007B; Tue, 27 Aug 2024 12:07:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E80F66B0088; Tue, 27 Aug 2024 12:07:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D21E06B0093; Tue, 27 Aug 2024 12:07:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id B0B846B007B for ; Tue, 27 Aug 2024 12:07:32 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 659AA121B11 for ; Tue, 27 Aug 2024 16:07:26 +0000 (UTC) X-FDA: 82498505292.24.78F1CE1 Received: from nyc.source.kernel.org (nyc.source.kernel.org [147.75.193.91]) by imf20.hostedemail.com (Postfix) with ESMTP id 440BB1C0024 for ; Tue, 27 Aug 2024 16:07:23 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=eHfC9fWz; spf=pass (imf20.hostedemail.com: domain of brauner@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=brauner@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1724774757; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=aLNXQS7NYLWIGU2yeQWtvdlxoSn9DuYQq3smEoSvHRY=; b=cbBRr/Dc/89ca9J6yn7cfxtk7A5oyAaC1qwykbJhvYsVrP2iA/3bZISCoq1CmCxhjC2GKj YbZEJdGWL+JtwwE4RRW6CmkNQABl1EQgf8qjAJD0xzKjizAYCzdoQN0kPKLj8ZU0FMSHG9 mQN+kC1nWydJBJZeBFhIpk9qeb2AWVw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1724774757; a=rsa-sha256; cv=none; b=YZl9l6P9Ls0J9OhNfrkEKSJyGeqoV5oCI/AgAWLLgbp2kCGEXIAZTkwoAmC8c6lz+yPkyO LrZR+RDsS+DD/xAvLtGwS0yEmZ9h3ls4dmRt2kizouEOKY8fO5UqmCtbnUqXoG7rsq8gvy zOq0u8498E8n4n7lD1lZeJdr3nx1S4Y= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=eHfC9fWz; spf=pass (imf20.hostedemail.com: domain of brauner@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=brauner@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by nyc.source.kernel.org (Postfix) with ESMTP id 24FB8A41FAA; Tue, 27 Aug 2024 16:07:16 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 38BFCC4FDE8; Tue, 27 Aug 2024 16:00:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1724774402; bh=1IKbiNGfmObraCG6cPeuTzv/L3Lu9auU2OH63obEB1o=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=eHfC9fWzsbA6YUnCXodcj+lciKGm5vf1SJgmid46erqddLezjgqoXx8w8UM5uFAQX Vg+GDTUfLTMOIj7Lmm0UC3qSI80uardtKVcAe+4vJxRDGVaaZ5jwXsflojMfqRVMdJ Aqtbn7Ra0ECmiaWzLHZe7Ip708yFAtVBeJgIj9c5Q2JlcZiEI8iJt7LcONTLnSP1tK mUJZzXNdd71ciQyB4luTO3P/NzA0h+u0AknL2A1AOgngt7tSO7mjWqYmbRyQMm1x4m k90avL5+GuDIEn5PmMSO0SfEWmHteZAgJ5qbLSxfrRmiLuoHHpZ9QB4csbUmF2D06N Wk1jDIro6YIKA== From: Christian Brauner Date: Tue, 27 Aug 2024 17:59:43 +0200 Subject: [PATCH v2 2/3] mm: add kmem_cache_create_rcu() MIME-Version: 1.0 Message-Id: <20240827-work-kmem_cache-rcu-v2-2-7bc9c90d5eef@kernel.org> References: <20240827-work-kmem_cache-rcu-v2-0-7bc9c90d5eef@kernel.org> In-Reply-To: <20240827-work-kmem_cache-rcu-v2-0-7bc9c90d5eef@kernel.org> To: Vlastimil Babka , Jens Axboe , "Paul E. McKenney" , Roman Gushchin , Jann Horn , Linus Torvalds , linux-mm@kvack.org Cc: Christian Brauner , linux-fsdevel@vger.kernel.org X-Mailer: b4 0.15-dev-37811 X-Developer-Signature: v=1; a=openpgp-sha256; l=11339; i=brauner@kernel.org; h=from:subject:message-id; bh=1IKbiNGfmObraCG6cPeuTzv/L3Lu9auU2OH63obEB1o=; b=owGbwMvMwCU28Zj0gdSKO4sYT6slMaSd/f57F/sVdrvz0+7prxDZfyRadwOr3T6W9ivM0spPS tlXPCni6yhlYRDjYpAVU2RxaDcJl1vOU7HZKFMDZg4rE8gQBi5OAZjIt+0M/xPuT2MWEnXdIco+ 87F/+6V77Ilzn60LPqbyf6X53ZZ+lksM/8wlNWfk/FG0nJzQHOBkbcDx74XxAu5JAZ25O6/Z3rT j5gIA X-Developer-Key: i=brauner@kernel.org; a=openpgp; fpr=4880B8C9BD0E5106FC070F4F7B3C391EFEA93624 X-Stat-Signature: pjid3cuk1yhhuxnr7ufwum4irx4j14ee X-Rspamd-Queue-Id: 440BB1C0024 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1724774843-882301 X-HE-Meta: U2FsdGVkX1/EgAI9+1Yt99hBQbmZk42bJfyVMIo8K7WL/HCyaSuK8JsxvuMFqu0myZYRobtJjrYeYtf6JuWEo5GC1MH7bFcFBPSeXyBadAuNCcI6PuJODEWfsg4dUp2Q3cwR7H1UHPiWXG5YGdyttTJEPxDPUXHd980LRCfrpgIrr7yHbP/hOriU348TM8neTqjnNl+yrZOIYFD79aY4VyH2lJHTZMIu2pKXyqgAlKJe7P3k0LZa3n4qplXY1rJzoohJm24Q6SgdsrTVLxFSQSGpcSPrVsj1DOBk8ctouSA7dJ3WwjmkJfbheBEMHTYxnNDSCg1Afadx81ZVPK3f6Sat88jF72vhCBTl6tKmVuCeD34sOFbi7oQ3DhzbCHKiemLhn3aW4HrFmC2x6j52KlbcwaQcw/EZvhqKzZ+7I1gdzG+MB5bpyizpMnuSd2AHv64UWiG/ePA9kPPg1Iir0fhxq9etRCSfAoZQ2iMO+q+4Lc9oI/vD+/RI5N5IDaqna5dY7WM40ty6lLOysK0Ar+Y5nxwCfQkwTYS+t+lVXHRrtL4xwCYzEQ24QU6b6Kl8wRN/+5jDSN65nEijkC7u4dV/cygFIRiXOLqflsKsV+YZPwdx4HNgFvM876UvNi8bZPygASm+TbJkUpKuhfBUnc7Er0F7B1QvLU1evwYHZ9vUfY9Vp0Eh3+Y1EdJmHS3JthrBlBLKS4obC9jtE8aUzBEaRYm+/v3imCUJOJhzx0bPVNYmYzO79pPjTPgyUm7QWY+VK4xoi5EURbZXUrOv2cM2+fR46XuwNaczwx/Q0yIGTb4ED6vl6IYXUfrJ33I7RYt0FwuV7qqtTdTYesSG/tutpLSnzRYJP0UDXgAMMVM+xtU/Sf7ia7l6ooHeqS6x+i9aeaz/6WrICqmzcM7chOZvlUUeHjpKmXcXfUyyEq2EANEzkVIeCEhDWaqu9M7BfFJTM5JO60aFwpQ5Q0M cnOAS8kf xUTmrFyLJJYVtaBdU0X+ANJlfmXj1Iy/L1jz+HfOZL9758+aX85HRYOs3zj5cKPm/oF76vT4FeyYg8qggXnbWK+SHjcTjyop8TR/Ayu/cMWzbTuA5Xd5hW4jOOnnkVi/I/O1Sr0uWIiUBiv6GCF8bwp9i1XrUMp+j51Z1h0ndxEPhLkjdmYYTVmyle5pxL8/rhyxa8knnbKX7uXDMRpMR076Xof2nrfDXBTeyibLubvSJHAnd9qcokVfRf5ku0y6wr7bXXw1qlRbxjrsfEI16ndDXmIgPEK8gVqnFRNHAVjdg9ytPHcS0QgiEtYfHDZokY3Y2 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When a kmem cache is created with SLAB_TYPESAFE_BY_RCU the free pointer must be located outside of the object because we don't know what part of the memory can safely be overwritten as it may be needed to prevent object recycling. That has the consequence that SLAB_TYPESAFE_BY_RCU may end up adding a new cacheline. This is the case for .e.g, struct file. After having it shrunk down by 40 bytes and having it fit in three cachelines we still have SLAB_TYPESAFE_BY_RCU adding a fourth cacheline because it needs to accomodate the free pointer and is hardware cacheline aligned. I tried to find ways to rectify this as struct file is pretty much everywhere and having it use less memory is a good thing. So here's a proposal. Signed-off-by: Christian Brauner --- include/linux/slab.h | 9 ++++ mm/slab.h | 1 + mm/slab_common.c | 133 ++++++++++++++++++++++++++++++++++++--------------- mm/slub.c | 17 ++++--- 4 files changed, 114 insertions(+), 46 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index eb2bf4629157..5b2da2cf31a8 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -212,6 +212,12 @@ enum _slab_flag_bits { #define SLAB_NO_OBJ_EXT __SLAB_FLAG_UNUSED #endif +/* + * freeptr_t represents a SLUB freelist pointer, which might be encoded + * and not dereferenceable if CONFIG_SLAB_FREELIST_HARDENED is enabled. + */ +typedef struct { unsigned long v; } freeptr_t; + /* * ZERO_SIZE_PTR will be returned for zero sized kmalloc requests. * @@ -242,6 +248,9 @@ struct kmem_cache *kmem_cache_create_usercopy(const char *name, slab_flags_t flags, unsigned int useroffset, unsigned int usersize, void (*ctor)(void *)); +struct kmem_cache *kmem_cache_create_rcu(const char *name, unsigned int size, + unsigned int freeptr_offset, + slab_flags_t flags); void kmem_cache_destroy(struct kmem_cache *s); int kmem_cache_shrink(struct kmem_cache *s); diff --git a/mm/slab.h b/mm/slab.h index dcdb56b8e7f5..b05512a14f07 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -261,6 +261,7 @@ struct kmem_cache { unsigned int object_size; /* Object size without metadata */ struct reciprocal_value reciprocal_size; unsigned int offset; /* Free pointer offset */ + unsigned int rcu_freeptr_offset; /* Specific free pointer requested */ #ifdef CONFIG_SLUB_CPU_PARTIAL /* Number of per cpu partial objects to keep around */ unsigned int cpu_partial; diff --git a/mm/slab_common.c b/mm/slab_common.c index c8dd7e08c5f6..c4beff642fff 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -202,9 +202,10 @@ struct kmem_cache *find_mergeable(unsigned int size, unsigned int align, } static struct kmem_cache *create_cache(const char *name, - unsigned int object_size, unsigned int align, - slab_flags_t flags, unsigned int useroffset, - unsigned int usersize, void (*ctor)(void *)) + unsigned int object_size, unsigned int freeptr_offset, + unsigned int align, slab_flags_t flags, + unsigned int useroffset, unsigned int usersize, + void (*ctor)(void *)) { struct kmem_cache *s; int err; @@ -212,6 +213,12 @@ static struct kmem_cache *create_cache(const char *name, if (WARN_ON(useroffset + usersize > object_size)) useroffset = usersize = 0; + err = -EINVAL; + if (freeptr_offset < UINT_MAX && + (freeptr_offset >= object_size || + (freeptr_offset && !(flags & SLAB_TYPESAFE_BY_RCU)))) + goto out; + err = -ENOMEM; s = kmem_cache_zalloc(kmem_cache, GFP_KERNEL); if (!s) @@ -219,13 +226,13 @@ static struct kmem_cache *create_cache(const char *name, s->name = name; s->size = s->object_size = object_size; + s->rcu_freeptr_offset = freeptr_offset; s->align = align; s->ctor = ctor; #ifdef CONFIG_HARDENED_USERCOPY s->useroffset = useroffset; s->usersize = usersize; #endif - err = __kmem_cache_create(s, flags); if (err) goto out_free_cache; @@ -240,38 +247,10 @@ static struct kmem_cache *create_cache(const char *name, return ERR_PTR(err); } -/** - * kmem_cache_create_usercopy - Create a cache with a region suitable - * for copying to userspace - * @name: A string which is used in /proc/slabinfo to identify this cache. - * @size: The size of objects to be created in this cache. - * @align: The required alignment for the objects. - * @flags: SLAB flags - * @useroffset: Usercopy region offset - * @usersize: Usercopy region size - * @ctor: A constructor for the objects. - * - * Cannot be called within a interrupt, but can be interrupted. - * The @ctor is run when new pages are allocated by the cache. - * - * The flags are - * - * %SLAB_POISON - Poison the slab with a known test pattern (a5a5a5a5) - * to catch references to uninitialised memory. - * - * %SLAB_RED_ZONE - Insert `Red` zones around the allocated memory to check - * for buffer overruns. - * - * %SLAB_HWCACHE_ALIGN - Align the objects in this cache to a hardware - * cacheline. This can be beneficial if you're counting cycles as closely - * as davem. - * - * Return: a pointer to the cache on success, NULL on failure. - */ -struct kmem_cache * -kmem_cache_create_usercopy(const char *name, - unsigned int size, unsigned int align, - slab_flags_t flags, +static struct kmem_cache * +do_kmem_cache_create_usercopy(const char *name, + unsigned int size, unsigned int freeptr_offset, + unsigned int align, slab_flags_t flags, unsigned int useroffset, unsigned int usersize, void (*ctor)(void *)) { @@ -331,7 +310,7 @@ kmem_cache_create_usercopy(const char *name, goto out_unlock; } - s = create_cache(cache_name, size, + s = create_cache(cache_name, size, freeptr_offset, calculate_alignment(flags, align, size), flags, useroffset, usersize, ctor); if (IS_ERR(s)) { @@ -355,6 +334,45 @@ kmem_cache_create_usercopy(const char *name, } return s; } + +/** + * kmem_cache_create_usercopy - Create a cache with a region suitable + * for copying to userspace + * @name: A string which is used in /proc/slabinfo to identify this cache. + * @size: The size of objects to be created in this cache. + * @freeptr_offset: Custom offset for the free pointer in RCU caches + * @align: The required alignment for the objects. + * @flags: SLAB flags + * @useroffset: Usercopy region offset + * @usersize: Usercopy region size + * @ctor: A constructor for the objects. + * + * Cannot be called within a interrupt, but can be interrupted. + * The @ctor is run when new pages are allocated by the cache. + * + * The flags are + * + * %SLAB_POISON - Poison the slab with a known test pattern (a5a5a5a5) + * to catch references to uninitialised memory. + * + * %SLAB_RED_ZONE - Insert `Red` zones around the allocated memory to check + * for buffer overruns. + * + * %SLAB_HWCACHE_ALIGN - Align the objects in this cache to a hardware + * cacheline. This can be beneficial if you're counting cycles as closely + * as davem. + * + * Return: a pointer to the cache on success, NULL on failure. + */ +struct kmem_cache * +kmem_cache_create_usercopy(const char *name, unsigned int size, + unsigned int align, slab_flags_t flags, + unsigned int useroffset, unsigned int usersize, + void (*ctor)(void *)) +{ + return do_kmem_cache_create_usercopy(name, size, UINT_MAX, align, flags, + useroffset, usersize, ctor); +} EXPORT_SYMBOL(kmem_cache_create_usercopy); /** @@ -386,11 +404,48 @@ struct kmem_cache * kmem_cache_create(const char *name, unsigned int size, unsigned int align, slab_flags_t flags, void (*ctor)(void *)) { - return kmem_cache_create_usercopy(name, size, align, flags, 0, 0, - ctor); + return do_kmem_cache_create_usercopy(name, size, UINT_MAX, align, flags, + 0, 0, ctor); } EXPORT_SYMBOL(kmem_cache_create); +/** + * kmem_cache_create_rcu - Create a SLAB_TYPESAFE_BY_RCU cache. + * @name: A string which is used in /proc/slabinfo to identify this cache. + * @size: The size of objects to be created in this cache. + * @freeptr_offset: The offset into the memory to the free pointer + * @flags: SLAB flags + * + * Cannot be called within a interrupt, but can be interrupted. + * The @ctor is run when new pages are allocated by the cache. + * + * See kmem_cache_create() for an explanation of possible @flags. + * + * By default SLAB_TYPESAFE_BY_RCU caches place free pointer outside of + * the object. This might cause the object to grow in size. Callers that + * have a reason to avoid this can specify a custom offset in their + * struct where the free pointer will be placed. + * + * Note that placing the free pointer inside the object requires the + * caller to ensure that no fields are invalidated that are required to + * guard against object recycling (See SLAB_TYPESAFE_BY_RCU for + * details.). + * + * Using zero as a value for @freeptr_offset is valid. To request no offset + * UINT_MAX can be specified. + * + * Return: a pointer to the cache on success, NULL on failure. + */ +struct kmem_cache *kmem_cache_create_rcu(const char *name, unsigned int size, + unsigned int freeptr_offset, + slab_flags_t flags) +{ + return do_kmem_cache_create_usercopy(name, size, freeptr_offset, 0, + flags | SLAB_TYPESAFE_BY_RCU, 0, 0, + NULL); +} +EXPORT_SYMBOL(kmem_cache_create_rcu); + static struct kmem_cache *kmem_buckets_cache __ro_after_init; /** diff --git a/mm/slub.c b/mm/slub.c index c9d8a2497fd6..b75f320e3963 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -465,12 +465,6 @@ static struct workqueue_struct *flushwq; * Core slab cache functions *******************************************************************/ -/* - * freeptr_t represents a SLUB freelist pointer, which might be encoded - * and not dereferenceable if CONFIG_SLAB_FREELIST_HARDENED is enabled. - */ -typedef struct { unsigned long v; } freeptr_t; - /* * Returns freelist pointer (ptr). With hardening, this is obfuscated * with an XOR of the address where the pointer is held and a per-cache @@ -5144,6 +5138,12 @@ static void set_cpu_partial(struct kmem_cache *s) #endif } +/* Was a valid freeptr offset requested? */ +static inline bool has_freeptr_offset(const struct kmem_cache *s) +{ + return s->rcu_freeptr_offset != UINT_MAX; +} + /* * calculate_sizes() determines the order and the distribution of data within * a slab object. @@ -5189,7 +5189,8 @@ static int calculate_sizes(struct kmem_cache *s) */ s->inuse = size; - if ((flags & (SLAB_TYPESAFE_BY_RCU | SLAB_POISON)) || s->ctor || + if (((flags & SLAB_TYPESAFE_BY_RCU) && !has_freeptr_offset(s)) || + (flags & SLAB_POISON) || s->ctor || ((flags & SLAB_RED_ZONE) && (s->object_size < sizeof(void *) || slub_debug_orig_size(s)))) { /* @@ -5210,6 +5211,8 @@ static int calculate_sizes(struct kmem_cache *s) */ s->offset = size; size += sizeof(void *); + } else if ((flags & SLAB_TYPESAFE_BY_RCU) && has_freeptr_offset(s)) { + s->offset = s->rcu_freeptr_offset; } else { /* * Store freelist pointer near middle of object to keep From patchwork Tue Aug 27 15:59:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Brauner X-Patchwork-Id: 13779787 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6DCF2C54731 for ; Tue, 27 Aug 2024 16:37:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B17A06B007B; Tue, 27 Aug 2024 12:37:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AC8606B0082; Tue, 27 Aug 2024 12:37:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 98F7C6B0083; Tue, 27 Aug 2024 12:37:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 7DD036B007B for ; Tue, 27 Aug 2024 12:37:26 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 2DB95A1946 for ; Tue, 27 Aug 2024 16:37:26 +0000 (UTC) X-FDA: 82498580892.11.C2FB830 Received: from nyc.source.kernel.org (nyc.source.kernel.org [147.75.193.91]) by imf23.hostedemail.com (Postfix) with ESMTP id 6D64814000F for ; Tue, 27 Aug 2024 16:37:23 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=XDQq7ZjD; spf=pass (imf23.hostedemail.com: domain of brauner@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=brauner@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1724776600; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=g7XDYs8Zib1E0KHu2lnyVJ0hUSitDf1ok71h59C/6Ng=; b=xvbDo/Cgo+ginwSQy7md6jqMRWrWX0flpWqfBl3X26PyaNiZlP41gZdSweCfn/EOrErMuY zfN86bOaVNXep3/wi77fnCsdN/6TXA0A4bqlmvUhrknx99XfqZy04IK9FnYDddJjspKtTJ HMfijApINwIF0HAyzvlDHYX6bujdPoo= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=XDQq7ZjD; spf=pass (imf23.hostedemail.com: domain of brauner@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=brauner@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1724776600; a=rsa-sha256; cv=none; b=JMGOH/dvfPEP2ABszFXv+BUxRG1EqJn8E1JpaeKPFy7TtkowtmqUO53bfo11MjZ/Z7pkq9 3UZlfYl8Gufux3XylDqiQAH2Ica0XZiQC9RTTBtEBxktM1DSVAhJfBy20p/NuVMKQBTJ2n XaXdLKE7tvB/JdMHr2De8FXgIUC4xQw= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by nyc.source.kernel.org (Postfix) with ESMTP id 96F21A4215D; Tue, 27 Aug 2024 16:37:15 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 783A1C4FE07; Tue, 27 Aug 2024 16:00:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1724774404; bh=DE+gGtjVtEol42ZCGLAdc2sv/qn+B/OiKlktxCin3uE=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=XDQq7ZjDwI8AC6Fj1oxAsuHrCGy9En+5jUbhNv0Hn009zPn4EP5k3gaSgiWsMIcmU v1aYprobuu5S8xNKWJJRLEpZ9Sn9FF9ZUjueUF9l2gAqAkHuB4G/via8om737bC4R7 s5ZFwup6mp0rtIPfyiegT5qfOvoG1HTnQupT70d7fQ2dqldnbQnk/T1tsD8Bfh+qGr JDkeBjOVndsuv1YbThNR/Y41t72q5vvxcypc6aj4ziqaQPe/U2JBo1+GjPpfSj1qPm yHjMC+JW2U2I1LdAjG6yiIVsNj5C6xrXW28N561WPBYxapRMufCgNcRPSZDMcUx28L 3xzdoo92Hg9NA== From: Christian Brauner Date: Tue, 27 Aug 2024 17:59:44 +0200 Subject: [PATCH v2 3/3] fs: use kmem_cache_create_rcu() MIME-Version: 1.0 Message-Id: <20240827-work-kmem_cache-rcu-v2-3-7bc9c90d5eef@kernel.org> References: <20240827-work-kmem_cache-rcu-v2-0-7bc9c90d5eef@kernel.org> In-Reply-To: <20240827-work-kmem_cache-rcu-v2-0-7bc9c90d5eef@kernel.org> To: Vlastimil Babka , Jens Axboe , "Paul E. McKenney" , Roman Gushchin , Jann Horn , Linus Torvalds , linux-mm@kvack.org Cc: Christian Brauner , linux-fsdevel@vger.kernel.org X-Mailer: b4 0.15-dev-37811 X-Developer-Signature: v=1; a=openpgp-sha256; l=1379; i=brauner@kernel.org; h=from:subject:message-id; bh=DE+gGtjVtEol42ZCGLAdc2sv/qn+B/OiKlktxCin3uE=; b=owGbwMvMwCU28Zj0gdSKO4sYT6slMaSd/f7767KcLA/hXX+3aE6tvXhpOj9fz6VNL8v+yF7fO GGj0dxcjo5SFgYxLgZZMUUWh3aTcLnlPBWbjTI1YOawMoEMYeDiFICJcKUw/C95ceaBUq2q+/RX X04fzUt9zn6zW5GxcvKWiepRa62Ss4UZ/lcZel3xF2AyljeveZNyn7PvRPK98DnzdspOlFy3QcT qCSMA X-Developer-Key: i=brauner@kernel.org; a=openpgp; fpr=4880B8C9BD0E5106FC070F4F7B3C391EFEA93624 X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: 6D64814000F X-Stat-Signature: m49xmq8sy85kjhqcoqhbr68sggz4j6wc X-HE-Tag: 1724776643-720914 X-HE-Meta: U2FsdGVkX183WTYpPaxQ/iCMFHwdV036MHa+5UPP90p1O6gOFL5ZIirzt2KNUa4MtJfrOVaZSOlsl9FByIGIdf05AlSSkGML489cmQc7IWKiwXMZpheEe750q5tT27h7AGvOCDwsoCKpC3nur/9sSBqx0bmarbGUR7BjU5zi1UcnKNM/Vy/E1cSD0tlBI3s5ysDiIkThwJNWgUODbPBvbGaPE+jb8Z5+i1IPkyCANesqM5cksBETBq0C9ljMnefyTzQIpcD3k4Si9+xJxoHFEv8AQS8drlUpvGMz7ypjSzijB9zrxFYNrMCVudFHmVND5pQ1arvgsXXBWJWwH3IFA6ViiSMDdoL6XATstgp7MrHASA072rsxvXZg8OAx0rSs5mlbBjdfn4PJHyHu47gX4KJeoktmG5DM3O2gtGmGUMQbeTbP6OOqT6vbgNYXyeaZaw1Op2IuMcqCzQgu6dGtovgmcv7WpujDnxs2CmnbK0j0ckSwCyuPG9qNLVCyKdsNId5q19Mpg30J79zp1Mm6SjetqT64BDSqkt8h71rSiJt6D/x1LQoVfNWdRQWoKBevKzYeahuogblWPiShe79BMGW+geEyaWLrJqlSgJ1l+YH3E5lYUewGlMWZMTUk9NAccyUvqbb7gwJgA+FW3fQLuuts1753kP+687aWcKSECT6EXUS4i7OOl439Wx/AeNBKdep5pTFy/Sh2v62ePyHs9tRYkUl1Oip62sNlttx0MZ2XDgANuXM/iViv3D31FoicAobGn1wuew4PkBekKuGplndUa1XXoSBkSSfdLCpkWLsYQguH9BQkIk0lMBc3OIuX24VByuUOL795BD0tvaqU/EpzQEGVZ/3MVvT1SX3R5bmAqn1JnsW+MpetdBFSP99Xz2NhTK3PgvymrTFPe47EkvDFe3cHkhQTSYB9iNwWKB3HaII+Lw+okiY3Bke8kpF9A5yrXJTqsj8Vcnomb5D vNfQISo4 jNUrqhORog5UETSoslfjySa+xIdYcPKB3Fv6kHrWemY6Rzl/V7Rkw9iH+gK16AHMl0vSme6l6T1soijdp1/FqeZeNGmX3nkc+WLb1FBXVS4mRyDthgQhVE6BErtWgtxEEi9HLAyur8CMqoPLvMxnrzBd/Et3UeLZiyj3sor+JsNmUTxWAA7sblu1feaKGpxNeG1u2COs6nZsGLYkykoXKh9l7HMgscdsOAtpxdFL+xqmkbCXi4S4o7yBUS0LlyJpkHTurRFHZeSY0fdo+GKifpGnzhfw51KVXG8oW7kr94wO5/BhAZK/dwdsu8c2H7YR92cCp X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Switch to the new kmem_cache_create_rcu() helper which allows us to use a custom free pointer offset avoiding the need to have an external free pointer which would grow struct file behind our backs. Signed-off-by: Christian Brauner --- fs/file_table.c | 6 +++--- include/linux/fs.h | 1 + 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/fs/file_table.c b/fs/file_table.c index 694199a1a966..83d5ac1fadc0 100644 --- a/fs/file_table.c +++ b/fs/file_table.c @@ -514,9 +514,9 @@ EXPORT_SYMBOL(__fput_sync); void __init files_init(void) { - filp_cachep = kmem_cache_create("filp", sizeof(struct file), 0, - SLAB_TYPESAFE_BY_RCU | SLAB_HWCACHE_ALIGN | - SLAB_PANIC | SLAB_ACCOUNT, NULL); + filp_cachep = kmem_cache_create_rcu("filp", sizeof(struct file), + offsetof(struct file, f_freeptr), + SLAB_HWCACHE_ALIGN | SLAB_PANIC | SLAB_ACCOUNT); percpu_counter_init(&nr_files, 0, GFP_KERNEL); } diff --git a/include/linux/fs.h b/include/linux/fs.h index 61097a9cf317..12a72f162da7 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -1057,6 +1057,7 @@ struct file { struct callback_head f_task_work; struct llist_node f_llist; struct file_ra_state f_ra; + freeptr_t f_freeptr; }; /* --- cacheline 3 boundary (192 bytes) --- */ } __randomize_layout