From patchwork Tue Sep 3 14:20:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Brauner X-Patchwork-Id: 13788765 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C5E736F315 for ; Tue, 3 Sep 2024 14:22:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725373345; cv=none; b=XgGJdaay4bwvWj27kWV8mgfMdFvmEJ2qwocLRS9tSkpHSpmvF6ZrCdEPq0c2Bk751dXSAHQ/HRLik6Pb29wy5XfWVAYUmtER50ciPltGmfpF3TxN8j2gzRRo1XBbFwOMchhJ8K2mE65NtpMV8HymHxvh2p3GIGzXUAmkUinZGrQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725373345; c=relaxed/simple; bh=aQiIKK7aypFaSSN7lgRJDvRI7/Ip5hPvhqIbIRsPc90=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=nMtsxvdnndDLfZff0ypP9IzQsj1VuEjfb6sGMB1QNyWUROF//UXV8uLrr2ATtcLHCrA/r4bATvIhlQeMKOGP/Wb21Bf778sHHw4MzJdSIhM0IEi7t90hqujLnUT7Wrsd57WUKOYyKTc7d4QIXbmI1VVVS/JEAMwR0SFQH3XZoaA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=j7WjnjG6; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="j7WjnjG6" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8DA9EC4CEC8; Tue, 3 Sep 2024 14:22:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1725373345; bh=aQiIKK7aypFaSSN7lgRJDvRI7/Ip5hPvhqIbIRsPc90=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=j7WjnjG6DU4tzE1BuBWHzDqxuj8IxdxSm8g0U15AEnbMBM8wttB1Y3G09ev5GLai5 W2xu715Lw9oCHM2xxc3zKsxgwCJlvcnbklI8orrNlXFMpmDE+/+EIkXe2HdsgB2AUw 7w5NIffNI05CLpefEIII4pV9TAVZt8+LH7bvSgW2lsoZFWpV/vjxSr8pH3VqHi15qD cQeUJebohkTHwEIHi5xUjNX8zFobHPdwFYu42KnOQBePw5MBtvUmDOOV8oi62Mo/Vk oziANUL3cw4Sh1Czk/9uSWtD9W51cHqOWLNO+FLe98JB/bZZtu4tHx/HPWN5Wnm3XP LAQw2zTFSQMBQ== From: Christian Brauner Date: Tue, 03 Sep 2024 16:20:42 +0200 Subject: [PATCH v2 01/15] sl*b: s/__kmem_cache_create/do_kmem_cache_create/g Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240903-work-kmem_cache_args-v2-1-76f97e9a4560@kernel.org> References: <20240903-work-kmem_cache_args-v2-0-76f97e9a4560@kernel.org> In-Reply-To: <20240903-work-kmem_cache_args-v2-0-76f97e9a4560@kernel.org> To: Vlastimil Babka , Jens Axboe , Jann Horn , Linus Torvalds , Mike Rapoport Cc: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Christian Brauner X-Mailer: b4 0.15-dev-37811 X-Developer-Signature: v=1; a=openpgp-sha256; l=1917; i=brauner@kernel.org; h=from:subject:message-id; bh=aQiIKK7aypFaSSN7lgRJDvRI7/Ip5hPvhqIbIRsPc90=; b=owGbwMvMwCU28Zj0gdSKO4sYT6slMaRdl55994PxvuwGq4XPVzybc53nc+GFXI8vs3/9VhDNM exdODGMt6OUhUGMi0FWTJHFod0kXG45T8Vmo0wNmDmsTCBDGLg4BWAidu8Z/mn7pr9d4vP/0D4O ibK9/vqX9kjKvtg1wTfqWyoXZ/XLma6MDPcTjRjfBf0+qqr9bb7gudfOv1ayefxlvdyk9OXzX5W HSWwA X-Developer-Key: i=brauner@kernel.org; a=openpgp; fpr=4880B8C9BD0E5106FC070F4F7B3C391EFEA93624 Free up reusing the double-underscore variant for follow-up patches. Signed-off-by: Christian Brauner Reviewed-by: Mike Rapoport (Microsoft) --- mm/slab.h | 2 +- mm/slab_common.c | 4 ++-- mm/slub.c | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/mm/slab.h b/mm/slab.h index a6051385186e..684bb48c4f39 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -424,7 +424,7 @@ kmalloc_slab(size_t size, kmem_buckets *b, gfp_t flags, unsigned long caller) gfp_t kmalloc_fix_flags(gfp_t flags); /* Functions provided by the slab allocators */ -int __kmem_cache_create(struct kmem_cache *, slab_flags_t flags); +int do_kmem_cache_create(struct kmem_cache *, slab_flags_t flags); void __init kmem_cache_init(void); extern void create_boot_cache(struct kmem_cache *, const char *name, diff --git a/mm/slab_common.c b/mm/slab_common.c index 95db3702f8d6..91e0e36e4379 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -234,7 +234,7 @@ static struct kmem_cache *create_cache(const char *name, s->useroffset = useroffset; s->usersize = usersize; #endif - err = __kmem_cache_create(s, flags); + err = do_kmem_cache_create(s, flags); if (err) goto out_free_cache; @@ -778,7 +778,7 @@ void __init create_boot_cache(struct kmem_cache *s, const char *name, s->usersize = usersize; #endif - err = __kmem_cache_create(s, flags); + err = do_kmem_cache_create(s, flags); if (err) panic("Creation of kmalloc slab %s size=%u failed. Reason %d\n", diff --git a/mm/slub.c b/mm/slub.c index 9aa5da1e8e27..23d9d783ff26 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -5902,7 +5902,7 @@ __kmem_cache_alias(const char *name, unsigned int size, unsigned int align, return s; } -int __kmem_cache_create(struct kmem_cache *s, slab_flags_t flags) +int do_kmem_cache_create(struct kmem_cache *s, slab_flags_t flags) { int err; From patchwork Tue Sep 3 14:20:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Brauner X-Patchwork-Id: 13788766 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 047186F315 for ; Tue, 3 Sep 2024 14:22:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725373348; cv=none; b=nyCnJZziJAYiq1y8ZT2KhFf4xjfPXUsyvmJFmrM/i7IidppEusKsRkxmax9Kuyp779N1MFYjAgdO7GoZtL1MlMbPJYauzKRoGmnSI7IQYKxUY4VuMWSroJK0PEn2TaNcGW7TzQxGibp/+KDX5Pd8lrmSZbqfVTumEqp6E38BqKU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725373348; c=relaxed/simple; bh=2h5DOrmEliYOEskMyLdOlS+a422lNEm7iLYnRuCn078=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=QEs10oCvu6StLC8czoUi8Aif5yMdRqqN6mIraI4sFJLw4HOa/Ai6GhLkatiEMzbr5zCszJJz3RHTFyAmi0gZzBqAUFUuthg/TGaL+EWXZeMKnC6cxIyTwQPc37iXqRCGQM/42H+SzpcwO9pZpjR1Dhu5Dden0P/wqiUjc9a+d/8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=iQJr2KSY; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="iQJr2KSY" Received: by smtp.kernel.org (Postfix) with ESMTPSA id D032DC4CEC7; Tue, 3 Sep 2024 14:22:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1725373347; bh=2h5DOrmEliYOEskMyLdOlS+a422lNEm7iLYnRuCn078=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=iQJr2KSYx/rknS/FfpX+JxiEROKLTM5PnAYZ+o7XEVv6NgP3Ss7tjJhNVLJDjAYmg V1w3/5f+g0CdyNTOTYOk+MyMT016PpeTQSQ9CVWlQjEmYiiidRhd5oBc/sSsG1zBIa e2QfgbzfX02r9yFo7QRX2G58gqj7RaPb+Tur00aybamWoo1hRqz8NyWQLbQXnL5bTq 9TJ8oEHrOK+F3C/N/OCCnqLEonwIvdHgPRKO+9swiWGr5ikMlWxa7lOaY4WvE2YHLg NAXHIZBEH3+euJk3IsDszxZJDMupI0XClXyaDEV4YW6f5o0YzZ0GNF3IuO1vHXJZhy CdnYHOYX2nb8w== From: Christian Brauner Date: Tue, 03 Sep 2024 16:20:43 +0200 Subject: [PATCH v2 02/15] slab: add struct kmem_cache_args Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240903-work-kmem_cache_args-v2-2-76f97e9a4560@kernel.org> References: <20240903-work-kmem_cache_args-v2-0-76f97e9a4560@kernel.org> In-Reply-To: <20240903-work-kmem_cache_args-v2-0-76f97e9a4560@kernel.org> To: Vlastimil Babka , Jens Axboe , Jann Horn , Linus Torvalds , Mike Rapoport Cc: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Christian Brauner X-Mailer: b4 0.15-dev-37811 X-Developer-Signature: v=1; a=openpgp-sha256; l=5053; i=brauner@kernel.org; h=from:subject:message-id; bh=2h5DOrmEliYOEskMyLdOlS+a422lNEm7iLYnRuCn078=; b=owGbwMvMwCU28Zj0gdSKO4sYT6slMaRdl559e5Opz1zVj/eFzHY8dLq47YuR2uY7dlX5wfVT8 wO/B7+80lHKwiDGxSArpsji0G4SLrecp2KzUaYGzBxWJpAhDFycAjCRgrmMDJvP/rf4f7iw78yF f8xdB6L/NnxZxs9957/CyR9zXj9dJerAyDC/eIqDhMKq5ZGlvj+2LF9+QOFBocTslZo74hdGpq9 /+YcBAA== X-Developer-Key: i=brauner@kernel.org; a=openpgp; fpr=4880B8C9BD0E5106FC070F4F7B3C391EFEA93624 Signed-off-by: Christian Brauner Reviewed-by: Mike Rapoport (Microsoft) --- include/linux/slab.h | 21 ++++++++++++++++ mm/slab_common.c | 67 +++++++++++++++++++++++++++++++++++++++------------- 2 files changed, 72 insertions(+), 16 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 5b2da2cf31a8..79d8c8bca4a4 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -240,6 +240,27 @@ struct mem_cgroup; */ bool slab_is_available(void); +/** + * @align: The required alignment for the objects. + * @useroffset: Usercopy region offset + * @usersize: Usercopy region size + * @freeptr_offset: Custom offset for the free pointer in RCU caches + * @use_freeptr_offset: Whether a @freeptr_offset is used + * @ctor: A constructor for the objects. + */ +struct kmem_cache_args { + unsigned int align; + unsigned int useroffset; + unsigned int usersize; + unsigned int freeptr_offset; + bool use_freeptr_offset; + void (*ctor)(void *); +}; + +struct kmem_cache *__kmem_cache_create_args(const char *name, + unsigned int object_size, + struct kmem_cache_args *args, + slab_flags_t flags); struct kmem_cache *kmem_cache_create(const char *name, unsigned int size, unsigned int align, slab_flags_t flags, void (*ctor)(void *)); diff --git a/mm/slab_common.c b/mm/slab_common.c index 91e0e36e4379..0f13c045b8d1 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -248,14 +248,24 @@ static struct kmem_cache *create_cache(const char *name, return ERR_PTR(err); } -static struct kmem_cache * -do_kmem_cache_create_usercopy(const char *name, - unsigned int size, unsigned int freeptr_offset, - unsigned int align, slab_flags_t flags, - unsigned int useroffset, unsigned int usersize, - void (*ctor)(void *)) +/** + * __kmem_cache_create_args - Create a kmem cache + * @name: A string which is used in /proc/slabinfo to identify this cache. + * @object_size: The size of objects to be created in this cache. + * @args: Arguments for the cache creation (see struct kmem_cache_args). + * @flags: See %SLAB_* flags for an explanation of individual @flags. + * + * Cannot be called within a interrupt, but can be interrupted. + * + * Return: a pointer to the cache on success, NULL on failure. + */ +struct kmem_cache *__kmem_cache_create_args(const char *name, + unsigned int object_size, + struct kmem_cache_args *args, + slab_flags_t flags) { struct kmem_cache *s = NULL; + unsigned int freeptr_offset = UINT_MAX; const char *cache_name; int err; @@ -275,7 +285,7 @@ do_kmem_cache_create_usercopy(const char *name, mutex_lock(&slab_mutex); - err = kmem_cache_sanity_check(name, size); + err = kmem_cache_sanity_check(name, object_size); if (err) { goto out_unlock; } @@ -296,12 +306,14 @@ do_kmem_cache_create_usercopy(const char *name, /* Fail closed on bad usersize of useroffset values. */ if (!IS_ENABLED(CONFIG_HARDENED_USERCOPY) || - WARN_ON(!usersize && useroffset) || - WARN_ON(size < usersize || size - usersize < useroffset)) - usersize = useroffset = 0; - - if (!usersize) - s = __kmem_cache_alias(name, size, align, flags, ctor); + WARN_ON(!args->usersize && args->useroffset) || + WARN_ON(object_size < args->usersize || + object_size - args->usersize < args->useroffset)) + args->usersize = args->useroffset = 0; + + if (!args->usersize) + s = __kmem_cache_alias(name, object_size, args->align, flags, + args->ctor); if (s) goto out_unlock; @@ -311,9 +323,11 @@ do_kmem_cache_create_usercopy(const char *name, goto out_unlock; } - s = create_cache(cache_name, size, freeptr_offset, - calculate_alignment(flags, align, size), - flags, useroffset, usersize, ctor); + if (args->use_freeptr_offset) + freeptr_offset = args->freeptr_offset; + s = create_cache(cache_name, object_size, freeptr_offset, + calculate_alignment(flags, args->align, object_size), + flags, args->useroffset, args->usersize, args->ctor); if (IS_ERR(s)) { err = PTR_ERR(s); kfree_const(cache_name); @@ -335,6 +349,27 @@ do_kmem_cache_create_usercopy(const char *name, } return s; } +EXPORT_SYMBOL(__kmem_cache_create_args); + +static struct kmem_cache * +do_kmem_cache_create_usercopy(const char *name, + unsigned int size, unsigned int freeptr_offset, + unsigned int align, slab_flags_t flags, + unsigned int useroffset, unsigned int usersize, + void (*ctor)(void *)) +{ + struct kmem_cache_args kmem_args = { + .align = align, + .use_freeptr_offset = freeptr_offset != UINT_MAX, + .freeptr_offset = freeptr_offset, + .useroffset = useroffset, + .usersize = usersize, + .ctor = ctor, + }; + + return __kmem_cache_create_args(name, size, &kmem_args, flags); +} + /** * kmem_cache_create_usercopy - Create a cache with a region suitable From patchwork Tue Sep 3 14:20:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Brauner X-Patchwork-Id: 13788767 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7A6151E502 for ; Tue, 3 Sep 2024 14:22:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725373350; cv=none; b=PX1ObkJIGsBODRm/cN9CGchD+Cb/Si/MSkA0dJwieVTEHJjBaHjNFXSgEGoWdi0ZBVxPM665Hq7415nkOxFPjDk/b6ij9Y3LUF+GcM02QbLOknpS6QA4ufqAGTuOJ+6PVP3FBAAy9Mpr7rD1uFPEj7AY4gzee4YJ7MrEArVcbLc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725373350; c=relaxed/simple; bh=B1znre1e72KpovENRq2WhUqNJBBTL5E5a2ArimpQcWQ=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=Rss0GLouExX8qAyccf/ao8KanOyENCIO0mtjh0lD2X2snn76LtbJCv8zgoAlvjUNb5gbRrUHnuPm15zRcMYQsIgMru5xnIIFXlfCjqcEdM8s8hZNcjY8/J5/psh/injPaIPbM7Xt/F4Eq11B31YuFk8F81yE0fajR8d07xFhT8g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=hGdsgjSh; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="hGdsgjSh" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2E388C4CEC4; Tue, 3 Sep 2024 14:22:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1725373349; bh=B1znre1e72KpovENRq2WhUqNJBBTL5E5a2ArimpQcWQ=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=hGdsgjShNU2trZTfO0WBjhdzdFWFkHmlXYOn8HDfCT0GXxvKeNgF4wR7j4364QNiY q1wD+/F1glEBVwIUOInyw4qaqmx4U/Oph+rmhif+EMEeXNW9a5ukgf44ThUWBmDYIi s6r+Feb7OUFSWPCiIeWlwCWwlrIDgqGQrCev2KyPGKL2d9Xrh3Kt5i2PDrOFFWYC2p Ps42dhybyCiqsS/P2ys4JCo0fFAdNK5dcBwvyUNllQ2ssvhGcnQjY7A3UMGp33KJlZ 14g0pUVTW37Xs8qGyENRNZfTRhVIZwvxc1frcIpy8vVjZFSQas0GJPOpGCeIHFzX4E 1f1beu4RpQiyw== From: Christian Brauner Date: Tue, 03 Sep 2024 16:20:44 +0200 Subject: [PATCH v2 03/15] slab: port kmem_cache_create() to struct kmem_cache_args Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240903-work-kmem_cache_args-v2-3-76f97e9a4560@kernel.org> References: <20240903-work-kmem_cache_args-v2-0-76f97e9a4560@kernel.org> In-Reply-To: <20240903-work-kmem_cache_args-v2-0-76f97e9a4560@kernel.org> To: Vlastimil Babka , Jens Axboe , Jann Horn , Linus Torvalds , Mike Rapoport Cc: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Christian Brauner X-Mailer: b4 0.15-dev-37811 X-Developer-Signature: v=1; a=openpgp-sha256; l=820; i=brauner@kernel.org; h=from:subject:message-id; bh=B1znre1e72KpovENRq2WhUqNJBBTL5E5a2ArimpQcWQ=; b=owGbwMvMwCU28Zj0gdSKO4sYT6slMaRdl57930r5SMjm0K8nn3zub9kt13bs1KNlE6zjn196/ CGhZP/b9x2lLAxiXAyyYoosDu0m4XLLeSo2G2VqwMxhZQIZwsDFKQAT2aTByPDq+TzWEP84jutP EhdmX+j96rHG9Zq3e8irf2+2pCxe53Sd4a/Ev+bF1kurI62YuGbzCcysF7Dsqp1xhqWiMWEC607 DC5wA X-Developer-Key: i=brauner@kernel.org; a=openpgp; fpr=4880B8C9BD0E5106FC070F4F7B3C391EFEA93624 Port kmem_cache_create() to struct kmem_cache_args. Signed-off-by: Christian Brauner Reviewed-by: Mike Rapoport (Microsoft) --- mm/slab_common.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/mm/slab_common.c b/mm/slab_common.c index 0f13c045b8d1..ac0832dac01e 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -439,8 +439,12 @@ struct kmem_cache * kmem_cache_create(const char *name, unsigned int size, unsigned int align, slab_flags_t flags, void (*ctor)(void *)) { - return do_kmem_cache_create_usercopy(name, size, UINT_MAX, align, flags, - 0, 0, ctor); + struct kmem_cache_args kmem_args = { + .align = align, + .ctor = ctor, + }; + + return __kmem_cache_create_args(name, size, &kmem_args, flags); } EXPORT_SYMBOL(kmem_cache_create); From patchwork Tue Sep 3 14:20:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Brauner X-Patchwork-Id: 13788768 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4C0F11E502 for ; Tue, 3 Sep 2024 14:22:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725373352; cv=none; b=tAi1mOf8qnYNduXjUoxXer0LQcUVx3HGVGlOMrOBlVhMN3Unvsht4JfIUCfJaV7lNCgGFuyl13ScM6jMw6uIBUakLkrZAsZwfbdTrmO1ZTZB7vlbrWvxgB2FaESW3cGPd8y5gmwU+eIzeIzCFGrKz8Ds+VrAzSUxNM8J5qyhPLs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725373352; c=relaxed/simple; bh=Kd1uMp+HUkedeShv2sPdOAndToo750mFLJVW3GciVU8=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=CeKct2dDiTkUDA+rR8qrfXxzsIE5Ztpog+NM6OulwBZozjgIn80lunW6Pha4fSOGuNYeke5xopCtIQzpaTE2BRz/ACcd7JmsB97jTBTQFeDBoPdCAillW/UwRDTdjv8nPhaH7XJRaSwiIkc00X18E0ZTRX2ZEpPay7WCZ4UffaY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=pVJqXNnq; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="pVJqXNnq" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 67AA4C4CECB; Tue, 3 Sep 2024 14:22:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1725373352; bh=Kd1uMp+HUkedeShv2sPdOAndToo750mFLJVW3GciVU8=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=pVJqXNnqYjpyuPF/kKeJ/msDo8LDirhJKxcVzSVn7jGoOyLMZ2EXXghasprE9pufQ nsM1+pfcpmCTfnI+Xk7jru584AUPl7jVdYtUCniXFOdwNzUUsY50ZS2tt62vETfhET zeEFzy9W1AshaSijXs20r8yLrPteIML1dTi9/GnZqons/TcQC6DryLuUPzDwBaJurq vqeEXvHKGC8YbzAqMK7hE/s+NQn91v8Kygf3A1/ZmWC3qwmFV6pQLZ6JJu+RVC4LmW 0kM1crsub4Y2/rH2WNdJgsRU9kKgMaKkPrkY0zIWAeIcQowvDzdck8rCkTDRGPf93d IyAJNVW5/3KdQ== From: Christian Brauner Date: Tue, 03 Sep 2024 16:20:45 +0200 Subject: [PATCH v2 04/15] slab: port kmem_cache_create_rcu() to struct kmem_cache_args Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240903-work-kmem_cache_args-v2-4-76f97e9a4560@kernel.org> References: <20240903-work-kmem_cache_args-v2-0-76f97e9a4560@kernel.org> In-Reply-To: <20240903-work-kmem_cache_args-v2-0-76f97e9a4560@kernel.org> To: Vlastimil Babka , Jens Axboe , Jann Horn , Linus Torvalds , Mike Rapoport Cc: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Christian Brauner X-Mailer: b4 0.15-dev-37811 X-Developer-Signature: v=1; a=openpgp-sha256; l=931; i=brauner@kernel.org; h=from:subject:message-id; bh=Kd1uMp+HUkedeShv2sPdOAndToo750mFLJVW3GciVU8=; b=owGbwMvMwCU28Zj0gdSKO4sYT6slMaRdl5793k/QulzHebby06KKRblnnhx44rZyyUXdmuem3 lNztXqVO0pZGMS4GGTFFFkc2k3C5ZbzVGw2ytSAmcPKBDKEgYtTACZincLIMPWl3gRFHUOh9u86 PqpMrrd2+//g/HxW5YVgcmtiMqPCU4b/0Wlrlt+fzrDsf16A6JusAplFhXIGM8y+v59xpv+cqPY 9HgA= X-Developer-Key: i=brauner@kernel.org; a=openpgp; fpr=4880B8C9BD0E5106FC070F4F7B3C391EFEA93624 Port kmem_cache_create_rcu() to struct kmem_cache_args. Signed-off-by: Christian Brauner Reviewed-by: Mike Rapoport (Microsoft) --- mm/slab_common.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/mm/slab_common.c b/mm/slab_common.c index ac0832dac01e..da62ed30f95d 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -481,9 +481,13 @@ struct kmem_cache *kmem_cache_create_rcu(const char *name, unsigned int size, unsigned int freeptr_offset, slab_flags_t flags) { - return do_kmem_cache_create_usercopy(name, size, freeptr_offset, 0, - flags | SLAB_TYPESAFE_BY_RCU, 0, 0, - NULL); + struct kmem_cache_args kmem_args = { + .freeptr_offset = freeptr_offset, + .use_freeptr_offset = true, + }; + + return __kmem_cache_create_args(name, size, &kmem_args, + flags | SLAB_TYPESAFE_BY_RCU); } EXPORT_SYMBOL(kmem_cache_create_rcu); From patchwork Tue Sep 3 14:20:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Brauner X-Patchwork-Id: 13788769 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9CD7A1E502 for ; Tue, 3 Sep 2024 14:22:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725373354; cv=none; b=E7uAYxXuGRRP03iBkoKK/7bFdNjaQjue4t2uvcHa3fYQLgBNCrICQDVfikZttKqwCVFKcBWeZH+kXyaPh9ES+n/JkCOdKVX96Byn7w1MboUn5FLc/0vuUK4wbrWeVUnTEXZ0RlqfgAGTZiAvskrwqvHlnBM2IVZQ9dO97LX2wl4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725373354; c=relaxed/simple; bh=qZgCti9Uoopmu2of9/KPoTLyubqgNTKqAU6SyDZvCz4=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=OKpbapQ1m3QvTTD71KY8vDDED034nUq0NyrkwaXnplhZSu5k/+T4w/3iXxKD5/IB6GJYC6LoOl9Mv9y5gcTzSYZHpicJimrDYdhfCqlz86WC8G7PHAmDo5hwSdZkHVDLK6YGblsVYIsgPn8f6o7W2JX6ztBYqkymREM7opjF7Kg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=lnwsuY/D; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="lnwsuY/D" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9D591C4CECA; Tue, 3 Sep 2024 14:22:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1725373354; bh=qZgCti9Uoopmu2of9/KPoTLyubqgNTKqAU6SyDZvCz4=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=lnwsuY/Dm0ZSsZkzBUqNed0O6oKMY249pFL4KNk8UDjxunv1NStiWR9awREnGzVT3 KBca6yWLYFHtQw02kRp1g6S93mGZZkvCh5SMf91cFE7Mul25Dqh4HgLh5x7QqOwd+a IASO+TMGu/s62B+KpNVrBsap/j/xFtqNvZeb+reXcuH6Akbp9Civ9Iho/TgGCsQpCA aS9Sl+/mzHzSVEnoGfBPXOvJcKIw76woMywoZxaL0W32AZxRLrz0Pnc7TYIap5nl7y A8+0eYI0Q8g9DM/vln56yslETXQgU1VF6SjvAlSG5UkpgA16SsQBPb+vrEyZK6c+6g lX57Vb026CJPA== From: Christian Brauner Date: Tue, 03 Sep 2024 16:20:46 +0200 Subject: [PATCH v2 05/15] slab: port kmem_cache_create_usercopy() to struct kmem_cache_args Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240903-work-kmem_cache_args-v2-5-76f97e9a4560@kernel.org> References: <20240903-work-kmem_cache_args-v2-0-76f97e9a4560@kernel.org> In-Reply-To: <20240903-work-kmem_cache_args-v2-0-76f97e9a4560@kernel.org> To: Vlastimil Babka , Jens Axboe , Jann Horn , Linus Torvalds , Mike Rapoport Cc: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Christian Brauner X-Mailer: b4 0.15-dev-37811 X-Developer-Signature: v=1; a=openpgp-sha256; l=1887; i=brauner@kernel.org; h=from:subject:message-id; bh=qZgCti9Uoopmu2of9/KPoTLyubqgNTKqAU6SyDZvCz4=; b=owGbwMvMwCU28Zj0gdSKO4sYT6slMaRdl55dWRq+dU34k3mzC1vupey0rti+eO9s/2fNCoxHp X382RXfdZSyMIhxMciKKbI4tJuEyy3nqdhslKkBM4eVCWQIAxenAEykKpPhr7Qby4t/pje7HY9u d25SzA3LY3tX8d15WbGbx7uqade+TGJkeHz0ycy1v7eviJaabRm/Ruw2154d1/h0nodxKkyfx73 3DRcA X-Developer-Key: i=brauner@kernel.org; a=openpgp; fpr=4880B8C9BD0E5106FC070F4F7B3C391EFEA93624 Pprt kmem_cache_create_usercopy() to struct kmem_cache_args and remove the now unused do_kmem_cache_create_usercopy() helper. Signed-off-by: Christian Brauner Reviewed-by: Mike Rapoport (Microsoft) --- mm/slab_common.c | 30 ++++++++---------------------- 1 file changed, 8 insertions(+), 22 deletions(-) diff --git a/mm/slab_common.c b/mm/slab_common.c index da62ed30f95d..16c36a946135 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -351,26 +351,6 @@ struct kmem_cache *__kmem_cache_create_args(const char *name, } EXPORT_SYMBOL(__kmem_cache_create_args); -static struct kmem_cache * -do_kmem_cache_create_usercopy(const char *name, - unsigned int size, unsigned int freeptr_offset, - unsigned int align, slab_flags_t flags, - unsigned int useroffset, unsigned int usersize, - void (*ctor)(void *)) -{ - struct kmem_cache_args kmem_args = { - .align = align, - .use_freeptr_offset = freeptr_offset != UINT_MAX, - .freeptr_offset = freeptr_offset, - .useroffset = useroffset, - .usersize = usersize, - .ctor = ctor, - }; - - return __kmem_cache_create_args(name, size, &kmem_args, flags); -} - - /** * kmem_cache_create_usercopy - Create a cache with a region suitable * for copying to userspace @@ -405,8 +385,14 @@ kmem_cache_create_usercopy(const char *name, unsigned int size, unsigned int useroffset, unsigned int usersize, void (*ctor)(void *)) { - return do_kmem_cache_create_usercopy(name, size, UINT_MAX, align, flags, - useroffset, usersize, ctor); + struct kmem_cache_args kmem_args = { + .align = align, + .ctor = ctor, + .useroffset = useroffset, + .usersize = usersize, + }; + + return __kmem_cache_create_args(name, size, &kmem_args, flags); } EXPORT_SYMBOL(kmem_cache_create_usercopy); From patchwork Tue Sep 3 14:20:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Brauner X-Patchwork-Id: 13788770 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 12A39626CB for ; Tue, 3 Sep 2024 14:22:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725373357; cv=none; b=O5Tx4ya9zhz88Wmulhh/0Qr17paWDjuO/meqd3/qj0JY9vC6ZA2pisexaW4j9d+kLOBm5lKf89dj1ubkDk7/aUrJWoLidHqmqPAURZBd1NC0P1esAp1wMjZia0aD6VRVkiunIbye6gv/xA4l9GPrjUbeQAtivMY2XPwk3VzezFU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725373357; c=relaxed/simple; bh=Kwi+I1VrtzNjFT2qOG43GSdAkcM45KGq+XLRDdRsBW4=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=phG79IvGbLQX9/ksKVg6C9tnVPddGJNy5dO2wau0/dIxc78bzCzbfN0S5STA6A9CZRr4xA1Y8ErDxRBcT2UYmTrWKBoTV3ZW+befaAYDZFSoAVFsE2IPUhmoyFSveBfyv00j9rlDP/X1UIBSAZrLKW+cmpZRw0yhAIEGzXmXr/Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=HU3gT6w0; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="HU3gT6w0" Received: by smtp.kernel.org (Postfix) with ESMTPSA id E6A06C4CEC9; Tue, 3 Sep 2024 14:22:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1725373356; bh=Kwi+I1VrtzNjFT2qOG43GSdAkcM45KGq+XLRDdRsBW4=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=HU3gT6w0kxMRCh5cjqQ6GVZT9zHNO+tW5OEaJBvo0wqjvITuNxXJnWE0javyOTqvr FQwWfOYB3XwW8uiTGL2BRtw0FoBY4nIFYHGzhueA7CW6+ZyWob9/v19WpyvgykMF4F 0P5Y++52gJVpGcEw0uSSOQl7l/1cihjlemmOLneEIhFzMqJeYJJTjt7Sh9wrB3qQZL zetPgbUDZvBTIRqtEdsoFD8Ulima2cMJVWDVNxfp/B1qF2Xnx2SVhJALGh05CPWFwm 6zvELTznDpr/2y7c4+PII7Y1eE8bujIG2SPWHhcRfP6+MLYRrjAPDYDXSFPKDYKOUF OXufpqNfUg41g== From: Christian Brauner Date: Tue, 03 Sep 2024 16:20:47 +0200 Subject: [PATCH v2 06/15] slab: pass struct kmem_cache_args to create_cache() Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240903-work-kmem_cache_args-v2-6-76f97e9a4560@kernel.org> References: <20240903-work-kmem_cache_args-v2-0-76f97e9a4560@kernel.org> In-Reply-To: <20240903-work-kmem_cache_args-v2-0-76f97e9a4560@kernel.org> To: Vlastimil Babka , Jens Axboe , Jann Horn , Linus Torvalds , Mike Rapoport Cc: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Christian Brauner X-Mailer: b4 0.15-dev-37811 X-Developer-Signature: v=1; a=openpgp-sha256; l=3048; i=brauner@kernel.org; h=from:subject:message-id; bh=Kwi+I1VrtzNjFT2qOG43GSdAkcM45KGq+XLRDdRsBW4=; b=owGbwMvMwCU28Zj0gdSKO4sYT6slMaRdl54t6HCjduvW5zuL87Xma60vmixz2EVi+v1EUzOu/ KIdp1n2dZSyMIhxMciKKbI4tJuEyy3nqdhslKkBM4eVCWQIAxenAEzkdTcjw1bOm71eDxO4vzrM tfp8IvronE09clPakure3Xb5PMPU4hQjw+M3m9gORTG+O3BeYtH19PclB7bZnHnHsN+5Xm53kE4 CIzsA X-Developer-Key: i=brauner@kernel.org; a=openpgp; fpr=4880B8C9BD0E5106FC070F4F7B3C391EFEA93624 Pass struct kmem_cache_args to create_cache() so that we can later simplify further helpers. Signed-off-by: Christian Brauner Reviewed-by: Mike Rapoport (Microsoft) --- mm/slab_common.c | 39 +++++++++++++++++++-------------------- 1 file changed, 19 insertions(+), 20 deletions(-) diff --git a/mm/slab_common.c b/mm/slab_common.c index 16c36a946135..9baa61c9c670 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -202,22 +202,22 @@ struct kmem_cache *find_mergeable(unsigned int size, unsigned int align, } static struct kmem_cache *create_cache(const char *name, - unsigned int object_size, unsigned int freeptr_offset, - unsigned int align, slab_flags_t flags, - unsigned int useroffset, unsigned int usersize, - void (*ctor)(void *)) + unsigned int object_size, + struct kmem_cache_args *args, + slab_flags_t flags) { struct kmem_cache *s; int err; - if (WARN_ON(useroffset + usersize > object_size)) - useroffset = usersize = 0; + if (WARN_ON(args->useroffset + args->usersize > object_size)) + args->useroffset = args->usersize = 0; /* If a custom freelist pointer is requested make sure it's sane. */ err = -EINVAL; - if (freeptr_offset != UINT_MAX && - (freeptr_offset >= object_size || !(flags & SLAB_TYPESAFE_BY_RCU) || - !IS_ALIGNED(freeptr_offset, sizeof(freeptr_t)))) + if (args->use_freeptr_offset && + (args->freeptr_offset >= object_size || + !(flags & SLAB_TYPESAFE_BY_RCU) || + !IS_ALIGNED(args->freeptr_offset, sizeof(freeptr_t)))) goto out; err = -ENOMEM; @@ -227,12 +227,15 @@ static struct kmem_cache *create_cache(const char *name, s->name = name; s->size = s->object_size = object_size; - s->rcu_freeptr_offset = freeptr_offset; - s->align = align; - s->ctor = ctor; + if (args->use_freeptr_offset) + s->rcu_freeptr_offset = args->freeptr_offset; + else + s->rcu_freeptr_offset = UINT_MAX; + s->align = args->align; + s->ctor = args->ctor; #ifdef CONFIG_HARDENED_USERCOPY - s->useroffset = useroffset; - s->usersize = usersize; + s->useroffset = args->useroffset; + s->usersize = args->usersize; #endif err = do_kmem_cache_create(s, flags); if (err) @@ -265,7 +268,6 @@ struct kmem_cache *__kmem_cache_create_args(const char *name, slab_flags_t flags) { struct kmem_cache *s = NULL; - unsigned int freeptr_offset = UINT_MAX; const char *cache_name; int err; @@ -323,11 +325,8 @@ struct kmem_cache *__kmem_cache_create_args(const char *name, goto out_unlock; } - if (args->use_freeptr_offset) - freeptr_offset = args->freeptr_offset; - s = create_cache(cache_name, object_size, freeptr_offset, - calculate_alignment(flags, args->align, object_size), - flags, args->useroffset, args->usersize, args->ctor); + args->align = calculate_alignment(flags, args->align, object_size); + s = create_cache(cache_name, object_size, args, flags); if (IS_ERR(s)) { err = PTR_ERR(s); kfree_const(cache_name); From patchwork Tue Sep 3 14:20:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Brauner X-Patchwork-Id: 13788771 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 613AF626CB for ; Tue, 3 Sep 2024 14:22:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725373359; cv=none; b=lwtVd7tXT/bMu277+hIvYpJvGfgUGbTIl1yl0WEEZZCiQtJcR+7RrVW/IFYmUIy6Sc1CYBREy+g2fsAH/EToZ74OY/cRANb49ppblTVnK4G9fvsWswlKdMh++CSdTdnxMpekhUzbaxgQK4J49X1a8ZMOZpVtPdfQ7TRfQpuL5DI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725373359; c=relaxed/simple; bh=NICKa5KUOQG+AjRPua/q+Zy7rozMk7chEiygU85Xzfw=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=LtfzUarwoMsLIIQvTI0Nf9GCvFWsZEp4MkUtbkUD4hNaidaAnNitUuf7SHMQYAfGHgAeadl5593S+/mwNGqeRWeFjk2nE5u5DsvJSAZsLDSejBcFLk8LA9d+HZSOTUhoZ9iVLLtBcY5gHaRxViiiWQyx0E91pYlUo6szSD82Qw4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=iC8yw4Sz; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="iC8yw4Sz" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2F9BDC4CECA; Tue, 3 Sep 2024 14:22:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1725373359; bh=NICKa5KUOQG+AjRPua/q+Zy7rozMk7chEiygU85Xzfw=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=iC8yw4SzaTmXOI0hC0s8OsuBU5aCzpjtbsOXChYJgSb+rOOw8TJqTrZReqPkClcGK ZNgy01bxU/uKUhxj7oMkaQhQAQx/3DyNu5CeaR/EV1moRwRvMHEMwMUAnyjEI/TDMW skD2yBxQE+vFthDy+IfReCtlevTldr3uCMhDIvE8Z+T4hQhePNd2ER/HHlrrFD/4JJ 0kGkpS2svFTSfpQC/Q40LPhG3ke3P4HRh3ee0tF902OveKbf7i3QXXtYpW3TGeBec6 Vrza4B7lPz457uYQqrZhBLocJthwFr5XglrmZMgfN9bATDcthv/vrHLusm04OqJSJH +goSwi9E619bA== From: Christian Brauner Date: Tue, 03 Sep 2024 16:20:48 +0200 Subject: [PATCH v2 07/15] slub: pull kmem_cache_open() into do_kmem_cache_create() Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240903-work-kmem_cache_args-v2-7-76f97e9a4560@kernel.org> References: <20240903-work-kmem_cache_args-v2-0-76f97e9a4560@kernel.org> In-Reply-To: <20240903-work-kmem_cache_args-v2-0-76f97e9a4560@kernel.org> To: Vlastimil Babka , Jens Axboe , Jann Horn , Linus Torvalds , Mike Rapoport Cc: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Christian Brauner X-Mailer: b4 0.15-dev-37811 X-Developer-Signature: v=1; a=openpgp-sha256; l=4182; i=brauner@kernel.org; h=from:subject:message-id; bh=NICKa5KUOQG+AjRPua/q+Zy7rozMk7chEiygU85Xzfw=; b=owGbwMvMwCU28Zj0gdSKO4sYT6slMaRdl55TEKXz8vYFK94ZvtNPxHhluBqE8J9eprm0vqG4t aFTqWhWRykLgxgXg6yYIotDu0m43HKeis1GmRowc1iZQIYwcHEKwETUFzAyHK382DX77OSW782/ Tyb0tolO+W8Z9G1hP99sobz366eZKTP8L/Lb0Wjtcnz14nu562PZJ+ZXWnRa+BqwelZ9XbLf9vo 9VgA= X-Developer-Key: i=brauner@kernel.org; a=openpgp; fpr=4880B8C9BD0E5106FC070F4F7B3C391EFEA93624 do_kmem_cache_create() is the only caller and we're going to pass down struct kmem_cache_args in a follow-up patch. Signed-off-by: Christian Brauner Reviewed-by: Mike Rapoport (Microsoft) --- mm/slub.c | 132 +++++++++++++++++++++++++++++--------------------------------- 1 file changed, 62 insertions(+), 70 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 23d9d783ff26..30f4ca6335c7 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -5290,65 +5290,6 @@ static int calculate_sizes(struct kmem_cache *s) return !!oo_objects(s->oo); } -static int kmem_cache_open(struct kmem_cache *s, slab_flags_t flags) -{ - s->flags = kmem_cache_flags(flags, s->name); -#ifdef CONFIG_SLAB_FREELIST_HARDENED - s->random = get_random_long(); -#endif - - if (!calculate_sizes(s)) - goto error; - if (disable_higher_order_debug) { - /* - * Disable debugging flags that store metadata if the min slab - * order increased. - */ - if (get_order(s->size) > get_order(s->object_size)) { - s->flags &= ~DEBUG_METADATA_FLAGS; - s->offset = 0; - if (!calculate_sizes(s)) - goto error; - } - } - -#ifdef system_has_freelist_aba - if (system_has_freelist_aba() && !(s->flags & SLAB_NO_CMPXCHG)) { - /* Enable fast mode */ - s->flags |= __CMPXCHG_DOUBLE; - } -#endif - - /* - * The larger the object size is, the more slabs we want on the partial - * list to avoid pounding the page allocator excessively. - */ - s->min_partial = min_t(unsigned long, MAX_PARTIAL, ilog2(s->size) / 2); - s->min_partial = max_t(unsigned long, MIN_PARTIAL, s->min_partial); - - set_cpu_partial(s); - -#ifdef CONFIG_NUMA - s->remote_node_defrag_ratio = 1000; -#endif - - /* Initialize the pre-computed randomized freelist if slab is up */ - if (slab_state >= UP) { - if (init_cache_random_seq(s)) - goto error; - } - - if (!init_kmem_cache_nodes(s)) - goto error; - - if (alloc_kmem_cache_cpus(s)) - return 0; - -error: - __kmem_cache_release(s); - return -EINVAL; -} - static void list_slab_objects(struct kmem_cache *s, struct slab *slab, const char *text) { @@ -5904,26 +5845,77 @@ __kmem_cache_alias(const char *name, unsigned int size, unsigned int align, int do_kmem_cache_create(struct kmem_cache *s, slab_flags_t flags) { - int err; + int err = -EINVAL; - err = kmem_cache_open(s, flags); - if (err) - return err; + s->flags = kmem_cache_flags(flags, s->name); +#ifdef CONFIG_SLAB_FREELIST_HARDENED + s->random = get_random_long(); +#endif + + if (!calculate_sizes(s)) + goto out; + if (disable_higher_order_debug) { + /* + * Disable debugging flags that store metadata if the min slab + * order increased. + */ + if (get_order(s->size) > get_order(s->object_size)) { + s->flags &= ~DEBUG_METADATA_FLAGS; + s->offset = 0; + if (!calculate_sizes(s)) + goto out; + } + } + +#ifdef system_has_freelist_aba + if (system_has_freelist_aba() && !(s->flags & SLAB_NO_CMPXCHG)) { + /* Enable fast mode */ + s->flags |= __CMPXCHG_DOUBLE; + } +#endif + + /* + * The larger the object size is, the more slabs we want on the partial + * list to avoid pounding the page allocator excessively. + */ + s->min_partial = min_t(unsigned long, MAX_PARTIAL, ilog2(s->size) / 2); + s->min_partial = max_t(unsigned long, MIN_PARTIAL, s->min_partial); + + set_cpu_partial(s); + +#ifdef CONFIG_NUMA + s->remote_node_defrag_ratio = 1000; +#endif + + /* Initialize the pre-computed randomized freelist if slab is up */ + if (slab_state >= UP) { + if (init_cache_random_seq(s)) + goto out; + } + + if (!init_kmem_cache_nodes(s)) + goto out; + + if (!alloc_kmem_cache_cpus(s)) + goto out; /* Mutex is not taken during early boot */ - if (slab_state <= UP) - return 0; + if (slab_state <= UP) { + err = 0; + goto out; + } err = sysfs_slab_add(s); - if (err) { - __kmem_cache_release(s); - return err; - } + if (err) + goto out; if (s->flags & SLAB_STORE_USER) debugfs_slab_add(s); - return 0; +out: + if (err) + __kmem_cache_release(s); + return err; } #ifdef SLAB_SUPPORTS_SYSFS From patchwork Tue Sep 3 14:20:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Brauner X-Patchwork-Id: 13788772 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 605BE69D31 for ; Tue, 3 Sep 2024 14:22:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725373361; cv=none; b=EAgafqW+usFMrV9mPejWrkWYe5e3uD85dLAmtrdadvMi1dTebehHhMSDnlQcd7/Cv5ui1ooQrV0lzU6DOMTbDvhx4NQg21+OKXyjppuv/x2SIuV4V0q65xQlYUACRNvwxwc+EZvBli/fFiNbQCl2wLOwPSPs83prbTydKv2EFcU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725373361; c=relaxed/simple; bh=LGtj/e6uWtTAC/2FfRENwjsuEXl8qpeXQfpfD6hAo/g=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=vB/cXUPZDaEbWlEMABW7l0XoeicSnfwxl2R3XQFVfSPToqulci9B+9sPjndg1ydpYRA0Vr5KPI/LdE5VMXdhjsgbOQudb5rrhqXQWb/fptAct6ZmnE9Aaqp6kS7hRfJEG/UYGDrgWmOdjNnqkyYHs29NorYaT0QT+DADMSgKo3A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=RsLWeAqc; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="RsLWeAqc" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8786DC4CEC4; Tue, 3 Sep 2024 14:22:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1725373361; bh=LGtj/e6uWtTAC/2FfRENwjsuEXl8qpeXQfpfD6hAo/g=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=RsLWeAqc+DgpjX7fIXp3seQzC7Jou5QA3YNKQA96AB5ngZNz4PJaisGMq3sCixh5r QOgA1QJdHZQNaRLR/leMrddYpzKBCE8vWScxNK+YiFqISZSXv84tau6h7mXHEVcpr0 HJYqlDlVWjsSc0SNm9i+pXjDw6ODPRoFutRQQfM3i/wA8qynyXkDQggEHsE/XfNbm/ mjXtmxEjDfgHqwCzhhcHS3NajIW1E/l41DICazy+Jl/afIq9+LHEEQ2buTp2tP1QlJ 1FKl0p/6/MdN+LRySJdd4+H+ILs5xRe5Gg8ifJHwc11AtaWaRaI8iB+luaLgDbU7YU N8CBqhbYiO10w== From: Christian Brauner Date: Tue, 03 Sep 2024 16:20:49 +0200 Subject: [PATCH v2 08/15] slab: pass struct kmem_cache_args to do_kmem_cache_create() Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240903-work-kmem_cache_args-v2-8-76f97e9a4560@kernel.org> References: <20240903-work-kmem_cache_args-v2-0-76f97e9a4560@kernel.org> In-Reply-To: <20240903-work-kmem_cache_args-v2-0-76f97e9a4560@kernel.org> To: Vlastimil Babka , Jens Axboe , Jann Horn , Linus Torvalds , Mike Rapoport Cc: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Christian Brauner X-Mailer: b4 0.15-dev-37811 X-Developer-Signature: v=1; a=openpgp-sha256; l=3814; i=brauner@kernel.org; h=from:subject:message-id; bh=LGtj/e6uWtTAC/2FfRENwjsuEXl8qpeXQfpfD6hAo/g=; b=owGbwMvMwCU28Zj0gdSKO4sYT6slMaRdl57zp/8H58sZpdqnOv5v877EcHrts//a9Txn2or0d 272eOaU1FHKwiDGxSArpsji0G4SLrecp2KzUaYGzBxWJpAhDFycAjCRDW8ZGTrXHOfxNkk++6Bu Ud50/7Xu9itYrLle9Rboh6fcuH56Axsjw7cazcD7wjNt1i5aLMG87YLj89YNW27YnctQMQqyfjp Bnx0A X-Developer-Key: i=brauner@kernel.org; a=openpgp; fpr=4880B8C9BD0E5106FC070F4F7B3C391EFEA93624 and initialize most things in do_kmem_cache_create(). In a follow-up patch we'll remove rcu_freeptr_offset from struct kmem_cache. Signed-off-by: Christian Brauner Reviewed-by: Mike Rapoport (Microsoft) --- mm/slab.h | 4 +++- mm/slab_common.c | 27 ++++++--------------------- mm/slub.c | 17 ++++++++++++++++- 3 files changed, 25 insertions(+), 23 deletions(-) diff --git a/mm/slab.h b/mm/slab.h index 684bb48c4f39..c7a4e0fc3cf1 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -424,7 +424,9 @@ kmalloc_slab(size_t size, kmem_buckets *b, gfp_t flags, unsigned long caller) gfp_t kmalloc_fix_flags(gfp_t flags); /* Functions provided by the slab allocators */ -int do_kmem_cache_create(struct kmem_cache *, slab_flags_t flags); +int do_kmem_cache_create(struct kmem_cache *s, const char *name, + unsigned int size, struct kmem_cache_args *args, + slab_flags_t flags); void __init kmem_cache_init(void); extern void create_boot_cache(struct kmem_cache *, const char *name, diff --git a/mm/slab_common.c b/mm/slab_common.c index 9baa61c9c670..19ae3dd6e36f 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -224,20 +224,7 @@ static struct kmem_cache *create_cache(const char *name, s = kmem_cache_zalloc(kmem_cache, GFP_KERNEL); if (!s) goto out; - - s->name = name; - s->size = s->object_size = object_size; - if (args->use_freeptr_offset) - s->rcu_freeptr_offset = args->freeptr_offset; - else - s->rcu_freeptr_offset = UINT_MAX; - s->align = args->align; - s->ctor = args->ctor; -#ifdef CONFIG_HARDENED_USERCOPY - s->useroffset = args->useroffset; - s->usersize = args->usersize; -#endif - err = do_kmem_cache_create(s, flags); + err = do_kmem_cache_create(s, name, object_size, args, flags); if (err) goto out_free_cache; @@ -788,9 +775,7 @@ void __init create_boot_cache(struct kmem_cache *s, const char *name, { int err; unsigned int align = ARCH_KMALLOC_MINALIGN; - - s->name = name; - s->size = s->object_size = size; + struct kmem_cache_args kmem_args = {}; /* * kmalloc caches guarantee alignment of at least the largest @@ -799,14 +784,14 @@ void __init create_boot_cache(struct kmem_cache *s, const char *name, */ if (flags & SLAB_KMALLOC) align = max(align, 1U << (ffs(size) - 1)); - s->align = calculate_alignment(flags, align, size); + kmem_args.align = calculate_alignment(flags, align, size); #ifdef CONFIG_HARDENED_USERCOPY - s->useroffset = useroffset; - s->usersize = usersize; + kmem_args.useroffset = useroffset; + kmem_args.usersize = usersize; #endif - err = do_kmem_cache_create(s, flags); + err = do_kmem_cache_create(s, name, size, &kmem_args, flags); if (err) panic("Creation of kmalloc slab %s size=%u failed. Reason %d\n", diff --git a/mm/slub.c b/mm/slub.c index 30f4ca6335c7..4719b60215b8 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -5843,14 +5843,29 @@ __kmem_cache_alias(const char *name, unsigned int size, unsigned int align, return s; } -int do_kmem_cache_create(struct kmem_cache *s, slab_flags_t flags) +int do_kmem_cache_create(struct kmem_cache *s, const char *name, + unsigned int size, struct kmem_cache_args *args, + slab_flags_t flags) { int err = -EINVAL; + s->name = name; + s->size = s->object_size = size; + s->flags = kmem_cache_flags(flags, s->name); #ifdef CONFIG_SLAB_FREELIST_HARDENED s->random = get_random_long(); #endif + if (args->use_freeptr_offset) + s->rcu_freeptr_offset = args->freeptr_offset; + else + s->rcu_freeptr_offset = UINT_MAX; + s->align = args->align; + s->ctor = args->ctor; +#ifdef CONFIG_HARDENED_USERCOPY + s->useroffset = args->useroffset; + s->usersize = args->usersize; +#endif if (!calculate_sizes(s)) goto out; From patchwork Tue Sep 3 14:20:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Brauner X-Patchwork-Id: 13788773 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9D9316F30B for ; Tue, 3 Sep 2024 14:22:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725373363; cv=none; b=ozvIZS5kVLGDML3HAA0gcYyCaNxXfwGZEwfefGPfKYmBmFBFSz9DD1DDt2i7yJtBHfz3OQTpS6zTVR+s5bZdcg1pdYOtJTBy/dlzo7Owm708ZBlqvO5t2zZ6QqUsvsEEl4GpPLZZegd4ejsIeLaIOuPdNIDCsSpcpBASRP7H1iQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725373363; c=relaxed/simple; bh=+kTwNa1m6b/Hnr0YtIDtx0q9W42j7lzgwCx+jUzI5rs=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=Fmh/FYNm6bbfAXnQDKg3NVydWBBCMiZRlol59BBY5Fj5Npqf+v3sSTSCsBycUSnROdjzenbhHRUwPc9KB6J3eeSj7Th0E/YbV9maxFr4vBNm76u7ul7uIZQxFohNaIttiDXY58BRMicrDP4zeM8hX6wAq85AkiZ0sNG1RIp/U9Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=ZIqDi3Qn; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="ZIqDi3Qn" Received: by smtp.kernel.org (Postfix) with ESMTPSA id BE620C4CEC4; Tue, 3 Sep 2024 14:22:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1725373363; bh=+kTwNa1m6b/Hnr0YtIDtx0q9W42j7lzgwCx+jUzI5rs=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=ZIqDi3QnJT6V+K28/jD+UNtuaEDHlnJWqYR8lxhqiapVrTdtG0h22hWUu81vD5O2+ jXFZjjBzQ8TZLgYQuZUDqWdgcND9G8Rz9UOfeQPM2XeImc+frsoPSL6onkde1jl1tY VAHgG1WziE9j9W4rFM9UIyvhpxLFqaxvTZkQ6SCPA8Gu4Mjyk55DAavUexXA9nChen uI00cTePs9fVCgmjPnmJbiR3TTTuRimy87cmxjPvCWtB3zu5XJAE3ncQJwG/bdg/RC 6t2651rDtNYBxxp2lsxTdiw2E/iwWxq/nt4y7EyD2YVjPFboFkKnXucewOKJb5i21q 76YuSfJsqpdfQ== From: Christian Brauner Date: Tue, 03 Sep 2024 16:20:50 +0200 Subject: [PATCH v2 09/15] sl*b: remove rcu_freeptr_offset from struct kmem_cache Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240903-work-kmem_cache_args-v2-9-76f97e9a4560@kernel.org> References: <20240903-work-kmem_cache_args-v2-0-76f97e9a4560@kernel.org> In-Reply-To: <20240903-work-kmem_cache_args-v2-0-76f97e9a4560@kernel.org> To: Vlastimil Babka , Jens Axboe , Jann Horn , Linus Torvalds , Mike Rapoport Cc: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Christian Brauner X-Mailer: b4 0.15-dev-37811 X-Developer-Signature: v=1; a=openpgp-sha256; l=3722; i=brauner@kernel.org; h=from:subject:message-id; bh=+kTwNa1m6b/Hnr0YtIDtx0q9W42j7lzgwCx+jUzI5rs=; b=owGbwMvMwCU28Zj0gdSKO4sYT6slMaRdl57jvoPX02rSPSWTt/ddD73463HOrSVxeqTj4aOzw 5g3SP541FHKwiDGxSArpsji0G4SLrecp2KzUaYGzBxWJpAhDFycAjCRBysY/unOE+Ll3uq9eeUC ieve17+IL5eOKJ3Vxv1pX/e6yVsDmu8yMvy9e2/elMKab78eFp99eugk338zkZNyMxfOisxszz/ 89zQ7AA== X-Developer-Key: i=brauner@kernel.org; a=openpgp; fpr=4880B8C9BD0E5106FC070F4F7B3C391EFEA93624 Now that we pass down struct kmem_cache_args to calculate_sizes() we don't need it anymore. Signed-off-by: Christian Brauner Reviewed-by: Mike Rapoport (Microsoft) --- mm/slab.h | 2 -- mm/slub.c | 25 +++++++------------------ 2 files changed, 7 insertions(+), 20 deletions(-) diff --git a/mm/slab.h b/mm/slab.h index c7a4e0fc3cf1..36ac38e21fcb 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -261,8 +261,6 @@ struct kmem_cache { unsigned int object_size; /* Object size without metadata */ struct reciprocal_value reciprocal_size; unsigned int offset; /* Free pointer offset */ - /* Specific free pointer requested (if not UINT_MAX) */ - unsigned int rcu_freeptr_offset; #ifdef CONFIG_SLUB_CPU_PARTIAL /* Number of per cpu partial objects to keep around */ unsigned int cpu_partial; diff --git a/mm/slub.c b/mm/slub.c index 4719b60215b8..a23c7036cd61 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3916,8 +3916,7 @@ static void *__slab_alloc_node(struct kmem_cache *s, * If the object has been wiped upon free, make sure it's fully initialized by * zeroing out freelist pointer. * - * Note that we also wipe custom freelist pointers specified via - * s->rcu_freeptr_offset. + * Note that we also wipe custom freelist pointers. */ static __always_inline void maybe_wipe_obj_freeptr(struct kmem_cache *s, void *obj) @@ -5141,17 +5140,11 @@ static void set_cpu_partial(struct kmem_cache *s) #endif } -/* Was a valid freeptr offset requested? */ -static inline bool has_freeptr_offset(const struct kmem_cache *s) -{ - return s->rcu_freeptr_offset != UINT_MAX; -} - /* * calculate_sizes() determines the order and the distribution of data within * a slab object. */ -static int calculate_sizes(struct kmem_cache *s) +static int calculate_sizes(struct kmem_cache_args *args, struct kmem_cache *s) { slab_flags_t flags = s->flags; unsigned int size = s->object_size; @@ -5192,7 +5185,7 @@ static int calculate_sizes(struct kmem_cache *s) */ s->inuse = size; - if (((flags & SLAB_TYPESAFE_BY_RCU) && !has_freeptr_offset(s)) || + if (((flags & SLAB_TYPESAFE_BY_RCU) && !args->use_freeptr_offset) || (flags & SLAB_POISON) || s->ctor || ((flags & SLAB_RED_ZONE) && (s->object_size < sizeof(void *) || slub_debug_orig_size(s)))) { @@ -5214,8 +5207,8 @@ static int calculate_sizes(struct kmem_cache *s) */ s->offset = size; size += sizeof(void *); - } else if ((flags & SLAB_TYPESAFE_BY_RCU) && has_freeptr_offset(s)) { - s->offset = s->rcu_freeptr_offset; + } else if ((flags & SLAB_TYPESAFE_BY_RCU) && args->use_freeptr_offset) { + s->offset = args->freeptr_offset; } else { /* * Store freelist pointer near middle of object to keep @@ -5856,10 +5849,6 @@ int do_kmem_cache_create(struct kmem_cache *s, const char *name, #ifdef CONFIG_SLAB_FREELIST_HARDENED s->random = get_random_long(); #endif - if (args->use_freeptr_offset) - s->rcu_freeptr_offset = args->freeptr_offset; - else - s->rcu_freeptr_offset = UINT_MAX; s->align = args->align; s->ctor = args->ctor; #ifdef CONFIG_HARDENED_USERCOPY @@ -5867,7 +5856,7 @@ int do_kmem_cache_create(struct kmem_cache *s, const char *name, s->usersize = args->usersize; #endif - if (!calculate_sizes(s)) + if (!calculate_sizes(args, s)) goto out; if (disable_higher_order_debug) { /* @@ -5877,7 +5866,7 @@ int do_kmem_cache_create(struct kmem_cache *s, const char *name, if (get_order(s->size) > get_order(s->object_size)) { s->flags &= ~DEBUG_METADATA_FLAGS; s->offset = 0; - if (!calculate_sizes(s)) + if (!calculate_sizes(args, s)) goto out; } } From patchwork Tue Sep 3 14:20:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Brauner X-Patchwork-Id: 13788774 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E8FBC626CB for ; Tue, 3 Sep 2024 14:22:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725373366; cv=none; b=EpllObYEIhVrZmn3tbe2G0wOY4PNOJbCXsQFREKG+L3ity0214xKlU/a1DnB8/txbE3YII+zbFEiYhkvG+OuD1knIQZLZS9VW7QbHmOep2JFBOPWzpWmMNuDrQl/4MxV+kC4CXqnko8U+shjgF80bqESEAREPm0crADTM2N35Ks= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725373366; c=relaxed/simple; bh=FR8F1qZbTb9zJTJgS+qNyN7qWMc09eHWUOjOyojvTgI=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=ex4ZaEMCgn/wzaAByf81BDorkuG7FvlAD0+lf/P9NASOIaQpe+R82Ea5yqCayD0G/qdznByISYL0Thpc2I3dwU4sU8B4W7VEdi+LPCo0DLoKhRIH2bJI69n9lOwX5Hy0Do3X1SPJz3k3CYa+nx6wW7XJEyAneYqFyPzZgCixOBY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=tKQpFLSf; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="tKQpFLSf" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0F330C4CEC9; Tue, 3 Sep 2024 14:22:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1725373365; bh=FR8F1qZbTb9zJTJgS+qNyN7qWMc09eHWUOjOyojvTgI=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=tKQpFLSfOXlziqcKn5RJLf9QiXJle0j40hcp8DAm2CZ1lv7nJtGL/phOIBav3PgIC lzdqEzEIhBfOm9AkvzY8eEKDiqdCLYQRCba9BYeFluRTZu9ju2EG8PJs/3cRxadAJF uPt/sGkTf1PyIhsEyxIoMg93KIak/gufudOUMg6FnObWCiaPK+LhWaxxIOE8PfkZEK Yu1wpiF4B94Q70AdFf0K6xv1dfOL3lwkGQmfvouRoV1y4ToXCfTiz2pajuDM/1AXke o2xlUd2qOCKEUaRipptLB6XVHbDhp/nQz88H6MzqsGe25WNixmOeeB16a0EZn0mwC+ 9YFW6udqq4n3g== From: Christian Brauner Date: Tue, 03 Sep 2024 16:20:51 +0200 Subject: [PATCH v2 10/15] slab: port KMEM_CACHE() to struct kmem_cache_args Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240903-work-kmem_cache_args-v2-10-76f97e9a4560@kernel.org> References: <20240903-work-kmem_cache_args-v2-0-76f97e9a4560@kernel.org> In-Reply-To: <20240903-work-kmem_cache_args-v2-0-76f97e9a4560@kernel.org> To: Vlastimil Babka , Jens Axboe , Jann Horn , Linus Torvalds , Mike Rapoport Cc: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Christian Brauner X-Mailer: b4 0.15-dev-37811 X-Developer-Signature: v=1; a=openpgp-sha256; l=1112; i=brauner@kernel.org; h=from:subject:message-id; bh=FR8F1qZbTb9zJTJgS+qNyN7qWMc09eHWUOjOyojvTgI=; b=owGbwMvMwCU28Zj0gdSKO4sYT6slMaRdl54TcJsj42vuU52ix9cm7n+RpX7vt0mi8vPZK59N9 mXqMBMW6yhlYRDjYpAVU2RxaDcJl1vOU7HZKFMDZg4rE8gQBi5OAZhI7xlGhlPX6wSmfa2LrRe9 f1Cu5+Prxe6Pbt0KtAqYwrF0fbL/1QOMDBdW/dzeJ5JSJeH+/mJSfZTb+vkfVf5WLjYunOaiZRj wkxMA X-Developer-Key: i=brauner@kernel.org; a=openpgp; fpr=4880B8C9BD0E5106FC070F4F7B3C391EFEA93624 Make KMEM_CACHE() use struct kmem_cache_args. Signed-off-by: Christian Brauner Reviewed-by: Mike Rapoport (Microsoft) --- include/linux/slab.h | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 79d8c8bca4a4..d9c2ed5bc02f 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -283,9 +283,12 @@ int kmem_cache_shrink(struct kmem_cache *s); * f.e. add ____cacheline_aligned_in_smp to the struct declaration * then the objects will be properly aligned in SMP configurations. */ -#define KMEM_CACHE(__struct, __flags) \ - kmem_cache_create(#__struct, sizeof(struct __struct), \ - __alignof__(struct __struct), (__flags), NULL) +#define KMEM_CACHE(__struct, __flags) \ + __kmem_cache_create_args(#__struct, sizeof(struct __struct), \ + &(struct kmem_cache_args) { \ + .align = __alignof__(struct __struct), \ + .ctor = NULL, \ + }, (__flags)) /* * To whitelist a single field for copying to/from usercopy, use this From patchwork Tue Sep 3 14:20:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Brauner X-Patchwork-Id: 13788775 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 813D51E502 for ; Tue, 3 Sep 2024 14:22:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725373368; cv=none; b=IYfHc0sUBg7c0JZFK1snfICZl6Gav7A78shVTHkm6Spt83bte7kYThSnTiqVCL2CIbBB+clgyoc1nCDUXM9MLWFDK7Zz84tHOBn36yqeq8gs+0AFR7lP2qWK6ojrPP+GKvSxcxvtI2L/QW58S672XMF89DzcIwlGgfzExcUIJjY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725373368; c=relaxed/simple; bh=PVJGD880SWN2QXXzV8G7Km5Za31W7UeV1k1iq+DB/nU=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=sKbyj7bYTgniX7AHh61N7RnLbyO1RGZn1XHjPcjoYsxJD5KRvL3javJnluvlaZaaXu3x/1X7BzDB35Mq55s/McEPIpue3SdusYGg4adFRTDCmtjbhtApgwr10nfh0PUbfyvzMRhWrSRuJ7q4OEwro2ohjBgesPe5ei3/FR17jmA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=elIp/6pQ; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="elIp/6pQ" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4AA01C4CEC8; Tue, 3 Sep 2024 14:22:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1725373368; bh=PVJGD880SWN2QXXzV8G7Km5Za31W7UeV1k1iq+DB/nU=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=elIp/6pQ1ITB14KbYVhhnY4NhT7QtlcVxjp9pClky3Odf1lSc8k/GlrrMjwudI3ax yZwcsK22IVTe+MaIv2LhcWKwuVfpxK1mrAd2BVCIrbu18yRcqDhU7IjFWBVk8AT4hG jYl22Y4aKeWbX368tp3tfoGRyKmfgpMXin1THiKLPKrzMgUj3BfM7eaCQRVfaOOIPC nITQ/g5+8+rj/moMX0B22xFmMvIGax8lLdryc6+GYcw+Bp2TAAL6OjI5THqFTzk93u D5aRUGE0ebxI68nZRNOO0yy94hDdzj+N9/pAeYFNqt9Gul75ONyfUQaNPfPZyF4DNg HPbjixYQEaTxQ== From: Christian Brauner Date: Tue, 03 Sep 2024 16:20:52 +0200 Subject: [PATCH v2 11/15] slab: port KMEM_CACHE_USERCOPY() to struct kmem_cache_args Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240903-work-kmem_cache_args-v2-11-76f97e9a4560@kernel.org> References: <20240903-work-kmem_cache_args-v2-0-76f97e9a4560@kernel.org> In-Reply-To: <20240903-work-kmem_cache_args-v2-0-76f97e9a4560@kernel.org> To: Vlastimil Babka , Jens Axboe , Jann Horn , Linus Torvalds , Mike Rapoport Cc: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Christian Brauner X-Mailer: b4 0.15-dev-37811 X-Developer-Signature: v=1; a=openpgp-sha256; l=1312; i=brauner@kernel.org; h=from:subject:message-id; bh=PVJGD880SWN2QXXzV8G7Km5Za31W7UeV1k1iq+DB/nU=; b=owGbwMvMwCU28Zj0gdSKO4sYT6slMaRdl55jWn9i6mW97Oe7fSc2ZE0PfnPFdUX6r3ju02dmP ovy4Eks7ihlYRDjYpAVU2RxaDcJl1vOU7HZKFMDZg4rE8gQBi5OAZjIfzaGvyJSqTpK1dVflng/ qYvIlF9x9mrjg4hrO3mTTmZE1UYddGD4w9NxXOtzZHCaj+Lnvbu38c7Iebb4xaWMg4pdJwxuROh q8AAA X-Developer-Key: i=brauner@kernel.org; a=openpgp; fpr=4880B8C9BD0E5106FC070F4F7B3C391EFEA93624 Make KMEM_CACHE_USERCOPY() use struct kmem_cache_args. Signed-off-by: Christian Brauner Reviewed-by: Mike Rapoport (Microsoft) --- include/linux/slab.h | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index d9c2ed5bc02f..aced16a08700 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -294,12 +294,14 @@ int kmem_cache_shrink(struct kmem_cache *s); * To whitelist a single field for copying to/from usercopy, use this * macro instead for KMEM_CACHE() above. */ -#define KMEM_CACHE_USERCOPY(__struct, __flags, __field) \ - kmem_cache_create_usercopy(#__struct, \ - sizeof(struct __struct), \ - __alignof__(struct __struct), (__flags), \ - offsetof(struct __struct, __field), \ - sizeof_field(struct __struct, __field), NULL) +#define KMEM_CACHE_USERCOPY(__struct, __flags, __field) \ + __kmem_cache_create_args(#__struct, sizeof(struct __struct), \ + &(struct kmem_cache_args) { \ + .align = __alignof__(struct __struct), \ + .useroffset = offsetof(struct __struct, __field), \ + .usersize = sizeof_field(struct __struct, __field), \ + .ctor = NULL, \ + }, (__flags)) /* * Common kmalloc functions provided by all allocators From patchwork Tue Sep 3 14:20:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Brauner X-Patchwork-Id: 13788776 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9DD6713D245 for ; Tue, 3 Sep 2024 14:22:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725373370; cv=none; b=MOkRqy624b1ZNyy+bS1/cbDj2w6f/D16MsoSRv6UFru0U9nY7O09S1czF4UKsZGey2QN0GKS9146RjN/+pftv5/0IMi+A4k5VZs0p+sVsM3PPENWTz+vBu2MWnKHtSbVG6TjUhHgDhnmpYnzXDP7ZNOsTgXxVbHbxV2rp977EdQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725373370; c=relaxed/simple; bh=t7al+h/ze5pARsr5Uki/kfE3prWidx+wEHKqUnyvBao=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=kXJGRrfngWzzriuFgpkhPkCYfwWfT3tMfflLsXw9FCUWdNLoLVAoWa2MisYJwIAjE2HXO8j/om2IV3Gz7b/dRNkjwRbJ4iqWON9WBOIJi4aYs6hL5nCCYZvKpVWIRgX07+mkJaF8DxqYPJv+3F8qPc7T+yczo/kTYkt2QwsdaFM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=GI5fCg6w; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="GI5fCg6w" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8E9E3C4CECA; Tue, 3 Sep 2024 14:22:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1725373370; bh=t7al+h/ze5pARsr5Uki/kfE3prWidx+wEHKqUnyvBao=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=GI5fCg6wE9JAbDFx38ntWbqoJ4WArMJXqs+IUMdUmibbQuKyzYDFU7hPngI66qys1 84hwKKRC0v19spyU7/Ipnv6ToPmsngH01J4aNuWZq+PqtCMZRclX2BEVtyRnXhuao8 fh95UuYYzzM4XjhZ7C/bRmv+10JsT54xgRGHRplcXiB21sYmQ0IYu/ePK7b3ul9xOC bVE1u96ZUhjkwFEFNiTs9fLMZbW45mYXTlJISeY08zRLxaBOtGu8yk6hXqlyYtCPqE 2vdul7RHzgcegdYpsD1x+41OOCNSIGvPIrjkKRkOYS089YiaU7mGycV1dG4FvFdDfA gLl9qU5veRI+g== From: Christian Brauner Date: Tue, 03 Sep 2024 16:20:53 +0200 Subject: [PATCH v2 12/15] slab: create kmem_cache_create() compatibility layer Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240903-work-kmem_cache_args-v2-12-76f97e9a4560@kernel.org> References: <20240903-work-kmem_cache_args-v2-0-76f97e9a4560@kernel.org> In-Reply-To: <20240903-work-kmem_cache_args-v2-0-76f97e9a4560@kernel.org> To: Vlastimil Babka , Jens Axboe , Jann Horn , Linus Torvalds , Mike Rapoport Cc: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Christian Brauner X-Mailer: b4 0.15-dev-37811 X-Developer-Signature: v=1; a=openpgp-sha256; l=3268; i=brauner@kernel.org; h=from:subject:message-id; bh=t7al+h/ze5pARsr5Uki/kfE3prWidx+wEHKqUnyvBao=; b=owGbwMvMwCU28Zj0gdSKO4sYT6slMaRdl57ze9V/3sgcXvbTlm733/ZNcmz8HvPm6rLvVVH3j H8EMYe97ChlYRDjYpAVU2RxaDcJl1vOU7HZKFMDZg4rE8gQBi5OAZiI6muGfyoS11b0Juc8q/Gb ynZ2SvHzuLkblG8F3Fh+ekXt4eVT0sIZ/pfzhm3/rHB5W/Mh18BrNztnn33+9KO/V8eEkCMimUK 68/gA X-Developer-Key: i=brauner@kernel.org; a=openpgp; fpr=4880B8C9BD0E5106FC070F4F7B3C391EFEA93624 Use _Generic() to create a compatibility layer that type switches on the third argument to either call __kmem_cache_create() or __kmem_cache_create_args(). This can be kept in place until all callers have been ported to struct kmem_cache_args. Signed-off-by: Christian Brauner Reviewed-by: Mike Rapoport (Microsoft) --- include/linux/slab.h | 13 ++++++++++--- mm/slab_common.c | 10 +++++----- 2 files changed, 15 insertions(+), 8 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index aced16a08700..4292d67094c3 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -261,9 +261,10 @@ struct kmem_cache *__kmem_cache_create_args(const char *name, unsigned int object_size, struct kmem_cache_args *args, slab_flags_t flags); -struct kmem_cache *kmem_cache_create(const char *name, unsigned int size, - unsigned int align, slab_flags_t flags, - void (*ctor)(void *)); + +struct kmem_cache *__kmem_cache_create(const char *name, unsigned int size, + unsigned int align, slab_flags_t flags, + void (*ctor)(void *)); struct kmem_cache *kmem_cache_create_usercopy(const char *name, unsigned int size, unsigned int align, slab_flags_t flags, @@ -272,6 +273,12 @@ struct kmem_cache *kmem_cache_create_usercopy(const char *name, struct kmem_cache *kmem_cache_create_rcu(const char *name, unsigned int size, unsigned int freeptr_offset, slab_flags_t flags); + +#define kmem_cache_create(__name, __object_size, __args, ...) \ + _Generic((__args), \ + struct kmem_cache_args *: __kmem_cache_create_args, \ + default: __kmem_cache_create)(__name, __object_size, __args, __VA_ARGS__) + void kmem_cache_destroy(struct kmem_cache *s); int kmem_cache_shrink(struct kmem_cache *s); diff --git a/mm/slab_common.c b/mm/slab_common.c index 19ae3dd6e36f..418459927670 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -383,7 +383,7 @@ kmem_cache_create_usercopy(const char *name, unsigned int size, EXPORT_SYMBOL(kmem_cache_create_usercopy); /** - * kmem_cache_create - Create a cache. + * __kmem_cache_create - Create a cache. * @name: A string which is used in /proc/slabinfo to identify this cache. * @size: The size of objects to be created in this cache. * @align: The required alignment for the objects. @@ -407,9 +407,9 @@ EXPORT_SYMBOL(kmem_cache_create_usercopy); * * Return: a pointer to the cache on success, NULL on failure. */ -struct kmem_cache * -kmem_cache_create(const char *name, unsigned int size, unsigned int align, - slab_flags_t flags, void (*ctor)(void *)) +struct kmem_cache *__kmem_cache_create(const char *name, unsigned int size, + unsigned int align, slab_flags_t flags, + void (*ctor)(void *)) { struct kmem_cache_args kmem_args = { .align = align, @@ -418,7 +418,7 @@ kmem_cache_create(const char *name, unsigned int size, unsigned int align, return __kmem_cache_create_args(name, size, &kmem_args, flags); } -EXPORT_SYMBOL(kmem_cache_create); +EXPORT_SYMBOL(__kmem_cache_create); /** * kmem_cache_create_rcu - Create a SLAB_TYPESAFE_BY_RCU cache. From patchwork Tue Sep 3 14:20:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Brauner X-Patchwork-Id: 13788777 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E9F1F1E502 for ; Tue, 3 Sep 2024 14:22:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725373373; cv=none; b=TSS+17tDnTvTg8zvGAdLl9aHLhfmDSAgWbRifiWszYH4SeOfPDw8N/Vh/XleGpUN9Dr9Jqt+ZWsBk3CqhFYuBA4NUt/ZabjSAWP9rfXw8JV7Z6G9q62IXgp+2IpCNO3eJL619yfhci0feuIXvKZgXESkmgvPmBBFZ4HaExcKguA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725373373; c=relaxed/simple; bh=CebY+vup8RjtcJO3CTsKAhV3jVPnEJg6xeCeD64qYmg=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=SXmvox/ue5q34j948rsD28g3vLRCliDERWaXEeQVKdDtpt7gFt6OSuCL8ZfcRclNkoUoTf6y7OSD7wmdE0bPQBiHfxK022vshcE9O9Ej5wtleiUgRAKMA4cwFIRJGQWzow2OWf3ewdqb+qQlcct+f3y9JVOtFy+57dbMCVFkjy4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=YuWuHWX/; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="YuWuHWX/" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0689BC4CEC8; Tue, 3 Sep 2024 14:22:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1725373372; bh=CebY+vup8RjtcJO3CTsKAhV3jVPnEJg6xeCeD64qYmg=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=YuWuHWX/bnShus2D9Wt4BPKhUFKhBoC2JuQHtpngis5DVhaSb80K8nhQ7YjfXM6eh ORHEfli3aPbBAyZDCOUv2avSMT8I4L/rZ9xbwqotShJf22bkmi/Rrsr1ksNZo8p2qB yhroIQSw/texBgSHn0y8XTPp/cu68rNXCymuGLjwkwRXwemv4lgxsurtLDaQ3C2FcE nT4hqV+8d3IuYjkB/SMfnwpTItI3r2Jc1KdztM38vzHm7i2WH/1vYsE6bbTaTwJBuL 6MbnkTx1QYYLjTqBqXuZXney+PfSi40v/QowMB35c1MvlznDbvrPeAYZ3tsEVPZt74 WRm/04WjO8UeQ== From: Christian Brauner Date: Tue, 03 Sep 2024 16:20:54 +0200 Subject: [PATCH v2 13/15] file: port to struct kmem_cache_args Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240903-work-kmem_cache_args-v2-13-76f97e9a4560@kernel.org> References: <20240903-work-kmem_cache_args-v2-0-76f97e9a4560@kernel.org> In-Reply-To: <20240903-work-kmem_cache_args-v2-0-76f97e9a4560@kernel.org> To: Vlastimil Babka , Jens Axboe , Jann Horn , Linus Torvalds , Mike Rapoport Cc: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Christian Brauner X-Mailer: b4 0.15-dev-37811 X-Developer-Signature: v=1; a=openpgp-sha256; l=938; i=brauner@kernel.org; h=from:subject:message-id; bh=CebY+vup8RjtcJO3CTsKAhV3jVPnEJg6xeCeD64qYmg=; b=owGbwMvMwCU28Zj0gdSKO4sYT6slMaRdl55zJ6dM2VhKS2/H/aW9Vbv7r6Yer3jIki0R3z/h6 +JyRfGojlIWBjEuBlkxRRaHdpNwueU8FZuNMjVg5rAygQxh4OIUgIko7GP4n2CVvqcx/IdlwHwL uX0fojbs3pu1ckf6cjH7jpOqtgznjzP807PtnHyx7T3v4Y1ySXwx8dNiSjMPyfivrI0JePt9s70 tBwA= X-Developer-Key: i=brauner@kernel.org; a=openpgp; fpr=4880B8C9BD0E5106FC070F4F7B3C391EFEA93624 Port filp_cache to struct kmem_cache_args. Signed-off-by: Christian Brauner Reviewed-by: Mike Rapoport (Microsoft) --- fs/file_table.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/fs/file_table.c b/fs/file_table.c index 3ef558f27a1c..861c03608e83 100644 --- a/fs/file_table.c +++ b/fs/file_table.c @@ -511,9 +511,14 @@ EXPORT_SYMBOL(__fput_sync); void __init files_init(void) { - filp_cachep = kmem_cache_create_rcu("filp", sizeof(struct file), - offsetof(struct file, f_freeptr), - SLAB_HWCACHE_ALIGN | SLAB_PANIC | SLAB_ACCOUNT); + struct kmem_cache_args args = { + .use_freeptr_offset = true, + .freeptr_offset = offsetof(struct file, f_freeptr), + }; + + filp_cachep = kmem_cache_create("filp", sizeof(struct file), &args, + SLAB_HWCACHE_ALIGN | SLAB_PANIC | + SLAB_ACCOUNT | SLAB_TYPESAFE_BY_RCU); percpu_counter_init(&nr_files, 0, GFP_KERNEL); } From patchwork Tue Sep 3 14:20:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Brauner X-Patchwork-Id: 13788778 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 94F2513C906 for ; Tue, 3 Sep 2024 14:22:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725373375; cv=none; b=qrzCy2mfxTs7wLRQNg6W/fLbGO1w6fsklUdRYsbWBhGiFgpucAQH6IBh31I4VseHwO+4juhElKsA0zM3TS/2TyDfPYMJF88IEXDykxzWIDShZ4l77BcumH3tNDfLnKlOCbtr66D9ly17NtajsMOSvMIHFx+N8tu8UYML+xMYTNw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725373375; c=relaxed/simple; bh=6bG1JSwASGPcSbXbn/SP58oY4O44BFhaq6fvlGpwWZw=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=Xep7bJHaLoJwHkUi+kw8BlnzpnYWYnxNh0f+sflDPOXvMskeQHenhNcPVgLLWxT13NHC97vM9FPrsJdHkTpPwk5ZcTzdCV3AlwXH3hiW4Ba9vvHPnFPd60tEj1W1CrCDEabMZcB3TEStLO7xi418FUUpZ+03/zHJMtM/bQ4bmN0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=JgSzT5kw; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="JgSzT5kw" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5CA42C4CEC8; Tue, 3 Sep 2024 14:22:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1725373375; bh=6bG1JSwASGPcSbXbn/SP58oY4O44BFhaq6fvlGpwWZw=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=JgSzT5kw/kqZjtu62rKEOhtPQaXg6+GNpK7HPlu2JAeGj9bEhx9j07h95VDy1OKzE O31WVPK0Aor2Wj0c2PJPqCQahWpa4akQrZOu+WDkA1EG1VHsmvxZMLB+IwkYLAS7+B R/zTjA9ThANEe8EYIwWi34xcct3yVJpVNb6u5JrENfp3n1grX2Zkf/UYwZbIecoyWL 5R0XAn9j6JCIHIhZN/V6PZqVH8c1zEohmwqYI8wyr/khfF7xn4DXQ9pmAOR7954BDD NiA1S6c/x6XZLb1vhIALBBWiPczImRVo3QYYPqxEV2Kg7cVyFUf2TJgorV4wcCsnHp 6WhEIvdcTTD/A== From: Christian Brauner Date: Tue, 03 Sep 2024 16:20:55 +0200 Subject: [PATCH v2 14/15] slab: remove kmem_cache_create_rcu() Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240903-work-kmem_cache_args-v2-14-76f97e9a4560@kernel.org> References: <20240903-work-kmem_cache_args-v2-0-76f97e9a4560@kernel.org> In-Reply-To: <20240903-work-kmem_cache_args-v2-0-76f97e9a4560@kernel.org> To: Vlastimil Babka , Jens Axboe , Jann Horn , Linus Torvalds , Mike Rapoport Cc: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Christian Brauner X-Mailer: b4 0.15-dev-37811 X-Developer-Signature: v=1; a=openpgp-sha256; l=3031; i=brauner@kernel.org; h=from:subject:message-id; bh=6bG1JSwASGPcSbXbn/SP58oY4O44BFhaq6fvlGpwWZw=; b=owGbwMvMwCU28Zj0gdSKO4sYT6slMaRdl54jXPX8rdx6/mVLDkyd06KQtJU9c0HW1+iL/6LX5 x2fsGtFdUcpC4MYF4OsmCKLQ7tJuNxynorNRpkaMHNYmUCGMHBxCsBE3ggyMrzjlnhzRLl04is9 m658Z+1bFmmFCb+0pq91PPIj48bNz6sZGZZfsks55tTGvGBllGHuUrZpJ/xDDt5X2HtBaAHH9Hc dTMwA X-Developer-Key: i=brauner@kernel.org; a=openpgp; fpr=4880B8C9BD0E5106FC070F4F7B3C391EFEA93624 Since we have kmem_cache_setup() and have ported kmem_cache_create_rcu() users over to it is unused and can be removed. Signed-off-by: Christian Brauner Reviewed-by: Mike Rapoport (Microsoft) --- include/linux/slab.h | 3 --- mm/slab_common.c | 43 ------------------------------------------- 2 files changed, 46 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 4292d67094c3..1176b30cd4b2 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -270,9 +270,6 @@ struct kmem_cache *kmem_cache_create_usercopy(const char *name, slab_flags_t flags, unsigned int useroffset, unsigned int usersize, void (*ctor)(void *)); -struct kmem_cache *kmem_cache_create_rcu(const char *name, unsigned int size, - unsigned int freeptr_offset, - slab_flags_t flags); #define kmem_cache_create(__name, __object_size, __args, ...) \ _Generic((__args), \ diff --git a/mm/slab_common.c b/mm/slab_common.c index 418459927670..9133b9fafcb1 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -420,49 +420,6 @@ struct kmem_cache *__kmem_cache_create(const char *name, unsigned int size, } EXPORT_SYMBOL(__kmem_cache_create); -/** - * kmem_cache_create_rcu - Create a SLAB_TYPESAFE_BY_RCU cache. - * @name: A string which is used in /proc/slabinfo to identify this cache. - * @size: The size of objects to be created in this cache. - * @freeptr_offset: The offset into the memory to the free pointer - * @flags: SLAB flags - * - * Cannot be called within an interrupt, but can be interrupted. - * - * See kmem_cache_create() for an explanation of possible @flags. - * - * By default SLAB_TYPESAFE_BY_RCU caches place the free pointer outside - * of the object. This might cause the object to grow in size. Callers - * that have a reason to avoid this can specify a custom free pointer - * offset in their struct where the free pointer will be placed. - * - * Note that placing the free pointer inside the object requires the - * caller to ensure that no fields are invalidated that are required to - * guard against object recycling (See SLAB_TYPESAFE_BY_RCU for - * details.). - * - * Using zero as a value for @freeptr_offset is valid. To request no - * offset UINT_MAX must be specified. - * - * Note that @ctor isn't supported with custom free pointers as a @ctor - * requires an external free pointer. - * - * Return: a pointer to the cache on success, NULL on failure. - */ -struct kmem_cache *kmem_cache_create_rcu(const char *name, unsigned int size, - unsigned int freeptr_offset, - slab_flags_t flags) -{ - struct kmem_cache_args kmem_args = { - .freeptr_offset = freeptr_offset, - .use_freeptr_offset = true, - }; - - return __kmem_cache_create_args(name, size, &kmem_args, - flags | SLAB_TYPESAFE_BY_RCU); -} -EXPORT_SYMBOL(kmem_cache_create_rcu); - static struct kmem_cache *kmem_buckets_cache __ro_after_init; /** From patchwork Tue Sep 3 14:20:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Brauner X-Patchwork-Id: 13788779 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D0C201420DD for ; Tue, 3 Sep 2024 14:22:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725373377; cv=none; b=HLRSxQYSbk6KqRVw5+YLB7pL/QsPICV/4v2HiuJJeu8iK4ceKrZGiekFnMlcIM6nNmMtm3xbgRF4zX4843wNjto5WtYUPK7zQ1jydfzA+2PfMwK1Xbj54ywQjxbI5PMrEBXY+ktcmdwbd8I1/InBuIWPHjmv/te8FCIRTa9xhUc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725373377; c=relaxed/simple; bh=3NCbIt7AK7tmjnKZtBo4rWvQELZGRKFxzNKzdNzEh+o=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=tVHjSZmym88+yz9H3wnVWWqAUitdl40N4hDL1dfay9k/BZ+LVQ8yv7UcAcpzKl+/XoDwFW+NqHqNdQwc3RFiIhFJ4RG7ZdybwTWvHRSg5/0LN3draNAolHJbM7rYHzESQRrTIc6Jntzr3/PLjB3Z1zj4ZhcvKyfXPq4u33cv6QM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=NIA6dpiB; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="NIA6dpiB" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9F4DEC4CEC7; Tue, 3 Sep 2024 14:22:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1725373377; bh=3NCbIt7AK7tmjnKZtBo4rWvQELZGRKFxzNKzdNzEh+o=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=NIA6dpiBbsE1OC9ubpMKp8jNcztrTDTatRxf5KWQbTjlNRL0XSoikHaYH8dpBad/t iQBWjOPBYr7F2hJIqWh1KHWyMLwTc0gaC0RunIT+Q0oQpZeoyPB8GGLUdhOw8irtF1 qbFVjlRZyEH7VMZ5Hv9JDpnK4Juc44MJqJh+uEOCaA5uPgtMgFvRdNmL+tmDM2VBuY Qxs7Y18c2eph5xk6SFRgiA6bR5BNORsx5hVl9nMzKNJSEjVWLvwjFklxT+weT11XLP YK1t4W5byhRnzilQiwcL7dGKwUnB0ES3FHG7KVbxBVKZQeyzZdkJf/fnMPTDWCmOuf lPTPbpjdMIbig== From: Christian Brauner Date: Tue, 03 Sep 2024 16:20:56 +0200 Subject: [PATCH v2 15/15] io_uring: port to struct kmem_cache_args Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240903-work-kmem_cache_args-v2-15-76f97e9a4560@kernel.org> References: <20240903-work-kmem_cache_args-v2-0-76f97e9a4560@kernel.org> In-Reply-To: <20240903-work-kmem_cache_args-v2-0-76f97e9a4560@kernel.org> To: Vlastimil Babka , Jens Axboe , Jann Horn , Linus Torvalds , Mike Rapoport Cc: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Christian Brauner X-Mailer: b4 0.15-dev-37811 X-Developer-Signature: v=1; a=openpgp-sha256; l=1540; i=brauner@kernel.org; h=from:subject:message-id; bh=3NCbIt7AK7tmjnKZtBo4rWvQELZGRKFxzNKzdNzEh+o=; b=owGbwMvMwCU28Zj0gdSKO4sYT6slMaRdl56jXCmdfbbN7sLDswvvtcefsxDlmOklJvBbS3OD1 g+uz2oVHaUsDGJcDLJiiiwO7Sbhcst5KjYbZWrAzGFlAhnCwMUpABPp4Wb4X7suxbHyXNDclqBt H/cV3d8rIrLblltMhat0pdlFZwOnSYwMZx3/pP+Yc9oxtfXD0wXrz8w0cVhxRmBV4tHgZbnpa58 wcgAA X-Developer-Key: i=brauner@kernel.org; a=openpgp; fpr=4880B8C9BD0E5106FC070F4F7B3C391EFEA93624 Port req_cachep to struct kmem_cache_args. Signed-off-by: Christian Brauner Reviewed-by: Mike Rapoport (Microsoft) --- io_uring/io_uring.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 3942db160f18..d9d721d1424e 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -3638,6 +3638,11 @@ SYSCALL_DEFINE2(io_uring_setup, u32, entries, static int __init io_uring_init(void) { + struct kmem_cache_args kmem_args = { + .useroffset = offsetof(struct io_kiocb, cmd.data), + .usersize = sizeof_field(struct io_kiocb, cmd.data), + }; + #define __BUILD_BUG_VERIFY_OFFSET_SIZE(stype, eoffset, esize, ename) do { \ BUILD_BUG_ON(offsetof(stype, ename) != eoffset); \ BUILD_BUG_ON(sizeof_field(stype, ename) != esize); \ @@ -3722,12 +3727,9 @@ static int __init io_uring_init(void) * range, and HARDENED_USERCOPY will complain if we haven't * correctly annotated this range. */ - req_cachep = kmem_cache_create_usercopy("io_kiocb", - sizeof(struct io_kiocb), 0, - SLAB_HWCACHE_ALIGN | SLAB_PANIC | - SLAB_ACCOUNT | SLAB_TYPESAFE_BY_RCU, - offsetof(struct io_kiocb, cmd.data), - sizeof_field(struct io_kiocb, cmd.data), NULL); + req_cachep = kmem_cache_create("io_kiocb", sizeof(struct io_kiocb), &kmem_args, + SLAB_HWCACHE_ALIGN | SLAB_PANIC | SLAB_ACCOUNT | + SLAB_TYPESAFE_BY_RCU); io_buf_cachep = KMEM_CACHE(io_buffer, SLAB_HWCACHE_ALIGN | SLAB_PANIC | SLAB_ACCOUNT);