From patchwork Mon Aug 7 11:08:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13343342 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3519C41513 for ; Mon, 7 Aug 2023 11:10:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4537C6B0075; Mon, 7 Aug 2023 07:10:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 404548D0003; Mon, 7 Aug 2023 07:10:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2A6478D0002; Mon, 7 Aug 2023 07:10:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 1AEE76B0075 for ; Mon, 7 Aug 2023 07:10:51 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id DF4A240995 for ; Mon, 7 Aug 2023 11:10:50 +0000 (UTC) X-FDA: 81097041060.10.01FCDD8 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) by imf04.hostedemail.com (Postfix) with ESMTP id 130B340016 for ; Mon, 7 Aug 2023 11:10:48 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=XqSgm7kB; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf04.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.173 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1691406649; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0+ir4b8RJjPS6OIbAMeQuS4120VPLjqRz3YtsLadct8=; b=aqE22c1og3sWVMjzw7J2qaMRbeqc02cN3QhhpA67Eh8z0rrkfJJZMbfX64MKvh4sezjn6z aXNeMP0RFV2y3fbmb+JpRuro2K25HzalbhZKuycRTITei8OVoB12u8gHKYZv7TQktcPCom MfkTvfDT6RaClVetMnZFrioGqZy6ewM= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=XqSgm7kB; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf04.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.173 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1691406649; a=rsa-sha256; cv=none; b=wWnCT87MqjxgPkpn+q2EbSSv6ErGShfnopBHJykFlaOdcx1zlB2JZ0aCECDj3XCkIRCieS jsP4oYlCfTxMhnj8m2YwcY3pN0uIijNcnlrWUJuMerOA24BpAzxUPWvMx/88xMCf6FRiaA 9SV4SFR97OnYFIF2in02GwVkSoOax5w= Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-1bb91c20602so9340915ad.0 for ; Mon, 07 Aug 2023 04:10:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1691406647; x=1692011447; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0+ir4b8RJjPS6OIbAMeQuS4120VPLjqRz3YtsLadct8=; b=XqSgm7kBpeCnE2gk/ybefyIcAxePmTjaifDT0pHnXDcb/Nc/UeCuy/rdgy4xRowAkQ aP4dUp/hrUWHR/4zWDCrbxFRixmFOlV8pnAOQyS1AwDBmMy6PTOmr3iG0yKzCbxujje0 t73ctUmXR29afAKI5JNarlJvM1vvoPYY3cx3UcuU87ePLchKbMsLak0/Jvs9wqFUCDZe F5bgU4otH0OCdVxWD0sXM2a1eraAUukOoDwvlCcB8mzgk3iB2I7g5XjW4+48PuHaFgCc uOLaUcpy4QuZWt68WuxcqsF9z/vzQ2XOsuEa2jtvaGJ4t8kUtmd0Q7Uf/YTK0wRowRTP E2tA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691406647; x=1692011447; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0+ir4b8RJjPS6OIbAMeQuS4120VPLjqRz3YtsLadct8=; b=Xr1Yko/yH3IUcMMzXhLa6eiqksex0CXOJSTIPN8WguRZx8R8WUQ6uiYBqFlFVrOCkM UlOQmdzMauyZCTQFTlNRz1llVjf+9iIuBRodMaBeFqjDGRa6QjoF7iPwjVi+TzXc5NKR rpYeda6z4aNiIbKtKsdbbPVniB6WbCvHNpJ7SQ2YTa36kqjAqEUCIiRJafhShMPBQt6X FNDR09OveNtqIS/VAuUqrYggmV+8i3xtO3wt5PCjC/6kM+IycvXipvjgw2wq7+57sD7e xdTyCkzLOqDyoZZSxYrLYByZ9UNN6ymy2LAbb8NU9Ij18uQXk7xj/XN1RQ8EFKzad8al 7GdQ== X-Gm-Message-State: ABy/qLaxR1gwVxHSUdMAhfGTSTOiWM+x60ERp6EYFDO8Dnm12qyTM4mH ZWYrVlCgYNT2yIv8hu4Yslozbg== X-Google-Smtp-Source: APBJJlHuLqAtKowl4cccNcqjA9V94U7s0kNn49/V0/tk7bu/NPlYiQpbpDIicm2vAutfhjwO22jVoA== X-Received: by 2002:a17:902:f54d:b0:1b8:9fc4:2733 with SMTP id h13-20020a170902f54d00b001b89fc42733mr32837074plf.3.1691406647588; Mon, 07 Aug 2023 04:10:47 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.146]) by smtp.gmail.com with ESMTPSA id y13-20020a17090aca8d00b0025be7b69d73sm5861191pjt.12.2023.08.07.04.10.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 07 Aug 2023 04:10:47 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, david@fromorbit.com, tkhai@ya.ru, vbabka@suse.cz, roman.gushchin@linux.dev, djwong@kernel.org, brauner@kernel.org, paulmck@kernel.org, tytso@mit.edu, steven.price@arm.com, cel@kernel.org, senozhatsky@chromium.org, yujie.liu@intel.com, gregkh@linuxfoundation.org, muchun.song@linux.dev, simon.horman@corigine.com, dlemoal@kernel.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org, kvm@vger.kernel.org, xen-devel@lists.xenproject.org, linux-erofs@lists.ozlabs.org, linux-f2fs-devel@lists.sourceforge.net, cluster-devel@redhat.com, linux-nfs@vger.kernel.org, linux-mtd@lists.infradead.org, rcu@vger.kernel.org, netdev@vger.kernel.org, dri-devel@lists.freedesktop.org, linux-arm-msm@vger.kernel.org, dm-devel@redhat.com, linux-raid@vger.kernel.org, linux-bcache@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-fsdevel@vger.kernel.org, linux-ext4@vger.kernel.org, linux-xfs@vger.kernel.org, linux-btrfs@vger.kernel.org, Qi Zheng Subject: [PATCH v4 04/48] mm: shrinker: add infrastructure for dynamically allocating shrinker Date: Mon, 7 Aug 2023 19:08:52 +0800 Message-Id: <20230807110936.21819-5-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230807110936.21819-1-zhengqi.arch@bytedance.com> References: <20230807110936.21819-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 130B340016 X-Stat-Signature: zdond1h9eaust41tiyzojzyfwoys9ucp X-HE-Tag: 1691406648-962551 X-HE-Meta: U2FsdGVkX1/YXIC2yq7wTz0Tavy+i77dQr/56TCNy1WjpPXeBZjXq7UVzAiX2XSwUGxi+5l6P/YqFNHRgPYx8q92D2QEss2EPn8WBB6S3/iLxqOG/wwb75SIzNWIgYaqTfdPxxMC/3Rt7TDfZw5BLmNnWrUHwozr5RpIUvI0AHRdBxbP464s9b1mGQaR0UX0z4VXRt2kZ01oizxGy8irvWSEOjkVKJhtMGDIGtJvrzJ0olQ5+VBgeFqldlTaWzC8m7pB3Jqn0L8yC52pT4Z8HCgipyTciOyMKIzAKXEhGSn8KWWFwQnNXCX7nJ5aoItweDLiB+vZCGcmHbJ4WNQ0Rwn1zYHrFDGwR7/lkpx5gxsKzfuxyiNOFX0QKmk5t1NlxiMzk5PMta4YUlux46Qhz1pSVldZzTDpev8PYcfyOQHSLH3/HjBB238Vgt2I9HvaVtzNtWPRc37dK8N7BnutNR2UTi6DeBvGAVa9Tj/qnMAAlC9waiP6e96lq1anDNs3/bOD1aP2QuTn353AhZXfJSshbAMn9/VAv3ctMWeyCcdu033eKSMZr0cPDrmLxgsTIBbl8eqEM8asDMDuIYpa8CAtRlTbp3GiN36TQeez9uDndg5AySX4UjD1I+Ch99bgxv4vFUmbvJPNiDQEgnmC9bPwApoCrJRHM7F1YHoDC8bHP/23/3I9fcEUEVNKciVWF+iIOAMpVBAYSHnNXl3ixUyJUYBgKA/lpuiWqc7XjY6e20uziRihlV7Mqb7BD6wx4+2RIwfHxaSYIpIR+Hog/475qzr5gtLSTx9Z9ZuZnvzRoNUBhhUooFmOoK8xISurK5WNxw5w7z3L+QI2oUzwnLxtGFyZvKC4tuSWmbfPueYbMgjxpF7rYf4mDt+bi4mrfmXlzVsMaB5jVe43ZhjrFOG8f2RD2ZaWZfkSU+LR2RV4ovDMZpb2R6DCZC+XrcY1SER5DfkLnctIce9PAdk N/ocD3oT 1eIAFMLYZnJSJdBRKQylSm3zeBBjrMFyW9yN/3miZMyEYwTWLWZwbAgmocBPnaWIPmHzfLYJ8OD4w6IHmGevUugbCteIJmoZk1kTPhun+/3axtr5zUENgwf3EpEPVoZoDJ1jeHS0DHqkw4EefX3AgX2O93t4r8AUtUaMmCLaUHNcS2/SXupN8zdosx7mYjjzGP56ASmJqjsQEeh9NaT8PnuMrZ/csOV+yo7Sx981hclxKpxaEpuaydUwlKHO6F687Ab+ikuzIgYDfjGttqFTrE0WlkCziCLRZx5+pNqvy7AEIx9IVaCDbf2Akvb1HaAnsEsGFh7gRiwgWytA6W/nB7B0vmOE3JR1p3X0ocvwQfbu7gb+QQjYPCurwiBXPj4xEKdpN+4R6hjn5X3d6boxmzcX3Hm0IJRKSOxZz X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently, the shrinker instances can be divided into the following three types: a) global shrinker instance statically defined in the kernel, such as workingset_shadow_shrinker. b) global shrinker instance statically defined in the kernel modules, such as mmu_shrinker in x86. c) shrinker instance embedded in other structures. For case a, the memory of shrinker instance is never freed. For case b, the memory of shrinker instance will be freed after synchronize_rcu() when the module is unloaded. For case c, the memory of shrinker instance will be freed along with the structure it is embedded in. In preparation for implementing lockless slab shrink, we need to dynamically allocate those shrinker instances in case c, then the memory can be dynamically freed alone by calling kfree_rcu(). So this commit adds the following new APIs for dynamically allocating shrinker, and add a private_data field to struct shrinker to record and get the original embedded structure. 1. shrinker_alloc() Used to allocate shrinker instance itself and related memory, it will return a pointer to the shrinker instance on success and NULL on failure. 2. shrinker_register() Used to register the shrinker instance, which is same as the current register_shrinker_prepared(). 3. shrinker_free() Used to unregister (if needed) and free the shrinker instance. In order to simplify shrinker-related APIs and make shrinker more independent of other kernel mechanisms, subsequent submissions will use the above API to convert all shrinkers (including case a and b) to dynamically allocated, and then remove all existing APIs. This will also have another advantage mentioned by Dave Chinner: ``` The other advantage of this is that it will break all the existing out of tree code and third party modules using the old API and will no longer work with a kernel using lockless slab shrinkers. They need to break (both at the source and binary levels) to stop bad things from happening due to using unconverted shrinkers in the new setup. ``` Signed-off-by: Qi Zheng --- include/linux/shrinker.h | 7 +++ mm/internal.h | 11 +++++ mm/shrinker.c | 101 +++++++++++++++++++++++++++++++++++++++ mm/shrinker_debug.c | 17 ++++++- 4 files changed, 134 insertions(+), 2 deletions(-) diff --git a/include/linux/shrinker.h b/include/linux/shrinker.h index 8dc15aa37410..cc23ff0aee20 100644 --- a/include/linux/shrinker.h +++ b/include/linux/shrinker.h @@ -70,6 +70,8 @@ struct shrinker { int seeks; /* seeks to recreate an obj */ unsigned flags; + void *private_data; + /* These are for internal use */ struct list_head list; #ifdef CONFIG_MEMCG @@ -95,6 +97,11 @@ struct shrinker { * non-MEMCG_AWARE shrinker should not have this flag set. */ #define SHRINKER_NONSLAB (1 << 3) +#define SHRINKER_ALLOCATED (1 << 4) + +struct shrinker *shrinker_alloc(unsigned int flags, const char *fmt, ...); +void shrinker_register(struct shrinker *shrinker); +void shrinker_free(struct shrinker *shrinker); extern int __printf(2, 3) prealloc_shrinker(struct shrinker *shrinker, const char *fmt, ...); diff --git a/mm/internal.h b/mm/internal.h index b98c29f0a471..7b882b903b82 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1152,6 +1152,9 @@ unsigned long shrink_slab(gfp_t gfp_mask, int nid, struct mem_cgroup *memcg, #ifdef CONFIG_SHRINKER_DEBUG extern int shrinker_debugfs_add(struct shrinker *shrinker); +extern int shrinker_debugfs_name_alloc(struct shrinker *shrinker, + const char *fmt, va_list ap); +extern void shrinker_debugfs_name_free(struct shrinker *shrinker); extern struct dentry *shrinker_debugfs_detach(struct shrinker *shrinker, int *debugfs_id); extern void shrinker_debugfs_remove(struct dentry *debugfs_entry, @@ -1161,6 +1164,14 @@ static inline int shrinker_debugfs_add(struct shrinker *shrinker) { return 0; } +static inline int shrinker_debugfs_name_alloc(struct shrinker *shrinker, + const char *fmt, va_list ap) +{ + return 0; +} +static inline void shrinker_debugfs_name_free(struct shrinker *shrinker) +{ +} static inline struct dentry *shrinker_debugfs_detach(struct shrinker *shrinker, int *debugfs_id) { diff --git a/mm/shrinker.c b/mm/shrinker.c index 043c87ccfab4..43a375f954f3 100644 --- a/mm/shrinker.c +++ b/mm/shrinker.c @@ -550,6 +550,107 @@ unsigned long shrink_slab(gfp_t gfp_mask, int nid, struct mem_cgroup *memcg, return freed; } +struct shrinker *shrinker_alloc(unsigned int flags, const char *fmt, ...) +{ + struct shrinker *shrinker; + unsigned int size; + va_list ap; + int err; + + shrinker = kzalloc(sizeof(struct shrinker), GFP_KERNEL); + if (!shrinker) + return NULL; + + va_start(ap, fmt); + err = shrinker_debugfs_name_alloc(shrinker, fmt, ap); + va_end(ap); + if (err) + goto err_name; + + shrinker->flags = flags | SHRINKER_ALLOCATED; + + if (flags & SHRINKER_MEMCG_AWARE) { + err = prealloc_memcg_shrinker(shrinker); + if (err == -ENOSYS) + shrinker->flags &= ~SHRINKER_MEMCG_AWARE; + else if (err == 0) + goto done; + else + goto err_flags; + } + + /* + * The nr_deferred is available on per memcg level for memcg aware + * shrinkers, so only allocate nr_deferred in the following cases: + * - non memcg aware shrinkers + * - !CONFIG_MEMCG + * - memcg is disabled by kernel command line + */ + size = sizeof(*shrinker->nr_deferred); + if (flags & SHRINKER_NUMA_AWARE) + size *= nr_node_ids; + + shrinker->nr_deferred = kzalloc(size, GFP_KERNEL); + if (!shrinker->nr_deferred) + goto err_flags; + +done: + return shrinker; + +err_flags: + shrinker_debugfs_name_free(shrinker); +err_name: + kfree(shrinker); + return NULL; +} +EXPORT_SYMBOL_GPL(shrinker_alloc); + +void shrinker_register(struct shrinker *shrinker) +{ + if (unlikely(!(shrinker->flags & SHRINKER_ALLOCATED))) { + pr_warn("Must use shrinker_alloc() to dynamically allocate the shrinker"); + return; + } + + down_write(&shrinker_rwsem); + list_add_tail(&shrinker->list, &shrinker_list); + shrinker->flags |= SHRINKER_REGISTERED; + shrinker_debugfs_add(shrinker); + up_write(&shrinker_rwsem); +} +EXPORT_SYMBOL_GPL(shrinker_register); + +void shrinker_free(struct shrinker *shrinker) +{ + struct dentry *debugfs_entry = NULL; + int debugfs_id; + + if (!shrinker) + return; + + down_write(&shrinker_rwsem); + if (shrinker->flags & SHRINKER_REGISTERED) { + list_del(&shrinker->list); + debugfs_entry = shrinker_debugfs_detach(shrinker, &debugfs_id); + shrinker->flags &= ~SHRINKER_REGISTERED; + } else { + shrinker_debugfs_name_free(shrinker); + } + + if (shrinker->flags & SHRINKER_MEMCG_AWARE) + unregister_memcg_shrinker(shrinker); + up_write(&shrinker_rwsem); + + if (debugfs_entry) + shrinker_debugfs_remove(debugfs_entry, debugfs_id); + + kfree(shrinker->nr_deferred); + shrinker->nr_deferred = NULL; + + kfree(shrinker); +} +EXPORT_SYMBOL_GPL(shrinker_free); + /* * Add a shrinker callback to be called from the vm. */ diff --git a/mm/shrinker_debug.c b/mm/shrinker_debug.c index 61702bdc1af4..aa2027075ed9 100644 --- a/mm/shrinker_debug.c +++ b/mm/shrinker_debug.c @@ -191,6 +191,20 @@ int shrinker_debugfs_add(struct shrinker *shrinker) return 0; } +int shrinker_debugfs_name_alloc(struct shrinker *shrinker, const char *fmt, + va_list ap) +{ + shrinker->name = kvasprintf_const(GFP_KERNEL, fmt, ap); + + return shrinker->name ? 0 : -ENOMEM; +} + +void shrinker_debugfs_name_free(struct shrinker *shrinker) +{ + kfree_const(shrinker->name); + shrinker->name = NULL; +} + int shrinker_debugfs_rename(struct shrinker *shrinker, const char *fmt, ...) { struct dentry *entry; @@ -239,8 +253,7 @@ struct dentry *shrinker_debugfs_detach(struct shrinker *shrinker, lockdep_assert_held(&shrinker_rwsem); - kfree_const(shrinker->name); - shrinker->name = NULL; + shrinker_debugfs_name_free(shrinker); *debugfs_id = entry ? shrinker->debugfs_id : -1; shrinker->debugfs_entry = NULL;