From patchwork Thu Feb 23 13:27:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13150325 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A662CC636D6 for ; Thu, 23 Feb 2023 13:27:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EB3D26B0078; Thu, 23 Feb 2023 08:27:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E647C6B007B; Thu, 23 Feb 2023 08:27:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D05EF6B007D; Thu, 23 Feb 2023 08:27:57 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id BF7766B0078 for ; Thu, 23 Feb 2023 08:27:57 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 93EA7A05F4 for ; Thu, 23 Feb 2023 13:27:57 +0000 (UTC) X-FDA: 80498634594.25.B2D65D4 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) by imf13.hostedemail.com (Postfix) with ESMTP id AE6B120023 for ; Thu, 23 Feb 2023 13:27:55 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=ZOzVR5Mq; spf=pass (imf13.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677158875; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HrlDcamq7Bi2YsIBW+HX78wqxYPHwSepVRJN6rWNlHI=; b=U6RdilrgJhgSuNkWK/rbI6WeGUR1Q6uTp19sD91dA/CGeKG93kh0qeX49h5SP0X/8qJy0J RqVTSazTUfyi7EaiqrQ3Xf4/RyR0SI+XSq46TqEM1pQiyL3Wj0c7gCCwRvSNwW/YuxnLHB QEEUAX6iEX0lstqIygzAjFOL/MBIxJ8= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=ZOzVR5Mq; spf=pass (imf13.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677158875; a=rsa-sha256; cv=none; b=P+Obun5dqtWG3BTeLSi40f0bIESnIq4ZAdvXz5x5X+PAJ687tzrkVs2aIWnw8Rk95WE2pE WdaAq20yIcQ3NQI+qd8JOgSkyar2cX4bxIvlO13s+GJY6KerreJHk6nCAdyM4uslZNYjxP ubN/nX5s6rO0WneYzfL70mPKbRE1yxs= Received: by mail-pl1-f175.google.com with SMTP id e9so7816670plh.2 for ; Thu, 23 Feb 2023 05:27:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HrlDcamq7Bi2YsIBW+HX78wqxYPHwSepVRJN6rWNlHI=; b=ZOzVR5Mq59QyAziI48QmC8waUhSJJ+JNAgYhrkofGCSeV4WavcSYIdZxOhkoYD32Uy lN22vIocIhlb1bTeM1MN3wOb2DPCrzoU9gEV0RzEOao1nVLaad65GzZ9eoRcdfu0yRIw 1AkgXLFFTpYJ6QCOy8HEgbyCf1dvNFXPCDLEPMYBpU2SBjVJ7fChrOZSBnJtmdulnM/u HHQn+mpyQ5p/8kzqT6hdqMUVjI0tcFTrz6VAVhYe2PsK4L3DmoGMWmnMb9trM8hOJWbn 4brJwLGR5uq6FPraW/MLHJfslc46DMOsDOrC999BVws6VMn/wx8kKhcEVvTP+hlgyV13 Yc0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HrlDcamq7Bi2YsIBW+HX78wqxYPHwSepVRJN6rWNlHI=; b=L32xXoO8Jq3NoHSinYzakUC4tPhHQw7JBZG4sb2TjKkGG+h8ucGPOpWmGXFlK0CB2W 5v61oCNOueBwJCI1ricpJS4WAKHKA6Ia+qsssq3ShIvZiU5vW6CHpg5L0unb4SUUPkou ajnyekfLK4bZHDrO9D/tOlKEJ7N7l6VgCAnkJx82BzrNzC+jbeOsCHHJ/QRnlPYYC/fx ZoottjwA5nN8y4+lKp9FxgHE3tjrz4hndoYE+1tHMZ0e5ri0VBy7pTcQIJUxFS7xEOH3 6R9kRasCz6X4Z1VzifgVXGpmz9TxTGI8w7lsgnYeLUex4EXlfyAuT59IpURlCxnuSZMU 4dRw== X-Gm-Message-State: AO0yUKX03/eBzCfVPu46OV79LbCT4fUYWYaQ7N9bVmMfnqS/4DFd3SRi tQCA6hQa6QvbWkPvI+/Hl3Kw3w== X-Google-Smtp-Source: AK7set/Q5vUZSF1XZejx+2VH547lt1MLTL0Wio2yOw08/SNhuOs+jhsXUnSmQScf+sU82FAyj/kmPA== X-Received: by 2002:a05:6a20:6914:b0:c7:5700:30bb with SMTP id q20-20020a056a20691400b000c7570030bbmr6050862pzj.4.1677158874314; Thu, 23 Feb 2023 05:27:54 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.245]) by smtp.gmail.com with ESMTPSA id g18-20020aa78752000000b005a9bf65b591sm3848591pfo.135.2023.02.23.05.27.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Feb 2023 05:27:54 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v2 1/7] mm: vmscan: add a map_nr_max field to shrinker_info Date: Thu, 23 Feb 2023 21:27:19 +0800 Message-Id: <20230223132725.11685-2-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230223132725.11685-1-zhengqi.arch@bytedance.com> References: <20230223132725.11685-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: AE6B120023 X-Rspam-User: X-Stat-Signature: zd77hzbowp4nmyuc11er588kpzkd3dw5 X-HE-Tag: 1677158875-639677 X-HE-Meta: U2FsdGVkX18/XFTkXzp/S7gt0aA3OWGSH+DOs9wWlEe4Wkv9Vyde9OxqaL9k6RKCKriX+tb+NDX8W+wFZLSns9o/BcEg379kew6STtYxu7HT9If4xLH6TGGu4HOVtTR0upWN+8mbEpJ/Lj8sXoYQMJkz41iS0lmnKu2nPxplxmocLEVxYaNP2Z//XabfoXa/ugRmJH95k9zYiTcPwb7K5GPK6k4L3PMSEkH67mZvmhQ4OoMGC43hpgYG5lU1UhWaJaErZ/4pfjQ7GWGlSyG62xyrw+6M1lK3CEBI4vfCiioUrYCa7lCKfXjOdGX5Ili0nsGccqBMghz29WHao/dSIqB24dEVO7+QZPw4Q8KP9ulooHQiov+76SZ+bd9OUfOvTyL2UMPoQbDmN9DC3pizBS862BVe4D96BOVaVL91D1f8uA9eFLZQlXIrBYkrOQqWhMNmarohFvKX87KLh9OMLZGt/CGp4ix8viDl9x1tQcvSb+NJm78oqXLUm6SZLeGpBcvloAruEs74S2fkAdiyAAbSp9M1Kx3i/BPD4BILxJm+rO7q5DqXUps36G3FO/xG6k03zXG3P/d+EK0ZfR4JtJFmhOFnGV80wO8EpSd+GYmhwCMkl5YEv8PggYjvlG7y4YDWFuAv0s0DWUd2Xwno4ef9rhY10Q7S0XIeEtNuf1WmuGzW2a2yq9N/SjKlrJMI/nP4qC+erPBUgjJpRlAxDj583bvd0qMrgk/XZZMvGKeiZ2B04S76UnqNZud9zFgLmzMz4RvSJqFxl0krpsVtcHyTE43H+WSBCHjKBYXFkiXybiyZcCBg7z9jwL5/LVYN9NzpHtZ9uSX7nYzjEeg7JD+sx975rn7E9dgKKCJoVqbJo3QuQD8SYWQEttERiOXh0E1Kyq02Ilws7qQU7A90n713AqQyeNeXqZ61loe+6zr0PMYPqnmwT7JwjNkrY73Rx9hUyb2nYzk4LYl/Nnw BxaDbnz3 p2wiiUySWFQEvvaMb2fy9tFFWt5gQL1dKf0/xcXauzPwAhptp2m+qVg+F0R80hQRIoa11oCPBNO4xk8zP8AfZyd3cqqwXMz88yvb2GiY5ehIbCC6G2/gJZ/NZDAuWfs+Kpq8LlnNaeIzf0/bGT45cSGipbblYSxQFqt3ke6j52UoWkbJrhDHpI1ve4rKAY3s7AKqOdRM0OPat+wgP8hGM3uAc2jurKciV59O2C4wOXjauMll7+WntuEfn9dTcCc6WUOiGUqJb8DIRjlAeSdHdFxPTz53u2SdNSasfj9jqGnQu71/D3w16X930DMQC7CXFNEa9iBL3xmQzjj0RZFuyQGjmn4WK/EMziJ/AL+/8hhtbI3x/MLuTQ7kjZD/fG+FTrsxOhrRqIyxX6Aeb4S+9CC+XLYIA0UAmKDRV8mD20A8Oybb3F+qzaUrkvSLwCE/DhnPcZWkp+ZJDVRKCgs7RTZjMBIlI3Um5ax0v X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: To prepare for the subsequent lockless memcg slab shrink, add a map_nr_max field to struct shrinker_info to records its own real shrinker_nr_max. No functional changes. Signed-off-by: Qi Zheng --- include/linux/memcontrol.h | 1 + mm/vmscan.c | 29 ++++++++++++++++++----------- 2 files changed, 19 insertions(+), 11 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index b6eda2ab205d..aa69ea98e2d8 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -97,6 +97,7 @@ struct shrinker_info { struct rcu_head rcu; atomic_long_t *nr_deferred; unsigned long *map; + int map_nr_max; }; struct lruvec_stats_percpu { diff --git a/mm/vmscan.c b/mm/vmscan.c index 9c1c5e8b24b8..9f895ca6216c 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -224,9 +224,16 @@ static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *memcg, lockdep_is_held(&shrinker_rwsem)); } +static inline bool need_expand(int new_nr_max, int old_nr_max) +{ + return round_up(new_nr_max, BITS_PER_LONG) > + round_up(old_nr_max, BITS_PER_LONG); +} + static int expand_one_shrinker_info(struct mem_cgroup *memcg, int map_size, int defer_size, - int old_map_size, int old_defer_size) + int old_map_size, int old_defer_size, + int new_nr_max) { struct shrinker_info *new, *old; struct mem_cgroup_per_node *pn; @@ -240,12 +247,16 @@ static int expand_one_shrinker_info(struct mem_cgroup *memcg, if (!old) return 0; + if (!need_expand(new_nr_max, old->map_nr_max)) + return 0; + new = kvmalloc_node(sizeof(*new) + size, GFP_KERNEL, nid); if (!new) return -ENOMEM; new->nr_deferred = (atomic_long_t *)(new + 1); new->map = (void *)new->nr_deferred + defer_size; + new->map_nr_max = new_nr_max; /* map: set all old bits, clear all new bits */ memset(new->map, (int)0xff, old_map_size); @@ -295,6 +306,7 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) } info->nr_deferred = (atomic_long_t *)(info + 1); info->map = (void *)info->nr_deferred + defer_size; + info->map_nr_max = shrinker_nr_max; rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_info, info); } up_write(&shrinker_rwsem); @@ -302,12 +314,6 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) return ret; } -static inline bool need_expand(int nr_max) -{ - return round_up(nr_max, BITS_PER_LONG) > - round_up(shrinker_nr_max, BITS_PER_LONG); -} - static int expand_shrinker_info(int new_id) { int ret = 0; @@ -316,7 +322,7 @@ static int expand_shrinker_info(int new_id) int old_map_size, old_defer_size = 0; struct mem_cgroup *memcg; - if (!need_expand(new_nr_max)) + if (!need_expand(new_nr_max, shrinker_nr_max)) goto out; if (!root_mem_cgroup) @@ -332,7 +338,8 @@ static int expand_shrinker_info(int new_id) memcg = mem_cgroup_iter(NULL, NULL, NULL); do { ret = expand_one_shrinker_info(memcg, map_size, defer_size, - old_map_size, old_defer_size); + old_map_size, old_defer_size, + new_nr_max); if (ret) { mem_cgroup_iter_break(NULL, memcg); goto out; @@ -432,7 +439,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) for_each_node(nid) { child_info = shrinker_info_protected(memcg, nid); parent_info = shrinker_info_protected(parent, nid); - for (i = 0; i < shrinker_nr_max; i++) { + for (i = 0; i < child_info->map_nr_max; i++) { nr = atomic_long_read(&child_info->nr_deferred[i]); atomic_long_add(nr, &parent_info->nr_deferred[i]); } @@ -899,7 +906,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, if (unlikely(!info)) goto unlock; - for_each_set_bit(i, info->map, shrinker_nr_max) { + for_each_set_bit(i, info->map, info->map_nr_max) { struct shrink_control sc = { .gfp_mask = gfp_mask, .nid = nid,