From patchwork Thu Feb 23 13:27:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13150325 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A662CC636D6 for ; Thu, 23 Feb 2023 13:27:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EB3D26B0078; Thu, 23 Feb 2023 08:27:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E647C6B007B; Thu, 23 Feb 2023 08:27:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D05EF6B007D; Thu, 23 Feb 2023 08:27:57 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id BF7766B0078 for ; Thu, 23 Feb 2023 08:27:57 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 93EA7A05F4 for ; Thu, 23 Feb 2023 13:27:57 +0000 (UTC) X-FDA: 80498634594.25.B2D65D4 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) by imf13.hostedemail.com (Postfix) with ESMTP id AE6B120023 for ; Thu, 23 Feb 2023 13:27:55 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=ZOzVR5Mq; spf=pass (imf13.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677158875; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HrlDcamq7Bi2YsIBW+HX78wqxYPHwSepVRJN6rWNlHI=; b=U6RdilrgJhgSuNkWK/rbI6WeGUR1Q6uTp19sD91dA/CGeKG93kh0qeX49h5SP0X/8qJy0J RqVTSazTUfyi7EaiqrQ3Xf4/RyR0SI+XSq46TqEM1pQiyL3Wj0c7gCCwRvSNwW/YuxnLHB QEEUAX6iEX0lstqIygzAjFOL/MBIxJ8= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=ZOzVR5Mq; spf=pass (imf13.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677158875; a=rsa-sha256; cv=none; b=P+Obun5dqtWG3BTeLSi40f0bIESnIq4ZAdvXz5x5X+PAJ687tzrkVs2aIWnw8Rk95WE2pE WdaAq20yIcQ3NQI+qd8JOgSkyar2cX4bxIvlO13s+GJY6KerreJHk6nCAdyM4uslZNYjxP ubN/nX5s6rO0WneYzfL70mPKbRE1yxs= Received: by mail-pl1-f175.google.com with SMTP id e9so7816670plh.2 for ; Thu, 23 Feb 2023 05:27:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HrlDcamq7Bi2YsIBW+HX78wqxYPHwSepVRJN6rWNlHI=; b=ZOzVR5Mq59QyAziI48QmC8waUhSJJ+JNAgYhrkofGCSeV4WavcSYIdZxOhkoYD32Uy lN22vIocIhlb1bTeM1MN3wOb2DPCrzoU9gEV0RzEOao1nVLaad65GzZ9eoRcdfu0yRIw 1AkgXLFFTpYJ6QCOy8HEgbyCf1dvNFXPCDLEPMYBpU2SBjVJ7fChrOZSBnJtmdulnM/u HHQn+mpyQ5p/8kzqT6hdqMUVjI0tcFTrz6VAVhYe2PsK4L3DmoGMWmnMb9trM8hOJWbn 4brJwLGR5uq6FPraW/MLHJfslc46DMOsDOrC999BVws6VMn/wx8kKhcEVvTP+hlgyV13 Yc0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HrlDcamq7Bi2YsIBW+HX78wqxYPHwSepVRJN6rWNlHI=; b=L32xXoO8Jq3NoHSinYzakUC4tPhHQw7JBZG4sb2TjKkGG+h8ucGPOpWmGXFlK0CB2W 5v61oCNOueBwJCI1ricpJS4WAKHKA6Ia+qsssq3ShIvZiU5vW6CHpg5L0unb4SUUPkou ajnyekfLK4bZHDrO9D/tOlKEJ7N7l6VgCAnkJx82BzrNzC+jbeOsCHHJ/QRnlPYYC/fx ZoottjwA5nN8y4+lKp9FxgHE3tjrz4hndoYE+1tHMZ0e5ri0VBy7pTcQIJUxFS7xEOH3 6R9kRasCz6X4Z1VzifgVXGpmz9TxTGI8w7lsgnYeLUex4EXlfyAuT59IpURlCxnuSZMU 4dRw== X-Gm-Message-State: AO0yUKX03/eBzCfVPu46OV79LbCT4fUYWYaQ7N9bVmMfnqS/4DFd3SRi tQCA6hQa6QvbWkPvI+/Hl3Kw3w== X-Google-Smtp-Source: AK7set/Q5vUZSF1XZejx+2VH547lt1MLTL0Wio2yOw08/SNhuOs+jhsXUnSmQScf+sU82FAyj/kmPA== X-Received: by 2002:a05:6a20:6914:b0:c7:5700:30bb with SMTP id q20-20020a056a20691400b000c7570030bbmr6050862pzj.4.1677158874314; Thu, 23 Feb 2023 05:27:54 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.245]) by smtp.gmail.com with ESMTPSA id g18-20020aa78752000000b005a9bf65b591sm3848591pfo.135.2023.02.23.05.27.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Feb 2023 05:27:54 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v2 1/7] mm: vmscan: add a map_nr_max field to shrinker_info Date: Thu, 23 Feb 2023 21:27:19 +0800 Message-Id: <20230223132725.11685-2-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230223132725.11685-1-zhengqi.arch@bytedance.com> References: <20230223132725.11685-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: AE6B120023 X-Rspam-User: X-Stat-Signature: zd77hzbowp4nmyuc11er588kpzkd3dw5 X-HE-Tag: 1677158875-639677 X-HE-Meta: U2FsdGVkX18/XFTkXzp/S7gt0aA3OWGSH+DOs9wWlEe4Wkv9Vyde9OxqaL9k6RKCKriX+tb+NDX8W+wFZLSns9o/BcEg379kew6STtYxu7HT9If4xLH6TGGu4HOVtTR0upWN+8mbEpJ/Lj8sXoYQMJkz41iS0lmnKu2nPxplxmocLEVxYaNP2Z//XabfoXa/ugRmJH95k9zYiTcPwb7K5GPK6k4L3PMSEkH67mZvmhQ4OoMGC43hpgYG5lU1UhWaJaErZ/4pfjQ7GWGlSyG62xyrw+6M1lK3CEBI4vfCiioUrYCa7lCKfXjOdGX5Ili0nsGccqBMghz29WHao/dSIqB24dEVO7+QZPw4Q8KP9ulooHQiov+76SZ+bd9OUfOvTyL2UMPoQbDmN9DC3pizBS862BVe4D96BOVaVL91D1f8uA9eFLZQlXIrBYkrOQqWhMNmarohFvKX87KLh9OMLZGt/CGp4ix8viDl9x1tQcvSb+NJm78oqXLUm6SZLeGpBcvloAruEs74S2fkAdiyAAbSp9M1Kx3i/BPD4BILxJm+rO7q5DqXUps36G3FO/xG6k03zXG3P/d+EK0ZfR4JtJFmhOFnGV80wO8EpSd+GYmhwCMkl5YEv8PggYjvlG7y4YDWFuAv0s0DWUd2Xwno4ef9rhY10Q7S0XIeEtNuf1WmuGzW2a2yq9N/SjKlrJMI/nP4qC+erPBUgjJpRlAxDj583bvd0qMrgk/XZZMvGKeiZ2B04S76UnqNZud9zFgLmzMz4RvSJqFxl0krpsVtcHyTE43H+WSBCHjKBYXFkiXybiyZcCBg7z9jwL5/LVYN9NzpHtZ9uSX7nYzjEeg7JD+sx975rn7E9dgKKCJoVqbJo3QuQD8SYWQEttERiOXh0E1Kyq02Ilws7qQU7A90n713AqQyeNeXqZ61loe+6zr0PMYPqnmwT7JwjNkrY73Rx9hUyb2nYzk4LYl/Nnw BxaDbnz3 p2wiiUySWFQEvvaMb2fy9tFFWt5gQL1dKf0/xcXauzPwAhptp2m+qVg+F0R80hQRIoa11oCPBNO4xk8zP8AfZyd3cqqwXMz88yvb2GiY5ehIbCC6G2/gJZ/NZDAuWfs+Kpq8LlnNaeIzf0/bGT45cSGipbblYSxQFqt3ke6j52UoWkbJrhDHpI1ve4rKAY3s7AKqOdRM0OPat+wgP8hGM3uAc2jurKciV59O2C4wOXjauMll7+WntuEfn9dTcCc6WUOiGUqJb8DIRjlAeSdHdFxPTz53u2SdNSasfj9jqGnQu71/D3w16X930DMQC7CXFNEa9iBL3xmQzjj0RZFuyQGjmn4WK/EMziJ/AL+/8hhtbI3x/MLuTQ7kjZD/fG+FTrsxOhrRqIyxX6Aeb4S+9CC+XLYIA0UAmKDRV8mD20A8Oybb3F+qzaUrkvSLwCE/DhnPcZWkp+ZJDVRKCgs7RTZjMBIlI3Um5ax0v X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: To prepare for the subsequent lockless memcg slab shrink, add a map_nr_max field to struct shrinker_info to records its own real shrinker_nr_max. No functional changes. Signed-off-by: Qi Zheng --- include/linux/memcontrol.h | 1 + mm/vmscan.c | 29 ++++++++++++++++++----------- 2 files changed, 19 insertions(+), 11 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index b6eda2ab205d..aa69ea98e2d8 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -97,6 +97,7 @@ struct shrinker_info { struct rcu_head rcu; atomic_long_t *nr_deferred; unsigned long *map; + int map_nr_max; }; struct lruvec_stats_percpu { diff --git a/mm/vmscan.c b/mm/vmscan.c index 9c1c5e8b24b8..9f895ca6216c 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -224,9 +224,16 @@ static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *memcg, lockdep_is_held(&shrinker_rwsem)); } +static inline bool need_expand(int new_nr_max, int old_nr_max) +{ + return round_up(new_nr_max, BITS_PER_LONG) > + round_up(old_nr_max, BITS_PER_LONG); +} + static int expand_one_shrinker_info(struct mem_cgroup *memcg, int map_size, int defer_size, - int old_map_size, int old_defer_size) + int old_map_size, int old_defer_size, + int new_nr_max) { struct shrinker_info *new, *old; struct mem_cgroup_per_node *pn; @@ -240,12 +247,16 @@ static int expand_one_shrinker_info(struct mem_cgroup *memcg, if (!old) return 0; + if (!need_expand(new_nr_max, old->map_nr_max)) + return 0; + new = kvmalloc_node(sizeof(*new) + size, GFP_KERNEL, nid); if (!new) return -ENOMEM; new->nr_deferred = (atomic_long_t *)(new + 1); new->map = (void *)new->nr_deferred + defer_size; + new->map_nr_max = new_nr_max; /* map: set all old bits, clear all new bits */ memset(new->map, (int)0xff, old_map_size); @@ -295,6 +306,7 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) } info->nr_deferred = (atomic_long_t *)(info + 1); info->map = (void *)info->nr_deferred + defer_size; + info->map_nr_max = shrinker_nr_max; rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_info, info); } up_write(&shrinker_rwsem); @@ -302,12 +314,6 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) return ret; } -static inline bool need_expand(int nr_max) -{ - return round_up(nr_max, BITS_PER_LONG) > - round_up(shrinker_nr_max, BITS_PER_LONG); -} - static int expand_shrinker_info(int new_id) { int ret = 0; @@ -316,7 +322,7 @@ static int expand_shrinker_info(int new_id) int old_map_size, old_defer_size = 0; struct mem_cgroup *memcg; - if (!need_expand(new_nr_max)) + if (!need_expand(new_nr_max, shrinker_nr_max)) goto out; if (!root_mem_cgroup) @@ -332,7 +338,8 @@ static int expand_shrinker_info(int new_id) memcg = mem_cgroup_iter(NULL, NULL, NULL); do { ret = expand_one_shrinker_info(memcg, map_size, defer_size, - old_map_size, old_defer_size); + old_map_size, old_defer_size, + new_nr_max); if (ret) { mem_cgroup_iter_break(NULL, memcg); goto out; @@ -432,7 +439,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) for_each_node(nid) { child_info = shrinker_info_protected(memcg, nid); parent_info = shrinker_info_protected(parent, nid); - for (i = 0; i < shrinker_nr_max; i++) { + for (i = 0; i < child_info->map_nr_max; i++) { nr = atomic_long_read(&child_info->nr_deferred[i]); atomic_long_add(nr, &parent_info->nr_deferred[i]); } @@ -899,7 +906,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, if (unlikely(!info)) goto unlock; - for_each_set_bit(i, info->map, shrinker_nr_max) { + for_each_set_bit(i, info->map, info->map_nr_max) { struct shrink_control sc = { .gfp_mask = gfp_mask, .nid = nid, From patchwork Thu Feb 23 13:27:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13150326 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 33AD5C61DA4 for ; Thu, 23 Feb 2023 13:28:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ABCA56B007B; Thu, 23 Feb 2023 08:28:04 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A6D3F6B007D; Thu, 23 Feb 2023 08:28:04 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 90D476B007E; Thu, 23 Feb 2023 08:28:04 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 8138C6B007B for ; Thu, 23 Feb 2023 08:28:04 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 33B18405CD for ; Thu, 23 Feb 2023 13:28:04 +0000 (UTC) X-FDA: 80498634888.17.81D5809 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) by imf13.hostedemail.com (Postfix) with ESMTP id 5001720012 for ; Thu, 23 Feb 2023 13:28:02 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=HuKGyT5g; spf=pass (imf13.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677158882; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=dmBjkO7XqRgvyMtVzOfBB5ntqP3V7YsqXGlIu6MjJGc=; b=ODxxpcvEiC5agUQbnj+QHydLbb+3fJ3edBtpD9IdX8NATrQ/jvnagc3150I1RmAQJlUTeW CLd2upINo3YDtdhEYsEMyIr/c8cJz8ZhK9D/025RIek93Nhy/BcAsi1JRHfdmMkHj9y3Ep DTL9HNtGPI8owxKD2Ke40recyDLaIuo= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=HuKGyT5g; spf=pass (imf13.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677158882; a=rsa-sha256; cv=none; b=T3Ug8U9hSZ7qDLgb2kxMFP7zHMuINWIFQHCIdZEgXiZDRpMFnZlYiaZqPr3WXyDNmL0G2N YocIBxyRKgZ7byPJs7nk0rsXBVxR1EBBBqbIy10VxjJ1W0iOwCsZmdr5AY0XVP3OCAoH5W 1b5l87vffSvYI42IXXk7Obqc/LNl47Q= Received: by mail-pl1-f175.google.com with SMTP id e9so7817125plh.2 for ; Thu, 23 Feb 2023 05:28:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=dmBjkO7XqRgvyMtVzOfBB5ntqP3V7YsqXGlIu6MjJGc=; b=HuKGyT5g9HX+trnuB9PFEZiZQnLZwCXwzriYOLJXDNbiFiIZPICp1ljy4qSOgSab2I NMzsv7Vmj7E/5NMfLc1BWIKOnCxL+BzoZpgKcnJYEltjWlGXo0fOwx9C6Pu3NsN9u46U Kv3wFXahyCB63qXHH0daVyjbjFbiPIcvdexuFVwLoVsLDYQ5kqjx3EO/D+d1pxqe6zU/ RO8l29x25cF9n7jVV5kbIhqnom267s/kJXkRQI4UJegeb5b/0C073tGKMlhkXpxOhbeS ztqiE80MiB/pXo60Z+CGiUXjkhVqipIt+okmhgJTC36MafGn2SoK2Bm4TUXoufO3abiD loIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dmBjkO7XqRgvyMtVzOfBB5ntqP3V7YsqXGlIu6MjJGc=; b=Xm1hDc74SaC+gD0LG7P635glwpIHzhq+yEEAyFZVUVuhQQFeM5lOL26x23re7XRDgO CaL9jdlonQHpijOi/vj6FknqVZn7YBnTxZKZ5ubRO5b4B9CZrV3fNLuea/Mva7GrBs0g k8gskMZsd5nOjwDJgMHAe+qjALwfFSR++AZ5DzcUfB9pI4TS6CJiizeKzibFpPbcQXWH uaXREDbV/jG5N84b9dCSjtaGpEWF4S79nN/yUQcUO/fdKBPkwYhB+KsdC/rNTwd7HEgu nNsOYepqR2xfNX81n5IMJtaz2d/mXuGgBb0mC4GmbctwpkeBe7mbQQV68PY6KGinWqKo ZxwQ== X-Gm-Message-State: AO0yUKVVlk4B6S0Q6dp+++AWr6szkO2RDt0W4oOx1KQPwdP82W21tsPQ 4J2WmP/+zhdO53QvHAagW03bHST3hjNiGisw X-Google-Smtp-Source: AK7set/Rzr6xcsH9ySEoci9FlTvKdtpASQ4i1a9Z/tDDR2gcVSm1DDUdIYED6NggeEbWXrwoEk7f6g== X-Received: by 2002:a05:6a20:7f9c:b0:cc:4118:65c4 with SMTP id d28-20020a056a207f9c00b000cc411865c4mr889206pzj.5.1677158881203; Thu, 23 Feb 2023 05:28:01 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.245]) by smtp.gmail.com with ESMTPSA id g18-20020aa78752000000b005a9bf65b591sm3848591pfo.135.2023.02.23.05.27.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Feb 2023 05:28:00 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v2 2/7] mm: vmscan: make global slab shrink lockless Date: Thu, 23 Feb 2023 21:27:20 +0800 Message-Id: <20230223132725.11685-3-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230223132725.11685-1-zhengqi.arch@bytedance.com> References: <20230223132725.11685-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 5001720012 X-Rspam-User: X-Stat-Signature: 68j5rayyw6de6jewk3nggwsxnfp3sfpg X-HE-Tag: 1677158882-49375 X-HE-Meta: U2FsdGVkX1/s1TP2pxRD6gDtWqPiVSoKSWPnQE15BML+b0zzCHvSX16MTEF0vUY1OMT2kva1PBszQ9wWRUewEyqWDCWXxKPcJzooholkPA/cu5XR2Mi3Ytge4GYdD6gT+HDTPs2qwNKWrx6d5zbkIYIPoMVau3GCq2xQUUxz+zmabkeoAprJdXAIkOUPF7MIlQKPCwssyR9DXvYD5kjQoLV35iYb8dX3Z/oCGKyslvPSMb17dKHQepO+GflHqZA+W2wjiJrAHrEXL7a+Wmr7JzSMo0lXkiS88D3dspe6iKUu3+qUNn4bCuXewHrO9BkYtNPuznypWs9k3SEy+NDXuEJPPt/optl4VT/wNeiJEAdg7BEQpzjUSJHwRGVwjGgzqhAFl2vefVPvN2mST+8Kx/TvGLxskPVamb6RZphX0L6FwEwVpKQOiTVYgm5RDeriUWmXWza09VuCR9MXCUaRBDIdD7dv//dQEfBI58ocN5ceF6hGw4/0Qsh+AsMwTQtDBsm+m3J0asPsd43KjhTw5JBY/cUCoAknaMSzO6KKwf5dmzGeblGQkyvYy5UVWqEFHrIT4/AhUPlq/13ivByesywvh1/hWUohAFCUAKRCi7B9lQVDFF6hwVMS0EafcNkyOupsK7JSLcA3N2aWkMDFd+t5HkXiXPefXkLdNRMsc/CL7aCT94+Gtug1FDTDOFHpqJlKMCUABhN/qU9wwY2CZ+X9A/lHwZMDT6/8Wsjgv1OpyGsyNdDLkEudL23OyjNZ3So2R9AYE8iXXwNxGOYTGzfHBjMZam3GlWi+k4Huq3srKJwwzjYRXEkjQ8/K0ey496yEWzImhtVjDKfvBMXnPRESDhZNQ0ku7QQGwF9To2YUzHNo4EzCqw3WnYZGYufILqBKQ1JQlkK0B1WbRnl0XfT2+nPoM2j0TxHk2Is2MfHV7XjoJDDp4miQedGvE8ePo8eANK6G8YlI801l63D UYqURBFs A0njVPivdcRMgNPXEOIvcKLJ2BK3q86hET2LlUXbhd8AhpvwK5/BueSJ3R9nqhfV3eL9tJmHUKircq0UVsRwiFppRQwdCI/D1N3xIDr/fueeD3oR7TOfD8HEYl4pXJiA6ROCTGy1t8OfKwgwCMxoNN9QgW6lzxJpWMlMTfdx4YB+q/h4Rd+UUM8uvjm1EvHB4MhxsAvm1M7m6UEFJwy2n9Qoe4KHIsiynPTIHyGio1vpOYddpkJ/GkLzkgWnTeXFj2ooCemBebdorSQsQeZeaDP+4LOgR/g28vpnesrnRyoCQQYf1mH+j/fvEKGoPfsF6BE6iqL4BS80DV++bCArEjxXxcKbEXT66cEGO+WelMq2u8TMVahvzbi0GzVZDIIa7mgI0Y8f4zIDGBzFGfLzIzf114BUPWRxn9tXza4WGj5/GdTdZn1k4GIEEYhYSApZWSaDw5vB3FtJxAKlsNazi3nuuJfM5wLVDtxx/yh6zyom1W11yng7iLMjunYh0GLTxME2iOt9A7Nf4yNij0vBnfsyCeUNPc9514K8oYILaHsVefZJXl0vvSqGnn84DdY9hj+Q89ihD3cag/64P1I5gYMBobEeZRzehzkdT9tOMHPl80k3rCOatw4WjgOaGxgKbW39M09fXKfZRQ7S4NkwysF8W3u4Ec3UYcsPq X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The shrinker_rwsem is a global lock in shrinkers subsystem, it is easy to cause blocking in the following cases: a. the write lock of shrinker_rwsem was held for too long. For example, there are many memcgs in the system, which causes some paths to hold locks and traverse it for too long. (e.g. expand_shrinker_info()) b. the read lock of shrinker_rwsem was held for too long, and a writer came at this time. Then this writer will be forced to wait and block all subsequent readers. For example: - be scheduled when the read lock of shrinker_rwsem is held in do_shrink_slab() - some shrinker are blocked for too long. Like the case mentioned in the patchset[1]. Therefore, many times in history ([2],[3],[4],[5]), some people wanted to replace shrinker_rwsem reader with SRCU, but they all gave up because SRCU was not unconditionally enabled. But now, since commit 1cd0bd06093c ("rcu: Remove CONFIG_SRCU"), the SRCU is unconditionally enabled. So it's time to use SRCU to protect readers who previously held shrinker_rwsem. [1]. https://lore.kernel.org/lkml/20191129214541.3110-1-ptikhomirov@virtuozzo.com/ [2]. https://lore.kernel.org/all/1437080113.3596.2.camel@stgolabs.net/ [3]. https://lore.kernel.org/lkml/1510609063-3327-1-git-send-email-penguin-kernel@I-love.SAKURA.ne.jp/ [4]. https://lore.kernel.org/lkml/153365347929.19074.12509495712735843805.stgit@localhost.localdomain/ [5]. https://lore.kernel.org/lkml/20210927074823.5825-1-sultan@kerneltoast.com/ Signed-off-by: Qi Zheng --- mm/vmscan.c | 27 +++++++++++---------------- 1 file changed, 11 insertions(+), 16 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 9f895ca6216c..02987a6f95d1 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -202,6 +202,7 @@ static void set_task_reclaim_state(struct task_struct *task, LIST_HEAD(shrinker_list); DECLARE_RWSEM(shrinker_rwsem); +DEFINE_SRCU(shrinker_srcu); #ifdef CONFIG_MEMCG static int shrinker_nr_max; @@ -706,7 +707,7 @@ void free_prealloced_shrinker(struct shrinker *shrinker) void register_shrinker_prepared(struct shrinker *shrinker) { down_write(&shrinker_rwsem); - list_add_tail(&shrinker->list, &shrinker_list); + list_add_tail_rcu(&shrinker->list, &shrinker_list); shrinker->flags |= SHRINKER_REGISTERED; shrinker_debugfs_add(shrinker); up_write(&shrinker_rwsem); @@ -760,13 +761,15 @@ void unregister_shrinker(struct shrinker *shrinker) return; down_write(&shrinker_rwsem); - list_del(&shrinker->list); + list_del_rcu(&shrinker->list); shrinker->flags &= ~SHRINKER_REGISTERED; if (shrinker->flags & SHRINKER_MEMCG_AWARE) unregister_memcg_shrinker(shrinker); debugfs_entry = shrinker_debugfs_remove(shrinker); up_write(&shrinker_rwsem); + synchronize_srcu(&shrinker_srcu); + debugfs_remove_recursive(debugfs_entry); kfree(shrinker->nr_deferred); @@ -786,6 +789,7 @@ void synchronize_shrinkers(void) { down_write(&shrinker_rwsem); up_write(&shrinker_rwsem); + synchronize_srcu(&shrinker_srcu); } EXPORT_SYMBOL(synchronize_shrinkers); @@ -996,6 +1000,7 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, { unsigned long ret, freed = 0; struct shrinker *shrinker; + int srcu_idx; /* * The root memcg might be allocated even though memcg is disabled @@ -1007,10 +1012,10 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, if (!mem_cgroup_disabled() && !mem_cgroup_is_root(memcg)) return shrink_slab_memcg(gfp_mask, nid, memcg, priority); - if (!down_read_trylock(&shrinker_rwsem)) - goto out; + srcu_idx = srcu_read_lock(&shrinker_srcu); - list_for_each_entry(shrinker, &shrinker_list, list) { + list_for_each_entry_srcu(shrinker, &shrinker_list, list, + srcu_read_lock_held(&shrinker_srcu)) { struct shrink_control sc = { .gfp_mask = gfp_mask, .nid = nid, @@ -1021,19 +1026,9 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, if (ret == SHRINK_EMPTY) ret = 0; freed += ret; - /* - * Bail out if someone want to register a new shrinker to - * prevent the registration from being stalled for long periods - * by parallel ongoing shrinking. - */ - if (rwsem_is_contended(&shrinker_rwsem)) { - freed = freed ? : 1; - break; - } } - up_read(&shrinker_rwsem); -out: + srcu_read_unlock(&shrinker_srcu, srcu_idx); cond_resched(); return freed; } From patchwork Thu Feb 23 13:27:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13150327 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6E687C636D6 for ; Thu, 23 Feb 2023 13:28:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DC8D26B007D; Thu, 23 Feb 2023 08:28:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D79656B007E; Thu, 23 Feb 2023 08:28:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C4B7B6B0080; Thu, 23 Feb 2023 08:28:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id B5C196B007D for ; Thu, 23 Feb 2023 08:28:10 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 63F371A015F for ; Thu, 23 Feb 2023 13:28:10 +0000 (UTC) X-FDA: 80498635140.20.A29112C Received: from mail-pg1-f177.google.com (mail-pg1-f177.google.com [209.85.215.177]) by imf12.hostedemail.com (Postfix) with ESMTP id 92ED140019 for ; Thu, 23 Feb 2023 13:28:08 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=hQ0T37qt; spf=pass (imf12.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.177 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677158888; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Syv0drMPggMMkttRMMkloSH5R6RYZ1YZAM5MesH77qc=; b=zMDrofz9kFfp6O/oCeQaAOnmtKwFCdk1VUTe+6ppObCNRg7i6h40dsRSSE2rEZeJGrKCBQ 41mx8PuFg8qWwpBfaQpKpwVyaZvzv/sZknRhf7VQxczriOsKQfofOwrB/3cj9uczdO7kKQ KLhvrsCREXBVhWQM5bmD41oJNEYI7Z4= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=hQ0T37qt; spf=pass (imf12.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.177 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677158888; a=rsa-sha256; cv=none; b=GA3zqLmSS3/UK41UhtGSUk2qD7H4nJ7au8cQpc/OIOEllMB98I4mkecAV5eFfUAHEPs5iy HZesfS+8Pak8yjL3WTNFhGiddKVEj5IBEnuMoHXT6K3BV6t3056RFx1LCJn3BZRpoAvVS0 UC70B9xBU4ZHHfEjFycnZ7CeugWczk4= Received: by mail-pg1-f177.google.com with SMTP id q189so5576449pga.9 for ; Thu, 23 Feb 2023 05:28:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1677158887; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Syv0drMPggMMkttRMMkloSH5R6RYZ1YZAM5MesH77qc=; b=hQ0T37qts3rJeuZtmDQARv10wVciADPxTYE7nHXMnr9/N5MAW0787FBKg4eYhKDGIR D+o6nrvNrTCJZgauHW1A4MDrBHjxksREY4lTnxwUa84Fsm5l7e+fbi03X46BgEoXdVQ+ dOAn3WrLOyf4bs99Rq8TgWRjbgf9v7Q1uHlPOpcxV2y0uxirjEy5+OIZMhdR26f/63fU P2Pmq2VOLIEelMQtA+O9p5Iee99sRG2fY+Tl1/DItQIUTAgSBqbuxd7az9ZKW/udBx0b cWC6J1VhJL/7sSkSUyQ5U+ZKGWW3NEEgGBmU1h341ArROsLUeQb5G50uiU2GeEJsA+K+ hEqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1677158887; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Syv0drMPggMMkttRMMkloSH5R6RYZ1YZAM5MesH77qc=; b=thIT5dxUJqwjFXJYxJRJractDXLHe9Int8iDEnx87qdVd4zLjo/NWFecbGPzXspspf XtFTbvElr5sqS5QaTp7aq7FD3H2sAhfw3DtriKjg6Vi3fAIE1MkuzGN/L1hURdU7jqaW 0xyP+/mA3lfAarf038TVC1+Us1iRsvLKZkMBpsXcx/vUGB9AoG+jKuEE1uXW/513RMak 6QBAnQkgnCEnbrL6xbRo8nfLXCWz6sgXlE/yORWTi87wCTu2uoaFE5qxiHsRyYHDoKdN neuobFTTbmdoKrMw3umD1nwAsVzjMtbJDkiPl4dGxcwqk5CTEF+rWstiXA5h3Dv8J58b 2big== X-Gm-Message-State: AO0yUKVG656hHp/a6cZOQGRcoteQ06qJ+VCSzhsqjzxDNHPUxHRbnJv7 JYz7knj1JobI9h0BxtVRro7lyA== X-Google-Smtp-Source: AK7set/3KhLevmt8IbsCAmbb8GxI559C7KBiMEGLPkGOAY2YkyfrjWjj15z30QZAy/t94RwKaPBlcQ== X-Received: by 2002:aa7:90cf:0:b0:5a9:c2d5:136c with SMTP id k15-20020aa790cf000000b005a9c2d5136cmr12399865pfk.3.1677158887363; Thu, 23 Feb 2023 05:28:07 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.245]) by smtp.gmail.com with ESMTPSA id g18-20020aa78752000000b005a9bf65b591sm3848591pfo.135.2023.02.23.05.28.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Feb 2023 05:28:07 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v2 3/7] mm: vmscan: make memcg slab shrink lockless Date: Thu, 23 Feb 2023 21:27:21 +0800 Message-Id: <20230223132725.11685-4-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230223132725.11685-1-zhengqi.arch@bytedance.com> References: <20230223132725.11685-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 92ED140019 X-Stat-Signature: y4geenyznqg3ptttmaertsipibm1zdkq X-HE-Tag: 1677158888-914044 X-HE-Meta: U2FsdGVkX18AfGViE2tXGNb7+wsWsMGh3FBi7yvULj2K4DlnZFJxZmT3iZFMWfKVC/4QXFHrzh3NK8ynJvqMyGfLb8kJAw6RHKujHBqMT62E3C04udq7qSu2+TSFuD4Zij4VPuZMH6szlBXOp49+edcFS6OgZcCCP6ReicAzChDprulylc4JeE6MWPmXi0d37+EKWdgCoqw1MaX8B77KRNuUxja7ArlUyXXRfXGkac0+MhoVEzyJ3nEciNpGNDA4K7tryFqGGit1bCcRWyubZK0QxrEWDFrfvNKEN7VJ2TI1z6EPkQij2J1aAXFJGQ4K/MgIcvwUaEDdZeWaz6MTxNDe77JlCLBOIXaOqPPBHxewqsAMQfIxKLy2vcTwsQX+NzbZd5h/t+gKyDmSettl3TUyqwQEXPN2DUBBngBmGQ3cSHwNhw/MrPNVvpTP3FCidQDCgXlLhfUYzmIfTXVw9+gjE35oX7Dwyvd48PlMYXR8EYSt2nacoO0QwazSU7L13+2luACOIMGC/X/cRuOPKw2wuVnWtcVIwVT1gB/Y5LZKsftmS/OJ0Y8/v4H8Z4UXhHl+ZT4pvQSbkyLBlwv62Sse35wZeA3m96B9mK/IT+HqkT0khm4Yc/U5wUZvRCxERKQQNOSVO5+LUPevh1Jxumn5zHGUNoddMU0kAOqfxr2M1FLi93YC/OlWeeqXI8rtT0m0/JQlyN+p4MHg1w34bUhU6He7DUE8ZGibZUs4+H89f1ZcrhK7rhWbNE6z6xCtQ6y4emYFvBwf2xIAg3f8uztXuaZKLjqKpwWEAaWJMDUtrCo/4TUsDyhr01LZU8beHHQY0g2iFPaZmwzQXQSnpbuotTJzejZaDyz56ea7Obc0fC29qlydJMVqhviXYD6NsqNmVMh6HIXVCey9KNauOyL4L3dRVDLsFOb4tTCJ6pzCLlgF7P/4RXwkFF64+D5tpwk1EO+9vc/22ByEiar wA1tn4vQ g4DevlWVUY69fXxpFwzElFnxxK4GMciNrA2aYVJhFlmiJGoCevU5okw29TYNdwEv51/XixJLOblNBwD9T8Pc1QAR1DI+PPhGdvdu0QSggXtvtx+3HQ6r2HOLWO93KHr92ANXb7Jk9rAeD3VHI6gOdLQ3j3aczb7ddxEldEyYeldgdWmc95mLUrdUihl+HhZtlEmt62M2juBHPwLNDC46uciGmYJuiv/NdmL+Q6n5Q/oqO29nTatu8eQvZkpDjnSUwQ/virN/Czw6tWCHO5XbJFZjdmOfhVvpjak58Tn8jfd4d63Xv5KCScAsklUlnHRdg4XeoGoHlo/cMODTKDjlU5Xe3wb+5rt0tLW8ta0c0zfj8cNEpy2EFdhWCLATgbJrpQdXrVjqtQPH6yTds5frHrvsbnuLROt/+A+MAKr5LvhnjG0k= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Like global slab shrink, since commit 1cd0bd06093c ("rcu: Remove CONFIG_SRCU"), it's time to use SRCU to protect readers who previously held shrinker_rwsem. We can test with the following script: ``` DIR="/root/shrinker/memcg/mnt" do_create() { mkdir /sys/fs/cgroup/memory/test echo 200M > /sys/fs/cgroup/memory/test/memory.limit_in_bytes for i in `seq 0 $1`; do mkdir /sys/fs/cgroup/memory/test/$i; echo $$ > /sys/fs/cgroup/memory/test/$i/cgroup.procs; mkdir -p $DIR/$i; done } do_mount() { for i in `seq $1 $2`; do mount -t tmpfs $i $DIR/$i; done } do_touch() { for i in `seq $1 $2`; do echo $$ > /sys/fs/cgroup/memory/test/$i/cgroup.procs; dd if=/dev/zero of=$DIR/$i/file$i bs=1M count=1 & done } do_create 2000 do_mount 0 2000 do_touch 0 1000 ``` Before applying: 46.60% [kernel] [k] down_read_trylock 18.70% [kernel] [k] up_read 15.44% [kernel] [k] shrink_slab 4.37% [kernel] [k] _find_next_bit 2.75% [kernel] [k] xa_load 2.07% [kernel] [k] idr_find 1.73% [kernel] [k] do_shrink_slab 1.42% [kernel] [k] shrink_lruvec 0.74% [kernel] [k] shrink_node 0.60% [kernel] [k] list_lru_count_one After applying: 19.53% [kernel] [k] _find_next_bit 14.63% [kernel] [k] do_shrink_slab 14.58% [kernel] [k] shrink_slab 11.83% [kernel] [k] shrink_lruvec 9.33% [kernel] [k] __blk_flush_plug 6.67% [kernel] [k] mem_cgroup_iter 3.73% [kernel] [k] list_lru_count_one 2.43% [kernel] [k] shrink_node 1.96% [kernel] [k] super_cache_count 1.78% [kernel] [k] __rcu_read_unlock 1.38% [kernel] [k] __srcu_read_lock 1.30% [kernel] [k] xas_descend We can see that the readers is no longer blocked. Signed-off-by: Qi Zheng --- mm/vmscan.c | 46 +++++++++++++++++++++++++++------------------- 1 file changed, 27 insertions(+), 19 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 02987a6f95d1..25a4a660e45f 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -57,6 +57,7 @@ #include #include #include +#include #include #include @@ -221,8 +222,21 @@ static inline int shrinker_defer_size(int nr_items) static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *memcg, int nid) { - return rcu_dereference_protected(memcg->nodeinfo[nid]->shrinker_info, - lockdep_is_held(&shrinker_rwsem)); + return srcu_dereference_check(memcg->nodeinfo[nid]->shrinker_info, + &shrinker_srcu, + lockdep_is_held(&shrinker_rwsem)); +} + +static struct shrinker_info *shrinker_info_srcu(struct mem_cgroup *memcg, + int nid) +{ + return srcu_dereference(memcg->nodeinfo[nid]->shrinker_info, + &shrinker_srcu); +} + +static void free_shrinker_info_rcu(struct rcu_head *head) +{ + kvfree(container_of(head, struct shrinker_info, rcu)); } static inline bool need_expand(int new_nr_max, int old_nr_max) @@ -268,7 +282,7 @@ static int expand_one_shrinker_info(struct mem_cgroup *memcg, defer_size - old_defer_size); rcu_assign_pointer(pn->shrinker_info, new); - kvfree_rcu(old, rcu); + call_srcu(&shrinker_srcu, &old->rcu, free_shrinker_info_rcu); } return 0; @@ -357,13 +371,14 @@ void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) { if (shrinker_id >= 0 && memcg && !mem_cgroup_is_root(memcg)) { struct shrinker_info *info; + int srcu_idx; - rcu_read_lock(); - info = rcu_dereference(memcg->nodeinfo[nid]->shrinker_info); + srcu_idx = srcu_read_lock(&shrinker_srcu); + info = shrinker_info_srcu(memcg, nid); /* Pairs with smp mb in shrink_slab() */ smp_mb__before_atomic(); set_bit(shrinker_id, info->map); - rcu_read_unlock(); + srcu_read_unlock(&shrinker_srcu, srcu_idx); } } @@ -377,7 +392,6 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) return -ENOSYS; down_write(&shrinker_rwsem); - /* This may call shrinker, so it must use down_read_trylock() */ id = idr_alloc(&shrinker_idr, shrinker, 0, 0, GFP_KERNEL); if (id < 0) goto unlock; @@ -411,7 +425,7 @@ static long xchg_nr_deferred_memcg(int nid, struct shrinker *shrinker, { struct shrinker_info *info; - info = shrinker_info_protected(memcg, nid); + info = shrinker_info_srcu(memcg, nid); return atomic_long_xchg(&info->nr_deferred[shrinker->id], 0); } @@ -420,7 +434,7 @@ static long add_nr_deferred_memcg(long nr, int nid, struct shrinker *shrinker, { struct shrinker_info *info; - info = shrinker_info_protected(memcg, nid); + info = shrinker_info_srcu(memcg, nid); return atomic_long_add_return(nr, &info->nr_deferred[shrinker->id]); } @@ -898,15 +912,14 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, { struct shrinker_info *info; unsigned long ret, freed = 0; + int srcu_idx; int i; if (!mem_cgroup_online(memcg)) return 0; - if (!down_read_trylock(&shrinker_rwsem)) - return 0; - - info = shrinker_info_protected(memcg, nid); + srcu_idx = srcu_read_lock(&shrinker_srcu); + info = shrinker_info_srcu(memcg, nid); if (unlikely(!info)) goto unlock; @@ -956,14 +969,9 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, set_shrinker_bit(memcg, nid, i); } freed += ret; - - if (rwsem_is_contended(&shrinker_rwsem)) { - freed = freed ? : 1; - break; - } } unlock: - up_read(&shrinker_rwsem); + srcu_read_unlock(&shrinker_srcu, srcu_idx); return freed; } #else /* CONFIG_MEMCG */ From patchwork Thu Feb 23 13:27:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13150328 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8092BC636D6 for ; Thu, 23 Feb 2023 13:28:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0F7CD6B007E; Thu, 23 Feb 2023 08:28:17 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0A8466B0080; Thu, 23 Feb 2023 08:28:17 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E8BCB6B0081; Thu, 23 Feb 2023 08:28:16 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id D97116B007E for ; Thu, 23 Feb 2023 08:28:16 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id AC39B160342 for ; Thu, 23 Feb 2023 13:28:16 +0000 (UTC) X-FDA: 80498635392.24.179E589 Received: from mail-pg1-f179.google.com (mail-pg1-f179.google.com [209.85.215.179]) by imf06.hostedemail.com (Postfix) with ESMTP id C90F0180003 for ; Thu, 23 Feb 2023 13:28:14 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=DpDn8Y+E; spf=pass (imf06.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.179 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677158894; a=rsa-sha256; cv=none; b=UVx1GlqLgvFySY1tFIk1xSAutZkhEzGOjpns4/LZvQqvg/sdICgidzbY7yS5ROnN8Z4KEd IK8+/xN1o9g6svoNcuHI1f4tWXxHdDURBH/HTrOjKozITJPYZ9d08ju72DnOWKz879bMUf EnmGfDFUMYJM67I35x6rWO5IvVfxI3g= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=DpDn8Y+E; spf=pass (imf06.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.179 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677158894; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4ra+jS9qPdJDcLL3tCNVjDgmNt8q5h7NH0ipc1XhyKg=; b=WxsEf3ArUmu6e/BfFMsnjspBjNFOoDUsaPKi2lSR1910/CKUKz0AV2XbbIwsvwBI6aQYpr bkmNJto3kkAajgqUsOn7i+lgLbC4OJJD05u7G+bWN9LaB7VxIDw+pr2W9IKnEXOhGXqN8x t4hrAZiCnMhGJcpHT+7xTF6aEsAdmCo= Received: by mail-pg1-f179.google.com with SMTP id bn17so78562pgb.10 for ; Thu, 23 Feb 2023 05:28:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4ra+jS9qPdJDcLL3tCNVjDgmNt8q5h7NH0ipc1XhyKg=; b=DpDn8Y+EU7riH4+ntcDG6C1rbsBVGtUkMmoAHDS+pC4P++8EynweyAV0JYqlyYGOtq X1Q+BLwoJwcnB84jpi52eVKNQ4Z7H9xuUGTHfERfoIzzzU6ACyTdYigUXyW6IU9Ious4 Dcv7MyI3TgRfz/xnXu5LeFECNtdpthD72ekZGWDFwliGf58bj+lRnE/qeFjqbyzPR/dU QS9YrLbbOfovo0TIt/QhGWwwvNLxWcvBspvW0zUIZAUavOJoYjXfSG6Wy0VfFlInlS7B HNvoy7xqBZa0nU3FIEoV776Q0WiC55EpZXUOtrYeZXGMXYVopQbU7AkcvEji/pZV0uXq Rv+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4ra+jS9qPdJDcLL3tCNVjDgmNt8q5h7NH0ipc1XhyKg=; b=ngEcht1NyWixXvIAwMGyP8ddOhB0WD3J9vjB7DEgoi33C5el99CXouJ9hsmbZ63DyH L2Zjz37PQkqxEbx1ajpzZFzZ9mEsfkhxZg5+jXC4omcmfqfz88e643mhnLJ7S7n4PxQE HFidi7c6ZdVIFbiE7YqhHCyNDBdkkluiqtcpSGfsA4RgF10WC7SFxGOLP2Y6kFd5DHSe 2ikfpyHW+LZKxhAOoFThlXPJlWWTcsYTEfLWoysUL2mJSmNxkLdM12AzwJWd9notjZsR WLLy9M2i/cy/Zoz5anouzJT9PvyHnkAAUugWI09AKBhRUFoTa8zKZjOhNE1De3uyWhRD IpIg== X-Gm-Message-State: AO0yUKX5OE0fd3T3IGuxQiTE0Ld8F/HBokRrS/aU3aMw+32ITJ4Vytqd TJvhB/dWAYYmUie7sbxLQDCEqQ== X-Google-Smtp-Source: AK7set8Z/82YLh8rwyoUePgrhBmvgU8h/2aIZeBSX/0142PpIqvK6RYABRkBwNOTr58e5iHS5l9UsQ== X-Received: by 2002:a05:6a00:299b:b0:5a8:ae97:25f2 with SMTP id cj27-20020a056a00299b00b005a8ae9725f2mr12832291pfb.0.1677158893792; Thu, 23 Feb 2023 05:28:13 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.245]) by smtp.gmail.com with ESMTPSA id g18-20020aa78752000000b005a9bf65b591sm3848591pfo.135.2023.02.23.05.28.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Feb 2023 05:28:13 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v2 4/7] mm: shrinkers: make count and scan in shrinker debugfs lockless Date: Thu, 23 Feb 2023 21:27:22 +0800 Message-Id: <20230223132725.11685-5-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230223132725.11685-1-zhengqi.arch@bytedance.com> References: <20230223132725.11685-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: C90F0180003 X-Rspamd-Server: rspam01 X-Stat-Signature: 4pdofrpf5c9k3gacpqo4f99x6a4n3fc6 X-HE-Tag: 1677158894-551687 X-HE-Meta: U2FsdGVkX1+Xk2HiVpa9abqLLmBz5Xqh6Nztpz/8Nj7yARh0UzPn30JyTGmH/JDm7bNle2AR/H1M5NayPHQcB55YhmHjvK8P/oDf+8GDRnGDGX0ssfCSdcEPZytEofMp0nbVZEYYziE0GoPQVpFhaaDtflKpnmJJMk6fhWlpwwyUAIf8cK1ETFUgvHHmmoRNNIxx95Nfrb+EE7KV4rICN3nW5oth8bCGk1tkmepc049G24yEOizFnOsJ3Wm/MCvCqf+MKt0elS/iPsNsMJxGftAt8HRF5Qz1XLt7MIZpbutCST3Z6aRoSP04R6EIDRRmeZ8GSQGsRCnFTARU5rjzga63vJZCBppqeOCReaA3BofRo71DmMbBtUWmiP0HIoH2BR9I1uy+uQpmGA7uGl9G7PxdBCx5Zr+P1SHjH8wLFU40ZLS1olz7JvGifgXVuBlxlm2ptUKMEyoSEZ+0dvcK4FE6Zvh1k4FPey200zNWJj8NVR72E0I++i9szRG+x0WmWWrpW0MdID97bRoVh6N4FrWW+TlOXDnKxjbyZ5dkvC9uInZqZaTuaiUcU4x//M1nh0Wv5k0FGlYdqwkPOByL8wpbz+x+Nk62qrf8i1kQdV2mR1eZTTKE8qz1DBou7pr2WQ+qLKJ3h387Qri8/IGc7Vpi7ZU682neC1u1vkRWS+CkyNWGyOaoXjwB5DvinESAJ3isOd518Y7Pa8pLsXqOr+h6Z/BSVLttwCwiGvcIVzAE9oMwBfhCx7+TGVRkQLswBTqQgPG3YvG3l94uw+XG7ATM8SBcQ9ibU/pK80buS6X6bDm3HPhxm5KVVSTc4t94qIBx5rY1Vz9juv4XfrpNkCE8jqkagaPJ5scppfmQQquIlVuy+gCI2/EbsqyR5EU9f5aBO+QOWuEIzGd+Q4HvQmmhZ7rHH/0rpgVrDfeEvJioICLzH+r1mVpGURUZG8ZPy0q/g3eUn0BeO5D3T/a od5Qn5PY p3eA8YIzkiffhajRvglJn99grGXm82oJ9rW4uTj1nZPC5RQBdnVsqBgD3nOCJf6A362tGxvOtt11LNsXqoZsdib4Bvwmz2xux7dl+3CUDB/M3wXgPFRSZX+/JTIK2QVhyWYPTPW1seTH+gMt+z+VlEHMrnkEeX59Kc/riv3yOQOa4+JXSNZ3/shkTTKe6ToWKUtYKKsj4LTqvb43M6Wnjb9hJua/13hBfAPfX/yXhpgTeXij2sJ+F6eEhHlySV2+OGbhmgr3kJYtHi1Ng/C9adiZO5fEc9GJsIy10cQDdIN+6r03NNUVy/U8DOqoQw0csUJNXWn0hy4IV8Bc4b4e0DMb5s2QM8zbt5afrhx5Pk3KD91RPgFlIW4lkXJ7MnUqdDUqNbkx0+rrtTC23OUX2xHrp2sjEyDuRxr48UCuKxuNLoiU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Like global and memcg slab shrink, also use SRCU to make count and scan operations in memory shrinker debugfs lockless. Signed-off-by: Qi Zheng --- mm/shrinker_debug.c | 24 +++++++----------------- 1 file changed, 7 insertions(+), 17 deletions(-) diff --git a/mm/shrinker_debug.c b/mm/shrinker_debug.c index 39c3491e28a3..6aa7a7ec69da 100644 --- a/mm/shrinker_debug.c +++ b/mm/shrinker_debug.c @@ -9,6 +9,7 @@ /* defined in vmscan.c */ extern struct rw_semaphore shrinker_rwsem; extern struct list_head shrinker_list; +extern struct srcu_struct shrinker_srcu; static DEFINE_IDA(shrinker_debugfs_ida); static struct dentry *shrinker_debugfs_root; @@ -49,18 +50,13 @@ static int shrinker_debugfs_count_show(struct seq_file *m, void *v) struct mem_cgroup *memcg; unsigned long total; bool memcg_aware; - int ret, nid; + int ret = 0, nid, srcu_idx; count_per_node = kcalloc(nr_node_ids, sizeof(unsigned long), GFP_KERNEL); if (!count_per_node) return -ENOMEM; - ret = down_read_killable(&shrinker_rwsem); - if (ret) { - kfree(count_per_node); - return ret; - } - rcu_read_lock(); + srcu_idx = srcu_read_lock(&shrinker_srcu); memcg_aware = shrinker->flags & SHRINKER_MEMCG_AWARE; @@ -91,8 +87,7 @@ static int shrinker_debugfs_count_show(struct seq_file *m, void *v) } } while ((memcg = mem_cgroup_iter(NULL, memcg, NULL)) != NULL); - rcu_read_unlock(); - up_read(&shrinker_rwsem); + srcu_read_unlock(&shrinker_srcu, srcu_idx); kfree(count_per_node); return ret; @@ -115,9 +110,8 @@ static ssize_t shrinker_debugfs_scan_write(struct file *file, .gfp_mask = GFP_KERNEL, }; struct mem_cgroup *memcg = NULL; - int nid; + int nid, srcu_idx; char kbuf[72]; - ssize_t ret; read_len = size < (sizeof(kbuf) - 1) ? size : (sizeof(kbuf) - 1); if (copy_from_user(kbuf, buf, read_len)) @@ -146,11 +140,7 @@ static ssize_t shrinker_debugfs_scan_write(struct file *file, return -EINVAL; } - ret = down_read_killable(&shrinker_rwsem); - if (ret) { - mem_cgroup_put(memcg); - return ret; - } + srcu_idx = srcu_read_lock(&shrinker_srcu); sc.nid = nid; sc.memcg = memcg; @@ -159,7 +149,7 @@ static ssize_t shrinker_debugfs_scan_write(struct file *file, shrinker->scan_objects(shrinker, &sc); - up_read(&shrinker_rwsem); + srcu_read_unlock(&shrinker_srcu, srcu_idx); mem_cgroup_put(memcg); return size; From patchwork Thu Feb 23 13:27:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13150329 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 36A93C61DA4 for ; Thu, 23 Feb 2023 13:28:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CDE506B0081; Thu, 23 Feb 2023 08:28:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C8E606B0082; Thu, 23 Feb 2023 08:28:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B30196B0083; Thu, 23 Feb 2023 08:28:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id A30B56B0081 for ; Thu, 23 Feb 2023 08:28:23 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 6BDB8160248 for ; Thu, 23 Feb 2023 13:28:23 +0000 (UTC) X-FDA: 80498635686.21.DDD0E69 Received: from mail-pf1-f180.google.com (mail-pf1-f180.google.com [209.85.210.180]) by imf17.hostedemail.com (Postfix) with ESMTP id 9AF9540010 for ; Thu, 23 Feb 2023 13:28:21 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=P9pp+ZyM; spf=pass (imf17.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.180 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677158901; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=B8oNy9C+Y/zmSEnFHZueuiP36FW/sue7PAxuyUaJKNA=; b=kSsLI9DhpKy5L6zOcWfdFlKy1w+PC+lFMp4ceUrd9g0e8jqxb4AugTVEDP893Hc3h96725 agzj8jBphqM9m8b6rWOMi8IiAeW9cHdKhxawaCjCMQUYI5XnLxQHw36CQ7NSZKSY9zA35E hms8IbEudDyKK+qW2CjeJ0K4IH/tZ7g= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=P9pp+ZyM; spf=pass (imf17.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.180 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677158901; a=rsa-sha256; cv=none; b=kLHFvJu4zMa6bQ5puXJiVJDNakTALuwvuyG6RwiU7W52AGWK6nsdFaOoUTvTboBl4ZgDg1 NYxTltIJ02sHBJvzbPFeIeWgCLdEHWPRQbIr9VQZU8Tat9MCXSEN+iVT3PxgC7pQZQVGyE wWdSYYp5ByDcZm6UJPyTO5nJ3BQYNLE= Received: by mail-pf1-f180.google.com with SMTP id b20so6677638pfo.6 for ; Thu, 23 Feb 2023 05:28:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1677158900; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=B8oNy9C+Y/zmSEnFHZueuiP36FW/sue7PAxuyUaJKNA=; b=P9pp+ZyMjawowl2CXNZ8AfWA4FDgA5DbRRV8Xgup0TXSnw3gGKc4YOluAUAPyTi+8N Eowu4hN9SdqhC2D6NScYL5anKi0WPpJZ/kDZVTjDQu+/GSTSXQemisE6KgdRU/vueJCH m5Z2d5bsrBDjNoX+11dfufhV13nj8waePZO9KexhDHGlwIWF2Eic+82YX2EcOzsUP/aF sdp2wkR1m8GXzkKkL1ndWZsnYALrWwQQa3JIsN3tgm5LwRWjlG9XLi3n5hjONfp4f/gx actRJ9hq1VYMZLcfQYjdow3np6b8hCQxlhtnE9rDWirXh8AxOKVOLv1vhYi0fp0ratLL LzNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1677158900; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=B8oNy9C+Y/zmSEnFHZueuiP36FW/sue7PAxuyUaJKNA=; b=ScjJH/YYIU3azHIBcjAZh5nKJNHUDFZcXFJUnZ0n27bn8zo6iHaOrkoKeOZbZu25+P 3z9z6WON/2jGGLVL+XOv25VqK+EzapPdjRIJytQxX5bHoaG7dWznP98/beQeM3cuUD6O 4MAkvYzc4wxiK/swyvJdAUMUCxOIwd2lPEHuLEcZ22wXzEOiWxptut1VppUw7VY+y+2M lCzEos1V2JJc3r31B+KoTJdakVBTxJayioeiBCJJYbf8A15V13OTmkvgNGtDJqTP4YLj tigBuKxJYxwio6Ie1fGT9siNeBqU+c+lAWk5dIA6gicBXTx9Kwsj0u+WSi5o1KuuL22G OeOg== X-Gm-Message-State: AO0yUKV90v8SAiZDMvdRYrYIqu9343bM3wzEoOtox90QSosaYppFwXfq 9PSEUZBaZT8JfXJTvENcEhmDvg== X-Google-Smtp-Source: AK7set/qxxKJpUWiKsq4lL0Upj2vUZx+VL5loQcaP10700XUcw0lQhPoWvn8hF2CG9W/WlEr4c7i8A== X-Received: by 2002:a05:6a00:2e83:b0:5d9:bfc9:a4f with SMTP id fd3-20020a056a002e8300b005d9bfc90a4fmr3754231pfb.3.1677158900390; Thu, 23 Feb 2023 05:28:20 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.245]) by smtp.gmail.com with ESMTPSA id g18-20020aa78752000000b005a9bf65b591sm3848591pfo.135.2023.02.23.05.28.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Feb 2023 05:28:20 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v2 5/7] mm: vmscan: hold write lock to reparent shrinker nr_deferred Date: Thu, 23 Feb 2023 21:27:23 +0800 Message-Id: <20230223132725.11685-6-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230223132725.11685-1-zhengqi.arch@bytedance.com> References: <20230223132725.11685-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 9AF9540010 X-Stat-Signature: 8ex3b7ygdp7ipit1dn6m6pcm6tannwxx X-HE-Tag: 1677158901-225673 X-HE-Meta: U2FsdGVkX1+6NJVpXRn/oSNU3nUot7Yj71CaFskqKmN/XKciWOTKHVQyPFy+pyR2WO5aaR9Qkzk2JxDaYv4PL/atOWBg68MNR4Tuu+/Lt1/usOCs9Q1i8WtdqqrgwLRpnGzUgX8/JNuOAfjxMYW43s2rjUjWFxdTcRi5yNuMnVXBGwaTR9ZnzB/MmwLo1LOPsaYF9OrCVNrJIF8fRsjK/bliJTmLviPy+Qf1t1EDXl8wBlA5v3eF8jSeM8Tu5du/7pW9ODj6AoyTLufGJ1cZZ7LYB7IL4fi4nw1U88HXQfJFQffCaX0oqztUz9Qz4flSIXKdzpuoqmGCttm4nbzX+EB0FWbRw2CFYDjuDf7NNHxcMTNDnEMCbxhw61qdeluYV/K/u7PzGgEI+fosIlK2fk7tliv4X3w1JuMf3xX3wsxt1nruy12kRnVuvpw5v3AjPBE2eUOnSG7SgjAzkq+Aq9S5KDrvheKfyHKSCE50utlONfVi97OaFDagKFwJgzDOgtosg0QYRqwlIawgfyi/ugci+DD5LKrlhDYkReSdDT13tADL/0ygO7E2jt7No1zTemr4uviSZx/b7pIjWCEtM8AS2DfN8agAakJyxvs5Usa7qU0YBEt3SX+IzhfCdfVoety0Fn4tZjfJDY9Zfieye5h0tZHi7r72Z3u9ak8RYY5kfqKPw+CiUfm9EYls7iBxQVepa8pkcbxlSzCa3voCmuL5RMTIJT/9NFQLjP1oVXIusFe8zW7szD9/R8CCS1HfU2zyjbl7KPvZYHrCVoj+qnhdZol7vXOXTsEjT1y27Vri7l6a6n/qMQKb4Q6LhTn5U/Px2ej93HJFUN16KOfoD9ALhDlIqMGH1ip5MfNxcCEAG26w2wlUaQ1YIVYaZrx4qUVzS/N1DNnK2/E7gbOVI4icetiiftKoDs2WhrW1ud2GUxnRucdMm9BmJv2VhjmhuuPyFYNT0NGwX/lhAOE QJ1QQn+o 1A6iupH3Gfx7DqGdKbFRO+ZqW9ipJxLJWU6YekUailJjaYp438LE8nqeWqp9QeJVm0T5TJkxNaNZlMkzycLqlSjAJ/iTTUUcCDvVop3KVeLeOQ+orJ93ndBexSfuFnQu2ZIVX3O0DNEwaV1jsdmARVZY+ZEXiyt+Cn4VkePjVoB9pStWuX+NUSrg8ZZ9NpBoqJYLv2dKWnDapX7xpwF7uGpPHfrSX+0Z45mHv3aH4WBPOzszfmIgsVVXCiqEnPbMWWfO1T4iqFCCD1tAwWMN1S0XTBYI745Bigtqr4H8d/KL5OO+FehN/27fmxS/TXHHO4KCXv9I+1A7GTY7JCGJX7zD+1Bg/QfEcyNOiyaIE0cGznHsbzob+Br2gBVd6J5geVIp1Wetdg4e21bPnaNJXreGmJpjouVHDhcderyz/148BZm5yIvNXUi03COOEsnXDhQ6pFm4RBL0z4DkqqOeffeT973WWVsuXdCd4 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: For now, reparent_shrinker_deferred() is the only holder of read lock of shrinker_rwsem. And it already holds the global cgroup_mutex, so it will not be called in parallel. Therefore, in order to convert shrinker_rwsem to shrinker_mutex later, here we change to hold the write lock of shrinker_rwsem to reparent. Signed-off-by: Qi Zheng --- mm/vmscan.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 25a4a660e45f..89602e97583a 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -450,7 +450,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) parent = root_mem_cgroup; /* Prevent from concurrent shrinker_info expand */ - down_read(&shrinker_rwsem); + down_write(&shrinker_rwsem); for_each_node(nid) { child_info = shrinker_info_protected(memcg, nid); parent_info = shrinker_info_protected(parent, nid); @@ -459,7 +459,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) atomic_long_add(nr, &parent_info->nr_deferred[i]); } } - up_read(&shrinker_rwsem); + up_write(&shrinker_rwsem); } static bool cgroup_reclaim(struct scan_control *sc) From patchwork Thu Feb 23 13:27:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13150330 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 349C8C636D6 for ; Thu, 23 Feb 2023 13:28:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CDEDA6B0071; Thu, 23 Feb 2023 08:28:29 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C8F526B0073; Thu, 23 Feb 2023 08:28:29 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B2F8F6B0082; Thu, 23 Feb 2023 08:28:29 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id A26BD6B0071 for ; Thu, 23 Feb 2023 08:28:29 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 7FB7E1A0681 for ; Thu, 23 Feb 2023 13:28:29 +0000 (UTC) X-FDA: 80498635938.27.8722F17 Received: from mail-pj1-f47.google.com (mail-pj1-f47.google.com [209.85.216.47]) by imf07.hostedemail.com (Postfix) with ESMTP id AAA9B40015 for ; Thu, 23 Feb 2023 13:28:27 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=lETgDORH; spf=pass (imf07.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.216.47 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677158907; a=rsa-sha256; cv=none; b=bRHVYEph7cc9SCAOSpxWqfvViTg3WhCYrWnLujqxYZ1Da8/7OiYg6wAltaXDYBXxZIbtK4 wsTUc/3f0wXHtQEOdMvY5Fa1BGufJv6lERi+UGp/6B6SaH1O7GJt+Ea8BCDx5xLtMe/1T/ wqoKkXDvP2ctMyTFZYLF6xIziDXum0M= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=lETgDORH; spf=pass (imf07.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.216.47 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677158907; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=CCg67xK9T4mK7fnQXbRxQLg+KO6dWMg4osRlqGhwwto=; b=wtG30G4t/6GmhDhnCoKBleLRAG7dvK/Mn+ZEUCCFSeTOqLnsIEXL/Y5P7zLTs5l4xSU1YR P4HBMcajml40lRi6ILC4CDKQ+QGQJVY9n50MhHHliRhSd3Ie4PPYUXeT2T306drtU54AlB xenWUnBWAdke9zLoptiwJ1csGtRTl6o= Received: by mail-pj1-f47.google.com with SMTP id m8-20020a17090a4d8800b002377bced051so28218pjh.0 for ; Thu, 23 Feb 2023 05:28:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1677158906; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=CCg67xK9T4mK7fnQXbRxQLg+KO6dWMg4osRlqGhwwto=; b=lETgDORHGvM9YstoHGbA1w0c/zosszK5Tl/tNTL5K5GecaMVH3OBn+bE4nEakg6egq Pvrj1imBQ3mvMDd4ZkkT+KTl+KI7731rrPWGJr93xhck6jDmBdEQXOnXXEO1QfhSK6RX ygXjpnptKdeVKz4W3Pi6mq9d0qV+R8FOe534PaajduVj/IHLnCHezzp/AdV++OCktNuP 8w/SRgZpNscp1wPYqJzwvIGsa1g6aKBO/N4rrdP+sT3ClXNrbTOpm5Hjwya0rBJVadLP 3Nf3Tv+Z4D5QSw8ejD088MsxJyTzHJO0ORVGeG8DQ79R3RuVcqlVVKIQp+9BWC5EuICj qEMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1677158906; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CCg67xK9T4mK7fnQXbRxQLg+KO6dWMg4osRlqGhwwto=; b=aJ/96k2gX2ES1hNQ/2JFqfxk8Zd9oZ/HxXtUU7YbM80OCtL7XeQKFsp+7P5hEUUDFx We3g1OLx/D8QWdB0330+YHYpyTufpcu98zGb0gLLv6dgFUyyVuzcnDi7S7ZjWP66dgWw BzPtxH4DvN35pHwYNm1ldBr77J409leTijG8so2euXDjWn/pHjWzjASUe/ztMdj5RBHX Wx8fJ7TpUlYBlU34HB/SSmbDkLTHHpQjpekQjSgyAMr3P5DHvfAEEMs9vPgMD/PRN/NA CPe679ggx7/bpBhk9g/WmUPSE7LjZgPAz116jA2Zf3y22g/LRrcG4VwoXx3bBWkqKJhr M0gg== X-Gm-Message-State: AO0yUKWIh31aZyXTMIJDL18cZDW7mwHGnZ5EtNFOjGVBL+uHn0PsjqQ3 srXKpLMXzwY57uLx7BrPyTBtvw== X-Google-Smtp-Source: AK7set/QZbVb36Sv4JLFW6OnERnzffyx6wAbjX3Cubg0yhkLOIgaCU3CSjc6M3m8hEj3yZzy5iBl6A== X-Received: by 2002:a05:6a21:33a4:b0:cc:35f5:1a84 with SMTP id yy36-20020a056a2133a400b000cc35f51a84mr1073433pzb.5.1677158906681; Thu, 23 Feb 2023 05:28:26 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.245]) by smtp.gmail.com with ESMTPSA id g18-20020aa78752000000b005a9bf65b591sm3848591pfo.135.2023.02.23.05.28.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Feb 2023 05:28:26 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v2 6/7] mm: vmscan: remove shrinker_rwsem from synchronize_shrinkers() Date: Thu, 23 Feb 2023 21:27:24 +0800 Message-Id: <20230223132725.11685-7-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230223132725.11685-1-zhengqi.arch@bytedance.com> References: <20230223132725.11685-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: AAA9B40015 X-Rspamd-Server: rspam01 X-Stat-Signature: 8x7mjyc4od13ixsnnjpaie7bc19rpcu9 X-HE-Tag: 1677158907-447489 X-HE-Meta: U2FsdGVkX19Uwb3IwYd8pt7KzMwvcy6HKbmsnGsnBGjIYhObU53FrBRjP7i6nN492nnm9Hab1+PJIGNaSI6QzgffnAq3lAlVV0IMKYhOXSPtg4/ylALoaQQ4uxPodroSQ00IwLLuQp7adGEtQUOIVGSqz3rystU94kd2ImoetppR3s1saOfMi1BTA6ZfMN4N4MvkNGCzCTgm+vLeibxCdhjLXC8fbVNGb55fd8k7mPGThrrRHWfVGFzE7wzbczwcsWJkpnCjEbStKUzBlvCOFEcUAtzQJktBpdPi30ZgroMspwrqKRy/M1/h5fR+qnq1ACKH14K/gjPdc0A5V3AJVk2YZManO6QIk7UWlGL5j1E+vLJ1hO2DcznZSIze/iDAv+xZ2HaF9Yujhj9qs7LLMNBUqy5QOlDuHBW1CIOrk17DHulHujCuc8YnbbHtcEeL4SSxt03gwZ9c99Rs8g4qQo4+66JVlXkf2TxDX1LUVi1c/3BSz+btf+CQrnxb5Z+nyBlHO4kTRqa6ndTS99yUHTNa1Qwe4/rjZSH3iA4c5UtJOd/gy+UqZcESmRfRYsnmdIIAz74SuCB5q+bUEvHKleWRsKoPCP57ZhYMHV+UNPKgw5phOtRM0weGQ4NqXrx6vLEln2VJMQL5Fr6ZVOzgg3r2UKyh1YhN13NbBiS/u9N/CoB3huDPK0SKhSpypcLM3+Wcvk/Ea0F60IfWhg4jentjnQe3RHeavLw/wgxcTXXKn2kQjnaM3akpa/q9iFbFFUriBXLT49X1lGdcGDaaKiu3zTYG5r7eeO3AtOjmdUdQ9VZKjdzrnqDCRHrn2Camr63tGxC76XKE1sQNZZaOYxPvr414w/sbETBiKPVeZB9tlmY8t21tCOx6vR7D6HHAvr5ZdPa8qSRv5NEX7BCCs9jCfyejGZ3kHD6S/Qih+GwbBr70L8xwtl+1cKcw1gnZ0jVyeoX7imWp5CvvXSl bu3STbp6 WdMYcRbRnvN9oCiRzZB/18+awVNgfn4WWKbL81Q+7/dpErrfdOJRfA8xpMr0slyuI60LZIhoMHjAosLrrGiSRzA1jaFcDZ5dN79VE7j5+UgCocAGwO40JnZL3Llaz2uVpFALPr+NDEf5zroM27gQCOlLroIj87bn2zw/f1H+Qww88o+7HM0RM5G9LmZLkTGWEWe8oeUDYXnYjNZJ3ZQDmCb0GfgxvMiv3wR59+EX0+nxF5XJRw0gkgo6bos+3gb97fXYI+7/zfYmn7zj7J7P1f0rTAQlFBAvNLhzfKwHUx6szXkwhBj3ghURxtkJZeUwcZ+fvdoWeUcpSRJXJ+MwHmDlTd9nk/j1zEa30c1iJqqqCqmRszPrr6odYHRtpIKJbrj1w5ZM1Me6yGZ2/fVnsjpj5FMQxjhWoY77Cu92DawMNNlumBivtFnE1crxhtz8BS+2VV5ghmbUGx6fCcn5azN5orQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now there are no readers of shrinker_rwsem, so synchronize_shrinkers() does not need to hold the writer of shrinker_rwsem to wait for all running shinkers to complete, synchronize_srcu() is enough. Signed-off-by: Qi Zheng --- mm/vmscan.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 89602e97583a..d1a95d60d127 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -794,15 +794,11 @@ EXPORT_SYMBOL(unregister_shrinker); /** * synchronize_shrinkers - Wait for all running shrinkers to complete. * - * This is equivalent to calling unregister_shrink() and register_shrinker(), - * but atomically and with less overhead. This is useful to guarantee that all - * shrinker invocations have seen an update, before freeing memory, similar to - * rcu. + * This is useful to guarantee that all shrinker invocations have seen an + * update, before freeing memory. */ void synchronize_shrinkers(void) { - down_write(&shrinker_rwsem); - up_write(&shrinker_rwsem); synchronize_srcu(&shrinker_srcu); } EXPORT_SYMBOL(synchronize_shrinkers); From patchwork Thu Feb 23 13:27:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13150331 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6478BC636D6 for ; Thu, 23 Feb 2023 13:28:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 082D46B0073; Thu, 23 Feb 2023 08:28:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 032DC6B0082; Thu, 23 Feb 2023 08:28:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E15FC6B0083; Thu, 23 Feb 2023 08:28:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id D41DD6B0073 for ; Thu, 23 Feb 2023 08:28:35 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id A0328804B1 for ; Thu, 23 Feb 2023 13:28:35 +0000 (UTC) X-FDA: 80498636190.04.CC3A086 Received: from mail-pg1-f171.google.com (mail-pg1-f171.google.com [209.85.215.171]) by imf21.hostedemail.com (Postfix) with ESMTP id C436D1C0002 for ; Thu, 23 Feb 2023 13:28:33 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=F520qmCv; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf21.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.171 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677158913; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=RPT0nfWhMLdXxcydHyGYHy0yeE/SVJFKwHgDhpkqgDM=; b=A394GCJ/3YTpDeAtXUhhelMEusA6U85Zk3GjbOYY1ZIrEL+PbbIYssvldDbYbtS/GZW91x 5wCNkDsx3trhXcCN36gzWB2YeGeSQt+hw0J8YRRrZjiUVKv+bc+DBfVX4VMSbkTKv0OPYF Ks1UDkqKeWjLQNIoPj2/2eB/Ur2CQO0= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=F520qmCv; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf21.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.171 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677158913; a=rsa-sha256; cv=none; b=AMWsSDagiMt8dlesipoUQu687WVqrlVKamaHdOL191PmpXvEG4rW4u+j/d6cwgZbJ3QY4d /H7RlpBWt5CKqeK/c698mE6EH5P0KFc5k0vyryv2MMsPZKq22GUqu8KEX8NsQszxiUV1t5 tggrwSLqcztAqPYxY9v5QSKMzDfnGzg= Received: by mail-pg1-f171.google.com with SMTP id d6so2394815pgu.2 for ; Thu, 23 Feb 2023 05:28:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1677158912; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RPT0nfWhMLdXxcydHyGYHy0yeE/SVJFKwHgDhpkqgDM=; b=F520qmCvl0UEhHNkuOAEZmp1N3sJcaWVf0fYb3nWYbFg7xaH1VreJ8NIxoJDOHmFWr fELDUoigMokRIqsdrxc8qGRde4TK1nfn/8m7o3tkhJa2mENEjDgadGrxnY5rqaQzkamU /RQ3vIZgVEe9OraPz+1/MMz1EuxiVvI+ZzojzqD2odCLdWNcrBtEzqVENWF6dvVswnZp a5ziG4nnVMEgF32yQD6np3wXBRQz+Q5XT41L4iM1wqrQtqSrf4Zc9extvwH12AOU6F09 0I920tJGW3EBzguj8MSWNnfH8JyQvo7rAULuvJmoQCsifU2+4i+3CRIKl8d3zaj+Lf9T fE0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1677158912; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RPT0nfWhMLdXxcydHyGYHy0yeE/SVJFKwHgDhpkqgDM=; b=WieeDClsvAJUQROir8oUtEaa9w1XGDQXyZeLFXdmeoNtypj8RfWaSNwMRTM7DHPex0 FWoPfostDcpC9Z5dkHmrIlrC8i6exklcnnZbxzB11Z976FHC0vEdgHnRL4vqVw7pxdpM +4LH542ZjgCtbbEbWlCWIPDwSyGwpthgmK+yCvc+Ni4S7GP8JtwGZrUnDlYVANLduLuB yJ6bZrZlKfkSglioeNCVqZDEzqXH5gvUnT83wI8PAHspInTqMd+Ey9Xo+7K8t6CoicMq ISqkuhk0RdnNlw+L/ooYQWkMJ9YbRCRzl7NzIqhnuCF31zNseYhwPOylsePoElyiR/pN fOZQ== X-Gm-Message-State: AO0yUKVXqYzzrxKyOEwA3Muwyp6ZniSdfQl9Ly1+tMdalAOQ1o2S0Ntx KA2wmTT4Ih2NIO+0M98co3EUyQ== X-Google-Smtp-Source: AK7set/SbEDbsO6C9L1y9CGpBIcDIvy9sr0sucapoT0kr6uR9zXluUZnIOxXK5HS7OjmFknadGlwnA== X-Received: by 2002:a05:6a00:4006:b0:5d9:f3a6:ef8e with SMTP id by6-20020a056a00400600b005d9f3a6ef8emr3764664pfb.2.1677158912664; Thu, 23 Feb 2023 05:28:32 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.245]) by smtp.gmail.com with ESMTPSA id g18-20020aa78752000000b005a9bf65b591sm3848591pfo.135.2023.02.23.05.28.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Feb 2023 05:28:32 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v2 7/7] mm: shrinkers: convert shrinker_rwsem to mutex Date: Thu, 23 Feb 2023 21:27:25 +0800 Message-Id: <20230223132725.11685-8-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230223132725.11685-1-zhengqi.arch@bytedance.com> References: <20230223132725.11685-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: C436D1C0002 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: g1gr8t9o6tohimsa85fq9bobh87zes8q X-HE-Tag: 1677158913-890708 X-HE-Meta: U2FsdGVkX1/mzQxBzAsqmYeNecTXxVkyiQsDBSYezqTzVh8ejf3mJWjYnf1lXonVRtDozH4EpXrHP3SRChz3/nzQsz/VOMD0GaVgCkgnVfSlcMurOaSWtmeKXSOJ14zapQBza4asofCwjD2npDcKzwrCasdQPrQaJIcCTHdrFQSxzXhVs9dLsrH4Iwpjqa/VPfi4F/W5D7zeR5vWFDwf84uko+L7Y9v7vUhR4ibmFutre+MKjUaB4opHFGe8YyOoUhaFwDf8wki0LtpFBj8pyZ3L99XLWED7cTZyJ8bKk1KX4q/GZ23xMUMZvr4VbYe+wZa2ygmP9znqglOfEAlOA/I0145VhNH3t/ZFH3a4EOf+iTMvhd8vR497ncU087/izYcHcBu1uTs0MKlDNSYv3SdFJDgfNnAfoRvBjGluElAm75HNkbLVglAjJXrTXgcEDc5BsFEyYutc8+p0LC0QsRkke+sOpQnQ9wRt2wxgzDJWJ0Vuab9Hcr9FyvdkZtm3HHzV0Q5VPDvHd7nMg/O4YlBDBGNTpYsvEVbwUWKoaUDhMKRpMNlYVWETr7UOfP0G6CBIO8unNqiMhETCM3w2xjTtO0qQ1apyjC7sOjO65t1wBWTgAQeCWmOhcTPGxEwwuZUVkdbYld0OCCLYJSSvuYLRLKkrj8vxDJpiERovQJx6oNuSc3V7GoEC5KK6IaFExkH0ZiQaGqz+4k611KobjJ5ALXKjaqO4ycYPQsw7fCxOscFaBG4a9J3sXFo/DHkF/lsksYdZPBFNwRDyI+OpwwM4wdJmD+n207jAlWRVDSKHVfLUb+ZX2EO9boTvou5L8Dtysfza/1bWU1Sio0oO6yzTgWI7DKEOZk2P3TeEZQ+ZVGEH6akUhx2oTN3EbO87uO7BRe/tko1oy4fPCw4NFrUhLbIY+agZc7GaJMX6FkM2Ap2Rgmbnj92Ebdab5q7HOn7ByW0cQF17zsHxx+i cF6ugK9J TljgHbdreHqhYrKf39YgIc+5TcU7yrs31KCpZZrj0t8+6U9XI9PnZjerFEN4npBKLH3tivJ8DIaxBjPs64P24Ao21mQE7ErS2JZbQ/w5xxcnBvojBE/1tdTjs8xwb5M2O6d1T+LQSsxDdLx7jCUW+zfrQrkRbYgh99HDCTAYSmzBdKwYfu6ZAMsqEyWJueUsv9d5bvFjGb2/Xpbd5WlA+YRS8F/qsq2wbANrKjLHaUf/x3JWpDRBZzMrtz3qFCe6veP0ZhvkV+WIyD80OCpIpKxAed+bfxiXMoRXYbFx/TGkIEGJheWd+SicklSsP2hfIDVoHjcIni79vInSsaoNcFK0+r3XrUWZLJQeIBeVlNu5xTUaXN39TihGva0jFaQqcl8GPBiGQcOOk5sndgQlxV/c3nym6CRjT2Nh3F8XDusFB/Vj8ta6AqLNqkVplCdbPBYjaK8b/HoH7yHWhXBaAdfInD4jSdPzVJFUu X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now there are no readers of shrinker_rwsem, so we can simply replace it with mutex lock. Signed-off-by: Qi Zheng --- drivers/md/dm-cache-metadata.c | 2 +- drivers/md/dm-thin-metadata.c | 2 +- fs/super.c | 2 +- mm/shrinker_debug.c | 14 +++++++------- mm/vmscan.c | 34 +++++++++++++++++----------------- 5 files changed, 27 insertions(+), 27 deletions(-) diff --git a/drivers/md/dm-cache-metadata.c b/drivers/md/dm-cache-metadata.c index acffed750e3e..9e0c69958587 100644 --- a/drivers/md/dm-cache-metadata.c +++ b/drivers/md/dm-cache-metadata.c @@ -1828,7 +1828,7 @@ int dm_cache_metadata_abort(struct dm_cache_metadata *cmd) * Replacement block manager (new_bm) is created and old_bm destroyed outside of * cmd root_lock to avoid ABBA deadlock that would result (due to life-cycle of * shrinker associated with the block manager's bufio client vs cmd root_lock). - * - must take shrinker_rwsem without holding cmd->root_lock + * - must take shrinker_mutex without holding cmd->root_lock */ new_bm = dm_block_manager_create(cmd->bdev, DM_CACHE_METADATA_BLOCK_SIZE << SECTOR_SHIFT, CACHE_MAX_CONCURRENT_LOCKS); diff --git a/drivers/md/dm-thin-metadata.c b/drivers/md/dm-thin-metadata.c index fd464fb024c3..9f5cb52c5763 100644 --- a/drivers/md/dm-thin-metadata.c +++ b/drivers/md/dm-thin-metadata.c @@ -1887,7 +1887,7 @@ int dm_pool_abort_metadata(struct dm_pool_metadata *pmd) * Replacement block manager (new_bm) is created and old_bm destroyed outside of * pmd root_lock to avoid ABBA deadlock that would result (due to life-cycle of * shrinker associated with the block manager's bufio client vs pmd root_lock). - * - must take shrinker_rwsem without holding pmd->root_lock + * - must take shrinker_mutex without holding pmd->root_lock */ new_bm = dm_block_manager_create(pmd->bdev, THIN_METADATA_BLOCK_SIZE << SECTOR_SHIFT, THIN_MAX_CONCURRENT_LOCKS); diff --git a/fs/super.c b/fs/super.c index 84332d5cb817..91a4037b1d95 100644 --- a/fs/super.c +++ b/fs/super.c @@ -54,7 +54,7 @@ static char *sb_writers_name[SB_FREEZE_LEVELS] = { * One thing we have to be careful of with a per-sb shrinker is that we don't * drop the last active reference to the superblock from within the shrinker. * If that happens we could trigger unregistering the shrinker from within the - * shrinker path and that leads to deadlock on the shrinker_rwsem. Hence we + * shrinker path and that leads to deadlock on the shrinker_mutex. Hence we * take a passive reference to the superblock to avoid this from occurring. */ static unsigned long super_cache_scan(struct shrinker *shrink, diff --git a/mm/shrinker_debug.c b/mm/shrinker_debug.c index 6aa7a7ec69da..b0f6aff372df 100644 --- a/mm/shrinker_debug.c +++ b/mm/shrinker_debug.c @@ -7,7 +7,7 @@ #include /* defined in vmscan.c */ -extern struct rw_semaphore shrinker_rwsem; +extern struct mutex shrinker_mutex; extern struct list_head shrinker_list; extern struct srcu_struct shrinker_srcu; @@ -167,7 +167,7 @@ int shrinker_debugfs_add(struct shrinker *shrinker) char buf[128]; int id; - lockdep_assert_held(&shrinker_rwsem); + lockdep_assert_held(&shrinker_mutex); /* debugfs isn't initialized yet, add debugfs entries later. */ if (!shrinker_debugfs_root) @@ -210,7 +210,7 @@ int shrinker_debugfs_rename(struct shrinker *shrinker, const char *fmt, ...) if (!new) return -ENOMEM; - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); old = shrinker->name; shrinker->name = new; @@ -228,7 +228,7 @@ int shrinker_debugfs_rename(struct shrinker *shrinker, const char *fmt, ...) shrinker->debugfs_entry = entry; } - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); kfree_const(old); @@ -240,7 +240,7 @@ struct dentry *shrinker_debugfs_remove(struct shrinker *shrinker) { struct dentry *entry = shrinker->debugfs_entry; - lockdep_assert_held(&shrinker_rwsem); + lockdep_assert_held(&shrinker_mutex); kfree_const(shrinker->name); shrinker->name = NULL; @@ -265,14 +265,14 @@ static int __init shrinker_debugfs_init(void) shrinker_debugfs_root = dentry; /* Create debugfs entries for shrinkers registered at boot */ - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); list_for_each_entry(shrinker, &shrinker_list, list) if (!shrinker->debugfs_entry) { ret = shrinker_debugfs_add(shrinker); if (ret) break; } - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); return ret; } diff --git a/mm/vmscan.c b/mm/vmscan.c index d1a95d60d127..27ef9946ae8a 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -35,7 +35,7 @@ #include #include #include -#include +#include #include #include #include @@ -202,7 +202,7 @@ static void set_task_reclaim_state(struct task_struct *task, } LIST_HEAD(shrinker_list); -DECLARE_RWSEM(shrinker_rwsem); +DEFINE_MUTEX(shrinker_mutex); DEFINE_SRCU(shrinker_srcu); #ifdef CONFIG_MEMCG @@ -224,7 +224,7 @@ static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *memcg, { return srcu_dereference_check(memcg->nodeinfo[nid]->shrinker_info, &shrinker_srcu, - lockdep_is_held(&shrinker_rwsem)); + lockdep_is_held(&shrinker_mutex)); } static struct shrinker_info *shrinker_info_srcu(struct mem_cgroup *memcg, @@ -308,7 +308,7 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) int nid, size, ret = 0; int map_size, defer_size = 0; - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); map_size = shrinker_map_size(shrinker_nr_max); defer_size = shrinker_defer_size(shrinker_nr_max); size = map_size + defer_size; @@ -324,7 +324,7 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) info->map_nr_max = shrinker_nr_max; rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_info, info); } - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); return ret; } @@ -343,7 +343,7 @@ static int expand_shrinker_info(int new_id) if (!root_mem_cgroup) goto out; - lockdep_assert_held(&shrinker_rwsem); + lockdep_assert_held(&shrinker_mutex); map_size = shrinker_map_size(new_nr_max); defer_size = shrinker_defer_size(new_nr_max); @@ -391,7 +391,7 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) if (mem_cgroup_disabled()) return -ENOSYS; - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); id = idr_alloc(&shrinker_idr, shrinker, 0, 0, GFP_KERNEL); if (id < 0) goto unlock; @@ -405,7 +405,7 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) shrinker->id = id; ret = 0; unlock: - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); return ret; } @@ -415,7 +415,7 @@ static void unregister_memcg_shrinker(struct shrinker *shrinker) BUG_ON(id < 0); - lockdep_assert_held(&shrinker_rwsem); + lockdep_assert_held(&shrinker_mutex); idr_remove(&shrinker_idr, id); } @@ -450,7 +450,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) parent = root_mem_cgroup; /* Prevent from concurrent shrinker_info expand */ - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); for_each_node(nid) { child_info = shrinker_info_protected(memcg, nid); parent_info = shrinker_info_protected(parent, nid); @@ -459,7 +459,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) atomic_long_add(nr, &parent_info->nr_deferred[i]); } } - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); } static bool cgroup_reclaim(struct scan_control *sc) @@ -708,9 +708,9 @@ void free_prealloced_shrinker(struct shrinker *shrinker) shrinker->name = NULL; #endif if (shrinker->flags & SHRINKER_MEMCG_AWARE) { - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); unregister_memcg_shrinker(shrinker); - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); return; } @@ -720,11 +720,11 @@ void free_prealloced_shrinker(struct shrinker *shrinker) void register_shrinker_prepared(struct shrinker *shrinker) { - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); list_add_tail_rcu(&shrinker->list, &shrinker_list); shrinker->flags |= SHRINKER_REGISTERED; shrinker_debugfs_add(shrinker); - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); } static int __register_shrinker(struct shrinker *shrinker) @@ -774,13 +774,13 @@ void unregister_shrinker(struct shrinker *shrinker) if (!(shrinker->flags & SHRINKER_REGISTERED)) return; - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); list_del_rcu(&shrinker->list); shrinker->flags &= ~SHRINKER_REGISTERED; if (shrinker->flags & SHRINKER_MEMCG_AWARE) unregister_memcg_shrinker(shrinker); debugfs_entry = shrinker_debugfs_remove(shrinker); - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); synchronize_srcu(&shrinker_srcu);