From patchwork Fri Apr 9 12:29:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12194041 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D870CC433B4 for ; Fri, 9 Apr 2021 12:32:00 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 61E8361177 for ; Fri, 9 Apr 2021 12:32:00 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 61E8361177 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id DDB8A6B0070; Fri, 9 Apr 2021 08:31:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CEDD56B0071; Fri, 9 Apr 2021 08:31:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B66206B0072; Fri, 9 Apr 2021 08:31:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0233.hostedemail.com [216.40.44.233]) by kanga.kvack.org (Postfix) with ESMTP id 9D50E6B0070 for ; Fri, 9 Apr 2021 08:31:59 -0400 (EDT) Received: from smtpin30.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 62C53B2AE for ; Fri, 9 Apr 2021 12:31:59 +0000 (UTC) X-FDA: 78012765558.30.64CFBBD Received: from mail-pj1-f46.google.com (mail-pj1-f46.google.com [209.85.216.46]) by imf19.hostedemail.com (Postfix) with ESMTP id BC8B990009F2 for ; Fri, 9 Apr 2021 12:31:48 +0000 (UTC) Received: by mail-pj1-f46.google.com with SMTP id k23-20020a17090a5917b02901043e35ad4aso4902302pji.3 for ; Fri, 09 Apr 2021 05:31:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=58kw73zULk1M3QRMYpGesyQhzu0BFPE8Fnems3AjAyI=; b=jSUh5hJWnp8EpV/wkefSJdZoi2UIab+EAWPvsDiKxRQZFhbegQ6L+Dl6eawxYH4xVP Z5VSbvD4uyhDlqJvsvJE7nm/k4W55DfqLKoNSuW9g0pMbgvsj6VyLm54oHXzQYVgaYOE 1eLC/i/3dlP0Ne5mvo6Zr5HbA191dng758K50uL4U8eC04wsiOi6hu06CioZQfggRe1l cTX182sfC625GWoWUSxyVFANLDFjhgIeRuJ8PabTFygDuLtG7KPWnb+YCntxJixUVreS fnBFBvxfh8qKkvW8XSdGg6kg0d4QTUqphO4HAAsi7LIkqyhWlhrUvL9nB9J62VMFJRCT AIQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=58kw73zULk1M3QRMYpGesyQhzu0BFPE8Fnems3AjAyI=; b=MzdJAOs76SVPO5+s6nfLo2+BSL6Syrs8Kxt0v4fm2g8VYKYokz5TBuT9TEzeDt4RIk r8z32YWMVmINEBYatCZFaa7KdJqvgd2u/8c+I7K6c6KABDzUv3LymEjFIF5x4fgbMOwE FZUI39rSRmf/tf4Uxb+T23xx6umWUzavZZwgsUGMJsJsJSSqTjMMDlvFqt9JrP68qhZA Al6kadnYKI62XYQKrcKKxtQaP3XI5ElQ66Jbat91zJR0qq5RUpwgQM+7ycl5DNPm3yzM 4twQJauh/Ehwnu/AnWQuy5OrrTX/8h+EWfkEtPLnf0cWB7qoJMb+wBwdOzjq7WLWiFM2 GYWA== X-Gm-Message-State: AOAM531Y+8VXP4YG21+VQpDRp5UplVWxHkJKr3kxu0CRUrzUVVOf47wh BONf3GEtkla0k62w8AORUMgNAg== X-Google-Smtp-Source: ABdhPJxT5HnFu/JWiq7mOhBolL3h47S0OslJDo9Gs2/ZnJOGN15SbV0U6S25bb3U+N8skATR+PwuPQ== X-Received: by 2002:a17:90a:e016:: with SMTP id u22mr13094679pjy.134.1617971518281; Fri, 09 Apr 2021 05:31:58 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.243]) by smtp.gmail.com with ESMTPSA id t3sm2167785pfg.176.2021.04.09.05.31.53 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 09 Apr 2021 05:31:57 -0700 (PDT) From: Muchun Song To: guro@fb.com, hannes@cmpxchg.org, mhocko@kernel.org, akpm@linux-foundation.org, shakeelb@google.com, vdavydov.dev@gmail.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, bsingharora@gmail.com, shy828301@gmail.com, alex.shi@linux.alibaba.com, Muchun Song Subject: [RFC PATCH v2 01/18] mm: memcontrol: fix page charging in page replacement Date: Fri, 9 Apr 2021 20:29:42 +0800 Message-Id: <20210409122959.82264-2-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210409122959.82264-1-songmuchun@bytedance.com> References: <20210409122959.82264-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: BC8B990009F2 X-Stat-Signature: ee8xe3oy8b4ar9ha1qh155noiya4r7qc Received-SPF: none (bytedance.com>: No applicable sender policy available) receiver=imf19; identity=mailfrom; envelope-from=""; helo=mail-pj1-f46.google.com; client-ip=209.85.216.46 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617971508-639661 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The pages aren't accounted at the root level, so do not charge the page to the root memcg in page replacement. Although we do not display the value (mem_cgroup_usage) so there shouldn't be any actual problem, but there is a WARN_ON_ONCE in the page_counter_cancel(). Who knows if it will trigger? So it is better to fix it. Signed-off-by: Muchun Song Acked-by: Johannes Weiner Reviewed-by: Shakeel Butt --- mm/memcontrol.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 64ada9e650a5..f229de925aa5 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -6806,9 +6806,11 @@ void mem_cgroup_migrate(struct page *oldpage, struct page *newpage) /* Force-charge the new page. The old one will be freed soon */ nr_pages = thp_nr_pages(newpage); - page_counter_charge(&memcg->memory, nr_pages); - if (do_memsw_account()) - page_counter_charge(&memcg->memsw, nr_pages); + if (!mem_cgroup_is_root(memcg)) { + page_counter_charge(&memcg->memory, nr_pages); + if (do_memsw_account()) + page_counter_charge(&memcg->memsw, nr_pages); + } css_get(&memcg->css); commit_charge(newpage, memcg); From patchwork Fri Apr 9 12:29:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12194043 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BE7AEC433B4 for ; Fri, 9 Apr 2021 12:32:06 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 22DC961165 for ; Fri, 9 Apr 2021 12:32:06 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 22DC961165 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B68AF6B0072; Fri, 9 Apr 2021 08:32:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B3E086B0073; Fri, 9 Apr 2021 08:32:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A06696B0075; Fri, 9 Apr 2021 08:32:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0249.hostedemail.com [216.40.44.249]) by kanga.kvack.org (Postfix) with ESMTP id 84FDE6B0072 for ; Fri, 9 Apr 2021 08:32:05 -0400 (EDT) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 299488248047 for ; Fri, 9 Apr 2021 12:32:05 +0000 (UTC) X-FDA: 78012765810.21.EA68DC9 Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) by imf07.hostedemail.com (Postfix) with ESMTP id 8A5F6A00014A for ; Fri, 9 Apr 2021 12:32:04 +0000 (UTC) Received: by mail-pl1-f172.google.com with SMTP id w8so344703plg.9 for ; Fri, 09 Apr 2021 05:32:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=i4a9s9eaaqji3vjC/5YOcby1VxJlho+mBaQ9dB0G314=; b=hxZly7/E5IKIeIk6u/EZCEyIoZ/xMYLvSN8rF8942v1a/+cwKmYifDXgBLVJKTj1Yz bpm5VsIWOcAU6I3tPtaofuBvT1HgIAsy26FDVfyheNOij9bcP4XLantQoaBp1IFNolOS zC92u44wbSgqqNgc3ljNUsqbqJVBA1NlDiwUZeaouYqD/Aoj4a24QBBz5o0/6o4MSXu/ 8ojwfYriBdUqDqEq3TbLXQQ3yIQnPoPXD/YYXaycOoMmVNgvC9hfAHsKB6aNYHo00rGo F2urOomjz+EwEnaZJgiTJLdM1ptrfPAqKst4fUI4fyx2KV+gnkUpaRaEzG0UtiHKGzWG 3qsA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=i4a9s9eaaqji3vjC/5YOcby1VxJlho+mBaQ9dB0G314=; b=JGlIze3GzLM7K0NW3bEnlJakaNqQgMOTTWK/s9iY+NQNcpcw2429nHD0HrwLRsXXe1 CzWko3xprVkp1Xu2WsP+kGnxTET46H4hyB4z1P7ONapcr2PNUw6qMe5suXmU+SFVDtqk xu2P3tAATlsJZI1AwmSRstLUZHv5eWu12DdyKKOnU7/y1BjZKsJjVEJz2LIWCXR3kY26 QyMNIBBvNyUSgJI+lqZwcMp6HsFiMmq69othcDAgRLO7ys+8Eak3FY31cKwFAU03WjZk DedyLbvAyw0ftqgIEKu7Sxe3XpWZNnqWlGJ77GqjESbOzNBK7kDM95afegh8P1EprNQP TXSw== X-Gm-Message-State: AOAM533ML07WfWlF3QDZiSi3l+kl3OqBFlDJ7R/vecjURdFfa6Y6+tDd mNaf3rNCahZ9SOZ/pkRxe8SRyg== X-Google-Smtp-Source: ABdhPJxUTMey7XOyvG1I+G4Ben3Ww5Kwa5QoiZhplW2u/vumLA6YeXi0uVU8sX4JsFQHGC5wctmeOw== X-Received: by 2002:a17:90a:4498:: with SMTP id t24mr13735310pjg.78.1617971523836; Fri, 09 Apr 2021 05:32:03 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.243]) by smtp.gmail.com with ESMTPSA id t3sm2167785pfg.176.2021.04.09.05.31.58 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 09 Apr 2021 05:32:03 -0700 (PDT) From: Muchun Song To: guro@fb.com, hannes@cmpxchg.org, mhocko@kernel.org, akpm@linux-foundation.org, shakeelb@google.com, vdavydov.dev@gmail.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, bsingharora@gmail.com, shy828301@gmail.com, alex.shi@linux.alibaba.com, Muchun Song Subject: [RFC PATCH v2 02/18] mm: memcontrol: bail out early when !mm in get_mem_cgroup_from_mm Date: Fri, 9 Apr 2021 20:29:43 +0800 Message-Id: <20210409122959.82264-3-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210409122959.82264-1-songmuchun@bytedance.com> References: <20210409122959.82264-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 8A5F6A00014A X-Stat-Signature: 5p17a8zedspn1do5moemz1yaf4fyozdr X-Rspamd-Server: rspam02 Received-SPF: none (bytedance.com>: No applicable sender policy available) receiver=imf07; identity=mailfrom; envelope-from=""; helo=mail-pl1-f172.google.com; client-ip=209.85.214.172 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617971524-65392 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When mm is NULL, we do not need to hold rcu lock and call css_tryget for the root memcg. And we also do not need to check !mm in every loop of while. So bail out early when !mm. Signed-off-by: Muchun Song Acked-by: Johannes Weiner Reviewed-by: Shakeel Butt --- mm/memcontrol.c | 21 ++++++++++----------- 1 file changed, 10 insertions(+), 11 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index f229de925aa5..9cbfff59b171 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -901,20 +901,19 @@ struct mem_cgroup *get_mem_cgroup_from_mm(struct mm_struct *mm) if (mem_cgroup_disabled()) return NULL; + /* + * Page cache insertions can happen without an + * actual mm context, e.g. during disk probing + * on boot, loopback IO, acct() writes etc. + */ + if (unlikely(!mm)) + return root_mem_cgroup; + rcu_read_lock(); do { - /* - * Page cache insertions can happen without an - * actual mm context, e.g. during disk probing - * on boot, loopback IO, acct() writes etc. - */ - if (unlikely(!mm)) + memcg = mem_cgroup_from_task(rcu_dereference(mm->owner)); + if (unlikely(!memcg)) memcg = root_mem_cgroup; - else { - memcg = mem_cgroup_from_task(rcu_dereference(mm->owner)); - if (unlikely(!memcg)) - memcg = root_mem_cgroup; - } } while (!css_tryget(&memcg->css)); rcu_read_unlock(); return memcg; From patchwork Fri Apr 9 12:29:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12194045 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 93046C433ED for ; Fri, 9 Apr 2021 12:32:23 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1E99261181 for ; Fri, 9 Apr 2021 12:32:23 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1E99261181 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B19416B0075; Fri, 9 Apr 2021 08:32:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AF03D6B0078; Fri, 9 Apr 2021 08:32:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 990F56B007B; Fri, 9 Apr 2021 08:32:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0059.hostedemail.com [216.40.44.59]) by kanga.kvack.org (Postfix) with ESMTP id 802AB6B0075 for ; Fri, 9 Apr 2021 08:32:22 -0400 (EDT) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 3A3878248047 for ; Fri, 9 Apr 2021 12:32:22 +0000 (UTC) X-FDA: 78012766524.11.4E46E38 Received: from mail-pg1-f174.google.com (mail-pg1-f174.google.com [209.85.215.174]) by imf16.hostedemail.com (Postfix) with ESMTP id 34B6F80192D5 for ; Fri, 9 Apr 2021 12:32:21 +0000 (UTC) Received: by mail-pg1-f174.google.com with SMTP id b17so3785967pgh.7 for ; Fri, 09 Apr 2021 05:32:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=KboWzAA0Vo1Ky5mj5yxHhRMCFO2+9SMKFGuBaMmV7vg=; b=Fn+4FB9vLM9YrQj9Me1zs6sH/n4f8nrDEsuMRHxSCrQ4oIBF0ErYthEs5c5YPYxpgF OKP9kF0CU5WEWtQF55Lfa7A2AiYr2FAnylG5iCg0FVDLvSyBRJiAdrkXjOi7qKwZLiCe aqCJxtuDhVYDe3UoeTvHnA1CCFNNyp3Xv8/F4ZgVSJUM0t5ycK0XqOUJdFhkvZFex811 /HjG9IBdMBp04eWC3BbTJy4MY3Nx1j6QPO8EcC289R+SYh5YK7rPZXJzskFqDfb10MrF jfIAvegBXUAdMiiQww1pVjvG5AeexX/BoWkTeZ6HZkwU2XCmTjyLrtj5G15IX1t9uKdg qFHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=KboWzAA0Vo1Ky5mj5yxHhRMCFO2+9SMKFGuBaMmV7vg=; b=jgDlVNuDtONMjWlndnfdAQq45qrM7Ty6bsP2e6U1lIYkR1TvoqCyNgZX7es8vpVTQd gWaICiWvkL/qw1Q+aqkaFspOr7NLDAQGrgAuAvd+Na8lknIbHAh7ZDTvq8IiU6VxqlXC 0nKI7TpRuXom/2q7TLKe0YPmKMOJiHUf/+dmluwxb0t8qsVvrzHXT5fSHTOFnARwqho3 6IrFMiXuc+ok/KnBAMzd93Euw9bvzRFiFDgNyifABymetlHhJs9Y1OTkQ5dhNoEGWz79 nSJi/HNfh/lSTspvNKoDr2+yZkamCzXlVp2a1Hugr3K2Yv9HbvHpyAedfVgaOUcr9N1y SreQ== X-Gm-Message-State: AOAM533zJbCNSbNCFiHJnEn9MRUpLbVgC9jFWeg4OhL72GftvsYcvIyo XVHjxMaRzpIQH6Z7WMSf82ob3w== X-Google-Smtp-Source: ABdhPJy4EUwCPJplsg2FCY/PfNvR8+q2FxSz6X7rdJ98QKB0KOwEImFa1X8fqxQvMxKBXBCgykJIXQ== X-Received: by 2002:a63:1d1:: with SMTP id 200mr9444674pgb.375.1617971540915; Fri, 09 Apr 2021 05:32:20 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.243]) by smtp.gmail.com with ESMTPSA id t3sm2167785pfg.176.2021.04.09.05.32.15 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 09 Apr 2021 05:32:20 -0700 (PDT) From: Muchun Song To: guro@fb.com, hannes@cmpxchg.org, mhocko@kernel.org, akpm@linux-foundation.org, shakeelb@google.com, vdavydov.dev@gmail.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, bsingharora@gmail.com, shy828301@gmail.com, alex.shi@linux.alibaba.com, Muchun Song Subject: [RFC PATCH v2 05/18] mm: memcontrol: simplify the logic of objcg pinning memcg Date: Fri, 9 Apr 2021 20:29:46 +0800 Message-Id: <20210409122959.82264-6-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210409122959.82264-1-songmuchun@bytedance.com> References: <20210409122959.82264-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 34B6F80192D5 X-Stat-Signature: 14ucsxreiuu51ap99oq4zc71d59jaojc X-Rspamd-Server: rspam02 Received-SPF: none (bytedance.com>: No applicable sender policy available) receiver=imf16; identity=mailfrom; envelope-from=""; helo=mail-pg1-f174.google.com; client-ip=209.85.215.174 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617971541-61339 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The obj_cgroup_release() and memcg_reparent_objcgs() are serialized by the css_set_lock. We do not need to care about objcg->memcg being released in the process of obj_cgroup_release(). So there is no need to pin memcg before releasing objcg. Remove those pinning logic to simplfy the code. There are only two places that modifies the objcg->memcg. One is the initialization to objcg->memcg in the memcg_online_kmem(), another is objcgs reparenting in the memcg_reparent_objcgs(). It is also impossible for the two to run in parallel. So xchg() is unnecessary and it is enough to use WRITE_ONCE(). Signed-off-by: Muchun Song Acked-by: Johannes Weiner --- mm/memcontrol.c | 20 +++++++------------- 1 file changed, 7 insertions(+), 13 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 1f807448233e..90c1ac58c64c 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -261,7 +261,6 @@ static void obj_cgroup_uncharge_pages(struct obj_cgroup *objcg, static void obj_cgroup_release(struct percpu_ref *ref) { struct obj_cgroup *objcg = container_of(ref, struct obj_cgroup, refcnt); - struct mem_cgroup *memcg; unsigned int nr_bytes; unsigned int nr_pages; unsigned long flags; @@ -291,11 +290,9 @@ static void obj_cgroup_release(struct percpu_ref *ref) nr_pages = nr_bytes >> PAGE_SHIFT; spin_lock_irqsave(&css_set_lock, flags); - memcg = obj_cgroup_memcg(objcg); if (nr_pages) obj_cgroup_uncharge_pages(objcg, nr_pages); list_del(&objcg->list); - mem_cgroup_put(memcg); spin_unlock_irqrestore(&css_set_lock, flags); percpu_ref_exit(ref); @@ -330,17 +327,14 @@ static void memcg_reparent_objcgs(struct mem_cgroup *memcg, spin_lock_irq(&css_set_lock); - /* Move active objcg to the parent's list */ - xchg(&objcg->memcg, parent); - css_get(&parent->css); - list_add(&objcg->list, &parent->objcg_list); + /* 1) Ready to reparent active objcg. */ + list_add(&objcg->list, &memcg->objcg_list); - /* Move already reparented objcgs to the parent's list */ - list_for_each_entry(iter, &memcg->objcg_list, list) { - css_get(&parent->css); - xchg(&iter->memcg, parent); - css_put(&memcg->css); - } + /* 2) Reparent active objcg and already reparented objcgs to parent. */ + list_for_each_entry(iter, &memcg->objcg_list, list) + WRITE_ONCE(iter->memcg, parent); + + /* 3) Move already reparented objcgs to the parent's list */ list_splice(&memcg->objcg_list, &parent->objcg_list); spin_unlock_irq(&css_set_lock); From patchwork Fri Apr 9 12:29:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12194047 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56543C433ED for ; Fri, 9 Apr 2021 12:32:36 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id EED7761165 for ; Fri, 9 Apr 2021 12:32:35 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EED7761165 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 8A7D76B007B; Fri, 9 Apr 2021 08:32:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 87E276B007D; Fri, 9 Apr 2021 08:32:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7465E6B007E; Fri, 9 Apr 2021 08:32:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0007.hostedemail.com [216.40.44.7]) by kanga.kvack.org (Postfix) with ESMTP id 59E676B007B for ; Fri, 9 Apr 2021 08:32:35 -0400 (EDT) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 150E88248047 for ; Fri, 9 Apr 2021 12:32:35 +0000 (UTC) X-FDA: 78012767070.25.3317D1F Received: from mail-pj1-f52.google.com (mail-pj1-f52.google.com [209.85.216.52]) by imf25.hostedemail.com (Postfix) with ESMTP id EF6C46000108 for ; Fri, 9 Apr 2021 12:32:32 +0000 (UTC) Received: by mail-pj1-f52.google.com with SMTP id ep1-20020a17090ae641b029014d48811e37so3110615pjb.4 for ; Fri, 09 Apr 2021 05:32:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=zEHZ5qG2Y9NIbhhZ3B17EZ5gBaH7ne5FjVmUdr7WNhw=; b=HhI9y7ykMy8qmjJd8T6jqQ4HxvvjCcyyeW1xVnQfYtZzlES1KSGQ85Q+DfM9GA7CQi SSIlTfC85Xfx/2gq/5GAnU3p9Foxs6ZApRZK2Ijcw2U8vzpqhy8cjpb9EaZgJEJf2Prs eX/U+LX5IkGaJlbTsgcf5zxP26U7l2ydVfxd865qVJYe7cdjeRVszFRwhdq/gNIwuJgm pCBHrk6ItplIEzUe8TbAm6EW+anIgZWthNuQ6hpKDJaf+0kyBN/LHB268iJFmdRMG1nE Dfi9W6U6lCMtkd5hUkMVeLoQUN1Tnl7ze1uAPrmGFcMe5HzgDSYaSCvoyA91sxWYLpid 3lwg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=zEHZ5qG2Y9NIbhhZ3B17EZ5gBaH7ne5FjVmUdr7WNhw=; b=EudclIuWuwVGMGWMPPhdhhwhJLhyVbMSWre+G/t1dTfjdE+y7pqGLKqfPXj9dDGnRB Z2Mb5LNYounlf9BQxuIysvWJWUUJcizY9efVnm57Rz7T/ooxlbpxwkXYWfqz23Weg8h7 6DcdtfkJoybPB51JTV+Or0Gd2B8XLvWiIYqtv7XkDZs9Wlmh5I0hHK2UF4pJsK7VhhpJ qeZEPFxwcTmgZlDI18g9zheMsYH3/tL0bBKkGA9HhRrVJWwts1/3TcGwymJBcuIyApni YaKjqQvp/xVnPFIAa+WZHTUrk7NFayXGT/ec5zF10JH3bgvaqHM4wbXjkSEChHl/Dj8W fSmQ== X-Gm-Message-State: AOAM531iUwqjiAZMSm2fPdxQbJgwjB3BbOCT1KtKZSWgAtPArViop8rW fpPydPESH788RIVOXIelC9eFcZIzeuGPGEfk X-Google-Smtp-Source: ABdhPJy7MgN+wnbbgO/53f8oazsZWPqfnfHqoxI3nVximXuOLtS2UPc/V+JY5nOP8tMuCP3A+H3F5g== X-Received: by 2002:a17:90a:156:: with SMTP id z22mr14391859pje.181.1617971553903; Fri, 09 Apr 2021 05:32:33 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.243]) by smtp.gmail.com with ESMTPSA id t3sm2167785pfg.176.2021.04.09.05.32.26 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 09 Apr 2021 05:32:33 -0700 (PDT) From: Muchun Song To: guro@fb.com, hannes@cmpxchg.org, mhocko@kernel.org, akpm@linux-foundation.org, shakeelb@google.com, vdavydov.dev@gmail.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, bsingharora@gmail.com, shy828301@gmail.com, alex.shi@linux.alibaba.com, Muchun Song Subject: [RFC PATCH v2 07/18] mm: memcontrol: introduce compact_lock_page_lruvec_irqsave Date: Fri, 9 Apr 2021 20:29:48 +0800 Message-Id: <20210409122959.82264-8-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210409122959.82264-1-songmuchun@bytedance.com> References: <20210409122959.82264-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: EF6C46000108 X-Stat-Signature: d3o7jj4q1i7ah76bj6w93be99b61jbe1 X-Rspamd-Server: rspam02 Received-SPF: none (bytedance.com>: No applicable sender policy available) receiver=imf25; identity=mailfrom; envelope-from=""; helo=mail-pj1-f52.google.com; client-ip=209.85.216.52 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617971552-863397 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If we reuse the objcg APIs to charge LRU pages, the page_memcg() can be changed when the LRU pages reparented. In this case, we need to acquire the new lruvec lock. lruvec = mem_cgroup_page_lruvec(page); // The page is reparented. compact_lock_irqsave(&lruvec->lru_lock, &flags, cc); // Acquired the wrong lruvec lock and need to retry. But compact_lock_irqsave() only take lruvec lock as the parameter, we cannot aware this change. If it can take the page as parameter to acquire the lruvec lock. When the page memcg is changed, we can use the page_memcg() detect whether we need to reacquire the new lruvec lock. So compact_lock_irqsave() is not suitable for us. Similar to lock_page_lruvec_irqsave(), introduce compact_lock_page_lruvec_irqsave() to acquire the lruvec lock in the compaction routine. Signed-off-by: Muchun Song --- mm/compaction.c | 29 +++++++++++++++++++++++++---- 1 file changed, 25 insertions(+), 4 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index e7da342003dd..c9efe3542b0a 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -511,6 +511,29 @@ static bool compact_lock_irqsave(spinlock_t *lock, unsigned long *flags, return true; } +static struct lruvec * +compact_lock_page_lruvec_irqsave(struct page *page, unsigned long *flags, + struct compact_control *cc) +{ + struct lruvec *lruvec; + + lruvec = mem_cgroup_page_lruvec(page); + + /* Track if the lock is contended in async mode */ + if (cc->mode == MIGRATE_ASYNC && !cc->contended) { + if (spin_trylock_irqsave(&lruvec->lru_lock, *flags)) + goto out; + + cc->contended = true; + } + + spin_lock_irqsave(&lruvec->lru_lock, *flags); +out: + lruvec_memcg_debug(lruvec, page); + + return lruvec; +} + /* * Compaction requires the taking of some coarse locks that are potentially * very heavily contended. The lock should be periodically unlocked to avoid @@ -1040,10 +1063,8 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, if (locked) unlock_page_lruvec_irqrestore(locked, flags); - compact_lock_irqsave(&lruvec->lru_lock, &flags, cc); - locked = lruvec; - - lruvec_memcg_debug(lruvec, page); + locked = compact_lock_page_lruvec_irqsave(page, &flags, cc); + lruvec = locked; /* Try get exclusive access under lock */ if (!skip_updated) { From patchwork Fri Apr 9 12:29:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12194049 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3FE97C433B4 for ; Fri, 9 Apr 2021 12:32:47 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E03A061178 for ; Fri, 9 Apr 2021 12:32:46 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E03A061178 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 7BDC56B007E; Fri, 9 Apr 2021 08:32:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 794076B0080; Fri, 9 Apr 2021 08:32:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 65BEE6B0081; Fri, 9 Apr 2021 08:32:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0118.hostedemail.com [216.40.44.118]) by kanga.kvack.org (Postfix) with ESMTP id 4DDCB6B007E for ; Fri, 9 Apr 2021 08:32:46 -0400 (EDT) Received: from smtpin34.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 0AD0E8248047 for ; Fri, 9 Apr 2021 12:32:46 +0000 (UTC) X-FDA: 78012767532.34.0ECCB05 Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) by imf30.hostedemail.com (Postfix) with ESMTP id 9D1BCE0001AC for ; Fri, 9 Apr 2021 12:32:38 +0000 (UTC) Received: by mail-pl1-f178.google.com with SMTP id h20so2686473plr.4 for ; Fri, 09 Apr 2021 05:32:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=AGnIcRUphb33Z52wrBzQRy2lP1/tCopQL4vrXFnFrmI=; b=S0AxdzbO4aXq30Vi35yMmb1Li/r3E9lW60nwWLlDMZRGKCvNrSuk9f6rMufTdhOyqF uGM5dcZ1TcR1fLjFFyIpJRKkjjDhlTNaqc5Ies/YjdlKakn02dWQ1V9zj1k2tXWloykp KcViVWqi0wDeQZoFgkHypenGAb2l/C8qnB/RTFfInSkCaTPj860ShomNWih9FAhRvtwR dUQj3pch7N74sEvRDw5muZ+tU8TCL9Ks18Tp/EOpqWvNbEPBoKAHXwT5/HHUz3MIeEvg UKnRw93yBCLGcNZUgOQr80Hfj3NdocAyk/VRGGBHm0O+YugvFmpwgS/3TWpCtBReAJON gv4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=AGnIcRUphb33Z52wrBzQRy2lP1/tCopQL4vrXFnFrmI=; b=rRpaEMVQC/GRpjIZOWyyJhVO1bUJW32TcQ4y6jz3RcLqLWrs72imTEI/iuC0dD0BUC Rg9igpcomGTsLgIuhY5oEYV5i4OF44LUN7+dD9KEkQDX2niWBCvsCgQcJ9jzatTiSuLy Ig8mxpwuu9o+a3wSfHrDCwULCDs5yMUIo7FitJ1RY0SXvGwDQN0IkwkpiY0FDnI2D6sX IMF3KdZNuqOj7IRJo1Nnlbl1gByQu0A5DqT8S3NQw4HuJ3MutyTWFTqhmoRBW24qEMtb FGNTCZTxtjIhJpUX9R95Bm5cEaYeYHOegwaFYkxf9IOXuVlTb0g4pPLv3pEHxmqMO2Yl AAmA== X-Gm-Message-State: AOAM533oIJILENsIBjvcnx+fMHBIz1Jxv5ENQ1y2MoAdo7fM25WDiYQq 9xsZKuTjcxX8/deiFMRwj5s4YcfN8N3BOIQh X-Google-Smtp-Source: ABdhPJwn8UgcJ7e8pfW/StQ/oe2JyK5NB/2gozqFYTM0tv5Cti79xM30O0H5IkxlZ8bpV8sY/ILxdQ== X-Received: by 2002:a17:90a:de89:: with SMTP id n9mr13461586pjv.180.1617971564855; Fri, 09 Apr 2021 05:32:44 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.243]) by smtp.gmail.com with ESMTPSA id t3sm2167785pfg.176.2021.04.09.05.32.39 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 09 Apr 2021 05:32:44 -0700 (PDT) From: Muchun Song To: guro@fb.com, hannes@cmpxchg.org, mhocko@kernel.org, akpm@linux-foundation.org, shakeelb@google.com, vdavydov.dev@gmail.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, bsingharora@gmail.com, shy828301@gmail.com, alex.shi@linux.alibaba.com, Muchun Song Subject: [RFC PATCH v2 09/18] mm: vmscan: remove noinline_for_stack Date: Fri, 9 Apr 2021 20:29:50 +0800 Message-Id: <20210409122959.82264-10-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210409122959.82264-1-songmuchun@bytedance.com> References: <20210409122959.82264-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 9D1BCE0001AC X-Stat-Signature: ce1sjd93rq7f1rocjrm6zqnjfgmmf1dz Received-SPF: none (bytedance.com>: No applicable sender policy available) receiver=imf30; identity=mailfrom; envelope-from=""; helo=mail-pl1-f178.google.com; client-ip=209.85.214.178 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617971558-111941 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The noinline_for_stack is introduced by commit 666356297ec4 ("vmscan: set up pagevec as late as possible in shrink_inactive_list()"), its purpose is to delay the allocation of pagevec as late as possible to save stack memory. But the commit 2bcf88796381 ("mm: take pagevecs off reclaim stack") replace pagevecs by lists of pages_to_free. So we do not need noinline_for_stack, just remove it (let the compiler decide whether to inline). Signed-off-by: Muchun Song Acked-by: Johannes Weiner Acked-by: Roman Gushchin Reviewed-by: Shakeel Butt --- mm/vmscan.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 64bf07cc20f2..e40b21298d77 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2015,8 +2015,8 @@ static int too_many_isolated(struct pglist_data *pgdat, int file, * * Returns the number of pages moved to the given lruvec. */ -static unsigned noinline_for_stack move_pages_to_lru(struct lruvec *lruvec, - struct list_head *list) +static unsigned int move_pages_to_lru(struct lruvec *lruvec, + struct list_head *list) { int nr_pages, nr_moved = 0; LIST_HEAD(pages_to_free); @@ -2096,7 +2096,7 @@ static int current_may_throttle(void) * shrink_inactive_list() is a helper for shrink_node(). It returns the number * of reclaimed pages */ -static noinline_for_stack unsigned long +static unsigned long shrink_inactive_list(unsigned long nr_to_scan, struct lruvec *lruvec, struct scan_control *sc, enum lru_list lru) { From patchwork Fri Apr 9 12:29:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12194051 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9484EC433B4 for ; Fri, 9 Apr 2021 12:32:52 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 3A7E161177 for ; Fri, 9 Apr 2021 12:32:52 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3A7E161177 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id CC6EB6B0081; Fri, 9 Apr 2021 08:32:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C9CB06B0082; Fri, 9 Apr 2021 08:32:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B3DF86B0083; Fri, 9 Apr 2021 08:32:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0031.hostedemail.com [216.40.44.31]) by kanga.kvack.org (Postfix) with ESMTP id 997A96B0081 for ; Fri, 9 Apr 2021 08:32:51 -0400 (EDT) Received: from smtpin31.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 598C66D9A for ; Fri, 9 Apr 2021 12:32:51 +0000 (UTC) X-FDA: 78012767742.31.2EEC2C7 Received: from mail-pg1-f179.google.com (mail-pg1-f179.google.com [209.85.215.179]) by imf19.hostedemail.com (Postfix) with ESMTP id B6D6490009ED for ; Fri, 9 Apr 2021 12:32:40 +0000 (UTC) Received: by mail-pg1-f179.google.com with SMTP id w10so3795166pgh.5 for ; Fri, 09 Apr 2021 05:32:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=1fMqyM142M/rLjd886U1IMp5j4aF6dHNuuaPr8ObPdI=; b=cQRyKiiKv4re6AmVdBzMfeckX6oHDJ8D7744bldCAKrRVgwddNhkYd9AeIdoDh06vy YrxFGz36GyHjdH740eUq03OlBfnWXq7wXGu0nWVNwpoFZc/DPH5v105jBfVtezgeA76i 2k6wx6pYY6tgLQwfzD3VUuyf9WRqjqtrjUhQhBfzZCX2XlUtbqD5SClktTezO2DguX5i FWgrLhA1CBlemaJk3Cu28VUfSvgfIezAuVUFwefwSXmLniEUhiaJ1DGYXmdnyjK3i0OM tqRXQ+tlp3fuzIllEyJ3srl9PO+cdXBS29bsYNiG0+uELvV7N+mfU6jTHVB5XiKeLTO+ H33w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=1fMqyM142M/rLjd886U1IMp5j4aF6dHNuuaPr8ObPdI=; b=YMB0Z9c7AOcp6OoLfSU8hhydYX77M26Y4rABp3mwrT0ZgsulHv8zIGoC2irfFHYJZI EPgZTu/x05LBZ11Scps+HnDvU/dKeyU7l7eIGB82i1JjZpn3b8B5bnWg4rRggD2W/lwS 7hHtEO4UkYvuuthjtxYKFd7WjYsKOR2axIloRftxcAKokqWDxGftIUk1seVkXahbQlcP m011RuOtn2hKIWXqdrHvRXdRGOP+wJZoYvJE1uQewN1F61r/X0yM2akLidk9R2j8inxO rXATQFT1oyhrHeZ/D7l/EgEmmU60e2K5meHkpoTP2NBopM4YJhGVSA6/5BCFka3rvbll HtuQ== X-Gm-Message-State: AOAM532K+dcHSP9RfdlKdThxOng0kNxlGPR1vaaDhDp3aSqAMwJJrIQy eq7TZEvWdnSO0MX8VW+jGc/ZGQ== X-Google-Smtp-Source: ABdhPJwJ8GhlVby5jiUHQpT1MsOC2gerlQ1Uo1BLndKNTDvWHnIEB7wkzawABrxZOijbSp+M55IsCQ== X-Received: by 2002:a62:7ccb:0:b029:247:2b6b:f42 with SMTP id x194-20020a627ccb0000b02902472b6b0f42mr2383323pfc.68.1617971570163; Fri, 09 Apr 2021 05:32:50 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.243]) by smtp.gmail.com with ESMTPSA id t3sm2167785pfg.176.2021.04.09.05.32.45 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 09 Apr 2021 05:32:49 -0700 (PDT) From: Muchun Song To: guro@fb.com, hannes@cmpxchg.org, mhocko@kernel.org, akpm@linux-foundation.org, shakeelb@google.com, vdavydov.dev@gmail.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, bsingharora@gmail.com, shy828301@gmail.com, alex.shi@linux.alibaba.com, Muchun Song Subject: [RFC PATCH v2 10/18] mm: vmscan: rework move_pages_to_lru() Date: Fri, 9 Apr 2021 20:29:51 +0800 Message-Id: <20210409122959.82264-11-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210409122959.82264-1-songmuchun@bytedance.com> References: <20210409122959.82264-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: B6D6490009ED X-Stat-Signature: b86sjf8mss9c3m84aj4yndyp1g4pfie8 Received-SPF: none (bytedance.com>: No applicable sender policy available) receiver=imf19; identity=mailfrom; envelope-from=""; helo=mail-pg1-f179.google.com; client-ip=209.85.215.179 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617971560-793502 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In the later patch, we will reparent the LRU pages. The pages which will move to appropriate LRU list can be reparented during the process of the move_pages_to_lru(). So holding a lruvec lock by the caller is wrong, we should use the more general interface of relock_page_lruvec_irq() to acquire the correct lruvec lock. Signed-off-by: Muchun Song --- mm/vmscan.c | 46 +++++++++++++++++++++++----------------------- 1 file changed, 23 insertions(+), 23 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index e40b21298d77..4431007825ad 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2013,23 +2013,27 @@ static int too_many_isolated(struct pglist_data *pgdat, int file, * move_pages_to_lru() moves pages from private @list to appropriate LRU list. * On return, @list is reused as a list of pages to be freed by the caller. * - * Returns the number of pages moved to the given lruvec. + * Returns the number of pages moved to the appropriate LRU list. + * + * Note: The caller must not hold any lruvec lock. */ -static unsigned int move_pages_to_lru(struct lruvec *lruvec, - struct list_head *list) +static unsigned int move_pages_to_lru(struct list_head *list) { - int nr_pages, nr_moved = 0; + int nr_moved = 0; + struct lruvec *lruvec = NULL; LIST_HEAD(pages_to_free); - struct page *page; while (!list_empty(list)) { - page = lru_to_page(list); + int nr_pages; + struct page *page = lru_to_page(list); + + lruvec = relock_page_lruvec_irq(page, lruvec); VM_BUG_ON_PAGE(PageLRU(page), page); list_del(&page->lru); if (unlikely(!page_evictable(page))) { - spin_unlock_irq(&lruvec->lru_lock); + unlock_page_lruvec_irq(lruvec); putback_lru_page(page); - spin_lock_irq(&lruvec->lru_lock); + lruvec = NULL; continue; } @@ -2050,19 +2054,15 @@ static unsigned int move_pages_to_lru(struct lruvec *lruvec, __clear_page_lru_flags(page); if (unlikely(PageCompound(page))) { - spin_unlock_irq(&lruvec->lru_lock); + unlock_page_lruvec_irq(lruvec); destroy_compound_page(page); - spin_lock_irq(&lruvec->lru_lock); + lruvec = NULL; } else list_add(&page->lru, &pages_to_free); continue; } - /* - * All pages were isolated from the same lruvec (and isolation - * inhibits memcg migration). - */ VM_BUG_ON_PAGE(!lruvec_holds_page_lru_lock(page, lruvec), page); add_page_to_lru_list(page, lruvec); nr_pages = thp_nr_pages(page); @@ -2071,6 +2071,8 @@ static unsigned int move_pages_to_lru(struct lruvec *lruvec, workingset_age_nonresident(lruvec, nr_pages); } + if (lruvec) + unlock_page_lruvec_irq(lruvec); /* * To save our caller's stack, now use input list for pages to free. */ @@ -2144,16 +2146,16 @@ shrink_inactive_list(unsigned long nr_to_scan, struct lruvec *lruvec, nr_reclaimed = shrink_page_list(&page_list, pgdat, sc, &stat, false); - spin_lock_irq(&lruvec->lru_lock); - move_pages_to_lru(lruvec, &page_list); + move_pages_to_lru(&page_list); + local_irq_disable(); __mod_node_page_state(pgdat, NR_ISOLATED_ANON + file, -nr_taken); item = current_is_kswapd() ? PGSTEAL_KSWAPD : PGSTEAL_DIRECT; if (!cgroup_reclaim(sc)) __count_vm_events(item, nr_reclaimed); __count_memcg_events(lruvec_memcg(lruvec), item, nr_reclaimed); __count_vm_events(PGSTEAL_ANON + file, nr_reclaimed); - spin_unlock_irq(&lruvec->lru_lock); + local_irq_enable(); lru_note_cost(lruvec, file, stat.nr_pageout); mem_cgroup_uncharge_list(&page_list); @@ -2280,18 +2282,16 @@ static void shrink_active_list(unsigned long nr_to_scan, /* * Move pages back to the lru list. */ - spin_lock_irq(&lruvec->lru_lock); - - nr_activate = move_pages_to_lru(lruvec, &l_active); - nr_deactivate = move_pages_to_lru(lruvec, &l_inactive); + nr_activate = move_pages_to_lru(&l_active); + nr_deactivate = move_pages_to_lru(&l_inactive); /* Keep all free pages in l_active list */ list_splice(&l_inactive, &l_active); + local_irq_disable(); __count_vm_events(PGDEACTIVATE, nr_deactivate); __count_memcg_events(lruvec_memcg(lruvec), PGDEACTIVATE, nr_deactivate); - __mod_node_page_state(pgdat, NR_ISOLATED_ANON + file, -nr_taken); - spin_unlock_irq(&lruvec->lru_lock); + local_irq_enable(); mem_cgroup_uncharge_list(&l_active); free_unref_page_list(&l_active); From patchwork Fri Apr 9 12:29:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12194053 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B1429C433B4 for ; Fri, 9 Apr 2021 12:33:03 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 59769610A8 for ; Fri, 9 Apr 2021 12:33:03 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 59769610A8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id DB0796B006E; Fri, 9 Apr 2021 08:33:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D87746B0083; Fri, 9 Apr 2021 08:33:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C75D66B0085; Fri, 9 Apr 2021 08:33:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0007.hostedemail.com [216.40.44.7]) by kanga.kvack.org (Postfix) with ESMTP id AD2AE6B006E for ; Fri, 9 Apr 2021 08:33:02 -0400 (EDT) Received: from smtpin36.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 6A5D26D9A for ; Fri, 9 Apr 2021 12:33:02 +0000 (UTC) X-FDA: 78012768204.36.2D3FF5D Received: from mail-pf1-f178.google.com (mail-pf1-f178.google.com [209.85.210.178]) by imf16.hostedemail.com (Postfix) with ESMTP id 5535380192DA for ; Fri, 9 Apr 2021 12:33:01 +0000 (UTC) Received: by mail-pf1-f178.google.com with SMTP id a85so3704729pfa.0 for ; Fri, 09 Apr 2021 05:33:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=EoY+xVUIMEYkMAOym+tsxa0CiEBJagAGYOcFqP2KHJM=; b=DsZMvhHWm4mrMrUgLxwe8rmKzWWGxf5n4KDgWVqWnWqFoXsMw/R6RWat4E5ynjciPk NX5/37x4XChb5bZbcKFnhd74XnMGTkmeesqaXJ+BjFZb1cs8nHlEo3NLugIlnUtS6ZqN RdbYWTmfPloA5h6gYb7ly2Ifp6esesBHDsIvYDowG1iBHvQ4hVO49/SnUyuKyxqNvkO/ 7bmPHKo9m7QPwRqWseQOIAxd4MGTZlmLswvyk2BFfbviZMyEFVdeUl6FforiFvqPCaTw Fcp6aBUn08w1ytjHCZGWAmPz6KeBflhR4bSL3oJKjbuBmiurrVlzQ6wjdAxEaE/0722/ OHmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=EoY+xVUIMEYkMAOym+tsxa0CiEBJagAGYOcFqP2KHJM=; b=Q/h6JVfCUBkvonWvRpzZO6R9JTZRIUksEb4lfZZdGsW0piNdXBPZRxAKvarRNZe7vQ dlFSTjEDz2RqO1oCq9IQb5MketUAx2B7hlrhLKdOnU0mgMqM6ujHYxkjSEQ9ZkyxLb+q vZW/mrAln/ujoFjcfcyKd2fChqhV+GMtf5S1aHJtwYXkwLXdf08nJKog6RPbBwK2m5LC kjdXMTHi+ZIxpEsZyfqMAPQ09KkRpiLfFlfdPEwtaQXcuVkk9M44TiBVBTOgPWohIYw1 gwgHLQO1m6RhO0C+hKYPiWnPv7a+5KRr8ZxqQ8aEAhuTbz+SxC2wwmQY2T3nq2OtUCJ6 B3WA== X-Gm-Message-State: AOAM533u2aNUwpmZTsCtF45xY6cqXnRPely/s5BJKQg8g1czMMeCP/og JJbE4Q/sSFt3IP2mvGzucBr7gQ== X-Google-Smtp-Source: ABdhPJw4jPv7cP2XDcv6q3vNe4wcTSoEXR3244myJw8jkKChOcifNjx+92CtSjt4a7uK5qWlSjnRNg== X-Received: by 2002:a05:6a00:1e:b029:241:72c2:695e with SMTP id h30-20020a056a00001eb029024172c2695emr12073445pfk.17.1617971581010; Fri, 09 Apr 2021 05:33:01 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.243]) by smtp.gmail.com with ESMTPSA id t3sm2167785pfg.176.2021.04.09.05.32.55 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 09 Apr 2021 05:33:00 -0700 (PDT) From: Muchun Song To: guro@fb.com, hannes@cmpxchg.org, mhocko@kernel.org, akpm@linux-foundation.org, shakeelb@google.com, vdavydov.dev@gmail.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, bsingharora@gmail.com, shy828301@gmail.com, alex.shi@linux.alibaba.com, Muchun Song Subject: [RFC PATCH v2 12/18] mm: thp: make deferred split queue lock safe when the LRU pages reparented Date: Fri, 9 Apr 2021 20:29:53 +0800 Message-Id: <20210409122959.82264-13-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210409122959.82264-1-songmuchun@bytedance.com> References: <20210409122959.82264-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Stat-Signature: 933y45ko4w6cfhccpw1tmhku6kf75jjc X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 5535380192DA Received-SPF: none (bytedance.com>: No applicable sender policy available) receiver=imf16; identity=mailfrom; envelope-from=""; helo=mail-pf1-f178.google.com; client-ip=209.85.210.178 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617971581-207390 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Similar to lruvec lock, we use the same approach to make the lock safe when the LRU pages reparented. Signed-off-by: Muchun Song --- mm/huge_memory.c | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 275dbfc8b2ae..aa5d7b72d5fc 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -497,6 +497,8 @@ static struct deferred_split *lock_split_queue(struct page *page) struct deferred_split *queue; struct mem_cgroup *memcg; + rcu_read_lock(); +retry: memcg = page_memcg(compound_head(page)); if (memcg) queue = &memcg->deferred_split_queue; @@ -504,6 +506,17 @@ static struct deferred_split *lock_split_queue(struct page *page) queue = &NODE_DATA(page_to_nid(page))->deferred_split_queue; spin_lock(&queue->split_queue_lock); + if (unlikely(memcg != page_memcg(page))) { + spin_unlock(&queue->split_queue_lock); + goto retry; + } + + /* + * Preemption is disabled in the internal of spin_lock, which can serve + * as RCU read-side critical sections. + */ + rcu_read_unlock(); + return queue; } @@ -513,6 +526,8 @@ static struct deferred_split *lock_split_queue_irqsave(struct page *page, struct deferred_split *queue; struct mem_cgroup *memcg; + rcu_read_lock(); +retry: memcg = page_memcg(compound_head(page)); if (memcg) queue = &memcg->deferred_split_queue; @@ -520,6 +535,14 @@ static struct deferred_split *lock_split_queue_irqsave(struct page *page, queue = &NODE_DATA(page_to_nid(page))->deferred_split_queue; spin_lock_irqsave(&queue->split_queue_lock, *flags); + if (unlikely(memcg != page_memcg(page))) { + spin_unlock_irqrestore(&queue->split_queue_lock, *flags); + goto retry; + } + + /* See the comments in lock_split_queue(). */ + rcu_read_unlock(); + return queue; } #else From patchwork Fri Apr 9 12:29:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12194055 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2D347C433B4 for ; Fri, 9 Apr 2021 12:33:14 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C33A3610FC for ; Fri, 9 Apr 2021 12:33:13 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C33A3610FC Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 5C1496B0085; Fri, 9 Apr 2021 08:33:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5976C6B0087; Fri, 9 Apr 2021 08:33:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 45FC86B0088; Fri, 9 Apr 2021 08:33:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0001.hostedemail.com [216.40.44.1]) by kanga.kvack.org (Postfix) with ESMTP id 2B52D6B0085 for ; Fri, 9 Apr 2021 08:33:13 -0400 (EDT) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id E337C18020C40 for ; Fri, 9 Apr 2021 12:33:12 +0000 (UTC) X-FDA: 78012768624.12.53A707D Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) by imf21.hostedemail.com (Postfix) with ESMTP id 5FD27E00010B for ; Fri, 9 Apr 2021 12:33:11 +0000 (UTC) Received: by mail-pl1-f178.google.com with SMTP id d8so2670951plh.11 for ; Fri, 09 Apr 2021 05:33:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=WkwygiVjWU/odQpPcj4LBTXPK99AOuCbqNnVZygcuVU=; b=OIiYGYgaLyMc9DaF2jATUJ0wdYKQCFK8qzvLjnnZ6Kljx67Go32v69xS7DY+3/vuWv yU/ZzuXVny826ac8DW0pY9WFt5tipn98gG4VJ0WgtYLUhp2mzt2RXZn5asU9Cy1Td7XS t/oaLfVetQ8ZwX9gFq5yHKbpvb94Q1apj71OWxnBO+q2araLy6SGlpbSmPTph7zp4ti1 g4lKnReN5wI23YRp9si7JJOTdVLGZvD0t23OE5/+Y6IwmhhLfzweLm1aB/UmDyeQkMHa uOwrKfhYg0guPR3WtSCsQd12J40WHj17QEtIpoDYZjEPH3flBijHiujYvey1CneOs5bg 8Bew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=WkwygiVjWU/odQpPcj4LBTXPK99AOuCbqNnVZygcuVU=; b=jAO6+G2rz6PJXrbAbu7Zr3btvQjkwA+XaZUiThmR5yR2aol1JhLqhd2uBFXJoJ2054 UG+eZQiyJMi/2YKc76DEScN3cz6pxlDxUySgwrgmDYwiEUo8D9XSV+HEqURhJzSy4dyy baa7pFsPVMXO6cin3HbQTlQGg7eCDLDB2UaC95TnD+CW94l+V1wh0kw3Y3Dm95wP1cuM kZJVcR83Q1L34pTqt7dTCQznk/cdaGHCLVljQvSzNK0d8GFuLFYNTJQKqQ39a31MHdDt LQHLSMF/1ImCfwcP0pCEWrX/h2z1X35FOzoDBFGEiS8JGy9TbS/jW7Koi49EzMiQzA2U PaOg== X-Gm-Message-State: AOAM532jd+dEA7g5mK7b43+nCw+GCYLrkwtj/PYBYc8WBr99dSmRRHOI i6zU3cmRw68Nu40d/arM851XyQ== X-Google-Smtp-Source: ABdhPJz+6M0EaKA7UonG/6FPqKDEcNlLgKwpnR3sggRvqNvIdbhIrdBCZgRDV6Brnp/QVho2vP1XbA== X-Received: by 2002:a17:90a:3904:: with SMTP id y4mr13371509pjb.125.1617971591798; Fri, 09 Apr 2021 05:33:11 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.243]) by smtp.gmail.com with ESMTPSA id t3sm2167785pfg.176.2021.04.09.05.33.06 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 09 Apr 2021 05:33:11 -0700 (PDT) From: Muchun Song To: guro@fb.com, hannes@cmpxchg.org, mhocko@kernel.org, akpm@linux-foundation.org, shakeelb@google.com, vdavydov.dev@gmail.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, bsingharora@gmail.com, shy828301@gmail.com, alex.shi@linux.alibaba.com, Muchun Song Subject: [RFC PATCH v2 14/18] mm: memcontrol: introduce memcg_reparent_ops Date: Fri, 9 Apr 2021 20:29:55 +0800 Message-Id: <20210409122959.82264-15-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210409122959.82264-1-songmuchun@bytedance.com> References: <20210409122959.82264-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 5FD27E00010B X-Stat-Signature: 9k8jwaa1u15apjuqzjj47imtntzcng6c Received-SPF: none (bytedance.com>: No applicable sender policy available) receiver=imf21; identity=mailfrom; envelope-from=""; helo=mail-pl1-f178.google.com; client-ip=209.85.214.178 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617971591-607555 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In the previous patch, we know how to make the lruvec lock safe when the LRU pages reparented. We should do something like following. memcg_reparent_objcgs(memcg) 1) lock // lruvec belongs to memcg and lruvec_parent belongs to parent memcg. spin_lock(&lruvec->lru_lock); spin_lock(&lruvec_parent->lru_lock); 2) do reparent // Move all the pages from the lruvec list to the parent lruvec list. 3) unlock spin_unlock(&lruvec_parent->lru_lock); spin_unlock(&lruvec->lru_lock); Apart from the page lruvec lock, the deferred split queue lock (THP only) also needs to do something similar. So we extracted the necessary 3 steps in the memcg_reparent_objcgs(). memcg_reparent_objcgs(memcg) 1) lock memcg_reparent_ops->lock(memcg, parent); 2) reparent memcg_reparent_ops->reparent(memcg, reparent); 3) unlock memcg_reparent_ops->unlock(memcg, reparent); Now there are two different locks (e.g. lruvec lock and deferred split queue lock) need to use this infrastructure. In the next patch, we will use those APIs to make those locks safe when the LRU pages reparented. Signed-off-by: Muchun Song --- include/linux/memcontrol.h | 11 +++++++++++ mm/memcontrol.c | 49 ++++++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 58 insertions(+), 2 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 7e15be2bd47a..fdc385ecff55 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -355,6 +355,17 @@ struct mem_cgroup { /* WARNING: nodeinfo must be the last member here */ }; +struct memcg_reparent_ops { + struct list_head list; + + /* Irq is disabled before calling those functions. */ + void (*lock)(struct mem_cgroup *memcg, struct mem_cgroup *parent); + void (*unlock)(struct mem_cgroup *memcg, struct mem_cgroup *parent); + void (*reparent)(struct mem_cgroup *memcg, struct mem_cgroup *parent); +}; + +void __init register_memcg_repatent(struct memcg_reparent_ops *ops); + /* * size of first charge trial. "32" comes from vmscan.c's magic value. * TODO: maybe necessary to use big numbers in big irons. diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 2f4fcb182883..536be7bdc98f 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -338,6 +338,41 @@ static struct obj_cgroup *obj_cgroup_alloc(void) return objcg; } +static LIST_HEAD(reparent_ops_head); + +static void memcg_reparent_lock(struct mem_cgroup *memcg, + struct mem_cgroup *parent) +{ + struct memcg_reparent_ops *ops; + + list_for_each_entry(ops, &reparent_ops_head, list) + ops->lock(memcg, parent); +} + +static void memcg_reparent_unlock(struct mem_cgroup *memcg, + struct mem_cgroup *parent) +{ + struct memcg_reparent_ops *ops; + + list_for_each_entry(ops, &reparent_ops_head, list) + ops->unlock(memcg, parent); +} + +static void memcg_do_reparent(struct mem_cgroup *memcg, + struct mem_cgroup *parent) +{ + struct memcg_reparent_ops *ops; + + list_for_each_entry(ops, &reparent_ops_head, list) + ops->reparent(memcg, parent); +} + +void __init register_memcg_repatent(struct memcg_reparent_ops *ops) +{ + BUG_ON(!ops->lock || !ops->unlock || !ops->reparent); + list_add(&ops->list, &reparent_ops_head); +} + static void memcg_reparent_objcgs(struct mem_cgroup *memcg) { struct obj_cgroup *objcg, *iter; @@ -347,9 +382,13 @@ static void memcg_reparent_objcgs(struct mem_cgroup *memcg) if (!parent) parent = root_mem_cgroup; + local_irq_disable(); + + memcg_reparent_lock(memcg, parent); + objcg = rcu_replace_pointer(memcg->objcg, NULL, true); - spin_lock_irq(&css_set_lock); + spin_lock(&css_set_lock); /* 1) Ready to reparent active objcg. */ list_add(&objcg->list, &memcg->objcg_list); @@ -361,7 +400,13 @@ static void memcg_reparent_objcgs(struct mem_cgroup *memcg) /* 3) Move already reparented objcgs to the parent's list */ list_splice(&memcg->objcg_list, &parent->objcg_list); - spin_unlock_irq(&css_set_lock); + spin_unlock(&css_set_lock); + + memcg_do_reparent(memcg, parent); + + memcg_reparent_unlock(memcg, parent); + + local_irq_enable(); percpu_ref_kill(&objcg->refcnt); }