From patchwork Wed Apr 21 07:00:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12215553 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 17E9FC433B4 for ; Wed, 21 Apr 2021 07:01:35 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9398061008 for ; Wed, 21 Apr 2021 07:01:34 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9398061008 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 300406B0072; Wed, 21 Apr 2021 03:01:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2AEDA6B0073; Wed, 21 Apr 2021 03:01:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 101396B0074; Wed, 21 Apr 2021 03:01:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0215.hostedemail.com [216.40.44.215]) by kanga.kvack.org (Postfix) with ESMTP id E8EEB6B0072 for ; Wed, 21 Apr 2021 03:01:33 -0400 (EDT) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 9F0033641 for ; Wed, 21 Apr 2021 07:01:33 +0000 (UTC) X-FDA: 78055478466.04.A7E3A91 Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by imf12.hostedemail.com (Postfix) with ESMTP id 39327F6 for ; Wed, 21 Apr 2021 07:01:25 +0000 (UTC) Received: by mail-pl1-f180.google.com with SMTP id h20so21104268plr.4 for ; Wed, 21 Apr 2021 00:01:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=fwWuYiWQGSbc8ycMu6DseDSOCB8T/8g49Sen6QaPJMk=; b=m2EZPGDxxaBT2seoPtdlskpRShBLXOsRTclW3ZU5paVEBHQb/Gx7RYe69DBH5Z7CFn 6a6RM4gMZavG6XPfeqE8GV4ZRzpVSq78WRq+FkH/uFRhbnmpuTvm3YI8iaJo5BZdyt+D /PsRsFhf/cw3sZ5HC558yDI61EniO5tUMActASgND0z5vHJ5ZUe08a+mDYZ831AciJbi Ox86jT2ezW8iZZ6VOnXdQJfTiRMI8wZLjoblvZ6aa4F/hCYrJ+GSmv6aDQbWceVIpOxz s2cKYTj1KQ1K/PX9nbTtE0NbVLZ1EnKfzgdjhjPO0ERu1ibOKYkdpcpMjyF1ewe0RFml Shlw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=fwWuYiWQGSbc8ycMu6DseDSOCB8T/8g49Sen6QaPJMk=; b=qxCtzprgdFkfTGmTIo53IOPCbHpi/fOuRLjAB7SG2/UslnABlrY/P/wcaaZ351bZ2v 1acrTgJNEOX/N2p34eoonn/u+2XCCuoclaCL3RWDaybdCpDXF+pCp9VG7bYxTTZGhKEm HnuMds/AGzNtnB5te8l63xV2uLNR5k64MXO8fpoUWRaWLuOVeLk88JS2ddPCdWJQ7vGi Tl4d2ubbZQ7T76tudkTOWiJpjGhyymc0Tdjxa9S3rBiWWz2zL9fjpgPJdA8OwgvLatuB Qiy+4j40jB69TXtdqLOKuX3hWIMWkimrQB3q0itNzFyOnfizBIE5rgbHYKMv4gmFTCh/ jgrg== X-Gm-Message-State: AOAM530FJLnhzb8+rhWnyKc5zgjpYgz7ySHseMdNsVO+btHDo+wYSRyb M0uX0CGuEYMxkONsxudKToEeOA== X-Google-Smtp-Source: ABdhPJxegXTguuiACBJsla3VyxWjtQa2UzPwLhHrGXGiGwKuPa6HjTCl4K05aAx9E/CQ2SNSgxOsWg== X-Received: by 2002:a17:902:4c:b029:ec:a39a:41ad with SMTP id 70-20020a170902004cb02900eca39a41admr15082872pla.52.1618988492393; Wed, 21 Apr 2021 00:01:32 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.255]) by smtp.gmail.com with ESMTPSA id c4sm929842pfb.94.2021.04.21.00.01.27 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 21 Apr 2021 00:01:32 -0700 (PDT) From: Muchun Song To: guro@fb.com, hannes@cmpxchg.org, mhocko@kernel.org, akpm@linux-foundation.org, shakeelb@google.com, vdavydov.dev@gmail.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, bsingharora@gmail.com, shy828301@gmail.com, alex.shi@linux.alibaba.com, Muchun Song Subject: [RFC PATCH v3 03/12] mm: memcontrol: make lruvec lock safe when the LRU pages reparented Date: Wed, 21 Apr 2021 15:00:50 +0800 Message-Id: <20210421070059.69361-4-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210421070059.69361-1-songmuchun@bytedance.com> References: <20210421070059.69361-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 39327F6 X-Stat-Signature: czrt6pkrtzwm6akiahji3q3bqxu1h186 Received-SPF: none (bytedance.com>: No applicable sender policy available) receiver=imf12; identity=mailfrom; envelope-from=""; helo=mail-pl1-f180.google.com; client-ip=209.85.214.180 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1618988485-671114 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The diagram below shows how to make the page lruvec lock safe when the LRU pages reparented. lock_page_lruvec(page) retry: lruvec = mem_cgroup_page_lruvec(page); // The page is reparented at this time. spin_lock(&lruvec->lru_lock); if (unlikely(lruvec_memcg(lruvec) != page_memcg(page))) // Acquired the wrong lruvec lock and need to retry. // Because this page is on the parent memcg lruvec list. goto retry; // If we reach here, it means that page_memcg(page) is stable. memcg_reparent_objcgs(memcg) // lruvec belongs to memcg and lruvec_parent belongs to parent memcg. spin_lock(&lruvec->lru_lock); spin_lock(&lruvec_parent->lru_lock); // Move all the pages from the lruvec list to the parent lruvec list. spin_unlock(&lruvec_parent->lru_lock); spin_unlock(&lruvec->lru_lock); After we acquire the lruvec lock, we need to check whether the page is reparented. If so, we need to reacquire the new lruvec lock. On the routine of the LRU pages reparenting, we will also acquire the lruvec lock (Will be implemented in the later patch). So page_memcg() cannot be changed when we hold the lruvec lock. Since lruvec_memcg(lruvec) is always equal to page_memcg(page) after we hold the lruvec lock, lruvec_memcg_debug() check is pointless. So remove it. This is a preparation for reparenting the LRU pages. Signed-off-by: Muchun Song Acked-by: Roman Gushchin --- include/linux/memcontrol.h | 16 +++------------ mm/compaction.c | 10 +++++++++- mm/memcontrol.c | 50 +++++++++++++++++++++++++++------------------- mm/swap.c | 5 +++++ 4 files changed, 47 insertions(+), 34 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 53a19db9f8eb..cb0d99583f77 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -746,7 +746,9 @@ static inline struct lruvec *mem_cgroup_lruvec(struct mem_cgroup *memcg, * mem_cgroup_page_lruvec - return lruvec for isolating/putting an LRU page * @page: the page * - * This function relies on page->mem_cgroup being stable. + * The lruvec can be changed to its parent lruvec when the page reparented. + * The caller need to recheck if it cares about this change (just like + * lock_page_lruvec() does). */ static inline struct lruvec *mem_cgroup_page_lruvec(struct page *page) { @@ -766,14 +768,6 @@ struct lruvec *lock_page_lruvec_irq(struct page *page); struct lruvec *lock_page_lruvec_irqsave(struct page *page, unsigned long *flags); -#ifdef CONFIG_DEBUG_VM -void lruvec_memcg_debug(struct lruvec *lruvec, struct page *page); -#else -static inline void lruvec_memcg_debug(struct lruvec *lruvec, struct page *page) -{ -} -#endif - static inline struct mem_cgroup *mem_cgroup_from_css(struct cgroup_subsys_state *css){ return css ? container_of(css, struct mem_cgroup, css) : NULL; @@ -1213,10 +1207,6 @@ static inline struct lruvec *mem_cgroup_page_lruvec(struct page *page) return &pgdat->__lruvec; } -static inline void lruvec_memcg_debug(struct lruvec *lruvec, struct page *page) -{ -} - static inline struct mem_cgroup *parent_mem_cgroup(struct mem_cgroup *memcg) { return NULL; diff --git a/mm/compaction.c b/mm/compaction.c index 082293587cc6..f88e2731d27c 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -517,6 +517,8 @@ compact_lock_page_lruvec_irqsave(struct page *page, unsigned long *flags, { struct lruvec *lruvec; + rcu_read_lock(); +retry: lruvec = mem_cgroup_page_lruvec(page); /* Track if the lock is contended in async mode */ @@ -529,7 +531,13 @@ compact_lock_page_lruvec_irqsave(struct page *page, unsigned long *flags, spin_lock_irqsave(&lruvec->lru_lock, *flags); out: - lruvec_memcg_debug(lruvec, page); + if (unlikely(lruvec_memcg(lruvec) != page_memcg(page))) { + spin_unlock_irqrestore(&lruvec->lru_lock, *flags); + goto retry; + } + + /* See the comments in lock_page_lruvec(). */ + rcu_read_unlock(); return lruvec; } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index dae7ab2b1406..fd8e2c242726 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1160,23 +1160,6 @@ int mem_cgroup_scan_tasks(struct mem_cgroup *memcg, return ret; } -#ifdef CONFIG_DEBUG_VM -void lruvec_memcg_debug(struct lruvec *lruvec, struct page *page) -{ - struct mem_cgroup *memcg; - - if (mem_cgroup_disabled()) - return; - - memcg = page_memcg(page); - - if (!memcg) - VM_BUG_ON_PAGE(lruvec_memcg(lruvec) != root_mem_cgroup, page); - else - VM_BUG_ON_PAGE(lruvec_memcg(lruvec) != memcg, page); -} -#endif - /** * lock_page_lruvec - lock and return lruvec for a given page. * @page: the page @@ -1191,10 +1174,21 @@ struct lruvec *lock_page_lruvec(struct page *page) { struct lruvec *lruvec; + rcu_read_lock(); +retry: lruvec = mem_cgroup_page_lruvec(page); spin_lock(&lruvec->lru_lock); - lruvec_memcg_debug(lruvec, page); + if (unlikely(lruvec_memcg(lruvec) != page_memcg(page))) { + spin_unlock(&lruvec->lru_lock); + goto retry; + } + + /* + * Preemption is disabled in the internal of spin_lock, which can serve + * as RCU read-side critical sections. + */ + rcu_read_unlock(); return lruvec; } @@ -1203,10 +1197,18 @@ struct lruvec *lock_page_lruvec_irq(struct page *page) { struct lruvec *lruvec; + rcu_read_lock(); +retry: lruvec = mem_cgroup_page_lruvec(page); spin_lock_irq(&lruvec->lru_lock); - lruvec_memcg_debug(lruvec, page); + if (unlikely(lruvec_memcg(lruvec) != page_memcg(page))) { + spin_unlock_irq(&lruvec->lru_lock); + goto retry; + } + + /* See the comments in lock_page_lruvec(). */ + rcu_read_unlock(); return lruvec; } @@ -1215,10 +1217,18 @@ struct lruvec *lock_page_lruvec_irqsave(struct page *page, unsigned long *flags) { struct lruvec *lruvec; + rcu_read_lock(); +retry: lruvec = mem_cgroup_page_lruvec(page); spin_lock_irqsave(&lruvec->lru_lock, *flags); - lruvec_memcg_debug(lruvec, page); + if (unlikely(lruvec_memcg(lruvec) != page_memcg(page))) { + spin_unlock_irqrestore(&lruvec->lru_lock, *flags); + goto retry; + } + + /* See the comments in lock_page_lruvec(). */ + rcu_read_unlock(); return lruvec; } diff --git a/mm/swap.c b/mm/swap.c index e0d5699213cc..f3ce307d09fa 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -313,6 +313,11 @@ void lru_note_cost(struct lruvec *lruvec, bool file, unsigned int nr_pages) void lru_note_cost_page(struct page *page) { + /* + * The rcu read lock is held by the caller, so we do not need to + * care about the lruvec returned by mem_cgroup_page_lruvec() being + * released. + */ lru_note_cost(mem_cgroup_page_lruvec(page), page_is_file_lru(page), thp_nr_pages(page)); }