From patchwork Fri Apr 9 12:29:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12194055 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2D347C433B4 for ; Fri, 9 Apr 2021 12:33:14 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C33A3610FC for ; Fri, 9 Apr 2021 12:33:13 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C33A3610FC Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 5C1496B0085; Fri, 9 Apr 2021 08:33:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5976C6B0087; Fri, 9 Apr 2021 08:33:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 45FC86B0088; Fri, 9 Apr 2021 08:33:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0001.hostedemail.com [216.40.44.1]) by kanga.kvack.org (Postfix) with ESMTP id 2B52D6B0085 for ; Fri, 9 Apr 2021 08:33:13 -0400 (EDT) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id E337C18020C40 for ; Fri, 9 Apr 2021 12:33:12 +0000 (UTC) X-FDA: 78012768624.12.53A707D Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) by imf21.hostedemail.com (Postfix) with ESMTP id 5FD27E00010B for ; Fri, 9 Apr 2021 12:33:11 +0000 (UTC) Received: by mail-pl1-f178.google.com with SMTP id d8so2670951plh.11 for ; Fri, 09 Apr 2021 05:33:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=WkwygiVjWU/odQpPcj4LBTXPK99AOuCbqNnVZygcuVU=; b=OIiYGYgaLyMc9DaF2jATUJ0wdYKQCFK8qzvLjnnZ6Kljx67Go32v69xS7DY+3/vuWv yU/ZzuXVny826ac8DW0pY9WFt5tipn98gG4VJ0WgtYLUhp2mzt2RXZn5asU9Cy1Td7XS t/oaLfVetQ8ZwX9gFq5yHKbpvb94Q1apj71OWxnBO+q2araLy6SGlpbSmPTph7zp4ti1 g4lKnReN5wI23YRp9si7JJOTdVLGZvD0t23OE5/+Y6IwmhhLfzweLm1aB/UmDyeQkMHa uOwrKfhYg0guPR3WtSCsQd12J40WHj17QEtIpoDYZjEPH3flBijHiujYvey1CneOs5bg 8Bew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=WkwygiVjWU/odQpPcj4LBTXPK99AOuCbqNnVZygcuVU=; b=jAO6+G2rz6PJXrbAbu7Zr3btvQjkwA+XaZUiThmR5yR2aol1JhLqhd2uBFXJoJ2054 UG+eZQiyJMi/2YKc76DEScN3cz6pxlDxUySgwrgmDYwiEUo8D9XSV+HEqURhJzSy4dyy baa7pFsPVMXO6cin3HbQTlQGg7eCDLDB2UaC95TnD+CW94l+V1wh0kw3Y3Dm95wP1cuM kZJVcR83Q1L34pTqt7dTCQznk/cdaGHCLVljQvSzNK0d8GFuLFYNTJQKqQ39a31MHdDt LQHLSMF/1ImCfwcP0pCEWrX/h2z1X35FOzoDBFGEiS8JGy9TbS/jW7Koi49EzMiQzA2U PaOg== X-Gm-Message-State: AOAM532jd+dEA7g5mK7b43+nCw+GCYLrkwtj/PYBYc8WBr99dSmRRHOI i6zU3cmRw68Nu40d/arM851XyQ== X-Google-Smtp-Source: ABdhPJz+6M0EaKA7UonG/6FPqKDEcNlLgKwpnR3sggRvqNvIdbhIrdBCZgRDV6Brnp/QVho2vP1XbA== X-Received: by 2002:a17:90a:3904:: with SMTP id y4mr13371509pjb.125.1617971591798; Fri, 09 Apr 2021 05:33:11 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.243]) by smtp.gmail.com with ESMTPSA id t3sm2167785pfg.176.2021.04.09.05.33.06 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 09 Apr 2021 05:33:11 -0700 (PDT) From: Muchun Song To: guro@fb.com, hannes@cmpxchg.org, mhocko@kernel.org, akpm@linux-foundation.org, shakeelb@google.com, vdavydov.dev@gmail.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, bsingharora@gmail.com, shy828301@gmail.com, alex.shi@linux.alibaba.com, Muchun Song Subject: [RFC PATCH v2 14/18] mm: memcontrol: introduce memcg_reparent_ops Date: Fri, 9 Apr 2021 20:29:55 +0800 Message-Id: <20210409122959.82264-15-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210409122959.82264-1-songmuchun@bytedance.com> References: <20210409122959.82264-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 5FD27E00010B X-Stat-Signature: 9k8jwaa1u15apjuqzjj47imtntzcng6c Received-SPF: none (bytedance.com>: No applicable sender policy available) receiver=imf21; identity=mailfrom; envelope-from=""; helo=mail-pl1-f178.google.com; client-ip=209.85.214.178 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617971591-607555 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In the previous patch, we know how to make the lruvec lock safe when the LRU pages reparented. We should do something like following. memcg_reparent_objcgs(memcg) 1) lock // lruvec belongs to memcg and lruvec_parent belongs to parent memcg. spin_lock(&lruvec->lru_lock); spin_lock(&lruvec_parent->lru_lock); 2) do reparent // Move all the pages from the lruvec list to the parent lruvec list. 3) unlock spin_unlock(&lruvec_parent->lru_lock); spin_unlock(&lruvec->lru_lock); Apart from the page lruvec lock, the deferred split queue lock (THP only) also needs to do something similar. So we extracted the necessary 3 steps in the memcg_reparent_objcgs(). memcg_reparent_objcgs(memcg) 1) lock memcg_reparent_ops->lock(memcg, parent); 2) reparent memcg_reparent_ops->reparent(memcg, reparent); 3) unlock memcg_reparent_ops->unlock(memcg, reparent); Now there are two different locks (e.g. lruvec lock and deferred split queue lock) need to use this infrastructure. In the next patch, we will use those APIs to make those locks safe when the LRU pages reparented. Signed-off-by: Muchun Song --- include/linux/memcontrol.h | 11 +++++++++++ mm/memcontrol.c | 49 ++++++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 58 insertions(+), 2 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 7e15be2bd47a..fdc385ecff55 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -355,6 +355,17 @@ struct mem_cgroup { /* WARNING: nodeinfo must be the last member here */ }; +struct memcg_reparent_ops { + struct list_head list; + + /* Irq is disabled before calling those functions. */ + void (*lock)(struct mem_cgroup *memcg, struct mem_cgroup *parent); + void (*unlock)(struct mem_cgroup *memcg, struct mem_cgroup *parent); + void (*reparent)(struct mem_cgroup *memcg, struct mem_cgroup *parent); +}; + +void __init register_memcg_repatent(struct memcg_reparent_ops *ops); + /* * size of first charge trial. "32" comes from vmscan.c's magic value. * TODO: maybe necessary to use big numbers in big irons. diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 2f4fcb182883..536be7bdc98f 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -338,6 +338,41 @@ static struct obj_cgroup *obj_cgroup_alloc(void) return objcg; } +static LIST_HEAD(reparent_ops_head); + +static void memcg_reparent_lock(struct mem_cgroup *memcg, + struct mem_cgroup *parent) +{ + struct memcg_reparent_ops *ops; + + list_for_each_entry(ops, &reparent_ops_head, list) + ops->lock(memcg, parent); +} + +static void memcg_reparent_unlock(struct mem_cgroup *memcg, + struct mem_cgroup *parent) +{ + struct memcg_reparent_ops *ops; + + list_for_each_entry(ops, &reparent_ops_head, list) + ops->unlock(memcg, parent); +} + +static void memcg_do_reparent(struct mem_cgroup *memcg, + struct mem_cgroup *parent) +{ + struct memcg_reparent_ops *ops; + + list_for_each_entry(ops, &reparent_ops_head, list) + ops->reparent(memcg, parent); +} + +void __init register_memcg_repatent(struct memcg_reparent_ops *ops) +{ + BUG_ON(!ops->lock || !ops->unlock || !ops->reparent); + list_add(&ops->list, &reparent_ops_head); +} + static void memcg_reparent_objcgs(struct mem_cgroup *memcg) { struct obj_cgroup *objcg, *iter; @@ -347,9 +382,13 @@ static void memcg_reparent_objcgs(struct mem_cgroup *memcg) if (!parent) parent = root_mem_cgroup; + local_irq_disable(); + + memcg_reparent_lock(memcg, parent); + objcg = rcu_replace_pointer(memcg->objcg, NULL, true); - spin_lock_irq(&css_set_lock); + spin_lock(&css_set_lock); /* 1) Ready to reparent active objcg. */ list_add(&objcg->list, &memcg->objcg_list); @@ -361,7 +400,13 @@ static void memcg_reparent_objcgs(struct mem_cgroup *memcg) /* 3) Move already reparented objcgs to the parent's list */ list_splice(&memcg->objcg_list, &parent->objcg_list); - spin_unlock_irq(&css_set_lock); + spin_unlock(&css_set_lock); + + memcg_do_reparent(memcg, parent); + + memcg_reparent_unlock(memcg, parent); + + local_irq_enable(); percpu_ref_kill(&objcg->refcnt); }