From patchwork Fri Jun 3 13:42:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 12869086 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39A67C433EF for ; Fri, 3 Jun 2022 13:44:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CC1DE8D0006; Fri, 3 Jun 2022 09:44:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C75E38D0001; Fri, 3 Jun 2022 09:44:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9FD678D0006; Fri, 3 Jun 2022 09:44:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 916F88D0001 for ; Fri, 3 Jun 2022 09:44:43 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 77CA315E5 for ; Fri, 3 Jun 2022 13:44:43 +0000 (UTC) X-FDA: 79537044846.16.7B34892 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf01.hostedemail.com (Postfix) with ESMTP id 8D1BD4007B for ; Fri, 3 Jun 2022 13:44:34 +0000 (UTC) Received: from pps.filterd (m0098394.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 253CFLgN002903; Fri, 3 Jun 2022 13:44:37 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=BUCn0BY2ZXsWri4dRdU95JjN/8kb3U+L/F9XlYqa2Bg=; b=BVnWdjjulZzHbLLJyBuVL9VJJrL5p7elwwnZV1I9qfuEPb78HGwg8g7g0KD6NX1rarjq 9qjNMLIwKb0RIrS5nNaHca8k78ZVCHLQ8UDKstvqXGXStThoW9crb6NyCYXUCVV/8bka OB3d55aXErE9cfaAdwDxqTijwxoiRq/45WXA+FnJSVjE/rm44Pq50a/CU+MRsqjJyW90 YySszjPwPz0HZ1eP1MxJwKLzIzuE+XYxSSyCOeKXXdLaBzDqyeH3hQN0kkwc8Qs30sYY eVN1TZCHKm0067GYxYT+DlTtyX1psgvifpxlXWehSaq3qrw5t95OfOlIfTfQo/dCdGXq zQ== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3gfgg8b99f-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 03 Jun 2022 13:44:36 +0000 Received: from m0098394.ppops.net (m0098394.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 253Cwqok004481; Fri, 3 Jun 2022 13:44:36 GMT Received: from ppma04dal.us.ibm.com (7a.29.35a9.ip4.static.sl-reverse.com [169.53.41.122]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3gfgg8b996-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 03 Jun 2022 13:44:36 +0000 Received: from pps.filterd (ppma04dal.us.ibm.com [127.0.0.1]) by ppma04dal.us.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 253DaUVL027611; Fri, 3 Jun 2022 13:44:35 GMT Received: from b01cxnp22036.gho.pok.ibm.com (b01cxnp22036.gho.pok.ibm.com [9.57.198.26]) by ppma04dal.us.ibm.com with ESMTP id 3gd4n68dq9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 03 Jun 2022 13:44:35 +0000 Received: from b01ledav004.gho.pok.ibm.com (b01ledav004.gho.pok.ibm.com [9.57.199.109]) by b01cxnp22036.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 253DiYHU9830668 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 3 Jun 2022 13:44:34 GMT Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 190F0112063; Fri, 3 Jun 2022 13:44:34 +0000 (GMT) Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 84AC2112062; Fri, 3 Jun 2022 13:44:26 +0000 (GMT) Received: from skywalker.ibmuc.com (unknown [9.43.93.173]) by b01ledav004.gho.pok.ibm.com (Postfix) with ESMTP; Fri, 3 Jun 2022 13:44:26 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org Cc: Wei Xu , Huang Ying , Greg Thelen , Yang Shi , Davidlohr Bueso , Tim C Chen , Brice Goglin , Michal Hocko , Linux Kernel Mailing List , Hesham Almatary , Dave Hansen , Jonathan Cameron , Alistair Popple , Dan Williams , Feng Tang , Jagdish Gediya , Baolin Wang , David Rientjes , "Aneesh Kumar K.V" Subject: [PATCH v5 9/9] mm/demotion: Update node_is_toptier to work with memory tiers Date: Fri, 3 Jun 2022 19:12:37 +0530 Message-Id: <20220603134237.131362-10-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220603134237.131362-1-aneesh.kumar@linux.ibm.com> References: <20220603134237.131362-1-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: h6x5ag3c0QcohwVDA3Yn5j1SGQueDgDx X-Proofpoint-GUID: xxrlc1sSbC-F3CxfgTlo2b9hGvdng3ck X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.874,Hydra:6.0.517,FMLib:17.11.64.514 definitions=2022-06-03_04,2022-06-03_01,2022-02-23_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 clxscore=1015 impostorscore=0 lowpriorityscore=0 malwarescore=0 bulkscore=0 mlxlogscore=999 mlxscore=0 adultscore=0 phishscore=0 suspectscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2204290000 definitions=main-2206030059 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 8D1BD4007B X-Stat-Signature: isti94t4g1q51jomweaunemy5zhxtcub X-Rspam-User: Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=BVnWdjju; spf=pass (imf01.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com X-HE-Tag: 1654263874-535798 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: With memory tiers support we can have memory on NUMA nodes in the top tier from which we want to avoid promotion tracking NUMA faults. Update node_is_toptier to work with memory tiers. To avoid taking locks, a nodemask is maintained for all demotion targets. All NUMA nodes are by default top tier nodes and as we add new lower memory tiers NUMA nodes get added to the demotion targets thereby moving them out of the top tier. Signed-off-by: Aneesh Kumar K.V --- include/linux/memory-tiers.h | 16 ++++++++++++++++ include/linux/node.h | 5 ----- mm/huge_memory.c | 1 + mm/memory-tiers.c | 10 ++++++++++ mm/migrate.c | 1 + mm/mprotect.c | 1 + 6 files changed, 29 insertions(+), 5 deletions(-) diff --git a/include/linux/memory-tiers.h b/include/linux/memory-tiers.h index cd6e71f702ad..32e0e6fabf02 100644 --- a/include/linux/memory-tiers.h +++ b/include/linux/memory-tiers.h @@ -16,12 +16,23 @@ #define MAX_MEMORY_TIERS 3 extern bool numa_demotion_enabled; +extern nodemask_t demotion_target_mask; int next_demotion_node(int node); void node_remove_from_memory_tier(int node); int node_get_memory_tier_id(int node); int node_set_memory_tier(int node, int tier); int node_reset_memory_tier(int node, int tier); void node_get_allowed_targets(int node, nodemask_t *targets); + +/* + * By default all nodes are top tiper. As we create new memory tiers + * we below top tiers we add them to NON_TOP_TIER state. + */ +static inline bool node_is_toptier(int node) +{ + return !node_isset(node, demotion_target_mask); +} + #else #define numa_demotion_enabled false static inline int next_demotion_node(int node) @@ -33,6 +44,11 @@ static inline void node_get_allowed_targets(int node, nodemask_t *targets) { *targets = NODE_MASK_NONE; } + +static inline bool node_is_toptier(int node) +{ + return true; +} #endif /* CONFIG_TIERED_MEMORY */ #endif diff --git a/include/linux/node.h b/include/linux/node.h index 40d641a8bfb0..9ec680dd607f 100644 --- a/include/linux/node.h +++ b/include/linux/node.h @@ -185,9 +185,4 @@ static inline void register_hugetlbfs_with_node(node_registration_func_t reg, #define to_node(device) container_of(device, struct node, dev) -static inline bool node_is_toptier(int node) -{ - return node_state(node, N_CPU); -} - #endif /* _LINUX_NODE_H_ */ diff --git a/mm/huge_memory.c b/mm/huge_memory.c index a77c78a2b6b5..294873d4be2b 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -35,6 +35,7 @@ #include #include #include +#include #include #include diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c index 592d939ec28d..df8e9910165a 100644 --- a/mm/memory-tiers.c +++ b/mm/memory-tiers.c @@ -31,6 +31,7 @@ static struct bus_type memory_tier_subsys = { static void establish_migration_targets(void); static DEFINE_MUTEX(memory_tier_lock); static LIST_HEAD(memory_tiers); +nodemask_t demotion_target_mask; /* * node_demotion[] examples: @@ -541,6 +542,15 @@ static void establish_migration_targets(void) */ list_for_each_entry(memtier, &memory_tiers, list) nodes_or(allowed, allowed, memtier->nodelist); + /* + * Add nodes to demotion target mask so that we can find + * top tier easily. + */ + memtier = list_first_entry(&memory_tiers, struct memory_tier, list); + if (memtier) + nodes_andnot(demotion_target_mask, allowed, memtier->nodelist); + else + demotion_target_mask = NODE_MASK_NONE; /* * Removes nodes not yet in N_MEMORY. */ diff --git a/mm/migrate.c b/mm/migrate.c index 0b554625a219..78615c48fc0f 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -50,6 +50,7 @@ #include #include #include +#include #include diff --git a/mm/mprotect.c b/mm/mprotect.c index ba5592655ee3..92a2fc0fa88b 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -31,6 +31,7 @@ #include #include #include +#include #include #include #include