From patchwork Fri Mar 1 08:22:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Ho-Ren (Jack) Chuang" X-Patchwork-Id: 13578137 Received: from mail-qv1-f42.google.com (mail-qv1-f42.google.com [209.85.219.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F172F69D38 for ; Fri, 1 Mar 2024 08:24:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709281445; cv=none; b=ji5tqUSq2RKJ9wyoOkXZE2A2a4xzy0cYkJiZzDBHQB0qQHPXd5uXqTKrqKC845PWoCANXHYyMVE2/kor8lMmm2cbx9CdWiLL877I9vpRhFhT07D5A+q0/wJ7fapchDc99kYii2rqxYSo+eumXu54jz/LQN6IW3/q3nm7eqSJfEc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709281445; c=relaxed/simple; bh=UataCGYo3G62/ULCC+IEnJOgFpYUGdT+U0HsTd/TvSo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=n/sp/zM6sR2XmjibNGbBRIScEgCijYroXFHfsw01Xw+L+bzdDuMteoA+L4AENk1CgM4XWB0EnpeWfevxF7lohUDDz7yZfJSNJa9sCd/sYa0hMfoREJeA02kGUPijvHw5j4jtoahwPFd+chZIp9yGp2nG6aI844JNRD5FgFOnycI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=PdpixPhM; arc=none smtp.client-ip=209.85.219.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="PdpixPhM" Received: by mail-qv1-f42.google.com with SMTP id 6a1803df08f44-68f901192afso7610356d6.1 for ; Fri, 01 Mar 2024 00:24:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1709281443; x=1709886243; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/6TShqh1i4+cFFfpKPT0xQIuYg06VEOVkdOiTtXAx9c=; b=PdpixPhMruAZAgHSjapjfKrqUVtgPikqaCmlnA2+G9Gh6OO2Od9R0lxRCKwC+C0dTZ v7ENMlrcDkQAyiQMCGZ1kP8Zn2QFq+Mzj+CGscteFPYjBRfJZ60R1NOwhzPtq/cUhv8D y0vwM5WFRJBx1Ghx4YnBKjLIYUbkLxp+5qz9mP1mVkU/5WtcsC691Xcbi2POhDJikAQ0 nPrJeQuZGbkPu5+FBy0ldxzq3s65WYgg6ZEEB35dCuS1ub7VDebOdNuYtFR6wQLgAjfL 18flJtnCsuLjVg94aSWEOpzkocmcilK2Dm7hTq9OoYuO0U8bV9chtFonp47XvEXronQA 0rNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709281443; x=1709886243; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/6TShqh1i4+cFFfpKPT0xQIuYg06VEOVkdOiTtXAx9c=; b=QdF9/0PVolTbh8ButkHMN4H768gQtjlq0Az5tAdd3qWS/Sm8GbfS+3DUMwigr+s7xR uQGUae8qEw6330e79sbn4l+Pynikt4g5iuCW2WUwQvNVH/dpbRrjOzcXRzvo/eRBaP+k W2ovQtizVx/lCRLglZoeQpmzzyie0PdY9OanhGd+tqorrFK+lMUwA5hTQke3PkYTD8dA O1qBlqjcjV4mAv/VIeyIB77qrIvNaqn086Oh6e2xHAyoq748Q3537JwgfYhC6zlkW9IV eLGAngNKsDKyc53DqOb3sybB3wv6TxDw+7VhHTTwlXX7hzPotcl0BVQNH8o82yvhqeTF GmdA== X-Forwarded-Encrypted: i=1; AJvYcCWeF8JMW6wuKuu/umbITO/FEZCF54bqxvaAsWvPdewbShijdtFu7M+GMKeFb9NktKgoK2Qf6QDUEI241Ia57Ex7vrvX9QvRLvG/ X-Gm-Message-State: AOJu0Ywop7z1ygdWhEr0+dLo1ER2qWOcHVlCOttmqt6g8C9L4n2veUsE WmEhYWIpZSAf5/2KgvOx5cPOcXZ3n8e8NicESyfpc7v4uqYfUC93W4H7wVybT1doP4qrlotn+lR kkgc= X-Google-Smtp-Source: AGHT+IFOE//de5gW8/t5EN6iDGtxoPupJsqmHB+gIs7CE75HdwkbfH6RIWi1eK5iB+kKO7THZmG8hQ== X-Received: by 2002:a05:6214:42:b0:68f:280f:14cc with SMTP id c2-20020a056214004200b0068f280f14ccmr982605qvr.35.1709281443055; Fri, 01 Mar 2024 00:24:03 -0800 (PST) Received: from n231-228-171.byted.org ([130.44.215.123]) by smtp.gmail.com with ESMTPSA id y19-20020a0cd993000000b0068fc392f526sm1631907qvj.127.2024.03.01.00.24.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Mar 2024 00:24:02 -0800 (PST) From: "Ho-Ren (Jack) Chuang" To: "Hao Xiang" , "Gregory Price" , aneesh.kumar@linux.ibm.com, mhocko@suse.com, tj@kernel.org, john@jagalactic.com, "Eishan Mirakhur" , "Vinicius Tavares Petrucci" , "Ravis OpenSrc" , "Alistair Popple" , "Rafael J. Wysocki" , Len Brown , Andrew Morton , Dave Jiang , Dan Williams , Jonathan Cameron , Huang Ying , "Ho-Ren (Jack) Chuang" , linux-acpi@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: "Ho-Ren (Jack) Chuang" , "Ho-Ren (Jack) Chuang" , linux-cxl@vger.kernel.org, qemu-devel@nongnu.org Subject: [PATCH v1 1/1] memory tier: acpi/hmat: create CPUless memory tiers after obtaining HMAT info Date: Fri, 1 Mar 2024 08:22:45 +0000 Message-Id: <20240301082248.3456086-2-horenchuang@bytedance.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20240301082248.3456086-1-horenchuang@bytedance.com> References: <20240301082248.3456086-1-horenchuang@bytedance.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 * Introduce `mt_init_with_hmat()` We defer memory tier initialization for those CPUless NUMA nodes until acquiring HMAT info. `mt_init_with_hmat()` is introduced to post-create CPUless memory tiers after obtaining HMAT info. It iterates through each CPUless memory node, creating memory tiers if necessary. Finally, it calculates demotion tables again at the end. * Introduce `hmat_find_alloc_memory_type()` Find or allocate a memory type in the `hmat_memory_types` list. * Make `set_node_memory_tier()` more generic This function can also be used for setting other memory types for a node. To do so, a new argument is added to specify a memory type. * Handle cases where there is no HMAT when creating memory tiers If no HMAT is specified, it falls back to using `default_dram_type`. * Change adist calculation code to use another new lock, mt_perf_lock. Iterating through CPUlist nodes requires holding the `memory_tier_lock`. However, `mt_calc_adistance()` will end up trying to acquire the same lock, leading to a potential deadlock. Therefore, we propose introducing a standalone `mt_perf_lock` to protect `default_dram_perf`. This approach not only avoids deadlock but also prevents holding a large lock simultaneously. Signed-off-by: Ho-Ren (Jack) Chuang Signed-off-by: Hao Xiang --- drivers/acpi/numa/hmat.c | 3 ++ include/linux/memory-tiers.h | 6 +++ mm/memory-tiers.c | 76 ++++++++++++++++++++++++++++++++---- 3 files changed, 77 insertions(+), 8 deletions(-) diff --git a/drivers/acpi/numa/hmat.c b/drivers/acpi/numa/hmat.c index d6b85f0f6082..9f57338b3cb5 100644 --- a/drivers/acpi/numa/hmat.c +++ b/drivers/acpi/numa/hmat.c @@ -1038,6 +1038,9 @@ static __init int hmat_init(void) if (!hmat_set_default_dram_perf()) register_mt_adistance_algorithm(&hmat_adist_nb); + /* Post-create CPUless memory tiers after getting HMAT info */ + mt_init_with_hmat(); + return 0; out_put: hmat_free_structures(); diff --git a/include/linux/memory-tiers.h b/include/linux/memory-tiers.h index 69e781900082..2f845e90c033 100644 --- a/include/linux/memory-tiers.h +++ b/include/linux/memory-tiers.h @@ -48,6 +48,7 @@ int mt_calc_adistance(int node, int *adist); int mt_set_default_dram_perf(int nid, struct access_coordinate *perf, const char *source); int mt_perf_to_adistance(struct access_coordinate *perf, int *adist); +void mt_init_with_hmat(void); #ifdef CONFIG_MIGRATION int next_demotion_node(int node); void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *targets); @@ -136,5 +137,10 @@ static inline int mt_perf_to_adistance(struct access_coordinate *perf, int *adis { return -EIO; } + +static inline void mt_init_with_hmat(void) +{ + +} #endif /* CONFIG_NUMA */ #endif /* _LINUX_MEMORY_TIERS_H */ diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c index 0537664620e5..7a0a579b3deb 100644 --- a/mm/memory-tiers.c +++ b/mm/memory-tiers.c @@ -35,7 +35,9 @@ struct node_memory_type_map { }; static DEFINE_MUTEX(memory_tier_lock); +static DEFINE_MUTEX(mt_perf_lock); static LIST_HEAD(memory_tiers); +static LIST_HEAD(hmat_memory_types); static struct node_memory_type_map node_memory_types[MAX_NUMNODES]; struct memory_dev_type *default_dram_type; @@ -502,7 +504,7 @@ static inline void __init_node_memory_type(int node, struct memory_dev_type *mem } } -static struct memory_tier *set_node_memory_tier(int node) +static struct memory_tier *set_node_memory_tier(int node, struct memory_dev_type *new_memtype) { struct memory_tier *memtier; struct memory_dev_type *memtype; @@ -514,7 +516,7 @@ static struct memory_tier *set_node_memory_tier(int node) if (!node_state(node, N_MEMORY)) return ERR_PTR(-EINVAL); - __init_node_memory_type(node, default_dram_type); + __init_node_memory_type(node, new_memtype); memtype = node_memory_types[node].memtype; node_set(node, memtype->nodes); @@ -623,6 +625,56 @@ void clear_node_memory_type(int node, struct memory_dev_type *memtype) } EXPORT_SYMBOL_GPL(clear_node_memory_type); +static struct memory_dev_type *hmat_find_alloc_memory_type(int adist) +{ + bool found = false; + struct memory_dev_type *mtype; + + list_for_each_entry(mtype, &hmat_memory_types, list) { + if (mtype->adistance == adist) { + found = true; + break; + } + } + if (!found) { + mtype = alloc_memory_type(adist); + if (!IS_ERR(mtype)) + list_add(&mtype->list, &hmat_memory_types); + } + return mtype; +} + +static void mt_create_with_hmat(int node) +{ + struct memory_dev_type *mtype = NULL; + int adist = MEMTIER_ADISTANCE_DRAM; + + mt_calc_adistance(node, &adist); + if (adist != MEMTIER_ADISTANCE_DRAM) { + mtype = hmat_find_alloc_memory_type(adist); + if (IS_ERR(mtype)) + pr_err("%s() failed to allocate a tier\n", __func__); + } else { + mtype = default_dram_type; + } + + set_node_memory_tier(node, mtype); +} + +void mt_init_with_hmat(void) +{ + int nid; + + mutex_lock(&memory_tier_lock); + for_each_node_state(nid, N_MEMORY) + if (!node_state(nid, N_CPU)) + mt_create_with_hmat(nid); + + establish_demotion_targets(); + mutex_unlock(&memory_tier_lock); +} +EXPORT_SYMBOL_GPL(mt_init_with_hmat); + static void dump_hmem_attrs(struct access_coordinate *coord, const char *prefix) { pr_info( @@ -636,7 +688,7 @@ int mt_set_default_dram_perf(int nid, struct access_coordinate *perf, { int rc = 0; - mutex_lock(&memory_tier_lock); + mutex_lock(&mt_perf_lock); if (default_dram_perf_error) { rc = -EIO; goto out; @@ -684,7 +736,7 @@ int mt_set_default_dram_perf(int nid, struct access_coordinate *perf, } out: - mutex_unlock(&memory_tier_lock); + mutex_unlock(&mt_perf_lock); return rc; } @@ -700,7 +752,7 @@ int mt_perf_to_adistance(struct access_coordinate *perf, int *adist) perf->read_bandwidth + perf->write_bandwidth == 0) return -EINVAL; - mutex_lock(&memory_tier_lock); + mutex_lock(&mt_perf_lock); /* * The abstract distance of a memory node is in direct proportion to * its memory latency (read + write) and inversely proportional to its @@ -713,7 +765,7 @@ int mt_perf_to_adistance(struct access_coordinate *perf, int *adist) (default_dram_perf.read_latency + default_dram_perf.write_latency) * (default_dram_perf.read_bandwidth + default_dram_perf.write_bandwidth) / (perf->read_bandwidth + perf->write_bandwidth); - mutex_unlock(&memory_tier_lock); + mutex_unlock(&mt_perf_lock); return 0; } @@ -797,7 +849,7 @@ static int __meminit memtier_hotplug_callback(struct notifier_block *self, break; case MEM_ONLINE: mutex_lock(&memory_tier_lock); - memtier = set_node_memory_tier(arg->status_change_nid); + memtier = set_node_memory_tier(arg->status_change_nid, default_dram_type); if (!IS_ERR(memtier)) establish_demotion_targets(); mutex_unlock(&memory_tier_lock); @@ -836,7 +888,15 @@ static int __init memory_tier_init(void) * types assigned. */ for_each_node_state(node, N_MEMORY) { - memtier = set_node_memory_tier(node); + if (!node_state(node, N_CPU)) + /* + * Defer memory tier initialization on CPUless numa nodes. + * These will be initialized when HMAT information is + * available. + */ + continue; + + memtier = set_node_memory_tier(node, default_dram_type); if (IS_ERR(memtier)) /* * Continue with memtiers we are able to setup