From patchwork Tue Jul 16 11:13:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Rapoport X-Patchwork-Id: 13734343 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7AFF6C3DA59 for ; Tue, 16 Jul 2024 11:16:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 081766B00AD; Tue, 16 Jul 2024 07:16:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0325E6B00AE; Tue, 16 Jul 2024 07:16:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DC76E6B00AF; Tue, 16 Jul 2024 07:16:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id BDDB86B00AD for ; Tue, 16 Jul 2024 07:16:51 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 7198C81330 for ; Tue, 16 Jul 2024 11:16:51 +0000 (UTC) X-FDA: 82345363422.30.0E24B94 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf07.hostedemail.com (Postfix) with ESMTP id B89B54000C for ; Tue, 16 Jul 2024 11:16:49 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=hbP8A0lE; spf=pass (imf07.hostedemail.com: domain of rppt@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=rppt@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1721128558; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=iBzGPZuMGD++mS9jOpUUPTzrt4xgBt4OXBSQWTSe814=; b=l52OLXYoJXkRZsm0EPIDkiJOKYrStEtiPcvK+q9OlXI7xYUhnrsCmVZQGBdm0Bpd/iL3eS 9+PxJ7jW1Hjq35nCMileSNNBzxoaI9Nez9TM7E2RsNwTrs3dJoWKvaSkNTVPiNtCxZMryX e7xvGVZYDVW+TBfPs93jTrm1P2xv6ow= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=hbP8A0lE; spf=pass (imf07.hostedemail.com: domain of rppt@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=rppt@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1721128558; a=rsa-sha256; cv=none; b=eotFKaSZAT77NKmNJpiVx43XrsbaROUXeXyFHJZuTDUS9CCnQG9hQa+Imc4j4BAmprAagQ 6iErR+mu4ZqhsseCr7JzGkMsfeP0EuJewIUtEekB0y9We4A9q7mIL089cVglrTCG9MgZF2 ep6eLp1RXP/f7Kuapuj7KI4Q2x14ncI= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id F1D9060B9F; Tue, 16 Jul 2024 11:16:48 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0EC11C4AF0D; Tue, 16 Jul 2024 11:16:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1721128608; bh=cw/dvVFpXQw8UFABdX3d8CGu73vGiQNRy5vsiOPnGKU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=hbP8A0lEpqUTNDzt8jAuMvv6yCcB+FkqjrOQXQgUWcNzcZ3+xmIR/Aq1Yfp/okNoV IxtfpfmtCpeARUNWee3xrWyJ+MV8+fqdOnXDXmDSbmggoHhS8hnAQRCM6WDuTW2B8o R6ovcDEBbii5Ob4bL2Mrw5doidNTSFYtK8nUfLb31saiKBa99WjoYVfh0rLKHC3BZa mysjvhMi7gKCDFNWYJfdmsaeFEBXPXyzNpLT8adWtIYuX5w/pzCWdyzxmtYtHzySlk JM0zImcPHXvpx62YrgbS+n47aQzi3RmFdvy8Y0k7NX2M73TkKaURcJbMZJrXC8nNrw ftINNXlBfq8ig== From: Mike Rapoport To: linux-kernel@vger.kernel.org Cc: Alexander Gordeev , Andreas Larsson , Andrew Morton , Arnd Bergmann , Borislav Petkov , Catalin Marinas , Christophe Leroy , Dan Williams , Dave Hansen , David Hildenbrand , "David S. Miller" , Greg Kroah-Hartman , Heiko Carstens , Huacai Chen , Ingo Molnar , Jiaxun Yang , John Paul Adrian Glaubitz , Jonathan Cameron , Michael Ellerman , Mike Rapoport , Palmer Dabbelt , "Rafael J. Wysocki" , Rob Herring , Thomas Bogendoerfer , Thomas Gleixner , Vasily Gorbik , Will Deacon , linux-arm-kernel@lists.infradead.org, loongarch@lists.linux.dev, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-sh@vger.kernel.org, sparclinux@vger.kernel.org, linux-acpi@vger.kernel.org, linux-cxl@vger.kernel.org, nvdimm@lists.linux.dev, devicetree@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Subject: [PATCH 16/17] arch_numa: switch over to numa_memblks Date: Tue, 16 Jul 2024 14:13:45 +0300 Message-ID: <20240716111346.3676969-17-rppt@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240716111346.3676969-1-rppt@kernel.org> References: <20240716111346.3676969-1-rppt@kernel.org> MIME-Version: 1.0 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: B89B54000C X-Stat-Signature: e7jh4hua4gr9pcnc8zzowm9y3eba3yz7 X-Rspam-User: X-HE-Tag: 1721128609-140972 X-HE-Meta: U2FsdGVkX187ZzzuoByXlMgXRhEMy/WeFrhQ3vTugLag1bU1PhWIPXEX1QD8G+fLr9BZP12YXt2i9HbO5LaoerxAYRnCdhqDbcUl6M54OrP/UiCm4CyjuYPQqfZCUJfVw9eqSqAgJisbCnkWNyJEuYfH6jxVeGt0H5diJM/ZW79u4XTx6mHXsNQqJQdewuLMOHcGuoGs5vgZdUQ/axSkhcznEmW/rKldd/NvQ7a6Pf/Obf7YMIcIqN6uk7qPWGqVKCz6qcdq8xHtNtpiwR8gscQxcewcyVzCEjKCQCohiickzn075BnuRBamxs9McaRhIraECs7K+LpAIfcYxi6rcmt5TeHMCbGcNMACtlup8oo9kUsTCLypWS+iwg2m/k4i2lDKWA1nvk0pB14GWx41OH4LDDcIl9GDJ+2ljKyxEkeyUqB2cjfBBTk6sBoQ+t4UsK34jlD1ZJxEusflndvTQXC3ncN/Rf0uPWv0kT9eDcTUT83KeuHOmuWiIp8icJDP7/ipcj7Ip3YKFq9c29Rjf0krsWwK6EWhPhjECfcHzqj5FG8CJSzqiB4WgxOJUu/oBcYwHoefhoiJUEC7UXW0Pq4pzVgwVN4TpGmZw8CCt4Ax0JIf8z7UfKHkw+9oqtJtWf+HDnpJGCR6ULe3NEFqNjitRmk8VFE1R5TFa5pbLp1OLof+feSFuivGy/EK2NhDvV3Uhqthw58Jae8umRx+8Jkuo/4bYjvVtbIY2twLEoZtIP4d5TDOuXHkU3PCs0bXvm/pIcSEYZ3qjWJU6RuBFIMqgoGS8ASZzia6h9Zh75Mvqc/y3vsauqR6Ds6j9jC4OhSOYLmHPHFxX6Rp0eeKj+RvzGutEesOr4wJm7JnRqs58HOUaC/ILJalSAV4zyv7Pp1mOPoqlDEOks8H6PzdQpBkm44V2pLWAw04XNwl13MH/VWCwxU6d2VRCwRA9ydWVtiEk4JEZiX/ZgR0wtG r9ffdK1m wWKvtGslHr8xbmHZ9vXChNhTYoBF4Bis0Rsh+TPH/+a9uMx6BZMb9vPSTM+iDvKtzQxoWOgdycNhiRrw3oqS1T8gGla09zswx7f0R9HGmxTLybG8FiciqsX1EZ4hnQbbyczTsBK4U5BIzEcVHUALSu69mqskFQQBJQRn7jsDF38KDcvFQ63FfyQxVATE5AEsenn+KXGiVEsegngZBGBE3IGizfcV9k5Xd9caBjRjiwsXHoTrOsF5OcV8QkLDf+VlACiFHENeryWa54oOaocg9IVk3EUet1+P1kw2c X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: "Mike Rapoport (Microsoft)" Until now arch_numa was directly translating firmware NUMA information to memblock. Using numa_memblks as an intermediate step has a few advantages: * alignment with more battle tested x86 implementation * availability of NUMA emulation * maintaining node information for not yet populated memory Replace current functionality related to numa_add_memblk() and __node_distance() with the implementation based on numa_memblks and add functions required by numa_emulation. Signed-off-by: Mike Rapoport (Microsoft) --- drivers/base/Kconfig | 1 + drivers/base/arch_numa.c | 200 +++++++++++-------------------------- include/asm-generic/numa.h | 6 +- 3 files changed, 64 insertions(+), 143 deletions(-) diff --git a/drivers/base/Kconfig b/drivers/base/Kconfig index 2b8fd6bb7da0..064eb52ff7e2 100644 --- a/drivers/base/Kconfig +++ b/drivers/base/Kconfig @@ -226,6 +226,7 @@ config GENERIC_ARCH_TOPOLOGY config GENERIC_ARCH_NUMA bool + select NUMA_MEMBLKS help Enable support for generic NUMA implementation. Currently, RISC-V and ARM64 use it. diff --git a/drivers/base/arch_numa.c b/drivers/base/arch_numa.c index 2ebf12eab99f..333cbbad7466 100644 --- a/drivers/base/arch_numa.c +++ b/drivers/base/arch_numa.c @@ -12,14 +12,12 @@ #include #include #include +#include #include -nodemask_t numa_nodes_parsed __initdata; static int cpu_to_node_map[NR_CPUS] = { [0 ... NR_CPUS-1] = NUMA_NO_NODE }; -static int numa_distance_cnt; -static u8 *numa_distance; bool numa_off; static __init int numa_parse_early_param(char *opt) @@ -28,6 +26,8 @@ static __init int numa_parse_early_param(char *opt) return -EINVAL; if (str_has_prefix(opt, "off")) numa_off = true; + if (!strncmp(opt, "fake=", 5)) + return numa_emu_cmdline(opt + 5); return 0; } @@ -59,6 +59,7 @@ EXPORT_SYMBOL(cpumask_of_node); #endif +#ifndef CONFIG_NUMA_EMU static void numa_update_cpu(unsigned int cpu, bool remove) { int nid = cpu_to_node(cpu); @@ -81,6 +82,7 @@ void numa_remove_cpu(unsigned int cpu) { numa_update_cpu(cpu, true); } +#endif void numa_clear_node(unsigned int cpu) { @@ -142,7 +144,7 @@ void __init early_map_cpu_to_node(unsigned int cpu, int nid) unsigned long __per_cpu_offset[NR_CPUS] __read_mostly; EXPORT_SYMBOL(__per_cpu_offset); -int __init early_cpu_to_node(int cpu) +int early_cpu_to_node(int cpu) { return cpu_to_node_map[cpu]; } @@ -187,30 +189,6 @@ void __init setup_per_cpu_areas(void) } #endif -/** - * numa_add_memblk() - Set node id to memblk - * @nid: NUMA node ID of the new memblk - * @start: Start address of the new memblk - * @end: End address of the new memblk - * - * RETURNS: - * 0 on success, -errno on failure. - */ -int __init numa_add_memblk(int nid, u64 start, u64 end) -{ - int ret; - - ret = memblock_set_node(start, (end - start), &memblock.memory, nid); - if (ret < 0) { - pr_err("memblock [0x%llx - 0x%llx] failed to add on node %d\n", - start, (end - 1), nid); - return ret; - } - - node_set(nid, numa_nodes_parsed); - return ret; -} - /* * Initialize NODE_DATA for a node on the local memory */ @@ -226,116 +204,9 @@ static void __init setup_node_data(int nid, u64 start_pfn, u64 end_pfn) NODE_DATA(nid)->node_spanned_pages = end_pfn - start_pfn; } -/* - * numa_free_distance - * - * The current table is freed. - */ -void __init numa_free_distance(void) -{ - size_t size; - - if (!numa_distance) - return; - - size = numa_distance_cnt * numa_distance_cnt * - sizeof(numa_distance[0]); - - memblock_free(numa_distance, size); - numa_distance_cnt = 0; - numa_distance = NULL; -} - -/* - * Create a new NUMA distance table. - */ -static int __init numa_alloc_distance(void) -{ - size_t size; - int i, j; - - size = nr_node_ids * nr_node_ids * sizeof(numa_distance[0]); - numa_distance = memblock_alloc(size, PAGE_SIZE); - if (WARN_ON(!numa_distance)) - return -ENOMEM; - - numa_distance_cnt = nr_node_ids; - - /* fill with the default distances */ - for (i = 0; i < numa_distance_cnt; i++) - for (j = 0; j < numa_distance_cnt; j++) - numa_distance[i * numa_distance_cnt + j] = i == j ? - LOCAL_DISTANCE : REMOTE_DISTANCE; - - pr_debug("Initialized distance table, cnt=%d\n", numa_distance_cnt); - - return 0; -} - -/** - * numa_set_distance() - Set inter node NUMA distance from node to node. - * @from: the 'from' node to set distance - * @to: the 'to' node to set distance - * @distance: NUMA distance - * - * Set the distance from node @from to @to to @distance. - * If distance table doesn't exist, a warning is printed. - * - * If @from or @to is higher than the highest known node or lower than zero - * or @distance doesn't make sense, the call is ignored. - */ -void __init numa_set_distance(int from, int to, int distance) -{ - if (!numa_distance) { - pr_warn_once("Warning: distance table not allocated yet\n"); - return; - } - - if (from >= numa_distance_cnt || to >= numa_distance_cnt || - from < 0 || to < 0) { - pr_warn_once("Warning: node ids are out of bound, from=%d to=%d distance=%d\n", - from, to, distance); - return; - } - - if ((u8)distance != distance || - (from == to && distance != LOCAL_DISTANCE)) { - pr_warn_once("Warning: invalid distance parameter, from=%d to=%d distance=%d\n", - from, to, distance); - return; - } - - numa_distance[from * numa_distance_cnt + to] = distance; -} - -/* - * Return NUMA distance @from to @to - */ -int __node_distance(int from, int to) -{ - if (from >= numa_distance_cnt || to >= numa_distance_cnt) - return from == to ? LOCAL_DISTANCE : REMOTE_DISTANCE; - return numa_distance[from * numa_distance_cnt + to]; -} -EXPORT_SYMBOL(__node_distance); - static int __init numa_register_nodes(void) { int nid; - struct memblock_region *mblk; - - /* Check that valid nid is set to memblks */ - for_each_mem_region(mblk) { - int mblk_nid = memblock_get_region_node(mblk); - phys_addr_t start = mblk->base; - phys_addr_t end = mblk->base + mblk->size - 1; - - if (mblk_nid == NUMA_NO_NODE || mblk_nid >= MAX_NUMNODES) { - pr_warn("Warning: invalid memblk node %d [mem %pap-%pap]\n", - mblk_nid, &start, &end); - return -EINVAL; - } - } /* Finally register nodes. */ for_each_node_mask(nid, numa_nodes_parsed) { @@ -360,11 +231,7 @@ static int __init numa_init(int (*init_func)(void)) nodes_clear(node_possible_map); nodes_clear(node_online_map); - ret = numa_alloc_distance(); - if (ret < 0) - return ret; - - ret = init_func(); + ret = numa_memblks_init(init_func, /* memblock_force_top_down */ false); if (ret < 0) goto out_free_distance; @@ -382,7 +249,7 @@ static int __init numa_init(int (*init_func)(void)) return 0; out_free_distance: - numa_free_distance(); + numa_reset_distance(); return ret; } @@ -454,3 +321,54 @@ void __init arch_numa_init(void) numa_init(dummy_numa_init); } + +#ifdef CONFIG_NUMA_EMU +void __init numa_emu_update_cpu_to_node(int *emu_nid_to_phys, + unsigned int nr_emu_nids) +{ + int i, j; + + /* + * Transform __apicid_to_node table to use emulated nids by + * reverse-mapping phys_nid. The maps should always exist but fall + * back to zero just in case. + */ + for (i = 0; i < ARRAY_SIZE(cpu_to_node_map); i++) { + if (cpu_to_node_map[i] == NUMA_NO_NODE) + continue; + for (j = 0; j < nr_emu_nids; j++) + if (cpu_to_node_map[i] == emu_nid_to_phys[j]) + break; + cpu_to_node_map[i] = j < nr_emu_nids ? j : 0; + } +} + +u64 __init numa_emu_dma_end(void) +{ + return PFN_PHYS(memblock_start_of_DRAM() + SZ_4G); +} + +void debug_cpumask_set_cpu(unsigned int cpu, int node, bool enable) +{ + struct cpumask *mask; + + if (node == NUMA_NO_NODE) + return; + + mask = node_to_cpumask_map[node]; + if (!cpumask_available(mask)) { + pr_err("node_to_cpumask_map[%i] NULL\n", node); + dump_stack(); + return; + } + + if (enable) + cpumask_set_cpu(cpu, mask); + else + cpumask_clear_cpu(cpu, mask); + + pr_debug("%s cpu %d node %d: mask now %*pbl\n", + enable ? "numa_add_cpu" : "numa_remove_cpu", + cpu, node, cpumask_pr_args(mask)); +} +#endif /* CONFIG_NUMA_EMU */ diff --git a/include/asm-generic/numa.h b/include/asm-generic/numa.h index c32e0cf23c90..c2b046d1fd82 100644 --- a/include/asm-generic/numa.h +++ b/include/asm-generic/numa.h @@ -32,8 +32,6 @@ static inline const struct cpumask *cpumask_of_node(int node) void __init arch_numa_init(void); int __init numa_add_memblk(int nodeid, u64 start, u64 end); -void __init numa_set_distance(int from, int to, int distance); -void __init numa_free_distance(void); void __init early_map_cpu_to_node(unsigned int cpu, int nid); int __init early_cpu_to_node(int cpu); void numa_store_cpu_info(unsigned int cpu); @@ -51,4 +49,8 @@ static inline int early_cpu_to_node(int cpu) { return 0; } #endif /* CONFIG_NUMA */ +#ifdef CONFIG_NUMA_EMU +void debug_cpumask_set_cpu(unsigned int cpu, int node, bool enable); +#endif + #endif /* __ASM_GENERIC_NUMA_H */