From patchwork Tue Jul 23 06:41:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Rapoport X-Patchwork-Id: 13739451 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 06AE0C3DA49 for ; Tue, 23 Jul 2024 06:46:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8CD076B00A7; Tue, 23 Jul 2024 02:46:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 861676B00BD; Tue, 23 Jul 2024 02:46:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6CEFE6B00B0; Tue, 23 Jul 2024 02:46:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 4C9C96B00BD for ; Tue, 23 Jul 2024 02:46:48 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 0726DA1C5B for ; Tue, 23 Jul 2024 06:46:48 +0000 (UTC) X-FDA: 82370084496.19.45F0FE2 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf13.hostedemail.com (Postfix) with ESMTP id 586F420005 for ; Tue, 23 Jul 2024 06:46:46 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=aifPEDXD; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf13.hostedemail.com: domain of rppt@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=rppt@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1721717154; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YwxCyFvpvd65E3n7xZa02al8cyZcfR+th7BShgSPqtI=; b=G6JqjgvRXftK+i9EdJ+mAQGa3HqrZctKx2s3K+3HKO5fOnDmQ8gzgQvijhKUOSYsaCbK8U fTRr4aSEwhTjVxBx4GJUpWWjRtNi4CjshNiSP2LRJl6ncFXPpH/NBySPA/QvAR9K0Ra0xI OYBV30k2qCUJygK4kxSav08iKvsVafk= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1721717154; a=rsa-sha256; cv=none; b=YVHkfKmxQnSqefjEOysNOR+YA8eBvP6ZxIwwLzAXlT1Wuuyi9a2eCdroZ4EST6wNK/phdi oESGQUQ+8bFU1CjpzZmAWCfZYIbvv+7E8s+RhTN4u/WowYHCt/c5QmUC8hxo3DAeKIRX0f mRL/vYg4CYvioBRJ08Smhc+MNztHyNk= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=aifPEDXD; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf13.hostedemail.com: domain of rppt@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=rppt@kernel.org Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 8667460187; Tue, 23 Jul 2024 06:46:45 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3718DC4AF13; Tue, 23 Jul 2024 06:46:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1721717205; bh=Mej9sxLQixWGdmK565gFwBx2ByPl7rtL0f0ZPnxf/S8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=aifPEDXDJfm59czghgzqn6gX2srkwMHjj0MDAYduHgsQYt3LGdewFa3vOyK8Dkumz 1HAs5r4jvJXbol55lz55vqk9ZcNeBu4vBBYJBnf5DTsEBF3Q6tPKOkOPEER240Uw29 4Gk6WKAuZBstvMWa7OmmJIJmjg37tfJXWJ9rKS3Q7lObr6Dd2VwwdoPNdtfsaGsdI2 vV0zrCVw7+zwfqJ/kHdVhStrolfdNSESoJho0dXlPRHtSnDLZia3RoJwCZ9oKUSo6C JEqAEo3ng6ENYfKRpvmqa4razDCj+uioqxOiis3qh54kCu2BTqAMZVkdqr5+j0zQ9z fWe8vNWTVnDlw== From: Mike Rapoport To: linux-kernel@vger.kernel.org Cc: Alexander Gordeev , Andreas Larsson , Andrew Morton , Arnd Bergmann , Borislav Petkov , Catalin Marinas , Christophe Leroy , Dan Williams , Dave Hansen , David Hildenbrand , "David S. Miller" , Davidlohr Bueso , Greg Kroah-Hartman , Heiko Carstens , Huacai Chen , Ingo Molnar , Jiaxun Yang , John Paul Adrian Glaubitz , Jonathan Cameron , Jonathan Corbet , Michael Ellerman , Mike Rapoport , Palmer Dabbelt , "Rafael J. Wysocki" , Rob Herring , Samuel Holland , Thomas Bogendoerfer , Thomas Gleixner , Vasily Gorbik , Will Deacon , Zi Yan , devicetree@vger.kernel.org, linux-acpi@vger.kernel.org, linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-cxl@vger.kernel.org, linux-doc@vger.kernel.org, linux-mips@vger.kernel.org, linux-mm@kvack.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-sh@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, loongarch@lists.linux.dev, nvdimm@lists.linux.dev, sparclinux@vger.kernel.org, x86@kernel.org Subject: [PATCH v2 23/25] arch_numa: switch over to numa_memblks Date: Tue, 23 Jul 2024 09:41:54 +0300 Message-ID: <20240723064156.4009477-24-rppt@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240723064156.4009477-1-rppt@kernel.org> References: <20240723064156.4009477-1-rppt@kernel.org> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 586F420005 X-Stat-Signature: qdg3cu579p4bd1168jm9mygtz4ep41f7 X-Rspam-User: X-HE-Tag: 1721717206-690258 X-HE-Meta: U2FsdGVkX1+a2AiUiZ1qYAGjFigtQ+jhPqqFCJ/qllHuVM5wTO+ZCM8FIjQd/lwnbrWQIa/boy8zLhG1widjJh+VOrEvJ9hXf1gwZrRAonSBbPq6UUM1HDobJneaY/PCOG02qL1uICO+MWsDBTYhpRsh27ltEPur+34OnpiMHN1pCD0OvaVHvDC4A2C5WmHTFLQRtyCsX1LaTyBb+lislMCad1NOii3grtU0vU3XKsY8jJ+4Zj20YZz0skFLd5VZZDW232SC+iXd8nTBCMf4N13gXdsrAklK6KGVU9g5NFLEUenti8lsP5rZOvePd/snsB/Lq1BUM8Us0RXBgAJI19KQzEueKne36CmUl2P+jhiYzUDqy1YC0NlU960IvjyKYYHd1Ic4f830OvUerHViImYY+hBP00n+fxF6kfkq9D3LPL7VpdZaAumuNQciDobFtNgD8q/hkPDuh+s/s5b6ikSyqNP0anOZBS8EYSA95QTEu17fAyFQJ27jz180WJ0uC3I/VImXRv4oKTnBnJ3TDuMY6nyx0QaiXr7KOafwvayv8s1vMTy2MIHNDG6fuJ8zvHAvusNovZ6T8+h6eT35H/W4Re/DmHAcBknvglAlWPmrDne5fWTtsjBJRTlI/WcWRtOe7acdkyEeA6TjRahVV2gFrLSyrqh2LvDX8z4JyK34JCzMrrpGDxm1xHvtqEGPilBmFJOu1ye8MMWw6d3gas1Rkqh4G9FUxT2Ij/3mdMdw7eZkt3w8Pg4ajzB9i49XDe+5YemMbTTJWnmh42331TQJyZYg7X74MaPVEaMjgQNoboLNA5kTFuOAmhUdW/83pwmgJB7cJGQiCckzH8yfLddfmPq18L6AklaamUTbA0yuZOksK1HvOOef9v674kwi303V9T4nUZdzsd9dtRgGQNbJo9bGxzXk3XLuxoL1YvD2/skG+orVX2nndICgjWbbuWVDwmxG2FKQtrZxNri StXNDZes Q9vmv8tG7tkmEs/8Uq1cfsWQoSzvNWtCgdOqhJaoIAE0/c8KPDnQNVdngbneuNTomcrmDUJKOWldIT1pPq4o26ill4HCdz+6hbTEunVlfbF3sK5ZXuEhpRXqJQaEbMDTTukpPRhQioZuwkR0QJlnLRk7+gAAYjUdELfPB/RQtu5AitIMRFtX4X21HtkQBwbU7U1sQ93KxD++17QJ8rasquXjk8Bcj6hePz2M+Ft5IpDcw3vqz2AbLgNiMArPgq2u04bkfwQ7vTycxhpfpf8zy9rfc3TnFRrjrTvXdunBW83WUSYmf10fCxNsytdLa30oGbmIDTRG8QqDsDgY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: "Mike Rapoport (Microsoft)" Until now arch_numa was directly translating firmware NUMA information to memblock. Using numa_memblks as an intermediate step has a few advantages: * alignment with more battle tested x86 implementation * availability of NUMA emulation * maintaining node information for not yet populated memory Replace current functionality related to numa_add_memblk() and __node_distance() with the implementation based on numa_memblks and add functions required by numa_emulation. Signed-off-by: Mike Rapoport (Microsoft) --- drivers/base/Kconfig | 1 + drivers/base/arch_numa.c | 200 +++++++++++-------------------------- include/asm-generic/numa.h | 6 +- 3 files changed, 64 insertions(+), 143 deletions(-) diff --git a/drivers/base/Kconfig b/drivers/base/Kconfig index 2b8fd6bb7da0..064eb52ff7e2 100644 --- a/drivers/base/Kconfig +++ b/drivers/base/Kconfig @@ -226,6 +226,7 @@ config GENERIC_ARCH_TOPOLOGY config GENERIC_ARCH_NUMA bool + select NUMA_MEMBLKS help Enable support for generic NUMA implementation. Currently, RISC-V and ARM64 use it. diff --git a/drivers/base/arch_numa.c b/drivers/base/arch_numa.c index 2ebf12eab99f..806550239d08 100644 --- a/drivers/base/arch_numa.c +++ b/drivers/base/arch_numa.c @@ -12,14 +12,12 @@ #include #include #include +#include #include -nodemask_t numa_nodes_parsed __initdata; static int cpu_to_node_map[NR_CPUS] = { [0 ... NR_CPUS-1] = NUMA_NO_NODE }; -static int numa_distance_cnt; -static u8 *numa_distance; bool numa_off; static __init int numa_parse_early_param(char *opt) @@ -28,6 +26,8 @@ static __init int numa_parse_early_param(char *opt) return -EINVAL; if (str_has_prefix(opt, "off")) numa_off = true; + if (!strncmp(opt, "fake=", 5)) + return numa_emu_cmdline(opt + 5); return 0; } @@ -59,6 +59,7 @@ EXPORT_SYMBOL(cpumask_of_node); #endif +#ifndef CONFIG_NUMA_EMU static void numa_update_cpu(unsigned int cpu, bool remove) { int nid = cpu_to_node(cpu); @@ -81,6 +82,7 @@ void numa_remove_cpu(unsigned int cpu) { numa_update_cpu(cpu, true); } +#endif void numa_clear_node(unsigned int cpu) { @@ -142,7 +144,7 @@ void __init early_map_cpu_to_node(unsigned int cpu, int nid) unsigned long __per_cpu_offset[NR_CPUS] __read_mostly; EXPORT_SYMBOL(__per_cpu_offset); -int __init early_cpu_to_node(int cpu) +int early_cpu_to_node(int cpu) { return cpu_to_node_map[cpu]; } @@ -187,30 +189,6 @@ void __init setup_per_cpu_areas(void) } #endif -/** - * numa_add_memblk() - Set node id to memblk - * @nid: NUMA node ID of the new memblk - * @start: Start address of the new memblk - * @end: End address of the new memblk - * - * RETURNS: - * 0 on success, -errno on failure. - */ -int __init numa_add_memblk(int nid, u64 start, u64 end) -{ - int ret; - - ret = memblock_set_node(start, (end - start), &memblock.memory, nid); - if (ret < 0) { - pr_err("memblock [0x%llx - 0x%llx] failed to add on node %d\n", - start, (end - 1), nid); - return ret; - } - - node_set(nid, numa_nodes_parsed); - return ret; -} - /* * Initialize NODE_DATA for a node on the local memory */ @@ -226,116 +204,9 @@ static void __init setup_node_data(int nid, u64 start_pfn, u64 end_pfn) NODE_DATA(nid)->node_spanned_pages = end_pfn - start_pfn; } -/* - * numa_free_distance - * - * The current table is freed. - */ -void __init numa_free_distance(void) -{ - size_t size; - - if (!numa_distance) - return; - - size = numa_distance_cnt * numa_distance_cnt * - sizeof(numa_distance[0]); - - memblock_free(numa_distance, size); - numa_distance_cnt = 0; - numa_distance = NULL; -} - -/* - * Create a new NUMA distance table. - */ -static int __init numa_alloc_distance(void) -{ - size_t size; - int i, j; - - size = nr_node_ids * nr_node_ids * sizeof(numa_distance[0]); - numa_distance = memblock_alloc(size, PAGE_SIZE); - if (WARN_ON(!numa_distance)) - return -ENOMEM; - - numa_distance_cnt = nr_node_ids; - - /* fill with the default distances */ - for (i = 0; i < numa_distance_cnt; i++) - for (j = 0; j < numa_distance_cnt; j++) - numa_distance[i * numa_distance_cnt + j] = i == j ? - LOCAL_DISTANCE : REMOTE_DISTANCE; - - pr_debug("Initialized distance table, cnt=%d\n", numa_distance_cnt); - - return 0; -} - -/** - * numa_set_distance() - Set inter node NUMA distance from node to node. - * @from: the 'from' node to set distance - * @to: the 'to' node to set distance - * @distance: NUMA distance - * - * Set the distance from node @from to @to to @distance. - * If distance table doesn't exist, a warning is printed. - * - * If @from or @to is higher than the highest known node or lower than zero - * or @distance doesn't make sense, the call is ignored. - */ -void __init numa_set_distance(int from, int to, int distance) -{ - if (!numa_distance) { - pr_warn_once("Warning: distance table not allocated yet\n"); - return; - } - - if (from >= numa_distance_cnt || to >= numa_distance_cnt || - from < 0 || to < 0) { - pr_warn_once("Warning: node ids are out of bound, from=%d to=%d distance=%d\n", - from, to, distance); - return; - } - - if ((u8)distance != distance || - (from == to && distance != LOCAL_DISTANCE)) { - pr_warn_once("Warning: invalid distance parameter, from=%d to=%d distance=%d\n", - from, to, distance); - return; - } - - numa_distance[from * numa_distance_cnt + to] = distance; -} - -/* - * Return NUMA distance @from to @to - */ -int __node_distance(int from, int to) -{ - if (from >= numa_distance_cnt || to >= numa_distance_cnt) - return from == to ? LOCAL_DISTANCE : REMOTE_DISTANCE; - return numa_distance[from * numa_distance_cnt + to]; -} -EXPORT_SYMBOL(__node_distance); - static int __init numa_register_nodes(void) { int nid; - struct memblock_region *mblk; - - /* Check that valid nid is set to memblks */ - for_each_mem_region(mblk) { - int mblk_nid = memblock_get_region_node(mblk); - phys_addr_t start = mblk->base; - phys_addr_t end = mblk->base + mblk->size - 1; - - if (mblk_nid == NUMA_NO_NODE || mblk_nid >= MAX_NUMNODES) { - pr_warn("Warning: invalid memblk node %d [mem %pap-%pap]\n", - mblk_nid, &start, &end); - return -EINVAL; - } - } /* Finally register nodes. */ for_each_node_mask(nid, numa_nodes_parsed) { @@ -360,11 +231,7 @@ static int __init numa_init(int (*init_func)(void)) nodes_clear(node_possible_map); nodes_clear(node_online_map); - ret = numa_alloc_distance(); - if (ret < 0) - return ret; - - ret = init_func(); + ret = numa_memblks_init(init_func, /* memblock_force_top_down */ false); if (ret < 0) goto out_free_distance; @@ -382,7 +249,7 @@ static int __init numa_init(int (*init_func)(void)) return 0; out_free_distance: - numa_free_distance(); + numa_reset_distance(); return ret; } @@ -454,3 +321,54 @@ void __init arch_numa_init(void) numa_init(dummy_numa_init); } + +#ifdef CONFIG_NUMA_EMU +void __init numa_emu_update_cpu_to_node(int *emu_nid_to_phys, + unsigned int nr_emu_nids) +{ + int i, j; + + /* + * Transform cpu_to_node_map table to use emulated nids by + * reverse-mapping phys_nid. The maps should always exist but fall + * back to zero just in case. + */ + for (i = 0; i < ARRAY_SIZE(cpu_to_node_map); i++) { + if (cpu_to_node_map[i] == NUMA_NO_NODE) + continue; + for (j = 0; j < nr_emu_nids; j++) + if (cpu_to_node_map[i] == emu_nid_to_phys[j]) + break; + cpu_to_node_map[i] = j < nr_emu_nids ? j : 0; + } +} + +u64 __init numa_emu_dma_end(void) +{ + return PFN_PHYS(memblock_start_of_DRAM() + SZ_4G); +} + +void debug_cpumask_set_cpu(unsigned int cpu, int node, bool enable) +{ + struct cpumask *mask; + + if (node == NUMA_NO_NODE) + return; + + mask = node_to_cpumask_map[node]; + if (!cpumask_available(mask)) { + pr_err("node_to_cpumask_map[%i] NULL\n", node); + dump_stack(); + return; + } + + if (enable) + cpumask_set_cpu(cpu, mask); + else + cpumask_clear_cpu(cpu, mask); + + pr_debug("%s cpu %d node %d: mask now %*pbl\n", + enable ? "numa_add_cpu" : "numa_remove_cpu", + cpu, node, cpumask_pr_args(mask)); +} +#endif /* CONFIG_NUMA_EMU */ diff --git a/include/asm-generic/numa.h b/include/asm-generic/numa.h index c32e0cf23c90..c2b046d1fd82 100644 --- a/include/asm-generic/numa.h +++ b/include/asm-generic/numa.h @@ -32,8 +32,6 @@ static inline const struct cpumask *cpumask_of_node(int node) void __init arch_numa_init(void); int __init numa_add_memblk(int nodeid, u64 start, u64 end); -void __init numa_set_distance(int from, int to, int distance); -void __init numa_free_distance(void); void __init early_map_cpu_to_node(unsigned int cpu, int nid); int __init early_cpu_to_node(int cpu); void numa_store_cpu_info(unsigned int cpu); @@ -51,4 +49,8 @@ static inline int early_cpu_to_node(int cpu) { return 0; } #endif /* CONFIG_NUMA */ +#ifdef CONFIG_NUMA_EMU +void debug_cpumask_set_cpu(unsigned int cpu, int node, bool enable); +#endif + #endif /* __ASM_GENERIC_NUMA_H */