From patchwork Tue Jul 18 11:41:36 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vijay Kilari X-Patchwork-Id: 9848097 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 62E9C60392 for ; Tue, 18 Jul 2017 11:45:22 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4F7B327480 for ; Tue, 18 Jul 2017 11:45:22 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 43D9D2857B; Tue, 18 Jul 2017 11:45:22 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.6 required=2.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED, FREEMAIL_FROM, RCVD_IN_DNSWL_MED, RCVD_IN_SORBS_WEB, T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id A18BC27480 for ; Tue, 18 Jul 2017 11:45:21 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dXQuK-00072W-BN; Tue, 18 Jul 2017 11:43:16 +0000 Received: from mail6.bemta5.messagelabs.com ([195.245.231.135]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dXQuJ-0006yE-2Y for xen-devel@lists.xen.org; Tue, 18 Jul 2017 11:43:15 +0000 Received: from [85.158.139.211] by server-6.bemta-5.messagelabs.com id FE/03-02175-254FD695; Tue, 18 Jul 2017 11:43:14 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFlrDIsWRWlGSWpSXmKPExsVyMfTAId3AL7m RBk+2aVos+biYxYHR4+ju30wBjFGsmXlJ+RUJrBmbf85mLThsX7GufyprA+NS/S5GLg4hgYmM Ek27N7OBOCwCL1kkPk98DuZICPSzStza/5e5i5ETyImT+Nm3mbWLkQPIrpC4tykEJCwkoCrR8 f8tO8SkNiaJ8+s3soEk2ARkJI48P8AEYosISEtc+3yZEaSIWWAbk8Sm61PZQRLCAj4SO5c9Zw WxWYAmvdx8D2wZr4CbxJ1rBxghlilIzJlkAxLmBAov6rrEDrHYVeJG30rWCYwCCxgZVjFqFKc WlaUW6Rob6CUVZaZnlOQmZuboGhqY6uWmFhcnpqfmJCYV6yXn525iBIZWPQMD4w7GCav8DjFK cjApifKuPpAbKcSXlJ9SmZFYnBFfVJqTWnyIUYaDQ0mCl+0zUE6wKDU9tSItMwcY5DBpCQ4eJ RHem5+A0rzFBYm5xZnpEKlTjJYcV66s+8LEMeXAdiD5asL/b0xCLHn5ealS4ryiIPMEQBoySv PgxsEi8RKjrJQwLyMDA4MQT0FqUW5mCar8K0ZxDkYlYd51IGt5MvNK4La+AjqICeggYd8ckIN KEhFSUg2Mi2fP+JlzUqbtQ3TcPUuf8s+n/wZ2O3n8S1h39/M0L8mHXyeo69mcTXoU5ZnIfniq X4yb32TzFQ/LLQq3a9fWTrB0KpA1erM5VveK4LqtOfnCS2TO3WHQcr4RlpvxTm/LLAfF16qTG kOfHPzfdLvh5/SFquv0WszPRilvKOg+X6du/iSgNFdXiaU4I9FQi7moOBEAGJB3Er8CAAA= X-Env-Sender: vijay.kilari@gmail.com X-Msg-Ref: server-7.tower-206.messagelabs.com!1500378192!100501007!1 X-Originating-IP: [209.85.192.194] X-SpamReason: No, hits=0.0 required=7.0 tests= X-StarScan-Received: X-StarScan-Version: 9.4.25; banners=-,-,- X-VirusChecked: Checked Received: (qmail 11081 invoked from network); 18 Jul 2017 11:43:13 -0000 Received: from mail-pf0-f194.google.com (HELO mail-pf0-f194.google.com) (209.85.192.194) by server-7.tower-206.messagelabs.com with AES128-GCM-SHA256 encrypted SMTP; 18 Jul 2017 11:43:13 -0000 Received: by mail-pf0-f194.google.com with SMTP id o88so2408651pfk.1 for ; Tue, 18 Jul 2017 04:43:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=/61W2CnaKmqXb+TzS3PefC2p4Ojb14+LMJAKOEpuimI=; b=n6WtNmE1HrrwEuX+WpXoutLWt/nJVhcbU3qhtMTgHOVIB4aGm3IgdKWG9FBn+Rnx+4 EJoCea7tWngzIDk8jrVcGJsQ00G/0u9zZ0Pex2sOOKpKJ538GZGTpUATnraKeHThoGt8 a566CJ54wycC+T4ImMFZLHOT/0SXHJCQD/gt1XeamCfypQ8/yBxTP1dmNqgkP7FHSIvC GavraSvvsDX8JwMVfrpdOXryXjVjI52fdiA6fxudnxk/goGWNeurR3GR8KQ4LmTyde7x lmi9UkSbOXbTn/nGZgJYjswsT7R3DWHk2ZW0rpCn7t3inxA6iYZQ7DCucoJFgqERyBOP /w6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=/61W2CnaKmqXb+TzS3PefC2p4Ojb14+LMJAKOEpuimI=; b=TxUik4BLpZ/Fzl/O23g/q4atK+hPUUYOSQPdGmqG/gT6GDkKNh37unKqxR7UyGFT1G fPjihfX40qiykH/JHrOqZbGgYIF1lS3vaK76hkyDk+06JjKs3m2xIUfFT0k9UgJ316YG 7vYXgKFuSDVcUbQd8bb9fCnJJabR3KR8J6RltkuWzlqCYEcDFarFnmrJeiP3fIq55/vA gSYDZkA/4W331CEKJ7nYJxCZ5Ji/PaTDVAZj/gLe/OSh9/TUtBraTin1mH79sf0OlrbI 8NIMGufOpdespVs0WMKOvzWpusjiiZGjTvfubeqhbKhoFC7RsYjeEaqtgNrRPHbTQqz/ waug== X-Gm-Message-State: AIVw111MxC7Uh9Kcau1u5ltt/2kbceUNrV4lLiquBz4Updq8/ltNex1+ WcYJ3hqJ37OYdy1M X-Received: by 10.98.200.24 with SMTP id z24mr655740pff.205.1500378191694; Tue, 18 Jul 2017 04:43:11 -0700 (PDT) Received: from cavium-Vostro-2520.caveonetworks.com ([111.93.218.67]) by smtp.gmail.com with ESMTPSA id r27sm4879707pfe.0.2017.07.18.04.43.07 (version=TLS1_1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 18 Jul 2017 04:43:11 -0700 (PDT) From: vijay.kilari@gmail.com To: xen-devel@lists.xen.org Date: Tue, 18 Jul 2017 17:11:36 +0530 Message-Id: <1500378106-2620-15-git-send-email-vijay.kilari@gmail.com> X-Mailer: git-send-email 1.7.9.5 In-Reply-To: <1500378106-2620-1-git-send-email-vijay.kilari@gmail.com> References: <1500378106-2620-1-git-send-email-vijay.kilari@gmail.com> Cc: kevin.tian@intel.com, sstabellini@kernel.org, wei.liu2@citrix.com, George.Dunlap@eu.citrix.com, andrew.cooper3@citrix.com, dario.faggioli@citrix.com, ian.jackson@eu.citrix.com, tim@xen.org, julien.grall@arm.com, jbeulich@suse.com, Vijaya Kumar K Subject: [Xen-devel] [RFC PATCH v3 14/24] ARM: NUMA: DT: Parse NUMA distance information X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: Vijaya Kumar K Parse distance-matrix and fetch node distance information. Store distance information in node_distance[]. Register dt_node_distance() function pointer with the ARM numa code. This approach can be later used for ACPI. Signed-off-by: Vijaya Kumar K --- v3: - Moved __node_distance() declaration to common header file - Use device_tree_node_compatible() instead of device_tree_node_matches() - Dropped xen/errno.h inclusion --- xen/arch/arm/numa/dt_numa.c | 131 ++++++++++++++++++++++++++++++++++++++++++++ xen/arch/arm/numa/numa.c | 22 ++++++++ xen/include/asm-arm/numa.h | 2 + xen/include/asm-x86/numa.h | 1 - xen/include/xen/numa.h | 3 + 5 files changed, 158 insertions(+), 1 deletion(-) diff --git a/xen/arch/arm/numa/dt_numa.c b/xen/arch/arm/numa/dt_numa.c index 84030e7..46c0346 100644 --- a/xen/arch/arm/numa/dt_numa.c +++ b/xen/arch/arm/numa/dt_numa.c @@ -23,6 +23,48 @@ #include #include +static uint8_t node_distance[MAX_NUMNODES][MAX_NUMNODES]; + +static uint8_t dt_node_distance(nodeid_t nodea, nodeid_t nodeb) +{ + if ( nodea >= MAX_NUMNODES || nodeb >= MAX_NUMNODES ) + return nodea == nodeb ? LOCAL_DISTANCE : REMOTE_DISTANCE; + + return node_distance[nodea][nodeb]; +} + +static int dt_numa_set_distance(uint32_t nodea, uint32_t nodeb, + uint32_t distance) +{ + /* node_distance is uint8_t. Ensure distance is less than 255 */ + if ( nodea >= MAX_NUMNODES || nodeb >= MAX_NUMNODES || distance > 255 ) + return -EINVAL; + + node_distance[nodea][nodeb] = distance; + + return 0; +} + +void init_dt_numa_distance(void) +{ + int i, j; + + for ( i = 0; i < MAX_NUMNODES; i++ ) + { + for ( j = 0; j < MAX_NUMNODES; j++ ) + { + /* + * Initialize distance 10 for local distance and + * 20 for remote distance. + */ + if ( i == j ) + node_distance[i][j] = LOCAL_DISTANCE; + else + node_distance[i][j] = REMOTE_DISTANCE; + } + } +} + /* * Even though we connect cpus to numa domains later in SMP * init, we need to know the node ids now for all cpus. @@ -58,6 +100,76 @@ static int __init dt_numa_process_cpu_node(const void *fdt) return 0; } +static int __init dt_numa_parse_distance_map(const void *fdt, int node, + const char *name, + uint32_t address_cells, + uint32_t size_cells) +{ + const struct fdt_property *prop; + const __be32 *matrix; + int entry_count, len, i; + + printk(XENLOG_INFO "NUMA: parsing numa-distance-map\n"); + + prop = fdt_get_property(fdt, node, "distance-matrix", &len); + if ( !prop ) + { + printk(XENLOG_WARNING + "NUMA: No distance-matrix property in distance-map\n"); + + return -EINVAL; + } + + if ( len % sizeof(uint32_t) != 0 ) + { + printk(XENLOG_WARNING + "distance-matrix in node is not a multiple of u32\n"); + + return -EINVAL; + } + + entry_count = len / sizeof(uint32_t); + if ( entry_count <= 0 ) + { + printk(XENLOG_WARNING "NUMA: Invalid distance-matrix\n"); + + return -EINVAL; + } + + matrix = (const __be32 *)prop->data; + for ( i = 0; i + 2 < entry_count; i += 3 ) + { + uint32_t nodea, nodeb, distance; + + nodea = dt_read_number(matrix, 1); + matrix++; + nodeb = dt_read_number(matrix, 1); + matrix++; + distance = dt_read_number(matrix, 1); + matrix++; + + if ( dt_numa_set_distance(nodea, nodeb, distance) ) + { + printk(XENLOG_WARNING + "NUMA: node-id out of range in distance matrix for [node%d -> node%d]\n", + nodea, nodeb); + return -EINVAL; + + } + printk(XENLOG_INFO "NUMA: distance[node%d -> node%d] = %d\n", + nodea, nodeb, distance); + + /* + * Set default distance of node B->A same as A->B. + * No need to check for return value of numa_set_distance. + */ + if ( nodeb > nodea ) + dt_numa_set_distance(nodeb, nodea, distance); + } + + return 0; +} + void __init dt_numa_process_memory_node(uint32_t nid, paddr_t start, paddr_t size) { @@ -90,11 +202,30 @@ void __init dt_numa_process_memory_node(uint32_t nid, paddr_t start, return; } +static int __init dt_numa_scan_distance_node(const void *fdt, int node, + const char *name, int depth, + uint32_t address_cells, + uint32_t size_cells, void *data) +{ + if ( device_tree_node_compatible(fdt, node, "numa-distance-map-v1") ) + return dt_numa_parse_distance_map(fdt, node, name, address_cells, + size_cells); + + return 0; +} + int __init dt_numa_init(void) { int ret; ret = dt_numa_process_cpu_node((void *)device_tree_flattened); + if ( ret ) + return ret; + + ret = device_tree_for_each_node((void *)device_tree_flattened, + dt_numa_scan_distance_node, NULL); + if ( !ret ) + register_node_distance(&dt_node_distance); return ret; } diff --git a/xen/arch/arm/numa/numa.c b/xen/arch/arm/numa/numa.c index 8227361..c00b92c 100644 --- a/xen/arch/arm/numa/numa.c +++ b/xen/arch/arm/numa/numa.c @@ -18,10 +18,30 @@ #include #include #include +#include + +static uint8_t (*node_distance_fn)(nodeid_t a, nodeid_t b); void numa_failed(void) { numa_off = true; + init_dt_numa_distance(); + node_distance_fn = NULL; +} + +uint8_t __node_distance(nodeid_t a, nodeid_t b) +{ + if ( node_distance_fn != NULL); + return node_distance_fn(a, b); + + return a == b ? LOCAL_DISTANCE : REMOTE_DISTANCE; +} + +EXPORT_SYMBOL(__node_distance); + +void register_node_distance(uint8_t (fn)(nodeid_t a, nodeid_t b)) +{ + node_distance_fn = fn; } void __init numa_init(void) @@ -29,6 +49,8 @@ void __init numa_init(void) int ret = 0; nodes_clear(processor_nodes_parsed); + init_dt_numa_distance(); + if ( numa_off ) goto no_numa; diff --git a/xen/include/asm-arm/numa.h b/xen/include/asm-arm/numa.h index 36cd782..d1dc83a 100644 --- a/xen/include/asm-arm/numa.h +++ b/xen/include/asm-arm/numa.h @@ -4,6 +4,8 @@ typedef uint8_t nodeid_t; void dt_numa_process_memory_node(uint32_t nid, paddr_t start, paddr_t size); +void register_node_distance(uint8_t (fn)(nodeid_t a, nodeid_t b)); +void init_dt_numa_distance(void); #ifdef CONFIG_NUMA void numa_init(void); diff --git a/xen/include/asm-x86/numa.h b/xen/include/asm-x86/numa.h index d8a0a44..ca0a2a6 100644 --- a/xen/include/asm-x86/numa.h +++ b/xen/include/asm-x86/numa.h @@ -18,7 +18,6 @@ extern nodeid_t apicid_to_node[]; extern void init_cpu_to_node(void); void srat_parse_regions(paddr_t addr); -extern uint8_t __node_distance(nodeid_t a, nodeid_t b); unsigned int arch_get_dma_bitsize(void); #endif diff --git a/xen/include/xen/numa.h b/xen/include/xen/numa.h index 110d5dc..10ef4c4 100644 --- a/xen/include/xen/numa.h +++ b/xen/include/xen/numa.h @@ -6,6 +6,8 @@ #include #define NUMA_NO_NODE 0xFF +#define LOCAL_DISTANCE 10 +#define REMOTE_DISTANCE 20 #define NUMA_NO_DISTANCE 0xFF #define MAX_NUMNODES NR_NODES @@ -70,6 +72,7 @@ int numa_add_memblk(nodeid_t nodeid, paddr_t start, uint64_t size); int get_num_node_memblks(void); bool arch_sanitize_nodes_memory(void); void numa_failed(void); +uint8_t __node_distance(nodeid_t a, nodeid_t b); #else static inline void numa_add_cpu(int cpu) { } static inline void numa_set_node(int cpu, nodeid_t node) { }