From patchwork Thu Apr 20 05:19:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yury Norov X-Patchwork-Id: 13217853 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4B032C77B7F for ; Thu, 20 Apr 2023 05:19:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232593AbjDTFTz (ORCPT ); Thu, 20 Apr 2023 01:19:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36286 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231944AbjDTFTy (ORCPT ); Thu, 20 Apr 2023 01:19:54 -0400 Received: from mail-pf1-x430.google.com (mail-pf1-x430.google.com [IPv6:2607:f8b0:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2AF901FE4; Wed, 19 Apr 2023 22:19:53 -0700 (PDT) Received: by mail-pf1-x430.google.com with SMTP id d2e1a72fcca58-63b64a32fd2so835263b3a.2; Wed, 19 Apr 2023 22:19:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1681967992; x=1684559992; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Yl+Y/+i+a8CEnavFpm7U7sXD4kX1xop+/VNm8WF14LE=; b=WV6LQEdx3aZMTz9ZBq6DSlLBb4pLKYIYOV11XYbeJt3xSsRVFYHtq5ggRX4L/sywoA pUYJG1ugEV9TTz94wRi3y0P8aj0nmIFPUD87jNz1y9EBXoN45bzM5cxq970FkqwW8qvv 0ibrVrkHRlneL4Lzu9ICuvUCJ2giIuXU4T4B9f8uiJGy6ZBgBDB22SF4gY33M4IORIct UPqm/ABNlEq1v1hXP4ay3x3OVmLib2YrRZRzpWzBbs8iPN6q3SIqYNMRY5KQ6OUSUqIZ OwoULCaMzA9JdLmXl2xkpdZUezUSvIA1RPXxCNWSv9fKVdJPDOHCPM0aatI2A8fKZZ9m Mzpg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681967992; x=1684559992; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Yl+Y/+i+a8CEnavFpm7U7sXD4kX1xop+/VNm8WF14LE=; b=YCi1/sWST22SiJclX/S35qmxpLCd/OQttP/A3tKAzf5fGcNQuK1EpJAjKr1WIQu33Y VJEDGTSMAagFeposSgeDQkm4NJWs9v1omVXJ+vKD1QMg8ZZ8aF9p5RYgBOo1n+3BDBiG SXAj+87xHd55OMl/RwciufCoR/0HNBloXVzzd8m0cBkcorvHtrOTZgIeO0rF4WHYSmrs OZ6D9lOOQEvZUAwEty+O7sy95f28kHA8nUY2LEXDNVtjq/qBBmsHQkrkUBaP+A47PK6c LPEHzzWu1iCY3joE6JsKz+Dlv9svFn+Ie90yxw8c2ruVkDmb8Ux+125mTz9jhk3E6zvr uI3w== X-Gm-Message-State: AAQBX9eo3Cq+IehH63/rozutsLN4pVUf4CSPv4iJJj8VK/4XXopOAz6V ziPsn9FGvcFSFum1QoixHGY= X-Google-Smtp-Source: AKy350bvOHRSZA/k/jeKOotwidMo14NW5H1n6tHobQF5b3jerKc62WuXaMfCCIDHR/cjcc51Bi57hA== X-Received: by 2002:a05:6a20:3ca2:b0:d9:9d04:2c73 with SMTP id b34-20020a056a203ca200b000d99d042c73mr452095pzj.45.1681967992446; Wed, 19 Apr 2023 22:19:52 -0700 (PDT) Received: from localhost ([2603:3024:e02:8500:653b:861d:e1ca:16ac]) by smtp.gmail.com with ESMTPSA id b23-20020a62a117000000b0063b64f1d6e9sm309903pff.33.2023.04.19.22.19.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Apr 2023 22:19:51 -0700 (PDT) From: Yury Norov To: Jakub Kicinski , netdev@vger.kernel.org, linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Yury Norov , Saeed Mahameed , Pawel Chmielewski , Leon Romanovsky , "David S. Miller" , Eric Dumazet , Paolo Abeni , Andy Shevchenko , Rasmus Villemoes , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira , Valentin Schneider , Tariq Toukan , Gal Pressman , Greg Kroah-Hartman , Heiko Carstens , Barry Song Subject: [PATCH v2 1/8] lib/find: add find_next_and_andnot_bit() Date: Wed, 19 Apr 2023 22:19:39 -0700 Message-Id: <20230420051946.7463-2-yury.norov@gmail.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20230420051946.7463-1-yury.norov@gmail.com> References: <20230420051946.7463-1-yury.norov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Similarly to find_nth_and_andnot_bit(), find_next_and_andnot_bit() is a convenient helper that allows traversing bitmaps without storing intermediate results in a temporary bitmap. In the following patches the function is used to implement NUMA-aware CPUs enumeration. Signed-off-by: Yury Norov --- include/linux/find.h | 43 +++++++++++++++++++++++++++++++++++++++++++ lib/find_bit.c | 12 ++++++++++++ 2 files changed, 55 insertions(+) diff --git a/include/linux/find.h b/include/linux/find.h index 5e4f39ef2e72..90b68d76c073 100644 --- a/include/linux/find.h +++ b/include/linux/find.h @@ -16,6 +16,9 @@ unsigned long _find_next_andnot_bit(const unsigned long *addr1, const unsigned l unsigned long nbits, unsigned long start); unsigned long _find_next_or_bit(const unsigned long *addr1, const unsigned long *addr2, unsigned long nbits, unsigned long start); +unsigned long _find_next_and_andnot_bit(const unsigned long *addr1, const unsigned long *addr2, + const unsigned long *addr3, unsigned long nbits, + unsigned long start); unsigned long _find_next_zero_bit(const unsigned long *addr, unsigned long nbits, unsigned long start); extern unsigned long _find_first_bit(const unsigned long *addr, unsigned long size); @@ -159,6 +162,40 @@ unsigned long find_next_or_bit(const unsigned long *addr1, } #endif +#ifndef find_next_and_andnot_bit +/** + * find_next_and_andnot_bit - find the next bit set in *addr1 and *addr2, + * excluding all the bits in *addr3 + * @addr1: The first address to base the search on + * @addr2: The second address to base the search on + * @addr3: The third address to base the search on + * @size: The bitmap size in bits + * @offset: The bitnumber to start searching at + * + * Return: the bit number for the next set bit + * If no bits are set, returns @size. + */ +static __always_inline +unsigned long find_next_and_andnot_bit(const unsigned long *addr1, + const unsigned long *addr2, + const unsigned long *addr3, + unsigned long size, + unsigned long offset) +{ + if (small_const_nbits(size)) { + unsigned long val; + + if (unlikely(offset >= size)) + return size; + + val = *addr1 & *addr2 & ~*addr3 & GENMASK(size - 1, offset); + return val ? __ffs(val) : size; + } + + return _find_next_and_andnot_bit(addr1, addr2, addr3, size, offset); +} +#endif + #ifndef find_next_zero_bit /** * find_next_zero_bit - find the next cleared bit in a memory region @@ -568,6 +605,12 @@ unsigned long find_next_bit_le(const void *addr, unsigned (bit) = find_next_andnot_bit((addr1), (addr2), (size), (bit)), (bit) < (size);\ (bit)++) +#define for_each_and_andnot_bit(bit, addr1, addr2, addr3, size) \ + for ((bit) = 0; \ + (bit) = find_next_and_andnot_bit((addr1), (addr2), (addr3), (size), (bit)),\ + (bit) < (size); \ + (bit)++) + #define for_each_or_bit(bit, addr1, addr2, size) \ for ((bit) = 0; \ (bit) = find_next_or_bit((addr1), (addr2), (size), (bit)), (bit) < (size);\ diff --git a/lib/find_bit.c b/lib/find_bit.c index 32f99e9a670e..4403e00890b1 100644 --- a/lib/find_bit.c +++ b/lib/find_bit.c @@ -182,6 +182,18 @@ unsigned long _find_next_andnot_bit(const unsigned long *addr1, const unsigned l EXPORT_SYMBOL(_find_next_andnot_bit); #endif +#ifndef find_next_and_andnot_bit +unsigned long _find_next_and_andnot_bit(const unsigned long *addr1, + const unsigned long *addr2, + const unsigned long *addr3, + unsigned long nbits, + unsigned long start) +{ + return FIND_NEXT_BIT(addr1[idx] & addr2[idx] & ~addr3[idx], /* nop */, nbits, start); +} +EXPORT_SYMBOL(_find_next_and_andnot_bit); +#endif + #ifndef find_next_or_bit unsigned long _find_next_or_bit(const unsigned long *addr1, const unsigned long *addr2, unsigned long nbits, unsigned long start) From patchwork Thu Apr 20 05:19:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yury Norov X-Patchwork-Id: 13217854 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0405FC77B73 for ; Thu, 20 Apr 2023 05:20:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233157AbjDTFT6 (ORCPT ); Thu, 20 Apr 2023 01:19:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36300 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232726AbjDTFT4 (ORCPT ); Thu, 20 Apr 2023 01:19:56 -0400 Received: from mail-pf1-x435.google.com (mail-pf1-x435.google.com [IPv6:2607:f8b0:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id ED72B1FE4; Wed, 19 Apr 2023 22:19:54 -0700 (PDT) Received: by mail-pf1-x435.google.com with SMTP id d2e1a72fcca58-63b509fe13eso516255b3a.1; Wed, 19 Apr 2023 22:19:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1681967994; x=1684559994; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=V+FrYzkNBTAOE1d+tJlAIPGbtNhc7MIPbyq0PvaiMlQ=; b=LC/ptG50ZvrrRq38X7yUGXTnM/QM/L7x6DX5gIVV7eLu7QKqzFVwj7oNJznvJB0Y6O 7dUjUkcE1VndvXAsae/q2mzR5JMyCGcxwXa72MNqzR+ndy1lktemW52ZiDVRLV47s7TL oczV0jo2FEAWc/efyumlapT2XAYAGOgaHzA0HD/BbTpUfDPLp4b9I7umP03WVfOJFIGF ZRpmquXPBnnYglWlr/qTX2iMCO3UcpiKGsGsIXfFAdZoadcL0Fuif03q/YGzbShdKcYj ItVhV43Nl14VsutaKqZBIjBj9ZjrLafHNDN+W9fvXLwfdwrOqJD2Gl6djAZJ5WYOnG5t Sx3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681967994; x=1684559994; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=V+FrYzkNBTAOE1d+tJlAIPGbtNhc7MIPbyq0PvaiMlQ=; b=eML4VnmKLYiBC4CeyUo5/gtgiG/C0g+iMoj7PtDAHKxKsRT+qOogS9SP1Xor3MBKfh r7bL4IzruSNMbuToYhwJCjWY8SUdEgMeHC9X7jS6mZ1JVWO7Hmeq40rTkcg+tZcFczBc H34wnPqRRUUCa5YtfELSZvk1zUI/y48jy1VVtR/alQQZ6mMVbfaF/GJq17lOxBbq6yYW cer8CfZcu5B7CL67x6wQ8GazvUez5C3kidIxQXvyfvMLQ2EEOTmoaru4qrhGzXhSfLlf IYaG0RnUlvz6j5Zh0roj2xoy4VYnXe0HjZkdnLyz6flPswK3Ov1QBzBP6H76ZiAvJnCj X7vA== X-Gm-Message-State: AAQBX9dSb9sSh0SY6CzOx2YY0kuzcIj1WSJK3p6mhRVpEbvEPUH6Dl3L eRLu4Kx3J9jj2NuiwqDtZSk= X-Google-Smtp-Source: AKy350YjV40kKxn5fuLvZMqaKD7FR8P7oN8GuPUJhPDWMHW7VEbwFb4wyk94IBjLhUV7BvbCauWgiA== X-Received: by 2002:a05:6a20:1583:b0:ee:524e:8426 with SMTP id h3-20020a056a20158300b000ee524e8426mr604979pzj.31.1681967994330; Wed, 19 Apr 2023 22:19:54 -0700 (PDT) Received: from localhost ([2603:3024:e02:8500:653b:861d:e1ca:16ac]) by smtp.gmail.com with ESMTPSA id f23-20020a656297000000b00514256c05c2sm303981pgv.7.2023.04.19.22.19.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Apr 2023 22:19:53 -0700 (PDT) From: Yury Norov To: Jakub Kicinski , netdev@vger.kernel.org, linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Yury Norov , Saeed Mahameed , Pawel Chmielewski , Leon Romanovsky , "David S. Miller" , Eric Dumazet , Paolo Abeni , Andy Shevchenko , Rasmus Villemoes , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira , Valentin Schneider , Tariq Toukan , Gal Pressman , Greg Kroah-Hartman , Heiko Carstens , Barry Song Subject: [PATCH v2 2/8] sched/topology: introduce sched_numa_find_next_cpu() Date: Wed, 19 Apr 2023 22:19:40 -0700 Message-Id: <20230420051946.7463-3-yury.norov@gmail.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20230420051946.7463-1-yury.norov@gmail.com> References: <20230420051946.7463-1-yury.norov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org The function searches for the next CPU in a given cpumask according to NUMA topology, so that it traverses cpus per-hop. If the CPU is the last cpu in a given hop, sched_numa_find_next_cpu() switches to the next hop, and picks the first CPU from there, excluding those already traversed. Signed-off-by: Yury Norov --- include/linux/topology.h | 7 +++++++ kernel/sched/topology.c | 39 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 46 insertions(+) diff --git a/include/linux/topology.h b/include/linux/topology.h index fea32377f7c7..13209095d6e2 100644 --- a/include/linux/topology.h +++ b/include/linux/topology.h @@ -247,6 +247,7 @@ static inline const struct cpumask *cpu_cpu_mask(int cpu) #ifdef CONFIG_NUMA int sched_numa_find_nth_cpu(const struct cpumask *cpus, int cpu, int node); +int sched_numa_find_next_cpu(const struct cpumask *cpus, int cpu, int node, unsigned int *hop); extern const struct cpumask *sched_numa_hop_mask(unsigned int node, unsigned int hops); #else static __always_inline int sched_numa_find_nth_cpu(const struct cpumask *cpus, int cpu, int node) @@ -254,6 +255,12 @@ static __always_inline int sched_numa_find_nth_cpu(const struct cpumask *cpus, i return cpumask_nth(cpu, cpus); } +static __always_inline +int sched_numa_find_next_cpu(const struct cpumask *cpus, int cpu, int node, unsigned int *hop) +{ + return find_next_bit(cpumask_bits(cpus), small_cpumask_bits, cpu); +} + static inline const struct cpumask * sched_numa_hop_mask(unsigned int node, unsigned int hops) { diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c index 051aaf65c749..fc163e4181e6 100644 --- a/kernel/sched/topology.c +++ b/kernel/sched/topology.c @@ -2130,6 +2130,45 @@ int sched_numa_find_nth_cpu(const struct cpumask *cpus, int cpu, int node) } EXPORT_SYMBOL_GPL(sched_numa_find_nth_cpu); +/* + * sched_numa_find_next_cpu() - given the NUMA topology, find the next cpu + * cpumask: cpumask to find a cpu from + * cpu: current cpu + * node: local node + * hop: (in/out) indicates distance order of current CPU to a local node + * + * The function searches for next cpu at a given NUMA distance, indicated + * by hop, and if nothing found, tries to find CPUs at a greater distance, + * starting from the beginning. + * + * Return: cpu, or >= nr_cpu_ids when nothing found. + */ +int sched_numa_find_next_cpu(const struct cpumask *cpus, int cpu, int node, unsigned int *hop) +{ + unsigned long *cur, *prev; + struct cpumask ***masks; + unsigned int ret; + + if (*hop >= sched_domains_numa_levels) + return nr_cpu_ids; + + masks = rcu_dereference(sched_domains_numa_masks); + cur = cpumask_bits(masks[*hop][node]); + if (*hop == 0) + ret = find_next_and_bit(cpumask_bits(cpus), cur, nr_cpu_ids, cpu); + else { + prev = cpumask_bits(masks[*hop - 1][node]); + ret = find_next_and_andnot_bit(cpumask_bits(cpus), cur, prev, nr_cpu_ids, cpu); + } + + if (ret < nr_cpu_ids) + return ret; + + *hop += 1; + return sched_numa_find_next_cpu(cpus, 0, node, hop); +} +EXPORT_SYMBOL_GPL(sched_numa_find_next_cpu); + /** * sched_numa_hop_mask() - Get the cpumask of CPUs at most @hops hops away from * @node From patchwork Thu Apr 20 05:19:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yury Norov X-Patchwork-Id: 13217855 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9F886C77B78 for ; Thu, 20 Apr 2023 05:20:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233659AbjDTFUE (ORCPT ); Thu, 20 Apr 2023 01:20:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36352 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232879AbjDTFT5 (ORCPT ); Thu, 20 Apr 2023 01:19:57 -0400 Received: from mail-pf1-x42e.google.com (mail-pf1-x42e.google.com [IPv6:2607:f8b0:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E05374C35; Wed, 19 Apr 2023 22:19:56 -0700 (PDT) Received: by mail-pf1-x42e.google.com with SMTP id d2e1a72fcca58-63b4dfead1bso586496b3a.3; Wed, 19 Apr 2023 22:19:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1681967996; x=1684559996; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kLNvrPFQliITteAo9upC56iHsY6zcpqba+MUcf3ofEM=; b=WlZAMdBLHMCaLqFR3I2eFzdqPm59UWfz+KGYXtcOW/MO0//QoQQ5CLKqZQGme5y8RS 53dlSb7LGcn/VoNs8WkLSWPCB3/u/6oBfLi/MGnAm9YvmGDNx28w4G8tzrAjIXrWSCtw DLpslLzfClllUH2eWmXwFJOsMioWZrE6R9n7xoYYXsLXY9dsSZN3sygetKxh37nrQIU/ jHCVy0QTeoyzGNSZYL4gkOTMD0nT9IjHqtqCk9FleBvx95L1q6bCbmhM4bTi7YHRid0z 4dVm4gwPeqBbCVt/VoOKd7wCAJzk8OQ/aF3RxEge04XngcGu7Wc6sqYnnP4xmwabZV2x OIJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681967996; x=1684559996; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kLNvrPFQliITteAo9upC56iHsY6zcpqba+MUcf3ofEM=; b=RUS/kMblVIhFT7JVFyhH1WCsHcjVD/7jgJYOray3xAX++5a68MHtUmXEGNnn3ohZMi YlcCin1MWPJjsXgdPrR5OeQK/7yVnhR4wpVzPEt+s1BbRurFkt0XjbnTEKejg50K0CsK Rm2XACvgNybHXIV4yYnaOT/w935xktVvtrZ1boQ7q+vpvv2bCDgjnLxgitYD5BXWZ6IF Lv+VK+K72XaBAm+qNuQSxbac7jUz/URYn+2FFErnSJ3LiQxUU31pxMNhTr+o0rWZGncG Y5ET+Xwo14VZcGLrB9Y7edbaG56WItOfKykZJCdctQrUKvQHCoLcdCBhcBp98+/yeOY9 D+Gw== X-Gm-Message-State: AAQBX9f3yOA+9hh9RZe5NIJN4TNylqxGiXoeIqASzv5GWC8j0C1GdAav I8eBk33nZ7nt+VFKWHOGGMc= X-Google-Smtp-Source: AKy350aU0t2+JZiJO+n5013tFnzCTP6dgs7dJGV+W237jVDegu4iHwv7ULjmis6CnBXcEsK5GfzooQ== X-Received: by 2002:a05:6a00:1255:b0:63b:2102:a1d4 with SMTP id u21-20020a056a00125500b0063b2102a1d4mr7203200pfi.13.1681967996230; Wed, 19 Apr 2023 22:19:56 -0700 (PDT) Received: from localhost ([2603:3024:e02:8500:653b:861d:e1ca:16ac]) by smtp.gmail.com with ESMTPSA id d17-20020a056a0024d100b0063d47bfcdd5sm306160pfv.111.2023.04.19.22.19.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Apr 2023 22:19:55 -0700 (PDT) From: Yury Norov To: Jakub Kicinski , netdev@vger.kernel.org, linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Yury Norov , Saeed Mahameed , Pawel Chmielewski , Leon Romanovsky , "David S. Miller" , Eric Dumazet , Paolo Abeni , Andy Shevchenko , Rasmus Villemoes , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira , Valentin Schneider , Tariq Toukan , Gal Pressman , Greg Kroah-Hartman , Heiko Carstens , Barry Song Subject: [PATCH v2 3/8] sched/topology: add for_each_numa_cpu() macro Date: Wed, 19 Apr 2023 22:19:41 -0700 Message-Id: <20230420051946.7463-4-yury.norov@gmail.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20230420051946.7463-1-yury.norov@gmail.com> References: <20230420051946.7463-1-yury.norov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org for_each_cpu() is widely used in the kernel, and it's beneficial to create a NUMA-aware version of the macro. Recently added for_each_numa_hop_mask() works, but switching existing codebase to using it is not an easy process. New for_each_numa_cpu() is designed to be similar to the for_each_cpu(). It allows to convert existing code to NUMA-aware as simple as adding a hop iterator variable and passing it inside new macro. for_each_numa_cpu() takes care of the rest. At the moment, we have 2 users of NUMA-aware enumerators. One is Melanox's in-tree driver, and another is Intel's in-review driver: https://lore.kernel.org/lkml/20230216145455.661709-1-pawel.chmielewski@intel.com/ Both real-life examples follow the same pattern: for_each_numa_hop_mask(cpus, prev, node) { for_each_cpu_andnot(cpu, cpus, prev) { if (cnt++ == max_num) goto out; do_something(cpu); } prev = cpus; } With the new macro, it would look like this: for_each_numa_cpu(cpu, hop, node, cpu_possible_mask) { if (cnt++ == max_num) break; do_something(cpu); } Straight conversion of existing for_each_cpu() codebase to NUMA-aware version with for_each_numa_hop_mask() is difficult because it doesn't take a user-provided cpu mask, and eventually ends up with open-coded double loop. With for_each_numa_cpu() it shouldn't be a brainteaser. Consider the NUMA-ignorant example: cpumask_t cpus = get_mask(); int cnt = 0, cpu; for_each_cpu(cpu, cpus) { if (cnt++ == max_num) break; do_something(cpu); } Converting it to NUMA-aware version would be as simple as: cpumask_t cpus = get_mask(); int node = get_node(); int cnt = 0, hop, cpu; for_each_numa_cpu(cpu, hop, node, cpus) { if (cnt++ == max_num) break; do_something(cpu); } The latter looks more verbose and avoids from open-coding that annoying double loop. Another advantage is that it works with a 'hop' parameter with the clear meaning of NUMA distance, and doesn't make people not familiar to enumerator internals bothering with current and previous masks machinery. Signed-off-by: Yury Norov --- include/linux/topology.h | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/include/linux/topology.h b/include/linux/topology.h index 13209095d6e2..01fb3a55d7ce 100644 --- a/include/linux/topology.h +++ b/include/linux/topology.h @@ -286,4 +286,20 @@ sched_numa_hop_mask(unsigned int node, unsigned int hops) !IS_ERR_OR_NULL(mask); \ __hops++) +/** + * for_each_numa_cpu - iterate over cpus in increasing order taking into account + * NUMA distances from a given node. + * @cpu: the (optionally unsigned) integer iterator + * @hop: the iterator variable, must be initialized to a desired minimal hop. + * @node: the NUMA node to start the search from. + * @mask: the cpumask pointer + * + * Requires rcu_lock to be held. + */ +#define for_each_numa_cpu(cpu, hop, node, mask) \ + for ((cpu) = 0, (hop) = 0; \ + (cpu) = sched_numa_find_next_cpu((mask), (cpu), (node), &(hop)),\ + (cpu) < nr_cpu_ids; \ + (cpu)++) + #endif /* _LINUX_TOPOLOGY_H */ From patchwork Thu Apr 20 05:19:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yury Norov X-Patchwork-Id: 13217856 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9E350C77B72 for ; Thu, 20 Apr 2023 05:20:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233790AbjDTFUd (ORCPT ); Thu, 20 Apr 2023 01:20:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36706 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233697AbjDTFUF (ORCPT ); Thu, 20 Apr 2023 01:20:05 -0400 Received: from mail-pf1-x42e.google.com (mail-pf1-x42e.google.com [IPv6:2607:f8b0:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DF3FA55AB; Wed, 19 Apr 2023 22:19:59 -0700 (PDT) Received: by mail-pf1-x42e.google.com with SMTP id d2e1a72fcca58-63b50a02bffso570009b3a.2; Wed, 19 Apr 2023 22:19:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1681967998; x=1684559998; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=owQqtMurqm2qI+24ndEGs/qWEaDzKKQeglmv6UMu1sw=; b=A5ITtIQzxLED1qZR1Mnw5hskw9J9pA2fHW1Fz6EgknrldXSQkt2U8nJXiJr8zWJ55U S60jhKV/oq8I8Z24Zq0fMc8GY7jFnfUTm/OK+ezJl8BVHZ1NIB+XN7soOgU6/Hn15dOQ 156MqX8nV+QIvo9dzrQs27d2xJ9+P3gUgClScF1ggfnrr9MecT9/C4Yfw96+r0imq+Fn 66lEzr9Pn98efMggd6jrxylqd2TuocFEtCv9qWNaT3+8VVeTcuAeNEjFRengib8pyjna bfCCsRuZRqwbbupmhDOOnn4AJih22+wiKuy1sVZtJKsQIMpamaAEYdS7dApF5cw3LdJh 09EQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681967998; x=1684559998; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=owQqtMurqm2qI+24ndEGs/qWEaDzKKQeglmv6UMu1sw=; b=Kzefz/6xvMAA1y1ac8EXKZJkOYqRot6XzvZjIEaHalyZSyca79vabTv8LEtWjTFPkf Ct7C58t3videeU3E5fnVESoIs6bu2/0yD0CqDqi7GqbQ1SzNqLLThEAsY4E2QUSbZaqx 3cdCajqKIVbqwzYxvLAHaK2cAp7h7crskBqeRToOjN9uUvARSZEVXqHGM5R4udPPzYUQ svg0NnUDiqYb6rTjO+W2jGKcUcB9ciqdpGvv95a0RS42x06JF3pwLlkgIk3mnzgzMp2W 2wkXAXHhddJ2pOeNakHAgGPpeomVmoyO1T6u9soIqnPs4S6i6Bp7br9w31WyDg3F282D 2NWg== X-Gm-Message-State: AAQBX9d2xpBROYf9P4z3bNiTKDW2ytkK50AzW4CO1Cle2OgfR9g5Z3jl KtUODxS1GndZbRmG3F90Fok= X-Google-Smtp-Source: AKy350b2TrJbqRKwQBXYvaIQBVuSaGrUUKe1brGFiJALP+1j6B7HK9k+M/FG34jQ7dhbD3XRQwH4aQ== X-Received: by 2002:a05:6a00:18a2:b0:639:c88b:c3e0 with SMTP id x34-20020a056a0018a200b00639c88bc3e0mr7518672pfh.22.1681967998283; Wed, 19 Apr 2023 22:19:58 -0700 (PDT) Received: from localhost ([2603:3024:e02:8500:653b:861d:e1ca:16ac]) by smtp.gmail.com with ESMTPSA id fa9-20020a056a002d0900b0062622ae3648sm309878pfb.78.2023.04.19.22.19.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Apr 2023 22:19:57 -0700 (PDT) From: Yury Norov To: Jakub Kicinski , netdev@vger.kernel.org, linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Yury Norov , Saeed Mahameed , Pawel Chmielewski , Leon Romanovsky , "David S. Miller" , Eric Dumazet , Paolo Abeni , Andy Shevchenko , Rasmus Villemoes , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira , Valentin Schneider , Tariq Toukan , Gal Pressman , Greg Kroah-Hartman , Heiko Carstens , Barry Song Subject: [PATCH v2 4/8] net: mlx5: switch comp_irqs_request() to using for_each_numa_cpu Date: Wed, 19 Apr 2023 22:19:42 -0700 Message-Id: <20230420051946.7463-5-yury.norov@gmail.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20230420051946.7463-1-yury.norov@gmail.com> References: <20230420051946.7463-1-yury.norov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org for_each_numa_cpu() is a more straightforward alternative to for_each_numa_hop_mask() + for_each_cpu_andnot(). Signed-off-by: Yury Norov Reviewed-by: Tariq Toukan --- drivers/net/ethernet/mellanox/mlx5/core/eq.c | 16 +++++----------- 1 file changed, 5 insertions(+), 11 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eq.c b/drivers/net/ethernet/mellanox/mlx5/core/eq.c index 38b32e98f3bd..80368952e9b1 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/eq.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/eq.c @@ -817,12 +817,10 @@ static void comp_irqs_release(struct mlx5_core_dev *dev) static int comp_irqs_request(struct mlx5_core_dev *dev) { struct mlx5_eq_table *table = dev->priv.eq_table; - const struct cpumask *prev = cpu_none_mask; - const struct cpumask *mask; int ncomp_eqs = table->num_comp_eqs; u16 *cpus; int ret; - int cpu; + int cpu, hop; int i; ncomp_eqs = table->num_comp_eqs; @@ -844,15 +842,11 @@ static int comp_irqs_request(struct mlx5_core_dev *dev) i = 0; rcu_read_lock(); - for_each_numa_hop_mask(mask, dev->priv.numa_node) { - for_each_cpu_andnot(cpu, mask, prev) { - cpus[i] = cpu; - if (++i == ncomp_eqs) - goto spread_done; - } - prev = mask; + for_each_numa_cpu(cpu, hop, dev->priv.numa_node, cpu_possible_mask) { + cpus[i] = cpu; + if (++i == ncomp_eqs) + break; } -spread_done: rcu_read_unlock(); ret = mlx5_irqs_request_vectors(dev, cpus, ncomp_eqs, table->comp_irqs); kfree(cpus); From patchwork Thu Apr 20 05:19:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yury Norov X-Patchwork-Id: 13217857 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F3F6C77B78 for ; Thu, 20 Apr 2023 05:20:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233840AbjDTFUq (ORCPT ); Thu, 20 Apr 2023 01:20:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37432 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233218AbjDTFUd (ORCPT ); Thu, 20 Apr 2023 01:20:33 -0400 Received: from mail-pg1-x52a.google.com (mail-pg1-x52a.google.com [IPv6:2607:f8b0:4864:20::52a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AADAA5FC2; Wed, 19 Apr 2023 22:20:01 -0700 (PDT) Received: by mail-pg1-x52a.google.com with SMTP id 41be03b00d2f7-517c840f181so347451a12.3; Wed, 19 Apr 2023 22:20:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1681968000; x=1684560000; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Nw654FlL6BUkwfjkIWsvpLLmEwiZB7gH0V0i7mwVLn0=; b=caaMCMxW9410Xn3lK3IxjlDtVt0D+8dKey3tlzaenQSmAdpDYe+tGHnk2N6G3E4hSG K0bM8/EQycWefdD8JSBRDuzm1L9PF8GILiLpUQK4DYmEwjouKc/kC6ZU1fmaiRfaosyC 41YaUEBWy83LIiEvpkuzG4JVu8J+q3zIf7Rh5l7SNcAnTQ2deX97DOuJX081V7F/BUHk cLgDfvSRNcnBXczVvDlbpXtQusjBiergqrq5KbKUf+TOAK/E0249AFh7BJywCZWu21ko F53i6I4y+M9n9erQbFGlvxdttMZWwqXqQs1vkHc6Mgc+ZzFtngsk4ePBeSmSBfzVGhC/ Ielw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681968000; x=1684560000; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Nw654FlL6BUkwfjkIWsvpLLmEwiZB7gH0V0i7mwVLn0=; b=fNNOA67fnNsuqfd8Osero470TICy7sJ1NKh398JPO5LjfTw6hLvAXi1OoVjNvfVFYl m8t4b8vNkQfRcH1to/6sjkN6Y2k/l6y1Lc4eNep/5+KKbtVs/M4yqtHolr3UJyNrCiuB lMgsOa36VUgqAE7yCeJ2s1eT2eqCEAQcevbRdyCutBbyKc43MsmrVS4ht2es3rpdprdU d7+KqWQLRPhNeVglFB3wlw0kV7qit5Zip3qwt0bj6HNRhwz3W4A7NNTMDseMuBurSHIh Bq5zU9Jhl+TIhwMZxTu54KXaSAJBgQ7EO0rlKovsvvjtEkJQVS5WtDjN8c7XUwDIqgWV CP5Q== X-Gm-Message-State: AAQBX9clS9CAXwzE5VW4mig09HD3eR8OpSiGXseUv7AuXtbKBHAkkRQb PageeyvZ5yTMw4q1cWArXOQ= X-Google-Smtp-Source: AKy350YobFSql6y5w6aBkEfFAhUb73JVMETAlJE/nTM985uHF9JUmaTDpo3tkrptRNRJu+kg0DYWlQ== X-Received: by 2002:a17:90a:f484:b0:249:6fc1:cd76 with SMTP id bx4-20020a17090af48400b002496fc1cd76mr451011pjb.43.1681968000378; Wed, 19 Apr 2023 22:20:00 -0700 (PDT) Received: from localhost ([2603:3024:e02:8500:653b:861d:e1ca:16ac]) by smtp.gmail.com with ESMTPSA id u10-20020a63234a000000b0050be8e0b94csm264166pgm.90.2023.04.19.22.19.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Apr 2023 22:19:59 -0700 (PDT) From: Yury Norov To: Jakub Kicinski , netdev@vger.kernel.org, linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Yury Norov , Saeed Mahameed , Pawel Chmielewski , Leon Romanovsky , "David S. Miller" , Eric Dumazet , Paolo Abeni , Andy Shevchenko , Rasmus Villemoes , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira , Valentin Schneider , Tariq Toukan , Gal Pressman , Greg Kroah-Hartman , Heiko Carstens , Barry Song Subject: [PATCH v2 5/8] lib/cpumask: update comment to cpumask_local_spread() Date: Wed, 19 Apr 2023 22:19:43 -0700 Message-Id: <20230420051946.7463-6-yury.norov@gmail.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20230420051946.7463-1-yury.norov@gmail.com> References: <20230420051946.7463-1-yury.norov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Now that we have a for_each_numa_cpu(), which is a more straightforward replacement to the cpumask_local_spread() when it comes to enumeration of CPUs with respect to NUMA topology, it's worth to update the comment. Signed-off-by: Yury Norov --- lib/cpumask.c | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/lib/cpumask.c b/lib/cpumask.c index e7258836b60b..151d1dc5c593 100644 --- a/lib/cpumask.c +++ b/lib/cpumask.c @@ -127,11 +127,8 @@ void __init free_bootmem_cpumask_var(cpumask_var_t mask) * * There's a better alternative based on for_each()-like iterators: * - * for_each_numa_hop_mask(mask, node) { - * for_each_cpu_andnot(cpu, mask, prev) - * do_something(cpu); - * prev = mask; - * } + * for_each_numa_cpu(cpu, hop, node, cpu_online_mask) + * do_something(cpu); * * It's simpler and more verbose than above. Complexity of iterator-based * enumeration is O(sched_domains_numa_levels * nr_cpu_ids), while From patchwork Thu Apr 20 05:19:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yury Norov X-Patchwork-Id: 13217858 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5FCB5C77B78 for ; Thu, 20 Apr 2023 05:20:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233803AbjDTFUu (ORCPT ); Thu, 20 Apr 2023 01:20:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37450 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233806AbjDTFUe (ORCPT ); Thu, 20 Apr 2023 01:20:34 -0400 Received: from mail-pf1-x42f.google.com (mail-pf1-x42f.google.com [IPv6:2607:f8b0:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 79D1959E5; Wed, 19 Apr 2023 22:20:03 -0700 (PDT) Received: by mail-pf1-x42f.google.com with SMTP id d2e1a72fcca58-63d2ba63dddso546881b3a.2; Wed, 19 Apr 2023 22:20:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1681968002; x=1684560002; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5usMpMDqVoc+ecsOtKE0Gtse+UU2DDnssjHsHE2rpV0=; b=EnspSQAPrQqvWP+FClFMlSsz4VeavJqTqfuNfrU+p2/isyACrtZ764pJLdjUFPzs5q UvF3NfeCiJDVsO5DlCEpMzHtuVnGBsOUFvM3TSWzDcJG3RoCMSu3mscfmG8pgNiwknSR KwludEpAlxzeEyNH332D4kslC3vmd1m3vOlhS1WFXmnJvy4oq27/VZ8NkvK9UIgodzvc vNW0Iu/hLrvQY43etyy/Vdb+8Ci7Advv7qfjpaiNxdSeiTdQWgOzAHJWNVg+s9ipU6SX KL5BcnTB7cFk9pO5HTzzUNYwx2YeBsZz0YO3k1rixwFp1S26i3/Za4AQacktEllyzGJH 1isg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681968002; x=1684560002; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5usMpMDqVoc+ecsOtKE0Gtse+UU2DDnssjHsHE2rpV0=; b=LkyB3dJ+gj9yx+K78aMVzF2gmjoX9OOrT67a1/lf5rgkRXwOWXyTt6WfRKs5buS6KR 6iOB3sG1mQYjyk+8vjhSFmPTNOGyw2O06zi1gJT1il340kGNlSG/eph/wnPqrlBG6v+W tL53n9Fr3z9lFTHi1rcj1vJKCDL8cimQGMDGCa4L4Qg+Dm1dPALjm+hedD4MKJrA1UBB 8BzScgre2Xs5Yx97iNbIwUa3kSp0AdpWH0DaB3bcsUG0MlIlijAEpnu3vBipdgGg4/Sf baJcGcQXuZ2XfyqP48Z2ZTek+nruaeEz5yV1snQUXGK42nMQ/NaWGAXFLV46qO90oZnR 1Utw== X-Gm-Message-State: AAQBX9cbWBaBTuHdIt0gzckPS8wWBRABryNLaJzUuo99l0zDvAyjADLr 8sswotkyW2RZXsrYf1PHp8I= X-Google-Smtp-Source: AKy350ZLX/cyqSBWzXE4XimpTZQpxiv+d9PhS6FwmYV5K0k7VFzQCqizXvgPPsjRknrgkYMSCpMY5Q== X-Received: by 2002:a05:6a20:8e13:b0:f1:bea6:a319 with SMTP id y19-20020a056a208e1300b000f1bea6a319mr548168pzj.25.1681968002080; Wed, 19 Apr 2023 22:20:02 -0700 (PDT) Received: from localhost ([2603:3024:e02:8500:653b:861d:e1ca:16ac]) by smtp.gmail.com with ESMTPSA id m11-20020a63580b000000b0051322a5aa64sm295875pgb.3.2023.04.19.22.20.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Apr 2023 22:20:01 -0700 (PDT) From: Yury Norov To: Jakub Kicinski , netdev@vger.kernel.org, linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Yury Norov , Saeed Mahameed , Pawel Chmielewski , Leon Romanovsky , "David S. Miller" , Eric Dumazet , Paolo Abeni , Andy Shevchenko , Rasmus Villemoes , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira , Valentin Schneider , Tariq Toukan , Gal Pressman , Greg Kroah-Hartman , Heiko Carstens , Barry Song Subject: [PATCH v2 6/8] sched/topology: export sched_domains_numa_levels Date: Wed, 19 Apr 2023 22:19:44 -0700 Message-Id: <20230420051946.7463-7-yury.norov@gmail.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20230420051946.7463-1-yury.norov@gmail.com> References: <20230420051946.7463-1-yury.norov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org The following patch adds a test for NUMA-aware CPU enumerators, and it requires an access to sched_domains_numa_levels. Signed-off-by: Yury Norov --- include/linux/topology.h | 7 +++++++ kernel/sched/topology.c | 10 ++++++---- 2 files changed, 13 insertions(+), 4 deletions(-) diff --git a/include/linux/topology.h b/include/linux/topology.h index 01fb3a55d7ce..7ebcc886dc76 100644 --- a/include/linux/topology.h +++ b/include/linux/topology.h @@ -43,6 +43,13 @@ for_each_online_node(node) \ if (nr_cpus_node(node)) +#ifdef CONFIG_NUMA +extern int __sched_domains_numa_levels; +#define sched_domains_numa_levels ((const int)__sched_domains_numa_levels) +#else +#define sched_domains_numa_levels (1) +#endif + int arch_update_cpu_topology(void); /* Conform to ACPI 2.0 SLIT distance definitions */ diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c index fc163e4181e6..56daa279c411 100644 --- a/kernel/sched/topology.c +++ b/kernel/sched/topology.c @@ -1508,7 +1508,9 @@ static void claim_allocations(int cpu, struct sched_domain *sd) #ifdef CONFIG_NUMA enum numa_topology_type sched_numa_topology_type; -static int sched_domains_numa_levels; +int __sched_domains_numa_levels; +EXPORT_SYMBOL_GPL(__sched_domains_numa_levels); + static int sched_domains_curr_level; int sched_max_numa_distance; @@ -1872,7 +1874,7 @@ void sched_init_numa(int offline_node) * * We reset it to 'nr_levels' at the end of this function. */ - sched_domains_numa_levels = 0; + __sched_domains_numa_levels = 0; masks = kzalloc(sizeof(void *) * nr_levels, GFP_KERNEL); if (!masks) @@ -1948,7 +1950,7 @@ void sched_init_numa(int offline_node) sched_domain_topology_saved = sched_domain_topology; sched_domain_topology = tl; - sched_domains_numa_levels = nr_levels; + __sched_domains_numa_levels = nr_levels; WRITE_ONCE(sched_max_numa_distance, sched_domains_numa_distance[nr_levels - 1]); init_numa_topology_type(offline_node); @@ -1961,7 +1963,7 @@ static void sched_reset_numa(void) struct cpumask ***masks; nr_levels = sched_domains_numa_levels; - sched_domains_numa_levels = 0; + __sched_domains_numa_levels = 0; sched_max_numa_distance = 0; sched_numa_topology_type = NUMA_DIRECT; distances = sched_domains_numa_distance; From patchwork Thu Apr 20 05:19:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yury Norov X-Patchwork-Id: 13217859 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7990CC77B78 for ; Thu, 20 Apr 2023 05:21:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233887AbjDTFVJ (ORCPT ); Thu, 20 Apr 2023 01:21:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37546 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233819AbjDTFUh (ORCPT ); Thu, 20 Apr 2023 01:20:37 -0400 Received: from mail-pl1-x634.google.com (mail-pl1-x634.google.com [IPv6:2607:f8b0:4864:20::634]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 051AC61AB; Wed, 19 Apr 2023 22:20:04 -0700 (PDT) Received: by mail-pl1-x634.google.com with SMTP id d9443c01a7336-1a6c5acf6ccso5829165ad.3; Wed, 19 Apr 2023 22:20:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1681968004; x=1684560004; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=OSjdqRXXXgYSdBLqK0Rzny0ztxBdvMaFW0s7IjJDzbk=; b=Ybxwe0Yed6TDQbZt4P4jJnwzwNYoazEymPMwfm7y2iWBuo10gwWPCtJXLWAI2jp88R 7G/ZrXA4ugRV40R+HD8eK6Uqf+RCesq67q/t+Fgb94vuYbz193xfsE9pcylZrhvDYkyO 4+r+C0yP0GYVslLqpFnHTmKKReOrIkrmSY9ZT/0/4F6Np20g+kPjTVTbGC9/OMCQ/7gc +kZllz8/kI893xCtYA+nPmryHa1BfbJTqxI0b+jkyUlgRvFVLDmRYztiohR6fCgWqXii ZsnoBGWw9v+EgD0Ae5WNtPCYiQahOybJSXPdCl4kazJ1X4mZqTHy3wLcH3n9ofx5Rn21 KqVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681968004; x=1684560004; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=OSjdqRXXXgYSdBLqK0Rzny0ztxBdvMaFW0s7IjJDzbk=; b=HauaI6ZWqKCYXs68jAM9390TmX3LB3RkbKdara/+suRee709/qHalqaZsb11/25cmb zQaHgJf8HfbxfWa8enSHYzDTyY7Ts/IfeI4kcgl6RrKTKQLUMZ50dw/X4e9Ql2+/JRBa EKouCmLYO5FqUFQvLcwaOoSiNeDkeI3zBiB+LZyaNQCgAU3ynbBgiifxVesdStLJyI/k cwSI+V6DZhvTr4Arb0Le6pi/2leGDi0KG8gA/Cdx2kV1LjdipjXhkU+CXKaw5mo95OnK /HpH0LnVqcF5IvETfMuolxgjVdqub7IzpXXwlg0EQqJNH0Wb9UloElvknWjYjtcBt43g B82g== X-Gm-Message-State: AAQBX9ei5KB6/pCS54HvPbEiWBVZRWDT1+wa101sbpbjfzRUvKbsWq35 wJdnfP20xDOcwf/5R70GpFw= X-Google-Smtp-Source: AKy350b9F8rpx7c4FugyLgTAE7jTlFSaz/LBmY7KHzcDVTxRnpAbBd/hzTt8vapTV+lrjWJcstWGQw== X-Received: by 2002:a17:902:ce88:b0:19d:1834:92b9 with SMTP id f8-20020a170902ce8800b0019d183492b9mr316932plg.56.1681968004129; Wed, 19 Apr 2023 22:20:04 -0700 (PDT) Received: from localhost ([2603:3024:e02:8500:653b:861d:e1ca:16ac]) by smtp.gmail.com with ESMTPSA id gq3-20020a17090b104300b0024b4a06a4fesm167781pjb.5.2023.04.19.22.20.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Apr 2023 22:20:03 -0700 (PDT) From: Yury Norov To: Jakub Kicinski , netdev@vger.kernel.org, linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Yury Norov , Saeed Mahameed , Pawel Chmielewski , Leon Romanovsky , "David S. Miller" , Eric Dumazet , Paolo Abeni , Andy Shevchenko , Rasmus Villemoes , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira , Valentin Schneider , Tariq Toukan , Gal Pressman , Greg Kroah-Hartman , Heiko Carstens , Barry Song Subject: [PATCH v2 7/8] lib: add test for for_each_numa_{cpu,hop_mask}() Date: Wed, 19 Apr 2023 22:19:45 -0700 Message-Id: <20230420051946.7463-8-yury.norov@gmail.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20230420051946.7463-1-yury.norov@gmail.com> References: <20230420051946.7463-1-yury.norov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org The test ensures that enumerators' output is consistent with cpumask_local_spread(). Signed-off-by: Yury Norov --- lib/test_bitmap.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/lib/test_bitmap.c b/lib/test_bitmap.c index a8005ad3bd58..1b5f805f6879 100644 --- a/lib/test_bitmap.c +++ b/lib/test_bitmap.c @@ -12,6 +12,7 @@ #include #include #include +#include #include #include "../tools/testing/selftests/kselftest_module.h" @@ -751,6 +752,33 @@ static void __init test_for_each_set_bit_wrap(void) } } +static void __init test_for_each_numa(void) +{ + unsigned int cpu, node; + + for (node = 0; node < sched_domains_numa_levels; node++) { + const struct cpumask *m, *p = cpu_none_mask; + unsigned int c = 0; + + rcu_read_lock(); + for_each_numa_hop_mask(m, node) { + for_each_cpu_andnot(cpu, m, p) + expect_eq_uint(cpumask_local_spread(c++, node), cpu); + p = m; + } + rcu_read_unlock(); + } + + for (node = 0; node < sched_domains_numa_levels; node++) { + unsigned int hop, c = 0; + + rcu_read_lock(); + for_each_numa_cpu(cpu, hop, node, cpu_online_mask) + expect_eq_uint(cpumask_local_spread(c++, node), cpu); + rcu_read_unlock(); + } +} + static void __init test_for_each_set_bit(void) { DECLARE_BITMAP(orig, 500); @@ -1237,6 +1265,7 @@ static void __init selftest(void) test_for_each_clear_bitrange_from(); test_for_each_set_clump8(); test_for_each_set_bit_wrap(); + test_for_each_numa(); } KSTM_MODULE_LOADERS(test_bitmap); From patchwork Thu Apr 20 05:19:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yury Norov X-Patchwork-Id: 13217860 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 43512C77B72 for ; Thu, 20 Apr 2023 05:21:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233904AbjDTFVU (ORCPT ); Thu, 20 Apr 2023 01:21:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36802 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233827AbjDTFUp (ORCPT ); Thu, 20 Apr 2023 01:20:45 -0400 Received: from mail-pf1-x431.google.com (mail-pf1-x431.google.com [IPv6:2607:f8b0:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9A4866A42; Wed, 19 Apr 2023 22:20:06 -0700 (PDT) Received: by mail-pf1-x431.google.com with SMTP id d2e1a72fcca58-63b4960b015so526582b3a.3; Wed, 19 Apr 2023 22:20:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1681968006; x=1684560006; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mEuVUyGkVR+9mS2OVPBgufzRK60Hjm+2X8xg7kaymT4=; b=QgcBC7QBOtNAraj6WWbRO+jlrX7uBp0bSSzwpYk1+EgdIenhJmGsDJSNCASqcre0q1 mcPsRHaKW8qS6pQQwQw85VL0X1YQvGlYrLrfnLdAIK8+eKiHtlYV5dctnW/23DNtY/Dj DKkBkIan3xYzRelahUP+Eq1NnM1rosgcTqYzHZYW65TokHUShgLIaJz0iiIOWzDhJ2dN dICuAVKsm4jiX/b+rHrz/ewanfQhkMEzUtcmvhhPi8v0jLySFRwTvxqeBhZkdXlaXm39 qHPOM6SBR9bC8AvG6wpm5jfl9xlVBlOV65ySAOvD+8pYTCthSBO8ugX8z4qeRKe3kMKN hSWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681968006; x=1684560006; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mEuVUyGkVR+9mS2OVPBgufzRK60Hjm+2X8xg7kaymT4=; b=Bh9NTq7s1xZfplIecYJBjPnPlAFnJ/KFQWpS2dZk3XV9aoKAZsVgdJ6ZTOoLp9Ud+G di0wyrY549v9L0fmj8BxJtjnuVizjBfmaselPgJA4T8nQiGqGLXS8fhCtKWtsJFmJDxs aTC7lMagNlittPL2B1dD3Dc7Jsl+MAcwclIO6FV5IiwgUSH6BKdT4bGRnme/aF4KjpCN cYrZ0KGMwcmhMh+Y/rPlhcSpgZ2NrsREoSpax6O2URqRpDHTkE1vl0RFXRxkFzo4Hp8u xnx8Yrdmq81MKP1nzEXLf30pg5syNMZxjFjjw8EpnAFI8k2u1EDOdj+C2X7OHlrbCU1I c/5Q== X-Gm-Message-State: AAQBX9dcU+Y3zhOFJ9BOqLg8Z7hRnUxHgTG1b+CzYeinYz5NRBzx3esC ONKBGGo8RjjmE5pcurQ7vEY= X-Google-Smtp-Source: AKy350bqU/yDYCrTiDgTOfdjRvtMMucqL2wiXqRDjP9IOAhI2Hi4gf4e1Khh7jHrIRVWJuC8Oxyaug== X-Received: by 2002:a05:6a20:548c:b0:ef:88c8:90ec with SMTP id i12-20020a056a20548c00b000ef88c890ecmr794792pzk.10.1681968005771; Wed, 19 Apr 2023 22:20:05 -0700 (PDT) Received: from localhost ([2603:3024:e02:8500:653b:861d:e1ca:16ac]) by smtp.gmail.com with ESMTPSA id p11-20020a654bcb000000b0050376cedb3asm295957pgr.24.2023.04.19.22.20.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Apr 2023 22:20:05 -0700 (PDT) From: Yury Norov To: Jakub Kicinski , netdev@vger.kernel.org, linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Yury Norov , Saeed Mahameed , Pawel Chmielewski , Leon Romanovsky , "David S. Miller" , Eric Dumazet , Paolo Abeni , Andy Shevchenko , Rasmus Villemoes , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira , Valentin Schneider , Tariq Toukan , Gal Pressman , Greg Kroah-Hartman , Heiko Carstens , Barry Song Subject: [PATCH v2 8/8] sched: drop for_each_numa_hop_mask() Date: Wed, 19 Apr 2023 22:19:46 -0700 Message-Id: <20230420051946.7463-9-yury.norov@gmail.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20230420051946.7463-1-yury.norov@gmail.com> References: <20230420051946.7463-1-yury.norov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Now that we have for_each_numa_cpu(), for_each_numa_hop_mask() and all related code is a dead code. Drop it. Signed-off-by: Yury Norov --- include/linux/topology.h | 25 ------------------------- kernel/sched/topology.c | 32 -------------------------------- lib/test_bitmap.c | 13 ------------- 3 files changed, 70 deletions(-) diff --git a/include/linux/topology.h b/include/linux/topology.h index 7ebcc886dc76..1225ade33053 100644 --- a/include/linux/topology.h +++ b/include/linux/topology.h @@ -255,7 +255,6 @@ static inline const struct cpumask *cpu_cpu_mask(int cpu) #ifdef CONFIG_NUMA int sched_numa_find_nth_cpu(const struct cpumask *cpus, int cpu, int node); int sched_numa_find_next_cpu(const struct cpumask *cpus, int cpu, int node, unsigned int *hop); -extern const struct cpumask *sched_numa_hop_mask(unsigned int node, unsigned int hops); #else static __always_inline int sched_numa_find_nth_cpu(const struct cpumask *cpus, int cpu, int node) { @@ -267,32 +266,8 @@ int sched_numa_find_next_cpu(const struct cpumask *cpus, int cpu, int node, unsi { return find_next_bit(cpumask_bits(cpus), small_cpumask_bits, cpu); } - -static inline const struct cpumask * -sched_numa_hop_mask(unsigned int node, unsigned int hops) -{ - return ERR_PTR(-EOPNOTSUPP); -} #endif /* CONFIG_NUMA */ -/** - * for_each_numa_hop_mask - iterate over cpumasks of increasing NUMA distance - * from a given node. - * @mask: the iteration variable. - * @node: the NUMA node to start the search from. - * - * Requires rcu_lock to be held. - * - * Yields cpu_online_mask for @node == NUMA_NO_NODE. - */ -#define for_each_numa_hop_mask(mask, node) \ - for (unsigned int __hops = 0; \ - mask = (node != NUMA_NO_NODE || __hops) ? \ - sched_numa_hop_mask(node, __hops) : \ - cpu_online_mask, \ - !IS_ERR_OR_NULL(mask); \ - __hops++) - /** * for_each_numa_cpu - iterate over cpus in increasing order taking into account * NUMA distances from a given node. diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c index 56daa279c411..9d08ffdbd2d8 100644 --- a/kernel/sched/topology.c +++ b/kernel/sched/topology.c @@ -2171,38 +2171,6 @@ int sched_numa_find_next_cpu(const struct cpumask *cpus, int cpu, int node, unsi } EXPORT_SYMBOL_GPL(sched_numa_find_next_cpu); -/** - * sched_numa_hop_mask() - Get the cpumask of CPUs at most @hops hops away from - * @node - * @node: The node to count hops from. - * @hops: Include CPUs up to that many hops away. 0 means local node. - * - * Return: On success, a pointer to a cpumask of CPUs at most @hops away from - * @node, an error value otherwise. - * - * Requires rcu_lock to be held. Returned cpumask is only valid within that - * read-side section, copy it if required beyond that. - * - * Note that not all hops are equal in distance; see sched_init_numa() for how - * distances and masks are handled. - * Also note that this is a reflection of sched_domains_numa_masks, which may change - * during the lifetime of the system (offline nodes are taken out of the masks). - */ -const struct cpumask *sched_numa_hop_mask(unsigned int node, unsigned int hops) -{ - struct cpumask ***masks; - - if (node >= nr_node_ids || hops >= sched_domains_numa_levels) - return ERR_PTR(-EINVAL); - - masks = rcu_dereference(sched_domains_numa_masks); - if (!masks) - return ERR_PTR(-EBUSY); - - return masks[hops][node]; -} -EXPORT_SYMBOL_GPL(sched_numa_hop_mask); - #endif /* CONFIG_NUMA */ static int __sdt_alloc(const struct cpumask *cpu_map) diff --git a/lib/test_bitmap.c b/lib/test_bitmap.c index 1b5f805f6879..6becb044a66f 100644 --- a/lib/test_bitmap.c +++ b/lib/test_bitmap.c @@ -756,19 +756,6 @@ static void __init test_for_each_numa(void) { unsigned int cpu, node; - for (node = 0; node < sched_domains_numa_levels; node++) { - const struct cpumask *m, *p = cpu_none_mask; - unsigned int c = 0; - - rcu_read_lock(); - for_each_numa_hop_mask(m, node) { - for_each_cpu_andnot(cpu, m, p) - expect_eq_uint(cpumask_local_spread(c++, node), cpu); - p = m; - } - rcu_read_unlock(); - } - for (node = 0; node < sched_domains_numa_levels; node++) { unsigned int hop, c = 0;