From patchwork Fri Feb 7 20:40:48 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrea Righi X-Patchwork-Id: 13965846 Received: from NAM04-DM6-obe.outbound.protection.outlook.com (mail-dm6nam04on2042.outbound.protection.outlook.com [40.107.102.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6E7121ADC7B; Fri, 7 Feb 2025 21:11:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.102.42 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738962692; cv=fail; b=g4e/fcnv9dBdd+lQaboo4Qhs5mq9NWGNXqsXzaqYovpjtSf76fFPhNssq/iMOHYofva7P23WdLIYRJqbKHNMw1XVGBe9bo/eMJUthqnsO+IxbaGYhNEEcoCBcQWwqA05IaLIuaO1GwjmVezDtxd5K07/AHCX9Khq2BM5s+29tIo= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738962692; c=relaxed/simple; bh=f1U0Vaev7Kgb3EC4ErBy/w/cG2iSMqnE7fLY1E2uFko=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=bWbAbYvQ8v6j+he1LyZHhl3yyJNfhfqsg9AJKsiUpqed3npsnl67+rzCdZj2SE4a2aTVEHAPxG7ckqxsY8TJmihuVnbD19yt8TH7quD1HDYQS3ISdy4bvNg6gDRM3RMbTjI2mZAjH0tLFjRy8httyimZOHMFIgBpVaxgq8qePCM= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=Gbb0CQYM; arc=fail smtp.client-ip=40.107.102.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="Gbb0CQYM" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=wo/sY1lAEllYhRnQMQl3ZcabmGwvV+x6F1N5WLdtyyd311Om5HjThYeptWLKlQT/hyq1IvsLzAnUvX9/4Y4I+FHbjpojxBHvfaEXhPWl22/GtHpC/3yoRM8MkRrflJJFUGNeykv+BDviVB7ncjCfmQcyriD+RTZJCc8LVQFh7bVBpglhewrppTupTG137funDp82JL7uDsMLW8C3VmUbdz6vqqCMKkLGsQFWIVtAcTOFBycqzLI3DQV+3uvawrbWORCCivQDeFUwM/+pk3MHPJEE6SOwQcmGyfuZCE4fpXFpZLjcfyLFTz1WWBLymxqujxtwJ53i+OUaK2VkkuT+Gg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ZRtNytf1M9PTm5NN6VFbbYEByuOaj7l/V6UFE1j+oj0=; b=iYSceCKqPpyHGCch94oDarVzQjsgCFvBgL0PYjNhmaCVftfq8z1fHidkvrwTc9Z4v7iPxgM9dSwRz1tGIJpkspIVLbJVqTFK/yFzfZFEMAdwZVB1yrUpXa52D2LGI4iZExWi8vFK3OZUNRbZ/pkIxOXxqp26Ha3Bv+wb9RRb0TfMRhEYHHQ+mPTnYkeLVaeDXeHC9hK3DnXw1nDZE9LfKH8ibl3f22wbIg1Y9TvZ7F3z9GgpidilhSXbwe+w+iqcqm/XdMAaBNHqBlCwNlC7gRCIwj9QAivk+ra0QG40WTFBk2YjrWa2xIE80JtWUzFqjJbKumq5WC9Hz+ZFz0QxkQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ZRtNytf1M9PTm5NN6VFbbYEByuOaj7l/V6UFE1j+oj0=; b=Gbb0CQYMLtxBax/WWpr6avRIoZjWuoC63mUpG8BGM9A0yLe+YWosSfrtuSm2aRiMrVG6gsGJ2njnv4jUF2bDhd+970uw7rclvJyu5rZNZDD97dgg3FQA1/0pwFdO5pUhml0RUEfSo1sdkfEPfDJ9IVH/lcRE8DOH1BPChddETuHR6wArCvsnuIJv6pWdkMTk0vqAk3/L2qq3we4bHh4lSlAFxywU1M1H5XxyJavjnkjKHUJyUeK26+f/4kqrKdDjR5vUTisItPoM9O9AhXZJncW2A4E2WgOQXLCEhFC7uAr8iWCQCeqtUqv28lVShlqtqZJty9piEEtItvU+4wmMZA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) by PH7PR12MB7380.namprd12.prod.outlook.com (2603:10b6:510:20f::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8422.14; Fri, 7 Feb 2025 21:11:25 +0000 Received: from CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5]) by CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5%7]) with mapi id 15.20.8422.010; Fri, 7 Feb 2025 21:11:24 +0000 From: Andrea Righi To: Tejun Heo , David Vernet , Changwoo Min Cc: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Ian May , bpf@vger.kernel.org, linux-kernel@vger.kernel.org, Yury Norov Subject: [PATCH 1/6] mm/numa: Introduce numa_nearest_nodemask() Date: Fri, 7 Feb 2025 21:40:48 +0100 Message-ID: <20250207211104.30009-2-arighi@nvidia.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250207211104.30009-1-arighi@nvidia.com> References: <20250207211104.30009-1-arighi@nvidia.com> X-ClientProxiedBy: FR3P281CA0201.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:a5::13) To CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY5PR12MB6405:EE_|PH7PR12MB7380:EE_ X-MS-Office365-Filtering-Correlation-Id: 00d8e71c-06e3-4f31-dbe5-08dd47bbfa4e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|366016|1800799024; X-Microsoft-Antispam-Message-Info: tvEsVVJ+fQP3jtGhWpWEgFE12Mok8SF6YohKH20L0OfrTevYh+RmQTIyQqhB2XQGGagBG3BKUw3PpcFd3QgJaCANhW2rax4032fsogj7aYQ86DlUleRWea3pX/uJCXFByd6Yb1SMT99VbIGVKSOrPZ48pykP53LBrDr8Y9o2hwN7b1xfuoEhKYa00+EGAG46gSLge3u3gQ5/BRRu7k0jacAQR3J0Vjljbz/JZBrwkHmpcIgpgjVnvVzRB6qqGeijylmhQeSKrkT9i9vdndAkVVdz6Vc2epCzF0lvLJq29Su7hTjLae7KIDLWsKSqyGn9w2P7+8HHxGlGKAp9qha1b2Qg0xwJBlFlTsbdFLjL8BwfW5b1IUfB9P6TK4UZuy9aafIjz8GyygGzNFmlQfREV4XQ9tqMczX3ek/W4+4x99FGMS9S7/t+pQCa5seiTUj+ec6Zdfrj+UWnchF6R4hgBF9chrS0XDizAW7mKpnvLBrDg2PMmyyhDEiY4mtJfNW+gmk5aqorJtcZfLKbfFKtEnsB2BvK8Sh40K6Pc9/Gm7CXpkF+qvevmKTJo6QZCNUyRHdL8cgAYHAPDJgST6Aw4VFJCsxJa8GW14dSiSvqruyyxcRrG0e9Lfd6dWuQef0XcSh3EVWn2kUxHnXDFGihCjFxCaLZgzOlnGlqdMiwTKr6XZMn3Yz1wJ1Vc8fyawE1FBWyZXYTRxV4Lb3t8C3c2ElqV+5NHSbUIvFMgM9ySoWrDoZWPBuSuP5PL3ufSN5xMC3kYKDyGT6gzQjmj8sxhJ8t0QLPGdW6qYo14/oW1PBYRDhSVOQCttwcjfKJK6MsBd1HxiFxCXLSqQoiNFGAAPenl/+ni+Wvju865MhVVnppe8YhMDcpdZb7x02zGOauYX2QF0aitCS+kJo/Uq8D3gemS5y99jCk2G+JTXf/4b4ugUzBRr6WplMAM8tL2XG8GPqrx38D1m59AjtI+LOxSbOmh3x079htigpWOCJMBCr/7s77JARJwIMwas/da56zyZ62pkmrY600wgX1DcyTyp3PYYd0DR7yby6enJfNUmfr0PNbIRjSKNemQl0vau5SIsHKru+2jn++uPKx2mxppRMIRpQ3vKVTmk3fqtDfxRTvepu5mT4ZmPKFJ8mfe3PkQhNhfwkKRPLRHt/mpj26WJ/F/dwY38DTO1nKZTC71oHeFSheM43UTc/ENnOfMb1ybN2FFSVPP4Tnl90xCOndAQ2eEYrgDUAqUr+VGoD7q8/lZBP2OVUDvy11DESY8BW6zrE14XoNZWPNMcYSLvHgj4CmnC+/16UiBkub4EW8wWWSVgoxhD9d0J2iavAu7/bzb+5PcyNeT2Tcn/XSPJ2zWdhEeK9v2/6MDe+dw+P5BjEQ0wi/fD0D8I7IJWLgeOQZ X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CY5PR12MB6405.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(366016)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: PDv1MNElaprqlfxseDBny46JX7fdI5jc7IntrAzqzUn8b24pJpXz5Exv+15qO1lYYbCtYrAWsCcgOcc4h5G7BJzsiwpc+Z1XwYyPJdFueyxdsJyeeIb7eFl6N0NoBMZq0JHBD4vMMt5QYrxnqfEFX4RLrQjnw388UqGkbdVpPo/fHa0KiaNa9uRpS9N7zLcfCBF920I5BEtMT/knvEfSI14fUmUVyDhYXW9IQ3U0c+q7LOHkdZcfbfb+GeA/wjKHk9mV6UeFlqBLkNVwaKAtb+S1KwWqRiTzZKKGGMQL7pYDLAjQNakZM2w5x3pWPu9shmK9mPzsh97T9lHdU6HUgZYbrGUCG7jmfVC/tMnJFHJ5c6eE0R2PWtwmou3t53wpGc13zHWaeLipVCP68bvPa4g9uOxyOyXqqfimPhV/RleL8HoOimySo4rLTeUIwUfboMlQT1KVxBt5SKxhH3yAVJh9czNp0rhQmykC0pXeUGptwVGb2A4GNOxFkBZHUJbN9qfveI0kPuuUT0T3EyCuI47y8AwWtgjidoZBS9gHcU+w9nzXk0TJzgesi371/K84W53tyX0eKlnxwDcP7mcwiehH6FxHEwg626m3s8Vowj9CTzGfRQyM90QJglI/vVsTByClCLcaIg9xIRPLuoTb+q26oHrx07wwONVsYHgTrofd7iuElCCF92uzxuhfSftTU7pE5l+1DSHUDKY57RXlxOJ+qbBxTqB8z6iphjUbxzqyt5ld1I04sVOQnCpxiyN6I9ljsGWXLseBs2YrDLp80SA2WSHS9sp8KPzAHsL0z0M97lEsP5h5azKkHh1gdV3SrVCfboSI9UQexQ5IemLCpAorqBf+ZeculHluRm0vhH1Osz+TJ6yiaIg4n1D19+78NQN1A76L3NXtoWHBxA2IbVlSVmL8H4Xxy2knZnMGTCenwhY/cVJ01+ADoOHI+vhWWP0oU4WgtQdBmhgjPuTtgAMG9bvHcFqilNpByET4AP0HwmXyRYpyVcVd+PPF1g81/UYvd8kGTgNZ2D1j3In8DttPYaeOHGoMdewsUyyoPkMOLqOtnG5bl2z1fc18I0rFVkctAV3aVBYbYou619bUX4bTX1oqSnpUrvaI2Yn3cBy8k9nSeexvJkSjsW/Gdn7T9BOw6uSnnyXFr3Ui32oIgrQclvKyWlEg92YaCYySr2zIWqObdRhA1FDYBjk51sh1npHt98pQHFIHACLDrF58a2nCFz+NyKUkeqNPucZkyi3XKd1lyXxmNBuKAAwMlBxY1dd+d9sIaZYCnyPBvOf3uBNkXwToAOUmSTs8EuiaA7LKlrcg1lwlqyEnB8P1NoVfjFiFHWLuzh8o+S+sfeFLimaGyvF/Mz4pxCogS3AtHH4pGlXclu0q4WtNhBhmIKImM28TDg8Bm7abJB9iI5t0PTgQI6RAZ7IRwPfG+Z2AA+DZSkntqMse+Nr9p63+9m4FlAW+xc3Pp6lGcncMl9bb0oR3NfgP9arQP0zeWd+cR1qxWUVtVLL3EtyeUwZjrOgC+QGN7ed0+wLZjTlY5OgMton55kZnOANLzdAonn4F0wtl9T6eT4vWdAFKO1bhbzn9 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 00d8e71c-06e3-4f31-dbe5-08dd47bbfa4e X-MS-Exchange-CrossTenant-AuthSource: CY5PR12MB6405.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Feb 2025 21:11:24.1861 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: vhu5GOO3+byh2IROErjYphS2MS+VJax6hZRNKT9SpY4oignBszYhkRP8D6zx3io5hYNQHFmbUAb8cyT7oRZLpw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB7380 Introduce the new helper numa_nearest_nodemask() to find the closest node, in a specified nodemask and state, from a given starting node. Returns MAX_NUMNODES if no node is found. Cc: Yury Norov Signed-off-by: Andrea Righi Signed-off-by: Yury Norov --- include/linux/nodemask_types.h | 6 +++++- include/linux/numa.h | 8 +++++++ mm/mempolicy.c | 38 ++++++++++++++++++++++++++++++++++ 3 files changed, 51 insertions(+), 1 deletion(-) diff --git a/include/linux/nodemask_types.h b/include/linux/nodemask_types.h index 6b28d97ea6ed0..8d0b7a66c3a49 100644 --- a/include/linux/nodemask_types.h +++ b/include/linux/nodemask_types.h @@ -5,6 +5,10 @@ #include #include -typedef struct { DECLARE_BITMAP(bits, MAX_NUMNODES); } nodemask_t; +struct nodemask { + DECLARE_BITMAP(bits, MAX_NUMNODES); +}; + +typedef struct nodemask nodemask_t; #endif /* __LINUX_NODEMASK_TYPES_H */ diff --git a/include/linux/numa.h b/include/linux/numa.h index 3567e40329ebc..a549b87d1fca5 100644 --- a/include/linux/numa.h +++ b/include/linux/numa.h @@ -27,6 +27,8 @@ static inline bool numa_valid_node(int nid) #define __initdata_or_meminfo __initdata #endif +struct nodemask; + #ifdef CONFIG_NUMA #include @@ -38,6 +40,7 @@ void __init alloc_offline_node_data(int nid); /* Generic implementation available */ int numa_nearest_node(int node, unsigned int state); +int numa_nearest_nodemask(int node, unsigned int state, struct nodemask *mask); #ifndef memory_add_physaddr_to_nid int memory_add_physaddr_to_nid(u64 start); @@ -55,6 +58,11 @@ static inline int numa_nearest_node(int node, unsigned int state) return NUMA_NO_NODE; } +static inline int numa_nearest_nodemask(int node, unsigned int state, struct nodemask *mask) +{ + return NUMA_NO_NODE; +} + static inline int memory_add_physaddr_to_nid(u64 start) { return 0; diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 162407fbf2bc7..1cfee509c7229 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -196,6 +196,44 @@ int numa_nearest_node(int node, unsigned int state) } EXPORT_SYMBOL_GPL(numa_nearest_node); +/** + * numa_nearest_nodemask - Find the node in @mask at the nearest distance + * from @node. + * + * @node: the node to start the search from. + * @state: the node state to filter nodes by. + * @mask: a pointer to a nodemask representing the allowed nodes. + * + * This function iterates over all nodes in the given state and calculates + * the distance to the starting node. + * + * Returns the node ID in @mask that is the closest in terms of distance + * from @node, or MAX_NUMNODES if no node is found. + */ +int numa_nearest_nodemask(int node, unsigned int state, nodemask_t *mask) +{ + int dist, n, min_dist = INT_MAX, min_node = MAX_NUMNODES; + + if (node == NUMA_NO_NODE) + return MAX_NUMNODES; + + if (node_state(node, state) && node_isset(node, *mask)) + return node; + + for_each_node_state(n, state) { + if (!node_isset(n, *mask)) + continue; + dist = node_distance(node, n); + if (dist < min_dist) { + min_dist = dist; + min_node = n; + } + } + + return min_node; +} +EXPORT_SYMBOL_GPL(numa_nearest_nodemask); + struct mempolicy *get_task_policy(struct task_struct *p) { struct mempolicy *pol = p->mempolicy; From patchwork Fri Feb 7 20:40:49 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrea Righi X-Patchwork-Id: 13965847 Received: from NAM04-DM6-obe.outbound.protection.outlook.com (mail-dm6nam04on2042.outbound.protection.outlook.com [40.107.102.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 509B41AC42B; Fri, 7 Feb 2025 21:11:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.102.42 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738962693; cv=fail; b=rJ41UQdDSLTyj8dm2eweCGV819NEI4k0svH9ZYUXNzzh5W9bu1+pRX/4upJAjtwerid/IwJodT+pEeqQcqEcMg+NM7/taPAc2rx42uKHX+Oxavs/TktRX5uu7uEhjjfebAivXyU1KEt+KC6DA2PLzxq41HtIv1bOkyTBNCZsItc= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738962693; c=relaxed/simple; bh=hPrN/NXYf9iyAOTRtvTf0OYnWVi26KWPiywnj21bTmY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=OieJuZDjh2xGvXiUlJAoXeLWHyVaotjIiUMSZaVW1xE7o71LRmgRhCU9sNRXHl+zR643Mk/yKVp84HxpNnYQ4Vvp4kWtT34SCv3HbH02peaq4XfYFs+guvxq9KXmZBqAmDTy7hqzCEKs5MTNTtwCgGUPfZ5aoHOwo0F+OTxr12A= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=tU54VVju; arc=fail smtp.client-ip=40.107.102.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="tU54VVju" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=KG+O2HzoHpfbso6fy75ZEyOW8mKLcvzLkRm2phAnfl55Rm1+vpxsqEECk7/IW1bCOq8Mgtkoi8gEbqgy67y8eeAtWpTO5S6Irf37AURdz66PC2NmCtIfVzHYQl235KP36e2sIJC0qIBTyMLnlL5oohZLqCTjIQGqEPLOUkMIUOy6QkFoQ+PEPmEQkzWIlt1Ez1EeXfmzfMFQPu3GEDiXG6r0fIjnOPiNwGXeYN4JmI0w4FiDdxxPtsZ+wb0L78SZSjelaYyf6d73glWd73rF/Eo5c7+uDai2R8QujRBVOm8I6XhUSAvrofHHOe5lyhdkyzA71zBzGhaW8hBlzFkblw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=V3vn7S5h43JQ7Wb1tCw5bcivFGNoOZwO7Y1hCvIEl/Q=; b=NNnxEyT4571pVQntmxFFevIljteKw+vXOBKBHW6bKXB0qS6BDG1Ep6eU/9bVm2LID4sHeRozehgUUS77EwNMOUg8U+Yx7X9Og+InVQaSdp/gaV/c9qvGVJFMuseShWlJHEGynedWJ803n9wFo+Q/8PkfA1jikkCq/0kIF6g4EfRDJCillybmBrA4V3G8RTNJQDMWX9K7hUaF2VwBRuPMSC7DG0DPBYn2TMMr9XSFCODdzMizNhFWLKwvFLpqfcxNB2l7ci+Cz08YGK7WmfQGEsdX7XCWlAVXjjaOVWSk4ov2hrvltwTqP/ssB4pkkqHeWZ7fpEVyz+dIbUwlis/avA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=V3vn7S5h43JQ7Wb1tCw5bcivFGNoOZwO7Y1hCvIEl/Q=; b=tU54VVjuPZp7DBo1KffxfRMJfDsF4BgQUbuvEPEKtVmG8n01btEtlr+6EjAX0eOelkumru6aUk7Wnp+pbeo2htQUkhqBpYk0VajkIp6ChnzQ4drFsfEEZkfu/mP2oPalW9eVYsnprGGTNXsX0LXBWeVYCSiGO4TBSb/iGWxwNaWjeOXFK/AxRmoT5kBcihChXouC6XIjBhEZrVTae2f+wpFVvGxf0AC9yTG7c0of+sekOI6jWw2sQS3XFwBm8bS7x6s6/14Bo35RFKMypbUIE9pFTWHwVVQurIrGsXjnHqU8WDhqqL0NqJHXWDXAIRkupCIqFA3d3K8LE85sZdSh0g== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) by PH7PR12MB7380.namprd12.prod.outlook.com (2603:10b6:510:20f::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8422.14; Fri, 7 Feb 2025 21:11:28 +0000 Received: from CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5]) by CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5%7]) with mapi id 15.20.8422.010; Fri, 7 Feb 2025 21:11:28 +0000 From: Andrea Righi To: Tejun Heo , David Vernet , Changwoo Min Cc: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Ian May , bpf@vger.kernel.org, linux-kernel@vger.kernel.org, Yury Norov Subject: [PATCH 2/6] sched/topology: Introduce for_each_numa_node() iterator Date: Fri, 7 Feb 2025 21:40:49 +0100 Message-ID: <20250207211104.30009-3-arighi@nvidia.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250207211104.30009-1-arighi@nvidia.com> References: <20250207211104.30009-1-arighi@nvidia.com> X-ClientProxiedBy: FR3P281CA0169.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:a0::8) To CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY5PR12MB6405:EE_|PH7PR12MB7380:EE_ X-MS-Office365-Filtering-Correlation-Id: ce99ebce-fb81-4ad0-0cbd-08dd47bbfc8b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|366016|1800799024; X-Microsoft-Antispam-Message-Info: BTlGdNfw5iagx7bYiMUHsBZgF+8PS3SDvN3rmtacGzXYjxkrenOfPNGVhZuz9aMVM1ehLIqYqALKUI+ROAaEnm/0EZDkUHXm7uwt2V7GvxG550Fq7687lUVKaMY+24k9SwQnfbTxWux9Prfqi1g6symu0rUJY+aJ83QWrL5EUBnEeyp8T0oRW213sXuKmgpo9rmUHcT0sOnamE3gFI1N1KB0GiDfB7xzkaz8+TkIcryAbQ60fgPO/UhThdgWRCJ8G6p6fsitmr4nxBHvvrRFd6Yz4B089KdrIGwZMoviUv2zVhHgwwy/cSZPajb5P4ELimXkBVzCj7W/S7N5CvvMnvgMkEjEJ5rqvfphYnU53PLHmUul/bCCq5+YXIQLIg2p0XBcDm58wXwuZ2lp/HjNGj0Njl+FGilQFr/qrI0q7rPdm/rg4kQaS6Mw5q41TjU5UderoXStzJfRzhE1AthAkuNlGtXE4ma5oq6rqhrp/mRROdmwKuV/i+Q/PE7nNsmXR1JIAWrMfncHPQ/fDP25Y1lpUmsFOkD9B8Zb1EJVjvgmL+PyYJx0iasXfZ2sJk9Q2S6aEwiJsQQrWLRW948/tSjRh8NhfBt1zgiLiiDarFxRhuqA0537GaCR5QQXx7AhtBYxSW3xjRtuIxwZBCFsBJVb5fAmH6Ri7gF3fMGdKRCd0A+zQUCvlVFr0i2UaHy358Hbfl1qXdzP0gmVELG+kjVr4lZPk0nCI3+rZH9lG9+VG+E1W5z4H81ie7cfNXrcmQCSsy9bRP89YsPyUaKfuAxRqXydChn9smB3Au7/YVIV65anmlSSqeKMuHiw9Vdmtu25hpxK+PhkgpvvAEaKyIst6Vmkw+r6taZy7d6zUzYm+9BPWUgl921mybcUpb9sEhWll1Tirk0peUeCaMmS0h4ICUDfaGeagn0oukeIqToriuTucPoTbva6/F+irawMgkTJlE77IY1qAryuJ5vMSuZ6YhzRAYh/OXRv87c5EdKKlXibHDRKZ/TcrHAB8O2zySQ0w+GoiMXeuMfYBJTQfcUtYy4oT4Ju8W7By/zyQ5cBthHuvpXmKrpx233SDE7SBIprernlGBuo2DnkL3tK/lBP6goKpxtk/xHFi9V0lY8Jm6JZi2v3UwvKC251zXRoE5TgGhifbsrunHwqF/FUwyq7SmhHxRFC9wIuNndH3j3ShYIudHZIQYp3e88S8g0nLW6w5neQcyuzIWsMIQ2pShdLvjzxp4Yxy24I/Kd4KAE4bc7WCKTWiYvsAdk4DN/pPLk/U9ChL4CxAEZO6a4vleBSuW07GBxKL5Hh3WfzSuKpxqYIigiurxinFyDZD8x9sUT9PcuwA13dFqjSEm+ri0Pv01gWiowqXHmfUcmGeP2GNhjFWJk7BZsi/PS0DwvQ X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CY5PR12MB6405.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(366016)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: CvJIKDDpAABgCaPO5n1U6r9KCwHU74tbphMuVjRuPuPG8iXgnJ9nqWbTeUAYzJEIrNXcKyiFOZzUWsZrE1Z1W365v/ufZ3mBud6E0e1lrvsAd8uSUQ1oFbw1BR8T7P6v9B7IeFcdTCvcrCX/boCrl7uwS0pSF01di0E6PnSG4JpQGgTBzHyEm3nBlVdzzL0fHvhFe6trDuiihU7ciXQ34vl1TnHhuZgoJuUgzVdYGXYplezSAu+pF1ERu4SRX4KZ+srQXY5eC891HPcyx+TpaZG3s5pSqpZ3vGv41KchyyW8iQVR6JYQ8/Roh0ALvbNj00QIGAttzQlOiRt5rVobF1gCwW5MCELjCuSg983xCVtxwe/yEkVK1ZTG3TgjIurHzv3Zbf+7okzLfPUotRfNBO6TlzXBzMu4m9mDfmpLZbbn50W1UICpCqnyt1/2o9klDcQXijrwkJlkwTVRC0U0nliI5Ghey6quIPiQnXqlpl7Fr5FkLWdae0jBiQiRy/LzO8nywvb9gRqY8F47CDRDftsttR7ba40TUqD34AAa8gmtmKan05bMPnhy5X+jN+i5LcajKbrsGaO+dJdpcMMOAjViGR72f9EERYp5pdRb1EuUPlvElRPBqF0jGRB8Llccj2k/ttEfkxMOfoRiqciiTbxS4NZUWB+pqfK4oI+9Co49bvfNLWWIRnm+Evr5cS0LbJjRBRvWYaHSObj+7G+rst6N/f2jeXiWWwd6sSVbZMOUhiBua0j07A6iVwGL2APw/5lg8T1spYpzKam2zNJo0bfant66Js0Ikl6SblrLtVkU8lXdVJPMuBib/ND32R1UEFGyy2mk1LzKO//dR8IviSl/8eTla3iUp57Hf0as9hfvuu3PPszttkiBt6Uz2LltE+eNjw+PySCE5hR09R03gJt97D3YbkfqMTH/ChrNtSWbj544FkaRyY/982UpTYHFf9DXmXzZzz1axyKOWF6OEVW2mg1AYGadTUzMYMaPoi3jGgnRr8JxkEahuc6Iv4mdh9k2/oyyY0Yp9o+J4TNzDSZxiCELP2Jz15HRUe/5nqDg7pZp/N12tXJl1hmFBBzTevv5omJQ0joJaTSvkMlCh84aq4uXkVJBiK6F9e4KwqrN3XGoOGH6Gmk4K+7reJ3GzQjET6MZxkEqB4GnkANCbecNoytR0sfkhnpdepjCyGd8Bdiw1QfZ4Q0s3Q21SwFckgvRBQo6jCYD45aTHyDXNvMPnWqXwN1qmYRlvQI3voE4cfsP0rWIOP8UZj7XNKrqrmxkOnNnuzA4UmYlEua8psZj3djf8hcv4fh1fKocy/GuITcxl+QjS3ZTuKZjM5X1F5NO/sYV5n2Koi+vT+q+LpXKpvpp9lSZ0HedlNIhniW0Y12yN4E98lVZwCSjl7c8JPOKtjT/TDRdL8QYHA/UnRHzrRc21klfJrC28BcRyIhLRLooFatLXmfCf20R07NLpGsNdtZQ6OQNZXKNGsFe3A18vRou173v/HAy7hzLPGA/CdAYF9ITLQDGIAAU9oOcZGHhgCXVM4CF1yQOj0mHjxLeju6hDDSD2tDw0tJtqPIoh7Kt+IgHD4Cqr7JGkPLY X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: ce99ebce-fb81-4ad0-0cbd-08dd47bbfc8b X-MS-Exchange-CrossTenant-AuthSource: CY5PR12MB6405.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Feb 2025 21:11:27.9943 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: I4I2EHHsOq+7Ba6arR+VeAVbMQgAc4uNKOsAMGghzNtjy4tiKiiGv85vS1mT75x9HySWj7daaDpnqPq8nbtP3g== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB7380 Introduce the new helper for_each_numa_node() to iterate over node IDs in order of increasing NUMA distance from a given starting node. This iterator is similar to for_each_numa_hop_mask(), but instead of providing a cpumask at each iteration, it provides a node ID. Example usage: nodemask_t unvisited = NODE_MASK_ALL; int node, start = cpu_to_node(smp_processor_id()); node = start; for_each_numa_node(node, unvisited, N_ONLINE) pr_info("node (%d, %d) -> %d\n", start, node, node_distance(start, node)); On a system with equidistant nodes: $ numactl -H ... node distances: node 0 1 2 3 0: 10 20 20 20 1: 20 10 20 20 2: 20 20 10 20 3: 20 20 20 10 Output of the example above (on node 0): [ 7.367022] node (0, 0) -> 10 [ 7.367151] node (0, 1) -> 20 [ 7.367186] node (0, 2) -> 20 [ 7.367247] node (0, 3) -> 20 On a system with non-equidistant nodes (simulated using virtme-ng): $ numactl -H ... node distances: node 0 1 2 3 0: 10 51 31 41 1: 51 10 21 61 2: 31 21 10 11 3: 41 61 11 10 Output of the example above (on node 0): [ 8.953644] node (0, 0) -> 10 [ 8.953712] node (0, 2) -> 31 [ 8.953764] node (0, 3) -> 41 [ 8.953817] node (0, 1) -> 51 Cc: Yury Norov Signed-off-by: Andrea Righi --- include/linux/topology.h | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/include/linux/topology.h b/include/linux/topology.h index 52f5850730b3e..09c18ee8be0eb 100644 --- a/include/linux/topology.h +++ b/include/linux/topology.h @@ -261,6 +261,34 @@ sched_numa_hop_mask(unsigned int node, unsigned int hops) } #endif /* CONFIG_NUMA */ +/** + * for_each_numa_node - iterate over nodes at increasing distances from a + * given starting node. + * @node: the iteration variable and the starting node. + * @unvisited: a nodemask to keep track of the unvisited nodes. + * @state: state of NUMA nodes to iterate. + * + * This macro iterates over NUMA node IDs in increasing distance from the + * starting @node and yields MAX_NUMNODES when all the nodes have been + * visited. + * + * The difference between for_each_node() and for_each_numa_node() is that + * the former allows to iterate over nodes in numerical order, whereas the + * latter iterates over nodes in increasing order of distance. + * + * This complexity of this iterator is O(N^2), where N represents the + * number of nodes, as each iteration involves scanning all nodes to + * find the one with the shortest distance. + * + * Requires rcu_lock to be held. + */ +#define for_each_numa_node(node, unvisited, state) \ + for (int start = (node), \ + node = numa_nearest_nodemask((start), (state), &(unvisited)); \ + node < MAX_NUMNODES; \ + node_clear(node, (unvisited)), \ + node = numa_nearest_nodemask((start), (state), &(unvisited))) + /** * for_each_numa_hop_mask - iterate over cpumasks of increasing NUMA distance * from a given node. From patchwork Fri Feb 7 20:40:50 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrea Righi X-Patchwork-Id: 13965848 Received: from NAM04-DM6-obe.outbound.protection.outlook.com (mail-dm6nam04on2042.outbound.protection.outlook.com [40.107.102.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1E6941B0406; Fri, 7 Feb 2025 21:11:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.102.42 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738962695; cv=fail; b=cycVPtkuOVFoje+OeZ0DMoopklyuhW71JmhEHUdOYc2dgjrlLdcAhv0G0fUwwiQzBY86dub3WNmU5vhxFWtDXlP++6IOejr3RuqffjPPr3+0eonRM1UlmcOK+ObqI7xG1G71sg9eq/mfZ8eDAAy5gZ8llO2PKaCGkmBfICQybQs= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738962695; c=relaxed/simple; bh=4KKf1PGCiOadWOWrEekrpAV8LWE4UxVYvoraeAmePpw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=Tr1i1j7uuw1L+6C2WoVsX1XAYnEbAhsKOfxUp5VSWjmI4fGoEXTI9jih8d2OmEFnTksHDZxeIf/8dXqVtDgmVIZ0Z/LLhD1/7cS3kUnyKrmp0xr4ySh+3bDUzTKKBPzi3lzbcd/PLI9Y1XDAzC66iqysq2c7CGem42zR4wYW+u4= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=K7lrtj16; arc=fail smtp.client-ip=40.107.102.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="K7lrtj16" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=r1CPhd/Lx0+d/2KM2bZFfpccn4Tqvxh3B6+Duk/sU9+fvCUsngX1zZTRwdf0tMuyP7NhFKqAw9xQeQJGOjvOhsGR5YmoI9a/Hv5LR+1xfi8JtwdGRGWMHxgzHhTKJh6WrH+X8uaEtTNNPRVnJw6ryO5yc09Ki+jmIGvopiN+g2RGb5KCpgpgb/xNXuw0Ix+vUQzvlodMPhk0/t952uZ6f06HT+91TG3NEAPBxyIGGrZ3LE4kQh2njsRFOYH9Ce+UOXLZI9VMvVgNpuMfq67RlZ6Lcq/U2xy2g7Kk3SHC8dKro4QKcXy+JsCGmrUtVVttRNIQuzzjwlBWwYXjRS6YXQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=7h3/imPYeugGLGjqlch/mf4pSavgICGPxzKjg2B/cYo=; b=iAwuPawW1qgLmvSR59p/TksqWOM+xntCKwK/O+f+7/EzKsoXpwC8VE8rJw1gJJWJldAqywunEtBXfzcsmL+jMKnMeVUq0AqBntzgMFgdLlENERJaVUBNPvKnPAdepfdW9OIIxjXqYGKsyfAdHWS7AjTlHRphF2LuKGX5YU4D+UaJmTVaHxt4TJ/Iff/oc3R+ckoUeD0Q/a/mOBe/fqvo1XbtmqgGgtTqaX0zft9sA0o29BQQSIf1JF52kpkbLmoTU/VXU7Q30TSF5grkJtinSXsXG4J7yjmM+nJqjaiGD9Nc1yI7V8UEOMiEUWqVhZmJq7phg2ZJhnNU4geKRXdhqg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=7h3/imPYeugGLGjqlch/mf4pSavgICGPxzKjg2B/cYo=; b=K7lrtj16xWXbVsiKdPNbe82HRWVPV+Y4imS1+ei9YtjlW/u6zYY5ByZvTO14/l/ICfv0sAjTTxepqd7a9ahISj+5MpAGGidVYIVdljG06bhC67/A1gYWCZsWvEHVko3svv5DQyNAlGgy8Ix0UoG5bucJRJq2uTuqohOzNcmbs5hl1diAXZBPNHjJ+qf148rl26br4to50UbXcmmSuTTOt9+q6GlD79E1sdT6C1mGCGYYetHpnFvH4OiuF2W1grPT1+KyztNe1QbnrUp/Dy9/i139MfU5QaSzhYxX9JZLzxkFousKPCHvHfTjZe6YqQbz4nVJSO4zxILEQLq8V7oAdA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) by PH7PR12MB7380.namprd12.prod.outlook.com (2603:10b6:510:20f::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8422.14; Fri, 7 Feb 2025 21:11:31 +0000 Received: from CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5]) by CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5%7]) with mapi id 15.20.8422.010; Fri, 7 Feb 2025 21:11:31 +0000 From: Andrea Righi To: Tejun Heo , David Vernet , Changwoo Min Cc: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Ian May , bpf@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 3/6] sched_ext: idle: Introduce SCX_OPS_BUILTIN_IDLE_PER_NODE Date: Fri, 7 Feb 2025 21:40:50 +0100 Message-ID: <20250207211104.30009-4-arighi@nvidia.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250207211104.30009-1-arighi@nvidia.com> References: <20250207211104.30009-1-arighi@nvidia.com> X-ClientProxiedBy: FR3P281CA0177.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:a0::14) To CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY5PR12MB6405:EE_|PH7PR12MB7380:EE_ X-MS-Office365-Filtering-Correlation-Id: 9ceabd10-d3ac-4720-28bc-08dd47bbfee2 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|366016|1800799024; X-Microsoft-Antispam-Message-Info: dFM7TXGiV0/zzsiJUSHhd5dN/s8J9veZI1yqcVC8y0ZJ+fmzSOC3feeC+o81un9ZyQoQtFvbdIkC5oJbPPCmimXdWOhL6wX85xXMGdskmyvy1POLQJqqMPTkMUncbyJAatGKE5TqMzKVsqAO/AUUtXl+nxrjZFdL03QbsrWTpXwl92Ig2+DsV+xwUvkZG/faSCcjaJp4ziTNv46PVU8x82H/CXPLaBDQ0R7Jawae6TXlGPkXLG7LafY+zTIES5615spg0vnocZ592UUFcGjVURYelqzDtDeyU9rd726EEZfh9fhAGxm+Qi+r5E2TTx2CU/aF/XyWTOkjfZchCbIWPW2WML1O50aDeggyQPlI/wcpWkTh2Ill+DG3zIETsNkbyHcFjX6xy9BRto7Udcs+wFq0v0eXFbP+WWnwW3uQpnnCJpP16YKWlSsYiWF5SyhgkuXoKON43XMt3kEOszpafQ6lSLaD0f9LUOteMEsZb7eB+iMB/YeaJydYHFvIsQCoLq8riFJqxscqd7vIah4Pxn7a409Hw+PpouHSHZyN+enVoVwPRvqTHzOIlA3KG66/z2ddUVPlzfejmyXYgg3VcUTduRpYdedNvDholFPW/iu34Lvu6L56QkHvvis4HpneN22deABfwBJ6ZL42J2ha9/ZZFid001qOkoEu8ZC44s7f03vSUq1xvYiwHOMUPN86uMWB3OWQcZMHmrD390BDdfHSPHWhPvXBkbbEc1fQ1/9krr5rVm79DVftuLZEl0ZwRRVgxJYa6qAVDD4DCtT7Lf6YUwpbVofztY1iQEA/KIHATKdNcuAbqTVD2IjT7+g/+jDji0DHZjUNb97osm0Jf+7xyMWfvPBtQOLjtNKyov9y47aPLoqcliGCA+AWfEIlUUEDGRp0kkfQHiz4RimutVp4YNMuNsSWd5naEyn/YhSFplravfibr44sfz8cBA7ejS5CfYC177N8KbmmvpRotnh5S9qFi/RmCbYemKKFip5EU+lcnJgndU/oIWEct1qwNMvnzlNM5zdSiDLHKsw6JrlrTUSasNBSziAOLJD9Y5SzI98gS6eweazaBXJDoEq/XWJUbco7FbECad/FO33UOs2KoaT220900SqePEtju5vqPydr8piDHdGQHLLVlkXTJCd/4o5IppMxeNejO9pMIgm3Wz96r97Wkk15snAYRx/s/GZNEvffABiSx2Sa4Vr7JkaKeR4Emmi9C/06quzA13ELijcp83piqUth+OzEzy5PZ/0Yr4hoDQBg9KVqJWN7nk9E9caCZHX26C2mAvuArivRoWtro1vvtWeSn8H6iPbKueYmgQTkIz6ZRz9VaPGEMNRJYQxGAGvXtoj0/i1hOdd1nMOEjuBJgSMgB9zKGuO9FzET2U7gmfxG6bef9tEU X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CY5PR12MB6405.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(366016)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: jiXXM+jEyXK2zoaRiapwmCgd5ljyOU9BAiWElu0FhU8SlpH1c96SxavmeSl8dZowY54CWwqaS3ESMb+Py6byyvZ5fZM5NBjPLsb5pez/SWjvCGFJ1HuSLgAxu1FZfpf7DlDnkC+FnQQSUiDxzCetRd1UU2Am1bEdDyU4vPLBX9PFQ4pic58HBlGY9Rx/SqUwslKteEfXOOGn+HhFAqIQ4dPlLQc+qz1UYBoVYBZUp8kcfesUptyiNY4IkUUxuHot7khNlmwwP1kWhSYmKj6d7xoUrwgOQARFpxATqk5GQpIcTuXowhbTBnuNMB3FnmoHxJ4ChHT6lHR/PqRr2ZoExmK6VLylic2ZJz8QjG8xMTqC6aZU3NlY7WwpsQzwR8MIxt4635Moa2fL1HUCyOE98/HOxA7jI/XKJpG2aybkYPS3YR/Gwo3IMwytNx1piRBnaN9+0JNdhdrgkEgpuWOBlEOtCpGqZvvQtdK0eMhronNmUO0DXcMzGAyEIqCZU8KYR3n8WdQVc29bV8R+E4Vzuhpb4MNRi1dBhFoA9YicMRDUfbx8+BDmPSSS28soXEEhuH0yYBpkkrWY4uCRkx1jXC5PVvmDERBy1zGFnO7Zj2qdW2evCUlFgRP3SV2VF+o1miv+SN8fVj159TVbnXrsLzQ0pAKxXMEVsBQwJaMuU8NfYlFKoNnCj+tvetJVWmrch/IcQDMNyk7xn1dN9U0PJ4HN8EDGPz/gvt/OYQnOWgiW8Okf/IMcCyngDZFBSCaEVm96TjD/aYnegA6hUsUh+KAAtEuM/s6RmmdgqvbrHKC357TC8IRkMywRDH+L6rwC+Lb7nMKTD7M7GcvT8XzkLCNKKgeg9E7sV6qD2K3NaUYZJbf/o3awweael9faFi4UOoiyJm2sIu7U8TLytSrAokOIvQ179KHwayY0nlLm+jsdD1M1AdXY6qluMqNvLQzGfvVSASKC0w8hAj/Sr2EO+UiszyOjqPGq5T2NdtYWt0u4la1d3H7uEL3hmGXE1xbb18lKhixcuqQ4LLapDsbL7R8iblxjYuiu+Rm5B+mn3/qNmepD1Gy9AuMOlDYQMBStomNFUjJpq25ZtXJOc0iZJ7NPK8PwoF70YdX4gt9F+19oPnRD4Bo55LX+qh6HcSYcfcVXjFyLozfK9IeNtbSPHMu19+DDYpIAqHZnQcQVbXYztBTci3xUHZ1zTRFZbtqxfKRY912Fp97hRCp0bSYpuPquXZPLUc4sAb0KYxnOXySw2GEehpF8jAG6KaHHBgc4G84tBCj61pzuvbP7uw68NnHGh3nZm4vB7RdEUNK0KQPsOe00F/vQvN6CzKHCsyX18p7Y9N0pHLvvl5Xeky4x0oGoii5StRG2ypE7yaFTfclFKUef9KrpAJ/ECx9wLqJh6qLVeA+QCeLZWQIwAiiHegJICNnbN2EdtHA70te89G+DpRUwntYn8tyslqGdNA5nGLmwAnbjEcFTVzgG7EJREa121fbnFwSLShWMgFym8POCeHGHh0UGPVYspViTIz+1saK0s+RUBdpsGqAGTD7itbDMU/zgOPm11VI7tP5aORbZS9/fBaZI2Zm3zf5MSAnj X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 9ceabd10-d3ac-4720-28bc-08dd47bbfee2 X-MS-Exchange-CrossTenant-AuthSource: CY5PR12MB6405.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Feb 2025 21:11:31.8036 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: D2O/D5N4bEFyZpNO0e27k9kdxi2rGtCLTk2ZKT8jckrgAa6yxmXFO4Y2Kp0+niaDz+OokkHpQD3efGZgD8iLdA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB7380 Add the new scheduler flag SCX_OPS_BUILTIN_IDLE_PER_NODE, which allows scx schedulers to select between using a global flat idle cpumask or multiple per-node cpumasks. This only introduces the flag and the mechanism to enable/disable this feature without affecting any scheduling behavior. Signed-off-by: Andrea Righi --- kernel/sched/ext.c | 21 +++++++++++++++++++-- kernel/sched/ext_idle.c | 33 +++++++++++++++++++++++++-------- kernel/sched/ext_idle.h | 6 ++++-- 3 files changed, 48 insertions(+), 12 deletions(-) diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c index 8a9a30895381a..0063a646124bc 100644 --- a/kernel/sched/ext.c +++ b/kernel/sched/ext.c @@ -125,6 +125,12 @@ enum scx_ops_flags { */ SCX_OPS_SWITCH_PARTIAL = 1LLU << 3, + /* + * If set, enable per-node idle cpumasks. If clear, use a single global + * flat idle cpumask. + */ + SCX_OPS_BUILTIN_IDLE_PER_NODE = 1LLU << 4, + /* * CPU cgroup support flags */ @@ -134,6 +140,7 @@ enum scx_ops_flags { SCX_OPS_ENQ_LAST | SCX_OPS_ENQ_EXITING | SCX_OPS_SWITCH_PARTIAL | + SCX_OPS_BUILTIN_IDLE_PER_NODE | SCX_OPS_HAS_CGROUP_WEIGHT, }; @@ -3344,7 +3351,7 @@ static void handle_hotplug(struct rq *rq, bool online) atomic_long_inc(&scx_hotplug_seq); if (scx_enabled()) - scx_idle_update_selcpu_topology(); + scx_idle_update_selcpu_topology(&scx_ops); if (online && SCX_HAS_OP(cpu_online)) SCX_CALL_OP(SCX_KF_UNLOCKED, cpu_online, cpu); @@ -5116,6 +5123,16 @@ static int validate_ops(const struct sched_ext_ops *ops) return -EINVAL; } + /* + * SCX_OPS_BUILTIN_IDLE_PER_NODE requires built-in CPU idle + * selection policy to be enabled. + */ + if ((ops->flags & SCX_OPS_BUILTIN_IDLE_PER_NODE) && + (ops->update_idle && !(ops->flags & SCX_OPS_KEEP_BUILTIN_IDLE))) { + scx_ops_error("SCX_OPS_BUILTIN_IDLE_PER_NODE requires CPU idle selection enabled"); + return -EINVAL; + } + return 0; } @@ -5240,7 +5257,7 @@ static int scx_ops_enable(struct sched_ext_ops *ops, struct bpf_link *link) static_branch_enable_cpuslocked(&scx_has_op[i]); check_hotplug_seq(ops); - scx_idle_update_selcpu_topology(); + scx_idle_update_selcpu_topology(ops); cpus_read_unlock(); diff --git a/kernel/sched/ext_idle.c b/kernel/sched/ext_idle.c index cb981956005b4..a3f2b00903ac2 100644 --- a/kernel/sched/ext_idle.c +++ b/kernel/sched/ext_idle.c @@ -14,6 +14,9 @@ /* Enable/disable built-in idle CPU selection policy */ DEFINE_STATIC_KEY_FALSE(scx_builtin_idle_enabled); +/* Enable/disable per-node idle cpumasks */ +DEFINE_STATIC_KEY_FALSE(scx_builtin_idle_per_node); + #ifdef CONFIG_SMP #ifdef CONFIG_CPUMASK_OFFSTACK #define CL_ALIGNED_IF_ONSTACK @@ -204,9 +207,9 @@ static bool llc_numa_mismatch(void) * CPU belongs to a single LLC domain, and that each LLC domain is entirely * contained within a single NUMA node. */ -void scx_idle_update_selcpu_topology(void) +void scx_idle_update_selcpu_topology(struct sched_ext_ops *ops) { - bool enable_llc = false, enable_numa = false; + bool enable_llc = false, enable_numa = false, enable_idle_node = false; unsigned int nr_cpus; s32 cpu = cpumask_first(cpu_online_mask); @@ -237,13 +240,21 @@ void scx_idle_update_selcpu_topology(void) * If all CPUs belong to the same NUMA node and the same LLC domain, * enabling both NUMA and LLC optimizations is unnecessary, as checking * for an idle CPU in the same domain twice is redundant. + * + * If SCX_OPS_BUILTIN_IDLE_PER_NODE is enabled ignore the NUMA + * optimization, as we would naturally select idle CPUs within + * specific NUMA nodes querying the corresponding per-node cpumask. */ - nr_cpus = numa_weight(cpu); - if (nr_cpus > 0) { - if (nr_cpus < num_online_cpus() && llc_numa_mismatch()) - enable_numa = true; - pr_debug("sched_ext: NUMA=%*pb weight=%u\n", - cpumask_pr_args(numa_span(cpu)), numa_weight(cpu)); + if (ops->flags & SCX_OPS_BUILTIN_IDLE_PER_NODE) { + enable_idle_node = true; + } else { + nr_cpus = numa_weight(cpu); + if (nr_cpus > 0) { + if (nr_cpus < num_online_cpus() && llc_numa_mismatch()) + enable_numa = true; + pr_debug("sched_ext: NUMA=%*pb weight=%u\n", + cpumask_pr_args(numa_span(cpu)), numa_weight(cpu)); + } } rcu_read_unlock(); @@ -251,6 +262,8 @@ void scx_idle_update_selcpu_topology(void) str_enabled_disabled(enable_llc)); pr_debug("sched_ext: NUMA idle selection %s\n", str_enabled_disabled(enable_numa)); + pr_debug("sched_ext: per-node idle cpumask %s\n", + str_enabled_disabled(enable_idle_node)); if (enable_llc) static_branch_enable_cpuslocked(&scx_selcpu_topo_llc); @@ -260,6 +273,10 @@ void scx_idle_update_selcpu_topology(void) static_branch_enable_cpuslocked(&scx_selcpu_topo_numa); else static_branch_disable_cpuslocked(&scx_selcpu_topo_numa); + if (enable_idle_node) + static_branch_enable_cpuslocked(&scx_builtin_idle_per_node); + else + static_branch_disable_cpuslocked(&scx_builtin_idle_per_node); } /* diff --git a/kernel/sched/ext_idle.h b/kernel/sched/ext_idle.h index 7a13a74815ba7..d005bd22c19a5 100644 --- a/kernel/sched/ext_idle.h +++ b/kernel/sched/ext_idle.h @@ -10,19 +10,21 @@ #ifndef _KERNEL_SCHED_EXT_IDLE_H #define _KERNEL_SCHED_EXT_IDLE_H +struct sched_ext_ops; + extern struct static_key_false scx_builtin_idle_enabled; #ifdef CONFIG_SMP extern struct static_key_false scx_selcpu_topo_llc; extern struct static_key_false scx_selcpu_topo_numa; -void scx_idle_update_selcpu_topology(void); +void scx_idle_update_selcpu_topology(struct sched_ext_ops *ops); void scx_idle_reset_masks(void); void scx_idle_init_masks(void); bool scx_idle_test_and_clear_cpu(int cpu); s32 scx_pick_idle_cpu(const struct cpumask *cpus_allowed, u64 flags); #else /* !CONFIG_SMP */ -static inline void scx_idle_update_selcpu_topology(void) {} +static inline void scx_idle_update_selcpu_topology(struct sched_ext_ops *ops) {} static inline void scx_idle_reset_masks(void) {} static inline void scx_idle_init_masks(void) {} static inline bool scx_idle_test_and_clear_cpu(int cpu) { return false; } From patchwork Fri Feb 7 20:40:51 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrea Righi X-Patchwork-Id: 13965849 Received: from NAM04-BN8-obe.outbound.protection.outlook.com (mail-bn8nam04on2057.outbound.protection.outlook.com [40.107.100.57]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0F8CB1D6193; Fri, 7 Feb 2025 21:11:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.100.57 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738962702; cv=fail; b=kclNFNJGaxrOYRLV90HX43KGN7eAQ+8YXrLgLm6YyNg4CF7n4+2Iwke/7eQFXtA3mNgIp0HSSThDLVKbmoH/JfLTr/z8rwomNqqZiYhRCUWUMmWI/8Xs7bdibmAZDemo6NSzaIdFyyoEdJkjVvAR68Rlyeh6sAVbmQlGQslkmDU= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738962702; c=relaxed/simple; bh=FvLuHn97ihMHWhIoMxr8TmtEIHIzwXHUbxbyZknVO9M=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=ocVHK5vtF1NslG769DdWl0Y7xO6NVV4eJbixwFlbtNnOb6tyEyZXTFcECbL4zA6REoueRvUBfmy85hmsabNCKgN1nMZ+oS1S9dT6YfA515lKGAWbNcHlhk58WPm+P4HDMgPfueN/N65M2vhMdqRqrzH/AJufX6fUZKkFBJEIjo0= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=S+9+vLLF; arc=fail smtp.client-ip=40.107.100.57 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="S+9+vLLF" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=CUWtzNYzkZLdcTf6YKdAuXA868JJy4SJTZSwqbi/+41A7iayuJDqaXzsoDAEcMfye6/um3pJflAQGMvTjq1FOJUIT+kpKkVoBBcRHfPQ91ZxnSufCPKfW3l02EC3JRNHBn2lIupWdMNhE9/1TyZiJwHiI4Cprkbp9sEW/d4a4uiU9gSYchHvD2Tarjqm/AoxDaIcopYixV7/H8sYmc8+DCTZeISAp/T/rXnQbHaW+brhy4Gike+aEVo6ZzhLEsItcgqRLb+/aOFDIZ7jUV/VJpmrNpp+ts8tHXiGURcCgsvqOcpFopm+kG0OWC336pk673fx0+rexxjmp5UsnBQGmA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=4Is+GHQDv+QhaaLW0HaFvTCx4kBiEf33Lc8haT3XMWI=; b=IlerCKkHoffAzCm60dWrwPMxk72+ruLsP9iuvCO7TJkgygHDLPwZm9TfZPM7qd9azflt0DHQ832z3+6Plw5B/c/mxAqVx7FCWCZH2zHYJHmwbkXSSjoWNvTcuFH3NZPTbYcKxBKzRxWsMW9Db5x3t1STFPC0Hmt6UX7vcVNAqWaoE6G0cMUSQDxOoMiHKeZ6fQwjd9+itHo9xv15eJb+aUibfN+RwviQdEYuKr0wSRqVGhfMs9eNzP/VCZhu6I8kPGEt6YRV4aO+wn/pH4OKywY3dA6l7TiN9Uz/jMrWSrSgOlTUE1t51Jh5Jg/M5j9E+GHopXX5IJf4GXtSCSmYbw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=4Is+GHQDv+QhaaLW0HaFvTCx4kBiEf33Lc8haT3XMWI=; b=S+9+vLLFH3VEY7kAHiVCsifRf5jgg5azf8s3O6e4GjjHcjXrn2Tw2Yhppv7V4u7gSDflzNlEjFhq1wnjRgH7EyLOH6U6r7piSlOu/pB/g4FwK7e2XkdBMWTZWzQnH4oPBH8M9npSxSGK0D8+IdmtuAcv2cOgeePGCcrNHhEMbNltwgRyLew4gCta0k58A0M18FzRVt6Mpwx8bE2f2vMMcLx4NC+Exj6n0YqSRFe21SEaIb/ep1NIlibp9q8qBvh74O3Qo0Zg0yskvvzrVRSw3TVQHreVXOd0pOEmIijNqFq0ye3UtcbVdmJiKuYTlmVV/0rUU0hAsTX2/1edxYTTCQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) by PH7PR12MB7380.namprd12.prod.outlook.com (2603:10b6:510:20f::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8422.14; Fri, 7 Feb 2025 21:11:36 +0000 Received: from CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5]) by CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5%7]) with mapi id 15.20.8422.010; Fri, 7 Feb 2025 21:11:35 +0000 From: Andrea Righi To: Tejun Heo , David Vernet , Changwoo Min Cc: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Ian May , bpf@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 4/6] sched_ext: idle: introduce SCX_PICK_IDLE_IN_NODE Date: Fri, 7 Feb 2025 21:40:51 +0100 Message-ID: <20250207211104.30009-5-arighi@nvidia.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250207211104.30009-1-arighi@nvidia.com> References: <20250207211104.30009-1-arighi@nvidia.com> X-ClientProxiedBy: FR5P281CA0052.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:f0::11) To CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY5PR12MB6405:EE_|PH7PR12MB7380:EE_ X-MS-Office365-Filtering-Correlation-Id: 4eed70eb-6b5d-44bc-7fcb-08dd47bc0157 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|366016|1800799024; X-Microsoft-Antispam-Message-Info: Y2GtVA0YKoRjdufJbit+hsnzCO3F/EpKuJfCk66Dxu84/elAmKHa+2Y4Q7MBH+3Ytjx6YYmppBumYxzWWUp/bkWxMpvsM/Ggj732TSyH307dRPyiBq/jqDHHTSjFVLqMgsNmOacrdECn2Zr8/iq13h6BwGExJv3eudfkAEytW+3kADupI7Sc59snSV97sxCqRMVkJ1ohd7ilgNBdPqwRlYi0/zw8YtF2GpE7S8UkAmAic3b1IwUSEbS22b9FPDqQjPWWHY3fxjo5fVzSx3UL+BLBbw+pQafdsvhzlKPQUlvGaf1EcheFMzQvXwUY7kac1b8JwrDdjURk+h0PKtvI5dvpOcMMCHBALQ+OzStM3ExEkP4HkPTtt0FhmheNdbIv4lgu47ZVJHNt+ie9IybKU+VcF2A91oeAjH6Mo1yA43ChTR6x1gJlZPOkqL1cX/t5YKCwaoJ7bxW+vILe+Ts9C6arEaOv6R7DjgTQjWm2Da3nYnHxKUYZjx/hLqDpdmKR8Qd9gRuUn4/Qo3w7ApVnxBz/OAnzWhi80gBMIEcopI8H5U54QrhcI/2JewZklxJbAgSopPJ4Pd/d7s511dn90shshmhx51ryT04TzW8Eq36hSJBtY4pBgmDn4Icwk7bHICyyrZdqvIAINKqDFCKes9DRkdywPYMHvpvFZtd4SZ32BKJ636MSGby2Rx/QP2IXtz8690zyXuM+5CypmlDKKyjZXioDYaJDX8YBsdvGrXxZcCtcVOJik5LK2r4/tv4L8cwjr7U215nAkmAig1YGbhL1MRQzCIPHy5kkRosTZyBPARhIw3dsbi433vI6TlLvL8leMgwoRpybT3MTV/1xwmRG5igZxQa+BzJ2gnKkZXYydMV/i9Qu5UI4eQhOe9qBIWc19vmY/bSSd7+PnD+NW9R9F5HffNlCXkxQF4hwAB+dC7hA8VtACzMonBhac47MAc6n9VROXNASy27mDjvFtSDzQ/Jqi5rpXxJgX5eMVtBSfvvjPdGZHY//QtCgH8Q40XEiJh8w9LfAnKj49nEVuE3ew1dH5lDZcmaeXQYfHICpx+wxhkR2lXRshDFYhckJHm5puNPc5/34PmvQm6LUXrFnwqXBzUa1PZKKLucs+bsPDiIqiri8nNovSxmk5X1oGV44C9Y1t+zNfT2mqXTp+XZvmzXLrssfbnWKka63tBZ/cZMcPLaQv81PT6jeWXpmMOb/J5fWZf3+MklV3FdIlL7C0EKjjeHvYWUtFqQD9CYTZtSDzVoTf0y46egUpu9ibiMvYhf8CQn4TC1/DW+Uv6qlrBnaRppq15jcBhPzuGPDTXjzgz9cgwc344ZLB5FwHqHPEgghON/x/kjcgd4PiszLKk/vU142YKRaGbnffU9/ayXEeuWXHPh73CXv7+Ro X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CY5PR12MB6405.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(366016)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: HTtYAUKF7ob1R3wPKpbKLJag/3DzyHNmq4/e1HSbeOx5bW8I1vQt2+k31oYy2XSbfonVDuStW+q9TE6MwbJ5CHZm5jfqNfTwkEsuu6HQ5ebqO3LJza1A53br+kWLAV4c+5qpome5H8Bt0hKFLZmdJ0Nx945YQqfSDgvYo07NCMmVyGPGp1rLyouLrekws1+njFqNW+CiPfbbSOvnRALQUvxQ1rb7Hhkj2/enre2dppxa60H/kmo/FVp0kQCan8Njbs3brssmJVlLYtR8kk8xoeXiVqqSgPRS759ZQUKoUsp8nAFrgG2eDAlMXCQfuMV9yyy3C1VQ+tZ6zovGLS5rs7CX8wTC1DOBYoFask4f+VnyFhMgl+E6h/WutTo9d492FRU9cWxjPo7/3S5rAIQtPeXzBJtvH+yWYuZt2alJClyc4qieVdryfiFIPq+vT2Z8rMpJ5FY99CQB46VetVQpLfb5QDFvYEtmlRYUj4m8k2kGZ9HmQ7KCvWK+G0He+wNSRa0Chb+6V8JvSunE7LB+qI8BtB80rdTRZr2D/wBNdwBixvspFBNL5xEDHRb8iG1fwN8gQYXJmkZdU/1066lpXRxEH6Ozvh0D6T2xK23Jn91/YFfyMKe9/mCEPvM4ED/M79Tty2jrqvuN5q5gZem1g7azXL2iL/qVaPSzdH/OgI3iRiF6HPS/90ZChInlCxoLhWK/Qfpij9BPMpFitc/wXlRLWYKFpWJyi/KRjgUyFMnP7nneRE7sk6lNccApNrvpsWJpWXUfTgj5ZT41Ui7s5mtzzGhFiC8Wmff5t6jVQmkE7R3My/zn5rOAKVqh7BA5tzI+KoKkjoI7Uf4b3Y3vVmrGiouKpqTUCGS8pxoFB52vXP47urJcIPTgDLZ95T5NisMz0C8/UZfvmccx+KHE4bWbzfFG15Ozr3rdYI34HHAt5uhOY6AodhqvpppDwNtPEwRauka4khTu4evc1uedvbmHG5Q6QcByoWLen3JzuUAM8slYUroMJLZgHWT4/wdIPWkG0kOs3ZjgBfKz+olD/wYHMIhaWz54bGpE28Fv75/GeSWq7c6Fe+GUvRNh82iqHeuSY+gswgy7kVCh+V0kL55tvqLRWJXqd8CQzrDFmoXYLYoUkwa7jpV/qIF/KXGcc2jk6A7+dYuqb5GL+cCDr0/S4yb59epcETXpOVn+87xfQpBTp7gPT9/z5mowxmeoBEbuX+XofK/cR7aNO24PAK72juodkKfQg8cF1gIazm4JbHnG7vmzUy61JPgN+tLF2kCvOCPiyB1gvzuuq7JCGhR8xM4bfnuIhwJi31IsUnKL2djqnvsqKfcGIaffXCnQmvVm78ESv8zMKwCuYNVn3mlfEP5e7y8UzS1QtjtPWcNwqkuVr07g3U6HiBFsHmD/rx69YOMHy/s77qC0h8QcS6tqFLTMd0vLjSrWqm3LOseoy0WNk7kR0ZfbFZHrEB9R3Jh/cK3YbMD93ExXd5a8RvpkOuBqzxY16NHs+XypU13F/VGS/hy4QHzfo4N5Nmee5SfiEYZxtHgsnt/0WxsUHGdrSrsbS2LqcNzvFEFRoecXlCl1cZo0Md9uPXEcm8O1 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4eed70eb-6b5d-44bc-7fcb-08dd47bc0157 X-MS-Exchange-CrossTenant-AuthSource: CY5PR12MB6405.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Feb 2025 21:11:35.9319 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: l2xjIb+H3Tlkbm+gdaY5s/150G1iYvDvm6Bl8bZ7x2XzNeowvDbs5HxAuIZ+sadUsP1Ye9ql61pwayaFe0heww== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB7380 Introduce a new flag to restrict the selection of an idle CPU to a single specific NUMA node. Signed-off-by: Andrea Righi --- kernel/sched/ext.c | 1 + 1 file changed, 1 insertion(+) diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c index 0063a646124bc..8dbe22167c158 100644 --- a/kernel/sched/ext.c +++ b/kernel/sched/ext.c @@ -775,6 +775,7 @@ enum scx_deq_flags { enum scx_pick_idle_cpu_flags { SCX_PICK_IDLE_CORE = 1LLU << 0, /* pick a CPU whose SMT siblings are also idle */ + SCX_PICK_IDLE_IN_NODE = 1LLU << 1, /* pick a CPU in the same target NUMA node */ }; enum scx_kick_flags { From patchwork Fri Feb 7 20:40:52 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrea Righi X-Patchwork-Id: 13965850 Received: from NAM04-BN8-obe.outbound.protection.outlook.com (mail-bn8nam04on2057.outbound.protection.outlook.com [40.107.100.57]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C742A1B415B; Fri, 7 Feb 2025 21:11:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.100.57 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738962705; cv=fail; b=TTK91sWDsOwfcHXweAoWjDZPS74gjhldrMmSn6k4+AF6N43pNB4RFdj8npsE0SewbPU5PnQN/C9K1RwLJ2FkZX6A64EHoZOm6cM1Gc4Uqo62Su25/wGNUOrNz/8nHVZ/ZcACNNIGEOlHTJRG6RTIyG31kDR4++XHEepg0b12OEA= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738962705; c=relaxed/simple; bh=68WAxPeVDa4JCnKcWiaH149hP2VakmWn4IiI/0OGzLQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=TblK4YqeHFivkiDns+udtJmINKBzBV/xnpUFHRibZtBdQMMG5Iz214i2jzYnWcGlDndLoAR7fn0QIUFlu3xX/wvCOLg1J5vRBXOP6/p+GziNWbjI8bVFS/e/6TzjpOa+JnEtphe0d/Ce4hj9VzQS2ouixT3qfz3bwjATeNBEEhQ= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=Ky9uEhFm; arc=fail smtp.client-ip=40.107.100.57 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="Ky9uEhFm" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=CnKd8kurOjJvqxhgXiCRnbYG7GHyYGuQXFRYIZ1XFwPXjhLqLTgokkcoDA8b5wNzzWiCVtTDXCblxY97HlgBA/jRzyFb6OaXYhpI1VkjuEDyhCN70dPhnyk12aVhWd2dzzEt6asIStVWDoDvNBLw11X9LCXNj6hT0ZaUnoZJnXqFAhnh1mz1EHkg+itRcixqMLr7xPR+AJfEgx36PYJGUqzqZlEELuR/dGLDQjPKxkyWOhW1/HLccE/lskbOxJPJMS3fUg9zlF6dhOvGM0HaburzAWZBdGqpC8VaY6qCyXgzjYW+MdbNKdIKziLViXikYWKRp1ZjGP+JSjzbnaj2Eg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=BQWKnNVX67hZvStXqKeIj9HUzXK7qGpHgtMk1iy4Mpw=; b=SnJY+SoYUIbdzkDE0zMyx54P7zhu3AgdCJnvLFGlsDQMWSQ+SND+zPN1J989PdBE62fSEBCI4UN7fOoVcKntFnPtX/KlNN4dIRwHsxUVQSZWPpskAC4jr2Va6WOuXeYMk/eYkjSd0osysiTn6F3XG9sX0YXIZQQJchrLzbfR8h5Voa6UZ9TQyEqFvUyteaZn2318jw6ztj+sAeVoUKH7k0Xm0DODHsDgN6H0UHHdVT/H9GKW+h2eImAt1OHIkuAeqCTiJb5pyQT+4oVspP5JIN6Tu76zIpBj07FJViWl/UsVi6M4AnrnZygsWx/qrAFenHgN5EfrnRYZUsPs8KYUJg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=BQWKnNVX67hZvStXqKeIj9HUzXK7qGpHgtMk1iy4Mpw=; b=Ky9uEhFm4f5dAmhgm1x2GIf9/gP2fuM+vgTlLs4L7ktbQ3fZVdxFmyxdTIFnV2glgM/kwlcoaQijTsnHQu7B6MI5ehOYkqz6MDJ3DKXGII73gUpjI1o7lv94X2h75qF2J4kF+yIb3euzyi+T+JpX16blmEU3EFv8LLB7m1aShEmBIZaLxcai661FSu5dMB3uAuz3Cz6c7RgL69z4PUHuImRv9nsimGRxnTccQZiEWfpXiv0/y1yn/QLMGB6tge5lHY4pz1lCsJf69W80U6StPnfnWHvSOyLsMIpBJx1D1P+ABbs6VMIbGfoSS57+MPsMF/v7MM7us6iHo8O/pqNFvA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) by PH7PR12MB7380.namprd12.prod.outlook.com (2603:10b6:510:20f::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8422.14; Fri, 7 Feb 2025 21:11:39 +0000 Received: from CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5]) by CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5%7]) with mapi id 15.20.8422.010; Fri, 7 Feb 2025 21:11:39 +0000 From: Andrea Righi To: Tejun Heo , David Vernet , Changwoo Min Cc: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Ian May , bpf@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 5/6] sched_ext: idle: Per-node idle cpumasks Date: Fri, 7 Feb 2025 21:40:52 +0100 Message-ID: <20250207211104.30009-6-arighi@nvidia.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250207211104.30009-1-arighi@nvidia.com> References: <20250207211104.30009-1-arighi@nvidia.com> X-ClientProxiedBy: FR3P281CA0172.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:a0::10) To CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY5PR12MB6405:EE_|PH7PR12MB7380:EE_ X-MS-Office365-Filtering-Correlation-Id: aa0830f8-4a82-426d-826d-08dd47bc0391 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|366016|1800799024; X-Microsoft-Antispam-Message-Info: 5eqdyvABnDGHfKc8TXy6LsITeRIbknifHOYJzemAKRWwDQXQbknXybGEzo/kJCrb+uXca63ydwafxFg/a+4gHRuTT+759DNM/jjVrMNL11MD6gJ3mEGFRoED4KbFtP603NeMNsEp0cy75Klcgh2j7yYScPccJEKJxIvKdB3XIFO3QRGryvwmYOSmqJWnNtsQxnt7d2rJ14h9pvyJ4hczufCTqZ9rv3SJ10EP/O4VaV1xzrDEUgbasnl/zbib7wB+HOxhooRhb+1vWOW1QCLuRZwsacrOEP1wWaPUGi4rkWoF9Z4XTFTkQwPDslr1Vkd72lOe9B0U6226bNt6qtW+1UOp4kaxrYm93QgWYThT9v2extNNkBmdPzerNidyfRdYVRuLaRdRmUMtiXvIpp49oUGrSWNqJAHo3QImLJFT3RfdaWRIAvCMDqO8h6EZ1g7bGJR5G48O6wMSdzJal8xnWLTF6a2k5o4t2Z24sUZeQgDjx1CFQFrTh4oPiCNaEq2xJGTK8mmWIFh6lvhqR696DTkSGxUWCVOQq2tA588QLIT+ae+zcN0hYnWV/FMJn99jssyCgGkqPjTMR7EO8A/QEHMnC+Ox/ts5lhktVIOzFXeq6KUYw0GFGeVOqur5nCnJ5q4mfPYMmDNXkbKimA8KYmHhewv3bDsYJV1ZVxwbf7DPEeIlHWJuKcZQHdYKnx9UXynyDZFj7xmDKsVsgU13fvxmVF6ED7di6FWMwO875FXdAbKlFNhaJCNdoXyCaUYJzaBvfWjjdI91GjyEzlrGTLkO3HQjCEyW/EjcM3oUQU3DtNDqEsnPTecUVssgZ9dSUPmgWfwkcjMIh9b1IF2WqU3SL+cDQTavHELKZKJB1fQqei+9tmScjFLXseX1NL+7vkE+700RBL5Fcs5BhMUrPbRjMm+wPq5UtTSwRjQgqHBVWdL3RikOXX9PlOY4o7ro4OCqZy/PnpJJm7iIGAiFbPUEYpPqKKQsZ7rn21cicPhjQoOzfZmxfhie6bhO4hzIp3l6CHCAuOIdWtf9o5Y3vIzVv1huWFDZ4tTppMFqXFvoHWKP8DdTtgmqR40n+EjPYdU5hZYuVhNTZqF6gfiof71dhASHc7GexK77ieLma69Ud1REqE9nBIem7EBv5u6rZ7Ud2J/+PDZuuwQATtDCI/NTBQDn/8mUl31fdK6dC+WYusyH8vstOmGA83fxZYFhc9zVpN6sCsWW5iMBjAuPjOgmxkxF9pZONjSvvRf94hH2+6KaAVAD7br8tME6VXENkXrtzoj71UJKNm5+nCXgdZl1P+bPQN2kDbhKWuHgFocG+Yiqg/vJNftcepWq48e2b1ScWWglUDgM0scGdTzxqeDeIsZ3X1rQ3lPWXO+GrKH9VRuEH2HwWo+cIVpqcI+G X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CY5PR12MB6405.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(366016)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: mJMTLOAhzXqZXT1YPRQutYKOOGUcm4b4w4T3/nEcRph1eoGTFbKlvWYl5ZT/B1FdCn2JM4d5BMhsqsETHjvtuo7lHNa/XSdlxud94c3b2XTE6dM6aZcSLXNhQuc8epcxoKPBn9kbOuEMNn9sMvEw7A5Ix/eAjvaQRoZ5T3KDExyxQzDJa5WI0iCQS3Od5hJ3Y4D9nBK4EPsloBjzKfvF2/z158PZZWP6bEsFXSQYrF5aPjeVXg8f/h4hRcXpj9xVDR1rr92kl6+kjOs97uKZ0rT93/e0+vT73avUmy420p76S/5kXtHlNbf5a+AjEWMaEdAElwF124r4/JNFTIdRBnu0SLicnxG8BSqj5GDGg++W715Cuxh47KG23Gx9ONMZjw8DvQLoSTYS0aRUUbfdWs8uYmNvnSc8+GoPTyOn2wYgbhHhBfsQrIaMvo0rSGQVMFCGAd+Cf3NIFL9PcqXIBYrtwX9VMVZpquNl3O7NQoMs12g79CY1g8nhSHNbs98STY1CLhK+F2j3c7nhRNBoWKpwyLBdk08yWi02Y1WwI3wLQ+RUsW2JvsmkRFACNk7YyouMjIJ7OWv7vb8uZi6yQQ9bZMS5j7eggXjl2pTcKwW1xaPtmDgknCAvXYyaJuKKgmDIeqxVZrsYrr6qcNG6s0Hixe9e9O0jlcBFRGnmNFOVZIIyWVWOf2H4bI2J4ZGiH61KW2H808rPb7KAIu8uT5+mKqt3Xls5EgcnOYzDRIZnEF5VKsrOZU97IybHORMyiNcyfdU184EU/XouljRJ4b2GZaRMB2WVTPMpU7TyVRXd+phPlL7QrvUr5nMRP3rOVjmvA/1y1McY/GiWPc5vhQlY+LNjS0OPK6u3yqPSdkNuSMWf37GLa6oiPJC6LoX7+fubZ1e18NpoLNpUClDWEKzvzFbmd371xeG7SfEnTIWC/d2BP0yFZNWXjXvkT7Ur96JYCpLKrwkVLBiDIP4PvkPjI6xPh1HcsTBldkYeMgU2BgThFw3E18be+Vh8NiMnbDO+qV88FkYBfci+PSwhg9Aun1BPvwVCSkhm67w4iSfeUAElrPKwj0t9Q5bsJxMEkWpDIr1SzTdGME78F2d3ce5o9jZHtDcfzfiOhnpsrUTk3O1vSlMmQ2AfefQogZb7ZYT6hvFi4m5AJWCtlpBAVjgsM58ixPXBAzLbnbTsdzPaB8bumDp3BAh6fgs5y5YPEm5iSPGwFwgWJMw2NSBGXZiO2GZc+FQoyj81K4IkPYHCta1S74WX8KKrhPQ/uoHcADafeEjD7lTP3t5cCJ5dAe+N7+MQ4nFmykidq9k1ymPD07/ZqQJ7OLUVpvRskmnRFc5hfoo9aFzjsqHlQjbwlqtWuUc8u7OOOt8bUiDigm7cB46Zw2uVqLfRkfVJgveR7nsrNT9G2okwcfVHo6TDf/GozITu6Ysl8IxVmtqzwAuQUq6cxnuhwol/D8OUw4Fi6e45VxE3HCr+L/KPK8fNFs5xxY6AIVdM+/KHQoOn7iBxDWZhKNXWUo3OLplqRPDX9gzv5JEbgZRwTC/HX8Npu0E+FiSYb9TJun3DGTSfHRF6OuD08XtJpQW2cMqqh1n+ X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: aa0830f8-4a82-426d-826d-08dd47bc0391 X-MS-Exchange-CrossTenant-AuthSource: CY5PR12MB6405.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Feb 2025 21:11:39.8346 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 1NN1h6NaGs0cS2uw8YZezPB3k6zpf3WI+D1ve8vRlBtPzF5XH1rG6S11SdBcdoIbV8Y10C0avItU3ctMv3q7ZQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB7380 Using a single global idle mask can lead to inefficiencies and a lot of stress on the cache coherency protocol on large systems with multiple NUMA nodes, since all the CPUs can create a really intense read/write activity on the single global cpumask. Therefore, split the global cpumask into multiple per-NUMA node cpumasks to improve scalability and performance on large systems. The concept is that each cpumask will track only the idle CPUs within its corresponding NUMA node, treating CPUs in other NUMA nodes as busy. In this way concurrent access to the idle cpumask will be restricted within each NUMA node. The split of multiple per-node idle cpumasks can be controlled using the SCX_OPS_BUILTIN_IDLE_PER_NODE flag. By default SCX_OPS_BUILTIN_IDLE_PER_NODE is not enabled and a global host-wide idle cpumask is used, maintaining the previous behavior. NOTE: if a scheduler explicitly enables the per-node idle cpumasks (via SCX_OPS_BUILTIN_IDLE_PER_NODE), scx_bpf_get_idle_cpu/smtmask() will trigger an scx error, since there are no system-wide cpumasks. Signed-off-by: Andrea Righi --- kernel/sched/ext_idle.c | 242 ++++++++++++++++++++++++++++++++-------- kernel/sched/ext_idle.h | 11 +- 2 files changed, 203 insertions(+), 50 deletions(-) diff --git a/kernel/sched/ext_idle.c b/kernel/sched/ext_idle.c index a3f2b00903ac2..4b90ec9018c1a 100644 --- a/kernel/sched/ext_idle.c +++ b/kernel/sched/ext_idle.c @@ -18,25 +18,88 @@ DEFINE_STATIC_KEY_FALSE(scx_builtin_idle_enabled); DEFINE_STATIC_KEY_FALSE(scx_builtin_idle_per_node); #ifdef CONFIG_SMP -#ifdef CONFIG_CPUMASK_OFFSTACK -#define CL_ALIGNED_IF_ONSTACK -#else -#define CL_ALIGNED_IF_ONSTACK __cacheline_aligned_in_smp -#endif - /* Enable/disable LLC aware optimizations */ DEFINE_STATIC_KEY_FALSE(scx_selcpu_topo_llc); /* Enable/disable NUMA aware optimizations */ DEFINE_STATIC_KEY_FALSE(scx_selcpu_topo_numa); -static struct { +/* + * cpumasks to track idle CPUs within each NUMA node. + * + * If SCX_OPS_BUILTIN_IDLE_PER_NODE is not enabled, a single global cpumask + * from is used to track all the idle CPUs in the system. + */ +struct idle_cpus { cpumask_var_t cpu; cpumask_var_t smt; -} idle_masks CL_ALIGNED_IF_ONSTACK; +}; + +/* + * Global host-wide idle cpumasks (used when SCX_OPS_BUILTIN_IDLE_PER_NODE + * is not enabled). + */ +static struct idle_cpus scx_idle_global_masks; + +/* + * Per-node idle cpumasks. + */ +static struct idle_cpus **scx_idle_node_masks; + +/* + * Initialize per-node idle cpumasks. + * + * In case of a single NUMA node or if NUMA support is disabled, only a + * single global host-wide cpumask will be initialized. + */ +void scx_idle_init_masks(void) +{ + int node; + + /* Allocate global idle cpumasks */ + BUG_ON(!alloc_cpumask_var(&scx_idle_global_masks.cpu, GFP_KERNEL)); + BUG_ON(!alloc_cpumask_var(&scx_idle_global_masks.smt, GFP_KERNEL)); + + /* Allocate per-node idle cpumasks */ + scx_idle_node_masks = kcalloc(num_possible_nodes(), + sizeof(*scx_idle_node_masks), GFP_KERNEL); + BUG_ON(!scx_idle_node_masks); + + for_each_node(node) { + scx_idle_node_masks[node] = kzalloc_node(sizeof(**scx_idle_node_masks), + GFP_KERNEL, node); + BUG_ON(!scx_idle_node_masks[node]); + + BUG_ON(!alloc_cpumask_var_node(&scx_idle_node_masks[node]->cpu, GFP_KERNEL, node)); + BUG_ON(!alloc_cpumask_var_node(&scx_idle_node_masks[node]->smt, GFP_KERNEL, node)); + } +} + +/* + * Return the idle masks associated to a target @node. + */ +static struct idle_cpus *idle_cpumask(int node) +{ + return node == NUMA_NO_NODE ? &scx_idle_global_masks : scx_idle_node_masks[node]; +} + +/* + * Return the node id associated to a target idle CPU (used to determine + * the proper idle cpumask). + */ +static int idle_cpu_to_node(int cpu) +{ + if (!static_branch_maybe(CONFIG_NUMA, &scx_builtin_idle_per_node)) + return NUMA_NO_NODE; + + return cpu_to_node(cpu); +} bool scx_idle_test_and_clear_cpu(int cpu) { + int node = idle_cpu_to_node(cpu); + struct cpumask *idle_cpus = idle_cpumask(node)->cpu; + #ifdef CONFIG_SCHED_SMT /* * SMT mask should be cleared whether we can claim @cpu or not. The SMT @@ -45,33 +108,38 @@ bool scx_idle_test_and_clear_cpu(int cpu) */ if (sched_smt_active()) { const struct cpumask *smt = cpu_smt_mask(cpu); + struct cpumask *idle_smts = idle_cpumask(node)->smt; /* * If offline, @cpu is not its own sibling and * scx_pick_idle_cpu() can get caught in an infinite loop as - * @cpu is never cleared from idle_masks.smt. Ensure that @cpu - * is eventually cleared. + * @cpu is never cleared from the idle SMT mask. Ensure that + * @cpu is eventually cleared. * * NOTE: Use cpumask_intersects() and cpumask_test_cpu() to * reduce memory writes, which may help alleviate cache * coherence pressure. */ - if (cpumask_intersects(smt, idle_masks.smt)) - cpumask_andnot(idle_masks.smt, idle_masks.smt, smt); - else if (cpumask_test_cpu(cpu, idle_masks.smt)) - __cpumask_clear_cpu(cpu, idle_masks.smt); + if (cpumask_intersects(smt, idle_smts)) + cpumask_andnot(idle_smts, idle_smts, smt); + else if (cpumask_test_cpu(cpu, idle_smts)) + __cpumask_clear_cpu(cpu, idle_smts); } #endif - return cpumask_test_and_clear_cpu(cpu, idle_masks.cpu); + + return cpumask_test_and_clear_cpu(cpu, idle_cpus); } -s32 scx_pick_idle_cpu(const struct cpumask *cpus_allowed, u64 flags) +/* + * Pick an idle CPU in a specific NUMA node. + */ +s32 pick_idle_cpu_from_node(const struct cpumask *cpus_allowed, int node, u64 flags) { int cpu; retry: if (sched_smt_active()) { - cpu = cpumask_any_and_distribute(idle_masks.smt, cpus_allowed); + cpu = cpumask_any_and_distribute(idle_cpumask(node)->smt, cpus_allowed); if (cpu < nr_cpu_ids) goto found; @@ -79,7 +147,7 @@ s32 scx_pick_idle_cpu(const struct cpumask *cpus_allowed, u64 flags) return -EBUSY; } - cpu = cpumask_any_and_distribute(idle_masks.cpu, cpus_allowed); + cpu = cpumask_any_and_distribute(idle_cpumask(node)->cpu, cpus_allowed); if (cpu >= nr_cpu_ids) return -EBUSY; @@ -90,6 +158,55 @@ s32 scx_pick_idle_cpu(const struct cpumask *cpus_allowed, u64 flags) goto retry; } +/* + * Find the best idle CPU in the system, relative to @node. + */ +s32 scx_pick_idle_cpu(const struct cpumask *cpus_allowed, int node, u64 flags) +{ + nodemask_t unvisited = NODE_MASK_ALL; + s32 cpu = -EBUSY; + + if (!static_branch_maybe(CONFIG_NUMA, &scx_builtin_idle_per_node)) + return pick_idle_cpu_from_node(cpus_allowed, NUMA_NO_NODE, flags); + + /* + * If an initial node is not specified, start with the current + * node. + */ + if (node == NUMA_NO_NODE) + node = numa_node_id(); + + /* + * Traverse all nodes in order of increasing distance, starting + * from @node. + * + * This loop is O(N^2), with N being the amount of NUMA nodes, + * which might be quite expensive in large NUMA systems. However, + * this complexity comes into play only when a scheduler enables + * SCX_OPS_BUILTIN_IDLE_PER_NODE and it's requesting an idle CPU + * without specifying a target NUMA node, so it shouldn't be a + * bottleneck is most cases. + * + * As a future optimization we may want to cache the list of hop + * nodes in a per-node array, instead of actually traversing them + * every time. + */ + for_each_numa_node(node, unvisited, N_POSSIBLE) { + cpu = pick_idle_cpu_from_node(cpus_allowed, node, flags); + if (cpu >= 0) + break; + + /* + * Check if the search is restricted to the same core or + * the same node. + */ + if (flags & SCX_PICK_IDLE_IN_NODE) + break; + } + + return cpu; +} + /* * Return the amount of CPUs in the same LLC domain of @cpu (or zero if the LLC * domain is not defined). @@ -310,6 +427,7 @@ s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, bool { const struct cpumask *llc_cpus = NULL; const struct cpumask *numa_cpus = NULL; + int node = idle_cpu_to_node(prev_cpu); s32 cpu; *found = false; @@ -367,9 +485,9 @@ s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, bool * piled up on it even if there is an idle core elsewhere on * the system. */ - if (!cpumask_empty(idle_masks.cpu) && - !(current->flags & PF_EXITING) && - cpu_rq(cpu)->scx.local_dsq.nr == 0) { + if (!(current->flags & PF_EXITING) && + cpu_rq(cpu)->scx.local_dsq.nr == 0 && + !cpumask_empty(idle_cpumask(cpu_to_node(cpu))->cpu)) { if (cpumask_test_cpu(cpu, p->cpus_ptr)) goto cpu_found; } @@ -383,7 +501,7 @@ s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, bool /* * Keep using @prev_cpu if it's part of a fully idle core. */ - if (cpumask_test_cpu(prev_cpu, idle_masks.smt) && + if (cpumask_test_cpu(prev_cpu, idle_cpumask(node)->smt) && scx_idle_test_and_clear_cpu(prev_cpu)) { cpu = prev_cpu; goto cpu_found; @@ -393,7 +511,7 @@ s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, bool * Search for any fully idle core in the same LLC domain. */ if (llc_cpus) { - cpu = scx_pick_idle_cpu(llc_cpus, SCX_PICK_IDLE_CORE); + cpu = pick_idle_cpu_from_node(llc_cpus, node, SCX_PICK_IDLE_CORE); if (cpu >= 0) goto cpu_found; } @@ -402,15 +520,19 @@ s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, bool * Search for any fully idle core in the same NUMA node. */ if (numa_cpus) { - cpu = scx_pick_idle_cpu(numa_cpus, SCX_PICK_IDLE_CORE); + cpu = pick_idle_cpu_from_node(numa_cpus, node, SCX_PICK_IDLE_CORE); if (cpu >= 0) goto cpu_found; } /* * Search for any full idle core usable by the task. + * + * If NUMA aware idle selection is enabled, the search will + * begin in prev_cpu's node and proceed to other nodes in + * order of increasing distance. */ - cpu = scx_pick_idle_cpu(p->cpus_ptr, SCX_PICK_IDLE_CORE); + cpu = scx_pick_idle_cpu(p->cpus_ptr, node, SCX_PICK_IDLE_CORE); if (cpu >= 0) goto cpu_found; } @@ -427,7 +549,7 @@ s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, bool * Search for any idle CPU in the same LLC domain. */ if (llc_cpus) { - cpu = scx_pick_idle_cpu(llc_cpus, 0); + cpu = pick_idle_cpu_from_node(llc_cpus, node, 0); if (cpu >= 0) goto cpu_found; } @@ -436,7 +558,7 @@ s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, bool * Search for any idle CPU in the same NUMA node. */ if (numa_cpus) { - cpu = scx_pick_idle_cpu(numa_cpus, 0); + cpu = pick_idle_cpu_from_node(numa_cpus, node, 0); if (cpu >= 0) goto cpu_found; } @@ -444,7 +566,7 @@ s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, bool /* * Search for any idle CPU usable by the task. */ - cpu = scx_pick_idle_cpu(p->cpus_ptr, 0); + cpu = scx_pick_idle_cpu(p->cpus_ptr, node, 0); if (cpu >= 0) goto cpu_found; @@ -460,38 +582,50 @@ s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, bool void scx_idle_reset_masks(void) { + int node; + + if (!static_branch_maybe(CONFIG_NUMA, &scx_builtin_idle_per_node)) { + cpumask_copy(idle_cpumask(NUMA_NO_NODE)->cpu, cpu_online_mask); + cpumask_copy(idle_cpumask(NUMA_NO_NODE)->smt, cpu_online_mask); + return; + } + /* * Consider all online cpus idle. Should converge to the actual state * quickly. */ - cpumask_copy(idle_masks.cpu, cpu_online_mask); - cpumask_copy(idle_masks.smt, cpu_online_mask); -} + for_each_node(node) { + const struct cpumask *node_mask = cpumask_of_node(node); + struct cpumask *idle_cpus = idle_cpumask(node)->cpu; + struct cpumask *idle_smts = idle_cpumask(node)->smt; -void scx_idle_init_masks(void) -{ - BUG_ON(!alloc_cpumask_var(&idle_masks.cpu, GFP_KERNEL)); - BUG_ON(!alloc_cpumask_var(&idle_masks.smt, GFP_KERNEL)); + cpumask_and(idle_cpus, cpu_online_mask, node_mask); + cpumask_copy(idle_smts, idle_cpus); + } } static void update_builtin_idle(int cpu, bool idle) { - assign_cpu(cpu, idle_masks.cpu, idle); + int node = idle_cpu_to_node(cpu); + struct cpumask *idle_cpus = idle_cpumask(node)->cpu; + + assign_cpu(cpu, idle_cpus, idle); #ifdef CONFIG_SCHED_SMT if (sched_smt_active()) { const struct cpumask *smt = cpu_smt_mask(cpu); + struct cpumask *idle_smts = idle_cpumask(node)->smt; if (idle) { /* - * idle_masks.smt handling is racy but that's fine as - * it's only for optimization and self-correcting. + * idle_smt handling is racy but that's fine as it's + * only for optimization and self-correcting. */ - if (!cpumask_subset(smt, idle_masks.cpu)) + if (!cpumask_subset(smt, idle_cpus)) return; - cpumask_or(idle_masks.smt, idle_masks.smt, smt); + cpumask_or(idle_smts, idle_smts, smt); } else { - cpumask_andnot(idle_masks.smt, idle_masks.smt, smt); + cpumask_andnot(idle_smts, idle_smts, smt); } } #endif @@ -599,15 +733,21 @@ __bpf_kfunc s32 scx_bpf_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, * scx_bpf_get_idle_cpumask - Get a referenced kptr to the idle-tracking * per-CPU cpumask. * - * Returns NULL if idle tracking is not enabled, or running on a UP kernel. + * Returns an empty mask if idle tracking is not enabled, or running on a + * UP kernel. */ __bpf_kfunc const struct cpumask *scx_bpf_get_idle_cpumask(void) { if (!check_builtin_idle_enabled()) return cpu_none_mask; + if (static_branch_unlikely(&scx_builtin_idle_per_node)) { + scx_ops_error("SCX_OPS_BUILTIN_IDLE_PER_NODE enabled"); + return cpu_none_mask; + } + #ifdef CONFIG_SMP - return idle_masks.cpu; + return idle_cpumask(NUMA_NO_NODE)->cpu; #else return cpu_none_mask; #endif @@ -618,18 +758,24 @@ __bpf_kfunc const struct cpumask *scx_bpf_get_idle_cpumask(void) * per-physical-core cpumask. Can be used to determine if an entire physical * core is free. * - * Returns NULL if idle tracking is not enabled, or running on a UP kernel. + * Returns an empty mask if idle tracking is not enabled, or running on a + * UP kernel. */ __bpf_kfunc const struct cpumask *scx_bpf_get_idle_smtmask(void) { if (!check_builtin_idle_enabled()) return cpu_none_mask; + if (static_branch_unlikely(&scx_builtin_idle_per_node)) { + scx_ops_error("SCX_OPS_BUILTIN_IDLE_PER_NODE enabled"); + return cpu_none_mask; + } + #ifdef CONFIG_SMP if (sched_smt_active()) - return idle_masks.smt; + return idle_cpumask(NUMA_NO_NODE)->smt; else - return idle_masks.cpu; + return idle_cpumask(NUMA_NO_NODE)->cpu; #else return cpu_none_mask; #endif @@ -696,7 +842,7 @@ __bpf_kfunc s32 scx_bpf_pick_idle_cpu(const struct cpumask *cpus_allowed, if (!check_builtin_idle_enabled()) return -EBUSY; - return scx_pick_idle_cpu(cpus_allowed, flags); + return scx_pick_idle_cpu(cpus_allowed, NUMA_NO_NODE, flags); } /** @@ -719,7 +865,7 @@ __bpf_kfunc s32 scx_bpf_pick_any_cpu(const struct cpumask *cpus_allowed, s32 cpu; if (static_branch_likely(&scx_builtin_idle_enabled)) { - cpu = scx_pick_idle_cpu(cpus_allowed, flags); + cpu = scx_pick_idle_cpu(cpus_allowed, NUMA_NO_NODE, flags); if (cpu >= 0) return cpu; } diff --git a/kernel/sched/ext_idle.h b/kernel/sched/ext_idle.h index d005bd22c19a5..b00ed5ad48e89 100644 --- a/kernel/sched/ext_idle.h +++ b/kernel/sched/ext_idle.h @@ -13,6 +13,7 @@ struct sched_ext_ops; extern struct static_key_false scx_builtin_idle_enabled; +extern struct static_key_false scx_builtin_idle_per_node; #ifdef CONFIG_SMP extern struct static_key_false scx_selcpu_topo_llc; @@ -22,13 +23,18 @@ void scx_idle_update_selcpu_topology(struct sched_ext_ops *ops); void scx_idle_reset_masks(void); void scx_idle_init_masks(void); bool scx_idle_test_and_clear_cpu(int cpu); -s32 scx_pick_idle_cpu(const struct cpumask *cpus_allowed, u64 flags); +s32 pick_idle_cpu_from_node(const struct cpumask *cpus_allowed, int node, u64 flags); +s32 scx_pick_idle_cpu(const struct cpumask *cpus_allowed, int node, u64 flags); #else /* !CONFIG_SMP */ static inline void scx_idle_update_selcpu_topology(struct sched_ext_ops *ops) {} static inline void scx_idle_reset_masks(void) {} static inline void scx_idle_init_masks(void) {} static inline bool scx_idle_test_and_clear_cpu(int cpu) { return false; } -static inline s32 scx_pick_idle_cpu(const struct cpumask *cpus_allowed, u64 flags) +static inline s32 pick_idle_cpu_from_node(const struct cpumask *cpus_allowed, int node, u64 flags) +{ + return -EBUSY; +} +static inline s32 scx_pick_idle_cpu(const struct cpumask *cpus_allowed, int node, u64 flags) { return -EBUSY; } @@ -36,6 +42,7 @@ static inline s32 scx_pick_idle_cpu(const struct cpumask *cpus_allowed, u64 flag s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, bool *found); +extern void scx_idle_init_masks(void); extern int scx_idle_init(void); #endif /* _KERNEL_SCHED_EXT_IDLE_H */ From patchwork Fri Feb 7 20:40:53 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrea Righi X-Patchwork-Id: 13965851 Received: from NAM04-DM6-obe.outbound.protection.outlook.com (mail-dm6nam04on2041.outbound.protection.outlook.com [40.107.102.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 204BE1D6193; Fri, 7 Feb 2025 21:11:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.102.41 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738962710; cv=fail; b=TFCrHjgSKcOJ/y6NOaRY+TAEHxQ4dxEz+OyobadTaF5Z3c48zdGmq9DVj0J0v0tWCAK5cxCn3x6wDbuvYmSsYBrovzDZ8xiQF5iVQ8nGr8l/ahv1bzaBGPQtmdJ90pLomIyUy82jHmOM7vOMd/CXySx2pKA5y9yWAhbl1tECwWs= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738962710; c=relaxed/simple; bh=vhJSlCLyUCZAwqWjgv3YeTWmabLIV9SytmPIcBf6bl8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=DOpRLnw+R1wwyDl6Lx0qek0OlBiC/ZA7SQ0WTY+yjH9Y+yUzH/Hm9N4OkNlzA3pIzpv2kAXy7PrFm55Bv7NH6CA+WkMuVL5Sm225yY8OVpWEBHX5+xlkqqLSCJKN1FH1tRxwx1kbzfEgYKGQsIorMRDVLC6kmejKiVrlBjEbAUg= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=iowgXcp2; arc=fail smtp.client-ip=40.107.102.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="iowgXcp2" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=voEEQW/kxNc2N4lXuITDrqwYRm+O2UT3lNtm5reF2smsrPFfDD395coIEMoDC4K+2W5Cqxpg6UpLEJBtVYY76b4gPhzadVz8wljNfBa4NCg+EUe+V3nd2+xNzA/kQs6BkMRtHpzRv2F61NgRsmba9Nt1cAedq+EVrGH1qbKWbg8UOdGdOvS+PSduB3bc7ohxNQRNooisgut8Yxnc+GdkhIKG4RGd5rD+glZDGWue8w8es5px/sLnEAIgtPRDR0O0207D+AYz00/XWQ8Ji1Q/QSh+T8nG33DVay33psM9xMvf8eRhBI5SWTAksmVemymdz4qx6hn5ZT/DFrFAedm89g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=n54boQg5TK+GwNXu96pP/LAtDrvsAvbwDGt4DvK65sM=; b=B8mEFnDq165YrmsLcSjYOdC4KyH0zhcHBwLsawi9p6qVMDLBPNTxeGIcyHSFYBsIdK3tfn+AX76Sg/RmtRNEI27Su59yd0GuuJ8GpoyhPfrzlydiZGZiNki76clBJBw/j6+Nc4IBLcIOgU9hdOwDDmNOQVyv3hg+sj6KQERe7i06artrEUhb7IS6AI8f3pfm1RLSUxnUcJgJsn1xuYcbNdHNVOOntp0u/mtSfgQjqdVhXsL8JyalIaL1zLaVldfHD3FGYiIeApOkjiMfLz5hQygMO9+OlgQkEchpFWrp9vvF+jnYcNu9DZf5SSxQ88CxxZyoFwl4h0vubQtEgh5CeQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=n54boQg5TK+GwNXu96pP/LAtDrvsAvbwDGt4DvK65sM=; b=iowgXcp2nLqAOvVwIvGD6P5xT9Npyhl6GFALlpIz/HIYbScZ0bASqDy1K/MP84bXAk10eDrG8omVX5yLOltJDAOku0EmMFGb7eBIfWJ60tBBc9Lu7MtWZFwLLUtDT19YqCd4Xys3gJrMT0yYQhryby44fn86A5VN4SH3iq54b5aS9dgVGOMCipQRsrBfjSAi8Sltg89IHlxKR2mlFJCaBZGR2qCzTzsojdYpkk3amR1oAwJivZJQYksff1JF6DUq/tl7Ugny3pKjCi5rDeF/BpeD0jQmFo7qEQ6rMgQb7zZiQQJxDzo9XfhoudPcdiPkqXvzdo8ljYTZ9wHy1aWiyQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) by PH7PR12MB7380.namprd12.prod.outlook.com (2603:10b6:510:20f::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8422.14; Fri, 7 Feb 2025 21:11:44 +0000 Received: from CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5]) by CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5%7]) with mapi id 15.20.8422.010; Fri, 7 Feb 2025 21:11:44 +0000 From: Andrea Righi To: Tejun Heo , David Vernet , Changwoo Min Cc: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Ian May , bpf@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 6/6] sched_ext: idle: Introduce node-aware idle cpu kfunc helpers Date: Fri, 7 Feb 2025 21:40:53 +0100 Message-ID: <20250207211104.30009-7-arighi@nvidia.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250207211104.30009-1-arighi@nvidia.com> References: <20250207211104.30009-1-arighi@nvidia.com> X-ClientProxiedBy: FR0P281CA0179.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:b4::19) To CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY5PR12MB6405:EE_|PH7PR12MB7380:EE_ X-MS-Office365-Filtering-Correlation-Id: 376eac06-7325-4b9c-c1a1-08dd47bc0620 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|366016|1800799024; X-Microsoft-Antispam-Message-Info: COxE1tCQsC8qn4rIjfJ2Od/VUePHJXWXTz24wP64HQ169LP6ML6fE27YgvKfOsHXOZmMUTW1W9qffDlJAl152P5sKiW4nOcDQY/Gk8/iBX44sUUHL04XkHCt7sNmYGfIGtsrHDvYjtSNpZ93m7qXt1dKKgL019acPDVQLNNHujIBzOKb2/415lTYXn9/w8pZTA1AHAJWyJDWPqZUCwIZDBOP2zkuRuvelaFveKf+GuSo/OOjzZXyWKBQV7s4KN7rUMUFAiB4Lz0E9xZbGXQ/QSA8isBUL/YMEJzrxBzny8ZAnC4KxKoSV5NJ/cacF1l/5H+o8pbsNkFF5o40LSZRLXpxWn1m7MjiJDYAQ82uykL79UHcM9xQiTonWRwYeW4bPmmQrbK2LgAdgIcD2kEBxW8mtk7bxKtm6INaMV4KOupcAIRXZvYbbGeQkMmu3w65V9L4o5VD4qLekPJvkYE8zrj+sf2pgrDi3rKLs6Us0o/EHjHyxR9hka7QAYopzGSsqGmBN/t+WtsZp9t4yuE8cFsIAztambVujVaI1IK7J1OUmB3/9x9shXfKLgcBKwXU0RTqBOCITHHWrZfFRAjyXUs+OaB1vvvz24FX2U9CW6jfB+nt/D9jsLj5fvvDueS/HYzxBIKofFLjYbnkQvdf17SxUidaEgRCxGQ/KV2dFUce3Yds4rwGF59OllIklt7BY0P4Bz14OVfgjVw+sd6lcqiILy1Px1YzPnJugdzsrzMLG6CSii7Hv788hKG4gyz1wm9vAcvo04zvvMTo3gb/sOT4MzvenDOm+j5691tzHldFaHpaXoq2t1YUP8amaDhAmU63MTYQtbLpwkrVdWq5ATWhWzEDZRYDMyd5LxapFhleQEDxlYpJBX6/S1QTS91g8h+KkmyOJFX5/rgwxWNgkRFRU9MPJlNL5Hu0pSs5cklC1x61G5JESqam1R9fcOZvzzvnyp+qGNDMtBFsheCI1NtOIUBNo/kqytDkOYr7neJEvan0GMpZ3qQ1ZSYQJMBSqa4ym+3/2iyTQ9eDAvgqTQbARm1b4NYarC45bLdHrPQsavdH73MrzSH4eAdRPb4T92BFM8DdRyx+lLFksu46Wvt3lUjBda5Ub4jrSukSGWv/Yq6/5TgxwOk1GDd9M317KrWVV4hotZK155TpgMNuGCYLYyBTFbO6tll5HdQgXSSQlMAVgNWuwxkbuLpDBe84ELdUHMlJtkDWIy8DuNyb2fZWFNDXgTFLrk0Mc73GLDosdc5wxIY1vBPZzknfLQnrCajiF0v5/SjnY8ySNt8RXU/2n2Njce6tsEQEUo8CThh10dl6wCJg+W4mxmWCa0PzD27Fc3m52GhsSs6cS/Mr6uaNve0KKqfnRT0QjMd3J6ZRAmTtxDuiTl3vs+2G3/0x X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CY5PR12MB6405.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(366016)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Np9Ll9F8Aat1ZKC1bS/ftmPJDfFHq/rftZgAwoJA0NmpJG9xSY8p+4ExP4+r/4dqnJN5c76Bzep6AIdgRWtlYFHlan9Shyf53NP9GC57h/cAUV6OZMkwNh36opQn6b/qA4w1uXpHmB3IbypZKgDdeKLDTVcn6YyC+xzOzbuGl7KwYHan8/4hC08zu5KjLlNU3QfXZF5an8STNBpkz0QY/97EmpqNtAkIz29wxihhgwEDSl2xiWtjHI3HZywSOs9fU1VKmUDh4eBvhcaapywuVXCQ/H5SHFWz2Rflm1ogYVeY8n/UmQ2Xvgvlf4aXcE50LlqbX36VGKb0B3FnQZJ2wH6ipd0NaETyp7OSQBRRiUSas1P6CF/WbYqUK1VZqf6S/V/ggSlPhKP3SEGm2Nx40UUjCH8ZIoAtEs5m/mlIrVQxEsKSu9lbNMsoAbX3raa2QAgbIKlbUFZPHcwyfsFgldy9RiUZpiZ1co2pie/hXavxDBrbBVhXjZA1agVXkVUORTy8hJKPWsWzU0u237aXwoFm19+L3ZvWNr6dmub9WhzOM6Tu6CJsMYLPE/zwUx0/OxE1H6wDSIIsbSyiZgQxXggaAmT6qgSaEftiJWI+17GbKNOkViAIkHyhbd8CMfbI4IgdHV2H6v/PRVsVSEWYSHfAGMSmAU6P27GbseXRQroD3zLen5VxNH+UzxYyfqUe6J5qyOuFN0fMZ5RlZzzC2du7ZI+0UAVKcqPAIGayGS+XLslSkCOD92Q0wvdgJHWDM9aW95mRsVGLMv0yEHg0jkVbyjscRNSn346bxxTJ2LjwjVuyW3Jem9imau6EF+de32sxVuaL9RixhOKAE3kxuKSN57TlGCt1zpK20wAkcSlUj47M9OqmZVXiYt+Lm7AKoSKyVystYeibFFtTlpf/aQw6qFuIrogcxirph58y9LuEsj7O1r9aDEo8nmZiV8DfcKfP9N2yKdfe19RgVNuXUh1ZKV0AF4vZZ/5hNAJ6+IfcGpwUHmMZ0Caf8ZjRX+9eiiq1FnHsXyuRMklFhWgn5JYYaWuUIS2bDkZEYWwBDnR4uaNdeoHne0WxpnOrNu9+UTx8qMCw0jzhz9oMO2KjaA9dD0vlu3TCrNOIbY4ZmtRjCwZZEXeN311qKAVPnpeLLm/fLnye8QFv4MhUkzAI0fVSjJkc04n3yVpa4GISbreDEUMiWuOKOvqsw/WDSJBPqAU6tOaJ9DyEw7IkXTy7NgIaIi5ol6hFFVUiGRVf30BKM/ts+7vL+6VIiQDfWR8o+FWSsaPidocexRMVRlt4Xuyn6GBgm6baZxLTjRmvBfwwAI9HXlFpJ0JAL6Q/eEpk8KKNfoUAffUfGekj52gr4ZfUmdTar7GsXvggfLmx2NZhbHoNLeOTEVH3jLZNgTGNgUPToHEDj2tVtRhaQ1UiiN4q5qfPdvD/6yACFIAF4XyIsCtVtb9nCpbuKVdmfc4APEfiL36XbyuLJx/JeP03fWj2ljKF4h82o5m/158xhe4qNkt1V0CqkWnFTQhOu3uw9tPGQXYd0lLK/geTo4rlY6WMtrDpDGxHwke58uqWRSXdXE5574XgMgakbiQeqe0D X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 376eac06-7325-4b9c-c1a1-08dd47bc0620 X-MS-Exchange-CrossTenant-AuthSource: CY5PR12MB6405.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Feb 2025 21:11:44.1103 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: u3ssZN7YO9YLbFZ/WWg6yXO4PKjbujraDCNMKgyM3KY8mQoXqZFCfJG8kPE4BROfCXr+9usfMveUWzFnc56f/w== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB7380 Add the following kfunc's to provide scx schedulers direct access to per-node idle cpumasks information: int scx_bpf_cpu_to_node(s32 cpu) const struct cpumask *scx_bpf_get_idle_cpumask_node(int node) const struct cpumask *scx_bpf_get_idle_smtmask_node(int node) s32 scx_bpf_pick_idle_cpu_node(const cpumask_t *cpus_allowed, int node, u64 flags) Signed-off-by: Andrea Righi --- kernel/sched/ext_idle.c | 121 +++++++++++++++++++++++ tools/sched_ext/include/scx/common.bpf.h | 4 + tools/sched_ext/include/scx/compat.bpf.h | 19 ++++ 3 files changed, 144 insertions(+) diff --git a/kernel/sched/ext_idle.c b/kernel/sched/ext_idle.c index 4b90ec9018c1a..260a26c6cb0b8 100644 --- a/kernel/sched/ext_idle.c +++ b/kernel/sched/ext_idle.c @@ -686,6 +686,33 @@ void __scx_update_idle(struct rq *rq, bool idle, bool do_notify) /******************************************************************************** * Helpers that can be called from the BPF scheduler. */ + +static int validate_node(int node) +{ + if (!static_branch_likely(&scx_builtin_idle_per_node)) { + scx_ops_error("per-node idle tracking is disabled"); + return -EOPNOTSUPP; + } + + /* Return no entry for NUMA_NO_NODE (not a critical scx error) */ + if (node == NUMA_NO_NODE) + return -ENOENT; + + /* Make sure node is in a valid range */ + if (node < 0 || node >= nr_node_ids) { + scx_ops_error("invalid node %d", node); + return -EINVAL; + } + + /* Make sure the node is part of the set of possible nodes */ + if (!node_possible(node)) { + scx_ops_error("unavailable node %d", node); + return -EINVAL; + } + + return node; +} + __bpf_kfunc_start_defs(); static bool check_builtin_idle_enabled(void) @@ -697,6 +724,21 @@ static bool check_builtin_idle_enabled(void) return false; } +/** + * scx_bpf_cpu_to_node - Return the NUMA node the given @cpu belongs to + */ +__bpf_kfunc int scx_bpf_cpu_to_node(s32 cpu) +{ +#ifdef CONFIG_NUMA + if (cpu < 0 || cpu >= nr_cpu_ids) + return -EINVAL; + + return idle_cpu_to_node(cpu); +#else + return 0; +#endif +} + /** * scx_bpf_select_cpu_dfl - The default implementation of ops.select_cpu() * @p: task_struct to select a CPU for @@ -729,6 +771,27 @@ __bpf_kfunc s32 scx_bpf_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, return prev_cpu; } +/** + * scx_bpf_get_idle_cpumask_node - Get a referenced kptr to the + * idle-tracking per-CPU cpumask of a target NUMA node. + * + * Returns an empty cpumask if idle tracking is not enabled, if @node is + * not valid, or running on a UP kernel. In this case the actual error will + * be reported to the BPF scheduler via scx_ops_error(). + */ +__bpf_kfunc const struct cpumask *scx_bpf_get_idle_cpumask_node(int node) +{ + node = validate_node(node); + if (node < 0) + return cpu_none_mask; + +#ifdef CONFIG_SMP + return idle_cpumask(node)->cpu; +#else + return cpu_none_mask; +#endif +} + /** * scx_bpf_get_idle_cpumask - Get a referenced kptr to the idle-tracking * per-CPU cpumask. @@ -753,6 +816,31 @@ __bpf_kfunc const struct cpumask *scx_bpf_get_idle_cpumask(void) #endif } +/** + * scx_bpf_get_idle_smtmask_node - Get a referenced kptr to the + * idle-tracking, per-physical-core cpumask of a target NUMA node. Can be + * used to determine if an entire physical core is free. + * + * Returns an empty cpumask if idle tracking is not enabled, if @node is + * not valid, or running on a UP kernel. In this case the actual error will + * be reported to the BPF scheduler via scx_ops_error(). + */ +__bpf_kfunc const struct cpumask *scx_bpf_get_idle_smtmask_node(int node) +{ + node = validate_node(node); + if (node < 0) + return cpu_none_mask; + +#ifdef CONFIG_SMP + if (sched_smt_active()) + return idle_cpumask(node)->smt; + else + return idle_cpumask(node)->cpu; +#else + return cpu_none_mask; +#endif +} + /** * scx_bpf_get_idle_smtmask - Get a referenced kptr to the idle-tracking, * per-physical-core cpumask. Can be used to determine if an entire physical @@ -817,6 +905,35 @@ __bpf_kfunc bool scx_bpf_test_and_clear_cpu_idle(s32 cpu) return false; } +/** + * scx_bpf_pick_idle_cpu_node - Pick and claim an idle cpu from a NUMA node + * @cpus_allowed: Allowed cpumask + * @node: target NUMA node + * @flags: %SCX_PICK_IDLE_* flags + * + * Pick and claim an idle cpu in @cpus_allowed from the NUMA node @node. + * + * Returns the picked idle cpu number on success, or -%EBUSY if no matching + * cpu was found. + * + * The search starts from @node and proceeds to other online NUMA nodes in + * order of increasing distance (unless SCX_PICK_IDLE_IN_NODE is specified, + * in which case the search is limited to the target @node). + * + * Always returns an error if ops.update_idle() is implemented and + * %SCX_OPS_KEEP_BUILTIN_IDLE is not set, or if + * %SCX_OPS_BUILTIN_IDLE_PER_NODE is not set. + */ +__bpf_kfunc s32 scx_bpf_pick_idle_cpu_node(const struct cpumask *cpus_allowed, + int node, u64 flags) +{ + node = validate_node(node); + if (node < 0) + return node; + + return scx_pick_idle_cpu(cpus_allowed, node, flags); +} + /** * scx_bpf_pick_idle_cpu - Pick and claim an idle cpu * @cpus_allowed: Allowed cpumask @@ -880,10 +997,14 @@ __bpf_kfunc s32 scx_bpf_pick_any_cpu(const struct cpumask *cpus_allowed, __bpf_kfunc_end_defs(); BTF_KFUNCS_START(scx_kfunc_ids_idle) +BTF_ID_FLAGS(func, scx_bpf_cpu_to_node) +BTF_ID_FLAGS(func, scx_bpf_get_idle_cpumask_node, KF_ACQUIRE) BTF_ID_FLAGS(func, scx_bpf_get_idle_cpumask, KF_ACQUIRE) +BTF_ID_FLAGS(func, scx_bpf_get_idle_smtmask_node, KF_ACQUIRE) BTF_ID_FLAGS(func, scx_bpf_get_idle_smtmask, KF_ACQUIRE) BTF_ID_FLAGS(func, scx_bpf_put_idle_cpumask, KF_RELEASE) BTF_ID_FLAGS(func, scx_bpf_test_and_clear_cpu_idle) +BTF_ID_FLAGS(func, scx_bpf_pick_idle_cpu_node, KF_RCU) BTF_ID_FLAGS(func, scx_bpf_pick_idle_cpu, KF_RCU) BTF_ID_FLAGS(func, scx_bpf_pick_any_cpu, KF_RCU) BTF_KFUNCS_END(scx_kfunc_ids_idle) diff --git a/tools/sched_ext/include/scx/common.bpf.h b/tools/sched_ext/include/scx/common.bpf.h index 7055400030241..7dd8ba2964553 100644 --- a/tools/sched_ext/include/scx/common.bpf.h +++ b/tools/sched_ext/include/scx/common.bpf.h @@ -63,13 +63,17 @@ u32 scx_bpf_cpuperf_cap(s32 cpu) __ksym __weak; u32 scx_bpf_cpuperf_cur(s32 cpu) __ksym __weak; void scx_bpf_cpuperf_set(s32 cpu, u32 perf) __ksym __weak; u32 scx_bpf_nr_cpu_ids(void) __ksym __weak; +int scx_bpf_cpu_to_node(s32 cpu) __ksym __weak; const struct cpumask *scx_bpf_get_possible_cpumask(void) __ksym __weak; const struct cpumask *scx_bpf_get_online_cpumask(void) __ksym __weak; void scx_bpf_put_cpumask(const struct cpumask *cpumask) __ksym __weak; +const struct cpumask *scx_bpf_get_idle_cpumask_node(int node) __ksym __weak; const struct cpumask *scx_bpf_get_idle_cpumask(void) __ksym; +const struct cpumask *scx_bpf_get_idle_smtmask_node(int node) __ksym __weak; const struct cpumask *scx_bpf_get_idle_smtmask(void) __ksym; void scx_bpf_put_idle_cpumask(const struct cpumask *cpumask) __ksym; bool scx_bpf_test_and_clear_cpu_idle(s32 cpu) __ksym; +s32 scx_bpf_pick_idle_cpu_node(const cpumask_t *cpus_allowed, int node, u64 flags) __ksym __weak; s32 scx_bpf_pick_idle_cpu(const cpumask_t *cpus_allowed, u64 flags) __ksym; s32 scx_bpf_pick_any_cpu(const cpumask_t *cpus_allowed, u64 flags) __ksym; bool scx_bpf_task_running(const struct task_struct *p) __ksym; diff --git a/tools/sched_ext/include/scx/compat.bpf.h b/tools/sched_ext/include/scx/compat.bpf.h index 50e1499ae0935..caa1a80f9a60c 100644 --- a/tools/sched_ext/include/scx/compat.bpf.h +++ b/tools/sched_ext/include/scx/compat.bpf.h @@ -130,6 +130,25 @@ bool scx_bpf_dispatch_vtime_from_dsq___compat(struct bpf_iter_scx_dsq *it__iter, scx_bpf_now() : \ bpf_ktime_get_ns()) +#define __COMPAT_scx_bpf_cpu_to_node(cpu) \ + (bpf_ksym_exists(scx_bpf_cpu_to_node) ? \ + scx_bpf_cpu_to_node(cpu) : 0) + +#define __COMPAT_scx_bpf_get_idle_cpumask_node(node) \ + (bpf_ksym_exists(scx_bpf_get_idle_cpumask_node) ? \ + scx_bpf_get_idle_cpumask_node(node) : \ + scx_bpf_get_idle_cpumask()) \ + +#define __COMPAT_scx_bpf_get_idle_smtmask_node(node) \ + (bpf_ksym_exists(scx_bpf_get_idle_smtmask_node) ? \ + scx_bpf_get_idle_smtmask_node(node) : \ + scx_bpf_get_idle_smtmask()) + +#define __COMPAT_scx_bpf_pick_idle_cpu_node(cpus_allowed, node, flags) \ + (bpf_ksym_exists(scx_bpf_pick_idle_cpu_node) ? \ + scx_bpf_pick_idle_cpu_node(cpus_allowed, node, flags) : \ + scx_bpf_pick_idle_cpu(cpus_allowed, flags)) + /* * Define sched_ext_ops. This may be expanded to define multiple variants for * backward compatibility. See compat.h::SCX_OPS_LOAD/ATTACH().