From patchwork Fri Dec 20 15:11:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrea Righi X-Patchwork-Id: 13916897 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2042.outbound.protection.outlook.com [40.107.94.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BCD1D218EBE; Fri, 20 Dec 2024 15:41:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.94.42 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734709290; cv=fail; b=MwXf85AAS43c5iLdP7l69h2iuZtUHMe8/hIibM/qNN3nWZn05XH470URAhCc2s8M/+1G/R8iUR4yOFgi6VaYTgDV8etluMOHz2SofUkEpBrO+lfdYAV1OP/zRN47XFZ87DzfTG405ajNkSEd7YpaOyuW9A1G43GSu+fspoLm35U= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734709290; c=relaxed/simple; bh=BrBs5mzicuG5ZdP3RUCKmuf2En7S5TxXbYK88PAnkQw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=odn/z+IHwpGlqJUhsqVH/+7rR83sMJ95BuaWW3dwT29F2YOPOnmHEXZClOOD1FlZDf/VXlVXEWnfMQFkb+rfi3fYzTVqR1+zsQkypoWGtJEkHi38g95vacnVQI7Fz8TbUGIRk610N3+FWptk9mY7vitckjRawfexvCzlQUWjves= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=JaB/Xt5b; arc=fail smtp.client-ip=40.107.94.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="JaB/Xt5b" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=YjHDEOeK3RWei08fKjosvnwtrmtQ3QdRKSEE7tOA03F+bkgKEBfN8VSxeTZKERoZdYhPRgPZqFwuyg9Q9sChK+I+qyFgWszSSn56gRQi5B4PmZEjN5jC4OkaIkNpOBnb5ziBs7dWFiSB9Ji1oW9WkFD4eBWPFhRTMsbBhNwyXbxN4x86F6MKD6R1G1GyhMMt2QPnhsvW9x9P0Z/72PGrz+sDUetgZa7JGCKq8DGwUDomBeXdh4PyTLH7wdhx4sylKPt8pt1x0DLCeUG5QRE/A0gmvqkKr9leHIX6PM3/C/SyNre+29x1srDJwQ7/cbxejKytnpdINLKHALijckwgXg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=fkOUdcjd3q8CSg46dJCG0LCHLYMhldMZ+SAfCYi6qdw=; b=bQnPDEL3WDB84AqtSvexNANsT6IWlFrTKqK3k5g6udpUd0T5ngigekVerSGKwI7XGkQx+z4UdDWcPEel4WG924TKEPZB0rP8ZaAOvBi/EinE4O8KndM61L3KS7gwuO1po3VUIFT+hAfcV74bSIDvR2Cvp0aJCp+hIn4BDr0HKt0GMwF6x+2/IZRk5f6+3Tw3QCzulkCltOYkfxdngWX0nrlAeyv/PompmEDp2pP2iiFop/DButpARRBvNsrysTtzoTSiCdWOEAjB/sVumkJVrpQjzVNqYk63OJIIxCQV44IiySqWnERQRUV0PvVY5DSHND4D1S6Fnt6ctmlyfGILzA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=fkOUdcjd3q8CSg46dJCG0LCHLYMhldMZ+SAfCYi6qdw=; b=JaB/Xt5bZCHUT//bAJE1s2dUvp0MY+kfY6x1jDXnnvcuEALTVJnHxXvW04CwjYmILOyEy8n2l+Cdd4Po1w3J512mozVMtEql1JE7L7V24S6rEIn7vs5ydCNuFGm22yglymFeRSB399SnxAAoD9XF2kmiQkdlCDlsXiswMRzeP0ADyo/++k9sB1tNtkzdfwXfSRhbL0j3O7mx03/eCaf7R2k5C4sICwQC18OWRN76Kty5jQiNdBUYgkblxEemxdbb3u1kVPG7gLa3DjchSJEtgjlgv1zIburGXpgr3VA2UCk4x5fJ+8RTKITij51GPOYmGl2wq5rq1VV9vLtOb/iXIg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) by MW4PR12MB7238.namprd12.prod.outlook.com (2603:10b6:303:229::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8272.17; Fri, 20 Dec 2024 15:41:20 +0000 Received: from CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5]) by CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5%3]) with mapi id 15.20.8272.005; Fri, 20 Dec 2024 15:41:20 +0000 From: Andrea Righi To: Tejun Heo , David Vernet , Changwoo Min Cc: Yury Norov , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , bpf@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 01/10] sched/topology: introduce for_each_numa_hop_node() / sched_numa_hop_node() Date: Fri, 20 Dec 2024 16:11:33 +0100 Message-ID: <20241220154107.287478-2-arighi@nvidia.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241220154107.287478-1-arighi@nvidia.com> References: <20241220154107.287478-1-arighi@nvidia.com> X-ClientProxiedBy: FR4P281CA0192.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:ca::10) To CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY5PR12MB6405:EE_|MW4PR12MB7238:EE_ X-MS-Office365-Filtering-Correlation-Id: 801a0846-d37b-49c6-19b2-08dd210cbfdc X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|376014|366016|1800799024; X-Microsoft-Antispam-Message-Info: Sf+DVSVdIrBhMzBRHy/PBKtmH44REcuzRhwGhthT5b32UlJVfTxVvZQeakklndh3IjTuGZVokIAP9WsE5VQKdguZ/UO0I1rEsytBdfgOnh9/rxJb7prgmae6vdAum/PvQ5039SE3kt3/sWmiGsyw+2h10i0SUPWDwe0KFUEPhfVeTAfYGsOt7UtpSHHxRsatmKe/AX7YACUav+vDALyb1nd+vtlDn4nCdP//IBsqXwjP3HuxVkj0wL/FswgRDuh/lln3rlAbSoVwhJnedbr50R9SXjUbVCihGEV0GFtUNnE6w9U/wO2K86ZrumYlsdt6SNnnSY/+0ncIAWo0ff85/2DCP3D1NX8OdFs3nrBuGpQH+kxEgRI1lPXR/sL8RIJgH/LNA1+vbJ+UGBco1xHSboaQqx7UQ/dDNnNx9irE9dZDZsFDm6iuAqzlM2DdddhZQguDG719r+JNTkVc/0M0K5s9EcddAmYb4S8i6cWyRMliiEGbU4bFnSb2MynyTxQmGIzp2DUSZK0i2JCz1mc9Vm0SGNcS/32G+d6uXCL3tqTXrFdxDx2lQ7kfZ1z3XC1fPr8u4KSWjEfUGyk0VH767x8qWfjSEZ4QSDXZe3o+PNVpSPW+gerA7EB6yiJA+oXfNJTnTeGrR3ZvPn/zpjA8bHWHcg8QPuqrmi3WMA7nRV1ptfmK6XN6kzzD2b6r8h0FINUdi/xlzNBk8tRVtamyDfYpjtpSHU4Pt0hbRMFs8Y7+newx078FX5EoysO8OoOYZ0Cas3kJpkmYQ82nWxKeRE3joecntMVgnR3xd2AeOMvDqOL0fSQDovRNyIxxkuwCdG7ZH6xV7LcgZjd+8vFSWoK9RyUq0CwY3u+OpgLYWujKcs5OqhIis9r1T209pxKeYDP4ooTS8V/5SxslhoMZs99PyCslM7TETBsPMh3Y+YPgGc2J9EFWv2PLJnICSwjvlZW07G7soH3BsZxgS+Bz7RNUnot9UawbNxcF9e+7O7quONo6tn1nDAfBR0QI3MuR4yGQVF+PJjjueGmhF/aYjj5jhLiLvQznh9U22z6jlHRS6UvaEuw4IdQdH2D3aBCCNtaynA0AbU5fXAKMNiXUY+F+34QGgHXoMokxlnG2rbAvPU/quPRIsTAUaQGMfOwXKpeVEX3rUM/AQ3mWHRIhwmua7AITthi7HskJGNzByS7cFOTQL4ZKpcQOKpxmpoXU9HWybEQPEGlgbvRUBxLNdXAVINB8K907iYu6DAMzOUtAjKnZ7iHNQ9JrpOz1ZWXXjuD1l739ibSgiFv/8T5+jYCEmkWuqgh8zHjYAiSOGmqg/TluS4DmhHw01eaUXsjzNGep1vWtr0OTKdXi6rTRare652EgrNClHfRfR+XG3vuJyaSbJUdw0odLup9nT+WE X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CY5PR12MB6405.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(7416014)(376014)(366016)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 6nWUw3q7fpBgcZcPaCpnWGVgfQi4HZUPyvokoozPE2pGqkCMImF8HQkJeZ/OzwVpn1ObpKx/dxGn87QUNAmhVEa0P1Bh/1KjyNV2bN7qR5dFNCqyed7wM/BoV8bwEMtkRuDcZm2ye1malv0yRrQzASSSLJB9HQcu5f9SXjhJzLLEVG8VOEcIxIbdsi/Dc5E6aTptUAoej5I9A0/Tf3ELitL05LI8J+CJTwx5nSHGD3Ewe89ulJeAxhLRC0dJd96DFgQhQKptpsAEGKuvy4It5kt4eHyb2Q2Nx55X7y/fGBa0imEpDskbb0AWi6TNeO3qV+ioO3PoR2cbWLRH8Nqew3lpECemDUldaU7p9R6tWUjB/V6oAADo0o9w64nLZbH4IytFmy9cEuiCdv3Y2WIewP0ylWHf0z06t8sBjqQfdWhzhrBt3IDFxBf1ee3HJLu+JP5WyhEecvoZOogV5EJdqimqG9VhkG/Lko33V8XTUY4yJAuIVki24Q6G+8QbIsBFq4eaMHY8UJzw+mCJI3ivOc6mvsnoGsUhUc4zs2w9TfzavflRaCiRoto4754HZUhpWVp6Te4j404vIyeHYiExT+bvjUIUJhpL24cJHNDYOVMOHS/qEr4+LG8Jeir8ANmCIZbbLlqdYSTR+llXH7YiRdcyg5KdLA8COPEyjc3mlEz2/0YoLp3N33Ud9hos0NYp1ZZNNiQT8L7iiPjvRGWupr0KrNvcxcswCRDcxwHh5Qz5UJ6J8oF7EhswzhhYORjYRy9pOdyOFYrACmXwQPpVdtLXFEN2Udfb6Jg1EssDzfzZ7/oBnobjkmDA2fK5K0hE28Lg3Dvpet5HO3gBt/Rug1PlVy4Agm5XeQ89PYXyUzrGvURasFkT+AUkIn15JoymDd/x9QSYaa2K7/taHXRlQZVu71Pdcgvlw1l01rIh3U5Tek0upqUFoPdyQwpvK0fKGUmeZ057mbl7PMCSlMFgp9FnU2/I14XzYjTbA88vnvmiJrurtKjMwjha7HCiZ/IR4LkICnGFBpo/PQmYnAY0Vereycei25Ht4+GApKMcuXxeNJaBKgpSRovNCeNQIRqz3EV8bMT2yvw03jKRsj4uTQxJ7qGXngaW1BeCPxpTfFG2DEQ2L5GHezAltgAkCYu80mjuWWoU3ZfVSZcaxyy9FufVNVgMelzhFmunGMkUJgUaQp9ULUsobY3CDEY1hT5fbiAZRhz7H8jVXgic6zMMiFzWQVBBvE1057Vi/3MI1n7x0Y6QNxXvYECEC3VwegJZSIK3pIcNdngjrcYDQI4qMxKYSKF3x+O8n264PxJyb9YqZp7+Mt1fbT7Spi5oMS3i7N0ODWRdEm9enSy5TEcAS1ryM7xANpNaGZBbuSYDr2PfhDWTSuwNH8AsV1y5uG6MMuRnu/U5XxfdAbo984BuYGhSayNeRvMDfe5RleRgubdUF3pflV0U/ela1+OgqyQJUa/efLVkY3R/zCxnqSzBGR00KIsrtMhgC0ueULQ6rwE2ovjMC3e8CLoU1h04y9bnT1R7pwmGya7IBtwjsuIuk67imsggFkoebVs5qSqOUKcN1BsTgmrOybaxxGHZu/YO X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 801a0846-d37b-49c6-19b2-08dd210cbfdc X-MS-Exchange-CrossTenant-AuthSource: CY5PR12MB6405.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Dec 2024 15:41:19.9546 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: +C8dY68YaADPlDSpY2kXeo7yU2cpoD643Zhs4EjV55m85mPsOF5uG+DYY9Jdr/uCvkmyAZSQqmfmvdviYPZ38w== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR12MB7238 Introduce for_each_numa_hop_node() and sched_numa_hop_node() to iterate over node IDs in order of increasing NUMA distance from a given starting node. These iterator functions are similar to for_each_numa_hop_mask() and sched_numa_hop_mask(), but instead of providing a cpumask at each iteration, they provide a node ID. Example usage: nodemask_t hop_nodes = NODE_MASK_NONE; int start = cpu_to_node(smp_processor_id()); for_each_numa_hop_node(node, start, hop_nodes, N_ONLINE) pr_info("node (%d, %d) -> \n", start, node, node_distance(start, node); Simulating the following NUMA topology in virtme-ng: $ numactl -H available: 4 nodes (0-3) node 0 cpus: 0 1 node 0 size: 1006 MB node 0 free: 928 MB node 1 cpus: 2 3 node 1 size: 1007 MB node 1 free: 986 MB node 2 cpus: 4 5 node 2 size: 889 MB node 2 free: 862 MB node 3 cpus: 6 7 node 3 size: 1006 MB node 3 free: 983 MB node distances: node 0 1 2 3 0: 10 51 31 41 1: 51 10 21 61 2: 31 21 10 11 3: 41 61 11 10 The output of the example above (on node 0) is the following: [ 84.953644] node (0, 0) -> 10 [ 84.953712] node (0, 2) -> 31 [ 84.953764] node (0, 3) -> 41 [ 84.953817] node (0, 1) -> 51 Cc: Yury Norov Signed-off-by: Andrea Righi --- include/linux/topology.h | 28 ++++++++++++++++++++++- kernel/sched/topology.c | 49 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 76 insertions(+), 1 deletion(-) diff --git a/include/linux/topology.h b/include/linux/topology.h index 52f5850730b3..d9014d90580d 100644 --- a/include/linux/topology.h +++ b/include/linux/topology.h @@ -248,12 +248,18 @@ static inline const struct cpumask *cpu_cpu_mask(int cpu) #ifdef CONFIG_NUMA int sched_numa_find_nth_cpu(const struct cpumask *cpus, int cpu, int node); extern const struct cpumask *sched_numa_hop_mask(unsigned int node, unsigned int hops); -#else +extern int sched_numa_hop_node(nodemask_t *hop_nodes, int start, unsigned int state); +#else /* !CONFIG_NUMA */ static __always_inline int sched_numa_find_nth_cpu(const struct cpumask *cpus, int cpu, int node) { return cpumask_nth_and(cpu, cpus, cpu_online_mask); } +static inline int sched_numa_hop_node(nodemask_t *hop_nodes, int start, unsigned int state) +{ + return NUMA_NO_NODE; +} + static inline const struct cpumask * sched_numa_hop_mask(unsigned int node, unsigned int hops) { @@ -261,6 +267,26 @@ sched_numa_hop_mask(unsigned int node, unsigned int hops) } #endif /* CONFIG_NUMA */ +/** + * for_each_numa_hop_node - iterate over NUMA nodes at increasing hop distances + * from a given starting node. + * @__node: the iteration variable, representing the current NUMA node. + * @__start: the NUMA node to start the iteration from. + * @__hop_nodes: a nodemask_t to track the visited nodes. + * @__state: state of NUMA nodes to iterate. + * + * Requires rcu_lock to be held. + * + * This macro iterates over NUMA nodes in increasing distance from + * @start_node. + * + * Yields NUMA_NO_NODE when all the nodes have been visited. + */ +#define for_each_numa_hop_node(__node, __start, __hop_nodes, __state) \ + for (int __node = __start; \ + __node != NUMA_NO_NODE; \ + __node = sched_numa_hop_node(&(__hop_nodes), __start, __state)) + /** * for_each_numa_hop_mask - iterate over cpumasks of increasing NUMA distance * from a given node. diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c index 9748a4c8d668..8e77c235ad9a 100644 --- a/kernel/sched/topology.c +++ b/kernel/sched/topology.c @@ -2185,6 +2185,55 @@ int sched_numa_find_nth_cpu(const struct cpumask *cpus, int cpu, int node) } EXPORT_SYMBOL_GPL(sched_numa_find_nth_cpu); +/** + * sched_numa_hop_node - Find the NUMA node at the closest hop distance + * from node @start. + * + * @hop_nodes: a pointer to a nodemask_t representing the visited nodes. + * @start: the NUMA node to start the hop search from. + * @state: the node state to filter nodes by. + * + * This function iterates over all NUMA nodes in the given state and + * calculates the hop distance to the starting node. It returns the NUMA + * node that is the closest in terms of hop distance + * that has not already been considered (not set in @hop_nodes). If the + * node is found, it is marked as considered in the @hop_nodes bitmask. + * + * The function checks if the node is not the start node and ensures it is + * not already part of the hop_nodes set. It then computes the distance to + * the start node using the node_distance() function. The closest node is + * chosen based on the minimum distance. + * + * Returns the NUMA node ID closest in terms of hop distance from the + * @start node, or NUMA_NO_NODE if no node is found (or all nodes have been + * visited). + */ +int sched_numa_hop_node(nodemask_t *hop_nodes, int start, unsigned int state) +{ + int dist, n, min_node, min_dist; + + if (state >= NR_NODE_STATES) + return NUMA_NO_NODE; + + min_node = NUMA_NO_NODE; + min_dist = INT_MAX; + + for_each_node_state(n, state) { + if (n == start || node_isset(n, *hop_nodes)) + continue; + dist = node_distance(start, n); + if (dist < min_dist) { + min_dist = dist; + min_node = n; + } + } + if (min_node != NUMA_NO_NODE) + node_set(min_node, *hop_nodes); + + return min_node; +} +EXPORT_SYMBOL_GPL(sched_numa_hop_node); + /** * sched_numa_hop_mask() - Get the cpumask of CPUs at most @hops hops away from * @node From patchwork Fri Dec 20 15:11:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrea Righi X-Patchwork-Id: 13916899 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2042.outbound.protection.outlook.com [40.107.94.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 77719219A83; Fri, 20 Dec 2024 15:41:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.94.42 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734709293; cv=fail; b=s8F35+JFe1+4EMYohJgkpMuJ+mQx1IrgAHaWnzeKkmQphsl2SbuoWKuZsRgQCXp5fxHMByF5vkpMZKOkfwTd+7PLyLHKErh+sJG+pUvzzsZTgYi0xD0S8cKFcyILhv7rCc/gEDrjBBaAVhWKHVX9fQ2Q8DimRBJPYRPOEfjwBw0= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734709293; c=relaxed/simple; bh=KlcYFBZqkx4sHZafnuBP7ghcKek87oc/S/koYFf9jIs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=pyoxiliR4dP5ri1qA8XGFXayK+JdFUK6bz/wQZZvxHxI6HDXx80JPK/7bZaC9Z2rI0yNec0BlcjHRqkZgGCgtInMQCvhn5DR8mhrDMsW1G/HLXfCeI4Tm0K/iKT+O0iyXjK2UtE/qh0ug9I6VnvEAX+1IjIBSljIZRceYTSwYMg= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=Yhkfd7K4; arc=fail smtp.client-ip=40.107.94.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="Yhkfd7K4" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=NFZpHuCNW3gnDDzR2rzr8IFzlQl5MqEm6SNVPvHY3bSrctqXF7utS7YMFAPvwB5Hiy4y9TDzqrRVqYhD0CkF6r9Kp4hztylKXbsfb6sPz0cxgjhkzCMEpfbu9cjJosgMfuEfSUbqmD7tsvH07W/zalAFPhDavMFzCPRAXN1pk2x+55BVJ1Ol8+lzLwzBc+2wHp549oUGQ5WO+8Pi2V813PsJuab8kbNczvM+jgd5wmExe2YDZRXPSsgauKCgV0zyEbHiddfPfQwdiiHSaVLk4QuqH+lOWPvB9VWs0xMoWPy2loX1k9st0VI9Pb6oBRb6zSdwyrzlx43EvYvtbBuy/g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=f9uSInKu/38EFQfNXL0CQk4Ggbv9bR33SfOxkDA4frM=; b=JtcuyhHlzX/gb8a/eT7OcFrEUBpDBiqI5XCI4hjIqsvgdipbHfEypLQlxfLBWIOdjwfHpslXfdkq0u+ITutDHFvhMNKO1CRtCt9JwM1SrN9nfb3IvQ5AznDQ96Gz+YTsb2azu3aaszgZE6TLzwxxoyE1zWPglqhXSPUkD9wkmY0GG6BDJR6MnYsiXMPHf6RHrbFmsqI/P1CRtTlHMfXYmolMnAcfQYD58IimaKSOWy1aKYCUsAZ2zJUgQ8ScEdjQ4hZ++U28tLwtupuMlgNseZnzJ8dKBrbbcI2Xv9DSvMUGUpR266imKfsQToTPHSWgB53STX9Zm9F7tqtjMtBTyA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=f9uSInKu/38EFQfNXL0CQk4Ggbv9bR33SfOxkDA4frM=; b=Yhkfd7K4Zha50OxrOrnGuH1mFdjP8cjbHOZtOlyclfrXauVl+Mzw4NwLa6yhvPsIzyuoQqRtVO+Qtlw6y3auh4DqIb0I0pHUzmPpUYUxogLEDNVQ5gQTpDqyXgPQ5BfiOX+AzjO4k9gqqUVrFk954BS6J9IXyeGadIpab2LGOF9EKIOTH0diB+B583e+3gomleUpMClytbkfUwist+/3g3tvFPz2BpLuofQrPA6z3aaSz4x477iH/bkb1kgBI1FD7dFegG9rRTkGW7jreCU3+6pYrS+vgndnbjbp7gFdix9whBvDbuIhlum0DSav/2uPfvhEJ5NDPcU/TljZdwxchw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) by MW4PR12MB7238.namprd12.prod.outlook.com (2603:10b6:303:229::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8272.17; Fri, 20 Dec 2024 15:41:24 +0000 Received: from CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5]) by CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5%3]) with mapi id 15.20.8272.005; Fri, 20 Dec 2024 15:41:24 +0000 From: Andrea Righi To: Tejun Heo , David Vernet , Changwoo Min Cc: Yury Norov , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , bpf@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 02/10] sched_ext: Move built-in idle CPU selection policy to a separate file Date: Fri, 20 Dec 2024 16:11:34 +0100 Message-ID: <20241220154107.287478-3-arighi@nvidia.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241220154107.287478-1-arighi@nvidia.com> References: <20241220154107.287478-1-arighi@nvidia.com> X-ClientProxiedBy: FR4P281CA0280.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:e6::16) To CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY5PR12MB6405:EE_|MW4PR12MB7238:EE_ X-MS-Office365-Filtering-Correlation-Id: db35f5ef-2f77-44a8-95e7-08dd210cc263 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|376014|366016|1800799024|7053199007; X-Microsoft-Antispam-Message-Info: TWElcB4F5U4aXJOFttBy/uk4yF8vvSgOa1Kak6lhH4RZxRBVQPYW0HgAfQSq6tSzPSvmqSn4EekflXZ4KXRLMTuMv6OAy28SYRNQq3blbnDRviMAYrwTuUsJGwFM+v+DMSMNhg0HSiR9VdNSdJQnja5gWadpBh+WnvH+T/BoMFDCVmDqbd7Xf8nP6tBP46i1Z8bkgyi6IwXiYqiLb6LgkAQdCYkru/yc0wP3xozVfwkBF6uyhkdviETeSNAiLqkUXrIgxVYshNxNHIGfex47YWkaHZDTUvlIlwFSoQzbyxnoYGatohZCakduuiWN3W/3Ug+KtMecaXkFhVI2WpxIlPtKdI/V8tPJxJ0CFOzddSTo0N68CLso9oIV10UpL+imrRQl1nPFxWst48TTxMmknuYoLT5rChKH3W+6W4Rfwd3QRUkTKxQ/WRbbp3aoAv+hTmhInts3bUhSdaQaLvJg68rtt1C7Ykf9ivoSnj7fwZvkWYjWyC5/2cOlBbh+9M22hC/4sRlrE7V+e2WsC/xvjmT/eajZgr7u6TJvcdxWiWvVlSLjgssVlV9uLWUMf8l+acEY03zFPAuhmZfi80djMN8rL4ujOe1rNLV4c6tyL0aM99/dcir7Be0YC0QXr6963I4xe7v/SVhD//hmDhARoeFp7J0/QKYsu9g9hH7wfLSmNFDCw7TsGQDN5SDqUy7Cx/Aflr4vB9icf8Qa3dizPPdesoylwUJC/gTBDhsOp8oy4la/G8H1jEvV5brKEgO+ryU1aaqn1sRm8fw5/tbh0pSpOVrQquF7QT5tjjIDKuvL3gBDK9IKh/5etUz8FEC0LT21K2v5ZvMf4OpLTjHua/kp5OA6Wtr8zcG1pQAaXJ0lQg1ZsGF/aLZcWGJrKGHvX8zH+Jea+EQrO1e5YcG6Kp8KxHV5qDk7XQuBEK61jZ+0tRnfaQhc6jLY3NCK/XtrxcsSimbsV7wNMWuwqljWoFo5hC/RJQv/LasbM00Af+XUn612YcKFaNwUn8ARKjvGYPx10bkYiZfNSQYkU8x6oR5+zoyINq8G+Lu4m/Dyj221XpgamPXMHuKWq97NpDIdPlCRwcI75issKP+qe4PQuS0vbVbWUuCdej+kRoebf44kjE13QLOBzYbIKEuz/9bccrIN/xpwzKLuKi1SA6eGpzS6ZqcQPLRBIk91IP5Hr4Z7cCjsVw2vcO6FSkOFFL4wRcwRMstd74nQGvdjO1c5Xsq2dQnxZTUYejArr8nmQuwuX/MnHkE/qYF4o3+v5QYs4kxNCsdQNUpPhRomotkwK1HuwTFvQAugHGbIMR5daZOhFm/unOuwjBw8Z8tQEYGtKRiGRf8RYou4VTOvIrAKRBgl492+3xqlcdCSqUdyGE4Sg+8c8X4QOA9xFjDktehr X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CY5PR12MB6405.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(7416014)(376014)(366016)(1800799024)(7053199007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: xZG9NRFnx4HRZBeEEB680eVaQemBztJUGymiwY2AS83iBAS7YM31j/vSAjoFLwhHhWsJ36N1FVIrz+VrtLdfkDZlnQT3FArKkcmyD9UcViEbs2UIx0XiPFq8OazPn4ot0tCegHBW6JwehIS9/AfXCzP+PG8h1vlAb75YjJ49oGV7SreWKvShrqcP144+s1BsM46wr02Oax9qVquivyP1OeODx+OGyLMILXfvZ06L4Z/6/YbFOPEXW26saxuhQq2QSZ/RbzMOqLStq41bz4F6w4T0vIeDRJTpFWC5Z5xIxUnU3D7xXL+AHbBXXkDF2+9LZpyGuHeCEnzAOCmt6W7B/ZcS8uDO+pO7Fqh27+lswQ3SDmsQTeRVZpsoEUJuzwuJmgMQCY+3HoibGzzMo3Nor3jgK9hDuURscRPMDusFXKXqJUgKnK0LHAyGrtPaGeYBwE4tJVCjsdnk6YRvLKfMIznLhdmWwFlwkw/EEg9RcUWynpYV75pc9FshyT7n42FNIklIfyUhwReNnuW0c9OjjKk00IdYHuOGyhK5N4EXn3QBaeHov2UctZbqejBSRqG3y4q6aEu7kW9GYVgAN8xIsX/Ip9lZRz0I2WUxraQfc5ocdkHIc4GMxGCuxZgegr7p0cWcCgMvJlCW7Tyvg4Sj8UghJ3aCFwE3SdeaetcnJzOTTQGzgwOEWp0SKEnIV28sWXy3/uLh5Kftkt3AoSRWUUwFLHEWjbUgHfm4vMiRdefsl8qRxYH8vEdE8oRysbNFg3SEigv09qXta3zOTxGoqmmouA5VQJuFH9yAxmeIG17VVlBVLA69r8jixpmJ9SXGx0ymmIRHyFvOI1hJfr7qodipcvRPcgI5j10n16Jiweri+fCKdRotGVW4+mdJix+Qa7Ht88wXfJ73VwBJ5WmD8UtvX7ubSmvxFRJ6GOIH5VKneVxwhZMlPUMnImE8pNXcopkPQM9IrgdQUKo7JqbB3WTkodopdKwCjUHg6LFe1D0mMZvGUe9bR2N3VsOdtyePqwFSQEz+YtxlTs+49XIFvvrSNV3y1Tk2CXr2qmTjp4/RgrfEx18aj8esfoCzKAtod0xvMAkxLBKrNqe4qO81dgozFJ8p7O3xXXvVe65s9/RQ3AbjCabQBdSyZttqN3og+tgYlEXoQq1VLOm4hvCiE0qO2R3WxFzSAMDBDLFGEMRywDZYzOIH1a5iGJeGUcNQ7+jX5JXnFp5mgnumg7AQwE1c0Jru21Kyd+v/XMoRmRod344jt+d8yiT4EqGcxGVlk17NYujsZa4+F6X2beuOhL2gigBH87sNPkeWPfNa2dSs4kR4Xnv4wqhCLUrr2q3qkJznp1DDd9iG9srV6nCoSp2PDmdVmZGCrnN5i/9QW1tBb/Svw4PEXz3eliRCZFFrh0ISCcDpJXghtzuPNcZd+Twe1Echc47Gt4mD6YpnIsRliJ9ksmaS8+bD2vUI8b6NdRpHukYbsVMlx5z4fa6HJjv/dcWLUVFzcPnwqPt5fuiFUDUS0VXGta65sjzupiShzZd/wVHo+j/Vwp8cg7xIUn3eHiMwBEE+XN8GZfZEBImEKQdA/85PfMUwhSmpESpb X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: db35f5ef-2f77-44a8-95e7-08dd210cc263 X-MS-Exchange-CrossTenant-AuthSource: CY5PR12MB6405.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Dec 2024 15:41:24.3237 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: D4x3uBWYEKl+OMh5t9S2Rh0x0wdD0c9HtyIp3Fm9v5YfPfDdU2PqUHF2d865QHgrpMFoAEpu0ZR2PmpjhwRiNg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR12MB7238 As ext.c is becoming quite large, move the idle CPU selection policy to a separate file (ext_idle.c) for better code readability. Also group together all the idle CPU selection kfunc's to the same btf_kfunc_id_set block. No functional change, this is purely code reorganization. Suggested-by: Yury Norov Signed-off-by: Andrea Righi --- MAINTAINERS | 1 + kernel/sched/ext.c | 702 +--------------------------------------- kernel/sched/ext_idle.c | 686 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 703 insertions(+), 686 deletions(-) create mode 100644 kernel/sched/ext_idle.c diff --git a/MAINTAINERS b/MAINTAINERS index 1e930c7a58b1..02960d1b9ee9 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -20909,6 +20909,7 @@ T: git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext.git F: include/linux/sched/ext.h F: kernel/sched/ext.h F: kernel/sched/ext.c +F: kernel/sched/ext_idle.c F: tools/sched_ext/ F: tools/testing/selftests/sched_ext diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c index 54e659ba9476..769e43fdea1e 100644 --- a/kernel/sched/ext.c +++ b/kernel/sched/ext.c @@ -6,6 +6,8 @@ * Copyright (c) 2022 Tejun Heo * Copyright (c) 2022 David Vernet */ +#include + #define SCX_OP_IDX(op) (offsetof(struct sched_ext_ops, op) / sizeof(void (*)(void))) enum scx_consts { @@ -882,12 +884,6 @@ static bool scx_warned_zero_slice; static DEFINE_STATIC_KEY_FALSE(scx_ops_enq_last); static DEFINE_STATIC_KEY_FALSE(scx_ops_enq_exiting); static DEFINE_STATIC_KEY_FALSE(scx_ops_cpu_preempt); -static DEFINE_STATIC_KEY_FALSE(scx_builtin_idle_enabled); - -#ifdef CONFIG_SMP -static DEFINE_STATIC_KEY_FALSE(scx_selcpu_topo_llc); -static DEFINE_STATIC_KEY_FALSE(scx_selcpu_topo_numa); -#endif static struct static_key_false scx_has_op[SCX_OPI_END] = { [0 ... SCX_OPI_END-1] = STATIC_KEY_FALSE_INIT }; @@ -895,6 +891,17 @@ static struct static_key_false scx_has_op[SCX_OPI_END] = static atomic_t scx_exit_kind = ATOMIC_INIT(SCX_EXIT_DONE); static struct scx_exit_info *scx_exit_info; +#define scx_ops_error_kind(err, fmt, args...) \ + scx_ops_exit_kind((err), 0, fmt, ##args) + +#define scx_ops_exit(code, fmt, args...) \ + scx_ops_exit_kind(SCX_EXIT_UNREG_KERN, (code), fmt, ##args) + +#define scx_ops_error(fmt, args...) \ + scx_ops_error_kind(SCX_EXIT_ERROR, fmt, ##args) + +#define SCX_HAS_OP(op) static_branch_likely(&scx_has_op[SCX_OP_IDX(op)]) + static atomic_long_t scx_nr_rejected = ATOMIC_LONG_INIT(0); static atomic_long_t scx_hotplug_seq = ATOMIC_LONG_INIT(0); @@ -922,21 +929,6 @@ static unsigned long scx_watchdog_timestamp = INITIAL_JIFFIES; static struct delayed_work scx_watchdog_work; -/* idle tracking */ -#ifdef CONFIG_SMP -#ifdef CONFIG_CPUMASK_OFFSTACK -#define CL_ALIGNED_IF_ONSTACK -#else -#define CL_ALIGNED_IF_ONSTACK __cacheline_aligned_in_smp -#endif - -static struct { - cpumask_var_t cpu; - cpumask_var_t smt; -} idle_masks CL_ALIGNED_IF_ONSTACK; - -#endif /* CONFIG_SMP */ - /* for %SCX_KICK_WAIT */ static unsigned long __percpu *scx_kick_cpus_pnt_seqs; @@ -1023,17 +1015,6 @@ static __printf(3, 4) void scx_ops_exit_kind(enum scx_exit_kind kind, s64 exit_code, const char *fmt, ...); -#define scx_ops_error_kind(err, fmt, args...) \ - scx_ops_exit_kind((err), 0, fmt, ##args) - -#define scx_ops_exit(code, fmt, args...) \ - scx_ops_exit_kind(SCX_EXIT_UNREG_KERN, (code), fmt, ##args) - -#define scx_ops_error(fmt, args...) \ - scx_ops_error_kind(SCX_EXIT_ERROR, fmt, ##args) - -#define SCX_HAS_OP(op) static_branch_likely(&scx_has_op[SCX_OP_IDX(op)]) - static long jiffies_delta_msecs(unsigned long at, unsigned long now) { if (time_after(at, now)) @@ -1540,6 +1521,9 @@ static int ops_sanitize_err(const char *ops_name, s32 err) return -EPROTO; } +/* Built-in idle CPU selection policy */ +#include "ext_idle.c" + static void run_deferred(struct rq *rq) { process_ddsp_deferred_locals(rq); @@ -3164,410 +3148,6 @@ bool scx_prio_less(const struct task_struct *a, const struct task_struct *b, #ifdef CONFIG_SMP -static bool test_and_clear_cpu_idle(int cpu) -{ -#ifdef CONFIG_SCHED_SMT - /* - * SMT mask should be cleared whether we can claim @cpu or not. The SMT - * cluster is not wholly idle either way. This also prevents - * scx_pick_idle_cpu() from getting caught in an infinite loop. - */ - if (sched_smt_active()) { - const struct cpumask *smt = cpu_smt_mask(cpu); - - /* - * If offline, @cpu is not its own sibling and - * scx_pick_idle_cpu() can get caught in an infinite loop as - * @cpu is never cleared from idle_masks.smt. Ensure that @cpu - * is eventually cleared. - */ - if (cpumask_intersects(smt, idle_masks.smt)) - cpumask_andnot(idle_masks.smt, idle_masks.smt, smt); - else if (cpumask_test_cpu(cpu, idle_masks.smt)) - __cpumask_clear_cpu(cpu, idle_masks.smt); - } -#endif - return cpumask_test_and_clear_cpu(cpu, idle_masks.cpu); -} - -static s32 scx_pick_idle_cpu(const struct cpumask *cpus_allowed, u64 flags) -{ - int cpu; - -retry: - if (sched_smt_active()) { - cpu = cpumask_any_and_distribute(idle_masks.smt, cpus_allowed); - if (cpu < nr_cpu_ids) - goto found; - - if (flags & SCX_PICK_IDLE_CORE) - return -EBUSY; - } - - cpu = cpumask_any_and_distribute(idle_masks.cpu, cpus_allowed); - if (cpu >= nr_cpu_ids) - return -EBUSY; - -found: - if (test_and_clear_cpu_idle(cpu)) - return cpu; - else - goto retry; -} - -/* - * Return the amount of CPUs in the same LLC domain of @cpu (or zero if the LLC - * domain is not defined). - */ -static unsigned int llc_weight(s32 cpu) -{ - struct sched_domain *sd; - - sd = rcu_dereference(per_cpu(sd_llc, cpu)); - if (!sd) - return 0; - - return sd->span_weight; -} - -/* - * Return the cpumask representing the LLC domain of @cpu (or NULL if the LLC - * domain is not defined). - */ -static struct cpumask *llc_span(s32 cpu) -{ - struct sched_domain *sd; - - sd = rcu_dereference(per_cpu(sd_llc, cpu)); - if (!sd) - return 0; - - return sched_domain_span(sd); -} - -/* - * Return the amount of CPUs in the same NUMA domain of @cpu (or zero if the - * NUMA domain is not defined). - */ -static unsigned int numa_weight(s32 cpu) -{ - struct sched_domain *sd; - struct sched_group *sg; - - sd = rcu_dereference(per_cpu(sd_numa, cpu)); - if (!sd) - return 0; - sg = sd->groups; - if (!sg) - return 0; - - return sg->group_weight; -} - -/* - * Return the cpumask representing the NUMA domain of @cpu (or NULL if the NUMA - * domain is not defined). - */ -static struct cpumask *numa_span(s32 cpu) -{ - struct sched_domain *sd; - struct sched_group *sg; - - sd = rcu_dereference(per_cpu(sd_numa, cpu)); - if (!sd) - return NULL; - sg = sd->groups; - if (!sg) - return NULL; - - return sched_group_span(sg); -} - -/* - * Return true if the LLC domains do not perfectly overlap with the NUMA - * domains, false otherwise. - */ -static bool llc_numa_mismatch(void) -{ - int cpu; - - /* - * We need to scan all online CPUs to verify whether their scheduling - * domains overlap. - * - * While it is rare to encounter architectures with asymmetric NUMA - * topologies, CPU hotplugging or virtualized environments can result - * in asymmetric configurations. - * - * For example: - * - * NUMA 0: - * - LLC 0: cpu0..cpu7 - * - LLC 1: cpu8..cpu15 [offline] - * - * NUMA 1: - * - LLC 0: cpu16..cpu23 - * - LLC 1: cpu24..cpu31 - * - * In this case, if we only check the first online CPU (cpu0), we might - * incorrectly assume that the LLC and NUMA domains are fully - * overlapping, which is incorrect (as NUMA 1 has two distinct LLC - * domains). - */ - for_each_online_cpu(cpu) - if (llc_weight(cpu) != numa_weight(cpu)) - return true; - - return false; -} - -/* - * Initialize topology-aware scheduling. - * - * Detect if the system has multiple LLC or multiple NUMA domains and enable - * cache-aware / NUMA-aware scheduling optimizations in the default CPU idle - * selection policy. - * - * Assumption: the kernel's internal topology representation assumes that each - * CPU belongs to a single LLC domain, and that each LLC domain is entirely - * contained within a single NUMA node. - */ -static void update_selcpu_topology(void) -{ - bool enable_llc = false, enable_numa = false; - unsigned int nr_cpus; - s32 cpu = cpumask_first(cpu_online_mask); - - /* - * Enable LLC domain optimization only when there are multiple LLC - * domains among the online CPUs. If all online CPUs are part of a - * single LLC domain, the idle CPU selection logic can choose any - * online CPU without bias. - * - * Note that it is sufficient to check the LLC domain of the first - * online CPU to determine whether a single LLC domain includes all - * CPUs. - */ - rcu_read_lock(); - nr_cpus = llc_weight(cpu); - if (nr_cpus > 0) { - if (nr_cpus < num_online_cpus()) - enable_llc = true; - pr_debug("sched_ext: LLC=%*pb weight=%u\n", - cpumask_pr_args(llc_span(cpu)), llc_weight(cpu)); - } - - /* - * Enable NUMA optimization only when there are multiple NUMA domains - * among the online CPUs and the NUMA domains don't perfectly overlaps - * with the LLC domains. - * - * If all CPUs belong to the same NUMA node and the same LLC domain, - * enabling both NUMA and LLC optimizations is unnecessary, as checking - * for an idle CPU in the same domain twice is redundant. - */ - nr_cpus = numa_weight(cpu); - if (nr_cpus > 0) { - if (nr_cpus < num_online_cpus() && llc_numa_mismatch()) - enable_numa = true; - pr_debug("sched_ext: NUMA=%*pb weight=%u\n", - cpumask_pr_args(numa_span(cpu)), numa_weight(cpu)); - } - rcu_read_unlock(); - - pr_debug("sched_ext: LLC idle selection %s\n", - enable_llc ? "enabled" : "disabled"); - pr_debug("sched_ext: NUMA idle selection %s\n", - enable_numa ? "enabled" : "disabled"); - - if (enable_llc) - static_branch_enable_cpuslocked(&scx_selcpu_topo_llc); - else - static_branch_disable_cpuslocked(&scx_selcpu_topo_llc); - if (enable_numa) - static_branch_enable_cpuslocked(&scx_selcpu_topo_numa); - else - static_branch_disable_cpuslocked(&scx_selcpu_topo_numa); -} - -/* - * Built-in CPU idle selection policy: - * - * 1. Prioritize full-idle cores: - * - always prioritize CPUs from fully idle cores (both logical CPUs are - * idle) to avoid interference caused by SMT. - * - * 2. Reuse the same CPU: - * - prefer the last used CPU to take advantage of cached data (L1, L2) and - * branch prediction optimizations. - * - * 3. Pick a CPU within the same LLC (Last-Level Cache): - * - if the above conditions aren't met, pick a CPU that shares the same LLC - * to maintain cache locality. - * - * 4. Pick a CPU within the same NUMA node, if enabled: - * - choose a CPU from the same NUMA node to reduce memory access latency. - * - * Step 3 and 4 are performed only if the system has, respectively, multiple - * LLC domains / multiple NUMA nodes (see scx_selcpu_topo_llc and - * scx_selcpu_topo_numa). - * - * NOTE: tasks that can only run on 1 CPU are excluded by this logic, because - * we never call ops.select_cpu() for them, see select_task_rq(). - */ -static s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, - u64 wake_flags, bool *found) -{ - const struct cpumask *llc_cpus = NULL; - const struct cpumask *numa_cpus = NULL; - s32 cpu; - - *found = false; - - /* - * This is necessary to protect llc_cpus. - */ - rcu_read_lock(); - - /* - * Determine the scheduling domain only if the task is allowed to run - * on all CPUs. - * - * This is done primarily for efficiency, as it avoids the overhead of - * updating a cpumask every time we need to select an idle CPU (which - * can be costly in large SMP systems), but it also aligns logically: - * if a task's scheduling domain is restricted by user-space (through - * CPU affinity), the task will simply use the flat scheduling domain - * defined by user-space. - */ - if (p->nr_cpus_allowed >= num_possible_cpus()) { - if (static_branch_maybe(CONFIG_NUMA, &scx_selcpu_topo_numa)) - numa_cpus = numa_span(prev_cpu); - - if (static_branch_maybe(CONFIG_SCHED_MC, &scx_selcpu_topo_llc)) - llc_cpus = llc_span(prev_cpu); - } - - /* - * If WAKE_SYNC, try to migrate the wakee to the waker's CPU. - */ - if (wake_flags & SCX_WAKE_SYNC) { - cpu = smp_processor_id(); - - /* - * If the waker's CPU is cache affine and prev_cpu is idle, - * then avoid a migration. - */ - if (cpus_share_cache(cpu, prev_cpu) && - test_and_clear_cpu_idle(prev_cpu)) { - cpu = prev_cpu; - goto cpu_found; - } - - /* - * If the waker's local DSQ is empty, and the system is under - * utilized, try to wake up @p to the local DSQ of the waker. - * - * Checking only for an empty local DSQ is insufficient as it - * could give the wakee an unfair advantage when the system is - * oversaturated. - * - * Checking only for the presence of idle CPUs is also - * insufficient as the local DSQ of the waker could have tasks - * piled up on it even if there is an idle core elsewhere on - * the system. - */ - if (!cpumask_empty(idle_masks.cpu) && - !(current->flags & PF_EXITING) && - cpu_rq(cpu)->scx.local_dsq.nr == 0) { - if (cpumask_test_cpu(cpu, p->cpus_ptr)) - goto cpu_found; - } - } - - /* - * If CPU has SMT, any wholly idle CPU is likely a better pick than - * partially idle @prev_cpu. - */ - if (sched_smt_active()) { - /* - * Keep using @prev_cpu if it's part of a fully idle core. - */ - if (cpumask_test_cpu(prev_cpu, idle_masks.smt) && - test_and_clear_cpu_idle(prev_cpu)) { - cpu = prev_cpu; - goto cpu_found; - } - - /* - * Search for any fully idle core in the same LLC domain. - */ - if (llc_cpus) { - cpu = scx_pick_idle_cpu(llc_cpus, SCX_PICK_IDLE_CORE); - if (cpu >= 0) - goto cpu_found; - } - - /* - * Search for any fully idle core in the same NUMA node. - */ - if (numa_cpus) { - cpu = scx_pick_idle_cpu(numa_cpus, SCX_PICK_IDLE_CORE); - if (cpu >= 0) - goto cpu_found; - } - - /* - * Search for any full idle core usable by the task. - */ - cpu = scx_pick_idle_cpu(p->cpus_ptr, SCX_PICK_IDLE_CORE); - if (cpu >= 0) - goto cpu_found; - } - - /* - * Use @prev_cpu if it's idle. - */ - if (test_and_clear_cpu_idle(prev_cpu)) { - cpu = prev_cpu; - goto cpu_found; - } - - /* - * Search for any idle CPU in the same LLC domain. - */ - if (llc_cpus) { - cpu = scx_pick_idle_cpu(llc_cpus, 0); - if (cpu >= 0) - goto cpu_found; - } - - /* - * Search for any idle CPU in the same NUMA node. - */ - if (numa_cpus) { - cpu = scx_pick_idle_cpu(numa_cpus, 0); - if (cpu >= 0) - goto cpu_found; - } - - /* - * Search for any idle CPU usable by the task. - */ - cpu = scx_pick_idle_cpu(p->cpus_ptr, 0); - if (cpu >= 0) - goto cpu_found; - - rcu_read_unlock(); - return prev_cpu; - -cpu_found: - rcu_read_unlock(); - - *found = true; - return cpu; -} - static int select_task_rq_scx(struct task_struct *p, int prev_cpu, int wake_flags) { /* @@ -3634,52 +3214,6 @@ static void set_cpus_allowed_scx(struct task_struct *p, (struct cpumask *)p->cpus_ptr); } -static void reset_idle_masks(void) -{ - /* - * Consider all online cpus idle. Should converge to the actual state - * quickly. - */ - cpumask_copy(idle_masks.cpu, cpu_online_mask); - cpumask_copy(idle_masks.smt, cpu_online_mask); -} - -void __scx_update_idle(struct rq *rq, bool idle) -{ - int cpu = cpu_of(rq); - - if (SCX_HAS_OP(update_idle) && !scx_rq_bypassing(rq)) { - SCX_CALL_OP(SCX_KF_REST, update_idle, cpu_of(rq), idle); - if (!static_branch_unlikely(&scx_builtin_idle_enabled)) - return; - } - - if (idle) - cpumask_set_cpu(cpu, idle_masks.cpu); - else - cpumask_clear_cpu(cpu, idle_masks.cpu); - -#ifdef CONFIG_SCHED_SMT - if (sched_smt_active()) { - const struct cpumask *smt = cpu_smt_mask(cpu); - - if (idle) { - /* - * idle_masks.smt handling is racy but that's fine as - * it's only for optimization and self-correcting. - */ - for_each_cpu(cpu, smt) { - if (!cpumask_test_cpu(cpu, idle_masks.cpu)) - return; - } - cpumask_or(idle_masks.smt, idle_masks.smt, smt); - } else { - cpumask_andnot(idle_masks.smt, idle_masks.smt, smt); - } - } -#endif -} - static void handle_hotplug(struct rq *rq, bool online) { int cpu = cpu_of(rq); @@ -3719,12 +3253,6 @@ static void rq_offline_scx(struct rq *rq) rq->scx.flags &= ~SCX_RQ_ONLINE; } -#else /* CONFIG_SMP */ - -static bool test_and_clear_cpu_idle(int cpu) { return false; } -static s32 scx_pick_idle_cpu(const struct cpumask *cpus_allowed, u64 flags) { return -EBUSY; } -static void reset_idle_masks(void) {} - #endif /* CONFIG_SMP */ static bool check_rq_for_timeouts(struct rq *rq) @@ -6290,55 +5818,6 @@ void __init init_sched_ext_class(void) /******************************************************************************** * Helpers that can be called from the BPF scheduler. */ -#include - -__bpf_kfunc_start_defs(); - -/** - * scx_bpf_select_cpu_dfl - The default implementation of ops.select_cpu() - * @p: task_struct to select a CPU for - * @prev_cpu: CPU @p was on previously - * @wake_flags: %SCX_WAKE_* flags - * @is_idle: out parameter indicating whether the returned CPU is idle - * - * Can only be called from ops.select_cpu() if the built-in CPU selection is - * enabled - ops.update_idle() is missing or %SCX_OPS_KEEP_BUILTIN_IDLE is set. - * @p, @prev_cpu and @wake_flags match ops.select_cpu(). - * - * Returns the picked CPU with *@is_idle indicating whether the picked CPU is - * currently idle and thus a good candidate for direct dispatching. - */ -__bpf_kfunc s32 scx_bpf_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, - u64 wake_flags, bool *is_idle) -{ - if (!static_branch_likely(&scx_builtin_idle_enabled)) { - scx_ops_error("built-in idle tracking is disabled"); - goto prev_cpu; - } - - if (!scx_kf_allowed(SCX_KF_SELECT_CPU)) - goto prev_cpu; - -#ifdef CONFIG_SMP - return scx_select_cpu_dfl(p, prev_cpu, wake_flags, is_idle); -#endif - -prev_cpu: - *is_idle = false; - return prev_cpu; -} - -__bpf_kfunc_end_defs(); - -BTF_KFUNCS_START(scx_kfunc_ids_select_cpu) -BTF_ID_FLAGS(func, scx_bpf_select_cpu_dfl, KF_RCU) -BTF_KFUNCS_END(scx_kfunc_ids_select_cpu) - -static const struct btf_kfunc_id_set scx_kfunc_set_select_cpu = { - .owner = THIS_MODULE, - .set = &scx_kfunc_ids_select_cpu, -}; - static bool scx_dsq_insert_preamble(struct task_struct *p, u64 enq_flags) { if (!scx_kf_allowed(SCX_KF_ENQUEUE | SCX_KF_DISPATCH)) @@ -7400,149 +6879,6 @@ __bpf_kfunc void scx_bpf_put_cpumask(const struct cpumask *cpumask) */ } -/** - * scx_bpf_get_idle_cpumask - Get a referenced kptr to the idle-tracking - * per-CPU cpumask. - * - * Returns NULL if idle tracking is not enabled, or running on a UP kernel. - */ -__bpf_kfunc const struct cpumask *scx_bpf_get_idle_cpumask(void) -{ - if (!static_branch_likely(&scx_builtin_idle_enabled)) { - scx_ops_error("built-in idle tracking is disabled"); - return cpu_none_mask; - } - -#ifdef CONFIG_SMP - return idle_masks.cpu; -#else - return cpu_none_mask; -#endif -} - -/** - * scx_bpf_get_idle_smtmask - Get a referenced kptr to the idle-tracking, - * per-physical-core cpumask. Can be used to determine if an entire physical - * core is free. - * - * Returns NULL if idle tracking is not enabled, or running on a UP kernel. - */ -__bpf_kfunc const struct cpumask *scx_bpf_get_idle_smtmask(void) -{ - if (!static_branch_likely(&scx_builtin_idle_enabled)) { - scx_ops_error("built-in idle tracking is disabled"); - return cpu_none_mask; - } - -#ifdef CONFIG_SMP - if (sched_smt_active()) - return idle_masks.smt; - else - return idle_masks.cpu; -#else - return cpu_none_mask; -#endif -} - -/** - * scx_bpf_put_idle_cpumask - Release a previously acquired referenced kptr to - * either the percpu, or SMT idle-tracking cpumask. - */ -__bpf_kfunc void scx_bpf_put_idle_cpumask(const struct cpumask *idle_mask) -{ - /* - * Empty function body because we aren't actually acquiring or releasing - * a reference to a global idle cpumask, which is read-only in the - * caller and is never released. The acquire / release semantics here - * are just used to make the cpumask a trusted pointer in the caller. - */ -} - -/** - * scx_bpf_test_and_clear_cpu_idle - Test and clear @cpu's idle state - * @cpu: cpu to test and clear idle for - * - * Returns %true if @cpu was idle and its idle state was successfully cleared. - * %false otherwise. - * - * Unavailable if ops.update_idle() is implemented and - * %SCX_OPS_KEEP_BUILTIN_IDLE is not set. - */ -__bpf_kfunc bool scx_bpf_test_and_clear_cpu_idle(s32 cpu) -{ - if (!static_branch_likely(&scx_builtin_idle_enabled)) { - scx_ops_error("built-in idle tracking is disabled"); - return false; - } - - if (ops_cpu_valid(cpu, NULL)) - return test_and_clear_cpu_idle(cpu); - else - return false; -} - -/** - * scx_bpf_pick_idle_cpu - Pick and claim an idle cpu - * @cpus_allowed: Allowed cpumask - * @flags: %SCX_PICK_IDLE_CPU_* flags - * - * Pick and claim an idle cpu in @cpus_allowed. Returns the picked idle cpu - * number on success. -%EBUSY if no matching cpu was found. - * - * Idle CPU tracking may race against CPU scheduling state transitions. For - * example, this function may return -%EBUSY as CPUs are transitioning into the - * idle state. If the caller then assumes that there will be dispatch events on - * the CPUs as they were all busy, the scheduler may end up stalling with CPUs - * idling while there are pending tasks. Use scx_bpf_pick_any_cpu() and - * scx_bpf_kick_cpu() to guarantee that there will be at least one dispatch - * event in the near future. - * - * Unavailable if ops.update_idle() is implemented and - * %SCX_OPS_KEEP_BUILTIN_IDLE is not set. - */ -__bpf_kfunc s32 scx_bpf_pick_idle_cpu(const struct cpumask *cpus_allowed, - u64 flags) -{ - if (!static_branch_likely(&scx_builtin_idle_enabled)) { - scx_ops_error("built-in idle tracking is disabled"); - return -EBUSY; - } - - return scx_pick_idle_cpu(cpus_allowed, flags); -} - -/** - * scx_bpf_pick_any_cpu - Pick and claim an idle cpu if available or pick any CPU - * @cpus_allowed: Allowed cpumask - * @flags: %SCX_PICK_IDLE_CPU_* flags - * - * Pick and claim an idle cpu in @cpus_allowed. If none is available, pick any - * CPU in @cpus_allowed. Guaranteed to succeed and returns the picked idle cpu - * number if @cpus_allowed is not empty. -%EBUSY is returned if @cpus_allowed is - * empty. - * - * If ops.update_idle() is implemented and %SCX_OPS_KEEP_BUILTIN_IDLE is not - * set, this function can't tell which CPUs are idle and will always pick any - * CPU. - */ -__bpf_kfunc s32 scx_bpf_pick_any_cpu(const struct cpumask *cpus_allowed, - u64 flags) -{ - s32 cpu; - - if (static_branch_likely(&scx_builtin_idle_enabled)) { - cpu = scx_pick_idle_cpu(cpus_allowed, flags); - if (cpu >= 0) - return cpu; - } - - cpu = cpumask_any_distribute(cpus_allowed); - if (cpu < nr_cpu_ids) - return cpu; - else - return -EBUSY; -} - /** * scx_bpf_task_running - Is task currently running? * @p: task of interest @@ -7620,12 +6956,6 @@ BTF_ID_FLAGS(func, scx_bpf_nr_cpu_ids) BTF_ID_FLAGS(func, scx_bpf_get_possible_cpumask, KF_ACQUIRE) BTF_ID_FLAGS(func, scx_bpf_get_online_cpumask, KF_ACQUIRE) BTF_ID_FLAGS(func, scx_bpf_put_cpumask, KF_RELEASE) -BTF_ID_FLAGS(func, scx_bpf_get_idle_cpumask, KF_ACQUIRE) -BTF_ID_FLAGS(func, scx_bpf_get_idle_smtmask, KF_ACQUIRE) -BTF_ID_FLAGS(func, scx_bpf_put_idle_cpumask, KF_RELEASE) -BTF_ID_FLAGS(func, scx_bpf_test_and_clear_cpu_idle) -BTF_ID_FLAGS(func, scx_bpf_pick_idle_cpu, KF_RCU) -BTF_ID_FLAGS(func, scx_bpf_pick_any_cpu, KF_RCU) BTF_ID_FLAGS(func, scx_bpf_task_running, KF_RCU) BTF_ID_FLAGS(func, scx_bpf_task_cpu, KF_RCU) BTF_ID_FLAGS(func, scx_bpf_cpu_rq) diff --git a/kernel/sched/ext_idle.c b/kernel/sched/ext_idle.c new file mode 100644 index 000000000000..9e8479dd7277 --- /dev/null +++ b/kernel/sched/ext_idle.c @@ -0,0 +1,686 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * BPF extensible scheduler class: Documentation/scheduler/sched-ext.rst + * + * Built-in idle CPU tracking policy. + * + * Copyright (c) 2022 Meta Platforms, Inc. and affiliates. + * Copyright (c) 2022 Tejun Heo + * Copyright (c) 2022 David Vernet + * Copyright (c) 2024 Andrea Righi + */ + +static DEFINE_STATIC_KEY_FALSE(scx_builtin_idle_enabled); + +#ifdef CONFIG_SMP +#ifdef CONFIG_CPUMASK_OFFSTACK +#define CL_ALIGNED_IF_ONSTACK +#else +#define CL_ALIGNED_IF_ONSTACK __cacheline_aligned_in_smp +#endif + +static struct { + cpumask_var_t cpu; + cpumask_var_t smt; +} idle_masks CL_ALIGNED_IF_ONSTACK; + +static DEFINE_STATIC_KEY_FALSE(scx_selcpu_topo_llc); +static DEFINE_STATIC_KEY_FALSE(scx_selcpu_topo_numa); + +static bool test_and_clear_cpu_idle(int cpu) +{ +#ifdef CONFIG_SCHED_SMT + /* + * SMT mask should be cleared whether we can claim @cpu or not. The SMT + * cluster is not wholly idle either way. This also prevents + * scx_pick_idle_cpu() from getting caught in an infinite loop. + */ + if (sched_smt_active()) { + const struct cpumask *smt = cpu_smt_mask(cpu); + + /* + * If offline, @cpu is not its own sibling and + * scx_pick_idle_cpu() can get caught in an infinite loop as + * @cpu is never cleared from idle_masks.smt. Ensure that @cpu + * is eventually cleared. + */ + if (cpumask_intersects(smt, idle_masks.smt)) + cpumask_andnot(idle_masks.smt, idle_masks.smt, smt); + else if (cpumask_test_cpu(cpu, idle_masks.smt)) + __cpumask_clear_cpu(cpu, idle_masks.smt); + } +#endif + return cpumask_test_and_clear_cpu(cpu, idle_masks.cpu); +} + +static s32 scx_pick_idle_cpu(const struct cpumask *cpus_allowed, u64 flags) +{ + int cpu; + +retry: + if (sched_smt_active()) { + cpu = cpumask_any_and_distribute(idle_masks.smt, cpus_allowed); + if (cpu < nr_cpu_ids) + goto found; + + if (flags & SCX_PICK_IDLE_CORE) + return -EBUSY; + } + + cpu = cpumask_any_and_distribute(idle_masks.cpu, cpus_allowed); + if (cpu >= nr_cpu_ids) + return -EBUSY; + +found: + if (test_and_clear_cpu_idle(cpu)) + return cpu; + else + goto retry; +} + +/* + * Return the amount of CPUs in the same LLC domain of @cpu (or zero if the LLC + * domain is not defined). + */ +static unsigned int llc_weight(s32 cpu) +{ + struct sched_domain *sd; + + sd = rcu_dereference(per_cpu(sd_llc, cpu)); + if (!sd) + return 0; + + return sd->span_weight; +} + +/* + * Return the cpumask representing the LLC domain of @cpu (or NULL if the LLC + * domain is not defined). + */ +static struct cpumask *llc_span(s32 cpu) +{ + struct sched_domain *sd; + + sd = rcu_dereference(per_cpu(sd_llc, cpu)); + if (!sd) + return 0; + + return sched_domain_span(sd); +} + +/* + * Return the amount of CPUs in the same NUMA domain of @cpu (or zero if the + * NUMA domain is not defined). + */ +static unsigned int numa_weight(s32 cpu) +{ + struct sched_domain *sd; + struct sched_group *sg; + + sd = rcu_dereference(per_cpu(sd_numa, cpu)); + if (!sd) + return 0; + sg = sd->groups; + if (!sg) + return 0; + + return sg->group_weight; +} + +/* + * Return the cpumask representing the NUMA domain of @cpu (or NULL if the NUMA + * domain is not defined). + */ +static struct cpumask *numa_span(s32 cpu) +{ + struct sched_domain *sd; + struct sched_group *sg; + + sd = rcu_dereference(per_cpu(sd_numa, cpu)); + if (!sd) + return NULL; + sg = sd->groups; + if (!sg) + return NULL; + + return sched_group_span(sg); +} + +/* + * Return true if the LLC domains do not perfectly overlap with the NUMA + * domains, false otherwise. + */ +static bool llc_numa_mismatch(void) +{ + int cpu; + + /* + * We need to scan all online CPUs to verify whether their scheduling + * domains overlap. + * + * While it is rare to encounter architectures with asymmetric NUMA + * topologies, CPU hotplugging or virtualized environments can result + * in asymmetric configurations. + * + * For example: + * + * NUMA 0: + * - LLC 0: cpu0..cpu7 + * - LLC 1: cpu8..cpu15 [offline] + * + * NUMA 1: + * - LLC 0: cpu16..cpu23 + * - LLC 1: cpu24..cpu31 + * + * In this case, if we only check the first online CPU (cpu0), we might + * incorrectly assume that the LLC and NUMA domains are fully + * overlapping, which is incorrect (as NUMA 1 has two distinct LLC + * domains). + */ + for_each_online_cpu(cpu) + if (llc_weight(cpu) != numa_weight(cpu)) + return true; + + return false; +} + +/* + * Initialize topology-aware scheduling. + * + * Detect if the system has multiple LLC or multiple NUMA domains and enable + * cache-aware / NUMA-aware scheduling optimizations in the default CPU idle + * selection policy. + * + * Assumption: the kernel's internal topology representation assumes that each + * CPU belongs to a single LLC domain, and that each LLC domain is entirely + * contained within a single NUMA node. + */ +static void update_selcpu_topology(void) +{ + bool enable_llc = false, enable_numa = false; + unsigned int nr_cpus; + s32 cpu = cpumask_first(cpu_online_mask); + + /* + * Enable LLC domain optimization only when there are multiple LLC + * domains among the online CPUs. If all online CPUs are part of a + * single LLC domain, the idle CPU selection logic can choose any + * online CPU without bias. + * + * Note that it is sufficient to check the LLC domain of the first + * online CPU to determine whether a single LLC domain includes all + * CPUs. + */ + rcu_read_lock(); + nr_cpus = llc_weight(cpu); + if (nr_cpus > 0) { + if (nr_cpus < num_online_cpus()) + enable_llc = true; + pr_debug("sched_ext: LLC=%*pb weight=%u\n", + cpumask_pr_args(llc_span(cpu)), llc_weight(cpu)); + } + + /* + * Enable NUMA optimization only when there are multiple NUMA domains + * among the online CPUs and the NUMA domains don't perfectly overlaps + * with the LLC domains. + * + * If all CPUs belong to the same NUMA node and the same LLC domain, + * enabling both NUMA and LLC optimizations is unnecessary, as checking + * for an idle CPU in the same domain twice is redundant. + */ + nr_cpus = numa_weight(cpu); + if (nr_cpus > 0) { + if (nr_cpus < num_online_cpus() && llc_numa_mismatch()) + enable_numa = true; + pr_debug("sched_ext: NUMA=%*pb weight=%u\n", + cpumask_pr_args(numa_span(cpu)), numa_weight(cpu)); + } + rcu_read_unlock(); + + pr_debug("sched_ext: LLC idle selection %s\n", + enable_llc ? "enabled" : "disabled"); + pr_debug("sched_ext: NUMA idle selection %s\n", + enable_numa ? "enabled" : "disabled"); + + if (enable_llc) + static_branch_enable_cpuslocked(&scx_selcpu_topo_llc); + else + static_branch_disable_cpuslocked(&scx_selcpu_topo_llc); + if (enable_numa) + static_branch_enable_cpuslocked(&scx_selcpu_topo_numa); + else + static_branch_disable_cpuslocked(&scx_selcpu_topo_numa); +} + +/* + * Built-in CPU idle selection policy: + * + * 1. Prioritize full-idle cores: + * - always prioritize CPUs from fully idle cores (both logical CPUs are + * idle) to avoid interference caused by SMT. + * + * 2. Reuse the same CPU: + * - prefer the last used CPU to take advantage of cached data (L1, L2) and + * branch prediction optimizations. + * + * 3. Pick a CPU within the same LLC (Last-Level Cache): + * - if the above conditions aren't met, pick a CPU that shares the same LLC + * to maintain cache locality. + * + * 4. Pick a CPU within the same NUMA node, if enabled: + * - choose a CPU from the same NUMA node to reduce memory access latency. + * + * Step 3 and 4 are performed only if the system has, respectively, multiple + * LLC domains / multiple NUMA nodes (see scx_selcpu_topo_llc and + * scx_selcpu_topo_numa). + * + * NOTE: tasks that can only run on 1 CPU are excluded by this logic, because + * we never call ops.select_cpu() for them, see select_task_rq(). + */ +static s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, + u64 wake_flags, bool *found) +{ + const struct cpumask *llc_cpus = NULL; + const struct cpumask *numa_cpus = NULL; + s32 cpu; + + *found = false; + + /* + * This is necessary to protect llc_cpus. + */ + rcu_read_lock(); + + /* + * Determine the scheduling domain only if the task is allowed to run + * on all CPUs. + * + * This is done primarily for efficiency, as it avoids the overhead of + * updating a cpumask every time we need to select an idle CPU (which + * can be costly in large SMP systems), but it also aligns logically: + * if a task's scheduling domain is restricted by user-space (through + * CPU affinity), the task will simply use the flat scheduling domain + * defined by user-space. + */ + if (p->nr_cpus_allowed >= num_possible_cpus()) { + if (static_branch_maybe(CONFIG_NUMA, &scx_selcpu_topo_numa)) + numa_cpus = numa_span(prev_cpu); + + if (static_branch_maybe(CONFIG_SCHED_MC, &scx_selcpu_topo_llc)) + llc_cpus = llc_span(prev_cpu); + } + + /* + * If WAKE_SYNC, try to migrate the wakee to the waker's CPU. + */ + if (wake_flags & SCX_WAKE_SYNC) { + cpu = smp_processor_id(); + + /* + * If the waker's CPU is cache affine and prev_cpu is idle, + * then avoid a migration. + */ + if (cpus_share_cache(cpu, prev_cpu) && + test_and_clear_cpu_idle(prev_cpu)) { + cpu = prev_cpu; + goto cpu_found; + } + + /* + * If the waker's local DSQ is empty, and the system is under + * utilized, try to wake up @p to the local DSQ of the waker. + * + * Checking only for an empty local DSQ is insufficient as it + * could give the wakee an unfair advantage when the system is + * oversaturated. + * + * Checking only for the presence of idle CPUs is also + * insufficient as the local DSQ of the waker could have tasks + * piled up on it even if there is an idle core elsewhere on + * the system. + */ + if (!cpumask_empty(idle_masks.cpu) && + !(current->flags & PF_EXITING) && + cpu_rq(cpu)->scx.local_dsq.nr == 0) { + if (cpumask_test_cpu(cpu, p->cpus_ptr)) + goto cpu_found; + } + } + + /* + * If CPU has SMT, any wholly idle CPU is likely a better pick than + * partially idle @prev_cpu. + */ + if (sched_smt_active()) { + /* + * Keep using @prev_cpu if it's part of a fully idle core. + */ + if (cpumask_test_cpu(prev_cpu, idle_masks.smt) && + test_and_clear_cpu_idle(prev_cpu)) { + cpu = prev_cpu; + goto cpu_found; + } + + /* + * Search for any fully idle core in the same LLC domain. + */ + if (llc_cpus) { + cpu = scx_pick_idle_cpu(llc_cpus, SCX_PICK_IDLE_CORE); + if (cpu >= 0) + goto cpu_found; + } + + /* + * Search for any fully idle core in the same NUMA node. + */ + if (numa_cpus) { + cpu = scx_pick_idle_cpu(numa_cpus, SCX_PICK_IDLE_CORE); + if (cpu >= 0) + goto cpu_found; + } + + /* + * Search for any full idle core usable by the task. + */ + cpu = scx_pick_idle_cpu(p->cpus_ptr, SCX_PICK_IDLE_CORE); + if (cpu >= 0) + goto cpu_found; + } + + /* + * Use @prev_cpu if it's idle. + */ + if (test_and_clear_cpu_idle(prev_cpu)) { + cpu = prev_cpu; + goto cpu_found; + } + + /* + * Search for any idle CPU in the same LLC domain. + */ + if (llc_cpus) { + cpu = scx_pick_idle_cpu(llc_cpus, 0); + if (cpu >= 0) + goto cpu_found; + } + + /* + * Search for any idle CPU in the same NUMA node. + */ + if (numa_cpus) { + cpu = scx_pick_idle_cpu(numa_cpus, 0); + if (cpu >= 0) + goto cpu_found; + } + + /* + * Search for any idle CPU usable by the task. + */ + cpu = scx_pick_idle_cpu(p->cpus_ptr, 0); + if (cpu >= 0) + goto cpu_found; + + rcu_read_unlock(); + return prev_cpu; + +cpu_found: + rcu_read_unlock(); + + *found = true; + return cpu; +} + +static void reset_idle_masks(void) +{ + /* + * Consider all online cpus idle. Should converge to the actual state + * quickly. + */ + cpumask_copy(idle_masks.cpu, cpu_online_mask); + cpumask_copy(idle_masks.smt, cpu_online_mask); +} + +void __scx_update_idle(struct rq *rq, bool idle) +{ + int cpu = cpu_of(rq); + + if (SCX_HAS_OP(update_idle) && !scx_rq_bypassing(rq)) { + SCX_CALL_OP(SCX_KF_REST, update_idle, cpu_of(rq), idle); + if (!static_branch_unlikely(&scx_builtin_idle_enabled)) + return; + } + + if (idle) + cpumask_set_cpu(cpu, idle_masks.cpu); + else + cpumask_clear_cpu(cpu, idle_masks.cpu); + +#ifdef CONFIG_SCHED_SMT + if (sched_smt_active()) { + const struct cpumask *smt = cpu_smt_mask(cpu); + + if (idle) { + /* + * idle_masks.smt handling is racy but that's fine as + * it's only for optimization and self-correcting. + */ + for_each_cpu(cpu, smt) { + if (!cpumask_test_cpu(cpu, idle_masks.cpu)) + return; + } + cpumask_or(idle_masks.smt, idle_masks.smt, smt); + } else { + cpumask_andnot(idle_masks.smt, idle_masks.smt, smt); + } + } +#endif +} + +#else /* !CONFIG_SMP */ + +static bool test_and_clear_cpu_idle(int cpu) { return false; } +static s32 scx_pick_idle_cpu(const struct cpumask *cpus_allowed, u64 flags) { return -EBUSY; } +static void reset_idle_masks(void) {} + +#endif /* CONFIG_SMP */ + + +/******************************************************************************** + * Helpers that can be called from the BPF scheduler. + */ +__bpf_kfunc_start_defs(); + +/** + * scx_bpf_select_cpu_dfl - The default implementation of ops.select_cpu() + * @p: task_struct to select a CPU for + * @prev_cpu: CPU @p was on previously + * @wake_flags: %SCX_WAKE_* flags + * @is_idle: out parameter indicating whether the returned CPU is idle + * + * Can only be called from ops.select_cpu() if the built-in CPU selection is + * enabled - ops.update_idle() is missing or %SCX_OPS_KEEP_BUILTIN_IDLE is set. + * @p, @prev_cpu and @wake_flags match ops.select_cpu(). + * + * Returns the picked CPU with *@is_idle indicating whether the picked CPU is + * currently idle and thus a good candidate for direct dispatching. + */ +__bpf_kfunc s32 scx_bpf_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, + u64 wake_flags, bool *is_idle) +{ + if (!static_branch_likely(&scx_builtin_idle_enabled)) { + scx_ops_error("built-in idle tracking is disabled"); + goto prev_cpu; + } + + if (!scx_kf_allowed(SCX_KF_SELECT_CPU)) + goto prev_cpu; + +#ifdef CONFIG_SMP + return scx_select_cpu_dfl(p, prev_cpu, wake_flags, is_idle); +#endif + +prev_cpu: + *is_idle = false; + return prev_cpu; +} + +/** + * scx_bpf_get_idle_cpumask - Get a referenced kptr to the idle-tracking + * per-CPU cpumask. + * + * Returns NULL if idle tracking is not enabled, or running on a UP kernel. + */ +__bpf_kfunc const struct cpumask *scx_bpf_get_idle_cpumask(void) +{ + if (!static_branch_likely(&scx_builtin_idle_enabled)) { + scx_ops_error("built-in idle tracking is disabled"); + return cpu_none_mask; + } + +#ifdef CONFIG_SMP + return idle_masks.cpu; +#else + return cpu_none_mask; +#endif +} + +/** + * scx_bpf_get_idle_smtmask - Get a referenced kptr to the idle-tracking, + * per-physical-core cpumask. Can be used to determine if an entire physical + * core is free. + * + * Returns NULL if idle tracking is not enabled, or running on a UP kernel. + */ +__bpf_kfunc const struct cpumask *scx_bpf_get_idle_smtmask(void) +{ + if (!static_branch_likely(&scx_builtin_idle_enabled)) { + scx_ops_error("built-in idle tracking is disabled"); + return cpu_none_mask; + } + +#ifdef CONFIG_SMP + if (sched_smt_active()) + return idle_masks.smt; + else + return idle_masks.cpu; +#else + return cpu_none_mask; +#endif +} + +/** + * scx_bpf_put_idle_cpumask - Release a previously acquired referenced kptr to + * either the percpu, or SMT idle-tracking cpumask. + */ +__bpf_kfunc void scx_bpf_put_idle_cpumask(const struct cpumask *idle_mask) +{ + /* + * Empty function body because we aren't actually acquiring or releasing + * a reference to a global idle cpumask, which is read-only in the + * caller and is never released. The acquire / release semantics here + * are just used to make the cpumask a trusted pointer in the caller. + */ +} + +/** + * scx_bpf_test_and_clear_cpu_idle - Test and clear @cpu's idle state + * @cpu: cpu to test and clear idle for + * + * Returns %true if @cpu was idle and its idle state was successfully cleared. + * %false otherwise. + * + * Unavailable if ops.update_idle() is implemented and + * %SCX_OPS_KEEP_BUILTIN_IDLE is not set. + */ +__bpf_kfunc bool scx_bpf_test_and_clear_cpu_idle(s32 cpu) +{ + if (!static_branch_likely(&scx_builtin_idle_enabled)) { + scx_ops_error("built-in idle tracking is disabled"); + return false; + } + + if (ops_cpu_valid(cpu, NULL)) + return test_and_clear_cpu_idle(cpu); + else + return false; +} + +/** + * scx_bpf_pick_idle_cpu - Pick and claim an idle cpu + * @cpus_allowed: Allowed cpumask + * @flags: %SCX_PICK_IDLE_CPU_* flags + * + * Pick and claim an idle cpu in @cpus_allowed. Returns the picked idle cpu + * number on success. -%EBUSY if no matching cpu was found. + * + * Idle CPU tracking may race against CPU scheduling state transitions. For + * example, this function may return -%EBUSY as CPUs are transitioning into the + * idle state. If the caller then assumes that there will be dispatch events on + * the CPUs as they were all busy, the scheduler may end up stalling with CPUs + * idling while there are pending tasks. Use scx_bpf_pick_any_cpu() and + * scx_bpf_kick_cpu() to guarantee that there will be at least one dispatch + * event in the near future. + * + * Unavailable if ops.update_idle() is implemented and + * %SCX_OPS_KEEP_BUILTIN_IDLE is not set. + */ +__bpf_kfunc s32 scx_bpf_pick_idle_cpu(const struct cpumask *cpus_allowed, + u64 flags) +{ + if (!static_branch_likely(&scx_builtin_idle_enabled)) { + scx_ops_error("built-in idle tracking is disabled"); + return -EBUSY; + } + + return scx_pick_idle_cpu(cpus_allowed, flags); +} + +/** + * scx_bpf_pick_any_cpu - Pick and claim an idle cpu if available or pick any CPU + * @cpus_allowed: Allowed cpumask + * @flags: %SCX_PICK_IDLE_CPU_* flags + * + * Pick and claim an idle cpu in @cpus_allowed. If none is available, pick any + * CPU in @cpus_allowed. Guaranteed to succeed and returns the picked idle cpu + * number if @cpus_allowed is not empty. -%EBUSY is returned if @cpus_allowed is + * empty. + * + * If ops.update_idle() is implemented and %SCX_OPS_KEEP_BUILTIN_IDLE is not + * set, this function can't tell which CPUs are idle and will always pick any + * CPU. + */ +__bpf_kfunc s32 scx_bpf_pick_any_cpu(const struct cpumask *cpus_allowed, + u64 flags) +{ + s32 cpu; + + if (static_branch_likely(&scx_builtin_idle_enabled)) { + cpu = scx_pick_idle_cpu(cpus_allowed, flags); + if (cpu >= 0) + return cpu; + } + + cpu = cpumask_any_distribute(cpus_allowed); + if (cpu < nr_cpu_ids) + return cpu; + else + return -EBUSY; +} + +__bpf_kfunc_end_defs(); + +BTF_KFUNCS_START(scx_kfunc_ids_select_cpu) +BTF_ID_FLAGS(func, scx_bpf_select_cpu_dfl, KF_RCU) +BTF_ID_FLAGS(func, scx_bpf_get_idle_cpumask, KF_ACQUIRE) +BTF_ID_FLAGS(func, scx_bpf_get_idle_smtmask, KF_ACQUIRE) +BTF_ID_FLAGS(func, scx_bpf_put_idle_cpumask, KF_RELEASE) +BTF_ID_FLAGS(func, scx_bpf_test_and_clear_cpu_idle) +BTF_ID_FLAGS(func, scx_bpf_pick_idle_cpu, KF_RCU) +BTF_ID_FLAGS(func, scx_bpf_pick_any_cpu, KF_RCU) +BTF_KFUNCS_END(scx_kfunc_ids_select_cpu) + +static const struct btf_kfunc_id_set scx_kfunc_set_select_cpu = { + .owner = THIS_MODULE, + .set = &scx_kfunc_ids_select_cpu, +}; From patchwork Fri Dec 20 15:11:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrea Righi X-Patchwork-Id: 13916898 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2059.outbound.protection.outlook.com [40.107.244.59]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AE261219A8F; Fri, 20 Dec 2024 15:41:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.244.59 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734709293; cv=fail; b=B/GL/wN7yJooJAgUQ6XJ89PiuFzRtlp7mgQO6fhAwMB+hVPCDfLgkGKwNomQ/r5ia6USRL1Knv+k3l3npdQjlMPzbhg/ZOaPHLlRiCoswH+Yc1zckfxfavusV4I7vjSuqjvUImc8v25SNHYHVhcbaaNEwveONQcl4rpUoZkT+Lo= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734709293; c=relaxed/simple; bh=rd1KLkv6/KG16obzT3mVY7o4zIbTwBv90ueReM766wE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=ay7MtJzJ64oqg1C7dGUgVVgqsnGllO+90pd6WXDdOTB7MDJMJlCbylbBKyofOt7wY2C5yr7Fcwg4kzl91EFz8KbgJP5BMTW2NME0quHiMNuk3XhWqkigFBP1aotssb5m5i8IL+F34hc9RZppX1La71Wi28atj3Zs1NYsX1yjZGo= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=JmPLjOzs; arc=fail smtp.client-ip=40.107.244.59 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="JmPLjOzs" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=oM6lZshcLBW9FoDMN74RVksAZIVkwZ8LqHOhNQDJB6Ovn8NX9l4qAB7fI8eGroM5i7w9c/lewDy46e3y4Uqifb13IJ07czSzpBruaXtYyfHN0HWaupdARNOGdtZNLX1nbYM/5qoK0b2Ydt1MOrSDBtCe3Ew6WMLiwkZOX1KuxboeIjitEn95i/T1DaQWDFcz6I5sZO8O2uvvKrW/OdAhDaJ9Xhszu92iCA2SXKX1XP+aJ3WOuwfBq8G7QJyTlAxFsKzZaPrTlPhmPWGHz5DwiLiCvFcvIKj8Ug1zkz00w8jk69UFpaS6Y11jqAmj7QoKEngQ/LEeVPbHdNqEVvdUtA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=MSAf/5oeljZeD2slGVutVoZN1cVbQQFKEBFtPdfHCcg=; b=Vh5ZKD52HlnMS4bHXXeIeddbI/VhZq8aDLXS98ld//5oe1KOvB8GpMjpsREZkPTvozwW7Gpe+B39i1zOL+uSegnSa7+Ouha3l3rdUm7I1EzzGU5g2xGqYyUlxmhtpgFPcg3fJxD4sRW0tWnnPS/TwNq1p5C0JlrX337z/7c8iZpFL774vDObXJF0Q6i1lhSQcCOQBuI6E0Kp+Re+I28by+2MVoHYQDLheyAb5ajQq/dxiVF/wYcIyROXAu12CNhHSWBz+VYA77WZDKPjiPg8fkeUQOKs4xx10DvLglm0FZIeMYvEpMGP8M2LzOumErkEWMD1J95pV0KOQ9DkFYx+oA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=MSAf/5oeljZeD2slGVutVoZN1cVbQQFKEBFtPdfHCcg=; b=JmPLjOzsbP0hEzYvfzW0FvghxfEaW/0NYb51PBHbU+xmt1z/0yhP1/YO0RByeDfDFjW3oPvoXGPohMV+uOGZVHnYCm7/pHRRitVEkd55pN+VzgcBJkZBKsHx7/+rHge+AAgIvz1TyNDgSNrDeFCWuYg2t/tNWRWCSWoSzFrYtc2yrH3liTFQRameSP13MA7YgBA7BMgFGQqdwnYqAhVAaWzQJ9mj68a+qa7mgTge0j4gBY/Q8q3SS3ECR1mHpKappTENYg7e8I+n9tWvfEXhvNwWyg0BTkMt5FhwMvQ9WqXSP3y62l3rD9fOyx+6sUKo9q76JMYWpp6q+CeX5lfPgg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) by SA1PR12MB7149.namprd12.prod.outlook.com (2603:10b6:806:29c::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8272.13; Fri, 20 Dec 2024 15:41:28 +0000 Received: from CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5]) by CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5%3]) with mapi id 15.20.8272.005; Fri, 20 Dec 2024 15:41:28 +0000 From: Andrea Righi To: Tejun Heo , David Vernet , Changwoo Min Cc: Yury Norov , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , bpf@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 03/10] sched_ext: idle: introduce check_builtin_idle_enabled() helper Date: Fri, 20 Dec 2024 16:11:35 +0100 Message-ID: <20241220154107.287478-4-arighi@nvidia.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241220154107.287478-1-arighi@nvidia.com> References: <20241220154107.287478-1-arighi@nvidia.com> X-ClientProxiedBy: FR4P281CA0281.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:e6::14) To CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY5PR12MB6405:EE_|SA1PR12MB7149:EE_ X-MS-Office365-Filtering-Correlation-Id: 3e06a7fe-e3f0-4e76-47f3-08dd210cc50f X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|366016|376014|7416014; X-Microsoft-Antispam-Message-Info: ZtnGpCYbYuEFNOO6WhlTIo9zPEgq9kBKETOk/C0eT3hf83ERzkN/D7vTGk6tzgZwP7pMxIL2HfSlDDiBZHgiDgNPJ77vzhgZoaWlcnufjbdhFGvpE4udNlX5v0GUixqqNgJx6ar8ZrzwT82QgfN8tCeVokvYHeehSA8fvH26w7FrjvnBH52ubqeIOSC3Vgh57gIRYhD7191qOVpNDWEIPV4GZJJy2r50DpW87NvnybFGqtdV4QRk0OW0dYMLvHPosBEfSTNacptVW4FuzrBQhmY0yQunRbG9MUC6kOSkS1As33B1K5XHOKX9kRID4o4oyku/Twz502IPtecEtgqdXlpTIcIfQffcReh7XGul+HAoVDjv/xLbZG1J9i3uujcOV4sSm5MWEHJHPa/oeOH6PX7mMIPlHCbYGp/YaZ3nkF7hDlhmAbiiQSl266TCjPE4X3hjpTcXxIO2kuNVTssX0UUVjb4x7VnDhqCkOZgCj6NPDl0mkhi69gkKHpxootm1YtI/h8Xoqx30WPR1LnQ3LRpgwU1HkHYzXUguKAEqBAV0GSKbs49Oe+MSAeMG/HzPaz2HTgI5tbDWS0lQ+cPrPac5yVfnzINq0jMTZTQtyjaayQgcECc823+ZcpvRkBx+G2pmf+Tbf3G+/pSBeclGVjnjcisazYBFGB9NWhrsWbHD0xCP6QWn6ebkpNMPP9/V9UhJbf/SEMruxc2Uwp22IRZ8uVqh0KRiKO4KbkFfwh5DTk2j6us05VuDiPAQT2p/dO1IGNCJJ9p60PVhDMC78iw+4DZwHg32ihVTnhBVIGAK+cZu3GE6bmw1EnWBskU+yjoBJRA24thtpoLBc2bGOK3D5ziQ4OuQEL1nB1oZESYIImDwnsr1I6WViZi0pXl3Ihjrb4DfbejQenMtSd2kpaLHMfd+ns0+AuqErp4PnhUzd/jjWSjXvReTskuLTwMlWsteD62STld6Py6eGn6SlFDRyw5SDoNcFkOSiTxKoRy3HXZ4Xske84bWP6mo0RU8D0JV+pB+gdaGL1igVGxdzZGMWqW2UdmsOesFQg0bjBVUgDHfEZ/MQ9Tp4Hdc6+GEmqjdEeGktA2BV3g8phZ5QChrJN/ahMe2UhIgKmjEH7fuiC2rXfND0E9ge9zETseR20mJyPHnU0f7EfktKJVYw/aBGYwfapCt9n7a1+EBGetdvrJ4+1/U+PHgfdPKDKC4a4M78GZfQ0dSGbSIPvwwdUwhLfpCsJlcZ5CP1A3sOj9EkAryzZSPCwnLp6adk2yi0QF3niWS5rSd9mIN1K2CgaU6+f1oPPSKmjU3YEgwDZQ4XDBY1S1zWG9nlSmvklCPuiKTnFpA6wNtADRdjQVettPe40WL58l29A3/vNnyGj+OEiqNF6bZ0If4ZFZ3WxE4 X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CY5PR12MB6405.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(366016)(376014)(7416014);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: PrI8qMo5usMejigqRvxbb76TjDHvK0MAmC2msC0Wy6EwmAp466IguVp8JAjjIm44MOle2Gn6BjAa3KXJcieEuVrvyEpBzsUwkfG81Pm94MYcyxgwgvvgzV02e9Iv+xeSN/O7du5QQpidf1juArAXt0mvdey5E3Xvz883I/4WcCi3WTPyTI/3xuKjSH3/t+bMGfYxEWPQBNUF/+hguUsrwYXd40GdW2LUd/z5t46dOr22lX5xhwlmJo+iNW0T0bcr/nVjvvwnSqZ/lW8NWDq/7OACeJOM+Cxgxih2RkkkHIgO2+a9qGMl90O92P0NYLzIo0DhjWpQZsJuYBDkzvbOiw8x2iX22yl4169BonL+BrBn23TRHfAfoL6QqSyrnGbOTP2ZmOcT9vQsiWaKxrPZqlRLKMKa3Z7TAoGi6VxQdqo+SEbgNbhS8G16AxRERF7g9oIFSTvjMDCfvigTOEmYhO2ZWb3K16ipIov4TZpz9xs4JT8XkgNcRrnGol8AGZNMS09dm7x1azizpsYeSgMG1fzpfJxSXXRF9qIje4mtIdNNSP1St490Gcqtcln9hx2vdYyNjxGdhwNkthc5hxq4DpfoK6/Vs+STNzFzFcXqI61WMXM2VuBbfMde6zSDtQYw+2lZOjwuD5Fn0s66TC7H8nGIUnoIGb4oXiGFDXaRXuTiM91HolRChWIRvxyY8mB9om/wtY7SzFs1xAGcfqknvnvGhIiZu7ktoSzSZUgYupaIS4EUPa/lElXDwp107Q3CH3YZ8n2SzQy3h1lPrBGJaVCIyrxRRj1MmjL/lROxL+KMbt8oqarf1hHgJLQTwjp1t7//y8rGB/pwmklAYjcRLgVyUofs3qQTBcF/C+GBX61mCcZoRaquQHwKGh8CVLHxB18QIiprkuUWUmLx3/TggINgR6Iw/XLadsjkYL/34KZf7KeQEyOWOXKUR6ziBJyeHm8vicHAjzXpMZhxR8xXoxE+YBN5h/TdyALMjP0UGauOPDXDIx9/inWnSg080miuayhISMdJRs7ep7NYcIdKS/Bp2fgtv1CVmDFim2ecDC8jabPcMXzbuRVbBq56H00HIParZWK/Pj2/5nR+I/Fl9gPzICRkq8vKjfIvJIi95YqfMD8iUF25fiGlEd00uwYgCtishKa76K1ONApx+Q3MXHXb1e5qO6FyeFuLFqGO0zKp9MtJN6aoJH19iXS38ZRvyWgpxMYFXCicOfd2lZ/96+gK3buT6phQS6kr7b6f45QY1Gq/jH5SdFf9uzt6qlDdE4qizUZs81lmmgnl3zCR6DstoKHzTpXI3ltZe5YSN5zygxFO2szDduXVP5L0rrtc2q3ZquyEJlm9HXxcH5hEl8KqVk6MdLsbEk3M+Hsj2ZrJbb4dbtXBPwNgVEuxk9c67y+1QJfQx4sL6dqa3xWoeeNsa7Ptlxk8MnFikDpHD3DzbpTknb7bKfHcQGlXxCO6TzTwEFgMJisg4loLlbrOJ47XOEW49QO23dV4XpoB1FNm66oN+wa+7O94R29gxjFnyZyaHMV6fP5WhbWB0Y9SSB/b1rFc8bG5xhBtpp9JRUqcid6zI8cL5DXhG0wmleRf X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 3e06a7fe-e3f0-4e76-47f3-08dd210cc50f X-MS-Exchange-CrossTenant-AuthSource: CY5PR12MB6405.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Dec 2024 15:41:28.6843 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: iuJi//y/MrSryQ0bqVyFf/WsPG+bCmNjmUMPtZyRACcuUIEKH5qzeVx6j/nQVYdBwzS7QBsO+ZwGGcLs4icrJA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB7149 Minor refactoring to add a helper function for checking if the built-in idle CPU selection policy is enabled. Signed-off-by: Andrea Righi --- kernel/sched/ext_idle.c | 29 ++++++++++++++--------------- 1 file changed, 14 insertions(+), 15 deletions(-) diff --git a/kernel/sched/ext_idle.c b/kernel/sched/ext_idle.c index 9e8479dd7277..0e57830072e4 100644 --- a/kernel/sched/ext_idle.c +++ b/kernel/sched/ext_idle.c @@ -12,6 +12,15 @@ static DEFINE_STATIC_KEY_FALSE(scx_builtin_idle_enabled); +static bool check_builtin_idle_enabled(void) +{ + if (static_branch_likely(&scx_builtin_idle_enabled)) + return true; + + scx_ops_error("built-in idle tracking is disabled"); + return false; +} + #ifdef CONFIG_SMP #ifdef CONFIG_CPUMASK_OFFSTACK #define CL_ALIGNED_IF_ONSTACK @@ -508,10 +517,8 @@ __bpf_kfunc_start_defs(); __bpf_kfunc s32 scx_bpf_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, bool *is_idle) { - if (!static_branch_likely(&scx_builtin_idle_enabled)) { - scx_ops_error("built-in idle tracking is disabled"); + if (!check_builtin_idle_enabled()) goto prev_cpu; - } if (!scx_kf_allowed(SCX_KF_SELECT_CPU)) goto prev_cpu; @@ -533,10 +540,8 @@ __bpf_kfunc s32 scx_bpf_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, */ __bpf_kfunc const struct cpumask *scx_bpf_get_idle_cpumask(void) { - if (!static_branch_likely(&scx_builtin_idle_enabled)) { - scx_ops_error("built-in idle tracking is disabled"); + if (!check_builtin_idle_enabled()) return cpu_none_mask; - } #ifdef CONFIG_SMP return idle_masks.cpu; @@ -554,10 +559,8 @@ __bpf_kfunc const struct cpumask *scx_bpf_get_idle_cpumask(void) */ __bpf_kfunc const struct cpumask *scx_bpf_get_idle_smtmask(void) { - if (!static_branch_likely(&scx_builtin_idle_enabled)) { - scx_ops_error("built-in idle tracking is disabled"); + if (!check_builtin_idle_enabled()) return cpu_none_mask; - } #ifdef CONFIG_SMP if (sched_smt_active()) @@ -595,10 +598,8 @@ __bpf_kfunc void scx_bpf_put_idle_cpumask(const struct cpumask *idle_mask) */ __bpf_kfunc bool scx_bpf_test_and_clear_cpu_idle(s32 cpu) { - if (!static_branch_likely(&scx_builtin_idle_enabled)) { - scx_ops_error("built-in idle tracking is disabled"); + if (!check_builtin_idle_enabled()) return false; - } if (ops_cpu_valid(cpu, NULL)) return test_and_clear_cpu_idle(cpu); @@ -628,10 +629,8 @@ __bpf_kfunc bool scx_bpf_test_and_clear_cpu_idle(s32 cpu) __bpf_kfunc s32 scx_bpf_pick_idle_cpu(const struct cpumask *cpus_allowed, u64 flags) { - if (!static_branch_likely(&scx_builtin_idle_enabled)) { - scx_ops_error("built-in idle tracking is disabled"); + if (!check_builtin_idle_enabled()) return -EBUSY; - } return scx_pick_idle_cpu(cpus_allowed, flags); } From patchwork Fri Dec 20 15:11:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrea Righi X-Patchwork-Id: 13916900 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2084.outbound.protection.outlook.com [40.107.244.84]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CD04721A440; Fri, 20 Dec 2024 15:41:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.244.84 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734709298; cv=fail; b=Kw7zJbiwn4oONKWFdtNsVCMrox6PXFF0GW2aHnOC2zOI/HmIyHsRge58eYNXyyWyH4NXR6USWzQgWZuAEwGUo4oxRpcSdtxg4p7+YtIs34faOkSz3XMzUxVs3d2nfe/GeBeWwc2kTIt1+cb/U/WjhXCRM/M8myIq2IB1IMl1qdc= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734709298; c=relaxed/simple; bh=KxRf734fvK6ZCQI4T3agHnNBNKtAxK+yTuBXzUj6Ok0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=fKud2rQa0EySsoedbwtbC+e98fOMx3tTKEpHCaOck+a9i8a/EXl0YZec/pSKPenyLfkk46RplFEwNaDqPq2OXlK3XtzNllghxximZs3JMoKjwnZxsSpVqO5kl8r4IVW5mwDNuDasAE+y/DzXvhEjK7k50CBL24Yo5mZabSrgXpY= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=OP51xZhc; arc=fail smtp.client-ip=40.107.244.84 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="OP51xZhc" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=VrT9WlLgLQSmAE2P9uZtf8e58wLTVpt6DmqCjnnZl6ea3ImugK9vm2BZvmp+PqOPW/sjjitoFzDfvlI3R97gjpdGzN9p5thA25qCpQ4La9aab7CdV4VtLuVZBzml4a/0dL+PY40ukJ8bSVEv3Cc0wuYgb1lb5UddT/jndwYBQByGar2bEKec8XwCj8aDDV12l2rMCGkg4Qc1YdwJkVmUDvN8j6k2GqmFtA6GK2iHhhFoUhdeiOKK17rgV8PmyPktvJhuCw+BmXNFkOThdbfLpQDZhGg4YgAueGlOH9WQCVtuwXEEUJmR3HgNpZJYunlT4mWBtk96PET0k64sN7zdLg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=LCBF3M6WWXNz4Ul2aZjpCiD52jYPCmU4fJP70kG7xLE=; b=gD6/jSBGBAV4kqewYuUh+YQjUjJkK3qzMJSm5+M1yj2QTgKeshHAVTIgmqWkyI/kv/zH+Ed1T1cZ3Di7SDdb9IuOUr5qW6rpLQO+1unhONy2SBxe7+ekqcg0S6Ld5/gLje2Cd94BypPu1yM/A0ELJDz5h+H/NiTt5o7WInNLo+C26Nz2exAXgfbP7vGzvWLo0Cp9L2Tt7twKdYeYm0T8YcC77T0ZdikAh25MqqNggEIO4Er7b92SvLUCzMUxUQ0T0vrXAku/2wO/E8o6YyJTRp1JdTdvKL+yBIC1LKkrH9lxlt6P6KHzNT9B59fhi4H4djaMTxuIWjnHHDfYmeaBQQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=LCBF3M6WWXNz4Ul2aZjpCiD52jYPCmU4fJP70kG7xLE=; b=OP51xZhc3fXq0mklOetK4ctvV9xEka92GE/aRjOgVuyE38k/wzQ/ViTNuwpMqOqMjLvcNkMbsEYSHFkmMKZE3AxiPU0/DjJcUzxraBUYitYpV2XXJPE1E8a9Ox2SV4v/6WeN5YaKYQU7z3gCQElIPS/0oHSTN5ydrl0075ujhEPq3kuReA5H6qgxal2yWWt95VpsaQvRvN8k1k6V0YVB+BB7MWQgY5PpoMDeGNZ+8b+A8ddhzrBgYFvI5wwPEZbwZux5NbH63I3FhZv0cVDd0wBiYWOM9+EgsgUUHI1WFjEQ7LqwSLQ46BthqfK/+F9sX+pw6fh9fHWkfeqMY9mr1g== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) by SA1PR12MB7149.namprd12.prod.outlook.com (2603:10b6:806:29c::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8272.13; Fri, 20 Dec 2024 15:41:33 +0000 Received: from CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5]) by CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5%3]) with mapi id 15.20.8272.005; Fri, 20 Dec 2024 15:41:33 +0000 From: Andrea Righi To: Tejun Heo , David Vernet , Changwoo Min Cc: Yury Norov , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , bpf@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 04/10] sched_ext: idle: use assign_cpu() to update the idle cpumask Date: Fri, 20 Dec 2024 16:11:36 +0100 Message-ID: <20241220154107.287478-5-arighi@nvidia.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241220154107.287478-1-arighi@nvidia.com> References: <20241220154107.287478-1-arighi@nvidia.com> X-ClientProxiedBy: FR4P281CA0184.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:ca::19) To CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY5PR12MB6405:EE_|SA1PR12MB7149:EE_ X-MS-Office365-Filtering-Correlation-Id: 7589d7e6-fe88-47c8-5d77-08dd210cc7c0 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|366016|376014|7416014; X-Microsoft-Antispam-Message-Info: qsCe0AmByE1qvZKGKGLptHhW0PwkFH6L/Y/A5NtNTezapk4n9QqoIRch4EVjSsUHBzFk4VhXbYHAuxQEB+Z7ht8xbSjYw2uUIZY8wE/DU48tHX0ske/o+X2V4ZNRZ06iUy2I+uF8fXQHPMB0xeqj9qSWvOFnfVpeIXXC/RNxZ7YlB++lCPqmYwzhR5LJZq8PFEVR/c6AWwRl05r+JxGrmlOXBprpJfypZbJBjMQs0/FQKBfR84v94c8OD7VLFq2NL3Qom0FFw1UgAmbvy9W1c+vgWftTaGfTvDZ5rL3qZ76HVyG4A4nrSXWa9rlocGSFteC+idML12SwrOx/oYCrKw0TMZ8FvOMhtNxtvRyn8vW5ig9802lvc6B68QJ05C+isHqfd+/l+tJGkts8EtF/hshJvxawiKpcdYP+4wwHXUzlOaYlOMk3foCYTM3RefRLFIKAk/J9xiLmtcoplybWGgfgnyOX3g1FrJZtu2Ih8nWkWgHOeSyUkparrug+QymnpZ1xUf+UkwtqycJG/JTWPrX2b7giBt9RF19BcrOzgqaj4HRbsYAt4XP8G1q3511K4jrrMGU/SOgJzY7DyPFpmJQIoZtfqxk6j61EMFAXVfONUxdIwr8tJ3NKfv/RVptAT9FBwed7ha1li8R0YzuZrlOkSz5Jp1nccbdMhhp6ob22HNRj8kY/hlkg5kpDFp7u/96GD5UFE5V4mEm1U8jHI2GuLFIZpnHAmom7UucqZMmC2yPIutODspOo/VaGUCdjATrvfo6bcP75bpKTgYIAfIco9FWrZRYMcOVRPBbh1DE2Po7s2ssi5MUE7iUhXn+F4LMY8sMXe6U2Z3l2bOGRJ038QMIjsRogSI5ncx6o9H8apcNPXQW2o+ossJYh/KW4alahVz6a/WLpbdojkminDOoe60MBpeDpd2BbH13C/2jVu9aNxKXwriWa6Xrhmq3iMbL487VB1QYqorTu7xQ5L4h91wzGhbLD1erBfMKBLLF6JznlA3Djdq2wr3LVghbz4SEfyg2nH8R4RA+yziStSdDDDdpTZqV+OWbsU18a02xfAL0jJgt/xpAY16josCPjkyeToQ9y+PMo5XXXgnmsvfeGlugwW7M8WluLPZP5o0YFmmAobYuYHiJOwnHrFVAzCVdS4lMu9nOgF/McO4KyzbRYSni/0Rs51iqO6FkdnN9YXu6oIuPxkOQLIPGuKnCjUcAPy7nm/Dfi7gUQkhwV+J2dEV44mxP5rcxYqtVdRY/UIuzpPb7gS+fTrSSvaTLxecGd1yqB2G35UgUlx2mV5gd0KePsT978EohfUIkjFbnELQIBrqzL9LWAswWezJv2lOwdKHwQN7r1U+PRBg8JwzwwEAIRklk7V5UL332Br9wD0u2C4cjQUR/0/5cAjuvO X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CY5PR12MB6405.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(366016)(376014)(7416014);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 1FZ6Z3KBGTETWqMS6MEqLtKStLhqDsNSx4dAA4NcOAvaL48cz4CrRaYVenfAnR1XX7BzKd1QVN4CURDQa6p8udQQXsVkGoIspH6HSxrJmLo3l+VhyUihYX5EsA/jBV0PHhLEkoDkqffHSUD6bx4sF0YB8bubRqwDQu7KvH4Speu+swVT0BZCHTFbfylWoi8falq6o7HQy6Lnearq6O091v1cNeVDO+GV54w7JbG60pIzxaE9MOW2t3kW16jz4/yaq9i6dH35UdfP2UsjFtQqTolNKnkk9wcrOLS0k7xnZoDGdXNWKPbI9CfHq9dr7u/cKtfJozBO3K6Rz0CA6bJpC0tlLVA/fNmLDR7kToDUnrSMkuhrxR9vOVvuR9RIvTLbbFlmhexPab7hjAyyZw8crhkTfUnK17+VZpQ6zHdI4K9tUEJ+y17R0F1vPLRkizWOjYFh1RGgtA598QignJOMuGvz2gWD782Ya7s2OajwWrpjn3I2S5mE1UQqW+xsCWVqatxQZ+JAImH3U/Nn5rgn1tAfD4gjU/jylVxpOeaKc+4v+9ENGin44M0BmNLX5e9pl9A1RIyvPJHdIlAwX3OCnXexCv5E7kyN9NkQPLX0XIg34EsGLCRsIXGpg5HCrAOJSbymavlW2D+CcIe40pOaI69fFYHlSCKCkj+U7+z6hUrMS3RS5q6zycf5PcLFjDi1HLWCKZcC4v7ZMNQBUO75fqhYzhFdXVb7YowhBWmsA319txeY0CyASQ7A9WV0Y5kvwChEvkoxhNg+aseWm+3CqihHSStnfyk5vZLw9URSGnpW9kKEdr4AIAGM2wrpoiQKweGoNMmMEaDtBidtlUayPJCiWprdeAQSTZ2XvFnb3v6kNmzSRniivrSFL+l19DoF0KvNCIegT2zC/lrGRscCQOmCP/J3L3HNoU5+Gy+g8N2Cr4n9EoriY0RRri6IKR6POwzts8AzwbRea2Pv2yUMkOaVBxyQg3VRah6cVl/sn0ri9ASQb387Z0BoQyv5MTTcahgM3WrGuEIEEatuYe6jmV7XBDMUKsa0xcipPYm/AHE1wVIrLnkQaS/Ttq1inyAHcPulh0R6mE7KzKQfGpNuViFetqQo7wucLk1X4fwasKRWJ1ZFOFYTAl3m96K6uQQKaIF3TAirCpIVsDF6kZBFX6xV9H1PmyBOkfmLr6AhE5iA/saxBlZWO22ZEfzyYx6yX7+9uOZsgmf4kXaVlkVbhxVL1vD/mpmSJYQmLCJb4uKihaqnDgV5YhveDByn5pwYN7U34xKMOtVs2Qe5jtr3mTzrU/wFcpT+qy7K8XTO30bVSEODwhJpDZQVxKkLttZxISx5dbD9JCgrAJAJcR4/Zb/wVXIADuqFju6gL1OddMGZ6x7//zHwHwR2zY7MVsJoC6WIWSMxS1mECj4EKuEtcgZ8YZYfguQ2TjyOXMwffjVQIm8yaJBb1SzWsKrWbPdqJMw4aGSIwyzZt2wc6Fw63m4utbuDro8v5NuLcuEDvh2SGQjvLemHryQDD3FydMVNjipXHluHT4lov8QM0+6GQfhm0hrwMuK2KXDvXLvyHfb/WLa2d5SgEq0RpYJ+VrQd X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 7589d7e6-fe88-47c8-5d77-08dd210cc7c0 X-MS-Exchange-CrossTenant-AuthSource: CY5PR12MB6405.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Dec 2024 15:41:33.1994 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: s6w4SSc/mgfm8Xa5u8/Z0I7YWBk61xa3PBjWXKTMdBwxUf0IHRa+5uSZ0MZlnsgN5cNMB5v2LCfveNqZITfjrw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB7149 Use the assign_cpu() helper to set or clear the CPU in the idle mask, based on the idle condition. Cc: Yury Norov Signed-off-by: Andrea Righi --- kernel/sched/ext_idle.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/kernel/sched/ext_idle.c b/kernel/sched/ext_idle.c index 0e57830072e4..dedd39febc88 100644 --- a/kernel/sched/ext_idle.c +++ b/kernel/sched/ext_idle.c @@ -460,10 +460,7 @@ void __scx_update_idle(struct rq *rq, bool idle) return; } - if (idle) - cpumask_set_cpu(cpu, idle_masks.cpu); - else - cpumask_clear_cpu(cpu, idle_masks.cpu); + assign_cpu(cpu, idle_masks.cpu, idle); #ifdef CONFIG_SCHED_SMT if (sched_smt_active()) { From patchwork Fri Dec 20 15:11:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrea Righi X-Patchwork-Id: 13916901 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2050.outbound.protection.outlook.com [40.107.244.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7E28C21A953; Fri, 20 Dec 2024 15:41:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.244.50 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734709301; cv=fail; b=umJdk/x+mcCuz7vWiFuD6IrUkfhb8lnfV/zrr8v5eRK/32rL4Bp4iXOovTDMM4NG25C3Uby/agXuTjtbwDZBwDjajCnu4YvCuetpt+HCFFrCeId7VMC+ANALD0SG9hUfIHfvW7XubFMIrU2FxHsLanqFAyeJ/1Rmo8YvxIFPVSc= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734709301; c=relaxed/simple; bh=+yzNxU5hWGYrNTloY1OLbYTTzkPrAhIlK6icyAUQFzY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=hq60LI605ML/OpImMj/MHB9ua0El0jZ08BGTLbdBvjttAHsRoBxWmxNVslgbCIc24mh8iOPvv2hP3nvQ5HyhC4hfuNzH+8L0WdvDbSjKJsSna4BQUbfHeMxKtqnhHotWKkC1io4t+HplziQJ0CNUJIb+PePgf80qlFAVEvvjA7k= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=GQnNt9uD; arc=fail smtp.client-ip=40.107.244.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="GQnNt9uD" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=MplauQk9AHfmr9AcegfAuI18PXCmj/a/kFq/jMvXu3lOwKlAUlDVGabnwI3I9QW1NvaNf0geG8VwweoeRIZAWA2KhKb2jbFpZ+KTU05e7PHtyVG3ZXS4XzjCEiWkQCIMEVVBOkKJuvEQvmz4JhImddfugey6AhuBKsI1nJRAakINBCJ5J7bdof3VUMdcEtGC0bv1ws5zIFLfmfMSkU4R4ZhBXsURnw9kqvK0urzkj16temHMq6Pf+kKUNUf71HeHP7d/C7yJeKyvtH7i9Mjjeqhnnfz7aIXNn8Q2axZ43oaGzwCXpWqZmvwPqDE2h0oVuMLam6v5KNE7PNhLNMhK/g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=YUw3coJ170tiKppqaO3qmAbuKUSzWHNxy7WYwCgYkPM=; b=rU0sJ1Ml5pCCJIZz7UTGs1a3w7gvztKTyHR/24NVbN5OcNJEivyU8Ed4ZHuXXo0HwLT+3IrKLVhEzdXXp9V28ShabbAG+mqj+y8mE1iMc82LIUspc/XBRIqcuard0bPYbxuZtyT4WhuugQaJLb8d2kIqKsdfuYARicHfezPXqVNxZGQx22iG6oNmFl84qiW2IGNgrY3xNE7t4RhSyAKDPlXi2Fp5nECIzzT15X75wEifaSEFTdCYQZ1hJJs8rDUZOGnO/g5YLwgfGa0e2ADrUt4V1D6Qnn1TjCSdGUYSsgomBeTz9q4Jks9yFTH7GbAZ05gCPsGrXXZ5Z51eYIv7zQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=YUw3coJ170tiKppqaO3qmAbuKUSzWHNxy7WYwCgYkPM=; b=GQnNt9uDZPqCmF6V6xA5JECe0PyQcyclzdRFUN+yQ4pgyEsmwhYI0TfLoTciNCMwdJ68DQ8Q8wsngJ0IODp63EDhhq+btmXqWCeziOTSOFyunXysv28ZkzLtUgkeQiEhpTn10KBw2iTcrMsKHmv7d3Z6EicxgVKsMWeQ4nD8/edErgdq6xk5ePVw/lahOQTDvN4edPqV+b2VpsLUoSH0c8aOwsn9lSwMUdiBopQZYZmgfX6dsPeV2AJxGqzbN3vWSHe7w1w/vuScDqq2WLHWyvwUgKNyphe7lzEIufc2pf6ZOniUt7pvcJryIQzy8fkQDCz9/XlcJ8aK1Duu6KTIAQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) by SA1PR12MB7149.namprd12.prod.outlook.com (2603:10b6:806:29c::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8272.13; Fri, 20 Dec 2024 15:41:37 +0000 Received: from CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5]) by CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5%3]) with mapi id 15.20.8272.005; Fri, 20 Dec 2024 15:41:37 +0000 From: Andrea Righi To: Tejun Heo , David Vernet , Changwoo Min Cc: Yury Norov , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , bpf@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 05/10] sched_ext: idle: clarify comments Date: Fri, 20 Dec 2024 16:11:37 +0100 Message-ID: <20241220154107.287478-6-arighi@nvidia.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241220154107.287478-1-arighi@nvidia.com> References: <20241220154107.287478-1-arighi@nvidia.com> X-ClientProxiedBy: FR3P281CA0103.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:a1::19) To CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY5PR12MB6405:EE_|SA1PR12MB7149:EE_ X-MS-Office365-Filtering-Correlation-Id: 0edb63ec-53e4-4638-2157-08dd210cca40 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|366016|376014|7416014; X-Microsoft-Antispam-Message-Info: 6RFiVLLnUrE87pyAC4lsncXVXVx4RuKuazTnj4FEU0F+9VWpHPhAXXh7iTz/h21tuYFSq9U0fcvHt0XmMp/Nri4TGkzhJaFMQDKhCfyBMKcn5WpwE5ZUmnq5gVvR/3UYqIq1FbCZRWE8Xv4AwYei7NNVm43Uzn0qVgbRk5tL/SVofT9zSPOCXdZ22tiXZ46Efs6JeV/tpLpA60WrwmC6DBH6fWJB+antTPSQ4BTAZEHU0KXoCcCuUY5rzP8qF+PHwPmznjEfTrrqgNkDmRzu1aXiAHhmizeibIGAD6f+z8tOqUOOK03fYugHcVt5CQ6ZX5g2Ur75cU2oM3fC9FUM6wTSonlZkgvCvAU4e9iR/fA5txHfciDwrKRCk/BCj9CPdMicBEBaYnYnSl1F3myRfNi5tVl65CailhssSMD7hoK0+v8WtSwUnGErCSK+5rmF2vlkV7+FTMv5X5PXc615ZRfWBfYw4XMY3oPA/In2/rEfSoRraeIyZ2YS5XkULjqCL1M7hGSf6aGmylvsWvjJBoBWX0MNvENpUg7jE6vM1uaCeLxnk/shntp5MHyUmvBgN0uKaZofDk35lUrKCZ6VhdQwgT0psN9J/QxU7nk2gkU7FQkXVkU+ZEpVBBpi4ognGi1d3qUyA1/89DHFS7WJTZAn8bagQbcufvxWIjK40yR8mXzrKlzephvf0tUtHxRvqvS5M5COJm+GtGZ0d61v5HaGjVQGFyRVChRsxzceVitW4bt/o3i0f3jt9VXsvwMY5o+6e7F/wDUnm/VWQI5XWKM4HSSCKH5RkkXOPYKtkHnNAbtMiVEWZsW3Es0t6vVZjOv8yfT+CEkU/KdPffd68wQzPW9pZmZqUY3PEuqftQXXsLnVL2rxQ2T8cYLhNjRxGItOCNh8xcnyjCJ3FpNKijrFhIUZW+rcijV7AYN0rIwUY0SjYaIE0jFkSwOhn4sAu81M7SwNjaPYScZ0PvRXSwjYjBctleO2XHSmoU4BPLofnslacimP11J0bpXtKfPmdopVIhpw51sGTVi1KVh8Ii6ZzqiHThV1l/NsH7AEg2Ceb1S4yMojGYpXUX2juNpazZZCywRLge6X3tVTnkyWxF1V45QhIn/Ty64cP4AjbTnzNDhQvrbuvMdd/IFSA8t77BwC4G0wEuW4xQxTDthrG1ics6AWR9Xi1WPs6dXNXqYjtmS68y10e6obXCBKoD6shchuQsnrKDyqikqKAQ5HTMK7S5tmrArxExwPBE9Elqv1SBnBSLeHO5/8eQVFFaDpgawEFNyjRBSNXR2J1FQZDViQ8dwYtxxoRGngZ4H61B5APwjEDjCy4ZHN00HeB3Qx0NINj/bymg2XGPfKBUme5/mT2/xpnFl+ZjZvN6b+WLpMaDv0b6K39RdXBEA2fgiU X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CY5PR12MB6405.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(366016)(376014)(7416014);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: wc3d0dUEwRTyydz0Ttpz2wVZ//TGSCmoqZR5WqSLwTpSHUAJiAneNWHsTxoIafmh6Y4u5Bdpp2RO81jSPs3k8if5yUf7YpgWaO9Q1IDZTnlMKM62cvjv6oDI2WoBF1+KN85g9SpVw5k0GxR2hjcRy+MtQNzdVO/6xDh+7ebZcQ3gLZH54KTVoXaBG38Wh4mMOhU5CztM0K8Cd+R+m2u0ZtmBvEM80ThWdLMasCNFtGX4zllqywW716ttlb0IvO7K1BfTki7EJftxKnyFvfmjzhH/03CPfxSLHEzugbR9XQ8i7/kU6vdiclPbaIH6M38ekZ19V+aKtEteQzQNB6Bh3VW8x6/TEWE+jCQaeAxGWngC4FHn9MX55hfbJWuXZAcuW8SEsalrM10lbgZqSsd98VWh/6INRiuSp+JtLn7Pq9jXR1XyZHhxU8Nqcs7CaKtvLVnYmcglfJ0ZkeBOIp9N4++JfZJNG0+8hFZ9vYS8Q/W4NZ157sj6YdDKvBPa5x0Xpm/RlY3OKKnZW54zdOEndUHyJP39zBleoOriEwitTFMGDhsV/76toOZGArByEd3XZRuhnrRMbfAqOuOd1i0hPvp7QLb6LzSkUvLSoWwzslBmhLz45M4lmrO+34Q0fr8kZQxsKlKJ1cPQL2zZfJ94TM21U2ynnC7R4q25MvIdmhNVA8l4Y0WVA6HcTUiuiBMlYRQCDN4XhYNOwfHhd15EwLApSSK+/0YY7QMm7SGvIhaJ0xeN3pY+n98EbHJIag9Na7MQ4UMg3+DjKoUR7aJ7uydZK522ctho8TBck7IzxkkaqOIsK2kigeJROteFPejdf9tQX5BV6elPp8x6doKC5oWSQuucwVcfRJCGVmfkOK1hhJLX2qUKFh31Uc1Jbf0ja7XDTLBCykVrKMBFcis3sKYu3cFRYyGtC8zsvQxZ6bgi8wbgFM3ww8IA5CkIYxtiddtjmclEBbcPqYsr/x1R7asvL05dnf0s/pjWLCGp+gXhBhH5GxPsw/K05YF2jkoNo74VA3MHnCrJaxGYZs3nrZtbXxFRDVyO5WypD2dhXteF2VqFuol7P4xhFFO2m9hLPezny0iSsVLhpzQaeJ83VJn0hJcxb0Qu3r30HT+OpF8YLSkb+GsX7LKpskRYp9pCcjl/UiUqNnLtJ9iY68bPSFB1cOq43t5ObGektvgmtjiuPHTyiOs685u5wNKdUFR2x4Rg3PjhbhnHx0a0zNCEAOqYiPqfRjFz9gaM3z4UJlUCRm76yduPqya9XmWbzN2+3Dw27+k5MJoKJ2ZYrzKU+nnlC02RgpQG0x0TWpyr9GfO0Etm8X7UspVwQJccOY2N87B08pkk51q2g9oeFHiknA6SMhBg8nubEs4DEUSyI+6lZ7Axf4knCwRYf7Ccmzu3j4OSjFh2YAp6VGJnNBBKDim9KM1/tWTNJ0XRznzpaMg6m3kIv+GSNuoMJ9PkmPOdxRCkex98XaDpvpug8JdiRN883ocFpIzr7ZD87cNH+4mgPwVb+6GOOkm0MpkZt/L66Ab1IPUYCFig7zDNV19Z+Q0Lyt1elQjUS0YM6ga+uC67wnjSqqIxBsZiShf/c6rB X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 0edb63ec-53e4-4638-2157-08dd210cca40 X-MS-Exchange-CrossTenant-AuthSource: CY5PR12MB6405.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Dec 2024 15:41:37.3991 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Exg385wsr/xPuYtQ2oKQlCaxAFkTjUZnLh6gXYcn/nLu/9pYKfWOG1C/7rQJR+w+EJKtyxnNthPSugB7r/cWtQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB7149 Add a comments to clarify about the usage of cpumask_intersects(). Moreover, update scx_select_cpu_dfl() description clarifying that the final step of the idle selection logic involves searching for any idle CPU in the system that the task can use. Signed-off-by: Andrea Righi --- kernel/sched/ext_idle.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/kernel/sched/ext_idle.c b/kernel/sched/ext_idle.c index dedd39febc88..4952e2793304 100644 --- a/kernel/sched/ext_idle.c +++ b/kernel/sched/ext_idle.c @@ -52,6 +52,10 @@ static bool test_and_clear_cpu_idle(int cpu) * scx_pick_idle_cpu() can get caught in an infinite loop as * @cpu is never cleared from idle_masks.smt. Ensure that @cpu * is eventually cleared. + * + * NOTE: Use cpumask_intersects() and cpumask_test_cpu() to + * reduce memory writes, which may help alleviate cache + * coherence pressure. */ if (cpumask_intersects(smt, idle_masks.smt)) cpumask_andnot(idle_masks.smt, idle_masks.smt, smt); @@ -280,6 +284,8 @@ static void update_selcpu_topology(void) * 4. Pick a CPU within the same NUMA node, if enabled: * - choose a CPU from the same NUMA node to reduce memory access latency. * + * 5. Pick any idle CPU usable by the task. + * * Step 3 and 4 are performed only if the system has, respectively, multiple * LLC domains / multiple NUMA nodes (see scx_selcpu_topo_llc and * scx_selcpu_topo_numa). From patchwork Fri Dec 20 15:11:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrea Righi X-Patchwork-Id: 13916902 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2072.outbound.protection.outlook.com [40.107.237.72]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7032821C178; Fri, 20 Dec 2024 15:41:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.237.72 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734709305; cv=fail; b=exNKw6YPsS7mLMEukzBBNpWPGDIufB//fzUCSk03tbxBxy63XbxYwy/CN2mXwt7KatvC+Ep0J/oUW9ePvV5OZByqmwknK4iWFfAJtajM4RuCsY6Tpd0K2QpJehL0u+dZ+r5e/ntSPH8p7jvHBP8ylkqvwujWZIKVvRXP3kdfQGc= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734709305; c=relaxed/simple; bh=S6Zmfb6gpHI2teqnPISLSr5irpWKYo7zBNPrWt9qUCU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=LtBE2bSh6r7ErWuGsvRjOJMr2CSVIwzEVqvVIFZNQqmaNFs+ihey+p7e0l/3t6qOOogL3WovgKuT5z3CKOlSBnNNSB3+8L4uTkvuD4FXmaG1qKof+3mq5RkLA50oSKeCcaCeGFY/RSQ9SoVMKdNSzGxboyZ6dlhvzKZPJXfubow= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=hqK6/nj7; arc=fail smtp.client-ip=40.107.237.72 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="hqK6/nj7" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=g0NPtggjCH3VmUQKRvLSiZ7OIuc4fsnv0OK3h30TWzYK7sB9M6jZSC78sketB7Br8bMBpeEMn5NZKJzXUR+OERLBHf/IotTLRlOdDwFrmXP95L1xziD9YRexLNoof+8K3kWnZ+2/0fFNk0SXYo+uKqkBpnARWXSGVCndXw+CYRaKu1z/B6JMSaCvPTll86aqSi140490Qy9abZym5BtFuHdP4cSs/JTCHLlu5IFOnYQtn50Ncy/sd2On0ldCPoFxp5uc/FMZe01Y2f6ThnMUjSeup8ZjgjhKIMVV+BHDTPmirRjdmpMf4pzUv4fZfqgaH90zMLOypO9IS8iptfAT5w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=JwGhBsKlpTx7ZGqvODEf/JNDaQwKZ74m5/Q3haVKAsI=; b=K1pijaeeXSwSu9U15RfoYA5WKWwbDbwcUj/ZfpmIbZS71xl/L7aawJ4Z6HrkTNeiBRqnqk+XscymGziue6xQuBxBD1fWeNWLi9PYuBzCT+ezs7lwSOVEZlF/OvmBTSIsjAtRrTNWnrO0NeAXm7pmBc5UlvTDnChYHHRC2I70TIUFA5p+Vt4lvdPSaG7ABRAHiJ1HxA0owHkGtrziC2jNiTB+qPs81go3a+ODP9unQoN0Q+OczNcSLpTahCATtV0uwFKa+aq6DEBOe5HR/8DBE6cNd2hKRUMwRkB66eWq3hp1gEJJxGcRSLCOenm4leHF2bF53gkre1bHsKdpWqJVjQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JwGhBsKlpTx7ZGqvODEf/JNDaQwKZ74m5/Q3haVKAsI=; b=hqK6/nj7h4sHWgaoXD42yjtVSWRrHivUrQAsWykIGnH823snpFtsF5dDerjb+z+0OemTP5fTITku+Tqh70Fok9v59wPg5K8FYP3EE5a2YNxO1r+JHjdSc0UCeB+zjKyGlaC+xSn7G+YV4o4+GhoKcXFyJlsew0WSgvSdU4IUx4Z6FLCPJ1j938oSeQwVbgsoW1QDzckYxrGRQY8WHtv8I32+hGWuxBjWkAOJ4JIdgRnwTXkVZAcQv+SvaI8gMVVqKwRsn0rZ2z9pu733rAqpILY6SeF4S7LfV8CyKlKuJ9tGpErGRIJg14/TRDYrXD/2HiQ11Gv7R6uymUYdLtC9qA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) by SA1PR12MB7149.namprd12.prod.outlook.com (2603:10b6:806:29c::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8272.13; Fri, 20 Dec 2024 15:41:41 +0000 Received: from CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5]) by CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5%3]) with mapi id 15.20.8272.005; Fri, 20 Dec 2024 15:41:41 +0000 From: Andrea Righi To: Tejun Heo , David Vernet , Changwoo Min Cc: Yury Norov , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , bpf@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 06/10] sched_ext: Introduce SCX_OPS_NODE_BUILTIN_IDLE Date: Fri, 20 Dec 2024 16:11:38 +0100 Message-ID: <20241220154107.287478-7-arighi@nvidia.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241220154107.287478-1-arighi@nvidia.com> References: <20241220154107.287478-1-arighi@nvidia.com> X-ClientProxiedBy: FR5P281CA0057.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:f0::15) To CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY5PR12MB6405:EE_|SA1PR12MB7149:EE_ X-MS-Office365-Filtering-Correlation-Id: 619af6a4-5f2b-4176-557c-08dd210cccc8 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|366016|376014|7416014; X-Microsoft-Antispam-Message-Info: ZWaUteLDkwE1CmjwaU74wqQWUJLVaHT2a8LIhu4ZUy2chvEXhRe40DBr2dBjeKs7/rL+Rbwd/afXUbck+CMNy//VuQuYzrFHkJmHYh42bQqci85vUyMrmAg6RuCW5Ab+6pa4OyxShYr2ayG/S9TcNZxLUOGcTynVTvYuJ1QXPvEoukaRSUmeAQkTuK+58YnY+XIDyhS/fxW0b2svZglmJwnhahYU+zLCnwNtfFJk1+I81kPEkXD2H6HJFaszC/zCNBTmlksn433VmnHffCOXJ1VEmRuJKukEeVfI+0WaB38iYR98K7kYEZuL00Q/poC39GRNQcxB34q+a9ahdj6QcVuXKvLJUkYE5De5Nq3AcT937/0MssyXXV4rWyYj53uFuhKfJ5zcsOIeBZ2YjlMjFqwX3vbK8sZSZSlxme4QInf7vnXhDse54PmkggwfxjGcxMkhpoLlRut0PfU73KatCl/bI36aykg2hUUfEzK9ryg0okYv+0aDO+E/TwIn/lFacsbtiSXuF1sglJfLwzByt3OLJRSWhHqcT8UrVQZm1lOF35LKxvcd9uzGAxt9kVp6Hpu9VscLrCPGn2vwaoJMJk+BerT2oAv+1KIUIWa0RpzP+KZOPzphKMfTD28XkNXjBOmWcIHhSWQ3H++IL6yy8Xw36aAXctV1YBqyndGzw83AFRKZSjJom6caNsQOPF/5MYp30RcJC8FsTYxdNjftROhLwcNUEfE5LqQ6reZlSLuW1hRiktmLqUe04bxZNA7Y7/p0l+wrAxH6RkuIX89//i+3LIIzyAMX+WGAZB98Hr802Yty+h1ppv4lQWuDPxBo/W1MTio7cgMeU/KDPmt3mako5+u2XqXv6Hzn24MZPhUg2s5L0AuyegkwgTGY78XEaeBsVGxCvy+8t7kOSq9z/yWP6mWe2T+Nme+vpLe5ojVZleDfmTy60KOa4Rjc5ZW2ze6wNK5G3FzUrx8OYGNgvUKiCik8C6Bp4Jg+K6/yYjCiznPyL6YkEw0BWT18u/pv1eYnaDKPwQg3r9azMlWYQlveiOOQY/jy6zPJG0wVv0iBrWohWBbaZxm6K/DW4IWG1j/99id8r0rf66XMti5unJVAURnWecdEUlZHAurOWJ/SrRZbjaBP4cU9ByqsGn/WE3sG3x4qK9dG3wrFNTeTNCtWIJeJWqQcEM2Bi/HQc7R7BS3tXCfrrAlJCzgyuY07JuDSmWTUCyAzqZ4R4WuhMxFkZulbVIMSO7HZVXfE8sO4xLgyKni2bsVKpPY7BRB5UgpxjFevZXBkqMlkWiNavHgtVnewv1gx5OyNBf9OvKywNlCnLrsMtHwQkFJtbu9Ii54K4ttdc2TCk9eF1UI8J69w8ppnfRkHeKNcdTDsUrkLD6jtc7sSOmiZsPe+ZcCM X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CY5PR12MB6405.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(366016)(376014)(7416014);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: bf0osG43fzlnp+TjflssIE6LYdGLUcWSWhIZz/p+44GM0++7+orEYohiI0qomOKU0dDT6dMvJ6AGiacQtWNqVYf3twV94uFL3SSQqvgEYdXum39YvwM+X/AWiy0KBITfnBRkaFfJSGEYEmN2yEC5q9o5rhRR9Pb+lVLE1vyL2trt3NV4UFRXysLwxbpB6F15vcUh5dSgkHf19p5nq6uYwpnuKFVTaYiV1+rcv6uTP0CDqWEIhGECA4sDge49/4Hjz/3eOBBlU5ZxXbHx4YX2YEEpaMRDF7QAk5eXrxtVZajPjB/wC93ehJkRRwHIKe9ae5KOfemVh8dUQIruCtUP1clG+BdwBoFPokWRxSHiZVKpxNJxYbkYUL+oyulpaS06YDavoOHPQFAD93YZkgIAs4xyCTRPC8VtZS0o5fP3NRk6veh9Nbn00GAOBRrrr0XdLPdpsrxl0fep2VxCwVa7GhkCaW7MiaWjEljwChCyLiib/BP/K8fR0tT1QntvgR52vEuqRPV3ncpN1s9F5Whrh2jYDC4fWcUDZSHIMSdE1sv9MfOIaB/kG7PeK7DR4X6r3xI3qndHyYFI/QYJNmy2UpQo4JHXO+TMOm2RLY3I1kqdxFhhkHwmc6fa2hpfPYr6l5lDtA6lorEHrSZddELpwg0b9w1lPlLzSvlT+BSvQ1pnAqor/8Gq1lqF9MU18CDoOxiAkT4Mqo2SPqqH3bFxnyjsQJ8uYGW8Na/I7cPXt/jCGotk/mVC+9VevPtr1QVAcAbLdwWtrR/58eMEBw9Evbby+9RebrXQpNkJXaqQ7+jrNkhUU0vXdWjNRTN8kpoOW/SaxY/KxluNAwMu9QDIG4fSNF3uSFm8xdlGoaCmPHButHKvNPqoTSrwJEc3udiuUMLmbt3bQ+af5zndtfXzBUGnChhairPCKud06/bawzhSkPnoGU5nT1sr58xEecfcNF8yFYlk+ckyeBovF2lrUvM9MbQY8l08KGoBrklnUsWoP/iU+ywpsv6hikCFrf1O/v3hu28WY2vAEOYCXgYpOy5FAro+X5qRpRgdnVbUN3EeIE9vt9ylMqlMUrh2EWkYWlVVpOBe1l71TJRhK8ucbtASl5MmjpzM0FIRnbCRIMgc4RcDy7uwo5ECVGEsHKRL206+dZEcRb81AyQ1AdJCzuws/f0Z83NQFdk+W/8F0J5vIknNbBuEppIziY1VfaBZ1lDxt/Tv4ozzpru+BYvO9QPLOBH8g2sJkW6k/tNI0I3/xl0CoA7rSGNS0qgg7slEhHuK5FbJI1+F4DNtVEG7ifJ96dkVM33ZKMuNPnEUJiE2WGomjc2Eckb+8lBqH8sGzxJV+TCbV/HNtRqbQuhe9KkiLD5QplKf0VUwGLpVyo1vCxCC6DKCEpTQJof/8F5StQE/s6wdkRyjXNgI5KjhrHWpF/0cmqBpFSXe4e20EBs7BdmlY4n0DifKYad0345eb2dZ0rWskc7xSgD1EkMzK1p+SJYI/rwn7XmNlPBq61DzEcGYv/tvylqSKp0jH5GqKZDuFtlh91K9jHXWxdCMz6uw16sq60iq+5JcknFIcAMdusXMMpCl05RmYzDMRHq7 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 619af6a4-5f2b-4176-557c-08dd210cccc8 X-MS-Exchange-CrossTenant-AuthSource: CY5PR12MB6405.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Dec 2024 15:41:41.6557 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: taqc122yK+pBwqii/jTOm7f95a2fE0iL8LMtcZ738HGhq1gEUHcsw9Wz5v7r4aGDVbDpioq2nBg+d09FNH01NQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB7149 Add the new scheduler flag SCX_OPS_NODE_BUILTIN_IDLE, which allows each scx scheduler to select between using a global flat idle cpumask or multiple per-node cpumasks. Signed-off-by: Andrea Righi --- kernel/sched/ext.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c index 769e43fdea1e..148ec04d4a0a 100644 --- a/kernel/sched/ext.c +++ b/kernel/sched/ext.c @@ -124,6 +124,12 @@ enum scx_ops_flags { */ SCX_OPS_SWITCH_PARTIAL = 1LLU << 3, + /* + * If set, enable per-node idle cpumasks. If clear, use a single global + * flat idle cpumask. + */ + SCX_OPS_BUILTIN_IDLE_PER_NODE = 1LLU << 4, + /* * CPU cgroup support flags */ @@ -133,6 +139,7 @@ enum scx_ops_flags { SCX_OPS_ENQ_LAST | SCX_OPS_ENQ_EXITING | SCX_OPS_SWITCH_PARTIAL | + SCX_OPS_BUILTIN_IDLE_PER_NODE | SCX_OPS_HAS_CGROUP_WEIGHT, }; @@ -4974,6 +4981,16 @@ static int validate_ops(const struct sched_ext_ops *ops) return -EINVAL; } + /* + * SCX_OPS_BUILTIN_IDLE_PER_NODE requires built-in CPU idle + * selection policy to be enabled. + */ + if ((ops->flags & SCX_OPS_BUILTIN_IDLE_PER_NODE) && + (ops->update_idle && !(ops->flags & SCX_OPS_KEEP_BUILTIN_IDLE))) { + scx_ops_error("SCX_OPS_BUILTIN_IDLE_PER_NODE requires CPU idle selection enabled"); + return -EINVAL; + } + return 0; } From patchwork Fri Dec 20 15:11:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrea Righi X-Patchwork-Id: 13916903 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2085.outbound.protection.outlook.com [40.107.237.85]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D7926218E92; Fri, 20 Dec 2024 15:41:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.237.85 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734709311; cv=fail; b=EhImvicErvO9QBIFpFpu2nKcKFcnBlp7RZ9JrUeNDmoRiNlZPt/JsDbGSaul4feKKoGYqjdhBUH3z6Z6yZQv0c0wAiYBbFa7bX7GvN0DcHvo1p/K/xztj04rjCyCMk6egrML9AQX3Qs+J5FUZb6ntS6YdF2kWMiBy7lTPlGaNW4= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734709311; c=relaxed/simple; bh=uC3p7KjO0G+PjrBaLp7k9ZAEqqZbITu0MhvbymU9r/Q=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=cJz2HdHPD9KAbhkAxz0XmOt1h6+n3bl4mSwRE20K0WK4cTk5AI29pyhwLeu+dMrxellmFIMIKgG+W8P4NMC1kYUzPWACgWX78/NSFXl8RG3K55d7fgaEzYSWy6rmDCOrXgUUafoiw9jJvKVDfborxmblFZ4eGZ6fJ1gndVuJ9s4= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=TZHzf1dc; arc=fail smtp.client-ip=40.107.237.85 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="TZHzf1dc" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=QWxLcDPxNRLda/9lGIEImHzJscX8g4i239iwgIdzyLiEand9vWwxn7gr4Vx7Ue8VxAfRnRRWXkMYfcPW1pnzuB8qxegpThwXuxS2+bgNtod6aoGjb/QxNRUfQyHoGc5fO1dRlj8otPcmU3xN+tNFkacQgqNbnneF4pu+POD1vZGr6rh/m6VYkpqpzByBzsIiAXjh0gioGYBgrY5IGZIUpMIghXop4rbhuyVAOtyHb7EgHtyeSj/bvGqSmvOBRLjnSNd6Y5Mkg+bpmB45UrqVVFGzF00Kd4l7OBrTbucLUg2JOi8AQ03XzgLnLkJHh4Sup9fLg8uLLjNf/rnYZY2GtA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ToXPRNuo+u60dYiobPFH7mXgY/Vc0hefTfWPMc6qNvo=; b=saRblrC18f8VP+hmv+fcP01xOQb2oOkpPyXbYjZJ7m68XXYseIGK7GOwvYuPljtHyPKPrQs2kpJOaVxGPY/GvoqJyJzyi4/N54qQ7O1tlAfx4AkAiOoumCvPqNUpnrYp3zVY7rxckwnQT1QpW3/Rm/j9YwUzZQa/OSRoWzBxMZEa956xLWS7W5x/9ogy5WtkU/QsBp4g+cf2f8MLkNONuIWCYaTIPblFVW7+fV/+OpGa1eNCoaNeLdLvgvUwqdLouyji4vfT4N25eXSu3Aca3U24piH7kknVrqLTaZqVzGuqWHm8NO3hJh2pFWkPPfXDCSa1MPLFgQhG/OYLkB44LQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ToXPRNuo+u60dYiobPFH7mXgY/Vc0hefTfWPMc6qNvo=; b=TZHzf1dcrDgFNGm9qX37pZJCqL3a3SYQmbjWhGzfDxJunhkW0Y1iKwWa4knuEePcxcZKu6OlGLrvs7wMfSZb9NOX+5Dx1JOrbqJdMXQOVBM4hagW7GpWqtYl3Z333fMay4Ggk+qljoKLvswEQt7jO6nDiwLd6L4VDWcKdCNzhKMkHl2nXvczMkq4l801dJuXdY+KlcXDmPBQDZPkFyYDnFKXxz4qM+TMUWoR4IrSoh2Mhku//N2XqMOtkDiywDbGnK2eRJHVikAiijbqQROns2Mr1Rl1o3wL8vq6S/sUEqme3J0GLU+XaKNSe0stCOfV6GQZC61u5soh6uky+63DJQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) by SA1PR12MB7149.namprd12.prod.outlook.com (2603:10b6:806:29c::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8272.13; Fri, 20 Dec 2024 15:41:46 +0000 Received: from CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5]) by CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5%3]) with mapi id 15.20.8272.005; Fri, 20 Dec 2024 15:41:46 +0000 From: Andrea Righi To: Tejun Heo , David Vernet , Changwoo Min Cc: Yury Norov , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , bpf@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 07/10] sched_ext: Introduce per-node idle cpumasks Date: Fri, 20 Dec 2024 16:11:39 +0100 Message-ID: <20241220154107.287478-8-arighi@nvidia.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241220154107.287478-1-arighi@nvidia.com> References: <20241220154107.287478-1-arighi@nvidia.com> X-ClientProxiedBy: FR5P281CA0052.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:f0::11) To CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY5PR12MB6405:EE_|SA1PR12MB7149:EE_ X-MS-Office365-Filtering-Correlation-Id: 2002e1fb-37cb-43ad-30eb-08dd210ccf42 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|366016|376014|7416014; X-Microsoft-Antispam-Message-Info: kAnO1DlxyuGE5SdjbGc3L3PMy28uy6v32FO/TwmodIrbHnDrf81i+9Jt6qMgwFVb1CYxjTCpsd3XNtXYRwjtXcLPF+cTYEPvNXrbplDVURTg9nvSExwvPtxzys+EegYRw21MR8onoHsBBwDBoAKHWkACL0v7kzTYbNFa0J5POkvSqWhO449+6C+HfJMk5dz/3dMg8UG8dtKXDE0ln7/5SA4qz2h+ANuyTg8zkMko4o6tqJqrONQc7uadPqvHcFRtx1/lCBY+v8t4bU3yL5NztaPBLNuv6KVebJK+3R6ZbtF7zUkcfSYz8CNoHROMZ3Uq5oXWJE6y7bk61iHsKReVMzSWzPrJEKRT4SRDy1+9+cWvuuR1k6D0MVteVw2dd+PWAl/Ey9eW4mPcvpIWeuj+70732ZKVAp8zUclP8SCNMRapr/URSvQ3asMTu7WmC5dj09XDdCgxaDEIdhE8Yx/KFDpdb69kyRyf2RgZ0Poohulko1c0ipNAq9aAcY90gvdOXU5VXnG822x3dbYg2OPGGsgmRrGPzNVP2p0bDUFw9+W+bJ2+bFznXLdZw+IwzY7qHj5zs6WXMMBxNhtKirTEe7g7NPUj0xyWhui1xUzyTP/T/ZCRKpPNcogxCyGSkWBR+50gBQilO3Txub46eTWentVAfM1/iD+iBXd9qQUZMmopY087b/OkDUBoUgsbtIKURIvaEAQmAjOPoYE7z7dOtBCIoZ/wSiOdjjJNF4oNIO36vvnXliXgVzGoMU5Ek1ywTX1+n/CE2nlBV2tVH74Z647wOuqAeAm2f/nRdR+WP6UNV8gT/4qAyO1c6yooI7+keon0Pf2s//ITbRNt1Rq4Oa6kmrfAVISWR0nJ+GE45B/UEE/TO9KUUc6764aJyOj9aicWT1rzN0tPLAWDeFyn45HMMJS8vJwiUHYxcttCh9sCgqzvhZm318Gpjo3zRKm/8C9gJZhadWG2iJGSxUHBl4T5u7Qr72wABti49I+L447Tr5mZZko5tf+z5QcPY686FCsokNwlwocT5hbej23yIenCLOfNIBUZrYOE7Kdh8chMGpsEXtgSbvZBn8qjHRcuXMhGao+YunWjPNEi3yZGgn1mtUq+GK/zG2hPPH3jNShHJTthaNEwYeITxapVtLWAZ9Vq4ZbrIwX0Ii2RbEP6k/+0lBhn3uK0Rff/UEzK1qOYf0G/Vic9hXSCUAzzKbeFzDNGIkFSR9vnSI20I05439tnWkPCI+Ts1S8+J6NQFgI8HzKt1mTKen8K05aCckuy5czmeNq6A493GiKD9/CTdEiSsH2v7x7r4kRpNME12mx0kJFxngTOvLhZX1BGipt6e1gbrWMpjh8zy6IBvrwwGlWeNppyU2BseORFeY7XbW8wjQWoJ8ShmdlZ9fG0xgoM X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CY5PR12MB6405.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(366016)(376014)(7416014);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: AV51/DYRACQQnVZjUTeWH+MzJhCQfy+gutdH8lgRH58Jp2Rr7Lx1pFtigYx3V2hy5kZPAnXPIXrehmK1OOPUXXvO30m5I+zcVSzHKtWpVbA0iq8yBWsGJ9MumVyRQjoNFALcEMhwMDKCA3hmuVLvBYgW+qNrJHPq0X50iDybEpAgFDRf+c2J0dNlTLYm+z4XYlE9jzco0elENa4nT98cMBDGqCMsLdSapF2/PP46F3VQ8WNeB2dfRQZdBL7IvdwlMJ4cuVlwO0iEW/SXHKcZ90FC2Wfl9B1aVcLSL7c66NQym3wwyWXXCdi8H0iCQzU0ijIHE8d2Klb2hZSag2zzqSThe2IwsWmCsR8mLe6aT2dsNLhJBeXpJqVUDH1rjO0PSX9opP1LxufJJwXrsif1zglXsH1bWSZH/OhzVsb5nHbvv4vGkkvUPl4mnzvTqJO6dTKX5ICbbdtWz8jfL1wa1CqmBRXPmjB7e/U0CjL6AxMKyRbXyqqoNBVa0zFO67jbXzb1xk5Yz5s+dnhJrK1KTwMypmPxv8tSwTXBwdWK73Vqm8Lcaw2Ix/405/fDFREq5lZFy0k5v+n/y9fA+wW8O8yVMkmw/YGI8KDBHLaJasef1D7aZseBKrYXSqOgMAf6mhoBZJNZTmBUGbLw5VLoN9ZfD/2k5EjJIPVCNlN3jWcAOfSAfGDq83m9YjnCu8GUMZ2LyWWK6x7AHUo/jkpj6zTtdH1yUw1LpYGYKGmQ9d5dIb2L5r/0M0Gq50QEQWCdvU+e7FAa96m+GZQxYZPQ1x983ylpPnTZpgXBb0xFsSyWkbCDByjIwBNbhwM1Oiw8kYVhzXmj/s8IiUon8xRdrsAVl/WyMBYFW5EdSXXAMUuYnFtPSR+83NrdGs8Qc/Ybhe8/eoSr+JjxK+VQMfj9vXcN8UpC5550aiABK4oXHm2WujKUYWtQHt/Plc1WH2azkIZgmyV01p2DjKz40OsKU3AQniVySs6KwhOR28JHEuZcPetjvH96s7EPY/D+LHS2E2Q2+gGdyxYhVIi80uieLzsR7YQQB8rqC8H7ArxNlXUuA2hHKWgbWWN3Ox4SF0dXE0khwvmk2ahRCNaMOMTg+iYKCMiF9dzhsdxv7aOhVJ/ZkMgnJco7OAEu4qaxNHcQngCivXppPypq79zehB+XHFslTgbREz7Gcnf8QJE5lS1QzZiTTlVGCCtVyOTpZQwfti49kmagUkq+mVfaIeE3iyFZ/BkbRanldBw/SLnBqJ5zYghUDcULCWpvHPXkrXJe1sW0EpDFo1GT6GGXEn5MFrU7mfV1lEjAcnEWIkcKLHmuk8+q43+bNIEcKbfPRH/c6ZZwiklqZS3PIiG0ntBbwmTf8uRpxBktI3bJ9q8gVTOSeYE7EEO7ZdnYbo7Rldcca351g2DiH2/l4nXPTo3FmuVu4E3QlqWZHCDYoZAmbl1AfgvRKz5qWPPmPgTC4wpBAqEhVhEjg4wfHiinY5ADWF9RJH56S6i6VjRl5AXy/3CAY2swNIM0f+TF9EQxp7cqDNJSYgZM5uil4w4ZiI2JJAKoDXC7d1xzofiW/tRqic/m7tu7moEKWuSx4w6oLjQB X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 2002e1fb-37cb-43ad-30eb-08dd210ccf42 X-MS-Exchange-CrossTenant-AuthSource: CY5PR12MB6405.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Dec 2024 15:41:45.9398 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 92xhQEEyi8KcAuvk8ncecCl6xw1q4D7OF3zGnZTXpo67O6oOT0+kD7yCK2raGTDrMxVPM2ApNukfnrF6kARpGQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB7149 Using a single global idle mask can lead to inefficiencies and a lot of stress on the cache coherency protocol on large systems with multiple NUMA nodes, since all the CPUs can create a really intense read/write activity on the single global cpumask. Therefore, split the global cpumask into multiple per-NUMA node cpumasks to improve scalability and performance on large systems. The concept is that each cpumask will track only the idle CPUs within its corresponding NUMA node, treating CPUs in other NUMA nodes as busy. In this way concurrent access to the idle cpumask will be restricted within each NUMA node. NOTE: if a scheduler enables the per-node idle cpumasks (via SCX_OPS_BUILTIN_IDLE_PER_NODE), scx_bpf_get_idle_cpu/smtmask() will trigger an scx error, since there are no system-wide cpumasks. By default (when SCX_OPS_BUILTIN_IDLE_PER_NODE is not enabled), only the cpumask of node 0 is used as a single global flat CPU mask, maintaining the previous behavior. Signed-off-by: Andrea Righi --- kernel/sched/ext.c | 7 +- kernel/sched/ext_idle.c | 258 +++++++++++++++++++++++++++++++--------- 2 files changed, 208 insertions(+), 57 deletions(-) diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c index 148ec04d4a0a..143938e935f1 100644 --- a/kernel/sched/ext.c +++ b/kernel/sched/ext.c @@ -3228,7 +3228,7 @@ static void handle_hotplug(struct rq *rq, bool online) atomic_long_inc(&scx_hotplug_seq); if (scx_enabled()) - update_selcpu_topology(); + update_selcpu_topology(&scx_ops); if (online && SCX_HAS_OP(cpu_online)) SCX_CALL_OP(SCX_KF_UNLOCKED, cpu_online, cpu); @@ -5107,7 +5107,7 @@ static int scx_ops_enable(struct sched_ext_ops *ops, struct bpf_link *link) check_hotplug_seq(ops); #ifdef CONFIG_SMP - update_selcpu_topology(); + update_selcpu_topology(ops); #endif cpus_read_unlock(); @@ -5800,8 +5800,7 @@ void __init init_sched_ext_class(void) BUG_ON(rhashtable_init(&dsq_hash, &dsq_hash_params)); #ifdef CONFIG_SMP - BUG_ON(!alloc_cpumask_var(&idle_masks.cpu, GFP_KERNEL)); - BUG_ON(!alloc_cpumask_var(&idle_masks.smt, GFP_KERNEL)); + idle_masks_init(); #endif scx_kick_cpus_pnt_seqs = __alloc_percpu(sizeof(scx_kick_cpus_pnt_seqs[0]) * nr_cpu_ids, diff --git a/kernel/sched/ext_idle.c b/kernel/sched/ext_idle.c index 4952e2793304..444f2a15f1d4 100644 --- a/kernel/sched/ext_idle.c +++ b/kernel/sched/ext_idle.c @@ -10,7 +10,14 @@ * Copyright (c) 2024 Andrea Righi */ +/* + * If NUMA awareness is disabled consider only node 0 as a single global + * NUMA node. + */ +#define NUMA_FLAT_NODE 0 + static DEFINE_STATIC_KEY_FALSE(scx_builtin_idle_enabled); +static DEFINE_STATIC_KEY_FALSE(scx_builtin_idle_per_node); static bool check_builtin_idle_enabled(void) { @@ -22,22 +29,82 @@ static bool check_builtin_idle_enabled(void) } #ifdef CONFIG_SMP -#ifdef CONFIG_CPUMASK_OFFSTACK -#define CL_ALIGNED_IF_ONSTACK -#else -#define CL_ALIGNED_IF_ONSTACK __cacheline_aligned_in_smp -#endif - -static struct { +struct idle_cpumask { cpumask_var_t cpu; cpumask_var_t smt; -} idle_masks CL_ALIGNED_IF_ONSTACK; +}; + +/* + * cpumasks to track idle CPUs within each NUMA node. + * + * If SCX_OPS_BUILTIN_IDLE_PER_NODE is not specified, a single flat cpumask + * from node 0 is used to track all idle CPUs system-wide. + */ +static struct idle_cpumask **scx_idle_masks; + +static struct idle_cpumask *get_idle_mask(int node) +{ + if (node == NUMA_NO_NODE) + node = numa_node_id(); + else if (WARN_ON_ONCE(node < 0 || node >= nr_node_ids)) + return NULL; + return scx_idle_masks[node]; +} + +static struct cpumask *get_idle_cpumask(int node) +{ + struct idle_cpumask *mask = get_idle_mask(node); + + return mask ? mask->cpu : cpu_none_mask; +} + +static struct cpumask *get_idle_smtmask(int node) +{ + struct idle_cpumask *mask = get_idle_mask(node); + + return mask ? mask->smt : cpu_none_mask; +} + +static void idle_masks_init(void) +{ + int node; + + scx_idle_masks = kcalloc(num_possible_nodes(), sizeof(*scx_idle_masks), GFP_KERNEL); + BUG_ON(!scx_idle_masks); + + for_each_node_state(node, N_POSSIBLE) { + scx_idle_masks[node] = kzalloc_node(sizeof(**scx_idle_masks), GFP_KERNEL, node); + BUG_ON(!scx_idle_masks[node]); + + BUG_ON(!alloc_cpumask_var_node(&scx_idle_masks[node]->cpu, GFP_KERNEL, node)); + BUG_ON(!alloc_cpumask_var_node(&scx_idle_masks[node]->smt, GFP_KERNEL, node)); + } +} static DEFINE_STATIC_KEY_FALSE(scx_selcpu_topo_llc); static DEFINE_STATIC_KEY_FALSE(scx_selcpu_topo_numa); +/* + * Return the node id associated to a target idle CPU (used to determine + * the proper idle cpumask). + */ +static int idle_cpu_to_node(int cpu) +{ + int node; + + if (static_branch_maybe(CONFIG_NUMA, &scx_builtin_idle_per_node)) + node = cpu_to_node(cpu); + else + node = NUMA_FLAT_NODE; + + return node; +} + static bool test_and_clear_cpu_idle(int cpu) { + int node = idle_cpu_to_node(cpu); + struct cpumask *idle_cpus = get_idle_cpumask(node); + #ifdef CONFIG_SCHED_SMT /* * SMT mask should be cleared whether we can claim @cpu or not. The SMT @@ -46,33 +113,37 @@ static bool test_and_clear_cpu_idle(int cpu) */ if (sched_smt_active()) { const struct cpumask *smt = cpu_smt_mask(cpu); + struct cpumask *idle_smts = get_idle_smtmask(node); /* * If offline, @cpu is not its own sibling and * scx_pick_idle_cpu() can get caught in an infinite loop as - * @cpu is never cleared from idle_masks.smt. Ensure that @cpu - * is eventually cleared. + * @cpu is never cleared from the idle SMT mask. Ensure that + * @cpu is eventually cleared. * * NOTE: Use cpumask_intersects() and cpumask_test_cpu() to * reduce memory writes, which may help alleviate cache * coherence pressure. */ - if (cpumask_intersects(smt, idle_masks.smt)) - cpumask_andnot(idle_masks.smt, idle_masks.smt, smt); - else if (cpumask_test_cpu(cpu, idle_masks.smt)) - __cpumask_clear_cpu(cpu, idle_masks.smt); + if (cpumask_intersects(smt, idle_smts)) + cpumask_andnot(idle_smts, idle_smts, smt); + else if (cpumask_test_cpu(cpu, idle_smts)) + __cpumask_clear_cpu(cpu, idle_smts); } #endif - return cpumask_test_and_clear_cpu(cpu, idle_masks.cpu); + return cpumask_test_and_clear_cpu(cpu, idle_cpus); } -static s32 scx_pick_idle_cpu(const struct cpumask *cpus_allowed, u64 flags) +/* + * Pick an idle CPU in a specific NUMA node. + */ +static s32 pick_idle_cpu_from_node(const struct cpumask *cpus_allowed, int node, u64 flags) { int cpu; retry: if (sched_smt_active()) { - cpu = cpumask_any_and_distribute(idle_masks.smt, cpus_allowed); + cpu = cpumask_any_and_distribute(get_idle_smtmask(node), cpus_allowed); if (cpu < nr_cpu_ids) goto found; @@ -80,15 +151,57 @@ static s32 scx_pick_idle_cpu(const struct cpumask *cpus_allowed, u64 flags) return -EBUSY; } - cpu = cpumask_any_and_distribute(idle_masks.cpu, cpus_allowed); + cpu = cpumask_any_and_distribute(get_idle_cpumask(node), cpus_allowed); if (cpu >= nr_cpu_ids) return -EBUSY; found: if (test_and_clear_cpu_idle(cpu)) return cpu; - else - goto retry; + goto retry; +} + +/* + * Find the best idle CPU in the system, relative to @node. + * + * If @node is NUMA_NO_NODE, start from the current node. + */ +static s32 scx_pick_idle_cpu(const struct cpumask *cpus_allowed, int node, u64 flags) +{ + nodemask_t hop_nodes = NODE_MASK_NONE; + s32 cpu = -EBUSY; + + if (!static_branch_maybe(CONFIG_NUMA, &scx_builtin_idle_per_node)) + return pick_idle_cpu_from_node(cpus_allowed, NUMA_FLAT_NODE, flags); + + /* + * If a NUMA node was not specified, start with the current one. + */ + if (node == NUMA_NO_NODE) + node = numa_node_id(); + + /* + * Traverse all nodes in order of increasing distance, starting + * from prev_cpu's node. + * + * This loop is O(N^2), with N being the amount of NUMA nodes, + * which might be quite expensive in large NUMA systems. However, + * this complexity comes into play only when a scheduler enables + * SCX_OPS_BUILTIN_IDLE_PER_NODE and it's requesting an idle CPU + * without specifying a target NUMA node, so it shouldn't be a + * bottleneck is most cases. + * + * As a future optimization we may want to cache the list of hop + * nodes in a per-node array, instead of actually traversing them + * every time. + */ + for_each_numa_hop_node(n, node, hop_nodes, N_POSSIBLE) { + cpu = pick_idle_cpu_from_node(cpus_allowed, n, flags); + if (cpu >= 0) + break; + } + + return cpu; } /* @@ -208,7 +321,7 @@ static bool llc_numa_mismatch(void) * CPU belongs to a single LLC domain, and that each LLC domain is entirely * contained within a single NUMA node. */ -static void update_selcpu_topology(void) +static void update_selcpu_topology(struct sched_ext_ops *ops) { bool enable_llc = false, enable_numa = false; unsigned int nr_cpus; @@ -298,6 +411,7 @@ static s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, { const struct cpumask *llc_cpus = NULL; const struct cpumask *numa_cpus = NULL; + int node = idle_cpu_to_node(prev_cpu); s32 cpu; *found = false; @@ -355,9 +469,9 @@ static s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, * piled up on it even if there is an idle core elsewhere on * the system. */ - if (!cpumask_empty(idle_masks.cpu) && - !(current->flags & PF_EXITING) && - cpu_rq(cpu)->scx.local_dsq.nr == 0) { + if (!(current->flags & PF_EXITING) && + cpu_rq(cpu)->scx.local_dsq.nr == 0 && + !cpumask_empty(get_idle_cpumask(idle_cpu_to_node(cpu)))) { if (cpumask_test_cpu(cpu, p->cpus_ptr)) goto cpu_found; } @@ -371,7 +485,7 @@ static s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, /* * Keep using @prev_cpu if it's part of a fully idle core. */ - if (cpumask_test_cpu(prev_cpu, idle_masks.smt) && + if (cpumask_test_cpu(prev_cpu, get_idle_smtmask(node)) && test_and_clear_cpu_idle(prev_cpu)) { cpu = prev_cpu; goto cpu_found; @@ -381,7 +495,7 @@ static s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, * Search for any fully idle core in the same LLC domain. */ if (llc_cpus) { - cpu = scx_pick_idle_cpu(llc_cpus, SCX_PICK_IDLE_CORE); + cpu = pick_idle_cpu_from_node(llc_cpus, node, SCX_PICK_IDLE_CORE); if (cpu >= 0) goto cpu_found; } @@ -390,15 +504,19 @@ static s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, * Search for any fully idle core in the same NUMA node. */ if (numa_cpus) { - cpu = scx_pick_idle_cpu(numa_cpus, SCX_PICK_IDLE_CORE); + cpu = scx_pick_idle_cpu(numa_cpus, node, SCX_PICK_IDLE_CORE); if (cpu >= 0) goto cpu_found; } /* * Search for any full idle core usable by the task. + * + * If NUMA aware idle selection is enabled, the search will + * begin in prev_cpu's node and proceed to other nodes in + * order of increasing distance. */ - cpu = scx_pick_idle_cpu(p->cpus_ptr, SCX_PICK_IDLE_CORE); + cpu = scx_pick_idle_cpu(p->cpus_ptr, node, SCX_PICK_IDLE_CORE); if (cpu >= 0) goto cpu_found; } @@ -415,7 +533,7 @@ static s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, * Search for any idle CPU in the same LLC domain. */ if (llc_cpus) { - cpu = scx_pick_idle_cpu(llc_cpus, 0); + cpu = pick_idle_cpu_from_node(llc_cpus, node, 0); if (cpu >= 0) goto cpu_found; } @@ -424,7 +542,7 @@ static s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, * Search for any idle CPU in the same NUMA node. */ if (numa_cpus) { - cpu = scx_pick_idle_cpu(numa_cpus, 0); + cpu = pick_idle_cpu_from_node(numa_cpus, node, 0); if (cpu >= 0) goto cpu_found; } @@ -432,7 +550,7 @@ static s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, /* * Search for any idle CPU usable by the task. */ - cpu = scx_pick_idle_cpu(p->cpus_ptr, 0); + cpu = scx_pick_idle_cpu(p->cpus_ptr, node, 0); if (cpu >= 0) goto cpu_found; @@ -448,17 +566,33 @@ static s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, static void reset_idle_masks(void) { + int node; + + if (!static_branch_maybe(CONFIG_NUMA, &scx_builtin_idle_per_node)) { + cpumask_copy(get_idle_cpumask(NUMA_FLAT_NODE), cpu_online_mask); + cpumask_copy(get_idle_smtmask(NUMA_FLAT_NODE), cpu_online_mask); + return; + } + /* * Consider all online cpus idle. Should converge to the actual state * quickly. */ - cpumask_copy(idle_masks.cpu, cpu_online_mask); - cpumask_copy(idle_masks.smt, cpu_online_mask); + for_each_node_state(node, N_POSSIBLE) { + const struct cpumask *node_mask = cpumask_of_node(node); + struct cpumask *idle_cpu = get_idle_cpumask(node); + struct cpumask *idle_smt = get_idle_smtmask(node); + + cpumask_and(idle_cpu, cpu_online_mask, node_mask); + cpumask_copy(idle_smt, idle_cpu); + } } void __scx_update_idle(struct rq *rq, bool idle) { int cpu = cpu_of(rq); + int node = idle_cpu_to_node(cpu); + struct cpumask *idle_cpu = get_idle_cpumask(node); if (SCX_HAS_OP(update_idle) && !scx_rq_bypassing(rq)) { SCX_CALL_OP(SCX_KF_REST, update_idle, cpu_of(rq), idle); @@ -466,24 +600,25 @@ void __scx_update_idle(struct rq *rq, bool idle) return; } - assign_cpu(cpu, idle_masks.cpu, idle); + assign_cpu(cpu, idle_cpu, idle); #ifdef CONFIG_SCHED_SMT if (sched_smt_active()) { const struct cpumask *smt = cpu_smt_mask(cpu); + struct cpumask *idle_smt = get_idle_smtmask(node); if (idle) { /* - * idle_masks.smt handling is racy but that's fine as - * it's only for optimization and self-correcting. + * idle_smt handling is racy but that's fine as it's + * only for optimization and self-correcting. */ for_each_cpu(cpu, smt) { - if (!cpumask_test_cpu(cpu, idle_masks.cpu)) + if (!cpumask_test_cpu(cpu, idle_cpu)) return; } - cpumask_or(idle_masks.smt, idle_masks.smt, smt); + cpumask_or(idle_smt, idle_smt, smt); } else { - cpumask_andnot(idle_masks.smt, idle_masks.smt, smt); + cpumask_andnot(idle_smt, idle_smt, smt); } } #endif @@ -491,8 +626,23 @@ void __scx_update_idle(struct rq *rq, bool idle) #else /* !CONFIG_SMP */ +static struct cpumask *get_idle_cpumask(int node) +{ + return cpu_none_mask; +} + +static struct cpumask *get_idle_smtmask(int node) +{ + return cpu_none_mask; +} + static bool test_and_clear_cpu_idle(int cpu) { return false; } -static s32 scx_pick_idle_cpu(const struct cpumask *cpus_allowed, u64 flags) { return -EBUSY; } + +static s32 scx_pick_idle_cpu(const struct cpumask *cpus_allowed, int node, u64 flags) +{ + return -EBUSY; +} + static void reset_idle_masks(void) {} #endif /* CONFIG_SMP */ @@ -546,11 +696,12 @@ __bpf_kfunc const struct cpumask *scx_bpf_get_idle_cpumask(void) if (!check_builtin_idle_enabled()) return cpu_none_mask; -#ifdef CONFIG_SMP - return idle_masks.cpu; -#else - return cpu_none_mask; -#endif + if (static_branch_unlikely(&scx_builtin_idle_per_node)) { + scx_ops_error("SCX_OPS_BUILTIN_IDLE_PER_NODE enabled"); + return cpu_none_mask; + } + + return get_idle_cpumask(NUMA_FLAT_NODE); } /** @@ -565,14 +716,15 @@ __bpf_kfunc const struct cpumask *scx_bpf_get_idle_smtmask(void) if (!check_builtin_idle_enabled()) return cpu_none_mask; -#ifdef CONFIG_SMP + if (static_branch_unlikely(&scx_builtin_idle_per_node)) { + scx_ops_error("SCX_OPS_BUILTIN_IDLE_PER_NODE enabled"); + return cpu_none_mask; + } + if (sched_smt_active()) - return idle_masks.smt; + return get_idle_smtmask(NUMA_FLAT_NODE); else - return idle_masks.cpu; -#else - return cpu_none_mask; -#endif + return get_idle_cpumask(NUMA_FLAT_NODE); } /** @@ -635,7 +787,7 @@ __bpf_kfunc s32 scx_bpf_pick_idle_cpu(const struct cpumask *cpus_allowed, if (!check_builtin_idle_enabled()) return -EBUSY; - return scx_pick_idle_cpu(cpus_allowed, flags); + return scx_pick_idle_cpu(cpus_allowed, NUMA_NO_NODE, flags); } /** @@ -658,7 +810,7 @@ __bpf_kfunc s32 scx_bpf_pick_any_cpu(const struct cpumask *cpus_allowed, s32 cpu; if (static_branch_likely(&scx_builtin_idle_enabled)) { - cpu = scx_pick_idle_cpu(cpus_allowed, flags); + cpu = scx_pick_idle_cpu(cpus_allowed, NUMA_NO_NODE, flags); if (cpu >= 0) return cpu; } From patchwork Fri Dec 20 15:11:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrea Righi X-Patchwork-Id: 13916904 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2043.outbound.protection.outlook.com [40.107.244.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CCCBF21C19B; Fri, 20 Dec 2024 15:41:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.244.43 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734709314; cv=fail; b=Ihir2xgW40gmYUdbkjXS7NeQZSmIejLIs2lNg5tye4pillqMGIVNciUCxzZM79FdYjLL+8OIMAwjAkiZnZ8aRTvNwqD7UIkyLhNkIP9yjkqXWnxaN5dlfcWyPEk6n6DrG6KpCE12a5pQ6slUIC93NZX+qV5x6dEI/aAD3h/ZMK4= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734709314; c=relaxed/simple; bh=Qs84QUO7aor9tM46u9EGnQJWA2ngENEZPX20Es/R9FM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=pxQAlKZuabh6W0g6kk/vAXbkIUNFzXylHekSNTtrIg666OqTVz0ffyfzkbS9DAH32ZpihJT8ITTySs8S5gWjujyd573EuqSSrP5VwC6LHASaKwp4nDg2sOijYRfde/gAN8nL5u/ztSKzRjA1gGV28U00j4dmXWrwTM7hpY1v9v8= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=lRVapCRw; arc=fail smtp.client-ip=40.107.244.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="lRVapCRw" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=D96wQSUEwhrrBalNI+xUQTc9oLXvrKJoLvyibb0+bGMt0pys+tAbPOtjdnvO1l36pw0HXCWMAGzOXjSbhWiOICgCmxc/R8FVJaRMMGOlT9fziW5/YnL2g4j76VcDM+1liCSIo8S8/TfhCZVRSIaFBQvpRPBIZqu5l2abiJIv34o8N04Gxx1tbZba4WEnzQy8RNERRRomSS5RxbcN61HRLVzAdRDijnWDZxXkoa375WStFdFx0SWD8TMR2IkVRLoRcpLv0Q5iGmmsLUkMy564j2PaPq/oZJ3CLHS/x8U4XCwjjpg+PZ4wmXNejdLx1btp1mSoiAXHX0EfnrVKUuCDYQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=DqwDWV2Mk7iopu6bUJMefBeBRVYvYWXIgOGq+9PlXXo=; b=NKC3n2Ti0j3kdwaEAVuwh5c7QM+Wj31RSYsKx581wz/kJ5zgsrJn/cx37sIxMHxe2E4yXBasvGgQQJx57m3nOS00Yfh98cNdZghPbo6UFsyiGgr3GRkqj5qf4wH143LFtrNNT24ds4kCB0xH90N4Au0Sl995d4LRdwp6yJFlqPOA4cwBWJ+9VZskqJJhN6wBW/LVtx0O0ipLm+X00Q/8J+XES1kCYM2iPhjHHp7KD0OysVHxg6IVFCuLWujp+ZBByF24z9Rb/dqLzC+ozHrXzdpnK8wdd1sZams3mnvY/qT/81VaAVXt0WvKM6STORJl4VAsP5CxybfeAmEElWwC3Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DqwDWV2Mk7iopu6bUJMefBeBRVYvYWXIgOGq+9PlXXo=; b=lRVapCRwri2XqGdpOCrDBMc83yPs8ks8r36za2fAD+GJBZvrNgmfYLdPJchld+00VO4nG1rXKbaORs17LHsZzEa1YGgMVPZVZ0XLcZCynrhazWh5pua7dckHH0xN9rzpt4Fk0XjFNLN7YCUf4+bTrH242HNcbEVbmIQN7mfpAKLkyDEHTn4lAY58Bw/vmkBKwJLXWjLM0ay6YBB5eXWLOlMfcjDRiCmiKMFmGTXD06dQwiOI9WYNjj4quQVL92+VdE/Pm9JZLe5lpPSboIsB4xy4OQIER55bsIcfhXb8imbrEuz2TA0OKncxItpG0DmhHwAZ9GNbetcHMiSuzO4D7g== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) by SA1PR12MB7149.namprd12.prod.outlook.com (2603:10b6:806:29c::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8272.13; Fri, 20 Dec 2024 15:41:50 +0000 Received: from CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5]) by CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5%3]) with mapi id 15.20.8272.005; Fri, 20 Dec 2024 15:41:50 +0000 From: Andrea Righi To: Tejun Heo , David Vernet , Changwoo Min Cc: Yury Norov , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , bpf@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 08/10] sched_ext: idle: introduce SCX_PICK_IDLE_NODE Date: Fri, 20 Dec 2024 16:11:40 +0100 Message-ID: <20241220154107.287478-9-arighi@nvidia.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241220154107.287478-1-arighi@nvidia.com> References: <20241220154107.287478-1-arighi@nvidia.com> X-ClientProxiedBy: FR3P281CA0168.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:a0::20) To CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY5PR12MB6405:EE_|SA1PR12MB7149:EE_ X-MS-Office365-Filtering-Correlation-Id: 642c8123-2ee2-41df-0736-08dd210cd1e6 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|366016|376014|7416014; X-Microsoft-Antispam-Message-Info: 6GwpifyQesgPDtQb3awmFMCsWNLwi8UPowz2HdI5pDAuIOj02IsdsNFXilopXTnc/35H0XrQAojMRhV3fnkrDT4ZOC2JGA/ufChK2L6jWxiQyHqasL0WpSZc+nSq96QZ4bVvnNYwMm132cfR209jf8vcIR943M91Q7oXX3Xax3mYjNEk5MHmQebXMUNu3I42GhZCPp4ckHOiHtYNb6qPwvKH0OsCZl8bziu1XBlFbcs9amI7RC7f66cMti6ZyPwNz6i9rAC2Hn3ziClNxMgFCLQLNALqGeKBaGkAaUJEjKgExf2t+xU+wPgcS7od6iDTV4awBE6o/HDOFFb3K7irpDWFe0KPCjwhSmUecCQK6kengc1QVRkVRjAGVIauJaGtihaaWPuuEOc50zIFxfaoN7pN4/PwQBE7ZugOumzVz+IGTgb0koau3T3ifzcaSkEEZOKRYQl7xy6pz+/R+Emw7g/bf4P7IpkpGtQWBYGh8QOTwWxV6yF/1PWdBNb8OrHz293Su+upnDM26w/wnrtmUfIOjSYFSFwubQ697GHHz+W5Vqx+zZkc0/Hm2mx8+oWd/3NsbnMXYL+QwMIccwx/x+bJQAmYIGt1eWkd2KYKM2SSGBLjAAlKporih15hyu2bahsrJhEd5STNnr7+Il7OLgvx/cY54mVZvc7JhJPn/70a+zF+0hCc6BjBuQ0exCIFn9DZqo3D1/S/p5asfPMyGe6nsMbKV7j/Mt5FzHALMwvchRcAwogvDJl+bPu8PPmlx0TsgZZdf4ZLnlvTSHQW7HahbrkLRCaTt2geGWAgkr/heZV6lCiUKU/n3nHKGyfW82KMA5OrDJink095F85MMLnBgh7/J4CCvhP4G5YRRaHM9wrWCq4ECn2vRKqfmANSqfNNCzhP3UlxL0jwkcqVC3GBK2RQpqvIOLo8NKV6BwIo7nCpEgdiE9YCLVjYzikS+uKzZRyHfqc1g9jviKVOX/ERX3t819PpwPwkSvn3t2YKB/1ZGwzGygUh8YPr2D2vi7WQJ+ote+jf+scTBWpjqEMwofMGdHYvzc1m1YIix0FSjWbi12AQd/AAGgYxUvntsDogXbQipIXAEQW9eN8dQ6HIdM5NPQgfRMSQe7LcSlc3P/roi85Wj5FGSybmDmibtK0rNr7AsvzjvSfU4mNtf6WfhxlJ3IHEn84Ek5O3nWKmgC90dKt3GBiu5nGMP9uiQlihubxq7gQtuUCDjflVmaGe/jKeHPLw+g0Sd+EVMOo3KS+euYGVySNWN+m0iASZgxtjW/kcLIyQgVXMKEmyZgt5oAUfr+a4XEQM7teYqvTHCh9+4nrfN6ViHITKTipgPYdq6ngOeaiQquRKfGQqMnbfoZpaGo/aIMHIB59KPFh/Z0QITZ0vgxMQhs5kXrCU X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CY5PR12MB6405.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(366016)(376014)(7416014);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: DEwyvehx9DRMrwiGTONwAyG8MUvimr6e/i4EQx3cga3ozHNMF/19I61GZwZtQeT1r3wodx0YI6YMeFxJCtkUaea9qESGmRKQ+NppwipJpwjNjbPTomOdzWrYF2qV6qYzwlY+3Q2Kp+7yVkugvUF3TkBYSWOyjjpMKBaL8ofCRBrHpuodKRMmXi8h5TnKLVREra6+SJ6lFeefXxjVRk1WJHlbUL4Y3cX9t/lD6Hj0THrgG/9e6YlA2MDtYhXdc+V5cFWYVwYEuvqjfqlwETX38u4b8jdNUx510t2TPafWjeaC0K4H/vvRpfDQJAOIahHjuqqtu94Zdk2dfc87BRfBwQ5P4dfLeXywN4IqXljtUS4IBWa6rqSPlq9sLFYgcue/yDQA9gcJzu6e0X7kR7Jqoijyx1fr3zLk1neEXvEXAauRAcHnyDdUuPZShMyE5JNw/b6tfUdQYGLhFa7ii15VjySnjQCE5ZDCrPepcO+Tu1et2uxqSPeYuADdFABdvSr5Ua8hAOzhaTEj6CCqkZ5RAPzr83rssnLmuV1BeD5AmeGFJGgsGVXfhp5df+dvi/IeKI1NLf90z3CLAaKQ6bsXhCzCEwBAI0Dgjmndwa1+5f/OIIaIu/nEFuKH7NeLCHwaA9mPUOgHJmyBQ0BC2s407yz3zzMXXf/6O7wPJ1irQpR6pEtMM1Zha0mM/RR4+3euMcQTl7sqgPKCsjAgDhTixlzjh3aJOXt++J7DJeygVqVvO2YzE4dIWaWTjXD1JXIROJDlXziceb81lMvTKh3OtUKbMjVz+MNGRjhVpnlEm9fCxicSGOCSAU0qYuvU6DLjTybr33wBJGSpi6IjMWr+M6BA+xrs1ufCJwbYliMuWxQHoWrIF2bOT7QLBzVAf//T2d7gZ05DgBUYXsSA3EYn3iPXsim9mCFC6WHWCMGhspVgoyrn8xmGjQ3rFLnlF8JL3rB3OY7zgXm7tREQIHIx2r1DyuKCBZTy6Yl2o52NpQtl3ybqECo2F+6N91becXqiCXjj6pco8eLzEP0bLlxjCL0CkoIenHGhoO9YqmD3buwJIYsnQHXxGMbUxpFZxD/D6DVz0+cP+mPvcq9zBqmoPF11DiKkX336pKMbAPr8zWsXd6Wefb+jDz0wjQG4tH32y6n1jVK4jLxQO+NMeE5hLLZ1nZA3yewmDfsbpswJOrz52k5V4nDH/dWXbmVgzBuHOmC8oJvMiGQF9sx+ItqIVq/2fc3IvHrIlBcqAJnjd6Pysb82sZaoXVjqqrRGcxOX2hFUA6I/V7z0ePD6abVqxXkDNLkyPIJkdr8bVK0oARXhx377/y1KLhUfhTzSQK9kCs6KOeCcPBotIwF2rBmbbXbzbQj7DF3UtjCw+asv+oBj3/5dXiGMmLWodDmb6Rsb6WFbwIoINaqvYr6QnlN1IpdLbfHHV6sQoeSET8omtDvcNdONYSt1f89Zp5C0V2WmmbGE9ATKdtPEkzALVlFDI2QE8ShqbNikkCgjHFvBKWYrFnxMFKR2xcSY97yOmtFk/RURYynsiBpK0D4W7AfwvSWYUQ7PsuE9zdvewg3bENGJSNl21LhRbwnSvUXkPwfo X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 642c8123-2ee2-41df-0736-08dd210cd1e6 X-MS-Exchange-CrossTenant-AuthSource: CY5PR12MB6405.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Dec 2024 15:41:50.2421 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: TDW3/1kuHQjnJEL+qSrj1xgBVUmvIiXe00/J+IW/5LzVFqHFU2F4j/eMoTI9dpC6Y38Z5bbbRZKGoaLadhEeAg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB7149 Introduce a flag to restrict the selection of an idle CPU to a specific NUMA node. Signed-off-by: Andrea Righi --- kernel/sched/ext.c | 1 + kernel/sched/ext_idle.c | 11 +++++++++-- 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c index 143938e935f1..da5c15bd3c56 100644 --- a/kernel/sched/ext.c +++ b/kernel/sched/ext.c @@ -773,6 +773,7 @@ enum scx_deq_flags { enum scx_pick_idle_cpu_flags { SCX_PICK_IDLE_CORE = 1LLU << 0, /* pick a CPU whose SMT siblings are also idle */ + SCX_PICK_IDLE_NODE = 1LLU << 1, /* pick a CPU in the same target NUMA node */ }; enum scx_kick_flags { diff --git a/kernel/sched/ext_idle.c b/kernel/sched/ext_idle.c index 444f2a15f1d4..013deaa08f12 100644 --- a/kernel/sched/ext_idle.c +++ b/kernel/sched/ext_idle.c @@ -199,6 +199,12 @@ static s32 scx_pick_idle_cpu(const struct cpumask *cpus_allowed, int node, u64 f cpu = pick_idle_cpu_from_node(cpus_allowed, n, flags); if (cpu >= 0) break; + /* + * Check if the search is restricted to the same core or + * the same node. + */ + if (flags & SCX_PICK_IDLE_NODE) + break; } return cpu; @@ -495,7 +501,8 @@ static s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, * Search for any fully idle core in the same LLC domain. */ if (llc_cpus) { - cpu = pick_idle_cpu_from_node(llc_cpus, node, SCX_PICK_IDLE_CORE); + cpu = scx_pick_idle_cpu(llc_cpus, node, + SCX_PICK_IDLE_CORE | SCX_PICK_IDLE_NODE); if (cpu >= 0) goto cpu_found; } @@ -533,7 +540,7 @@ static s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, * Search for any idle CPU in the same LLC domain. */ if (llc_cpus) { - cpu = pick_idle_cpu_from_node(llc_cpus, node, 0); + cpu = scx_pick_idle_cpu(llc_cpus, node, SCX_PICK_IDLE_NODE); if (cpu >= 0) goto cpu_found; } From patchwork Fri Dec 20 15:11:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrea Righi X-Patchwork-Id: 13916905 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2055.outbound.protection.outlook.com [40.107.237.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1806221C9F9; Fri, 20 Dec 2024 15:41:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.237.55 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734709318; cv=fail; b=etSsQuXmTQMT8bjtmBYmB97uBqctIYhX/kdRhKSQjkfLspJ9twSlhBUaw/CUmL+/28GiO2P5cSZH0JfcbvVe3WLB7w200Rd02Eb+8f/4CKaIj7vJP+ymKPLi41it5xam02PKKyXCZbHbdgOGJvLWECEr9IpEEAW4F/mmiZLIl9g= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734709318; c=relaxed/simple; bh=ESQcF1i56YYgYIeb0Crfwv/4aAPjSVs3lWR2OU4E74Q=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=C+nJPYUJ8H+eIVx9/OGZspwJkDuFh0zMXce9aH0GLTaDbyLcpW+Z0Nax1KR+IUEqFw8F32PLSqCMmoAnF7w9+XoBNMu23zVpfqiHq69v4ZPu0hokVtf9maPcuaTnPrBm1pb1zIuKedlUaAS0nO9ZIfJ7/wNYmgdX5LIfpuaen2I= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=NVXvoQhG; arc=fail smtp.client-ip=40.107.237.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="NVXvoQhG" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=nvk7al2RushLuGtQev1Z9iqRGzRdshakp36fR5rbNv+ruvOBYYAnWDtfxobeHSSPuTjKlLlTY1QtqbmeQxhr1a0UAm2WIDH6ys7WXa/Rm6pTuiZwaszQTw4v5rb+G/CqEEG1KwilOQq07cVx8Kk+DWGicBclUthQyildVBh8AelUjQZhkQIKdQhj3hThRYTkjxY70+EdAXedPaGCZ1YwF52Wlp8NQ1e6VBMssU0xMjQAubhhssUvJB5dsA1SBDNQAMaPDeGl1e5vmWm2hKI6O8EnlJBAyaKCQlFdKYEHRs8tTBf5KB06GhjVawtQyKGtwi8EWcrN/ABWEefwI7WEXA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=LRXrXTMMNrCTkmDj6IgGD9v4tudpGN3CU+dmB3C8qxw=; b=m6UqioeSpQHbN7WIyzVyVY4fg/4G/GPRNCfITMowJ0jRox4LiBYoYW6P8nrFUZOkbakPpfWC7zzZYIHaNGP35q/kSkImrGO4EMEvSMeoJczFawCqNySxVfyIdg8pEUW6CwimkVuj6qcGH1Ore4ZAUXgMhjvVJpscBoaVLNDpaiGiSJ/+UirDWTF200UH/8z1K84uFDbCpvvcwcFg7qBnzwWeFn2OGcrOjYJLlSdH+BpmD98IRMpjH9Scrhnax49S3EoHd1xne/wCGUZ97juldV2E9f7CBDy+Qw9/0ZH/NRralqoBxhvKC7LLr4tdhSBKVUDLEGDQ8Wl8Dg3Fjz1Rbw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=LRXrXTMMNrCTkmDj6IgGD9v4tudpGN3CU+dmB3C8qxw=; b=NVXvoQhGhbVapnGeV7/qSwlUT5Gpm0rAdT9vywlZBtrPn6osH/Dd+lAaMDnKrudQ6VLXHo7FTaE8SDr9INIsMUm0G/fDhs5TEBy2P1IplNdiASsCfFtHMeiv0K7LLOabcLKBjReBZ9suIyQRT3rRGKHtO2XgVchmqjRNYxEddAx7dvVdzLlpfNZ90J+Pd/2n5uXtkb3Fo/E2Dvnyb5X3AqLXRl4EmCkhQTblnqp2kmg+vncWZfGYWTqNG+J59devo+ZxqKvHBV4y2GYnJUuA9mm5yaIJkDuAQ3fm+DW8reQVDEbyTl1s9yjFgGfHYvyofWHH1jd3opkRgaUjCxqwvg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) by SA1PR12MB7149.namprd12.prod.outlook.com (2603:10b6:806:29c::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8272.13; Fri, 20 Dec 2024 15:41:54 +0000 Received: from CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5]) by CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5%3]) with mapi id 15.20.8272.005; Fri, 20 Dec 2024 15:41:54 +0000 From: Andrea Righi To: Tejun Heo , David Vernet , Changwoo Min Cc: Yury Norov , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , bpf@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 09/10] sched_ext: idle: Get rid of the scx_selcpu_topo_numa logic Date: Fri, 20 Dec 2024 16:11:41 +0100 Message-ID: <20241220154107.287478-10-arighi@nvidia.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241220154107.287478-1-arighi@nvidia.com> References: <20241220154107.287478-1-arighi@nvidia.com> X-ClientProxiedBy: FR0P281CA0045.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:48::14) To CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY5PR12MB6405:EE_|SA1PR12MB7149:EE_ X-MS-Office365-Filtering-Correlation-Id: 84759d9e-8b26-4019-ac6e-08dd210cd443 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|366016|376014|7416014; X-Microsoft-Antispam-Message-Info: KcD5cu/jPtBfWeM2ezQWpQvqL1cupNsTgQ7Uv435zfGjJaiYYhqFIOdH58/TOjGEQ0A4IeiXetVSUOYFbrg5PS327sgUVRF35uFQWK4D9Hcn/0Zov0o9Gr/B4chqBmRCPjmQE6rrStYuAEGAxNSCUXW7RaqwHxObTJpNluOAM7+GYtdpqQxb2Pd1iwKA750+bbzkFq8yRp1+nM0HegocyFVVgULcCg9J4f6M7vtfWq5VvIGtn0NFtM7tW3X+Yb2DQpgDTF7867iYCiG5eoyFp++xxAEoAzNr8ZaL5LrKM9X4S8Km31Kay8W8fsdjCO0oWOWuYVKYF6g2yknXNqdD6A6qPL1rKuZT9L+KQUQJslDoXwrhN2IrBESz9EVYFHvcWNTrkjKsMpc72Ez60TQbaqp8h11V3Ad6kQI8NJtGy7A8y6r7Fgy/7zXFBd1BB3ZIIxuN92AOjCJF10hPEqURX9GAiHZDMpWPjuHe+f8okRghSN0zN7tGaxP4GE1Ak3LpxDG7c/lX/h/N52nzxeGxk61nmrIx3x2EK1ssnAcON2Yn4GTY1vcrI+9DlTqNt68ne7V1D+uU72IR/zMss3gLapwH26DuhRCO92vko2hoIDFXQ9wbIBgW5Mx2QE6B7CILK064+AV7TLTTGKbHs7Mrm+gXW+S+xGFSpdLwG6j+BDmdekLOy2oYN6zhJAB0JcbTVuxpk9sPmMmZw10mUWkOd026zuDICWWzoSoNmWjt5j8qNJ7yFNuJLDc0OgwMwu3XyIndMxuZtOcMXyp4rhVKh4B9/2V66lr83nwT6nKbj9fHlyaeqkAVf0lwTBMeqeP0Lh6dpnfujWcTnNR6U1Dzbv5VwoxCAqFdqjMd8ZpYC2B5GMlZ+LurXIir8oPic+v+tXfiOo8scr6yArNTfQDL+SZjDzfsy/QQB+CNseFwI5rOwrQ546vzAkcwpV77ldYHHTEF05k/b0G3DUsg4teLehz4XgqkVmtccrFeeUb+cHsF9u5nusgm6caowsXmeHaqkNLHBYhb5zE3U0jeHQ25D0Pe6pWrscJWm7E1iVt7hyxGi2n155W/hJl8629DTG9bjYZV9NlSrFaTZ8mh5+G3TIJ7L1VRF827OYc3oZgRZXAo/PS5ENA8TdXfVdgxNJK7orVyAednBSCEbnTsxdwrZv3Mx9L4cjtFuww+Xs8/fLMF5ao3HOTCzelPOQc3CQlDoMlUjiA7XTzPrWbIb6UZywBQVYabjHRTuNtbvQb7zp96cVOvyMAJdTayUzSUwIP+QGvZTTTvSidpc0SXVpFWm13ZHWE8XQuOQUYmoM8DZfSVngEqdHwEzaVPqojj9q/dIcwd91jO9c/j9orbydMN1onvbvlPsZJ1zMMP2ljq69wBfi2bzFXBldBQs1hHFh/O X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CY5PR12MB6405.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(366016)(376014)(7416014);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: DHGkQnc31vquAvFuNh+eH8hX5ybufk/0UdV4HzxyrKuyqO6ApsHNprFoDIVaj5IifzQIsveN/Tp+7jdHE8fq7/76T+gQjnUsbpAm2P4LlnFk5XFMuqHi2+73ksZKBgR3cT3Vpq3MQjiQm78e9tWtBDZcyvFy9q+uEmem8a0sAPs+1Hm735qzQxkfkKip4wST41Jmy3FWit4j9Y9Xl0FFtAy6GcdAw+p/YtLJNEsWhKD5affKVMUD93m6lxLTJudXJxXUacZfSUntcvWY34AtbocNgRqevSnjUlBS/NDAtz6KwN+nSzlCICpyMl8L8VoJKRO9qK9VeXdlH0zZlnpW1GXwuPzP4WjgdUZJQmO8Ncg2o5i6b1HONAKJKWiy7bhAlVJjCpj26HCd54ppjLyopjJKxD/S05hz9lIhesFGG44NgLNRWf4KS204f2ErMckvCR63bNZOxxMIMsx0LDp9l8R+BzFx2aYfRYXDxC+qQzTPCeJZ5N2MxaB1cmY2mO0wE33XyUoA7/y+zEzEHnUaNg9M/aHhoM+BR9iO0o3/3IIBE7i6m9BsyE07kBAgPYcJ3HppcHbZgGLi6iOIeL17nQ6thRoFNkEP3D543JsHkTJvj58nCvaIJU8A/TnuNO4jsIZZ1DlbcPOV8DsNx1dtRxKKFwntp39vFpVIYLsQTwAe0Z5Fpc+Imzp34/iVKlHZ11gBdwP1ZZKrPUS/wwOTKtZPY03xANYjYEZXN25mGh9lQNP446gFGDKi97NZp4ClzgOI1WormPynYArcfrvf1EpELJAeJDE/ltpYL5iyE4ycasHgZxTm2dO2JYiHexvZzdO1W1EpUmUPQzlAVsIk2sYW7QpU1JEWddCaYX+Nf9d4xovi9VreXLd3GwgG25r3iaTYGMUTQvf4k+ntX8dhPsmwVxkjZtfPtABVTqMFwL594GOIHnp3/qUJN1wTWO6sUKY35/UQTiMFjkP3+RzoL6oGz2XKVOHMxJbST90Tm18t78MmPSjteV5MnBAjCXkklDuYfy3P7N/YnVA+MQWRmrcoS5gNFJEjsP42Gc410xWOpOX11in8IrVXEWaAUJOXe92rWyIK6I749NXi/Miik3qzwQp+RJhhJGp4DefVBjGnMO8V1Emrj+YAUIp/UTU0IU8ml2LpIq9sNUzIlNf6o9xGRYygtBrI2yjTDltACMxlbgaH54dhe1hM+b8mv9egfjMeUfy78e80AEF9HExdDt6zMe7hCEpwdot4qqPMYnxbnA/LfJIYUgGkX3aS3rCrh6w7fY94G0XoSoW3D0WdfH0Gddkdv1j/ULRid42YLs3hvm09J8IV6bo6NTXLxDIkGizcap1jDJsbxLlo2AEvqlPELhK8imcVEQFmc6cXjV/6voIfDZo4iZCaFqEKud8ncPaFBV9GioJx/fAzkuXGy2Vb9Z2M9r9admgqRsRQv4CQnIg+X/DLjGrVuI+XjEiSuRI2+yGQdU8sR/Sw7wPM59BJXTGxXS/imxXZBqcHCiN/w6cWs+XfVzeP31UofVBhMtvjx7pkKupSfQfvfq43hFq528LdrNtfQKbZvnx6J7b/pyyrjwGaC7tEyHYlLx8Z X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 84759d9e-8b26-4019-ac6e-08dd210cd443 X-MS-Exchange-CrossTenant-AuthSource: CY5PR12MB6405.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Dec 2024 15:41:54.1759 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: jpp+A6B9T/4sbj26auVp7EAfTNqYgqLPRv5cTiPrJ2tPh1qIu2wyu4wZ/A6GkO+ULqY+B/WJXpTQG9mS8pR/BQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB7149 With the introduction of separate per-NUMA node cpumasks, we automatically track idle CPUs within each NUMA node. This makes the special logic for determining idle CPUs in each NUMA node redundant and unnecessary, so we can get rid of it. Signed-off-by: Andrea Righi --- kernel/sched/ext_idle.c | 93 ++++++++++------------------------------- 1 file changed, 23 insertions(+), 70 deletions(-) diff --git a/kernel/sched/ext_idle.c b/kernel/sched/ext_idle.c index 013deaa08f12..b36e93da1b75 100644 --- a/kernel/sched/ext_idle.c +++ b/kernel/sched/ext_idle.c @@ -82,7 +82,6 @@ static void idle_masks_init(void) } static DEFINE_STATIC_KEY_FALSE(scx_selcpu_topo_llc); -static DEFINE_STATIC_KEY_FALSE(scx_selcpu_topo_numa); /* * Return the node id associated to a target idle CPU (used to determine @@ -259,25 +258,6 @@ static unsigned int numa_weight(s32 cpu) return sg->group_weight; } -/* - * Return the cpumask representing the NUMA domain of @cpu (or NULL if the NUMA - * domain is not defined). - */ -static struct cpumask *numa_span(s32 cpu) -{ - struct sched_domain *sd; - struct sched_group *sg; - - sd = rcu_dereference(per_cpu(sd_numa, cpu)); - if (!sd) - return NULL; - sg = sd->groups; - if (!sg) - return NULL; - - return sched_group_span(sg); -} - /* * Return true if the LLC domains do not perfectly overlap with the NUMA * domains, false otherwise. @@ -329,7 +309,7 @@ static bool llc_numa_mismatch(void) */ static void update_selcpu_topology(struct sched_ext_ops *ops) { - bool enable_llc = false, enable_numa = false; + bool enable_llc = false; unsigned int nr_cpus; s32 cpu = cpumask_first(cpu_online_mask); @@ -348,41 +328,34 @@ static void update_selcpu_topology(struct sched_ext_ops *ops) if (nr_cpus > 0) { if (nr_cpus < num_online_cpus()) enable_llc = true; + /* + * No need to enable LLC optimization if the LLC domains are + * perfectly overlapping with the NUMA domains when per-node + * cpumasks are enabled. + */ + if ((ops->flags & SCX_OPS_BUILTIN_IDLE_PER_NODE) && + !llc_numa_mismatch()) + enable_llc = false; pr_debug("sched_ext: LLC=%*pb weight=%u\n", cpumask_pr_args(llc_span(cpu)), llc_weight(cpu)); } - - /* - * Enable NUMA optimization only when there are multiple NUMA domains - * among the online CPUs and the NUMA domains don't perfectly overlaps - * with the LLC domains. - * - * If all CPUs belong to the same NUMA node and the same LLC domain, - * enabling both NUMA and LLC optimizations is unnecessary, as checking - * for an idle CPU in the same domain twice is redundant. - */ - nr_cpus = numa_weight(cpu); - if (nr_cpus > 0) { - if (nr_cpus < num_online_cpus() && llc_numa_mismatch()) - enable_numa = true; - pr_debug("sched_ext: NUMA=%*pb weight=%u\n", - cpumask_pr_args(numa_span(cpu)), numa_weight(cpu)); - } rcu_read_unlock(); pr_debug("sched_ext: LLC idle selection %s\n", enable_llc ? "enabled" : "disabled"); - pr_debug("sched_ext: NUMA idle selection %s\n", - enable_numa ? "enabled" : "disabled"); if (enable_llc) static_branch_enable_cpuslocked(&scx_selcpu_topo_llc); else static_branch_disable_cpuslocked(&scx_selcpu_topo_llc); - if (enable_numa) - static_branch_enable_cpuslocked(&scx_selcpu_topo_numa); + + /* + * Check if we need to enable per-node cpumasks. + */ + if (ops->flags & SCX_OPS_BUILTIN_IDLE_PER_NODE) + static_branch_enable_cpuslocked(&scx_builtin_idle_per_node); else - static_branch_disable_cpuslocked(&scx_selcpu_topo_numa); + static_branch_disable_cpuslocked(&scx_builtin_idle_per_node); } /* @@ -405,9 +378,8 @@ static void update_selcpu_topology(struct sched_ext_ops *ops) * * 5. Pick any idle CPU usable by the task. * - * Step 3 and 4 are performed only if the system has, respectively, multiple - * LLC domains / multiple NUMA nodes (see scx_selcpu_topo_llc and - * scx_selcpu_topo_numa). + * Step 3 is performed only if the system has multiple LLC domains that are not + * perfectly overlapping with the NUMA domains (see scx_selcpu_topo_llc). * * NOTE: tasks that can only run on 1 CPU are excluded by this logic, because * we never call ops.select_cpu() for them, see select_task_rq(). @@ -416,7 +388,6 @@ static s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, bool *found) { const struct cpumask *llc_cpus = NULL; - const struct cpumask *numa_cpus = NULL; int node = idle_cpu_to_node(prev_cpu); s32 cpu; @@ -438,13 +409,9 @@ static s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, * CPU affinity), the task will simply use the flat scheduling domain * defined by user-space. */ - if (p->nr_cpus_allowed >= num_possible_cpus()) { - if (static_branch_maybe(CONFIG_NUMA, &scx_selcpu_topo_numa)) - numa_cpus = numa_span(prev_cpu); - + if (p->nr_cpus_allowed >= num_possible_cpus()) if (static_branch_maybe(CONFIG_SCHED_MC, &scx_selcpu_topo_llc)) llc_cpus = llc_span(prev_cpu); - } /* * If WAKE_SYNC, try to migrate the wakee to the waker's CPU. @@ -507,15 +474,6 @@ static s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, goto cpu_found; } - /* - * Search for any fully idle core in the same NUMA node. - */ - if (numa_cpus) { - cpu = scx_pick_idle_cpu(numa_cpus, node, SCX_PICK_IDLE_CORE); - if (cpu >= 0) - goto cpu_found; - } - /* * Search for any full idle core usable by the task. * @@ -545,17 +503,12 @@ static s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, goto cpu_found; } - /* - * Search for any idle CPU in the same NUMA node. - */ - if (numa_cpus) { - cpu = pick_idle_cpu_from_node(numa_cpus, node, 0); - if (cpu >= 0) - goto cpu_found; - } - /* * Search for any idle CPU usable by the task. + * + * If NUMA aware idle selection is enabled, the search will begin + * in prev_cpu's node and proceed to other nodes in order of + * increasing distance. */ cpu = scx_pick_idle_cpu(p->cpus_ptr, node, 0); if (cpu >= 0) From patchwork Fri Dec 20 15:11:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrea Righi X-Patchwork-Id: 13916906 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2048.outbound.protection.outlook.com [40.107.237.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D4DFF219A9E; Fri, 20 Dec 2024 15:42:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.237.48 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734709322; cv=fail; b=GEfjLBfNYDr7cL4dqTDsOgSnMeE/flgZWIHGYHNa+X3vgRIxWEwiFTQ9u/g30TAl/y8qycGChDMyIGqjHBMZvAjAtq2/N6ruEIk3lf+Cq1vggzCZPCOLIIeAOOuBeH7AkHingmiGZ8T8VJv10FPbkpLX/cfmaPKl3VsmUSsJQnc= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734709322; c=relaxed/simple; bh=KZeK4ClwTny2/gfLMDMRSxisGEdK47Lg3yaRSXWZTTI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=GF679X68untl3CmwmcAzRK062iPhpGLUKAtZ/2RhTcL3qIZAhzrwe9Rt+boOaMDxPi2N/AlvoX6vV7uZak6yonbirUIad79puO8NHumms5X4cOQdQ+AI+StLAozxuk1GduZbxerTwcv7IxUTmF7nAXveRS1+YjLyJI0nKZUknGg= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=rNiV/Kgb; arc=fail smtp.client-ip=40.107.237.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="rNiV/Kgb" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=V7BzjTKktOtY1B0S2//DxF54DWuPmmyAgd1566xH1WFQsZp5WAdF5Xq5MYloE53DKmphByB41lEfTKsqC+NbiKt/6SxI5eDieb89OmDAZJK3LF8AWwt2c6MGRsqz3quP5gJF9mNI98es4VwjVwL/frPtZhOB15NO9dWgHMU3i7bKZaUev2kmkyBxSIYeMS+vbPDhOqLdWcqeDGnH7bUVnwesC2pSS2X/yXhtr8k++Yq2FzVBzClUIloW0uSDIKcuoeptLyZ9bnSgPXW0lKJnMP/q2X2pYnrZGABLQKDlJUkXbqVmJ14qj/G20q0vLgb7v3yIzEffm7F+/bcQRi2gpQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ptBiQ1MawOcrAFmtYivwhcV+dm0PWgiQgldTX5dVmAY=; b=oEMof54Y0AUPyW2pSdGGkevn65rrWI7JBiyuOJ7zCCwBnE8dItCO7t4WKShoICxrw1orcBzHdUhpV6mt7skSO3LIAtT88nI9jbDQaU9sTQLVS4/z0/1EfqiPp+zeBRVqZiu5ScOBA7r6YOq2O7quzUKYZo8L9rg+jhlJEJH/Zn8Kfik79giP4VFsm23PKOhLra3Lit88Vpy5cmCBYH/pIO204PkkiJqcKTf2+otnhziUDnRg1pwDbwDc6LpP6r0QR0JeFDM6Kj+BFRZYqTsuejl9vIgO+Y5fnou2/1sZrNoCJ66HjQYjuD82Mact/ssp0FYjNoiL5ZebNmnxs2DyzQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ptBiQ1MawOcrAFmtYivwhcV+dm0PWgiQgldTX5dVmAY=; b=rNiV/KgbpWYVV3mrHUzt5GQHvfKiPUKnQZ13cOJnCH5Uqe51bB0OGNwD0uQBvyGcJv8x0BtWAq1jAkG4e4FueYbMEGxGvd8JxiO9Df1p7GFl0JfGtTHXPLDJQ/5OWCwOnKKMPyGSw29b1/K+RtNl+RmDODbpJl8cCXMcfHI0XnCbRA7syHtlUhB9M116lQRXfn6ROmPO/riAgfMwDV/95C/cNY3ry6lIpxr86O7wBitd4K/oD2CQsVlwp0qtMxxHTUGbFh36YmryWwON36WTMJPdPA4kLK3JfVoBnPQDy7xrOQqEI8O5LVWmc6u7/jkM3aCUzRL/r+PeAfIQ/r/0VA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) by SA1PR12MB7149.namprd12.prod.outlook.com (2603:10b6:806:29c::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8272.13; Fri, 20 Dec 2024 15:41:58 +0000 Received: from CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5]) by CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5%3]) with mapi id 15.20.8272.005; Fri, 20 Dec 2024 15:41:58 +0000 From: Andrea Righi To: Tejun Heo , David Vernet , Changwoo Min Cc: Yury Norov , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , bpf@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 10/10] sched_ext: idle: Introduce NUMA aware idle cpu kfunc helpers Date: Fri, 20 Dec 2024 16:11:42 +0100 Message-ID: <20241220154107.287478-11-arighi@nvidia.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241220154107.287478-1-arighi@nvidia.com> References: <20241220154107.287478-1-arighi@nvidia.com> X-ClientProxiedBy: FR3P281CA0179.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:a0::16) To CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY5PR12MB6405:EE_|SA1PR12MB7149:EE_ X-MS-Office365-Filtering-Correlation-Id: 3ee63f29-8376-4011-19ca-08dd210cd680 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|366016|376014|7416014; X-Microsoft-Antispam-Message-Info: q2ZkoiwNzu5H4FTLFSPsOQxzehhwHOq4DhQ3VS4JUJMN9JcDcbex3WLUK4tO/OuNQlaBbO1BH+f28cpb8bGrPgItFKAgVLFd6cKzlPokNrviSRg3jgh5ncZ06eIm5IRBdiXwFygLbhH1c9sdWfn6mhO4IjCd32o6ZDdWqgOFmDa2srfoXqLMvNzHEjWWP9nEwd0tsqG9zTmTJSAiIrEDVDp2qkzC826f0BTQIoyeM3QVz9mNlLkTwAZBwwpceAXvpo+NhrDMCacqDr2K6JraeBY5/Px4WWGtT5itdFisP9VhJf5+dzwHaUMwyLJxGeObCLlIAhDQyyWAEkjZ1sNL3tQJSjfClDYqXwbUFp4lmLWQ+CeM6lbD6FnydqzGwZ6d5v0TOSrz4RgcCv8BGP2NQYUBto+AGAEDJwzQy0z6Z+vkbqmkkmWkTvRqJsLHTr9laqbZew9/856ohMyq0S3wmGwjhulwoVUspDGUafMGBzrokyBa10AWxXvi94KRn8kxh/z7Jg6PXsmyutI1qejsoFqdeWBeXEP0aTb3a4szZyHxB/RlEWYLZUU/GEpa8/LJ+OS1PneMwwwg3yovBnjFJS/UJPcGxFMOQQgif4H/R+4nE+usj8urQLoKRPrfv+Py47bo/rwNXuK9jQqFgB9bNkWIRQ1U5QSp9SalOxYnuRZiRhzOgRlpCtvv/G2WIkHzP/rbvvXqdexHJ6OrkY7oBvFxsS9JfEpEhMgEMucFQXwhn3le9NFwoZKRhzKozmPrfAg5jncaKxMygs4kCga58nsCdz2BTxa8owjxNA75wzbDDr8nqNt5ULQWRbGSXwM04ofaBh8+9WJpkY8AUE6+E3fltLaHr4lvKT1oj8deNOzqEr8DZhLU+A6Bfia0X1nAa5nNddhLIlT+dljkhFh638bwLqya8l2ZRxd1g9OMA5z2/XrKDOF0BnDEH5Xdc/tKAtoIe1gSiBv9MoeVuVmWr3Ir4yZQNdwKXiE1wXzzzVD3wiwuoIU9nYr/49xcWFfqy41G29guFzRLCJYjbkrpOK56Bq2mNTTgTVh3TMM2sUKU7hdwLOeImXJOOej/kniOnJVvUrMr9IItgKW6KHSlTXVaneJ2YGLi83/lYYt+NdKxenLGhtWEDDCtz6yu82raA4ka0gB3q1wBU8Yv4B8pL6wxZTpeeSTBOA0Xh/9C09Q7bAkBSnhn2v9g40tJvTsv33GZDQrB/F8qdK5rVpYlzhVl4Quc16fKu3Emgrqqkyf80gXFskBM8IF9zbaXfym8PS+02akoAZYOD5prozbGabzKjnzHU9SbzIKMNSGWK5bX6o26jTqcu0qNJxpWaURlh9r1DGSx2az2ambBiH7ddZ2FHQk7GfhWTqm+ab+vPcwtes7JGcj83p02qhZGzRLz X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CY5PR12MB6405.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(366016)(376014)(7416014);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: cYVSP2IXFJLuQfs9ebd34xI/Yw5JI9u1/7RU7+fE5Wos83E7sG+hrH+uYi61yf8DaUWkJUGF6h0vBCv40VBaS0vweYK1Xpt8c09yQiBKnbpCRCbur6Lp9h/lLS9UMugXBnwXZyJ9O/6dkrCkEpEVstBXNk0sOOPhDWykLd84DU3Se2VyUaxMSMvSR6EL9ZGI4wtAVbWEfnQQcZRcgVfMHnJEZAPdXvjlVs+0YNp2unAJ2oKbJByG3d2JVGBSk/rnPpyc1dpHwfuxVWwt5K+bsFYxx5G222w0GVp0HkfLY+Q20PTiSL0uxorVmnxLmw/4LB5kZDm4Gx6pm7YmwjW3NBS+m03wdgmnHJcsP15pxvUjaMIIvNszexD3eH4hvFNYOh0yft1P/emCXv2/DpyNEcD/WDqL6YZgTuJbekAFw9toDtuMrcSjnWdciGdKeS39BOuClh/O9RX5qLxDqvfkMxjlepc6nLdFaYViStvKfbYANK+oQB3Jv93tR9T+Iwt9ECztLyMXOjU739wsnyU+RzAdZ/QaDtBup88DA0GEOQwFPpS0dpDbKVEHm8dtr4jtjPVg4STZxSKSsIB2Q3a+K850Ja8Ha3fbeXPt+5yIgdZhCJEp27Z+sc3qnIL9G+xDiconYCs8hBIOsgkYNk4lpJT80xJ3uJ0sS3rpqo09JLLuUwJrHOfpdAYpsS8GGyE3rK5e82wfxcFWG3F+4qYqErhAu+2ealAAHJGj6QpEx3j2SUpx5S7tOXeOWIXds5uVyVbJr2kFT3iTz/VYLqqDMSWfb6zPySelSFhdZtUEnKT2/lHk6TCvAVUGDRECprLeDK9vx25DG52uBNWiDeVUuB+YJ1237+fHqR+aqDlCKAAAbllY9IPkWJggXineNlZMxybxWWVCIhTwA86oq0EXXPQ9OdPSWQpuc5tGGtE5AWhwlv3sVULbe0N/grVCvq3ylMulgLH8jtG7duTfTaEBbaTVfFP/sjXvNkEhCMFYmilM2AVtjkmcN1xzL/FVPbnhPG4iBV0MAsEXU95sTjYIJA5HTqp6E0v44Z//uW/vLAU+77rUeqgzgxnaAHI4Uy1Oayo/V4xHfTrEkYoWx8/wx3q4aQmkRjVTtndAxNtikL+ZKypWN5g0ytttmn7y12Clz60hVk/I6cJqJBFvpqsFMlhHINhD1nsefRilkUZ0SCuMXqS68DIl3J9L9h+ZIymtJ6sYBiTxUsYGh4297p1T85XXsy2eTH/Eim1yeEv2HQGMPceD6kwBY8ZOD3/PImUM3+VFkMmYhX0T8gqH5/rQ6l4U82MgDyiTIUP4XEvzSDYk4OH7aubIN1SeQ1wxAptR8YveYa8eKY0+mgg3D60vUAF78QRphe4dM2u180b1RfzOnya9LrKsuLYqEKVrSFBmekZ3qy14FFy4GHLVY+GBBXT+fDkNm5owHGUJa2bk/HFufuaq7uGlgT14u2rQV6ep8/vqLK5TlNYOsmashDhR++GehVvbPy8S1WUQxGoGU6mp4leOMs20MbqeVqKLzVXGHJ/CiMNJHi5whLd2SgkVgx+jloRWNV2KmfjuaPt/ssXRlh8anZ0i2qc7b/3t1MM8 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 3ee63f29-8376-4011-19ca-08dd210cd680 X-MS-Exchange-CrossTenant-AuthSource: CY5PR12MB6405.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Dec 2024 15:41:58.0859 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: jOMiYvg9Q7YmgrsvlHJpoinQaPs97R8v2/Et1EfAbh3XRMHU/9sXCmigaUw2KhvRlISkgr1qs4///9i2Be7o7A== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB7149 Add the following kfunc's to provide scx schedulers direct access to per-node idle cpumasks information: const struct cpumask *scx_bpf_get_idle_cpumask_node(int node) const struct cpumask *scx_bpf_get_idle_smtmask_node(int node) s32 scx_bpf_pick_idle_cpu_node(const cpumask_t *cpus_allowed, int node, u64 flags) int scx_bpf_cpu_to_node(s32 cpu) Signed-off-by: Andrea Righi --- kernel/sched/ext_idle.c | 163 ++++++++++++++++++++--- tools/sched_ext/include/scx/common.bpf.h | 4 + tools/sched_ext/include/scx/compat.bpf.h | 19 +++ 3 files changed, 170 insertions(+), 16 deletions(-) diff --git a/kernel/sched/ext_idle.c b/kernel/sched/ext_idle.c index b36e93da1b75..0f8ccc1e290e 100644 --- a/kernel/sched/ext_idle.c +++ b/kernel/sched/ext_idle.c @@ -28,6 +28,60 @@ static bool check_builtin_idle_enabled(void) return false; } +static bool check_builtin_idle_per_node_enabled(void) +{ + if (static_branch_likely(&scx_builtin_idle_per_node)) + return true; + + scx_ops_error("per-node idle tracking is disabled"); + return false; +} + +/* + * Validate and resolve a NUMA node. + * + * Return the resolved node ID on success or a negative value otherwise. + */ +static int validate_node(int node) +{ + if (!check_builtin_idle_per_node_enabled()) + return -EINVAL; + + /* If no node is specified, use the current one */ + if (node == NUMA_NO_NODE) + return numa_node_id(); + + /* Make sure node is in a valid range */ + if (node < 0 || node >= nr_node_ids) { + scx_ops_error("invalid node %d", node); + return -ENOENT; + } + + /* Make sure the node is part of the set of possible nodes */ + if (!node_possible(node)) { + scx_ops_error("unavailable node %d", node); + return -EINVAL; + } + + return node; +} + +/* + * Return the node id associated to a target idle CPU (used to determine + * the proper idle cpumask). + */ +static int idle_cpu_to_node(int cpu) +{ + int node; + + if (static_branch_maybe(CONFIG_NUMA, &scx_builtin_idle_per_node)) + node = cpu_to_node(cpu); + else + node = NUMA_FLAT_NODE; + + return node; +} + #ifdef CONFIG_SMP struct idle_cpumask { cpumask_var_t cpu; @@ -83,22 +137,6 @@ static void idle_masks_init(void) static DEFINE_STATIC_KEY_FALSE(scx_selcpu_topo_llc); -/* - * Return the node id associated to a target idle CPU (used to determine - * the proper idle cpumask). - */ -static int idle_cpu_to_node(int cpu) -{ - int node; - - if (static_branch_maybe(CONFIG_NUMA, &scx_builtin_idle_per_node)) - node = cpu_to_node(cpu); - else - node = NUMA_FLAT_NODE; - - return node; -} - static bool test_and_clear_cpu_idle(int cpu) { int node = idle_cpu_to_node(cpu); @@ -613,6 +651,17 @@ static void reset_idle_masks(void) {} */ __bpf_kfunc_start_defs(); +/** + * scx_bpf_cpu_to_node - Return the NUMA node the given @cpu belongs to + */ +__bpf_kfunc int scx_bpf_cpu_to_node(s32 cpu) +{ + if (cpu < 0 || cpu >= nr_cpu_ids) + return -EINVAL; + + return idle_cpu_to_node(cpu); +} + /** * scx_bpf_select_cpu_dfl - The default implementation of ops.select_cpu() * @p: task_struct to select a CPU for @@ -645,6 +694,28 @@ __bpf_kfunc s32 scx_bpf_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, return prev_cpu; } +/** + * scx_bpf_get_idle_cpumask_node - Get a referenced kptr to the idle-tracking + * per-CPU cpumask of a target NUMA node. + * + * NUMA_NO_NODE is interpreted as the current node. + * + * Returns an empty cpumask if idle tracking is not enabled, if @node is not + * valid, or running on a UP kernel. + */ +__bpf_kfunc const struct cpumask *scx_bpf_get_idle_cpumask_node(int node) +{ + node = validate_node(node); + if (node < 0) + return cpu_none_mask; + +#ifdef CONFIG_SMP + return get_idle_cpumask(node); +#else + return cpu_none_mask; +#endif +} + /** * scx_bpf_get_idle_cpumask - Get a referenced kptr to the idle-tracking * per-CPU cpumask. @@ -664,6 +735,32 @@ __bpf_kfunc const struct cpumask *scx_bpf_get_idle_cpumask(void) return get_idle_cpumask(NUMA_FLAT_NODE); } +/** + * scx_bpf_get_idle_smtmask_node - Get a referenced kptr to the idle-tracking, + * per-physical-core cpumask of a target NUMA node. Can be used to determine + * if an entire physical core is free. + * + * NUMA_NO_NODE is interpreted as the current node. + * + * Returns an empty cpumask if idle tracking is not enabled, if @node is not + * valid, or running on a UP kernel. + */ +__bpf_kfunc const struct cpumask *scx_bpf_get_idle_smtmask_node(int node) +{ + node = validate_node(node); + if (node < 0) + return cpu_none_mask; + +#ifdef CONFIG_SMP + if (sched_smt_active()) + return get_idle_smtmask(node); + else + return get_idle_cpumask(node); +#else + return cpu_none_mask; +#endif +} + /** * scx_bpf_get_idle_smtmask - Get a referenced kptr to the idle-tracking, * per-physical-core cpumask. Can be used to determine if an entire physical @@ -722,6 +819,36 @@ __bpf_kfunc bool scx_bpf_test_and_clear_cpu_idle(s32 cpu) return false; } +/** + * scx_bpf_pick_idle_cpu_node - Pick and claim an idle cpu from a NUMA node + * @cpus_allowed: Allowed cpumask + * @node: target NUMA node + * @flags: %SCX_PICK_IDLE_CPU_* flags + * + * Pick and claim an idle cpu in @cpus_allowed from the NUMA node @node. + * Returns the picked idle cpu number on success. -%EBUSY if no matching cpu + * was found. + * + * If @node is NUMA_NO_NODE, the search is restricted to the current NUMA + * node. Otherwise, the search starts from @node and proceeds to other + * online NUMA nodes in order of increasing distance (unless + * SCX_PICK_IDLE_NODE is specified, in which case the search is limited to + * the target @node). + * + * Unavailable if ops.update_idle() is implemented and + * %SCX_OPS_KEEP_BUILTIN_IDLE is not set or if %SCX_OPS_KEEP_BUILTIN_IDLE is + * not set. + */ +__bpf_kfunc s32 scx_bpf_pick_idle_cpu_node(const struct cpumask *cpus_allowed, + int node, u64 flags) +{ + node = validate_node(node); + if (node < 0) + return node; + + return scx_pick_idle_cpu(cpus_allowed, node, flags); +} + /** * scx_bpf_pick_idle_cpu - Pick and claim an idle cpu * @cpus_allowed: Allowed cpumask @@ -785,11 +912,15 @@ __bpf_kfunc s32 scx_bpf_pick_any_cpu(const struct cpumask *cpus_allowed, __bpf_kfunc_end_defs(); BTF_KFUNCS_START(scx_kfunc_ids_select_cpu) +BTF_ID_FLAGS(func, scx_bpf_cpu_to_node) BTF_ID_FLAGS(func, scx_bpf_select_cpu_dfl, KF_RCU) +BTF_ID_FLAGS(func, scx_bpf_get_idle_cpumask_node, KF_ACQUIRE) BTF_ID_FLAGS(func, scx_bpf_get_idle_cpumask, KF_ACQUIRE) +BTF_ID_FLAGS(func, scx_bpf_get_idle_smtmask_node, KF_ACQUIRE) BTF_ID_FLAGS(func, scx_bpf_get_idle_smtmask, KF_ACQUIRE) BTF_ID_FLAGS(func, scx_bpf_put_idle_cpumask, KF_RELEASE) BTF_ID_FLAGS(func, scx_bpf_test_and_clear_cpu_idle) +BTF_ID_FLAGS(func, scx_bpf_pick_idle_cpu_node, KF_RCU) BTF_ID_FLAGS(func, scx_bpf_pick_idle_cpu, KF_RCU) BTF_ID_FLAGS(func, scx_bpf_pick_any_cpu, KF_RCU) BTF_KFUNCS_END(scx_kfunc_ids_select_cpu) diff --git a/tools/sched_ext/include/scx/common.bpf.h b/tools/sched_ext/include/scx/common.bpf.h index 858ba1f438f6..fe0433f7c4d9 100644 --- a/tools/sched_ext/include/scx/common.bpf.h +++ b/tools/sched_ext/include/scx/common.bpf.h @@ -63,13 +63,17 @@ u32 scx_bpf_cpuperf_cap(s32 cpu) __ksym __weak; u32 scx_bpf_cpuperf_cur(s32 cpu) __ksym __weak; void scx_bpf_cpuperf_set(s32 cpu, u32 perf) __ksym __weak; u32 scx_bpf_nr_cpu_ids(void) __ksym __weak; +int scx_bpf_cpu_to_node(s32 cpu) __ksym __weak; const struct cpumask *scx_bpf_get_possible_cpumask(void) __ksym __weak; const struct cpumask *scx_bpf_get_online_cpumask(void) __ksym __weak; void scx_bpf_put_cpumask(const struct cpumask *cpumask) __ksym __weak; +const struct cpumask *scx_bpf_get_idle_cpumask_node(int node) __ksym __weak; const struct cpumask *scx_bpf_get_idle_cpumask(void) __ksym; +const struct cpumask *scx_bpf_get_idle_smtmask_node(int node) __ksym __weak; const struct cpumask *scx_bpf_get_idle_smtmask(void) __ksym; void scx_bpf_put_idle_cpumask(const struct cpumask *cpumask) __ksym; bool scx_bpf_test_and_clear_cpu_idle(s32 cpu) __ksym; +s32 scx_bpf_pick_idle_cpu_node(const cpumask_t *cpus_allowed, int node, u64 flags) __ksym __weak; s32 scx_bpf_pick_idle_cpu(const cpumask_t *cpus_allowed, u64 flags) __ksym; s32 scx_bpf_pick_any_cpu(const cpumask_t *cpus_allowed, u64 flags) __ksym; bool scx_bpf_task_running(const struct task_struct *p) __ksym; diff --git a/tools/sched_ext/include/scx/compat.bpf.h b/tools/sched_ext/include/scx/compat.bpf.h index d56520100a26..dfc329d5a91e 100644 --- a/tools/sched_ext/include/scx/compat.bpf.h +++ b/tools/sched_ext/include/scx/compat.bpf.h @@ -125,6 +125,25 @@ bool scx_bpf_dispatch_vtime_from_dsq___compat(struct bpf_iter_scx_dsq *it__iter, false; \ }) +#define __COMPAT_scx_bpf_cpu_to_node(cpu) \ + (bpf_ksym_exists(scx_bpf_cpu_to_node) ? \ + scx_bpf_cpu_to_node(cpu) : 0) + +#define __COMPAT_scx_bpf_get_idle_cpumask_node(node) \ + (bpf_ksym_exists(scx_bpf_get_idle_cpumask_node) ? \ + scx_bpf_get_idle_cpumask_node(node) : \ + scx_bpf_get_idle_cpumask()) \ + +#define __COMPAT_scx_bpf_get_idle_smtmask_node(node) \ + (bpf_ksym_exists(scx_bpf_get_idle_smtmask_node) ? \ + scx_bpf_get_idle_smtmask_node(node) : \ + scx_bpf_get_idle_smtmask()) + +#define __COMPAT_scx_bpf_pick_idle_cpu_node(cpus_allowed, node, flags) \ + (bpf_ksym_exists(scx_bpf_pick_idle_cpu_node) ? \ + scx_bpf_pick_idle_cpu_node(cpus_allowed, node, flags) : \ + scx_bpf_pick_idle_cpu(cpus_allowed, flags)) + /* * Define sched_ext_ops. This may be expanded to define multiple variants for * backward compatibility. See compat.h::SCX_OPS_LOAD/ATTACH().