From patchwork Mon Nov 28 13:20:39 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ricardo Neri X-Patchwork-Id: 13057478 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D76F6C433FE for ; Mon, 28 Nov 2022 13:13:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231675AbiK1NNp (ORCPT ); Mon, 28 Nov 2022 08:13:45 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39936 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231656AbiK1NNd (ORCPT ); Mon, 28 Nov 2022 08:13:33 -0500 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5727713D5E; Mon, 28 Nov 2022 05:13:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1669641212; x=1701177212; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=bVqpVo1Q9MYSWk8Td5o+e8rn4ip/1+KMBJaJ/M8s+/Y=; b=U5vi/ORGokvMveSG0Qai9S11M08uOBRjdSHA8jXTTzLa/lPDNCuT7VvG 8B9LKyJjj3eCneQEjqVHlJNnyHMu2TGaEC8GGCNV5yTk4KHa++thsMMeb l5iCSupGVtJK7hfRJqg6g0EPqyy2ZI60zCZRjn8tZu62mSuhuZeRaBYbm fmNEFhICVDfE3Co+d5zeqdzCqk47bwvo8zk1Ff+ryj+1b69WPwhxi79Mk 34f1KPLeonfpCRCiQwyYtvxX8sENJcSDv2cDCNqvRfPErv12Jn7FFtfA3 X3ZuadxzzzstMV4a3hPI7mZIy5kQ03No4d1QBqNJ30D/3GNT/XVn9C+eo g==; X-IronPort-AV: E=McAfee;i="6500,9779,10544"; a="401117058" X-IronPort-AV: E=Sophos;i="5.96,200,1665471600"; d="scan'208";a="401117058" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Nov 2022 05:13:31 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10544"; a="749381321" X-IronPort-AV: E=Sophos;i="5.96,200,1665471600"; d="scan'208";a="749381321" Received: from ranerica-svr.sc.intel.com ([172.25.110.23]) by fmsmga002.fm.intel.com with ESMTP; 28 Nov 2022 05:13:31 -0800 From: Ricardo Neri To: "Peter Zijlstra (Intel)" , Juri Lelli , Vincent Guittot Cc: Ricardo Neri , "Ravi V. Shankar" , Ben Segall , Daniel Bristot de Oliveira , Dietmar Eggemann , Len Brown , Mel Gorman , "Rafael J. Wysocki" , Srinivas Pandruvada , Steven Rostedt , Tim Chen , Valentin Schneider , x86@kernel.org, "Joel Fernandes (Google)" , linux-kernel@vger.kernel.org, linux-pm@vger.kernel.org, Ricardo Neri , "Tim C . Chen" Subject: [PATCH v2 01/22] sched/task_struct: Introduce IPC classes of tasks Date: Mon, 28 Nov 2022 05:20:39 -0800 Message-Id: <20221128132100.30253-2-ricardo.neri-calderon@linux.intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20221128132100.30253-1-ricardo.neri-calderon@linux.intel.com> References: <20221128132100.30253-1-ricardo.neri-calderon@linux.intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org On hybrid processors, the architecture differences between the types of CPUs lead to different instructions-per-cycle (IPC) on each type of CPU. IPCs may differ further by the type of instructions. Instructions can be grouped into classes of similar IPCs. Hence, tasks can be classified into groups based on the type of instructions they execute. Add a new member task_struct::ipcc to associate a particular task to an IPC class that depends on the instructions it executes. The scheduler may use the IPC class of a task and data about the performance among CPUs of a given IPC class to improve throughput. It may, for instance, place certain classes of tasks on CPUs of higher performance. The methods to determine the classification of a task and its relative IPC score are specific to each CPU architecture. Cc: Ben Segall Cc: Daniel Bristot de Oliveira Cc: Dietmar Eggemann Cc: Joel Fernandes (Google) Cc: Len Brown Cc: Mel Gorman Cc: Rafael J. Wysocki Cc: Srinivas Pandruvada Cc: Steven Rostedt Cc: Tim C. Chen Cc: Valentin Schneider Cc: x86@kernel.org Cc: linux-pm@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Ricardo Neri --- Changes since v1: * Renamed task_struct::class as task::struct_ipcc. (Joel) * Use task_struct::ipcc = 0 for unclassified tasks. (PeterZ) * Renamed CONFIG_SCHED_TASK_CLASSES as CONFIG_IPC_CLASSES. (PeterZ, Joel) --- include/linux/sched.h | 10 ++++++++++ init/Kconfig | 12 ++++++++++++ 2 files changed, 22 insertions(+) diff --git a/include/linux/sched.h b/include/linux/sched.h index 68c07ae0d7ff..47ae3557ba07 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -127,6 +127,8 @@ struct task_group; __TASK_TRACED | EXIT_DEAD | EXIT_ZOMBIE | \ TASK_PARKED) +#define IPC_CLASS_UNCLASSIFIED 0 + #define task_is_running(task) (READ_ONCE((task)->__state) == TASK_RUNNING) #define task_is_traced(task) ((READ_ONCE(task->jobctl) & JOBCTL_TRACED) != 0) @@ -1525,6 +1527,14 @@ struct task_struct { union rv_task_monitor rv[RV_PER_TASK_MONITORS]; #endif +#ifdef CONFIG_IPC_CLASSES + /* + * A hardware-defined classification of task based on the number + * of instructions per cycle. + */ + unsigned int ipcc; +#endif + /* * New fields for task_struct should be added above here, so that * they are included in the randomized portion of task_struct. diff --git a/init/Kconfig b/init/Kconfig index abf65098f1b6..cd17dd4d3718 100644 --- a/init/Kconfig +++ b/init/Kconfig @@ -865,6 +865,18 @@ config UCLAMP_BUCKETS_COUNT If in doubt, use the default value. +config IPC_CLASSES + bool "IPC classes of tasks" + depends on SMP + help + If selected, each task is assigned a classification value that + reflects the type of instructions that the task executes. This + classification reflects but is not equal to the number of + instructions retired per cycle. + + The scheduler uses the classification value to improve the placement + of tasks. + endmenu # From patchwork Mon Nov 28 13:20:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ricardo Neri X-Patchwork-Id: 13057482 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 22FD5C46467 for ; Mon, 28 Nov 2022 13:13:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231701AbiK1NNt (ORCPT ); Mon, 28 Nov 2022 08:13:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39378 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231453AbiK1NNg (ORCPT ); Mon, 28 Nov 2022 08:13:36 -0500 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E22F71CB36; Mon, 28 Nov 2022 05:13:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1669641214; x=1701177214; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=PygCb/lFlxpA6HfJbBXFOfXsvHEBrsUtKwJwsV3g7Oo=; b=MB5KWN8Hs73tFgcXJ1rUQbxcugySDvxKDxIqy2ada6vR8zTy5Ce63b0s WvuhiLAgTbZYjYeSIu3yji1SNy2hVhWrUY7txId5iRlkKu520H7PEJ44b HuM1Ry7dltAUyRt+RmyqhcviLeTFMZkDq8Rr81v8/DLANjSyrrw2lKzmX gvqoowsX56aTuzSc3e/LdbOCpZJiC+YrwPMg38U9KBnmBBxfzSsnb5D12 XJB4EbUEOHYZ6cZxiD8s8nIOyKHanDeddF4yPGYLv3guBQWtaRS3gToPM yreGGCV/0xpbRcXIhcx1K6ENWrkkLjlv9r2yts7xF/vDc4synfkRJ4zFh Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10544"; a="401117061" X-IronPort-AV: E=Sophos;i="5.96,200,1665471600"; d="scan'208";a="401117061" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Nov 2022 05:13:31 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10544"; a="749381327" X-IronPort-AV: E=Sophos;i="5.96,200,1665471600"; d="scan'208";a="749381327" Received: from ranerica-svr.sc.intel.com ([172.25.110.23]) by fmsmga002.fm.intel.com with ESMTP; 28 Nov 2022 05:13:31 -0800 From: Ricardo Neri To: "Peter Zijlstra (Intel)" , Juri Lelli , Vincent Guittot Cc: Ricardo Neri , "Ravi V. Shankar" , Ben Segall , Daniel Bristot de Oliveira , Dietmar Eggemann , Len Brown , Mel Gorman , "Rafael J. Wysocki" , Srinivas Pandruvada , Steven Rostedt , Tim Chen , Valentin Schneider , x86@kernel.org, "Joel Fernandes (Google)" , linux-kernel@vger.kernel.org, linux-pm@vger.kernel.org, Ricardo Neri , "Tim C . Chen" Subject: [PATCH v2 02/22] sched: Add interfaces for IPC classes Date: Mon, 28 Nov 2022 05:20:40 -0800 Message-Id: <20221128132100.30253-3-ricardo.neri-calderon@linux.intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20221128132100.30253-1-ricardo.neri-calderon@linux.intel.com> References: <20221128132100.30253-1-ricardo.neri-calderon@linux.intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org Add the interfaces that architectures shall implement to convey the data to support IPC classes. arch_update_ipcc() updates the IPC classification of the current task as given by hardware. arch_get_ipcc_score() provides a performance score for a given IPC class when placed on a specific CPU. Higher scores indicate higher performance. The number of classes and the score of each class of task are determined by hardware. Cc: Ben Segall Cc: Daniel Bristot de Oliveira Cc: Dietmar Eggemann Cc: Joel Fernandes (Google) Cc: Len Brown Cc: Mel Gorman Cc: Rafael J. Wysocki Cc: Srinivas Pandruvada Cc: Steven Rostedt Cc: Tim C. Chen Cc: Valentin Schneider Cc: x86@kernel.org Cc: linux-pm@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Ricardo Neri --- Changes since v1: * Shortened the names of the IPCC interfaces (PeterZ): sched_task_classes_enabled >> sched_ipcc_enabled arch_has_task_classes >> arch_has_ipc_classes arch_update_task_class >> arch_update_ipcc arch_get_task_class_score >> arch_get_ipcc_score * Removed smt_siblings_idle argument from arch_update_ipcc(). (PeterZ) --- kernel/sched/sched.h | 60 +++++++++++++++++++++++++++++++++++++++++ kernel/sched/topology.c | 8 ++++++ 2 files changed, 68 insertions(+) diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index b1d338a740e5..75e22baa2622 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -2531,6 +2531,66 @@ void arch_scale_freq_tick(void) } #endif +#ifdef CONFIG_IPC_CLASSES +DECLARE_STATIC_KEY_FALSE(sched_ipcc); + +static inline bool sched_ipcc_enabled(void) +{ + return static_branch_unlikely(&sched_ipcc); +} + +#ifndef arch_has_ipc_classes +/** + * arch_has_ipc_classes() - Check whether hardware supports IPC classes of tasks + * + * Returns: true of IPC classes of tasks are supported. + */ +static __always_inline +bool arch_has_ipc_classes(void) +{ + return false; +} +#endif + +#ifndef arch_update_ipcc +/** + * arch_update_ipcc() - Update the IPC class of the current task + * @curr: The current task + * + * Request that the IPC classification of @curr is updated. + * + * Returns: none + */ +static __always_inline +void arch_update_ipcc(struct task_struct *curr) +{ +} +#endif + +#ifndef arch_get_ipcc_score +/** + * arch_get_ipcc_score() - Get the IPC score of a class of task + * @ipcc: The IPC class + * @cpu: A CPU number + * + * Returns the performance score of an IPC class when running on @cpu. + * Error when either @class or @cpu are invalid. + */ +static __always_inline +int arch_get_ipcc_score(unsigned short ipcc, int cpu) +{ + return 1; +} +#endif +#else /* CONFIG_IPC_CLASSES */ + +#define arch_get_ipcc_score(ipcc, cpu) (-EINVAL) +#define arch_update_ipcc(curr) + +static inline bool sched_ipcc_enabled(void) { return false; } + +#endif /* CONFIG_IPC_CLASSES */ + #ifndef arch_scale_freq_capacity /** * arch_scale_freq_capacity - get the frequency scale factor of a given CPU. diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c index 8154ef590b9f..eb1654b64df7 100644 --- a/kernel/sched/topology.c +++ b/kernel/sched/topology.c @@ -669,6 +669,9 @@ DEFINE_PER_CPU(struct sched_domain __rcu *, sd_numa); DEFINE_PER_CPU(struct sched_domain __rcu *, sd_asym_packing); DEFINE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity); DEFINE_STATIC_KEY_FALSE(sched_asym_cpucapacity); +#ifdef CONFIG_IPC_CLASSES +DEFINE_STATIC_KEY_FALSE(sched_ipcc); +#endif static void update_top_cache_domain(int cpu) { @@ -2388,6 +2391,11 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att if (has_asym) static_branch_inc_cpuslocked(&sched_asym_cpucapacity); +#ifdef CONFIG_IPC_CLASSES + if (arch_has_ipc_classes()) + static_branch_enable_cpuslocked(&sched_ipcc); +#endif + if (rq && sched_debug_verbose) { pr_info("root domain span: %*pbl (max cpu_capacity = %lu)\n", cpumask_pr_args(cpu_map), rq->rd->max_cpu_capacity); From patchwork Mon Nov 28 13:20:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ricardo Neri X-Patchwork-Id: 13057479 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 059DBC43217 for ; Mon, 28 Nov 2022 13:13:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231432AbiK1NNr (ORCPT ); Mon, 28 Nov 2022 08:13:47 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39028 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231624AbiK1NNf (ORCPT ); Mon, 28 Nov 2022 08:13:35 -0500 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 90105120A3; Mon, 28 Nov 2022 05:13:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1669641213; x=1701177213; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=SGRJsWlMFBDt9YHKWDWh4VtdInjeycHfz68CW0YMoNE=; b=V03l0WTHV5S3FfrqNguzqJB+2yysCVjerBdkNqpNDS0sQrIsS8sZLIaN sSyY2JCBkkPJILUo6kgK304SCUfnjw/Dp4bTn8MgUagOG88dI4R9QQ3pA +9Ym90KFbjrggAafgIiATCA/E7bLKHEHiFn6wHUEXat2Eh/7AW01OpNkB Hp9Y8aH3r/CR5X9ZjydRdr4p2PB/dCa/eLjAHKNVk1DWptAwI+Q9a/ylf m9qfr6Gwcas3FbwWWARjUrG8GmcHV2TT4lfA/xlh1s4Vm33LIMghHmnes 6wI1QXQmUmCLC1Khr1ydkrDAjT5ioPmo0nDJV1apxSmlupUG0bmFcSSgY w==; X-IronPort-AV: E=McAfee;i="6500,9779,10544"; a="401117071" X-IronPort-AV: E=Sophos;i="5.96,200,1665471600"; d="scan'208";a="401117071" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Nov 2022 05:13:32 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10544"; a="749381332" X-IronPort-AV: E=Sophos;i="5.96,200,1665471600"; d="scan'208";a="749381332" Received: from ranerica-svr.sc.intel.com ([172.25.110.23]) by fmsmga002.fm.intel.com with ESMTP; 28 Nov 2022 05:13:31 -0800 From: Ricardo Neri To: "Peter Zijlstra (Intel)" , Juri Lelli , Vincent Guittot Cc: Ricardo Neri , "Ravi V. Shankar" , Ben Segall , Daniel Bristot de Oliveira , Dietmar Eggemann , Len Brown , Mel Gorman , "Rafael J. Wysocki" , Srinivas Pandruvada , Steven Rostedt , Tim Chen , Valentin Schneider , x86@kernel.org, "Joel Fernandes (Google)" , linux-kernel@vger.kernel.org, linux-pm@vger.kernel.org, Ricardo Neri , "Tim C . Chen" Subject: [PATCH v2 03/22] sched/core: Initialize the IPC class of a new task Date: Mon, 28 Nov 2022 05:20:41 -0800 Message-Id: <20221128132100.30253-4-ricardo.neri-calderon@linux.intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20221128132100.30253-1-ricardo.neri-calderon@linux.intel.com> References: <20221128132100.30253-1-ricardo.neri-calderon@linux.intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org New tasks shall start life as unclassified. They will be classified by hardware when they run. Cc: Ben Segall Cc: Daniel Bristot de Oliveira Cc: Dietmar Eggemann Cc: Joel Fernandes (Google) Cc: Len Brown Cc: Mel Gorman Cc: Rafael J. Wysocki Cc: Srinivas Pandruvada Cc: Steven Rostedt Cc: Tim C. Chen Cc: Valentin Schneider Cc: x86@kernel.org Cc: linux-pm@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Ricardo Neri --- Changes since v1: * None --- kernel/sched/core.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 78b2d5cabcc5..8dd43ee05534 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -4372,6 +4372,9 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p) p->se.prev_sum_exec_runtime = 0; p->se.nr_migrations = 0; p->se.vruntime = 0; +#ifdef CONFIG_IPC_CLASSES + p->ipcc = IPC_CLASS_UNCLASSIFIED; +#endif INIT_LIST_HEAD(&p->se.group_node); #ifdef CONFIG_FAIR_GROUP_SCHED From patchwork Mon Nov 28 13:20:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ricardo Neri X-Patchwork-Id: 13057483 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AC9B0C4708C for ; Mon, 28 Nov 2022 13:13:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231638AbiK1NNt (ORCPT ); Mon, 28 Nov 2022 08:13:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40018 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231510AbiK1NNh (ORCPT ); Mon, 28 Nov 2022 08:13:37 -0500 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 43DDB1CFCF; Mon, 28 Nov 2022 05:13:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1669641215; x=1701177215; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=C8o1EqWi47EMS3NLEboiqwaO6Cqj8PqboSa/beXp5NY=; b=CIquFBhoaOO+JcnTLi7Rt1q5gZb9/AhDXDPV2ooUICbRVFq3AHQFlfSD WoyyIfAMrneTWaWYNTpYjfhLsB+y/nk+cVRBnNqKoOp2lOgePfoZGw9Eo 9ysUk4bmTWUaPYoxRJ/QPmaY4QPZ7LCBei/RpGL09Wzzq62QTeGvXUQH0 VEWBACqkmUvgwONsKSMr/FNFreA2nu+2lwo2Gjv6GjtpINguY9QUm++bO St+BlwWawy1G8AhdGmg491/qACjerpzgug7iF9Tz4rIk2fSJL0eQ1Y0zn kU2NA5JTmZF5apbH9lPXpX6NvQiyrZHSNES3n2f1Rh8P7w0QwrjnBT+K9 g==; X-IronPort-AV: E=McAfee;i="6500,9779,10544"; a="401117084" X-IronPort-AV: E=Sophos;i="5.96,200,1665471600"; d="scan'208";a="401117084" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Nov 2022 05:13:32 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10544"; a="749381335" X-IronPort-AV: E=Sophos;i="5.96,200,1665471600"; d="scan'208";a="749381335" Received: from ranerica-svr.sc.intel.com ([172.25.110.23]) by fmsmga002.fm.intel.com with ESMTP; 28 Nov 2022 05:13:32 -0800 From: Ricardo Neri To: "Peter Zijlstra (Intel)" , Juri Lelli , Vincent Guittot Cc: Ricardo Neri , "Ravi V. Shankar" , Ben Segall , Daniel Bristot de Oliveira , Dietmar Eggemann , Len Brown , Mel Gorman , "Rafael J. Wysocki" , Srinivas Pandruvada , Steven Rostedt , Tim Chen , Valentin Schneider , x86@kernel.org, "Joel Fernandes (Google)" , linux-kernel@vger.kernel.org, linux-pm@vger.kernel.org, Ricardo Neri , "Tim C . Chen" Subject: [PATCH v2 04/22] sched/core: Add user_tick as argument to scheduler_tick() Date: Mon, 28 Nov 2022 05:20:42 -0800 Message-Id: <20221128132100.30253-5-ricardo.neri-calderon@linux.intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20221128132100.30253-1-ricardo.neri-calderon@linux.intel.com> References: <20221128132100.30253-1-ricardo.neri-calderon@linux.intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org Differentiate between user and kernel ticks so that the scheduler updates the IPC class of the current task during the latter. Cc: Ben Segall Cc: Daniel Bristot de Oliveira Cc: Dietmar Eggemann Cc: Joel Fernandes (Google) Cc: Len Brown Cc: Mel Gorman Cc: Rafael J. Wysocki Cc: Srinivas Pandruvada Cc: Steven Rostedt Cc: Tim C. Chen Cc: Valentin Schneider Cc: x86@kernel.org Cc: linux-pm@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Ricardo Neri --- Changes since v1: * None --- include/linux/sched.h | 2 +- kernel/sched/core.c | 2 +- kernel/time/timer.c | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 47ae3557ba07..ddabc7449edd 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -293,7 +293,7 @@ enum { TASK_COMM_LEN = 16, }; -extern void scheduler_tick(void); +extern void scheduler_tick(bool user_tick); #define MAX_SCHEDULE_TIMEOUT LONG_MAX diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 8dd43ee05534..8bb6f597c42b 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -5487,7 +5487,7 @@ static inline u64 cpu_resched_latency(struct rq *rq) { return 0; } * This function gets called by the timer code, with HZ frequency. * We call it with interrupts disabled. */ -void scheduler_tick(void) +void scheduler_tick(bool user_tick) { int cpu = smp_processor_id(); struct rq *rq = cpu_rq(cpu); diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 63a8ce7177dd..e15e24105891 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -2073,7 +2073,7 @@ void update_process_times(int user_tick) if (in_irq()) irq_work_tick(); #endif - scheduler_tick(); + scheduler_tick(user_tick); if (IS_ENABLED(CONFIG_POSIX_TIMERS)) run_posix_cpu_timers(); } From patchwork Mon Nov 28 13:20:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ricardo Neri X-Patchwork-Id: 13057481 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 67B92C433FE for ; Mon, 28 Nov 2022 13:13:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231699AbiK1NNs (ORCPT ); Mon, 28 Nov 2022 08:13:48 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39380 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231470AbiK1NNg (ORCPT ); Mon, 28 Nov 2022 08:13:36 -0500 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 30DEFE66; Mon, 28 Nov 2022 05:13:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1669641215; x=1701177215; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=IPfAz2o04h3wMx9MPOZMNtyWtq/ctVtFFpyHR+YbVE4=; b=MFqN/Seue60Te17UU6g1FX7dvMqWh65ZFIrG2uEvFCXNdZg+iMovFME0 aoiqctpQnp2v+2eGPxRaiZVWzq3FsUh1kE2qwAw4DpOXYdCoPbnEEpKuu K7iRKfzBJQnovxtDG9zZ560tt6wogH3U4R/kjTND/cWPopURCK4PFB1rC GA3marC4rnpWf9pQODn8DBQHUNMdZboRjCj/wlKvAfRtDaIY1IWvgrDhC iAqtbXGDZSzo0IIq3qcOvy+Ui+NeQadxix1PtZEk2Jf+yo4baPCIpwYSL fxHOHSXSmOwfde7g6Ihtw8E4IQ4LGqPbTeRdBBabhKapvm7wzJbO5r1Kf w==; X-IronPort-AV: E=McAfee;i="6500,9779,10544"; a="401117095" X-IronPort-AV: E=Sophos;i="5.96,200,1665471600"; d="scan'208";a="401117095" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Nov 2022 05:13:32 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10544"; a="749381339" X-IronPort-AV: E=Sophos;i="5.96,200,1665471600"; d="scan'208";a="749381339" Received: from ranerica-svr.sc.intel.com ([172.25.110.23]) by fmsmga002.fm.intel.com with ESMTP; 28 Nov 2022 05:13:32 -0800 From: Ricardo Neri To: "Peter Zijlstra (Intel)" , Juri Lelli , Vincent Guittot Cc: Ricardo Neri , "Ravi V. Shankar" , Ben Segall , Daniel Bristot de Oliveira , Dietmar Eggemann , Len Brown , Mel Gorman , "Rafael J. Wysocki" , Srinivas Pandruvada , Steven Rostedt , Tim Chen , Valentin Schneider , x86@kernel.org, "Joel Fernandes (Google)" , linux-kernel@vger.kernel.org, linux-pm@vger.kernel.org, Ricardo Neri , "Tim C . Chen" Subject: [PATCH v2 05/22] sched/core: Update the IPC class of the current task Date: Mon, 28 Nov 2022 05:20:43 -0800 Message-Id: <20221128132100.30253-6-ricardo.neri-calderon@linux.intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20221128132100.30253-1-ricardo.neri-calderon@linux.intel.com> References: <20221128132100.30253-1-ricardo.neri-calderon@linux.intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org When supported, hardware monitors the instruction stream to classify the current task. Hence, at userspace tick, we are ready to read the most recent classification result for the current task. Cc: Ben Segall Cc: Daniel Bristot de Oliveira Cc: Dietmar Eggemann Cc: Joel Fernandes (Google) Cc: Len Brown Cc: Mel Gorman Cc: Rafael J. Wysocki Cc: Srinivas Pandruvada Cc: Steven Rostedt Cc: Tim C. Chen Cc: Valentin Schneider Cc: x86@kernel.org Cc: linux-pm@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Ricardo Neri --- Changes since v1: * Removed argument smt_siblings_idle from call to arch_ipcc_update(). * Used the new IPCC interfaces names. --- kernel/sched/core.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 8bb6f597c42b..2cd409536b72 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -5496,6 +5496,9 @@ void scheduler_tick(bool user_tick) unsigned long thermal_pressure; u64 resched_latency; + if (sched_ipcc_enabled() && user_tick) + arch_update_ipcc(curr); + arch_scale_freq_tick(); sched_clock_tick(); From patchwork Mon Nov 28 13:20:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ricardo Neri X-Patchwork-Id: 13057486 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7CD09C43217 for ; Mon, 28 Nov 2022 13:14:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231764AbiK1NN6 (ORCPT ); Mon, 28 Nov 2022 08:13:58 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40088 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231660AbiK1NNi (ORCPT ); Mon, 28 Nov 2022 08:13:38 -0500 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 80F121D675; Mon, 28 Nov 2022 05:13:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1669641217; x=1701177217; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=bGQyQKNPbV4jkYdbDYkNg5GZCIoRHz5qvQ3UspaGSq0=; b=aS5gpUB51cG2KwtKRyqqMPr/zzbMb3muopZIkuvj2n9GJlu08ulRQAVM sp/YONifCZxsl+vDaQaHzCMLfWRCxyvsKL9MBdwNKXxMQJNZikhQ58SgZ 8YtqjHIxJ2Nn9PegIYdj7xa3Qi9fnWrv9OvRxJ/hFIGEQThkBC7Y4PigB 5TtYthlWfGoAzzCyXn47u6Gb5yn3Z9bWzaC3kf5Uax9XOm0ZdqkfZdrGp 7hCFNA5sKqV3pJmdBIyArMg6TUnW7W2F+0QIETBZGZwISm7/d0BT+ikr3 AGiZ8qPbIRU9uCIcyGKtuKQ6WoXZntzLVhTRezo3FoPV84ak0JiRNXPk/ Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10544"; a="401117105" X-IronPort-AV: E=Sophos;i="5.96,200,1665471600"; d="scan'208";a="401117105" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Nov 2022 05:13:32 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10544"; a="749381342" X-IronPort-AV: E=Sophos;i="5.96,200,1665471600"; d="scan'208";a="749381342" Received: from ranerica-svr.sc.intel.com ([172.25.110.23]) by fmsmga002.fm.intel.com with ESMTP; 28 Nov 2022 05:13:32 -0800 From: Ricardo Neri To: "Peter Zijlstra (Intel)" , Juri Lelli , Vincent Guittot Cc: Ricardo Neri , "Ravi V. Shankar" , Ben Segall , Daniel Bristot de Oliveira , Dietmar Eggemann , Len Brown , Mel Gorman , "Rafael J. Wysocki" , Srinivas Pandruvada , Steven Rostedt , Tim Chen , Valentin Schneider , x86@kernel.org, "Joel Fernandes (Google)" , linux-kernel@vger.kernel.org, linux-pm@vger.kernel.org, Ricardo Neri , "Tim C . Chen" Subject: [PATCH v2 06/22] sched/fair: Collect load-balancing stats for IPC classes Date: Mon, 28 Nov 2022 05:20:44 -0800 Message-Id: <20221128132100.30253-7-ricardo.neri-calderon@linux.intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20221128132100.30253-1-ricardo.neri-calderon@linux.intel.com> References: <20221128132100.30253-1-ricardo.neri-calderon@linux.intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org When selecting a busiest scheduling group, the IPC class of the current task can be used to select between two scheduling groups of equal asym_packing priority and number of running tasks. Compute a new IPC class performance score for a scheduling group. It is the sum of the performance of the current tasks of all the runqueues. Also, keep track of the task with the lowest IPC class score on the scheduling group. These two metrics will be used during idle load balancing to compute the current and the prospective task-class performance of a scheduling group. Cc: Ben Segall Cc: Daniel Bristot de Oliveira Cc: Dietmar Eggemann Cc: Joel Fernandes (Google) Cc: Len Brown Cc: Mel Gorman Cc: Rafael J. Wysocki Cc: Srinivas Pandruvada Cc: Steven Rostedt Cc: Tim C. Chen Cc: Valentin Schneider Cc: x86@kernel.org Cc: linux-pm@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Ricardo Neri --- Changes since v1: * Implemented cleanups and reworks from PeterZ. Thanks! * Used the new interface names. --- kernel/sched/fair.c | 55 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 224107278471..3a1d6c50a19b 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -9100,6 +9100,57 @@ group_type group_classify(unsigned int imbalance_pct, return group_has_spare; } +struct sg_lb_ipcc_stats { + int min_score; /* Min(score(rq->curr->ipcc)) */ + int min_ipcc; /* Min(rq->curr->ipcc) */ + long sum_score; /* Sum(score(rq->curr->ipcc)) */ +}; + +#ifdef CONFIG_IPC_CLASSES +static void init_rq_ipcc_stats(struct sg_lb_ipcc_stats *sgcs) +{ + *sgcs = (struct sg_lb_ipcc_stats) { + .min_score = INT_MAX, + }; +} + +/** Called only if cpu_of(@rq) is not idle and has tasks running. */ +static void update_sg_lb_ipcc_stats(struct sg_lb_ipcc_stats *sgcs, + struct rq *rq) +{ + struct task_struct *curr; + unsigned short ipcc; + int score; + + if (!sched_ipcc_enabled()) + return; + + curr = rcu_dereference(rq->curr); + if (!curr || (curr->flags & PF_EXITING) || is_idle_task(curr)) + return; + + ipcc = curr->ipcc; + score = arch_get_ipcc_score(ipcc, cpu_of(rq)); + + sgcs->sum_score += score; + + if (score < sgcs->min_score) { + sgcs->min_score = score; + sgcs->min_ipcc = ipcc; + } +} + +#else /* CONFIG_IPC_CLASSES */ +static void update_sg_lb_ipcc_stats(struct sg_lb_ipcc_stats *sgcs, + struct rq *rq) +{ +} + +static void init_rq_ipcc_stats(struct sg_lb_ipcc_stats *class_sgs) +{ +} +#endif /* CONFIG_IPC_CLASSES */ + /** * asym_smt_can_pull_tasks - Check whether the load balancing CPU can pull tasks * @dst_cpu: Destination CPU of the load balancing @@ -9212,9 +9263,11 @@ static inline void update_sg_lb_stats(struct lb_env *env, struct sg_lb_stats *sgs, int *sg_status) { + struct sg_lb_ipcc_stats sgcs; int i, nr_running, local_group; memset(sgs, 0, sizeof(*sgs)); + init_rq_ipcc_stats(&sgcs); local_group = group == sds->local; @@ -9264,6 +9317,8 @@ static inline void update_sg_lb_stats(struct lb_env *env, if (sgs->group_misfit_task_load < load) sgs->group_misfit_task_load = load; } + + update_sg_lb_ipcc_stats(&sgcs, rq); } sgs->group_capacity = group->sgc->capacity; From patchwork Mon Nov 28 13:20:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ricardo Neri X-Patchwork-Id: 13057484 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3205EC433FE for ; Mon, 28 Nov 2022 13:13:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231708AbiK1NNu (ORCPT ); Mon, 28 Nov 2022 08:13:50 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40078 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231657AbiK1NNi (ORCPT ); Mon, 28 Nov 2022 08:13:38 -0500 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 80F671D676; Mon, 28 Nov 2022 05:13:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1669641217; x=1701177217; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=2uPfNNplw7xOwTi269F+VNr3fJJfs2c0TJ+7Nw/WlL4=; b=W2mD71dOmlZY9WM53EkJ/Uy9G7hyZOgGnRJ14qwTAHhYW/RLkVjqhltN CuKkQs3ExkJPIt4eO0JvXCYhXHVkpoh8vBDkouHeSKDrzRmO7F0LsX0Wb dWiEwUb+UKsNOXNloc8CSUTIgZ7pJ3PhTtdrM+NN8opTVzbjye6aTVkSe BqnP71iOS6AFuL8v2jSc4DfLB8ghR6Gk6683P3ZRFRl6O+DEUNEuQOQus ZVLNyI/m+NjIJYpbHlHas1VJMAcNnSUh90JrqcnmR0Ojh6Skyq+1sFrOy TPec0XhDsegeMrYxVUnlaLsXWsyKFILWR3RTCHS4kocgHVMRrEiAt2vWL g==; X-IronPort-AV: E=McAfee;i="6500,9779,10544"; a="401117115" X-IronPort-AV: E=Sophos;i="5.96,200,1665471600"; d="scan'208";a="401117115" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Nov 2022 05:13:33 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10544"; a="749381345" X-IronPort-AV: E=Sophos;i="5.96,200,1665471600"; d="scan'208";a="749381345" Received: from ranerica-svr.sc.intel.com ([172.25.110.23]) by fmsmga002.fm.intel.com with ESMTP; 28 Nov 2022 05:13:32 -0800 From: Ricardo Neri To: "Peter Zijlstra (Intel)" , Juri Lelli , Vincent Guittot Cc: Ricardo Neri , "Ravi V. Shankar" , Ben Segall , Daniel Bristot de Oliveira , Dietmar Eggemann , Len Brown , Mel Gorman , "Rafael J. Wysocki" , Srinivas Pandruvada , Steven Rostedt , Tim Chen , Valentin Schneider , x86@kernel.org, "Joel Fernandes (Google)" , linux-kernel@vger.kernel.org, linux-pm@vger.kernel.org, Ricardo Neri , "Tim C . Chen" Subject: [PATCH v2 07/22] sched/fair: Compute IPC class scores for load balancing Date: Mon, 28 Nov 2022 05:20:45 -0800 Message-Id: <20221128132100.30253-8-ricardo.neri-calderon@linux.intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20221128132100.30253-1-ricardo.neri-calderon@linux.intel.com> References: <20221128132100.30253-1-ricardo.neri-calderon@linux.intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org Compute the joint total (both current and prospective) IPC class score of a scheduling group and the local scheduling group. These IPCC statistics are used during asym_packing load balancing. It implies that the candidate sched group will have one fewer busy CPU after load balancing. This observation is important for physical cores with SMT support. The IPCC score of scheduling groups composed of SMT siblings needs to consider that the siblings share CPU resources. When computing the total IPCC score of the scheduling group, divide score from each sibilng by the number of busy siblings. Cc: Ben Segall Cc: Daniel Bristot de Oliveira Cc: Dietmar Eggemann Cc: Joel Fernandes (Google) Cc: Len Brown Cc: Mel Gorman Cc: Rafael J. Wysocki Cc: Srinivas Pandruvada Cc: Steven Rostedt Cc: Tim C. Chen Cc: Valentin Schneider Cc: x86@kernel.org Cc: linux-pm@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Ricardo Neri --- Changes since v1: * Implemented cleanups and reworks from PeterZ. I took all his suggestions, except the computation of the IPC score before and after load balancing. We are computing not the average score, but the *total*. * Check for the SD_SHARE_CPUCAPACITY to compute the throughput of the SMT siblings of a physical core. * Used the new interface names. * Reworded commit message for clarity. --- kernel/sched/fair.c | 45 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 3a1d6c50a19b..e333f9623b3a 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -8766,6 +8766,10 @@ struct sg_lb_stats { unsigned int nr_numa_running; unsigned int nr_preferred_running; #endif +#ifdef CONFIG_IPC_CLASSES + long ipcc_score_after; /* Prospective IPCC score after load balancing */ + long ipcc_score_before; /* IPCC score before load balancing */ +#endif }; /* @@ -9140,6 +9144,38 @@ static void update_sg_lb_ipcc_stats(struct sg_lb_ipcc_stats *sgcs, } } +static void update_sg_lb_stats_scores(struct sg_lb_ipcc_stats *sgcs, + struct sg_lb_stats *sgs, + struct sched_group *sg, + int dst_cpu) +{ + int busy_cpus, score_on_dst_cpu; + long before, after; + + if (!sched_ipcc_enabled()) + return; + + busy_cpus = sgs->group_weight - sgs->idle_cpus; + /* No busy CPUs in the group. No tasks to move. */ + if (!busy_cpus) + return; + + score_on_dst_cpu = arch_get_ipcc_score(sgcs->min_ipcc, dst_cpu); + + before = sgcs->sum_score; + after = before - sgcs->min_score; + + /* SMT siblings share throughput. */ + if (busy_cpus > 1 && sg->flags & SD_SHARE_CPUCAPACITY) { + before /= busy_cpus; + /* One sibling will become idle after load balance. */ + after /= busy_cpus - 1; + } + + sgs->ipcc_score_after = after + score_on_dst_cpu; + sgs->ipcc_score_before = before; +} + #else /* CONFIG_IPC_CLASSES */ static void update_sg_lb_ipcc_stats(struct sg_lb_ipcc_stats *sgcs, struct rq *rq) @@ -9149,6 +9185,14 @@ static void update_sg_lb_ipcc_stats(struct sg_lb_ipcc_stats *sgcs, static void init_rq_ipcc_stats(struct sg_lb_ipcc_stats *class_sgs) { } + +static void update_sg_lb_stats_scores(struct sg_lb_ipcc_stats *sgcs, + struct sg_lb_stats *sgs, + struct sched_group *sg, + int dst_cpu) +{ +} + #endif /* CONFIG_IPC_CLASSES */ /** @@ -9329,6 +9373,7 @@ static inline void update_sg_lb_stats(struct lb_env *env, if (!local_group && env->sd->flags & SD_ASYM_PACKING && env->idle != CPU_NOT_IDLE && sgs->sum_h_nr_running && sched_asym(env, sds, sgs, group)) { + update_sg_lb_stats_scores(&sgcs, sgs, group, env->dst_cpu); sgs->group_asym_packing = 1; } From patchwork Mon Nov 28 13:20:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ricardo Neri X-Patchwork-Id: 13057485 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0421BC433FE for ; Mon, 28 Nov 2022 13:13:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231652AbiK1NN5 (ORCPT ); Mon, 28 Nov 2022 08:13:57 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40086 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231577AbiK1NNi (ORCPT ); Mon, 28 Nov 2022 08:13:38 -0500 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 812071D678; Mon, 28 Nov 2022 05:13:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1669641217; x=1701177217; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=GsviGa0DcVIC2LcEIZUUyDmrwHqAyGZci2nKyeSjNto=; b=Mpkr6eFStFskEuiJLekkFdqLeBG5DoXW4DtoVyy0Wywdo0aoFwILAJ4I bk/HdXFqz0Llb7RMabI7fawAxTeOmeYJPr2BjYSKwKXTE5jj/nLXdUm4J /pMf7EY18eE5kB4XjjUVWu4cqbFSkGuQgf6/spZDo1HtJkCw24MpxsABa jmPrWoQgas7ObviX9c7Y9RhK4tSLHGVukeyIbFX0otcEj9cIGGRIjebPn nL7RadxxVaqJMuU84hYaU+86Bt6ENYV9BoCCQyChvKBswqWomcHx2e4k9 vzOvq5Q4PrEEvmPjhOqd7RX7Zi3rHd1td8fNlYS2R55oGBsxICA9UE/fK w==; X-IronPort-AV: E=McAfee;i="6500,9779,10544"; a="401117125" X-IronPort-AV: E=Sophos;i="5.96,200,1665471600"; d="scan'208";a="401117125" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Nov 2022 05:13:33 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10544"; a="749381349" X-IronPort-AV: E=Sophos;i="5.96,200,1665471600"; d="scan'208";a="749381349" Received: from ranerica-svr.sc.intel.com ([172.25.110.23]) by fmsmga002.fm.intel.com with ESMTP; 28 Nov 2022 05:13:33 -0800 From: Ricardo Neri To: "Peter Zijlstra (Intel)" , Juri Lelli , Vincent Guittot Cc: Ricardo Neri , "Ravi V. Shankar" , Ben Segall , Daniel Bristot de Oliveira , Dietmar Eggemann , Len Brown , Mel Gorman , "Rafael J. Wysocki" , Srinivas Pandruvada , Steven Rostedt , Tim Chen , Valentin Schneider , x86@kernel.org, "Joel Fernandes (Google)" , linux-kernel@vger.kernel.org, linux-pm@vger.kernel.org, Ricardo Neri , "Tim C . Chen" Subject: [PATCH v2 08/22] sched/fair: Use IPC class to pick the busiest group Date: Mon, 28 Nov 2022 05:20:46 -0800 Message-Id: <20221128132100.30253-9-ricardo.neri-calderon@linux.intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20221128132100.30253-1-ricardo.neri-calderon@linux.intel.com> References: <20221128132100.30253-1-ricardo.neri-calderon@linux.intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org As it iterates, update_sd_pick_busiest() keeps on selecting as busiest sched groups of identical priority. Since both groups have the same priority, either group is a good choice. The IPCC score of the tasks placed a sched group can break this tie. Pick as busiest the sched group that yields a higher IPCC score after load balancing. Cc: Ben Segall Cc: Daniel Bristot de Oliveira Cc: Dietmar Eggemann Cc: Joel Fernandes (Google) Cc: Len Brown Cc: Mel Gorman Cc: Rafael J. Wysocki Cc: Srinivas Pandruvada Cc: Steven Rostedt Cc: Tim C. Chen Cc: Valentin Schneider Cc: x86@kernel.org Cc: linux-pm@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Ricardo Neri --- Changes since v1: * Added a comment to clarify why sched_asym_prefer() needs a tie breaker only in update_sd_pick_busiest(). (PeterZ) * Renamed functions for accuracy: sched_asym_class_prefer() >> sched_asym_ipcc_prefer() sched_asym_class_pick() >> sched_asym_ipcc_pick() * Reworded commit message for clarity. --- kernel/sched/fair.c | 75 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 75 insertions(+) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index e333f9623b3a..e8b181c31842 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -9176,6 +9176,63 @@ static void update_sg_lb_stats_scores(struct sg_lb_ipcc_stats *sgcs, sgs->ipcc_score_before = before; } +/** + * sched_asym_ipcc_prefer - Select a sched group based on its IPCC score + * @a: Load balancing statistics of @sg_a + * @b: Load balancing statistics of @sg_b + * + * Returns: true if preferring @a has a higher IPCC score than @b after + * balancing load. Returns false otherwise. + */ +static bool sched_asym_ipcc_prefer(struct sg_lb_stats *a, + struct sg_lb_stats *b) +{ + if (!sched_ipcc_enabled()) + return false; + + /* @a increases overall throughput after load balance. */ + if (a->ipcc_score_after > b->ipcc_score_after) + return true; + + /* + * If @a and @b yield the same overall throughput, pick @a if + * its current throughput is lower than that of @b. + */ + if (a->ipcc_score_after == b->ipcc_score_after) + return a->ipcc_score_before < b->ipcc_score_before; + + return false; +} + +/** + * sched_asym_ipcc_pick - Select a sched group based on its IPCC score + * @a: A scheduling group + * @b: A second scheduling group + * @a_stats: Load balancing statistics of @a + * @b_stats: Load balancing statistics of @b + * + * Returns: true if @a has the same priority and @a has tasks with IPCC classes + * that yield higher overall throughput after load balance. + * Returns false otherwise. + */ +static bool sched_asym_ipcc_pick(struct sched_group *a, + struct sched_group *b, + struct sg_lb_stats *a_stats, + struct sg_lb_stats *b_stats) +{ + /* + * Only use the class-specific preference selection if both sched + * groups have the same priority. We are not looking at a specific + * CPU. We do not care about the idle state of the groups' + * preferred CPU. + */ + if (arch_asym_cpu_priority(a->asym_prefer_cpu, false) != + arch_asym_cpu_priority(b->asym_prefer_cpu, false)) + return false; + + return sched_asym_ipcc_prefer(a_stats, b_stats); +} + #else /* CONFIG_IPC_CLASSES */ static void update_sg_lb_ipcc_stats(struct sg_lb_ipcc_stats *sgcs, struct rq *rq) @@ -9193,6 +9250,14 @@ static void update_sg_lb_stats_scores(struct sg_lb_ipcc_stats *sgcs, { } +static bool sched_asym_ipcc_pick(struct sched_group *a, + struct sched_group *b, + struct sg_lb_stats *a_stats, + struct sg_lb_stats *b_stats) +{ + return false; +} + #endif /* CONFIG_IPC_CLASSES */ /** @@ -9452,6 +9517,16 @@ static bool update_sd_pick_busiest(struct lb_env *env, sds->busiest->asym_prefer_cpu, false)) return false; + + /* + * Unlike other callers of sched_asym_prefer(), here both @sg + * and @sds::busiest have tasks running. When they have equal + * priority, their IPC class scores can be used to select a + * better busiest. + */ + if (sched_asym_ipcc_pick(sds->busiest, sg, &sds->busiest_stat, sgs)) + return false; + break; case group_misfit_task: From patchwork Mon Nov 28 13:20:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ricardo Neri X-Patchwork-Id: 13057488 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 08B66C43217 for ; Mon, 28 Nov 2022 13:14:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231787AbiK1NOA (ORCPT ); Mon, 28 Nov 2022 08:14:00 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39072 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231665AbiK1NNj (ORCPT ); Mon, 28 Nov 2022 08:13:39 -0500 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 697E21DA63; Mon, 28 Nov 2022 05:13:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1669641218; x=1701177218; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=QuRfJtcjeJpezXilnQW60/bNiOmEuQ40ZOh0UG1TU4A=; b=TIsGMRdxitwIt/fUtN7V2nT9X9lBWZouyurJtfOyx5dZUdoKk5707IcM ntl1vDzJKPn8i6poAh9txSBxi0C2G5iI/h8rjS1y2oP7ctsQRaht66/n1 uqOaFjze8szn7zOvzPmIBeRoYwUMJyMTM6tBQFzhmfq37GvBQnHWedF5z aWOCLe9AJHDitFtq/Ate8xOn2siFldEWRvIAvBMpk2ArqyAJ9YcUcE5fC 3J/9+YbwR0Vmm8raxkSNZRwe033ILMwJ7yuw26li5ap4Cwc026U+exMq8 KZKXB3hHUiWzqhBx8eApRgGKCtjM55iU4taaFJM2vb4sxXokII/iClpyt w==; X-IronPort-AV: E=McAfee;i="6500,9779,10544"; a="401117137" X-IronPort-AV: E=Sophos;i="5.96,200,1665471600"; d="scan'208";a="401117137" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Nov 2022 05:13:33 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10544"; a="749381354" X-IronPort-AV: E=Sophos;i="5.96,200,1665471600"; d="scan'208";a="749381354" Received: from ranerica-svr.sc.intel.com ([172.25.110.23]) by fmsmga002.fm.intel.com with ESMTP; 28 Nov 2022 05:13:33 -0800 From: Ricardo Neri To: "Peter Zijlstra (Intel)" , Juri Lelli , Vincent Guittot Cc: Ricardo Neri , "Ravi V. Shankar" , Ben Segall , Daniel Bristot de Oliveira , Dietmar Eggemann , Len Brown , Mel Gorman , "Rafael J. Wysocki" , Srinivas Pandruvada , Steven Rostedt , Tim Chen , Valentin Schneider , x86@kernel.org, "Joel Fernandes (Google)" , linux-kernel@vger.kernel.org, linux-pm@vger.kernel.org, Ricardo Neri , "Tim C . Chen" Subject: [PATCH v2 09/22] sched/fair: Use IPC class score to select a busiest runqueue Date: Mon, 28 Nov 2022 05:20:47 -0800 Message-Id: <20221128132100.30253-10-ricardo.neri-calderon@linux.intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20221128132100.30253-1-ricardo.neri-calderon@linux.intel.com> References: <20221128132100.30253-1-ricardo.neri-calderon@linux.intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org For two runqueues of equal priority and equal number of running of tasks, select the one whose current task would have the highest IPC class score if placed on the destination CPU. Cc: Ben Segall Cc: Daniel Bristot de Oliveira Cc: Dietmar Eggemann Cc: Joel Fernandes (Google) Cc: Len Brown Cc: Mel Gorman Cc: Rafael J. Wysocki Cc: Srinivas Pandruvada Cc: Steven Rostedt Cc: Tim C. Chen Cc: Valentin Schneider Cc: x86@kernel.org Cc: linux-pm@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Ricardo Neri --- Changes since v1: * Fixed a bug when selecting a busiest runqueue: when comparing two runqueues with equal nr_running, we must compute the IPCC score delta of both. * Renamed local variables to improve the layout of the code block. (PeterZ) * Used the new interface names. --- kernel/sched/fair.c | 54 ++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 53 insertions(+), 1 deletion(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index e8b181c31842..113470bbd7a5 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -9233,6 +9233,24 @@ static bool sched_asym_ipcc_pick(struct sched_group *a, return sched_asym_ipcc_prefer(a_stats, b_stats); } +/** + * ipcc_score_delta - Get the IPCC score delta on a different CPU + * @p: A task + * @alt_cpu: A prospective CPU to place @p + * + * Returns: The IPCC score delta that @p would get if placed on @alt_cpu + */ +static int ipcc_score_delta(struct task_struct *p, int alt_cpu) +{ + unsigned long ipcc = p->ipcc; + + if (!sched_ipcc_enabled()) + return INT_MIN; + + return arch_get_ipcc_score(ipcc, alt_cpu) - + arch_get_ipcc_score(ipcc, task_cpu(p)); +} + #else /* CONFIG_IPC_CLASSES */ static void update_sg_lb_ipcc_stats(struct sg_lb_ipcc_stats *sgcs, struct rq *rq) @@ -9258,6 +9276,11 @@ static bool sched_asym_ipcc_pick(struct sched_group *a, return false; } +static int ipcc_score_delta(struct task_struct *p, int alt_cpu) +{ + return INT_MIN; +} + #endif /* CONFIG_IPC_CLASSES */ /** @@ -10419,8 +10442,8 @@ static struct rq *find_busiest_queue(struct lb_env *env, { struct rq *busiest = NULL, *rq; unsigned long busiest_util = 0, busiest_load = 0, busiest_capacity = 1; + int i, busiest_ipcc_delta = INT_MIN; unsigned int busiest_nr = 0; - int i; for_each_cpu_and(i, sched_group_span(group), env->cpus) { unsigned long capacity, load, util; @@ -10526,8 +10549,37 @@ static struct rq *find_busiest_queue(struct lb_env *env, case migrate_task: if (busiest_nr < nr_running) { + struct task_struct *curr; + busiest_nr = nr_running; busiest = rq; + + /* + * Remember the IPC score delta of busiest::curr. + * We may need it to break a tie with other queues + * with equal nr_running. + */ + curr = rcu_dereference(busiest->curr); + busiest_ipcc_delta = ipcc_score_delta(curr, + env->dst_cpu); + /* + * If rq and busiest have the same number of running + * tasks, pick rq if doing so would give rq::curr a + * bigger IPC boost on dst_cpu. + */ + } else if (sched_ipcc_enabled() && + busiest_nr == nr_running) { + struct task_struct *curr; + int delta; + + curr = rcu_dereference(rq->curr); + delta = ipcc_score_delta(curr, env->dst_cpu); + + if (busiest_ipcc_delta < delta) { + busiest_ipcc_delta = delta; + busiest_nr = nr_running; + busiest = rq; + } } break; From patchwork Mon Nov 28 13:20:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ricardo Neri X-Patchwork-Id: 13057487 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1A366C433FE for ; Mon, 28 Nov 2022 13:14:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231776AbiK1NN7 (ORCPT ); Mon, 28 Nov 2022 08:13:59 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39688 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231653AbiK1NNj (ORCPT ); Mon, 28 Nov 2022 08:13:39 -0500 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7C4391D67C; Mon, 28 Nov 2022 05:13:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1669641218; x=1701177218; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=Ts4gZATy16m9TfKQnjzi/8+7PtA65oHBAtgxhfYL63g=; b=dUhTCKb9+Deq3nD5cKNThUR5G3vwq71Bg7saPFZ5um7Wlua7/F/upQPJ RRhTDsz7xqH2nud8wqpz2x+cjS2FUh9DA8y5OaMabpdhwpPlnpD+JebNT I64o+SdsjNXkSJafO02AOp2dNx4l6jQ/wOeGCh/5CTDSkjxgYaL176zvp RnQblt2WgZVSPuYjmIeacSm+RFs6KtICOWyLzMp3JWyV/hBo2RohGJinx tSRsslZnuK83Ho8zgLGyo7HBvbJ3tt7NJQXXrVcDzP2cwqKLlcEIqvrUM 2Ok9NSyUMCJER11DT73mRPUUPMv8NYriZnKCr3Z2Y6oYG6y5TC/jOOWoY Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10544"; a="401117148" X-IronPort-AV: E=Sophos;i="5.96,200,1665471600"; d="scan'208";a="401117148" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Nov 2022 05:13:33 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10544"; a="749381357" X-IronPort-AV: E=Sophos;i="5.96,200,1665471600"; d="scan'208";a="749381357" Received: from ranerica-svr.sc.intel.com ([172.25.110.23]) by fmsmga002.fm.intel.com with ESMTP; 28 Nov 2022 05:13:33 -0800 From: Ricardo Neri To: "Peter Zijlstra (Intel)" , Juri Lelli , Vincent Guittot Cc: Ricardo Neri , "Ravi V. Shankar" , Ben Segall , Daniel Bristot de Oliveira , Dietmar Eggemann , Len Brown , Mel Gorman , "Rafael J. Wysocki" , Srinivas Pandruvada , Steven Rostedt , Tim Chen , Valentin Schneider , x86@kernel.org, "Joel Fernandes (Google)" , linux-kernel@vger.kernel.org, linux-pm@vger.kernel.org, Ricardo Neri , "Tim C . Chen" Subject: [PATCH v2 10/22] thermal: intel: hfi: Introduce Intel Thread Director classes Date: Mon, 28 Nov 2022 05:20:48 -0800 Message-Id: <20221128132100.30253-11-ricardo.neri-calderon@linux.intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20221128132100.30253-1-ricardo.neri-calderon@linux.intel.com> References: <20221128132100.30253-1-ricardo.neri-calderon@linux.intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org On Intel hybrid parts, each type of CPU has specific performance and energy efficiency capabilities. The Intel Thread Director technology extends the Hardware Feedback Interface (HFI) to provide performance and energy efficiency data for advanced classes of instructions. Add support to parse and parse per-class capabilities. Cc: Ben Segall Cc: Daniel Bristot de Oliveira Cc: Dietmar Eggemann Cc: Joel Fernandes (Google) Cc: Len Brown Cc: Mel Gorman Cc: Rafael J. Wysocki Cc: Srinivas Pandruvada Cc: Steven Rostedt Cc: Tim C. Chen Cc: Valentin Schneider Cc: x86@kernel.org Cc: linux-pm@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Ricardo Neri --- Changes since v1: * Removed a now obsolete comment. --- drivers/thermal/intel/intel_hfi.c | 30 ++++++++++++++++++++++++------ 1 file changed, 24 insertions(+), 6 deletions(-) diff --git a/drivers/thermal/intel/intel_hfi.c b/drivers/thermal/intel/intel_hfi.c index a0640f762dc5..df4dc50e19fb 100644 --- a/drivers/thermal/intel/intel_hfi.c +++ b/drivers/thermal/intel/intel_hfi.c @@ -79,7 +79,7 @@ union cpuid6_edx { * @ee_cap: Energy efficiency capability * * Capabilities of a logical processor in the HFI table. These capabilities are - * unitless. + * unitless and specific to each HFI class. */ struct hfi_cpu_data { u8 perf_cap; @@ -91,7 +91,8 @@ struct hfi_cpu_data { * @perf_updated: Hardware updated performance capabilities * @ee_updated: Hardware updated energy efficiency capabilities * - * Properties of the data in an HFI table. + * Properties of the data in an HFI table. There exists one header per each + * HFI class. */ struct hfi_hdr { u8 perf_updated; @@ -129,16 +130,21 @@ struct hfi_instance { /** * struct hfi_features - Supported HFI features + * @nr_classes: Number of classes supported * @nr_table_pages: Size of the HFI table in 4KB pages * @cpu_stride: Stride size to locate the capability data of a logical * processor within the table (i.e., row stride) + * @class_stride: Stride size to locate a class within the capability + * data of a logical processor or the HFI table header * @hdr_size: Size of the table header * * Parameters and supported features that are common to all HFI instances */ struct hfi_features { + unsigned int nr_classes; unsigned int nr_table_pages; unsigned int cpu_stride; + unsigned int class_stride; unsigned int hdr_size; }; @@ -325,8 +331,8 @@ static void init_hfi_cpu_index(struct hfi_cpu_info *info) } /* - * The format of the HFI table depends on the number of capabilities that the - * hardware supports. Keep a data structure to navigate the table. + * The format of the HFI table depends on the number of capabilities and classes + * that the hardware supports. Keep a data structure to navigate the table. */ static void init_hfi_instance(struct hfi_instance *hfi_instance) { @@ -507,18 +513,30 @@ static __init int hfi_parse_features(void) /* The number of 4KB pages required by the table */ hfi_features.nr_table_pages = edx.split.table_pages + 1; + /* + * Capability fields of an HFI class are grouped together. Classes are + * contiguous in memory. Hence, use the number of supported features to + * locate a specific class. + */ + hfi_features.class_stride = nr_capabilities; + + /* For now, use only one class of the HFI table */ + hfi_features.nr_classes = 1; + /* * The header contains change indications for each supported feature. * The size of the table header is rounded up to be a multiple of 8 * bytes. */ - hfi_features.hdr_size = DIV_ROUND_UP(nr_capabilities, 8) * 8; + hfi_features.hdr_size = DIV_ROUND_UP(nr_capabilities * + hfi_features.nr_classes, 8) * 8; /* * Data of each logical processor is also rounded up to be a multiple * of 8 bytes. */ - hfi_features.cpu_stride = DIV_ROUND_UP(nr_capabilities, 8) * 8; + hfi_features.cpu_stride = DIV_ROUND_UP(nr_capabilities * + hfi_features.nr_classes, 8) * 8; return 0; } From patchwork Mon Nov 28 13:20:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ricardo Neri X-Patchwork-Id: 13057489 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39316C4167D for ; Mon, 28 Nov 2022 13:14:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231791AbiK1NOC (ORCPT ); Mon, 28 Nov 2022 08:14:02 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39876 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229896AbiK1NNj (ORCPT ); Mon, 28 Nov 2022 08:13:39 -0500 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8F1A51DA53; Mon, 28 Nov 2022 05:13:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1669641218; x=1701177218; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=xRdgNgGdXQiORGcgdnIBmgL4e1VMD8FuTc66jRXlDMQ=; b=eP3if2/17XG58Aa8gCs/yoHPvTQLjlMwsRb/GHZZrNwMSJXhaeb6x5+r uUDoskJFU7j9fkN1QVb7UR4UUy7SsC7JH3IIRwqG7T6gFXG90nEA43f64 mpc+3oJeU8kKtKxxhS+zq1JAfrT7X9cL76ewDU7rMMKQoctRL7iXmgbIm DM/HTwt5k0aVt5fwSJOcEuqaia37mB6umldvwazWGZQM1lpP3DKqg4OjR 9NAsMwIROJJb0MkzT+O2TCop5NfHvBjRkl3QMfAAM7a3T0hgiEbSjbF2M SPBJukrmxYQBZKfWUb3tpNHLQf/+6OMfoPp5EFbuR0SApUEwFuFgs3p9s Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10544"; a="401117159" X-IronPort-AV: E=Sophos;i="5.96,200,1665471600"; d="scan'208";a="401117159" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Nov 2022 05:13:34 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10544"; a="749381363" X-IronPort-AV: E=Sophos;i="5.96,200,1665471600"; d="scan'208";a="749381363" Received: from ranerica-svr.sc.intel.com ([172.25.110.23]) by fmsmga002.fm.intel.com with ESMTP; 28 Nov 2022 05:13:33 -0800 From: Ricardo Neri To: "Peter Zijlstra (Intel)" , Juri Lelli , Vincent Guittot Cc: Ricardo Neri , "Ravi V. Shankar" , Ben Segall , Daniel Bristot de Oliveira , Dietmar Eggemann , Len Brown , Mel Gorman , "Rafael J. Wysocki" , Srinivas Pandruvada , Steven Rostedt , Tim Chen , Valentin Schneider , x86@kernel.org, "Joel Fernandes (Google)" , linux-kernel@vger.kernel.org, linux-pm@vger.kernel.org, Ricardo Neri , "Tim C . Chen" Subject: [PATCH v2 11/22] thermal: intel: hfi: Store per-CPU IPCC scores Date: Mon, 28 Nov 2022 05:20:49 -0800 Message-Id: <20221128132100.30253-12-ricardo.neri-calderon@linux.intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20221128132100.30253-1-ricardo.neri-calderon@linux.intel.com> References: <20221128132100.30253-1-ricardo.neri-calderon@linux.intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org The scheduler reads the IPCC scores when balancing load. These reads can be quite frequent. Hardware can also update the HFI table frequently. Concurrent access may cause a lot of contention. It gets worse as the number of CPUs increases. Instead, create separate per-CPU IPCC scores that the scheduler can read without the HFI table lock. Cc: Ben Segall Cc: Daniel Bristot de Oliveira Cc: Dietmar Eggemann Cc: Joel Fernandes (Google) Cc: Len Brown Cc: Mel Gorman Cc: Rafael J. Wysocki Cc: Srinivas Pandruvada Cc: Steven Rostedt Cc: Tim C. Chen Cc: Valentin Schneider Cc: x86@kernel.org Cc: linux-pm@vger.kernel.org Cc: linux-kernel@vger.kernel.org Suggested-by: Peter Zijlstra (Intel) Signed-off-by: Ricardo Neri --- Changes since v1: * Added this patch. --- drivers/thermal/intel/intel_hfi.c | 38 +++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) diff --git a/drivers/thermal/intel/intel_hfi.c b/drivers/thermal/intel/intel_hfi.c index df4dc50e19fb..56dba967849c 100644 --- a/drivers/thermal/intel/intel_hfi.c +++ b/drivers/thermal/intel/intel_hfi.c @@ -29,6 +29,7 @@ #include #include #include +#include #include #include #include @@ -172,6 +173,35 @@ static struct workqueue_struct *hfi_updates_wq; #define HFI_UPDATE_INTERVAL HZ #define HFI_MAX_THERM_NOTIFY_COUNT 16 +#ifdef CONFIG_IPC_CLASSES +static int __percpu *hfi_ipcc_scores; + +static int alloc_hfi_ipcc_scores(void) +{ + hfi_ipcc_scores = __alloc_percpu(sizeof(*hfi_ipcc_scores) * + hfi_features.nr_classes, + sizeof(*hfi_ipcc_scores)); + + return !hfi_ipcc_scores; +} + +static void set_hfi_ipcc_score(void *caps, int cpu) +{ + int i, *hfi_class = per_cpu_ptr(hfi_ipcc_scores, cpu); + + for (i = 0; i < hfi_features.nr_classes; i++) { + struct hfi_cpu_data *class_caps; + + class_caps = caps + i * hfi_features.class_stride; + WRITE_ONCE(hfi_class[i], class_caps->perf_cap); + } +} + +#else +static int alloc_hfi_ipcc_scores(void) { return 0; } +static void set_hfi_ipcc_score(void *caps, int cpu) { } +#endif /* CONFIG_IPC_CLASSES */ + static void get_hfi_caps(struct hfi_instance *hfi_instance, struct thermal_genl_cpu_caps *cpu_caps) { @@ -194,6 +224,8 @@ static void get_hfi_caps(struct hfi_instance *hfi_instance, cpu_caps[i].efficiency = caps->ee_cap << 2; ++i; + + set_hfi_ipcc_score(caps, cpu); } raw_spin_unlock_irq(&hfi_instance->table_lock); } @@ -572,8 +604,14 @@ void __init intel_hfi_init(void) if (!hfi_updates_wq) goto err_nomem; + if (alloc_hfi_ipcc_scores()) + goto err_ipcc; + return; +err_ipcc: + destroy_workqueue(hfi_updates_wq); + err_nomem: for (j = 0; j < i; ++j) { hfi_instance = &hfi_instances[j]; From patchwork Mon Nov 28 13:20:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ricardo Neri X-Patchwork-Id: 13057490 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 83A0BC433FE for ; Mon, 28 Nov 2022 13:14:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231806AbiK1NOI (ORCPT ); Mon, 28 Nov 2022 08:14:08 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39874 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231534AbiK1NNk (ORCPT ); Mon, 28 Nov 2022 08:13:40 -0500 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5CD919FF5; Mon, 28 Nov 2022 05:13:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1669641219; x=1701177219; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=NJihmyOVPjkl7Qdl9zakJCCiujspncSo1d7+r1Yn1vU=; b=j1tgSnHGyUwiofNoNQJraegM0j4LuNnx+zCCM2jYqRiLkxoP/Vwx4J5z BGBQ5d3Ipfu2FcCVOFjP8G7HtbRyU4wyjRXbpeFPjGB7bT13LysulJch7 Vd0HPc+2czVZKdUmwEWEaLNokZGSRvMV4T7+LqbZWx42lGizr3FM0Uk82 jiLpXJIr5oeGVne9gEVsEKpMBlc7ubqoz8j9N5CMrz8Wq18POzBnlNZQQ OGwDN+q59jzQpEU9jKMJ1lhXb0h0eu+HpIIuDN7u5LdrD+slRVIV08Hgm +Bfc4HhdF++bwTGnu7rYBEblsaCIhx8oe5ywV71Hul+upydruHaZxrhMu Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10544"; a="401117169" X-IronPort-AV: E=Sophos;i="5.96,200,1665471600"; d="scan'208";a="401117169" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Nov 2022 05:13:34 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10544"; a="749381366" X-IronPort-AV: E=Sophos;i="5.96,200,1665471600"; d="scan'208";a="749381366" Received: from ranerica-svr.sc.intel.com ([172.25.110.23]) by fmsmga002.fm.intel.com with ESMTP; 28 Nov 2022 05:13:34 -0800 From: Ricardo Neri To: "Peter Zijlstra (Intel)" , Juri Lelli , Vincent Guittot Cc: Ricardo Neri , "Ravi V. Shankar" , Ben Segall , Daniel Bristot de Oliveira , Dietmar Eggemann , Len Brown , Mel Gorman , "Rafael J. Wysocki" , Srinivas Pandruvada , Steven Rostedt , Tim Chen , Valentin Schneider , x86@kernel.org, "Joel Fernandes (Google)" , linux-kernel@vger.kernel.org, linux-pm@vger.kernel.org, Ricardo Neri , "Tim C . Chen" Subject: [PATCH v2 12/22] x86/cpufeatures: Add the Intel Thread Director feature definitions Date: Mon, 28 Nov 2022 05:20:50 -0800 Message-Id: <20221128132100.30253-13-ricardo.neri-calderon@linux.intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20221128132100.30253-1-ricardo.neri-calderon@linux.intel.com> References: <20221128132100.30253-1-ricardo.neri-calderon@linux.intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org Intel Thread Director (ITD) provides hardware resources to classify the current task. The classification reflects the type of instructions that a task currently executes. ITD extends the Hardware Feedback Interface table to provide performance and energy efficiency capabilities for each of the supported classes of tasks. Cc: Ben Segall Cc: Daniel Bristot de Oliveira Cc: Dietmar Eggemann Cc: Joel Fernandes (Google) Cc: Len Brown Cc: Mel Gorman Cc: Rafael J. Wysocki Cc: Srinivas Pandruvada Cc: Steven Rostedt Cc: Tim C. Chen Cc: Valentin Schneider Cc: x86@kernel.org Cc: linux-pm@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Ricardo Neri --- Changes since v1: * Removed dependency on CONFIG_INTEL_THREAD_DIRECTOR. Instead, depend on CONFIG_IPC_CLASSES. * Added DISABLE_ITD to the correct DISABLE_MASK: 14 instead of 13. --- arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/disabled-features.h | 8 +++++++- arch/x86/kernel/cpu/cpuid-deps.c | 1 + 3 files changed, 9 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h index b6525491a41b..80b2beafc81e 100644 --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -344,6 +344,7 @@ #define X86_FEATURE_HWP_EPP (14*32+10) /* HWP Energy Perf. Preference */ #define X86_FEATURE_HWP_PKG_REQ (14*32+11) /* HWP Package Level Request */ #define X86_FEATURE_HFI (14*32+19) /* Hardware Feedback Interface */ +#define X86_FEATURE_ITD (14*32+23) /* Intel Thread Director */ /* AMD SVM Feature Identification, CPUID level 0x8000000a (EDX), word 15 */ #define X86_FEATURE_NPT (15*32+ 0) /* Nested Page Table support */ diff --git a/arch/x86/include/asm/disabled-features.h b/arch/x86/include/asm/disabled-features.h index c44b56f7ffba..0edd9bef7f2e 100644 --- a/arch/x86/include/asm/disabled-features.h +++ b/arch/x86/include/asm/disabled-features.h @@ -99,6 +99,12 @@ # define DISABLE_TDX_GUEST (1 << (X86_FEATURE_TDX_GUEST & 31)) #endif +#ifdef CONFIG_IPC_CLASSES +# define DISABLE_ITD 0 +#else +# define DISABLE_ITD (1 << (X86_FEATURE_ITD & 31)) +#endif + /* * Make sure to add features to the correct mask */ @@ -117,7 +123,7 @@ DISABLE_CALL_DEPTH_TRACKING) #define DISABLED_MASK12 0 #define DISABLED_MASK13 0 -#define DISABLED_MASK14 0 +#define DISABLED_MASK14 (DISABLE_ITD) #define DISABLED_MASK15 0 #define DISABLED_MASK16 (DISABLE_PKU|DISABLE_OSPKE|DISABLE_LA57|DISABLE_UMIP| \ DISABLE_ENQCMD) diff --git a/arch/x86/kernel/cpu/cpuid-deps.c b/arch/x86/kernel/cpu/cpuid-deps.c index d95221117129..277f157e067e 100644 --- a/arch/x86/kernel/cpu/cpuid-deps.c +++ b/arch/x86/kernel/cpu/cpuid-deps.c @@ -79,6 +79,7 @@ static const struct cpuid_dep cpuid_deps[] = { { X86_FEATURE_XFD, X86_FEATURE_XSAVES }, { X86_FEATURE_XFD, X86_FEATURE_XGETBV1 }, { X86_FEATURE_AMX_TILE, X86_FEATURE_XFD }, + { X86_FEATURE_ITD, X86_FEATURE_HFI }, {} }; From patchwork Mon Nov 28 13:20:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ricardo Neri X-Patchwork-Id: 13057491 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 46D4AC46467 for ; Mon, 28 Nov 2022 13:14:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231600AbiK1NOK (ORCPT ); Mon, 28 Nov 2022 08:14:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39690 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231531AbiK1NNk (ORCPT ); Mon, 28 Nov 2022 08:13:40 -0500 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8FDEFA46F; Mon, 28 Nov 2022 05:13:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1669641219; x=1701177219; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=ElamVYUpgDqJdxkmls7IkTtE4f7JXYRsemQvg1xBFFc=; b=Ku9+ujnj0jt1DV4ZEqWbFcflA/Y5vkn34TyYe0Bz1h7aC2yrSAKha9K5 N0lM62DkG2hcWWgGMPqRYervSHCKX1FgnHYN3bZzBOWw/nCdF4cZAAabx yZ2Dn00nr8wrnuvhUKP5tlO6rpzbEBdpJCn/lt6G0gFV1IrD6WR2uxqNw jhRlK+Lk6xMXvkEsVtrRQkUG8Kn269QlSWABtxPaP6abUCeGp3gdoEbmN GsQL5cZzxMf4HiBs8/QNTD1JqmCWHnCDH1xGSPH1/BxS8Usx1NMF1s3vj Vy5wNR1uow/QWbttI5ioYBPHhIvvXGgfpnz/YNrnp6OPd0tXLEH2GRQYB Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10544"; a="401117179" X-IronPort-AV: E=Sophos;i="5.96,200,1665471600"; d="scan'208";a="401117179" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Nov 2022 05:13:34 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10544"; a="749381369" X-IronPort-AV: E=Sophos;i="5.96,200,1665471600"; d="scan'208";a="749381369" Received: from ranerica-svr.sc.intel.com ([172.25.110.23]) by fmsmga002.fm.intel.com with ESMTP; 28 Nov 2022 05:13:34 -0800 From: Ricardo Neri To: "Peter Zijlstra (Intel)" , Juri Lelli , Vincent Guittot Cc: Ricardo Neri , "Ravi V. Shankar" , Ben Segall , Daniel Bristot de Oliveira , Dietmar Eggemann , Len Brown , Mel Gorman , "Rafael J. Wysocki" , Srinivas Pandruvada , Steven Rostedt , Tim Chen , Valentin Schneider , x86@kernel.org, "Joel Fernandes (Google)" , linux-kernel@vger.kernel.org, linux-pm@vger.kernel.org, Ricardo Neri , "Tim C . Chen" Subject: [PATCH v2 13/22] thermal: intel: hfi: Update the IPC class of the current task Date: Mon, 28 Nov 2022 05:20:51 -0800 Message-Id: <20221128132100.30253-14-ricardo.neri-calderon@linux.intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20221128132100.30253-1-ricardo.neri-calderon@linux.intel.com> References: <20221128132100.30253-1-ricardo.neri-calderon@linux.intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org Use Intel Thread Director classification to update the IPC class of a task. Implement the needed scheduler interfaces. Cc: Ben Segall Cc: Daniel Bristot de Oliveira Cc: Dietmar Eggemann Cc: Joel Fernandes (Google) Cc: Len Brown Cc: Mel Gorman Cc: Rafael J. Wysocki Cc: Srinivas Pandruvada Cc: Steven Rostedt Cc: Tim C. Chen Cc: Valentin Schneider Cc: x86@kernel.org Cc: linux-pm@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Ricardo Neri --- Changes since v1: * Adjusted the result the classification of Intel Thread Director to start at class 1. Class 0 for the scheduler means that the task is unclassified. * Redefined union hfi_thread_feedback_char_msr to ensure all bit-fields are packed. (PeterZ) * Removed CONFIG_INTEL_THREAD_DIRECTOR. (PeterZ) * Shortened the names of the functions that implement IPC classes. * Removed argument smt_siblings_idle from intel_hfi_update_ipcc(). (PeterZ) --- arch/x86/include/asm/topology.h | 8 +++++++ drivers/thermal/intel/intel_hfi.c | 37 +++++++++++++++++++++++++++++++ 2 files changed, 45 insertions(+) diff --git a/arch/x86/include/asm/topology.h b/arch/x86/include/asm/topology.h index 458c891a8273..cf46a3aea283 100644 --- a/arch/x86/include/asm/topology.h +++ b/arch/x86/include/asm/topology.h @@ -227,4 +227,12 @@ void init_freq_invariance_cppc(void); #define arch_init_invariance_cppc init_freq_invariance_cppc #endif +#if defined(CONFIG_IPC_CLASSES) && defined(CONFIG_INTEL_HFI_THERMAL) +int intel_hfi_has_ipc_classes(void); +void intel_hfi_update_ipcc(struct task_struct *curr); + +#define arch_has_ipc_classes intel_hfi_has_ipc_classes +#define arch_update_ipcc intel_hfi_update_ipcc +#endif /* defined(CONFIG_IPC_CLASSES) && defined(CONFIG_INTEL_HFI_THERMAL) */ + #endif /* _ASM_X86_TOPOLOGY_H */ diff --git a/drivers/thermal/intel/intel_hfi.c b/drivers/thermal/intel/intel_hfi.c index 56dba967849c..f85394b532a7 100644 --- a/drivers/thermal/intel/intel_hfi.c +++ b/drivers/thermal/intel/intel_hfi.c @@ -74,6 +74,17 @@ union cpuid6_edx { u32 full; }; +#ifdef CONFIG_IPC_CLASSES +union hfi_thread_feedback_char_msr { + struct { + u64 classid : 8; + u64 __reserved : 55; + u64 valid : 1; + } split; + u64 full; +}; +#endif + /** * struct hfi_cpu_data - HFI capabilities per CPU * @perf_cap: Performance capability @@ -176,6 +187,32 @@ static struct workqueue_struct *hfi_updates_wq; #ifdef CONFIG_IPC_CLASSES static int __percpu *hfi_ipcc_scores; +int intel_hfi_has_ipc_classes(void) +{ + return cpu_feature_enabled(X86_FEATURE_ITD); +} + +void intel_hfi_update_ipcc(struct task_struct *curr) +{ + union hfi_thread_feedback_char_msr msr; + + /* We should not be here if ITD is not supported. */ + if (!cpu_feature_enabled(X86_FEATURE_ITD)) { + pr_warn_once("task classification requested but not supported!"); + return; + } + + rdmsrl(MSR_IA32_HW_FEEDBACK_CHAR, msr.full); + if (!msr.split.valid) + return; + + /* + * 0 is a valid classification for Intel Thread Director. A scheduler + * IPCC class of 0 means that the task is unclassified. Adjust. + */ + curr->ipcc = msr.split.classid + 1; +} + static int alloc_hfi_ipcc_scores(void) { hfi_ipcc_scores = __alloc_percpu(sizeof(*hfi_ipcc_scores) * From patchwork Mon Nov 28 13:20:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ricardo Neri X-Patchwork-Id: 13057492 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EC538C433FE for ; Mon, 28 Nov 2022 13:14:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231812AbiK1NOL (ORCPT ); Mon, 28 Nov 2022 08:14:11 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39076 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231571AbiK1NNk (ORCPT ); Mon, 28 Nov 2022 08:13:40 -0500 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2A0681706C; Mon, 28 Nov 2022 05:13:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1669641220; x=1701177220; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=ChhBXDi6i4cO+o51+tVzAGA9ljkMrmhRzO0pv8QvdKc=; b=ePLdhY1iJLhxEHUW+R/17wYTQOjttdYPbQAMe9rF7Juv61tJWUUhN1Pf CeVD10Hmwt2tSgAHdCWhkFMmDrJ2pzxLnAF+TbBR0H3b30puxwVHjyaEH zpl7/dV9D0DPpVYv9tWhPya+aOuL7Pu5e6O6hlyxrM3a1WyRXjwoEDyvU m2/bZyISvkNrZj8bsBbdgr7p1w+eaxIN89k7nDapS6Xpum/9VligD797g 1nH45GwJAfrA2O7e3gGEyVvwsQDW9ZKk5tttnbm18bqBnyHaLRIawpDVb lnzkYVWmAfB/kfVQqGAWc2/ImZJzfu6VVXis0dfI8WxHgCLKBZFY2X1Zj w==; X-IronPort-AV: E=McAfee;i="6500,9779,10544"; a="401117189" X-IronPort-AV: E=Sophos;i="5.96,200,1665471600"; d="scan'208";a="401117189" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Nov 2022 05:13:34 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10544"; a="749381374" X-IronPort-AV: E=Sophos;i="5.96,200,1665471600"; d="scan'208";a="749381374" Received: from ranerica-svr.sc.intel.com ([172.25.110.23]) by fmsmga002.fm.intel.com with ESMTP; 28 Nov 2022 05:13:34 -0800 From: Ricardo Neri To: "Peter Zijlstra (Intel)" , Juri Lelli , Vincent Guittot Cc: Ricardo Neri , "Ravi V. Shankar" , Ben Segall , Daniel Bristot de Oliveira , Dietmar Eggemann , Len Brown , Mel Gorman , "Rafael J. Wysocki" , Srinivas Pandruvada , Steven Rostedt , Tim Chen , Valentin Schneider , x86@kernel.org, "Joel Fernandes (Google)" , linux-kernel@vger.kernel.org, linux-pm@vger.kernel.org, Ricardo Neri , "Tim C . Chen" Subject: [PATCH v2 14/22] thermal: intel: hfi: Report the IPC class score of a CPU Date: Mon, 28 Nov 2022 05:20:52 -0800 Message-Id: <20221128132100.30253-15-ricardo.neri-calderon@linux.intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20221128132100.30253-1-ricardo.neri-calderon@linux.intel.com> References: <20221128132100.30253-1-ricardo.neri-calderon@linux.intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org Implement the arch_get_ipcc_score() interface of the scheduler. Use the performance capabilities of the extended Hardware Feedback Interface table as the IPC score of a class of tasks when placed on a given CPU. Cc: Ben Segall Cc: Daniel Bristot de Oliveira Cc: Dietmar Eggemann Cc: Joel Fernandes (Google) Cc: Len Brown Cc: Mel Gorman Cc: Rafael J. Wysocki Cc: Srinivas Pandruvada Cc: Steven Rostedt Cc: Tim C. Chen Cc: Valentin Schneider Cc: x86@kernel.org Cc: linux-pm@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Ricardo Neri --- Changes since v1: * Adjusted the returned HFI class (which starts at 0) to match the scheduler IPCC class (which starts at 1). (PeterZ) * Used the new interface names. --- arch/x86/include/asm/topology.h | 2 ++ drivers/thermal/intel/intel_hfi.c | 27 +++++++++++++++++++++++++++ 2 files changed, 29 insertions(+) diff --git a/arch/x86/include/asm/topology.h b/arch/x86/include/asm/topology.h index cf46a3aea283..0fae13058f01 100644 --- a/arch/x86/include/asm/topology.h +++ b/arch/x86/include/asm/topology.h @@ -230,9 +230,11 @@ void init_freq_invariance_cppc(void); #if defined(CONFIG_IPC_CLASSES) && defined(CONFIG_INTEL_HFI_THERMAL) int intel_hfi_has_ipc_classes(void); void intel_hfi_update_ipcc(struct task_struct *curr); +int intel_hfi_get_ipcc_score(unsigned short ipcc, int cpu); #define arch_has_ipc_classes intel_hfi_has_ipc_classes #define arch_update_ipcc intel_hfi_update_ipcc +#define arch_get_ipcc_score intel_hfi_get_ipcc_score #endif /* defined(CONFIG_IPC_CLASSES) && defined(CONFIG_INTEL_HFI_THERMAL) */ #endif /* _ASM_X86_TOPOLOGY_H */ diff --git a/drivers/thermal/intel/intel_hfi.c b/drivers/thermal/intel/intel_hfi.c index f85394b532a7..1f7b18198bd4 100644 --- a/drivers/thermal/intel/intel_hfi.c +++ b/drivers/thermal/intel/intel_hfi.c @@ -213,6 +213,33 @@ void intel_hfi_update_ipcc(struct task_struct *curr) curr->ipcc = msr.split.classid + 1; } +int intel_hfi_get_ipcc_score(unsigned short ipcc, int cpu) +{ + unsigned short hfi_class; + int *scores; + + if (cpu < 0 || cpu >= nr_cpu_ids) + return -EINVAL; + + if (ipcc == IPC_CLASS_UNCLASSIFIED) + return -EINVAL; + + /* + * Scheduler IPC classes start at 1. HFI classes start at 0. + * See note intel_hfi_update_ipcc(). + */ + hfi_class = ipcc - 1; + + if (hfi_class >= hfi_features.nr_classes) + return -EINVAL; + + scores = per_cpu_ptr(hfi_ipcc_scores, cpu); + if (!scores) + return -ENODEV; + + return READ_ONCE(scores[hfi_class]); +} + static int alloc_hfi_ipcc_scores(void) { hfi_ipcc_scores = __alloc_percpu(sizeof(*hfi_ipcc_scores) * From patchwork Mon Nov 28 13:20:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ricardo Neri X-Patchwork-Id: 13057494 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D4BA5C433FE for ; Mon, 28 Nov 2022 13:14:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231828AbiK1NOT (ORCPT ); Mon, 28 Nov 2022 08:14:19 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38800 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231591AbiK1NNk (ORCPT ); Mon, 28 Nov 2022 08:13:40 -0500 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 269C4B7CD; Mon, 28 Nov 2022 05:13:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1669641220; x=1701177220; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=qxJuo8ldTPva48VrC4qfDcG1QduoBAzNsZcRAMiSMzk=; b=SQSfzVDV4acHSgxO+V4fEB7SGuM6IXwhmyu3xFdfQ1kjwSwIb4r/quzz BJG7BVzw/Xmgo9oAsDUdGYlPzAMqQO9l4OvYeg4JKsHKANxXdnHkhSm09 E2dPiyApFjz4Hn1G2al7R+nFmQ/g5+nSmiXUkM2W6ZCucSJEv5+0/ONVe 9NEhRhedJSzYn3AFfHRUMVtMly1x5cn8g64spKA5FYbiE5IOWIGNQqtgn FO9TNgK/AcbrixzE6ZySqpBq+pveGInvudQWM/A8TTApwPBlCECpVichx pYzZL8vOkvjrPe2x574h1d1xJZSuf233YV3TKjtsc7gfAU3XDOPr19J7/ w==; X-IronPort-AV: E=McAfee;i="6500,9779,10544"; a="401117200" X-IronPort-AV: E=Sophos;i="5.96,200,1665471600"; d="scan'208";a="401117200" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Nov 2022 05:13:35 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10544"; a="749381381" X-IronPort-AV: E=Sophos;i="5.96,200,1665471600"; d="scan'208";a="749381381" Received: from ranerica-svr.sc.intel.com ([172.25.110.23]) by fmsmga002.fm.intel.com with ESMTP; 28 Nov 2022 05:13:34 -0800 From: Ricardo Neri To: "Peter Zijlstra (Intel)" , Juri Lelli , Vincent Guittot Cc: Ricardo Neri , "Ravi V. Shankar" , Ben Segall , Daniel Bristot de Oliveira , Dietmar Eggemann , Len Brown , Mel Gorman , "Rafael J. Wysocki" , Srinivas Pandruvada , Steven Rostedt , Tim Chen , Valentin Schneider , x86@kernel.org, "Joel Fernandes (Google)" , linux-kernel@vger.kernel.org, linux-pm@vger.kernel.org, Ricardo Neri , "Tim C . Chen" Subject: [PATCH v2 15/22] thermal: intel: hfi: Define a default class for unclassified tasks Date: Mon, 28 Nov 2022 05:20:53 -0800 Message-Id: <20221128132100.30253-16-ricardo.neri-calderon@linux.intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20221128132100.30253-1-ricardo.neri-calderon@linux.intel.com> References: <20221128132100.30253-1-ricardo.neri-calderon@linux.intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org A task may be unclassified if it has been recently created, spend most of its lifetime sleeping, or hardware has not provided a classification. Most tasks will be eventually classified as scheduler's IPC class 1 (HFI class 0). This class corresponds to the capabilities in the legacy, classless, HFI table. IPC class 1 is a reasonable choice until hardware provides an actual classification. Meanwhile, the scheduler will place other tasks with higher scores on higher-performance CPUs. Cc: Ben Segall Cc: Daniel Bristot de Oliveira Cc: Dietmar Eggemann Cc: Len Brown Cc: Mel Gorman Cc: Rafael J. Wysocki Cc: Srinivas Pandruvada Cc: Steven Rostedt Cc: Tim C. Chen Cc: Valentin Schneider Cc: x86@kernel.org Cc: linux-pm@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Ricardo Neri --- Changes since v1: * Now the default class is 1. --- drivers/thermal/intel/intel_hfi.c | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/drivers/thermal/intel/intel_hfi.c b/drivers/thermal/intel/intel_hfi.c index 1f7b18198bd4..1b3fd704ae9a 100644 --- a/drivers/thermal/intel/intel_hfi.c +++ b/drivers/thermal/intel/intel_hfi.c @@ -187,6 +187,19 @@ static struct workqueue_struct *hfi_updates_wq; #ifdef CONFIG_IPC_CLASSES static int __percpu *hfi_ipcc_scores; +/* + * A task may be unclassified if it has been recently created, spend most of + * its lifetime sleeping, or hardware has not provided a classification. + * + * Most tasks will be classified as scheduler's IPC class 1 (HFI class 0) + * eventually. Meanwhile, the scheduler will place tasks of higher IPC score + * on higher-performance CPUs. + * + * IPC class 1 is a reasonable choice. It matches the performance capability + * of the legacy, classless, HFI table. + */ +#define HFI_UNCLASSIFIED_DEFAULT 1 + int intel_hfi_has_ipc_classes(void) { return cpu_feature_enabled(X86_FEATURE_ITD); @@ -222,7 +235,7 @@ int intel_hfi_get_ipcc_score(unsigned short ipcc, int cpu) return -EINVAL; if (ipcc == IPC_CLASS_UNCLASSIFIED) - return -EINVAL; + ipcc = HFI_UNCLASSIFIED_DEFAULT; /* * Scheduler IPC classes start at 1. HFI classes start at 0. From patchwork Mon Nov 28 13:20:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ricardo Neri X-Patchwork-Id: 13057493 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A6BAEC4167D for ; Mon, 28 Nov 2022 13:14:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231815AbiK1NON (ORCPT ); Mon, 28 Nov 2022 08:14:13 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40118 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231567AbiK1NNk (ORCPT ); Mon, 28 Nov 2022 08:13:40 -0500 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A62372DCD; Mon, 28 Nov 2022 05:13:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1669641219; x=1701177219; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=6edsBDhndxufd+3EwJXKfsHyRlYl4SM2zX6wq8H/56U=; b=OzSNHXalsHBm3jZdROfgz1/kUfd36h+u4RkznczD5jae31akEp2HLtoI GU+8kO2KdFMWV/HLxBZHebThJKcj54O8vgTIp6AIv3RjhGDVpcWya+MQ8 YujLMP2fiO6j9iWyTx3yhzoG49/IJ85wd8/UgYwGCOmpMzHVID0HzgDCO 6o7mGpKPv3+u7ia2XzUBjywjqdp4RZbUJG6Ur0LyrhvbAH9eoPw+g5uus gZcbH+OewhalZbqqqkwGIsd/OlZusxkcJDFvV8r0Qt/Yv1NQ7GIvDQGJQ N4kZyvB1OpvwGWwM0hwA9zun0qqIl7cz6tgPTlL07cHkHyEFYOH6a86WR g==; X-IronPort-AV: E=McAfee;i="6500,9779,10544"; a="401117214" X-IronPort-AV: E=Sophos;i="5.96,200,1665471600"; d="scan'208";a="401117214" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Nov 2022 05:13:35 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10544"; a="749381384" X-IronPort-AV: E=Sophos;i="5.96,200,1665471600"; d="scan'208";a="749381384" Received: from ranerica-svr.sc.intel.com ([172.25.110.23]) by fmsmga002.fm.intel.com with ESMTP; 28 Nov 2022 05:13:35 -0800 From: Ricardo Neri To: "Peter Zijlstra (Intel)" , Juri Lelli , Vincent Guittot Cc: Ricardo Neri , "Ravi V. Shankar" , Ben Segall , Daniel Bristot de Oliveira , Dietmar Eggemann , Len Brown , Mel Gorman , "Rafael J. Wysocki" , Srinivas Pandruvada , Steven Rostedt , Tim Chen , Valentin Schneider , x86@kernel.org, "Joel Fernandes (Google)" , linux-kernel@vger.kernel.org, linux-pm@vger.kernel.org, Ricardo Neri , "Tim C . Chen" Subject: [PATCH v2 16/22] thermal: intel: hfi: Enable the Intel Thread Director Date: Mon, 28 Nov 2022 05:20:54 -0800 Message-Id: <20221128132100.30253-17-ricardo.neri-calderon@linux.intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20221128132100.30253-1-ricardo.neri-calderon@linux.intel.com> References: <20221128132100.30253-1-ricardo.neri-calderon@linux.intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org Enable Intel Thread Director from the CPU hotplug callback: globally from CPU0 and then enable the thread-classification hardware in each logical processor individually. Also, initialize the number of classes supported. Cc: Ben Segall Cc: Daniel Bristot de Oliveira Cc: Dietmar Eggemann Cc: Joel Fernandes (Google) Cc: Len Brown Cc: Mel Gorman Cc: Rafael J. Wysocki Cc: Srinivas Pandruvada Cc: Steven Rostedt Cc: Tim C. Chen Cc: Valentin Schneider Cc: x86@kernel.org Cc: linux-pm@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Ricardo Neri --- Changes since v1: * None --- arch/x86/include/asm/msr-index.h | 2 ++ drivers/thermal/intel/intel_hfi.c | 30 ++++++++++++++++++++++++++++-- 2 files changed, 30 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/msr-index.h b/arch/x86/include/asm/msr-index.h index 37ff47552bcb..96303330223b 100644 --- a/arch/x86/include/asm/msr-index.h +++ b/arch/x86/include/asm/msr-index.h @@ -1075,6 +1075,8 @@ /* Hardware Feedback Interface */ #define MSR_IA32_HW_FEEDBACK_PTR 0x17d0 #define MSR_IA32_HW_FEEDBACK_CONFIG 0x17d1 +#define MSR_IA32_HW_FEEDBACK_THREAD_CONFIG 0x17d4 +#define MSR_IA32_HW_FEEDBACK_CHAR 0x17d2 /* x2APIC locked status */ #define MSR_IA32_XAPIC_DISABLE_STATUS 0xBD diff --git a/drivers/thermal/intel/intel_hfi.c b/drivers/thermal/intel/intel_hfi.c index 1b3fd704ae9a..8287bfd7d6b6 100644 --- a/drivers/thermal/intel/intel_hfi.c +++ b/drivers/thermal/intel/intel_hfi.c @@ -50,6 +50,8 @@ /* Hardware Feedback Interface MSR configuration bits */ #define HW_FEEDBACK_PTR_VALID_BIT BIT(0) #define HW_FEEDBACK_CONFIG_HFI_ENABLE_BIT BIT(0) +#define HW_FEEDBACK_CONFIG_ITD_ENABLE_BIT BIT(1) +#define HW_FEEDBACK_THREAD_CONFIG_ENABLE_BIT BIT(0) /* CPUID detection and enumeration definitions for HFI */ @@ -74,6 +76,15 @@ union cpuid6_edx { u32 full; }; +union cpuid6_ecx { + struct { + u32 dont_care0:8; + u32 nr_classes:8; + u32 dont_care1:16; + } split; + u32 full; +}; + #ifdef CONFIG_IPC_CLASSES union hfi_thread_feedback_char_msr { struct { @@ -495,6 +506,11 @@ void intel_hfi_online(unsigned int cpu) init_hfi_cpu_index(info); + if (cpu_feature_enabled(X86_FEATURE_ITD)) { + msr_val = HW_FEEDBACK_THREAD_CONFIG_ENABLE_BIT; + wrmsrl(MSR_IA32_HW_FEEDBACK_THREAD_CONFIG, msr_val); + } + /* * Now check if the HFI instance of the package/die of @cpu has been * initialized (by checking its header). In such case, all we have to @@ -550,6 +566,10 @@ void intel_hfi_online(unsigned int cpu) */ rdmsrl(MSR_IA32_HW_FEEDBACK_CONFIG, msr_val); msr_val |= HW_FEEDBACK_CONFIG_HFI_ENABLE_BIT; + + if (cpu_feature_enabled(X86_FEATURE_ITD)) + msr_val |= HW_FEEDBACK_CONFIG_ITD_ENABLE_BIT; + wrmsrl(MSR_IA32_HW_FEEDBACK_CONFIG, msr_val); unlock: @@ -629,8 +649,14 @@ static __init int hfi_parse_features(void) */ hfi_features.class_stride = nr_capabilities; - /* For now, use only one class of the HFI table */ - hfi_features.nr_classes = 1; + if (cpu_feature_enabled(X86_FEATURE_ITD)) { + union cpuid6_ecx ecx; + + ecx.full = cpuid_ecx(CPUID_HFI_LEAF); + hfi_features.nr_classes = ecx.split.nr_classes; + } else { + hfi_features.nr_classes = 1; + } /* * The header contains change indications for each supported feature. From patchwork Mon Nov 28 13:20:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ricardo Neri X-Patchwork-Id: 13057495 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C6E68C4167D for ; Mon, 28 Nov 2022 13:14:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231831AbiK1NOT (ORCPT ); Mon, 28 Nov 2022 08:14:19 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39080 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231599AbiK1NNl (ORCPT ); Mon, 28 Nov 2022 08:13:41 -0500 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BF74122D; Mon, 28 Nov 2022 05:13:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1669641220; x=1701177220; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=p7BCBvy/ijiGArW5gb4LxSYPxM/Arci+QhS/bMOw9bo=; b=gSFrBgEqXr5rBXnI3IFSYy+cx+9uhnT4QhrkD3L0uF9xdrzRceWmNDjh ZKMKJJarF4v+ACLXe8Sh03JqoLVY8TjXdK2yU0sQZONt8nuVXGDSe4apo rMCVuxYe9m8XJw9xn4W49uNfPB2J2Sk9DZ0V6QfNNjlR37RAMisrOlJST iWwWrHksfFikdxNVa8UqgYtvEXKkVh2EaXDlzCMrFVJ3eLAYqq6KZqXws c5xI2L11dv4buBjsJwsTvn0AyDd0VoIaq6Vz4is3dg1nQBE+rKPV+VXBV 2h9YaQnS9bR5/oDopZ3nPh2pflANk57kPXVFFnnMCKzINW1Zl8GrG2FFQ Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10544"; a="401117226" X-IronPort-AV: E=Sophos;i="5.96,200,1665471600"; d="scan'208";a="401117226" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Nov 2022 05:13:35 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10544"; a="749381387" X-IronPort-AV: E=Sophos;i="5.96,200,1665471600"; d="scan'208";a="749381387" Received: from ranerica-svr.sc.intel.com ([172.25.110.23]) by fmsmga002.fm.intel.com with ESMTP; 28 Nov 2022 05:13:35 -0800 From: Ricardo Neri To: "Peter Zijlstra (Intel)" , Juri Lelli , Vincent Guittot Cc: Ricardo Neri , "Ravi V. Shankar" , Ben Segall , Daniel Bristot de Oliveira , Dietmar Eggemann , Len Brown , Mel Gorman , "Rafael J. Wysocki" , Srinivas Pandruvada , Steven Rostedt , Tim Chen , Valentin Schneider , x86@kernel.org, "Joel Fernandes (Google)" , linux-kernel@vger.kernel.org, linux-pm@vger.kernel.org, Ricardo Neri , "Tim C . Chen" Subject: [PATCH v2 17/22] sched/task_struct: Add helpers for IPC classification Date: Mon, 28 Nov 2022 05:20:55 -0800 Message-Id: <20221128132100.30253-18-ricardo.neri-calderon@linux.intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20221128132100.30253-1-ricardo.neri-calderon@linux.intel.com> References: <20221128132100.30253-1-ricardo.neri-calderon@linux.intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org The unprocessed classification that hardware provides for a task may not be usable by the scheduler: the classification may change too frequently or architectures may want to consider extra factors. For instance, some processors with Intel Thread Director need to consider the state of the SMT siblings of a core. Provide per-task helper variables that architectures can use to post- process the classification that hardware provides. Cc: Ben Segall Cc: Daniel Bristot de Oliveira Cc: Dietmar Eggemann Cc: Joel Fernandes (Google) Cc: Len Brown Cc: Mel Gorman Cc: Rafael J. Wysocki Cc: Srinivas Pandruvada Cc: Steven Rostedt Cc: Tim C. Chen Cc: Valentin Schneider Cc: x86@kernel.org Cc: linux-pm@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Ricardo Neri --- Changes since v1: * Used bit-fields to fit all the IPC class data in 4 bytes. (PeterZ) * Shortened names of the helpers. * Renamed helpers with the ipcc_ prefix. * Reworded commit message for clarity --- include/linux/sched.h | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index ddabc7449edd..8a99aa316c37 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1532,7 +1532,17 @@ struct task_struct { * A hardware-defined classification of task based on the number * of instructions per cycle. */ - unsigned int ipcc; + unsigned int ipcc : 9; + /* + * A candidate classification that arch-specific implementations + * qualify for correctness. + */ + unsigned int ipcc_tmp : 9; + /* + * Counter to filter out transient the candidate classification + * of a task + */ + unsigned int ipcc_cntr : 14; #endif /* From patchwork Mon Nov 28 13:20:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ricardo Neri X-Patchwork-Id: 13057497 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1092BC4167D for ; Mon, 28 Nov 2022 13:14:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230502AbiK1NOU (ORCPT ); Mon, 28 Nov 2022 08:14:20 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39894 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231607AbiK1NNl (ORCPT ); Mon, 28 Nov 2022 08:13:41 -0500 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C07FE1CFC2; Mon, 28 Nov 2022 05:13:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1669641220; x=1701177220; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=KWVB7nlsf20aLmkJw9PMc9fnASyxtCkpqLu/IPZ9sIY=; b=gNCDRFydNHxGrZ32vgvgI8pBRG7LjpNHtdZzMzyuZ8fXB50Uwp9lS3kA zRoRQPUFn5de46HRr7ZQ0XtAZjS5JTFIKMxG6YaJQ8yhdeAb692U3a5lz ddTIozmiUIT/bUIamOMSsrzn2+clHiq2Bbf27x66B0kySRWFuc+HrSSI3 AUQXSVS4riThDleOtdzqdRFCp1EU8JYHP/JptWOUl42lRUMyceLqsS+cS WlehERQ4rOdZkTyepAbsEHN1ulF3wM+Q7NBexfVzod03maujcwqaCd/fT MQl2rA2qzsFcYqON4Vzc9K56Rj4InE42wLT0QjnYCqvOH+kp1+kp0uZ6x A==; X-IronPort-AV: E=McAfee;i="6500,9779,10544"; a="401117238" X-IronPort-AV: E=Sophos;i="5.96,200,1665471600"; d="scan'208";a="401117238" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Nov 2022 05:13:36 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10544"; a="749381390" X-IronPort-AV: E=Sophos;i="5.96,200,1665471600"; d="scan'208";a="749381390" Received: from ranerica-svr.sc.intel.com ([172.25.110.23]) by fmsmga002.fm.intel.com with ESMTP; 28 Nov 2022 05:13:35 -0800 From: Ricardo Neri To: "Peter Zijlstra (Intel)" , Juri Lelli , Vincent Guittot Cc: Ricardo Neri , "Ravi V. Shankar" , Ben Segall , Daniel Bristot de Oliveira , Dietmar Eggemann , Len Brown , Mel Gorman , "Rafael J. Wysocki" , Srinivas Pandruvada , Steven Rostedt , Tim Chen , Valentin Schneider , x86@kernel.org, "Joel Fernandes (Google)" , linux-kernel@vger.kernel.org, linux-pm@vger.kernel.org, Ricardo Neri , "Tim C . Chen" Subject: [PATCH v2 18/22] sched/core: Initialize helpers of task classification Date: Mon, 28 Nov 2022 05:20:56 -0800 Message-Id: <20221128132100.30253-19-ricardo.neri-calderon@linux.intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20221128132100.30253-1-ricardo.neri-calderon@linux.intel.com> References: <20221128132100.30253-1-ricardo.neri-calderon@linux.intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org Just as tasks start life unclassified, initialize the classification auxiliar variables. Cc: Ben Segall Cc: Daniel Bristot de Oliveira Cc: Dietmar Eggemann Cc: Joel Fernandes (Google) Cc: Len Brown Cc: Mel Gorman Cc: Rafael J. Wysocki Cc: Srinivas Pandruvada Cc: Steven Rostedt Cc: Tim C. Chen Cc: Valentin Schneider Cc: x86@kernel.org Cc: linux-pm@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Ricardo Neri --- Changes since v1: * None --- kernel/sched/core.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 2cd409536b72..0406b07c51a0 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -4374,6 +4374,8 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p) p->se.vruntime = 0; #ifdef CONFIG_IPC_CLASSES p->ipcc = IPC_CLASS_UNCLASSIFIED; + p->ipcc_tmp = IPC_CLASS_UNCLASSIFIED; + p->ipcc_cntr = 0; #endif INIT_LIST_HEAD(&p->se.group_node); From patchwork Mon Nov 28 13:20:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ricardo Neri X-Patchwork-Id: 13057496 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0016EC433FE for ; Mon, 28 Nov 2022 13:14:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231838AbiK1NOU (ORCPT ); Mon, 28 Nov 2022 08:14:20 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39878 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231258AbiK1NNl (ORCPT ); Mon, 28 Nov 2022 08:13:41 -0500 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C06A41CB37; Mon, 28 Nov 2022 05:13:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1669641220; x=1701177220; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=6bnvjZwJXP3wVx/GzfxHDbk1nNW034G+XQZFyKkDwVI=; b=amtFdyXMYRlgqZ6C2IJoEYMFqv7J+hI5ztwU8lFKkAn6ZsCKsJ7iJI31 76gQv550LresMwB59yQfho+lV3JuSd2+VWD4eJlgGxhMGnjP1+ltI6CoZ ESJ+Nb6v0BM5B1yigVQLtxul6plcd7magRzbu3XQMIJBU3odV4uBFgs6T w9zb7VflXdnNwYfJExNjRi58n4bQbEXk6P5rq/G7ALyoNBs/v2Hv6YTW6 juTbRSK8QOCS04SY/Hv41JXvRvCvAicaYTn1kgQfGh+szSCXm7r4ugsJn Rd5DnlCWbL0Ziwn4vXFUqIfNLS7gVzmWli7LSp5SKo55Gm9aFnuY2xz1m Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10544"; a="401117249" X-IronPort-AV: E=Sophos;i="5.96,200,1665471600"; d="scan'208";a="401117249" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Nov 2022 05:13:36 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10544"; a="749381393" X-IronPort-AV: E=Sophos;i="5.96,200,1665471600"; d="scan'208";a="749381393" Received: from ranerica-svr.sc.intel.com ([172.25.110.23]) by fmsmga002.fm.intel.com with ESMTP; 28 Nov 2022 05:13:35 -0800 From: Ricardo Neri To: "Peter Zijlstra (Intel)" , Juri Lelli , Vincent Guittot Cc: Ricardo Neri , "Ravi V. Shankar" , Ben Segall , Daniel Bristot de Oliveira , Dietmar Eggemann , Len Brown , Mel Gorman , "Rafael J. Wysocki" , Srinivas Pandruvada , Steven Rostedt , Tim Chen , Valentin Schneider , x86@kernel.org, "Joel Fernandes (Google)" , linux-kernel@vger.kernel.org, linux-pm@vger.kernel.org, Ricardo Neri , "Tim C . Chen" Subject: [PATCH v2 19/22] thermal: intel: hfi: Implement model-specific checks for task classification Date: Mon, 28 Nov 2022 05:20:57 -0800 Message-Id: <20221128132100.30253-20-ricardo.neri-calderon@linux.intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20221128132100.30253-1-ricardo.neri-calderon@linux.intel.com> References: <20221128132100.30253-1-ricardo.neri-calderon@linux.intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org In Alderlake and Raptorlake, the result of thread classification is more accurate when only one SMT sibling is busy. Classification results for class 2 and 3 that are always reliable. To avoid unnecessary migrations, only update the class of a task if it has been the same during 4 consecutive ticks. Cc: Ben Segall Cc: Daniel Bristot de Oliveira Cc: Dietmar Eggemann Cc: Joel Fernandes (Google) Cc: Len Brown Cc: Mel Gorman Cc: Rafael J. Wysocki Cc: Srinivas Pandruvada Cc: Steven Rostedt Cc: Tim C. Chen Cc: Valentin Schneider Cc: x86@kernel.org Cc: linux-pm@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Ricardo Neri --- Changes since v1: * Adjusted the result the classification of Intel Thread Director to start at class 1. Class 0 for the scheduler means that the task is unclassified. * Used the new names of the IPC classes members in task_struct. * Reworked helper functions to use sched_smt_siblings_idle() to query the idle state of the SMT siblings of a CPU. --- drivers/thermal/intel/intel_hfi.c | 60 ++++++++++++++++++++++++++++++- 1 file changed, 59 insertions(+), 1 deletion(-) diff --git a/drivers/thermal/intel/intel_hfi.c b/drivers/thermal/intel/intel_hfi.c index 8287bfd7d6b6..a9ae09036909 100644 --- a/drivers/thermal/intel/intel_hfi.c +++ b/drivers/thermal/intel/intel_hfi.c @@ -40,6 +40,7 @@ #include #include +#include #include "../thermal_core.h" #include "intel_hfi.h" @@ -216,9 +217,64 @@ int intel_hfi_has_ipc_classes(void) return cpu_feature_enabled(X86_FEATURE_ITD); } +#define CLASS_DEBOUNCER_SKIPS 4 + +/** + * debounce_and_update_class() - Process and update a task's classification + * + * @p: The task of which the classification will be updated + * @new_ipcc: The new IPC classification + * + * Update the classification of @p with the new value that hardware provides. + * Only update the classification of @p if it has been the same during + * CLASS_DEBOUNCER_SKIPS consecutive ticks. + */ +static void debounce_and_update_class(struct task_struct *p, u8 new_ipcc) +{ + u16 debounce_skip; + + /* The class of @p changed, only restart the debounce counter. */ + if (p->ipcc_tmp != new_ipcc) { + p->ipcc_cntr = 1; + goto out; + } + + /* + * The class of @p did not change. Update it if it has been the same + * for CLASS_DEBOUNCER_SKIPS user ticks. + */ + debounce_skip = p->ipcc_cntr + 1; + if (debounce_skip < CLASS_DEBOUNCER_SKIPS) + p->ipcc_cntr++; + else + p->ipcc = new_ipcc; + +out: + p->ipcc_tmp = new_ipcc; +} + +static bool classification_is_accurate(u8 hfi_class, bool smt_siblings_idle) +{ + switch (boot_cpu_data.x86_model) { + case INTEL_FAM6_ALDERLAKE: + case INTEL_FAM6_ALDERLAKE_L: + case INTEL_FAM6_RAPTORLAKE: + case INTEL_FAM6_RAPTORLAKE_P: + case INTEL_FAM6_RAPTORLAKE_S: + if (hfi_class == 3 || hfi_class == 2 || smt_siblings_idle) + return true; + + return false; + + default: + return true; + } +} + void intel_hfi_update_ipcc(struct task_struct *curr) { union hfi_thread_feedback_char_msr msr; + bool idle; /* We should not be here if ITD is not supported. */ if (!cpu_feature_enabled(X86_FEATURE_ITD)) { @@ -234,7 +290,9 @@ void intel_hfi_update_ipcc(struct task_struct *curr) * 0 is a valid classification for Intel Thread Director. A scheduler * IPCC class of 0 means that the task is unclassified. Adjust. */ - curr->ipcc = msr.split.classid + 1; + idle = sched_smt_siblings_idle(task_cpu(curr)); + if (classification_is_accurate(msr.split.classid, idle)) + debounce_and_update_class(curr, msr.split.classid + 1); } int intel_hfi_get_ipcc_score(unsigned short ipcc, int cpu) From patchwork Mon Nov 28 13:20:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ricardo Neri X-Patchwork-Id: 13057498 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B710DC46467 for ; Mon, 28 Nov 2022 13:14:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231841AbiK1NOX (ORCPT ); Mon, 28 Nov 2022 08:14:23 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40168 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231655AbiK1NNm (ORCPT ); Mon, 28 Nov 2022 08:13:42 -0500 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8A3921DA5F; Mon, 28 Nov 2022 05:13:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1669641221; x=1701177221; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=U9TIyYUMDTGWF/am2zEWFCJshAZh6Eamp8JHDo02tn8=; b=d1X0izXQsq8bimLuxoByKBtb50BbTS+NFW017jW42z9oH2Kx7+MdQCvw DHnuol0lmQF9TtLyWyC9pnifMZ6+TH27FiVyCEb+1YQCl8zJxw0yR66sC 36dmkJm5y9XZnJlFOwIz2+5KvQjomBjdDvjFhNaRP5Ul8okM6MEXs4/ye USKMXdv4UnRDepjeAdDyn0KDAjbFBjzsgF4vL0iZlVu74c8/SwZwBbjts 8xc/pV7265ZSHrW0afZzECNLxJfa5oCmRb/fIJ9UcTwdmC9nc9xopYDm4 /SsGNA2pbDDXf4C+sMg/TOVhQyuget34YhnmjDe9LaDLPeCBNYsa7699Z g==; X-IronPort-AV: E=McAfee;i="6500,9779,10544"; a="401117261" X-IronPort-AV: E=Sophos;i="5.96,200,1665471600"; d="scan'208";a="401117261" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Nov 2022 05:13:36 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10544"; a="749381398" X-IronPort-AV: E=Sophos;i="5.96,200,1665471600"; d="scan'208";a="749381398" Received: from ranerica-svr.sc.intel.com ([172.25.110.23]) by fmsmga002.fm.intel.com with ESMTP; 28 Nov 2022 05:13:36 -0800 From: Ricardo Neri To: "Peter Zijlstra (Intel)" , Juri Lelli , Vincent Guittot Cc: Ricardo Neri , "Ravi V. Shankar" , Ben Segall , Daniel Bristot de Oliveira , Dietmar Eggemann , Len Brown , Mel Gorman , "Rafael J. Wysocki" , Srinivas Pandruvada , Steven Rostedt , Tim Chen , Valentin Schneider , x86@kernel.org, "Joel Fernandes (Google)" , linux-kernel@vger.kernel.org, linux-pm@vger.kernel.org, Ricardo Neri , "Tim C . Chen" Subject: [PATCH v2 20/22] x86/cpufeatures: Add feature bit for HRESET Date: Mon, 28 Nov 2022 05:20:58 -0800 Message-Id: <20221128132100.30253-21-ricardo.neri-calderon@linux.intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20221128132100.30253-1-ricardo.neri-calderon@linux.intel.com> References: <20221128132100.30253-1-ricardo.neri-calderon@linux.intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org The HRESET instruction prevents the classification of the current task from influencing the classification of the next task when running serially on the same logical processor. Cc: Ben Segall Cc: Daniel Bristot de Oliveira Cc: Dietmar Eggemann Cc: Joel Fernandes (Google) Cc: Len Brown Cc: Mel Gorman Cc: Rafael J. Wysocki Cc: Srinivas Pandruvada Cc: Steven Rostedt Cc: Tim C. Chen Cc: Valentin Schneider Cc: x86@kernel.org Cc: linux-pm@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Ricardo Neri --- Changes since v1: * None --- arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/msr-index.h | 4 +++- arch/x86/kernel/cpu/scattered.c | 1 + 3 files changed, 5 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h index 80b2beafc81e..281a7c861b8d 100644 --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -308,6 +308,7 @@ #define X86_FEATURE_CALL_DEPTH (11*32+19) /* "" Call depth tracking for RSB stuffing */ #define X86_FEATURE_MSR_TSX_CTRL (11*32+20) /* "" MSR IA32_TSX_CTRL (Intel) implemented */ +#define X86_FEATURE_HRESET (11*32+21) /* Hardware history reset instruction */ /* Intel-defined CPU features, CPUID level 0x00000007:1 (EAX), word 12 */ #define X86_FEATURE_AVX_VNNI (12*32+ 4) /* AVX VNNI instructions */ diff --git a/arch/x86/include/asm/msr-index.h b/arch/x86/include/asm/msr-index.h index 96303330223b..7a3ff73164bd 100644 --- a/arch/x86/include/asm/msr-index.h +++ b/arch/x86/include/asm/msr-index.h @@ -1078,6 +1078,9 @@ #define MSR_IA32_HW_FEEDBACK_THREAD_CONFIG 0x17d4 #define MSR_IA32_HW_FEEDBACK_CHAR 0x17d2 +/* Hardware History Reset */ +#define MSR_IA32_HW_HRESET_ENABLE 0x17da + /* x2APIC locked status */ #define MSR_IA32_XAPIC_DISABLE_STATUS 0xBD #define LEGACY_XAPIC_DISABLED BIT(0) /* @@ -1085,5 +1088,4 @@ * disabling x2APIC will cause * a #GP */ - #endif /* _ASM_X86_MSR_INDEX_H */ diff --git a/arch/x86/kernel/cpu/scattered.c b/arch/x86/kernel/cpu/scattered.c index f53944fb8f7f..66bc5713644d 100644 --- a/arch/x86/kernel/cpu/scattered.c +++ b/arch/x86/kernel/cpu/scattered.c @@ -28,6 +28,7 @@ static const struct cpuid_bit cpuid_bits[] = { { X86_FEATURE_EPB, CPUID_ECX, 3, 0x00000006, 0 }, { X86_FEATURE_INTEL_PPIN, CPUID_EBX, 0, 0x00000007, 1 }, { X86_FEATURE_RRSBA_CTRL, CPUID_EDX, 2, 0x00000007, 2 }, + { X86_FEATURE_HRESET, CPUID_EAX, 22, 0x00000007, 1 }, { X86_FEATURE_CQM_LLC, CPUID_EDX, 1, 0x0000000f, 0 }, { X86_FEATURE_CQM_OCCUP_LLC, CPUID_EDX, 0, 0x0000000f, 1 }, { X86_FEATURE_CQM_MBM_TOTAL, CPUID_EDX, 1, 0x0000000f, 1 }, From patchwork Mon Nov 28 13:20:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ricardo Neri X-Patchwork-Id: 13057499 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0AF62C433FE for ; Mon, 28 Nov 2022 13:14:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231847AbiK1NOY (ORCPT ); Mon, 28 Nov 2022 08:14:24 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40170 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231666AbiK1NNm (ORCPT ); Mon, 28 Nov 2022 08:13:42 -0500 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 907011B9C4; Mon, 28 Nov 2022 05:13:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1669641221; x=1701177221; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=PIe0SGbn3SQDW/BsVmI7VngYYmBv3E8AzHVNV7RbEPw=; b=E/Bedru4SRJxz0kyZ8401LjnEyu0QpOhrf3kb9UrBjy1Xr5fxQb/8Y2k znOEIgvlr+EQq7fnSoGKQ9HS+kbJuBc3cvjqN9/zDtWDt/gr5pySlWN5X 0rJycqYZZu+wdBpBKCT/M6ky265Kq+lHdjd1JxCz7D9mgoPRJqc7mTs2O QflTBSm/RsqFXxttcvIUT6XTO5HcNjyung4F2iygT9MBaxN5JbpAzjRcz fh+bGDm8l3q5wzjoHUwN4pdvQBa/cCIRA/i+TWr+AX4hieC6YdFoldlu2 6qTAEBSoiYgWZTibirPYMYVlJ/VVZ/7OgTjmlwodhbpsgTM+brpWv09gm w==; X-IronPort-AV: E=McAfee;i="6500,9779,10544"; a="401117273" X-IronPort-AV: E=Sophos;i="5.96,200,1665471600"; d="scan'208";a="401117273" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Nov 2022 05:13:36 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10544"; a="749381402" X-IronPort-AV: E=Sophos;i="5.96,200,1665471600"; d="scan'208";a="749381402" Received: from ranerica-svr.sc.intel.com ([172.25.110.23]) by fmsmga002.fm.intel.com with ESMTP; 28 Nov 2022 05:13:36 -0800 From: Ricardo Neri To: "Peter Zijlstra (Intel)" , Juri Lelli , Vincent Guittot Cc: Ricardo Neri , "Ravi V. Shankar" , Ben Segall , Daniel Bristot de Oliveira , Dietmar Eggemann , Len Brown , Mel Gorman , "Rafael J. Wysocki" , Srinivas Pandruvada , Steven Rostedt , Tim Chen , Valentin Schneider , x86@kernel.org, "Joel Fernandes (Google)" , linux-kernel@vger.kernel.org, linux-pm@vger.kernel.org, Ricardo Neri , "Tim C . Chen" Subject: [PATCH v2 21/22] x86/hreset: Configure history reset Date: Mon, 28 Nov 2022 05:20:59 -0800 Message-Id: <20221128132100.30253-22-ricardo.neri-calderon@linux.intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20221128132100.30253-1-ricardo.neri-calderon@linux.intel.com> References: <20221128132100.30253-1-ricardo.neri-calderon@linux.intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org Configure the MSR that controls the behavior of HRESET on each logical processor. Cc: Ben Segall Cc: Daniel Bristot de Oliveira Cc: Dietmar Eggemann Cc: Joel Fernandes (Google) Cc: Len Brown Cc: Mel Gorman Cc: Rafael J. Wysocki Cc: Srinivas Pandruvada Cc: Steven Rostedt Cc: Tim C. Chen Cc: Valentin Schneider Cc: x86@kernel.org Cc: linux-pm@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Ricardo Neri --- Changes since v1: * Marked hardware_history_features as __ro_after_init instead of __read_mostly. (PeterZ) --- arch/x86/kernel/cpu/common.c | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index 73cc546e024d..f8630da2a6dd 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -412,6 +412,26 @@ static __always_inline void setup_umip(struct cpuinfo_x86 *c) cr4_clear_bits(X86_CR4_UMIP); } +static u32 hardware_history_features __ro_after_init; + +static __always_inline void setup_hreset(struct cpuinfo_x86 *c) +{ + if (!cpu_feature_enabled(X86_FEATURE_HRESET)) + return; + + /* + * Use on all CPUs the hardware history features that the boot + * CPU supports. + */ + if (c == &boot_cpu_data) + hardware_history_features = cpuid_ebx(0x20); + + if (!hardware_history_features) + return; + + wrmsrl(MSR_IA32_HW_HRESET_ENABLE, hardware_history_features); +} + /* These bits should not change their value after CPU init is finished. */ static const unsigned long cr4_pinned_mask = X86_CR4_SMEP | X86_CR4_SMAP | X86_CR4_UMIP | @@ -1844,10 +1864,11 @@ static void identify_cpu(struct cpuinfo_x86 *c) /* Disable the PN if appropriate */ squash_the_stupid_serial_number(c); - /* Set up SMEP/SMAP/UMIP */ + /* Set up SMEP/SMAP/UMIP/HRESET */ setup_smep(c); setup_smap(c); setup_umip(c); + setup_hreset(c); /* Enable FSGSBASE instructions if available. */ if (cpu_has(c, X86_FEATURE_FSGSBASE)) { From patchwork Mon Nov 28 13:21:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ricardo Neri X-Patchwork-Id: 13057500 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C0BE9C47089 for ; Mon, 28 Nov 2022 13:14:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231860AbiK1NO1 (ORCPT ); Mon, 28 Nov 2022 08:14:27 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39908 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231611AbiK1NNm (ORCPT ); Mon, 28 Nov 2022 08:13:42 -0500 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E08141DA61; Mon, 28 Nov 2022 05:13:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1669641221; x=1701177221; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=HY2fKdj5Se/fueVNKplVVj75mjSgfhPku4CZtL+A+14=; b=Dgkl7iS7+1RSuICNFpLw6aI+H/O68dtl5VdMfg/wzxMvbWp9G5lVZ1YT jd+ha2kfxx8hLtZOjpfBXdztE9POFNkKS2x1BYYMxDnzYxEy2KLkWNYjH gbyNBH76I8VZwrtC2Vb4iJqEuwbDhDQqg8ts3vNTpAarns66dsfZbKstx rY48qOddNnO6LjeAR4+EouCHWPvq0hsv79dWlZfjtxYZ28OMvbOw4QZGq OSbKYGD/Ig5gDgyWlJ5JxoUOzTm4S1TqcOQGQXkaMYX523PDGkwzI5ojh Iy1j+xh2IRRufyubgiNNQ7MM2/I7NkJ6Frz7DWBWsWjXR5s4jNQjRLUOA A==; X-IronPort-AV: E=McAfee;i="6500,9779,10544"; a="401117286" X-IronPort-AV: E=Sophos;i="5.96,200,1665471600"; d="scan'208";a="401117286" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Nov 2022 05:13:37 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10544"; a="749381406" X-IronPort-AV: E=Sophos;i="5.96,200,1665471600"; d="scan'208";a="749381406" Received: from ranerica-svr.sc.intel.com ([172.25.110.23]) by fmsmga002.fm.intel.com with ESMTP; 28 Nov 2022 05:13:36 -0800 From: Ricardo Neri To: "Peter Zijlstra (Intel)" , Juri Lelli , Vincent Guittot Cc: Ricardo Neri , "Ravi V. Shankar" , Ben Segall , Daniel Bristot de Oliveira , Dietmar Eggemann , Len Brown , Mel Gorman , "Rafael J. Wysocki" , Srinivas Pandruvada , Steven Rostedt , Tim Chen , Valentin Schneider , x86@kernel.org, "Joel Fernandes (Google)" , linux-kernel@vger.kernel.org, linux-pm@vger.kernel.org, Ricardo Neri , "Tim C . Chen" Subject: [PATCH v2 22/22] x86/process: Reset hardware history in context switch Date: Mon, 28 Nov 2022 05:21:00 -0800 Message-Id: <20221128132100.30253-23-ricardo.neri-calderon@linux.intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20221128132100.30253-1-ricardo.neri-calderon@linux.intel.com> References: <20221128132100.30253-1-ricardo.neri-calderon@linux.intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org Reset the classification history of the current task when switching to the next task. Hardware will start the classification of the next task from scratch. Cc: Ben Segall Cc: Daniel Bristot de Oliveira Cc: Dietmar Eggemann Cc: Joel Fernandes (Google) Cc: Len Brown Cc: Mel Gorman Cc: Rafael J. Wysocki Cc: Srinivas Pandruvada Cc: Steven Rostedt Cc: Tim C. Chen Cc: Valentin Schneider Cc: x86@kernel.org Cc: linux-pm@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Ricardo Neri --- Changes since v1: * Measurements of the cost of the HRESET instruction Methodology: I created a tight loop with interrupts and preemption disabled. I recorded the value of the TSC counter before and after executing HRESET or RDTSC. I repeated the measurement 100,000 times. I performed the experiment using an Alder Lake S system. I set the frequency of the CPUs at a fixed value. The table below compares the cost of HRESET with RDTSC (expressed in the elapsed TSC count). The cost of the two instructions is comparable. PCore ECore Frequency (GHz) 5.0 3.8 HRESET (avg) 28.5 44.7 HRESET (stdev %) 3.6 2.3 RDTSC (avg) 25.2 35.7 RDTSC (stdev %) 3.9 2.6 * Used an ALTERNATIVE macro instead of static_cpu_has() to execute HRESET when supported. (PeterZ) --- arch/x86/include/asm/hreset.h | 30 ++++++++++++++++++++++++++++++ arch/x86/kernel/cpu/common.c | 7 +++++++ arch/x86/kernel/process_32.c | 3 +++ arch/x86/kernel/process_64.c | 3 +++ 4 files changed, 43 insertions(+) create mode 100644 arch/x86/include/asm/hreset.h diff --git a/arch/x86/include/asm/hreset.h b/arch/x86/include/asm/hreset.h new file mode 100644 index 000000000000..d68ca2fb8642 --- /dev/null +++ b/arch/x86/include/asm/hreset.h @@ -0,0 +1,30 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_HRESET_H + +/** + * HRESET - History reset. Available since binutils v2.36. + * + * Request the processor to reset the history of task classification on the + * current logical processor. The history components to be + * reset are specified in %eax. Only bits specified in CPUID(0x20).EBX + * and enabled in the IA32_HRESET_ENABLE MSR can be selected. + * + * The assembly code looks like: + * + * hreset %eax + * + * The corresponding machine code looks like: + * + * F3 0F 3A F0 ModRM Imm + * + * The value of ModRM is 0xc0 to specify %eax register addressing. + * The ignored immediate operand is set to 0. + * + * The instruction is documented in the Intel SDM. + */ + +#define __ASM_HRESET ".byte 0xf3, 0xf, 0x3a, 0xf0, 0xc0, 0x0" + +void reset_hardware_history(void); + +#endif /* _ASM_X86_HRESET_H */ diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index f8630da2a6dd..6c2b9768698e 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -53,6 +53,7 @@ #include #include #include +#include #include #include #include @@ -414,6 +415,12 @@ static __always_inline void setup_umip(struct cpuinfo_x86 *c) static u32 hardware_history_features __ro_after_init; +void reset_hardware_history(void) +{ + asm_inline volatile (ALTERNATIVE("", __ASM_HRESET, X86_FEATURE_HRESET) + : : "a" (hardware_history_features) : "memory"); +} + static __always_inline void setup_hreset(struct cpuinfo_x86 *c) { if (!cpu_feature_enabled(X86_FEATURE_HRESET)) diff --git a/arch/x86/kernel/process_32.c b/arch/x86/kernel/process_32.c index 470c128759ea..397a6e6f4e61 100644 --- a/arch/x86/kernel/process_32.c +++ b/arch/x86/kernel/process_32.c @@ -52,6 +52,7 @@ #include #include #include +#include #include #include "process.h" @@ -214,6 +215,8 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p) /* Load the Intel cache allocation PQR MSR. */ resctrl_sched_in(); + reset_hardware_history(); + return prev_p; } diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 084ec467dbb1..ac9b3d44c1bd 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -53,6 +53,7 @@ #include #include #include +#include #include #include #ifdef CONFIG_IA32_EMULATION @@ -658,6 +659,8 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p) /* Load the Intel cache allocation PQR MSR. */ resctrl_sched_in(); + reset_hardware_history(); + return prev_p; }