From patchwork Wed Apr 17 09:38:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Beata Michalska X-Patchwork-Id: 13633115 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 45FA3C4345F for ; Wed, 17 Apr 2024 09:39:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=PJEgSA8+QPm7QpB3ZfleRPJpO7+VyxflJT3Nqyg/mQQ=; b=v5XzNp7PR64smp X/E2xXlg6DufyQ7tB87QLq9W1SGX0lqGMYTbbFbmeBIAetu9XHYZHZ5w+7WLkQMwv07Be3OIIBbDA e94ub1fBnHxkE1l/tMWzHmXO7av0drDb9tqhGBLuUuHp/f19SgbjkaSettag+O7jilwaO2oI1LE4E 6yLnZqv9QoXStvLYlDi+c3WEhfrlvpU9jXVPntOXyC5nugfmfatqzG+b32SC/P5wBsOj6qDBJSYjS QX6Mvub/WlBvXttWDyXeINGdx5ztGC0Pwdc8D2AfTV9K7YtN6qnJhAhzGlRh3OzReKu8paqKtdUBM +g4vOQAjXMIEcPc52HEQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rx1l3-0000000FRkM-1f7P; Wed, 17 Apr 2024 09:39:13 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rx1kt-0000000FRgc-0aM7 for linux-arm-kernel@lists.infradead.org; Wed, 17 Apr 2024 09:39:04 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 20C32DA7; Wed, 17 Apr 2024 02:39:30 -0700 (PDT) Received: from e125905.cambridge.arm.com (e125905.cambridge.arm.com [10.1.194.73]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 71D283F64C; Wed, 17 Apr 2024 02:39:00 -0700 (PDT) From: Beata Michalska To: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, ionela.voinescu@arm.com, vanshikonda@os.amperecomputing.com Cc: sudeep.holla@arm.com, will@kernel.org, catalin.marinas@arm.com, vincent.guittot@linaro.org, sumitg@nvidia.com, yang@os.amperecomputing.com, lihuisong@huawei.com, viresh.kumar@linaro.org Subject: [PATCH v5 1/5] arch_topology: init capacity_freq_ref to 0 Date: Wed, 17 Apr 2024 10:38:44 +0100 Message-Id: <20240417093848.1555462-2-beata.michalska@arm.com> In-Reply-To: <20240417093848.1555462-1-beata.michalska@arm.com> References: <20240417093848.1555462-1-beata.michalska@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240417_023903_264287_C9B698C5 X-CRM114-Status: UNSURE ( 9.60 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ionela Voinescu It's useful to have capacity_freq_ref initialized to 0 for users of arch_scale_freq_ref() to detect when capacity_freq_ref was not yet set. The only scenario affected by this change in the init value is when a cpufreq driver is never loaded. As a result, the only setter of a cpu scale factor remains the call of topology_normalize_cpu_scale() from parse_dt_topology(). There we cannot use the value 0 of capacity_freq_ref so we have to compensate for its uninitialized state. Signed-off-by: Ionela Voinescu Signed-off-by: Beata Michalska Reviewed-by: Vincent Guittot Reviewed-by: Sudeep Holla --- drivers/base/arch_topology.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/drivers/base/arch_topology.c b/drivers/base/arch_topology.c index 024b78a0cfc1..7d4c92cd2bad 100644 --- a/drivers/base/arch_topology.c +++ b/drivers/base/arch_topology.c @@ -27,7 +27,7 @@ static DEFINE_PER_CPU(struct scale_freq_data __rcu *, sft_data); static struct cpumask scale_freq_counters_mask; static bool scale_freq_invariant; -DEFINE_PER_CPU(unsigned long, capacity_freq_ref) = 1; +DEFINE_PER_CPU(unsigned long, capacity_freq_ref) = 0; EXPORT_PER_CPU_SYMBOL_GPL(capacity_freq_ref); static bool supports_scale_freq_counters(const struct cpumask *cpus) @@ -292,13 +292,15 @@ void topology_normalize_cpu_scale(void) capacity_scale = 1; for_each_possible_cpu(cpu) { - capacity = raw_capacity[cpu] * per_cpu(capacity_freq_ref, cpu); + capacity = raw_capacity[cpu] * + (per_cpu(capacity_freq_ref, cpu) ?: 1); capacity_scale = max(capacity, capacity_scale); } pr_debug("cpu_capacity: capacity_scale=%llu\n", capacity_scale); for_each_possible_cpu(cpu) { - capacity = raw_capacity[cpu] * per_cpu(capacity_freq_ref, cpu); + capacity = raw_capacity[cpu] * + (per_cpu(capacity_freq_ref, cpu) ?: 1); capacity = div64_u64(capacity << SCHED_CAPACITY_SHIFT, capacity_scale); topology_set_cpu_scale(cpu, capacity); From patchwork Wed Apr 17 09:38:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Beata Michalska X-Patchwork-Id: 13633113 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4D496C4345F for ; Wed, 17 Apr 2024 09:39:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=w0Ublsh64zBgybwvDVlOZPL9YOLEhnkTEQ3MZksCxpo=; b=OtHpc20aHxnymz 2Vei0x4nKVkbzks9h3SXrlbOevkJdh5AtWZmtpEWsJItHIHa98f6TW4rXpKlLA9f08txbThptAW0G AnRudX49BSHWcG4m2FVPdC1LmX5x4HYr6w58GdrXV7kn5sf0FABD7mm1kTd5a6i+HX9pfBOI8Xh77 AO6yoIMIHI9ySBKMVjsyijiGCxboSD4jy5XOsITfi6YWY/qUw85eX9YKRNW/qZCWi+U1A6VwkRHVB s+vra1SjKIe5VLQlrni+GFY6kwiKicAjv7Tl9RaCKRsuGuHok4zVoeTm/iHj95ASilbffGhMX1WK9 eAJROvjDZeUNDkBO2oWg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rx1l4-0000000FRkg-0Mh8; Wed, 17 Apr 2024 09:39:14 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rx1ku-0000000FRh6-3wmU for linux-arm-kernel@lists.infradead.org; Wed, 17 Apr 2024 09:39:06 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 3870B1477; Wed, 17 Apr 2024 02:39:32 -0700 (PDT) Received: from e125905.cambridge.arm.com (e125905.cambridge.arm.com [10.1.194.73]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 808E13F64C; Wed, 17 Apr 2024 02:39:02 -0700 (PDT) From: Beata Michalska To: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, ionela.voinescu@arm.com, vanshikonda@os.amperecomputing.com Cc: sudeep.holla@arm.com, will@kernel.org, catalin.marinas@arm.com, vincent.guittot@linaro.org, sumitg@nvidia.com, yang@os.amperecomputing.com, lihuisong@huawei.com, viresh.kumar@linaro.org Subject: [PATCH v5 2/5] arm64: amu: Rule out potential use after free Date: Wed, 17 Apr 2024 10:38:45 +0100 Message-Id: <20240417093848.1555462-3-beata.michalska@arm.com> In-Reply-To: <20240417093848.1555462-1-beata.michalska@arm.com> References: <20240417093848.1555462-1-beata.michalska@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240417_023905_105600_A1243D91 X-CRM114-Status: GOOD ( 10.00 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org For the time being, the amu_fie_cpus cpumask is being exclusively used by the AMU-related internals of FIE support and is guaranteed to be valid on every access currently made. Still the mask is not being invalidated on one of the error handling code paths, which leaves a soft spot with potential risk of uaf for CPUMASK_OFFSTACK cases. To make things sound, set the cpumaks pointer explicitly to NULL upon failing to register the cpufreq notifier. Note that, due to the quirks of CPUMASK_OFFSTACK, this change needs to be wrapped with grim ifdefing (it would be better served by incorporating this into free_cpumask_var ...) Signed-off-by: Beata Michalska --- arch/arm64/kernel/topology.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kernel/topology.c b/arch/arm64/kernel/topology.c index 1a2c72f3e7f8..3c814a278534 100644 --- a/arch/arm64/kernel/topology.c +++ b/arch/arm64/kernel/topology.c @@ -244,8 +244,12 @@ static int __init init_amu_fie(void) ret = cpufreq_register_notifier(&init_amu_fie_notifier, CPUFREQ_POLICY_NOTIFIER); - if (ret) + if (ret) { free_cpumask_var(amu_fie_cpus); +#ifdef CONFIG_CPUMASK_OFFSTACK + amu_fie_cpus = NULL; +#endif + } return ret; } From patchwork Wed Apr 17 09:38:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Beata Michalska X-Patchwork-Id: 13633116 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 94B73C001CC for ; Wed, 17 Apr 2024 09:39:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=f/9AG5g7Pp+2ESv69ql6Rn9h6VNUfV9EmJADA7an3CU=; b=g3XxOMCtp55e3A i84Dlj/Lujtr8SyCKDb0lwBkCv99C25De9/VJEUMYY0h4gwqfApmaOA+Lo/axnscxAExgP07XecQ8 R2GMMjgSjE0Ayacab3wRtfm1degp+GB9287ltyFO0cIQJGofMMGVANuWklZzi9RKhpTTR/QS5ikXn QC/9YePyYY9KN6WvmSUdGS8uAPbahSngErmrh4GzjcS2X/RqfCexMLmAAnDndYr4xOecK5D/bR4gc YCbE/XvD3lCiHqIeI8FQbCRKu5s3a6SWaMnWd/xcsCzEFgtIoVOObOBJssjhcUyKjeqKrQAGPnxlm znFhHiReIWxaOB9crAMw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rx1l4-0000000FRlE-3gVm; Wed, 17 Apr 2024 09:39:14 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rx1kw-0000000FRhb-3MMx for linux-arm-kernel@lists.infradead.org; Wed, 17 Apr 2024 09:39:09 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 47314339; Wed, 17 Apr 2024 02:39:34 -0700 (PDT) Received: from e125905.cambridge.arm.com (e125905.cambridge.arm.com [10.1.194.73]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 9881C3F64C; Wed, 17 Apr 2024 02:39:04 -0700 (PDT) From: Beata Michalska To: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, ionela.voinescu@arm.com, vanshikonda@os.amperecomputing.com Cc: sudeep.holla@arm.com, will@kernel.org, catalin.marinas@arm.com, vincent.guittot@linaro.org, sumitg@nvidia.com, yang@os.amperecomputing.com, lihuisong@huawei.com, viresh.kumar@linaro.org Subject: [PATCH v5 3/5] arm64: Provide an AMU-based version of arch_freq_get_on_cpu Date: Wed, 17 Apr 2024 10:38:46 +0100 Message-Id: <20240417093848.1555462-4-beata.michalska@arm.com> In-Reply-To: <20240417093848.1555462-1-beata.michalska@arm.com> References: <20240417093848.1555462-1-beata.michalska@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240417_023907_308422_A16E7570 X-CRM114-Status: GOOD ( 18.68 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org With the Frequency Invariance Engine (FIE) being already wired up with sched tick and making use of relevant (core counter and constant counter) AMU counters, getting the current frequency for a given CPU, can be achieved by utilizing the frequency scale factor which reflects an average CPU frequency for the last tick period length. The solution is partially based on APERF/MPERF implementation of arch_freq_get_on_cpu. Suggested-by: Ionela Voinescu Signed-off-by: Beata Michalska --- arch/arm64/kernel/topology.c | 110 +++++++++++++++++++++++++++++++---- 1 file changed, 100 insertions(+), 10 deletions(-) diff --git a/arch/arm64/kernel/topology.c b/arch/arm64/kernel/topology.c index 3c814a278534..475fdbf3032a 100644 --- a/arch/arm64/kernel/topology.c +++ b/arch/arm64/kernel/topology.c @@ -17,6 +17,7 @@ #include #include #include +#include #include #include @@ -88,18 +89,28 @@ int __init parse_acpi_topology(void) * initialized. */ static DEFINE_PER_CPU_READ_MOSTLY(unsigned long, arch_max_freq_scale) = 1UL << (2 * SCHED_CAPACITY_SHIFT); -static DEFINE_PER_CPU(u64, arch_const_cycles_prev); -static DEFINE_PER_CPU(u64, arch_core_cycles_prev); static cpumask_var_t amu_fie_cpus; +struct amu_cntr_sample { + u64 arch_const_cycles_prev; + u64 arch_core_cycles_prev; + unsigned long last_update; +}; + +static DEFINE_PER_CPU_SHARED_ALIGNED(struct amu_cntr_sample, cpu_amu_samples); + void update_freq_counters_refs(void) { - this_cpu_write(arch_core_cycles_prev, read_corecnt()); - this_cpu_write(arch_const_cycles_prev, read_constcnt()); + struct amu_cntr_sample *amu_sample = this_cpu_ptr(&cpu_amu_samples); + + amu_sample->arch_core_cycles_prev = read_corecnt(); + amu_sample->arch_const_cycles_prev = read_constcnt(); } static inline bool freq_counters_valid(int cpu) { + struct amu_cntr_sample *amu_sample = per_cpu_ptr(&cpu_amu_samples, cpu); + if ((cpu >= nr_cpu_ids) || !cpumask_test_cpu(cpu, cpu_present_mask)) return false; @@ -108,8 +119,8 @@ static inline bool freq_counters_valid(int cpu) return false; } - if (unlikely(!per_cpu(arch_const_cycles_prev, cpu) || - !per_cpu(arch_core_cycles_prev, cpu))) { + if (unlikely(!amu_sample->arch_const_cycles_prev || + !amu_sample->arch_core_cycles_prev)) { pr_debug("CPU%d: cycle counters are not enabled.\n", cpu); return false; } @@ -152,17 +163,22 @@ void freq_inv_set_max_ratio(int cpu, u64 max_rate) static void amu_scale_freq_tick(void) { + struct amu_cntr_sample *amu_sample = this_cpu_ptr(&cpu_amu_samples); u64 prev_core_cnt, prev_const_cnt; u64 core_cnt, const_cnt, scale; - prev_const_cnt = this_cpu_read(arch_const_cycles_prev); - prev_core_cnt = this_cpu_read(arch_core_cycles_prev); + prev_const_cnt = amu_sample->arch_const_cycles_prev; + prev_core_cnt = amu_sample->arch_core_cycles_prev; update_freq_counters_refs(); - const_cnt = this_cpu_read(arch_const_cycles_prev); - core_cnt = this_cpu_read(arch_core_cycles_prev); + const_cnt = amu_sample->arch_const_cycles_prev; + core_cnt = amu_sample->arch_core_cycles_prev; + /* + * This should not happen unless the AMUs have been reset and the + * counter values have not been restored - unlikely + */ if (unlikely(core_cnt <= prev_core_cnt || const_cnt <= prev_const_cnt)) return; @@ -182,6 +198,8 @@ static void amu_scale_freq_tick(void) scale = min_t(unsigned long, scale, SCHED_CAPACITY_SCALE); this_cpu_write(arch_freq_scale, (unsigned long)scale); + + amu_sample->last_update = jiffies; } static struct scale_freq_data amu_sfd = { @@ -189,6 +207,78 @@ static struct scale_freq_data amu_sfd = { .set_freq_scale = amu_scale_freq_tick, }; +static __always_inline bool amu_fie_cpu_supported(unsigned int cpu) +{ + return cpumask_available(amu_fie_cpus) && + cpumask_test_cpu(cpu, amu_fie_cpus); +} + +#define AMU_SAMPLE_EXP_MS 20 + +unsigned int arch_freq_get_on_cpu(int cpu) +{ + struct amu_cntr_sample *amu_sample; + unsigned int start_cpu = cpu; + unsigned long last_update; + unsigned int freq = 0; + u64 scale; + + if (!amu_fie_cpu_supported(cpu) || !arch_scale_freq_ref(cpu)) + return 0; + +retry: + amu_sample = per_cpu_ptr(&cpu_amu_samples, cpu); + + last_update = amu_sample->last_update; + + /* + * For those CPUs that are in full dynticks mode, + * and those that have not seen tick for a while + * try an alternative source for the counters (and thus freq scale), + * if available, for given policy: + * this boils down to identifying an active cpu within the same freq + * domain, if any. + */ + if (!housekeeping_cpu(cpu, HK_TYPE_TICK) || + time_is_before_jiffies(last_update + msecs_to_jiffies(AMU_SAMPLE_EXP_MS))) { + struct cpufreq_policy *policy = cpufreq_cpu_get(cpu); + int ref_cpu = cpu; + + if (!policy) + goto leave; + + if (!policy_is_shared(policy)) { + cpufreq_cpu_put(policy); + goto leave; + } + + do { + ref_cpu = cpumask_next_wrap(ref_cpu, policy->cpus, + start_cpu, false); + + } while (ref_cpu < nr_cpu_ids && idle_cpu(ref_cpu)); + + cpufreq_cpu_put(policy); + + if (ref_cpu >= nr_cpu_ids) + /* No alternative to pull info from */ + goto leave; + + cpu = ref_cpu; + goto retry; + } + /* + * Reversed computation to the one used to determine + * the arch_freq_scale value + * (see amu_scale_freq_tick for details) + */ + scale = arch_scale_freq_capacity(cpu); + freq = scale * arch_scale_freq_ref(cpu); + freq >>= SCHED_CAPACITY_SHIFT; +leave: + return freq; +} + static void amu_fie_setup(const struct cpumask *cpus) { int cpu; From patchwork Wed Apr 17 09:38:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Beata Michalska X-Patchwork-Id: 13633114 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6830EC001CC for ; Wed, 17 Apr 2024 09:39:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=t2Jg1CAAXRjRXQZHTNTAuK8YRF6VxXO2VxgyDHOetsg=; b=14b867kXo3vXoR WhKNgS/k19pt+/w7W7bt1o/hVE5AsuxpCK01f2CzJcsPERfrAoTmbYpXdiN+/k1xxq8YSPN7PE8sJ uMfIqv3o26ZQTD0gad0/sT4itxbCePsNCVbMm1rnInOMlU1H8zjkoxWgCgy4LFSSYbnEuh4i9A618 7sxCzJtPCfpCAQr6m0/2XwCWNHlLTTs+bR3VBpugGMAxNFXVOHYiHIIB7SS7akLj8TbDhuO7XchaG 00z2xCK5w8YVvv8Vh5K0eOO4lpyeLfkQQHB+hu+NHg2JSJhMNF9AvcmGzTvyZNkdgrs6mfBMi9Y8W A1deH4F1DAuNnsdefyqQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rx1l5-0000000FRlm-2cu8; Wed, 17 Apr 2024 09:39:15 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rx1kz-0000000FRj1-15dX for linux-arm-kernel@lists.infradead.org; Wed, 17 Apr 2024 09:39:10 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 55EA7DA7; Wed, 17 Apr 2024 02:39:36 -0700 (PDT) Received: from e125905.cambridge.arm.com (e125905.cambridge.arm.com [10.1.194.73]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id A741E3F64C; Wed, 17 Apr 2024 02:39:06 -0700 (PDT) From: Beata Michalska To: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, ionela.voinescu@arm.com, vanshikonda@os.amperecomputing.com Cc: sudeep.holla@arm.com, will@kernel.org, catalin.marinas@arm.com, vincent.guittot@linaro.org, sumitg@nvidia.com, yang@os.amperecomputing.com, lihuisong@huawei.com, viresh.kumar@linaro.org Subject: [PATCH v5 4/5] arm64: Update AMU-based frequency scale factor on entering idle Date: Wed, 17 Apr 2024 10:38:47 +0100 Message-Id: <20240417093848.1555462-5-beata.michalska@arm.com> In-Reply-To: <20240417093848.1555462-1-beata.michalska@arm.com> References: <20240417093848.1555462-1-beata.michalska@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240417_023909_447618_00E61B6F X-CRM114-Status: GOOD ( 11.99 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Now that the frequency scale factor has been activated for retrieving current frequency on a given CPU, trigger its update upon entering idle. This will, to an extent, allow querying last known frequency in a non-invasive way. It will also improve the frequency scale factor accuracy when a CPU entering idle did not receive a tick for a while. As a consequence, for idle cores, the reported frequency will be the last one observed before entering the idle state. Suggested-by: Vanshidhar Konda Signed-off-by: Beata Michalska --- arch/arm64/kernel/topology.c | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/arch/arm64/kernel/topology.c b/arch/arm64/kernel/topology.c index 475fdbf3032a..3110863ee18c 100644 --- a/arch/arm64/kernel/topology.c +++ b/arch/arm64/kernel/topology.c @@ -213,6 +213,19 @@ static __always_inline bool amu_fie_cpu_supported(unsigned int cpu) cpumask_test_cpu(cpu, amu_fie_cpus); } +void arch_cpu_idle_enter(void) +{ + unsigned int cpu = smp_processor_id(); + + if (!amu_fie_cpu_supported(cpu)) + return; + + /* Kick in AMU update but only if one has not happened already */ + if (housekeeping_cpu(cpu, HK_TYPE_TICK) && + time_is_before_jiffies(per_cpu(cpu_amu_samples.last_update, cpu))) + amu_scale_freq_tick(); +} + #define AMU_SAMPLE_EXP_MS 20 unsigned int arch_freq_get_on_cpu(int cpu) @@ -239,8 +252,8 @@ unsigned int arch_freq_get_on_cpu(int cpu) * this boils down to identifying an active cpu within the same freq * domain, if any. */ - if (!housekeeping_cpu(cpu, HK_TYPE_TICK) || - time_is_before_jiffies(last_update + msecs_to_jiffies(AMU_SAMPLE_EXP_MS))) { + if (!housekeeping_cpu(cpu, HK_TYPE_TICK) || (!idle_cpu(cpu) && + time_is_before_jiffies(last_update + msecs_to_jiffies(AMU_SAMPLE_EXP_MS)))) { struct cpufreq_policy *policy = cpufreq_cpu_get(cpu); int ref_cpu = cpu; From patchwork Wed Apr 17 09:38:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Beata Michalska X-Patchwork-Id: 13633117 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 22C23C4345F for ; Wed, 17 Apr 2024 09:39:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=GmDF+O2fb7sIvJOLY7djBE2JirlA4lxipgS95VGJcRk=; b=P5+4NTYUcInrxF 03hIrcS5MLso4eutoTlOpUiO68HuUi2Jps0WrpDmOdTKrMkM3+Fg5V0/4QIro/RM5j9LDbMtNOkcN s2DBvhmjbvapPTlYvc5HRAypQX+KwqhY18JlhzQsZFYvVZ+KqIOY0jwMxzPuAzGRsDGS1+fsM1LGb NTny7nlu5jpWUyzBKBeFfo/C8vDIw+nlt1YPLm7r41HVJW7xu6uKTE7PAp3P/xAwH2Y1NN/EnVU0h ikOl25zN5XgnqtmR+ehN0tVuuEgp/w7Fo/Ex8jasbRaWw+QXXDVhN2mZCGpntd26LSPgjupUSK6VP icTSJw86myg77o6FrSmQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rx1l6-0000000FRmH-1cTo; Wed, 17 Apr 2024 09:39:16 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rx1l0-0000000FRjn-3t7F for linux-arm-kernel@lists.infradead.org; Wed, 17 Apr 2024 09:39:12 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6D3A21477; Wed, 17 Apr 2024 02:39:38 -0700 (PDT) Received: from e125905.cambridge.arm.com (e125905.cambridge.arm.com [10.1.194.73]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id B5D333F64C; Wed, 17 Apr 2024 02:39:08 -0700 (PDT) From: Beata Michalska To: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, ionela.voinescu@arm.com, vanshikonda@os.amperecomputing.com Cc: sudeep.holla@arm.com, will@kernel.org, catalin.marinas@arm.com, vincent.guittot@linaro.org, sumitg@nvidia.com, yang@os.amperecomputing.com, lihuisong@huawei.com, viresh.kumar@linaro.org Subject: [PATCH v5 5/5] cpufreq: Use arch specific feedback for cpuinfo_cur_freq Date: Wed, 17 Apr 2024 10:38:48 +0100 Message-Id: <20240417093848.1555462-6-beata.michalska@arm.com> In-Reply-To: <20240417093848.1555462-1-beata.michalska@arm.com> References: <20240417093848.1555462-1-beata.michalska@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240417_023911_040033_6C0E0CD8 X-CRM114-Status: UNSURE ( 9.53 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Some architectures provide a way to determine an average frequency over a certain period of time based on available performance monitors (AMU on ARM or APERF/MPERf on x86). With those at hand, enroll arch_freq_get_on_cpu into cpuinfo_cur_freq policy sysfs attribute handler, which is expected to represent the current frequency of a given CPU,as obtained by the hardware. This is the type of feedback that counters do provide. Signed-off-by: Beata Michalska --- drivers/cpufreq/cpufreq.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/drivers/cpufreq/cpufreq.c b/drivers/cpufreq/cpufreq.c index f6f8d7f450e7..89118406ec68 100644 --- a/drivers/cpufreq/cpufreq.c +++ b/drivers/cpufreq/cpufreq.c @@ -793,8 +793,10 @@ store_one(scaling_max_freq, max); static ssize_t show_cpuinfo_cur_freq(struct cpufreq_policy *policy, char *buf) { - unsigned int cur_freq = __cpufreq_get(policy); + unsigned int cur_freq = arch_freq_get_on_cpu(policy->cpu); + if (!cur_freq) + cur_freq = __cpufreq_get(policy); if (cur_freq) return sprintf(buf, "%u\n", cur_freq);