From patchwork Thu Jul 6 09:49:48 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dietmar Eggemann X-Patchwork-Id: 9827887 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 2DFAD604D9 for ; Thu, 6 Jul 2017 09:51:03 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1F67728574 for ; Thu, 6 Jul 2017 09:51:03 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 13CD728628; Thu, 6 Jul 2017 09:51:03 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7B9B22861D for ; Thu, 6 Jul 2017 09:51:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752149AbdGFJuy (ORCPT ); Thu, 6 Jul 2017 05:50:54 -0400 Received: from foss.arm.com ([217.140.101.70]:36192 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751059AbdGFJuw (ORCPT ); Thu, 6 Jul 2017 05:50:52 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 3CFF21713; Thu, 6 Jul 2017 02:50:38 -0700 (PDT) Received: from e107985-lin.cambridge.arm.com (e107985-lin.cambridge.arm.com [10.1.210.41]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 3C5AB3F3E1; Thu, 6 Jul 2017 02:50:36 -0700 (PDT) From: Dietmar Eggemann To: linux-kernel@vger.kernel.org Cc: linux-pm@vger.kernel.org, linux@arm.linux.org.uk, Greg Kroah-Hartman , Russell King , Catalin Marinas , Will Deacon , Juri Lelli , Vincent Guittot , Peter Zijlstra , Morten Rasmussen , Viresh Kumar , "Rafael J . Wysocki" Subject: [PATCH v2 10/10] drivers base/arch_topology: inline cpu- and frequency-invariant accounting Date: Thu, 6 Jul 2017 10:49:48 +0100 Message-Id: <20170706094948.8779-11-dietmar.eggemann@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20170706094948.8779-1-dietmar.eggemann@arm.com> References: <20170706094948.8779-1-dietmar.eggemann@arm.com> Sender: linux-pm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP To speed up the cpu- and frequency-invariant accounting of the task scheduler make sure that the CIE (topology_get_cpu_scale()) and FIE (topology_get_freq_scale() get completely inlined into the task scheduler consumer functions (e.g. __update_load_avg_se()). This patch-set changes the interface for CIE and FIE from: drivers/base/arch_topology.c: static DEFINE_PER_CPU(unsigned long, item); unsigned long topology_get_item_scale(...) { return per_cpu(item, cpu) } include/linux/arch_topology.h: unsigned long topology_get_item_scale(...); to: drivers/base/arch_topology.c: DEFINE_PER_CPU(unsigned long, item); include/linux/arch_topology.h: DECLARE_PER_CPU(unsigned long, item); static inline unsigned long topology_get_item_scale(...) { return per_cpu(item, cpu) } An uplift in performance could be detected running the kernel with the following test patch on top (on JUNO R0 (arm64)): @@ -2812,10 +2812,18 @@ accumulate_sum(u64 delta, int cpu, struct sched_avg *sa, unsigned long scale_freq, scale_cpu; u32 contrib = (u32)delta; /* p == 0 -> delta < 1024 */ u64 periods; + u64 t1, t2; + + t1 = sched_clock_cpu(cpu); scale_freq = arch_scale_freq_capacity(NULL, cpu); scale_cpu = arch_scale_cpu_capacity(NULL, cpu); + t2 = sched_clock_cpu(cpu); + + trace_printk("cpu=%d t1=%llu t2=%llu diff=%llu\n", + cpu, t1, t2, t2 - t1); + delta += sa->period_contrib; periods = delta / 1024; /* A period is * 1024us * (~1ms) */ The following test results (3 test runs each) have been obtained by tracing this trace printk (diff=x) for Cortex A-53 (LITTLE) and Cortex A-57 (big) cpus w/ (inline) and w/o (non-inline) this patch. mean max min A-57 inline: 119.6 300 60 96.8 280 60 110.2 660 60 A-57 non-inline: 142.8 460 80 157.6 680 80 153.4 720 80 A-53 inline: 141.6 360 100 118.8 500 100 148.6 380 100 A-53 non-inline: 293 840 120 253.2 840 120 299.6 1060 140 Cc: Greg Kroah-Hartman Cc: Juri Lelli Signed-off-by: Dietmar Eggemann --- drivers/base/arch_topology.c | 14 ++------------ include/linux/arch_topology.h | 15 +++++++++++++-- 2 files changed, 15 insertions(+), 14 deletions(-) diff --git a/drivers/base/arch_topology.c b/drivers/base/arch_topology.c index 63fb3f945d21..b4481cff14bf 100644 --- a/drivers/base/arch_topology.c +++ b/drivers/base/arch_topology.c @@ -22,12 +22,7 @@ #include #include -static DEFINE_PER_CPU(unsigned long, freq_scale) = SCHED_CAPACITY_SCALE; - -unsigned long topology_get_freq_scale(struct sched_domain *sd, int cpu) -{ - return per_cpu(freq_scale, cpu); -} +DEFINE_PER_CPU(unsigned long, freq_scale) = SCHED_CAPACITY_SCALE; void topology_set_freq_scale(struct cpumask *cpus, unsigned long cur_freq, unsigned long max_freq) @@ -43,12 +38,7 @@ void topology_set_freq_scale(struct cpumask *cpus, unsigned long cur_freq, static DEFINE_MUTEX(cpu_scale_mutex); -static DEFINE_PER_CPU(unsigned long, cpu_scale) = SCHED_CAPACITY_SCALE; - -unsigned long topology_get_cpu_scale(struct sched_domain *sd, int cpu) -{ - return per_cpu(cpu_scale, cpu); -} +DEFINE_PER_CPU(unsigned long, cpu_scale) = SCHED_CAPACITY_SCALE; void topology_set_cpu_scale(unsigned int cpu, unsigned long capacity) { diff --git a/include/linux/arch_topology.h b/include/linux/arch_topology.h index 168104d2d2cf..361e85a30151 100644 --- a/include/linux/arch_topology.h +++ b/include/linux/arch_topology.h @@ -11,12 +11,23 @@ void topology_normalize_cpu_scale(void); struct device_node; int topology_parse_cpu_capacity(struct device_node *cpu_node, int cpu); +DECLARE_PER_CPU(unsigned long, cpu_scale); +DECLARE_PER_CPU(unsigned long, freq_scale); + struct sched_domain; -unsigned long topology_get_cpu_scale(struct sched_domain *sd, int cpu); +static inline +unsigned long topology_get_cpu_scale(struct sched_domain *sd, int cpu) +{ + return per_cpu(cpu_scale, cpu); +} void topology_set_cpu_scale(unsigned int cpu, unsigned long capacity); -unsigned long topology_get_freq_scale(struct sched_domain *sd, int cpu); +static inline +unsigned long topology_get_freq_scale(struct sched_domain *sd, int cpu) +{ + return per_cpu(freq_scale, cpu); +} void topology_set_freq_scale(struct cpumask *cpus, unsigned long cur_freq, unsigned long max_freq);