From patchwork Thu Dec 3 14:11:21 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 11948951 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5E0C8C43211 for ; Thu, 3 Dec 2020 14:13:35 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id EA01420705 for ; Thu, 3 Dec 2020 14:13:34 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EA01420705 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=techsingularity.net Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=sAjTlZlQTUtU7Pu3wbwZN5uwzlpH1t1krug9kPmCmfQ=; b=YLT06fiAEwcGmleCnN8yLdoPI aV6WPuv+hg0DdzTg0JeZUi7Tp4N7yvfcNh8DhOjPc6yEABpWpfvsTmRXsfRnFViSJWemANvuVN11Q EJky74nyaa2hwpzXErYKBV+QuN9lNuMYpGlfdhh+853FQZv/3SFWD9COcRENSH2DbqBXlJ5ps3XeO mE5qe7bAo+Ayqq61tlGRnNjdKHNuqlMtoN6j1Fdpr4xz0j2NLCEl0dLPLJWA/hYIPiEtngRht3kTT hYLmOMt7BbzP+r43txwckTkP0+m+LSn4fsRlBg8IqGelqI9cfuGiOYGVrsph2rgVg6sXXYjlfiHb6 lkiHFGulA==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kkpL5-0003ws-M4; Thu, 03 Dec 2020 14:12:07 +0000 Received: from outbound-smtp57.blacknight.com ([46.22.136.241]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kkpKT-0003gn-3m for linux-arm-kernel@lists.infradead.org; Thu, 03 Dec 2020 14:11:37 +0000 Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp57.blacknight.com (Postfix) with ESMTPS id 4F68CFA913 for ; Thu, 3 Dec 2020 14:11:26 +0000 (GMT) Received: (qmail 22879 invoked from network); 3 Dec 2020 14:11:26 -0000 Received: from unknown (HELO stampy.112glenside.lan) (mgorman@techsingularity.net@[84.203.22.4]) by 81.17.254.9 with ESMTPA; 3 Dec 2020 14:11:26 -0000 From: Mel Gorman To: LKML Subject: [PATCH 07/10] sched/fair: Account for the idle cpu/smt search cost Date: Thu, 3 Dec 2020 14:11:21 +0000 Message-Id: <20201203141124.7391-8-mgorman@techsingularity.net> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20201203141124.7391-1-mgorman@techsingularity.net> References: <20201203141124.7391-1-mgorman@techsingularity.net> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201203_091129_494770_4129E28B X-CRM114-Status: GOOD ( 18.50 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Barry Song , Juri Lelli , Vincent Guittot , Peter Ziljstra , Aubrey Li , Ingo Molnar , Mel Gorman , Valentin Schneider , Linux-ARM Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org select_idle_cpu() accounts average search cost for the purposes of conducting a limited proportional search if SIS_PROP is enabled. The issue is that select_idle_cpu() does not account for the cost if a candidate is found and select_idle_smt() is ignored. This patch moves the accounting of avg_cost to cover the cpu/smt search costs. select_idle_core() costs could be accounted for but it has its own throttling mechanism by tracking depending on whether idle cores are expected to exist. This patch is a bisection hazard becuse SIS_PROP and how it balances avg_cost vs avg_idle was probably guided by the fact that avg_cost was not always accounted for. Signed-off-by: Mel Gorman --- kernel/sched/fair.c | 82 +++++++++++++++++++++++++-------------------- 1 file changed, 46 insertions(+), 36 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 1d8f5c4b4936..185fc6e28f8e 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -6006,6 +6006,29 @@ static inline int find_idlest_cpu(struct sched_domain *sd, struct task_struct *p return new_cpu; } +static int sis_search_depth(struct sched_domain *sd, struct sched_domain *this_sd) +{ + u64 avg_cost, avg_idle, span_avg; + int nr = INT_MAX; + + if (sched_feat(SIS_PROP)) { + /* + * Due to large variance we need a large fuzz factor; hackbench in + * particularly is sensitive here. + */ + avg_idle = this_rq()->avg_idle / 512; + avg_cost = this_sd->avg_scan_cost + 1; + + span_avg = sd->span_weight * avg_idle; + if (span_avg > 4*avg_cost) + nr = div_u64(span_avg, avg_cost); + else + nr = 4; + } + + return nr; +} + #ifdef CONFIG_SCHED_SMT DEFINE_STATIC_KEY_FALSE(sched_smt_present); EXPORT_SYMBOL_GPL(sched_smt_present); @@ -6151,35 +6174,11 @@ static inline int select_idle_smt(struct task_struct *p, struct sched_domain *sd * comparing the average scan cost (tracked in sd->avg_scan_cost) against the * average idle time for this rq (as found in rq->avg_idle). */ -static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int target) +static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, + int target, int nr) { struct cpumask *cpus = this_cpu_cpumask_var_ptr(select_idle_mask); - struct sched_domain *this_sd; - u64 avg_cost, avg_idle; - u64 time; - int this = smp_processor_id(); - int cpu, nr = INT_MAX; - - this_sd = rcu_dereference(*this_cpu_ptr(&sd_llc)); - if (!this_sd) - return -1; - - /* - * Due to large variance we need a large fuzz factor; hackbench in - * particularly is sensitive here. - */ - avg_idle = this_rq()->avg_idle / 512; - avg_cost = this_sd->avg_scan_cost + 1; - - if (sched_feat(SIS_PROP)) { - u64 span_avg = sd->span_weight * avg_idle; - if (span_avg > 4*avg_cost) - nr = div_u64(span_avg, avg_cost); - else - nr = 4; - } - - time = cpu_clock(this); + int cpu; cpumask_and(cpus, sched_domain_span(sd), p->cpus_ptr); __cpumask_clear_cpu(target, cpus); @@ -6192,9 +6191,6 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t break; } - time = cpu_clock(this) - time; - update_avg(&this_sd->avg_scan_cost, time); - return cpu; } @@ -6245,9 +6241,10 @@ static inline bool asym_fits_capacity(int task_util, int cpu) */ static int select_idle_sibling(struct task_struct *p, int prev, int target) { - struct sched_domain *sd; + struct sched_domain *sd, *this_sd; unsigned long task_util; - int i, recent_used_cpu; + int i, recent_used_cpu, depth; + u64 time; schedstat_inc(this_rq()->sis_search); @@ -6337,21 +6334,34 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target) if (!sd) return target; + this_sd = rcu_dereference(*this_cpu_ptr(&sd_llc)); + if (!this_sd) + return target; + + depth = sis_search_depth(sd, this_sd); + schedstat_inc(this_rq()->sis_domain_search); i = select_idle_core(p, sd, target); if ((unsigned)i < nr_cpumask_bits) return i; - i = select_idle_cpu(p, sd, target); + time = cpu_clock(smp_processor_id()); + i = select_idle_cpu(p, sd, target, depth); if ((unsigned)i < nr_cpumask_bits) - return i; + goto acct_cost; i = select_idle_smt(p, sd, target); if ((unsigned)i < nr_cpumask_bits) - return i; + goto acct_cost; schedstat_inc(this_rq()->sis_failed); - return target; + i = target; + +acct_cost: + time = cpu_clock(smp_processor_id()) - time; + update_avg(&this_sd->avg_scan_cost, time); + + return i; } /**