From patchwork Thu Dec 3 14:11:15 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 11948949 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C0FDEC43211 for ; Thu, 3 Dec 2020 14:13:29 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 56F1B20705 for ; Thu, 3 Dec 2020 14:13:29 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 56F1B20705 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=techsingularity.net Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=e625aBYYoF6f0hO0ac6Iil8TVpBUh36qEXUyEFdaUSM=; b=toZOTSHofTkqjxabkS7XMVKLR JExnFpYZ7zVOT03mjXQkGCOxHYuQsIuE4Oh/IYuWApiMnodUrhKK1l6su61k1Vc1REdDdnB2H0TXZ jb52ze5nK06aEczDixIBCuJDG0HOCFe/sbDvLAFFg35bgETMR0Lynn3McpGaCubvsRgnNTrnljtON 8woKMiGVelHY/adtSIkoteqEIhbbBMAnv64+azlpkAsBkElJYc2qREUqGo/+89vqyVxiHGZwyqywj nELoRJQQT0Kc2wzfUL3K827cXyu+FwOCxuDAu4JZIRYSbPENfgvjJiCcZfi93FmaYdcx4edBOz+gM cPtSgSxDw==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kkpKv-0003rh-JZ; Thu, 03 Dec 2020 14:11:57 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kkpKZ-0003ly-Ah for linux-arm-kernel@merlin.infradead.org; Thu, 03 Dec 2020 14:11:35 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=0QZAHnSLXr8NA84NYT7K190k0jIimN150iWEiUbF3ko=; b=Q4bKGaAC9YCAJVE7Eq4w1ImKFV fMCeNq8GhumDpsqmxJFJ1XitkgkuTJdE5oCWIzkuV6cmMl+v3jkYaL84xTo9Q5CVa6WEDcfOUFpzy EEAuUovrGZoTqBhxXx2ZwWkWAJUyrRIHcT8VQgJ1Cf9Za7i9qr5sPCf/rwbMS2q3IcY/bUODtsgAO NVh0HDP03JFYXfjpt5uTG2jbq7BBusJIfw3Q56+D4cxLHPa4mC1BOBtRGnQI6Rqn/I7pVMAmUPhJ6 tFfWmJxlar3yiXJrOk5tMgmb4+fbHE7dyfuEUuQ2eGWcPNwRma9nz4PZOnCeTDXv5aYKTcc5rgQ/O DFOpj+AQ==; Received: from outbound-smtp54.blacknight.com ([46.22.136.238]) by casper.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kkpKT-0005fL-Ue for linux-arm-kernel@lists.infradead.org; Thu, 03 Dec 2020 14:11:34 +0000 Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp54.blacknight.com (Postfix) with ESMTPS id F309EFA8F1 for ; Thu, 3 Dec 2020 14:11:24 +0000 (GMT) Received: (qmail 22661 invoked from network); 3 Dec 2020 14:11:24 -0000 Received: from unknown (HELO stampy.112glenside.lan) (mgorman@techsingularity.net@[84.203.22.4]) by 81.17.254.9 with ESMTPA; 3 Dec 2020 14:11:24 -0000 From: Mel Gorman To: LKML Subject: [PATCH 01/10] sched/fair: Track efficiency of select_idle_sibling Date: Thu, 3 Dec 2020 14:11:15 +0000 Message-Id: <20201203141124.7391-2-mgorman@techsingularity.net> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20201203141124.7391-1-mgorman@techsingularity.net> References: <20201203141124.7391-1-mgorman@techsingularity.net> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201203_141130_292324_8A4D1AE6 X-CRM114-Status: GOOD ( 19.09 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Barry Song , Juri Lelli , Vincent Guittot , Peter Ziljstra , Aubrey Li , Ingo Molnar , Mel Gorman , Valentin Schneider , Linux-ARM Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org select_idle_sibling is an important path that finds a nearby idle CPU on wakeup. As it is examining other CPUs state, it can be expensive in terms of cache usage. This patch tracks the search efficiency if schedstats are enabled. In general, this is only useful for kernel developers but schedstats are typically disabled by default so it is convenient for development and mostly free otherwise. It is not required that this patch be merged with the series but if we are looking at time or search complexity, the stats generate hard data on what the search costs actually are. SIS Search: Number of calls to select_idle_sibling SIS Domain Search: Number of times the domain was searched because the fast path failed. SIS Scanned: Generally the number of runqueues scanned but the fast path counts as 1 regardless of the values for target, prev and recent. SIS Domain Scanned: Number of runqueues scanned during a search of the LLC domain. SIS Failures: Number of SIS calls that failed to find an idle CPU SIS Search Efficiency: A ratio expressed as a percentage of runqueues scanned versus idle CPUs found. A 100% efficiency indicates that the target, prev or recent CPU of a task was idle at wakeup. The lower the efficiency, the more runqueues were scanned before an idle CPU was found. SIS Domain Search Efficiency: Similar, except only for the slower SIS patch. SIS Fast Success Rate: Percentage of SIS that used target, prev or recent CPUs. SIS Success rate: Percentage of scans that found an idle CPU. Signed-off-by: Mel Gorman --- kernel/sched/debug.c | 4 ++++ kernel/sched/fair.c | 14 ++++++++++++++ kernel/sched/sched.h | 6 ++++++ kernel/sched/stats.c | 8 +++++--- 4 files changed, 29 insertions(+), 3 deletions(-) diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c index 2357921580f9..2386cc5e79e5 100644 --- a/kernel/sched/debug.c +++ b/kernel/sched/debug.c @@ -714,6 +714,10 @@ do { \ P(sched_goidle); P(ttwu_count); P(ttwu_local); + P(sis_search); + P(sis_domain_search); + P(sis_scanned); + P(sis_failed); } #undef P diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 98075f9ea9a8..494ba01f3414 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -6081,6 +6081,7 @@ static int select_idle_core(struct task_struct *p, struct sched_domain *sd, int bool idle = true; for_each_cpu(cpu, cpu_smt_mask(core)) { + schedstat_inc(this_rq()->sis_scanned); if (!available_idle_cpu(cpu)) { idle = false; break; @@ -6112,6 +6113,7 @@ static int select_idle_smt(struct task_struct *p, struct sched_domain *sd, int t return -1; for_each_cpu(cpu, cpu_smt_mask(target)) { + schedstat_inc(this_rq()->sis_scanned); if (!cpumask_test_cpu(cpu, p->cpus_ptr) || !cpumask_test_cpu(cpu, sched_domain_span(sd))) continue; @@ -6177,6 +6179,7 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t cpumask_and(cpus, sched_domain_span(sd), p->cpus_ptr); for_each_cpu_wrap(cpu, cpus, target) { + schedstat_inc(this_rq()->sis_scanned); if (!--nr) return -1; if (available_idle_cpu(cpu) || sched_idle_cpu(cpu)) @@ -6240,6 +6243,15 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target) unsigned long task_util; int i, recent_used_cpu; + schedstat_inc(this_rq()->sis_search); + + /* + * Checking if prev, target and recent is treated as one scan. A + * perfect hit on one of those is considered 100% efficiency. + * Further scanning impairs efficiency. + */ + schedstat_inc(this_rq()->sis_scanned); + /* * On asymmetric system, update task utilization because we will check * that the task fits with cpu's capacity. @@ -6315,6 +6327,7 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target) if (!sd) return target; + schedstat_inc(this_rq()->sis_domain_search); i = select_idle_core(p, sd, target); if ((unsigned)i < nr_cpumask_bits) return i; @@ -6327,6 +6340,7 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target) if ((unsigned)i < nr_cpumask_bits) return i; + schedstat_inc(this_rq()->sis_failed); return target; } diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index f5acb6c5ce49..90a62dd9293d 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -1049,6 +1049,12 @@ struct rq { /* try_to_wake_up() stats */ unsigned int ttwu_count; unsigned int ttwu_local; + + /* select_idle_sibling stats */ + unsigned int sis_search; + unsigned int sis_domain_search; + unsigned int sis_scanned; + unsigned int sis_failed; #endif #ifdef CONFIG_CPU_IDLE diff --git a/kernel/sched/stats.c b/kernel/sched/stats.c index 750fb3c67eed..390bfcc3842c 100644 --- a/kernel/sched/stats.c +++ b/kernel/sched/stats.c @@ -10,7 +10,7 @@ * Bump this up when changing the output format or the meaning of an existing * format, so that tools can adapt (or abort) */ -#define SCHEDSTAT_VERSION 15 +#define SCHEDSTAT_VERSION 16 static int show_schedstat(struct seq_file *seq, void *v) { @@ -30,12 +30,14 @@ static int show_schedstat(struct seq_file *seq, void *v) /* runqueue-specific stats */ seq_printf(seq, - "cpu%d %u 0 %u %u %u %u %llu %llu %lu", + "cpu%d %u 0 %u %u %u %u %llu %llu %lu %u %u %u %u", cpu, rq->yld_count, rq->sched_count, rq->sched_goidle, rq->ttwu_count, rq->ttwu_local, rq->rq_cpu_time, - rq->rq_sched_info.run_delay, rq->rq_sched_info.pcount); + rq->rq_sched_info.run_delay, rq->rq_sched_info.pcount, + rq->sis_search, rq->sis_domain_search, + rq->sis_scanned, rq->sis_failed); seq_printf(seq, "\n"); From patchwork Thu Dec 3 14:11:16 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 11948947 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7639FC00130 for ; Thu, 3 Dec 2020 14:13:20 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 0E28420757 for ; Thu, 3 Dec 2020 14:13:20 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0E28420757 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=techsingularity.net Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=bmumf/KAe/x8Od8ggOL6s8VCcCxRUiwN96dD8HTKD80=; b=IJuktPDarP2F7ykUVvyQW4+wf MMfCo7woHu07rz2kYUGM5lmS8buC/AGa7Mti9rTMlQhNT8x4GywE41JoEAFGXvFvo7FnjlwK9/EyX rXzPmYTq0y+xtTDcZruWsQeZDMelhPNnZQG8tiEIxZcH5VJE5bs8y/Se3zDqmkh2kC1SeSV4RteYA cY/FKW7r8OaniWDksNvj3J62Oy6CW5qG/6pTplePwkJdx2d0sUV5n+z8iw+HT2p6L1sYjufVkZXgl 7XjnMS11Rf1HlZ6jUAC8WgbcjAvFz9vqKzqeKJOfL/QOR4NG7ofXpPu7W7P6eWe1pFyiNup3Olane c/VsDfH2A==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kkpKp-0003pI-Au; Thu, 03 Dec 2020 14:11:51 +0000 Received: from outbound-smtp45.blacknight.com ([46.22.136.57]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kkpKS-0003gd-KL for linux-arm-kernel@lists.infradead.org; Thu, 03 Dec 2020 14:11:32 +0000 Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp45.blacknight.com (Postfix) with ESMTPS id 39555FA8FC for ; Thu, 3 Dec 2020 14:11:25 +0000 (GMT) Received: (qmail 22688 invoked from network); 3 Dec 2020 14:11:25 -0000 Received: from unknown (HELO stampy.112glenside.lan) (mgorman@techsingularity.net@[84.203.22.4]) by 81.17.254.9 with ESMTPA; 3 Dec 2020 14:11:24 -0000 From: Mel Gorman To: LKML Subject: [PATCH 02/10] sched/fair: Track efficiency of task recent_used_cpu Date: Thu, 3 Dec 2020 14:11:16 +0000 Message-Id: <20201203141124.7391-3-mgorman@techsingularity.net> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20201203141124.7391-1-mgorman@techsingularity.net> References: <20201203141124.7391-1-mgorman@techsingularity.net> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201203_091128_938031_1D7643F8 X-CRM114-Status: GOOD ( 16.51 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Barry Song , Juri Lelli , Vincent Guittot , Peter Ziljstra , Aubrey Li , Ingo Molnar , Mel Gorman , Valentin Schneider , Linux-ARM Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org This simply tracks the efficiency of the recent_used_cpu. The hit rate of this matters as it can avoid a domain search. Similarly, the miss rate matters because each miss is a penalty to the fast path. It is not required that this patch be merged with the series but if we are looking at the usefulness of p->recent_used_cpu, the stats generate hard data on what the hit rate is. MMTests uses this to generate additional metrics. SIS Recent Used Hit: A recent CPU was eligible and used. Each hit is a domain search avoided. SIS Recent Used Miss: A recent CPU was eligible but unavailable. Each time this is miss, there was a small penalty to the fast path before a domain search happened. SIS Recent Success Rate: A percentage of the number of hits versus the total attempts to use the recent CPU. SIS Recent Attempts: The total number of times the recent CPU was examined. A high number of Recent Attempts with a low Success Rate implies the fast path is being punished severely. This could have been presented as a weighting of hits and misses but calculating an appropriate weight for misses is problematic. Signed-off-by: Mel Gorman --- kernel/sched/debug.c | 2 ++ kernel/sched/fair.c | 23 +++++++++++++---------- kernel/sched/sched.h | 2 ++ kernel/sched/stats.c | 7 ++++--- 4 files changed, 21 insertions(+), 13 deletions(-) diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c index 2386cc5e79e5..8f933a9e8c25 100644 --- a/kernel/sched/debug.c +++ b/kernel/sched/debug.c @@ -718,6 +718,8 @@ do { \ P(sis_domain_search); P(sis_scanned); P(sis_failed); + P(sis_recent_hit); + P(sis_recent_miss); } #undef P diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 494ba01f3414..d9acd55d309b 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -6291,16 +6291,19 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target) recent_used_cpu = p->recent_used_cpu; if (recent_used_cpu != prev && recent_used_cpu != target && - cpus_share_cache(recent_used_cpu, target) && - (available_idle_cpu(recent_used_cpu) || sched_idle_cpu(recent_used_cpu)) && - cpumask_test_cpu(p->recent_used_cpu, p->cpus_ptr) && - asym_fits_capacity(task_util, recent_used_cpu)) { - /* - * Replace recent_used_cpu with prev as it is a potential - * candidate for the next wake: - */ - p->recent_used_cpu = prev; - return recent_used_cpu; + cpus_share_cache(recent_used_cpu, target)) { + if ((available_idle_cpu(recent_used_cpu) || sched_idle_cpu(recent_used_cpu)) && + cpumask_test_cpu(p->recent_used_cpu, p->cpus_ptr) && + asym_fits_capacity(task_util, recent_used_cpu)) { + /* + * Replace recent_used_cpu with prev as it is a potential + * candidate for the next wake: + */ + p->recent_used_cpu = prev; + schedstat_inc(this_rq()->sis_recent_hit); + return recent_used_cpu; + } + schedstat_inc(this_rq()->sis_recent_miss); } /* diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 90a62dd9293d..6a6578c4c24b 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -1055,6 +1055,8 @@ struct rq { unsigned int sis_domain_search; unsigned int sis_scanned; unsigned int sis_failed; + unsigned int sis_recent_hit; + unsigned int sis_recent_miss; #endif #ifdef CONFIG_CPU_IDLE diff --git a/kernel/sched/stats.c b/kernel/sched/stats.c index 390bfcc3842c..402fab75aa14 100644 --- a/kernel/sched/stats.c +++ b/kernel/sched/stats.c @@ -10,7 +10,7 @@ * Bump this up when changing the output format or the meaning of an existing * format, so that tools can adapt (or abort) */ -#define SCHEDSTAT_VERSION 16 +#define SCHEDSTAT_VERSION 17 static int show_schedstat(struct seq_file *seq, void *v) { @@ -30,14 +30,15 @@ static int show_schedstat(struct seq_file *seq, void *v) /* runqueue-specific stats */ seq_printf(seq, - "cpu%d %u 0 %u %u %u %u %llu %llu %lu %u %u %u %u", + "cpu%d %u 0 %u %u %u %u %llu %llu %lu %u %u %u %u %u %u", cpu, rq->yld_count, rq->sched_count, rq->sched_goidle, rq->ttwu_count, rq->ttwu_local, rq->rq_cpu_time, rq->rq_sched_info.run_delay, rq->rq_sched_info.pcount, rq->sis_search, rq->sis_domain_search, - rq->sis_scanned, rq->sis_failed); + rq->sis_scanned, rq->sis_failed, + rq->sis_recent_hit, rq->sis_recent_miss); seq_printf(seq, "\n"); From patchwork Thu Dec 3 14:11:17 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 11948935 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 89904C001B1 for ; Thu, 3 Dec 2020 14:12:59 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1309A2079A for ; Thu, 3 Dec 2020 14:12:59 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1309A2079A Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=techsingularity.net Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=UbZl70KBpvr03GnDXFrGlI/xLigs9BxtZOkScr4MJN0=; b=y1YZtee1tI116WjkWZOfhwhhk Kd0ir4Jo9C3ECQIJOUojQW1v40lnzIdFGrM7oqEC/nM+WXM7rGsPETwpAi+46OzIO80Mwp0AvVo9R aMb7hI1YTwdj/oMe+9Ohn/WcecTMKJy3dSpWNm8Wil28fbeq80GaTm8LlwouzgR5FIT5e1Xx7W3pE Yh8i38/6eXx4iHU8RUiAemDwq7aeFHKH6fNH9ROwMr92dq1vHYvHjTop8J+AlxxWoW2LNHiWj7KM1 98INAzO89p6pQmXkS9bexKffU8y/3ZJtsD8l49o883QjZEd4C7M9TSUL317e5k+BYI56bWfRWAVVN T/JTCCYOg==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kkpKX-0003kv-Cs; Thu, 03 Dec 2020 14:11:33 +0000 Received: from outbound-smtp11.blacknight.com ([46.22.139.106]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kkpKS-0003gc-G6 for linux-arm-kernel@lists.infradead.org; Thu, 03 Dec 2020 14:11:30 +0000 Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp11.blacknight.com (Postfix) with ESMTPS id 693971C3734 for ; Thu, 3 Dec 2020 14:11:25 +0000 (GMT) Received: (qmail 22720 invoked from network); 3 Dec 2020 14:11:25 -0000 Received: from unknown (HELO stampy.112glenside.lan) (mgorman@techsingularity.net@[84.203.22.4]) by 81.17.254.9 with ESMTPA; 3 Dec 2020 14:11:25 -0000 From: Mel Gorman To: LKML Subject: [PATCH 03/10] sched/fair: Remove SIS_AVG_CPU Date: Thu, 3 Dec 2020 14:11:17 +0000 Message-Id: <20201203141124.7391-4-mgorman@techsingularity.net> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20201203141124.7391-1-mgorman@techsingularity.net> References: <20201203141124.7391-1-mgorman@techsingularity.net> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201203_091128_658273_00C8E098 X-CRM114-Status: GOOD ( 11.34 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Barry Song , Juri Lelli , Vincent Guittot , Peter Ziljstra , Aubrey Li , Ingo Molnar , Mel Gorman , Valentin Schneider , Linux-ARM Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org SIS_AVG_CPU was introduced as a means of avoiding a search when the average search cost indicated that the search would likely fail. It was a blunt instrument and disabled by 4c77b18cf8b7 ("sched/fair: Make select_idle_cpu() more aggressive") and later replaced with a proportional search depth by 1ad3aaf3fcd2 ("sched/core: Implement new approach to scale select_idle_cpu()"). While there are corner cases where SIS_AVG_CPU is better, it has now been disabled for almost three years. As the intent of SIS_PROP is to reduce the time complexity of select_idle_cpu(), lets drop SIS_AVG_CPU and focus on SIS_PROP as a throttling mechanism. Signed-off-by: Mel Gorman --- kernel/sched/fair.c | 3 --- kernel/sched/features.h | 1 - 2 files changed, 4 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index d9acd55d309b..fc48cc99b03d 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -6163,9 +6163,6 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t avg_idle = this_rq()->avg_idle / 512; avg_cost = this_sd->avg_scan_cost + 1; - if (sched_feat(SIS_AVG_CPU) && avg_idle < avg_cost) - return -1; - if (sched_feat(SIS_PROP)) { u64 span_avg = sd->span_weight * avg_idle; if (span_avg > 4*avg_cost) diff --git a/kernel/sched/features.h b/kernel/sched/features.h index 68d369cba9e4..e875eabb6600 100644 --- a/kernel/sched/features.h +++ b/kernel/sched/features.h @@ -54,7 +54,6 @@ SCHED_FEAT(TTWU_QUEUE, true) /* * When doing wakeups, attempt to limit superfluous scans of the LLC domain. */ -SCHED_FEAT(SIS_AVG_CPU, false) SCHED_FEAT(SIS_PROP, true) /* From patchwork Thu Dec 3 14:11:18 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 11948937 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D50A4C433C1 for ; Thu, 3 Dec 2020 14:13:01 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 6B63720798 for ; Thu, 3 Dec 2020 14:13:01 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6B63720798 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=techsingularity.net Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=oubLeose4pM/gNn7LXm0VDJd/XJx/Ct2QWRDrr4FwRE=; b=PdJPslWcwsfdP4rm7eTa1N3/W NgdgMNL0us+Uns2kL2S1Ypafm6lckgYhf4u5w4GoNkpEec+xuq699PAGswbqcRdCymq3TfP5PZivx DVN0Zb1i72w2Of06qM+kR2be9F2GG9EInXcIXPPF5ZSP8LZYWgCNF8d0m6Qot3ziQc/3jqYhJioSg W8w4OzF1wCWNTCtkVsMiEQhijyODpLHISJwLSIzsW1bRemVx6bqnzn/tdTDWeVvj9LiKRVSotqnX0 snb8EuodRdpzmwPATVLFXgYT0OagQpIJAkRLPaFYeH9NTTsOSKcNIeBmyiIA2dB6m7c5SnkRFZfsl v+6qBoXPw==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kkpKb-0003mM-RL; Thu, 03 Dec 2020 14:11:37 +0000 Received: from outbound-smtp19.blacknight.com ([46.22.139.246]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kkpKS-0003gg-Ks for linux-arm-kernel@lists.infradead.org; Thu, 03 Dec 2020 14:11:30 +0000 Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp19.blacknight.com (Postfix) with ESMTPS id 984681C373A for ; Thu, 3 Dec 2020 14:11:25 +0000 (GMT) Received: (qmail 22748 invoked from network); 3 Dec 2020 14:11:25 -0000 Received: from unknown (HELO stampy.112glenside.lan) (mgorman@techsingularity.net@[84.203.22.4]) by 81.17.254.9 with ESMTPA; 3 Dec 2020 14:11:25 -0000 From: Mel Gorman To: LKML Subject: [PATCH 04/10] sched/fair: Return an idle cpu if one is found after a failed search for an idle core Date: Thu, 3 Dec 2020 14:11:18 +0000 Message-Id: <20201203141124.7391-5-mgorman@techsingularity.net> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20201203141124.7391-1-mgorman@techsingularity.net> References: <20201203141124.7391-1-mgorman@techsingularity.net> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201203_091128_827318_08395EDD X-CRM114-Status: GOOD ( 15.98 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Barry Song , Juri Lelli , Vincent Guittot , Peter Ziljstra , Aubrey Li , Ingo Molnar , Mel Gorman , Valentin Schneider , Linux-ARM Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org select_idle_core is called when SMT is active and there is likely a free core available. It may find idle CPUs but this information is simply discarded and the scan starts over again with select_idle_cpu. This patch caches information on idle CPUs found during the search for a core and uses one if no core is found. This is a tradeoff. There may be a slight impact when utilisation is low and an idle core can be found quickly. It provides improvements as the number of busy CPUs approaches 50% of the domain size when SMT is enabled. With tbench on a 2-socket CascadeLake machine, 80 logical CPUs, HT enabled 5.10.0-rc6 5.10.0-rc6 schedstat idlecandidate Hmean 1 500.06 ( 0.00%) 505.67 * 1.12%* Hmean 2 975.90 ( 0.00%) 974.06 * -0.19%* Hmean 4 1902.95 ( 0.00%) 1904.43 * 0.08%* Hmean 8 3761.73 ( 0.00%) 3721.02 * -1.08%* Hmean 16 6713.93 ( 0.00%) 6769.17 * 0.82%* Hmean 32 10435.31 ( 0.00%) 10312.58 * -1.18%* Hmean 64 12325.51 ( 0.00%) 13792.01 * 11.90%* Hmean 128 21225.21 ( 0.00%) 20963.44 * -1.23%* Hmean 256 20532.83 ( 0.00%) 20335.62 * -0.96%* Hmean 320 20334.81 ( 0.00%) 20147.25 * -0.92%* In this particular test, the cost/benefit is marginal except for 64 which was a point where the machine was over 50% busy but not fully utilised. Signed-off-by: Mel Gorman --- kernel/sched/fair.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index fc48cc99b03d..845bc0cd9158 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -6066,6 +6066,7 @@ void __update_idle_core(struct rq *rq) */ static int select_idle_core(struct task_struct *p, struct sched_domain *sd, int target) { + int idle_candidate = -1; struct cpumask *cpus = this_cpu_cpumask_var_ptr(select_idle_mask); int core, cpu; @@ -6084,7 +6085,13 @@ static int select_idle_core(struct task_struct *p, struct sched_domain *sd, int schedstat_inc(this_rq()->sis_scanned); if (!available_idle_cpu(cpu)) { idle = false; - break; + if (idle_candidate != -1) + break; + } + + if (idle_candidate == -1 && + cpumask_test_cpu(cpu, p->cpus_ptr)) { + idle_candidate = cpu; } } @@ -6099,7 +6106,7 @@ static int select_idle_core(struct task_struct *p, struct sched_domain *sd, int */ set_idle_cores(target, 0); - return -1; + return idle_candidate; } /* From patchwork Thu Dec 3 14:11:19 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 11948943 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1B1A4C43211 for ; Thu, 3 Dec 2020 14:13:11 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 990E020705 for ; Thu, 3 Dec 2020 14:13:10 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 990E020705 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=techsingularity.net Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=MferWYc3FTx6fWLSjEObDVYxJoLWVD55cM9O57KlNlk=; b=XrVsI3WG8kqpNgoKbNmVNFSTf GbOvNGXcwIHadW1dEsv3U0YcuIdoWiKLRTt511cz68FuBbd+oR97/szWWHYQ6X3/XlFipHUzSgOYz u3F0uuH9lIMY8mXZ/sq1AOhdEHUm7dD1R3GirC/kBAc1+76mvdcfR+qz2eus36pOyQdUKB8iXT8Ps iMWnr9v1e+g0LTm6v1khhtAYxg7zGqUriuM9Wxm36DJN1Ap9pE/6B/k6r+eduJMvewGYaAClMEiT0 /OYh0XIdkLtZG6Y8z68RYke7JjnFr8WnqGxQ8L43/fHFh78qITkuRFWNQQxezFzgPv89anTQvrmEG mfrLHMnAg==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kkpKi-0003np-On; Thu, 03 Dec 2020 14:11:44 +0000 Received: from outbound-smtp11.blacknight.com ([46.22.139.106]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kkpKS-0003gh-MI for linux-arm-kernel@lists.infradead.org; Thu, 03 Dec 2020 14:11:31 +0000 Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp11.blacknight.com (Postfix) with ESMTPS id D84BD1C3737 for ; Thu, 3 Dec 2020 14:11:25 +0000 (GMT) Received: (qmail 22768 invoked from network); 3 Dec 2020 14:11:25 -0000 Received: from unknown (HELO stampy.112glenside.lan) (mgorman@techsingularity.net@[84.203.22.4]) by 81.17.254.9 with ESMTPA; 3 Dec 2020 14:11:25 -0000 From: Mel Gorman To: LKML Subject: [PATCH 05/10] sched/fair: Do not replace recent_used_cpu with the new target Date: Thu, 3 Dec 2020 14:11:19 +0000 Message-Id: <20201203141124.7391-6-mgorman@techsingularity.net> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20201203141124.7391-1-mgorman@techsingularity.net> References: <20201203141124.7391-1-mgorman@techsingularity.net> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201203_091128_881918_4DA4A16C X-CRM114-Status: GOOD ( 15.52 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Barry Song , Juri Lelli , Vincent Guittot , Peter Ziljstra , Aubrey Li , Ingo Molnar , Mel Gorman , Valentin Schneider , Linux-ARM Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org After select_idle_sibling, p->recent_used_cpu is set to the new target. However on the next wakeup, prev will be the same as recent_used_cpu unless the load balancer has moved the task since the last wakeup. It still works, but is less efficient than it can be after all the changes that went in since that reduce unnecessary migrations, load balancer changes etc. This patch preserves recent_used_cpu for longer. With tbench on a 2-socket CascadeLake machine, 80 logical CPUs, HT enabled 5.10.0-rc6 5.10.0-rc6 idlecandidate-v1r10 altrecent-v1r10 Hmean 1 505.67 ( 0.00%) 501.34 * -0.86%* Hmean 2 974.06 ( 0.00%) 981.39 * 0.75%* Hmean 4 1904.43 ( 0.00%) 1926.13 * 1.14%* Hmean 8 3721.02 ( 0.00%) 3799.86 * 2.12%* Hmean 16 6769.17 ( 0.00%) 6938.40 * 2.50%* Hmean 32 10312.58 ( 0.00%) 10632.11 * 3.10%* Hmean 64 13792.01 ( 0.00%) 13670.17 * -0.88%* Hmean 128 20963.44 ( 0.00%) 21456.33 * 2.35%* Hmean 256 20335.62 ( 0.00%) 21070.24 * 3.61%* Hmean 320 20147.25 ( 0.00%) 20624.92 * 2.37%* The benefit is marginal, the main impact is on how it affects p->recent_used_cpu and whether a domain search happens. From the schedstats patches and schedstat enabled Ops SIS Search 5653107942.00 5726545742.00 Ops SIS Domain Search 3365067916.00 3319768543.00 Ops SIS Scanned 112173512543.00 99194352541.00 Ops SIS Domain Scanned 109885472517.00 96787575342.00 Ops SIS Failures 2923185114.00 2950166441.00 Ops SIS Recent Used Hit 56547.00 118064916.00 Ops SIS Recent Used Miss 1590899250.00 354942791.00 Ops SIS Recent Attempts 1590955797.00 473007707.00 Ops SIS Search Efficiency 5.04 5.77 Ops SIS Domain Search Eff 3.06 3.43 Ops SIS Fast Success Rate 40.47 42.03 Ops SIS Success Rate 48.29 48.48 Ops SIS Recent Success Rate 0.00 24.96 (First interesting point is the ridiculous number of times runqueues are enabled -- almost 97 billion times over the course of 40 minutes) Note "Recent Used Hit" is over 2000 times more likely to succeed. The failure rate also increases by quite a lot but the cost is marginal even if the "Fast Success Rate" only increases by 2% overall. What cannot be observed from these stats is where the biggest impact as these stats cover low utilisation to over saturation. If graphed over time, the graphs show that the sched domain is only scanned at negligible rates until the machine is fully busy. With low utilisation, the "Fast Success Rate" is almost 100% until the machine is fully busy. For 320 clients, the success rate is close to 0% which is unsurprising. Signed-off-by: Mel Gorman --- kernel/sched/fair.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 845bc0cd9158..68dd9cd62fbd 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -6293,6 +6293,7 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target) /* Check a recently used CPU as a potential idle candidate: */ recent_used_cpu = p->recent_used_cpu; + p->recent_used_cpu = prev; if (recent_used_cpu != prev && recent_used_cpu != target && cpus_share_cache(recent_used_cpu, target)) { @@ -6789,9 +6790,6 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags) } else if (wake_flags & WF_TTWU) { /* XXX always ? */ /* Fast path */ new_cpu = select_idle_sibling(p, prev_cpu, new_cpu); - - if (want_affine) - current->recent_used_cpu = cpu; } rcu_read_unlock(); From patchwork Thu Dec 3 14:11:20 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 11948939 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DD9E8C43211 for ; Thu, 3 Dec 2020 14:13:07 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5830520705 for ; Thu, 3 Dec 2020 14:13:07 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5830520705 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=techsingularity.net Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=XNJ/s/qkRi7uSsDgx2vLFUvDRrchYME5etCA8Cy3/kg=; b=auiiXYUChRIh8hEwp8d2OwyDe XtrCdHr62VS9D4JzfyMvvvT7yHVfETvTNQztkPeiiNbE8t3B6nUBomHz8Iiig12YAGiyGrZxsn5q7 TYfjhUT8B/auzguE9plRyrwh2hdilqPAxRB0ZUSlXxQNEg5DEWSsGSMMqj8eHXNrr0bLVkR4RmmvM YPgJpLZzZEfhS29H1Rjb110zCF3pzx73AoWuwVouF5fLU70+DrIbX/xcM6PiqnC+N4yQ0x5yQIUyg QLBsUyJAQLQUZx08BSb8aH2/sfwb1K6HMBb4sJBTcpDk6KPNabd6c53ipmUQ7ly0tcQN7Uxo0y/HV bhBWoKs6w==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kkpKf-0003n7-Hz; Thu, 03 Dec 2020 14:11:41 +0000 Received: from [46.22.139.220] (helo=outbound-smtp37.blacknight.com) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kkpKT-0003gi-14 for linux-arm-kernel@lists.infradead.org; Thu, 03 Dec 2020 14:11:31 +0000 Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp37.blacknight.com (Postfix) with ESMTPS id 1FCA41780 for ; Thu, 3 Dec 2020 14:11:26 +0000 (GMT) Received: (qmail 22842 invoked from network); 3 Dec 2020 14:11:25 -0000 Received: from unknown (HELO stampy.112glenside.lan) (mgorman@techsingularity.net@[84.203.22.4]) by 81.17.254.9 with ESMTPA; 3 Dec 2020 14:11:25 -0000 From: Mel Gorman To: LKML Subject: [PATCH 06/10] sched/fair: Clear the target CPU from the cpumask of CPUs searched Date: Thu, 3 Dec 2020 14:11:20 +0000 Message-Id: <20201203141124.7391-7-mgorman@techsingularity.net> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20201203141124.7391-1-mgorman@techsingularity.net> References: <20201203141124.7391-1-mgorman@techsingularity.net> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201203_091129_189212_1A314EEF X-CRM114-Status: GOOD ( 10.98 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Barry Song , Juri Lelli , Vincent Guittot , Peter Ziljstra , Aubrey Li , Ingo Molnar , Mel Gorman , Valentin Schneider , Linux-ARM Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The target CPU is definitely not idle in both select_idle_core and select_idle_cpu. For select_idle_core(), the SMT is potentially checked unnecessarily as the core is definitely not idle if the target is busy. For select_idle_cpu(), the first CPU checked is simply a waste. Signed-off-by: Mel Gorman --- kernel/sched/fair.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 68dd9cd62fbd..1d8f5c4b4936 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -6077,6 +6077,7 @@ static int select_idle_core(struct task_struct *p, struct sched_domain *sd, int return -1; cpumask_and(cpus, sched_domain_span(sd), p->cpus_ptr); + __cpumask_clear_cpu(target, cpus); for_each_cpu_wrap(core, cpus, target) { bool idle = true; @@ -6181,6 +6182,7 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t time = cpu_clock(this); cpumask_and(cpus, sched_domain_span(sd), p->cpus_ptr); + __cpumask_clear_cpu(target, cpus); for_each_cpu_wrap(cpu, cpus, target) { schedstat_inc(this_rq()->sis_scanned); From patchwork Thu Dec 3 14:11:21 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 11948951 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5E0C8C43211 for ; Thu, 3 Dec 2020 14:13:35 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id EA01420705 for ; Thu, 3 Dec 2020 14:13:34 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EA01420705 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=techsingularity.net Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=sAjTlZlQTUtU7Pu3wbwZN5uwzlpH1t1krug9kPmCmfQ=; b=YLT06fiAEwcGmleCnN8yLdoPI aV6WPuv+hg0DdzTg0JeZUi7Tp4N7yvfcNh8DhOjPc6yEABpWpfvsTmRXsfRnFViSJWemANvuVN11Q EJky74nyaa2hwpzXErYKBV+QuN9lNuMYpGlfdhh+853FQZv/3SFWD9COcRENSH2DbqBXlJ5ps3XeO mE5qe7bAo+Ayqq61tlGRnNjdKHNuqlMtoN6j1Fdpr4xz0j2NLCEl0dLPLJWA/hYIPiEtngRht3kTT hYLmOMt7BbzP+r43txwckTkP0+m+LSn4fsRlBg8IqGelqI9cfuGiOYGVrsph2rgVg6sXXYjlfiHb6 lkiHFGulA==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kkpL5-0003ws-M4; Thu, 03 Dec 2020 14:12:07 +0000 Received: from outbound-smtp57.blacknight.com ([46.22.136.241]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kkpKT-0003gn-3m for linux-arm-kernel@lists.infradead.org; Thu, 03 Dec 2020 14:11:37 +0000 Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp57.blacknight.com (Postfix) with ESMTPS id 4F68CFA913 for ; Thu, 3 Dec 2020 14:11:26 +0000 (GMT) Received: (qmail 22879 invoked from network); 3 Dec 2020 14:11:26 -0000 Received: from unknown (HELO stampy.112glenside.lan) (mgorman@techsingularity.net@[84.203.22.4]) by 81.17.254.9 with ESMTPA; 3 Dec 2020 14:11:26 -0000 From: Mel Gorman To: LKML Subject: [PATCH 07/10] sched/fair: Account for the idle cpu/smt search cost Date: Thu, 3 Dec 2020 14:11:21 +0000 Message-Id: <20201203141124.7391-8-mgorman@techsingularity.net> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20201203141124.7391-1-mgorman@techsingularity.net> References: <20201203141124.7391-1-mgorman@techsingularity.net> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201203_091129_494770_4129E28B X-CRM114-Status: GOOD ( 18.50 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Barry Song , Juri Lelli , Vincent Guittot , Peter Ziljstra , Aubrey Li , Ingo Molnar , Mel Gorman , Valentin Schneider , Linux-ARM Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org select_idle_cpu() accounts average search cost for the purposes of conducting a limited proportional search if SIS_PROP is enabled. The issue is that select_idle_cpu() does not account for the cost if a candidate is found and select_idle_smt() is ignored. This patch moves the accounting of avg_cost to cover the cpu/smt search costs. select_idle_core() costs could be accounted for but it has its own throttling mechanism by tracking depending on whether idle cores are expected to exist. This patch is a bisection hazard becuse SIS_PROP and how it balances avg_cost vs avg_idle was probably guided by the fact that avg_cost was not always accounted for. Signed-off-by: Mel Gorman --- kernel/sched/fair.c | 82 +++++++++++++++++++++++++-------------------- 1 file changed, 46 insertions(+), 36 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 1d8f5c4b4936..185fc6e28f8e 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -6006,6 +6006,29 @@ static inline int find_idlest_cpu(struct sched_domain *sd, struct task_struct *p return new_cpu; } +static int sis_search_depth(struct sched_domain *sd, struct sched_domain *this_sd) +{ + u64 avg_cost, avg_idle, span_avg; + int nr = INT_MAX; + + if (sched_feat(SIS_PROP)) { + /* + * Due to large variance we need a large fuzz factor; hackbench in + * particularly is sensitive here. + */ + avg_idle = this_rq()->avg_idle / 512; + avg_cost = this_sd->avg_scan_cost + 1; + + span_avg = sd->span_weight * avg_idle; + if (span_avg > 4*avg_cost) + nr = div_u64(span_avg, avg_cost); + else + nr = 4; + } + + return nr; +} + #ifdef CONFIG_SCHED_SMT DEFINE_STATIC_KEY_FALSE(sched_smt_present); EXPORT_SYMBOL_GPL(sched_smt_present); @@ -6151,35 +6174,11 @@ static inline int select_idle_smt(struct task_struct *p, struct sched_domain *sd * comparing the average scan cost (tracked in sd->avg_scan_cost) against the * average idle time for this rq (as found in rq->avg_idle). */ -static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int target) +static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, + int target, int nr) { struct cpumask *cpus = this_cpu_cpumask_var_ptr(select_idle_mask); - struct sched_domain *this_sd; - u64 avg_cost, avg_idle; - u64 time; - int this = smp_processor_id(); - int cpu, nr = INT_MAX; - - this_sd = rcu_dereference(*this_cpu_ptr(&sd_llc)); - if (!this_sd) - return -1; - - /* - * Due to large variance we need a large fuzz factor; hackbench in - * particularly is sensitive here. - */ - avg_idle = this_rq()->avg_idle / 512; - avg_cost = this_sd->avg_scan_cost + 1; - - if (sched_feat(SIS_PROP)) { - u64 span_avg = sd->span_weight * avg_idle; - if (span_avg > 4*avg_cost) - nr = div_u64(span_avg, avg_cost); - else - nr = 4; - } - - time = cpu_clock(this); + int cpu; cpumask_and(cpus, sched_domain_span(sd), p->cpus_ptr); __cpumask_clear_cpu(target, cpus); @@ -6192,9 +6191,6 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t break; } - time = cpu_clock(this) - time; - update_avg(&this_sd->avg_scan_cost, time); - return cpu; } @@ -6245,9 +6241,10 @@ static inline bool asym_fits_capacity(int task_util, int cpu) */ static int select_idle_sibling(struct task_struct *p, int prev, int target) { - struct sched_domain *sd; + struct sched_domain *sd, *this_sd; unsigned long task_util; - int i, recent_used_cpu; + int i, recent_used_cpu, depth; + u64 time; schedstat_inc(this_rq()->sis_search); @@ -6337,21 +6334,34 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target) if (!sd) return target; + this_sd = rcu_dereference(*this_cpu_ptr(&sd_llc)); + if (!this_sd) + return target; + + depth = sis_search_depth(sd, this_sd); + schedstat_inc(this_rq()->sis_domain_search); i = select_idle_core(p, sd, target); if ((unsigned)i < nr_cpumask_bits) return i; - i = select_idle_cpu(p, sd, target); + time = cpu_clock(smp_processor_id()); + i = select_idle_cpu(p, sd, target, depth); if ((unsigned)i < nr_cpumask_bits) - return i; + goto acct_cost; i = select_idle_smt(p, sd, target); if ((unsigned)i < nr_cpumask_bits) - return i; + goto acct_cost; schedstat_inc(this_rq()->sis_failed); - return target; + i = target; + +acct_cost: + time = cpu_clock(smp_processor_id()) - time; + update_avg(&this_sd->avg_scan_cost, time); + + return i; } /** From patchwork Thu Dec 3 14:11:22 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 11948945 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 20BF8C43211 for ; Thu, 3 Dec 2020 14:13:19 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id ACF9020705 for ; Thu, 3 Dec 2020 14:13:18 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org ACF9020705 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=techsingularity.net Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=1ZLr1ObO9PsTVOb8N1rWJi1otzv8+Ys/DWH+hSVarzU=; b=2Bv3DQYRbjML3Xe0zXqRBgNs/ typtbaNQQZE5O8Nz1m9NT23BDYjv9/fvOZ2d10NW+f2WoBiP+YgDcz54v4lKmz2jfTN/N/J8wNoA4 uAPJvlOGrlIXTElXr74NIog0AAvFmqem3CuCTkqufZ9UFzvyBougc8NwloB+8Qgih1X0RKRTYHAoK nnsVQoNe9Wt2AaDcCin0JKCZft8Zo6dI759nZOqY/T6En7kFokBKFGSj0K/6ZJz4OKHUzhH8hqnhp NZLgJ+44bEBgPGn7pzq4jwfPCWdo4cwePXg0spLahM8r/KL0+B91xIkUepDzo0C8aEBza5PobbYqm +JbLyz1mw==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kkpKt-0003qq-8f; Thu, 03 Dec 2020 14:11:55 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kkpKZ-0003lv-Ab for linux-arm-kernel@merlin.infradead.org; Thu, 03 Dec 2020 14:11:35 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=RKWjwt0ValWwD5pq+efQIOTaeFEdeNs/1sZsRbm/UUA=; b=m0yx1iMxnj1NRCUFsNmNfmoY1a Tl6HwZOMmgDfUHdSTc/rtz4QkLkLOo5+2uPEFh31z2pNwET8UrcS+47+oowPIj8EhaHj+3BysSnrn Jm8Yc704qxtQ4Rse6/tudewoWZYMPlT4TvBeIeV5qsLP1vM+8ctWK8pPfaLGmmgTfzLRKdKzqF6AN tF7hQ+klTFThSV6Q8jphHuYPm41FeBqGIUO5DW+XB5+hviLhjpBVUoYuLOcp9nzNKefig+c6DgwdG hOa2PYfZ0cSg6OWya77kDEs5YSHRsSH7YROLWT4Sw8DqN7y+jfskaU1IUn/dxBXPK3fMW+OH2c86N 4fG/R4nw==; Received: from outbound-smtp31.blacknight.com ([81.17.249.62]) by casper.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kkpKU-0005fv-MU for linux-arm-kernel@lists.infradead.org; Thu, 03 Dec 2020 14:11:33 +0000 Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp31.blacknight.com (Postfix) with ESMTPS id 8D989C0B10 for ; Thu, 3 Dec 2020 14:11:26 +0000 (GMT) Received: (qmail 22909 invoked from network); 3 Dec 2020 14:11:26 -0000 Received: from unknown (HELO stampy.112glenside.lan) (mgorman@techsingularity.net@[84.203.22.4]) by 81.17.254.9 with ESMTPA; 3 Dec 2020 14:11:26 -0000 From: Mel Gorman To: LKML Subject: [PATCH 08/10] sched/fair: Reintroduce SIS_AVG_CPU but in the context of SIS_PROP to reduce search depth Date: Thu, 3 Dec 2020 14:11:22 +0000 Message-Id: <20201203141124.7391-9-mgorman@techsingularity.net> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20201203141124.7391-1-mgorman@techsingularity.net> References: <20201203141124.7391-1-mgorman@techsingularity.net> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201203_141131_038599_DEBBC09E X-CRM114-Status: UNSURE ( 9.66 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Barry Song , Juri Lelli , Vincent Guittot , Peter Ziljstra , Aubrey Li , Ingo Molnar , Mel Gorman , Valentin Schneider , Linux-ARM Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Subject says it all but no supporting data at this time. This might help the hackbench case in isolation or throw other workloads under the bus. Final version will have proper data. Signed-off-by: Mel Gorman --- kernel/sched/fair.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 185fc6e28f8e..33ce65b67381 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -6024,6 +6024,14 @@ static int sis_search_depth(struct sched_domain *sd, struct sched_domain *this_s nr = div_u64(span_avg, avg_cost); else nr = 4; + + /* + * Throttle the depth search futher if average idle time is + * below the average cost. This is primarily to deal with + * the saturated case where searches are likely to fail. + */ + if (avg_idle < avg_cost) + nr >>= 1; } return nr; From patchwork Thu Dec 3 14:11:23 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 11948941 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2756DC433C1 for ; Thu, 3 Dec 2020 14:13:10 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id AF7F420705 for ; Thu, 3 Dec 2020 14:13:09 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AF7F420705 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=techsingularity.net Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=5/2Mv0DOJ6X+ZMzqdF+QqBeuQPL546REf+hfbNH1MGM=; b=y50ZDq369Curbsjnz7/Nshoyo bYzgQrpSfF7nFOhGnh/otgGTPtie2qbRw65nc8TKzjrG2J5X0UBXsrspo7LCWzxMFRq4mmwEmzNIt l5wqh2Lye1SLbI3sYcu/8866Kb5YwRLd5sngWul0AhrqGS8Pck39AvvRyjJb9ts0LPXy8117PFq1L VhHjotZTdCz9UYgq8fCKjhoTjPKdnJCGz9+XqhvVboV8GHLfA4S8NERvWLjgTQGHBXvoPv8IJnk7P /lJwBNeiEB1g94Sz4FMQjzHMz0QdAyDNt6k5wYfuGJqnULIN7TMhEwri2RYDVkARQjX2UyktMXSB7 ha7dHP/Cw==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kkpKm-0003oc-Gv; Thu, 03 Dec 2020 14:11:48 +0000 Received: from outbound-smtp57.blacknight.com ([46.22.136.241]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kkpKS-0003gp-W7 for linux-arm-kernel@lists.infradead.org; Thu, 03 Dec 2020 14:11:32 +0000 Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp57.blacknight.com (Postfix) with ESMTPS id C3D5AFA91B for ; Thu, 3 Dec 2020 14:11:26 +0000 (GMT) Received: (qmail 22934 invoked from network); 3 Dec 2020 14:11:26 -0000 Received: from unknown (HELO stampy.112glenside.lan) (mgorman@techsingularity.net@[84.203.22.4]) by 81.17.254.9 with ESMTPA; 3 Dec 2020 14:11:26 -0000 From: Mel Gorman To: LKML Subject: [PATCH 09/10] sched/fair: Limit the search for an idle core Date: Thu, 3 Dec 2020 14:11:23 +0000 Message-Id: <20201203141124.7391-10-mgorman@techsingularity.net> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20201203141124.7391-1-mgorman@techsingularity.net> References: <20201203141124.7391-1-mgorman@techsingularity.net> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201203_091129_159608_DE732998 X-CRM114-Status: GOOD ( 15.13 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Barry Song , Juri Lelli , Vincent Guittot , Peter Ziljstra , Aubrey Li , Ingo Molnar , Mel Gorman , Valentin Schneider , Linux-ARM Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Note: This is a bad idea, it's for illustration only to show how the search space can be filtered at each stage. Searching an idle_cpu_mask would be a potential option. select_idle_core() would be left alone as it has its own throttling mechanism select_idle_core() may search a full domain for an idle core even if idle CPUs exist result in an excessive search. This patch partially limits the search for an idle core similar to select_idle_cpu() once an idle candidate is found. Note that this patch can *increase* the number of runqueues considered. Any searching done by select_idle_core() is duplicated by select_idle_cpu() if an idle candidate is not found. If there is an idle CPU then aborting select_idle_core() can have a negative impact. This is addressed in the next patch. Signed-off-by: Mel Gorman --- kernel/sched/fair.c | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 33ce65b67381..cd95daf9f53e 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -6095,7 +6095,8 @@ void __update_idle_core(struct rq *rq) * there are no idle cores left in the system; tracked through * sd_llc->shared->has_idle_cores and enabled through update_idle_core() above. */ -static int select_idle_core(struct task_struct *p, struct sched_domain *sd, int target) +static int select_idle_core(struct task_struct *p, struct sched_domain *sd, + int target, int nr) { int idle_candidate = -1; struct cpumask *cpus = this_cpu_cpumask_var_ptr(select_idle_mask); @@ -6115,6 +6116,11 @@ static int select_idle_core(struct task_struct *p, struct sched_domain *sd, int for_each_cpu(cpu, cpu_smt_mask(core)) { schedstat_inc(this_rq()->sis_scanned); + + /* Apply limits if there is an idle candidate */ + if (idle_candidate != -1) + nr--; + if (!available_idle_cpu(cpu)) { idle = false; if (idle_candidate != -1) @@ -6130,6 +6136,9 @@ static int select_idle_core(struct task_struct *p, struct sched_domain *sd, int if (idle) return core; + if (!nr) + break; + cpumask_andnot(cpus, cpus, cpu_smt_mask(core)); } @@ -6165,7 +6174,8 @@ static int select_idle_smt(struct task_struct *p, struct sched_domain *sd, int t #else /* CONFIG_SCHED_SMT */ -static inline int select_idle_core(struct task_struct *p, struct sched_domain *sd, int target) +static inline int select_idle_core(struct task_struct *p, struct sched_domain *sd, + int target, int nr) { return -1; } @@ -6349,7 +6359,7 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target) depth = sis_search_depth(sd, this_sd); schedstat_inc(this_rq()->sis_domain_search); - i = select_idle_core(p, sd, target); + i = select_idle_core(p, sd, target, depth); if ((unsigned)i < nr_cpumask_bits) return i; From patchwork Thu Dec 3 14:20:11 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 11948959 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_SANE_1 autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E665FC433C1 for ; Thu, 3 Dec 2020 14:21:30 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 732B120793 for ; Thu, 3 Dec 2020 14:21:30 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 732B120793 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=techsingularity.net Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References:Message-ID: Subject:To:From:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=FR6fxXCP3xnjoSlitPqnk0jXy1sy307R5uZPo5/Py0U=; b=1/7+DHPibwo6l1Tr3XsmHLLrY hpMusmeNmPjyEx09oPRfKdZ41xmkkE2Y5FvmefRzoWCv+dpAQDfAoVjtJo2R7TwviPxdbZo/CrolW RQkdY48ohh2ZPz9L5+vzyvZwXyrHFT5oRlxStIHTXUuUIvs0tqDHpIoLbb/rtP+hYjTJtyRfYZENQ NOyrKrr4hl40WNHx9gif1dpgWa1GSlBwxy7v4dbcAh+LpUIJYMxtd7izmcUF9nmUnHUj5ZPStbc/Q gijo6vxuia38pBP8mZXjTDPQCEHrm9+uACyi5DSw2wIvYO6Aa/oJrcsn8cee2bzQ/qp2lo5fg6xtY PsagQc5ag==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kkpT0-0007Td-Oh; Thu, 03 Dec 2020 14:20:18 +0000 Received: from outbound-smtp49.blacknight.com ([46.22.136.233]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kkpSx-0007Rq-5U for linux-arm-kernel@lists.infradead.org; Thu, 03 Dec 2020 14:20:16 +0000 Received: from mail.blacknight.com (pemlinmail05.blacknight.ie [81.17.254.26]) by outbound-smtp49.blacknight.com (Postfix) with ESMTPS id 7577EFA8F1 for ; Thu, 3 Dec 2020 14:20:14 +0000 (GMT) Received: (qmail 27425 invoked from network); 3 Dec 2020 14:20:14 -0000 Received: from unknown (HELO techsingularity.net) (mgorman@techsingularity.net@[84.203.22.4]) by 81.17.254.9 with ESMTPSA (AES256-SHA encrypted, authenticated); 3 Dec 2020 14:20:13 -0000 Date: Thu, 3 Dec 2020 14:20:11 +0000 From: Mel Gorman To: LKML Subject: [PATCH 10/10] sched/fair: Avoid revisiting CPUs multiple times during select_idle_sibling Message-ID: <20201203142011.GW3371@techsingularity.net> References: <20201203141124.7391-1-mgorman@techsingularity.net> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20201203141124.7391-1-mgorman@techsingularity.net> User-Agent: Mutt/1.10.1 (2018-07-13) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201203_092015_389368_40944DEB X-CRM114-Status: GOOD ( 15.80 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Barry Song , Juri Lelli , Vincent Guittot , Peter Ziljstra , Aubrey Li , Ingo Molnar , Valentin Schneider , Linux-ARM Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Note: While this is done in the context of select_idle_core(), I would not expect it to be done like this. The intent is to illustrate how idle_cpu_mask could be filtered before select_idle_cpus() scans the rest of a domain or a wider scan was done across a cluster. select_idle_core() potentially searches a number of CPUs for idle candidates before select_idle_cpu() clears the mask and revisits the same CPUs. This patch moves the initialisation of select_idle_mask to the top-level and reuses the same mask across both select_idle_core and select_idle_cpu. select_idle_smt() is left alone as the cost of checking one SMT sibling is marginal relative to calling __clear_cpumask_cpu() for evey CPU visited by select_idle_core(). Signed-off-by: Mel Gorman --- kernel/sched/fair.c | 29 ++++++++++++++++------------- 1 file changed, 16 insertions(+), 13 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index cd95daf9f53e..af2e108c20c0 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -6096,10 +6096,9 @@ void __update_idle_core(struct rq *rq) * sd_llc->shared->has_idle_cores and enabled through update_idle_core() above. */ static int select_idle_core(struct task_struct *p, struct sched_domain *sd, - int target, int nr) + int target, int nr, struct cpumask *cpus) { int idle_candidate = -1; - struct cpumask *cpus = this_cpu_cpumask_var_ptr(select_idle_mask); int core, cpu; if (!static_branch_likely(&sched_smt_present)) @@ -6108,9 +6107,6 @@ static int select_idle_core(struct task_struct *p, struct sched_domain *sd, if (!test_idle_cores(target, false)) return -1; - cpumask_and(cpus, sched_domain_span(sd), p->cpus_ptr); - __cpumask_clear_cpu(target, cpus); - for_each_cpu_wrap(core, cpus, target) { bool idle = true; @@ -6175,7 +6171,7 @@ static int select_idle_smt(struct task_struct *p, struct sched_domain *sd, int t #else /* CONFIG_SCHED_SMT */ static inline int select_idle_core(struct task_struct *p, struct sched_domain *sd, - int target, int nr) + int target, int nr, struct cpumask *cpus) { return -1; } @@ -6193,14 +6189,10 @@ static inline int select_idle_smt(struct task_struct *p, struct sched_domain *sd * average idle time for this rq (as found in rq->avg_idle). */ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, - int target, int nr) + int target, int nr, struct cpumask *cpus) { - struct cpumask *cpus = this_cpu_cpumask_var_ptr(select_idle_mask); int cpu; - cpumask_and(cpus, sched_domain_span(sd), p->cpus_ptr); - __cpumask_clear_cpu(target, cpus); - for_each_cpu_wrap(cpu, cpus, target) { schedstat_inc(this_rq()->sis_scanned); if (!--nr) @@ -6260,6 +6252,7 @@ static inline bool asym_fits_capacity(int task_util, int cpu) static int select_idle_sibling(struct task_struct *p, int prev, int target) { struct sched_domain *sd, *this_sd; + struct cpumask *cpus_visited; unsigned long task_util; int i, recent_used_cpu, depth; u64 time; @@ -6358,13 +6351,23 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target) depth = sis_search_depth(sd, this_sd); + /* + * Init the select_idle_mask. select_idle_core() will mask + * out the CPUs that have already been limited to limit the + * search in select_idle_cpu(). Further clearing is not + * done as select_idle_smt checks only one CPU. + */ + cpus_visited = this_cpu_cpumask_var_ptr(select_idle_mask); + cpumask_and(cpus_visited, sched_domain_span(sd), p->cpus_ptr); + __cpumask_clear_cpu(target, cpus_visited); + schedstat_inc(this_rq()->sis_domain_search); - i = select_idle_core(p, sd, target, depth); + i = select_idle_core(p, sd, target, depth, cpus_visited); if ((unsigned)i < nr_cpumask_bits) return i; time = cpu_clock(smp_processor_id()); - i = select_idle_cpu(p, sd, target, depth); + i = select_idle_cpu(p, sd, target, depth, cpus_visited); if ((unsigned)i < nr_cpumask_bits) goto acct_cost;