From patchwork Tue Sep 15 10:31:58 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 11776017 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4D8E56CA for ; Tue, 15 Sep 2020 10:44:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1E8A921D91 for ; Tue, 15 Sep 2020 10:44:39 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="FoIo/T2b" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726441AbgIOKo3 (ORCPT ); Tue, 15 Sep 2020 06:44:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55682 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726390AbgIOKoH (ORCPT ); Tue, 15 Sep 2020 06:44:07 -0400 Received: from merlin.infradead.org (merlin.infradead.org [IPv6:2001:8b0:10b:1231::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CCAE8C06178C; Tue, 15 Sep 2020 03:44:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=merlin.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=ppSQUOpE02c8Naaz+T4DUk8II1mP6aR5xHXHK1TOwZg=; b=FoIo/T2bPOlpyEqDB81v+CHi3d vkcM6PkwqA5k3WYoZ1qPUwpPk3ohTrs+rs3AkEWfvJZSTiJIYbT90GANjfPwx0zUujxd+sd/FJY+l CCly9MLsbCsHp8lHTByYGh4Wo+KDFVSjjr8/eBf+GzZ4TghJWeAx5XavyYAfAxD+UERUOEEr5OQpn cdQPN7seqyeK/QShtvo5a2cBeYwMTTq5t7A9oh2Qooi9uoJlpJAnxaQr5AnquzHAUdpV5CtkBeX1+ lBVusKyD280P+zROmq+pqRTXRzyRjbAjKFsAlMDUtS8TuFlgI0C7+3cvR/kC6WSlMdG07OB8fGWmp +G6sK31g==; Received: from j217100.upc-j.chello.nl ([24.132.217.100] helo=noisy.programming.kicks-ass.net) by merlin.infradead.org with esmtpsa (Exim 4.92.3 #3 (Red Hat Linux)) id 1kI8R6-0007lG-0q; Tue, 15 Sep 2020 10:43:44 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id B52FD3075F1; Tue, 15 Sep 2020 12:43:40 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 90069214EDD49; Tue, 15 Sep 2020 12:43:40 +0200 (CEST) Message-ID: <20200915103806.280265587@infradead.org> User-Agent: quilt/0.66 Date: Tue, 15 Sep 2020 12:31:58 +0200 From: Peter Zijlstra To: rjw@rjwysocki.net, bp@alien8.de Cc: x86@kernel.org, tony.luck@intel.com, lenb@kernel.org, daniel.lezcano@linaro.org, linux-kernel@vger.kernel.org, linux-acpi@vger.kernel.org, linux-pm@vger.kernel.org, ulf.hansson@linaro.org, paulmck@kernel.org, tglx@linutronix.de, naresh.kamboju@linaro.org, peterz@infradead.org Subject: [RFC][PATCH 1/4] acpi: Use CPUIDLE_FLAG_TIMER_STOP References: <20200915103157.345404192@infradead.org> MIME-Version: 1.0 Sender: linux-pm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org Make acpi_processor_idle use the common broadcast code, there's no reason not to. This also removes some RCU usage after rcu_idle_enter(). Signed-off-by: Peter Zijlstra (Intel) Acked-by: Rafael J. Wysocki Reported-by: Borislav Petkov Tested-by: Borislav Petkov Reported-by: Guenter Roeck Signed-off-by: Rafael J. Wysocki --- drivers/acpi/processor_idle.c | 49 +++++++++++++----------------------------- 1 file changed, 16 insertions(+), 33 deletions(-) --- a/drivers/acpi/processor_idle.c +++ b/drivers/acpi/processor_idle.c @@ -161,18 +161,10 @@ static void lapic_timer_propagate_broadc } /* Power(C) State timer broadcast control */ -static void lapic_timer_state_broadcast(struct acpi_processor *pr, - struct acpi_processor_cx *cx, - int broadcast) -{ - int state = cx - pr->power.states; - - if (state >= pr->power.timer_broadcast_on_state) { - if (broadcast) - tick_broadcast_enter(); - else - tick_broadcast_exit(); - } +static bool lapic_timer_needs_broadcast(struct acpi_processor *pr, + struct acpi_processor_cx *cx) +{ + return cx - pr->power.states >= pr->power.timer_broadcast_on_state; } #else @@ -180,9 +172,9 @@ static void lapic_timer_state_broadcast( static void lapic_timer_check_state(int state, struct acpi_processor *pr, struct acpi_processor_cx *cstate) { } static void lapic_timer_propagate_broadcast(struct acpi_processor *pr) { } -static void lapic_timer_state_broadcast(struct acpi_processor *pr, - struct acpi_processor_cx *cx, - int broadcast) + +static bool lapic_timer_needs_broadcast(struct acpi_processor *pr, + struct acpi_processor_cx *cx) { } @@ -568,21 +560,13 @@ static DEFINE_RAW_SPINLOCK(c3_lock); * acpi_idle_enter_bm - enters C3 with proper BM handling * @pr: Target processor * @cx: Target state context - * @timer_bc: Whether or not to change timer mode to broadcast */ static void acpi_idle_enter_bm(struct acpi_processor *pr, - struct acpi_processor_cx *cx, bool timer_bc) + struct acpi_processor_cx *cx) { acpi_unlazy_tlb(smp_processor_id()); /* - * Must be done before busmaster disable as we might need to - * access HPET ! - */ - if (timer_bc) - lapic_timer_state_broadcast(pr, cx, 1); - - /* * disable bus master * bm_check implies we need ARB_DIS * bm_control implies whether we can do ARB_DIS @@ -609,9 +593,6 @@ static void acpi_idle_enter_bm(struct ac c3_cpu_count--; raw_spin_unlock(&c3_lock); } - - if (timer_bc) - lapic_timer_state_broadcast(pr, cx, 0); } static int acpi_idle_enter(struct cpuidle_device *dev, @@ -630,7 +611,7 @@ static int acpi_idle_enter(struct cpuidl cx = per_cpu(acpi_cstate[index], dev->cpu); } else if (cx->type == ACPI_STATE_C3 && pr->flags.bm_check) { if (cx->bm_sts_skip || !acpi_idle_bm_check()) { - acpi_idle_enter_bm(pr, cx, true); + acpi_idle_enter_bm(pr, cx); return index; } else if (drv->safe_state_index >= 0) { index = drv->safe_state_index; @@ -642,15 +623,11 @@ static int acpi_idle_enter(struct cpuidl } } - lapic_timer_state_broadcast(pr, cx, 1); - if (cx->type == ACPI_STATE_C3) ACPI_FLUSH_CPU_CACHE(); acpi_idle_do_entry(cx); - lapic_timer_state_broadcast(pr, cx, 0); - return index; } @@ -666,7 +643,7 @@ static int acpi_idle_enter_s2idle(struct return 0; if (pr->flags.bm_check) { - acpi_idle_enter_bm(pr, cx, false); + acpi_idle_enter_bm(pr, cx); return 0; } else { ACPI_FLUSH_CPU_CACHE(); @@ -682,6 +659,7 @@ static int acpi_processor_setup_cpuidle_ { int i, count = ACPI_IDLE_STATE_START; struct acpi_processor_cx *cx; + struct cpuidle_state *state; if (max_cstate == 0) max_cstate = 1; @@ -694,6 +672,11 @@ static int acpi_processor_setup_cpuidle_ per_cpu(acpi_cstate[count], dev->cpu) = cx; + if (lapic_timer_needs_broadcast(pr, cx)) { + state = &acpi_idle_driver.states[count]; + state->flags |= CPUIDLE_FLAG_TIMER_STOP; + } + count++; if (count == CPUIDLE_STATE_MAX) break; From patchwork Tue Sep 15 10:31:59 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 11776039 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 11712618 for ; Tue, 15 Sep 2020 10:46:26 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E5D5721D7F for ; Tue, 15 Sep 2020 10:46:25 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="bpANDfAq" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726466AbgIOKqB (ORCPT ); Tue, 15 Sep 2020 06:46:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55652 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726384AbgIOKn5 (ORCPT ); Tue, 15 Sep 2020 06:43:57 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DCC2AC061788; Tue, 15 Sep 2020 03:43:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=mRb05QRhEgX5DGrRuAZaR2rGnerDrceJein4gsxe/Oc=; b=bpANDfAqLOkvapnR4e8RYeRGLJ K14vHjTl8QL0w6Xcm3ZFmf4Tdafh1Wc9Mo7zsHtdzLbO1fGYwsSNmy0XpgBbx3PDD5SDdv4EoyC6U Q9mMIZIfHIBTTGsB0KdkWx1DN7TUa5VRolyCJI5xWwZ5QFSLcS0S9yRyS7gk3uohejGjJvukkNWWt QyXSNJWvKj+WIyQ3jWKEIG3rgQy477DzRYEAMLw9TSJCCiYxWWGEcG1A4Z02Pzz4CUtDYz/qUyqry NAovZHIMbwOcby9iY3WKAVOGWKNtWAViAZBrjT6qXOfMTcpvBWkz6H3BPsrkNWRRm46m2shVRGfgg NJti/h/w==; Received: from j217100.upc-j.chello.nl ([24.132.217.100] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.92.3 #3 (Red Hat Linux)) id 1kI8R3-0006uk-Jk; Tue, 15 Sep 2020 10:43:41 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id B452E30705A; Tue, 15 Sep 2020 12:43:40 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 9249A214EDD4D; Tue, 15 Sep 2020 12:43:40 +0200 (CEST) Message-ID: <20200915103806.348668768@infradead.org> User-Agent: quilt/0.66 Date: Tue, 15 Sep 2020 12:31:59 +0200 From: Peter Zijlstra To: rjw@rjwysocki.net, bp@alien8.de Cc: x86@kernel.org, tony.luck@intel.com, lenb@kernel.org, daniel.lezcano@linaro.org, linux-kernel@vger.kernel.org, linux-acpi@vger.kernel.org, linux-pm@vger.kernel.org, ulf.hansson@linaro.org, paulmck@kernel.org, tglx@linutronix.de, naresh.kamboju@linaro.org, peterz@infradead.org Subject: [RFC][PATCH 2/4] acpi: Use CPUIDLE_FLAG_TLB_FLUSHED References: <20200915103157.345404192@infradead.org> MIME-Version: 1.0 Sender: linux-pm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org Make acpi_processor_idle() use the generic TLB flushing code. This again removes RCU usage after rcu_idle_enter(). (XXX make every C3 invalidate TLBs, not just C3-BM) Signed-off-by: Peter Zijlstra (Intel) --- arch/ia64/include/asm/acpi.h | 2 -- arch/x86/include/asm/acpi.h | 2 -- drivers/acpi/processor_idle.c | 10 +++++----- 3 files changed, 5 insertions(+), 9 deletions(-) --- a/arch/ia64/include/asm/acpi.h +++ b/arch/ia64/include/asm/acpi.h @@ -74,8 +74,6 @@ static inline void arch_acpi_set_pdc_bit buf[2] |= ACPI_PDC_EST_CAPABILITY_SMP; } -#define acpi_unlazy_tlb(x) - #ifdef CONFIG_ACPI_NUMA extern cpumask_t early_cpu_possible_map; #define for_each_possible_early_cpu(cpu) \ --- a/arch/x86/include/asm/acpi.h +++ b/arch/x86/include/asm/acpi.h @@ -159,8 +159,6 @@ static inline u64 x86_default_get_root_p extern int x86_acpi_numa_init(void); #endif /* CONFIG_ACPI_NUMA */ -#define acpi_unlazy_tlb(x) leave_mm(x) - #ifdef CONFIG_ACPI_APEI static inline pgprot_t arch_apei_get_mem_attribute(phys_addr_t addr) { --- a/drivers/acpi/processor_idle.c +++ b/drivers/acpi/processor_idle.c @@ -565,8 +565,6 @@ static DEFINE_RAW_SPINLOCK(c3_lock); static void acpi_idle_enter_bm(struct acpi_processor *pr, struct acpi_processor_cx *cx) { - acpi_unlazy_tlb(smp_processor_id()); - /* * disable bus master * bm_check implies we need ARB_DIS @@ -666,6 +664,7 @@ static int acpi_processor_setup_cpuidle_ max_cstate = 1; for (i = 1; i < ACPI_PROCESSOR_MAX_POWER && i <= max_cstate; i++) { + state = &acpi_idle_driver.states[count]; cx = &pr->power.states[i]; if (!cx->valid) @@ -673,10 +672,11 @@ static int acpi_processor_setup_cpuidle_ per_cpu(acpi_cstate[count], dev->cpu) = cx; - if (lapic_timer_needs_broadcast(pr, cx)) { - state = &acpi_idle_driver.states[count]; + if (lapic_timer_needs_broadcast(pr, cx)) state->flags |= CPUIDLE_FLAG_TIMER_STOP; - } + + if (cx->type == ACPI_STATE_C3) + state->flags |= CPUIDLE_FLAG_TLB_FLUSHED; count++; if (count == CPUIDLE_STATE_MAX) From patchwork Tue Sep 15 10:32:00 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 11776007 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EB6446CA for ; Tue, 15 Sep 2020 10:44:30 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A7D7220B1F for ; Tue, 15 Sep 2020 10:44:30 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="wZb27JSi" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726437AbgIOKo0 (ORCPT ); Tue, 15 Sep 2020 06:44:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55680 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726145AbgIOKoG (ORCPT ); Tue, 15 Sep 2020 06:44:06 -0400 Received: from merlin.infradead.org (merlin.infradead.org [IPv6:2001:8b0:10b:1231::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CCA91C06178B; Tue, 15 Sep 2020 03:44:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=merlin.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=k8OJ3LBTy7ZYlXue2CjZpOH2bHs7RL6MvRpNpTjIcbE=; b=wZb27JSiyu5tgLC8O7nVMw8HLw l4iBFnzeCisEHqyRBlc3fsbYf4gTe+bBIPAOtAD2lfoTevdEAIEKU6TmeAgMSHyfDZhQJ8xBn796w LxaimyshQMlMgIOUMAZCE8pPMC9nIUwtvYnw53r55n3hofLvFNeZUb4gK4tAo2aplsNXrm528X22/ fd1qGF5yG/m4TLA9jCGFxgoPjp/ntni+9sOItFF9U1OCpi4KJwJcNTRTQ8Ktb7bayLjt0uSxn6hDA 9CyJIBDfTBYdpiDc9MXjyK5eOE+Z7C6SlLHHf6ENkhTu8DHLuO55ce6y3CsUkqZTO+Hj4gp1KkXst z75EGaPw==; Received: from j217100.upc-j.chello.nl ([24.132.217.100] helo=noisy.programming.kicks-ass.net) by merlin.infradead.org with esmtpsa (Exim 4.92.3 #3 (Red Hat Linux)) id 1kI8R6-0007mK-MV; Tue, 15 Sep 2020 10:43:45 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id B36CA3060F2; Tue, 15 Sep 2020 12:43:40 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 97E1B214EDD4F; Tue, 15 Sep 2020 12:43:40 +0200 (CEST) Message-ID: <20200915103806.411374792@infradead.org> User-Agent: quilt/0.66 Date: Tue, 15 Sep 2020 12:32:00 +0200 From: Peter Zijlstra To: rjw@rjwysocki.net, bp@alien8.de Cc: x86@kernel.org, tony.luck@intel.com, lenb@kernel.org, daniel.lezcano@linaro.org, linux-kernel@vger.kernel.org, linux-acpi@vger.kernel.org, linux-pm@vger.kernel.org, ulf.hansson@linaro.org, paulmck@kernel.org, tglx@linutronix.de, naresh.kamboju@linaro.org, peterz@infradead.org Subject: [RFC][PATCH 3/4] cpuidle: Allow cpuidle drivers to take over RCU-idle References: <20200915103157.345404192@infradead.org> MIME-Version: 1.0 Sender: linux-pm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org Some drivers have to do significant work, some of which relies on RCU still being active. Instead of using RCU_NONIDLE in the drivers and flipping RCU back on, allow drivers to take over RCU-idle duty. Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Ulf Hansson --- drivers/cpuidle/cpuidle.c | 15 ++++++++++----- include/linux/cpuidle.h | 1 + 2 files changed, 11 insertions(+), 5 deletions(-) --- a/drivers/cpuidle/cpuidle.c +++ b/drivers/cpuidle/cpuidle.c @@ -138,6 +138,7 @@ static void enter_s2idle_proper(struct c struct cpuidle_device *dev, int index) { ktime_t time_start, time_end; + struct cpuidle_state *target_state = &drv->states[index]; time_start = ns_to_ktime(local_clock()); @@ -153,8 +154,9 @@ static void enter_s2idle_proper(struct c * suspended is generally unsafe. */ stop_critical_timings(); - rcu_idle_enter(); - drv->states[index].enter_s2idle(dev, drv, index); + if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE)) + rcu_idle_enter(); + target_state->enter_s2idle(dev, drv, index); if (WARN_ON_ONCE(!irqs_disabled())) local_irq_disable(); /* @@ -162,7 +164,8 @@ static void enter_s2idle_proper(struct c * first CPU executing it calls functions containing RCU read-side * critical sections, so tell RCU about that. */ - rcu_idle_exit(); + if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE)) + rcu_idle_exit(); tick_unfreeze(); start_critical_timings(); @@ -239,9 +242,11 @@ int cpuidle_enter_state(struct cpuidle_d time_start = ns_to_ktime(local_clock()); stop_critical_timings(); - rcu_idle_enter(); + if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE)) + rcu_idle_enter(); entered_state = target_state->enter(dev, drv, index); - rcu_idle_exit(); + if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE)) + rcu_idle_exit(); start_critical_timings(); sched_clock_idle_wakeup_event(); --- a/include/linux/cpuidle.h +++ b/include/linux/cpuidle.h @@ -82,6 +82,7 @@ struct cpuidle_state { #define CPUIDLE_FLAG_UNUSABLE BIT(3) /* avoid using this state */ #define CPUIDLE_FLAG_OFF BIT(4) /* disable this state by default */ #define CPUIDLE_FLAG_TLB_FLUSHED BIT(5) /* idle-state flushes TLBs */ +#define CPUIDLE_FLAG_RCU_IDLE BIT(6) /* idle-state takes care of RCU */ struct cpuidle_device_kobj; struct cpuidle_state_kobj; From patchwork Tue Sep 15 10:32:01 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 11776031 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 043D76CA for ; Tue, 15 Sep 2020 10:45:50 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D85F32078D for ; Tue, 15 Sep 2020 10:45:49 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="PIzaub4o" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726370AbgIOKpq (ORCPT ); Tue, 15 Sep 2020 06:45:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55654 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726428AbgIOKn5 (ORCPT ); Tue, 15 Sep 2020 06:43:57 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E56E0C06178A; Tue, 15 Sep 2020 03:43:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=ww+8+f29UThP3VzHThfXkwodAndD/WG4c0ho1R/aaxE=; b=PIzaub4oY69we/Pk0FNrQiLcQ5 HUJ4xb9pQVaharBMU9Xqnu88a3EoI1ghgwgdAW9ybzpL4TUszGcxFw9jMnJh2B/1nCuobc9BwFV90 rQ7T6dZNimgR687YH7M88a5C2DEn/x10eVRjVYla81bM+rmNtvksvAAC3QT8tLuM7qj8V0r7fX3rk VT/0IUvqzF22iWsSCrm5Cq2CO1BE9DvfeE6j6gArN2Bq9ZzWIrGRhBcklKbx581Q+gAmbrfyJ5br4 mUMkp4O8HusizMKoH6UGUA0sHt3xqsw3FLNM6OW/6rfYilFj3ofiwlG2w/YgjAW4Rb31EUTFL87zk fWkhrYWg==; Received: from j217100.upc-j.chello.nl ([24.132.217.100] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.92.3 #3 (Red Hat Linux)) id 1kI8R5-0006v7-7n; Tue, 15 Sep 2020 10:43:43 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id AE4E43059C6; Tue, 15 Sep 2020 12:43:40 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 9B736214EDD4E; Tue, 15 Sep 2020 12:43:40 +0200 (CEST) Message-ID: <20200915103806.479637218@infradead.org> User-Agent: quilt/0.66 Date: Tue, 15 Sep 2020 12:32:01 +0200 From: Peter Zijlstra To: rjw@rjwysocki.net, bp@alien8.de Cc: x86@kernel.org, tony.luck@intel.com, lenb@kernel.org, daniel.lezcano@linaro.org, linux-kernel@vger.kernel.org, linux-acpi@vger.kernel.org, linux-pm@vger.kernel.org, ulf.hansson@linaro.org, paulmck@kernel.org, tglx@linutronix.de, naresh.kamboju@linaro.org, peterz@infradead.org Subject: [RFC][PATCH 4/4] acpi: Take over RCU-idle for C3-BM idle References: <20200915103157.345404192@infradead.org> MIME-Version: 1.0 Sender: linux-pm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org The C3 BusMaster idle code takes lock in a number of places, some deep inside the ACPI code. Instead of wrapping it all in RCU_NONIDLE, have the driver take over RCU-idle duty and avoid flipping RCU state back and forth a lot. ( by marking 'C3 && bm_check' as RCU_IDLE, we _must_ call enter_bm() for that combination, otherwise we'll loose RCU-idle, this requires shuffling some code around ) Signed-off-by: Peter Zijlstra (Intel) --- drivers/acpi/processor_idle.c | 69 +++++++++++++++++++++++++++++------------- 1 file changed, 49 insertions(+), 20 deletions(-) --- a/drivers/acpi/processor_idle.c +++ b/drivers/acpi/processor_idle.c @@ -558,22 +558,43 @@ static DEFINE_RAW_SPINLOCK(c3_lock); /** * acpi_idle_enter_bm - enters C3 with proper BM handling + * @drv: cpuidle driver * @pr: Target processor * @cx: Target state context + * @index: index of target state */ -static void acpi_idle_enter_bm(struct acpi_processor *pr, - struct acpi_processor_cx *cx) +static int acpi_idle_enter_bm(struct cpuidle_driver *drv, + struct acpi_processor *pr, + struct acpi_processor_cx *cx, + int index) { + static struct acpi_processor_cx safe_cx = { + .entry_method = ACPI_CSTATE_HALT, + }; + /* * disable bus master * bm_check implies we need ARB_DIS * bm_control implies whether we can do ARB_DIS * - * That leaves a case where bm_check is set and bm_control is - * not set. In that case we cannot do much, we enter C3 - * without doing anything. + * That leaves a case where bm_check is set and bm_control is not set. + * In that case we cannot do much, we enter C3 without doing anything. */ - if (pr->flags.bm_control) { + bool dis_bm = pr->flags.bm_control; + + /* If we can skip BM, demote to a safe state. */ + if (!cx->bm_sts_skip && acpi_idle_bm_check()) { + dis_bm = false; + index = drv->safe_state_index; + if (index >= 0) { + cx = this_cpu_read(acpi_cstate[index]); + } else { + cx = &safe_cx; + index = -EBUSY; + } + } + + if (dis_bm) { raw_spin_lock(&c3_lock); c3_cpu_count++; /* Disable bus master arbitration when all CPUs are in C3 */ @@ -582,15 +603,21 @@ static void acpi_idle_enter_bm(struct ac raw_spin_unlock(&c3_lock); } + rcu_idle_enter(); + acpi_idle_do_entry(cx); + rcu_idle_exit(); + /* Re-enable bus master arbitration */ - if (pr->flags.bm_control) { + if (dis_bm) { raw_spin_lock(&c3_lock); acpi_write_bit_register(ACPI_BITREG_ARB_DISABLE, 0); c3_cpu_count--; raw_spin_unlock(&c3_lock); } + + return index; } static int acpi_idle_enter(struct cpuidle_device *dev, @@ -604,20 +631,13 @@ static int acpi_idle_enter(struct cpuidl return -EINVAL; if (cx->type != ACPI_STATE_C1) { + if (cx->type == ACPI_STATE_C3 && pr->flags.bm_check) + return acpi_idle_enter_bm(drv, pr, cx, index); + + /* C2 to C1 demotion. */ if (acpi_idle_fallback_to_c1(pr) && num_online_cpus() > 1) { index = ACPI_IDLE_STATE_START; cx = per_cpu(acpi_cstate[index], dev->cpu); - } else if (cx->type == ACPI_STATE_C3 && pr->flags.bm_check) { - if (cx->bm_sts_skip || !acpi_idle_bm_check()) { - acpi_idle_enter_bm(pr, cx); - return index; - } else if (drv->safe_state_index >= 0) { - index = drv->safe_state_index; - cx = per_cpu(acpi_cstate[index], dev->cpu); - } else { - acpi_safe_halt(); - return -EBUSY; - } } } @@ -641,7 +661,13 @@ static int acpi_idle_enter_s2idle(struct return 0; if (pr->flags.bm_check) { - acpi_idle_enter_bm(pr, cx); + u8 bm_sts_skip = cx->bm_sts_skip; + + /* Don't check BM_STS, do an unconditional ARB_DIS for S2IDLE */ + cx->bm_sts_skip = 1; + acpi_idle_enter_bm(drv, pr, cx, index); + cx->bm_sts_skip = bm_sts_skip; + return 0; } else { ACPI_FLUSH_CPU_CACHE(); @@ -674,8 +700,11 @@ static int acpi_processor_setup_cpuidle_ if (lapic_timer_needs_broadcast(pr, cx)) state->flags |= CPUIDLE_FLAG_TIMER_STOP; - if (cx->type == ACPI_STATE_C3) + if (cx->type == ACPI_STATE_C3) { state->flags |= CPUIDLE_FLAG_TLB_FLUSHED; + if (pr->flags.bm_check) + state->flags |= CPUIDLE_FLAG_RCU_IDLE; + } count++; if (count == CPUIDLE_STATE_MAX)