From patchwork Thu Jan 2 15:01:55 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 13924517 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C58511B413B; Thu, 2 Jan 2025 15:02:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735830130; cv=none; b=Q+4vPgqKhcD7WW+PNswohAgvojqep6vIAmuAIhtW/LR9xr5PeyUa6RLATmBjx1kwr9GjWvGe9QBSsEq1sDgGxNEbArPLeGxuxnBxhnbz0uiholmn0Ly8/uL/7+7vPrpjfX5U2u8D6Fqf/9uzlsSB9RjjHJ6fC2rm7VMrbw0TfNY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735830130; c=relaxed/simple; bh=6M3MwJVPQVJfwZCyUXnKFfzUiBOjUgHdAX5hBQtJ7Q0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nARbfYI+X2Z+hWDcq7T/ASQynDg7G0yH6u3zW70XyacVsQbh+UiqGpMVW0d37YQWjgLkkzN6ByGgZgG8CXu6jMiZ/tqRVH85a/5GLvNa0ofmCgb5yIHsfMoVM/da5fvm766geyw1rBOl1nKhQRUz5Er8AG9HhKKrQvTRa0aIS8s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=L63/ynnU; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="L63/ynnU" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 24548C4CED7; Thu, 2 Jan 2025 15:02:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1735830130; bh=6M3MwJVPQVJfwZCyUXnKFfzUiBOjUgHdAX5hBQtJ7Q0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=L63/ynnUEauCbVUAEBzIyL8lnuVzbb6Q6MQwj/R5Qj4zxbUTVGNhAR6BYL59OZa9j PReQL/bPHabbBuhHMfpTFUOziL6CnDFf9bZYRPUc0J6S2PBUsSP2tiQuwkuB+zWRUg t2/nV8Ts607pMLfqadssTmz/4hTUZGNTVgiEsplhGR8qiBB4kGyxo99NH4lR7TTRUc amHMpddZ8QqPLCoicVfitIVrPSyg5XLgt1JMo0LPPPJ+jYA+ZwjFPQlC/Wc1XucIS2 LEYNUGCVJKuVo/cAPMT+Voa1g6ybXWrnoST1nWVWUNqMAfjItkqu9Jcqf0MxGcZRWK 5tQSVEIelL/SA== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , "Rafael J . Wysocki" , Daniel Lezcano , linux-pm@vger.kernel.org, Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Peter Zijlstra , Marcelo Tosatti Subject: [PATCH 1/6] cpuidle: Remove unnecessary current_clr_polling_and_test() from haltpoll Date: Thu, 2 Jan 2025 16:01:55 +0100 Message-ID: <20250102150201.21639-2-frederic@kernel.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20250102150201.21639-1-frederic@kernel.org> References: <20250102150201.21639-1-frederic@kernel.org> Precedence: bulk X-Mailing-List: linux-pm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 When cpuidle drivers ->enter() callback are called, the TIF_NR_POLLING flag is cleared already and TIF_NEED_RESCHED checked by call_cpuidle(). Therefore calling current_clr_polling_and_test() is redundant here and further setting of TIF_NEED_RESCHED will result in an IPI and thus an idle loop exit. This call can be safely removed. Cc: Marcelo Tosatti Acked-by: Rafael J. Wysocki Signed-off-by: Frederic Weisbecker --- drivers/cpuidle/cpuidle-haltpoll.c | 3 --- 1 file changed, 3 deletions(-) diff --git a/drivers/cpuidle/cpuidle-haltpoll.c b/drivers/cpuidle/cpuidle-haltpoll.c index bcd03e893a0a..07dd9f000273 100644 --- a/drivers/cpuidle/cpuidle-haltpoll.c +++ b/drivers/cpuidle/cpuidle-haltpoll.c @@ -28,9 +28,6 @@ static enum cpuhp_state haltpoll_hp_state; static __cpuidle int default_enter_idle(struct cpuidle_device *dev, struct cpuidle_driver *drv, int index) { - if (current_clr_polling_and_test()) - return index; - arch_cpu_idle(); return index; } From patchwork Thu Jan 2 15:01:56 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 13924518 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DE7AC1B4237; Thu, 2 Jan 2025 15:02:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735830134; cv=none; b=SVxVpktWnCAuGTeinuZm1IoQrUcUxb8xNoXhz2EGOjKGLsIDknHkf5v1Q95H70dFps/Hiloa52NlQSm1rM5qRit0XCocrh+G/CtRw45Gagd8kWmFGMtGYyEbHjmj9MwxELBImZ+oM8YIebjWzumBo4OXVNe79Xa5L0GGCoFvv68= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735830134; c=relaxed/simple; bh=eDrsAGqnYYDkW7fmOHZJ0e9gFrLnk1cHsS3rf1os5kE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Ogh+A2vi8+pKXMbEa9mRRW/J0e4FxOxPylmskjN348NGdJVMflJan6CjDzK6QVmLNSWNC7szYNXJSBnUSUiPTkvgrsQmww0Q2CbbvC9WQ0ars7/3pPBoVdeBz19y/Eh2DfFPOjIEYN7awKgu39Cdfg5FFiUGqH01yu1Q9LZEaeg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=hZY7fUM0; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="hZY7fUM0" Received: by smtp.kernel.org (Postfix) with ESMTPSA id D5E72C4CEDC; Thu, 2 Jan 2025 15:02:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1735830133; bh=eDrsAGqnYYDkW7fmOHZJ0e9gFrLnk1cHsS3rf1os5kE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=hZY7fUM0BLbSTCOQRGxwBv+TGOmEkHwfNEUfDupmtZ7HcHxQHbPiKfTy2zgcXI+b5 z8DCxcpDW2o618S89Inq8Blmg3FfdWEDJnSLzTXE/Ozo7hW3j9sCg1zOQful4KtYao yngNZtRYRGgGMpvMXqu3/7ZcY3doZqrqL2HHvtB7n39edBcnrXLuITQnt9QS5tUGWf e8JiLzXH7aNWDADBhoKRNb2NfB9cC6dQMfYqiL8fWbBO+uAa5F4jRbLcgnnWqT4Dwf a/skfGfN19rkXHsPL0PJEwVGV9jpKlvyk1c2saDwWIr6TtPdI/aA6S0VcX4ve+aw6B dsw1cR30n09IQ== From: Frederic Weisbecker To: LKML Cc: Peter Zijlstra , "Rafael J . Wysocki" , Daniel Lezcano , linux-pm@vger.kernel.org, Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Jacob Pan , Frederic Weisbecker Subject: [PATCH 2/6] cpuidle: Introduce CPUIDLE_FLAG_MWAIT Date: Thu, 2 Jan 2025 16:01:56 +0100 Message-ID: <20250102150201.21639-3-frederic@kernel.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20250102150201.21639-1-frederic@kernel.org> References: <20250102150201.21639-1-frederic@kernel.org> Precedence: bulk X-Mailing-List: linux-pm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Peter Zijlstra Provide a way to tell the cpuidle core about states monitoring TIF_NEED_RESCHED on the hardware level, monitor/mwait users being the only examples in use. This will allow cpuidle core to manage TIF_NR_POLLING on behalf of all kinds of TIF_NEED_RESCHED watching states while keeping a necessary distinction for the governors between software loops polling on TIF_NEED_RESCHED and hardware monitored writes to thread flags. [fweisbec: _ Initialize flag from acpi_processor_setup_cstates() instead of acpi_processor_setup_lpi_states(), as the latter seem to be about arm64... _ Rename CPUIDLE_FLAG_NO_IPI to CPUIDLE_FLAG_MWAIT] Signed-off-by: Peter Zijlstra Signed-off-by: Frederic Weisbecker --- drivers/acpi/processor_idle.c | 3 +++ drivers/idle/intel_idle.c | 5 ++++- include/linux/cpuidle.h | 1 + 3 files changed, 8 insertions(+), 1 deletion(-) diff --git a/drivers/acpi/processor_idle.c b/drivers/acpi/processor_idle.c index 698897b29de2..66cb5536d91e 100644 --- a/drivers/acpi/processor_idle.c +++ b/drivers/acpi/processor_idle.c @@ -806,6 +806,9 @@ static int acpi_processor_setup_cstates(struct acpi_processor *pr) if (cx->type == ACPI_STATE_C1 || cx->type == ACPI_STATE_C2) drv->safe_state_index = count; + if (cx->entry_method == ACPI_CSTATE_FFH) + state->flags |= CPUIDLE_FLAG_MWAIT; + /* * Halt-induced C1 is not good for ->enter_s2idle, because it * re-enables interrupts on exit. Moreover, C1 is generally not diff --git a/drivers/idle/intel_idle.c b/drivers/idle/intel_idle.c index ac4d8faa3886..d52723fbeb04 100644 --- a/drivers/idle/intel_idle.c +++ b/drivers/idle/intel_idle.c @@ -1787,7 +1787,8 @@ static void __init intel_idle_init_cstates_acpi(struct cpuidle_driver *drv) if (cx->type > ACPI_STATE_C1) state->target_residency *= 3; - state->flags = MWAIT2flg(cx->address); + state->flags = MWAIT2flg(cx->address) | CPUIDLE_FLAG_MWAIT; + if (cx->type > ACPI_STATE_C2) state->flags |= CPUIDLE_FLAG_TLB_FLUSHED; @@ -2072,6 +2073,8 @@ static bool __init intel_idle_verify_cstate(unsigned int mwait_hint) static void state_update_enter_method(struct cpuidle_state *state, int cstate) { + state->flags |= CPUIDLE_FLAG_MWAIT; + if (state->flags & CPUIDLE_FLAG_INIT_XSTATE) { /* * Combining with XSTATE with IBRS or IRQ_ENABLE flags diff --git a/include/linux/cpuidle.h b/include/linux/cpuidle.h index a9ee4fe55dcf..b8084617aa27 100644 --- a/include/linux/cpuidle.h +++ b/include/linux/cpuidle.h @@ -85,6 +85,7 @@ struct cpuidle_state { #define CPUIDLE_FLAG_OFF BIT(4) /* disable this state by default */ #define CPUIDLE_FLAG_TLB_FLUSHED BIT(5) /* idle-state flushes TLBs */ #define CPUIDLE_FLAG_RCU_IDLE BIT(6) /* idle-state takes care of RCU */ +#define CPUIDLE_FLAG_MWAIT BIT(7) /* hardware need_resched() monitoring */ struct cpuidle_device_kobj; struct cpuidle_state_kobj; From patchwork Thu Jan 2 15:01:57 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 13924519 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D546E1B4120; Thu, 2 Jan 2025 15:02:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735830138; cv=none; b=D8lU+5NLTi+DxIh1NZEfISHrzASz47zxYSQ5pn2OuDTZlv9fwV8p+fVtAq6SYsIikR2AJbJDlju92wlyATy0iHeFZEkIh2VgzlJvgUQiTmXmxZYYHgCvM9j/FWSd23wu2pax/d6uDdnY1F5FAHJEFkKRB3n7wvE+0sTqu7jhJ1s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735830138; c=relaxed/simple; bh=F1FmzVp6iCOT8qiM1kjCr6AQCFvi69gi8t0iIwn4s5I=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ElFTOrIUP+6o+iC9j7q+rMhvf+w00w+Uz/i+x6k/GhkwcgUvrLbuCYEdO0xbfqcls5NgZB5tWU9cOJu+RdkqcKrFgMzIteoF+SymZhLPUzqS6svM+nMoB07Mp3C7vZhmlGcX89lKPxouQiYCM+xL4/YVAoADsmh1ER9ufDYjtik= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=StxHrsmm; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="StxHrsmm" Received: by smtp.kernel.org (Postfix) with ESMTPSA id D74A6C4CEDD; Thu, 2 Jan 2025 15:02:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1735830138; bh=F1FmzVp6iCOT8qiM1kjCr6AQCFvi69gi8t0iIwn4s5I=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=StxHrsmmFW/2gj6R3DymlKgYReiQApi2t3zi3CclCNOiILo4eEg/vU4OKWXuPvo67 W6qBi5R1cL28xjBz6dWLY/oCJvJf+2erTsE3GYnQINohLp6oHuJ/rmYVUei1NPTyYL B74GNMHBj+rH6TEPEYB5fRrQ/MQlXabbGDVrhluoWAM7PcooKFdhfL19p+z12ksR1G Qk+hc6C4byaPyg2g3H6JFjB6x84gpPCsodZJzzfXY+uHK14IqCaG0Sp0NjsSAcx23N T6AvpdeOyx/iN4B8Y7U/Q8bd5D5U8R/5Hi34pQ8sgiDiMJiV1G4Y44xFcJR/75711u rRfmZP72UYpZA== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Jacob Pan , Peter Zijlstra , "Rafael J . Wysocki" , Daniel Lezcano , linux-pm@vger.kernel.org, Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Russell King , Catalin Marinas , Will Deacon , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Madhavan Srinivasan , Paul Walmsley , Palmer Dabbelt , Albert Ou , Len Brown , Arnd Bergmann , linux-arch@vger.kernel.org Subject: [PATCH 3/6] x86/cpuidle: Move buggy mwait implementations away from CPUIDLE_FLAG_MWAIT Date: Thu, 2 Jan 2025 16:01:57 +0100 Message-ID: <20250102150201.21639-4-frederic@kernel.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20250102150201.21639-1-frederic@kernel.org> References: <20250102150201.21639-1-frederic@kernel.org> Precedence: bulk X-Mailing-List: linux-pm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Buggy MWAIT implementations can't carry the CPUIDLE_FLAG_MWAIT flag because they require IPIs to wake up. Therefore they shouldn't be called with TIF_NR_POLLING. Reported-by: Rafael J. Wysocki Signed-off-by: Frederic Weisbecker --- arch/arm/include/asm/cpuidle.h | 2 ++ arch/arm64/include/asm/cpuidle.h | 3 +++ arch/powerpc/include/asm/cpuidle.h | 4 ++++ arch/riscv/include/asm/cpuidle.h | 2 ++ arch/x86/include/asm/cpuidle.h | 12 ++++++++++++ drivers/acpi/processor_idle.c | 4 +++- drivers/idle/intel_idle.c | 9 +++++++-- include/asm-generic/Kbuild | 1 + include/asm-generic/cpuidle.h | 10 ++++++++++ 9 files changed, 44 insertions(+), 3 deletions(-) create mode 100644 arch/x86/include/asm/cpuidle.h create mode 100644 include/asm-generic/cpuidle.h diff --git a/arch/arm/include/asm/cpuidle.h b/arch/arm/include/asm/cpuidle.h index 397be5ed30e7..0ea1d2ec837d 100644 --- a/arch/arm/include/asm/cpuidle.h +++ b/arch/arm/include/asm/cpuidle.h @@ -55,4 +55,6 @@ struct arm_cpuidle_irq_context { }; #define arm_cpuidle_save_irq_context(c) (void)c #define arm_cpuidle_restore_irq_context(c) (void)c +#include + #endif diff --git a/arch/arm64/include/asm/cpuidle.h b/arch/arm64/include/asm/cpuidle.h index 2047713e097d..ef49124135a7 100644 --- a/arch/arm64/include/asm/cpuidle.h +++ b/arch/arm64/include/asm/cpuidle.h @@ -38,4 +38,7 @@ struct arm_cpuidle_irq_context { }; #define arm_cpuidle_save_irq_context(c) (void)c #define arm_cpuidle_restore_irq_context(c) (void)c #endif + +#include + #endif diff --git a/arch/powerpc/include/asm/cpuidle.h b/arch/powerpc/include/asm/cpuidle.h index 0cce5dc7fb1c..788706bc04ec 100644 --- a/arch/powerpc/include/asm/cpuidle.h +++ b/arch/powerpc/include/asm/cpuidle.h @@ -102,4 +102,8 @@ static inline void report_invalid_psscr_val(u64 psscr_val, int err) #endif +#ifndef __ASSEMBLY__ +#include +#endif + #endif diff --git a/arch/riscv/include/asm/cpuidle.h b/arch/riscv/include/asm/cpuidle.h index 71fdc607d4bc..1f1b24901d86 100644 --- a/arch/riscv/include/asm/cpuidle.h +++ b/arch/riscv/include/asm/cpuidle.h @@ -21,4 +21,6 @@ static inline void cpu_do_idle(void) wait_for_interrupt(); } +#include + #endif diff --git a/arch/x86/include/asm/cpuidle.h b/arch/x86/include/asm/cpuidle.h new file mode 100644 index 000000000000..a59db1a3314a --- /dev/null +++ b/arch/x86/include/asm/cpuidle.h @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_CPUIDLE_H +#define _ASM_X86_CPUIDLE_H + +#include + +static inline bool arch_cpuidle_mwait_needs_ipi(void) +{ + return boot_cpu_has_bug(X86_BUG_MONITOR); +} + +#endif /* _ASM_X86_CPUIDLE_H */ diff --git a/drivers/acpi/processor_idle.c b/drivers/acpi/processor_idle.c index 66cb5536d91e..0f29dd92b346 100644 --- a/drivers/acpi/processor_idle.c +++ b/drivers/acpi/processor_idle.c @@ -23,6 +23,7 @@ #include #include #include +#include /* * Include the apic definitions for x86 to have the APIC timer related defines @@ -806,7 +807,8 @@ static int acpi_processor_setup_cstates(struct acpi_processor *pr) if (cx->type == ACPI_STATE_C1 || cx->type == ACPI_STATE_C2) drv->safe_state_index = count; - if (cx->entry_method == ACPI_CSTATE_FFH) + if (cx->entry_method == ACPI_CSTATE_FFH && + !arch_cpuidle_mwait_needs_ipi()) state->flags |= CPUIDLE_FLAG_MWAIT; /* diff --git a/drivers/idle/intel_idle.c b/drivers/idle/intel_idle.c index d52723fbeb04..b2f494effd4a 100644 --- a/drivers/idle/intel_idle.c +++ b/drivers/idle/intel_idle.c @@ -56,6 +56,7 @@ #include #include #include +#include #define INTEL_IDLE_VERSION "0.5.1" @@ -1787,7 +1788,10 @@ static void __init intel_idle_init_cstates_acpi(struct cpuidle_driver *drv) if (cx->type > ACPI_STATE_C1) state->target_residency *= 3; - state->flags = MWAIT2flg(cx->address) | CPUIDLE_FLAG_MWAIT; + state->flags = MWAIT2flg(cx->address); + + if (!arch_cpuidle_mwait_needs_ipi()) + state->flags |= CPUIDLE_FLAG_MWAIT; if (cx->type > ACPI_STATE_C2) state->flags |= CPUIDLE_FLAG_TLB_FLUSHED; @@ -2073,7 +2077,8 @@ static bool __init intel_idle_verify_cstate(unsigned int mwait_hint) static void state_update_enter_method(struct cpuidle_state *state, int cstate) { - state->flags |= CPUIDLE_FLAG_MWAIT; + if (!arch_cpuidle_mwait_needs_ipi()) + state->flags |= CPUIDLE_FLAG_MWAIT; if (state->flags & CPUIDLE_FLAG_INIT_XSTATE) { /* diff --git a/include/asm-generic/Kbuild b/include/asm-generic/Kbuild index 1b43c3a77012..7754da499d16 100644 --- a/include/asm-generic/Kbuild +++ b/include/asm-generic/Kbuild @@ -13,6 +13,7 @@ mandatory-y += cacheflush.h mandatory-y += cfi.h mandatory-y += checksum.h mandatory-y += compat.h +mandatory-y += cpuidle.h mandatory-y += current.h mandatory-y += delay.h mandatory-y += device.h diff --git a/include/asm-generic/cpuidle.h b/include/asm-generic/cpuidle.h new file mode 100644 index 000000000000..748a2022ed2a --- /dev/null +++ b/include/asm-generic/cpuidle.h @@ -0,0 +1,10 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __ASM_CPUIDLE_H +#define __ASM_CPUIDLE_H + +static inline bool arch_cpuidle_mwait_needs_ipi(void) +{ + return true; +} + +#endif /* __ASM_CPUIDLE_H */ From patchwork Thu Jan 2 15:01:58 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 13924520 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2E9771B4F0F; Thu, 2 Jan 2025 15:02:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735830141; cv=none; b=ljajt9GXtK5yperxXnCw8inn0oI66zuo0HCV8T/XFoAgBTnMKcxK+nX1+Jd7+4NygD6TZJ3suajC2CGeC7VVQ6Fm96LV4Nqcom1jqTUfzLHkwjZvaOnAy9wqMf1Z8fWdRGE77gw6wmS/VaCACYJ3PLC0b8TnaY+7hy+4AMkLSvg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735830141; c=relaxed/simple; bh=NavR6UqlfYLDkNot5pquuMbNapnDO70AeqgbD4rubNQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=JfcWwXR63RFULIXaIGq2YPN+X8ZfjzIzHdHP1wmmCy7+pkMY4dJiqyMewbp6pLlYJ+aKJ2SEF8M+WMKWAyJER8B6HbuND5FYVFXs85o6Kba7LMtnkQZz47FxMdSCajbtWtJg6srSdov5V3BcGY97PjFjNVIIRZZO6UzRYPYbWME= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=ii1zFYfY; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="ii1zFYfY" Received: by smtp.kernel.org (Postfix) with ESMTPSA id B50B0C4CEDC; Thu, 2 Jan 2025 15:02:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1735830141; bh=NavR6UqlfYLDkNot5pquuMbNapnDO70AeqgbD4rubNQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ii1zFYfY+IdRCDIfGXBqRZ9EQrbkQuQHJbyiNYcm1DLQdsiqh33r9b8VYJ4MoCs4e 0Yrd74ltYRuBi3yv90JL8VcjEtHH2oLdd/b1g5x1fndmmqhfvTN3SJb67pIn6eE3Hr i4bHnUDVclbQnH3TKuwHHX76hydFQdnbJ/JCIN3VPP7cRM7VFrShaHKXFB5SmBe/yS VQ2WFrmVZdoNfZ2z/53Hb8649SVMesCTwoFuTv0jPFhBAksToL771Lnn4b4k9I/4CZ tAkG6DQx8RBpplNVlcGjkHjgJc9swYpLG2r5wjcf11f35Qoag8n+CiNjQCdQJhi+vl DmK5aLVyecDIg== From: Frederic Weisbecker To: LKML Cc: Peter Zijlstra , Jacob Pan , "Rafael J . Wysocki" , Daniel Lezcano , linux-pm@vger.kernel.org, Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Frederic Weisbecker Subject: [PATCH 4/6] cpuidle: Handle TIF_NR_POLLING on behalf of CPUIDLE_FLAG_MWAIT states Date: Thu, 2 Jan 2025 16:01:58 +0100 Message-ID: <20250102150201.21639-5-frederic@kernel.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20250102150201.21639-1-frederic@kernel.org> References: <20250102150201.21639-1-frederic@kernel.org> Precedence: bulk X-Mailing-List: linux-pm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Peter Zijlstra The current handling of TIF_NR_POLLING is a bit of a maze: 1) TIF_NR_POLLING is set on idle entry (one atomic set) 2) Once cpuidle has selected an appropriate state and the tick is evaluated and then possibly stopped, TIF_NR_POLLING is cleared (one RmW operation) 2) The cpuidle state is then called with TIF_NR_POLLING cleared but if the state polls on (or monitors) need_resched() it sets again TIF_NR_POLLING before sleeping and clears it on wake-up. Summary: another pair of set/clear 3) Set back TIF_NR_POLLING (one atomic set) 4) goto 2) if need_resched() is not set All those costly atomic operations, fully ordered RmW for some of them, could be avoided if the cpuidle core knew in advance if the target state polls on (or monitors) need_resched(). If so, TIF_NR_POLLING could simply be set once upon entering the idle loop and cleared once after idle loop exit. Start dealing with that with handling TIF_NR_POLLING on behalf of mwait based states. [fweisbec: _ Handle broadcast properly _ Ignore mwait_idle() as it can be used by default_idle_call()] Signed-off-by: Peter Zijlstra Signed-off-by: Frederic Weisbecker --- arch/x86/include/asm/mwait.h | 27 +++++++++++------------- drivers/cpuidle/cpuidle.c | 22 +++++++++++++++++++- include/linux/sched/idle.h | 7 ++++++- kernel/sched/idle.c | 40 +++++++++++++----------------------- 4 files changed, 53 insertions(+), 43 deletions(-) diff --git a/arch/x86/include/asm/mwait.h b/arch/x86/include/asm/mwait.h index 920426d691ce..3e06a7f3bf5a 100644 --- a/arch/x86/include/asm/mwait.h +++ b/arch/x86/include/asm/mwait.h @@ -116,25 +116,22 @@ static __always_inline void __sti_mwait(unsigned long eax, unsigned long ecx) */ static __always_inline void mwait_idle_with_hints(unsigned long eax, unsigned long ecx) { - if (static_cpu_has_bug(X86_BUG_MONITOR) || !current_set_polling_and_test()) { - if (static_cpu_has_bug(X86_BUG_CLFLUSH_MONITOR)) { - mb(); - clflush((void *)¤t_thread_info()->flags); - mb(); - } + if (static_cpu_has_bug(X86_BUG_CLFLUSH_MONITOR)) { + mb(); + clflush((void *)¤t_thread_info()->flags); + mb(); + } - __monitor((void *)¤t_thread_info()->flags, 0, 0); + __monitor((void *)¤t_thread_info()->flags, 0, 0); - if (!need_resched()) { - if (ecx & 1) { - __mwait(eax, ecx); - } else { - __sti_mwait(eax, ecx); - raw_local_irq_disable(); - } + if (!need_resched()) { + if (ecx & 1) { + __mwait(eax, ecx); + } else { + __sti_mwait(eax, ecx); + raw_local_irq_disable(); } } - current_clr_polling(); } /* diff --git a/drivers/cpuidle/cpuidle.c b/drivers/cpuidle/cpuidle.c index 0835da449db8..46c0a2726f67 100644 --- a/drivers/cpuidle/cpuidle.c +++ b/drivers/cpuidle/cpuidle.c @@ -217,10 +217,10 @@ noinstr int cpuidle_enter_state(struct cpuidle_device *dev, int index) { int entered_state; - struct cpuidle_state *target_state = &drv->states[index]; bool broadcast = !!(target_state->flags & CPUIDLE_FLAG_TIMER_STOP); ktime_t time_start, time_end; + bool polling; instrumentation_begin(); @@ -237,6 +237,23 @@ noinstr int cpuidle_enter_state(struct cpuidle_device *dev, broadcast = false; } + polling = target_state->flags & CPUIDLE_FLAG_MWAIT; + + /* + * If the target state doesn't poll on need_resched(), this is + * the last check after which further TIF_NEED_RESCHED remote setting + * will involve an IPI. + */ + if (!polling && current_clr_polling_and_test()) { + if (broadcast) + tick_broadcast_exit(); + dev->last_residency_ns = 0; + local_irq_enable(); + instrumentation_end(); + return -EBUSY; + } + + if (target_state->flags & CPUIDLE_FLAG_TLB_FLUSHED) leave_mm(); @@ -336,6 +353,9 @@ noinstr int cpuidle_enter_state(struct cpuidle_device *dev, dev->states_usage[index].rejected++; } + if (!polling) + __current_set_polling(); + instrumentation_end(); return entered_state; diff --git a/include/linux/sched/idle.h b/include/linux/sched/idle.h index e670ac282333..3e3482bfb028 100644 --- a/include/linux/sched/idle.h +++ b/include/linux/sched/idle.h @@ -68,6 +68,8 @@ static __always_inline bool __must_check current_set_polling_and_test(void) static __always_inline bool __must_check current_clr_polling_and_test(void) { + bool ret; + __current_clr_polling(); /* @@ -76,7 +78,10 @@ static __always_inline bool __must_check current_clr_polling_and_test(void) */ smp_mb__after_atomic(); - return unlikely(tif_need_resched()); + ret = unlikely(tif_need_resched()); + if (ret) + __current_set_polling(); + return ret; } #else diff --git a/kernel/sched/idle.c b/kernel/sched/idle.c index 621696269584..9eece3df1080 100644 --- a/kernel/sched/idle.c +++ b/kernel/sched/idle.c @@ -114,12 +114,13 @@ void __cpuidle default_idle_call(void) stop_critical_timings(); ct_cpuidle_enter(); - arch_cpu_idle(); + arch_cpu_idle(); // XXX assumes !polling ct_cpuidle_exit(); start_critical_timings(); trace_cpu_idle(PWR_EVENT_EXIT, smp_processor_id()); cond_tick_broadcast_exit(); + __current_set_polling(); } local_irq_enable(); instrumentation_end(); @@ -128,31 +129,14 @@ void __cpuidle default_idle_call(void) static int call_cpuidle_s2idle(struct cpuidle_driver *drv, struct cpuidle_device *dev) { + int ret; + if (current_clr_polling_and_test()) return -EBUSY; - return cpuidle_enter_s2idle(drv, dev); -} - -static int call_cpuidle(struct cpuidle_driver *drv, struct cpuidle_device *dev, - int next_state) -{ - /* - * The idle task must be scheduled, it is pointless to go to idle, just - * update no idle residency and return. - */ - if (current_clr_polling_and_test()) { - dev->last_residency_ns = 0; - local_irq_enable(); - return -EBUSY; - } - - /* - * Enter the idle state previously returned by the governor decision. - * This function will block until an interrupt occurs and will take - * care of re-enabling the local interrupts - */ - return cpuidle_enter(drv, dev, next_state); + ret = cpuidle_enter_s2idle(drv, dev); + __current_set_polling(); + return ret; } /** @@ -213,7 +197,7 @@ static void cpuidle_idle_call(void) tick_nohz_idle_stop_tick(); next_state = cpuidle_find_deepest_state(drv, dev, max_latency_ns); - call_cpuidle(drv, dev, next_state); + cpuidle_enter(drv, dev, next_state); } else { bool stop_tick = true; @@ -227,7 +211,12 @@ static void cpuidle_idle_call(void) else tick_nohz_idle_retain_tick(); - entered_state = call_cpuidle(drv, dev, next_state); + /* + * Enter the idle state previously returned by the governor decision. + * This function will block until an interrupt occurs and will take + * care of re-enabling the local interrupts. + */ + entered_state = cpuidle_enter(drv, dev, next_state); /* * Give the governor an opportunity to reflect on the outcome */ @@ -235,7 +224,6 @@ static void cpuidle_idle_call(void) } exit_idle: - __current_set_polling(); /* * It is up to the idle functions to re-enable local interrupts From patchwork Thu Jan 2 15:01:59 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 13924521 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EF1FF1B4137; Thu, 2 Jan 2025 15:02:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735830144; cv=none; b=VaHvH9dzQUYYIahqv62MJR38uA8fPcYc9sJqZQu0uasSJVDdZOmCNyyZ4jkNIki5V9kDsCfmMSgxL2DY6NFSEKOmF3ZhvS75ZgY8d05A8FnMAiUxvCLg8uAYxbGFxRIB3yNd3iJoJ/1pOhGCyJ68IKmvcLO/vdh5FUO+3u95mYY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735830144; c=relaxed/simple; bh=3JoSvgVSyWPdcDhFxJbqeFmv3Z/rfrPtZzh7qZxBCX8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=R1SS52VgUroEt4Vuom4jsH1g/Vl3lOitetHgQdqdQqb3iqcW5/cpgGKtojj/ecSUAoY3iMHKO0qIPV1g7xl+bOQosjoWHnlW8WWJvdk58u0Dp6/OSy/alb+g4xByt0/K1aukoKw09Tm/EEE9MXAzEWqkifmooI9SEyWrlGENlPs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Mv6syPWb; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Mv6syPWb" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 762CBC4CEDC; Thu, 2 Jan 2025 15:02:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1735830143; bh=3JoSvgVSyWPdcDhFxJbqeFmv3Z/rfrPtZzh7qZxBCX8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Mv6syPWbiN7Tx5+Yoc4UulxCwUKy4O0bmvLn47nwBqpuOJV3bDPky3UCqfVa00XlZ kuEZMFBO8Q3xLU7yp4lhywRa3sluwbmEfxQeEX37PBU2xkbfewGM/OMC3/JG2NvFAw T0FydsonY8VHD59oNfaO2Gpn68BnxMlTdpvnacSBZa18MYX7fpgEDvyELBkrtOHOaK hEtKr4uI3fyfgOyMrSO2NyU3A125yipjINU232/G4cIE+PkBNgsTTnMcasPxw4uXrq wtHo0PacbNXJtsbqC24dgvA0kzat8tn3sSnrAJnEs2i8i3nZENuoBunLP0SMRTsZ/X QmTtadfXX2y0A== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Peter Zijlstra , "Rafael J . Wysocki" , Daniel Lezcano , linux-pm@vger.kernel.org, Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "Rafael J. Wysocki" Subject: [PATCH 5/6] cpuidle: Remove call_cpuidle_s2idle() Date: Thu, 2 Jan 2025 16:01:59 +0100 Message-ID: <20250102150201.21639-6-frederic@kernel.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20250102150201.21639-1-frederic@kernel.org> References: <20250102150201.21639-1-frederic@kernel.org> Precedence: bulk X-Mailing-List: linux-pm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 This middle call is unecessary, especially now that its counterpart call_cpuidle() has been removed. Suggested-by: Rafael J. Wysocki Signed-off-by: Frederic Weisbecker --- kernel/sched/idle.c | 23 ++++++----------------- 1 file changed, 6 insertions(+), 17 deletions(-) diff --git a/kernel/sched/idle.c b/kernel/sched/idle.c index 9eece3df1080..86b902eb24fe 100644 --- a/kernel/sched/idle.c +++ b/kernel/sched/idle.c @@ -126,19 +126,6 @@ void __cpuidle default_idle_call(void) instrumentation_end(); } -static int call_cpuidle_s2idle(struct cpuidle_driver *drv, - struct cpuidle_device *dev) -{ - int ret; - - if (current_clr_polling_and_test()) - return -EBUSY; - - ret = cpuidle_enter_s2idle(drv, dev); - __current_set_polling(); - return ret; -} - /** * cpuidle_idle_call - the main idle function * @@ -184,10 +171,12 @@ static void cpuidle_idle_call(void) u64 max_latency_ns; if (idle_should_enter_s2idle()) { - - entered_state = call_cpuidle_s2idle(drv, dev); - if (entered_state > 0) - goto exit_idle; + if (!current_clr_polling_and_test()) { + entered_state = cpuidle_enter_s2idle(drv, dev); + __current_set_polling(); + if (entered_state > 0) + goto exit_idle; + } max_latency_ns = U64_MAX; } else { From patchwork Thu Jan 2 15:02:00 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 13924522 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 725C61B414A; Thu, 2 Jan 2025 15:02:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735830146; cv=none; b=XGxGDWqbDi5VRt31cPM2sj4ZbhZgZihKxgN3rnwfPEeu3N1xcq7wPSUv3JbGF+SwPEGH2fkyocjs6fJDM6cbcNKNeKtzTQ/cVqPS5Surv3Ucuiu+YyPC7JFWXFfvHNr2L3CEVqDDTHjyuCO1UWZ/GHfdfX3DHWFYFT0gxR3EkEA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735830146; c=relaxed/simple; bh=pAiIzXceRjp6FxVnLb+1NpJxcsd0OOyzRdXDgocx5VE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=rjReXWtXx8T0zHAXHnQM+uWBrWBebO9A7Bilom401gixLHk9ufO5HmxmBpbWwJPKRGISOAnB6JldyzUX7QYE7C9w8RPdCK1PaNnwyPAyZZ6qxxHhIzKWAR7hKw097DFW3nrV0Z6hkuTY5AQJZX3ae9333SihT7AUwqdRdle6vp8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=oKX9u/P9; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="oKX9u/P9" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 068C1C4CED7; Thu, 2 Jan 2025 15:02:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1735830146; bh=pAiIzXceRjp6FxVnLb+1NpJxcsd0OOyzRdXDgocx5VE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=oKX9u/P9iy4qqcOJA5BOhYzW+q3TLEVQNBUl8oJG2Gzo+ip+mrH087BUxWnV5TBUY r70G0h/sR53kXvwqtU8ofZbF6uxZYAfz37EF8JpOez7Xe99YgHdmCPu6G5excX89lf 2Lj+s8Lca6rUCAux9vKIcTksItE6L0CCflAQEzg8JXL/1mPk1i4/R2ksMRUfxCx6oT L69C27KHWDH8Hmodxp0M5rxRSexDw7ZIzyWOSzxjYpBcnOMepkMfvIfNiFvceAB89t cyYVKu2V0IMz1n4AI2mWT6RkVSxk2t5Fzxi0tQgp3vIj3ZRAppLKjHHzIzVFzgXEv7 x5JICyLPpD6dQ== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Peter Zijlstra , "Rafael J . Wysocki" , Daniel Lezcano , linux-pm@vger.kernel.org, Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen Subject: [PATCH 6/6] cpuidle: Handle TIF_NR_POLLING on behalf of software polling idle states Date: Thu, 2 Jan 2025 16:02:00 +0100 Message-ID: <20250102150201.21639-7-frederic@kernel.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20250102150201.21639-1-frederic@kernel.org> References: <20250102150201.21639-1-frederic@kernel.org> Precedence: bulk X-Mailing-List: linux-pm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Software polling idle states set again TIF_NR_POLLING and clear it upon exit. This involves error prone duplicated code and wasted cycles performing atomic operations, sometimes RmW fully ordered. To avoid this, benefit instead from the same generic TIF_NR_POLLING handling that is currently in use for hardware monitoring states. Signed-off-by: Frederic Weisbecker --- drivers/cpuidle/cpuidle-powernv.c | 10 ---------- drivers/cpuidle/cpuidle-pseries.c | 11 ----------- drivers/cpuidle/cpuidle.c | 2 +- drivers/cpuidle/poll_state.c | 30 ++++++++++++------------------ 4 files changed, 13 insertions(+), 40 deletions(-) diff --git a/drivers/cpuidle/cpuidle-powernv.c b/drivers/cpuidle/cpuidle-powernv.c index 9ebedd972df0..1bf0d2234016 100644 --- a/drivers/cpuidle/cpuidle-powernv.c +++ b/drivers/cpuidle/cpuidle-powernv.c @@ -71,8 +71,6 @@ static int snooze_loop(struct cpuidle_device *dev, { u64 snooze_exit_time; - set_thread_flag(TIF_POLLING_NRFLAG); - local_irq_enable(); snooze_exit_time = get_tb() + get_snooze_timeout(dev, drv, index); @@ -81,21 +79,13 @@ static int snooze_loop(struct cpuidle_device *dev, HMT_very_low(); while (!need_resched()) { if (likely(snooze_timeout_en) && get_tb() > snooze_exit_time) { - /* - * Task has not woken up but we are exiting the polling - * loop anyway. Require a barrier after polling is - * cleared to order subsequent test of need_resched(). - */ - clear_thread_flag(TIF_POLLING_NRFLAG); dev->poll_time_limit = true; - smp_mb(); break; } } HMT_medium(); ppc64_runlatch_on(); - clear_thread_flag(TIF_POLLING_NRFLAG); local_irq_disable(); diff --git a/drivers/cpuidle/cpuidle-pseries.c b/drivers/cpuidle/cpuidle-pseries.c index f68c65f1d023..704bb01d9e9e 100644 --- a/drivers/cpuidle/cpuidle-pseries.c +++ b/drivers/cpuidle/cpuidle-pseries.c @@ -40,8 +40,6 @@ int snooze_loop(struct cpuidle_device *dev, struct cpuidle_driver *drv, { u64 snooze_exit_time; - set_thread_flag(TIF_POLLING_NRFLAG); - pseries_idle_prolog(); raw_local_irq_enable(); snooze_exit_time = get_tb() + snooze_timeout; @@ -51,21 +49,12 @@ int snooze_loop(struct cpuidle_device *dev, struct cpuidle_driver *drv, HMT_low(); HMT_very_low(); if (likely(snooze_timeout_en) && get_tb() > snooze_exit_time) { - /* - * Task has not woken up but we are exiting the polling - * loop anyway. Require a barrier after polling is - * cleared to order subsequent test of need_resched(). - */ dev->poll_time_limit = true; - clear_thread_flag(TIF_POLLING_NRFLAG); - smp_mb(); break; } } HMT_medium(); - clear_thread_flag(TIF_POLLING_NRFLAG); - raw_local_irq_disable(); pseries_idle_epilog(); diff --git a/drivers/cpuidle/cpuidle.c b/drivers/cpuidle/cpuidle.c index 46c0a2726f67..fecc50c2860e 100644 --- a/drivers/cpuidle/cpuidle.c +++ b/drivers/cpuidle/cpuidle.c @@ -237,7 +237,7 @@ noinstr int cpuidle_enter_state(struct cpuidle_device *dev, broadcast = false; } - polling = target_state->flags & CPUIDLE_FLAG_MWAIT; + polling = target_state->flags & (CPUIDLE_FLAG_MWAIT | CPUIDLE_FLAG_POLLING); /* * If the target state doesn't poll on need_resched(), this is diff --git a/drivers/cpuidle/poll_state.c b/drivers/cpuidle/poll_state.c index 9b6d90a72601..d69936e2517e 100644 --- a/drivers/cpuidle/poll_state.c +++ b/drivers/cpuidle/poll_state.c @@ -13,35 +13,29 @@ static int __cpuidle poll_idle(struct cpuidle_device *dev, struct cpuidle_driver *drv, int index) { - u64 time_start; - - time_start = local_clock_noinstr(); + u64 time_start = local_clock_noinstr(); + unsigned int loop_count = 0; + u64 limit; dev->poll_time_limit = false; raw_local_irq_enable(); - if (!current_set_polling_and_test()) { - unsigned int loop_count = 0; - u64 limit; - limit = cpuidle_poll_time(drv, dev); + limit = cpuidle_poll_time(drv, dev); - while (!need_resched()) { - cpu_relax(); - if (loop_count++ < POLL_IDLE_RELAX_COUNT) - continue; + while (!need_resched()) { + cpu_relax(); + if (loop_count++ < POLL_IDLE_RELAX_COUNT) + continue; - loop_count = 0; - if (local_clock_noinstr() - time_start > limit) { - dev->poll_time_limit = true; - break; - } + loop_count = 0; + if (local_clock_noinstr() - time_start > limit) { + dev->poll_time_limit = true; + break; } } raw_local_irq_disable(); - current_clr_polling(); - return index; }