From patchwork Wed Aug 31 18:07:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 12961204 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C6625ECAAD4 for ; Wed, 31 Aug 2022 18:08:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232269AbiHaSIV (ORCPT ); Wed, 31 Aug 2022 14:08:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42980 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232226AbiHaSIM (ORCPT ); Wed, 31 Aug 2022 14:08:12 -0400 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 75BE3E397F; Wed, 31 Aug 2022 11:08:10 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id BE28ACE1EA8; Wed, 31 Aug 2022 18:08:08 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3734EC433C1; Wed, 31 Aug 2022 18:08:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1661969287; bh=HYR3hoM0Ydmx6dDsk73iVgFWt9139ZE43iky0FI7k8o=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=E4KXR+uCJLQzTIhgXky3CODmPEstNfZhZHCvQc5yWFxrM6zvl0RBWLUzxESa3wYZ2 YhFK0UWPGkcmjedf+yzkE2rV14QXLv5YzDRf6cM0h7jfHYNWctzOXXef8mtdFarbH0 kq9QtElnBHr04r4/z4SSfRdrt/qUm/n6ynQVpf4KIQsYErvoDuso7RBfCv1caMzO/f KbzprbFI84uo9PpE2q9VPhVcF00bu2YCYYO4te+9nTsB3KTLSMzRhtVQoE3YK28Vpq WW7VjJovstgQSgzFMp4iBNFNtk08qyxSki8jjHre15aMk8Z/OpTeeDmEdVHcjx4vV9 S3m3viBbKL7TQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id EAC195C015D; Wed, 31 Aug 2022 11:08:06 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, Zqiang , "Paul E . McKenney" Subject: [PATCH rcu 01/10] rcu: Fix rcu_read_unlock_strict() strict QS reporting Date: Wed, 31 Aug 2022 11:07:56 -0700 Message-Id: <20220831180805.2693546-1-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220831180759.GA2693289@paulmck-ThinkPad-P17-Gen-1> References: <20220831180759.GA2693289@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org From: Zqiang Kernels built with CONFIG_PREEMPT=n and CONFIG_RCU_STRICT_GRACE_PERIOD=y report the quiescent state directly from the outermost rcu_read_unlock(). However, the current CPU's rcu_data structure's ->cpu_no_qs.b.norm might still be set, in which case rcu_report_qs_rdp() will exit early, thus failing to report quiescent state. This commit therefore causes rcu_read_unlock_strict() to clear CPU's rcu_data structure's ->cpu_no_qs.b.norm field before invoking rcu_report_qs_rdp(). Signed-off-by: Zqiang Signed-off-by: Paul E. McKenney --- kernel/rcu/tree_plugin.h | 1 + 1 file changed, 1 insertion(+) diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h index 438ecae6bd7e7..86772c95ed0ae 100644 --- a/kernel/rcu/tree_plugin.h +++ b/kernel/rcu/tree_plugin.h @@ -824,6 +824,7 @@ void rcu_read_unlock_strict(void) if (irqs_disabled() || preempt_count() || !rcu_state.gp_kthread) return; rdp = this_cpu_ptr(&rcu_data); + rdp->cpu_no_qs.b.norm = false; rcu_report_qs_rdp(rdp); udelay(rcu_unlock_delay); } From patchwork Wed Aug 31 18:07:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 12961198 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D0351ECAAD3 for ; Wed, 31 Aug 2022 18:08:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232268AbiHaSIN (ORCPT ); Wed, 31 Aug 2022 14:08:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42914 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232131AbiHaSIL (ORCPT ); Wed, 31 Aug 2022 14:08:11 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 40B06E1AA7; Wed, 31 Aug 2022 11:08:08 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id D522161C14; Wed, 31 Aug 2022 18:08:07 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 41D48C433D7; Wed, 31 Aug 2022 18:08:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1661969287; bh=IINFzSXV050MZ7p39aFiCw3ksGDcAruZYr0+y+xvqvE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=nitWNXI+In3pEwI/3JSH9oJN/skv/4jMUZfznb0dIKMjmHkxKYHR0NYhuQ+sWQHmm dm3GRPHd7xNUr3NUzGn1dgxFuZcZbrub5L5XCOvSafld81VyrIWWH0TjtqDMwr8Fzo vKSF1F64wiz6kbXk+fX5kZlEm3Q4rCwOeX733xQd5Mx3rYufF6mpDQAoY86d39QUDZ pUz88AwlWFBSyw4oJdmB2s7guG1pYH8pbd5y8nO0tD9DTMOVDxQQOzBYoWvPWsEzwM d8vPu3GMorxk2sYeWSDX1cfaiMgRApkc+NF2I3trBJyWyKNqtujMzlawVUCfkIynq8 eLqnRSyz6x9XA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id ECEAA5C019C; Wed, 31 Aug 2022 11:08:06 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, Zqiang , "Paul E . McKenney" Subject: [PATCH rcu 02/10] rcu: Update rcu_preempt_deferred_qs() comments for !PREEMPT kernels Date: Wed, 31 Aug 2022 11:07:57 -0700 Message-Id: <20220831180805.2693546-2-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220831180759.GA2693289@paulmck-ThinkPad-P17-Gen-1> References: <20220831180759.GA2693289@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org From: Zqiang In non-premptible kernels, tasks never do context switches within RCU read-side critical sections. Therefore, in such kernels, each leaf rcu_node structure's ->blkd_tasks list will always be empty. The comment on the non-preemptible version of rcu_preempt_deferred_qs() confuses this point, so this commit therefore fixes it. Signed-off-by: Zqiang Signed-off-by: Paul E. McKenney --- kernel/rcu/tree_plugin.h | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h index 86772c95ed0ae..4152816dd29f6 100644 --- a/kernel/rcu/tree_plugin.h +++ b/kernel/rcu/tree_plugin.h @@ -932,10 +932,13 @@ static notrace bool rcu_preempt_need_deferred_qs(struct task_struct *t) return false; } -// Except that we do need to respond to a request by an expedited grace -// period for a quiescent state from this CPU. Note that requests from -// tasks are handled when removing the task from the blocked-tasks list -// below. +// Except that we do need to respond to a request by an expedited +// grace period for a quiescent state from this CPU. Note that in +// non-preemptible kernels, there can be no context switches within RCU +// read-side critical sections, which in turn means that the leaf rcu_node +// structure's blocked-tasks list is always empty. is therefore no need to +// actually check it. Instead, a quiescent state from this CPU suffices, +// and this function is only called from such a quiescent state. notrace void rcu_preempt_deferred_qs(struct task_struct *t) { struct rcu_data *rdp = this_cpu_ptr(&rcu_data); From patchwork Wed Aug 31 18:07:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 12961202 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 77386ECAAD4 for ; Wed, 31 Aug 2022 18:08:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232324AbiHaSIS (ORCPT ); Wed, 31 Aug 2022 14:08:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42976 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232204AbiHaSIM (ORCPT ); Wed, 31 Aug 2022 14:08:12 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EEC33E3962; Wed, 31 Aug 2022 11:08:09 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 75DB2B8220A; Wed, 31 Aug 2022 18:08:08 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3C62CC433D6; Wed, 31 Aug 2022 18:08:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1661969287; bh=UBju2okSl8/q5Sdd+pxIbphkj3m0c6OMyYxyLduTv9E=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=bHZVuyX6uOE/uI7HlKdl9Z73vVGpgsOpKfsNmSa4xB1ZpBGOqJ5M+c7sMI8DequyL Zc9QkNuWSu9O+bYt3RdISY24OjXpiFrQod3PpOZP4QTCseEXzuit6NPEPPo5Vg6hoc 2hGfvngdCFKo8lGD+16cxuQofqSijU9f5a1l3rNbcRQAWoberduJKyP/WG8iazseZ2 P62gCmdIn9ZzTJiyDGyGS2469Jo3U6icjvxNUUnMnZerRDVdTXpNdg1C2j51QbLphS TNWNjgkeOqWnB6gkeD4rmETuDdL//1+hKFJ6S0LfQ/ob/vspm5+2l+LBNn4h12FViO 22DSDHw6ZjysQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id EEDB95C02A9; Wed, 31 Aug 2022 11:08:06 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, Zqiang , "Paul E . McKenney" Subject: [PATCH rcu 03/10] rcu: Add QS check in rcu_exp_handler() for non-preemptible kernels Date: Wed, 31 Aug 2022 11:07:58 -0700 Message-Id: <20220831180805.2693546-3-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220831180759.GA2693289@paulmck-ThinkPad-P17-Gen-1> References: <20220831180759.GA2693289@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org From: Zqiang Kernels built with CONFIG_PREEMPTION=n and CONFIG_PREEMPT_COUNT=y maintain preempt_count() state. Because such kernels map __rcu_read_lock() and __rcu_read_unlock() to preempt_disable() and preempt_enable(), respectively, this allows the expedited grace period's !CONFIG_PREEMPT_RCU version of the rcu_exp_handler() IPI handler function to use preempt_count() to detect quiescent states. This preempt_count() usage might seem to risk failures due to use of implicit RCU readers in portions of the kernel under #ifndef CONFIG_PREEMPTION, except that rcu_core() already disallows such implicit RCU readers. The moral of this story is that you must use explicit read-side markings such as rcu_read_lock() or preempt_disable() even if the code knows that this kernel does not support preemption. This commit therefore adds a preempt_count()-based check for a quiescent state in the !CONFIG_PREEMPT_RCU version of the rcu_exp_handler() function for kernels built with CONFIG_PREEMPT_COUNT=y, reporting an immediate quiescent state when the interrupted code had both preemption and softirqs enabled. This change results in about a 2% reduction in expedited grace-period latency in kernels built with both CONFIG_PREEMPT_RCU=n and CONFIG_PREEMPT_COUNT=y. Signed-off-by: Zqiang Signed-off-by: Paul E. McKenney Link: https://lore.kernel.org/all/20220622103549.2840087-1-qiang1.zhang@intel.com/ --- kernel/rcu/tree_exp.h | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h index be667583a5547..b07998159d1fa 100644 --- a/kernel/rcu/tree_exp.h +++ b/kernel/rcu/tree_exp.h @@ -828,11 +828,13 @@ static void rcu_exp_handler(void *unused) { struct rcu_data *rdp = this_cpu_ptr(&rcu_data); struct rcu_node *rnp = rdp->mynode; + bool preempt_bh_enabled = !(preempt_count() & (PREEMPT_MASK | SOFTIRQ_MASK)); if (!(READ_ONCE(rnp->expmask) & rdp->grpmask) || __this_cpu_read(rcu_data.cpu_no_qs.b.exp)) return; - if (rcu_is_cpu_rrupt_from_idle()) { + if (rcu_is_cpu_rrupt_from_idle() || + (IS_ENABLED(CONFIG_PREEMPT_COUNT) && preempt_bh_enabled)) { rcu_report_exp_rdp(this_cpu_ptr(&rcu_data)); return; } From patchwork Wed Aug 31 18:07:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 12961197 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 97D3EECAAD1 for ; Wed, 31 Aug 2022 18:08:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232144AbiHaSIM (ORCPT ); Wed, 31 Aug 2022 14:08:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42912 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232127AbiHaSIL (ORCPT ); Wed, 31 Aug 2022 14:08:11 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3DB8FE1902; Wed, 31 Aug 2022 11:08:08 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id D017161BF0; Wed, 31 Aug 2022 18:08:07 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3F270C433B5; Wed, 31 Aug 2022 18:08:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1661969287; bh=TVPaGAMvm71KkeIDiWEUCZyNJ+spX0wF8WrKcuO37bg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=PepPYCxiB3sPg2EwiI5jDe0jPLlsz2hApLeQU2kOk1kfjE+gXRui8Dqj/cTpxYt90 nx4yRsEYmyKQp1tfiImfJDTSlY6zggmrFZLJSUR5qmY66OKApDI/V1djbYPdJpqRRY OQ2KhD1I2enuAFOj8UjxIHQzefNMIelwi6bcZj/8EMNJZlKcJotxdXnWDlwE3jf8yn O7kHkb5KykmB5s4rzF7gi5VvjX9ASqvyHBaIWVTXbQkiiCGJpSsU5RBlEVgFSgApaz T+XdBl2lnkYP2DYxUiAaqcZMlRTT3AJGYc4axSp50TlklMNXvgElwCCnAhwRKxQaGq 9yNj6y+JSlWfw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id F0BF15C0513; Wed, 31 Aug 2022 11:08:06 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, Zqiang , "Paul E . McKenney" Subject: [PATCH rcu 04/10] rcu: Make tiny RCU support leak callbacks for debug-object errors Date: Wed, 31 Aug 2022 11:07:59 -0700 Message-Id: <20220831180805.2693546-4-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220831180759.GA2693289@paulmck-ThinkPad-P17-Gen-1> References: <20220831180759.GA2693289@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org From: Zqiang Currently, only Tree RCU leaks callbacks setting when it detects a duplicate call_rcu(). This commit causes Tiny RCU to also leak callbacks in this situation. Because this is Tiny RCU, kernel size is important: 1. CONFIG_TINY_RCU=y and CONFIG_DEBUG_OBJECTS_RCU_HEAD=n (Production kernel) Original: text data bss dec hex filename 26290663 20159823 15212544 61663030 3ace736 vmlinux With this commit: text data bss dec hex filename 26290663 20159823 15212544 61663030 3ace736 vmlinux 2. CONFIG_TINY_RCU=y and CONFIG_DEBUG_OBJECTS_RCU_HEAD=y (Debugging kernel) Original: text data bss dec hex filename 26291319 20160143 15212544 61664006 3aceb06 vmlinux With this commit: text data bss dec hex filename 26291319 20160431 15212544 61664294 3acec26 vmlinux These results show that the kernel size is unchanged for production kernels, as desired. Signed-off-by: Zqiang Signed-off-by: Paul E. McKenney --- kernel/rcu/tiny.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/tiny.c b/kernel/rcu/tiny.c index f0561ee16b9c2..943d431b908f6 100644 --- a/kernel/rcu/tiny.c +++ b/kernel/rcu/tiny.c @@ -158,6 +158,10 @@ void synchronize_rcu(void) } EXPORT_SYMBOL_GPL(synchronize_rcu); +static void tiny_rcu_leak_callback(struct rcu_head *rhp) +{ +} + /* * Post an RCU callback to be invoked after the end of an RCU grace * period. But since we have but one CPU, that would be after any @@ -165,9 +169,20 @@ EXPORT_SYMBOL_GPL(synchronize_rcu); */ void call_rcu(struct rcu_head *head, rcu_callback_t func) { + static atomic_t doublefrees; unsigned long flags; - debug_rcu_head_queue(head); + if (debug_rcu_head_queue(head)) { + if (atomic_inc_return(&doublefrees) < 4) { + pr_err("%s(): Double-freed CB %p->%pS()!!! ", __func__, head, head->func); + mem_dump_obj(head); + } + + if (!__is_kvfree_rcu_offset((unsigned long)head->func)) + WRITE_ONCE(head->func, tiny_rcu_leak_callback); + return; + } + head->func = func; head->next = NULL; From patchwork Wed Aug 31 18:08:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 12961203 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5B706ECAAD3 for ; Wed, 31 Aug 2022 18:08:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232333AbiHaSIT (ORCPT ); Wed, 31 Aug 2022 14:08:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42978 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232221AbiHaSIM (ORCPT ); Wed, 31 Aug 2022 14:08:12 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 09295E396C; Wed, 31 Aug 2022 11:08:10 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 98B2DB82012; Wed, 31 Aug 2022 18:08:08 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 443AAC4347C; Wed, 31 Aug 2022 18:08:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1661969287; bh=VSLfzxIxzjNIGO2+Wdg4oAuIvSgu0DsDoavaH3Q4Te4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Ms2NOJdXRA75zhlq+5y+vkC2qiJDRLx4i6jvKaigRKsqTdjH+Spxh6EZjsthBrHC3 SMFMeHSBDMkvA1FvD/fjrAzO81MFv+NGEYzsJgbPfWurOewIvU2PG+kSCyLHCZ2Ec9 OSbdcCz3jH28BheGzxAPp/3EP4ERKU7NM07WSvEnZXWqgrxWfCmB19rg+WoN8R+fpP ZV/TPUPAXL/ZKyMiXJLZz8uYyjLJhAsXEjNkRamn1cCpzLliHjsGpL3nYfB0r5fexS aaf16vAqsKXxzNSNWfPj9IymKt4WRWskQRikj5VEy4G9y7BaY+p4RLcSq5MxBZqTk2 ryvVYG/9RnWRg== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id F2A6D5C06A7; Wed, 31 Aug 2022 11:08:06 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" , Neeraj Upadhyay , Boqun Feng , Frederic Weisbecker Subject: [PATCH rcu 05/10] rcu: Document reason for rcu_all_qs() call to preempt_disable() Date: Wed, 31 Aug 2022 11:08:00 -0700 Message-Id: <20220831180805.2693546-5-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220831180759.GA2693289@paulmck-ThinkPad-P17-Gen-1> References: <20220831180759.GA2693289@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org Given that rcu_all_qs() is in non-preemptible kernels, why on earth should it invoke preempt_disable()? This commit adds the reason, which is to work nicely with debugging enabled in CONFIG_PREEMPT_COUNT=y kernels. Reported-by: Neeraj Upadhyay Reported-by: Boqun Feng Reported-by: Frederic Weisbecker Signed-off-by: Paul E. McKenney --- kernel/rcu/tree_plugin.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h index 4152816dd29f6..c46b3c74dad1f 100644 --- a/kernel/rcu/tree_plugin.h +++ b/kernel/rcu/tree_plugin.h @@ -870,7 +870,7 @@ void rcu_all_qs(void) if (!raw_cpu_read(rcu_data.rcu_urgent_qs)) return; - preempt_disable(); + preempt_disable(); // For CONFIG_PREEMPT_COUNT=y kernels /* Load rcu_urgent_qs before other flags. */ if (!smp_load_acquire(this_cpu_ptr(&rcu_data.rcu_urgent_qs))) { preempt_enable(); From patchwork Wed Aug 31 18:08:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 12961205 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C54FAECAAD3 for ; Wed, 31 Aug 2022 18:08:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232296AbiHaSIV (ORCPT ); Wed, 31 Aug 2022 14:08:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42944 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232262AbiHaSIN (ORCPT ); Wed, 31 Aug 2022 14:08:13 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 759D2E396E; Wed, 31 Aug 2022 11:08:10 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 24AABB82271; Wed, 31 Aug 2022 18:08:09 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9292BC43146; Wed, 31 Aug 2022 18:08:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1661969287; bh=cuZvf2I96/EDoARq2cU0qVdQPo0+rYUi76tBGq7KBaI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=MLFXKa0NyXxac//Joj5H/FBsrhbUeAfIMRkMIVWysxNVppG0h38AQsEQdhXqJ7EjM W65nDQ/bger9XUj3e2zoxMmKUoueNOmfjCFi2FwPIpYjYtRqZv+7QWFpYnQJVe92r7 gPHoExaZJrPKySqq2mHVvLnYFTyBP//o/xa9sfFHrig2YyJ9L89NnGgnkAoX1J34SF 6f7ELKE0SMR0sOPn8+dDJnVKo/V+z17lFSUmoEzkl7fhzbyHJxTOZszJ9Mjduy0grd 9eNcFLYSL6gaLR48YQ7FOBEb7n6Yi6DJAwHZUEc9Vb1VjiAAC74eKTOABUMWX2uJlR QBb/0UWIwAMoA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 0049C5C090A; Wed, 31 Aug 2022 11:08:06 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" , Maxim Mikityanskiy Subject: [PATCH rcu 06/10] rcu: Update rcu_access_pointer() header for rcu_dereference_protected() Date: Wed, 31 Aug 2022 11:08:01 -0700 Message-Id: <20220831180805.2693546-6-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220831180759.GA2693289@paulmck-ThinkPad-P17-Gen-1> References: <20220831180759.GA2693289@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org The rcu_access_pointer() docbook header correctly notes that it may be used during post-grace-period teardown. However, it is usually better to use rcu_dereference_protected() for this purpose. This commit therefore calls out this preferred usage. Reported-by: Maxim Mikityanskiy Signed-off-by: Paul E. McKenney --- include/linux/rcupdate.h | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index f527f27e64387..61a1a85c720c3 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -496,13 +496,21 @@ do { \ * against NULL. Although rcu_access_pointer() may also be used in cases * where update-side locks prevent the value of the pointer from changing, * you should instead use rcu_dereference_protected() for this use case. + * Within an RCU read-side critical section, there is little reason to + * use rcu_access_pointer(). + * + * It is usually best to test the rcu_access_pointer() return value + * directly in order to avoid accidental dereferences being introduced + * by later inattentive changes. In other words, assigning the + * rcu_access_pointer() return value to a local variable results in an + * accident waiting to happen. * * It is also permissible to use rcu_access_pointer() when read-side - * access to the pointer was removed at least one grace period ago, as - * is the case in the context of the RCU callback that is freeing up - * the data, or after a synchronize_rcu() returns. This can be useful - * when tearing down multi-linked structures after a grace period - * has elapsed. + * access to the pointer was removed at least one grace period ago, as is + * the case in the context of the RCU callback that is freeing up the data, + * or after a synchronize_rcu() returns. This can be useful when tearing + * down multi-linked structures after a grace period has elapsed. However, + * rcu_dereference_protected() is normally preferred for this use case. */ #define rcu_access_pointer(p) __rcu_access_pointer((p), __UNIQUE_ID(rcu), __rcu) From patchwork Wed Aug 31 18:08:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 12961200 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9AA96ECAAD1 for ; Wed, 31 Aug 2022 18:08:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232292AbiHaSIQ (ORCPT ); Wed, 31 Aug 2022 14:08:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42938 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230249AbiHaSIL (ORCPT ); Wed, 31 Aug 2022 14:08:11 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D1616E3426; Wed, 31 Aug 2022 11:08:08 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 6EA4D61BFB; Wed, 31 Aug 2022 18:08:08 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 92A4FC43147; Wed, 31 Aug 2022 18:08:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1661969287; bh=/uL5W9eLHl+SeaTGYxjhOKKa+37RcoDRIc6rSK6aGYY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Ya7lrUH6BuPPFoMDt5cTFT8U63Rj19ssY0sIun5dQA7NevNz8LavQbqDHbSFtqfk3 RWbx1nPgIdMjxpfJuT5T1x6xNQFl9NtkJtKfdYgR1WZz6zbeYDBIq+9ZFzM77f/J6b vZ7iWq0FuBAh2NcunU4Hv2KLHTQdwli0b2VVdeMl0+zgUygvNZ0dgDe/m6KVTDJgNV IXq+p21r1Ww/B6U3q47VemMbwYWCwYNALdUgUP5b/zNeu3dpugBJrKqTRD85AncDmt fziOsWN4xSNi0kwUTiIk9tNZbNzlUzRBoBXhTnqHtDEusZHSOHt6RwuaB8CWP/ksf4 xiKWAZqiHP5aQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 022C75C0950; Wed, 31 Aug 2022 11:08:07 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, Zhen Lei , "Paul E . McKenney" , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira , Valentin Schneider Subject: [PATCH rcu 07/10] sched/debug: Try trigger_single_cpu_backtrace(cpu) in dump_cpu_task() Date: Wed, 31 Aug 2022 11:08:02 -0700 Message-Id: <20220831180805.2693546-7-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220831180759.GA2693289@paulmck-ThinkPad-P17-Gen-1> References: <20220831180759.GA2693289@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org From: Zhen Lei The trigger_all_cpu_backtrace() function attempts to send an NMI to the target CPU, which usually provides much better stack traces than the dump_cpu_task() function's approach of dumping that stack from some other CPU. So much so that most calls to dump_cpu_task() only happen after a call to trigger_all_cpu_backtrace() has failed. And the exception to this rule really should attempt to use trigger_all_cpu_backtrace() first. Therefore, move the trigger_all_cpu_backtrace() invocation into dump_cpu_task(). Signed-off-by: Zhen Lei Signed-off-by: Paul E. McKenney Cc: Ingo Molnar Cc: Peter Zijlstra Cc: Juri Lelli Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Ben Segall Cc: Mel Gorman Cc: Daniel Bristot de Oliveira Cc: Valentin Schneider --- kernel/rcu/tree_stall.h | 5 ++--- kernel/sched/core.c | 3 +++ kernel/smp.c | 3 +-- 3 files changed, 6 insertions(+), 5 deletions(-) diff --git a/kernel/rcu/tree_stall.h b/kernel/rcu/tree_stall.h index c3fbbcc09327f..5653560573e22 100644 --- a/kernel/rcu/tree_stall.h +++ b/kernel/rcu/tree_stall.h @@ -368,7 +368,7 @@ static void rcu_dump_cpu_stacks(void) if (rnp->qsmask & leaf_node_cpu_bit(rnp, cpu)) { if (cpu_is_offline(cpu)) pr_err("Offline CPU %d blocking current GP.\n", cpu); - else if (!trigger_single_cpu_backtrace(cpu)) + else dump_cpu_task(cpu); } raw_spin_unlock_irqrestore_rcu_node(rnp, flags); @@ -511,8 +511,7 @@ static void rcu_check_gp_kthread_starvation(void) pr_err("RCU GP kthread last ran on offline CPU %d.\n", cpu); } else { pr_err("Stack dump where RCU GP kthread last ran:\n"); - if (!trigger_single_cpu_backtrace(cpu)) - dump_cpu_task(cpu); + dump_cpu_task(cpu); } } wake_up_process(gpk); diff --git a/kernel/sched/core.c b/kernel/sched/core.c index ee28253c9ac0c..e15b6a7f34f47 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -11183,6 +11183,9 @@ struct cgroup_subsys cpu_cgrp_subsys = { void dump_cpu_task(int cpu) { + if (trigger_single_cpu_backtrace(cpu)) + return; + pr_info("Task dump for CPU %d:\n", cpu); sched_show_task(cpu_curr(cpu)); } diff --git a/kernel/smp.c b/kernel/smp.c index 650810a6f29b3..e8cdc025a046f 100644 --- a/kernel/smp.c +++ b/kernel/smp.c @@ -370,8 +370,7 @@ static bool csd_lock_wait_toolong(struct __call_single_data *csd, u64 ts0, u64 * if (cpu >= 0) { if (static_branch_unlikely(&csdlock_debug_extended)) csd_lock_print_extended(csd, cpu); - if (!trigger_single_cpu_backtrace(cpu)) - dump_cpu_task(cpu); + dump_cpu_task(cpu); if (!cpu_cur_csd) { pr_alert("csd: Re-sending CSD lock (#%d) IPI from CPU#%02d to CPU#%02d\n", *bug_id, raw_smp_processor_id(), cpu); arch_send_call_function_single_ipi(cpu); From patchwork Wed Aug 31 18:08:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 12961206 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 10437ECAAD1 for ; Wed, 31 Aug 2022 18:08:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232305AbiHaSIX (ORCPT ); Wed, 31 Aug 2022 14:08:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43010 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232286AbiHaSIO (ORCPT ); Wed, 31 Aug 2022 14:08:14 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 75D3FE3981; Wed, 31 Aug 2022 11:08:10 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 49E7AB82272; Wed, 31 Aug 2022 18:08:09 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 960CCC43143; Wed, 31 Aug 2022 18:08:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1661969287; bh=o9XgzRlPiWVyRgUYLtR8EAg++iHahsXyN3N9pj5GojQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=WSVrT/MWKdKtPeROyci6f6kHR0kAWFcDmm5cP62dDJshO+HkeLZA6HaQzBLIMXFnu aIYHb3pEzEsfpVCNRBDj66qJJJbuXS4o2ZXtrIK7t2sI9SCjQ8AvMYPzBIsTJtGUc5 ufEDfC2fGiRQFRiB7Ux7U4y1mprWuWVy3nX+95r8kDJ8OGwWaW6V1x1WOglPzR2DEb 6j+qx/YzPUiHHmPr46a9TNcegMGCP3EChFzOy8s5kgLu46OKVdwdv2V7n75BG+NKDP 9Iqkh1v4REhatHOawxTqDp84fSlal0j+r5pTIElFklBQw4WGmoPCagt1VSHSrcH3Pm 3pWG9g9xnGshA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 045DF5C0981; Wed, 31 Aug 2022 11:08:07 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, Zhen Lei , "Paul E . McKenney" , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira , Valentin Schneider Subject: [PATCH rcu 08/10] sched/debug: Show the registers of 'current' in dump_cpu_task() Date: Wed, 31 Aug 2022 11:08:03 -0700 Message-Id: <20220831180805.2693546-8-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220831180759.GA2693289@paulmck-ThinkPad-P17-Gen-1> References: <20220831180759.GA2693289@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org From: Zhen Lei The dump_cpu_task() function does not print registers on architectures that do not support NMIs. However, registers can be useful for debugging. Fortunately, in the case where dump_cpu_task() is invoked from an interrupt handler and is dumping the current CPU's stack, the get_irq_regs() function can be used to get the registers. Therefore, this commit makes dump_cpu_task() check to see if it is being asked to dump the current CPU's stack from within an interrupt handler, and, if so, it uses the get_irq_regs() function to obtain the registers. On systems that do support NMIs, this commit has the further advantage of avoiding a self-NMI in this case. This is an example of rcu self-detected stall on arm64, which does not support NMIs: [ 27.501721] rcu: INFO: rcu_preempt self-detected stall on CPU [ 27.502238] rcu: 0-....: (1250 ticks this GP) idle=4f7/1/0x4000000000000000 softirq=2594/2594 fqs=619 [ 27.502632] (t=1251 jiffies g=2989 q=29 ncpus=4) [ 27.503845] CPU: 0 PID: 306 Comm: test0 Not tainted 5.19.0-rc7-00009-g1c1a6c29ff99-dirty #46 [ 27.504732] Hardware name: linux,dummy-virt (DT) [ 27.504947] pstate: 20000005 (nzCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 27.504998] pc : arch_counter_read+0x18/0x24 [ 27.505301] lr : arch_counter_read+0x18/0x24 [ 27.505328] sp : ffff80000b29bdf0 [ 27.505345] x29: ffff80000b29bdf0 x28: 0000000000000000 x27: 0000000000000000 [ 27.505475] x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000 [ 27.505553] x23: 0000000000001f40 x22: ffff800009849c48 x21: 000000065f871ae0 [ 27.505627] x20: 00000000000025ec x19: ffff80000a6eb300 x18: ffffffffffffffff [ 27.505654] x17: 0000000000000001 x16: 0000000000000000 x15: ffff80000a6d0296 [ 27.505681] x14: ffffffffffffffff x13: ffff80000a29bc18 x12: 0000000000000426 [ 27.505709] x11: 0000000000000162 x10: ffff80000a2f3c18 x9 : ffff80000a29bc18 [ 27.505736] x8 : 00000000ffffefff x7 : ffff80000a2f3c18 x6 : 00000000759bd013 [ 27.505761] x5 : 01ffffffffffffff x4 : 0002dc6c00000000 x3 : 0000000000000017 [ 27.505787] x2 : 00000000000025ec x1 : ffff80000b29bdf0 x0 : 0000000075a30653 [ 27.505937] Call trace: [ 27.506002] arch_counter_read+0x18/0x24 [ 27.506171] ktime_get+0x48/0xa0 [ 27.506207] test_task+0x70/0xf0 [ 27.506227] kthread+0x10c/0x110 [ 27.506243] ret_from_fork+0x10/0x20 This is a marked improvement over the old output: [ 27.944550] rcu: INFO: rcu_preempt self-detected stall on CPU [ 27.944980] rcu: 0-....: (1249 ticks this GP) idle=cbb/1/0x4000000000000000 softirq=2610/2610 fqs=614 [ 27.945407] (t=1251 jiffies g=2681 q=28 ncpus=4) [ 27.945731] Task dump for CPU 0: [ 27.945844] task:test0 state:R running task stack: 0 pid: 306 ppid: 2 flags:0x0000000a [ 27.946073] Call trace: [ 27.946151] dump_backtrace.part.0+0xc8/0xd4 [ 27.946378] show_stack+0x18/0x70 [ 27.946405] sched_show_task+0x150/0x180 [ 27.946427] dump_cpu_task+0x44/0x54 [ 27.947193] rcu_dump_cpu_stacks+0xec/0x130 [ 27.947212] rcu_sched_clock_irq+0xb18/0xef0 [ 27.947231] update_process_times+0x68/0xac [ 27.947248] tick_sched_handle+0x34/0x60 [ 27.947266] tick_sched_timer+0x4c/0xa4 [ 27.947281] __hrtimer_run_queues+0x178/0x360 [ 27.947295] hrtimer_interrupt+0xe8/0x244 [ 27.947309] arch_timer_handler_virt+0x38/0x4c [ 27.947326] handle_percpu_devid_irq+0x88/0x230 [ 27.947342] generic_handle_domain_irq+0x2c/0x44 [ 27.947357] gic_handle_irq+0x44/0xc4 [ 27.947376] call_on_irq_stack+0x2c/0x54 [ 27.947415] do_interrupt_handler+0x80/0x94 [ 27.947431] el1_interrupt+0x34/0x70 [ 27.947447] el1h_64_irq_handler+0x18/0x24 [ 27.947462] el1h_64_irq+0x64/0x68 <--- the above backtrace is worthless [ 27.947474] arch_counter_read+0x18/0x24 [ 27.947487] ktime_get+0x48/0xa0 [ 27.947501] test_task+0x70/0xf0 [ 27.947520] kthread+0x10c/0x110 [ 27.947538] ret_from_fork+0x10/0x20 Signed-off-by: Zhen Lei Signed-off-by: Paul E. McKenney Cc: Ingo Molnar Cc: Peter Zijlstra Cc: Juri Lelli Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Ben Segall Cc: Mel Gorman Cc: Daniel Bristot de Oliveira Cc: Valentin Schneider --- kernel/sched/core.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index e15b6a7f34f47..60fdc0faf1c9d 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -73,6 +73,7 @@ #include +#include #include #include @@ -11183,6 +11184,16 @@ struct cgroup_subsys cpu_cgrp_subsys = { void dump_cpu_task(int cpu) { + if (cpu == smp_processor_id() && in_hardirq()) { + struct pt_regs *regs; + + regs = get_irq_regs(); + if (regs) { + show_regs(regs); + return; + } + } + if (trigger_single_cpu_backtrace(cpu)) return; From patchwork Wed Aug 31 18:08:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 12961201 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D4D82C0502A for ; Wed, 31 Aug 2022 18:08:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232115AbiHaSIR (ORCPT ); Wed, 31 Aug 2022 14:08:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42946 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232141AbiHaSIL (ORCPT ); Wed, 31 Aug 2022 14:08:11 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 24C49E3943; Wed, 31 Aug 2022 11:08:09 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 9A03161C19; Wed, 31 Aug 2022 18:08:08 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9652FC4314A; Wed, 31 Aug 2022 18:08:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1661969287; bh=DfIvCam7iGu1hPFxklqGzZcjvwrU/yGB++vrN26Fk/w=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=usadxwQdhiAWoS7zwJlmjAdFT4pokIH1Cu8Mo3ttdlp2zJrrjagbSh+VdgzIfZwN8 InzRn01jWX3tPupYsaaLmuwGw7LTxgI5+yjriSfPlpihQdwzQGZ55JMcBgoliwX1KC ebuJsMurLgWMi7cNPHLftgYsD55dED0XcKw/WCYowCbLPsIu75PchFH9Oiv1yCdi8F pQGonP9uEKTXvG6O64VoQcUAV4RAzrTgwFbjvzakmuCmhlFr0oQq2/0U0MqLFeZEmw hnWwykXblNuPhnetUaYWUskEo+UuTbY2FTMAOl4BDAvUeW7/lnso5+Ci0QWVfYU+ic g1ZO69gIeugxQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 061E25C09C2; Wed, 31 Aug 2022 11:08:07 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, Zqiang , "Paul E . McKenney" Subject: [PATCH rcu 09/10] rcu: Avoid triggering strict-GP irq-work when RCU is idle Date: Wed, 31 Aug 2022 11:08:04 -0700 Message-Id: <20220831180805.2693546-9-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220831180759.GA2693289@paulmck-ThinkPad-P17-Gen-1> References: <20220831180759.GA2693289@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org From: Zqiang Kernels built with PREEMPT_RCU=y and RCU_STRICT_GRACE_PERIOD=y trigger irq-work from rcu_read_unlock(), and the resulting irq-work handler invokes rcu_preempt_deferred_qs_handle(). The point of this triggering is to force grace periods to end quickly in order to give tools like KASAN a better chance of detecting RCU usage bugs such as leaking RCU-protected pointers out of an RCU read-side critical section. However, this irq-work triggering is unconditional. This works, but there is no point in doing this irq-work unless the current grace period is waiting on the running CPU or task, which is not the common case. After all, in the common case there are many rcu_read_unlock() calls per CPU per grace period. This commit therefore triggers the irq-work only when the current grace period is waiting on the running CPU or task. This change was tested as follows on a four-CPU system: echo rcu_preempt_deferred_qs_handler > /sys/kernel/debug/tracing/set_ftrace_filter echo 1 > /sys/kernel/debug/tracing/function_profile_enabled insmod rcutorture.ko sleep 20 rmmod rcutorture.ko echo 0 > /sys/kernel/debug/tracing/function_profile_enabled echo > /sys/kernel/debug/tracing/set_ftrace_filter This procedure produces results in this per-CPU set of files: /sys/kernel/debug/tracing/trace_stat/function* Sample output from one of these files is as follows: Function Hit Time Avg s^2 -------- --- ---- --- --- rcu_preempt_deferred_qs_handle 838746 182650.3 us 0.217 us 0.004 us The baseline sum of the "Hit" values (the number of calls to this function) was 3,319,015. With this commit, that sum was 1,140,359, for a 2.9x reduction. The worst-case variance across the CPUs was less than 25%, so this large effect size is statistically significant. The raw data is available in the Link: URL. Link: https://lore.kernel.org/all/20220808022626.12825-1-qiang1.zhang@intel.com/ Signed-off-by: Zqiang Signed-off-by: Paul E. McKenney --- kernel/rcu/tree_plugin.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h index c46b3c74dad1f..207617f69aa56 100644 --- a/kernel/rcu/tree_plugin.h +++ b/kernel/rcu/tree_plugin.h @@ -641,7 +641,8 @@ static void rcu_read_unlock_special(struct task_struct *t) expboost = (t->rcu_blocked_node && READ_ONCE(t->rcu_blocked_node->exp_tasks)) || (rdp->grpmask & READ_ONCE(rnp->expmask)) || - IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD) || + (IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD) && + ((rdp->grpmask & READ_ONCE(rnp->qsmask)) || t->rcu_blocked_node)) || (IS_ENABLED(CONFIG_RCU_BOOST) && irqs_were_disabled && t->rcu_blocked_node); // Need to defer quiescent state until everything is enabled. From patchwork Wed Aug 31 18:08:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 12961199 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0A0F1ECAAD4 for ; Wed, 31 Aug 2022 18:08:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232114AbiHaSIO (ORCPT ); Wed, 31 Aug 2022 14:08:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42924 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232140AbiHaSIL (ORCPT ); Wed, 31 Aug 2022 14:08:11 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CF73BE3424; Wed, 31 Aug 2022 11:08:08 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 6B3DB61BF8; Wed, 31 Aug 2022 18:08:08 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 963EDC43149; Wed, 31 Aug 2022 18:08:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1661969287; bh=wIElk+8Yjt/y58bEutr6XNPojpOeAx/3LFQtXQ/TtcE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=YaXY99Imw3oIa57i7HAurBomcTi31N1v/B+LkocyLOlrmS5hp9JVCoCM1hYVXHqSu pCoP7/ReVlSQGZum6AAfYDgXulJe1h8XkahlfHBj4YeRiXm1cDAoPOIBSrv/4bvO4T keD4INjNRnVP8Wn+VWi6Zcp0E/85LyGh4JEASvAK7RzR4GTfFkFTlqOBjB0WgnH9VD RrzGcGC0zYyFwtHwB2XhhMnd8SehizwVuB0LUFtgo+/FMpC54Gq/tgZDVC/Evx8oTb IYjZtxZsKPNLW8hpdES63CLxXNz8lww5fvAh/oo7UEhKQ1l6TikJD2SwFt9iQNo86f +bTaIBjHLJjkg== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 0814B5C0A6B; Wed, 31 Aug 2022 11:08:07 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH rcu 10/10] rcu: Exclude outgoing CPU when it is the last to leave Date: Wed, 31 Aug 2022 11:08:05 -0700 Message-Id: <20220831180805.2693546-10-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220831180759.GA2693289@paulmck-ThinkPad-P17-Gen-1> References: <20220831180759.GA2693289@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org The rcu_boost_kthread_setaffinity() function removes the outgoing CPU from the set_cpus_allowed() mask for the corresponding leaf rcu_node structure's rcub priority-boosting kthread. Except that if the outgoing CPU will leave that structure without any online CPUs, the mask is set to the housekeeping CPU mask from housekeeping_cpumask(). Which is fine unless the outgoing CPU happens to be a housekeeping CPU. This commit therefore removes the outgoing CPU from the housekeeping mask. This would of course be problematic if the outgoing CPU was the last online housekeeping CPU, but in that case you are in a world of hurt anyway. If someone comes up with a valid use case for a system needing all the housekeeping CPUs to be offline, further adjustments can be made. Signed-off-by: Paul E. McKenney --- kernel/rcu/tree_plugin.h | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h index 207617f69aa56..32b424b571bd9 100644 --- a/kernel/rcu/tree_plugin.h +++ b/kernel/rcu/tree_plugin.h @@ -1243,8 +1243,11 @@ static void rcu_boost_kthread_setaffinity(struct rcu_node *rnp, int outgoingcpu) cpu != outgoingcpu) cpumask_set_cpu(cpu, cm); cpumask_and(cm, cm, housekeeping_cpumask(HK_TYPE_RCU)); - if (cpumask_empty(cm)) + if (cpumask_empty(cm)) { cpumask_copy(cm, housekeeping_cpumask(HK_TYPE_RCU)); + if (outgoingcpu >= 0) + cpumask_clear_cpu(outgoingcpu, cm); + } set_cpus_allowed_ptr(t, cm); mutex_unlock(&rnp->boost_kthread_mutex); free_cpumask_var(cm);