From patchwork Wed May 10 17:06:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 13237100 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3FB20C7EE22 for ; Wed, 10 May 2023 17:06:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236003AbjEJRG0 (ORCPT ); Wed, 10 May 2023 13:06:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49816 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236367AbjEJRGX (ORCPT ); Wed, 10 May 2023 13:06:23 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8ABD6358C; Wed, 10 May 2023 10:06:11 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 1FE75636DF; Wed, 10 May 2023 17:06:11 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7F747C433EF; Wed, 10 May 2023 17:06:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1683738370; bh=kQwhUcHnUV1oB8K+SYCdQwdWT6mtPz7OHiOwXCmWFXs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Anbm9VOAYtEtoX7qGLRyHb8ln3njJ14N8ou716QX9jLtd0EpbHyNlc2Ij8OpNmEPC JfpGlBBiSXXmTdoJMZpLaSTmod4JNdZFsA1Ok2UZhRgVbN2DRHUNBj9fNr0kb3KYg9 i3Csz5mZgQ5uogU9/JBg3+gQp/LAULludnnBBF6OSjDCl8M92XV9w//2ZWovyGFt3o STHmsrW6Cq2+bTeTAQE95svezqRBeyWRELebPS1TNyPVnLK0MIkhw/nM5Jc0nrszke 9Vcc7J+uTIlgoKD7UFBqjIq3UCbr9XwsW4kqeqd6NFM9hjW6Qpa4tOLLlpNCgnFjlz PkGcd1uMMKl5A== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 2BB0ECE126E; Wed, 10 May 2023 10:06:10 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, Frederic Weisbecker , "Paul E . McKenney" Subject: [PATCH rcu 1/4] rcu/nocb: Protect lazy shrinker against concurrent (de-)offloading Date: Wed, 10 May 2023 10:06:04 -0700 Message-Id: <20230510170607.2187978-1-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <9c5fae22-e25e-4548-b6e9-117495b29e62@paulmck-laptop> References: <9c5fae22-e25e-4548-b6e9-117495b29e62@paulmck-laptop> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org From: Frederic Weisbecker The shrinker may run concurrently with callbacks (de-)offloading. As such, calling rcu_nocb_lock() is very dangerous because it does a conditional locking. The worst outcome is that rcu_nocb_lock() doesn't lock but rcu_nocb_unlock() eventually unlocks, or the reverse, creating an imbalance. Fix this with protecting against (de-)offloading using the barrier mutex. Although if the barrier mutex is contended, which should be rare, then step aside so as not to trigger a mutex VS allocation dependency chain. Signed-off-by: Frederic Weisbecker Signed-off-by: Paul E. McKenney --- kernel/rcu/tree_nocb.h | 25 ++++++++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index f2280616f9d5..1a86883902ce 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1336,13 +1336,33 @@ lazy_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) unsigned long flags; unsigned long count = 0; + /* + * Protect against concurrent (de-)offloading. Otherwise nocb locking + * may be ignored or imbalanced. + */ + if (!mutex_trylock(&rcu_state.barrier_mutex)) { + /* + * But really don't insist if barrier_mutex is contended since we + * can't guarantee that it will never engage in a dependency + * chain involving memory allocation. The lock is seldom contended + * anyway. + */ + return 0; + } + /* Snapshot count of all CPUs */ for_each_possible_cpu(cpu) { struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); - int _count = READ_ONCE(rdp->lazy_len); + int _count; + + if (!rcu_rdp_is_offloaded(rdp)) + continue; + + _count = READ_ONCE(rdp->lazy_len); if (_count == 0) continue; + rcu_nocb_lock_irqsave(rdp, flags); WRITE_ONCE(rdp->lazy_len, 0); rcu_nocb_unlock_irqrestore(rdp, flags); @@ -1352,6 +1372,9 @@ lazy_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) if (sc->nr_to_scan <= 0) break; } + + mutex_unlock(&rcu_state.barrier_mutex); + return count ? count : SHRINK_STOP; } From patchwork Wed May 10 17:06:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 13237101 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E9D6EC77B7D for ; Wed, 10 May 2023 17:06:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236447AbjEJRG1 (ORCPT ); Wed, 10 May 2023 13:06:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49918 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236319AbjEJRGX (ORCPT ); Wed, 10 May 2023 13:06:23 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D0A70A8; Wed, 10 May 2023 10:06:11 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 352D163F2F; Wed, 10 May 2023 17:06:11 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 88A69C433D2; Wed, 10 May 2023 17:06:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1683738370; bh=XH5sB07f3Ey9Nxv2UrTDaQPS9YGvRkyAeAe39j1qiZg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=hwaz01RxSf5T8D6+ZZpqvWlINKfEHlDRz/ABaTt1G1sGGGfhJx4n2w29hOaNxWGIm Jxtfn5KGGnL/LOS+HDItbHCrRKnfeJpQAYDOTvkHG+6/dayMtQjv2B/scYCtI5HmPC 0GJ2bTDErbVBuQlPk3Z/iS7mnNZEOekQcqlkG+CFhfVKmLzEE/xjDH48uPeS4yPSvj nBT8tj+IOKOo+RiYSCgw4dprENveCuFMsC1M73Bo+tYJiUXwyQsj5lwh9DteNLkGWa eS7hpqr2yLr12VOFpv4EVvCm4jUWBexqcMNj35teRBAfM5S4ptaYZGQPya7zUqiJvj HD2FrfZQyHzJw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 2F6C4CE1279; Wed, 10 May 2023 10:06:10 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, Frederic Weisbecker , "Paul E . McKenney" Subject: [PATCH rcu 2/4] rcu/nocb: Fix shrinker race against callback enqueuer Date: Wed, 10 May 2023 10:06:05 -0700 Message-Id: <20230510170607.2187978-2-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <9c5fae22-e25e-4548-b6e9-117495b29e62@paulmck-laptop> References: <9c5fae22-e25e-4548-b6e9-117495b29e62@paulmck-laptop> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org From: Frederic Weisbecker The shrinker resets the lazy callbacks counter in order to trigger the pending lazy queue flush though the rcuog kthread. The counter reset is protected by the ->nocb_lock against concurrent accesses...except for one of them. Here is a list of existing synchronized readers/writer: 1) The first lazy enqueuer (incrementing ->lazy_len to 1) does so under ->nocb_lock and ->nocb_bypass_lock. 2) The further lazy enqueuers (incrementing ->lazy_len above 1) do so under ->nocb_bypass_lock _only_. 3) The lazy flush checks and resets to 0 under ->nocb_lock and ->nocb_bypass_lock. The shrinker protects its ->lazy_len reset against cases 1) and 3) but not against 2). As such, setting ->lazy_len to 0 under the ->nocb_lock may be cancelled right away by an overwrite from an enqueuer, leading rcuog to ignore the flush. To avoid that, use the proper bypass flush API which takes care of all those details. Signed-off-by: Frederic Weisbecker Signed-off-by: Paul E. McKenney --- kernel/rcu/tree_nocb.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 1a86883902ce..c321fce2af8e 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1364,7 +1364,7 @@ lazy_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) continue; rcu_nocb_lock_irqsave(rdp, flags); - WRITE_ONCE(rdp->lazy_len, 0); + WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies, false)); rcu_nocb_unlock_irqrestore(rdp, flags); wake_nocb_gp(rdp, false); sc->nr_to_scan -= _count; From patchwork Wed May 10 17:06:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 13237102 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DF99CC7EE24 for ; Wed, 10 May 2023 17:06:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233852AbjEJRG1 (ORCPT ); Wed, 10 May 2023 13:06:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49832 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236114AbjEJRGX (ORCPT ); Wed, 10 May 2023 13:06:23 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BF96840DC; Wed, 10 May 2023 10:06:11 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 3B1D963F38; Wed, 10 May 2023 17:06:11 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9803CC4339C; Wed, 10 May 2023 17:06:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1683738370; bh=H90atByaaZG57Ja5si+08MrZWAJclHUD/Fm7cc4BBy0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Zb5FyCgFyVq9kEZcciczkbQBiJssX9KxpfXgBdP7Op/QoO5qstS/dC0UqXuOLZHB5 Q2cmdAtVzBAV29p+W15urN4LldqNFusI3zwWFg2ZSsuRkrqjIpn4K5nMUpobeldqQ+ hsTmP1GiQKHjCWnmgHSXuTm5ikzekrNyBtDvlo8FlfmdpUQZh+fMS8kaw+GZ3ccx5C QDmGfsJvStbf8ONS6KNNdOmMbhqsfMBvvJRjT6oU2d7v/+Fv0qlZbiyR2WgneJRrPf PMxcff/79aG4EXbUpvTbIra9xrpDZFKhQGbhYl6oPh7eGiHmHmKT+sOL9sdUwq8qHm c3sHKU5324qZQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 32131CE127F; Wed, 10 May 2023 10:06:10 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, Frederic Weisbecker , "Paul E . McKenney" Subject: [PATCH rcu 3/4] rcu/nocb: Recheck lazy callbacks under the ->nocb_lock from shrinker Date: Wed, 10 May 2023 10:06:06 -0700 Message-Id: <20230510170607.2187978-3-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <9c5fae22-e25e-4548-b6e9-117495b29e62@paulmck-laptop> References: <9c5fae22-e25e-4548-b6e9-117495b29e62@paulmck-laptop> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org From: Frederic Weisbecker The ->lazy_len is only checked locklessly. Recheck again under the ->nocb_lock to avoid spending more time on flushing/waking if not necessary. The ->lazy_len can still increment concurrently (from 1 to infinity) but under the ->nocb_lock we at least know for sure if there are lazy callbacks at all (->lazy_len > 0). Signed-off-by: Frederic Weisbecker Signed-off-by: Paul E. McKenney --- kernel/rcu/tree_nocb.h | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index c321fce2af8e..dfa9c10d6727 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1358,12 +1358,20 @@ lazy_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) if (!rcu_rdp_is_offloaded(rdp)) continue; - _count = READ_ONCE(rdp->lazy_len); - - if (_count == 0) + if (!READ_ONCE(rdp->lazy_len)) continue; rcu_nocb_lock_irqsave(rdp, flags); + /* + * Recheck under the nocb lock. Since we are not holding the bypass + * lock we may still race with increments from the enqueuer but still + * we know for sure if there is at least one lazy callback. + */ + _count = READ_ONCE(rdp->lazy_len); + if (!_count) { + rcu_nocb_unlock_irqrestore(rdp, flags); + continue; + } WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies, false)); rcu_nocb_unlock_irqrestore(rdp, flags); wake_nocb_gp(rdp, false); From patchwork Wed May 10 17:06:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 13237103 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C19A3C77B7D for ; Wed, 10 May 2023 17:06:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236367AbjEJRGb (ORCPT ); Wed, 10 May 2023 13:06:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49936 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230016AbjEJRGY (ORCPT ); Wed, 10 May 2023 13:06:24 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D27DE2110; Wed, 10 May 2023 10:06:11 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 43FD063F47; Wed, 10 May 2023 17:06:11 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 982EEC4339E; Wed, 10 May 2023 17:06:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1683738370; bh=U+Q4wxr8ZsSuY8eMqt0mzW+sVG9CBmqF3HyB4hgOzwQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Ml32gSFwseQxjkto07DNvhQVAkMiMJt2v6tyTcFUNSliaFMkfCAVHt96WWYubRqDR q0bOqW9QejsnUciAh3uaqNh6qdB18VNsMgaF8WRAA4OuOzi+xvoER4GLnPSUbg0MAY SLxoV+rbKrPEC4LNORrv/kefb1L6ERl+aKJhUQdyCBM6q0gRXZ4w6WXaesSbfTV2cu bMusu1up87tm9m9pF+3+uaT1jBo1Wyf1Bgl7EMvRI8K+Tw1aFE5qT8qe6UO0tnNiuU gF4+gZeFzh39/GAER8lxRAd3FcG0C9XcwPDebFhURaBHrzEKAOBYJUwapSNzDIYw+T owjVLe4pVvE5g== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 346DDCE1348; Wed, 10 May 2023 10:06:10 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, Frederic Weisbecker , "Paul E . McKenney" Subject: [PATCH rcu 4/4] rcu/nocb: Make shrinker to iterate only NOCB CPUs Date: Wed, 10 May 2023 10:06:07 -0700 Message-Id: <20230510170607.2187978-4-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <9c5fae22-e25e-4548-b6e9-117495b29e62@paulmck-laptop> References: <9c5fae22-e25e-4548-b6e9-117495b29e62@paulmck-laptop> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org From: Frederic Weisbecker Callbacks can only be queued as lazy on NOCB CPUs, therefore iterating over the NOCB mask is enough for both counting and scanning. Just lock the mostly uncontended barrier mutex on counting as well in order to keep rcu_nocb_mask stable. Signed-off-by: Frederic Weisbecker Signed-off-by: Paul E. McKenney --- kernel/rcu/tree_nocb.h | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index dfa9c10d6727..43229d2b0c44 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1319,13 +1319,22 @@ lazy_rcu_shrink_count(struct shrinker *shrink, struct shrink_control *sc) int cpu; unsigned long count = 0; + if (WARN_ON_ONCE(!cpumask_available(rcu_nocb_mask))) + return 0; + + /* Protect rcu_nocb_mask against concurrent (de-)offloading. */ + if (!mutex_trylock(&rcu_state.barrier_mutex)) + return 0; + /* Snapshot count of all CPUs */ - for_each_possible_cpu(cpu) { + for_each_cpu(cpu, rcu_nocb_mask) { struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); count += READ_ONCE(rdp->lazy_len); } + mutex_unlock(&rcu_state.barrier_mutex); + return count ? count : SHRINK_EMPTY; } @@ -1336,6 +1345,8 @@ lazy_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) unsigned long flags; unsigned long count = 0; + if (WARN_ON_ONCE(!cpumask_available(rcu_nocb_mask))) + return 0; /* * Protect against concurrent (de-)offloading. Otherwise nocb locking * may be ignored or imbalanced. @@ -1351,11 +1362,11 @@ lazy_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) } /* Snapshot count of all CPUs */ - for_each_possible_cpu(cpu) { + for_each_cpu(cpu, rcu_nocb_mask) { struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); int _count; - if (!rcu_rdp_is_offloaded(rdp)) + if (WARN_ON_ONCE(!rcu_rdp_is_offloaded(rdp))) continue; if (!READ_ONCE(rdp->lazy_len))