From patchwork Tue Nov 12 14:51:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 13872347 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0223C2309A3; Tue, 12 Nov 2024 14:52:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731423130; cv=none; b=NHuZAzMSEdtf5lc3zB8ckbcrMLzHZ4BAmIjL4qQg/L91TWzn3car0ibY+UV5rl3kd3SZvEcOXZXn3BjO6tFNoyAPK0ha0nYl/LtsFED0II80jUNeOX9sJeDwVSCHluMF7bi9rrJchLql/82o32zEFXUiPd4YQsNcFHKgoh/oXZw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731423130; c=relaxed/simple; bh=2S9CoFvH8vbZLEEph+lBWbRkjxdGKOOkqpjrAj9RVRc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=peMPEFqRmqXhnR0c+EUQoE0nOXQpUKKFFxQKCRZR1xxnC9eupKWFzNPboIghX5V5+daj8USuhGcB/XgrfvoWn1E3mJirKKel1Vt2swGT8aF6+8NQ2cXiHZe+8HS1btMmRSP/6nMHkHxCeSv8bIf91x4pl0w+tHXbHW6cLxzGXN4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=tF0fHLsr; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="tF0fHLsr" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 97F91C4CECD; Tue, 12 Nov 2024 14:52:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1731423129; bh=2S9CoFvH8vbZLEEph+lBWbRkjxdGKOOkqpjrAj9RVRc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=tF0fHLsraQbTEUlY2w6EjUJjdlJ5+hBIb1UPKIWTrjp87X9jnqo2HeElhjxXD3eyJ 0I7CPmyX3fzTDaYhkAJ+ezymDQ0cHg2TXujj8CA3MQI+ZcEFPDFyHq/5reG/MlyFjp Hyx5L+YupujyOuyDW9qWmFX1PHkyEU/wFm0ho5UDLRo1fHicU4JcjLian7N2WwaeC4 BdL7r93mhM2drsJca+g3W8C6IdJ6rnBVvhrCxnMyq+Gp+tFIvaPMDF+4FrYtlX2zUt 9OD+GG1jrlY0qhu0Ra8rBopSAU9eB9qH49KOZuhFdmwzvT1hoAiRW3VmB+HEXKn9m1 mY5do7WtLJOcQ== From: Frederic Weisbecker To: LKML Cc: Zhen Lei , Boqun Feng , Joel Fernandes , Josh Triplett , Lai Jiangshan , Mathieu Desnoyers , Neeraj Upadhyay , "Paul E . McKenney" , Steven Rostedt , Uladzislau Rezki , Zqiang , rcu , Neeraj Upadhyay , Frederic Weisbecker Subject: [PATCH 01/16] srcu: Replace WARN_ON_ONCE() with BUILD_BUG_ON() if possible Date: Tue, 12 Nov 2024 15:51:44 +0100 Message-ID: <20241112145159.23032-2-frederic@kernel.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20241112145159.23032-1-frederic@kernel.org> References: <20241112145159.23032-1-frederic@kernel.org> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Zhen Lei The value of ARRAY_SIZE() can be determined at compile time, so if both sides of the equation are ARRAY_SIZE(), using BUILD_BUG_ON() can help us catch the problem earlier. While there are cases where unequal array sizes will work, there is no point in allowing them, so it makes more sense to force them to be equal using BUILD_BUG_ON(). Signed-off-by: Zhen Lei Signed-off-by: Neeraj Upadhyay Signed-off-by: Frederic Weisbecker --- kernel/rcu/srcutree.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 31706e3293bc..78afaffd1b26 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -128,7 +128,7 @@ static void init_srcu_struct_data(struct srcu_struct *ssp) * Initialize the per-CPU srcu_data array, which feeds into the * leaves of the srcu_node tree. */ - WARN_ON_ONCE(ARRAY_SIZE(sdp->srcu_lock_count) != + BUILD_BUG_ON(ARRAY_SIZE(sdp->srcu_lock_count) != ARRAY_SIZE(sdp->srcu_unlock_count)); for_each_possible_cpu(cpu) { sdp = per_cpu_ptr(ssp->sda, cpu); @@ -187,7 +187,7 @@ static bool init_srcu_struct_nodes(struct srcu_struct *ssp, gfp_t gfp_flags) /* Each pass through this loop initializes one srcu_node structure. */ srcu_for_each_node_breadth_first(ssp, snp) { spin_lock_init(&ACCESS_PRIVATE(snp, lock)); - WARN_ON_ONCE(ARRAY_SIZE(snp->srcu_have_cbs) != + BUILD_BUG_ON(ARRAY_SIZE(snp->srcu_have_cbs) != ARRAY_SIZE(snp->srcu_data_have_cbs)); for (i = 0; i < ARRAY_SIZE(snp->srcu_have_cbs); i++) { snp->srcu_have_cbs[i] = SRCU_SNP_INIT_SEQ; From patchwork Tue Nov 12 14:51:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 13872348 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0F81E2309A3; Tue, 12 Nov 2024 14:52:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731423136; cv=none; b=tgiUn2/LUD701x8yvCwo1DDjPwXsKhbVe0f3gjRp4ATfCCIhgi8milzXVL1HJTMltW//r5dcFXPWye7EmY9GAmoKO2E25MvKxbOcxmbt7TOY8/rCDQjSJ4bQpmJATxeGM/Ukr5ykR4BV+Xsf3cB0FOFVabmRzKlGWSKeSYc9FJ0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731423136; c=relaxed/simple; bh=z1VXHQ8nm+thKUAOq8fF+b7rvfScDMLO8mSbw0MCNro=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=BFQVVl2sfn809r5jjoC3R/dJ+vOeyd/YKBdtQPeD3J01cP9uTaw/PNi61iOmFS9z1cAxaL9sVLpoCVCHC3bjUdkSHHaLFoEVSfa2Fs7ojVDpzS77NoxmPVylSvXHQgFhynxCIyqXUXuLfq61sV/377y6HVMNlattbEn+gu+Huo8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=T933MiDd; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="T933MiDd" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 10E22C4CED0; Tue, 12 Nov 2024 14:52:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1731423133; bh=z1VXHQ8nm+thKUAOq8fF+b7rvfScDMLO8mSbw0MCNro=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=T933MiDdj+qYN+IK2kYLW4s3EmTvllNqGp0Y+/7B3jmv6+D4wg/FUx8Ro3h19b57M 5IINo09bsWhFRaCitBUgTDMeCzOPmkIA64my1IksBllOCVVgH7X+8AwLB+3YYx13Ve vlXMileo9CvfBG8H+ggvTtXOm+BTtv8UivK/89oI8X7Q7dZuoibFzIsytBXJMPUjwZ XsvsfYVZi1BIZrOjOlfMlPQq2tAjo/Ll9jEBwXae9TjXc2QLAKdUjDHMPcA4xy1j/w fD/fcogS2cmvcUwq+g08zVELkQeREKE4AnwNYDsUkC0ctI6zJQkmOKjKkrlfBYoiJs SGq84tMDNgfMA== From: Frederic Weisbecker To: LKML Cc: "Paul E. McKenney" , Boqun Feng , Joel Fernandes , Josh Triplett , Lai Jiangshan , Mathieu Desnoyers , Neeraj Upadhyay , Steven Rostedt , Uladzislau Rezki , Zqiang , rcu , Alexei Starovoitov , Andrii Nakryiko , Peter Zijlstra , Kent Overstreet , bpf@vger.kernel.org, Neeraj Upadhyay , Frederic Weisbecker Subject: [PATCH 02/16] srcu: Rename srcu_might_be_idle() to srcu_should_expedite() Date: Tue, 12 Nov 2024 15:51:45 +0100 Message-ID: <20241112145159.23032-3-frederic@kernel.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20241112145159.23032-1-frederic@kernel.org> References: <20241112145159.23032-1-frederic@kernel.org> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" SRCU auto-expedites grace periods that follow a sufficiently long idle period, and the srcu_might_be_idle() function is used to make this decision. However, the upcoming light-weight SRCU readers will not do auto-expediting because doing so would cause the grace-period machinery to invoke synchronize_rcu_expedited() twice, with IPIs all around. However, software-engineering considerations force this determination to remain in srcu_might_be_idle(). This commit therefore changes the name of srcu_might_be_idle() to srcu_should_expedite(), thus moving from what it currently does to why it does it, this latter being more future-proof. Signed-off-by: Paul E. McKenney Cc: Alexei Starovoitov Cc: Andrii Nakryiko Cc: Peter Zijlstra Cc: Kent Overstreet Cc: Reviewed-by: Neeraj Upadhyay Signed-off-by: Frederic Weisbecker --- kernel/rcu/srcutree.c | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 78afaffd1b26..2fe0abade9c0 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -1139,7 +1139,8 @@ static void srcu_flip(struct srcu_struct *ssp) } /* - * If SRCU is likely idle, return true, otherwise return false. + * If SRCU is likely idle, in other words, the next SRCU grace period + * should be expedited, return true, otherwise return false. * * Note that it is OK for several current from-idle requests for a new * grace period from idle to specify expediting because they will all end @@ -1159,7 +1160,7 @@ static void srcu_flip(struct srcu_struct *ssp) * negligible when amortized over that time period, and the extra latency * of a needlessly non-expedited grace period is similarly negligible. */ -static bool srcu_might_be_idle(struct srcu_struct *ssp) +static bool srcu_should_expedite(struct srcu_struct *ssp) { unsigned long curseq; unsigned long flags; @@ -1469,14 +1470,15 @@ EXPORT_SYMBOL_GPL(synchronize_srcu_expedited); * Implementation of these memory-ordering guarantees is similar to * that of synchronize_rcu(). * - * If SRCU is likely idle, expedite the first request. This semantic - * was provided by Classic SRCU, and is relied upon by its users, so TREE - * SRCU must also provide it. Note that detecting idleness is heuristic - * and subject to both false positives and negatives. + * If SRCU is likely idle as determined by srcu_should_expedite(), + * expedite the first request. This semantic was provided by Classic SRCU, + * and is relied upon by its users, so TREE SRCU must also provide it. + * Note that detecting idleness is heuristic and subject to both false + * positives and negatives. */ void synchronize_srcu(struct srcu_struct *ssp) { - if (srcu_might_be_idle(ssp) || rcu_gp_is_expedited()) + if (srcu_should_expedite(ssp) || rcu_gp_is_expedited()) synchronize_srcu_expedited(ssp); else __synchronize_srcu(ssp, true); From patchwork Tue Nov 12 14:51:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 13872349 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0D13620010B; Tue, 12 Nov 2024 14:52:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731423138; cv=none; b=blvAE/txZHXtUw72M2Hd68zBZ6d/ZgLeSA39OGz2r7mYYeUTYJVzsSsRzsbseUOzH+TlrP1VgdboQexrdjTe4Hehuf39FA9cy+wfASEBngPGalkO+/tSnvcHd1rpXDIRQLycgbe/tp2IRuvD09U4bjsTfTyg/0HvGzb4pq4WWUE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731423138; c=relaxed/simple; bh=5UY7zE2xDmZ/aGLmtXPVX1tGAih/WdNZc9gDIB70zsc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=p0ebkmKRY8lLRsXIwTYy9w+Ol4zzpbEK2G2IvP+vSbgpHBxRBu1y6E1SJG4gPZqR0m1LgWx2slKBxYeFfMjPOYm4Q4cxjx5t0SRWwP90XNytfKJ+5BHyd/Rj0qiQ/xK+YCWsFMzZeLsG6eU/IJ48nCcqPdlTCbmJdrsUd3JlvFY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=s9DUX3WW; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="s9DUX3WW" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 09482C4CECD; Tue, 12 Nov 2024 14:52:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1731423137; bh=5UY7zE2xDmZ/aGLmtXPVX1tGAih/WdNZc9gDIB70zsc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=s9DUX3WWJRRloLMHnjLBrMjIzfxWmhraywqbrSLtMP0pBdcsgVNveghEZEOFsYsTS hiaTF3uRo5BmUl1IBbRoX6BrqaBSMxn0gdTIQHOmsCLWTW31LBfPGhzr9A8hJm5l+w Q9tY4WQzCw+FuP7F0PCK3N9N/pmDmEpElM0JkqH757udj3UTKyyZOpw8m6Ly3eJ9Jg gdZbrYV/FHpEKiiapcrlHqRD8S+iSRcH6QRaEj5NguI5pRBvvHF+xizCu5b3hbjZdJ tR+SE4e19PF7tFidMBIpFeZd2zyHFX5GYAri4VhHEdIJqfSfW08qgV9UadYaNcHk0X ex/f0xvtrPdGg== From: Frederic Weisbecker To: LKML Cc: "Paul E. McKenney" , Boqun Feng , Joel Fernandes , Josh Triplett , Lai Jiangshan , Mathieu Desnoyers , Neeraj Upadhyay , Steven Rostedt , Uladzislau Rezki , Zqiang , rcu , Alexei Starovoitov , Andrii Nakryiko , Peter Zijlstra , Kent Overstreet , bpf@vger.kernel.org, Neeraj Upadhyay , Frederic Weisbecker Subject: [PATCH 03/16] srcu: Introduce srcu_gp_is_expedited() helper function Date: Tue, 12 Nov 2024 15:51:46 +0100 Message-ID: <20241112145159.23032-4-frederic@kernel.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20241112145159.23032-1-frederic@kernel.org> References: <20241112145159.23032-1-frederic@kernel.org> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" Even though the open-coded expressions usually fit on one line, this commit replaces them with a call to a new srcu_gp_is_expedited() helper function in order to improve readability. Signed-off-by: Paul E. McKenney Cc: Alexei Starovoitov Cc: Andrii Nakryiko Cc: Peter Zijlstra Cc: Kent Overstreet Cc: Reviewed-by: Neeraj Upadhyay Signed-off-by: Frederic Weisbecker --- kernel/rcu/srcutree.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 2fe0abade9c0..5b1a315f77bc 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -418,6 +418,16 @@ static void check_init_srcu_struct(struct srcu_struct *ssp) spin_unlock_irqrestore_rcu_node(ssp->srcu_sup, flags); } +/* + * Is the current or any upcoming grace period to be expedited? + */ +static bool srcu_gp_is_expedited(struct srcu_struct *ssp) +{ + struct srcu_usage *sup = ssp->srcu_sup; + + return ULONG_CMP_LT(READ_ONCE(sup->srcu_gp_seq), READ_ONCE(sup->srcu_gp_seq_needed_exp)); +} + /* * Returns approximate total of the readers' ->srcu_lock_count[] values * for the rank of per-CPU counters specified by idx. @@ -622,7 +632,7 @@ static unsigned long srcu_get_delay(struct srcu_struct *ssp) unsigned long jbase = SRCU_INTERVAL; struct srcu_usage *sup = ssp->srcu_sup; - if (ULONG_CMP_LT(READ_ONCE(sup->srcu_gp_seq), READ_ONCE(sup->srcu_gp_seq_needed_exp))) + if (srcu_gp_is_expedited(ssp)) jbase = 0; if (rcu_seq_state(READ_ONCE(sup->srcu_gp_seq))) { j = jiffies - 1; @@ -867,7 +877,7 @@ static void srcu_gp_end(struct srcu_struct *ssp) spin_lock_irq_rcu_node(sup); idx = rcu_seq_state(sup->srcu_gp_seq); WARN_ON_ONCE(idx != SRCU_STATE_SCAN2); - if (ULONG_CMP_LT(READ_ONCE(sup->srcu_gp_seq), READ_ONCE(sup->srcu_gp_seq_needed_exp))) + if (srcu_gp_is_expedited(ssp)) cbdelay = 0; WRITE_ONCE(sup->srcu_last_gp_end, ktime_get_mono_fast_ns()); From patchwork Tue Nov 12 14:51:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 13872350 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2BF621FF038; Tue, 12 Nov 2024 14:52:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731423142; cv=none; b=Y607TxczhGi61Whwxd1h90jrBJ1zr1yA9lpghMpUY3LrfPrh2M0FoQHNcRgtsCOp4gZDR5lkiNvBmOyIkpbagDxNhA8l7DHuB1WqHwfQj9PxHtACGQWhbBln50bqBqXW8aVCzO+jOM/F0dLTR182SMot1JExhY+7SnFtFKJy+BA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731423142; c=relaxed/simple; bh=NQcbyP0eQeX9acimkw+BybSioEiW16tdxX92lZ4lFVI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=tYQmC2LVxiGf0GH+C7A5ZlL891B140bhUMmUlljhPll94moVJNc4FqRlWj5y6XumSsnXF8wqcEpA47YVnxEBSnWUh4mephPGdc2qCXzs/Es0rppjRMpSnIBgwA7Dos/ClIYwJ22kiiFiBmivDVZVE/mnA1sqQ0Gw8O23erJh4Gs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=r8fOVVv3; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="r8fOVVv3" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0E392C4CED0; Tue, 12 Nov 2024 14:52:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1731423141; bh=NQcbyP0eQeX9acimkw+BybSioEiW16tdxX92lZ4lFVI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=r8fOVVv3fToXQjNH06pmkV2G2DsuoVynWcJqyuYAs1L5K3WOddoeFGJgkWyOG4Od3 5D9fkAbrhZXqP7Tks8fHjUaqzTz58/jWjw1I28ehbgISxQydY9g4wTBi+0qGcvqfk1 Os7l8SElJzFTOqu9jAEykZpiBoOAfQszZQgS6OObCHDxagwaHVbYjazmMf+PK/qJea p76dnEGNIYtMmyv2boBmctRn11dzxckTt/zggfUKkpyJmM9PvffTrCgsH7spY+ftK/ cEDZCAdYga34IfouXHfg3rpC0TAUdcnOPBtlpxfEhOwbKwrWIx6ZRYotv9B/MevQ4e N+mpIiWntUTIw== From: Frederic Weisbecker To: LKML Cc: "Paul E. McKenney" , Boqun Feng , Joel Fernandes , Josh Triplett , Lai Jiangshan , Mathieu Desnoyers , Neeraj Upadhyay , Steven Rostedt , Uladzislau Rezki , Zqiang , rcu , Alexei Starovoitov , Andrii Nakryiko , Peter Zijlstra , Kent Overstreet , bpf@vger.kernel.org, Neeraj Upadhyay , Frederic Weisbecker Subject: [PATCH 04/16] srcu: Renaming in preparation for additional reader flavor Date: Tue, 12 Nov 2024 15:51:47 +0100 Message-ID: <20241112145159.23032-5-frederic@kernel.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20241112145159.23032-1-frederic@kernel.org> References: <20241112145159.23032-1-frederic@kernel.org> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" Currently, there are only two flavors of readers, normal and NMI-safe. A number of fields, functions, and types reflect this restriction. This renaming-only commit prepares for the addition of light-weight (as in memory-barrier-free) readers. OK, OK, there is also a drive-by white-space fixeup! Signed-off-by: Paul E. McKenney Cc: Alexei Starovoitov Cc: Andrii Nakryiko Cc: Peter Zijlstra Cc: Kent Overstreet Cc: Reviewed-by: Neeraj Upadhyay Signed-off-by: Frederic Weisbecker --- include/linux/srcu.h | 21 ++++++++++----------- include/linux/srcutree.h | 2 +- kernel/rcu/srcutree.c | 22 +++++++++++----------- 3 files changed, 22 insertions(+), 23 deletions(-) diff --git a/include/linux/srcu.h b/include/linux/srcu.h index 835bbb2d1f88..06728ef6f32a 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -181,10 +181,9 @@ static inline int srcu_read_lock_held(const struct srcu_struct *ssp) #define SRCU_NMI_SAFE 0x2 #if defined(CONFIG_PROVE_RCU) && defined(CONFIG_TREE_SRCU) -void srcu_check_nmi_safety(struct srcu_struct *ssp, bool nmi_safe); +void srcu_check_read_flavor(struct srcu_struct *ssp, int read_flavor); #else -static inline void srcu_check_nmi_safety(struct srcu_struct *ssp, - bool nmi_safe) { } +static inline void srcu_check_read_flavor(struct srcu_struct *ssp, int read_flavor) { } #endif @@ -245,7 +244,7 @@ static inline int srcu_read_lock(struct srcu_struct *ssp) __acquires(ssp) { int retval; - srcu_check_nmi_safety(ssp, false); + srcu_check_read_flavor(ssp, false); retval = __srcu_read_lock(ssp); srcu_lock_acquire(&ssp->dep_map); return retval; @@ -262,7 +261,7 @@ static inline int srcu_read_lock_nmisafe(struct srcu_struct *ssp) __acquires(ssp { int retval; - srcu_check_nmi_safety(ssp, true); + srcu_check_read_flavor(ssp, true); retval = __srcu_read_lock_nmisafe(ssp); rcu_try_lock_acquire(&ssp->dep_map); return retval; @@ -274,7 +273,7 @@ srcu_read_lock_notrace(struct srcu_struct *ssp) __acquires(ssp) { int retval; - srcu_check_nmi_safety(ssp, false); + srcu_check_read_flavor(ssp, false); retval = __srcu_read_lock(ssp); return retval; } @@ -303,7 +302,7 @@ srcu_read_lock_notrace(struct srcu_struct *ssp) __acquires(ssp) static inline int srcu_down_read(struct srcu_struct *ssp) __acquires(ssp) { WARN_ON_ONCE(in_nmi()); - srcu_check_nmi_safety(ssp, false); + srcu_check_read_flavor(ssp, false); return __srcu_read_lock(ssp); } @@ -318,7 +317,7 @@ static inline void srcu_read_unlock(struct srcu_struct *ssp, int idx) __releases(ssp) { WARN_ON_ONCE(idx & ~0x1); - srcu_check_nmi_safety(ssp, false); + srcu_check_read_flavor(ssp, false); srcu_lock_release(&ssp->dep_map); __srcu_read_unlock(ssp, idx); } @@ -334,7 +333,7 @@ static inline void srcu_read_unlock_nmisafe(struct srcu_struct *ssp, int idx) __releases(ssp) { WARN_ON_ONCE(idx & ~0x1); - srcu_check_nmi_safety(ssp, true); + srcu_check_read_flavor(ssp, true); rcu_lock_release(&ssp->dep_map); __srcu_read_unlock_nmisafe(ssp, idx); } @@ -343,7 +342,7 @@ static inline void srcu_read_unlock_nmisafe(struct srcu_struct *ssp, int idx) static inline notrace void srcu_read_unlock_notrace(struct srcu_struct *ssp, int idx) __releases(ssp) { - srcu_check_nmi_safety(ssp, false); + srcu_check_read_flavor(ssp, false); __srcu_read_unlock(ssp, idx); } @@ -360,7 +359,7 @@ static inline void srcu_up_read(struct srcu_struct *ssp, int idx) { WARN_ON_ONCE(idx & ~0x1); WARN_ON_ONCE(in_nmi()); - srcu_check_nmi_safety(ssp, false); + srcu_check_read_flavor(ssp, false); __srcu_read_unlock(ssp, idx); } diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index ed57598394de..ab7d8d215b84 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -25,7 +25,7 @@ struct srcu_data { /* Read-side state. */ atomic_long_t srcu_lock_count[2]; /* Locks per CPU. */ atomic_long_t srcu_unlock_count[2]; /* Unlocks per CPU. */ - int srcu_nmi_safety; /* NMI-safe srcu_struct structure? */ + int srcu_reader_flavor; /* Reader flavor for srcu_struct structure? */ /* Update-side state. */ spinlock_t __private lock ____cacheline_internodealigned_in_smp; diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 5b1a315f77bc..f259dd834272 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -460,7 +460,7 @@ static unsigned long srcu_readers_unlock_idx(struct srcu_struct *ssp, int idx) sum += atomic_long_read(&cpuc->srcu_unlock_count[idx]); if (IS_ENABLED(CONFIG_PROVE_RCU)) - mask = mask | READ_ONCE(cpuc->srcu_nmi_safety); + mask = mask | READ_ONCE(cpuc->srcu_reader_flavor); } WARN_ONCE(IS_ENABLED(CONFIG_PROVE_RCU) && (mask & (mask >> 1)), "Mixed NMI-safe readers for srcu_struct at %ps.\n", ssp); @@ -699,25 +699,25 @@ EXPORT_SYMBOL_GPL(cleanup_srcu_struct); #ifdef CONFIG_PROVE_RCU /* - * Check for consistent NMI safety. + * Check for consistent reader flavor. */ -void srcu_check_nmi_safety(struct srcu_struct *ssp, bool nmi_safe) +void srcu_check_read_flavor(struct srcu_struct *ssp, int read_flavor) { - int nmi_safe_mask = 1 << nmi_safe; - int old_nmi_safe_mask; + int reader_flavor_mask = 1 << read_flavor; + int old_reader_flavor_mask; struct srcu_data *sdp; /* NMI-unsafe use in NMI is a bad sign */ - WARN_ON_ONCE(!nmi_safe && in_nmi()); + WARN_ON_ONCE(!read_flavor && in_nmi()); sdp = raw_cpu_ptr(ssp->sda); - old_nmi_safe_mask = READ_ONCE(sdp->srcu_nmi_safety); - if (!old_nmi_safe_mask) { - WRITE_ONCE(sdp->srcu_nmi_safety, nmi_safe_mask); + old_reader_flavor_mask = READ_ONCE(sdp->srcu_reader_flavor); + if (!old_reader_flavor_mask) { + WRITE_ONCE(sdp->srcu_reader_flavor, reader_flavor_mask); return; } - WARN_ONCE(old_nmi_safe_mask != nmi_safe_mask, "CPU %d old state %d new state %d\n", sdp->cpu, old_nmi_safe_mask, nmi_safe_mask); + WARN_ONCE(old_reader_flavor_mask != reader_flavor_mask, "CPU %d old state %d new state %d\n", sdp->cpu, old_reader_flavor_mask, reader_flavor_mask); } -EXPORT_SYMBOL_GPL(srcu_check_nmi_safety); +EXPORT_SYMBOL_GPL(srcu_check_read_flavor); #endif /* CONFIG_PROVE_RCU */ /* From patchwork Tue Nov 12 14:51:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 13872351 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5339E1FF056; Tue, 12 Nov 2024 14:52:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731423146; cv=none; b=tvN8Eq43NQhskGSamRjzk2qTmEw3WRUHoOp0c5y71fpklWNMNqiBthyCoLz8Wmt0xTjikKezEvjAB/3JRVIdQDZTWXcGCemcYw7CVcslay5jD7kz5wnoiPO0OOFRanp7+f6RNsfBCGc/wQ7HBIxXtTbcVi/2DIN0Ctvlspukixw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731423146; c=relaxed/simple; bh=cdJ+xBnPORsx2g7uxvtjLS56Zpr2XJsr87vnsr2jyos=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=aqniYbOlEeIxMwnje3Cvny43nmJ/tHTBHh5MSwgtw3iYZFGQt8cvAl0cgo+8Lat1J1+VnpcJbSU/tTrjGBeCznzxrlqsYyA9kb/bMFAzEpiNBjqljzYGo8Qp2dTeZfDoc/erXrwptmJSd3hZY1QroJnvN72MKBy+CXGHSJdy6zk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=OHk8JAtZ; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="OHk8JAtZ" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1CFFBC4CED9; Tue, 12 Nov 2024 14:52:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1731423145; bh=cdJ+xBnPORsx2g7uxvtjLS56Zpr2XJsr87vnsr2jyos=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=OHk8JAtZ4DmTg1qGtX9TXL3Xq6bcOHS8jRo5I6rlok3iZW8m89sw985/l8ImsRwHT 2XlKB1WF83tYrLHTFkE5WQ6pf0aNa3BSZsyOUnLIzT5NwJ7JBXQPdBaweGkA3I7inQ VIsIIBTtFoNUwFj4ny5Vo9EjPyh9IZGvEH4yy3Fle1ZpkQ8+qXicH6gpSRXhXE87pl Ccu/HbfVKm0OAGQ4zc91gU7k+yQwC95R7kertZh7tgr+hKdnzdyNDmXHuu0iTtEpZJ f/NM+PapRVldfY3kL2r1xPHI0//A9jJwpf/4+KZ3X/t/nyvr87fjvWVDWd5KwqKF+J 9kYpTr9pA9n5Q== From: Frederic Weisbecker To: LKML Cc: "Paul E. McKenney" , Boqun Feng , Joel Fernandes , Josh Triplett , Lai Jiangshan , Mathieu Desnoyers , Neeraj Upadhyay , Steven Rostedt , Uladzislau Rezki , Zqiang , rcu , Alexei Starovoitov , Andrii Nakryiko , Peter Zijlstra , Kent Overstreet , bpf@vger.kernel.org, Neeraj Upadhyay , Frederic Weisbecker Subject: [PATCH 05/16] srcu: Bit manipulation changes for additional reader flavor Date: Tue, 12 Nov 2024 15:51:48 +0100 Message-ID: <20241112145159.23032-6-frederic@kernel.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20241112145159.23032-1-frederic@kernel.org> References: <20241112145159.23032-1-frederic@kernel.org> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" Currently, there are only two flavors of readers, normal and NMI-safe. Very straightforward state updates suffice to check for erroneous mixing of reader flavors on a given srcu_struct structure. This commit upgrades the checking in preparation for the addition of light-weight (as in memory-barrier-free) readers. Signed-off-by: Paul E. McKenney Cc: Alexei Starovoitov Cc: Andrii Nakryiko Cc: Peter Zijlstra Cc: Kent Overstreet Cc: Reviewed-by: Neeraj Upadhyay Signed-off-by: Frederic Weisbecker --- kernel/rcu/srcutree.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index f259dd834272..9774bc500de5 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -462,7 +462,7 @@ static unsigned long srcu_readers_unlock_idx(struct srcu_struct *ssp, int idx) if (IS_ENABLED(CONFIG_PROVE_RCU)) mask = mask | READ_ONCE(cpuc->srcu_reader_flavor); } - WARN_ONCE(IS_ENABLED(CONFIG_PROVE_RCU) && (mask & (mask >> 1)), + WARN_ONCE(IS_ENABLED(CONFIG_PROVE_RCU) && (mask & (mask - 1)), "Mixed NMI-safe readers for srcu_struct at %ps.\n", ssp); return sum; } @@ -712,8 +712,9 @@ void srcu_check_read_flavor(struct srcu_struct *ssp, int read_flavor) sdp = raw_cpu_ptr(ssp->sda); old_reader_flavor_mask = READ_ONCE(sdp->srcu_reader_flavor); if (!old_reader_flavor_mask) { - WRITE_ONCE(sdp->srcu_reader_flavor, reader_flavor_mask); - return; + old_reader_flavor_mask = cmpxchg(&sdp->srcu_reader_flavor, 0, reader_flavor_mask); + if (!old_reader_flavor_mask) + return; } WARN_ONCE(old_reader_flavor_mask != reader_flavor_mask, "CPU %d old state %d new state %d\n", sdp->cpu, old_reader_flavor_mask, reader_flavor_mask); } From patchwork Tue Nov 12 14:51:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 13872352 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 399491FF056; Tue, 12 Nov 2024 14:52:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731423150; cv=none; b=CRffaEMkr1ylxt/+x5aR+8K+hXrqkcdAQZJZWShjRsVEGJRwtCOvszTtybRn+VAjoIJnurow45PoeCo0Rfhhc6ysfTkYinP4L3rjEpKm9DgUKuLR8VxkxqsfPTj9F/nfz+OIuKVhsVaxmsXrnPPfXRpGzdyRkvP3feISW4JfYsQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731423150; c=relaxed/simple; bh=QJOH44h7yAycsL8NQ2uV4M+pwPZccjc5Vep5GFxRuxY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=o7hHsLTdJZ1rgaRYt+ulvfmHXqO6bQBFoODDVqDEWiXxKeNyIeNSYLN+TFfSGeqJh+CKLPaoqXFLcyXEPXtHANBtk574OpNVPSU9pn9JDAhS8tJzfsPtQtsTw7+segxjgOUjg810yebUH5rr5C7VSKes0NPZbOGRsTVuQ5u9nCo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=U91YU9b6; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="U91YU9b6" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4EBF3C4CECD; Tue, 12 Nov 2024 14:52:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1731423150; bh=QJOH44h7yAycsL8NQ2uV4M+pwPZccjc5Vep5GFxRuxY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=U91YU9b6TSi3vNs1qAwTlqUkUE5tp6I/w1TxduxAySrONB5SrOuDu6lghpnAWjFm2 2klk3QUsTT0OvHf2CA/Fezc8uMzqNrKcoE3e0bEMV+JntBCkZru6qVJ1SlWe6KJFuA 9kCpjGq7hXYQ6m77potKf3lONXQ7VZlEKFEceKkBL3/Qsqd82J5HKO5DuExkmMJIkU JrchAsaDmllSLBD7Hhb6OQxbP41yLjKQYFbdxcJ0xq7QZhZWLmgRNOSrq9TCNbQOIQ wbVzygkFB4hFPW95Z7alW9bxsnByjEycomRcsroJx+FsEYU/SkqrNGOO8gEuhZWC7x dMnoLQqzaKkKA== From: Frederic Weisbecker To: LKML Cc: "Paul E. McKenney" , Boqun Feng , Joel Fernandes , Josh Triplett , Lai Jiangshan , Mathieu Desnoyers , Neeraj Upadhyay , Steven Rostedt , Uladzislau Rezki , Zqiang , rcu , Alexei Starovoitov , Andrii Nakryiko , Peter Zijlstra , Kent Overstreet , bpf@vger.kernel.org, Neeraj Upadhyay , Frederic Weisbecker Subject: [PATCH 06/16] srcu: Standardize srcu_data pointers to "sdp" and similar Date: Tue, 12 Nov 2024 15:51:49 +0100 Message-ID: <20241112145159.23032-7-frederic@kernel.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20241112145159.23032-1-frederic@kernel.org> References: <20241112145159.23032-1-frederic@kernel.org> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" This commit changes a few "cpuc" variables to "sdp" to align with usage elsewhere. [ paulmck: Apply Neeraj Upadhyay feedback. ] Signed-off-by: Paul E. McKenney Cc: Alexei Starovoitov Cc: Andrii Nakryiko Cc: Peter Zijlstra Cc: Kent Overstreet Cc: Reviewed-by: Neeraj Upadhyay Signed-off-by: Frederic Weisbecker --- kernel/rcu/srcutree.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 9774bc500de5..b85da944d794 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -438,9 +438,9 @@ static unsigned long srcu_readers_lock_idx(struct srcu_struct *ssp, int idx) unsigned long sum = 0; for_each_possible_cpu(cpu) { - struct srcu_data *cpuc = per_cpu_ptr(ssp->sda, cpu); + struct srcu_data *sdp = per_cpu_ptr(ssp->sda, cpu); - sum += atomic_long_read(&cpuc->srcu_lock_count[idx]); + sum += atomic_long_read(&sdp->srcu_lock_count[idx]); } return sum; } @@ -456,11 +456,11 @@ static unsigned long srcu_readers_unlock_idx(struct srcu_struct *ssp, int idx) unsigned long sum = 0; for_each_possible_cpu(cpu) { - struct srcu_data *cpuc = per_cpu_ptr(ssp->sda, cpu); + struct srcu_data *sdp = per_cpu_ptr(ssp->sda, cpu); - sum += atomic_long_read(&cpuc->srcu_unlock_count[idx]); + sum += atomic_long_read(&sdp->srcu_unlock_count[idx]); if (IS_ENABLED(CONFIG_PROVE_RCU)) - mask = mask | READ_ONCE(cpuc->srcu_reader_flavor); + mask = mask | READ_ONCE(sdp->srcu_reader_flavor); } WARN_ONCE(IS_ENABLED(CONFIG_PROVE_RCU) && (mask & (mask - 1)), "Mixed NMI-safe readers for srcu_struct at %ps.\n", ssp); @@ -564,12 +564,12 @@ static bool srcu_readers_active(struct srcu_struct *ssp) unsigned long sum = 0; for_each_possible_cpu(cpu) { - struct srcu_data *cpuc = per_cpu_ptr(ssp->sda, cpu); + struct srcu_data *sdp = per_cpu_ptr(ssp->sda, cpu); - sum += atomic_long_read(&cpuc->srcu_lock_count[0]); - sum += atomic_long_read(&cpuc->srcu_lock_count[1]); - sum -= atomic_long_read(&cpuc->srcu_unlock_count[0]); - sum -= atomic_long_read(&cpuc->srcu_unlock_count[1]); + sum += atomic_long_read(&sdp->srcu_lock_count[0]); + sum += atomic_long_read(&sdp->srcu_lock_count[1]); + sum -= atomic_long_read(&sdp->srcu_unlock_count[0]); + sum -= atomic_long_read(&sdp->srcu_unlock_count[1]); } return sum; } From patchwork Tue Nov 12 14:51:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 13872353 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F16C21FF056; Tue, 12 Nov 2024 14:52:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731423155; cv=none; b=EErHVKjMUDlvcT2aN5on9xp8jgnMMbEe9/p3FnHugjyQ7uBK5jNw75FeJUSBNkbnPehB6RPrWgblfovZGk6z4oKv3AToK87om92Q1tQiaJT4C+kr8Y25uqvgihGhycBiHG3uXIZem/AoO7uoU2+qUsc8zZz9op7KT2gZFr8DQCo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731423155; c=relaxed/simple; bh=+k06mgf6RuNLp0+mrhqlZ/eG8Y0LssknZjipMi01OMk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=MCjs7e3VB66n2+GlUES0HiaDE+jZEyB2+PNDNcpll3037u71F03y9bm/OTPkuigKseEt/Y8b3NWiABpHP/+dsCoKGsDiKQXpxgi6wiDwBW0oxSdExLWFWi2GZnE7/apsBqs2PyGxftixgNdCwYYkV5OBKse1CUKdzthpQQZMV0A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=WX0vmyto; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="WX0vmyto" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 84714C4CED0; Tue, 12 Nov 2024 14:52:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1731423153; bh=+k06mgf6RuNLp0+mrhqlZ/eG8Y0LssknZjipMi01OMk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=WX0vmytoeIRvQ2poozrmqIpFfJvnrJpqZzpsRJ52HLFEQCdC+KQ0NWG2KwkQ4Luwb /y3pR2b4cbjiCJ8d+Rj48f5cP1rkPVpHP02ftCPcOi2dOFOjnNJfDv0rOj1/6LgoXd 7ttYjEZIUo9y9BEVFt0YiOsUOivlpMrP6/x5/aKa2LfDZKcXJy4jzpVi/QzkN//zbp oBB2AvyPIqbZK0VbEi7fm8x+6wtNx3mspmULayL+XTYczHodwLGXXAEnWXxSfrMZmI OM6uL8wjzuPoThilq7u3bX/w4hFntyIo9dB0Gpt3dLFxPSPH1seMSJACLglh0AEu9o 0Iv19U8Vt+fXQ== From: Frederic Weisbecker To: LKML Cc: "Paul E. McKenney" , Boqun Feng , Joel Fernandes , Josh Triplett , Lai Jiangshan , Mathieu Desnoyers , Neeraj Upadhyay , Steven Rostedt , Uladzislau Rezki , Zqiang , rcu , Andrii Nakryiko , Neeraj Upadhyay , Frederic Weisbecker Subject: [PATCH 07/16] srcu: Improve srcu_read_lock{,_nmisafe}() comments Date: Tue, 12 Nov 2024 15:51:50 +0100 Message-ID: <20241112145159.23032-8-frederic@kernel.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20241112145159.23032-1-frederic@kernel.org> References: <20241112145159.23032-1-frederic@kernel.org> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" This commit adds some additional usage constraints to the kernel-doc headers of srcu_read_lock() and srcu_read_lock_nmi_safe(). Suggested-by: Andrii Nakryiko Signed-off-by: Paul E. McKenney Reviewed-by: Neeraj Upadhyay Signed-off-by: Frederic Weisbecker --- include/linux/srcu.h | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/include/linux/srcu.h b/include/linux/srcu.h index 06728ef6f32a..46fd06b212ba 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -235,10 +235,13 @@ static inline void srcu_check_read_flavor(struct srcu_struct *ssp, int read_flav * a mutex that is held elsewhere while calling synchronize_srcu() or * synchronize_srcu_expedited(). * - * Note that srcu_read_lock() and the matching srcu_read_unlock() must - * occur in the same context, for example, it is illegal to invoke - * srcu_read_unlock() in an irq handler if the matching srcu_read_lock() - * was invoked in process context. + * The return value from srcu_read_lock() must be passed unaltered + * to the matching srcu_read_unlock(). Note that srcu_read_lock() and + * the matching srcu_read_unlock() must occur in the same context, for + * example, it is illegal to invoke srcu_read_unlock() in an irq handler + * if the matching srcu_read_lock() was invoked in process context. Or, + * for that matter to invoke srcu_read_unlock() from one task and the + * matching srcu_read_lock() from another. */ static inline int srcu_read_lock(struct srcu_struct *ssp) __acquires(ssp) { @@ -256,6 +259,10 @@ static inline int srcu_read_lock(struct srcu_struct *ssp) __acquires(ssp) * * Enter an SRCU read-side critical section, but in an NMI-safe manner. * See srcu_read_lock() for more information. + * + * If srcu_read_lock_nmisafe() is ever used on an srcu_struct structure, + * then none of the other flavors may be used, whether before, during, + * or after. */ static inline int srcu_read_lock_nmisafe(struct srcu_struct *ssp) __acquires(ssp) { From patchwork Tue Nov 12 14:51:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 13872354 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3144B1FF056; Tue, 12 Nov 2024 14:52:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731423158; cv=none; b=lD9MhSFo0OjGUh8ldpPIU3zh2CnkHwZie9LMfXAyWoAYScXBqkhPGp/VqPfJT5CfX3WVrWXWLtjYcshJowM984y8kYFikq+GC++0Abp2y11qdIKmGCFFPyEfqSaKIWFUjiU7g/ScbUa26h0jP36hrc8bNeLxI2EM5SFMtpu/RpA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731423158; c=relaxed/simple; bh=XHUlBJ+mKYT+6k4IYtYtDy0V0TjePNkPX4AfboMkkeo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=an1H7s5XVZ3dsI46boT+5x2LGyo220zhcBAR4LBJxoXj/wjpmn36ZAYo7ca8u/0/fxvoyazlRj4k2xi195SZHtCi4QQW/406slXMTqmlnB05g1xib3FROYtAxPr96iY9hYdMyFKe4SlyPXru0gK4z3+ba05mPvs8KxC4Z3UIExs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=UIaby/0O; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="UIaby/0O" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 01806C4CECD; Tue, 12 Nov 2024 14:52:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1731423157; bh=XHUlBJ+mKYT+6k4IYtYtDy0V0TjePNkPX4AfboMkkeo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=UIaby/0OJfZLGuEU9xoWWlvVpghxrlhsQ57edyJGRLUBbHXZtb9z1n5F41v3cDyOJ 6hc1e5kOsHBDJR8tcwP8vcGYd+/CwWdqiFOrzF4vV9P75DocB8Y0ixrOuZkRDbHOxi mSqh5oKexrap/owERE6mdWStr3g/G+YrWjI6w+iZdu3A8aG2xEUcPLZnjc5YZ6mfu8 s5NR2Hzwb4Mr6VrBZ7SSvRPY1NaPip+57jgDxXL0dW3GsO4Ts7eooAij/ExWRoRPuT ojoKlx4Lt/SLUPnqbC0qi70IY83iXj1xBzrO6lcKFcekMfFDH1Q6hU7qhyVV/dAKYt lJHKgDeJnA5kg== From: Frederic Weisbecker To: LKML Cc: "Paul E. McKenney" , Boqun Feng , Joel Fernandes , Josh Triplett , Lai Jiangshan , Mathieu Desnoyers , Neeraj Upadhyay , Steven Rostedt , Uladzislau Rezki , Zqiang , rcu , Alexei Starovoitov , Andrii Nakryiko , Peter Zijlstra , Kent Overstreet , bpf@vger.kernel.org, Neeraj Upadhyay , Frederic Weisbecker Subject: [PATCH 08/16] srcu: Create CPP macros for normal and NMI-safe SRCU readers Date: Tue, 12 Nov 2024 15:51:51 +0100 Message-ID: <20241112145159.23032-9-frederic@kernel.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20241112145159.23032-1-frederic@kernel.org> References: <20241112145159.23032-1-frederic@kernel.org> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" This commit creates SRCU_READ_FLAVOR_NORMAL and SRCU_READ_FLAVOR_NMI C-preprocessor macros for srcu_read_lock() and srcu_read_lock_nmisafe(), respectively. These replace the old true/false values that were previously passed to srcu_check_read_flavor(). In addition, the srcu_check_read_flavor() function itself requires a bit of rework to handle bitmasks instead of true/false values. Signed-off-by: Paul E. McKenney Cc: Alexei Starovoitov Cc: Andrii Nakryiko Cc: Peter Zijlstra Cc: Kent Overstreet Cc: Signed-off-by: Paul E. McKenney Reviewed-by: Neeraj Upadhyay Signed-off-by: Frederic Weisbecker --- include/linux/srcu.h | 20 ++++++++------------ include/linux/srcutree.h | 4 ++++ kernel/rcu/srcutree.c | 21 +++++++++++---------- 3 files changed, 23 insertions(+), 22 deletions(-) diff --git a/include/linux/srcu.h b/include/linux/srcu.h index 46fd06b212ba..84daaa33ea0a 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -176,10 +176,6 @@ static inline int srcu_read_lock_held(const struct srcu_struct *ssp) #endif /* #else #ifdef CONFIG_DEBUG_LOCK_ALLOC */ -#define SRCU_NMI_UNKNOWN 0x0 -#define SRCU_NMI_UNSAFE 0x1 -#define SRCU_NMI_SAFE 0x2 - #if defined(CONFIG_PROVE_RCU) && defined(CONFIG_TREE_SRCU) void srcu_check_read_flavor(struct srcu_struct *ssp, int read_flavor); #else @@ -247,7 +243,7 @@ static inline int srcu_read_lock(struct srcu_struct *ssp) __acquires(ssp) { int retval; - srcu_check_read_flavor(ssp, false); + srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_NORMAL); retval = __srcu_read_lock(ssp); srcu_lock_acquire(&ssp->dep_map); return retval; @@ -268,7 +264,7 @@ static inline int srcu_read_lock_nmisafe(struct srcu_struct *ssp) __acquires(ssp { int retval; - srcu_check_read_flavor(ssp, true); + srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_NMI); retval = __srcu_read_lock_nmisafe(ssp); rcu_try_lock_acquire(&ssp->dep_map); return retval; @@ -280,7 +276,7 @@ srcu_read_lock_notrace(struct srcu_struct *ssp) __acquires(ssp) { int retval; - srcu_check_read_flavor(ssp, false); + srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_NORMAL); retval = __srcu_read_lock(ssp); return retval; } @@ -309,7 +305,7 @@ srcu_read_lock_notrace(struct srcu_struct *ssp) __acquires(ssp) static inline int srcu_down_read(struct srcu_struct *ssp) __acquires(ssp) { WARN_ON_ONCE(in_nmi()); - srcu_check_read_flavor(ssp, false); + srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_NORMAL); return __srcu_read_lock(ssp); } @@ -324,7 +320,7 @@ static inline void srcu_read_unlock(struct srcu_struct *ssp, int idx) __releases(ssp) { WARN_ON_ONCE(idx & ~0x1); - srcu_check_read_flavor(ssp, false); + srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_NORMAL); srcu_lock_release(&ssp->dep_map); __srcu_read_unlock(ssp, idx); } @@ -340,7 +336,7 @@ static inline void srcu_read_unlock_nmisafe(struct srcu_struct *ssp, int idx) __releases(ssp) { WARN_ON_ONCE(idx & ~0x1); - srcu_check_read_flavor(ssp, true); + srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_NMI); rcu_lock_release(&ssp->dep_map); __srcu_read_unlock_nmisafe(ssp, idx); } @@ -349,7 +345,7 @@ static inline void srcu_read_unlock_nmisafe(struct srcu_struct *ssp, int idx) static inline notrace void srcu_read_unlock_notrace(struct srcu_struct *ssp, int idx) __releases(ssp) { - srcu_check_read_flavor(ssp, false); + srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_NORMAL); __srcu_read_unlock(ssp, idx); } @@ -366,7 +362,7 @@ static inline void srcu_up_read(struct srcu_struct *ssp, int idx) { WARN_ON_ONCE(idx & ~0x1); WARN_ON_ONCE(in_nmi()); - srcu_check_read_flavor(ssp, false); + srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_NORMAL); __srcu_read_unlock(ssp, idx); } diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index ab7d8d215b84..79ad809c7f03 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -43,6 +43,10 @@ struct srcu_data { struct srcu_struct *ssp; }; +/* Values for ->srcu_reader_flavor. */ +#define SRCU_READ_FLAVOR_NORMAL 0x1 // srcu_read_lock(). +#define SRCU_READ_FLAVOR_NMI 0x2 // srcu_read_lock_nmisafe(). + /* * Node in SRCU combining tree, similar in function to rcu_data. */ diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index b85da944d794..602b4b8c4b89 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -463,7 +463,7 @@ static unsigned long srcu_readers_unlock_idx(struct srcu_struct *ssp, int idx) mask = mask | READ_ONCE(sdp->srcu_reader_flavor); } WARN_ONCE(IS_ENABLED(CONFIG_PROVE_RCU) && (mask & (mask - 1)), - "Mixed NMI-safe readers for srcu_struct at %ps.\n", ssp); + "Mixed reader flavors for srcu_struct at %ps.\n", ssp); return sum; } @@ -703,20 +703,21 @@ EXPORT_SYMBOL_GPL(cleanup_srcu_struct); */ void srcu_check_read_flavor(struct srcu_struct *ssp, int read_flavor) { - int reader_flavor_mask = 1 << read_flavor; - int old_reader_flavor_mask; + int old_read_flavor; struct srcu_data *sdp; - /* NMI-unsafe use in NMI is a bad sign */ - WARN_ON_ONCE(!read_flavor && in_nmi()); + /* NMI-unsafe use in NMI is a bad sign, as is multi-bit read_flavor values. */ + WARN_ON_ONCE((read_flavor != SRCU_READ_FLAVOR_NMI) && in_nmi()); + WARN_ON_ONCE(read_flavor & (read_flavor - 1)); + sdp = raw_cpu_ptr(ssp->sda); - old_reader_flavor_mask = READ_ONCE(sdp->srcu_reader_flavor); - if (!old_reader_flavor_mask) { - old_reader_flavor_mask = cmpxchg(&sdp->srcu_reader_flavor, 0, reader_flavor_mask); - if (!old_reader_flavor_mask) + old_read_flavor = READ_ONCE(sdp->srcu_reader_flavor); + if (!old_read_flavor) { + old_read_flavor = cmpxchg(&sdp->srcu_reader_flavor, 0, read_flavor); + if (!old_read_flavor) return; } - WARN_ONCE(old_reader_flavor_mask != reader_flavor_mask, "CPU %d old state %d new state %d\n", sdp->cpu, old_reader_flavor_mask, reader_flavor_mask); + WARN_ONCE(old_read_flavor != read_flavor, "CPU %d old state %d new state %d\n", sdp->cpu, old_read_flavor, read_flavor); } EXPORT_SYMBOL_GPL(srcu_check_read_flavor); #endif /* CONFIG_PROVE_RCU */ From patchwork Tue Nov 12 14:51:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 13872355 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9BFCD1FF056; Tue, 12 Nov 2024 14:52:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731423162; cv=none; b=G9xlX8qGoztkK1ErW2mGF4vuC21UZj/xuw4fZnOnzfn8MpUjGjoHyYgXXJ0op95h3Z/r0J8PDHzPbljs/NGf/Qfr9LuuiOjgtn/7u6JHtOZD4SonrjAcbEjvR7XMYPVfszviytyNi1qV6pjAeM9QX+d9BCl9IH2oedVLX0Nsfdw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731423162; c=relaxed/simple; bh=yltyxHaqD19z2KwpYjA0voRol2zhvum/Rg11NyAJhTU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=vFMlb4c+fyoeHG+k4JyfTqhrczjz9BismpXYEFFD9bBnWV8cac1W/KtzbxIsJhiY2TIjg0VuQQ9eK7XbW4wAl2HBUEtobYNhFz5BejGX7FLu6zTBCJvhi0mmD9NPL+36tY/Vlz9VdMYMTPxFvch5BzTz5BUTrCWE8kZAYEeBGNA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=leMx6E9k; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="leMx6E9k" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4917DC4CED6; Tue, 12 Nov 2024 14:52:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1731423162; bh=yltyxHaqD19z2KwpYjA0voRol2zhvum/Rg11NyAJhTU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=leMx6E9kkc2gKzyI0QzAgK/HDoUFGxV1ugLfq0yeFS0Fgmslu0vKI22jYFGw9L0+H T+cfOIRyW4y4AW122+qLKLDojKAif7EbF0ddKMlR/MEYFWbkiGMBNjkiv5d4H6cYoU ge9khr7vdlsAehV9cKVJu1bG6ZOLfaStXY1fsITCPhYqsSjDU8kgPiuZ4bQeYialbQ aoVNbOGaS66iofd92jcuuzzn2fVR1UNPiXmhBJW8KaMl22OV3MLD8P/TTADcIQWZmz 3Uyy6aeta+dCjJax0YV472o83v3betVn1y6wqLAqw7mAT0kkF21KoAH5fVwKeCtNIt 4yq+Mt8eQt8ag== From: Frederic Weisbecker To: LKML Cc: "Paul E. McKenney" , Boqun Feng , Joel Fernandes , Josh Triplett , Lai Jiangshan , Mathieu Desnoyers , Neeraj Upadhyay , Steven Rostedt , Uladzislau Rezki , Zqiang , rcu , kernel test robot , Alexei Starovoitov , Andrii Nakryiko , Peter Zijlstra , Kent Overstreet , bpf@vger.kernel.org, Neeraj Upadhyay , Frederic Weisbecker Subject: [PATCH 09/16] srcu: Add srcu_read_lock_lite() and srcu_read_unlock_lite() Date: Tue, 12 Nov 2024 15:51:52 +0100 Message-ID: <20241112145159.23032-10-frederic@kernel.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20241112145159.23032-1-frederic@kernel.org> References: <20241112145159.23032-1-frederic@kernel.org> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" This patch adds srcu_read_lock_lite() and srcu_read_unlock_lite(), which dispense with the read-side smp_mb() but also are restricted to code regions that RCU is watching. If a given srcu_struct structure uses srcu_read_lock_lite() and srcu_read_unlock_lite(), it is not permitted to use any other SRCU read-side marker, before, during, or after. Another price of light-weight readers is heavier weight grace periods. Such readers mean that SRCU grace periods on srcu_struct structures used by light-weight readers will incur at least two calls to synchronize_rcu(). In addition, normal SRCU grace periods for light-weight-reader srcu_struct structures never auto-expedite. Note that expedited SRCU grace periods for light-weight-reader srcu_struct structures still invoke synchronize_rcu(), not synchronize_srcu_expedited(). Something about wishing to keep the IPIs down to a dull roar. The srcu_read_lock_lite() and srcu_read_unlock_lite() functions may not (repeat, *not*) be used from NMI handlers, but if this is needed, an additional flavor of SRCU reader can be added by some future commit. [ paulmck: Apply Alexei Starovoitov expediting feedback. ] [ paulmck: Apply kernel test robot feedback. ] Signed-off-by: Paul E. McKenney Tested-by: kernel test robot Cc: Alexei Starovoitov Cc: Andrii Nakryiko Cc: Peter Zijlstra Cc: Kent Overstreet Cc: Reviewed-by: Neeraj Upadhyay Signed-off-by: Frederic Weisbecker --- include/linux/srcu.h | 51 ++++++++++++++++++++++++- include/linux/srcutree.h | 1 + kernel/rcu/srcutree.c | 82 ++++++++++++++++++++++++++++++++++------ 3 files changed, 122 insertions(+), 12 deletions(-) diff --git a/include/linux/srcu.h b/include/linux/srcu.h index 84daaa33ea0a..4ba96e2cfa40 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -56,6 +56,13 @@ void call_srcu(struct srcu_struct *ssp, struct rcu_head *head, void cleanup_srcu_struct(struct srcu_struct *ssp); int __srcu_read_lock(struct srcu_struct *ssp) __acquires(ssp); void __srcu_read_unlock(struct srcu_struct *ssp, int idx) __releases(ssp); +#ifdef CONFIG_TINY_SRCU +#define __srcu_read_lock_lite __srcu_read_lock +#define __srcu_read_unlock_lite __srcu_read_unlock +#else // #ifdef CONFIG_TINY_SRCU +int __srcu_read_lock_lite(struct srcu_struct *ssp) __acquires(ssp); +void __srcu_read_unlock_lite(struct srcu_struct *ssp, int idx) __releases(ssp); +#endif // #else // #ifdef CONFIG_TINY_SRCU void synchronize_srcu(struct srcu_struct *ssp); #define SRCU_GET_STATE_COMPLETED 0x1 @@ -179,7 +186,7 @@ static inline int srcu_read_lock_held(const struct srcu_struct *ssp) #if defined(CONFIG_PROVE_RCU) && defined(CONFIG_TREE_SRCU) void srcu_check_read_flavor(struct srcu_struct *ssp, int read_flavor); #else -static inline void srcu_check_read_flavor(struct srcu_struct *ssp, int read_flavor) { } +#define srcu_check_read_flavor(ssp, read_flavor) do { } while (0) #endif @@ -249,6 +256,32 @@ static inline int srcu_read_lock(struct srcu_struct *ssp) __acquires(ssp) return retval; } +/** + * srcu_read_lock_lite - register a new reader for an SRCU-protected structure. + * @ssp: srcu_struct in which to register the new reader. + * + * Enter an SRCU read-side critical section, but for a light-weight + * smp_mb()-free reader. See srcu_read_lock() for more information. + * + * If srcu_read_lock_lite() is ever used on an srcu_struct structure, + * then none of the other flavors may be used, whether before, during, + * or after. Note that grace-period auto-expediting is disabled for _lite + * srcu_struct structures because auto-expedited grace periods invoke + * synchronize_rcu_expedited(), IPIs and all. + * + * Note that srcu_read_lock_lite() can be invoked only from those contexts + * where RCU is watching. Otherwise, lockdep will complain. + */ +static inline int srcu_read_lock_lite(struct srcu_struct *ssp) __acquires(ssp) +{ + int retval; + + srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_LITE); + retval = __srcu_read_lock_lite(ssp); + rcu_try_lock_acquire(&ssp->dep_map); + return retval; +} + /** * srcu_read_lock_nmisafe - register a new reader for an SRCU-protected structure. * @ssp: srcu_struct in which to register the new reader. @@ -325,6 +358,22 @@ static inline void srcu_read_unlock(struct srcu_struct *ssp, int idx) __srcu_read_unlock(ssp, idx); } +/** + * srcu_read_unlock_lite - unregister a old reader from an SRCU-protected structure. + * @ssp: srcu_struct in which to unregister the old reader. + * @idx: return value from corresponding srcu_read_lock(). + * + * Exit a light-weight SRCU read-side critical section. + */ +static inline void srcu_read_unlock_lite(struct srcu_struct *ssp, int idx) + __releases(ssp) +{ + WARN_ON_ONCE(idx & ~0x1); + srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_LITE); + srcu_lock_release(&ssp->dep_map); + __srcu_read_unlock(ssp, idx); +} + /** * srcu_read_unlock_nmisafe - unregister a old reader from an SRCU-protected structure. * @ssp: srcu_struct in which to unregister the old reader. diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index 79ad809c7f03..8074138cbd62 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -46,6 +46,7 @@ struct srcu_data { /* Values for ->srcu_reader_flavor. */ #define SRCU_READ_FLAVOR_NORMAL 0x1 // srcu_read_lock(). #define SRCU_READ_FLAVOR_NMI 0x2 // srcu_read_lock_nmisafe(). +#define SRCU_READ_FLAVOR_LITE 0x4 // srcu_read_lock_lite(). /* * Node in SRCU combining tree, similar in function to rcu_data. diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 602b4b8c4b89..8632a3caeb33 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -429,20 +429,29 @@ static bool srcu_gp_is_expedited(struct srcu_struct *ssp) } /* - * Returns approximate total of the readers' ->srcu_lock_count[] values - * for the rank of per-CPU counters specified by idx. + * Computes approximate total of the readers' ->srcu_lock_count[] values + * for the rank of per-CPU counters specified by idx, and returns true if + * the caller did the proper barrier (gp), and if the count of the locks + * matches that of the unlocks passed in. */ -static unsigned long srcu_readers_lock_idx(struct srcu_struct *ssp, int idx) +static bool srcu_readers_lock_idx(struct srcu_struct *ssp, int idx, bool gp, unsigned long unlocks) { int cpu; + unsigned long mask = 0; unsigned long sum = 0; for_each_possible_cpu(cpu) { struct srcu_data *sdp = per_cpu_ptr(ssp->sda, cpu); sum += atomic_long_read(&sdp->srcu_lock_count[idx]); + if (IS_ENABLED(CONFIG_PROVE_RCU)) + mask = mask | READ_ONCE(sdp->srcu_reader_flavor); } - return sum; + WARN_ONCE(IS_ENABLED(CONFIG_PROVE_RCU) && (mask & (mask - 1)), + "Mixed reader flavors for srcu_struct at %ps.\n", ssp); + if (mask & SRCU_READ_FLAVOR_LITE && !gp) + return false; + return sum == unlocks; } /* @@ -473,6 +482,7 @@ static unsigned long srcu_readers_unlock_idx(struct srcu_struct *ssp, int idx) */ static bool srcu_readers_active_idx_check(struct srcu_struct *ssp, int idx) { + bool did_gp = !!(raw_cpu_read(ssp->sda->srcu_reader_flavor) & SRCU_READ_FLAVOR_LITE); unsigned long unlocks; unlocks = srcu_readers_unlock_idx(ssp, idx); @@ -482,13 +492,16 @@ static bool srcu_readers_active_idx_check(struct srcu_struct *ssp, int idx) * unlock is counted. Needs to be a smp_mb() as the read side may * contain a read from a variable that is written to before the * synchronize_srcu() in the write side. In this case smp_mb()s - * A and B act like the store buffering pattern. + * A and B (or X and Y) act like the store buffering pattern. * - * This smp_mb() also pairs with smp_mb() C to prevent accesses - * after the synchronize_srcu() from being executed before the - * grace period ends. + * This smp_mb() also pairs with smp_mb() C (or, in the case of X, + * Z) to prevent accesses after the synchronize_srcu() from being + * executed before the grace period ends. */ - smp_mb(); /* A */ + if (!did_gp) + smp_mb(); /* A */ + else + synchronize_rcu(); /* X */ /* * If the locks are the same as the unlocks, then there must have @@ -546,7 +559,7 @@ static bool srcu_readers_active_idx_check(struct srcu_struct *ssp, int idx) * which are unlikely to be configured with an address space fully * populated with memory, at least not anytime soon. */ - return srcu_readers_lock_idx(ssp, idx) == unlocks; + return srcu_readers_lock_idx(ssp, idx, did_gp, unlocks); } /** @@ -750,6 +763,47 @@ void __srcu_read_unlock(struct srcu_struct *ssp, int idx) } EXPORT_SYMBOL_GPL(__srcu_read_unlock); +/* + * Counts the new reader in the appropriate per-CPU element of the + * srcu_struct. Returns an index that must be passed to the matching + * srcu_read_unlock_lite(). + * + * Note that this_cpu_inc() is an RCU read-side critical section either + * because it disables interrupts, because it is a single instruction, + * or because it is a read-modify-write atomic operation, depending on + * the whims of the architecture. + */ +int __srcu_read_lock_lite(struct srcu_struct *ssp) +{ + int idx; + + RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_lock_lite()."); + idx = READ_ONCE(ssp->srcu_idx) & 0x1; + this_cpu_inc(ssp->sda->srcu_lock_count[idx].counter); /* Y */ + barrier(); /* Avoid leaking the critical section. */ + return idx; +} +EXPORT_SYMBOL_GPL(__srcu_read_lock_lite); + +/* + * Removes the count for the old reader from the appropriate + * per-CPU element of the srcu_struct. Note that this may well be a + * different CPU than that which was incremented by the corresponding + * srcu_read_lock_lite(), but it must be within the same task. + * + * Note that this_cpu_inc() is an RCU read-side critical section either + * because it disables interrupts, because it is a single instruction, + * or because it is a read-modify-write atomic operation, depending on + * the whims of the architecture. + */ +void __srcu_read_unlock_lite(struct srcu_struct *ssp, int idx) +{ + barrier(); /* Avoid leaking the critical section. */ + this_cpu_inc(ssp->sda->srcu_unlock_count[idx].counter); /* Z */ + RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_unlock_lite()."); +} +EXPORT_SYMBOL_GPL(__srcu_read_unlock_lite); + #ifdef CONFIG_NEED_SRCU_NMI_SAFE /* @@ -1134,6 +1188,8 @@ static void srcu_flip(struct srcu_struct *ssp) * it stays until either (1) Compilers learn about this sort of * control dependency or (2) Some production workload running on * a production system is unduly delayed by this slowpath smp_mb(). + * Except for _lite() readers, where it is inoperative, which + * means that it is a good thing that it is redundant. */ smp_mb(); /* E */ /* Pairs with B and C. */ @@ -1152,7 +1208,8 @@ static void srcu_flip(struct srcu_struct *ssp) /* * If SRCU is likely idle, in other words, the next SRCU grace period - * should be expedited, return true, otherwise return false. + * should be expedited, return true, otherwise return false. Except that + * in the presence of _lite() readers, always return false. * * Note that it is OK for several current from-idle requests for a new * grace period from idle to specify expediting because they will all end @@ -1181,6 +1238,9 @@ static bool srcu_should_expedite(struct srcu_struct *ssp) unsigned long tlast; check_init_srcu_struct(ssp); + /* If _lite() readers, don't do unsolicited expediting. */ + if (this_cpu_read(ssp->sda->srcu_reader_flavor) & SRCU_READ_FLAVOR_LITE) + return false; /* If the local srcu_data structure has callbacks, not idle. */ sdp = raw_cpu_ptr(ssp->sda); spin_lock_irqsave_rcu_node(sdp, flags); From patchwork Tue Nov 12 14:51:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 13872356 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C96BB20110A; Tue, 12 Nov 2024 14:52:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731423166; cv=none; b=JndNwrO7awSG+eAgadsiX8bcLxavdcONW/pDzcC0cUmnJKVdRJhLKEtelPXEky4neUDQxgmunkOKi4KuEktEJjk+Sl544xH/KpOi2SRz+xtPTF6IDGSqEPOoZswtrylJACCFUndwwiPhxLiFj00EVVvlxcNj/9FsXVVWCVWrGlg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731423166; c=relaxed/simple; bh=bGavktbS0y6IOUlaxf8zrMDlcwBDfIwStxF+Yf1IcC4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=SM9hlWRH8/NxuauPTWCjG6OVNIsSU5z99JUC/ngS3J9XS/dqgWBVarCvziSKDjTt/PsG72WnMxoG6YY+3ZqI5BX8p2U2npiIjlOtflKddg9ZsVtKphrqaSzLRQCpo/KK8ISt9KIod7xvXZQWjx4graWtScUIky8dIdAv8HyqdUs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=fbZ6uZg4; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="fbZ6uZg4" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 993A5C4CED5; Tue, 12 Nov 2024 14:52:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1731423166; bh=bGavktbS0y6IOUlaxf8zrMDlcwBDfIwStxF+Yf1IcC4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=fbZ6uZg4jKJq2uakfy+DBQf0EhEaBBtaiLkatmDhtTqAFVa+uKno7yqcmgKPkpsYp yl73xNOy6jd/kTPOBO6/fbFE/yZmIQ+wOnxTrXKjyDjEyIxPOrpj4uUUFaoMwbXU6W YBFWdr8ejxCKrGrQN4nag3FEVKJTSlp1iIwaWwIhIg4el3yeVfEUIjbxVSNb4EyUxQ yVn92ZVDZVG2VoQIZE+dhaLQwjfRBx6MD2H9I4tIp1yu7S7p8XggORLfk7RF5qwJja xVk0eX3qwrD+AGxFcZRmGjUcoAzRUpIzxpwFVyPBMBTogdQrytgUkPVz8EXED1KjlV emOke24XiZuuQ== From: Frederic Weisbecker To: LKML Cc: "Paul E. McKenney" , Boqun Feng , Joel Fernandes , Josh Triplett , Lai Jiangshan , Mathieu Desnoyers , Neeraj Upadhyay , Steven Rostedt , Uladzislau Rezki , Zqiang , rcu , Alexei Starovoitov , Andrii Nakryiko , Peter Zijlstra , Kent Overstreet , bpf@vger.kernel.org, Neeraj Upadhyay , Frederic Weisbecker Subject: [PATCH 10/16] srcu: Allow inlining of __srcu_read_{,un}lock_lite() Date: Tue, 12 Nov 2024 15:51:53 +0100 Message-ID: <20241112145159.23032-11-frederic@kernel.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20241112145159.23032-1-frederic@kernel.org> References: <20241112145159.23032-1-frederic@kernel.org> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" This commit moves __srcu_read_lock_lite() and __srcu_read_unlock_lite() into include/linux/srcu.h and marks them "static inline" so that they can be inlined into srcu_read_lock_lite() and srcu_read_unlock_lite(), respectively. They are not hand-inlined due to Tree SRCU and Tiny SRCU having different implementations. The earlier removal of smp_mb() combined with the inlining produce significant single-percentage performance wins. Link: https://lore.kernel.org/all/CAEf4BzYgiNmSb=ZKQ65tm6nJDi1UX2Gq26cdHSH1mPwXJYZj5g@mail.gmail.com/ Reported-by: Alexei Starovoitov Signed-off-by: Paul E. McKenney Cc: Alexei Starovoitov Cc: Andrii Nakryiko Cc: Peter Zijlstra Cc: Kent Overstreet Cc: Reviewed-by: Neeraj Upadhyay Signed-off-by: Frederic Weisbecker --- include/linux/srcutree.h | 39 ++++++++++++++++++++++++++++++++++++++ kernel/rcu/srcutree.c | 41 ---------------------------------------- 2 files changed, 39 insertions(+), 41 deletions(-) diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index 8074138cbd62..778eb61542e1 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -209,4 +209,43 @@ void synchronize_srcu_expedited(struct srcu_struct *ssp); void srcu_barrier(struct srcu_struct *ssp); void srcu_torture_stats_print(struct srcu_struct *ssp, char *tt, char *tf); +/* + * Counts the new reader in the appropriate per-CPU element of the + * srcu_struct. Returns an index that must be passed to the matching + * srcu_read_unlock_lite(). + * + * Note that this_cpu_inc() is an RCU read-side critical section either + * because it disables interrupts, because it is a single instruction, + * or because it is a read-modify-write atomic operation, depending on + * the whims of the architecture. + */ +static inline int __srcu_read_lock_lite(struct srcu_struct *ssp) +{ + int idx; + + RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_lock_lite()."); + idx = READ_ONCE(ssp->srcu_idx) & 0x1; + this_cpu_inc(ssp->sda->srcu_lock_count[idx].counter); /* Y */ + barrier(); /* Avoid leaking the critical section. */ + return idx; +} + +/* + * Removes the count for the old reader from the appropriate + * per-CPU element of the srcu_struct. Note that this may well be a + * different CPU than that which was incremented by the corresponding + * srcu_read_lock_lite(), but it must be within the same task. + * + * Note that this_cpu_inc() is an RCU read-side critical section either + * because it disables interrupts, because it is a single instruction, + * or because it is a read-modify-write atomic operation, depending on + * the whims of the architecture. + */ +static inline void __srcu_read_unlock_lite(struct srcu_struct *ssp, int idx) +{ + barrier(); /* Avoid leaking the critical section. */ + this_cpu_inc(ssp->sda->srcu_unlock_count[idx].counter); /* Z */ + RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_unlock_lite()."); +} + #endif diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 8632a3caeb33..d3a0c76ce590 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -763,47 +763,6 @@ void __srcu_read_unlock(struct srcu_struct *ssp, int idx) } EXPORT_SYMBOL_GPL(__srcu_read_unlock); -/* - * Counts the new reader in the appropriate per-CPU element of the - * srcu_struct. Returns an index that must be passed to the matching - * srcu_read_unlock_lite(). - * - * Note that this_cpu_inc() is an RCU read-side critical section either - * because it disables interrupts, because it is a single instruction, - * or because it is a read-modify-write atomic operation, depending on - * the whims of the architecture. - */ -int __srcu_read_lock_lite(struct srcu_struct *ssp) -{ - int idx; - - RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_lock_lite()."); - idx = READ_ONCE(ssp->srcu_idx) & 0x1; - this_cpu_inc(ssp->sda->srcu_lock_count[idx].counter); /* Y */ - barrier(); /* Avoid leaking the critical section. */ - return idx; -} -EXPORT_SYMBOL_GPL(__srcu_read_lock_lite); - -/* - * Removes the count for the old reader from the appropriate - * per-CPU element of the srcu_struct. Note that this may well be a - * different CPU than that which was incremented by the corresponding - * srcu_read_lock_lite(), but it must be within the same task. - * - * Note that this_cpu_inc() is an RCU read-side critical section either - * because it disables interrupts, because it is a single instruction, - * or because it is a read-modify-write atomic operation, depending on - * the whims of the architecture. - */ -void __srcu_read_unlock_lite(struct srcu_struct *ssp, int idx) -{ - barrier(); /* Avoid leaking the critical section. */ - this_cpu_inc(ssp->sda->srcu_unlock_count[idx].counter); /* Z */ - RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_unlock_lite()."); -} -EXPORT_SYMBOL_GPL(__srcu_read_unlock_lite); - #ifdef CONFIG_NEED_SRCU_NMI_SAFE /* From patchwork Tue Nov 12 14:51:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 13872357 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 98F731FF7AF; Tue, 12 Nov 2024 14:52:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731423170; cv=none; b=AO9uBYg3CliH5qJBTbV5KodS8sGVmSEh+z59zDUgfBckOJQab2HhtdNt8ztdRsbcFlBX2n63uNun+AG370W2oM5QbQ6rXp/GVYEYqYKfoCnyyV21QAthmdwyghoHQ0R4VlJqUPCtb4x2KKCEf8R0gtWMEwdEQKJq/vYJLNa0s/M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731423170; c=relaxed/simple; bh=eTwF1Vp7R6rmPCXlAVbdnCiKJbBVd/JbVkLnHoPeyJk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=iWB/vs0XX3nnHo+9lsTF28whA3IaT3udp649sxPCZjMVxIAw0dXbiz3VgpDlXAMI+6lP928RJlUhXy16q9QMdUPD4ZvTNFXAE05myDDZjL6xpo9ykRO2FVyVK/CCDzh05ISkIuhC3qgk3o2AyIkdJxmCRA70a5figmE7tOstons= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=m3b8kn2A; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="m3b8kn2A" Received: by smtp.kernel.org (Postfix) with ESMTPSA id C14EEC4CED0; Tue, 12 Nov 2024 14:52:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1731423170; bh=eTwF1Vp7R6rmPCXlAVbdnCiKJbBVd/JbVkLnHoPeyJk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=m3b8kn2AjbO/2Hzdk1eUXP2nXskxOj8F3tA5UULmRxDtQ8RDIC+4tECD3kpz776Xl Lziwr4SdXX6FdOFbL381W2JqtWvcet79hBFFSo3sVBBHQdzaCoWkiATP9ZoAT0ZOPI z42i9QxILWsAAt+nyON57yjt8hkf0uWD1nN7wPc4FNNv3l0VvQSMJ4gTl4elIZusuV Y0lsts/mTbr42pejx2VW9zxNiOC8BiZUX60F8KhOxWcHXW5i9F+qOLVlP6WjfSrETl f5+p4r3675cdX8zAZi5JiPJOgGUayyk8hFilnZRAbvsB00tfiWr5XViYkT4R4/3eKC 9RS9KYHuGtJNg== From: Frederic Weisbecker To: LKML Cc: "Paul E. McKenney" , Boqun Feng , Joel Fernandes , Josh Triplett , Lai Jiangshan , Mathieu Desnoyers , Neeraj Upadhyay , Steven Rostedt , Uladzislau Rezki , Zqiang , rcu , Alexei Starovoitov , Andrii Nakryiko , Peter Zijlstra , Kent Overstreet , bpf@vger.kernel.org, Neeraj Upadhyay , Frederic Weisbecker Subject: [PATCH 11/16] rcutorture: Expand RCUTORTURE_RDR_MASK_[12] to eight bits Date: Tue, 12 Nov 2024 15:51:54 +0100 Message-ID: <20241112145159.23032-12-frederic@kernel.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20241112145159.23032-1-frederic@kernel.org> References: <20241112145159.23032-1-frederic@kernel.org> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" This commit prepares for testing of multiple SRCU reader flavors by expanding RCUTORTURE_RDR_MASK_1 and RCUTORTURE_RDR_MASK_2 from a single bit to eight bits, allowing them to accommodate the return values from multiple calls to srcu_read_lock*(). This will in turn permit better testing coverage for these SRCU reader flavors, including testing of the diagnostics for inproper use of mixed reader flavors. Signed-off-by: Paul E. McKenney Cc: Alexei Starovoitov Cc: Andrii Nakryiko Cc: Peter Zijlstra Cc: Kent Overstreet Cc: Reviewed-by: Neeraj Upadhyay Signed-off-by: Frederic Weisbecker --- kernel/rcu/rcutorture.c | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c index bb75dbf5c800..f96ab98f8182 100644 --- a/kernel/rcu/rcutorture.c +++ b/kernel/rcu/rcutorture.c @@ -57,9 +57,9 @@ MODULE_AUTHOR("Paul E. McKenney and Josh Triplett extendables field, extendables param, and related definitions. */ #define RCUTORTURE_RDR_SHIFT_1 8 /* Put SRCU index in upper bits. */ -#define RCUTORTURE_RDR_MASK_1 (1 << RCUTORTURE_RDR_SHIFT_1) -#define RCUTORTURE_RDR_SHIFT_2 9 /* Put SRCU index in upper bits. */ -#define RCUTORTURE_RDR_MASK_2 (1 << RCUTORTURE_RDR_SHIFT_2) +#define RCUTORTURE_RDR_MASK_1 (0xff << RCUTORTURE_RDR_SHIFT_1) +#define RCUTORTURE_RDR_SHIFT_2 16 /* Put SRCU index in upper bits. */ +#define RCUTORTURE_RDR_MASK_2 (0xff << RCUTORTURE_RDR_SHIFT_2) #define RCUTORTURE_RDR_BH 0x01 /* Extend readers by disabling bh. */ #define RCUTORTURE_RDR_IRQ 0x02 /* ... disabling interrupts. */ #define RCUTORTURE_RDR_PREEMPT 0x04 /* ... disabling preemption. */ @@ -71,6 +71,9 @@ MODULE_AUTHOR("Paul E. McKenney and Josh Triplett > RCUTORTURE_RDR_SHIFT_2) > 1); + WARN_ON_ONCE(idxold2 & ~RCUTORTURE_RDR_ALLBITS); rtrsp->rt_readstate = newstate; /* First, put new protection in place to avoid critical-section gap. */ @@ -1835,9 +1838,9 @@ static void rcutorture_one_extend(int *readstate, int newstate, if (statesnew & RCUTORTURE_RDR_SCHED) rcu_read_lock_sched(); if (statesnew & RCUTORTURE_RDR_RCU_1) - idxnew1 = (cur_ops->readlock() & 0x1) << RCUTORTURE_RDR_SHIFT_1; + idxnew1 = (cur_ops->readlock() << RCUTORTURE_RDR_SHIFT_1) & RCUTORTURE_RDR_MASK_1; if (statesnew & RCUTORTURE_RDR_RCU_2) - idxnew2 = (cur_ops->readlock() & 0x1) << RCUTORTURE_RDR_SHIFT_2; + idxnew2 = (cur_ops->readlock() << RCUTORTURE_RDR_SHIFT_2) & RCUTORTURE_RDR_MASK_2; /* * Next, remove old protection, in decreasing order of strength @@ -1857,7 +1860,7 @@ static void rcutorture_one_extend(int *readstate, int newstate, if (statesold & RCUTORTURE_RDR_RBH) rcu_read_unlock_bh(); if (statesold & RCUTORTURE_RDR_RCU_2) { - cur_ops->readunlock((idxold2 >> RCUTORTURE_RDR_SHIFT_2) & 0x1); + cur_ops->readunlock((idxold2 & RCUTORTURE_RDR_MASK_2) >> RCUTORTURE_RDR_SHIFT_2); WARN_ON_ONCE(idxnew2 != -1); idxold2 = 0; } @@ -1867,7 +1870,7 @@ static void rcutorture_one_extend(int *readstate, int newstate, lockit = !cur_ops->no_pi_lock && !statesnew && !(torture_random(trsp) & 0xffff); if (lockit) raw_spin_lock_irqsave(¤t->pi_lock, flags); - cur_ops->readunlock((idxold1 >> RCUTORTURE_RDR_SHIFT_1) & 0x1); + cur_ops->readunlock((idxold1 & RCUTORTURE_RDR_MASK_1) >> RCUTORTURE_RDR_SHIFT_1); WARN_ON_ONCE(idxnew1 != -1); idxold1 = 0; if (lockit) @@ -1882,16 +1885,13 @@ static void rcutorture_one_extend(int *readstate, int newstate, if (idxnew1 == -1) idxnew1 = idxold1 & RCUTORTURE_RDR_MASK_1; WARN_ON_ONCE(idxnew1 < 0); - if (WARN_ON_ONCE((idxnew1 >> RCUTORTURE_RDR_SHIFT_1) > 1)) - pr_info("Unexpected idxnew1 value of %#x\n", idxnew1); if (idxnew2 == -1) idxnew2 = idxold2 & RCUTORTURE_RDR_MASK_2; WARN_ON_ONCE(idxnew2 < 0); - WARN_ON_ONCE((idxnew2 >> RCUTORTURE_RDR_SHIFT_2) > 1); *readstate = idxnew1 | idxnew2 | newstate; WARN_ON_ONCE(*readstate < 0); - if (WARN_ON_ONCE((*readstate >> RCUTORTURE_RDR_SHIFT_2) > 1)) - pr_info("Unexpected idxnew2 value of %#x\n", idxnew2); + if (WARN_ON_ONCE(*readstate & ~RCUTORTURE_RDR_ALLBITS)) + pr_info("Unexpected readstate value of %#x\n", *readstate); } /* Return the biggest extendables mask given current RCU and boot parameters. */ @@ -1916,7 +1916,7 @@ rcutorture_extend_mask(int oldmask, struct torture_random_state *trsp) unsigned long preempts_irq = preempts | RCUTORTURE_RDR_IRQ; unsigned long bhs = RCUTORTURE_RDR_BH | RCUTORTURE_RDR_RBH; - WARN_ON_ONCE(mask >> RCUTORTURE_RDR_SHIFT_1); + WARN_ON_ONCE(mask >> RCUTORTURE_RDR_SHIFT_1); // Can't have reader idx bits. /* Mostly only one bit (need preemption!), sometimes lots of bits. */ if (!(randmask1 & 0x7)) mask = mask & randmask2; From patchwork Tue Nov 12 14:51:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 13872358 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E956D1FC7FF; Tue, 12 Nov 2024 14:52:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731423175; cv=none; b=bdXk2PJH71nGFkBuT7pz44HZEJfBYB8YCq7cYAXUk+NcCie8P/4/dOlGg6ZIV4UFsTe2DrsPvAlFbZpaqKgqpzuf9xaYI5GjjVRSANf/sUdS8X1TfdWrypJ9M2/h+EXOffA5Jp0Vr/cjqfMS+juEQVW9x0KSyX/c4hs8PJ8SJWk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731423175; c=relaxed/simple; bh=nwvp2zP98ddDicB9bTU8Bfx4YUep7GYbMP51FFzAyzI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Bwife1PDpoyOUTVsivN/b2OeowlTtCrEcpFlZPaBcr4IvADongW4OgcMdoz6kYZ7WNZ21w/yjzAeyn6rbKxz5ja9apN3Bv/ht6OnZUSxfrilrZeriQtZYCsefuQ/3FT1s8a9yWkrXtozgwMp8yFigqZVz2y9+C/BxpJTJWNbRNo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=tfbzR/zI; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="tfbzR/zI" Received: by smtp.kernel.org (Postfix) with ESMTPSA id D8A4EC4CED5; Tue, 12 Nov 2024 14:52:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1731423174; bh=nwvp2zP98ddDicB9bTU8Bfx4YUep7GYbMP51FFzAyzI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=tfbzR/zIwPybRU4kHWdRwXJC90OaHRePLkpfDo6BXICoCko2XfJ3ITVSRCo8f57pY EOoVZySC+3ln6C3ei5iOL3L9IFMINF6htbqW2Vc/vWhqHlV77Of2B/ogmT3RNauq5C k28UrH8JTCg+VWkxxuQwMqyN03puoGx0gfvTQDJWaFAfBu3IiYaAWvqqXm1qF6cqrk YtWcJeM3U61O1M0+fAej6UgO5+s7LcwJoXkjda1NUG1V7e+qUOgtcXxKApmmuNnZQ8 irl2ywmN84qDYqB4ECwqWTH34DY8jZ/g2hvwxTXvM25DKAbIzEv+dRFVmhUgk3qQ+x sYDC3FEKB45iA== From: Frederic Weisbecker To: LKML Cc: "Paul E. McKenney" , Boqun Feng , Joel Fernandes , Josh Triplett , Lai Jiangshan , Mathieu Desnoyers , Neeraj Upadhyay , Steven Rostedt , Uladzislau Rezki , Zqiang , rcu , Alexei Starovoitov , Andrii Nakryiko , Peter Zijlstra , Kent Overstreet , bpf@vger.kernel.org, Neeraj Upadhyay , Frederic Weisbecker Subject: [PATCH 12/16] rcutorture: Add reader_flavor parameter for SRCU readers Date: Tue, 12 Nov 2024 15:51:55 +0100 Message-ID: <20241112145159.23032-13-frederic@kernel.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20241112145159.23032-1-frederic@kernel.org> References: <20241112145159.23032-1-frederic@kernel.org> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" This commit adds an rcutorture.reader_flavor parameter whose bits correspond to reader flavors. For example, SRCU's readers are 0x1 for normal and 0x2 for NMI-safe. Signed-off-by: Paul E. McKenney Cc: Alexei Starovoitov Cc: Andrii Nakryiko Cc: Peter Zijlstra Cc: Kent Overstreet Cc: Reviewed-by: Neeraj Upadhyay Signed-off-by: Frederic Weisbecker --- .../admin-guide/kernel-parameters.txt | 8 +++++ kernel/rcu/rcutorture.c | 30 ++++++++++++++----- 2 files changed, 30 insertions(+), 8 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 1518343bbe22..52922727006f 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -5426,6 +5426,14 @@ The delay, in seconds, between successive read-then-exit testing episodes. + rcutorture.reader_flavor= [KNL] + A bit mask indicating which readers to use. + If there is more than one bit set, the readers + are entered from low-order bit up, and are + exited in the opposite order. For SRCU, the + 0x1 bit is normal readers and the 0x2 bit is + for NMI-safe readers. + rcutorture.shuffle_interval= [KNL] Set task-shuffle interval (s). Shuffling tasks allows some CPUs to go into dyntick-idle mode diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c index f96ab98f8182..405decec3367 100644 --- a/kernel/rcu/rcutorture.c +++ b/kernel/rcu/rcutorture.c @@ -111,6 +111,7 @@ torture_param(int, nocbs_nthreads, 0, "Number of NOCB toggle threads, 0 to disab torture_param(int, nocbs_toggle, 1000, "Time between toggling nocb state (ms)"); torture_param(int, read_exit_delay, 13, "Delay between read-then-exit episodes (s)"); torture_param(int, read_exit_burst, 16, "# of read-then-exit bursts per episode, zero to disable"); +torture_param(int, reader_flavor, 0x1, "Reader flavors to use, one per bit."); torture_param(int, shuffle_interval, 3, "Number of seconds between shuffles"); torture_param(int, shutdown_secs, 0, "Shutdown time (s), <= zero to disable."); torture_param(int, stall_cpu, 0, "Stall duration (s), zero to disable."); @@ -644,10 +645,20 @@ static void srcu_get_gp_data(int *flags, unsigned long *gp_seq) static int srcu_torture_read_lock(void) { - if (cur_ops == &srcud_ops) - return srcu_read_lock_nmisafe(srcu_ctlp); - else - return srcu_read_lock(srcu_ctlp); + int idx; + int ret = 0; + + if ((reader_flavor & 0x1) || !(reader_flavor & 0x7)) { + idx = srcu_read_lock(srcu_ctlp); + WARN_ON_ONCE(idx & ~0x1); + ret += idx; + } + if (reader_flavor & 0x2) { + idx = srcu_read_lock_nmisafe(srcu_ctlp); + WARN_ON_ONCE(idx & ~0x1); + ret += idx << 1; + } + return ret; } static void @@ -671,10 +682,11 @@ srcu_read_delay(struct torture_random_state *rrsp, struct rt_read_seg *rtrsp) static void srcu_torture_read_unlock(int idx) { - if (cur_ops == &srcud_ops) - srcu_read_unlock_nmisafe(srcu_ctlp, idx); - else - srcu_read_unlock(srcu_ctlp, idx); + WARN_ON_ONCE((reader_flavor && (idx & ~reader_flavor)) || (!reader_flavor && (idx & ~0x1))); + if (reader_flavor & 0x2) + srcu_read_unlock_nmisafe(srcu_ctlp, (idx & 0x2) >> 1); + if ((reader_flavor & 0x1) || !(reader_flavor & 0x7)) + srcu_read_unlock(srcu_ctlp, idx & 0x1); } static int torture_srcu_read_lock_held(void) @@ -2389,6 +2401,7 @@ rcu_torture_print_module_parms(struct rcu_torture_ops *cur_ops, const char *tag) "n_barrier_cbs=%d " "onoff_interval=%d onoff_holdoff=%d " "read_exit_delay=%d read_exit_burst=%d " + "reader_flavor=%x " "nocbs_nthreads=%d nocbs_toggle=%d " "test_nmis=%d\n", torture_type, tag, nrealreaders, nfakewriters, @@ -2401,6 +2414,7 @@ rcu_torture_print_module_parms(struct rcu_torture_ops *cur_ops, const char *tag) n_barrier_cbs, onoff_interval, onoff_holdoff, read_exit_delay, read_exit_burst, + reader_flavor, nocbs_nthreads, nocbs_toggle, test_nmis); } From patchwork Tue Nov 12 14:51:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 13872359 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9DEDB1FC7FF; Tue, 12 Nov 2024 14:52:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731423180; cv=none; b=iDsXcaNgYEGY7qwCyZLB7dPTPapjsc8kvX4dv4A3zh3oe79USoIACwXyv5kKlXg1Dyfak9oV9Kyc3oAhMMF7YHkrY66C08Xtnq8URaC4pPbXkYJ5CnA27ghCrpGxqeV7HqwKwGBrmihU/EJdfwuu6/Ou+GjXARvLqc28FEqsvmI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731423180; c=relaxed/simple; bh=89PLNAGurys0mJBxDA/ZpZpRTaoDP8T4R09w6xw7TlU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=rUELgmAT3zjCHc9M568NuinEdiESeyqX83ceOo84vHrwfO35XlaphvHbBhfOrj+lL0CmMKj3mkz9YSdEbRC3ewCMx+QyZ+N8s7W8rwDWwr5Y+6ZjE7CvhwNBKxiQd+V80Ho6PwJdSl7iseECAGrFvk0l5vGns4tc8YF8rlz2oUU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=XXoJnIX9; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="XXoJnIX9" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 36055C4CED8; Tue, 12 Nov 2024 14:52:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1731423179; bh=89PLNAGurys0mJBxDA/ZpZpRTaoDP8T4R09w6xw7TlU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=XXoJnIX9pE8GJIgPVMl6V0UxchgE3PpNmtPgDuER5xM7+oAv98KyfhztiNFbZvv3I Gv0irBUQjJBkLD4m7c/aTcktoCrfPCtma5YaS6M9/8jEVVaRmGA03wbFM5L8p+kxCy MU6yC1wN3RSeXo7B9TEfOySx1wsZ3wSRgBQVCGTYdgC+jfYgszTra20vOzefUoFiZ1 HSyvwVjm2nfHA6sQsF/9SfYYR2/8QIzn6VVBWVOYfhOXMfEPsYLMspw4nnPkHRpBRO afw7nCn2A6tIfzPqNzwqFmZ/wjhDS9BibAJxmTLZK5fYaIqr90NCZqDPMgugY9KUtz fJ8B3OOOOX2lA== From: Frederic Weisbecker To: LKML Cc: "Paul E. McKenney" , Boqun Feng , Joel Fernandes , Josh Triplett , Lai Jiangshan , Mathieu Desnoyers , Neeraj Upadhyay , Steven Rostedt , Uladzislau Rezki , Zqiang , rcu , Alexei Starovoitov , Andrii Nakryiko , Peter Zijlstra , Kent Overstreet , bpf@vger.kernel.org, Neeraj Upadhyay , Frederic Weisbecker Subject: [PATCH 13/16] rcutorture: Add srcu_read_lock_lite() support to rcutorture.reader_flavor Date: Tue, 12 Nov 2024 15:51:56 +0100 Message-ID: <20241112145159.23032-14-frederic@kernel.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20241112145159.23032-1-frederic@kernel.org> References: <20241112145159.23032-1-frederic@kernel.org> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" This commit causes bit 0x4 of rcutorture.reader_flavor to select the new srcu_read_lock_lite() and srcu_read_unlock_lite() functions. Signed-off-by: Paul E. McKenney Cc: Alexei Starovoitov Cc: Andrii Nakryiko Cc: Peter Zijlstra Cc: Kent Overstreet Cc: Reviewed-by: Neeraj Upadhyay Signed-off-by: Frederic Weisbecker --- Documentation/admin-guide/kernel-parameters.txt | 4 ++-- kernel/rcu/rcutorture.c | 7 +++++++ 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 52922727006f..203ec51e41d4 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -5431,8 +5431,8 @@ If there is more than one bit set, the readers are entered from low-order bit up, and are exited in the opposite order. For SRCU, the - 0x1 bit is normal readers and the 0x2 bit is - for NMI-safe readers. + 0x1 bit is normal readers, 0x2 NMI-safe readers, + and 0x4 light-weight readers. rcutorture.shuffle_interval= [KNL] Set task-shuffle interval (s). Shuffling tasks diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c index 405decec3367..a313cdcb0960 100644 --- a/kernel/rcu/rcutorture.c +++ b/kernel/rcu/rcutorture.c @@ -658,6 +658,11 @@ static int srcu_torture_read_lock(void) WARN_ON_ONCE(idx & ~0x1); ret += idx << 1; } + if (reader_flavor & 0x4) { + idx = srcu_read_lock_lite(srcu_ctlp); + WARN_ON_ONCE(idx & ~0x1); + ret += idx << 2; + } return ret; } @@ -683,6 +688,8 @@ srcu_read_delay(struct torture_random_state *rrsp, struct rt_read_seg *rtrsp) static void srcu_torture_read_unlock(int idx) { WARN_ON_ONCE((reader_flavor && (idx & ~reader_flavor)) || (!reader_flavor && (idx & ~0x1))); + if (reader_flavor & 0x4) + srcu_read_unlock_lite(srcu_ctlp, (idx & 0x4) >> 2); if (reader_flavor & 0x2) srcu_read_unlock_nmisafe(srcu_ctlp, (idx & 0x2) >> 1); if ((reader_flavor & 0x1) || !(reader_flavor & 0x7)) From patchwork Tue Nov 12 14:51:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 13872360 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9E1E11FF7DF; Tue, 12 Nov 2024 14:53:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731423183; cv=none; b=iYim1ZWYX3k0nJSaBVP0iFMd80keuMn2wYBgLPa1Klhh2g6KrUA9WARqRfrI95wjcoDsqT2SqyRA7jp4zBm/cHxlXdQcKPpdqpnia7qeawT3PVquukukJdhyYHnfx4rtRK6LXh9NwqkA9ON/de75EWA/2z42ZqhPL4yuke0vNcY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731423183; c=relaxed/simple; bh=C5DvJ9v6NDVzUTyLbZO2R6cXGi2WrGyjqmelZ/LOsj4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Vy+fQmPj4c06AsfnWR6RIOzOHzGCZl8FKAMgwkpCtsJsEEp2RhDM1ICic7ssBgPVnJIjZRLsTVIHoOfZMMo6c7HHiX2HuNf5/65A4IQV3GlC1OZTYIYwpfRADoTdgzGnzOlB9MEBhzYuUlf/xSHcKvAgMYFuGPOItFVfJfHhRn4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=s5ERMtBi; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="s5ERMtBi" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9CC68C4CECD; Tue, 12 Nov 2024 14:52:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1731423183; bh=C5DvJ9v6NDVzUTyLbZO2R6cXGi2WrGyjqmelZ/LOsj4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=s5ERMtBinGrWLNvuQa3lnWb/5AfUahCSWRxf+4D0BDpSjtTVtwbG5fyxUShkfZC2H uFmqO7387uTsRHvQUAg4lkgaJMjwzxQi4WKak2J191ixJrnAySNb7JKQe8mPur/Mxg I+kgPu3wT3teEhCr6R9LDpLL1lQRd7cJSVOM0OhphJ/Rn8e9ZeNg5fePYcad01q3tH 9crG0HBPopZ1RDgNQ+aR26/8cH+GDdtjDbfNu/dus5O1Iz+2mZH1l8zjWjxdBn8Nhl Ba0avAhxESRMZzCwjTcgPbQYjpDcjrL4AyLXyP4M8NTuJQz5IN+7t9IcWn4KyxouGZ 6N1ojdybd+Jlg== From: Frederic Weisbecker To: LKML Cc: "Paul E. McKenney" , Boqun Feng , Joel Fernandes , Josh Triplett , Lai Jiangshan , Mathieu Desnoyers , Neeraj Upadhyay , Steven Rostedt , Uladzislau Rezki , Zqiang , rcu , Alexei Starovoitov , Andrii Nakryiko , Peter Zijlstra , Kent Overstreet , bpf@vger.kernel.org, Neeraj Upadhyay , Frederic Weisbecker Subject: [PATCH 14/16] rcutorture: Add light-weight SRCU scenario Date: Tue, 12 Nov 2024 15:51:57 +0100 Message-ID: <20241112145159.23032-15-frederic@kernel.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20241112145159.23032-1-frederic@kernel.org> References: <20241112145159.23032-1-frederic@kernel.org> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" This commit adds an rcutorture scenario that tests light-weight SRCU readers. While in the area, it adjusts the size of the TREE10 scenario. Signed-off-by: Paul E. McKenney Cc: Alexei Starovoitov Cc: Andrii Nakryiko Cc: Peter Zijlstra Cc: Kent Overstreet Cc: Reviewed-by: Neeraj Upadhyay Signed-off-by: Frederic Weisbecker --- tools/testing/selftests/rcutorture/configs/rcu/CFLIST | 1 + tools/testing/selftests/rcutorture/configs/rcu/SRCU-L | 10 ++++++++++ .../selftests/rcutorture/configs/rcu/SRCU-L.boot | 3 +++ .../selftests/rcutorture/configs/rcu/SRCU-N.boot | 1 + tools/testing/selftests/rcutorture/configs/rcu/TREE10 | 2 +- 5 files changed, 16 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/rcutorture/configs/rcu/SRCU-L create mode 100644 tools/testing/selftests/rcutorture/configs/rcu/SRCU-L.boot diff --git a/tools/testing/selftests/rcutorture/configs/rcu/CFLIST b/tools/testing/selftests/rcutorture/configs/rcu/CFLIST index 98b6175e5aa0..45f572570a8c 100644 --- a/tools/testing/selftests/rcutorture/configs/rcu/CFLIST +++ b/tools/testing/selftests/rcutorture/configs/rcu/CFLIST @@ -5,6 +5,7 @@ TREE04 TREE05 TREE07 TREE09 +SRCU-L SRCU-N SRCU-P SRCU-T diff --git a/tools/testing/selftests/rcutorture/configs/rcu/SRCU-L b/tools/testing/selftests/rcutorture/configs/rcu/SRCU-L new file mode 100644 index 000000000000..3b4fa8dbef8a --- /dev/null +++ b/tools/testing/selftests/rcutorture/configs/rcu/SRCU-L @@ -0,0 +1,10 @@ +CONFIG_RCU_TRACE=n +CONFIG_SMP=y +CONFIG_NR_CPUS=6 +CONFIG_HOTPLUG_CPU=y +CONFIG_PREEMPT_NONE=y +CONFIG_PREEMPT_VOLUNTARY=n +CONFIG_PREEMPT=n +#CHECK#CONFIG_RCU_EXPERT=n +CONFIG_KPROBES=n +CONFIG_FTRACE=n diff --git a/tools/testing/selftests/rcutorture/configs/rcu/SRCU-L.boot b/tools/testing/selftests/rcutorture/configs/rcu/SRCU-L.boot new file mode 100644 index 000000000000..0207b3138c5b --- /dev/null +++ b/tools/testing/selftests/rcutorture/configs/rcu/SRCU-L.boot @@ -0,0 +1,3 @@ +rcutorture.torture_type=srcu +rcutorture.reader_flavor=0x4 +rcutorture.fwd_progress=3 diff --git a/tools/testing/selftests/rcutorture/configs/rcu/SRCU-N.boot b/tools/testing/selftests/rcutorture/configs/rcu/SRCU-N.boot index ce0694fd9b92..b54cf87dc110 100644 --- a/tools/testing/selftests/rcutorture/configs/rcu/SRCU-N.boot +++ b/tools/testing/selftests/rcutorture/configs/rcu/SRCU-N.boot @@ -1,2 +1,3 @@ rcutorture.torture_type=srcu +rcutorture.reader_flavor=0x2 rcutorture.fwd_progress=3 diff --git a/tools/testing/selftests/rcutorture/configs/rcu/TREE10 b/tools/testing/selftests/rcutorture/configs/rcu/TREE10 index a323d8948b7c..759ee51d3ddc 100644 --- a/tools/testing/selftests/rcutorture/configs/rcu/TREE10 +++ b/tools/testing/selftests/rcutorture/configs/rcu/TREE10 @@ -1,5 +1,5 @@ CONFIG_SMP=y -CONFIG_NR_CPUS=56 +CONFIG_NR_CPUS=74 CONFIG_PREEMPT_NONE=y CONFIG_PREEMPT_VOLUNTARY=n CONFIG_PREEMPT=n From patchwork Tue Nov 12 14:51:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 13872361 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 36908202F9E; Tue, 12 Nov 2024 14:53:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731423188; cv=none; b=I0Q8hUOnP5j6qgP2hsyCK5dmVkJHie6c+fwYEoXl0Z+GEoGWhxDZa1HPAlxuXEuXU4REaDjiRpT3VENeayLSVMBrty2MkIvhun8Tp5hgAeDMLKJCvmsu1S46sso7pDaV4uiF5M5eUtET1alrNAg5X/M28Qnlg03ynoyF7+Miges= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731423188; c=relaxed/simple; bh=DSNbca3lQ8LSXso6jqYZBWuI3HmmlzLJZscELXWEgfM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=DPdN6G9DxNiLLsoT5gv5afnOVDfNVbsuvFyrhHpnHutiv4C/ZwBG6M59t64DNxSUgwEz3mXcw7ouGGuvKFPVEB9SnPQJZ+ru4zYNIuZm88Gf3VtYN+Mh/wV0C7bWb7rWVy08YHpU0KOwXZPq+fhA0AJT/dDTrldm1gDfIrI9pDc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=lsLl9uQv; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="lsLl9uQv" Received: by smtp.kernel.org (Postfix) with ESMTPSA id D3F2BC4CEDB; Tue, 12 Nov 2024 14:53:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1731423187; bh=DSNbca3lQ8LSXso6jqYZBWuI3HmmlzLJZscELXWEgfM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=lsLl9uQvLt8thmWmeMZSNMaub4BZv51O5iIP//APEUCfgN+bjQ1CZ0MVStIZmS3b+ hu13ktakC0BZhufXS5q+zV6kXHDF5BfegCMlNst9SX4dJV5SHY/2IQkl8TTMw2VXVB 8EYtfrQ/3ifIbHLbEKFzAVvlt8FXyjb8f72itSwXtENX0m3Pga+QZ+BwltzddhfucA LV/oJmR7CyqxJ7H2n+MIcfvhTV9IU/0m6VvXTv9K1M08knmIFXVzelFXEp/6H+RdNm ijMQrQmvib3MXp2qmJ9SVrRKFVuSbr1ffRdXmc5f6qboI0ZKmPR171otuGyYSf17uk S9O9wym6AR+NQ== From: Frederic Weisbecker To: LKML Cc: "Paul E. McKenney" , Boqun Feng , Joel Fernandes , Josh Triplett , Lai Jiangshan , Mathieu Desnoyers , Neeraj Upadhyay , Steven Rostedt , Uladzislau Rezki , Zqiang , rcu , Alexei Starovoitov , Andrii Nakryiko , Peter Zijlstra , Kent Overstreet , bpf@vger.kernel.org, Neeraj Upadhyay , Frederic Weisbecker Subject: [PATCH 15/16] refscale: Add srcu_read_lock_lite() support using "srcu-lite" Date: Tue, 12 Nov 2024 15:51:58 +0100 Message-ID: <20241112145159.23032-16-frederic@kernel.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20241112145159.23032-1-frederic@kernel.org> References: <20241112145159.23032-1-frederic@kernel.org> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" This commit creates a new srcu-lite option for the refscale.scale_type module parameter that selects srcu_read_lock_lite() and srcu_read_unlock_lite(). [ paulmck: Apply Dan Carpenter feedback. ] Signed-off-by: Paul E. McKenney Cc: Alexei Starovoitov Cc: Andrii Nakryiko Cc: Peter Zijlstra Cc: Kent Overstreet Cc: Reviewed-by: Neeraj Upadhyay Signed-off-by: Frederic Weisbecker --- kernel/rcu/refscale.c | 37 ++++++++++++++++++++++++++++++++++--- 1 file changed, 34 insertions(+), 3 deletions(-) diff --git a/kernel/rcu/refscale.c b/kernel/rcu/refscale.c index 0db9db73f57f..338e7c5ac44a 100644 --- a/kernel/rcu/refscale.c +++ b/kernel/rcu/refscale.c @@ -212,6 +212,36 @@ static const struct ref_scale_ops srcu_ops = { .name = "srcu" }; +static void srcu_lite_ref_scale_read_section(const int nloops) +{ + int i; + int idx; + + for (i = nloops; i >= 0; i--) { + idx = srcu_read_lock_lite(srcu_ctlp); + srcu_read_unlock_lite(srcu_ctlp, idx); + } +} + +static void srcu_lite_ref_scale_delay_section(const int nloops, const int udl, const int ndl) +{ + int i; + int idx; + + for (i = nloops; i >= 0; i--) { + idx = srcu_read_lock_lite(srcu_ctlp); + un_delay(udl, ndl); + srcu_read_unlock_lite(srcu_ctlp, idx); + } +} + +static const struct ref_scale_ops srcu_lite_ops = { + .init = rcu_sync_scale_init, + .readsection = srcu_lite_ref_scale_read_section, + .delaysection = srcu_lite_ref_scale_delay_section, + .name = "srcu-lite" +}; + #ifdef CONFIG_TASKS_RCU // Definitions for RCU Tasks ref scale testing: Empty read markers. @@ -1082,9 +1112,10 @@ ref_scale_init(void) long i; int firsterr = 0; static const struct ref_scale_ops *scale_ops[] = { - &rcu_ops, &srcu_ops, RCU_TRACE_OPS RCU_TASKS_OPS &refcnt_ops, &rwlock_ops, - &rwsem_ops, &lock_ops, &lock_irq_ops, &acqrel_ops, &clock_ops, &jiffies_ops, - &typesafe_ref_ops, &typesafe_lock_ops, &typesafe_seqlock_ops, + &rcu_ops, &srcu_ops, &srcu_lite_ops, RCU_TRACE_OPS RCU_TASKS_OPS + &refcnt_ops, &rwlock_ops, &rwsem_ops, &lock_ops, &lock_irq_ops, &acqrel_ops, + &clock_ops, &jiffies_ops, &typesafe_ref_ops, &typesafe_lock_ops, + &typesafe_seqlock_ops, }; if (!torture_init_begin(scale_type, verbose)) From patchwork Tue Nov 12 14:51:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 13872362 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 325B620011A; Tue, 12 Nov 2024 14:53:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731423191; cv=none; b=LOZfL6nnsr8dXHX7TFI9ifuGwtNbces2bRZ8Gj2wk5BxJdy/wFZpMnecC85MOIk9VoWA81B6SuzbKxA0DH0+eWzQBsYD16pz5lTQtsgV48UQ+AfpUqfXMQzTrsXy0XcC5DAiVQOE+R09vh1ikKXFCVgacdkjAnJyl0kOAlP5two= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731423191; c=relaxed/simple; bh=GIhZh93QuvVeoudWwuzoyIzNtlHYTdDkGBR8g5zW14w=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=JVwM3tQxr4UadtfyOOK/jEuyxbERoX50y6a/F12zbababSbP4NsMGEve3MxVgz7GflvtfhTpTnnsYR1+zoKA8LDJn6WdzU/SDJaDIHxo92wfac0+wV3A/YXNTKbxqOECxjg3G8f/B+TYJoC5SfqKBWwHERdjdkBavRrGoYYhgJc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Ce0i2qR9; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Ce0i2qR9" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 19C86C4CED5; Tue, 12 Nov 2024 14:53:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1731423191; bh=GIhZh93QuvVeoudWwuzoyIzNtlHYTdDkGBR8g5zW14w=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Ce0i2qR9yIfjpjRAm99cOcCBS0s6QD0DO3Thlkt7SyEryCtT757nKtt0uogRZA6Fh xBo3l9Ulw99OEts2/KUDWZ8f2PvZA5nuLKOa5HoGSFvQfOHTERDSgsEOP92OsPRhLS A1Eatq5N2VYCQ/RhE6t6jMZ6sublSkWsWSjIyvhtxtm/5DXytkWbrvEgfsGshVI0Iu YJI3FvMNPMRpo1+lL/7Fa/3ZKATZKzxqXK6VkzqrUi+h8BMQqifqkjFQzTBU3yuXHn 3nAh0Uwy8GNAasMnwqLNsQdAM1Rtow/zoUvYbwFUs/NN8KjVASlH6uF2/d/nevCrOh 5vlTvYHftY3TA== From: Frederic Weisbecker To: LKML Cc: "Paul E. McKenney" , Boqun Feng , Joel Fernandes , Josh Triplett , Lai Jiangshan , Mathieu Desnoyers , Neeraj Upadhyay , Steven Rostedt , Uladzislau Rezki , Zqiang , rcu , Andrii Nakryiko , Neeraj Upadhyay , Frederic Weisbecker Subject: [PATCH 16/16] srcu: Improve srcu_read_lock_lite() kernel-doc comment Date: Tue, 12 Nov 2024 15:51:59 +0100 Message-ID: <20241112145159.23032-17-frederic@kernel.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20241112145159.23032-1-frederic@kernel.org> References: <20241112145159.23032-1-frederic@kernel.org> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" Where RCU is watching is where it is OK to invoke rcu_read_lock(). Reported-by: Andrii Nakryiko Signed-off-by: Paul E. McKenney Acked-by: Andrii Nakryiko Reviewed-by: Neeraj Upadhyay Signed-off-by: Frederic Weisbecker --- include/linux/srcu.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/include/linux/srcu.h b/include/linux/srcu.h index 4ba96e2cfa40..bab1dae3f69e 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -270,7 +270,8 @@ static inline int srcu_read_lock(struct srcu_struct *ssp) __acquires(ssp) * synchronize_rcu_expedited(), IPIs and all. * * Note that srcu_read_lock_lite() can be invoked only from those contexts - * where RCU is watching. Otherwise, lockdep will complain. + * where RCU is watching, that is, from contexts where it would be legal + * to invoke rcu_read_lock(). Otherwise, lockdep will complain. */ static inline int srcu_read_lock_lite(struct srcu_struct *ssp) __acquires(ssp) {