From patchwork Tue Nov 12 01:31:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 13871621 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8AB14205E1E; Tue, 12 Nov 2024 01:31:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731375105; cv=none; b=qW6p8McxPMbd7JsjMIrHq7bzvdcQrFOgYX6CZj7dSPHs5F7Q+P4hmE+UQR9SRXabLWVJMy+Q2s/gomJN8+lePyor1OyEr50/KttLZsFiNvoL0v0Z3hqs+m9mjrYnro9EYdOttQWyWrdTPZd2sDv1tY0mDutKphfSadIDmZlaRPs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731375105; c=relaxed/simple; bh=MVgOKsZ5cymmfkHb8KBHOc8ZHTR30pecMj78iSO0C7U=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=X/WrDVVmFMbnJbASToqrWLzevlHxLNRLmd6qrf76sp8Amm5rbI5P/rIc/EspQ1gdZAQqEsf1PSRSzzr6vHQQuYlE2m85Y5u4CKV3Gdk7YCWWlQRIY4vEoUF9bYlLwl4Nqiwy+fMOVyNQaitEwtsM61tPGv0NY06JWTg6RVteUSc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=DtG8i4eM; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="DtG8i4eM" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2647FC4CED0; Tue, 12 Nov 2024 01:31:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1731375105; bh=MVgOKsZ5cymmfkHb8KBHOc8ZHTR30pecMj78iSO0C7U=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DtG8i4eM3UsNa3nG+ZzccEsiQAuu10OYc8P+mGfUYpqZilqfXNMy6JvlDP7VNKKqb mys0kK+xz1N+kg/CpN9MckbCzBr0vMhnCurpbCHnas/mxIJFtiYo21ZM5gGhR7Vr/q 3LCz5qRSP6u7/1qXO+bn+eGKdwPb5m9c6zdx+/i6jGSwgR1qBU9MnaWY3GqcsZPbyH /FBAJhRE6Wnt1kJMVrK+ISN5C5pUYcm2/BX3q1eqDGAFXSRaim0FfJ1nP4+8+UjEOC vu24WCAr0/5NRpbbParGt7UbAkQqzVxF9keAn6zieqfHVddtc+RhnbR4S3XsCHE3rv UzPSF5nZ/aZVw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id BA84FCE0BA3; Mon, 11 Nov 2024 17:31:44 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, frederic@kernel.org, "Paul E. McKenney" , Neeraj Upadhyay Subject: [PATCH rcu 1/3] srcu: Remove smp_mb() from srcu_read_unlock_lite() Date: Mon, 11 Nov 2024 17:31:41 -0800 Message-Id: <20241112013143.1926484-1-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 The srcu_read_unlock_lite() function invokes __srcu_read_unlock() instead of __srcu_read_unlock_lite(), which means that it is doing an unnecessary smp_mb(). This is harmless other than the performance degradation. This commit therefore switches to __srcu_read_unlock_lite(). Reported-by: Neeraj Upadhyay Closes: https://lore.kernel.org/all/d07e8f4a-d5ff-4c8e-8e61-50db285c57e9@amd.com/ Fixes: c0f08d6b5a61 ("srcu: Add srcu_read_lock_lite() and srcu_read_unlock_lite()") Signed-off-by: Paul E. McKenney Cc: Frederic Weisbecker Reviewed-by: Neeraj Upadhyay --- include/linux/srcu.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/linux/srcu.h b/include/linux/srcu.h index 512a8c54ba5ba..fff6265818d4f 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -373,7 +373,7 @@ static inline void srcu_read_unlock_lite(struct srcu_struct *ssp, int idx) WARN_ON_ONCE(idx & ~0x1); srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_LITE); srcu_lock_release(&ssp->dep_map); - __srcu_read_unlock(ssp, idx); + __srcu_read_unlock_lite(ssp, idx); } /** From patchwork Tue Nov 12 01:31:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 13871620 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8AA35205ABF; Tue, 12 Nov 2024 01:31:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731375105; cv=none; b=Nz8l4DxWJcoo0CLNsgS6Lli8K6obteqoOp6tzyJXT6rWiVjPopK5aIesM7e7/2+wieoBk3Tb/EAjanwdEXOaKRp/7qC8lPJBn+MvhnP8UcZsLftJSrM4efcClB/F2TDkio19QC+16QLg4kygGBGyfrLKSTZh5rBPGbpCqoptQEo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731375105; c=relaxed/simple; bh=QYsZv/IYpEobJEvqATWqh2kv3A3XK8nogTl6lynsHwM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=aQjJqn8VaTV8ppRfK5v2ikB7DO7fPopeEYAj0oqWtxcuaG5jaFWb5jPZIrFSs9Wza+QzAFYShaui6lX6jkfzjZBGxc1cOLRC4CGqrQo03+k+ytaVLUAmnk+XCOXGUn+wnyuOAiNse/BcSYvRUxGqIwQACnx9rhr3d5H2m9Xr4Ys= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=V9ideWVA; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="V9ideWVA" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 22C38C4CECF; Tue, 12 Nov 2024 01:31:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1731375105; bh=QYsZv/IYpEobJEvqATWqh2kv3A3XK8nogTl6lynsHwM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=V9ideWVAj+QEnCQikWVK34pz64e5HcUo6BtxwdLYSyJtzRmhsUVzYIpfLMzIMpKEd ismKT27ypW2PcYJ2haLLculposdkCpbQwLC4f4YNU4xgiakTZzEIzpxwlbXomSGyMS L250Ou/S4sPg181Ilq9unzDkXdP9pfubI5C/X3t9tSpiFRzgaS2H4eYTHwueW7xS7y 7jIM/LWlrLLKFwJfQ4k/kadj8ws4B0UIK7yNZ+AqVw07N/u3vZsu4PUCxtWFkRG2PV L2rw6T3poEURM20YT0R/L3But/LWfmri9rw/Hdt+6bKyW3SXQqpDHVCwv9HxU+8WtW Z70N2DMBZLHJQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id BD0CACE0BC4; Mon, 11 Nov 2024 17:31:44 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, frederic@kernel.org, "Paul E. McKenney" , Neeraj Upadhyay Subject: [PATCH rcu 2/3] srcu: Check for srcu_read_lock_lite() across all CPUs Date: Mon, 11 Nov 2024 17:31:42 -0800 Message-Id: <20241112013143.1926484-2-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 If srcu_read_lock_lite() is used on a given srcu_struct structure, then the grace-period processing must to synchronize_rcu() instead of smp_mb() between the scans of the ->srcu_unlock_count[] and ->srcu_lock_count[] counters. Currently, it does that by testing the SRCU_READ_FLAVOR_LITE bit of the ->srcu_reader_flavor mask, which works well. But only if the CPU running that srcu_struct structure's grace period has previously executed srcu_read_lock_lite(), which might not be the case, especially just after that srcu_struct structure has been created and initialized. This commit therefore updates the srcu_readers_unlock_idx() function to OR together the ->srcu_reader_flavor masks from all CPUs, and then make the srcu_readers_active_idx_check() function that test the SRCU_READ_FLAVOR_LITE bit in the resulting mask. Note that the srcu_readers_unlock_idx() function is already scanning all the CPUs to sum up the ->srcu_unlock_count[] fields and that this is on the grace-period slow path, hence no concerns about the small amount of extra work. Reported-by: Neeraj Upadhyay Closes: https://lore.kernel.org/all/d07e8f4a-d5ff-4c8e-8e61-50db285c57e9@amd.com/ Fixes: c0f08d6b5a61 ("srcu: Add srcu_read_lock_lite() and srcu_read_unlock_lite()") Signed-off-by: Paul E. McKenney Cc: Frederic Weisbecker Reviewed-by: Neeraj Upadhyay --- kernel/rcu/srcutree.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 70979f294768c..5991381b44383 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -458,7 +458,7 @@ static bool srcu_readers_lock_idx(struct srcu_struct *ssp, int idx, bool gp, uns * Returns approximate total of the readers' ->srcu_unlock_count[] values * for the rank of per-CPU counters specified by idx. */ -static unsigned long srcu_readers_unlock_idx(struct srcu_struct *ssp, int idx) +static unsigned long srcu_readers_unlock_idx(struct srcu_struct *ssp, int idx, unsigned long *rdm) { int cpu; unsigned long mask = 0; @@ -468,11 +468,11 @@ static unsigned long srcu_readers_unlock_idx(struct srcu_struct *ssp, int idx) struct srcu_data *sdp = per_cpu_ptr(ssp->sda, cpu); sum += atomic_long_read(&sdp->srcu_unlock_count[idx]); - if (IS_ENABLED(CONFIG_PROVE_RCU)) - mask = mask | READ_ONCE(sdp->srcu_reader_flavor); + mask = mask | READ_ONCE(sdp->srcu_reader_flavor); } WARN_ONCE(IS_ENABLED(CONFIG_PROVE_RCU) && (mask & (mask - 1)), "Mixed reader flavors for srcu_struct at %ps.\n", ssp); + *rdm = mask; return sum; } @@ -482,10 +482,11 @@ static unsigned long srcu_readers_unlock_idx(struct srcu_struct *ssp, int idx) */ static bool srcu_readers_active_idx_check(struct srcu_struct *ssp, int idx) { - bool did_gp = !!(raw_cpu_read(ssp->sda->srcu_reader_flavor) & SRCU_READ_FLAVOR_LITE); + unsigned long rdm; unsigned long unlocks; - unlocks = srcu_readers_unlock_idx(ssp, idx); + unlocks = srcu_readers_unlock_idx(ssp, idx, &rdm); + bool did_gp = !!(rdm & SRCU_READ_FLAVOR_LITE); /* * Make sure that a lock is always counted if the corresponding From patchwork Tue Nov 12 01:31:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 13871622 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8AA8B205AC2; Tue, 12 Nov 2024 01:31:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731375105; cv=none; b=qYr/TZ+6fq4MHcYXq4dbIslhOAYxkGo3l1YBIy2bUhgKLd/pCnEgN8xVhadXVuWk0BekAJXjA0+s8DKxhDerlnxKQTYhFN3vOYoOM9vJxPAQBm2qb3PfG56W2iyKTK+H/XlfLFfoGM1S0qA1YykCBv2EUw9cVcxTGh48GQzoGMk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731375105; c=relaxed/simple; bh=+MUnT2ixsByPRqgZLMCXaVOCAlMUjkJamTfkr8KcD0c=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=tNakR7+NjHmUSjnOvNXQWFQiBWVBQqq59aQRZQrhkhRrXzHbnqr45c8YakUJk+gIelVguR8aHnHtqogNYLflRlJXd7tHf7ZPtZ+YDRs5MDp2gFuJRjI+ty88ewr3Osi3o86TlV1C0G4JY61mOPg6PbJD+j6nU4Rvvg5IOzMS+5M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=fzzpMlWt; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="fzzpMlWt" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 33C17C4CED5; Tue, 12 Nov 2024 01:31:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1731375105; bh=+MUnT2ixsByPRqgZLMCXaVOCAlMUjkJamTfkr8KcD0c=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=fzzpMlWthttpkPhQSgqy31GAfo9FZJfvIkMZDyIvoaoa4UAz2QOkzznOZRWCxFd36 LZ4ruAnbM+cYFQDTpemIyv0elL7o09TEwyO1+e+kYNJlRWrl9qOrX/ltjw9p3faxEx idX1BJQi1Dkf5CjwlVlIFGxrOTbjDu6zEIfMcIOfs5vqDa8jO+tuY/A+gbusHJsAi2 L8eYUEJMvL6KlYNG2Qr+aBK5i1ZEIMHM2cBuMMC80aoVptkbJ5uEfkYPgoDwR0RB03 3UqVS7wY8MPIx0XxMTLBkgF0vobQAFC4jvsiH37YEEFHfkOTzANmCq+q4V5c5jE2Sk vmrMpKeET3Vgg== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id C052FCE0D89; Mon, 11 Nov 2024 17:31:44 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, frederic@kernel.org, "Paul E. McKenney" , Neeraj Upadhyay Subject: [PATCH rcu 3/3] srcu: Unconditionally record srcu_read_lock_lite() in ->srcu_reader_flavor Date: Mon, 11 Nov 2024 17:31:43 -0800 Message-Id: <20241112013143.1926484-3-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Currently, srcu_read_lock_lite() uses the SRCU_READ_FLAVOR_LITE bit in ->srcu_reader_flavor to communicate to the grace-period processing in srcu_readers_active_idx_check() that the smp_mb() must be replaced by a synchronize_rcu(). Unfortunately, ->srcu_reader_flavor is not updated unless the kernel is built with CONFIG_PROVE_RCU=y. Therefore in all kernels built with CONFIG_PROVE_RCU=n, srcu_readers_active_idx_check() incorrectly uses smp_mb() instead of synchronize_rcu() for srcu_struct structures whose readers use srcu_read_lock_lite(). This commit therefore causes Tree SRCU srcu_read_lock_lite() to unconditionally update ->srcu_reader_flavor so that srcu_readers_active_idx_check() can make the correct choice. Reported-by: Neeraj Upadhyay Closes: https://lore.kernel.org/all/d07e8f4a-d5ff-4c8e-8e61-50db285c57e9@amd.com/ Fixes: c0f08d6b5a61 ("srcu: Add srcu_read_lock_lite() and srcu_read_unlock_lite()" Signed-off-by: Paul E. McKenney Cc: Frederic Weisbecker Reviewed-by: Neeraj Upadhyay --- include/linux/srcu.h | 8 +------- include/linux/srcutiny.h | 3 +++ include/linux/srcutree.h | 21 +++++++++++++++++++++ kernel/rcu/srcutree.c | 6 ++---- 4 files changed, 27 insertions(+), 11 deletions(-) diff --git a/include/linux/srcu.h b/include/linux/srcu.h index fff6265818d4f..abaddd7e6ddf7 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -183,12 +183,6 @@ static inline int srcu_read_lock_held(const struct srcu_struct *ssp) #endif /* #else #ifdef CONFIG_DEBUG_LOCK_ALLOC */ -#if defined(CONFIG_PROVE_RCU) && defined(CONFIG_TREE_SRCU) -void srcu_check_read_flavor(struct srcu_struct *ssp, int read_flavor); -#else -#define srcu_check_read_flavor(ssp, read_flavor) do { } while (0) -#endif - /** * srcu_dereference_check - fetch SRCU-protected pointer for later dereferencing @@ -278,7 +272,7 @@ static inline int srcu_read_lock_lite(struct srcu_struct *ssp) __acquires(ssp) { int retval; - srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_LITE); + srcu_check_read_flavor_lite(ssp); retval = __srcu_read_lock_lite(ssp); rcu_try_lock_acquire(&ssp->dep_map); return retval; diff --git a/include/linux/srcutiny.h b/include/linux/srcutiny.h index 4d96bbdb45f08..1321da803274d 100644 --- a/include/linux/srcutiny.h +++ b/include/linux/srcutiny.h @@ -81,6 +81,9 @@ static inline void srcu_barrier(struct srcu_struct *ssp) synchronize_srcu(ssp); } +#define srcu_check_read_flavor(ssp, read_flavor) do { } while (0) +#define srcu_check_read_flavor_lite(ssp) do { } while (0) + /* Defined here to avoid size increase for non-torture kernels. */ static inline void srcu_torture_stats_print(struct srcu_struct *ssp, char *tt, char *tf) diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index 778eb61542e18..490aeecc6bb47 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -248,4 +248,25 @@ static inline void __srcu_read_unlock_lite(struct srcu_struct *ssp, int idx) RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_unlock_lite()."); } +void __srcu_check_read_flavor(struct srcu_struct *ssp, int read_flavor); + +// Record _lite() usage even for CONFIG_PROVE_RCU=n kernels. +static inline void srcu_check_read_flavor_lite(struct srcu_struct *ssp) +{ + struct srcu_data *sdp = raw_cpu_ptr(ssp->sda); + + if (likely(READ_ONCE(sdp->srcu_reader_flavor) & SRCU_READ_FLAVOR_LITE)) + return; + + // Note that the cmpxchg() in srcu_check_read_flavor() is fully ordered. + __srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_LITE); +} + +// Record non-_lite() usage only for CONFIG_PROVE_RCU=y kernels. +static inline void srcu_check_read_flavor(struct srcu_struct *ssp, int read_flavor) +{ + if (IS_ENABLED(CONFIG_PROVE_RCU)) + __srcu_check_read_flavor(ssp, read_flavor); +} + #endif diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 5991381b44383..ee3cbb8308664 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -711,11 +711,10 @@ void cleanup_srcu_struct(struct srcu_struct *ssp) } EXPORT_SYMBOL_GPL(cleanup_srcu_struct); -#ifdef CONFIG_PROVE_RCU /* * Check for consistent reader flavor. */ -void srcu_check_read_flavor(struct srcu_struct *ssp, int read_flavor) +void __srcu_check_read_flavor(struct srcu_struct *ssp, int read_flavor) { int old_read_flavor; struct srcu_data *sdp; @@ -733,8 +732,7 @@ void srcu_check_read_flavor(struct srcu_struct *ssp, int read_flavor) } WARN_ONCE(old_read_flavor != read_flavor, "CPU %d old state %d new state %d\n", sdp->cpu, old_read_flavor, read_flavor); } -EXPORT_SYMBOL_GPL(srcu_check_read_flavor); -#endif /* CONFIG_PROVE_RCU */ +EXPORT_SYMBOL_GPL(__srcu_check_read_flavor); /* * Counts the new reader in the appropriate per-CPU element of the