[03/10] rcu: Add QS check in rcu_exp_handler() for non-preemptible kernels

Commit Message

Paul E. McKenney Aug. 31, 2022, 6:07 p.m. UTC

From: Zqiang <[e-mail]>

Kernels built with CONFIG_PREEMPTION=n and CONFIG_PREEMPT_COUNT=y maintain
preempt_count() state.  Because such kernels map __rcu_read_lock()
and __rcu_read_unlock() to preempt_disable() and preempt_enable(),
respectively, this allows the expedited grace period's !CONFIG_PREEMPT_RCU
version of the rcu_exp_handler() IPI handler function to use
preempt_count() to detect quiescent states.

This preempt_count() usage might seem to risk failures due to
use of implicit RCU readers in portions of the kernel under #ifndef
CONFIG_PREEMPTION, except that rcu_core() already disallows such implicit
RCU readers.  The moral of this story is that you must use explicit
read-side markings such as rcu_read_lock() or preempt_disable() even if
the code knows that this kernel does not support preemption.

This commit therefore adds a preempt_count()-based check for a quiescent
state in the !CONFIG_PREEMPT_RCU version of the rcu_exp_handler()
function for kernels built with CONFIG_PREEMPT_COUNT=y, reporting an
immediate quiescent state when the interrupted code had both preemption
and softirqs enabled.

This change results in about a 2% reduction in expedited grace-period
latency in kernels built with both CONFIG_PREEMPT_RCU=n and
CONFIG_PREEMPT_COUNT=y.

Signed-off-by: Zqiang <[e-mail]>
Signed-off-by: Paul E. McKenney <[e-mail]>
Link: https://lore.kernel.org/all/[e-mail]/
---
 kernel/rcu/tree_exp.h | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

Comments

Frederic Weisbecker Sept. 7, 2022, 12:10 p.m. UTC | #1

On Wed, Aug 31, 2022 at 11:07:58AM -0700, Paul E. McKenney wrote:
> From: Zqiang <[e-mail]>
> 
> Kernels built with CONFIG_PREEMPTION=n and CONFIG_PREEMPT_COUNT=y maintain
> preempt_count() state.  Because such kernels map __rcu_read_lock()
> and __rcu_read_unlock() to preempt_disable() and preempt_enable(),
> respectively, this allows the expedited grace period's !CONFIG_PREEMPT_RCU
> version of the rcu_exp_handler() IPI handler function to use
> preempt_count() to detect quiescent states.
> 
> This preempt_count() usage might seem to risk failures due to
> use of implicit RCU readers in portions of the kernel under #ifndef
> CONFIG_PREEMPTION, except that rcu_core() already disallows such implicit
> RCU readers.  The moral of this story is that you must use explicit
> read-side markings such as rcu_read_lock() or preempt_disable() even if
> the code knows that this kernel does not support preemption.
> 
> This commit therefore adds a preempt_count()-based check for a quiescent
> state in the !CONFIG_PREEMPT_RCU version of the rcu_exp_handler()
> function for kernels built with CONFIG_PREEMPT_COUNT=y, reporting an
> immediate quiescent state when the interrupted code had both preemption
> and softirqs enabled.
> 
> This change results in about a 2% reduction in expedited grace-period
> latency in kernels built with both CONFIG_PREEMPT_RCU=n and
> CONFIG_PREEMPT_COUNT=y.
> 
> Signed-off-by: Zqiang <[e-mail]>
> Signed-off-by: Paul E. McKenney <[e-mail]>
> Link: https://lore.kernel.org/all/[e-mail]/
> ---
>  kernel/rcu/tree_exp.h | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h
> index be667583a5547..b07998159d1fa 100644
> --- a/kernel/rcu/tree_exp.h
> +++ b/kernel/rcu/tree_exp.h
> @@ -828,11 +828,13 @@ static void rcu_exp_handler(void *unused)
>  {
>  	struct rcu_data *rdp = this_cpu_ptr(&rcu_data);
>  	struct rcu_node *rnp = rdp->mynode;
> +	bool preempt_bh_enabled = !(preempt_count() & (PREEMPT_MASK |
> SOFTIRQ_MASK));

I don't know if nested hardirqs still exist. I only heard old rumours
about broken drivers. Should we take care of them?

Also are we sure that all callers of flush_smp_call_function_queue()
are QS?

Let's see we know that rcu_exp_handler() can either be executed from:

* hardirqs

Or from process context, expected to be RCU QS states at least in idle
as the comment above flush_smp_call_function_queue() in idle says
(but I'd rather check all the in-process callers before stating all
of them are in QS)

* idle (in which case preemption is disabled unfortunately so the current
test won't help)
* stop_machine
  _ When CPU is dead and out of RCU (rcutree_dead_cpu() called)
    so that should be a QS.
  _ When CPU is migrating (is it a QS then?)

If we check further that all non-IRQ callers of flush_smp_call_function_queue()
are always quiescent states then we could deduce that !in_hardirq() means we are in
a quiescent state, whether preemption is disabled or not.

In any case for the current patch, perhaps a more robust test against nested
hardirqs would be:

unsigned long cnt = preempt_count();
bool preempt_bh_enabled = (!cnt || cnt == HARDIRQ_OFFSET)

Thanks.

>  
>  	if (!(READ_ONCE(rnp->expmask) & rdp->grpmask) ||
>  	    __this_cpu_read(rcu_data.cpu_no_qs.b.exp))
>  		return;
> -	if (rcu_is_cpu_rrupt_from_idle()) {
> +	if (rcu_is_cpu_rrupt_from_idle() ||
> +	    (IS_ENABLED(CONFIG_PREEMPT_COUNT) && preempt_bh_enabled)) {
>  		rcu_report_exp_rdp(this_cpu_ptr(&rcu_data));
>  		return;
>  	}
> -- 
> 2.31.1.189.g2e36527f23
>

Paul E. McKenney Sept. 7, 2022, 2:57 p.m. UTC | #2

On Wed, Sep 07, 2022 at 02:10:10PM +0200, Frederic Weisbecker wrote:
> On Wed, Aug 31, 2022 at 11:07:58AM -0700, Paul E. McKenney wrote:
> > From: Zqiang <[e-mail]>
> > 
> > Kernels built with CONFIG_PREEMPTION=n and CONFIG_PREEMPT_COUNT=y maintain
> > preempt_count() state.  Because such kernels map __rcu_read_lock()
> > and __rcu_read_unlock() to preempt_disable() and preempt_enable(),
> > respectively, this allows the expedited grace period's !CONFIG_PREEMPT_RCU
> > version of the rcu_exp_handler() IPI handler function to use
> > preempt_count() to detect quiescent states.
> > 
> > This preempt_count() usage might seem to risk failures due to
> > use of implicit RCU readers in portions of the kernel under #ifndef
> > CONFIG_PREEMPTION, except that rcu_core() already disallows such implicit
> > RCU readers.  The moral of this story is that you must use explicit
> > read-side markings such as rcu_read_lock() or preempt_disable() even if
> > the code knows that this kernel does not support preemption.
> > 
> > This commit therefore adds a preempt_count()-based check for a quiescent
> > state in the !CONFIG_PREEMPT_RCU version of the rcu_exp_handler()
> > function for kernels built with CONFIG_PREEMPT_COUNT=y, reporting an
> > immediate quiescent state when the interrupted code had both preemption
> > and softirqs enabled.
> > 
> > This change results in about a 2% reduction in expedited grace-period
> > latency in kernels built with both CONFIG_PREEMPT_RCU=n and
> > CONFIG_PREEMPT_COUNT=y.
> > 
> > Signed-off-by: Zqiang <[e-mail]>
> > Signed-off-by: Paul E. McKenney <[e-mail]>
> > Link: https://lore.kernel.org/all/[e-mail]/
> > ---
> >  kernel/rcu/tree_exp.h | 4 +++-
> >  1 file changed, 3 insertions(+), 1 deletion(-)
> > 
> > diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h
> > index be667583a5547..b07998159d1fa 100644
> > --- a/kernel/rcu/tree_exp.h
> > +++ b/kernel/rcu/tree_exp.h
> > @@ -828,11 +828,13 @@ static void rcu_exp_handler(void *unused)
> >  {
> >  	struct rcu_data *rdp = this_cpu_ptr(&rcu_data);
> >  	struct rcu_node *rnp = rdp->mynode;
> > +	bool preempt_bh_enabled = !(preempt_count() & (PREEMPT_MASK |
> > SOFTIRQ_MASK));
> 
> I don't know if nested hardirqs still exist. I only heard old rumours
> about broken drivers. Should we take care of them?

Last I checked, certain tracing scenarios from irq handlers looked
to RCU like nested irq handlers.  Given that, does your more robust
approach below work correctly?

						Thanx, Paul

> Also are we sure that all callers of flush_smp_call_function_queue()
> are QS?
> 
> Let's see we know that rcu_exp_handler() can either be executed from:
> 
> * hardirqs
> 
> Or from process context, expected to be RCU QS states at least in idle
> as the comment above flush_smp_call_function_queue() in idle says
> (but I'd rather check all the in-process callers before stating all
> of them are in QS)
> 
> * idle (in which case preemption is disabled unfortunately so the current
> test won't help)
> * stop_machine
>   _ When CPU is dead and out of RCU (rcutree_dead_cpu() called)
>     so that should be a QS.
>   _ When CPU is migrating (is it a QS then?)
> 
> If we check further that all non-IRQ callers of flush_smp_call_function_queue()
> are always quiescent states then we could deduce that !in_hardirq() means we are in
> a quiescent state, whether preemption is disabled or not.
> 
> In any case for the current patch, perhaps a more robust test against nested
> hardirqs would be:
> 
> unsigned long cnt = preempt_count();
> bool preempt_bh_enabled = (!cnt || cnt == HARDIRQ_OFFSET)
> 
> Thanks.
> 
> >  
> >  	if (!(READ_ONCE(rnp->expmask) & rdp->grpmask) ||
> >  	    __this_cpu_read(rcu_data.cpu_no_qs.b.exp))
> >  		return;
> > -	if (rcu_is_cpu_rrupt_from_idle()) {
> > +	if (rcu_is_cpu_rrupt_from_idle() ||
> > +	    (IS_ENABLED(CONFIG_PREEMPT_COUNT) && preempt_bh_enabled)) {
> >  		rcu_report_exp_rdp(this_cpu_ptr(&rcu_data));
> >  		return;
> >  	}
> > -- 
> > 2.31.1.189.g2e36527f23
> >

Frederic Weisbecker Sept. 7, 2022, 3:14 p.m. UTC | #3

On Wed, Sep 07, 2022 at 07:57:59AM -0700, Paul E. McKenney wrote:
> On Wed, Sep 07, 2022 at 02:10:10PM +0200, Frederic Weisbecker wrote:
> > On Wed, Aug 31, 2022 at 11:07:58AM -0700, Paul E. McKenney wrote:
> > > From: Zqiang <[e-mail]>
> > > 
> > > Kernels built with CONFIG_PREEMPTION=n and CONFIG_PREEMPT_COUNT=y maintain
> > > preempt_count() state.  Because such kernels map __rcu_read_lock()
> > > and __rcu_read_unlock() to preempt_disable() and preempt_enable(),
> > > respectively, this allows the expedited grace period's !CONFIG_PREEMPT_RCU
> > > version of the rcu_exp_handler() IPI handler function to use
> > > preempt_count() to detect quiescent states.
> > > 
> > > This preempt_count() usage might seem to risk failures due to
> > > use of implicit RCU readers in portions of the kernel under #ifndef
> > > CONFIG_PREEMPTION, except that rcu_core() already disallows such implicit
> > > RCU readers.  The moral of this story is that you must use explicit
> > > read-side markings such as rcu_read_lock() or preempt_disable() even if
> > > the code knows that this kernel does not support preemption.
> > > 
> > > This commit therefore adds a preempt_count()-based check for a quiescent
> > > state in the !CONFIG_PREEMPT_RCU version of the rcu_exp_handler()
> > > function for kernels built with CONFIG_PREEMPT_COUNT=y, reporting an
> > > immediate quiescent state when the interrupted code had both preemption
> > > and softirqs enabled.
> > > 
> > > This change results in about a 2% reduction in expedited grace-period
> > > latency in kernels built with both CONFIG_PREEMPT_RCU=n and
> > > CONFIG_PREEMPT_COUNT=y.
> > > 
> > > Signed-off-by: Zqiang <[e-mail]>
> > > Signed-off-by: Paul E. McKenney <[e-mail]>
> > > Link: https://lore.kernel.org/all/[e-mail]/
> > > ---
> > >  kernel/rcu/tree_exp.h | 4 +++-
> > >  1 file changed, 3 insertions(+), 1 deletion(-)
> > > 
> > > diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h
> > > index be667583a5547..b07998159d1fa 100644
> > > --- a/kernel/rcu/tree_exp.h
> > > +++ b/kernel/rcu/tree_exp.h
> > > @@ -828,11 +828,13 @@ static void rcu_exp_handler(void *unused)
> > >  {
> > >  	struct rcu_data *rdp = this_cpu_ptr(&rcu_data);
> > >  	struct rcu_node *rnp = rdp->mynode;
> > > +	bool preempt_bh_enabled = !(preempt_count() & (PREEMPT_MASK |
> > > SOFTIRQ_MASK));
> > 
> > I don't know if nested hardirqs still exist. I only heard old rumours
> > about broken drivers. Should we take care of them?
> 
> Last I checked, certain tracing scenarios from irq handlers looked
> to RCU like nested irq handlers.  Given that, does your more robust
> approach below work correctly?

I haven't observed that but in any case, the check I propose
is more strict than the one on this patch. So in the worst case it's
a QS not reported if a nested interrupt is detected.

Thanks.

Patch

diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h
index be667583a5547..b07998159d1fa 100644
--- a/kernel/rcu/tree_exp.h
+++ b/kernel/rcu/tree_exp.h
@@ -828,11 +828,13 @@  static void rcu_exp_handler(void *unused)
 {
 	struct rcu_data *rdp = this_cpu_ptr(&rcu_data);
 	struct rcu_node *rnp = rdp->mynode;
+	bool preempt_bh_enabled = !(preempt_count() & (PREEMPT_MASK | SOFTIRQ_MASK));
 
 	if (!(READ_ONCE(rnp->expmask) & rdp->grpmask) ||
 	    __this_cpu_read(rcu_data.cpu_no_qs.b.exp))
 		return;
-	if (rcu_is_cpu_rrupt_from_idle()) {
+	if (rcu_is_cpu_rrupt_from_idle() ||
+	    (IS_ENABLED(CONFIG_PREEMPT_COUNT) && preempt_bh_enabled)) {
 		rcu_report_exp_rdp(this_cpu_ptr(&rcu_data));
 		return;
 	}