[PATCHv4,3/3] arm64: entry: avoid double-accounting IRQ RCU entry

Commit Message

Pingfan Liu Oct. 1, 2021, 2:44 p.m. UTC

When an IRQ is taken, some accounting needs to be performed to enter and
exit IRQ context around the IRQ handler. On arm64 some of this
accounting is performed by both the architecture code and the IRQ domain
code, resulting in calling rcu_irq_enter() twice per exception entry,
violating the expectations of the core RCU code, and resulting in
failing to identify quiescent periods correctly (e.g. in
rcu_is_cpu_rrupt_from_idle()).

To fix this, we must perform all the accounting from the architecture
code. We prevent the IRQ domain code from performing any accounting by
selecting HAVE_ARCH_IRQENTRY, and must call irq_enter_rcu() and
irq_exit_rcu() around invoking the root IRQ handler.

When we take a pNMI from a context with IRQs disabled, we'll perform the
necessary accounting as part of arm64_enter_nmi() and arm64_exit_nmi(),
and should only call irq_enter_rcu() and irq_exit_rcu() when we may have
taken a regular interrupt.

Signed-off-by: Pingfan Liu <[e-mail]>
Reviewed-by: Mark Rutland <[e-mail]>
Reviewed-by: Marc Zyngier <[e-mail]>
Cc: "Paul E. McKenney" <[e-mail]>
Cc: Catalin Marinas <[e-mail]>
Cc: Will Deacon <[e-mail]>
Cc: Joey Gouly <[e-mail]>
Cc: Sami Tolvanen <[e-mail]>
Cc: Julien Thierry <[e-mail]>
Cc: Thomas Gleixner <[e-mail]>
Cc: Yuichi Ito <[e-mail]>
Cc: [e-mail]
To: [e-mail]
---
 arch/arm64/Kconfig               | 1 +
 arch/arm64/kernel/entry-common.c | 4 ++++
 2 files changed, 5 insertions(+)

Patch

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 5c7ae4c3954b..c0ba052116b2 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -144,6 +144,7 @@  config ARM64
 	select HAVE_ARCH_BITREVERSE
 	select HAVE_ARCH_COMPILER_H
 	select HAVE_ARCH_HUGE_VMAP
+	select HAVE_ARCH_IRQENTRY
 	select HAVE_ARCH_JUMP_LABEL
 	select HAVE_ARCH_JUMP_LABEL_RELATIVE
 	select HAVE_ARCH_KASAN if !(ARM64_16K_PAGES && ARM64_VA_BITS_48)
diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c
index fecf046f0708..af931e63dc59 100644
--- a/arch/arm64/kernel/entry-common.c
+++ b/arch/arm64/kernel/entry-common.c
@@ -428,7 +428,9 @@  static __always_inline void
 __el1_interrupt(struct pt_regs *regs, void (*handler)(struct pt_regs *))
 {
 	enter_from_kernel_mode(regs);
+	irq_enter_rcu();
 	do_interrupt_handler(regs, handler);
+	irq_exit_rcu();
 	/*
 	 * Note: thread_info::preempt_count includes both thread_info::count
 	 * and thread_info::need_resched, and is not equivalent to
@@ -666,7 +668,9 @@  static void noinstr el0_interrupt(struct pt_regs *regs,
 	if (regs->pc & BIT(55))
 		arm64_apply_bp_hardening();
 
+	irq_enter_rcu();
 	do_interrupt_handler(regs, handler);
+	irq_exit_rcu();
 
 	exit_to_user_mode(regs);
 }