From patchwork Fri Aug 16 04:02:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neeraj Upadhyay X-Patchwork-Id: 13765394 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 240176F077; Fri, 16 Aug 2024 04:03:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723780999; cv=none; b=oHsQtGzi0J3NxXNEeUr4BezYhqkE6stgiqZRUWijylm2ykcAKJUYCXg8//ioulFTkNxG8L/FUakBxav159mBxoizmP+YTYEM8cYpd9qweRgOzgm099SxDsuXPS/Y80vUniUPtS+DuzSoh1x9SHlsSXaP8DtASgDTlcRxGxa4NoI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723780999; c=relaxed/simple; bh=Y5UbxV5Jyf6/dkj1ZZX1svOhGW+35AdNr+MhtBOj3OE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=BIbNkPaM1cm+CpI9CZfMOrq6M182CHVTz/gDLmi3Vl4lH+JdGaaFAVvq9N7q91nleYPALYe1vucC+FOjR34jkX91gmRcCPILcybOS4jMbjMvaUx98knKWUdDsR5whQsQNQ5LWjZm11kreYgE+NARuCT39NCkvMhZRiIehPX5meY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=GoNz66Pi; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="GoNz66Pi" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5281DC32782; Fri, 16 Aug 2024 04:03:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723780998; bh=Y5UbxV5Jyf6/dkj1ZZX1svOhGW+35AdNr+MhtBOj3OE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GoNz66PiS7K0T8geYloBMuLAx2vfNFADkyvNvDvWfxyKXzQUSqCpJpe/tAHIJmfqH 0pHQTcRQN+lz6HjE8Re7EP/iHlZOXtgKZYm+Cy90m6kZyrP/re47ZlWPbNzgM6pCb3 OXd79BomByhXb7zCO7SSW3JDpFMwhZKMa6Oz1DE8PvLHji3biqoWTXgJaHKTLTlLti lzpNoEDEF7cfwvxR1xsmAw0iwDqWTRD9VS9i6gDoLmCJboljXQvCix2RysZ6veRdkp DOb5x7l7d7jwGaGe1ZuO22VHhcRW2J1HVhEWwjZkSirl+GfkaYnyTR9zYFlCEYEad8 lPmABeGM6G6QQ== From: neeraj.upadhyay@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, paulmck@kernel.org, neeraj.upadhyay@kernel.org, neeraj.upadhyay@amd.com, boqun.feng@gmail.com, joel@joelfernandes.org, urezki@gmail.com, frederic@kernel.org, Valentin Schneider , Thomas Gleixner Subject: [PATCH rcu 01/26] treewide: context_tracking: Rename CONTEXT_* into CT_STATE_* Date: Fri, 16 Aug 2024 09:32:16 +0530 Message-Id: <20240816040241.17776-1-neeraj.upadhyay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240816040030.GA16986@neeraj.linux> References: <20240816040030.GA16986@neeraj.linux> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Valentin Schneider Context tracking state related symbols currently use a mix of the CONTEXT_ (e.g. CONTEXT_KERNEL) and CT_SATE_ (e.g. CT_STATE_MASK) prefixes. Clean up the naming and make the ctx_state enum use the CT_STATE_ prefix. Suggested-by: Frederic Weisbecker Signed-off-by: Valentin Schneider Acked-by: Frederic Weisbecker Acked-by: Thomas Gleixner Signed-off-by: Neeraj Upadhyay --- arch/Kconfig | 2 +- arch/arm64/kernel/entry-common.c | 2 +- arch/powerpc/include/asm/interrupt.h | 6 +++--- arch/powerpc/kernel/interrupt.c | 6 +++--- arch/powerpc/kernel/syscall.c | 2 +- arch/x86/entry/common.c | 2 +- include/linux/context_tracking.h | 16 ++++++++-------- include/linux/context_tracking_state.h | 20 ++++++++++---------- include/linux/entry-common.h | 2 +- kernel/context_tracking.c | 12 ++++++------ kernel/entry/common.c | 2 +- kernel/sched/core.c | 4 ++-- 12 files changed, 38 insertions(+), 38 deletions(-) diff --git a/arch/Kconfig b/arch/Kconfig index 975dd22a2dbd..4e2eaba9e305 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -862,7 +862,7 @@ config HAVE_CONTEXT_TRACKING_USER_OFFSTACK Architecture neither relies on exception_enter()/exception_exit() nor on schedule_user(). Also preempt_schedule_notrace() and preempt_schedule_irq() can't be called in a preemptible section - while context tracking is CONTEXT_USER. This feature reflects a sane + while context tracking is CT_STATE_USER. This feature reflects a sane entry implementation where the following requirements are met on critical entry code, ie: before user_exit() or after user_enter(): diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c index b77a15955f28..3fcd9d080bf2 100644 --- a/arch/arm64/kernel/entry-common.c +++ b/arch/arm64/kernel/entry-common.c @@ -103,7 +103,7 @@ static void noinstr exit_to_kernel_mode(struct pt_regs *regs) static __always_inline void __enter_from_user_mode(void) { lockdep_hardirqs_off(CALLER_ADDR0); - CT_WARN_ON(ct_state() != CONTEXT_USER); + CT_WARN_ON(ct_state() != CT_STATE_USER); user_exit_irqoff(); trace_hardirqs_off_finish(); mte_disable_tco_entry(current); diff --git a/arch/powerpc/include/asm/interrupt.h b/arch/powerpc/include/asm/interrupt.h index 2d6c886b40f4..23638d4e73ac 100644 --- a/arch/powerpc/include/asm/interrupt.h +++ b/arch/powerpc/include/asm/interrupt.h @@ -177,7 +177,7 @@ static inline void interrupt_enter_prepare(struct pt_regs *regs) if (user_mode(regs)) { kuap_lock(); - CT_WARN_ON(ct_state() != CONTEXT_USER); + CT_WARN_ON(ct_state() != CT_STATE_USER); user_exit_irqoff(); account_cpu_user_entry(); @@ -189,8 +189,8 @@ static inline void interrupt_enter_prepare(struct pt_regs *regs) * so avoid recursion. */ if (TRAP(regs) != INTERRUPT_PROGRAM) - CT_WARN_ON(ct_state() != CONTEXT_KERNEL && - ct_state() != CONTEXT_IDLE); + CT_WARN_ON(ct_state() != CT_STATE_KERNEL && + ct_state() != CT_STATE_IDLE); INT_SOFT_MASK_BUG_ON(regs, is_implicit_soft_masked(regs)); INT_SOFT_MASK_BUG_ON(regs, arch_irq_disabled_regs(regs) && search_kernel_restart_table(regs->nip)); diff --git a/arch/powerpc/kernel/interrupt.c b/arch/powerpc/kernel/interrupt.c index eca293794a1e..af62ec974b97 100644 --- a/arch/powerpc/kernel/interrupt.c +++ b/arch/powerpc/kernel/interrupt.c @@ -266,7 +266,7 @@ notrace unsigned long syscall_exit_prepare(unsigned long r3, unsigned long ret = 0; bool is_not_scv = !IS_ENABLED(CONFIG_PPC_BOOK3S_64) || !scv; - CT_WARN_ON(ct_state() == CONTEXT_USER); + CT_WARN_ON(ct_state() == CT_STATE_USER); kuap_assert_locked(); @@ -344,7 +344,7 @@ notrace unsigned long interrupt_exit_user_prepare(struct pt_regs *regs) BUG_ON(regs_is_unrecoverable(regs)); BUG_ON(arch_irq_disabled_regs(regs)); - CT_WARN_ON(ct_state() == CONTEXT_USER); + CT_WARN_ON(ct_state() == CT_STATE_USER); /* * We don't need to restore AMR on the way back to userspace for KUAP. @@ -386,7 +386,7 @@ notrace unsigned long interrupt_exit_kernel_prepare(struct pt_regs *regs) if (!IS_ENABLED(CONFIG_PPC_BOOK3E_64) && TRAP(regs) != INTERRUPT_PROGRAM && TRAP(regs) != INTERRUPT_PERFMON) - CT_WARN_ON(ct_state() == CONTEXT_USER); + CT_WARN_ON(ct_state() == CT_STATE_USER); kuap = kuap_get_and_assert_locked(); diff --git a/arch/powerpc/kernel/syscall.c b/arch/powerpc/kernel/syscall.c index f6f868e817e6..be159ad4b77b 100644 --- a/arch/powerpc/kernel/syscall.c +++ b/arch/powerpc/kernel/syscall.c @@ -27,7 +27,7 @@ notrace long system_call_exception(struct pt_regs *regs, unsigned long r0) trace_hardirqs_off(); /* finish reconciling */ - CT_WARN_ON(ct_state() == CONTEXT_KERNEL); + CT_WARN_ON(ct_state() == CT_STATE_KERNEL); user_exit_irqoff(); BUG_ON(regs_is_unrecoverable(regs)); diff --git a/arch/x86/entry/common.c b/arch/x86/entry/common.c index 51cc9c7cb9bd..94941c5a10ac 100644 --- a/arch/x86/entry/common.c +++ b/arch/x86/entry/common.c @@ -150,7 +150,7 @@ early_param("ia32_emulation", ia32_emulation_override_cmdline); #endif /* - * Invoke a 32-bit syscall. Called with IRQs on in CONTEXT_KERNEL. + * Invoke a 32-bit syscall. Called with IRQs on in CT_STATE_KERNEL. */ static __always_inline void do_syscall_32_irqs_on(struct pt_regs *regs, int nr) { diff --git a/include/linux/context_tracking.h b/include/linux/context_tracking.h index 6e76b9dba00e..28fcfa184903 100644 --- a/include/linux/context_tracking.h +++ b/include/linux/context_tracking.h @@ -26,26 +26,26 @@ extern void user_exit_callable(void); static inline void user_enter(void) { if (context_tracking_enabled()) - ct_user_enter(CONTEXT_USER); + ct_user_enter(CT_STATE_USER); } static inline void user_exit(void) { if (context_tracking_enabled()) - ct_user_exit(CONTEXT_USER); + ct_user_exit(CT_STATE_USER); } /* Called with interrupts disabled. */ static __always_inline void user_enter_irqoff(void) { if (context_tracking_enabled()) - __ct_user_enter(CONTEXT_USER); + __ct_user_enter(CT_STATE_USER); } static __always_inline void user_exit_irqoff(void) { if (context_tracking_enabled()) - __ct_user_exit(CONTEXT_USER); + __ct_user_exit(CT_STATE_USER); } static inline enum ctx_state exception_enter(void) @@ -57,7 +57,7 @@ static inline enum ctx_state exception_enter(void) return 0; prev_ctx = __ct_state(); - if (prev_ctx != CONTEXT_KERNEL) + if (prev_ctx != CT_STATE_KERNEL) ct_user_exit(prev_ctx); return prev_ctx; @@ -67,7 +67,7 @@ static inline void exception_exit(enum ctx_state prev_ctx) { if (!IS_ENABLED(CONFIG_HAVE_CONTEXT_TRACKING_USER_OFFSTACK) && context_tracking_enabled()) { - if (prev_ctx != CONTEXT_KERNEL) + if (prev_ctx != CT_STATE_KERNEL) ct_user_enter(prev_ctx); } } @@ -75,7 +75,7 @@ static inline void exception_exit(enum ctx_state prev_ctx) static __always_inline bool context_tracking_guest_enter(void) { if (context_tracking_enabled()) - __ct_user_enter(CONTEXT_GUEST); + __ct_user_enter(CT_STATE_GUEST); return context_tracking_enabled_this_cpu(); } @@ -83,7 +83,7 @@ static __always_inline bool context_tracking_guest_enter(void) static __always_inline void context_tracking_guest_exit(void) { if (context_tracking_enabled()) - __ct_user_exit(CONTEXT_GUEST); + __ct_user_exit(CT_STATE_GUEST); } #define CT_WARN_ON(cond) WARN_ON(context_tracking_enabled() && (cond)) diff --git a/include/linux/context_tracking_state.h b/include/linux/context_tracking_state.h index bbff5f7f8803..f1c53125edee 100644 --- a/include/linux/context_tracking_state.h +++ b/include/linux/context_tracking_state.h @@ -10,18 +10,18 @@ #define DYNTICK_IRQ_NONIDLE ((LONG_MAX / 2) + 1) enum ctx_state { - CONTEXT_DISABLED = -1, /* returned by ct_state() if unknown */ - CONTEXT_KERNEL = 0, - CONTEXT_IDLE = 1, - CONTEXT_USER = 2, - CONTEXT_GUEST = 3, - CONTEXT_MAX = 4, + CT_STATE_DISABLED = -1, /* returned by ct_state() if unknown */ + CT_STATE_KERNEL = 0, + CT_STATE_IDLE = 1, + CT_STATE_USER = 2, + CT_STATE_GUEST = 3, + CT_STATE_MAX = 4, }; /* Even value for idle, else odd. */ -#define RCU_DYNTICKS_IDX CONTEXT_MAX +#define RCU_DYNTICKS_IDX CT_STATE_MAX -#define CT_STATE_MASK (CONTEXT_MAX - 1) +#define CT_STATE_MASK (CT_STATE_MAX - 1) #define CT_DYNTICKS_MASK (~CT_STATE_MASK) struct context_tracking { @@ -123,14 +123,14 @@ static inline bool context_tracking_enabled_this_cpu(void) * * Returns the current cpu's context tracking state if context tracking * is enabled. If context tracking is disabled, returns - * CONTEXT_DISABLED. This should be used primarily for debugging. + * CT_STATE_DISABLED. This should be used primarily for debugging. */ static __always_inline int ct_state(void) { int ret; if (!context_tracking_enabled()) - return CONTEXT_DISABLED; + return CT_STATE_DISABLED; preempt_disable(); ret = __ct_state(); diff --git a/include/linux/entry-common.h b/include/linux/entry-common.h index b0fb775a600d..1e50cdb83ae5 100644 --- a/include/linux/entry-common.h +++ b/include/linux/entry-common.h @@ -108,7 +108,7 @@ static __always_inline void enter_from_user_mode(struct pt_regs *regs) arch_enter_from_user_mode(regs); lockdep_hardirqs_off(CALLER_ADDR0); - CT_WARN_ON(__ct_state() != CONTEXT_USER); + CT_WARN_ON(__ct_state() != CT_STATE_USER); user_exit_irqoff(); instrumentation_begin(); diff --git a/kernel/context_tracking.c b/kernel/context_tracking.c index 24b1e1143260..4bb5751af994 100644 --- a/kernel/context_tracking.c +++ b/kernel/context_tracking.c @@ -317,7 +317,7 @@ void noinstr ct_nmi_enter(void) void noinstr ct_idle_enter(void) { WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !raw_irqs_disabled()); - ct_kernel_exit(false, RCU_DYNTICKS_IDX + CONTEXT_IDLE); + ct_kernel_exit(false, RCU_DYNTICKS_IDX + CT_STATE_IDLE); } EXPORT_SYMBOL_GPL(ct_idle_enter); @@ -335,7 +335,7 @@ void noinstr ct_idle_exit(void) unsigned long flags; raw_local_irq_save(flags); - ct_kernel_enter(false, RCU_DYNTICKS_IDX - CONTEXT_IDLE); + ct_kernel_enter(false, RCU_DYNTICKS_IDX - CT_STATE_IDLE); raw_local_irq_restore(flags); } EXPORT_SYMBOL_GPL(ct_idle_exit); @@ -485,7 +485,7 @@ void noinstr __ct_user_enter(enum ctx_state state) * user_exit() or ct_irq_enter(). Let's remove RCU's dependency * on the tick. */ - if (state == CONTEXT_USER) { + if (state == CT_STATE_USER) { instrumentation_begin(); trace_user_enter(0); vtime_user_enter(current); @@ -621,7 +621,7 @@ void noinstr __ct_user_exit(enum ctx_state state) * run a RCU read side critical section anytime. */ ct_kernel_enter(true, RCU_DYNTICKS_IDX - state); - if (state == CONTEXT_USER) { + if (state == CT_STATE_USER) { instrumentation_begin(); vtime_user_exit(current); trace_user_exit(0); @@ -634,12 +634,12 @@ void noinstr __ct_user_exit(enum ctx_state state) * In this we case we don't care about any concurrency/ordering. */ if (!IS_ENABLED(CONFIG_CONTEXT_TRACKING_IDLE)) - raw_atomic_set(&ct->state, CONTEXT_KERNEL); + raw_atomic_set(&ct->state, CT_STATE_KERNEL); } else { if (!IS_ENABLED(CONFIG_CONTEXT_TRACKING_IDLE)) { /* Tracking for vtime only, no concurrent RCU EQS accounting */ - raw_atomic_set(&ct->state, CONTEXT_KERNEL); + raw_atomic_set(&ct->state, CT_STATE_KERNEL); } else { /* * Tracking for vtime and RCU EQS. Make sure we don't race diff --git a/kernel/entry/common.c b/kernel/entry/common.c index 90843cc38588..5b6934e23c21 100644 --- a/kernel/entry/common.c +++ b/kernel/entry/common.c @@ -182,7 +182,7 @@ static void syscall_exit_to_user_mode_prepare(struct pt_regs *regs) unsigned long work = READ_ONCE(current_thread_info()->syscall_work); unsigned long nr = syscall_get_nr(current, regs); - CT_WARN_ON(ct_state() != CONTEXT_KERNEL); + CT_WARN_ON(ct_state() != CT_STATE_KERNEL); if (IS_ENABLED(CONFIG_PROVE_LOCKING)) { if (WARN(irqs_disabled(), "syscall %lu left IRQs disabled", nr)) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index a9f655025607..93a845fe8449 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -5762,7 +5762,7 @@ static inline void schedule_debug(struct task_struct *prev, bool preempt) preempt_count_set(PREEMPT_DISABLED); } rcu_sleep_check(); - SCHED_WARN_ON(ct_state() == CONTEXT_USER); + SCHED_WARN_ON(ct_state() == CT_STATE_USER); profile_hit(SCHED_PROFILING, __builtin_return_address(0)); @@ -6658,7 +6658,7 @@ asmlinkage __visible void __sched schedule_user(void) * we find a better solution. * * NB: There are buggy callers of this function. Ideally we - * should warn if prev_state != CONTEXT_USER, but that will trigger + * should warn if prev_state != CT_STATE_USER, but that will trigger * too frequently to make sense yet. */ enum ctx_state prev_state = exception_enter(); From patchwork Fri Aug 16 04:02:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neeraj Upadhyay X-Patchwork-Id: 13765395 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 70E5078C6D; Fri, 16 Aug 2024 04:03:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723781009; cv=none; b=qJA0SuHhnh4+Av1yCtUpnsZNmDFqryKsn+4hX2lpXYOip/pSiHCqsgX/NGoPm4UHMeNhaenukhaJ4V28QfwsIRSEh5bTEno+AHpHXQxgdayz2VnrPIycpsdeCoRawCD+t7L5CrW6LSI98dFV7aBX86dTpvQh8z8mR1F0Zi0y600= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723781009; c=relaxed/simple; bh=k7rZkydd8ZvDl6ML5poNkTPePqihNQL2z42FUtsuzHk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=S/7HzDKFQqRIrrRt8TMWOIAMlX+by+SDQd2m9OfJGBP7vmytgsGqn8F/n/fOU5rdKqO+5wccoYYfaLT0uwD8lSXVkz4v9z8ppXXtk5uX0YDMbwZWuSYwRFVoZ9eLqq7H1xm3Rag6GSQyy4tG9tofgVx1avX2gAerBKnbLe0mztI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=MQjVwdnu; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="MQjVwdnu" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 55FFDC4AF0B; Fri, 16 Aug 2024 04:03:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723781008; bh=k7rZkydd8ZvDl6ML5poNkTPePqihNQL2z42FUtsuzHk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=MQjVwdnuNoBGDjnFYSFqy6OOUNZ3DhzyJKs+q1hOAzNEbJeVsNNKTUXRlxlX8UGEb bHglVfvZe5AJVufVRQDt8ZHtsMtifYgIJgD8q5nUs+EYzuAtpqB69KYVrwfwmoKqh1 Coh1CLhWLuiGT1adXxMxKXQzd9m2cvcvxFXFuxJljPSvc17xbH96E8uRlKdWe0SfHq j7Lc4jpauZpqpEDMgNVPNPtaR8rpzXonl4A/GYLtxbkyYQqvOGuXHjzhXz6kUMXmqk y/LB8YvXvfBGmrXViZ/Rbnpeahksy/R30be3hGxSq9TPvarWWizRbxUNFOpSm/O9ub aPCvzr7O8DG3A== From: neeraj.upadhyay@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, paulmck@kernel.org, neeraj.upadhyay@kernel.org, neeraj.upadhyay@amd.com, boqun.feng@gmail.com, joel@joelfernandes.org, urezki@gmail.com, frederic@kernel.org, Valentin Schneider , Thomas Gleixner Subject: [PATCH rcu 02/26] context_tracking, rcu: Rename RCU_DYNTICKS_IDX into CT_RCU_WATCHING Date: Fri, 16 Aug 2024 09:32:17 +0530 Message-Id: <20240816040241.17776-2-neeraj.upadhyay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240816040030.GA16986@neeraj.linux> References: <20240816040030.GA16986@neeraj.linux> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Valentin Schneider The symbols relating to the CT_STATE part of context_tracking.state are now all prefixed with CT_STATE. The RCU dynticks counter part of that atomic variable still involves symbols with different prefixes, align them all to be prefixed with CT_RCU_WATCHING. Suggested-by: "Paul E. McKenney" Signed-off-by: Valentin Schneider Reviewed-by: Frederic Weisbecker Acked-by: Thomas Gleixner Signed-off-by: Neeraj Upadhyay --- include/linux/context_tracking.h | 6 +++--- include/linux/context_tracking_state.h | 12 ++++++------ kernel/context_tracking.c | 22 +++++++++++----------- kernel/rcu/tree.c | 12 ++++++------ 4 files changed, 26 insertions(+), 26 deletions(-) diff --git a/include/linux/context_tracking.h b/include/linux/context_tracking.h index 28fcfa184903..a6c36780cc3b 100644 --- a/include/linux/context_tracking.h +++ b/include/linux/context_tracking.h @@ -119,7 +119,7 @@ extern void ct_idle_exit(void); */ static __always_inline bool rcu_dynticks_curr_cpu_in_eqs(void) { - return !(raw_atomic_read(this_cpu_ptr(&context_tracking.state)) & RCU_DYNTICKS_IDX); + return !(raw_atomic_read(this_cpu_ptr(&context_tracking.state)) & CT_RCU_WATCHING); } /* @@ -142,7 +142,7 @@ static __always_inline bool warn_rcu_enter(void) preempt_disable_notrace(); if (rcu_dynticks_curr_cpu_in_eqs()) { ret = true; - ct_state_inc(RCU_DYNTICKS_IDX); + ct_state_inc(CT_RCU_WATCHING); } return ret; @@ -151,7 +151,7 @@ static __always_inline bool warn_rcu_enter(void) static __always_inline void warn_rcu_exit(bool rcu) { if (rcu) - ct_state_inc(RCU_DYNTICKS_IDX); + ct_state_inc(CT_RCU_WATCHING); preempt_enable_notrace(); } diff --git a/include/linux/context_tracking_state.h b/include/linux/context_tracking_state.h index f1c53125edee..94d6a935af3b 100644 --- a/include/linux/context_tracking_state.h +++ b/include/linux/context_tracking_state.h @@ -18,11 +18,11 @@ enum ctx_state { CT_STATE_MAX = 4, }; -/* Even value for idle, else odd. */ -#define RCU_DYNTICKS_IDX CT_STATE_MAX +/* Odd value for watching, else even. */ +#define CT_RCU_WATCHING CT_STATE_MAX #define CT_STATE_MASK (CT_STATE_MAX - 1) -#define CT_DYNTICKS_MASK (~CT_STATE_MASK) +#define CT_RCU_WATCHING_MASK (~CT_STATE_MASK) struct context_tracking { #ifdef CONFIG_CONTEXT_TRACKING_USER @@ -58,21 +58,21 @@ static __always_inline int __ct_state(void) #ifdef CONFIG_CONTEXT_TRACKING_IDLE static __always_inline int ct_dynticks(void) { - return atomic_read(this_cpu_ptr(&context_tracking.state)) & CT_DYNTICKS_MASK; + return atomic_read(this_cpu_ptr(&context_tracking.state)) & CT_RCU_WATCHING_MASK; } static __always_inline int ct_dynticks_cpu(int cpu) { struct context_tracking *ct = per_cpu_ptr(&context_tracking, cpu); - return atomic_read(&ct->state) & CT_DYNTICKS_MASK; + return atomic_read(&ct->state) & CT_RCU_WATCHING_MASK; } static __always_inline int ct_dynticks_cpu_acquire(int cpu) { struct context_tracking *ct = per_cpu_ptr(&context_tracking, cpu); - return atomic_read_acquire(&ct->state) & CT_DYNTICKS_MASK; + return atomic_read_acquire(&ct->state) & CT_RCU_WATCHING_MASK; } static __always_inline long ct_dynticks_nesting(void) diff --git a/kernel/context_tracking.c b/kernel/context_tracking.c index 4bb5751af994..b2589bc59e18 100644 --- a/kernel/context_tracking.c +++ b/kernel/context_tracking.c @@ -31,7 +31,7 @@ DEFINE_PER_CPU(struct context_tracking, context_tracking) = { .dynticks_nesting = 1, .dynticks_nmi_nesting = DYNTICK_IRQ_NONIDLE, #endif - .state = ATOMIC_INIT(RCU_DYNTICKS_IDX), + .state = ATOMIC_INIT(CT_RCU_WATCHING), }; EXPORT_SYMBOL_GPL(context_tracking); @@ -90,7 +90,7 @@ static noinstr void ct_kernel_exit_state(int offset) rcu_dynticks_task_trace_enter(); // Before ->dynticks update! seq = ct_state_inc(offset); // RCU is no longer watching. Better be in extended quiescent state! - WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && (seq & RCU_DYNTICKS_IDX)); + WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && (seq & CT_RCU_WATCHING)); } /* @@ -110,7 +110,7 @@ static noinstr void ct_kernel_enter_state(int offset) seq = ct_state_inc(offset); // RCU is now watching. Better not be in an extended quiescent state! rcu_dynticks_task_trace_exit(); // After ->dynticks update! - WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !(seq & RCU_DYNTICKS_IDX)); + WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !(seq & CT_RCU_WATCHING)); } /* @@ -236,7 +236,7 @@ void noinstr ct_nmi_exit(void) instrumentation_end(); // RCU is watching here ... - ct_kernel_exit_state(RCU_DYNTICKS_IDX); + ct_kernel_exit_state(CT_RCU_WATCHING); // ... but is no longer watching here. if (!in_nmi()) @@ -277,7 +277,7 @@ void noinstr ct_nmi_enter(void) rcu_dynticks_task_exit(); // RCU is not watching here ... - ct_kernel_enter_state(RCU_DYNTICKS_IDX); + ct_kernel_enter_state(CT_RCU_WATCHING); // ... but is watching here. instrumentation_begin(); @@ -317,7 +317,7 @@ void noinstr ct_nmi_enter(void) void noinstr ct_idle_enter(void) { WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !raw_irqs_disabled()); - ct_kernel_exit(false, RCU_DYNTICKS_IDX + CT_STATE_IDLE); + ct_kernel_exit(false, CT_RCU_WATCHING + CT_STATE_IDLE); } EXPORT_SYMBOL_GPL(ct_idle_enter); @@ -335,7 +335,7 @@ void noinstr ct_idle_exit(void) unsigned long flags; raw_local_irq_save(flags); - ct_kernel_enter(false, RCU_DYNTICKS_IDX - CT_STATE_IDLE); + ct_kernel_enter(false, CT_RCU_WATCHING - CT_STATE_IDLE); raw_local_irq_restore(flags); } EXPORT_SYMBOL_GPL(ct_idle_exit); @@ -504,7 +504,7 @@ void noinstr __ct_user_enter(enum ctx_state state) * CPU doesn't need to maintain the tick for RCU maintenance purposes * when the CPU runs in userspace. */ - ct_kernel_exit(true, RCU_DYNTICKS_IDX + state); + ct_kernel_exit(true, CT_RCU_WATCHING + state); /* * Special case if we only track user <-> kernel transitions for tickless @@ -534,7 +534,7 @@ void noinstr __ct_user_enter(enum ctx_state state) /* * Tracking for vtime and RCU EQS. Make sure we don't race * with NMIs. OTOH we don't care about ordering here since - * RCU only requires RCU_DYNTICKS_IDX increments to be fully + * RCU only requires CT_RCU_WATCHING increments to be fully * ordered. */ raw_atomic_add(state, &ct->state); @@ -620,7 +620,7 @@ void noinstr __ct_user_exit(enum ctx_state state) * Exit RCU idle mode while entering the kernel because it can * run a RCU read side critical section anytime. */ - ct_kernel_enter(true, RCU_DYNTICKS_IDX - state); + ct_kernel_enter(true, CT_RCU_WATCHING - state); if (state == CT_STATE_USER) { instrumentation_begin(); vtime_user_exit(current); @@ -644,7 +644,7 @@ void noinstr __ct_user_exit(enum ctx_state state) /* * Tracking for vtime and RCU EQS. Make sure we don't race * with NMIs. OTOH we don't care about ordering here since - * RCU only requires RCU_DYNTICKS_IDX increments to be fully + * RCU only requires CT_RCU_WATCHING increments to be fully * ordered. */ raw_atomic_sub(state, &ct->state); diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index e641cc681901..04f87c44385c 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -294,9 +294,9 @@ void rcu_softirq_qs(void) */ static void rcu_dynticks_eqs_online(void) { - if (ct_dynticks() & RCU_DYNTICKS_IDX) + if (ct_dynticks() & CT_RCU_WATCHING) return; - ct_state_inc(RCU_DYNTICKS_IDX); + ct_state_inc(CT_RCU_WATCHING); } /* @@ -305,7 +305,7 @@ static void rcu_dynticks_eqs_online(void) */ static bool rcu_dynticks_in_eqs(int snap) { - return !(snap & RCU_DYNTICKS_IDX); + return !(snap & CT_RCU_WATCHING); } /* @@ -335,7 +335,7 @@ bool rcu_dynticks_zero_in_eqs(int cpu, int *vp) int snap; // If not quiescent, force back to earlier extended quiescent state. - snap = ct_dynticks_cpu(cpu) & ~RCU_DYNTICKS_IDX; + snap = ct_dynticks_cpu(cpu) & ~CT_RCU_WATCHING; smp_rmb(); // Order ->dynticks and *vp reads. if (READ_ONCE(*vp)) return false; // Non-zero, so report failure; @@ -361,9 +361,9 @@ notrace void rcu_momentary_dyntick_idle(void) int seq; raw_cpu_write(rcu_data.rcu_need_heavy_qs, false); - seq = ct_state_inc(2 * RCU_DYNTICKS_IDX); + seq = ct_state_inc(2 * CT_RCU_WATCHING); /* It is illegal to call this from idle state. */ - WARN_ON_ONCE(!(seq & RCU_DYNTICKS_IDX)); + WARN_ON_ONCE(!(seq & CT_RCU_WATCHING)); rcu_preempt_deferred_qs(current); } EXPORT_SYMBOL_GPL(rcu_momentary_dyntick_idle); From patchwork Fri Aug 16 04:02:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neeraj Upadhyay X-Patchwork-Id: 13765396 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 657787DA8B; Fri, 16 Aug 2024 04:03:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723781018; cv=none; b=FJ9h7aZYH1JHKpsAGz/GIGcdCLIMk7d5qNPTIYo6Xj/D7IvX45+Lo9+Zei6iHbEhlq+B1ESdLdcWGr7oXavfmXHCtDDFDnjaoIq4NJrnSmsn4Fej+xdkjqhO5KdoXdF7Lav2jGn4gS7JzPxKQSA9CU1WDUH6YiHPEIBt1EOQmWU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723781018; c=relaxed/simple; bh=++0c0XEDnidLQRlUlDS91vCsbWjdldBAxPFLpDWGdwU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=N4kybxCa1TnLsgU1VOf6i8mmIC0IAGRH97T/0xzOWnCNGDPFji4NlOKvY7D15MeCdCywF3Ne06yWd+JrCe1W3j8s1Pvce50MPUUjXX2enSAnzMt8Y1JkLYAGlekDNWEsUcvyj9K5w8RbGdoxxGhIZUA0QfovagUld64pHmmvtGI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=B0VwhuE1; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="B0VwhuE1" Received: by smtp.kernel.org (Postfix) with ESMTPSA id EEC3AC4AF09; Fri, 16 Aug 2024 04:03:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723781016; bh=++0c0XEDnidLQRlUlDS91vCsbWjdldBAxPFLpDWGdwU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=B0VwhuE1rAYTlNgAUWRuEvXob87Z4G8L/RV5WidbMSjHaJqA2ANNkrHAM5BExf7ZF qldMWX/dwTqX2VcDlOwZoRlEMCrUzoOYrw518F+tDonm5kzehRyM1rv4BB6kl5AnMP QgvZdegFbDQiyofVjxDOPTA6QmUkoH4MiCo3PSucZgwmqDwRXImpwlKVrXnZPbAHmb jP9oNlzJA/MRE01kAk1KRWO5nQ64k7LNhtj0onjNV9hGdA7ViKDcs6Y0aIA7M0Y2xK yuSwA6OgG0+hnsS2tbz1LMPqLz1E7IDeMrXPx/ACg3VvO4bi+EBARQKN8IxqKv2qKT qMo7ShM4aJxpw== From: neeraj.upadhyay@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, paulmck@kernel.org, neeraj.upadhyay@kernel.org, neeraj.upadhyay@amd.com, boqun.feng@gmail.com, joel@joelfernandes.org, urezki@gmail.com, frederic@kernel.org, Valentin Schneider Subject: [PATCH rcu 03/26] context_tracking, rcu: Rename ct_dynticks() into ct_rcu_watching() Date: Fri, 16 Aug 2024 09:32:18 +0530 Message-Id: <20240816040241.17776-3-neeraj.upadhyay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240816040030.GA16986@neeraj.linux> References: <20240816040030.GA16986@neeraj.linux> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Valentin Schneider The context_tracking.state RCU_DYNTICKS subvariable has been renamed to RCU_WATCHING, reflect that change in the related helpers. Signed-off-by: Valentin Schneider Reviewed-by: Frederic Weisbecker Signed-off-by: Neeraj Upadhyay --- include/linux/context_tracking_state.h | 2 +- kernel/context_tracking.c | 10 +++++----- kernel/rcu/tree.c | 2 +- 3 files changed, 7 insertions(+), 7 deletions(-) diff --git a/include/linux/context_tracking_state.h b/include/linux/context_tracking_state.h index 94d6a935af3b..cb90d8c17810 100644 --- a/include/linux/context_tracking_state.h +++ b/include/linux/context_tracking_state.h @@ -56,7 +56,7 @@ static __always_inline int __ct_state(void) #endif #ifdef CONFIG_CONTEXT_TRACKING_IDLE -static __always_inline int ct_dynticks(void) +static __always_inline int ct_rcu_watching(void) { return atomic_read(this_cpu_ptr(&context_tracking.state)) & CT_RCU_WATCHING_MASK; } diff --git a/kernel/context_tracking.c b/kernel/context_tracking.c index b2589bc59e18..868ae0bcd4be 100644 --- a/kernel/context_tracking.c +++ b/kernel/context_tracking.c @@ -137,7 +137,7 @@ static void noinstr ct_kernel_exit(bool user, int offset) instrumentation_begin(); lockdep_assert_irqs_disabled(); - trace_rcu_dyntick(TPS("Start"), ct_dynticks_nesting(), 0, ct_dynticks()); + trace_rcu_dyntick(TPS("Start"), ct_dynticks_nesting(), 0, ct_rcu_watching()); WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !user && !is_idle_task(current)); rcu_preempt_deferred_qs(current); @@ -182,7 +182,7 @@ static void noinstr ct_kernel_enter(bool user, int offset) // instrumentation for the noinstr ct_kernel_enter_state() instrument_atomic_write(&ct->state, sizeof(ct->state)); - trace_rcu_dyntick(TPS("End"), ct_dynticks_nesting(), 1, ct_dynticks()); + trace_rcu_dyntick(TPS("End"), ct_dynticks_nesting(), 1, ct_rcu_watching()); WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !user && !is_idle_task(current)); WRITE_ONCE(ct->dynticks_nesting, 1); WARN_ON_ONCE(ct_dynticks_nmi_nesting()); @@ -220,7 +220,7 @@ void noinstr ct_nmi_exit(void) */ if (ct_dynticks_nmi_nesting() != 1) { trace_rcu_dyntick(TPS("--="), ct_dynticks_nmi_nesting(), ct_dynticks_nmi_nesting() - 2, - ct_dynticks()); + ct_rcu_watching()); WRITE_ONCE(ct->dynticks_nmi_nesting, /* No store tearing. */ ct_dynticks_nmi_nesting() - 2); instrumentation_end(); @@ -228,7 +228,7 @@ void noinstr ct_nmi_exit(void) } /* This NMI interrupted an RCU-idle CPU, restore RCU-idleness. */ - trace_rcu_dyntick(TPS("Startirq"), ct_dynticks_nmi_nesting(), 0, ct_dynticks()); + trace_rcu_dyntick(TPS("Startirq"), ct_dynticks_nmi_nesting(), 0, ct_rcu_watching()); WRITE_ONCE(ct->dynticks_nmi_nesting, 0); /* Avoid store tearing. */ // instrumentation for the noinstr ct_kernel_exit_state() @@ -296,7 +296,7 @@ void noinstr ct_nmi_enter(void) trace_rcu_dyntick(incby == 1 ? TPS("Endirq") : TPS("++="), ct_dynticks_nmi_nesting(), - ct_dynticks_nmi_nesting() + incby, ct_dynticks()); + ct_dynticks_nmi_nesting() + incby, ct_rcu_watching()); instrumentation_end(); WRITE_ONCE(ct->dynticks_nmi_nesting, /* Prevent store tearing. */ ct_dynticks_nmi_nesting() + incby); diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 04f87c44385c..3a2f0325aa43 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -294,7 +294,7 @@ void rcu_softirq_qs(void) */ static void rcu_dynticks_eqs_online(void) { - if (ct_dynticks() & CT_RCU_WATCHING) + if (ct_rcu_watching() & CT_RCU_WATCHING) return; ct_state_inc(CT_RCU_WATCHING); } From patchwork Fri Aug 16 04:02:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neeraj Upadhyay X-Patchwork-Id: 13765397 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 57F4874418; Fri, 16 Aug 2024 04:03:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723781026; cv=none; b=Y8+zTlvYZLQkSi7zYnfB0rLXIyVsLLSJdt8QidvCS/29l5pvFK2GmpdcvoTGwj3vpLsgNXZY0PE0KiNxAZNGIk8v3pLXnaBc/eQAa6z24ClzSzCVN4LxhuLe+VzyAOoH4JB72ojNFNqqaQdHI92j+yuHzTDr4CkwKBkCs5Wv8GQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723781026; c=relaxed/simple; bh=xOSNuSYiybYrwARpAf5EHCoOy7qhk0v8Ujjxxy15+so=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=RUPD811kas8LRS/g66GNXUfQ/jmb65VSbslFtmg1UCExHd+RFf/NqrMJ8LPlI+57vIA9L11j2gYmpTw+qIXYw4fXmoXnawHmTNubPkz3ivoent2dIgmeUpbkDQAsbSBEtu35WZgICf4eJ/1AdgcCZL6zXkJeg5koIZfU+TrbLr0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=mTd+BdpV; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="mTd+BdpV" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6EC2EC4AF09; Fri, 16 Aug 2024 04:03:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723781026; bh=xOSNuSYiybYrwARpAf5EHCoOy7qhk0v8Ujjxxy15+so=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=mTd+BdpVpVK5DsXh4z81ObLJrJzyzUNlhN9cxx2Ho3UJGIwytm0E3AVBhnIljXbao 3rOy7x2y/8wvEw8zuBCtA/GbYss++iDjtu/hX6QXhoITMqOcHNn0kIjJju0iuwz0gG +43VEZC3baw8rUFr0IKBX2YHlvNLjxH4izvnhTEhZeHnrQHadXTCejmJ98ryO3tsj0 /41oNDiuWbKPGG3UjUPGmsbY1FD7TjygKoEGzNTVkeJNnjQRCQfpmWT0GEC5qAwAky xRgjg7UWXuiMiBGL9hFcqdYepNzn861GH7hY/47PDh5sDSbUGYrrJHG6va7GAJMyNs EbCRxDII/Notg== From: neeraj.upadhyay@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, paulmck@kernel.org, neeraj.upadhyay@kernel.org, neeraj.upadhyay@amd.com, boqun.feng@gmail.com, joel@joelfernandes.org, urezki@gmail.com, frederic@kernel.org, Valentin Schneider Subject: [PATCH rcu 04/26] context_tracking, rcu: Rename ct_dynticks_cpu() into ct_rcu_watching_cpu() Date: Fri, 16 Aug 2024 09:32:19 +0530 Message-Id: <20240816040241.17776-4-neeraj.upadhyay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240816040030.GA16986@neeraj.linux> References: <20240816040030.GA16986@neeraj.linux> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Valentin Schneider The context_tracking.state RCU_DYNTICKS subvariable has been renamed to RCU_WATCHING, reflect that change in the related helpers. Signed-off-by: Valentin Schneider Reviewed-by: Frederic Weisbecker Signed-off-by: Neeraj Upadhyay --- include/linux/context_tracking_state.h | 2 +- kernel/rcu/tree.c | 10 +++++----- kernel/rcu/tree_stall.h | 4 ++-- 3 files changed, 8 insertions(+), 8 deletions(-) diff --git a/include/linux/context_tracking_state.h b/include/linux/context_tracking_state.h index cb90d8c17810..ad5a06a42b4a 100644 --- a/include/linux/context_tracking_state.h +++ b/include/linux/context_tracking_state.h @@ -61,7 +61,7 @@ static __always_inline int ct_rcu_watching(void) return atomic_read(this_cpu_ptr(&context_tracking.state)) & CT_RCU_WATCHING_MASK; } -static __always_inline int ct_dynticks_cpu(int cpu) +static __always_inline int ct_rcu_watching_cpu(int cpu) { struct context_tracking *ct = per_cpu_ptr(&context_tracking, cpu); diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 3a2f0325aa43..e7f612e9f7e5 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -335,14 +335,14 @@ bool rcu_dynticks_zero_in_eqs(int cpu, int *vp) int snap; // If not quiescent, force back to earlier extended quiescent state. - snap = ct_dynticks_cpu(cpu) & ~CT_RCU_WATCHING; - smp_rmb(); // Order ->dynticks and *vp reads. + snap = ct_rcu_watching_cpu(cpu) & ~CT_RCU_WATCHING; + smp_rmb(); // Order CT state and *vp reads. if (READ_ONCE(*vp)) return false; // Non-zero, so report failure; - smp_rmb(); // Order *vp read and ->dynticks re-read. + smp_rmb(); // Order *vp read and CT state re-read. // If still in the same extended quiescent state, we are good! - return snap == ct_dynticks_cpu(cpu); + return snap == ct_rcu_watching_cpu(cpu); } /* @@ -4805,7 +4805,7 @@ rcu_boot_init_percpu_data(int cpu) rdp->grpmask = leaf_node_cpu_bit(rdp->mynode, cpu); INIT_WORK(&rdp->strict_work, strict_work_handler); WARN_ON_ONCE(ct->dynticks_nesting != 1); - WARN_ON_ONCE(rcu_dynticks_in_eqs(ct_dynticks_cpu(cpu))); + WARN_ON_ONCE(rcu_dynticks_in_eqs(ct_rcu_watching_cpu(cpu))); rdp->barrier_seq_snap = rcu_state.barrier_sequence; rdp->rcu_ofl_gp_seq = rcu_state.gp_seq; rdp->rcu_ofl_gp_state = RCU_GP_CLEANED; diff --git a/kernel/rcu/tree_stall.h b/kernel/rcu/tree_stall.h index 4b0e9d7c4c68..d65974448e81 100644 --- a/kernel/rcu/tree_stall.h +++ b/kernel/rcu/tree_stall.h @@ -501,7 +501,7 @@ static void print_cpu_stall_info(int cpu) } delta = rcu_seq_ctr(rdp->mynode->gp_seq - rdp->rcu_iw_gp_seq); falsepositive = rcu_is_gp_kthread_starving(NULL) && - rcu_dynticks_in_eqs(ct_dynticks_cpu(cpu)); + rcu_dynticks_in_eqs(ct_rcu_watching_cpu(cpu)); rcuc_starved = rcu_is_rcuc_kthread_starving(rdp, &j); if (rcuc_starved) // Print signed value, as negative values indicate a probable bug. @@ -515,7 +515,7 @@ static void print_cpu_stall_info(int cpu) rdp->rcu_iw_pending ? (int)min(delta, 9UL) + '0' : "!."[!delta], ticks_value, ticks_title, - ct_dynticks_cpu(cpu) & 0xffff, + ct_rcu_watching_cpu(cpu) & 0xffff, ct_dynticks_nesting_cpu(cpu), ct_dynticks_nmi_nesting_cpu(cpu), rdp->softirq_snap, kstat_softirqs_cpu(RCU_SOFTIRQ, cpu), data_race(rcu_state.n_force_qs) - rcu_state.n_force_qs_gpstart, From patchwork Fri Aug 16 04:02:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neeraj Upadhyay X-Patchwork-Id: 13765398 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1975878B4E; Fri, 16 Aug 2024 04:03:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723781038; cv=none; b=iityYegEcLlWULVaY+FzIWJDp5zfAeBNp2kyrJMd0xeDuGKuU8liXtWyRp10/jhetsqUAotaLq5XmHqwK69o7zSAJLUu09MwPyg/Ssf4XK8GafDUHblhD63d/XAXagbH+Rv3mdQGPfUop3BOs4jgUnVY19QdvtgXpj+SVS8CbE0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723781038; c=relaxed/simple; bh=NyulHrMVLNYTfXy3AWfdScqtYEeVT/u7nVYWdKpf0a0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=A13DKV+ffyasoM73SALNC8dJ273Wyi53vVDQZxUCBLCWfr1kthIBVhOdX6IBxE1B+diH6Tox4Vi0DXY5494Kg6IY31LnUps3EMG86bW72I4ZEBvAP4tC747UiZnDI/koS+ZKYLNoVs33UvRbcR4SQh6bIKSaxqQlJ4sFIyKKVvo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=sbcMF76n; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="sbcMF76n" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 43BB7C32782; Fri, 16 Aug 2024 04:03:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723781038; bh=NyulHrMVLNYTfXy3AWfdScqtYEeVT/u7nVYWdKpf0a0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=sbcMF76n+xKc5K9ln7rXe2TQ0G3TZx21NGvxesc39Dgx1bX1f63g28o0nmpBgawRV eOL70jZgHdons0z47xud22dj4kK7grmUvMuB8XcDKhf9Q7juTNuPhgoBN2iWd6cLkf bi1C7utE2R7x3TaLGWnZzn8rtWXhSLNSdwI+tbpExthrNTW4MQdFx4vIjmWatomA54 GC1bK/7OgQ2XlYhOuMbKpeW/VP8M+Tq4RiE+uG45FFAdkw2o4QmRblSlaRQbcO0poU I+NIgD+ToIuytEDSX8ZK59wJY3dzyUf6mEvcXwuEmNzM6bb5nlVRBEZQMfuHffOfCb QAHcjx5CzeyDw== From: neeraj.upadhyay@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, paulmck@kernel.org, neeraj.upadhyay@kernel.org, neeraj.upadhyay@amd.com, boqun.feng@gmail.com, joel@joelfernandes.org, urezki@gmail.com, frederic@kernel.org, Valentin Schneider Subject: [PATCH rcu 05/26] context_tracking, rcu: Rename ct_dynticks_cpu_acquire() into ct_rcu_watching_cpu_acquire() Date: Fri, 16 Aug 2024 09:32:20 +0530 Message-Id: <20240816040241.17776-5-neeraj.upadhyay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240816040030.GA16986@neeraj.linux> References: <20240816040030.GA16986@neeraj.linux> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Valentin Schneider The context_tracking.state RCU_DYNTICKS subvariable has been renamed to RCU_WATCHING, reflect that change in the related helpers. Signed-off-by: Valentin Schneider Reviewed-by: Frederic Weisbecker Signed-off-by: Neeraj Upadhyay --- .../RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst | 2 +- include/linux/context_tracking_state.h | 2 +- kernel/rcu/tree.c | 4 ++-- kernel/rcu/tree_exp.h | 2 +- 4 files changed, 5 insertions(+), 5 deletions(-) diff --git a/Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst b/Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst index 728b1e690c64..2d7036ad7476 100644 --- a/Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst +++ b/Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst @@ -149,7 +149,7 @@ This case is handled by calls to the strongly ordered ``atomic_add_return()`` read-modify-write atomic operation that is invoked within ``rcu_dynticks_eqs_enter()`` at idle-entry time and within ``rcu_dynticks_eqs_exit()`` at idle-exit time. -The grace-period kthread invokes first ``ct_dynticks_cpu_acquire()`` +The grace-period kthread invokes first ``ct_rcu_watching_cpu_acquire()`` (preceded by a full memory barrier) and ``rcu_dynticks_in_eqs_since()`` (both of which rely on acquire semantics) to detect idle CPUs. diff --git a/include/linux/context_tracking_state.h b/include/linux/context_tracking_state.h index ad5a06a42b4a..ad6570ffeff3 100644 --- a/include/linux/context_tracking_state.h +++ b/include/linux/context_tracking_state.h @@ -68,7 +68,7 @@ static __always_inline int ct_rcu_watching_cpu(int cpu) return atomic_read(&ct->state) & CT_RCU_WATCHING_MASK; } -static __always_inline int ct_dynticks_cpu_acquire(int cpu) +static __always_inline int ct_rcu_watching_cpu_acquire(int cpu) { struct context_tracking *ct = per_cpu_ptr(&context_tracking, cpu); diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index e7f612e9f7e5..45a9f3667c2a 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -323,7 +323,7 @@ static bool rcu_dynticks_in_eqs_since(struct rcu_data *rdp, int snap) * performed by the remote CPU prior to entering idle and therefore can * rely solely on acquire semantics. */ - return snap != ct_dynticks_cpu_acquire(rdp->cpu); + return snap != ct_rcu_watching_cpu_acquire(rdp->cpu); } /* @@ -782,7 +782,7 @@ static int dyntick_save_progress_counter(struct rcu_data *rdp) * Ordering between remote CPU's pre idle accesses and post grace period * updater's accesses is enforced by the below acquire semantic. */ - rdp->dynticks_snap = ct_dynticks_cpu_acquire(rdp->cpu); + rdp->dynticks_snap = ct_rcu_watching_cpu_acquire(rdp->cpu); if (rcu_dynticks_in_eqs(rdp->dynticks_snap)) { trace_rcu_fqs(rcu_state.name, rdp->gp_seq, rdp->cpu, TPS("dti")); rcu_gpnum_ovf(rdp->mynode, rdp); diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h index 4acd29d16fdb..daa87fec703f 100644 --- a/kernel/rcu/tree_exp.h +++ b/kernel/rcu/tree_exp.h @@ -376,7 +376,7 @@ static void __sync_rcu_exp_select_node_cpus(struct rcu_exp_work *rewp) * post grace period updater's accesses is enforced by the * below acquire semantic. */ - snap = ct_dynticks_cpu_acquire(cpu); + snap = ct_rcu_watching_cpu_acquire(cpu); if (rcu_dynticks_in_eqs(snap)) mask_ofl_test |= mask; else From patchwork Fri Aug 16 04:02:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neeraj Upadhyay X-Patchwork-Id: 13765399 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 296487DA64; Fri, 16 Aug 2024 04:04:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723781047; cv=none; b=bGAnCoZAxhGj5rrmVw5DKvGCOlb5hQDE9Yute0rGP/a8S+MOyJ1F69k7oF+zIvEUUY/2mWTR7LmHhEqRtSEhokDVUSTKcsU16EJy5ePVDl83wdP+eWHbOlVDNCWvD0Kvg9v9T4I50b0V5MvxE9+YJF0gu8QcM5ai7uwzJVKvdj4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723781047; c=relaxed/simple; bh=EJMjMl2JpJHeHT8KAgtkO4gbK13CfBULyS/uGa67NYU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=iWAv9a4OwrV5ZO/WwyIMBUtDoB3G/674nt0tDRPvu/6YKNLBaGb7Bh/jjFApnID2yMvZd3pQFV2izx3yzOtcSugdTz7TxWuWWFnR6eaU+LWCfOmE9O7mVZQj1e5N89pwBosgzwx6PgMfiMrZKGfi6oGMCpsBIzJfM0m0G8Qqtkc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=MLuE+A8e; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="MLuE+A8e" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 60D22C4AF09; Fri, 16 Aug 2024 04:03:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723781046; bh=EJMjMl2JpJHeHT8KAgtkO4gbK13CfBULyS/uGa67NYU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=MLuE+A8eJoXqHy79K1PA3LnVNoHw+elr5ZISjKMs0nnOh9AusIITfSERECRXqmMtu 7h18ib6QrvFyr16lDRncuJ7cyliv8MxmZecdcjjG8tD985ZVZH59uLUiZMfkaLYuXZ s94vy3TaEASy6zfdqZFFO9jSWt8U01fRF1p47ZSPLEt11KeeyLk9DKwXUb8FjHihMa 5xP6tRHJoI24dUIFEqr/kBby29ndoH8BTdQeufgUR5ZeXMfE0K9PsV7ZaGMLsxbh1I lQwMO/9jxJtNLnJbLfDMXF0EdCIBUv0WXFUFWEBRa0UPOnfDWFTFSt4KL4E7iguiPa dL4m6wgxsJKMg== From: neeraj.upadhyay@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, paulmck@kernel.org, neeraj.upadhyay@kernel.org, neeraj.upadhyay@amd.com, boqun.feng@gmail.com, joel@joelfernandes.org, urezki@gmail.com, frederic@kernel.org, Valentin Schneider Subject: [PATCH rcu 06/26] context_tracking, rcu: Rename struct context_tracking .dynticks_nesting into .nesting Date: Fri, 16 Aug 2024 09:32:21 +0530 Message-Id: <20240816040241.17776-6-neeraj.upadhyay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240816040030.GA16986@neeraj.linux> References: <20240816040030.GA16986@neeraj.linux> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Valentin Schneider The context_tracking.state RCU_DYNTICKS subvariable has been renamed to RCU_WATCHING, reflect that change in the related helpers. [ neeraj.upadhyay: Fix htmldocs build error reported by Stephen Rothwell ] Suggested-by: Frederic Weisbecker Signed-off-by: Valentin Schneider Reviewed-by: Frederic Weisbecker Signed-off-by: Neeraj Upadhyay --- .../RCU/Design/Data-Structures/Data-Structures.rst | 10 +++++----- include/linux/context_tracking_state.h | 6 +++--- include/trace/events/rcu.h | 2 +- kernel/context_tracking.c | 10 +++++----- kernel/rcu/tree.c | 8 ++++---- 5 files changed, 18 insertions(+), 18 deletions(-) diff --git a/Documentation/RCU/Design/Data-Structures/Data-Structures.rst b/Documentation/RCU/Design/Data-Structures/Data-Structures.rst index b34990c7c377..69860bbec202 100644 --- a/Documentation/RCU/Design/Data-Structures/Data-Structures.rst +++ b/Documentation/RCU/Design/Data-Structures/Data-Structures.rst @@ -935,7 +935,7 @@ This portion of the rcu_data structure is declared as follows: :: - 1 long dynticks_nesting; + 1 long nesting; 2 long dynticks_nmi_nesting; 3 atomic_t dynticks; 4 bool rcu_need_heavy_qs; @@ -945,7 +945,7 @@ These fields in the rcu_data structure maintain the per-CPU dyntick-idle state for the corresponding CPU. The fields may be accessed only from the corresponding CPU (and from tracing) unless otherwise stated. -The ``->dynticks_nesting`` field counts the nesting depth of process +The ``->nesting`` field counts the nesting depth of process execution, so that in normal circumstances this counter has value zero or one. NMIs, irqs, and tracers are counted by the ``->dynticks_nmi_nesting`` field. Because NMIs cannot be masked, changes @@ -960,9 +960,9 @@ process-level transitions. However, it turns out that when running in non-idle kernel context, the Linux kernel is fully capable of entering interrupt handlers that never exit and perhaps also vice versa. Therefore, whenever the -``->dynticks_nesting`` field is incremented up from zero, the +``->nesting`` field is incremented up from zero, the ``->dynticks_nmi_nesting`` field is set to a large positive number, and -whenever the ``->dynticks_nesting`` field is decremented down to zero, +whenever the ``->nesting`` field is decremented down to zero, the ``->dynticks_nmi_nesting`` field is set to zero. Assuming that the number of misnested interrupts is not sufficient to overflow the counter, this approach corrects the ``->dynticks_nmi_nesting`` field @@ -992,7 +992,7 @@ code. +-----------------------------------------------------------------------+ | **Quick Quiz**: | +-----------------------------------------------------------------------+ -| Why not simply combine the ``->dynticks_nesting`` and | +| Why not simply combine the ``->nesting`` and | | ``->dynticks_nmi_nesting`` counters into a single counter that just | | counts the number of reasons that the corresponding CPU is non-idle? | +-----------------------------------------------------------------------+ diff --git a/include/linux/context_tracking_state.h b/include/linux/context_tracking_state.h index ad6570ffeff3..65290e7677e6 100644 --- a/include/linux/context_tracking_state.h +++ b/include/linux/context_tracking_state.h @@ -39,7 +39,7 @@ struct context_tracking { atomic_t state; #endif #ifdef CONFIG_CONTEXT_TRACKING_IDLE - long dynticks_nesting; /* Track process nesting level. */ + long nesting; /* Track process nesting level. */ long dynticks_nmi_nesting; /* Track irq/NMI nesting level. */ #endif }; @@ -77,14 +77,14 @@ static __always_inline int ct_rcu_watching_cpu_acquire(int cpu) static __always_inline long ct_dynticks_nesting(void) { - return __this_cpu_read(context_tracking.dynticks_nesting); + return __this_cpu_read(context_tracking.nesting); } static __always_inline long ct_dynticks_nesting_cpu(int cpu) { struct context_tracking *ct = per_cpu_ptr(&context_tracking, cpu); - return ct->dynticks_nesting; + return ct->nesting; } static __always_inline long ct_dynticks_nmi_nesting(void) diff --git a/include/trace/events/rcu.h b/include/trace/events/rcu.h index 31b3e0d3e65f..4066b6d51e46 100644 --- a/include/trace/events/rcu.h +++ b/include/trace/events/rcu.h @@ -469,7 +469,7 @@ TRACE_EVENT(rcu_stall_warning, * polarity: "Start", "End", "StillNonIdle" for entering, exiting or still not * being in dyntick-idle mode. * context: "USER" or "IDLE" or "IRQ". - * NMIs nested in IRQs are inferred with dynticks_nesting > 1 in IRQ context. + * NMIs nested in IRQs are inferred with nesting > 1 in IRQ context. * * These events also take a pair of numbers, which indicate the nesting * depth before and after the event of interest, and a third number that is diff --git a/kernel/context_tracking.c b/kernel/context_tracking.c index 868ae0bcd4be..5cfdfc03b401 100644 --- a/kernel/context_tracking.c +++ b/kernel/context_tracking.c @@ -28,7 +28,7 @@ DEFINE_PER_CPU(struct context_tracking, context_tracking) = { #ifdef CONFIG_CONTEXT_TRACKING_IDLE - .dynticks_nesting = 1, + .nesting = 1, .dynticks_nmi_nesting = DYNTICK_IRQ_NONIDLE, #endif .state = ATOMIC_INIT(CT_RCU_WATCHING), @@ -131,7 +131,7 @@ static void noinstr ct_kernel_exit(bool user, int offset) ct_dynticks_nesting() == 0); if (ct_dynticks_nesting() != 1) { // RCU will still be watching, so just do accounting and leave. - ct->dynticks_nesting--; + ct->nesting--; return; } @@ -145,7 +145,7 @@ static void noinstr ct_kernel_exit(bool user, int offset) instrument_atomic_write(&ct->state, sizeof(ct->state)); instrumentation_end(); - WRITE_ONCE(ct->dynticks_nesting, 0); /* Avoid irq-access tearing. */ + WRITE_ONCE(ct->nesting, 0); /* Avoid irq-access tearing. */ // RCU is watching here ... ct_kernel_exit_state(offset); // ... but is no longer watching here. @@ -170,7 +170,7 @@ static void noinstr ct_kernel_enter(bool user, int offset) WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && oldval < 0); if (oldval) { // RCU was already watching, so just do accounting and leave. - ct->dynticks_nesting++; + ct->nesting++; return; } rcu_dynticks_task_exit(); @@ -184,7 +184,7 @@ static void noinstr ct_kernel_enter(bool user, int offset) trace_rcu_dyntick(TPS("End"), ct_dynticks_nesting(), 1, ct_rcu_watching()); WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !user && !is_idle_task(current)); - WRITE_ONCE(ct->dynticks_nesting, 1); + WRITE_ONCE(ct->nesting, 1); WARN_ON_ONCE(ct_dynticks_nmi_nesting()); WRITE_ONCE(ct->dynticks_nmi_nesting, DYNTICK_IRQ_NONIDLE); instrumentation_end(); diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 45a9f3667c2a..0d5a539acd58 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -389,7 +389,7 @@ static int rcu_is_cpu_rrupt_from_idle(void) /* Check for counter underflows */ RCU_LOCKDEP_WARN(ct_dynticks_nesting() < 0, - "RCU dynticks_nesting counter underflow!"); + "RCU nesting counter underflow!"); RCU_LOCKDEP_WARN(ct_dynticks_nmi_nesting() <= 0, "RCU dynticks_nmi_nesting counter underflow/zero!"); @@ -597,7 +597,7 @@ void rcu_irq_exit_check_preempt(void) lockdep_assert_irqs_disabled(); RCU_LOCKDEP_WARN(ct_dynticks_nesting() <= 0, - "RCU dynticks_nesting counter underflow/zero!"); + "RCU nesting counter underflow/zero!"); RCU_LOCKDEP_WARN(ct_dynticks_nmi_nesting() != DYNTICK_IRQ_NONIDLE, "Bad RCU dynticks_nmi_nesting counter\n"); @@ -4804,7 +4804,7 @@ rcu_boot_init_percpu_data(int cpu) /* Set up local state, ensuring consistent view of global state. */ rdp->grpmask = leaf_node_cpu_bit(rdp->mynode, cpu); INIT_WORK(&rdp->strict_work, strict_work_handler); - WARN_ON_ONCE(ct->dynticks_nesting != 1); + WARN_ON_ONCE(ct->nesting != 1); WARN_ON_ONCE(rcu_dynticks_in_eqs(ct_rcu_watching_cpu(cpu))); rdp->barrier_seq_snap = rcu_state.barrier_sequence; rdp->rcu_ofl_gp_seq = rcu_state.gp_seq; @@ -4898,7 +4898,7 @@ int rcutree_prepare_cpu(unsigned int cpu) rdp->qlen_last_fqs_check = 0; rdp->n_force_qs_snap = READ_ONCE(rcu_state.n_force_qs); rdp->blimit = blimit; - ct->dynticks_nesting = 1; /* CPU not up, no tearing. */ + ct->nesting = 1; /* CPU not up, no tearing. */ raw_spin_unlock_rcu_node(rnp); /* irqs remain disabled. */ /* From patchwork Fri Aug 16 04:02:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neeraj Upadhyay X-Patchwork-Id: 13765400 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E86D46F077; Fri, 16 Aug 2024 04:04:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723781057; cv=none; b=cfDMWKghfKrhT9lTfZEd9DscTqopvQNPwhaJw+O861q2BEmyu1jRkRQEubtkkInG68k4zoOT93cnKJkFJWnmyxDorQ6Tq59PAF1Ng8ogtX4P2ENtaaLVq+Tr4yy8SdxKxo06vOj5ARINjSIO3eeD3E+/vDVx+B9lCzPfOiJRc+Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723781057; c=relaxed/simple; bh=ekhSu6/CJ43f9nrJP4ALZW5Tq/0idh0P3w/90Gcl9f8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=glChQKfVsQKIPiUE+/kfhadRJSowYiT7PpnihzmuJLlq0mnqfK5qYWnHy0fAnSdxXQHmNK59ZLhqveyqhxLI8sdOi511XrsZ0jiD/R4LAzV2DWltX8YwcWqjN/LKzCsOaysvvN6Z/KZWBQZV2iSW4znYoc6l4B1qve+khfrr1C0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Axi+u7g8; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Axi+u7g8" Received: by smtp.kernel.org (Postfix) with ESMTPSA id D96FAC4AF09; Fri, 16 Aug 2024 04:04:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723781056; bh=ekhSu6/CJ43f9nrJP4ALZW5Tq/0idh0P3w/90Gcl9f8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Axi+u7g8OfRbktStpUAF6wDcyUNNzeDzDfTV6eg0QspgcFFReXqKwkK8A+fYSPx5B hF0FrAmcprfemgFvltXwRqBV8PVlij+zE82AFl2kMz0u/N5rLxhhhmxr9P6uLEcIZL q81ClEXqINZXfznpS68UBdI3IL9+pXvqhq1P5DC34kf/YEW5dqa/meHrT4c9pBpGNX MK9L/GEqeRi19iolVuqkUVgnFa3qUBuD/FGJ1YOx8pK6sBNqQx7rfRjtHMXIYluHWL 50RjrAWKU5iTTJOOJT+jcw21P9+Ydk/A35KhJ3fSZnW39S3AGvt4I3LsIjz4Am7lQc PTbh+hN2g67MQ== From: neeraj.upadhyay@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, paulmck@kernel.org, neeraj.upadhyay@kernel.org, neeraj.upadhyay@amd.com, boqun.feng@gmail.com, joel@joelfernandes.org, urezki@gmail.com, frederic@kernel.org, Valentin Schneider Subject: [PATCH rcu 07/26] context_tracking, rcu: Rename ct_dynticks_nesting() into ct_nesting() Date: Fri, 16 Aug 2024 09:32:22 +0530 Message-Id: <20240816040241.17776-7-neeraj.upadhyay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240816040030.GA16986@neeraj.linux> References: <20240816040030.GA16986@neeraj.linux> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Valentin Schneider The context_tracking.state RCU_DYNTICKS subvariable has been renamed to RCU_WATCHING, reflect that change in the related helpers. Suggested-by: Frederic Weisbecker Signed-off-by: Valentin Schneider Reviewed-by: Frederic Weisbecker Signed-off-by: Neeraj Upadhyay --- include/linux/context_tracking_state.h | 2 +- kernel/context_tracking.c | 10 +++++----- kernel/rcu/tree.c | 6 +++--- 3 files changed, 9 insertions(+), 9 deletions(-) diff --git a/include/linux/context_tracking_state.h b/include/linux/context_tracking_state.h index 65290e7677e6..586c1ff22c2e 100644 --- a/include/linux/context_tracking_state.h +++ b/include/linux/context_tracking_state.h @@ -75,7 +75,7 @@ static __always_inline int ct_rcu_watching_cpu_acquire(int cpu) return atomic_read_acquire(&ct->state) & CT_RCU_WATCHING_MASK; } -static __always_inline long ct_dynticks_nesting(void) +static __always_inline long ct_nesting(void) { return __this_cpu_read(context_tracking.nesting); } diff --git a/kernel/context_tracking.c b/kernel/context_tracking.c index 5cfdfc03b401..a951bde0bbcb 100644 --- a/kernel/context_tracking.c +++ b/kernel/context_tracking.c @@ -128,8 +128,8 @@ static void noinstr ct_kernel_exit(bool user, int offset) WARN_ON_ONCE(ct_dynticks_nmi_nesting() != DYNTICK_IRQ_NONIDLE); WRITE_ONCE(ct->dynticks_nmi_nesting, 0); WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && - ct_dynticks_nesting() == 0); - if (ct_dynticks_nesting() != 1) { + ct_nesting() == 0); + if (ct_nesting() != 1) { // RCU will still be watching, so just do accounting and leave. ct->nesting--; return; @@ -137,7 +137,7 @@ static void noinstr ct_kernel_exit(bool user, int offset) instrumentation_begin(); lockdep_assert_irqs_disabled(); - trace_rcu_dyntick(TPS("Start"), ct_dynticks_nesting(), 0, ct_rcu_watching()); + trace_rcu_dyntick(TPS("Start"), ct_nesting(), 0, ct_rcu_watching()); WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !user && !is_idle_task(current)); rcu_preempt_deferred_qs(current); @@ -166,7 +166,7 @@ static void noinstr ct_kernel_enter(bool user, int offset) long oldval; WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !raw_irqs_disabled()); - oldval = ct_dynticks_nesting(); + oldval = ct_nesting(); WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && oldval < 0); if (oldval) { // RCU was already watching, so just do accounting and leave. @@ -182,7 +182,7 @@ static void noinstr ct_kernel_enter(bool user, int offset) // instrumentation for the noinstr ct_kernel_enter_state() instrument_atomic_write(&ct->state, sizeof(ct->state)); - trace_rcu_dyntick(TPS("End"), ct_dynticks_nesting(), 1, ct_rcu_watching()); + trace_rcu_dyntick(TPS("End"), ct_nesting(), 1, ct_rcu_watching()); WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !user && !is_idle_task(current)); WRITE_ONCE(ct->nesting, 1); WARN_ON_ONCE(ct_dynticks_nmi_nesting()); diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 0d5a539acd58..cf69a234080f 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -388,7 +388,7 @@ static int rcu_is_cpu_rrupt_from_idle(void) lockdep_assert_irqs_disabled(); /* Check for counter underflows */ - RCU_LOCKDEP_WARN(ct_dynticks_nesting() < 0, + RCU_LOCKDEP_WARN(ct_nesting() < 0, "RCU nesting counter underflow!"); RCU_LOCKDEP_WARN(ct_dynticks_nmi_nesting() <= 0, "RCU dynticks_nmi_nesting counter underflow/zero!"); @@ -404,7 +404,7 @@ static int rcu_is_cpu_rrupt_from_idle(void) WARN_ON_ONCE(!nesting && !is_idle_task(current)); /* Does CPU appear to be idle from an RCU standpoint? */ - return ct_dynticks_nesting() == 0; + return ct_nesting() == 0; } #define DEFAULT_RCU_BLIMIT (IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD) ? 1000 : 10) @@ -596,7 +596,7 @@ void rcu_irq_exit_check_preempt(void) { lockdep_assert_irqs_disabled(); - RCU_LOCKDEP_WARN(ct_dynticks_nesting() <= 0, + RCU_LOCKDEP_WARN(ct_nesting() <= 0, "RCU nesting counter underflow/zero!"); RCU_LOCKDEP_WARN(ct_dynticks_nmi_nesting() != DYNTICK_IRQ_NONIDLE, From patchwork Fri Aug 16 04:02:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neeraj Upadhyay X-Patchwork-Id: 13765401 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D3DB978B4C; Fri, 16 Aug 2024 04:04:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723781064; cv=none; b=LYac1xyVtPkjBUdsq9o4MfF9U5CZKo/WOKi/kpmxOd1vPSZTflH/tP2fSkzsRQuXkivQCSEMs0Fmq+i2QnOyDN1BSvZDi8YZVZhvdd38yfEhHj78EVHimSN+lm4iFDvGH1c2uRVKkesqPMLa+8cf+HCk2Z5UjKtN/7l11S6sL2U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723781064; c=relaxed/simple; bh=ft6U4z6A+mqbAhVQaGYeAhHApoONNXfw6evxiIkMcp4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=LyhCiiP13vP7dqeioiNhu5zONtxOxdqPwpKI/Rg7ZN5/1Z+x33nS1z/oTSm/LCRiMeCU279L7HAK0526m3idOfe9Jy+gVRl0sFx8SCseYQjbzHtiIJ0uTBGc8R3QLMhS1ADa3N3MZO7gAKD6ToH/gUZyNnyqTvx0u9zCSfAXQyU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=kvw9TnEO; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="kvw9TnEO" Received: by smtp.kernel.org (Postfix) with ESMTPSA id BFBA0C4AF0B; Fri, 16 Aug 2024 04:04:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723781064; bh=ft6U4z6A+mqbAhVQaGYeAhHApoONNXfw6evxiIkMcp4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=kvw9TnEOTpzob4x8jpuEwk8hhEXeNFd0RDpPM/9wdZ92gmoL4TMzZkiKz9Di/IVca pQEbv/g+Q6b//912E9rCP4JVeBotjKzyZcUs/9T7TTajskczsroGDo2x2ar6iqcUtv 9kL1Piak0pqza1/ZXWTF+GJJwiBkzs+ScG0xqwayhrEYm67JVgLgkj/D6VMQwfzYWp HhQaKvyh+CDzz/x/ySl4eOQgcWv5zUtKwjzxj9QdZMfd4cUsoDIThBODlkzSxlJQY6 ide4sBkfRkxzD9Jztz4+gatikxxLJAH9x0ki/8WV9aet2GGemMtNG6165480H9B1FN ZPTrAqvI8TaQw== From: neeraj.upadhyay@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, paulmck@kernel.org, neeraj.upadhyay@kernel.org, neeraj.upadhyay@amd.com, boqun.feng@gmail.com, joel@joelfernandes.org, urezki@gmail.com, frederic@kernel.org, Valentin Schneider Subject: [PATCH rcu 08/26] context_tracking, rcu: Rename ct_dynticks_nesting_cpu() into ct_nesting_cpu() Date: Fri, 16 Aug 2024 09:32:23 +0530 Message-Id: <20240816040241.17776-8-neeraj.upadhyay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240816040030.GA16986@neeraj.linux> References: <20240816040030.GA16986@neeraj.linux> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Valentin Schneider The context_tracking.state RCU_DYNTICKS subvariable has been renamed to RCU_WATCHING, and the 'dynticks' prefix can be dropped without losing any meaning. Suggested-by: Frederic Weisbecker Signed-off-by: Valentin Schneider Reviewed-by: Frederic Weisbecker Signed-off-by: Neeraj Upadhyay --- include/linux/context_tracking_state.h | 2 +- kernel/rcu/tree_stall.h | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/include/linux/context_tracking_state.h b/include/linux/context_tracking_state.h index 586c1ff22c2e..fd42d8120ac2 100644 --- a/include/linux/context_tracking_state.h +++ b/include/linux/context_tracking_state.h @@ -80,7 +80,7 @@ static __always_inline long ct_nesting(void) return __this_cpu_read(context_tracking.nesting); } -static __always_inline long ct_dynticks_nesting_cpu(int cpu) +static __always_inline long ct_nesting_cpu(int cpu) { struct context_tracking *ct = per_cpu_ptr(&context_tracking, cpu); diff --git a/kernel/rcu/tree_stall.h b/kernel/rcu/tree_stall.h index d65974448e81..59b1d84a4749 100644 --- a/kernel/rcu/tree_stall.h +++ b/kernel/rcu/tree_stall.h @@ -516,7 +516,7 @@ static void print_cpu_stall_info(int cpu) "!."[!delta], ticks_value, ticks_title, ct_rcu_watching_cpu(cpu) & 0xffff, - ct_dynticks_nesting_cpu(cpu), ct_dynticks_nmi_nesting_cpu(cpu), + ct_nesting_cpu(cpu), ct_dynticks_nmi_nesting_cpu(cpu), rdp->softirq_snap, kstat_softirqs_cpu(RCU_SOFTIRQ, cpu), data_race(rcu_state.n_force_qs) - rcu_state.n_force_qs_gpstart, rcuc_starved ? buf : "", From patchwork Fri Aug 16 04:02:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neeraj Upadhyay X-Patchwork-Id: 13765402 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 06D6678B60; Fri, 16 Aug 2024 04:04:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723781072; cv=none; b=ahGnXm42+GCHePDNkHn07OeQ+/a5y/l/wRB6IsBSCmQ9SOJUdCPyp/O5RZKqV6x+IlQVph/i/I1830hpd/LqudwDv3eQ6BDOQF0xezJQObCbzbdoiaRbPLxiakUo2doxUh0XOAU3ybVXt9jJLHZSlhmDTFta2B6BcQOcGKroeZo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723781072; c=relaxed/simple; bh=WKrFlGc311b417Ae3nBtrZ9FiooyNB5o0plOQxiWj2A=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=RfF7WT28ubTqwohxrosgK5sBsFozLewiw4f0GqfUEDaCwmESgyPGxuxJwEDUnLbG6KKbxXIU34CA6yncOFWlwkttcLSdYjI3exiMBv30qwDANiJ4g0A1lqGzCvi1W9QjPKC4uvkj+czBHN2/a8Wpz0WEbHQTJBtmRyhCONb1nTs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Z4fuhgTD; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Z4fuhgTD" Received: by smtp.kernel.org (Postfix) with ESMTPSA id D4CABC4AF0B; Fri, 16 Aug 2024 04:04:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723781071; bh=WKrFlGc311b417Ae3nBtrZ9FiooyNB5o0plOQxiWj2A=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Z4fuhgTDHN+mcCtJhvJaeMAEXWSp8pDBuR7x2kaaFP+6OfRrJD5+8wiR0DRopMZuT J6o/l9p/+4Qq3B7qSW2bWTd5ZHtONLrOvEpZHJLrbS1aYkTOpO8OthLC8/PsdJTvgm 2hWnu/2sJQwRSiqK7PTfF8PdB7JMyhuV8wrBck/STeS7ZrgZQQ7AtLv+zlRI6cIhMJ SZURHVHE1TGwJMtL9CSuW+p7jaRsEyXrgijdh4mPPuNYP/MVc/0gmZmk/qfSU1cYAT k8LT8BUFT1FcKawfBXCxbXc7x9iyx3pY11JwQ19gq5RcXFE1mB2PwlexU393JMrpxD MyP1EVJeIdQ5Q== From: neeraj.upadhyay@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, paulmck@kernel.org, neeraj.upadhyay@kernel.org, neeraj.upadhyay@amd.com, boqun.feng@gmail.com, joel@joelfernandes.org, urezki@gmail.com, frederic@kernel.org, Valentin Schneider Subject: [PATCH rcu 09/26] context_tracking, rcu: Rename struct context_tracking .dynticks_nmi_nesting into .nmi_nesting Date: Fri, 16 Aug 2024 09:32:24 +0530 Message-Id: <20240816040241.17776-9-neeraj.upadhyay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240816040030.GA16986@neeraj.linux> References: <20240816040030.GA16986@neeraj.linux> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Valentin Schneider The context_tracking.state RCU_DYNTICKS subvariable has been renamed to RCU_WATCHING, and the 'dynticks' prefix can be dropped without losing any meaning. [ neeraj.upadhyay: Fix htmldocs build error reported by Stephen Rothwell ] Suggested-by: Frederic Weisbecker Signed-off-by: Valentin Schneider Reviewed-by: Frederic Weisbecker Signed-off-by: Neeraj Upadhyay --- .../Data-Structures/Data-Structures.rst | 14 ++++----- include/linux/context_tracking_state.h | 6 ++-- kernel/context_tracking.c | 30 +++++++++---------- kernel/rcu/tree.c | 4 +-- 4 files changed, 27 insertions(+), 27 deletions(-) diff --git a/Documentation/RCU/Design/Data-Structures/Data-Structures.rst b/Documentation/RCU/Design/Data-Structures/Data-Structures.rst index 69860bbec202..28d9f9e1783f 100644 --- a/Documentation/RCU/Design/Data-Structures/Data-Structures.rst +++ b/Documentation/RCU/Design/Data-Structures/Data-Structures.rst @@ -936,7 +936,7 @@ This portion of the rcu_data structure is declared as follows: :: 1 long nesting; - 2 long dynticks_nmi_nesting; + 2 long nmi_nesting; 3 atomic_t dynticks; 4 bool rcu_need_heavy_qs; 5 bool rcu_urgent_qs; @@ -948,11 +948,11 @@ the corresponding CPU (and from tracing) unless otherwise stated. The ``->nesting`` field counts the nesting depth of process execution, so that in normal circumstances this counter has value zero or one. NMIs, irqs, and tracers are counted by the -``->dynticks_nmi_nesting`` field. Because NMIs cannot be masked, changes +``->nmi_nesting`` field. Because NMIs cannot be masked, changes to this variable have to be undertaken carefully using an algorithm provided by Andy Lutomirski. The initial transition from idle adds one, and nested transitions add two, so that a nesting level of five is -represented by a ``->dynticks_nmi_nesting`` value of nine. This counter +represented by a ``->nmi_nesting`` value of nine. This counter can therefore be thought of as counting the number of reasons why this CPU cannot be permitted to enter dyntick-idle mode, aside from process-level transitions. @@ -961,11 +961,11 @@ However, it turns out that when running in non-idle kernel context, the Linux kernel is fully capable of entering interrupt handlers that never exit and perhaps also vice versa. Therefore, whenever the ``->nesting`` field is incremented up from zero, the -``->dynticks_nmi_nesting`` field is set to a large positive number, and +``->nmi_nesting`` field is set to a large positive number, and whenever the ``->nesting`` field is decremented down to zero, -the ``->dynticks_nmi_nesting`` field is set to zero. Assuming that +the ``->nmi_nesting`` field is set to zero. Assuming that the number of misnested interrupts is not sufficient to overflow the -counter, this approach corrects the ``->dynticks_nmi_nesting`` field +counter, this approach corrects the ``->nmi_nesting`` field every time the corresponding CPU enters the idle loop from process context. @@ -993,7 +993,7 @@ code. | **Quick Quiz**: | +-----------------------------------------------------------------------+ | Why not simply combine the ``->nesting`` and | -| ``->dynticks_nmi_nesting`` counters into a single counter that just | +| ``->nmi_nesting`` counters into a single counter that just | | counts the number of reasons that the corresponding CPU is non-idle? | +-----------------------------------------------------------------------+ | **Answer**: | diff --git a/include/linux/context_tracking_state.h b/include/linux/context_tracking_state.h index fd42d8120ac2..12d00adf29e1 100644 --- a/include/linux/context_tracking_state.h +++ b/include/linux/context_tracking_state.h @@ -40,7 +40,7 @@ struct context_tracking { #endif #ifdef CONFIG_CONTEXT_TRACKING_IDLE long nesting; /* Track process nesting level. */ - long dynticks_nmi_nesting; /* Track irq/NMI nesting level. */ + long nmi_nesting; /* Track irq/NMI nesting level. */ #endif }; @@ -89,14 +89,14 @@ static __always_inline long ct_nesting_cpu(int cpu) static __always_inline long ct_dynticks_nmi_nesting(void) { - return __this_cpu_read(context_tracking.dynticks_nmi_nesting); + return __this_cpu_read(context_tracking.nmi_nesting); } static __always_inline long ct_dynticks_nmi_nesting_cpu(int cpu) { struct context_tracking *ct = per_cpu_ptr(&context_tracking, cpu); - return ct->dynticks_nmi_nesting; + return ct->nmi_nesting; } #endif /* #ifdef CONFIG_CONTEXT_TRACKING_IDLE */ diff --git a/kernel/context_tracking.c b/kernel/context_tracking.c index a951bde0bbcb..ae94215aa132 100644 --- a/kernel/context_tracking.c +++ b/kernel/context_tracking.c @@ -29,7 +29,7 @@ DEFINE_PER_CPU(struct context_tracking, context_tracking) = { #ifdef CONFIG_CONTEXT_TRACKING_IDLE .nesting = 1, - .dynticks_nmi_nesting = DYNTICK_IRQ_NONIDLE, + .nmi_nesting = DYNTICK_IRQ_NONIDLE, #endif .state = ATOMIC_INIT(CT_RCU_WATCHING), }; @@ -117,7 +117,7 @@ static noinstr void ct_kernel_enter_state(int offset) * Enter an RCU extended quiescent state, which can be either the * idle loop or adaptive-tickless usermode execution. * - * We crowbar the ->dynticks_nmi_nesting field to zero to allow for + * We crowbar the ->nmi_nesting field to zero to allow for * the possibility of usermode upcalls having messed up our count * of interrupt nesting level during the prior busy period. */ @@ -126,7 +126,7 @@ static void noinstr ct_kernel_exit(bool user, int offset) struct context_tracking *ct = this_cpu_ptr(&context_tracking); WARN_ON_ONCE(ct_dynticks_nmi_nesting() != DYNTICK_IRQ_NONIDLE); - WRITE_ONCE(ct->dynticks_nmi_nesting, 0); + WRITE_ONCE(ct->nmi_nesting, 0); WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && ct_nesting() == 0); if (ct_nesting() != 1) { @@ -156,7 +156,7 @@ static void noinstr ct_kernel_exit(bool user, int offset) * Exit an RCU extended quiescent state, which can be either the * idle loop or adaptive-tickless usermode execution. * - * We crowbar the ->dynticks_nmi_nesting field to DYNTICK_IRQ_NONIDLE to + * We crowbar the ->nmi_nesting field to DYNTICK_IRQ_NONIDLE to * allow for the possibility of usermode upcalls messing up our count of * interrupt nesting level during the busy period that is just now starting. */ @@ -186,7 +186,7 @@ static void noinstr ct_kernel_enter(bool user, int offset) WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !user && !is_idle_task(current)); WRITE_ONCE(ct->nesting, 1); WARN_ON_ONCE(ct_dynticks_nmi_nesting()); - WRITE_ONCE(ct->dynticks_nmi_nesting, DYNTICK_IRQ_NONIDLE); + WRITE_ONCE(ct->nmi_nesting, DYNTICK_IRQ_NONIDLE); instrumentation_end(); } @@ -194,7 +194,7 @@ static void noinstr ct_kernel_enter(bool user, int offset) * ct_nmi_exit - inform RCU of exit from NMI context * * If we are returning from the outermost NMI handler that interrupted an - * RCU-idle period, update ct->state and ct->dynticks_nmi_nesting + * RCU-idle period, update ct->state and ct->nmi_nesting * to let the RCU grace-period handling know that the CPU is back to * being RCU-idle. * @@ -207,7 +207,7 @@ void noinstr ct_nmi_exit(void) instrumentation_begin(); /* - * Check for ->dynticks_nmi_nesting underflow and bad ->dynticks. + * Check for ->nmi_nesting underflow and bad ->dynticks. * (We are exiting an NMI handler, so RCU better be paying attention * to us!) */ @@ -221,7 +221,7 @@ void noinstr ct_nmi_exit(void) if (ct_dynticks_nmi_nesting() != 1) { trace_rcu_dyntick(TPS("--="), ct_dynticks_nmi_nesting(), ct_dynticks_nmi_nesting() - 2, ct_rcu_watching()); - WRITE_ONCE(ct->dynticks_nmi_nesting, /* No store tearing. */ + WRITE_ONCE(ct->nmi_nesting, /* No store tearing. */ ct_dynticks_nmi_nesting() - 2); instrumentation_end(); return; @@ -229,7 +229,7 @@ void noinstr ct_nmi_exit(void) /* This NMI interrupted an RCU-idle CPU, restore RCU-idleness. */ trace_rcu_dyntick(TPS("Startirq"), ct_dynticks_nmi_nesting(), 0, ct_rcu_watching()); - WRITE_ONCE(ct->dynticks_nmi_nesting, 0); /* Avoid store tearing. */ + WRITE_ONCE(ct->nmi_nesting, 0); /* Avoid store tearing. */ // instrumentation for the noinstr ct_kernel_exit_state() instrument_atomic_write(&ct->state, sizeof(ct->state)); @@ -247,7 +247,7 @@ void noinstr ct_nmi_exit(void) * ct_nmi_enter - inform RCU of entry to NMI context * * If the CPU was idle from RCU's viewpoint, update ct->state and - * ct->dynticks_nmi_nesting to let the RCU grace-period handling know + * ct->nmi_nesting to let the RCU grace-period handling know * that the CPU is active. This implementation permits nested NMIs, as * long as the nesting level does not overflow an int. (You will probably * run out of stack space first.) @@ -264,10 +264,10 @@ void noinstr ct_nmi_enter(void) WARN_ON_ONCE(ct_dynticks_nmi_nesting() < 0); /* - * If idle from RCU viewpoint, atomically increment ->dynticks - * to mark non-idle and increment ->dynticks_nmi_nesting by one. - * Otherwise, increment ->dynticks_nmi_nesting by two. This means - * if ->dynticks_nmi_nesting is equal to one, we are guaranteed + * If idle from RCU viewpoint, atomically increment CT state + * to mark non-idle and increment ->nmi_nesting by one. + * Otherwise, increment ->nmi_nesting by two. This means + * if ->nmi_nesting is equal to one, we are guaranteed * to be in the outermost NMI handler that interrupted an RCU-idle * period (observation due to Andy Lutomirski). */ @@ -298,7 +298,7 @@ void noinstr ct_nmi_enter(void) ct_dynticks_nmi_nesting(), ct_dynticks_nmi_nesting() + incby, ct_rcu_watching()); instrumentation_end(); - WRITE_ONCE(ct->dynticks_nmi_nesting, /* Prevent store tearing. */ + WRITE_ONCE(ct->nmi_nesting, /* Prevent store tearing. */ ct_dynticks_nmi_nesting() + incby); barrier(); } diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index cf69a234080f..934f6b34a551 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -391,7 +391,7 @@ static int rcu_is_cpu_rrupt_from_idle(void) RCU_LOCKDEP_WARN(ct_nesting() < 0, "RCU nesting counter underflow!"); RCU_LOCKDEP_WARN(ct_dynticks_nmi_nesting() <= 0, - "RCU dynticks_nmi_nesting counter underflow/zero!"); + "RCU nmi_nesting counter underflow/zero!"); /* Are we at first interrupt nesting level? */ nesting = ct_dynticks_nmi_nesting(); @@ -600,7 +600,7 @@ void rcu_irq_exit_check_preempt(void) "RCU nesting counter underflow/zero!"); RCU_LOCKDEP_WARN(ct_dynticks_nmi_nesting() != DYNTICK_IRQ_NONIDLE, - "Bad RCU dynticks_nmi_nesting counter\n"); + "Bad RCU nmi_nesting counter\n"); RCU_LOCKDEP_WARN(rcu_dynticks_curr_cpu_in_eqs(), "RCU in extended quiescent state!"); } From patchwork Fri Aug 16 04:02:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neeraj Upadhyay X-Patchwork-Id: 13765403 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ACD8B82488; Fri, 16 Aug 2024 04:04:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723781078; cv=none; b=Lm2wqOxBGahuTJkc+KpLe/VStNnRJQ6uorOKfPscQQ3BOCULoYWhDuqePe8fidUPSPmEAWFg+pVKvMF/UdxBTFQJ6FXX6bicBt9e4V2hkpWO1Uo77/RtO7R2P3TTOkMlMN43l8ZtnISnF65BB7MU1vtT7V6qeQhwjx0A1BbJlzw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723781078; c=relaxed/simple; bh=GRWciJuVIN13exrIEry43JWRpUFvmg9EQQUVwxm6HNI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=uvyTTpHDRp53D9JQQX8lO86gtDIK85S1bvvwERfE037Wg+3vXYLkK5ArqmlrxweqFvliXBzPj3IyxkLoYC5UvmXCt0ebDo/k0st+UY4L6ySTbZrrQM+kFiZMPKmIokMIbDg+1Bqocxm0eSlqZ1xSKJHK+RGeMgO80g4v0gdNj1g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=fRpOi3hS; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="fRpOi3hS" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9BAE0C32782; Fri, 16 Aug 2024 04:04:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723781078; bh=GRWciJuVIN13exrIEry43JWRpUFvmg9EQQUVwxm6HNI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=fRpOi3hSopzo/Xx/+yXZ4yiEXJ5G5Oy/tdeM7y4kNUbd2JB07Ku0telDMj9Y9ohNp 4Glvt3oJiyjsvjQ4BbinZ5sWW2quNfrnDHH2cmh1Z95RYcXQ2vuwDRT0tgLppSXTw7 J/PpFXTd+w9Iz5YYnsmptbNiC2qvNceULEhqAcA9aWxYcQHsSTuwOrdsURKq07ze1b A/55MkE7oacPqA4V+vmZexvCpRXYEChWjy1UbtW18y2RCR3k70N3mZVOB/R81e0A0r zlYzjZO5ERzrSTE9we0GHC4GoHfqM6giIwOI81fIbSMzUNPOeZSvHya744ERc21M+I sWin8xkwdqX3g== From: neeraj.upadhyay@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, paulmck@kernel.org, neeraj.upadhyay@kernel.org, neeraj.upadhyay@amd.com, boqun.feng@gmail.com, joel@joelfernandes.org, urezki@gmail.com, frederic@kernel.org, Valentin Schneider Subject: [PATCH rcu 10/26] context_tracking, rcu: Rename ct_dynticks_nmi_nesting() into ct_nmi_nesting() Date: Fri, 16 Aug 2024 09:32:25 +0530 Message-Id: <20240816040241.17776-10-neeraj.upadhyay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240816040030.GA16986@neeraj.linux> References: <20240816040030.GA16986@neeraj.linux> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Valentin Schneider The context_tracking.state RCU_DYNTICKS subvariable has been renamed to RCU_WATCHING, and the 'dynticks' prefix can be dropped without losing any meaning. Suggested-by: Frederic Weisbecker Signed-off-by: Valentin Schneider Reviewed-by: Frederic Weisbecker Signed-off-by: Neeraj Upadhyay --- include/linux/context_tracking_state.h | 2 +- kernel/context_tracking.c | 24 ++++++++++++------------ kernel/rcu/tree.c | 6 +++--- 3 files changed, 16 insertions(+), 16 deletions(-) diff --git a/include/linux/context_tracking_state.h b/include/linux/context_tracking_state.h index 12d00adf29e1..8f32fe599c5c 100644 --- a/include/linux/context_tracking_state.h +++ b/include/linux/context_tracking_state.h @@ -87,7 +87,7 @@ static __always_inline long ct_nesting_cpu(int cpu) return ct->nesting; } -static __always_inline long ct_dynticks_nmi_nesting(void) +static __always_inline long ct_nmi_nesting(void) { return __this_cpu_read(context_tracking.nmi_nesting); } diff --git a/kernel/context_tracking.c b/kernel/context_tracking.c index ae94215aa132..115843eeb030 100644 --- a/kernel/context_tracking.c +++ b/kernel/context_tracking.c @@ -125,7 +125,7 @@ static void noinstr ct_kernel_exit(bool user, int offset) { struct context_tracking *ct = this_cpu_ptr(&context_tracking); - WARN_ON_ONCE(ct_dynticks_nmi_nesting() != DYNTICK_IRQ_NONIDLE); + WARN_ON_ONCE(ct_nmi_nesting() != DYNTICK_IRQ_NONIDLE); WRITE_ONCE(ct->nmi_nesting, 0); WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && ct_nesting() == 0); @@ -185,7 +185,7 @@ static void noinstr ct_kernel_enter(bool user, int offset) trace_rcu_dyntick(TPS("End"), ct_nesting(), 1, ct_rcu_watching()); WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !user && !is_idle_task(current)); WRITE_ONCE(ct->nesting, 1); - WARN_ON_ONCE(ct_dynticks_nmi_nesting()); + WARN_ON_ONCE(ct_nmi_nesting()); WRITE_ONCE(ct->nmi_nesting, DYNTICK_IRQ_NONIDLE); instrumentation_end(); } @@ -207,28 +207,28 @@ void noinstr ct_nmi_exit(void) instrumentation_begin(); /* - * Check for ->nmi_nesting underflow and bad ->dynticks. + * Check for ->nmi_nesting underflow and bad CT state. * (We are exiting an NMI handler, so RCU better be paying attention * to us!) */ - WARN_ON_ONCE(ct_dynticks_nmi_nesting() <= 0); + WARN_ON_ONCE(ct_nmi_nesting() <= 0); WARN_ON_ONCE(rcu_dynticks_curr_cpu_in_eqs()); /* * If the nesting level is not 1, the CPU wasn't RCU-idle, so * leave it in non-RCU-idle state. */ - if (ct_dynticks_nmi_nesting() != 1) { - trace_rcu_dyntick(TPS("--="), ct_dynticks_nmi_nesting(), ct_dynticks_nmi_nesting() - 2, + if (ct_nmi_nesting() != 1) { + trace_rcu_dyntick(TPS("--="), ct_nmi_nesting(), ct_nmi_nesting() - 2, ct_rcu_watching()); WRITE_ONCE(ct->nmi_nesting, /* No store tearing. */ - ct_dynticks_nmi_nesting() - 2); + ct_nmi_nesting() - 2); instrumentation_end(); return; } /* This NMI interrupted an RCU-idle CPU, restore RCU-idleness. */ - trace_rcu_dyntick(TPS("Startirq"), ct_dynticks_nmi_nesting(), 0, ct_rcu_watching()); + trace_rcu_dyntick(TPS("Startirq"), ct_nmi_nesting(), 0, ct_rcu_watching()); WRITE_ONCE(ct->nmi_nesting, 0); /* Avoid store tearing. */ // instrumentation for the noinstr ct_kernel_exit_state() @@ -261,7 +261,7 @@ void noinstr ct_nmi_enter(void) struct context_tracking *ct = this_cpu_ptr(&context_tracking); /* Complain about underflow. */ - WARN_ON_ONCE(ct_dynticks_nmi_nesting() < 0); + WARN_ON_ONCE(ct_nmi_nesting() < 0); /* * If idle from RCU viewpoint, atomically increment CT state @@ -295,11 +295,11 @@ void noinstr ct_nmi_enter(void) } trace_rcu_dyntick(incby == 1 ? TPS("Endirq") : TPS("++="), - ct_dynticks_nmi_nesting(), - ct_dynticks_nmi_nesting() + incby, ct_rcu_watching()); + ct_nmi_nesting(), + ct_nmi_nesting() + incby, ct_rcu_watching()); instrumentation_end(); WRITE_ONCE(ct->nmi_nesting, /* Prevent store tearing. */ - ct_dynticks_nmi_nesting() + incby); + ct_nmi_nesting() + incby); barrier(); } diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 934f6b34a551..14cc314eedad 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -390,11 +390,11 @@ static int rcu_is_cpu_rrupt_from_idle(void) /* Check for counter underflows */ RCU_LOCKDEP_WARN(ct_nesting() < 0, "RCU nesting counter underflow!"); - RCU_LOCKDEP_WARN(ct_dynticks_nmi_nesting() <= 0, + RCU_LOCKDEP_WARN(ct_nmi_nesting() <= 0, "RCU nmi_nesting counter underflow/zero!"); /* Are we at first interrupt nesting level? */ - nesting = ct_dynticks_nmi_nesting(); + nesting = ct_nmi_nesting(); if (nesting > 1) return false; @@ -598,7 +598,7 @@ void rcu_irq_exit_check_preempt(void) RCU_LOCKDEP_WARN(ct_nesting() <= 0, "RCU nesting counter underflow/zero!"); - RCU_LOCKDEP_WARN(ct_dynticks_nmi_nesting() != + RCU_LOCKDEP_WARN(ct_nmi_nesting() != DYNTICK_IRQ_NONIDLE, "Bad RCU nmi_nesting counter\n"); RCU_LOCKDEP_WARN(rcu_dynticks_curr_cpu_in_eqs(), From patchwork Fri Aug 16 04:02:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neeraj Upadhyay X-Patchwork-Id: 13765404 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F3EA183CD2; Fri, 16 Aug 2024 04:04:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723781089; cv=none; b=XrgZAG3/ALCS7kis4K8p9hq45RM7cXg36DURtIbX63urUvfNewExtVTq1BqnKnvBDF53n7j+YYMBIjDJIgmw687jG1+WqOSDXxtqGB8r5mN9edAgUkY7Tpe/BigZGIrxqADpE9i50GBEbUDoNxV/sJ2Pb6pUbxTFjUVf4ioq4Ko= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723781089; c=relaxed/simple; bh=7Y5L9kQlhh/suY4IvkdPD01biH3wXCMEYqTW+T86dAU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=tvo2ulKxno1UWcFXx43+YpR8wPkBfFxp4GhEqkGK3mzWGgNiLPkhSW0wAKA8VhbB7MCgVBNAN8dHnqe/kTuuljTV0IgO2PzPXbMnx0SxZxCGp7y8Pp1W0m51GLr6sP893BLjFQN26aIsvNb/npGVgGm/vUqm4H+ugxOtbjyRtfk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=rb5sJcYb; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="rb5sJcYb" Received: by smtp.kernel.org (Postfix) with ESMTPSA id D218BC4AF10; Fri, 16 Aug 2024 04:04:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723781088; bh=7Y5L9kQlhh/suY4IvkdPD01biH3wXCMEYqTW+T86dAU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=rb5sJcYbosa1IBgN8WnGGRBd0IXRW6iLl1J9N2jlwrkO+9zdo5balWdKoNVxjQ6dN iok41qibjtjV5nkQruK9Kj2cFLD1D5PkCGW3c6jBLUrWUYwX/9mx6l3mFchAkYS+95 W3XiuwJdOWKCRDvr7IzySEsnUUXd3msQ2TXmjSQJtJeHfkIC7bmI4WvYO3BHADT6Ve r6gk3mUaKpd5q5HvH6ScQX1keIXJksjS2XPI1dJxmrrO8TwVHR7qb6a3Vpn7PRoG/g 2eEpj+NUCOnbc8ZD4+WuQvtVFRIGlgqFF5Z3Me1n2A8GChBIF4iwrrziYVO45E8mv4 TSpZD2lJvD9BA== From: neeraj.upadhyay@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, paulmck@kernel.org, neeraj.upadhyay@kernel.org, neeraj.upadhyay@amd.com, boqun.feng@gmail.com, joel@joelfernandes.org, urezki@gmail.com, frederic@kernel.org, Valentin Schneider Subject: [PATCH rcu 11/26] context_tracking, rcu: Rename ct_dynticks_nmi_nesting_cpu() into ct_nmi_nesting_cpu() Date: Fri, 16 Aug 2024 09:32:26 +0530 Message-Id: <20240816040241.17776-11-neeraj.upadhyay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240816040030.GA16986@neeraj.linux> References: <20240816040030.GA16986@neeraj.linux> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Valentin Schneider The context_tracking.state RCU_DYNTICKS subvariable has been renamed to RCU_WATCHING, and the 'dynticks' prefix can be dropped without losing any meaning. Suggested-by: Frederic Weisbecker Signed-off-by: Valentin Schneider Reviewed-by: Frederic Weisbecker Signed-off-by: Neeraj Upadhyay --- include/linux/context_tracking_state.h | 2 +- kernel/rcu/tree_stall.h | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/include/linux/context_tracking_state.h b/include/linux/context_tracking_state.h index 8f32fe599c5c..34fd504e53a8 100644 --- a/include/linux/context_tracking_state.h +++ b/include/linux/context_tracking_state.h @@ -92,7 +92,7 @@ static __always_inline long ct_nmi_nesting(void) return __this_cpu_read(context_tracking.nmi_nesting); } -static __always_inline long ct_dynticks_nmi_nesting_cpu(int cpu) +static __always_inline long ct_nmi_nesting_cpu(int cpu) { struct context_tracking *ct = per_cpu_ptr(&context_tracking, cpu); diff --git a/kernel/rcu/tree_stall.h b/kernel/rcu/tree_stall.h index 59b1d84a4749..ec49f0155bec 100644 --- a/kernel/rcu/tree_stall.h +++ b/kernel/rcu/tree_stall.h @@ -516,7 +516,7 @@ static void print_cpu_stall_info(int cpu) "!."[!delta], ticks_value, ticks_title, ct_rcu_watching_cpu(cpu) & 0xffff, - ct_nesting_cpu(cpu), ct_dynticks_nmi_nesting_cpu(cpu), + ct_nesting_cpu(cpu), ct_nmi_nesting_cpu(cpu), rdp->softirq_snap, kstat_softirqs_cpu(RCU_SOFTIRQ, cpu), data_race(rcu_state.n_force_qs) - rcu_state.n_force_qs_gpstart, rcuc_starved ? buf : "", From patchwork Fri Aug 16 04:02:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neeraj Upadhyay X-Patchwork-Id: 13765405 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4F0DF84DE4; Fri, 16 Aug 2024 04:04:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723781096; cv=none; b=Gnok/U565D4djiRDCOERHdozJ8OzcTN+q215lqNRkk8ZjIo9GEMWaaeEjFFnZVbatG8bnd6z4nU4lC2XIPJf7s+ucXY2pHyLvulCxbJmWkP4UNV21qVSz3BqNxgBSe98l2c1IpTq7YI7t2TMcCb6rS40Y/O0prxDc+7mIBNIE7E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723781096; c=relaxed/simple; bh=THKTu+qqZwMD5L0kBlzwGcrwFIAlx/sAcIbij6QiS+4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=b1GVFGsCvgtgMpixs0zlyxO1SfFeIXvm655ZWzmcw1tNXYqktQtqvfzXnFLIBrpgsJKGwe/L4D0BuA3cIrsL0auZeeMMkFbrHljymAcCK62xR/QZDkPxFTj5LxFLWczTXMt0320G2TG4+JGcWSUutkRmPdiA1/kk9771Nm7DQHU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=H7ZTaUJ1; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="H7ZTaUJ1" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 738EBC4AF0B; Fri, 16 Aug 2024 04:04:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723781096; bh=THKTu+qqZwMD5L0kBlzwGcrwFIAlx/sAcIbij6QiS+4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=H7ZTaUJ1d5wA92LmRxQ/7LnW83Eaz2keopv7OXCO9ZBMf7LAVSMv+D4uV9reH1OLx ZbKqNsql38TNYfAX2HnpPKhRt1bw49TwlMU2LHAQqxvbcT011U3SbMkI8NNA5aWXnU QT4m8ObktHWkL1c4ocPeysmlhOQdf8cdtwfMyv0Zwp5OwKfZ347TfcJiKegNT7XrNU gyzK7T/KsiQS4WJwmGETyZefM0gzCBfCTc1G7cZhssufSqAa+hIrrAvAUEqJkjTpEl /BAdm0b8Rv+sOdN2r9Fvk5aS8cZuB3diuzIpl0bd3uZtTm+K3eg1DL2xNZbT9jNz/p VIcNkcNprYP4A== From: neeraj.upadhyay@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, paulmck@kernel.org, neeraj.upadhyay@kernel.org, neeraj.upadhyay@amd.com, boqun.feng@gmail.com, joel@joelfernandes.org, urezki@gmail.com, frederic@kernel.org, Valentin Schneider Subject: [PATCH rcu 12/26] context_tracking, rcu: Rename DYNTICK_IRQ_NONIDLE into CT_NESTING_IRQ_NONIDLE Date: Fri, 16 Aug 2024 09:32:27 +0530 Message-Id: <20240816040241.17776-12-neeraj.upadhyay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240816040030.GA16986@neeraj.linux> References: <20240816040030.GA16986@neeraj.linux> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Valentin Schneider The context_tracking.state RCU_DYNTICKS subvariable has been renamed to RCU_WATCHING, and the 'dynticks' prefix can be dropped without losing any meaning. Signed-off-by: Valentin Schneider Reviewed-by: Frederic Weisbecker Signed-off-by: Neeraj Upadhyay --- include/linux/context_tracking_state.h | 2 +- kernel/context_tracking.c | 8 ++++---- kernel/rcu/tree.c | 2 +- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/include/linux/context_tracking_state.h b/include/linux/context_tracking_state.h index 34fd504e53a8..0dbda59c9f37 100644 --- a/include/linux/context_tracking_state.h +++ b/include/linux/context_tracking_state.h @@ -7,7 +7,7 @@ #include /* Offset to allow distinguishing irq vs. task-based idle entry/exit. */ -#define DYNTICK_IRQ_NONIDLE ((LONG_MAX / 2) + 1) +#define CT_NESTING_IRQ_NONIDLE ((LONG_MAX / 2) + 1) enum ctx_state { CT_STATE_DISABLED = -1, /* returned by ct_state() if unknown */ diff --git a/kernel/context_tracking.c b/kernel/context_tracking.c index 115843eeb030..8262f57a4363 100644 --- a/kernel/context_tracking.c +++ b/kernel/context_tracking.c @@ -29,7 +29,7 @@ DEFINE_PER_CPU(struct context_tracking, context_tracking) = { #ifdef CONFIG_CONTEXT_TRACKING_IDLE .nesting = 1, - .nmi_nesting = DYNTICK_IRQ_NONIDLE, + .nmi_nesting = CT_NESTING_IRQ_NONIDLE, #endif .state = ATOMIC_INIT(CT_RCU_WATCHING), }; @@ -125,7 +125,7 @@ static void noinstr ct_kernel_exit(bool user, int offset) { struct context_tracking *ct = this_cpu_ptr(&context_tracking); - WARN_ON_ONCE(ct_nmi_nesting() != DYNTICK_IRQ_NONIDLE); + WARN_ON_ONCE(ct_nmi_nesting() != CT_NESTING_IRQ_NONIDLE); WRITE_ONCE(ct->nmi_nesting, 0); WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && ct_nesting() == 0); @@ -156,7 +156,7 @@ static void noinstr ct_kernel_exit(bool user, int offset) * Exit an RCU extended quiescent state, which can be either the * idle loop or adaptive-tickless usermode execution. * - * We crowbar the ->nmi_nesting field to DYNTICK_IRQ_NONIDLE to + * We crowbar the ->nmi_nesting field to CT_NESTING_IRQ_NONIDLE to * allow for the possibility of usermode upcalls messing up our count of * interrupt nesting level during the busy period that is just now starting. */ @@ -186,7 +186,7 @@ static void noinstr ct_kernel_enter(bool user, int offset) WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !user && !is_idle_task(current)); WRITE_ONCE(ct->nesting, 1); WARN_ON_ONCE(ct_nmi_nesting()); - WRITE_ONCE(ct->nmi_nesting, DYNTICK_IRQ_NONIDLE); + WRITE_ONCE(ct->nmi_nesting, CT_NESTING_IRQ_NONIDLE); instrumentation_end(); } diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 14cc314eedad..4778c873f4ac 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -599,7 +599,7 @@ void rcu_irq_exit_check_preempt(void) RCU_LOCKDEP_WARN(ct_nesting() <= 0, "RCU nesting counter underflow/zero!"); RCU_LOCKDEP_WARN(ct_nmi_nesting() != - DYNTICK_IRQ_NONIDLE, + CT_NESTING_IRQ_NONIDLE, "Bad RCU nmi_nesting counter\n"); RCU_LOCKDEP_WARN(rcu_dynticks_curr_cpu_in_eqs(), "RCU in extended quiescent state!"); From patchwork Fri Aug 16 04:02:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neeraj Upadhyay X-Patchwork-Id: 13765406 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AD7E9770F1; Fri, 16 Aug 2024 04:05:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723781114; cv=none; b=d319jfyaWs2uKJXkCVx9LuutVSSU1WqG5iBXfY5jLv6HGnAnuDZTRl3eIrDu5OIBo1JnuIIis4NqOTS3thxGkPANBLoje6yWskrDlDu1nISY8sh7geXiNBetkbEiCuhwr2WgV9zKxzRRTcZxR6dcyVGwcU0ZWNl7gdOKndE+9x0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723781114; c=relaxed/simple; bh=0tBGnHusoipi1DnYmvCYm+XQmQDY34ztMhfItPHweQ4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=k3+l8ffuf6CVGTLUKxHCFvFs1YhVItCjWfj3jamZEn+/Kq4PHLVw+lNRYuTdCtl8OX1J8usxKO/idwlGCSDkwFuTZus9kXgCJvAuLlj0Glqp7eMmKpCWmh3mP286NnealU8NKXAj9LWymYhPoxVhlDQGyB9wOX4wPhKAkqa+Utw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=jeTTMhK8; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="jeTTMhK8" Received: by smtp.kernel.org (Postfix) with ESMTPSA id D048AC32782; Fri, 16 Aug 2024 04:05:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723781113; bh=0tBGnHusoipi1DnYmvCYm+XQmQDY34ztMhfItPHweQ4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=jeTTMhK8AqLx7C0EN9f5PZZ+YYEKLu/NIzKxksH2eKpftdXSob/8sWPKPOnz64y/E aRfZ/3HWcO4ZwHKdCq+kRqhkIXTRT3DqFFRhs0+pA/XPMlOG74GJLeu642KAxW/qGK sLSHtdEd+njuifDOM3uM6C9lBVAMWhvH49AgxCtkZ9uiGgkPPkSBXKnoFRYqTH43HR Bn8Xh1ZTatbnqFguV5dU1IRAJzHmps3c1zFT9tG+sQVryLiRin8UFF5RzqFun8uXYS 4aAX6IzZI2+LJn8ZTSc6l+3KOgpMdE4IacdKW6HgvDfeiJk7Cy9F0kU65q8pufL+9Q i/Bp8py5OIG0g== From: neeraj.upadhyay@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, paulmck@kernel.org, neeraj.upadhyay@kernel.org, neeraj.upadhyay@amd.com, boqun.feng@gmail.com, joel@joelfernandes.org, urezki@gmail.com, frederic@kernel.org, Valentin Schneider Subject: [PATCH rcu 13/26] context_tracking, rcu: Rename rcu_dynticks_task*() into rcu_task*() Date: Fri, 16 Aug 2024 09:32:28 +0530 Message-Id: <20240816040241.17776-13-neeraj.upadhyay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240816040030.GA16986@neeraj.linux> References: <20240816040030.GA16986@neeraj.linux> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Valentin Schneider The context_tracking.state RCU_DYNTICKS subvariable has been renamed to RCU_WATCHING, and the 'dynticks' prefix can be dropped without losing any meaning. While at it, flip the suffixes of these helpers. We are not telling that we are entering dynticks mode from an RCU-task perspective anymore; we are telling that we are exiting RCU-tasks because we are in eqs mode. [ neeraj.upadhyay: Incorporate Frederic Weisbecker feedback. ] Suggested-by: Frederic Weisbecker Signed-off-by: Valentin Schneider Reviewed-by: Frederic Weisbecker Signed-off-by: Neeraj Upadhyay --- kernel/context_tracking.c | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/kernel/context_tracking.c b/kernel/context_tracking.c index 8262f57a4363..00c8f128885a 100644 --- a/kernel/context_tracking.c +++ b/kernel/context_tracking.c @@ -38,24 +38,24 @@ EXPORT_SYMBOL_GPL(context_tracking); #ifdef CONFIG_CONTEXT_TRACKING_IDLE #define TPS(x) tracepoint_string(x) -/* Record the current task on dyntick-idle entry. */ -static __always_inline void rcu_dynticks_task_enter(void) +/* Record the current task on exiting RCU-tasks (dyntick-idle entry). */ +static __always_inline void rcu_task_exit(void) { #if defined(CONFIG_TASKS_RCU) && defined(CONFIG_NO_HZ_FULL) WRITE_ONCE(current->rcu_tasks_idle_cpu, smp_processor_id()); #endif /* #if defined(CONFIG_TASKS_RCU) && defined(CONFIG_NO_HZ_FULL) */ } -/* Record no current task on dyntick-idle exit. */ -static __always_inline void rcu_dynticks_task_exit(void) +/* Record no current task on entering RCU-tasks (dyntick-idle exit). */ +static __always_inline void rcu_task_enter(void) { #if defined(CONFIG_TASKS_RCU) && defined(CONFIG_NO_HZ_FULL) WRITE_ONCE(current->rcu_tasks_idle_cpu, -1); #endif /* #if defined(CONFIG_TASKS_RCU) && defined(CONFIG_NO_HZ_FULL) */ } -/* Turn on heavyweight RCU tasks trace readers on idle/user entry. */ -static __always_inline void rcu_dynticks_task_trace_enter(void) +/* Turn on heavyweight RCU tasks trace readers on kernel exit. */ +static __always_inline void rcu_task_trace_heavyweight_enter(void) { #ifdef CONFIG_TASKS_TRACE_RCU if (IS_ENABLED(CONFIG_TASKS_TRACE_RCU_READ_MB)) @@ -63,8 +63,8 @@ static __always_inline void rcu_dynticks_task_trace_enter(void) #endif /* #ifdef CONFIG_TASKS_TRACE_RCU */ } -/* Turn off heavyweight RCU tasks trace readers on idle/user exit. */ -static __always_inline void rcu_dynticks_task_trace_exit(void) +/* Turn off heavyweight RCU tasks trace readers on kernel entry. */ +static __always_inline void rcu_task_trace_heavyweight_exit(void) { #ifdef CONFIG_TASKS_TRACE_RCU if (IS_ENABLED(CONFIG_TASKS_TRACE_RCU_READ_MB)) @@ -87,7 +87,7 @@ static noinstr void ct_kernel_exit_state(int offset) * critical sections, and we also must force ordering with the * next idle sojourn. */ - rcu_dynticks_task_trace_enter(); // Before ->dynticks update! + rcu_task_trace_heavyweight_enter(); // Before CT state update! seq = ct_state_inc(offset); // RCU is no longer watching. Better be in extended quiescent state! WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && (seq & CT_RCU_WATCHING)); @@ -109,7 +109,7 @@ static noinstr void ct_kernel_enter_state(int offset) */ seq = ct_state_inc(offset); // RCU is now watching. Better not be in an extended quiescent state! - rcu_dynticks_task_trace_exit(); // After ->dynticks update! + rcu_task_trace_heavyweight_exit(); // After CT state update! WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !(seq & CT_RCU_WATCHING)); } @@ -149,7 +149,7 @@ static void noinstr ct_kernel_exit(bool user, int offset) // RCU is watching here ... ct_kernel_exit_state(offset); // ... but is no longer watching here. - rcu_dynticks_task_enter(); + rcu_task_exit(); } /* @@ -173,7 +173,7 @@ static void noinstr ct_kernel_enter(bool user, int offset) ct->nesting++; return; } - rcu_dynticks_task_exit(); + rcu_task_enter(); // RCU is not watching here ... ct_kernel_enter_state(offset); // ... but is watching here. @@ -240,7 +240,7 @@ void noinstr ct_nmi_exit(void) // ... but is no longer watching here. if (!in_nmi()) - rcu_dynticks_task_enter(); + rcu_task_exit(); } /** @@ -274,7 +274,7 @@ void noinstr ct_nmi_enter(void) if (rcu_dynticks_curr_cpu_in_eqs()) { if (!in_nmi()) - rcu_dynticks_task_exit(); + rcu_task_enter(); // RCU is not watching here ... ct_kernel_enter_state(CT_RCU_WATCHING); From patchwork Fri Aug 16 04:02:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neeraj Upadhyay X-Patchwork-Id: 13765407 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A8C0078C6D; Fri, 16 Aug 2024 04:05:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723781120; cv=none; b=lLIaEdY0YoHda7FE6ytjzz1r8K1yJI8b6XidVnOvJcTQxyQfnht4Pc2KSVBKMul1H6IC757hs2vacQv78sOUL5MXWytnVfQ6yU9qtsapIQSCjpxeStj16m6o6AzLdEGBW8Wz+zKb7ZI+qhsaNaxXP4Xc/0SF9AyieKdw1KtZ6Mo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723781120; c=relaxed/simple; bh=38Y/IJhB87OLzqPmhC++nD4wUogBNY48hQVAJtlQdIc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Ln02VB721z5lquTNKSNjc2I743scerHfowx6cJsABR35NmgUbkMGc/ygvTljJi7V9HlXCAETdVz2aCNWvDEvpGKLmrww2s96gCpmR+Nz+OMcq4kk7RwIVLCTuImyqLPkOTkZ4oPxTHy949aEOBIYeHRi9JH6cB1ec9kk3CQstX4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=meGHsZhp; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="meGHsZhp" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0077AC32782; Fri, 16 Aug 2024 04:05:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723781120; bh=38Y/IJhB87OLzqPmhC++nD4wUogBNY48hQVAJtlQdIc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=meGHsZhp4zLkXmZpcBfgyvbfyRf4lMQhOQbheLxorgupFPrZjMbbO32uFk/8Url84 +3KoPnS+7VbK6wg0H7m6eSxSEgCjg1TZqb/hVs8qPLc3uNMcuym4jrJsFYqY59NieA 2AMEQr7T4PXeUrrs78UeuBOIf50P/scr8cCepvrROYZ1tsF8O/HW98WCMLu0RrvJdI X9i4gWwaE8tP51G2I6ORKFJw1nbqfCIkJ7PmZdKJAiP4oYIjUzyMXt2UI0/p7UPPOq pIHfpF1tERmyOLLFTJleBxCarF1/bdbyB9gYlPOiO3PCaYo8d0/BmTy3yKQpj9Iy1o 6yN9cKifkXTSg== From: neeraj.upadhyay@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, paulmck@kernel.org, neeraj.upadhyay@kernel.org, neeraj.upadhyay@amd.com, boqun.feng@gmail.com, joel@joelfernandes.org, urezki@gmail.com, frederic@kernel.org, Valentin Schneider Subject: [PATCH rcu 14/26] context_tracking, rcu: Rename rcu_dynticks_curr_cpu_in_eqs() into rcu_is_watching_curr_cpu() Date: Fri, 16 Aug 2024 09:32:29 +0530 Message-Id: <20240816040241.17776-14-neeraj.upadhyay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240816040030.GA16986@neeraj.linux> References: <20240816040030.GA16986@neeraj.linux> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Valentin Schneider The context_tracking.state RCU_DYNTICKS subvariable has been renamed to RCU_WATCHING, reflect that change in the related helpers. Note that "watching" is the opposite of "in EQS", so the negation is lifted out of the helper and into the callsites. Signed-off-by: Valentin Schneider Reviewed-by: Frederic Weisbecker Signed-off-by: Neeraj Upadhyay --- include/linux/context_tracking.h | 12 ++++++++---- kernel/context_tracking.c | 6 +++--- kernel/rcu/tree.c | 6 +++--- 3 files changed, 14 insertions(+), 10 deletions(-) diff --git a/include/linux/context_tracking.h b/include/linux/context_tracking.h index a6c36780cc3b..d53092ffa9db 100644 --- a/include/linux/context_tracking.h +++ b/include/linux/context_tracking.h @@ -113,13 +113,17 @@ extern void ct_idle_enter(void); extern void ct_idle_exit(void); /* - * Is the current CPU in an extended quiescent state? + * Is RCU watching the current CPU (IOW, it is not in an extended quiescent state)? + * + * Note that this returns the actual boolean data (watching / not watching), + * whereas ct_rcu_watching() returns the RCU_WATCHING subvariable of + * context_tracking.state. * * No ordering, as we are sampling CPU-local information. */ -static __always_inline bool rcu_dynticks_curr_cpu_in_eqs(void) +static __always_inline bool rcu_is_watching_curr_cpu(void) { - return !(raw_atomic_read(this_cpu_ptr(&context_tracking.state)) & CT_RCU_WATCHING); + return raw_atomic_read(this_cpu_ptr(&context_tracking.state)) & CT_RCU_WATCHING; } /* @@ -140,7 +144,7 @@ static __always_inline bool warn_rcu_enter(void) * lots of the actual reporting also relies on RCU. */ preempt_disable_notrace(); - if (rcu_dynticks_curr_cpu_in_eqs()) { + if (!rcu_is_watching_curr_cpu()) { ret = true; ct_state_inc(CT_RCU_WATCHING); } diff --git a/kernel/context_tracking.c b/kernel/context_tracking.c index 00c8f128885a..31c6d0c1b79d 100644 --- a/kernel/context_tracking.c +++ b/kernel/context_tracking.c @@ -212,7 +212,7 @@ void noinstr ct_nmi_exit(void) * to us!) */ WARN_ON_ONCE(ct_nmi_nesting() <= 0); - WARN_ON_ONCE(rcu_dynticks_curr_cpu_in_eqs()); + WARN_ON_ONCE(!rcu_is_watching_curr_cpu()); /* * If the nesting level is not 1, the CPU wasn't RCU-idle, so @@ -271,7 +271,7 @@ void noinstr ct_nmi_enter(void) * to be in the outermost NMI handler that interrupted an RCU-idle * period (observation due to Andy Lutomirski). */ - if (rcu_dynticks_curr_cpu_in_eqs()) { + if (!rcu_is_watching_curr_cpu()) { if (!in_nmi()) rcu_task_enter(); @@ -281,7 +281,7 @@ void noinstr ct_nmi_enter(void) // ... but is watching here. instrumentation_begin(); - // instrumentation for the noinstr rcu_dynticks_curr_cpu_in_eqs() + // instrumentation for the noinstr rcu_is_watching_curr_cpu() instrument_atomic_read(&ct->state, sizeof(ct->state)); // instrumentation for the noinstr ct_kernel_enter_state() instrument_atomic_write(&ct->state, sizeof(ct->state)); diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 4778c873f4ac..68eca7d3fdd1 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -601,7 +601,7 @@ void rcu_irq_exit_check_preempt(void) RCU_LOCKDEP_WARN(ct_nmi_nesting() != CT_NESTING_IRQ_NONIDLE, "Bad RCU nmi_nesting counter\n"); - RCU_LOCKDEP_WARN(rcu_dynticks_curr_cpu_in_eqs(), + RCU_LOCKDEP_WARN(!rcu_is_watching_curr_cpu(), "RCU in extended quiescent state!"); } #endif /* #ifdef CONFIG_PROVE_RCU */ @@ -641,7 +641,7 @@ void __rcu_irq_enter_check_tick(void) if (in_nmi()) return; - RCU_LOCKDEP_WARN(rcu_dynticks_curr_cpu_in_eqs(), + RCU_LOCKDEP_WARN(!rcu_is_watching_curr_cpu(), "Illegal rcu_irq_enter_check_tick() from extended quiescent state"); if (!tick_nohz_full_cpu(rdp->cpu) || @@ -723,7 +723,7 @@ notrace bool rcu_is_watching(void) bool ret; preempt_disable_notrace(); - ret = !rcu_dynticks_curr_cpu_in_eqs(); + ret = rcu_is_watching_curr_cpu(); preempt_enable_notrace(); return ret; } From patchwork Fri Aug 16 04:02:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neeraj Upadhyay X-Patchwork-Id: 13765408 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 046F278C6D; Fri, 16 Aug 2024 04:05:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723781128; cv=none; b=Qs7/bp2M7B9OtbLobojhC0HJem3hh/EdqsQd+ajnw+hPokRCNxNpAmQv5jATfW7EZ6JIwLwyLHfHkMcLEQa2FTbkOjapqlbW52O7ThdLlDc4DHRAT6wWsepb8F+oCJSVIJhjxQ0P6ZhjpEfbJB8mTBTdD7y3De4O+oQ+MOmdRaM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723781128; c=relaxed/simple; bh=pdYpSdwEaOBgNhubrdJXez5sTv6oyOsjgknZPbrT68k=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=JqyBym9w+fwlJeHMdIa5YH+kDb0x/nxZTp25NLpeasAXMERJH8m5UhTi6MQRcnIrXxW9GuyDDMOjEadP3wwxJLoO+WH4jHtsmNVcb15Po9EAmZR7V2HeOEVKZtgGtAqPHzYLzxDfLkXghJtoIDW5TntdqZfnDLckRDtbzSmtQ1Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=vIzXlMhb; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="vIzXlMhb" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1D550C32782; Fri, 16 Aug 2024 04:05:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723781127; bh=pdYpSdwEaOBgNhubrdJXez5sTv6oyOsjgknZPbrT68k=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=vIzXlMhb0J2GP9xF6ZNzMRBHdvVHGmQGpSsYVbuD/attf0hAGX0Jd1fYUVeXnJx+S Bm0Das7XGiuXpKBIgVjHoIWmQXyXic+cR5dxrl/v7rcecxuh2iffjTY6+NqnA854mW cEBMkbviS/t3irDril6AiOZV4m9/5ShgpkiOLlBRhCrNKjEQbtwBClzBI0qnfm5Zv6 Eqfc0wtvb6u7U4IsvV6KaIQ15mKDxu58iqgb4IiVir1KWjmm9yZ4x/qnpIq60Iu47W NYQReYSdFEtcL/sjc1q8qc8Vg5YWhjdObvVIV8RZhlxESfxcxEzB3IQD8jdNCcDGoj iwnf3ykCc5X4Q== From: neeraj.upadhyay@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, paulmck@kernel.org, neeraj.upadhyay@kernel.org, neeraj.upadhyay@amd.com, boqun.feng@gmail.com, joel@joelfernandes.org, urezki@gmail.com, frederic@kernel.org, Valentin Schneider Subject: [PATCH rcu 15/26] rcu: Rename rcu_dynticks_eqs_online() into rcu_watching_online() Date: Fri, 16 Aug 2024 09:32:30 +0530 Message-Id: <20240816040241.17776-15-neeraj.upadhyay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240816040030.GA16986@neeraj.linux> References: <20240816040030.GA16986@neeraj.linux> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Valentin Schneider The context_tracking.state RCU_DYNTICKS subvariable has been renamed to RCU_WATCHING, reflect that change in the related helpers. Signed-off-by: Valentin Schneider Reviewed-by: Frederic Weisbecker Signed-off-by: Neeraj Upadhyay --- kernel/rcu/tree.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 68eca7d3fdd1..47ec9f2daaed 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -283,16 +283,16 @@ void rcu_softirq_qs(void) } /* - * Reset the current CPU's ->dynticks counter to indicate that the + * Reset the current CPU's RCU_WATCHING counter to indicate that the * newly onlined CPU is no longer in an extended quiescent state. * This will either leave the counter unchanged, or increment it * to the next non-quiescent value. * * The non-atomic test/increment sequence works because the upper bits - * of the ->dynticks counter are manipulated only by the corresponding CPU, + * of the ->state variable are manipulated only by the corresponding CPU, * or when the corresponding CPU is offline. */ -static void rcu_dynticks_eqs_online(void) +static void rcu_watching_online(void) { if (ct_rcu_watching() & CT_RCU_WATCHING) return; @@ -5058,7 +5058,7 @@ void rcutree_report_cpu_starting(unsigned int cpu) rnp = rdp->mynode; mask = rdp->grpmask; arch_spin_lock(&rcu_state.ofl_lock); - rcu_dynticks_eqs_online(); + rcu_watching_online(); raw_spin_lock(&rcu_state.barrier_lock); raw_spin_lock_rcu_node(rnp); WRITE_ONCE(rnp->qsmaskinitnext, rnp->qsmaskinitnext | mask); From patchwork Fri Aug 16 04:02:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neeraj Upadhyay X-Patchwork-Id: 13765409 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6B9E782488; Fri, 16 Aug 2024 04:05:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723781136; cv=none; b=FX9nKy89w2SsocVCaOotRqRadwn2YKW8PSsUaLxzdte1Q7rNb7mnHR49DC5VHvKTuJMle/nthMsGYwVEthddmcnE74kMzzTw8jUAuk8pzPCLf1Q9wa320awziwyT/X/5mj1+xSS2OLAFhKYoFkwZEl7IvFWGyhqtUhzwbkk/EEo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723781136; c=relaxed/simple; bh=+Mo5q7GWPN7t00hvEjG+UUC6SUC5HOsEA7aWBEXTzsU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=rDW/CoU/AhVCag2jcewjCpiDZTpqYDmWnJG9PU+ki/b+zPeSTzfG44gqaknpsTGArHhnsWCVouWnqhuZXQ/zwg3VbDUG90Y5wP21h9H5jN4x9frKouYjibgjwrHhBwU6BWkY/TwFm+QfYyYUAjIdcDbDvbQHnFsY2+EJb/STlZo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=c7mD8cd1; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="c7mD8cd1" Received: by smtp.kernel.org (Postfix) with ESMTPSA id EC5E0C4AF09; Fri, 16 Aug 2024 04:05:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723781136; bh=+Mo5q7GWPN7t00hvEjG+UUC6SUC5HOsEA7aWBEXTzsU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=c7mD8cd1wS1kVA4uCzXB07gAdF9jgv8mZ8Mkm/YVcBOQrMaIrbExGsIR2piAIuB0M VNgOtC9fOZgI0I05g0Ao1XjXRhAcY+xZUNIQqgq7Nlg3Gvu0NDzVr06GmW4tfkBi1y CV70aFdtdyLOp6BfhQZvc3P5l1nnPtyjphtN+UR+rktkdAc3JCiDy7F1Vs/3Ba71js YZZq0H2OtNyxJ96ma9viOhjgYM6sJZhziq4jFo6vcVUN20lozB7G1yFsZbFEuXzyd7 se1sM+KGWBSwMEVoH8s7F1cA8j3bGKDfzh7NZKfFwNpI8MVnPT7g/LNzRUoZsE8oqr eHqaCy1SVZdcw== From: neeraj.upadhyay@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, paulmck@kernel.org, neeraj.upadhyay@kernel.org, neeraj.upadhyay@amd.com, boqun.feng@gmail.com, joel@joelfernandes.org, urezki@gmail.com, frederic@kernel.org, Valentin Schneider Subject: [PATCH rcu 16/26] rcu: Rename rcu_dynticks_in_eqs() into rcu_watching_snap_in_eqs() Date: Fri, 16 Aug 2024 09:32:31 +0530 Message-Id: <20240816040241.17776-16-neeraj.upadhyay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240816040030.GA16986@neeraj.linux> References: <20240816040030.GA16986@neeraj.linux> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Valentin Schneider The context_tracking.state RCU_DYNTICKS subvariable has been renamed to RCU_WATCHING, reflect that change in the related helpers. While at it, update a comment that still refers to rcu_dynticks_snap(), which was removed by commit: 7be2e6323b9b ("rcu: Remove full memory barrier on RCU stall printout") Signed-off-by: Valentin Schneider Reviewed-by: Frederic Weisbecker Signed-off-by: Neeraj Upadhyay --- kernel/rcu/tree.c | 8 ++++---- kernel/rcu/tree_exp.h | 2 +- kernel/rcu/tree_stall.h | 2 +- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 47ec9f2daaed..73516d76b70a 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -300,10 +300,10 @@ static void rcu_watching_online(void) } /* - * Return true if the snapshot returned from rcu_dynticks_snap() + * Return true if the snapshot returned from ct_rcu_watching() * indicates that RCU is in an extended quiescent state. */ -static bool rcu_dynticks_in_eqs(int snap) +static bool rcu_watching_snap_in_eqs(int snap) { return !(snap & CT_RCU_WATCHING); } @@ -783,7 +783,7 @@ static int dyntick_save_progress_counter(struct rcu_data *rdp) * updater's accesses is enforced by the below acquire semantic. */ rdp->dynticks_snap = ct_rcu_watching_cpu_acquire(rdp->cpu); - if (rcu_dynticks_in_eqs(rdp->dynticks_snap)) { + if (rcu_watching_snap_in_eqs(rdp->dynticks_snap)) { trace_rcu_fqs(rcu_state.name, rdp->gp_seq, rdp->cpu, TPS("dti")); rcu_gpnum_ovf(rdp->mynode, rdp); return 1; @@ -4805,7 +4805,7 @@ rcu_boot_init_percpu_data(int cpu) rdp->grpmask = leaf_node_cpu_bit(rdp->mynode, cpu); INIT_WORK(&rdp->strict_work, strict_work_handler); WARN_ON_ONCE(ct->nesting != 1); - WARN_ON_ONCE(rcu_dynticks_in_eqs(ct_rcu_watching_cpu(cpu))); + WARN_ON_ONCE(rcu_watching_snap_in_eqs(ct_rcu_watching_cpu(cpu))); rdp->barrier_seq_snap = rcu_state.barrier_sequence; rdp->rcu_ofl_gp_seq = rcu_state.gp_seq; rdp->rcu_ofl_gp_state = RCU_GP_CLEANED; diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h index daa87fec703f..137559c84309 100644 --- a/kernel/rcu/tree_exp.h +++ b/kernel/rcu/tree_exp.h @@ -377,7 +377,7 @@ static void __sync_rcu_exp_select_node_cpus(struct rcu_exp_work *rewp) * below acquire semantic. */ snap = ct_rcu_watching_cpu_acquire(cpu); - if (rcu_dynticks_in_eqs(snap)) + if (rcu_watching_snap_in_eqs(snap)) mask_ofl_test |= mask; else rdp->exp_dynticks_snap = snap; diff --git a/kernel/rcu/tree_stall.h b/kernel/rcu/tree_stall.h index ec49f0155bec..e933c6a58d5f 100644 --- a/kernel/rcu/tree_stall.h +++ b/kernel/rcu/tree_stall.h @@ -501,7 +501,7 @@ static void print_cpu_stall_info(int cpu) } delta = rcu_seq_ctr(rdp->mynode->gp_seq - rdp->rcu_iw_gp_seq); falsepositive = rcu_is_gp_kthread_starving(NULL) && - rcu_dynticks_in_eqs(ct_rcu_watching_cpu(cpu)); + rcu_watching_snap_in_eqs(ct_rcu_watching_cpu(cpu)); rcuc_starved = rcu_is_rcuc_kthread_starving(rdp, &j); if (rcuc_starved) // Print signed value, as negative values indicate a probable bug. From patchwork Fri Aug 16 04:02:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neeraj Upadhyay X-Patchwork-Id: 13765410 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C615A12BF32; Fri, 16 Aug 2024 04:05:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723781145; cv=none; b=uldbGewNbpMcc1EZlUwsZVqqq3PwIFzvo+0dKGSTn8zMkFepGGDtlJG6Jcc1/fGPTF3xVPPXnD/spmeeLoy+arHFYfoYzo0X7hcuXKn3B7T9+bSdywzz4dP66tRflspaogP1j43otz7UqFq/1y5NUHQsKIGdTXgL5pe7SFXS1qU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723781145; c=relaxed/simple; bh=hjaBXcJ9dBRYxycecncIBwBcXltyNWJ49cx74mvOUCY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=NtMhxj/aN5zdeY7M4EeC4bgTkZMTkYxtbCJN3KdgZCtuVDqZ8YynNzH6WudGlLPifj1quzEGVLR7OgBUcachfeRIJVciA5Cxa+mALBW5i/Kk3xjYwx1Ogib5Tzh9518D0mg5InKN+QoL8miyPG3daFlXzKd1n6DmiiYeo5AK1nw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=VXqMY7hq; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="VXqMY7hq" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8CFD4C4AF09; Fri, 16 Aug 2024 04:05:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723781144; bh=hjaBXcJ9dBRYxycecncIBwBcXltyNWJ49cx74mvOUCY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=VXqMY7hqy44coyaWMN90bF3Q+JmvLt4aHS6542BsIURCbNe2q0xoOmt9tFM8wAQhz tkNRt/S+/LCa0JVgCkIH7blcUKBW8yOW0TmxhiTBnPZ+eiTe85z1PovOmpCv3BREBZ TLL2w/UiRC4x78H6lqSjvqyA+HU/2TBXySL53a+Ep4XjW/bhFNwUXSN0Iqz+jJIVTP gplne0YhFQIOjxKUDpTqJlSw+A4lfmCvwRmX7mwCY/AWqcMY1sSfARCgk8S0Jc3SAF gUl1EKHY1GDlGdbpDrDSto9V2ewdN3GcsmppaujhLu1UywnACI3cunzSeYO2tNcMhs JmwYEOMIvJKqQ== From: neeraj.upadhyay@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, paulmck@kernel.org, neeraj.upadhyay@kernel.org, neeraj.upadhyay@amd.com, boqun.feng@gmail.com, joel@joelfernandes.org, urezki@gmail.com, frederic@kernel.org, Valentin Schneider Subject: [PATCH rcu 17/26] rcu: Rename rcu_dynticks_in_eqs_since() into rcu_watching_snap_stopped_since() Date: Fri, 16 Aug 2024 09:32:32 +0530 Message-Id: <20240816040241.17776-17-neeraj.upadhyay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240816040030.GA16986@neeraj.linux> References: <20240816040030.GA16986@neeraj.linux> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Valentin Schneider The context_tracking.state RCU_DYNTICKS subvariable has been renamed to RCU_WATCHING, the dynticks prefix can go. While at it, this helper is only meant to be called after failing an earlier call to rcu_watching_snap_in_eqs(), document this in the comments and add a WARN_ON_ONCE() for good measure. Signed-off-by: Valentin Schneider Reviewed-by: Frederic Weisbecker Signed-off-by: Neeraj Upadhyay --- .../Tree-RCU-Memory-Ordering.rst | 2 +- kernel/rcu/tree.c | 23 ++++++++++++++----- kernel/rcu/tree_exp.h | 2 +- 3 files changed, 19 insertions(+), 8 deletions(-) diff --git a/Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst b/Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst index 2d7036ad7476..7163d0def34e 100644 --- a/Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst +++ b/Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst @@ -150,7 +150,7 @@ This case is handled by calls to the strongly ordered is invoked within ``rcu_dynticks_eqs_enter()`` at idle-entry time and within ``rcu_dynticks_eqs_exit()`` at idle-exit time. The grace-period kthread invokes first ``ct_rcu_watching_cpu_acquire()`` -(preceded by a full memory barrier) and ``rcu_dynticks_in_eqs_since()`` +(preceded by a full memory barrier) and ``rcu_watching_snap_stopped_since()`` (both of which rely on acquire semantics) to detect idle CPUs. +-----------------------------------------------------------------------+ diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 73516d76b70a..93570cb5e99e 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -308,12 +308,20 @@ static bool rcu_watching_snap_in_eqs(int snap) return !(snap & CT_RCU_WATCHING); } -/* - * Return true if the CPU corresponding to the specified rcu_data - * structure has spent some time in an extended quiescent state since - * rcu_dynticks_snap() returned the specified snapshot. +/** + * rcu_watching_snap_stopped_since() - Has RCU stopped watching a given CPU + * since the specified @snap? + * + * @rdp: The rcu_data corresponding to the CPU for which to check EQS. + * @snap: rcu_watching snapshot taken when the CPU wasn't in an EQS. + * + * Returns true if the CPU corresponding to @rdp has spent some time in an + * extended quiescent state since @snap. Note that this doesn't check if it + * /still/ is in an EQS, just that it went through one since @snap. + * + * This is meant to be used in a loop waiting for a CPU to go through an EQS. */ -static bool rcu_dynticks_in_eqs_since(struct rcu_data *rdp, int snap) +static bool rcu_watching_snap_stopped_since(struct rcu_data *rdp, int snap) { /* * The first failing snapshot is already ordered against the accesses @@ -323,6 +331,9 @@ static bool rcu_dynticks_in_eqs_since(struct rcu_data *rdp, int snap) * performed by the remote CPU prior to entering idle and therefore can * rely solely on acquire semantics. */ + if (WARN_ON_ONCE(rcu_watching_snap_in_eqs(snap))) + return true; + return snap != ct_rcu_watching_cpu_acquire(rdp->cpu); } @@ -815,7 +826,7 @@ static int rcu_implicit_dynticks_qs(struct rcu_data *rdp) * read-side critical section that started before the beginning * of the current RCU grace period. */ - if (rcu_dynticks_in_eqs_since(rdp, rdp->dynticks_snap)) { + if (rcu_watching_snap_stopped_since(rdp, rdp->dynticks_snap)) { trace_rcu_fqs(rcu_state.name, rdp->gp_seq, rdp->cpu, TPS("dti")); rcu_gpnum_ovf(rnp, rdp); return 1; diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h index 137559c84309..48ad8b868d83 100644 --- a/kernel/rcu/tree_exp.h +++ b/kernel/rcu/tree_exp.h @@ -400,7 +400,7 @@ static void __sync_rcu_exp_select_node_cpus(struct rcu_exp_work *rewp) unsigned long mask = rdp->grpmask; retry_ipi: - if (rcu_dynticks_in_eqs_since(rdp, rdp->exp_dynticks_snap)) { + if (rcu_watching_snap_stopped_since(rdp, rdp->exp_dynticks_snap)) { mask_ofl_test |= mask; continue; } From patchwork Fri Aug 16 04:02:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neeraj Upadhyay X-Patchwork-Id: 13765411 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5CF5112FF7B; Fri, 16 Aug 2024 04:05:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723781152; cv=none; b=ZEKm+8ssTlKILxD15rHmpbejY70VM7yzpKKUTs4/TTGxPaLv/md3zr6NWUSrWB0dQss1hJoJk9Gnw/lyoOQfhfMSmpbKQ728IBxt0gDH7uP5PHHe0gC6Mt4qnplqrD8B4N+u6KtXJE7aBcsPfPtYmHuHMy4hwbjN7b0cI97ne8s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723781152; c=relaxed/simple; bh=Z1poKp0DfIzf8G/KTV53Tc5rMLzCgOhwSFtPjx4JNGM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=hIU5A7cwUmHl7TZecQJHIhQ+u29MNQa5NRp7eE4H9xY6XZ71bizCkyI9f/MAHqrwzj+wpy28S8dZDW0ThXuGPJ9Dv3wuqFm0+P2e/l0hNhTUmQETJrLdBuX8ztY4GJm8uh5QMPqohEvVAr/yT70obVqqoH7GCQ9ES5S91PivUXk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=PEONPJAe; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="PEONPJAe" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 866FDC32782; Fri, 16 Aug 2024 04:05:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723781152; bh=Z1poKp0DfIzf8G/KTV53Tc5rMLzCgOhwSFtPjx4JNGM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=PEONPJAeuBPIrRZ3alUOLqIl1FMmpUbyv7tQfuBXPOCmAihA5IiJ833gkTjgqMGQL NB79kyWSBJoBGpms+3YzOHMqg7Vc8DBV1OQxXR6AowP+KWt7TUhsSJdRL1QWwmGjZ6 XvzRDeboc7+kGT/N9OkXbOioyTucSZZf4TqoVgLaVB+9EEgdTkS1Xf06VmFHNvaYKh IftRe2irirvAbmOnreGbkmVNLOOlQe61Ckn/CywdEWsjvoyF6otNrbnB4FoAXrYs62 EVHwquLdKboV8gMWF3z6Se2URFis/A7063lb3Y96I0Hibghl0QgVMwTfxfV63f+4hk WskZ9wwZi0rrQ== From: neeraj.upadhyay@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, paulmck@kernel.org, neeraj.upadhyay@kernel.org, neeraj.upadhyay@amd.com, boqun.feng@gmail.com, joel@joelfernandes.org, urezki@gmail.com, frederic@kernel.org, Valentin Schneider Subject: [PATCH rcu 18/26] rcu: Rename rcu_dynticks_zero_in_eqs() into rcu_watching_zero_in_eqs() Date: Fri, 16 Aug 2024 09:32:33 +0530 Message-Id: <20240816040241.17776-18-neeraj.upadhyay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240816040030.GA16986@neeraj.linux> References: <20240816040030.GA16986@neeraj.linux> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Valentin Schneider The context_tracking.state RCU_DYNTICKS subvariable has been renamed to RCU_WATCHING, reflect that change in the related helpers. Signed-off-by: Valentin Schneider Reviewed-by: Frederic Weisbecker Signed-off-by: Neeraj Upadhyay --- kernel/rcu/rcu.h | 4 ++-- kernel/rcu/tasks.h | 2 +- kernel/rcu/tree.c | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/kernel/rcu/rcu.h b/kernel/rcu/rcu.h index 38238e595a61..5564402af4cb 100644 --- a/kernel/rcu/rcu.h +++ b/kernel/rcu/rcu.h @@ -606,7 +606,7 @@ void srcutorture_get_gp_data(struct srcu_struct *sp, int *flags, #endif #ifdef CONFIG_TINY_RCU -static inline bool rcu_dynticks_zero_in_eqs(int cpu, int *vp) { return false; } +static inline bool rcu_watching_zero_in_eqs(int cpu, int *vp) { return false; } static inline unsigned long rcu_get_gp_seq(void) { return 0; } static inline unsigned long rcu_exp_batches_completed(void) { return 0; } static inline unsigned long @@ -619,7 +619,7 @@ static inline void rcu_fwd_progress_check(unsigned long j) { } static inline void rcu_gp_slow_register(atomic_t *rgssp) { } static inline void rcu_gp_slow_unregister(atomic_t *rgssp) { } #else /* #ifdef CONFIG_TINY_RCU */ -bool rcu_dynticks_zero_in_eqs(int cpu, int *vp); +bool rcu_watching_zero_in_eqs(int cpu, int *vp); unsigned long rcu_get_gp_seq(void); unsigned long rcu_exp_batches_completed(void); unsigned long srcu_batches_completed(struct srcu_struct *sp); diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index ba3440a45b6d..2484c1a8e051 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -1613,7 +1613,7 @@ static int trc_inspect_reader(struct task_struct *t, void *bhp_in) // However, we cannot safely change its state. n_heavy_reader_attempts++; // Check for "running" idle tasks on offline CPUs. - if (!rcu_dynticks_zero_in_eqs(cpu, &t->trc_reader_nesting)) + if (!rcu_watching_zero_in_eqs(cpu, &t->trc_reader_nesting)) return -EINVAL; // No quiescent state, do it the hard way. n_heavy_reader_updates++; nesting = 0; diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 93570cb5e99e..ac3577ac10bd 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -341,7 +341,7 @@ static bool rcu_watching_snap_stopped_since(struct rcu_data *rdp, int snap) * Return true if the referenced integer is zero while the specified * CPU remains within a single extended quiescent state. */ -bool rcu_dynticks_zero_in_eqs(int cpu, int *vp) +bool rcu_watching_zero_in_eqs(int cpu, int *vp) { int snap; From patchwork Fri Aug 16 04:02:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neeraj Upadhyay X-Patchwork-Id: 13765412 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 713F97BB01; Fri, 16 Aug 2024 04:06:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723781161; cv=none; b=lGZvX/+zvqJEwztfUjotFOlNIGzj4O8cMkr+Mm9ZNhfuUBFwHDbi7sOqAMKg78GDY9meFJGcOzlSOYzUmcFWX5VSYQG/Psybv67YkMiFOTmAr0udwXYb62DqsRDHQ+npwJ8itDvgnqSBewrsgTFE4iD22jNdOr93mj4CULsNSNc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723781161; c=relaxed/simple; bh=eezzsYtU0JEgacHwaXnRZKwm2QxEqloqluBFnlXLfZw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=d1ULUtNDSbMtlUrRpZ2YevDcWUQmRLu66OtGxTM04aLEQ8kQclsElyPg6163Lr1VGYSX9JTN2K7O0c5zLCIqgtoiB7quRFLePyhGiGPsTRK5xRqfUvoaPbFWT1cz3nI//KYZRQLYgeiZZtShFL3ah0AX8vgf10svA1/U43QYJ6c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=ryEuqN1b; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="ryEuqN1b" Received: by smtp.kernel.org (Postfix) with ESMTPSA id EE434C32782; Fri, 16 Aug 2024 04:05:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723781161; bh=eezzsYtU0JEgacHwaXnRZKwm2QxEqloqluBFnlXLfZw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ryEuqN1boUtjAxV2kmP11XcJQ0NX4LcJFiAEIKO5N075HWyEBhbx2VdTPXbgCKDSI aGu0VNDKGbawUCbgtGoTg+nWrONg2MXF2sb8njL66hjK8xblcGv/17PF06c8sceqml XoxlOLkOmtX3+aa2WZrtmdzkv2GpitxKxWSCPz+UH/PsXiP3xAs1T9yGwi60lOMaXQ r+koiGRJAhiu74ueQrk0iYws8el41a+RQhNS3Da2Bx5bpFIZnScld9Gflk3hifS0mt 5AcsXWuw4XRJz1Ktypgn+5/DZ2jFsf4f/xa2pQKSFOjBKqbwumTsRm0h4oU6fGcCRj oaDUpeijjca6Q== From: neeraj.upadhyay@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, paulmck@kernel.org, neeraj.upadhyay@kernel.org, neeraj.upadhyay@amd.com, boqun.feng@gmail.com, joel@joelfernandes.org, urezki@gmail.com, frederic@kernel.org, Valentin Schneider Subject: [PATCH rcu 19/26] rcu: Rename struct rcu_data .dynticks_snap into .watching_snap Date: Fri, 16 Aug 2024 09:32:34 +0530 Message-Id: <20240816040241.17776-19-neeraj.upadhyay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240816040030.GA16986@neeraj.linux> References: <20240816040030.GA16986@neeraj.linux> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Valentin Schneider The context_tracking.state RCU_DYNTICKS subvariable has been renamed to RCU_WATCHING, and the snapshot helpers are now prefix by "rcu_watching". Reflect that change into the storage variables for these snapshots. Signed-off-by: Valentin Schneider Reviewed-by: Frederic Weisbecker Signed-off-by: Neeraj Upadhyay --- .../RCU/Design/Data-Structures/Data-Structures.rst | 4 ++-- kernel/rcu/tree.c | 6 +++--- kernel/rcu/tree.h | 2 +- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/Documentation/RCU/Design/Data-Structures/Data-Structures.rst b/Documentation/RCU/Design/Data-Structures/Data-Structures.rst index 28d9f9e1783f..04e16775c752 100644 --- a/Documentation/RCU/Design/Data-Structures/Data-Structures.rst +++ b/Documentation/RCU/Design/Data-Structures/Data-Structures.rst @@ -921,10 +921,10 @@ This portion of the ``rcu_data`` structure is declared as follows: :: - 1 int dynticks_snap; + 1 int watching_snap; 2 unsigned long dynticks_fqs; -The ``->dynticks_snap`` field is used to take a snapshot of the +The ``->watching_snap`` field is used to take a snapshot of the corresponding CPU's dyntick-idle state when forcing quiescent states, and is therefore accessed from other CPUs. Finally, the ``->dynticks_fqs`` field is used to count the number of times this CPU diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index ac3577ac10bd..a969d8ddc58e 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -793,8 +793,8 @@ static int dyntick_save_progress_counter(struct rcu_data *rdp) * Ordering between remote CPU's pre idle accesses and post grace period * updater's accesses is enforced by the below acquire semantic. */ - rdp->dynticks_snap = ct_rcu_watching_cpu_acquire(rdp->cpu); - if (rcu_watching_snap_in_eqs(rdp->dynticks_snap)) { + rdp->watching_snap = ct_rcu_watching_cpu_acquire(rdp->cpu); + if (rcu_watching_snap_in_eqs(rdp->watching_snap)) { trace_rcu_fqs(rcu_state.name, rdp->gp_seq, rdp->cpu, TPS("dti")); rcu_gpnum_ovf(rdp->mynode, rdp); return 1; @@ -826,7 +826,7 @@ static int rcu_implicit_dynticks_qs(struct rcu_data *rdp) * read-side critical section that started before the beginning * of the current RCU grace period. */ - if (rcu_watching_snap_stopped_since(rdp, rdp->dynticks_snap)) { + if (rcu_watching_snap_stopped_since(rdp, rdp->watching_snap)) { trace_rcu_fqs(rcu_state.name, rdp->gp_seq, rdp->cpu, TPS("dti")); rcu_gpnum_ovf(rnp, rdp); return 1; diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h index fcf2b4aa3441..f5361a7d7269 100644 --- a/kernel/rcu/tree.h +++ b/kernel/rcu/tree.h @@ -206,7 +206,7 @@ struct rcu_data { long blimit; /* Upper limit on a processed batch */ /* 3) dynticks interface. */ - int dynticks_snap; /* Per-GP tracking for dynticks. */ + int watching_snap; /* Per-GP tracking for dynticks. */ bool rcu_need_heavy_qs; /* GP old, so heavy quiescent state! */ bool rcu_urgent_qs; /* GP old need light quiescent state. */ bool rcu_forced_tick; /* Forced tick to provide QS. */ From patchwork Fri Aug 16 04:02:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neeraj Upadhyay X-Patchwork-Id: 13765413 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 87D478287E; Fri, 16 Aug 2024 04:06:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723781168; cv=none; b=mh182Wxs0Dpafbc+hl2LK205HXt6aEPD2uJ0nx4j1gseSUGv2slDUY83nkln5XoIyq22kj24K87JTFTL3blhQ+ySMDjuCXq017tKQTTewEFdVL7jqAE92yKHuSJ1oYxhOvRZC5IPHA7zZANR2YY0U46I6xNH2rwqR3Vr23pKqxU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723781168; c=relaxed/simple; bh=/alW98fFIxhaforztOaK21XXuT5F3kiWNzbUR3+Uyto=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=cpJ3Mc1AZmn8BZJP/ej6XQZ4xYxxZBRKukx/JTbcwvY3Hn7MrigdxySmkoVcuZvpet45gBOWyAMiR06Ev0RW8vHPbGHN2uJ+xKZu+Vs4U2izAK6USK6MoHwOt1RHW6e08/bg2Clf+RWv/EImS+h/ZB8m3wusgGRxGPdQkJOnGEo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=pEZz02E2; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="pEZz02E2" Received: by smtp.kernel.org (Postfix) with ESMTPSA id A6F61C32782; Fri, 16 Aug 2024 04:06:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723781168; bh=/alW98fFIxhaforztOaK21XXuT5F3kiWNzbUR3+Uyto=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=pEZz02E27heWMWQ1tZ3OA81pbUt700h5ptXZPzMjpt01h16T8Bzr6r77x+cquS0cy gFIrMXkuaFmBjf1/0OZRng3y+Pl7cCcvYuXfG9Q+Y0fxQ+dBB3aYWbZ4MC2FCucdsE YehIXfC57wlbbRq6CDtsBXsHf1IPwkshETl0lWP6abOxfWvK/xC4+LrtPHTV6LIovi fqusDHaDmEhYg92MtVZion5ujKFnh4H9rB8+Ygz01WikBPc0opvUsJCBApM8/NzwvP ZqMdd8cwVFbtspGwlloHc4sil9oDA6TtZYk52XNmINyTwtKUJQn/89ZC95L6VXoKvr 3xEm1EnpYzrfA== From: neeraj.upadhyay@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, paulmck@kernel.org, neeraj.upadhyay@kernel.org, neeraj.upadhyay@amd.com, boqun.feng@gmail.com, joel@joelfernandes.org, urezki@gmail.com, frederic@kernel.org, Valentin Schneider Subject: [PATCH rcu 20/26] rcu: Rename struct rcu_data .exp_dynticks_snap into .exp_watching_snap Date: Fri, 16 Aug 2024 09:32:35 +0530 Message-Id: <20240816040241.17776-20-neeraj.upadhyay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240816040030.GA16986@neeraj.linux> References: <20240816040030.GA16986@neeraj.linux> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Valentin Schneider The context_tracking.state RCU_DYNTICKS subvariable has been renamed to RCU_WATCHING, and the snapshot helpers are now prefix by "rcu_watching". Reflect that change into the storage variables for these snapshots. Signed-off-by: Valentin Schneider Reviewed-by: Frederic Weisbecker Signed-off-by: Neeraj Upadhyay --- kernel/rcu/tree.h | 2 +- kernel/rcu/tree_exp.h | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h index f5361a7d7269..13fdcc2aa081 100644 --- a/kernel/rcu/tree.h +++ b/kernel/rcu/tree.h @@ -215,7 +215,7 @@ struct rcu_data { /* 4) rcu_barrier(), OOM callbacks, and expediting. */ unsigned long barrier_seq_snap; /* Snap of rcu_state.barrier_sequence. */ struct rcu_head barrier_head; - int exp_dynticks_snap; /* Double-check need for IPI. */ + int exp_watching_snap; /* Double-check need for IPI. */ /* 5) Callback offloading. */ #ifdef CONFIG_RCU_NOCB_CPU diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h index 48ad8b868d83..eec42df0c018 100644 --- a/kernel/rcu/tree_exp.h +++ b/kernel/rcu/tree_exp.h @@ -380,7 +380,7 @@ static void __sync_rcu_exp_select_node_cpus(struct rcu_exp_work *rewp) if (rcu_watching_snap_in_eqs(snap)) mask_ofl_test |= mask; else - rdp->exp_dynticks_snap = snap; + rdp->exp_watching_snap = snap; } } mask_ofl_ipi = rnp->expmask & ~mask_ofl_test; @@ -400,7 +400,7 @@ static void __sync_rcu_exp_select_node_cpus(struct rcu_exp_work *rewp) unsigned long mask = rdp->grpmask; retry_ipi: - if (rcu_watching_snap_stopped_since(rdp, rdp->exp_dynticks_snap)) { + if (rcu_watching_snap_stopped_since(rdp, rdp->exp_watching_snap)) { mask_ofl_test |= mask; continue; } From patchwork Fri Aug 16 04:02:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neeraj Upadhyay X-Patchwork-Id: 13765417 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 23AB585260; Fri, 16 Aug 2024 04:06:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723781175; cv=none; b=B0PwK57BjZ6hf/3DGfL6bEXEgz71Hqo4giFxLsRhaQmMKndTvzWSQLDil0xHuzVju/aNUprPAyE3jFhH7h7WZIJu71cVuPvfLHgVdcF2pUiIXIn21yjgfeb+OrEICc7UEMmBi8uxPbNudVYtwkF73HITOieOFf2xs8XC700cNA0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723781175; c=relaxed/simple; bh=s/028LW28I+qDCNzGGYVqaLGBDBEMLt7vSM49+sYWOA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=ZX8CtWfK3u2r+iE1GNhJBW2CHtxHAFiEEQYy4zBmN9SYI149LjYbSMW33jENd7Kjb0Q7czRnG9k5z5rGnM30lcfp0h8dCSY/WAiXnp3JXpGdocNlXXgEe5jQK5WjhDVnSra8yXfwqBYognPPmpPCLsSCk7ea+ddQSEtbD2h2KiA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=HvyZn1Rl; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="HvyZn1Rl" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 30F61C4AF0E; Fri, 16 Aug 2024 04:06:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723781174; bh=s/028LW28I+qDCNzGGYVqaLGBDBEMLt7vSM49+sYWOA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=HvyZn1RlsXQ4tgOSELTS3t1dLKYvJH+zFc8TLt4lgJC1L5r7lgPAubAsDzJok6a1f L3QLyXzyd0pYNJdcJsLylwU3lWVtKeJST29mT+FNo+CjjlZKZMDwzyRzWW9TXq8zkE kSYrRhGF2jGTFLGPej7+RTdC667jvMlDrQ9A5miUtpIziZnEILLMzKNUlyFcDIy9L0 e0/EMl2UQLILp/IXJkIa7iETuwhY7ItvAhRWxiCTc9tiIJFAZhSH4kdNOz3fZgBTwy aU2kb0d2LSDk1biwklFWp3Bw0JumV6BSKxl6GQ3ro0z8GFhriuGy2cUIWLZLXDZoGu 4xI+RBUnB232Q== From: neeraj.upadhyay@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, paulmck@kernel.org, neeraj.upadhyay@kernel.org, neeraj.upadhyay@amd.com, boqun.feng@gmail.com, joel@joelfernandes.org, urezki@gmail.com, frederic@kernel.org, Valentin Schneider Subject: [PATCH rcu 21/26] rcu: Rename dyntick_save_progress_counter() into rcu_watching_snap_save() Date: Fri, 16 Aug 2024 09:32:36 +0530 Message-Id: <20240816040241.17776-21-neeraj.upadhyay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240816040030.GA16986@neeraj.linux> References: <20240816040030.GA16986@neeraj.linux> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Valentin Schneider The context_tracking.state RCU_DYNTICKS subvariable has been renamed to RCU_WATCHING, and the 'dynticks' prefix can be dropped without losing any meaning. Suggested-by: Frederic Weisbecker Signed-off-by: Valentin Schneider Reviewed-by: Frederic Weisbecker Signed-off-by: Neeraj Upadhyay --- .../RCU/Design/Memory-Ordering/TreeRCU-dyntick.svg | 2 +- .../RCU/Design/Memory-Ordering/TreeRCU-gp-fqs.svg | 2 +- Documentation/RCU/Design/Memory-Ordering/TreeRCU-gp.svg | 2 +- .../RCU/Design/Memory-Ordering/TreeRCU-hotplug.svg | 2 +- kernel/rcu/tree.c | 8 ++++---- 5 files changed, 8 insertions(+), 8 deletions(-) diff --git a/Documentation/RCU/Design/Memory-Ordering/TreeRCU-dyntick.svg b/Documentation/RCU/Design/Memory-Ordering/TreeRCU-dyntick.svg index 423df00c4df9..13956baf748a 100644 --- a/Documentation/RCU/Design/Memory-Ordering/TreeRCU-dyntick.svg +++ b/Documentation/RCU/Design/Memory-Ordering/TreeRCU-dyntick.svg @@ -528,7 +528,7 @@ font-style="normal" y="-8652.5312" x="2466.7822" - xml:space="preserve">dyntick_save_progress_counter() + xml:space="preserve">rcu_watching_snap_save() dyntick_save_progress_counter() + xml:space="preserve">rcu_watching_snap_save() dyntick_save_progress_counter() + xml:space="preserve">rcu_watching_snap_save() dyntick_save_progress_counter() + xml:space="preserve">rcu_watching_snap_save() X-Patchwork-Id: 13765418 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E5C1654F87; Fri, 16 Aug 2024 04:06:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723781182; cv=none; b=Ef7Ro2irpPkdgjweMxLO4oMPr7uM5ceHVtx7YkaWvBH5DdX3DyBG1A7W4jESzN5xJYaDDyyoA8L67gkwMtwzi6ppWf7n+0jWI+Xlmo74ouLEaokgzkrgjwXhHl+1Bgr5JvZQcDTGocGg+A4tBVfDkEfKujUcfJZHhJX3U99eKgk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723781182; c=relaxed/simple; bh=3WRner0RMhGFgnkPtCcFs9g6yG8s5bs0mSd98/aW7I0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=AwZL4cJlZfDKK46wNK+Ytb5kGkHTN+A/sSyxY4YycxU1R6VHH+Sp+O1NaQUQuYKnEr1gFJ1gnFZ2l2zaQn7SNtLPEvj/nB+0W2kEOCgvuiY/o7+HrRdYdhiat83+9zxWbRtfGDLEih0upObYlMHm5J4xCswRh1/ghCCgxBqMjqo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=ElUiE6sD; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="ElUiE6sD" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7FAC6C4AF0B; Fri, 16 Aug 2024 04:06:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723781181; bh=3WRner0RMhGFgnkPtCcFs9g6yG8s5bs0mSd98/aW7I0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ElUiE6sDSBSDn3OiqGnUJOZsCkEvNjTEjW8SgwSF7qWlUVI4XFjJ0wtUuEmOyUVzf tvdX4UjZY9c6zEOIMzLry76HV4qinYwYhqz0PcwEH0MG5u1U19gzT2cHv9RzteHDk3 RiL1aUeDYa10VcWgmB3XvjvpsF7q+RaTPMp0vYJsJ/b8yKMMuHQBR6x9c9aqcC2cmZ 9DN/BUfTAzNyQT08Wz0QrqVqAHVfD4dAPUqeNEh7s2B7lRkfco8sidi8o1nIUJqLHT kncS+CZjCm+nrjKQctLscR/h7Vs/IMRbLFO8aVoDWhGR9rg/E+robBF2fU6Hp3zEXl yNViFP6kgTISQ== From: neeraj.upadhyay@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, paulmck@kernel.org, neeraj.upadhyay@kernel.org, neeraj.upadhyay@amd.com, boqun.feng@gmail.com, joel@joelfernandes.org, urezki@gmail.com, frederic@kernel.org, Valentin Schneider Subject: [PATCH rcu 22/26] rcu: Rename rcu_implicit_dynticks_qs() into rcu_watching_snap_recheck() Date: Fri, 16 Aug 2024 09:32:37 +0530 Message-Id: <20240816040241.17776-22-neeraj.upadhyay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240816040030.GA16986@neeraj.linux> References: <20240816040030.GA16986@neeraj.linux> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Valentin Schneider The context_tracking.state RCU_DYNTICKS subvariable has been renamed to RCU_WATCHING, drop the dyntick reference and update the name of this helper to express that it rechecks rdp->watching_snap after an earlier rcu_watching_snap_save(). Signed-off-by: Valentin Schneider Reviewed-by: Frederic Weisbecker Signed-off-by: Neeraj Upadhyay --- Documentation/RCU/Design/Memory-Ordering/TreeRCU-dyntick.svg | 2 +- Documentation/RCU/Design/Memory-Ordering/TreeRCU-gp-fqs.svg | 2 +- Documentation/RCU/Design/Memory-Ordering/TreeRCU-gp.svg | 2 +- Documentation/RCU/Design/Memory-Ordering/TreeRCU-hotplug.svg | 2 +- kernel/rcu/tree.c | 4 ++-- 5 files changed, 6 insertions(+), 6 deletions(-) diff --git a/Documentation/RCU/Design/Memory-Ordering/TreeRCU-dyntick.svg b/Documentation/RCU/Design/Memory-Ordering/TreeRCU-dyntick.svg index 13956baf748a..ab9707f04e66 100644 --- a/Documentation/RCU/Design/Memory-Ordering/TreeRCU-dyntick.svg +++ b/Documentation/RCU/Design/Memory-Ordering/TreeRCU-dyntick.svg @@ -537,7 +537,7 @@ font-style="normal" y="-8368.1475" x="2463.3262" - xml:space="preserve">rcu_implicit_dynticks_qs() + xml:space="preserve">rcu_watching_snap_recheck() rcu_implicit_dynticks_qs() + xml:space="preserve">rcu_watching_snap_recheck() rcu_implicit_dynticks_qs() + xml:space="preserve">rcu_watching_snap_recheck() rcu_implicit_dynticks_qs() + xml:space="preserve">rcu_watching_snap_recheck() X-Patchwork-Id: 13765419 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0D2DC13B597; Fri, 16 Aug 2024 04:06:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723781189; cv=none; b=Wwhqvu0l2tihLnTbPecNd7M6K+2nDx/7Rdv/Jeq9nSze9NyHAzpDLVgHaP3ahGnsHCpEdr2NwjMYUXKBypJT05xNXRWRqfgkfrtAcn2lH7tzMFEjaf6bx9vRzF8tZdlKJA+acLGLxybKpuWUodv5z47dz/hIMP86qe/+FpZB/sc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723781189; c=relaxed/simple; bh=1AKpEgn3U4vxGjMUiUyEqACsLBPHdiJlIcWsaU4OHL0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=H+n3UHYUC33xy0sgsAnJzmkiUYvgH8b1MrqTRx0pTn5j+q2U+/nHh5HnF6jau1wkg4ZR08sOGzYzYtCCoa6TeAcGvwOYu78aaF2pbJWeafG+MRv/E+reoSYajGfzLKxVarpzafWqiXmOvV0k/lNWA25s25H7cLelUgufmiYoNPk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=u7cjazo6; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="u7cjazo6" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 65C1DC32782; Fri, 16 Aug 2024 04:06:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723781188; bh=1AKpEgn3U4vxGjMUiUyEqACsLBPHdiJlIcWsaU4OHL0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=u7cjazo6+WgNuDELxbgr+tCuWahDGvpThg0ZKXSOy99VkebuLy72vgRuRIj//Udgy zBUcYIe80FsNRnG1kXvjN8PYzLzF1grXUYVb+xLom6AOV8R2HgWZNaGUdq603c25LP RVomR5JqHuWi395Ewb6kSCmgoGFG/ylz8XLWaSGqPfrwKWXuuDJEOTh93/H4ND+qre ZLOcOK8iCr9vmqvtH95gpLy+hUEx1XOXvIDqIrBYORgVlwhxHc9P0S14le8H2kTjvr +jNYTwbYvO/xwDvFCMq/ByjOqU4flIGel7aBuNTKm0UNQng8QePA/9iRWqUZJIp09a phbUbMDFgcIJg== From: neeraj.upadhyay@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, paulmck@kernel.org, neeraj.upadhyay@kernel.org, neeraj.upadhyay@amd.com, boqun.feng@gmail.com, joel@joelfernandes.org, urezki@gmail.com, frederic@kernel.org, Valentin Schneider Subject: [PATCH rcu 23/26] rcu: Rename rcu_momentary_dyntick_idle() into rcu_momentary_eqs() Date: Fri, 16 Aug 2024 09:32:38 +0530 Message-Id: <20240816040241.17776-23-neeraj.upadhyay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240816040030.GA16986@neeraj.linux> References: <20240816040030.GA16986@neeraj.linux> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Valentin Schneider The context_tracking.state RCU_DYNTICKS subvariable has been renamed to RCU_WATCHING, replace "dyntick_idle" into "eqs" to drop the dyntick reference. Signed-off-by: Valentin Schneider Reviewed-by: Frederic Weisbecker Signed-off-by: Neeraj Upadhyay --- include/linux/rcutiny.h | 2 +- include/linux/rcutree.h | 2 +- kernel/rcu/rcutorture.c | 4 ++-- kernel/rcu/tree.c | 4 ++-- kernel/rcu/tree_nocb.h | 2 +- kernel/rcu/tree_plugin.h | 6 +++--- kernel/stop_machine.c | 2 +- kernel/trace/trace_osnoise.c | 4 ++-- 8 files changed, 13 insertions(+), 13 deletions(-) diff --git a/include/linux/rcutiny.h b/include/linux/rcutiny.h index d9ac7b136aea..cf2b5a188f78 100644 --- a/include/linux/rcutiny.h +++ b/include/linux/rcutiny.h @@ -158,7 +158,7 @@ void rcu_scheduler_starting(void); static inline void rcu_end_inkernel_boot(void) { } static inline bool rcu_inkernel_boot_has_ended(void) { return true; } static inline bool rcu_is_watching(void) { return true; } -static inline void rcu_momentary_dyntick_idle(void) { } +static inline void rcu_momentary_eqs(void) { } static inline void kfree_rcu_scheduler_running(void) { } static inline bool rcu_gp_might_be_stalled(void) { return false; } diff --git a/include/linux/rcutree.h b/include/linux/rcutree.h index 254244202ea9..7dbde2b6f714 100644 --- a/include/linux/rcutree.h +++ b/include/linux/rcutree.h @@ -37,7 +37,7 @@ void synchronize_rcu_expedited(void); void kvfree_call_rcu(struct rcu_head *head, void *ptr); void rcu_barrier(void); -void rcu_momentary_dyntick_idle(void); +void rcu_momentary_eqs(void); void kfree_rcu_scheduler_running(void); bool rcu_gp_might_be_stalled(void); diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c index 08bf7c669dd3..ef757c3e84a2 100644 --- a/kernel/rcu/rcutorture.c +++ b/kernel/rcu/rcutorture.c @@ -2680,7 +2680,7 @@ static unsigned long rcu_torture_fwd_prog_cbfree(struct rcu_fwd *rfp) rcu_torture_fwd_prog_cond_resched(freed); if (tick_nohz_full_enabled()) { local_irq_save(flags); - rcu_momentary_dyntick_idle(); + rcu_momentary_eqs(); local_irq_restore(flags); } } @@ -2830,7 +2830,7 @@ static void rcu_torture_fwd_prog_cr(struct rcu_fwd *rfp) rcu_torture_fwd_prog_cond_resched(n_launders + n_max_cbs); if (tick_nohz_full_enabled()) { local_irq_save(flags); - rcu_momentary_dyntick_idle(); + rcu_momentary_eqs(); local_irq_restore(flags); } } diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index b53b354417ae..e0f6fc272906 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -367,7 +367,7 @@ bool rcu_watching_zero_in_eqs(int cpu, int *vp) * * The caller must have disabled interrupts and must not be idle. */ -notrace void rcu_momentary_dyntick_idle(void) +notrace void rcu_momentary_eqs(void) { int seq; @@ -377,7 +377,7 @@ notrace void rcu_momentary_dyntick_idle(void) WARN_ON_ONCE(!(seq & CT_RCU_WATCHING)); rcu_preempt_deferred_qs(current); } -EXPORT_SYMBOL_GPL(rcu_momentary_dyntick_idle); +EXPORT_SYMBOL_GPL(rcu_momentary_eqs); /** * rcu_is_cpu_rrupt_from_idle - see if 'interrupted' from idle diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 3ce30841119a..e629646ac6f5 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -917,7 +917,7 @@ static void nocb_cb_wait(struct rcu_data *rdp) WARN_ON_ONCE(!rcu_rdp_is_offloaded(rdp)); local_irq_save(flags); - rcu_momentary_dyntick_idle(); + rcu_momentary_eqs(); local_irq_restore(flags); /* * Disable BH to provide the expected environment. Also, when diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h index c569da65b421..354fbf2e411c 100644 --- a/kernel/rcu/tree_plugin.h +++ b/kernel/rcu/tree_plugin.h @@ -869,7 +869,7 @@ static void rcu_qs(void) /* * Register an urgently needed quiescent state. If there is an - * emergency, invoke rcu_momentary_dyntick_idle() to do a heavy-weight + * emergency, invoke rcu_momentary_eqs() to do a heavy-weight * dyntick-idle quiescent state visible to other CPUs, which will in * some cases serve for expedited as well as normal grace periods. * Either way, register a lightweight quiescent state. @@ -889,7 +889,7 @@ void rcu_all_qs(void) this_cpu_write(rcu_data.rcu_urgent_qs, false); if (unlikely(raw_cpu_read(rcu_data.rcu_need_heavy_qs))) { local_irq_save(flags); - rcu_momentary_dyntick_idle(); + rcu_momentary_eqs(); local_irq_restore(flags); } rcu_qs(); @@ -909,7 +909,7 @@ void rcu_note_context_switch(bool preempt) goto out; this_cpu_write(rcu_data.rcu_urgent_qs, false); if (unlikely(raw_cpu_read(rcu_data.rcu_need_heavy_qs))) - rcu_momentary_dyntick_idle(); + rcu_momentary_eqs(); out: rcu_tasks_qs(current, preempt); trace_rcu_utilization(TPS("End context switch")); diff --git a/kernel/stop_machine.c b/kernel/stop_machine.c index cedb17ba158a..da821ce258ea 100644 --- a/kernel/stop_machine.c +++ b/kernel/stop_machine.c @@ -251,7 +251,7 @@ static int multi_cpu_stop(void *data) */ touch_nmi_watchdog(); } - rcu_momentary_dyntick_idle(); + rcu_momentary_eqs(); } while (curstate != MULTI_STOP_EXIT); local_irq_restore(flags); diff --git a/kernel/trace/trace_osnoise.c b/kernel/trace/trace_osnoise.c index 66a871553d4a..ad86b1ba2e43 100644 --- a/kernel/trace/trace_osnoise.c +++ b/kernel/trace/trace_osnoise.c @@ -1535,7 +1535,7 @@ static int run_osnoise(void) * This will eventually cause unwarranted noise as PREEMPT_RCU * will force preemption as the means of ending the current * grace period. We avoid this problem by calling - * rcu_momentary_dyntick_idle(), which performs a zero duration + * rcu_momentary_eqs(), which performs a zero duration * EQS allowing PREEMPT_RCU to end the current grace period. * This call shouldn't be wrapped inside an RCU critical * section. @@ -1547,7 +1547,7 @@ static int run_osnoise(void) if (!disable_irq) local_irq_disable(); - rcu_momentary_dyntick_idle(); + rcu_momentary_eqs(); if (!disable_irq) local_irq_enable(); From patchwork Fri Aug 16 04:02:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neeraj Upadhyay X-Patchwork-Id: 13765420 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C1D64770E5; Fri, 16 Aug 2024 04:06:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723781195; cv=none; b=TifAFhtSFdErSKLDu2QZNUKYDDK8cV/FApSI9yM1+atGZ3P7Sq0869mTCepNKQkpe//Rcfq7cs/t1eJ4UuCobOGFVq56FBuDVBPVr/zzBXstT48nrZU+sXxYN28cErVPhh8xgT6d4El+1SV4Zpo0lVCvNYIeW7d3D2nxKXQ2JlU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723781195; c=relaxed/simple; bh=jrkMkNE9TDHxapA8A5E3kTJWW0PhzMSOdacp5FUaQG4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=D18Dnr6IUlg2oG72Ig1J2glW8wHlTlUsdXDjdz0IGEGn1Z7qdsdsXk2BTpUVgx2eatCrtBJKLVTH23162jpeVNAwi0YWrn1sDXAdLmnSotfz7GH1pSwA0Msb1xdifp4stl9puwTEJ10sx0xqWEbT1ohg+PrEXtgQPDUeazxRe+s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=CVafHjHk; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="CVafHjHk" Received: by smtp.kernel.org (Postfix) with ESMTPSA id C113DC4AF17; Fri, 16 Aug 2024 04:06:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723781195; bh=jrkMkNE9TDHxapA8A5E3kTJWW0PhzMSOdacp5FUaQG4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=CVafHjHksmZRa3uu1DsC3DJLqr2+aGhDVEeORS4r8r2j9qRQzmcRw8T93annbTZIa GSBlcOzBJJ/r087eIxzJtR+8ixwKHG8t/JM9FvL1Rd1swiKkAQPs8BsZVgGAsesnNb yQ3f46qGfBJmB2+1HqABZnWZU/PGzuZ/Jki/SIDx9ntV7oc+s4dImoF8eMn8NIsSEV 6hbISrRUJ59l+sf2lEto3I+98WOqz3PGNFclSipxT/3bQ12mEeEWE5C9PMIIgNMbtO rfC4LcBNbIRhU7GfabKittFU9pZwSmfMbFIAlAkEYjyxouxXb4P2MGUu8NONJE97Rf SWkqhXNH41Jng== From: neeraj.upadhyay@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, paulmck@kernel.org, neeraj.upadhyay@kernel.org, neeraj.upadhyay@amd.com, boqun.feng@gmail.com, joel@joelfernandes.org, urezki@gmail.com, frederic@kernel.org, Valentin Schneider Subject: [PATCH rcu 24/26] rcu: Update stray documentation references to rcu_dynticks_eqs_{enter, exit}() Date: Fri, 16 Aug 2024 09:32:39 +0530 Message-Id: <20240816040241.17776-24-neeraj.upadhyay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240816040030.GA16986@neeraj.linux> References: <20240816040030.GA16986@neeraj.linux> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Valentin Schneider rcu_dynticks_eqs_{enter, exit}() have been replaced by their context-tracking counterparts since commit: 171476775d32 ("context_tracking: Convert state to atomic_t") Update the stray documentation references. Signed-off-by: Valentin Schneider Reviewed-by: Frederic Weisbecker Signed-off-by: Neeraj Upadhyay --- .../RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst | 4 ++-- Documentation/RCU/Design/Memory-Ordering/TreeRCU-dyntick.svg | 4 ++-- Documentation/RCU/Design/Memory-Ordering/TreeRCU-gp-fqs.svg | 4 ++-- Documentation/RCU/Design/Memory-Ordering/TreeRCU-gp.svg | 4 ++-- 4 files changed, 8 insertions(+), 8 deletions(-) diff --git a/Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst b/Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst index 7163d0def34e..1a5ff1a9f02e 100644 --- a/Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst +++ b/Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst @@ -147,8 +147,8 @@ RCU read-side critical sections preceding and following the current idle sojourn. This case is handled by calls to the strongly ordered ``atomic_add_return()`` read-modify-write atomic operation that -is invoked within ``rcu_dynticks_eqs_enter()`` at idle-entry -time and within ``rcu_dynticks_eqs_exit()`` at idle-exit time. +is invoked within ``ct_kernel_exit_state()`` at idle-entry +time and within ``ct_kernel_enter_state()`` at idle-exit time. The grace-period kthread invokes first ``ct_rcu_watching_cpu_acquire()`` (preceded by a full memory barrier) and ``rcu_watching_snap_stopped_since()`` (both of which rely on acquire semantics) to detect idle CPUs. diff --git a/Documentation/RCU/Design/Memory-Ordering/TreeRCU-dyntick.svg b/Documentation/RCU/Design/Memory-Ordering/TreeRCU-dyntick.svg index ab9707f04e66..3fbc19c48a58 100644 --- a/Documentation/RCU/Design/Memory-Ordering/TreeRCU-dyntick.svg +++ b/Documentation/RCU/Design/Memory-Ordering/TreeRCU-dyntick.svg @@ -607,7 +607,7 @@ font-weight="bold" font-size="192" id="text202-7-5-3-27-6" - style="font-size:192px;font-style:normal;font-weight:bold;text-anchor:start;fill:#000000;stroke-width:0.025in;font-family:Courier">rcu_dynticks_eqs_enter() + style="font-size:192px;font-style:normal;font-weight:bold;text-anchor:start;fill:#000000;stroke-width:0.025in;font-family:Courier">ct_kernel_exit_state() rcu_dynticks_eqs_exit() + style="font-size:192px;font-style:normal;font-weight:bold;text-anchor:start;fill:#000000;stroke-width:0.025in;font-family:Courier">ct_kernel_enter_state() rcu_dynticks_eqs_enter() + style="font-size:192px;font-style:normal;font-weight:bold;text-anchor:start;fill:#000000;stroke-width:0.025in;font-family:Courier">ct_kernel_exit_state() rcu_dynticks_eqs_exit() + style="font-size:192px;font-style:normal;font-weight:bold;text-anchor:start;fill:#000000;stroke-width:0.025in;font-family:Courier">ct_kernel_enter_state() rcu_dynticks_eqs_enter() + style="font-size:192px;font-style:normal;font-weight:bold;text-anchor:start;fill:#000000;stroke-width:0.025in;font-family:Courier">ct_kernel_exit_state() rcu_dynticks_eqs_exit() + style="font-size:192px;font-style:normal;font-weight:bold;text-anchor:start;fill:#000000;stroke-width:0.025in;font-family:Courier">ct_kernel_enter_state() X-Patchwork-Id: 13765421 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5B32113F012; Fri, 16 Aug 2024 04:06:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723781202; cv=none; b=BzBJVQoruPShfq4ik4F6phl4t41Q1xumVR3XrjV2/lPCS1De7BpYHXHL1RKnwD5hEDbKBm8neCXEGKKQlZREGYVxEnykaf49J3aFX36fvzUbeH6rTaq6ac9l0/K/NBZVH3j1eV6m37DPE5jvDeMmr+zXmN7J/N5cUh4zYEHiTlk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723781202; c=relaxed/simple; bh=rGaOumaoYMNca3yrX/V+ugm7iILw3Vxku/c6CrZHQjk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=iHsaVgX9BKAWEnGs+Z56lxBjWz2DDzRIoh1svPejuBQgs9bduRIqTGFhFZ/WDPSd1Y/Clj9FaLiuGklFITbpetBlZ2UgxPpBYR35PQeXdWBvQPzCAA5Ho2fb3mKEdAVKkmTqX1uz6Sb92aFud/rZ/K17oJgMYeD2G/CJ0b2piJI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=ckI0wkv0; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="ckI0wkv0" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2017FC32782; Fri, 16 Aug 2024 04:06:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723781201; bh=rGaOumaoYMNca3yrX/V+ugm7iILw3Vxku/c6CrZHQjk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ckI0wkv0XxWhUGNFY5JA9IcQ7XRNzJkDUvmnLavF/or4BzW3Q78EzSPHAcfVnFxOV RnDZFX3cydsVQThpimTn5dF+y6I2/mPL9aY4iL9GiNq8NOmdYGHvUw6DNNg+8V3BDI kfdtl3jUv9eN0BdBdSwonPvgVoDLixYQQDDGx2Hefht7jHKchdk0QJ6tEfP6y1AORZ JIPgg7KFAjWhd+TNZETi/hcIdDxChFh+8vesreRK8qNOvUJ57EuLvBi+38aqsihYW4 aE7Lq2psSXoL8BYaoc3IhmoQabcjqHjot7ueXfLzmaJZHx/tX2JYg0zQ1G4Eeuge3b Q7BuRbNvcBJgQ== From: neeraj.upadhyay@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, paulmck@kernel.org, neeraj.upadhyay@kernel.org, neeraj.upadhyay@amd.com, boqun.feng@gmail.com, joel@joelfernandes.org, urezki@gmail.com, frederic@kernel.org, Valentin Schneider Subject: [PATCH rcu 25/26] context_tracking, rcu: Rename rcu_dyntick trace event into rcu_watching Date: Fri, 16 Aug 2024 09:32:40 +0530 Message-Id: <20240816040241.17776-25-neeraj.upadhyay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240816040030.GA16986@neeraj.linux> References: <20240816040030.GA16986@neeraj.linux> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Valentin Schneider The "rcu_dyntick" naming convention has been turned into "rcu_watching" for all helpers now, align the trace event to that. To add to the confusion, the strings passed to the trace event are now reversed: when RCU "starts" the dyntick / EQS state, it "stops" watching. Signed-off-by: Valentin Schneider Reviewed-by: Frederic Weisbecker Signed-off-by: Neeraj Upadhyay --- include/trace/events/rcu.h | 18 +++++++++--------- kernel/context_tracking.c | 10 +++++----- 2 files changed, 14 insertions(+), 14 deletions(-) diff --git a/include/trace/events/rcu.h b/include/trace/events/rcu.h index 4066b6d51e46..e81431deaa50 100644 --- a/include/trace/events/rcu.h +++ b/include/trace/events/rcu.h @@ -466,40 +466,40 @@ TRACE_EVENT(rcu_stall_warning, /* * Tracepoint for dyntick-idle entry/exit events. These take 2 strings * as argument: - * polarity: "Start", "End", "StillNonIdle" for entering, exiting or still not - * being in dyntick-idle mode. + * polarity: "Start", "End", "StillWatching" for entering, exiting or still not + * being in EQS mode. * context: "USER" or "IDLE" or "IRQ". * NMIs nested in IRQs are inferred with nesting > 1 in IRQ context. * * These events also take a pair of numbers, which indicate the nesting * depth before and after the event of interest, and a third number that is - * the ->dynticks counter. Note that task-related and interrupt-related + * the RCU_WATCHING counter. Note that task-related and interrupt-related * events use two separate counters, and that the "++=" and "--=" events * for irq/NMI will change the counter by two, otherwise by one. */ -TRACE_EVENT_RCU(rcu_dyntick, +TRACE_EVENT_RCU(rcu_watching, - TP_PROTO(const char *polarity, long oldnesting, long newnesting, int dynticks), + TP_PROTO(const char *polarity, long oldnesting, long newnesting, int counter), - TP_ARGS(polarity, oldnesting, newnesting, dynticks), + TP_ARGS(polarity, oldnesting, newnesting, counter), TP_STRUCT__entry( __field(const char *, polarity) __field(long, oldnesting) __field(long, newnesting) - __field(int, dynticks) + __field(int, counter) ), TP_fast_assign( __entry->polarity = polarity; __entry->oldnesting = oldnesting; __entry->newnesting = newnesting; - __entry->dynticks = dynticks; + __entry->counter = counter; ), TP_printk("%s %lx %lx %#3x", __entry->polarity, __entry->oldnesting, __entry->newnesting, - __entry->dynticks & 0xfff) + __entry->counter & 0xfff) ); /* diff --git a/kernel/context_tracking.c b/kernel/context_tracking.c index 31c6d0c1b79d..938c48952d26 100644 --- a/kernel/context_tracking.c +++ b/kernel/context_tracking.c @@ -137,7 +137,7 @@ static void noinstr ct_kernel_exit(bool user, int offset) instrumentation_begin(); lockdep_assert_irqs_disabled(); - trace_rcu_dyntick(TPS("Start"), ct_nesting(), 0, ct_rcu_watching()); + trace_rcu_watching(TPS("End"), ct_nesting(), 0, ct_rcu_watching()); WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !user && !is_idle_task(current)); rcu_preempt_deferred_qs(current); @@ -182,7 +182,7 @@ static void noinstr ct_kernel_enter(bool user, int offset) // instrumentation for the noinstr ct_kernel_enter_state() instrument_atomic_write(&ct->state, sizeof(ct->state)); - trace_rcu_dyntick(TPS("End"), ct_nesting(), 1, ct_rcu_watching()); + trace_rcu_watching(TPS("Start"), ct_nesting(), 1, ct_rcu_watching()); WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !user && !is_idle_task(current)); WRITE_ONCE(ct->nesting, 1); WARN_ON_ONCE(ct_nmi_nesting()); @@ -219,7 +219,7 @@ void noinstr ct_nmi_exit(void) * leave it in non-RCU-idle state. */ if (ct_nmi_nesting() != 1) { - trace_rcu_dyntick(TPS("--="), ct_nmi_nesting(), ct_nmi_nesting() - 2, + trace_rcu_watching(TPS("--="), ct_nmi_nesting(), ct_nmi_nesting() - 2, ct_rcu_watching()); WRITE_ONCE(ct->nmi_nesting, /* No store tearing. */ ct_nmi_nesting() - 2); @@ -228,7 +228,7 @@ void noinstr ct_nmi_exit(void) } /* This NMI interrupted an RCU-idle CPU, restore RCU-idleness. */ - trace_rcu_dyntick(TPS("Startirq"), ct_nmi_nesting(), 0, ct_rcu_watching()); + trace_rcu_watching(TPS("Endirq"), ct_nmi_nesting(), 0, ct_rcu_watching()); WRITE_ONCE(ct->nmi_nesting, 0); /* Avoid store tearing. */ // instrumentation for the noinstr ct_kernel_exit_state() @@ -294,7 +294,7 @@ void noinstr ct_nmi_enter(void) instrumentation_begin(); } - trace_rcu_dyntick(incby == 1 ? TPS("Endirq") : TPS("++="), + trace_rcu_watching(incby == 1 ? TPS("Startirq") : TPS("++="), ct_nmi_nesting(), ct_nmi_nesting() + incby, ct_rcu_watching()); instrumentation_end(); From patchwork Fri Aug 16 04:02:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neeraj Upadhyay X-Patchwork-Id: 13765422 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6F8BD144D01; Fri, 16 Aug 2024 04:06:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723781209; cv=none; b=PW7KZCftiEYov79ADTnbAr4lQog4XPMMme3YTY3KPlDD+Idw8HzUo60JzaBfyUpB6KerrRUWCCkLVrEkKWORrnxJC0ZnWQ5yIi+m1g1NslwB6he04sRBBWP38VboZlOlUxsZS24SroiVB6DmCfLpQZf3lhd1xOdHgztH+ScE4m0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723781209; c=relaxed/simple; bh=oGg1sTOOfik2DIsN98TerQ4lcdNiCmH/sdT5tuPDOKg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=oBqTUQNW1TT6HBb1JSvpiN3lLVGF4lSupAe6Jokkb4X7vgYQKHETuhoBf0xla7b4oa6kHCKT0rkkSQ7JaTzOcyWDt87MNkKEO1f+Ti6oWmjSE4WQAi6NURt3RsBl6iiuY+MvdF79E9aMOfjBcf2JZiWLRRU3eQG/nwLqpFjHD9Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=qiRQZR7C; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="qiRQZR7C" Received: by smtp.kernel.org (Postfix) with ESMTPSA id D7E70C4AF12; Fri, 16 Aug 2024 04:06:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723781209; bh=oGg1sTOOfik2DIsN98TerQ4lcdNiCmH/sdT5tuPDOKg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=qiRQZR7C/op9ywCAK4rWBIAea4GQQv8G3Cj5hH9DFoTEvMDvoeVr8cN46mdwbsouD aATEm8ul/wFm2Kruw0JxwbIS1DDan+bJGNG9KXYkCCEAECaHQI0cCWvxU4d5X8OBaQ 6CVcoS7PoO6FzkAHIX4gZOHGkg1zrGanTUsyRZB8pOz1x5uJumqAl4ceQliS3z687R 2b7IG6pHVB0fSl8AG6PuvM1kML486tdzfz/coghizERU0TVNzsPmqZo0x98R7KQX6m yB1MX/nhfqs4TXqoYmVrabEG7yjUZuJM+I7tN9Hutwpy1TybTDfwoaEDNQWd2Z+jv5 GMX8bjGm8dY0g== From: neeraj.upadhyay@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, paulmck@kernel.org, neeraj.upadhyay@kernel.org, neeraj.upadhyay@amd.com, boqun.feng@gmail.com, joel@joelfernandes.org, urezki@gmail.com, frederic@kernel.org, Sean Christopherson Subject: [PATCH rcu 26/26] context_tracking: Tag context_tracking_enabled_this_cpu() __always_inline Date: Fri, 16 Aug 2024 09:32:41 +0530 Message-Id: <20240816040241.17776-26-neeraj.upadhyay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240816040030.GA16986@neeraj.linux> References: <20240816040030.GA16986@neeraj.linux> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Sean Christopherson Force context_tracking_enabled_this_cpu() to be inlined so that invoking it from guest_context_enter_irqoff(), which KVM uses in non-instrumentable code, doesn't unexpectedly leave a noinstr section. vmlinux.o: warning: objtool: vmx_vcpu_enter_exit+0x1c7: call to context_tracking_enabled_this_cpu() leaves .noinstr.text section vmlinux.o: warning: objtool: svm_vcpu_enter_exit+0x83: call to context_tracking_enabled_this_cpu() leaves .noinstr.text section Note, the CONFIG_CONTEXT_TRACKING_USER=n stub is already __always_inline. Signed-off-by: Sean Christopherson Signed-off-by: Paul E. McKenney Signed-off-by: Neeraj Upadhyay --- include/linux/context_tracking_state.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/linux/context_tracking_state.h b/include/linux/context_tracking_state.h index 0dbda59c9f37..7b8433d5a8ef 100644 --- a/include/linux/context_tracking_state.h +++ b/include/linux/context_tracking_state.h @@ -113,7 +113,7 @@ static __always_inline bool context_tracking_enabled_cpu(int cpu) return context_tracking_enabled() && per_cpu(context_tracking.active, cpu); } -static inline bool context_tracking_enabled_this_cpu(void) +static __always_inline bool context_tracking_enabled_this_cpu(void) { return context_tracking_enabled() && __this_cpu_read(context_tracking.active); }