From patchwork Sat Mar 25 17:33:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 13187831 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0AA43C6FD1C for ; Sat, 25 Mar 2023 17:34:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230292AbjCYReE (ORCPT ); Sat, 25 Mar 2023 13:34:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36642 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231575AbjCYReD (ORCPT ); Sat, 25 Mar 2023 13:34:03 -0400 Received: from mail-qv1-xf34.google.com (mail-qv1-xf34.google.com [IPv6:2607:f8b0:4864:20::f34]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 33A77D50D for ; Sat, 25 Mar 2023 10:33:57 -0700 (PDT) Received: by mail-qv1-xf34.google.com with SMTP id m16so3707950qvi.12 for ; Sat, 25 Mar 2023 10:33:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; t=1679765636; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ttYDdJ2Zx276ZWlJoVAWoRza17Ux4kZUYP3LM4W/4/Q=; b=TBoO+zTCa6TlvCdSFINwGhqY8IkwKRzIpUuMBfJ00Ks7v4MNhxO5xhvGMwRhaK8P9r Gn7ol1JTk3hslUvRYVP2KSNYdJj6l80Fwotm3y4bciBkbTl7+U9pgO5y+hZKzkz8ZyyY aElsltBE3c6EIzkIKdIrj8qosyBuHLivrfW1E= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679765636; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ttYDdJ2Zx276ZWlJoVAWoRza17Ux4kZUYP3LM4W/4/Q=; b=2Py2+iF+lQqkdrdyREKrmnW1WFC8HnivFcJpR6WLL+z6Z1DKtswFGWybgFF2oxQCBm dxQTpw+xssIymKVo+JpkdwlFF7T5t8X2bqDXJomhmvvAQgzfA9kz7gp4t7Dsp9FoUgWb P+fa6kSQ/udiij/vz7LXgGbNPB0bRtXKH5M4LpA4KZtz9tnfpt/C9BZwRSIF3wmQIeQe mi8c+oZvN4Sb6nSoIBiyokxSL1alT112gF5pzV/qDwVW28LesRhT9xUFMttVsJH9OiCr M8QiyhUKt2+oIv30WMJvb0dmY8O9PCNdp1qGe3s/Tv2Dw7fhPLdKf0laEXKS8qUb5BNo s71Q== X-Gm-Message-State: AAQBX9cm2WxarXSKg8BI3zVbzGZEcwPy6B1huLeDykpgnnEoocmN0/ga lOfvs8s5GtCAs/1eLmdhpe/tzz9FPMjHEsKdXxk= X-Google-Smtp-Source: AKy350YQWqfmMrWDtG0WEVQO3hv3uuYGkz9FiACGNcg3/FZZLaU9RFaRrPjTEGVNb2IOqMP/p/ZRmg== X-Received: by 2002:a05:6214:f0f:b0:56e:a9d4:428a with SMTP id gw15-20020a0562140f0f00b0056ea9d4428amr9766799qvb.1.1679765636225; Sat, 25 Mar 2023 10:33:56 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (129.239.188.35.bc.googleusercontent.com. [35.188.239.129]) by smtp.gmail.com with ESMTPSA id m4-20020ac84444000000b003e37ee54b5dsm6762764qtn.90.2023.03.25.10.33.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 25 Mar 2023 10:33:55 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org, Greg Kroah-Hartman , "Rafael J. Wysocki" , Frederic Weisbecker , Thomas Gleixner , Ingo Molnar , "Paul E. McKenney" , Josh Triplett Cc: "Joel Fernandes (Google)" , Frederic Weisbecker , Zhouyi Zhou , Will Deacon , Marc Zyngier , rcu , stable@vger.kernel.org Subject: [PATCH v2 04/13] tick/nohz: Fix cpu_is_hotpluggable() by checking with nohz subsystem Date: Sat, 25 Mar 2023 17:33:07 +0000 Message-Id: <20230325173316.3118674-5-joel@joelfernandes.org> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog In-Reply-To: <20230325173316.3118674-1-joel@joelfernandes.org> References: <20230325173316.3118674-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org For CONFIG_NO_HZ_FULL systems, the tick_do_timer_cpu cannot be offlined. However, cpu_is_hotpluggable() still returns true for those CPUs. This causes torture tests that do offlining to end up trying to offline this CPU causing test failures. Such failure happens on all architectures. Fix it by asking the opinion of the nohz subsystem on whether the CPU can be hotplugged. [ Apply Frederic Weisbecker feedback on refactoring tick_nohz_cpu_down(). ] For drivers/base/ portion: Acked-by: Greg Kroah-Hartman Cc: Frederic Weisbecker Cc: "Paul E. McKenney" Cc: Zhouyi Zhou Cc: Will Deacon Cc: Marc Zyngier Cc: rcu Cc: stable@vger.kernel.org Fixes: 2987557f52b9 ("driver-core/cpu: Expose hotpluggability to the rest of the kernel") Signed-off-by: Paul E. McKenney Signed-off-by: Joel Fernandes (Google) Acked-by: Frederic Weisbecker Acked-by: Greg Kroah-Hartman Acked-by: Frederic Weisbecker Signed-off-by: Paul E. McKenney Signed-off-by: Joel Fernandes (Google) --- drivers/base/cpu.c | 3 ++- include/linux/tick.h | 2 ++ kernel/time/tick-sched.c | 11 ++++++++--- 3 files changed, 12 insertions(+), 4 deletions(-) diff --git a/drivers/base/cpu.c b/drivers/base/cpu.c index 182c6122f815..c1815b9dae68 100644 --- a/drivers/base/cpu.c +++ b/drivers/base/cpu.c @@ -487,7 +487,8 @@ static const struct attribute_group *cpu_root_attr_groups[] = { bool cpu_is_hotpluggable(unsigned int cpu) { struct device *dev = get_cpu_device(cpu); - return dev && container_of(dev, struct cpu, dev)->hotpluggable; + return dev && container_of(dev, struct cpu, dev)->hotpluggable + && tick_nohz_cpu_hotpluggable(cpu); } EXPORT_SYMBOL_GPL(cpu_is_hotpluggable); diff --git a/include/linux/tick.h b/include/linux/tick.h index bfd571f18cfd..9459fef5b857 100644 --- a/include/linux/tick.h +++ b/include/linux/tick.h @@ -216,6 +216,7 @@ extern void tick_nohz_dep_set_signal(struct task_struct *tsk, enum tick_dep_bits bit); extern void tick_nohz_dep_clear_signal(struct signal_struct *signal, enum tick_dep_bits bit); +extern bool tick_nohz_cpu_hotpluggable(unsigned int cpu); /* * The below are tick_nohz_[set,clear]_dep() wrappers that optimize off-cases @@ -280,6 +281,7 @@ static inline void tick_nohz_full_add_cpus_to(struct cpumask *mask) { } static inline void tick_nohz_dep_set_cpu(int cpu, enum tick_dep_bits bit) { } static inline void tick_nohz_dep_clear_cpu(int cpu, enum tick_dep_bits bit) { } +static inline bool tick_nohz_cpu_hotpluggable(unsigned int cpu) { return true; } static inline void tick_dep_set(enum tick_dep_bits bit) { } static inline void tick_dep_clear(enum tick_dep_bits bit) { } diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c index b0e3c9205946..68d81a4283c8 100644 --- a/kernel/time/tick-sched.c +++ b/kernel/time/tick-sched.c @@ -527,7 +527,7 @@ void __init tick_nohz_full_setup(cpumask_var_t cpumask) tick_nohz_full_running = true; } -static int tick_nohz_cpu_down(unsigned int cpu) +bool tick_nohz_cpu_hotpluggable(unsigned int cpu) { /* * The tick_do_timer_cpu CPU handles housekeeping duty (unbound @@ -535,8 +535,13 @@ static int tick_nohz_cpu_down(unsigned int cpu) * CPUs. It must remain online when nohz full is enabled. */ if (tick_nohz_full_running && tick_do_timer_cpu == cpu) - return -EBUSY; - return 0; + return false; + return true; +} + +static int tick_nohz_cpu_down(unsigned int cpu) +{ + return tick_nohz_cpu_hotpluggable(cpu) ? 0 : -EBUSY; } void __init tick_nohz_init(void) From patchwork Sat Mar 25 17:33:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 13187832 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E9A12C761AF for ; Sat, 25 Mar 2023 17:34:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231820AbjCYReH (ORCPT ); Sat, 25 Mar 2023 13:34:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36812 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231778AbjCYReG (ORCPT ); Sat, 25 Mar 2023 13:34:06 -0400 Received: from mail-qt1-x833.google.com (mail-qt1-x833.google.com [IPv6:2607:f8b0:4864:20::833]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6503A9763 for ; Sat, 25 Mar 2023 10:33:59 -0700 (PDT) Received: by mail-qt1-x833.google.com with SMTP id r5so4507060qtp.4 for ; Sat, 25 Mar 2023 10:33:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; t=1679765638; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=E6Aj0mbJD/qZB0r2J2tLnHasIPmuosonkIp69GrjB7E=; b=Y0ARwJXZKYTnxN/qpI5CVYzmurO4mC3453dcb5GLWg6JjriSKj0CM3lb+Msiqq3XWO 8HbIL+kk+3wbPn+tXBcGbmQKysL7d06+QRHdQYwQ3p0g0UcR6SNyLI7wxwREnGr+UCHq aSKCZrC2vRIuhioqVrVNyibAqvhLuYcxJlqJo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679765638; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=E6Aj0mbJD/qZB0r2J2tLnHasIPmuosonkIp69GrjB7E=; b=391qVWqwOu/rb0KCNd/moSkZRY7IOgJC1DmKDJVYSlkrRZg3kQlmGg9IZnIWI4T/G7 D22qEnkuhywg0d8MFlrvCl9dTjkzQbvwTLEE0keft2Da/uDsDJQb+rMjDO9G5j+xPyct NH8qs3aZrKsU1s3M9l6JeBdrnOWXErgcrxaYZQWfV4QAiyWZgWjGpLrWnkW1fvwYO1tY gIuhwo3qLov6VIctnxhTR70oLgzWl+KSUOpEw4eFGG9n7m5pXOdNmDTH26zJek8xPcq0 wHR7g7+hznb50pNvYB90VWN1dh9obQtue1X3Jgx2z2gVy59FLkoEhoVdGkfBE940G01N WOSg== X-Gm-Message-State: AO0yUKXZ2BUQBMMtIhJDvCPq8nBjV9wTHxngu9rKKvGTz0VY9n2BNYtt HgVw7yRlIkQxfD1e4AI1x+iuoA== X-Google-Smtp-Source: AK7set/CmKjhmjvVfu3VgHk3NFiY73TLBKFZ9sR9voGoRKmwo6xyEu0cGZniNtSZAxfO2TZZVtCdJQ== X-Received: by 2002:ac8:5b96:0:b0:3bf:c9d1:4276 with SMTP id a22-20020ac85b96000000b003bfc9d14276mr11467681qta.5.1679765638478; Sat, 25 Mar 2023 10:33:58 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (129.239.188.35.bc.googleusercontent.com. [35.188.239.129]) by smtp.gmail.com with ESMTPSA id m4-20020ac84444000000b003e37ee54b5dsm6762764qtn.90.2023.03.25.10.33.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 25 Mar 2023 10:33:58 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org, "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang Cc: rcu@vger.kernel.org Subject: [PATCH v2 06/13] rcu: Fix set/clear TICK_DEP_BIT_RCU_EXP bitmask race Date: Sat, 25 Mar 2023 17:33:09 +0000 Message-Id: <20230325173316.3118674-7-joel@joelfernandes.org> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog In-Reply-To: <20230325173316.3118674-1-joel@joelfernandes.org> References: <20230325173316.3118674-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org From: Zqiang For kernels built with CONFIG_NO_HZ_FULL=y, the following scenario can result in the scheduling-clock interrupt remaining enabled on a holdout CPU after its quiescent state has been reported: CPU1 CPU2 rcu_report_exp_cpu_mult synchronize_rcu_expedited_wait acquires rnp->lock mask = rnp->expmask; for_each_leaf_node_cpu_mask(rnp, cpu, mask) rnp->expmask = rnp->expmask & ~mask; rdp = per_cpu_ptr(&rcu_data, cpu1); for_each_leaf_node_cpu_mask(rnp, cpu, mask) rdp = per_cpu_ptr(&rcu_data, cpu1); if (!rdp->rcu_forced_tick_exp) continue; rdp->rcu_forced_tick_exp = true; tick_dep_set_cpu(cpu1, TICK_DEP_BIT_RCU_EXP); The problem is that CPU2's sampling of rnp->expmask is obsolete by the time it invokes tick_dep_set_cpu(), and CPU1 is not guaranteed to see CPU2's store to ->rcu_forced_tick_exp in time to clear it. And even if CPU1 does see that store, it might invoke tick_dep_clear_cpu() before CPU2 got around to executing its tick_dep_set_cpu(), which would still leave the victim CPU with its scheduler-clock tick running. Either way, an nohz_full real-time application running on the victim CPU would have its latency needlessly degraded. Note that expedited RCU grace periods look at context-tracking information, and so if the CPU is executing in nohz_full usermode throughout, that CPU cannot be victimized in this manner. This commit therefore causes synchronize_rcu_expedited_wait to hold the rcu_node structure's ->lock when checking for holdout CPUs, setting TICK_DEP_BIT_RCU_EXP, and invoking tick_dep_set_cpu(), thus preventing this race. Signed-off-by: Zqiang Reviewed-by: Frederic Weisbecker Signed-off-by: Paul E. McKenney Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree_exp.h | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h index 249c2967d9e6..7cc4856da081 100644 --- a/kernel/rcu/tree_exp.h +++ b/kernel/rcu/tree_exp.h @@ -594,6 +594,7 @@ static void synchronize_rcu_expedited_wait(void) struct rcu_data *rdp; struct rcu_node *rnp; struct rcu_node *rnp_root = rcu_get_root(); + unsigned long flags; trace_rcu_exp_grace_period(rcu_state.name, rcu_exp_gp_seq_endval(), TPS("startwait")); jiffies_stall = rcu_exp_jiffies_till_stall_check(); @@ -602,17 +603,17 @@ static void synchronize_rcu_expedited_wait(void) if (synchronize_rcu_expedited_wait_once(1)) return; rcu_for_each_leaf_node(rnp) { + raw_spin_lock_irqsave_rcu_node(rnp, flags); mask = READ_ONCE(rnp->expmask); for_each_leaf_node_cpu_mask(rnp, cpu, mask) { rdp = per_cpu_ptr(&rcu_data, cpu); if (rdp->rcu_forced_tick_exp) continue; rdp->rcu_forced_tick_exp = true; - preempt_disable(); if (cpu_online(cpu)) tick_dep_set_cpu(cpu, TICK_DEP_BIT_RCU_EXP); - preempt_enable(); } + raw_spin_unlock_irqrestore_rcu_node(rnp, flags); } j = READ_ONCE(jiffies_till_first_fqs); if (synchronize_rcu_expedited_wait_once(j + HZ)) From patchwork Sat Mar 25 17:33:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 13187833 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 10278C76196 for ; Sat, 25 Mar 2023 17:34:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230094AbjCYReM (ORCPT ); Sat, 25 Mar 2023 13:34:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36748 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231520AbjCYReH (ORCPT ); Sat, 25 Mar 2023 13:34:07 -0400 Received: from mail-qt1-x82c.google.com (mail-qt1-x82c.google.com [IPv6:2607:f8b0:4864:20::82c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CFDC79EEF for ; Sat, 25 Mar 2023 10:34:01 -0700 (PDT) Received: by mail-qt1-x82c.google.com with SMTP id bz27so4514902qtb.1 for ; Sat, 25 Mar 2023 10:34:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; t=1679765641; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=QyrXPYue4+kcgnuVhPQbl4hTj4jaTNBfGwV5VXHSeY8=; b=wY7sjFYpq8YfHJuBRa5ndpAjxilKOxdXrz02f1B0SQldgeClltUK0+9Vtv54p+zDPm dRF+8T5ZRft8VqUNlEvzSiiq2rG7I7wFmObEc8Dm9u9xHEzCF0D2P+oJfF8dJ3lmlyql fjeWspj+q+fR+IxsTM6N1CXB47IUmZZ28GsLs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679765641; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QyrXPYue4+kcgnuVhPQbl4hTj4jaTNBfGwV5VXHSeY8=; b=J9o4vE/UqLZOzoiqxeYnXbsjqv+N4nvOxNQaWpY6fla38g8vlcdYxZxdXhl0Wio7t1 yJsCk6kPpYEOaeoyf63e1FIkVe8pXJf+8Y95Prph/Uh78i2VpIEiEpL1EyCC6VepNcXj CYxOLeXJhxhy/J0KsWUWQtFQK7KrenHOfSAT6gqDXIuUDY1sU5rmwMpErT/slXqKUvML MGFGIMXLCaqGBqfntMet8YARLM2JGmuaKovgfnyRGPZ4bkdpg3kJmrXhEbFhPTXhvn7c Z5LhmBDPDA0rtMuOucGbX3QJ4HYEs2YAlUNDc88WgZUVwWVWCo4ISAIxEp1kqw0nVGUb MtjA== X-Gm-Message-State: AO0yUKUuSJs6b1ClOapv5tHnuNuZNMMYl3UkK8/W0f8W7F/qwvud6LYS WOyNufmiNBVE01izDmHFdx5tVg== X-Google-Smtp-Source: AK7set9Jlw5IeClQNc9dTI42kGK/qMIDrAXw7kkLsJ9JP3829DzmHHgq2Y4Qu45ppUcd1z+iFWox1w== X-Received: by 2002:a05:622a:1749:b0:3e3:a298:7928 with SMTP id l9-20020a05622a174900b003e3a2987928mr12730734qtk.9.1679765640939; Sat, 25 Mar 2023 10:34:00 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (129.239.188.35.bc.googleusercontent.com. [35.188.239.129]) by smtp.gmail.com with ESMTPSA id m4-20020ac84444000000b003e37ee54b5dsm6762764qtn.90.2023.03.25.10.34.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 25 Mar 2023 10:34:00 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org, "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang Cc: rcu@vger.kernel.org Subject: [PATCH v2 08/13] rcu: Register rcu-lazy shrinker only for CONFIG_RCU_LAZY=y kernels Date: Sat, 25 Mar 2023 17:33:11 +0000 Message-Id: <20230325173316.3118674-9-joel@joelfernandes.org> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog In-Reply-To: <20230325173316.3118674-1-joel@joelfernandes.org> References: <20230325173316.3118674-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org From: Zqiang The lazy_rcu_shrink_count() shrinker function is registered even in kernels built with CONFIG_RCU_LAZY=n, in which case this function uselessly consumes cycles learning that no CPU has any lazy callbacks queued. This commit therefore registers this shrinker function only in the kernels built with CONFIG_RCU_LAZY=y, where it might actually do something useful. Signed-off-by: Zqiang Reviewed-by: Frederic Weisbecker Reviewed-by: Joel Fernandes (Google) Signed-off-by: Paul E. McKenney Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree_nocb.h | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 9e1c8caec5ce..f2280616f9d5 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1312,6 +1312,7 @@ int rcu_nocb_cpu_offload(int cpu) } EXPORT_SYMBOL_GPL(rcu_nocb_cpu_offload); +#ifdef CONFIG_RCU_LAZY static unsigned long lazy_rcu_shrink_count(struct shrinker *shrink, struct shrink_control *sc) { @@ -1360,6 +1361,7 @@ static struct shrinker lazy_rcu_shrinker = { .batch = 0, .seeks = DEFAULT_SEEKS, }; +#endif // #ifdef CONFIG_RCU_LAZY void __init rcu_init_nohz(void) { @@ -1391,8 +1393,10 @@ void __init rcu_init_nohz(void) if (!rcu_state.nocb_is_setup) return; +#ifdef CONFIG_RCU_LAZY if (register_shrinker(&lazy_rcu_shrinker, "rcu-lazy")) pr_err("Failed to register lazy_rcu shrinker!\n"); +#endif // #ifdef CONFIG_RCU_LAZY if (!cpumask_subset(rcu_nocb_mask, cpu_possible_mask)) { pr_info("\tNote: kernel parameter 'rcu_nocbs=', 'nohz_full', or 'isolcpus=' contains nonexistent CPUs.\n"); From patchwork Sat Mar 25 17:33:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 13187834 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 47916C76196 for ; Sat, 25 Mar 2023 17:34:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231782AbjCYReP (ORCPT ); Sat, 25 Mar 2023 13:34:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36888 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231575AbjCYReH (ORCPT ); Sat, 25 Mar 2023 13:34:07 -0400 Received: from mail-qt1-x82b.google.com (mail-qt1-x82b.google.com [IPv6:2607:f8b0:4864:20::82b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 87579B779 for ; Sat, 25 Mar 2023 10:34:02 -0700 (PDT) Received: by mail-qt1-x82b.google.com with SMTP id x1so4499479qtr.7 for ; Sat, 25 Mar 2023 10:34:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; t=1679765642; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=QibGVAzBv0gTWQqE5TFgxflup7tk3PummO3A7I5nlFU=; b=Bitiwir0A9IEj0P20k33ypH0Yu5zV4sW3y+VbbmcQ3jA6Nw5b2VS13k98mZrzbC630 re3JkfEMsrpKvdzMDtvWdQOgOvAHYSdj4qWTk+3WFFSEZJYUii/o8lbJ/uIyU9P+yTcs eil2loMErO49vAu9dR/zDfZq5DfkTdczeMp8c= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679765642; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QibGVAzBv0gTWQqE5TFgxflup7tk3PummO3A7I5nlFU=; b=YzIC87IIZd6RmhB1x3i55XCHwOrHDrwDQ1tYL4SXzPygipMYATMlF7hwj9W9m44WMO QK5n8U/cCNWOCtH2dMWXwcajob7K4Pi9Sl1zP98iSOgZ2FND2N4b5TAjxyV9wi1ORH8x acR35LipHl5+oAAHQwmJjynya0+FQ1ML9oOmt5ygEcLacP+ndHO4BjqPzBCAjCZI5fl/ teEznwgVeXZ7AIDkGWJoa2/MpRAtH2nzQwlmQ01K8eY5KEK1lKUiWLtU5h3ZoHtwIhyW VjfvEweJKS/aQtcxFUwi8Xpnpps2kAdhAsvO2w/WdqzGvkvWK8OFOQ7jIDVbR9HQ4LXT sUgg== X-Gm-Message-State: AAQBX9dggHwOCvcS/g0m4qmmftvfRYt6PbZg8uho4Kj4bGwuN5sT/Jfu i6U5FxyjaiSohzN08Gkpv1rRyw== X-Google-Smtp-Source: AK7set/c+utZD7KsGUN+joNbxXwpbhjNKkljJ/zT/1BqFUiu6jbDVCKWmia3xO9V96K6lYkSph7hzA== X-Received: by 2002:ac8:7c48:0:b0:3e3:8bcd:23cb with SMTP id o8-20020ac87c48000000b003e38bcd23cbmr10851438qtv.29.1679765642081; Sat, 25 Mar 2023 10:34:02 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (129.239.188.35.bc.googleusercontent.com. [35.188.239.129]) by smtp.gmail.com with ESMTPSA id m4-20020ac84444000000b003e37ee54b5dsm6762764qtn.90.2023.03.25.10.34.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 25 Mar 2023 10:34:01 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org, "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang Cc: rcu@vger.kernel.org Subject: [PATCH v2 09/13] rcu: Remove never-set needwake assignment from rcu_report_qs_rdp() Date: Sat, 25 Mar 2023 17:33:12 +0000 Message-Id: <20230325173316.3118674-10-joel@joelfernandes.org> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog In-Reply-To: <20230325173316.3118674-1-joel@joelfernandes.org> References: <20230325173316.3118674-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org From: Zqiang The rcu_accelerate_cbs() function is invoked by rcu_report_qs_rdp() only if there is a grace period in progress that is still blocked by at least one CPU on this rcu_node structure. This means that rcu_accelerate_cbs() should never return the value true, and thus that this function should never set the needwake variable and in turn never invoke rcu_gp_kthread_wake(). This commit therefore removes the needwake variable and the invocation of rcu_gp_kthread_wake() in favor of a WARN_ON_ONCE() on the call to rcu_accelerate_cbs(). The purpose of this new WARN_ON_ONCE() is to detect situations where the system's opinion differs from ours. Signed-off-by: Zqiang Reviewed-by: Frederic Weisbecker Signed-off-by: Paul E. McKenney Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 8e880c09ab59..e80e8f128c57 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -1955,7 +1955,6 @@ rcu_report_qs_rdp(struct rcu_data *rdp) { unsigned long flags; unsigned long mask; - bool needwake = false; bool needacc = false; struct rcu_node *rnp; @@ -1987,7 +1986,12 @@ rcu_report_qs_rdp(struct rcu_data *rdp) * NOCB kthreads have their own way to deal with that... */ if (!rcu_rdp_is_offloaded(rdp)) { - needwake = rcu_accelerate_cbs(rnp, rdp); + /* + * The current GP has not yet ended, so it + * should not be possible for rcu_accelerate_cbs() + * to return true. So complain, but don't awaken. + */ + WARN_ON_ONCE(rcu_accelerate_cbs(rnp, rdp)); } else if (!rcu_segcblist_completely_offloaded(&rdp->cblist)) { /* * ...but NOCB kthreads may miss or delay callbacks acceleration @@ -1999,8 +2003,6 @@ rcu_report_qs_rdp(struct rcu_data *rdp) rcu_disable_urgency_upon_qs(rdp); rcu_report_qs_rnp(mask, rnp, rnp->gp_seq, flags); /* ^^^ Released rnp->lock */ - if (needwake) - rcu_gp_kthread_wake(); if (needacc) { rcu_nocb_lock_irqsave(rdp, flags); From patchwork Sat Mar 25 17:33:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 13187835 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 07BECC6FD1F for ; Sat, 25 Mar 2023 17:34:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231833AbjCYReQ (ORCPT ); Sat, 25 Mar 2023 13:34:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36934 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231777AbjCYReH (ORCPT ); Sat, 25 Mar 2023 13:34:07 -0400 Received: from mail-qt1-x832.google.com (mail-qt1-x832.google.com [IPv6:2607:f8b0:4864:20::832]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9E8C2EC52 for ; Sat, 25 Mar 2023 10:34:04 -0700 (PDT) Received: by mail-qt1-x832.google.com with SMTP id t19so4486356qta.12 for ; Sat, 25 Mar 2023 10:34:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; t=1679765643; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Q6C53sn27NuA32AX3ykr9qVTka3YNyXyLo2FHjcHAPk=; b=yhQE5JitzdBNkzY0TYfTzuvOS/VKmMwn5KP2dPiUdYcgdHLPrmvZZ6Np/lG32lNBAd lt9VxOLDD0Yxnj/P9thYO1M/R2JLG12ggb55LdZcQNrb8ZOv8E4OD9ybmGmBVQE7U7t4 vwfVjwDCDMeJ7G6L/7UIbp5+MZFMuw6ebBtYE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679765643; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Q6C53sn27NuA32AX3ykr9qVTka3YNyXyLo2FHjcHAPk=; b=D73FRGS1oKeBVVR9sHOqtuFC1zzt6pTI5yzeIjl9ldr8mUAT+Ok3qtyG+enBI7ztEP nWimAN+gX13qPlWn4BUpL073fVMIPtvacCbztv919Ch7i022vDyhJ/CYB4EgkTPmCgH7 efqvNNB40aw+uuu+MAMYHk3tPD/4NsH/dclFx1aL87162FZsnRgjWykqMST2Xz2z5EdR 89jOB9NsJOMR01BY+PCybvGTyQX0IcjVASNocBiyfDt8ABxiA4WuPVNiixJd43KRc1bo 4oDtbjfhF4SNrqGlpzGTKnbdyDsKKQlrsDcOhBdYpbwU0deNxr6if/u1qM5jT1YcLVLJ PezQ== X-Gm-Message-State: AO0yUKXqao/7bxT2qaft/0HQQXZtkuA2YaRrQzFkGPKeH1QdeyDAuk0Y gso6Miq3ee+0dpCCPIksGt9nfw== X-Google-Smtp-Source: AK7set/kBRoccFhabLv3Raii1q34w76PXtiC0I5XZu7Br5sHC8TyaR8GiL2D8zsuukPc+azN0dL7kw== X-Received: by 2002:ac8:5787:0:b0:3e3:98af:5de9 with SMTP id v7-20020ac85787000000b003e398af5de9mr12140109qta.63.1679765643455; Sat, 25 Mar 2023 10:34:03 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (129.239.188.35.bc.googleusercontent.com. [35.188.239.129]) by smtp.gmail.com with ESMTPSA id m4-20020ac84444000000b003e37ee54b5dsm6762764qtn.90.2023.03.25.10.34.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 25 Mar 2023 10:34:03 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org, "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang Cc: rcu@vger.kernel.org Subject: [PATCH v2 10/13] rcu: Permit start_poll_synchronize_rcu_expedited() to be invoked early Date: Sat, 25 Mar 2023 17:33:13 +0000 Message-Id: <20230325173316.3118674-11-joel@joelfernandes.org> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog In-Reply-To: <20230325173316.3118674-1-joel@joelfernandes.org> References: <20230325173316.3118674-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org From: Zqiang According to the commit log of the patch that added it to the kernel, start_poll_synchronize_rcu_expedited() can be invoked very early, as in long before rcu_init() has been invoked. But before rcu_init(), the rcu_data structure's ->mynode field has not yet been initialized. This means that the start_poll_synchronize_rcu_expedited() function's attempt to set the CPU's leaf rcu_node structure's ->exp_seq_poll_rq field will result in a segmentation fault. This commit therefore causes start_poll_synchronize_rcu_expedited() to set ->exp_seq_poll_rq only after rcu_init() has initialized all CPUs' rcu_data structures' ->mynode fields. It also removes the check from the rcu_init() function so that start_poll_synchronize_rcu_expedited( is unconditionally invoked. Yes, this might result in an unnecessary boot-time grace period, but this is down in the noise. Signed-off-by: Zqiang Reviewed-by: Frederic Weisbecker Reviewed-by: Joel Fernandes (Google) Signed-off-by: Paul E. McKenney Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree.c | 5 ++--- kernel/rcu/tree_exp.h | 5 +++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index e80e8f128c57..90d54571126a 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -4942,9 +4942,8 @@ void __init rcu_init(void) else qovld_calc = qovld; - // Kick-start any polled grace periods that started early. - if (!(per_cpu_ptr(&rcu_data, cpu)->mynode->exp_seq_poll_rq & 0x1)) - (void)start_poll_synchronize_rcu_expedited(); + // Kick-start in case any polled grace periods started early. + (void)start_poll_synchronize_rcu_expedited(); rcu_test_sync_prims(); } diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h index 7cc4856da081..5343f32e7d67 100644 --- a/kernel/rcu/tree_exp.h +++ b/kernel/rcu/tree_exp.h @@ -1066,9 +1066,10 @@ unsigned long start_poll_synchronize_rcu_expedited(void) if (rcu_init_invoked()) raw_spin_lock_irqsave(&rnp->exp_poll_lock, flags); if (!poll_state_synchronize_rcu(s)) { - rnp->exp_seq_poll_rq = s; - if (rcu_init_invoked()) + if (rcu_init_invoked()) { + rnp->exp_seq_poll_rq = s; queue_work(rcu_gp_wq, &rnp->exp_poll_wq); + } } if (rcu_init_invoked()) raw_spin_unlock_irqrestore(&rnp->exp_poll_lock, flags); From patchwork Sat Mar 25 17:33:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 13187836 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E0868C76196 for ; Sat, 25 Mar 2023 17:34:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231929AbjCYReV (ORCPT ); Sat, 25 Mar 2023 13:34:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37162 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231841AbjCYReK (ORCPT ); Sat, 25 Mar 2023 13:34:10 -0400 Received: from mail-qt1-x835.google.com (mail-qt1-x835.google.com [IPv6:2607:f8b0:4864:20::835]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C4CE5EC6E for ; Sat, 25 Mar 2023 10:34:05 -0700 (PDT) Received: by mail-qt1-x835.google.com with SMTP id c19so4481216qtn.13 for ; Sat, 25 Mar 2023 10:34:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; t=1679765645; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kDgsb2d+hZmc+VQbS7ffR7EbpJzTOYghTZrLGYmTtWc=; b=jhaF7jMZ0YB7rLiRDFZrRklRXt1OS7zPKXAlOrC6Ofk00510pCD4FnDFVqfLxhpUjD BgjbV9/qa/zMtSEVZ9i8O8NwPGlXueA5d8pnRbjRaOifUjtUqVzrrbg4wXpX7QQON0Ga h2cXsbZfUrcDoLDyKLO+AqiHanYO7hZFWtMyw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679765645; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kDgsb2d+hZmc+VQbS7ffR7EbpJzTOYghTZrLGYmTtWc=; b=kGmXtgoEhSA7+M7xzwz0HCRkhJOEk+7G6EPo4SLSI1RMNIRn+3f/PacYhVHrVhihM1 sRgm1wkj6W/SZ/oiIFI+UMa4jH74EXgbcuneN4HED5V5dsE0Tb3iduCBtD+bENOoZNK9 0ZqLdCupTbr+7VF+Z5Xc9YqPI8++2+0JRo5i78ojyUw3VGzUrf0ZkOZEPaaRhuRx+kvJ GIY0/yu2bXnRnY4m2K0GSifgfQHNFOPoLqH+iGHrdtcOzshd8J26+KusBq3rpcZQ10n5 WQuGoSX0GC9Ip0iBmxrlk71zycFsbF4j+pnnag+ocRv1DoqtPJcmxrdF0kXLfKLW5K6u FtyQ== X-Gm-Message-State: AO0yUKU621newZ04iYd6GLUXbSHOLy0YcRWGZ9ElQLdkEAnHnS3f1MCF aqyFxoyiVmrFW8RfFppwM1slHKMx0JPZbI+YKok= X-Google-Smtp-Source: AKy350ZKIT2t3gxR4Zylf8zYlF3puKJt9xTfHezSW7rx/lBHNOBE+j7Z/Q9dY0lrU51bFaQa0S0IRA== X-Received: by 2002:ac8:5755:0:b0:3e1:7465:91f5 with SMTP id 21-20020ac85755000000b003e1746591f5mr12423604qtx.10.1679765644872; Sat, 25 Mar 2023 10:34:04 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (129.239.188.35.bc.googleusercontent.com. [35.188.239.129]) by smtp.gmail.com with ESMTPSA id m4-20020ac84444000000b003e37ee54b5dsm6762764qtn.90.2023.03.25.10.34.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 25 Mar 2023 10:34:04 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org, "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang Cc: Mark Brown , rcu@vger.kernel.org Subject: [PATCH v2 11/13] rcu-tasks: Report stalls during synchronize_srcu() in rcu_tasks_postscan() Date: Sat, 25 Mar 2023 17:33:14 +0000 Message-Id: <20230325173316.3118674-12-joel@joelfernandes.org> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog In-Reply-To: <20230325173316.3118674-1-joel@joelfernandes.org> References: <20230325173316.3118674-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org From: Neeraj Upadhyay The call to synchronize_srcu() from rcu_tasks_postscan() can be stalled by a task getting stuck in do_exit() between that function's calls to exit_tasks_rcu_start() and exit_tasks_rcu_finish(). To ease diagnosis of this situation, print a stall warning message every rcu_task_stall_info period when rcu_tasks_postscan() is stalled. [ paulmck: Adjust to handle CONFIG_SMP=n. ] Reported-by: Mark Brown Link: https://lore.kernel.org/rcu/20230111212736.GA1062057@paulmck-ThinkPad-P17-Gen-1/ Signed-off-by: Neeraj Upadhyay Signed-off-by: Paul E. McKenney Signed-off-by: Joel Fernandes (Google) Acked-by: Frederic Weisbecker --- kernel/rcu/tasks.h | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index bfb5e1549f2b..baf7ec178155 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -139,6 +139,12 @@ static struct rcu_tasks rt_name = \ /* Track exiting tasks in order to allow them to be waited for. */ DEFINE_STATIC_SRCU(tasks_rcu_exit_srcu); +#ifdef CONFIG_TASKS_RCU +/* Report delay in synchronize_srcu() completion in rcu_tasks_postscan(). */ +static void tasks_rcu_exit_srcu_stall(struct timer_list *unused); +static DEFINE_TIMER(tasks_rcu_exit_srcu_stall_timer, tasks_rcu_exit_srcu_stall); +#endif + /* Avoid IPIing CPUs early in the grace period. */ #define RCU_TASK_IPI_DELAY (IS_ENABLED(CONFIG_TASKS_TRACE_RCU_READ_MB) ? HZ / 2 : 0) static int rcu_task_ipi_delay __read_mostly = RCU_TASK_IPI_DELAY; @@ -830,6 +836,13 @@ static void rcu_tasks_pertask(struct task_struct *t, struct list_head *hop) /* Processing between scanning taskslist and draining the holdout list. */ static void rcu_tasks_postscan(struct list_head *hop) { + int rtsi = READ_ONCE(rcu_task_stall_info); + + if (!IS_ENABLED(CONFIG_TINY_RCU)) { + tasks_rcu_exit_srcu_stall_timer.expires = jiffies + rtsi; + add_timer(&tasks_rcu_exit_srcu_stall_timer); + } + /* * Exiting tasks may escape the tasklist scan. Those are vulnerable * until their final schedule() with TASK_DEAD state. To cope with @@ -848,6 +861,9 @@ static void rcu_tasks_postscan(struct list_head *hop) * call to synchronize_rcu(). */ synchronize_srcu(&tasks_rcu_exit_srcu); + + if (!IS_ENABLED(CONFIG_TINY_RCU)) + del_timer_sync(&tasks_rcu_exit_srcu_stall_timer); } /* See if tasks are still holding out, complain if so. */ @@ -923,6 +939,21 @@ static void rcu_tasks_postgp(struct rcu_tasks *rtp) void call_rcu_tasks(struct rcu_head *rhp, rcu_callback_t func); DEFINE_RCU_TASKS(rcu_tasks, rcu_tasks_wait_gp, call_rcu_tasks, "RCU Tasks"); +static void tasks_rcu_exit_srcu_stall(struct timer_list *unused) +{ +#ifndef CONFIG_TINY_RCU + int rtsi; + + rtsi = READ_ONCE(rcu_task_stall_info); + pr_info("%s: %s grace period number %lu (since boot) gp_state: %s is %lu jiffies old.\n", + __func__, rcu_tasks.kname, rcu_tasks.tasks_gp_seq, + tasks_gp_state_getname(&rcu_tasks), jiffies - rcu_tasks.gp_jiffies); + pr_info("Please check any exiting tasks stuck between calls to exit_tasks_rcu_start() and exit_tasks_rcu_finish()\n"); + tasks_rcu_exit_srcu_stall_timer.expires = jiffies + rtsi; + add_timer(&tasks_rcu_exit_srcu_stall_timer); +#endif // #ifndef CONFIG_TINY_RCU +} + /** * call_rcu_tasks() - Queue an RCU for invocation task-based grace period * @rhp: structure to be used for queueing the RCU updates. From patchwork Sat Mar 25 17:33:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 13187837 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 231BBC6FD1F for ; Sat, 25 Mar 2023 17:34:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231984AbjCYRe2 (ORCPT ); Sat, 25 Mar 2023 13:34:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36934 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231886AbjCYReO (ORCPT ); Sat, 25 Mar 2023 13:34:14 -0400 Received: from mail-qt1-x82d.google.com (mail-qt1-x82d.google.com [IPv6:2607:f8b0:4864:20::82d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1C5D8F744 for ; Sat, 25 Mar 2023 10:34:07 -0700 (PDT) Received: by mail-qt1-x82d.google.com with SMTP id ga7so4521133qtb.2 for ; Sat, 25 Mar 2023 10:34:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; t=1679765646; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8qmbLy4FeD1wXnpI5jFiDQ+/P/XNzoLaPeANuPc2CnI=; b=EMuepLXHAx1qWBBkKwrBLaXDPtiy7wo5Nx6Gbt+/79BQ1l1YhLOKw+DvIsAKNacCVV XLEvZQXEJVNl4Wt9PIGjTLoHgIxkQUfoyMmJqKf0/PvCUQ5q//7G/zQHgcKYkDUfFB7s C8yPwfoLXSJEB9JPxc1hkYf/Gi+q1qB5Gu24U= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679765646; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8qmbLy4FeD1wXnpI5jFiDQ+/P/XNzoLaPeANuPc2CnI=; b=fg44+zg2mEu8lFawckblZUPuQ8YhtoAPelr5GFmFaHY6gqJ6RsLaflfe70ZelShnsg Hua+lx6yLR60yk80oeYJD901/HFmaDp9K5MelBc6NUypCh7qK25Poz9EfGYSZ0ergpd5 RCyF3WfqemTZP4b/wf+6eclr9n9RYz6R+a8RQYVlfoF78vZQZuoigRrOx0M64XuGHWSN zJfedc93DGT3zx5Bb6gebrORQVtvP/5y0cL/aDH9POx3Ajrkx6y/i7D3T3zYGFD6WMYn Bdi266Kr6iFDmXbyJjjX3zVYFPyJrlneX6I9/3iIZ+T/s9sfZj4fB0prfkcjzZFrKfX2 zIfg== X-Gm-Message-State: AO0yUKWaYnP0Ix3suo7+HaCU8kumta/xZBpqCJZioDKjto0ofOh5Se7H vQg7G8qJdYEhC/cBwrCES+w9sA== X-Google-Smtp-Source: AK7set8gLQCta4FYIlst2AUFh/hE7rHGLWZ46fe8EwGZn7VJz0oJc9ujYIdovRFZpYujEKW41/8PTg== X-Received: by 2002:a05:622a:88:b0:3dd:a703:f7df with SMTP id o8-20020a05622a008800b003dda703f7dfmr12570788qtw.2.1679765646066; Sat, 25 Mar 2023 10:34:06 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (129.239.188.35.bc.googleusercontent.com. [35.188.239.129]) by smtp.gmail.com with ESMTPSA id m4-20020ac84444000000b003e37ee54b5dsm6762764qtn.90.2023.03.25.10.34.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 25 Mar 2023 10:34:05 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org, "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Thomas Gleixner , Ingo Molnar Cc: Zheng Yejian , stable@vger.kernel.org, rcu@vger.kernel.org Subject: [PATCH v2 12/13] rcu: Avoid stack overflow due to __rcu_irq_enter_check_tick() being kprobe-ed Date: Sat, 25 Mar 2023 17:33:15 +0000 Message-Id: <20230325173316.3118674-13-joel@joelfernandes.org> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog In-Reply-To: <20230325173316.3118674-1-joel@joelfernandes.org> References: <20230325173316.3118674-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org From: Zheng Yejian Registering a kprobe on __rcu_irq_enter_check_tick() can cause kernel stack overflow as shown below. This issue can be reproduced by enabling CONFIG_NO_HZ_FULL and booting the kernel with argument "nohz_full=", and then giving the following commands at the shell prompt: # cd /sys/kernel/tracing/ # echo 'p:mp1 __rcu_irq_enter_check_tick' >> kprobe_events # echo 1 > events/kprobes/enable This commit therefore adds __rcu_irq_enter_check_tick() to the kprobes blacklist using NOKPROBE_SYMBOL(). Insufficient stack space to handle exception! ESR: 0x00000000f2000004 -- BRK (AArch64) FAR: 0x0000ffffccf3e510 Task stack: [0xffff80000ad30000..0xffff80000ad38000] IRQ stack: [0xffff800008050000..0xffff800008058000] Overflow stack: [0xffff089c36f9f310..0xffff089c36fa0310] CPU: 5 PID: 190 Comm: bash Not tainted 6.2.0-rc2-00320-g1f5abbd77e2c #19 Hardware name: linux,dummy-virt (DT) pstate: 400003c5 (nZcv DAIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--) pc : __rcu_irq_enter_check_tick+0x0/0x1b8 lr : ct_nmi_enter+0x11c/0x138 sp : ffff80000ad30080 x29: ffff80000ad30080 x28: ffff089c82e20000 x27: 0000000000000000 x26: 0000000000000000 x25: ffff089c02a8d100 x24: 0000000000000000 x23: 00000000400003c5 x22: 0000ffffccf3e510 x21: ffff089c36fae148 x20: ffff80000ad30120 x19: ffffa8da8fcce148 x18: 0000000000000000 x17: 0000000000000000 x16: 0000000000000000 x15: ffffa8da8e44ea6c x14: ffffa8da8e44e968 x13: ffffa8da8e03136c x12: 1fffe113804d6809 x11: ffff6113804d6809 x10: 0000000000000a60 x9 : dfff800000000000 x8 : ffff089c026b404f x7 : 00009eec7fb297f7 x6 : 0000000000000001 x5 : ffff80000ad30120 x4 : dfff800000000000 x3 : ffffa8da8e3016f4 x2 : 0000000000000003 x1 : 0000000000000000 x0 : 0000000000000000 Kernel panic - not syncing: kernel stack overflow CPU: 5 PID: 190 Comm: bash Not tainted 6.2.0-rc2-00320-g1f5abbd77e2c #19 Hardware name: linux,dummy-virt (DT) Call trace: dump_backtrace+0xf8/0x108 show_stack+0x20/0x30 dump_stack_lvl+0x68/0x84 dump_stack+0x1c/0x38 panic+0x214/0x404 add_taint+0x0/0xf8 panic_bad_stack+0x144/0x160 handle_bad_stack+0x38/0x58 __bad_stack+0x78/0x7c __rcu_irq_enter_check_tick+0x0/0x1b8 arm64_enter_el1_dbg.isra.0+0x14/0x20 el1_dbg+0x2c/0x90 el1h_64_sync_handler+0xcc/0xe8 el1h_64_sync+0x64/0x68 __rcu_irq_enter_check_tick+0x0/0x1b8 arm64_enter_el1_dbg.isra.0+0x14/0x20 el1_dbg+0x2c/0x90 el1h_64_sync_handler+0xcc/0xe8 el1h_64_sync+0x64/0x68 __rcu_irq_enter_check_tick+0x0/0x1b8 arm64_enter_el1_dbg.isra.0+0x14/0x20 el1_dbg+0x2c/0x90 el1h_64_sync_handler+0xcc/0xe8 el1h_64_sync+0x64/0x68 __rcu_irq_enter_check_tick+0x0/0x1b8 [...] el1_dbg+0x2c/0x90 el1h_64_sync_handler+0xcc/0xe8 el1h_64_sync+0x64/0x68 __rcu_irq_enter_check_tick+0x0/0x1b8 arm64_enter_el1_dbg.isra.0+0x14/0x20 el1_dbg+0x2c/0x90 el1h_64_sync_handler+0xcc/0xe8 el1h_64_sync+0x64/0x68 __rcu_irq_enter_check_tick+0x0/0x1b8 arm64_enter_el1_dbg.isra.0+0x14/0x20 el1_dbg+0x2c/0x90 el1h_64_sync_handler+0xcc/0xe8 el1h_64_sync+0x64/0x68 __rcu_irq_enter_check_tick+0x0/0x1b8 el1_interrupt+0x28/0x60 el1h_64_irq_handler+0x18/0x28 el1h_64_irq+0x64/0x68 __ftrace_set_clr_event_nolock+0x98/0x198 __ftrace_set_clr_event+0x58/0x80 system_enable_write+0x144/0x178 vfs_write+0x174/0x738 ksys_write+0xd0/0x188 __arm64_sys_write+0x4c/0x60 invoke_syscall+0x64/0x180 el0_svc_common.constprop.0+0x84/0x160 do_el0_svc+0x48/0xe8 el0_svc+0x34/0xd0 el0t_64_sync_handler+0xb8/0xc0 el0t_64_sync+0x190/0x194 SMP: stopping secondary CPUs Kernel Offset: 0x28da86000000 from 0xffff800008000000 PHYS_OFFSET: 0xfffff76600000000 CPU features: 0x00000,01a00100,0000421b Memory Limit: none Link: https://lore.kernel.org/all/20221119040049.795065-1-zhengyejian1@huawei.com/ Fixes: aaf2bc50df1f ("rcu: Abstract out rcu_irq_enter_check_tick() from rcu_nmi_enter()") Signed-off-by: Zheng Yejian Cc: stable@vger.kernel.org Signed-off-by: Paul E. McKenney Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree.c | 1 + 1 file changed, 1 insertion(+) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 90d54571126a..ee27a03d7576 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -640,6 +640,7 @@ void __rcu_irq_enter_check_tick(void) } raw_spin_unlock_rcu_node(rdp->mynode); } +NOKPROBE_SYMBOL(__rcu_irq_enter_check_tick); #endif /* CONFIG_NO_HZ_FULL */ /* From patchwork Sat Mar 25 17:33:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 13187838 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9FF2AC6FD1C for ; Sat, 25 Mar 2023 17:34:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231928AbjCYReb (ORCPT ); Sat, 25 Mar 2023 13:34:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37162 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231840AbjCYReU (ORCPT ); Sat, 25 Mar 2023 13:34:20 -0400 Received: from mail-qt1-x836.google.com (mail-qt1-x836.google.com [IPv6:2607:f8b0:4864:20::836]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C06F5F75B for ; Sat, 25 Mar 2023 10:34:07 -0700 (PDT) Received: by mail-qt1-x836.google.com with SMTP id g19so4488997qts.9 for ; Sat, 25 Mar 2023 10:34:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; t=1679765647; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ONoHk+Rj1ufMcYHSOwfNTPmuuZ3mXzko5D4ACMs/eOA=; b=ikQ37MdCcvJ+qjCxLa0DlYnQ1qgqtu0+7/KD0Bb/Tc4mBBI7pxErsaA3YnRKCfkpmX 3p0P+l2DU6LzWXS3KmQPuTqncSpWI/pcmz1ufIhInc5srxC3za/4WZhFXfWgLnnaNFNZ Nk9m6/9CnTBL+94W3Acb2UnO12oHOjCG9k8vg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679765647; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ONoHk+Rj1ufMcYHSOwfNTPmuuZ3mXzko5D4ACMs/eOA=; b=HRBDPp8HmAnCHfAd/JdwOwagVA7Tm1UP39wgB1rZBQjQVTNiQcllpH127MeeVbAiro K025551FGUMhJpZyM0CHihYR0q+4+mpQ1o4sLwVj09YXx9fxwCvo3hlCKAVOuNki+Lxd nLW1N/ubV9FvovHtf7rytq9vtCIlgQ4hCMx7EPQK9E8dUwgqZIo+Rm2lAhx3ytJfjdH7 JMlkgaEtexOsnJpC8OP/1mxJQMUja2XMSxorOqcV+inCKu5V8LAxgnIYjhzRyn2WCDxd 6KMGfBg7Mhtm1IRY+Sjx69q4Bo/fZtX+EmKoFZmUSvhz8zy5o4CAMmtWaLt1z5YWh7dZ wdCA== X-Gm-Message-State: AAQBX9ekBequUcObbBUFZ9+RtroNv+1XDqnaONbS6hI+q1d6skbqwO2U rfKMUuB0pCYmD6VTlPKMmBx5yg== X-Google-Smtp-Source: AKy350YEgc1Eq00eYoCtjBB7hMJjZEs/02/OUcwF5Nmwzwr8rFs2fkTW9CVn5OaLI4MsCVSYmWRBKw== X-Received: by 2002:a05:622a:1a0b:b0:3e4:dbaa:8a05 with SMTP id f11-20020a05622a1a0b00b003e4dbaa8a05mr4599709qtb.0.1679765647265; Sat, 25 Mar 2023 10:34:07 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (129.239.188.35.bc.googleusercontent.com. [35.188.239.129]) by smtp.gmail.com with ESMTPSA id m4-20020ac84444000000b003e37ee54b5dsm6762764qtn.90.2023.03.25.10.34.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 25 Mar 2023 10:34:06 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org, "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang Cc: rcu@vger.kernel.org Subject: [PATCH v2 13/13] rcu: Protect rcu_print_task_exp_stall() ->exp_tasks access Date: Sat, 25 Mar 2023 17:33:16 +0000 Message-Id: <20230325173316.3118674-14-joel@joelfernandes.org> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog In-Reply-To: <20230325173316.3118674-1-joel@joelfernandes.org> References: <20230325173316.3118674-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org From: Zqiang For kernels built with CONFIG_PREEMPT_RCU=y, the following scenario can result in a NULL-pointer dereference: CPU1 CPU2 rcu_preempt_deferred_qs_irqrestore rcu_print_task_exp_stall if (special.b.blocked) READ_ONCE(rnp->exp_tasks) != NULL raw_spin_lock_rcu_node np = rcu_next_node_entry(t, rnp) if (&t->rcu_node_entry == rnp->exp_tasks) WRITE_ONCE(rnp->exp_tasks, np) .... raw_spin_unlock_irqrestore_rcu_node raw_spin_lock_irqsave_rcu_node t = list_entry(rnp->exp_tasks->prev, struct task_struct, rcu_node_entry) (if rnp->exp_tasks is NULL, this will dereference a NULL pointer) The problem is that CPU2 accesses the rcu_node structure's->exp_tasks field without holding the rcu_node structure's ->lock and CPU2 did not observe CPU1's change to rcu_node structure's ->exp_tasks in time. Therefore, if CPU1 sets rcu_node structure's->exp_tasks pointer to NULL, then CPU2 might dereference that NULL pointer. This commit therefore holds the rcu_node structure's ->lock while accessing that structure's->exp_tasks field. [ paulmck: Apply Frederic Weisbecker feedback. ] Signed-off-by: Zqiang Signed-off-by: Paul E. McKenney Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree_exp.h | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h index 5343f32e7d67..3b7abb58157d 100644 --- a/kernel/rcu/tree_exp.h +++ b/kernel/rcu/tree_exp.h @@ -803,9 +803,11 @@ static int rcu_print_task_exp_stall(struct rcu_node *rnp) int ndetected = 0; struct task_struct *t; - if (!READ_ONCE(rnp->exp_tasks)) - return 0; raw_spin_lock_irqsave_rcu_node(rnp, flags); + if (!rnp->exp_tasks) { + raw_spin_unlock_irqrestore_rcu_node(rnp, flags); + return 0; + } t = list_entry(rnp->exp_tasks->prev, struct task_struct, rcu_node_entry); list_for_each_entry_continue(t, &rnp->blkd_tasks, rcu_node_entry) {