From patchwork Fri Aug 2 00:42:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 13750983 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6B1D815E89; Fri, 2 Aug 2024 00:43:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722559391; cv=none; b=m+Ll7vtZRwLqmkNM5gQUSgHqS8VyXKOBBDJZ/RaZekaDwsVgdHi6s1vvGIgPvhYd9nFFJcE1kGQ0nZDdGVqDGNn95K4DSFRlVj6hv5YgKu4lfNJek3bOOpPgzRTO7+cUMy3xQlh1QlombhuCdci4f1xnf0Tee8k3QSEGMO7HmA8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722559391; c=relaxed/simple; bh=cISelkdGqljgDeI2+QcSA5s1XZGf3TYnjcH1wmM0OdM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=uSnWYmSYpv8cmJWGzzbrfZeIM1J2GvZdZGvbyte/VUv0/htgglCViGETOlYlD1q4PRbbjrjFa4pyk89LBwVDUyZ3wwQOtUVcsalq6l7UC5eJ+FW9dlNl93x76zIkDWX2Z8WhcLqMVgYZYOimGyil+AulCPxCNz1lOkRlZ66d1YQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=TjDgN8vn; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="TjDgN8vn" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 08670C32786; Fri, 2 Aug 2024 00:43:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1722559391; bh=cISelkdGqljgDeI2+QcSA5s1XZGf3TYnjcH1wmM0OdM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=TjDgN8vnv0H0AAhrda5LwUO5Kyi1T+RuOPT+grl1MdPMyG2S2Mb0EsDHMO4zftjwW nu0/3oZ03SBMpbRwmS6CTXhqEVo5SXJkgr3V2nLP9PC4ixvv2g8k02QEBI6n0GmYcH nYhANZM/+MPikZUyT6/0qPnUww5hB1SVtwcvix6Y/qDmAVnZICn9yGkCez5zuFRd4D dR0EItufThTc083wZUa8wT7WZmMvhExThz1L0sWYi8PEpAF5Ph9DV6haYJAB3nF6Pr KrsM6sOgi08GkL7D3SgVDiBIeC2wpu32Rkw6JzmFzVAWk0hvWoRHWyjLRV2CuqS+wR C7eIrUoYvzgxQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id B6AB8CE09F8; Thu, 1 Aug 2024 17:43:10 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH rcu 01/11] rcuscale: Save a few lines with whitespace-only change Date: Thu, 1 Aug 2024 17:42:58 -0700 Message-Id: <20240802004308.4134731-1-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <917e8cc8-8688-428a-9122-25544c5cc101@paulmck-laptop> References: <917e8cc8-8688-428a-9122-25544c5cc101@paulmck-laptop> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 This whitespace-only commit fuses a few lines of code, taking advantage of the newish 100-character-per-line limit to save a few lines of code. Signed-off-by: Paul E. McKenney --- kernel/rcu/rcuscale.c | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/kernel/rcu/rcuscale.c b/kernel/rcu/rcuscale.c index d534d4ec23147..3269dd9c639f7 100644 --- a/kernel/rcu/rcuscale.c +++ b/kernel/rcu/rcuscale.c @@ -1015,13 +1015,9 @@ rcu_scale_init(void) } while (atomic_read(&n_rcu_scale_reader_started) < nrealreaders) schedule_timeout_uninterruptible(1); - writer_tasks = kcalloc(nrealwriters, sizeof(reader_tasks[0]), - GFP_KERNEL); - writer_durations = kcalloc(nrealwriters, sizeof(*writer_durations), - GFP_KERNEL); - writer_n_durations = - kcalloc(nrealwriters, sizeof(*writer_n_durations), - GFP_KERNEL); + writer_tasks = kcalloc(nrealwriters, sizeof(reader_tasks[0]), GFP_KERNEL); + writer_durations = kcalloc(nrealwriters, sizeof(*writer_durations), GFP_KERNEL); + writer_n_durations = kcalloc(nrealwriters, sizeof(*writer_n_durations), GFP_KERNEL); if (!writer_tasks || !writer_durations || !writer_n_durations) { SCALEOUT_ERRSTRING("out of memory"); firsterr = -ENOMEM; From patchwork Fri Aug 2 00:42:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 13750984 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6B1A115AC4; Fri, 2 Aug 2024 00:43:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722559391; cv=none; b=TosHeh1zYrFHNRcjPh7zexxL6XazbqIX6zNLifKHRsibTfNg/KD9JgdUDbIYL6ilTRvUf7uNEmIhr59E3jNIQ+t/VpcPlLD0oQcr+B2rWMXsqvPF4BnW40s0sTytQ9mMpPAILBkYtx/mcJFBY3+hRmRAG6H5CSli01ZUteW90Oo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722559391; c=relaxed/simple; bh=DqMDZ/+cWkW/MQt6snJKfAxQMZLfZafULDqWCEsrGjE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=tZBLW4YcddZasecWxjjk5b7nL1rjK2ljd1YLdZ/7/jlDoWZTsQWTaR4hP1aMHIepQj/jt/cS2BZaej1VQtkkbY/23e0K9D9W9V7Eyjl5vxI1jae7Q0K/55n5UTiNe6phccRjXQF45Be2k64rNdj9lsSTk+ZAj6ROcc35djh+2u8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=CcZ0cvtm; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="CcZ0cvtm" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 206DFC4AF0C; Fri, 2 Aug 2024 00:43:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1722559391; bh=DqMDZ/+cWkW/MQt6snJKfAxQMZLfZafULDqWCEsrGjE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=CcZ0cvtm3iviePhBC2csX/OmEVvnkyUSux2AdC6qQ718KGrvLQcH9FHvMuQgNBFf6 yYqaas4G5VStUTero6h+1r2+8hYJXrXWxfR0bgdEoCDNMNGyCsmKKIneyG/I3iSpg1 xyz3yUKRq0ZzcLdHvAHBEZkauWp9rlO4QE71fo7aUYiPcs5Dy5t4odImG7j74lu1Gx oS0MaAJ8t6WwR87ugKnrtuv6JkUCNZmLIOWC6lcshCsLucu76IAqwY7j6ze8KY39YN sV+JtQ2LJVpTc7hOR/a4qcsJBU9Mv1r4Np/LCADGXAwO/142oTv6bPcpc+tsCDaJc5 kWu3+qhpFj+tQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id BF25FCE0A01; Thu, 1 Aug 2024 17:43:10 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH rcu 02/11] rcuscale: Dump stacks of stalled rcu_scale_writer() instances Date: Thu, 1 Aug 2024 17:42:59 -0700 Message-Id: <20240802004308.4134731-2-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <917e8cc8-8688-428a-9122-25544c5cc101@paulmck-laptop> References: <917e8cc8-8688-428a-9122-25544c5cc101@paulmck-laptop> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 This commit improves debuggability by dumping the stacks of rcu_scale_writer() instances that have not completed in a reasonable timeframe. These stacks are dumped remotely, but they will be accurate in the thus-far common case where the stalled rcu_scale_writer() instances are blocked. Signed-off-by: Paul E. McKenney --- kernel/rcu/rcuscale.c | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/rcuscale.c b/kernel/rcu/rcuscale.c index 3269dd9c639f7..c34a8e64edc30 100644 --- a/kernel/rcu/rcuscale.c +++ b/kernel/rcu/rcuscale.c @@ -39,6 +39,7 @@ #include #include #include +#include #include "rcu.h" @@ -111,6 +112,7 @@ static struct task_struct **reader_tasks; static struct task_struct *shutdown_task; static u64 **writer_durations; +static bool *writer_done; static int *writer_n_durations; static atomic_t n_rcu_scale_reader_started; static atomic_t n_rcu_scale_writer_started; @@ -524,6 +526,7 @@ rcu_scale_writer(void *arg) started = true; if (!done && i >= MIN_MEAS && time_after(jiffies, jdone)) { done = true; + WRITE_ONCE(writer_done[me], true); sched_set_normal(current, 0); pr_alert("%s%s rcu_scale_writer %ld has %d measurements\n", scale_type, SCALE_FLAG, me, MIN_MEAS); @@ -549,6 +552,19 @@ rcu_scale_writer(void *arg) if (done && !alldone && atomic_read(&n_rcu_scale_writer_finished) >= nrealwriters) alldone = true; + if (done && !alldone && time_after(jiffies, jdone + HZ * 60)) { + static atomic_t dumped; + int i; + + if (!atomic_xchg(&dumped, 1)) { + for (i = 0; i < nrealwriters; i++) { + if (writer_done[i]) + continue; + pr_info("%s: Task %ld flags writer %d:\n", __func__, me, i); + sched_show_task(writer_tasks[i]); + } + } + } if (started && !alldone && i < MAX_MEAS - 1) i++; rcu_scale_wait_shutdown(); @@ -1015,10 +1031,11 @@ rcu_scale_init(void) } while (atomic_read(&n_rcu_scale_reader_started) < nrealreaders) schedule_timeout_uninterruptible(1); - writer_tasks = kcalloc(nrealwriters, sizeof(reader_tasks[0]), GFP_KERNEL); + writer_tasks = kcalloc(nrealwriters, sizeof(writer_tasks[0]), GFP_KERNEL); writer_durations = kcalloc(nrealwriters, sizeof(*writer_durations), GFP_KERNEL); writer_n_durations = kcalloc(nrealwriters, sizeof(*writer_n_durations), GFP_KERNEL); - if (!writer_tasks || !writer_durations || !writer_n_durations) { + writer_done = kcalloc(nrealwriters, sizeof(writer_done[0]), GFP_KERNEL); + if (!writer_tasks || !writer_durations || !writer_n_durations || !writer_done) { SCALEOUT_ERRSTRING("out of memory"); firsterr = -ENOMEM; goto unwind; From patchwork Fri Aug 2 00:43:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 13750985 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6B204168DA; Fri, 2 Aug 2024 00:43:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722559391; cv=none; b=G6qZMS6glDeu726f2cFydYspwpVTvXRNzn6yBDcqB59uOwYk8Aid7PNz9maLzVpa94hj0JAnQyAld+RjFM0dyCkKkGNYCzNemeX0epvknWWXOzzaA8mB3ssCQ5IBIRahu3Vl5Bk2gLjsVwlWYxVJR6DEAUosfcPGuQEIXb9XoFs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722559391; c=relaxed/simple; bh=WaiDAqrc+b+wo00YSTTvMD3HfVQ61YAzmXrAlI6Fp8Q=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=hit9AtoigyCAQ4yC3G+KJB5Jq0xioz2gH4PvzggrJpIv5G92H1Roh4TEtBNGFoOBmLRk5eCAe641N5Yv1kfLRJSTkfytJH4KH99jHVO8pe7vP0xLRWQFx8iRhPU2Va7gPNi0wqvuC2LdToO/h44r0kCsPB9Ppal6uopERtSiGAo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=gVshaJlX; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="gVshaJlX" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2654EC4AF0A; Fri, 2 Aug 2024 00:43:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1722559391; bh=WaiDAqrc+b+wo00YSTTvMD3HfVQ61YAzmXrAlI6Fp8Q=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=gVshaJlXSvLX1piW3D/+90Q478VYOhywWGiWIUa9J/IUpvm5nUj27eZX+8bK2Gmvb ri1GSJKNY6qYKhZKPwA7nV+KP/5N0aCnKTkZmHFD6fQh/cQmmuAKPv2Scaxej7vqtG LoROBImkBtO4Sr3+id86NtvXfEQEAAjIbaZl+imBeF2g8oDaRdtxoE2N7WuhRr1okX XK9CTJyD+TwnIr+8HNoR41zzARzHbBEN50tP3Tc+eN0ICAlscfs6j06GIT7nfnMjD8 Q/TN+aCWiW1GE2m+qhqaRgDEHC2y5MUIbYQLvqzSHwsFHFIk/InOxRRimQBY9E4qXb Ag9sYKhzQCXFw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id C3624CE0BC3; Thu, 1 Aug 2024 17:43:10 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH rcu 03/11] rcuscale: Dump grace-period statistics when rcu_scale_writer() stalls Date: Thu, 1 Aug 2024 17:43:00 -0700 Message-Id: <20240802004308.4134731-3-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <917e8cc8-8688-428a-9122-25544c5cc101@paulmck-laptop> References: <917e8cc8-8688-428a-9122-25544c5cc101@paulmck-laptop> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 This commit adds a .stats function pointer to the rcu_scale_ops structure, and if this is non-NULL, it is invoked after stack traces are dumped in response to a rcu_scale_writer() stall. Signed-off-by: Paul E. McKenney --- kernel/rcu/rcuscale.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/kernel/rcu/rcuscale.c b/kernel/rcu/rcuscale.c index c34a8e64edc30..ddfb96e5a4e1e 100644 --- a/kernel/rcu/rcuscale.c +++ b/kernel/rcu/rcuscale.c @@ -145,6 +145,7 @@ struct rcu_scale_ops { void (*sync)(void); void (*exp_sync)(void); struct task_struct *(*rso_gp_kthread)(void); + void (*stats)(void); const char *name; }; @@ -226,6 +227,11 @@ static void srcu_scale_synchronize(void) synchronize_srcu(srcu_ctlp); } +static void srcu_scale_stats(void) +{ + srcu_torture_stats_print(srcu_ctlp, scale_type, SCALE_FLAG); +} + static void srcu_scale_synchronize_expedited(void) { synchronize_srcu_expedited(srcu_ctlp); @@ -243,6 +249,7 @@ static struct rcu_scale_ops srcu_ops = { .gp_barrier = srcu_rcu_barrier, .sync = srcu_scale_synchronize, .exp_sync = srcu_scale_synchronize_expedited, + .stats = srcu_scale_stats, .name = "srcu" }; @@ -272,6 +279,7 @@ static struct rcu_scale_ops srcud_ops = { .gp_barrier = srcu_rcu_barrier, .sync = srcu_scale_synchronize, .exp_sync = srcu_scale_synchronize_expedited, + .stats = srcu_scale_stats, .name = "srcud" }; @@ -563,6 +571,8 @@ rcu_scale_writer(void *arg) pr_info("%s: Task %ld flags writer %d:\n", __func__, me, i); sched_show_task(writer_tasks[i]); } + if (cur_ops->stats) + cur_ops->stats(); } } if (started && !alldone && i < MAX_MEAS - 1) From patchwork Fri Aug 2 00:43:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 13750982 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6B236171BB; Fri, 2 Aug 2024 00:43:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722559391; cv=none; b=Xia9J1H7Ev4IDLlFBwoPhMgh4uiESQby1onFB+WRn+lbj3IxT73+q743sA+HKjsw2qaHQM2+/XA2LThkwma8oPS8If4vwPiOmowjKtnsnlTHp29yEfCf8hb41+PNqErB58jaIftChS3Dy6NhsaScE7S6f+AXRQS0NCj4abJndZI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722559391; c=relaxed/simple; bh=ii3Lw31Du2kvj80qfw2EEctLsKo4MDiaG/6o1ncMupY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=tM9SZDfnItqaVaTHjg9D13iTHbyTFwuKfiyYsoah900+f5kO2P/yrX+0uoNJIbMEehSXX1RkDg9oyJTanQOM1hHhUYv8nkIGVORejsTf6k6YdMDfajlIRAMlhqM0BdFPfpKkGB1GNMr9c7IPTz6iAhtp56yCsYLGf2Dsc47X0KQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=aMeU4s/a; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="aMeU4s/a" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 307DBC4AF0E; Fri, 2 Aug 2024 00:43:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1722559391; bh=ii3Lw31Du2kvj80qfw2EEctLsKo4MDiaG/6o1ncMupY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=aMeU4s/aSSRJlqfRV4FMOOjAF7clBboARTEy5Yoq1lkRyTdQshKvq/daAIp31TMTp OrPHWLq4ZUks+fxlzu7XuBf4D3ktpEusH04uZdcdyxy83jAdDvIkZeSz2yFI0LO7Rx RSOTGUhyiJ0oHZPZCw8WkFnwcyXRz4gwrM9WQukAQsTid1zFaHcgiCWzynRrXu143Y yZv38O/xnWQ5FJ3WV0f4TPRNkOuoRpF5MoDZetjCipgJe320ZyDSPn4JOpixeirvTH 0DEzKSPzNrzWRD9FRKGMyAsbC1HdvL+iJ7kvYsajRsCyZwYK/oeysKhA4XfBPMewuD BUJgOSVEP33jw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id C641FCE0E0B; Thu, 1 Aug 2024 17:43:10 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH rcu 04/11] rcu: Mark callbacks not currently participating in barrier operation Date: Thu, 1 Aug 2024 17:43:01 -0700 Message-Id: <20240802004308.4134731-4-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <917e8cc8-8688-428a-9122-25544c5cc101@paulmck-laptop> References: <917e8cc8-8688-428a-9122-25544c5cc101@paulmck-laptop> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 RCU keeps a count of the number of callbacks that the current rcu_barrier() is waiting on, but there is currently no easy way to work out which callback is stuck. One way to do this is to mark idle RCU-barrier callbacks by making the ->next pointer point to the callback itself, and this commit does just that. Later commits will use this for debug output. Signed-off-by: Paul E. McKenney --- kernel/rcu/tree.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 77b5b39e19a80..930846f06bee5 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -4383,6 +4383,7 @@ static void rcu_barrier_callback(struct rcu_head *rhp) { unsigned long __maybe_unused s = rcu_state.barrier_sequence; + rhp->next = rhp; // Mark the callback as having been invoked. if (atomic_dec_and_test(&rcu_state.barrier_cpu_count)) { rcu_barrier_trace(TPS("LastCB"), -1, s); complete(&rcu_state.barrier_completion); @@ -5404,6 +5405,8 @@ static void __init rcu_init_one(void) while (i > rnp->grphi) rnp++; per_cpu_ptr(&rcu_data, i)->mynode = rnp; + per_cpu_ptr(&rcu_data, i)->barrier_head.next = + &per_cpu_ptr(&rcu_data, i)->barrier_head; rcu_boot_init_percpu_data(i); } } From patchwork Fri Aug 2 00:43:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 13750986 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8DD3D17547; Fri, 2 Aug 2024 00:43:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722559391; cv=none; b=GSjobiXWEW/KSVqjALm0iy56bCwiiUpYFMPYYVXsfIjdMelJqU/0nKZwMJmkV0qkJMb2MPGBKhIlzryZ1hS1pLbgQP2nxuGZtuLQd53JYoSenVpGTgVGmloGtN6vMHXnky1ZEbU9nW1IOiT9QWlbekle3UBIyUcZ5BYA/FM3EUs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722559391; c=relaxed/simple; bh=YyIr4NQsi1NoigTr7BSqbvmIgl5FYqqqdCsPh4aHIdY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=tZAMKqqnlu0ueHLJsMA03QRSi3ZSQhKb6NzW5A4Ylp38hEYlpjQFSsT70MpTgMR+CEqk1UAzmCLrnmcfRBrV8MhSX5i69HbXm8ln8TEdBmiyGqA0Clx+G2sjHtndCa7VIgxhHrgloW+ZV2304AONpg60VqDIjaLWu4RaRnON0nc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=FXJx/+bK; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="FXJx/+bK" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3C6C2C4AF0F; Fri, 2 Aug 2024 00:43:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1722559391; bh=YyIr4NQsi1NoigTr7BSqbvmIgl5FYqqqdCsPh4aHIdY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=FXJx/+bKPMGudXqBCHmmQnyn4qjw/tsJw1EhJadZ9La7z5vZraNExkFghbpjaRJEI 7YGxr4fSz1ZzrK9GAg60x9+YNp11O5gXy8fZcvV7wnfwbqV7Ju1DkO9xZUqWPFRnlu C9+i2ez+dnD6f6insr2AcYuULd6yM0un4uv2pCLx4xcF+vGy5swIzMDNz4Jtozaiin Zv/i0W1IM5ycDC3+k0IFobaGMwk6J2IKvkkUuLX6zb+GjatKxiu42lFaecvk6GvHcU pjiMM11I3T/Z3FKXaLeWeU1a+UCtnVU6sAW8n0J4gf6AIiYS/G2JADQkjiZpH5q6/z 3rqE0N+VtS+jA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id C8A89CE0F69; Thu, 1 Aug 2024 17:43:10 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH rcu 05/11] rcuscale: Print detailed grace-period and barrier diagnostics Date: Thu, 1 Aug 2024 17:43:02 -0700 Message-Id: <20240802004308.4134731-5-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <917e8cc8-8688-428a-9122-25544c5cc101@paulmck-laptop> References: <917e8cc8-8688-428a-9122-25544c5cc101@paulmck-laptop> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 This commit uses the new rcu_tasks_torture_stats_print(), rcu_tasks_trace_torture_stats_print(), and rcu_tasks_rude_torture_stats_print() functions in order to provide detailed diagnostics on grace-period, callback, and barrier state when rcu_scale_writer() hangs. [ paulmck: Apply kernel test robot feedback. ] Signed-off-by: Paul E. McKenney --- kernel/rcu/rcuscale.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/kernel/rcu/rcuscale.c b/kernel/rcu/rcuscale.c index ddfb96e5a4e1e..933014b381ec0 100644 --- a/kernel/rcu/rcuscale.c +++ b/kernel/rcu/rcuscale.c @@ -298,6 +298,11 @@ static void tasks_scale_read_unlock(int idx) { } +static void rcu_tasks_scale_stats(void) +{ + rcu_tasks_torture_stats_print(scale_type, SCALE_FLAG); +} + static struct rcu_scale_ops tasks_ops = { .ptype = RCU_TASKS_FLAVOR, .init = rcu_sync_scale_init, @@ -310,6 +315,7 @@ static struct rcu_scale_ops tasks_ops = { .sync = synchronize_rcu_tasks, .exp_sync = synchronize_rcu_tasks, .rso_gp_kthread = get_rcu_tasks_gp_kthread, + .stats = IS_ENABLED(CONFIG_TINY_RCU) ? NULL : rcu_tasks_scale_stats, .name = "tasks" }; @@ -336,6 +342,11 @@ static void tasks_rude_scale_read_unlock(int idx) { } +static void rcu_tasks_rude_scale_stats(void) +{ + rcu_tasks_rude_torture_stats_print(scale_type, SCALE_FLAG); +} + static struct rcu_scale_ops tasks_rude_ops = { .ptype = RCU_TASKS_RUDE_FLAVOR, .init = rcu_sync_scale_init, @@ -346,6 +357,7 @@ static struct rcu_scale_ops tasks_rude_ops = { .sync = synchronize_rcu_tasks_rude, .exp_sync = synchronize_rcu_tasks_rude, .rso_gp_kthread = get_rcu_tasks_rude_gp_kthread, + .stats = IS_ENABLED(CONFIG_TINY_RCU) ? NULL : rcu_tasks_rude_scale_stats, .name = "tasks-rude" }; @@ -374,6 +386,11 @@ static void tasks_trace_scale_read_unlock(int idx) rcu_read_unlock_trace(); } +static void rcu_tasks_trace_scale_stats(void) +{ + rcu_tasks_trace_torture_stats_print(scale_type, SCALE_FLAG); +} + static struct rcu_scale_ops tasks_tracing_ops = { .ptype = RCU_TASKS_FLAVOR, .init = rcu_sync_scale_init, @@ -386,6 +403,7 @@ static struct rcu_scale_ops tasks_tracing_ops = { .sync = synchronize_rcu_tasks_trace, .exp_sync = synchronize_rcu_tasks_trace, .rso_gp_kthread = get_rcu_tasks_trace_gp_kthread, + .stats = IS_ENABLED(CONFIG_TINY_RCU) ? NULL : rcu_tasks_trace_scale_stats, .name = "tasks-tracing" }; From patchwork Fri Aug 2 00:43:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 13750989 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AF42317BB7; Fri, 2 Aug 2024 00:43:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722559391; cv=none; b=EaO2StBo4fsM5udS4rjJrKFVm8gWGE69l5FtVmHX4j+aMFxkcETvjVC1uK4TAjeIlG3X5rQ9vnAstdjcG/LF054BO+esi+GSYydprOKjd7U4aAf9izQpgCMLBsuOEVhmb3jRJSS42JFdNssPOwxtV8WPuZMYpJk6agQzm7Zeh+k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722559391; c=relaxed/simple; bh=AEzkTPCKcNGzv5Un7TZuguhoYqpHfaG2unUtIDA5KrA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=JSfGwciN5yq7XWr1Tw/2vVBlJuvKqfZbH6iu2QI/l2vAl7EKroXqAZDmIeXx8ulbG8xkHNL9fZ45T867NYKBRgk2vu8V5XHhKdbUl3+rixQT/be2KF1hnmDWkiS+m71XYiDVRwz/uejpDjl48XojLmjQTOAwEqT83xhyBKmxMDw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=UEzGCdj3; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="UEzGCdj3" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6564CC4AF11; Fri, 2 Aug 2024 00:43:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1722559391; bh=AEzkTPCKcNGzv5Un7TZuguhoYqpHfaG2unUtIDA5KrA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=UEzGCdj3zGWsz6VSJzTbOsaxJUwMoKotUWDArHHs2FuhtYnnF1l5Lp88f9+8nBbt6 P/TRaFkM5RgnTzSe9fA/JeiIIFflq9jfyZO7aVOFhGIjozs2qudAOC4Jzf70b3gCOw N7o+6kDaF4uTVQhtdBRgdZVxBZterodI9/ARJ1zBohjXKfIDgYSsaQwmO9ThBDJjSA fikkQcgErkv6CMrU136FHmEf615Ee+GKYl2PFxQ6HZ55VDUbqQWqpUEpAlL83V3z0h YQyghbRlaen5yseQa5c0KSXuu7xwZJGEyfMq9SCiZo0jYXQSu//yS96sNPpHh3CxAE YGKMbmxEYO+IQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id CB57ECE0FA1; Thu, 1 Aug 2024 17:43:10 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH rcu 06/11] rcuscale: Provide clear error when async specified without primitives Date: Thu, 1 Aug 2024 17:43:03 -0700 Message-Id: <20240802004308.4134731-6-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <917e8cc8-8688-428a-9122-25544c5cc101@paulmck-laptop> References: <917e8cc8-8688-428a-9122-25544c5cc101@paulmck-laptop> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Currently, if the rcuscale module's async module parameter is specified for RCU implementations that do not have sync primitives such as call_rcu(), there will be a series of splats due to calls to a NULL pointer. This commit therefore warns of this situation, but switches to non-async testing. Signed-off-by: Paul E. McKenney --- kernel/rcu/rcuscale.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/rcuscale.c b/kernel/rcu/rcuscale.c index 933014b381ec0..315ced63ec105 100644 --- a/kernel/rcu/rcuscale.c +++ b/kernel/rcu/rcuscale.c @@ -525,7 +525,7 @@ rcu_scale_writer(void *arg) schedule_timeout_idle(torture_random(&tr) % writer_holdoff_jiffies + 1); wdp = &wdpp[i]; *wdp = ktime_get_mono_fast_ns(); - if (gp_async) { + if (gp_async && !WARN_ON_ONCE(!cur_ops->async)) { retry: if (!rhp) rhp = kmalloc(sizeof(*rhp), GFP_KERNEL); @@ -597,7 +597,7 @@ rcu_scale_writer(void *arg) i++; rcu_scale_wait_shutdown(); } while (!torture_must_stop()); - if (gp_async) { + if (gp_async && cur_ops->async) { cur_ops->gp_barrier(); } writer_n_durations[me] = i_max + 1; From patchwork Fri Aug 2 00:43:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 13750987 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 99F5917BA4; Fri, 2 Aug 2024 00:43:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722559391; cv=none; b=MuxkPARxVYD6J5flMJY/AVilk7KVNCRdereCGGKjZsKsG7gZlgEXKzZCN87xyFAK2wdpbZVAFRObNymJvn7G4Z1S/vlRusxlChKaHIKwICyYY1s0cJJkSZcRmjTv5AcSEKaJ2SjJeXUGms2z7kNY+zHjQYAC9pahVbJlov/oM70= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722559391; c=relaxed/simple; bh=wtNU9ny0pb6w6vugnHTvWS5lxc3HG5A0ixYSJLK9w1g=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=FqM6iQQPsk3QSKW1+gWmlMY1ljxKcs9yFFWJwqymiPGFy2QT7AwYazRIhN9F0ZlWXg2SFI7OWAtcsnI6FXv8sgl0NFu0ANsuDEqWqKZfSPmpqv2FixH1D2/VqDCzzxUK8RbfZZnraZ30n/slR1fhDcaG+rMPobcBeg+HB1I7lMU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=ul8bPBSU; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="ul8bPBSU" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6E582C4AF15; Fri, 2 Aug 2024 00:43:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1722559391; bh=wtNU9ny0pb6w6vugnHTvWS5lxc3HG5A0ixYSJLK9w1g=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ul8bPBSU2jl+Z3cs5u/wyRDgudETm4TcVugUU2h8u86K132S3wDN8zZa0ZbYCXc6K XoiOLqHRqc0XvVCn8zmNflupW3neXlF/AZh8qsm+74Z2ekP5OT50riSAaxX+44OQeL UCCuTFzbZ/JbA5Fo51cyakhEEXHqmngEzcNnpbrCv1r/a019UrKrFhkUyeFMjYGIH9 YQNFjzFtRntD7HxT8cIoFyXrFYpHkdKstUMl5Y6MediuXNzOZBCNdBOm57Jvv3R6Gw 2RL5EU85qhQ84JkLVa6Dc1sw3yL5ZhaZebNqZL9Fz7DfTDhBttuAzs7lfbLKiW/tGW 4gninT4KMfK4w== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id CDF0FCE0FA9; Thu, 1 Aug 2024 17:43:10 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH rcu 07/11] rcuscale: Make all writer tasks report upon hang Date: Thu, 1 Aug 2024 17:43:04 -0700 Message-Id: <20240802004308.4134731-7-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <917e8cc8-8688-428a-9122-25544c5cc101@paulmck-laptop> References: <917e8cc8-8688-428a-9122-25544c5cc101@paulmck-laptop> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 This commit causes all writer tasks to provide a brief report after a hang has been reported, spaced at one-second intervals. Signed-off-by: Paul E. McKenney --- kernel/rcu/rcuscale.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/kernel/rcu/rcuscale.c b/kernel/rcu/rcuscale.c index 315ced63ec105..a820f11b19444 100644 --- a/kernel/rcu/rcuscale.c +++ b/kernel/rcu/rcuscale.c @@ -483,6 +483,7 @@ rcu_scale_writer(void *arg) unsigned long jdone; long me = (long)arg; struct rcu_head *rhp = NULL; + bool selfreport = false; bool started = false, done = false, alldone = false; u64 t; DEFINE_TORTURE_RANDOM(tr); @@ -593,6 +594,11 @@ rcu_scale_writer(void *arg) cur_ops->stats(); } } + if (!selfreport && time_after(jiffies, jdone + HZ * (70 + me))) { + pr_info("%s: Writer %ld self-report: started %d done %d/%d->%d i %d jdone %lu.\n", + __func__, me, started, done, writer_done[me], atomic_read(&n_rcu_scale_writer_finished), i, jiffies - jdone); + selfreport = true; + } if (started && !alldone && i < MAX_MEAS - 1) i++; rcu_scale_wait_shutdown(); From patchwork Fri Aug 2 00:43:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 13750988 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AF45317BD3; Fri, 2 Aug 2024 00:43:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722559391; cv=none; b=r/jfvWkZo/lXIYZsymbnyB4NiA/dVBftjNCB0WRTYrDZj5QpZbm+bVrjaUgSGm6Mqf+Z/sWxDrInNJMCgxGWjMnKxWb2nb3OVG0kJfldGGwlgOjx4ZNv/RCa66NNv63X5CHN/KeI6qDU0UnrIqYYHaVjWZG666KJBu0zqtpjfyo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722559391; c=relaxed/simple; bh=kxIQmbcr9Bs4WnWw6UbzaEHba/xrsV4JB7mQCevE2OM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=TYN/voCFcj5nX/8p732mqz13fCfrTY4FCD380YCy/UVhiFVijuTxqb4ie5rHHUy6RF/6IYeHUqocrd9p/3nfIdQrSR6FjHYaUH0t8L98H9CG6gXEVh/J8633rUQP649N/9ZptSk9Gqm12k1swKmZh8VLxlHqb0som7Yd8gsAAwQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=bhvWbTD4; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="bhvWbTD4" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 79D50C4AF16; Fri, 2 Aug 2024 00:43:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1722559391; bh=kxIQmbcr9Bs4WnWw6UbzaEHba/xrsV4JB7mQCevE2OM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=bhvWbTD4Z9rNhhvFIR9TzST0qkP+o2j2AWZEJEoEJWSpRiV2C0Ua0bEYBUivxm50x wqy13tCfS1rrVnxlg8eR9JHh4Y9MRokrq7AGtxry4G8pMDdAcIHDcfE37Ey+gZjNC6 1nkl9MBPCADmOTWvd9AZ4pLcz6UBHEE4LI7E7vJgTIMFTAecAeq/yN7f4nOrt9wMeX TJA8mYjZr/AI+fODSY8OAaARwn//V9P2GOy70lz6shaYBOoUAh2gjcB8lYwk7G3UvJ AB2CkFvPViYNwaoBMQfXTC4GxpPlZduJuepmqZUZFf7+8z5zEQSbe7rFjQ9ieBkvC0 FQRb06jWyBkuQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id D0FEFCE1073; Thu, 1 Aug 2024 17:43:10 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH rcu 08/11] rcuscale: Make rcu_scale_writer() tolerate repeated GFP_KERNEL failure Date: Thu, 1 Aug 2024 17:43:05 -0700 Message-Id: <20240802004308.4134731-8-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <917e8cc8-8688-428a-9122-25544c5cc101@paulmck-laptop> References: <917e8cc8-8688-428a-9122-25544c5cc101@paulmck-laptop> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Under some conditions, kmalloc(GFP_KERNEL) allocations have been observed to repeatedly fail. This situation has been observed to cause one of the rcu_scale_writer() instances to loop indefinitely retrying memory allocation for an asynchronous grace-period primitive. The problem is that if memory is short, all the other instances will allocate all available memory before the looping task is awakened from its rcu_barrier*() call. This in turn results in hangs, so that rcuscale fails to complete. This commit therefore removes the tight retry loop, so that when this condition occurs, the affected task is still passing through the full loop with its full set of termination checks. This spreads the risk of indefinite memory-allocation retry failures across all instances of rcu_scale_writer() tasks, which in turn prevents the hangs. Signed-off-by: Paul E. McKenney --- kernel/rcu/rcuscale.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/kernel/rcu/rcuscale.c b/kernel/rcu/rcuscale.c index a820f11b19444..01d48eb753b41 100644 --- a/kernel/rcu/rcuscale.c +++ b/kernel/rcu/rcuscale.c @@ -520,6 +520,8 @@ rcu_scale_writer(void *arg) jdone = jiffies + minruntime * HZ; do { + bool gp_succeeded = false; + if (writer_holdoff) udelay(writer_holdoff); if (writer_holdoff_jiffies) @@ -527,23 +529,24 @@ rcu_scale_writer(void *arg) wdp = &wdpp[i]; *wdp = ktime_get_mono_fast_ns(); if (gp_async && !WARN_ON_ONCE(!cur_ops->async)) { -retry: if (!rhp) rhp = kmalloc(sizeof(*rhp), GFP_KERNEL); if (rhp && atomic_read(this_cpu_ptr(&n_async_inflight)) < gp_async_max) { atomic_inc(this_cpu_ptr(&n_async_inflight)); cur_ops->async(rhp, rcu_scale_async_cb); rhp = NULL; + gp_succeeded = true; } else if (!kthread_should_stop()) { cur_ops->gp_barrier(); - goto retry; } else { kfree(rhp); /* Because we are stopping. */ } } else if (gp_exp) { cur_ops->exp_sync(); + gp_succeeded = true; } else { cur_ops->sync(); + gp_succeeded = true; } t = ktime_get_mono_fast_ns(); *wdp = t - *wdp; @@ -599,7 +602,7 @@ rcu_scale_writer(void *arg) __func__, me, started, done, writer_done[me], atomic_read(&n_rcu_scale_writer_finished), i, jiffies - jdone); selfreport = true; } - if (started && !alldone && i < MAX_MEAS - 1) + if (gp_succeeded && started && !alldone && i < MAX_MEAS - 1) i++; rcu_scale_wait_shutdown(); } while (!torture_must_stop()); From patchwork Fri Aug 2 00:43:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 13750990 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B9DE31804F; Fri, 2 Aug 2024 00:43:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722559391; cv=none; b=JpKlO8xS0C8TIYQMGN4LVPNVnHDYnJzsrTUGXqgQ9FEK7aTAHS16HnPj8ceHPRQWhIzQGUf+5GTSKkM40qDp7mGc+aN2LDAwgAtCD1rpsz87VnoRudXKfDIEh+lc660Xa2AU5s+xcaDc2mfqfDLFslqCIDqYCfbA5Oxhmcl8pso= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722559391; c=relaxed/simple; bh=dza1hueVRZhO/EgWh2ZF/9iuT1tip62LhNQOQgi0Wp4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=KTmvPEOYAuTJ+DQ4pQuGHz1FXHHqVhC8/9dpubzTF/GR15heMvGa7eOVuY0GTYtWwfrZniQr6ZEGNUjpvT53+Rt9PfJtdnKt/g6UJTCb7iDgIBFzZIPWmPS6VkjpBaMldupoJTK6GdQaXwhXGARq40Zxt23r87IGwNDSYew2F2k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=F3ntq0wz; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="F3ntq0wz" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7CA70C4AF0D; Fri, 2 Aug 2024 00:43:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1722559391; bh=dza1hueVRZhO/EgWh2ZF/9iuT1tip62LhNQOQgi0Wp4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=F3ntq0wznyOzH+sPLvAtl185wnMS4ZgsMg7hQ1+ZLeY6KCHCiMZJyxtKzIg66ps4m EvUPqeBqpvrqQhCYowv5AENeR+xjs4eko0RCpzWMNVJiQFdrWS/CaDHYZV08zPAwc8 9wkzn1iiSdA3MOnk1r3AlrJeA+RagbYLTpEpdslWx2YY5qOe1UQrzGFtD2v8kfmDgX Kh7TdFMab4rd3hC9j6z9ngYqvkc0dJ1oer9q2O7JvriV+fW6xjN8PyxvWM1y74J6y6 YaUvBmP6obfDqTkFFT3D3wwIPr94716o8yjYrDjeTl1CR1kmYKnLSTms6na7dq4pTG R1QbykiUGDpAg== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id D3A53CE10D8; Thu, 1 Aug 2024 17:43:10 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH rcu 09/11] rcuscale: Use special allocator for rcu_scale_writer() Date: Thu, 1 Aug 2024 17:43:06 -0700 Message-Id: <20240802004308.4134731-9-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <917e8cc8-8688-428a-9122-25544c5cc101@paulmck-laptop> References: <917e8cc8-8688-428a-9122-25544c5cc101@paulmck-laptop> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 The rcu_scale_writer() function needs only a fixed number of rcu_head structures per kthread, which means that a trivial allocator suffices. This commit therefore uses an llist-based allocator using a fixed array of structures per kthread. This allows aggressive testing of RCU performance without stressing the slab allocators. Signed-off-by: Paul E. McKenney --- kernel/rcu/rcuscale.c | 123 ++++++++++++++++++++++++++++++++++++++---- 1 file changed, 113 insertions(+), 10 deletions(-) diff --git a/kernel/rcu/rcuscale.c b/kernel/rcu/rcuscale.c index 01d48eb753b41..f945f8175e859 100644 --- a/kernel/rcu/rcuscale.c +++ b/kernel/rcu/rcuscale.c @@ -105,6 +105,19 @@ static char *scale_type = "rcu"; module_param(scale_type, charp, 0444); MODULE_PARM_DESC(scale_type, "Type of RCU to scalability-test (rcu, srcu, ...)"); +// Structure definitions for custom fixed-per-task allocator. +struct writer_mblock { + struct rcu_head wmb_rh; + struct llist_node wmb_node; + struct writer_freelist *wmb_wfl; +}; + +struct writer_freelist { + struct llist_head ws_lhg; + struct llist_head ____cacheline_internodealigned_in_smp ws_lhp; + struct writer_mblock *ws_mblocks; +}; + static int nrealreaders; static int nrealwriters; static struct task_struct **writer_tasks; @@ -113,6 +126,7 @@ static struct task_struct *shutdown_task; static u64 **writer_durations; static bool *writer_done; +static struct writer_freelist *writer_freelists; static int *writer_n_durations; static atomic_t n_rcu_scale_reader_started; static atomic_t n_rcu_scale_writer_started; @@ -463,13 +477,52 @@ rcu_scale_reader(void *arg) return 0; } +/* + * Allocate a writer_mblock structure for the specified rcu_scale_writer + * task. + */ +static struct writer_mblock *rcu_scale_alloc(long me) +{ + struct llist_node *llnp; + struct writer_freelist *wflp; + struct writer_mblock *wmbp; + + if (WARN_ON_ONCE(!writer_freelists)) + return NULL; + wflp = &writer_freelists[me]; + if (llist_empty(&wflp->ws_lhp)) { + // ->ws_lhp is private to its rcu_scale_writer task. + wmbp = container_of(llist_del_all(&wflp->ws_lhg), struct writer_mblock, wmb_node); + wflp->ws_lhp.first = &wmbp->wmb_node; + } + llnp = llist_del_first(&wflp->ws_lhp); + if (!llnp) + return NULL; + return container_of(llnp, struct writer_mblock, wmb_node); +} + +/* + * Free a writer_mblock structure to its rcu_scale_writer task. + */ +static void rcu_scale_free(struct writer_mblock *wmbp) +{ + struct writer_freelist *wflp; + + if (!wmbp) + return; + wflp = wmbp->wmb_wfl; + llist_add(&wmbp->wmb_node, &wflp->ws_lhg); +} + /* * Callback function for asynchronous grace periods from rcu_scale_writer(). */ static void rcu_scale_async_cb(struct rcu_head *rhp) { + struct writer_mblock *wmbp = container_of(rhp, struct writer_mblock, wmb_rh); + atomic_dec(this_cpu_ptr(&n_async_inflight)); - kfree(rhp); + rcu_scale_free(wmbp); } /* @@ -482,13 +535,13 @@ rcu_scale_writer(void *arg) int i_max; unsigned long jdone; long me = (long)arg; - struct rcu_head *rhp = NULL; bool selfreport = false; bool started = false, done = false, alldone = false; u64 t; DEFINE_TORTURE_RANDOM(tr); u64 *wdp; u64 *wdpp = writer_durations[me]; + struct writer_mblock *wmbp = NULL; VERBOSE_SCALEOUT_STRING("rcu_scale_writer task started"); WARN_ON(!wdpp); @@ -529,17 +582,18 @@ rcu_scale_writer(void *arg) wdp = &wdpp[i]; *wdp = ktime_get_mono_fast_ns(); if (gp_async && !WARN_ON_ONCE(!cur_ops->async)) { - if (!rhp) - rhp = kmalloc(sizeof(*rhp), GFP_KERNEL); - if (rhp && atomic_read(this_cpu_ptr(&n_async_inflight)) < gp_async_max) { + if (!wmbp) + wmbp = rcu_scale_alloc(me); + if (wmbp && atomic_read(this_cpu_ptr(&n_async_inflight)) < gp_async_max) { atomic_inc(this_cpu_ptr(&n_async_inflight)); - cur_ops->async(rhp, rcu_scale_async_cb); - rhp = NULL; + cur_ops->async(&wmbp->wmb_rh, rcu_scale_async_cb); + wmbp = NULL; gp_succeeded = true; } else if (!kthread_should_stop()) { cur_ops->gp_barrier(); } else { - kfree(rhp); /* Because we are stopping. */ + rcu_scale_free(wmbp); /* Because we are stopping. */ + wmbp = NULL; } } else if (gp_exp) { cur_ops->exp_sync(); @@ -607,6 +661,7 @@ rcu_scale_writer(void *arg) rcu_scale_wait_shutdown(); } while (!torture_must_stop()); if (gp_async && cur_ops->async) { + rcu_scale_free(wmbp); cur_ops->gp_barrier(); } writer_n_durations[me] = i_max + 1; @@ -970,10 +1025,28 @@ rcu_scale_cleanup(void) schedule_timeout_uninterruptible(1); } kfree(writer_durations[i]); + if (writer_freelists) { + int ctr = 0; + struct llist_node *llnp; + struct writer_freelist *wflp = &writer_freelists[i]; + + if (wflp->ws_mblocks) { + llist_for_each(llnp, wflp->ws_lhg.first) + ctr++; + llist_for_each(llnp, wflp->ws_lhp.first) + ctr++; + WARN_ONCE(ctr != gp_async_max, + "%s: ctr = %d gp_async_max = %d\n", + __func__, ctr, gp_async_max); + kfree(wflp->ws_mblocks); + } + } } kfree(writer_tasks); kfree(writer_durations); kfree(writer_n_durations); + kfree(writer_freelists); + writer_freelists = NULL; } /* Do torture-type-specific cleanup operations. */ @@ -1000,8 +1073,9 @@ rcu_scale_shutdown(void *arg) static int __init rcu_scale_init(void) { - long i; int firsterr = 0; + long i; + long j; static struct rcu_scale_ops *scale_ops[] = { &rcu_ops, &srcu_ops, &srcud_ops, TASKS_OPS TASKS_RUDE_OPS TASKS_TRACING_OPS }; @@ -1072,7 +1146,18 @@ rcu_scale_init(void) writer_durations = kcalloc(nrealwriters, sizeof(*writer_durations), GFP_KERNEL); writer_n_durations = kcalloc(nrealwriters, sizeof(*writer_n_durations), GFP_KERNEL); writer_done = kcalloc(nrealwriters, sizeof(writer_done[0]), GFP_KERNEL); - if (!writer_tasks || !writer_durations || !writer_n_durations || !writer_done) { + if (gp_async) { + if (gp_async_max <= 0) { + pr_warn("%s: gp_async_max = %d must be greater than zero.\n", + __func__, gp_async_max); + WARN_ON_ONCE(IS_BUILTIN(CONFIG_RCU_TORTURE_TEST)); + firsterr = -EINVAL; + goto unwind; + } + writer_freelists = kcalloc(nrealwriters, sizeof(writer_freelists[0]), GFP_KERNEL); + } + if (!writer_tasks || !writer_durations || !writer_n_durations || !writer_done || + (gp_async && !writer_freelists)) { SCALEOUT_ERRSTRING("out of memory"); firsterr = -ENOMEM; goto unwind; @@ -1085,6 +1170,24 @@ rcu_scale_init(void) firsterr = -ENOMEM; goto unwind; } + if (writer_freelists) { + struct writer_freelist *wflp = &writer_freelists[i]; + + init_llist_head(&wflp->ws_lhg); + init_llist_head(&wflp->ws_lhp); + wflp->ws_mblocks = kcalloc(gp_async_max, sizeof(wflp->ws_mblocks[0]), + GFP_KERNEL); + if (!wflp->ws_mblocks) { + firsterr = -ENOMEM; + goto unwind; + } + for (j = 0; j < gp_async_max; j++) { + struct writer_mblock *wmbp = &wflp->ws_mblocks[j]; + + wmbp->wmb_wfl = wflp; + llist_add(&wmbp->wmb_node, &wflp->ws_lhp); + } + } firsterr = torture_create_kthread(rcu_scale_writer, (void *)i, writer_tasks[i]); if (torture_init_error(firsterr)) From patchwork Fri Aug 2 00:43:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 13750991 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B9DAF17C95; Fri, 2 Aug 2024 00:43:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722559391; cv=none; b=RnnBJIca0HFqgP8qYkSkDI93aHhZq61sZ+jnvD8YsR49CNZMSLWF+W0/vki9LpmW0T9/1ZmdtTdzNgwWPVuBJx/+lotT33WqwysSV+FLO0G91tWaha8+CEjHJrfZprOOaQzk/eXxtj0H55I4osMfwLfSL/0AkbYBh8QQF2KRVbE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722559391; c=relaxed/simple; bh=Touz5IJtnmv5BN3ovY+yPXkt+2bTcKDGka/zHv5HyVE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=uQYwMBVfylzsTQXTZThZX7JCU23m1SqJscryn2s0FJ8WSIatREnX6vl015qHpuILAbEg7uLGuos8HHe4us+m8e2wnrMja34dT1nvGXQtwasL5+QwBVe5JH4WATJOm65KffScwGuxxUTezmcBUMaClLu2ei1tgnPWfGWAMhfWDZg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=QtQAd0xE; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="QtQAd0xE" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7ED2BC4AF19; Fri, 2 Aug 2024 00:43:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1722559391; bh=Touz5IJtnmv5BN3ovY+yPXkt+2bTcKDGka/zHv5HyVE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=QtQAd0xE/H0Wz0rg2pZtEAxWDGcqxYEDhrLfq12j3EhM5cyiXqRbmcyTFiI2ruUrd 3ulvPmC3rpbArtKxtGRgkuiU/OjNh7J0Zf1BeYoEdCfe4dZaOKJLMTIolhy2kA/Wlu qVtVaZ5eH9TVqh1mK0GYwL7RBiXIJBLoxeagAPnfW1rWzi1zWYb9SsZ//wGxMjN5YQ juzqsHrGPR0W4pgx9snWqCm7TanSDDIQeiv5p+rkNr9WVEn7iHVBihWyabl2Fa2X3O ttWpPAsgpqnYUuFcvQTCvTDv4S3UpEDpv64M6+ntzNMK8j/W4B73aiETiYWtyfNPZb +EwFnjvDUoYNQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id D6443CE10F9; Thu, 1 Aug 2024 17:43:10 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH rcu 10/11] rcuscale: NULL out top-level pointers to heap memory Date: Thu, 1 Aug 2024 17:43:07 -0700 Message-Id: <20240802004308.4134731-10-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <917e8cc8-8688-428a-9122-25544c5cc101@paulmck-laptop> References: <917e8cc8-8688-428a-9122-25544c5cc101@paulmck-laptop> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Currently, if someone modprobes and rmmods rcuscale successfully, but the next run errors out during the modprobe, non-NULL pointers to freed memory will remain. If the run after that also errors out during the modprobe, there will be double-free bugs. This commit therefore NULLs out top-level pointers to memory that has just been freed. Signed-off-by: Paul E. McKenney --- kernel/rcu/rcuscale.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/kernel/rcu/rcuscale.c b/kernel/rcu/rcuscale.c index f945f8175e859..13d379c179248 100644 --- a/kernel/rcu/rcuscale.c +++ b/kernel/rcu/rcuscale.c @@ -819,6 +819,7 @@ kfree_scale_cleanup(void) torture_stop_kthread(kfree_scale_thread, kfree_reader_tasks[i]); kfree(kfree_reader_tasks); + kfree_reader_tasks = NULL; } torture_cleanup_end(); @@ -987,6 +988,7 @@ rcu_scale_cleanup(void) torture_stop_kthread(rcu_scale_reader, reader_tasks[i]); kfree(reader_tasks); + reader_tasks = NULL; } if (writer_tasks) { @@ -1043,8 +1045,11 @@ rcu_scale_cleanup(void) } } kfree(writer_tasks); + writer_tasks = NULL; kfree(writer_durations); + writer_durations = NULL; kfree(writer_n_durations); + writer_n_durations = NULL; kfree(writer_freelists); writer_freelists = NULL; } From patchwork Fri Aug 2 00:43:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 13750992 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C81DA18E06; Fri, 2 Aug 2024 00:43:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722559391; cv=none; b=aeh3zrVxtt5TFgXjqSERqV/Cni6TMdX5qjz++eyvERNUL/WBjiPH/pPXFQ+bMNoYfpcxQtg+SjaLcTAawBA47HckDAD2ysnYGP7JBHtGfFjOy9mhkl82pOzxVV+tyEbsK+6goPZmJENuxV5SVxzVNYHKl0EkWjMW9er73ge8TNg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722559391; c=relaxed/simple; bh=0uXJ7XpqMMOTU5D9L7DX8BbJ/THNXMlQczyB3O6xc1c=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Q6MpY4pBnTk04hdDNcrjUKySPIrMw8NM2TtGL19DMZ0UX0PzY498jQlcnUi7AoTeFsUaOZ5dpWao4B6RGNWD3NCVlO6ki+3D+3nPY67SD0Gk2o7j96PkGWUjUqO5MuZXvy8pj+BPcg3Wz7qKy+w9U9TV1U/AlyOxLYp6AfkWC4U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=OIqaK0Xk; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="OIqaK0Xk" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8251FC4AF17; Fri, 2 Aug 2024 00:43:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1722559391; bh=0uXJ7XpqMMOTU5D9L7DX8BbJ/THNXMlQczyB3O6xc1c=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=OIqaK0XkfevEFyIoWZ+h2IqA31zqlYISem5Wb/IElhq3wvGBRVM0hFZDUNkb6UKjC c5ckoYj31+3opPDh5z9cGrQaGmgPetTc6TuyVC7W/Zxux6hRNTPENrPnukR7l+gtrD 8iw9Lc2kDbbotzikFwHm51fFi4EGFmUGKrIUACQeVcIDU9waFpviRWQDzOst+A9kYP g2jCzi1rFQW0KNniIT+i8iHyxDm+2Rc/wpX+dNkzvTkMLzVHGvLtMpZNnyda6YaUR+ j5WmyzvrG2pI478uSPMln9zNfkecs/pzwzo9roGRl7gZQsTKlruMo7OTGsbFNW4S0k 6GKRYfsQZhD5A== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id D8C1BCE11CD; Thu, 1 Aug 2024 17:43:10 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Vlastimil Babka Subject: [PATCH rcu 11/11] rcuscale: Count outstanding callbacks per-task rather than per-CPU Date: Thu, 1 Aug 2024 17:43:08 -0700 Message-Id: <20240802004308.4134731-11-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <917e8cc8-8688-428a-9122-25544c5cc101@paulmck-laptop> References: <917e8cc8-8688-428a-9122-25544c5cc101@paulmck-laptop> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 The current rcu_scale_writer() asynchronous grace-period testing uses a per-CPU counter to track the number of outstanding callbacks. This is subject to CPU-imbalance errors when tasks migrate from one CPU to another between the time that the counter is incremented and the callback is queued, and additionally in kernels configured such that callbacks can be invoked on some CPU other than the one that queued it. This commit therefore arranges for per-task callback counts, thus avoiding any issues with migration of either tasks or callbacks. Reported-by: Vlastimil Babka Signed-off-by: Paul E. McKenney --- kernel/rcu/rcuscale.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/kernel/rcu/rcuscale.c b/kernel/rcu/rcuscale.c index 13d379c179248..b1c50df142eba 100644 --- a/kernel/rcu/rcuscale.c +++ b/kernel/rcu/rcuscale.c @@ -114,6 +114,7 @@ struct writer_mblock { struct writer_freelist { struct llist_head ws_lhg; + atomic_t ws_inflight; struct llist_head ____cacheline_internodealigned_in_smp ws_lhp; struct writer_mblock *ws_mblocks; }; @@ -136,7 +137,6 @@ static u64 t_rcu_scale_writer_started; static u64 t_rcu_scale_writer_finished; static unsigned long b_rcu_gp_test_started; static unsigned long b_rcu_gp_test_finished; -static DEFINE_PER_CPU(atomic_t, n_async_inflight); #define MAX_MEAS 10000 #define MIN_MEAS 100 @@ -520,8 +520,9 @@ static void rcu_scale_free(struct writer_mblock *wmbp) static void rcu_scale_async_cb(struct rcu_head *rhp) { struct writer_mblock *wmbp = container_of(rhp, struct writer_mblock, wmb_rh); + struct writer_freelist *wflp = wmbp->wmb_wfl; - atomic_dec(this_cpu_ptr(&n_async_inflight)); + atomic_dec(&wflp->ws_inflight); rcu_scale_free(wmbp); } @@ -541,6 +542,7 @@ rcu_scale_writer(void *arg) DEFINE_TORTURE_RANDOM(tr); u64 *wdp; u64 *wdpp = writer_durations[me]; + struct writer_freelist *wflp = &writer_freelists[me]; struct writer_mblock *wmbp = NULL; VERBOSE_SCALEOUT_STRING("rcu_scale_writer task started"); @@ -584,8 +586,8 @@ rcu_scale_writer(void *arg) if (gp_async && !WARN_ON_ONCE(!cur_ops->async)) { if (!wmbp) wmbp = rcu_scale_alloc(me); - if (wmbp && atomic_read(this_cpu_ptr(&n_async_inflight)) < gp_async_max) { - atomic_inc(this_cpu_ptr(&n_async_inflight)); + if (wmbp && atomic_read(&wflp->ws_inflight) < gp_async_max) { + atomic_inc(&wflp->ws_inflight); cur_ops->async(&wmbp->wmb_rh, rcu_scale_async_cb); wmbp = NULL; gp_succeeded = true;