From patchwork Fri Aug 16 07:02:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neeraj Upadhyay X-Patchwork-Id: 13765530 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EC34075817; Fri, 16 Aug 2024 07:03:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723791839; cv=none; b=JAC9NqoTGuROqB3kGWqTnr7jZBsx5QTfTC+M0ZsOioEvtcEm8zDp+TrCWw3HOPQ7xMY9Z+pbzDt2YKQ8qV80xhTN6rAw0H/55YXv+s7kv1XpUmquu50UFGFeloMe7/uZObw8OqOLMqy/xBEYT8hfzmksmKvRvBN6A47IGJ20HCc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723791839; c=relaxed/simple; bh=CvlFaPcyQA+Nkp6oUPDzrGfQYWuO6ut8DozmNgSRw4Y=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=WoZZqc5TrMvs+gToKpfQ3b38NFLJpOcVvT1Iqx94bOmj9irJBz3dFw09okBvyILLJNL7KjegYNWf0hlAFoOpEu4VyN+1/yksF+FnDp2AkDQNDjqJHDzJXThlldRe+7JhUmvm8/o/M4LlTsO28qkrfFnx91dXpwrlUgjhGmgSPLw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=FgvXIuse; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="FgvXIuse" Received: by smtp.kernel.org (Postfix) with ESMTPSA id E02E5C32782; Fri, 16 Aug 2024 07:03:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723791837; bh=CvlFaPcyQA+Nkp6oUPDzrGfQYWuO6ut8DozmNgSRw4Y=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=FgvXIuseQaCUaFPWNzTYBnjU/MhUF0BKKnvG8MVdlmpfTgBWnbgda3+G9IZMdH1J0 T66pAcIYdWFyFF2+fe4pH8k+9+PNeZzQfX63Z2F0w3kqJmCtJgpoBNT6xTL8nv9V35 LpipqI7cOaHjQjtFvzi5dXLfs+t2M9H9OUaDiU98owo3MomdbX6ailGgKMqMmqSQpo OMezHVzhCzAKKm7U236da9gdhejt0x7Iz4czloftE2zIKDbC+/V20vNP+UxtjvB64t +GL3pPBRblUFaT46MoG2tSpkt8zTUna4EzRpxxSqwjLj4+AUVKUn8LFtjTXZK5/40i nNs6KPL2m3dtg== From: neeraj.upadhyay@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, paulmck@kernel.org, neeraj.upadhyay@kernel.org, neeraj.upadhyay@amd.com, boqun.feng@gmail.com, joel@joelfernandes.org, urezki@gmail.com, frederic@kernel.org Subject: [PATCH rcu 01/14] refscale: Add TINY scenario Date: Fri, 16 Aug 2024 12:32:43 +0530 Message-Id: <20240816070256.60993-1-neeraj.upadhyay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240816070209.GA60666@neeraj.linux> References: <20240816070209.GA60666@neeraj.linux> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" This commit adds a TINY scenario in order to support tests of Tiny RCU and Tiny SRCU. Signed-off-by: Paul E. McKenney Signed-off-by: Neeraj Upadhyay --- .../rcutorture/configs/refscale/TINY | 20 +++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100644 tools/testing/selftests/rcutorture/configs/refscale/TINY diff --git a/tools/testing/selftests/rcutorture/configs/refscale/TINY b/tools/testing/selftests/rcutorture/configs/refscale/TINY new file mode 100644 index 000000000000..759343980b80 --- /dev/null +++ b/tools/testing/selftests/rcutorture/configs/refscale/TINY @@ -0,0 +1,20 @@ +CONFIG_SMP=n +CONFIG_PREEMPT_NONE=y +CONFIG_PREEMPT_VOLUNTARY=n +CONFIG_PREEMPT=n +CONFIG_PREEMPT_DYNAMIC=n +#CHECK#CONFIG_PREEMPT_RCU=n +CONFIG_HZ_PERIODIC=n +CONFIG_NO_HZ_IDLE=y +CONFIG_NO_HZ_FULL=n +CONFIG_HOTPLUG_CPU=n +CONFIG_SUSPEND=n +CONFIG_HIBERNATION=n +CONFIG_RCU_NOCB_CPU=n +CONFIG_DEBUG_LOCK_ALLOC=n +CONFIG_PROVE_LOCKING=n +CONFIG_RCU_BOOST=n +CONFIG_DEBUG_OBJECTS_RCU_HEAD=n +CONFIG_RCU_EXPERT=y +CONFIG_KPROBES=n +CONFIG_FTRACE=n From patchwork Fri Aug 16 07:02:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neeraj Upadhyay X-Patchwork-Id: 13765531 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 86D2D12C549; Fri, 16 Aug 2024 07:04:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723791845; cv=none; b=KOPTXQvLofi5gziYwwr8qZH9d3uDAlTjlWzY4QPpNR8yIIFyhIII9aa4u53ORZ3gK2U9y8a0KqT7gOwAXi464/MWd4T+EKahARMn0sfjd+gYXlORIMJGnqonNhX9dgT39hKFWcIOWczumzNMVNEpMw9sXhrMvmdEN1Xaa9RJ+8Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723791845; c=relaxed/simple; bh=Onny6ymZLcW/+CIvxBmp98gGmBO/rcJMG4CCHHJNEDs=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=ox6l4+QT9Vzr9xYM6Rri12p6EUsKb4KlKRGy4rREYV53bwnRW8PgejFnlzsJP5+rbAAdKwXVa8WRmFEBTGU0Cgz3FzIfiPp1VNhlEXDohZa1BVz4Ih2ynlnvFqfSjy9MglZ8igdb1kPOYleb810LPuTunAJFDwSEkUzdEpcH9Hk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=PdRvd0na; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="PdRvd0na" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 73C72C4AF0B; Fri, 16 Aug 2024 07:03:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723791845; bh=Onny6ymZLcW/+CIvxBmp98gGmBO/rcJMG4CCHHJNEDs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=PdRvd0naZTFrNDqaNb1PcQccdXC9bogemPuXvicl9+yAfMPedxU+eqhlmR12yq/T6 MSIeTDOLFtYQPo3/Fe3iuTonEy4jxSeYzgN+47KUiTicdYgZIrkUw3+3Mg4rXh33Bs j29ybHk2uGZDcm06wod/6wkd2tFszlfYaOC2V+JjgkwNd5ps+RsFEy/6epad5fvVwg pFVC83JvaW4DtS1ycVk37ugo1FMd2I9iEkW0ynVS50yvFJRPw1Iv1PfX3KO4s0GiXL htU/c5SkhBVJg63ke/agO7JFKxf4GKNEKvIkibtU+uGEVfA8ZptIdwSwB5HAxc9qk8 f1jwae3fwM8+w== From: neeraj.upadhyay@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, paulmck@kernel.org, neeraj.upadhyay@kernel.org, neeraj.upadhyay@amd.com, boqun.feng@gmail.com, joel@joelfernandes.org, urezki@gmail.com, frederic@kernel.org, Christophe JAILLET , Davidlohr Bueso Subject: [PATCH rcu 02/14] refscale: Optimize process_durations() Date: Fri, 16 Aug 2024 12:32:44 +0530 Message-Id: <20240816070256.60993-2-neeraj.upadhyay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240816070209.GA60666@neeraj.linux> References: <20240816070209.GA60666@neeraj.linux> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Christophe JAILLET process_durations() is not a hot path, but there is no good reason to iterate over and over the data already in 'buf'. Using a seq_buf saves some useless strcat() and the need of a temp buffer. Data is written directly at the correct place. Signed-off-by: Christophe JAILLET Tested-by: "Paul E. McKenney" Reviewed-by: Davidlohr Bueso Signed-off-by: Neeraj Upadhyay --- kernel/rcu/refscale.c | 25 ++++++++++++++----------- 1 file changed, 14 insertions(+), 11 deletions(-) diff --git a/kernel/rcu/refscale.c b/kernel/rcu/refscale.c index f4ea5b1ec068..cfec0648e141 100644 --- a/kernel/rcu/refscale.c +++ b/kernel/rcu/refscale.c @@ -28,6 +28,7 @@ #include #include #include +#include #include #include #include @@ -891,32 +892,34 @@ static u64 process_durations(int n) { int i; struct reader_task *rt; - char buf1[64]; + struct seq_buf s; char *buf; u64 sum = 0; buf = kmalloc(800 + 64, GFP_KERNEL); if (!buf) return 0; - buf[0] = 0; - sprintf(buf, "Experiment #%d (Format: :)", - exp_idx); + seq_buf_init(&s, buf, 800 + 64); + + seq_buf_printf(&s, "Experiment #%d (Format: :)", + exp_idx); for (i = 0; i < n && !torture_must_stop(); i++) { rt = &(reader_tasks[i]); - sprintf(buf1, "%d: %llu\t", i, rt->last_duration_ns); if (i % 5 == 0) - strcat(buf, "\n"); - if (strlen(buf) >= 800) { - pr_alert("%s", buf); - buf[0] = 0; + seq_buf_putc(&s, '\n'); + + if (seq_buf_used(&s) >= 800) { + pr_alert("%s", seq_buf_str(&s)); + seq_buf_clear(&s); } - strcat(buf, buf1); + + seq_buf_printf(&s, "%d: %llu\t", i, rt->last_duration_ns); sum += rt->last_duration_ns; } - pr_alert("%s\n", buf); + pr_alert("%s\n", seq_buf_str(&s)); kfree(buf); return sum; From patchwork Fri Aug 16 07:02:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neeraj Upadhyay X-Patchwork-Id: 13765532 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EFB17132117; Fri, 16 Aug 2024 07:04:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723791851; cv=none; b=Qsb++ECBXW3SpkTDioifA7ObQ5lIMrgVNsV51bzoViyHaghwSU0I/GP4eJV7xGZXEXt3n8YT5bKev40DBvAB93kRusowK1Ekmpy02IVVCyHKQixdaG0v+oXPJV3F4iT45tewinIMkw7R8NTfCVEfpPm79hGfr0+wasVy43LqGd8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723791851; c=relaxed/simple; bh=JlWfWfOshVrjXO5oIEB2X3rEfPtzchaaADjnIcIzEFE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Q822B4Lzt6ZTp74WZsvnumM7fAELZefymk7UI7Na6CzzQMUOB5aULxJ+rHh+a1TegDA3uhK+uA63rMVCQbvauQDizZbZSqqikRAPc2XQhy+3AYyue4IyLZZt5+BiZ0UROMhseVL0BxqGOftZ0SZfG/YVnwTZbK4bwpSyLwWlRYo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=BKxhLb1M; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="BKxhLb1M" Received: by smtp.kernel.org (Postfix) with ESMTPSA id B4B94C4AF0E; Fri, 16 Aug 2024 07:04:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723791850; bh=JlWfWfOshVrjXO5oIEB2X3rEfPtzchaaADjnIcIzEFE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=BKxhLb1MY75/OZAGComEnwRS/pBf3jgL8bSzOGUDa3prPQPZ75ELbgohuZmkekMtF 8+Z/kQLolrRA8FOj/Y6KbdoR1Yb38MXsnij00tmV5i6oDRiOaFCiE2PCAEYrkXnzwf 54eMY26vl2xr2VqCoTGcIxvMjZ4U+6xXaAqU3m8W92aNf2zeXuj7mVzSikjtwHIOhC i0QHXQjKnylX7W9Gw3jSdmjLDBF1sXIVP8a6OfyxWxb32vgG0g1zT7QPD3gT+SP/7F dE0kA9ZFYJkjS+sX3KaPvraDbCMv5HoKd2lOY72iV8GaaO0cFq5xGn91P1r4JZTHdm J4UO/XrivHi+w== From: neeraj.upadhyay@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, paulmck@kernel.org, neeraj.upadhyay@kernel.org, neeraj.upadhyay@amd.com, boqun.feng@gmail.com, joel@joelfernandes.org, urezki@gmail.com, frederic@kernel.org Subject: [PATCH rcu 03/14] rcuscale: Save a few lines with whitespace-only change Date: Fri, 16 Aug 2024 12:32:45 +0530 Message-Id: <20240816070256.60993-3-neeraj.upadhyay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240816070209.GA60666@neeraj.linux> References: <20240816070209.GA60666@neeraj.linux> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" This whitespace-only commit fuses a few lines of code, taking advantage of the newish 100-character-per-line limit to save a few lines of code. Signed-off-by: "Paul E. McKenney" Signed-off-by: Neeraj Upadhyay --- kernel/rcu/rcuscale.c | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/kernel/rcu/rcuscale.c b/kernel/rcu/rcuscale.c index d534d4ec2314..3269dd9c639f 100644 --- a/kernel/rcu/rcuscale.c +++ b/kernel/rcu/rcuscale.c @@ -1015,13 +1015,9 @@ rcu_scale_init(void) } while (atomic_read(&n_rcu_scale_reader_started) < nrealreaders) schedule_timeout_uninterruptible(1); - writer_tasks = kcalloc(nrealwriters, sizeof(reader_tasks[0]), - GFP_KERNEL); - writer_durations = kcalloc(nrealwriters, sizeof(*writer_durations), - GFP_KERNEL); - writer_n_durations = - kcalloc(nrealwriters, sizeof(*writer_n_durations), - GFP_KERNEL); + writer_tasks = kcalloc(nrealwriters, sizeof(reader_tasks[0]), GFP_KERNEL); + writer_durations = kcalloc(nrealwriters, sizeof(*writer_durations), GFP_KERNEL); + writer_n_durations = kcalloc(nrealwriters, sizeof(*writer_n_durations), GFP_KERNEL); if (!writer_tasks || !writer_durations || !writer_n_durations) { SCALEOUT_ERRSTRING("out of memory"); firsterr = -ENOMEM; From patchwork Fri Aug 16 07:02:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neeraj Upadhyay X-Patchwork-Id: 13765533 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 11B0113A256; Fri, 16 Aug 2024 07:04:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723791857; cv=none; b=A9CbfHBtr4aPp1urUmaeOA0y0oxvL6ZXoBzQ+YGLfzipuOLSQT984IG2SjljB4e0tNfC40NPTtdd0Ttl/6OT6V5GwS+TfwJJrgpJJWFXoGljaaMGD/v39afhhydecJ7nMa8Vk6sZ0+X/0c1ja/gd3NeeedSoqNc61HrHfgQNU+g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723791857; c=relaxed/simple; bh=vtELkXCUFMg5oRqM2b92OFdwmNazPUkHQIvr96CsLRI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=GYgxMTNCzcil2jHvkaOWibrv6+8LjPlMi+sZ6oxY2H0FbhTCnU9+sdDCgyClURwJnOTNZsAeNgvibycrlOVjTm8tqC4t+lyqObz+HBtfBs5Sgeoj6Hn13ug2J7dwm3lb3qEE0XHO71UqlnKrpYFUglXwHU3TrDKcFwpxIMrlU9w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=J3RtWp/h; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="J3RtWp/h" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8441FC4AF13; Fri, 16 Aug 2024 07:04:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723791856; bh=vtELkXCUFMg5oRqM2b92OFdwmNazPUkHQIvr96CsLRI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=J3RtWp/h57Ally7NN/Jk6EZGMseop72P0G6LKxEWPJbjtEC7pxtLnccf59tfmZJ9d a7+GVSIV8BiHOW2hUKDIVmPeXja6kS4EjuXrYGyHwlsnKwUBjtNQzCAxh6jhSJd+G7 0XuHQk6GqiZ48sRZTXXaQhDAI7h8YI+54h3Dmfx2W90streaJ4aSoPjtwITa+mbdJB ltPd5KuDTzT09UvdFnzHmhEjs8ZHdSKnCpcu2WVgwQ7rIWfEnIVUGKZdqzfPI9KEkh ESJL+69ZFXqHO/QKIvWcCqLhCP5G6MeN8JhxVufLGy+vrM/0fkLnTj0oFJAyuEId4F MATMb47OazoAA== From: neeraj.upadhyay@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, paulmck@kernel.org, neeraj.upadhyay@kernel.org, neeraj.upadhyay@amd.com, boqun.feng@gmail.com, joel@joelfernandes.org, urezki@gmail.com, frederic@kernel.org Subject: [PATCH rcu 04/14] rcuscale: Dump stacks of stalled rcu_scale_writer() instances Date: Fri, 16 Aug 2024 12:32:46 +0530 Message-Id: <20240816070256.60993-4-neeraj.upadhyay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240816070209.GA60666@neeraj.linux> References: <20240816070209.GA60666@neeraj.linux> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" This commit improves debuggability by dumping the stacks of rcu_scale_writer() instances that have not completed in a reasonable timeframe. These stacks are dumped remotely, but they will be accurate in the thus-far common case where the stalled rcu_scale_writer() instances are blocked. [ paulmck: Apply kernel test robot feedback. ] Signed-off-by: "Paul E. McKenney" Signed-off-by: Neeraj Upadhyay --- kernel/rcu/rcuscale.c | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/rcuscale.c b/kernel/rcu/rcuscale.c index 3269dd9c639f..5087ca7062d9 100644 --- a/kernel/rcu/rcuscale.c +++ b/kernel/rcu/rcuscale.c @@ -39,6 +39,7 @@ #include #include #include +#include #include "rcu.h" @@ -111,6 +112,7 @@ static struct task_struct **reader_tasks; static struct task_struct *shutdown_task; static u64 **writer_durations; +static bool *writer_done; static int *writer_n_durations; static atomic_t n_rcu_scale_reader_started; static atomic_t n_rcu_scale_writer_started; @@ -524,6 +526,7 @@ rcu_scale_writer(void *arg) started = true; if (!done && i >= MIN_MEAS && time_after(jiffies, jdone)) { done = true; + WRITE_ONCE(writer_done[me], true); sched_set_normal(current, 0); pr_alert("%s%s rcu_scale_writer %ld has %d measurements\n", scale_type, SCALE_FLAG, me, MIN_MEAS); @@ -549,6 +552,19 @@ rcu_scale_writer(void *arg) if (done && !alldone && atomic_read(&n_rcu_scale_writer_finished) >= nrealwriters) alldone = true; + if (done && !alldone && time_after(jiffies, jdone + HZ * 60)) { + static atomic_t dumped; + int i; + + if (!atomic_xchg(&dumped, 1)) { + for (i = 0; i < nrealwriters; i++) { + if (writer_done[i]) + continue; + pr_info("%s: Task %ld flags writer %d:\n", __func__, me, i); + sched_show_task(writer_tasks[i]); + } + } + } if (started && !alldone && i < MAX_MEAS - 1) i++; rcu_scale_wait_shutdown(); @@ -921,6 +937,8 @@ rcu_scale_cleanup(void) kfree(writer_tasks); kfree(writer_durations); kfree(writer_n_durations); + kfree(writer_done); + writer_done = NULL; } /* Do torture-type-specific cleanup operations. */ @@ -1015,10 +1033,11 @@ rcu_scale_init(void) } while (atomic_read(&n_rcu_scale_reader_started) < nrealreaders) schedule_timeout_uninterruptible(1); - writer_tasks = kcalloc(nrealwriters, sizeof(reader_tasks[0]), GFP_KERNEL); + writer_tasks = kcalloc(nrealwriters, sizeof(writer_tasks[0]), GFP_KERNEL); writer_durations = kcalloc(nrealwriters, sizeof(*writer_durations), GFP_KERNEL); writer_n_durations = kcalloc(nrealwriters, sizeof(*writer_n_durations), GFP_KERNEL); - if (!writer_tasks || !writer_durations || !writer_n_durations) { + writer_done = kcalloc(nrealwriters, sizeof(writer_done[0]), GFP_KERNEL); + if (!writer_tasks || !writer_durations || !writer_n_durations || !writer_done) { SCALEOUT_ERRSTRING("out of memory"); firsterr = -ENOMEM; goto unwind; From patchwork Fri Aug 16 07:02:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neeraj Upadhyay X-Patchwork-Id: 13765534 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8C79513B58E; Fri, 16 Aug 2024 07:04:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723791862; cv=none; b=JsHUcuXHjeEhNhBFhrnQKVhio0ZTucEaYj6Pe4s0fqV1sd31bmQRmClo2lbjP31w71LBpUCXYh2AARpTUnCnTX4jIXytmh4xd5Mgt03dkYxkjKEASpgPtUW4d2Q7ejfCO8kYJxOT+x5gS3VtJocq/yAMNAr0hN/Cb26PSvKJtaI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723791862; c=relaxed/simple; bh=K5JsMoDhxJRgPGCUcAXtKOF5CRFQ1NO+CwBhPjeO2Nk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=AOJ4K2oTnw5fz8jLYQ7WMQCBn8ff5k5CwGfMSgTpFr5UGM7SQ945X2Uzc+EZk+MGAvrMzPb6QwR8dzadIDPFvesoGkCiYf0kaXCOJ0uTXpZEsDkLcbSSnM0j81l8d/vdmIqBlWLgsht8QGLA995OqJoXsWCcvyJRra3bsOJ566o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=aGYON0pT; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="aGYON0pT" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 77EFCC4AF0E; Fri, 16 Aug 2024 07:04:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723791862; bh=K5JsMoDhxJRgPGCUcAXtKOF5CRFQ1NO+CwBhPjeO2Nk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=aGYON0pTvlUIk1eXtfmjBNkEf2mTpveSnkH87A0K2U8x0eak5xJcfTCtf0or260ej vMy3Lrdbl6Ru62aR0sxLNoYOfqEUQyg+gGPTwBpe3sSxseYIYXdXssCpTPW3rmXYjx O+sQUTrua61Pe7Gl/ePVnzaySXx5i5qd/7bLnHbjJsewrt7ALgF8dBM8EpoQvk+zB4 aOWhptKTFFRhhMQQ6KVLCtTBSd37garo7f2+yGM22uFR5XBpOCO1X3N8ccC5YEVJLd BVIN+/CuzQvm6nQpTOuBk2qhlY9MTqaRI1nGAbf/Gn6bsGeHAoAuVlxapQw9BuNbs0 Ffaj2vTwzu3pw== From: neeraj.upadhyay@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, paulmck@kernel.org, neeraj.upadhyay@kernel.org, neeraj.upadhyay@amd.com, boqun.feng@gmail.com, joel@joelfernandes.org, urezki@gmail.com, frederic@kernel.org Subject: [PATCH rcu 05/14] rcuscale: Dump grace-period statistics when rcu_scale_writer() stalls Date: Fri, 16 Aug 2024 12:32:47 +0530 Message-Id: <20240816070256.60993-5-neeraj.upadhyay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240816070209.GA60666@neeraj.linux> References: <20240816070209.GA60666@neeraj.linux> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" This commit adds a .stats function pointer to the rcu_scale_ops structure, and if this is non-NULL, it is invoked after stack traces are dumped in response to a rcu_scale_writer() stall. Signed-off-by: "Paul E. McKenney" Signed-off-by: Neeraj Upadhyay --- kernel/rcu/rcuscale.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/kernel/rcu/rcuscale.c b/kernel/rcu/rcuscale.c index 5087ca7062d9..1b9a43653d7e 100644 --- a/kernel/rcu/rcuscale.c +++ b/kernel/rcu/rcuscale.c @@ -145,6 +145,7 @@ struct rcu_scale_ops { void (*sync)(void); void (*exp_sync)(void); struct task_struct *(*rso_gp_kthread)(void); + void (*stats)(void); const char *name; }; @@ -226,6 +227,11 @@ static void srcu_scale_synchronize(void) synchronize_srcu(srcu_ctlp); } +static void srcu_scale_stats(void) +{ + srcu_torture_stats_print(srcu_ctlp, scale_type, SCALE_FLAG); +} + static void srcu_scale_synchronize_expedited(void) { synchronize_srcu_expedited(srcu_ctlp); @@ -243,6 +249,7 @@ static struct rcu_scale_ops srcu_ops = { .gp_barrier = srcu_rcu_barrier, .sync = srcu_scale_synchronize, .exp_sync = srcu_scale_synchronize_expedited, + .stats = srcu_scale_stats, .name = "srcu" }; @@ -272,6 +279,7 @@ static struct rcu_scale_ops srcud_ops = { .gp_barrier = srcu_rcu_barrier, .sync = srcu_scale_synchronize, .exp_sync = srcu_scale_synchronize_expedited, + .stats = srcu_scale_stats, .name = "srcud" }; @@ -563,6 +571,8 @@ rcu_scale_writer(void *arg) pr_info("%s: Task %ld flags writer %d:\n", __func__, me, i); sched_show_task(writer_tasks[i]); } + if (cur_ops->stats) + cur_ops->stats(); } } if (started && !alldone && i < MAX_MEAS - 1) From patchwork Fri Aug 16 07:02:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neeraj Upadhyay X-Patchwork-Id: 13765535 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 90E1713D258; Fri, 16 Aug 2024 07:04:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723791868; cv=none; b=KTbLRFEMhuQylUbsKbkvec2YAWD+58/g6yhebDi8jh656OAalcXPNF3uBh1u4L47NpTy2l+thBAceLN8WOtHlxlLzNr1ae0ur/D3SsKiN6OOb0OJC5gD3bkmOEC0VZuB0m5i8y9s3dKpIz8OD4goWdJCdLdpZ8v4bsHcpy5JgGI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723791868; c=relaxed/simple; bh=Kpne2lKthjgBTkbEjIqIgS4eVjzf8xKIxkMykFK9Pns=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=off2x3YFraWiGf2rWl0tPJN34jM2pEf0ZdhU+jlgVLbVVGxwUQ5jb+ULuGUufXtazebxbhVW9o7WKa/scKc5z4nRuQxeyb3U9lPYiHLXL/HFEuXTTWfjUMAq+irw2+w0hRjEurHB+vuSNEfj4TKtrMylNMExbiqvHDM1XylQeEE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=NRdZebFG; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="NRdZebFG" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3BDDBC4AF09; Fri, 16 Aug 2024 07:04:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723791868; bh=Kpne2lKthjgBTkbEjIqIgS4eVjzf8xKIxkMykFK9Pns=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=NRdZebFGVm2sg2vgZYC+5MaF82Cz1Yp9ZknVXeBtd7X0B/PArVBwFnfwUiRGDFW5h BwGheP3bEc9SNopA5tyTjC8Qb909ouTovoRWhLN+KG9yBG5/9lROKbwfYq7ZolztQ9 2fdK1Y0tSVZyaP5VjzJ+cfR809T9WsnHW5NG6HgAksfeuplXjYY05jC9rfK/0vwwus D84c18oNkEttiLHm73YMw3KRywyVuJlyxTm8Vbz7sD4rJ8oLRVV8lJr9kUtrb/vNv1 OXWyMpw1VK4IzAkyuebpn7BrCkPt/UoVh1P+6P397BkJnvuOZ/00dMI5sMNxejec7N RXcQrMA+IDB1w== From: neeraj.upadhyay@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, paulmck@kernel.org, neeraj.upadhyay@kernel.org, neeraj.upadhyay@amd.com, boqun.feng@gmail.com, joel@joelfernandes.org, urezki@gmail.com, frederic@kernel.org Subject: [PATCH rcu 06/14] rcu: Mark callbacks not currently participating in barrier operation Date: Fri, 16 Aug 2024 12:32:48 +0530 Message-Id: <20240816070256.60993-6-neeraj.upadhyay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240816070209.GA60666@neeraj.linux> References: <20240816070209.GA60666@neeraj.linux> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" RCU keeps a count of the number of callbacks that the current rcu_barrier() is waiting on, but there is currently no easy way to work out which callback is stuck. One way to do this is to mark idle RCU-barrier callbacks by making the ->next pointer point to the callback itself, and this commit does just that. Later commits will use this for debug output. Signed-off-by: "Paul E. McKenney" Signed-off-by: Neeraj Upadhyay --- kernel/rcu/tree.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index e641cc681901..f931171daecd 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -4403,6 +4403,7 @@ static void rcu_barrier_callback(struct rcu_head *rhp) { unsigned long __maybe_unused s = rcu_state.barrier_sequence; + rhp->next = rhp; // Mark the callback as having been invoked. if (atomic_dec_and_test(&rcu_state.barrier_cpu_count)) { rcu_barrier_trace(TPS("LastCB"), -1, s); complete(&rcu_state.barrier_completion); @@ -5424,6 +5425,8 @@ static void __init rcu_init_one(void) while (i > rnp->grphi) rnp++; per_cpu_ptr(&rcu_data, i)->mynode = rnp; + per_cpu_ptr(&rcu_data, i)->barrier_head.next = + &per_cpu_ptr(&rcu_data, i)->barrier_head; rcu_boot_init_percpu_data(i); } } From patchwork Fri Aug 16 07:02:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neeraj Upadhyay X-Patchwork-Id: 13765536 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1910B126F2A; Fri, 16 Aug 2024 07:04:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723791875; cv=none; b=cy5/EPrBOUY13qrnsdJUREEas8Kj9UZZbR5x4XcdIrV0INa+9bGhDXspGCRN28mBSdbdfaMGEl0Bq0JoFJKVvOWbtq71CBcQljCrsZF+vLxPkbdNYCOzUUgE7NTKHyPpkHexzI7DQhNJMH694NVfj8ZIbnG9f7+suZDgNIJc9Pk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723791875; c=relaxed/simple; bh=rGx98LRQ4fzsdM/wfrRs6MgCPfxa8nno7GmWNehrPtM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=P1+6R5Ev8rRjn9cPcH8Xo493/R8aBy/r6CsbE1Ey040x1/ZQepct8t4qZ1IN/q9ny2KqJPAC3p/SZM/7PLto1zE+srSXW6UlbxISAW42iyCTb2MkKIrGvqtSKYN3qg3Et9nM6MwTM22Ifv0K3OjsahxBqrwdyTCshC16vpy6RSU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Kn4ggiKe; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Kn4ggiKe" Received: by smtp.kernel.org (Postfix) with ESMTPSA id A78C6C4AF09; Fri, 16 Aug 2024 07:04:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723791874; bh=rGx98LRQ4fzsdM/wfrRs6MgCPfxa8nno7GmWNehrPtM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Kn4ggiKeNAh9c04GZchvKGjegP3P12bDm4ZTGfiOoAPkOjtgqXAmKrirLmdDij/M4 lcpQe/SIz2dGLmK4CACHOMYKszNuhSKOA5yDekXA1R/PxFjGkiQEUy9B4molU4Ph/B k9Zf4so2ro69ApbuLFxN9Qwrhttil8SIX8LXS8g3cJTFYObinuiwGgM4HYeiNh/BWd mGAam9Hr0sgWWSumW9EgWzEj57ZvZpyBLjOmsXNT4bqYtKXX3bRBAbW8QaBUAC78gp Fjun2YP0u52AckriwQ+D2DEE3/BWocU8r9ElvTGsXoE1WYKtQRwKe8b4gxXlnhihng f9etaXIDM/xJw== From: neeraj.upadhyay@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, paulmck@kernel.org, neeraj.upadhyay@kernel.org, neeraj.upadhyay@amd.com, boqun.feng@gmail.com, joel@joelfernandes.org, urezki@gmail.com, frederic@kernel.org Subject: [PATCH rcu 07/14] rcuscale: Print detailed grace-period and barrier diagnostics Date: Fri, 16 Aug 2024 12:32:49 +0530 Message-Id: <20240816070256.60993-7-neeraj.upadhyay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240816070209.GA60666@neeraj.linux> References: <20240816070209.GA60666@neeraj.linux> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" This commit uses the new rcu_tasks_torture_stats_print(), rcu_tasks_trace_torture_stats_print(), and rcu_tasks_rude_torture_stats_print() functions in order to provide detailed diagnostics on grace-period, callback, and barrier state when rcu_scale_writer() hangs. [ paulmck: Apply kernel test robot feedback. ] Signed-off-by: "Paul E. McKenney" Signed-off-by: Neeraj Upadhyay --- kernel/rcu/rcuscale.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/kernel/rcu/rcuscale.c b/kernel/rcu/rcuscale.c index 1b9a43653d7e..c507750e94d8 100644 --- a/kernel/rcu/rcuscale.c +++ b/kernel/rcu/rcuscale.c @@ -298,6 +298,11 @@ static void tasks_scale_read_unlock(int idx) { } +static void rcu_tasks_scale_stats(void) +{ + rcu_tasks_torture_stats_print(scale_type, SCALE_FLAG); +} + static struct rcu_scale_ops tasks_ops = { .ptype = RCU_TASKS_FLAVOR, .init = rcu_sync_scale_init, @@ -310,6 +315,7 @@ static struct rcu_scale_ops tasks_ops = { .sync = synchronize_rcu_tasks, .exp_sync = synchronize_rcu_tasks, .rso_gp_kthread = get_rcu_tasks_gp_kthread, + .stats = IS_ENABLED(CONFIG_TINY_RCU) ? NULL : rcu_tasks_scale_stats, .name = "tasks" }; @@ -336,6 +342,11 @@ static void tasks_rude_scale_read_unlock(int idx) { } +static void rcu_tasks_rude_scale_stats(void) +{ + rcu_tasks_rude_torture_stats_print(scale_type, SCALE_FLAG); +} + static struct rcu_scale_ops tasks_rude_ops = { .ptype = RCU_TASKS_RUDE_FLAVOR, .init = rcu_sync_scale_init, @@ -346,6 +357,7 @@ static struct rcu_scale_ops tasks_rude_ops = { .sync = synchronize_rcu_tasks_rude, .exp_sync = synchronize_rcu_tasks_rude, .rso_gp_kthread = get_rcu_tasks_rude_gp_kthread, + .stats = IS_ENABLED(CONFIG_TINY_RCU) ? NULL : rcu_tasks_rude_scale_stats, .name = "tasks-rude" }; @@ -374,6 +386,11 @@ static void tasks_trace_scale_read_unlock(int idx) rcu_read_unlock_trace(); } +static void rcu_tasks_trace_scale_stats(void) +{ + rcu_tasks_trace_torture_stats_print(scale_type, SCALE_FLAG); +} + static struct rcu_scale_ops tasks_tracing_ops = { .ptype = RCU_TASKS_FLAVOR, .init = rcu_sync_scale_init, @@ -386,6 +403,7 @@ static struct rcu_scale_ops tasks_tracing_ops = { .sync = synchronize_rcu_tasks_trace, .exp_sync = synchronize_rcu_tasks_trace, .rso_gp_kthread = get_rcu_tasks_trace_gp_kthread, + .stats = IS_ENABLED(CONFIG_TINY_RCU) ? NULL : rcu_tasks_trace_scale_stats, .name = "tasks-tracing" }; From patchwork Fri Aug 16 07:02:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neeraj Upadhyay X-Patchwork-Id: 13765537 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1D73912D742; Fri, 16 Aug 2024 07:04:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723791881; cv=none; b=clNNyZvOrhecsmyEBDXjD0YvwU8bQkIxL8jDIVc/jd0Its9x63CyrbCszeNqc4q6cxmUD1O5j/Jpsu9aaV6qWnH6mtpS8QME4z7tB8sLFb3mjsif7dQ49qs00e+7dC0ePOx348uTIfnjztv91snSiipuOlEJ08KR/OJNjyfaOGg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723791881; c=relaxed/simple; bh=7kERyCASbzE1YDDe0Of+LxZ9CqXpNEA+iRkLzJTLXbs=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=fZYCSbwdpmE86m7Q0Dvj4dbpfSZ1n5tlzoADDBYaGxsfYkJ8wTfUrR3L3bOsa2QLk54p/qDRNJRbTYd5Gz0qdSKupHuPowgqUr547JgfoqnjR6sRd0mvU9V8VnN/KB/P0D/PivCD91UgVXpiwali7khNN82udZTxN9qlcoNs/mg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=UbTgVMrZ; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="UbTgVMrZ" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 898F6C4AF09; Fri, 16 Aug 2024 07:04:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723791881; bh=7kERyCASbzE1YDDe0Of+LxZ9CqXpNEA+iRkLzJTLXbs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=UbTgVMrZDZTvbPIShvdWAXng721tYqKsA+GQ6ionUflthr2pdv7MW7rPxya61xiyc 1WsWHXMhrS3j5RcAuOOs5VWXQ7esNYJjRbwbB5uAlbNms2R/DrHMakHtko6YKOdQa4 Fjr7zUie1tiukMCntgEQfTrsxUxcQzx0+ifRdiLmzQOIStQz3Bqxe3srcyQ547a19n tSnysHHIQ/3PqB4EymmS0VQGF8mR39E+MEl4UWkXozML6NmA7yB2T9eK5kn/qbyX0h Ab451v/xfevi3W8lNHekYnyDX0lAskKqSTTGMeRQaqMxJlDTpd/86xHT4QHlpDQKiZ j5ctei/zpeMHg== From: neeraj.upadhyay@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, paulmck@kernel.org, neeraj.upadhyay@kernel.org, neeraj.upadhyay@amd.com, boqun.feng@gmail.com, joel@joelfernandes.org, urezki@gmail.com, frederic@kernel.org Subject: [PATCH rcu 08/14] rcuscale: Provide clear error when async specified without primitives Date: Fri, 16 Aug 2024 12:32:50 +0530 Message-Id: <20240816070256.60993-8-neeraj.upadhyay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240816070209.GA60666@neeraj.linux> References: <20240816070209.GA60666@neeraj.linux> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" Currently, if the rcuscale module's async module parameter is specified for RCU implementations that do not have async primitives such as RCU Tasks Rude (which now lacks a call_rcu_tasks_rude() function), there will be a series of splats due to calls to a NULL pointer. This commit therefore warns of this situation, but switches to non-async testing. Signed-off-by: "Paul E. McKenney" Signed-off-by: Neeraj Upadhyay --- kernel/rcu/rcuscale.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/rcuscale.c b/kernel/rcu/rcuscale.c index c507750e94d8..79e1c32d5c0f 100644 --- a/kernel/rcu/rcuscale.c +++ b/kernel/rcu/rcuscale.c @@ -525,7 +525,7 @@ rcu_scale_writer(void *arg) schedule_timeout_idle(torture_random(&tr) % writer_holdoff_jiffies + 1); wdp = &wdpp[i]; *wdp = ktime_get_mono_fast_ns(); - if (gp_async) { + if (gp_async && !WARN_ON_ONCE(!cur_ops->async)) { retry: if (!rhp) rhp = kmalloc(sizeof(*rhp), GFP_KERNEL); @@ -597,7 +597,7 @@ rcu_scale_writer(void *arg) i++; rcu_scale_wait_shutdown(); } while (!torture_must_stop()); - if (gp_async) { + if (gp_async && cur_ops->async) { cur_ops->gp_barrier(); } writer_n_durations[me] = i_max + 1; From patchwork Fri Aug 16 07:02:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neeraj Upadhyay X-Patchwork-Id: 13765538 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A18FA12EBE7; Fri, 16 Aug 2024 07:04:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723791887; cv=none; b=o9TtT2khRhxCbAQEM0uq+68O4Y4CSHNsJ0eKEtZvKYPkv+CsmhMIsFmbSNNUcbPqYHHryVjgvZw+YOZGLaiauxQG2uEaIxhLb7keBlEnTdt0v7KxRTnMuImD3OyZSaxNwsCcnVdmww5niYYE54R4ODiwOWjwOA7M3Hfz98ojttY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723791887; c=relaxed/simple; bh=OoxDmktJaOMty2kevzZelHWkNA6J/CFdJiE+1Wq6wDg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=eQLkVXRwFbEFoOfCEocgq68jSN1s/TKnRRJtLGbR7GfG7MdOUL5SDbXNs450+z9rHpw4SSXxp0XVXRzOOTcVg+yw+crSF/0Fdg/Et1GANST6eiByRRYUUVhDN4ijdM/JVxJUSS3r0aK8EVQfmC/SsM9a0l/ko9dhEEo71K/nG2M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=p6CthFmz; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="p6CthFmz" Received: by smtp.kernel.org (Postfix) with ESMTPSA id D7209C4AF12; Fri, 16 Aug 2024 07:04:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723791887; bh=OoxDmktJaOMty2kevzZelHWkNA6J/CFdJiE+1Wq6wDg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=p6CthFmzSeJWwU3gTcaEbHhvNxPVRqeyu8Nk7k4rUUfXPVsrEj9QBEmEek3jmctVQ DNQdUVR0/X+BfQ3eCKIJoaYzmAYyRHtLOhQnBL4nwKEgAmoTriyeIZUhDbqNQeuW8h CdViMglCSusg4opjixCPdW0LU5UmPXZKxul9BGl0TYSyUH6FJjcMYv6iC88RH2e6Uh K/RqFyJ5pwQx5nKNYqcvfXSs96MZfq3t/LJunYMv3k4AeWxk/DrSuy61A2fjj8uDq5 eKOp1J3+4PvslkXsHwNbM0RIMuY2h79FYlLl0AF0cX1BAyQ+QDXORH/pK2fQQwSzvL 5Nz9ocPVkx1iw== From: neeraj.upadhyay@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, paulmck@kernel.org, neeraj.upadhyay@kernel.org, neeraj.upadhyay@amd.com, boqun.feng@gmail.com, joel@joelfernandes.org, urezki@gmail.com, frederic@kernel.org Subject: [PATCH rcu 09/14] rcuscale: Make all writer tasks report upon hang Date: Fri, 16 Aug 2024 12:32:51 +0530 Message-Id: <20240816070256.60993-9-neeraj.upadhyay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240816070209.GA60666@neeraj.linux> References: <20240816070209.GA60666@neeraj.linux> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" This commit causes all writer tasks to provide a brief report after a hang has been reported, spaced at one-second intervals. Signed-off-by: "Paul E. McKenney" Signed-off-by: Neeraj Upadhyay --- kernel/rcu/rcuscale.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/kernel/rcu/rcuscale.c b/kernel/rcu/rcuscale.c index 79e1c32d5c0f..dfe8e0faa4d8 100644 --- a/kernel/rcu/rcuscale.c +++ b/kernel/rcu/rcuscale.c @@ -483,6 +483,7 @@ rcu_scale_writer(void *arg) unsigned long jdone; long me = (long)arg; struct rcu_head *rhp = NULL; + bool selfreport = false; bool started = false, done = false, alldone = false; u64 t; DEFINE_TORTURE_RANDOM(tr); @@ -593,6 +594,11 @@ rcu_scale_writer(void *arg) cur_ops->stats(); } } + if (!selfreport && time_after(jiffies, jdone + HZ * (70 + me))) { + pr_info("%s: Writer %ld self-report: started %d done %d/%d->%d i %d jdone %lu.\n", + __func__, me, started, done, writer_done[me], atomic_read(&n_rcu_scale_writer_finished), i, jiffies - jdone); + selfreport = true; + } if (started && !alldone && i < MAX_MEAS - 1) i++; rcu_scale_wait_shutdown(); From patchwork Fri Aug 16 07:02:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neeraj Upadhyay X-Patchwork-Id: 13765539 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AD3B713790B; Fri, 16 Aug 2024 07:04:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723791892; cv=none; b=NluyrdkVXwvhHDzBvhYhFXbiItu9YjxpwkjkwTteacsxoQeyRTJZUqpjZy1H6COP8K/aPgj186h7nh775+tJzN8Jx+3kiIscrXPGi7/W04y0h9O4zohv2/mWYhgxLqmDqGuYdMBPse97kaOA/OS0pt25ZAf5/Jca1S6+I9KFP18= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723791892; c=relaxed/simple; bh=BJJbVM0oLX+bNBsDn9howoFV7Y/+W9jzYAXgiWat4O4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=c6ZsV8yhQs22gk5IETJwQQSuguSvjLUkN3qsMOG2Lvmz1Cea7Gsqm+tCNvLErNR7cXyv8yIFLCOBUlNJCA/Lqei1BIXK0yRXBJ8w/Pw3oJAbRZwS8JfAnxhtF1WplX/da+TWSRXKC4JiaW3ZvVnJl4IyuQYt9lx26eCzqYoGyvQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=ADDC4TZw; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="ADDC4TZw" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 105D0C4AF10; Fri, 16 Aug 2024 07:04:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723791892; bh=BJJbVM0oLX+bNBsDn9howoFV7Y/+W9jzYAXgiWat4O4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ADDC4TZwINLeeN08WtjxfuFVf8gaQWgI0EK638hF3sDThRjNMddQLMTRScxKulqkF qStHirPe81laziSxdBVRbJhltYyZLEh6+fIEKsBbPwINKnwAGF/olbTAOZhU8UsCUS E+UQwl0LtrI0MokkgF/wTRVqMy/WvULUU0Z4ycFZWawqubPq3WV2CWw+vSkPZ23jzX 8E7rksqWKdjiU7CdZ0yLHDhSs0pxB5IqzkLIdniC06pICU+Ftt5A3hEqQAf0BTgTQ0 Yy3hq0vpAHD/mb9D5JrJHAWvJ9K98dOdNIqbmZbnqg3njPx16yVtcRq2Vc+xSp4LF9 IxslYlOTqW8Ug== From: neeraj.upadhyay@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, paulmck@kernel.org, neeraj.upadhyay@kernel.org, neeraj.upadhyay@amd.com, boqun.feng@gmail.com, joel@joelfernandes.org, urezki@gmail.com, frederic@kernel.org Subject: [PATCH rcu 10/14] rcuscale: Make rcu_scale_writer() tolerate repeated GFP_KERNEL failure Date: Fri, 16 Aug 2024 12:32:52 +0530 Message-Id: <20240816070256.60993-10-neeraj.upadhyay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240816070209.GA60666@neeraj.linux> References: <20240816070209.GA60666@neeraj.linux> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" Under some conditions, kmalloc(GFP_KERNEL) allocations have been observed to repeatedly fail. This situation has been observed to cause one of the rcu_scale_writer() instances to loop indefinitely retrying memory allocation for an asynchronous grace-period primitive. The problem is that if memory is short, all the other instances will allocate all available memory before the looping task is awakened from its rcu_barrier*() call. This in turn results in hangs, so that rcuscale fails to complete. This commit therefore removes the tight retry loop, so that when this condition occurs, the affected task is still passing through the full loop with its full set of termination checks. This spreads the risk of indefinite memory-allocation retry failures across all instances of rcu_scale_writer() tasks, which in turn prevents the hangs. Signed-off-by: "Paul E. McKenney" Signed-off-by: Neeraj Upadhyay --- kernel/rcu/rcuscale.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/kernel/rcu/rcuscale.c b/kernel/rcu/rcuscale.c index dfe8e0faa4d8..80518662273b 100644 --- a/kernel/rcu/rcuscale.c +++ b/kernel/rcu/rcuscale.c @@ -520,6 +520,8 @@ rcu_scale_writer(void *arg) jdone = jiffies + minruntime * HZ; do { + bool gp_succeeded = false; + if (writer_holdoff) udelay(writer_holdoff); if (writer_holdoff_jiffies) @@ -527,23 +529,24 @@ rcu_scale_writer(void *arg) wdp = &wdpp[i]; *wdp = ktime_get_mono_fast_ns(); if (gp_async && !WARN_ON_ONCE(!cur_ops->async)) { -retry: if (!rhp) rhp = kmalloc(sizeof(*rhp), GFP_KERNEL); if (rhp && atomic_read(this_cpu_ptr(&n_async_inflight)) < gp_async_max) { atomic_inc(this_cpu_ptr(&n_async_inflight)); cur_ops->async(rhp, rcu_scale_async_cb); rhp = NULL; + gp_succeeded = true; } else if (!kthread_should_stop()) { cur_ops->gp_barrier(); - goto retry; } else { kfree(rhp); /* Because we are stopping. */ } } else if (gp_exp) { cur_ops->exp_sync(); + gp_succeeded = true; } else { cur_ops->sync(); + gp_succeeded = true; } t = ktime_get_mono_fast_ns(); *wdp = t - *wdp; @@ -599,7 +602,7 @@ rcu_scale_writer(void *arg) __func__, me, started, done, writer_done[me], atomic_read(&n_rcu_scale_writer_finished), i, jiffies - jdone); selfreport = true; } - if (started && !alldone && i < MAX_MEAS - 1) + if (gp_succeeded && started && !alldone && i < MAX_MEAS - 1) i++; rcu_scale_wait_shutdown(); } while (!torture_must_stop()); From patchwork Fri Aug 16 07:02:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neeraj Upadhyay X-Patchwork-Id: 13765540 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 02E7A136E30; Fri, 16 Aug 2024 07:04:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723791899; cv=none; b=goFtkzc293zGELoXqWOQihDsTg32iyLembZTBdwyl0gFOdV2XHdMC3IYDau+d+MEv0/jNB+t95jy41XhR4t6paPVLkP2kX/+dJaiMd4BugXTNdNvHCPT7eH+9J+9u6rhbT5Z1lrpzbGADGlOYWZJ8YQwsct9kpy72391TIjAryA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723791899; c=relaxed/simple; bh=geQJg3YD9qfSjT0GW6Ak27p78kCmBFeReYEfLDCVq64=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=kh9I/0aaTGNiaZE+X5QUX5U0QquSKYTfXH1jBv17VZxlZAA5IXYqJsRw1nX4dlrPhLTw90ON2MauEFblXM/JPMnqmi7B4P53XmuPHbMOJwv/IK5c+DN1X1ebkJTHG8ZPSmAE4sHRFLMuB5+gUAR4TZQLkxIpEZm5wRWo23Xg/Fg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=ALIDhqg9; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="ALIDhqg9" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 70025C4AF09; Fri, 16 Aug 2024 07:04:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723791898; bh=geQJg3YD9qfSjT0GW6Ak27p78kCmBFeReYEfLDCVq64=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ALIDhqg9aEB7Zji6gTjqPHwkEu1Q31xsj8rtIqOfrWl9Oq5aTId2bHlrlTvwaQ1on G8+DMI6XL6uklFxLiRN+KtkDpa9HyK3MB/h9OXrhKerqkMsmBhPSpPHmrqJnJ25Cok HH2zWtwJQ3eGuDT2JDQMdic5Do2qvt89bD+bDPlCr/BPSwqyuMXfHVHrTaIZcPodpZ JMH52AGh3MEgBjyV/1Vu6z6irScE7Uk6Ro8J5V6+wZvprkswwTTWOlmHeNsMQWbXr2 Yi7GCnVCPx9B1sLDtFZt9UsGoYosOtaI0cj2lUK20w4QPg47gVfr25jfJrhn+cpge9 LJMh4nfykgjzg== From: neeraj.upadhyay@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, paulmck@kernel.org, neeraj.upadhyay@kernel.org, neeraj.upadhyay@amd.com, boqun.feng@gmail.com, joel@joelfernandes.org, urezki@gmail.com, frederic@kernel.org Subject: [PATCH rcu 11/14] rcuscale: Use special allocator for rcu_scale_writer() Date: Fri, 16 Aug 2024 12:32:53 +0530 Message-Id: <20240816070256.60993-11-neeraj.upadhyay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240816070209.GA60666@neeraj.linux> References: <20240816070209.GA60666@neeraj.linux> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" The rcu_scale_writer() function needs only a fixed number of rcu_head structures per kthread, which means that a trivial allocator suffices. This commit therefore uses an llist-based allocator using a fixed array of structures per kthread. This allows aggressive testing of RCU performance without stressing the slab allocators. Signed-off-by: "Paul E. McKenney" Signed-off-by: Neeraj Upadhyay --- kernel/rcu/rcuscale.c | 123 ++++++++++++++++++++++++++++++++++++++---- 1 file changed, 113 insertions(+), 10 deletions(-) diff --git a/kernel/rcu/rcuscale.c b/kernel/rcu/rcuscale.c index 80518662273b..bc7cca979c06 100644 --- a/kernel/rcu/rcuscale.c +++ b/kernel/rcu/rcuscale.c @@ -105,6 +105,19 @@ static char *scale_type = "rcu"; module_param(scale_type, charp, 0444); MODULE_PARM_DESC(scale_type, "Type of RCU to scalability-test (rcu, srcu, ...)"); +// Structure definitions for custom fixed-per-task allocator. +struct writer_mblock { + struct rcu_head wmb_rh; + struct llist_node wmb_node; + struct writer_freelist *wmb_wfl; +}; + +struct writer_freelist { + struct llist_head ws_lhg; + struct llist_head ____cacheline_internodealigned_in_smp ws_lhp; + struct writer_mblock *ws_mblocks; +}; + static int nrealreaders; static int nrealwriters; static struct task_struct **writer_tasks; @@ -113,6 +126,7 @@ static struct task_struct *shutdown_task; static u64 **writer_durations; static bool *writer_done; +static struct writer_freelist *writer_freelists; static int *writer_n_durations; static atomic_t n_rcu_scale_reader_started; static atomic_t n_rcu_scale_writer_started; @@ -463,13 +477,52 @@ rcu_scale_reader(void *arg) return 0; } +/* + * Allocate a writer_mblock structure for the specified rcu_scale_writer + * task. + */ +static struct writer_mblock *rcu_scale_alloc(long me) +{ + struct llist_node *llnp; + struct writer_freelist *wflp; + struct writer_mblock *wmbp; + + if (WARN_ON_ONCE(!writer_freelists)) + return NULL; + wflp = &writer_freelists[me]; + if (llist_empty(&wflp->ws_lhp)) { + // ->ws_lhp is private to its rcu_scale_writer task. + wmbp = container_of(llist_del_all(&wflp->ws_lhg), struct writer_mblock, wmb_node); + wflp->ws_lhp.first = &wmbp->wmb_node; + } + llnp = llist_del_first(&wflp->ws_lhp); + if (!llnp) + return NULL; + return container_of(llnp, struct writer_mblock, wmb_node); +} + +/* + * Free a writer_mblock structure to its rcu_scale_writer task. + */ +static void rcu_scale_free(struct writer_mblock *wmbp) +{ + struct writer_freelist *wflp; + + if (!wmbp) + return; + wflp = wmbp->wmb_wfl; + llist_add(&wmbp->wmb_node, &wflp->ws_lhg); +} + /* * Callback function for asynchronous grace periods from rcu_scale_writer(). */ static void rcu_scale_async_cb(struct rcu_head *rhp) { + struct writer_mblock *wmbp = container_of(rhp, struct writer_mblock, wmb_rh); + atomic_dec(this_cpu_ptr(&n_async_inflight)); - kfree(rhp); + rcu_scale_free(wmbp); } /* @@ -482,13 +535,13 @@ rcu_scale_writer(void *arg) int i_max; unsigned long jdone; long me = (long)arg; - struct rcu_head *rhp = NULL; bool selfreport = false; bool started = false, done = false, alldone = false; u64 t; DEFINE_TORTURE_RANDOM(tr); u64 *wdp; u64 *wdpp = writer_durations[me]; + struct writer_mblock *wmbp = NULL; VERBOSE_SCALEOUT_STRING("rcu_scale_writer task started"); WARN_ON(!wdpp); @@ -529,17 +582,18 @@ rcu_scale_writer(void *arg) wdp = &wdpp[i]; *wdp = ktime_get_mono_fast_ns(); if (gp_async && !WARN_ON_ONCE(!cur_ops->async)) { - if (!rhp) - rhp = kmalloc(sizeof(*rhp), GFP_KERNEL); - if (rhp && atomic_read(this_cpu_ptr(&n_async_inflight)) < gp_async_max) { + if (!wmbp) + wmbp = rcu_scale_alloc(me); + if (wmbp && atomic_read(this_cpu_ptr(&n_async_inflight)) < gp_async_max) { atomic_inc(this_cpu_ptr(&n_async_inflight)); - cur_ops->async(rhp, rcu_scale_async_cb); - rhp = NULL; + cur_ops->async(&wmbp->wmb_rh, rcu_scale_async_cb); + wmbp = NULL; gp_succeeded = true; } else if (!kthread_should_stop()) { cur_ops->gp_barrier(); } else { - kfree(rhp); /* Because we are stopping. */ + rcu_scale_free(wmbp); /* Because we are stopping. */ + wmbp = NULL; } } else if (gp_exp) { cur_ops->exp_sync(); @@ -607,6 +661,7 @@ rcu_scale_writer(void *arg) rcu_scale_wait_shutdown(); } while (!torture_must_stop()); if (gp_async && cur_ops->async) { + rcu_scale_free(wmbp); cur_ops->gp_barrier(); } writer_n_durations[me] = i_max + 1; @@ -970,12 +1025,30 @@ rcu_scale_cleanup(void) schedule_timeout_uninterruptible(1); } kfree(writer_durations[i]); + if (writer_freelists) { + int ctr = 0; + struct llist_node *llnp; + struct writer_freelist *wflp = &writer_freelists[i]; + + if (wflp->ws_mblocks) { + llist_for_each(llnp, wflp->ws_lhg.first) + ctr++; + llist_for_each(llnp, wflp->ws_lhp.first) + ctr++; + WARN_ONCE(ctr != gp_async_max, + "%s: ctr = %d gp_async_max = %d\n", + __func__, ctr, gp_async_max); + kfree(wflp->ws_mblocks); + } + } } kfree(writer_tasks); kfree(writer_durations); kfree(writer_n_durations); kfree(writer_done); writer_done = NULL; + kfree(writer_freelists); + writer_freelists = NULL; } /* Do torture-type-specific cleanup operations. */ @@ -1002,8 +1075,9 @@ rcu_scale_shutdown(void *arg) static int __init rcu_scale_init(void) { - long i; int firsterr = 0; + long i; + long j; static struct rcu_scale_ops *scale_ops[] = { &rcu_ops, &srcu_ops, &srcud_ops, TASKS_OPS TASKS_RUDE_OPS TASKS_TRACING_OPS }; @@ -1074,7 +1148,18 @@ rcu_scale_init(void) writer_durations = kcalloc(nrealwriters, sizeof(*writer_durations), GFP_KERNEL); writer_n_durations = kcalloc(nrealwriters, sizeof(*writer_n_durations), GFP_KERNEL); writer_done = kcalloc(nrealwriters, sizeof(writer_done[0]), GFP_KERNEL); - if (!writer_tasks || !writer_durations || !writer_n_durations || !writer_done) { + if (gp_async) { + if (gp_async_max <= 0) { + pr_warn("%s: gp_async_max = %d must be greater than zero.\n", + __func__, gp_async_max); + WARN_ON_ONCE(IS_BUILTIN(CONFIG_RCU_TORTURE_TEST)); + firsterr = -EINVAL; + goto unwind; + } + writer_freelists = kcalloc(nrealwriters, sizeof(writer_freelists[0]), GFP_KERNEL); + } + if (!writer_tasks || !writer_durations || !writer_n_durations || !writer_done || + (gp_async && !writer_freelists)) { SCALEOUT_ERRSTRING("out of memory"); firsterr = -ENOMEM; goto unwind; @@ -1087,6 +1172,24 @@ rcu_scale_init(void) firsterr = -ENOMEM; goto unwind; } + if (writer_freelists) { + struct writer_freelist *wflp = &writer_freelists[i]; + + init_llist_head(&wflp->ws_lhg); + init_llist_head(&wflp->ws_lhp); + wflp->ws_mblocks = kcalloc(gp_async_max, sizeof(wflp->ws_mblocks[0]), + GFP_KERNEL); + if (!wflp->ws_mblocks) { + firsterr = -ENOMEM; + goto unwind; + } + for (j = 0; j < gp_async_max; j++) { + struct writer_mblock *wmbp = &wflp->ws_mblocks[j]; + + wmbp->wmb_wfl = wflp; + llist_add(&wmbp->wmb_node, &wflp->ws_lhp); + } + } firsterr = torture_create_kthread(rcu_scale_writer, (void *)i, writer_tasks[i]); if (torture_init_error(firsterr)) From patchwork Fri Aug 16 07:02:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neeraj Upadhyay X-Patchwork-Id: 13765541 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4A27313D61A; Fri, 16 Aug 2024 07:05:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723791905; cv=none; b=PTHinGHUCBvsoaJbSg38uufj5wXxBB1tCZdKTMMJPAdVJdO4SPrLRZBvTNu0h/HeE0Ub3PF1gm4r6DL15tD+i/SvJzdRm5Aak9R5a8sa95S96k68l1bFZYap3HYMLxvkK9X5PSd1d/wgxxdmIiryxhMFF5V+KsLN9GRhFVCK59I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723791905; c=relaxed/simple; bh=LLMOR7Lho0MmgHN1fCBe5uVw2BOnsaGjuDnrIUmfdsQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=qVeG3WE2adeLi4dtcEPEzmbHp03lvfuIt81AHLmxxB+43/RWBA8PaQTFNLu81GKAnn3O2Ghbh+PyKUwNOLQyaEHW8rcohpxpei11iEbuvNEtI182TVXP+ZRua8cgWUc6CFwGudu61YTXckq1SUVTPsu1P1CqM/UBmAAHMstGpLY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=FcuzodBp; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="FcuzodBp" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 03CC2C4AF12; Fri, 16 Aug 2024 07:04:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723791904; bh=LLMOR7Lho0MmgHN1fCBe5uVw2BOnsaGjuDnrIUmfdsQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=FcuzodBp6qop+A+/AgZb5spP+GIxDSdZ8Xnu93CwDSL1fUdTb966t/gM2+Lq4Myhw 2e8Mk7P4i/XeY4oI4PzNP3pH6DipsYkxIhE8FACq+hQL3xC+nmBmAXBrs8wY3aISDm VRLnQCuV2eudJDqnJy/wxgZsqzAJSh3iHQOEvqlbsTwqRADgqnhLPwRnOWXUUGtCFW s3ss/tdOu9kYh43Wi3nrOOzQX4Tv26whA7MgxHFZLJwkb3beUwwujQGxzZzCrExqoI W+KPmbLmzGZJQqatiJeTANW2nxjaRRLSPTU7HUfvVcn+7LpJqApcJr0c8PmSz8Z3Ms 5DQEopW6ujKCQ== From: neeraj.upadhyay@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, paulmck@kernel.org, neeraj.upadhyay@kernel.org, neeraj.upadhyay@amd.com, boqun.feng@gmail.com, joel@joelfernandes.org, urezki@gmail.com, frederic@kernel.org Subject: [PATCH rcu 12/14] rcuscale: NULL out top-level pointers to heap memory Date: Fri, 16 Aug 2024 12:32:54 +0530 Message-Id: <20240816070256.60993-12-neeraj.upadhyay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240816070209.GA60666@neeraj.linux> References: <20240816070209.GA60666@neeraj.linux> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" Currently, if someone modprobes and rmmods rcuscale successfully, but the next run errors out during the modprobe, non-NULL pointers to freed memory will remain. If the run after that also errors out during the modprobe, there will be double-free bugs. This commit therefore NULLs out top-level pointers to memory that has just been freed. Signed-off-by: "Paul E. McKenney" Signed-off-by: Neeraj Upadhyay --- kernel/rcu/rcuscale.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/kernel/rcu/rcuscale.c b/kernel/rcu/rcuscale.c index bc7cca979c06..61a178914256 100644 --- a/kernel/rcu/rcuscale.c +++ b/kernel/rcu/rcuscale.c @@ -819,6 +819,7 @@ kfree_scale_cleanup(void) torture_stop_kthread(kfree_scale_thread, kfree_reader_tasks[i]); kfree(kfree_reader_tasks); + kfree_reader_tasks = NULL; } torture_cleanup_end(); @@ -987,6 +988,7 @@ rcu_scale_cleanup(void) torture_stop_kthread(rcu_scale_reader, reader_tasks[i]); kfree(reader_tasks); + reader_tasks = NULL; } if (writer_tasks) { @@ -1043,8 +1045,11 @@ rcu_scale_cleanup(void) } } kfree(writer_tasks); + writer_tasks = NULL; kfree(writer_durations); + writer_durations = NULL; kfree(writer_n_durations); + writer_n_durations = NULL; kfree(writer_done); writer_done = NULL; kfree(writer_freelists); From patchwork Fri Aug 16 07:02:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neeraj Upadhyay X-Patchwork-Id: 13765542 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A6E2413E033; Fri, 16 Aug 2024 07:05:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723791911; cv=none; b=cKkywFFCEwruYeXlC5DXCzJPb62kzqNGEw+02wfAA8TpC7pCx0YwJgcZTzOEWnv+Zpv1k2yndYRpwD9lSHfjzmBrlr9oXqEgU9xMiAnraS8bRi0sLXEYbdcj8maVrJD/s2PuQDvG+uz4tkUpzcIoH6ixPSsiyaYeOLYs7dJPmdU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723791911; c=relaxed/simple; bh=3CHmrIAyszwFM0IWdsUcipqJUlHGF9O40d3/y8Ctn4A=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=OsjlQEChadmWvp6w0af3F395BmHSzI1Ol7sX54ofDqVD8T5KLLjGwOsQ6scjUALdpnYle0JpPA5XdJ8xqB4f1YSrV3b6nk6DbtKu8SGCDafvgJs5ZBrquHX93/gqbmtuJNLxobjII7Mc/LS0Cc/fYkOfxsjRpvQhfuPtaEXJ42Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=uCcTGXNv; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="uCcTGXNv" Received: by smtp.kernel.org (Postfix) with ESMTPSA id B76A0C32782; Fri, 16 Aug 2024 07:05:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723791911; bh=3CHmrIAyszwFM0IWdsUcipqJUlHGF9O40d3/y8Ctn4A=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=uCcTGXNvKuiEBEs4A/3AWTdoPqqkAkGw9bsS1Zy9XUwGjtQqdBgwl02KLA1MBGFad 4EvejIRjNVg4QuLuGgtesKHlZNWuEQNNpRK48mM/UqFuWILGzl4x+qmc63qXn2xWM7 OcPVP1/5n2dhicaMyG4zYZ88o1n3VNGthLpBaQtl9LT1+Mj0wZ6wdYEiArlXUGol9x sPJyovQN/kM8k4Q2KM6caJrm/NGZhgVDfMsmcKAdJytWurbp8lZeFyHfy+9v1TvUAw EqP5YXoiv6IClhh/jt5C1W03nsa0R3LWRTEOU0pTmUbZVaXkLmTQ3SFPY896KVCBBR cCTLpaTSyUEyA== From: neeraj.upadhyay@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, paulmck@kernel.org, neeraj.upadhyay@kernel.org, neeraj.upadhyay@amd.com, boqun.feng@gmail.com, joel@joelfernandes.org, urezki@gmail.com, frederic@kernel.org, Vlastimil Babka Subject: [PATCH rcu 13/14] rcuscale: Count outstanding callbacks per-task rather than per-CPU Date: Fri, 16 Aug 2024 12:32:55 +0530 Message-Id: <20240816070256.60993-13-neeraj.upadhyay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240816070209.GA60666@neeraj.linux> References: <20240816070209.GA60666@neeraj.linux> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" The current rcu_scale_writer() asynchronous grace-period testing uses a per-CPU counter to track the number of outstanding callbacks. This is subject to CPU-imbalance errors when tasks migrate from one CPU to another between the time that the counter is incremented and the callback is queued, and additionally in kernels configured such that callbacks can be invoked on some CPU other than the one that queued it. This commit therefore arranges for per-task callback counts, thus avoiding any issues with migration of either tasks or callbacks. Reported-by: Vlastimil Babka Signed-off-by: "Paul E. McKenney" Signed-off-by: Neeraj Upadhyay --- kernel/rcu/rcuscale.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/kernel/rcu/rcuscale.c b/kernel/rcu/rcuscale.c index 61a178914256..6d37596deb1f 100644 --- a/kernel/rcu/rcuscale.c +++ b/kernel/rcu/rcuscale.c @@ -114,6 +114,7 @@ struct writer_mblock { struct writer_freelist { struct llist_head ws_lhg; + atomic_t ws_inflight; struct llist_head ____cacheline_internodealigned_in_smp ws_lhp; struct writer_mblock *ws_mblocks; }; @@ -136,7 +137,6 @@ static u64 t_rcu_scale_writer_started; static u64 t_rcu_scale_writer_finished; static unsigned long b_rcu_gp_test_started; static unsigned long b_rcu_gp_test_finished; -static DEFINE_PER_CPU(atomic_t, n_async_inflight); #define MAX_MEAS 10000 #define MIN_MEAS 100 @@ -520,8 +520,9 @@ static void rcu_scale_free(struct writer_mblock *wmbp) static void rcu_scale_async_cb(struct rcu_head *rhp) { struct writer_mblock *wmbp = container_of(rhp, struct writer_mblock, wmb_rh); + struct writer_freelist *wflp = wmbp->wmb_wfl; - atomic_dec(this_cpu_ptr(&n_async_inflight)); + atomic_dec(&wflp->ws_inflight); rcu_scale_free(wmbp); } @@ -541,6 +542,7 @@ rcu_scale_writer(void *arg) DEFINE_TORTURE_RANDOM(tr); u64 *wdp; u64 *wdpp = writer_durations[me]; + struct writer_freelist *wflp = &writer_freelists[me]; struct writer_mblock *wmbp = NULL; VERBOSE_SCALEOUT_STRING("rcu_scale_writer task started"); @@ -584,8 +586,8 @@ rcu_scale_writer(void *arg) if (gp_async && !WARN_ON_ONCE(!cur_ops->async)) { if (!wmbp) wmbp = rcu_scale_alloc(me); - if (wmbp && atomic_read(this_cpu_ptr(&n_async_inflight)) < gp_async_max) { - atomic_inc(this_cpu_ptr(&n_async_inflight)); + if (wmbp && atomic_read(&wflp->ws_inflight) < gp_async_max) { + atomic_inc(&wflp->ws_inflight); cur_ops->async(&wmbp->wmb_rh, rcu_scale_async_cb); wmbp = NULL; gp_succeeded = true; From patchwork Fri Aug 16 07:02:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neeraj Upadhyay X-Patchwork-Id: 13765543 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0F56213BC1B; Fri, 16 Aug 2024 07:05:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723791918; cv=none; b=TNXKdimyuwB9BrrvAJUuSi1s0CP7Rq7K2rk8g2UYRsXE9ZjkMSceHGNX7+3Zzbe1+QV1AGpiUSoNdAdRfmvfQM59Hqu0GevtRPaM4UGcdpK6j0lrBRK/V8X9takwjAovWuww1bMR+1E2yECkUniWcmbpsiCa9fY9sFDufOu/eJA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723791918; c=relaxed/simple; bh=8pjw6uKpTTYrTKlKksDSIG0o84ZLVtTrpHbW5hmvNOs=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=YbPjsBLP31Zbk/Xj61umyuEocJhMLW7ArDfcnO3llYHu3yrtrSGJaskFxjQ1TqZk3G6r1vYRH3jFseghJZnnEmzy7omzqjOReP9FKJbIDxZgGgfVUTdlstC/ZkjcsRnFXUqIsBqfWwdQsAG1vN2hsPi8ivu/7th0t2t2wRoNN54= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=AtpWgPG2; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="AtpWgPG2" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 67210C32782; Fri, 16 Aug 2024 07:05:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723791917; bh=8pjw6uKpTTYrTKlKksDSIG0o84ZLVtTrpHbW5hmvNOs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=AtpWgPG2K+2P8vca/MtMmTfCTwXci91pdkbVwHq5qc3C/doTL/hc09MC26LrZJ5HT 0ZLoTOGZkPZgVIRsSI488mxxOvWZmsqiPDdy1m8Wv1BOJI5r5GTutm7gFneoeKCtPP vw6HQWZf8F/1FPH+2VU+BAyyx9kL2h0oXWW53nfJJ+ToSI17rlgJdOOUw6hvvZggRM FXa/f142S8FloRs8RNJ+kfl2MLByI1KnLO9RRPgNGyWRuaLbNnQZl/nKmnEeEwwBRa Q3E0TjtBbLwHg6859YD6zWoWCQo1/jF7tMvt0V1XXHvEaAepHKyUuxdcjHiJk7hLsR 0uVRDwH/HEdow== From: neeraj.upadhyay@kernel.org To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, paulmck@kernel.org, neeraj.upadhyay@kernel.org, neeraj.upadhyay@amd.com, boqun.feng@gmail.com, joel@joelfernandes.org, urezki@gmail.com, frederic@kernel.org, Christophe JAILLET Subject: [PATCH rcu 14/14] refscale: Constify struct ref_scale_ops Date: Fri, 16 Aug 2024 12:32:56 +0530 Message-Id: <20240816070256.60993-14-neeraj.upadhyay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240816070209.GA60666@neeraj.linux> References: <20240816070209.GA60666@neeraj.linux> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Christophe JAILLET 'struct ref_scale_ops' are not modified in these drivers. Constifying this structure moves some data to a read-only section, so increase overall security. On a x86_64, with allmodconfig: Before: ====== text data bss dec hex filename 34231 4167 736 39134 98de kernel/rcu/refscale.o After: ===== text data bss dec hex filename 35175 3239 736 39150 98ee kernel/rcu/refscale.o Signed-off-by: Christophe JAILLET Tested-by: "Paul E. McKenney" Signed-off-by: Neeraj Upadhyay --- kernel/rcu/refscale.c | 42 +++++++++++++++++++++--------------------- 1 file changed, 21 insertions(+), 21 deletions(-) diff --git a/kernel/rcu/refscale.c b/kernel/rcu/refscale.c index cfec0648e141..0db9db73f57f 100644 --- a/kernel/rcu/refscale.c +++ b/kernel/rcu/refscale.c @@ -135,7 +135,7 @@ struct ref_scale_ops { const char *name; }; -static struct ref_scale_ops *cur_ops; +static const struct ref_scale_ops *cur_ops; static void un_delay(const int udl, const int ndl) { @@ -171,7 +171,7 @@ static bool rcu_sync_scale_init(void) return true; } -static struct ref_scale_ops rcu_ops = { +static const struct ref_scale_ops rcu_ops = { .init = rcu_sync_scale_init, .readsection = ref_rcu_read_section, .delaysection = ref_rcu_delay_section, @@ -205,7 +205,7 @@ static void srcu_ref_scale_delay_section(const int nloops, const int udl, const } } -static struct ref_scale_ops srcu_ops = { +static const struct ref_scale_ops srcu_ops = { .init = rcu_sync_scale_init, .readsection = srcu_ref_scale_read_section, .delaysection = srcu_ref_scale_delay_section, @@ -232,7 +232,7 @@ static void rcu_tasks_ref_scale_delay_section(const int nloops, const int udl, c un_delay(udl, ndl); } -static struct ref_scale_ops rcu_tasks_ops = { +static const struct ref_scale_ops rcu_tasks_ops = { .init = rcu_sync_scale_init, .readsection = rcu_tasks_ref_scale_read_section, .delaysection = rcu_tasks_ref_scale_delay_section, @@ -271,7 +271,7 @@ static void rcu_trace_ref_scale_delay_section(const int nloops, const int udl, c } } -static struct ref_scale_ops rcu_trace_ops = { +static const struct ref_scale_ops rcu_trace_ops = { .init = rcu_sync_scale_init, .readsection = rcu_trace_ref_scale_read_section, .delaysection = rcu_trace_ref_scale_delay_section, @@ -310,7 +310,7 @@ static void ref_refcnt_delay_section(const int nloops, const int udl, const int } } -static struct ref_scale_ops refcnt_ops = { +static const struct ref_scale_ops refcnt_ops = { .init = rcu_sync_scale_init, .readsection = ref_refcnt_section, .delaysection = ref_refcnt_delay_section, @@ -347,7 +347,7 @@ static void ref_rwlock_delay_section(const int nloops, const int udl, const int } } -static struct ref_scale_ops rwlock_ops = { +static const struct ref_scale_ops rwlock_ops = { .init = ref_rwlock_init, .readsection = ref_rwlock_section, .delaysection = ref_rwlock_delay_section, @@ -384,7 +384,7 @@ static void ref_rwsem_delay_section(const int nloops, const int udl, const int n } } -static struct ref_scale_ops rwsem_ops = { +static const struct ref_scale_ops rwsem_ops = { .init = ref_rwsem_init, .readsection = ref_rwsem_section, .delaysection = ref_rwsem_delay_section, @@ -419,7 +419,7 @@ static void ref_lock_delay_section(const int nloops, const int udl, const int nd preempt_enable(); } -static struct ref_scale_ops lock_ops = { +static const struct ref_scale_ops lock_ops = { .readsection = ref_lock_section, .delaysection = ref_lock_delay_section, .name = "lock" @@ -454,7 +454,7 @@ static void ref_lock_irq_delay_section(const int nloops, const int udl, const in preempt_enable(); } -static struct ref_scale_ops lock_irq_ops = { +static const struct ref_scale_ops lock_irq_ops = { .readsection = ref_lock_irq_section, .delaysection = ref_lock_irq_delay_section, .name = "lock-irq" @@ -490,7 +490,7 @@ static void ref_acqrel_delay_section(const int nloops, const int udl, const int preempt_enable(); } -static struct ref_scale_ops acqrel_ops = { +static const struct ref_scale_ops acqrel_ops = { .readsection = ref_acqrel_section, .delaysection = ref_acqrel_delay_section, .name = "acqrel" @@ -524,7 +524,7 @@ static void ref_clock_delay_section(const int nloops, const int udl, const int n stopopts = x; } -static struct ref_scale_ops clock_ops = { +static const struct ref_scale_ops clock_ops = { .readsection = ref_clock_section, .delaysection = ref_clock_delay_section, .name = "clock" @@ -556,7 +556,7 @@ static void ref_jiffies_delay_section(const int nloops, const int udl, const int stopopts = x; } -static struct ref_scale_ops jiffies_ops = { +static const struct ref_scale_ops jiffies_ops = { .readsection = ref_jiffies_section, .delaysection = ref_jiffies_delay_section, .name = "jiffies" @@ -706,9 +706,9 @@ static void refscale_typesafe_ctor(void *rtsp_in) preempt_enable(); } -static struct ref_scale_ops typesafe_ref_ops; -static struct ref_scale_ops typesafe_lock_ops; -static struct ref_scale_ops typesafe_seqlock_ops; +static const struct ref_scale_ops typesafe_ref_ops; +static const struct ref_scale_ops typesafe_lock_ops; +static const struct ref_scale_ops typesafe_seqlock_ops; // Initialize for a typesafe test. static bool typesafe_init(void) @@ -769,7 +769,7 @@ static void typesafe_cleanup(void) } // The typesafe_init() function distinguishes these structures by address. -static struct ref_scale_ops typesafe_ref_ops = { +static const struct ref_scale_ops typesafe_ref_ops = { .init = typesafe_init, .cleanup = typesafe_cleanup, .readsection = typesafe_read_section, @@ -777,7 +777,7 @@ static struct ref_scale_ops typesafe_ref_ops = { .name = "typesafe_ref" }; -static struct ref_scale_ops typesafe_lock_ops = { +static const struct ref_scale_ops typesafe_lock_ops = { .init = typesafe_init, .cleanup = typesafe_cleanup, .readsection = typesafe_read_section, @@ -785,7 +785,7 @@ static struct ref_scale_ops typesafe_lock_ops = { .name = "typesafe_lock" }; -static struct ref_scale_ops typesafe_seqlock_ops = { +static const struct ref_scale_ops typesafe_seqlock_ops = { .init = typesafe_init, .cleanup = typesafe_cleanup, .readsection = typesafe_read_section, @@ -1026,7 +1026,7 @@ static int main_func(void *arg) } static void -ref_scale_print_module_parms(struct ref_scale_ops *cur_ops, const char *tag) +ref_scale_print_module_parms(const struct ref_scale_ops *cur_ops, const char *tag) { pr_alert("%s" SCALE_FLAG "--- %s: verbose=%d verbose_batched=%d shutdown=%d holdoff=%d lookup_instances=%ld loops=%ld nreaders=%d nruns=%d readdelay=%d\n", scale_type, tag, @@ -1081,7 +1081,7 @@ ref_scale_init(void) { long i; int firsterr = 0; - static struct ref_scale_ops *scale_ops[] = { + static const struct ref_scale_ops *scale_ops[] = { &rcu_ops, &srcu_ops, RCU_TRACE_OPS RCU_TASKS_OPS &refcnt_ops, &rwlock_ops, &rwsem_ops, &lock_ops, &lock_irq_ops, &acqrel_ops, &clock_ops, &jiffies_ops, &typesafe_ref_ops, &typesafe_lock_ops, &typesafe_seqlock_ops,