From patchwork Sat Jul 29 14:27:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 13333188 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 98324C001DF for ; Sat, 29 Jul 2023 14:28:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231977AbjG2O2j (ORCPT ); Sat, 29 Jul 2023 10:28:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56958 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232004AbjG2O2f (ORCPT ); Sat, 29 Jul 2023 10:28:35 -0400 Received: from mail-yb1-xb2b.google.com (mail-yb1-xb2b.google.com [IPv6:2607:f8b0:4864:20::b2b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 615B246B4 for ; Sat, 29 Jul 2023 07:28:18 -0700 (PDT) Received: by mail-yb1-xb2b.google.com with SMTP id 3f1490d57ef6-cb19b1b9a36so3004588276.0 for ; Sat, 29 Jul 2023 07:28:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; t=1690640896; x=1691245696; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=iwxEGA9bB7b0DElBaIYcQs9XoGIY+NHTZRuaPpVwP9M=; b=bgVc8RgM8MUE3/FR/qxJAAVnZPDSL8B/L8Y55nDBH4dQN+gKdet8DkRYtYGE5rhJKB 5PFNZ2s+lGod3ijPAm+7YYjP7qeW1FBV3BCBrlG7+C52Vjzf4HQApIQy+8ztJrApuvFA jh0kfZEj3pdfu2H7JHKa7COI15JG/bknJpcRI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690640896; x=1691245696; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=iwxEGA9bB7b0DElBaIYcQs9XoGIY+NHTZRuaPpVwP9M=; b=Zqn7ETlW6f3sZArzi7ZK0n8AyveN3//kAUTB31rbePucGt/8pg8A4n1uqYWpRfN9Uu F3WPpuRTjHfnye0spGz9qLYMPVzGb+CFLABKOaDS6rCuBT/oNqLjyyZA1JtQnzsi3ypL A/3ejUNgaeGa9OU57p1s50XiEiIxTr/GeZ5jNWkDnEGemAvRl0GSOYGUPb+JA38ZMryh shO5372HQc0egqH+q5q2ZuaAObZQ2vho4e4+53wOASyU60F0CA0Qzb2pOg8ZJpbE8wQP MGc3IgsVftzrJTX51QWUnIKdBVP2nk6xybxZOlxIEKUo9PyiG9+uOGt1QpJPgvljeKAk 6Ozw== X-Gm-Message-State: ABy/qLZYUSkelkn7KuShGMFtSmLH0/uP7HLGw0vRxkRalX2N/jBZJEbt lohi262kLK291cxs8rdrwHX4rW+k/lfdA9zcGig= X-Google-Smtp-Source: APBJJlFk+pVi5ucHAZ4YN78zym81iXhYotkmEVyM8rGXoNXNjoUsMje3qOOrHrpWOnLI579JZtCj9g== X-Received: by 2002:a92:cdab:0:b0:348:8050:d7a with SMTP id g11-20020a92cdab000000b0034880500d7amr3082832ild.30.1690640875892; Sat, 29 Jul 2023 07:27:55 -0700 (PDT) Received: from joelboxx5.c.googlers.com.com (254.82.172.34.bc.googleusercontent.com. [34.172.82.254]) by smtp.gmail.com with ESMTPSA id q9-20020a0566380ec900b0042b35c7b8c5sm1855362jas.61.2023.07.29.07.27.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 29 Jul 2023 07:27:55 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org Cc: rcu@vger.kernel.org, paulmck@kernel.org, "Joel Fernandes (Google)" , stable@vger.kernel.org, Davidlohr Bueso , Josh Triplett Subject: [PATCH v2 1/5] rcutorture: Fix stuttering races and other issues Date: Sat, 29 Jul 2023 14:27:31 +0000 Message-ID: <20230729142738.222208-2-joel@joelfernandes.org> X-Mailer: git-send-email 2.41.0.487.g6d72f3e995-goog In-Reply-To: <20230729142738.222208-1-joel@joelfernandes.org> References: <20230729142738.222208-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org The stuttering code isn't functioning as expected. Ideally, it should pause the torture threads for a designated period before resuming. Yet, it fails to halt the test for the correct duration. Additionally, a race condition exists, potentially causing the stuttering code to pause for an extended period if the 'spt' variable is non-zero due to the stutter orchestration thread's inadequate CPU time. Moreover, over-stuttering can hinder RCU's progress on TREE07 kernels. This happens as the stuttering code may run within a softirq due to RCU callbacks. Consequently, ksoftirqd keeps a CPU busy for several seconds, thus obstructing RCU's progress. This situation triggers a warning message in the logs: [ 2169.481783] rcu_torture_writer: rtort_pipe_count: 9 This warning suggests that an RCU torture object, although invisible to RCU readers, couldn't make it past the pipe array and be freed -- a strong indication that there weren't enough grace periods during the stutter interval. To address these issues, this patch sets the "stutter end" time to an absolute point in the future set by the main stutter thread. This is then used for waiting in stutter_wait(). While the stutter thread still defines this absolute time, the waiters' waiting logic doesn't rely on the stutter thread receiving sufficient CPU time to halt the stuttering as the halting is now self-controlled. Cc: stable@vger.kernel.org Signed-off-by: Joel Fernandes (Google) --- kernel/torture.c | 45 ++++++++++++--------------------------------- 1 file changed, 12 insertions(+), 33 deletions(-) diff --git a/kernel/torture.c b/kernel/torture.c index 6ba62e5993e7..fd353f98162f 100644 --- a/kernel/torture.c +++ b/kernel/torture.c @@ -720,7 +720,7 @@ static void torture_shutdown_cleanup(void) * suddenly applied to or removed from the system. */ static struct task_struct *stutter_task; -static int stutter_pause_test; +static ktime_t stutter_till_abs_time; static int stutter; static int stutter_gap; @@ -730,30 +730,16 @@ static int stutter_gap; */ bool stutter_wait(const char *title) { - unsigned int i = 0; bool ret = false; - int spt; + ktime_t till_ns; cond_resched_tasks_rcu_qs(); - spt = READ_ONCE(stutter_pause_test); - for (; spt; spt = READ_ONCE(stutter_pause_test)) { - if (!ret && !rt_task(current)) { - sched_set_normal(current, MAX_NICE); - ret = true; - } - if (spt == 1) { - torture_hrtimeout_jiffies(1, NULL); - } else if (spt == 2) { - while (READ_ONCE(stutter_pause_test)) { - if (!(i++ & 0xffff)) - torture_hrtimeout_us(10, 0, NULL); - cond_resched(); - } - } else { - torture_hrtimeout_jiffies(round_jiffies_relative(HZ), NULL); - } - torture_shutdown_absorb(title); + till_ns = READ_ONCE(stutter_till_abs_time); + if (till_ns && ktime_before(ktime_get(), till_ns)) { + torture_hrtimeout_ns(till_ns, 0, HRTIMER_MODE_ABS, NULL); + ret = true; } + torture_shutdown_absorb(title); return ret; } EXPORT_SYMBOL_GPL(stutter_wait); @@ -764,23 +750,16 @@ EXPORT_SYMBOL_GPL(stutter_wait); */ static int torture_stutter(void *arg) { - DEFINE_TORTURE_RANDOM(rand); - int wtime; + ktime_t till_ns; VERBOSE_TOROUT_STRING("torture_stutter task started"); do { if (!torture_must_stop() && stutter > 1) { - wtime = stutter; - if (stutter > 2) { - WRITE_ONCE(stutter_pause_test, 1); - wtime = stutter - 3; - torture_hrtimeout_jiffies(wtime, &rand); - wtime = 2; - } - WRITE_ONCE(stutter_pause_test, 2); - torture_hrtimeout_jiffies(wtime, NULL); + till_ns = ktime_add_ns(ktime_get(), + jiffies_to_nsecs(stutter)); + WRITE_ONCE(stutter_till_abs_time, till_ns); + torture_hrtimeout_jiffies(stutter - 1, NULL); } - WRITE_ONCE(stutter_pause_test, 0); if (!torture_must_stop()) torture_hrtimeout_jiffies(stutter_gap, NULL); torture_shutdown_absorb("torture_stutter"); From patchwork Sat Jul 29 14:27:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 13333183 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A174BC001E0 for ; Sat, 29 Jul 2023 14:28:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231878AbjG2O17 (ORCPT ); Sat, 29 Jul 2023 10:27:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56458 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231894AbjG2O16 (ORCPT ); Sat, 29 Jul 2023 10:27:58 -0400 Received: from mail-il1-x131.google.com (mail-il1-x131.google.com [IPv6:2607:f8b0:4864:20::131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 88DF8E7A for ; Sat, 29 Jul 2023 07:27:57 -0700 (PDT) Received: by mail-il1-x131.google.com with SMTP id e9e14a558f8ab-348de515667so13590075ab.1 for ; Sat, 29 Jul 2023 07:27:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; t=1690640877; x=1691245677; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=K/AP9GOcXVQHz8AOI7UjuaU+sYzEqAzG6F3W2fxwbBg=; b=J4kyjvgZ82W4wgXCeEeMZsVCguRFpikKkcE2HcgJj1qeY7vt+QnMIYGWOOT+WB4w/d QxBIq/qctXIL+Pq+QR5781uOJJWRU23mvLFOkz/5oH2U307qLuDCMVxppr9LWan2wYgB x4GjVCbkk7A28bEYLtEBKdloUvZLupH9Z4E6M= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690640877; x=1691245677; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=K/AP9GOcXVQHz8AOI7UjuaU+sYzEqAzG6F3W2fxwbBg=; b=A8r9GcJeh0g3EkrZrOHVxtHUdcO60GNG6suqlheDGMhc4aDroEac9ya2KjS0xtyLsr geGviMn+xVj2YtH72Pos470tnz/Mly6EZ3oPsTFaAt0lMaWQR7H3rTiqTjLsq/cw9kv8 v2m0d8bXB0kZDHpW83iIYnVbIie0k7i6vQj15g+tvtj4wBjtQqbwZ5biwJl0xdhXHbwi fmsfNY62XAygNg0H6CjHXaPbcJ+yDQIHYr2gouwaMdboodosg2Cq1j9+g95rCWIqQA+b 6nrU+ZnVFxUn06JhOIMZKGqLLT1JCAZl6BW9U8k8/lqsqhs0FQDZfEUVQnAXthLiynDj vLEw== X-Gm-Message-State: ABy/qLYfE4IaDP9W5GhGHvQBreeZeCkhpo+E1cJbosgwc5npiG9cbrRn mxBQ6aqV2jLyCyN+h5AVJiGf/Q== X-Google-Smtp-Source: APBJJlE8nBeVkXcUMALJORNmqTh8c7r1ht4vw0YPrIrwKi+ZvPe7AoO4UA+ZLaLbe2+FsI5VCRzahg== X-Received: by 2002:a92:c7c1:0:b0:345:ad81:ecaf with SMTP id g1-20020a92c7c1000000b00345ad81ecafmr2084447ilk.3.1690640876885; Sat, 29 Jul 2023 07:27:56 -0700 (PDT) Received: from joelboxx5.c.googlers.com.com (254.82.172.34.bc.googleusercontent.com. [34.172.82.254]) by smtp.gmail.com with ESMTPSA id q9-20020a0566380ec900b0042b35c7b8c5sm1855362jas.61.2023.07.29.07.27.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 29 Jul 2023 07:27:56 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org Cc: rcu@vger.kernel.org, paulmck@kernel.org, "Joel Fernandes (Google)" , Lai Jiangshan , Josh Triplett , Steven Rostedt , Mathieu Desnoyers Subject: [PATCH v2 2/5] srcu: Fix error handling in init_srcu_struct_fields() Date: Sat, 29 Jul 2023 14:27:32 +0000 Message-ID: <20230729142738.222208-3-joel@joelfernandes.org> X-Mailer: git-send-email 2.41.0.487.g6d72f3e995-goog In-Reply-To: <20230729142738.222208-1-joel@joelfernandes.org> References: <20230729142738.222208-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org The current error handling in init_srcu_struct_fields() is a bit inconsistent. If init_srcu_struct_nodes() fails, the function either returns -ENOMEM or 0 depending on whether ssp->sda_is_static is true or false. This can make init_srcu_struct_fields() return 0 even if memory allocation failed! Simplify the error handling by always returning -ENOMEM if either init_srcu_struct_nodes() or the per-CPU allocation fails. This makes the control flow easier to follow and avoids the inconsistent return values. Add goto labels to avoid duplicating the error cleanup code. Link: https://lore.kernel.org/r/20230404003508.GA254019@google.com Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/srcutree.c | 32 +++++++++++++++++--------------- 1 file changed, 17 insertions(+), 15 deletions(-) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 20d7a238d675..f1a905200fc2 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -255,29 +255,31 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp, bool is_static) ssp->srcu_sup->sda_is_static = is_static; if (!is_static) ssp->sda = alloc_percpu(struct srcu_data); - if (!ssp->sda) { - if (!is_static) - kfree(ssp->srcu_sup); - return -ENOMEM; - } + if (!ssp->sda) + goto err_free_sup; init_srcu_struct_data(ssp); ssp->srcu_sup->srcu_gp_seq_needed_exp = 0; ssp->srcu_sup->srcu_last_gp_end = ktime_get_mono_fast_ns(); if (READ_ONCE(ssp->srcu_sup->srcu_size_state) == SRCU_SIZE_SMALL && SRCU_SIZING_IS_INIT()) { - if (!init_srcu_struct_nodes(ssp, GFP_ATOMIC)) { - if (!ssp->srcu_sup->sda_is_static) { - free_percpu(ssp->sda); - ssp->sda = NULL; - kfree(ssp->srcu_sup); - return -ENOMEM; - } - } else { - WRITE_ONCE(ssp->srcu_sup->srcu_size_state, SRCU_SIZE_BIG); - } + if (!init_srcu_struct_nodes(ssp, GFP_ATOMIC)) + goto err_free_sda; + WRITE_ONCE(ssp->srcu_sup->srcu_size_state, SRCU_SIZE_BIG); } ssp->srcu_sup->srcu_ssp = ssp; smp_store_release(&ssp->srcu_sup->srcu_gp_seq_needed, 0); /* Init done. */ return 0; + +err_free_sda: + if (!is_static) { + free_percpu(ssp->sda); + ssp->sda = NULL; + } +err_free_sup: + if (!is_static) { + kfree(ssp->srcu_sup); + ssp->srcu_sup = NULL; + } + return -ENOMEM; } #ifdef CONFIG_DEBUG_LOCK_ALLOC From patchwork Sat Jul 29 14:27:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 13333184 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E3485C001DF for ; Sat, 29 Jul 2023 14:28:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231890AbjG2O2A (ORCPT ); Sat, 29 Jul 2023 10:28:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56470 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231919AbjG2O17 (ORCPT ); Sat, 29 Jul 2023 10:27:59 -0400 Received: from mail-il1-x132.google.com (mail-il1-x132.google.com [IPv6:2607:f8b0:4864:20::132]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 808F430F4 for ; Sat, 29 Jul 2023 07:27:58 -0700 (PDT) Received: by mail-il1-x132.google.com with SMTP id e9e14a558f8ab-34911a634edso1242395ab.0 for ; Sat, 29 Jul 2023 07:27:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; t=1690640878; x=1691245678; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=B6faQrb3vVDF+1ZA3HcLro9nEgsaXGn2RtvbxbNNEvc=; b=plXTHquWdh4bDvLFn2iXXCHJmvv/+MoXo8lRqYopE64G7FgOHv+cnT2c05bBwokNCo f2+s+ItwB5Hhqd7VCNCKAVOINhdBw91B9XtHD1TQBm0bxbtcj/QaCeXNKMY+P2vShgB/ AyVO2DIkMfhumM/yGpIR5gkL7/1244kpfN2Zs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690640878; x=1691245678; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=B6faQrb3vVDF+1ZA3HcLro9nEgsaXGn2RtvbxbNNEvc=; b=ZOVIeUNwHnciHzn1kafWjukUCq2lAklwyjW8TVhmXyCIpQ3RHFDTRDoPIMbTW1DAwj gqMyc3D6PpYZZGGvcZhef7N9KUOf781M3/bPPuLTHzRbtErWcO2eaEGbnce0Nb7/zaUO vuD183p2Dg5igQCOR2SND+jGI43Qo3CVpZ2ld0Fo7/saTYSVpSOdCYwni99OJiaf4nHN ghoTIXV+/G/9yZOuvmvZanBSZaknAjuvj05sUIho1q6JF3HK8V7whD1jB83ZcLlKX13H 21XebgL6GKxYqgzHAzcNjKv2l3TpRHG4aaJM3gXwVDUMviroce1WmR2woGnahG4RCkaH 7xIg== X-Gm-Message-State: ABy/qLYQxAlNS5LxsPEkqI7xI/094KE+6J91phev0y57fgI7DjP8XByK h4rZFvjbBTSQEeSaPwLRqv49Qw== X-Google-Smtp-Source: APBJJlHK/5GmyzQgFG1Ksqn6znBGEhc+zRSHdPwsSlZO5dCGgHvfR+k/CmjuBsoXHLn5orj/tiR73g== X-Received: by 2002:a05:6e02:1d94:b0:348:f6cb:e585 with SMTP id h20-20020a056e021d9400b00348f6cbe585mr2922758ila.30.1690640877759; Sat, 29 Jul 2023 07:27:57 -0700 (PDT) Received: from joelboxx5.c.googlers.com.com (254.82.172.34.bc.googleusercontent.com. [34.172.82.254]) by smtp.gmail.com with ESMTPSA id q9-20020a0566380ec900b0042b35c7b8c5sm1855362jas.61.2023.07.29.07.27.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 29 Jul 2023 07:27:57 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org Cc: rcu@vger.kernel.org, paulmck@kernel.org, "Joel Fernandes (Google)" , Frederic Weisbecker , Neeraj Upadhyay , Josh Triplett , Boqun Feng , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang Subject: [PATCH v2 3/5] tree/nocb: Adjust RCU_NOCB_WAKE_* macros from weaker to stronger Date: Sat, 29 Jul 2023 14:27:33 +0000 Message-ID: <20230729142738.222208-4-joel@joelfernandes.org> X-Mailer: git-send-email 2.41.0.487.g6d72f3e995-goog In-Reply-To: <20230729142738.222208-1-joel@joelfernandes.org> References: <20230729142738.222208-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org This is needed to make the next patch work correctly as we rely on the strength of the wakeup when comparing deferred-wakeup types across different CPUs. Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h index 192536916f9a..0f40a9c2b78d 100644 --- a/kernel/rcu/tree.h +++ b/kernel/rcu/tree.h @@ -288,8 +288,8 @@ struct rcu_data { /* Values for nocb_defer_wakeup field in struct rcu_data. */ #define RCU_NOCB_WAKE_NOT 0 -#define RCU_NOCB_WAKE_BYPASS 1 -#define RCU_NOCB_WAKE_LAZY 2 +#define RCU_NOCB_WAKE_LAZY 1 +#define RCU_NOCB_WAKE_BYPASS 2 #define RCU_NOCB_WAKE 3 #define RCU_NOCB_WAKE_FORCE 4 From patchwork Sat Jul 29 14:27:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 13333187 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 22A2DC001E0 for ; Sat, 29 Jul 2023 14:28:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231931AbjG2O2B (ORCPT ); Sat, 29 Jul 2023 10:28:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56482 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231899AbjG2O2A (ORCPT ); Sat, 29 Jul 2023 10:28:00 -0400 Received: from mail-io1-xd33.google.com (mail-io1-xd33.google.com [IPv6:2607:f8b0:4864:20::d33]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 722DC2686 for ; Sat, 29 Jul 2023 07:27:59 -0700 (PDT) Received: by mail-io1-xd33.google.com with SMTP id ca18e2360f4ac-7909307ecd1so16793439f.1 for ; Sat, 29 Jul 2023 07:27:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; t=1690640879; x=1691245679; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0DIiXFGh73Irk8k8jfSTlgn3pzuU9BPg/xYZpIge3SA=; b=HsW5SYF4O4C1xrNtb8sm4pRTfnrrViTLGL7v+Gn/Ah7UCbBgZ2opbMTyrHiGdhoaEn gcNOhQcOz0QL2+Q2P4xe34TxFbuIhXlf9EExkeLpiag6Pz3CQW2C2PScS3kCNl4lYs5+ 6y9zYLXR2axTa6ZjwfeIofwo3CIgGps6icYMg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690640879; x=1691245679; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0DIiXFGh73Irk8k8jfSTlgn3pzuU9BPg/xYZpIge3SA=; b=Q+wB6kyQ0BNNINmu3gyCG52LR2w29q/LoNP08R5kBGng6tZ69mADG6XEy3SJm3Ngbc U6pepjeUplkhlxKQ+h/3mol4crCrzMIesCwfl9wRRJhSEQRc7Dhd0AZReNrxL6hWMFUq 57HBe7WC4OsJ7/VRQyely8UcX9p3yLN6uQjwZbMmfFn9RtJXvYxxospA+1kSYIg5GIy3 4DD+Y6nZq0JsXV5RqACJ6tUN9dGoNjTmlGFDtxRBqwGK7RGFOE93gsDxR17jesqwx/AC U0WR8yR7wyiQcpO0GAM9zA2L8sNAmP6nYHzI6RkYVexxjPpEL0MjZEeehYqpxhym1pAg ILyw== X-Gm-Message-State: ABy/qLYgOXlgq81YpKUIUKCq92iAcIHtXn22ugpMeNxz9iXHKkeaNqBd sRzdgUau7roMnDOKBzig4u6QVg== X-Google-Smtp-Source: APBJJlGcgY471iiH1xZgoLcliih5C9aIWb1vc/SB0/AH9RPxwH+38J4y3R/0f9F30AOyyZrL/iE+Xg== X-Received: by 2002:a5e:a90c:0:b0:787:34d:f223 with SMTP id c12-20020a5ea90c000000b00787034df223mr2373322iod.11.1690640878695; Sat, 29 Jul 2023 07:27:58 -0700 (PDT) Received: from joelboxx5.c.googlers.com.com (254.82.172.34.bc.googleusercontent.com. [34.172.82.254]) by smtp.gmail.com with ESMTPSA id q9-20020a0566380ec900b0042b35c7b8c5sm1855362jas.61.2023.07.29.07.27.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 29 Jul 2023 07:27:58 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org Cc: rcu@vger.kernel.org, paulmck@kernel.org, "Joel Fernandes (Google)" , Frederic Weisbecker , Neeraj Upadhyay , Josh Triplett , Boqun Feng , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang Subject: [PATCH v2 4/5] tree/nocb: Improve readability of nocb_gp_wait() Date: Sat, 29 Jul 2023 14:27:34 +0000 Message-ID: <20230729142738.222208-5-joel@joelfernandes.org> X-Mailer: git-send-email 2.41.0.487.g6d72f3e995-goog In-Reply-To: <20230729142738.222208-1-joel@joelfernandes.org> References: <20230729142738.222208-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org The nocb_gp_wait() function contains logic to check each rdp's bypass list, flush if needed, and decide on wakeups. This makes the function hard to follow. Split out the bypass checking and flushing into a separate helper nocb_gp_flush_wake(). The new function encapsulates the logic to: - Check if the bypass needs to be flushed - Flush if needed - Return info on wakeups (lazy, bypass or none) nocb_gp_wait() now becomes simpler by calling the helper to handle the bypass flushing and waking logic. This splitting improves the readability and maintainability of the code by encapsulating related logic into a function with a clear purpose. Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree_nocb.h | 113 ++++++++++++++++++++++++----------------- 1 file changed, 66 insertions(+), 47 deletions(-) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 5598212d1f27..c805825c3f00 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -670,22 +670,68 @@ static void nocb_gp_sleep(struct rcu_data *my_rdp, int cpu) trace_rcu_nocb_wake(rcu_state.name, cpu, TPS("EndSleep")); } +/* + * Given an rdp, flush its bypass list if needed and return information about + * if a deferred-wakeup needs to be organized depending on whether things are + * still in the bypass list. Also tell caller if the list was flushed and if it + * is still empty after any flushing. + */ +static int nocb_gp_flush_wake(struct rcu_data *rdp, bool *empty, bool *flush) +{ + long bypass_ncbs; + long lazy_ncbs; + unsigned long j = jiffies; + + trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("Check")); + lockdep_assert_held(&rdp->nocb_lock); + bypass_ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass); + lazy_ncbs = READ_ONCE(rdp->lazy_len); + + *flush = false; + *empty = false; + if (bypass_ncbs && (lazy_ncbs == bypass_ncbs) && + (time_after(j, READ_ONCE(rdp->nocb_bypass_first) + jiffies_till_flush) || + bypass_ncbs > 2 * qhimark)) { + *flush = true; + } else if (bypass_ncbs && (lazy_ncbs != bypass_ncbs) && + (time_after(j, READ_ONCE(rdp->nocb_bypass_first) + 1) || + bypass_ncbs > 2 * qhimark)) { + *flush = true; + } else if (!bypass_ncbs && rcu_segcblist_empty(&rdp->cblist)) { + *empty = true; + return RCU_NOCB_WAKE_NOT; + } + + if (*flush) { + // Bypass full or old, so flush it. + (void)rcu_nocb_try_flush_bypass(rdp, j); + bypass_ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass); + lazy_ncbs = READ_ONCE(rdp->lazy_len); + } + + if (bypass_ncbs) { + trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, + bypass_ncbs == lazy_ncbs ? TPS("Lazy") : TPS("Bypass")); + return (bypass_ncbs == lazy_ncbs ? RCU_NOCB_WAKE_LAZY : + RCU_NOCB_WAKE_BYPASS); + } + return RCU_NOCB_WAKE_NOT; +} + /* * No-CBs GP kthreads come here to wait for additional callbacks to show up * or for grace periods to end. */ static void nocb_gp_wait(struct rcu_data *my_rdp) { - bool bypass = false; int __maybe_unused cpu = my_rdp->cpu; unsigned long cur_gp_seq; unsigned long flags; bool gotcbs = false; - unsigned long j = jiffies; - bool lazy = false; bool needwait_gp = false; // This prevents actual uninitialized use. bool needwake; bool needwake_gp; + int defer_wake_type = RCU_NOCB_WAKE_NOT; struct rcu_data *rdp, *rdp_toggling = NULL; struct rcu_node *rnp; unsigned long wait_gp_seq = 0; // Suppress "use uninitialized" warning. @@ -712,44 +758,24 @@ static void nocb_gp_wait(struct rcu_data *my_rdp) * won't be ignored for long. */ list_for_each_entry(rdp, &my_rdp->nocb_head_rdp, nocb_entry_rdp) { - long bypass_ncbs; - bool flush_bypass = false; - long lazy_ncbs; + int defer_wake_type_one = RCU_NOCB_WAKE_NOT; + bool flushed; + bool empty; - trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("Check")); rcu_nocb_lock_irqsave(rdp, flags); - lockdep_assert_held(&rdp->nocb_lock); - bypass_ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass); - lazy_ncbs = READ_ONCE(rdp->lazy_len); + defer_wake_type_one = nocb_gp_flush_wake(rdp, &empty, &flushed); - if (bypass_ncbs && (lazy_ncbs == bypass_ncbs) && - (time_after(j, READ_ONCE(rdp->nocb_bypass_first) + jiffies_till_flush) || - bypass_ncbs > 2 * qhimark)) { - flush_bypass = true; - } else if (bypass_ncbs && (lazy_ncbs != bypass_ncbs) && - (time_after(j, READ_ONCE(rdp->nocb_bypass_first) + 1) || - bypass_ncbs > 2 * qhimark)) { - flush_bypass = true; - } else if (!bypass_ncbs && rcu_segcblist_empty(&rdp->cblist)) { - rcu_nocb_unlock_irqrestore(rdp, flags); - continue; /* No callbacks here, try next. */ - } + // We may need to do a deferred wakeup later for bypass/lazy + // So note down what we learnt from the rdp. + defer_wake_type = max(defer_wake_type_one, defer_wake_type); - if (flush_bypass) { - // Bypass full or old, so flush it. - (void)rcu_nocb_try_flush_bypass(rdp, j); - bypass_ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass); - lazy_ncbs = READ_ONCE(rdp->lazy_len); + // Did we make any updates to main cblist? If not, no + // non-deferred wake up to do for this rdp. + if (!flushed && empty) { + rcu_nocb_unlock_irqrestore(rdp, flags); + continue; } - if (bypass_ncbs) { - trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, - bypass_ncbs == lazy_ncbs ? TPS("Lazy") : TPS("Bypass")); - if (bypass_ncbs == lazy_ncbs) - lazy = true; - else - bypass = true; - } rnp = rdp->mynode; // Advance callbacks if helpful and low contention. @@ -792,23 +818,16 @@ static void nocb_gp_wait(struct rcu_data *my_rdp) rcu_gp_kthread_wake(); } - my_rdp->nocb_gp_bypass = bypass; + my_rdp->nocb_gp_bypass = (defer_wake_type == RCU_NOCB_WAKE_BYPASS); my_rdp->nocb_gp_gp = needwait_gp; my_rdp->nocb_gp_seq = needwait_gp ? wait_gp_seq : 0; // At least one child with non-empty ->nocb_bypass, so set // timer in order to avoid stranding its callbacks. - if (!rcu_nocb_poll) { - // If bypass list only has lazy CBs. Add a deferred lazy wake up. - if (lazy && !bypass) { - wake_nocb_gp_defer(my_rdp, RCU_NOCB_WAKE_LAZY, - TPS("WakeLazyIsDeferred")); - // Otherwise add a deferred bypass wake up. - } else if (bypass) { - wake_nocb_gp_defer(my_rdp, RCU_NOCB_WAKE_BYPASS, - TPS("WakeBypassIsDeferred")); - } - } + if (!rcu_nocb_poll && defer_wake_type != RCU_NOCB_WAKE_NOT) + wake_nocb_gp_defer(my_rdp, defer_wake_type, + defer_wake_type == RCU_NOCB_WAKE_LAZY ? + TPS("WakeLazyIsDeferred") : TPS("WakeBypassIsDeferred")); if (rcu_nocb_poll) { /* Polling, so trace if first poll in the series. */ From patchwork Sat Jul 29 14:27:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 13333185 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 03929C001DC for ; Sat, 29 Jul 2023 14:28:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231962AbjG2O2F (ORCPT ); Sat, 29 Jul 2023 10:28:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56502 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231939AbjG2O2C (ORCPT ); Sat, 29 Jul 2023 10:28:02 -0400 Received: from mail-il1-x12c.google.com (mail-il1-x12c.google.com [IPv6:2607:f8b0:4864:20::12c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5E0A1E7A for ; Sat, 29 Jul 2023 07:28:00 -0700 (PDT) Received: by mail-il1-x12c.google.com with SMTP id e9e14a558f8ab-348f5193c12so9783725ab.2 for ; Sat, 29 Jul 2023 07:28:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; t=1690640879; x=1691245679; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ynoALsv1aca8y8xpjhuiunc/CTMUKjN2qeAE0MoxzWk=; b=w+eXq0b+xVCeRwLimsIFDmDvgRjcNTvCTtckAuYL+NxLFVlICU0f2/cWdwaY8mOG/Z q9zwNJwGGpEOV3xOiYldmz0Ht7ao1zWta1BbZEEjXcNOnH9EHP0dAD99T3uBoMbvXql5 a9TVjh54Y30EGW5rDrtC/vX3tjxDAXxWkwGk0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690640879; x=1691245679; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ynoALsv1aca8y8xpjhuiunc/CTMUKjN2qeAE0MoxzWk=; b=SUTM2Dlt52Tpc+vSXu7/1GCgMp9Laj1R+A5k1Cazr4EmGfs3fWaadu3IR0Uy3dbu9p 54f7NnYrxX7yyJo8r+TVYivLsjcl4xBHCbUIKCqmJI8Swb2xYwOw2Q2kM7g4nh0ZCtIp GFhLjNdJxJb/jGetFuOr44oC1HwAgRi2Df5uZSQTHdFpDXyN2VR1yGly4Ww91RaJb4kS rwWjfUna545D7lrG7g2h2nb5SdeoJjZKEDZ5n4/hS28LnNperjyUCLuJ6Nc/SZlaicIx Da1JkOPIYcuRvLKVpwUVWfdwOxJxB465kwGALzKfIZw52tx1JOq0nBN6CzUcw8slHzDZ iBfg== X-Gm-Message-State: ABy/qLY4u/TIuAuyoqiX0O/+nOAWp7H2cDjOkiLgZqX6UKv1aOxrvRH1 TYkaNXY5oJoVOf40bKdur++8dA== X-Google-Smtp-Source: APBJJlFNIZPWUROiB89przJra0uGXRoZjzPe5M07xGkhX3SWIVZXoL08l9IiN4nUOnSpcobgTvOV7A== X-Received: by 2002:a5e:d717:0:b0:790:9460:6304 with SMTP id v23-20020a5ed717000000b0079094606304mr657721iom.5.1690640879700; Sat, 29 Jul 2023 07:27:59 -0700 (PDT) Received: from joelboxx5.c.googlers.com.com (254.82.172.34.bc.googleusercontent.com. [34.172.82.254]) by smtp.gmail.com with ESMTPSA id q9-20020a0566380ec900b0042b35c7b8c5sm1855362jas.61.2023.07.29.07.27.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 29 Jul 2023 07:27:59 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org Cc: rcu@vger.kernel.org, paulmck@kernel.org, "Joel Fernandes (Google)" , Andy Whitcroft , Joe Perches , Dwaipayan Ray , Lukas Bulwahn Subject: [PATCH 5/5] Revert "checkpatch: Error out if deprecated RCU API used" Date: Sat, 29 Jul 2023 14:27:35 +0000 Message-ID: <20230729142738.222208-6-joel@joelfernandes.org> X-Mailer: git-send-email 2.41.0.487.g6d72f3e995-goog In-Reply-To: <20230729142738.222208-1-joel@joelfernandes.org> References: <20230729142738.222208-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org The check for single-argument kfree_rcu() is no longer needed as all users have been converted. This reverts commit 1eacac3255495be7502d406e2ba5444fb5c3607c. Signed-off-by: Joel Fernandes (Google) --- scripts/checkpatch.pl | 9 --------- 1 file changed, 9 deletions(-) diff --git a/scripts/checkpatch.pl b/scripts/checkpatch.pl index a9841148cde2..528f619520eb 100755 --- a/scripts/checkpatch.pl +++ b/scripts/checkpatch.pl @@ -6422,15 +6422,6 @@ sub process { } } -# check for soon-to-be-deprecated single-argument k[v]free_rcu() API - if ($line =~ /\bk[v]?free_rcu\s*\([^(]+\)/) { - if ($line =~ /\bk[v]?free_rcu\s*\([^,]+\)/) { - ERROR("DEPRECATED_API", - "Single-argument k[v]free_rcu() API is deprecated, please pass rcu_head object or call k[v]free_rcu_mightsleep()." . $herecurr); - } - } - - # check for unnecessary "Out of Memory" messages if ($line =~ /^\+.*\b$logFunctions\s*\(/ && $prevline =~ /^[ \+]\s*if\s*\(\s*(\!\s*|NULL\s*==\s*)?($Lval)(\s*==\s*NULL\s*)?\s*\)/ &&