From patchwork Sat Sep 17 16:41:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 12979200 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 766C6C6FA82 for ; Sat, 17 Sep 2022 16:42:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229527AbiIQQmh (ORCPT ); Sat, 17 Sep 2022 12:42:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42310 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229703AbiIQQmg (ORCPT ); Sat, 17 Sep 2022 12:42:36 -0400 Received: from mail-qv1-xf34.google.com (mail-qv1-xf34.google.com [IPv6:2607:f8b0:4864:20::f34]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9DC682F393 for ; Sat, 17 Sep 2022 09:42:33 -0700 (PDT) Received: by mail-qv1-xf34.google.com with SMTP id m9so18934807qvv.7 for ; Sat, 17 Sep 2022 09:42:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=YtXTLA3M+nWqodgxztJaMgLvSRyU03TOMEk16miDVSg=; b=ShOLZGx75Ii8OwDydiUuECJDaVrzXu7dm1LvXWu1uMAHXy8OWxy3+sY1KgJKUVy9sc lJGUEgu7gVKBhWs0Wz0ZhBtUO3JLjqMoq0kpgajDLoDILUKTL2HipbaT+ktS1fFKI787 zPzc24o/oNUffNEGEUH9lEkxMlvjJIA+zFR6g= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=YtXTLA3M+nWqodgxztJaMgLvSRyU03TOMEk16miDVSg=; b=iahv5tWXuFvKFYy+HAE3DNFdDykQ4GVbLGlQG0fCfUuFVyhH65IPipl0YCuZItZFsm 9vYdfJWmE06JCkISiAOVw6y+BrrehATBSTzwIb7OJgzZnw/z4h3jLe4HRVNnIAHXPliR VHS+8jU0X52P5LR0Q4ImPH25cSOnaPl+o1MQtQ9TO/+ai1HDUvR8MYfCpBZJUa1Slvpu 7+oU17uWHaqh8WKl6KbpjQFxwyK8OLkMPyz4K1TwV1iubOPtLxk079laAN5yquRHVrX3 IigkK11AD1dsMUceyjxvyg+G9RhkMGrc9s3t7gmiTmLRuaGszKjzfkZwLFj17w20HIfP 8QiQ== X-Gm-Message-State: ACrzQf1YvunS7vWVXw3h4bQIxsmM4nrm0J7hcvgDPVccC+zLpzn8kWPR 8MSEx2cXvUdUGOxjs2dRz1fPJ2CwVlLQbA== X-Google-Smtp-Source: AMsMyM4IYHlTU6MQt7EpgMPbONgN5h3lvYXkOHlxQ4yG2OYh9ORFH9JAXc2T1IMbliGqy+f9+mm10Q== X-Received: by 2002:a05:6214:1d21:b0:4ad:1361:befa with SMTP id f1-20020a0562141d2100b004ad1361befamr6419573qvd.111.1663432952613; Sat, 17 Sep 2022 09:42:32 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id q31-20020a05620a2a5f00b006bb0e5ca4bbsm9479239qkp.85.2022.09.17.09.42.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 17 Sep 2022 09:42:32 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, "Joel Fernandes (Google)" Subject: [PATCH rcu/next 1/3] rcu/tree: Use READ_ONCE() for lockless read of rnp->qsmask Date: Sat, 17 Sep 2022 16:41:58 +0000 Message-Id: <20220917164200.511783-2-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.3.968.ga6b4b080e4-goog In-Reply-To: <20220917164200.511783-1-joel@joelfernandes.org> References: <20220917164200.511783-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org The rnp->qsmask is locklessly accessed from rcutree_dying_cpu(). This may help avoid load tearing due to concurrent access, KCSAN issues, and preserve sanity of people reading the mask in tracing. Reviewed-by: Frederic Weisbecker Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 0ca21ac0f064..5ec97e3f7468 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2106,7 +2106,7 @@ int rcutree_dying_cpu(unsigned int cpu) if (!IS_ENABLED(CONFIG_HOTPLUG_CPU)) return 0; - blkd = !!(rnp->qsmask & rdp->grpmask); + blkd = !!(READ_ONCE(rnp->qsmask) & rdp->grpmask); trace_rcu_grace_period(rcu_state.name, READ_ONCE(rnp->gp_seq), blkd ? TPS("cpuofl-bgp") : TPS("cpuofl")); return 0; From patchwork Sat Sep 17 16:41:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 12979202 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 34B03ECAAD3 for ; Sat, 17 Sep 2022 16:42:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229731AbiIQQmi (ORCPT ); Sat, 17 Sep 2022 12:42:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42318 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229709AbiIQQmg (ORCPT ); Sat, 17 Sep 2022 12:42:36 -0400 Received: from mail-qv1-xf2a.google.com (mail-qv1-xf2a.google.com [IPv6:2607:f8b0:4864:20::f2a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 381A8303E1 for ; Sat, 17 Sep 2022 09:42:34 -0700 (PDT) Received: by mail-qv1-xf2a.google.com with SMTP id i15so18942032qvp.5 for ; Sat, 17 Sep 2022 09:42:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=W20sVr4rHsYJL/FKcEAa6dXGu1RHDwQdwaDFAlx3ULo=; b=uHIj9ohx6d/ikjQUH9MJsZfu/j/LooTs088y2mOsZ7N/UW6p5VqrtfTdkxJu4dTLO9 dI+OHDQizo/UIdP8y7bRLnQMGJcr0bulMfrIjDxsnsn1rINWrRQ7Qc7W+R6iba6C9DUL /mZGnML0/7+6LWpSZGZC5FgZlRWwDkLl3avUQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=W20sVr4rHsYJL/FKcEAa6dXGu1RHDwQdwaDFAlx3ULo=; b=GM2A/9PyFS53vPdQmzKMP9Sez3wD6oKoA5c5LxLlaVdCYf/Kb2F4awAe2RzNyhUBw/ oiMjgjQd/s2s10xm97Fbeu2/THFfbL9fi1/Y4fR81neH+xGlJ4gRIgPPlcAetwUmu8Dt m6px83nNKzmFojl8jn0seWSoMTiss0/RDf3D6WcY2Ydl18R62xeF44RZlTWd4EI0aahR omIY7A1UJWQfbXayMWwGu9VLeERwpkiXQ9wIYHvUnODAXcFw+eT9TlPSL1nEPTl6S8GY G8w/utEHHLyJ0prPKjgx8K2tlQrFPOGFy2/NDJ9ao2gtlaLM8QPNs9Fw0SgQoWpOaVGP HTRw== X-Gm-Message-State: ACrzQf0+eosQ0AaOzYdt8c5wo8RBxu1OYksWtz4aKDSTe5S7x/FJI4N5 ELnLVoouiHhl+QZp/coRlBZaKD9/9J5Q4w== X-Google-Smtp-Source: AMsMyM6CVm//iVr2UMSIXomdq/E4kcyBPnoaIHoLb5h5yloPlLl+1C4yIDBz1aMUo9ZUdvIXPLlYig== X-Received: by 2002:a0c:f084:0:b0:49e:7bc1:26a3 with SMTP id g4-20020a0cf084000000b0049e7bc126a3mr8842257qvk.107.1663432953226; Sat, 17 Sep 2022 09:42:33 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id q31-20020a05620a2a5f00b006bb0e5ca4bbsm9479239qkp.85.2022.09.17.09.42.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 17 Sep 2022 09:42:32 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, "Joel Fernandes (Google)" Subject: [PATCH rcu/next 2/3] rcu: Fix late wakeup when flush of bypass cblist happens (v6) Date: Sat, 17 Sep 2022 16:41:59 +0000 Message-Id: <20220917164200.511783-3-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.3.968.ga6b4b080e4-goog In-Reply-To: <20220917164200.511783-1-joel@joelfernandes.org> References: <20220917164200.511783-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org When the bypass cblist gets too big or its timeout has occurred, it is flushed into the main cblist. However, the bypass timer is still running and the behavior is that it would eventually expire and wake the GP thread. Since we are going to use the bypass cblist for lazy CBs, do the wakeup soon as the flush for "too big or too long" bypass list happens. Otherwise, long delays can happen for callbacks which get promoted from lazy to non-lazy. This is a good thing to do anyway (regardless of future lazy patches), since it makes the behavior consistent with behavior of other code paths where flushing into the ->cblist makes the GP kthread into a non-sleeping state quickly. [ Frederic Weisbec: changes to not do wake up GP thread unless needed, comment changes ]. Reviewed-by: Frederic Weisbecker Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree_nocb.h | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 0a5f0ef41484..04c87f250e01 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -433,8 +433,9 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, if ((ncbs && j != READ_ONCE(rdp->nocb_bypass_first)) || ncbs >= qhimark) { rcu_nocb_lock(rdp); + *was_alldone = !rcu_segcblist_pend_cbs(&rdp->cblist); + if (!rcu_nocb_flush_bypass(rdp, rhp, j)) { - *was_alldone = !rcu_segcblist_pend_cbs(&rdp->cblist); if (*was_alldone) trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("FirstQ")); @@ -447,7 +448,12 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, rcu_advance_cbs_nowake(rdp->mynode, rdp); rdp->nocb_gp_adv_time = j; } - rcu_nocb_unlock_irqrestore(rdp, flags); + + // The flush succeeded and we moved CBs into the regular list. + // Don't wait for the wake up timer as it may be too far ahead. + // Wake up the GP thread now instead, if the cblist was empty. + __call_rcu_nocb_wake(rdp, *was_alldone, flags); + return true; // Callback already enqueued. } From patchwork Sat Sep 17 16:42:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 12979201 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C18BFC6FA86 for ; Sat, 17 Sep 2022 16:42:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229725AbiIQQmj (ORCPT ); Sat, 17 Sep 2022 12:42:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42322 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229718AbiIQQmh (ORCPT ); Sat, 17 Sep 2022 12:42:37 -0400 Received: from mail-qk1-x736.google.com (mail-qk1-x736.google.com [IPv6:2607:f8b0:4864:20::736]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F057130555 for ; Sat, 17 Sep 2022 09:42:34 -0700 (PDT) Received: by mail-qk1-x736.google.com with SMTP id i3so13191289qkl.3 for ; Sat, 17 Sep 2022 09:42:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=LbqcEwrWK8xzGA015vucIgdiMEo2clH8LFFcAywvHQQ=; b=WyMg2FBbm/t2DCQzBs99+MvCJHW0x8rLhbPX/j2RGxjVYxVYRycoEUDAFecP1EX/fT PBhwMWhxq/aF+d2r1RTwRn4QXPhdGt495Yrkfd2kaQ6mz625FYHGPGTA4+tmP9W3cKNf 0qnLlsyWq/TNJi0sw1SE2aJ99lvvIKJkHci5M= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=LbqcEwrWK8xzGA015vucIgdiMEo2clH8LFFcAywvHQQ=; b=cqePdP9iYoXAUjgFRgY+CFtdTvbY0a1svR8Y4Z91BupmIkSZH6HUPZscTNqyPCwSJ1 OAPOSM20KGSO/I/ab1bCEzSLIydMfle10Z8CxNhNjxkWHj0xrOltU+KkhPR3q/3dWU86 864QRYAMsqR4yJJjBmfNHOCHpCf4zAi3xkD0kjv8p2GZOWwEAy8cDmkYyCNM54PGYsTt 9olKGijb2at9NwqwYZmsI9z98+s5xztlA1lnwwPc0nyxYTBmbz8S3xEPb/ULdD9NFMQF Q8jrlC8yKVH+r90ASl5Ox4zfUtJIuNxueOoR3ay26JiPpjAiaRE6nmynsfA+EVzznfmu dVhA== X-Gm-Message-State: ACrzQf1S0LODw+F3s2jb0UOs+0TmmzJVcVLrAtZNxgPyv2ktN516aLb/ 0QsLHIiGl9aPEaaA083CFk2oLg0LwnNjAQ== X-Google-Smtp-Source: AMsMyM7joNQtb0/LD1ROJWXNvXDA7dYnaBUzWEFaVKlNWDRFSERJUYlikz42Yc+wRk0MtQOPEq6Hwg== X-Received: by 2002:a05:620a:2681:b0:6b5:b60c:1e66 with SMTP id c1-20020a05620a268100b006b5b60c1e66mr7841048qkp.99.1663432953813; Sat, 17 Sep 2022 09:42:33 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id q31-20020a05620a2a5f00b006bb0e5ca4bbsm9479239qkp.85.2022.09.17.09.42.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 17 Sep 2022 09:42:33 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, "Joel Fernandes (Google)" Subject: [PATCH rcu/next 3/3] rcu: Call trace_rcu_callback() also for bypass queuing (v2) Date: Sat, 17 Sep 2022 16:42:00 +0000 Message-Id: <20220917164200.511783-4-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.3.968.ga6b4b080e4-goog In-Reply-To: <20220917164200.511783-1-joel@joelfernandes.org> References: <20220917164200.511783-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org If any callback is queued into the bypass list, then trace_rcu_callback() does not show it. This makes it unclear when a callback was actually queued, as the resulting trace only includes a rcu_invoke_callback event. Fix it by calling the tracing function even if queuing into bypass. This is needed for the future rcutop tool which monitors enqueuing of callbacks. Note that, in case of bypass queuing, the new tracing happens without the nocb_lock. This should be OK since on CONFIG_RCU_NOCB_CPU systems, the total callbacks is represented by an atomic counter. Also, other paths like rcu_barrier() also sample the total number of callback without the nocb_lock. Also, while at it, optimize the tracing so that rcu_state is not accessed if tracing is disabled, because that's useless if we are not tracing. A quick inspection of the generated assembler shows that rcu_state is accessed even if the jump label for the tracepoint is disabled. Here is gcc -S output of the bad asm (note that I un-inlined it just for testing and illustration however the final __trace_rcu_callback in the patch is marked static inline): __trace_rcu_callback: movq 8(%rdi), %rcx movq rcu_state+3640(%rip), %rax movq %rdi, %rdx cmpq $4095, %rcx ja .L3100 movq 192(%rsi), %r8 1:jmp .L3101 # objtool NOPs this .pushsection __jump_table, "aw" .balign 8 .long 1b - . .long .L3101 - . .quad __tracepoint_rcu_kvfree_callback+8 + 2 - . .popsection With this change, the jump label check which is NOOPed is moved to the beginning of the function. Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree.c | 30 ++++++++++++++++++++++-------- 1 file changed, 22 insertions(+), 8 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 5ec97e3f7468..18f07e167d5e 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2728,6 +2728,22 @@ static void check_cb_ovld(struct rcu_data *rdp) raw_spin_unlock_rcu_node(rnp); } +/* + * Trace RCU callback helper, call after enqueuing callback. + */ +static inline void __trace_rcu_callback(struct rcu_head *head, + struct rcu_data *rdp) +{ + if (trace_rcu_kvfree_callback_enabled() && + __is_kvfree_rcu_offset((unsigned long)head->func)) + trace_rcu_kvfree_callback(rcu_state.name, head, + (unsigned long)head->func, + rcu_segcblist_n_cbs(&rdp->cblist)); + else if (trace_rcu_callback_enabled()) + trace_rcu_callback(rcu_state.name, head, + rcu_segcblist_n_cbs(&rdp->cblist)); +} + /** * call_rcu() - Queue an RCU callback for invocation after a grace period. * @head: structure to be used for queueing the RCU updates. @@ -2809,17 +2825,15 @@ void call_rcu(struct rcu_head *head, rcu_callback_t func) } check_cb_ovld(rdp); - if (rcu_nocb_try_bypass(rdp, head, &was_alldone, flags)) + + if (rcu_nocb_try_bypass(rdp, head, &was_alldone, flags)) { + __trace_rcu_callback(head, rdp); return; // Enqueued onto ->nocb_bypass, so just leave. + } + // If no-CBs CPU gets here, rcu_nocb_try_bypass() acquired ->nocb_lock. rcu_segcblist_enqueue(&rdp->cblist, head); - if (__is_kvfree_rcu_offset((unsigned long)func)) - trace_rcu_kvfree_callback(rcu_state.name, head, - (unsigned long)func, - rcu_segcblist_n_cbs(&rdp->cblist)); - else - trace_rcu_callback(rcu_state.name, head, - rcu_segcblist_n_cbs(&rdp->cblist)); + __trace_rcu_callback(head, rdp); trace_rcu_segcb_stats(&rdp->cblist, TPS("SegCBQueued"));