From patchwork Thu Sep 15 00:14:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 12976710 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2D375C6FA82 for ; Thu, 15 Sep 2022 00:14:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229947AbiIOAOg (ORCPT ); Wed, 14 Sep 2022 20:14:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41172 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229924AbiIOAOg (ORCPT ); Wed, 14 Sep 2022 20:14:36 -0400 Received: from mail-qk1-x730.google.com (mail-qk1-x730.google.com [IPv6:2607:f8b0:4864:20::730]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 378C43883 for ; Wed, 14 Sep 2022 17:14:34 -0700 (PDT) Received: by mail-qk1-x730.google.com with SMTP id f13so7905183qkk.6 for ; Wed, 14 Sep 2022 17:14:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=A1vsCw7qsSbg6LG6s3lCEoHneJ7sDD1rTCb2gsMBhiI=; b=PVl0MtbuC8jP0UnDsRzD+DryiQcwfi3cdRxo1mYiLGB8fjmQS20KFseigIWbxllokR COcSb6OcEDsD5gucsUYep4hzLW6CW/pF7H6hf7a7LE9R+G6dJUVv+ekd/+7NxLMJCCVL 4QUu2IzMbm6D2kPkwWMzqte+24i+TJwQtpCTU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=A1vsCw7qsSbg6LG6s3lCEoHneJ7sDD1rTCb2gsMBhiI=; b=mVkbHbT+/01NGq2L4EW1XDjVF4GrF9Tl/YbIoZDuiq+IK2nZ7voJA+hcYCZgt5vFa3 LvwV/Q1rMlIONdWoa6xR8gvTD8HQ6rJPVlS7WV1+cyUP+p3CHg8FMrEv1Cmh/PqhIx9c I0QeQxZeqVlZEag7hLsks8iS24ls8omz04O0WmOwwwNRfIqfs1L4tNj2a/UZ4vSHYTEz 5BHnsn5MOphhrfuDxJY/v45AcTjwcnblxpH1GV4y8wKyj2OiZEiwRnkn2Ul/PFZXprgA 2TkfkU4g3OL695qBhT7vJ6Wltjnnc5KyVMF2jsTaor3fkYFkv6k6hAtZOGl5zDDpl0z3 SXYg== X-Gm-Message-State: ACgBeo17C/NLQwBDRlOEVTSN9eIdaczycAiajiuoGJAm8wQ1k9WIOj/N z5KGGl6hoTFZaYWcZAPm4IiFnRf/GDRYVQMXAyQ= X-Google-Smtp-Source: AA6agR7skx+qmUrnXxbC9eolRQX/E9WCYME4MPsQJ6mEMoqK9nsQdr33ypJE00CInZ5jxffolQVMlw== X-Received: by 2002:a05:620a:454e:b0:6ce:3c11:24ad with SMTP id u14-20020a05620a454e00b006ce3c1124admr13034044qkp.606.1663200873386; Wed, 14 Sep 2022 17:14:33 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id do11-20020a05620a2b0b00b006bbb07ebd83sm2872422qkb.108.2022.09.14.17.14.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Sep 2022 17:14:32 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org Cc: "Joel Fernandes (Google)" , Frederic Weisbecker , Josh Triplett , Lai Jiangshan , Mathieu Desnoyers , Neeraj Upadhyay , "Paul E. McKenney" , rcu@vger.kernel.org, Steven Rostedt Subject: [PATCH rcu/next 1/3] rcu/tree: Use READ_ONCE() for lockless read of rnp->qsmask Date: Thu, 15 Sep 2022 00:14:17 +0000 Message-Id: <20220915001419.55617-2-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog In-Reply-To: <20220915001419.55617-1-joel@joelfernandes.org> References: <20220915001419.55617-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org The rnp->qsmask is locklessly accessed from rcutree_dying_cpu(). This may help avoid load tearing due to concurrent access, KCSAN issues, and preserve sanity of people reading the mask in tracing. Signed-off-by: Joel Fernandes (Google) Reviewed-by: Frederic Weisbecker --- kernel/rcu/tree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 0ca21ac0f064..5ec97e3f7468 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2106,7 +2106,7 @@ int rcutree_dying_cpu(unsigned int cpu) if (!IS_ENABLED(CONFIG_HOTPLUG_CPU)) return 0; - blkd = !!(rnp->qsmask & rdp->grpmask); + blkd = !!(READ_ONCE(rnp->qsmask) & rdp->grpmask); trace_rcu_grace_period(rcu_state.name, READ_ONCE(rnp->gp_seq), blkd ? TPS("cpuofl-bgp") : TPS("cpuofl")); return 0; From patchwork Thu Sep 15 00:14:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 12976711 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3E64C6FA82 for ; Thu, 15 Sep 2022 00:14:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229960AbiIOAOk (ORCPT ); Wed, 14 Sep 2022 20:14:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41174 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229962AbiIOAOi (ORCPT ); Wed, 14 Sep 2022 20:14:38 -0400 Received: from mail-qk1-x734.google.com (mail-qk1-x734.google.com [IPv6:2607:f8b0:4864:20::734]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4EACF64E0 for ; Wed, 14 Sep 2022 17:14:35 -0700 (PDT) Received: by mail-qk1-x734.google.com with SMTP id d17so11372216qko.13 for ; Wed, 14 Sep 2022 17:14:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=7NTlW4AVEaEifdUDNJKG5n5TSALhIocTrw3kCCpTxNA=; b=aufasvy68P6Ewg+Se2UzkhM+dsbLyQxC4Y8rMgj0/dn7wakhHYWGMOKsnUKG3S4ABp 6EucXs9y5wMo4i5DJQlt8fqTu/P8aaNCEAyBL7IzBTgXEuHN/qUGlR8gR2BWlMi8Mm7/ 9WKL7uRA0DCxmbHJoZcawp2vF9OiRs32DNSDU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=7NTlW4AVEaEifdUDNJKG5n5TSALhIocTrw3kCCpTxNA=; b=6PyqAKBF75WnlYAbKplvWx7Y5dDLMJzuAlHlCem/ALcuKV5XH4ls+qQJZf2dM4JuZQ cS9OlQTQ9gBVo7XAWHkVp/VVSD6elINpjtqstsRxFwua+hAwrX6krE0aBsHO2xYdx4AP QxHD0cDZSlNaX7cvzbuDwv682TObOlp+L4n4M4pI/ufdFyW94f2cwUoYurQRtrmoTl9H i0VSIl7YpFwp/ACSkjwfphf+IJer3+sibGZeW25q65EseYUB16E88LsqRAPfPGoqBDz6 WeCdqkLEW0ioRtLjZOOJEVHLwLCgTfxiEqULC84+jd1DPJJl5uSr0r/6j7veDEP2EDKY L5qA== X-Gm-Message-State: ACgBeo0w/HEKp3nyWdl21/c73j5BWGsyQ9FDs75QsHbk1Bjt6SO8WQrn pybnKiAWxVDLmH2FTnYpXVvpFQ== X-Google-Smtp-Source: AA6agR4dx+hoiCK3T5tFVcbkeQk/wU2na31F054HHTCbDwmPVtIe+kG4dyVZZowUw7+G+FYkfxNT4A== X-Received: by 2002:a05:620a:2491:b0:6bb:8982:8f66 with SMTP id i17-20020a05620a249100b006bb89828f66mr28190732qkn.665.1663200874404; Wed, 14 Sep 2022 17:14:34 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id do11-20020a05620a2b0b00b006bbb07ebd83sm2872422qkb.108.2022.09.14.17.14.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Sep 2022 17:14:33 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org Cc: "Joel Fernandes (Google)" , Frederic Weisbecker , Josh Triplett , Lai Jiangshan , Mathieu Desnoyers , Neeraj Upadhyay , "Paul E. McKenney" , rcu@vger.kernel.org, Steven Rostedt Subject: [PATCH rcu/next 2/3] rcu: Move trace_rcu_callback() before bypassing Date: Thu, 15 Sep 2022 00:14:18 +0000 Message-Id: <20220915001419.55617-3-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog In-Reply-To: <20220915001419.55617-1-joel@joelfernandes.org> References: <20220915001419.55617-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org If any CB is queued into the bypass list, then trace_rcu_callback() does not show it. This makes it not clear when a callback was actually queued, as you only end up getting a trace_rcu_invoke_callback() trace. Fix it by moving trace_rcu_callback() before trace_rcu_nocb_try_bypass(). Signed-off-by: Joel Fernandes (Google) Signed-off-by: Joel Fernandes (Google) Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 5ec97e3f7468..9fe581be8696 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2809,10 +2809,7 @@ void call_rcu(struct rcu_head *head, rcu_callback_t func) } check_cb_ovld(rdp); - if (rcu_nocb_try_bypass(rdp, head, &was_alldone, flags)) - return; // Enqueued onto ->nocb_bypass, so just leave. - // If no-CBs CPU gets here, rcu_nocb_try_bypass() acquired ->nocb_lock. - rcu_segcblist_enqueue(&rdp->cblist, head); + if (__is_kvfree_rcu_offset((unsigned long)func)) trace_rcu_kvfree_callback(rcu_state.name, head, (unsigned long)func, @@ -2821,6 +2818,11 @@ void call_rcu(struct rcu_head *head, rcu_callback_t func) trace_rcu_callback(rcu_state.name, head, rcu_segcblist_n_cbs(&rdp->cblist)); + if (rcu_nocb_try_bypass(rdp, head, &was_alldone, flags)) + return; // Enqueued onto ->nocb_bypass, so just leave. + // If no-CBs CPU gets here, rcu_nocb_try_bypass() acquired ->nocb_lock. + rcu_segcblist_enqueue(&rdp->cblist, head); + trace_rcu_segcb_stats(&rdp->cblist, TPS("SegCBQueued")); /* Go handle any RCU core processing required. */ From patchwork Thu Sep 15 00:14:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 12976712 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 618CFECAAD3 for ; Thu, 15 Sep 2022 00:14:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229936AbiIOAOl (ORCPT ); Wed, 14 Sep 2022 20:14:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41320 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229992AbiIOAOi (ORCPT ); Wed, 14 Sep 2022 20:14:38 -0400 Received: from mail-qk1-x72d.google.com (mail-qk1-x72d.google.com [IPv6:2607:f8b0:4864:20::72d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B61DB3883 for ; Wed, 14 Sep 2022 17:14:36 -0700 (PDT) Received: by mail-qk1-x72d.google.com with SMTP id u28so9202072qku.2 for ; Wed, 14 Sep 2022 17:14:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=SqiD/rmTmUnZN+sxp6myZ+MDQ3Vwh3eOQpZ6rdR1rxE=; b=ZLcx7NHB52/qzkYWSR8ckkf1gareZ4GmzTiq/hCOgGnRBEHkz0NxGweHu1K3EVjRIg G3fMULXiX7AFK9DZkvhNqmCZECLF0CkEWQAp6MDA9wyeA6dFgX5uV1lxs9PA0qPeJ9Kz BW7OgVuE0HCV8Sui9C6J77DMSvj8TO6/fRzB8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=SqiD/rmTmUnZN+sxp6myZ+MDQ3Vwh3eOQpZ6rdR1rxE=; b=2ID/Du5w/0OI1GQc3gKdPj5F6OVMxOBOAJTCqd35YXZxoga7hx456dNRRHpUgW50ef 1Abynk1+V0DsFVdtf/JSETxaHsElmZzfiVI2EiBqY5BeTgY04IjRpnfddvOD41eUbMWL kRB66Z0dmcAgNZWw+u9Qz/5n5CT1E5g08UYjZMUCTNa97MjdK9DWqMmOjcljjY/yT8xW sU5bGCk68ddzUp0k76C5mHxWtTomSg1ZP+v/VbedZY3OpwjqnMHspIP+oHZZQ4XnPp0b 3U9dZvUsn2I2mjUGKSFr0gtT8UgZPmwdyoTheuTG89D+xbNpYz+n2XIblzwWnSGTd5tu oHJQ== X-Gm-Message-State: ACgBeo192OazuVegyybFnFOUWSPVV1XDwd4SSBBY5GgNcQDE7sc4W9ps R7m36L0xFyti4UBEoa4FlnmRtw== X-Google-Smtp-Source: AA6agR6Jaelol8J/6+LJ4V0dMBTzfQIqYm7WXyULloG4qJkDuOxHTYKXfQ9I2xuKCi40lhnAtDyzpw== X-Received: by 2002:a05:620a:318a:b0:6ce:732a:f92 with SMTP id bi10-20020a05620a318a00b006ce732a0f92mr6969342qkb.347.1663200875110; Wed, 14 Sep 2022 17:14:35 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id do11-20020a05620a2b0b00b006bbb07ebd83sm2872422qkb.108.2022.09.14.17.14.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Sep 2022 17:14:34 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org Cc: "Joel Fernandes (Google)" , Frederic Weisbecker , Josh Triplett , Lai Jiangshan , Mathieu Desnoyers , Neeraj Upadhyay , "Paul E. McKenney" , rcu@vger.kernel.org, Steven Rostedt Subject: [PATCH rcu/next 3/3] rcu: Fix late wakeup when flush of bypass cblist happens Date: Thu, 15 Sep 2022 00:14:19 +0000 Message-Id: <20220915001419.55617-4-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog In-Reply-To: <20220915001419.55617-1-joel@joelfernandes.org> References: <20220915001419.55617-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org When the bypass cblist gets too big or its timeout has occurred, it is flushed into the main cblist. However, the bypass timer is still running and the behavior is that it would eventually expire and wake the GP thread. Since we are going to use the bypass cblist for lazy CBs, do the wakeup soon as the flush happens. Otherwise, the lazy-timer will go off much later and the now-non-lazy cblist CBs can get stranded for the duration of the timer. This is a good thing to do anyway, since it makes the behavior consistent with behavior of other code paths where queueing something into the ->cblist makes the GP kthread in a non-sleeping state quickly. Signed-off-by: Joel Fernandes (Google) Reviewed-by: Frederic Weisbecker --- kernel/rcu/tree_nocb.h | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 0a5f0ef41484..04c87f250e01 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -433,8 +433,9 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, if ((ncbs && j != READ_ONCE(rdp->nocb_bypass_first)) || ncbs >= qhimark) { rcu_nocb_lock(rdp); + *was_alldone = !rcu_segcblist_pend_cbs(&rdp->cblist); + if (!rcu_nocb_flush_bypass(rdp, rhp, j)) { - *was_alldone = !rcu_segcblist_pend_cbs(&rdp->cblist); if (*was_alldone) trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("FirstQ")); @@ -447,7 +448,12 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, rcu_advance_cbs_nowake(rdp->mynode, rdp); rdp->nocb_gp_adv_time = j; } - rcu_nocb_unlock_irqrestore(rdp, flags); + + // The flush succeeded and we moved CBs into the regular list. + // Don't wait for the wake up timer as it may be too far ahead. + // Wake up the GP thread now instead, if the cblist was empty. + __call_rcu_nocb_wake(rdp, *was_alldone, flags); + return true; // Callback already enqueued. }