From patchwork Thu Feb 1 01:40:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boqun Feng X-Patchwork-Id: 13540567 Received: from mail-qv1-f52.google.com (mail-qv1-f52.google.com [209.85.219.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D06AA443E; Thu, 1 Feb 2024 01:42:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706751734; cv=none; b=q/Vgdb62OOhhRPXen2cwqyMSwA33hNHO6dDT2ttt/izVBfMMJTHonFkx7rSTbFh5YqM7Dk35goRjryN98M6VBBJ36LhqHbyI/hs9E+rNtOm0IhMmucpqbd3+Xyw/xl0a1pMB5kRTXEf11CFXNWaG6wbmVFB2eS6q5VKI0xyJ0eE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706751734; c=relaxed/simple; bh=JYMQcq8E1zGYMJ0i7nVikl7w2oNjNCpEj0nSc8NMMoM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=dm0FWI6r0KGjYGa1deE/wSgOfEWEMUM+po3Pg5tyE9CmVAf2ITqPeV4xPRMjTNDMv1DnX8tFsNkbOYqBumhwUChgNvna1v+wTkyNGFu5YSgMS//m9grFiQfGAtY0D37mOMJqwx4AlBSA0CEJc7UdIOZKwT/hmLrHeaZrwHJgBGI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=VoUBmB5/; arc=none smtp.client-ip=209.85.219.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="VoUBmB5/" Received: by mail-qv1-f52.google.com with SMTP id 6a1803df08f44-6818a9fe380so2470446d6.2; Wed, 31 Jan 2024 17:42:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1706751731; x=1707356531; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=zLJWc1KOEuIrLOS4qhIHcNcluYpoORRWU7wyfjkuepk=; b=VoUBmB5/NTIxGqwdwplE+GmmEsEDc2b4PLVocFyysIXCoTcVyuidpPnpUmJLuDS8rY KkEZpeM47JbigKSk1Z95LmgV/8xsbSI0GlfV9DGD2Gc66+LQwlo/8au/T6KepqOO56Lv vzlPzNo66u2vVMdCBXoQAqMwGuD1jBIlheBzchEjOJ9M2eOW7IZ9YotCAVrN0KpHrHmD koryuz7qn+g/OhMG/WOZfBi8XLmCiDkgWQhtIsnBT1PTwItossmRtbECoPNWZMHvsVFM ovT9ccqZvECnw73ughgvFQz47memrI0uUgaSroUKcCxMwcfoIURILB55/M/7MxllgnTg K6eQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706751731; x=1707356531; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=zLJWc1KOEuIrLOS4qhIHcNcluYpoORRWU7wyfjkuepk=; b=awoPAK5pSLUx81swMxemKimGPD3rWEo5xEf4n6vkCPc9eqm4lCf06a2nOAaH3FJ9XU eRNOy12SlcbOhimMD7DP11MeDNRE+owhupQAPeijg5gbMDvp37MBFU46ujIjRbLce2u9 ugXq7Nmq7FSl06GTmjIkkhOWQT+oGngmMTDUhSYu8v8CE4vlfgE8dPurBY159aKapRmF nyKmOstW2PYY6hSwDUp9wOGTQvdVHJZzsXSPuORy1gmaHt6XMTEuAhOHyCQ340SFMfGB wdCpr5rljxU+VMNsAqOrXD/3s7JxCGepMAxHJtEANFciWCBEDp/uVos2nxvcKyhue/or ZlTg== X-Gm-Message-State: AOJu0YwKS77FCzXQTFvI2aYDFnU+URRZ/esyLkAVEMV/eQKHdRfCwpp2 AEBfK2P3Dsj8l06vPgHyfB5U44B+bfaoc7FoJFDvwQF6vBfXciBr X-Google-Smtp-Source: AGHT+IFfdivcP9CfPPG+370KcqaFjtRQ/uxNHw9YiEyYwjl2DybCtI9N6qTerOdOTave++dQWIQJwg== X-Received: by 2002:a0c:9a91:0:b0:68c:7a2f:1c0b with SMTP id y17-20020a0c9a91000000b0068c7a2f1c0bmr461906qvd.62.1706751731594; Wed, 31 Jan 2024 17:42:11 -0800 (PST) X-Forwarded-Encrypted: i=0; AJvYcCUptm6AUOJltqfcS2AtCtlsJsb1yHlur08z+RnNtjG+RLixsXaPqKBfcIAfMkLUGEQRDMiOE+x5UtJKVII8pPqVKwzgwpxE4HpTJVl9f65F3Kvh+5puy/8pobwzkXOpTU+mE2IEu2YDSmhK2TSr4l3KcFwZ90Pj6nF/a50lL5X53tYcx3hCGl07UGR4dkvb22ZarKUiQaxOp/0civ0rzoFHo2hC+BCeL2K0w4aRuZhGkJV9fgvIzZxyqYLGeZweWkqkJSubiIJFhSfNWLOM6h8trZFPpTIoBRx7KzaigUXdXukNE3g3p8PYSeY96ZNz7hdrJqVkY75zerOko9dhV0GxE1iJr5Iza/r1fg11lxK2gtOqTnr+BNmTSjbtTP79 Received: from auth1-smtp.messagingengine.com (auth1-smtp.messagingengine.com. [66.111.4.227]) by smtp.gmail.com with ESMTPSA id pc19-20020a056214489300b0068c6e9fa24asm1018586qvb.10.2024.01.31.17.42.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jan 2024 17:42:11 -0800 (PST) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailauth.nyi.internal (Postfix) with ESMTP id D58CA27C005B; Wed, 31 Jan 2024 20:42:10 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Wed, 31 Jan 2024 20:42:10 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrfedutddgfeejucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvvefufffkofgjfhgggfestdekredtredttdenucfhrhhomhepuehoqhhu nhcuhfgvnhhguceosghoqhhunhdrfhgvnhhgsehgmhgrihhlrdgtohhmqeenucggtffrrg htthgvrhhnpeegleejiedthedvheeggfejveefjeejkefgveffieeujefhueeigfegueeh geeggfenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpe gsohhquhhnodhmvghsmhhtphgruhhthhhpvghrshhonhgrlhhithihqdeiledvgeehtdei gedqudejjeekheehhedvqdgsohhquhhnrdhfvghngheppehgmhgrihhlrdgtohhmsehfih igmhgvrdhnrghmvg X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 31 Jan 2024 20:42:10 -0500 (EST) From: Boqun Feng To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org Cc: Frederic Weisbecker , "Paul E . McKenney" , Boqun Feng , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang Subject: [PATCH 1/6] rcu/nocb: Remove needless LOAD-ACQUIRE Date: Wed, 31 Jan 2024 17:40:53 -0800 Message-ID: <20240201014100.3204532-2-boqun.feng@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240201014100.3204532-1-boqun.feng@gmail.com> References: <20240201014100.3204532-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Frederic Weisbecker The LOAD-ACQUIRE access performed on rdp->nocb_cb_sleep advertizes ordering callback execution against grace period completion. However this is contradicted by the following: * This LOAD-ACQUIRE doesn't pair with anything. The only counterpart barrier that can be found is the smp_mb() placed after callbacks advancing in nocb_gp_wait(). However the barrier is placed _after_ ->nocb_cb_sleep write. * Callbacks can be concurrently advanced between the LOAD-ACQUIRE on ->nocb_cb_sleep and the call to rcu_segcblist_extract_done_cbs() in rcu_do_batch(), making any ordering based on ->nocb_cb_sleep broken. * Both rcu_segcblist_extract_done_cbs() and rcu_advance_cbs() are called under the nocb_lock, the latter hereby providing already the desired ACQUIRE semantics. Therefore it is safe to access ->nocb_cb_sleep with a simple compiler barrier. Signed-off-by: Frederic Weisbecker Reviewed-by: Paul E. McKenney Signed-off-by: Boqun Feng --- kernel/rcu/tree_nocb.h | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 4efbf7333d4e..785946834c6b 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -933,8 +933,7 @@ static void nocb_cb_wait(struct rcu_data *rdp) swait_event_interruptible_exclusive(rdp->nocb_cb_wq, nocb_cb_wait_cond(rdp)); - // VVV Ensure CB invocation follows _sleep test. - if (smp_load_acquire(&rdp->nocb_cb_sleep)) { // ^^^ + if (READ_ONCE(rdp->nocb_cb_sleep)) { WARN_ON(signal_pending(current)); trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("WokeEmpty")); } From patchwork Thu Feb 1 01:40:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boqun Feng X-Patchwork-Id: 13540568 Received: from mail-qt1-f181.google.com (mail-qt1-f181.google.com [209.85.160.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CD5195255; Thu, 1 Feb 2024 01:42:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706751736; cv=none; b=HrVu/gbVvO74T15ThLIowdFrCIQ8pKGl0NSfYHWSqq1JvVLG2IBF1w4mauCO5yVy//MUhq4hB2cBexA95/a/Mp0rOUGpsImz8gU+rHfgTsdU+892zvtRHm9ek/2jbCUFavDxNFLDndGTgyOmECQa0+urUMFihiazbY1ONlRIBoc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706751736; c=relaxed/simple; bh=pSSj7EvIHKtcO8CNxnXkeBO80f/T+be+cHyUt2QsZkM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=GNbExlc3auSM7UhfiIfLHe/Wt93ez3p3MwiTz+jcGVtbTUzcyPeS+PyJHJiuBADwJi0pf+oledVSvph08oHeFgoRTqnPHg+jF+2sYLRZdENNMYAowHfFP1gDuAxoAL6e0ff/mDfhovN2KhL3FkmjxYY3raHWSWEM9nhl8IcJwOc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=hv+ft3aY; arc=none smtp.client-ip=209.85.160.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="hv+ft3aY" Received: by mail-qt1-f181.google.com with SMTP id d75a77b69052e-42a99f34cc8so2367231cf.2; Wed, 31 Jan 2024 17:42:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1706751733; x=1707356533; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=p4HlLJA6ewOwBM3r/aagQRFX9+hzF0ZwosHf3q2R4gw=; b=hv+ft3aYUzIpq6tWxxvhor34MCO+ATd6ORApMYmwOMfN6IHrGPz9g+QGKQ0ibUUmWf aEnUvZYs4fkKFrsicTlHd6WW8nAuXVXbEdJ+qv0iAXj9+8i9K87jTMkpvGtH8YCCj7hk 66byqc6tYj72H9mz9wYTz1+cWnuqWVK+mPVGOoT5JTB1RoKC3V+tPeQObG6Z8j49leIr Ir+II3ndh3TDfYY+4y5MafXPIhzA0nGT/kdFk9kiPHeiBPxfocNgJjsQ4xG8KBDmiDkI DP0qWUUiwceLl02pPB2LWG2Y63QmSZcyA6t9d734tr0++Xv1Jhx9Zw2wYjAahx586ji9 kHiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706751733; x=1707356533; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=p4HlLJA6ewOwBM3r/aagQRFX9+hzF0ZwosHf3q2R4gw=; b=EHsreN1XVT87/jeKPPqs4eRihJKB6tN+X8ewsJes/Qr5tuuwxB2QKvzhAiAbu3qsje fOMzHQNcpzwZj/7zuWCsC9ny5YyCpPEcHmzx/MXsVu/KRzO05TDvwtVFOI5jqaio90Rv +sNIAFBSVjahp9mA0dBONpOk5xPlAE0UevGwc3yGxOG52NziLjgYRX19aokKgDeclNkM ZiBgQKR15+iIU0ir63TrOHFvFSBQWIn9NF0bJbFgizI57V7qXLpvG9DW9TwVYqSxdC17 kO4BDNurLHUfXeCYNMRVxTcsi22EdaG2nVXN6T1ynwaykl/1gVBefdhqpgGuYacCUWf6 9mGw== X-Gm-Message-State: AOJu0YwlHYab+2tivuV12uOILRNmLijDm8/u2cApb4TB8LOadAEruMFF HkPV64cHwAcWrNWczjYQKpoN5VIg0WIJgD3ELSQszwVkVrHxFkxA X-Google-Smtp-Source: AGHT+IHR2I14AziyvzHaxK+XQyZBT4pd6y+PWW9W4yxucjAybiYJy8TRkriFTnLC2wJ0575mM3SFXw== X-Received: by 2002:ac8:5795:0:b0:42a:a934:1f4c with SMTP id v21-20020ac85795000000b0042aa9341f4cmr1433192qta.2.1706751733703; Wed, 31 Jan 2024 17:42:13 -0800 (PST) X-Forwarded-Encrypted: i=0; AJvYcCXD5i1//U9u8ubT2vzi0K2o+/GwzeSTAQDt4PPCY2iGVlZvaeJzwvA34bwkEtmcQoMOulFFSQdo9AD4Y/ZpmEHbfvf7mb9DFbXm5iGTLa+ZqJze9TZ87NASpoXm26/h30XAyDauZldtN8u51C0sdQe4lkp5HFIyiYP07OpR7boPbMxcPqxXbTORJ0tnPUgwZkZZ3WIl5SCBwjLCM0NEA8mKiKQZKcAiF+J6o3sX9SxmZ0Br0zS2qnpXJ/dt23lJtos2fRlTDlJDclZrhbr9PA7E+LmSYtBwJSyNV3wZYBV91/r/cYwcLJ1E4rIIpSl/YR24UWwstvBtpL6ZRdBAAqwE6a7QhL6qjAl+kadWlIlJdpZlXufQiME1Tg09ektM Received: from auth1-smtp.messagingengine.com (auth1-smtp.messagingengine.com. [66.111.4.227]) by smtp.gmail.com with ESMTPSA id fv6-20020a05622a4a0600b00429be14d3bbsm3638748qtb.13.2024.01.31.17.42.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jan 2024 17:42:13 -0800 (PST) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailauth.nyi.internal (Postfix) with ESMTP id E295F27C005B; Wed, 31 Jan 2024 20:42:12 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Wed, 31 Jan 2024 20:42:12 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrfedutddgfeeiucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvvefufffkofgjfhgggfestdekredtredttdenucfhrhhomhepuehoqhhu nhcuhfgvnhhguceosghoqhhunhdrfhgvnhhgsehgmhgrihhlrdgtohhmqeenucggtffrrg htthgvrhhnpeegleejiedthedvheeggfejveefjeejkefgveffieeujefhueeigfegueeh geeggfenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpe gsohhquhhnodhmvghsmhhtphgruhhthhhpvghrshhonhgrlhhithihqdeiledvgeehtdei gedqudejjeekheehhedvqdgsohhquhhnrdhfvghngheppehgmhgrihhlrdgtohhmsehfih igmhgvrdhnrghmvg X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 31 Jan 2024 20:42:11 -0500 (EST) From: Boqun Feng To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org Cc: Frederic Weisbecker , "Paul E . McKenney" , Boqun Feng , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang Subject: [PATCH 2/6] rcu/nocb: Remove needless full barrier after callback advancing Date: Wed, 31 Jan 2024 17:40:54 -0800 Message-ID: <20240201014100.3204532-3-boqun.feng@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240201014100.3204532-1-boqun.feng@gmail.com> References: <20240201014100.3204532-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Frederic Weisbecker A full barrier is issued from nocb_gp_wait() upon callbacks advancing to order grace period completion with callbacks execution. However these two events are already ordered by the smp_mb__after_unlock_lock() barrier within the call to raw_spin_lock_rcu_node() that is necessary for callbacks advancing to happen. The following litmus test shows the kind of guarantee that this barrier provides: C smp_mb__after_unlock_lock {} // rcu_gp_cleanup() P0(spinlock_t *rnp_lock, int *gpnum) { // Grace period cleanup increase gp sequence number spin_lock(rnp_lock); WRITE_ONCE(*gpnum, 1); spin_unlock(rnp_lock); } // nocb_gp_wait() P1(spinlock_t *rnp_lock, spinlock_t *nocb_lock, int *gpnum, int *cb_ready) { int r1; // Call rcu_advance_cbs() from nocb_gp_wait() spin_lock(nocb_lock); spin_lock(rnp_lock); smp_mb__after_unlock_lock(); r1 = READ_ONCE(*gpnum); WRITE_ONCE(*cb_ready, 1); spin_unlock(rnp_lock); spin_unlock(nocb_lock); } // nocb_cb_wait() P2(spinlock_t *nocb_lock, int *cb_ready, int *cb_executed) { int r2; // rcu_do_batch() -> rcu_segcblist_extract_done_cbs() spin_lock(nocb_lock); r2 = READ_ONCE(*cb_ready); spin_unlock(nocb_lock); // Actual callback execution WRITE_ONCE(*cb_executed, 1); } P3(int *cb_executed, int *gpnum) { int r3; WRITE_ONCE(*cb_executed, 2); smp_mb(); r3 = READ_ONCE(*gpnum); } exists (1:r1=1 /\ 2:r2=1 /\ cb_executed=2 /\ 3:r3=0) (* Bad outcome. *) Here the bad outcome only occurs if the smp_mb__after_unlock_lock() is removed. This barrier orders the grace period completion against callbacks advancing and even later callbacks invocation, thanks to the opportunistic propagation via the ->nocb_lock to nocb_cb_wait(). Therefore the smp_mb() placed after callbacks advancing can be safely removed. Signed-off-by: Frederic Weisbecker Reviewed-by: Paul E. McKenney Signed-off-by: Boqun Feng --- kernel/rcu/tree.c | 6 ++++++ kernel/rcu/tree_nocb.h | 1 - 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index b2bccfd37c38..d540d210e5c7 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2145,6 +2145,12 @@ static void rcu_do_batch(struct rcu_data *rdp) * Extract the list of ready callbacks, disabling IRQs to prevent * races with call_rcu() from interrupt handlers. Leave the * callback counts, as rcu_barrier() needs to be conservative. + * + * Callbacks execution is fully ordered against preceding grace period + * completion (materialized by rnp->gp_seq update) thanks to the + * smp_mb__after_unlock_lock() upon node locking required for callbacks + * advancing. In NOCB mode this ordering is then further relayed through + * the nocb locking that protects both callbacks advancing and extraction. */ rcu_nocb_lock_irqsave(rdp, flags); WARN_ON_ONCE(cpu_is_offline(smp_processor_id())); diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 785946834c6b..b2c3145c4c13 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -779,7 +779,6 @@ static void nocb_gp_wait(struct rcu_data *my_rdp) if (rcu_segcblist_ready_cbs(&rdp->cblist)) { needwake = rdp->nocb_cb_sleep; WRITE_ONCE(rdp->nocb_cb_sleep, false); - smp_mb(); /* CB invocation -after- GP end. */ } else { needwake = false; } From patchwork Thu Feb 1 01:40:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boqun Feng X-Patchwork-Id: 13540569 Received: from mail-qv1-f41.google.com (mail-qv1-f41.google.com [209.85.219.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8247463A0; Thu, 1 Feb 2024 01:42:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706751738; cv=none; b=WTTEMcTbp2/WHaBh3yj9NKgmuqdF1grpd9AoljSbUSnNXlCy6dwuoga4twi+rEHGRlvxUL+WiDYtLabD4M/hYUeWfpbTU4b4JeZesBR1JfwN6ta2Y5b2M/pMJod4D8G/aHy11e1MA0aFvKwiFGj50mglnMU9R0sswdrcRLgjyEw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706751738; c=relaxed/simple; bh=URsVmEp6Jr873vr5jenxJaWo72Ku3rabQQ+gAmu0f1E=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=gXGpFfytrlHZ4wevifCyEUNggv6h7755Jp9lQpPszX3R7zB26QBv7/9/J1xeqFgsh3azPkHe3ypVsxDlT+vJPeyw8RwoCN3PAeJmEa3r8RCYXNbGAEEmO9Ebd//95M2e+os9sMnnWjChiBUFhWCaIlEkIsGB173lGOBvDddf/t8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=LZQUa7Mh; arc=none smtp.client-ip=209.85.219.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="LZQUa7Mh" Received: by mail-qv1-f41.google.com with SMTP id 6a1803df08f44-68c44100327so2017606d6.2; Wed, 31 Jan 2024 17:42:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1706751735; x=1707356535; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=98uYtdjZqt8mPN8u771GmaA9N12/7R+F5RUUjJgYUPQ=; b=LZQUa7MhUF0BNfXRKevudcd1zOzldfHlWvlQbbyG94tvN4k+wYl1veVoII+0e/Yo8g HMYRv0PEFk6NEyPDcp1XWYHRIjggnCGXunk1E8Sc5xAMy2zZgVjnA21BHVHWFcQQXmhK dTDxePnOo1zHb982AgdeqK2zga4Ixc33cSGAKXspn6gz5tntGnqfLshVYxnYg1HAL2SJ 6jw9kWK2y/638TNb+IA8Qy3arkGXLf8L0ekD3l7QfjHhKkTWx6uPUO0KyMI/0eNT+fox Qwc+Bhe1tpd7LMURQMoKOjeXuKPoLtXUyIIWzbcIhfIqhT6Bdxj6eVNeExqqiijXNc8z TgqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706751735; x=1707356535; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=98uYtdjZqt8mPN8u771GmaA9N12/7R+F5RUUjJgYUPQ=; b=lOpcmiQgWDgf2rQ1fU0jywC3TFB/0CnokAN3nN1L2P2rnI9lIjItS/eQOJU9y5sU+8 KDQbuIScdXrNYppvZyTsUWReP+7LOSkqblhHaofaXdL0vFYu913RpY8CqJkd4lUYgkZy /oiaUszXWTch5u4sKFZUh2ZP+4tKscyM5fURFVpD/pilqOjY2n+lYzGSr3SDIDYfr7x+ QR2AbegCpMcqpF6zg6Z74cLYYXkOPa/ZPTSv8henDp0Nk3geFVOtgXFhJXPOmgCxLTWb QAPLHQQLO9yRcpPjYILrXrgzSNTtJtyuY1yPq/CBXW+x4az/EMegpY7aGOG0IOpKrT9g 1QdQ== X-Gm-Message-State: AOJu0YyxIrOgtmw5bVghEZLpt4rkJbEDwsi++/S3woE2qcL/f1ncUSV2 hkYMR/5L04X1zUFaouEeyhwImNby1QaiJC/YrLmZgk0coMjQKKFo X-Google-Smtp-Source: AGHT+IGmZu+egkcf1f6ysn8wFESXHeVIjltXqOdkSszNHlpulRGuA1ijCixLr+Ou66HSNAkuYpV4eQ== X-Received: by 2002:a05:6214:20e6:b0:68c:4caa:dd17 with SMTP id 6-20020a05621420e600b0068c4caadd17mr4015452qvk.6.1706751735429; Wed, 31 Jan 2024 17:42:15 -0800 (PST) X-Forwarded-Encrypted: i=0; AJvYcCV5H58O5j3sQdOL+vJNGvEAGH3Se5j3EHzm0jbl5Emvk7HuvTTSdqJiI8P12gEB7VHZI0tMOi+oHbFCB8tcfbHWoqSr8eVDiLAz+Nd1kH7UNxPM7BHiMTLWibDtND+WY1s2/dkmpcbkiSfKmfPxhMAehNYm3ygbG6ahESqeKE4zHyPaWiAuEUaNs/jnIT5MUD16IcTUJIpCgFIkaRQzsXPy7rqnG/kNrzZv0WiwgmNvEyS4fcO0d+4c5AUReRm4cgeacFQueuIrzyC/+iBOzlMV4Gubck2GXuZ44mIvqVIujzNMIJdyoFJQChJh0gpSVfn+E3Kvcv5Gyn3BCxOBeg0uJ9O73QE/G27CDzlpDLfnFTMQcJIEvV5WwYdV5Zl9f+J/jaG9DSbD+g/+o4xRXKXUMxr6moZVUw== Received: from auth1-smtp.messagingengine.com (auth1-smtp.messagingengine.com. [66.111.4.227]) by smtp.gmail.com with ESMTPSA id on6-20020a056214448600b00685ad9090basm2453293qvb.97.2024.01.31.17.42.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jan 2024 17:42:15 -0800 (PST) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailauth.nyi.internal (Postfix) with ESMTP id A91B027C005B; Wed, 31 Jan 2024 20:42:14 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute5.internal (MEProxy); Wed, 31 Jan 2024 20:42:14 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrfedutddgfeejucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvvefufffkofgjfhgggfestdekredtredttdenucfhrhhomhepuehoqhhu nhcuhfgvnhhguceosghoqhhunhdrfhgvnhhgsehgmhgrihhlrdgtohhmqeenucggtffrrg htthgvrhhnpeegleejiedthedvheeggfejveefjeejkefgveffieeujefhueeigfegueeh geeggfenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpe gsohhquhhnodhmvghsmhhtphgruhhthhhpvghrshhonhgrlhhithihqdeiledvgeehtdei gedqudejjeekheehhedvqdgsohhquhhnrdhfvghngheppehgmhgrihhlrdgtohhmsehfih igmhgvrdhnrghmvg X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 31 Jan 2024 20:42:14 -0500 (EST) From: Boqun Feng To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org Cc: Frederic Weisbecker , Neeraj Upadhyay , "Paul E . McKenney" , Boqun Feng , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang Subject: [PATCH 3/6] rcu/nocb: Make IRQs disablement symmetric Date: Wed, 31 Jan 2024 17:40:55 -0800 Message-ID: <20240201014100.3204532-4-boqun.feng@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240201014100.3204532-1-boqun.feng@gmail.com> References: <20240201014100.3204532-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Frederic Weisbecker Currently IRQs are disabled on call_rcu() and then depending on the context: * If the CPU is in nocb mode: - If the callback is enqueued in the bypass list, IRQs are re-enabled implictly by rcu_nocb_try_bypass() - If the callback is enqueued in the normal list, IRQs are re-enabled implicitly by __call_rcu_nocb_wake() * If the CPU is NOT in nocb mode, IRQs are reenabled explicitly from call_rcu() This makes the code a bit hard to follow, especially as it interleaves with nocb locking. To make the IRQ flags coverage clearer and also in order to prepare for moving all the nocb enqueue code to its own function, always re-enable the IRQ flags explicitly from call_rcu(). Reviewed-by: Neeraj Upadhyay (AMD) Signed-off-by: Frederic Weisbecker Reviewed-by: Paul E. McKenney Signed-off-by: Boqun Feng --- kernel/rcu/tree.c | 9 ++++++--- kernel/rcu/tree_nocb.h | 20 +++++++++----------- 2 files changed, 15 insertions(+), 14 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index d540d210e5c7..a402dc4e9a9c 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2735,8 +2735,10 @@ __call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy_in) } check_cb_ovld(rdp); - if (rcu_nocb_try_bypass(rdp, head, &was_alldone, flags, lazy)) + if (rcu_nocb_try_bypass(rdp, head, &was_alldone, flags, lazy)) { + local_irq_restore(flags); return; // Enqueued onto ->nocb_bypass, so just leave. + } // If no-CBs CPU gets here, rcu_nocb_try_bypass() acquired ->nocb_lock. rcu_segcblist_enqueue(&rdp->cblist, head); if (__is_kvfree_rcu_offset((unsigned long)func)) @@ -2754,8 +2756,8 @@ __call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy_in) __call_rcu_nocb_wake(rdp, was_alldone, flags); /* unlocks */ } else { __call_rcu_core(rdp, head, flags); - local_irq_restore(flags); } + local_irq_restore(flags); } #ifdef CONFIG_RCU_LAZY @@ -4646,8 +4648,9 @@ void rcutree_migrate_callbacks(int cpu) __call_rcu_nocb_wake(my_rdp, true, flags); } else { rcu_nocb_unlock(my_rdp); /* irqs remain disabled. */ - raw_spin_unlock_irqrestore_rcu_node(my_rnp, flags); + raw_spin_unlock_rcu_node(my_rnp); /* irqs remain disabled. */ } + local_irq_restore(flags); if (needwake) rcu_gp_kthread_wake(); lockdep_assert_irqs_enabled(); diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index b2c3145c4c13..1d5c03c5c702 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -532,9 +532,7 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, // 2. Both of these conditions are met: // a. The bypass list previously had only lazy CBs, and: // b. The new CB is non-lazy. - if (ncbs && (!bypass_is_lazy || lazy)) { - local_irq_restore(flags); - } else { + if (!ncbs || (bypass_is_lazy && !lazy)) { // No-CBs GP kthread might be indefinitely asleep, if so, wake. rcu_nocb_lock(rdp); // Rare during call_rcu() flood. if (!rcu_segcblist_pend_cbs(&rdp->cblist)) { @@ -544,7 +542,7 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, } else { trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("FirstBQnoWake")); - rcu_nocb_unlock_irqrestore(rdp, flags); + rcu_nocb_unlock(rdp); } } return true; // Callback already enqueued. @@ -570,7 +568,7 @@ static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_alldone, // If we are being polled or there is no kthread, just leave. t = READ_ONCE(rdp->nocb_gp_kthread); if (rcu_nocb_poll || !t) { - rcu_nocb_unlock_irqrestore(rdp, flags); + rcu_nocb_unlock(rdp); trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("WakeNotPoll")); return; @@ -583,17 +581,17 @@ static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_alldone, rdp->qlen_last_fqs_check = len; // Only lazy CBs in bypass list if (lazy_len && bypass_len == lazy_len) { - rcu_nocb_unlock_irqrestore(rdp, flags); + rcu_nocb_unlock(rdp); wake_nocb_gp_defer(rdp, RCU_NOCB_WAKE_LAZY, TPS("WakeLazy")); } else if (!irqs_disabled_flags(flags)) { /* ... if queue was empty ... */ - rcu_nocb_unlock_irqrestore(rdp, flags); + rcu_nocb_unlock(rdp); wake_nocb_gp(rdp, false); trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("WakeEmpty")); } else { - rcu_nocb_unlock_irqrestore(rdp, flags); + rcu_nocb_unlock(rdp); wake_nocb_gp_defer(rdp, RCU_NOCB_WAKE, TPS("WakeEmptyIsDeferred")); } @@ -611,15 +609,15 @@ static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_alldone, if ((rdp->nocb_cb_sleep || !rcu_segcblist_ready_cbs(&rdp->cblist)) && !timer_pending(&rdp->nocb_timer)) { - rcu_nocb_unlock_irqrestore(rdp, flags); + rcu_nocb_unlock(rdp); wake_nocb_gp_defer(rdp, RCU_NOCB_WAKE_FORCE, TPS("WakeOvfIsDeferred")); } else { - rcu_nocb_unlock_irqrestore(rdp, flags); + rcu_nocb_unlock(rdp); trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("WakeNot")); } } else { - rcu_nocb_unlock_irqrestore(rdp, flags); + rcu_nocb_unlock(rdp); trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("WakeNot")); } } From patchwork Thu Feb 1 01:40:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Boqun Feng X-Patchwork-Id: 13540570 Received: from mail-qk1-f174.google.com (mail-qk1-f174.google.com [209.85.222.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B8794FBE7; Thu, 1 Feb 2024 01:42:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706751740; cv=none; b=QPqyCbRroOYylcHDPSjL4+F6i7FiyHy6ZzMybEg6fsRVMeHmYpZsFN+8mfjpEnZeECdOh+qHdaazz8S4x+yss9p41Wp35EWXMQ1ZGNOK16UtKLBEcozYOOuy1Zmnhn2nBXqzxHBhtJd8XfbFJ4jiqFEIxtrmH19+3R+AHQrue1U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706751740; c=relaxed/simple; bh=vN+b/ifMf2GpJznfNVNFzi7NjJGu81cMQy2PppVHW9I=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=JBV1ECOF8yKYSFv0/BxpOMbQRzxy46hUUMt/b1SrH9XkRKg5DZTVV4gSQr/Rlz/s1NLlN00iMa7Rvma43piwlda2ODSko/hPujz1Oa4o/UK82RpuPMGK+n7s41QEyrj7sVa8jltoeEiZgd1mTzEyP4E1M2PfnTzvu78Z/QVRZto= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=DUvovgep; arc=none smtp.client-ip=209.85.222.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="DUvovgep" Received: by mail-qk1-f174.google.com with SMTP id af79cd13be357-783f27300b1so25369385a.1; Wed, 31 Jan 2024 17:42:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1706751737; x=1707356537; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=HkR3gzB832b3FW96NlBLB1xXylKJv2Axeh5KafyCU8E=; b=DUvovgep9E9pgBwh78jqlcgwESoSlQmrkW6esP7dXciXNMwMH+7oNhoL5dH6OoKm8O dNwG1rnje4peYgTqMtn/h3k2U4G8tQOykEoMSrZ4Km/GLWrvEd9KQS83GjYEa3cmS4/T VJEhvAETw5i6IFPkKosKSgvp1gsKC5s8d5bJHjIz3/4t4lw7qpzjA4agkcz2+9RnL8Uv G+xCZtiYWlQnlw4U9D+xXzwKWBaMCza/LeWYK5xEHfasBbTE8VTNWvHa5WlKJSOkWaBd ocCrntCV8pkZ/g5tsq/rF4RvKCvyZT2qqLVoYl/fRMXDa2gwUfvpf57wGrInI8W5iyZ7 7ykA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706751737; x=1707356537; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=HkR3gzB832b3FW96NlBLB1xXylKJv2Axeh5KafyCU8E=; b=Qrhzb4D7IGXlIPhWAG2K3FSEGn5zRY0mXyUeQr14odkhLPTWnqrDeBX+yhGauLF1Ij Kb3CSvGf7IVaWshj/JdJnYdnbdeSs1VTCvnFkeb87WKtpgjsctxMudJQNmNbcaKkh1Qz 91PrZ5Ts2fGQR6QaxFSGu8lkjYmySe3amdeWonxRas9PF30JnR6mCN7ttjHDAA4qm9yf 55R2UaOpZVXbsbkgU5CJLBquSzQbnE0SMfNc93bZ2kanQr3O4+pcm9yvS1kjBjvKI5sJ lPxtvcmOOdxKxZVJFqFVcBjmpjyr4iExisngzDNxABX3UjA0JqvyAhq+maHeZq4hKarQ yefQ== X-Gm-Message-State: AOJu0YyLndnOdGRbwref6tAcFBVtAOmglcLCVtyY4+RlJsu3aUhvuXKL jPvjyXyZY+rvJjmo/Xouy3N5+xN23hi68ddLII3gO991vDSiLSP/ X-Google-Smtp-Source: AGHT+IHN0DKmr04jB8Sfqo23EfmK7Ir1HrkOhBMZpNw7emhEOlxsTJPat1oMsnvXTApc9asZZBrsvQ== X-Received: by 2002:a05:6214:20ea:b0:68c:7aab:696f with SMTP id 10-20020a05621420ea00b0068c7aab696fmr637624qvk.56.1706751737591; Wed, 31 Jan 2024 17:42:17 -0800 (PST) X-Forwarded-Encrypted: i=0; AJvYcCX3uAeKx2NmzGlq4BnurQEwL6ItQoP/F3SbwkDACvEGgNeBwQxAo/QKhyoFI2JF9X77wB3dAdO9A3VSv83pA7F1yVKeMe0wXdAYcd2PLRm+Qr+PCLG3AQHrT5d6AigUqPpX3UyTI6OvKEkqaSS5H9YoTcES8XCDZqcwop9PID5rNq9NKzIUPOII3c0vnnIWDtMATMFe4ymughYaqyIpqi9LAJbeV289cLo21MZuCnDvpvWVGiE5ICi3b8pkicsrBHAAQNw3tdK8+Q/S7B9rn2jpdmRf79mnUPCqoWzEfZCdaYBl4uzx/+LNgSWuS2IolvrynmESZvtL5ER0ZrpyudDO6rPBYcxz8vcU6KYrcZeopiahm+XO6V4z2p86dT5Q Received: from fauth1-smtp.messagingengine.com (fauth1-smtp.messagingengine.com. [103.168.172.200]) by smtp.gmail.com with ESMTPSA id eo10-20020ad4594a000000b0068179afede6sm3263621qvb.100.2024.01.31.17.42.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jan 2024 17:42:17 -0800 (PST) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailfauth.nyi.internal (Postfix) with ESMTP id 84C091200043; Wed, 31 Jan 2024 20:42:16 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Wed, 31 Jan 2024 20:42:16 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrfedutddgfeejucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvvefufffkofgjfhggtgfgsehtkeertdertdejnecuhfhrohhmpeeuohhq uhhnucfhvghnghcuoegsohhquhhnrdhfvghnghesghhmrghilhdrtghomheqnecuggftrf grthhtvghrnhepheffieehhefhveehhfelfeejtdeitdejhfevkeefjeeiudegkeekueeu hfeiveeknecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomh epsghoqhhunhdomhgvshhmthhprghuthhhphgvrhhsohhnrghlihhthidqieelvdeghedt ieegqddujeejkeehheehvddqsghoqhhunhdrfhgvnhhgpeepghhmrghilhdrtghomhesfh higihmvgdrnhgrmhgv X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 31 Jan 2024 20:42:15 -0500 (EST) From: Boqun Feng To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org Cc: Frederic Weisbecker , "Paul E . McKenney" , Boqun Feng , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang Subject: [PATCH 4/6] rcu/nocb: Re-arrange call_rcu() NOCB specific code Date: Wed, 31 Jan 2024 17:40:56 -0800 Message-ID: <20240201014100.3204532-5-boqun.feng@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240201014100.3204532-1-boqun.feng@gmail.com> References: <20240201014100.3204532-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Frederic Weisbecker Currently the call_rcu() function interleaves NOCB and !NOCB enqueue code in a complicated way such that: * The bypass enqueue code may or may not have enqueued and may or may not have locked the ->nocb_lock. Everything that follows is in a Schrödinger locking state for the unwary reviewer's eyes. * The was_alldone is always set but only used in NOCB related code. * The NOCB wake up is distantly related to the locking hopefully performed by the bypass enqueue code that did not enqueue on the bypass list. Unconfuse the whole and gather NOCB and !NOCB specific enqueue code to their own functions. Signed-off-by: Frederic Weisbecker Reviewed-by: Paul E. McKenney Signed-off-by: Boqun Feng --- kernel/rcu/tree.c | 44 +++++++++++++++++++----------------------- kernel/rcu/tree.h | 9 ++++----- kernel/rcu/tree_nocb.h | 18 ++++++++++++++--- 3 files changed, 39 insertions(+), 32 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index a402dc4e9a9c..cc0e169e299a 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2597,12 +2597,26 @@ static int __init rcu_spawn_core_kthreads(void) return 0; } +static void rcutree_enqueue(struct rcu_data *rdp, struct rcu_head *head, rcu_callback_t func) +{ + rcu_segcblist_enqueue(&rdp->cblist, head); + if (__is_kvfree_rcu_offset((unsigned long)func)) + trace_rcu_kvfree_callback(rcu_state.name, head, + (unsigned long)func, + rcu_segcblist_n_cbs(&rdp->cblist)); + else + trace_rcu_callback(rcu_state.name, head, + rcu_segcblist_n_cbs(&rdp->cblist)); + trace_rcu_segcb_stats(&rdp->cblist, TPS("SegCBQueued")); +} + /* * Handle any core-RCU processing required by a call_rcu() invocation. */ -static void __call_rcu_core(struct rcu_data *rdp, struct rcu_head *head, - unsigned long flags) +static void call_rcu_core(struct rcu_data *rdp, struct rcu_head *head, + rcu_callback_t func, unsigned long flags) { + rcutree_enqueue(rdp, head, func); /* * If called from an extended quiescent state, invoke the RCU * core in order to force a re-evaluation of RCU's idleness. @@ -2698,7 +2712,6 @@ __call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy_in) unsigned long flags; bool lazy; struct rcu_data *rdp; - bool was_alldone; /* Misaligned rcu_head! */ WARN_ON_ONCE((unsigned long)head & (sizeof(void *) - 1)); @@ -2735,28 +2748,11 @@ __call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy_in) } check_cb_ovld(rdp); - if (rcu_nocb_try_bypass(rdp, head, &was_alldone, flags, lazy)) { - local_irq_restore(flags); - return; // Enqueued onto ->nocb_bypass, so just leave. - } - // If no-CBs CPU gets here, rcu_nocb_try_bypass() acquired ->nocb_lock. - rcu_segcblist_enqueue(&rdp->cblist, head); - if (__is_kvfree_rcu_offset((unsigned long)func)) - trace_rcu_kvfree_callback(rcu_state.name, head, - (unsigned long)func, - rcu_segcblist_n_cbs(&rdp->cblist)); - else - trace_rcu_callback(rcu_state.name, head, - rcu_segcblist_n_cbs(&rdp->cblist)); - - trace_rcu_segcb_stats(&rdp->cblist, TPS("SegCBQueued")); - /* Go handle any RCU core processing required. */ - if (unlikely(rcu_rdp_is_offloaded(rdp))) { - __call_rcu_nocb_wake(rdp, was_alldone, flags); /* unlocks */ - } else { - __call_rcu_core(rdp, head, flags); - } + if (unlikely(rcu_rdp_is_offloaded(rdp))) + call_rcu_nocb(rdp, head, func, flags, lazy); + else + call_rcu_core(rdp, head, func, flags); local_irq_restore(flags); } diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h index e9821a8422db..bf478da89a8f 100644 --- a/kernel/rcu/tree.h +++ b/kernel/rcu/tree.h @@ -467,11 +467,10 @@ static void rcu_init_one_nocb(struct rcu_node *rnp); static bool wake_nocb_gp(struct rcu_data *rdp, bool force); static bool rcu_nocb_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp, unsigned long j, bool lazy); -static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, - bool *was_alldone, unsigned long flags, - bool lazy); -static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_empty, - unsigned long flags); +static void call_rcu_nocb(struct rcu_data *rdp, struct rcu_head *head, + rcu_callback_t func, unsigned long flags, bool lazy); +static void __maybe_unused __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_empty, + unsigned long flags); static int rcu_nocb_need_deferred_wakeup(struct rcu_data *rdp, int level); static bool do_nocb_deferred_wakeup(struct rcu_data *rdp); static void rcu_boot_init_nocb_percpu_data(struct rcu_data *rdp); diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 1d5c03c5c702..9e8052ba14b9 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -622,6 +622,18 @@ static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_alldone, } } +static void call_rcu_nocb(struct rcu_data *rdp, struct rcu_head *head, + rcu_callback_t func, unsigned long flags, bool lazy) +{ + bool was_alldone; + + if (!rcu_nocb_try_bypass(rdp, head, &was_alldone, flags, lazy)) { + /* Not enqueued on bypass but locked, do regular enqueue */ + rcutree_enqueue(rdp, head, func); + __call_rcu_nocb_wake(rdp, was_alldone, flags); /* unlocks */ + } +} + static int nocb_gp_toggle_rdp(struct rcu_data *rdp, bool *wake_state) { @@ -1764,10 +1776,10 @@ static bool rcu_nocb_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp, return true; } -static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, - bool *was_alldone, unsigned long flags, bool lazy) +static void call_rcu_nocb(struct rcu_data *rdp, struct rcu_head *head, + rcu_callback_t func, unsigned long flags, bool lazy) { - return false; + WARN_ON_ONCE(1); /* Should be dead code! */ } static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_empty, From patchwork Thu Feb 1 01:40:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boqun Feng X-Patchwork-Id: 13540571 Received: from mail-qt1-f170.google.com (mail-qt1-f170.google.com [209.85.160.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3764419BDC; Thu, 1 Feb 2024 01:42:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706751741; cv=none; b=USTIHFdH+04gRHNsr1+Hd5fc5MAeycDaN/WycXibE3ScqHd2PwLwOQaVjMoZq+NCzevuxuV1sSXbNPsqZJLDmK2RHL6aGPOX6r/IuE2Cw60n/9fdcEoGnhgcY5vtJoW53S31loikxXlgxPhE/wPZT1f0d7BxXlo4t8Cw8MXiDNg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706751741; c=relaxed/simple; bh=O8nU5bEULJUElIjpZZSaHK6+7EZs2Vxmkgn27DuOtNg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=OH/KiaTktl71BvdPb7EE/CU4gubHves2PX9f998FgnmxIib1t4dspGrBqGMO4u/qFe3nhkhAW1jtZQ9DHnhQGYl5yuasAprx+C33afpl993jsiQAHgF4i8XEokB8UvSpMA2dtZdPcJgaAEQzy3AkrGO9lY2EB3fUMVh69lrC96w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=ODXak1IS; arc=none smtp.client-ip=209.85.160.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ODXak1IS" Received: by mail-qt1-f170.google.com with SMTP id d75a77b69052e-42993124fa1so2500961cf.3; Wed, 31 Jan 2024 17:42:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1706751739; x=1707356539; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=kITZv0k9fxTTu9rhEhSjW/uQbfGz/wypbxDOcIf6Ytk=; b=ODXak1ISq3RHSWZkzysRoiT0/3TxPFnaRFie//clqDP80Q3g3grT/GdOK449OdLs5f 3183wCIfRVA3/vwTXRo3lyfN4PnrJyYX8oq70q/Gr6Z+3BA14wrbL2+y4p4ZLW/0eHPo u9IhFKG4B9c4UgtuQE3d01QSre1RcaKv50OANTWxFK++w5319eXt8TgmUrZJhv2lk41u XAldT2rZKMNEWlTKoisLuxBLbL4Bjxi7RsaVhfdw2ZwJZRHwzmU2Zps6HFsaZCENr8BJ BgJaNshSNXhQnbYw6a+7q5k8gCDqwBNnYVOfG77OqACKl5peEIoqIA9hVfxLjgqp1Yoa ZTqA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706751739; x=1707356539; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=kITZv0k9fxTTu9rhEhSjW/uQbfGz/wypbxDOcIf6Ytk=; b=wlRkShvR7xaeJgzZMzgpOdJTMEJjMDLs8F1w/pNf3xAWfIpw+e177ixKp0Xdhm+KCv f8BrlER90VeRXXZHAnkxusLbze6hV2P1X5YGQNyAJleIPv7knar4WU0AJfaFsr4MaBdK uh81BVj4wcGGsEMjaFd9hgMwekPrmmUzSgwiIRSFguU4yLzPkbquAJEgIUw3eGoi47hL RhLoT4gnvl7jc52307KEvBAhzsKubdHn9MEwDJFzXYPdpg9FoQb+GeubXiAR/A4H1wg2 nBXSTp6/7vxqxrzLEqiOAYKDOJQ/cJtpJ8FjHH+pL7DzX5IEcrXlO9iSMqNfOH6jAo6M /acw== X-Gm-Message-State: AOJu0YxK0fdgJNfCfktbGmu7Hd0IrFKIDqFzEwoEMlqnrNoiwKkDZ6R7 1RQttLCfxuC+3jAEQ8vP3/TfU/N+oSUsx91JjgvQ7UNSdugM+rvc X-Google-Smtp-Source: AGHT+IF54iKoob1lK/h1KqyUuXfADz8JsCv1IYCAn+54L6vf2AXkZQQbqJz9nYKw5YPhd78F/D79Jw== X-Received: by 2002:ac8:5947:0:b0:42b:ec11:1a6b with SMTP id 7-20020ac85947000000b0042bec111a6bmr3654721qtz.67.1706751739040; Wed, 31 Jan 2024 17:42:19 -0800 (PST) X-Forwarded-Encrypted: i=0; AJvYcCWyQ/sB2p8wMYMTrAO5UrG4Qn303OKDFKXkbM494Xo3znQVI0CPuSatGuIIl7tjl5Ipx9gD9xmEvBfHDcdi3f6XqxCMfJ3CgzxfZmt5xR1sZOrcOvv9+qWeZ0fkIovuhKs8au6MeHk2kCqz/mFa3SEBbv/JtYn2/jtB5WkIVJBb91c5R5pWwwd9G2zZSI3L0N61ZMZI1PikHXWuRYBTyMy2qLEvB56XnqscgUK3ifJo2aKdzRfMmiqUogxqeWSyqVaUeM8umARUCyZsJR8TAkLBTzDKSKYcmBVlqhWfZ5j0++XMWMrho1MHAIl+whO0Eajq+zcxdYWHEeeJsf/+1TWeUK/HnJkih5BCWXdS7MJMmywmtZ15nZFg Received: from fauth1-smtp.messagingengine.com (fauth1-smtp.messagingengine.com. [103.168.172.200]) by smtp.gmail.com with ESMTPSA id gd13-20020a05622a5c0d00b0042992b06012sm1644885qtb.2.2024.01.31.17.42.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jan 2024 17:42:18 -0800 (PST) Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailfauth.nyi.internal (Postfix) with ESMTP id 438251200043; Wed, 31 Jan 2024 20:42:18 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute6.internal (MEProxy); Wed, 31 Jan 2024 20:42:18 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrfedutddgfeejucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvvefufffkofgjfhgggfestdekredtredttdenucfhrhhomhepuehoqhhu nhcuhfgvnhhguceosghoqhhunhdrfhgvnhhgsehgmhgrihhlrdgtohhmqeenucggtffrrg htthgvrhhnpeegleejiedthedvheeggfejveefjeejkefgveffieeujefhueeigfegueeh geeggfenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpe gsohhquhhnodhmvghsmhhtphgruhhthhhpvghrshhonhgrlhhithihqdeiledvgeehtdei gedqudejjeekheehhedvqdgsohhquhhnrdhfvghngheppehgmhgrihhlrdgtohhmsehfih igmhgvrdhnrghmvg X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 31 Jan 2024 20:42:17 -0500 (EST) From: Boqun Feng To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org Cc: Zqiang , Joel Fernandes , Frederic Weisbecker , "Paul E . McKenney" , Boqun Feng , Neeraj Upadhyay , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan Subject: [PATCH 5/6] rcu/nocb: Fix WARN_ON_ONCE() in the rcu_nocb_bypass_lock() Date: Wed, 31 Jan 2024 17:40:57 -0800 Message-ID: <20240201014100.3204532-6-boqun.feng@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240201014100.3204532-1-boqun.feng@gmail.com> References: <20240201014100.3204532-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Zqiang For the kernels built with CONFIG_RCU_NOCB_CPU_DEFAULT_ALL=y and CONFIG_RCU_LAZY=y, the following scenarios will trigger WARN_ON_ONCE() in the rcu_nocb_bypass_lock() and rcu_nocb_wait_contended() functions: CPU2 CPU11 kthread rcu_nocb_cb_kthread ksys_write rcu_do_batch vfs_write rcu_torture_timer_cb proc_sys_write __kmem_cache_free proc_sys_call_handler kmemleak_free drop_caches_sysctl_handler delete_object_full drop_slab __delete_object shrink_slab put_object lazy_rcu_shrink_scan call_rcu rcu_nocb_flush_bypass __call_rcu_commn rcu_nocb_bypass_lock raw_spin_trylock(&rdp->nocb_bypass_lock) fail atomic_inc(&rdp->nocb_lock_contended); rcu_nocb_wait_contended WARN_ON_ONCE(smp_processor_id() != rdp->cpu); WARN_ON_ONCE(atomic_read(&rdp->nocb_lock_contended)) | |_ _ _ _ _ _ _ _ _ _same rdp and rdp->cpu != 11_ _ _ _ _ _ _ _ _ __| Reproduce this bug with "echo 3 > /proc/sys/vm/drop_caches". This commit therefore uses rcu_nocb_try_flush_bypass() instead of rcu_nocb_flush_bypass() in lazy_rcu_shrink_scan(). If the nocb_bypass queue is being flushed, then rcu_nocb_try_flush_bypass will return directly. Signed-off-by: Zqiang Reviewed-by: Joel Fernandes (Google) Reviewed-by: Frederic Weisbecker Reviewed-by: Paul E. McKenney Signed-off-by: Boqun Feng --- kernel/rcu/tree_nocb.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 9e8052ba14b9..ffa69a5e18f4 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1391,7 +1391,7 @@ lazy_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) rcu_nocb_unlock_irqrestore(rdp, flags); continue; } - WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies, false)); + rcu_nocb_try_flush_bypass(rdp, jiffies); rcu_nocb_unlock_irqrestore(rdp, flags); wake_nocb_gp(rdp, false); sc->nr_to_scan -= _count; From patchwork Thu Feb 1 01:40:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boqun Feng X-Patchwork-Id: 13540572 Received: from mail-qk1-f171.google.com (mail-qk1-f171.google.com [209.85.222.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D6B3F4C7B; Thu, 1 Feb 2024 01:42:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706751743; cv=none; b=NVJGuMpcfFDwA6F7NnNYay3crQGsqyTHpDXLi3JkZW/EBFYrT7eO1aHCi5jULXv9zaDEZq5Pr80qeOJMmQfA9I92BGGHQxIvePfVjpX59vcyIYu5Jxm2QhPxHK8iKPDH+PkYu37BciMNYD47EfrMTOdWPjnbElKrJLgk+WOylRk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706751743; c=relaxed/simple; bh=6oMVGx/JwGgsizKVfPO1lHIPkL1xgjV9NuMQhjvciWc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=tudURNa+bwSd/RYoc8m1Aer1fBOD6WtfD9FARAGJ32yVWeR2HYTbEG/RTACr+yteeqP7ZpC8ZgXD860HTyn5mpQaOf43jifIw0o3utbzPpKMQUb3Lnzf+wi6Ve2LGrcndr9tFPi9tRBvFrpVgZlmXuZtsD7mr+3jnKrBV0vqff8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=ZuLWh71B; arc=none smtp.client-ip=209.85.222.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ZuLWh71B" Received: by mail-qk1-f171.google.com with SMTP id af79cd13be357-783ced12f9bso19792585a.3; Wed, 31 Jan 2024 17:42:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1706751740; x=1707356540; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=4WRKA+8oB7qFwtR1n75bKmIl1ybu1sGsFv65Z7DNMuk=; b=ZuLWh71BL1EJNOFSk/gid87c2vx7R3bxoiGdyL8PffH0ma/Utl7wwCbs0PoXeed9X5 +wE0iMkAyMv2A7yysekrh88dGErbyN7rlV9/UZc868cRe74aYPxKswDl+LsvMU4FY62+ MzcIlBLHbwuK0KcWOpWqbGle/qniXSoNZfQNX+8Rxw8yRADYaLNxbSWjN9yi38euSb0E MTgqxvu5Y9GTu3y31KssoK3YBqQLnSLVZtKTdC2jH2ORUl7Bs0wfAD6IjACaOUh8G7bo TsoUZjC50SHX66/JE82hoZZA4UKSiWJCOOKwt29gqdECLI0KQDh/7sWNEaEXkR7RlTHT R34A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706751740; x=1707356540; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=4WRKA+8oB7qFwtR1n75bKmIl1ybu1sGsFv65Z7DNMuk=; b=U8fb9UNf1hCOxXDd5X2KDpAoZJT2WaRcY8QJYIKyhaUQLJshAzgSraDz+86URiMvF4 ffZRCsA+thjJUr+n0si6AIOsy0pbk5haI6x/2koXaYSdB8f8z8Pgh70K14YRGXb6LBtl 6JWiiMlHakn03rV9CHbnJY8bgqTndXeApNmF6A9o1zUrYkqw3GdRdV0To4Pm/dZEEZgy gRh6FykbyNvPzJhwcntBl/fXrI3Tn8rMFEzG9c+jdm2DWGScd0U19eCf3l56tYMs7JEF kvBZJJL7BJYIgh4M91U0WUgVhUM8z/3tMKTC9oB6OX0+ddriUEC+O/xpanaqhLHP4vVj 9HXg== X-Gm-Message-State: AOJu0Yzltku3mK5WOrUL2olSJmBK50Ed1ah8qEFzxEKbdAnccuVF4NQM HxLp2p9cRRrYGMCVtsdgmPEq2TrizdsoAGrSYcegbYfCBOr2JVPuy2qOXRpn X-Google-Smtp-Source: AGHT+IH7gXEnhRV6dQ/hlBl1ymm90ftsMLEZXl2lXUDTf5mMK0KACFhC6A6WDgvulzQq3n9XQg8mjw== X-Received: by 2002:a05:620a:124c:b0:783:4a89:f1d9 with SMTP id a12-20020a05620a124c00b007834a89f1d9mr1233836qkl.59.1706751740697; Wed, 31 Jan 2024 17:42:20 -0800 (PST) X-Forwarded-Encrypted: i=0; AJvYcCWHG9KQtwYecuVAw3zxj3CBVPr63QVPiot25Ti3U39+yLxoXmX2tiyTEzAMYm2wos+U1uG/a36SWyhSXx0T+NMz0XuoixOz4zXYn5QMYtMo8DQ4xhy1OYv1U2EyWVgWItmvawjkFZMDd822hmOW/ZBcZ/Jho9XnKLij7ig8pGhCnA0BVGlqKr9bF/LAEq7cffAh1q4siV6BVIttYaBaO3yGpyLnd92oFQ1z3rwwN5OXA5erZ8afTzxG+DS9KoWAplGRveeVz2MY4/p1XQ7w6RHbfPMZKlkRwpS1ktIiWZeMThX0t10Mp4k3K8fC/AMFVHncaHXI6k+ApnZkS7nXhRyYiccemfoIxRslq0PxKIwc5kJvDFNf8L4q Received: from auth1-smtp.messagingengine.com (auth1-smtp.messagingengine.com. [66.111.4.227]) by smtp.gmail.com with ESMTPSA id qp15-20020a05620a388f00b007840843f3b2sm2079154qkn.18.2024.01.31.17.42.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jan 2024 17:42:20 -0800 (PST) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailauth.nyi.internal (Postfix) with ESMTP id E1BC627C005B; Wed, 31 Jan 2024 20:42:19 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute5.internal (MEProxy); Wed, 31 Jan 2024 20:42:19 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrfedutddgfeejucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvvefufffkofgjfhgggfestdekredtredttdenucfhrhhomhepuehoqhhu nhcuhfgvnhhguceosghoqhhunhdrfhgvnhhgsehgmhgrihhlrdgtohhmqeenucggtffrrg htthgvrhhnpeegleejiedthedvheeggfejveefjeejkefgveffieeujefhueeigfegueeh geeggfenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpe gsohhquhhnodhmvghsmhhtphgruhhthhhpvghrshhonhgrlhhithihqdeiledvgeehtdei gedqudejjeekheehhedvqdgsohhquhhnrdhfvghngheppehgmhgrihhlrdgtohhmsehfih igmhgvrdhnrghmvg X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 31 Jan 2024 20:42:19 -0500 (EST) From: Boqun Feng To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org Cc: Zqiang , Frederic Weisbecker , "Paul E . McKenney" , Boqun Feng , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan Subject: [PATCH 6/6] rcu/nocb: Check rdp_gp->nocb_timer in __call_rcu_nocb_wake() Date: Wed, 31 Jan 2024 17:40:58 -0800 Message-ID: <20240201014100.3204532-7-boqun.feng@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240201014100.3204532-1-boqun.feng@gmail.com> References: <20240201014100.3204532-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Zqiang Currently, only rdp_gp->nocb_timer is used, for nocb_timer of no-rdp_gp structure, the timer_pending() is always return false, this commit therefore need to check rdp_gp->nocb_timer in __call_rcu_nocb_wake(). Signed-off-by: Zqiang Reviewed-by: Frederic Weisbecker Reviewed-by: Paul E. McKenney Signed-off-by: Boqun Feng --- kernel/rcu/tree_nocb.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index ffa69a5e18f4..f124d4d45ce6 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -564,6 +564,7 @@ static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_alldone, long lazy_len; long len; struct task_struct *t; + struct rcu_data *rdp_gp = rdp->nocb_gp_rdp; // If we are being polled or there is no kthread, just leave. t = READ_ONCE(rdp->nocb_gp_kthread); @@ -608,7 +609,7 @@ static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_alldone, smp_mb(); /* Enqueue before timer_pending(). */ if ((rdp->nocb_cb_sleep || !rcu_segcblist_ready_cbs(&rdp->cblist)) && - !timer_pending(&rdp->nocb_timer)) { + !timer_pending(&rdp_gp->nocb_timer)) { rcu_nocb_unlock(rdp); wake_nocb_gp_defer(rdp, RCU_NOCB_WAKE_FORCE, TPS("WakeOvfIsDeferred"));