From patchwork Mon Feb 24 02:21:55 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boqun Feng X-Patchwork-Id: 13987314 Received: from mail-qk1-f177.google.com (mail-qk1-f177.google.com [209.85.222.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0D40024B29; Mon, 24 Feb 2025 02:22:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363741; cv=none; b=Mobu/Dazgl9M8LOunZ8l0heLyWOb/+xpMnylnfkLOVyBBKHoRUpArHTPXQfj6PkjzK+WCy3S4vTWgGnBm+LH7v2y71KTJ1vcSwbg+MaWtLkgKbOAdcZaAkG2bCvGwWWu7ZVaYQ/x1qLGv2NBw9d8zBZ36tTWEY4CO5g4Col3t98= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363741; c=relaxed/simple; bh=3hLfqkqwajj486Pe46ojiChhC3AdPfNGZK5wX465TGA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=OAhhxwLkRP4I0GQp9OH1IZO6dQbEWeRgZjJ0Ix9yX4Sj2BATJ+k5Dymvigwg0xytOqsl1dG+iOeqhYWgGtJbMZkEclp9Mk9r4AvL4K00TecTGXTtIFxoftweIr+TVy9WEBJ0PfsWtA05R1pNL8uVRuOJSigmgrKMvWpNTzZhQDU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=kHjcPRdY; arc=none smtp.client-ip=209.85.222.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="kHjcPRdY" Received: by mail-qk1-f177.google.com with SMTP id af79cd13be357-7c0a3d6a6e4so376569485a.1; Sun, 23 Feb 2025 18:22:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740363739; x=1740968539; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=mJvBAuzH2IJvhDI3I6/wuJNYmCk2oz4gPvTHRsBnqEo=; b=kHjcPRdYVefMIkwLS8brhXQY6HO22Yu75OClvgI3QUpvARTiu6ISgJa6EU2cJQp2sE qtyUGWulCLaylQ8pe8aZdUAQl75pwp0lRpdi1m27Z5b+JiPJw1OhHIDkt+F7tA9FwWGu KozfVhEaZo/LlcxxrQsHg5kxGlg570ipt2yjhfdAiflPLoFZJCgLyH2AFahS+BxdW6Gn Jy1euB4cK5gniWJIdcocat2QcEKqwgAsA695nu97Lklxay8kPX8HlPYz0RNoAkEDsVAK 5sdt8nZEVv6aPu+F8lP+XHUagcoBmIsYnQyzBG9GPcwEGsoT/H/L+/dKGX3F2z2fiJl9 q9uQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740363739; x=1740968539; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=mJvBAuzH2IJvhDI3I6/wuJNYmCk2oz4gPvTHRsBnqEo=; b=aYnpzpwVbCEVlITq4ya5RvdZkKVtuDjkBQ1T0Bx1VrvW10lZ83PhbKkZEU8nDmOwXa YsIViGBvYmPjUOji7+YIvS7uTj9x6us3nX3xyAMm3ua01vg5/X+VUVCoZifXoDKsq5jz 0BZq0klQ5Ib4atW8akH3HI8I/FhEVrbCUEfOmZTq4D5rreVHTlf/08hINzgnqr8UIdaS sa4K0ELjylVGtEwzwW5Qu+/lB7G0+WkQRK22LVes2rz6NHpxdUVkBCfoVy466OlrqVs9 s3Hhuu2U5RC+4PGbE4rnWrDjljsoETCRXfqATO86X5vwpj/G8nDHWLCZ6QNmR+QwJoU/ bzUg== X-Forwarded-Encrypted: i=1; AJvYcCUoe/v2ht5Tt1AWx0gXyCsleASIZoh8SmbSDMMX8Oga81ICrVgegYmMQUWNluz0e4rsT8ms2vVCaorpoMm1@vger.kernel.org, AJvYcCVG4yAkWKkpJHGyIVFEn9QpDJQE2SrB2czii/Yg/2L0SxjlPMG03K54q2WiYqaXfQIlcto=@vger.kernel.org, AJvYcCXLIPVMaGYsOhkDjVv3B+ucTErkNFvyU25h/ig+82nb/W1hiq//mEKtdJNT4PmLv0SStQ0iYsrZnHjZugrJJgon@vger.kernel.org X-Gm-Message-State: AOJu0YwVTSeAAvLFsozvPFEC1d7jU0oacjdCylz5Vo6Nqh/QRsArwglG RHc0U7/J6bRBIMtqVRTdCtTzsxdVYj99yWlkFHbOMwCkEatZSeMC X-Gm-Gg: ASbGncsjbWgtSpK6MMJXGGdE7w+45BO5zddFsB67biDE/qoCxZ38/SOye9IRNFI5P+5 RYoTcag8z1qpAl80ouM2tUW4AbWNUsyfFqfa/zgZuIKyEYERlnAFx68P2qViOZQJdwJgLsTV7L0 WBRmpiRwdjEt/ldRirkPVVZvwintVPzHw1D7S9ZiX5XN8n+7AEKHacsdf6fL28UQyPQUXg5J08k xctEglUUc4c80yZ2ZQTqCOR+T9E0ntX6ek8oEZMktb1NlSIa2pVVLMgILTowN9i0pyg2ihJgkwy xRt7FwN+VRfvJ1akKnYG2hviaBrVBNcmuX8ayoJ6K6IqQMOOEi3dmS+5bS1oK3SpL5OPdk039Ki cd96u58CKgzbAW9n/ X-Google-Smtp-Source: AGHT+IFUBJ/19SzIMd4phxmUKiWctiSDZpRIqcsMy7b55y/0AHhQ+k0TqMz944j7Fg4RJXmaFX+y3Q== X-Received: by 2002:a05:620a:24c4:b0:7c0:b185:a952 with SMTP id af79cd13be357-7c0ceee6234mr1410812585a.2.1740363738851; Sun, 23 Feb 2025 18:22:18 -0800 (PST) Received: from fauth-a2-smtp.messagingengine.com (fauth-a2-smtp.messagingengine.com. [103.168.172.201]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7c0b0d81b2fsm744810485a.84.2025.02.23.18.22.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 23 Feb 2025 18:22:18 -0800 (PST) Received: from phl-compute-01.internal (phl-compute-01.phl.internal [10.202.2.41]) by mailfauth.phl.internal (Postfix) with ESMTP id EBDA81200043; Sun, 23 Feb 2025 21:22:17 -0500 (EST) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-01.internal (MEProxy); Sun, 23 Feb 2025 21:22:17 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgdejjeehgecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvfevufffkffojghfggfgsedtkeertdertddt necuhfhrohhmpeeuohhquhhnucfhvghnghcuoegsohhquhhnrdhfvghnghesghhmrghilh drtghomheqnecuggftrfgrthhtvghrnhepgeeljeeitdehvdehgefgjeevfeejjeekgfev ffeiueejhfeuiefggeeuheeggefgnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrg hmpehmrghilhhfrhhomhepsghoqhhunhdomhgvshhmthhprghuthhhphgvrhhsohhnrghl ihhthidqieelvdeghedtieegqddujeejkeehheehvddqsghoqhhunhdrfhgvnhhgpeepgh hmrghilhdrtghomhesfhhigihmvgdrnhgrmhgvpdhnsggprhgtphhtthhopedvvddpmhho uggvpehsmhhtphhouhhtpdhrtghpthhtoheprhgtuhesvhhgvghrrdhkvghrnhgvlhdroh hrghdprhgtphhtthhopehjihgrnhhgshhhrghnlhgrihesghhmrghilhdrtghomhdprhgt phhtthhopehprghulhhmtghksehkvghrnhgvlhdrohhrghdprhgtphhtthhopehjohhshh esjhhoshhhthhrihhplhgvthhtrdhorhhgpdhrtghpthhtoheprhhoshhtvgguthesghho ohgumhhishdrohhrghdprhgtphhtthhopehmrghthhhivghurdguvghsnhhohigvrhhsse gvfhhfihgtihhoshdrtghomhdprhgtphhtthhopehfrhgvuggvrhhitgeskhgvrhhnvghl rdhorhhgpdhrtghpthhtohepnhgvvghrrghjrdhuphgrughhhigrhieskhgvrhhnvghlrd horhhgpdhrtghpthhtohepjhhovghlsehjohgvlhhfvghrnhgrnhguvghsrdhorhhg X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 23 Feb 2025 21:22:17 -0500 (EST) From: Boqun Feng To: rcu@vger.kernel.org Cc: Lai Jiangshan , "Paul E. McKenney" , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Uladzislau Rezki , Zqiang , Davidlohr Bueso , Shuah Khan , linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, bpf@vger.kernel.org, Ankur Arora , Alexei Starovoitov , Andrii Nakryiko , Peter Zijlstra , Kent Overstreet Subject: [PATCH rcu 01/20] srcu: Make Tiny SRCU able to operate in preemptible kernels Date: Sun, 23 Feb 2025 18:21:55 -0800 Message-Id: <20250224022214.12037-2-boqun.feng@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250224022214.12037-1-boqun.feng@gmail.com> References: <20250224022214.12037-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" Given that SRCU allows its read-side critical sections are not just preemptible, but also allow general blocking, there is not much reason to restrict Tiny SRCU to non-preemptible kernels. This commit therefore removes Tiny SRCU dependencies on non-preemptibility, primarily surrounding its interaction with rcutorture and early boot. Signed-off-by: Paul E. McKenney Cc: Ankur Arora Cc: Alexei Starovoitov Cc: Andrii Nakryiko Cc: Peter Zijlstra Cc: Kent Overstreet Cc: Signed-off-by: Boqun Feng --- kernel/rcu/rcu.h | 9 ++++++--- kernel/rcu/srcutiny.c | 6 ++++++ 2 files changed, 12 insertions(+), 3 deletions(-) diff --git a/kernel/rcu/rcu.h b/kernel/rcu/rcu.h index feb3ac1dc5d5..2909662c805f 100644 --- a/kernel/rcu/rcu.h +++ b/kernel/rcu/rcu.h @@ -611,8 +611,6 @@ void srcutorture_get_gp_data(struct srcu_struct *sp, int *flags, static inline bool rcu_watching_zero_in_eqs(int cpu, int *vp) { return false; } static inline unsigned long rcu_get_gp_seq(void) { return 0; } static inline unsigned long rcu_exp_batches_completed(void) { return 0; } -static inline unsigned long -srcu_batches_completed(struct srcu_struct *sp) { return 0; } static inline void rcu_force_quiescent_state(void) { } static inline bool rcu_check_boost_fail(unsigned long gp_state, int *cpup) { return true; } static inline void show_rcu_gp_kthreads(void) { } @@ -624,7 +622,6 @@ static inline void rcu_gp_slow_unregister(atomic_t *rgssp) { } bool rcu_watching_zero_in_eqs(int cpu, int *vp); unsigned long rcu_get_gp_seq(void); unsigned long rcu_exp_batches_completed(void); -unsigned long srcu_batches_completed(struct srcu_struct *sp); bool rcu_check_boost_fail(unsigned long gp_state, int *cpup); void show_rcu_gp_kthreads(void); int rcu_get_gp_kthreads_prio(void); @@ -636,6 +633,12 @@ void rcu_gp_slow_register(atomic_t *rgssp); void rcu_gp_slow_unregister(atomic_t *rgssp); #endif /* #else #ifdef CONFIG_TINY_RCU */ +#ifdef CONFIG_TINY_SRCU +static inline unsigned long srcu_batches_completed(struct srcu_struct *sp) { return 0; } +#else // #ifdef CONFIG_TINY_SRCU +unsigned long srcu_batches_completed(struct srcu_struct *sp); +#endif // #else // #ifdef CONFIG_TINY_SRCU + #ifdef CONFIG_RCU_NOCB_CPU void rcu_bind_current_to_nocb(void); #else diff --git a/kernel/rcu/srcutiny.c b/kernel/rcu/srcutiny.c index 4dcbf8aa80ff..2a94f0e65606 100644 --- a/kernel/rcu/srcutiny.c +++ b/kernel/rcu/srcutiny.c @@ -20,7 +20,11 @@ #include "rcu_segcblist.h" #include "rcu.h" +#ifndef CONFIG_TREE_RCU int rcu_scheduler_active __read_mostly; +#else // #ifndef CONFIG_TREE_RCU +extern int rcu_scheduler_active; +#endif // #else // #ifndef CONFIG_TREE_RCU static LIST_HEAD(srcu_boot_list); static bool srcu_init_done; @@ -282,11 +286,13 @@ bool poll_state_synchronize_srcu(struct srcu_struct *ssp, unsigned long cookie) } EXPORT_SYMBOL_GPL(poll_state_synchronize_srcu); +#ifndef CONFIG_TREE_RCU /* Lockdep diagnostics. */ void __init rcu_scheduler_starting(void) { rcu_scheduler_active = RCU_SCHEDULER_RUNNING; } +#endif // #ifndef CONFIG_TREE_RCU /* * Queue work for srcu_struct structures with early boot callbacks. From patchwork Mon Feb 24 02:21:56 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boqun Feng X-Patchwork-Id: 13987315 Received: from mail-qk1-f174.google.com (mail-qk1-f174.google.com [209.85.222.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 67E7C1A08B8; Mon, 24 Feb 2025 02:22:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363743; cv=none; b=sE/aa19PIn2xX4BNbLXxcsQWmuaVVsGt8vluoHiQiTBkmzmwATpQo2NI0O+RfOjJYifIEDaJ2J09tHH2hQJ28h90cfoFZDlmyFRFSDvtmR4oFXc8+OBkeE+v6zSpha73tBwyUX+l2fKrkWyMna+WFnedEBiO9bOR7QTqSGveVMs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363743; c=relaxed/simple; bh=Tl0eobmzKFR8IE72hC7Onl6yHZ8FilaP/b+maBCQA8E=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Fz3Cw7M3HRDuh/NQY+6sTHE0wUTcQxJlnIRlyZIsO1wru7weTEzPJNIO747q4c20yWmncHzuRabkFCP8Thwe+yMaUEXOFGnRCT7AOq5THXe6CjUNoRPyv5GMEWmRX7pRPy55V4T6vP81MYWxukfxiHF31wBSWvDPaIjx1GbD3BM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=MltGf9+X; arc=none smtp.client-ip=209.85.222.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="MltGf9+X" Received: by mail-qk1-f174.google.com with SMTP id af79cd13be357-7c0892e4b19so484888585a.3; Sun, 23 Feb 2025 18:22:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740363740; x=1740968540; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=D1poumkMbwz8WUDhgcTGwAxfP5yNlPlZcTKSuNaDv0I=; b=MltGf9+XcGiqNjM1BPFni22Upfufkq6Vi0EDWNs4hfHR/vYGWf/Bdjb61sYsj8gwaV JPpgaYEn62qrEmfkDWxrGyyXyjdyDVchqZyt49mm+7JMaKqJ8BBm5w48Pb95J7LChrvv iExHsbmZUJqE792IKfdVxrLccq8c9hfqojX55iQYadYfy+vMvQhWMqOaOTTDrL+boZ2x Ao4F4A9Mt4DhVZ5DgfXloIVaw9Vz48smlviMSgwbYvuUh1C7cBe+nIVd0iHUsrvhNyqs pMW9N468LBvESiosj9gibkj/xRmptqjpTohHXRe8mnpo5H/2rrzX/izC+PCGC3CN4n/Z TQxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740363740; x=1740968540; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=D1poumkMbwz8WUDhgcTGwAxfP5yNlPlZcTKSuNaDv0I=; b=LadNns5EwlcH2KWeG96IfZMAJ2QSdmMtbkDzhxs4n9qE51/xDa/HPElYsAjaZUTLEE Xv5/8DLHfHHHUCFeHKhA6eTlP6mQeBdO8G73/hkrXgV/XKPSSeTIVyN5zovIEJM4fHS4 ShjD0GP+swMEZ0CuTeBuxCziMVLwqU4Tt7fFoEe5EGuwPEGqrb89bE7JCnn6QPZyKm4I AZSJUIVsa2f50xagswtbAmNK3Y8qJ/yn0Zr/DVuOLHDioDCO37MVfBppnzGUr8FtnS5b vLXGOZAa+vBBGeBz0gB97sjVOWP9jNZfwUAW3apvwSS+r0B6dOWzgVVCC57LFk3EKNf8 fRoQ== X-Forwarded-Encrypted: i=1; AJvYcCVWhhEPPjF98DI4O5iN/q4s6T8Tm2Cb2QZj6dR8lqS+Vk/c3xkYbZ2Ty+K1/HU8Ixm+O78=@vger.kernel.org, AJvYcCVaq+sMkKIV15Q0rCrFFyo4kWz/StQ9jsmwNe1I7uiNjc40eEakjO+Bl9abJK2q2bQ0sQrmK8yglHEEAnRo@vger.kernel.org, AJvYcCWVkl8HGdMyo0iVP9YGHPjT5lazk10VGBAygMWze8L7SuAvTb4SwIY0MTT+pOkj3Dtyu6Zz0iIleQfr2v+L9NjH@vger.kernel.org X-Gm-Message-State: AOJu0YyyaNXKFbowWH+NJxEeoBctGe2aWb6RvPuB43wasS+vIZxyX1yL sm+ePgCeeiJ13sq1Xysg3p3fjMBqL/TMtDYDdi9lOrf9mAo6FHqs X-Gm-Gg: ASbGncvooaTYHaEj5f8OYWsBQcy2R94dtG9/8Kx9sh4lL6Ng7x43XGLQo4LK++2lqiX xdiLk9xknXda/6Dcu36hmPiUCAYt3+Goy0D2armR4NyZR+8yijc2ixLfEpbaBzZZrGfw3f7gB9p /nEo7W4jeq3gXMc9xoklVNpfD74kBBHfuleAVHyIvLYOIb4iH76NjgSmtEFNUjoiRqHwQf+oUSa XsTVLLJ6A3SHLZZWj7N0wdlWOX3Ua/QbruO4IDA4zzHErUJ8onauExqO419kaqqloj7UgWRKyqG ZckyIBV+TEc/p6oR5H8wqdIQFLBMeqH7mecpvEzGXJsIJGUbiSxYVfbpye8N6KbnE0LQUInwGSN rdIfjcGJdzn3xTdGM X-Google-Smtp-Source: AGHT+IEbG51k5BQfXyudC0joXeQ0yrs56uBl53PyH/gx5UidLvBI1pZnngtdFjGKW1kJtA5lDdnEOQ== X-Received: by 2002:a05:620a:4113:b0:7c0:a389:f23c with SMTP id af79cd13be357-7c0cef7a394mr1888342285a.49.1740363740269; Sun, 23 Feb 2025 18:22:20 -0800 (PST) Received: from fauth-a2-smtp.messagingengine.com (fauth-a2-smtp.messagingengine.com. [103.168.172.201]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7c0a609cb1bsm856081785a.111.2025.02.23.18.22.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 23 Feb 2025 18:22:19 -0800 (PST) Received: from phl-compute-12.internal (phl-compute-12.phl.internal [10.202.2.52]) by mailfauth.phl.internal (Postfix) with ESMTP id 58A4F1200043; Sun, 23 Feb 2025 21:22:19 -0500 (EST) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-12.internal (MEProxy); Sun, 23 Feb 2025 21:22:19 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgdejjeehgecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvfevufffkffojghfggfgsedtkeertdertddt necuhfhrohhmpeeuohhquhhnucfhvghnghcuoegsohhquhhnrdhfvghnghesghhmrghilh drtghomheqnecuggftrfgrthhtvghrnhepgeeljeeitdehvdehgefgjeevfeejjeekgfev ffeiueejhfeuiefggeeuheeggefgnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrg hmpehmrghilhhfrhhomhepsghoqhhunhdomhgvshhmthhprghuthhhphgvrhhsohhnrghl ihhthidqieelvdeghedtieegqddujeejkeehheehvddqsghoqhhunhdrfhgvnhhgpeepgh hmrghilhdrtghomhesfhhigihmvgdrnhgrmhgvpdhnsggprhgtphhtthhopedvvddpmhho uggvpehsmhhtphhouhhtpdhrtghpthhtoheprhgtuhesvhhgvghrrdhkvghrnhgvlhdroh hrghdprhgtphhtthhopehjihgrnhhgshhhrghnlhgrihesghhmrghilhdrtghomhdprhgt phhtthhopehprghulhhmtghksehkvghrnhgvlhdrohhrghdprhgtphhtthhopehjohhshh esjhhoshhhthhrihhplhgvthhtrdhorhhgpdhrtghpthhtoheprhhoshhtvgguthesghho ohgumhhishdrohhrghdprhgtphhtthhopehmrghthhhivghurdguvghsnhhohigvrhhsse gvfhhfihgtihhoshdrtghomhdprhgtphhtthhopehfrhgvuggvrhhitgeskhgvrhhnvghl rdhorhhgpdhrtghpthhtohepnhgvvghrrghjrdhuphgrughhhigrhieskhgvrhhnvghlrd horhhgpdhrtghpthhtohepjhhovghlsehjohgvlhhfvghrnhgrnhguvghsrdhorhhg X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 23 Feb 2025 21:22:18 -0500 (EST) From: Boqun Feng To: rcu@vger.kernel.org Cc: Lai Jiangshan , "Paul E. McKenney" , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Uladzislau Rezki , Zqiang , Davidlohr Bueso , Shuah Khan , linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, bpf@vger.kernel.org, Neeraj Upadhyay , Alexei Starovoitov , Andrii Nakryiko , Peter Zijlstra , Kent Overstreet Subject: [PATCH rcu 02/20] srcu: Define SRCU_READ_FLAVOR_ALL in terms of symbols Date: Sun, 23 Feb 2025 18:21:56 -0800 Message-Id: <20250224022214.12037-3-boqun.feng@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250224022214.12037-1-boqun.feng@gmail.com> References: <20250224022214.12037-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" This commit defines SRCU_READ_FLAVOR_ALL in terms of the SRCU_READ_FLAVOR_* definitions instead of a hexadecimal constant. Suggested-by: Neeraj Upadhyay Signed-off-by: Paul E. McKenney Cc: Alexei Starovoitov Cc: Andrii Nakryiko Cc: Peter Zijlstra Cc: Kent Overstreet Cc: Signed-off-by: Boqun Feng --- include/linux/srcu.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/include/linux/srcu.h b/include/linux/srcu.h index d7ba46e74f58..f6f779b9d9ff 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -47,7 +47,8 @@ int init_srcu_struct(struct srcu_struct *ssp); #define SRCU_READ_FLAVOR_NORMAL 0x1 // srcu_read_lock(). #define SRCU_READ_FLAVOR_NMI 0x2 // srcu_read_lock_nmisafe(). #define SRCU_READ_FLAVOR_LITE 0x4 // srcu_read_lock_lite(). -#define SRCU_READ_FLAVOR_ALL 0x7 // All of the above. +#define SRCU_READ_FLAVOR_ALL (SRCU_READ_FLAVOR_NORMAL | SRCU_READ_FLAVOR_NMI | \ + SRCU_READ_FLAVOR_LITE) // All of the above. #ifdef CONFIG_TINY_SRCU #include From patchwork Mon Feb 24 02:21:57 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boqun Feng X-Patchwork-Id: 13987316 Received: from mail-qk1-f181.google.com (mail-qk1-f181.google.com [209.85.222.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D3BB21A3153; Mon, 24 Feb 2025 02:22:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363744; cv=none; b=jikj8cgYZJoqdkvdmCfJrpg/hznI/tZRUAFCN4uN0Tj8/y0G+mTeHwiXhy8eHxS6okbynSRXx5/t3rAD7oQfJZBjIyEqqbLZjC7uwH3KedE4hOi0wxU6OdpznmkadAucH44YF2qop5g0KJitQAfSDVLcs7seMHcs3Fa0BRXYAJ4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363744; c=relaxed/simple; bh=IWWFvnRQmBN9XFMhXfPj63PsErr3J9ivkWHm1D9uPTY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=MFkseZgb/NkDQTBF8olR4dJbESt5l6irT9HjeBkYC1beUgZYs3fjo1D0IazzWTl5oacjfAUQJKst35MXx6amQ6rS6G791VVSBN6lmW1Z+uiFQYiG8K+zJn7sWjBjI0KtK8JGEpExMhwmVtC4QtBmQpHVUdXKOPhdFeF8iVvsjnE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=bDUqj5aA; arc=none smtp.client-ip=209.85.222.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="bDUqj5aA" Received: by mail-qk1-f181.google.com with SMTP id af79cd13be357-7c0818add57so411383285a.3; Sun, 23 Feb 2025 18:22:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740363741; x=1740968541; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=DdxxxEQunYArASFbJzD0EbW0TcJj5v5m/sYqOFYiFGc=; b=bDUqj5aANHVi929lFyjp9T+NDkv3FrukazGK17egG6gnR/rISjnGGAclRSxq/yVgwG GbRiEZhrgNwB8kFQ6dR59TyPL0GV4vL/KzDIjfAtBYrts7UYxxux2Y4uuK4Jr3b3jWXu Mx4tc/g/OOBN6oiGWaoovxZk/qOT0n5W36xhuSm70NcF9NCN9aZ577GV4QIioxqorT1y /6VW3Anh3UgVH4qIkq3aMkJcsv4BkrtxDpDWfb/pyNQcTi3swkzI9ixIOKgX0EWkRPs7 Mi7W5gSh8u7a+uQEonRZwcULoWR2Goa+7wl1bf0Pjcur4T+xs3R+kAbRjiS/pRvG/5bJ 4OVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740363741; x=1740968541; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=DdxxxEQunYArASFbJzD0EbW0TcJj5v5m/sYqOFYiFGc=; b=sP+4qufvSmM7Eakhi6OOpwLkGMZFGWSH927A54iR1X5FUZbh1AMCuxJ4ivUCgTqT5j Is+aEKx9p06TMBe5Z+8coXk4w2TEr3YazNNDLzF/JEpLCID4lj/Gkv6CUenevBf9AcnI FXxqrxdVOo+LhL38P6eYFmYYgCwKR83uOjicZlIYPsXOXBHIoeJmBqrkcWsIkiDR6CW9 ioxMCAtyMNOUC+57pA4XTRhDiuoiwDkawrFE+3hny2cTbhET+PUyoJVYY7O0t2vpS51B FUw69bsXo8U5fqCJtz61QW+8cnQ8PyttLdc/QBf8TvH3geTSllTTE3Z09+QGyirQtlsM 5MQw== X-Forwarded-Encrypted: i=1; AJvYcCWm4gzH6lHBsi/bkGyBX0YopDk/SgvgrQP9q9NCvWICu2xdvGqT7rZeWpkNmNJ7DiHW8Og=@vger.kernel.org, AJvYcCWqv6t2sOCzPqdBlUTyxZxY/xyN7JRoM635Y4ydSp5z/Jlm0n6X03c3QQVgo6ghVhhTwXf5En7h7kAYz3z3JIF4@vger.kernel.org, AJvYcCXxW6D4Om1IltbdxDsLusmIw1Vg7poYLs9ZX7VKzvKy5hTxXESSj3F+7RlZETFd4Y+XOkvSSDqSLG5M/P9i@vger.kernel.org X-Gm-Message-State: AOJu0YweftAajX/YZptu4UWfYxskVbbGhbvz2FoYFJuOmhtdRQdGDJm1 gD+dytR5FK8zCX7UhC9SW24PjvCJTY45LRfkiinql7SwFBc3UMOZ X-Gm-Gg: ASbGnctbWJxF5fDsHIT5SvVqrLd4nD6803gr1MqcgXdNvXL4zcT494l3N346LcaX+Hq 9K/33h5TrSC2lXP2J9CZj6AttbwGmUAM/YB1oFgYhqksSz4ntuBcHHuveVK6a4Rn86UXUzEAezp F5TWXUGA6siGf0Ig/qC8YnAhM0hw/3pBXDv72RuTmJq66Pja+ymOm6tI/Av3f7Oa8h8e0JQTWaO GOEEG+UoZKfXxgkbASt8FKOIJRRqJnehxuKiiz1i86Y0OxFg0motlt+R7AomnmIYUwj60+D9PFa in3sVyhn0lcvGOwNCfDxN3UJ1ChzosN/jn3JiPdO471mNtWxwMEGLixOVv4xmwQOXaarD9Gri3Z E5AEkMk1h4tID0yaZ X-Google-Smtp-Source: AGHT+IEDsAUAVip8lsPccMpgEXsuo884vRHK3H4G+8C+AfawM9a3nx+CeNFx3jftdkE9f5tmKS5q2g== X-Received: by 2002:a05:620a:f14:b0:7c2:2a54:8810 with SMTP id af79cd13be357-7c22a54882dmr408863385a.23.1740363741631; Sun, 23 Feb 2025 18:22:21 -0800 (PST) Received: from fauth-a2-smtp.messagingengine.com (fauth-a2-smtp.messagingengine.com. [103.168.172.201]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7c0a3dcda55sm879624685a.58.2025.02.23.18.22.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 23 Feb 2025 18:22:21 -0800 (PST) Received: from phl-compute-08.internal (phl-compute-08.phl.internal [10.202.2.48]) by mailfauth.phl.internal (Postfix) with ESMTP id B80881200043; Sun, 23 Feb 2025 21:22:20 -0500 (EST) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-08.internal (MEProxy); Sun, 23 Feb 2025 21:22:20 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgdejjeehgecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvfevufffkffojghfggfgsedtkeertdertddt necuhfhrohhmpeeuohhquhhnucfhvghnghcuoegsohhquhhnrdhfvghnghesghhmrghilh drtghomheqnecuggftrfgrthhtvghrnhepgeeljeeitdehvdehgefgjeevfeejjeekgfev ffeiueejhfeuiefggeeuheeggefgnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrg hmpehmrghilhhfrhhomhepsghoqhhunhdomhgvshhmthhprghuthhhphgvrhhsohhnrghl ihhthidqieelvdeghedtieegqddujeejkeehheehvddqsghoqhhunhdrfhgvnhhgpeepgh hmrghilhdrtghomhesfhhigihmvgdrnhgrmhgvpdhnsggprhgtphhtthhopedvuddpmhho uggvpehsmhhtphhouhhtpdhrtghpthhtoheprhgtuhesvhhgvghrrdhkvghrnhgvlhdroh hrghdprhgtphhtthhopehjihgrnhhgshhhrghnlhgrihesghhmrghilhdrtghomhdprhgt phhtthhopehprghulhhmtghksehkvghrnhgvlhdrohhrghdprhgtphhtthhopehjohhshh esjhhoshhhthhrihhplhgvthhtrdhorhhgpdhrtghpthhtoheprhhoshhtvgguthesghho ohgumhhishdrohhrghdprhgtphhtthhopehmrghthhhivghurdguvghsnhhohigvrhhsse gvfhhfihgtihhoshdrtghomhdprhgtphhtthhopehfrhgvuggvrhhitgeskhgvrhhnvghl rdhorhhgpdhrtghpthhtohepnhgvvghrrghjrdhuphgrughhhigrhieskhgvrhhnvghlrd horhhgpdhrtghpthhtohepjhhovghlsehjohgvlhhfvghrnhgrnhguvghsrdhorhhg X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 23 Feb 2025 21:22:20 -0500 (EST) From: Boqun Feng To: rcu@vger.kernel.org Cc: Lai Jiangshan , "Paul E. McKenney" , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Uladzislau Rezki , Zqiang , Davidlohr Bueso , Shuah Khan , linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, bpf@vger.kernel.org, Alexei Starovoitov , Andrii Nakryiko , Peter Zijlstra , Kent Overstreet Subject: [PATCH rcu 03/20] srcu: Use ->srcu_gp_seq for rcutorture reader batch Date: Sun, 23 Feb 2025 18:21:57 -0800 Message-Id: <20250224022214.12037-4-boqun.feng@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250224022214.12037-1-boqun.feng@gmail.com> References: <20250224022214.12037-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" This commit stops using ->srcu_idx for rcutorture's reader-batch consistency checking, using ->srcu_gp_seq instead. This is a first step towards a faster srcu_read_{,un}lock_lite() that avoids the array accesses that use ->srcu_idx. Signed-off-by: Paul E. McKenney Cc: Alexei Starovoitov Cc: Andrii Nakryiko Cc: Peter Zijlstra Cc: Kent Overstreet Cc: Signed-off-by: Boqun Feng --- kernel/rcu/rcutorture.c | 2 ++ kernel/rcu/srcutree.c | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c index d26fb1d33ed9..1d2de50fb5d6 100644 --- a/kernel/rcu/rcutorture.c +++ b/kernel/rcu/rcutorture.c @@ -791,6 +791,7 @@ static struct rcu_torture_ops srcu_ops = { .readunlock = srcu_torture_read_unlock, .readlock_held = torture_srcu_read_lock_held, .get_gp_seq = srcu_torture_completed, + .gp_diff = rcu_seq_diff, .deferred_free = srcu_torture_deferred_free, .sync = srcu_torture_synchronize, .exp_sync = srcu_torture_synchronize_expedited, @@ -834,6 +835,7 @@ static struct rcu_torture_ops srcud_ops = { .readunlock = srcu_torture_read_unlock, .readlock_held = torture_srcu_read_lock_held, .get_gp_seq = srcu_torture_completed, + .gp_diff = rcu_seq_diff, .deferred_free = srcu_torture_deferred_free, .sync = srcu_torture_synchronize, .exp_sync = srcu_torture_synchronize_expedited, diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index b83c74c4dcc0..e69ce9d59abf 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -1675,7 +1675,7 @@ EXPORT_SYMBOL_GPL(srcu_barrier); */ unsigned long srcu_batches_completed(struct srcu_struct *ssp) { - return READ_ONCE(ssp->srcu_idx); + return READ_ONCE(ssp->srcu_sup->srcu_gp_seq); } EXPORT_SYMBOL_GPL(srcu_batches_completed); From patchwork Mon Feb 24 02:21:58 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boqun Feng X-Patchwork-Id: 13987317 Received: from mail-qt1-f172.google.com (mail-qt1-f172.google.com [209.85.160.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 57F811A7253; Mon, 24 Feb 2025 02:22:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363746; cv=none; b=K/t5bf10PlHWwWSKPyykL3f9zGTe/JQeJ1YG+KewgotueeOpWYgKOJFhOvp25dwJJ5ICwB+Xuh4EFbA7ce32snpByY41hIszpe5Wg1tKGlnn7BLDYpsTIr7h3frzM2NYlLV8/Qqdd2TRTnOgbh6t+sM2aagzaRq+n4ksnT9PeqA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363746; c=relaxed/simple; bh=wANPTy97bY/zTdpU+z3ZCDuJvLVgmYnNsjBC6Zp8iT4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=an1i3JkNjHodotAUaMID8IFFgcpv3wwRV5yS9VX6la00eQ89P6ru8dTYMcuCfwtG1DHyVJI/yxblZ0c/8gFMtLra8ZjGNIuk7E4Zf4eNILaD8BCNhK+EUf95EXWDh6cCnLe6oWf9KnLy3UAIl9ADDcAADcBwTlhiSsIiTyZgeNA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=V8ZQi1vX; arc=none smtp.client-ip=209.85.160.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="V8ZQi1vX" Received: by mail-qt1-f172.google.com with SMTP id d75a77b69052e-471fabc5bf5so22118501cf.3; Sun, 23 Feb 2025 18:22:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740363743; x=1740968543; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=qOLMEk+pK7yaifZ1uynRQqEOYQw0oxUKxlJ/oKqOFnU=; b=V8ZQi1vXaLPxirPuDrtavSTG2HOBhbfCAW00ECngBvTufAhCBT6O+L58jy2yrVQHld Gc79KMaOSmAsBzWJ6B8jjyJyFnNyPctW9IiNjNV6Hk3X/PM9G0SARpLUxPSglK7W4pCz bWe8wQ6ZyVKNbq7Dp1YKNACQypHiXwP/Qlm7kg+z+yf0T7eyp3nWeosqDx/JUlfFnP+1 7s+Iy27xrvaazxVQ1WIptU3tA4in+7T5pytxdt9WAMqJ9/cFuMqAEJlH1pMgz0wfw5vc tFz5ndHwVDwiizdRxLP1ykSGKGNSoaPNXHY6ynOtQpaRnBysC3e6wk8cmml1s2B+lNSB 21MA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740363743; x=1740968543; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=qOLMEk+pK7yaifZ1uynRQqEOYQw0oxUKxlJ/oKqOFnU=; b=LeU4av4pworWOsjijcpFWNhjFe1pLiK7Qv0ivj3Lvc+xKXsEzY4vQslG4iXpIh4JHp lXQj0cYF/38V4AvnKC5eiSqlAHmyW3k1bYbNsxzVyGZ5Pt3eKHvlk9X7eNxm5j4JpHLe tuRb9hMUY1EKeBRor36q/o40wHOG/XTTMYlSXg4UYZ83yhnuhLn+3wZ2ucKhIcOiXK0G /+7qrqqhQkUicxS+9eAteqF6taemPShTs1NeonyxLM52KAG2zOuJ3IhCFWQH2E0y83Vs jzB7tJ+F4a/z1awzJpR8LgyPBJKoFssuFHhPgLmNtSxgTEJgERZ31XhbDJBgAB1zpcqJ kBhA== X-Forwarded-Encrypted: i=1; AJvYcCUkD1depFC10FcWeGdkQVZ2zXJ6vg1vzCsGJQFkfySNMMVBMFYnJdDYIx/utwVJyKocpj98yKgvm3h/9WH9CpGU@vger.kernel.org, AJvYcCVTG0XxA+RJwbJ1SekjDI9BA0yFZJXoLiLN6huaJwmkaxTENWiEzE0GRikgNEMe+VmWZdU=@vger.kernel.org, AJvYcCVUZLeVZu5riZRluogMryASgRa5BwZMPu6GFnRvbZoBt6RpelT+hsgLJUKDsu+WJWZDpH5HBuqkQttKr4iI@vger.kernel.org X-Gm-Message-State: AOJu0YxxIck14CiLslo8iMknwKUJ3EMZtwCRuv9JLtsQGleEizGWSnTO YrDtw43kY+IxSx0/81OZhX0BLrpQpTz26+nstFLc/rwNPCZLedGo X-Gm-Gg: ASbGnctDeUEF3OobbIquwTqrPkLejXtL0KtLKrsiuhcyCfslg3S9ekZsm0KOA+Ac14C uxIijixTK0wTuoL5PwBNYTR/uOoR91iVCl+g/pSjwprw/MY0qrL1FUUo2RFX+zVYBVsgPzAF28n vgaIkOGbS54IozVSGivEdwbrHOT0kEpalzyFb0UX0lYEkr9e/+k0oRQh7kYiF/nueyMrP+WYoL9 8SBiN8L2R28Uxg5MQtmzjhDUo6/8CvaWtOTwW7vxHoWqrNTiOj34ziWFrQC4maZqEEq1QzpP7Os gsOT4xlYwnXqoe4M98R9YMziZ357tJt3CYRHNn8gZItslq7YDMhXwmB2rGQXd2cbqB+IViLFlzr H6ZDUWn0pIHGtmpOR X-Google-Smtp-Source: AGHT+IFx4DEUzY8iXU9N91XMG8aD4j9QaEqr6DyvWhkoV6w4KF1Me5GNMAhFv0AUVhGfqJH7aUxvhQ== X-Received: by 2002:a05:622a:2d2:b0:471:a523:6ac1 with SMTP id d75a77b69052e-472228a7e9bmr150011431cf.6.1740363743099; Sun, 23 Feb 2025 18:22:23 -0800 (PST) Received: from fauth-a2-smtp.messagingengine.com (fauth-a2-smtp.messagingengine.com. [103.168.172.201]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-471ed70947fsm89446881cf.14.2025.02.23.18.22.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 23 Feb 2025 18:22:22 -0800 (PST) Received: from phl-compute-01.internal (phl-compute-01.phl.internal [10.202.2.41]) by mailfauth.phl.internal (Postfix) with ESMTP id 166D91200043; Sun, 23 Feb 2025 21:22:22 -0500 (EST) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-01.internal (MEProxy); Sun, 23 Feb 2025 21:22:22 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgdejjeehgecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvfevufffkffojghfggfgsedtkeertdertddt necuhfhrohhmpeeuohhquhhnucfhvghnghcuoegsohhquhhnrdhfvghnghesghhmrghilh drtghomheqnecuggftrfgrthhtvghrnhepgeeljeeitdehvdehgefgjeevfeejjeekgfev ffeiueejhfeuiefggeeuheeggefgnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrg hmpehmrghilhhfrhhomhepsghoqhhunhdomhgvshhmthhprghuthhhphgvrhhsohhnrghl ihhthidqieelvdeghedtieegqddujeejkeehheehvddqsghoqhhunhdrfhgvnhhgpeepgh hmrghilhdrtghomhesfhhigihmvgdrnhgrmhgvpdhnsggprhgtphhtthhopedvuddpmhho uggvpehsmhhtphhouhhtpdhrtghpthhtoheprhgtuhesvhhgvghrrdhkvghrnhgvlhdroh hrghdprhgtphhtthhopehjihgrnhhgshhhrghnlhgrihesghhmrghilhdrtghomhdprhgt phhtthhopehprghulhhmtghksehkvghrnhgvlhdrohhrghdprhgtphhtthhopehjohhshh esjhhoshhhthhrihhplhgvthhtrdhorhhgpdhrtghpthhtoheprhhoshhtvgguthesghho ohgumhhishdrohhrghdprhgtphhtthhopehmrghthhhivghurdguvghsnhhohigvrhhsse gvfhhfihgtihhoshdrtghomhdprhgtphhtthhopehfrhgvuggvrhhitgeskhgvrhhnvghl rdhorhhgpdhrtghpthhtohepnhgvvghrrghjrdhuphgrughhhigrhieskhgvrhhnvghlrd horhhgpdhrtghpthhtohepjhhovghlsehjohgvlhhfvghrnhgrnhguvghsrdhorhhg X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 23 Feb 2025 21:22:21 -0500 (EST) From: Boqun Feng To: rcu@vger.kernel.org Cc: Lai Jiangshan , "Paul E. McKenney" , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Uladzislau Rezki , Zqiang , Davidlohr Bueso , Shuah Khan , linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, bpf@vger.kernel.org, Alexei Starovoitov , Andrii Nakryiko , Peter Zijlstra , Kent Overstreet Subject: [PATCH rcu 04/20] srcu: Pull ->srcu_{un,}lock_count into a new srcu_ctr structure Date: Sun, 23 Feb 2025 18:21:58 -0800 Message-Id: <20250224022214.12037-5-boqun.feng@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250224022214.12037-1-boqun.feng@gmail.com> References: <20250224022214.12037-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" This commit prepares for array-index-free srcu_read_lock*() by moving the ->srcu_{un,}lock_count fields into a new srcu_ctr structure. This will permit ->srcu_index to be replaced by a per-CPU pointer to this structure. Signed-off-by: Paul E. McKenney Cc: Alexei Starovoitov Cc: Andrii Nakryiko Cc: Peter Zijlstra Cc: Kent Overstreet Cc: Signed-off-by: Boqun Feng --- include/linux/srcutree.h | 13 +++-- kernel/rcu/srcutree.c | 115 +++++++++++++++++++-------------------- 2 files changed, 66 insertions(+), 62 deletions(-) diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index b17814c9d1c7..c794d599db5c 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -17,14 +17,19 @@ struct srcu_node; struct srcu_struct; +/* One element of the srcu_data srcu_ctrs array. */ +struct srcu_ctr { + atomic_long_t srcu_locks; /* Locks per CPU. */ + atomic_long_t srcu_unlocks; /* Unlocks per CPU. */ +}; + /* * Per-CPU structure feeding into leaf srcu_node, similar in function * to rcu_node. */ struct srcu_data { /* Read-side state. */ - atomic_long_t srcu_lock_count[2]; /* Locks per CPU. */ - atomic_long_t srcu_unlock_count[2]; /* Unlocks per CPU. */ + struct srcu_ctr srcu_ctrs[2]; /* Locks and unlocks per CPU. */ int srcu_reader_flavor; /* Reader flavor for srcu_struct structure? */ /* Values: SRCU_READ_FLAVOR_.* */ @@ -221,7 +226,7 @@ static inline int __srcu_read_lock_lite(struct srcu_struct *ssp) RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_lock_lite()."); idx = READ_ONCE(ssp->srcu_idx) & 0x1; - this_cpu_inc(ssp->sda->srcu_lock_count[idx].counter); /* Y */ + this_cpu_inc(ssp->sda->srcu_ctrs[idx].srcu_locks.counter); /* Y */ barrier(); /* Avoid leaking the critical section. */ return idx; } @@ -240,7 +245,7 @@ static inline int __srcu_read_lock_lite(struct srcu_struct *ssp) static inline void __srcu_read_unlock_lite(struct srcu_struct *ssp, int idx) { barrier(); /* Avoid leaking the critical section. */ - this_cpu_inc(ssp->sda->srcu_unlock_count[idx].counter); /* Z */ + this_cpu_inc(ssp->sda->srcu_ctrs[idx].srcu_unlocks.counter); /* Z */ RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_unlock_lite()."); } diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index e69ce9d59abf..d7ee2f345e19 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -116,8 +116,9 @@ do { \ /* * Initialize SRCU per-CPU data. Note that statically allocated * srcu_struct structures might already have srcu_read_lock() and - * srcu_read_unlock() running against them. So if the is_static parameter - * is set, don't initialize ->srcu_lock_count[] and ->srcu_unlock_count[]. + * srcu_read_unlock() running against them. So if the is_static + * parameter is set, don't initialize ->srcu_ctrs[].srcu_locks and + * ->srcu_ctrs[].srcu_unlocks. */ static void init_srcu_struct_data(struct srcu_struct *ssp) { @@ -128,8 +129,6 @@ static void init_srcu_struct_data(struct srcu_struct *ssp) * Initialize the per-CPU srcu_data array, which feeds into the * leaves of the srcu_node tree. */ - BUILD_BUG_ON(ARRAY_SIZE(sdp->srcu_lock_count) != - ARRAY_SIZE(sdp->srcu_unlock_count)); for_each_possible_cpu(cpu) { sdp = per_cpu_ptr(ssp->sda, cpu); spin_lock_init(&ACCESS_PRIVATE(sdp, lock)); @@ -429,10 +428,10 @@ static bool srcu_gp_is_expedited(struct srcu_struct *ssp) } /* - * Computes approximate total of the readers' ->srcu_lock_count[] values - * for the rank of per-CPU counters specified by idx, and returns true if - * the caller did the proper barrier (gp), and if the count of the locks - * matches that of the unlocks passed in. + * Computes approximate total of the readers' ->srcu_ctrs[].srcu_locks + * values for the rank of per-CPU counters specified by idx, and returns + * true if the caller did the proper barrier (gp), and if the count of + * the locks matches that of the unlocks passed in. */ static bool srcu_readers_lock_idx(struct srcu_struct *ssp, int idx, bool gp, unsigned long unlocks) { @@ -443,7 +442,7 @@ static bool srcu_readers_lock_idx(struct srcu_struct *ssp, int idx, bool gp, uns for_each_possible_cpu(cpu) { struct srcu_data *sdp = per_cpu_ptr(ssp->sda, cpu); - sum += atomic_long_read(&sdp->srcu_lock_count[idx]); + sum += atomic_long_read(&sdp->srcu_ctrs[idx].srcu_locks); if (IS_ENABLED(CONFIG_PROVE_RCU)) mask = mask | READ_ONCE(sdp->srcu_reader_flavor); } @@ -455,8 +454,8 @@ static bool srcu_readers_lock_idx(struct srcu_struct *ssp, int idx, bool gp, uns } /* - * Returns approximate total of the readers' ->srcu_unlock_count[] values - * for the rank of per-CPU counters specified by idx. + * Returns approximate total of the readers' ->srcu_ctrs[].srcu_unlocks + * values for the rank of per-CPU counters specified by idx. */ static unsigned long srcu_readers_unlock_idx(struct srcu_struct *ssp, int idx, unsigned long *rdm) { @@ -467,7 +466,7 @@ static unsigned long srcu_readers_unlock_idx(struct srcu_struct *ssp, int idx, u for_each_possible_cpu(cpu) { struct srcu_data *sdp = per_cpu_ptr(ssp->sda, cpu); - sum += atomic_long_read(&sdp->srcu_unlock_count[idx]); + sum += atomic_long_read(&sdp->srcu_ctrs[idx].srcu_unlocks); mask = mask | READ_ONCE(sdp->srcu_reader_flavor); } WARN_ONCE(IS_ENABLED(CONFIG_PROVE_RCU) && (mask & (mask - 1)), @@ -510,9 +509,9 @@ static bool srcu_readers_active_idx_check(struct srcu_struct *ssp, int idx) * been no readers on this index at some point in this function. * But there might be more readers, as a task might have read * the current ->srcu_idx but not yet have incremented its CPU's - * ->srcu_lock_count[idx] counter. In fact, it is possible + * ->srcu_ctrs[idx].srcu_locks counter. In fact, it is possible * that most of the tasks have been preempted between fetching - * ->srcu_idx and incrementing ->srcu_lock_count[idx]. And there + * ->srcu_idx and incrementing ->srcu_ctrs[idx].srcu_locks. And there * could be almost (ULONG_MAX / sizeof(struct task_struct)) tasks * in a system whose address space was fully populated with memory. * Call this quantity Nt. @@ -521,36 +520,36 @@ static bool srcu_readers_active_idx_check(struct srcu_struct *ssp, int idx) * code for a long time. That now-preempted updater has already * flipped ->srcu_idx (possibly during the preceding grace period), * done an smp_mb() (again, possibly during the preceding grace - * period), and summed up the ->srcu_unlock_count[idx] counters. + * period), and summed up the ->srcu_ctrs[idx].srcu_unlocks counters. * How many times can a given one of the aforementioned Nt tasks - * increment the old ->srcu_idx value's ->srcu_lock_count[idx] + * increment the old ->srcu_idx value's ->srcu_ctrs[idx].srcu_locks * counter, in the absence of nesting? * * It can clearly do so once, given that it has already fetched - * the old value of ->srcu_idx and is just about to use that value - * to index its increment of ->srcu_lock_count[idx]. But as soon as - * it leaves that SRCU read-side critical section, it will increment - * ->srcu_unlock_count[idx], which must follow the updater's above - * read from that same value. Thus, as soon the reading task does - * an smp_mb() and a later fetch from ->srcu_idx, that task will be - * guaranteed to get the new index. Except that the increment of - * ->srcu_unlock_count[idx] in __srcu_read_unlock() is after the - * smp_mb(), and the fetch from ->srcu_idx in __srcu_read_lock() - * is before the smp_mb(). Thus, that task might not see the new - * value of ->srcu_idx until the -second- __srcu_read_lock(), - * which in turn means that this task might well increment - * ->srcu_lock_count[idx] for the old value of ->srcu_idx twice, - * not just once. + * the old value of ->srcu_idx and is just about to use that + * value to index its increment of ->srcu_ctrs[idx].srcu_locks. + * But as soon as it leaves that SRCU read-side critical section, + * it will increment ->srcu_ctrs[idx].srcu_unlocks, which must + * follow the updater's above read from that same value. Thus, + * as soon the reading task does an smp_mb() and a later fetch from + * ->srcu_idx, that task will be guaranteed to get the new index. + * Except that the increment of ->srcu_ctrs[idx].srcu_unlocks + * in __srcu_read_unlock() is after the smp_mb(), and the fetch + * from ->srcu_idx in __srcu_read_lock() is before the smp_mb(). + * Thus, that task might not see the new value of ->srcu_idx until + * the -second- __srcu_read_lock(), which in turn means that this + * task might well increment ->srcu_ctrs[idx].srcu_locks for the + * old value of ->srcu_idx twice, not just once. * * However, it is important to note that a given smp_mb() takes * effect not just for the task executing it, but also for any * later task running on that same CPU. * - * That is, there can be almost Nt + Nc further increments of - * ->srcu_lock_count[idx] for the old index, where Nc is the number - * of CPUs. But this is OK because the size of the task_struct - * structure limits the value of Nt and current systems limit Nc - * to a few thousand. + * That is, there can be almost Nt + Nc further increments + * of ->srcu_ctrs[idx].srcu_locks for the old index, where Nc + * is the number of CPUs. But this is OK because the size of + * the task_struct structure limits the value of Nt and current + * systems limit Nc to a few thousand. * * OK, but what about nesting? This does impose a limit on * nesting of half of the size of the task_struct structure @@ -581,10 +580,10 @@ static bool srcu_readers_active(struct srcu_struct *ssp) for_each_possible_cpu(cpu) { struct srcu_data *sdp = per_cpu_ptr(ssp->sda, cpu); - sum += atomic_long_read(&sdp->srcu_lock_count[0]); - sum += atomic_long_read(&sdp->srcu_lock_count[1]); - sum -= atomic_long_read(&sdp->srcu_unlock_count[0]); - sum -= atomic_long_read(&sdp->srcu_unlock_count[1]); + sum += atomic_long_read(&sdp->srcu_ctrs[0].srcu_locks); + sum += atomic_long_read(&sdp->srcu_ctrs[1].srcu_locks); + sum -= atomic_long_read(&sdp->srcu_ctrs[0].srcu_unlocks); + sum -= atomic_long_read(&sdp->srcu_ctrs[1].srcu_unlocks); } return sum; } @@ -746,7 +745,7 @@ int __srcu_read_lock(struct srcu_struct *ssp) int idx; idx = READ_ONCE(ssp->srcu_idx) & 0x1; - this_cpu_inc(ssp->sda->srcu_lock_count[idx].counter); + this_cpu_inc(ssp->sda->srcu_ctrs[idx].srcu_locks.counter); smp_mb(); /* B */ /* Avoid leaking the critical section. */ return idx; } @@ -760,7 +759,7 @@ EXPORT_SYMBOL_GPL(__srcu_read_lock); void __srcu_read_unlock(struct srcu_struct *ssp, int idx) { smp_mb(); /* C */ /* Avoid leaking the critical section. */ - this_cpu_inc(ssp->sda->srcu_unlock_count[idx].counter); + this_cpu_inc(ssp->sda->srcu_ctrs[idx].srcu_unlocks.counter); } EXPORT_SYMBOL_GPL(__srcu_read_unlock); @@ -777,7 +776,7 @@ int __srcu_read_lock_nmisafe(struct srcu_struct *ssp) struct srcu_data *sdp = raw_cpu_ptr(ssp->sda); idx = READ_ONCE(ssp->srcu_idx) & 0x1; - atomic_long_inc(&sdp->srcu_lock_count[idx]); + atomic_long_inc(&sdp->srcu_ctrs[idx].srcu_locks); smp_mb__after_atomic(); /* B */ /* Avoid leaking the critical section. */ return idx; } @@ -793,7 +792,7 @@ void __srcu_read_unlock_nmisafe(struct srcu_struct *ssp, int idx) struct srcu_data *sdp = raw_cpu_ptr(ssp->sda); smp_mb__before_atomic(); /* C */ /* Avoid leaking the critical section. */ - atomic_long_inc(&sdp->srcu_unlock_count[idx]); + atomic_long_inc(&sdp->srcu_ctrs[idx].srcu_unlocks); } EXPORT_SYMBOL_GPL(__srcu_read_unlock_nmisafe); @@ -1123,17 +1122,17 @@ static void srcu_flip(struct srcu_struct *ssp) /* * Because the flip of ->srcu_idx is executed only if the * preceding call to srcu_readers_active_idx_check() found that - * the ->srcu_unlock_count[] and ->srcu_lock_count[] sums matched - * and because that summing uses atomic_long_read(), there is - * ordering due to a control dependency between that summing and - * the WRITE_ONCE() in this call to srcu_flip(). This ordering - * ensures that if this updater saw a given reader's increment from - * __srcu_read_lock(), that reader was using a value of ->srcu_idx - * from before the previous call to srcu_flip(), which should be - * quite rare. This ordering thus helps forward progress because - * the grace period could otherwise be delayed by additional - * calls to __srcu_read_lock() using that old (soon to be new) - * value of ->srcu_idx. + * the ->srcu_ctrs[].srcu_unlocks and ->srcu_ctrs[].srcu_locks sums + * matched and because that summing uses atomic_long_read(), + * there is ordering due to a control dependency between that + * summing and the WRITE_ONCE() in this call to srcu_flip(). + * This ordering ensures that if this updater saw a given reader's + * increment from __srcu_read_lock(), that reader was using a value + * of ->srcu_idx from before the previous call to srcu_flip(), + * which should be quite rare. This ordering thus helps forward + * progress because the grace period could otherwise be delayed + * by additional calls to __srcu_read_lock() using that old (soon + * to be new) value of ->srcu_idx. * * This sum-equality check and ordering also ensures that if * a given call to __srcu_read_lock() uses the new value of @@ -1914,8 +1913,8 @@ void srcu_torture_stats_print(struct srcu_struct *ssp, char *tt, char *tf) struct srcu_data *sdp; sdp = per_cpu_ptr(ssp->sda, cpu); - u0 = data_race(atomic_long_read(&sdp->srcu_unlock_count[!idx])); - u1 = data_race(atomic_long_read(&sdp->srcu_unlock_count[idx])); + u0 = data_race(atomic_long_read(&sdp->srcu_ctrs[!idx].srcu_unlocks)); + u1 = data_race(atomic_long_read(&sdp->srcu_ctrs[idx].srcu_unlocks)); /* * Make sure that a lock is always counted if the corresponding @@ -1923,8 +1922,8 @@ void srcu_torture_stats_print(struct srcu_struct *ssp, char *tt, char *tf) */ smp_rmb(); - l0 = data_race(atomic_long_read(&sdp->srcu_lock_count[!idx])); - l1 = data_race(atomic_long_read(&sdp->srcu_lock_count[idx])); + l0 = data_race(atomic_long_read(&sdp->srcu_ctrs[!idx].srcu_locks)); + l1 = data_race(atomic_long_read(&sdp->srcu_ctrs[idx].srcu_locks)); c0 = l0 - u0; c1 = l1 - u1; From patchwork Mon Feb 24 02:21:59 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boqun Feng X-Patchwork-Id: 13987318 Received: from mail-qk1-f172.google.com (mail-qk1-f172.google.com [209.85.222.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 110651B0413; Mon, 24 Feb 2025 02:22:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363747; cv=none; b=kwD+fI2z1nHbIVIDXknhyKV1aHuAmBWZjIadEGs8cNHXtfo+2uTmbF35UK/9NfmRYBwrfDuu286/cmdCeTzpCatXeyXMk4UJSyAFWDgAmDvOKkcAJ/tpunUn1Kvig78QcnRSAuqnCwxZG8EZ50gIBYnTQvbUPUy1YjDY/2uG98I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363747; c=relaxed/simple; bh=zRra4zBBZtDQO3WoVybDDJwUOBjEV98iMvYd/Qf7oxo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=UZI4spgjbzTQeMmXiOygvT1+arDQboommqHmt7Y+MjG3OButsS/tnLpfl/W0G3kWGdVvuYFNaVl9mzH35Kc2TeJeubc0AueXlsB4YMv8oWwAI9ReR3jo9a5HocIokgHurmrz/EVObqKe9WeDxk3gKrTJ5Lkd2/XuSuN8rjxYV7Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=JKcWl11q; arc=none smtp.client-ip=209.85.222.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="JKcWl11q" Received: by mail-qk1-f172.google.com with SMTP id af79cd13be357-7c0a1677aebso381003285a.0; Sun, 23 Feb 2025 18:22:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740363745; x=1740968545; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=h9z6N5Bvl2pb8/I5e6YQjTUdeZNzH7F0rRvppc4nnQA=; b=JKcWl11qrjp4qaPyL+L2KEdieCKWJqyDBYV/4Kjq33/xWF2cn0T+yML5XmG/OqaoAX FWt9ZsYJqy+ZIrnZouA6ahkSijlCG9onxsmwV5Ubm7q/4EhAHsxRxpuhnZT1YevEqUjq 5hDw64uCpklSm1eTrD7bQjGbZI5oTUonU9NUaOzgULAy42T6bJKOuHPMaWiAkX6wZbnq bRur1gK0lDtOsWoq/8buwYGBeKe/E/X+UAk9wA2uf+gDEtvvJRNqmYKkX88tWrb3H4Eg tJXF73svRPhjoEL9tVv1mplaG0VCDQ8hyVph0Hyn++xgwOpQP43XDkK3skS3gQm+63Kq FvtQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740363745; x=1740968545; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=h9z6N5Bvl2pb8/I5e6YQjTUdeZNzH7F0rRvppc4nnQA=; b=a3dYCltreXimbNHPzUe/bacFZpzW/hJ7BGTxi9SfXDW+Cq4+8i4Lm+0k+C1/kKD4Z0 ZKKemNKLGMFzgsUN2ut7k1xc4ePA5StJKmk+ZueDxSA62d+d6p0XBw2ulQdFNpPrd3Cz WQziQ1eO89z8Sh6UPE88JJX98s9HUX6iYnxOCF2W26UqHgkYKu2ZcIQP+KEoYoN0EZED q7o+qXsSRNtatpTW24wANd00/VRlxFi8aVkpyK0z5esxexNGVT+tSUVEQggcOMBGdC8j UyxufxFTFtKsh8Vg58XocgEq6zO+DPA2XqrSMiQsy+YAd81nko2iNJMf7lJWufpniUi4 zfSw== X-Forwarded-Encrypted: i=1; AJvYcCUNVY7REWT4Qo28RZJUwY8yAKBpoR/d8SgiXvEwqyAPnR7QvOhI2X3GbufHlGc28V/8PofRF70k14TL0MS96Kcd@vger.kernel.org, AJvYcCWzD3A87ociruc5cMmUcvmcp/pAiNQoee/I3MlZCvu/IvXFqwaq81XtSl4ZZUj1utpvXvY=@vger.kernel.org, AJvYcCXoJLV3vaMU5va3AACzKMWHJbRNgWAaHwm6NrvlJ+5c++Vlm2qDhyUa8Pbi6De6+Hp6h5yFdzeZw/vECCTm@vger.kernel.org X-Gm-Message-State: AOJu0Yz+fGDTl2vEo48Pfz2h5rXN8v2L5gJ2uA4o6C/WMB9LjQUK6Ko3 kflmBle9kdt2xo6TQbPmK4w6udoEszSfSk8KXvXgBDYPSv6gpLUKhzS1MA== X-Gm-Gg: ASbGncuKBnvGlF2lWVgQC1xOeh9q/GXjqwEdkyLmFK+krYTvLbDBLOa6h2bQWVCydMr S+OAQJ88QGqewlQZJ1YFg0kDkg3S7yc5gugSfD2scbUjbN5dJtimjRR3tDuzbZiIOCo18asY39j JcmjzefpcMzTLvq3jrBN5/zqkuElA8Wf8LLIezMgUxThC3y0Uz1OrsVkrX8ZzbsBMiGAdEthmNN An8wOvpjBL1mv9rX1+X3thHODo2IgfGX3OVP2FZKhwhIF8KNPxOOCKfmnWeG1LOg5020XkISHoo 2gS1UqiyhUQ73GT+n9wF83kOedOKq7iKvSu9r/+DVI7z6lUIwZ+Ba/RQNCHbAS+QuN9h1xToKID WUMue/CPWlc9qTTJZ X-Google-Smtp-Source: AGHT+IFgbIWJVDI5TX05kb/ZTvdYGWwbSJPyqHQrPiTksA7vZMhwnuGLYKGh02gnGO7Mv+ckvF8gdg== X-Received: by 2002:a05:620a:800b:b0:7c0:c264:6737 with SMTP id af79cd13be357-7c0cf8b015emr1564047085a.4.1740363744776; Sun, 23 Feb 2025 18:22:24 -0800 (PST) Received: from fauth-a2-smtp.messagingengine.com (fauth-a2-smtp.messagingengine.com. [103.168.172.201]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7c09e95b68fsm938235185a.12.2025.02.23.18.22.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 23 Feb 2025 18:22:24 -0800 (PST) Received: from phl-compute-08.internal (phl-compute-08.phl.internal [10.202.2.48]) by mailfauth.phl.internal (Postfix) with ESMTP id 7D8C81200043; Sun, 23 Feb 2025 21:22:23 -0500 (EST) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-08.internal (MEProxy); Sun, 23 Feb 2025 21:22:23 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgdejjeehgecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvfevufffkffojghfggfgsedtkeertdertddt necuhfhrohhmpeeuohhquhhnucfhvghnghcuoegsohhquhhnrdhfvghnghesghhmrghilh drtghomheqnecuggftrfgrthhtvghrnhepgeeljeeitdehvdehgefgjeevfeejjeekgfev ffeiueejhfeuiefggeeuheeggefgnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrg hmpehmrghilhhfrhhomhepsghoqhhunhdomhgvshhmthhprghuthhhphgvrhhsohhnrghl ihhthidqieelvdeghedtieegqddujeejkeehheehvddqsghoqhhunhdrfhgvnhhgpeepgh hmrghilhdrtghomhesfhhigihmvgdrnhgrmhgvpdhnsggprhgtphhtthhopedvvddpmhho uggvpehsmhhtphhouhhtpdhrtghpthhtoheprhgtuhesvhhgvghrrdhkvghrnhgvlhdroh hrghdprhgtphhtthhopehjihgrnhhgshhhrghnlhgrihesghhmrghilhdrtghomhdprhgt phhtthhopehprghulhhmtghksehkvghrnhgvlhdrohhrghdprhgtphhtthhopehjohhshh esjhhoshhhthhrihhplhgvthhtrdhorhhgpdhrtghpthhtoheprhhoshhtvgguthesghho ohgumhhishdrohhrghdprhgtphhtthhopehmrghthhhivghurdguvghsnhhohigvrhhsse gvfhhfihgtihhoshdrtghomhdprhgtphhtthhopehfrhgvuggvrhhitgeskhgvrhhnvghl rdhorhhgpdhrtghpthhtohepnhgvvghrrghjrdhuphgrughhhigrhieskhgvrhhnvghlrd horhhgpdhrtghpthhtohepjhhovghlsehjohgvlhhfvghrnhgrnhguvghsrdhorhhg X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 23 Feb 2025 21:22:22 -0500 (EST) From: Boqun Feng To: rcu@vger.kernel.org Cc: Lai Jiangshan , "Paul E. McKenney" , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Uladzislau Rezki , Zqiang , Davidlohr Bueso , Shuah Khan , linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, bpf@vger.kernel.org, kernel test robot , Alexei Starovoitov , Andrii Nakryiko , Peter Zijlstra , Kent Overstreet Subject: [PATCH rcu 05/20] srcu: Make SRCU readers use ->srcu_ctrs for counter selection Date: Sun, 23 Feb 2025 18:21:59 -0800 Message-Id: <20250224022214.12037-6-boqun.feng@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250224022214.12037-1-boqun.feng@gmail.com> References: <20250224022214.12037-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" This commit causes SRCU readers to use ->srcu_ctrs for counter selection instead of ->srcu_idx. This takes another step towards array-indexing-free SRCU readers. [ paulmck: Apply kernel test robot feedback. ] Co-developed-by: Z qiang Signed-off-by: Z qiang Signed-off-by: Paul E. McKenney Tested-by: kernel test robot Cc: Alexei Starovoitov Cc: Andrii Nakryiko Cc: Peter Zijlstra Cc: Kent Overstreet Cc: Signed-off-by: Boqun Feng --- include/linux/srcutree.h | 9 +++++---- kernel/rcu/srcutree.c | 23 +++++++++++++---------- 2 files changed, 18 insertions(+), 14 deletions(-) diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index c794d599db5c..1b01ced61a45 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -101,6 +101,7 @@ struct srcu_usage { */ struct srcu_struct { unsigned int srcu_idx; /* Current rdr array element. */ + struct srcu_ctr __percpu *srcu_ctrp; struct srcu_data __percpu *sda; /* Per-CPU srcu_data array. */ struct lockdep_map dep_map; struct srcu_usage *srcu_sup; /* Update-side data. */ @@ -167,6 +168,7 @@ struct srcu_struct { #define __SRCU_STRUCT_INIT(name, usage_name, pcpu_name) \ { \ .sda = &pcpu_name, \ + .srcu_ctrp = &pcpu_name.srcu_ctrs[0], \ __SRCU_STRUCT_INIT_COMMON(name, usage_name) \ } @@ -222,13 +224,12 @@ void srcu_torture_stats_print(struct srcu_struct *ssp, char *tt, char *tf); */ static inline int __srcu_read_lock_lite(struct srcu_struct *ssp) { - int idx; + struct srcu_ctr __percpu *scp = READ_ONCE(ssp->srcu_ctrp); RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_lock_lite()."); - idx = READ_ONCE(ssp->srcu_idx) & 0x1; - this_cpu_inc(ssp->sda->srcu_ctrs[idx].srcu_locks.counter); /* Y */ + this_cpu_inc(scp->srcu_locks.counter); /* Y */ barrier(); /* Avoid leaking the critical section. */ - return idx; + return scp - &ssp->sda->srcu_ctrs[0]; } /* diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index d7ee2f345e19..7efde1a2344e 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -253,8 +253,10 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp, bool is_static) atomic_set(&ssp->srcu_sup->srcu_barrier_cpu_cnt, 0); INIT_DELAYED_WORK(&ssp->srcu_sup->work, process_srcu); ssp->srcu_sup->sda_is_static = is_static; - if (!is_static) + if (!is_static) { ssp->sda = alloc_percpu(struct srcu_data); + ssp->srcu_ctrp = &ssp->sda->srcu_ctrs[0]; + } if (!ssp->sda) goto err_free_sup; init_srcu_struct_data(ssp); @@ -742,12 +744,11 @@ EXPORT_SYMBOL_GPL(__srcu_check_read_flavor); */ int __srcu_read_lock(struct srcu_struct *ssp) { - int idx; + struct srcu_ctr __percpu *scp = READ_ONCE(ssp->srcu_ctrp); - idx = READ_ONCE(ssp->srcu_idx) & 0x1; - this_cpu_inc(ssp->sda->srcu_ctrs[idx].srcu_locks.counter); + this_cpu_inc(scp->srcu_locks.counter); smp_mb(); /* B */ /* Avoid leaking the critical section. */ - return idx; + return scp - &ssp->sda->srcu_ctrs[0]; } EXPORT_SYMBOL_GPL(__srcu_read_lock); @@ -772,13 +773,12 @@ EXPORT_SYMBOL_GPL(__srcu_read_unlock); */ int __srcu_read_lock_nmisafe(struct srcu_struct *ssp) { - int idx; - struct srcu_data *sdp = raw_cpu_ptr(ssp->sda); + struct srcu_ctr __percpu *scpp = READ_ONCE(ssp->srcu_ctrp); + struct srcu_ctr *scp = raw_cpu_ptr(scpp); - idx = READ_ONCE(ssp->srcu_idx) & 0x1; - atomic_long_inc(&sdp->srcu_ctrs[idx].srcu_locks); + atomic_long_inc(&scp->srcu_locks); smp_mb__after_atomic(); /* B */ /* Avoid leaking the critical section. */ - return idx; + return scpp - &ssp->sda->srcu_ctrs[0]; } EXPORT_SYMBOL_GPL(__srcu_read_lock_nmisafe); @@ -1152,6 +1152,8 @@ static void srcu_flip(struct srcu_struct *ssp) smp_mb(); /* E */ /* Pairs with B and C. */ WRITE_ONCE(ssp->srcu_idx, ssp->srcu_idx + 1); // Flip the counter. + WRITE_ONCE(ssp->srcu_ctrp, + &ssp->sda->srcu_ctrs[!(ssp->srcu_ctrp - &ssp->sda->srcu_ctrs[0])]); /* * Ensure that if the updater misses an __srcu_read_unlock() @@ -2000,6 +2002,7 @@ static int srcu_module_coming(struct module *mod) ssp->sda = alloc_percpu(struct srcu_data); if (WARN_ON_ONCE(!ssp->sda)) return -ENOMEM; + ssp->srcu_ctrp = &ssp->sda->srcu_ctrs[0]; } return 0; } From patchwork Mon Feb 24 02:22:00 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boqun Feng X-Patchwork-Id: 13987319 Received: from mail-qk1-f169.google.com (mail-qk1-f169.google.com [209.85.222.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 28DF51C6FEC; Mon, 24 Feb 2025 02:22:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363749; cv=none; b=T8VY8Bulv0z5VlHDARi8y4XFblvfVXN4MGlUEt9JNiFXUvXpF/yXtn90KFWZHBzi23kfKNwMFr7CCbC3sOyuXm9Ja3V9XsaKWO6b5iShKngna4VCzcvcD4RUVWBGrIfEYn2DLswGTgwMumsQ6xVih5P39zdUv8Sq8rjbA+S+NOY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363749; c=relaxed/simple; bh=n8KYK+ApLOt1uHZCuF43++Q8gIRMXbNEKX0ciRrabXg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=GhjGqXDTnWgcltalZb535Qc6pLtOG2CILGCZUkeE0yA7gGl6SI/hn97IEbp2vX0RfCitBcToIlUWuNDsgUjQ4/1ytsMdC5xkT6ueu4NyGOFGKFOIGsCTgwi1wsb7lZjYcgKDQCoAPuFCzgLMQNoM6lq061HGdWGmb/t8EZIyznw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=mVVtcLTl; arc=none smtp.client-ip=209.85.222.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="mVVtcLTl" Received: by mail-qk1-f169.google.com with SMTP id af79cd13be357-7c09a30e14dso688316085a.0; Sun, 23 Feb 2025 18:22:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740363746; x=1740968546; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=AcTo3KBDVWrf1oe45FpKbyw6ahcrt2yvV77yr4g8K3g=; b=mVVtcLTlHQXj/KKTCK2fvFuZOEce7Px7xt25KIwN188XTTvCCg3Nh7XvMEdxAMeA3l vANHClV2dRNlBLZpmls87ufEHGxqBKbVluz6uVRmjIzy4Bc7mKYVDw3Bf7oXgByWrnDK i2FnoSpVRxhb0bLfRXLy5/jqa9svJJ8oCh5ulKBb6XbMAzaXEvchNJlc9dWhuzwLKSQD 6Xc8HDLv0qPZNhfUy4VYtQIgd1I2GxAjLBT95KgZhBFKRQoy6QqPLZo0JKmqdvErXnwe ecItlmryEyDnpeYwHETbmlgI/XjDtcndrlMcDpD1ETTdDpa/b67UcDG7dsv1+NAOoFV8 HjrA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740363746; x=1740968546; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=AcTo3KBDVWrf1oe45FpKbyw6ahcrt2yvV77yr4g8K3g=; b=I0t9xiSj8qHOf3kBnqD6F2BxImfj5fZSCiaPz+gf/mo55L07/SSOw+4zjC7d1gsPAo ZFmRTu9+vhZLuCNF5DRkmB+xQ9qsZUSW4DCBBnawg53LzVwD8eKcaBas7axmhxu6P4k1 gYg7POYQDXhD9gto+7JbprgycYVwbqqTjLbwDQXDL7+eGG152LOynkFe+nayeFEBvvK+ osMtrBnchCUb4cS9oKk7u4bzCVqpC4f3FlRzRMt9cpjiNtiyBvu9pq9j8lLwR0M/ZOj5 Cgo0zPwiPY8T2bLk8WjEgin4fC85mKGq75bMTW365Dq13KTYcn3GvZsPvYBNeHmDAjxX VRAg== X-Forwarded-Encrypted: i=1; AJvYcCWM78JkMnXhH1pWrNyhiUqcVin3V72pimR1BGlrprdqAeh1nDUmwenygI7YVnSoxOFXruV5YnJBx0QoC2bjLOg2@vger.kernel.org, AJvYcCXO+2NTXleUgNrYDpQ368legic6Et5/O6gRKr624yZL2hsmgmsvjUKysHji2QdECE6Zs+PqG6KI8Nl0HneS@vger.kernel.org, AJvYcCXdb9GeWNFz0DB5/Ny3f/5Gt87gqAB3U3ip6t+w17fO0jgpC8vKbjlmQ5gO9CkfIr219/E=@vger.kernel.org X-Gm-Message-State: AOJu0Yw4uBmsIMixYnqloJbs64c9xdAai9r2aLO6RM9F/kBRCog6fWdz wWVcIrzp5wIw1DsJzMVPXsqvb7HR409vzSuzZeqgeaFpNbemmU5O X-Gm-Gg: ASbGncvB5rbADtlTojnk3m4JvJumGBoRC8XxKkAc7Z4ZNEhIi4aKYKJJxL40BajAwPP a/5Scr397XNqXH7VdazElfpb1jxMyJJ4CE7T74uenfDXzHtOcCNSSKmbzAoUgg7Iu4xd9ncsoMp Yj/Y+SsOnuvEz1VRhHUTyVbCSAyUWHK4Ve9Y1DfsvIaDAa85ia6kds0lcTsTLQNuRyyy6T5Oz4/ cPXB7qqZm1ubzBQNiYkY5TBYsu0DhoVc3KYE8Cf3yBcla97ylN2/oW5SExPYPpUDxcZwRz7KbNy s+HII2w8CtP6i/aJooY08Olm/FxoQH0xkrmbVDY2QYnMCzgqjLwTDkexdS7X2qrrxtdbZXYkSvI UEZOBq6jxOshncrEl X-Google-Smtp-Source: AGHT+IGEM+/HZwVspmar+q03xk2siTeyFzKw3jnpholWrfwhHL/2SXZ2X7t+5uARsJftW+6Gd12JeA== X-Received: by 2002:a05:620a:4045:b0:7c0:c282:702d with SMTP id af79cd13be357-7c0cef561aemr1595736785a.39.1740363745924; Sun, 23 Feb 2025 18:22:25 -0800 (PST) Received: from fauth-a2-smtp.messagingengine.com (fauth-a2-smtp.messagingengine.com. [103.168.172.201]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7c0c2d9f679sm552767685a.5.2025.02.23.18.22.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 23 Feb 2025 18:22:25 -0800 (PST) Received: from phl-compute-08.internal (phl-compute-08.phl.internal [10.202.2.48]) by mailfauth.phl.internal (Postfix) with ESMTP id D791F1200066; Sun, 23 Feb 2025 21:22:24 -0500 (EST) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-08.internal (MEProxy); Sun, 23 Feb 2025 21:22:24 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgdejjeehgecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvfevufffkffojghfggfgsedtkeertdertddt necuhfhrohhmpeeuohhquhhnucfhvghnghcuoegsohhquhhnrdhfvghnghesghhmrghilh drtghomheqnecuggftrfgrthhtvghrnhepgeeljeeitdehvdehgefgjeevfeejjeekgfev ffeiueejhfeuiefggeeuheeggefgnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrg hmpehmrghilhhfrhhomhepsghoqhhunhdomhgvshhmthhprghuthhhphgvrhhsohhnrghl ihhthidqieelvdeghedtieegqddujeejkeehheehvddqsghoqhhunhdrfhgvnhhgpeepgh hmrghilhdrtghomhesfhhigihmvgdrnhgrmhgvpdhnsggprhgtphhtthhopedvuddpmhho uggvpehsmhhtphhouhhtpdhrtghpthhtoheprhgtuhesvhhgvghrrdhkvghrnhgvlhdroh hrghdprhgtphhtthhopehjihgrnhhgshhhrghnlhgrihesghhmrghilhdrtghomhdprhgt phhtthhopehprghulhhmtghksehkvghrnhgvlhdrohhrghdprhgtphhtthhopehjohhshh esjhhoshhhthhrihhplhgvthhtrdhorhhgpdhrtghpthhtoheprhhoshhtvgguthesghho ohgumhhishdrohhrghdprhgtphhtthhopehmrghthhhivghurdguvghsnhhohigvrhhsse gvfhhfihgtihhoshdrtghomhdprhgtphhtthhopehfrhgvuggvrhhitgeskhgvrhhnvghl rdhorhhgpdhrtghpthhtohepnhgvvghrrghjrdhuphgrughhhigrhieskhgvrhhnvghlrd horhhgpdhrtghpthhtohepjhhovghlsehjohgvlhhfvghrnhgrnhguvghsrdhorhhg X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 23 Feb 2025 21:22:24 -0500 (EST) From: Boqun Feng To: rcu@vger.kernel.org Cc: Lai Jiangshan , "Paul E. McKenney" , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Uladzislau Rezki , Zqiang , Davidlohr Bueso , Shuah Khan , linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, bpf@vger.kernel.org, Alexei Starovoitov , Andrii Nakryiko , Peter Zijlstra , Kent Overstreet Subject: [PATCH rcu 06/20] srcu: Make Tree SRCU updates independent of ->srcu_idx Date: Sun, 23 Feb 2025 18:22:00 -0800 Message-Id: <20250224022214.12037-7-boqun.feng@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250224022214.12037-1-boqun.feng@gmail.com> References: <20250224022214.12037-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" This commit makes Tree SRCU updates independent of ->srcu_idx, then drop ->srcu_idx. Signed-off-by: Paul E. McKenney Cc: Alexei Starovoitov Cc: Andrii Nakryiko Cc: Peter Zijlstra Cc: Kent Overstreet Cc: Signed-off-by: Boqun Feng --- include/linux/srcutree.h | 1 - kernel/rcu/srcutree.c | 68 ++++++++++++++++++++-------------------- 2 files changed, 34 insertions(+), 35 deletions(-) diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index 1b01ced61a45..6b7eba59f384 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -100,7 +100,6 @@ struct srcu_usage { * Per-SRCU-domain structure, similar in function to rcu_state. */ struct srcu_struct { - unsigned int srcu_idx; /* Current rdr array element. */ struct srcu_ctr __percpu *srcu_ctrp; struct srcu_data __percpu *sda; /* Per-CPU srcu_data array. */ struct lockdep_map dep_map; diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 7efde1a2344e..247bdf42fb54 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -246,7 +246,6 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp, bool is_static) ssp->srcu_sup->node = NULL; mutex_init(&ssp->srcu_sup->srcu_cb_mutex); mutex_init(&ssp->srcu_sup->srcu_gp_mutex); - ssp->srcu_idx = 0; ssp->srcu_sup->srcu_gp_seq = SRCU_GP_SEQ_INITIAL_VAL; ssp->srcu_sup->srcu_barrier_seq = 0; mutex_init(&ssp->srcu_sup->srcu_barrier_mutex); @@ -510,38 +509,39 @@ static bool srcu_readers_active_idx_check(struct srcu_struct *ssp, int idx) * If the locks are the same as the unlocks, then there must have * been no readers on this index at some point in this function. * But there might be more readers, as a task might have read - * the current ->srcu_idx but not yet have incremented its CPU's + * the current ->srcu_ctrp but not yet have incremented its CPU's * ->srcu_ctrs[idx].srcu_locks counter. In fact, it is possible * that most of the tasks have been preempted between fetching - * ->srcu_idx and incrementing ->srcu_ctrs[idx].srcu_locks. And there - * could be almost (ULONG_MAX / sizeof(struct task_struct)) tasks - * in a system whose address space was fully populated with memory. - * Call this quantity Nt. + * ->srcu_ctrp and incrementing ->srcu_ctrs[idx].srcu_locks. And + * there could be almost (ULONG_MAX / sizeof(struct task_struct)) + * tasks in a system whose address space was fully populated + * with memory. Call this quantity Nt. * - * So suppose that the updater is preempted at this point in the - * code for a long time. That now-preempted updater has already - * flipped ->srcu_idx (possibly during the preceding grace period), - * done an smp_mb() (again, possibly during the preceding grace - * period), and summed up the ->srcu_ctrs[idx].srcu_unlocks counters. - * How many times can a given one of the aforementioned Nt tasks - * increment the old ->srcu_idx value's ->srcu_ctrs[idx].srcu_locks - * counter, in the absence of nesting? + * So suppose that the updater is preempted at this + * point in the code for a long time. That now-preempted + * updater has already flipped ->srcu_ctrp (possibly during + * the preceding grace period), done an smp_mb() (again, + * possibly during the preceding grace period), and summed up + * the ->srcu_ctrs[idx].srcu_unlocks counters. How many times + * can a given one of the aforementioned Nt tasks increment the + * old ->srcu_ctrp value's ->srcu_ctrs[idx].srcu_locks counter, + * in the absence of nesting? * * It can clearly do so once, given that it has already fetched - * the old value of ->srcu_idx and is just about to use that + * the old value of ->srcu_ctrp and is just about to use that * value to index its increment of ->srcu_ctrs[idx].srcu_locks. * But as soon as it leaves that SRCU read-side critical section, * it will increment ->srcu_ctrs[idx].srcu_unlocks, which must - * follow the updater's above read from that same value. Thus, - * as soon the reading task does an smp_mb() and a later fetch from - * ->srcu_idx, that task will be guaranteed to get the new index. + * follow the updater's above read from that same value. Thus, + as soon the reading task does an smp_mb() and a later fetch from + * ->srcu_ctrp, that task will be guaranteed to get the new index. * Except that the increment of ->srcu_ctrs[idx].srcu_unlocks * in __srcu_read_unlock() is after the smp_mb(), and the fetch - * from ->srcu_idx in __srcu_read_lock() is before the smp_mb(). - * Thus, that task might not see the new value of ->srcu_idx until + * from ->srcu_ctrp in __srcu_read_lock() is before the smp_mb(). + * Thus, that task might not see the new value of ->srcu_ctrp until * the -second- __srcu_read_lock(), which in turn means that this * task might well increment ->srcu_ctrs[idx].srcu_locks for the - * old value of ->srcu_idx twice, not just once. + * old value of ->srcu_ctrp twice, not just once. * * However, it is important to note that a given smp_mb() takes * effect not just for the task executing it, but also for any @@ -1095,7 +1095,7 @@ static void srcu_funnel_gp_start(struct srcu_struct *ssp, struct srcu_data *sdp, /* * Wait until all readers counted by array index idx complete, but * loop an additional time if there is an expedited grace period pending. - * The caller must ensure that ->srcu_idx is not changed while checking. + * The caller must ensure that ->srcu_ctrp is not changed while checking. */ static bool try_check_zero(struct srcu_struct *ssp, int idx, int trycount) { @@ -1113,14 +1113,14 @@ static bool try_check_zero(struct srcu_struct *ssp, int idx, int trycount) } /* - * Increment the ->srcu_idx counter so that future SRCU readers will + * Increment the ->srcu_ctrp counter so that future SRCU readers will * use the other rank of the ->srcu_(un)lock_count[] arrays. This allows * us to wait for pre-existing readers in a starvation-free manner. */ static void srcu_flip(struct srcu_struct *ssp) { /* - * Because the flip of ->srcu_idx is executed only if the + * Because the flip of ->srcu_ctrp is executed only if the * preceding call to srcu_readers_active_idx_check() found that * the ->srcu_ctrs[].srcu_unlocks and ->srcu_ctrs[].srcu_locks sums * matched and because that summing uses atomic_long_read(), @@ -1128,15 +1128,15 @@ static void srcu_flip(struct srcu_struct *ssp) * summing and the WRITE_ONCE() in this call to srcu_flip(). * This ordering ensures that if this updater saw a given reader's * increment from __srcu_read_lock(), that reader was using a value - * of ->srcu_idx from before the previous call to srcu_flip(), + * of ->srcu_ctrp from before the previous call to srcu_flip(), * which should be quite rare. This ordering thus helps forward * progress because the grace period could otherwise be delayed * by additional calls to __srcu_read_lock() using that old (soon - * to be new) value of ->srcu_idx. + * to be new) value of ->srcu_ctrp. * * This sum-equality check and ordering also ensures that if * a given call to __srcu_read_lock() uses the new value of - * ->srcu_idx, this updater's earlier scans cannot have seen + * ->srcu_ctrp, this updater's earlier scans cannot have seen * that reader's increments, which is all to the good, because * this grace period need not wait on that reader. After all, * if those earlier scans had seen that reader, there would have @@ -1151,7 +1151,6 @@ static void srcu_flip(struct srcu_struct *ssp) */ smp_mb(); /* E */ /* Pairs with B and C. */ - WRITE_ONCE(ssp->srcu_idx, ssp->srcu_idx + 1); // Flip the counter. WRITE_ONCE(ssp->srcu_ctrp, &ssp->sda->srcu_ctrs[!(ssp->srcu_ctrp - &ssp->sda->srcu_ctrs[0])]); @@ -1466,8 +1465,9 @@ EXPORT_SYMBOL_GPL(synchronize_srcu_expedited); * * Wait for the count to drain to zero of both indexes. To avoid the * possible starvation of synchronize_srcu(), it waits for the count of - * the index=((->srcu_idx & 1) ^ 1) to drain to zero at first, - * and then flip the srcu_idx and wait for the count of the other index. + * the index=!(ssp->srcu_ctrp - &ssp->sda->srcu_ctrs[0]) to drain to zero + * at first, and then flip the ->srcu_ctrp and wait for the count of the + * other index. * * Can block; must be called from process context. * @@ -1693,7 +1693,7 @@ static void srcu_advance_state(struct srcu_struct *ssp) /* * Because readers might be delayed for an extended period after - * fetching ->srcu_idx for their index, at any point in time there + * fetching ->srcu_ctrp for their index, at any point in time there * might well be readers using both idx=0 and idx=1. We therefore * need to wait for readers to clear from both index values before * invoking a callback. @@ -1721,7 +1721,7 @@ static void srcu_advance_state(struct srcu_struct *ssp) } if (rcu_seq_state(READ_ONCE(ssp->srcu_sup->srcu_gp_seq)) == SRCU_STATE_SCAN1) { - idx = 1 ^ (ssp->srcu_idx & 1); + idx = !(ssp->srcu_ctrp - &ssp->sda->srcu_ctrs[0]); if (!try_check_zero(ssp, idx, 1)) { mutex_unlock(&ssp->srcu_sup->srcu_gp_mutex); return; /* readers present, retry later. */ @@ -1739,7 +1739,7 @@ static void srcu_advance_state(struct srcu_struct *ssp) * SRCU read-side critical sections are normally short, * so check at least twice in quick succession after a flip. */ - idx = 1 ^ (ssp->srcu_idx & 1); + idx = !(ssp->srcu_ctrp - &ssp->sda->srcu_ctrs[0]); if (!try_check_zero(ssp, idx, 2)) { mutex_unlock(&ssp->srcu_sup->srcu_gp_mutex); return; /* readers present, retry later. */ @@ -1897,7 +1897,7 @@ void srcu_torture_stats_print(struct srcu_struct *ssp, char *tt, char *tf) int ss_state = READ_ONCE(ssp->srcu_sup->srcu_size_state); int ss_state_idx = ss_state; - idx = ssp->srcu_idx & 0x1; + idx = ssp->srcu_ctrp - &ssp->sda->srcu_ctrs[0]; if (ss_state < 0 || ss_state >= ARRAY_SIZE(srcu_size_state_name)) ss_state_idx = ARRAY_SIZE(srcu_size_state_name) - 1; pr_alert("%s%s Tree SRCU g%ld state %d (%s)", From patchwork Mon Feb 24 02:22:01 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boqun Feng X-Patchwork-Id: 13987320 Received: from mail-qv1-f53.google.com (mail-qv1-f53.google.com [209.85.219.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 864FF1C84B5; Mon, 24 Feb 2025 02:22:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363750; cv=none; b=aEpLJGVwgB/36uQ51h5Yu+Ujg3E/jt30wnV7yoYA5LwXydsDAkkoJ2MiCvKaKoXe9LH8Bfd70q8P+EkuxKuG//D+FpVvz0l3SeUe+ntQ3Hs3B1Z+qjFGxYa7H8YBsg4yGbjTFelt7hTCEzD4+XORiMB0uEKJDHEhiNtBKTgfo1A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363750; c=relaxed/simple; bh=jXfx3DcR/gIMfS6LpI1FhxoXAND9nxcBAQS0dO6cBLA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=iuxqGCJ0pxL4X5S8Wz7iyQZMtymJkenzfIaDdyOmYBXlu/BQO+TOD7sk9EkPbtDP/xr2ar/yYNVLKoRfDlFrUOxl9GzA57j3eNJZuwO2zD0KXYfLZBNp+UV1RB4QSBiyJNvUNRbS+ri95nNVk8ph2B1PnBu/af8v6FLW7rhh0Yc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=gGWzCXtQ; arc=none smtp.client-ip=209.85.219.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="gGWzCXtQ" Received: by mail-qv1-f53.google.com with SMTP id 6a1803df08f44-6e65e0a3f7eso17748536d6.0; Sun, 23 Feb 2025 18:22:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740363747; x=1740968547; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=NgjdExsgtBixOhKRmg8fzqCy/iqJaNe7nEbvDmdzUZw=; b=gGWzCXtQKhfi/UThlcLtwCXxc+jJD4wajILX16xM08uvwbaS3w4PdDQ5UFM1wQDvj5 1/wcuch0GTSAPTHtp8SSEVjRFkCRFiRWldw4jSR2qwOwNP6eU8M9+3XwmqwRqT8KpXnm kxsH4IQairg5rCAa1GwvNcFIfBhg707GxwLL7pr5XHP2MhTreeSKJ1jXmRHLbV0HWVms GElavAHTc++XHMGhAnJr6AqchJozGisMJeE1ZIoukA/bD72EONES/mILOSPMS5eMvxJ7 Hn9yhS0KYE538Awed6CrkQyMtGL8v6FWc5cI9aTlZkLPuXpuLTYHDGz8HesNoKJuaITg fNGA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740363747; x=1740968547; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=NgjdExsgtBixOhKRmg8fzqCy/iqJaNe7nEbvDmdzUZw=; b=LvgjcxpWzKlMwy8iICQ9dB/lj7Kp4oOa8l+NSdv4u+2j/l+Z4K0ODqGHla4XL/bPOE kyb5jt8J1o7B4tfPq3fg7XJJHrGNJosTMwtEXppgEIexztpXkZvUDnkl4A039VB6FjsU csmrys9oUJEj4Pu4GcV3cPYZikkAnGTotjD1BWG1XIiDPX6UGDSh7EL65v8y0LJZop5c jwX9z/w0eX8TNpg4QH3fETrVgD6tVnCBGlyJn/eK1v9zzy1yLvaL0utmWMnayUlWdmiM YOfK0Zar/xF5+tWqef/9vsJKmZVP37tGma4rBcTE4dI40gjuhcPs5juqPBLiK4tTgEJi zV/A== X-Forwarded-Encrypted: i=1; AJvYcCV60UnT9NI+O5Px9mE9A+jayLmWHTDfmuazsELy0WlCTenZez3hSBT3z/+SaS1CR15klGo=@vger.kernel.org, AJvYcCVuHiXgyj+iB2riE5/MVGwnJlMPdqqz86g4LlVSJm31KPM7FYcA0yvwbXjH1m0Bc3RNz7Udst73Jjlp9onY@vger.kernel.org, AJvYcCW/bcysgTJO2Ohuf8ZPm+R9FA79yP1F8pRv3I6Ioxdg8cpbXmogyiQjXBvAkeaN0zJvhTwthEeWPfTD/nyPgpmT@vger.kernel.org X-Gm-Message-State: AOJu0Yy5/yN/AzJTpHA9sQp2AOl7fSStATZUm9O7XO37uGxM4433DV2x R3QEJaw6WZc0XOH+5ZLqeRyI5Y625PKbfoSJELbd3w55y6fEfuh0 X-Gm-Gg: ASbGnctjHjjh6c6EtmZqFmTSpAhv0asLSZYwsHsP92Gp/M/WjHDpmwIcbNt4qJiLzi/ d1Wmjt8MEKqgOJ9fueI/QOgVixjGtyr2YpjfgvANYWQHiSmrt+8/laigVUTAyTkrQJtwINOtIOD nBYQ4Mx/3GvBwVTN+g3mWAJGO+B3i84mitSemjhgibdz8sqRXYkq6GJwu9uvfHFn2wC+DinrXlc t2feIQPOkm4ugoJllka1/O0pMaaKh4IceozBcazhLpWm3I25wcBPj4I0oYAaFs6lj+98ZdzCrMV Zz83Pz6nuWU4Lqgyg0OagfQbtjGzFcMriv7MKkuNDjfvgqSQWBpkFqZqWQIaDSvCP9gpS9TQDJ2 KW060J/gIaIv6HIv/ X-Google-Smtp-Source: AGHT+IHgsDZI/0jVHQs54YPHxU/PatQqYvtF2q+bMn6TEOIqoQinIEowN8zhQPDQGC/qynR4HXp0gA== X-Received: by 2002:a05:6214:48c:b0:6e6:6089:4978 with SMTP id 6a1803df08f44-6e6ae82f74dmr134500206d6.24.1740363747192; Sun, 23 Feb 2025 18:22:27 -0800 (PST) Received: from fauth-a2-smtp.messagingengine.com (fauth-a2-smtp.messagingengine.com. [103.168.172.201]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6e65d779266sm128758466d6.25.2025.02.23.18.22.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 23 Feb 2025 18:22:26 -0800 (PST) Received: from phl-compute-08.internal (phl-compute-08.phl.internal [10.202.2.48]) by mailfauth.phl.internal (Postfix) with ESMTP id 35ACF1200043; Sun, 23 Feb 2025 21:22:26 -0500 (EST) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-08.internal (MEProxy); Sun, 23 Feb 2025 21:22:26 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgdejjeehgecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvfevufffkffojghfggfgsedtkeertdertddt necuhfhrohhmpeeuohhquhhnucfhvghnghcuoegsohhquhhnrdhfvghnghesghhmrghilh drtghomheqnecuggftrfgrthhtvghrnhepgeeljeeitdehvdehgefgjeevfeejjeekgfev ffeiueejhfeuiefggeeuheeggefgnecuvehluhhsthgvrhfuihiivgepfeenucfrrghrrg hmpehmrghilhhfrhhomhepsghoqhhunhdomhgvshhmthhprghuthhhphgvrhhsohhnrghl ihhthidqieelvdeghedtieegqddujeejkeehheehvddqsghoqhhunhdrfhgvnhhgpeepgh hmrghilhdrtghomhesfhhigihmvgdrnhgrmhgvpdhnsggprhgtphhtthhopedvvddpmhho uggvpehsmhhtphhouhhtpdhrtghpthhtoheprhgtuhesvhhgvghrrdhkvghrnhgvlhdroh hrghdprhgtphhtthhopehjihgrnhhgshhhrghnlhgrihesghhmrghilhdrtghomhdprhgt phhtthhopehprghulhhmtghksehkvghrnhgvlhdrohhrghdprhgtphhtthhopehjohhshh esjhhoshhhthhrihhplhgvthhtrdhorhhgpdhrtghpthhtoheprhhoshhtvgguthesghho ohgumhhishdrohhrghdprhgtphhtthhopehmrghthhhivghurdguvghsnhhohigvrhhsse gvfhhfihgtihhoshdrtghomhdprhgtphhtthhopehfrhgvuggvrhhitgeskhgvrhhnvghl rdhorhhgpdhrtghpthhtohepnhgvvghrrghjrdhuphgrughhhigrhieskhgvrhhnvghlrd horhhgpdhrtghpthhtohepjhhovghlsehjohgvlhhfvghrnhgrnhguvghsrdhorhhg X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 23 Feb 2025 21:22:25 -0500 (EST) From: Boqun Feng To: rcu@vger.kernel.org Cc: Lai Jiangshan , "Paul E. McKenney" , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Uladzislau Rezki , Zqiang , Davidlohr Bueso , Shuah Khan , linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, bpf@vger.kernel.org, syzbot+16a19b06125a2963eaee@syzkaller.appspotmail.com, Alexei Starovoitov , Andrii Nakryiko , Peter Zijlstra , Kent Overstreet Subject: [PATCH rcu 07/20] srcu: Force synchronization for srcu_get_delay() Date: Sun, 23 Feb 2025 18:22:01 -0800 Message-Id: <20250224022214.12037-8-boqun.feng@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250224022214.12037-1-boqun.feng@gmail.com> References: <20250224022214.12037-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" Currently, srcu_get_delay() can be called concurrently, for example, by a CPU that is the first to request a new grace period and the CPU processing the current grace period. Although concurrent access is harmless, it unnecessarily expands the state space. Additionally, all calls to srcu_get_delay() are from slow paths. This commit therefore protects all calls to srcu_get_delay() with ssp->srcu_sup->lock, which is already held on the invocation from the srcu_funnel_gp_start() function. While in the area, this commit also adds a lockdep_assert_held() to srcu_get_delay() itself. Reported-by: syzbot+16a19b06125a2963eaee@syzkaller.appspotmail.com Signed-off-by: Paul E. McKenney Cc: Alexei Starovoitov Cc: Andrii Nakryiko Cc: Peter Zijlstra Cc: Kent Overstreet Cc: Signed-off-by: Boqun Feng --- kernel/rcu/srcutree.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 247bdf42fb54..121dd290cae1 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -648,6 +648,7 @@ static unsigned long srcu_get_delay(struct srcu_struct *ssp) unsigned long jbase = SRCU_INTERVAL; struct srcu_usage *sup = ssp->srcu_sup; + lockdep_assert_held(&ACCESS_PRIVATE(ssp->srcu_sup, lock)); if (srcu_gp_is_expedited(ssp)) jbase = 0; if (rcu_seq_state(READ_ONCE(sup->srcu_gp_seq))) { @@ -675,9 +676,13 @@ static unsigned long srcu_get_delay(struct srcu_struct *ssp) void cleanup_srcu_struct(struct srcu_struct *ssp) { int cpu; + unsigned long delay; struct srcu_usage *sup = ssp->srcu_sup; - if (WARN_ON(!srcu_get_delay(ssp))) + spin_lock_irq_rcu_node(ssp->srcu_sup); + delay = srcu_get_delay(ssp); + spin_unlock_irq_rcu_node(ssp->srcu_sup); + if (WARN_ON(!delay)) return; /* Just leak it! */ if (WARN_ON(srcu_readers_active(ssp))) return; /* Just leak it! */ @@ -1101,7 +1106,9 @@ static bool try_check_zero(struct srcu_struct *ssp, int idx, int trycount) { unsigned long curdelay; + spin_lock_irq_rcu_node(ssp->srcu_sup); curdelay = !srcu_get_delay(ssp); + spin_unlock_irq_rcu_node(ssp->srcu_sup); for (;;) { if (srcu_readers_active_idx_check(ssp, idx)) @@ -1850,7 +1857,9 @@ static void process_srcu(struct work_struct *work) ssp = sup->srcu_ssp; srcu_advance_state(ssp); + spin_lock_irq_rcu_node(ssp->srcu_sup); curdelay = srcu_get_delay(ssp); + spin_unlock_irq_rcu_node(ssp->srcu_sup); if (curdelay) { WRITE_ONCE(sup->reschedule_count, 0); } else { From patchwork Mon Feb 24 02:22:02 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boqun Feng X-Patchwork-Id: 13987321 Received: from mail-qv1-f50.google.com (mail-qv1-f50.google.com [209.85.219.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 83FC61D86C6; Mon, 24 Feb 2025 02:22:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363752; cv=none; b=aEvUdNqlpqFbpA1t0/9s7fyYC1ZN23bBYsTi/jDq3K3OyIGymzbDxQUlPsI18qGSI+WpF2YU2s+yLQyfXHLauZ2ouCcQOgH1jPEArpLInCVj7XuxpFM4RCjDABpMVj13my5kp1jvAviLTife5o5xSU+Zy6rlCV3z/m4tGwVgR0U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363752; c=relaxed/simple; bh=/9l1ioIqmq/Hpm/G/Z1p7cG+v9hvH+kj+ihFoubN/j8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=bEvj1osPqBVYJRfgQy8R23sRkbBosmucYm8sK42h1jAxA4WUGvn7Bch913/nVM6H7lykxgngSUP3SDp2q8cbaHJ0cxH//7so3NbWWrGNQlbREib4VCAu2oobtWdNj4e6RUBWlE/uqYND57PTlG/wvmilZt95em4SxRynn2MwkGw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=QK2NElW1; arc=none smtp.client-ip=209.85.219.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="QK2NElW1" Received: by mail-qv1-f50.google.com with SMTP id 6a1803df08f44-6e67f377236so32018176d6.1; Sun, 23 Feb 2025 18:22:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740363749; x=1740968549; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=fYMRQ6Hd0Y9BVpEL39V5BvX9VXJ2sUq5YH9vLc7Ts4Y=; b=QK2NElW1rhlj58o8BUG188Zp91jbJ3A73bSHYwoUdrooJqq4T2eOGscnCwFgUveoi2 QMPed5ccUIRJQnfLpxqhVxloD7j4Z+9K8z/z6tUQdKN6rmpuQr5DgXn0f3iB1GLLYueQ oYueNfKRAleLi5lBx8xHiyTeupTe+QUYZQ8DFuDgWQDiAaW+drjqOsDKdG8qKNCFXzXK TfXDCngwdz0X3bF+x2+vLTqNS2GY9OxTMIndpGRrszK2lw7VE3P+VkFb/VyCQTTpNfck QrKlgAj6v9+sJ+LdtNt+qV3j2w90y9FWFri6tJrfW4vUFIK+GPH+XVaBVAs9it7PJr1j PlZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740363749; x=1740968549; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=fYMRQ6Hd0Y9BVpEL39V5BvX9VXJ2sUq5YH9vLc7Ts4Y=; b=W1qmnV1jml8wTbVvT4/UKHTWxdQFzLnYm3mK2NZ52ZgljTH/3YvivNPmUJVrGkr1p7 emwV0nostRh2TyrO3LiMOaQuTeuAIJXJFkFDSb5DLZiIc//6yJrmrOSQYleZIS4OgKXO /H5ATtUlDii2sTwPs+y0IsfyN4F0DGwolfspp8XBd8f1xxhEmJy0cKXcH1hwmAsYxOaJ +tsAPy0+N5OQ7InZtXd4iVirLpdcvTex6XO2tGETddW/L1/CvMyfgJ3fMBovqVjneMXB EXbT3T+JzOas3AVFLxvDDWfaif+cAf3vllqXKntdNFRWSPnPliXliog0s2zSZV4MhSib VUuA== X-Forwarded-Encrypted: i=1; AJvYcCU7P/XtN6Yu7Wa6jj4Hd6KjJiqB/4VMnp9AkOP8wJMEfYpI6pCdT7u8HNIsKAu4wkl6e3HrUjbEuPHVvRMbbesk@vger.kernel.org, AJvYcCUmfpGGIWG5jtr1TSv6b5sgFUCHbFGKtukRuy9wxruczzupQzhux+YT5IhD3/F622jY9/M=@vger.kernel.org, AJvYcCVBECuNADc+6TIHXEzBefQlxtTTjSed3j3YqMpm4cpustNJlSNedjbm1+BNgS+XL5oxzKQfhx9jWe1Td6MR@vger.kernel.org X-Gm-Message-State: AOJu0YyJNvOiBka9wC5JTSx3/CSQluiIsSZbrwCxa4hcAfou8onbREK9 oyjH3Hrc3ZpaOSFNKo07WswwVuusU7gj8WEBta0W7xTfIuTk4yXa X-Gm-Gg: ASbGnctiQp/UhMPszkXHiOql9PqoOZC1GjtrJidCvTWQC8fVTxW2H83j4S6IdxjLbfD TjZu0zIdQ39u6sa8fTspaDM4pYYILfUm5ljlBXIBqNUIMv9SbY3zzzcXWCvgRek5CMrJDbrJ6x/ 2xniM6pcK2IVJHUfYgdMNGGLJbdHOxolecLO1aybS7fmshUUi2JX/olhS3Bw6SR5Y+RAzMY9k2x qlhgvKtWv8+5sWyAZroeLJB3ixTkYrdAJH5Vt5ramkCUm7cJZMRP6kqHcIu5Ija7h+2OgtYLriC qi1KdDSQvpfOUxO+IC3gH6fdMPnw9+qeD1NrjiipU7dzmy92WTuWorb+C+KFcl/rC1d28+xzVZ5 /x17nj8fsO3IaM4He X-Google-Smtp-Source: AGHT+IHClRbnQ4ctEnkp5istwWDgkUN2nCCWyWoj8RsUc0KxJ45c4T22hIVU190OvqEKtZUeDqyb/A== X-Received: by 2002:a05:6214:194c:b0:6e6:5d9a:9171 with SMTP id 6a1803df08f44-6e6b00fc43bmr180572626d6.23.1740363749320; Sun, 23 Feb 2025 18:22:29 -0800 (PST) Received: from fauth-a2-smtp.messagingengine.com (fauth-a2-smtp.messagingengine.com. [103.168.172.201]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6e65d779201sm128381556d6.2.2025.02.23.18.22.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 23 Feb 2025 18:22:29 -0800 (PST) Received: from phl-compute-03.internal (phl-compute-03.phl.internal [10.202.2.43]) by mailfauth.phl.internal (Postfix) with ESMTP id 79A0D1200043; Sun, 23 Feb 2025 21:22:28 -0500 (EST) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-03.internal (MEProxy); Sun, 23 Feb 2025 21:22:28 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgdejjeehgecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvfevufffkffojghfggfgsedtkeertdertddt necuhfhrohhmpeeuohhquhhnucfhvghnghcuoegsohhquhhnrdhfvghnghesghhmrghilh drtghomheqnecuggftrfgrthhtvghrnhepgeeljeeitdehvdehgefgjeevfeejjeekgfev ffeiueejhfeuiefggeeuheeggefgnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrg hmpehmrghilhhfrhhomhepsghoqhhunhdomhgvshhmthhprghuthhhphgvrhhsohhnrghl ihhthidqieelvdeghedtieegqddujeejkeehheehvddqsghoqhhunhdrfhgvnhhgpeepgh hmrghilhdrtghomhesfhhigihmvgdrnhgrmhgvpdhnsggprhgtphhtthhopedvuddpmhho uggvpehsmhhtphhouhhtpdhrtghpthhtoheprhgtuhesvhhgvghrrdhkvghrnhgvlhdroh hrghdprhgtphhtthhopehjihgrnhhgshhhrghnlhgrihesghhmrghilhdrtghomhdprhgt phhtthhopehprghulhhmtghksehkvghrnhgvlhdrohhrghdprhgtphhtthhopehjohhshh esjhhoshhhthhrihhplhgvthhtrdhorhhgpdhrtghpthhtoheprhhoshhtvgguthesghho ohgumhhishdrohhrghdprhgtphhtthhopehmrghthhhivghurdguvghsnhhohigvrhhsse gvfhhfihgtihhoshdrtghomhdprhgtphhtthhopehfrhgvuggvrhhitgeskhgvrhhnvghl rdhorhhgpdhrtghpthhtohepnhgvvghrrghjrdhuphgrughhhigrhieskhgvrhhnvghlrd horhhgpdhrtghpthhtohepjhhovghlsehjohgvlhhfvghrnhgrnhguvghsrdhorhhg X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 23 Feb 2025 21:22:27 -0500 (EST) From: Boqun Feng To: rcu@vger.kernel.org Cc: Lai Jiangshan , "Paul E. McKenney" , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Uladzislau Rezki , Zqiang , Davidlohr Bueso , Shuah Khan , linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, bpf@vger.kernel.org, Alexei Starovoitov , Andrii Nakryiko , Peter Zijlstra , Kent Overstreet Subject: [PATCH rcu 08/20] srcu: Rename srcu_check_read_flavor_lite() to srcu_check_read_flavor_force() Date: Sun, 23 Feb 2025 18:22:02 -0800 Message-Id: <20250224022214.12037-9-boqun.feng@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250224022214.12037-1-boqun.feng@gmail.com> References: <20250224022214.12037-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" This commit renames the srcu_check_read_flavor_lite() function to srcu_check_read_flavor_force() and adds a read_flavor argument in order to support an srcu_read_lock_fast() variant that is to avoid array indexing in both the lock and unlock primitives. Signed-off-by: Paul E. McKenney Cc: Alexei Starovoitov Cc: Andrii Nakryiko Cc: Peter Zijlstra Cc: Kent Overstreet Cc: Signed-off-by: Boqun Feng --- include/linux/srcu.h | 2 +- include/linux/srcutiny.h | 2 +- include/linux/srcutree.h | 10 ++++++---- 3 files changed, 8 insertions(+), 6 deletions(-) diff --git a/include/linux/srcu.h b/include/linux/srcu.h index f6f779b9d9ff..ca00b9af7c23 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -279,7 +279,7 @@ static inline int srcu_read_lock_lite(struct srcu_struct *ssp) __acquires(ssp) { int retval; - srcu_check_read_flavor_lite(ssp); + srcu_check_read_flavor_force(ssp, SRCU_READ_FLAVOR_LITE); retval = __srcu_read_lock_lite(ssp); rcu_try_lock_acquire(&ssp->dep_map); return retval; diff --git a/include/linux/srcutiny.h b/include/linux/srcutiny.h index 1321da803274..b347bde1aac2 100644 --- a/include/linux/srcutiny.h +++ b/include/linux/srcutiny.h @@ -82,7 +82,7 @@ static inline void srcu_barrier(struct srcu_struct *ssp) } #define srcu_check_read_flavor(ssp, read_flavor) do { } while (0) -#define srcu_check_read_flavor_lite(ssp) do { } while (0) +#define srcu_check_read_flavor_force(ssp, read_flavor) do { } while (0) /* Defined here to avoid size increase for non-torture kernels. */ static inline void srcu_torture_stats_print(struct srcu_struct *ssp, diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index 6b7eba59f384..e29cc57eac81 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -251,16 +251,18 @@ static inline void __srcu_read_unlock_lite(struct srcu_struct *ssp, int idx) void __srcu_check_read_flavor(struct srcu_struct *ssp, int read_flavor); -// Record _lite() usage even for CONFIG_PROVE_RCU=n kernels. -static inline void srcu_check_read_flavor_lite(struct srcu_struct *ssp) +// Record reader usage even for CONFIG_PROVE_RCU=n kernels. This is +// needed only for flavors that require grace-period smp_mb() calls to be +// promoted to synchronize_rcu(). +static inline void srcu_check_read_flavor_force(struct srcu_struct *ssp, int read_flavor) { struct srcu_data *sdp = raw_cpu_ptr(ssp->sda); - if (likely(READ_ONCE(sdp->srcu_reader_flavor) & SRCU_READ_FLAVOR_LITE)) + if (likely(READ_ONCE(sdp->srcu_reader_flavor) & read_flavor)) return; // Note that the cmpxchg() in __srcu_check_read_flavor() is fully ordered. - __srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_LITE); + __srcu_check_read_flavor(ssp, read_flavor); } // Record non-_lite() usage only for CONFIG_PROVE_RCU=y kernels. From patchwork Mon Feb 24 02:22:03 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boqun Feng X-Patchwork-Id: 13987322 Received: from mail-qv1-f45.google.com (mail-qv1-f45.google.com [209.85.219.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AB67B1DF988; Mon, 24 Feb 2025 02:22:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363754; cv=none; b=eCvmhZmTNYj8gR7cqY8ps5E9dEFEUWwN5KxrDhAGA57Y/kw+7FSvLZ9lsYw8ZxBKvPv1jEmhSQOwYPwzzQ4HZKCgjgM/2Jk8u+9aBP928v+nhYME0p5vJzvvJyeOxzvf4gd5xPLa0e3Hx0warTyv6zXamY28QpdSd4LsVOGD1ps= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363754; c=relaxed/simple; bh=FJ4Rb41z1ufJ+wTZ0uFJtruWUxDLdj5Ng58pCay5gII=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=EuoXERcY7lq5+rP3vf4ITWMicX9XLTze8WkOhc8I1XojZjU4Wx0QqvujXZy+VW9spYQxBRDMNmDA9/uc2ZEEXqmxt5JqY4UoFNRRwmUp+XMMT9d3I7q+qjAqFs0jpCBC37zxhBV+3X6tDrEqjui++5TrzaF3ggh8FYbaarBmT44= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Om0+Va3F; arc=none smtp.client-ip=209.85.219.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Om0+Va3F" Received: by mail-qv1-f45.google.com with SMTP id 6a1803df08f44-6df83fd01cbso17405806d6.2; Sun, 23 Feb 2025 18:22:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740363751; x=1740968551; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=kUQeyMIXkWewc+GJgkcRvcJST3TpV64E6KpD7DCbPAI=; b=Om0+Va3F/ozLUm1PmUc9VuI414kcUD70dXinX7R37W7iBJgbn3BPbgwF9EqVKGJUKK zTS4spNGFh2fma1NWqQwtuVjvRiGrSmCAUIkm1yu8kuZyjtgMVfxf4nhdbJdjCliS2tT 4DwSkFOC7cvX15qxwbVf47wt5kOdf/0CD+1D7bWZT1MaRVFiwSU0gdncsQlFnYVQOpMe z5W0uQ8Bn7SkQx8493g4e+yD/8dRS67i3gaBNkXPPdH6KUqA6lGou7yHWKvMzyZyFQ9R raIP/HJYLV5k8MDd6xgJ1AJFUtnMJ0R6jKKTZPwPVhcTjsNgcQ9UqAC5ivisNRYWGjds 2PGA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740363751; x=1740968551; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=kUQeyMIXkWewc+GJgkcRvcJST3TpV64E6KpD7DCbPAI=; b=bZizlq/Er9ZVbOymMeDpuH5j8tnSvbpt7mC0VpPAGU5zaVFWPLW4vBVHisaOpV7unV SnrNx/mFcQ8mAnTP5b4a4EeNedi9OALR+cKps6amd0ljZ3Vd4Dm4xjVQXjIIrwn0Cbg3 g8iJ3bCvKwp9BAaRZVz2Mu+uliYuHGPQKVZKYs1AeX8tE4S/XSGVfbBl7/jY3MnxCHig XVHGY4Xc4Ndt27D5XvnJcaR5t3UHTpUp5pSR9XND5T+N9/8RzcJ07iFghuz5UjH/ol1i 4nWvZIFdHfV+two8ySSRJTR9sYbDeUe7BCdtKQ+BBCNOUtQGG7gI66tqEYlEm058TYOp y3NQ== X-Forwarded-Encrypted: i=1; AJvYcCU8+3Ba0HmqP+iWrbGNfla/Ak2OlAHAaG8MtN4TLn2ql+HVQz4kjzZGV07e6HKRD+idnKszN+bpx3Kspdrbdtjk@vger.kernel.org, AJvYcCXFwXfMcCMKmUr3nQIKhIHOfdo8MsH1VV8FEOlxCxznceptTAzSaEVVYcZ6yjPBkEdcg8s=@vger.kernel.org, AJvYcCXt3yWa6vwKUXUyXLrxMP3vXI4CSszvB8dRiFOE8EiPxdi8eWuVuoeKhocC1OZ5mpmm2lRwbJwM2YPRpRmf@vger.kernel.org X-Gm-Message-State: AOJu0YySxK9Y7QQ7ZIl3VC3FptffamKi/Qzb8FpWIUXKRgXlazol4/tl o6nw9DaBH0ySAbCAZpvahZpl0T7qrb3fndwzQ836sR/3vUWLcR2B X-Gm-Gg: ASbGncuTELLDp4xxH1ePLry9Tl+edRvHiDgZYMJRQuOPGcOfuB0gw5QqjGGFeFRNSuI Dr/hx2FSLZKLxa8SBXSDS5VRACLrvgcsJUKQtiGDl8c1v1bc/357ESnJfpiyVB+uu7ecJ2sPZRe tdL5GIcJWq8E50McwdFzDF2Vt97Qd7Ub7wt+MnAfh0CTIIbNmkDME8vKb3Jfp4umImE7jDS5UwM vcqPQ+BhQe5Ff3nLb88xpZgdvjyEF7DuIBzDDEuz7koltJPxsDW42/9ZnGxjFdptkA1MPIr/aQD tJfCrmrAKOUV/fRO4A41MctoGYIUD/LLuh2M9QQ1cxxbEIstOKOydxnHpg9wtEns48GPSX4AjQP hqgZFfvZEee9u3/uC X-Google-Smtp-Source: AGHT+IGhKtPOfbP0/vWC3oiyxINHUXqLzfQbfiT0IAYpiHTHPSjosaEHla0jWewE7AnvlRlUVCbdbw== X-Received: by 2002:a05:6214:1c4c:b0:6e6:a60f:24cf with SMTP id 6a1803df08f44-6e6ae829f69mr127020026d6.19.1740363751540; Sun, 23 Feb 2025 18:22:31 -0800 (PST) Received: from fauth-a2-smtp.messagingengine.com (fauth-a2-smtp.messagingengine.com. [103.168.172.201]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6e65dcf86a7sm127798886d6.119.2025.02.23.18.22.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 23 Feb 2025 18:22:31 -0800 (PST) Received: from phl-compute-08.internal (phl-compute-08.phl.internal [10.202.2.48]) by mailfauth.phl.internal (Postfix) with ESMTP id BB41C1200043; Sun, 23 Feb 2025 21:22:30 -0500 (EST) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-08.internal (MEProxy); Sun, 23 Feb 2025 21:22:30 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgdejjeehgecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvfevufffkffojghfggfgsedtkeertdertddt necuhfhrohhmpeeuohhquhhnucfhvghnghcuoegsohhquhhnrdhfvghnghesghhmrghilh drtghomheqnecuggftrfgrthhtvghrnhepgeeljeeitdehvdehgefgjeevfeejjeekgfev ffeiueejhfeuiefggeeuheeggefgnecuvehluhhsthgvrhfuihiivgepfeenucfrrghrrg hmpehmrghilhhfrhhomhepsghoqhhunhdomhgvshhmthhprghuthhhphgvrhhsohhnrghl ihhthidqieelvdeghedtieegqddujeejkeehheehvddqsghoqhhunhdrfhgvnhhgpeepgh hmrghilhdrtghomhesfhhigihmvgdrnhgrmhgvpdhnsggprhgtphhtthhopedvuddpmhho uggvpehsmhhtphhouhhtpdhrtghpthhtoheprhgtuhesvhhgvghrrdhkvghrnhgvlhdroh hrghdprhgtphhtthhopehjihgrnhhgshhhrghnlhgrihesghhmrghilhdrtghomhdprhgt phhtthhopehprghulhhmtghksehkvghrnhgvlhdrohhrghdprhgtphhtthhopehjohhshh esjhhoshhhthhrihhplhgvthhtrdhorhhgpdhrtghpthhtoheprhhoshhtvgguthesghho ohgumhhishdrohhrghdprhgtphhtthhopehmrghthhhivghurdguvghsnhhohigvrhhsse gvfhhfihgtihhoshdrtghomhdprhgtphhtthhopehfrhgvuggvrhhitgeskhgvrhhnvghl rdhorhhgpdhrtghpthhtohepnhgvvghrrghjrdhuphgrughhhigrhieskhgvrhhnvghlrd horhhgpdhrtghpthhtohepjhhovghlsehjohgvlhhfvghrnhgrnhguvghsrdhorhhg X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 23 Feb 2025 21:22:29 -0500 (EST) From: Boqun Feng To: rcu@vger.kernel.org Cc: Lai Jiangshan , "Paul E. McKenney" , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Uladzislau Rezki , Zqiang , Davidlohr Bueso , Shuah Khan , linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, bpf@vger.kernel.org, Alexei Starovoitov , Andrii Nakryiko , Peter Zijlstra , Kent Overstreet Subject: [PATCH rcu 09/20] srcu: Add SRCU_READ_FLAVOR_SLOWGP to flag need for synchronize_rcu() Date: Sun, 23 Feb 2025 18:22:03 -0800 Message-Id: <20250224022214.12037-10-boqun.feng@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250224022214.12037-1-boqun.feng@gmail.com> References: <20250224022214.12037-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" This commit switches from a direct test of SRCU_READ_FLAVOR_LITE to a new SRCU_READ_FLAVOR_SLOWGP macro to check for substituting synchronize_rcu() for smp_mb() in SRCU grace periods. Right now, SRCU_READ_FLAVOR_SLOWGP is exactly SRCU_READ_FLAVOR_LITE, but the addition of the _fast() flavor of SRCU will change that. Signed-off-by: Paul E. McKenney Cc: Alexei Starovoitov Cc: Andrii Nakryiko Cc: Peter Zijlstra Cc: Kent Overstreet Cc: Signed-off-by: Boqun Feng --- include/linux/srcu.h | 3 +++ kernel/rcu/srcutree.c | 6 +++--- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/include/linux/srcu.h b/include/linux/srcu.h index ca00b9af7c23..505f5bdce444 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -49,6 +49,9 @@ int init_srcu_struct(struct srcu_struct *ssp); #define SRCU_READ_FLAVOR_LITE 0x4 // srcu_read_lock_lite(). #define SRCU_READ_FLAVOR_ALL (SRCU_READ_FLAVOR_NORMAL | SRCU_READ_FLAVOR_NMI | \ SRCU_READ_FLAVOR_LITE) // All of the above. +#define SRCU_READ_FLAVOR_SLOWGP SRCU_READ_FLAVOR_LITE + // Flavors requiring synchronize_rcu() + // instead of smp_mb(). #ifdef CONFIG_TINY_SRCU #include diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 121dd290cae1..8b5c50bc98e5 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -449,7 +449,7 @@ static bool srcu_readers_lock_idx(struct srcu_struct *ssp, int idx, bool gp, uns } WARN_ONCE(IS_ENABLED(CONFIG_PROVE_RCU) && (mask & (mask - 1)), "Mixed reader flavors for srcu_struct at %ps.\n", ssp); - if (mask & SRCU_READ_FLAVOR_LITE && !gp) + if (mask & SRCU_READ_FLAVOR_SLOWGP && !gp) return false; return sum == unlocks; } @@ -487,7 +487,7 @@ static bool srcu_readers_active_idx_check(struct srcu_struct *ssp, int idx) unsigned long unlocks; unlocks = srcu_readers_unlock_idx(ssp, idx, &rdm); - did_gp = !!(rdm & SRCU_READ_FLAVOR_LITE); + did_gp = !!(rdm & SRCU_READ_FLAVOR_SLOWGP); /* * Make sure that a lock is always counted if the corresponding @@ -1205,7 +1205,7 @@ static bool srcu_should_expedite(struct srcu_struct *ssp) check_init_srcu_struct(ssp); /* If _lite() readers, don't do unsolicited expediting. */ - if (this_cpu_read(ssp->sda->srcu_reader_flavor) & SRCU_READ_FLAVOR_LITE) + if (this_cpu_read(ssp->sda->srcu_reader_flavor) & SRCU_READ_FLAVOR_SLOWGP) return false; /* If the local srcu_data structure has callbacks, not idle. */ sdp = raw_cpu_ptr(ssp->sda); From patchwork Mon Feb 24 02:22:04 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boqun Feng X-Patchwork-Id: 13987323 Received: from mail-qt1-f172.google.com (mail-qt1-f172.google.com [209.85.160.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 999751A2390; Mon, 24 Feb 2025 02:22:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363756; cv=none; b=K9/OVLAJo/9B2W1AzhCxS0CSJ9q+AboECmbBSqk0MW6Xvj8SBcEdnZfsuEFVWQ7WyAxNpheTNPwztuZ0DhV504QT97LYAN2VCI3tGPaJSgZpj1kLU4rkKWpcMkTEDSmAsHSjV2fB7Z+dMU8kHvj+6s4vOsrr03EWHoUOeRTK/ds= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363756; c=relaxed/simple; bh=TgHuhjSSnyReNfhJRKsOVKxVBGTwz2BqA8pdzW/huhg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=VzZaWczv32ozCbk7ieSP66y2T0ao60QJxeLs+d8oCVWvFNW0Wver755zP4FPbL6g0e9a+UU/t0WQ+xGGDdzoUY+vCc3LT3lb2wcreimUOYNUD/jlo6j7SRYWrynW9YTTjDyDtkZGLM+ag+HIntQrpOqt4N+NL2najYLf2mteXCk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=M4/Go+t2; arc=none smtp.client-ip=209.85.160.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="M4/Go+t2" Received: by mail-qt1-f172.google.com with SMTP id d75a77b69052e-471fe5e0a80so34221141cf.1; Sun, 23 Feb 2025 18:22:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740363753; x=1740968553; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=YGLAsuKK/f3ek23KfjWTWNPSOmHZjb22aDmgZC2NlGs=; b=M4/Go+t2K82cr8/gqI6/u/FWnhuuxGwEaAKiFEeiX+cJHiHUdeKZbK2S2HLg2IT2q1 plsfqURXB9wAX9hzohIc1PzYWEST579uWwHa6Et14zVV/BKnet/MHtPU20YndoTUBN22 Ecapjr636Z9LIAddpV14FfJCktlI8VdIWHJW/bJMNhlCpYgCoX+RIHkSfwA/82aFOudB x1Z47+7vPbQBIl5Y1JdU9wis0kEfDNRaBzbJVIS+6e2RBBUbiIPic/73L/eKvLQf5t0o UPkBLARvgNPTycuQMHon27puRKcAzIIjL1iRlTvaWVhdwNp6VaUQCsWX4JacDBfNt9Vz +piw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740363753; x=1740968553; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=YGLAsuKK/f3ek23KfjWTWNPSOmHZjb22aDmgZC2NlGs=; b=J4o1tRCM51P0eLpzT+Cw6ko0BvWGq/gpDEQK6+dJ8LnpBE4oJjTNZhiEcMDrtOHhnR vAqAp+M8UC8o7p10V3tpBxwqTeF+7BEzNxAGq6Vo6G7Y1zG1EWQDxdmwAcIq91LCtCSL 0Fyb+L2q7q7DORas2AZ3S5nIFlCyBPKkFRgt36uqsplcDNS7LMIu6NASW4TNxMIOp096 Hh4iHHU/PZsPMSqxjVMMWwU1jDVKh3HfjjrKK5jkimBCaE+zJyEs3x7uO/6+ffpbqTSV XrAsydcyl6/e+WtVaVDWzd0lVBj+DdHnGs1yTd97aBO5eRpMsaTh/FnhlWBrlU4QPadd BJ/g== X-Forwarded-Encrypted: i=1; AJvYcCWHnPXicsQ2xC5Fq0wAeEgikdXPb9hyBd+a3AgiUTApG5zmav4QrG+eDJhejwM71NCdeEk=@vger.kernel.org, AJvYcCXHyUkJrUeffTncDx2lY1SQY1dztyS90FwpPRxKFi1+ZqfX0UEqEsAduRmOQnk1CGoMz9x0AqAG2cXcL6re@vger.kernel.org, AJvYcCXgMsN5RxWUDWsrZSJfEkmzqdF9Te9W8MO8tWn2tmkvog6kIONEiFXcUeKhB38n/rj0ZnKn5endN4nDuWCnPzra@vger.kernel.org X-Gm-Message-State: AOJu0Yz1X5RiN/u8eFobXJDYME39RO6ULqjawtqfB1kVL6Gxzl+LTKFg Uy3o0N2O9CX1qPAOjJi5ORyXGa/i3TR8vH7Dj/9w4p2v9Xo4YjOe X-Gm-Gg: ASbGncuYQx4qUn980X2tapHbjBlFaVJVHfVEwfviHJpw+1k2w+AwdNF2jUET5OBMHAk 2UDyQM5ktT/NmTKH9wyDNzSHPYbGf3NRPdHOJAMkPdLC3e5qQmKF7DGFc5J8j9XQFs73WwH/kUg yz8sTR9chhvPuI/u9ep4CBq+sqX7VfaVViZEGjBu40V3iLS219MimjrskjnKExYOf2fRpwCuea2 lr6M3fkhhntRK7dERfBta6icGQ/mhdGDvKUPfQlhdrW8qU7zwbpOzqrdVOZoJf3/YNnSCNIiv4l 97vHnFoFVpF0RIW+WruGw1lII806VeQn2Cyn200sZR1bOKDlB1FdfBcNxDcAL03V1VVrUAHeT+J 6FdAK7fXYjAOE/qDK X-Google-Smtp-Source: AGHT+IGUSog6nwxcNA18fQoC5dGEQkV9c1YwPYJYckbB/HQopiZwY63sHGbjZDO/LBtKd+Y4ciduqg== X-Received: by 2002:a05:622a:10c:b0:472:3e4:53fe with SMTP id d75a77b69052e-47224611fb7mr150683901cf.0.1740363753370; Sun, 23 Feb 2025 18:22:33 -0800 (PST) Received: from fauth-a2-smtp.messagingengine.com (fauth-a2-smtp.messagingengine.com. [103.168.172.201]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-47209a1b133sm62076051cf.70.2025.02.23.18.22.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 23 Feb 2025 18:22:33 -0800 (PST) Received: from phl-compute-03.internal (phl-compute-03.phl.internal [10.202.2.43]) by mailfauth.phl.internal (Postfix) with ESMTP id 78D701200043; Sun, 23 Feb 2025 21:22:32 -0500 (EST) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-03.internal (MEProxy); Sun, 23 Feb 2025 21:22:32 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgdejjeehgecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvfevufffkffojghfggfgsedtkeertdertddt necuhfhrohhmpeeuohhquhhnucfhvghnghcuoegsohhquhhnrdhfvghnghesghhmrghilh drtghomheqnecuggftrfgrthhtvghrnhepgeeljeeitdehvdehgefgjeevfeejjeekgfev ffeiueejhfeuiefggeeuheeggefgnecuvehluhhsthgvrhfuihiivgepudenucfrrghrrg hmpehmrghilhhfrhhomhepsghoqhhunhdomhgvshhmthhprghuthhhphgvrhhsohhnrghl ihhthidqieelvdeghedtieegqddujeejkeehheehvddqsghoqhhunhdrfhgvnhhgpeepgh hmrghilhdrtghomhesfhhigihmvgdrnhgrmhgvpdhnsggprhgtphhtthhopedvuddpmhho uggvpehsmhhtphhouhhtpdhrtghpthhtoheprhgtuhesvhhgvghrrdhkvghrnhgvlhdroh hrghdprhgtphhtthhopehjihgrnhhgshhhrghnlhgrihesghhmrghilhdrtghomhdprhgt phhtthhopehprghulhhmtghksehkvghrnhgvlhdrohhrghdprhgtphhtthhopehjohhshh esjhhoshhhthhrihhplhgvthhtrdhorhhgpdhrtghpthhtoheprhhoshhtvgguthesghho ohgumhhishdrohhrghdprhgtphhtthhopehmrghthhhivghurdguvghsnhhohigvrhhsse gvfhhfihgtihhoshdrtghomhdprhgtphhtthhopehfrhgvuggvrhhitgeskhgvrhhnvghl rdhorhhgpdhrtghpthhtohepnhgvvghrrghjrdhuphgrughhhigrhieskhgvrhhnvghlrd horhhgpdhrtghpthhtohepjhhovghlsehjohgvlhhfvghrnhgrnhguvghsrdhorhhg X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 23 Feb 2025 21:22:31 -0500 (EST) From: Boqun Feng To: rcu@vger.kernel.org Cc: Lai Jiangshan , "Paul E. McKenney" , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Uladzislau Rezki , Zqiang , Davidlohr Bueso , Shuah Khan , linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, bpf@vger.kernel.org, Alexei Starovoitov , Andrii Nakryiko , Peter Zijlstra , Kent Overstreet Subject: [PATCH rcu 10/20] srcu: Pull pointer-to-integer conversion into __srcu_ptr_to_ctr() Date: Sun, 23 Feb 2025 18:22:04 -0800 Message-Id: <20250224022214.12037-11-boqun.feng@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250224022214.12037-1-boqun.feng@gmail.com> References: <20250224022214.12037-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" This commit abstracts the srcu_read_lock*() pointer-to-integer conversion into a new __srcu_ptr_to_ctr(). This will be used in rcutorture for testing an srcu_read_lock_fast() that returns a pointer rather than an integer. Signed-off-by: Paul E. McKenney Cc: Alexei Starovoitov Cc: Andrii Nakryiko Cc: Peter Zijlstra Cc: Kent Overstreet Cc: Signed-off-by: Boqun Feng --- include/linux/srcutree.h | 9 ++++++++- kernel/rcu/srcutree.c | 4 ++-- 2 files changed, 10 insertions(+), 3 deletions(-) diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index e29cc57eac81..f41bb3a55a04 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -211,6 +211,13 @@ void synchronize_srcu_expedited(struct srcu_struct *ssp); void srcu_barrier(struct srcu_struct *ssp); void srcu_torture_stats_print(struct srcu_struct *ssp, char *tt, char *tf); +// Converts a per-CPU pointer to an ->srcu_ctrs[] array element to that +// element's index. +static inline bool __srcu_ptr_to_ctr(struct srcu_struct *ssp, struct srcu_ctr __percpu *scpp) +{ + return scpp - &ssp->sda->srcu_ctrs[0]; +} + /* * Counts the new reader in the appropriate per-CPU element of the * srcu_struct. Returns an index that must be passed to the matching @@ -228,7 +235,7 @@ static inline int __srcu_read_lock_lite(struct srcu_struct *ssp) RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_lock_lite()."); this_cpu_inc(scp->srcu_locks.counter); /* Y */ barrier(); /* Avoid leaking the critical section. */ - return scp - &ssp->sda->srcu_ctrs[0]; + return __srcu_ptr_to_ctr(ssp, scp); } /* diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 8b5c50bc98e5..a91651866485 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -753,7 +753,7 @@ int __srcu_read_lock(struct srcu_struct *ssp) this_cpu_inc(scp->srcu_locks.counter); smp_mb(); /* B */ /* Avoid leaking the critical section. */ - return scp - &ssp->sda->srcu_ctrs[0]; + return __srcu_ptr_to_ctr(ssp, scp); } EXPORT_SYMBOL_GPL(__srcu_read_lock); @@ -783,7 +783,7 @@ int __srcu_read_lock_nmisafe(struct srcu_struct *ssp) atomic_long_inc(&scp->srcu_locks); smp_mb__after_atomic(); /* B */ /* Avoid leaking the critical section. */ - return scpp - &ssp->sda->srcu_ctrs[0]; + return __srcu_ptr_to_ctr(ssp, scpp); } EXPORT_SYMBOL_GPL(__srcu_read_lock_nmisafe); From patchwork Mon Feb 24 02:22:05 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boqun Feng X-Patchwork-Id: 13987324 Received: from mail-qk1-f176.google.com (mail-qk1-f176.google.com [209.85.222.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D58C01A23A4; Mon, 24 Feb 2025 02:22:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363757; cv=none; b=cbc9yTacV7bywOxCzPzMcgzwK6eZ/TGpJkYkYHIaHhu7gLyZFyek6Y/bCGfTnJliG02ahJTO3AJkUpr1yVA0USpbLLDexNllChb1zTVDXMN1nQaOsDCB5byplcDqn+/v+vqJkxgObxrKUysLinZNeccYB7erUR5z7AwtBHxwKZI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363757; c=relaxed/simple; bh=YrHT1FYi1+OzGm9Y+UlE/vWNj9k1nIL2jAmcMpxhAJs=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=GOyCj22EcRv1y0qYhC6Ngi3bhdhjfDmEjtZYSMColdtvweVWA005cLQCAZcEdcgWnp20fsMY3nW3aAm7V+huFLaOJYF2ofCDNnyHR78GAK0jrOcgVWb69nYfx7mZmO15qJL+2EBynBtk8QWXsPW4eiOXlfQ3++jnJftSd/h9pAE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=jup5R1a8; arc=none smtp.client-ip=209.85.222.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="jup5R1a8" Received: by mail-qk1-f176.google.com with SMTP id af79cd13be357-7be8f281714so496476385a.1; Sun, 23 Feb 2025 18:22:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740363755; x=1740968555; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=b4UbglbVJKQ61dofpRaNQb5dypoLHjOhU9tZFxlz6Bc=; b=jup5R1a86jcAl3VM3zxMtAbP0Te58weFlknwBvoT7br5OWuPYqhlP4jwlFkSyKkDP+ vkV6XNas+/pf48gKE1DpqhPO21/xsDkAakLr4tygXorBzFZWOmsbkFkC3dmd+/yKeUs0 U7mh1DFRLwwg1UiOmg4cPwZeqsJ07XVcWSW0sXEoouLMSJ3eH+Cs+Ou1vY1E8tQJ0dOh 95quRa5UNt+CRS8SKgipYSudZbEjibua42Gi3kLmz1zpd4qmWa/tIF8hcuo0ux697kSD 1Q4ojUfpRRzS5p8LWLjZb0qQhu0P82sBG9ZJZgB9D61rJSjiGYnsJToK9lQzcTIZB9p9 qxXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740363755; x=1740968555; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=b4UbglbVJKQ61dofpRaNQb5dypoLHjOhU9tZFxlz6Bc=; b=vkBQQFc11HymwggBzD0LVp8U0/KeGAmFtfQA4/RwKUak4nXsulIq3HSvsEwVIRXJsO U3qxkygVfS8GoSbASljjX9/16/XCznf14WT8nYV82f3ZmswMOPKDyVmxU6KiGZmizwpz yc8XtmEq/S9NC4B4B1h1JtYhrK1moC2B+Ru4qIAG1Jm/HuTLG9oDsXE2yJAzSaLQXW2I TzpyPEqzUygn15AgJJ5P4wW7FMsi81Ha8Y/FRmGOkRZ2DY0W/IrHN/LtkSGhR1qxwhIe cGwhHjxYubbXF2sFfj6yYxHfNvlZjlM+SwHQmI5BntN91qFl12aatszllmgesBcSjsJx 2BHw== X-Forwarded-Encrypted: i=1; AJvYcCV13iIWB4PTaMdNhngWdzO7QWIMm0BhMCTyCXlB5b/nOICy1YSyLJDAKQ+QePcV3DNbcRE=@vger.kernel.org, AJvYcCX+sMxBpt1ahnN7ttznoPbHpjXHfkluO08OAqMHYSmDphWSsEQAFnY+XZZ+gnpCOhgpro/Jq0uDuLPdBGWh@vger.kernel.org, AJvYcCXr8qW9jb0GufzFgjffbeZj0uDs5M8pjRFqCfykF1R7Eh48NXDYUdiD7KHKtWUJ+Kpmt+97bIFhKAYLaTZt2A3D@vger.kernel.org X-Gm-Message-State: AOJu0YxQXKhL2MHqNQP1K2vs+5W/KCgYEcFWYTDU0F+wni0BAWntPgbv LVbuq0fIC5DHk48qAgJtNlhNSac0wcghKgJper5l/OUM1wC7lShZ X-Gm-Gg: ASbGncsdHhkOOCVeqOk7TA9VcCz+8UsThKMiFLq7Zyt2jZ66vXUOydqu53OU3H3ahBk ufCRAffA0CO6rZrzVyFfO1laA3/QNGf256z6NlV9RO9Ds7khGBnFdow9UyfrW2wpLOERl4dcWVL GBvfCuXOvvaA9tu/7K4zGWNJUn8yVHJgFTt6dhuJoL+BH4SFOV+kJFJd9BbAxVGw+Jgv4oPM1Yq vrd9ymtdV4n0YXbPxRXhgo5mxY1okbq4/IjrCyRvodGRH4Zf0/zcfwyDrFnpnxYaMWxKOVYx39a laV00hKkJ6yROP3BdxHR7PZWXVinohR/0FxSM9fsMPQtrxZwiYL34JpYBTSj8LGd4tGhLgzxWYf ewTZ93g1BPTcC97gd X-Google-Smtp-Source: AGHT+IFJ0c4dK35a+uqh/H+mgY4++zlo+aC7ovaC3i7iZFv0MXgycDMLedqpsay7Y6JGaqjxgV3XgA== X-Received: by 2002:a05:620a:1a04:b0:7c0:9df3:a0d7 with SMTP id af79cd13be357-7c0cef49da6mr1415529885a.41.1740363754658; Sun, 23 Feb 2025 18:22:34 -0800 (PST) Received: from fauth-a2-smtp.messagingengine.com (fauth-a2-smtp.messagingengine.com. [103.168.172.201]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6e65d785c09sm127670766d6.36.2025.02.23.18.22.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 23 Feb 2025 18:22:34 -0800 (PST) Received: from phl-compute-11.internal (phl-compute-11.phl.internal [10.202.2.51]) by mailfauth.phl.internal (Postfix) with ESMTP id CCFCF1200043; Sun, 23 Feb 2025 21:22:33 -0500 (EST) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-11.internal (MEProxy); Sun, 23 Feb 2025 21:22:33 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgdejjeehgecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvfevufffkffojghfggfgsedtkeertdertddt necuhfhrohhmpeeuohhquhhnucfhvghnghcuoegsohhquhhnrdhfvghnghesghhmrghilh drtghomheqnecuggftrfgrthhtvghrnhepgeeljeeitdehvdehgefgjeevfeejjeekgfev ffeiueejhfeuiefggeeuheeggefgnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrg hmpehmrghilhhfrhhomhepsghoqhhunhdomhgvshhmthhprghuthhhphgvrhhsohhnrghl ihhthidqieelvdeghedtieegqddujeejkeehheehvddqsghoqhhunhdrfhgvnhhgpeepgh hmrghilhdrtghomhesfhhigihmvgdrnhgrmhgvpdhnsggprhgtphhtthhopedvuddpmhho uggvpehsmhhtphhouhhtpdhrtghpthhtoheprhgtuhesvhhgvghrrdhkvghrnhgvlhdroh hrghdprhgtphhtthhopehjihgrnhhgshhhrghnlhgrihesghhmrghilhdrtghomhdprhgt phhtthhopehprghulhhmtghksehkvghrnhgvlhdrohhrghdprhgtphhtthhopehjohhshh esjhhoshhhthhrihhplhgvthhtrdhorhhgpdhrtghpthhtoheprhhoshhtvgguthesghho ohgumhhishdrohhrghdprhgtphhtthhopehmrghthhhivghurdguvghsnhhohigvrhhsse gvfhhfihgtihhoshdrtghomhdprhgtphhtthhopehfrhgvuggvrhhitgeskhgvrhhnvghl rdhorhhgpdhrtghpthhtohepnhgvvghrrghjrdhuphgrughhhigrhieskhgvrhhnvghlrd horhhgpdhrtghpthhtohepjhhovghlsehjohgvlhhfvghrnhgrnhguvghsrdhorhhg X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 23 Feb 2025 21:22:33 -0500 (EST) From: Boqun Feng To: rcu@vger.kernel.org Cc: Lai Jiangshan , "Paul E. McKenney" , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Uladzislau Rezki , Zqiang , Davidlohr Bueso , Shuah Khan , linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, bpf@vger.kernel.org, Alexei Starovoitov , Andrii Nakryiko , Peter Zijlstra , Kent Overstreet Subject: [PATCH rcu 11/20] srcu: Pull integer-to-pointer conversion into __srcu_ctr_to_ptr() Date: Sun, 23 Feb 2025 18:22:05 -0800 Message-Id: <20250224022214.12037-12-boqun.feng@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250224022214.12037-1-boqun.feng@gmail.com> References: <20250224022214.12037-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" This commit abstracts the srcu_read_unlock*() integer-to-pointer conversion into a new __srcu_ctr_to_ptr(). This will be used in rcutorture for testing an srcu_read_unlock_fast() that avoids array-indexing overhead by taking a pointer rather than an integer. [ paulmck: Apply kernel test robot feedback. ] Signed-off-by: Paul E. McKenney Cc: Alexei Starovoitov Cc: Andrii Nakryiko Cc: Peter Zijlstra Cc: Kent Overstreet Cc: Signed-off-by: Boqun Feng --- include/linux/srcutree.h | 9 ++++++++- kernel/rcu/srcutree.c | 6 ++---- 2 files changed, 10 insertions(+), 5 deletions(-) diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index f41bb3a55a04..55fa400624bb 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -218,6 +218,13 @@ static inline bool __srcu_ptr_to_ctr(struct srcu_struct *ssp, struct srcu_ctr __ return scpp - &ssp->sda->srcu_ctrs[0]; } +// Converts an integer to a per-CPU pointer to the corresponding +// ->srcu_ctrs[] array element. +static inline struct srcu_ctr __percpu *__srcu_ctr_to_ptr(struct srcu_struct *ssp, int idx) +{ + return &ssp->sda->srcu_ctrs[idx]; +} + /* * Counts the new reader in the appropriate per-CPU element of the * srcu_struct. Returns an index that must be passed to the matching @@ -252,7 +259,7 @@ static inline int __srcu_read_lock_lite(struct srcu_struct *ssp) static inline void __srcu_read_unlock_lite(struct srcu_struct *ssp, int idx) { barrier(); /* Avoid leaking the critical section. */ - this_cpu_inc(ssp->sda->srcu_ctrs[idx].srcu_unlocks.counter); /* Z */ + this_cpu_inc(__srcu_ctr_to_ptr(ssp, idx)->srcu_unlocks.counter); /* Z */ RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_unlock_lite()."); } diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index a91651866485..7a8ace83c98d 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -765,7 +765,7 @@ EXPORT_SYMBOL_GPL(__srcu_read_lock); void __srcu_read_unlock(struct srcu_struct *ssp, int idx) { smp_mb(); /* C */ /* Avoid leaking the critical section. */ - this_cpu_inc(ssp->sda->srcu_ctrs[idx].srcu_unlocks.counter); + this_cpu_inc(__srcu_ctr_to_ptr(ssp, idx)->srcu_unlocks.counter); } EXPORT_SYMBOL_GPL(__srcu_read_unlock); @@ -794,10 +794,8 @@ EXPORT_SYMBOL_GPL(__srcu_read_lock_nmisafe); */ void __srcu_read_unlock_nmisafe(struct srcu_struct *ssp, int idx) { - struct srcu_data *sdp = raw_cpu_ptr(ssp->sda); - smp_mb__before_atomic(); /* C */ /* Avoid leaking the critical section. */ - atomic_long_inc(&sdp->srcu_ctrs[idx].srcu_unlocks); + atomic_long_inc(&raw_cpu_ptr(__srcu_ctr_to_ptr(ssp, idx))->srcu_unlocks); } EXPORT_SYMBOL_GPL(__srcu_read_unlock_nmisafe); From patchwork Mon Feb 24 02:22:06 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boqun Feng X-Patchwork-Id: 13987325 Received: from mail-qt1-f169.google.com (mail-qt1-f169.google.com [209.85.160.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0A7FF1FC0F3; Mon, 24 Feb 2025 02:22:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363758; cv=none; b=tBCyP+H/NqXXG6U83/jWA2CJDpzQryDjOTMoDBQU9vZ2+sd5a7FboALQidTlSJjttPON+HHFrJRoZ0Bl1KoJB5Fjn6qAIv8esgovWuvw+3BaDpEjiOLNZbL37sPjxyw4eSiC3hkDfTf/mI5uXdkvSbIgnlcVbAnwSOlAsYwzvVM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363758; c=relaxed/simple; bh=vYGr4oase6oAhv/S3mUkEU7TFl9z4yEqQk+LuMvaIG0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=ajf4yj5QAE6bknpFi5wt4gufo5+XxFkzoI8pVY35wB67GEq6J7jopGY/Nx0JW03J/LU8iMJwIIoLqYa0g0IKWlXvV1vLyrt9HvHyxCvf/U0MwiLnG7RwTjDT+B/wixKM73lFyeQQO3UQHbQtSe9AfULXK3vJuMSyg/N77aNnwpc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Wyrlz9Ge; arc=none smtp.client-ip=209.85.160.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Wyrlz9Ge" Received: by mail-qt1-f169.google.com with SMTP id d75a77b69052e-47210ab1283so57653511cf.1; Sun, 23 Feb 2025 18:22:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740363756; x=1740968556; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=9xc+k+HYzIVy9Ic6oMLdV0EroWNk+iKQ9YparoGG0Ik=; b=Wyrlz9GeIxY2YjJXe5tWrd95p9p7msqGEXxXi7/xqTBOWk15M67/8i6xjkyjPEnxfY 7whIWckBQfsOppeoMzX8gqiDmThtLSLLx7CNewck+VgH5IJkC1iwKJf5iN/bZOiTc4OL NKhF3NEHxA2Q5IFTOetiKYqIavYrWiQRYF0g+mZLItX2t6zviYxXoLtlkiHRFPDPwtlK H3b0/jHXKExi8KFpXcC8TlMT0UlYsqiK6jGTqfpBEiEOABU45LTPaMGA9WCLHqYRSBxL IbBcZ27krs1gKqlvRQmqJ57HZGCaMQMN/iM1T1zxVBtiuk8Bd5q3fPp9UCo2BYzHwiVI cbFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740363756; x=1740968556; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=9xc+k+HYzIVy9Ic6oMLdV0EroWNk+iKQ9YparoGG0Ik=; b=XAE0xVenYtwI2iEnypHaVncaOZqYupa3hnDtyZ7ZBsAFyw3ELng6Z1MJJkebQVcP62 Ah2cTVjzHbxz1C1zEUyn59iMrfDmuYoHT7WX/qLup+GYqFXH/eS+8CJjRHRLayxdPXSH Xec7Mp983Y1iOgZXz+QYFq/sbup4Xc1aRoT5qZavA9p7iKtDOGreM2Ezq4ItGeJDSOSZ +eVZFflxVnvIZFS8Qdd4fWDdzHOtR/YQQ9BJt1uu2NH0e/Br9BO08DD/eT+bX78d+UyU ggKAuwZQ+LAiJrH1kbJ5rYG3CrE/UVZvyD3m541b3+nldk+qSPgo0f6+Nk/mZyc/1P7W 72OQ== X-Forwarded-Encrypted: i=1; AJvYcCUYVXgGI3Zhbm/451ylKto2Ngp9o9DlE/s1UC69F1nXqBsu6qsxAdaSEooXf3P2gv4SYVU=@vger.kernel.org, AJvYcCV7hMTbtS+HFsQDNoNrGKiBA4375C3eYAOqWfBQHqkv++sCUMSrIMfgK+hg2njCuPStzrRH4RE8GR+EqgIxyeib@vger.kernel.org, AJvYcCWECpV444VOB/ZhaCFlRFBaz6KZBnvGKirat0hzS+OdIT+CsaVSIbbFybjEdiax1rjAxrOU3ktMyvb/Bfr4@vger.kernel.org X-Gm-Message-State: AOJu0YzacwxkFy1U4fXaZEG9uhAgBn6kIpF5pzn3ch39XObqenW8uHL0 MiK6DKfYGCkMdfRNfKAZ7a6M9A7n/yzD/iTG8lrLtsO81fLCne/2 X-Gm-Gg: ASbGncuk+uIJ5IBm0xjHXwHwGLSBkzx9TEGNCKQEXUkzF2ov4zrWTARI0HozmIV+QMt +/+A6W9szofq0x1Q/vMWVAZgVsY5qRLCHHK7aTgCnX+piE6BiNTykKy18DfCV0YNNpcWUFgd7HE JxtPmMNEcqCWfdz5BkCKrX9jsqfzpQsSBzEJXluB73NAqyoS257qkzIPsDmnzCIddzGE4LEq+ti MS45pMURDV0rDEpgxQbY5z/U6qO9DQVNvbTvH4QOzmWw7U7ORpuwhTFWOZPmfrEkmpo6yLf2Zml t32jIFk83dfPJ3T58Fv4T6G+QvkcoTHBhJA+Bz1ZwfBbylKjM8IeRaXjz0MKYnt4oGBx2m/Du4z rhGNJVYEgLd2Klof+ X-Google-Smtp-Source: AGHT+IFYTMVp6JYLVO/whvadX5BcZfW43b4gbDKbaMnb7W7u3y3MHECALfKNOxhOC+OdWqH3s0bwIw== X-Received: by 2002:a05:622a:388:b0:471:fde4:f09e with SMTP id d75a77b69052e-4722295d3c7mr159638211cf.36.1740363755990; Sun, 23 Feb 2025 18:22:35 -0800 (PST) Received: from fauth-a2-smtp.messagingengine.com (fauth-a2-smtp.messagingengine.com. [103.168.172.201]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7c09d5980e1sm952821385a.13.2025.02.23.18.22.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 23 Feb 2025 18:22:35 -0800 (PST) Received: from phl-compute-04.internal (phl-compute-04.phl.internal [10.202.2.44]) by mailfauth.phl.internal (Postfix) with ESMTP id 2987F1200043; Sun, 23 Feb 2025 21:22:35 -0500 (EST) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-04.internal (MEProxy); Sun, 23 Feb 2025 21:22:35 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgdejjeehgecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvfevufffkffojghfggfgsedtkeertdertddt necuhfhrohhmpeeuohhquhhnucfhvghnghcuoegsohhquhhnrdhfvghnghesghhmrghilh drtghomheqnecuggftrfgrthhtvghrnhepgeeljeeitdehvdehgefgjeevfeejjeekgfev ffeiueejhfeuiefggeeuheeggefgnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrg hmpehmrghilhhfrhhomhepsghoqhhunhdomhgvshhmthhprghuthhhphgvrhhsohhnrghl ihhthidqieelvdeghedtieegqddujeejkeehheehvddqsghoqhhunhdrfhgvnhhgpeepgh hmrghilhdrtghomhesfhhigihmvgdrnhgrmhgvpdhnsggprhgtphhtthhopedvuddpmhho uggvpehsmhhtphhouhhtpdhrtghpthhtoheprhgtuhesvhhgvghrrdhkvghrnhgvlhdroh hrghdprhgtphhtthhopehjihgrnhhgshhhrghnlhgrihesghhmrghilhdrtghomhdprhgt phhtthhopehprghulhhmtghksehkvghrnhgvlhdrohhrghdprhgtphhtthhopehjohhshh esjhhoshhhthhrihhplhgvthhtrdhorhhgpdhrtghpthhtoheprhhoshhtvgguthesghho ohgumhhishdrohhrghdprhgtphhtthhopehmrghthhhivghurdguvghsnhhohigvrhhsse gvfhhfihgtihhoshdrtghomhdprhgtphhtthhopehfrhgvuggvrhhitgeskhgvrhhnvghl rdhorhhgpdhrtghpthhtohepnhgvvghrrghjrdhuphgrughhhigrhieskhgvrhhnvghlrd horhhgpdhrtghpthhtohepjhhovghlsehjohgvlhhfvghrnhgrnhguvghsrdhorhhg X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 23 Feb 2025 21:22:34 -0500 (EST) From: Boqun Feng To: rcu@vger.kernel.org Cc: Lai Jiangshan , "Paul E. McKenney" , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Uladzislau Rezki , Zqiang , Davidlohr Bueso , Shuah Khan , linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, bpf@vger.kernel.org, Alexei Starovoitov , Andrii Nakryiko , Peter Zijlstra , Kent Overstreet Subject: [PATCH rcu 12/20] srcu: Move SRCU Tree/Tiny definitions from srcu.h Date: Sun, 23 Feb 2025 18:22:06 -0800 Message-Id: <20250224022214.12037-13-boqun.feng@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250224022214.12037-1-boqun.feng@gmail.com> References: <20250224022214.12037-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" There are a couple of definitions under "#ifdef CONFIG_TINY_SRCU" in include/linux/srcu.h. There is no point in them being there, so this commit moves them to include/linux/srcutiny.h and include/linux/srcutree.c, thus eliminating that #ifdef. Signed-off-by: Paul E. McKenney Cc: Alexei Starovoitov Cc: Andrii Nakryiko Cc: Peter Zijlstra Cc: Kent Overstreet Cc: Signed-off-by: Boqun Feng --- include/linux/srcu.h | 10 +--------- include/linux/srcutiny.h | 3 +++ include/linux/srcutree.h | 1 + 3 files changed, 5 insertions(+), 9 deletions(-) diff --git a/include/linux/srcu.h b/include/linux/srcu.h index 505f5bdce444..2bd0e24e9b55 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -52,6 +52,7 @@ int init_srcu_struct(struct srcu_struct *ssp); #define SRCU_READ_FLAVOR_SLOWGP SRCU_READ_FLAVOR_LITE // Flavors requiring synchronize_rcu() // instead of smp_mb(). +void __srcu_read_unlock(struct srcu_struct *ssp, int idx) __releases(ssp); #ifdef CONFIG_TINY_SRCU #include @@ -64,15 +65,6 @@ int init_srcu_struct(struct srcu_struct *ssp); void call_srcu(struct srcu_struct *ssp, struct rcu_head *head, void (*func)(struct rcu_head *head)); void cleanup_srcu_struct(struct srcu_struct *ssp); -int __srcu_read_lock(struct srcu_struct *ssp) __acquires(ssp); -void __srcu_read_unlock(struct srcu_struct *ssp, int idx) __releases(ssp); -#ifdef CONFIG_TINY_SRCU -#define __srcu_read_lock_lite __srcu_read_lock -#define __srcu_read_unlock_lite __srcu_read_unlock -#else // #ifdef CONFIG_TINY_SRCU -int __srcu_read_lock_lite(struct srcu_struct *ssp) __acquires(ssp); -void __srcu_read_unlock_lite(struct srcu_struct *ssp, int idx) __releases(ssp); -#endif // #else // #ifdef CONFIG_TINY_SRCU void synchronize_srcu(struct srcu_struct *ssp); #define SRCU_GET_STATE_COMPLETED 0x1 diff --git a/include/linux/srcutiny.h b/include/linux/srcutiny.h index b347bde1aac2..a6194f7a7e34 100644 --- a/include/linux/srcutiny.h +++ b/include/linux/srcutiny.h @@ -71,6 +71,9 @@ static inline int __srcu_read_lock(struct srcu_struct *ssp) return idx; } +#define __srcu_read_lock_lite __srcu_read_lock +#define __srcu_read_unlock_lite __srcu_read_unlock + static inline void synchronize_srcu_expedited(struct srcu_struct *ssp) { synchronize_srcu(ssp); diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index 55fa400624bb..ef3065c0cadc 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -207,6 +207,7 @@ struct srcu_struct { #define DEFINE_SRCU(name) __DEFINE_SRCU(name, /* not static */) #define DEFINE_STATIC_SRCU(name) __DEFINE_SRCU(name, static) +int __srcu_read_lock(struct srcu_struct *ssp) __acquires(ssp); void synchronize_srcu_expedited(struct srcu_struct *ssp); void srcu_barrier(struct srcu_struct *ssp); void srcu_torture_stats_print(struct srcu_struct *ssp, char *tt, char *tf); From patchwork Mon Feb 24 02:22:07 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boqun Feng X-Patchwork-Id: 13987326 Received: from mail-qk1-f171.google.com (mail-qk1-f171.google.com [209.85.222.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8BC811FDA79; Mon, 24 Feb 2025 02:22:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363760; cv=none; b=LKrsnwzC6dNbvbYnyVTr5ncDgHRkvFU4u+pj7RlcX32NzWvLo7i+1p2t/gTF4BW476x0J5y6leBtt4T4AjoDD+0XbrBQWG68Qx5wX+agIcIUWbEdkHkGzHeHGUipwhEG7yMEdnGJdJICP7evEFuGK+DNOdcqOhZ27HwZLGiMju0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363760; c=relaxed/simple; bh=eWuqWsxlsJdHvX9J4PpjZoW4YF4OyOGuqZuyk6j8YHw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=QyxtL0p60C82IS/TocS6XM2lE9V9QcRJztmzD6ZpZcvkRkWLNYEcgTybpOBaPZwOwfRPYqd14m5cPPmak53daLuu0SlMhN0w7OUoIjS0hoEkgR5rZ4Eyhlxb7r2HsscYTrlYe0CdiWqBDV3N0PKsc5Fmf7u9imOt7bayEAbPXUk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=AoTSSQUT; arc=none smtp.client-ip=209.85.222.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="AoTSSQUT" Received: by mail-qk1-f171.google.com with SMTP id af79cd13be357-7c2303a56d6so43915285a.3; Sun, 23 Feb 2025 18:22:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740363757; x=1740968557; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=cs7/tYIYyEnsMEUFXtBt9RoaeQsr5/p8PlPMftFjBMc=; b=AoTSSQUTE0406cBg2euHav0H58Vo+pU5Axd2/OTKDYBy+LOCuNRmhD0Muv5DnWrzfw V7lh046ELyQr5BLELmMyHVbSuzyGJ0/lHznGLevkWoFt7XAEq5nn0oZW0FHRrkIfhKtA ArTM83Dc+hKR9fZFHlQzhsjlXHjkSbrES6KLhZueRcJM8itli8mQ6LR6ddkC87rfDl85 PIrggkcsQjIkcqRxkWITbIIxGDJ6WLssYE8aGntFkHzK/acrfAEzu9HxWL09OBcTpzDO LEOGbc1VbO0n39pkcgOAmypAvmOgNEroCyc/iZNgXtCL8FMfB88DKoDt02GG49oulg5O hjkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740363757; x=1740968557; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=cs7/tYIYyEnsMEUFXtBt9RoaeQsr5/p8PlPMftFjBMc=; b=aW9vfaVfcePk6W/FDdmGxcLMo5RIyjeBEpJSo9Xw25vwRVvYd7hUtzbTWC0+H2bdgM sbwu2aZiA6Q6HP5c+QOIaw6EVfXnEkij8thb6kGPw3cGnOsHTRcVmImUIbaJMKM106C6 hn/XEfJT2vbFeuF0Px3PeBhqntrp8rkCI5bWt9Ea5tRezrSIP9JNRUnAaiH1YMQoCMbv rhRdNZDOU3jkR1QPrDI5x3snYp2hhR3avSeas6BZQS0RTb1aXB1XEJ+dYpWT5RYRVNYU I0HbM1nNji08RC2RkdW73yPDLcHSfVJfHmfdRwYLhe5RKvB3yFhV1HrcuDVt2ceWjGuj e/yg== X-Forwarded-Encrypted: i=1; AJvYcCVAsh9Yvx1g/6KYb0bTnStvtalzzB5gYqYi2Fy2JKn5Ruw3+JEBW0SMMeb9vT3/gMNUDds=@vger.kernel.org, AJvYcCVgDgjmrUHq0FMICyzEmQSK6PP8WAKSyJrSj5dcLmmrz197W52KXy1akFUOocFhLe+q2XEN2/JYmhUOE0/yBlVU@vger.kernel.org, AJvYcCVwudu1WyEmLXrC8AIk2K3E0/kZiKQl2nUc4wHo3YgGnxqwmcCqf4gqm4yOVQnduPnA5my5SSOCgVFw8D6+@vger.kernel.org X-Gm-Message-State: AOJu0YxvHVxPcAxbw2ChJzDOn9ZJWQYmSPjDx1cdjLgYJb0VVPWtodLd 8KB0uVUhl4p0zEvj6qbptgXWco5ANSrwfnbp+M3o8apd+PbUmq2i X-Gm-Gg: ASbGnct1HhAjblOj+2tKklcQQiuQfFa2gje3zZaEk9+u5vDTXqJLna9l2zmID7Gh1NN /bpAl1QMJrhhAIXdRmbL6V8MOBPlU6Bw0zsw8UOu4PCBexa94bn+kSfOzilV60BBKFMxFNUIHbn H6O8PMddt8ToAc+WBU3pJAt0eNiDpAqWqYBOrteGgbSIvKW1w+RAWXGEgy2qbr/bChXOasZVceQ Dfik2UD3bFluJvasDcQBmkiVSuacGIUVUS4xf7rxZ4EWZsGmqrcMLkWTfC7PlD/5YeYU8xAEoMX 1mdWQsMBMPVCRHvpMgvAOybu7cTkZpjQVWrTPSpQ8afaTGmJlwA6M8f77/566O6OINz7i4CIV5j Z3ZuzJI89pUjhORmH X-Google-Smtp-Source: AGHT+IEjwxU9YmoI3nKfcIVCjMPgByTWVYCjFgmn68xYM5Z/nUOsRS89BtQNBjkqirCkPx4hVFuwgA== X-Received: by 2002:a05:6214:1c0f:b0:6d3:f6bd:ca04 with SMTP id 6a1803df08f44-6e6b01d79c6mr129720876d6.40.1740363757345; Sun, 23 Feb 2025 18:22:37 -0800 (PST) Received: from fauth-a2-smtp.messagingengine.com (fauth-a2-smtp.messagingengine.com. [103.168.172.201]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6e65d785ce5sm127500186d6.41.2025.02.23.18.22.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 23 Feb 2025 18:22:37 -0800 (PST) Received: from phl-compute-08.internal (phl-compute-08.phl.internal [10.202.2.48]) by mailfauth.phl.internal (Postfix) with ESMTP id 883821200043; Sun, 23 Feb 2025 21:22:36 -0500 (EST) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-08.internal (MEProxy); Sun, 23 Feb 2025 21:22:36 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgdejjeehgecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvfevufffkffojghfggfgsedtkeertdertddt necuhfhrohhmpeeuohhquhhnucfhvghnghcuoegsohhquhhnrdhfvghnghesghhmrghilh drtghomheqnecuggftrfgrthhtvghrnhepgeeljeeitdehvdehgefgjeevfeejjeekgfev ffeiueejhfeuiefggeeuheeggefgnecuvehluhhsthgvrhfuihiivgepheenucfrrghrrg hmpehmrghilhhfrhhomhepsghoqhhunhdomhgvshhmthhprghuthhhphgvrhhsohhnrghl ihhthidqieelvdeghedtieegqddujeejkeehheehvddqsghoqhhunhdrfhgvnhhgpeepgh hmrghilhdrtghomhesfhhigihmvgdrnhgrmhgvpdhnsggprhgtphhtthhopedvuddpmhho uggvpehsmhhtphhouhhtpdhrtghpthhtoheprhgtuhesvhhgvghrrdhkvghrnhgvlhdroh hrghdprhgtphhtthhopehjihgrnhhgshhhrghnlhgrihesghhmrghilhdrtghomhdprhgt phhtthhopehprghulhhmtghksehkvghrnhgvlhdrohhrghdprhgtphhtthhopehjohhshh esjhhoshhhthhrihhplhgvthhtrdhorhhgpdhrtghpthhtoheprhhoshhtvgguthesghho ohgumhhishdrohhrghdprhgtphhtthhopehmrghthhhivghurdguvghsnhhohigvrhhsse gvfhhfihgtihhoshdrtghomhdprhgtphhtthhopehfrhgvuggvrhhitgeskhgvrhhnvghl rdhorhhgpdhrtghpthhtohepnhgvvghrrghjrdhuphgrughhhigrhieskhgvrhhnvghlrd horhhgpdhrtghpthhtohepjhhovghlsehjohgvlhhfvghrnhgrnhguvghsrdhorhhg X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 23 Feb 2025 21:22:36 -0500 (EST) From: Boqun Feng To: rcu@vger.kernel.org Cc: Lai Jiangshan , "Paul E. McKenney" , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Uladzislau Rezki , Zqiang , Davidlohr Bueso , Shuah Khan , linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, bpf@vger.kernel.org, Alexei Starovoitov , Andrii Nakryiko , Peter Zijlstra , Kent Overstreet Subject: [PATCH rcu 13/20] srcu: Add SRCU-fast readers Date: Sun, 23 Feb 2025 18:22:07 -0800 Message-Id: <20250224022214.12037-14-boqun.feng@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250224022214.12037-1-boqun.feng@gmail.com> References: <20250224022214.12037-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" This commit adds srcu_read_{,un}lock_fast(), which is similar to srcu_read_{,un}lock_lite(), but avoids the array-indexing and pointer-following overhead. On a microbenchmark featuring tight loops around empty readers, this results in about a 20% speedup compared to RCU Tasks Trace on my x86 laptop. Please note that SRCU-fast has drawbacks compared to RCU Tasks Trace, including: o Lack of CPU stall warnings. o SRCU-fast readers permitted only where rcu_is_watching(). o A pointer-sized return value from srcu_read_lock_fast() must be passed to the corresponding srcu_read_unlock_fast(). o In the absence of readers, a synchronize_srcu() having _fast() readers will incur the latency of at least two normal RCU grace periods. o RCU Tasks Trace priority boosting could be easily added. Boosting SRCU readers is more difficult. SRCU-fast also has a drawback compared to SRCU-lite, namely that the return value from srcu_read_lock_fast()-fast is a 64-bit pointer and that from srcu_read_lock_lite() is only a 32-bit int. [ paulmck: Apply feedback from Akira Yokosawa. ] Signed-off-by: Paul E. McKenney Cc: Alexei Starovoitov Cc: Andrii Nakryiko Cc: Peter Zijlstra Cc: Kent Overstreet Cc: Signed-off-by: Boqun Feng --- include/linux/srcu.h | 47 ++++++++++++++++++++++++++++++++++++++-- include/linux/srcutiny.h | 22 +++++++++++++++++++ include/linux/srcutree.h | 38 ++++++++++++++++++++++++++++++++ 3 files changed, 105 insertions(+), 2 deletions(-) diff --git a/include/linux/srcu.h b/include/linux/srcu.h index 2bd0e24e9b55..63bddc301423 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -47,9 +47,10 @@ int init_srcu_struct(struct srcu_struct *ssp); #define SRCU_READ_FLAVOR_NORMAL 0x1 // srcu_read_lock(). #define SRCU_READ_FLAVOR_NMI 0x2 // srcu_read_lock_nmisafe(). #define SRCU_READ_FLAVOR_LITE 0x4 // srcu_read_lock_lite(). +#define SRCU_READ_FLAVOR_FAST 0x8 // srcu_read_lock_fast(). #define SRCU_READ_FLAVOR_ALL (SRCU_READ_FLAVOR_NORMAL | SRCU_READ_FLAVOR_NMI | \ - SRCU_READ_FLAVOR_LITE) // All of the above. -#define SRCU_READ_FLAVOR_SLOWGP SRCU_READ_FLAVOR_LITE + SRCU_READ_FLAVOR_LITE | SRCU_READ_FLAVOR_FAST) // All of the above. +#define SRCU_READ_FLAVOR_SLOWGP (SRCU_READ_FLAVOR_LITE | SRCU_READ_FLAVOR_FAST) // Flavors requiring synchronize_rcu() // instead of smp_mb(). void __srcu_read_unlock(struct srcu_struct *ssp, int idx) __releases(ssp); @@ -253,6 +254,33 @@ static inline int srcu_read_lock(struct srcu_struct *ssp) __acquires(ssp) return retval; } +/** + * srcu_read_lock_fast - register a new reader for an SRCU-protected structure. + * @ssp: srcu_struct in which to register the new reader. + * + * Enter an SRCU read-side critical section, but for a light-weight + * smp_mb()-free reader. See srcu_read_lock() for more information. + * + * If srcu_read_lock_fast() is ever used on an srcu_struct structure, + * then none of the other flavors may be used, whether before, during, + * or after. Note that grace-period auto-expediting is disabled for _fast + * srcu_struct structures because auto-expedited grace periods invoke + * synchronize_rcu_expedited(), IPIs and all. + * + * Note that srcu_read_lock_fast() can be invoked only from those contexts + * where RCU is watching, that is, from contexts where it would be legal + * to invoke rcu_read_lock(). Otherwise, lockdep will complain. + */ +static inline struct srcu_ctr __percpu *srcu_read_lock_fast(struct srcu_struct *ssp) __acquires(ssp) +{ + struct srcu_ctr __percpu *retval; + + srcu_check_read_flavor_force(ssp, SRCU_READ_FLAVOR_FAST); + retval = __srcu_read_lock_fast(ssp); + rcu_try_lock_acquire(&ssp->dep_map); + return retval; +} + /** * srcu_read_lock_lite - register a new reader for an SRCU-protected structure. * @ssp: srcu_struct in which to register the new reader. @@ -356,6 +384,21 @@ static inline void srcu_read_unlock(struct srcu_struct *ssp, int idx) __srcu_read_unlock(ssp, idx); } +/** + * srcu_read_unlock_fast - unregister a old reader from an SRCU-protected structure. + * @ssp: srcu_struct in which to unregister the old reader. + * @scp: return value from corresponding srcu_read_lock_fast(). + * + * Exit a light-weight SRCU read-side critical section. + */ +static inline void srcu_read_unlock_fast(struct srcu_struct *ssp, struct srcu_ctr __percpu *scp) + __releases(ssp) +{ + srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST); + srcu_lock_release(&ssp->dep_map); + __srcu_read_unlock_fast(ssp, scp); +} + /** * srcu_read_unlock_lite - unregister a old reader from an SRCU-protected structure. * @ssp: srcu_struct in which to unregister the old reader. diff --git a/include/linux/srcutiny.h b/include/linux/srcutiny.h index a6194f7a7e34..e271f9f96bfc 100644 --- a/include/linux/srcutiny.h +++ b/include/linux/srcutiny.h @@ -71,6 +71,28 @@ static inline int __srcu_read_lock(struct srcu_struct *ssp) return idx; } +struct srcu_ctr; + +static inline bool __srcu_ptr_to_ctr(struct srcu_struct *ssp, struct srcu_ctr __percpu *scpp) +{ + return (int)(intptr_t)(struct srcu_ctr __force __kernel *)scpp; +} + +static inline struct srcu_ctr __percpu *__srcu_ctr_to_ptr(struct srcu_struct *ssp, int idx) +{ + return (struct srcu_ctr __percpu *)(intptr_t)idx; +} + +static inline struct srcu_ctr __percpu *__srcu_read_lock_fast(struct srcu_struct *ssp) +{ + return __srcu_ctr_to_ptr(ssp, __srcu_read_lock(ssp)); +} + +static inline void __srcu_read_unlock_fast(struct srcu_struct *ssp, struct srcu_ctr __percpu *scp) +{ + __srcu_read_unlock(ssp, __srcu_ptr_to_ctr(ssp, scp)); +} + #define __srcu_read_lock_lite __srcu_read_lock #define __srcu_read_unlock_lite __srcu_read_unlock diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index ef3065c0cadc..bdc467efce3a 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -226,6 +226,44 @@ static inline struct srcu_ctr __percpu *__srcu_ctr_to_ptr(struct srcu_struct *ss return &ssp->sda->srcu_ctrs[idx]; } +/* + * Counts the new reader in the appropriate per-CPU element of the + * srcu_struct. Returns a pointer that must be passed to the matching + * srcu_read_unlock_fast(). + * + * Note that this_cpu_inc() is an RCU read-side critical section either + * because it disables interrupts, because it is a single instruction, + * or because it is a read-modify-write atomic operation, depending on + * the whims of the architecture. + */ +static inline struct srcu_ctr __percpu *__srcu_read_lock_fast(struct srcu_struct *ssp) +{ + struct srcu_ctr __percpu *scp = READ_ONCE(ssp->srcu_ctrp); + + RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_lock_fast()."); + this_cpu_inc(scp->srcu_locks.counter); /* Y */ + barrier(); /* Avoid leaking the critical section. */ + return scp; +} + +/* + * Removes the count for the old reader from the appropriate + * per-CPU element of the srcu_struct. Note that this may well be a + * different CPU than that which was incremented by the corresponding + * srcu_read_lock_fast(), but it must be within the same task. + * + * Note that this_cpu_inc() is an RCU read-side critical section either + * because it disables interrupts, because it is a single instruction, + * or because it is a read-modify-write atomic operation, depending on + * the whims of the architecture. + */ +static inline void __srcu_read_unlock_fast(struct srcu_struct *ssp, struct srcu_ctr __percpu *scp) +{ + barrier(); /* Avoid leaking the critical section. */ + this_cpu_inc(scp->srcu_unlocks.counter); /* Z */ + RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_unlock_fast()."); +} + /* * Counts the new reader in the appropriate per-CPU element of the * srcu_struct. Returns an index that must be passed to the matching From patchwork Mon Feb 24 02:22:08 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boqun Feng X-Patchwork-Id: 13987327 Received: from mail-qv1-f41.google.com (mail-qv1-f41.google.com [209.85.219.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DB6F0202C44; Mon, 24 Feb 2025 02:22:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363761; cv=none; b=QzRYVMCIvYU4fambvBZFZq/HH6zIAaCzz0DkWENbCUg/jp8wciA3+Lbd3qC1RrjIr6Vb+X5d5CyUNIzxi9iBBVYuEPxMcfn+ENg78/T3IhWOmWlf/X+F5eELGaJMkedTHELH7B/gzJ281FYE5Mw50lpge6Jr00bB/gzHQjpQXVk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363761; c=relaxed/simple; bh=x1RagGnEsGngNw3EY8nNKEZy+pH7qTCaixBwp6WaeUM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=jvdPrjBFDbn4V/XaKIRGAlM485biFgPD3CinoAJyFBYvl03qIe9luwyO/svYXZlld6bRYnk9g9QrddBdxsQbBZaaozXW10RKtR02gUDoG0PsQhFVXtOScqyoJVILcleAEpmm5HL+I+b2JNWJRQoO/PKeeoW0uNamzP6G+J5Pdxo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=X84TZ4gq; arc=none smtp.client-ip=209.85.219.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="X84TZ4gq" Received: by mail-qv1-f41.google.com with SMTP id 6a1803df08f44-6e6621960eeso48879686d6.0; Sun, 23 Feb 2025 18:22:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740363759; x=1740968559; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=LYVXngARAHEiVLhrUvNNltGXmcHFU3g/uWrZ+f3iCoY=; b=X84TZ4gq6tQ3qHFKwFwJosUXNsyME9HArhW4ireHougv+e7EZR410YOyreB5r5MqVM WGWd4L4lqaRPXJGbInj8XxzlXlxtWGOU6tHZxHl6V1t3nvrboHD/U5M+K2Xw4zVuQjug KNc8UcuY58rSClsN8NL88XTQH3YuePTZ9zTKaabOz/b+p5pBiIyqwRyWKBz9H/fvjx8+ o3DyPk4c9ovVXU3Y27jTCZ4E6XANvJdEgZcWnRZHzFFtFL0ozvBHIXZIc4n0qXl5sFSD 3ENHD8muRKo29rMfzQUvNsyCyPQOv1no5BL2ODDINwP6bd3MjDPSa6LZHrnwLZz/IBRH r38A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740363759; x=1740968559; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=LYVXngARAHEiVLhrUvNNltGXmcHFU3g/uWrZ+f3iCoY=; b=vojje5ZMZoEZNuZL5UxlcLe59C97ENJyTQ6PRem2dGl4Mi7dAR0SXBK5x7XlKxvMft kX+VUrE+4/2n1hNY2XkWrdtlgKabqheC+PdVmEKy+anIQzh7k8b5pOy9IfWq6TQCCp4n o7YqulNTXZaV+elBEARtEx1m/0lmU6qwxt0zeBpdr2HbUTsEH/gRk9dZYp2civjQhyOJ VQ3Prd2pRM8GRwvT5qMQJEs7pj35fEFKb9wudWqyjGGQx70wvAI7FrTCuK7t9yN2zist HoB7NVWqutLYzaziq3v/26TUqTGGnL47+/94uNHXW0gmasgz76e0GK5sjJPFgm9c7HTc dzBg== X-Forwarded-Encrypted: i=1; AJvYcCUqkc6Owwky4vy2RGsuifaLbKw6FMOp3VknVwRRLL5NCy/CNCwjqy9QcT9RPGY8zWJW7XjAfPJLBANKjmbcuEvW@vger.kernel.org, AJvYcCV7fZieXfRWUnnPM5a9wpzBatFbZbmM4i19FKGXwuwYcgvq0+kRmvAILYX/Ly29A6MmHus=@vger.kernel.org, AJvYcCWCS+u4g/FBbCW1HAIkEps0IZsSSLEwVlgm+Le4pv70/q79kePxsHzAv5+COMrRoS7t7SIuvgyu5o4cEuC+@vger.kernel.org X-Gm-Message-State: AOJu0Yyktw6NfxiIbpsG0uVmMIml3fZZ/3gGGSSrcaA5to7NRsEcRyFn glQSCxC7Mkhzf1t/ZbTWdmWxgJyi3rbDCI0GuQ8rhwa9ODpwkmOT X-Gm-Gg: ASbGnct5epEnVQyZDF878nneLW3Xflj/GFgHNvfU636AiktYwRisbvc/9noTg8GWCA7 wIpChg/VgUlUdKMBdE0Oka2xt/RhCTgOCW8eIOrleok4UL9S+d4EZPxtb6Rt4d522ram2HWPk9n DB9cVUYoGUmCoKfg3rh0X41YO3dbjK+MdLxqU6xSmKbkkWMByJTGE1pccDyfSKlVTDOE87P06wu tUSjEydr31wojt99zbZna25tKiqG/YVvfDjZZvcBXk2x/abamcB48e+b1xlHwfwt2lyU/7fvCCi Oo7FyOhGYJNMSi5yPToj4FE9tcFv78P650p2sErdyNU9LcD+6OVZZr4aJ7ScMRRJW1t3fcJhomT ZXKM7jr/UpEizxhVL X-Google-Smtp-Source: AGHT+IHZEXr3FBaKPE3j25LAFJ0ZFPpAKJCewtDs9aZxscFGONzRa6lLREjHVfy2UJiJ5PQf0mtAdQ== X-Received: by 2002:a05:6214:1250:b0:6e2:3761:71b0 with SMTP id 6a1803df08f44-6e6ae72d99bmr175752656d6.5.1740363758761; Sun, 23 Feb 2025 18:22:38 -0800 (PST) Received: from fauth-a2-smtp.messagingengine.com (fauth-a2-smtp.messagingengine.com. [103.168.172.201]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6e65d71721csm128183396d6.0.2025.02.23.18.22.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 23 Feb 2025 18:22:38 -0800 (PST) Received: from phl-compute-01.internal (phl-compute-01.phl.internal [10.202.2.41]) by mailfauth.phl.internal (Postfix) with ESMTP id DD8B81200043; Sun, 23 Feb 2025 21:22:37 -0500 (EST) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-01.internal (MEProxy); Sun, 23 Feb 2025 21:22:37 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgdejjeehgecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvfevufffkffojghfggfgsedtkeertdertddt necuhfhrohhmpeeuohhquhhnucfhvghnghcuoegsohhquhhnrdhfvghnghesghhmrghilh drtghomheqnecuggftrfgrthhtvghrnhepgeeljeeitdehvdehgefgjeevfeejjeekgfev ffeiueejhfeuiefggeeuheeggefgnecuvehluhhsthgvrhfuihiivgepvdenucfrrghrrg hmpehmrghilhhfrhhomhepsghoqhhunhdomhgvshhmthhprghuthhhphgvrhhsohhnrghl ihhthidqieelvdeghedtieegqddujeejkeehheehvddqsghoqhhunhdrfhgvnhhgpeepgh hmrghilhdrtghomhesfhhigihmvgdrnhgrmhgvpdhnsggprhgtphhtthhopedvuddpmhho uggvpehsmhhtphhouhhtpdhrtghpthhtoheprhgtuhesvhhgvghrrdhkvghrnhgvlhdroh hrghdprhgtphhtthhopehjihgrnhhgshhhrghnlhgrihesghhmrghilhdrtghomhdprhgt phhtthhopehprghulhhmtghksehkvghrnhgvlhdrohhrghdprhgtphhtthhopehjohhshh esjhhoshhhthhrihhplhgvthhtrdhorhhgpdhrtghpthhtoheprhhoshhtvgguthesghho ohgumhhishdrohhrghdprhgtphhtthhopehmrghthhhivghurdguvghsnhhohigvrhhsse gvfhhfihgtihhoshdrtghomhdprhgtphhtthhopehfrhgvuggvrhhitgeskhgvrhhnvghl rdhorhhgpdhrtghpthhtohepnhgvvghrrghjrdhuphgrughhhigrhieskhgvrhhnvghlrd horhhgpdhrtghpthhtohepjhhovghlsehjohgvlhhfvghrnhgrnhguvghsrdhorhhg X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 23 Feb 2025 21:22:37 -0500 (EST) From: Boqun Feng To: rcu@vger.kernel.org Cc: Lai Jiangshan , "Paul E. McKenney" , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Uladzislau Rezki , Zqiang , Davidlohr Bueso , Shuah Khan , linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, bpf@vger.kernel.org, Alexei Starovoitov , Andrii Nakryiko , Peter Zijlstra , Kent Overstreet Subject: [PATCH rcu 14/20] rcutorture: Add ability to test srcu_read_{,un}lock_fast() Date: Sun, 23 Feb 2025 18:22:08 -0800 Message-Id: <20250224022214.12037-15-boqun.feng@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250224022214.12037-1-boqun.feng@gmail.com> References: <20250224022214.12037-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" This commit permits rcutorture to test srcu_read_{,un}lock_fast(), which is specified by the rcutorture.reader_flavor=0x8 kernel boot parameter. Signed-off-by: Paul E. McKenney Cc: Alexei Starovoitov Cc: Andrii Nakryiko Cc: Peter Zijlstra Cc: Kent Overstreet Cc: Signed-off-by: Boqun Feng --- kernel/rcu/rcutorture.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c index 1d2de50fb5d6..1bd3eaa0b8e7 100644 --- a/kernel/rcu/rcutorture.c +++ b/kernel/rcu/rcutorture.c @@ -677,6 +677,7 @@ static void srcu_get_gp_data(int *flags, unsigned long *gp_seq) static int srcu_torture_read_lock(void) { int idx; + struct srcu_ctr __percpu *scp; int ret = 0; if ((reader_flavor & SRCU_READ_FLAVOR_NORMAL) || !(reader_flavor & SRCU_READ_FLAVOR_ALL)) { @@ -694,6 +695,12 @@ static int srcu_torture_read_lock(void) WARN_ON_ONCE(idx & ~0x1); ret += idx << 2; } + if (reader_flavor & SRCU_READ_FLAVOR_FAST) { + scp = srcu_read_lock_fast(srcu_ctlp); + idx = __srcu_ptr_to_ctr(srcu_ctlp, scp); + WARN_ON_ONCE(idx & ~0x1); + ret += idx << 3; + } return ret; } @@ -719,6 +726,8 @@ srcu_read_delay(struct torture_random_state *rrsp, struct rt_read_seg *rtrsp) static void srcu_torture_read_unlock(int idx) { WARN_ON_ONCE((reader_flavor && (idx & ~reader_flavor)) || (!reader_flavor && (idx & ~0x1))); + if (reader_flavor & SRCU_READ_FLAVOR_FAST) + srcu_read_unlock_fast(srcu_ctlp, __srcu_ctr_to_ptr(srcu_ctlp, (idx & 0x8) >> 3)); if (reader_flavor & SRCU_READ_FLAVOR_LITE) srcu_read_unlock_lite(srcu_ctlp, (idx & 0x4) >> 2); if (reader_flavor & SRCU_READ_FLAVOR_NMI) From patchwork Mon Feb 24 02:22:09 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boqun Feng X-Patchwork-Id: 13987328 Received: from mail-qv1-f41.google.com (mail-qv1-f41.google.com [209.85.219.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 21EEF2063CB; Mon, 24 Feb 2025 02:22:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363763; cv=none; b=n5ErI8I8vdGe+h4dnlrjv7WbTBp538t3dQ8Q5lIq5ZjpvQS4J/UI4lZFdx0sgYPLocIrYTO9ds4VuGGFzdCoBr8AQGCDb5wzGhwFySpSjvP0Uvm1qdKlPBZiu2bOwD7tLbzJJ9flxoz3BARu2iim9bEttRNaQjQcI4B/94X+ckc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363763; c=relaxed/simple; bh=h8HBUzXuphlZNQxNxs1uJ1SrYCmrEvzUpER8NpS6wz0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=JNH6UL63rnQnzb4yetv41m/GgjSlBaFAKz5gLCD7iT5kUj48PlH/HnXzDa27JfnWe1TeNqPn8CYF7+ti1MPRxtgc2KUUgQ0kK47EWBmW2+M13RkGz2MYlaNG87BXzfbhJ3Ld+Tu51/U/bLvj6FgUumxweslepkxKjsv3ppWwPgI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=fHLfz5jX; arc=none smtp.client-ip=209.85.219.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="fHLfz5jX" Received: by mail-qv1-f41.google.com with SMTP id 6a1803df08f44-6e68943a295so35295096d6.0; Sun, 23 Feb 2025 18:22:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740363760; x=1740968560; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=Oog4qI5g2vg4T9pens3ux9X508RzykqUXStM/G8prXE=; b=fHLfz5jXqb694TcHdpEDQnSdy5XYuzZEnyr488I+Jq4ELylqM8qNRPhqzy3OWKmm8G T71m0aLXAxU+mvyfKfzU7OZTW4PfnMEEShfePHAkhPZtNVHsiVJyQDYmSr40WehZGTqI 9Mx5N3rt/ZFbf6+a18VvTcZx0DNLIxlWwYUDwcFAVot6Xsh1yBAF4f16Xi4xKBjLtHJz NynwxfVyPvmS3bh9HV8nksKtZjGgucnVwVVvKVDkvsR+CxurfjqKV4lCU9mkasCCyDYh SH/lDXMfQN9Ub62greux0krO/qt+kD/jxAFZuP2CytryLdiyl72YBK+htL9518BEGOb9 HFOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740363760; x=1740968560; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=Oog4qI5g2vg4T9pens3ux9X508RzykqUXStM/G8prXE=; b=cVyu4mbXxGm0cSXOnVkqSswE5ztia53AAZj/MJoMBq+alESE4COtTka7cTiuktY1qr t5zFebpjWfFMzvsQ7qJZHS0UH8XaqZJ/OHhHo3UaSbwxtvzUuwNZ47E9rGIcVYDY2F9d QW662YvBlfIBrLxxSsZseXb4R/8fwP8ZU/IT+zEuA4aR1XHidDEZ0wEs0X5ktFJbI7uj HVxccM0jPX4JOUy+n8cunqkntOyCQ5yf7V4NEIEQavXZwtyzn+56fJbWBX1f007rAPEc G/C6c36c94hcm2ARjyyHyI4zRY0SITWuJeecSBzmSif3hNM/g3C2sDqof+ePZE87o7X8 dxNg== X-Forwarded-Encrypted: i=1; AJvYcCV9vlPIAI3DOBnBynf/D6WHTO3ZvAAzwL5bnmqWA9S9ntlj6Nui2OOom4OmLAhmhnI0reShjFyWqMuxpmhnKzg9@vger.kernel.org, AJvYcCVXM63GcVEywOiwgcHhnBoSdxKIgsSso3l0a3uD/XNXoN5H7pK3z3mrJaYW1V/EGv2L7yc=@vger.kernel.org, AJvYcCW9bPstb8oXI0LADyaEjJtfz/moMtWXP8+Dm8vX7npa0F6qlVkpPfYGLnTiQsj0fCO1tIv4sHQIoF0+fWh5@vger.kernel.org X-Gm-Message-State: AOJu0YxftwsZdSdDOQT+0AuQOxLZoFZQsSLZW+DT/Fg8Mq4t2xJJ1Yc+ bvHP0HqJTMnSwHZO5i9JnHswei4H3ybPBqp4SB3wMwOQuPcU1KIK X-Gm-Gg: ASbGncvXfVzMWb1bBzpYB1z9FcoS6+NlgpxImtM1KpbLYtBrN8AessD0GciNtnvo9PW ZKte/IBlMVBhjUeH4rrK9cdMsJIPAOvBPGkgMbKUk+QNVv2CelpVsOQHt8ZmwPUbrE65GiNQWEp Ca/Y7NaCmFvKdVCYEZtukmjyhLNwSa5E4twYibjBTrGDF+smOB+qK5EtizGRVPdW2LkDShRaNb8 XvKUHTQo6l366lBG4LRRUMe/yryK/BPUz7YdaboSv0eUwdvrDVEoEcz5z75pGXQPOc/SzVF3mb6 iiPeHIFcUb7GHqjNuNsQIzrzHh2/64XUvmBanI3EpnXNO4kNVcKEZZtH9gPIeCJJqn1eqpgV7iB LGLuC1YWSBrRcJzMI X-Google-Smtp-Source: AGHT+IGAajfRFO2xGulo8n/40NXHr0Led+2dTbHy2lKt/rZwX0131IbbeJmnf9Cue1pOnk2ANk5LyQ== X-Received: by 2002:a05:6214:230b:b0:6e6:61f1:458a with SMTP id 6a1803df08f44-6e6b00935b2mr154788296d6.14.1740363760063; Sun, 23 Feb 2025 18:22:40 -0800 (PST) Received: from fauth-a2-smtp.messagingengine.com (fauth-a2-smtp.messagingengine.com. [103.168.172.201]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6e65d779271sm128282586d6.3.2025.02.23.18.22.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 23 Feb 2025 18:22:39 -0800 (PST) Received: from phl-compute-01.internal (phl-compute-01.phl.internal [10.202.2.41]) by mailfauth.phl.internal (Postfix) with ESMTP id 37F341200043; Sun, 23 Feb 2025 21:22:39 -0500 (EST) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-01.internal (MEProxy); Sun, 23 Feb 2025 21:22:39 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgdejjeehgecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvfevufffkffojghfggfgsedtkeertdertddt necuhfhrohhmpeeuohhquhhnucfhvghnghcuoegsohhquhhnrdhfvghnghesghhmrghilh drtghomheqnecuggftrfgrthhtvghrnhepgeeljeeitdehvdehgefgjeevfeejjeekgfev ffeiueejhfeuiefggeeuheeggefgnecuvehluhhsthgvrhfuihiivgepvdenucfrrghrrg hmpehmrghilhhfrhhomhepsghoqhhunhdomhgvshhmthhprghuthhhphgvrhhsohhnrghl ihhthidqieelvdeghedtieegqddujeejkeehheehvddqsghoqhhunhdrfhgvnhhgpeepgh hmrghilhdrtghomhesfhhigihmvgdrnhgrmhgvpdhnsggprhgtphhtthhopedvuddpmhho uggvpehsmhhtphhouhhtpdhrtghpthhtoheprhgtuhesvhhgvghrrdhkvghrnhgvlhdroh hrghdprhgtphhtthhopehjihgrnhhgshhhrghnlhgrihesghhmrghilhdrtghomhdprhgt phhtthhopehprghulhhmtghksehkvghrnhgvlhdrohhrghdprhgtphhtthhopehjohhshh esjhhoshhhthhrihhplhgvthhtrdhorhhgpdhrtghpthhtoheprhhoshhtvgguthesghho ohgumhhishdrohhrghdprhgtphhtthhopehmrghthhhivghurdguvghsnhhohigvrhhsse gvfhhfihgtihhoshdrtghomhdprhgtphhtthhopehfrhgvuggvrhhitgeskhgvrhhnvghl rdhorhhgpdhrtghpthhtohepnhgvvghrrghjrdhuphgrughhhigrhieskhgvrhhnvghlrd horhhgpdhrtghpthhtohepjhhovghlsehjohgvlhhfvghrnhgrnhguvghsrdhorhhg X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 23 Feb 2025 21:22:38 -0500 (EST) From: Boqun Feng To: rcu@vger.kernel.org Cc: Lai Jiangshan , "Paul E. McKenney" , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Uladzislau Rezki , Zqiang , Davidlohr Bueso , Shuah Khan , linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, bpf@vger.kernel.org, Alexei Starovoitov , Andrii Nakryiko , Peter Zijlstra , Kent Overstreet Subject: [PATCH rcu 15/20] refscale: Add srcu_read_lock_fast() support using "srcu-fast" Date: Sun, 23 Feb 2025 18:22:09 -0800 Message-Id: <20250224022214.12037-16-boqun.feng@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250224022214.12037-1-boqun.feng@gmail.com> References: <20250224022214.12037-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" This commit creates a new srcu-fast option for the refscale.scale_type module parameter that selects srcu_read_lock_fast() and srcu_read_unlock_fast(). Signed-off-by: Paul E. McKenney Cc: Alexei Starovoitov Cc: Andrii Nakryiko Cc: Peter Zijlstra Cc: Kent Overstreet Cc: Signed-off-by: Boqun Feng --- kernel/rcu/refscale.c | 32 +++++++++++++++++++++++++++++++- 1 file changed, 31 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/refscale.c b/kernel/rcu/refscale.c index 1b47376acdc4..f11a7c2af778 100644 --- a/kernel/rcu/refscale.c +++ b/kernel/rcu/refscale.c @@ -216,6 +216,36 @@ static const struct ref_scale_ops srcu_ops = { .name = "srcu" }; +static void srcu_fast_ref_scale_read_section(const int nloops) +{ + int i; + struct srcu_ctr __percpu *scp; + + for (i = nloops; i >= 0; i--) { + scp = srcu_read_lock_fast(srcu_ctlp); + srcu_read_unlock_fast(srcu_ctlp, scp); + } +} + +static void srcu_fast_ref_scale_delay_section(const int nloops, const int udl, const int ndl) +{ + int i; + struct srcu_ctr __percpu *scp; + + for (i = nloops; i >= 0; i--) { + scp = srcu_read_lock_fast(srcu_ctlp); + un_delay(udl, ndl); + srcu_read_unlock_fast(srcu_ctlp, scp); + } +} + +static const struct ref_scale_ops srcu_fast_ops = { + .init = rcu_sync_scale_init, + .readsection = srcu_fast_ref_scale_read_section, + .delaysection = srcu_fast_ref_scale_delay_section, + .name = "srcu-fast" +}; + static void srcu_lite_ref_scale_read_section(const int nloops) { int i; @@ -1163,7 +1193,7 @@ ref_scale_init(void) long i; int firsterr = 0; static const struct ref_scale_ops *scale_ops[] = { - &rcu_ops, &srcu_ops, &srcu_lite_ops, RCU_TRACE_OPS RCU_TASKS_OPS + &rcu_ops, &srcu_ops, &srcu_fast_ops, &srcu_lite_ops, RCU_TRACE_OPS RCU_TASKS_OPS &refcnt_ops, &rwlock_ops, &rwsem_ops, &lock_ops, &lock_irq_ops, &acqrel_ops, &sched_clock_ops, &clock_ops, &jiffies_ops, &typesafe_ref_ops, &typesafe_lock_ops, &typesafe_seqlock_ops, From patchwork Mon Feb 24 02:22:10 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boqun Feng X-Patchwork-Id: 13987329 Received: from mail-qk1-f182.google.com (mail-qk1-f182.google.com [209.85.222.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 566A32066F2; Mon, 24 Feb 2025 02:22:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363763; cv=none; b=GAw6Wu93R14ZZuL2b4iV0hKWHlbpT1vCCSfg08+Kfc9rdt1WnCTI5mUX1dxTH+IJs8ympySLXC1N5j0puOhs4LxxjwOgA3399IX2TlejHSylmMmDKA748QF6nYWjqKBdGF70wEmA/LTIvXKziLBKDv0yZ3EOjzP3dTcI1wbJ6cU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363763; c=relaxed/simple; bh=kAUdkuuHIe0hHKVg4sC/0Qh5Fb6pN442dDVYb1BISTo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=G1gDYd4f7mNNFdU/KH0zWXN99Z66Ac5Zv3oLqY8ZGXaoBRqZiC6tYawpzbSHwLM0Kovunq4DL+XWSr/pocuglPpsKFUuULraPiiq0JyKgW2uYrkPiXt8dA8MyIGniRuPuRS9rPs9omh/cQzmBWJqloHXDgHL2/o3VuTRA/6VpVI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=UheFGIen; arc=none smtp.client-ip=209.85.222.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="UheFGIen" Received: by mail-qk1-f182.google.com with SMTP id af79cd13be357-7c0ade6036aso492449185a.0; Sun, 23 Feb 2025 18:22:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740363761; x=1740968561; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=HoXXo7te2qdCoqRU2DAsTQhDKL9OLvKsbtcGFLuSrXc=; b=UheFGIenFPdOK3I3vZbk7qW684gRr4c72LvY3IBr+Ob+BwejHRuFSgACYSvYru7o8G AhAsbmkd0uZHfFLyOpJNRO1ZwIgBEq9tF1WctxAJgZuHmDQe3hh4Fj6gmp8ih22K0MJx FaRFRRRCbh+01+Gth/+DzB1GL15L93UdmtXu2anPQdP4mS2xj5A00eWppFuWlKRuuqum Fz6xh/3UtE7/Ii07EXftNW0qEvBVN/Yyv2Wkyzw8IxswpQAwGe5lK+aoAqwgvTeJbps9 2CtCDt7PuHFCHPeWnVkcMAY26YCfGSBPZuEMccjgMQQykF6ubdjh9a00DhCvoQkTlJPR yY6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740363761; x=1740968561; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=HoXXo7te2qdCoqRU2DAsTQhDKL9OLvKsbtcGFLuSrXc=; b=OOiRb9VxLk/Sl45uyYK1vGrYZX2hmkRuKW2REauAD6mSn7QfKPlHkLTHvkhfHev2vE jerxJ3lWHAVdCTB47tKU/mL27cJPibRRcxhDaMJDCrmnD7xDTK6SOnF99A8ACaRkdxDp D16Ech6aXysrKjHI/ynOFht1xgIgJ+Vvpf0sV5CXeN1By+afYr8IbaTNkI8/bDkUaJmb PioIt8qA15YoB7lymsXUmXI4bZYYb0ZgppNtmALYcvisdqV+awqVUVH77cwIOhancSMM qDnenmac+kw8yb2TGu1mw4ZkrxIMzowOVpYIsxfzNtwnIovz4P7cTWn1uD5EUreN1iJe RYgQ== X-Forwarded-Encrypted: i=1; AJvYcCV9j8HQCrArVAJmxwHAhIRsc34Dhxl3YT1EhQpzHznDTPYRh3e6BXYHqURsx9u2a1I8MNI=@vger.kernel.org, AJvYcCX2iCRegL8FV/NH5OpxK+wuvwnh7bjgTjoPWc4ipyTBm1EjIhtxB7vji0uJHslQwxWrnA0a5RAnOFMxy8de2Vlp@vger.kernel.org, AJvYcCXZjdV0aKsYfvFIQVdn0VwacPZvpjvhammA6ocpETxPA3qtKwvQijQNZP4pOYUfoOr9P3VToJe94CIP1LO/@vger.kernel.org X-Gm-Message-State: AOJu0YzSVItVtFvl+pv12tHp4U5xL/I4rr9eEkJGqMq0rKOq4Baei1m7 Gnbjt4sNZEVbLkIAec2Rekbgrqn2P7rMbbi9bmteALH4dZ/IBFdq X-Gm-Gg: ASbGncvtcprgay+9DY4IwvYc2xsrQhReVwCYSnqJ58reZlVSzQt8ob1EDgwhGXd+GcO JvqLIRte4iHPozauSABDvEO1Maz4XigS5WObvd3eDmxrORMC3ckCLSXHlJ9pvr1fmTcMJXNISSv UTTE6G4ObP3REPQNVgCr9h8kY1s/peYOVoQu7LlhFh34wDW5RrZ13IqBuymcPTNYP37E9xXTykd 8upkcfkrYKxklWKXqZRcwTGaDrWG2HExGM9pLVhSolOfFUXOi1wdmn/kSJGlJ1r6XWeEY08fbJj 4z8gVljCtCLIOz3Z8uPiw/x+xtBXR28c9Qj95uawOCznILp3vH74ge3H+to593V9fXOfjGwMY5N 62nEpTHZILGw27AJR X-Google-Smtp-Source: AGHT+IEZNGEvL+X6WQsjqqBoVy8Xl/SkDXAOELoRPMIY4g9ibmVObQC07YvQ0QLcC5b11t6JdM6nkw== X-Received: by 2002:a05:6214:d07:b0:6e6:69e4:825e with SMTP id 6a1803df08f44-6e6ae8f3b6cmr123259076d6.21.1740363761352; Sun, 23 Feb 2025 18:22:41 -0800 (PST) Received: from fauth-a2-smtp.messagingengine.com (fauth-a2-smtp.messagingengine.com. [103.168.172.201]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6e69c93e8d0sm56723336d6.108.2025.02.23.18.22.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 23 Feb 2025 18:22:41 -0800 (PST) Received: from phl-compute-12.internal (phl-compute-12.phl.internal [10.202.2.52]) by mailfauth.phl.internal (Postfix) with ESMTP id 880FF1200043; Sun, 23 Feb 2025 21:22:40 -0500 (EST) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-12.internal (MEProxy); Sun, 23 Feb 2025 21:22:40 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgdejjeehhecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvfevufffkffojghfggfgsedtkeertdertddt necuhfhrohhmpeeuohhquhhnucfhvghnghcuoegsohhquhhnrdhfvghnghesghhmrghilh drtghomheqnecuggftrfgrthhtvghrnhepgeeljeeitdehvdehgefgjeevfeejjeekgfev ffeiueejhfeuiefggeeuheeggefgnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrg hmpehmrghilhhfrhhomhepsghoqhhunhdomhgvshhmthhprghuthhhphgvrhhsohhnrghl ihhthidqieelvdeghedtieegqddujeejkeehheehvddqsghoqhhunhdrfhgvnhhgpeepgh hmrghilhdrtghomhesfhhigihmvgdrnhgrmhgvpdhnsggprhgtphhtthhopedujedpmhho uggvpehsmhhtphhouhhtpdhrtghpthhtoheprhgtuhesvhhgvghrrdhkvghrnhgvlhdroh hrghdprhgtphhtthhopehjihgrnhhgshhhrghnlhgrihesghhmrghilhdrtghomhdprhgt phhtthhopehprghulhhmtghksehkvghrnhgvlhdrohhrghdprhgtphhtthhopehjohhshh esjhhoshhhthhrihhplhgvthhtrdhorhhgpdhrtghpthhtoheprhhoshhtvgguthesghho ohgumhhishdrohhrghdprhgtphhtthhopehmrghthhhivghurdguvghsnhhohigvrhhsse gvfhhfihgtihhoshdrtghomhdprhgtphhtthhopehfrhgvuggvrhhitgeskhgvrhhnvghl rdhorhhgpdhrtghpthhtohepnhgvvghrrghjrdhuphgrughhhigrhieskhgvrhhnvghlrd horhhgpdhrtghpthhtohepjhhovghlsehjohgvlhhfvghrnhgrnhguvghsrdhorhhg X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 23 Feb 2025 21:22:40 -0500 (EST) From: Boqun Feng To: rcu@vger.kernel.org Cc: Lai Jiangshan , "Paul E. McKenney" , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Uladzislau Rezki , Zqiang , Davidlohr Bueso , Shuah Khan , linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH rcu 16/20] rcutorture: Make scenario SRCU-P use srcu_read_lock_fast() Date: Sun, 23 Feb 2025 18:22:10 -0800 Message-Id: <20250224022214.12037-17-boqun.feng@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250224022214.12037-1-boqun.feng@gmail.com> References: <20250224022214.12037-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" This commit causes the rcutorture SRCU-P scenario use the srcu_read_lock_fast() and srcu_read_unlock_fast() functions. This will cause these two functions to be regularly tested by several developers (myself included), for example, those who use torture.sh as an RCU acceptance test. Signed-off-by: Paul E. McKenney Signed-off-by: Boqun Feng --- tools/testing/selftests/rcutorture/configs/rcu/SRCU-P.boot | 1 + 1 file changed, 1 insertion(+) diff --git a/tools/testing/selftests/rcutorture/configs/rcu/SRCU-P.boot b/tools/testing/selftests/rcutorture/configs/rcu/SRCU-P.boot index 2db39f298d18..fb61703690cb 100644 --- a/tools/testing/selftests/rcutorture/configs/rcu/SRCU-P.boot +++ b/tools/testing/selftests/rcutorture/configs/rcu/SRCU-P.boot @@ -2,3 +2,4 @@ rcutorture.torture_type=srcud rcupdate.rcu_self_test=1 rcutorture.fwd_progress=3 srcutree.big_cpu_lim=5 +rcutorture.reader_flavor=0x8 From patchwork Mon Feb 24 02:22:11 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boqun Feng X-Patchwork-Id: 13987330 Received: from mail-qv1-f50.google.com (mail-qv1-f50.google.com [209.85.219.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BB65C227E86; Mon, 24 Feb 2025 02:22:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363765; cv=none; b=lqgwVNoW/2G1mKc1b5gTou5DfHVUinQMBFR/t0CYOtqPOgNGC/DpjJKoB1PKhXyID2uqXOAdg/1NGqj25cLn29+yu/ZcpaKDzWZme5eZaMGFPaMk6RPYmLupgB4XnLrrVh8OU4OyfZ9GkAmjMB2iQ7Q/zN36KacvBIDpaVCuGR0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363765; c=relaxed/simple; bh=ivvgjhZ1dzBCvoS8KY01scRvocfceY53UhjRI5+E+TE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Uy/IXe0CCL2PCJy3XE1idjKoBu6cdXIn6p2b0lDSOU1LxpNvECUsBDQykzHxfZWb4z/+5lIrWJVRAbtbAiHi/RWaC6FraNKhQMrR3pYDIcUebtWyIVTyk3J8u/JN8cTil8kBjS09tQNqBsI2i4NAz/xT0kQ6JwkM2NrcgEq6XD0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=gzX2nhIX; arc=none smtp.client-ip=209.85.219.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="gzX2nhIX" Received: by mail-qv1-f50.google.com with SMTP id 6a1803df08f44-6e6719e6e82so33524416d6.2; Sun, 23 Feb 2025 18:22:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740363762; x=1740968562; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=uQqtypxCvU6d0bgYV6SXXcCNS5AjtvZwsSrkwgJ8kLo=; b=gzX2nhIXn2Nh/pvLdN7wqbfvqepU62uD6KT30JplemCP+GjqNNfSMEJjT1Cw9xz8x6 S6yPmQIoFgX5IubmwI2qeXz+7KqyG9sQh4aY6aPLNm5uNkyiF7JzjsCVZ/a7/wkOhA5H qWpQre8vhwzQCNxgUF1OryljqFu9AXzY+pdGXSCWn9o5NuxnaYvLBjhcNRInuTVKW8aL u48EECMPjdR7EJunyKE7yjTu6eDoSydBTOBhIuVgxq/JoIpo/jU5uyNc0ZEi9dzVQLMy dl2rv8B6z2hNj2pGBWVsOQWQjO++HolSgz3520fTNWsVwyysoo9Ij2vhmEFkmIp4tVnC 9uIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740363762; x=1740968562; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=uQqtypxCvU6d0bgYV6SXXcCNS5AjtvZwsSrkwgJ8kLo=; b=ILR7KxqHskMKZJ3LATsegRIpDmr1s0+6BArqBC5BE80lp7EPi4q1KlkSS8YvlqC42k 9rdU841fiFDGCRfoW4+sKdSRbI25j1EE8pMNEb4P2OWjuKljxhzwf6k7wpFN6RloBMtT YEb93XPGb/H+F7zuZsohwQKCJEthK2CGW7CuHZTgDfBLlviCfk//qx3zWvokXzEg2qDg NZHkhI9jqP8wX27o7yJPwoZMgRpy9bY6DnHHGE6S1qPLwIMDKLYqBobnP6mhmO3uhXzK t1jao8pZAM+QVnJWt7ttrpCQnO5wxEFuy5XvEx0zn8pCp2fiRCRvgx1slFw9ixH/Nrvy 5huw== X-Forwarded-Encrypted: i=1; AJvYcCVevp1S8jRgvzPTJQyQHHt+lOvLc+abC5W62jw/Gb+R7impZ3eXGpLIth/L/RhuyIUf1W4TRGcumIyabKcusakj@vger.kernel.org, AJvYcCWJIA7JGjXjAUaSPaOArVnuwMHDcutRhgVgtHTrCMkjxvGtiKT9LLJCQkhF6lKR8xfsSEM=@vger.kernel.org, AJvYcCX7WqADk52m41NZT5sPE0Yc0TLzShm9wWI8J2+Y41qJLAb1Lt8Xs+rdwHg/lduRXso5mgLz8o6+EdrHUEmL@vger.kernel.org X-Gm-Message-State: AOJu0Yy+PCzzQYxM5unZCxnVgzHARQKSvM2JWX5GmrMQRy47y+fqRuyz 6dlcnzCaJ2WodpzfsJVRkz9x6fcZf0Fzmynajf9eGBZq2J70TbB0 X-Gm-Gg: ASbGncvDAI5Q3rG5b0SeAuFDSiBhywewsqgmOB/uirbUTgifXuulTOWw15xvpIHNgdP 6OehkRh+6DcjCrVLapWY2SOiN5+Dwvz3WkN7zLHiHYVFzBgPZRiKd22wmzFzhyjqmsvWJtXyhYh 7mAykGa6jMKPPsUN1zMDpy0A4POO+eDD4VELE8JEBBO4BwfwonWO34biCT8tx11EFBA4Y5dk91N jQv8IYTauNlQOrlU/bUgfLfSdwwQp5S6AzfCIQNNI/7Lfavae5JFTKxvlmpaMUBXZBvMZJi3zhE gS9+N/zxcq31088QLE/cRZGqbY9aGoYU7i7yylicbYLygioRkfa47IVH49Q1Pnwrvk415n4yZKM UMS5q+bw6wBkj2I6g X-Google-Smtp-Source: AGHT+IEkuhFM1Iq0XCFaLoezdyIrc0QYsLOXTpVv+69A5554K4E8nebum/GazflySakZWD8qUdK27Q== X-Received: by 2002:a05:6214:20e7:b0:6e6:62e0:886a with SMTP id 6a1803df08f44-6e6b01dc31amr153086266d6.43.1740363762621; Sun, 23 Feb 2025 18:22:42 -0800 (PST) Received: from fauth-a2-smtp.messagingengine.com (fauth-a2-smtp.messagingengine.com. [103.168.172.201]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6e65d7792f5sm127714046d6.18.2025.02.23.18.22.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 23 Feb 2025 18:22:42 -0800 (PST) Received: from phl-compute-01.internal (phl-compute-01.phl.internal [10.202.2.41]) by mailfauth.phl.internal (Postfix) with ESMTP id E51991200043; Sun, 23 Feb 2025 21:22:41 -0500 (EST) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-01.internal (MEProxy); Sun, 23 Feb 2025 21:22:41 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgdejjeehgecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvfevufffkffojghfggfgsedtkeertdertddt necuhfhrohhmpeeuohhquhhnucfhvghnghcuoegsohhquhhnrdhfvghnghesghhmrghilh drtghomheqnecuggftrfgrthhtvghrnhepgeeljeeitdehvdehgefgjeevfeejjeekgfev ffeiueejhfeuiefggeeuheeggefgnecuvehluhhsthgvrhfuihiivgepgeenucfrrghrrg hmpehmrghilhhfrhhomhepsghoqhhunhdomhgvshhmthhprghuthhhphgvrhhsohhnrghl ihhthidqieelvdeghedtieegqddujeejkeehheehvddqsghoqhhunhdrfhgvnhhgpeepgh hmrghilhdrtghomhesfhhigihmvgdrnhgrmhgvpdhnsggprhgtphhtthhopedujedpmhho uggvpehsmhhtphhouhhtpdhrtghpthhtoheprhgtuhesvhhgvghrrdhkvghrnhgvlhdroh hrghdprhgtphhtthhopehjihgrnhhgshhhrghnlhgrihesghhmrghilhdrtghomhdprhgt phhtthhopehprghulhhmtghksehkvghrnhgvlhdrohhrghdprhgtphhtthhopehjohhshh esjhhoshhhthhrihhplhgvthhtrdhorhhgpdhrtghpthhtoheprhhoshhtvgguthesghho ohgumhhishdrohhrghdprhgtphhtthhopehmrghthhhivghurdguvghsnhhohigvrhhsse gvfhhfihgtihhoshdrtghomhdprhgtphhtthhopehfrhgvuggvrhhitgeskhgvrhhnvghl rdhorhhgpdhrtghpthhtohepnhgvvghrrghjrdhuphgrughhhigrhieskhgvrhhnvghlrd horhhgpdhrtghpthhtohepjhhovghlsehjohgvlhhfvghrnhgrnhguvghsrdhorhhg X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 23 Feb 2025 21:22:41 -0500 (EST) From: Boqun Feng To: rcu@vger.kernel.org Cc: Lai Jiangshan , "Paul E. McKenney" , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Uladzislau Rezki , Zqiang , Davidlohr Bueso , Shuah Khan , linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH rcu 17/20] srcu: Fix srcu_read_unlock_{lite,nmisafe}() kernel-doc Date: Sun, 23 Feb 2025 18:22:11 -0800 Message-Id: <20250224022214.12037-18-boqun.feng@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250224022214.12037-1-boqun.feng@gmail.com> References: <20250224022214.12037-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" The srcu_read_unlock_lite() and srcu_read_unlock_nmisafe() both say that their idx parameters must come from srcu_read_lock(). This would be bad, because a given srcu_struct structure may be used only with one flavor of SRCU reader. This commit therefore updates the srcu_read_unlock_lite() kernel-doc header to say that its idx parameter must be obtained from srcu_read_lock_lite() and the srcu_read_unlock_nmisafe() kernel-doc header to say that its idx parameter must be obtained from srcu_read_lock_nmisafe(). Signed-off-by: Paul E. McKenney Signed-off-by: Boqun Feng --- include/linux/srcu.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/include/linux/srcu.h b/include/linux/srcu.h index 63bddc301423..a0df80baaccf 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -402,7 +402,7 @@ static inline void srcu_read_unlock_fast(struct srcu_struct *ssp, struct srcu_ct /** * srcu_read_unlock_lite - unregister a old reader from an SRCU-protected structure. * @ssp: srcu_struct in which to unregister the old reader. - * @idx: return value from corresponding srcu_read_lock(). + * @idx: return value from corresponding srcu_read_lock_lite(). * * Exit a light-weight SRCU read-side critical section. */ @@ -418,7 +418,7 @@ static inline void srcu_read_unlock_lite(struct srcu_struct *ssp, int idx) /** * srcu_read_unlock_nmisafe - unregister a old reader from an SRCU-protected structure. * @ssp: srcu_struct in which to unregister the old reader. - * @idx: return value from corresponding srcu_read_lock(). + * @idx: return value from corresponding srcu_read_lock_nmisafe(). * * Exit an SRCU read-side critical section, but in an NMI-safe manner. */ From patchwork Mon Feb 24 02:22:12 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boqun Feng X-Patchwork-Id: 13987331 Received: from mail-qv1-f42.google.com (mail-qv1-f42.google.com [209.85.219.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 24EA724060C; Mon, 24 Feb 2025 02:22:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363766; cv=none; b=phjiAzXHsgPj000u6mr8q3s1CjfiDH1Xr0j7lMSeWxv6DXL8wgt/80u+YFYFbqfNmnpWIROR53HUBYKu4C5kWyefMXTMkForlPxJ626V1IGdD4gby8CtumKPKmkLyZf/CJ8lqbgRY0PEuWu0BeSBtM8P90GeswE7POFQpelKL+8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363766; c=relaxed/simple; bh=mRU9WCM67lJH5/lbA8uOg4PhTTj8coyG+QePvNGba3k=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Iz/E8RKGavcLcA9O+Tx5r4lsEOFirbxFZmOgUYLXRKMR1k06Mcp6sm6eYHQGFXcR2qCgwKMY6W2Rowd3Dv7MJmOsNn9rxqsfNfLnqftBd4adeZz0gRSxbmsbyLZp6Vm+D4C0d5qKozQZQbyZKrhlATiqtIQJujnsxFefckjcxEw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=STa3TsHG; arc=none smtp.client-ip=209.85.219.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="STa3TsHG" Received: by mail-qv1-f42.google.com with SMTP id 6a1803df08f44-6e67fad4671so35417956d6.1; Sun, 23 Feb 2025 18:22:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740363764; x=1740968564; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=eg6dUqpGoKhiV+cpu434M/fIW4z0zJ0cT+ywlwyprkA=; b=STa3TsHGJtVb+MCCGHC2sB6gvb5cnWXjN892Yue9Jq0rRDfO5B6LsCbQ2O79PP9llf XeTe8iY5qTueKaqUtCQE5XGkjFFfKhPDs/4PunaCEH1+Bp/LErEY4bHQQOliqEPSqQP8 X8TZFKEdARr1Qi4gKXTgwRlYmAYmBZ/S6dmH6BN3Cw078SJlG+FUpqgq9hWfpduqrttg EP5W0RQTZxykJYB4YcLuZHspxiDwMhdzrO0QHvEYdwS6Dj1yQ0ZQBvArBzsCtgoxJods W1l0n6yClTMmUnfa4pkqWmff8WbCH97dqilf/0WkgpHsmsiZlP3U4pZ6BshT92sGpkCb 2mnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740363764; x=1740968564; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=eg6dUqpGoKhiV+cpu434M/fIW4z0zJ0cT+ywlwyprkA=; b=P6JgmEPlTWPEEpPo61yT7mEKnV8sPqV7feH29rRluMgJSFnaCr/FW8fsS2hVKwIw4w T3+IpupwRxUIIexn1V+NxK6/lkCDbLk58qtQnhfZVSshjOYnrxUoAup8iAVo14ZhI7Bu ZR6gKEf5cAY4rsUlpJb7rIzu+SttnaHOkQ0dvA1dI0g65+3FJ2tH+aCIHHEt2pWFmRFi XYQEv8bcBMCfeKWmTpn9MZ3bsDOawsVdMTpb86c2LmN8MspWbMRWRx7qHH28dvWuvnjZ 9HPnGZLRum5HXWftmpvMheIBWsGdpwyoHfS6XtIA3JX4TbCCEEmFFJ5r8vcDJwfxsmAt iD0Q== X-Forwarded-Encrypted: i=1; AJvYcCWBf4aFLi3TAcMMCcglyC2x1blEYzEHWVe0SArJYvy62s1jPsTqQcnoxDYb1ZMqATOOm5gA9iaWKvwi0E8U@vger.kernel.org, AJvYcCWK0UD0UnHItb49ZFez36bm5Z5Uhrpk55u8X9LBwHHilUXa/tpvGjJtDxzFskJFC3NN2cGDlxI6Bkohswqbv7WD@vger.kernel.org, AJvYcCWiq2n5Sx/nQ/TRi4fa/Ml6MdUxFhy2S+KZj83ONKN22VAEgR28RdNGAV24/2sO7dl/cUY=@vger.kernel.org X-Gm-Message-State: AOJu0Yy/E8f16js9tvlpk7F1RfRwiUgDUhaZi5TPhXHVxvs6DXFppBqx lpygdtc5iLfbU8r9zxUQk5ScEOUec1ibWfbQ3K7j/KTI8e4RQPNW X-Gm-Gg: ASbGncvD+ML09oDBiJ19x7uzYqt8Cpkw+Bq68wifSP2hZX7UZVvxdd9cGIpubPITmBX 6h9JVTIKXy4GYy16cXKinaspR4Xd2Uv2PzkHHz5blubKCO/J3ITfKy2AcIiIa34isgzon/SJVjC i3mDowyISWjku8OChMGVpCtlaBXUs83+S8/6ZpJu+2iaN5uxaTH8ODoX39sx1BKRGCfSpImXIY3 yB0VhDYL3riOQ/NMfkysRR/8Jb95IPBsgYc2hK161wxzr8/9SXiAWxPBT4K7COA/Eavd/FQECZJ JaGyPjggmPMnIRvCq8bAEXXjHnNnjjorKj/AUhhMzc4htyU2qnTK3lhWWcDcOewhHMRPnbljK+O aFCptKovjORYMkGv+ X-Google-Smtp-Source: AGHT+IGzIhpbr9iU21ineA2QNZuQdz9ARj2vtfEGBqgEatKHIidqLPnjQqCQPKzUU874OlLqnK72cg== X-Received: by 2002:ad4:5fcd:0:b0:6e4:4011:9df7 with SMTP id 6a1803df08f44-6e6ae7f876emr161379806d6.16.1740363764052; Sun, 23 Feb 2025 18:22:44 -0800 (PST) Received: from fauth-a2-smtp.messagingengine.com (fauth-a2-smtp.messagingengine.com. [103.168.172.201]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6e6b1759af6sm36879046d6.93.2025.02.23.18.22.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 23 Feb 2025 18:22:43 -0800 (PST) Received: from phl-compute-09.internal (phl-compute-09.phl.internal [10.202.2.49]) by mailfauth.phl.internal (Postfix) with ESMTP id 404341200066; Sun, 23 Feb 2025 21:22:43 -0500 (EST) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-09.internal (MEProxy); Sun, 23 Feb 2025 21:22:43 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgdejjeehgecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvfevufffkffojghfggfgsedtkeertdertddt necuhfhrohhmpeeuohhquhhnucfhvghnghcuoegsohhquhhnrdhfvghnghesghhmrghilh drtghomheqnecuggftrfgrthhtvghrnhepgeeljeeitdehvdehgefgjeevfeejjeekgfev ffeiueejhfeuiefggeeuheeggefgnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrg hmpehmrghilhhfrhhomhepsghoqhhunhdomhgvshhmthhprghuthhhphgvrhhsohhnrghl ihhthidqieelvdeghedtieegqddujeejkeehheehvddqsghoqhhunhdrfhgvnhhgpeepgh hmrghilhdrtghomhesfhhigihmvgdrnhgrmhgvpdhnsggprhgtphhtthhopedvuddpmhho uggvpehsmhhtphhouhhtpdhrtghpthhtoheprhgtuhesvhhgvghrrdhkvghrnhgvlhdroh hrghdprhgtphhtthhopehjihgrnhhgshhhrghnlhgrihesghhmrghilhdrtghomhdprhgt phhtthhopehprghulhhmtghksehkvghrnhgvlhdrohhrghdprhgtphhtthhopehjohhshh esjhhoshhhthhrihhplhgvthhtrdhorhhgpdhrtghpthhtoheprhhoshhtvgguthesghho ohgumhhishdrohhrghdprhgtphhtthhopehmrghthhhivghurdguvghsnhhohigvrhhsse gvfhhfihgtihhoshdrtghomhdprhgtphhtthhopehfrhgvuggvrhhitgeskhgvrhhnvghl rdhorhhgpdhrtghpthhtohepnhgvvghrrghjrdhuphgrughhhigrhieskhgvrhhnvghlrd horhhgpdhrtghpthhtohepjhhovghlsehjohgvlhhfvghrnhgrnhguvghsrdhorhhg X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 23 Feb 2025 21:22:42 -0500 (EST) From: Boqun Feng To: rcu@vger.kernel.org Cc: Lai Jiangshan , "Paul E. McKenney" , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Uladzislau Rezki , Zqiang , Davidlohr Bueso , Shuah Khan , linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, bpf@vger.kernel.org, Alexei Starovoitov , Andrii Nakryiko , Peter Zijlstra , Kent Overstreet Subject: [PATCH rcu 18/20] srcu: Document that srcu_{read_lock,down_read}() can share srcu_struct Date: Sun, 23 Feb 2025 18:22:12 -0800 Message-Id: <20250224022214.12037-19-boqun.feng@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250224022214.12037-1-boqun.feng@gmail.com> References: <20250224022214.12037-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" This commit adds a sentence to the srcu_down_read() function's kernel-doc header noting that it is permissible to use srcu_down_read() and srcu_read_lock() on the same srcu_struct, even concurrently. Signed-off-by: Paul E. McKenney Cc: Alexei Starovoitov Cc: Andrii Nakryiko Cc: Peter Zijlstra Cc: Kent Overstreet Cc: Signed-off-by: Boqun Feng --- include/linux/srcu.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/include/linux/srcu.h b/include/linux/srcu.h index a0df80baaccf..317eab82a5f0 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -359,7 +359,8 @@ srcu_read_lock_notrace(struct srcu_struct *ssp) __acquires(ssp) * srcu_down_read() nor srcu_up_read() may be invoked from an NMI handler. * * Calls to srcu_down_read() may be nested, similar to the manner in - * which calls to down_read() may be nested. + * which calls to down_read() may be nested. The same srcu_struct may be + * used concurrently by srcu_down_read() and srcu_read_lock(). */ static inline int srcu_down_read(struct srcu_struct *ssp) __acquires(ssp) { From patchwork Mon Feb 24 02:22:13 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boqun Feng X-Patchwork-Id: 13987332 Received: from mail-qv1-f42.google.com (mail-qv1-f42.google.com [209.85.219.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 971F1241682; Mon, 24 Feb 2025 02:22:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363768; cv=none; b=CTrwMJld4nTA6/aHxbUvcoqhy7cMgEg9RKhcPhAbfUrkuIwmskktl5hXzGnRUh5Z47cp7Iy85ikRUsluHIVq9Aj3wWHdMCGJOq2GdPhfR/Jp6PxifyNmcZzVLjJsE+okygybmRGVWpiMfrTcFwmy2MoA9h+N2gIV0TcaGVSLBiA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363768; c=relaxed/simple; bh=aBreVKJt9M2Mo1GiqPEfTdZ2p2QA3tyIinn8cNmB6YY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=pcTlLe8mF2eA4vD2skjC0r6B+CEwzxJWUFDohGJQ+dQdLoOBpiao8TTpEvTdNgnNys3pNnE4zl4VKdSDrA9pVxUJxVJmuhOHUTTIZE2qp7mDmQFqF3U01H1IId+RtSWMleSkAAHJPK6Raf8ZJSR5PDp48OREArSr5BUDAhUeanQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=ZuecK9Wz; arc=none smtp.client-ip=209.85.219.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ZuecK9Wz" Received: by mail-qv1-f42.google.com with SMTP id 6a1803df08f44-6dd0d09215aso27612806d6.2; Sun, 23 Feb 2025 18:22:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740363765; x=1740968565; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=6iNUHb0+00l4f0ZTUqicqdX9tAj3JGSCI33HuVMMHHs=; b=ZuecK9WzL86Xk164a3yDwSXpHV9oDwIlsGxNW2lWuqCrHf/DuVq03yeTyQWcpl8VSz kx/aTzcMTkyT+As/kde8OykVlm8+R7+7haMnG0fsmcu/+uWa855GiMTlqsI+x0AfAZrW 6FDFrsvdS2pKxAOwNAh1gncYJTK6GJy7bUXquZggbRh4B6AyCYF01zhMElgZMj/q05h6 q6RhfQgdVwyLeV6u1aCARMZbKeTf0jq7AYUaFXgUMDNMR4uVRQbOXGj5KcS58JxOogbg SVK3SqNhb5RmY4f3G2WFCLHo6qNo6tCFo5s52+c+pOvBNn7TSYkC7B3v4AX71jqKp+P+ m1Uw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740363765; x=1740968565; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=6iNUHb0+00l4f0ZTUqicqdX9tAj3JGSCI33HuVMMHHs=; b=J8r89a4jJH2lzcs0UKfgUV6quiZWAgxDt4q/drbpMvnNuSGWIzJX+iNe0nriXKckPd rY85ZFg7nR0p/slG2MK+wbe1/q+A+3ACW0F5IMPX9nsRKsDYCdLWRBPQs2u+AJsL4oK8 iqGAUywrhcP5rDdhacwMr31hSu4q4SMb2djBrMZufSoWiQW6XfZdXB6J7duopOCqhzeh Yfr/YW8b4JQRC0oC3RJL7yx/jHv7Pl4+zBnIkeBJPSe+QNNbCO3qCzBbwTUuQDzrgYdu dy8sfCyfGTbvpC47Z9Yz9zW5UbXe2JajzCLBOYuvJiJCB/QElElKxcv/06IPgeypJhBV neNg== X-Forwarded-Encrypted: i=1; AJvYcCU309EVXTrRaBt8O861RE4bNxL24LAIfDj/0VH2Egm/BWc8bhrDNwJTM1tHn786GOJm7jVb8dvoace1tCDjimbI@vger.kernel.org, AJvYcCWSw3RtKyMCwCJG/DPoJCEADd7PpELSCyTN5j3mIup8pn46MxZ8laV8DJG1WPW0sVOE3fc=@vger.kernel.org, AJvYcCX9I6uH7mGvxf17t+DWkBkvpbphR52XfObmDKiaPJK/++vOi678WPdLTgcCRsauUCvK+QFGAbAr7skDQHsK@vger.kernel.org X-Gm-Message-State: AOJu0YxokuSubP1BO3C62Yujlr66JQGMe/LfHkV3/HLwHIEBmRB4uxSP P27bnbNWGh/WrBn0VEwaoaa7+kkqAocjwXX78E/COPFeATbSR76b X-Gm-Gg: ASbGnctpCoPQF3OCnLRQjmDPKLwnjVy/YvmPjPEpRVp2eC1+J4dK9p585VWTrq8hvbZ 9ufkGFFH8HLFRIirslnE9BeaaeVTyViA5DwE0R2n9TczLRRVwVxB7KlaIm2uxdHI/jelBSK+umJ WBdcdQX8/eMEm9/dmbHCvNJtwHuyZiC1Kgy/mp1PWmocwD9i5M8ueoWRkoaNdgob7yZqt5v5LjV 0a7BbYeZ4RAYU5fN06jsxi5/Xv9tQqr5MDpHI/bna8T3MhxInbTTT+NDd2VzDqZ32dnjgGFkYNq 1DP8DcNMRpsmHOQ89HIMkYEe2bNla/1U57ZKrnA/oEgfEd3J5y6r6wla0TvaxWlLupdlawDIaH4 RWMytRfJyuHUJ6CGn X-Google-Smtp-Source: AGHT+IFGnIWRDDkB+R8MV7hu1vyojxOiEmgcPh7SpQHpo6uSxFjLJ1i7br7f+Plyna8lLf0sf8pNkg== X-Received: by 2002:a05:6214:21e9:b0:6e6:5aa2:4e4f with SMTP id 6a1803df08f44-6e6b01a9dcemr175224056d6.32.1740363765483; Sun, 23 Feb 2025 18:22:45 -0800 (PST) Received: from fauth-a2-smtp.messagingengine.com (fauth-a2-smtp.messagingengine.com. [103.168.172.201]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6e6ac353309sm43626336d6.12.2025.02.23.18.22.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 23 Feb 2025 18:22:45 -0800 (PST) Received: from phl-compute-04.internal (phl-compute-04.phl.internal [10.202.2.44]) by mailfauth.phl.internal (Postfix) with ESMTP id 9A6DE1200043; Sun, 23 Feb 2025 21:22:44 -0500 (EST) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-04.internal (MEProxy); Sun, 23 Feb 2025 21:22:44 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgdejjeehgecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvfevufffkffojghfggfgsedtkeertdertddt necuhfhrohhmpeeuohhquhhnucfhvghnghcuoegsohhquhhnrdhfvghnghesghhmrghilh drtghomheqnecuggftrfgrthhtvghrnhepgeeljeeitdehvdehgefgjeevfeejjeekgfev ffeiueejhfeuiefggeeuheeggefgnecuvehluhhsthgvrhfuihiivgepudenucfrrghrrg hmpehmrghilhhfrhhomhepsghoqhhunhdomhgvshhmthhprghuthhhphgvrhhsohhnrghl ihhthidqieelvdeghedtieegqddujeejkeehheehvddqsghoqhhunhdrfhgvnhhgpeepgh hmrghilhdrtghomhesfhhigihmvgdrnhgrmhgvpdhnsggprhgtphhtthhopedvuddpmhho uggvpehsmhhtphhouhhtpdhrtghpthhtoheprhgtuhesvhhgvghrrdhkvghrnhgvlhdroh hrghdprhgtphhtthhopehjihgrnhhgshhhrghnlhgrihesghhmrghilhdrtghomhdprhgt phhtthhopehprghulhhmtghksehkvghrnhgvlhdrohhrghdprhgtphhtthhopehjohhshh esjhhoshhhthhrihhplhgvthhtrdhorhhgpdhrtghpthhtoheprhhoshhtvgguthesghho ohgumhhishdrohhrghdprhgtphhtthhopehmrghthhhivghurdguvghsnhhohigvrhhsse gvfhhfihgtihhoshdrtghomhdprhgtphhtthhopehfrhgvuggvrhhitgeskhgvrhhnvghl rdhorhhgpdhrtghpthhtohepnhgvvghrrghjrdhuphgrughhhigrhieskhgvrhhnvghlrd horhhgpdhrtghpthhtohepjhhovghlsehjohgvlhhfvghrnhgrnhguvghsrdhorhhg X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 23 Feb 2025 21:22:44 -0500 (EST) From: Boqun Feng To: rcu@vger.kernel.org Cc: Lai Jiangshan , "Paul E. McKenney" , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Uladzislau Rezki , Zqiang , Davidlohr Bueso , Shuah Khan , linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, bpf@vger.kernel.org, Andrii Nakryiko , Alexei Starovoitov , Peter Zijlstra , Kent Overstreet Subject: [PATCH rcu 19/20] srcu: Add srcu_down_read_fast() and srcu_up_read_fast() Date: Sun, 23 Feb 2025 18:22:13 -0800 Message-Id: <20250224022214.12037-20-boqun.feng@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250224022214.12037-1-boqun.feng@gmail.com> References: <20250224022214.12037-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" A pair of matching srcu_read_lock_fast() and srcu_read_unlock_fast() invocations must take place within the same context, for example, within the same task. Otherwise, lockdep complains, as is the right thing to do for most use cases. However, there are use cases involving tracing (for example, uretprobes) in which an SRCU reader needs to begin in one task and end in a timer handler, which might interrupt some other task. This commit therefore supplies the semaphore-like srcu_down_read_fast() and srcu_up_read_fast() functions, which act like srcu_read_lock_fast() and srcu_read_unlock_fast(), but permitting srcu_up_read_fast() to be invoked in a different context than was the matching srcu_down_read_fast(). Neither srcu_down_read_fast() nor srcu_up_read_fast() may be invoked from an NMI handler. Reported-by: Andrii Nakryiko Signed-off-by: Paul E. McKenney Cc: Alexei Starovoitov Cc: Peter Zijlstra Cc: Kent Overstreet Cc: Signed-off-by: Boqun Feng --- include/linux/srcu.h | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/include/linux/srcu.h b/include/linux/srcu.h index 317eab82a5f0..900b0d5c05f5 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -281,6 +281,24 @@ static inline struct srcu_ctr __percpu *srcu_read_lock_fast(struct srcu_struct * return retval; } +/** + * srcu_down_read_fast - register a new reader for an SRCU-protected structure. + * @ssp: srcu_struct in which to register the new reader. + * + * Enter a semaphore-like SRCU read-side critical section, but for + * a light-weight smp_mb()-free reader. See srcu_read_lock_fast() and + * srcu_down_read() for more information. + * + * The same srcu_struct may be used concurrently by srcu_down_read_fast() + * and srcu_read_lock_fast(). + */ +static inline struct srcu_ctr __percpu *srcu_down_read_fast(struct srcu_struct *ssp) __acquires(ssp) +{ + WARN_ON_ONCE(IS_ENABLED(CONFIG_PROVE_RCU) && in_nmi()); + srcu_check_read_flavor_force(ssp, SRCU_READ_FLAVOR_FAST); + return __srcu_read_lock_fast(ssp); +} + /** * srcu_read_lock_lite - register a new reader for an SRCU-protected structure. * @ssp: srcu_struct in which to register the new reader. @@ -400,6 +418,22 @@ static inline void srcu_read_unlock_fast(struct srcu_struct *ssp, struct srcu_ct __srcu_read_unlock_fast(ssp, scp); } +/** + * srcu_up_read_fast - unregister a old reader from an SRCU-protected structure. + * @ssp: srcu_struct in which to unregister the old reader. + * @scp: return value from corresponding srcu_read_lock_fast(). + * + * Exit an SRCU read-side critical section, but not necessarily from + * the same context as the maching srcu_down_read_fast(). + */ +static inline void srcu_up_read_fast(struct srcu_struct *ssp, struct srcu_ctr __percpu *scp) + __releases(ssp) +{ + WARN_ON_ONCE(IS_ENABLED(CONFIG_PROVE_RCU) && in_nmi()); + srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST); + __srcu_read_unlock_fast(ssp, scp); +} + /** * srcu_read_unlock_lite - unregister a old reader from an SRCU-protected structure. * @ssp: srcu_struct in which to unregister the old reader. From patchwork Mon Feb 24 02:22:14 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boqun Feng X-Patchwork-Id: 13987333 Received: from mail-qt1-f170.google.com (mail-qt1-f170.google.com [209.85.160.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1C06F241C87; Mon, 24 Feb 2025 02:22:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363769; cv=none; b=i9D3NoZk1FrM7o/PtsdPN9RuN+k6PYGYiM2X3gcxe5uqs8KWu6m0D/6RZ68PmcDZIITdz9gFl0TwkIgtb7DwCuV4ccFNaj0vnxbtbXeGLYiTt5bm5Wtpi78EYIlUOoBvbhamUCecT/E5rb8aeOaBcPL9YHO+6CRJa+qFoNg9FnA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740363769; c=relaxed/simple; bh=Z7wcVTQ2PMbca+VamTK60Bfots1ruiWDLXTJSx+zoDc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=DRTRyZMb1krESw/GuSAbqsndk2fl0cAlweIjuj9p6XKtHVPoKgWHb1XGMzUR3Kf8p+jAWpnNTsG64IbEDSIUtEMbTwOSGAe9YKES9jXdBbM6bPy/tW7LMwRJaWDKliYmcvYJief5dBti/G/Wz/cBiYAZmOGfr+Jpc+wci64me1g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=UTIdZnLC; arc=none smtp.client-ip=209.85.160.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="UTIdZnLC" Received: by mail-qt1-f170.google.com with SMTP id d75a77b69052e-47210ab1283so57654981cf.1; Sun, 23 Feb 2025 18:22:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740363767; x=1740968567; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=Meuv+MydLFL2FaXVwlNSjl8ifFrRUaf5oa8eloYWNAY=; b=UTIdZnLC/We+kcPwiP+quF3DmhU7+nqO6mZukeiFLr/z0iFRtVbn8yd1MHijbCSxcj EDxrxCfWN8rNocFsx4xt06RT+34A1t0EeoA+PDp4tPiN1hBK1P86ZNUK9Tclu3W9pAQZ 1nIyzFNG2Y3V6fBmMedi8pvBcPCqNAnRBQl2Eobnt6nMul63IaydXeI4gV2r8ViCE1RO 1AIjLR8MUBLDS2AXE+jvS2lWqGxA8/i68rF5QdZiTAbtYjs0ZEF70yBGAuawzuxQbYgA jiY8mOfxJhujYHpoa2fNetkSeirlMuY/1bgHg7CmZ6XWmFw5i0CdusBg3m1l/CiLh15X iudA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740363767; x=1740968567; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=Meuv+MydLFL2FaXVwlNSjl8ifFrRUaf5oa8eloYWNAY=; b=BoemCvro8iGuVbkte5zGSW4mLaTpJJvqZBXQUaWDOQEd3uTe5NYbfkVLYcBekVdofz ovtbhSSbP+VQH3FPMHwRi56onZIsNWajGqMRhp+8LR+37/LeTduDo0ZopZsbNQ5NGC8c PgV0PefAd1zegVR8DWFVg/QrX2F1dxqz5eH10JAFm08c2e1b7Ia0hr00Eg9e2leuSUCe noP5DZxkIN+MvO8rVniHHo6tVtBFA00jRyWsOOPPs2HoUDEwppD5DxyY2grL4pmJN2XH tKrfOY4AaI5CPbtkFh+cDWkoBh4rxQjM2cV65otETw3V9LIu467mWJSlVBl3XcBaUISW SEPg== X-Forwarded-Encrypted: i=1; AJvYcCU9nohnHBZH8xsO5AyH6umHDefn6FRH0Eo17QAs6M1kq7QaieJBfKXMYWLYBbQEj2K9fUg=@vger.kernel.org, AJvYcCUGywl/xdpWZmvl/G32A7kNc6iqpAjGNHc5lrK1KA2yHVulytKP3QcYqV+gkeZ1No8A9IbCYq5IdTSDG45yng2p@vger.kernel.org, AJvYcCXsnEDSpVA6W0NBvNxITCtIeMuiIY1HRsy0fKvtXpdJDCY06GvsHpfFERuyAbNEkgofRDMpDOAjPCI+Vfmm@vger.kernel.org X-Gm-Message-State: AOJu0YyNrfqKPNF95weA39UXJlbmZ3qYLkZfAJ5CoprvPhkcgTIzSDdm QuNn05tdGf5w+70kXSVGe37GDT5LTqfL1SvNwco4VlzupcuYJmCU X-Gm-Gg: ASbGnctVTHvrE2GE706rYrp+LlRPNE0GFG3fN7g45tytpv0cUvpFLHQy4a+hY6/AOsp cYejlhe8//T31lNDC0Qb3BMWDOn8eFigXJSN69fHOUCYLzyVJ8rxD7Y0REPAdZdd3hIFfrd0E7I hS+VSLWnfvG0gZ5LouRbWSOmZf0PHU1nzACgBe66IBlBe8GYLiKOeFFc4BZp6Mj2diRtBgqN66n tmSElMaZrFceAOghW1afa1YHTucr9o7yq1+Ti9QbGVLlpzGOSMD4RpGIobhqEBBk8QBkU71mon1 eH6QjAB4SnrXT/JAUZXJRiyVPaabJGP9YR+wWTFZGZfBfPo5TtSed7fBxdbgBA14JfkiomSrKPY rLPkGSSDOScL5beOm X-Google-Smtp-Source: AGHT+IFiS5JCKSf9kpXYqQvd4C20SO9WeFyiqD0ODJRSs4FwlxUKUEaIQb+jwmPk1FCG3Lz3izvoTA== X-Received: by 2002:ac8:5804:0:b0:472:9ea:887b with SMTP id d75a77b69052e-472228b3738mr106238181cf.10.1740363766868; Sun, 23 Feb 2025 18:22:46 -0800 (PST) Received: from fauth-a2-smtp.messagingengine.com (fauth-a2-smtp.messagingengine.com. [103.168.172.201]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-4720b1fe010sm59942551cf.60.2025.02.23.18.22.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 23 Feb 2025 18:22:46 -0800 (PST) Received: from phl-compute-08.internal (phl-compute-08.phl.internal [10.202.2.48]) by mailfauth.phl.internal (Postfix) with ESMTP id F0E6C1200043; Sun, 23 Feb 2025 21:22:45 -0500 (EST) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-08.internal (MEProxy); Sun, 23 Feb 2025 21:22:45 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgdejjeehgecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvfevufffkffojghfggfgsedtkeertdertddt necuhfhrohhmpeeuohhquhhnucfhvghnghcuoegsohhquhhnrdhfvghnghesghhmrghilh drtghomheqnecuggftrfgrthhtvghrnhepgeeljeeitdehvdehgefgjeevfeejjeekgfev ffeiueejhfeuiefggeeuheeggefgnecuvehluhhsthgvrhfuihiivgepieenucfrrghrrg hmpehmrghilhhfrhhomhepsghoqhhunhdomhgvshhmthhprghuthhhphgvrhhsohhnrghl ihhthidqieelvdeghedtieegqddujeejkeehheehvddqsghoqhhunhdrfhgvnhhgpeepgh hmrghilhdrtghomhesfhhigihmvgdrnhgrmhgvpdhnsggprhgtphhtthhopedukedpmhho uggvpehsmhhtphhouhhtpdhrtghpthhtoheprhgtuhesvhhgvghrrdhkvghrnhgvlhdroh hrghdprhgtphhtthhopehjihgrnhhgshhhrghnlhgrihesghhmrghilhdrtghomhdprhgt phhtthhopehprghulhhmtghksehkvghrnhgvlhdrohhrghdprhgtphhtthhopehjohhshh esjhhoshhhthhrihhplhgvthhtrdhorhhgpdhrtghpthhtoheprhhoshhtvgguthesghho ohgumhhishdrohhrghdprhgtphhtthhopehmrghthhhivghurdguvghsnhhohigvrhhsse gvfhhfihgtihhoshdrtghomhdprhgtphhtthhopehfrhgvuggvrhhitgeskhgvrhhnvghl rdhorhhgpdhrtghpthhtohepnhgvvghrrghjrdhuphgrughhhigrhieskhgvrhhnvghlrd horhhgpdhrtghpthhtohepjhhovghlsehjohgvlhhfvghrnhgrnhguvghsrdhorhhg X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 23 Feb 2025 21:22:45 -0500 (EST) From: Boqun Feng To: rcu@vger.kernel.org Cc: Lai Jiangshan , "Paul E. McKenney" , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Uladzislau Rezki , Zqiang , Davidlohr Bueso , Shuah Khan , linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, bpf@vger.kernel.org, Alexei Starovoitov Subject: [PATCH rcu 20/20] srcu: Make SRCU-fast also be NMI-safe Date: Sun, 23 Feb 2025 18:22:14 -0800 Message-Id: <20250224022214.12037-21-boqun.feng@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250224022214.12037-1-boqun.feng@gmail.com> References: <20250224022214.12037-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: "Paul E. McKenney" BPF uses rcu_read_lock_trace() in NMI context, so srcu_read_lock_fast() must be NMI-safe if it is to have any chance of addressing RCU Tasks Trace use cases. This commit therefore causes srcu_read_lock_fast() and srcu_read_unlock_fast() to use atomic_long_inc() instead of this_cpu_inc() on architectures that support NMIs but do not have NMI-safe implementations of this_cpu_inc(). Note that both x86 and arm64 have NMI-safe implementations of this_cpu_inc(), and thus do not pay the performance penalty inherent in atomic_inc_long(). It is tempting to use this trick to fold srcu_read_lock_nmisafe() into srcu_read_lock(), but this would need careful thought, review, and performance analysis. Though those smp_mb() calls might well make performance a non-issue. Reported-by: Alexei Starovoitov Signed-off-by: Paul E. McKenney Signed-off-by: Boqun Feng --- include/linux/srcutree.h | 34 ++++++++++++++++++++++++---------- 1 file changed, 24 insertions(+), 10 deletions(-) diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index bdc467efce3a..8bed7e6cc4c1 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -231,17 +231,24 @@ static inline struct srcu_ctr __percpu *__srcu_ctr_to_ptr(struct srcu_struct *ss * srcu_struct. Returns a pointer that must be passed to the matching * srcu_read_unlock_fast(). * - * Note that this_cpu_inc() is an RCU read-side critical section either - * because it disables interrupts, because it is a single instruction, - * or because it is a read-modify-write atomic operation, depending on - * the whims of the architecture. + * Note that both this_cpu_inc() and atomic_long_inc() are RCU read-side + * critical sections either because they disables interrupts, because they + * are a single instruction, or because they are a read-modify-write atomic + * operation, depending on the whims of the architecture. + * + * This means that __srcu_read_lock_fast() is not all that fast + * on architectures that support NMIs but do not supply NMI-safe + * implementations of this_cpu_inc(). */ static inline struct srcu_ctr __percpu *__srcu_read_lock_fast(struct srcu_struct *ssp) { struct srcu_ctr __percpu *scp = READ_ONCE(ssp->srcu_ctrp); RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_lock_fast()."); - this_cpu_inc(scp->srcu_locks.counter); /* Y */ + if (!IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE)) + this_cpu_inc(scp->srcu_locks.counter); /* Y */ + else + atomic_long_inc(raw_cpu_ptr(&scp->srcu_locks)); /* Z */ barrier(); /* Avoid leaking the critical section. */ return scp; } @@ -252,15 +259,22 @@ static inline struct srcu_ctr __percpu *__srcu_read_lock_fast(struct srcu_struct * different CPU than that which was incremented by the corresponding * srcu_read_lock_fast(), but it must be within the same task. * - * Note that this_cpu_inc() is an RCU read-side critical section either - * because it disables interrupts, because it is a single instruction, - * or because it is a read-modify-write atomic operation, depending on - * the whims of the architecture. + * Note that both this_cpu_inc() and atomic_long_inc() are RCU read-side + * critical sections either because they disables interrupts, because they + * are a single instruction, or because they are a read-modify-write atomic + * operation, depending on the whims of the architecture. + * + * This means that __srcu_read_unlock_fast() is not all that fast + * on architectures that support NMIs but do not supply NMI-safe + * implementations of this_cpu_inc(). */ static inline void __srcu_read_unlock_fast(struct srcu_struct *ssp, struct srcu_ctr __percpu *scp) { barrier(); /* Avoid leaking the critical section. */ - this_cpu_inc(scp->srcu_unlocks.counter); /* Z */ + if (!IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE)) + this_cpu_inc(scp->srcu_unlocks.counter); /* Z */ + else + atomic_long_inc(raw_cpu_ptr(&scp->srcu_unlocks)); /* Z */ RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_unlock_fast()."); }