From patchwork Tue Feb 11 22:38:07 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 13970786 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2080.outbound.protection.outlook.com [40.107.93.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C02DA27109F; Tue, 11 Feb 2025 22:38:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.93.80 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739313517; cv=fail; b=TZ9Fph+LrZajWzL9VsuzRUd65ql31HiyH2qMkPe5xJGnUqW3mL483YFuZB6lt0xRg491QUK1m55qelgB9XV6WjR7nd6V45SGsUlWK3/UjXIceilxzK3yqAf6EwcIpRtZmzOT2xRUu/QykaT0CFbD6tz0QwryxwWTuFvbGf0Ito8= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739313517; c=relaxed/simple; bh=LBkdCqN0Id3nK0UxZoHndqTuxaoNU5PnfNkev/wh85Y=; h=From:To:Cc:Subject:Date:Message-ID:Content-Type:MIME-Version; b=MzSzyDZH9Kz/LTRuymfhg+5HtZF86E68IawbqP4CVR0iXcRTyK2mi4n6Zy4enKPV8lFyTTyPf7SHOMRljA3nA80ZIy6ifXEEgG4L5kbSgBVHOchoLP3xgR2N3xDjvoTziCPERzxN8SYQVjeucMILuCFYLE+Cs3sMjnmoj0lV6uA= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=OJkIR4vN; arc=fail smtp.client-ip=40.107.93.80 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="OJkIR4vN" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Z+gFwTzeN+RLthpbdQ6ZLNsznDr18tvtjPZ2sQT+R6PKNcFeN07g/aP4TXytU3yaqtx6q5ueK1vUMlg6aZ07DddF3ZCAZVUn+pw7dawooe7l+mqM07xrq6TVwJR/SFkOzZbxYd3mC4IQT8b7kM8kier4A89l2z6mgQktTGiUSZbWTS6xHyB/l+WwSDkizdugU1HMmynNv1VuPbETPfwmSD/rvS2HbopWm1rTGrUTQf/ycR/kPwoiIvzfBdKgacGOZIKdMEyhaCWerwaLz9pfx3rFcaa6eWiXGtbz8JYbXiOVHQGWuIycoqPfXw8Kox/Kv04oLiDGiBNhSnk0gYoPlA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=o8vSGC7S2NY/OTpU5vSgsMN8JTXOr3JRndjFpZK5UwI=; b=l4tswPiDquOcpaEvSb6oVQVmKo1PHpOdpLtZU8KXx1GIQhpHhiss7gTUqnAJDkYLQPszjavo0PL+SNzBCh3NarPo0eOsp208PAmjcM3/GCMWP8ypHDcK1WprHPeC3qi6MYq3Ehyozs7sNjtvUP1J1PpwJ9qONKryYMo+il7UEU2j/d9JYHMqXyO29ZAwIy+GaxwPHqC4f7MSiASPYQjnsbCI3GVoFhhH1+36UORADJNRGwysTrVEUMYXLM5Ie/hBi7G1UuBMJhwg3U9IXSn2lsQhp4qeVYC7+afJNmcatpzlO83SYvKFdBrOpmXngTVOPr9TDYrzL9Nn0wVjvoVcSg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=o8vSGC7S2NY/OTpU5vSgsMN8JTXOr3JRndjFpZK5UwI=; b=OJkIR4vNAVVqsPUSUrw4MwlF7jBOtsNjs2reS7pzcEg6+aX1WY/iSvOb3cLMga8wxZG6PxO9KMKLF9HZ6NGA1lPcvqAUCSbxbixwAkFVIbgST7h+oZOZazklDyaGobsWVc/KwL5mmSVdfNh2HWbLTefqJ9LikLQl4MA9xsQYh1YdYfq0Jl2bWbLraoZNoRzqGLKuOetZ8G+qPyygADGPmbu/COFBKRJ2hAw81V1Uj0fHCB+jsP3Za0gZVBGOFfI2VpQcjrk5lId0EOMjxVsLNcv1amT5hMwcN5tpHeV0VAunVwtYF+kXN9HiId+q/D3zLpxKhbwtdZVccXjIsJqE8Q== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from SN7PR12MB8059.namprd12.prod.outlook.com (2603:10b6:806:32b::7) by SA0PR12MB7001.namprd12.prod.outlook.com (2603:10b6:806:2c0::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8422.18; Tue, 11 Feb 2025 22:38:32 +0000 Received: from SN7PR12MB8059.namprd12.prod.outlook.com ([fe80::4ee2:654e:1fe8:4b91]) by SN7PR12MB8059.namprd12.prod.outlook.com ([fe80::4ee2:654e:1fe8:4b91%5]) with mapi id 15.20.8422.015; Tue, 11 Feb 2025 22:38:32 +0000 From: Joel Fernandes To: linux-kernel@vger.kernel.org, "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Uladzislau Rezki , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang Cc: "Joel Fernandes" , rcu@vger.kernel.org Subject: [PATCH v2] rcu: Merge rcu_seq_done_exact() logic into rcu_seq_done() Date: Tue, 11 Feb 2025 17:38:07 -0500 Message-ID: <20250211223807.1834978-1-joelagnelf@nvidia.com> X-Mailer: git-send-email 2.43.0 X-ClientProxiedBy: BY1P220CA0003.NAMP220.PROD.OUTLOOK.COM (2603:10b6:a03:59d::10) To SN7PR12MB8059.namprd12.prod.outlook.com (2603:10b6:806:32b::7) Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN7PR12MB8059:EE_|SA0PR12MB7001:EE_ X-MS-Office365-Filtering-Correlation-Id: 95d877d9-7d6d-494a-fb38-08dd4aecd049 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|366016|7416014|376014|921020; X-Microsoft-Antispam-Message-Info: N3nPupFNMu/kf8DUsYGdZkwnYp4vhN2Wmws2sTCTM16ITYQ1KNtPjp9oSMpLKzA2Wa/YGhI8WNpK6kYO4+Be1/ud+ZcfLOmkF2nX4LJ7/QHXhto8YzVByQMhj2HZfwW9V1nWROMLRlVINdsEtM3MNQeQdKVawlgVOJ2DiOmC058rTYLmvsv5AI5UbLAyURL8nU3ESYqjhGDvVFyxCPzhNcN8maw+27d2fsfiZnrTWYgG7omBuCfJzG7nuCdK9An2yA2vsFyrerwOKdSN+vcERF4I3H5UZkXpLCBPl2jjZoZJsNGQ/ras+/LDYqfWxN6kf3zDibN/gge9xDY+gN63kQ9fXW5OcoM9+tPmFIR1EuUob5I83sFgMrXE7453Qyscw0TjyaL0aiF9OntafEABOGb3NZk0S3yJX7450Ro6sbOUTru6s/O6NW5TsAr7uw72ZreZ6bDw/dXYUB2aIa196QrZ+I91XaJz7cU5oYtGAnhePPh9XL4YI4ZQfv9IEQTSgPjEYS9mlfrn1Xsn+hrkdI+ncQAv0bhhxQpPselJzovlOELGOOeYRng+rju75dDwBU02YIlUkP8n91azbeq2s1MzGzTVf8+V/EQIx7MZGcNpwr4375O0nf+2m0urvd+VVU1obDqq8SEUZkNb8aJbz7vFcSTj19buxtgsWGHxbUtlXBrrVq63hxGSpGezzoklorbtKtuK8LHSbmCJeS0pJWDQR5xcox+9+SiCwiG0gWCxe39Dmf7zbiEcqd9ru25a/pnbao7BxNczDJZLzPF9d3lWV9upcbbSnWNA3yB1WlLwsQH3QGv2UcLrIauKNbF8iC6RbDYnKBwqut6/4luaq1sD5aTB92Is7HbiOdbqsJfJoh7b9qt3HUfEevhmGFvX3+K01TLB5LGQLzP5Fh+vCrQmziRwzLO/JW437TccNADjKJp2jfucGGhDD8ObXIbux620SzRDSIXWWE8PH9+PXUJ6Q49BnoBOrU9fCqyGBTEyG7DoCqbeHyyTi0ICTA7A7jOhtMmfhVJQsx/2JAAj6igkgMZW2ijWoengQRODUZ04VX5C91DBYfFbkbym4tOm8otVckVtXeJuEg3nPW5/6xZy+I8i52yWLgSAkEO1HTWTOyWlVKkLyrL2HsJ7u/NcirGsdd9B74znR1EKI1oafU5TPuYzQ49h71yi+EYfDXvsSs4IBC56iRKlJbZXZcr/PabRhUXVld2/DmMEWtaJ/2bj4LQOJfmnHQ6lgTHSbZx2W3nXW1Bsq9GQBl1bHS/cfqbIcSJIjdSt50v7jKTn93mhdDrYk7zVs8Z+/XQnuP5+eHndK3C8pzfDmusTupmzmZ2z1FvGuSyTOz/0XSHeLLw8UMh2IQ8wc1OtyXOHSw8evoHn7fDPN5VVf2+b1E3GShSDvZ3cYCAK+eDyQWhmN8XiAeZfBNxkquBLTw4gzPA= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SN7PR12MB8059.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(366016)(7416014)(376014)(921020);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: ApkVWVJSOAYUCLSMi7T25BB3CfY5OetMxKu7cc7l6Yv0NoNX9EX4XpYY4fr7e6ae07WVxTneUUJmKZrj/MH/o74VdJB3uTFp5Q1e49ki7LASvNqD6v8wTPswIS05iDINq47WxuigI5lrGnsSpKt9C4ZHtHTgi7PX5D5OpIMEJ6TFJgSIThwJ6RH3Bcb5uhsNGiODSNncPiXeGGkz85w1bZVyEK2VXgSai0qTCL2lAJS/tW8xLTTg7rwW0LvcRwO1uU93/DZTjzuE0w+Lm/QkJdMNOke3AR0eA5ebHFWzmMIo5nk44NPlA8+0KumgpSYJH74DRdIlbI2DavHYm+YyLnhV/zC1pSEaukjCtreOXTNHU/Y4BXn0eOF4WccKVPO+HFOMeexFqh14nBClp1CrZ1O5MljWY9tlmt+mZ0AWkChVSLtL+in2tCP0s/gWWkdtsirPTavF5d1NG6x57khfq4iVsx0OiFnrifo6bQZ7GBzUutdASBeJ8/1hWZlH91hQSDffPYrEnOS9GJRXb/sgH82rID2VYI+UAiNWcdChQ/BgzWLVQyap7ldHM0ZlTNX+eOnNewQrFO9+vuai9F/Nc7xNmXHPcdHcflWGhzXFtjP3aQF56/15zX/NakId4YtWvNj1ytDFE7z832ugW2PvafmEnRUYOj32e15oMq7GTsU956Yo+FY5XZZECKFvp+P4C7mxh5FIQ7MJxQa6/CBykVVtSVV8vIIhzBpIQrG0MMN25EHua30fR/nsKMR26xPrPnuFx+cFpBE2X5ZB1VE/voNys1cpBPlM1WS/AuxtzSAiWyVnME5roaQR5/ZNs/xC29/5/y2ZZZ2hPbkc059miwsRZOQLbwTCeWFS/yhKpDLE9/q66pUCHQrYIncqWl8N2R/Fk4sQPaMUbtAyzYJO0DrD46BN9xCAkBeeR3dxIxK6bXdekWHtgtpV7bb1fWtwCs/d5U8WYLuhtOkXsmzT6oOlFY1l6rVKRr1xFov7Z/JIwh9Yzsa9d+LtGGczuTxvdJZff2m3VxL2GkmRNOE+7NKj6lVb+ZcYyJdbzEKKoxvyg80jSCgFIjhn5Vn1psoNLMSg3+xh8Z6DVAg3+XcuP1NdUUKNtBmaIGGNNOdNIZ23WycBge9axUDtIU7EEUqJhFY3QXMOv2QD48WUwmdyW9lw9z1NudWTHSPcH+ZOxNkh18AmeHWzLgmIm9dN/zGg2aQSJA3z9erXjL+cx322ldfL5PWUUjyYDDbCYKUXqa/RfmAKcyq8bwiO5A3ICmw10B+QXA7qbdKsw8knlbmFqy/pZjQMRGWb2mP6VsYduFQwbq2JoOOTJkBZJ0l47+6bHge0OL83NeB3Mg8Sv1Frn1H4PRTDEmf0WXTefuOV1AJ36opQl34CrP9Y1NhQmTH8tG/ZclPr0QfLtzs5if/kT/apxXwzPo15PZ8zj6uzShOXEQRcHFhnRcFR3aWXq/3innLp/OdQSgIJ5fCz4FbpkwTa26BIMw2volj7igBI6Fjv7XDA/AJXBwcISDAW6mH6FPMcLcFcaIwaa4cu3Yf5jN8DlypzjtKVp3W7AsZqp+oPG2C+niDNE3J+UaXz3c6g X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 95d877d9-7d6d-494a-fb38-08dd4aecd049 X-MS-Exchange-CrossTenant-AuthSource: SN7PR12MB8059.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Feb 2025 22:38:32.5258 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Toow7FaXM0wDZCWHzycn3/UxyzGqFL1EtWoFVIn1tp4Sg3qjwyuDaO74xoS1+Io8LEeNgbehnvTHeH8adefXaA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA0PR12MB7001 From: "Joel Fernandes" The rcu_seq_done() API has a large "false-negative" windows of size ULONG_MAX/2, where after wrap around, it is possible that it will think that a GP has not completed if a wrap around happens and the delta is large. One place this might cause a possible problem is SRCU: poll_state_synchronize_srcu() uses rcu_seq_done() unlike poll_state_synchronize_rcu() which uses rcu_seq_done_exact(). The rcu_seq_done_exact() makes more sense for polling API, as there is a higher chance that there is a significant delay between the get_state..() and poll_state..() calls. Another place where this seems scary is if the condition for the wakeup was false causing missed wakeups, example in tree-nocb: swait_event_interruptible_exclusive( rnp->nocb_gp_wq[rcu_seq_ctr(wait_gp_seq) & 0x1], rcu_seq_done(&rnp->gp_seq, wait_gp_seq) || !READ_ONCE(my_rdp->nocb_gp_sleep)); The shorter false-negative window of rcu_seq_done_exact() would improve robustness as rcu_seq_done_exact() makes the window of false-negativity by only ~2-3 GPs versus ULONG_MAX/2. It also results in a negative code delta and could potentially avoid issues in the future where rcu_seq_done() was reporting false-negatives for too long. One downside of this change is the slightly higher computation, but it is trivial computation and I think is worth it. rcutorture runs of all scenarios for 15 minutes passed. Code inspection was done thoroughly for all users to convince the change would work. Further inspection reveals it is more robust so it is more than a cleanup. Signed-off-by: Joel Fernandes --- v1->v2: Updated commit message with more analysis and points. kernel/rcu/rcu.h | 13 ++----------- kernel/rcu/tree.c | 6 +++--- 2 files changed, 5 insertions(+), 14 deletions(-) diff --git a/kernel/rcu/rcu.h b/kernel/rcu/rcu.h index eed2951a4962..c2ca196907cb 100644 --- a/kernel/rcu/rcu.h +++ b/kernel/rcu/rcu.h @@ -146,19 +146,10 @@ static inline bool rcu_seq_started(unsigned long *sp, unsigned long s) /* * Given a snapshot from rcu_seq_snap(), determine whether or not a - * full update-side operation has occurred. + * full update-side operation has occurred while also handling + * wraparounds that exceed the (ULONG_MAX / 2) safety-factor/guard-band. */ static inline bool rcu_seq_done(unsigned long *sp, unsigned long s) -{ - return ULONG_CMP_GE(READ_ONCE(*sp), s); -} - -/* - * Given a snapshot from rcu_seq_snap(), determine whether or not a - * full update-side operation has occurred, but do not allow the - * (ULONG_MAX / 2) safety-factor/guard-band. - */ -static inline bool rcu_seq_done_exact(unsigned long *sp, unsigned long s) { unsigned long cur_s = READ_ONCE(*sp); diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index b77ccc55557b..835600cec9ba 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -4300,7 +4300,7 @@ EXPORT_SYMBOL_GPL(start_poll_synchronize_rcu_full); bool poll_state_synchronize_rcu(unsigned long oldstate) { if (oldstate == RCU_GET_STATE_COMPLETED || - rcu_seq_done_exact(&rcu_state.gp_seq_polled, oldstate)) { + rcu_seq_done(&rcu_state.gp_seq_polled, oldstate)) { smp_mb(); /* Ensure GP ends before subsequent accesses. */ return true; } @@ -4347,9 +4347,9 @@ bool poll_state_synchronize_rcu_full(struct rcu_gp_oldstate *rgosp) smp_mb(); // Order against root rcu_node structure grace-period cleanup. if (rgosp->rgos_norm == RCU_GET_STATE_COMPLETED || - rcu_seq_done_exact(&rnp->gp_seq, rgosp->rgos_norm) || + rcu_seq_done(&rnp->gp_seq, rgosp->rgos_norm) || rgosp->rgos_exp == RCU_GET_STATE_COMPLETED || - rcu_seq_done_exact(&rcu_state.expedited_sequence, rgosp->rgos_exp)) { + rcu_seq_done(&rcu_state.expedited_sequence, rgosp->rgos_exp)) { smp_mb(); /* Ensure GP ends before subsequent accesses. */ return true; }