From patchwork Sun Apr 7 09:05:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lai Jiangshan X-Patchwork-Id: 13620086 Received: from mail-pf1-f171.google.com (mail-pf1-f171.google.com [209.85.210.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 306CE156E4; Sun, 7 Apr 2024 09:03:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712480614; cv=none; b=mGWONMP1qULugokf+ff1MaPJMHhZXYL0DJrpEinHkr0+vmtkJz0cTH9/BZwfrowYcDroHvCnmC7yhmeoKfrT/BjwmM9BLJX2fG4IHUA1coYJE+px+R3xRZ4JxksiGg7pWEeZ7N1Tf2NQ18SCue6CHHmO8jar3XUTWJMxi4IVUdc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712480614; c=relaxed/simple; bh=3Gos612aoLPrsPFZpY6vzxas7L2U1yTGHX46XVU7dYQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=FpQVbCR69AgR9n1tLcDp1wac568T8ARkHdU42G9nsPFg1Lem6oBCVF3LUl7AMeCyzXhOSOYIxqR1E3hEDqq4e8sn3TrLOgcTuYRF/KYhLj/6Zw9bbvJMcMHfHRuk3ACjOxt0UxumLfWQ63bfJp/Jks3x84iJAiAg5HrGcjWxx/8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=UpGLFiR6; arc=none smtp.client-ip=209.85.210.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="UpGLFiR6" Received: by mail-pf1-f171.google.com with SMTP id d2e1a72fcca58-6ed054f282aso1516534b3a.0; Sun, 07 Apr 2024 02:03:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712480612; x=1713085412; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=c1Nt907o8edMLRxAYjTwIu4B/dw5DVN79x2+si+sMmU=; b=UpGLFiR6XiGSy72KoXCIfrXlV1ckg6ap9YDZIxWsydi86nwNYyH2/EmRcSLuUdx+ep 3XFOTNkH17QuISCjf9RamU1+/oxA5p090FErc6l1oFExNGaRqU2cMfF0xn9Yl+Bucjbt jIqtlhvwcISVYqUXo7lrb0bKO8LyAvz/CXVO8zMfCxMoQpu9eMtufJ2nuv24khrAt8rU xzM48W1eC7dZ4B1JpOUyMDZBUEFpwVp0pJ7HIJtOqFAr+nu2BYoG7P0gcpj1CQsyFNOO 6K1oBUZgJl6NwX5P8sARWzH7+fV7h6SkICRzLa4zDqwhdK3vc8oqEKYs07rMcf327lRE pJfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712480612; x=1713085412; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=c1Nt907o8edMLRxAYjTwIu4B/dw5DVN79x2+si+sMmU=; b=CR3mWv1QX0o6/G/opSwrwCC/P97dVQrym9B/riTPQKDKYhEgez80vmTrm7aDPyDwFG YsowHswI0v7ssZCKlodrbmxiXj+VndrX6P9oPz/YjH/8d6Qp+pt+8m5kvnPrH5JFKjnN b4d5DbpFBLaLl7d6ZcSo064SpitVIkG8n0k9b5YGPZmG5Wu+SFqBgGs5hY44FO71+sUP 47bv5q09io+6TbfbeLMyfsFwLlZHW1ND6xX0YCZVgy7KNMYxLEE9E0kWr9M8Vx1Y/jtH xNo8lV2Ut1j/b5iYBNzhkz2+u7yQlFTIsgp0PvlkghNRtMIw1fsxIw0Oqqlyij5xVf5z sgxw== X-Gm-Message-State: AOJu0YylI3EhTgMg33+1udNEiI44A8EBoepK5FaSJnCT56WmVCzWLyt3 RkTavsUjTPTEsKTM2cN8eVJvK0EEnuozvYS4W4uTWz81ZWy8mmq5RcsdeTnn X-Google-Smtp-Source: AGHT+IEZWBC54z6Qvf92SZcg4NE/Us7RCUpxhKtDVKx3i9t4ub/JtDuGslUTjHfh2I5x8mKskbAYIw== X-Received: by 2002:a17:90a:df85:b0:2a1:fb5e:6028 with SMTP id p5-20020a17090adf8500b002a1fb5e6028mr4428255pjv.20.1712480612016; Sun, 07 Apr 2024 02:03:32 -0700 (PDT) Received: from localhost ([47.89.225.180]) by smtp.gmail.com with ESMTPSA id i13-20020a63220d000000b005cd8044c6fesm4267344pgi.23.2024.04.07.02.03.31 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 07 Apr 2024 02:03:31 -0700 (PDT) From: Lai Jiangshan To: linux-kernel@vger.kernel.org Cc: rcu@vger.kernel.org, x86@kernel.org, Lai Jiangshan , "Paul E. McKenney" , Peter Zijlstra , Frederic Weisbecker , Andrew Morton , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng Subject: [PATCH V2 01/11] lib: Use rcu_preempt_depth() to replace current->rcu_read_lock_nesting Date: Sun, 7 Apr 2024 17:05:48 +0800 Message-Id: <20240407090558.3395-2-jiangshanlai@gmail.com> X-Mailer: git-send-email 2.19.1.6.gb485710b In-Reply-To: <20240407090558.3395-1-jiangshanlai@gmail.com> References: <20240407090558.3395-1-jiangshanlai@gmail.com> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Lai Jiangshan Use the general wrapper rcu_preempt_depth() instead of the open code. Prepare for enabling per-cpu rcu_preempt_count, in which case current->rcu_read_lock_nesting might not be synced with real depth. Cc: "Paul E. McKenney" Cc: Peter Zijlstra Cc: Frederic Weisbecker Signed-off-by: Lai Jiangshan --- lib/locking-selftest.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/lib/locking-selftest.c b/lib/locking-selftest.c index 6f6a5fc85b42..9bb41fb18088 100644 --- a/lib/locking-selftest.c +++ b/lib/locking-selftest.c @@ -1434,7 +1434,7 @@ static void dotest(void (*testcase_fn)(void), int expected, int lockclass_mask) #ifdef CONFIG_SMP int saved_mgd_count = current->migration_disabled; #endif - int saved_rcu_count = current->rcu_read_lock_nesting; + int saved_rcu_count = rcu_preempt_depth(); #endif WARN_ON(irqs_disabled()); @@ -1476,9 +1476,9 @@ static void dotest(void (*testcase_fn)(void), int expected, int lockclass_mask) migrate_enable(); #endif - while (current->rcu_read_lock_nesting > saved_rcu_count) + while (rcu_preempt_depth() > saved_rcu_count) rcu_read_unlock(); - WARN_ON_ONCE(current->rcu_read_lock_nesting < saved_rcu_count); + WARN_ON_ONCE(rcu_preempt_depth() < saved_rcu_count); #endif #ifdef CONFIG_TRACE_IRQFLAGS From patchwork Sun Apr 7 09:05:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lai Jiangshan X-Patchwork-Id: 13620087 Received: from mail-pg1-f180.google.com (mail-pg1-f180.google.com [209.85.215.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 838B21DA24; Sun, 7 Apr 2024 09:03:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712480618; cv=none; b=psyqen/guuzYIDUGmDR1VLaOW3Z0yGGYRO2NDi+V7APjWP8enmvTph99xIckT5wdWT9aCUXv/f8AbXobvlU0sEF4flDdFkdlzC41esQymuyEGfeeVPcSWLHAVFGlT3Daror543EoUBQzD1ztzwu88EW+evbnBy+m6sLJFuB2ZmY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712480618; c=relaxed/simple; bh=83Npl5u33AsT7Ew5r4IPt7zF9FD8Clznb3uYTsdUbDE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=gji+WE/G0f+HxKEs3pw9MCO+WPOPcI/w5iRckdcpR6T0ZQTu+J28EHhqq7l0G+aofHV0J5dhlOuc5FOc7PluWFQiaH3mGZOg2SrfMf9I1EfqdJf/Fq/0UJ4A7DoWZayv9XuW/ceWIbcdgiD3MZXSZ0zEcTWWZCltUviso+wNXSA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=lnQcOZ9b; arc=none smtp.client-ip=209.85.215.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="lnQcOZ9b" Received: by mail-pg1-f180.google.com with SMTP id 41be03b00d2f7-5dbcfa0eb5dso2580691a12.3; Sun, 07 Apr 2024 02:03:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712480616; x=1713085416; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0RqVJfN/RcgKGdk3KqHQb6BibGIpzFn9cI1YBM8NAdA=; b=lnQcOZ9b0PQ1lkjNQQ2lFM31IS053zHwNA3ncDX6TdBQNIueFcTZoPjV/hqG14LEsl yEaLR9wW684n5nrXPnqM+vyMuTubVEfYMocjCVdombzvTvT66a3pIb8Op7jsBUZjxadD SGmBDHWhXRHFoPcPvxwpoCv+ZcuNMQUIq9XMrQflPPFZdKT+ZPoyUyt61hYm2Ky/LyNY NTgfRJlHJZ/xcIzEZFkOsceWMiHJqqMjCCsGnrOb5W6HL+/SV0p45vzfXlmIoZHQJT9x YbTFe57WS9NIyZubx1zDoOhcW6eFiUvtUGtsm2Xam0MaVXp7vCByXf/zoBadeN6RZniw hBjg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712480616; x=1713085416; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0RqVJfN/RcgKGdk3KqHQb6BibGIpzFn9cI1YBM8NAdA=; b=pMP9/8UWoOg0zEVM7HzQWuxvjnPIrsx71z7nbOVHVO9HZpBzN6TnD1nE0qcd4tmc4c TA43Zp3fpX28cnYx642Z3cIiKyn0iWVQJ3+gbKtcIw/fSYQ76jn9knl/7/rd0UbAtLFW ZpIDc8N8Z0mOW1rUTYE1CAKwFHrqW9w2WNKulmVLlCuX6sgFf623/dQOmGaug0RQg4Id s8h+VkHKm56owgn3bsYGmjmR32vlqh2DbZsZ+b1HpmikAzQflMzZ3EH0TQaeM8qYenZ+ ceiedBWUj9U52cbsZ0TGGdAZXTECqxt5D5wyI52eG207MGOjgrzROTAjwwBduklb04On aZUA== X-Gm-Message-State: AOJu0YxbE86CvQ2+sl7pzYP0GJud2LXto0mj0oYH5nEETIotFdL/XrdN PrXZ29QOqmQFWKIJXm8VLUm+6WjgvGjRKl6xRlcKBT2vLgm1eZDZx6HJEtZt X-Google-Smtp-Source: AGHT+IHidET2V2zDq2JEY2AQ2oyH0YB/n+RTzr+56t5d/1xTL7S25sWEPxmaw9RVF38l1lrFhZc1BQ== X-Received: by 2002:a17:903:11d0:b0:1df:f681:3cd8 with SMTP id q16-20020a17090311d000b001dff6813cd8mr6169822plh.12.1712480616461; Sun, 07 Apr 2024 02:03:36 -0700 (PDT) Received: from localhost ([198.11.176.14]) by smtp.gmail.com with ESMTPSA id o22-20020a17090aac1600b002a26245adffsm4156102pjq.41.2024.04.07.02.03.35 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 07 Apr 2024 02:03:36 -0700 (PDT) From: Lai Jiangshan To: linux-kernel@vger.kernel.org Cc: rcu@vger.kernel.org, x86@kernel.org, Lai Jiangshan , "Paul E. McKenney" , Peter Zijlstra , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang Subject: [PATCH V2 02/11] rcu: Move rcu_preempt_depth_set() to rcupdate.h Date: Sun, 7 Apr 2024 17:05:49 +0800 Message-Id: <20240407090558.3395-3-jiangshanlai@gmail.com> X-Mailer: git-send-email 2.19.1.6.gb485710b In-Reply-To: <20240407090558.3395-1-jiangshanlai@gmail.com> References: <20240407090558.3395-1-jiangshanlai@gmail.com> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Lai Jiangshan Prepare for arch-specific-defined rcu_preempt_depth_set(). No functionality change intended, but it has to be defined as a macro as rcupdate.h is a very low level header included from areas that don't even know about the task struct "current". Cc: "Paul E. McKenney" Cc: Peter Zijlstra Cc: Frederic Weisbecker Signed-off-by: Lai Jiangshan --- include/linux/rcupdate.h | 1 + kernel/rcu/tree_plugin.h | 5 ----- 2 files changed, 1 insertion(+), 5 deletions(-) diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index 17d7ed5f3ae6..ebe9bfc700bb 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -78,6 +78,7 @@ void __rcu_read_unlock(void); * types of kernel builds, the rcu_read_lock() nesting depth is unknowable. */ #define rcu_preempt_depth() READ_ONCE(current->rcu_read_lock_nesting) +#define rcu_preempt_depth_set(val) WRITE_ONCE(current->rcu_read_lock_nesting, (val)) #else /* #ifdef CONFIG_PREEMPT_RCU */ diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h index 36a8b5dbf5b5..b1264096d03a 100644 --- a/kernel/rcu/tree_plugin.h +++ b/kernel/rcu/tree_plugin.h @@ -387,11 +387,6 @@ static int rcu_preempt_read_exit(void) return ret; } -static void rcu_preempt_depth_set(int val) -{ - WRITE_ONCE(current->rcu_read_lock_nesting, val); -} - /* * Preemptible RCU implementation for rcu_read_lock(). * Just increment ->rcu_read_lock_nesting, shared state will be updated From patchwork Sun Apr 7 09:05:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lai Jiangshan X-Patchwork-Id: 13620088 Received: from mail-pf1-f182.google.com (mail-pf1-f182.google.com [209.85.210.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9D333225D0; Sun, 7 Apr 2024 09:03:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712480623; cv=none; b=G9UQGj3p5Wyd/nZv4WmSKkc/dmVdO0LWOn6BtEs+IMZQziUeX04t6kU7IoaHTvAFvvlId9mY9a7xnJMRDOJJvMt34awSlcdMtqdy+pQNmFSpdgSMFpUngouvxiqWmkKhoRvUQdvhwGzn/6UwGTd0LlWMOrW+A7FiRY7stAL53Eo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712480623; c=relaxed/simple; bh=fbc/eh8HxR9mhBJCVcgaSscwMF8MFg41ZBJxKMGrRA0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=pzdR/+xpQ2mKeEH6xncu6GyAXENSI/irhtlerkaLnlyx6eBFcKGy7WAdMR88v+fSHyeMMj+Z21Z32FO6mlJta/QnNbDnjZtqM1TTgSNHuA0c6PC+xeEak+y5+YZ9B5nCAHzuXmsUEq/aust9QHxFx4GVT1oYKeDOZsSnucnwhmc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=jc79AA6G; arc=none smtp.client-ip=209.85.210.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="jc79AA6G" Received: by mail-pf1-f182.google.com with SMTP id d2e1a72fcca58-6ed2dc03df6so12903b3a.1; Sun, 07 Apr 2024 02:03:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712480620; x=1713085420; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RryskTliZ3IXjIMEW4QLTJUCIPdL2atrGVjHqL54oO4=; b=jc79AA6Gso7gENaAKp+SmB7dB9pe17z50Cj5CRYFoG/mKJUo4Ij8QG2d94CIN3mFyi c8pxqhlTbjRFmmHPOjdYq4YppZudK91fha8BQyvmcuGC6iLaxtjbPioMvMima8zfkksY kfpBoONJH5h9HAEHqKj98jq/WwR7eyFjf6RyYNJl6ppXXftTj4dbel7/Xlrl6hFOfa0C YRhG4UrPokICxNr3PHvYJz8KIxObn3LKWbVoaOY3qixCQfgs92QifA+2KQGggRW+dTm2 oRpg9Q6AN2tAgxNrX3VQEQVO5RkEGKfkECxgvslkaTxIYXSsP1AUElaz8UOyIaL70ZFm /Bcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712480620; x=1713085420; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RryskTliZ3IXjIMEW4QLTJUCIPdL2atrGVjHqL54oO4=; b=BzjIgO9XzKeLHMmfDGz6MQcRU/LDEV0XYMBdXunPRkTZ6/9hjTcFphIyANP4eLi4Ha qwKI3xJVQf90XoXinpUWNXqHFyHqofWpgK+KyDaehlzSLppmmeAXsmFKWedkSPfzMoXl PvlN8dDrCL7s2t7C3R0f7ErgFNaOiScuExw2SeYyxSr6G9nzRqDijOvtUwDH5DFvieYN 0u8MI83mgiS0eXPh2bra+zUaJo5GoCjn2v8VsEEwtGPuuZGLtPJm9oS1HPLDtG9nFJJn 3Qmc+SkqNU76YQrmKTVFRG2nFTdZVkZY0p8Jxq4wv9pP+71LysP9f6iUrAdjlTXOrtdV NZog== X-Gm-Message-State: AOJu0Yy0cVS8+b3S6Mz78Et9LVrlvqNLEEw4E+i7XIiL3VEoBjPweElw 1ba3wlIVjX2d/GhOHfafgh+tRgjR+7U1/l3NfmhmFgx4lrm09J+DYAziIlmc X-Google-Smtp-Source: AGHT+IGbg1psQDNl4aGTCjaoHOZ/zbY+xXBJoXh6D1RemuxFUcbUJQU2ZTzC3jwjhyKQjLgtVQ8Z/g== X-Received: by 2002:a05:6a20:394f:b0:1a3:a637:9753 with SMTP id r15-20020a056a20394f00b001a3a6379753mr8330961pzg.12.1712480620477; Sun, 07 Apr 2024 02:03:40 -0700 (PDT) Received: from localhost ([198.11.176.14]) by smtp.gmail.com with ESMTPSA id p37-20020a056a0026e500b006e6813f2301sm2587566pfw.56.2024.04.07.02.03.39 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 07 Apr 2024 02:03:40 -0700 (PDT) From: Lai Jiangshan To: linux-kernel@vger.kernel.org Cc: rcu@vger.kernel.org, x86@kernel.org, Lai Jiangshan , "Paul E. McKenney" , Peter Zijlstra , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang Subject: [PATCH V2 03/11] rcu: Reorder tree_exp.h after tree_plugin.h Date: Sun, 7 Apr 2024 17:05:50 +0800 Message-Id: <20240407090558.3395-4-jiangshanlai@gmail.com> X-Mailer: git-send-email 2.19.1.6.gb485710b In-Reply-To: <20240407090558.3395-1-jiangshanlai@gmail.com> References: <20240407090558.3395-1-jiangshanlai@gmail.com> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Lai Jiangshan Enable tree_exp.h using some rcu preempt macros introduced in the next patch. The new macros touch core rcu-preempt fields and are better to be implemented in tree_plugin.h. Cc: "Paul E. McKenney" Cc: Peter Zijlstra Cc: Frederic Weisbecker Signed-off-by: Lai Jiangshan --- kernel/rcu/tree.c | 2 +- kernel/rcu/tree_plugin.h | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index d9642dd06c25..57d1ae26861f 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -5245,6 +5245,6 @@ void __init rcu_init(void) } #include "tree_stall.h" -#include "tree_exp.h" #include "tree_nocb.h" #include "tree_plugin.h" +#include "tree_exp.h" diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h index b1264096d03a..d899b4afc21c 100644 --- a/kernel/rcu/tree_plugin.h +++ b/kernel/rcu/tree_plugin.h @@ -103,6 +103,7 @@ static void __init rcu_bootup_announce_oddness(void) #ifdef CONFIG_PREEMPT_RCU static void rcu_report_exp_rnp(struct rcu_node *rnp, bool wake); +static bool sync_rcu_exp_done(struct rcu_node *rnp); static void rcu_read_unlock_special(struct task_struct *t); /* From patchwork Sun Apr 7 09:05:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lai Jiangshan X-Patchwork-Id: 13620089 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9B72D25622; Sun, 7 Apr 2024 09:03:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712480627; cv=none; b=TDnjuH02AUdFC5tKYlTYJYa0urBsEo2mXVcKPTo4uk+mO8GGMbKoZPQp9n/VZgNGJgakBCMZH1Ncxq1p9ad86RwkVXz/y9dbgFpshNlMbilBG0xNxMQOQzxQqtKu3CR7qdaVHjizt6CT+XyexnMjywCMkX1UHJMea44IZUEgKkw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712480627; c=relaxed/simple; bh=dyDjW0emqk3ra0mcQ2J2Pyvf/leGXIhR8J75jqjVAuw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=RYpxy59x7vnRh12VRko9ZRi3DtasRv1xzErjcvdyfkdTjfyqv4cMqS29FWokC4LmwErjdlskf1vz7WlKF6rxGWwIc67G/LrKoq6RIULSTIUchNjEb+sd9No/T+DQjmI9+ANZ5e2+hS0wPxzNPl6UdQtOH0B16pE6RM6ePnw1xcs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=NhGNgTCr; arc=none smtp.client-ip=209.85.214.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="NhGNgTCr" Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-1e2bbc2048eso28539385ad.3; Sun, 07 Apr 2024 02:03:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712480624; x=1713085424; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ZvoBd6HF4JbE2BHoe97A/COYGhUtV93EmRmWHbLWiDc=; b=NhGNgTCrnINYBIH9NlRJR8YRRDYyek45I4AfQ7DClp/s/RrBv18TJP9YfyVC0ClX4v gs+cXXzubLLRdR5dbZl+nbQbhyV//Ai9RgujZAoSSFnwZavCjfdP41KOYWrPkmY5zEkN pMsK4oA7AaNa4A+pNcGWhz0YXj1PcAIuaE7uYBlisATUPqtkBQI6JBVZQwdLx16/9vGG 4yTcezCaTy6LXX8QMnpyjJS/T14V02iof9KMGo5/p/N0zMWd5vjdsP6L56C5JS9iJ+PB QPzH2vrbEARyMmYnDOdI+9rgq69/Y5D6bm8ANZk3qHTi781XuyRbVcs8aJk/5QC9UtiZ I6Sg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712480624; x=1713085424; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ZvoBd6HF4JbE2BHoe97A/COYGhUtV93EmRmWHbLWiDc=; b=YYveQfB4+3zC6bz7iraBzIYJKF1VeCew5tVzaCZDOfHAP0RU8Ak0/lRGv0YVPfV8TA cod14T0kfJeh8dSWuIvvTQpVvZ0/XWlErQPISOFXSCu+w1kByeBQPr3lIqy8d/8bnZGa Ocwf2KgmI5sIYC4FkEDZMCXavOQxhTxTqKoZBgicWSZz+/eJZRvcMyePYeRk2eZxp5SO 0RAz0+nrpDziXqWa5voY+HFgD1Chl81rG/zjRWsmQCSR3VvTBRQeP0gKYHmQViJ1sop4 63Yx9p6R/PWxeetJvMY+RkQE2dXyrU1RsyTSFDzca4TN98PTzebhkCMu1bzoFluffQrV ypRA== X-Gm-Message-State: AOJu0YzVR3rn/EKNdyc6ICViBuMarVsDj94PoMQ0/oNtSqtWbAT0Ano7 e62kFlWqE5sXEdmoQJNngu7ckl25XTPIT8xyMeh0PufUO3inXtk3XHcxmwJH X-Google-Smtp-Source: AGHT+IHm+MSwmnlLu34S3eEMB6KiFa2TY5nq3L1NGeIEDXeLzqfKCXuvVHVNTMQDGC2xidm4Is6oaw== X-Received: by 2002:a17:902:ac86:b0:1e2:aa07:37d7 with SMTP id h6-20020a170902ac8600b001e2aa0737d7mr5210462plr.22.1712480624566; Sun, 07 Apr 2024 02:03:44 -0700 (PDT) Received: from localhost ([198.11.176.14]) by smtp.gmail.com with ESMTPSA id jx3-20020a170903138300b001d9a91af8a4sm4565578plb.28.2024.04.07.02.03.43 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 07 Apr 2024 02:03:44 -0700 (PDT) From: Lai Jiangshan To: linux-kernel@vger.kernel.org Cc: rcu@vger.kernel.org, x86@kernel.org, Lai Jiangshan , "Paul E. McKenney" , Peter Zijlstra , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang Subject: [PATCH V2 04/11] rcu: Add macros set_rcu_preempt_special() and clear_rcu_preempt_special() Date: Sun, 7 Apr 2024 17:05:51 +0800 Message-Id: <20240407090558.3395-5-jiangshanlai@gmail.com> X-Mailer: git-send-email 2.19.1.6.gb485710b In-Reply-To: <20240407090558.3395-1-jiangshanlai@gmail.com> References: <20240407090558.3395-1-jiangshanlai@gmail.com> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Lai Jiangshan Add wrappers accessing to t->rcu_read_unlock_special.b so that the wrappers can be adapted for synching the value with the later-introduced per-cpu rcu_preempt_count. Cc: "Paul E. McKenney" Cc: Peter Zijlstra Cc: Frederic Weisbecker Signed-off-by: Lai Jiangshan --- kernel/rcu/tree_exp.h | 2 +- kernel/rcu/tree_plugin.h | 20 +++++++++++++------- 2 files changed, 14 insertions(+), 8 deletions(-) diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h index 6b83537480b1..4e91414552e6 100644 --- a/kernel/rcu/tree_exp.h +++ b/kernel/rcu/tree_exp.h @@ -725,7 +725,7 @@ static void rcu_exp_handler(void *unused) raw_spin_lock_irqsave_rcu_node(rnp, flags); if (rnp->expmask & rdp->grpmask) { WRITE_ONCE(rdp->cpu_no_qs.b.exp, true); - t->rcu_read_unlock_special.b.exp_hint = true; + set_rcu_preempt_special(exp_hint); } raw_spin_unlock_irqrestore_rcu_node(rnp, flags); return; diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h index d899b4afc21c..3dbd5609185e 100644 --- a/kernel/rcu/tree_plugin.h +++ b/kernel/rcu/tree_plugin.h @@ -106,6 +106,14 @@ static void rcu_report_exp_rnp(struct rcu_node *rnp, bool wake); static bool sync_rcu_exp_done(struct rcu_node *rnp); static void rcu_read_unlock_special(struct task_struct *t); +#define set_rcu_preempt_special(reason) do { \ + WRITE_ONCE(current->rcu_read_unlock_special.b.reason, true); \ + } while (0) + +#define clear_rcu_preempt_special(reason) do { \ + WRITE_ONCE(current->rcu_read_unlock_special.b.reason, false); \ + } while (0) + /* * Tell them what RCU they are running. */ @@ -293,7 +301,7 @@ static void rcu_qs(void) TPS("cpuqs")); __this_cpu_write(rcu_data.cpu_no_qs.b.norm, false); barrier(); /* Coordinate with rcu_flavor_sched_clock_irq(). */ - WRITE_ONCE(current->rcu_read_unlock_special.b.need_qs, false); + clear_rcu_preempt_special(need_qs); } } @@ -325,7 +333,7 @@ void rcu_note_context_switch(bool preempt) /* Possibly blocking in an RCU read-side critical section. */ rnp = rdp->mynode; raw_spin_lock_rcu_node(rnp); - t->rcu_read_unlock_special.b.blocked = true; + set_rcu_preempt_special(blocked); t->rcu_blocked_node = rnp; /* @@ -399,7 +407,7 @@ void __rcu_read_lock(void) if (IS_ENABLED(CONFIG_PROVE_LOCKING)) WARN_ON_ONCE(rcu_preempt_depth() > RCU_NEST_PMAX); if (IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD) && rcu_state.gp_kthread) - WRITE_ONCE(current->rcu_read_unlock_special.b.need_qs, true); + set_rcu_preempt_special(need_qs); barrier(); /* critical section after entry code. */ } EXPORT_SYMBOL_GPL(__rcu_read_lock); @@ -738,7 +746,7 @@ static void rcu_flavor_sched_clock_irq(int user) __this_cpu_read(rcu_data.cpu_no_qs.b.norm) && !t->rcu_read_unlock_special.b.need_qs && time_after(jiffies, rcu_state.gp_start + HZ)) - t->rcu_read_unlock_special.b.need_qs = true; + set_rcu_preempt_special(need_qs); } /* @@ -751,12 +759,10 @@ static void rcu_flavor_sched_clock_irq(int user) */ void exit_rcu(void) { - struct task_struct *t = current; - if (unlikely(!list_empty(¤t->rcu_node_entry))) { rcu_preempt_depth_set(1); barrier(); - WRITE_ONCE(t->rcu_read_unlock_special.b.blocked, true); + set_rcu_preempt_special(blocked); } else if (unlikely(rcu_preempt_depth())) { rcu_preempt_depth_set(1); } else { From patchwork Sun Apr 7 09:05:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lai Jiangshan X-Patchwork-Id: 13620090 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EFE7B2BAF2; Sun, 7 Apr 2024 09:03:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712480631; cv=none; b=hkd4U9UaJX/4+8qu8mg9+23MXcyRTtdresRXhh3CvI+BANdaJRlZbQkDBSW4uKmYr3gCFgYsula9AXHq5T+DTRGmMfqq0eUIHPtixfVsqGYS0kDkWlEo5P9iapZ0lsT6BnSUWaypoDBxjn6mZIqt0cAXWGXqJHEAwfkaIb1U/tQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712480631; c=relaxed/simple; bh=WQzpAah6Oo2bkMAnx/zWo1QQcwsq2Z/42lDo9dpY4/s=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=bSyX0UAy2o13e69fSSpHuY633NjLqE8MY1DoXVIYqi91zQTL5nVd9JvL6tOsunpQibSKorzvL0cJLi49DWCr8BJXeIVkOkuIf5ACqCrFyst806kQqY8xpsEZzRA+DaYix7ICxEb1+IyBk53sJtF1Ocf5y2wUtKsp1P+Azkt1za4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=FTEgtn7Z; arc=none smtp.client-ip=209.85.214.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="FTEgtn7Z" Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-1e2a7b5ef7bso28881515ad.1; Sun, 07 Apr 2024 02:03:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712480629; x=1713085429; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=OwgJg6qGDb4F/2vLVULuXMEXIMCZTw7ICcUrvera85s=; b=FTEgtn7Zt1EOobuHeT/vt/VT462oSr1qdMiG6mmuwkTV0znRAXAr2UO1MUpj+TwlqT cbIUclZdEXMAK0ADqGAKfbhSQiszUk77TEX1NEDu3j9axLO5sPhNKNNYmwDcNGhNv6QI rdwnOYI8Hd0FyFNQ23/DKsjF4H/esIJBGo3xB86d8iVlS/XG6HDu3Ilr4/vJzEsI0Z7P ANe0wAZXtnRGlTP9ompFOVAhUXVwwC0CS+Z6tsu4vkc+Nhxukobj1JrDYG/+hNRPMGcE 5BCRZm0HSFnrrgYnkStR9Cuwe5CVupupV1++UzjBvei7YElYy6t//SHfR6SgbuzS5GsI 01qQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712480629; x=1713085429; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=OwgJg6qGDb4F/2vLVULuXMEXIMCZTw7ICcUrvera85s=; b=bWKrJjUc3gG30LBjP5dm5eNsj0n1NBZHg7kp9hC7/7z+YCjRLHIBLGqzsjsjFkm+y8 PE/PEcbTTwiur42bcyWtqf6awjUPtfmt4u+m9EndYY+g7FLJFKmkui350yM7OdST2hg/ b2h53tP0RLkiSc3gndfncguBWPbdYlJheACARfbyAgS62+X5GrQXVlTeghMBzfX6LeAc ABcCoXl3LLFaHosRBxUxDLfa6w6F5niTtXsyUKqxzLcsqcWsQ9/be7obbeoagCoQH8RM KodB03cfKWEEiO5qPUJ5OYStuouUWcynTGXBbOHPhGL4ZPGtvQs/iX+GJrFj2yscuMXF bZWg== X-Gm-Message-State: AOJu0YzoFlfmaAaPgma6Stcnoeft4MwzE0RCYn0DUXfsehgZ5QyvL1Xf /zG7I0gd5h7yWAfXOAJluVPznmj0CKaeWXO6q9djoVnpq6+tcm2XfCyYPn6p X-Google-Smtp-Source: AGHT+IFNSd0I3oJLebO3gMhdolrLbFNc1uMWBKx4Hq0g7mWJqEwiqFZrjSqYatrRa2/RMSDzojoBog== X-Received: by 2002:a17:903:1112:b0:1e2:3d61:e266 with SMTP id n18-20020a170903111200b001e23d61e266mr8168295plh.49.1712480628868; Sun, 07 Apr 2024 02:03:48 -0700 (PDT) Received: from localhost ([47.89.225.180]) by smtp.gmail.com with ESMTPSA id i12-20020a17090332cc00b001ddc83fda95sm4508784plr.186.2024.04.07.02.03.47 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 07 Apr 2024 02:03:48 -0700 (PDT) From: Lai Jiangshan To: linux-kernel@vger.kernel.org Cc: rcu@vger.kernel.org, x86@kernel.org, Lai Jiangshan , "Paul E. McKenney" , Peter Zijlstra , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang Subject: [PATCH V2 05/11] rcu: Make rcu_read_unlock_special() global Date: Sun, 7 Apr 2024 17:05:52 +0800 Message-Id: <20240407090558.3395-6-jiangshanlai@gmail.com> X-Mailer: git-send-email 2.19.1.6.gb485710b In-Reply-To: <20240407090558.3395-1-jiangshanlai@gmail.com> References: <20240407090558.3395-1-jiangshanlai@gmail.com> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Lai Jiangshan Make it global so that it can be used in the future inlined rcu_read_unlock(). Make it exported so that the inlined rcu_read_unlock() can be used in modules. Make it taking a void-argument so that the caller can reduce instructions obtaining the "current" task and it can be used in THUNK later. Cc: "Paul E. McKenney" Cc: Peter Zijlstra Cc: Frederic Weisbecker Signed-off-by: Lai Jiangshan --- include/linux/rcupdate.h | 2 ++ kernel/rcu/tree_plugin.h | 7 ++++--- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index ebe9bfc700bb..328667ae8086 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -68,6 +68,8 @@ static inline bool same_state_synchronize_rcu(unsigned long oldstate1, unsigned #ifdef CONFIG_PREEMPT_RCU +void rcu_read_unlock_special(void); + void __rcu_read_lock(void); void __rcu_read_unlock(void); diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h index 3dbd5609185e..880b3fef1158 100644 --- a/kernel/rcu/tree_plugin.h +++ b/kernel/rcu/tree_plugin.h @@ -104,7 +104,6 @@ static void __init rcu_bootup_announce_oddness(void) static void rcu_report_exp_rnp(struct rcu_node *rnp, bool wake); static bool sync_rcu_exp_done(struct rcu_node *rnp); -static void rcu_read_unlock_special(struct task_struct *t); #define set_rcu_preempt_special(reason) do { \ WRITE_ONCE(current->rcu_read_unlock_special.b.reason, true); \ @@ -427,7 +426,7 @@ void __rcu_read_unlock(void) if (rcu_preempt_read_exit() == 0) { barrier(); // critical-section exit before .s check. if (unlikely(READ_ONCE(t->rcu_read_unlock_special.s))) - rcu_read_unlock_special(t); + rcu_read_unlock_special(); } if (IS_ENABLED(CONFIG_PROVE_LOCKING)) { int rrln = rcu_preempt_depth(); @@ -627,8 +626,9 @@ static void rcu_preempt_deferred_qs_handler(struct irq_work *iwp) * notify RCU core processing or task having blocked during the RCU * read-side critical section. */ -static void rcu_read_unlock_special(struct task_struct *t) +void rcu_read_unlock_special(void) { + struct task_struct *t = current; unsigned long flags; bool irqs_were_disabled; bool preempt_bh_were_disabled = @@ -684,6 +684,7 @@ static void rcu_read_unlock_special(struct task_struct *t) } rcu_preempt_deferred_qs_irqrestore(t, flags); } +EXPORT_SYMBOL_GPL(rcu_read_unlock_special); /* * Check that the list of blocked tasks for the newly completed grace From patchwork Sun Apr 7 09:05:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lai Jiangshan X-Patchwork-Id: 13620091 Received: from mail-pg1-f176.google.com (mail-pg1-f176.google.com [209.85.215.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2A3BC22F02; Sun, 7 Apr 2024 09:03:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712480635; cv=none; b=JsMUejhSOD/p9WDs0Z5ccP69t+EK+BCIX1hgVFvoRPSghs/TqMYDe//mzfz78ZzZU2bOWJKXLO8GCAihmpSQ0ESwPu6qCzza0JW+j06L+j/YYCWY17vY7dGfi77TvJCbZTEPhHA4dKU7JOsXM9lA55n1b1xPTAXv6HArdK7fajM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712480635; c=relaxed/simple; bh=3TturtFYIwfV1u8ACm2gm3tHdeACQEBcKlzkWT/2VSg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=KsJfJ4rkUx7Ti05n00K2Q1THzY3hGnym0gvw68dy3Hrqz0dZ8h5ucdxokJBXhJ9oOwcIfkwxJC7I6aZb6wvoP3es1BOztrQT7IWtiNbsVwIODb9uL0XNAUT78MsLxMz3h/fmdGTQBgTXAP2W579t4SnFe2+f/CN9jpQTp71B0O8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=M1xsNfx0; arc=none smtp.client-ip=209.85.215.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="M1xsNfx0" Received: by mail-pg1-f176.google.com with SMTP id 41be03b00d2f7-5dbcfa0eb5dso2580802a12.3; Sun, 07 Apr 2024 02:03:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712480633; x=1713085433; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=U7zjP/fD7ntbdBgOTsfd20ge3XzeOlmyb2f9DwxHaY0=; b=M1xsNfx0NmYoubyMUScrEDrph+U8x/PfXlmiFJft93QuVlGx1Xc9u8FmDxQ1w7SOfu zDJvfuGr3K4kL1Xw5AvSM0k7whx12HNXPPmBp3HldfjA7vH6+u4BT8XYf2SRelAgoNU6 p3weNc/P7KQi5A4aO76VKHIkT19b6uVMhSkkFxj7JGAmFythsSgG6AjWcgIyyaV7bqF6 9+varSYGy4yfIGDJuY7JudIbuf4PE2KlacMsVHANa6dgiQIecfzb4xWnM/ikRZGEaMsZ 92C1K2WC/1qtPyaSmPMDzysDJeb1nz+hbeuzeNQrV83/QQuQgglHiiJI8mkJuSQ2iuu7 u6BA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712480633; x=1713085433; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=U7zjP/fD7ntbdBgOTsfd20ge3XzeOlmyb2f9DwxHaY0=; b=BJBRYsqWNYA6Sox7Jekcx9Cw0pXEY4oXUpNcdLIy62gpV1VZgUk3qmQn9suGDuexRk f1TgpmLnt47CQuRWsS0zD3BxyPL4a7Uym+28s41CXF7wpVJdnqulJkaiKl6PIXMDlCJW 6C+uINsBR4Tadq/Wg+0IUv0IHwmCxW93hYxjvPJnz9V5ygU/tnfFJFb/bIXYsNIiGV/4 H+i3jY3xvjzLDUGJwsadsPlDy+ACXHr6l3l1zBow7BeA4VYp5EoawSzn9NN1sTIXM6dk zSp2Qtnr807VU9rcq0eSgWl3Bz18AkIsTQSDh4B8yqZCUJMBbTLP4/A4JBaKK3Z8DXHf VECg== X-Gm-Message-State: AOJu0YxkuSd3vvqi6Gm3AIj8z+o+msgIT1dR4sffQLFMxoCk3FE2H/6O /RJIBbKKLHfVqWjRa7PksupvHZCA/8JESbxBHUs2icNqqgSjSvAWW6EVjscE X-Google-Smtp-Source: AGHT+IGUXYj82DWPk7jq9yGQJ0difrAS0sPahh0a1BfbXK03ecPROvUt1tXny7PR6YY3g1QvTtJxog== X-Received: by 2002:a17:902:d386:b0:1e0:119e:f935 with SMTP id e6-20020a170902d38600b001e0119ef935mr4703483pld.15.1712480633128; Sun, 07 Apr 2024 02:03:53 -0700 (PDT) Received: from localhost ([47.254.32.37]) by smtp.gmail.com with ESMTPSA id h7-20020a170902680700b001e2b4f513e1sm4632217plk.106.2024.04.07.02.03.52 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 07 Apr 2024 02:03:52 -0700 (PDT) From: Lai Jiangshan To: linux-kernel@vger.kernel.org Cc: rcu@vger.kernel.org, x86@kernel.org, Lai Jiangshan , "Paul E. McKenney" , Peter Zijlstra , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang Subject: [PATCH V2 06/11] rcu: Rename marco __LINUX_RCU_H to __KERNEL_RCU_H Date: Sun, 7 Apr 2024 17:05:53 +0800 Message-Id: <20240407090558.3395-7-jiangshanlai@gmail.com> X-Mailer: git-send-email 2.19.1.6.gb485710b In-Reply-To: <20240407090558.3395-1-jiangshanlai@gmail.com> References: <20240407090558.3395-1-jiangshanlai@gmail.com> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Lai Jiangshan to reflect the path of kernel/rcu/rcu.h. Cc: "Paul E. McKenney" Cc: Peter Zijlstra Cc: Frederic Weisbecker Signed-off-by: Lai Jiangshan --- kernel/rcu/rcu.h | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/kernel/rcu/rcu.h b/kernel/rcu/rcu.h index 86fce206560e..b17b2ed657fc 100644 --- a/kernel/rcu/rcu.h +++ b/kernel/rcu/rcu.h @@ -7,8 +7,8 @@ * Author: Paul E. McKenney */ -#ifndef __LINUX_RCU_H -#define __LINUX_RCU_H +#ifndef __KERNEL_RCU_H +#define __KERNEL_RCU_H #include #include @@ -668,4 +668,4 @@ int rcu_stall_notifier_call_chain(unsigned long val, void *v); static inline int rcu_stall_notifier_call_chain(unsigned long val, void *v) { return NOTIFY_DONE; } #endif // #else // #if defined(CONFIG_RCU_STALL_COMMON) && defined(CONFIG_RCU_CPU_STALL_NOTIFIER) -#endif /* __LINUX_RCU_H */ +#endif /* __KERNEL_RCU_H */ From patchwork Sun Apr 7 09:05:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lai Jiangshan X-Patchwork-Id: 13620092 Received: from mail-pf1-f171.google.com (mail-pf1-f171.google.com [209.85.210.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 31B5717577; Sun, 7 Apr 2024 09:03:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712480639; cv=none; b=Vb9Mr5iVQZhmAW5gPQzcd2l8nySg1DlxiA6YYNne7DJVeBZNObcCcJ5w5DN5CnavQ4ZgVOmYW3UmOGFfKdPkDJ6VXWJTQOD8Taww2+IEuIbdCOmJNAuk3ymhwpDkMmNRDPkxcWq8JrgzcmmImDsi+x/72MSXWxs5yAk4tBo/C/g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712480639; c=relaxed/simple; bh=WZdyr+NknpomhAuRPc/9gTXcGeciCnFG1T1UfMulHtw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=EjTrCYLaRPpyjMOIQP+aA9i9AmyuI91mmwAEUa4ZU738RC/uaJIcdgLQKyA8CvUaESKtY7rK3oSVqMRIIPUxHp/4BYx6ixkFSXW05TEyHT94tbJelFzVT4N8A6lYrWUzWqIn7Bmy63IbwncOeNJcdRD8bKz0ddg8EM11hjM4J+Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=ImjrK7vQ; arc=none smtp.client-ip=209.85.210.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ImjrK7vQ" Received: by mail-pf1-f171.google.com with SMTP id d2e1a72fcca58-6ed2dbf3c92so10756b3a.2; Sun, 07 Apr 2024 02:03:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712480637; x=1713085437; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=GpTyCyqlk6BGMCLifNqvgRTCltbDUUo/xKTC4G72UYQ=; b=ImjrK7vQ3SpazvjiazZjJRRR8QJGBogDhVeLV9OQeFoVn9OasdnrdXUXiInFZOljk4 UqZ4yt4gbev8jAjk4lnOg+S6asstT1zVdAoq9RgSTfgbHTCKiQ/D8JkADyhHXFMRcQDU AGKNMyeKmCupZxANYFv272VUFccusZlgeV7UwF8BqzxZ0swq+EpIawZXe/vIfeDExfSy hPXSRUs4bWeh9ykPjdteY24+0rlIMuhHic0JTGSBIFoQ4m+FCEgB8AXlJdGfvALWa1f8 A5VQagrCH+rPfGm9XLgFZk2UPSO/QgI18mafYq7uBqKQGOFTA12w/I9/KDYO81lTv80D jkvw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712480637; x=1713085437; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GpTyCyqlk6BGMCLifNqvgRTCltbDUUo/xKTC4G72UYQ=; b=Aa4N5fv3wb1cDfObTghe/mtIEtISN5O+DVHxsftPitI3VCv47g1jgXpmmlSbJ/0a/F 9Rh5IGxt+IC5svdWnQJy8Qsw0wIl9cRM4i+XCGQjYVXZG1kn6THMQQgG8kl1zJhQg35W P0rghSBz1YsNUGXs2H2k+iHgbQ2fyAnY3Ook0WUSgLWOP4scrqFxWuDZlKUqg68/4ZYD SQLCBPOAnOQSF75wgSVTPMmAuefK0JMW7dC4Z3wwiv0FVutB9EGEeQzYnzRabJ1baDIU QfGVDoS3dgimrQTLuJ6NXs7LahgeRsLlprFDiaieubvldEZVy0/8nDpqzQINxVe1Tpxv NZRA== X-Gm-Message-State: AOJu0YzCbkcDlBf1ZobgZL67MNN9x5DoyrxxmMdFvifBpXep6Xv2trbe a33jXybuzXj+Wqq7WQdo6jkWQ5rrZgXYu+rqnPvxg3iVBlMQZAzvSUzdgM80 X-Google-Smtp-Source: AGHT+IGt7lhWPbhDk/feHJIC0p6YENAzxvvr8u7QkT8swnrmIKOUIBIf8sPv6ly2Voskt8dfuV6jvg== X-Received: by 2002:a05:6a20:ce4b:b0:1a1:8312:6dfb with SMTP id id11-20020a056a20ce4b00b001a183126dfbmr6281164pzb.58.1712480637225; Sun, 07 Apr 2024 02:03:57 -0700 (PDT) Received: from localhost ([47.89.225.180]) by smtp.gmail.com with ESMTPSA id s35-20020a17090a2f2600b0029baf24ee51sm6648516pjd.48.2024.04.07.02.03.56 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 07 Apr 2024 02:03:56 -0700 (PDT) From: Lai Jiangshan To: linux-kernel@vger.kernel.org Cc: rcu@vger.kernel.org, x86@kernel.org, Lai Jiangshan , "Paul E. McKenney" , Peter Zijlstra , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Ingo Molnar , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira , Valentin Schneider Subject: [PATCH V2 07/11] sched/core: Add rcu_preempt_switch() Date: Sun, 7 Apr 2024 17:05:54 +0800 Message-Id: <20240407090558.3395-8-jiangshanlai@gmail.com> X-Mailer: git-send-email 2.19.1.6.gb485710b In-Reply-To: <20240407090558.3395-1-jiangshanlai@gmail.com> References: <20240407090558.3395-1-jiangshanlai@gmail.com> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Lai Jiangshan When the per-cpu rcu_preempt_count is used, it has to be switched too on context-switching. And the instructions to switch the per-cpu rcu_preempt_count are few, so it is inlined to avoid the overhead in the scheduler. Cc: "Paul E. McKenney" Cc: Peter Zijlstra Cc: Frederic Weisbecker Signed-off-by: Lai Jiangshan --- kernel/rcu/rcu.h | 5 +++++ kernel/sched/core.c | 2 ++ 2 files changed, 7 insertions(+) diff --git a/kernel/rcu/rcu.h b/kernel/rcu/rcu.h index b17b2ed657fc..ea5ae957c687 100644 --- a/kernel/rcu/rcu.h +++ b/kernel/rcu/rcu.h @@ -668,4 +668,9 @@ int rcu_stall_notifier_call_chain(unsigned long val, void *v); static inline int rcu_stall_notifier_call_chain(unsigned long val, void *v) { return NOTIFY_DONE; } #endif // #else // #if defined(CONFIG_RCU_STALL_COMMON) && defined(CONFIG_RCU_CPU_STALL_NOTIFIER) +static inline void +rcu_preempt_switch(struct task_struct *prev, struct task_struct *next) +{ +} + #endif /* __KERNEL_RCU_H */ diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 7019a40457a6..1d9e3c51c913 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -95,6 +95,7 @@ #include "../workqueue_internal.h" #include "../../io_uring/io-wq.h" #include "../smpboot.h" +#include "../rcu/rcu.h" EXPORT_TRACEPOINT_SYMBOL_GPL(ipi_send_cpu); EXPORT_TRACEPOINT_SYMBOL_GPL(ipi_send_cpumask); @@ -6737,6 +6738,7 @@ static void __sched notrace __schedule(unsigned int sched_mode) */ ++*switch_count; + rcu_preempt_switch(prev, next); migrate_disable_switch(rq, prev); psi_sched_switch(prev, next, !task_on_rq_queued(prev)); From patchwork Sun Apr 7 09:05:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lai Jiangshan X-Patchwork-Id: 13620093 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 47F9133CF5; Sun, 7 Apr 2024 09:04:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712480645; cv=none; b=HNlxf5EX0LqkmnBEpwD/elOlcKSyXKaj2PhrfD+vi28sYNkcqZbzsuOT8UKGRiKKAkfAA4LQbJthF7418KaranIVw7EMH7bG+oO/exb9nEvztKo6kwv1aB3lNhdwwksVlsz6z5YFb7IykKiNNvG206v2U6RRdGwDgzu7Y+SqqCE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712480645; c=relaxed/simple; bh=FJ4sCFu41FqELkYNcQeO9GHd0HS+4qyW6t8CF+WAzbc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=UT473uI7wRtiNuaJQDd/LCP0NpYUH7s+I5ssCHlT1zNoopMFVhB70g4GXfamISfNKAI56E6pb97RUfw8b7tPa3caVWNEpWqw2ISbiyP+DdZCbP2GdNkgP6cz+cl44vr9PaW1+q2KAFdHr+qcmCeFmhXfaapp9YSa55DDzDIM8M4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=VB6XghBs; arc=none smtp.client-ip=209.85.214.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="VB6XghBs" Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-1e2bbc2048eso28541015ad.3; Sun, 07 Apr 2024 02:04:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712480643; x=1713085443; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=P6Ao7lt8bYptuXYSouvezeqYjGeG+t4ZB0d0AMHEq3o=; b=VB6XghBs0CEh3lGFxzG0vfzU8XnskjjBc28EmlAfLDDEXxW3PeFMTJ+gpys3sCptsc vfp9QKaa97BQQKS5ZtxeZzl+89HYuRz4bCQZLOQHvKrAGRF4CpfaLBg6dl7Yy6TgN+Fv J4x2RBEjhVrfeWZTnVpLLcRB3Qd+DuLleDAOlrP0i6stfWXf3TqCo3UoV4ugpC3t8YiM VSnjiaQpuhx4owipL6RIFc1kOlN8Yi1xgiE3dxwHC16TrJmtzgL/7lD1El3DPOBB+bT9 jBlAXVqp4opNQkk+LQaVbgfAU0tnAFew5TMtpCeyXVNrNwlhXKIwyljCxIH4wNbf9jQr 62Lw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712480643; x=1713085443; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=P6Ao7lt8bYptuXYSouvezeqYjGeG+t4ZB0d0AMHEq3o=; b=qxgggXfNMe9YSzqoc/toYMzeDM3YhZWATbGUBy97kTWDs8kaIQpkaaOw1t3Akkd2+p yEBl6V8wOEyyMeDWSOLFlqLrya4Cep9VWtHHPV9aAcz5LP5dwPiF5pUrUHX+Ahp12Ixc +fvodOIDsxWnfWdhMsPOec4P7JBsNt/F/BcGd5kG4fQhR7fJ6w0O55ISXYN+p9Yb0h+g hnSSzWqM74IYKxMx6gyo556c6pAagvP8vePEJKvLocxr/aLPZ0BofSDbLAvYJSSRZ9EB mmUk1tTETnRVoaTB6AF09FqBEKNDmvAZtUtg12PWXnzu0EPSZLhMCucuTaQTXToRs26Y ewMg== X-Gm-Message-State: AOJu0YwMgvjVJjkLtcqqYpazOumXYBQqEOF/jOD7JMPG6sGjzY+8laR0 Uo/jdnAXNc6M7rRw8sI/qoUKmKqB9Gqk/75I3m84r8Dj3CfEztqzeRD/P8yK X-Google-Smtp-Source: AGHT+IFkaG5AT1Vcbcxhl+vZNr+Yss32Phiw8WSKNjNbCK2mZPlpEBzEAmEdTwic4cfsu/0SJkgRvw== X-Received: by 2002:a17:902:ce03:b0:1e4:43f:64d4 with SMTP id k3-20020a170902ce0300b001e4043f64d4mr940632plg.49.1712480643183; Sun, 07 Apr 2024 02:04:03 -0700 (PDT) Received: from localhost ([47.254.32.37]) by smtp.gmail.com with ESMTPSA id w26-20020a63161a000000b005f0a1a46f08sm3743660pgl.11.2024.04.07.02.04.02 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 07 Apr 2024 02:04:02 -0700 (PDT) From: Lai Jiangshan To: linux-kernel@vger.kernel.org Cc: rcu@vger.kernel.org, x86@kernel.org, Lai Jiangshan , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Subject: [PATCH V2 08/11] x86/entry: Merge thunk_64.S and thunk_32.S into thunk.S Date: Sun, 7 Apr 2024 17:05:55 +0800 Message-Id: <20240407090558.3395-9-jiangshanlai@gmail.com> X-Mailer: git-send-email 2.19.1.6.gb485710b In-Reply-To: <20240407090558.3395-1-jiangshanlai@gmail.com> References: <20240407090558.3395-1-jiangshanlai@gmail.com> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Lai Jiangshan The code in thunk_64.S and thunk_32.S are exactly the same except for the comments. Merge them in to thunk.S. And since thunk_32.S was originated from thunk_64.S, the new merged thunk.S is actually renamed from thunk_64.S. Signed-off-by: Lai Jiangshan --- arch/x86/entry/Makefile | 2 +- arch/x86/entry/{thunk_64.S => thunk.S} | 0 arch/x86/entry/thunk_32.S | 18 ------------------ 3 files changed, 1 insertion(+), 19 deletions(-) rename arch/x86/entry/{thunk_64.S => thunk.S} (100%) delete mode 100644 arch/x86/entry/thunk_32.S diff --git a/arch/x86/entry/Makefile b/arch/x86/entry/Makefile index c93e7f5c2a06..ce1cc1622385 100644 --- a/arch/x86/entry/Makefile +++ b/arch/x86/entry/Makefile @@ -17,7 +17,7 @@ obj-y += common.o obj-y += vdso/ obj-y += vsyscall/ -obj-$(CONFIG_PREEMPTION) += thunk_$(BITS).o +obj-$(CONFIG_PREEMPTION) += thunk.o CFLAGS_entry_fred.o += -fno-stack-protector CFLAGS_REMOVE_entry_fred.o += -pg $(CC_FLAGS_FTRACE) obj-$(CONFIG_X86_FRED) += entry_64_fred.o entry_fred.o diff --git a/arch/x86/entry/thunk_64.S b/arch/x86/entry/thunk.S similarity index 100% rename from arch/x86/entry/thunk_64.S rename to arch/x86/entry/thunk.S diff --git a/arch/x86/entry/thunk_32.S b/arch/x86/entry/thunk_32.S deleted file mode 100644 index da37f42f4549..000000000000 --- a/arch/x86/entry/thunk_32.S +++ /dev/null @@ -1,18 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-only */ -/* - * Trampoline to trace irqs off. (otherwise CALLER_ADDR1 might crash) - * Copyright 2008 by Steven Rostedt, Red Hat, Inc - * (inspired by Andi Kleen's thunk_64.S) - */ - -#include -#include -#include - -#include "calling.h" - -THUNK preempt_schedule_thunk, preempt_schedule -THUNK preempt_schedule_notrace_thunk, preempt_schedule_notrace -EXPORT_SYMBOL(preempt_schedule_thunk) -EXPORT_SYMBOL(preempt_schedule_notrace_thunk) - From patchwork Sun Apr 7 09:05:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lai Jiangshan X-Patchwork-Id: 13620094 Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9E6DF376E4; Sun, 7 Apr 2024 09:04:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712480650; cv=none; b=GP4whrBO2tvEo/wEhyfyADx38SQOmCkQPomc4aLxnhqiRGqD7rAThmGd0b/Nbs+qs94q+VRgp5KvUw79UZvZMLYhFuC+neT8FE3lmE25VsHRmL54SrlX6A8Zhx/dY3bQm1fKcrsBG+KH9BTRM7KZL3a8EEZNZcPFj4pmD1N7ykU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712480650; c=relaxed/simple; bh=SKi1Hw6pqGBhLR48UF2gLw2ljWBn/PhxuaBl95AwpXI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=C9LFDxmv9djHLii7uImf8t6kscOqp3JSoYXqGJW8u4W09dYGXd9MYKPhSwOaIIN4Gr3Jtb7bvjwrCB0vJXOfyHXN8w9e4HpdrgvlujD3mDxlWwX3GRoue1KW2TgjpSfZJ4QKBUIYWaUpcZCFfbZxyth0pbE4tH7l2wAYE8fhuA0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=V3BOo+Gs; arc=none smtp.client-ip=209.85.214.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="V3BOo+Gs" Received: by mail-pl1-f179.google.com with SMTP id d9443c01a7336-1e419d1f080so331525ad.1; Sun, 07 Apr 2024 02:04:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712480647; x=1713085447; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=uvmZr4F4KrE2SgRrk5vM9yP83Kb1iDKJQwNWlrc52tI=; b=V3BOo+Gs71UmD4/rCg29MqQgSf/Zh5ALCtL3yUh8JnWS17TaS8y3atqmhTLn+VjakX C2pPNwzVgMVKbhwOk7/JjNgxYnXBP6UKd7X+enZGZCv/qod0sESVNlJ9Cj0IxE3gUWmU xBg7NMyOhXLInOVjrS3HE1d0Sj+SAufDOjoNRe54sNYSkL87OFDYKGmyNQn4sOvrupTq zBSb75OVnjA8Xt80gG3WqoNjjYn7qHZ6oU27nwFK7eaI+/TeW9S1btUuwQ7e5P5QCjNJ uXhabc14UdXOJIqqFjdEj2lPrloiWm3YNX1nepIr+tcGW40ghmC4i55pxleg0DH0Ouk5 Ui/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712480647; x=1713085447; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=uvmZr4F4KrE2SgRrk5vM9yP83Kb1iDKJQwNWlrc52tI=; b=e8nddDsO8sFu/nTv9uFZCIc+jgxDc24OBzUmLpIKZYDEekJR+fVL5pMR16eI3wqk+g qzW6lLTEsyYWlMXwV6KZeswRwij4O0cGjrgOm33ZZWN7+tduFeOECZw9NV4NsHJEKc+F 8wAnauLXB1IeyNF1lfp8nuZwpvEOW9hMbygq9RSrWeswrCzdCKzcyzEqCIDvkBk3NlNL dIb+2fY3cwpKb/XR8khJTmiP/uuKW5ScHbegfKtatdO2lIK764/ubAZdD1XmSJZ6YUl/ AeZXM2/dWsvEsQUJi1ilS+NWDEN1lgYEV5AbNtilgr0Tn8BbhJ5irp19NdnLO7IJPRbi 9lJw== X-Gm-Message-State: AOJu0YyH8IFt6Ah9mDnTGsLwf7C6cWcQteTtrVAbMr9keeF+L1P2qODH K5DZtx7dbUGb+u67eBHKCcZi6luBexRcciaOqCWjvez/xZWRVT9nKoXCxFzX X-Google-Smtp-Source: AGHT+IGj1b4HAvBohPJYYhY/r/gbQG5Zn4YiBDo+LF5oMqonLV2JPcEw09TM6F1adDIHhvF8GCd35g== X-Received: by 2002:a17:902:ce87:b0:1e2:c6ed:d1c with SMTP id f7-20020a170902ce8700b001e2c6ed0d1cmr5117315plg.42.1712480647622; Sun, 07 Apr 2024 02:04:07 -0700 (PDT) Received: from localhost ([198.11.178.15]) by smtp.gmail.com with ESMTPSA id h12-20020a170902f54c00b001dd99fe365dsm4517205plf.42.2024.04.07.02.04.06 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 07 Apr 2024 02:04:07 -0700 (PDT) From: Lai Jiangshan To: linux-kernel@vger.kernel.org Cc: rcu@vger.kernel.org, x86@kernel.org, Lai Jiangshan , "Paul E. McKenney" , Peter Zijlstra , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang Subject: [PATCH V2 09/11] rcu: Implement PCPU_RCU_PREEMPT_COUNT framework Date: Sun, 7 Apr 2024 17:05:56 +0800 Message-Id: <20240407090558.3395-10-jiangshanlai@gmail.com> X-Mailer: git-send-email 2.19.1.6.gb485710b In-Reply-To: <20240407090558.3395-1-jiangshanlai@gmail.com> References: <20240407090558.3395-1-jiangshanlai@gmail.com> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Lai Jiangshan When the arch code provides HAVE_PCPU_RCU_PREEMPT_COUNT and the corresponding functions, rcu core uses the functions to implement rcu_preempt_depth(), special bits, switching and so on. Cc: "Paul E. McKenney" Cc: Peter Zijlstra Cc: Frederic Weisbecker Signed-off-by: Lai Jiangshan --- include/linux/rcupdate.h | 33 +++++++++++++++++++++++++++++++++ kernel/rcu/Kconfig | 8 ++++++++ kernel/rcu/rcu.h | 4 ++++ kernel/rcu/tree_plugin.h | 8 ++++++++ 4 files changed, 53 insertions(+) diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index 328667ae8086..e3e5ce44c7dc 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -70,6 +70,8 @@ static inline bool same_state_synchronize_rcu(unsigned long oldstate1, unsigned void rcu_read_unlock_special(void); +#ifndef CONFIG_PCPU_RCU_PREEMPT_COUNT + void __rcu_read_lock(void); void __rcu_read_unlock(void); @@ -81,6 +83,37 @@ void __rcu_read_unlock(void); */ #define rcu_preempt_depth() READ_ONCE(current->rcu_read_lock_nesting) #define rcu_preempt_depth_set(val) WRITE_ONCE(current->rcu_read_lock_nesting, (val)) +#define pcpu_rcu_preempt_special_set() do { } while (0) +#define pcpu_rcu_preempt_special_clear() do { } while (0) + +#else /* #ifndef CONFIG_PCPU_RCU_PREEMPT_COUNT */ + +#include + +static __always_inline void __rcu_read_lock(void) +{ + pcpu_rcu_preempt_count_add(1); + barrier(); +} + +static __always_inline void __rcu_read_unlock(void) +{ + barrier(); + if (unlikely(pcpu_rcu_preempt_count_dec_and_test())) + pcpu_rcu_read_unlock_special(); +} + +static inline int rcu_preempt_depth(void) +{ + return pcpu_rcu_preempt_count(); +} + +static inline void rcu_preempt_depth_set(int val) +{ + pcpu_rcu_preempt_count_set(val); +} + +#endif /* #else #ifndef CONFIG_PCPU_RCU_PREEMPT_COUNT */ #else /* #ifdef CONFIG_PREEMPT_RCU */ diff --git a/kernel/rcu/Kconfig b/kernel/rcu/Kconfig index e7d2dd267593..5d91147bc9a3 100644 --- a/kernel/rcu/Kconfig +++ b/kernel/rcu/Kconfig @@ -345,4 +345,12 @@ config RCU_DOUBLE_CHECK_CB_TIME Say Y here if you need tighter callback-limit enforcement. Say N here if you are unsure. +config HAVE_PCPU_RCU_PREEMPT_COUNT + bool + +config PCPU_RCU_PREEMPT_COUNT + def_bool y + depends on PREEMPT_RCU && HAVE_PCPU_RCU_PREEMPT_COUNT + depends on !PROVE_LOCKING && !RCU_STRICT_GRACE_PERIOD + endmenu # "RCU Subsystem" diff --git a/kernel/rcu/rcu.h b/kernel/rcu/rcu.h index ea5ae957c687..2322b040c5cd 100644 --- a/kernel/rcu/rcu.h +++ b/kernel/rcu/rcu.h @@ -671,6 +671,10 @@ static inline int rcu_stall_notifier_call_chain(unsigned long val, void *v) { re static inline void rcu_preempt_switch(struct task_struct *prev, struct task_struct *next) { +#ifdef CONFIG_PCPU_RCU_PREEMPT_COUNT + prev->rcu_read_lock_nesting = rcu_preempt_depth(); + pcpu_rcu_preempt_switch(next->rcu_read_lock_nesting, next->rcu_read_unlock_special.s); +#endif // #ifdef CONFIG_PCPU_RCU_PREEMPT_COUNT } #endif /* __KERNEL_RCU_H */ diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h index 880b3fef1158..db68d0c1c1f2 100644 --- a/kernel/rcu/tree_plugin.h +++ b/kernel/rcu/tree_plugin.h @@ -107,10 +107,13 @@ static bool sync_rcu_exp_done(struct rcu_node *rnp); #define set_rcu_preempt_special(reason) do { \ WRITE_ONCE(current->rcu_read_unlock_special.b.reason, true); \ + pcpu_rcu_preempt_special_set(); \ } while (0) #define clear_rcu_preempt_special(reason) do { \ WRITE_ONCE(current->rcu_read_unlock_special.b.reason, false); \ + if (!current->rcu_read_unlock_special.s) \ + pcpu_rcu_preempt_special_clear(); \ } while (0) /* @@ -379,6 +382,8 @@ static int rcu_preempt_blocked_readers_cgp(struct rcu_node *rnp) return READ_ONCE(rnp->gp_tasks) != NULL; } +#ifndef CONFIG_PCPU_RCU_PREEMPT_COUNT + /* limit value for ->rcu_read_lock_nesting. */ #define RCU_NEST_PMAX (INT_MAX / 2) @@ -436,6 +441,8 @@ void __rcu_read_unlock(void) } EXPORT_SYMBOL_GPL(__rcu_read_unlock); +#endif /* #ifndef CONFIG_PCPU_RCU_PREEMPT_COUNT */ + /* * Advance a ->blkd_tasks-list pointer to the next entry, instead * returning NULL if at the end of the list. @@ -489,6 +496,7 @@ rcu_preempt_deferred_qs_irqrestore(struct task_struct *t, unsigned long flags) return; } t->rcu_read_unlock_special.s = 0; + pcpu_rcu_preempt_special_clear(); if (special.b.need_qs) { if (IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD)) { rdp->cpu_no_qs.b.norm = false; From patchwork Sun Apr 7 09:05:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lai Jiangshan X-Patchwork-Id: 13620095 Received: from mail-pf1-f176.google.com (mail-pf1-f176.google.com [209.85.210.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2C3BF38DEC; Sun, 7 Apr 2024 09:04:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712480661; cv=none; b=FNuo6gIRX/c9zH7morNLEIA2PwOCt+AkL++PjdpPd7hp+WR5kQcS8TpDec7XjQlcPuUgdRzqDTvz4imt4JvxAYUpLnLdkZHnva3lUPsik8Nh3u0qfOFwaLixBxiRLMwu8t9enFb2Br3S70iQZAKXWVYkyau2owFFyZgecxbqM9Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712480661; c=relaxed/simple; bh=biS7KzzHTKcOQAMJum93/nBTbQjerhk67V4BVUndbxs=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=KFvkIINNQBRGw1CmZS6wMCn7zJzicxsehnVPY0BNNQcNSpioW+qZx6OuHxgBZTFrluvMgUYy/nhDnUBDokxEN09s8YdCFJBHFctl+p/2xdqUGl5113oLidwUIdVTu/Z5rhKz2qlP5N/rDgBz418krlosv7mclnaWc6DiUdxzuho= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=U8N5sEYz; arc=none smtp.client-ip=209.85.210.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="U8N5sEYz" Received: by mail-pf1-f176.google.com with SMTP id d2e1a72fcca58-6ed04c91c46so1752574b3a.0; Sun, 07 Apr 2024 02:04:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712480659; x=1713085459; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8pFdud31CO30bbz/KPNSHwu6wK19wUtd+HcNwJYEJfY=; b=U8N5sEYztOMLwG/sg0hjXHkBMQbgbp9eOeMEK2mkl7XE3EDtQObQ1afeIZ5kBv9vME 3BfNA+oUm8ptdHbZcdPYlHdDPBmIZobkRsjKwX/xhphkW7sg0O65kI7aExLBZbGL0aRR EQCSmaqnUpIVQ1UGFTHWdQydM6IiVBAHFUazkMXxKrV2jAJMAzrUNpnL+uxBnboCaQhc HXiAVGOVfS4i7hycABprt5mgT91MlW5cfrVZhj9DOe81MmlbW0WeHHWU/bMzZWspYCHO SNEQU9jibNay/CrVXdF8i1zsRs5H0dLh6AASuadR4roLiSO4exZWDsTTKdATyOLEJbKR Av0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712480659; x=1713085459; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8pFdud31CO30bbz/KPNSHwu6wK19wUtd+HcNwJYEJfY=; b=KIFOF9iMiXd7TdPgRxWlOpZcU79uudNDa6SQDGbjKsGNJ75gXtF70lp9Amzfj8lMoU 5VgJ+Jk6WcjQFYexQV2B5C6ur74jbX7GLyFZNrRGILgRZlGDOUJAIDIAq/nX7wnVMu4u bkDatfYXUSip+Rabl1YcxgnPBS2mp7gBt88ttv2cXfHBesB+eW4o+b8NRFTyZBIa2p6v iRHG0dhqgXsGMPNtpbbFaSX84dybUlg9yx1JHrhN1DSoRpQqfLzoVKq5YfzaB1OXvX/F R9CSIIzquOl/2LS+PsxEfqGaLRFKWy0t5mSRyzFbAbQt4qNoR+/0oTd4kDIQHvBZ3QLt t7Xg== X-Gm-Message-State: AOJu0YxPbsQ3oncf7vL2b/Id46mp4CC9quLdHRUSn6Fo0B4hFTxtdrPT UqN67OPq/j7wdh9QXV/sAcCzqbpy7tvpjLF1Pp3cOYg+fZnt8svrGhRzImRB X-Google-Smtp-Source: AGHT+IEaKJcXHWyEbaXHBNXfVOIsKNaj3lY8ob579KruAgIbKEgl/YO0KV/ToaFfMleSxyGz42XHwA== X-Received: by 2002:a05:6a21:329b:b0:1a7:1cb:3e20 with SMTP id yt27-20020a056a21329b00b001a701cb3e20mr7113512pzb.54.1712480659129; Sun, 07 Apr 2024 02:04:19 -0700 (PDT) Received: from localhost ([198.11.178.15]) by smtp.gmail.com with ESMTPSA id r6-20020aa78446000000b006eb0027f2b8sm4322191pfn.9.2024.04.07.02.04.18 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 07 Apr 2024 02:04:18 -0700 (PDT) From: Lai Jiangshan To: linux-kernel@vger.kernel.org Cc: rcu@vger.kernel.org, x86@kernel.org, Lai Jiangshan , "Paul E. McKenney" , Peter Zijlstra , Frederic Weisbecker , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Uros Bizjak , Josh Poimboeuf , Breno Leitao , Kent Overstreet , Pawan Gupta , Rick Edgecombe , Vegard Nossum , Daniel Sneddon , Nikolay Borisov Subject: [PATCH V2 10/11] x86/rcu: Add rcu_preempt_count Date: Sun, 7 Apr 2024 17:05:57 +0800 Message-Id: <20240407090558.3395-11-jiangshanlai@gmail.com> X-Mailer: git-send-email 2.19.1.6.gb485710b In-Reply-To: <20240407090558.3395-1-jiangshanlai@gmail.com> References: <20240407090558.3395-1-jiangshanlai@gmail.com> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Lai Jiangshan Implement PCPU_RCU_PREEMPT_COUNT for x86. Mainly copied from asm/preempt.h Make rcu_read_[un]lock() inlined for rcu-preempt. Make rcu_read_lock() only one instruction. Make rcu_read_unlock() only two instructions in the fast path. Cc: "Paul E. McKenney" Cc: Peter Zijlstra Cc: Frederic Weisbecker Signed-off-by: Lai Jiangshan --- arch/x86/Kconfig | 1 + arch/x86/include/asm/current.h | 3 + arch/x86/include/asm/rcu_preempt.h | 107 +++++++++++++++++++++++++++++ arch/x86/kernel/cpu/common.c | 4 ++ 4 files changed, 115 insertions(+) create mode 100644 arch/x86/include/asm/rcu_preempt.h diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 4fff6ed46e90..e805cac3763d 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -256,6 +256,7 @@ config X86 select HAVE_OBJTOOL if X86_64 select HAVE_OPTPROBES select HAVE_PAGE_SIZE_4KB + select HAVE_PCPU_RCU_PREEMPT_COUNT select HAVE_PCSPKR_PLATFORM select HAVE_PERF_EVENTS select HAVE_PERF_EVENTS_NMI diff --git a/arch/x86/include/asm/current.h b/arch/x86/include/asm/current.h index bf5953883ec3..dcc2ef784120 100644 --- a/arch/x86/include/asm/current.h +++ b/arch/x86/include/asm/current.h @@ -24,6 +24,9 @@ struct pcpu_hot { unsigned long top_of_stack; void *hardirq_stack_ptr; u16 softirq_pending; +#ifdef CONFIG_PCPU_RCU_PREEMPT_COUNT + int rcu_preempt_count; +#endif // #ifdef CONFIG_PCPU_RCU_PREEMPT_COUNT #ifdef CONFIG_X86_64 bool hardirq_stack_inuse; #else diff --git a/arch/x86/include/asm/rcu_preempt.h b/arch/x86/include/asm/rcu_preempt.h new file mode 100644 index 000000000000..cb25ebe038a5 --- /dev/null +++ b/arch/x86/include/asm/rcu_preempt.h @@ -0,0 +1,107 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __ASM_RCU_PREEMPT_H +#define __ASM_RCU_PREEMPT_H + +#include +#include +#include + +#ifdef CONFIG_PCPU_RCU_PREEMPT_COUNT + +/* We use the MSB mostly because its available */ +#define RCU_PREEMPT_UNLOCK_SPECIAL_INVERTED 0x80000000 + +/* + * We use the RCU_PREEMPT_UNLOCK_SPECIAL_INVERTED bit as an inverted + * current->rcu_read_unlock_special.s such that a decrement hitting 0 + * means we can and should call rcu_read_unlock_special(). + */ +#define RCU_PREEMPT_INIT (0 + RCU_PREEMPT_UNLOCK_SPECIAL_INVERTED) + +/* + * We mask the RCU_PREEMPT_UNLOCK_SPECIAL_INVERTED bit so as not to + * confuse all current users that think a non-zero value indicates we + * are in a critical section. + */ +static inline int pcpu_rcu_preempt_count(void) +{ + return raw_cpu_read_4(pcpu_hot.rcu_preempt_count) & ~RCU_PREEMPT_UNLOCK_SPECIAL_INVERTED; +} + +static inline void pcpu_rcu_preempt_count_set(int count) +{ + int old, new; + + old = raw_cpu_read_4(pcpu_hot.rcu_preempt_count); + do { + new = (old & RCU_PREEMPT_UNLOCK_SPECIAL_INVERTED) | + (count & ~RCU_PREEMPT_UNLOCK_SPECIAL_INVERTED); + } while (!raw_cpu_try_cmpxchg_4(pcpu_hot.rcu_preempt_count, &old, new)); +} + +/* + * We fold the RCU_PREEMPT_UNLOCK_SPECIAL_INVERTED bit into the RCU + * preempt count such that rcu_read_unlock() can decrement and test for + * the need of unlock-special handling with a single instruction. + * + * We invert the actual bit, so that when the decrement hits 0 we know + * we both reach a quiescent state (no rcu preempt count) and need to + * handle unlock-special (the bit is cleared), normally to report the + * quiescent state immediately. + */ + +static inline void pcpu_rcu_preempt_special_set(void) +{ + raw_cpu_and_4(pcpu_hot.rcu_preempt_count, ~RCU_PREEMPT_UNLOCK_SPECIAL_INVERTED); +} + +static inline void pcpu_rcu_preempt_special_clear(void) +{ + raw_cpu_or_4(pcpu_hot.rcu_preempt_count, RCU_PREEMPT_UNLOCK_SPECIAL_INVERTED); +} + +static inline bool pcpu_rcu_preempt_special_test(void) +{ + return !(raw_cpu_read_4(pcpu_hot.rcu_preempt_count) & RCU_PREEMPT_UNLOCK_SPECIAL_INVERTED); +} + +static inline void pcpu_rcu_preempt_switch(int count, bool special) +{ + if (likely(!special)) + raw_cpu_write(pcpu_hot.rcu_preempt_count, count | RCU_PREEMPT_UNLOCK_SPECIAL_INVERTED); + else + raw_cpu_write(pcpu_hot.rcu_preempt_count, count); +} + +/* + * The various rcu_preempt_count add/sub methods + */ + +static __always_inline void pcpu_rcu_preempt_count_add(int val) +{ + raw_cpu_add_4(pcpu_hot.rcu_preempt_count, val); +} + +static __always_inline void pcpu_rcu_preempt_count_sub(int val) +{ + raw_cpu_add_4(pcpu_hot.rcu_preempt_count, -val); +} + +/* + * Because we keep RCU_PREEMPT_UNLOCK_SPECIAL_INVERTED set when we do + * _not_ need to handle unlock-special for a fast-path decrement. + */ +static __always_inline bool pcpu_rcu_preempt_count_dec_and_test(void) +{ + return GEN_UNARY_RMWcc("decl", __my_cpu_var(pcpu_hot.rcu_preempt_count), e, + __percpu_arg([var])); +} + +#define pcpu_rcu_read_unlock_special() \ +do { \ + rcu_read_unlock_special(); \ +} while (0) + +#endif // #ifdef CONFIG_PCPU_RCU_PREEMPT_COUNT + +#endif /* __ASM_RCU_PREEMPT_H */ diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index 5c1e6d6be267..918b1f5cb75d 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -1995,6 +1995,10 @@ DEFINE_PER_CPU_ALIGNED(struct pcpu_hot, pcpu_hot) = { .current_task = &init_task, .preempt_count = INIT_PREEMPT_COUNT, .top_of_stack = TOP_OF_INIT_STACK, + +#ifdef CONFIG_PCPU_RCU_PREEMPT_COUNT + .rcu_preempt_count = RCU_PREEMPT_INIT, +#endif // #ifdef CONFIG_PCPU_RCU_PREEMPT_COUNT }; EXPORT_PER_CPU_SYMBOL(pcpu_hot); EXPORT_PER_CPU_SYMBOL(const_pcpu_hot); From patchwork Sun Apr 7 09:05:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lai Jiangshan X-Patchwork-Id: 13620096 Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EFF661B5A4; Sun, 7 Apr 2024 09:04:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712480668; cv=none; b=TyapFc+O78+4YbvM8pN8TSo3YDuKSN5Nqqi4VfG4Z1bt8n2Dx7Al6ZUbnws4FylWrdHz8CPGsW+lJMoeN6aJegBJjIG7bYcbY3bDqllyUryPxcHeICS6dEYsw8pyOQMplSmNsg7plc+cexYnpZa490uiujFehwrxExR1XC74gTY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712480668; c=relaxed/simple; bh=IhZYosKK4YOMcbHez/fMmD2UL1qwCvnS6/0P7o8GM80=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=djd9VTuemOWPpKuTNZuj0iTV5FdcitQx8w9fPvbpV0pFMS5oyaoPucgbujh3tGsTisanBwDKb7FltslT3Cx4mWCd52YhO8QJZIv5gE2woi+N10XO8P+YZKNnW71yrcPD+wuwSCcF4lYqHPsBkBpTYnuXwaoCyHpp/T8T8xRemCY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=jNnBC+JN; arc=none smtp.client-ip=209.85.214.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="jNnBC+JN" Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-1e3f17c6491so3927195ad.2; Sun, 07 Apr 2024 02:04:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712480666; x=1713085466; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=LkIq2OM32fM/A9YPDdmXDhI8BJtLMOXuFU9dLok8GaA=; b=jNnBC+JNqzBiqk0HHJBE1z2R+nvrFM5al1tSFCs94uY77p9fwN8yUKfk7JPCI68d0x C1LTCiEB89E/CPxehd1VIQ/nrmkYPmH5Usuav/z5OmvP18AQY3AWD7ySue4dUstF4OIP tuiDgLocKVsn75i+TLtXuJRM6p61fP1ZTrpnvHPsSZUEG3EQgMwc5kUN0fbCBF81nfmA DhIX1h445S4SwdP2hdi3kw/Hdak+Yl8RanFX7UOboBk41CgaswE+gJ8yOzsuTLEM3jeS FskUviZ1kTZVsy7OQ1beVw/WYU/haDGiatZ76p4GILc/7rJ1sK39SBYXurMhOxusZjhX NdEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712480666; x=1713085466; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LkIq2OM32fM/A9YPDdmXDhI8BJtLMOXuFU9dLok8GaA=; b=m45CIo3WESkRpGbwEKRqygujIxAUXwtOwaKttrFU5UGfWjHfKanSXEHF7hOFPJLEeJ T67jYwp74go/pM+bLRSLIcVs05XlqZSBqequlD3P+67n7/p4mhv3zVdoNeKnJv/yDJKP DTWM1X6j6VE6FIVBGDFhKzLDUQfc7Gag1wzxb7Duy0xORs1eLTGx89IO/3iK0rgv3D05 AE/8PKf7FZGD7Lq24rbVhcNeMPcMGXITrRw7XXf/VYtd6x6JSKZblmE13+wdMTBNN22d Yww4PMSE6eQc1e45Tq3DL+bXf/j6cWv3MWkJ4Gs5d8yKqQ26QVBTIaaYzXtOyDhol4J+ rNsw== X-Gm-Message-State: AOJu0YyQ+fI6ukUTphnJObU/Sqfd8PniemmmtDuY5Iw9D0hH+IlbH49B 8jD5hYeSt9Fo9jcBH4wSgv9348hJMIzz/VW8pusvea59kngZCA0KB6ExGKGJ X-Google-Smtp-Source: AGHT+IF9PcRqZDnUhHy3vl6624twg2/Jp4GtKYwOmGa3ZycNF1LGiu6DrMU4VbWU9vWZoHd1u0jlIw== X-Received: by 2002:a05:6a21:a5aa:b0:1a7:647a:420e with SMTP id gd42-20020a056a21a5aa00b001a7647a420emr710671pzc.3.1712480666154; Sun, 07 Apr 2024 02:04:26 -0700 (PDT) Received: from localhost ([47.89.225.180]) by smtp.gmail.com with ESMTPSA id fa12-20020a056a002d0c00b006e694719fa0sm2412840pfb.147.2024.04.07.02.04.25 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 07 Apr 2024 02:04:25 -0700 (PDT) From: Lai Jiangshan To: linux-kernel@vger.kernel.org Cc: rcu@vger.kernel.org, x86@kernel.org, Lai Jiangshan , "Paul E. McKenney" , Peter Zijlstra , Frederic Weisbecker , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Subject: [PATCH V2 11/11] x86/rcu: Add THUNK rcu_read_unlock_special_thunk Date: Sun, 7 Apr 2024 17:05:58 +0800 Message-Id: <20240407090558.3395-12-jiangshanlai@gmail.com> X-Mailer: git-send-email 2.19.1.6.gb485710b In-Reply-To: <20240407090558.3395-1-jiangshanlai@gmail.com> References: <20240407090558.3395-1-jiangshanlai@gmail.com> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Lai Jiangshan Add rcu_read_unlock_special_thunk(), so that the inlined rcu_read_unlock() doesn't need any code to save the caller-saved registers. Make rcu_read_unlock() only two instructions in the slow path at the caller site. Cc: "Paul E. McKenney" Cc: Peter Zijlstra Cc: Frederic Weisbecker Signed-off-by: Lai Jiangshan --- arch/x86/entry/thunk.S | 5 +++++ arch/x86/include/asm/rcu_preempt.h | 4 +++- 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/arch/x86/entry/thunk.S b/arch/x86/entry/thunk.S index 119ebdc3d362..10c60369a67c 100644 --- a/arch/x86/entry/thunk.S +++ b/arch/x86/entry/thunk.S @@ -13,3 +13,8 @@ THUNK preempt_schedule_thunk, preempt_schedule THUNK preempt_schedule_notrace_thunk, preempt_schedule_notrace EXPORT_SYMBOL(preempt_schedule_thunk) EXPORT_SYMBOL(preempt_schedule_notrace_thunk) + +#ifdef CONFIG_PCPU_RCU_PREEMPT_COUNT +THUNK rcu_read_unlock_special_thunk, rcu_read_unlock_special +EXPORT_SYMBOL_GPL(rcu_read_unlock_special_thunk) +#endif /* #ifdef CONFIG_PCPU_RCU_PREEMPT_COUNT */ diff --git a/arch/x86/include/asm/rcu_preempt.h b/arch/x86/include/asm/rcu_preempt.h index cb25ebe038a5..acdd73b74c05 100644 --- a/arch/x86/include/asm/rcu_preempt.h +++ b/arch/x86/include/asm/rcu_preempt.h @@ -97,9 +97,11 @@ static __always_inline bool pcpu_rcu_preempt_count_dec_and_test(void) __percpu_arg([var])); } +extern asmlinkage void rcu_read_unlock_special_thunk(void); + #define pcpu_rcu_read_unlock_special() \ do { \ - rcu_read_unlock_special(); \ + asm volatile ("call rcu_read_unlock_special_thunk" : ASM_CALL_CONSTRAINT);\ } while (0) #endif // #ifdef CONFIG_PCPU_RCU_PREEMPT_COUNT