From patchwork Fri Mar 8 17:34:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Uladzislau Rezki (Sony)" X-Patchwork-Id: 13587139 Received: from mail-lf1-f51.google.com (mail-lf1-f51.google.com [209.85.167.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 47836568A; Fri, 8 Mar 2024 17:34:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709919256; cv=none; b=u0AhrShlM2NiEPOIy/oIyK+qOupQA+QMj6QHAXWppSqFbQ95j9DK2zuHUkM/QSNOl/mnMvR4Zi1jdL0t4y609GzklV13fz+kJJsuVVgvUZJ1VghZahUW6BSV/UKQneIqT5Yve1//4Ef+M7YZ1ifE/xxZZu3JB/eNmF+pmDN+lQU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709919256; c=relaxed/simple; bh=+sM30cOyHqXSmmhxaV2sjUAEf1nC4m+PeirwUjejQqo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=VZHnoy9lqdM+bzzL/MGRRDj9LahXpmm6SFBDu71GHbjP6cZ+1tABTlcS54RUerUpRqMyNYMAZDU9GgpOACy5T6Ci3G5xHJtPF2DurP5i5gmUDf/s+PYo0Ns8gTxPCi6vTwqRL/1B9SZ5EtzPoEepP4hT2CtOwMyhPMagmtISE8w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=DBhrncvN; arc=none smtp.client-ip=209.85.167.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="DBhrncvN" Received: by mail-lf1-f51.google.com with SMTP id 2adb3069b0e04-5131c48055cso2668960e87.1; Fri, 08 Mar 2024 09:34:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709919253; x=1710524053; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jOpDJiuEMajeibzl8V5uZUX1gqeJEBwIDoQaXh828aE=; b=DBhrncvNjUivOKrZOm7f9SmW6War2ElLSRGpgp3r/SZS/59sI1gcRx97Atw2HQXWPV iiDrOHRHI7ss+P7PX+JMMVBbK0tG3t2WWzBv5N/x5nuJPAxp5eC/9wifvGtnaC8MmoKl MAaPTFtd2akeiu05ZkPIBlqObDTz9XSTVDgWc+Q+DeCfYIfzbJS/cIsaDvTLSfIa+CR/ yrqimrcd6H3nbQezQn9Ac4mBvZ9tLiat/lccnLrbPfeQwnP1PZqyiUqjJZEMUvnbUt7I oGqVhDz8jC8Cewig4cG5exq12H24Fz2Bg71oJtn0CaGr2Tea6LVrQSotfTgjzqtebCJC suZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709919253; x=1710524053; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jOpDJiuEMajeibzl8V5uZUX1gqeJEBwIDoQaXh828aE=; b=NxKWBffEMRX+ZFcyjG86N/GpzdhdUUNrilMpQ5LR2Y7ImceWzDKA2fTQiriwP+UX2g gMxddTqDNUn+LrDgGklTpNBavz77ikKMtdEz7gALuKoBOjAeDwvNZYNcBWZf9t/z2upD rUSq9EQvO6FSgpXmPRTm/gWlkemdNWgoUkOuG5RyZMMMYmqnF0Zv1f/5j46l6Qpxj91X P61hzRZTmMhbxzhITp4mamEkHqdxVRVsDLGIKFmR2jtOp/yjEDQR+WOsKdLY2iuhOeLK KCVYUFp4lqCBHOOoU58CzDlkppVvGeAG1plRuDpGtS3wAAEJqzdekHrMw9smVa5rp8AY RCUw== X-Forwarded-Encrypted: i=1; AJvYcCU+a96gV7WLoJjfLM1Joh5xwdLZwzVAwu1orz0RRGHRoggWfMGmW0sj3E7mFJve24xz+ahtI1Ex3T9bV9q7T5yNKhJQQM+gal7wl/db X-Gm-Message-State: AOJu0YweDyGH3ERWGx/Q2vyUqenn5XsBo1UBbYftMCbyy/mx8CYDUuZE t2rHHCbxKBxRIkN323r3DxgsbHfiFVyQ4FLTGbD1wIH1C66OEs3n X-Google-Smtp-Source: AGHT+IH+PQeVlCzofdLfMc8Gj/FLZaAGqr6IVOI8KbixI6Z9BFsnX/bMxCChXoNFbGLsem/FB+rV/g== X-Received: by 2002:ac2:4289:0:b0:513:1385:c943 with SMTP id m9-20020ac24289000000b005131385c943mr3475035lfh.40.1709919253367; Fri, 08 Mar 2024 09:34:13 -0800 (PST) Received: from pc638.lan (host-185-121-47-193.sydskane.nu. [185.121.47.193]) by smtp.gmail.com with ESMTPSA id a23-20020a195f57000000b0051333e2f55dsm3150504lfj.159.2024.03.08.09.34.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Mar 2024 09:34:13 -0800 (PST) From: "Uladzislau Rezki (Sony)" To: "Paul E . McKenney" Cc: RCU , Neeraj upadhyay , Boqun Feng , Hillf Danton , Joel Fernandes , LKML , Uladzislau Rezki , Oleksiy Avramchenko , Frederic Weisbecker Subject: [PATCH v6 1/6] rcu: Add data structures for synchronize_rcu() Date: Fri, 8 Mar 2024 18:34:04 +0100 Message-Id: <20240308173409.335345-2-urezki@gmail.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240308173409.335345-1-urezki@gmail.com> References: <20240308173409.335345-1-urezki@gmail.com> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 The synchronize_rcu() call is going to be reworked, thus this patch adds dedicated fields into the rcu_state structure. Signed-off-by: Uladzislau Rezki (Sony) --- kernel/rcu/tree.h | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h index df48160b3136..b942b9437438 100644 --- a/kernel/rcu/tree.h +++ b/kernel/rcu/tree.h @@ -315,6 +315,13 @@ do { \ __set_current_state(TASK_RUNNING); \ } while (0) +#define SR_NORMAL_GP_WAIT_HEAD_MAX 5 + +struct sr_wait_node { + atomic_t inuse; + struct llist_node node; +}; + /* * RCU global state, including node hierarchy. This hierarchy is * represented in "heap" form in a dense array. The root (first level) @@ -400,6 +407,13 @@ struct rcu_state { /* Synchronize offline with */ /* GP pre-initialization. */ int nocb_is_setup; /* nocb is setup from boot */ + + /* synchronize_rcu() part. */ + struct llist_head srs_next; /* request a GP users. */ + struct llist_node *srs_wait_tail; /* wait for GP users. */ + struct llist_node *srs_done_tail; /* ready for GP users. */ + struct sr_wait_node srs_wait_nodes[SR_NORMAL_GP_WAIT_HEAD_MAX]; + struct work_struct srs_cleanup_work; }; /* Values for rcu_state structure's gp_flags field. */ From patchwork Fri Mar 8 17:34:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Uladzislau Rezki (Sony)" X-Patchwork-Id: 13587140 Received: from mail-lf1-f51.google.com (mail-lf1-f51.google.com [209.85.167.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 86AF43309E; Fri, 8 Mar 2024 17:34:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709919259; cv=none; b=ufhwMpMKXDbuiUSgnMQTDcSigGk8fWo8BA9Z2K37X/ufADIa9BgOC9AWwtCaJovE7Hx10Y6YPP0Zr/o7sfK4ld3HlWploiZADuBTpQdKK00H9GPLCbs+AD24OjSMPRMtijnt0BL+HbWXtbTC0vn51lMujcPWNFzcEQ/u9Cl2wH0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709919259; c=relaxed/simple; bh=vMyTuQgrNDK9laSjDAGcqYxayeYM2fB6YaZpcst67PA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=OM0tZXXMDcDD5o3TgRuyXenimYGQR0PaZkia4aSifu9PnzYVUHhSde0IuyPOr0kYYXpGaW9m2X5/qe9mGKUEOkGKT6KmMy1eE/yc7D3z/UZ7LSX1IQ/TmQRC0KUzv0LkFWUfQJ2WLDFqW0AMILMImsriKX1/8RuRiF/ldZfYMGQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=V8JLy93k; arc=none smtp.client-ip=209.85.167.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="V8JLy93k" Received: by mail-lf1-f51.google.com with SMTP id 2adb3069b0e04-51325c38d10so3317952e87.1; Fri, 08 Mar 2024 09:34:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709919255; x=1710524055; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BoVvGEQ3KdyN70gq1OxGDPTrqn4ioezIrJ4KB8Ijm20=; b=V8JLy93kom2/S6yny7Y8OYQOm3IqrG81SGK0dIGLWNUurQMImVQkyLd0DmCyFVdW3O ilPv57luDt+CjHtYDsjViz4GxdPm2vD8HtU4h2Aaq2lBC2L93RguP5ttElS+qs1/h851 0u5GeEJpOMUd1lK/QP1SiDDEMs7fmqwb90dloi7VJQDLj4qOUmpSbjQGnIaEURBtLbwe MRe2we+3lQ2GaQmynMsefYXGccI2VjhD+IqmFK93n/VZNGQ7H8bTKq3Nv+dBcFTGgWrB me+dMGVNznQrWpAuai5fEYFOXTLCGcLIIuy10EVGaZ0IX52t0ZSosH6EVwjDyg/DMU4C 4xiA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709919255; x=1710524055; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BoVvGEQ3KdyN70gq1OxGDPTrqn4ioezIrJ4KB8Ijm20=; b=mdhcnIZdKJbfCgf1x4EMLZWfyNIEvyl0rAWiu1nDh/OwntNerJvE+b68KvByS+BKT8 I9aXNoMcrC2xpSEVFUe9R/vgCr8oxvcy+SyuZwFVL1SBFOFIc11dzphXaQs6zm0Udvkb nOZ8IAp9Ywiy78HhZL8KfvyCisqqgbXID0o4rdwmOLuznt7KQ7vKdlgH6C0SlSwGJCXD Up/tYtMLiHCvICmPycjlf0jI9hxJxKxNnfIleJYQXv615rCgnIjA+VS3RbZQ4D3OWUyf 3hlvTavm2mtkZ8jdFEJgSw9HJWUhvopbW0uO2n/JavLMHc/wD6wNtG71jYyCzUj+rU0X l6DA== X-Forwarded-Encrypted: i=1; AJvYcCXhU1q9XkxAlgbGuw5yFAeBFPLPwdZVBYSqHDdm6Gy9RsFRZZJKbOf+yBqK/qAYyv3cKz1SLgGBSodD87idL+eRywkMZN/mYav3Q1So X-Gm-Message-State: AOJu0YzftA/1yYRsQojqZcZ+AmafSX5yzfizDhkIz9tF2PFi+OeTOmqT w0xshW31Cy2LJIX11VD9Wh1h+fKx3w1aMtGQ1SUFQ+8ZhR1i/fgXJXESN8zWofQ= X-Google-Smtp-Source: AGHT+IGrP7fiUCsZvn5l+Ul9AL/ZO/m9HsNMe3vnHzNX8QTB26WFlrxEfOn1r9s1z8SwGPDFMlAS+w== X-Received: by 2002:ac2:55aa:0:b0:513:4105:fd7f with SMTP id y10-20020ac255aa000000b005134105fd7fmr293553lfg.6.1709919254293; Fri, 08 Mar 2024 09:34:14 -0800 (PST) Received: from pc638.lan (host-185-121-47-193.sydskane.nu. [185.121.47.193]) by smtp.gmail.com with ESMTPSA id a23-20020a195f57000000b0051333e2f55dsm3150504lfj.159.2024.03.08.09.34.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Mar 2024 09:34:13 -0800 (PST) From: "Uladzislau Rezki (Sony)" To: "Paul E . McKenney" Cc: RCU , Neeraj upadhyay , Boqun Feng , Hillf Danton , Joel Fernandes , LKML , Uladzislau Rezki , Oleksiy Avramchenko , Frederic Weisbecker Subject: [PATCH v6 2/6] rcu: Reduce synchronize_rcu() latency Date: Fri, 8 Mar 2024 18:34:05 +0100 Message-Id: <20240308173409.335345-3-urezki@gmail.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240308173409.335345-1-urezki@gmail.com> References: <20240308173409.335345-1-urezki@gmail.com> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 A call to a synchronize_rcu() can be optimized from a latency point of view. Workloads which depend on this can benefit of it. The delay of wakeme_after_rcu() callback, which unblocks a waiter, depends on several factors: - how fast a process of offloading is started. Combination of: - !CONFIG_RCU_NOCB_CPU/CONFIG_RCU_NOCB_CPU; - !CONFIG_RCU_LAZY/CONFIG_RCU_LAZY; - other. - when started, invoking path is interrupted due to: - time limit; - need_resched(); - if limit is reached. - where in a nocb list it is located; - how fast previous callbacks completed; Example: 1. On our embedded devices i can easily trigger the scenario when it is a last in the list out of ~3600 callbacks: <...>-29 [001] d..1. 21950.145313: rcu_batch_start: rcu_preempt CBs=3613 bl=28 ... <...>-29 [001] ..... 21950.152578: rcu_invoke_callback: rcu_preempt rhp=00000000b2d6dee8 func=__free_vm_area_struct.cfi_jt <...>-29 [001] ..... 21950.152579: rcu_invoke_callback: rcu_preempt rhp=00000000a446f607 func=__free_vm_area_struct.cfi_jt <...>-29 [001] ..... 21950.152580: rcu_invoke_callback: rcu_preempt rhp=00000000a5cab03b func=__free_vm_area_struct.cfi_jt <...>-29 [001] ..... 21950.152581: rcu_invoke_callback: rcu_preempt rhp=0000000013b7e5ee func=__free_vm_area_struct.cfi_jt <...>-29 [001] ..... 21950.152582: rcu_invoke_callback: rcu_preempt rhp=000000000a8ca6f9 func=__free_vm_area_struct.cfi_jt <...>-29 [001] ..... 21950.152583: rcu_invoke_callback: rcu_preempt rhp=000000008f162ca8 func=wakeme_after_rcu.cfi_jt <...>-29 [001] d..1. 21950.152625: rcu_batch_end: rcu_preempt CBs-invoked=3612 idle=.... 2. We use cpuset/cgroup to classify tasks and assign them into different cgroups. For example "backgrond" group which binds tasks only to little CPUs or "foreground" which makes use of all CPUs. Tasks can be migrated between groups by a request if an acceleration is needed. See below an example how "surfaceflinger" task gets migrated. Initially it is located in the "system-background" cgroup which allows to run only on little cores. In order to speed it up it can be temporary moved into "foreground" cgroup which allows to use big/all CPUs: cgroup_attach_task(): -> cgroup_migrate_execute() -> cpuset_can_attach() -> percpu_down_write() -> rcu_sync_enter() -> synchronize_rcu() -> now move tasks to the new cgroup. -> cgroup_migrate_finish() rcuop/1-29 [000] ..... 7030.528570: rcu_invoke_callback: rcu_preempt rhp=00000000461605e0 func=wakeme_after_rcu.cfi_jt PERFD-SERVER-1855 [000] d..1. 7030.530293: cgroup_attach_task: dst_root=3 dst_id=22 dst_level=1 dst_path=/foreground pid=1900 comm=surfaceflinger TimerDispatch-2768 [002] d..5. 7030.537542: sched_migrate_task: comm=surfaceflinger pid=1900 prio=98 orig_cpu=0 dest_cpu=4 "Boosting a task" depends on synchronize_rcu() latency: - first trace shows a completion of synchronize_rcu(); - second shows attaching a task to a new group; - last shows a final step when migration occurs. 3. To address this drawback, maintain a separate track that consists of synchronize_rcu() callers only. After completion of a grace period users are deferred to a dedicated worker to process requests. 4. This patch reduces the latency of synchronize_rcu() approximately by ~30-40% on synthetic tests. The real test case, camera launch time, shows(time is in milliseconds): 1-run 542 vs 489 improvement 9% 2-run 540 vs 466 improvement 13% 3-run 518 vs 468 improvement 9% 4-run 531 vs 457 improvement 13% 5-run 548 vs 475 improvement 13% 6-run 509 vs 484 improvement 4% Synthetic test(no "noise" from other callbacks): Hardware: x86_64 64 CPUs, 64GB of memory Linux-6.6 - 10K tasks(simultaneous); - each task does(1000 loops) synchronize_rcu(); kfree(p); default: CONFIG_RCU_NOCB_CPU: takes 54 seconds to complete all users; patch: CONFIG_RCU_NOCB_CPU: takes 35 seconds to complete all users. Running 60K gives approximately same results on my setup. Please note it is without any interaction with another type of callbacks, otherwise it will impact a lot a default case. 5. By default it is disabled. To enable this perform one of the below sequence: echo 1 > /sys/module/rcutree/parameters/rcu_normal_wake_from_gp or pass a boot parameter "rcutree.rcu_normal_wake_from_gp=1" Reviewed-by: Frederic Weisbecker Co-developed-by: Neeraj Upadhyay Signed-off-by: Uladzislau Rezki (Sony) --- .../admin-guide/kernel-parameters.txt | 14 + kernel/rcu/tree.c | 330 +++++++++++++++++- kernel/rcu/tree_exp.h | 2 +- 3 files changed, 344 insertions(+), 2 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 94314d0eb301..3f894fbb4916 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -5052,6 +5052,20 @@ delay, memory pressure or callback list growing too big. + rcutree.rcu_normal_wake_from_gp= [KNL] + Reduces a latency of synchronize_rcu() call. This approach + maintains its own track of synchronize_rcu() callers, so it + does not interact with regular callbacks because it does not + use a call_rcu[_hurry]() path. Please note, this is for a + normal grace period. + + How to enable it: + + echo 1 > /sys/module/rcutree/parameters/rcu_normal_wake_from_gp + or pass a boot parameter "rcutree.rcu_normal_wake_from_gp=1" + + Default is 0. + rcuscale.gp_async= [KNL] Measure performance of asynchronous grace-period primitives such as call_rcu(). diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 559f2d0d271f..42a31d4d279b 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -75,6 +75,7 @@ #define MODULE_PARAM_PREFIX "rcutree." /* Data structures. */ +static void rcu_sr_normal_gp_cleanup_work(struct work_struct *); static DEFINE_PER_CPU_SHARED_ALIGNED(struct rcu_data, rcu_data) = { .gpwrap = true, @@ -93,6 +94,8 @@ static struct rcu_state rcu_state = { .exp_mutex = __MUTEX_INITIALIZER(rcu_state.exp_mutex), .exp_wake_mutex = __MUTEX_INITIALIZER(rcu_state.exp_wake_mutex), .ofl_lock = __ARCH_SPIN_LOCK_UNLOCKED, + .srs_cleanup_work = __WORK_INITIALIZER(rcu_state.srs_cleanup_work, + rcu_sr_normal_gp_cleanup_work), }; /* Dump rcu_node combining tree at boot to verify correct setup. */ @@ -1450,6 +1453,281 @@ static void rcu_poll_gp_seq_end_unlocked(unsigned long *snap) raw_spin_unlock_irqrestore_rcu_node(rnp, flags); } +/* + * There is a single llist, which is used for handling + * synchronize_rcu() users' enqueued rcu_synchronize nodes. + * Within this llist, there are two tail pointers: + * + * wait tail: Tracks the set of nodes, which need to + * wait for the current GP to complete. + * done tail: Tracks the set of nodes, for which grace + * period has elapsed. These nodes processing + * will be done as part of the cleanup work + * execution by a kworker. + * + * At every grace period init, a new wait node is added + * to the llist. This wait node is used as wait tail + * for this new grace period. Given that there are a fixed + * number of wait nodes, if all wait nodes are in use + * (which can happen when kworker callback processing + * is delayed) and additional grace period is requested. + * This means, a system is slow in processing callbacks. + * + * TODO: If a slow processing is detected, a first node + * in the llist should be used as a wait-tail for this + * grace period, therefore users which should wait due + * to a slow process are handled by _this_ grace period + * and not next. + * + * Below is an illustration of how the done and wait + * tail pointers move from one set of rcu_synchronize nodes + * to the other, as grace periods start and finish and + * nodes are processed by kworker. + * + * + * a. Initial llist callbacks list: + * + * +----------+ +--------+ +-------+ + * | | | | | | + * | head |---------> | cb2 |--------->| cb1 | + * | | | | | | + * +----------+ +--------+ +-------+ + * + * + * + * b. New GP1 Start: + * + * WAIT TAIL + * | + * | + * v + * +----------+ +--------+ +--------+ +-------+ + * | | | | | | | | + * | head ------> wait |------> cb2 |------> | cb1 | + * | | | head1 | | | | | + * +----------+ +--------+ +--------+ +-------+ + * + * + * + * c. GP completion: + * + * WAIT_TAIL == DONE_TAIL + * + * DONE TAIL + * | + * | + * v + * +----------+ +--------+ +--------+ +-------+ + * | | | | | | | | + * | head ------> wait |------> cb2 |------> | cb1 | + * | | | head1 | | | | | + * +----------+ +--------+ +--------+ +-------+ + * + * + * + * d. New callbacks and GP2 start: + * + * WAIT TAIL DONE TAIL + * | | + * | | + * v v + * +----------+ +------+ +------+ +------+ +-----+ +-----+ +-----+ + * | | | | | | | | | | | | | | + * | head ------> wait |--->| cb4 |--->| cb3 |--->|wait |--->| cb2 |--->| cb1 | + * | | | head2| | | | | |head1| | | | | + * +----------+ +------+ +------+ +------+ +-----+ +-----+ +-----+ + * + * + * + * e. GP2 completion: + * + * WAIT_TAIL == DONE_TAIL + * DONE TAIL + * | + * | + * v + * +----------+ +------+ +------+ +------+ +-----+ +-----+ +-----+ + * | | | | | | | | | | | | | | + * | head ------> wait |--->| cb4 |--->| cb3 |--->|wait |--->| cb2 |--->| cb1 | + * | | | head2| | | | | |head1| | | | | + * +----------+ +------+ +------+ +------+ +-----+ +-----+ +-----+ + * + * + * While the llist state transitions from d to e, a kworker + * can start executing rcu_sr_normal_gp_cleanup_work() and + * can observe either the old done tail (@c) or the new + * done tail (@e). So, done tail updates and reads need + * to use the rel-acq semantics. If the concurrent kworker + * observes the old done tail, the newly queued work + * execution will process the updated done tail. If the + * concurrent kworker observes the new done tail, then + * the newly queued work will skip processing the done + * tail, as workqueue semantics guarantees that the new + * work is executed only after the previous one completes. + * + * f. kworker callbacks processing complete: + * + * + * DONE TAIL + * | + * | + * v + * +----------+ +--------+ + * | | | | + * | head ------> wait | + * | | | head2 | + * +----------+ +--------+ + * + */ +static bool rcu_sr_is_wait_head(struct llist_node *node) +{ + return &(rcu_state.srs_wait_nodes)[0].node <= node && + node <= &(rcu_state.srs_wait_nodes)[SR_NORMAL_GP_WAIT_HEAD_MAX - 1].node; +} + +static struct llist_node *rcu_sr_get_wait_head(void) +{ + struct sr_wait_node *sr_wn; + int i; + + for (i = 0; i < SR_NORMAL_GP_WAIT_HEAD_MAX; i++) { + sr_wn = &(rcu_state.srs_wait_nodes)[i]; + + if (!atomic_cmpxchg_acquire(&sr_wn->inuse, 0, 1)) + return &sr_wn->node; + } + + return NULL; +} + +static void rcu_sr_put_wait_head(struct llist_node *node) +{ + struct sr_wait_node *sr_wn = container_of(node, struct sr_wait_node, node); + atomic_set_release(&sr_wn->inuse, 0); +} + +/* Disabled by default. */ +static int rcu_normal_wake_from_gp; +module_param(rcu_normal_wake_from_gp, int, 0644); + +static void rcu_sr_normal_complete(struct llist_node *node) +{ + struct rcu_synchronize *rs = container_of( + (struct rcu_head *) node, struct rcu_synchronize, head); + unsigned long oldstate = (unsigned long) rs->head.func; + + WARN_ONCE(IS_ENABLED(CONFIG_PROVE_RCU) && + !poll_state_synchronize_rcu(oldstate), + "A full grace period is not passed yet: %lu", + rcu_seq_diff(get_state_synchronize_rcu(), oldstate)); + + /* Finally. */ + complete(&rs->completion); +} + +static void rcu_sr_normal_gp_cleanup_work(struct work_struct *work) +{ + struct llist_node *done, *rcu, *next, *head; + + /* + * This work execution can potentially execute + * while a new done tail is being updated by + * grace period kthread in rcu_sr_normal_gp_cleanup(). + * So, read and updates of done tail need to + * follow acq-rel semantics. + * + * Given that wq semantics guarantees that a single work + * cannot execute concurrently by multiple kworkers, + * the done tail list manipulations are protected here. + */ + done = smp_load_acquire(&rcu_state.srs_done_tail); + if (!done) + return; + + WARN_ON_ONCE(!rcu_sr_is_wait_head(done)); + head = done->next; + done->next = NULL; + + /* + * The dummy node, which is pointed to by the + * done tail which is acq-read above is not removed + * here. This allows lockless additions of new + * rcu_synchronize nodes in rcu_sr_normal_add_req(), + * while the cleanup work executes. The dummy + * nodes is removed, in next round of cleanup + * work execution. + */ + llist_for_each_safe(rcu, next, head) { + if (!rcu_sr_is_wait_head(rcu)) { + rcu_sr_normal_complete(rcu); + continue; + } + + rcu_sr_put_wait_head(rcu); + } +} + +/* + * Helper function for rcu_gp_cleanup(). + */ +static void rcu_sr_normal_gp_cleanup(void) +{ + struct llist_node *wait_tail; + + wait_tail = rcu_state.srs_wait_tail; + if (wait_tail == NULL) + return; + + rcu_state.srs_wait_tail = NULL; + ASSERT_EXCLUSIVE_WRITER(rcu_state.srs_wait_tail); + + // concurrent sr_normal_gp_cleanup work might observe this update. + smp_store_release(&rcu_state.srs_done_tail, wait_tail); + ASSERT_EXCLUSIVE_WRITER(rcu_state.srs_done_tail); + + schedule_work(&rcu_state.srs_cleanup_work); +} + +/* + * Helper function for rcu_gp_init(). + */ +static bool rcu_sr_normal_gp_init(void) +{ + struct llist_node *first; + struct llist_node *wait_head; + bool start_new_poll = false; + + first = READ_ONCE(rcu_state.srs_next.first); + if (!first || rcu_sr_is_wait_head(first)) + return start_new_poll; + + wait_head = rcu_sr_get_wait_head(); + if (!wait_head) { + // Kick another GP to retry. + start_new_poll = true; + return start_new_poll; + } + + /* Inject a wait-dummy-node. */ + llist_add(wait_head, &rcu_state.srs_next); + + /* + * A waiting list of rcu_synchronize nodes should be empty on + * this step, since a GP-kthread, rcu_gp_init() -> gp_cleanup(), + * rolls it over. If not, it is a BUG, warn a user. + */ + WARN_ON_ONCE(rcu_state.srs_wait_tail != NULL); + rcu_state.srs_wait_tail = wait_head; + ASSERT_EXCLUSIVE_WRITER(rcu_state.srs_wait_tail); + + return start_new_poll; +} + +static void rcu_sr_normal_add_req(struct rcu_synchronize *rs) +{ + llist_add((struct llist_node *) &rs->head, &rcu_state.srs_next); +} + /* * Initialize a new grace period. Return false if no grace period required. */ @@ -1460,6 +1738,7 @@ static noinline_for_stack bool rcu_gp_init(void) unsigned long mask; struct rcu_data *rdp; struct rcu_node *rnp = rcu_get_root(); + bool start_new_poll; WRITE_ONCE(rcu_state.gp_activity, jiffies); raw_spin_lock_irq_rcu_node(rnp); @@ -1484,10 +1763,24 @@ static noinline_for_stack bool rcu_gp_init(void) /* Record GP times before starting GP, hence rcu_seq_start(). */ rcu_seq_start(&rcu_state.gp_seq); ASSERT_EXCLUSIVE_WRITER(rcu_state.gp_seq); + start_new_poll = rcu_sr_normal_gp_init(); trace_rcu_grace_period(rcu_state.name, rcu_state.gp_seq, TPS("start")); rcu_poll_gp_seq_start(&rcu_state.gp_seq_polled_snap); raw_spin_unlock_irq_rcu_node(rnp); + /* + * The "start_new_poll" is set to true, only when this GP is not able + * to handle anything and there are outstanding users. It happens when + * the rcu_sr_normal_gp_init() function was not able to insert a dummy + * separator to the llist, because there were no left any dummy-nodes. + * + * Number of dummy-nodes is fixed, it could be that we are run out of + * them, if so we start a new pool request to repeat a try. It is rare + * and it means that a system is doing a slow processing of callbacks. + */ + if (start_new_poll) + (void) start_poll_synchronize_rcu(); + /* * Apply per-leaf buffered online and offline operations to * the rcu_node tree. Note that this new grace period need not @@ -1852,6 +2145,9 @@ static noinline void rcu_gp_cleanup(void) } raw_spin_unlock_irq_rcu_node(rnp); + // Make synchronize_rcu() users aware of the end of old grace period. + rcu_sr_normal_gp_cleanup(); + // If strict, make all CPUs aware of the end of the old grace period. if (IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD)) on_each_cpu(rcu_strict_gp_boundary, NULL, 0); @@ -3584,6 +3880,38 @@ static int rcu_blocking_is_gp(void) return true; } +/* + * Helper function for the synchronize_rcu() API. + */ +static void synchronize_rcu_normal(void) +{ + struct rcu_synchronize rs; + + if (!READ_ONCE(rcu_normal_wake_from_gp)) { + wait_rcu_gp(call_rcu_hurry); + return; + } + + init_rcu_head_on_stack(&rs.head); + init_completion(&rs.completion); + + /* + * This code might be preempted, therefore take a GP + * snapshot before adding a request. + */ + if (IS_ENABLED(CONFIG_PROVE_RCU)) + rs.head.func = (void *) get_state_synchronize_rcu(); + + rcu_sr_normal_add_req(&rs); + + /* Kick a GP and start waiting. */ + (void) start_poll_synchronize_rcu(); + + /* Now we can wait. */ + wait_for_completion(&rs.completion); + destroy_rcu_head_on_stack(&rs.head); +} + /** * synchronize_rcu - wait until a grace period has elapsed. * @@ -3635,7 +3963,7 @@ void synchronize_rcu(void) if (rcu_gp_is_expedited()) synchronize_rcu_expedited(); else - wait_rcu_gp(call_rcu_hurry); + synchronize_rcu_normal(); return; } diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h index 6b83537480b1..8a1d9c8bd9f7 100644 --- a/kernel/rcu/tree_exp.h +++ b/kernel/rcu/tree_exp.h @@ -930,7 +930,7 @@ void synchronize_rcu_expedited(void) /* If expedited grace periods are prohibited, fall back to normal. */ if (rcu_gp_is_normal()) { - wait_rcu_gp(call_rcu_hurry); + synchronize_rcu_normal(); return; } From patchwork Fri Mar 8 17:34:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Uladzislau Rezki (Sony)" X-Patchwork-Id: 13587141 Received: from mail-lf1-f52.google.com (mail-lf1-f52.google.com [209.85.167.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 56B6C3FBB9; Fri, 8 Mar 2024 17:34:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709919259; cv=none; b=qrpkgO7qEoQzqmV/aT3EWyU0wkbY+dU5b97fiKEVei0RTTJQ/Sefbl5QPKZ8qdnIMKm0Eu8CCvqslV8dRJPyb20VKsyQ9lrmBYCgTHEjI9s5QrZ5GoTsDFYPwIStS4vh6oE+VFWnvOQoqtFfMgoNCkdLNkDsWvbzo1QPZ71gIRU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709919259; c=relaxed/simple; bh=i9xh4PFq53VVHMU9LfrMaittt3a26iMVfWpygIZ7oew=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Q9LgwEuwlX0+/ZSzc7niKoHxkYr4HZz60ate8TNToHvesrV3GQ86vHuJZuT/w16XcayphvZs2nODrUeGPllbt3O6e9oQWCLTqq97IJUSn4J7VDJkbKE79ROq2FYsDhrolU6mUFThpJ0maOAewS/GqZeYRo2VfDvkNzBrjMMbOo8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=FZ/O0FuU; arc=none smtp.client-ip=209.85.167.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="FZ/O0FuU" Received: by mail-lf1-f52.google.com with SMTP id 2adb3069b0e04-5132181d54bso3164987e87.3; Fri, 08 Mar 2024 09:34:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709919255; x=1710524055; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=C/cUkmoDb/vMEZdBp6an+/laLEoOrf8mMtp4dQwMC1s=; b=FZ/O0FuU5oAzAxwhsFj6aiuBMmWC8dV75plF0OetUMKVUnZELCKbxHshZmuCwQ5hyZ YNKYme20UMEAy3mp0OWna9jd5yhqj1P7KhEQouRHd+na21yAo6Wp+KO1Y//dhEGTArL+ 2QNwkMCgPhFAloSSoG1GLp59iFmv5GryeFpBPbV/wLekOpNmI4k1V/RhxLBE5co0AtkK IzLPmKTCfSMfPFwy7kU0+JXv4SubhhGuY3iwq+oEgH14AIwv2Uoyu2wqr5JawNhn0FC4 i7h+FaeSWgBiEU4Jvz8R04a258n2JHqWQA8e/hq4Kkfb4Ul/NTSgsSc8KUlZP9EPuT3M WJCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709919255; x=1710524055; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=C/cUkmoDb/vMEZdBp6an+/laLEoOrf8mMtp4dQwMC1s=; b=pFOkKG5d5XB/8ytb0iTlW2gTOWDzwsnwuxizWegdp2ndOmCp6CPKOOdySkiyKZVenL Rr36xBn8q6JB6pU/aYFXoThNb+8ZQZW2i9fgo7Lg+QOHB9+idhT1+ubBetsRHPkgKvXR +jBSVii7qrPmMEKBuMwOeUpqpUs+OHzO7d14MC5+hQSm82sD8zQHYimPCYZPYQUpTXZJ fEMfUzTG4Rd7qTiFh8TWVTWf9rE6wMZRDtlTnG06BE4+HwNaNcU3SA1++B17n6f8+bj/ lUQTDWa8p3dj6f+iRyOKh8lM+XUSpH79z6XAfL1jfj8wjPCbdS07RBVLPLFkqHgU6w6M +x2Q== X-Forwarded-Encrypted: i=1; AJvYcCXyiFxtSesl8Fkpkj0lFcq5HMPieKPYYtv2dks0R7cR7npgjDjRZQScWePytgocReftWrdcID5zUpt2UN6BdmGZqgfpkTvxBe40Sdqa X-Gm-Message-State: AOJu0YwXKLRFqRaxjSZwh9LRLW606kjvpGDMJCUwx5Y1VO2/deJ7CWz6 ByFLagLKC3byo23hOzGNtYvUTiTw95ZIkAk7co2bxIycflssi617 X-Google-Smtp-Source: AGHT+IHsyDLgEOxeS/q3et3wykWVMvudFD9nQnMwi9PEaLRJ7xOrTOkUhTZ/eH2T/UqD6XoP2nTnvw== X-Received: by 2002:a19:c20c:0:b0:512:bb47:967c with SMTP id l12-20020a19c20c000000b00512bb47967cmr3382421lfc.19.1709919255167; Fri, 08 Mar 2024 09:34:15 -0800 (PST) Received: from pc638.lan (host-185-121-47-193.sydskane.nu. [185.121.47.193]) by smtp.gmail.com with ESMTPSA id a23-20020a195f57000000b0051333e2f55dsm3150504lfj.159.2024.03.08.09.34.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Mar 2024 09:34:14 -0800 (PST) From: "Uladzislau Rezki (Sony)" To: "Paul E . McKenney" Cc: RCU , Neeraj upadhyay , Boqun Feng , Hillf Danton , Joel Fernandes , LKML , Uladzislau Rezki , Oleksiy Avramchenko , Frederic Weisbecker Subject: [PATCH v6 3/6] rcu: Add a trace event for synchronize_rcu_normal() Date: Fri, 8 Mar 2024 18:34:06 +0100 Message-Id: <20240308173409.335345-4-urezki@gmail.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240308173409.335345-1-urezki@gmail.com> References: <20240308173409.335345-1-urezki@gmail.com> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Add an rcu_sr_normal() trace event. It takes three arguments first one is the name of RCU flavour, second one is a user id which triggeres synchronize_rcu_normal() and last one is an event. There are two traces in the synchronize_rcu_normal(). On entry, when a new request is registered and on exit point when request is completed. Please note, CONFIG_RCU_TRACE=y is required to activate traces. Signed-off-by: Uladzislau Rezki (Sony) --- include/trace/events/rcu.h | 27 +++++++++++++++++++++++++++ kernel/rcu/tree.c | 7 ++++++- 2 files changed, 33 insertions(+), 1 deletion(-) diff --git a/include/trace/events/rcu.h b/include/trace/events/rcu.h index 2ef9c719772a..31b3e0d3e65f 100644 --- a/include/trace/events/rcu.h +++ b/include/trace/events/rcu.h @@ -707,6 +707,33 @@ TRACE_EVENT_RCU(rcu_invoke_kfree_bulk_callback, __entry->rcuname, __entry->p, __entry->nr_records) ); +/* + * Tracepoint for a normal synchronize_rcu() states. The first argument + * is the RCU flavor, the second argument is a pointer to rcu_head the + * last one is an event. + */ +TRACE_EVENT_RCU(rcu_sr_normal, + + TP_PROTO(const char *rcuname, struct rcu_head *rhp, const char *srevent), + + TP_ARGS(rcuname, rhp, srevent), + + TP_STRUCT__entry( + __field(const char *, rcuname) + __field(void *, rhp) + __field(const char *, srevent) + ), + + TP_fast_assign( + __entry->rcuname = rcuname; + __entry->rhp = rhp; + __entry->srevent = srevent; + ), + + TP_printk("%s rhp=0x%p event=%s", + __entry->rcuname, __entry->rhp, __entry->srevent) +); + /* * Tracepoint for exiting rcu_do_batch after RCU callbacks have been * invoked. The first argument is the name of the RCU flavor, diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 42a31d4d279b..dcc1764e8fad 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -3887,9 +3887,11 @@ static void synchronize_rcu_normal(void) { struct rcu_synchronize rs; + trace_rcu_sr_normal(rcu_state.name, &rs.head, TPS("request")); + if (!READ_ONCE(rcu_normal_wake_from_gp)) { wait_rcu_gp(call_rcu_hurry); - return; + goto trace_complete_out; } init_rcu_head_on_stack(&rs.head); @@ -3910,6 +3912,9 @@ static void synchronize_rcu_normal(void) /* Now we can wait. */ wait_for_completion(&rs.completion); destroy_rcu_head_on_stack(&rs.head); + +trace_complete_out: + trace_rcu_sr_normal(rcu_state.name, &rs.head, TPS("complete")); } /** From patchwork Fri Mar 8 17:34:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Uladzislau Rezki (Sony)" X-Patchwork-Id: 13587142 Received: from mail-lf1-f49.google.com (mail-lf1-f49.google.com [209.85.167.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D8624495CC; Fri, 8 Mar 2024 17:34:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709919259; cv=none; b=rKgWZAI2dK1i4T/2gvXuIvkD6FAfA8tskQ5MAEf424oqMlz7b3BTkUNw86Ig38c7jlD3DTDfb3UDsvcSB8s2H0WupqA3YPAJVfarOh4WzJkN5sGjDzy5MHcpeOuijhN58GUvxAr/yxWs6Hh9+NXe2yux+H4SPjyBO0vydPsjWmw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709919259; c=relaxed/simple; bh=pUOWqKH8F0AZqFoW5l76mY0as9X0NZxjquxwH8plNXs=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=IczVtfMs4eLOzQ4osQUncfq4OhXDoI6uCJbyM3Bi9qu/LEu0NzDBFJ1zTfea3pXwbn+5DRZRCivkgl3ScH0WoP3eqILZeZsWYllOmzb33w/gPVnJ2PhajIeN7sgyc3G9qAx6ZC0oz1ynmRlG8nDxcCV5pfhn6kpIRhvxJAs9v2M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=izCSQZNy; arc=none smtp.client-ip=209.85.167.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="izCSQZNy" Received: by mail-lf1-f49.google.com with SMTP id 2adb3069b0e04-51331634948so3198585e87.0; Fri, 08 Mar 2024 09:34:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709919256; x=1710524056; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VJXBLMujukXP0aFm/GcWV39azCd58NyPFINXJlQ3LLY=; b=izCSQZNya9ere1OjKW/lmF537VUYhHimpqfXsF0HHZv+oYu2FdiE/V6jtJIfwj6L4g Lax3FHt9gKDVERiU7za0z2/whO+VV8KMOoOrfDzkztgj6YpAI3SI6msADsjFPO/K9P85 vtrGNWd25aTnSSYySKwmIpbFLMb6nk/Q5tIiDo8r9oZt51YczpFtp4qyuLzvhqvW3L42 3QfLCvHTpGgA2HzymPClOL+tae5c3B9krG6IhNwKJ5Lf3mXr+NFtujIGvEXFRYT4+Wvr ezNmm5ke277EIn1LgWP2IlBwSac/cKSxupBIH3XCETv7WAsCHvXkhLasI+FfCtjTudgH adrg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709919256; x=1710524056; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VJXBLMujukXP0aFm/GcWV39azCd58NyPFINXJlQ3LLY=; b=ebvycSz3b6sjOVieP6JK7aibjQ6uwv001h7Eskea/o/3h/r3D4UfNvDq2zOkvnMCY1 7EYhSurRAcfwlQdMe1NDDstfcVwOscbC1OJ2g5JdA9Icr9D7zahhLVV4Z+WM73a6EILw tJU9shUD0egaBU+E9/pm3S+7t3iTlDaguxqwmAwpGDkynkov6ptBg0gDfbVpC1Umupjl lBweUe4M4S//h/WNgXKyhfYrfKHHpe/Ot5ru3qC8/qgdSDw+6yjazxXDMHDsVatxKZAw zY+8WnRuD5YigsCCw/UW2y46HH9hoCeAy0oPfqafo4/pH4+BX57JnFrDTDWAdQHIhxEQ 5xdg== X-Forwarded-Encrypted: i=1; AJvYcCWNITypfRTnS29aneGIOrz6zMnPYB2sV7SRVjw8Yc1McQbSXQrnnx2snlyV+96dTdmmOZcnNCdvj/RO3oCKczgFmbyVnOWZnBdv3cFy X-Gm-Message-State: AOJu0YxzE+UDqoiqFPAYk1vS0LzABjWCXpWwfmLAGsZzP7V788H6ZklG 4bnYM49RTJslM0XyEesPNJ7Jw9tEdStCHwv0z71pfeb9GNC0SZNy X-Google-Smtp-Source: AGHT+IEHRrZfsLAXmwfC0TLEfea+DaUlZDiB/BZsG73+jVdIprdrukbUYpmRDwzyTfT6rFZfayKehA== X-Received: by 2002:a05:6512:3194:b0:513:4965:8d4e with SMTP id i20-20020a056512319400b0051349658d4emr352641lfe.26.1709919256044; Fri, 08 Mar 2024 09:34:16 -0800 (PST) Received: from pc638.lan (host-185-121-47-193.sydskane.nu. [185.121.47.193]) by smtp.gmail.com with ESMTPSA id a23-20020a195f57000000b0051333e2f55dsm3150504lfj.159.2024.03.08.09.34.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Mar 2024 09:34:15 -0800 (PST) From: "Uladzislau Rezki (Sony)" To: "Paul E . McKenney" Cc: RCU , Neeraj upadhyay , Boqun Feng , Hillf Danton , Joel Fernandes , LKML , Uladzislau Rezki , Oleksiy Avramchenko , Frederic Weisbecker Subject: [PATCH v6 4/6] rcu: Support direct wake-up of synchronize_rcu() users Date: Fri, 8 Mar 2024 18:34:07 +0100 Message-Id: <20240308173409.335345-5-urezki@gmail.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240308173409.335345-1-urezki@gmail.com> References: <20240308173409.335345-1-urezki@gmail.com> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 This patch introduces a small enhancement which allows to do a direct wake-up of synchronize_rcu() callers. It occurs after a completion of grace period, thus by the gp-kthread. Number of clients is limited by the hard-coded maximum allowed threshold. The remaining part, if still exists is deferred to a main worker. Signed-off-by: Uladzislau Rezki (Sony) --- kernel/rcu/tree.c | 32 ++++++++++++++++++++++++++++++-- kernel/rcu/tree.h | 6 ++++++ 2 files changed, 36 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index dcc1764e8fad..063200613444 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -1672,7 +1672,8 @@ static void rcu_sr_normal_gp_cleanup_work(struct work_struct *work) */ static void rcu_sr_normal_gp_cleanup(void) { - struct llist_node *wait_tail; + struct llist_node *wait_tail, *next, *rcu; + int done = 0; wait_tail = rcu_state.srs_wait_tail; if (wait_tail == NULL) @@ -1680,12 +1681,39 @@ static void rcu_sr_normal_gp_cleanup(void) rcu_state.srs_wait_tail = NULL; ASSERT_EXCLUSIVE_WRITER(rcu_state.srs_wait_tail); + WARN_ON_ONCE(!rcu_sr_is_wait_head(wait_tail)); + + /* + * Process (a) and (d) cases. See an illustration. Apart of + * that it handles the scenario when all clients are done, + * wait-head is released if last. The worker is not kicked. + */ + llist_for_each_safe(rcu, next, wait_tail->next) { + if (rcu_sr_is_wait_head(rcu)) { + if (!rcu->next) { + rcu_sr_put_wait_head(rcu); + wait_tail->next = NULL; + } else { + wait_tail->next = rcu; + } + + break; + } + + rcu_sr_normal_complete(rcu); + // It can be last, update a next on this step. + wait_tail->next = next; + + if (++done == SR_MAX_USERS_WAKE_FROM_GP) + break; + } // concurrent sr_normal_gp_cleanup work might observe this update. smp_store_release(&rcu_state.srs_done_tail, wait_tail); ASSERT_EXCLUSIVE_WRITER(rcu_state.srs_done_tail); - schedule_work(&rcu_state.srs_cleanup_work); + if (wait_tail->next) + schedule_work(&rcu_state.srs_cleanup_work); } /* diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h index b942b9437438..2832787cee1d 100644 --- a/kernel/rcu/tree.h +++ b/kernel/rcu/tree.h @@ -315,6 +315,12 @@ do { \ __set_current_state(TASK_RUNNING); \ } while (0) +/* + * A max threshold for synchronize_rcu() users which are + * awaken directly by the rcu_gp_kthread(). Left part is + * deferred to the main worker. + */ +#define SR_MAX_USERS_WAKE_FROM_GP 5 #define SR_NORMAL_GP_WAIT_HEAD_MAX 5 struct sr_wait_node { From patchwork Fri Mar 8 17:34:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Uladzislau Rezki (Sony)" X-Patchwork-Id: 13587143 Received: from mail-lf1-f47.google.com (mail-lf1-f47.google.com [209.85.167.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D207154BC1; Fri, 8 Mar 2024 17:34:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.47 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709919260; cv=none; b=KGuTQFDKzqLBQCqeHBbl4nm7NpheP/Y6IxLbOjq+tEvWDf2CD+g07F/IVl7rPeHqtvlv4mz6XY/9kifSXsjzOkBjyjrUy+sykgBZa09h3Qny1+YkbYd/wbor4xyyJl17OfBMIUDmIWKK62fOroktsAJwFnnxrviCSQ7+VgswEaA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709919260; c=relaxed/simple; bh=v+sxR+Ob4ZQPLQ09ulDfHwcTMlnB/RStbMjd236wci8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=jjDqDylajryT2ST+6KO4d/GGjNV4hI0QmXKyB3sOseqQ/TrPb+eVempm6qBbyf5NZ5btOSBKNWCLgis5EE0e+ZwhI/qKN34K3N8bzNoJDgdCWFDfk+ZghJ4ofsdehfdl8RLCoouDZl36flQyoFDCqxkeYCXFlF0MfMczJ3nekS4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=U1gQq0Ef; arc=none smtp.client-ip=209.85.167.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="U1gQq0Ef" Received: by mail-lf1-f47.google.com with SMTP id 2adb3069b0e04-51340e89df1so2514387e87.1; Fri, 08 Mar 2024 09:34:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709919257; x=1710524057; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=UQxHmqgduMjmouOggzpR5Nl/EtQ16DC5+fzglQp0ujU=; b=U1gQq0EfO2xQ9QsFXklSTXUPSa5wipTXiGWQaJrIMf5OsYKwQ1Hwr9ZFfJLuOjnooW WMoxS2E1rY41sI75Og/FjXqXuIzefL8VnfYZlfh5/AY2jZezqT1JmblvOz5mvBMg9rEA /vJ2cgF1AYPuyeDBBJHjJZzDEDAaHdduX1cG+EgIMXBHbrs4zK3pCIz/tbNxck01PcXK OcvX8I7KMr+GS/n5xjMZbfOsW1nUAQHvuWEjCrqpks0jbZb3UKgckMObIvDOMnvMXoRY YVX6RK8LvhXh9qhhwwoJn7Rz0X+OAcpVY8s7Ud072HSbAdk0VYi4RYKnKqh6RXpISldW nGWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709919257; x=1710524057; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=UQxHmqgduMjmouOggzpR5Nl/EtQ16DC5+fzglQp0ujU=; b=V42b++bqXx2S7fNsDrYKuKrgcr+kEEO6rejgFF3MRVl+rx9yNo9O2gqb+Gt27RCbSV XlP/PJpdbeb7l1fAyMsEftQ8a6piYyw5nSB9g44fi04iTYledwTiPeCxKV26J/otdrmQ zruO5KlAO68f7VrGv6SSHedXtqVTuL8T2Nhy76wsYUrVR8ymeOmg8txvENH7JpFG1GXS hs7EK2PXjojJRkdAFCXgVhPTQwhZgw1EsefWNfG3ViEH88fweBvKZECTlfA99VSIhD3z DhB/cjeQwUNB8/b1y7nSOsAbXi68iUxYEqGCkyoxdC9O6jNDe0nAB87mejnKwd6A8LNx mLOw== X-Forwarded-Encrypted: i=1; AJvYcCVw2vqc6MaLnCYc4A3lNSduUTFgdxQ6IerOSMV8+aNd1BoiAHWF0E36RBr/8avoqst5x3b2KUVHGedgwcKb5bJjzxem9FZUPvYKg5Hc X-Gm-Message-State: AOJu0YzXEGMvw9yYyrKI1a7rA4XC93Id6BPJxtleCtYI+e1R12dtSm/O tfkfDq5k6dyPWbykzzWWqN4aU0/ovghOIWdEj5wRMpcnbRpl07mB X-Google-Smtp-Source: AGHT+IE5WRbqVE9uNkknnNbc6iyH+E0ACPKnVHNkrm+ZiUnPl2tgqWmTBO//NvpQNPmCgJVsQcKczA== X-Received: by 2002:a05:6512:238f:b0:512:b27c:18b5 with SMTP id c15-20020a056512238f00b00512b27c18b5mr6044752lfv.30.1709919256950; Fri, 08 Mar 2024 09:34:16 -0800 (PST) Received: from pc638.lan (host-185-121-47-193.sydskane.nu. [185.121.47.193]) by smtp.gmail.com with ESMTPSA id a23-20020a195f57000000b0051333e2f55dsm3150504lfj.159.2024.03.08.09.34.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Mar 2024 09:34:16 -0800 (PST) From: "Uladzislau Rezki (Sony)" To: "Paul E . McKenney" Cc: RCU , Neeraj upadhyay , Boqun Feng , Hillf Danton , Joel Fernandes , LKML , Uladzislau Rezki , Oleksiy Avramchenko , Frederic Weisbecker Subject: [PATCH v6 5/6] rcu: Do not release a wait-head from a GP kthread Date: Fri, 8 Mar 2024 18:34:08 +0100 Message-Id: <20240308173409.335345-6-urezki@gmail.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240308173409.335345-1-urezki@gmail.com> References: <20240308173409.335345-1-urezki@gmail.com> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Fix a below race by not releasing a wait-head from the GP-kthread as it can lead for reusing it whereas a worker can still access it thus execute newly added callbacks too early. CPU 0 CPU 1 ----- ----- // wait_tail == HEAD1 rcu_sr_normal_gp_cleanup() { // has passed SR_MAX_USERS_WAKE_FROM_GP wait_tail->next = next; // done_tail = HEAD1 smp_store_release(&rcu_state.srs_done_tail, wait_tail); queue_work() { test_and_set_bit(WORK_STRUCT_PENDING_BIT, work_data_bits(work) __queue_work() } } set_work_pool_and_clear_pending() rcu_sr_normal_gp_cleanup_work() { // new GP, wait_tail == HEAD2 rcu_sr_normal_gp_cleanup() { // executes all completion, but stop at HEAD1 wait_tail->next = HEAD1; // done_tail = HEAD2 smp_store_release(&rcu_state.srs_done_tail, wait_tail); queue_work() { test_and_set_bit(WORK_STRUCT_PENDING_BIT, work_data_bits(work) __queue_work() } } // done = HEAD2 done = smp_load_acquire(&rcu_state.srs_done_tail); // head = HEAD1 head = done->next; done->next = NULL; llist_for_each_safe() { // completes all callbacks, release HEAD1 } } // Process second queue set_work_pool_and_clear_pending() rcu_sr_normal_gp_cleanup_work() { // done = HEAD2 done = smp_load_acquire(&rcu_state.srs_done_tail); // new GP, wait_tail == HEAD3 rcu_sr_normal_gp_cleanup() { // Finds HEAD2 with ->next == NULL at the end rcu_sr_put_wait_head(HEAD2) ... // A few more GPs later rcu_sr_normal_gp_init() { HEAD2 = rcu_sr_get_wait_head(); llist_add(HEAD2, &rcu_state.srs_next); // head == rcu_state.srs_next head = done->next; done->next = NULL; llist_for_each_safe() { // EXECUTE CALLBACKS TOO EARLY!!! } } Reviewed-by: Frederic Weisbecker Reviewed-by: Joel Fernandes (Google) Reported-by: Frederic Weisbecker Fixes: 05a10b921000 ("rcu: Support direct wake-up of synchronize_rcu() users") Signed-off-by: Uladzislau Rezki (Sony) --- kernel/rcu/tree.c | 22 ++++++++-------------- 1 file changed, 8 insertions(+), 14 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 063200613444..d854115ff11f 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -1684,21 +1684,11 @@ static void rcu_sr_normal_gp_cleanup(void) WARN_ON_ONCE(!rcu_sr_is_wait_head(wait_tail)); /* - * Process (a) and (d) cases. See an illustration. Apart of - * that it handles the scenario when all clients are done, - * wait-head is released if last. The worker is not kicked. + * Process (a) and (d) cases. See an illustration. */ llist_for_each_safe(rcu, next, wait_tail->next) { - if (rcu_sr_is_wait_head(rcu)) { - if (!rcu->next) { - rcu_sr_put_wait_head(rcu); - wait_tail->next = NULL; - } else { - wait_tail->next = rcu; - } - + if (rcu_sr_is_wait_head(rcu)) break; - } rcu_sr_normal_complete(rcu); // It can be last, update a next on this step. @@ -1712,8 +1702,12 @@ static void rcu_sr_normal_gp_cleanup(void) smp_store_release(&rcu_state.srs_done_tail, wait_tail); ASSERT_EXCLUSIVE_WRITER(rcu_state.srs_done_tail); - if (wait_tail->next) - schedule_work(&rcu_state.srs_cleanup_work); + /* + * We schedule a work in order to perform a final processing + * of outstanding users(if still left) and releasing wait-heads + * added by rcu_sr_normal_gp_init() call. + */ + schedule_work(&rcu_state.srs_cleanup_work); } /* From patchwork Fri Mar 8 17:34:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Uladzislau Rezki (Sony)" X-Patchwork-Id: 13587144 Received: from mail-lf1-f43.google.com (mail-lf1-f43.google.com [209.85.167.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 97E5F5675B; Fri, 8 Mar 2024 17:34:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709919261; cv=none; b=d9ZQuVmeYTbGbBKkEOX3IK8Ep86mZ8R3p5GsIGQFNyXr6TDfRsxnDtetyI3ENhA7FAFGaN5A5kL5FVq5AwNZAGTBeJdviPbdXwtgDzGcSZ2hQkY03nU7BDWqWEotAdU8D4zO8eblfbwTEhtNWaQeZw3mtmwBbL3ehPXs7bhLOqM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709919261; c=relaxed/simple; bh=LbObmE6sW1VI50NF8hBx5GYZYD1ZL7EHGJyWwt2uywI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=DYRtt9RSums5L3gvhlvNvj22mkSNxds75vNHj/w06ewyVEyLZqUkkRHAsv0+mavWVe4T+n4CuT0X05mef1Fmgu1LiiszTfVRxMZMjXkekXYC40xODsxYDelLKFNS+nP39nfYOyIGLQ8ZwepvvbOIUThuu5rLgzE4k/JWycMAL3o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=U+sp2PHx; arc=none smtp.client-ip=209.85.167.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="U+sp2PHx" Received: by mail-lf1-f43.google.com with SMTP id 2adb3069b0e04-51323dfce59so1231041e87.3; Fri, 08 Mar 2024 09:34:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709919258; x=1710524058; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Ar3ui94sCfHSHLOMDUIl3IQJOAbXBJ+xjHyVr1i8Qfs=; b=U+sp2PHx/IWH3GC8IDZfyxNfJRHx7cs7KoUvZLY942S9DFnXNC8MWsAo8YTb9LpltS xaqBrifAGTEgdhQtcHbc024PBwLDS/gfdSXiipq+9gIEJ2Mu6GFTEDCgM7htPcPF1nn8 pvdCyMk946xmTsJ/4y84dUTOt/NvhvNyi6ZZMqq1RkUxu6B3khCs6UNTWw4Vq6jqomjJ 6JzBMqTdgaDOXAAG8dbKFMVNB8jawXdaCuY9KBx4RNHXCAb3Cgh7SI3lKfLEt0r1Q9tx BbAqT12D1K2fEbekJR94AY8KDFskT/2VYMuJkMSeR1D2Ps6qihvqubW4z9WqWrv75bt9 Ellw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709919258; x=1710524058; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Ar3ui94sCfHSHLOMDUIl3IQJOAbXBJ+xjHyVr1i8Qfs=; b=oTPnuYV6DXjoSwgo3QYr4xfZziEASm84iwzL6ufvpPOadMxS+okxmJaS8DBP/S9pL+ 3lVHCdO4PFMPeu9KLlnWoQJHFFVgNyQudC84kW4jY54TZ/1Xk+XxuMMEYdtBh6bAhmzl gIW52+dCzb4EttmC6uRhmD7I6O57v6i1BM2T61c9/+/NwAP7IbZJT9iCnhDixO8tXx70 SU3NXulchx4ZcS1pXIHFWMLcakLg5SpdEVcC0+GHOmGMMXLtbTsIwEFMm8WhP2Sls7wu NxindzEpIkyox3ZP4vMQsUIoR3wjU3s74uFx6vYuyFRaV1ScAkw2/sR61Egscw+mNDZt bPdA== X-Forwarded-Encrypted: i=1; AJvYcCU3utwdRKx3S2ELMjJ4X6OYm8RMNYHx+h7f16jXMBgy79Ued3VCvqVNSTHGTW9rD2pD2ZYxwiOaf6cA0Ef2Cf+oDOHG3Cct+rWUvDAW X-Gm-Message-State: AOJu0YyOkgt3xYz1Z8S8vtKxVKzN9Amg3xTDQuVssva+qMip/gjGMtrN t84pjfNMOUltk6/6RHBnenCvO7AXP7zBL8eBby42L0XPWq6gBFKd X-Google-Smtp-Source: AGHT+IFOuH6v/8Kdjsj0UKr8zhImUNwpyXkZeVqhGT0QyG0KNUHE1/rN+3yvUn5TpNmkm7Teuy1aow== X-Received: by 2002:ac2:4113:0:b0:513:19df:4eb6 with SMTP id b19-20020ac24113000000b0051319df4eb6mr3653497lfi.12.1709919257852; Fri, 08 Mar 2024 09:34:17 -0800 (PST) Received: from pc638.lan (host-185-121-47-193.sydskane.nu. [185.121.47.193]) by smtp.gmail.com with ESMTPSA id a23-20020a195f57000000b0051333e2f55dsm3150504lfj.159.2024.03.08.09.34.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Mar 2024 09:34:17 -0800 (PST) From: "Uladzislau Rezki (Sony)" To: "Paul E . McKenney" Cc: RCU , Neeraj upadhyay , Boqun Feng , Hillf Danton , Joel Fernandes , LKML , Uladzislau Rezki , Oleksiy Avramchenko , Frederic Weisbecker Subject: [PATCH v6 6/6] rcu: Allocate WQ with WQ_MEM_RECLAIM bit set Date: Fri, 8 Mar 2024 18:34:09 +0100 Message-Id: <20240308173409.335345-7-urezki@gmail.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240308173409.335345-1-urezki@gmail.com> References: <20240308173409.335345-1-urezki@gmail.com> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 synchronize_rcu() users have to be processed regardless of memory pressure so our private WQ needs to have at least one execution context what WQ_MEM_RECLAIM flag guarantees. Reviewed-by: Joel Fernandes (Google) --- kernel/rcu/tree.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index d854115ff11f..fe2e15c320dd 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -1609,6 +1609,7 @@ static void rcu_sr_put_wait_head(struct llist_node *node) /* Disabled by default. */ static int rcu_normal_wake_from_gp; module_param(rcu_normal_wake_from_gp, int, 0644); +static struct workqueue_struct *sync_wq; static void rcu_sr_normal_complete(struct llist_node *node) { @@ -1707,7 +1708,7 @@ static void rcu_sr_normal_gp_cleanup(void) * of outstanding users(if still left) and releasing wait-heads * added by rcu_sr_normal_gp_init() call. */ - schedule_work(&rcu_state.srs_cleanup_work); + queue_work(sync_wq, &rcu_state.srs_cleanup_work); } /* @@ -5609,6 +5610,9 @@ void __init rcu_init(void) rcu_gp_wq = alloc_workqueue("rcu_gp", WQ_MEM_RECLAIM, 0); WARN_ON(!rcu_gp_wq); + sync_wq = alloc_workqueue("sync_wq", WQ_MEM_RECLAIM, 0); + WARN_ON(!sync_wq); + /* Fill in default value for rcutree.qovld boot parameter. */ /* -After- the rcu_node ->lock fields are initialized! */ if (qovld < 0)