From patchwork Thu May 12 03:04:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 12846882 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 85E84C433FE for ; Thu, 12 May 2022 03:05:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242984AbiELDFD (ORCPT ); Wed, 11 May 2022 23:05:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37186 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244299AbiELDFA (ORCPT ); Wed, 11 May 2022 23:05:00 -0400 Received: from mail-qv1-xf30.google.com (mail-qv1-xf30.google.com [IPv6:2607:f8b0:4864:20::f30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CCAEF1FD866 for ; Wed, 11 May 2022 20:04:59 -0700 (PDT) Received: by mail-qv1-xf30.google.com with SMTP id n10so3469710qvi.5 for ; Wed, 11 May 2022 20:04:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=TONViqYIQx6lxNplcStVpG+Rk+ubHZb2eJ/SOygdl3M=; b=gFB92lOmFtfM+xmcxsj9coXYagNjnFP8WZqbjJYZ58K3M1cv8P/CwVy4w0iBP1Q+qd cQTikBm/JJhJ3VUDwVZu771czMekWQHz91B1w8bQyr+m39dvGmjXXiREy1HqWAybp5V+ ebX5MKBira6/vwpYDijDMUgoKI6/Gt7eMg9PA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=TONViqYIQx6lxNplcStVpG+Rk+ubHZb2eJ/SOygdl3M=; b=OcdihprySpT35P4lmMc2e+MvQTsY8uOeoMUBInCVT5qgovb5+QgGWp0Omd7Mcj4W+8 uwGWBuYE7xfqWmropE4Nr2ugAMRP90/v22hVEn3K0SEGBmUqGG9E+ffy+cy4JcJ9aehQ +GXntPsY1gR+j51IsRZYbgndhSLRl5sYWnemeDXpFAEm141DJNCXRmC6fIgC1503zYx1 +UMvbRvpbQzp3LCUlllmwPopRYw6yYiHMpTMtfpQE+cevlY9pquaZoKRU9gNfBy/bamg V+nwGwKynINvBVCXXptYuLEqW4DbfzjyBIRcjXDeTSm5ROr05KXsz6wuK127H6xD92VQ 7xcg== X-Gm-Message-State: AOAM532638AErVD+qpnLrCov9I7EpC8BiT6/3P/Bu4EiPYozPlSkE8LJ 4f7igka1FppPq7SYS1GyCa+0Xf+tuVN1lg== X-Google-Smtp-Source: ABdhPJyfZpvxUICAuNsWMa5CxKFYh+0q83frtSAXsIkrDVLpRjlwptonXmW2teDfbVOUmVYZvpttug== X-Received: by 2002:ad4:5c88:0:b0:45a:e934:4730 with SMTP id o8-20020ad45c88000000b0045ae9344730mr24340822qvh.61.1652324698730; Wed, 11 May 2022 20:04:58 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (29.46.245.35.bc.googleusercontent.com. [35.245.46.29]) by smtp.gmail.com with ESMTPSA id h124-20020a376c82000000b0069fc13ce203sm2270334qkc.52.2022.05.11.20.04.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 May 2022 20:04:57 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, "Joel Fernandes (Google)" Subject: [RFC v1 01/14] rcu: Add a lock-less lazy RCU implementation Date: Thu, 12 May 2022 03:04:29 +0000 Message-Id: <20220512030442.2530552-2-joel@joelfernandes.org> X-Mailer: git-send-email 2.36.0.550.gb090851708-goog In-Reply-To: <20220512030442.2530552-1-joel@joelfernandes.org> References: <20220512030442.2530552-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org Implement timer-based RCU callback batching. The batch is flushed whenever a certain amount of time has passed, or the batch on a particular CPU grows too big. Also memory pressure can flush it. Locking is avoided to reduce lock contention when queuing and dequeuing happens on different CPUs of a per-cpu list, such as when shrinker context is running on different CPU. Also not having to use locks keeps the per-CPU structure size small. Signed-off-by: Joel Fernandes (Google) --- include/linux/rcupdate.h | 6 ++ kernel/rcu/Kconfig | 8 +++ kernel/rcu/Makefile | 1 + kernel/rcu/lazy.c | 145 +++++++++++++++++++++++++++++++++++++++ kernel/rcu/rcu.h | 5 ++ kernel/rcu/tree.c | 2 + 6 files changed, 167 insertions(+) create mode 100644 kernel/rcu/lazy.c diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index 88b42eb46406..d0a6c4f5172c 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -82,6 +82,12 @@ static inline int rcu_preempt_depth(void) #endif /* #else #ifdef CONFIG_PREEMPT_RCU */ +#ifdef CONFIG_RCU_LAZY +void call_rcu_lazy(struct rcu_head *head, rcu_callback_t func); +#else +#define call_rcu_lazy(head, func) call_rcu(head, func) +#endif + /* Internal to kernel */ void rcu_init(void); extern int rcu_scheduler_active __read_mostly; diff --git a/kernel/rcu/Kconfig b/kernel/rcu/Kconfig index bf8e341e75b4..c09715079829 100644 --- a/kernel/rcu/Kconfig +++ b/kernel/rcu/Kconfig @@ -241,4 +241,12 @@ config TASKS_TRACE_RCU_READ_MB Say N here if you hate read-side memory barriers. Take the default if you are unsure. +config RCU_LAZY + bool "RCU callback lazy invocation functionality" + depends on RCU_NOCB_CPU + default y + help + To save power, batch RCU callbacks and flush after delay, memory + pressure or callback list growing too big. + endmenu # "RCU Subsystem" diff --git a/kernel/rcu/Makefile b/kernel/rcu/Makefile index 0cfb009a99b9..8968b330d6e0 100644 --- a/kernel/rcu/Makefile +++ b/kernel/rcu/Makefile @@ -16,3 +16,4 @@ obj-$(CONFIG_RCU_REF_SCALE_TEST) += refscale.o obj-$(CONFIG_TREE_RCU) += tree.o obj-$(CONFIG_TINY_RCU) += tiny.o obj-$(CONFIG_RCU_NEED_SEGCBLIST) += rcu_segcblist.o +obj-$(CONFIG_RCU_LAZY) += lazy.o diff --git a/kernel/rcu/lazy.c b/kernel/rcu/lazy.c new file mode 100644 index 000000000000..55e406cfc528 --- /dev/null +++ b/kernel/rcu/lazy.c @@ -0,0 +1,145 @@ +/* + * Lockless lazy-RCU implementation. + */ +#include +#include +#include +#include "rcu.h" + +// How much to batch before flushing? +#define MAX_LAZY_BATCH 2048 + +// How much to wait before flushing? +#define MAX_LAZY_JIFFIES 10000 + +// We cast lazy_rcu_head to rcu_head and back. This keeps the API simple while +// allowing us to use lockless list node in the head. Also, we use BUILD_BUG_ON +// later to ensure that rcu_head and lazy_rcu_head are of the same size. +struct lazy_rcu_head { + struct llist_node llist_node; + void (*func)(struct callback_head *head); +} __attribute__((aligned(sizeof(void *)))); + +struct rcu_lazy_pcp { + struct llist_head head; + struct delayed_work work; + atomic_t count; +}; +DEFINE_PER_CPU(struct rcu_lazy_pcp, rcu_lazy_pcp_ins); + +// Lockless flush of CPU, can be called concurrently. +static void lazy_rcu_flush_cpu(struct rcu_lazy_pcp *rlp) +{ + struct llist_node *node = llist_del_all(&rlp->head); + struct lazy_rcu_head *cursor, *temp; + + if (!node) + return; + + llist_for_each_entry_safe(cursor, temp, node, llist_node) { + struct rcu_head *rh = (struct rcu_head *)cursor; + debug_rcu_head_unqueue(rh); + call_rcu(rh, rh->func); + atomic_dec(&rlp->count); + } +} + +void call_rcu_lazy(struct rcu_head *head_rcu, rcu_callback_t func) +{ + struct lazy_rcu_head *head = (struct lazy_rcu_head *)head_rcu; + struct rcu_lazy_pcp *rlp; + + preempt_disable(); + rlp = this_cpu_ptr(&rcu_lazy_pcp_ins); + preempt_enable(); + + if (debug_rcu_head_queue((void *)head)) { + // Probable double call_rcu(), just leak. + WARN_ONCE(1, "%s(): Double-freed call. rcu_head %p\n", + __func__, head); + + // Mark as success and leave. + return; + } + + // Queue to per-cpu llist + head->func = func; + llist_add(&head->llist_node, &rlp->head); + + // Flush queue if too big + if (atomic_inc_return(&rlp->count) >= MAX_LAZY_BATCH) { + lazy_rcu_flush_cpu(rlp); + } else { + if (!delayed_work_pending(&rlp->work)) { + schedule_delayed_work(&rlp->work, MAX_LAZY_JIFFIES); + } + } +} + +static unsigned long +lazy_rcu_shrink_count(struct shrinker *shrink, struct shrink_control *sc) +{ + unsigned long count = 0; + int cpu; + + /* Snapshot count of all CPUs */ + for_each_possible_cpu(cpu) { + struct rcu_lazy_pcp *rlp = per_cpu_ptr(&rcu_lazy_pcp_ins, cpu); + + count += atomic_read(&rlp->count); + } + + return count; +} + +static unsigned long +lazy_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) +{ + int cpu, freed = 0; + + for_each_possible_cpu(cpu) { + struct rcu_lazy_pcp *rlp = per_cpu_ptr(&rcu_lazy_pcp_ins, cpu); + unsigned long count; + + count = atomic_read(&rlp->count); + lazy_rcu_flush_cpu(rlp); + sc->nr_to_scan -= count; + freed += count; + if (sc->nr_to_scan <= 0) + break; + } + + return freed == 0 ? SHRINK_STOP : freed; +} + +/* + * This function is invoked after MAX_LAZY_JIFFIES timeout. + */ +static void lazy_work(struct work_struct *work) +{ + struct rcu_lazy_pcp *rlp = container_of(work, struct rcu_lazy_pcp, work.work); + + lazy_rcu_flush_cpu(rlp); +} + +static struct shrinker lazy_rcu_shrinker = { + .count_objects = lazy_rcu_shrink_count, + .scan_objects = lazy_rcu_shrink_scan, + .batch = 0, + .seeks = DEFAULT_SEEKS, +}; + +void __init rcu_lazy_init(void) +{ + int cpu; + + BUILD_BUG_ON(sizeof(struct lazy_rcu_head) != sizeof(struct rcu_head)); + + for_each_possible_cpu(cpu) { + struct rcu_lazy_pcp *rlp = per_cpu_ptr(&rcu_lazy_pcp_ins, cpu); + INIT_DELAYED_WORK(&rlp->work, lazy_work); + } + + if (register_shrinker(&lazy_rcu_shrinker)) + pr_err("Failed to register lazy_rcu shrinker!\n"); +} diff --git a/kernel/rcu/rcu.h b/kernel/rcu/rcu.h index 24b5f2c2de87..a5f4b44f395f 100644 --- a/kernel/rcu/rcu.h +++ b/kernel/rcu/rcu.h @@ -561,4 +561,9 @@ void show_rcu_tasks_trace_gp_kthread(void); static inline void show_rcu_tasks_trace_gp_kthread(void) {} #endif +#ifdef CONFIG_RCU_LAZY +void rcu_lazy_init(void); +#else +static inline void rcu_lazy_init(void) {} +#endif #endif /* __LINUX_RCU_H */ diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index a4c25a6283b0..ebdf6f7c9023 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -4775,6 +4775,8 @@ void __init rcu_init(void) qovld_calc = DEFAULT_RCU_QOVLD_MULT * qhimark; else qovld_calc = qovld; + + rcu_lazy_init(); } #include "tree_stall.h" From patchwork Thu May 12 03:04:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 12846883 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 000B7C433F5 for ; Thu, 12 May 2022 03:05:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238889AbiELDFE (ORCPT ); Wed, 11 May 2022 23:05:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37198 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244406AbiELDFC (ORCPT ); Wed, 11 May 2022 23:05:02 -0400 Received: from mail-qv1-xf2d.google.com (mail-qv1-xf2d.google.com [IPv6:2607:f8b0:4864:20::f2d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D79981FC2E2 for ; Wed, 11 May 2022 20:05:01 -0700 (PDT) Received: by mail-qv1-xf2d.google.com with SMTP id js14so3455620qvb.12 for ; Wed, 11 May 2022 20:05:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=h+Ucrgo0EM0maL7FfExwGPTxb3huWv/XPq/8ObhEBrc=; b=jqRk3dVBT7IZGqdmImZ8I3lH98wyBphzHFiz1x1pPIRKIDC45+H5ky4EMeztLFO0wK HQF8O5eSVUBJmWvd30U2tp1ibO7N0IkrvMn+gR0l2EVtTfiCIgru9oLeNH+1Bd0Zl8b2 KBdCbhVzLihTlvIaV63Tf/3zQzehAYzofLxTI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=h+Ucrgo0EM0maL7FfExwGPTxb3huWv/XPq/8ObhEBrc=; b=LZeTFzVvth4UYh8PXK1HEUYKAXoNftRjJ6EtxrQWmajgvSXAddxmjpz7gRmIVf5rIh y0rO69fFgwgoEzW5rR53E/+SFqeqBKf1GDAw1bm5CyW7PcO7H/D+ST1VAZdTozZHREyo 06FX38M87KwkMQ/nC5IBQJRuBxnw6YMACZcf5Ek4HssKBdg+Fc/lQheayNeOhWPUAnC/ fxrQnP5bo8HTWjcNee9ayg2DF+cL01NGqsYNU6vwrsoLXscH0Dow702uZOS00idPoRNF GGGZuTh75LlJ5CbU71WnoSfc4ldMpkD7tyfhSD6fTRh1Ou9OS+ji7AgHovmL44t5niWq qcTA== X-Gm-Message-State: AOAM531N/9FIURAHYWwMkCaIeo0uZtevxOjSqv/HxYjR/STlBhESUUA6 v8VBbBcZ1qXbUL4VL9Xn/uMW3w6Td2ieBg== X-Google-Smtp-Source: ABdhPJxl1v/9fRdONzTSO0gtRcKmBrpCm5hTRw3CRpzGVSLVjxXiKT4W/IvYpwSyF6hgqdg9pyrtmw== X-Received: by 2002:a05:6214:e45:b0:45a:ba94:11 with SMTP id o5-20020a0562140e4500b0045aba940011mr25326095qvc.38.1652324700826; Wed, 11 May 2022 20:05:00 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (29.46.245.35.bc.googleusercontent.com. [35.245.46.29]) by smtp.gmail.com with ESMTPSA id h124-20020a376c82000000b0069fc13ce203sm2270334qkc.52.2022.05.11.20.04.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 May 2022 20:04:59 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, "Joel Fernandes (Google)" Subject: [RFC v1 02/14] workqueue: Add a lazy version of queue_rcu_work() Date: Thu, 12 May 2022 03:04:30 +0000 Message-Id: <20220512030442.2530552-3-joel@joelfernandes.org> X-Mailer: git-send-email 2.36.0.550.gb090851708-goog In-Reply-To: <20220512030442.2530552-1-joel@joelfernandes.org> References: <20220512030442.2530552-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org This will be used in kfree_rcu() later to make it do call_rcu() lazily. Signed-off-by: Joel Fernandes (Google) --- include/linux/workqueue.h | 1 + kernel/workqueue.c | 25 +++++++++++++++++++++++++ 2 files changed, 26 insertions(+) diff --git a/include/linux/workqueue.h b/include/linux/workqueue.h index 7fee9b6cfede..2678a6b5b3f3 100644 --- a/include/linux/workqueue.h +++ b/include/linux/workqueue.h @@ -444,6 +444,7 @@ extern bool queue_delayed_work_on(int cpu, struct workqueue_struct *wq, extern bool mod_delayed_work_on(int cpu, struct workqueue_struct *wq, struct delayed_work *dwork, unsigned long delay); extern bool queue_rcu_work(struct workqueue_struct *wq, struct rcu_work *rwork); +extern bool queue_rcu_work_lazy(struct workqueue_struct *wq, struct rcu_work *rwork); extern void flush_workqueue(struct workqueue_struct *wq); extern void drain_workqueue(struct workqueue_struct *wq); diff --git a/kernel/workqueue.c b/kernel/workqueue.c index 33f1106b4f99..9444949cc148 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -1796,6 +1796,31 @@ bool queue_rcu_work(struct workqueue_struct *wq, struct rcu_work *rwork) } EXPORT_SYMBOL(queue_rcu_work); +/** + * queue_rcu_work_lazy - queue work after a RCU grace period + * @wq: workqueue to use + * @rwork: work to queue + * + * Return: %false if @rwork was already pending, %true otherwise. Note + * that a full RCU grace period is guaranteed only after a %true return. + * While @rwork is guaranteed to be executed after a %false return, the + * execution may happen before a full RCU grace period has passed. + */ +bool queue_rcu_work_lazy(struct workqueue_struct *wq, struct rcu_work *rwork) +{ + struct work_struct *work = &rwork->work; + + if (!test_and_set_bit(WORK_STRUCT_PENDING_BIT, work_data_bits(work))) { + rwork->wq = wq; + call_rcu_lazy(&rwork->rcu, rcu_work_rcufn); + return true; + } + + return false; +} +EXPORT_SYMBOL(queue_rcu_work_lazy); + + /** * worker_enter_idle - enter idle state * @worker: worker which is entering idle state From patchwork Thu May 12 03:04:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 12846884 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 38E33C433EF for ; Thu, 12 May 2022 03:05:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244459AbiELDFF (ORCPT ); Wed, 11 May 2022 23:05:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37228 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244299AbiELDFE (ORCPT ); Wed, 11 May 2022 23:05:04 -0400 Received: from mail-qk1-x72e.google.com (mail-qk1-x72e.google.com [IPv6:2607:f8b0:4864:20::72e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C197D1FE3CE for ; Wed, 11 May 2022 20:05:02 -0700 (PDT) Received: by mail-qk1-x72e.google.com with SMTP id 185so3742109qke.7 for ; Wed, 11 May 2022 20:05:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=6iRjJs/b10qWS973iRQCDfhG0rdBQpqDx7mITDoIFnQ=; b=DoI3pBi249bWsPQZbGGRoOL3nGoc4pzBY1UM02yWAL+Jx40INZHBcPy36V3xpZpxHN nHktj24BxoTKBxmNTygSNlHA91tkI78CDWtuEYWDVNrEDYdv7im1842GuVvO+PeOBjP/ TN3orCb8HaCJyMvmQJDS5xI0UuEwjAbPt9gp4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=6iRjJs/b10qWS973iRQCDfhG0rdBQpqDx7mITDoIFnQ=; b=0RqMusd48YPE++fInLQS8ex4Y71Rcz/ZgaM+yhJtFeMLYBtc/HbUjFiCH9vNYhpAFN ukzHROA0Yy9j56QMJ7WvA56ZpKxIFA/B0Uk9dX5EOycW49Eekr0dmd6jXE6FSG8NuMd1 L66G36BFPKJYEk5iWhonTJ+S/KSZIEV0N72jinvX45yzPw2ZGKTRYM3nHEBOmW1f2Xhk Y/Tk4kDYJu+bieTqC0tg1Nw7DSQ0v0pt16mpBAmBx7DVhFoub0HXQ7EbzA2rX69c5T1E ogequ6mdZ3dCMU6i140Ul6sXjkThQRNwKOPz4o90wzdVHBBD+PH6ElQPpLEa1WQeuJ9u rO5A== X-Gm-Message-State: AOAM531Is2pDDi9xTrkC4kAMNN+2LPcbj1lv5rcOwBfOPgdytm9Gmtx0 Lf0/CWyG6jnGvn6BmhubirkVyOYaJ5PbtQ== X-Google-Smtp-Source: ABdhPJyfR17mcxMbPisRHjZwHlS8+J0SwCgE73HA/4TvFiqQrYO7+vC4MeTRsxa8wz+Wm7IDacc7ug== X-Received: by 2002:a05:620a:d4e:b0:69b:f2e5:b8cd with SMTP id o14-20020a05620a0d4e00b0069bf2e5b8cdmr20840842qkl.535.1652324701550; Wed, 11 May 2022 20:05:01 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (29.46.245.35.bc.googleusercontent.com. [35.245.46.29]) by smtp.gmail.com with ESMTPSA id h124-20020a376c82000000b0069fc13ce203sm2270334qkc.52.2022.05.11.20.05.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 May 2022 20:05:01 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, "Joel Fernandes (Google)" Subject: [RFC v1 03/14] block/blk-ioc: Move call_rcu() to call_rcu_lazy() Date: Thu, 12 May 2022 03:04:31 +0000 Message-Id: <20220512030442.2530552-4-joel@joelfernandes.org> X-Mailer: git-send-email 2.36.0.550.gb090851708-goog In-Reply-To: <20220512030442.2530552-1-joel@joelfernandes.org> References: <20220512030442.2530552-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org This is required to prevent callbacks triggering RCU machinery too quickly and too often, which adds more power to the system. Signed-off-by: Joel Fernandes (Google) --- block/blk-ioc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/block/blk-ioc.c b/block/blk-ioc.c index 11f49f78db32..96d5de5df0b6 100644 --- a/block/blk-ioc.c +++ b/block/blk-ioc.c @@ -98,7 +98,7 @@ static void ioc_destroy_icq(struct io_cq *icq) */ icq->__rcu_icq_cache = et->icq_cache; icq->flags |= ICQ_DESTROYED; - call_rcu(&icq->__rcu_head, icq_free_icq_rcu); + call_rcu_lazy(&icq->__rcu_head, icq_free_icq_rcu); } /* From patchwork Thu May 12 03:04:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 12846885 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C5883C433F5 for ; Thu, 12 May 2022 03:05:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244406AbiELDFG (ORCPT ); Wed, 11 May 2022 23:05:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37244 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244468AbiELDFF (ORCPT ); Wed, 11 May 2022 23:05:05 -0400 Received: from mail-qk1-x735.google.com (mail-qk1-x735.google.com [IPv6:2607:f8b0:4864:20::735]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DB7BB1FC2E2 for ; Wed, 11 May 2022 20:05:03 -0700 (PDT) Received: by mail-qk1-x735.google.com with SMTP id bs17so3784519qkb.0 for ; Wed, 11 May 2022 20:05:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=tZYwvc4nSRfwkY1hI9gFtWU/c+R8K3b6EhO2tYCnlpU=; b=aTE/unhoMaGxOUCCrBIgKbzySr3cULCQ5TAoDSL8kyEauR7AtDV8CejE+04dYhccJy KcH7+4QnzwLXqblklJ0ME3ugHcEpvag39UX7OoY/7U+OJ1cs0dZzafMFFJEGI+Zi7hLo s0LvhNHP0h18uC2G2IAFvMamoCdOi5YwY3Afg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=tZYwvc4nSRfwkY1hI9gFtWU/c+R8K3b6EhO2tYCnlpU=; b=r1ThJtZT1kmmLUrlHoKi13UQOWL964eX79zVVRAxNXmMWI9OatHolGG6A2uWBD2l2m sphsnr1FnYD1AuCWend9TzOFCfPNnGGKtkFdMWSAmFqeXB64DfhpKYMfPCVeic7ORr8s YcfyozbiBlKDAbArbsPIGAAKL9z8ISzflPr7H4C4vGBbXEJ/LjihIAgoIDYKL+EXQkqS 3eGExxQ6QSelRcsZ8u4XW/syHBFJI3H7tvaB6Bc5h3HEwRp65cxvC3wNcLNWNMEa4rzs M9VBPnNydE2IaGn6YHZP846o5HxTOzjPEq63bS8AEifqqvv5+bJYfF6uCZZDElzBjjGb KJtw== X-Gm-Message-State: AOAM532etLAaaFMvGvvjIhXjP9ItLuNxFdDmvEIN4UuFJK94C03wJTdR Gy0WBqJPtSqyDjAcGvLYVTTd3AlH/J70yg== X-Google-Smtp-Source: ABdhPJwirgdZpfilGyH+mvunVo+eLsRAK7ZAzSvBCeQ/lqZ4EuTlJMqQ/s+5n6K6FyBEBFgMNrR33Q== X-Received: by 2002:a37:48e:0:b0:69f:9bc8:720d with SMTP id 136-20020a37048e000000b0069f9bc8720dmr21080039qke.268.1652324702891; Wed, 11 May 2022 20:05:02 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (29.46.245.35.bc.googleusercontent.com. [35.245.46.29]) by smtp.gmail.com with ESMTPSA id h124-20020a376c82000000b0069fc13ce203sm2270334qkc.52.2022.05.11.20.05.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 May 2022 20:05:02 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, "Joel Fernandes (Google)" Subject: [RFC v1 04/14] cred: Move call_rcu() to call_rcu_lazy() Date: Thu, 12 May 2022 03:04:32 +0000 Message-Id: <20220512030442.2530552-5-joel@joelfernandes.org> X-Mailer: git-send-email 2.36.0.550.gb090851708-goog In-Reply-To: <20220512030442.2530552-1-joel@joelfernandes.org> References: <20220512030442.2530552-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org This is required to prevent callbacks triggering RCU machinery too quickly and too often, which adds more power to the system. Signed-off-by: Joel Fernandes (Google) --- kernel/cred.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/cred.c b/kernel/cred.c index 933155c96922..f4d69d7f2763 100644 --- a/kernel/cred.c +++ b/kernel/cred.c @@ -150,7 +150,7 @@ void __put_cred(struct cred *cred) if (cred->non_rcu) put_cred_rcu(&cred->rcu); else - call_rcu(&cred->rcu, put_cred_rcu); + call_rcu_lazy(&cred->rcu, put_cred_rcu); } EXPORT_SYMBOL(__put_cred); From patchwork Thu May 12 03:04:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 12846886 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4F112C433FE for ; Thu, 12 May 2022 03:05:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244468AbiELDFH (ORCPT ); Wed, 11 May 2022 23:05:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37250 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244299AbiELDFF (ORCPT ); Wed, 11 May 2022 23:05:05 -0400 Received: from mail-qv1-xf30.google.com (mail-qv1-xf30.google.com [IPv6:2607:f8b0:4864:20::f30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1C2211FE3CE for ; Wed, 11 May 2022 20:05:05 -0700 (PDT) Received: by mail-qv1-xf30.google.com with SMTP id f3so3486845qvi.2 for ; Wed, 11 May 2022 20:05:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=invpnJO/pL9cqkXCSVg2CTU14ipkwGYESRvGU1ShOSw=; b=XPyG6kT2eKR/Mt9EfUyGMibPfy5f/CGskcH0N1nkpFc9yt9AIbPp7S1uqqhfsZ+FQd Ye2WUQG8v8JOxvOK7mmkTqQcDOi1Op0vRhplB7mWWhyYtDn6xJ8JDqki3XcxD67Wzssc 8JJsq7ULmKW13yx0wXaxNcP1brDecWNab7TbQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=invpnJO/pL9cqkXCSVg2CTU14ipkwGYESRvGU1ShOSw=; b=GKJWFjiajm2X2qg/kqejrg4j7ywn+xw58G/m5vjcAb9HogGk6EqsYDRQihXAoMoMuL uN9PXoaRKBMgDyiclz/rS3uPksJiCmXyiCp8tpvnOwcn3TkScAdZikJ4EGgwbHUDQYAQ uKl7U87epQTx0rzIFPYitkGC9Gh3Hve3LlejkjyKEhqQhght2TzJ7YP7/ESibKgvJ3Qm HdDthLqOWBGq5dR4aqfdneluD+CBL973TfvJCeMvRSasegalRmST8bDLeZNk7SS04UbB 2GWrwfm4NeDOBfVmuTA52tKc6BXeQGPXJB2XwVxPlxLRs6FBGF3svzsnydbwSBW09iVq yDUw== X-Gm-Message-State: AOAM531rjdpGI/J/UFGfRgjElGRlMJv9klbZwW7IhSuvbxQiDhXWwNCg 79Lp2nuryAvjOpv1ZgKbO81b59NWFGN/Ag== X-Google-Smtp-Source: ABdhPJz7Qo74KWQjFmgHXsxBlbBgMkDon7y+iEK3xtmizs5I+GQ8De+SZ725yfUpY9nPfniAvlhAqA== X-Received: by 2002:a05:6214:e6b:b0:45b:474:1035 with SMTP id jz11-20020a0562140e6b00b0045b04741035mr18262385qvb.39.1652324704087; Wed, 11 May 2022 20:05:04 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (29.46.245.35.bc.googleusercontent.com. [35.245.46.29]) by smtp.gmail.com with ESMTPSA id h124-20020a376c82000000b0069fc13ce203sm2270334qkc.52.2022.05.11.20.05.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 May 2022 20:05:03 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, "Joel Fernandes (Google)" Subject: [RFC v1 05/14] fs: Move call_rcu() to call_rcu_lazy() in some paths Date: Thu, 12 May 2022 03:04:33 +0000 Message-Id: <20220512030442.2530552-6-joel@joelfernandes.org> X-Mailer: git-send-email 2.36.0.550.gb090851708-goog In-Reply-To: <20220512030442.2530552-1-joel@joelfernandes.org> References: <20220512030442.2530552-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org This is required to prevent callbacks triggering RCU machinery too quickly and too often, which adds more power to the system. When testing, we found that these paths were invoked often when the system is not doing anything (screen is ON but otherwise idle). Signed-off-by: Joel Fernandes (Google) --- fs/dcache.c | 4 ++-- fs/eventpoll.c | 2 +- fs/file_table.c | 3 ++- fs/inode.c | 2 +- 4 files changed, 6 insertions(+), 5 deletions(-) diff --git a/fs/dcache.c b/fs/dcache.c index c84269c6e8bf..517e02cde103 100644 --- a/fs/dcache.c +++ b/fs/dcache.c @@ -366,7 +366,7 @@ static void dentry_free(struct dentry *dentry) if (unlikely(dname_external(dentry))) { struct external_name *p = external_name(dentry); if (likely(atomic_dec_and_test(&p->u.count))) { - call_rcu(&dentry->d_u.d_rcu, __d_free_external); + call_rcu_lazy(&dentry->d_u.d_rcu, __d_free_external); return; } } @@ -374,7 +374,7 @@ static void dentry_free(struct dentry *dentry) if (dentry->d_flags & DCACHE_NORCU) __d_free(&dentry->d_u.d_rcu); else - call_rcu(&dentry->d_u.d_rcu, __d_free); + call_rcu_lazy(&dentry->d_u.d_rcu, __d_free); } /* diff --git a/fs/eventpoll.c b/fs/eventpoll.c index e2daa940ebce..10a24cca2cff 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -728,7 +728,7 @@ static int ep_remove(struct eventpoll *ep, struct epitem *epi) * ep->mtx. The rcu read side, reverse_path_check_proc(), does not make * use of the rbn field. */ - call_rcu(&epi->rcu, epi_rcu_free); + call_rcu_lazy(&epi->rcu, epi_rcu_free); percpu_counter_dec(&ep->user->epoll_watches); diff --git a/fs/file_table.c b/fs/file_table.c index 7d2e692b66a9..415815d3ef80 100644 --- a/fs/file_table.c +++ b/fs/file_table.c @@ -56,7 +56,8 @@ static inline void file_free(struct file *f) security_file_free(f); if (!(f->f_mode & FMODE_NOACCOUNT)) percpu_counter_dec(&nr_files); - call_rcu(&f->f_u.fu_rcuhead, file_free_rcu); + + call_rcu_lazy(&f->f_u.fu_rcuhead, file_free_rcu); } /* diff --git a/fs/inode.c b/fs/inode.c index 63324df6fa27..b288a5bef4c7 100644 --- a/fs/inode.c +++ b/fs/inode.c @@ -312,7 +312,7 @@ static void destroy_inode(struct inode *inode) return; } inode->free_inode = ops->free_inode; - call_rcu(&inode->i_rcu, i_callback); + call_rcu_lazy(&inode->i_rcu, i_callback); } /** From patchwork Thu May 12 03:04:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 12846887 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3399FC433EF for ; Thu, 12 May 2022 03:05:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244480AbiELDFI (ORCPT ); Wed, 11 May 2022 23:05:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37256 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244450AbiELDFG (ORCPT ); Wed, 11 May 2022 23:05:06 -0400 Received: from mail-qt1-x82a.google.com (mail-qt1-x82a.google.com [IPv6:2607:f8b0:4864:20::82a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 143081FC2E2 for ; Wed, 11 May 2022 20:05:06 -0700 (PDT) Received: by mail-qt1-x82a.google.com with SMTP id u35so3409694qtc.13 for ; Wed, 11 May 2022 20:05:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=EfV4yWDJGiChIrululMwLYVp74K3O8ar9DeNHie78NA=; b=D69o/mFEs6JlcjHLexiUzbRKIB8i2MrJgyyP+hFR26flu0CZjJrFxDDq7b7HBJgWk2 KzTtJSBcXkXxMB5J4gmMBWlpJ37LeRM5hmsHBeKlfUjmzv9sjzm968Cudz7eWGqHzb9Z pXaDXHzg123ti55tg82tN8DC2P/hHZXuMY8Ek= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=EfV4yWDJGiChIrululMwLYVp74K3O8ar9DeNHie78NA=; b=Z7V12xQ+YMWB8veuVBjF87xCNuWhXzYALO1YuCoYavC67W2t1nog4WoeTgH74lGWI8 b5uZqf+sBSuxBPXq5XSnmQKMxChswfp3BBzJZPLaID5eAwEW0j6YmcIVHZXICACkovpF c9I/GlC/1guZIIDkiXt5fN6nJgurt8KDYoAcjNt2DOMkTMv4xD1bEN3cHCe/C5mAD1GX BgS08a8WS8xk0Rs73UcSz7GBvV3AoO2nMaHzfh4MOgoL0s6PEglc8fNUm9j0J2gebsEg 11rVBHk7qKdgx8Nz3CEB2l7tFORc357wpT22quQtp0iPnDJXo7+sjGK9HjW3g79etntv Wnjg== X-Gm-Message-State: AOAM533W+dKwhyJqoSC3PO0ITJ0wqvggXKHw/O6jdt7I0o2kYy6BO3g5 fspm9SP51ghyP8KLM/NG7/SsXy96rKm+SA== X-Google-Smtp-Source: ABdhPJzAJ6la+p18oLB6wyrN2jmE1J7R3XulQnk0Scm9w/JxFeN+NubmFZ/1BQMpt14tRX3/IUMiug== X-Received: by 2002:ac8:5852:0:b0:2f3:cdbd:d5a0 with SMTP id h18-20020ac85852000000b002f3cdbdd5a0mr23561982qth.207.1652324705089; Wed, 11 May 2022 20:05:05 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (29.46.245.35.bc.googleusercontent.com. [35.245.46.29]) by smtp.gmail.com with ESMTPSA id h124-20020a376c82000000b0069fc13ce203sm2270334qkc.52.2022.05.11.20.05.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 May 2022 20:05:04 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, "Joel Fernandes (Google)" Subject: [RFC v1 06/14] kernel: Move various core kernel usages to call_rcu_lazy() Date: Thu, 12 May 2022 03:04:34 +0000 Message-Id: <20220512030442.2530552-7-joel@joelfernandes.org> X-Mailer: git-send-email 2.36.0.550.gb090851708-goog In-Reply-To: <20220512030442.2530552-1-joel@joelfernandes.org> References: <20220512030442.2530552-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org Signed-off-by: Joel Fernandes (Google) --- kernel/exit.c | 2 +- kernel/pid.c | 2 +- kernel/time/posix-timers.c | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/kernel/exit.c b/kernel/exit.c index b00a25bb4ab9..6d84ec8e1fd0 100644 --- a/kernel/exit.c +++ b/kernel/exit.c @@ -177,7 +177,7 @@ static void delayed_put_task_struct(struct rcu_head *rhp) void put_task_struct_rcu_user(struct task_struct *task) { if (refcount_dec_and_test(&task->rcu_users)) - call_rcu(&task->rcu, delayed_put_task_struct); + call_rcu_lazy(&task->rcu, delayed_put_task_struct); } void release_task(struct task_struct *p) diff --git a/kernel/pid.c b/kernel/pid.c index 2fc0a16ec77b..5a5144519d70 100644 --- a/kernel/pid.c +++ b/kernel/pid.c @@ -153,7 +153,7 @@ void free_pid(struct pid *pid) } spin_unlock_irqrestore(&pidmap_lock, flags); - call_rcu(&pid->rcu, delayed_put_pid); + call_rcu_lazy(&pid->rcu, delayed_put_pid); } struct pid *alloc_pid(struct pid_namespace *ns, pid_t *set_tid, diff --git a/kernel/time/posix-timers.c b/kernel/time/posix-timers.c index 1cd10b102c51..04e191dfa91e 100644 --- a/kernel/time/posix-timers.c +++ b/kernel/time/posix-timers.c @@ -485,7 +485,7 @@ static void release_posix_timer(struct k_itimer *tmr, int it_id_set) } put_pid(tmr->it_pid); sigqueue_free(tmr->sigq); - call_rcu(&tmr->rcu, k_itimer_rcu_free); + call_rcu_lazy(&tmr->rcu, k_itimer_rcu_free); } static int common_timer_create(struct k_itimer *new_timer) From patchwork Thu May 12 03:04:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 12846888 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9D7DCC433EF for ; Thu, 12 May 2022 03:05:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244516AbiELDFL (ORCPT ); Wed, 11 May 2022 23:05:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37274 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244299AbiELDFI (ORCPT ); Wed, 11 May 2022 23:05:08 -0400 Received: from mail-qt1-x836.google.com (mail-qt1-x836.google.com [IPv6:2607:f8b0:4864:20::836]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1E6B21FD866 for ; Wed, 11 May 2022 20:05:07 -0700 (PDT) Received: by mail-qt1-x836.google.com with SMTP id x22so3459861qto.2 for ; Wed, 11 May 2022 20:05:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=GlPq9iL7yZbXCRdaYgr9qcrcREB/mxZJb+fO+2O0bUg=; b=MzvsJjzbupJa9p2oAyYninBW2dHy++R8QT4mGrPJMnEFtCqV77ok/4eZfMG5f6FPla 7Xj38x48Z9Q820s9MedRT3iaUMUt3JPdwRCV5HHQxijqRRppIpG2EUKinOIJejXx7ir9 ScrazMyDrJYcAMoS9Pm+Rm0ML3hgndZSX+O7w= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=GlPq9iL7yZbXCRdaYgr9qcrcREB/mxZJb+fO+2O0bUg=; b=Ba7P0w6ecE5YECmzfSh78JqG2NtAQYvR0kw+MuPIzfcsDaXEHDheqsp56pp7i7cQ2V 9GflCm8x18ApW8SDK7Q8gomUtSO+u+C8MIIjzSBhsGqijgP1UkxX8MWCXtfnTv2gTnmQ HwmRvKyMJKvLJr6z6OwvPfULCGKv/RmrwXMGNuLcdVA7xv60434NvFhUL26n9Bf5zmWj J4M938If1SlxjgKHIhge16j9Kg6YepiBlv4X7v388Wal2DksoXf4qfSGwx3ti+Wyym/C c4ekC84ZizwPGSsc7GntYFlFxVL4Ke3kUxU4BHMrmAOWG526OBqN13TlgekwF5jQ85Of 4GNg== X-Gm-Message-State: AOAM533Clz/jBDSf6cTw6Ob7UnrO3W9UofU/8MmaKhCyazmwGl+P/KzA 7DwHwsdKylv/XSXp4aXRAcWAqDgM1RPMpA== X-Google-Smtp-Source: ABdhPJygGO+SWbl/tzMkaYMcWnVCLIIUzqbRTcmMnO+jNhUP/t27rObsTov9kHAY4rGEfytFp/5t0g== X-Received: by 2002:ac8:5bd0:0:b0:2f3:bb55:4612 with SMTP id b16-20020ac85bd0000000b002f3bb554612mr26997113qtb.391.1652324706044; Wed, 11 May 2022 20:05:06 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (29.46.245.35.bc.googleusercontent.com. [35.245.46.29]) by smtp.gmail.com with ESMTPSA id h124-20020a376c82000000b0069fc13ce203sm2270334qkc.52.2022.05.11.20.05.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 May 2022 20:05:05 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, "Joel Fernandes (Google)" Subject: [RFC v1 07/14] security: Move call_rcu() to call_rcu_lazy() Date: Thu, 12 May 2022 03:04:35 +0000 Message-Id: <20220512030442.2530552-8-joel@joelfernandes.org> X-Mailer: git-send-email 2.36.0.550.gb090851708-goog In-Reply-To: <20220512030442.2530552-1-joel@joelfernandes.org> References: <20220512030442.2530552-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org This is required to prevent callbacks triggering RCU machinery too quickly and too often, which adds more power to the system. Signed-off-by: Joel Fernandes (Google) --- security/security.c | 2 +- security/selinux/avc.c | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/security/security.c b/security/security.c index 22261d79f333..42c165631e8f 100644 --- a/security/security.c +++ b/security/security.c @@ -1039,7 +1039,7 @@ void security_inode_free(struct inode *inode) * The inode will be freed after the RCU grace period too. */ if (inode->i_security) - call_rcu((struct rcu_head *)inode->i_security, + call_rcu_lazy((struct rcu_head *)inode->i_security, inode_free_by_rcu); } diff --git a/security/selinux/avc.c b/security/selinux/avc.c index abcd9740d10f..8c639c6df955 100644 --- a/security/selinux/avc.c +++ b/security/selinux/avc.c @@ -442,7 +442,7 @@ static void avc_node_free(struct rcu_head *rhead) static void avc_node_delete(struct selinux_avc *avc, struct avc_node *node) { hlist_del_rcu(&node->list); - call_rcu(&node->rhead, avc_node_free); + call_rcu_lazy(&node->rhead, avc_node_free); atomic_dec(&avc->avc_cache.active_nodes); } @@ -458,7 +458,7 @@ static void avc_node_replace(struct selinux_avc *avc, struct avc_node *new, struct avc_node *old) { hlist_replace_rcu(&old->list, &new->list); - call_rcu(&old->rhead, avc_node_free); + call_rcu_lazy(&old->rhead, avc_node_free); atomic_dec(&avc->avc_cache.active_nodes); } From patchwork Thu May 12 03:04:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 12846889 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CCC0FC433EF for ; Thu, 12 May 2022 03:05:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244299AbiELDFM (ORCPT ); Wed, 11 May 2022 23:05:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37276 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244489AbiELDFI (ORCPT ); Wed, 11 May 2022 23:05:08 -0400 Received: from mail-qt1-x835.google.com (mail-qt1-x835.google.com [IPv6:2607:f8b0:4864:20::835]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AEB191FC2E2 for ; Wed, 11 May 2022 20:05:07 -0700 (PDT) Received: by mail-qt1-x835.google.com with SMTP id h3so3455056qtn.4 for ; Wed, 11 May 2022 20:05:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Nug8nbY87iEa+XSWPZu0aU2M2l7wYnXsOW/cyW7eajY=; b=tfAZ5r4bh3NA6e7T//RBhRkcSq9FjWi8hDp8OY/sROlUB0RuMAFrDjdx0RzpTCsGfR fDRT8PvhCjT0q/Rd+J6+q2nZVSUOke/HxBiDDWJ+tEJjewYaWE7KpkvknRmQNlH8NdAN f4Y/RJSFvFr4sna2sKphUCPaJE/zjyK/XPqfU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Nug8nbY87iEa+XSWPZu0aU2M2l7wYnXsOW/cyW7eajY=; b=JsqblsPL4wTa04WJr+JPN0DTnF7y/UnClQElUuMg6FRxJTxbkEj0TbPzJB2Tj6nvsV st8dZeHPFIKmcHoqU5yPFKexW2PW+BVeRhjWXuPGWvM47/g/nIg4/FGiUNeiTOJga5hx PhzFeHNP+HVUmOlY6FZJx3xOvzG9NqKawPpAg2ihJs8e10jJcDYdBKDZbTIRmwnLwps/ lQr6VDblk7JpHce/ScXLUQsGvSPOq4qmWDmD1WlGGuT41BmFDEjrWTz0pIZg+Q7coCOz fEhkeamzaXE34q1UDQyftyAUWci8nNLGqUIdQ65D6rTKawYyFdGapY5m+SefocdUOhX9 i6xA== X-Gm-Message-State: AOAM533mRNm31JJ2q5CZBbrV8WKYk8JgFOooANZIrshMjvI8/yMl7RaO 5rnP3qbp66bz/OSFyOIG7a3/OsNO/SsXZQ== X-Google-Smtp-Source: ABdhPJw8vCshps1jBtfrj9hugLug1THWkweJKWw53CHfGWpfw06HInnRxbTomhM8ALgtj3MfWMdPvw== X-Received: by 2002:a05:622a:6206:b0:2f1:e40e:41b3 with SMTP id hj6-20020a05622a620600b002f1e40e41b3mr26722019qtb.196.1652324706678; Wed, 11 May 2022 20:05:06 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (29.46.245.35.bc.googleusercontent.com. [35.245.46.29]) by smtp.gmail.com with ESMTPSA id h124-20020a376c82000000b0069fc13ce203sm2270334qkc.52.2022.05.11.20.05.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 May 2022 20:05:06 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, "Joel Fernandes (Google)" Subject: [RFC v1 08/14] net/core: Move call_rcu() to call_rcu_lazy() Date: Thu, 12 May 2022 03:04:36 +0000 Message-Id: <20220512030442.2530552-9-joel@joelfernandes.org> X-Mailer: git-send-email 2.36.0.550.gb090851708-goog In-Reply-To: <20220512030442.2530552-1-joel@joelfernandes.org> References: <20220512030442.2530552-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org This is required to prevent callbacks triggering RCU machinery too quickly and too often, which adds more power to the system. Signed-off-by: Joel Fernandes (Google) --- net/core/dst.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/net/core/dst.c b/net/core/dst.c index d16c2c9bfebd..68c240a4a0d7 100644 --- a/net/core/dst.c +++ b/net/core/dst.c @@ -174,7 +174,7 @@ void dst_release(struct dst_entry *dst) net_warn_ratelimited("%s: dst:%p refcnt:%d\n", __func__, dst, newrefcnt); if (!newrefcnt) - call_rcu(&dst->rcu_head, dst_destroy_rcu); + call_rcu_lazy(&dst->rcu_head, dst_destroy_rcu); } } EXPORT_SYMBOL(dst_release); From patchwork Thu May 12 03:04:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 12846890 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D4E17C433FE for ; Thu, 12 May 2022 03:05:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244489AbiELDFN (ORCPT ); Wed, 11 May 2022 23:05:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37292 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244450AbiELDFJ (ORCPT ); Wed, 11 May 2022 23:05:09 -0400 Received: from mail-qt1-x833.google.com (mail-qt1-x833.google.com [IPv6:2607:f8b0:4864:20::833]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EED1E1FE1F2 for ; Wed, 11 May 2022 20:05:08 -0700 (PDT) Received: by mail-qt1-x833.google.com with SMTP id k2so3467275qtp.1 for ; Wed, 11 May 2022 20:05:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=rZAMLkDSz2UfdY/AqwPRElqLdOiMu3ZxjzKwUOJzmh4=; b=FEawNa57ren25RELKctNZipR4nrle5P4f62ZSPSmqimIpXMZhzZL8fNQuubHgkNHH9 3YhrIQZZuyPKsRTS4kkMOCpT7wqQLcP5ww/SBCR1XcFUrFbrjee3AEA59UEN8V6GsuSa PuU5lSjrWrQIP41lwxaqXj/PJk4P6Ne5IY5AM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=rZAMLkDSz2UfdY/AqwPRElqLdOiMu3ZxjzKwUOJzmh4=; b=ND1mIvN994B2kR91hQMMzylTAK2oEDuZllr/ePwbRkxDTZDK/SnkHGoDe6duQE4r9d xD5MlYrTXzOVmj1nTkhUQ60Hk1q3ThvGOvQ7EIVKUDaeGXo0UTh8DYBOEV+utwzbnP6a iiauN22SLqSTWgXPJwgKGIcKPWmmHBb7gn02VsiwKvJGYr1sic3j9gvYQKKWzvfaHNBS PycStxuBK2wu2eB9Mb8UVzznbn8KzQFjXOEuaO4Y2k1TCqdpS/1p5czY1TApF/W9C+oI zP+1if2LsobgJcJRptTyI/vRqfxd3r8+cgtOt0+NTqlInkhWHcZrOrpiuF28N5VvPZkJ ngOg== X-Gm-Message-State: AOAM530Ix+2oXRNSI0C8ZSKcnZ+iaZxdX7xiYYChDs3YNozAReh+L1t6 o/Zp7EN9xffDwnHqW4R1Jt4IFt7Gr7GwkQ== X-Google-Smtp-Source: ABdhPJyaLEwkQ6xzAUPMoregPIWuyFIXyOQKu1SbM2u5iFKQiwp9aExGwRWpIV/PJRBqqLnH7ul8SQ== X-Received: by 2002:a05:622a:110c:b0:2f3:d347:6f8d with SMTP id e12-20020a05622a110c00b002f3d3476f8dmr20297078qty.403.1652324707964; Wed, 11 May 2022 20:05:07 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (29.46.245.35.bc.googleusercontent.com. [35.245.46.29]) by smtp.gmail.com with ESMTPSA id h124-20020a376c82000000b0069fc13ce203sm2270334qkc.52.2022.05.11.20.05.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 May 2022 20:05:07 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, "Joel Fernandes (Google)" Subject: [RFC v1 09/14] lib: Move call_rcu() to call_rcu_lazy() Date: Thu, 12 May 2022 03:04:37 +0000 Message-Id: <20220512030442.2530552-10-joel@joelfernandes.org> X-Mailer: git-send-email 2.36.0.550.gb090851708-goog In-Reply-To: <20220512030442.2530552-1-joel@joelfernandes.org> References: <20220512030442.2530552-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org Move radix-tree and xarray to call_rcu_lazy(). This is required to prevent callbacks triggering RCU machinery too quickly and too often, which adds more power to the system. Signed-off-by: Joel Fernandes (Google) --- lib/radix-tree.c | 2 +- lib/xarray.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/radix-tree.c b/lib/radix-tree.c index b3afafe46fff..1526dc9e1d93 100644 --- a/lib/radix-tree.c +++ b/lib/radix-tree.c @@ -305,7 +305,7 @@ void radix_tree_node_rcu_free(struct rcu_head *head) static inline void radix_tree_node_free(struct radix_tree_node *node) { - call_rcu(&node->rcu_head, radix_tree_node_rcu_free); + call_rcu_lazy(&node->rcu_head, radix_tree_node_rcu_free); } /* diff --git a/lib/xarray.c b/lib/xarray.c index 6f47f6375808..29d6abf6b1ff 100644 --- a/lib/xarray.c +++ b/lib/xarray.c @@ -255,7 +255,7 @@ static void xa_node_free(struct xa_node *node) { XA_NODE_BUG_ON(node, !list_empty(&node->private_list)); node->array = XA_RCU_FREE; - call_rcu(&node->rcu_head, radix_tree_node_rcu_free); + call_rcu_lazy(&node->rcu_head, radix_tree_node_rcu_free); } /* From patchwork Thu May 12 03:04:38 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 12846891 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2882CC433F5 for ; Thu, 12 May 2022 03:05:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244450AbiELDFQ (ORCPT ); Wed, 11 May 2022 23:05:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37298 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244497AbiELDFK (ORCPT ); Wed, 11 May 2022 23:05:10 -0400 Received: from mail-qk1-x735.google.com (mail-qk1-x735.google.com [IPv6:2607:f8b0:4864:20::735]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 653BE1FE3C9 for ; Wed, 11 May 2022 20:05:09 -0700 (PDT) Received: by mail-qk1-x735.google.com with SMTP id bs17so3784519qkb.0 for ; Wed, 11 May 2022 20:05:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ubeCwGbbIca8I/2Mu+0A56F+uXWjxN0ith+OykgDGUE=; b=Tv8/8rp1258TG/oHxE8QnGMI11Nf0JIPrD0Voc0mwoaQEarQYIZiCDLYCkubMdaGrS rs/ADIDbVf8HG6IFDYon/iNP89nUay/mAkNdDpqcTYLkUx11uYNTvpLSb/VZuiPm0hwr X5jq60H7J95wr88SGhywS6L8lPZttuZXpLUS4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ubeCwGbbIca8I/2Mu+0A56F+uXWjxN0ith+OykgDGUE=; b=A1Ca+CuSsY46tTD5FcTkoEQqZMy+Q8lE4kp6/PQBBzsKN9nAgTmU7zNDb6XYLIqrBj Z1JG5r190pmeulfQ0vEjOScz3UFdYdlibgJOYjyptJANxWzOQ7k0mB6qqJptvdOqKTib aQ4n4+X1Bg57qUXDViYZR3CtWdb59gAelYSof0/8QWaSRYiL85nYUCQNHKAJIP0BL7pb DwNY5jQWalZxgDsa6bEKrkwZcZvkyHBHzhBA3N/ivpOENkz7Bd6FGcr2qzKaWr63HynJ 8xxP6Zp2n+38K0u3Jnj6k4Fnw5JbCML2EQHih0QY0dymcW5AM24u4t7YhbxKwk8dLYXU MnVA== X-Gm-Message-State: AOAM533fMAqVvnLxReL/uZ8IP9sD8/x7yAbr+aeNcCb27qjvQLknzWgf 382dV7QaB3eouU3kMtBMUX7TTOo960/0BQ== X-Google-Smtp-Source: ABdhPJwYusyqb64lt76aBKKFepAgSqiuLLaIZRktkqKQj0y5RhDW/1dgCHkF6xA+oV3Y7llMoV21uQ== X-Received: by 2002:ae9:ed8e:0:b0:69f:cc2c:3d6b with SMTP id c136-20020ae9ed8e000000b0069fcc2c3d6bmr21655890qkg.381.1652324708956; Wed, 11 May 2022 20:05:08 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (29.46.245.35.bc.googleusercontent.com. [35.245.46.29]) by smtp.gmail.com with ESMTPSA id h124-20020a376c82000000b0069fc13ce203sm2270334qkc.52.2022.05.11.20.05.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 May 2022 20:05:08 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, "Joel Fernandes (Google)" Subject: [RFC v1 10/14] kfree/rcu: Queue RCU work via queue_rcu_work_lazy() Date: Thu, 12 May 2022 03:04:38 +0000 Message-Id: <20220512030442.2530552-11-joel@joelfernandes.org> X-Mailer: git-send-email 2.36.0.550.gb090851708-goog In-Reply-To: <20220512030442.2530552-1-joel@joelfernandes.org> References: <20220512030442.2530552-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index ebdf6f7c9023..3baf29014f86 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -3407,7 +3407,7 @@ static void kfree_rcu_monitor(struct work_struct *work) // be that the work is in the pending state when // channels have been detached following by each // other. - queue_rcu_work(system_wq, &krwp->rcu_work); + queue_rcu_work_lazy(system_wq, &krwp->rcu_work); } } From patchwork Thu May 12 03:04:39 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 12846892 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3318AC433F5 for ; Thu, 12 May 2022 03:05:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244217AbiELDFT (ORCPT ); Wed, 11 May 2022 23:05:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37314 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244529AbiELDFM (ORCPT ); Wed, 11 May 2022 23:05:12 -0400 Received: from mail-qt1-x831.google.com (mail-qt1-x831.google.com [IPv6:2607:f8b0:4864:20::831]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 679B01FE3CE for ; Wed, 11 May 2022 20:05:11 -0700 (PDT) Received: by mail-qt1-x831.google.com with SMTP id hh4so3427927qtb.10 for ; Wed, 11 May 2022 20:05:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=au0WoB+u99j5yQ6lmarkVSHpLzgPDYFuYSvbCBirDpk=; b=TGcnN/8wF1fa16DoEg1TA5IThvSDVSqnwmG2s4kijAaJ71BF54l8I823QaWnXGw02w yU+VEV+Lm6wL3okVAWOXHuRLKGSfFtPo2gMjp5JhFty/oPxfJKYUuL9G3eos66hyIMk5 wYNhHYmP0I1M8cO3txWEJD9BAYQhtS+TYe0Ew= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=au0WoB+u99j5yQ6lmarkVSHpLzgPDYFuYSvbCBirDpk=; b=JS5oL/wpa2aqeezs5yclD+gpVUMlP3KQkNCyNsrRVdYPMYHILShmT+d4iorzSXiuc1 A6WMNmOSmMABSPpG9GnrVxA7zXrheLnbBgByS1ZTyh0i4KVC5Yqcw3e+CiTOJgA76Mcx Z00tJq/fdyb/+/uXopLzrX24+ibb0uH0WTHPC2n04cbvGncG2FBLo88en9fpaGpSKFPj 3FfRmwoJu8nxpbsLv3jLRAI+ree2uNenZPAczjiZQryWOS++AkK21bSNNjsmp4jSUogC E5A23jszLN1EQiNgLaEquxV6IhwDf6dk/XnHf8GpaFEeeV6BdVHKM3Z8YXe8P+WN4rxO IadA== X-Gm-Message-State: AOAM5337yJ3SxYFM+sIKuYZl84urFQ00Jkt7bbCon1LrAZgAclC+eEQm s/ANFafPqdPii9O2pQwNHewbwLzkEVBldQ== X-Google-Smtp-Source: ABdhPJxOga9EoctgpdwhL7qg/2+dgrYEWWdAgD5wYdcuPXWi6bTyq/3ttWuNgsYJGbcOXFoDhBCnUQ== X-Received: by 2002:ac8:5b8e:0:b0:2f3:f12d:bab5 with SMTP id a14-20020ac85b8e000000b002f3f12dbab5mr5520609qta.183.1652324710178; Wed, 11 May 2022 20:05:10 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (29.46.245.35.bc.googleusercontent.com. [35.245.46.29]) by smtp.gmail.com with ESMTPSA id h124-20020a376c82000000b0069fc13ce203sm2270334qkc.52.2022.05.11.20.05.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 May 2022 20:05:09 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, "Joel Fernandes (Google)" Subject: [RFC v1 11/14] i915: Move call_rcu() to call_rcu_lazy() Date: Thu, 12 May 2022 03:04:39 +0000 Message-Id: <20220512030442.2530552-12-joel@joelfernandes.org> X-Mailer: git-send-email 2.36.0.550.gb090851708-goog In-Reply-To: <20220512030442.2530552-1-joel@joelfernandes.org> References: <20220512030442.2530552-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org This is required to prevent callbacks triggering RCU machinery too quickly and too often, which adds more power to the system. Signed-off-by: Joel Fernandes (Google) --- drivers/gpu/drm/i915/gem/i915_gem_object.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object.c b/drivers/gpu/drm/i915/gem/i915_gem_object.c index d87b508b59b1..b54d1be3ee68 100644 --- a/drivers/gpu/drm/i915/gem/i915_gem_object.c +++ b/drivers/gpu/drm/i915/gem/i915_gem_object.c @@ -343,7 +343,7 @@ static void __i915_gem_free_objects(struct drm_i915_private *i915, __i915_gem_free_object(obj); /* But keep the pointer alive for RCU-protected lookups */ - call_rcu(&obj->rcu, __i915_gem_free_object_rcu); + call_rcu_lazy(&obj->rcu, __i915_gem_free_object_rcu); cond_resched(); } } From patchwork Thu May 12 03:04:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 12846893 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4BE14C433EF for ; Thu, 12 May 2022 03:05:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244497AbiELDFU (ORCPT ); Wed, 11 May 2022 23:05:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37292 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244533AbiELDFN (ORCPT ); Wed, 11 May 2022 23:05:13 -0400 Received: from mail-qk1-x736.google.com (mail-qk1-x736.google.com [IPv6:2607:f8b0:4864:20::736]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3D4971FC2E2 for ; Wed, 11 May 2022 20:05:12 -0700 (PDT) Received: by mail-qk1-x736.google.com with SMTP id z126so3756942qkb.2 for ; Wed, 11 May 2022 20:05:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=oeVCD28BZkKVAoFgJA0eT8P+UE+HQoj/imzw07R+UyA=; b=LA+We+R2/2fOk4tZjx61kMwo3Z96H6NWU9bCb7Ue5UFhME2AuZVFEj8DfSyT/7tvBb RlWBRbrguE/iuCXvBo+l/3x7grMqV3YR8e0o1vt5zdyO6V6AChJH4019LztmyG5V2RhX qc1mOsdDXWx5mHXaNq6Fs7yZGjx1b1yVqJX9Y= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=oeVCD28BZkKVAoFgJA0eT8P+UE+HQoj/imzw07R+UyA=; b=16ynmBs8rgn6vv3jifyG9sMn+oaohp3Q3QsUON6UVUzMvImtGRJ8zfD0aOx2c8koW3 Di9GNf1ttmqxpwnbRl6WFK3m3E9oE4v17BmW/EXJrD11xSEzLd74vEHhQQHPKbhuoMX0 OdmMqdoeFV+jtMZjCKFZKyLe3Z9ODS8J7kbK0t+bZ8nxgFtZkyZYfFVVG26yLJuXluLE 015K1Mbq3hu1IQn6KCVbb1asEjm82vPJ6Hz1CIKxm3y//ZzNiZalW+qNwHaVQZtx4BSH 6MpNCF3Qig6+kDgjp8ntboAgVR+tF868jJ35g3aiWoPAItMzIeVvQgN10L4tA5WMdiUG qefw== X-Gm-Message-State: AOAM531QpaSIKvaF59ChvJPUJO414MywsHYwvxxxpJ44hYOSPvAvR7sV NQCluWhqQ3qCmdeHjeoMX1o8t8OHN7SW3w== X-Google-Smtp-Source: ABdhPJyWKNe3rKNuaPfIwj4ZWoycKgk939N7IuqQqUSnRr8uP00q4tO40J3wFiEKe2wbwU4JCPuk9Q== X-Received: by 2002:a05:620a:1903:b0:67d:243b:a8ae with SMTP id bj3-20020a05620a190300b0067d243ba8aemr21149211qkb.142.1652324711253; Wed, 11 May 2022 20:05:11 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (29.46.245.35.bc.googleusercontent.com. [35.245.46.29]) by smtp.gmail.com with ESMTPSA id h124-20020a376c82000000b0069fc13ce203sm2270334qkc.52.2022.05.11.20.05.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 May 2022 20:05:10 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, "Joel Fernandes (Google)" Subject: [RFC v1 12/14] rcu/kfree: remove useless monitor_todo flag Date: Thu, 12 May 2022 03:04:40 +0000 Message-Id: <20220512030442.2530552-13-joel@joelfernandes.org> X-Mailer: git-send-email 2.36.0.550.gb090851708-goog In-Reply-To: <20220512030442.2530552-1-joel@joelfernandes.org> References: <20220512030442.2530552-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org monitor_todo is not needed as the work struct already tracks if work is pending. Just use that to know if work is pending using delayed_work_pending() helper. Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree.c | 22 +++++++--------------- 1 file changed, 7 insertions(+), 15 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 3baf29014f86..3828ac3bf1c4 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -3155,7 +3155,6 @@ struct kfree_rcu_cpu_work { * @krw_arr: Array of batches of kfree_rcu() objects waiting for a grace period * @lock: Synchronize access to this structure * @monitor_work: Promote @head to @head_free after KFREE_DRAIN_JIFFIES - * @monitor_todo: Tracks whether a @monitor_work delayed work is pending * @initialized: The @rcu_work fields have been initialized * @count: Number of objects for which GP not started * @bkvcache: @@ -3180,7 +3179,6 @@ struct kfree_rcu_cpu { struct kfree_rcu_cpu_work krw_arr[KFREE_N_BATCHES]; raw_spinlock_t lock; struct delayed_work monitor_work; - bool monitor_todo; bool initialized; int count; @@ -3416,9 +3414,7 @@ static void kfree_rcu_monitor(struct work_struct *work) // of the channels that is still busy we should rearm the // work to repeat an attempt. Because previous batches are // still in progress. - if (!krcp->bkvhead[0] && !krcp->bkvhead[1] && !krcp->head) - krcp->monitor_todo = false; - else + if (krcp->bkvhead[0] || krcp->bkvhead[1] || krcp->head) schedule_delayed_work(&krcp->monitor_work, KFREE_DRAIN_JIFFIES); raw_spin_unlock_irqrestore(&krcp->lock, flags); @@ -3607,10 +3603,8 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func) // Set timer to drain after KFREE_DRAIN_JIFFIES. if (rcu_scheduler_active == RCU_SCHEDULER_RUNNING && - !krcp->monitor_todo) { - krcp->monitor_todo = true; + !delayed_work_pending(&krcp->monitor_work)) schedule_delayed_work(&krcp->monitor_work, KFREE_DRAIN_JIFFIES); - } unlock_return: krc_this_cpu_unlock(krcp, flags); @@ -3685,14 +3679,12 @@ void __init kfree_rcu_scheduler_running(void) struct kfree_rcu_cpu *krcp = per_cpu_ptr(&krc, cpu); raw_spin_lock_irqsave(&krcp->lock, flags); - if ((!krcp->bkvhead[0] && !krcp->bkvhead[1] && !krcp->head) || - krcp->monitor_todo) { - raw_spin_unlock_irqrestore(&krcp->lock, flags); - continue; + if (krcp->bkvhead[0] || krcp->bkvhead[1] || krcp->head) { + if (delayed_work_pending(&krcp->monitor_work)) { + schedule_delayed_work_on(cpu, &krcp->monitor_work, + KFREE_DRAIN_JIFFIES); + } } - krcp->monitor_todo = true; - schedule_delayed_work_on(cpu, &krcp->monitor_work, - KFREE_DRAIN_JIFFIES); raw_spin_unlock_irqrestore(&krcp->lock, flags); } } From patchwork Thu May 12 03:04:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 12846894 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EDFD4C433FE for ; Thu, 12 May 2022 03:05:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244513AbiELDFV (ORCPT ); Wed, 11 May 2022 23:05:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37298 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244562AbiELDFO (ORCPT ); Wed, 11 May 2022 23:05:14 -0400 Received: from mail-qt1-x82c.google.com (mail-qt1-x82c.google.com [IPv6:2607:f8b0:4864:20::82c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2923C1FE1F2 for ; Wed, 11 May 2022 20:05:13 -0700 (PDT) Received: by mail-qt1-x82c.google.com with SMTP id x9so3446261qts.6 for ; Wed, 11 May 2022 20:05:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=9VguBeQ+12qdz5ORPPz8A3LziW7+dOXzz8U6/Bhny2s=; b=g3sd8zXr4eY3YPK7kiTFuglS9ynRQb3gfdlxBVFgd+S29k9TT5QtDXpPjGGe/iRsCz ik9dY9qNonxIBRQhFScSfkZmvPRlt8rhsl09x4lq10E0aAOkHdj4I1y4a3UlHHBG+zYN rcEXbO8LeByperlUP551BQE038Vg4u7MUf0Tc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=9VguBeQ+12qdz5ORPPz8A3LziW7+dOXzz8U6/Bhny2s=; b=JAtMN6MIT8wjk0fUo+hNN+8NycYmnTJRrqHF2uVFX3LcfLBZ3a6RatvLGNe5SmfNK9 aRGnZTd8VyMIOuZgCedq5rfCectCEdkhQVjx3yzKOeV764B6sflDaryfwbkEIF+mtfnX Kc7S+T17hkS0VEhCMefIZUnx8oycsXnBvS+10TXIS3ZQJ/iIgA/dSkPF9POVfLwFQajL 3k++0YQRqYt6NZtSUam1UcUWsRiwPGTl24/rfcH1/gPQ31W88MD647o2kMMbGeYg8URk Gf4AW/5grEd/wyJqK58Xj53AfI5g0MpDF0emEC/q+6ppdRu5OXEp9hbBwgvupjd5wTQr MMsw== X-Gm-Message-State: AOAM53016sj2MuokR9bAE1iDf63IdYDtAIrnopMjLZB9rlm8ZX2xHh/Z GtoOhhaT3Is/koc87rsUmI4BSSHsFTDJsA== X-Google-Smtp-Source: ABdhPJy3hdaEwQV6kb236TcQN6B2ZwyBAK4FV8lCkQvL4nZHeVDWGM7FQhBwpyoQmxmXzFO1h0ZTIA== X-Received: by 2002:a05:622a:15cb:b0:2f3:f90f:1939 with SMTP id d11-20020a05622a15cb00b002f3f90f1939mr815053qty.374.1652324712078; Wed, 11 May 2022 20:05:12 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (29.46.245.35.bc.googleusercontent.com. [35.245.46.29]) by smtp.gmail.com with ESMTPSA id h124-20020a376c82000000b0069fc13ce203sm2270334qkc.52.2022.05.11.20.05.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 May 2022 20:05:11 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, "Joel Fernandes (Google)" Subject: [RFC v1 13/14] rcu/kfree: Fix kfree_rcu_shrink_count() return value Date: Thu, 12 May 2022 03:04:41 +0000 Message-Id: <20220512030442.2530552-14-joel@joelfernandes.org> X-Mailer: git-send-email 2.36.0.550.gb090851708-goog In-Reply-To: <20220512030442.2530552-1-joel@joelfernandes.org> References: <20220512030442.2530552-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org As per the comments in include/linux/shrinker.h, .count_objects callback should return the number of freeable items, but if there are no objects to free, SHRINK_EMPTY should be returned. The only time 0 is returned should be when we are unable to determine the number of objects, or the cache should be skipped for another reason. Signed-off-by: Joel Fernandes (Google) Reviewed-by: Uladzislau Rezki (Sony) --- kernel/rcu/tree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 3828ac3bf1c4..f191542cdf5e 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -3637,7 +3637,7 @@ kfree_rcu_shrink_count(struct shrinker *shrink, struct shrink_control *sc) atomic_set(&krcp->backoff_page_cache_fill, 1); } - return count; + return count == 0 ? SHRINK_EMPTY : count; } static unsigned long From patchwork Thu May 12 03:04:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 12846895 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 55DC1C4332F for ; Thu, 12 May 2022 03:05:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244529AbiELDFW (ORCPT ); Wed, 11 May 2022 23:05:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37344 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244586AbiELDFO (ORCPT ); Wed, 11 May 2022 23:05:14 -0400 Received: from mail-qt1-x82a.google.com (mail-qt1-x82a.google.com [IPv6:2607:f8b0:4864:20::82a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6CD4B200F5C for ; Wed, 11 May 2022 20:05:13 -0700 (PDT) Received: by mail-qt1-x82a.google.com with SMTP id u35so3409694qtc.13 for ; Wed, 11 May 2022 20:05:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=krloPx/WxS98VS1vS96cNgmL7YiotWv16x5KX/rZNwk=; b=erC2NorT3UzdBeIUO7Tvs9XfYymqMTGkuZ665aGCZw/PFXnbYwEscWPhhJC/mnxK/s w3JKXgLjq0AgUbIlDC/AUpxW8yVKgCiFP5aTIvogTkVO6KRC1w2mgp6TZGWEHDrrrybG XFc1so/fm/wXMD1/yzvfyYjXGDU07bh8fxLx4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=krloPx/WxS98VS1vS96cNgmL7YiotWv16x5KX/rZNwk=; b=eaK3DkXKQQcb82ILMJuXSAxFQSjTvzsW8tQMkq4tSwDJe7E2LBSb3gzGyysRAQZnPo HNy/NXBs0qUeMYF6pusMa7yXCpU0Za6sNEy+bzvb7L1nDSpOcM0tJTFQw6+5CE3LXN/c 7VNGuMDRzDdNo5M5tC5UAS9O2R+6HTkt4Vbb4yk04PQdVrYSn3mZHC93B97r2Riie8eQ tgt/qSFKLYKMMRNrGZ1DJidI4Ldy6Bf22Sqnq52f6Vv7nH2ecYFALHRoqzJOic8xuaih 9zYRpKbR3o2u3W4v5HtPupT8jIBb0pHb1Icv6sAeYCMjLxoh5arIdtxFAvK0cMJZTum3 TSsw== X-Gm-Message-State: AOAM533NYnqMXT5fQ2V9z70aJ7YBFMliWTv8iBNTGHM298+REgDqh3aD LwPEExtlRQ27GBT5ML2VMNq2DjaeLugMog== X-Google-Smtp-Source: ABdhPJxpFwDKzLMJIvkSUabCAm16C4Iznxq4wawVZ64fhCyJqTNvXd8da+90kv5Zp31aq4CB0h68Aw== X-Received: by 2002:ac8:7dcf:0:b0:2f3:d53a:add2 with SMTP id c15-20020ac87dcf000000b002f3d53aadd2mr18976974qte.128.1652324712862; Wed, 11 May 2022 20:05:12 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (29.46.245.35.bc.googleusercontent.com. [35.245.46.29]) by smtp.gmail.com with ESMTPSA id h124-20020a376c82000000b0069fc13ce203sm2270334qkc.52.2022.05.11.20.05.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 May 2022 20:05:12 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, "Joel Fernandes (Google)" Subject: [RFC v1 14/14] DEBUG: Toggle rcu_lazy and tune at runtime Date: Thu, 12 May 2022 03:04:42 +0000 Message-Id: <20220512030442.2530552-15-joel@joelfernandes.org> X-Mailer: git-send-email 2.36.0.550.gb090851708-goog In-Reply-To: <20220512030442.2530552-1-joel@joelfernandes.org> References: <20220512030442.2530552-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org Add sysctl knobs just for easier debugging/testing, to tune the maximum batch size, maximum time to wait before flush, and turning off the feature entirely. Signed-off-by: Joel Fernandes (Google) --- include/linux/sched/sysctl.h | 4 ++++ kernel/rcu/lazy.c | 12 ++++++++++-- kernel/sysctl.c | 23 +++++++++++++++++++++++ 3 files changed, 37 insertions(+), 2 deletions(-) diff --git a/include/linux/sched/sysctl.h b/include/linux/sched/sysctl.h index c19dd5a2c05c..55ffc61beed1 100644 --- a/include/linux/sched/sysctl.h +++ b/include/linux/sched/sysctl.h @@ -16,6 +16,10 @@ enum { sysctl_hung_task_timeout_secs = 0 }; extern unsigned int sysctl_sched_child_runs_first; +extern unsigned int sysctl_rcu_lazy; +extern unsigned int sysctl_rcu_lazy_batch; +extern unsigned int sysctl_rcu_lazy_jiffies; + enum sched_tunable_scaling { SCHED_TUNABLESCALING_NONE, SCHED_TUNABLESCALING_LOG, diff --git a/kernel/rcu/lazy.c b/kernel/rcu/lazy.c index 55e406cfc528..0af9fb67c92b 100644 --- a/kernel/rcu/lazy.c +++ b/kernel/rcu/lazy.c @@ -12,6 +12,10 @@ // How much to wait before flushing? #define MAX_LAZY_JIFFIES 10000 +unsigned int sysctl_rcu_lazy_batch = MAX_LAZY_BATCH; +unsigned int sysctl_rcu_lazy_jiffies = MAX_LAZY_JIFFIES; +unsigned int sysctl_rcu_lazy = 1; + // We cast lazy_rcu_head to rcu_head and back. This keeps the API simple while // allowing us to use lockless list node in the head. Also, we use BUILD_BUG_ON // later to ensure that rcu_head and lazy_rcu_head are of the same size. @@ -49,6 +53,10 @@ void call_rcu_lazy(struct rcu_head *head_rcu, rcu_callback_t func) struct lazy_rcu_head *head = (struct lazy_rcu_head *)head_rcu; struct rcu_lazy_pcp *rlp; + if (!sysctl_rcu_lazy) { + return call_rcu(head_rcu, func); + } + preempt_disable(); rlp = this_cpu_ptr(&rcu_lazy_pcp_ins); preempt_enable(); @@ -67,11 +75,11 @@ void call_rcu_lazy(struct rcu_head *head_rcu, rcu_callback_t func) llist_add(&head->llist_node, &rlp->head); // Flush queue if too big - if (atomic_inc_return(&rlp->count) >= MAX_LAZY_BATCH) { + if (atomic_inc_return(&rlp->count) >= sysctl_rcu_lazy_batch) { lazy_rcu_flush_cpu(rlp); } else { if (!delayed_work_pending(&rlp->work)) { - schedule_delayed_work(&rlp->work, MAX_LAZY_JIFFIES); + schedule_delayed_work(&rlp->work, sysctl_rcu_lazy_jiffies); } } } diff --git a/kernel/sysctl.c b/kernel/sysctl.c index 5ae443b2882e..2ba830ca71ec 100644 --- a/kernel/sysctl.c +++ b/kernel/sysctl.c @@ -1659,6 +1659,29 @@ static struct ctl_table kern_table[] = { .mode = 0644, .proc_handler = proc_dointvec, }, +#ifdef CONFIG_RCU_LAZY + { + .procname = "rcu_lazy", + .data = &sysctl_rcu_lazy, + .maxlen = sizeof(unsigned int), + .mode = 0644, + .proc_handler = proc_dointvec, + }, + { + .procname = "rcu_lazy_batch", + .data = &sysctl_rcu_lazy_batch, + .maxlen = sizeof(unsigned int), + .mode = 0644, + .proc_handler = proc_dointvec, + }, + { + .procname = "rcu_lazy_jiffies", + .data = &sysctl_rcu_lazy_jiffies, + .maxlen = sizeof(unsigned int), + .mode = 0644, + .proc_handler = proc_dointvec, + }, +#endif #ifdef CONFIG_SCHEDSTATS { .procname = "sched_schedstats",