From patchwork Wed Jun 22 22:50:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 12891550 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 24718C43334 for ; Wed, 22 Jun 2022 22:51:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232538AbiFVWvp (ORCPT ); Wed, 22 Jun 2022 18:51:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45594 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237106AbiFVWvm (ORCPT ); Wed, 22 Jun 2022 18:51:42 -0400 Received: from mail-qv1-xf35.google.com (mail-qv1-xf35.google.com [IPv6:2607:f8b0:4864:20::f35]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AD9FD40E72 for ; Wed, 22 Jun 2022 15:51:41 -0700 (PDT) Received: by mail-qv1-xf35.google.com with SMTP id c1so27038065qvi.11 for ; Wed, 22 Jun 2022 15:51:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=K8zLqAuQGGInMEO/tC5oFW9jqhiZEM0UzbeIkjAgT/E=; b=u6ATeb3gtaca1zArGlTV20bvGnJYCnyGi0lrVTh4JYP4rTbIjKfKdeM9RDdwdI9RAi ujrJtVDrjaCtmGg5lfrv26gOYUGzGdvprVDIpOwUgQm35vlAlgg6z/uQvruLfpgAqFnf LBh4FHH8YPXW9tWhspYhEKaA5r2GbBI4MSxtM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=K8zLqAuQGGInMEO/tC5oFW9jqhiZEM0UzbeIkjAgT/E=; b=ZKLZ3zlsw3JePG8KjL54IN2+mDNCFkV/BdCsEVjYQxlarExWLdxwEji6k329RdrO6/ 39ifMwCsmxoonqT5/qxdshl21x5JjKpSl+bFyHyKpJMI6C1+qdVSEuqsFNXo/uuCFed8 qXXGqh/PpZ87TKgaZNAjOXkrHDAngAz7BujCy7r+LGkbnv6UGlZLwgpBks78JvA1z0ys QHBf7rT9CQxVeOhyY+ljmEYqRneFduRriFBTTocT5czKNMS+0M8omhDG4WRUdL5oYQV/ Ywln4zUefwLC+iPzZBJKbb9VsH6AcuM5WKo/baYZSWpZXIXEqEeYV3WcMc0NNLoziNsK sg+A== X-Gm-Message-State: AJIora81EDd01kRALJRkamaPi71bcY2qV49gcIDRvt67NLDQkBNoAqlr gxvLe0R2kbD2LtE3CA3CZ8WtGL1+Rs95yg== X-Google-Smtp-Source: AGRyM1vNLmCVtta0shMDBQ0BoHzenJeOoo6ATvfQjsRBrP8Iu05+Inum8C5HKWzZ9QzXs2v3TExN7g== X-Received: by 2002:a05:6214:2501:b0:470:3f6d:c075 with SMTP id gf1-20020a056214250100b004703f6dc075mr16486903qvb.95.1655938300584; Wed, 22 Jun 2022 15:51:40 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id k66-20020a37a145000000b006ab91fd03fasm1794022qke.19.2022.06.22.15.51.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Jun 2022 15:51:39 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, Marco Elver Subject: [PATCH v2 1/1] context_tracking: Use arch_atomic_read() in __ct_state for KASAN Date: Wed, 22 Jun 2022 22:50:54 +0000 Message-Id: <20220622225102.2112026-2-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.0.rc0.104.g0611611a94-goog In-Reply-To: <20220622225102.2112026-1-joel@joelfernandes.org> References: <20220622225102.2112026-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org From: "Paul E. McKenney" Context tracking's __ct_state() function can be invoked from noinstr state where RCU is not watching. This means that its use of atomic_read() causes KASAN to invoke the non-noinstr __kasan_check_read() function from the noinstr function __ct_state(). This is problematic because someone tracing the __kasan_check_read() function could get a nasty surprise because of RCU not watching. This commit therefore replaces the __ct_state() function's use of atomic_read() with arch_atomic_read(), which KASAN does not attempt to add instrumention to. Signed-off-by: Paul E. McKenney Cc: Frederic Weisbecker Cc: Marco Elver Reviewed-by: Marco Elver --- include/linux/context_tracking_state.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/linux/context_tracking_state.h b/include/linux/context_tracking_state.h index 0aecc07fb4f5..81c51e5f0314 100644 --- a/include/linux/context_tracking_state.h +++ b/include/linux/context_tracking_state.h @@ -49,7 +49,7 @@ DECLARE_PER_CPU(struct context_tracking, context_tracking); static __always_inline int __ct_state(void) { - return atomic_read(this_cpu_ptr(&context_tracking.state)) & CT_STATE_MASK; + return arch_atomic_read(this_cpu_ptr(&context_tracking.state)) & CT_STATE_MASK; } #endif From patchwork Wed Jun 22 22:50:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 12891553 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 79F16CCA47F for ; Wed, 22 Jun 2022 22:51:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1359766AbiFVWvs (ORCPT ); Wed, 22 Jun 2022 18:51:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45634 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347009AbiFVWvq (ORCPT ); Wed, 22 Jun 2022 18:51:46 -0400 Received: from mail-qv1-xf36.google.com (mail-qv1-xf36.google.com [IPv6:2607:f8b0:4864:20::f36]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0BFC540E72 for ; Wed, 22 Jun 2022 15:51:44 -0700 (PDT) Received: by mail-qv1-xf36.google.com with SMTP id 59so4714759qvb.3 for ; Wed, 22 Jun 2022 15:51:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=tZ2VmzE2lHGW4TDS5ibElbKctouxZ1GAoFxv9DWSNFQ=; b=uAvtO1wTymA1OKGzMjhdAg4Xrl4FVIKHWotoJraXcn07le/b0XJ7l01X76a8WK4Q90 kmUbwQlNMXoi/6t3IXkCKFj3BN/t49X5NpYdWKYhWTCqUvfHy+wkYe934MhNMPgDiTFa hN6xqJX72Ktoan5doV3c3TtrMl4Bo7T3X2iuo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=tZ2VmzE2lHGW4TDS5ibElbKctouxZ1GAoFxv9DWSNFQ=; b=aNCa1weMnednvKaDxuHLapU7votxf7ESZylE0nZFoKqWfCmIV39UmY4wzNAYNHhIPx dG1A6OcgyNCKXWnyYAS6Rf3xiTQ5LN2VTvqgX3WO8es72KIXorrUefTUglixt5kbOPIb Nfv4N51z24+8ecdqRRsoW/vLSqxFDbGiNyAHO5ki/G6Dc67edJawsaa6TsEmsZeiyBoJ o9sbQFC5NjoNADJ9ygkz7UzpON6HLGnn/nQlcJgAfVFPEgSijMKurYYgL8MUz7KQtYZu DLURW17yt2bH8J0WNFZ8Vv/67t/Fuzmi6omyUKa7GXTn3tQOi2hmsM2aSaFeQWh6Vcx5 N0hw== X-Gm-Message-State: AJIora9vERdp3XcdbUzAIwYUhTfUqH31gErSC/1jqzhfrOgKQ+T3efBz 5lSsJuKF2TFvuZizKbFS7lDC/JiTOg8KJQ== X-Google-Smtp-Source: AGRyM1tPJC4HgfQNhb89caYKVnJktcVYAzMTrPx9I8PteeedjGdtVYdEKZzoNlG7gLoLRzY4NHCDLQ== X-Received: by 2002:a05:6214:f08:b0:470:519a:168a with SMTP id gw8-20020a0562140f0800b00470519a168amr9962716qvb.29.1655938303002; Wed, 22 Jun 2022 15:51:43 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id k66-20020a37a145000000b006ab91fd03fasm1794022qke.19.2022.06.22.15.51.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Jun 2022 15:51:42 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, Joel Fernandes Subject: [PATCH v2 2/8] rcu: shrinker for lazy rcu Date: Wed, 22 Jun 2022 22:50:56 +0000 Message-Id: <20220622225102.2112026-4-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.0.rc0.104.g0611611a94-goog In-Reply-To: <20220622225102.2112026-1-joel@joelfernandes.org> References: <20220622225102.2112026-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org From: Vineeth Pillai The shrinker is used to speed up the free'ing of memory potentially held by RCU lazy callbacks. RCU kernel module test cases show this to be effective. Test is introduced in a later patch. Signed-off-by: Vineeth Pillai Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/rcu_segcblist.h | 14 +++++++++-- kernel/rcu/tree_nocb.h | 48 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 60 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/rcu_segcblist.h b/kernel/rcu/rcu_segcblist.h index c3d7de65b689..cf71425dbb5e 100644 --- a/kernel/rcu/rcu_segcblist.h +++ b/kernel/rcu/rcu_segcblist.h @@ -15,16 +15,26 @@ static inline long rcu_cblist_n_cbs(struct rcu_cblist *rclp) return READ_ONCE(rclp->len); } +#ifdef CONFIG_RCU_LAZY /* Return number of callbacks in the specified callback list. */ static inline long rcu_cblist_n_lazy_cbs(struct rcu_cblist *rclp) { -#ifdef CONFIG_RCU_LAZY return READ_ONCE(rclp->lazy_len); +} + +static inline void rcu_cblist_reset_lazy_len(struct rcu_cblist *rclp) +{ + WRITE_ONCE(rclp->lazy_len, 0); +} #else +static inline long rcu_cblist_n_lazy_cbs(struct rcu_cblist *rclp) +{ return 0; -#endif } +static inline void rcu_cblist_reset_lazy_len(struct rcu_cblist *rclp) {} +#endif + /* Return number of callbacks in segmented callback list by summing seglen. */ long rcu_segcblist_n_segment_cbs(struct rcu_segcblist *rsclp); diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index b9244f22e102..2f5da12811a5 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1207,6 +1207,51 @@ int rcu_nocb_cpu_offload(int cpu) } EXPORT_SYMBOL_GPL(rcu_nocb_cpu_offload); +static unsigned long +lazy_rcu_shrink_count(struct shrinker *shrink, struct shrink_control *sc) { + int cpu; + unsigned long count = 0; + + /* Snapshot count of all CPUs */ + for_each_possible_cpu(cpu) { + struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); + count += rcu_cblist_n_lazy_cbs(&rdp->nocb_bypass); + } + + return count ? count : SHRINK_EMPTY; +} + +static unsigned long +lazy_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) { + int cpu; + unsigned long flags; + unsigned long count = 0; + + /* Snapshot count of all CPUs */ + for_each_possible_cpu(cpu) { + struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); + int _count = rcu_cblist_n_lazy_cbs(&rdp->nocb_bypass); + if (_count == 0) + continue; + rcu_nocb_lock_irqsave(rdp, flags); + rcu_cblist_reset_lazy_len(&rdp->nocb_bypass); + rcu_nocb_unlock_irqrestore(rdp, flags); + wake_nocb_gp(rdp, false); + sc->nr_to_scan -= _count; + count += _count; + if (sc->nr_to_scan <= 0) + break; + } + return count ? count : SHRINK_STOP; +} + +static struct shrinker lazy_rcu_shrinker = { + .count_objects = lazy_rcu_shrink_count, + .scan_objects = lazy_rcu_shrink_scan, + .batch = 0, + .seeks = DEFAULT_SEEKS, +}; + void __init rcu_init_nohz(void) { int cpu; @@ -1244,6 +1289,9 @@ void __init rcu_init_nohz(void) if (!rcu_state.nocb_is_setup) return; + if (register_shrinker(&lazy_rcu_shrinker)) + pr_err("Failed to register lazy_rcu shrinker!\n"); + #if defined(CONFIG_NO_HZ_FULL) if (tick_nohz_full_running) cpumask_or(rcu_nocb_mask, rcu_nocb_mask, tick_nohz_full_mask); From patchwork Wed Jun 22 22:50:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 12891552 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 36B21C433EF for ; Wed, 22 Jun 2022 22:51:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1359741AbiFVWvs (ORCPT ); Wed, 22 Jun 2022 18:51:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45656 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237106AbiFVWvq (ORCPT ); Wed, 22 Jun 2022 18:51:46 -0400 Received: from mail-qv1-xf29.google.com (mail-qv1-xf29.google.com [IPv6:2607:f8b0:4864:20::f29]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AA8F741303 for ; Wed, 22 Jun 2022 15:51:44 -0700 (PDT) Received: by mail-qv1-xf29.google.com with SMTP id q4so15321643qvq.8 for ; Wed, 22 Jun 2022 15:51:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=H+kSqBl3RB9I7OY9hm0XkeWhykcK3oDF7t29sGiTzsw=; b=UXqJkdf/KUdA/7F69IfFzfqaGez+SJKEsYw7nuoWGBu0X1hTbSLO5mv5SvEHwxyffi v6O/+XZ6kAy6ihIhq5dQgHhxeYv1dabpAGdmOWyyEl2imSX0RJXhmRwGEI5Yg9J0bB3R MPHCpMk4uVaFRmEAqQ1tWKl/a8e4oKq/zkQgE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=H+kSqBl3RB9I7OY9hm0XkeWhykcK3oDF7t29sGiTzsw=; b=nje2ki/z/V748mxmvLis2Vlh3cuFAIfmApmCLcnGK2xGFaGNocPkF0JLXxuKzPlfxh s20tGqYBiwJcg2rAS+0OpshQusJcZX39bM/Hb3+lggRnZTi3+Sb5Qq/rNL23gT4qchLs z87AUXG/wzl/J84gMNcVPTmc8JWGuYXJQO+F785pXi5k2x0eZE3+Pb6P7fJdR6oW1VJx LCLC1QhAUrX48gH6+CHA9+20brJkf6x2JnmOT840cg4qL6TYWqvFk7Q70d9tZ+03vM4O 2vA10Rk9eFxddcL2iQwu6en0eNyOo2iB+JbnkW8rvY4m73q+Z7wtIIU0j1eTrRD2UQiX uzEQ== X-Gm-Message-State: AJIora/dM3VVJ4MIVWWX8URpT7A74LvOoQ4u3ImYgwRqyVv9eaN38I1U 3v8ncAuFrBToqUD1zewsVJ9UbPABmNHJEg== X-Google-Smtp-Source: AGRyM1uuVGVPfaJ0Ie+qNySjbSbMPeGHjQQx1Ig5cgUKgMzyi+kfvrEg7TFmRbp9baN9SneC9y1kag== X-Received: by 2002:ac8:594d:0:b0:305:1a14:de6c with SMTP id 13-20020ac8594d000000b003051a14de6cmr5135737qtz.677.1655938303882; Wed, 22 Jun 2022 15:51:43 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id k66-20020a37a145000000b006ab91fd03fasm1794022qke.19.2022.06.22.15.51.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Jun 2022 15:51:43 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, "Joel Fernandes (Google)" Subject: [PATCH v2 3/8] fs: Move call_rcu() to call_rcu_lazy() in some paths Date: Wed, 22 Jun 2022 22:50:57 +0000 Message-Id: <20220622225102.2112026-5-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.0.rc0.104.g0611611a94-goog In-Reply-To: <20220622225102.2112026-1-joel@joelfernandes.org> References: <20220622225102.2112026-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org This is required to prevent callbacks triggering RCU machinery too quickly and too often, which adds more power to the system. When testing, we found that these paths were invoked often when the system is not doing anything (screen is ON but otherwise idle). Signed-off-by: Joel Fernandes (Google) --- fs/dcache.c | 4 ++-- fs/eventpoll.c | 2 +- fs/file_table.c | 2 +- fs/inode.c | 2 +- 4 files changed, 5 insertions(+), 5 deletions(-) diff --git a/fs/dcache.c b/fs/dcache.c index 93f4f5ee07bf..7f51bac390c8 100644 --- a/fs/dcache.c +++ b/fs/dcache.c @@ -366,7 +366,7 @@ static void dentry_free(struct dentry *dentry) if (unlikely(dname_external(dentry))) { struct external_name *p = external_name(dentry); if (likely(atomic_dec_and_test(&p->u.count))) { - call_rcu(&dentry->d_u.d_rcu, __d_free_external); + call_rcu_lazy(&dentry->d_u.d_rcu, __d_free_external); return; } } @@ -374,7 +374,7 @@ static void dentry_free(struct dentry *dentry) if (dentry->d_flags & DCACHE_NORCU) __d_free(&dentry->d_u.d_rcu); else - call_rcu(&dentry->d_u.d_rcu, __d_free); + call_rcu_lazy(&dentry->d_u.d_rcu, __d_free); } /* diff --git a/fs/eventpoll.c b/fs/eventpoll.c index 971f98af48ff..57b3f781760c 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -729,7 +729,7 @@ static int ep_remove(struct eventpoll *ep, struct epitem *epi) * ep->mtx. The rcu read side, reverse_path_check_proc(), does not make * use of the rbn field. */ - call_rcu(&epi->rcu, epi_rcu_free); + call_rcu_lazy(&epi->rcu, epi_rcu_free); percpu_counter_dec(&ep->user->epoll_watches); diff --git a/fs/file_table.c b/fs/file_table.c index 5424e3a8df5f..417f57e9cb30 100644 --- a/fs/file_table.c +++ b/fs/file_table.c @@ -56,7 +56,7 @@ static inline void file_free(struct file *f) security_file_free(f); if (!(f->f_mode & FMODE_NOACCOUNT)) percpu_counter_dec(&nr_files); - call_rcu(&f->f_u.fu_rcuhead, file_free_rcu); + call_rcu_lazy(&f->f_u.fu_rcuhead, file_free_rcu); } /* diff --git a/fs/inode.c b/fs/inode.c index bd4da9c5207e..38fe040ddbd6 100644 --- a/fs/inode.c +++ b/fs/inode.c @@ -312,7 +312,7 @@ static void destroy_inode(struct inode *inode) return; } inode->free_inode = ops->free_inode; - call_rcu(&inode->i_rcu, i_callback); + call_rcu_lazy(&inode->i_rcu, i_callback); } /** From patchwork Wed Jun 22 22:50:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 12891554 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D560FCCA480 for ; Wed, 22 Jun 2022 22:51:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1359799AbiFVWvt (ORCPT ); Wed, 22 Jun 2022 18:51:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45668 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1358620AbiFVWvr (ORCPT ); Wed, 22 Jun 2022 18:51:47 -0400 Received: from mail-qv1-xf2a.google.com (mail-qv1-xf2a.google.com [IPv6:2607:f8b0:4864:20::f2a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 01D2141638 for ; Wed, 22 Jun 2022 15:51:46 -0700 (PDT) Received: by mail-qv1-xf2a.google.com with SMTP id o43so27420978qvo.4 for ; Wed, 22 Jun 2022 15:51:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=czRWmqXwJRiSEH25SM5GYb5Yl+/vPCw6VIvQPTxRUu8=; b=TffKA6dV8lgb0owObCDYZAoszUr3Wlohvc9Oe7lMyM8i3Tp+lw7j5I5jYbuBeUeBHr hw6rj2Ic8XvDgkEB7GWWhmAB28oH0ZodN41wjG+/n3WraZU8356yTc+2MMq9mW58RFg1 SJPUS+ANKIuHUETM3nyiMJpm3QA/edyOhnM0M= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=czRWmqXwJRiSEH25SM5GYb5Yl+/vPCw6VIvQPTxRUu8=; b=i7RNh0Eji4SWdU7GGIoDEdoCqG8r7DwGeluFokYMhZYE6OalyihpV4MEw2OR36Oxsi X1v1/9wMfAMdcCZgDswC8AGvvPrrqnhcZh21fta53A8GMxUHb1FLgEvnT1+cPMqj4Cv9 qELBlFPnokfDCKi9iueEIO0Ev+S/I1QpBOzDhCH0tNymzOh5G9sJbx2/pDMU1nnxIsP0 Fi7cOjn+xl1EBISG01D/BmOXijXMn1TFTvdNCfn5pioBW6XTxG5F0W5Jx1lSnlEWczSY qHf22ClrJrm5Gyyz/dPXEZgyfxT4ir3eY2FIw23O31LvzoE3FmZPl4p5iWsmQ3I29IkS GwhQ== X-Gm-Message-State: AJIora9dOzrMW06ovfInrVb6Ku31PVEVNXw1hvV7wV+/LIth5ED9XzAi Ur+sHBoX++h9EF5wbKOeFdY3cQTra9jnrg== X-Google-Smtp-Source: AGRyM1uo/5L9en8Zb8UJ/9q6iw0BqvqvJQYPuudXxkgc81Cjbz58Q0qLgY+GAxCRwsEEJphkgx2Ofg== X-Received: by 2002:a05:6214:ac7:b0:470:6dd0:c3da with SMTP id g7-20020a0562140ac700b004706dd0c3damr3578574qvi.43.1655938304958; Wed, 22 Jun 2022 15:51:44 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id k66-20020a37a145000000b006ab91fd03fasm1794022qke.19.2022.06.22.15.51.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Jun 2022 15:51:44 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, "Joel Fernandes (Google)" Subject: [PATCH v2 4/8] rcu/nocb: Add option to force all call_rcu() to lazy Date: Wed, 22 Jun 2022 22:50:58 +0000 Message-Id: <20220622225102.2112026-6-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.0.rc0.104.g0611611a94-goog In-Reply-To: <20220622225102.2112026-1-joel@joelfernandes.org> References: <20220622225102.2112026-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org This will be used in the rcu scale test, to ensure that fly-by call_rcu()s do no cause call_rcu_lazy() CBs to be flushed to the rdp ->cblist. Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/rcu.h | 2 ++ kernel/rcu/tree.c | 11 ++++++++++- 2 files changed, 12 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/rcu.h b/kernel/rcu/rcu.h index 4916077119f3..71c0f45e70c3 100644 --- a/kernel/rcu/rcu.h +++ b/kernel/rcu/rcu.h @@ -472,6 +472,7 @@ void do_trace_rcu_torture_read(const char *rcutorturename, unsigned long c_old, unsigned long c); void rcu_gp_set_torture_wait(int duration); +void rcu_force_call_rcu_to_lazy(bool force); #else static inline void rcutorture_get_gp_data(enum rcutorture_type test_type, int *flags, unsigned long *gp_seq) @@ -490,6 +491,7 @@ void do_trace_rcu_torture_read(const char *rcutorturename, do { } while (0) #endif static inline void rcu_gp_set_torture_wait(int duration) { } +static inline void rcu_force_call_rcu_to_lazy(bool force) { } #endif #if IS_ENABLED(CONFIG_RCU_TORTURE_TEST) || IS_MODULE(CONFIG_RCU_TORTURE_TEST) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index d2e3d6e176d2..711679d10cbb 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -3171,9 +3171,18 @@ void call_rcu_lazy(struct rcu_head *head, rcu_callback_t func) EXPORT_SYMBOL_GPL(call_rcu_lazy); #endif +static bool force_call_rcu_to_lazy; + +void rcu_force_call_rcu_to_lazy(bool force) +{ + if (IS_ENABLED(CONFIG_RCU_SCALE_TEST)) + WRITE_ONCE(force_call_rcu_to_lazy, force); +} +EXPORT_SYMBOL_GPL(rcu_force_call_rcu_to_lazy); + void call_rcu(struct rcu_head *head, rcu_callback_t func) { - return __call_rcu_common(head, func, false); + return __call_rcu_common(head, func, force_call_rcu_to_lazy); } EXPORT_SYMBOL_GPL(call_rcu); From patchwork Wed Jun 22 22:50:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 12891555 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56911C43334 for ; Wed, 22 Jun 2022 22:51:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1359797AbiFVWvt (ORCPT ); Wed, 22 Jun 2022 18:51:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45680 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1358911AbiFVWvr (ORCPT ); Wed, 22 Jun 2022 18:51:47 -0400 Received: from mail-qv1-xf30.google.com (mail-qv1-xf30.google.com [IPv6:2607:f8b0:4864:20::f30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 69E2B419A0 for ; Wed, 22 Jun 2022 15:51:46 -0700 (PDT) Received: by mail-qv1-xf30.google.com with SMTP id n15so23395363qvh.12 for ; Wed, 22 Jun 2022 15:51:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Vd4a4Sb5uy4qMpkEMdY0l97I8Oa96Mx9jouj7Tqv17A=; b=m94cURFdCK6y/WgXeNVeJSsjpwtrlSDB3PxHYiJdPWnOgrL7pP1TMYNYz1W7b+MV+D o4mjJJIuJZdtP6EZIKXzc4DK8WuAV8kIBrLtnHGBvt0N/j4sCit/6477vompbS+f68Rl jrHVbbjKj6qyxBdoOzjlkNClgq4YCQxnGirjQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Vd4a4Sb5uy4qMpkEMdY0l97I8Oa96Mx9jouj7Tqv17A=; b=2XargAtss0H8ElnLeYUy7nxeucu1f9HBSQxSH50p4DsMarCaeiPgz2bWU7OuN2YLxq OxuOmL5uVzoNMIm2vHqxhGKfvCXnDC+Yz1dEtEQEPPFXuyOYE2RWaQMDCmNlDW0EoOmF nu+z6zee7Cr2/ru0eDSpDnyYheDi6w1ysKWA5WE6imaTsFmE0hw9PMqyhoOlk8vSW6I+ uBaIFmJfuGDsKknOiatlE9DXC4SnULu08YO2QV266m5RT2NTN0mdIWT0hFimqSe5RS1D z9yMkFaTpX55sifTKDzHa+1cUijXr9Dv7r9kOszGJsP/YRJ6ydOKNBlVPRS5AjKn8ERe t/vg== X-Gm-Message-State: AJIora/uBghwx6AHYdqWQt60bogUs4ZO1k3j0UXdWYsaMUo1hphxjFKi xAAZi+ycY3LHcEGbJvutiRiOrv0tjxNh5g== X-Google-Smtp-Source: AGRyM1sVrXdFvKoX00v5/yoAGUyGmQGrFinrp1P16vG+Qt7RLn229XFOM58LGmehXc+yFXMyMumP1g== X-Received: by 2002:a05:622a:148d:b0:305:1a61:1003 with SMTP id t13-20020a05622a148d00b003051a611003mr5335099qtx.269.1655938305958; Wed, 22 Jun 2022 15:51:45 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id k66-20020a37a145000000b006ab91fd03fasm1794022qke.19.2022.06.22.15.51.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Jun 2022 15:51:45 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, "Joel Fernandes (Google)" Subject: [PATCH v2 5/8] rcu/nocb: Wake up gp thread when flushing Date: Wed, 22 Jun 2022 22:50:59 +0000 Message-Id: <20220622225102.2112026-7-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.0.rc0.104.g0611611a94-goog In-Reply-To: <20220622225102.2112026-1-joel@joelfernandes.org> References: <20220622225102.2112026-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org We notice that rcu_barrier() can take a really long time. It appears that this can happen when all CBs are lazy and the timer does not fire yet. So after flushing, nothing wakes up GP thread. This patch forces GP thread to wake when bypass flushing happens, this fixes the rcu_barrier() delays with lazy CBs. Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree_nocb.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 2f5da12811a5..b481f1ea57c0 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -325,6 +325,8 @@ static bool rcu_nocb_do_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp, rcu_segcblist_insert_pend_cbs(&rdp->cblist, &rcl); WRITE_ONCE(rdp->nocb_bypass_first, j); rcu_nocb_bypass_unlock(rdp); + + wake_nocb_gp(rdp, true); return true; } From patchwork Wed Jun 22 22:51:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 12891556 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0CFC0CCA483 for ; Wed, 22 Jun 2022 22:51:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1358620AbiFVWvv (ORCPT ); Wed, 22 Jun 2022 18:51:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45656 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1359757AbiFVWvs (ORCPT ); Wed, 22 Jun 2022 18:51:48 -0400 Received: from mail-qk1-x732.google.com (mail-qk1-x732.google.com [IPv6:2607:f8b0:4864:20::732]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F033040E72 for ; Wed, 22 Jun 2022 15:51:47 -0700 (PDT) Received: by mail-qk1-x732.google.com with SMTP id v6so2948412qkh.2 for ; Wed, 22 Jun 2022 15:51:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=VLpRj7ISmpLc7bYzOd0iAFFn8wSs/6G7/zuKg/d2BV4=; b=uUovH7zphZj5j44vTCjc6fQKvn5vsTqcZUI+oVj1KSCucV7WrcEJQ8xUffNEA3EpXw GDZIrTbjd/nV4pr6O7V+SKKTEYqB8ykmpL9P4Z5A7FiTLFLHAH9e+1C715gu+U1RgZmp p9M3xiILqZ6xnVI8MP1Vs70BvMK0jUzTgAYIs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=VLpRj7ISmpLc7bYzOd0iAFFn8wSs/6G7/zuKg/d2BV4=; b=LNB37VUDhEtJ/GqI5gDNlen20xv31CjV54/d5K2Zg2e4uWBG3I7b/SailMtPzxhGF/ +Wq2rADsAel2ighL+3SeBTmMClm7CWW8BPf2pdou2qH8D7P2CMqrPTXOJASTysG0wagY d0DvviX9wHbVIqS8AcGLlS7yUsQIJyHZ84p4XFDflTE29kM1yueZZuhfyQKFzrJYreZY doQXpyUceu27h3/odVN4HKlsAqskTAxd7CznNbwbR85Pupaua8lb0U/Zt1BRxIcuqnJv yD+1kdCpUIIk2Dwq+lqq92KUqzof6xPXzOsTzhceIcuw8rF7AjrqXogVEgSbTyRHHPDD E2hA== X-Gm-Message-State: AJIora8XydgYxmMO9i8SxT3Yt1V312xYpyUySv9LDhvCkZntyJTUJVlw Yw+Sd/BF2juibUDNYyFxzGHwn7qoUI0Usw== X-Google-Smtp-Source: AGRyM1sPh4hJnzcz0VNfhHC4UeQg+s3UcijJzqY655+yVI85xyoOc6cAptRFlkJHV19NUU17LL9gZg== X-Received: by 2002:a37:c86:0:b0:6ae:ded5:2002 with SMTP id 128-20020a370c86000000b006aeded52002mr3616328qkm.594.1655938306664; Wed, 22 Jun 2022 15:51:46 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id k66-20020a37a145000000b006ab91fd03fasm1794022qke.19.2022.06.22.15.51.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Jun 2022 15:51:46 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, "Joel Fernandes (Google)" Subject: [PATCH v2 6/8] rcuscale: Add test for using call_rcu_lazy() to emulate kfree_rcu() Date: Wed, 22 Jun 2022 22:51:00 +0000 Message-Id: <20220622225102.2112026-8-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.0.rc0.104.g0611611a94-goog In-Reply-To: <20220622225102.2112026-1-joel@joelfernandes.org> References: <20220622225102.2112026-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org Reuse the kfree_rcu() test in order to be able to compare the memory reclaiming properties of call_rcu_lazy() with kfree_rcu(). With this test, we find similar memory footprint and time call_rcu_lazy() free'ing takes compared to kfree_rcu(). Also we confirm that call_rcu_lazy() can survive OOM during extremely frequent calls. If we really push it, i.e. boot system with low memory and compare kfree_rcu() with call_rcu_lazy(), I find that call_rcu_lazy() is more resilient and is much harder to produce OOM as compared to kfree_rcu(). Signed-off-by: Joel Fernandes (Google) Reported-by: kernel test robot Reported-by: kernel test robot Reported-by: kernel test robot --- kernel/rcu/rcu.h | 6 ++++ kernel/rcu/rcuscale.c | 64 +++++++++++++++++++++++++++++++++++++++++- kernel/rcu/tree_nocb.h | 17 ++++++++++- 3 files changed, 85 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/rcu.h b/kernel/rcu/rcu.h index 71c0f45e70c3..436faf80a66b 100644 --- a/kernel/rcu/rcu.h +++ b/kernel/rcu/rcu.h @@ -473,6 +473,12 @@ void do_trace_rcu_torture_read(const char *rcutorturename, unsigned long c); void rcu_gp_set_torture_wait(int duration); void rcu_force_call_rcu_to_lazy(bool force); + +#if IS_ENABLED(CONFIG_RCU_SCALE_TEST) +unsigned long rcu_scale_get_jiffies_till_flush(void); +void rcu_scale_set_jiffies_till_flush(unsigned long j); +#endif + #else static inline void rcutorture_get_gp_data(enum rcutorture_type test_type, int *flags, unsigned long *gp_seq) diff --git a/kernel/rcu/rcuscale.c b/kernel/rcu/rcuscale.c index 277a5bfb37d4..58ee5c2cb37b 100644 --- a/kernel/rcu/rcuscale.c +++ b/kernel/rcu/rcuscale.c @@ -95,6 +95,7 @@ torture_param(int, verbose, 1, "Enable verbose debugging printk()s"); torture_param(int, writer_holdoff, 0, "Holdoff (us) between GPs, zero to disable"); torture_param(int, kfree_rcu_test, 0, "Do we run a kfree_rcu() scale test?"); torture_param(int, kfree_mult, 1, "Multiple of kfree_obj size to allocate."); +torture_param(int, kfree_rcu_by_lazy, 0, "Use call_rcu_lazy() to emulate kfree_rcu()?"); static char *scale_type = "rcu"; module_param(scale_type, charp, 0444); @@ -658,6 +659,13 @@ struct kfree_obj { struct rcu_head rh; }; +/* Used if doing RCU-kfree'ing via call_rcu_lazy(). */ +void kfree_rcu_lazy(struct rcu_head *rh) +{ + struct kfree_obj *obj = container_of(rh, struct kfree_obj, rh); + kfree(obj); +} + static int kfree_scale_thread(void *arg) { @@ -695,6 +703,11 @@ kfree_scale_thread(void *arg) if (!alloc_ptr) return -ENOMEM; + if (kfree_rcu_by_lazy) { + call_rcu_lazy(&(alloc_ptr->rh), kfree_rcu_lazy); + continue; + } + // By default kfree_rcu_test_single and kfree_rcu_test_double are // initialized to false. If both have the same value (false or true) // both are randomly tested, otherwise only the one with value true @@ -737,6 +750,9 @@ kfree_scale_cleanup(void) { int i; + if (kfree_rcu_by_lazy) + rcu_force_call_rcu_to_lazy(false); + if (torture_cleanup_begin()) return; @@ -766,11 +782,55 @@ kfree_scale_shutdown(void *arg) return -EINVAL; } +// Used if doing RCU-kfree'ing via call_rcu_lazy(). +unsigned long jiffies_at_lazy_cb; +struct rcu_head lazy_test1_rh; +int rcu_lazy_test1_cb_called; +void call_rcu_lazy_test1(struct rcu_head *rh) +{ + jiffies_at_lazy_cb = jiffies; + WRITE_ONCE(rcu_lazy_test1_cb_called, 1); +} + static int __init kfree_scale_init(void) { long i; int firsterr = 0; + unsigned long orig_jif, jif_start; + + // Force all call_rcu() to call_rcu_lazy() so that non-lazy CBs + // do not remove laziness of the lazy ones (since the test tries + // to stress call_rcu_lazy() for OOM). + // + // Also, do a quick self-test to ensure laziness is as much as + // expected. + if (kfree_rcu_by_lazy) { + /* do a test to check the timeout. */ + orig_jif = rcu_scale_get_jiffies_till_flush(); + + rcu_force_call_rcu_to_lazy(true); + rcu_scale_set_jiffies_till_flush(2 * HZ); + rcu_barrier(); + + jif_start = jiffies; + jiffies_at_lazy_cb = 0; + call_rcu_lazy(&lazy_test1_rh, call_rcu_lazy_test1); + + smp_cond_load_relaxed(&rcu_lazy_test1_cb_called, VAL == 1); + + rcu_scale_set_jiffies_till_flush(orig_jif); + + if (WARN_ON_ONCE(jiffies_at_lazy_cb - jif_start < 2 * HZ)) { + pr_alert("Lazy CBs are not being lazy as expected!\n"); + return -1; + } + + if (WARN_ON_ONCE(jiffies_at_lazy_cb - jif_start > 3 * HZ)) { + pr_alert("Lazy CBs are being too lazy!\n"); + return -1; + } + } kfree_nrealthreads = compute_real(kfree_nthreads); /* Start up the kthreads. */ @@ -783,7 +843,9 @@ kfree_scale_init(void) schedule_timeout_uninterruptible(1); } - pr_alert("kfree object size=%zu\n", kfree_mult * sizeof(struct kfree_obj)); + pr_alert("kfree object size=%zu, kfree_rcu_by_lazy=%d\n", + kfree_mult * sizeof(struct kfree_obj), + kfree_rcu_by_lazy); kfree_reader_tasks = kcalloc(kfree_nrealthreads, sizeof(kfree_reader_tasks[0]), GFP_KERNEL); diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index b481f1ea57c0..255f2945b0fc 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -257,6 +257,21 @@ static bool wake_nocb_gp(struct rcu_data *rdp, bool force) } #define LAZY_FLUSH_JIFFIES (10 * HZ) +unsigned long jiffies_till_flush = LAZY_FLUSH_JIFFIES; + +#ifdef CONFIG_RCU_SCALE_TEST +void rcu_scale_set_jiffies_till_flush(unsigned long jif) +{ + jiffies_till_flush = jif; +} +EXPORT_SYMBOL(rcu_scale_set_jiffies_till_flush); + +unsigned long rcu_scale_get_jiffies_till_flush(void) +{ + return jiffies_till_flush; +} +EXPORT_SYMBOL(rcu_scale_get_jiffies_till_flush); +#endif /* * Arrange to wake the GP kthread for this NOCB group at some future @@ -275,7 +290,7 @@ static void wake_nocb_gp_defer(struct rcu_data *rdp, int waketype, * of callback storm, no need to wake up too early. */ if (waketype == RCU_NOCB_WAKE_LAZY) { - mod_timer(&rdp_gp->nocb_timer, jiffies + LAZY_FLUSH_JIFFIES); + mod_timer(&rdp_gp->nocb_timer, jiffies + jiffies_till_flush); WRITE_ONCE(rdp_gp->nocb_defer_wakeup, waketype); } else if (waketype == RCU_NOCB_WAKE_BYPASS) { mod_timer(&rdp_gp->nocb_timer, jiffies + 2); From patchwork Wed Jun 22 22:51:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 12891557 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C9BE6C433EF for ; Wed, 22 Jun 2022 22:51:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1359796AbiFVWvx (ORCPT ); Wed, 22 Jun 2022 18:51:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45710 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347009AbiFVWvt (ORCPT ); Wed, 22 Jun 2022 18:51:49 -0400 Received: from mail-qv1-xf32.google.com (mail-qv1-xf32.google.com [IPv6:2607:f8b0:4864:20::f32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EC8DC41302 for ; Wed, 22 Jun 2022 15:51:48 -0700 (PDT) Received: by mail-qv1-xf32.google.com with SMTP id o43so27421109qvo.4 for ; Wed, 22 Jun 2022 15:51:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=oulAy7NlmqgAX/AG2BIWch0UpN6lWOBq/Xx+LqGxyiU=; b=Zd4Acj6vFejlAuM4LGySk0mn5EpwW09kqb0tPrWdZuUSIL1zdVj6CpnngjAG5CU35U 9yra0A4qcuU3QelVy9Thq3pL7e37068nnzNVRkCJG9aRq+zBR8VCYjCfHgtlXa90Ue4H IGj41tdwPLrWondJBlm13KXtRnLZJprFieFjI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=oulAy7NlmqgAX/AG2BIWch0UpN6lWOBq/Xx+LqGxyiU=; b=XjMsobBfkwY8fjURtmK4/VmmfT1nJ4mh74czP4kS3JWKkcu/robF7DpaGx4NB/eM6f V3y4g8C2ys0k7nVONCoMoLrA5Qxcp0AzFjD8QwXMj/7eVnXv2cdti2xLihvASQtP7QYe IIsAsqeKUQ6YIU4TPEmLXoxUylD0nWP5aki2hXIdXjIQnoqC42YQ+M6TgITL9tTqXt6L z6Opfkyxp1IhOzoM+AB4qd9OH9Sp9/x7C4/XUZqOvsierrkGFJjQeK+5BLTuCGXWV9GF qwq9ktKF3OqCJUo/Kn8JnYrJBwXQqCxBlJyhj/Hgw+Yq4Mn4Htz2uKWsDv1xoV+gYwYq +38A== X-Gm-Message-State: AJIora/fhHHev9Cmrhc7pzOJr7xdKs0gV6GjYqXTBwOoBr1MDuzuOX2i 6pFWjt00rfF/NaVNFgA1b1zdo+ZyhyG7dg== X-Google-Smtp-Source: AGRyM1uNf6I5+fQ6KcX/9MOZDknEktSDE/Z0l3YRPxDg86JuWUz6QknaienDhYHazulLUj/lBR6DPA== X-Received: by 2002:a0c:9066:0:b0:470:2b7a:2078 with SMTP id o93-20020a0c9066000000b004702b7a2078mr22020533qvo.51.1655938307947; Wed, 22 Jun 2022 15:51:47 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id k66-20020a37a145000000b006ab91fd03fasm1794022qke.19.2022.06.22.15.51.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Jun 2022 15:51:47 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, "Joel Fernandes (Google)" Subject: [PATCH v2 7/8] rcu/nocb: Rewrite deferred wake up logic to be more clean Date: Wed, 22 Jun 2022 22:51:01 +0000 Message-Id: <20220622225102.2112026-9-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.0.rc0.104.g0611611a94-goog In-Reply-To: <20220622225102.2112026-1-joel@joelfernandes.org> References: <20220622225102.2112026-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org There are 2 things this function does: 1. modify the gp wake timer. 2. save the value of the strongest requested wake up so far. The strongest is "wake force" and the weakest is "lazy". The existing logic already does the following: 1. if the existing deferred wake is stronger than the requested one (requested in waketype), modify the gp timer to be more in the future. For example, if the existing one is WAKE and the new waketype requested is BYPASS, then the timer is made to expire later than earlier. 2. even though the timer is modified in #1, a weaker waketype does not end up changing rdp->nocb_gp_defer to be weaker. In other words, ->nocb_gp_defer records the strongest waketype requested so far, even though the timer may or may not be the soonest expiry possible. For simplicity, we write this logic using switch statements and consolidate some of the timer modification operations. Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree_nocb.h | 35 ++++++++++++++++++++++++----------- 1 file changed, 24 insertions(+), 11 deletions(-) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 255f2945b0fc..67b0bd5d233a 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -282,6 +282,7 @@ static void wake_nocb_gp_defer(struct rcu_data *rdp, int waketype, { unsigned long flags; struct rcu_data *rdp_gp = rdp->nocb_gp_rdp; + unsigned long mod_jif = 0; raw_spin_lock_irqsave(&rdp_gp->nocb_gp_lock, flags); @@ -289,19 +290,31 @@ static void wake_nocb_gp_defer(struct rcu_data *rdp, int waketype, * Bypass wakeup overrides previous deferments. In case * of callback storm, no need to wake up too early. */ - if (waketype == RCU_NOCB_WAKE_LAZY) { - mod_timer(&rdp_gp->nocb_timer, jiffies + jiffies_till_flush); - WRITE_ONCE(rdp_gp->nocb_defer_wakeup, waketype); - } else if (waketype == RCU_NOCB_WAKE_BYPASS) { - mod_timer(&rdp_gp->nocb_timer, jiffies + 2); - WRITE_ONCE(rdp_gp->nocb_defer_wakeup, waketype); - } else { - if (rdp_gp->nocb_defer_wakeup < RCU_NOCB_WAKE) - mod_timer(&rdp_gp->nocb_timer, jiffies + 1); - if (rdp_gp->nocb_defer_wakeup < waketype) - WRITE_ONCE(rdp_gp->nocb_defer_wakeup, waketype); + switch (waketype) { + case RCU_NOCB_WAKE_LAZY: + mod_jif = jiffies_till_flush; + break; + + case RCU_NOCB_WAKE_BYPASS: + mod_jif = 2; + break; + + case RCU_NOCB_WAKE: + case RCU_NOCB_WAKE_FORCE: + // If the type of deferred wake is "stronger" + // than it was before, make it wake up the soonest. + if (rdp_gp->nocb_defer_wakeup < RCU_NOCB_WAKE) + mod_jif = 1; + break; } + if (mod_jif) + mod_timer(&rdp_gp->nocb_timer, jiffies + mod_jif); + + // If new type of wake up is strong than before, promote. + if (rdp_gp->nocb_defer_wakeup < waketype) + WRITE_ONCE(rdp_gp->nocb_defer_wakeup, waketype); + raw_spin_unlock_irqrestore(&rdp_gp->nocb_gp_lock, flags); trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, reason); From patchwork Wed Jun 22 22:51:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 12891558 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B7217CCA479 for ; Wed, 22 Jun 2022 22:51:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1359870AbiFVWvy (ORCPT ); Wed, 22 Jun 2022 18:51:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45656 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1359812AbiFVWvt (ORCPT ); Wed, 22 Jun 2022 18:51:49 -0400 Received: from mail-qv1-xf35.google.com (mail-qv1-xf35.google.com [IPv6:2607:f8b0:4864:20::f35]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2FC1940A3E for ; Wed, 22 Jun 2022 15:51:49 -0700 (PDT) Received: by mail-qv1-xf35.google.com with SMTP id c1so27038065qvi.11 for ; Wed, 22 Jun 2022 15:51:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=FIRVjLUngtzvaOGGJhPX9zs36OvO+dWcUJ0YzPStXJo=; b=RP4Xw5PjLXl1/ttonHyWA6UJqEVlPBaI/VbWuKT2GdpriQ5RbqwHgHOBGOdfOTxTyq TDGa++K8gS/R+55d58vj0W+rk6ukDyJmrWWx/adgrlWS5p6iXgZaCXklmrAQ3/o06bKv sjXJp95JkuuCiMNbBqArpXD5cBjAQs44IfTfg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=FIRVjLUngtzvaOGGJhPX9zs36OvO+dWcUJ0YzPStXJo=; b=A35dENIHBhvC6cxevBhU3WdSDPc2HbSNOIxIVipzwycP44zu0NQLBO799LQnw6fqG2 lGMIGf3RfmAU9vlYGKeyr4mUYFWSNS8VGEP/sF5p+tpqM3gL8jIB/lIe1cdsAY2WCWVG e868pOwRiljzD83YoF4BWwV3XetqDfWT3fphfwH/ZCQWIHyY9ARvDH63YfjpsEDXFcvA /Q0PiSRJXpQlBTuKpcPXu4ZHELptIajG6FKMFBRhh4IaOWpXBoZOxXczhCkS3/wt9QgK uH6bLwLc2933xCsl5m5I3pflgffYeNoqh0owXKirDDiJI3j/p1iUrVYQ6WtskBsyvzRA EBiw== X-Gm-Message-State: AJIora/G2ZOoAdxAsojWrrwwSwUIubBfnNda/YLrYXnrka6dxoE32nN8 mGD5P3uLJxtj7ctwSSC2U/NFMjDjNrlnoQ== X-Google-Smtp-Source: AGRyM1vaA5Ao94/E0rX5gloX6owI1QnEQU1eoreWEcsbtCYin0ct2kXn34bL0xAor+gWulrg/0CnCA== X-Received: by 2002:a05:6214:76b:b0:46e:2704:7c42 with SMTP id f11-20020a056214076b00b0046e27047c42mr30212669qvz.90.1655938308765; Wed, 22 Jun 2022 15:51:48 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id k66-20020a37a145000000b006ab91fd03fasm1794022qke.19.2022.06.22.15.51.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Jun 2022 15:51:48 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, "Joel Fernandes (Google)" Subject: [PATCH v2 8/8] rcu/kfree: Fix kfree_rcu_shrink_count() return value Date: Wed, 22 Jun 2022 22:51:02 +0000 Message-Id: <20220622225102.2112026-10-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.0.rc0.104.g0611611a94-goog In-Reply-To: <20220622225102.2112026-1-joel@joelfernandes.org> References: <20220622225102.2112026-1-joel@joelfernandes.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: rcu@vger.kernel.org As per the comments in include/linux/shrinker.h, .count_objects callback should return the number of freeable items, but if there are no objects to free, SHRINK_EMPTY should be returned. The only time 0 is returned should be when we are unable to determine the number of objects, or the cache should be skipped for another reason. Signed-off-by: Joel Fernandes (Google) Reviewed-by: Uladzislau Rezki (Sony) --- kernel/rcu/tree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 711679d10cbb..935788e8d2d7 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -3722,7 +3722,7 @@ kfree_rcu_shrink_count(struct shrinker *shrink, struct shrink_control *sc) atomic_set(&krcp->backoff_page_cache_fill, 1); } - return count; + return count == 0 ? SHRINK_EMPTY : count; } static unsigned long