From patchwork Tue Mar 28 22:16:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13191614 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A4CECC761A6 for ; Tue, 28 Mar 2023 22:16:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 330066B0072; Tue, 28 Mar 2023 18:16:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2E0606B0074; Tue, 28 Mar 2023 18:16:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1A9096B0075; Tue, 28 Mar 2023 18:16:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 005656B0072 for ; Tue, 28 Mar 2023 18:16:55 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id AA00EC0B15 for ; Tue, 28 Mar 2023 22:16:55 +0000 (UTC) X-FDA: 80619717990.06.ABAF227 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) by imf11.hostedemail.com (Postfix) with ESMTP id E28454000B for ; Tue, 28 Mar 2023 22:16:53 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="hC/hCzGf"; spf=pass (imf11.hostedemail.com: domain of 3VGcjZAoKCNcRHLKR3AF769HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--yosryahmed.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3VGcjZAoKCNcRHLKR3AF769HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680041814; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ECd9n7cK9XN/T+IJ39MyPXAm65DzgHuyZCFnDGQgZIE=; b=2mK07MVPQDHUzFrihBzWZ9bQWDlbz6pqiD2FsE3r3uiDOWUHe1rRa1i/Yao15ucqk1m17f 2LBXSoL5OdMevnhIvFZ0zaiD4JxDBrcW0gHGImzrXsz+XvQxNWhhbmp4JwzrZgxzi3ohbu IL5zXSTlALa0qhX17pJI6/YFXH0tCtA= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="hC/hCzGf"; spf=pass (imf11.hostedemail.com: domain of 3VGcjZAoKCNcRHLKR3AF769HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--yosryahmed.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3VGcjZAoKCNcRHLKR3AF769HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680041814; a=rsa-sha256; cv=none; b=Yu7Xf8b/GHrBMP6uQmtqKiYf0vMfIYtu+SwQe13yI57yIBJy/kFo3OmSTSMnBbCFpQoDz7 gGW3a/jkTTZjuR8qC3znSNDYU/jwkOOyeFO0+VLh49k+QAXmJEQz5/V4HiKblhn9HUohJV BIPmJok9FlwwWX3k80IL1BQgsHYjVos= Received: by mail-pl1-f202.google.com with SMTP id k3-20020a170902ce0300b0019ca6e66303so8351509plg.18 for ; Tue, 28 Mar 2023 15:16:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1680041813; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ECd9n7cK9XN/T+IJ39MyPXAm65DzgHuyZCFnDGQgZIE=; b=hC/hCzGf8kQKLtuU/Lq4g/PH8Bikoaov/kMCU55X6hS3y0cgLvVqN0Ui9SeHKqoI2Y MLR9yv7Q6ZpTQchtNNsVDSGumEgnJDRGOrdjWF1vQ04Hu4VkyeJ+agrrQETPfZvxGuwE fYcWpRh33+ZEKqsq/V4FxFGt/SOC7Xson0BSVI7SL+S7Djz8oPuCrC/TQcGQu540vmXS GyCet5dMYq6C9cPRyqnOFXjOod08thOhwnsgDFjm//XeVxGxDJp0Yqlgz0MeTNLqnkrX POTGQvY7O/HgL6iaisQLmX0ut7TKBC/YmOGLurxCyE9UYzyCbGnroCIDWbRRsYz1/a88 d5vA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680041813; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ECd9n7cK9XN/T+IJ39MyPXAm65DzgHuyZCFnDGQgZIE=; b=Xl01KPNtcUpxTYwdDe4sEkXhJfKe5/hqLVdbSOMq2Az5EXyyans8WyjYFSLXGSX0C2 KD/wqBGw+JzHjmBTfAlGDLq9FqHMWweiOVBiMFtzD2z0Si/JYDBUtuYWJvfV35WXDc2c 9mZgoPMnCjisUITZOH/hxZrhAxPFQMlM7158Pv4bfEboZNI02xaN7akzmtar6ROriUrN ELmJSo5jRC0/M9adtpzd9twvCI9uq4r6A1l9FggBvpYLQYykSh2xAFAmDwq5WYouNRBL LNvAXnLu6TteyZafqE2+31KFDJpFPe4kuho/xe5esskGxDdpcQw9pvdCt4yKJ/UQmYn4 Hyzw== X-Gm-Message-State: AAQBX9elvz02lxT7HyuZalnwQ23wnE/iJ9cMwkWnPsA+VSgPGE780TFv FO9kg4xfB+OZZdr0jdAL+drJO5tFo16YcnRj X-Google-Smtp-Source: AKy350bt7vkQiq6OhtHhv/j8fBGrKxEfoXOdpNRwyGSGJDzpsapd3WOuPFqtBqzF+StnrMen5GR5vLj6Xa5jtVdE X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a17:90a:6943:b0:236:a1f9:9a9d with SMTP id j3-20020a17090a694300b00236a1f99a9dmr64331pjm.2.1680041812843; Tue, 28 Mar 2023 15:16:52 -0700 (PDT) Date: Tue, 28 Mar 2023 22:16:36 +0000 In-Reply-To: <20230328221644.803272-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230328221644.803272-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230328221644.803272-2-yosryahmed@google.com> Subject: [PATCH v2 1/9] cgroup: rename cgroup_rstat_flush_"irqsafe" to "atomic" From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , " =?utf-8?q?Michal_Koutn=C3=BD?= " Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: E28454000B X-Stat-Signature: w9hcxwjbbrtzemx7g7nsxr73wtqwj16m X-HE-Tag: 1680041813-836379 X-HE-Meta: U2FsdGVkX18lzMyMpwTSQ4cvu/wizlmhh+skRD6LOmx6Vbt3a3o2iJcPMfhvb9xI1vMaZCcfINIrsF04x+7lqQd/iqHkiOHWhzabIirRHhBCcXmKJPZaWX5EqJt6QafrJoUfxPg6w2EIC5vW9FAemCT907+wJURx9vbwMNOfmu4qLQQbc6W0/jUAZDdfdq638jpRNukmh4ZyJO7szu7Q8hm7pF01dSuFyrGqMBwx4u79IMu9NELFaFdO1QnzDrMylJJpYneh1QWvmH8ESbbaD436WNpiR+IHLnsic98Fulp0y5z80QCuAVcUu9SwJeXZoV6L5TV5liuTebwG++d0DuNqqcIuFHpkch9yreav4iRibKV6RO6nUPpIxHWDWhL0iXoXm3DGrBUBEuKWVh8dITXqya3df+6ut9ChLKsNacZgs733xnlqEuwxH5AhfOKJOr/Qu/vqjwVf98c6ErecOS5v3pF4LjELx4OqInb3Wq6sl05V8P8UFRC8pXQcY9OSEnmv6IpVaic8uckawuOyqJ2d4j48UWXx3enNxNxYGPmH6fmcNzDAXpX3zMNwoeA1qtqyI4ex78TVeN+0u7ZY2tuJrOboMYEvHVHxVdxDcJwhsXUO9O1t0bJlVtUILukzqt98cWQgmNPzu5cj8mgF74xwisz0A4hXFglr43sXuZl0ZoHpHUF3xOt7ast8/aVot33WMfl2dqQb53O2r9vBLNRVP99eOBoyQgqzTjJmuyRcafVtkKpVbd7tLVDUWbXX0YgIEKGt0GT7tXHxixM4Kcr354RHSN2uvQ3cOxyepj7+tnScM7Bk3sGMZxLnAxRAw4qQrwwg838PeikLQa5giI6TOViOWptbAB1mhzYcl9Qk/qNpsTS2kAs/ZQ7IgSFxKEntx7cB0TnSHZ5I24zmCzI/Hj77y/PDQ3WYSHqZhCH2smADb2nwoYp0jte5PNHlKvUjbQwHu88p8ynkAW3 vyUM95+c Q51aR1HhWxjPtjnhNUadM+sNyU8uWRwfrOZ7q1GDoXkhutMzA19HsRsflrLMf38/2CaN7rgpsD9bLzqqqBrbnE7DxspYabzrxNN2+iWbF4jY2JkygglFVpAVVDGQqBe+2PR7zdWoT3zkMNbTnez2/Yo4LrmX2KZc+AYtcoQZU163yPPCs99P/J8R6Omy2710Z/JuwskWdSf6lFn9nNKRPewqL2zjS4V/E0AC5PyXRgLqHPfpEtLQBFZMJ9ca6zQXzrOqVmqZKKBkcZD2bK1jl231VENC2G152hiCQbruTQPwPKb1LDhBES1UCG3Nnac2gSOSQYgUsO4nkY4tAiS49XCvv0cWQ13E86t+IAd5zgB3siXa3RpTOmljjaVCF+YM8Mr7tC9+o01XdHMXe5lvGKlUnrh3AUFsRARvix57/ugi3ercv2A+uNC0jY5+C7IZIToN2JUNpMIa8tc8r01W+DCvLk4fq2iYZoRyvfiQBOTJvKUN3S0MsZo7BNDvMYztwgTL0m9S/hcqRHAhA7aauUhFPjCuTqiZ9JL1c8oqrlk53uLpnigRt2W1594aTeGxcLGM41+o/uV+3JB2BRI8BPzO/Wdwc3Ap5Mw1rBdT476Tppc26yW8vRLUjmR61U1+r7N1R1/zAlJgfbwEPSJqhf+prSillrS2JgsJ6eIat4Nln8R7lYHZm8+dqjiWfMd5ze4ZRhB9LWEk8D+MQHs1DnIp9wh+wCotOcPol42zV8gY7ES0ZyamMfve8xXNgnAEltT9MoPPYrVhuCjQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: cgroup_rstat_flush_irqsafe() can be a confusing name. It may read as "irqs are disabled throughout", which is what the current implementation does (currently under discussion [1]), but is not the intention. The intention is that this function is safe to call from atomic contexts. Name it as such. Suggested-by: Johannes Weiner Signed-off-by: Yosry Ahmed Acked-by: Shakeel Butt Acked-by: Johannes Weiner --- include/linux/cgroup.h | 2 +- kernel/cgroup/rstat.c | 4 ++-- mm/memcontrol.c | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/include/linux/cgroup.h b/include/linux/cgroup.h index 3410aecffdb4..885f5395fcd0 100644 --- a/include/linux/cgroup.h +++ b/include/linux/cgroup.h @@ -692,7 +692,7 @@ static inline void cgroup_path_from_kernfs_id(u64 id, char *buf, size_t buflen) */ void cgroup_rstat_updated(struct cgroup *cgrp, int cpu); void cgroup_rstat_flush(struct cgroup *cgrp); -void cgroup_rstat_flush_irqsafe(struct cgroup *cgrp); +void cgroup_rstat_flush_atomic(struct cgroup *cgrp); void cgroup_rstat_flush_hold(struct cgroup *cgrp); void cgroup_rstat_flush_release(void); diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c index 831f1f472bb8..d3252b0416b6 100644 --- a/kernel/cgroup/rstat.c +++ b/kernel/cgroup/rstat.c @@ -241,12 +241,12 @@ __bpf_kfunc void cgroup_rstat_flush(struct cgroup *cgrp) } /** - * cgroup_rstat_flush_irqsafe - irqsafe version of cgroup_rstat_flush() + * cgroup_rstat_flush_atomic- atomic version of cgroup_rstat_flush() * @cgrp: target cgroup * * This function can be called from any context. */ -void cgroup_rstat_flush_irqsafe(struct cgroup *cgrp) +void cgroup_rstat_flush_atomic(struct cgroup *cgrp) { unsigned long flags; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 5abffe6f8389..0205e58ea430 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -642,7 +642,7 @@ static void __mem_cgroup_flush_stats(void) return; flush_next_time = jiffies_64 + 2*FLUSH_TIME; - cgroup_rstat_flush_irqsafe(root_mem_cgroup->css.cgroup); + cgroup_rstat_flush_atomic(root_mem_cgroup->css.cgroup); atomic_set(&stats_flush_threshold, 0); spin_unlock_irqrestore(&stats_flush_lock, flag); } From patchwork Tue Mar 28 22:16:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13191615 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0B776C77B60 for ; Tue, 28 Mar 2023 22:16:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 94B7E6B0074; Tue, 28 Mar 2023 18:16:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 85F016B0075; Tue, 28 Mar 2023 18:16:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6D8A56B0078; Tue, 28 Mar 2023 18:16:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 5DF376B0074 for ; Tue, 28 Mar 2023 18:16:57 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 3985E1A0431 for ; Tue, 28 Mar 2023 22:16:57 +0000 (UTC) X-FDA: 80619718074.27.709234E Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf19.hostedemail.com (Postfix) with ESMTP id 7BED71A0019 for ; Tue, 28 Mar 2023 22:16:55 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=aB5XInM5; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf19.hostedemail.com: domain of 3VmcjZAoKCNkTJNMT5CH98BJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--yosryahmed.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3VmcjZAoKCNkTJNMT5CH98BJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--yosryahmed.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680041815; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kpfCPbcrVlI467nPrWnClcfZlWEh8/oKLk4mWLWwJ5M=; b=vI/r7hl1v/6xo8++ylnOiElA8dxbwEcblEeMwIsvtPyXCYqtKLNzL9xdAnyGltIiJN3jP3 BJwV/pgRmCteRn77+xipgZeT8lRLKjvdIwv9HHleGmKkmiaJQmm9y/6hlt3vIm4H3j2St+ CjuqSXb9UqazUs1iyNaVH3ytjT4CLP4= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=aB5XInM5; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf19.hostedemail.com: domain of 3VmcjZAoKCNkTJNMT5CH98BJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--yosryahmed.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3VmcjZAoKCNkTJNMT5CH98BJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--yosryahmed.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680041815; a=rsa-sha256; cv=none; b=ueFMHOL4jz/fCXzx/h8D3vvACVYuKZyCx+ihbVBORpqK60hGZotPOMdN7WTCffIIBLvgLf JkwfiZvIJGnrQwVy5950VoWbE1O7+Fvt7ux45zUjzd5Om7n8lMUhzXaquo31JtGdVRgqO2 5spp6JtR0AXIR26/AmHPzSJ5zbp1XqQ= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-5461fe3163eso3297007b3.17 for ; Tue, 28 Mar 2023 15:16:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1680041814; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=kpfCPbcrVlI467nPrWnClcfZlWEh8/oKLk4mWLWwJ5M=; b=aB5XInM5shRFEupjVRfOVTwIp8OT/8GOCdpf0mMVeXPooztYFeqB/4fTyYFky42c2L cHJL0WAYSCnTVIPjouGLoaqK+AnedliRs9QJra7EnwTz3y2TYL063GYspGDlo69C/aKX Q4WCP9t0FZSy4z5mYrJKLFahZFndlIErOfAb2mjxfSIGPKnJVKuUODD216rqjI4tCXyS D6OSq62KYNM8d3Y/gjIsH2sn135NHZXDjDoSFmzJ5ShWqmWT9/5W9qO3Eof0i2wwkL4l 11B1AMdH7KMsOMtqPWcbVxPahm2yYwvYDsj0ARVEud+qGQ3Lehzgwg9PonWNXUtgVgTf eH8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680041814; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=kpfCPbcrVlI467nPrWnClcfZlWEh8/oKLk4mWLWwJ5M=; b=xnrmXCtnQbeRLecFhb8Ya7ju1vbf9cIv0et6LAXRQlS3377lSQ/Bmq7NiaQe6ZoXYu quahEn/Fhnd8ZcPTOfdhWg4hgG1ALvlk+xuDu1NM2Z5x5Rq0PrzjvtVaPBCJ0WCbJNo5 TKVwHLw/BzKR4MJITvYYHwNXFTKEe/6DQ8yN10TlEQ9lKZON9W8JALmjJDZYDzvJ7YFF f1nXN/NMnbltcRGpXv4kebns2Lh6noHKKbrXqLxMZW033JmGzmzI18cJK1z87k2zg5gq fUoIQfoiXZqe0YZWhp3OVCmqMIRI9MjVcEAprouncUdv07YrGSTkmTMrzX69GgXQMXpW H/+w== X-Gm-Message-State: AAQBX9cjH6ntpLEgMTdP4JujUY1AjJA1RneKiCfkXZfYOyWbYjZUQ0Cf 3j5SbcmS2uXdQJSWjGGIv1GHPgd1uugdZ8Nm X-Google-Smtp-Source: AKy350a5zKI96b1WoEMsf2Uk1BbmTXMpgnT5uYIBtloHj2QDwXA1RiFaSK6WQDs33/Qs1TEwSL1tU23SVOrmYfBj X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a05:6902:1586:b0:b33:531b:3dd4 with SMTP id k6-20020a056902158600b00b33531b3dd4mr8636279ybu.1.1680041814635; Tue, 28 Mar 2023 15:16:54 -0700 (PDT) Date: Tue, 28 Mar 2023 22:16:37 +0000 In-Reply-To: <20230328221644.803272-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230328221644.803272-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230328221644.803272-3-yosryahmed@google.com> Subject: [PATCH v2 2/9] memcg: rename mem_cgroup_flush_stats_"delayed" to "ratelimited" From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , " =?utf-8?q?Michal_Koutn=C3=BD?= " Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 7BED71A0019 X-Stat-Signature: oerwh4pw9xbbgwiz97porgi3mqxju75e X-HE-Tag: 1680041815-171651 X-HE-Meta: U2FsdGVkX18i0nZHgMZT7HaMScjQVOxOu1i+qDC3oCaOWJLT3yW5vN/l7k+Hq//Q8pdntEpna31L8so5M/5YsWQHDFeW8s9O33hDf3PGdKrV7PmXNrbR2gNwl74tV7TgHaEk1HUB3I3hTN9xax6lUwZTH+5WrfqFuK1VsV5vY4mPwyGlsKHjlH0yKTtRcfuy4eZ7keicvw+VwRJQueIy6UGwN7q3RA2ROL+h2IWAXM7mUEPfUTv03hTLfm+W3pXVQZlYnyURbXUEAr1RF4KFGELjiBgGlyiQ9ygMaP7to1vijsV7uBoS25rQiIIOTx+8WDqSa3hT08/JTSf/6U3AINfPnL4w4dLoo0PX2gni4s4XfNGPnyt4FTLBpNeLEePnVzVsfStP9jqPmh/DmczPjmRuun7HGRoCm/IRv6NR1kjxdWhPwa3LQczEQ3Obas5Fg/O+fx3v526I1N/wlHgH/BM/gf/p+DAr/fI0PDiYVhwanOnjcziPWxlhm96rWMR2QFU3+WrnsmsfHdYMobbTTm4SaXEcSxuomwf7MEnkp07HVGVr75z7x4PbMbBkOk9QvXz2a6ZYpIkRF38+y+/XgS/J0MIsabozmDrVHXL5iRGViBcrYIVeT30acBMkciLrd9DF8PXp7QjEWRUvVAvjortKHEmevSjpK+fo1OEY9DoRNT9v9LQnCxnUKCIAiNp7IV/QWTr48LLmo9kKRIrc3xt7P1r1qihCWUqaUbaeZMpsAIxovV++vml7yLmHsmYizcGurZt9TaRNcoe7UmL+ohvLBd0EyT5MGuVJTf2G07VQrOwfFk1NOS57QK6Z5XkBTrx7z9K44/EO8r6bj5w9YsNPp+81rH39/zeFAmRA527QA8od4riaDB5UUAWzVt+IdynnQvlpbv5AFu2g1spaZQNKnN1LvINoKtp/AXBVFUSnwCIX8qMzDaJkDnozPS2DmkXZup+iiF16V5Cc1x4 tZJq0FZ0 hkzIPGAuu7J7A/mpaUMTcbTBC0Sya8W/0rUVWZzejkyAxvM9UybieGKA9DWEqeOF03iNR8VbvvQSHfXvWEXSIcClYjDNxYWMBT00SO6zYTd0qqHDvdEpmApa6xlgxlRrF3vFXS/ZLD/ay3uPv0tjhYYQq3ZHkPa4QPVyRtHCte3KEzn1HBV3KCu0bQp/bRyDUhx9l9R3npTEM2Rnkwm2ypZu1jnDj5+vwZLF8OolFeFUE62IHFJzUI6VtW3sMqpvg+k8MBUTcMjfc6sjQvcvwY/K/ZDoF3/K9fcTLJ75UkH9UB2DuksL/CYJiM/L+8N+zKjOGxWq9RTWShLNejvwU/JH0EvfiCQCPjODjgrkd5mW6dEl1Pm9Jhwc8goTxxrmBeYluqn1WGkRcZzM7izMPxnTeHVB7S/UCrkcz+dJAF/LMJnRv/yHeYqFy9vvd2L11g1tI+/4sdznWNd2/pMU0qDOI1oTxLW8XjbVBnn9OvRu8d7lP31CUfbbbVxrbh530e/UgBxLV07mJ6atjXYq0h69P7I5YaOz890GbixzW+S+W780KRuRwlm9VbVA+FN8K+Dh0QDU4FO6ezkSWauwjOto6YYWkbMql42mk6sO2sCBaGDQ8NyqPLPDnrHT8ohXEqFf1Y7PVRq0BqNAzPtig67pjd/4t8p6XqjM4AHc7s0q3xF6kkHOsoNM8X4OjUk9gpHRSdOaUc3eleWnZFNgc5zOjInew4cbpj0MVjr3kPWvTycpKaRVpvZ3PBNNfzHctuKa7zhLzBFVw55A= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: mem_cgroup_flush_stats_delayed() suggests his is using a delayed_work, but this is actually sometimes flushing directly from the callsite. What it's doing is ratelimited calls. A better name would be mem_cgroup_flush_stats_ratelimited(). Suggested-by: Johannes Weiner Signed-off-by: Yosry Ahmed Acked-by: Shakeel Butt Acked-by: Johannes Weiner Acked-by: Michal Hocko --- include/linux/memcontrol.h | 4 ++-- mm/memcontrol.c | 2 +- mm/workingset.c | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index b6eda2ab205d..ac3f3b3a45e2 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1037,7 +1037,7 @@ static inline unsigned long lruvec_page_state_local(struct lruvec *lruvec, } void mem_cgroup_flush_stats(void); -void mem_cgroup_flush_stats_delayed(void); +void mem_cgroup_flush_stats_ratelimited(void); void __mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, int val); @@ -1535,7 +1535,7 @@ static inline void mem_cgroup_flush_stats(void) { } -static inline void mem_cgroup_flush_stats_delayed(void) +static inline void mem_cgroup_flush_stats_ratelimited(void) { } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 0205e58ea430..c3b6aae78901 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -653,7 +653,7 @@ void mem_cgroup_flush_stats(void) __mem_cgroup_flush_stats(); } -void mem_cgroup_flush_stats_delayed(void) +void mem_cgroup_flush_stats_ratelimited(void) { if (time_after64(jiffies_64, flush_next_time)) mem_cgroup_flush_stats(); diff --git a/mm/workingset.c b/mm/workingset.c index 00c6f4d9d9be..af862c6738c3 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -462,7 +462,7 @@ void workingset_refault(struct folio *folio, void *shadow) mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + file, nr); - mem_cgroup_flush_stats_delayed(); + mem_cgroup_flush_stats_ratelimited(); /* * Compare the distance to the existing workingset size. We * don't activate pages that couldn't stay resident even if From patchwork Tue Mar 28 22:16:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13191616 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C24CC761A6 for ; Tue, 28 Mar 2023 22:17:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 974B76B007D; Tue, 28 Mar 2023 18:16:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 921A16B0078; Tue, 28 Mar 2023 18:16:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7C6316B007D; Tue, 28 Mar 2023 18:16:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 5E24F6B0075 for ; Tue, 28 Mar 2023 18:16:59 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 285A8A046B for ; Tue, 28 Mar 2023 22:16:59 +0000 (UTC) X-FDA: 80619718158.13.313A8C4 Received: from mail-pf1-f201.google.com (mail-pf1-f201.google.com [209.85.210.201]) by imf12.hostedemail.com (Postfix) with ESMTP id 58C5D4000E for ; Tue, 28 Mar 2023 22:16:57 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=TcS0DaK7; spf=pass (imf12.hostedemail.com: domain of 3WGcjZAoKCNsVLPOV7EJBADLLDIB.9LJIFKRU-JJHS79H.LOD@flex--yosryahmed.bounces.google.com designates 209.85.210.201 as permitted sender) smtp.mailfrom=3WGcjZAoKCNsVLPOV7EJBADLLDIB.9LJIFKRU-JJHS79H.LOD@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680041817; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/WDDqk1nTT7D9mKcEzqzCjbueo/E6L9K7q/9wjNO8LY=; b=7GIxKzjYugjjIQ6bdR2nJtitQ+d2PWYjBLFDFufNETOGlkeNdJwe/VlR5rktMFLj0nUVTA yMSTJG0u1FiQ8wbKTiw9MQe9pHyD7BHESO9eQUGN2UB195kL4E18qdzR+wgTN09bbIlKjX u27GfzeL8vxou97nrQSvPpv8LghRvI0= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=TcS0DaK7; spf=pass (imf12.hostedemail.com: domain of 3WGcjZAoKCNsVLPOV7EJBADLLDIB.9LJIFKRU-JJHS79H.LOD@flex--yosryahmed.bounces.google.com designates 209.85.210.201 as permitted sender) smtp.mailfrom=3WGcjZAoKCNsVLPOV7EJBADLLDIB.9LJIFKRU-JJHS79H.LOD@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680041817; a=rsa-sha256; cv=none; b=QnqXlNhbdnPwjm04/eUQ+l3yxyXEb/b66uSvgcva085KA39ymf1y3XU4kQ6bHLIbIU77hP OQ48qegzT8BtclX58HqOFzZyW4Z5hra1m49m3PgdFimzJLoY/7h7FMcrNKfxbTmCsCDb2D vbEaqVkRTLFYkQmp4Ttqkrg/KBSb77I= Received: by mail-pf1-f201.google.com with SMTP id y186-20020a62cec3000000b00627df3d6ec4so6400885pfg.12 for ; Tue, 28 Mar 2023 15:16:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1680041816; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=/WDDqk1nTT7D9mKcEzqzCjbueo/E6L9K7q/9wjNO8LY=; b=TcS0DaK7zSX/dgbsp92dhBvU1YXIfJw7DU8Bxymw385Iypv0i9ByV9836BmH3XGIri GA3G9Ah6/ynKVVXAJLor7BukC88mhb5FcK9F0N2UppuVbKofnYFrmwIOVN2419Sgn/T3 Leo2J62N2CQ7p4vQ5tLG3l0ISmiaqnyiPoXjgBX2M4RYA1wXrmenFymFmJXjeUJkODde IPUO/XjdrDDGWWgE3vWyhfVxnLq+1UNPnYiIQva9A9S4iya4Iw/wc2ln/EZk6xTcIXjg k22ctJfIxPj6HJVEsaVfFSF1/WN3feXNcAQlYcNsdD7yhK296UbpIQqZDQ1X1PAq/IZq ToQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680041816; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/WDDqk1nTT7D9mKcEzqzCjbueo/E6L9K7q/9wjNO8LY=; b=fadlhk7lyDPwhJGMmgTLuyiDUKhvB3HgW2W2jwxh7n/fglP67JAte+QaLZIzz3yY9A 3uUe9cwbT1r3htvqBnRb1AE6oSR8k2ikwIzXpKko9RsKGr+HOTeEmAC+3ExBWZto2JMK AMDeKKXJpskapYNkBzJts9/27eBut/+eyKC8mLfqsiaAlq+B3HK67Ptm1Qf6R0/kO0E9 LJ+Y6wTB/72FszWJcJXf4iO7i+PCXACjOkO2AY+9Z+sGBBGTpJ4lw8I6wzdnzUyqK2tJ KDW1BQZuCT9qNbCkjVF6ZYm9wB4Qar1lcBbtArXtFL3Cuf8HsQfS8RJeffHPSQOP8bRV 13GA== X-Gm-Message-State: AAQBX9c5Bx9ijVbDFu1ECVUUsT7c6zr9kTJBbwVO6W5FifUfUChfqM1l RMLen/txNuB93Z+4o1RPz1oO1JQzl95g4/NQ X-Google-Smtp-Source: AKy350axjsRVVcKhkqlJ581QnQNXv5GyXPvAYEbbNcmbDp26pUkQK7hJHNgyYA5XT7+ktiHVuLWG9T6jy5rcV85F X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a63:455:0:b0:50b:e523:3cd2 with SMTP id 82-20020a630455000000b0050be5233cd2mr4608055pge.11.1680041816312; Tue, 28 Mar 2023 15:16:56 -0700 (PDT) Date: Tue, 28 Mar 2023 22:16:38 +0000 In-Reply-To: <20230328221644.803272-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230328221644.803272-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230328221644.803272-4-yosryahmed@google.com> Subject: [PATCH v2 3/9] memcg: do not flush stats in irq context From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , " =?utf-8?q?Michal_Koutn=C3=BD?= " Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 58C5D4000E X-Stat-Signature: ezmzwk7i9nu6e79e7a7n9jppxsy7t743 X-HE-Tag: 1680041817-233747 X-HE-Meta: U2FsdGVkX19k9zNjGg4IscwixXmFdb5oIAJfJ7iSNt/8Br/1yMMzp90wZpTEkkaYHB9lU/foP7S6NYcT/4z5U9sM0vfzePAsotLuZlShVCDLOcZkl+ttGPBScaZKxYtRH6WM7CEWv0pKQBWySt+kN2cUTKehCzTP7SgdoWGdFltycZtd+eEFjTK1H27awmXTznuvfUs9s67ZEbBSUzT82TeXsp6iIldsY8EhCxwzq7lk82SsxpXA4mOO5rBBSYTYr936bxVEWP/w76+9IhlUz5l/IY2slGTvqXXBap3YpA4Imtg1Po3SCyXXPtPNDHcr5/q6ZXmNdwa0bbBqWspGlkUOhhP6eEdTQEnVsDSHOtymJk22i0emcysvfGpsDwyQphdySBjNpHnKfRFyfHkcI7PphBe58wDLcpepLvrsVQ94iQRbsqFfT78t/ZQWyalYFnia+J2U5lNuigCspZr46+70XvStTIhxXKdWyptZML1Pzet4HGemDZ+jHEhHBwWSY0ge87TbgYZKNAICySzzQhJfvgOT697ByGRRefpfCXCKauc4YukZ/7AQWaBnJyw4Oeen/DYWV5cvFoAqe66G+vyjgUnNCdYmLd085QD+T0bmUkefn6ooOGgHZAFfuzTsWIYZvNh6A4MYanUrdwRudnYNUg1o9dbpWfux8quW2quRL5OGqdjngek9gnTHZ4UqjR7nhRtEE/e3uNwgVyJEX9HQYkDPckB2rDaJeFWRtboX3xMfbnecBH2peUDoZbaKrWGdZTP+EJiN96Y0BZpgKVHsDMlfQwz5DWqkrkHsb+tTm9s5a+eK+SthvFcrYn62le5GCN1VsSXYD0o4hK74MnKxYqm/CsXn1ByTWUAKrF25SxNnK0b33nGMKjboGISitlcLQV9v54v11yM/IUSE9D/juxKD/d8ApUYo8QrqmGSZdVwuRFRid/f3FXAyV+ZRPn+Vye+6wEodsysK2xJ bJCRSz6+ SIOyuT8xpUMePxvUwPX8hNcqZ7EbfMWNgrlSjgGtM05xiHaeMY4B/nnFhYNPpzejGNjLW6VgflydME6E4SsSnx1yOHiiZWAIa2WpRT6geq7oZjmOLyTQ5M8dqQHBB50xLj8T4G8S6tst2PHmy/+Wdf66jMEbWaqr2sWNpdUKbbGRR18vJcpd3thmE+jhsWz1sQokv+GckKaXX+gR83i1Hw0yhCBS5mDpo1qkQlxg4D6ypLijcd5QiyybvL5iUAPPbZl8iG+mFHUUDLiZLrng6zWt1aYCMCF0cIPl6yIR5lnUSTa/LCio3V5qewClg3cmxJ8GAduDxjIHGkoN5NWXhe9qn67AX/4DW7aPLcjuPQHvDBVcSIvRZNFAffWWQF8ulVZYahKJ2k/LTVkxTPO4glRv5gPvjJhHjOxyLoGGLc02XYhaVSfs9v9ZTb1rUO4Q+MUf3af7715viWvnXWNmL0XSeVOBZvivaTeGSZy6RKx/0MRftg9V3biUM3QtNswRv8z0bE9mFypLbZMmntTrwxeB81/KkVPRVOLsiMUq6Xq2xbcF04Sx0LbV9aRYbnJ1Q2/Pd0LZll7ryVZTV229mlhw4WeIh31dKGDJlB/ixCdmSZnQVkp0vD9M7Q1F90edDqblqy94ePO/A6ZcG55cWjUIrbHG7/OHUYuIBMNN5U/XCD+JkE2KsNLTvliUCXMipS6P0sbDh7E7xhM8J69wT+2Db3A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently, the only context in which we can invoke an rstat flush from irq context is through mem_cgroup_usage() on the root memcg when called from memcg_check_events(). An rstat flush is an expensive operation that should not be done in irq context, so do not flush stats and use the stale stats in this case. Arguably, usage threshold events are not reliable on the root memcg anyway since its usage is ill-defined. Suggested-by: Johannes Weiner Suggested-by: Shakeel Butt Signed-off-by: Yosry Ahmed Acked-by: Shakeel Butt Acked-by: Johannes Weiner Acked-by: Michal Hocko --- mm/memcontrol.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index c3b6aae78901..ff39f78f962e 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3669,7 +3669,21 @@ static unsigned long mem_cgroup_usage(struct mem_cgroup *memcg, bool swap) unsigned long val; if (mem_cgroup_is_root(memcg)) { - mem_cgroup_flush_stats(); + /* + * We can reach here from irq context through: + * uncharge_batch() + * |--memcg_check_events() + * |--mem_cgroup_threshold() + * |--__mem_cgroup_threshold() + * |--mem_cgroup_usage + * + * rstat flushing is an expensive operation that should not be + * done from irq context; use stale stats in this case. + * Arguably, usage threshold events are not reliable on the root + * memcg anyway since its usage is ill-defined. + */ + if (in_task()) + mem_cgroup_flush_stats(); val = memcg_page_state(memcg, NR_FILE_PAGES) + memcg_page_state(memcg, NR_ANON_MAPPED); if (swap) From patchwork Tue Mar 28 22:16:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13191617 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 04317C77B60 for ; Tue, 28 Mar 2023 22:17:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1AA966B0075; Tue, 28 Mar 2023 18:17:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1599B6B0078; Tue, 28 Mar 2023 18:17:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F3D1C6B007E; Tue, 28 Mar 2023 18:17:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id E25B26B0075 for ; Tue, 28 Mar 2023 18:17:00 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id BCE5C80C20 for ; Tue, 28 Mar 2023 22:17:00 +0000 (UTC) X-FDA: 80619718200.01.D5A7400 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf24.hostedemail.com (Postfix) with ESMTP id E3C1018000F for ; Tue, 28 Mar 2023 22:16:58 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=BwpN5IEN; spf=pass (imf24.hostedemail.com: domain of 3WmcjZAoKCN0XNRQX9GLDCFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--yosryahmed.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3WmcjZAoKCN0XNRQX9GLDCFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680041818; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ZQQVGwjs4PjltBBm07Wln67ACtAz9IV2/pUsgTkN4l8=; b=5BNFdQ67qHLMzbDeDi+oFJgcHJqTbwdZP/srJ0dN1guqIX/1+dhpoqeJtSF21PRjBpz1Ni nJz90S2KBdhGdqRSrFoO7F9SE4UREgCxzlOFPy5FG4HlWwDel/FiVTS3TWDFobBBw1EXzz i3rXhHb48s6zfY5YeldEjkM9FdqBrL4= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=BwpN5IEN; spf=pass (imf24.hostedemail.com: domain of 3WmcjZAoKCN0XNRQX9GLDCFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--yosryahmed.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3WmcjZAoKCN0XNRQX9GLDCFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680041818; a=rsa-sha256; cv=none; b=YQMAXPUIBRJM1bNH+fOWoNClHonLJwSa385mwjoPXbmgCQDqkawb0+ysb0+FtMWgaFm10d E+7dTQVSsliZS+EO75IMxV81dUZSPbJMbzDm9rUooUiN6pnCtVK1p7mFC4lKxCcFq25Aan iNnWwpBo+GSemsIrfLGN7vZqul680ro= Received: by mail-yb1-f201.google.com with SMTP id w5-20020a253005000000b00aedd4305ff2so13543755ybw.13 for ; Tue, 28 Mar 2023 15:16:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1680041818; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ZQQVGwjs4PjltBBm07Wln67ACtAz9IV2/pUsgTkN4l8=; b=BwpN5IENfDJAO3k0fvlZBkjsW+/4dRHFvKU3Y61vx7dQ2RtgMRwRfqyo0DTmjCAyVc CnLkW84f17NTLqbLQGhvt1qFEPE3lJHVmINhnviNxj716C70wOSUz3DOzWHSZ0RhU5XN r07qEoT8pGKoQFAOapju4r5aUWkkVs0eQmIgIci7KEjffCx1HyDeGdNPdwdlAJbu1aOk 7WwXMQu8pO8A54UInZRnQw9kbNQZxf/x1jM4losJCUKwsJkxT/8pf/ZH8ftd03KJhEdg XtPwGtxKxPzrVZOjpFL3fFaXSbnVLYOlrmJD9NS/UpXXC6msV74XQ/lannH6QIJ1sQ32 hXEw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680041818; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ZQQVGwjs4PjltBBm07Wln67ACtAz9IV2/pUsgTkN4l8=; b=D24Sx7gVbeYG47TYHf0dd4zKMsB7kEbt2yk1p0UM3/PFQfMAcR7PjxEcBQf8UbVSC/ Oz/Tuoi6mkLpc3X8Hwi/UT3//gp/3+DL/QiarSOtIBuQuPxYC9WYEnX9pBVolSG10iJn ShBcV5LHM4h0yFGlrjoUGytAvFQzG/lJrn3rL2yqolrxl8alPxKzDQbjiFMPhqN07565 bejPo7eu8ucB7ADl2Ma7mJWubkUfmyP/WU6quP8reQW153x+mENpXX6MAvyoqSctG+jX dKxkRRxNFdNB5RNn9VPIFNQW5eLroo/2jlmmtWe8fZ9cvlyNbgMpTQez1m/j4g5q3cQP Hbcg== X-Gm-Message-State: AAQBX9dI0HCCN9RvNzFy/folW34wSg3Z70padhzy9GP/wD1EYsEDb5c9 dcGDvPQcXFzuUmgJtMK70Q1v16am6ulyCr8l X-Google-Smtp-Source: AKy350a+cGo/rQiavcF9zU9F0niI9qIhWFxPTrQKaAQYONdIyzIaIiEtY9m/Hx7sNG18TtPt6miuAFUHkt3sB0SU X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a05:6902:1003:b0:b1d:5061:98e3 with SMTP id w3-20020a056902100300b00b1d506198e3mr11533410ybt.6.1680041818092; Tue, 28 Mar 2023 15:16:58 -0700 (PDT) Date: Tue, 28 Mar 2023 22:16:39 +0000 In-Reply-To: <20230328221644.803272-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230328221644.803272-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230328221644.803272-5-yosryahmed@google.com> Subject: [PATCH v2 4/9] cgroup: rstat: add WARN_ON_ONCE() if flushing outside task context From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , " =?utf-8?q?Michal_Koutn=C3=BD?= " Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: E3C1018000F X-Stat-Signature: gjg6x1j6atjaqj86xk6y8un6g488pyp3 X-Rspam-User: X-HE-Tag: 1680041818-572597 X-HE-Meta: U2FsdGVkX1/Oc+HH8jEiXPopsCnNnEqyNnBiXrCECJmz3MH5H962NdJUi9BfcqVaXfS28Pod54Vkk/POWGQIIBfhIcu+NPy1/6BBSYBiBGhadiwccr56LbJisa7p3dkyTjqwHK8Pq0Uuv9o6TnA5jDX/klnYoVhmSWjHklZoG5mnWZrrfdHXQiy6xNLdF257FhmwxRRVLbxVqwdq39HnKROT3kDg0ts3ptiBki3l8+Yqbs8x3osFXwyDhuUyc7upvYWbljve99lC+EivQYcBRqkAqFL1486M9e5lyablqgaXsOFIUCMIc824gD9tzw/Mmdfz69vSV2+2Eyux+VtPxpEWhmAbt1oNhvDjRZygLAU2DmSDSFezDO9R401dE8TnUH0AQWyf4NxnzrRoWoQhhs9yH/xnaK2dmPydluYfXEnQajenpIZ8OLradsS8FgVH+T8m22z0+cAnhcr1gDL8rIBZK/UF7+JKFq0IPkrm2HpVBQCaj/rdlWTAqWMerc1vADhUyU9dySCCJ9V+sdVWbJsVkz6anmAI/d1XgGe7Qc37PGIhijgkZrdT2aQ+axk2yxukjEydaDVoprU2bkvF/ovzP88YvzakpnSU8HbdZvF+KLwbbMZGU3/1Yjw8tUuUcPyOsXJXBTevbstb1P2mhBHXg7UhLteIGJiqBlBpKEy0r996/XrBWRv0XIqhjGnpqsgjF9ZanuoShKG4D0rShD9G3TpzL6VurcAz8uUlcKz2wHZ5PRG3AZV/xOdxDkVfegzJ0jZDq5Ud9n6oA2O22CkO8Ry+jP6ZWPk10Lj3zP3i9vW6cjNCU1EF978wlnCa2D+V4mjaiyafEm1YkIokIZ4YE0LSq+TLS/druHon10+y85slZ+I5TcTIPg0PZ7+kmrr0M0BIBnnn2E+D52qQ3ctLpuMOPvgRAv3P3n34vs8/XPLdYzAh0FmRkKT6Fn5IG46Y+D/pLubBalVILkr PTWTgN1r AA0KRIjRbVMi2Xtn+fXlm70uCkKRgb+SLwAB5ecrnxGsY3foAMiakkVCmoBb1O3rCN4ZQVGZwNXgpDy2rojqmgGNlhiscZeZRJIGEqrPUmSuDgKGDiqwGzHz8pI/iDHGbAe2ngIMVGpG/rNs0BIhq9Pp/RDc9/CszUUqXpFdpkxaUhdNh9lmRdcA29d3khex7Kbfcz36mlX6mPKsXN7BUapl5H+d4SGArX3lsjIBEv3xoLxQF3HGVXCCEQC7/cJU+EvNhz71dJ7YeCnRqWWh8mxq8VBzjZxGay4oLj0/1sNUkoof915a8Xz6wAzwSWo5lBJpdDtseTXPCaEF/xIg5jmcdvPj+SsFZ1HSd+eyoN9tUW0ogmL95s2b/HPS+Fpn50/hryX0yUCE/pl6/x7iD0QuYmSkPqHm+GjtLZn1LwsVmVAR4h/J9M7U2ci1OkoAiCy1SEPQ06kkpTwGjKXnj2XATTh9HWMN3SOljdLz7XRyGvUwryfR7VhIUpHZ/529vqdI0hw0OFrXylnfJqZwsV1yrOhz+rgmZD1lg00JoPA5QqBix8o2l5rCAbrOpY82zuP+bWMin15RvEWGqHpQYNs3P0mY9NAv2qHpPgVbrhJcBBE5xx8X97jSWiAvV74nzDGRXaCvbmb9XqzHlLyetQsuCjt9BQYHd6DHtX3XnunC+JiVQ2e2Oxdr8cVGhOVcXPSUKUtXX0dELACk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: rstat flushing is too expensive to perform in irq context. The previous patch removed the only context that may invoke an rstat flush from irq context, add a WARN_ON_ONCE() to detect future violations, or those that we are not aware of. Ideally, we wouldn't flush with irqs disabled either, but we have one context today that does so in mem_cgroup_usage(). Forbid callers from irq context for now, and hopefully we can also forbid callers with irqs disabled in the future when we can get rid of this callsite. Signed-off-by: Yosry Ahmed Reviewed-by: Shakeel Butt --- kernel/cgroup/rstat.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c index d3252b0416b6..c2571939139f 100644 --- a/kernel/cgroup/rstat.c +++ b/kernel/cgroup/rstat.c @@ -176,6 +176,8 @@ static void cgroup_rstat_flush_locked(struct cgroup *cgrp, bool may_sleep) { int cpu; + /* rstat flushing is too expensive for irq context */ + WARN_ON_ONCE(!in_task()); lockdep_assert_held(&cgroup_rstat_lock); for_each_possible_cpu(cpu) { From patchwork Tue Mar 28 22:16:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13191618 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EE930C6FD18 for ; Tue, 28 Mar 2023 22:17:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3F6096B0078; Tue, 28 Mar 2023 18:17:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 355A06B007E; Tue, 28 Mar 2023 18:17:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1CF726B0080; Tue, 28 Mar 2023 18:17:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 0C92E6B0078 for ; Tue, 28 Mar 2023 18:17:03 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id D38F280B35 for ; Tue, 28 Mar 2023 22:17:02 +0000 (UTC) X-FDA: 80619718284.15.1BB758F Received: from mail-pf1-f202.google.com (mail-pf1-f202.google.com [209.85.210.202]) by imf12.hostedemail.com (Postfix) with ESMTP id 13EC940011 for ; Tue, 28 Mar 2023 22:17:00 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=h3KlqW+j; spf=pass (imf12.hostedemail.com: domain of 3W2cjZAoKCN4YOSRYAHMEDGOOGLE.COMLINUX-MMKVACK.ORG@flex--yosryahmed.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=3W2cjZAoKCN4YOSRYAHMEDGOOGLE.COMLINUX-MMKVACK.ORG@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680041821; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PicCaNLBAprYzcnNZKIFMVmM/pcbqkg3z9jyIWcMc+U=; b=n8jZfsAuBDRS5C380KWR27ZvdC9bzqnifKhvh6uwhBHgmopC2oZzZ6xiYYq5hR2wkwlQ3d uKnyGzPYL4z7s6ktAh/X7DVAk+TRfRBxL+4GZOelLp4X2wbvQbubZmhLWyLakVEMBU1WPP ngQBL6vz/7Emq5IMoPexZEkmWib2hQ4= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=h3KlqW+j; spf=pass (imf12.hostedemail.com: domain of 3W2cjZAoKCN4YOSRYAHMEDGOOGLE.COMLINUX-MMKVACK.ORG@flex--yosryahmed.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=3W2cjZAoKCN4YOSRYAHMEDGOOGLE.COMLINUX-MMKVACK.ORG@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680041821; a=rsa-sha256; cv=none; b=vr+sQ5Vp5LVmLRZbaUylIT9ll9ganvbH/0LBGWG/zhrx8upOT2Mb2r6xbgsMKD9t/7onGn vZe7OXDDlmnrpW4rMzAJH+x+R/xMNN1Xe2/CnPfxsSw/79X9LuhbuZampsMIFhXgOAV0dD c5RLB6uUmaj+vH8looK7sVAnxBJHKWw= Received: by mail-pf1-f202.google.com with SMTP id m12-20020a62f20c000000b0062612a76a08so6323742pfh.2 for ; Tue, 28 Mar 2023 15:17:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1680041820; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=PicCaNLBAprYzcnNZKIFMVmM/pcbqkg3z9jyIWcMc+U=; b=h3KlqW+jIEJYTK+BpLRlcd0fDHvnJwiAwYWj6YfVTBV4VsUspeq7grO2+Nyd5Hs9eL i4P1tnbfhFqPfNn0TjyAIHaXO2gZSr29zn48TaONjgFqCTDEnUQ3f8yJSse0IzidGN/G NpCoYbM12CNmhz16TMo5PtvGfqzYl/w2IuM1QOzqxIgbpDSLm4BrT8d/pKei+Ju4GCP/ Ogx40HCdcd1U7cHn91PPNKl9xn/1CmyhrjIjg+lC++Gh679/KOFvr1XoPNmeY+cl/pIr M7KlNiqZzpwmUKaPk8gBIyxokfcnZJdVNjXvvh+f/KGJuZhwnOInjtgGjgRxKbQyAU78 gp4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680041820; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=PicCaNLBAprYzcnNZKIFMVmM/pcbqkg3z9jyIWcMc+U=; b=DCvYYeECX4TlRII+uMIwxyQlVaj58N+vsninMhg/QMGfnvA971b/qT3T2KdORtsXfI EOhV724a86dJfTNY63kKn92SB0jv6HWMr0w74QuaHleE1YrGO8fgnPoxBViKQBSZp9Gi YYVus5vIVKu3IyfHAF+VoQ3sTF+35GizEsetHy6CYftaCz72N4NaHdV82yn5k7HfreIZ O/AwmN8yt2SUYGT6U3YJ8gBEMzhPk/Qg1Q9YyEnEmbu9Mby+yJ+lAlmgi/FLVqvYp9eh dvEC4zeghW9Z9IihPAjLvySTY+HKR23YwiMw7QtoAHrZXsZB0VPzf/3pQm05ZKf5HHO+ kRXw== X-Gm-Message-State: AAQBX9cxivUXUdu38v8Za4SpT217Bdaqwvyea9SeQbt6C10873DGFctD 9Dah+69nB3fNlHMMLf0z/zzSmZqJcDOuGQQ+ X-Google-Smtp-Source: AKy350aocwWe2yH7hemOfHMFd7M5GQ6mfc2FkU5LSUuK9d8xU2zKVz1ThP6UHoZndV91Ud7xpyV60PqDU9oxBKYZ X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a63:ce08:0:b0:50c:6cd:cace with SMTP id y8-20020a63ce08000000b0050c06cdcacemr4745231pgf.2.1680041819916; Tue, 28 Mar 2023 15:16:59 -0700 (PDT) Date: Tue, 28 Mar 2023 22:16:40 +0000 In-Reply-To: <20230328221644.803272-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230328221644.803272-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230328221644.803272-6-yosryahmed@google.com> Subject: [PATCH v2 5/9] memcg: replace stats_flush_lock with an atomic From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , " =?utf-8?q?Michal_Koutn=C3=BD?= " Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 13EC940011 X-Stat-Signature: chmjdirwniqefe4npg6m649ak8gqtgrf X-HE-Tag: 1680041820-689296 X-HE-Meta: U2FsdGVkX1/643WY/nvm/hqA2+54/4wt6lHg2nc7keazz02jofRT8wSykKOp/OKrRjPQA4KPkHIDu2G7PQJip14iNwk9d6oo/1j28nLFyO0buuE9P9/vVooUbAEK/aSamR8UiwQlcL9hfv5EqmGMb99bwQO2UmMax6NzaWuvp9Z3JKIWPnPXhdw4U+6Kg0K2ttk/k5uyCCgKhmKqEcl8+3H/pR5mFlpBI3ivQkkZpx6G/2aW4KgCpTRRVyZAy4qmOpQ8BlDL/PFZmrh5pjDarNMQ/HnLkZN0ep0YuzntNFT7bzMlPfQFNhLi4GSN0RzICIRpf8KYEkM/5yBwaQPuvu5LZvato0eONFevf8GEc2fczAREAJIP0ZuErIdal9fzR3V+SirlUBYrU6/McOst2fyA7HIVZ0fX1hunaa7ReztqOdYflWf+nAniQpsjq/bOrJW2rwiisyLPpxRit2UG91IZrmXBDsMcaJ7CbdaWISTW7ZhX/aLHIjXfjrKfwXkeWKeThdj/IYibFZm6gR3X2cJs4ezgkh7skLQwsfyBTmi1pHWwM/3+FNKG6SStleQNC/g5lgz1dvOrTjHVqBgMjEUvOlEieUktoicV6pgMjAyQHGtw6OvrNv+QqCrk6zExF1o5Hl72xgDgkOxMQYpSkPhO5gJ+17EptLPRZeBj/lcXFpAm0BuXwH46AmOcPW+rGPUNSmjE3s+BVLcZxGBj9H+VulKN1X4ZK2frEndZVO+7HNOw48+x9Yx390+x/IP1MBaKEp5IcQZCb8+gzdUJ8PpUAMooeU+LNqGO+j+eWbnq7ZVxCrJh1XMmjl9piL/5Og1Ip+moQhZmZK35U3MCH8GUS0MdQGUGxPSUT68nkCAfClwc39TK5qq9S39i+UzbKHbSpLLlZEqkvpEbLTgwL/NsX8D9RmiBvPDeKG/S8NnzISxRKxDj+gM5vpCaA8WVUW/J3J3nHCuzAt1bZgr EAEcM5Nf GwqD7p6Cn7ShZ+So83dtulgSMS4wR5uHjkTBXGb3MKreOGxCk/V/5NfBqrmWPmQTXm49j3yX1vPPImXkZkaW66BnBKhHIbVVzfadAsthZLmN0Vy4zPXEnnRSB8qUIZTAWNh26sY1qyo8yJ+PQau2tRCLcn7yVJbUbF1W0H9aLylpvrCX3aUdz/0l3dEitcEH5eXHEZvka2X8OlQ4+FNG26LT3AJbdi9k6w8QSId5qrswIATAMTM+HtBMch8/P9A1mXxpzrVwu0b4DS2uankZmLd1ddX/7VTsT+bfJGGv2j0ltrNMhFcV1nILa9Jz/RukQ5j+19tg1VnoIQI4l6YIOtQkolB1WHCn52Y10prM5n2fwTLdPqMumxP/Jk6cfMtskXScvX6nFsRjsAJvWwBSLPMFKDPNqJC1rehCZi4/kR916F+H6nzfEFJ8jH8TYxIcTkNWrMmRFzVkeaY/AnG44yRMGnl7iigaqm/slGGCzJoUEsdwJPGoRA/L2gNxdEx+3U1DgTXagb1v/CBrtQApoa9LlufD6c88ROUlwHCL3gX83M5njNxMrYXXrr3fPnEPZVd/ROMaB0cMHHYFOHYAGCkNJ11H3jEstjm53+fLZOZ3C+BvogHdWt+nkU7XJUP+bPX2iBzqhaZyEvuaRPJjAo4JERr2VQDs7GAstKiO3Cr6h6tLuqiHCqJU3N3EReugjs5XlxZoZHGemrfz96YjPzL5mNOqECkINF/nPZc69plHQRHGCGOh+LEQXlt5fnwHm5DqrJF2Q03DHmpxYGZHifNWMg9TSEReu1dNWmYgPeIR18UPpe691sL6ocxawr4tZxIWd X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: As Johannes notes in [1], stats_flush_lock is currently used to: (a) Protect updated to stats_flush_threshold. (b) Protect updates to flush_next_time. (c) Serializes calls to cgroup_rstat_flush() based on those ratelimits. However: 1. stats_flush_threshold is already an atomic 2. flush_next_time is not atomic. The writer is locked, but the reader is lockless. If the reader races with a flush, you could see this: if (time_after(jiffies, flush_next_time)) spin_trylock() flush_next_time = now + delay flush() spin_unlock() spin_trylock() flush_next_time = now + delay flush() spin_unlock() which means we already can get flushes at a higher frequency than FLUSH_TIME during races. But it isn't really a problem. The reader could also see garbled partial updates, so it needs at least READ_ONCE and WRITE_ONCE protection. 3. Serializing cgroup_rstat_flush() calls against the ratelimit factors is currently broken because of the race in 2. But the race is actually harmless, all we might get is the occasional earlier flush. If there is no delta, the flush won't do much. And if there is, the flush is justified. So the lock can be removed all together. However, the lock also served the purpose of preventing a thundering herd problem for concurrent flushers, see [2]. Use an atomic instead to serve the purpose of unifying concurrent flushers. [1]https://lore.kernel.org/lkml/20230323172732.GE739026@cmpxchg.org/ [2]https://lore.kernel.org/lkml/20210716212137.1391164-2-shakeelb@google.com/ Signed-off-by: Yosry Ahmed Acked-by: Johannes Weiner Acked-by: Shakeel Butt Acked-by: Michal Hocko --- mm/memcontrol.c | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index ff39f78f962e..65750f8b8259 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -585,8 +585,8 @@ mem_cgroup_largest_soft_limit_node(struct mem_cgroup_tree_per_node *mctz) */ static void flush_memcg_stats_dwork(struct work_struct *w); static DECLARE_DEFERRABLE_WORK(stats_flush_dwork, flush_memcg_stats_dwork); -static DEFINE_SPINLOCK(stats_flush_lock); static DEFINE_PER_CPU(unsigned int, stats_updates); +static atomic_t stats_flush_ongoing = ATOMIC_INIT(0); static atomic_t stats_flush_threshold = ATOMIC_INIT(0); static u64 flush_next_time; @@ -636,15 +636,19 @@ static inline void memcg_rstat_updated(struct mem_cgroup *memcg, int val) static void __mem_cgroup_flush_stats(void) { - unsigned long flag; - - if (!spin_trylock_irqsave(&stats_flush_lock, flag)) + /* + * We always flush the entire tree, so concurrent flushers can just + * skip. This avoids a thundering herd problem on the rstat global lock + * from memcg flushers (e.g. reclaim, refault, etc). + */ + if (atomic_read(&stats_flush_ongoing) || + atomic_xchg(&stats_flush_ongoing, 1)) return; - flush_next_time = jiffies_64 + 2*FLUSH_TIME; + WRITE_ONCE(flush_next_time, jiffies_64 + 2*FLUSH_TIME); cgroup_rstat_flush_atomic(root_mem_cgroup->css.cgroup); atomic_set(&stats_flush_threshold, 0); - spin_unlock_irqrestore(&stats_flush_lock, flag); + atomic_set(&stats_flush_ongoing, 0); } void mem_cgroup_flush_stats(void) @@ -655,7 +659,7 @@ void mem_cgroup_flush_stats(void) void mem_cgroup_flush_stats_ratelimited(void) { - if (time_after64(jiffies_64, flush_next_time)) + if (time_after64(jiffies_64, READ_ONCE(flush_next_time))) mem_cgroup_flush_stats(); } From patchwork Tue Mar 28 22:16:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13191619 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8FE20C77B60 for ; Tue, 28 Mar 2023 22:17:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0C0256B007E; Tue, 28 Mar 2023 18:17:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 073876B0080; Tue, 28 Mar 2023 18:17:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E04E26B0081; Tue, 28 Mar 2023 18:17:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id D133A6B007E for ; Tue, 28 Mar 2023 18:17:04 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 964A440694 for ; Tue, 28 Mar 2023 22:17:04 +0000 (UTC) X-FDA: 80619718368.06.11CF17E Received: from mail-pf1-f201.google.com (mail-pf1-f201.google.com [209.85.210.201]) by imf19.hostedemail.com (Postfix) with ESMTP id C63751A0020 for ; Tue, 28 Mar 2023 22:17:02 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=IxPRAGCq; spf=pass (imf19.hostedemail.com: domain of 3XWcjZAoKCOAaQUTaCJOGFIQQING.EQONKPWZ-OOMXCEM.QTI@flex--yosryahmed.bounces.google.com designates 209.85.210.201 as permitted sender) smtp.mailfrom=3XWcjZAoKCOAaQUTaCJOGFIQQING.EQONKPWZ-OOMXCEM.QTI@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680041822; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rItEK+KYvsSIiuNXlyB23Cs11TTT5fJ8aFIXbrMiIEw=; b=ueb8HmuOvmZZQk88/pEN0qAqNE8w667blEAAJ6h2yGDPW5OpHFaTTk+1IedWwq/fbYMuK7 3b1cixZvHfOf2sS1/XaeZ1RmC/A6U1oXvBqVuRLplH+BiRg9VDoRWVLevqDNr75g/d6mH8 blUULSUHAhEe9OrOgW/gajaZJjbXDYQ= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=IxPRAGCq; spf=pass (imf19.hostedemail.com: domain of 3XWcjZAoKCOAaQUTaCJOGFIQQING.EQONKPWZ-OOMXCEM.QTI@flex--yosryahmed.bounces.google.com designates 209.85.210.201 as permitted sender) smtp.mailfrom=3XWcjZAoKCOAaQUTaCJOGFIQQING.EQONKPWZ-OOMXCEM.QTI@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680041822; a=rsa-sha256; cv=none; b=heZg2904H/tKbgmKGiSrjOnlZFLDtrY6n46O4mLdSIOEH3/uZI6Edg4Yascq5+GpoASrWI tui/2QKmKrGFo5OaQ2BBHMj6kLj5OMcFT38sfnpr+xzROrfvDofs63OyaVwJdUQhkhkawA 67CFPUKBM62QkcFqw96Hb5rCSMt9FW0= Received: by mail-pf1-f201.google.com with SMTP id 16-20020a056a00071000b00627e9b4871eso6416343pfl.11 for ; Tue, 28 Mar 2023 15:17:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1680041822; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=rItEK+KYvsSIiuNXlyB23Cs11TTT5fJ8aFIXbrMiIEw=; b=IxPRAGCqLmO64hebIOFNkgw1YGUDJSVyfgVVBwxUcvdgL59f7l4LN5zMPrXb+jjbv/ 44AuhCK3u7FtNfz2jZoS0YDY1UIo89Iub/aEWQsolPiMTjtNQ86ze7JWsd1EKsY5Y1Aw 07n0xvSJnL2lwlEm697GGmrpvMBX0F+5BNT3XjdQx09Cev4XnMOYZ7hcuMbACVnhAT8t T+Q0avDfvu5GCgeBYFMaudoWaGLkL6vKFSabNVsNzlmPSiAqWuR+oN/II5DxoATas6tB N+Po8vyHrMmHA1KarR8lOaWajBlxLkTT4ktZV7Et/fSUfAvv7rnsRtElYoPsoIvIEGG6 moFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680041822; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=rItEK+KYvsSIiuNXlyB23Cs11TTT5fJ8aFIXbrMiIEw=; b=rq8X8HNzXmCML5m4bVS3YxKkkDAeuTzpBi0804fZ1q2Z5CFJ6z2a5wnE0oQrqK8Y1W WPzOsj3CJppoYFiOAEqGkdd3PV87tBFc8qQbIJONphhSZ1QsGHtgHFblwzfcD6J6gG9e sazMvr+6+a+vJJi8CffhANoFxeqto/XL0sxfssMbNUSXkM/qRwP0mmgfTsiVsXWhatMn r0EksB4YH06WJL+4G9Vv7lWO9L5csuWziF2N8WW/rIuwWadj0+uryHYbIeIaPHBXtSls AaNcjIkRPeebqOjaM09o2Xl1hi9mlwYJFF97xbMeAxgKTPqJi4JU0Ndi+9gbJLCs8L2H 03EA== X-Gm-Message-State: AAQBX9ffEjFN9tFgy4e6JNuiHhqOudWZcMRMFuQD5wsZJsYks9TFRzwr z+UQpSHSZFf0OI6Eeve3nXEwCEE3TQHt3mIx X-Google-Smtp-Source: AKy350bXvyCoGIAXSg9szgc2JGdZUl73nsfa2Et7WueltHgrvj0bY0SpfT6d/5uVqOXZZR9I4CnMF+3pkDT4wLPq X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a63:ce08:0:b0:50c:6cd:cace with SMTP id y8-20020a63ce08000000b0050c06cdcacemr4745262pgf.2.1680041821773; Tue, 28 Mar 2023 15:17:01 -0700 (PDT) Date: Tue, 28 Mar 2023 22:16:41 +0000 In-Reply-To: <20230328221644.803272-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230328221644.803272-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230328221644.803272-7-yosryahmed@google.com> Subject: [PATCH v2 6/9] memcg: sleep during flushing stats in safe contexts From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , " =?utf-8?q?Michal_Koutn=C3=BD?= " Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: C63751A0020 X-Stat-Signature: rpsfrqybu3goyamgiczsk7f8uqgefeem X-HE-Tag: 1680041822-549801 X-HE-Meta: U2FsdGVkX1+4u9hogyJA4/b9byHOb2npALiIsjjCntW+cqKSq3uJ4sUhYfCuG+vAettLDD51a5a4ZpEoGi9LNvmOHU1n5m3+Jp54OY2nTGNnkkQAscuRpC/t39XpU4BqDiKyJkwgwP1dyh1UchzvYXfWHCzzm3PNWZxtONYkOsF+V/9QKuTy2EmhAAXdTGu/BIm16xPoXb0DLCMSn9tVW4ujK0E7dTyW1oMZPBT6OZb44P6oRZK9QsoMRxLknMVw9xROLVYn0IrN5OVJKc3fgfvyeMzpFLp41aRjh9wsVMYY++w5vsFBOX+FGmEdZujNM6Ej4KgZUszNrYjbIx0L4p5+EMbIp/gfcQnUaBGOjl50HFqs8wxV8p9qTyONGeMP5oZjWWfG9sQYvRrk6QNhb1o4pW0pSyrhl8UBU2MzCVCsDgBrUS5REM7AZ8hg/JnNQp2X1h03BWMa1k2DBscTIj7Rtbf9C5sWEEVx1425jQl43xQaCGnu2qaZWCfw09H5B7jf5/XfNF1faCvwPYxK2zPslSj+EmurLYmxKop+Gk5LC12ZINm+C2KCg+2O+kVTUgAOoEN7iUT85YeIeYXAWrQnaK4eUUxdx2CnBp2vv9Aha/M2awdLZ701hDq54R4PAY2lc7BBLDdpuQxvtI2MEIbYNAkEraDI+7+Jl64WTYT3XQmWzFqvrcuXnSKXU0VOyD+w8VucxUgY2qCj3GsfmEJx6vwrkA5BmTucmGAQUKVMIcUjCbC47JEvb7MT1C/OM2yVeGG9a2xGtBtyYEslBg+0Nm9/vaK5WWkDO6rTliONrZHww8QgOJJ5noI36eGj+Wceg9sz1Y+DLWlEA5aD06gRAMzGR17YF2Ka0pZpxy/LotEO5RNoKMNQEhIFqy29uht4WH+8MmzcP+a3CO7GhcrNIbmIOMG5CCW9z9LvVX1O30i0V1glXAfkXoSfgE7DO9YR5mUGbIT1PZ9YNTn qhZ8eLCw D4D7PCfd4fjD8Xr0jX7bvK2hCzRdnjN1QIILVKsgsiyfX3DsjayH0GTzIgB0WPqcpiAsibUv1vhIvMz5yhQOT/gJ5GRMMYGxqVW33qUM0GmFxjVhljUU3xPmx9gCOQJ9ddU5Dbepm2RQvNIaVL/54XoY85qjFBkNIO+Bo16FJP3POwt8raZgNHo0SmsXTjDT8+Rzsiesk9+cKkrfaPPOYKQ9G9pU+OmRpnddAaBCoAh3vXQ6+jiw8h74f2jfz0zyBKdO32IxVDOsGFdG5WzK/rk7dhYB/zMSANXAKFJUuQXVEYQzJA7mFqyCSrlEZbx4ysbcnVHl3lm+0FXILTpd5aHxMavmh65ysyi78z/dyLCx3TAqSd5n34lhaXmi3xUsczi0HxJJMlIgMWU2OM7w6gQvSuKLxVGNmANCN0wsLIcL+mP2SqcXe5Dx1GBQYM7IHms6NGJ7HXY8QxV3uT9oh/AuIhwDpFfL4o1cXB7dsSM559VZjq0HgMwway3FyHPfIRKK/4Ji0LizftAwr1T5N1UkcCLJjW/lAw3bBaEHOxwitTMx9QveRYAMIgqQumtF3oUrDOgdTqCOnLyc2ivHyCFwZGgsY4zqrqugmjcrnXt4ciu6UO/Q3mQqUBjJ4u/eGJzpnWRy3vKJdAOPIGySukxcmTibk9YRfHOTs X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently, all contexts that flush memcg stats do so with sleeping not allowed. Some of these contexts are perfectly safe to sleep in, such as reading cgroup files from userspace or the background periodic flusher. Refactor the code to make mem_cgroup_flush_stats() non-atomic (aka sleepable), and provide a separate atomic version. The atomic version is used in reclaim, refault, writeback, and in mem_cgroup_usage(). All other code paths are left to use the non-atomic version. This includes callbacks for userspace reads and the periodic flusher. Since refault is the only caller of mem_cgroup_flush_stats_ratelimited(), change it to mem_cgroup_flush_stats_atomic_ratelimited(). Reclaim and refault code paths are modified to do non-atomic flushing in separate later patches -- so it will eventually be changed back to mem_cgroup_flush_stats_ratelimited(). Signed-off-by: Yosry Ahmed Acked-by: Shakeel Butt Acked-by: Michal Hocko --- include/linux/memcontrol.h | 9 ++++++-- mm/memcontrol.c | 45 ++++++++++++++++++++++++++++++-------- mm/vmscan.c | 2 +- mm/workingset.c | 2 +- 4 files changed, 45 insertions(+), 13 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index ac3f3b3a45e2..b424ba3ebd09 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1037,7 +1037,8 @@ static inline unsigned long lruvec_page_state_local(struct lruvec *lruvec, } void mem_cgroup_flush_stats(void); -void mem_cgroup_flush_stats_ratelimited(void); +void mem_cgroup_flush_stats_atomic(void); +void mem_cgroup_flush_stats_atomic_ratelimited(void); void __mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, int val); @@ -1535,7 +1536,11 @@ static inline void mem_cgroup_flush_stats(void) { } -static inline void mem_cgroup_flush_stats_ratelimited(void) +static inline void mem_cgroup_flush_stats_atomic(void) +{ +} + +static inline void mem_cgroup_flush_stats_atomic_ratelimited(void) { } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 65750f8b8259..a2ce3aa10d94 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -634,7 +634,7 @@ static inline void memcg_rstat_updated(struct mem_cgroup *memcg, int val) } } -static void __mem_cgroup_flush_stats(void) +static void do_flush_stats(bool atomic) { /* * We always flush the entire tree, so concurrent flushers can just @@ -646,26 +646,46 @@ static void __mem_cgroup_flush_stats(void) return; WRITE_ONCE(flush_next_time, jiffies_64 + 2*FLUSH_TIME); - cgroup_rstat_flush_atomic(root_mem_cgroup->css.cgroup); + + if (atomic) + cgroup_rstat_flush_atomic(root_mem_cgroup->css.cgroup); + else + cgroup_rstat_flush(root_mem_cgroup->css.cgroup); + atomic_set(&stats_flush_threshold, 0); atomic_set(&stats_flush_ongoing, 0); } +static bool should_flush_stats(void) +{ + return atomic_read(&stats_flush_threshold) > num_online_cpus(); +} + void mem_cgroup_flush_stats(void) { - if (atomic_read(&stats_flush_threshold) > num_online_cpus()) - __mem_cgroup_flush_stats(); + if (should_flush_stats()) + do_flush_stats(false); } -void mem_cgroup_flush_stats_ratelimited(void) +void mem_cgroup_flush_stats_atomic(void) +{ + if (should_flush_stats()) + do_flush_stats(true); +} + +void mem_cgroup_flush_stats_atomic_ratelimited(void) { if (time_after64(jiffies_64, READ_ONCE(flush_next_time))) - mem_cgroup_flush_stats(); + mem_cgroup_flush_stats_atomic(); } static void flush_memcg_stats_dwork(struct work_struct *w) { - __mem_cgroup_flush_stats(); + /* + * Always flush here so that flushing in latency-sensitive paths is + * as cheap as possible. + */ + do_flush_stats(false); queue_delayed_work(system_unbound_wq, &stats_flush_dwork, FLUSH_TIME); } @@ -3685,9 +3705,12 @@ static unsigned long mem_cgroup_usage(struct mem_cgroup *memcg, bool swap) * done from irq context; use stale stats in this case. * Arguably, usage threshold events are not reliable on the root * memcg anyway since its usage is ill-defined. + * + * Additionally, other call paths through memcg_check_events() + * disable irqs, so make sure we are flushing stats atomically. */ if (in_task()) - mem_cgroup_flush_stats(); + mem_cgroup_flush_stats_atomic(); val = memcg_page_state(memcg, NR_FILE_PAGES) + memcg_page_state(memcg, NR_ANON_MAPPED); if (swap) @@ -4610,7 +4633,11 @@ void mem_cgroup_wb_stats(struct bdi_writeback *wb, unsigned long *pfilepages, struct mem_cgroup *memcg = mem_cgroup_from_css(wb->memcg_css); struct mem_cgroup *parent; - mem_cgroup_flush_stats(); + /* + * wb_writeback() takes a spinlock and calls + * wb_over_bg_thresh()->mem_cgroup_wb_stats(). Do not sleep. + */ + mem_cgroup_flush_stats_atomic(); *pdirty = memcg_page_state(memcg, NR_FILE_DIRTY); *pwriteback = memcg_page_state(memcg, NR_WRITEBACK); diff --git a/mm/vmscan.c b/mm/vmscan.c index 9c1c5e8b24b8..a9511ccb936f 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2845,7 +2845,7 @@ static void prepare_scan_count(pg_data_t *pgdat, struct scan_control *sc) * Flush the memory cgroup stats, so that we read accurate per-memcg * lruvec stats for heuristics. */ - mem_cgroup_flush_stats(); + mem_cgroup_flush_stats_atomic(); /* * Determine the scan balance between anon and file LRUs. diff --git a/mm/workingset.c b/mm/workingset.c index af862c6738c3..dab0c362b9e3 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -462,7 +462,7 @@ void workingset_refault(struct folio *folio, void *shadow) mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + file, nr); - mem_cgroup_flush_stats_ratelimited(); + mem_cgroup_flush_stats_atomic_ratelimited(); /* * Compare the distance to the existing workingset size. We * don't activate pages that couldn't stay resident even if From patchwork Tue Mar 28 22:16:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13191620 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5034DC76196 for ; Tue, 28 Mar 2023 22:17:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 30AA86B0080; Tue, 28 Mar 2023 18:17:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 21EF96B0081; Tue, 28 Mar 2023 18:17:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 097BA6B0082; Tue, 28 Mar 2023 18:17:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id E47F76B0080 for ; Tue, 28 Mar 2023 18:17:05 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id C82C740B3A for ; Tue, 28 Mar 2023 22:17:05 +0000 (UTC) X-FDA: 80619718410.27.4B7105C Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) by imf11.hostedemail.com (Postfix) with ESMTP id 140564000C for ; Tue, 28 Mar 2023 22:17:03 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=j2h7VlEd; spf=pass (imf11.hostedemail.com: domain of 3X2cjZAoKCOIcSWVcELQIHKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--yosryahmed.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3X2cjZAoKCOIcSWVcELQIHKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680041824; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=a7F6fk9fZnPdw9ddOVIzAjBdiTpqd88vdaggqxhF/qs=; b=HBVoOEL3Vv+dVdM+XgmwNukui/JBRuImZuSf5wXocBdicvCMSXEQdJ6KBNtLtECYxir2FE ogrhblvwfsYG7ozuid8DfVo4EgQFeV1WDV2SMD4X+uGQweZyERD+bgYMoQ2gQONa+GeXBA +xIrPFS7RcHmGC7zxUGv5rgGL/MLYx4= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=j2h7VlEd; spf=pass (imf11.hostedemail.com: domain of 3X2cjZAoKCOIcSWVcELQIHKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--yosryahmed.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3X2cjZAoKCOIcSWVcELQIHKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680041824; a=rsa-sha256; cv=none; b=RTktQ4zUW9CQo7sAF+SnBfVtGC0qezLSMm0pAHtPzMvu5ZjXGshTHC48+5YNgCWFaNdEs0 PclHbt85aMdU0pLJaelY5w81wx1BVABvsy4qvcBpRQUYmkOheGet/MI7pT8+EDm609VqEF bxSanzzfGfvILEMzGtLQ2hMi5AFf5GM= Received: by mail-pl1-f202.google.com with SMTP id k3-20020a170902ce0300b0019ca6e66303so8351661plg.18 for ; Tue, 28 Mar 2023 15:17:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1680041823; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=a7F6fk9fZnPdw9ddOVIzAjBdiTpqd88vdaggqxhF/qs=; b=j2h7VlEdeGIL4dqtCRH6krZRXCzlm8zdD/VKFrhl2Ix5hroKwD9u6hjC+9l9HOi6UV H1JPSNu1LdgWvdxHE9hg+SM93x1Jht2L9pd8vgLQEuBZlRpZngZUgkiaZwaZa4gCkUC+ zgspBsM6/B8TFiwy4IbbIfRW24OLGYSFku2vitXovcEwbuAN1td3zyOf9AyMfligZ5Lp BhTKdBAPhJ+a7QI4Cj7LZxYyn4A2yXYMpirZryPEzjspDaCYUkIzUrMU1Kd6ErRoShgT XFMZOBTc6edK5ZIe1IhscJC8wUQ57Xb89+9zAOCcDb3JVfb8T7SfV7MPXlrp4JguLYCz aCRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680041823; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=a7F6fk9fZnPdw9ddOVIzAjBdiTpqd88vdaggqxhF/qs=; b=SQVgVKhQkv9+WDCj1aHG1IZ1s3FSqYQvZV6ZevhaqjsrDh9qhh31yHuoSpK249CxXw fuNtHsYBz16b+5zeOcf9WfwlmSxX5dG4apdZ6Ql7W4xJLGwMLNWK4H2T5m131Z4PMnI0 zHx2gHnuCQujoxXg7G2DQuZa3/agipnHQGIxI3nyo9a85yVaJzyzza94E4HP/ZJv5qOu LAqHNZrUk/pBNZEikX6NGnkmGaZabuDDuf7taQcJ9ij81Q+YKIYY5NsSclmVqFHX1RZ9 EKoXVFXBXKADAx1wRnMeKNrRjsmAZG/zvaTfXl6RNmmapZJH0+88ldBAKH1Bb+l+kUbQ nRlg== X-Gm-Message-State: AAQBX9eXu0Jo8PSpi8+QzYkk1A+kpoKJ/VdbJ8o5prUR36FO9QB29IZu 5AlcJqsufivXtM9P/r1dKRzHF5zNs7z+oTG5 X-Google-Smtp-Source: AKy350YX9lhtezGewKcmJEZ6+PWl0qdkFOVRrH2qPmewBQAi0Jj2O6028tHTh6zpbuIXp3Yq/V52VHvowCQfk3sU X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a17:902:ec90:b0:1a0:763d:6c2a with SMTP id x16-20020a170902ec9000b001a0763d6c2amr6633613plg.10.1680041823533; Tue, 28 Mar 2023 15:17:03 -0700 (PDT) Date: Tue, 28 Mar 2023 22:16:42 +0000 In-Reply-To: <20230328221644.803272-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230328221644.803272-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230328221644.803272-8-yosryahmed@google.com> Subject: [PATCH v2 7/9] workingset: memcg: sleep when flushing stats in workingset_refault() From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , " =?utf-8?q?Michal_Koutn=C3=BD?= " Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 140564000C X-Stat-Signature: rt6y6zyj95sunfgkirizmfxqxck4pir4 X-HE-Tag: 1680041823-269731 X-HE-Meta: U2FsdGVkX1/5ZaYk5qBOFr+5ecwgEGw0InDzox3Ae4bEJ2JtDjUugGUhEIM4ODSF9YBtBfphyNP2ZYqTaQat2q29tG7dw7x1qU1ekMDAXC5zDOwaRynm1wcGMmn13s4EWtCiqRmUkCe7RBzMJ7g8UKDrdB8Zur+uzdyWEgxplngfL7RNi9NAlSepa3N+38Ihzk9U8/t+QgdAy1fSo6Ci2yuODaH7dutvLVzgQsG/i6G00YhHM5oJMFBl5ll7J4X+3EmCWFHNnvfdhaIbyE3gVjQSy5Cys5npO3KR703h7SEYJV6Egbk5Y0TVj9vhbwPA/A+y+KLFCFWl8XSIllNDB1II+fP0i2EJ7/LSgZ+ACWH11wOIxQ5Pap1NnUyPWX1rMRnQLKt7BWAhx8ZaZP/5n2pL8Hz8Q6dhJdoPHIxU0uPgGzLOWpXJK1cCrhtiJk2yswmOaErxUsoxZj3jkQ40354F9mUGiTMv18njqPIWqTAWUazQ4snshUGPo1hzz6IOpZpvthC7fiL75+ovYu0GA1A9PYFxpDrq+08ZA3ujri0zGI0iiV36TfVaz1cL+qX18T8b4EyofmvIzZKX67izryq8UkfkhBYN4wQRILZoR6i6a6rOJUGv9lwEySSpOmGHhgqg1JIOxrpBbVb2HUrbP7d6lJEe/LvpZwDxjIIRbPBr+Czk/J8KG4PDuLGqBRyPZJhw8/684DeznV41D7rZH5jfPXOxkmgDL/GJHBIweBfYvLgv1FRMK2Y9C0f4DvRmNcj2F5sURHxkTmosx4/nKO461y1+3KYQJVFEODFDl27G6Ht/EYMkrROwUncGnadWjmsEo1sTbhFIK65NEgQQhUIXrGK3/RJ1+sAi7umEuh9jEZJIweUu/CVoGdvhCHGR5tMGm8+MNhnMKDmhd1H3hZwAhfYr4oXPGvsdp+DZvFEUXejg6Ait30Vmw4VYpAoHkaoPDkx/frA1pby9T+9 aEBlMbWL 8FsyIXIY7fzIc5QMWdGKBLdm7Jvf6vFd642Z+GIgfCdIsk4BlUA5+FlXDPFKkWPnJrdotVsHTParGEDPIs+iVpAXHIEDUKMq6rs+2p1T/bnk9Y0/NK55fpGwvfw0rCaEF8Q9mkzaIZkOL9GY8+py/HOZDNU2+/Nu5ifFDQ27DoatyWcbdXzLjWxl2RqV674z4y0rzVy+tfc0PYLcZ4ZKTK2xbfGyYF0hDyijTpIL9SjnwCfWVI8zd6hmrZnbDTlXyz82KKGOMRNwE+9k2kjDE8MbPO7ETqsh2ZnBA5+RdRBCd0iOsQly8lvRh2JfIT1AsBvpJEsUya1pbHDWwogJVcv47sGkNNvGZYaBELfhQbZe4aI+8h3rxXNkc3s04Cgz506TTxVPEUERQao7KGMLCoxsoc4/SUMUamoMIV6Rj29taLaOmHtwDo//BiD6HjyWiTYDae/xxPRJk/xO2uKZ1/L0v3vvCceKC/+KoecN7fXSauYavKjHlWnhkAcwh60aahoTRYEgtErbulosJ670U23a8rKtQundXaqhU+SEgVnGSQxsvrM7XSzq063t+kyR5/+M0ph67vRsSDa4EH/3QBU9no0FB0qZb7E7Ftalb0kbiIVHKywf6iZCu4PCahlk1gF391y6y0CQsg1gNp4DGrzvKh1XwVhuPi3BWsxBcgJdnDI7ziUGIdc6401vSE/PnzawPt51AWokcZ5XCmVW0C5psvTCxRUixwiXoEGp8/sp76JNJQS5D22JUym2UBhNstIx+kAqh3Zw8eAM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In workingset_refault(), we call mem_cgroup_flush_stats_atomic_ratelimited() to flush stats within an RCU read section and with sleeping disallowed. Move the call above the RCU read section and allow sleeping to avoid unnecessarily performing a lot of work without sleeping. Since workingset_refault() is the only caller of mem_cgroup_flush_stats_atomic_ratelimited(), just make it non-atomic, and rename it to mem_cgroup_flush_stats_ratelimited(). Signed-off-by: Yosry Ahmed Acked-by: Shakeel Butt Acked-by: Johannes Weiner Acked-by: Michal Hocko --- include/linux/memcontrol.h | 4 ++-- mm/memcontrol.c | 4 ++-- mm/workingset.c | 5 +++-- 3 files changed, 7 insertions(+), 6 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index b424ba3ebd09..a4bc3910a2eb 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1038,7 +1038,7 @@ static inline unsigned long lruvec_page_state_local(struct lruvec *lruvec, void mem_cgroup_flush_stats(void); void mem_cgroup_flush_stats_atomic(void); -void mem_cgroup_flush_stats_atomic_ratelimited(void); +void mem_cgroup_flush_stats_ratelimited(void); void __mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, int val); @@ -1540,7 +1540,7 @@ static inline void mem_cgroup_flush_stats_atomic(void) { } -static inline void mem_cgroup_flush_stats_atomic_ratelimited(void) +static inline void mem_cgroup_flush_stats_ratelimited(void) { } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index a2ce3aa10d94..361c0bbf7283 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -673,10 +673,10 @@ void mem_cgroup_flush_stats_atomic(void) do_flush_stats(true); } -void mem_cgroup_flush_stats_atomic_ratelimited(void) +void mem_cgroup_flush_stats_ratelimited(void) { if (time_after64(jiffies_64, READ_ONCE(flush_next_time))) - mem_cgroup_flush_stats_atomic(); + mem_cgroup_flush_stats(); } static void flush_memcg_stats_dwork(struct work_struct *w) diff --git a/mm/workingset.c b/mm/workingset.c index dab0c362b9e3..3025beee9b34 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -406,6 +406,9 @@ void workingset_refault(struct folio *folio, void *shadow) unpack_shadow(shadow, &memcgid, &pgdat, &eviction, &workingset); eviction <<= bucket_order; + /* Flush stats (and potentially sleep) before holding RCU read lock */ + mem_cgroup_flush_stats_ratelimited(); + rcu_read_lock(); /* * Look up the memcg associated with the stored ID. It might @@ -461,8 +464,6 @@ void workingset_refault(struct folio *folio, void *shadow) lruvec = mem_cgroup_lruvec(memcg, pgdat); mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + file, nr); - - mem_cgroup_flush_stats_atomic_ratelimited(); /* * Compare the distance to the existing workingset size. We * don't activate pages that couldn't stay resident even if From patchwork Tue Mar 28 22:16:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13191621 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1355FC77B60 for ; Tue, 28 Mar 2023 22:17:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 220FF6B0081; Tue, 28 Mar 2023 18:17:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1817B6B0082; Tue, 28 Mar 2023 18:17:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EC77A6B0083; Tue, 28 Mar 2023 18:17:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id D74C66B0081 for ; Tue, 28 Mar 2023 18:17:07 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id B2FEF1204E8 for ; Tue, 28 Mar 2023 22:17:07 +0000 (UTC) X-FDA: 80619718494.09.0EA95A7 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf01.hostedemail.com (Postfix) with ESMTP id F02C940013 for ; Tue, 28 Mar 2023 22:17:05 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=s2Rhy6bh; spf=pass (imf01.hostedemail.com: domain of 3YWcjZAoKCOQeUYXeGNSKJMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--yosryahmed.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3YWcjZAoKCOQeUYXeGNSKJMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680041826; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=AvW0vRtvUGz6RfpHjuJf5DcrLtUCw4uDLtzj7kPXkaI=; b=sT10cfpzk9V9W2cTNYc9Jk1gwaDHG21+j5Aup/YnNafzkBztWJGbGmGmXQWA5b5yis4ze2 1GnSBzfZHHCz9vbZHr0YswdkzAvQ7Xcdy/xiv13I5+kYyzMJ8ptmyyuJ8qFkbRMDWQM4gl xr+rirw536dhhIVPT7PnMTExefgSrhw= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=s2Rhy6bh; spf=pass (imf01.hostedemail.com: domain of 3YWcjZAoKCOQeUYXeGNSKJMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--yosryahmed.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3YWcjZAoKCOQeUYXeGNSKJMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680041826; a=rsa-sha256; cv=none; b=fof3oGczjNrdnN3K8cxBlN8KH32cBSpa/udG2smTFaSnYnSlm7kPJz1e7XBofmzeB69U7B idnNHwcNBTsKaDxavw/tmP7MzkIy4y2/bgHojz7/dBPRyfjH9/AWOAjLMesUxhPV5C2xfC pS3gEUUcr9BxPtCJavy9n8gSXvWqXT4= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-536a5a0b6e3so136900407b3.10 for ; Tue, 28 Mar 2023 15:17:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1680041825; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=AvW0vRtvUGz6RfpHjuJf5DcrLtUCw4uDLtzj7kPXkaI=; b=s2Rhy6bhmsxOtwUgNmtWPZ1OIy2px3+ZgcKdui8SYbOasl9GBmkC/HgOP+eca1phsP jP5oqLAVmD71L+yTcF9Qf7RNoZtsSSFPX0S6Mq6S5HfUed3nW8TUcaJtaUm/tsI8O9Am d5OJg/swbll5+1GgM/EDnx0GyYjs89Yzqk1B1Gc+8UCS388fxVzDgMBm93/UJRyrTJYr f1CeTGEfaowg6ARY0FoahfZMcN2kOGoi1UzCuTmmzShA86ZvRPo5Rcmo6znhWB9xOjYc VRAX/Gquhhjvyizfw+QszvCjk77JT0+yZFVam4Xqz3PGsmr5F2QrldBe6NoQ8SARTtvR pWDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680041825; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=AvW0vRtvUGz6RfpHjuJf5DcrLtUCw4uDLtzj7kPXkaI=; b=Dm5/CAySmkrxApGXmakl4TS5kEpAnPr795xHyEx9VriJfgnUUeXMFkEH4sRopw419s 36KOhNWVrtOyA8VocHPojLQwUGwLlGa42x1hfRIJKmGuoTSJsw74+Wjgae0MDgDlohgy fLGW4Vee6aLMDslcde/8f16Yr8LRvEHWgA25uAa9l1AS5ag0cokLiEG2D7lo5pdVo1/x /SpmudfiO9x/vAKzk3Veyu7u6LiAj24nUZkvsgCQ+3drkCKg4Rj8ohbIZRwWpErCouEt 5ao8ppMrvFAAFAG27rVrOpedPdxHD5KDLCmAE+ro52Hcan7NeTVCiVL4ZEjXGwTdKb9m 7nDw== X-Gm-Message-State: AAQBX9cIHPDq60Qb6/NC8RzM7f3iqukzVhitHC9fH3cjt68fpvC63a/0 75I9ULuALkgn5/pDvhVoerwJroY+p6tt7rd2 X-Google-Smtp-Source: AKy350aC/5JUEL/+7c/dIUP+7KT0Wk9JcWhzZQ4Xw7OfNYXAfRzlQJVntfC4MTRTs3t2TIUHulQT5W2wJHMWSZ0v X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a05:6902:722:b0:a09:314f:a3ef with SMTP id l2-20020a056902072200b00a09314fa3efmr10887537ybt.12.1680041825151; Tue, 28 Mar 2023 15:17:05 -0700 (PDT) Date: Tue, 28 Mar 2023 22:16:43 +0000 In-Reply-To: <20230328221644.803272-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230328221644.803272-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230328221644.803272-9-yosryahmed@google.com> Subject: [PATCH v2 8/9] vmscan: memcg: sleep when flushing stats during reclaim From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , " =?utf-8?q?Michal_Koutn=C3=BD?= " Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed X-Rspamd-Queue-Id: F02C940013 X-Stat-Signature: siaip7r3bt8t3y1ww98pzar4broktjci X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1680041825-162608 X-HE-Meta: U2FsdGVkX19wyVJ+7a1Q9qNT3BTmhd+SZgc74qds4B+p9iA+aHkc9pFKD5UIiG2xlFccCTFcLkba7cwfZWzMpchE/JX5597M+EOSfzqLSF7dzqJxujf6/Pc4gXlqF1E/Cb7h7/PFSe7e956iNhEJUGjFI81iEOGjbRYk4M7/R5W26chmyisM4If9TYDYhWGMEXWOqPTtzzw2lGP6zgE7R1Dp0qUrJlG1IoFFkSSp4P/lXYJ3MvkwlnE0UjA1RJXQmlBbT9ufcmszppUKebCEfGAAmsUzPrBfxiJeHGM0HL6XY11MYSdqLo5gY/6PxwsuAZPMoEmaCw4tN6Oizq2xAoQKjsoXtylS9Po6rtkii6mZzPHDmdD77wbA/LvqWDQfazD+P7MYl/nb8YoDV8VADGsWNbRyQFnzQ/6nwlJtdlgzny6UMRYOp/+/PcO2YyVcZc2f2koJyKZzLOWXzCtJ/Cnwq0U0l30TJ+UJLHcCMJnD7D0UKWHVEJ4HiaRkIzP4AnPO3xtShmrGWebdjYPMLOmYw3dkbvqYU4ILAkyn47nfD2GjLfHyr6tWA6P98rtUejaNDFq8xf+rBNraY54qFPy0/9TbzwUiY+7WzHh01wpbGSTsGkCwutmMnI7CPhMZ6fQMncUO0iyiMKaGwNyd2uBNJ6LNX1oojDkFtiwrtYGjnwGKGro/fMJWF6qyCAhluMs7NVDORHTCBjU0bDXz/xWSIoPSv92RBNCu7lrBoijLl8tFUDsODhmm7B6xBuwygkAtD4FwXZjyIaUawCBLNXekYAhJMTLjwEN9bbCPKB7VQDGpCXfvZcmFrJZGzC9HQEzO3T2SbRNvNE45Ih+vzING3JL6Y1+SY+wZJRgNmkv0Hrd7YARiweVBTIpqJ2O7pDZQeWezrHE8C6pnLDVxcsWv7qd+JnFvac5VrR3YGASEdaCvLp2RB56ICp0GAu9KzC6pneDOiWS/j5iFw3P Sd0n61Ll u8JL8FSeYl3b6WGfJlSdjlR/LdOAGDdY6AH+eDAfuP5ts85jcGjDAz1wytr9/FqKr9Gjm9JVgRRPR9n2Xyt7FUCsWj7PC9PxtmB2LcjWV/0jgUsPsBulgF+r/pfabAtnMcnh8mTpfpAA0y3Xxc713d4ZtO41s3jJLHG9wGt9pvEemNRipkgmoX1+8xAq9b+/lRY6aln2sdLZHq0OuSb0WImTuIqD2QasUEqlrN6FgF38WgZ+CeYfakAYrbGcYk9aObV4q/ulBibVjrcVtSyAdSuJ53J1FJ/wQ8G9QzcLCltL0nJRywcnS0fq50gQlV3ri2R272comAmF6glqu0iHpe+U4z4NBEOmdA+NUHn+1GOH3CGtGNwLGgTg7Cj4pBupM8iz8KL+olxJXyMMQcQdpWT3sVI1d8vA5f8HOMzMe+VmcqUXcAy/PnQScUYeP1+l280q83wQPG+9dxVKp3m1O0ydhKQro5e8WbiXuGN79jBMPiM6CvRNNG83yrJBYPQb+ksUp2GS5V+/eLCK0oruGFq+7wmgxFLDe6vsB4XYWmeNdyDI2FUXhsESRozvIeJMCYb2wWzWTjowGz7q8fGMGfNLzcT5G1SNnQgoTvhcjD3CxpwEmr+pHl57PFFlmkVnGxpfbJxLS3FKr1TbAWitayB9voyRoUdFCyNCupEC7q6gYZ4fj9mQn92TkgUQfGtkTFBUjOU//q9sWyENMG5Ji9/jcbA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Memory reclaim is a sleepable context. Allow sleeping when flushing memcg stats to avoid unnecessarily performing a lot of work without sleeping. This can slow down reclaim code if flushing stats is taking too long, but there is already multiple cond_resched()'s in reclaim code. Signed-off-by: Yosry Ahmed Acked-by: Shakeel Butt Acked-by: Johannes Weiner Acked-by: Michal Hocko --- mm/vmscan.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index a9511ccb936f..9c1c5e8b24b8 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2845,7 +2845,7 @@ static void prepare_scan_count(pg_data_t *pgdat, struct scan_control *sc) * Flush the memory cgroup stats, so that we read accurate per-memcg * lruvec stats for heuristics. */ - mem_cgroup_flush_stats_atomic(); + mem_cgroup_flush_stats(); /* * Determine the scan balance between anon and file LRUs. From patchwork Tue Mar 28 22:16:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13191622 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C110DC6FD18 for ; Tue, 28 Mar 2023 22:17:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 57BD26B0082; Tue, 28 Mar 2023 18:17:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5085C6B0083; Tue, 28 Mar 2023 18:17:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2B9E56B0085; Tue, 28 Mar 2023 18:17:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 16D726B0082 for ; Tue, 28 Mar 2023 18:17:10 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id EB5F2A039D for ; Tue, 28 Mar 2023 22:17:09 +0000 (UTC) X-FDA: 80619718578.20.2936FB5 Received: from mail-pf1-f201.google.com (mail-pf1-f201.google.com [209.85.210.201]) by imf21.hostedemail.com (Postfix) with ESMTP id 2B8D11C0004 for ; Tue, 28 Mar 2023 22:17:07 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Jh0ANfJd; spf=pass (imf21.hostedemail.com: domain of 3YmcjZAoKCOUfVZYfHOTLKNVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--yosryahmed.bounces.google.com designates 209.85.210.201 as permitted sender) smtp.mailfrom=3YmcjZAoKCOUfVZYfHOTLKNVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680041828; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3EqK7sQZOfVTLBQ7dVPZO4hRmfVfAzm1laH3BYvfeCs=; b=5QtNHSq+7c2VQVpPQwamFQ6WwsO+uYFA1/U4WYLHv8j/nwmvYoopE9yzZnUU+QAeVjDQbi /F2obVbYm+70AciCQcxNInqNBpUpzkv7fbC8RXiXLJwoYhFIjgtRGbRXNg8KDcvlYT4U9K UK+FdMXVBoiYCIdDB6KDw39RQpB9E2o= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Jh0ANfJd; spf=pass (imf21.hostedemail.com: domain of 3YmcjZAoKCOUfVZYfHOTLKNVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--yosryahmed.bounces.google.com designates 209.85.210.201 as permitted sender) smtp.mailfrom=3YmcjZAoKCOUfVZYfHOTLKNVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680041828; a=rsa-sha256; cv=none; b=NCO7ZCpL+jexi5k9/9EtHtmfPFQD2LWHcKKgKWAQnPBdd2udJUuxt+YnO44Kd5IGHkPgkt BnJTJp7Ih741A114NNJAF8Nqu+cEsjCm+X4cgMHRTuoCwUtrzBb+DFGxIRrFbuwlunUwm7 WKz2wZIhaq/G+iLYZB/zoEiySzCuu64= Received: by mail-pf1-f201.google.com with SMTP id a6-20020aa795a6000000b006262c174d64so6315027pfk.7 for ; Tue, 28 Mar 2023 15:17:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1680041827; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=3EqK7sQZOfVTLBQ7dVPZO4hRmfVfAzm1laH3BYvfeCs=; b=Jh0ANfJdLtvcj14Ycx8o0NJ4r9qhZa68Pa0uc2IX0QIjCZKetrfT/Nq/ZDVr5AllWU XClT4UcNh1sPOxMcsh6sWOQo9/2lETExd/HKLVBB2M4wtgPvzRmNcVBtdBrQWB7ALdhZ R5a35ECD7K8gif2eOzvhZ1FDX+1AdzcBxQSZ/AU6EUVqL3HECb2nu5rbm6oZEkaJmqE+ JiSGi5TQPR/5oszlj+b2yUkvaRVAqFo8Ka4HGQ6Ui7wSkXSWPGkTPTug5dc09uPeC/es /a05azr2/gL04QjqP7TTdBynMB9W71leZpwkNhiYtT2r27tRHGjbrwcjlQFLDRhwcOoU qhHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680041827; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=3EqK7sQZOfVTLBQ7dVPZO4hRmfVfAzm1laH3BYvfeCs=; b=x2q8790ATHb4TBO1b9VpDqtlE/jFlqRzzvCNQX5hLg7/T0PxSiyp4PO5iXjUzA0WdW 4zQ4105MQt+ZIoGbJ2QYJuvoqKXlLI5tSnZv2QPIFDepIfnHngggFLEwcBR1CUEqirIG /EWVIDRyAn/W8uDZEDUHsXIlfwZmwK5RsOy/IEqfvyVr4tn8qSj1ma1CtOYfPTGq1VEz VF+EzHRZXOb4HLe0fWDCaEOG60NtsS42m0l7tTUNe0ULkbueIzHNQtrKxZeQGe+jUyD4 7Cf5SUGshujA2WuI77FgqtKbjd6tESrBXbfNelR7Yw6uLuJHDHcwrLuG9/6Hf6Q5Lpv6 of/A== X-Gm-Message-State: AAQBX9c3wXhHRSwhdyVl7UAyc2dSFo4WHXR4CDQ5z8GBICulrt4VUS/S JqbRy7OjBIP6Vuu11LGV92LddIheOKp9sXgp X-Google-Smtp-Source: AKy350b8EHObFwTAz134ielB7Lfw4iTDwRchFeXjbgsPVENWxNL5SgSGxt3wM+3XFB7OgKxVRznr5DrlZ29gVq8P X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a17:90a:fb57:b0:23d:30c2:c5b7 with SMTP id iq23-20020a17090afb5700b0023d30c2c5b7mr60436pjb.3.1680041826556; Tue, 28 Mar 2023 15:17:06 -0700 (PDT) Date: Tue, 28 Mar 2023 22:16:44 +0000 In-Reply-To: <20230328221644.803272-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230328221644.803272-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230328221644.803272-10-yosryahmed@google.com> Subject: [PATCH v2 9/9] memcg: do not modify rstat tree for zero updates From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , " =?utf-8?q?Michal_Koutn=C3=BD?= " Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 2B8D11C0004 X-Stat-Signature: xkfp74dkqxdqtc6dufz97djqq9zgu365 X-Rspam-User: X-HE-Tag: 1680041827-380623 X-HE-Meta: U2FsdGVkX19UbVRlDZd7M2hPnoWU1MwFJZT/ufPTJhQDuQQuFbdzb9SK9czmaAFeBejvSgbo/kJexD2LEJB+ZXYCMDIwHrK0NC4uYXbCaPJa2W3ec3Hrn/IawcM2y69MbRWf0a99HwYqeCwO6SAo1xEn1Dh7d4jP+pIBpyIUsgXkgHKOmHOkdcJ67EVmpEtY1j/M+wYHuQolX0WGsyFsrxK7yjHAtWbTmjfsp6xHbhOhOKu1543ME3f3XD6G3zhKZ4C5XmZKKGKMBtt9GG1s2xlGwjV+B5HKzLJkjeq5b/87wHuQg3e+M8aWftUkFQfcGx3OXIzTt/vYm2m0h0yAk6M2rgFC8R6JWtnrTmjIwMXLPMsXdHwnPjb2rECvKUXSh4WcAutgNQleoQJBPOsufKZIFcsL7NxFO+v3mxRAFDc4RgS9ME6TG+dAF4BS3ThIp/BEKWPSZdvue7oTdEggEPIJF+ykaoSDeXG1lNlIBTdFCfCXTc6Yd+t5H8ggzFt2L8uITtdoH4UlL3QdC3Ftb3DAryZwqAdVS9xF5e06fcEtSEFbSE/YA2W36VrZYStGQXLIyfBcOv4K9kgEOgfCy4WGrFkbVeIW70woDkTrb31JLKQdqu+gYGcPrFwEHqmxpQj0WjeTlKV2Qms5xfYOgsBxt7FD7c7vQMyHo+mcNUPRt+HJt7OkBxbEhOWmXpjr2W2Wq2EoUY/0AYuMNuUzKN7/SxI6jmg0vUpfK7gqGR5AeBWgRDq6K48BVvf7h/sUTmXZVbXSWZTWKrE9eEwvjW7JbRiS7Q7i5F4JX9KUuEZybrBRk2dKTwh7I98VySVWhkKRZ+3GZu8FwZ1rbEePcStou9whlnehmzX7cmlCNq4+StMB4R+OSqDDgn8ijJcmq96eLv0EreSCNMDfLzCXlA0dLq9qgkp6PtmLmt9cGtvMTc67KCkhmtILQ2mg4q3Is4s9NiiG5QwwdR2PmBe GqknPwc6 oFxTZZ4czTuk78oHhyVEX0slm8a0/FMrbLKPQiMPFgLdI05f8VaQMlzxc/+bkCYP07WZlyHqMXZFOhwYuxKKc4RpnZOzEWGN/w3LHv43eO47ICbkctoJr9mllrxJ/q7gbBsuRAPSkLbHjtMCGDYje9rv7SvxkB/hWyrEnHQ5rZe7WxR9ScnBQH3Px7AbZurvJyucSfJjA6tiZ9kGF2Lwp/lZWFYO3OID7+e82KHOb3L4RRS9iX6t2fCdf8i/rmGgBoyZX1NIoMxtNkYim3GUvIdM2pNCex4rsglaPzdqMrG+EKWjDK80HMBy2TvcE0JdYA217TbZW8v3XIxol0pv/wICelqA/iBPmWT279eprhbFKKroJ+fbyFRdpPgYVFzjUL3EovQgSgKKBk5vQYCsfRKmxx9zTRSfJfsLk6t0J/V6TTjjuX8a+P7zq+FMHWmkP+aExNy1JuNn5hbhPv/NPFo8edA2qm4BRuBW3zDytVK2P1CPuR86Iwb0QtwFo5X0aJbfgbJfBSfQOGXnjHvU+v0Z1X9reL5YKgxXmlQGL10Z8RQasqJCPdJ3y5Om31YUhJdVeVfHyIReVwhLtCAuTMlh3t+CkaM545/6V1Kge4TjOLIf3wpGkekXAYvk7YWq0q8eMAUNZ5K9UlyDYeOeaD73LqStrt7UDiBUarsVrxA/RRVKHJUegzhuF6i2LTfU2xEsBCHb7jNgm64/qOuvQV14/rA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In some situations, we may end up calling memcg_rstat_updated() with a value of 0, which means the stat was not actually updated. An example is if we fail to reclaim any pages in shrink_folio_list(). Do not add the cgroup to the rstat updated tree in this case, to avoid unnecessarily flushing it. Signed-off-by: Yosry Ahmed Acked-by: Shakeel Butt Acked-by: Johannes Weiner Acked-by: Michal Hocko --- mm/memcontrol.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 361c0bbf7283..a63ee2efa780 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -618,6 +618,9 @@ static inline void memcg_rstat_updated(struct mem_cgroup *memcg, int val) { unsigned int x; + if (!val) + return; + cgroup_rstat_updated(memcg->css.cgroup, smp_processor_id()); x = __this_cpu_add_return(stats_updates, abs(val));