From patchwork Thu Mar 23 04:00:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13184904 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9C09EC6FD1D for ; Thu, 23 Mar 2023 04:00:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 29AC76B0078; Thu, 23 Mar 2023 00:00:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 222D36B007B; Thu, 23 Mar 2023 00:00:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0C4D26B007D; Thu, 23 Mar 2023 00:00:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 004386B0078 for ; Thu, 23 Mar 2023 00:00:46 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id C8B71AAF23 for ; Thu, 23 Mar 2023 04:00:46 +0000 (UTC) X-FDA: 80598811692.15.0F0028B Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) by imf13.hostedemail.com (Postfix) with ESMTP id D2C6620005 for ; Thu, 23 Mar 2023 04:00:44 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=q0IiHI38; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf13.hostedemail.com: domain of 3684bZAoKCCAUKONU6DIA9CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--yosryahmed.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3684bZAoKCCAUKONU6DIA9CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--yosryahmed.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679544044; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zlRHMwQCwPxTz7URsIPFYEiZ1y/NkKG6JVNDPEffcgo=; b=QHZQwEPWOUe+QSWV0elJW1eWOKZQuj++jPL2hjHgC8U0T8VVR3yqnTcs4lsDuaxGRMGbwk qL4RJ6cA2paWQ+/T4VzaXhf0gD1w10uGwo45EcvrQR7ZiCTZAT8kXO6FZSUdteEQclxzdQ 7Bp/dskwoatN38qzAqPv1BzMzmeB2I8= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=q0IiHI38; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf13.hostedemail.com: domain of 3684bZAoKCCAUKONU6DIA9CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--yosryahmed.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3684bZAoKCCAUKONU6DIA9CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--yosryahmed.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679544044; a=rsa-sha256; cv=none; b=EZ+XWslbcGdPBWuQ1Fo0DfxPkAwv60xae2upNCkuQ/wzX9+T1jdN6ttSA+7upvBF+wiCcW NimeY4VkaEV0cdF9AeSfbVDUKQkEdjY+APHcw9mPVSMkAEGsQ+qEtKe9ee4VYwl0ti8G64 1lFg+VvcdsyRb4k9aS0bzKtjau3lBuM= Received: by mail-pl1-f202.google.com with SMTP id a9-20020a170902b58900b0019e2eafafddso11623758pls.7 for ; Wed, 22 Mar 2023 21:00:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1679544043; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=zlRHMwQCwPxTz7URsIPFYEiZ1y/NkKG6JVNDPEffcgo=; b=q0IiHI38SIToezuWJ5YCNMq/43EaxUo3i4AMEPHcC9bZCfWe3qfaYbxabY9EbcCoqm LUESQ/1v1D2aalE9SO+twkVNP5hnW+UNiSPbsUMxGnfaQG+8mDUTyeXBv18fXpGW3SWD GCVDR2eWTWBw72p7FFHYiZQYGqUWBUdEyjEquw53iNIuEWGhIQhSBNfbqBw3sfwQmtJy zq0COAvO1AYz/HqEKQJev5lhSkkjfsk/GYKhQDhBPMJgp6hmFHeDaNzuC/4om1icb3Uk IS34ZPkCnh/Dxdz7tz/xvLp+bFG8mdt7D5IhxCaGQj3LQws3RSkaD2yIkaXnXsqGxGWA GdAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679544043; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=zlRHMwQCwPxTz7URsIPFYEiZ1y/NkKG6JVNDPEffcgo=; b=BwjjgHeeq+NrEGNpwW/GFY+9R6k+Yq+xXogBirjMLI8TGdkjPk55720j0u2UqyJUXw r8h8c5UNe6cwMNZ1n8gZePI+uT7Yp7pJioYvlw8IUcGLKDcZzgAPumD2dJqJ4KfjO1+s x7EhJUtEZyEz2ygqz3lXVAYt7W2fUyVzXJncG5vtuiVbsNTzJuf1fcyXeckfrZu3bKnD rQ3sWTdZb3k2R9FzYFYk7UNX33mygOHs6g0yozj0PO24Uq87YhaBD5CaagyrqRYj9VpW 3uEoXa8+dmdb3CD0s1k3kOsKauhHRsAeuobJWR+ZZgDNRe9oUNnMwFeYfXzA/jnZtKyk 3ktQ== X-Gm-Message-State: AO0yUKW+5rNLetdoYOYRZyBOCoIkiP7CdItew3BJA9HF7ntHLVMG2xFx +KRH4RUm+5ESepeymAU5w7yehzgs2aQrIMyF X-Google-Smtp-Source: AK7set97jrlAfVvPpHOrM2Vr0RCmWVnsZU6sMMMwHyygC2dGwvgADEuqjxobUXV27YUvQlS1Att+r/ZLiyOif2dS X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a17:902:e745:b0:1a0:4aa3:3a9a with SMTP id p5-20020a170902e74500b001a04aa33a9amr1983132plf.2.1679544043324; Wed, 22 Mar 2023 21:00:43 -0700 (PDT) Date: Thu, 23 Mar 2023 04:00:31 +0000 In-Reply-To: <20230323040037.2389095-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230323040037.2389095-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.rc1.284.g88254d51c5-goog Message-ID: <20230323040037.2389095-2-yosryahmed@google.com> Subject: [RFC PATCH 1/7] cgroup: rstat: only disable interrupts for the percpu lock From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed X-Rspamd-Queue-Id: D2C6620005 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: 9uy9bo4ycfbfjcgenagq5c7nt58ap7as X-HE-Tag: 1679544044-637766 X-HE-Meta: U2FsdGVkX19X/yKQFvpNJ0OjuoPNf78hMg4Gt//ru24nP9zJAeCdVo1hKvROWAb05h7jPjhOA8HDGvZ19yfGWZx+mwyqrXtk3EsKO04MM2gs8wgh25z3omcuPDWtCChuYcMD061n2cIr8AyK+DUZ6FiWerBd98+EwiBZlMLR+5hBO89z2IkuWnG0VXRZHq3xv0BH/RwPkuOyHemrNsb+AzB4FlXHUoNYV0QLgiy25DVSgVq6A27jQFNR3+8LgMzzHY5SxrN4sz5jTtSKgOZQZ1iClY6yJdYtmutbRautyWbmzzP/giuL3fg82tP2R1IL5ItoJadAPWCAa/lC947NVtCgI1X3UMB/tjvaSnIcgQVL2kRv7rDbiqenPYc+dt+a4vCxgKi+4kXh8dkTj4OXRFy8WalOsMnET47QjQVnzO+45Au1ikc5iuxRHQ0n73Wb2opxOnVqxvERu6/fqrYSm4Z20r2fRkkawpH/lsP/z+XOdTqzRnCnnPjrSdPZTCdfRP9K/k8qfoUPP+JgKkNF0IVOLDpXCSDbuSFYPYSXqPfa8dKJWacZfsrJXJJmEcHMBDPNd8VIHHvyn6BN/qZxp0cw2mPx8/a+mNz0AYzyZxDTaqy1tRF/2wDD+u2T4cxrtPN3ZI56rN/ZjxCJfzjnc4myHkpShAtZf0VucRYqeCPhY+9TL0fg6Qwv05Vj6DrpqaxZaKMbVfDl3+FKnvVyqlLNujJLGa4+DAqeJSFNp2PwzBzXnEjJj3xtf/sMXac1wuPRfIegiGAasIzoyldKT9nnJ3liWIz8cQNCqFq+VsBei9Encsge9KeReLu+oAcigy7QK7BY3mjebuyRYKcnfdFZhfpQ0NDGgkgGbg60nFQTsNl+HT4TfLfGSk7kvgfqfO1W6QoHXeScF/Q7I6qf6KMBfCmmRXR1u+ERj7X+yvwsZ6mmrfuQGFK2wA0c1vvnC2su2PCur21Z2IFGpez QT0wZKvG ZDEu4KjHJXE5WEpO2riCHfWBs2HzoKdynjSx04YhTQfeR4tGWYmGZ/euQcpIYxn8aSk/S/pjg86JHIE7ztVnKyisjYasXfnIxKBOgwUbu78ENHchRdHmUQFbJ2d6REnNhkI7Rx3XteyfAUZ/I57KhOEoSfUxf3paF/zoRQ+mapjhSj4Wa0qq/MhMPzp3FNuGoa3yunI9xxKqaAQnqBBpcAzEercDR7kTkxKAufSGMOtjbAxkcHTqJNZy3EOE2GOmseQYHjTkz+JoBsTMU0hrrgo6VByXkPZXMJSavxiHbbjH+/yR+/0dDt92coUEyHkNRyCllH8Rd95C0GjSu8w/+rr9RfWFRtZftgBKHlStYT53WM/FvxEokwxSMU8ifakF3UQFiY1/tJCHQ2vKhSgxQNaXFWV6gU3ssX80YqQOc7R58Rw+W2shOVXRlyKYtYRUNugex8x7M6X0ruxxo1+paaDKB2iimMRq2l/5eXuelCr86rObTko0tJZ79vhadGzmU1o22NKEzrbJ9iZVl4vdN3ScV1lfJejFucanuIhrhFE0pvKXbuxdfFztQJqE4ce87Ef2fr+p+n1G0Fsm9pPq9hWX7DlkS9VHV21s+juhfURtJwWUtAXk/fS1BSBXfzdxd81mXghuoNAIfIbD2yXmD9rLyTw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently, when sleeping is not allowed during rstat flushing, we hold the global rstat lock with interrupts disabled throughout the entire flush operation. Flushing in an O(# cgroups * # cpus) operation, and having interrupts disabled throughout is dangerous. For some contexts, we may not want to sleep, but can be interrupted (e.g. while holding a spinlock or RCU read lock). As such, do not disable interrupts throughout rstat flushing, only when holding the percpu lock. This breaks down the O(# cgroups * # cpus) duration with interrupts disabled to a series of O(# cgroups) durations. Furthermore, if a cpu spinning waiting for the global rstat lock, it doesn't need to spin with interrupts disabled anymore. If the caller of rstat flushing needs interrupts to be disabled, it's up to them to decide that, and it should be fine to hold the global rstat lock with interrupts disabled. There is currently a single context that may invoke rstat flushing with interrupts disabled, the mem_cgroup_flush_stats() call in mem_cgroup_usage(), if called from mem_cgroup_threshold(). To make it safe to hold the global rstat lock with interrupts enabled, make sure we only flush from in_task() contexts. The side effect of that we read stale stats in interrupt context, but this should be okay, as flushing in interrupt context is dangerous anyway as it is an expensive operation, so reading stale stats is safer. Signed-off-by: Yosry Ahmed --- kernel/cgroup/rstat.c | 40 +++++++++++++++++++++++++++++----------- 1 file changed, 29 insertions(+), 11 deletions(-) diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c index 831f1f472bb8..af11e28bd055 100644 --- a/kernel/cgroup/rstat.c +++ b/kernel/cgroup/rstat.c @@ -7,6 +7,7 @@ #include #include +/* This lock should only be held from task context */ static DEFINE_SPINLOCK(cgroup_rstat_lock); static DEFINE_PER_CPU(raw_spinlock_t, cgroup_rstat_cpu_lock); @@ -210,14 +211,24 @@ static void cgroup_rstat_flush_locked(struct cgroup *cgrp, bool may_sleep) /* if @may_sleep, play nice and yield if necessary */ if (may_sleep && (need_resched() || spin_needbreak(&cgroup_rstat_lock))) { - spin_unlock_irq(&cgroup_rstat_lock); + spin_unlock(&cgroup_rstat_lock); if (!cond_resched()) cpu_relax(); - spin_lock_irq(&cgroup_rstat_lock); + spin_lock(&cgroup_rstat_lock); } } } +static bool should_skip_flush(void) +{ + /* + * We acquire cgroup_rstat_lock without disabling interrupts, so we + * should not try to acquire from interrupt contexts to avoid deadlocks. + * It can be expensive to flush stats in interrupt context anyway. + */ + return !in_task(); +} + /** * cgroup_rstat_flush - flush stats in @cgrp's subtree * @cgrp: target cgroup @@ -229,15 +240,18 @@ static void cgroup_rstat_flush_locked(struct cgroup *cgrp, bool may_sleep) * This also gets all cgroups in the subtree including @cgrp off the * ->updated_children lists. * - * This function may block. + * This function is safe to call from contexts that disable interrupts, but + * @may_sleep must be set to false, otherwise the function may block. */ __bpf_kfunc void cgroup_rstat_flush(struct cgroup *cgrp) { - might_sleep(); + if (should_skip_flush()) + return; - spin_lock_irq(&cgroup_rstat_lock); + might_sleep(); + spin_lock(&cgroup_rstat_lock); cgroup_rstat_flush_locked(cgrp, true); - spin_unlock_irq(&cgroup_rstat_lock); + spin_unlock(&cgroup_rstat_lock); } /** @@ -248,11 +262,12 @@ __bpf_kfunc void cgroup_rstat_flush(struct cgroup *cgrp) */ void cgroup_rstat_flush_irqsafe(struct cgroup *cgrp) { - unsigned long flags; + if (should_skip_flush()) + return; - spin_lock_irqsave(&cgroup_rstat_lock, flags); + spin_lock(&cgroup_rstat_lock); cgroup_rstat_flush_locked(cgrp, false); - spin_unlock_irqrestore(&cgroup_rstat_lock, flags); + spin_unlock(&cgroup_rstat_lock); } /** @@ -267,8 +282,11 @@ void cgroup_rstat_flush_irqsafe(struct cgroup *cgrp) void cgroup_rstat_flush_hold(struct cgroup *cgrp) __acquires(&cgroup_rstat_lock) { + if (should_skip_flush()) + return; + might_sleep(); - spin_lock_irq(&cgroup_rstat_lock); + spin_lock(&cgroup_rstat_lock); cgroup_rstat_flush_locked(cgrp, true); } @@ -278,7 +296,7 @@ void cgroup_rstat_flush_hold(struct cgroup *cgrp) void cgroup_rstat_flush_release(void) __releases(&cgroup_rstat_lock) { - spin_unlock_irq(&cgroup_rstat_lock); + spin_unlock(&cgroup_rstat_lock); } int cgroup_rstat_init(struct cgroup *cgrp) From patchwork Thu Mar 23 04:00:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13184905 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 689CFC6FD1C for ; Thu, 23 Mar 2023 04:00:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CD6F26B007D; Thu, 23 Mar 2023 00:00:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C0E366B007E; Thu, 23 Mar 2023 00:00:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AAE336B0080; Thu, 23 Mar 2023 00:00:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 9A9E86B007D for ; Thu, 23 Mar 2023 00:00:48 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 65D48A032E for ; Thu, 23 Mar 2023 04:00:48 +0000 (UTC) X-FDA: 80598811776.06.150ABDA Received: from mail-pf1-f202.google.com (mail-pf1-f202.google.com [209.85.210.202]) by imf17.hostedemail.com (Postfix) with ESMTP id 96B6340009 for ; Thu, 23 Mar 2023 04:00:46 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=nvFwhwj7; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf17.hostedemail.com: domain of 37c4bZAoKCCIWMQPW8FKCBEMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--yosryahmed.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=37c4bZAoKCCIWMQPW8FKCBEMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--yosryahmed.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679544046; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0V6KepxSWjC0U8YUMhyf5sEsW80RflfKR4tDJnFyjUM=; b=U9MXZmRcj0w6Y8llg554o8MLapiOO0c1JFAOtX/pYDJHh4/oNeCgpYDacDx+OOnmhOOa/g y21a+cG4WXRtf/yHmE96wGst3t49NgakNkFgJ/aPRUWKXKSwS3Iu2+j7w0ZFoMBXjXsnVD 1wSgJqvS9JRxqLC18VW1kYyqMLtXBoc= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=nvFwhwj7; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf17.hostedemail.com: domain of 37c4bZAoKCCIWMQPW8FKCBEMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--yosryahmed.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=37c4bZAoKCCIWMQPW8FKCBEMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--yosryahmed.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679544046; a=rsa-sha256; cv=none; b=MB7xGOf6oF5qrMmlWXzr9tfrY3P4JL8KM8VjCOtcE2eQH25Ru0eWKS/O+l//eE0S9hKVxJ 5iY71NjxBCgGapAoXLn8IfGtaxLx9lpsOvdMHabhTuKHWJtocQiCbn0lYKRLJu3Xr4G0HS XJnGtpnTilPPgnGIjz+iRdMzKcICTR8= Received: by mail-pf1-f202.google.com with SMTP id i15-20020aa78d8f000000b005edc307b103so9972307pfr.17 for ; Wed, 22 Mar 2023 21:00:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1679544045; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=0V6KepxSWjC0U8YUMhyf5sEsW80RflfKR4tDJnFyjUM=; b=nvFwhwj7qGyMPQaTPrzYFw+T0XNSe66OQzM5b4pjwLTwVBRy9Yg/AbwpYFnRMIM5gG L465fbRhjctsBw/iYduTrtP7LUUTB2pfnwQVaagEQM0GyAtfd40drA654+PpJ27hrtIB BUWcp/c32KQ98SFLCZ9FSHYNM6vaide1p2wRQYwdJvG/bU3/l2g1YAjLINeXoxbcrV2W 3/G5Y/6fhAJDfUsgxXfVWzP78YqVFCMwh/+/7IKP9AVvA+7qfhsyQkLxneMASA+CreO0 VfuwrktKMl3lyTDdGpOXeKO210ebUx1dz4XnfSvMg+6i1Ne/G/0TP1oqkrL/txGwmyNx VJtg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679544045; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=0V6KepxSWjC0U8YUMhyf5sEsW80RflfKR4tDJnFyjUM=; b=nIWFSlpiLrtrAEQowd7WqwAv4gJaNPovz9hHers5zvy4UIxiFmscHw7r3w8oQ0IKou uhFgjYCYSfEAs7x3yhdfrjeiGTNnS6wNO0C/bOEBsslU8AJdyChCzJKjyiHoAEkvQ9nU A5W/vSEjRIevmTdsnKauVc6qofeg5mO723reiFiSN7Sq4nT0W5Qr8PHOZR1kRiBFA5mL Ibci4s+KCawaE7ZM35EI5d4YoKtH7Sr5UPwmoyk6EuLashSgzsrDwhsRdT5BWOCNfK8Y iWFlMrpIwSRViCcqCm3/0lKy5w+d/JcDtFpfbTFoTfMAeIOOhmcXZbhsf9badeTwkeYE uZzQ== X-Gm-Message-State: AO0yUKUBq1YS3B656VGLpi4aYL9sXsbY1t+/zNK5L0OsueyFXUjOuZbh H3NnFAwpyUk23HOwbppzu08dphFaAlI+vkrQ X-Google-Smtp-Source: AK7set+NjW/4AXEB5d0IG7XiVNOSp7doL59qiTn9uvLqiFnoG/BqWGCoWZHV7SR3pQA/2uxuP1t6d8Miz+ziUaX1 X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a05:6a00:851:b0:628:30d:2d2f with SMTP id q17-20020a056a00085100b00628030d2d2fmr2879444pfk.5.1679544045305; Wed, 22 Mar 2023 21:00:45 -0700 (PDT) Date: Thu, 23 Mar 2023 04:00:32 +0000 In-Reply-To: <20230323040037.2389095-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230323040037.2389095-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.rc1.284.g88254d51c5-goog Message-ID: <20230323040037.2389095-3-yosryahmed@google.com> Subject: [RFC PATCH 2/7] memcg: do not disable interrupts when holding stats_flush_lock From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed X-Rspamd-Queue-Id: 96B6340009 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: ch79rfttckgap5gdpczhqti75fk1uc1w X-HE-Tag: 1679544046-993972 X-HE-Meta: U2FsdGVkX18Rf/+3OQvoDa+OjiDOd8iI0cEv5nsfj9hId4aiEInV1MasphSj8eEm/W4TL50qQ9orQ9gaSFVpooL2atv/ViQWmVJghw3Kru56n+qMCHpwsCyt7+8Ho3bxTXERV4GxSMhuMTMPZLWqZh5txKqEMt/FUKDptrhvAJnP4LZwbWPyfHRde8LxgfsNF4zoc0UeHzR2vdkPVY6O8XGXXki66Ryz6Z+9YA2Ye/5bUXOb9wlcBDhFh93/LgAumAClQmqPyoL4zg4eLYHK3mIvHZ6vWBFd2jP6lxy+pJq3mlcNjbwWVFcCDpsgZ5lIGMR+xG02ZYWxGdzfc0SNw8dEHG9iddt4hM4QUtTBo39Fqi+xBbAtijktmLfV1+pRkV31J60ovMh+FLetXR3KpPPDXVIYRE+VAf1hknH6MXqwEgU1PzyL/Tm9269gYs1X2LDdl9BVs2TO6+rRM16SILgd+uh4dHMqGCvHNCM2YIlgjTK8KPQPAVqeIpLYDXosPL1DoTVeN/LskxibES1yN62a3U1PDkFLv9f+d1It0acdzQltwUiSB15o8mXxMFRjtTYZ2Y2s9RJA/eFMwxceiA6rwwha9RxvFamRGWQgoh2/u4+nzooGga+kaTlXIyzODO/KkO5Os50Eb14YT0SI205QdgSrHMWlKP7hI4/qYiqYXlqLMw+a+DuYvjCuiq6bB2G0At05kxGEJAB+M9M+gG5pIwLvY+HJqbwxHLoeEe+DRWUd31bU3QAfFN3YCTNt9anXJ3ThNTm3DI7aF//kj3o3sUXjydPKdTnBHBxib4QPoJVXsJBixCtQ6qmD0Vng0ekEg4yX+cq92+wmS3ZnoVjoRYmhv8SeQ1KrgYc2o99V3eLcfo+I1h/ZPvaVo6nnAxrd+TliyFm5HfQCbx6qU2gyckNzmlFJhV7nlblhpaGoRVN1YNImbCLOlUpenZl6nyjyMdN52e1iZ1jLlUj Xi6gPNuv 7fl2MBc4IwX8zjAgYT5pgfJsvXbF5CzSmJN7/mKg7QCfyxUyR0re8bIZCsylZlJkOYeq7SQo0NIkP6caVPPAdo+8+aJ/qwSggN00BywyWyMwyibVOhVUwLQG7IWetUSVrQbhBpM0+QOaJVrypUxvhKE2AcdbyR6wS2TdqKKvEb2SWEWKFy02vzlVAy/E49/i4O6GWtusXo0hq3ikyaPJtlhbd5j1bjx/rQTU+DF2bFEnQ/ym931vSEvdkq122BjfBB3nlt85Uer/jj7YAcyrfArcpwZqVapR2gW0vYiCn77kqnuf/XxtGfO9uQA2iAjxjTqeu4YbtMe39lCQHJlTL8CREgIhB27lgFtr8DxYXVkJi2lxlI+8TTbRKzIDEzGSRJ/910MuGAunzJw7Q4UYpkA9QKZ0e2cuDxkPQzNbP9KbIsm6tctxjpmRdWy8e58GZJPurQF+zkTRMa0ruXaHbYj0rPDoR85xcDtDM5z8FnKMUUOvbeClyz2f3IoqEN+ZVzCGTrWYKrs3vSMNeJL9oFtVFn0pDyVbw4oRook0PpHQ/17ev+q773ZkzUKBwMxgkoCwvqUfsj5WwAI4K5/GQX8RAMiet+MhNINO8qaI1GTgNjGDWtAIYOFp6m3uoLrwnp/kDx98RLlobFJBqFhKfrMWObupFuyYMY81+2c5vV5DxhVLy3IlR24whT54BEILDvS913o9ozaOrrNQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The rstat flushing code was modified so that we do not disable interrupts when we hold the global rstat lock. Do the same for stats_flush_lock on the memcg side to avoid unnecessarily disabling interrupts throughout flushing. Since the code exclusively uses trylock to acquire this lock, it should be fine to hold from interrupt contexts or normal contexts without disabling interrupts as a deadlock cannot occur. For interrupt contexts we will return immediately without flushing anyway. Signed-off-by: Yosry Ahmed --- mm/memcontrol.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 5abffe6f8389..e0e92b38fa51 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -636,15 +636,17 @@ static inline void memcg_rstat_updated(struct mem_cgroup *memcg, int val) static void __mem_cgroup_flush_stats(void) { - unsigned long flag; - - if (!spin_trylock_irqsave(&stats_flush_lock, flag)) + /* + * This lock can be acquired from interrupt context, but we only acquire + * using trylock so it should be fine as we cannot cause a deadlock. + */ + if (!spin_trylock(&stats_flush_lock)) return; flush_next_time = jiffies_64 + 2*FLUSH_TIME; cgroup_rstat_flush_irqsafe(root_mem_cgroup->css.cgroup); atomic_set(&stats_flush_threshold, 0); - spin_unlock_irqrestore(&stats_flush_lock, flag); + spin_unlock(&stats_flush_lock); } void mem_cgroup_flush_stats(void) From patchwork Thu Mar 23 04:00:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13184906 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 41524C76196 for ; Thu, 23 Mar 2023 04:00:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 950A96B0080; Thu, 23 Mar 2023 00:00:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 900E06B0081; Thu, 23 Mar 2023 00:00:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7C93C6B0082; Thu, 23 Mar 2023 00:00:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 643E86B0080 for ; Thu, 23 Mar 2023 00:00:50 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 3941AAAEDA for ; Thu, 23 Mar 2023 04:00:50 +0000 (UTC) X-FDA: 80598811860.24.1D8763B Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf06.hostedemail.com (Postfix) with ESMTP id 1B3B218001A for ; Thu, 23 Mar 2023 04:00:47 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=GwAMe3Sd; spf=pass (imf06.hostedemail.com: domain of 3784bZAoKCCQYOSRYAHMEDGOOGLE.COMLINUX-MMKVACK.ORG@flex--yosryahmed.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3784bZAoKCCQYOSRYAHMEDGOOGLE.COMLINUX-MMKVACK.ORG@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679544048; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=736+92Vy7w+whzoBxwgGPc/WG8vZYtmz8qFOhZmUk9w=; b=olpwhO8QHsVWAiXgK9oNc0tQ4KOsaqX+PEkPtCCuz9ScGqm9uDF8RlI01vXHtzWvIUvQDY nJQUiRLWGn01WsNHtKVoYYsduI+DhE/kbiGE9iFmPc2Mwlly5Mb5ocLyRRNwNBVBSNvUkl /Y1iePlHNfWaLVn8O/bJaDuVgCYtbwQ= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=GwAMe3Sd; spf=pass (imf06.hostedemail.com: domain of 3784bZAoKCCQYOSRYAHMEDGOOGLE.COMLINUX-MMKVACK.ORG@flex--yosryahmed.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3784bZAoKCCQYOSRYAHMEDGOOGLE.COMLINUX-MMKVACK.ORG@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679544048; a=rsa-sha256; cv=none; b=2gdHHq3zqEB6zRNAVAqaRKSqhYYu4TMAUrGezAFSYIpfpSdZ7sjJaATOyvLN1VturwXn26 +kqf4cPcrjMvoBrvCaPwUmKiqJDhIVnatjIIkMfurmP5FeDSD0AdUM5HeIQd4mlgYyPp2V DcXAEHZB2E9FOG4Ehb0EC5WYEGw41KA= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-5458dde029bso1695597b3.13 for ; Wed, 22 Mar 2023 21:00:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1679544047; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=736+92Vy7w+whzoBxwgGPc/WG8vZYtmz8qFOhZmUk9w=; b=GwAMe3Sdh/YfT7TszWBGAWb/Omm+6/db9kDSH3rx0U2jTcaU6cdo97INaHv86kxNQA jhXBaZOARcF2wynwqEfwzWZoINYtyHtPiv9iRsKtX2yswvpMBD6qg4pgoYL8XYK3JdrY NUg095+EPmMHIupEKGzGCL9BJH8z0FwgmxWYVXMQd4hSYkBDyIXOQyRVbTCpsq34KRPV CyG3E5BvF3wTZXEl8tQiaUrIBU2ScbP5rypiT3jSEorEK7YISHgjHIMI4g2IycEeQTfl s5/P1y3KHUJjZSE6PJ3e2OlZHSMu+2KD/QDk8kFiQ8Hi9jpOcpijxBB3B0Ib/9iMoeNk 6zZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679544047; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=736+92Vy7w+whzoBxwgGPc/WG8vZYtmz8qFOhZmUk9w=; b=faOK+1UjyjdFZ/RosVApTQcRAGE5k7c2+OUWiPssSLsCBtetY/HTYNurlDdeEX4I+U uqsC1ZxTajyQ61XjQTIfckT0bAiwxaNQ+c6TuT4iohiu1e8K6iuG91kNvKl4AiJpOMWb 2llhPFXb6F8VLVbdUJkzxWaEOhOFAFUItDj3HqOtybzTbtrpyX8mI0KjbVWMJunGcrjg rvY7nEmVhOZsM2i7XlUgEvVHy535/D1/beyhHsO8kyymuN6RADGxu38latU5fRmzab+M zjJ0rLOJ1r5qAcQfN+eO07GYLlBPQV8P206P6KfqD4ouAJrR8hxGJNZQjO5RrM1WOsV9 IfYg== X-Gm-Message-State: AAQBX9c+306vjRjSVQTCniSBK10J3ucIkRxdu+8a4XwFnLMi64y6GsgQ 9Vl1rJLsTbnmOaFp0ft41FA69vHV0iIJ6LsK X-Google-Smtp-Source: AKy350YCGOdNWpxPNSJcwpjqKe6Apct4Cbzrlv7LWeidWe3sKdVG3fkxwH2jR1od8/69JvygqRYEaeYcvlRO26C1 X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a81:ad1b:0:b0:541:69bc:8626 with SMTP id l27-20020a81ad1b000000b0054169bc8626mr1137749ywh.10.1679544047097; Wed, 22 Mar 2023 21:00:47 -0700 (PDT) Date: Thu, 23 Mar 2023 04:00:33 +0000 In-Reply-To: <20230323040037.2389095-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230323040037.2389095-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.rc1.284.g88254d51c5-goog Message-ID: <20230323040037.2389095-4-yosryahmed@google.com> Subject: [RFC PATCH 3/7] cgroup: rstat: remove cgroup_rstat_flush_irqsafe() From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 1B3B218001A X-Stat-Signature: b1ibp1t4o4cax1u79fyhgtiz6u9woxfp X-Rspam-User: X-HE-Tag: 1679544047-535888 X-HE-Meta: U2FsdGVkX18PNtb/+XqYObQoXasri4CD+Dm+YrXwaf+QHm8V1bd74FVbwMcl6zm/b9a20m0BI/UFhOOiiH4e0d+xBKdSo1I2kg/q1CJklLvkkKmVqhao5UuMyUPPp9IhE3dUgi7NRGzhqmcKKbFfKdB5YBvjHinEkjwusCs3psf/y+Z3j7VhZwfmngiVWDFzLxbm5hJtfiNxKrhbrGyPSK42QHvTZPcQxkXxP+WoHd+bvh3lVA1QrSFb7lJANzXvCioEfgPQpkAelI9vrVI411r4+/69yuMPjj9KJkBG2OqHTV4hjpxxQSxb55SkI92MfCKKBDVKKF+qShZApmLE1NpQHrnHD/FMlItfiumLIoXbG5AzXG66qYG9T1j0ZcJG/f+DVcFXx9KcMB8kZMiePsEmC4Y8SHnLtbe6kBy37MyDyQhCQOMSrFtbYzcMFKMMGMk5pzupNLiegcM6mqOFcOlJH8BYfH00EHbX0awLASFQs+2zmiGmiTmCd3ZWA/UMuWggIXYTO86neHDsWoa5689sTId1ItZstHfPQDH7255zZnkI1OcQdKfy/rKRtpQcbew69zV6ijCgaAocXcGDIGp5q/HKxWJXZ4JPth3H9RfICkXI2akuSSbVCOkG+AOi1GCTmPlt/NNEaV8gcRuAeF+CNrNUEg7wo3JxS7cH1b11DEyyQ/aWoopyujqRjJhE38AkZJ3tr1OqK+C149S8BUSCbqVGzig+TfBH4oiT45ulTWEnw1tVoXKn1SofU0BqtlfIhwv52PfKqRtbYbAnx+TL8SEox/rT4F1XFVxLMDgONg6rEELlB9FZbyh2CsbufJRupCy63QAMjWxVLj3Ve10l25EK3H8i8hzNrWtThTrR8gzMKpU2c4/Bvb4QHvLZ+gIx7sduVYRM7EpL1wtjpzg30KwD2+WVZojbTyRYbvT+/FBv2W79XpJwx2xiaO+15U45+MS83sz2T3Bhfyk tJmtKIfV jRvsl/vXAfAm1JMLOv4/9ZuAFcJ8kfOAeWCPj4/FX67ZQ1iaxt1OpgHyvh2mWaktCcycHcycKp/xpSBvHR9NlmgCroBTfBuRjNkl8TOUQVgxUpP7um4beJ8tswgjLAcrgXQ9mz1EQIzg9KoLOB2BKxAqYwrqOMMCylFCOCTjX/0D2rdo5Ru7c7I52o8BI76l7XmkkYu3C7fYydL4XrYvsVLDnIPYBWniP4Ah6wOYE0t+SLDZ7hpbOj7CJEdUhZpqY6Nv1/wmV8hd5ggSGgSRqCBGFyZj6vTWUG29+GJFynFGA0KCFb28NUris6HKmbYS54HPLzbL8BQUIH+t7dV5S/sF58DRSMmhPW38CmCy4BjvF7vrJxKw+ctYkOJS6rIyC0XGQde6b7Y4hps3oG3bBvTmjC0OcSZg0cEZ96EqLKNk1GbThrgmbhjmOED8UN+izjTMozkDgFxlPEJBLciRsRsQZweplfDStZTLQbOcMddjU1NsZIFGMeM56GdDp/NpCOALpdomaiXnvTx1sRkDCPMoH3nQ4hNmV7m8R1Owz84eUp5gfLEFSZuxhOuBWH310ukRryxVFeKkU0AavtYZWlorpfpHv+OpGvisvg8GlLNypGY0mORo8OQwjVziFgKXaMlxe9Ka28L4G8SyPdUYopv5JGtFI4Szas+vg9os4aDceDxp9MR926hlLqqI7XghCFeurUP+p4lxCiAE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The naming of cgroup_rstat_flush_irqsafe() can be confusing. It can read like "irqsave", which means that it disables interrupts throughout, but it actually doesn't. It is just "safe" to call from contexts with interrupts disabled. Furthermore, this is only used today by mem_cgroup_flush_stats(), which will be changed by a later patch to optionally allow sleeping. Simplify the code and make it easier to reason about by instead passing in an explicit @may_sleep argument to cgroup_rstat_flush(), which gets passed directly to cgroup_rstat_flush_locked(). Signed-off-by: Yosry Ahmed --- block/blk-cgroup.c | 2 +- include/linux/cgroup.h | 3 +-- kernel/cgroup/cgroup.c | 4 ++-- kernel/cgroup/rstat.c | 24 +++++------------------- mm/memcontrol.c | 6 +++--- 5 files changed, 12 insertions(+), 27 deletions(-) diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c index bd50b55bdb61..3fe313ce5e6b 100644 --- a/block/blk-cgroup.c +++ b/block/blk-cgroup.c @@ -1043,7 +1043,7 @@ static int blkcg_print_stat(struct seq_file *sf, void *v) if (!seq_css(sf)->parent) blkcg_fill_root_iostats(); else - cgroup_rstat_flush(blkcg->css.cgroup); + cgroup_rstat_flush(blkcg->css.cgroup, true); rcu_read_lock(); hlist_for_each_entry_rcu(blkg, &blkcg->blkg_list, blkcg_node) { diff --git a/include/linux/cgroup.h b/include/linux/cgroup.h index 3410aecffdb4..743c345b6a3f 100644 --- a/include/linux/cgroup.h +++ b/include/linux/cgroup.h @@ -691,8 +691,7 @@ static inline void cgroup_path_from_kernfs_id(u64 id, char *buf, size_t buflen) * cgroup scalable recursive statistics. */ void cgroup_rstat_updated(struct cgroup *cgrp, int cpu); -void cgroup_rstat_flush(struct cgroup *cgrp); -void cgroup_rstat_flush_irqsafe(struct cgroup *cgrp); +void cgroup_rstat_flush(struct cgroup *cgrp, bool may_sleep); void cgroup_rstat_flush_hold(struct cgroup *cgrp); void cgroup_rstat_flush_release(void); diff --git a/kernel/cgroup/cgroup.c b/kernel/cgroup/cgroup.c index 935e8121b21e..58df0fc379f6 100644 --- a/kernel/cgroup/cgroup.c +++ b/kernel/cgroup/cgroup.c @@ -5393,7 +5393,7 @@ static void css_release_work_fn(struct work_struct *work) if (ss) { /* css release path */ if (!list_empty(&css->rstat_css_node)) { - cgroup_rstat_flush(cgrp); + cgroup_rstat_flush(cgrp, true); list_del_rcu(&css->rstat_css_node); } @@ -5406,7 +5406,7 @@ static void css_release_work_fn(struct work_struct *work) /* cgroup release path */ TRACE_CGROUP_PATH(release, cgrp); - cgroup_rstat_flush(cgrp); + cgroup_rstat_flush(cgrp, true); spin_lock_irq(&css_set_lock); for (tcgrp = cgroup_parent(cgrp); tcgrp; diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c index af11e28bd055..fe8690bb1e1c 100644 --- a/kernel/cgroup/rstat.c +++ b/kernel/cgroup/rstat.c @@ -243,30 +243,16 @@ static bool should_skip_flush(void) * This function is safe to call from contexts that disable interrupts, but * @may_sleep must be set to false, otherwise the function may block. */ -__bpf_kfunc void cgroup_rstat_flush(struct cgroup *cgrp) +__bpf_kfunc void cgroup_rstat_flush(struct cgroup *cgrp, bool may_sleep) { if (should_skip_flush()) return; - might_sleep(); - spin_lock(&cgroup_rstat_lock); - cgroup_rstat_flush_locked(cgrp, true); - spin_unlock(&cgroup_rstat_lock); -} - -/** - * cgroup_rstat_flush_irqsafe - irqsafe version of cgroup_rstat_flush() - * @cgrp: target cgroup - * - * This function can be called from any context. - */ -void cgroup_rstat_flush_irqsafe(struct cgroup *cgrp) -{ - if (should_skip_flush()) - return; + if (may_sleep) + might_sleep(); spin_lock(&cgroup_rstat_lock); - cgroup_rstat_flush_locked(cgrp, false); + cgroup_rstat_flush_locked(cgrp, may_sleep); spin_unlock(&cgroup_rstat_lock); } @@ -325,7 +311,7 @@ void cgroup_rstat_exit(struct cgroup *cgrp) { int cpu; - cgroup_rstat_flush(cgrp); + cgroup_rstat_flush(cgrp, true); /* sanity check */ for_each_possible_cpu(cpu) { diff --git a/mm/memcontrol.c b/mm/memcontrol.c index e0e92b38fa51..72cd44f88d97 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -644,7 +644,7 @@ static void __mem_cgroup_flush_stats(void) return; flush_next_time = jiffies_64 + 2*FLUSH_TIME; - cgroup_rstat_flush_irqsafe(root_mem_cgroup->css.cgroup); + cgroup_rstat_flush(root_mem_cgroup->css.cgroup, false); atomic_set(&stats_flush_threshold, 0); spin_unlock(&stats_flush_lock); } @@ -7745,7 +7745,7 @@ bool obj_cgroup_may_zswap(struct obj_cgroup *objcg) break; } - cgroup_rstat_flush(memcg->css.cgroup); + cgroup_rstat_flush(memcg->css.cgroup, true); pages = memcg_page_state(memcg, MEMCG_ZSWAP_B) / PAGE_SIZE; if (pages < max) continue; @@ -7810,7 +7810,7 @@ void obj_cgroup_uncharge_zswap(struct obj_cgroup *objcg, size_t size) static u64 zswap_current_read(struct cgroup_subsys_state *css, struct cftype *cft) { - cgroup_rstat_flush(css->cgroup); + cgroup_rstat_flush(css->cgroup, true); return memcg_page_state(mem_cgroup_from_css(css), MEMCG_ZSWAP_B); } From patchwork Thu Mar 23 04:00:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13184907 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1FA2DC6FD1C for ; Thu, 23 Mar 2023 04:00:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D21336B0082; Thu, 23 Mar 2023 00:00:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CD3F96B0083; Thu, 23 Mar 2023 00:00:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B72AD6B0085; Thu, 23 Mar 2023 00:00:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 9BAB36B0082 for ; Thu, 23 Mar 2023 00:00:51 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 723AA802FB for ; Thu, 23 Mar 2023 04:00:51 +0000 (UTC) X-FDA: 80598811902.11.03BE0BB Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf17.hostedemail.com (Postfix) with ESMTP id ACDCC4000C for ; Thu, 23 Mar 2023 04:00:49 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=cgw7YjSY; spf=pass (imf17.hostedemail.com: domain of 38M4bZAoKCCUZPTSZBINFEHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--yosryahmed.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=38M4bZAoKCCUZPTSZBINFEHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679544049; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xrG0SezuE3D1BzHk1bOIN+tOwhU+wFsyLck18RLDkiM=; b=EQKCJqORdEdavwRM2I7/0GNFxFKtQ6d5vtV3qJ+2+m9EoMRqjnCAPDM5oRKrWST1zv9R5A 4ok7PISfV3sNpyoaXy7ZcIpEjZUaJCTwVPJt61EB39Vdjtd0MzJ/UCQ3+gWbrt+5of3PZ9 YJ+wrFxgyQ2Zk5lk7HjLa677YbbIGS8= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=cgw7YjSY; spf=pass (imf17.hostedemail.com: domain of 38M4bZAoKCCUZPTSZBINFEHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--yosryahmed.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=38M4bZAoKCCUZPTSZBINFEHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679544049; a=rsa-sha256; cv=none; b=WbwZdpZiOGwaQ5hDX6TL8mT30V3xzpf4mtLQlru7mwS3x0doM4oStvvoJEPnLuScLzhutk jEdwVwuXkxSru1dLnNTYCS+M2t6UmB+4oe3eLgmvOWCZ0JOBjwQkPEnNSgYqGn3xWykf47 Ra3QMtWUHLOQpbGFI8TY4yhm4S4qVYc= Received: by mail-yb1-f202.google.com with SMTP id 185-20020a250ac2000000b00b6d0cdc8e3bso10222880ybk.4 for ; Wed, 22 Mar 2023 21:00:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1679544049; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=xrG0SezuE3D1BzHk1bOIN+tOwhU+wFsyLck18RLDkiM=; b=cgw7YjSYw+yZCrYg4MWNWyL1ocWVlMfhI6U/IH5SNNI/KXZEirBQTnVHw1Jk19ca7x YZMRDXY2GIXRShVJyUBfGcwrm5OQ54Bn3XOEBif3BbHD9ArewU0b2vLyFm38bp8+NRx8 4lwk3DHRAIpY5JhlVhZi+qJFnS5DTPOFtneU65+9wwMhEPN3ScjeohcSLADEGreopxgz hPSmYPd9gVS979yLkwNh9j+9QGCIU5M1yATespEadqsIAym+pG18x2O6VFUlYJCK8ejc v8yJF8c/6m0MiyUkNdlDzwM4KjC9ikuULQ0wGXuneSrTmETcMqRGnUz3Mhll67JgWKef njxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679544049; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=xrG0SezuE3D1BzHk1bOIN+tOwhU+wFsyLck18RLDkiM=; b=GzBbwf8pTFOJOApFDK0vTdxvThnNdWD+1jFpoMrGsoMw7l5yrhv1WFCUe8+qxY7fY9 jDQtspJyApzpUxqAVx1YWz6cI31ecyRqB1fnfyPMJIkKEvSika5c4bpt2oVpHfaVJLUJ QeqFT8q3DAhkBnF+kxi+ScrLrSNltZL7SgueGS0Q+XbkDrV7ldSeJNQSq0jzgseyAhGZ HpbWRJbla4vlDS2sFLZRzzllSZadOAj9AShX8WSSwqHIQlrZl25sABbcOlR1pzuSgMQr g5bcRBOtEjktSlNi5HziKUV7RYO5u/ZWwzXuPaGqDHPk9r8HTKbVSowS+pfNDHj77ioE njpQ== X-Gm-Message-State: AAQBX9cH5VkkBQKEOOnSo1QKqJJbakfyIps8oiBHP+biVa9k6kOiDLMI UzV3Hh1Xzt/GDWOi+P2gVlr6QSzWqKHV13e1 X-Google-Smtp-Source: AKy350bTQGWCAJk1730fCXLD+nAem5ueAaNE2q7H+XY1RCF27byWvmJXxhgyQR2PhcI0/Laslob9u/xZD8mfS05D X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a81:b149:0:b0:544:bb1e:f9cf with SMTP id p70-20020a81b149000000b00544bb1ef9cfmr1185027ywh.4.1679544048914; Wed, 22 Mar 2023 21:00:48 -0700 (PDT) Date: Thu, 23 Mar 2023 04:00:34 +0000 In-Reply-To: <20230323040037.2389095-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230323040037.2389095-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.rc1.284.g88254d51c5-goog Message-ID: <20230323040037.2389095-5-yosryahmed@google.com> Subject: [RFC PATCH 4/7] memcg: sleep during flushing stats in safe contexts From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: ACDCC4000C X-Stat-Signature: utbidxtpnqows33acgpku3zsa63n6zw1 X-HE-Tag: 1679544049-482795 X-HE-Meta: U2FsdGVkX18Ljcvw/XJ3SVkNjkae76nOR3sBIOR54B9WJnQKIRmDQsVHfMU20hqNjPupv7jTs3WmZcaq1x0KFcgtBs6itruZEBknY1CFlMX9gZgPavCuGABLRsAoI7WXO0uwKVT9lWa4t7wGJt2sUrLjFL1S2ZAc6FUlszKC4Jv1onJtso38Rfo8ErtKnDBC4MBsusBjKJbtCqBSsYZ2u2csYXFD2Gmm7K9oCS6C3upAunwtt/a4KadUIuWdadusiB2l/5TWK6FMiITc6PcWB4O53qp28/F976InsMH0tqSZVkXeGJqpjuLZYXplm0vJHg1FBZ3Vrf8Y6IdOGsGl8hhMZ6Kht6+z/Sk1MGZ5FJKlzaAkbV2VxElFP8m9NbzvCnAk7n1orw/db+ZdFHg1jExg21aiTRmm5n7DnVMiKB0sxAcNs0RFfRo404/hgAPxNHStvpiangLFC/df7NOnH0IwVifkKuXyQ11v3QVWh1mQGMq4nRYo6jb1PfLe1GKM/4hlIX27mBK7oPA9JVC8BQpzNoWLWFLqRLYNwNmZNhslWQO9DJC8MZmLzl8giFEytVSS0wu6gGIGVuvifrkr3P8W10D0i5v11lk7RoLGgIKtComibqUATQnxieDKf9OUzIXyW9FuOdWGIulgRQZ2GdoI+B7kI9SN13i9cgHnAvW06u+YvgZ24pxxhKxDDpzbcNbGgCCjWrCEifdjnhhs+b8Jez2H/MVVj2Cc6HKt2SBHyP/okNsLATMjmpo5Y7kicJTn4wac8nlit0q+SK9z46VocTxJhjXUc9tztZHiW5egShe6Gbc5jhl2rS14oBYF2URGBr5U4xKARLpeaCzKmihLLsvCmq8sjnasgRaGJlrSPw+jBMTaG3WMegTPqClFE8bvAVMwPeOFEZeKZK6wdxLm5umhX5S3D8Y+gakJHn+vsFknnTwRpQYiNmQCYW9wcuORb8unhhFTm6JBdIC HynioMl9 9usRz2ywr6c+cybu1yVxRXcimaYLE2nL/tXEs8tk1tRlKqeusPavmNBbipEknLu6EU+/O6xd5oYEXyxegBeXYym578fHuO9sLaQXDViXtkHxAtL+p8a+p74XOElv1k0H0ogqaDEwMai6ZkyBh9XTI4TodyerBJjoegGRiFcbxB1FnbC8URitgrnPBfEe7jlKIsxXYuTnzcdHWsqtAgsG3+6AxBdOsDdKdr42R5yBA7yKUuhF7bXqfcsyO5wmMQkaU19Vjs9lXibN2kbgT6Tgip7C+S2d9AMj/+ORcFl/2oqX4616ly+C5AIZ/rjOg6vIH/mlg25Xl6AcKTPjcFyngP1f5GPEPzP11yzWqW2AaBxSUmShgt4IRiE1WPJXpMBkrsDISvIy97EEnj5qWC3fjaS7PJ5CvRoWtuhfbRfTfWUQzXzGEJNfIqF1F6yEYIAoRrOY/eMBpw1RTtC2Crkv/UfA31693qrC0Wn/WhtMIukNWvoPu/g91t29eYDgfkImX0iH5NAc3JDwNu76+HN5Ujlrk125vkMfmd14unhmw4QXplDrp8U/WmHLtN8tq+/m+0V7l2R9G6B/qNhRiQJgc0NL9YUCqqp37GkZY5o3hIqQO5PZPd3BkdrM+Bo4gpK9B82pxnFkV9hB0kmUcPbcA2IYwZA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently, all contexts that flush memcg stats do so with sleeping not allowed. Some of these contexts are perfectly safe to sleep in, such as reading cgroup files from userspace or the background periodic flusher. Enable choosing whether sleeping is allowed or not when flushing memcg stats, and allow sleeping in safe contexts to avoid unnecessarily performing a lot of work without sleeping. Signed-off-by: Yosry Ahmed --- include/linux/memcontrol.h | 8 ++++---- mm/memcontrol.c | 35 ++++++++++++++++++++++------------- mm/vmscan.c | 2 +- mm/workingset.c | 3 ++- 4 files changed, 29 insertions(+), 19 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index b6eda2ab205d..0c7b286f2caf 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1036,8 +1036,8 @@ static inline unsigned long lruvec_page_state_local(struct lruvec *lruvec, return x; } -void mem_cgroup_flush_stats(void); -void mem_cgroup_flush_stats_delayed(void); +void mem_cgroup_flush_stats(bool may_sleep); +void mem_cgroup_flush_stats_delayed(bool may_sleep); void __mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, int val); @@ -1531,11 +1531,11 @@ static inline unsigned long lruvec_page_state_local(struct lruvec *lruvec, return node_page_state(lruvec_pgdat(lruvec), idx); } -static inline void mem_cgroup_flush_stats(void) +static inline void mem_cgroup_flush_stats(bool may_sleep) { } -static inline void mem_cgroup_flush_stats_delayed(void) +static inline void mem_cgroup_flush_stats_delayed(bool may_sleep) { } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 72cd44f88d97..39a9c7a978ae 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -634,7 +634,7 @@ static inline void memcg_rstat_updated(struct mem_cgroup *memcg, int val) } } -static void __mem_cgroup_flush_stats(void) +static void __mem_cgroup_flush_stats(bool may_sleep) { /* * This lock can be acquired from interrupt context, but we only acquire @@ -644,26 +644,26 @@ static void __mem_cgroup_flush_stats(void) return; flush_next_time = jiffies_64 + 2*FLUSH_TIME; - cgroup_rstat_flush(root_mem_cgroup->css.cgroup, false); + cgroup_rstat_flush(root_mem_cgroup->css.cgroup, may_sleep); atomic_set(&stats_flush_threshold, 0); spin_unlock(&stats_flush_lock); } -void mem_cgroup_flush_stats(void) +void mem_cgroup_flush_stats(bool may_sleep) { if (atomic_read(&stats_flush_threshold) > num_online_cpus()) - __mem_cgroup_flush_stats(); + __mem_cgroup_flush_stats(may_sleep); } -void mem_cgroup_flush_stats_delayed(void) +void mem_cgroup_flush_stats_delayed(bool may_sleep) { if (time_after64(jiffies_64, flush_next_time)) - mem_cgroup_flush_stats(); + mem_cgroup_flush_stats(may_sleep); } static void flush_memcg_stats_dwork(struct work_struct *w) { - __mem_cgroup_flush_stats(); + __mem_cgroup_flush_stats(true); queue_delayed_work(system_unbound_wq, &stats_flush_dwork, FLUSH_TIME); } @@ -1570,7 +1570,7 @@ static void memory_stat_format(struct mem_cgroup *memcg, char *buf, int bufsize) * * Current memory state: */ - mem_cgroup_flush_stats(); + mem_cgroup_flush_stats(true); for (i = 0; i < ARRAY_SIZE(memory_stats); i++) { u64 size; @@ -3671,7 +3671,11 @@ static unsigned long mem_cgroup_usage(struct mem_cgroup *memcg, bool swap) unsigned long val; if (mem_cgroup_is_root(memcg)) { - mem_cgroup_flush_stats(); + /* + * mem_cgroup_threshold() calls here from irqsafe context. + * Don't sleep. + */ + mem_cgroup_flush_stats(false); val = memcg_page_state(memcg, NR_FILE_PAGES) + memcg_page_state(memcg, NR_ANON_MAPPED); if (swap) @@ -4014,7 +4018,7 @@ static int memcg_numa_stat_show(struct seq_file *m, void *v) int nid; struct mem_cgroup *memcg = mem_cgroup_from_seq(m); - mem_cgroup_flush_stats(); + mem_cgroup_flush_stats(true); for (stat = stats; stat < stats + ARRAY_SIZE(stats); stat++) { seq_printf(m, "%s=%lu", stat->name, @@ -4090,7 +4094,7 @@ static int memcg_stat_show(struct seq_file *m, void *v) BUILD_BUG_ON(ARRAY_SIZE(memcg1_stat_names) != ARRAY_SIZE(memcg1_stats)); - mem_cgroup_flush_stats(); + mem_cgroup_flush_stats(true); for (i = 0; i < ARRAY_SIZE(memcg1_stats); i++) { unsigned long nr; @@ -4594,7 +4598,12 @@ void mem_cgroup_wb_stats(struct bdi_writeback *wb, unsigned long *pfilepages, struct mem_cgroup *memcg = mem_cgroup_from_css(wb->memcg_css); struct mem_cgroup *parent; - mem_cgroup_flush_stats(); + /* + * wb_writeback() takes a spinlock and calls + * wb_over_bg_thresh()->mem_cgroup_wb_stats(). + * Do not sleep. + */ + mem_cgroup_flush_stats(false); *pdirty = memcg_page_state(memcg, NR_FILE_DIRTY); *pwriteback = memcg_page_state(memcg, NR_WRITEBACK); @@ -6596,7 +6605,7 @@ static int memory_numa_stat_show(struct seq_file *m, void *v) int i; struct mem_cgroup *memcg = mem_cgroup_from_seq(m); - mem_cgroup_flush_stats(); + mem_cgroup_flush_stats(true); for (i = 0; i < ARRAY_SIZE(memory_stats); i++) { int nid; diff --git a/mm/vmscan.c b/mm/vmscan.c index 9c1c5e8b24b8..59d1830d08ac 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2845,7 +2845,7 @@ static void prepare_scan_count(pg_data_t *pgdat, struct scan_control *sc) * Flush the memory cgroup stats, so that we read accurate per-memcg * lruvec stats for heuristics. */ - mem_cgroup_flush_stats(); + mem_cgroup_flush_stats(false); /* * Determine the scan balance between anon and file LRUs. diff --git a/mm/workingset.c b/mm/workingset.c index 00c6f4d9d9be..042eabbb43f6 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -462,7 +462,8 @@ void workingset_refault(struct folio *folio, void *shadow) mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + file, nr); - mem_cgroup_flush_stats_delayed(); + /* Do not sleep with RCU lock held */ + mem_cgroup_flush_stats_delayed(false); /* * Compare the distance to the existing workingset size. We * don't activate pages that couldn't stay resident even if From patchwork Thu Mar 23 04:00:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13184908 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D786C6FD1D for ; Thu, 23 Mar 2023 04:00:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C5E196B0083; Thu, 23 Mar 2023 00:00:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BE6CB6B0088; Thu, 23 Mar 2023 00:00:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A39EB6B0087; Thu, 23 Mar 2023 00:00:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 919AF6B0083 for ; Thu, 23 Mar 2023 00:00:53 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 5B7E8120169 for ; Thu, 23 Mar 2023 04:00:53 +0000 (UTC) X-FDA: 80598811986.26.A5C2F9E Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) by imf13.hostedemail.com (Postfix) with ESMTP id 8EF9F20018 for ; Thu, 23 Mar 2023 04:00:51 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=VdCCijFi; spf=pass (imf13.hostedemail.com: domain of 38s4bZAoKCCcbRVUbDKPHGJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--yosryahmed.bounces.google.com designates 209.85.215.202 as permitted sender) smtp.mailfrom=38s4bZAoKCCcbRVUbDKPHGJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679544051; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9m7S+Ebo8MkLWAncbvKsOW8wVqp1O+0Jr3lM7bEbkLI=; b=7sMZ5fDy3RPYQ+guBjaZXTHGSS55FwIFGFVSoh1TjvRGPbj1em299GDJc6cYfLYObUcCfl jplYBTxz9POc4QpfpKhDzTlZVC4Io62Yu+UeGgeD3TBeandWdtRGlc3/Www0gLXAsjFIcP 0ey+eOPbvYbks6R2ANBCz9ry9q8xSpg= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=VdCCijFi; spf=pass (imf13.hostedemail.com: domain of 38s4bZAoKCCcbRVUbDKPHGJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--yosryahmed.bounces.google.com designates 209.85.215.202 as permitted sender) smtp.mailfrom=38s4bZAoKCCcbRVUbDKPHGJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679544051; a=rsa-sha256; cv=none; b=BJEJiV3DBGFIwFVmIf6B1P9s4ANJIjDg9loE3u4VdCIdwHGcLhVLHARNN7Qls9L2Lprndk qMVbayKRes5wHViAOqyociweHTR6B8pGJjE+dgb6VKdSOriPZQez2t/ogNLZmMUh90+5+x QDlWkEvOeoOYXca4boPyVSPNfyOMYTE= Received: by mail-pg1-f202.google.com with SMTP id e36-20020a631e24000000b0050f76fb84b6so3544146pge.8 for ; Wed, 22 Mar 2023 21:00:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1679544050; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=9m7S+Ebo8MkLWAncbvKsOW8wVqp1O+0Jr3lM7bEbkLI=; b=VdCCijFiKDJ4yDz8tFMap2tBXNfAakMH2cmowf1HUrZHuzx98c171wgP+Q37bsMmOY dvm3L1704oXKLAgAkOK0GjUnYQoP54+fQujrx3ACvUW9YBMoC0EV4BPolgDqFHWKSqHY vC+M0AbP8Thw306r5f0tU0s0RnClNRrpM7iLZQZCh4urjjm1nPlLVN2cvKbhB0gks3OE JIza4tgsYELq7gTHgeHRM8vhlVtpzgxg6m7gbPXptQ7q4d0Q9K++kyZO3UiZG82ponKY DM+ht1D3A+Eg8Co2dt+ym/gky/eIx2ki3TeVSJ2PxhVMHdlX5/2KaImaACWOeUxKy2jU FXPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679544050; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=9m7S+Ebo8MkLWAncbvKsOW8wVqp1O+0Jr3lM7bEbkLI=; b=hI8omRw54QQcJAJmy4sQKbNzhdmYpnwH0YKGWmLdy4MrqrUNtQ/+Ddh2xoRC9RqjHn pt+993N3Mjhdbn919WUlBGN5D7zWwsfa3Nb1AG8MZC3z4700Y8PkPSWM8Ga3s16p1iDQ XqVlmxMWozF0YnOe19vKJgjUUjtSJ9XHM80Qt3PKAQQDFVIb/5cqgSCgURc+5i8hIunY 7VFyPhDhDb3gelDvDuESwGQ/p7BlcaRAvcTEGGi+igk6otOCCnVHZ484PDDIOxXYyMf7 Et2C4cBZqRqmslwKT85xj/GwbqJ/Lkn1iTq1pIJStMKPO1SEiS6bEf0e/dTAHn3jv234 yMfA== X-Gm-Message-State: AO0yUKW2jin9BbLvxOtZpQFyKOBO2RcHqmDp94+Ckaf6KE6ScnCuB1p1 I/zJP2jC2VWboCvivYOKm7WUY+AzHGq6cvuP X-Google-Smtp-Source: AK7set+m4AnWZnXPXOtgLtJe2rjiQdXqc3xUA0g5EOBUkvh8xUumZjwEW1aaQclKH89BoDnkrewjWDyAajrYgNTk X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a05:6a00:80e6:b0:623:8a88:1bba with SMTP id ei38-20020a056a0080e600b006238a881bbamr2937696pfb.2.1679544050539; Wed, 22 Mar 2023 21:00:50 -0700 (PDT) Date: Thu, 23 Mar 2023 04:00:35 +0000 In-Reply-To: <20230323040037.2389095-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230323040037.2389095-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.rc1.284.g88254d51c5-goog Message-ID: <20230323040037.2389095-6-yosryahmed@google.com> Subject: [RFC PATCH 5/7] vmscan: memcg: sleep when flushing stats during reclaim From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed X-Stat-Signature: 9tu3srq5dcfqm4jek1a3k15mend4urc6 X-Rspam-User: X-Rspamd-Queue-Id: 8EF9F20018 X-Rspamd-Server: rspam06 X-HE-Tag: 1679544051-41225 X-HE-Meta: U2FsdGVkX1/zcdI2lpAb0iEiujsDuVrYueVN0G87fVNqOCD6FqLQ91WkeRQjxVFutxWt4Te7V78F/hVQZIWPQtE642hFGSMVfmHy43Gn+WjKz8/82sCVkDh8FGtGFkRsFZHO91lxoGprpe2XBFxe51ZR6nWdbfOPEo8J+T5pJG8PNBDHpmg062URFT0Nfe3HFcCA5jFtksTlCGKMueXOS2AtBhqyUnqulDzYxA+TSaC2zoQ6GpLie3RVXJkcd6sag8VVqCaUmtRGoXT6hwmsKpsa63nPI6rtPTiNHfaB82ZtQ9KlKxNjjj7TSKwKUKMbDfC8m80YW8T0odH6QG/Am49fZ5mV8WlIVxeNcLE4f4tLuG2rPG8WNpTLblzmggVKFqNZ12i67Fz0FdaMRQ/+MEDmtFFXzFxZmc1c6iaV/vFWzhfRABzUZIRlEZtoi6DAqRdCglxsFFs8zcSI6HC2+tevEkkCN2jxp1oOHXmZn/XufMyf7j1Zyz7idcLqTSVQG/i+rVXcAZf/dm/nb1NuOBf6G+LrCKdby2XMulj8W/PVo61c55V8xQxn3uSZm1tUeJJkSXpfYmsMP2qnyftebIE52mnjMMyDP1kxmPBIpEwnPxVIev+e/EWedjTWrcygxwMDy6CRUlxhXvtXXdVgt1BA4Rz5H/xNmZG6ihdrEFsnjajmcylpY0q/HR3WeGlJ2jcubrU7/GcYuY+dco6mmy710p8RRUJCrEtSGcuS6/+sFmkavd5ZUu+q/1hSe0XtSdTCYLLZlOKAFIyi9V+CZppF7KA6NS4FAz1/LHN8oJYOuuL2NQvKAGxq1JyI8EVXoueh6NerrdRF7jnuQBohJXNdG85Rgbi0vDHIfspWoIaFZHT+T7taRl9IYgEe6s6NovJkHHCC5OW3OFQTmWIoiXAG074xpf7/uc5Q/YiFto4JIqYFitInnU80Q8PCJ9AmUJ89ECTWZ+HOQNPJ5ez HskN9O9d goe+cVA+jOAfVimzH15457zD/1QSfCCoBQbP4/OQUkY7GXyUZAoe9bTQEQ8MrK7lL3uunsbzYST+HATz3ed4Vy1ne8hFkCTZyQPJuNPRVeus+9DmEluyRvcThvVVLUmrzclM+9Moz3JJNPlLk6yjeHUz3bZSqQTmuNWp1PtJ/Q0NXcb0C2xowQSBb6q+F7nrm4NizJQaleEWlHQCHLN3Mqwr3AXwxKTofTwcM+gq3hiZyej0PuwRuPHPcHqITTOHWPNf73bYENUJvOeCgitouKv6FeEv70Lf2U1z26By2a01VJtYLLdrTZ8xJj93Mij+iQrVRh2olvMAddyQNZ+0PYEspc19ub4fdqGIK+Yy0VKRHYReurkKtPoqrhK8JX7pDFfVOSsvgcqmDsXq0UIjTXMBQXtWCbz7NBfqsgbUfm29kvCJhWBSlLqhd4SUOy83e82RqMPbpKxOkTpIVMnHa0JlH7+gEvXG8kErhoA8nrSC4S2/EvBNsrl20fxDvrKhqZDN3W0+7iokmymV73tbgJquVtuNd+9jrCOwFE8UO0mi5o89eVUIQfzmCAcGB5pqGxTQ+MrD9FRpmP4hiTwl8oq7IAO6pIDR8+FjDSqPfH7JkYCO0Ux4+VJt2ofeTds3dyoFBCg/bqHCvauUwATOEIRo0Ow== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Memory reclaim should be a sleepable context. Allow sleeping when flushing memcg stats to avoid unnecessarily performing a lot of work without sleeping. This can slow down reclaim code if flushing stats is taking too long, but there is already multiple cond_resched()'s in reclaim code. Signed-off-by: Yosry Ahmed --- mm/vmscan.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 59d1830d08ac..bae35cfb33c8 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2845,7 +2845,7 @@ static void prepare_scan_count(pg_data_t *pgdat, struct scan_control *sc) * Flush the memory cgroup stats, so that we read accurate per-memcg * lruvec stats for heuristics. */ - mem_cgroup_flush_stats(false); + mem_cgroup_flush_stats(true); /* * Determine the scan balance between anon and file LRUs. From patchwork Thu Mar 23 04:00:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13184909 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1CE8EC6FD1C for ; Thu, 23 Mar 2023 04:00:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 88F436B0089; Thu, 23 Mar 2023 00:00:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8644D6B008A; Thu, 23 Mar 2023 00:00:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7533F6B008C; Thu, 23 Mar 2023 00:00:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 59E6A6B0089 for ; Thu, 23 Mar 2023 00:00:55 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 2E1A6A01A1 for ; Thu, 23 Mar 2023 04:00:55 +0000 (UTC) X-FDA: 80598812070.11.8DF25B2 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) by imf01.hostedemail.com (Postfix) with ESMTP id 6A91340013 for ; Thu, 23 Mar 2023 04:00:53 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=CYUJafqL; spf=pass (imf01.hostedemail.com: domain of 39M4bZAoKCCkdTXWdFMRJILTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--yosryahmed.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=39M4bZAoKCCkdTXWdFMRJILTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679544053; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IoeZYZryJtZ5YKLINZigwUk+eCMrWPKZYyl5pNjfs0s=; b=L1ZjT3IOmCjXrIFn6bAW7OyuPgvQHoetrTyQVvSwKZvkjHKaxvid1rhiYdLUA5SMln+fPj IA4rjEcmzLuzdhXUbpwi6ZXyKbLEPZDhMd8MTBfTZV8mAGpUTteWBFMcIbJ8Y5Jftac7I4 ncx3coFs0GObg9R9rSFtwI9KkCv/1uA= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=CYUJafqL; spf=pass (imf01.hostedemail.com: domain of 39M4bZAoKCCkdTXWdFMRJILTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--yosryahmed.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=39M4bZAoKCCkdTXWdFMRJILTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679544053; a=rsa-sha256; cv=none; b=PbO98yvuBCmBq7rjfEDNCavtSKJJePmi4VDL6n5/KHnVI/XzqLLsUUW30+SXkz2YPC0FjP 7Cbi5VmqnymKJ1tIURwwunXACOrEOLfXa2/E8LYIMQRI1oOODorMXk5fRy/Le4Ny4Zyzkv KzQpgiaaZZDv+oT8rV1ROPA5wj6Mp+Y= Received: by mail-pl1-f202.google.com with SMTP id p9-20020a170902e74900b001a1c7b2e7afso7040174plf.0 for ; Wed, 22 Mar 2023 21:00:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1679544052; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=IoeZYZryJtZ5YKLINZigwUk+eCMrWPKZYyl5pNjfs0s=; b=CYUJafqLeSTzFoXixoXxJfg+d8a1R4JvCUerDh64ifDjpdaxbdbWqowybN/gvJ19LK Og473fSzkkpOeMA58DC64BMRm6M0MCGDM/VGSfHLkAlVcw8rDdEYpMTrVbzG7kCaO9yo XEy99SSbQbOpczsL5Zy04pgal2f5dv+k8luu/GDpUWQB1RZl9qYQhWpR8p3p3vD98PGc ce78QmYoavhlvFbHk6YMCQ65CM/lnXEYztj/xUq2pBOa3UnU3uBgApeimhahme8eLkyC CQZkL9MM7bPQMzx+6ozclKMKwr8T8qPF+ZR71Vewcxu4v3esM5hXIWD8hj0gOLxV+qOY L9ww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679544052; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=IoeZYZryJtZ5YKLINZigwUk+eCMrWPKZYyl5pNjfs0s=; b=H6ZUayMNcLAPerp6dBv7HZOe8jFnUQp9fOoe1K1LQCUF3SaxGpw9Kc0+bRU59rZIc9 LOmwCfWJz+8Gsto0CYdK2ub7EtDEEruHcwdyFcV4Aj7/cDug0sTUshme1pLo+0+yMPgL a2mmJ/2ZyfLcSY1O1mU//b7MLfaNWDAOnCHBm7zqGh6D5NJ06vgguLM25VWOLllRm6Lu vwzHypMYmSQ87+Ati1OU/+Pod0twolhbQZ42080Q/SOT5wrQ7jgLb9Apzivvvk74U7bb r3weA2au2nKu3LaV9OVs0DeXjCnMpmuzw6l1UF9O5gYP3Xikq4HEAq+EhnhBTva1sQnM nwRQ== X-Gm-Message-State: AO0yUKULjpmpGhJE+q//sf/MI7UqE3H+xYP8S/BI1KXB3S1M4r2eQ87l SIFYPfamluIh3QZxnnqRKbFrsTyfbhoTL5/k X-Google-Smtp-Source: AK7set9aF1O2O+BZVk6rqDRrbbtYYQ/yqA23fBe+9ZzPXGEhcCIhooyW8p0Z2epSuqN1jjR7FDL5kfmsz1ZKE44X X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a17:90a:ce11:b0:23f:d473:dd44 with SMTP id f17-20020a17090ace1100b0023fd473dd44mr1881490pju.3.1679544052257; Wed, 22 Mar 2023 21:00:52 -0700 (PDT) Date: Thu, 23 Mar 2023 04:00:36 +0000 In-Reply-To: <20230323040037.2389095-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230323040037.2389095-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.rc1.284.g88254d51c5-goog Message-ID: <20230323040037.2389095-7-yosryahmed@google.com> Subject: [RFC PATCH 6/7] workingset: memcg: sleep when flushing stats in workingset_refault() From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed X-Rspamd-Queue-Id: 6A91340013 X-Stat-Signature: zjuihm3wt7r9azktpcnqxb6877hmdzbe X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1679544053-475883 X-HE-Meta: U2FsdGVkX1+hchwr3HPCLL0ADODj6e7fH5k578ZfiUPfyfMzTuP9Gn0GMSuWid75YJaJP75RD73jcjJIRlpetxXrqCH237reFEcjIPAyqCFRk4ZMDs38HvXsOLwZLCAo0yZnDI/TrknzGg3d+D9WP3s3l1dcoGMPloIZEEOChWYTQvm1nexi26FSo7nZ0YmGCg7E1cmctXN7uRnBJAFQB/P+O6PlhR2qWBX9T4DC5aQc9j73t+7gswzw88yDm3FSQwOMZ02DYEcO6DY1qWsYCxqiY68soDDubnnqWx1B9u9AntA7DzUB8vyqr7EkRdEPMUeD2Kr/UROQ0H2HU7K1ci+nWU3ScOiiQE1o2ePD9Trps5GttSHVgatxwTo9JyvAb7RvtGFXaLRvUe2zkeJBj0dt814G6byZFwJKTQ+f0DMxYRCO6k9qFpfAIF7rTaCOssVgQqv1+CdsWRZviXkHtQWhQzDAzw3TvMM5H4KjAO1K48tbWZas+mwyX32CldkxUIib/X/HDY7+sJlrpXwzZ9T/dnEbWCNy+bpgvFfPkNOJhVpUt9twmG+LlBcstZ8NDZCjWAlk4JUiNqfFgBuaJLFivkMZ8t0BmMSSaxwG81JEU2pAhiGC+NiXmm7nbKnw+S1UksCoO7evhyBQzgQ608vVgwQ7BwAqxCWAEC5hKY5QlBG3RiHJWMMQ+vIJHtZM32loT6cWSm1XDNyUr9eqXYRineIZrfqLB7yP/M0V/NnksGhtC0fbru6N6tT+zHejeZio+hWCCZaNriOq5aOapNq9pzP+vWfs3rg8zYLfNAlA7gUe1yk8Vw/QZRMEmbL0SQqj9wEXsQAl1bJ/HqAi0c5WxnVOAppvG9SbPCuzAES36HJz+ZkYqRWuTZATaSwodv3iaKZOThVXqf87NACJfSJm3X/jQVqAJrUebquooukiMEmKauqjgYNLMaRCBt4opmjR6NNW+I4TZAt+++R Bb+J483h 9N+T/xGlc6cbmNwmCJmHdrutM6aZFIaky3KH4o5d0saXyE3az7kPAaTBIWRfboMdGn85qAobPkNyM1v8CzAEqis9bIt1tQV4aSeaPEfVCAut0+o6rFkMbMFnskHKFCvtFoBR5WXab4GCfuQxpV4QT9a3CIsWeki+pidjn7RpPa9IFOTo5DzV4RHJqQXRnVPtsQF18ymXF2Vlu7nLQykMqqDckRaUgIKeu0BznTuMk91sQdta9hZ2LKOQ2Hpf8XtmUjMiEgC+f+8UDfSuu2oA3p1m2PZP/z3ywEj7dEyWQwVsbB0Uiaj3ThWgK8Ej+7UKgmZvHDY03W5Y5vCnVVwTWelkhIWIcRUk0kAAx8HZQmDPBxsWftKKISGdpYb/cUPqw73UFUqxZN8g4aE1PqEqGxx8rrTDDv3MYyVIy0goY0iR6kAb6ihf3dvBYD8VA/0ekOtmDNgCFrnIcmGBYYPq1RniZ5aQSySk4+rYLZDi6FZ10dRzHOncmAoGDVLkputbf6x+bhBlcdcc1Q5z85LaCsb/fVBwhSi3p//r5QW/xx6eocSkR+YgplxQgG61nnjC7981oVXcwhSl954ME7RCpzCdvs6jhBFTfvB7SMXcLoA+CAJHJCYAjTybPgl4or86oFJ8i1GXdjxaf57qUZHeNftFImNCAlLkTpGGrrYgrnSLocYOb7VOoXCwKMMpf0hLVc+6tuBnQQlh1buM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000154, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In workingset_refault(), we call mem_cgroup_flush_stats_delayed() to flush stats within an RCU read section and with sleeping disallowed. Move the call to mem_cgroup_flush_stats_delayed() above the RCU read section and allow sleeping to avoid unnecessarily performing a lot of work without sleeping. Signed-off-by: Yosry Ahmed --- A lot of code paths call into workingset_refault(), so I am not generally sure at all whether it's okay to sleep in all contexts or not. Feedback here would be very helpful. --- mm/workingset.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/mm/workingset.c b/mm/workingset.c index 042eabbb43f6..410bc6684ea7 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -406,6 +406,8 @@ void workingset_refault(struct folio *folio, void *shadow) unpack_shadow(shadow, &memcgid, &pgdat, &eviction, &workingset); eviction <<= bucket_order; + /* Flush stats (and potentially sleep) before holding RCU read lock */ + mem_cgroup_flush_stats_delayed(true); rcu_read_lock(); /* * Look up the memcg associated with the stored ID. It might @@ -461,9 +463,6 @@ void workingset_refault(struct folio *folio, void *shadow) lruvec = mem_cgroup_lruvec(memcg, pgdat); mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + file, nr); - - /* Do not sleep with RCU lock held */ - mem_cgroup_flush_stats_delayed(false); /* * Compare the distance to the existing workingset size. We * don't activate pages that couldn't stay resident even if From patchwork Thu Mar 23 04:00:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13184910 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D6968C7619A for ; Thu, 23 Mar 2023 04:00:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E5A946B008A; Thu, 23 Mar 2023 00:00:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E0AE56B008C; Thu, 23 Mar 2023 00:00:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CD2316B0092; Thu, 23 Mar 2023 00:00:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id BEFAA6B008A for ; Thu, 23 Mar 2023 00:00:56 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 90105C01D6 for ; Thu, 23 Mar 2023 04:00:56 +0000 (UTC) X-FDA: 80598812112.26.ADE2A55 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf13.hostedemail.com (Postfix) with ESMTP id BE3BD20005 for ; Thu, 23 Mar 2023 04:00:54 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Hsck6MLI; spf=pass (imf13.hostedemail.com: domain of 39c4bZAoKCCoeUYXeGNSKJMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--yosryahmed.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=39c4bZAoKCCoeUYXeGNSKJMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679544054; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GbeROioJTSRzpGZIx65CHc5YHmhlz1VRWOdmKxovUco=; b=VvkiUjU1YvOcq4woS3rUi5l7vNowkJPQu3DW+YPTb/flv5rkvcyuSynh/w8Vq8R5RrQ8D0 9ffc8r+Ve05vJjiQF8QeS+QpCXfqV5juDlp3h2OnFuLzDC0pAZhlU1awlohHptY5yLGI5p nlMQxy04YYcE/8mpqiC6IJD4PyvoUj4= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Hsck6MLI; spf=pass (imf13.hostedemail.com: domain of 39c4bZAoKCCoeUYXeGNSKJMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--yosryahmed.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=39c4bZAoKCCoeUYXeGNSKJMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679544054; a=rsa-sha256; cv=none; b=GRu3KSZ8+Y19UaVLBCrnmvaALo+YPSOaDB1Txl5dEvA5srmvhuOqH2EdsVQZE1/kpvchU7 iDW9xGP+MktJjZP2600IPfbIMnBfmx2E16q3lhmQ5Uda2TuqY42ogKt36EXevyY1SukD8m Hwv4f6wCFZW8n3mIbfFrC0Qrj4gJ+Y8= Received: by mail-yb1-f201.google.com with SMTP id i11-20020a256d0b000000b0086349255277so21517532ybc.8 for ; Wed, 22 Mar 2023 21:00:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1679544054; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=GbeROioJTSRzpGZIx65CHc5YHmhlz1VRWOdmKxovUco=; b=Hsck6MLI0adIPASRaTY3d/eUkBNtZFq5fHTozHNB8yOtl2ZkHkJQQ4GNrxQh4iSoWz JkY9CbdFxu+ujh+Pyx1oKlS53Vj0EANz1HJ/mNRwoFFcZsbYiHuLCMd4Lx5axIZ0gB5l yaad/tHsVvn1tHU57blkHy/+dS8g9HHclDAV/QT3pdal8fCNCqCcJD9lMmCv5J074fQc TEWaFK69tR0mz3hWUAlm/mUgTwQep/HE638zK7huhLNEr0Yv6vPjLeQWwk4UQCFbwI31 KRefecpL1tiBaS/Y5Rnpmz8ntCNmT0BogLXtNGk9m7i1zmm8BH4noVNiIwwFJbgEqIWE ZH9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679544054; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=GbeROioJTSRzpGZIx65CHc5YHmhlz1VRWOdmKxovUco=; b=hByDhs+Lz8ZMm0hY5O8NZro/FYqxaCfAWp2Qr2M2sBQbmrTxkiR1WECiKXVmgDDvLK M+Fie5qum4pMWSSMw1aAp+Ldr6jPeOpDpCPwPDc3EZUhCEcVsyN/0DGcUT9Yb11/pDjb EX8AXt2eVBC53Y8dH3/HSb2/820YSlO0YBjsUTBy/dbDNu0O1SoEhPZn3lNZa9A/iNYt 5BExArlU0NDr4iJ+pa7YjR9jS8ywKLdx92uqYsYx/iaI5NNYj2QUayRviHOsQQnlL8+x Dt3XPAjP8xLfSUTosL0I7tk/bN2je2vlxiRKRIwaMucZkTjQd/KBdQmmJnrNybvTdAd4 ytFA== X-Gm-Message-State: AAQBX9dvljcfonRIFRJxlBr48vDvHZiVGSdvfio64q2clFR46tlQnWdM YN8wuUxNaeCPtquieNQA7qAbK7qmSMZT7jLK X-Google-Smtp-Source: AKy350aHC0fJxZVNDyOM4vrsmksnjt2c5UEEiO2IoOQY1+2DycjUjKQKdK6RunG6Yd33ZztPbzyijCwEZgPQmgPG X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a05:6902:1549:b0:b73:4a25:fd36 with SMTP id r9-20020a056902154900b00b734a25fd36mr1286158ybu.2.1679544053971; Wed, 22 Mar 2023 21:00:53 -0700 (PDT) Date: Thu, 23 Mar 2023 04:00:37 +0000 In-Reply-To: <20230323040037.2389095-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230323040037.2389095-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.rc1.284.g88254d51c5-goog Message-ID: <20230323040037.2389095-8-yosryahmed@google.com> Subject: [RFC PATCH 7/7] memcg: do not modify rstat tree for zero updates From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed X-Stat-Signature: cza439n5tcr5cg6b8w66udmj9orhbj17 X-Rspam-User: X-Rspamd-Queue-Id: BE3BD20005 X-Rspamd-Server: rspam06 X-HE-Tag: 1679544054-380071 X-HE-Meta: U2FsdGVkX19S4pTU/G8E+BNqfDB3T5xyW4t5JdrREz4rqOySNrkufGOyjRL69VGKafmuKREaGHhOjC5KmfDDQHLB8dxbynPvofXCUcSSOEsLmYl2GGSRbZyFcB4D/QR9ky9TGlIEnMICJx43nHn6XcSWXftkchsKaMU89IHNT7y303EawPdmTNdv2bF7vC8tc8a63UjAiMlDurUpCBiBbZTixxZCPOl19rNiuPvFd4zsEZNjNzlGPLwPWWvPWWzLvw93pkXZ+1bFhenwLhOPmaUstDAQEf73VQCiWDovleQxCnMhYNeTsVoMc7Aaftp0+uBco1myA7edsbZbjKjYuUi2ilOWMrnS/DTCV95lvkx/+aP+76oWt79/JXRhI5Fs+UhfXv1u/c+GbvOYdJqRlSRkuOpy3JDZJkssgOf7q8W1sJaslxvZR+eJwRo+HdZP8mW1njiTHnye4zXTsR24o+XeIi7wXq6rtspBPtjNpgSkDYhhGmavT9s/hLpJ39MA8ZmzYdXzkZNXeQ3g0V0w0jyWBR3y9ZmWwfiLrFBUdD5QuHCKqRl0FVke4ctLO7g0qLAWH+nbXFlbV0xOJ5+HYvP1DU0yv4J71vC+GPX1nB0+4CuclzDuiPEMIZhBMomCOZlcI0XE80yh+JLJet9ZzBvd4nU56dOaiu4dSRXC9B0GjTWYVDlWct1MNnUxPt5tFYpWEoDXiuqjpwLcq4Lk9LF2HamQEkAoF/RNwb9zeJJ0hSnZ4PMxoRB3gL4K3QyEUA9Yr1B7zZ/x/4TJSYs2KYE1XxUuVndAE859bennrRuGg1SsfbJFf3Yy6bz+nhrNQbblxfDUdi5MUevQKNfGbo42stFiP+q4i6fSUprlOKg3OxZA0udCeIX5QDTf6PmL3KJg2Kok3iOyP67kfWBg/9Z0fLSs7vhjhAUTxDVDrhZOoRL0reqC+OQuDOOS5m5WOQuOhRgUH6F+F1fW5Uo FnABCgiL CYoqB/exYTo6Yb7GkWNCaKDxY52qiiBZ453FLHTlDkmrz2aAetBO8b9eRQS8v1iOPAS+L0un8g0S0pbiQERsYDeNymYMLFfDQs0WFcjgclFhRz+DS+Sl29ObVAZozuM4wg8mVgQlMgl+MT+37JovDJrRznWokiJO2gx8PGAhy79hpKDbTHwnZP7diMH1PjDxfPCuVeVxgPzbSAMigELzQyhZWnQ+g57cIhiuCKuh64mdBGKIbai4nfQxdbvfw9cjDW3EnjFVeOh04CTNxcmLSWYKT6cx09+BoGJhoRrUaPTAK5m4py5cY3dSLv0+4H/wCmumuwCVuBPST4uGAxFH1IO+NmADDOur6l69frDGprYcPaJtDy+oMfwBPFI8+rGEJS3wP8i8Rgc1X34WEmJuOydPcknfBkPLtiBpnJRiqUizL8MwfsiC6wXFMccRucA7t6NTLABQ9C9pPnoVpvHItG/3qu45ReuzxKefX0TvtYzD1/yNAMz4xS9/QgsWvosL+1YUvLDQ+eK/uUM/2SVAJjo1bLWxu8Wgs53ulqEFP4yrBBy308ZHWSD4+mnSMXKLCPLW/akQB+zzR07SJCWvlOp37WR10TuBrvj7zrv1UG+GqKFtoGXE0/SZq27uw80Xf3P/51N+rYmVziW4X/ckzMiv2+w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In some situations, we may end up calling memcg_rstat_updated() with a value of 0, which means the stat was not actually updated. An example is if we fail to reclaim any pages in shrink_folio_list(). Do not add the cgroup to the rstat updated tree in this case, to avoid unnecessarily flushing it. Signed-off-by: Yosry Ahmed --- mm/memcontrol.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 39a9c7a978ae..7afd29399409 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -618,6 +618,9 @@ static inline void memcg_rstat_updated(struct mem_cgroup *memcg, int val) { unsigned int x; + if (!val) + return; + cgroup_rstat_updated(memcg->css.cgroup, smp_processor_id()); x = __this_cpu_add_return(stats_updates, abs(val));