From patchwork Tue Mar 28 06:16:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13190553 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 987ACC76196 for ; Tue, 28 Mar 2023 06:16:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 04898900002; Tue, 28 Mar 2023 02:16:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F3C016B0074; Tue, 28 Mar 2023 02:16:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E29A1900002; Tue, 28 Mar 2023 02:16:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id D20346B0072 for ; Tue, 28 Mar 2023 02:16:44 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 977F3407EE for ; Tue, 28 Mar 2023 06:16:44 +0000 (UTC) X-FDA: 80617298328.24.9436FB7 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf12.hostedemail.com (Postfix) with ESMTP id E998040011 for ; Tue, 28 Mar 2023 06:16:42 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=NeuDT5i+; spf=pass (imf12.hostedemail.com: domain of 3SYYiZAoKCAg6w0z6ipumlowwotm.kwutqv25-uus3iks.wzo@flex--yosryahmed.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3SYYiZAoKCAg6w0z6ipumlowwotm.kwutqv25-uus3iks.wzo@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679984203; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=sp9O7DL9obBb0muMtifinlg/m6vcys5sIiJC9lNDWi4=; b=4zRYSg6PMW7RJd64+v+D+XWZYP8EdCjMyhRb4vbew7zj092CIHgT4ejniAEMN7BBq9mOTO JloSrkLW21vNO8KAAQ1BZD7gdpSfG+SJIqNApxu2ezdAHo29dZ347Y9CiGUDZInGQeuems LzeeZhmwiuIGXwhVREZPcmh+GM35TFE= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=NeuDT5i+; spf=pass (imf12.hostedemail.com: domain of 3SYYiZAoKCAg6w0z6ipumlowwotm.kwutqv25-uus3iks.wzo@flex--yosryahmed.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3SYYiZAoKCAg6w0z6ipumlowwotm.kwutqv25-uus3iks.wzo@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679984203; a=rsa-sha256; cv=none; b=gn0UKoHgIRBkVrw2AkpNbg7l8uhsAzBQOMyYW2WFfISsZNpcmOqOldwZI+2F/WVNcI/ZhJ mQ1i9cF+jT5n44lt/oyQJb45rK4i95GvQ2Y6Md1dy7Fa9yXVTmKcgr129VNH0AduGdnSHk FJkx6799O+OfMsaxBU39A1Sg1RSB/LE= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-54476ef9caeso109101707b3.6 for ; Mon, 27 Mar 2023 23:16:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1679984202; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=sp9O7DL9obBb0muMtifinlg/m6vcys5sIiJC9lNDWi4=; b=NeuDT5i+IX7EsYuGxrlQHGfM/O44K/q8azh0N01/JWHajTpr/8bCoJ2QTmweAtvhfe gYoeDE3hEA/7en26XdMklOvz+naw4YK/G+YhpbjekM9lK1gn62mOub1rqZqAuOD3Va4S 1QIsws0AsakJv6xmZVg2tR6CGN3YqUveLDQmgizdGWo93cuUB1v+gh6AulTa+cAtsQ1E KKArWRQY0GrdStBwfUXdbR6yPsdQlbb61EZd9PCwqGtcxA7k1j294s/M/nt3EKFywNtt l3OyHVKVYr6ScGHLswWLNtnzcPUbA+BqIwaG4p3phWhy3WEkn1PyyK7m1rhPq1yMmAEF q2mw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679984202; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=sp9O7DL9obBb0muMtifinlg/m6vcys5sIiJC9lNDWi4=; b=QNGjha4k9pebI2VpmJiTCYi4mNKbQ/1b+WU//xNRVCbAAJIJN25gyHunviQ9ZkZtjR sLa8OecCK7baA9R+W3181RwYmSHeztkBGpy1EXTp4VaKKiumTSEWllS9G26VoPGTj9dW /Z5wUFwhQ+E5Yb269rU2VYCNFDsE442SO7UMYBbjWUaZwAXvX2XVqIpEYr/NejIkp9cA hhNArctcomn3q4hrlxp6T2/e4AGSILN8E02SKoWXhS9PfGR062Feytoj9SuULHG5qfNI bNl5OZ8omlxBqqi41+XWvdmxGujpHKwYMb4q8sHVaYx+a3goPw6AfK4iCFWnZzT1pG47 ddmw== X-Gm-Message-State: AAQBX9dG8VH4AUZiFsPcTlqVa+5bm4AWgTbSWm2eLFl5NRJYhqQyj8gA 9/9BvrvJ4u7Snqs1AnkbbmsUBzb7bXHRgVRA X-Google-Smtp-Source: AKy350bZY7FA4NsJ0fqBTDQ1vhS8yI9N8IskrUxgaH2uFCVGGGkuCT76RaKDFHndt2HqiHyVKmyLU1Kd/HHTyvK/ X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a25:2749:0:b0:b4a:e062:3576 with SMTP id n70-20020a252749000000b00b4ae0623576mr6830636ybn.13.1679984201935; Mon, 27 Mar 2023 23:16:41 -0700 (PDT) Date: Tue, 28 Mar 2023 06:16:29 +0000 Mime-Version: 1.0 X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230328061638.203420-1-yosryahmed@google.com> Subject: [PATCH v1 0/9] memcg: make rstat flushing irq and sleep friendly From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , " =?utf-8?q?Michal_Koutn=C3=BD?= " Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed X-Rspamd-Queue-Id: E998040011 X-Stat-Signature: tmdangcw11t8s7akt7j4txwrw49dxsma X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1679984202-420614 X-HE-Meta: U2FsdGVkX18NV9w3VehdgmlSdsZbHmCXQ7LcCYP4nMNyuZ+avkuD3hQzy4duxxfZ/oJwVbRTADJmWTwO7D9bgMMljQ4cPYI9ebtNaI9zSJZhgke0vDthXq9K0Elz9X0T1iUpzddr2VzOPWyqYwu+9kRh80+VkVYDLEG/Egp8X8B4UynHMerBJkvj1dPvOt0fgnNjgR1UwxBiCdbDIMhndAXcCkJPDzTQZOWUMvymgTX7rwOkdcHdG+VWCuOPTk/eGj0OUYQMUEBUTi1vLe2tuqoY3ktngchTihcTrB5bm5qkvAmvt+3VUgLoNMWkxTO2sHjaIjmUuh1SH/9JKAY2NZuXA/0joL4n5yqExu5bbA99GlgCOoCngEa9idUfVG8cvYQE49bYyR7iQcFs8Uu3kXmlzKPeYzqouxJc5QGa0aUWMlFjOTaAVX4IcZlBHLljCEnr2K15x0wdKMvDerqwBvBH2Dc75MI9r9jBoOR7neJyuYXiMQ+05jhUe3UjBcK7TSgxN8zWz6cr0feHwnOMK8RwAmNdH85uI+eghctiaJNakvJKnLcuSplPBcYSvR+e+wEcUS/kyl2ovT6QXMuCJC68JHZa3h/LwN98Vh6DYVf41lXUHCxA8JsHuQ/mpA8XEf2YX3PNiQi2dq4HpgM/vVhRHtLQacTGbEAL/3epPXdeaEUYL34gpUX0Oid01OYadNlD24o+HMrZ54TTzLAX+bsboBqk2kz22aWfdRbcz+RiCwLgBghgpgjjBcnHbSEAV2fxa+YWc/y8cHvDjFpbBwcSVnquDOLkrOuhZdjSln4O8kBgdv3Zsm0KK/CrC7+IOkSA15I/IeVeOBjarAMa5Hkgo3Qc0c4b3ws2TOOUW37ka5rttzp1K3wj2ZTIiDwYh3QLQH4ffkvAfNHgLN9IkyCjfZP0bUv844TJc0es00jeW1Af19H3csNf2C/xK80ULPrhaDRjm7M7E1dkBEM 8cCtxHxi 8Ju0u/jw0Hih05f3uCawi3O49Wcl4Y32a5123yL3L48FCJLyisBTFx1v2C3RCW+wjYO7aCU8lYU3fmESFptSLI/XdDVZcidMKLI0Gkf8/RkLqoXkL2DFx5Jva9OeWYJVlKIXBb7VC7VWr/nCyvYNzxvO0Kaj9lMhjLVMU2AlSWYVCZHcOWKIuOCrTeiR3UORI11a6JetK1P8mVedm9ln+leagiTEGp4pSA8DRwWAs6TRadq5dFJky+YV1dgIFfa+de1Js1NMnH7c2i/JdBtH8ePza0PQEAMcVwHyKijxRVk2dboNSBL9Afe1CFQ/Jv3Q0eTkMMV4tY4NFvO7RWzyq2oxbyzyxTb9DMCehitxST9g3/EUvBS5ggBS3FwZuqyh8LfImjrh3mfaJJl80n0kF66B1n74sulMMdw0wfqquv4Og34h5NLuto21GJ9wi3Vl1k6ro5emsI5BZkwAO2w5HJUsI/VdDMC9GdSfiquHRUMS0jxp/Aznxwsjcdq4OPSJD7K4gOaLGk6tgRFVimCY7SJ0PurD2GWblioEcj4pS/wHo1ZAWAZo5FDTkYXXNhoPp5oxU4ycX4H92uDvXytd7HKGxY/C+qJDBwRKoJC2/dbdrje++biwUxaCUFCLiTsABh8SA X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently, all calls to flush memcg stats use the atomic variant for rstat flushing, cgroup_rstat_flush_irqsafe(), which keeps interrupts disabled throughout flushing and does not sleep. Flushing stats is an expensive operation, and we should avoid doing it atomically where possible. Otherwise, we may end up doing a lot of work without rescheduling and with interrupts disabled unnecessarily. Patches 1 and 2 are cleanups requested during reviews of prior versions of this series. Patch 3 makes sure we never try to flush from within an irq context, and patch 4 adds a WARN_ON_ONCE() to make sure we catch any violations. Patches 5 to 8 introduce separate variants of mem_cgroup_flush_stats() for atomic and non-atomic flushing, and make sure we only flush the stats atomically when necessary. Patch 9 is a slightly tangential optimization that limits the work done by rstat flushing in some scenarios. RFC -> v1: - Dropped patch 1 that attempted to make the global rstat lock a non-irq lock, will follow up on that separetly (Shakeel). - Dropped stats_flush_lock entirely, replaced by an atomic (Johannes). - Renamed cgroup_rstat_flush_irqsafe() to cgroup_rstat_flush_atomic() instead of removing it (Johannes). - Added a patch to rename mem_cgroup_flush_stats_delayed() to mem_cgroup_flush_stats_ratelimited() (Johannes). - Separate APIs for flushing memcg stats in atomic and non-atomic contexts instead of a boolean argument (Johannes). - Added patches 3 & 4 to make sure we never flush from irq context (Shakeel & Johannes). Yosry Ahmed (9): cgroup: rename cgroup_rstat_flush_"irqsafe" to "atomic" memcg: rename mem_cgroup_flush_stats_"delayed" to "ratelimited" memcg: do not flush stats in irq context cgroup: rstat: add WARN_ON_ONCE() if flushing outside task context memcg: replace stats_flush_lock with an atomic memcg: sleep during flushing stats in safe contexts workingset: memcg: sleep when flushing stats in workingset_refault() vmscan: memcg: sleep when flushing stats during reclaim memcg: do not modify rstat tree for zero updates include/linux/cgroup.h | 2 +- include/linux/memcontrol.h | 9 +++- kernel/cgroup/rstat.c | 6 ++- mm/memcontrol.c | 86 ++++++++++++++++++++++++++++++++------ mm/workingset.c | 4 +- 5 files changed, 87 insertions(+), 20 deletions(-)