From patchwork Tue Oct 3 23:14:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nhat Pham X-Patchwork-Id: 13408052 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 351D1E8FDC1 for ; Tue, 3 Oct 2023 23:14:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7D0738D003B; Tue, 3 Oct 2023 19:14:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7593F8D0003; Tue, 3 Oct 2023 19:14:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5AC158D003B; Tue, 3 Oct 2023 19:14:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 406CA8D0003 for ; Tue, 3 Oct 2023 19:14:28 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 09051C049D for ; Tue, 3 Oct 2023 23:14:28 +0000 (UTC) X-FDA: 81305706216.01.3FCF2C2 Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) by imf07.hostedemail.com (Postfix) with ESMTP id 38CF140010 for ; Tue, 3 Oct 2023 23:14:25 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=F2zDl5aX; spf=pass (imf07.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.214.176 as permitted sender) smtp.mailfrom=nphamcs@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696374865; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xWjIhVNT1E7PfrfKwNLVQZ455aStkJaWOS9he+QZwc4=; b=1R8eYlycrFDgEnI05GP+ugaxi93KJ7bYGFaiFaNuXUSbi88ChgCTkJSIomcxGpD3X8hnoj qZMEb/b1XldyAv4GT5t35mCPTWbhX4OlNaDFZxc5V5JzQNSmnrA0yAuYLJz9531weIKfcp 42m5AEqXMvc0NmnJ6sld8S6zsjtITvk= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696374865; a=rsa-sha256; cv=none; b=wnKt+7eluNVjrgVQAUQT2BNsQJx0T6MQSF3lJnjblvT+g5sS1sv4Bwa6Bq0Dy84nT3KRgM YL20d+4+J28qDAobfxcoM7KHsZqRaZaTy3f9pFsiR8qIjS5yrrElZUegdxafpJLwYgXsvJ iw/znNtVQn+pHbM7fEAnnWvAcyLlPWM= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=F2zDl5aX; spf=pass (imf07.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.214.176 as permitted sender) smtp.mailfrom=nphamcs@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-1c3d8fb23d9so11568875ad.0 for ; Tue, 03 Oct 2023 16:14:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1696374864; x=1696979664; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xWjIhVNT1E7PfrfKwNLVQZ455aStkJaWOS9he+QZwc4=; b=F2zDl5aXFF+gXURnk12etWBBj3hiF2wzOwGpYjTw02bd+75aaOpTT5yS6POz+xhnzK BQIjQNDG1bM8hEJ3T6KXhqyzUeZE/qZK1YhFKdJGwWSJ3A4rGkmpV+dS9YjpwU5G6JmG 9pd+bMuABT5t12Y0Q8icEspOrMc0ugvSVjH8l3Kr5VRv4RcImUpg16mm2frdbs2Xrxd9 vLLQs+G5PfFizbAj8yppwc4yvEqF/ACRTZAxVfCYoY4DwrtH9YNMzMwvjapfRFRejIjJ L9Ex1Wc3w+Rc3XC6qh7FNJyRg6juiIGkZw1x7UNSMZfG8Nmv6ZJCCwNBlIpmEAmvf2Fh HiNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696374864; x=1696979664; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xWjIhVNT1E7PfrfKwNLVQZ455aStkJaWOS9he+QZwc4=; b=viriCGvms2P7a94YK7HUon4WMpvglO1fURnB5CI+KWmW36FGhmnCzhPSKG8QB79NBa acB8OoNwXvAM98GH8m0JUyrw5sr+uqyDaNTNM4LqAWFpppen67RjyNArbQA8Atslm7kQ hIiYF9m/pD1rfGllvmGQ3u2meIn5ZhEyg8m2+5JFz5bee53WL8p0m3cSuQu491EO444j ZCAA1K71yKCiXRiW5Q2nlv2Un9OfyiG+NE2D6RLuWXIyjI/noKtHBnXisXSZtwrzFnQE Wwzj9nuW4O979v9CwHXqMUZE0m6LaAPz1/iBJBcsBsa5mjlMAS3Z/qQFW3zXUpkuFhSi lJJQ== X-Gm-Message-State: AOJu0YzNF/WHtuvhf3nfyKtdyAbXYCcJ6/TaRO6wl6TZE48LunhP8swM PHPmO/C8WoW3dBd9R/g+3R8= X-Google-Smtp-Source: AGHT+IETIduuzQcGbX3ju74VeGurMPVK+mZzhRAQKI3L+wYgIJF0Bkfu8w2t03oMVilyhWuyfFZ02g== X-Received: by 2002:a17:902:bc45:b0:1c7:7916:e87b with SMTP id t5-20020a170902bc4500b001c77916e87bmr1071338plz.14.1696374863896; Tue, 03 Oct 2023 16:14:23 -0700 (PDT) Received: from localhost (fwdproxy-prn-116.fbsv.net. [2a03:2880:ff:74::face:b00c]) by smtp.gmail.com with ESMTPSA id u5-20020a17090282c500b001c446dea2c5sm2173607plz.143.2023.10.03.16.14.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Oct 2023 16:14:23 -0700 (PDT) From: Nhat Pham To: akpm@linux-foundation.org Cc: riel@surriel.com, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, muchun.song@linux.dev, tj@kernel.org, lizefan.x@bytedance.com, shuah@kernel.org, mike.kravetz@oracle.com, yosryahmed@google.com, fvdl@google.com, linux-mm@kvack.org, kernel-team@meta.com, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org Subject: [PATCH] memcontrol: only transfer the memcg data for migration Date: Tue, 3 Oct 2023 16:14:22 -0700 Message-Id: <20231003231422.4046187-1-nphamcs@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231003171329.GB314430@monkey> References: <20231003171329.GB314430@monkey> MIME-Version: 1.0 X-Stat-Signature: izh3kz6w9qe1np98pwt3eqwzqx3wrkdc X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 38CF140010 X-Rspam-User: X-HE-Tag: 1696374865-462361 X-HE-Meta: U2FsdGVkX19slxlPhTfZWWuFo7VGosr6+YPf7pbzqi9wogvG1CU4bF/pNp0Ezw3x9AtSpUP/ebtpoFlBz9uzkyvb4AKCFuuIYCo7sMUMyppDcP16xjHZxVvrzRoUE8t74vmRUVCTA4B4uAEj8ftnWNLQ8mFHGtzwu1buTbusbdnQaxyamyGPj1t3qmsQ1FQKG2NZGZWaFyCZPILPZ2MS0PEukzhVQ+U0dcgTtc2oIfzGLmbbbzx7cjOnwQO4U+dqB4LGXKmjR+7CToHilz1wMF36j9T4zdnlErI0FTboFHN8KP2A0pRcE4/Vg9TKBli4Zou4olJChMfCXzGzs8kvN0cA2jKywj0hV0l8+AR0KFllz13PS+yXHnBZnDIabIYTPlc4Np+o+p1num++bCNQWEjuZXzKSNkJJWjfZKKfkrM0iBGEWnUBb7rI+eYtI171orBZaWD1OzKL0GyAp5CBwjNr4u5VBpYLYmxEU6PhLfN9XDJ4rqfyoJWiLLbvymnKDlfgwx5jVQ37VlyxWkix7FGiPr33PSP+rLjA8BO8aNS48vJdnpVgX8AFlvdyk6Q7Rkg/zbUlaLi+sjMGenyNk8+1eS2twEB5huu3Gs+MXCGQTyytRF9gKOLD23tYqsb1hFOV5tlepmVQZOk2plsCeNZCiyeLOGZ9/Mq/6PR23NOsxaqAi4IEflhvDmbP6jAs2XZdwvbyY1imSRK7CDoylaLrhR/o0t76t+UiPOzyky2TQHaO/HpPX4lN94QDWWYGZza3h08Xbqdnmuxnu2lWxHRACXaFsF2AYXbxIFONYjzNi6MC5vFLSBKHdfvGt2zSs2DxS7L+dosmGrDhflED47fA98lt/DIAOjTU8ZtuFn4qCNAS81hCa31gSeiskGusRF/sTbMl64zxSPr4BijvqRZWTkDsDMvrIFmsRh2zOYlM1Byl+8kQT3vqo2ami8M0x2SC5KzbnpPRrnJY6U6 25+ujlq4 OLuSEItQtFf5OLLhlLI10YFCgNXD7kg7l8Y3AF5LNYMNOazcFqsySf6neAoXwwdl0OISTCDXU0ZJ/FUINv3vdkZbxJ2VZ7UyNPGZrSBa7X/9o3aFEs1tHDf8AQkyP7A+/kOrxf3oh8MBMsPC0SrFrT+nQJFHVaVWF+QPexbcFjqOYMLFIfHepPpoHOTFBfXrJtMO7Wpha1cNHxhUHZGY/DS57sktbg0EiZ+ltr61grvMQ8IiRAkWSAtLgEQZis5et9O+powuinZHNKz2aPUNp1dppc5RaFFkMNJHpV1/fE6fhXjHcTkqGr7LjEbJz6kRyiJlbNiBbK8MuQZUUuQVj0YGWFDb+c/vm3NM38EeOzA1+yfGIVV7Z5X65bcaFXPOqE74VlL9d8JQfN/Cm4H0hPXynAoo09+oUEJnpBAjK9edcEjEIy7c9XwnSwdcBI1XRdfmVRmxdE0YfD0fnCWiVnnOmqW9Bs9CKKRxzw9ZnRqKbFC+w4lxz1XkHW+X5KBFSF/NiYfxgHHFghU6ZDViHK5lT/QJkYMwUvvz3 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: For most migration use cases, only transfer the memcg data from the old folio to the new folio, and clear the old folio's memcg data. No charging and uncharging will be done. These use cases include the new hugetlb memcg accounting behavior (which was not previously handled). This shaves off some work on the migration path, and avoids the temporary double charging of a folio during its migration. The only exception is replace_page_cache_folio(), which will use the old mem_cgroup_migrate() (now renamed to mem_cgroup_replace_folio). In that context, the isolation of the old page isn't quite as thorough as with migration, so we cannot use our new implementation directly. This patch is the result of the following discussion on the new hugetlb memcg accounting behavior: https://lore.kernel.org/lkml/20231003171329.GB314430@monkey/ Reported-by: Mike Kravetz Closes: https://lore.kernel.org/lkml/20231003171329.GB314430@monkey/ Suggested-by: Johannes Weiner Signed-off-by: Nhat Pham --- include/linux/memcontrol.h | 7 ++++++ mm/filemap.c | 2 +- mm/memcontrol.c | 45 +++++++++++++++++++++++++++++++++++--- mm/migrate.c | 3 +-- 4 files changed, 51 insertions(+), 6 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index a827e2129790..e3eaa123256b 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -711,6 +711,8 @@ static inline void mem_cgroup_uncharge_list(struct list_head *page_list) void mem_cgroup_cancel_charge(struct mem_cgroup *memcg, unsigned int nr_pages); +void mem_cgroup_replace_folio(struct folio *old, struct folio *new); + void mem_cgroup_migrate(struct folio *old, struct folio *new); /** @@ -1294,6 +1296,11 @@ static inline void mem_cgroup_cancel_charge(struct mem_cgroup *memcg, { } +static inline void mem_cgroup_replace_folio(struct folio *old, + struct folio *new) +{ +} + static inline void mem_cgroup_migrate(struct folio *old, struct folio *new) { } diff --git a/mm/filemap.c b/mm/filemap.c index 9481ffaf24e6..673745219c82 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -819,7 +819,7 @@ void replace_page_cache_folio(struct folio *old, struct folio *new) new->mapping = mapping; new->index = offset; - mem_cgroup_migrate(old, new); + mem_cgroup_replace_folio(old, new); xas_lock_irq(&xas); xas_store(&xas, new); diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 6660684f6f97..cbaa26605b3d 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -7316,16 +7316,17 @@ void __mem_cgroup_uncharge_list(struct list_head *page_list) } /** - * mem_cgroup_migrate - Charge a folio's replacement. + * mem_cgroup_replace_folio - Charge a folio's replacement. * @old: Currently circulating folio. * @new: Replacement folio. * * Charge @new as a replacement folio for @old. @old will - * be uncharged upon free. + * be uncharged upon free. This is only used by the page cache + * (in replace_page_cache_folio()). * * Both folios must be locked, @new->mapping must be set up. */ -void mem_cgroup_migrate(struct folio *old, struct folio *new) +void mem_cgroup_replace_folio(struct folio *old, struct folio *new) { struct mem_cgroup *memcg; long nr_pages = folio_nr_pages(new); @@ -7364,6 +7365,44 @@ void mem_cgroup_migrate(struct folio *old, struct folio *new) local_irq_restore(flags); } +/** + * mem_cgroup_migrate - Transfer the memcg data from the old to the new folio. + * @old: Currently circulating folio. + * @new: Replacement folio. + * + * Transfer the memcg data from the old folio to the new folio for migration. + * The old folio's data info will be cleared. Note that the memory counters + * will remain unchanged throughout the process. + * + * Both folios must be locked, @new->mapping must be set up. + */ +void mem_cgroup_migrate(struct folio *old, struct folio *new) +{ + struct mem_cgroup *memcg; + + VM_BUG_ON_FOLIO(!folio_test_locked(old), old); + VM_BUG_ON_FOLIO(!folio_test_locked(new), new); + VM_BUG_ON_FOLIO(folio_test_anon(old) != folio_test_anon(new), new); + VM_BUG_ON_FOLIO(folio_nr_pages(old) != folio_nr_pages(new), new); + + if (mem_cgroup_disabled()) + return; + + memcg = folio_memcg(old); + /* + * Note that it is normal to see !memcg for a hugetlb folio. + * It could have been allocated when memory_hugetlb_accounting was not + * selected, for e.g. + */ + VM_WARN_ON_ONCE_FOLIO(!memcg, old); + if (!memcg) + return; + + /* Transfer the charge and the css ref */ + commit_charge(new, memcg); + old->memcg_data = 0; +} + DEFINE_STATIC_KEY_FALSE(memcg_sockets_enabled_key); EXPORT_SYMBOL(memcg_sockets_enabled_key); diff --git a/mm/migrate.c b/mm/migrate.c index 7d1804c4a5d9..6034c7ed1d65 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -633,8 +633,7 @@ void folio_migrate_flags(struct folio *newfolio, struct folio *folio) folio_copy_owner(newfolio, folio); - if (!folio_test_hugetlb(folio)) - mem_cgroup_migrate(folio, newfolio); + mem_cgroup_migrate(folio, newfolio); } EXPORT_SYMBOL(folio_migrate_flags);