From patchwork Fri Oct 6 18:46:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nhat Pham X-Patchwork-Id: 13411836 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5567CE92FFC for ; Fri, 6 Oct 2023 18:46:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4FF3880009; Fri, 6 Oct 2023 14:46:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 462B780008; Fri, 6 Oct 2023 14:46:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2645180009; Fri, 6 Oct 2023 14:46:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 1007680008 for ; Fri, 6 Oct 2023 14:46:37 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id CD8A28036B for ; Fri, 6 Oct 2023 18:46:36 +0000 (UTC) X-FDA: 81315917592.15.071AADA Received: from mail-pg1-f173.google.com (mail-pg1-f173.google.com [209.85.215.173]) by imf14.hostedemail.com (Postfix) with ESMTP id 09855100020 for ; Fri, 6 Oct 2023 18:46:34 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=C6sK2kBw; spf=pass (imf14.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.215.173 as permitted sender) smtp.mailfrom=nphamcs@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696617995; a=rsa-sha256; cv=none; b=4P6tNr4tCru48oXyfnxmAPTGJJ95odx9MxU1y06n1baEYfLwQ3uRI/AopyIs2TP2NastTX cyxsuWWkmHYypiaLt/cQo2pgUPrI6QwEoBGvCsaiwr1SChSm2FaAxTsyBH3oPiRaYE1Ms+ /Emijmh3TCQl4viQ4KtXpnIOV9dWut4= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=C6sK2kBw; spf=pass (imf14.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.215.173 as permitted sender) smtp.mailfrom=nphamcs@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696617995; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6PLbRh1GdQSuzgMGxcjYajdp+r42f9X7B8/Cs8IIw1E=; b=dYmOIV01VsnTO7BzK9wLT3OtIj/rFfkAD+Q5c7uKiTMwvBXGNGE7NAGmiOf8hkteq0XO4i kP5UxLBm5vajU0aLkWO9qxjsxlKEks5a6pfTfe3GjZYQg/dDbmPzqru2pl+TU0SBAaINbU bX/Il269k2UMVGoRc8bUTfFYDc07KxU= Received: by mail-pg1-f173.google.com with SMTP id 41be03b00d2f7-565334377d0so1887277a12.2 for ; Fri, 06 Oct 2023 11:46:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1696617994; x=1697222794; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6PLbRh1GdQSuzgMGxcjYajdp+r42f9X7B8/Cs8IIw1E=; b=C6sK2kBwIcpXUKBtnyBTJsq5mLZVwme/uHkH6T6ZPdjq6st1FNQF3wJ3io+MOJ2KSG wLEDRSIsJf9sZRR/PpwrPWuu/YjqmU53kPSBkQQz3iRVXa1cfQdFveC4ViCEJhUnDeeF /O+KxEtwRTk568isgAFl4NfVy1XTro1Sq2RHLYSL2JthCviZ5fD6NyNRPYZQj++NtCaS OF4xQV/i88flnAXUf7OdaxmKCS382HCpT+0TJGQY2AHYdusJ+mYhj+tE41bS9TlUJah0 /6Srl2rFkCGSDmeADRDYkxPOjOJHIvAHPe7Jx6iLRBpVesomxJKo52FTEOCKNQ76JvB/ gl8A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696617994; x=1697222794; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6PLbRh1GdQSuzgMGxcjYajdp+r42f9X7B8/Cs8IIw1E=; b=BPi8l1pACDXeTiJH2+RYmdyHKvJGV4lFhgM7zOX2ypdpdKq2QTSCE+6XuApb3z3f09 o7kk8IsJwwj5QEG9vkMQCcMhOCSzsMYLQY2u8Eed+eI0bWboWGeG7m87NsZsiQPYv05O Cf/6Utjlr/kYBJJrK1pxSSYGpvXeGDhEIQYvVMCafgyL5F2R1yRy/pA7KQxjE6ilvJLg DlGfoU4HewRI2q1RnFhhm5TB+nImhqSJsMUtlj19gQz2fg3fzsZYkiy6bQwsm2TfuNs8 V6eVhwR7jd7ZzKUG/cHaYNwWldjw9Jl+v9XFxIDXyAT6Wu4hoU3kv7/rOk3ygDWs6NrV oiXQ== X-Gm-Message-State: AOJu0YxJtkNNGTBCOAna7iM1Ax5EhrZeRc7Jdu1QDPDpPHKgQfRcaa4t dd6jLRXx6EoVUWBLC29jAShPmx1QPeM= X-Google-Smtp-Source: AGHT+IEe+6yqU5r07JsaeSeLI4O7kp/29vLl/f9OnIgJYThbQQ0o75CPnRFGXWPYmDXr5YACLKuB3g== X-Received: by 2002:a05:6a21:798a:b0:16c:bd7e:d524 with SMTP id bh10-20020a056a21798a00b0016cbd7ed524mr566168pzc.57.1696617993719; Fri, 06 Oct 2023 11:46:33 -0700 (PDT) Received: from localhost (fwdproxy-prn-015.fbsv.net. [2a03:2880:ff:f::face:b00c]) by smtp.gmail.com with ESMTPSA id g22-20020aa78756000000b00682669dc19bsm1800647pfo.201.2023.10.06.11.46.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Oct 2023 11:46:32 -0700 (PDT) From: Nhat Pham To: akpm@linux-foundation.org Cc: riel@surriel.com, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, muchun.song@linux.dev, tj@kernel.org, lizefan.x@bytedance.com, shuah@kernel.org, mike.kravetz@oracle.com, yosryahmed@google.com, fvdl@google.com, linux-mm@kvack.org, kernel-team@meta.com, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org Subject: [PATCH v4 2/4] memcontrol: only transfer the memcg data for migration Date: Fri, 6 Oct 2023 11:46:27 -0700 Message-Id: <20231006184629.155543-3-nphamcs@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231006184629.155543-1-nphamcs@gmail.com> References: <20231006184629.155543-1-nphamcs@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 09855100020 X-Stat-Signature: knxs7zc5xdjwzpifjo3dqjp88wrp9hp6 X-Rspam-User: X-HE-Tag: 1696617994-118405 X-HE-Meta: U2FsdGVkX18d0KMLkNLsagA2W9DNZ3x2bxtghdwFyVILXPm8yCtu2Ig03hGmTWfVN/P6SWflX1NymsMojALYQo+0OK9NunlomIEc9QSj0rjOFDeSWXdIXQOdigFLuNZA0mMSzWvFgEy9D8yWl/3yJ9l8qSjcP3tYvDx1/ahOGfg0pCuezQUfPHuJZSb3j/Pzs6qe/0j5kwok49gCZOnZEQQAmsLeYg5O+6i48VDkK/r+uuPp8YvC1+ZOichMx9SmNr1uZlYqY2YhxvpBD4U6AokVPy58RCpw9axOY6Tk91qsF2RLkkQ3+jrNJWd2OOgvFZenGvJKmVYXXh2worvEwIT5YqNrZqpYCogNuCdh9a+RbjrynN0Sp/i145PA2prxmpeKU6AuphbrWrBG8Lx+i6Cx9BNsGCpEvPFwMZSKS/4J24fGmshv6wxAabQmumvqNUNy6C1sLDNb5f9cieazYAZBNOtO0+/cehvlpMkmG7bKZyGXKwETUxlZL/3vK21d9mbG2E+ceB9o9BqA2oe1MlnmNPy+3Pxniv/B9FqOo4ThR0UyuMJRGGNo9dog6qz7tsk0q6FQchIlkizR5G4rZYbE0bK3DyNMnD6cvZqJMuCJctMqs4u0TDsVpHef5lh2C4NwMvCcFK3e1Z3M8RDZGosOGLywiYgXauw6euJ5ci9iix/k7n6DJc96xwkhAW00MNEIN+GVlDnNIxL3WnKHZ7M2FNH88Hk/O2WboreEnUGhzQHk+tAqdwV7KM9B+Svpm2WRCyV9Kljtugc4t++kVWuB5xwAYqT4z0gJ8wrg89YtedjzdlxsZrE/24InveG/o32nukdObISZ+r6LoyqYbslByH6d+9aWY/YCd88LJy2BUkxeXhK5L3oDRUVi16ouK3qkVBOcaGgefkaeFfUbl26H0MqVG8CrTe8HwsuMllXcN/Eq0jzq5/YuqOhn6hnMFKmZsUTFagek7xeJUoL y6V1aqx/ bEgreZkU3+EA1W6Ieck/fnM0XuPdaZ2+I/xv3WSZAoHezDvFhTkKw0Y13LXdJWBXqi505sIjmGUrNvjcYfifLWPx++jNjGMROCQ2tVVtzEowbtuuPWhBimDeenv71YV1f8AByXM6Ym+FktMzOA7VPwZRLuf5Vt7HmXLEYE8rugEw39Eg1mv3U5mv60Ft+KRujvb+ZzyatknErDBVcYNJ6MjJU8JXpX4Io8gYPJAIolZjjM9riiwI6C+WpRrT23dx9eviHOQZMxi8Jt2ZRm62MJFAsfigtZ91nbadXn6fJgD3CcVbH0O5NWfNjJciGpWmGJycC+LKXdT30JGk8jF8Adfl4gtO0t1LQ3xoV5K/ckbRrlpFzb4ouhiMMEgHYUxFOnOkxB+W4W7qptW4d37oScgfuQinu51kM4jEAEZDw6FLCxkujI6c0JP5Acm1qVnd1Td8OKwTXWn54/MyjIR8mkU1+sXVK/yMzf3bB/QYWzOQHoePiLf5jc6nG8vf69LbJm9fZ/ZqX+Vgmr9yLN6xPGk8evxw9u3v4ecU4YQcNLbQ5Zmo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: For most migration use cases, only transfer the memcg data from the old folio to the new folio, and clear the old folio's memcg data. No charging and uncharging will be done. This shaves off some work on the migration path, and avoids the temporary double charging of a folio during its migration. The only exception is replace_page_cache_folio(), which will use the old mem_cgroup_migrate() (now renamed to mem_cgroup_replace_folio). In that context, the isolation of the old page isn't quite as thorough as with migration, so we cannot use our new implementation directly. This patch is the result of the following discussion on the new hugetlb memcg accounting behavior: https://lore.kernel.org/lkml/20231003171329.GB314430@monkey/ Suggested-by: Johannes Weiner Signed-off-by: Nhat Pham Acked-by: Johannes Weiner --- include/linux/memcontrol.h | 7 +++++++ mm/filemap.c | 2 +- mm/memcontrol.c | 40 +++++++++++++++++++++++++++++++++++--- 3 files changed, 45 insertions(+), 4 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 42bf7e9b1a2f..5daf14da3759 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -708,6 +708,8 @@ static inline void mem_cgroup_uncharge_list(struct list_head *page_list) void mem_cgroup_cancel_charge(struct mem_cgroup *memcg, unsigned int nr_pages); +void mem_cgroup_replace_folio(struct folio *old, struct folio *new); + void mem_cgroup_migrate(struct folio *old, struct folio *new); /** @@ -1285,6 +1287,11 @@ static inline void mem_cgroup_cancel_charge(struct mem_cgroup *memcg, { } +static inline void mem_cgroup_replace_folio(struct folio *old, + struct folio *new) +{ +} + static inline void mem_cgroup_migrate(struct folio *old, struct folio *new) { } diff --git a/mm/filemap.c b/mm/filemap.c index 9481ffaf24e6..673745219c82 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -819,7 +819,7 @@ void replace_page_cache_folio(struct folio *old, struct folio *new) new->mapping = mapping; new->index = offset; - mem_cgroup_migrate(old, new); + mem_cgroup_replace_folio(old, new); xas_lock_irq(&xas); xas_store(&xas, new); diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 0219befeae38..b9c479d768e2 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -7281,16 +7281,17 @@ void __mem_cgroup_uncharge_list(struct list_head *page_list) } /** - * mem_cgroup_migrate - Charge a folio's replacement. + * mem_cgroup_replace_folio - Charge a folio's replacement. * @old: Currently circulating folio. * @new: Replacement folio. * * Charge @new as a replacement folio for @old. @old will - * be uncharged upon free. + * be uncharged upon free. This is only used by the page cache + * (in replace_page_cache_folio()). * * Both folios must be locked, @new->mapping must be set up. */ -void mem_cgroup_migrate(struct folio *old, struct folio *new) +void mem_cgroup_replace_folio(struct folio *old, struct folio *new) { struct mem_cgroup *memcg; long nr_pages = folio_nr_pages(new); @@ -7329,6 +7330,39 @@ void mem_cgroup_migrate(struct folio *old, struct folio *new) local_irq_restore(flags); } +/** + * mem_cgroup_migrate - Transfer the memcg data from the old to the new folio. + * @old: Currently circulating folio. + * @new: Replacement folio. + * + * Transfer the memcg data from the old folio to the new folio for migration. + * The old folio's data info will be cleared. Note that the memory counters + * will remain unchanged throughout the process. + * + * Both folios must be locked, @new->mapping must be set up. + */ +void mem_cgroup_migrate(struct folio *old, struct folio *new) +{ + struct mem_cgroup *memcg; + + VM_BUG_ON_FOLIO(!folio_test_locked(old), old); + VM_BUG_ON_FOLIO(!folio_test_locked(new), new); + VM_BUG_ON_FOLIO(folio_test_anon(old) != folio_test_anon(new), new); + VM_BUG_ON_FOLIO(folio_nr_pages(old) != folio_nr_pages(new), new); + + if (mem_cgroup_disabled()) + return; + + memcg = folio_memcg(old); + VM_WARN_ON_ONCE_FOLIO(!memcg, old); + if (!memcg) + return; + + /* Transfer the charge and the css ref */ + commit_charge(new, memcg); + old->memcg_data = 0; +} + DEFINE_STATIC_KEY_FALSE(memcg_sockets_enabled_key); EXPORT_SYMBOL(memcg_sockets_enabled_key);