From patchwork Fri Jul 12 02:44:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 13731251 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BCC06C3DA45 for ; Fri, 12 Jul 2024 02:45:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9BE416B008A; Thu, 11 Jul 2024 22:45:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 96E596B0099; Thu, 11 Jul 2024 22:45:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 835466B009A; Thu, 11 Jul 2024 22:45:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 67C5C6B008A for ; Thu, 11 Jul 2024 22:45:10 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 07515A0572 for ; Fri, 12 Jul 2024 02:45:10 +0000 (UTC) X-FDA: 82329558780.20.2E2A5B9 Received: from fout6-smtp.messagingengine.com (fout6-smtp.messagingengine.com [103.168.172.149]) by imf28.hostedemail.com (Postfix) with ESMTP id 07817C0006 for ; Fri, 12 Jul 2024 02:45:07 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=sent.com header.s=fm2 header.b=lz8yYnWl; dkim=pass header.d=messagingengine.com header.s=fm2 header.b="F 1B8jHJ"; spf=pass (imf28.hostedemail.com: domain of zi.yan@sent.com designates 103.168.172.149 as permitted sender) smtp.mailfrom=zi.yan@sent.com; dmarc=pass (policy=none) header.from=sent.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720752291; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jNGroiyo4BuTb3hCWpWux7PfCpk0jLOtvpYsdbCWP/I=; b=BDMDF2pPM4HXwb5MtXqGyFPR6IVQXHlxPkcgUBoA8VlooyW/wdcF3qPQ7VeXclGR3HPZpj +PZLdbcvNuqsL8/3GH8Kp1nyDF7pLF4uXMPNV9k6aSjuiGxs2f3V4uBHCCiGyv5xkdZGCX +1RyVM+swcH4Xz2Ob3Q6PLEK0jfIwlw= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=sent.com header.s=fm2 header.b=lz8yYnWl; dkim=pass header.d=messagingengine.com header.s=fm2 header.b="F 1B8jHJ"; spf=pass (imf28.hostedemail.com: domain of zi.yan@sent.com designates 103.168.172.149 as permitted sender) smtp.mailfrom=zi.yan@sent.com; dmarc=pass (policy=none) header.from=sent.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720752291; a=rsa-sha256; cv=none; b=cokbUiaCFn9Ju/sdc8Xnbx6R1q/gBhVPo5bcs9JZUJPVncSCaYUEHB4Iv1Hq7ifii5UWm/ muuDtqfS13bmMqnTBh8ZJ+eINak067HahQ2xmFr5wldTMqYsE8sS2fnZodS/yoyHbQHrCg b4WaF0Th8yZXqZL1bg0ygNbJYLM2hCE= Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailfout.nyi.internal (Postfix) with ESMTP id 5CDE01388638; Thu, 11 Jul 2024 22:45:07 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Thu, 11 Jul 2024 22:45:07 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sent.com; h=cc :cc:content-transfer-encoding:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:reply-to:subject:subject:to:to; s=fm2; t=1720752307; x=1720838707; bh=jNGroiyo4BuTb3hCWpWux7PfCpk0jLOtvpYsdbCWP/I=; b= lz8yYnWlO09wfHb83ln8BKkBoq5gRUQUwFV6dKkRSNHwT4rFIM9zWzLIqahj4zcY nIKYIY0qBuykHnAcw9kWMVsgUcfpGapKaEEYwFcjePBPb9J+vOKMqVWSuhQdCUgi X/ywcj41J/8GmeqbecKCObHP2Gl3Zr1ZqExtA96VyEde9CU/pdggiB24rp7Ynkr3 KmvsfUc7zb0dwxGPPC5RGBYlTmzbvDgrDGjuoYWCIjKEyIi2CsDuZ8gZCHfRMDAy iA7+METH6x27l+qkdLRd79V01z1npc7+W91QR4PbAE2opafwTdtzGiAXOodyqxH/ P+RcVgx5iCn98RjXD2Wk8A== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:reply-to:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm2; t=1720752307; x= 1720838707; bh=jNGroiyo4BuTb3hCWpWux7PfCpk0jLOtvpYsdbCWP/I=; b=F 1B8jHJ4ViBoTClRPWsQtEFz8ORDTWX/HBWUZXJMc/mK0/CscC9ecK2cnFdnms0Be Di/RSin7dfh9ANGM62MH/uM50t0aqSb1MHPuEYdPLltOQ15RQGlK0fnZVZGRt97l VE31CoxTR5wQr+uQojH/Oy1DWxLiiC80wUV7S7nzO3hbRu1IXbvqTq2JlXFJd5de aGcgLCth+TsO3jtZOGz0cqf/9IFDBzNrZP83iQ5jAddkb2UXrMHU0OurI1IE7twH t/zF5WbD9PIw2WTeQ2QEpLUvG8j901OJkltRg91Lsk1qggqlaE5i2tb4TNhXt3Ru ZojyNMaWjOD7ZZ300CnuQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeftddrfeehgdeigecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvfevufffkffojghfrhgggfestdekredtredttdenucfhrhhomhepkghiucgj rghnuceoiihirdihrghnsehsvghnthdrtghomheqnecuggftrfgrthhtvghrnhepudfgfe elhfekiefgtedvudfghfduffefieefffejvefhlefhhffgkedtfeevhfevnecuvehluhhs thgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepiihirdihrghnsehsvg hnthdrtghomh X-ME-Proxy: Feedback-ID: iccd040f4:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 11 Jul 2024 22:45:05 -0400 (EDT) From: Zi Yan To: David Hildenbrand , "Huang, Ying" , linux-mm@kvack.org Cc: Zi Yan , Andrew Morton , Baolin Wang , linux-kernel@vger.kernel.org Subject: [RFC PATCH 1/3] memory tiering: read last_cpupid correctly in do_huge_pmd_numa_page() Date: Thu, 11 Jul 2024 22:44:53 -0400 Message-ID: <20240712024455.163543-2-zi.yan@sent.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240712024455.163543-1-zi.yan@sent.com> References: <20240712024455.163543-1-zi.yan@sent.com> Reply-To: Zi Yan MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: 07817C0006 X-Stat-Signature: pfsjaswgqcp3n9qnrhd9moksn31mjrie X-HE-Tag: 1720752307-323765 X-HE-Meta: U2FsdGVkX19+IKf29bPwg74eyU+1eAYEc9aHWB7I60fnwHmdVhEh3AaLELcAvoU8fAYB3iWhVVswbr9ynibcxnt4Zt2Arv7WFfV7N4F3K985I2QHXY0ut1ZCUfaAM58COjA/VmCC7/soa1KT8nBXwDpR6px5k5cCGPx4sjzmNU0tm0TIeSob06rcsw6Iz2POAa9P0WRt+IcmR5bhlWn/YWW0rRNmVg5jV6lRfkcblOqzTQXHtkur+TI07/qWGcm8P4q4x3b1ieD8TxK+dZJMcMyALne+paR36O/nyhKufb5GFKQcAG7Qj4i/gh9iIED7suLd9eLiuamZfFuyHzIv6sW37R3+dpwL7to7VECZfcQT891jcYyqUvDiwvTcIBpHSO7n6XFIEAzFlkIlQOh/UgYDy06punPp1Bw9/pPq24D4+pvEDxf7v4Nq5BYWh7GfHORv2+qGCa6BcTF3fFUZrFDWu0HZpRdd8LbfsuD0AV3v++7+Cj7fhs+qWFaXBhVAe7pzUDicb/LfcrvAtNmxSDC0f506eTZf2HuGR/JCqyN+8saUT0lqucE6bqc5ILXWzJvQkOUskA0Zo42wS/Di3LWcPF/IazYN698QV6gETkyECpBKs42++76ZV643218ixOXVcF9p++m0wg5krncF5vz4dyxKnTdxAr/C/XOTiKQWqGS1EYJFjb4plclnetLWLfbOxsqOAF5OVC6c8gFttfx+EBqeMEFGRez6fa82LMKsFJrprOT+HC9m1fotpaG6McbArJCrFxTwTtfl9cJXRB3vDDwSClv75+sihNJenLcba6/RRtBmJ/mc2XZP0+1R1wDUX+M2GcbOvxG0UW6T4h2ojEsXAzHSikMWh6JmJZZc52mkCUd1kbhKFGMoV2C/NhvzwLmSPe48SOv3dKUbtI8HApMqVq3FjB7NBSROtgIW45zJaYzGz6DUMSFdd+nqyYroDiWKCNIZ1pDikZT 96QQl17A FNIclLJRWj8A9a0p9S49dqlKpxHsC+ilZ6yF76ef2jweQq12yx1sAHs/MCQAZCjBMCVfvETiDX0AMNAUYBFvMqSkBQakLO0zoab4GWA0Sfg66ZkioxIMI2XnuR0iFv8xSP8mycpuhcqwBzUwUyG0oALUcno0Ie6S7/VfXuuQz35NcQpcvjX2WbI7DHhJ4p936Wl3ZnEXYrR68vf0tje30dpfzy/jG1Pm/7BUA0W7gojnkF4HKSYl65MBGi502g5Ove2wrthMwc2Xa7iLha/c5+myxrP4mHVL16xFHKQvhzUGnm/jkCX6rDSNF3Hl2Jp7+9ijNLdzlfOGxOjUym2UXDjCmreNjefKUmKU06wkP/ka1R3gtzPJVylf+/bMBV5FzCPrlUW8tcvP9y77Wx6e97Jb1KCVc9WqxuyhB8A+aepEiGLmtXbi6pC1zAer/dSBTAGt5SMe1ZK+2GyU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Zi Yan last_cpupid is only available when memory tiering is off or the folio is in toptier node. Complete the check to read last_cpupid when it is available. Before the fix, the default last_cpupid will be used even if memory tiering mode is turned off at runtime instead of the actual value. This can prevent task_numa_fault() from getting right numa fault stats, but should not cause any crash. User might see performance changes after the fix. Fixes: 33024536bafd ("memory tiering: hot page selection with hint page fault latency") Signed-off-by: Zi Yan Reviewed-by: "Huang, Ying" Reviewed-by: Baolin Wang Reported-by: ... Acked-by: David Hildenbrand Reported-by: David Hildenbrand --- mm/huge_memory.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index d7c84480f1a4..07d9dde4ca33 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1705,7 +1705,8 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) * For memory tiering mode, cpupid of slow memory page is used * to record page access time. So use default value. */ - if (node_is_toptier(nid)) + if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) || + node_is_toptier(nid)) last_cpupid = folio_last_cpupid(folio); target_nid = numa_migrate_prep(folio, vmf, haddr, nid, &flags); if (target_nid == NUMA_NO_NODE) From patchwork Fri Jul 12 02:44:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 13731252 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0E2A1C3DA49 for ; Fri, 12 Jul 2024 02:45:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2BA4E6B009A; Thu, 11 Jul 2024 22:45:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 21D636B009C; Thu, 11 Jul 2024 22:45:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 06EFE6B009B; Thu, 11 Jul 2024 22:45:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id DFCAB6B0099 for ; Thu, 11 Jul 2024 22:45:11 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 994A01407A9 for ; Fri, 12 Jul 2024 02:45:11 +0000 (UTC) X-FDA: 82329558822.12.1A26719 Received: from fhigh3-smtp.messagingengine.com (fhigh3-smtp.messagingengine.com [103.168.172.154]) by imf27.hostedemail.com (Postfix) with ESMTP id 9360940013 for ; Fri, 12 Jul 2024 02:45:09 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=sent.com header.s=fm2 header.b=nZefoTlK; dkim=pass header.d=messagingengine.com header.s=fm2 header.b="M TIp+Eg"; spf=pass (imf27.hostedemail.com: domain of zi.yan@sent.com designates 103.168.172.154 as permitted sender) smtp.mailfrom=zi.yan@sent.com; dmarc=pass (policy=none) header.from=sent.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720752292; a=rsa-sha256; cv=none; b=kpEidhAH5damv/6cU6i+6GEVGHSVh8qkgpU4pVmHJ6iF4Da1qGo72vZon8YR6y0e42KopF PNpsHL5EY8ZrIPLFo9jOsxFR3+YRkEAhfLxO7yBN5l/zIt1CFswUkBFY4AB5eHT3Z22+aR wHYCbGwfR4kf4l8o3glaFkgpa+uHyNU= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=sent.com header.s=fm2 header.b=nZefoTlK; dkim=pass header.d=messagingengine.com header.s=fm2 header.b="M TIp+Eg"; spf=pass (imf27.hostedemail.com: domain of zi.yan@sent.com designates 103.168.172.154 as permitted sender) smtp.mailfrom=zi.yan@sent.com; dmarc=pass (policy=none) header.from=sent.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720752292; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=to4wnXa/PHhQKWIeEkU0XgFAJrxVOY/Z7C32sHOwamo=; b=2BeONkQuLPSjAB777Y5l23qXRCKETmJM2xWZO/MrJfcO8ciSZ19TvUiKPUZCNQHdcRivoM AipaKIuKrW4wW+aG2YLhMKn9ujyShDj0kLMpUy0f7mzCXSD+Ck7V4HSQ9WGzTdrethaCO8 cvezlH54YTcjF/8rW3KNvuPrlA7JXYk= Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailfhigh.nyi.internal (Postfix) with ESMTP id DC88711403D2; Thu, 11 Jul 2024 22:45:08 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute2.internal (MEProxy); Thu, 11 Jul 2024 22:45:08 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sent.com; h=cc :cc:content-transfer-encoding:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:reply-to:subject:subject:to:to; s=fm2; t=1720752308; x=1720838708; bh=to4wnXa/PHhQKWIeEkU0XgFAJrxVOY/Z7C32sHOwamo=; b= nZefoTlKwQe9WlVRNw7pzFUFeLBTgJkzPIKJJcV7IkC8gOOEOkpF/+sUO4S9C4Jk 1C2FaybJ4DXMwA7Pez1e80TZ3vZc8awY36g8T3RuZtpvx92qWbrUohOL6q1juGt5 5ZXBFk0mcJsAFUMM+IkE4Rjs8GjRRdll7kSkl+v6VlXTMr3wPoPl6TMEp5VY3NfV WEBOxvHbTZSM/rFmP/EhwvWIP5kXhcFPP0HV/6EOI0mJq5VQHyfUQBWIRC8/wiiG KTjRsswYWjfhH6w/2cZ1cagkZx43LbrJAHHgUPP0WaqqEJcVkKE+R8C/EYAFiCA6 KdNztpxOHA3QrIjFj5GURA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:reply-to:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm2; t=1720752308; x= 1720838708; bh=to4wnXa/PHhQKWIeEkU0XgFAJrxVOY/Z7C32sHOwamo=; b=M TIp+Eg20hq6uumef+91PiN0qYVH0tM/c8uB8w49KQCgTFTGrGtWPWdZe15NWsy6n T0AA5SOVHVb9WJKmUZoI4mmIHgcTavU/OFrpxgZdneIAOfE2kwsFTLxBoj3xq7ql UwC/ALDV6I/LMKPKwn7XUD1g31YN8bzInB5V2FdVVBX54OGWcLIYfuyyd1nOybWL 8bI1hXYwvT+OMpNzRJQuWYHV74FdETCbCSSSx8Fy352mPoWFqzigFMv0TC0pkTV0 hH3VPQXGcus4psENBuTi/tiMXGh/nqRVyAbipDYxmCLqpQ0TFdz3s14ZXIq9Vy/n p/NKrviCBDq2kFrzFVFgg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeftddrfeehgdeigecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvfevufffkffojghfrhgggfestdekredtredttdenucfhrhhomhepkghiucgj rghnuceoiihirdihrghnsehsvghnthdrtghomheqnecuggftrfgrthhtvghrnhepudfgfe elhfekiefgtedvudfghfduffefieefffejvefhlefhhffgkedtfeevhfevnecuvehluhhs thgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepiihirdihrghnsehsvg hnthdrtghomh X-ME-Proxy: Feedback-ID: iccd040f4:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 11 Jul 2024 22:45:07 -0400 (EDT) From: Zi Yan To: David Hildenbrand , "Huang, Ying" , linux-mm@kvack.org Cc: Zi Yan , Andrew Morton , Baolin Wang , linux-kernel@vger.kernel.org Subject: [RFC PATCH 2/3] memory tiering: introduce folio_has_cpupid() check Date: Thu, 11 Jul 2024 22:44:54 -0400 Message-ID: <20240712024455.163543-3-zi.yan@sent.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240712024455.163543-1-zi.yan@sent.com> References: <20240712024455.163543-1-zi.yan@sent.com> Reply-To: Zi Yan MIME-Version: 1.0 X-Stat-Signature: rufu83g77azqufgksrjz3tyojxtbzc9d X-Rspamd-Queue-Id: 9360940013 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1720752309-983918 X-HE-Meta: U2FsdGVkX1/KbiI5T4K5WNIG9v8KW3/r/MICa3OYpulqcW60Gkit1APHr1Lr+DrSBdwRrkoH7k3lDw0Ogz520UJKnapG/z6LVlblzIQKt8nJGRVfRqZcee5zTK/asi9/L8MUfvlSycPVFxiTUQMaaYFaChyIpP3LeaRUu+yuKgzT6cTsvB+F1cx2shmh1r27kDe9i7r3ePTusgpHaLqR2vd6y/8sMIk5DVsh2hcjQ+YNQyx49yXdVO/FopelEeyRRKbfvwxKw/MjnZUsMgYbdQ0NO714oJ94xUzpdzgaoAsNTkx7Y7tGJixaI34GrDXlKWV4EoopNLfcgRp+42NF223z2s7AQhDV4PUZhOyKfPipaA7KE9wBB0Dp2D6EVR6ZLBHbzCgj83etJuMZO6lLd2CFeoAC6lZMN3YLFtkzI0mco7hpYeEtEYNJOIqkhhfRKO6PGsQRLO6rkpD3fE2rd77gxicaWKUrCbX7qmAkBTKrnXOubKL0CjWoKgrs9UMhfNkSWOz71iF3AR49V1Y4POj4ouPHTwpJ3M8heK7Dogwlpk3G+MfZ+1nkfJmLR8AiLM65oeej4Ciy/wMHBUtaBcWdymFegUBHP3ox0NHqmBBbQYai5xWx0zmhPBfZKfNIuKirxZFSgj/xP2kWKBZGCXs9mAIOD/HpHi5EIcn7E1+HKQz57aDMoBniKAr3bGCDQEqYVPxLK2xTYUKbFmiCDQMW9T7T0jHEsb2IjpsT76X79FiGzarDxogLtUG1naw3NT261cwp1tJkrWk/STrRUp+k/SrPlTr/dFu5I9NChNb0QocNyqf5dnfkycQ37a0VvRI+IXtgD17tYUvT5Qwn0EP0Rc5sx69jW0HSOnU9YoIX0Oii8uN11qDRGZXYLSPk3bSMRe7q2hxUQsnQz/J2Px4uj+pfsDosE1F6Jv0YwlGy9UHP1onoibb4hkVTLVr/dxrX+dp5iCC/1yR6ob0 1DV00PBP ZYp/99pm80JV3IT7wMhlBvI2HGStX+NOHDHMyQD0t+245rYwVC9inedm33u7hf9zH1AYE6FPXLGcSx1FUQ6Sp1BERNe/SY8ro1Ydr3Hd5AJDCA8SC0ips1I4Gh8YDAT0zFhZ3eq1hA/pV/wcW4W9OvfcgZVX1xMbOrkNOpUqOjp9JsFy8pRQrNuGAWwLiWWqQSlHIskbb/A7a4v3Y1JqHIC4DhsCVF1iHFo+zqevJtXCy4sAf58bv8LFy1gfxsbYUkI9yQ+Sp9kzgLVelfjwMAsLIrp/Ey9mZG5C4hXLgsUp/4bPMdk4Z5kUCMAhnaK/l8KFds8OxubCPUdHiaHXPqJLFP73ecZmqZ0KlU0oHBE+R6EhrzkaBjfTcnX/kb55vSNsQCznGPWPiEx5MR8+lp3kcz1d8HojPMvy9t9qgY3FX0KiDC+DXEt13g3IwaN0XhNU1JNPA4Ie/Njc5Cf7MrfIk0fnEp3nMbfGzWNkm9IqKSX0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Zi Yan Instead of open coded check for if memory tiering mode is on and a folio is in the top tier memory, use a function to encapsulate the check. Signed-off-by: Zi Yan Reviewed-by: "Huang, Ying" --- include/linux/memory-tiers.h | 8 ++++++++ kernel/sched/fair.c | 3 +-- mm/huge_memory.c | 6 ++---- mm/memory-tiers.c | 17 +++++++++++++++++ mm/memory.c | 3 +-- mm/mprotect.c | 3 +-- 6 files changed, 30 insertions(+), 10 deletions(-) diff --git a/include/linux/memory-tiers.h b/include/linux/memory-tiers.h index 0dc0cf2863e2..10c127d461c4 100644 --- a/include/linux/memory-tiers.h +++ b/include/linux/memory-tiers.h @@ -73,6 +73,10 @@ static inline bool node_is_toptier(int node) } #endif + +bool folio_has_cpupid(struct folio *folio); + + #else #define numa_demotion_enabled false @@ -151,5 +155,9 @@ static inline struct memory_dev_type *mt_find_alloc_memory_type(int adist, static inline void mt_put_memory_types(struct list_head *memory_types) { } +static inline bool folio_has_cpupid(struct folio *folio) +{ + return true; +} #endif /* CONFIG_NUMA */ #endif /* _LINUX_MEMORY_TIERS_H */ diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 8a5b1ae0aa55..03de808cb3cc 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -1840,8 +1840,7 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio, * The pages in slow memory node should be migrated according * to hot/cold instead of private/shared. */ - if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING && - !node_is_toptier(src_nid)) { + if (!folio_has_cpupid(folio)) { struct pglist_data *pgdat; unsigned long rate_limit; unsigned int latency, th, def_th; diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 07d9dde4ca33..8c11d6da4b36 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1705,8 +1705,7 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) * For memory tiering mode, cpupid of slow memory page is used * to record page access time. So use default value. */ - if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) || - node_is_toptier(nid)) + if (folio_has_cpupid(folio)) last_cpupid = folio_last_cpupid(folio); target_nid = numa_migrate_prep(folio, vmf, haddr, nid, &flags); if (target_nid == NUMA_NO_NODE) @@ -2059,8 +2058,7 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, toptier) goto unlock; - if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING && - !toptier) + if (!folio_has_cpupid(folio)) folio_xchg_access_time(folio, jiffies_to_msecs(jiffies)); } diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c index 4775b3a3dabe..7f0360d4e3a0 100644 --- a/mm/memory-tiers.c +++ b/mm/memory-tiers.c @@ -6,6 +6,7 @@ #include #include #include +#include #include "internal.h" @@ -50,6 +51,22 @@ static const struct bus_type memory_tier_subsys = { .dev_name = "memory_tier", }; +/** + * folio_has_cpupid - check if a folio has cpupid information + * @folio: folio to check + * + * folio's _last_cpupid field is repurposed by memory tiering. In memory + * tiering mode, cpupid of slow memory folio (not toptier memory) is used to + * record page access time. + * + * Return: the folio _last_cpupid is used as cpupid + */ +bool folio_has_cpupid(struct folio *folio) +{ + return !(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) || + node_is_toptier(folio_nid(folio)); +} + #ifdef CONFIG_MIGRATION static int top_tier_adistance; /* diff --git a/mm/memory.c b/mm/memory.c index dceb62f3fa34..96c2f5b3d796 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5344,8 +5344,7 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) * For memory tiering mode, cpupid of slow memory page is used * to record page access time. So use default value. */ - if ((sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) && - !node_is_toptier(nid)) + if (!folio_has_cpupid(folio)) last_cpupid = (-1 & LAST_CPUPID_MASK); else last_cpupid = folio_last_cpupid(folio); diff --git a/mm/mprotect.c b/mm/mprotect.c index 222ab434da54..787c3c2bf1b6 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -161,8 +161,7 @@ static long change_pte_range(struct mmu_gather *tlb, if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_NORMAL) && toptier) continue; - if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING && - !toptier) + if (!folio_has_cpupid(folio)) folio_xchg_access_time(folio, jiffies_to_msecs(jiffies)); } From patchwork Fri Jul 12 02:44:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 13731253 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7A9B5C3DA4B for ; Fri, 12 Jul 2024 02:45:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CF4006B009B; Thu, 11 Jul 2024 22:45:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C7FCB6B009C; Thu, 11 Jul 2024 22:45:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A5B976B009D; Thu, 11 Jul 2024 22:45:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 828236B009B for ; Thu, 11 Jul 2024 22:45:13 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 3DDFDC0821 for ; Fri, 12 Jul 2024 02:45:13 +0000 (UTC) X-FDA: 82329558906.28.8396A56 Received: from fout6-smtp.messagingengine.com (fout6-smtp.messagingengine.com [103.168.172.149]) by imf24.hostedemail.com (Postfix) with ESMTP id 37132180015 for ; Fri, 12 Jul 2024 02:45:11 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=sent.com header.s=fm2 header.b=UzhrMi6K; dkim=pass header.d=messagingengine.com header.s=fm2 header.b="A t1TRM8"; spf=pass (imf24.hostedemail.com: domain of zi.yan@sent.com designates 103.168.172.149 as permitted sender) smtp.mailfrom=zi.yan@sent.com; dmarc=pass (policy=none) header.from=sent.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720752294; a=rsa-sha256; cv=none; b=xZja6sR14rAiZeukrc5KGTqtqzzYl/uT8YSS5KgtgW85KWT8HBtMtfxYVM3/rBMD82gnfg 4vwobAZnNTf/93hLAY6ksnSKQHkhhnG3E0v8S57WUT4EEtyaSM+dfk7TD4dNTUq2b9/ugM Zk2vD0N2hHup0r7EjfTvra1GI11B+1w= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=sent.com header.s=fm2 header.b=UzhrMi6K; dkim=pass header.d=messagingengine.com header.s=fm2 header.b="A t1TRM8"; spf=pass (imf24.hostedemail.com: domain of zi.yan@sent.com designates 103.168.172.149 as permitted sender) smtp.mailfrom=zi.yan@sent.com; dmarc=pass (policy=none) header.from=sent.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720752294; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=qOLH68BAMXmJaL/17ag+VBuU12QsIrePjVshhM6Mfe0=; b=RTt8jAK0tB0gphpcBQfRuTOlGMftYf3Hthd4I+l8xKQf3DriMc6qmPiO2yiyt2KGfCfzUh AOIBthQYYsiexru9N+U9NItnyX2v/WLut2yTLISdRzn5uxbGM4qh4tFZv9O+WEd9JdGXfr LiXI+IdZMNxh5i5jNmdQdBc/gRMRpqM= Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailfout.nyi.internal (Postfix) with ESMTP id 7F0D11388636; Thu, 11 Jul 2024 22:45:10 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute2.internal (MEProxy); Thu, 11 Jul 2024 22:45:10 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sent.com; h=cc :cc:content-transfer-encoding:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:reply-to:subject:subject:to:to; s=fm2; t=1720752310; x=1720838710; bh=qOLH68BAMXmJaL/17ag+VBuU12QsIrePjVshhM6Mfe0=; b= UzhrMi6KRCDGISUUfYy8VY1O+HNyLY+YsDCfn7vyNqLyKNGyvCiSp9lSx4DIlrdH Wz+A0c1HnuV444dIAMSJgLwkaNNhtz4lhfpX+gTZnqbMokGPCg7uxg8j0zaAtrnr EWxIehnk0SV9nHHyDxbY5U7RiN7QcbagxPxt/rOmTQVH+1epuEnFG0iYOIUOCVfG XgncIdsqnEve7p5ol3AcczTUYpTAXmqf2ow/fkmJaK4zZhrSsewoCoxNgq1mcoQf /fqXqJW4PGGdUQZiIhi3ZDvNr/Zo5QpZ7OaHn12I9iLFunXjwZGLOVapaaxzpiHH fxJGpWsU1BKUjU5L/bydTQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:reply-to:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm2; t=1720752310; x= 1720838710; bh=qOLH68BAMXmJaL/17ag+VBuU12QsIrePjVshhM6Mfe0=; b=A t1TRM8pYiC+yDef6ArSXq6vIeVDSlwZcsQDCaVLGNIxE4y0f/Vhbnx2bSXtCH/Ki Wui7qzOwtG6EfE/TvSOU8G+iFjPFvFISE5JCp4PJ1M21fMSKoKD9mrRaTxEhIj6j 6O0/Xlo1boRKbeFqZeKuidvP05NIVdUBWzKO7qRECwyYbtGYqh8OHWnyflHUMvZ5 q7nXoI7kEY4k+bQmvSGAbpfXR/qck3i9ctyTJmlrh2/coeF06LvghZoDLzpyDnhG qZQ6lrNiySmt77S9XhQ8BJBeKfv1GurePGryUThlB7g7ALS7nA0QzLZ0Lybi6N/P /VLW4r2XyjIn89dqDqGKw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeftddrfeehgdeigecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvfevufffkffojghfrhgggfestdekredtredttdenucfhrhhomhepkghiucgj rghnuceoiihirdihrghnsehsvghnthdrtghomheqnecuggftrfgrthhtvghrnhepudfgfe elhfekiefgtedvudfghfduffefieefffejvefhlefhhffgkedtfeevhfevnecuvehluhhs thgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepiihirdihrghnsehsvg hnthdrtghomh X-ME-Proxy: Feedback-ID: iccd040f4:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 11 Jul 2024 22:45:08 -0400 (EDT) From: Zi Yan To: David Hildenbrand , "Huang, Ying" , linux-mm@kvack.org Cc: Zi Yan , Andrew Morton , Baolin Wang , linux-kernel@vger.kernel.org Subject: [RFC PATCH 3/3] mm/migrate: move common code to numa_migrate_check (was numa_migrate_prep) Date: Thu, 11 Jul 2024 22:44:55 -0400 Message-ID: <20240712024455.163543-4-zi.yan@sent.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240712024455.163543-1-zi.yan@sent.com> References: <20240712024455.163543-1-zi.yan@sent.com> Reply-To: Zi Yan MIME-Version: 1.0 X-Stat-Signature: 6xejnd17nu3mwuwqra7bi3ohbxo9fjsg X-Rspamd-Queue-Id: 37132180015 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1720752311-882798 X-HE-Meta: U2FsdGVkX1/oykv1SGVKXpj4SCM3v0fNFD43Snj2DpMqJy0q5Omo3sFvg669O43QpovLdjJJTnpRqGzUySwOFsui76URq9To1iTu/YKXw6nrJKESZOb2X3aDY641O0IsG60jHDf06dv0/f4f/5kX7RDJtzn4RwrGgrIksPfGuwb/Jr3USxFENcijvWaivaBlPkIhc6Uff3s+x6n2Df9tqLhMVJIDa/fTStFCpyiIWR7VJVYrtWITq0f+nups3Nrc5mHxGohqaVdrTpBlJegbDTTgxk/IzY008Z8DPVtBrqCOqninBQnC2DCEmDbO2y7RrNkppKTo6e2AhIcp6xYtVGEjy/iFQFu0LQsaXNBgG6y8C9SOq3Kr9Etzsa8ZFFRZAM1m6bjeYB+I0x7+y2GuTZjSifxGRC9hiWoh9xP4seWqshEOQ6hv0D2+szILbyFi4woFvhH4/btw9g8bRZ4FicmLJmaOjcw0COJSf1dUGtlPM49/B7S2pSSteUm7KkGlTl8BwYBRkC5aNN4n9HQFr/yn9xdX147LxPaloMISRqdctNILSKrOUABpqIt5kZIAZkco2FoPPpmDekM5SpzEgrsc/eiv86fuLaxuPBeZsZRgIzyYjJEapSYNxOTjznoJbiNAe6Eifaa6gqMkZuQGkSu7Nitv4ODRVqQ2j2HBxiIgd0NxzGAf9zlib5kfk4YZh1KIcDLl5SSL2IS6J4tZjmE4NfJmYfvTcdfkHlYz1GOcoflbFjbOpDiWvO1sTsLoXD4trELx54FO+pbjRl1yKV0K3EyFQWIOP414k3PA6Lryxjq36aK/B8moju8vKMSLJj7ESYhB861xAqpEjkVwWAuPKG0kid66NrmUA2V+CkDwxhZpQjohxvcDLT+sJdfrmTAGie/VPGvY82Q6PuNE/ZNlWdovKXsMB9AFi+agMHUS0N71pn6jSrHhZsWwq5mh/OUOy0qnZQY7/VJUY/e rALnrOQs 0BcUta/8VNC6YBlXcY8fqWPsypl3fQzLem6oLwe7DZ+RKLY/gvkU8KdnljQ/90IdEFKqIgIS3jPb4MISkPYG8YeucFUCkZDL1/8lYkAIUDCkvG2wXjXIRJo1jjt+/j21dcXUryPR60JSI5+ZF5iUIu1skrpf2eH733vNSzTpEdNjjAyaUBfNZi3hzSArXYH2LZwDBNL7iqtB/B1CgUwXUvPEgMOY8GQgzOfC8NPhMqBBeusHB88Q/ogkLCScVaBziRdqb7IL5CS7gQtUJguKUJm0i4N/ZMgNGLs6JL0ttPJ9GahDsqOYDy991n4zVurGueuCPobMHcX2+0cwybKjVl6eqbIB+2P08wH6OMJQ1N09b8/XTkQFX+PO/TBO1R/LTktkJ0FK4xktoodwBFQRaJ2K9x2XHnvWIODjJIFtCPqogT4hP3FqnkXnb7QzF78c4QzcSDAe8sZLFf/M= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Zi Yan do_numa_page() and do_huge_pmd_numa_page() share a lot of common code. To reduce redundancy, move common code to numa_migrate_prep() and rename the function to numa_migrate_check() to reflect its functionality. There is some code difference between do_numa_page() and do_huge_pmd_numa_page() before the code move: 1. do_huge_pmd_numa_page() did not check shared folios to set TNF_SHARED. 2. do_huge_pmd_numa_page() did not check and skip zone device folios. Signed-off-by: Zi Yan --- mm/huge_memory.c | 28 ++++++----------- mm/internal.h | 5 +-- mm/memory.c | 81 +++++++++++++++++++++++------------------------- 3 files changed, 52 insertions(+), 62 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 8c11d6da4b36..66d67d13e0dc 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1670,10 +1670,10 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) pmd_t pmd; struct folio *folio; unsigned long haddr = vmf->address & HPAGE_PMD_MASK; - int nid = NUMA_NO_NODE; - int target_nid, last_cpupid = (-1 & LAST_CPUPID_MASK); + int target_nid = NUMA_NO_NODE; + int last_cpupid = (-1 & LAST_CPUPID_MASK); bool writable = false; - int flags = 0; + int flags = 0, nr_pages; vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd); if (unlikely(!pmd_same(oldpmd, *vmf->pmd))) { @@ -1693,21 +1693,13 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) writable = true; folio = vm_normal_folio_pmd(vma, haddr, pmd); - if (!folio) + if (!folio || folio_is_zone_device(folio)) goto out_map; - /* See similar comment in do_numa_page for explanation */ - if (!writable) - flags |= TNF_NO_GROUP; + nr_pages = folio_nr_pages(folio); - nid = folio_nid(folio); - /* - * For memory tiering mode, cpupid of slow memory page is used - * to record page access time. So use default value. - */ - if (folio_has_cpupid(folio)) - last_cpupid = folio_last_cpupid(folio); - target_nid = numa_migrate_prep(folio, vmf, haddr, nid, &flags); + target_nid = numa_migrate_check(folio, vmf, haddr, writable, + &flags, &last_cpupid); if (target_nid == NUMA_NO_NODE) goto out_map; if (migrate_misplaced_folio_prepare(folio, vma, target_nid)) { @@ -1720,8 +1712,8 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) if (!migrate_misplaced_folio(folio, vma, target_nid)) { flags |= TNF_MIGRATED; - nid = target_nid; } else { + target_nid = NUMA_NO_NODE; flags |= TNF_MIGRATE_FAIL; vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd); if (unlikely(!pmd_same(oldpmd, *vmf->pmd))) { @@ -1732,8 +1724,8 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) } out: - if (nid != NUMA_NO_NODE) - task_numa_fault(last_cpupid, nid, HPAGE_PMD_NR, flags); + if (target_nid != NUMA_NO_NODE) + task_numa_fault(last_cpupid, target_nid, nr_pages, flags); return 0; diff --git a/mm/internal.h b/mm/internal.h index b4d86436565b..7782b7bb3383 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1217,8 +1217,9 @@ void vunmap_range_noflush(unsigned long start, unsigned long end); void __vunmap_range_noflush(unsigned long start, unsigned long end); -int numa_migrate_prep(struct folio *folio, struct vm_fault *vmf, - unsigned long addr, int page_nid, int *flags); +int numa_migrate_check(struct folio *folio, struct vm_fault *vmf, + unsigned long addr, bool writable, + int *flags, int *last_cpupid); void free_zone_device_folio(struct folio *folio); int migrate_device_coherent_page(struct page *page); diff --git a/mm/memory.c b/mm/memory.c index 96c2f5b3d796..a252c0f13755 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5209,16 +5209,42 @@ static vm_fault_t do_fault(struct vm_fault *vmf) return ret; } -int numa_migrate_prep(struct folio *folio, struct vm_fault *vmf, - unsigned long addr, int page_nid, int *flags) +int numa_migrate_check(struct folio *folio, struct vm_fault *vmf, + unsigned long addr, bool writable, + int *flags, int *last_cpupid) { struct vm_area_struct *vma = vmf->vma; + /* + * Avoid grouping on RO pages in general. RO pages shouldn't hurt as + * much anyway since they can be in shared cache state. This misses + * the case where a mapping is writable but the process never writes + * to it but pte_write gets cleared during protection updates and + * pte_dirty has unpredictable behaviour between PTE scan updates, + * background writeback, dirty balancing and application behaviour. + */ + if (!writable) + *flags |= TNF_NO_GROUP; + + /* + * Flag if the folio is shared between multiple address spaces. This + * is later used when determining whether to group tasks together + */ + if (folio_likely_mapped_shared(folio) && (vma->vm_flags & VM_SHARED)) + *flags |= TNF_SHARED; + + /* + * For memory tiering mode, cpupid of slow memory page is used + * to record page access time. + */ + if (folio_has_cpupid(folio)) + *last_cpupid = folio_last_cpupid(folio); + /* Record the current PID acceesing VMA */ vma_set_access_pid_bit(vma); count_vm_numa_event(NUMA_HINT_FAULTS); - if (page_nid == numa_node_id()) { + if (folio_nid(folio) == numa_node_id()) { count_vm_numa_event(NUMA_HINT_FAULTS_LOCAL); *flags |= TNF_FAULT_LOCAL; } @@ -5284,12 +5310,11 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) { struct vm_area_struct *vma = vmf->vma; struct folio *folio = NULL; - int nid = NUMA_NO_NODE; + int target_nid = NUMA_NO_NODE; bool writable = false, ignore_writable = false; bool pte_write_upgrade = vma_wants_manual_pte_write_upgrade(vma); - int last_cpupid; - int target_nid; - pte_t pte, old_pte; + int last_cpupid = (-1 & LAST_CPUPID_MASK); + pte_t pte, old_pte = vmf->orig_pte; int flags = 0, nr_pages; /* @@ -5297,10 +5322,7 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) * table lock, that its contents have not changed during fault handling. */ spin_lock(vmf->ptl); - /* Read the live PTE from the page tables: */ - old_pte = ptep_get(vmf->pte); - - if (unlikely(!pte_same(old_pte, vmf->orig_pte))) { + if (unlikely(!pte_same(old_pte, *vmf->pte))) { pte_unmap_unlock(vmf->pte, vmf->ptl); goto out; } @@ -5320,35 +5342,10 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) if (!folio || folio_is_zone_device(folio)) goto out_map; - /* - * Avoid grouping on RO pages in general. RO pages shouldn't hurt as - * much anyway since they can be in shared cache state. This misses - * the case where a mapping is writable but the process never writes - * to it but pte_write gets cleared during protection updates and - * pte_dirty has unpredictable behaviour between PTE scan updates, - * background writeback, dirty balancing and application behaviour. - */ - if (!writable) - flags |= TNF_NO_GROUP; - - /* - * Flag if the folio is shared between multiple address spaces. This - * is later used when determining whether to group tasks together - */ - if (folio_likely_mapped_shared(folio) && (vma->vm_flags & VM_SHARED)) - flags |= TNF_SHARED; - - nid = folio_nid(folio); nr_pages = folio_nr_pages(folio); - /* - * For memory tiering mode, cpupid of slow memory page is used - * to record page access time. So use default value. - */ - if (!folio_has_cpupid(folio)) - last_cpupid = (-1 & LAST_CPUPID_MASK); - else - last_cpupid = folio_last_cpupid(folio); - target_nid = numa_migrate_prep(folio, vmf, vmf->address, nid, &flags); + + target_nid = numa_migrate_check(folio, vmf, vmf->address, writable, + &flags, &last_cpupid); if (target_nid == NUMA_NO_NODE) goto out_map; if (migrate_misplaced_folio_prepare(folio, vma, target_nid)) { @@ -5362,9 +5359,9 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) /* Migrate to the requested node */ if (!migrate_misplaced_folio(folio, vma, target_nid)) { - nid = target_nid; flags |= TNF_MIGRATED; } else { + target_nid = NUMA_NO_NODE; flags |= TNF_MIGRATE_FAIL; vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); @@ -5378,8 +5375,8 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) } out: - if (nid != NUMA_NO_NODE) - task_numa_fault(last_cpupid, nid, nr_pages, flags); + if (target_nid != NUMA_NO_NODE) + task_numa_fault(last_cpupid, target_nid, nr_pages, flags); return 0; out_map: /*