From patchwork Thu Jul 4 04:31:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oscar Salvador X-Patchwork-Id: 13723200 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CC58CC30653 for ; Thu, 4 Jul 2024 04:32:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E28046B00AD; Thu, 4 Jul 2024 00:32:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D86206B00AE; Thu, 4 Jul 2024 00:32:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AEEED6B00AF; Thu, 4 Jul 2024 00:32:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 853DE6B00AD for ; Thu, 4 Jul 2024 00:32:21 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 3EB6DA0FC5 for ; Thu, 4 Jul 2024 04:32:21 +0000 (UTC) X-FDA: 82300798482.04.571AD1E Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) by imf29.hostedemail.com (Postfix) with ESMTP id 2447C120014 for ; Thu, 4 Jul 2024 04:32:18 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=suse.de; spf=pass (imf29.hostedemail.com: domain of osalvador@suse.de designates 195.135.223.131 as permitted sender) smtp.mailfrom=osalvador@suse.de ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720067519; a=rsa-sha256; cv=none; b=2b6fZyml7NswzHHh5GSfO3/VKsHBVrDHjcTBjdr1WodnpJzlbAlEYc8/bkke95xY++kDyH 1PQ2Q91MhdzJY163ow6ngYVmZc929u4mLdhQY0GW0k8tsqrYh0x6XmMgGmJvNb5ZNDb6db nAIT0o2zbvn7kZUBi7qyo2T8gBazCE4= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=suse.de; spf=pass (imf29.hostedemail.com: domain of osalvador@suse.de designates 195.135.223.131 as permitted sender) smtp.mailfrom=osalvador@suse.de ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720067519; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Uzs8V2oOmleXqErej25OEJn8fB6VvJqW1r2F1UQZ8IY=; b=So6ICzJITOkOEqeevkNWXDwF86lelWEsHa5Io/WpqtiraFpHtRJB9oKhiLTvxOJn39z444 Z7DRhnYHVMtahQNYyM278uGbqbBHiOzzv1+HL2cmLwMX5eCrDnmq76EIcpDL7SpjZSf9x7 rFifE7q/gGPSOCTE6YE9IgohS8OdX1o= Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id DB2C41FCFD; Thu, 4 Jul 2024 04:32:17 +0000 (UTC) Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id C2D9D13889; Thu, 4 Jul 2024 04:32:16 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id iBXOKtAlhmbnDAAAD6G6ig (envelope-from ); Thu, 04 Jul 2024 04:32:16 +0000 From: Oscar Salvador To: Andrew Morton Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Peter Xu , Muchun Song , David Hildenbrand , SeongJae Park , Miaohe Lin , Michal Hocko , Matthew Wilcox , Christophe Leroy , Oscar Salvador Subject: [PATCH 22/45] mm/mempolicy: Enable queue_folios_pmd to handle hugetlb vmas Date: Thu, 4 Jul 2024 06:31:09 +0200 Message-ID: <20240704043132.28501-23-osalvador@suse.de> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240704043132.28501-1-osalvador@suse.de> References: <20240704043132.28501-1-osalvador@suse.de> MIME-Version: 1.0 X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Action: no action X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 2447C120014 X-Stat-Signature: rb65t377f4owhkck5d9kdsgdg3wgxujz X-Rspam-User: X-HE-Tag: 1720067538-231927 X-HE-Meta: U2FsdGVkX192cj9jR4RLgEnekYX1y1NIgnxbhCMXl0ZTWtahW1B4tQDv2KLJ9HTuiUmkr9xxxIuhDQ0JtbkK0UVfwck/I7H07edkGwT1Kx52xEG0rOZyMBgpHC4QS0qjI/344jnb9tXyBVoiVG5t2x98yLJcdGoPXLcnp8VGJiuf+ben/Zsr4mlunjG5rFVUvGl7qI28yHF6QjwsICwdrcnCJ7TXNRqwgz1vBlYEtiGe9yYdkzJ6t5Z2ipx4IGV7qzbfvq5HgR4mvMCsH/3BGTMuH+/qD0+NkVVNu9sROpm8CrbMhJsLsK00NHtAtIBDsUoE4GRl+NGwAyEmx/n9oUKq5bFR7UtQuMIe8dnfy3UPFtd1Z45CxqVoi4/s1SVnoeW1MSXFdfUqEr9eZenmgg/77AxywHtqepNVOOSYxlTwtdYTzuMTwQadLrJmP1yVExmx85WxV0w03bJ2s9CcjaK1qhdeBvPilqSZRZ82KUmr60WqfxhCpdHWWvUPcTO19YcdDWDyhi4pivvuKIi2fAe/m5YHSTizaHGgNcdT4z7h0WQ3iaqQh6NOuRsdrEGdeqzvulC2I4KS2evQsQ5EMwNblk6o41AfAid0tQNOaMEdNN/oQZMO++ZSMyaZSRsvca0ck+0T3+i6kaXwxd905OnBOj2js4sb7rpCgMV52NFwkf76+UYFrsxptTx7+cDNuOB9qDAsm7m8xWOKZaw2XgjIfwf6fEfMXc9Uh+/utrDgG8XNnzeNz3els9hZSxPyX++KOjmhL7TsOBe0H7kqqfjZ0prhkhIYRDWBwen77ZUlblKL5vX4T1r18Zw5WJwJpKQnbR8oYCUKEij2fXf58txKwIVoPRtE+kxqpPZZdr6WO57esHqdFTpZMIxvwud5NkOOhnnMFwpnlKmnlNlQR7DcQJ8akVgrXqVpK2cthBSmXpBKEl8kAmcI+6X7ZYNQHx1bsrRO8Pvssf+mtdb Feydtx0d e62Yh2n2JY94pwL1zl5/P1mdlfqnI/fFceimtXJbEP4YxUqxQVv4NMNdIT9M85kQsoEN0gdINNI2G4KJliDS1KIcjW1UFV1l/wsacqJqTZsvFQ5XJJlTG+1IcmaB6pmfMXB+S/XlQnAUkeibDMdoTQLgD+PFi7kKOF98ZdP7APmr5tiM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: PMD-mapped hugetlb vmas will also reach smaps_pmd_entry. Add the required code so it knows how to handle those there. Signed-off-by: Oscar Salvador --- include/linux/mm_inline.h | 7 +++++++ mm/mempolicy.c | 42 ++++++++++++++++++++++++--------------- 2 files changed, 33 insertions(+), 16 deletions(-) diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h index 93e3eb86ef4e..521a001429d2 100644 --- a/include/linux/mm_inline.h +++ b/include/linux/mm_inline.h @@ -591,6 +591,13 @@ static inline bool vma_has_recency(struct vm_area_struct *vma) return true; } +static inline bool is_shared_pmd(pmd_t *pmd, struct vm_area_struct *vma) +{ + if (!is_vm_hugetlb_page(vma)) + return false; + return hugetlb_pmd_shared((pte_t *)pmd); +} + static inline spinlock_t *pmd_huge_lock(pmd_t *pmd, struct vm_area_struct *vma) { spinlock_t *ptl; diff --git a/mm/mempolicy.c b/mm/mempolicy.c index f8703feb68b7..5baf29da198c 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -455,7 +455,8 @@ static const struct mempolicy_operations mpol_ops[MPOL_MAX] = { }; static bool migrate_folio_add(struct folio *folio, struct list_head *foliolist, - unsigned long flags); + unsigned long flags, struct vm_area_struct *vma, + bool shared); static nodemask_t *policy_nodemask(gfp_t gfp, struct mempolicy *pol, pgoff_t ilx, int *nid); @@ -518,7 +519,8 @@ static void queue_folios_pmd(pmd_t *pmd, struct mm_walk *walk) return; if (!(qp->flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) || !vma_migratable(walk->vma) || - !migrate_folio_add(folio, qp->pagelist, qp->flags)) + !migrate_folio_add(folio, qp->pagelist, qp->flags, walk->vma, + is_shared_pmd(pmd, walk->vma))) qp->nr_failed++; } @@ -543,7 +545,7 @@ static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr, pte_t ptent; spinlock_t *ptl; - ptl = pmd_trans_huge_lock(pmd, vma); + ptl = pmd_huge_lock(pmd, vma); if (ptl) { queue_folios_pmd(pmd, walk); spin_unlock(ptl); @@ -598,7 +600,7 @@ static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr, } if (!(flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) || !vma_migratable(vma) || - !migrate_folio_add(folio, qp->pagelist, flags)) { + !migrate_folio_add(folio, qp->pagelist, flags, vma, false)) { qp->nr_failed++; if (strictly_unmovable(flags)) break; @@ -1025,8 +1027,11 @@ static long do_get_mempolicy(int *policy, nodemask_t *nmask, #ifdef CONFIG_MIGRATION static bool migrate_folio_add(struct folio *folio, struct list_head *foliolist, - unsigned long flags) + unsigned long flags, struct vm_area_struct *vma, + bool shared) { + bool ret = true; + bool is_hugetlb = is_vm_hugetlb_page(vma); /* * Unless MPOL_MF_MOVE_ALL, we try to avoid migrating a shared folio. * Choosing not to migrate a shared folio is not counted as a failure. @@ -1034,23 +1039,27 @@ static bool migrate_folio_add(struct folio *folio, struct list_head *foliolist, * See folio_likely_mapped_shared() on possible imprecision when we * cannot easily detect if a folio is shared. */ - if ((flags & MPOL_MF_MOVE_ALL) || !folio_likely_mapped_shared(folio)) { - if (folio_isolate_lru(folio)) { - list_add_tail(&folio->lru, foliolist); - node_stat_mod_folio(folio, - NR_ISOLATED_ANON + folio_is_file_lru(folio), - folio_nr_pages(folio)); - } else { + if ((flags & MPOL_MF_MOVE_ALL) || + (!folio_likely_mapped_shared(folio) && !shared)) { + if (is_hugetlb) + return isolate_hugetlb(folio, foliolist); + + ret = folio_isolate_lru(folio); + if (!ret) /* * Non-movable folio may reach here. And, there may be * temporary off LRU folios or non-LRU movable folios. * Treat them as unmovable folios since they can't be * isolated, so they can't be moved at the moment. */ - return false; - } + return ret; + + list_add_tail(&folio->lru, foliolist); + node_stat_mod_folio(folio, + NR_ISOLATED_ANON + folio_is_file_lru(folio), + folio_nr_pages(folio)); } - return true; + return ret; } /* @@ -1239,7 +1248,8 @@ static struct folio *alloc_migration_target_by_mpol(struct folio *src, #else static bool migrate_folio_add(struct folio *folio, struct list_head *foliolist, - unsigned long flags) + unsigned long flags, struct vm_area_struct *vma, + bool shared) { return false; }