From patchwork Wed Mar 20 01:45:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 13597183 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8F081C54E68 for ; Wed, 20 Mar 2024 01:45:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 043B96B007B; Tue, 19 Mar 2024 21:45:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F354D6B0082; Tue, 19 Mar 2024 21:45:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DD8136B0083; Tue, 19 Mar 2024 21:45:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id CD6166B007B for ; Tue, 19 Mar 2024 21:45:26 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 7626BA1966 for ; Wed, 20 Mar 2024 01:45:26 +0000 (UTC) X-FDA: 81915725052.26.192E9FB Received: from fhigh6-smtp.messagingengine.com (fhigh6-smtp.messagingengine.com [103.168.172.157]) by imf01.hostedemail.com (Postfix) with ESMTP id 589634000C for ; Wed, 20 Mar 2024 01:45:24 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=sent.com header.s=fm2 header.b="M Zgnlph"; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=vaMgqZNG; spf=pass (imf01.hostedemail.com: domain of zi.yan@sent.com designates 103.168.172.157 as permitted sender) smtp.mailfrom=zi.yan@sent.com; dmarc=pass (policy=none) header.from=sent.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710899124; a=rsa-sha256; cv=none; b=BHR6czLxG822Q9tPzPASVksLb2M+xUbk5QuAAPDjJtXa5KbeZ6Xw6e2vEzfLv578t1EDGO aSXyXVj6dbC8u0fOb8JrqkUC5XcdmM/6Zd1rMcDkrhylPwx3iAOs5E077gr6PIeN9+lH2c VOBf86WTleF1vTo2dbUwSvH7JWSiclc= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=sent.com header.s=fm2 header.b="M Zgnlph"; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=vaMgqZNG; spf=pass (imf01.hostedemail.com: domain of zi.yan@sent.com designates 103.168.172.157 as permitted sender) smtp.mailfrom=zi.yan@sent.com; dmarc=pass (policy=none) header.from=sent.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710899124; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=OllGBd/e2P4hWoSOipzEy0ACyR7T+q17LVRPTxf0DMQ=; b=vWtsWJmOltSDbNeuGTTXSD6iwpPoAixePnkJIKKEUE0bcRcMHuiH/Wi9rEdZEaQbQof/P8 9Az7ZJruHKllrf47NNkxBdp7yWAu7d3CNVcm92Vq10ODHw4G2lhqQV0Y8pzEbUQbI3RMAE EO2/mtJxFWi1N7C5BOH2Xip1FduxzW4= Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailfhigh.nyi.internal (Postfix) with ESMTP id 8B156114011F; Tue, 19 Mar 2024 21:45:23 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute5.internal (MEProxy); Tue, 19 Mar 2024 21:45:23 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sent.com; h=cc :cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:message-id:mime-version:reply-to :reply-to:subject:subject:to:to; s=fm2; t=1710899123; x= 1710985523; bh=OllGBd/e2P4hWoSOipzEy0ACyR7T+q17LVRPTxf0DMQ=; b=M ZgnlphEblWkAjWrxCxWUh6gDkJ5GAShRjT40ftx3ETe8GbZ0T7ZpKNkw8VkB8WH1 rfcXw6cW5XxutEDJBKUBvHQphk+w3I5bpnuAqN6v3fh4Slg3IzsCs2T80b5PaLJv 6TG3ADK1rCUdlWx5GD8ugznonTzm0Zea/8hu+m8JJ0ZVOipdO3qJRFSuf5Cr8yuW OCtCBkp+LMGYKABMJNs2/eWnRR6vFxJTZIFOrcaj+mIfDF/79uXURqxbMM/G2bdG r46AZ6xBsM6TXwRpM3O2SluGTbwRbLezWpBcL/ND9yrBeEV7RxLk3fblNULCFBMg oQRSFsrs60o0C9EeRZbYA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:message-id:mime-version:reply-to:reply-to :subject:subject:to:to:x-me-proxy:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm2; t=1710899123; x=1710985523; bh=O llGBd/e2P4hWoSOipzEy0ACyR7T+q17LVRPTxf0DMQ=; b=vaMgqZNGFpNqXGQoF y1LPPJaBw97LhzZ7VLVWAAYKoc96ems8x6Y+b5r0zig/5T/XYml4QJLFmubuKTTc /n4vLq12ylEYTomaJEK76+lBXv2+zdNd/+2P8bWLv55q+U790AB+gDBHIQY293J5 zeU7DdSUK5xPMXyXOes5WwqzwLNvsWkCvO9OrEFc/iBVQoTUhRaeqokz1yjuRQH8 Y5EyKcc/3XTdnKgfQ3XzHilAfxSVljDzqPzjRe3uRfFqf4aoX7kvv6jl+lNC1WAY DKKkTZ3AihZd4RprQ0aJ4u57Dc6EsicRnXuwXIWSXDgJjy0xWtYeIOK4tDLFfQUq x8ipg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrledvgddtlecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvfevufffkfforhggtgfgsehtkeertdertdejnecuhfhrohhmpegkihcujggr nhcuoeiiihdrhigrnhesshgvnhhtrdgtohhmqeenucggtffrrghtthgvrhhnpeetudevhe etheehieetiefhjeevjeeltdfgvdeiueeiudetffdtvedthfetvedtffenucffohhmrghi nhepkhgvrhhnvghlrdhorhhgnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpe hmrghilhhfrhhomhepiihirdihrghnsehsvghnthdrtghomh X-ME-Proxy: Feedback-ID: iccd040f4:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 19 Mar 2024 21:45:22 -0400 (EDT) From: Zi Yan To: linux-mm@kvack.org Cc: Zi Yan , Andrew Morton , "Matthew Wilcox (Oracle)" , Yang Shi , Huang Ying , "Kirill A . Shutemov" , Ryan Roberts , Baolin Wang , "Yin, Fengwei" , SeongJae Park , linux-kernel@vger.kernel.org Subject: [PATCH v4] mm/migrate: split source folio if it is on deferred split list Date: Tue, 19 Mar 2024 21:45:11 -0400 Message-ID: <20240320014511.306128-1-zi.yan@sent.com> X-Mailer: git-send-email 2.43.0 Reply-To: Zi Yan MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 589634000C X-Stat-Signature: 6uz3fkqh1twtzijjc1aaa3bgpm4f1it4 X-Rspam-User: X-HE-Tag: 1710899124-821891 X-HE-Meta: U2FsdGVkX1/tnW45Yql3Ob7vz8gZhuulzTWGb92pvcFnxb+24O9k+bnn0GA3AtjDGfJ0sqTi8T06Nr8zTfyg2ipgJNmuEIYcuHlnYxTdlnIALv0EbqsIsoVZ/ffI9G5eH0H/MetbDTmXXOonU2J2pOZMpXwS6vUIGBgFg8rO29q8NYDml4pVI+4JSmTy+LK1HZpynYw5ahthOsIV5y8iouOFHbxauB5+5UrkYHf6czoeZstsGLSa7zYoSAaqkhDWb3TJrJLyjoF6WsL62rp8DauiDhN0LRmTqurh9RCpdFI7SsqYp/TT0QCnq5Ds+m1/VbYiMqLCDsTlu03PS4BTub/j/yRLxVuzS5jfqCW4R4PoRSx64vC61DVX06eccTVA+bY2rO7RfUt0IMzIcW34pM1n4jJI7hLPHDORUzi2EohucRxgpzTthaGfQuhRppYxyPXVKdLpZrCq48gDJl9FAUgAqszzmniaiWHskl42icESX5RFPNW1wtMy6sWlotxsZ28a9BmmDLbquSBeFlgHsQL9v84xINUzxxdDwbF3cyMIJKDM05aqUSa2jYf/MT0e8Zn7Vx1lfQ1xP6yLbc1xdG3Tgomq4mO5eiXaIZyuzZDoVwPTwLOhAO2DZ14pyDUYMXGLPpRjBoIXmyTwHwYg4LevoB1T2NXD1PI6/U3nOECf9x4nafU84VV//kq+W3C2HBhiOTIJ+MuhTzAyQOLyRSaP155XuKvKtIW+P0689sQfi/vXcc/MD+BSggSZe6E42vShvi0c8D6TLuMYiqm6p+TlOm562JqGMLvTV6LsCMe8BlGPoFeoibNc/P+cLmXGkGtyDCE6jH/1/a71j+/zOH/5km2J0QNCboXlyxvAHmXe0HpRhsFR9uqv9LnSSWMsiaayPXK2x0aXd2FBO5KVbbNgHnHN5VQr5dIz+k0DCaDuu9BoV8h0To3DRliHfUip77LDS5C5Ub0b4yBAEm9 v7K2OpsK K8M2AWG83vBVlXM3SAngATovh4JA0nv7+fFMJ7aNiFEu5poRu1mr7S34/Us2ETYpLuuaqHuIJYsdKHbei3ebL/S+MJp/AnFAtwJUj0YdsHm5JIVf9qoBE+koR9dXKq4Ijtk4L5uUU2scEfP+ihjVmeQAEW1mvG/S+LsIQrSxcUHMCOATZBpsELk5ckudbnY5G1dy89qFSK6fIZorMwyZpY434cZZL+ASjfgYz2q2YB3jObKoS2kbq7r9992GxIyij/rJp1ITdeVoLKkWbgUnnQ3rNfcUMLIfDkm2TI9Y+7MDdRjaf3J5FwjNQK3MmUejiXt5mpf5lVsbjtWxhnXSJhvESy50lexzvs6d7yW9B7/Iy+2wOR92RMjyXsZdtA626zwGd1enN4OtGnDiwcSS2BjnG30ih1R38tEJ91FfHFY4wajtbNqFZ6fcEvapXctKEAuoKqkHscSxIL9yBX7Xk/NHnY39kn1CTV1VHw/zokDjLhcipd8x4DHT7gWuzWAscRTG1S0vI6whHpu6POgwGlMZLo45+MkHI9EOo X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Zi Yan If the source folio is on deferred split list, it is likely some subpages are not used. Split it before migration to avoid migrating unused subpages. Commit 616b8371539a6 ("mm: thp: enable thp migration in generic path") did not check if a THP is on deferred split list before migration, thus, the destination THP is never put on deferred split list even if the source THP might be. The opportunity of reclaiming free pages in a partially mapped THP during deferred list scanning is lost, but no other harmful consequence is present[1]. From v3: 1. Guarded deferred list code behind CONFIG_TRANSPARENT_HUGEPAGE to avoid compilation error (per SeongJae Park). From v2: 1. Split the source folio instead of migrating it (per Matthew Wilcox)[2]. From v1: 1. Used dst to get correct deferred split list after migration (per Ryan Roberts). [1]: https://lore.kernel.org/linux-mm/03CE3A00-917C-48CC-8E1C-6A98713C817C@nvidia.com/ [2]: https://lore.kernel.org/linux-mm/Ze_P6xagdTbcu1Kz@casper.infradead.org/ Fixes: 616b8371539a ("mm: thp: enable thp migration in generic path") Signed-off-by: Zi Yan --- mm/huge_memory.c | 22 ----------------- mm/internal.h | 25 +++++++++++++++++++ mm/migrate.c | 62 +++++++++++++++++++++++++++++++++++++++--------- 3 files changed, 76 insertions(+), 33 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index b40bd9f3ead5..c77cedf45f3a 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -766,28 +766,6 @@ pmd_t maybe_pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma) return pmd; } -#ifdef CONFIG_MEMCG -static inline -struct deferred_split *get_deferred_split_queue(struct folio *folio) -{ - struct mem_cgroup *memcg = folio_memcg(folio); - struct pglist_data *pgdat = NODE_DATA(folio_nid(folio)); - - if (memcg) - return &memcg->deferred_split_queue; - else - return &pgdat->deferred_split_queue; -} -#else -static inline -struct deferred_split *get_deferred_split_queue(struct folio *folio) -{ - struct pglist_data *pgdat = NODE_DATA(folio_nid(folio)); - - return &pgdat->deferred_split_queue; -} -#endif - void folio_prep_large_rmappable(struct folio *folio) { if (!folio || !folio_test_large(folio)) diff --git a/mm/internal.h b/mm/internal.h index 85c3db43454d..56cf2051cb88 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1106,6 +1106,31 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmd, unsigned int flags); +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +#ifdef CONFIG_MEMCG +static inline +struct deferred_split *get_deferred_split_queue(struct folio *folio) +{ + struct mem_cgroup *memcg = folio_memcg(folio); + struct pglist_data *pgdat = NODE_DATA(folio_nid(folio)); + + if (memcg) + return &memcg->deferred_split_queue; + else + return &pgdat->deferred_split_queue; +} +#else +static inline +struct deferred_split *get_deferred_split_queue(struct folio *folio) +{ + struct pglist_data *pgdat = NODE_DATA(folio_nid(folio)); + + return &pgdat->deferred_split_queue; +} +#endif +#endif + + /* * mm/mmap.c */ diff --git a/mm/migrate.c b/mm/migrate.c index 73a052a382f1..e80cb0f46342 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1654,25 +1654,65 @@ static int migrate_pages_batch(struct list_head *from, /* * Large folio migration might be unsupported or - * the allocation might be failed so we should retry - * on the same folio with the large folio split + * the folio is on deferred split list so we should + * retry on the same folio with the large folio split * to normal folios. * * Split folios are put in split_folios, and * we will migrate them after the rest of the * list is processed. */ - if (!thp_migration_supported() && is_thp) { - nr_failed++; - stats->nr_thp_failed++; - if (!try_split_folio(folio, split_folios)) { - stats->nr_thp_split++; - stats->nr_split++; + if (is_thp) { + bool is_on_deferred_list = false; + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + /* + * Check without taking split_queue_lock to + * reduce locking overheads. The worst case is + * that if the folio is put on the deferred + * split list after the check, it will be + * migrated and not put back on the list. + * The migrated folio will not be split + * via shrinker during memory pressure. + */ + if (!data_race(list_empty(&folio->_deferred_list))) { + struct deferred_split *ds_queue; + unsigned long flags; + + ds_queue = + get_deferred_split_queue(folio); + spin_lock_irqsave(&ds_queue->split_queue_lock, + flags); + /* + * Only check if the folio is on + * deferred split list without removing + * it. Since the folio can be on + * deferred_split_scan() local list and + * removing it can cause the local list + * corruption. Folio split process + * below can handle it with the help of + * folio_ref_freeze(). + */ + is_on_deferred_list = + !list_empty(&folio->_deferred_list); + spin_unlock_irqrestore(&ds_queue->split_queue_lock, + flags); + } +#endif + if (!thp_migration_supported() || + is_on_deferred_list) { + nr_failed++; + stats->nr_thp_failed++; + if (!try_split_folio(folio, + split_folios)) { + stats->nr_thp_split++; + stats->nr_split++; + continue; + } + stats->nr_failed_pages += nr_pages; + list_move_tail(&folio->lru, ret_folios); continue; } - stats->nr_failed_pages += nr_pages; - list_move_tail(&folio->lru, ret_folios); - continue; } rc = migrate_folio_unmap(get_new_folio, put_new_folio,