From patchwork Mon Apr 3 20:18:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 13198731 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 74D31C77B6E for ; Mon, 3 Apr 2023 20:19:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 909696B0082; Mon, 3 Apr 2023 16:19:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8435F6B0083; Mon, 3 Apr 2023 16:19:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6BCC5900002; Mon, 3 Apr 2023 16:19:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 57EA56B0082 for ; Mon, 3 Apr 2023 16:19:10 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 1D49012073A for ; Mon, 3 Apr 2023 20:19:10 +0000 (UTC) X-FDA: 80641194060.15.8549960 Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by imf20.hostedemail.com (Postfix) with ESMTP id 074CF1C000D for ; Mon, 3 Apr 2023 20:19:07 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=sent.com header.s=fm2 header.b=bYdnJ5QB; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=Z9UKPLXQ; dmarc=pass (policy=none) header.from=sent.com; spf=pass (imf20.hostedemail.com: domain of zi.yan@sent.com designates 66.111.4.27 as permitted sender) smtp.mailfrom=zi.yan@sent.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680553148; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=aksTP0wsXOFiVzEDaAS1LD1g3uMK0049FZHtL5zgUuw=; b=aoub4Wxc4tj+rh6Ee8Hjas2SkG75L/Jfy6oaHlnKexrYCeqrq4897LMS9zSdSGmh34ouf2 HomhDERgfl0pi3lmWkFkInsxkH5qN8XL+r/kQx4YUnTYkpVPYCpWAjhN4a45HVA3aaTBeh LgnwBT8UuhPhWkamWe52t96v38j8Qe8= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=sent.com header.s=fm2 header.b=bYdnJ5QB; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=Z9UKPLXQ; dmarc=pass (policy=none) header.from=sent.com; spf=pass (imf20.hostedemail.com: domain of zi.yan@sent.com designates 66.111.4.27 as permitted sender) smtp.mailfrom=zi.yan@sent.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680553148; a=rsa-sha256; cv=none; b=DhyT4IP1Pgj0NaHDRLTgERDpBvla6+pFJguRvFWkXqkm1N51UGhFBVqxViYAFZt7jHoWBm w+Kve04+JpHR3BeooEGSmMOCgXOG5oIhR60vR09ZuWFKqsbsvCSuLmeYdYw60Ft7GpjO1o UzKqvnzHuXAgbwuOptbDUITvaPbgJeU= Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id 5773E5C01DB; Mon, 3 Apr 2023 16:19:07 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Mon, 03 Apr 2023 16:19:07 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sent.com; h=cc :cc:content-transfer-encoding:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:reply-to:sender:subject:subject:to:to; s=fm2; t= 1680553147; x=1680639547; bh=aksTP0wsXOFiVzEDaAS1LD1g3uMK0049FZH tL5zgUuw=; b=bYdnJ5QBNBJu3ZJXstuE2YWzj77VMJt3PPI5qvUwBTn5Mfvy6UP OvN6rJlshP4EJE/ppLHgOV+6woXIBSQn48IXJBGdXnJBpsEQHQJGoByyWYll0HHY IoT23jhtTOFUuPptabv2bP1LT88IASZKqFaa3iDH8/QpPiz7yYRMWoNrrTnkQnc/ izzYHzEcu1F887QvGyepRAojOkwFJA060KYDciqk34TXsBVs38lV/RBCMrSh+paD h28yFdOzDRiLcuwb3EGHDTULuUf6ngjiv3J1O1bhKeGEDX8cwCh1+p0HBDJYnSGd 2Lefjnz0M4pVUhTwN+EkqA2U1q8tcVXqx8Q== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:reply-to:sender:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; t= 1680553147; x=1680639547; bh=aksTP0wsXOFiVzEDaAS1LD1g3uMK0049FZH tL5zgUuw=; b=Z9UKPLXQGewR1AAUglbCW2XnniXMqwNYdZYYljH4El75rSI1PZS UisJFudWXq9mKF5YWxQGeDbsTI3kcylaQu8cD9jIMvrjR1JTZ4uYIdddaQoCOBT4 cevzys70blRonh504OZXmmz7vVCsXJohYm7my1N69wJOt9scgSZpx/TUlh6FygrX EyOzUs9NNS7PdWLiUTkeKNJYzCqQNIaKVca0dxWYTW7YSGbBFsaHV9RlLFtX1NF5 Cr87vOYzTa08mYpes6IbdLJilwUdy/lcke2pugO1hCalzuuLbtIi+9OfWf7mGZ0s erknzwyW48OM86MsafK1s2+8rzpSiNmYX5A== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrvdeijedgudegkecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefhvfevufffkffojghfrhgggfestdhqredtredttdenucfhrhhomhepkghi ucgjrghnuceoiihirdihrghnsehsvghnthdrtghomheqnecuggftrfgrthhtvghrnhepge eghedugfduuddvleehheetgeeltdetieevuefhffevkefhveeufeeiieejgedvnecuvehl uhhsthgvrhfuihiivgepudenucfrrghrrghmpehmrghilhhfrhhomhepiihirdihrghnse hsvghnthdrtghomh X-ME-Proxy: Feedback-ID: iccd040f4:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 3 Apr 2023 16:19:06 -0400 (EDT) From: Zi Yan To: "Matthew Wilcox (Oracle)" , Yang Shi , Yu Zhao , linux-mm@kvack.org Cc: Zi Yan , "Kirill A . Shutemov" , Ryan Roberts , =?utf-8?q?Michal_Koutn=C3=BD?= , Roman Gushchin , "Zach O'Keefe" , Andrew Morton , linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kselftest@vger.kernel.org Subject: [PATCH v3 5/7] mm: thp: split huge page to any lower order pages. Date: Mon, 3 Apr 2023 16:18:37 -0400 Message-Id: <20230403201839.4097845-6-zi.yan@sent.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230403201839.4097845-1-zi.yan@sent.com> References: <20230403201839.4097845-1-zi.yan@sent.com> Reply-To: Zi Yan MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 074CF1C000D X-Stat-Signature: 6exw6osywpre7jw4afouh67wafus483e X-HE-Tag: 1680553147-895859 X-HE-Meta: U2FsdGVkX1+dOUaJnQWRHmK2S9GQLqn+0hAefzsu5kaXqwRO78iGBJs5ApBWfK2Om6EpMLka2Mw9m9/eEePoo7m3768+y4GxQf3+nZifqUhSvzlrmTdAorsIJ0sKucE6l8ToucSIlDUsZ9IlUlEx0OFkrsQeiUrLepxKbvdJ4Xo3WSJBqQnNrb4ery+Qk7tWGkQTgPFBxQlJcCmj9rmpPfFDBayAfIR22zA3ALZ4zwh2DTqpyoegcS6KCf1OG3XVqVT9M8gjnl1MRATHmBWWMnGYgjo6sg83muB+QpvrMFQ/tHoOAw6f36gHi2ol6n2KLYVQJIMPvjwMsw6IScJNN8BTH7qxH57IwCzplNMpd1NbJpFl82z2jWD5YtDKm2dZBFavElkjWwxmhTOWx9AFbE0yF7uI2tudw2n/9Hzh0hxtp7QAZGS91ys3LAM1tNuFKrPFPDIBsw+qPZRRVa4hIPGT9LmgvoaAijXFxqXq9omzS516s4r6L/G6hM0RiuNlRa1oHhvx30srX7tTdNNMiQpd82A0nRNraMJpvhc+aMGXgTF6/gcAWqr5quZIo7+2TzvFjiTB1ToGugxei3aXFEa6kfICFTRPHkWm30ESiqnOjO9pDjpx+nBKxFvOZK+h5Dcb+JpL3ZLtqo9ngZKLZOhMQESpgbbOyTBP1U2b5fZujOusoaE3v71fOQMayqzmW4uVmLZ3bh68W/5BqbzvjFXylBm5fqtxOPiNY7QiMPDRchxR9ehGIdVN2wQBxdzi8LKdCf5o36A9lMetS78gH1mOc0o2g1/B/lIpmm8yzd5ic31/MKJ3Mr0fzKqhJtXe6rgPpkGYYjGRQqq5bLg1io10MX/0TubCoak/WcDs/kwQlfj4Y9jDnjHfK4tAQXOEURebRtjZPh1IMKuAag70ew9nOpUxji/suwxAWqlF6uL7g0IqWw1Ha4/mamW+MeUVp/sxulEJ47waTZq4UXs gkK80kCS AcTHIhv2DXQnMcUXGtslwnmWmyJORQH1uLTT0/D4hCy7/RHsXuhXc/6IRAp/7rSNoZBy+VfuW0iTAD5LHOPNP2ma/n+99VH4sNYRU20Xvhi9vlyQsKb5PCl+mYF984d/dcreL2XZJoqjvZkkFoFixSjYUDtrrce1YP9P1pUHXDkVN1I8Eq8uIiaAPzU3T8rcSH68rrMGAXg9qlRaVaVBrDVrB4iz7TmYnlxPvL2yUbt0OGoX8olhYVQibGsFgWlEYiio4Bzw6PtH/OmJTRSIHExuLPaQyn/P8Wk6pYwqS32ODT0QME89OvmVqCA78ntGJOxw+/PjLn2VW1C8T3SvAeje8jHRuMOONCZrO8VBFqcP9zOgyjQWwvbeXbZuhxIn/5oaVw+sKpPMa8oyqQt/GKOSlGvL0EQiLXI1x X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Zi Yan To split a THP to any lower order pages, we need to reform THPs on subpages at given order and add page refcount based on the new page order. Also we need to reinitialize page_deferred_list after removing the page from the split_queue, otherwise a subsequent split will see list corruption when checking the page_deferred_list again. It has many uses, like minimizing the number of pages after truncating a huge pagecache page. For anonymous THPs, we can only split them to order-0 like before until we add support for any size anonymous THPs. Signed-off-by: Zi Yan --- include/linux/huge_mm.h | 10 ++-- mm/huge_memory.c | 102 +++++++++++++++++++++++++++++----------- 2 files changed, 81 insertions(+), 31 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 20284387b841..32c91e1b59cd 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -147,10 +147,11 @@ void prep_transhuge_page(struct page *page); void free_transhuge_page(struct page *page); bool can_split_folio(struct folio *folio, int *pextra_pins); -int split_huge_page_to_list(struct page *page, struct list_head *list); +int split_huge_page_to_list_to_order(struct page *page, struct list_head *list, + unsigned int new_order); static inline int split_huge_page(struct page *page) { - return split_huge_page_to_list(page, NULL); + return split_huge_page_to_list_to_order(page, NULL, 0); } void deferred_split_folio(struct folio *folio); @@ -297,7 +298,8 @@ can_split_folio(struct folio *folio, int *pextra_pins) return false; } static inline int -split_huge_page_to_list(struct page *page, struct list_head *list) +split_huge_page_to_list_to_order(struct page *page, struct list_head *list, + unsigned int new_order) { return 0; } @@ -397,7 +399,7 @@ static inline bool thp_migration_supported(void) static inline int split_folio_to_list(struct folio *folio, struct list_head *list) { - return split_huge_page_to_list(&folio->page, list); + return split_huge_page_to_list_to_order(&folio->page, list, 0); } static inline int split_folio(struct folio *folio) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index f8a8a72b207d..619d25278340 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2359,11 +2359,13 @@ void vma_adjust_trans_huge(struct vm_area_struct *vma, static void unmap_folio(struct folio *folio) { - enum ttu_flags ttu_flags = TTU_RMAP_LOCKED | TTU_SPLIT_HUGE_PMD | - TTU_SYNC; + enum ttu_flags ttu_flags = TTU_RMAP_LOCKED | TTU_SYNC; VM_BUG_ON_FOLIO(!folio_test_large(folio), folio); + if (folio_test_pmd_mappable(folio)) + ttu_flags |= TTU_SPLIT_HUGE_PMD; + /* * Anon pages need migration entries to preserve them, but file * pages can simply be left unmapped, then faulted back on demand. @@ -2395,7 +2397,6 @@ static void lru_add_page_tail(struct page *head, struct page *tail, struct lruvec *lruvec, struct list_head *list) { VM_BUG_ON_PAGE(!PageHead(head), head); - VM_BUG_ON_PAGE(PageCompound(tail), head); VM_BUG_ON_PAGE(PageLRU(tail), head); lockdep_assert_held(&lruvec->lru_lock); @@ -2416,7 +2417,7 @@ static void lru_add_page_tail(struct page *head, struct page *tail, } static void __split_huge_page_tail(struct page *head, int tail, - struct lruvec *lruvec, struct list_head *list) + struct lruvec *lruvec, struct list_head *list, unsigned int new_order) { struct page *page_tail = head + tail; @@ -2483,10 +2484,15 @@ static void __split_huge_page_tail(struct page *head, int tail, * which needs correct compound_head(). */ clear_compound_head(page_tail); + if (new_order) { + prep_compound_page(page_tail, new_order); + prep_transhuge_page(page_tail); + } /* Finally unfreeze refcount. Additional reference from page cache. */ - page_ref_unfreeze(page_tail, 1 + (!PageAnon(head) || - PageSwapCache(head))); + page_ref_unfreeze(page_tail, 1 + ((!PageAnon(head) || + PageSwapCache(head)) ? + thp_nr_pages(page_tail) : 0)); if (page_is_young(head)) set_page_young(page_tail); @@ -2504,7 +2510,7 @@ static void __split_huge_page_tail(struct page *head, int tail, } static void __split_huge_page(struct page *page, struct list_head *list, - pgoff_t end) + pgoff_t end, unsigned int new_order) { struct folio *folio = page_folio(page); struct page *head = &folio->page; @@ -2512,11 +2518,12 @@ static void __split_huge_page(struct page *page, struct list_head *list, struct address_space *swap_cache = NULL; unsigned long offset = 0; unsigned int nr = thp_nr_pages(head); + unsigned int new_nr = 1 << new_order; int order = folio_order(folio); int i; /* complete memcg works before add pages to LRU */ - split_page_memcg(head, order, 0); + split_page_memcg(head, order, new_order); if (PageAnon(head) && PageSwapCache(head)) { swp_entry_t entry = { .val = page_private(head) }; @@ -2531,14 +2538,14 @@ static void __split_huge_page(struct page *page, struct list_head *list, ClearPageHasHWPoisoned(head); - for (i = nr - 1; i >= 1; i--) { - __split_huge_page_tail(head, i, lruvec, list); + for (i = nr - new_nr; i >= new_nr; i -= new_nr) { + __split_huge_page_tail(head, i, lruvec, list, new_order); /* Some pages can be beyond EOF: drop them from page cache */ if (head[i].index >= end) { struct folio *tail = page_folio(head + i); if (shmem_mapping(head->mapping)) - shmem_uncharge(head->mapping->host, 1); + shmem_uncharge(head->mapping->host, new_nr); else if (folio_test_clear_dirty(tail)) folio_account_cleaned(tail, inode_to_wb(folio->mapping->host)); @@ -2548,29 +2555,38 @@ static void __split_huge_page(struct page *page, struct list_head *list, __xa_store(&head->mapping->i_pages, head[i].index, head + i, 0); } else if (swap_cache) { + /* + * split anonymous THPs (including swapped out ones) to + * non-zero order not supported + */ + VM_WARN_ONCE(new_order, + "Split swap-cached anon folio to non-0 order not supported"); __xa_store(&swap_cache->i_pages, offset + i, head + i, 0); } } - ClearPageCompound(head); + if (!new_order) + ClearPageCompound(head); + else + set_compound_order(head, new_order); unlock_page_lruvec(lruvec); /* Caller disabled irqs, so they are still disabled here */ - split_page_owner(head, order, 0); + split_page_owner(head, order, new_order); /* See comment in __split_huge_page_tail() */ if (PageAnon(head)) { /* Additional pin to swap cache */ if (PageSwapCache(head)) { - page_ref_add(head, 2); + page_ref_add(head, 1 + new_nr); xa_unlock(&swap_cache->i_pages); } else { page_ref_inc(head); } } else { /* Additional pin to page cache */ - page_ref_add(head, 2); + page_ref_add(head, 1 + new_nr); xa_unlock(&head->mapping->i_pages); } local_irq_enable(); @@ -2583,7 +2599,15 @@ static void __split_huge_page(struct page *page, struct list_head *list, split_swap_cluster(entry); } - for (i = 0; i < nr; i++) { + /* + * set page to its compound_head when split to non order-0 pages, so + * we can skip unlocking it below, since PG_locked is transferred to + * the compound_head of the page and the caller will unlock it. + */ + if (new_order) + page = compound_head(page); + + for (i = 0; i < nr; i += new_nr) { struct page *subpage = head + i; if (subpage == page) continue; @@ -2617,29 +2641,31 @@ bool can_split_folio(struct folio *folio, int *pextra_pins) } /* - * This function splits huge page into normal pages. @page can point to any - * subpage of huge page to split. Split doesn't change the position of @page. + * This function splits huge page into pages in @new_order. @page can point to + * any subpage of huge page to split. Split doesn't change the position of + * @page. * * Only caller must hold pin on the @page, otherwise split fails with -EBUSY. * The huge page must be locked. * * If @list is null, tail pages will be added to LRU list, otherwise, to @list. * - * Both head page and tail pages will inherit mapping, flags, and so on from - * the hugepage. + * Pages in new_order will inherit mapping, flags, and so on from the hugepage. * - * GUP pin and PG_locked transferred to @page. Rest subpages can be freed if - * they are not mapped. + * GUP pin and PG_locked transferred to @page or the compound page @page belongs + * to. Rest subpages can be freed if they are not mapped. * * Returns 0 if the hugepage is split successfully. * Returns -EBUSY if the page is pinned or if anon_vma disappeared from under * us. */ -int split_huge_page_to_list(struct page *page, struct list_head *list) +int split_huge_page_to_list_to_order(struct page *page, struct list_head *list, + unsigned int new_order) { struct folio *folio = page_folio(page); struct deferred_split *ds_queue = get_deferred_split_queue(folio); - XA_STATE(xas, &folio->mapping->i_pages, folio->index); + /* reset xarray order to new order after split */ + XA_STATE_ORDER(xas, &folio->mapping->i_pages, folio->index, new_order); struct anon_vma *anon_vma = NULL; struct address_space *mapping = NULL; int extra_pins, ret; @@ -2649,6 +2675,18 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); VM_BUG_ON_FOLIO(!folio_test_large(folio), folio); + /* Cannot split THP to order-1 (no order-1 THPs) */ + if (new_order == 1) { + VM_WARN_ONCE(1, "Cannot split to order-1 folio"); + return -EINVAL; + } + + /* Split anonymous folio to non-zero order not support */ + if (folio_test_anon(folio) && new_order) { + VM_WARN_ONCE(1, "Split anon folio to non-0 order not support"); + return -EINVAL; + } + is_hzp = is_huge_zero_page(&folio->page); VM_WARN_ON_ONCE_FOLIO(is_hzp, folio); if (is_hzp) @@ -2744,7 +2782,13 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) if (folio_ref_freeze(folio, 1 + extra_pins)) { if (!list_empty(&folio->_deferred_list)) { ds_queue->split_queue_len--; - list_del(&folio->_deferred_list); + /* + * Reinitialize page_deferred_list after removing the + * page from the split_queue, otherwise a subsequent + * split will see list corruption when checking the + * page_deferred_list. + */ + list_del_init(&folio->_deferred_list); } spin_unlock(&ds_queue->split_queue_lock); if (mapping) { @@ -2754,14 +2798,18 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) if (folio_test_swapbacked(folio)) { __lruvec_stat_mod_folio(folio, NR_SHMEM_THPS, -nr); - } else { + } else if (!new_order) { + /* + * Decrease THP stats only if split to normal + * pages + */ __lruvec_stat_mod_folio(folio, NR_FILE_THPS, -nr); filemap_nr_thps_dec(mapping); } } - __split_huge_page(page, list, end); + __split_huge_page(page, list, end, new_order); ret = 0; } else { spin_unlock(&ds_queue->split_queue_lock);