From patchwork Thu Aug 22 13:50:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pankaj Raghav (Samsung)" X-Patchwork-Id: 13773525 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1083AC52D7C for ; Thu, 22 Aug 2024 13:50:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9CBD66B026D; Thu, 22 Aug 2024 09:50:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 97B946B026E; Thu, 22 Aug 2024 09:50:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7CDFC6B026F; Thu, 22 Aug 2024 09:50:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 5AF5B6B026D for ; Thu, 22 Aug 2024 09:50:57 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 1BB28816D0 for ; Thu, 22 Aug 2024 13:50:57 +0000 (UTC) X-FDA: 82480017354.11.E1727A7 Received: from mout-p-102.mailbox.org (mout-p-102.mailbox.org [80.241.56.152]) by imf09.hostedemail.com (Postfix) with ESMTP id 403F214001B for ; Thu, 22 Aug 2024 13:50:53 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=pankajraghav.com header.s=MBO0001 header.b=TI3XkqrO; dmarc=pass (policy=quarantine) header.from=pankajraghav.com; spf=pass (imf09.hostedemail.com: domain of kernel@pankajraghav.com designates 80.241.56.152 as permitted sender) smtp.mailfrom=kernel@pankajraghav.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1724334564; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=x4+UJEIcZLaw+P6a+NKFEvwnnHiOsE5MmywSgHb9XjU=; b=nXPVUo02vCZO+2t2o/ECK03IAmIb05wbrcdMBC4XG9C/Bwf3AxSuqarUux/DSIi/56h94v 2jJFC44CNCv03na12MQpBHOPjPdqgHZK9Te55KFU+xDJSTS3/hYREqfKeCptsKfZED4Y5i 0bEGOxoqoWeMTIaohSIJNfJ3XJSptjo= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1724334564; a=rsa-sha256; cv=none; b=ptVLCRAiK/2smq8DS70H65Ku+UdkgQP2neBgdVvIgCCFKSufU5oF5eG2f1SKpVSUtuzrIS 7dmHEvndFRd81LbQlGK2tb4iP1U472k0HFNY4gZA2+69VdIqsdum2jSr5PgXQtbbbGnPuv vGMOlKat6f5I/p9B4lvRlHU3VYMGMhQ= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=pankajraghav.com header.s=MBO0001 header.b=TI3XkqrO; dmarc=pass (policy=quarantine) header.from=pankajraghav.com; spf=pass (imf09.hostedemail.com: domain of kernel@pankajraghav.com designates 80.241.56.152 as permitted sender) smtp.mailfrom=kernel@pankajraghav.com Received: from smtp202.mailbox.org (smtp202.mailbox.org [IPv6:2001:67c:2050:b231:465::202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mout-p-102.mailbox.org (Postfix) with ESMTPS id 4WqPjk6K4bz9shX; Thu, 22 Aug 2024 15:50:50 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pankajraghav.com; s=MBO0001; t=1724334650; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=x4+UJEIcZLaw+P6a+NKFEvwnnHiOsE5MmywSgHb9XjU=; b=TI3XkqrOA1G5xwXEyCR1F9QdslfABIfK9sZB8HKy2iWovXDXulySJg3g87dZTPCkTT5C44 k4f8YSeu80Sbez6lb6ok73vzCwncZBmUYVJKzxwOG+Aw8p9A2YCq4Cd5reW0vK5LF/gYuY 23Q+7tPshAIXPIhqVbNRdyVEWa4tU9loSCNPaXo5yLr42au/8Kawto53+jwyjVFfKurQMD EsqyNjAiWtoV50XL7TfQealdHd01Q7HN/iIc1Mn7e3+saSIzpB4u2YHJLSTluwfivIXipl yYrPITnpRSP0mSTCPpx6KDZZ5K0836MrDORPqIy1r3Z+e42nnlmT57QA40k0pg== From: "Pankaj Raghav (Samsung)" To: brauner@kernel.org, akpm@linux-foundation.org Cc: chandan.babu@oracle.com, linux-fsdevel@vger.kernel.org, djwong@kernel.org, hare@suse.de, gost.dev@samsung.com, linux-xfs@vger.kernel.org, kernel@pankajraghav.com, hch@lst.de, david@fromorbit.com, Zi Yan , yang@os.amperecomputing.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, willy@infradead.org, john.g.garry@oracle.com, cl@os.amperecomputing.com, p.raghav@samsung.com, mcgrof@kernel.org, ryan.roberts@arm.com, David Howells Subject: [PATCH v13 04/10] mm: split a folio in minimum folio order chunks Date: Thu, 22 Aug 2024 15:50:12 +0200 Message-ID: <20240822135018.1931258-5-kernel@pankajraghav.com> In-Reply-To: <20240822135018.1931258-1-kernel@pankajraghav.com> References: <20240822135018.1931258-1-kernel@pankajraghav.com> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 403F214001B X-Stat-Signature: xrttywz4phcamgyzuu8fjoet6qcj94oc X-Rspam-User: X-HE-Tag: 1724334653-256354 X-HE-Meta: U2FsdGVkX18bRwJsVFR9lHVbAxRh22SrJ7yFnXLrDdU2dwuYourfQSJCwsaAYfmu4sNDbM+Xofyv6IX1MX/1FAGeW5T6dfFDYOozyNfBQ3mUyT2pcr9RTGXuasLTu3BLdgVQdSM7e29oHzDfcJdHrEJTv7F2atQC91JI5bwrH3XQtmfhLn1tHwG6tSJEtqwofQd2DXgUQjXrI02H2WxaFDvf8WHxKyaypxS8sd3+OQc6ScvcRdQXrxa8n/slKedGqn2cI4E1/IGbMMuCGNrVtaFqI37BUFl7X3nx967DyDr445yQlx51STvo4f0Or8IDWSjSW8W6+IihbzSJbvVkJcmxFBTxEzNGAXjj8Jm26ad0+Pa5q50ZlEMtIR8UlY/b9xkCMvNbPukh0rus3fgW9PT7mwHthRmMvVGJhrnfLuuUxwsvbBZvjV/zuGcSfC34AhcNkGGGepsrlVzvDeCoFjnOpiF4tM7D7t7LQwe9id8fqiyNpAd4eOnvoWKIU0LB++So/yBWLJWPm8zY+L2AgOEH3E/gyeY1OhN3Xjx67AMYGXnWFMCQoAVBXrZQtU4yo1WqDlq+FkFKfwDVG5eiKeJfXyPpEEG/iAWLsivdbJlXMNLNCox3HsfwFf94AsZbsLi4Ca3Mzrv5LAa0I6aBAoNRNU9d3oWd/Ihk/ZdtFBguzcUOF2u+qbr8ysDcdrOOqBcYnx4Ch27sCOoHDpjUh23TrlxtqTrE1Rh6y+/3UbdYak/Hte9VZn+a0FicDy9hGeIQIM8ZYyUEeQo0ZMGRMKBWjwHY+RM5Nmtkqw52dAvTsahawyhNqJXvTmJGi+qCKApoU3nVhkPBvEpmNunH3pVqDONicVVnzHo6zGkb+UWEAj34DPwkgrIRsE54YJPuOsIq8fhO4yCQCV1sMsM2bvj0+dQ8mlRFubExgVOVRx8v9Vw92+9YuqnwSxAZLVltdZZjnUiumdAAtI6BT75 IORZA7tF wdfHhgZfKUlICuJU8U53LqojHrskLaRqqcmKxCoNhBeDcx+TNXk9sVgwIgBL9d4+fGxBsHaJgOaMh+u4pKWfZNJGv5FYG6FmzeWj7gwEboNiHiAtuxqiMR5ylVPwgZsliXwE3+4OVwlzVVOgiVKrRxKSI3LWuiQ6+pZezgviehYIZYdB2RmlaEohFP/RPoZlmohdWUgp74OaGTfgUufYUVOekXT22lHZUZauGOcPCzeMZ/55Ptb7rTfM2z+Mis8Wq4yJMdDmD0msT0DB8N5qEi52hPgu1UZdQxaehEN2tgZJCyrQQDis9qE4d5iQmbB+ezP/BYv3+QQ14ykdJBCrQx1Yk/D02JNlq2KhYlyEuahx4yvc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Luis Chamberlain split_folio() and split_folio_to_list() assume order 0, to support minorder for non-anonymous folios, we must expand these to check the folio mapping order and use that. Set new_order to be at least minimum folio order if it is set in split_huge_page_to_list() so that we can maintain minimum folio order requirement in the page cache. Update the debugfs write files used for testing to ensure the order is respected as well. We simply enforce the min order when a file mapping is used. Signed-off-by: Luis Chamberlain Signed-off-by: Pankaj Raghav Reviewed-by: Hannes Reinecke Reviewed-by: Zi Yan Tested-by: David Howells --- include/linux/huge_mm.h | 14 +++++++--- mm/huge_memory.c | 60 ++++++++++++++++++++++++++++++++++++++--- 2 files changed, 66 insertions(+), 8 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 4c32058cacfec..70424d55da088 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -96,6 +96,8 @@ extern struct kobj_attribute thpsize_shmem_enabled_attr; #define thp_vma_allowable_order(vma, vm_flags, tva_flags, order) \ (!!thp_vma_allowable_orders(vma, vm_flags, tva_flags, BIT(order))) +#define split_folio(f) split_folio_to_list(f, NULL) + #ifdef CONFIG_PGTABLE_HAS_HUGE_LEAVES #define HPAGE_PMD_SHIFT PMD_SHIFT #define HPAGE_PUD_SHIFT PUD_SHIFT @@ -317,9 +319,10 @@ unsigned long thp_get_unmapped_area_vmflags(struct file *filp, unsigned long add bool can_split_folio(struct folio *folio, int caller_pins, int *pextra_pins); int split_huge_page_to_list_to_order(struct page *page, struct list_head *list, unsigned int new_order); +int split_folio_to_list(struct folio *folio, struct list_head *list); static inline int split_huge_page(struct page *page) { - return split_huge_page_to_list_to_order(page, NULL, 0); + return split_folio(page_folio(page)); } void deferred_split_folio(struct folio *folio); @@ -495,6 +498,12 @@ static inline int split_huge_page(struct page *page) { return 0; } + +static inline int split_folio_to_list(struct folio *folio, struct list_head *list) +{ + return 0; +} + static inline void deferred_split_folio(struct folio *folio) {} #define split_huge_pmd(__vma, __pmd, __address) \ do { } while (0) @@ -622,7 +631,4 @@ static inline int split_folio_to_order(struct folio *folio, int new_order) return split_folio_to_list_to_order(folio, NULL, new_order); } -#define split_folio_to_list(f, l) split_folio_to_list_to_order(f, l, 0) -#define split_folio(f) split_folio_to_order(f, 0) - #endif /* _LINUX_HUGE_MM_H */ diff --git a/mm/huge_memory.c b/mm/huge_memory.c index cf8e34f62976f..06384b85a3a20 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3303,6 +3303,9 @@ bool can_split_folio(struct folio *folio, int caller_pins, int *pextra_pins) * released, or if some unexpected race happened (e.g., anon VMA disappeared, * truncation). * + * Callers should ensure that the order respects the address space mapping + * min-order if one is set for non-anonymous folios. + * * Returns -EINVAL when trying to split to an order that is incompatible * with the folio. Splitting to order 0 is compatible with all folios. */ @@ -3384,6 +3387,7 @@ int split_huge_page_to_list_to_order(struct page *page, struct list_head *list, mapping = NULL; anon_vma_lock_write(anon_vma); } else { + unsigned int min_order; gfp_t gfp; mapping = folio->mapping; @@ -3394,6 +3398,14 @@ int split_huge_page_to_list_to_order(struct page *page, struct list_head *list, goto out; } + min_order = mapping_min_folio_order(folio->mapping); + if (new_order < min_order) { + VM_WARN_ONCE(1, "Cannot split mapped folio below min-order: %u", + min_order); + ret = -EINVAL; + goto out; + } + gfp = current_gfp_context(mapping_gfp_mask(mapping) & GFP_RECLAIM_MASK); @@ -3506,6 +3518,25 @@ int split_huge_page_to_list_to_order(struct page *page, struct list_head *list, return ret; } +int split_folio_to_list(struct folio *folio, struct list_head *list) +{ + unsigned int min_order = 0; + + if (folio_test_anon(folio)) + goto out; + + if (!folio->mapping) { + if (folio_test_pmd_mappable(folio)) + count_vm_event(THP_SPLIT_PAGE_FAILED); + return -EBUSY; + } + + min_order = mapping_min_folio_order(folio->mapping); +out: + return split_huge_page_to_list_to_order(&folio->page, list, + min_order); +} + void __folio_undo_large_rmappable(struct folio *folio) { struct deferred_split *ds_queue; @@ -3736,6 +3767,8 @@ static int split_huge_pages_pid(int pid, unsigned long vaddr_start, struct vm_area_struct *vma = vma_lookup(mm, addr); struct folio_walk fw; struct folio *folio; + struct address_space *mapping; + unsigned int target_order = new_order; if (!vma) break; @@ -3753,7 +3786,13 @@ static int split_huge_pages_pid(int pid, unsigned long vaddr_start, if (!is_transparent_hugepage(folio)) goto next; - if (new_order >= folio_order(folio)) + if (!folio_test_anon(folio)) { + mapping = folio->mapping; + target_order = max(new_order, + mapping_min_folio_order(mapping)); + } + + if (target_order >= folio_order(folio)) goto next; total++; @@ -3771,9 +3810,14 @@ static int split_huge_pages_pid(int pid, unsigned long vaddr_start, folio_get(folio); folio_walk_end(&fw, vma); - if (!split_folio_to_order(folio, new_order)) + if (!folio_test_anon(folio) && folio->mapping != mapping) + goto unlock; + + if (!split_folio_to_order(folio, target_order)) split++; +unlock: + folio_unlock(folio); folio_put(folio); @@ -3802,6 +3846,8 @@ static int split_huge_pages_in_file(const char *file_path, pgoff_t off_start, pgoff_t index; int nr_pages = 1; unsigned long total = 0, split = 0; + unsigned int min_order; + unsigned int target_order; file = getname_kernel(file_path); if (IS_ERR(file)) @@ -3815,6 +3861,8 @@ static int split_huge_pages_in_file(const char *file_path, pgoff_t off_start, file_path, off_start, off_end); mapping = candidate->f_mapping; + min_order = mapping_min_folio_order(mapping); + target_order = max(new_order, min_order); for (index = off_start; index < off_end; index += nr_pages) { struct folio *folio = filemap_get_folio(mapping, index); @@ -3829,15 +3877,19 @@ static int split_huge_pages_in_file(const char *file_path, pgoff_t off_start, total++; nr_pages = folio_nr_pages(folio); - if (new_order >= folio_order(folio)) + if (target_order >= folio_order(folio)) goto next; if (!folio_trylock(folio)) goto next; - if (!split_folio_to_order(folio, new_order)) + if (folio->mapping != mapping) + goto unlock; + + if (!split_folio_to_order(folio, target_order)) split++; +unlock: folio_unlock(folio); next: folio_put(folio);