From patchwork Fri Mar 1 21:47:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13579164 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 80812C5478C for ; Fri, 1 Mar 2024 21:47:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DB4436B009D; Fri, 1 Mar 2024 16:47:19 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D64886B009F; Fri, 1 Mar 2024 16:47:19 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C2D316B00A0; Fri, 1 Mar 2024 16:47:19 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id AE51F6B009D for ; Fri, 1 Mar 2024 16:47:19 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 82FBE1A0512 for ; Fri, 1 Mar 2024 21:47:19 +0000 (UTC) X-FDA: 81849806598.15.74BE445 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf11.hostedemail.com (Postfix) with ESMTP id ED14240006 for ; Fri, 1 Mar 2024 21:47:17 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=VmUiyKwR; spf=none (imf11.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709329638; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=lAm88NCh7qCyZozAaLMF5xc2naIQx+TsIZO0GlDocOQ=; b=lvHu1WLnRS3tOYb53d6tQH7wcF+FyGUGwmcCTA/PYrik81K8wOH9OQNdQ3EmJy338AGTRv 7DK9n12D+V54m8wfac22yk3wGaxbECxd32EmJXKkxpFsHMUFNdYZafRSAE6lNyglZ/8x2p O4Rcg4n/6ZTt1vYTixk0lAUYumwz4ls= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709329638; a=rsa-sha256; cv=none; b=iaWRWkdb4WEpu7FH/zpGJpl3Cnuubv690SYQulmPdnmh603clEkgk2sX1Qm0M5gUdZn4vD TWQyvqCT15jRLF54kZer4UwuQ3cEkGnjsIrBeef0zPwGnf2diDgQYQZ03eFqVPDnxKggwx xbO06oJ2XBSxQUFql40fZutEguZLTj8= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=VmUiyKwR; spf=none (imf11.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=lAm88NCh7qCyZozAaLMF5xc2naIQx+TsIZO0GlDocOQ=; b=VmUiyKwRbFkgNyqK0y5t4p7bph yBavLscAMxee/Z7kaZX6UntAWDDMV7a1BjbjxIp74Cn+SWXc579e+63eOQiYj6A1lkSDgMPRi+SHi K8IUy0GrNAmYhjIq0EbBWvjDWgkOuIH0qi7o70Y5J3mPMR6tzlh1Ckpc3CJqf7UmBo2NmGGbUsoNo eAy8Uyf2qXp/xl96vbtu7gEsAHfC734IKZrm8SvojItzgeKGpBt0/H+pKerJ0Vd4Mrke0M4JmqnCa oRwZcyHpnrzRDpvxM8EXfIcEDjRkFKL8JL4hDXgw4TClS8Wkxiiw5YcMtSTt7dHaTegSAn8cW6cRJ GZTx1W/g==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rgAin-0000000ByEh-2NQT; Fri, 01 Mar 2024 21:47:13 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , Oscar Salvador Subject: [PATCH 1/5] hugetlb: Make folio_test_hugetlb safer to call Date: Fri, 1 Mar 2024 21:47:06 +0000 Message-ID: <20240301214712.2853147-2-willy@infradead.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240301214712.2853147-1-willy@infradead.org> References: <20240301214712.2853147-1-willy@infradead.org> MIME-Version: 1.0 X-Stat-Signature: cfqtk6cofaaekrdxn1xysefikp3zdqjk X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: ED14240006 X-Rspam-User: X-HE-Tag: 1709329637-367744 X-HE-Meta: U2FsdGVkX18o7J5Yf1/drz/C2ApKzlbh9yeV/ECzFZCzXdVCkS4b4DlmFk5RMjL2XrWVEdwZWHSHZHJ6knlfQjKmDLGYmdoorRDlNb5YveTG8ZlPDAeht/U+nCONI6wOdHtvV7xxMDE+6igPaTGYqjQIWUweScmeUk+XegJMaedNUFVxH6J3hkLaVKXkBbwNfT4l3tRZris8cljsQVQxuXbyYCpZASZ/Oq4GDO4crzSd/Tbc4eg5CU59ygSyTYsgs6giDhaDo90c4Bd7uWyz4vx7gmH7hM9oU2rrZDqPbpeC/ScB/qGUWu4MyA8HGUoRKcdrGxxPzX/6TdBvIS8buu8utgGWF1+i0OGS3p4/1eQj6NY6+52x+r0RhpbQt7Ssue/k2lgrdV8yDBINHHY/mhzYcY6GM/1YI25VB5tpZIJV4MupwSxLaExoM81gsylRzuDl7q6AK7jWUr3iGV5r3nNBeHYm94zHMbKroGt3o8H89iI/zSytzXSsvmYN+9VcIwQunat5fnp3tI7kasoGe+fFEwYqRlfoULXShcEZIDiVJ2VktApnYqDgHkLUbPD2wjGoow6QU+39r+owjWY1wvh5sfcR/49Z2zVsT3JaA5AJeMW/cvZJZCAjivQPysZs9E/BBmt3AYPwWlx81qxBtlhQRxZNg7+57dc/3TYiM2uP5cValOzS/xOiqc3eI7yYbOLFDaUi18EEEvWP5fSg2xryZyNljNh6gc44E0Qx9yoohdbP486XNZREwZXaVUBKUWONmF6F/BUacqUFJlAuS5PUU3W5SKQDwV3rbI1zh0zbVHmCbN7jLORzjh+sW3MgaCLHRQJNfnmWO/1KlDJwMPaUJj+gQqBPigm4IDO3S1zHmpoYnC+ybEylDImrZoiEmnZZ5B/bI590cSXreK2tHe17LxLVQTpmWNBnBHWkgc8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: At least two places (memory failure and page migration) need to call folio_test_hugetlb() without a reference on the folio. This can currently result in false positives (returning true when the folio doesn't belong to hugetlb) and more commonly in VM_BUG_ON() when a folio is split. The new way to distinguish a hugetlb folio is to see if (1) the page is compound (or the folio is large) and (2) page[1].mapping is set to the address of hugetlb_lock. If the folio is (or has been) large then page[1] is guaranteed to exist. If the folio is split between the two tests, page[1].mapping will be set to something which definitely isn't the address of hugetlb_lock. Because we shift around the layout of struct folio a bit, we now use page[1].private, which means we need to adjust __split_huge_page_tail() a little. We also need to annoy the vmcore_info people again. Sorry. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Oscar Salvador --- include/linux/mm_types.h | 4 +++- include/linux/page-flags.h | 25 ++++--------------------- kernel/vmcore_info.c | 3 ++- mm/huge_memory.c | 10 ++-------- mm/hugetlb.c | 24 ++++++++++++++++++++++++ 5 files changed, 35 insertions(+), 31 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index a7223ba3ea1e..fd80bf8b5d8a 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -289,6 +289,7 @@ typedef struct { * @virtual: Virtual address in the kernel direct map. * @_last_cpupid: IDs of last CPU and last process that accessed the folio. * @_entire_mapcount: Do not use directly, call folio_entire_mapcount(). + * @large_id: May identify the type of a large folio. * @_nr_pages_mapped: Do not use directly, call folio_mapcount(). * @_pincount: Do not use directly, call folio_maybe_dma_pinned(). * @_folio_nr_pages: Do not use directly, call folio_nr_pages(). @@ -348,9 +349,9 @@ struct folio { struct { unsigned long _flags_1; unsigned long _head_1; - unsigned long _folio_avail; /* public: */ atomic_t _entire_mapcount; + void *large_id; atomic_t _nr_pages_mapped; atomic_t _pincount; #ifdef CONFIG_64BIT @@ -407,6 +408,7 @@ FOLIO_MATCH(_last_cpupid, _last_cpupid); offsetof(struct page, pg) + sizeof(struct page)) FOLIO_MATCH(flags, _flags_1); FOLIO_MATCH(compound_head, _head_1); +FOLIO_MATCH(mapping, large_id); #undef FOLIO_MATCH #define FOLIO_MATCH(pg, fl) \ static_assert(offsetof(struct folio, fl) == \ diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 652d77805e99..75bead4a5f09 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -188,10 +188,9 @@ enum pageflags { * PF_ANY. */ + PG_large_rmappable = PG_active, /* anon or file-backed */ /* At least one page in this folio has the hwpoison flag set */ - PG_has_hwpoisoned = PG_error, - PG_hugetlb = PG_active, - PG_large_rmappable = PG_workingset, /* anon or file-backed */ + PG_has_hwpoisoned = PG_workingset, }; #define PAGEFLAGS_MASK ((1UL << NR_PAGEFLAGS) - 1) @@ -857,23 +856,7 @@ TESTPAGEFLAG_FALSE(LargeRmappable, large_rmappable) #ifdef CONFIG_HUGETLB_PAGE int PageHuge(const struct page *page); -SETPAGEFLAG(HugeTLB, hugetlb, PF_SECOND) -CLEARPAGEFLAG(HugeTLB, hugetlb, PF_SECOND) - -/** - * folio_test_hugetlb - Determine if the folio belongs to hugetlbfs - * @folio: The folio to test. - * - * Context: Any context. Caller should have a reference on the folio to - * prevent it from being turned into a tail page. - * Return: True for hugetlbfs folios, false for anon folios or folios - * belonging to other filesystems. - */ -static inline bool folio_test_hugetlb(const struct folio *folio) -{ - return folio_test_large(folio) && - test_bit(PG_hugetlb, const_folio_flags(folio, 1)); -} +bool folio_test_hugetlb(const struct folio *folio); #else TESTPAGEFLAG_FALSE(Huge, hugetlb) #endif @@ -1118,7 +1101,7 @@ static __always_inline void __ClearPageAnonExclusive(struct page *page) */ #define PAGE_FLAGS_SECOND \ (0xffUL /* order */ | 1UL << PG_has_hwpoisoned | \ - 1UL << PG_hugetlb | 1UL << PG_large_rmappable) + 1UL << PG_large_rmappable) #define PAGE_FLAGS_PRIVATE \ (1UL << PG_private | 1UL << PG_private_2) diff --git a/kernel/vmcore_info.c b/kernel/vmcore_info.c index f95516cd45bb..453cd44a9c9c 100644 --- a/kernel/vmcore_info.c +++ b/kernel/vmcore_info.c @@ -14,6 +14,7 @@ #include #include #include +#include #include #include @@ -206,7 +207,7 @@ static int __init crash_save_vmcoreinfo_init(void) #define PAGE_BUDDY_MAPCOUNT_VALUE (~PG_buddy) VMCOREINFO_NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE); #ifdef CONFIG_HUGETLB_PAGE - VMCOREINFO_NUMBER(PG_hugetlb); + VMCOREINFO_NUMBER(&hugetlb_lock); #define PAGE_OFFLINE_MAPCOUNT_VALUE (~PG_offline) VMCOREINFO_NUMBER(PAGE_OFFLINE_MAPCOUNT_VALUE); #endif diff --git a/mm/huge_memory.c b/mm/huge_memory.c index a81a09236c16..5731f28cba5f 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2840,16 +2840,10 @@ static void __split_huge_page_tail(struct folio *folio, int tail, page_tail->mapping = head->mapping; page_tail->index = head->index + tail; - /* - * page->private should not be set in tail pages. Fix up and warn once - * if private is unexpectedly set. - */ - if (unlikely(page_tail->private)) { - VM_WARN_ON_ONCE_PAGE(true, page_tail); - page_tail->private = 0; - } if (folio_test_swapcache(folio)) new_folio->swap.val = folio->swap.val + tail; + else + new_folio->private = NULL; /* Page flags must be visible before we make the page non-compound. */ smp_wmb(); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index bb17e5c22759..963c25963b5e 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -100,6 +100,30 @@ static void hugetlb_unshare_pmds(struct vm_area_struct *vma, unsigned long start, unsigned long end); static struct resv_map *vma_resv_map(struct vm_area_struct *vma); +static void folio_set_hugetlb(struct folio *folio) +{ + folio->large_id = &hugetlb_lock; +} + +static void folio_clear_hugetlb(struct folio *folio) +{ + folio->large_id = NULL; +} + +/** + * folio_test_hugetlb - Determine if the folio belongs to hugetlbfs. + * @folio: The folio to test. + * + * Context: Any context. + * Return: True for hugetlbfs folios, false for anon folios or folios + * belonging to other filesystems. + */ +bool folio_test_hugetlb(const struct folio *folio) +{ + return folio_test_large(folio) && folio->large_id == &hugetlb_lock; +} +EXPORT_SYMBOL_GPL(folio_test_hugetlb); + static inline bool subpool_is_free(struct hugepage_subpool *spool) { if (spool->count) From patchwork Fri Mar 1 21:47:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13579161 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8BEC0C5475B for ; Fri, 1 Mar 2024 21:47:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B69F86B0098; Fri, 1 Mar 2024 16:47:16 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B19E56B009B; Fri, 1 Mar 2024 16:47:16 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A30006B009C; Fri, 1 Mar 2024 16:47:16 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 908986B0098 for ; Fri, 1 Mar 2024 16:47:16 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 62E671205E5 for ; Fri, 1 Mar 2024 21:47:16 +0000 (UTC) X-FDA: 81849806472.10.C8810C5 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf28.hostedemail.com (Postfix) with ESMTP id E8E27C000C for ; Fri, 1 Mar 2024 21:47:14 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=gEKfwRsD; dmarc=none; spf=none (imf28.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709329635; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=m55YQGBGw9qVxoXEA/93FX4UlDUH9XWWX9notPlTSC8=; b=uQmXO8giiZo/WziiHsGM2ApRmuxAYGob1TX/iSa6C9srzZ26knW0IDgLc/Ikx3Q+WzEVyM ej8wQpmMGA3biZRrYUsxyiGO8XFVGivJi1PEVtNDJe2bTxA0+bJZ5iInodD4Ohf6AAIrq0 1fYBRbYka1fFm8mCr/QQLMtyBbI1Ac0= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=gEKfwRsD; dmarc=none; spf=none (imf28.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709329635; a=rsa-sha256; cv=none; b=1RCJeYrX/6vIJk7oBW/MX3E2I/Jn4IawvQfTaH/VVVQDDoCJPWw1EuVcBoVHUYvRJLbDsr Se+OYyy+CxyN7fz0gQblqxpGQK/CQ/swQec772bDKfXsWzHrgsa3jRtUS5yLYKoSACeLCD RVKq2cvOU4CKVVnZeMdgkzLX0knrPak= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=m55YQGBGw9qVxoXEA/93FX4UlDUH9XWWX9notPlTSC8=; b=gEKfwRsDt5A11MFHZYQ9jnXYbz srP/61zMNtPq+C7wvY6T8/OulnjowP3il+775wbFWXeXls612mjoGruI9aYIjVZXoZjXux17ZVBEB 7YRhpZdcInzgYHzRIAU0DUJP4BDynKqzlF9k7kWuLhioM2RJ21vJIrnZg9tfYSC3Hsn4YoV0nvZ0B Fvg6oYQwTZF4iXfpbyCYt/lmgIQgBHnV/jKdV7NQ4ZRRuqXfPu/Z9wrerKv7ZrViEkNBpu7L65sdL qE0353tz5VE2BsJDS5iNvfhb13adZ7SSF9FFhqnUHIwpA3dXLxGR1kL/ufANEfvLGET6RSOgzpw3l 0bTcHzbg==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rgAin-0000000ByEj-2hEW; Fri, 01 Mar 2024 21:47:13 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , Oscar Salvador Subject: [PATCH 2/5] hugetlb: Add hugetlb_pfn_folio Date: Fri, 1 Mar 2024 21:47:07 +0000 Message-ID: <20240301214712.2853147-3-willy@infradead.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240301214712.2853147-1-willy@infradead.org> References: <20240301214712.2853147-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: E8E27C000C X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: fsf9erf7yd91gnggndgwkzfxqirctk56 X-HE-Tag: 1709329634-976420 X-HE-Meta: U2FsdGVkX1/9nbWQlLxZvuLOAPlzQ4o3GnWidif2C8xSRgj700DyhkBVKufuWOUfPqaAagXq4zbiflLMsU397ryGKno4E0imL/G03fsGrSoCWd/5iZO3xvd7vVWRs50w7pWXaKkfZwYEy4ca1k6FYo4BC6PR6gXmkf/rZ7HkYsF5NnIsCRX7pTWWaagtvQqicjtvgdspC3N3FLy0d4ObGU1CK1+NHbhRpVrBZIWm4uQJ5d8291N5jFEAUWoIsmKhbTGcNM/fRD449tFh8x5+9SoTjLoIKXkd0b4o0LNVTywxffgt1eUfgGzusKEyh42bTs/IAV2A5MJsK+UjlJlR2f9OhRXgeMMemjlKkJwI4JQn5Lq4xK3DZaa75lG5XDu5ODgfvUG8FicrXbMdD8AXo1tEo4EVHajqrrR1f139BAgptLGDyicYNgZ0QZDvsD6QuOKhzmdznF5BAowwLmnPmb6whoSsQ4dTE4u/rKQrEN5CnvDGAzkEccxSGt7iyUmWfdetfhi4+ueQ5/w54+pCw9GwzW/CYk38nBuOiRr4VB2bSCpmw+WhGb51VIsr/jP8KYpc8/4vVoa661FMcmSCgs5LHoigad0GDnGMoLsSyj71q7zBBi34+v9HGhStwdZ1PEBy8DWkQ+uNQeBsel9o9ALDkAqujRxkvzeDZiL1oYVngGJcvZikKYVMYtN5HdfnlydxeaMmS0MxF3GnAuApqRlhI6lUFMFM897ji8We4WsuFnmcNjg7pRiAg+xxT8ysQgzqmrGkrpp8qcBO5CKSPfa2zuYIoVm3XAZ+3j9+fhYkVriOpJmuJYQTk82QMyeYFKfrAqp12Bn7/AHAvMZmRPi3EWfGaI4KqW9WowPvBWIfcB04hLI6dDU8PNx9MfRgGeYYeDIr2WEdgsokzao1FrQf3k6xSCcH X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Returns the hugetlb folio containing this PFN or NULL if the PFN is not within a hugetlb folio. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Oscar Salvador --- include/linux/hugetlb.h | 1 + mm/hugetlb.c | 27 +++++++++++++++++++++++++++ 2 files changed, 28 insertions(+) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 77b30a8c6076..c1d23ab53752 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -163,6 +163,7 @@ bool hugetlb_reserve_pages(struct inode *inode, long from, long to, long hugetlb_unreserve_pages(struct inode *inode, long start, long end, long freed); bool isolate_hugetlb(struct folio *folio, struct list_head *list); +struct folio *hugetlb_pfn_folio(unsigned long pfn); int get_hwpoison_hugetlb_folio(struct folio *folio, bool *hugetlb, bool unpoison); int get_huge_page_for_hwpoison(unsigned long pfn, int flags, bool *migratable_cleared); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 963c25963b5e..0895debc36f3 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -124,6 +124,33 @@ bool folio_test_hugetlb(const struct folio *folio) } EXPORT_SYMBOL_GPL(folio_test_hugetlb); +/** + * hugetlb_pfn_folio - Convert a pfn to a hugetlb folio. + * @pfn: The Page Frame Number. + * + * If this function returns NULL, the pfn definitely wasn't part of + * a hugetlb folio. If a folio pointer is returned, this pfn was + * part of that hugetlb folio at some point. If the caller holds + * the @hugetlb_lock, hugetlb folios may not be returned to the page + * allocator, so the folio will still be valid (it may or may not + * be free). + * + * Return: NULL if the pfn is not part of a hugetlb folio. + */ +struct folio *hugetlb_pfn_folio(unsigned long pfn) +{ + struct page *page = pfn_to_page(pfn); + struct folio *folio; + + if (!PageCompound(page)) + return NULL; + folio = page_folio(page); + if (folio->large_id != &hugetlb_lock) + return NULL; + + return folio; +} + static inline bool subpool_is_free(struct hugepage_subpool *spool) { if (spool->count) From patchwork Fri Mar 1 21:47:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13579165 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 91F41C5475B for ; Fri, 1 Mar 2024 21:47:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E04706B009F; Fri, 1 Mar 2024 16:47:22 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D8D896B00A0; Fri, 1 Mar 2024 16:47:22 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BDF0D6B00A1; Fri, 1 Mar 2024 16:47:22 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id AB9816B009F for ; Fri, 1 Mar 2024 16:47:22 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 7F2E11C0E6E for ; Fri, 1 Mar 2024 21:47:22 +0000 (UTC) X-FDA: 81849806724.23.9C1B414 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf25.hostedemail.com (Postfix) with ESMTP id E7403A001E for ; Fri, 1 Mar 2024 21:47:20 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=ld9FyidN; dmarc=none; spf=none (imf25.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709329641; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/DHmW4lsy5cWc2w+CyTR97vxYADmnbZh/vvNjDWdKlo=; b=V8gVHJLnT5oijY9zEASoB2U2hewyE3AOW7ZITuY8RNTMAE1mqCqhY1g3yUDx7d8EMVCAhX oOd0lsNO34XRZ590M6l72XMyF66hGrQVB+AW/A8uB5xGqM6QmRp6mvc9hRmwXdeH2ZJYvn Vozl+mpqszep3D65NrGbnfVkoiYSPio= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=ld9FyidN; dmarc=none; spf=none (imf25.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709329641; a=rsa-sha256; cv=none; b=eXb8r7a6wRoWyrMkBpCs2LR1zMbZOAWrHcDN4gRTRqEQfecKPb5F91UcIwEdu6aSGkJCdc GAiyoJHKq6pJKUtmwACQuINB3NxhEq3qHKi1NhxOSaSlYY19CvJkw5tpWakqyjn74LFdk5 eiZ+bniwPQBkOSYVoPgdMA6oXHNBybc= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=/DHmW4lsy5cWc2w+CyTR97vxYADmnbZh/vvNjDWdKlo=; b=ld9FyidNHd6q9I4dl5lgyHFUPD 1XPOy1eBvmwS9n3uEABjERxW54V5aC1RDVmO4vZhHusKXBV/QeHis+OMsOs08fR2M3tWeL6ZOIdu2 MM03S40zP4n+bSNeJJxDnl4P9z+ubmhTywEOOrJN6oqhUBzYmWk3FHQng8XDmogEIKJtkUMv9Y4UQ KJzvyxdbSgVJp3UkbMnb3GG815mraGeVr1c0D0S6aZRD3yfxisIAXEsErggXkbjY6Tl9XWpBU9RGO vxfRWYddzhqpxYY+tWtqNnh93r+zsyo2LL4FA5+ws36JDVPDBtnWLgiJpmwJ+4H7oOGxXPrF3d2QD 4zm3FguA==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rgAin-0000000ByEl-32Yw; Fri, 01 Mar 2024 21:47:13 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , Oscar Salvador Subject: [PATCH 3/5] memory-failure: Use hugetlb_pfn_folio Date: Fri, 1 Mar 2024 21:47:08 +0000 Message-ID: <20240301214712.2853147-4-willy@infradead.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240301214712.2853147-1-willy@infradead.org> References: <20240301214712.2853147-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: w3bnbidc6iqfiw8qje7b8aferiar7k3e X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: E7403A001E X-HE-Tag: 1709329640-768788 X-HE-Meta: U2FsdGVkX1/LlBbSjfyhTwgAjFyDo2dW91v29aP7rkYVrXY+Nr0EearOW14hdFhhDB9kMv24kFENdRBL6dur+HGPYDnY7uXGdJi2NdcUCMQHk/HvnfzWerKrpFYLcVn/TLLlkDwwsGWhlhwfgPivk5pAvBolyDS3YomiAxyIQjauLcxxbJUOTInQG1Qycokv6hXHpsvJF211+7oeOzIKor2PFXJLHHinOaOCRcw42ISS9V3CsU7V2G2G8UMWPLbjNGq6XZ5di/A7wuet18aGUoNcHrCKw6GUO4kQ4w9AmRVH5LXVre6LwZHkLHVoSgiDy3gWeHu0yqlyGc/bBuOJlzOsq34ra3AMp8x1KGCLUNiw+5dg0yhfv6cF3uqc9Kl4M90SiJmgR1xYVrpNY82koYvSXSqz+8wNk5JfVlDSEFufLe627GikqZHk9+rfNJjYz52xFipciQ0nbJnVpG/Na27VnuuR47EbbjNDGt/tdMHhvpLcj5Jqesw2EUwJRGdz9m1z/49u0SLie/kYQMEOdLtOz5SR/4YloFah0tOVwFwHSqw5DIRy85ZSnTZYQC35pfc11BP6BI8OQ0DZYd1hFUCVbJTLibW+MNlWpr9PGnfps7XMLhFYMp9EFjR5/gucNIv39ML55gP3LGBNMART1OixbqkxXD1/lOc4E5YybdMxqW1wzlFbYlpOvRImm+2h3dp86zQ8w4p7IaW+uea/rlcy5zlK4406NyGJ8s2bFzpnhhTs+JQkIeaH79k0cYpzvRnjxJ41t1daioVbHy9AmPIyehNZLcnVtjHViHXL7RpPfqp0JjMMHpaF2uu2Tk6iOBEslJl1f9QAJTKtgyYLH29zeTsN1f75JimQDSiQ1oIOgCbQsuPFe1bX0CI6JOtsKtN9mtExnsNkoS1SMfwQBQHB9bNiSIk0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Calling folio_test_hugetlb() is inefficient; hugetlb_pfn_folio() does less work. This is a safe use of hugetlb_pfn_folio() as __get_huge_page_for_hwpoison() holds the hugetlb_lock. Signed-off-by: Matthew Wilcox (Oracle) --- mm/memory-failure.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mm/memory-failure.c b/mm/memory-failure.c index 9349948f1abf..e42d0cd7888d 100644 --- a/mm/memory-failure.c +++ b/mm/memory-failure.c @@ -1987,12 +1987,12 @@ void folio_clear_hugetlb_hwpoison(struct folio *folio) int __get_huge_page_for_hwpoison(unsigned long pfn, int flags, bool *migratable_cleared) { - struct page *page = pfn_to_page(pfn); - struct folio *folio = page_folio(page); + struct folio *folio; int ret = 2; /* fallback to normal page handling */ bool count_increased = false; - if (!folio_test_hugetlb(folio)) + folio = hwpoison_pfn_folio(pfn); + if (!folio) goto out; if (flags & MF_COUNT_INCREASED) { From patchwork Fri Mar 1 21:47:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13579163 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8CF3BC5475B for ; Fri, 1 Mar 2024 21:47:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C7E186B009C; Fri, 1 Mar 2024 16:47:17 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C2C3D6B009D; Fri, 1 Mar 2024 16:47:17 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 977936B00A0; Fri, 1 Mar 2024 16:47:17 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 629456B009C for ; Fri, 1 Mar 2024 16:47:17 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 1BD971404D3 for ; Fri, 1 Mar 2024 21:47:17 +0000 (UTC) X-FDA: 81849806514.05.549039C Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf27.hostedemail.com (Postfix) with ESMTP id 7D35E4001A for ; Fri, 1 Mar 2024 21:47:15 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=nGoNpGBE; dmarc=none; spf=none (imf27.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709329635; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6KeEeju/hMI7K6EYqbMB4GB0Eui15rJAsgnCJmbCFsY=; b=pi2+oirLCkDs7h0ug746L44HBB5hMVJ4R2FA6gdpRciEzbQF6voyyK4e9jM7wsMp9mZLuP GxI9nGxENhNCtXUZb3gQfYUfu8oPF3ccl/I5nsABJfkJTXrNePto2pwm2KfTWK56cKqNkY uhI/Fym6cLlp/NXjMDFnjmYBh/nM4ug= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=nGoNpGBE; dmarc=none; spf=none (imf27.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709329635; a=rsa-sha256; cv=none; b=roK0qCwTaMjKVAztt2IpM8WrjCaXXgJfEJP4ptkidRNDpdFRjRYcKGpGaHYwwpJOdkRPB1 KCFNo8n6E9PuyOvWvDLA0JBTGWFDru3o76wuknnWOtPwkIZU1zVM3alTHQPuhW18wbq/pG wmiEjZwos2vpAUnCZDs1KxoJ5eLIgqM= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=6KeEeju/hMI7K6EYqbMB4GB0Eui15rJAsgnCJmbCFsY=; b=nGoNpGBEvpxTXESstmjqc/TLQ6 6WywmjqMQNZkIZIiEWu+wyGpatscC9dptVrDqrbw7468JlwTiV5cmh9ttWD1aDP4Spam9plYNzncb sMtrrNXSpvNLbP1qPIVGaIKdN0xjmI6H08t6jScG4yWif2Z2FZ8i/d7cXqm1a5prdpnXJaopZBz5H QS9Fa1kf2kdA7B/7RG/of/GtE+SHWlB3DeHox4AMysHbppB9JQNF6Xxa0z5NvaMkwj15EUOqtgEgk ugxf6wNoVAnaTfWHprwsA0CfkE8ZcgdJU27d0cxMBaIT5hnmtwV9GiIp3u7iRYL03q0lno/UHghTw vIU0oYpQ==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rgAin-0000000ByEn-3NYS; Fri, 01 Mar 2024 21:47:13 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , Oscar Salvador Subject: [PATCH 4/5] memory-failure: Reorganise get_huge_page_for_hwpoison() Date: Fri, 1 Mar 2024 21:47:09 +0000 Message-ID: <20240301214712.2853147-5-willy@infradead.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240301214712.2853147-1-willy@infradead.org> References: <20240301214712.2853147-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 7D35E4001A X-Stat-Signature: n5hrr4uktc8f7g9qr7h1668e99xfiyg9 X-Rspam-User: X-HE-Tag: 1709329635-799284 X-HE-Meta: U2FsdGVkX1/77v7BhjvZkEHjMLaUhh7vsnvSwS+XOFj0uQanHqhmYAGjw8LZJGrkUk2DmkzzlJL4ZWO++aQW4jffyoV/nfPw1Hq7joLMDazvO6WPnJfuRWGxnU6AX985kXddcbZc1D7HohPnuINUg6VDNfCcfU5jr72DDtoCiSbgB5wz+NmW+OH0LHxwLMVQz+NLIze3lL1FXbnXqDYFWKX+K/mZFpRMBd72SlIrCw6vrrXXV+CeamgL7/fkp3hwF4fChdBpk2WQB9WSym6I6VdFjDPO9fvIjWUeO/8lPrdKPBxa1qphiZea1EPzJM1RM8qnRJ9+yWA2GVCwqPkpxJr390wWeRQyII5XoB98euVww6yC4AzpqCc7lxObmpqchx5FHnfqqnW0FCeAl3eVgaH2m7OmXVcG+bhH8mAVOVd9aKrWfq+wcZiEaXGJfB0AEnkfFbksVOUNpgx3W4DADkoH/SjFp3999kchNJoiDE+csSpfXnvvsTcdJuFIoCenRNK0V5gG9K8z9uSRpVYkvS9iBDlm68b+k3z267qySIjpj7oQyWWWnow7UH2OI/PNE5+j0laCkpKxfSOaXKTTqLOGVFOnmrvRgM1ot2T9ZjU2VoZEiX3E+Y7EpbIwgMTviEd0ZdDJ7KN9f7o+8uybhxSv185N1PBzpX3S+OQv86jT8COKlD9RRQ4VArPlm7tQPYvgel4W0uHglgtqD6yU3+7ao9+hFn2mn3h24biu/ktseCVd4wDK2cmuYgcRK2hSjR51Wvh+tffyQ0CUcbLk1knk1YNVXFEanhr/5ASJgiQ2TIAdb5whg+Xlhy2zsUCnvGKvWrxaYSYDyTWFI4oP54yEt2U6679h3cJiVKFYleqoxi/hQAtdOAeWhCHR9PY5hCZH+Bw3b1A8XtU4GSmyGEdbMpjJh90HMfxkdSeJV5p70pl8Aga+FltUy/F11C5Ag7USIx8mP6x7Rmwc+uC jC3rdsxq RBt+50bHMLnDV+ZaY46V8n73/UcK4kD5pNQOMTLcIDlp3s5SukQ/Z/2cgwumsZ17sr7zPvCihk3CvzdwQ4KZGBr7HIn/VNrB+QfTLupg6ZYzlAbUlS8lBFgdmKJacucxDPedSs/VD7/QwB4ryfZt5Rc1V9EdhUnNz9Nl7J0KJVasfjZA8aqnFEEcLy1chJyD579/KbZZ/9bVWp/RLR1nNVyTJ4W7ldBMQHJLt X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: There's no need to split this function across hugetlb.c and memory-failure.c now that hugetlb_lock is visible outside hugetlb.c (since 2012). Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/hugetlb.h | 8 -------- include/linux/mm.h | 8 -------- mm/hugetlb.c | 11 ----------- mm/memory-failure.c | 8 ++++++-- 4 files changed, 6 insertions(+), 29 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index c1d23ab53752..89f4b90eec68 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -165,8 +165,6 @@ long hugetlb_unreserve_pages(struct inode *inode, long start, long end, bool isolate_hugetlb(struct folio *folio, struct list_head *list); struct folio *hugetlb_pfn_folio(unsigned long pfn); int get_hwpoison_hugetlb_folio(struct folio *folio, bool *hugetlb, bool unpoison); -int get_huge_page_for_hwpoison(unsigned long pfn, int flags, - bool *migratable_cleared); void folio_putback_active_hugetlb(struct folio *folio); void move_hugetlb_state(struct folio *old_folio, struct folio *new_folio, int reason); void hugetlb_fix_reserve_counts(struct inode *inode); @@ -452,12 +450,6 @@ static inline int get_hwpoison_hugetlb_folio(struct folio *folio, bool *hugetlb, return 0; } -static inline int get_huge_page_for_hwpoison(unsigned long pfn, int flags, - bool *migratable_cleared) -{ - return 0; -} - static inline void folio_putback_active_hugetlb(struct folio *folio) { } diff --git a/include/linux/mm.h b/include/linux/mm.h index 0436b919f1c7..cfbf2bbc6200 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3978,8 +3978,6 @@ extern int soft_offline_page(unsigned long pfn, int flags); */ extern const struct attribute_group memory_failure_attr_group; extern void memory_failure_queue(unsigned long pfn, int flags); -extern int __get_huge_page_for_hwpoison(unsigned long pfn, int flags, - bool *migratable_cleared); void num_poisoned_pages_inc(unsigned long pfn); void num_poisoned_pages_sub(unsigned long pfn, long i); struct task_struct *task_early_kill(struct task_struct *tsk, int force_early); @@ -3988,12 +3986,6 @@ static inline void memory_failure_queue(unsigned long pfn, int flags) { } -static inline int __get_huge_page_for_hwpoison(unsigned long pfn, int flags, - bool *migratable_cleared) -{ - return 0; -} - static inline void num_poisoned_pages_inc(unsigned long pfn) { } diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 0895debc36f3..2e6ebedb75a8 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -7674,17 +7674,6 @@ int get_hwpoison_hugetlb_folio(struct folio *folio, bool *hugetlb, bool unpoison return ret; } -int get_huge_page_for_hwpoison(unsigned long pfn, int flags, - bool *migratable_cleared) -{ - int ret; - - spin_lock_irq(&hugetlb_lock); - ret = __get_huge_page_for_hwpoison(pfn, flags, migratable_cleared); - spin_unlock_irq(&hugetlb_lock); - return ret; -} - void folio_putback_active_hugetlb(struct folio *folio) { spin_lock_irq(&hugetlb_lock); diff --git a/mm/memory-failure.c b/mm/memory-failure.c index e42d0cd7888d..fe9d2a0cb962 100644 --- a/mm/memory-failure.c +++ b/mm/memory-failure.c @@ -1984,13 +1984,15 @@ void folio_clear_hugetlb_hwpoison(struct folio *folio) * -EBUSY - the hugepage is busy (try to retry) * -EHWPOISON - the hugepage is already hwpoisoned */ -int __get_huge_page_for_hwpoison(unsigned long pfn, int flags, +static int get_huge_page_for_hwpoison(unsigned long pfn, int flags, bool *migratable_cleared) { struct folio *folio; int ret = 2; /* fallback to normal page handling */ bool count_increased = false; + spin_lock_irq(&hugetlb_lock); + folio = hwpoison_pfn_folio(pfn); if (!folio) goto out; @@ -2024,10 +2026,12 @@ int __get_huge_page_for_hwpoison(unsigned long pfn, int flags, *migratable_cleared = true; } - return ret; + goto unlock; out: if (count_increased) folio_put(folio); +unlock: + spin_unlock_irq(&hugetlb_lock); return ret; } From patchwork Fri Mar 1 21:47:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13579162 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6760CC54E4A for ; Fri, 1 Mar 2024 21:47:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6DD896B009B; Fri, 1 Mar 2024 16:47:17 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 66BB46B009F; Fri, 1 Mar 2024 16:47:17 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4D84F6B009D; Fri, 1 Mar 2024 16:47:17 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 3F2456B009B for ; Fri, 1 Mar 2024 16:47:17 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 119BD8012C for ; Fri, 1 Mar 2024 21:47:17 +0000 (UTC) X-FDA: 81849806514.05.DA9B004 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf25.hostedemail.com (Postfix) with ESMTP id 92E15A000E for ; Fri, 1 Mar 2024 21:47:15 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Bi6Jm4kc; dmarc=none; spf=none (imf25.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709329635; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=F8cxoSYHCxUiA/zgEJyY7ynTvnpkxtkAiF+Pei7dmM4=; b=PLeUVlI+CdMq2AhnsgNSL0VFuYcp8FBUNSGs4T8KJKYo8+THxqHzLssKwBkXHhe9+zczrd Bb//n0D6Ii9SdHFYZ8huas6psjpzEFdr7C9Bgew7YLzgJyFfGg/nHdmkrSIIK5zecVtA0j QqPOWieyMwO5IkeErXp8gRF+NBGqYAI= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Bi6Jm4kc; dmarc=none; spf=none (imf25.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709329635; a=rsa-sha256; cv=none; b=fcW3B+4lx3aFBCd3bVLWK1PQoajTKYjvX66LIZzV54d0kRX0eKDoff0veqOc7fVDeRKcQu iySlJqeq7Q8tZPKjA8JgMos6UKPmiCv+s2SAQJ14iOFAjhZvmYLYHwWdsGTUFPpF8KiMe8 JwHLpLrNc0TOejl8INikfH+T/2NSFfA= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=F8cxoSYHCxUiA/zgEJyY7ynTvnpkxtkAiF+Pei7dmM4=; b=Bi6Jm4kc8KkQNzDOW3clachwrc nMJCuqTG2pV86TVGZFVfaQ9H99Dc7fpYVvZgKNFiN1rTmeKNXvpQV0+vjifUGzD32ZcTjWtNvoopc wggsbh/bo7qxOQf12VB7CeWebC+mZQs7LYIsliUe5HLzgNQHgM/ZE49ZtRAjLeLwbgmD5XFZ7+eOW 0+lC/PDJu/+iFYHCxInZ7nuLcKxJhtGsMLA4KdNxSegag7Vj6y3Pmc4LAf8hKqV9Gq/Hsq+fR1J8k TJER4b9NUiOqNrXLwC4g5tVk7HCLUJ0B1FUKj2Q+dbV3GRn4w9xguuAtKmuAHdK20+AeNIGJJ+wAT BwqT7VCQ==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rgAin-0000000ByEw-3rRH; Fri, 01 Mar 2024 21:47:13 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , Oscar Salvador Subject: [PATCH 5/5] compaction: Use hugetlb_pfn_folio in isolate_migratepages_block Date: Fri, 1 Mar 2024 21:47:10 +0000 Message-ID: <20240301214712.2853147-6-willy@infradead.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240301214712.2853147-1-willy@infradead.org> References: <20240301214712.2853147-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 92E15A000E X-Stat-Signature: 1cyubfzmdb6w88yjhm4uj998h1msmcy8 X-HE-Tag: 1709329635-831103 X-HE-Meta: U2FsdGVkX1/hN/2sWhV+fyPG+NEeZ6TfqfIrUG4CFDZgy+A0faHuI5b5gPJkwh+uX7d8zV0YRySongLtEQacIGmpamM233ZCLFk0wvokQUFDlL8HI+qqDH2yO+Pzh+Jd1PPlokKd3rkujzzDKm3DUoqoSE02ppMNBHBHhzkRtgKWHOfqGTfHyNhIp8PFiEgo2V4ztiuYdO/vI8oznuUJh2W4fEc1Q6I6pEbyuDhxLJ1tVj5n4ifV5D602ccgMxH9mA3m5OS6es+gJ6Dz2m1suhDGPQS+Sno111uOeXxFOoYjUtlXudlnq1QhOPkXw2ueRYdBFKnRTEEnmwanEQE9FPhvQV/DBJlFQfSL2amwnmc+ZjpRJCxtJuymu/+qX2cFlP66ZKQlIeTeghDiJQ6X5A0q7jVUymRW50bcnwtpJTwXdhdm7xDWXZJDPYhYtLAgFASMY3KDNyaA2hv5hAwFPKqGgk5uo/RdLfu5ljW2Hvz6KAfItd/FVFbklnLy6D2MQLSo+NkdxhIvH155HYmQaBK9cgMOfVRcscM+zOWgI5Eansnf2vxlvc3EDvd8N7KURAGmy1kWIu+iAD89luxGDYDs9FZe+usrLq2WtZxLZRaDpnse+d0hLsmxwlr1eXX9oOOrdTHgd0rCc98DES+tt7d7BATZP+8m1jzfRHYKXM/99vyfXukrT9lzfBo462DncyJ4QNfalJbg5XFSpCbE01kNFZiBvbnSUQe8aOSZky2Prq7ZCg2LycQ1kEwamPj0AHVU2xm9yoroxc5ZjiwbU+N1TmH0HjglT2zv/P43TXIVlUsUy7mfSsa//1yHgL+fpJ9NHax1wkBpSA3/yw0sPafR0WvEkur/7JndLM1tR4Ehdl9yTNzaITK0VclkgXCDfrfnMCu5yjkh6B8XDCuWSLBfAYIWWuhl/rCwnbISj7Q= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This is more efficient than calling page_folio() ourselves, and removes a call to PageHuge() which we're trying to get rid of. We do not hold the hugetlb_lock here, so the folio may be dissolved under us, but we check for that in hugetlb_isolate_or_dissolve() (which now takes a folio argument instead of converting to a folio itself). Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/hugetlb.h | 4 ++-- mm/compaction.c | 16 ++++++++-------- mm/hugetlb.c | 10 +++++----- 3 files changed, 15 insertions(+), 15 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 89f4b90eec68..436086a300f5 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -736,7 +736,7 @@ struct huge_bootmem_page { struct hstate *hstate; }; -int isolate_or_dissolve_huge_page(struct page *page, struct list_head *list); +int hugetlb_isolate_or_dissolve(struct folio *folio, struct list_head *list); struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, unsigned long addr, int avoid_reserve); struct folio *alloc_hugetlb_folio_nodemask(struct hstate *h, int preferred_nid, @@ -1043,7 +1043,7 @@ static inline struct folio *filemap_lock_hugetlb_folio(struct hstate *h, return NULL; } -static inline int isolate_or_dissolve_huge_page(struct page *page, +static inline int hugetlb_isolate_or_dissolve(struct folio *folio, struct list_head *list) { return -ENOMEM; diff --git a/mm/compaction.c b/mm/compaction.c index 807b58e6eb68..ad3430c22ede 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -1001,17 +1001,18 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, valid_page = page; } - if (PageHuge(page)) { + folio = hugetlb_pfn_folio(low_pfn); + if (folio) { /* * skip hugetlbfs if we are not compacting for pages * bigger than its order. THPs and other compound pages * are handled below. */ if (!cc->alloc_contig) { - const unsigned int order = compound_order(page); + const unsigned int order = folio_order(folio); if (order <= MAX_PAGE_ORDER) { - low_pfn += (1UL << order) - 1; + low_pfn |= (1UL << order) - 1; nr_scanned += (1UL << order) - 1; } goto isolate_fail; @@ -1022,7 +1023,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, locked = NULL; } - ret = isolate_or_dissolve_huge_page(page, &cc->migratepages); + ret = hugetlb_isolate_or_dissolve(folio, &cc->migratepages); /* * Fail isolation in case isolate_or_dissolve_huge_page() @@ -1032,17 +1033,16 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, /* Do not report -EBUSY down the chain */ if (ret == -EBUSY) ret = 0; - low_pfn += compound_nr(page) - 1; - nr_scanned += compound_nr(page) - 1; + low_pfn += folio_nr_pages(folio) - 1; + nr_scanned += folio_nr_pages(folio) - 1; goto isolate_fail; } - if (PageHuge(page)) { + if (folio_test_hugetlb(folio)) { /* * Hugepage was successfully isolated and placed * on the cc->migratepages list. */ - folio = page_folio(page); low_pfn += folio_nr_pages(folio) - 1; goto isolate_success_no_list; } diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 2e6ebedb75a8..7832b0730e80 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3160,16 +3160,16 @@ static int alloc_and_dissolve_hugetlb_folio(struct hstate *h, return ret; } -int isolate_or_dissolve_huge_page(struct page *page, struct list_head *list) +int hugetlb_isolate_or_dissolve(struct folio *folio, struct list_head *list) { struct hstate *h; - struct folio *folio = page_folio(page); int ret = -EBUSY; /* - * The page might have been dissolved from under our feet, so make sure - * to carefully check the state under the lock. - * Return success when racing as if we dissolved the page ourselves. + * The folio might have been dissolved from under our feet, + * so make sure to carefully check the state under the lock. + * Return success when racing as if we dissolved the folio + * ourselves. */ spin_lock_irq(&hugetlb_lock); if (folio_test_hugetlb(folio)) {