From patchwork Thu Mar 21 14:24:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13598779 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 04769C54E58 for ; Thu, 21 Mar 2024 14:25:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 585CA6B0099; Thu, 21 Mar 2024 10:25:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 50FEF6B009A; Thu, 21 Mar 2024 10:25:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2EE566B009B; Thu, 21 Mar 2024 10:25:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 19D886B0099 for ; Thu, 21 Mar 2024 10:25:18 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id D7C2BC160F for ; Thu, 21 Mar 2024 14:25:17 +0000 (UTC) X-FDA: 81921268674.23.4F18708 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf02.hostedemail.com (Postfix) with ESMTP id 1289A80005 for ; Thu, 21 Mar 2024 14:25:15 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=rG6FPrBm; dmarc=none; spf=none (imf02.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711031116; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YSXRYNjFOMopeMCyQudR+fTuGGBUc9i8jkm6+jE9EkQ=; b=Y+Et24czPn70/TMyAdfx5gPUeqYrJOYrsT2Gtv0kwhHClj+2eBBh5uhn4k1Gk3AxlRUQuN atEGB0zYLEAibmXgI+xehQinvoTmjk4g78rf3otcdRRBClnWDsG2BYJOhdckPUaSRQbJvs aVM6qxO6vEzALClr6MwHyMFTHHNYlmQ= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=rG6FPrBm; dmarc=none; spf=none (imf02.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711031116; a=rsa-sha256; cv=none; b=qlqZ2b4KaJ9YAefX7i1UwtXC4hPlgBXzFBxUY79LLHNB8BB+mo5eDYQodGZm2r1qCsZHDt rNLWcilfqGSeE3xfsFGshNbKzr9s3u5YdIv4XivVm/PiZp++r6u5cOuusooNMgqfassyUC D3JYHg2R386EsSg36G8BqJMNdGIJG1k= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=YSXRYNjFOMopeMCyQudR+fTuGGBUc9i8jkm6+jE9EkQ=; b=rG6FPrBmd1O5jT8zp4YGxsfqjK pJAqGO1dH5yRnC8dzPWMuh63NMP2CcMYqUjGsrT4iyIeyhisq3To8X/GRrI2WZrcjsBfaSxHaJUb7 WGUzkL0Yon/lAtdP/M2WyzSzxPYkY1PQmz5KtHaX1glDMUabi11CZ+fDruOEZjP2EFR4D+LYLLUYL VF3D9Esoo6Y8HV8rPanb/ViJ1LhqKbPDtMdIROHhP4yssmRTk3enfbiak2nWGWN+rjyKy4vIcAF47 pCEloN8T0O/IHN1zrDeIaaFYI351F94ZOr6wveP88RtgBJ7u1VYXrtViINApocLYHGx+0BMJu0cJQ ffubHvtw==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rnJLf-00000006u42-2m9a; Thu, 21 Mar 2024 14:24:51 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, David Hildenbrand , Vlastimil Babka , Miaohe Lin , Muchun Song , Oscar Salvador Subject: [PATCH 5/9] mm: Turn folio_test_hugetlb into a PageType Date: Thu, 21 Mar 2024 14:24:43 +0000 Message-ID: <20240321142448.1645400-6-willy@infradead.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240321142448.1645400-1-willy@infradead.org> References: <20240321142448.1645400-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: h1o3zgffaru8gmp6f8do5bdnwiefddyq X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 1289A80005 X-HE-Tag: 1711031115-264285 X-HE-Meta: U2FsdGVkX1/I8qqhZs2z/8iQ+mj4VfjFZrzRAtUxTWov537jzz1Iig2PkQTqo+Nvxr/3ncVRk71J3ve3sUQk3y3EZK0AazNHIPsdDXmqtxk0/h6BZh1GFpgavpMHtXInOMn7xHOIpx3qcRJz0UbWf5JVKqJq59tGP5zlpGxLInBzLjoxhwwM8o66ZyphDXp0HvYAvmfNMOfzYZ9iuKbjE6RvLkQR/LzJ/MIyoTLxFEjfsIhnG3+I4CWx0iSfUgNOgn3o699DMW5ek4lI/mLGn85Neq9xp5FH86qVijG0CL0QgI6dz5CnugYgMa12zCivivVdCqoMGBR4/5+jnbAuWK4w9gYf9SzLoZgDH96E7zaPXvi3SVFT0a1Ju8HoUeUYWnJKE5akIZNIhaFuflSSX/yffbg1O14Jihueex52Bf+5rVmlOIqt6kWSIb3AyikLJVjvpUqWNHx1B6nH8Fxx3wYIyTrap0gtxutxGM4kAO6rfUTgTmmfa/5bfCTJ2zKX/1SLoEHPuXdpgMiBeZXDaTFHy6+grnrzF/9chXUYAZEGdU+JXSCI8+dwbnvZ2Xe4jj4f92VSmL6MfSbmVPZi4SLaBvayMVS3BymL3XsOFBLVjISnIdsWckbiRos3u9eD2Gu5WE/K0AQ0Y5GdI4PLc+dXdEHrZXHmDpF56BcVJ0PzQuMQI9xrwVdrhHK8UD0a21bFyjHLfKqkGcT7gXIjBlc09Jh3GzqYUaoEHWWRZcpzEMHNCw9YOAh9nVfg2XPRhLOlh1fi45vdQ/FX/6XL5GaB2GbHCPgt3zS4USlcV6szvPgPF51Ng0raXIcLbI5/lYS2z77M3slWceqLU5IjGQFVyNQu7eNP/nM2FhZWZjhg4DU8gp7VRYRroS0MUcpA531r5gcGRGjgQ/K3NbTMHLlRPS3RU+x5C3N+sG3Z1rkXrvjkGqMXQgB6svssJpVYPAp2ZITvV7ew+c0WPVS 6orS5efy k0Ct8ddNbiVwAC0PUbLFd6w6fMe1rK3zrVmKAEcarTkBLpVfI+JydWOZkFu3qF4IAYlFJw9sZsslQ/FEbWHrtXZ8/21B959Mnr88DTzXllrTjMneva7jXKFAMtuetvxvkdwb5t8v0dybJCDmnC2vYmF0eV/n3bzGYNHDY6UFYJ6u/eUm6wvyxLMF1/ibbg5OWKk2t29M5Yvgr/rnkDA64OsBiKK93jB8mRec77xeHTHoWjRFkbTdN5CKfX6fLNjcXCaIB X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The current folio_test_hugetlb() can be fooled by a concurrent folio split into returning true for a folio which has never belonged to hugetlbfs. This can't happen if the caller holds a refcount on it, but we have a few places (memory-failure, compaction, procfs) which do not and should not take a speculative reference. Since hugetlb pages do not use individual page mapcounts (they are always fully mapped and use the entire_mapcount field to record the number of mappings), the PageType field is available now that page_mapcount() ignores the value in this field. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: David Hildenbrand Acked-by: Vlastimil Babka Reported-by: Luis Chamberlain --- include/linux/page-flags.h | 70 ++++++++++++++++------------------ include/trace/events/mmflags.h | 1 + mm/hugetlb.c | 22 ++--------- 3 files changed, 37 insertions(+), 56 deletions(-) diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 5852f967c640..6fb3cd42ee59 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -190,7 +190,6 @@ enum pageflags { /* At least one page in this folio has the hwpoison flag set */ PG_has_hwpoisoned = PG_error, - PG_hugetlb = PG_active, PG_large_rmappable = PG_workingset, /* anon or file-backed */ }; @@ -876,29 +875,6 @@ FOLIO_FLAG_FALSE(large_rmappable) #define PG_head_mask ((1UL << PG_head)) -#ifdef CONFIG_HUGETLB_PAGE -int PageHuge(const struct page *page); -SETPAGEFLAG(HugeTLB, hugetlb, PF_SECOND) -CLEARPAGEFLAG(HugeTLB, hugetlb, PF_SECOND) - -/** - * folio_test_hugetlb - Determine if the folio belongs to hugetlbfs - * @folio: The folio to test. - * - * Context: Any context. Caller should have a reference on the folio to - * prevent it from being turned into a tail page. - * Return: True for hugetlbfs folios, false for anon folios or folios - * belonging to other filesystems. - */ -static inline bool folio_test_hugetlb(const struct folio *folio) -{ - return folio_test_large(folio) && - test_bit(PG_hugetlb, const_folio_flags(folio, 1)); -} -#else -TESTPAGEFLAG_FALSE(Huge, hugetlb) -#endif - #ifdef CONFIG_TRANSPARENT_HUGEPAGE /* * PageHuge() only returns true for hugetlbfs pages, but not for @@ -954,18 +930,6 @@ PAGEFLAG_FALSE(HasHWPoisoned, has_hwpoisoned) TESTSCFLAG_FALSE(HasHWPoisoned, has_hwpoisoned) #endif -/* - * Check if a page is currently marked HWPoisoned. Note that this check is - * best effort only and inherently racy: there is no way to synchronize with - * failing hardware. - */ -static inline bool is_page_hwpoison(struct page *page) -{ - if (PageHWPoison(page)) - return true; - return PageHuge(page) && PageHWPoison(compound_head(page)); -} - /* * For pages that are never mapped to userspace (and aren't PageSlab), * page_type may be used. Because it is initialised to -1, we invert the @@ -982,6 +946,7 @@ static inline bool is_page_hwpoison(struct page *page) #define PG_offline 0x00000100 #define PG_table 0x00000200 #define PG_guard 0x00000400 +#define PG_hugetlb 0x00000800 #define PageType(page, flag) \ ((page->page_type & (PAGE_TYPE_BASE | flag)) == PAGE_TYPE_BASE) @@ -1076,6 +1041,37 @@ PAGE_TYPE_OPS(Table, table, pgtable) */ PAGE_TYPE_OPS(Guard, guard, guard) +#ifdef CONFIG_HUGETLB_PAGE +FOLIO_TYPE_OPS(hugetlb, hugetlb) +#else +FOLIO_TEST_FLAG_FALSE(hugetlb) +#endif + +/** + * PageHuge - Determine if the page belongs to hugetlbfs + * @page: The page to test. + * + * Context: Any context. + * Return: True for hugetlbfs pages, false for anon pages or pages + * belonging to other filesystems. + */ +static inline bool PageHuge(const struct page *page) +{ + return folio_test_hugetlb(page_folio(page)); +} + +/* + * Check if a page is currently marked HWPoisoned. Note that this check is + * best effort only and inherently racy: there is no way to synchronize with + * failing hardware. + */ +static inline bool is_page_hwpoison(struct page *page) +{ + if (PageHWPoison(page)) + return true; + return PageHuge(page) && PageHWPoison(compound_head(page)); +} + extern bool is_free_buddy_page(struct page *page); PAGEFLAG(Isolated, isolated, PF_ANY); @@ -1142,7 +1138,7 @@ static __always_inline void __ClearPageAnonExclusive(struct page *page) */ #define PAGE_FLAGS_SECOND \ (0xffUL /* order */ | 1UL << PG_has_hwpoisoned | \ - 1UL << PG_hugetlb | 1UL << PG_large_rmappable) + 1UL << PG_large_rmappable) #define PAGE_FLAGS_PRIVATE \ (1UL << PG_private | 1UL << PG_private_2) diff --git a/include/trace/events/mmflags.h b/include/trace/events/mmflags.h index d801409b33cf..d55e53ac91bd 100644 --- a/include/trace/events/mmflags.h +++ b/include/trace/events/mmflags.h @@ -135,6 +135,7 @@ IF_HAVE_PG_ARCH_X(arch_3) #define DEF_PAGETYPE_NAME(_name) { PG_##_name, __stringify(_name) } #define __def_pagetype_names \ + DEF_PAGETYPE_NAME(hugetlb), \ DEF_PAGETYPE_NAME(offline), \ DEF_PAGETYPE_NAME(guard), \ DEF_PAGETYPE_NAME(table), \ diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 7e9a766059aa..bdcbb62096cf 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1624,7 +1624,7 @@ static inline void __clear_hugetlb_destructor(struct hstate *h, { lockdep_assert_held(&hugetlb_lock); - folio_clear_hugetlb(folio); + __folio_clear_hugetlb(folio); } /* @@ -1711,7 +1711,7 @@ static void add_hugetlb_folio(struct hstate *h, struct folio *folio, h->surplus_huge_pages_node[nid]++; } - folio_set_hugetlb(folio); + __folio_set_hugetlb(folio); folio_change_private(folio, NULL); /* * We have to set hugetlb_vmemmap_optimized again as above @@ -2050,7 +2050,7 @@ static void __prep_account_new_huge_page(struct hstate *h, int nid) static void init_new_hugetlb_folio(struct hstate *h, struct folio *folio) { - folio_set_hugetlb(folio); + __folio_set_hugetlb(folio); INIT_LIST_HEAD(&folio->lru); hugetlb_set_folio_subpool(folio, NULL); set_hugetlb_cgroup(folio, NULL); @@ -2160,22 +2160,6 @@ static bool prep_compound_gigantic_folio_for_demote(struct folio *folio, return __prep_compound_gigantic_folio(folio, order, true); } -/* - * PageHuge() only returns true for hugetlbfs pages, but not for normal or - * transparent huge pages. See the PageTransHuge() documentation for more - * details. - */ -int PageHuge(const struct page *page) -{ - const struct folio *folio; - - if (!PageCompound(page)) - return 0; - folio = page_folio(page); - return folio_test_hugetlb(folio); -} -EXPORT_SYMBOL_GPL(PageHuge); - /* * Find and lock address space (mapping) in write mode. *