From patchwork Wed Jan 11 14:28:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13096757 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A01E0C46467 for ; Wed, 11 Jan 2023 14:29:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3FB4A8E0002; Wed, 11 Jan 2023 09:29:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C36868E0008; Wed, 11 Jan 2023 09:29:09 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8B28B900003; Wed, 11 Jan 2023 09:29:09 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 3387C8E000A for ; Wed, 11 Jan 2023 09:29:09 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id D7A8EAB337 for ; Wed, 11 Jan 2023 14:29:08 +0000 (UTC) X-FDA: 80342750376.08.4EE33D5 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf03.hostedemail.com (Postfix) with ESMTP id 35A6520004 for ; Wed, 11 Jan 2023 14:29:06 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=mrQ7M36+; dmarc=none; spf=none (imf03.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673447347; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=T9gQL9FPQQL45CgTP3zUp/wqTgVsvP9BYcSSsl4zn3o=; b=Uk/qeAuoP/uS/LzkqWbFqJ9/hHUVyZxRHgYrEvYgOCVJ3+Gln7iIR0OF/7Sw1cXzpQT2oW SMV4RycbncWVq9JpX/Mw+ymelP2WhrhRiFDsqEMmz0FkqccPFyNVs40DNdMFOdCoc53fPa 4wTv8bf9bwkC8BwwaLQba1tUQOTfkTY= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=mrQ7M36+; dmarc=none; spf=none (imf03.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673447347; a=rsa-sha256; cv=none; b=uLTTm+t+wfBO2vfodpG/HynrDFNcoZ7npIdiBmdKFlXXIceUN4z3ohO2prQUwww7FXZ2HQ hjx+8/xj6cOTxbTz4587iuJjgiViMlLyujReiZ9e0XuXslbltwIbQyvZwvuo7MWzx6cXvH KCQ3sO+kXhb1cvkQH+Rtq2m4BZcaO40= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=T9gQL9FPQQL45CgTP3zUp/wqTgVsvP9BYcSSsl4zn3o=; b=mrQ7M36+jB5FuEevCuLuNYxuVJ /KNWPKC5V/hABAJE3V75XBeibCeG6ud94kkqOAxBRxeDVgw1a7Y3PUcbAU+seDFOeuhs0nX7w+9Vr j6mXa0CoZa1Jm9FyLwKaAdbta99tb6kt11DLJ72SgGkZGjIoCBDW9O1d4F8inWSmKiNMkWNA/NPZB K8FAiRT9nCyE58Z8T6rfD7U0uKkL+YxhEaOqrhM9KDKbLblc8VMX1Xfj6pCBZhxacv0WBBHCiItsE 6H+HrdhHuYmXawo+svjWaLT/M46f/OqzoR5vqPLGcQN8U9d47eqLVV7gr9kF21VcMUogMLTP/o9lf wgITzIkA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pFc6O-004CXu-ON; Wed, 11 Jan 2023 14:29:16 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Hugh Dickins Subject: [PATCH 01/28] mm: Remove folio_pincount_ptr() and head_compound_pincount() Date: Wed, 11 Jan 2023 14:28:47 +0000 Message-Id: <20230111142915.1001531-2-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230111142915.1001531-1-willy@infradead.org> References: <20230111142915.1001531-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 35A6520004 X-Stat-Signature: w3j7chmy8nfnbsqz3qn637en9h6r9ghr X-HE-Tag: 1673447346-583577 X-HE-Meta: U2FsdGVkX19Y+4L3G01iWvHxzP0pOS0+BA4qS8iz7ojnfxKnWW8d6IEuXqBBSJm7j8VBgnzNzJgdDQiVmbSmtIiVv/WfdoIljbLdDOsf+IbWmmkTgZA0j1wvryUSLfa6tpxgjsVTh5FBOJXCzBvP/2urjrv60WKNNdUOtszVywBnF2iiWsKKopIFbaBd5C8gRg5J9rD91vwGB8BhVbObRefpfXGIMzDfAPjt4chRKRNOe9hu+b64GSYFLelodaVG+QR4i7J91OFY+6NfQa+3IKDwiRY1m87nObmrcCOMe32DkkD8q5H5wtnnBRZetdEvRRwctqN9r0SkJjMN0fN3GeRnsANOPAm130XP0jiI9RHd0/44dMeqMeOhVGsRa0snAmZX0VF4lOoUxzBHEwQURd2guCssOOPFPpDDPxQlqlqUsTm049YGW7UIuD/pCMp1sstftywbKfPa/R1kEVBueUicnUD2hkxzmCY2ajFLuLwXBEtAVjd5+os93nZzV8MpfNuD/kM9g5GImVdRb5oqO7OSNrsSwcrnuH8kDd6lenF/6WA/0C2LMXBQFFedCtGG3RMlj+3eA6gOVtd/H/WwK6XemddMZJKT+GpGa8fs2Hf1VjVe3kBatiwUESSeENSC02jPep6CtTfCwQzTY9Nx7X8pQ3PUtnuxqrLJpeIbH1vjjmBTLjLJnubLT2WCRdwG3+stnNIgW/bIsNCaXnVsHhwa/pXqFL1ajvCkJj0O6/WHpjcNTS9VGOUbwmsxA+LCFYLDLefSwuZjO3vCb0m3BROttDa2Ts0CsCkpNAQPU0vKF8iA/VDxCKYGGtY/34m4pA6DiUc9RVIOYFnwgm6U8sODHkrOpw9CScii+FG1AtFV5MmpnCn/LQ6WrFdH45UcTImGjBd5mwRsW8oP+OXBwZ80I0Az6vKbTHQUnT9gfQIMVXPyFsakSA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We can use folio->_pincount directly, since all users are guarded by tests of compound/large. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: John Hubbard --- Documentation/core-api/pin_user_pages.rst | 29 +++++++++++------------ include/linux/mm.h | 14 ++--------- include/linux/mm_types.h | 5 ---- mm/debug.c | 4 ++-- mm/gup.c | 8 +++---- mm/huge_memory.c | 4 ++-- mm/hugetlb.c | 4 ++-- mm/page_alloc.c | 9 ++++--- 8 files changed, 32 insertions(+), 45 deletions(-) diff --git a/Documentation/core-api/pin_user_pages.rst b/Documentation/core-api/pin_user_pages.rst index facafbdecb95..9fb0b1080d3b 100644 --- a/Documentation/core-api/pin_user_pages.rst +++ b/Documentation/core-api/pin_user_pages.rst @@ -55,18 +55,17 @@ flags the caller provides. The caller is required to pass in a non-null struct pages* array, and the function then pins pages by incrementing each by a special value: GUP_PIN_COUNTING_BIAS. -For compound pages, the GUP_PIN_COUNTING_BIAS scheme is not used. Instead, -an exact form of pin counting is achieved, by using the 2nd struct page -in the compound page. A new struct page field, compound_pincount, has -been added in order to support this. - -This approach for compound pages avoids the counting upper limit problems that -are discussed below. Those limitations would have been aggravated severely by -huge pages, because each tail page adds a refcount to the head page. And in -fact, testing revealed that, without a separate compound_pincount field, -page overflows were seen in some huge page stress tests. - -This also means that huge pages and compound pages do not suffer +For large folios, the GUP_PIN_COUNTING_BIAS scheme is not used. Instead, +the extra space available in the struct folio is used to store the +pincount directly. + +This approach for large folios avoids the counting upper limit problems +that are discussed below. Those limitations would have been aggravated +severely by huge pages, because each tail page adds a refcount to the +head page. And in fact, testing revealed that, without a separate pincount +field, refcount overflows were seen in some huge page stress tests. + +This also means that huge pages and large folios do not suffer from the false positives problem that is mentioned below.:: Function @@ -264,9 +263,9 @@ place.) Other diagnostics ================= -dump_page() has been enhanced slightly, to handle these new counting -fields, and to better report on compound pages in general. Specifically, -for compound pages, the exact (compound_pincount) pincount is reported. +dump_page() has been enhanced slightly to handle these new counting +fields, and to better report on large folios in general. Specifically, +for large folios, the exact pincount is reported. References ========== diff --git a/include/linux/mm.h b/include/linux/mm.h index 49e40766adb6..5683a25ce08e 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1011,11 +1011,6 @@ static inline void folio_set_compound_dtor(struct folio *folio, void destroy_large_folio(struct folio *folio); -static inline int head_compound_pincount(struct page *head) -{ - return atomic_read(compound_pincount_ptr(head)); -} - static inline void set_compound_order(struct page *page, unsigned int order) { page[1].compound_order = order; @@ -1641,11 +1636,6 @@ static inline struct folio *pfn_folio(unsigned long pfn) return page_folio(pfn_to_page(pfn)); } -static inline atomic_t *folio_pincount_ptr(struct folio *folio) -{ - return &folio_page(folio, 1)->compound_pincount; -} - /** * folio_maybe_dma_pinned - Report if a folio may be pinned for DMA. * @folio: The folio. @@ -1663,7 +1653,7 @@ static inline atomic_t *folio_pincount_ptr(struct folio *folio) * expected to be able to deal gracefully with a false positive. * * For large folios, the result will be exactly correct. That's because - * we have more tracking data available: the compound_pincount is used + * we have more tracking data available: the _pincount field is used * instead of the GUP_PIN_COUNTING_BIAS scheme. * * For more information, please see Documentation/core-api/pin_user_pages.rst. @@ -1674,7 +1664,7 @@ static inline atomic_t *folio_pincount_ptr(struct folio *folio) static inline bool folio_maybe_dma_pinned(struct folio *folio) { if (folio_test_large(folio)) - return atomic_read(folio_pincount_ptr(folio)) > 0; + return atomic_read(&folio->_pincount) > 0; /* * folio_ref_count() is signed. If that refcount overflows, then diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 2b0a0595fc9e..c225d81eae83 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -443,11 +443,6 @@ static inline atomic_t *subpages_mapcount_ptr(struct page *page) return &page[1].subpages_mapcount; } -static inline atomic_t *compound_pincount_ptr(struct page *page) -{ - return &page[1].compound_pincount; -} - /* * Used for sizing the vmemmap region on some architectures */ diff --git a/mm/debug.c b/mm/debug.c index 7f8e5f744e42..893c9dbf76ca 100644 --- a/mm/debug.c +++ b/mm/debug.c @@ -94,11 +94,11 @@ static void __dump_page(struct page *page) page, page_ref_count(head), mapcount, mapping, page_to_pgoff(page), page_to_pfn(page)); if (compound) { - pr_warn("head:%p order:%u compound_mapcount:%d subpages_mapcount:%d compound_pincount:%d\n", + pr_warn("head:%p order:%u compound_mapcount:%d subpages_mapcount:%d pincount:%d\n", head, compound_order(head), head_compound_mapcount(head), head_subpages_mapcount(head), - head_compound_pincount(head)); + atomic_read(&folio->_pincount)); } #ifdef CONFIG_MEMCG diff --git a/mm/gup.c b/mm/gup.c index f45a3a5be53a..38ba1697dd61 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -111,7 +111,7 @@ static inline struct folio *try_get_folio(struct page *page, int refs) * FOLL_GET: folio's refcount will be incremented by @refs. * * FOLL_PIN on large folios: folio's refcount will be incremented by - * @refs, and its compound_pincount will be incremented by @refs. + * @refs, and its pincount will be incremented by @refs. * * FOLL_PIN on single-page folios: folio's refcount will be incremented by * @refs * GUP_PIN_COUNTING_BIAS. @@ -157,7 +157,7 @@ struct folio *try_grab_folio(struct page *page, int refs, unsigned int flags) * try_get_folio() is left intact. */ if (folio_test_large(folio)) - atomic_add(refs, folio_pincount_ptr(folio)); + atomic_add(refs, &folio->_pincount); else folio_ref_add(folio, refs * (GUP_PIN_COUNTING_BIAS - 1)); @@ -182,7 +182,7 @@ static void gup_put_folio(struct folio *folio, int refs, unsigned int flags) if (flags & FOLL_PIN) { node_stat_mod_folio(folio, NR_FOLL_PIN_RELEASED, refs); if (folio_test_large(folio)) - atomic_sub(refs, folio_pincount_ptr(folio)); + atomic_sub(refs, &folio->_pincount); else refs *= GUP_PIN_COUNTING_BIAS; } @@ -232,7 +232,7 @@ int __must_check try_grab_page(struct page *page, unsigned int flags) */ if (folio_test_large(folio)) { folio_ref_add(folio, 1); - atomic_add(1, folio_pincount_ptr(folio)); + atomic_add(1, &folio->_pincount); } else { folio_ref_add(folio, GUP_PIN_COUNTING_BIAS); } diff --git a/mm/huge_memory.c b/mm/huge_memory.c index c13b1f67d14e..9570f03cdee4 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2477,9 +2477,9 @@ static void __split_huge_page_tail(struct page *head, int tail, * of swap cache pages that store the swp_entry_t in tail pages. * Fix up and warn once if private is unexpectedly set. * - * What of 32-bit systems, on which head[1].compound_pincount overlays + * What of 32-bit systems, on which folio->_pincount overlays * head[1].private? No problem: THP_SWAP is not enabled on 32-bit, and - * compound_pincount must be 0 for folio_ref_freeze() to have succeeded. + * pincount must be 0 for folio_ref_freeze() to have succeeded. */ if (!folio_test_swapcache(page_folio(head))) { VM_WARN_ON_ONCE_PAGE(page_tail->private != 0, page_tail); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 273a6522aa4c..15b2707c1600 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1476,7 +1476,7 @@ static void __destroy_compound_gigantic_folio(struct folio *folio, atomic_set(folio_mapcount_ptr(folio), 0); atomic_set(folio_subpages_mapcount_ptr(folio), 0); - atomic_set(folio_pincount_ptr(folio), 0); + atomic_set(&folio->_pincount, 0); for (i = 1; i < nr_pages; i++) { p = folio_page(folio, i); @@ -1998,7 +1998,7 @@ static bool __prep_compound_gigantic_folio(struct folio *folio, } atomic_set(folio_mapcount_ptr(folio), -1); atomic_set(folio_subpages_mapcount_ptr(folio), 0); - atomic_set(folio_pincount_ptr(folio), 0); + atomic_set(&folio->_pincount, 0); return true; out_error: diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 4d9afa1048ea..d1e5ec875fd0 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -775,11 +775,13 @@ void free_compound_page(struct page *page) static void prep_compound_head(struct page *page, unsigned int order) { + struct folio *folio = (struct folio *)page; + set_compound_page_dtor(page, COMPOUND_PAGE_DTOR); set_compound_order(page, order); atomic_set(compound_mapcount_ptr(page), -1); atomic_set(subpages_mapcount_ptr(page), 0); - atomic_set(compound_pincount_ptr(page), 0); + atomic_set(&folio->_pincount, 0); } static void prep_compound_tail(struct page *head, int tail_idx) @@ -1291,6 +1293,7 @@ static inline bool free_page_is_bad(struct page *page) static int free_tail_pages_check(struct page *head_page, struct page *page) { + struct folio *folio = (struct folio *)head_page; int ret = 1; /* @@ -1314,8 +1317,8 @@ static int free_tail_pages_check(struct page *head_page, struct page *page) bad_page(page, "nonzero subpages_mapcount"); goto out; } - if (unlikely(head_compound_pincount(head_page))) { - bad_page(page, "nonzero compound_pincount"); + if (unlikely(atomic_read(&folio->_pincount))) { + bad_page(page, "nonzero pincount"); goto out; } break; From patchwork Wed Jan 11 14:28:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13096777 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 05F3CC46467 for ; Wed, 11 Jan 2023 14:29:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4974190000B; Wed, 11 Jan 2023 09:29:14 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4213F8E000F; Wed, 11 Jan 2023 09:29:14 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2007F90000E; Wed, 11 Jan 2023 09:29:14 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id DCE3790000B for ; Wed, 11 Jan 2023 09:29:13 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id B9056AB392 for ; Wed, 11 Jan 2023 14:29:13 +0000 (UTC) X-FDA: 80342750586.21.307C423 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf30.hostedemail.com (Postfix) with ESMTP id 1F1B380011 for ; Wed, 11 Jan 2023 14:29:11 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=cQyBhdlg; spf=none (imf30.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673447352; a=rsa-sha256; cv=none; b=P6M4fszg+rLIKLkvuCF2TSGozkyZz91dRKv7/Bk4Zqg6dQfoCjeWRHlKnWgw9XDVBuF1Gl yzQRPAzHp6lPrW+kNRGdSws6lxBHk5wGVBYTmiKGX1mXIQzLtiEaEIdiiYYOn962Zxm7Yp 28apypy2si++VJr3JXfjYG/uCZnNmLs= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=cQyBhdlg; spf=none (imf30.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673447352; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=h24dY4ACiNWUVtfL6tx/eGZ3lYIxcBUu5K9dPTzNH2Q=; b=Mw6+NNbSHrxU7NtAgeOKxTpzT026umRde08/F9TCJpOhYALvTrCpEPvXuihWGBbBqhqU0a gBwwkIDMUflCGHtx+b6V2IOMBW1Qb/BM0RGdBZm0ATpd0djAhe6lj2oOmHldxZoFpzGlIU aAC/l+Y0YgYRbG/JezVJI4S+JTAKuo0= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=h24dY4ACiNWUVtfL6tx/eGZ3lYIxcBUu5K9dPTzNH2Q=; b=cQyBhdlggRlKrktuvKGGXKhrC9 R2fgeX3GHnLDQ/E3OpoEaYs1t6wx0Isg7xcfNcfjQsSB/yxn1L1AapDTQYXWQNZdrX5neQUaSDlfN GSTJqddtTfcSi6gcG+eSYf+mSKA3PCVH/OjcuXAdsqP3PeF3X/Q6CMAp7O1kqEL+hATtsoQ/+tVbg iZqhjNVZGVHio3WQB4ofM/nsmCV9Z4ckTpqI2rL3r8owj7HPDu/6s5S6yRlRoN9QxWdwk9rYXiEmG xkXKgx+Qcpmaz9luHV7jSp79zc77wSOgFT04aAufDEu3OIuVvFa/BQeYyZhXjEAscBlmB4k6x2ocr v1huh6mA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pFc6O-004CXw-Qn; Wed, 11 Jan 2023 14:29:16 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Hugh Dickins Subject: [PATCH 02/28] mm: Convert head_subpages_mapcount() into folio_nr_pages_mapped() Date: Wed, 11 Jan 2023 14:28:48 +0000 Message-Id: <20230111142915.1001531-3-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230111142915.1001531-1-willy@infradead.org> References: <20230111142915.1001531-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 1F1B380011 X-Rspamd-Server: rspam01 X-Stat-Signature: 11m7tc5i5kskrpjpx7w63iiyjnjmwbgb X-HE-Tag: 1673447351-206138 X-HE-Meta: U2FsdGVkX1/c1/nV5UQ44JkscOgwo4ARsWoN9yG3tVUtxKAdGUyB+9ubkCxdbw+XXva69P2YXhhEg3bslLZttMg3OpYM9CoR/DmMuL30zA78i7Ha12Ti1cXh8j7aQIxTc9a/lkMa2PKA7vMY52RLRLHqvUYW3s8l3jl7GPriJeKsk+E7EHw+ZLG1EbiVNPgJ7Iz8rHqKFxaqzHnJz+D3z0q92Y7gnX5Nzq3EigcHwK+FL0qNNh3ISdu1zvRJ4wWjOdH+E0I3hZPGVVa6gZ+YrXKXblCR+6w/M+lEoqrLfk36Hz+dNDc5zdo+Jfokz9manntZRk8y9p3QxIWoTb4wpz4MP6g1hx3IIfKBQ55pBwitapDNC8mMpqWsY9Q3Yr9EcsAEFVy6UGMHsKDiYRhmkdY44p5PQmkqEfVb/IJWILqH9ZcEFbRow1ANcPM1dVg+2J7miduC5w+GqSEtTYwblZLWC58ptGvG/ISEBWISk0dqucOYz0N2jutGw1YGsdauhVtfKqs0ZBSrsmIr8WKpQyD1I4Om0BJaqt6MHt4O62i1if3MkNihdoRbUYmZmXXzo7HXw66jbzE3ug0yEmhYwAVSLnAb3xanGpEs48A0dcuuQwwwFN8aADyvy8l35AaIFDZ8Vcm91/EWC2x/P0jjU3MWi7AA7oEmNCvbPQNTvhjTb2LWKXNuT/B//jw918zqByy3JEc6eyuFvX4V1QmbbopitQ9qMFq/DxPqKW3KRGniDkng0EJhVETafM7BAD1agv4UQmagXSXLSVi1EaSPbkHHFT32AJSs2KleVsF7AppFdYwLVsy6mCFaQD5Ja5e6at4uCMu+KeUeegDNpxjiyEl/gZMIWqGhmFJAPPb8kilEty/tPZN0XIVCLFY0G8KBzVJnepOLPrSDavRZ5j+aL3UfSrFoS0b+jxqnUkqTReeaPz42cUSv8A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Calling this 'mapcount' is confusing since mapcount is usually the number of times something is mapped; instead this is the number of mapped pages. It's also better to enforce that this is a folio rather than a head page. Move folio_nr_pages_mapped() into mm/internal.h since this is not something we want device drivers or filesystems poking at. Get rid of folio_subpages_mapcount_ptr() and use folio->_nr_pages_mapped directly. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/mm.h | 22 ++-------------------- include/linux/mm_types.h | 12 +++--------- mm/debug.c | 4 ++-- mm/hugetlb.c | 4 ++-- mm/internal.h | 18 ++++++++++++++++++ mm/rmap.c | 9 +++++---- 6 files changed, 32 insertions(+), 37 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 5683a25ce08e..aa151e69416d 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -843,24 +843,6 @@ static inline int head_compound_mapcount(struct page *head) return atomic_read(compound_mapcount_ptr(head)) + 1; } -/* - * If a 16GB hugetlb page were mapped by PTEs of all of its 4kB sub-pages, - * its subpages_mapcount would be 0x400000: choose the COMPOUND_MAPPED bit - * above that range, instead of 2*(PMD_SIZE/PAGE_SIZE). Hugetlb currently - * leaves subpages_mapcount at 0, but avoid surprise if it participates later. - */ -#define COMPOUND_MAPPED 0x800000 -#define SUBPAGES_MAPPED (COMPOUND_MAPPED - 1) - -/* - * Number of sub-pages mapped by PTE, does not include compound mapcount. - * Must be called only on head of compound page. - */ -static inline int head_subpages_mapcount(struct page *head) -{ - return atomic_read(subpages_mapcount_ptr(head)) & SUBPAGES_MAPPED; -} - /* * The atomic page->_mapcount, starts from -1: so that transitions * both from it and to it can be tracked, using atomic_inc_and_test @@ -920,9 +902,9 @@ static inline bool folio_large_is_mapped(struct folio *folio) { /* * Reading folio_mapcount_ptr() below could be omitted if hugetlb - * participated in incrementing subpages_mapcount when compound mapped. + * participated in incrementing nr_pages_mapped when compound mapped. */ - return atomic_read(folio_subpages_mapcount_ptr(folio)) > 0 || + return atomic_read(&folio->_nr_pages_mapped) > 0 || atomic_read(folio_mapcount_ptr(folio)) >= 0; } diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index c225d81eae83..aa2039f18f4d 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -307,7 +307,7 @@ static inline struct page *encoded_page_ptr(struct encoded_page *page) * @_folio_dtor: Which destructor to use for this folio. * @_folio_order: Do not use directly, call folio_order(). * @_compound_mapcount: Do not use directly, call folio_entire_mapcount(). - * @_subpages_mapcount: Do not use directly, call folio_mapcount(). + * @_nr_pages_mapped: Do not use directly, call folio_mapcount(). * @_pincount: Do not use directly, call folio_maybe_dma_pinned(). * @_folio_nr_pages: Do not use directly, call folio_nr_pages(). * @_flags_2: For alignment. Do not use. @@ -361,7 +361,7 @@ struct folio { unsigned char _folio_dtor; unsigned char _folio_order; atomic_t _compound_mapcount; - atomic_t _subpages_mapcount; + atomic_t _nr_pages_mapped; atomic_t _pincount; #ifdef CONFIG_64BIT unsigned int _folio_nr_pages; @@ -404,7 +404,7 @@ FOLIO_MATCH(compound_head, _head_1); FOLIO_MATCH(compound_dtor, _folio_dtor); FOLIO_MATCH(compound_order, _folio_order); FOLIO_MATCH(compound_mapcount, _compound_mapcount); -FOLIO_MATCH(subpages_mapcount, _subpages_mapcount); +FOLIO_MATCH(subpages_mapcount, _nr_pages_mapped); FOLIO_MATCH(compound_pincount, _pincount); #ifdef CONFIG_64BIT FOLIO_MATCH(compound_nr, _folio_nr_pages); @@ -427,12 +427,6 @@ static inline atomic_t *folio_mapcount_ptr(struct folio *folio) return &tail->compound_mapcount; } -static inline atomic_t *folio_subpages_mapcount_ptr(struct folio *folio) -{ - struct page *tail = &folio->page + 1; - return &tail->subpages_mapcount; -} - static inline atomic_t *compound_mapcount_ptr(struct page *page) { return &page[1].compound_mapcount; diff --git a/mm/debug.c b/mm/debug.c index 893c9dbf76ca..8e58e8dab0b2 100644 --- a/mm/debug.c +++ b/mm/debug.c @@ -94,10 +94,10 @@ static void __dump_page(struct page *page) page, page_ref_count(head), mapcount, mapping, page_to_pgoff(page), page_to_pfn(page)); if (compound) { - pr_warn("head:%p order:%u compound_mapcount:%d subpages_mapcount:%d pincount:%d\n", + pr_warn("head:%p order:%u compound_mapcount:%d nr_pages_mapped:%d pincount:%d\n", head, compound_order(head), head_compound_mapcount(head), - head_subpages_mapcount(head), + folio_nr_pages_mapped(folio), atomic_read(&folio->_pincount)); } diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 15b2707c1600..c9702224931c 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1475,7 +1475,7 @@ static void __destroy_compound_gigantic_folio(struct folio *folio, struct page *p; atomic_set(folio_mapcount_ptr(folio), 0); - atomic_set(folio_subpages_mapcount_ptr(folio), 0); + atomic_set(&folio->_nr_pages_mapped, 0); atomic_set(&folio->_pincount, 0); for (i = 1; i < nr_pages; i++) { @@ -1997,7 +1997,7 @@ static bool __prep_compound_gigantic_folio(struct folio *folio, set_compound_head(p, &folio->page); } atomic_set(folio_mapcount_ptr(folio), -1); - atomic_set(folio_subpages_mapcount_ptr(folio), 0); + atomic_set(&folio->_nr_pages_mapped, 0); atomic_set(&folio->_pincount, 0); return true; diff --git a/mm/internal.h b/mm/internal.h index f04b8fb57d90..57b4a6992ed4 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -52,6 +52,24 @@ struct folio_batch; void page_writeback_init(void); +/* + * If a 16GB hugetlb folio were mapped by PTEs of all of its 4kB pages, + * its nr_pages_mapped would be 0x400000: choose the COMPOUND_MAPPED bit + * above that range, instead of 2*(PMD_SIZE/PAGE_SIZE). Hugetlb currently + * leaves nr_pages_mapped at 0, but avoid surprise if it participates later. + */ +#define COMPOUND_MAPPED 0x800000 +#define FOLIO_PAGES_MAPPED (COMPOUND_MAPPED - 1) + +/* + * How many individual pages have an elevated _mapcount. Excludes + * the folio's entire_mapcount. + */ +static inline int folio_nr_pages_mapped(struct folio *folio) +{ + return atomic_read(&folio->_nr_pages_mapped) & FOLIO_PAGES_MAPPED; +} + static inline void *folio_raw_mapping(struct folio *folio) { unsigned long mapping = (unsigned long)folio->mapping; diff --git a/mm/rmap.c b/mm/rmap.c index 7bc3b1fa7bc7..f08685054d0a 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1080,12 +1080,13 @@ int pfn_mkclean_range(unsigned long pfn, unsigned long nr_pages, pgoff_t pgoff, int total_compound_mapcount(struct page *head) { + struct folio *folio = (struct folio *)head; int mapcount = head_compound_mapcount(head); int nr_subpages; int i; /* In the common case, avoid the loop when no subpages mapped by PTE */ - if (head_subpages_mapcount(head) == 0) + if (folio_nr_pages_mapped(folio) == 0) return mapcount; /* * Add all the PTE mappings of those subpages mapped by PTE. @@ -1233,7 +1234,7 @@ void page_add_anon_rmap(struct page *page, nr = atomic_add_return_relaxed(COMPOUND_MAPPED, mapped); if (likely(nr < COMPOUND_MAPPED + COMPOUND_MAPPED)) { nr_pmdmapped = thp_nr_pages(page); - nr = nr_pmdmapped - (nr & SUBPAGES_MAPPED); + nr = nr_pmdmapped - (nr & FOLIO_PAGES_MAPPED); /* Raced ahead of a remove and another add? */ if (unlikely(nr < 0)) nr = 0; @@ -1337,7 +1338,7 @@ void page_add_file_rmap(struct page *page, nr = atomic_add_return_relaxed(COMPOUND_MAPPED, mapped); if (likely(nr < COMPOUND_MAPPED + COMPOUND_MAPPED)) { nr_pmdmapped = thp_nr_pages(page); - nr = nr_pmdmapped - (nr & SUBPAGES_MAPPED); + nr = nr_pmdmapped - (nr & FOLIO_PAGES_MAPPED); /* Raced ahead of a remove and another add? */ if (unlikely(nr < 0)) nr = 0; @@ -1399,7 +1400,7 @@ void page_remove_rmap(struct page *page, nr = atomic_sub_return_relaxed(COMPOUND_MAPPED, mapped); if (likely(nr < COMPOUND_MAPPED)) { nr_pmdmapped = thp_nr_pages(page); - nr = nr_pmdmapped - (nr & SUBPAGES_MAPPED); + nr = nr_pmdmapped - (nr & FOLIO_PAGES_MAPPED); /* Raced ahead of another remove and an add? */ if (unlikely(nr < 0)) nr = 0; From patchwork Wed Jan 11 14:28:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13096780 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 338FCC54EBC for ; Wed, 11 Jan 2023 14:29:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 24C8D8E000F; Wed, 11 Jan 2023 09:29:20 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1FE5F940009; Wed, 11 Jan 2023 09:29:20 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 075EA8E0010; Wed, 11 Jan 2023 09:29:19 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id DE8038E000F for ; Wed, 11 Jan 2023 09:29:19 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id BAD0D80C96 for ; Wed, 11 Jan 2023 14:29:19 +0000 (UTC) X-FDA: 80342750838.10.47F214C Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf06.hostedemail.com (Postfix) with ESMTP id 22B33180003 for ; Wed, 11 Jan 2023 14:29:17 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=qKNSjuF5; spf=none (imf06.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673447358; a=rsa-sha256; cv=none; b=QZtWI79ylCvsd5NxquWfZpRNwFgtlQ0aUFM5mK8PpnTgvCpehI0nJjDKSC+uYmjPv8poDs BbBtSfrzfVyLPkDIh+Tt8umnzMTiQP3rudIKi03kFgEZQ0C2LprvWfKDAabpYKeDGGlknV Aw/TOqcwMYRrvbfJLkRycwjFJGkcHaE= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=qKNSjuF5; spf=none (imf06.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673447358; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=WvVzapyi0dCw+kNmNHNzQOI1Ac8jKfCSQjkrQYngcng=; b=HKrUMkSqmxGFDq3BbbbEIpE2HIyr+oSONM6ZkYgyVjDcwEtjyCaSGgpz9K3E7/MSn82mmf /DsifRZHJndfI3hqeaTKVORMVI5vEeCL2AqrXu85dhlMWejm5jtlMZki+6Sm8+qCM0h7rK B60Z+V16CyKqQ1PjY5HGt8C3MoQX86w= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=WvVzapyi0dCw+kNmNHNzQOI1Ac8jKfCSQjkrQYngcng=; b=qKNSjuF5P+j8qQnMSSkvNbZkKJ RHmXnsAulsaJlUa7U/NHAjB8ojVPa71qiUAb4IdBLQyR/mnTotDKHSUQYp0BC1yHgc3ZT5s4DBm4s oiajxoHSweJ+L+qFx3sH8OdwFPQsjtgECv++9Cz9VIFrnUYJQ4D12r7ohB8Fvw9hOOuf5PxMae3EH ZRcPQSVGx9g4zSeuAjob2eShpJblxCZvEf6c9XDONYq27i1ZLfMGxkto93X9wI8kUbzYq/4Kr9yxh kfrmVtXXu7dLijEkuKVxDyyLuqCqsGNBjvVbYH61qeE7lnfslo6fFu8nRSy5uhU9zc/XErsZ8AUHI 6eUctIMg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pFc6O-004CXy-TE; Wed, 11 Jan 2023 14:29:16 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Hugh Dickins Subject: [PATCH 03/28] doc: Clarify refcount section by referring to folios & pages Date: Wed, 11 Jan 2023 14:28:49 +0000 Message-Id: <20230111142915.1001531-4-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230111142915.1001531-1-willy@infradead.org> References: <20230111142915.1001531-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 22B33180003 X-Rspamd-Server: rspam01 X-Stat-Signature: nu3tz1nbkm4u48fzsy94ukyeibmoz549 X-HE-Tag: 1673447357-762443 X-HE-Meta: U2FsdGVkX18e5c5xrcrSgVlCQuFVv8dU4smzvg0RGZadWWTVWDRDJTRQC5CVulTZUVjGEQW0UtfGxV0qKAY07YkLPIhxlOYy3cULKM6WQEPDquKlXzcSXTLq+5DohCoVcNuWVLRFJFX0PhRWdnI3b145U2fqB3COqeAhseqgO9XMPaNSvx3SzELaAqAynfdhd/7ya7t488YlEtaImutkYVqIxrX/ZyNZZ7X1fKQqESkvA5XFlHeHTrR21avkTfRmUJuVfdam18eXKSVBmohtRoPTEyLIEJ6w6BM8ohFq2AlVruACuuQmOGd9EHIVHeG2gMp/7x+SjsLEhrS10Ci/tEdJFW2A4RBlermXMBBWdat7crzBCOD971+oy2fH2X6jKvEYbUILYmADZEwtxVVDE2TK21VucyiOr1lOyzknZjEU3cK5I3W6D/LLD1x/12T+/95asVIufNmFEfKuuH7ee9MaavISwkP5LR8UBPvePS/gyP/wDmeAcghRP5bNl5ZMNqhVJYbG/V/bphK/Oe/QYPdgXWtsjRdztjhReOjDHWTycP9+6HTuPt5qIQu/pRDQxKHka8qm43fwUJwml+0jW0Gir0fNnX3uCZrP5sSTzP9YsQgLsrQgDLnlE1eoD5ycuXVUi1AnwanZDDcGw7Vr0Rr2xE9xy37oM24Z162cs5nocONqGSMxIHm4rpWY5Tyo1+5A4iFGrY3Bf2WDoLtXkpwd4U1/DcvAJ9x7Rw895SD4BcU1Af5saWMt80r/p0AUJXGWm+tvhLuoVd8Tq4SIaoPgk0ADJcVvy03nPC7wcCdFvjMSVohwdrM9Pguyht0FDt+ngCSUHtPXXvrJMTDlIRyE3/4r2J5ZrYiInGJ4ochV3695Rxnp5cc4HcJTaPzH X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Include the rename of subpages_mapcount to _nr_pages_mapped. Signed-off-by: Matthew Wilcox (Oracle) --- Documentation/mm/transhuge.rst | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/Documentation/mm/transhuge.rst b/Documentation/mm/transhuge.rst index ec3dc5b04226..03bbd0a19041 100644 --- a/Documentation/mm/transhuge.rst +++ b/Documentation/mm/transhuge.rst @@ -112,20 +112,20 @@ Refcounts and transparent huge pages Refcounting on THP is mostly consistent with refcounting on other compound pages: - - get_page()/put_page() and GUP operate on head page's ->_refcount. + - get_page()/put_page() and GUP operate on the folio->_refcount. - ->_refcount in tail pages is always zero: get_page_unless_zero() never succeeds on tail pages. - - map/unmap of PMD entry for the whole compound page increment/decrement - ->compound_mapcount, stored in the first tail page of the compound page; - and also increment/decrement ->subpages_mapcount (also in the first tail) - by COMPOUND_MAPPED when compound_mapcount goes from -1 to 0 or 0 to -1. + - map/unmap of a PMD entry for the whole THP increment/decrement + folio->_entire_mapcount and also increment/decrement + folio->_nr_pages_mapped by COMPOUND_MAPPED when _entire_mapcount + goes from -1 to 0 or 0 to -1. - - map/unmap of sub-pages with PTE entry increment/decrement ->_mapcount - on relevant sub-page of the compound page, and also increment/decrement - ->subpages_mapcount, stored in first tail page of the compound page, when - _mapcount goes from -1 to 0 or 0 to -1: counting sub-pages mapped by PTE. + - map/unmap of individual pages with PTE entry increment/decrement + page->_mapcount and also increment/decrement folio->_nr_pages_mapped + when page->_mapcount goes from -1 to 0 or 0 to -1 as this counts + the number of pages mapped by PTE. split_huge_page internally has to distribute the refcounts in the head page to the tail pages before clearing all PG_head/tail bits from the page From patchwork Wed Jan 11 14:28:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13096779 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 60828C5479D for ; Wed, 11 Jan 2023 14:29:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F1E4E940007; Wed, 11 Jan 2023 09:29:16 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id ED0EB8E000F; Wed, 11 Jan 2023 09:29:16 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CAABD940007; Wed, 11 Jan 2023 09:29:16 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id B104E8E000F for ; Wed, 11 Jan 2023 09:29:16 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 89D251A022C for ; Wed, 11 Jan 2023 14:29:16 +0000 (UTC) X-FDA: 80342750712.11.D914025 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf07.hostedemail.com (Postfix) with ESMTP id 065BA40013 for ; Wed, 11 Jan 2023 14:29:14 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=ZpsDp1ZU; dmarc=none; spf=none (imf07.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673447355; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PWk3rt37uRiBd6nhhszu2D/+LUr+ocb9my4gOz9jlaI=; b=lsWkSh5Z98/IsHY973ueA306Z9Nul9HRYjxL3/LMBCc7DLXqeXe1kwSxCAMoqs97ljdl7x WmWv/tqpssM9F0zAvYsZK1Ms5QE+Vt2LQ71VNse563l1SFuXPhEE6v56oH7GUrFrq0d+Yq Gy8bBfkMNNbeFtWTVDRwhQtBtshom5U= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=ZpsDp1ZU; dmarc=none; spf=none (imf07.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673447355; a=rsa-sha256; cv=none; b=OWVUmWHAjrzAUnXNgVIezGSc8G0VcHqKa3KaxPWKLeZVaG6ICTIPaU4USsWkpH3Rq2SY2C j+70Z8KRjrTk63Y4m9ZbcAWJcOhsy8beGtVcuAFbWKiKCLx82GEKW9Edw4rHi/bp5bWplz +A42wl+ka5zJY7k86Dj2az3Rg5Gwoq4= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=PWk3rt37uRiBd6nhhszu2D/+LUr+ocb9my4gOz9jlaI=; b=ZpsDp1ZUPNMWSo0GDMlRzA27vB VmgcHtEzOm2DuRtNqpaIvZe3JuYN46phjaE+cD+PuSMREJjWFF9HgcUMW9J0jrqJUF7tzaq0Ja5nd nppZl1Cm1749Y35o81K2yroIXsTf/9OK9dTHaRZnt5D/2LecRW00RXMoQoD/kR5JB/NnSTtYRZSWa T2h0KhfjFk7J7+WKrdqiLpl11CW+G12sw4SMLV7m0O9g6o2zQbfrJOS845oiMe/06YpS4dPAR142e Dnv/WbtGCelaAlzaJc8AzGvSQAPQ6sg09gzuW2bE7fXkYl01VDWbB7QtSNOfNMnMhIXDV87VNXTMC kjot5TsQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pFc6O-004CY0-VM; Wed, 11 Jan 2023 14:29:16 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Hugh Dickins Subject: [PATCH 04/28] mm: Convert total_compound_mapcount() to folio_total_mapcount() Date: Wed, 11 Jan 2023 14:28:50 +0000 Message-Id: <20230111142915.1001531-5-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230111142915.1001531-1-willy@infradead.org> References: <20230111142915.1001531-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 065BA40013 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: nntymwydmzzt54peeenhacr4uh3p4o5j X-HE-Tag: 1673447354-892198 X-HE-Meta: U2FsdGVkX1+p7eWqc9ovTn86xoTPfI4gyPrpdAnP6D8EglmmKJrQzZ8nuIWAygn3xRN8DJiEDBgFoZk90MlDtw6bAOp8doT0U8oSjM0pRcsB280S7SZKOVbZU2J6R+OhEojTx9eThaaUeLHA4e6gr8EnswKiGyHt1N6ZsJIl/OvbSDEUN3Llo2If8JGOGhV7tnA7sEm+cz+3V5MzDa1eU+Ydb/wc51XH0/BchYJuViNvzR50NXKaqg7l8pPyNtDsKCkmS4Ebg5R7iel8XyFqGXYBxaNtK6XH3AnZ3IIPXwMs/Z7jup3i+ZPOLsB1xBVBDIDz5jeAR8KjYfYoDaNyg5EfcGLkqwr6nRRMWlGQF7n0ZzWph4UO40WkSYoJVaEw61NrSrWw+lR0AOkzvgwJky8P2gTdxzxVkkyN2sQakJwvXvWwv/1vHdpL/9wjWN8ryK3pIdFzO7FPF03l6ib58VaFeioWr39dFzsDAQGV0p74Fa1DXwshg/bcFVhyTIl39vF0MZYInpuRQNYL4+Bm6fsnARD9abyiLYdekcdoMwsy+c5drpb+zf1w0hHT9yLN2iuUSDjvuL5Z6hk5fO4kJWukgvTXBWmkZtN5OnIyHV6eqH+L4soa1SywytvaEA6b7r5oqtvnYVIjLP2u9Xdm6l1i6mvLPXRwFBCTmHHPziyL1fAlJIprnRgvUIh7WrQAnzbswkCwHwAle3zTvnWacWZaHQd/fC6VIPCF/fFt8ChAkYn/t43h5lVGTM8OIVMnhl0dEM7WTG2poyaxuVUmjY6YC1ewxhxdRue5AyrsPM56/W/Z5rGsAyFZrMq/6aR15GjW6cylt6r7ixQQj75wWbqUbci0KtqOl42qaQIvV2JbxI/EYPmimj+49E6MjoP1aFVPoOMBEwPYJotCbxqJWZZoVncjXdws0XbCZ9H3m6oqXnsYa7LL5g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Instead of enforcing that the argument must be a head page by naming, enforce it with the compiler by making it a folio. Also rename the counter in struct folio from _compound_mapcount to _entire_mapcount. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/mm.h | 6 +++--- include/linux/mm_types.h | 6 +++--- mm/rmap.c | 21 ++++++++++----------- 3 files changed, 16 insertions(+), 17 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index aa151e69416d..8bddc7810f78 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -871,7 +871,7 @@ static inline int page_mapcount(struct page *page) return head_compound_mapcount(page) + mapcount; } -int total_compound_mapcount(struct page *head); +int folio_total_mapcount(struct folio *folio); /** * folio_mapcount() - Calculate the number of mappings of this folio. @@ -888,14 +888,14 @@ static inline int folio_mapcount(struct folio *folio) { if (likely(!folio_test_large(folio))) return atomic_read(&folio->_mapcount) + 1; - return total_compound_mapcount(&folio->page); + return folio_total_mapcount(folio); } static inline int total_mapcount(struct page *page) { if (likely(!PageCompound(page))) return atomic_read(&page->_mapcount) + 1; - return total_compound_mapcount(compound_head(page)); + return folio_total_mapcount(page_folio(page)); } static inline bool folio_large_is_mapped(struct folio *folio) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index aa2039f18f4d..a66054a9f0b6 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -306,7 +306,7 @@ static inline struct page *encoded_page_ptr(struct encoded_page *page) * @_head_1: Points to the folio. Do not use. * @_folio_dtor: Which destructor to use for this folio. * @_folio_order: Do not use directly, call folio_order(). - * @_compound_mapcount: Do not use directly, call folio_entire_mapcount(). + * @_entire_mapcount: Do not use directly, call folio_entire_mapcount(). * @_nr_pages_mapped: Do not use directly, call folio_mapcount(). * @_pincount: Do not use directly, call folio_maybe_dma_pinned(). * @_folio_nr_pages: Do not use directly, call folio_nr_pages(). @@ -360,7 +360,7 @@ struct folio { unsigned long _head_1; unsigned char _folio_dtor; unsigned char _folio_order; - atomic_t _compound_mapcount; + atomic_t _entire_mapcount; atomic_t _nr_pages_mapped; atomic_t _pincount; #ifdef CONFIG_64BIT @@ -403,7 +403,7 @@ FOLIO_MATCH(flags, _flags_1); FOLIO_MATCH(compound_head, _head_1); FOLIO_MATCH(compound_dtor, _folio_dtor); FOLIO_MATCH(compound_order, _folio_order); -FOLIO_MATCH(compound_mapcount, _compound_mapcount); +FOLIO_MATCH(compound_mapcount, _entire_mapcount); FOLIO_MATCH(subpages_mapcount, _nr_pages_mapped); FOLIO_MATCH(compound_pincount, _pincount); #ifdef CONFIG_64BIT diff --git a/mm/rmap.c b/mm/rmap.c index f08685054d0a..675d8401c2da 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1078,27 +1078,26 @@ int pfn_mkclean_range(unsigned long pfn, unsigned long nr_pages, pgoff_t pgoff, return page_vma_mkclean_one(&pvmw); } -int total_compound_mapcount(struct page *head) +int folio_total_mapcount(struct folio *folio) { - struct folio *folio = (struct folio *)head; - int mapcount = head_compound_mapcount(head); - int nr_subpages; + int mapcount = folio_entire_mapcount(folio); + int nr_pages; int i; - /* In the common case, avoid the loop when no subpages mapped by PTE */ + /* In the common case, avoid the loop when no pages mapped by PTE */ if (folio_nr_pages_mapped(folio) == 0) return mapcount; /* - * Add all the PTE mappings of those subpages mapped by PTE. - * Limit the loop, knowing that only subpages_mapcount are mapped? + * Add all the PTE mappings of those pages mapped by PTE. + * Limit the loop to folio_nr_pages_mapped()? * Perhaps: given all the raciness, that may be a good or a bad idea. */ - nr_subpages = thp_nr_pages(head); - for (i = 0; i < nr_subpages; i++) - mapcount += atomic_read(&head[i]._mapcount); + nr_pages = folio_nr_pages(folio); + for (i = 0; i < nr_pages; i++) + mapcount += atomic_read(&folio_page(folio, i)->_mapcount); /* But each of those _mapcounts was based on -1 */ - mapcount += nr_subpages; + mapcount += nr_pages; return mapcount; } From patchwork Wed Jan 11 14:28:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13096768 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 304F7C54EBC for ; Wed, 11 Jan 2023 14:29:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 953BE940008; Wed, 11 Jan 2023 09:29:12 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 49D1B90000E; Wed, 11 Jan 2023 09:29:12 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D970490000D; Wed, 11 Jan 2023 09:29:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id DC2C6940007 for ; Wed, 11 Jan 2023 09:29:10 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id AAE41140929 for ; Wed, 11 Jan 2023 14:29:10 +0000 (UTC) X-FDA: 80342750460.23.276D99C Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf15.hostedemail.com (Postfix) with ESMTP id 19CA7A000C for ; Wed, 11 Jan 2023 14:29:08 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=TgqfnK9N; spf=none (imf15.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673447349; a=rsa-sha256; cv=none; b=L+qbZ1Q8kJgntgDBR30J72RTN5DVdkww63ZTdxjhv6f2Bujf+ANHg2KCFKWK5RJX4i6fCq MhXvS2J8Ce2S4VXhNEmt0PLYIRvGIA3GI6MeRMAG7EtlD5frGTJiubO5LX859/7Lvo6l7N PSeI2cD0Vr/I1USTbP0XTiuIyFxptSo= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=TgqfnK9N; spf=none (imf15.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673447349; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=k563aUCO0woe019PthIukGmZp4g+v5x7heD1X9jV3wA=; b=D2Lz7/vnonSddrW6q2Lzl52IKT9ZTgLAWE8jW8JVXjnPihtCIxv1yWH41o1eyAnEV/N+Fy Teg+rUtqXsLKLeHdL0CjLLWJnRv1yHppFf9tMbn29yBaRaCKWPc0dlKzS+GESLIReN/stH jq3o7y27qMuo3V1nGLTW6+9BRhhd+E8= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=k563aUCO0woe019PthIukGmZp4g+v5x7heD1X9jV3wA=; b=TgqfnK9NYbf/dPaBd2igQnq4pc vjgKExtNpFgQpWSTWltsHdgkZhL05TP5BFTe4eSC6+6w1PtezM+4AtCWaVmUDLj7r6CVplAmDwKn9 PASQQuj5D1Gl5ghsFTGc6JvJ5eEkFl2HwvThS4kH8pwaKlw49daGOcYpigC1Y7Qo5lYIQNk8NmIil RsqH1zoFtXVb637N9z18XqxFV43iF8QCNuDNt0CR4rfOsAcRF9ScyTeNSvpAEUlEy40ag5/r9UdHN lsKrSiiTLmyXbtm42qmzXxWtaUql85HnD0mcmX50GNndWUWJE74dIUz7bS9BdPI61Au7J5AKpMFuR /lQnMxxA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pFc6P-004CY2-1L; Wed, 11 Jan 2023 14:29:17 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Hugh Dickins Subject: [PATCH 05/28] mm: Convert page_remove_rmap() to use a folio internally Date: Wed, 11 Jan 2023 14:28:51 +0000 Message-Id: <20230111142915.1001531-6-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230111142915.1001531-1-willy@infradead.org> References: <20230111142915.1001531-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 19CA7A000C X-Stat-Signature: atpxmtnnh6s9wfmy4c9stcb7ig6fx3m7 X-HE-Tag: 1673447348-49670 X-HE-Meta: U2FsdGVkX19E/uRSPuEteoKcZnNNwiDbe5doW5dDjukI+Jtkgaf1rbASqHs3t6Cgv7+XSkimHXfNUxs2axj37Dh8V/3tZYw51LbJTcXfKzYUAdH6oOGObBtpSxKd+dy6ELGLZSJOb4F9eu9sLYg7KaOmIMrvlFq5jS0izDJJ1yS4v8ySkhDoICWTWkXs+4SD831nlkX+PwBegBjX90zc00+hdbfWw8HQ0OwnLNun6smpqCCJuqVrwQLyRxT4ZYFuFUBI261G+VBQgc7uQf3V5xxeSbZyv2WYRLM+5TrfIp4WoeACuP+/qnPnhY7UJUlFaBW4pDvn1eGKaz+0QhUrewDImNMcxdxcjZ/UP7MKwrym2uQraGs4ayl/LuL/xdI/sX+h7JR0SexUBAfL4pflEdadImjMTkUzMX6O6yD9fEayTN5+6CP5lA0/+zH2qVzisdaEY5olQpr4AWADfNXm3L8CLJGZolEw/VAWnER81VIeYENLe7KVtJqgPaDtA8siYUwFrRhCM9vDqA00dX7iiYzqJ6qrPvh1IjoDygu4cfYtCl3kvowmDgez3JLh0G9bOgXT2kWlyATxGQEDSWIO6rc9zN4PCEbYBdQKdEnxN93rLUG0R83Zl16Pzr9a/YOKbLpLICdBertzDTQ1v1bHurhS1Q6SGdLYRZtanwmBXHL4CW/2phuZYVbQByaYu9/LFCNk5ABGJdhiFbgzEr9dmf/HrpLwzgWmi0dk5FvEJQ/XCXoIeT5danDWtHp6v6Rw3/eVPw+Moqw51/E32qPN3cqbGI/YbAABoFpVL11zIcIajnk9zAGn9UB3FvUGBsiue1UOrAvgcqABw/a8IPCEAqUaxHsjSaR7nwcJZeCjrFz/x7WJSKwvcH+mTB4ViAZpD9AunsMaVshk+r7XY9Nu6vZNYl97ary4kQD82eVK4VvZcEsOxMOZHA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The API for page_remove_rmap() needs to be page-based, because we can remove mappings of pages individually. But inside the function, we want to only call compound_head() once and then use the folio APIs instead of the page APIs that each call compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/rmap.c | 47 ++++++++++++++++++++++++++--------------------- 1 file changed, 26 insertions(+), 21 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index 675d8401c2da..d137bd8e5309 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1365,19 +1365,21 @@ void page_add_file_rmap(struct page *page, * * The caller needs to hold the pte lock. */ -void page_remove_rmap(struct page *page, - struct vm_area_struct *vma, bool compound) +void page_remove_rmap(struct page *page, struct vm_area_struct *vma, + bool compound) { - atomic_t *mapped; + struct folio *folio = page_folio(page); + atomic_t *mapped = &folio->_nr_pages_mapped; int nr = 0, nr_pmdmapped = 0; bool last; + enum node_stat_item idx; VM_BUG_ON_PAGE(compound && !PageHead(page), page); /* Hugetlb pages are not counted in NR_*MAPPED */ - if (unlikely(PageHuge(page))) { + if (unlikely(folio_test_hugetlb(folio))) { /* hugetlb pages are always mapped with pmds */ - atomic_dec(compound_mapcount_ptr(page)); + atomic_dec(&folio->_entire_mapcount); return; } @@ -1385,20 +1387,18 @@ void page_remove_rmap(struct page *page, if (likely(!compound)) { last = atomic_add_negative(-1, &page->_mapcount); nr = last; - if (last && PageCompound(page)) { - mapped = subpages_mapcount_ptr(compound_head(page)); + if (last && folio_test_large(folio)) { nr = atomic_dec_return_relaxed(mapped); nr = (nr < COMPOUND_MAPPED); } - } else if (PageTransHuge(page)) { + } else if (folio_test_pmd_mappable(folio)) { /* That test is redundant: it's for safety or to optimize out */ - last = atomic_add_negative(-1, compound_mapcount_ptr(page)); + last = atomic_add_negative(-1, &folio->_entire_mapcount); if (last) { - mapped = subpages_mapcount_ptr(page); nr = atomic_sub_return_relaxed(COMPOUND_MAPPED, mapped); if (likely(nr < COMPOUND_MAPPED)) { - nr_pmdmapped = thp_nr_pages(page); + nr_pmdmapped = folio_nr_pages(folio); nr = nr_pmdmapped - (nr & FOLIO_PAGES_MAPPED); /* Raced ahead of another remove and an add? */ if (unlikely(nr < 0)) @@ -1411,21 +1411,26 @@ void page_remove_rmap(struct page *page, } if (nr_pmdmapped) { - __mod_lruvec_page_state(page, PageAnon(page) ? NR_ANON_THPS : - (PageSwapBacked(page) ? NR_SHMEM_PMDMAPPED : - NR_FILE_PMDMAPPED), -nr_pmdmapped); + if (folio_test_anon(folio)) + idx = NR_ANON_THPS; + else if (folio_test_swapbacked(folio)) + idx = NR_SHMEM_PMDMAPPED; + else + idx = NR_FILE_PMDMAPPED; + __lruvec_stat_mod_folio(folio, idx, -nr_pmdmapped); } if (nr) { - __mod_lruvec_page_state(page, PageAnon(page) ? NR_ANON_MAPPED : - NR_FILE_MAPPED, -nr); + idx = folio_test_anon(folio) ? NR_ANON_MAPPED : NR_FILE_MAPPED; + __lruvec_stat_mod_folio(folio, idx, -nr); + /* - * Queue anon THP for deferred split if at least one small - * page of the compound page is unmapped, but at least one - * small page is still mapped. + * Queue anon THP for deferred split if at least one + * page of the folio is unmapped and at least one page + * is still mapped. */ - if (PageTransCompound(page) && PageAnon(page)) + if (folio_test_pmd_mappable(folio) && folio_test_anon(folio)) if (!compound || nr < nr_pmdmapped) - deferred_split_huge_page(compound_head(page)); + deferred_split_huge_page(&folio->page); } /* From patchwork Wed Jan 11 14:28:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13096763 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 899B0C46467 for ; Wed, 11 Jan 2023 14:29:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 88CA98E0011; Wed, 11 Jan 2023 09:29:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5583B8E0001; Wed, 11 Jan 2023 09:29:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B7226900007; Wed, 11 Jan 2023 09:29:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id D5E758E000A for ; Wed, 11 Jan 2023 09:29:09 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id B42B11C6414 for ; Wed, 11 Jan 2023 14:29:09 +0000 (UTC) X-FDA: 80342750418.26.0547C59 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf01.hostedemail.com (Postfix) with ESMTP id D5C3E40004 for ; Wed, 11 Jan 2023 14:29:06 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=TyPmpJkC; dmarc=none; spf=none (imf01.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673447347; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=S5RQv/iHXB0LHnXKaLRNqyQBA4vTXTkj3jwk4WmiUds=; b=DgK5POXBuuEHeUbzXlSPgUbhIeWd2F7tKUiVuvbCLPX1+FH1PDcKTzvXZcfH0xAF/+NZv/ eQhecb1HhKQyyD4btM+2V1SiYiPkkidqEa+Okx9l9uT5HN0RlilKuW8iGnyJ+KEl2fQHJZ 70bKVyUbZAUhrvl0Hx4vcN3S5zrbOac= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=TyPmpJkC; dmarc=none; spf=none (imf01.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673447347; a=rsa-sha256; cv=none; b=JyAWgRBW77qea4zVWUNyqJFQaFHU1u+t3XkOKushFED/Ka/ICe70zBFE/KLksiE39s0z/v AYC4kjLTHJVSqjtSAb4Z5cM8n8HMggmIIr3ILXWRPgr6O15AB/UH04IBvecM+83m/1Wqad 9h5AljxkA/Cd4qH8/vnYfzBMDlsxlkU= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=S5RQv/iHXB0LHnXKaLRNqyQBA4vTXTkj3jwk4WmiUds=; b=TyPmpJkC0TCDVOH1Xf7hFGHEhF ljnBaA0NG4to6fCNil2897PRSAOn9gxuOgLUVUcr1cJvaZaW7cnU/HjGcit7ubSnb/IWfRAQgp/We fIx7eb04Vxi69wNs6VaG0bploau4bGuEi4a5QN+Zb1xksumYJeSKvYNtsKw/JBQKcrOEMexZnp6vP zpgnC8QqgB3e+pX4BaYKdrFbIfnNCdMRBnHW9XIlvs07vgnr6/ugJYGz4bW9IYUxK6ttGHXmiaYjz wx2WAcUvTupzQshPK7Yr7/5QSdJCLXyy4Nwu6SS+TFgU3epUBIOj6+7WPZOrlFiQbcP3Ps5XdL9CH b7nZ3Piw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pFc6P-004CY4-44; Wed, 11 Jan 2023 14:29:17 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Hugh Dickins Subject: [PATCH 06/28] mm: Convert page_add_anon_rmap() to use a folio internally Date: Wed, 11 Jan 2023 14:28:52 +0000 Message-Id: <20230111142915.1001531-7-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230111142915.1001531-1-willy@infradead.org> References: <20230111142915.1001531-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: D5C3E40004 X-Stat-Signature: 9exg13j58mbydjj9fuj5qpj6p95kmk86 X-HE-Tag: 1673447346-68404 X-HE-Meta: U2FsdGVkX1+4K7AdfnoNSxtodcsL5g60UoKZx/H9ox+QR00EbaxCwHCKUxf6vHX187UJNFr9gtWDRSzCemD+dxo12ibPWb5txAWecHwwnUo7iyhyY4weqbk45Tnf+FPrZRQ5j/jahrnCJRzQMhbBfnn+rTbHD9s86RpIx2O5odUAciTggx2kW76avzPv5i1Nqhr1lmOv0G3C62dxhcSEdIQhtjenAuoC+tmq3d+ApdZ4yc+xFRolrP+5zj9zzlfpWzWV85kOyYFTuJatY1i55c4lP0wMvxnREv770noNwnhvcVbNNaj5rH8zrwLCGtmC0QXz4LTuAsxKVEmw4v7U08gThZg9cKBFLN2Wy+vcumHJAW8kqUpVKACbc0Gn96NmyLqlX2O5UTv0tlHQiw41WUUv4xaV2NkNNt+VXKquCYMOkWFAs4dhFS5Zm3CnxOy/P+yhp5SyStusYmE/WJG6dmZzTu/9NFOC8y4vvJoMBtZob1Erb+9cab5y9RjrlA/d/352MVYHAENRkrwk/0PjAbUfzjN4t3h/dVfPt6aO04WXtVL/e5MJoGyq9qWu97pmOzRl3UwIYs4hcSHYeitzHTcgyFC5SxLmU675Arv/svGrjCjAsd3KEEyxVM5eSDXtX5UcGycTF4k9WEdowJO+WdAAwvqSL+KQIKT7KIRSaR+WnCzLrZME6IeI9FPLya9i9vVyDFhUPVc476kOHpEIiJHhfRpIqlI7UAnEdFdSbnXhJCECgDOQbaj5FZy0Wjj3Mtxwjam64gnu8AXLEKQZ0dlnir+s5qNoRVl+Z6nyCCzHkUCj4xeAYifCc5AoH3WnmH9hWrckt6eY8N7ImrpoR0TbkTPw/ZGBWw1ibto9EScLHkrOxTq9nYoYbu88Co6MXl1C2CSX5TDiH9Skp/RsNZYywzvKnNSMiLz2cjk5Ax79OO1xK6DGew== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The API for page_add_anon_rmap() needs to be page-based, because we can add mappings of individual pages. But inside the function, we want to only call compound_head() once and then use the folio APIs instead of the page APIs that each call compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/rmap.c | 23 +++++++++++------------ 1 file changed, 11 insertions(+), 12 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index d137bd8e5309..187c7c832111 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1207,10 +1207,11 @@ static void __page_check_anon_rmap(struct page *page, * and to ensure that PageAnon is not being upgraded racily to PageKsm * (but PageKsm is never downgraded to PageAnon). */ -void page_add_anon_rmap(struct page *page, - struct vm_area_struct *vma, unsigned long address, rmap_t flags) +void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, + unsigned long address, rmap_t flags) { - atomic_t *mapped; + struct folio *folio = page_folio(page); + atomic_t *mapped = &folio->_nr_pages_mapped; int nr = 0, nr_pmdmapped = 0; bool compound = flags & RMAP_COMPOUND; bool first = true; @@ -1219,20 +1220,18 @@ void page_add_anon_rmap(struct page *page, if (likely(!compound)) { first = atomic_inc_and_test(&page->_mapcount); nr = first; - if (first && PageCompound(page)) { - mapped = subpages_mapcount_ptr(compound_head(page)); + if (first && folio_test_large(folio)) { nr = atomic_inc_return_relaxed(mapped); nr = (nr < COMPOUND_MAPPED); } - } else if (PageTransHuge(page)) { + } else if (folio_test_pmd_mappable(folio)) { /* That test is redundant: it's for safety or to optimize out */ - first = atomic_inc_and_test(compound_mapcount_ptr(page)); + first = atomic_inc_and_test(&folio->_entire_mapcount); if (first) { - mapped = subpages_mapcount_ptr(page); nr = atomic_add_return_relaxed(COMPOUND_MAPPED, mapped); if (likely(nr < COMPOUND_MAPPED + COMPOUND_MAPPED)) { - nr_pmdmapped = thp_nr_pages(page); + nr_pmdmapped = folio_nr_pages(folio); nr = nr_pmdmapped - (nr & FOLIO_PAGES_MAPPED); /* Raced ahead of a remove and another add? */ if (unlikely(nr < 0)) @@ -1248,11 +1247,11 @@ void page_add_anon_rmap(struct page *page, VM_BUG_ON_PAGE(!first && PageAnonExclusive(page), page); if (nr_pmdmapped) - __mod_lruvec_page_state(page, NR_ANON_THPS, nr_pmdmapped); + __lruvec_stat_mod_folio(folio, NR_ANON_THPS, nr_pmdmapped); if (nr) - __mod_lruvec_page_state(page, NR_ANON_MAPPED, nr); + __lruvec_stat_mod_folio(folio, NR_ANON_MAPPED, nr); - if (likely(!PageKsm(page))) { + if (likely(!folio_test_ksm(folio))) { /* address might be in next vma when migration races vma_adjust */ if (first) __page_set_anon_rmap(page, vma, address, From patchwork Wed Jan 11 14:28:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13096753 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D1729C46467 for ; Wed, 11 Jan 2023 14:29:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2FE7C8E0009; Wed, 11 Jan 2023 09:29:09 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 222C78E0002; Wed, 11 Jan 2023 09:29:09 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 09DCA8E0005; Wed, 11 Jan 2023 09:29:09 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id E91DA8E0001 for ; Wed, 11 Jan 2023 09:29:08 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id C2DE21C6342 for ; Wed, 11 Jan 2023 14:29:08 +0000 (UTC) X-FDA: 80342750376.12.BB73B62 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf26.hostedemail.com (Postfix) with ESMTP id F11DB140004 for ; Wed, 11 Jan 2023 14:29:06 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=rVMUVJDd; spf=none (imf26.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673447347; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+M9PYetxBWVQcE/WzZKSgATAyynY7Xkz/FsMlEYD1jw=; b=WgJEtSZz1tELaHHGSfA6Rj0Mxr4vJE8a9r1EaLD/Eh44ZBzc7QuJIruTF5hSWzgGwTFz5B I7FoqCGxrbYLm0tpmBG1SSunyBsAmEMFxSfRe0YSRGq1zFS9AdaeoimwymM1lc6X8pgyJ5 vygqmu1unMjLMuWqaVxWP6v/hC7Lq8w= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=rVMUVJDd; spf=none (imf26.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673447347; a=rsa-sha256; cv=none; b=FNneasn89ULNy93q18dyPeuZT5/9JKm17REjhSRVEeuClFrKqzE2xFSDRzZgLx4dtYtn/l iPL3d8M+6vGvQEtiPzbpTGiL9hXdi0wvxj1nD1+WKgiV9p4o9fgGAtgF0ICtxKq3djT6y8 MJzdbJScsEhbLruHYxvy5RyfTygYzx4= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=+M9PYetxBWVQcE/WzZKSgATAyynY7Xkz/FsMlEYD1jw=; b=rVMUVJDdW5xPOeYfbS5d0cShsD 5cLbkjfcH/32sd6DGZyRN7SjoAL6kzTH/atPlZejvJcdVGCFcO8Bsecd/ABDDpE80/gXBJnDFVIOG 5AMtK+927M0Bv7mQeBdgyzoD3amBSKmE/6h0u+n+f/4NB+KiG6HQLFje6pYJ9w6tlSy/rGVyU2bMv QeB7ykoxiaSiVkWnYKzLSZyBUh7y+rl8TpCggTOM5B9RRpCLTokdeaGh3vOjzKbyu+jG5nNtn0HUa 8/cFmRtQGjBZdrcUD5IgLP+6beRuB9ptYzKdYs4DaHw0MD1F0C2Pxz0zW4Nzimfmmb6txPT7QeE25 t8DcM8aw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pFc6P-004CYH-Hl; Wed, 11 Jan 2023 14:29:17 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Hugh Dickins Subject: [PATCH 07/28] mm: Convert page_add_file_rmap() to use a folio internally Date: Wed, 11 Jan 2023 14:28:53 +0000 Message-Id: <20230111142915.1001531-8-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230111142915.1001531-1-willy@infradead.org> References: <20230111142915.1001531-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: F11DB140004 X-Stat-Signature: n93f4ikxfybzwa83x7sao5dbk5xx8q97 X-Rspam-User: X-HE-Tag: 1673447346-492687 X-HE-Meta: U2FsdGVkX18y1zkzsvdsupgPXSPLbxztOqX1HkHCnbMseWfDWm9e48Xr3RqUI+7u5YpPH3Dqm3bAT1IwmveTJ9dQ6z1+KwB398HtCazDgGvTXu4DWZes/PXuBFVIFbtitexq5ve8xo3nohrRXqZ+Ln+NasaziYIhaGjiEwNYwG+CjvIbL11XMm3bAQabAEG3VImscsmOCNmX0BMdPQgfkhtlrZYyHhA/V5kZ7Xz+iLLNxZJ+7x1K1ipQm/z4lOwDK07g3bDB3KKv4H5fOwebdo5yS8vqWMNN0xDpYl0RXTWNMmfcIiWJIEH+hcVHtcTsKOBOn0Xwr7toRFzMGLhMJb/jo/7HaxXp2/9gu7DS1dJh/SGboN7wh2+o/uZg7+1lgWWhMYUvm4jHfpHbY9vKZE8qqsqONBTcNGOwpEvy4+GoFooEh9Z2Fwqg9tw6Gz4ZAvcO7V2ztC8dOOuFQiw5Sjr92UHF2gcTAT5ec3f81nvc94QSIwlIEi8V+k1M7Y/N/cWW/qc7ckyzVDelYvis3Vz+WUctGANeUljM3W1yq77k9OPJfGE4RVSRsdQKtxN/2beb4OFT9KOn1cgcicFNr7zyqlrRpBIw8iMtamPU3p7cMYPpU/0njwowrv7uoZJA0I6kQPd/M6eSTgzUy7L2LuSRbycZCqGV59o2iFy4hqScPojHr3Qd55t/X/69HMVQhILkGMWn8lshhTUfi5rKUJ1nRqT9iIKeXQg/U/ejlTlwkgqF4DfRBurjXXQ/M1vbnHMkJt4s2KNsqHbeR+/KYYwgFQtfKrrYwRGoj6FKqLBIjLM6mbAfz+8ap8Y9oUkV4t/+ETKLNFbYIiUvYojgCOYybuLGd3eaCGzx/LUgGXwq0gGHUWMN/TSsUBKryztbAxcNgvym+qHcDBrcMQxnCM7nDJNygWHQ3YSxCdbutHHP7d3DQY55cA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The API for page_add_file_rmap() needs to be page-based, because we can add mappings of individual pages. But inside the function, we want to only call compound_head() once and then use the folio APIs instead of the page APIs that each call compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/rmap.c | 21 ++++++++++----------- 1 file changed, 10 insertions(+), 11 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index 187c7c832111..7b83b56d5603 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1309,10 +1309,11 @@ void page_add_new_anon_rmap(struct page *page, * * The caller needs to hold the pte lock. */ -void page_add_file_rmap(struct page *page, - struct vm_area_struct *vma, bool compound) +void page_add_file_rmap(struct page *page, struct vm_area_struct *vma, + bool compound) { - atomic_t *mapped; + struct folio *folio = page_folio(page); + atomic_t *mapped = &folio->_nr_pages_mapped; int nr = 0, nr_pmdmapped = 0; bool first; @@ -1322,20 +1323,18 @@ void page_add_file_rmap(struct page *page, if (likely(!compound)) { first = atomic_inc_and_test(&page->_mapcount); nr = first; - if (first && PageCompound(page)) { - mapped = subpages_mapcount_ptr(compound_head(page)); + if (first && folio_test_large(folio)) { nr = atomic_inc_return_relaxed(mapped); nr = (nr < COMPOUND_MAPPED); } - } else if (PageTransHuge(page)) { + } else if (folio_test_pmd_mappable(folio)) { /* That test is redundant: it's for safety or to optimize out */ - first = atomic_inc_and_test(compound_mapcount_ptr(page)); + first = atomic_inc_and_test(&folio->_entire_mapcount); if (first) { - mapped = subpages_mapcount_ptr(page); nr = atomic_add_return_relaxed(COMPOUND_MAPPED, mapped); if (likely(nr < COMPOUND_MAPPED + COMPOUND_MAPPED)) { - nr_pmdmapped = thp_nr_pages(page); + nr_pmdmapped = folio_nr_pages(folio); nr = nr_pmdmapped - (nr & FOLIO_PAGES_MAPPED); /* Raced ahead of a remove and another add? */ if (unlikely(nr < 0)) @@ -1348,10 +1347,10 @@ void page_add_file_rmap(struct page *page, } if (nr_pmdmapped) - __mod_lruvec_page_state(page, PageSwapBacked(page) ? + __lruvec_stat_mod_folio(folio, folio_test_swapbacked(folio) ? NR_SHMEM_PMDMAPPED : NR_FILE_PMDMAPPED, nr_pmdmapped); if (nr) - __mod_lruvec_page_state(page, NR_FILE_MAPPED, nr); + __lruvec_stat_mod_folio(folio, NR_FILE_MAPPED, nr); mlock_vma_page(page, vma, compound); } From patchwork Wed Jan 11 14:28:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13096754 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5AFAEC5479D for ; Wed, 11 Jan 2023 14:29:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 666E18E000B; Wed, 11 Jan 2023 09:29:09 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 55797900002; Wed, 11 Jan 2023 09:29:09 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 13AD68E0001; Wed, 11 Jan 2023 09:29:09 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id EFE8E8E0002 for ; Wed, 11 Jan 2023 09:29:08 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id C24501401EF for ; Wed, 11 Jan 2023 14:29:08 +0000 (UTC) X-FDA: 80342750376.16.5031402 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf22.hostedemail.com (Postfix) with ESMTP id D009DC0019 for ; Wed, 11 Jan 2023 14:29:06 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="J8+3C/3v"; spf=none (imf22.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673447347; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=smKdNZSEfoOItNhulwcaWNs98KWqjgVXzV6tgn0lCxw=; b=HiXADWqYMceO7embY/ZBkJuwhvOnyLuySOIRw5xv8Tom6RYztMft73Ne08fOCmk2O7Wm8o fAMo/SZk9xHH4RWiDtnpGgJwyYYWdGR2bSNfv7auX3e9lMUp84m0IFAk/sj7LhV85FT1HW 5Qjqt/Nn7/SM37yvbAqxnC3XiRGKkv8= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="J8+3C/3v"; spf=none (imf22.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673447347; a=rsa-sha256; cv=none; b=1dUJl/WXnsdSuUvY7kAP8GDTiTBcK4joWKLwnMmN2XWeM2/fiKdpvHcMInppoQ8w0baDOu Kx/pZX6ssO7njHNAjYll1mb1GSg3twu/O1+kacb0UNlvv7Wj3wD9NRXox9xmkS1+sU1L9j uONNiufiyCycKfTr2Y6OQXyyaG8Q/uI= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=smKdNZSEfoOItNhulwcaWNs98KWqjgVXzV6tgn0lCxw=; b=J8+3C/3vUG9uR+wRCdJx0v3GEa ZKmbcIu3wxmkU9Xs08G09aHfo5gTcgtVYWg7OHkWiLtOCMuFg2qMVhzgkPLTsIM/17sg+EFZ35GiM nYFRThMprqAxwGLbIAzPQ+6p0tzPpQOC3MaswYu0/omVEldSBeEX5dHFrbRsRt+s7xfCEdLfCKzfY bQaIvCEUIaIXvmNQ3vnmPGkIQKgvag2GelzxGrdgp67J3B+Ynk+V8UZOKS0eaJ7qwwyDBEsRGR+Cs tWZYUuYFfH7f/G3YCJflBZFUU3ku8kEL2nL38WP/EDHQl/hY0RXkwQqtjUlzMjW8uiQHW66XPGAig ZdIHaSSA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pFc6P-004CYN-M4; Wed, 11 Jan 2023 14:29:17 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Hugh Dickins Subject: [PATCH 08/28] mm: Add folio_add_new_anon_rmap() Date: Wed, 11 Jan 2023 14:28:54 +0000 Message-Id: <20230111142915.1001531-9-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230111142915.1001531-1-willy@infradead.org> References: <20230111142915.1001531-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: D009DC0019 X-Stat-Signature: akz81ys6irdup6aosm8syc4xtsxybtmc X-HE-Tag: 1673447346-194789 X-HE-Meta: U2FsdGVkX19/ld0dEoNQBQ28EHMEheLFENr/LRGfP1XoIYoEC88RnO/fNhfZsts8Dw8WZvrHxaAdupES03Gsbm4XfTos6UUMZSdhucSjuCUNj2Dod89syNuemegvjAUYjfGz3lzUwWpLNoIhOqVR99ALERpfJKn0rqAbbKGVVA0xegyTr3QxeQYmg9M8mmgRPPmiYwJB+y8OAUe2o9Iu8xu1iwpvY74I3BQ83AkktDpn0Ffckh61TfcfMX9t78DDZBv+XwKzT8Qh6f8k0J7OS/AP+Yu597UggKJXqJV0lKHDDSZdYFaHFP0W5mM6NYXBK1NNmFaqmvVemzPgjesoHaE2nYX3Y0AJsmsCHA7GSjsdKYZ2ugxQNrncWzfG7UhsdY3+8QlCpxur+wBbf00GzfqiAGsk/bcHa2Xts/42K4PnnTz6vyu0/MCnDjV4Se+5dCseU2ERWjQHCNtyJ+GRWLBdmtMMsJAOMJrdqjzZnaxX2jLEMVAdsbXsnRQgDGJwuIySHIhAvwxZ5zX6WkXaL9sqPIluMX+SqOimSfM2S+h55XFby04gVHKq6gsRjPLRi/awAB6zVqV55jkSyzuNcG9As22SxMBsDzdYc4Pjzfz/341UOp6N1SbynXKWsWj8Y153i23gOd+y3s+4IjKZX0FW8aT48RvHN/k7m7vwq/Id572dPmmRwmekgFFPtmhCRC9bne+Whw3snelFSziJ4fSiKA4IDwdZlbu7OmDVeqRjCGkT6kO0pppsFhZUgNGuPEeInv+nzmdxG+27V0o4ResXvGgOYjhNtoMjCN2mlPmzKyCPpp72lBkaYlw6MUvcFqIVwcDoRgC9m/VTcSS9GEP65eGhFZTw8MA2FjZWedd4E0/nZ0Ht/rPXysXYLFv85GZsNJargaZGwglF4ySYuLR6znss+FmVKzmH6kOh3vGtLsJxgXppkA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In contrast to other rmap functions, page_add_new_anon_rmap() is always called with a freshly allocated page. That means it can't be called with a tail page. Turn page_add_new_anon_rmap() into folio_add_new_anon_rmap() and add a page_add_new_anon_rmap() wrapper. Callers can be converted individually. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/rmap.h | 2 ++ mm/folio-compat.c | 8 ++++++++ mm/rmap.c | 37 ++++++++++++++++++------------------- 3 files changed, 28 insertions(+), 19 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index bd3504d11b15..aa682a2a93ce 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -194,6 +194,8 @@ void page_add_anon_rmap(struct page *, struct vm_area_struct *, unsigned long address, rmap_t flags); void page_add_new_anon_rmap(struct page *, struct vm_area_struct *, unsigned long address); +void folio_add_new_anon_rmap(struct folio *, struct vm_area_struct *, + unsigned long address); void page_add_file_rmap(struct page *, struct vm_area_struct *, bool compound); void page_remove_rmap(struct page *, struct vm_area_struct *, diff --git a/mm/folio-compat.c b/mm/folio-compat.c index 69ed25790c68..92f53adc0dd9 100644 --- a/mm/folio-compat.c +++ b/mm/folio-compat.c @@ -123,3 +123,11 @@ void putback_lru_page(struct page *page) { folio_putback_lru(page_folio(page)); } + +void page_add_new_anon_rmap(struct page *page, struct vm_area_struct *vma, + unsigned long address) +{ + VM_BUG_ON_PAGE(PageTail(page), page); + + return folio_add_new_anon_rmap((struct folio *)page, vma, address); +} diff --git a/mm/rmap.c b/mm/rmap.c index 7b83b56d5603..2749e1466b09 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1264,41 +1264,40 @@ void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, } /** - * page_add_new_anon_rmap - add mapping to a new anonymous page - * @page: the page to add the mapping to + * folio_add_new_anon_rmap - Add mapping to a new anonymous folio. + * @folio: The folio to add the mapping to. * @vma: the vm area in which the mapping is added * @address: the user virtual address mapped * - * If it's a compound page, it is accounted as a compound page. As the page - * is new, it's assume to get mapped exclusively by a single process. - * - * Same as page_add_anon_rmap but must only be called on *new* pages. + * Like page_add_anon_rmap() but must only be called on *new* folios. * This means the inc-and-test can be bypassed. - * Page does not have to be locked. + * The folio does not have to be locked. + * + * If the folio is large, it is accounted as a THP. As the folio + * is new, it's assumed to be mapped exclusively by a single process. */ -void page_add_new_anon_rmap(struct page *page, - struct vm_area_struct *vma, unsigned long address) +void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, + unsigned long address) { int nr; VM_BUG_ON_VMA(address < vma->vm_start || address >= vma->vm_end, vma); - __SetPageSwapBacked(page); + __folio_set_swapbacked(folio); - if (likely(!PageCompound(page))) { + if (likely(!folio_test_pmd_mappable(folio))) { /* increment count (starts at -1) */ - atomic_set(&page->_mapcount, 0); + atomic_set(&folio->_mapcount, 0); nr = 1; } else { - VM_BUG_ON_PAGE(!PageTransHuge(page), page); /* increment count (starts at -1) */ - atomic_set(compound_mapcount_ptr(page), 0); - atomic_set(subpages_mapcount_ptr(page), COMPOUND_MAPPED); - nr = thp_nr_pages(page); - __mod_lruvec_page_state(page, NR_ANON_THPS, nr); + atomic_set(&folio->_entire_mapcount, 0); + atomic_set(&folio->_nr_pages_mapped, COMPOUND_MAPPED); + nr = folio_nr_pages(folio); + __lruvec_stat_mod_folio(folio, NR_ANON_THPS, nr); } - __mod_lruvec_page_state(page, NR_ANON_MAPPED, nr); - __page_set_anon_rmap(page, vma, address, 1); + __lruvec_stat_mod_folio(folio, NR_ANON_MAPPED, nr); + __page_set_anon_rmap(&folio->page, vma, address, 1); } /** From patchwork Wed Jan 11 14:28:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13096759 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C9FAC54EBC for ; Wed, 11 Jan 2023 14:29:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C5B228E0005; Wed, 11 Jan 2023 09:29:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3154D900003; Wed, 11 Jan 2023 09:29:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B739D900006; Wed, 11 Jan 2023 09:29:09 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 405338E000C for ; Wed, 11 Jan 2023 09:29:09 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id D6FFA1202B1 for ; Wed, 11 Jan 2023 14:29:08 +0000 (UTC) X-FDA: 80342750376.08.BD57899 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf15.hostedemail.com (Postfix) with ESMTP id 61C87A000C for ; Wed, 11 Jan 2023 14:29:07 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=lOa7QyLO; spf=none (imf15.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673447347; a=rsa-sha256; cv=none; b=ycXSiJoWI/lF9j0Dw/rloCkv+MQFVpe8mAFmHL2ravtuiwLAVrq7UEeI4LP1ySxHJK/iF2 JhZTwC24CH3qrGIXBr03VwHnQ68x6S73MTu4ri511ukHs3wKm2Wn6BXmRrE07rycW+9pOZ UqRrDUFusKEJA2JvUeFYgLLXWdnVkQk= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=lOa7QyLO; spf=none (imf15.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673447347; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7dSp84UZ0ULVEnUK/8GfgUxk5eRD63Nb9UCo8eq6tck=; b=5gzJKZr50+Ifie2z1sSo//Eqqkc8l51FZwS4KFhV14Daxepa2qCupkMiO5oJbMrKjJ/WZL yahjoH7MoZ4ViHUyvIl2GurbiPMyxTw69TZ/JM/nPGJ/s9QqKBTBg8CU7dmfAbPMNmuMKW z1FNykLrOfw1EqtuU/MtmVlixUqeQhc= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=7dSp84UZ0ULVEnUK/8GfgUxk5eRD63Nb9UCo8eq6tck=; b=lOa7QyLOd2L6Dp/uQBISmjFY3L n+N+NOQuYEYILIXv06FPxR3z2BsEPwXxtUOdHsZsT5THBTwOT/B8JbxxKOqLQampLB3Uxnf5MaN3t eZ+Jf1O7iJDnPWhfIDD/Y43NBoL4inP4clotLqSIaylJVctK1iSJI/zCM0PR0Pl/WhyZ05RxgC8RY M23TgSwlRi0Em4awfVO94zttvkQvPUY/cFyYPIRa/D1L3cOvw+7YFYSTbRW4BSKMqAeNGHq3WbYTt zfMHNb9B8lP5Kdy423AB5msNUsvdW6cP6SXOE+9NcokDDaQa1ILDSKH1LlC1xIK7hSbeQ0o5Lxqbq tSocZqMA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pFc6P-004CYT-QX; Wed, 11 Jan 2023 14:29:17 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Hugh Dickins Subject: [PATCH 09/28] page_alloc: Use folio fields directly Date: Wed, 11 Jan 2023 14:28:55 +0000 Message-Id: <20230111142915.1001531-10-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230111142915.1001531-1-willy@infradead.org> References: <20230111142915.1001531-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 61C87A000C X-Stat-Signature: xitqx9kndjqfjbcrrpusan9b7tw3f6zt X-HE-Tag: 1673447347-742591 X-HE-Meta: U2FsdGVkX18jo8hLvM6hhvb12kWULu9zATWsKMl26hCDkwU2MaGogyjHLdZbTwFCN3GpQV44HQBGAlRjqtUbx3TTn2cTmPfjw7hg3qMWF5M7rN8L0LWaE9e0CZrayJf8tc8f+BGkxbXflr+xqLN3Qft//QvjnwrvqGxDPv4cUx242WerQoWZqhbdw7LSzfRQhaTT6ySyzG24DEjpwk9RkLQr5u36stwObAXyy0rBnxMnscVnf/Hjh36kLg/+rzE/3HTFOsyVoXqykomYrioef9qTnj4yAaXxkNgjctJQpJ/Lexf20T+7mqkvh9nBHcm5ppgNWCVPJuMUbxtL9MyaqwiGo/qGDT8pfTgtUGA7/EQB6g31e2o2KRGrZiiPScN4r5eXrfAawIuqHiMCqdE+Kb3z6yuY0M/B0GFquBjghvQywbKsX3XJX0FMjZaHJyYMznm/JImrVP93WGtdoHDBARTeNNyMECmttMBMs7j5WuE8HZbavLHHMsA+jh74WdmqOqivG9Lf1CbGdevSIkSAQMUCTohu+URCp0K50ih7SY+i6BC54QGr7aF6YKH+bfyRzumSRJRs1LyeYLBKHfzAyqvVgfs+VAhivRr7/tqhbaEyty1sv2RVu2QWRjrn4p3HgM6v3I/MO2u3XS8H0mAET29zgfVjbvSu8pvnXAow1Kk6OT8o/kfG7b2fvT4/1WZ3lfKl223mvpn4a8xuyfldxePS9lYK/zTiD662fZByNx5qQ67CmokOVd9fkFoc2kjdf+x8SuvmGpF64NW9T323wtwR/HfHMY3NRVKe3OY8CPlm3sucM1FBA5Qc7USfft8oelw3JtN5NKBMyc5Zw4xNjtt+Wjk4To8TL6s/z42AJTg2pSIKZ3ymelVY3cxTBDkX5w6yv/emuzFs1QC2+WkRqFtmCOtKggRSq/ETCXfdVeoCr7A0G2r9nA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Rmove the uses of compound_mapcount_ptr(), head_compound_mapcount() and subpages_mapcount_ptr() Signed-off-by: Matthew Wilcox (Oracle) --- mm/page_alloc.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index d1e5ec875fd0..946a4ab7b278 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -779,8 +779,8 @@ static void prep_compound_head(struct page *page, unsigned int order) set_compound_page_dtor(page, COMPOUND_PAGE_DTOR); set_compound_order(page, order); - atomic_set(compound_mapcount_ptr(page), -1); - atomic_set(subpages_mapcount_ptr(page), 0); + atomic_set(&folio->_entire_mapcount, -1); + atomic_set(&folio->_nr_pages_mapped, 0); atomic_set(&folio->_pincount, 0); } @@ -1309,12 +1309,12 @@ static int free_tail_pages_check(struct page *head_page, struct page *page) switch (page - head_page) { case 1: /* the first tail page: these may be in place of ->mapping */ - if (unlikely(head_compound_mapcount(head_page))) { - bad_page(page, "nonzero compound_mapcount"); + if (unlikely(folio_entire_mapcount(folio))) { + bad_page(page, "nonzero entire_mapcount"); goto out; } - if (unlikely(atomic_read(subpages_mapcount_ptr(head_page)))) { - bad_page(page, "nonzero subpages_mapcount"); + if (unlikely(atomic_read(&folio->_nr_pages_mapped))) { + bad_page(page, "nonzero nr_pages_mapped"); goto out; } if (unlikely(atomic_read(&folio->_pincount))) { From patchwork Wed Jan 11 14:28:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13096758 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ED425C5479D for ; Wed, 11 Jan 2023 14:29:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7AB098E0008; Wed, 11 Jan 2023 09:29:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0F0A18E000D; Wed, 11 Jan 2023 09:29:09 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9C503900004; Wed, 11 Jan 2023 09:29:09 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 2E6B28E0008 for ; Wed, 11 Jan 2023 09:29:09 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id EFF2EA0CE0 for ; Wed, 11 Jan 2023 14:29:08 +0000 (UTC) X-FDA: 80342750376.05.5BFB569 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf13.hostedemail.com (Postfix) with ESMTP id 340152000F for ; Wed, 11 Jan 2023 14:29:06 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=SRBWuAZr; spf=none (imf13.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673447347; a=rsa-sha256; cv=none; b=YZTU+FdEbDE8ezBuDZ1OYxXqYqfgHSOQxP2XkbVSZhB37nclLqBX2yf7asvx1M+00xR6xV D/Qw9BrweEVK+DOHkFrJaGPQ2/enncOpBaiLTO3bHafvMDYiYO/k6hf105nN+vFJLIz0uq owlRg6UThMYPDc4J3ySK7jq74od2FXs= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=SRBWuAZr; spf=none (imf13.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673447347; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2gT3v7v1xMDlHHMu0m2jx4osbPSH8IBycrujL01k8Zc=; b=xkROCtB4Zc3ko2usaVmLM0lvoyjmLd1/Oix6vIAo3seJXjyiYigsXX9G8dtZOIxl3Lh85L TZ5xojVD0A95aD/sF6UIxszkPCi7wWve0QgfuoVSrnPVSnE2TUkRu8uv2v5wFyNw7hC0Te EvEtvd5nSGKQcmog56U5NGSiOdQ+bXc= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=2gT3v7v1xMDlHHMu0m2jx4osbPSH8IBycrujL01k8Zc=; b=SRBWuAZrAjQGkXq1XmS4D8PJA8 R8ZONMPB5yWb/VarORzV4HvDppIp3hqyW1HAIIAxOdEfzTw/LiiphIvN4MgC31SZaR+uOxPhvkLav tHxoqf4sekXd6JPmpAat9SD6kPnG0+71f/zkm/rWdNqy+1GF/j/hnAnFoBdJQ2wDlBqWQARyxwIx8 oNy48vhzdYGKORuoV11hcAQOVMKJ1kPHjPQn+OpCRmwm1NTQWVdTMDT/q8LLvpTPUyW/MfbwpmJY3 T2Brv1TC9sGLdgsupAmzeaAukFdF5rLDbIE0l0iHeBQ5lWewj3UAJpMW2ESUe1nYq96WrAX5C7VxQ jvfCKpOg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pFc6P-004CYZ-UU; Wed, 11 Jan 2023 14:29:17 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Hugh Dickins Subject: [PATCH 10/28] mm: Use a folio in hugepage_add_anon_rmap() and hugepage_add_new_anon_rmap() Date: Wed, 11 Jan 2023 14:28:56 +0000 Message-Id: <20230111142915.1001531-11-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230111142915.1001531-1-willy@infradead.org> References: <20230111142915.1001531-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 340152000F X-Rspamd-Server: rspam01 X-Stat-Signature: 9o9awdygnrbg4got8fx4xgzpyjg36kw4 X-HE-Tag: 1673447346-594784 X-HE-Meta: U2FsdGVkX1/bNPu9JBX9o1I479nlja5fXS2vP3a/zTZjbK0k6/RH82C/7mY7RfVIogUdlbNy4HYzDxYwAs1TqBwJa35/UsulWEWtm0TmO52WUx+I7RQQ8qpMM8N6VFDIie3sCPvjy5z5HOv6SN8wCpjYRmdm6CIE3VLKWE58hxGFOuoQaVEH6mUlt4PUpKG7LODqrIlH0Z5DSrr0D8RkE/EmK+jxhLxaUT5wiSPih9zZB1K56TWbAJ9f5h73D2rSLMtXrTuisQ/7GV8FVYexFmAHIbsiFpqrTDc20wEK+8WcbsNWMpsxFi/Mf5D3AXgn9ynUhjmMRww+Lvgb49c7uQipBWNr1CefeIbbtKvXTWzlkW6XPnm1AFfuuHOFh8Rg54r+2p8U5rhHYc6+9nDyuegtOczAopm3Z5klFRXG98M+gYBJj1dsZtX2B2WuD91eQ72hE3zL+s2XCSc4xQvOdQ/H12VxCWzfCJqaCB9mV77r6A5v4j8SXGqXRMDSiMNvyTmdV/2+4fJ/FVr0vx54OMU4iYOT3l63Vo4nU+j7wYgJ/8BmHKA38Orv71Pej8Mr+mmRLAZtdvugRrrzH+mabD3hfTY29FM6Utdkif8y5lUIeOa+LlGXM0/MtiRTeycBXO1zFGzJRvxC9RhwcgqvwlN7eyglQ0Nlp2t+bkwJeH366uacw7a6euXmKEaTgx3HfSurStdWMAm2jPZbWiw6JzRYdy+q8X8SsB75g2W4an8rnxt+OF0KwNGxSDHzUqrvG4UuL5et3z388tcnFQgDZtc5TB3PI6oofPXBZ4GRjxmNYWnBQBv01WfjxyLlqeydRPRJ7nZLKwBFmIc9DFlryf4ZhzwXIAfNT6tHM7a3RmsaFYdjJB7C74H92BJh+wj7a3wvwpdKEzA82BRDJTiaz8V0Y9xh7ScH5XbANUKMO5T2/qAkslZTHA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Remove uses of compound_mapcount_ptr() Signed-off-by: Matthew Wilcox (Oracle) --- mm/rmap.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index 2749e1466b09..462b334d6842 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2542,13 +2542,14 @@ void rmap_walk_locked(struct folio *folio, struct rmap_walk_control *rwc) void hugepage_add_anon_rmap(struct page *page, struct vm_area_struct *vma, unsigned long address, rmap_t flags) { + struct folio *folio = page_folio(page); struct anon_vma *anon_vma = vma->anon_vma; int first; - BUG_ON(!PageLocked(page)); + BUG_ON(!folio_test_locked(folio)); BUG_ON(!anon_vma); /* address might be in next vma when migration races vma_adjust */ - first = atomic_inc_and_test(compound_mapcount_ptr(page)); + first = atomic_inc_and_test(&folio->_entire_mapcount); VM_BUG_ON_PAGE(!first && (flags & RMAP_EXCLUSIVE), page); VM_BUG_ON_PAGE(!first && PageAnonExclusive(page), page); if (first) @@ -2559,10 +2560,12 @@ void hugepage_add_anon_rmap(struct page *page, struct vm_area_struct *vma, void hugepage_add_new_anon_rmap(struct page *page, struct vm_area_struct *vma, unsigned long address) { + struct folio *folio = page_folio(page); + BUG_ON(address < vma->vm_start || address >= vma->vm_end); /* increment count (starts at -1) */ - atomic_set(compound_mapcount_ptr(page), 0); - ClearHPageRestoreReserve(page); + atomic_set(&folio->_entire_mapcount, 0); + folio_clear_hugetlb_restore_reserve(folio); __page_set_anon_rmap(page, vma, address, 1); } #endif /* CONFIG_HUGETLB_PAGE */ From patchwork Wed Jan 11 14:28:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13096760 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 92293C63797 for ; Wed, 11 Jan 2023 14:29:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 063BF900006; Wed, 11 Jan 2023 09:29:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 648FD900004; Wed, 11 Jan 2023 09:29:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F1EE38E000C; Wed, 11 Jan 2023 09:29:09 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 45BF68E0001 for ; Wed, 11 Jan 2023 09:29:09 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id D9A311204EF for ; Wed, 11 Jan 2023 14:29:08 +0000 (UTC) X-FDA: 80342750376.09.3E98DF4 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf04.hostedemail.com (Postfix) with ESMTP id 59C8140023 for ; Wed, 11 Jan 2023 14:29:07 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=hVtu3kGA; dmarc=none; spf=none (imf04.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673447347; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kmo9D0eOQK4gl5x/N93BIarcQ1MlfQ8S0IUSdbRs2jA=; b=Yi4H0cwb9GrN4Dki2+6Y/OFZFse7mwKj5WJHuuQoQ9h+QJCt19Ru9j6bPq5Te6lULlXDYn M+T9qcB89S/Vd6xTlYo3ZuKza42RRB/Q0Fe1rYzHTriiQXdhA0bniY5oTMYr9W739mBl9N e3wqENl9RSnDfNkhqmLhGuBEP8Neea8= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=hVtu3kGA; dmarc=none; spf=none (imf04.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673447347; a=rsa-sha256; cv=none; b=QMGpCUaHMY8i4hXH4TKs7qQ9Zr5GfS4uORGfWaFuxunYFp/T2IUV4EGsBylPMB1OEsAmHJ Kp2HVeEYMkUtY+SR6W7NHSdZytwmRoS5OuGWl67dCu3xmlwka0FjyPv2Jbmkv8RDlIpt4q o+5/enKS3FH0ZhO770mF7UvVcZx6PSE= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=kmo9D0eOQK4gl5x/N93BIarcQ1MlfQ8S0IUSdbRs2jA=; b=hVtu3kGAtuXR8w87XpIyiGCbDe S4LTCqI8a6fm3G11+cai1Vy4QulMK+MuaMns02flG/AH4NYVXddUwchZCRCTdvTxx0QBKja0pdHxC gKYYigCA1Vr06JTZWJdRHlJZdsLbZ3scprPcCk5KeinEfmCZsOL4Y7bAHttu99Nw+nvE96oaZ7gdP gCbO1PJ0yl3xA5HzBJHF+/bI7H99AneJaZtvDhhJQ9rN2tA+uOPowWinfZHuMYW+Y8pMbzsGlQnJB sFBSFcmnjU6VUl41oPTEN3erH59WsiFLrVAks+w+vqyO3v+oTZiqwZSevlFRoYvLDim//b2nF8A0m l/LNukJw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pFc6Q-004CYf-23; Wed, 11 Jan 2023 14:29:18 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Hugh Dickins Subject: [PATCH 11/28] mm: Use entire_mapcount in __page_dup_rmap() Date: Wed, 11 Jan 2023 14:28:57 +0000 Message-Id: <20230111142915.1001531-12-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230111142915.1001531-1-willy@infradead.org> References: <20230111142915.1001531-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 59C8140023 X-Stat-Signature: y5xi5b9nchw8gmhq3qr5qej3jfjzs9xm X-HE-Tag: 1673447347-5288 X-HE-Meta: U2FsdGVkX19hYd5O6DH/bN4BZKEI1W7O6BI/BJP4Mwvjd/yGHlFrCeD7dkgnUecBJYtRbBqgTAfH/3i/DhGrVWn0Y+BdpH5Hb6MhgxwnJrNFTsCWHQaa8xx7sc0UF2KKLfKgHFKbCay8tSDJ7z7HQFUUM9NtWLtiQcKp/agwqf7OfQH+DKRNb6GUE1XHuVCaT2uU2bm7mxeCeim9ra3xqAVl+fC0GBWQ12Ou9TtdrDVHDcf7C81OIwEdWzlt2Uozfy4LeThUWp1R4OFDxnkxOH81xYqfmxqnCIViO6DbiTnP/aHoX/jFATSs7ygls73TUGLwQ9JFScZ8Tk0m/nBrQaPYLssa6U7cWBQkIIOmUJ403SQdqjS8d/+xX0C3KcVfpZLHKc5HSabpY5LCfbgJWwSySXgHGHb3p+4pO4KQGDU7IcNZ3z7WGgjg6cpeUMrrPFIN+MM21SDmcpDAAPEZyF2eemVToqJkESJ2zXvxtDUbrL0CQyy7X/1YPq5qsTBGvwowhM0EBW4mX2ESK0V6i7RoyfEjnntLY2kLaTNS1VCRNH2RHBtFtTO9yKafvZqjb5geum2PnLDx+480McQ/DTFwFb9qUV5/0ybFBd2Ay8OIil5oiuDSpa1r6DozQC9EiJD5pzjucJJVHBy2QhlzyTOeavNb9fm8LTz7F5FXSMKz5YxGideG5d1Z7sAO9Fd1dUjrYt3wgFVOqGzk4jpm3KpdiNZXESx7ejfiRN6IiDtS0/niWyaKR+G5GnK2aesr8vX1lJ8hOl+YKyDvAW6UXFJCmJaktdXODHGrpwKcESWmSD5HIE1zAhMnPG8D6dRb4/U4LkWLL0E7J1xVTb56/HmNawsrrBRfRfEhthDv/R2TH+nFRvj2Qp/pK3azZVZtzvwQBVjwMfWRuFaGZYRSrTYpPvRkUXfHjPOPQL1uDLk6BCIuGXMsMg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Remove the use of the compound_mapcount_ptr() wrapper, and add an assertion that we're not passing a tail page if we're duplicating a PMD. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/rmap.h | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index aa682a2a93ce..a6bd1f0a183d 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -208,7 +208,14 @@ void hugepage_add_new_anon_rmap(struct page *, struct vm_area_struct *, static inline void __page_dup_rmap(struct page *page, bool compound) { - atomic_inc(compound ? compound_mapcount_ptr(page) : &page->_mapcount); + if (compound) { + struct folio *folio = (struct folio *)page; + + VM_BUG_ON_PAGE(compound && !PageHead(page), page); + atomic_inc(&folio->_entire_mapcount); + } else { + atomic_inc(&page->_mapcount); + } } static inline void page_dup_file_rmap(struct page *page, bool compound) From patchwork Wed Jan 11 14:28:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13096776 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A7D3CC54EBC for ; Wed, 11 Jan 2023 14:29:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F003E90000C; Wed, 11 Jan 2023 09:29:13 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BC6538E000F; Wed, 11 Jan 2023 09:29:13 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EF646900003; Wed, 11 Jan 2023 09:29:12 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 999DA900003 for ; Wed, 11 Jan 2023 09:29:11 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 6BCCB1C6499 for ; Wed, 11 Jan 2023 14:29:11 +0000 (UTC) X-FDA: 80342750502.09.B0614AC Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf17.hostedemail.com (Postfix) with ESMTP id E8B814001A for ; Wed, 11 Jan 2023 14:29:09 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=LFNd83qA; spf=none (imf17.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673447350; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=oBSy+RoKw5Qche5S64mF1NtwyV9/tXVwrjS1i8/b8Xc=; b=yrD+PoJimRGs4WSS9++M9SbEZqli3MAFzXb+VtmqB+6ZJNPmFqpNUxmC0b4RteynwR5XHY 1FllogyzEL9FXE8E37qEJLFkXST20vj0F+6daxXTR22zxVz3m8Qnc6CUldLc7l5e3WM0Oa 9+frpOuJtztYHYX1vFqIGeLdeasnAaM= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=LFNd83qA; spf=none (imf17.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673447350; a=rsa-sha256; cv=none; b=DxpTgcNo8RMG1ij2qK2Va4H5UNpUVImPzzSpPs+xkdEZUS+ieElmPggNxpcJVl0q9lJjWp GJM+A0hpsXeQV9LFqiSSuvH48PrlVzlVe5kVlnx/km4xw/ooStVTTU9aybj/EA+mv3bmis NRSVeb16Ag6pCh6Q5dlSUj6lRINk1XE= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=oBSy+RoKw5Qche5S64mF1NtwyV9/tXVwrjS1i8/b8Xc=; b=LFNd83qAIIAIxwnXz2doZUJ5Fo 26aslhdCSwbGff4HO6ZNnIAe8QVv4lH4B0wNKHaiC/yJdybCIPUKafcHpWGCLUQWzPY6Ds2MZnAv0 d3HxYKJhlqtuu6/7fyYPcItddserJydNRCGt2B8oJ1iIbfC0W2D0ECIeWwipiWOHm8XpPutQbgk8c sdfSxKbj0kMpuSxC8aXKir9ysisURraw9QFq3vF9XCL+AJoWbnpVGjTJblf1KMUv/ajUjS7JafBUI YixQJPLae9Q4NG+HoTSlie9iFFPqk0TSA7QBsFiPgCbCQmSonjG3lIG2CjP/FbJUFckBG9CZnv/mT yV9l6DvQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pFc6Q-004CYn-5y; Wed, 11 Jan 2023 14:29:18 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Hugh Dickins Subject: [PATCH 12/28] mm/debug: Remove call to head_compound_mapcount() Date: Wed, 11 Jan 2023 14:28:58 +0000 Message-Id: <20230111142915.1001531-13-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230111142915.1001531-1-willy@infradead.org> References: <20230111142915.1001531-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: E8B814001A X-Stat-Signature: cfszuq8zow8wfbxbji31wo1rjozmeooe X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1673447349-184308 X-HE-Meta: U2FsdGVkX1/KH+/yU6nEA+INsBF7WnK8N1lKTmez1ErzYZxR04P7TPMFQcESpQGCxXUrheKXnaUOqay55nwhcm8l02Pn/EH7OKgl1EBAJ35pimsJYconCl0k90PPP64umqnyf0dXkbTJ7ElxCFvJawPFs7CWFrwQR9rjdClO2FCU2amAsDJRex14lxcNa4C81BlxM1kCy9ThF2vRpSrYuLKcMtkOIuBhiLib44MrECRPxUNQLS+OLA2KXsn61zSDjc7nTEDSo8pmZwYMl5/TO1pbdWL48XhqsarQkmJhj122CXpmenvogNzojYaqe9QVPut5c+G14LZzKTbXzTPBB9vxZ5wmSpEnRg9XRac8UP0slEbQFYLwZUZbuHYVkTwZ8gUnxb/5UEbrfIWbsnqOZHFcQ/VDQr03B5gnZEFUClCPrf0OEBoiWDVOunxym2PnzNtJAt4NJl93m5pBDCXdAiwZtPEP0gwHF3+fsrCM41SZ3Ku1uSHwnFWrynTyjPWIvqaIQvbbfKKDbRvgWRdBgSKj/ZYyp7f7n/25RXTawoVA+66c4RpmCFJl38R5hoc14gW66lc6XpuubAJ8Aza8Dz1vAeOwZWlfnzAy6HN1AVrmFpoz3O7xHBfDgRJYiCFbkcaW9uQ0x3dLVQ/+PFugt7RqL3wNzlsCfLhEEoXmCxZ6wbVuea7iAVYmHkNRB6RyY5/1sMJx/08FZ7oR69sydhxyJ/2nbe3ITMAVq3VHVem/q4wDT+Ih0zvI/3BbTjc7LkeaW2Vlk5I1CvyjtbJLXO7JLz5OkvME0IoWcx8gAMZjbdYJEAOEmT1sr/HfryzCF9V5ceMvvS5Bdkt48zo5vJebT0vbDeWXZNe/z+cQGdkE8RmS38TYTzuxHpRcwXcGU6RAaXTHNFNM2/3YOO57gPIiF5hsNoKiChe2awPM6T68QKCJzP2Otw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Call folio_entire_mapcount() instead. Signed-off-by: Matthew Wilcox (Oracle) --- mm/debug.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/debug.c b/mm/debug.c index 8e58e8dab0b2..9d3d893dc7f4 100644 --- a/mm/debug.c +++ b/mm/debug.c @@ -94,9 +94,9 @@ static void __dump_page(struct page *page) page, page_ref_count(head), mapcount, mapping, page_to_pgoff(page), page_to_pfn(page)); if (compound) { - pr_warn("head:%p order:%u compound_mapcount:%d nr_pages_mapped:%d pincount:%d\n", + pr_warn("head:%p order:%u entire_mapcount:%d nr_pages_mapped:%d pincount:%d\n", head, compound_order(head), - head_compound_mapcount(head), + folio_entire_mapcount(folio), folio_nr_pages_mapped(folio), atomic_read(&folio->_pincount)); } From patchwork Wed Jan 11 14:28:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13096761 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DC893C5479D for ; Wed, 11 Jan 2023 14:29:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2B672900004; Wed, 11 Jan 2023 09:29:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B067F900005; Wed, 11 Jan 2023 09:29:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 471888E0005; Wed, 11 Jan 2023 09:29:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 599AA8E0002 for ; Wed, 11 Jan 2023 09:29:09 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 04156120C0C for ; Wed, 11 Jan 2023 14:29:08 +0000 (UTC) X-FDA: 80342750418.04.FC6DEEC Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf25.hostedemail.com (Postfix) with ESMTP id 52FD9A0016 for ; Wed, 11 Jan 2023 14:29:07 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="i+N/Wkxa"; dmarc=none; spf=none (imf25.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673447347; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=G5oV+bri3hYrpZpECQtheMeOQewtuwycnzyaDeAOK/k=; b=oCIEt5cAYM+cmc+yYudISY9KGPWm3w5jY19X+LoYfT9afmMHmonA/4PjC+R5H1ZVipyajA nwUycOIDfFVB+pQtPPj250z0QU9RoUQD/2xIwK2AO9CIOOMEpQUbPsBVnIezDgdquJhgpf A/dbd9p9dgSc+Xjgfn4rnMn6sUs7oP8= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="i+N/Wkxa"; dmarc=none; spf=none (imf25.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673447347; a=rsa-sha256; cv=none; b=T7rkoN1NN3pKfuz2aQTPaUOlGzp8EGRjgdc8gi9NHIh69vzeF1i+sQz2b2G8icaivDyO4t Hqd0j4jxn7rwOV/nl+NT7/9qDo+LUbquF3BYnh/Sbexn8mgcGH51b12d/JCBqmK8neYaYn qPTg39u985DWemdeljZdc0nxJJuxANU= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=G5oV+bri3hYrpZpECQtheMeOQewtuwycnzyaDeAOK/k=; b=i+N/WkxaC3wYdpN/S8J/mfMnwE jMxiv46zKhGjN29ZExlZotL/nk4JpzzigPYVNdn6/9dv0I3qe0+uP9ILqUNf8AsSZpzBiN/HSR+gn BluTxegBzsKaHKme99inIexZZV3UCmbrUcAVYnlf22Zu8D6+Tk/lKuk2tRHa1nVW6nlyIcaPULz/v ms/CVUlAsXjHMm512QtwB0QBQxBlcOMmxTylLZIZtByLdsIrJE10/3WB9K+37SLouS8XL/xPT1KHg EXS+FfQTSSflvyC5DIiUqvnOG02mrE2sDY2v1N0dXFrus3FkNDAp37C8jseC+4+sBYlrRx6rfc5tw tmJPqmfQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pFc6Q-004CYx-AG; Wed, 11 Jan 2023 14:29:18 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Hugh Dickins Subject: [PATCH 13/28] hugetlb: Remove uses of folio_mapcount_ptr Date: Wed, 11 Jan 2023 14:28:59 +0000 Message-Id: <20230111142915.1001531-14-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230111142915.1001531-1-willy@infradead.org> References: <20230111142915.1001531-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 52FD9A0016 X-Stat-Signature: zxr8ohjp37zcfjpw36uqdp7wioryozai X-HE-Tag: 1673447347-56850 X-HE-Meta: U2FsdGVkX18rlteaq9NgxxH0vUnLDsIIxJG/k5ZphTTHfEESHL7GC5JBHJ9NtTEXhXnTjfG8ZC/YGOoAN3Y/xtNxAbfZE2ojcq2oTMfoRtwglQ7T0HkjZLXMZAOc8dPV0i2ddi1ujBiX/3v2T5FzNZge8YyKc2NmYfcp/G3ixHw4UmrjKGGw6VOle9JU8x8gXoepw2hD+LdmMysvX41VXLOn0ts2NcHBpVo6fg1u0yde0a5f88zSdP9CqW2FWTI72qpboKeqnWjFHhIbUkZrRiRETJlEqkTs5hVo0BreeIFzCLg0VEFNjax5vNrSD9Mls/gnxzfUOgMxJ/tY0Q3xGJS1xvsvBBt6rREf+tkK0aghNr6fvg2mvArByCsAQ1mdCyMevCBKUq0aAZQ0FxC2ZpogDpTsG9TI1Ee6iZieoXLdXf1AxpklwWiEKbqpcwO8eLJvoHHGRTydXgwLnFetPNZOPrGDhItaQBAXx7aMzH5n64YRvW7rjRnD9gH3VKPNgV3JlHic70LY2KDhFZb+YJ15bqO7Um9zQl33CHjv9GuXj3TiKvNZ4LNSqGNqte1dZi6xV+R/J6v/BHscJb2YWvAhirAUujwkqolownpvcx/kAzzV3i5uv6SdQFpQhUsHSRIL6ZSj7rPEtRpNSWLL5ms4cssfMTGsheBi56wnWev/lHvzp8rck7er3kObDqgCGkyw+435yI5t79oSjxaMjQvvZ/l+6QR5rcZ2OvyIudXErLOR9dwImJc/O0hfiRsQTRzazHLR5+pVKS/uVlQOxZxhawhTPZ0pMqTAAccASR9NggzUDFAbWBe2ke4jJo0Z7zSLdUjPDkE7h0i6bWcrOGA9yzuhWemXeFZhyh9MoWRw71fMYVFuhE1+Wd5m4PDXqStPGZsPY35N+a+Nr98wwW+Vt96uSG08p82lKKj9YNtwcf6gqMFybg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use the entire_mapcount field directly. Signed-off-by: Matthew Wilcox (Oracle) --- mm/hugetlb.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index c9702224931c..a68e0e597a8f 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1474,7 +1474,7 @@ static void __destroy_compound_gigantic_folio(struct folio *folio, int nr_pages = 1 << order; struct page *p; - atomic_set(folio_mapcount_ptr(folio), 0); + atomic_set(&folio->_entire_mapcount, 0); atomic_set(&folio->_nr_pages_mapped, 0); atomic_set(&folio->_pincount, 0); @@ -1996,7 +1996,7 @@ static bool __prep_compound_gigantic_folio(struct folio *folio, if (i != 0) set_compound_head(p, &folio->page); } - atomic_set(folio_mapcount_ptr(folio), -1); + atomic_set(&folio->_entire_mapcount, -1); atomic_set(&folio->_nr_pages_mapped, 0); atomic_set(&folio->_pincount, 0); return true; From patchwork Wed Jan 11 14:29:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13096755 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B5B20C54EBC for ; Wed, 11 Jan 2023 14:29:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ADC08900002; Wed, 11 Jan 2023 09:29:09 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 865468E000E; Wed, 11 Jan 2023 09:29:09 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 407EF8E000D; Wed, 11 Jan 2023 09:29:09 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 22B2F8E0005 for ; Wed, 11 Jan 2023 09:29:09 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id F3FBD1A0B20 for ; Wed, 11 Jan 2023 14:29:08 +0000 (UTC) X-FDA: 80342750418.08.3EF3F79 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf14.hostedemail.com (Postfix) with ESMTP id 6FC7310001A for ; Wed, 11 Jan 2023 14:29:07 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=XR8fttJD; spf=none (imf14.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673447347; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IKp5GUfaVn0YeovLGI5UqQOKYLaV3dJc1SkhOFhWgXM=; b=Jfvap9ROfJoHapgtu1HxoKpErBANG4jBCXtUAKDsbv2fqL6weqdRb5mwpkfmlP+Niiy9sl eFIOVjaXFB28bAPi3IclVbWGg+AqrHeMsnZtrxPInq7XCIakVnOY3XIlqaB9GI0caNwMBT In0cglBKsrJ3RzHCTKOlSqmLWJmOrtM= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=XR8fttJD; spf=none (imf14.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673447347; a=rsa-sha256; cv=none; b=3BRZxQljF473iqXxmdFNgH66HTtSch0DwPF4up8pjtPf5P8jSCzSg1OrEBee51dRPq0gjz 9W9ZQ9/1X0vkthdtytH82O3GXDmvWVeyw8jwhPBQYJbOP9NR8fBU1NqZx7a4CGjBKssJYJ OPKfgccCNKYGWhUzXUfm7HNEORLrWNk= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=IKp5GUfaVn0YeovLGI5UqQOKYLaV3dJc1SkhOFhWgXM=; b=XR8fttJDzKYrjrQ4R6795opsOH HGeF/Zb711cDs2TmdF/Yzu2hYX7shlDmHdLqgxoYCOjT/GurzPesChu/XbVtnPkMfYF84nDEMjkTU NWN/M3/gf7YsgZ4JIp9kf+06urIwu50K2PmHGXymb0GZVjWrEaPHA2B6r7xACj6EVjf3heUn8VXfk v3ALV/XSEuu9JHqepMXHORIU7NMiLJnvrmoa7LIiEwA3d2wHQO6v+ykvrJCilp+RujaYY8Jj2rxA7 V6KXThgdXjqjDoFIkppVKFFLp1Wnpd7xXVgDEc2f/SdwQag5Nuc+zZasZiMkoSEp8A12oqLHiOYgt D6DKe5kw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pFc6Q-004CZ5-Ea; Wed, 11 Jan 2023 14:29:18 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Hugh Dickins Subject: [PATCH 14/28] mm: Convert page_mapcount() to use folio_entire_mapcount() Date: Wed, 11 Jan 2023 14:29:00 +0000 Message-Id: <20230111142915.1001531-15-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230111142915.1001531-1-willy@infradead.org> References: <20230111142915.1001531-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 6FC7310001A X-Rspam-User: X-Stat-Signature: yaz3fmx1reixtafugydopg9mwo9yagke X-HE-Tag: 1673447347-368794 X-HE-Meta: U2FsdGVkX1/zEpfTfUypMOYWIxy3YVLU8+SNFViTxJYcZJT2umtt5cGqGigIKI2KpeYa5c8QjjPYHSAqGL9KqeInQmMqH+uALRpyQ41cZ75c+04Msgi/usbIi6S2jIptUYqs21K6JVv78I2dL9XZMo8oThFQuFOdVkNRKHqT5INysvRFfI5nVSKssb6uDqpcEvAvK5Vd4LySQ5i74exYbiRYWLzcMOyP8UBizHp8dZul2R6VgkgQ5PL6WudoMKgmdv3y94pz0v+YBq/PFC9+u+UtKfB7MyXGTSJQ09hVb7PzDq9gj5AQPlb29chiMDyvOxoIP8/S8ue6pyWFUJOqdlCvI6GqlqzN+S2tWqBft3NVJ8WHm818Chk3qafO+b6qHlBu+Bj0zz/l4CWQQtfQ+CgVcqV0hIrjJUpUVewatU6U8usvfuX1htrMN/t1mOY67jOEPOCAOGTbQeFoitumX6WzfUtEF/X0sbe3JslXVRurxbGFEYhsK+8qSCIDRkgaoDhDCrUUfcqoKQ4tMOPjZLl8L17JCnqc/fV3JWx7QS4l5WhMFl/yhe+q3k680uTwZtfre2BRywUJ8/DcZZ4UIwprLU3hMgcxcksjYNgFbVmnrqqSOeZluZ5TrVrshI8LbuDgBXqomA2ieB4EKnRcYgc97uATjYAQ4nzYTtLAp+/RYypqtfK0KmiQb9eknxqdRT1eIL/bbE2m11i5VrXu3X6udPOts4FNNrk6ddOvjN4byUYlygZZ7Bmj+Z8TI0BZm2Fj/szBhpxFT7tq72TtjJs6wSbWqKAiCbt6ITdnBDrVK4q0yBzcT4FjTzJmxy1+DI8+i2waAU28AFF6d6r/myciG2J5ujzUXYt9RW7MD8nVc3NOi3RbqZgsz5AwdSvsn9mOZ17qP1AgrveGIjLs2XsCBMYUpVCSErb1XKpojycX7k2ejNqzmA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Remove a use of head_compound_mapcount(). Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/mm.h | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 8bddc7810f78..554b73dae188 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -853,22 +853,26 @@ static inline void page_mapcount_reset(struct page *page) atomic_set(&(page)->_mapcount, -1); } -/* - * Mapcount of 0-order page; when compound sub-page, includes - * compound_mapcount of compound_head of page. +/** + * page_mapcount() - Number of times this precise page is mapped. + * @page: The page. + * + * The number of times this page is mapped. If this page is part of + * a large folio, it includes the number of times this page is mapped + * as part of that folio. * - * Result is undefined for pages which cannot be mapped into userspace. + * The result is undefined for pages which cannot be mapped into userspace. * For example SLAB or special types of pages. See function page_has_type(). - * They use this place in struct page differently. + * They use this field in struct page differently. */ static inline int page_mapcount(struct page *page) { int mapcount = atomic_read(&page->_mapcount) + 1; - if (likely(!PageCompound(page))) - return mapcount; - page = compound_head(page); - return head_compound_mapcount(page) + mapcount; + if (unlikely(PageCompound(page))) + mapcount += folio_entire_mapcount(page_folio(page)); + + return mapcount; } int folio_total_mapcount(struct folio *folio); From patchwork Wed Jan 11 14:29:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13096756 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4AA2CC5479D for ; Wed, 11 Jan 2023 14:29:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 111048E000E; Wed, 11 Jan 2023 09:29:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A60A28E0005; Wed, 11 Jan 2023 09:29:09 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5CC118E0005; Wed, 11 Jan 2023 09:29:09 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 350768E000B for ; Wed, 11 Jan 2023 09:29:09 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 069ED1C6354 for ; Wed, 11 Jan 2023 14:29:09 +0000 (UTC) X-FDA: 80342750418.04.4A7D66C Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf08.hostedemail.com (Postfix) with ESMTP id 85BD516000A for ; Wed, 11 Jan 2023 14:29:07 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=YeiWvpAL; dmarc=none; spf=none (imf08.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673447347; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=glmfC0MZHhVXgE0NJOKhxVOTmYgJGmkWJ0aBpVK64Po=; b=61ROGnZRmObfRzNHq97AZUaRGL1o7gXL22aPAdju7ai8ZmfdQi8YdFPb1/iSy3r6nJQyex X/G/DyJeiKJ1bwxvJV8v3J+Y6zQWz0vhxKhdFk3fWDU6PYTUQHmXx+/bglSKLf0cVImXUw 5XHCx/RAwkbctd/dLidFB38J2VsKmQ0= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=YeiWvpAL; dmarc=none; spf=none (imf08.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673447347; a=rsa-sha256; cv=none; b=KITFPNTQNQyp/8JhHFl9WOZxAJ0yiveq0cyuduZeOY8vR0+KM2JIp8c7zc37P7n+Ops7Ya a5lVOEaiCHsBu5Fa/QNcZClHT6MPgmfwYtwtcAmiXYhbfuj4iFhev1D2mJluneKSfhK7OY B0dJQTzhVP7eBtLvv18Y4b/lBaHWcD0= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=glmfC0MZHhVXgE0NJOKhxVOTmYgJGmkWJ0aBpVK64Po=; b=YeiWvpALCFMluvgkgDmEDDk7a8 VQp5mlYfDByvKj+Z3QiT6LA1iPXXcFh5Rla56zAE3jun+fmkwK1T7dPKOwOWYUaCkUFr5a1L4ya7l YDEM6pNe6d1S+Hx0mVrrClyiFNb4HoijZO7LYuK4r9CPIT8q8+rtdL3ltyrITxyVpgj9Mta6xE91v hz4nqQgIESKZR2l+FGbQsi1Mn5wdDm7fIgOakEHKU5iA3R0VaLk2ZkUaKMs4v+VZ6fNfVyr7hQf1A K0yZ5819kfcddKbOV28gdoGrYZUa4A4VPAEQIRZISKbfQHs8J0HJ8zgsLi3TVAg7zcCmOrKEA9HRd B5ARv2Eg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pFc6Q-004CZF-K6; Wed, 11 Jan 2023 14:29:18 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Hugh Dickins Subject: [PATCH 15/28] mm: Remove head_compound_mapcount() and _ptr functions Date: Wed, 11 Jan 2023 14:29:01 +0000 Message-Id: <20230111142915.1001531-16-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230111142915.1001531-1-willy@infradead.org> References: <20230111142915.1001531-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 85BD516000A X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: dwugdfdxt77sxgjd6epgh1pnw34zqg8i X-HE-Tag: 1673447347-123229 X-HE-Meta: U2FsdGVkX1/yrw/CnGv0sufVh+JpVEJHaO5VBXHDFinjCPqE6BMpf1/COfcWLHUi1oBYn9NMsw0f/5JxqaJWjwVJNjp0gQbsUssc8zvYq2PqUsXH2P1HX/cp2nyehk2mZ6S7eXYKDokdecPS9AHvYy5uDelrHnyvTj2weUlQuRr07hh3ZGEXvIdA3KLAzLfU5KQ2i/4Q+8Grefkq5Tcqe6H/33CZT0RYf52MqP74mk4BLbzzB3pX+cM25sS+/ymNty8Nnb1wTrKR6wDWS3Bl15v9vzMd6qRvg/c4RE4TiE8U5rHN38ONBFuhhkYz6YQseBqpf46Oa0fPN6UrWGZ0xhVKbN7UgtXw9ct8r1e2uNsBxSneMKHVkGw90OrL/G6o/CVZVjf911aAsvbjwjPdFM00RjtMytmQc1VI87QxbRKH3yP3GA+Kj8+SOyGZ1hyo67b3PO+ZB/58Za6Cks9DCy2HxOZF7bKi32FXGH3d4JsPAELnuyh9VOq7DbF35O/1U2zY0w3iFLYexVwNtOmDlJ7/SqaKVKao3oqmzsHUVy1dWobQU4yvOyg8aEPGLkSF/pipZtSBfyAgjjry7qSzKtpTaRAaGR/ih7LeUxbYbc9bQSDfzXxRXIXI9LN+h4k0cex8BaF4uG+HqxGAIVGBd3U1EXMg+LwuTrn7WiXYDtYVUB9h6oSlpp8U1xGwhPZ5ENiSevWSKzx+LBzIIUhUgld6XMv4g78B0iGSEBwMysrOb2bUKEdvRfbPg3R8b98K4VxA6BAIBa7uYGw72esQs6D5H6PpDu7rQ6hasolCFpwypeqy/ZAUZD0NKe1jbGvvQ+abEYx8SUSnMn2AqV2pMpBGcRasuW3TuVXsWFcEasMcrTlnxpVTP2dY31dRdl+zvP4vP9NNcJbm5/li0CKdqARdFk//AGnlQs9B/Ymqm1of6cUnb9VAZQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: folio_mapcount_ptr(), compound_mapcount_ptr() and subpages_mapcount_ptr() are all now unused. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/mm.h | 15 +++------------ include/linux/mm_types.h | 16 ---------------- 2 files changed, 3 insertions(+), 28 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 554b73dae188..5002dd4db544 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -831,16 +831,7 @@ static inline int is_vmalloc_or_module_addr(const void *x) static inline int folio_entire_mapcount(struct folio *folio) { VM_BUG_ON_FOLIO(!folio_test_large(folio), folio); - return atomic_read(folio_mapcount_ptr(folio)) + 1; -} - -/* - * Mapcount of compound page as a whole, does not include mapped sub-pages. - * Must be called only on head of compound page. - */ -static inline int head_compound_mapcount(struct page *head) -{ - return atomic_read(compound_mapcount_ptr(head)) + 1; + return atomic_read(&folio->_entire_mapcount) + 1; } /* @@ -905,11 +896,11 @@ static inline int total_mapcount(struct page *page) static inline bool folio_large_is_mapped(struct folio *folio) { /* - * Reading folio_mapcount_ptr() below could be omitted if hugetlb + * Reading _entire_mapcount below could be omitted if hugetlb * participated in incrementing nr_pages_mapped when compound mapped. */ return atomic_read(&folio->_nr_pages_mapped) > 0 || - atomic_read(folio_mapcount_ptr(folio)) >= 0; + atomic_read(&folio->_entire_mapcount) >= 0; } /** diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index a66054a9f0b6..1cf0fcd99d49 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -421,22 +421,6 @@ FOLIO_MATCH(hugetlb_cgroup_rsvd, _hugetlb_cgroup_rsvd); FOLIO_MATCH(hugetlb_hwpoison, _hugetlb_hwpoison); #undef FOLIO_MATCH -static inline atomic_t *folio_mapcount_ptr(struct folio *folio) -{ - struct page *tail = &folio->page + 1; - return &tail->compound_mapcount; -} - -static inline atomic_t *compound_mapcount_ptr(struct page *page) -{ - return &page[1].compound_mapcount; -} - -static inline atomic_t *subpages_mapcount_ptr(struct page *page) -{ - return &page[1].subpages_mapcount; -} - /* * Used for sizing the vmemmap region on some architectures */ From patchwork Wed Jan 11 14:29:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13096774 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B04D6C5479D for ; Wed, 11 Jan 2023 14:29:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9F67F8E0001; Wed, 11 Jan 2023 09:29:13 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 66E8390000C; Wed, 11 Jan 2023 09:29:13 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BC1AC8E000F; Wed, 11 Jan 2023 09:29:12 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 74A688E000F for ; Wed, 11 Jan 2023 09:29:11 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 51CB61A0B20 for ; Wed, 11 Jan 2023 14:29:11 +0000 (UTC) X-FDA: 80342750502.14.7891EAA Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf01.hostedemail.com (Postfix) with ESMTP id A2DD74001E for ; Wed, 11 Jan 2023 14:29:09 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=uiiSCM8U; dmarc=none; spf=none (imf01.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673447349; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BMbYaVMlHD0AsYDSTxv30HdoHwDdnYnBxZnWDD3/zMk=; b=GRHEoIC4WSI8276s/dBDx3W6BQ5gu973enDeDSkFsLpF+OVE723Xg0O3wo4z2mqyVjMsYy Wb3QJ5ybYZ/th+gLfZnuM1XnXH4AGfvK8waAC0BvIYKenlxBjlg5PnctQTpKIenokAmqHN Lqgjt0QUMuv6cQd3FJaLy1hm1xSQEiw= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=uiiSCM8U; dmarc=none; spf=none (imf01.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673447349; a=rsa-sha256; cv=none; b=cZfSeVUb82ik1NJwCYWyYMRoz4AM9PZU11D4BeMW8S0HQcF18RtZ6uNgFgVMBprYF3gnA2 a4TZSmajROBCh2aRk1/+RHfkJCUVygsWaHISaGLYVSU7IjcgPIh9iv6S5Qupy80Fta5lJV e4bsWBMqMhwby3b9cegI7R2TuuW77Eo= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=BMbYaVMlHD0AsYDSTxv30HdoHwDdnYnBxZnWDD3/zMk=; b=uiiSCM8U/hyGbtUl2HnxpQ5uOa jDNWPMglAGc5EMjvKMt/hA09To3NQfiI7nhFZsZID4OezZjgjRU+YUcVkqS5ZWziXdUw+sFwUTwwq nzhZaZobsWLUvQKTH63/W0G7vjDVkfqfoo/ov5Zuv45EjO1mYirkwm+KbT1dwCk8bOOaY9QpitSv0 yaOpdagvOQlghHCV8xVFdYEaAxaKRlmUT/eauVlLVMBuykCwAANf4yFPiPg5Nrw6ssc5rcPUC9t1K GqcB73WsN6iqaRnIzekAfHWGvVXbiuNcLliHTV7tJDdbrsW0BQRjzqkJo5Wt70IbiAotLc+RAbcZl IOjbWGng==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pFc6Q-004CZL-OK; Wed, 11 Jan 2023 14:29:18 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Hugh Dickins Subject: [PATCH 16/28] mm: Reimplement compound_order() Date: Wed, 11 Jan 2023 14:29:02 +0000 Message-Id: <20230111142915.1001531-17-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230111142915.1001531-1-willy@infradead.org> References: <20230111142915.1001531-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: A2DD74001E X-Stat-Signature: 1irortqeogtubz5c3pwtc854xc8z38sk X-HE-Tag: 1673447349-908371 X-HE-Meta: U2FsdGVkX1+2jsPLDe13lSFHSGbrJmOGHqCvKODxkL4n6jKO8hCmv7GtTh0M27s5eXxT8EJEiki7NfDYaDJNekHZs2Egpq9cbKajpwbcu8WWubJ0ud0r8szi89y7z+2Dy/8NEenJCZJ1oE96HBhpMJImo/EiNp9ylVOdzmXvIeWy9UCZqHwO+jzP8FdOixdb4PAOD4OXeaOdkxRTyz1ONocsEtsmG/aLZp8/aMsnyYBQPdFKy4pDOP0czgyTOofbsl8aHjBSU+BzMoAv/c+LlxTtO0ceQuSnOcFJJfDMsT+piAE+mCu0zm69oj0gCUu2Awmgx9uuBYEOQ9gtFVZrGClUE8UEM1cbS49mtTN/j6EldUgjkyaaLSxZ3jrVmHRZRWY49LgO0gu+pJ68vdiRLIvFYlLq0AimyIa1CqYZ13fbmSqDIcJl9Y2PyGGs4ukzUZ6wC1LF+ih6K5cnGKUiHs02TsZnLqFDHMe3VU5cLT1t9C4nhMclIR/2FSdjp3LdN7S1oVkpL1PTvyEZKI0+n6en4eQmlLP7CBIifST7I3VX2dlaIgRTJII55l87xDOBYm8MNHyJjJ5nO8CbbdE7BPeUOLJKQjOFwmIp39Tn0ZZRnOavbyn6oS135qU7RnKxtX41+iPhbijs64LtItIZJ9gXcbHVTGH8RXGA0lwcLMHlFUBAiSvzqAS4RKMxYmshRXWHVFtFE+2AnrUMXEMYvtUZvlfgbxYhDeuHFAoFUmH25JDKku+z0F5lNttvMlPBdeHQXP6Nr7Os1Ram0olrPxcX+WnQ348W6B3OpkLmpwMFg83tfXWb8maHJuXodMLgmAWhkPE0GBnw2430HTdYVEa5n9nHEBD9DPmews9/S+do1nwa304UFCVJAEPQ6JNCGdQe99O24MRTN0/g86pl9SQdm7j3pySHDfmeRRGFecwBwr7f0h+GdA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make compound_order() use struct folio. It can't be turned into a wrapper around folio_order() as a page can be turned into a tail page between a check in compound_order() and the assertion in folio_test_large(). Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/mm.h | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 5002dd4db544..ddf09522c0d3 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -719,11 +719,20 @@ int vma_is_stack_for_current(struct vm_area_struct *vma); struct mmu_gather; struct inode; +/* + * compound_order() can be called without holding a reference, which means + * that niceties like page_folio() don't work. These callers should be + * prepared to handle wild return values. For example, PG_head may be + * set before _folio_order is initialised, or this may be a tail page. + * See compaction.c for some good examples. + */ static inline unsigned int compound_order(struct page *page) { - if (!PageHead(page)) + struct folio *folio = (struct folio *)page; + + if (!test_bit(PG_head, &folio->flags)) return 0; - return page[1].compound_order; + return folio->_folio_order; } /** From patchwork Wed Jan 11 14:29:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13096762 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2E6A3C54EBC for ; Wed, 11 Jan 2023 14:29:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 55A868E000C; Wed, 11 Jan 2023 09:29:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id F0A4D900003; Wed, 11 Jan 2023 09:29:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6913F900006; Wed, 11 Jan 2023 09:29:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 9CD54900005 for ; Wed, 11 Jan 2023 09:29:09 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 85E7FA0CE0 for ; Wed, 11 Jan 2023 14:29:09 +0000 (UTC) X-FDA: 80342750418.29.CC31F31 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf29.hostedemail.com (Postfix) with ESMTP id D7A02120011 for ; Wed, 11 Jan 2023 14:29:07 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Ta7ihsCN; spf=none (imf29.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673447347; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FxqAkPcytKuw5YjKq8uigj0qwlZCH6XVIp0l1lIOOVE=; b=iLs+GvcdocmFyIn9BKgpMDrZUNfDszEcbfPunvE4dtpR0iScl99mk8ohMjhJfFWAcy3Rle YIk5VJYQ2WX3MXJwlUOgIy5GnpkViCt1SfmoGxkOLNUWdhUiN5C9HA3qNwtx40FHpWyLCn lmiwDuVlza0wEB8FL/LRUwvVvwP5m0I= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Ta7ihsCN; spf=none (imf29.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673447348; a=rsa-sha256; cv=none; b=PDnfd2jMo94ztTIb7fDZT9emuhRB3mF8VCt5lE9wBYOHCDkx4oNdF4kON4ljn0bBGlQRo/ Lr98bXfemJUwRkIitwiqYRkt7I6Wa3twxa4B6X0xEjCY1zw+ZEAs7v1REV3chO4TKKkd8j UAfFUWv2YLC6NWwiG3XXxoj85R6DP1k= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=FxqAkPcytKuw5YjKq8uigj0qwlZCH6XVIp0l1lIOOVE=; b=Ta7ihsCNrkX1wy4/lB+0I83MRJ G32WLk8oRk3UMzkN+1L/gVFqSU5H2kFatCztXlnj1/FcGiN+Gy8WrpV3e34nrv2ZMH7ZrYY7kUmkc FrMI+7ItiyD/RDNbBjtjb8eC9HpJaA+Op2n0SA3JuroJ277CctaK+OJ/TXBTghR/CBVAW085wDuRc Z/Wbq9zzzWeTKUddNNYBP9NRjBOaY7hNQIesaUypXEMGDz5RSicPi9S8mFKBhve9rYjvQyTxeFpTn gogqGw0Z8kVK8CO1FGUV+Y4IRC7TZf7wsDyKRxEbC2Rf2vngOYkRcY6zcx+svF3AhdOJeKxsoEknQ 2etB8Xmw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pFc6Q-004CZV-TT; Wed, 11 Jan 2023 14:29:18 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Hugh Dickins Subject: [PATCH 17/28] mm: Reimplement compound_nr() Date: Wed, 11 Jan 2023 14:29:03 +0000 Message-Id: <20230111142915.1001531-18-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230111142915.1001531-1-willy@infradead.org> References: <20230111142915.1001531-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: D7A02120011 X-Stat-Signature: dz41ehb61bbgxohmqjjfz4ttwam3d77x X-HE-Tag: 1673447347-895213 X-HE-Meta: U2FsdGVkX19aUxs2WXrB9WEdexkALD4XdbvTjSJICxTvA0Xb0XfNTlS2utU1U453HlNLr/B6zfbLx8qKPW2voDP3pj6QmcKj32tKOnmM4ZcAvmzI582pLdg75GU2i5ZV9cnmybau32x+P7TSosX9YTkL8ZLbWr/etpry2kRNbMgbIdQWVNSziZsstA6KcC/rDu6lSTCsMzl1QdpT/e8+flPtiL0FXYKjn77mjW4gGv652kJqsTGWaHcNJYY8rNCLVkJZn12aAS6xJvOAQ7BrcMv8SHx/Il9m3RvVJ+F6ZADJd420tn33JabYc46E2Cw31J1MHLxOTM7AsYye0Mms7lXveVHUH5Oj9WzUUkfbM+xvDkyjSD6kahbXP9X1Fs1WYAy7+mskOj6ZHO3vB64mE7h9ao2qx4v0fIQivi7JaCKIGzHO1LED1Q/duPEIKLfiG6xnatDUvgPqK+N7Zt0H5AdJ5R3RNAoevQ2mLuPrYjlglBmPuJujifHXA3omaWnwzXftRs803/vWNke71tF4/X5bYP6iq2oADS/IcWNm9Z0K8fCcp4DuB0wz6THQrElyIM1D17LoNKLxHHkqQX4LP5YLZhrfnhIkqLpAwSNvqT1BHHuppeKI3GLytzsCkX2ITLsSuTTr/g7Hx77rFZUTed6AjGcMJhekL/IjZVK7YAMmtq9Uc9njGnGOurjnU1HLkdmxezRixGt35M52C0ZCSTqJLcpWYr1BLvMc9GnRMNiZbLEk+d3ZVmfBHNcuuZyJr1M6v8wJ193YttqbrQIB2rDp6nuwdFdFAvW568COu0VrfmlLifh7E2jsM4N66cQnpJr0x7e2jjQu+akk4uXrz17sqVRaMzsxK50OPwAOrAP0Vp03pzvydZPZjMdYe1AtMm20JQHLp+ebXB9WOMnZ8xIuqGD24zBcbrsu5ZHmNZeMzd1l1B5b8g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Turn compound_nr() into a wrapper around folio_nr_pages(). Similarly to compound_order(), casting the struct page directly to struct folio preserves the existing behaviour, while calling page_folio() would change the behaviour. Move thp_nr_pages() down in the file so that compound_nr() can be after folio_nr_pages(). Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/mm.h | 38 ++++++++++++++++---------------------- 1 file changed, 16 insertions(+), 22 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index ddf09522c0d3..0b1cdaf0fa90 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1005,18 +1005,6 @@ static inline void set_compound_order(struct page *page, unsigned int order) #endif } -/* Returns the number of pages in this potentially compound page. */ -static inline unsigned long compound_nr(struct page *page) -{ - if (!PageHead(page)) - return 1; -#ifdef CONFIG_64BIT - return page[1].compound_nr; -#else - return 1UL << compound_order(page); -#endif -} - /* Returns the number of bytes in this potentially compound page. */ static inline unsigned long page_size(struct page *page) { @@ -1039,16 +1027,6 @@ static inline unsigned int thp_order(struct page *page) return compound_order(page); } -/** - * thp_nr_pages - The number of regular pages in this huge page. - * @page: The head page of a huge page. - */ -static inline int thp_nr_pages(struct page *page) -{ - VM_BUG_ON_PGFLAGS(PageTail(page), page); - return compound_nr(page); -} - /** * thp_size - Size of a transparent huge page. * @page: Head page of a transparent huge page. @@ -1758,6 +1736,22 @@ static inline long folio_nr_pages(struct folio *folio) #endif } +/* Returns the number of pages in this potentially compound page. */ +static inline unsigned long compound_nr(struct page *page) +{ + return folio_nr_pages((struct folio *)page); +} + +/** + * thp_nr_pages - The number of regular pages in this huge page. + * @page: The head page of a huge page. + */ +static inline int thp_nr_pages(struct page *page) +{ + VM_BUG_ON_PGFLAGS(PageTail(page), page); + return compound_nr(page); +} + /** * folio_next - Move to the next physical folio. * @folio: The folio we're currently operating on. From patchwork Wed Jan 11 14:29:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13096773 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 40AFAC54EBC for ; Wed, 11 Jan 2023 14:29:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7E17190000D; Wed, 11 Jan 2023 09:29:13 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 425B490000B; Wed, 11 Jan 2023 09:29:13 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8435F940009; Wed, 11 Jan 2023 09:29:12 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 0DA0990000B for ; Wed, 11 Jan 2023 09:29:11 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id CCAC2AB31E for ; Wed, 11 Jan 2023 14:29:10 +0000 (UTC) X-FDA: 80342750460.11.1826AE2 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf13.hostedemail.com (Postfix) with ESMTP id 301D12000F for ; Wed, 11 Jan 2023 14:29:08 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=FI5tQYBA; spf=none (imf13.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673447349; a=rsa-sha256; cv=none; b=aoSEIwL0+eGcbC34jfx/tdf2fuck0ns+yTJ6TR49YO646UUlIFJVofT/IdyY3wtLYMgKCF HRptprS90KiCkLFtJm+9RowRjYEzhZaMgvY0KuZp+7SDjDbx7o8qFNjflCIr7/dFFDp1bA EzOCzyep0NM3Z0xdLrvuj0HPL8dgTYY= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=FI5tQYBA; spf=none (imf13.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673447349; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GoCqV7MoGixzU//QTHO85uusqFb3KGVK6jprnb5YbpM=; b=HcPLwCu1aW/B0cKLRcX4YedBY5/efWsNoMQ5KM5dCPQATUlr8+CSUHfLTVjcpGJ76M+biU Rcs1Ql11OrLImQdjbM5M1oyUg4TMgQUqyLgnGPU+XL6aa9VBAKjui26SovTbRvL1ZVQGJI kqOs9R6LF4qdmGqIJ0oqkBeYxGKQrVY= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=GoCqV7MoGixzU//QTHO85uusqFb3KGVK6jprnb5YbpM=; b=FI5tQYBADyTXIX4FF+uS3ooESH fMDFopapyTDJ1/EK1ndUay2aHpvA7H1u38h3dv+B+4AMgcjoWL8F25kOAquIxhxmpOyuEKkZPrRlI wpYqfOFMbbY+ukk/9+Yjt4D7fFNTYGBfyonGmMLch/VomnWdDpbmiA1J7C/I3Fs+3hz67uGuf39KL 7X6zS9tqp974xpGGcvfZ5fD2phaJzCSTaEEGNkfWjOjvIR+qIwtt0qNBwBIiD58H7EJJPMioDNnHa ZWDUzHKG5bM575V3AtbeaFNCO4DA6zr3jQCr9hgGz8wZdhHP/7qUJpSuFLvo1IEdQ/bUouuPkvU0k OX7XfGgA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pFc6R-004CZg-1r; Wed, 11 Jan 2023 14:29:19 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Hugh Dickins Subject: [PATCH 18/28] mm: Convert set_compound_page_dtor() and set_compound_order() to folios Date: Wed, 11 Jan 2023 14:29:04 +0000 Message-Id: <20230111142915.1001531-19-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230111142915.1001531-1-willy@infradead.org> References: <20230111142915.1001531-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 301D12000F X-Rspamd-Server: rspam01 X-Stat-Signature: gz9ygdfnyju7jq684ieko8uo6uds6m3n X-HE-Tag: 1673447348-165119 X-HE-Meta: U2FsdGVkX18FB+uNIhDQGQnSxL+WmLMGLkhyAStTkCZDGSGBuc0qVz987YU5H2Y0X0BDI0xxPMBqD2EPMLsJS5ygiqQ4689mFVygJlz3XwDe1SeRXrmtX9E8SMKlju9LaGLL1XJkcP4LgJzwh5OwqNjTd+oVezZM3XmNQVy9RotqbVUcfcAG55PRtCIyRJ7B0a2cdUkfRSECYnTIoHuPV+7wWfvla2SeuMTNvaoo95Qe8jDdzk6OFVPqVVUXvPa2al5pPHPKr1mLsNdfvrPM/E4luNdLkt/Ev8vbYwrMY3U/xknO1huWbSOsi0lsUsDnYHQikURjfzc9Cdmea6SYT1CNegwnYtmZKBjsramLfYSh93zcx4cjmSmdYvtG+cWBiG6m0Lh9THRoF4/m23+qLx/YUreDuVyyfQPwSksTKE5DBrNC8ADGue80uJ1jN8vEXSOQW1FZFD+HxugkwaSbLwVwjnsOcwb21CKsCacgjOXsEfgSfaI2felCTz2za8fYn0ylRQcED4rKdzf7cIT7r/ZiokzkZULzN8xw2N5Vjk0IwPOhDsd0p6M2qSLRdOQFiXI6l1u5MHNcoCdb/Fgg5imrvpQBmCipTfBYOiWo5ViJ+k0pgBzgsuU3p+EJ/KCyVnMuETSzeTQSo8/avOYYcgkjr1Midkb4vbFWRZNL3qC7XFX+pDldClvZ1Wf8sVDztkka8wZWMXKR5MBlEQlyzlFixFcIBiftkoIWn7RZXNv1KckWm+Qy9dYtgAMWQBxZX97rN/ASi5EopFX2DSuhLqM81nJa95xKLUXHV0Sv4biIKWgPd5I9UmvTkT9WvLwaq8Vcsd8XE629oxbcDSzYDU0ISI2EvwQit9kEjdnLxBjQNu5TnEDohXR+aiQhpG4nfcUVoUnCeDnTWKNMMFeHaMXoTnnWdIGD/jc2Fmi57pWkMSftHmeFyA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Replace uses of compound_dtor, compound_order and compound_nr by their folio equivalents. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/mm.h | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 0b1cdaf0fa90..57d702fc8677 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -984,8 +984,11 @@ extern compound_page_dtor * const compound_page_dtors[NR_COMPOUND_DTORS]; static inline void set_compound_page_dtor(struct page *page, enum compound_dtor_id compound_dtor) { + struct folio *folio = (struct folio *)page; + VM_BUG_ON_PAGE(compound_dtor >= NR_COMPOUND_DTORS, page); - page[1].compound_dtor = compound_dtor; + VM_BUG_ON_PAGE(!PageHead(page), page); + folio->_folio_dtor = compound_dtor; } static inline void folio_set_compound_dtor(struct folio *folio, @@ -999,9 +1002,11 @@ void destroy_large_folio(struct folio *folio); static inline void set_compound_order(struct page *page, unsigned int order) { - page[1].compound_order = order; + struct folio *folio = (struct folio *)page; + + folio->_folio_order = order; #ifdef CONFIG_64BIT - page[1].compound_nr = 1U << order; + folio->_folio_nr_pages = 1U << order; #endif } From patchwork Wed Jan 11 14:29:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13096765 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 96830C5479D for ; Wed, 11 Jan 2023 14:29:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D9E40900009; Wed, 11 Jan 2023 09:29:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A41C7900007; Wed, 11 Jan 2023 09:29:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1FF3790000D; Wed, 11 Jan 2023 09:29:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 0B3F58E0001 for ; Wed, 11 Jan 2023 09:29:10 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id CB474C01CA for ; Wed, 11 Jan 2023 14:29:09 +0000 (UTC) X-FDA: 80342750418.27.161E4AE Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf17.hostedemail.com (Postfix) with ESMTP id 4A9F040016 for ; Wed, 11 Jan 2023 14:29:07 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Zscb4R+1; spf=none (imf17.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673447348; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xm1Fxzrz0ns4lKtzFJtwY/4ITuGXi0DU/+c1DUN/2L0=; b=EiEEPHDXWWibj7DTeKnhWTSUPeQ0jEKlDOV3M8HUtzuvKD0nQFP7Lh8EU6+huc9oEk1Xrl +RJUhrag+OTUD7coqaVJpocQCukuGXQbMTGkOld1gydYURkHsde2iPrqZwndYBp3oqbkJL ZE07bW7AIzVxONvrm7iehIyFUTq59p0= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Zscb4R+1; spf=none (imf17.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673447348; a=rsa-sha256; cv=none; b=ttGrZLFkC/2zHtQ3vkOnXBBt/rdfZxgrwoSLMbXBO8pBPaf7N86ayv7oMJe1eoMaMUZ1aV qexTmMEPHcxgjAhK4R6+vLQ/WTMUZb74tj8W2DwkIJy6oRcbnttixQuU9g1wtxqCHaygAb hqp8vlUhkroKqIa9WuxMAJuKqJZp7Y8= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=xm1Fxzrz0ns4lKtzFJtwY/4ITuGXi0DU/+c1DUN/2L0=; b=Zscb4R+10NwuCaikA+xO49j0lF tHPtBZT3rywnSzAw5kLesDkA/5789V7rnlir/+gDSLv7tYC6r8rBLmIjPfLjisXrupUpqmknEGRFi 1yd1ZTSwRoCHanjtGjMr6Q+kLw1o37NNFA3e3jL5qSqw12FHmfAPJj3kwOTxieq8e4zP7lU7xy9Kj Vq9NGk+AU7BaTvR8Hay1YBJ+GCQHJ/1DnbrxF7HaMg1v6NN0leYCftcvdc5B3IELkzBh5KSHEzbCE wN3GIT5xhvHQAq6Kb3uDwsFXHnxy3bnHq9SlDcH4Hdwcvjk1AvLBO97sEsRuGhMaqQ6oOJoZOZqhB xpLQPrQA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pFc6R-004CZo-5t; Wed, 11 Jan 2023 14:29:19 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Hugh Dickins Subject: [PATCH 19/28] mm: Convert is_transparent_hugepage() to use a folio Date: Wed, 11 Jan 2023 14:29:05 +0000 Message-Id: <20230111142915.1001531-20-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230111142915.1001531-1-willy@infradead.org> References: <20230111142915.1001531-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 4A9F040016 X-Stat-Signature: giyc9t3yeus7rfi9nac8ysrham5o5d1b X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1673447347-292382 X-HE-Meta: U2FsdGVkX1+qWenAURqn04pRee5UK0kh1E8Fa9TtqsDppPJqPSn1gXXlMoShgv4FUdHKzdN0u/R6oeA8cp/3jeOZUYvoNDTaBY3jTIwDpRE0z7xadO4BxrflEZzp7t2mSPxOKT375c1bTQiHBfePwhTLjy/j9MrsFR3HvE7dYMBo/9w3GiXMGQcaaXKoFyslUMODxFMuexQgLY4S1moDS425Mwt+KKKR4Yz4Wm+YNK1Y9JMhyu0M3QD4jvoXjBI+02tyD/9vBsdNake1WJ6WZKfRdgleaxzyEU/gAmUN/3CCXuHu9kVyqvZqr44uplSwzvNcxwVqPD0QLRtGwEbC6FRnl/RSZmrcZBX783P6kgew+/KULxCrEI3HC30U/O6OdgDwkcp9itmU3Z9Zg6585z9E9HEgy/8fUFfJPHsxGwpv4+9wFz+BpeOpZLcQFX0xcGMPR5Fs7a8gnuMuvwjuqvSWdiYGP/v2W3oBWOogd+LfnpRgQQuGEts4NexSQAafsPx+RthkFcmK2twBt00N4fXiEV3WzcRLJz3Z6VaHxJhBgDoZ6xCHeBdOs55c9Xwbc2Bx4rg6IiHmVlIPE/EtW1QEtSM1iV0CisRyqJ2xScs3ba6SMWdE31nQv70mGZ8ocDXlINfhTrruY9DFLMXHFZaK6BSPR0IWNoEsl2+MwBGy01mxaLxoWywHItpMSTqhmL5qQ4m8TWBcy+quh2TKBfYaPBLdbQKNqJUqrwcMH35KYAaDl/07Uy4LSBCoE/8iZplKOT3Z9ELfog7L8TK8CAsDLgq1OQSGU+LW9sBZxh/Qqo4l2dTFGE4gpjnETii4U9nNCgrisSwi18ayAutz8rwyCadBRtWcgnpiv0jQ9/dWXa91HfXk8SkoH5PqOf6ATmS4mRXoUyRmIZLcBG3bI2yjlw0WegKbnMTQ2Q8tGYsV8mRNERs2uQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Replace a use of page->compound_dtor with its folio equivalent. Signed-off-by: Matthew Wilcox (Oracle) --- mm/huge_memory.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 9570f03cdee4..bfa960f012fa 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -591,12 +591,14 @@ void prep_transhuge_page(struct page *page) static inline bool is_transparent_hugepage(struct page *page) { + struct folio *folio; + if (!PageCompound(page)) return false; - page = compound_head(page); - return is_huge_zero_page(page) || - page[1].compound_dtor == TRANSHUGE_PAGE_DTOR; + folio = page_folio(page); + return is_huge_zero_page(&folio->page) || + folio->_folio_dtor == TRANSHUGE_PAGE_DTOR; } static unsigned long __thp_get_unmapped_area(struct file *filp, From patchwork Wed Jan 11 14:29:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13096775 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4AA89C46467 for ; Wed, 11 Jan 2023 14:29:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C64A3900003; Wed, 11 Jan 2023 09:29:13 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 97444940007; Wed, 11 Jan 2023 09:29:13 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CE2788E0013; Wed, 11 Jan 2023 09:29:12 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 74FD18E0010 for ; Wed, 11 Jan 2023 09:29:11 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 5464D1A0B7A for ; Wed, 11 Jan 2023 14:29:11 +0000 (UTC) X-FDA: 80342750502.12.732D854 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf29.hostedemail.com (Postfix) with ESMTP id 8A0F0120015 for ; Wed, 11 Jan 2023 14:29:09 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="nZ/Nk5L9"; spf=none (imf29.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673447349; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xal2brZlLk3XQXt3Yb5MjnF1PPmoWq6AZ7/YaAtTH/o=; b=eQRORgJT4Sv5s83tiCvlcM6X2vqB8U+Bx/PrAC0z+YTkqYw0BZAP91vbaCwThGu5O9o9GS lGbmZoRn1r6/6dOzjcZlCQIK3J+lMZ60lEAIv4wmv+TEw017OCybN04td+H4C5LCfIt6bx 8rSV1ZwhqdwLBUXQxXBLB0pO0h5ELbQ= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="nZ/Nk5L9"; spf=none (imf29.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673447349; a=rsa-sha256; cv=none; b=Efw3+JVdptn69kPVaLQExy31NV/I9+u5mNvqFm3S+czBNfG4Gum1ksunZ2yY7q+0iU4Rsh dXMvTTKzAnjsdCn73w4y0gY7MbSzr/AR/tovO74I50FeTkgeQ5aEGilkEp2gfSmyEcO342 C5gz8q8qqqaAVpV46NvBNMVkw388SmE= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=xal2brZlLk3XQXt3Yb5MjnF1PPmoWq6AZ7/YaAtTH/o=; b=nZ/Nk5L9+a0OsKzV/Vn1BC1KAA FrPMezbuQKJfjAGcMIF5tkOc8f98mUYoSEccTNiPkOvPBYR4Wj3XC08ymx3a7B0fmjJfsaBoTd9zK TzhxUPwufWSpkDkAT/rOafdoK0LehF6OxDziu3q74Zc/WJBHXqI2Kj/8z2+7hnMxf3TFbKdcqcY7O KwOJ1AtJFdgNatyehb2chDnNEHFMjFJkjHqWVOOdVB78LKPfBTpLDofUqaWa2CCDK/CjO27X0x5zA ZkMqqn3zOEExvBqQNdBDiwquUhAklB+GGeqrcuDDnIRuicu5dXo1QMOwoKheCOmtWZVm4vwlklRBr JtyI3J6Q==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pFc6R-004CZy-AV; Wed, 11 Jan 2023 14:29:19 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Hugh Dickins Subject: [PATCH 20/28] mm: Convert destroy_large_folio() to use folio_dtor Date: Wed, 11 Jan 2023 14:29:06 +0000 Message-Id: <20230111142915.1001531-21-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230111142915.1001531-1-willy@infradead.org> References: <20230111142915.1001531-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 8A0F0120015 X-Stat-Signature: 8nnh835rqdhu44yzq7kpq6qdqr5878ym X-HE-Tag: 1673447349-731262 X-HE-Meta: U2FsdGVkX19jw7KRgBNcB5N9m7Er3AbNN24tv19LlUml6nqnwc6FWFlw4FCqdh9hlFgl4Mw3lW8fsD/NQvC5hfnhn4poZoUzWEYfr3Mhv/yB16o8btiS5vsUdpkkA7ewsEYXDzGNnLT47NcE6SUgIsPjI9XxLq6MV9eIW++llqQSrWWDq3F+CVWDW8i6zyCQmvcdt+xbpbTcgJkPAOYAFzznlOm5F1+vJIM7K8I4x9BVXP9z+HwD1AYXgCBtrqLJlooowznNdTvqQcgbKQ3yYVxCAL5yMzg+rdGFz+vHhQZ268aJWFlTx2A2gPzYDOkDPppETKlx/vgUWjz7ssbQTHDqBTNzwEyrt6qm4X7C1RqfWsPFMUJdsB4sgsXPg4C8uE59hKXIiIA0L02I1CyvMxhG49hvMq3stcxnmmOraU2lx2sfFQBX6yVkiJ8aW0ZO3iGLaJzIGj7wSzzrAHLdcuPd/Tduar6a8HNYSOUkYAIBVvBwXIGEMZ58eFE0QqmxZuJDNKzrjP/lVib6yFYMrViWw6vk3v9H3tau9gnA05OXtb/ueAj+pYIqjmuuHas7ZKYJ/Md0ylCy87XQ0rQsyKoyQyH8cFS86cds7CLeP2uFePdPel8okgLwRxu+qTjZ5zy4G7JkgCGNHr4XtSQxJhRSr9NDIVsx/nHg+JlLIalMVDyD2g/eI8jZxuDE5/E+pVttlphTa3baIsIPeKojhuDHeXDs0Zc4U6p5WQAXMtG+4fNs1anrBhb5xOmzycydgQjy5BsoLa7inIQiDqNOl7FJ7s3Jpfwg0175dOYIEZIiBGKBD2S+kR+5LrXORCcKqQ17ATYz64xWt+Gpo823WBqpM39lQ/aVbXFdmK7w1evOCI1yBxgxdpGarGaOW4jHMmC2uw39Fr7kpiwCWyqExYx9QtkSY9acEG/o4blehs/1bA4/rKsIEQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Replace a use of compound_dtor. Signed-off-by: Matthew Wilcox (Oracle) --- mm/page_alloc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 946a4ab7b278..41a239ce4692 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -807,7 +807,7 @@ void prep_compound_page(struct page *page, unsigned int order) void destroy_large_folio(struct folio *folio) { - enum compound_dtor_id dtor = folio_page(folio, 1)->compound_dtor; + enum compound_dtor_id dtor = folio->_folio_dtor; VM_BUG_ON_FOLIO(dtor >= NR_COMPOUND_DTORS, folio); compound_page_dtors[dtor](&folio->page); From patchwork Wed Jan 11 14:29:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13096770 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 45241C46467 for ; Wed, 11 Jan 2023 14:29:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EFA9690000A; Wed, 11 Jan 2023 09:29:12 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AFFBD94000A; Wed, 11 Jan 2023 09:29:12 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2BFF8900005; Wed, 11 Jan 2023 09:29:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 02D51900008 for ; Wed, 11 Jan 2023 09:29:11 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id DB981A01A6 for ; Wed, 11 Jan 2023 14:29:10 +0000 (UTC) X-FDA: 80342750460.18.6D459A0 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf08.hostedemail.com (Postfix) with ESMTP id 396D3160008 for ; Wed, 11 Jan 2023 14:29:08 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=WuFCtoL7; dmarc=none; spf=none (imf08.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673447349; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=fiJqAGhssq/2TqYGILm9QDcQEpK39OAP0z5tPzA/mtg=; b=fCe4Twdiv9NB8kkiDRwkUJo3XftGxI0MdpBu4J0+vrYGfaEiX35wzkXCwIPhIatHgmuA4w 8V5tTZJQFy07J6pVkUKrVdyW5/J09+yM6UioS7/1mjOVLZpDXSKBy5rhzmGVujgsxmbAjL bb+GnfnrGMtw5crg0BrUXsmA5iXSLgE= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=WuFCtoL7; dmarc=none; spf=none (imf08.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673447349; a=rsa-sha256; cv=none; b=1Ac7qcD7GCBZevTY8WOf+H/ib1SPn2Sng0RuL1uNpZyLnz99VOQaXf5N9Pa2QY8jPy7VIf QeyCWX1dBmR//va6cl0GcRy8YsY6xFUDvZFDCfdSpOc+n820mwE0D9t+rwgwnAIaIeI5fy l1hY6iwPCkdt9DsIpPtQumilsZivNBs= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=fiJqAGhssq/2TqYGILm9QDcQEpK39OAP0z5tPzA/mtg=; b=WuFCtoL7YEeTBn8iVX8HRjtGlr RlbLN4GD2f8Bc7XDlFWDfaSH8BUHHvEFAcl5mE1opI0yR2HYGyRR3tWwpYjBF1/7fxCkssrkzIaN/ TI/U0YLd8RC+CnhKcdN+ZZkfsmbR98bO4cp/gjLrNHaLX6gg7zNEL2KJVcK1dGHjAdmukJzgoklM4 J5Q3za9MICIwbVQpM2AVdL6BzwCc0u6ZfdLfgNrZeGdWnfvmmr3DzGIf1h+I6C1gJYWRDLIaBdyKw vvpmlB36hX78MQKT1Rjjj1kw7mS4cmNU6A+rH5pA3NRMDQwJ30oefGYAKWyglQ7JOryu0BaUNRd4p ZEUvZ1rA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pFc6R-004Ca9-Fm; Wed, 11 Jan 2023 14:29:19 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Hugh Dickins Subject: [PATCH 21/28] hugetlb: Remove uses of compound_dtor and compound_nr Date: Wed, 11 Jan 2023 14:29:07 +0000 Message-Id: <20230111142915.1001531-22-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230111142915.1001531-1-willy@infradead.org> References: <20230111142915.1001531-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 396D3160008 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: xn6w7eserxnmxcdj5h7nosabysbz9np9 X-HE-Tag: 1673447348-412157 X-HE-Meta: U2FsdGVkX1/APm7pkvdMVv48Fmfb2ITQ9S2Qyw3V5Flb13eboiRDIIWXVr55D6/LKLnkJHw7oGvTJ4ICX7d2vArmf/iqZlZRuDrNM0qysGoK7neQ+HRDvEPCC/ZmtTJm23zNNul4KYXhXlijfz0lLeAhfEsTW2jnqjy3hNGADnL95vS2Psuc5WRNrtFo99t2hbV24tp3ZyC7P7QJLzP95Lq0wRpzIkjdEv67eU3GNHcaJr+dEKyG509c4mL5ydyzE+PjYZ5NYw3gkzYgq+yVeJNwvDHq6PgE0zkxMLmmTnNYohpAfQiTlUtgms4VzQLatJwfzXiqo4Va87Ru70dCKeaFc3PxQn4qG0XrmoK984HB7lrwvqU401J92s6qkYMPBXn/Eeh5VEf7dviHhIevcnnQqFQeowtd1BvlMmJ8rlHLLI0CpBetS8VXoTzVUAolAckRuq4aXH16IIFPegOvqsqDFmecuzjFwM8Q139SRFuDv9mjnSrPZG17wl0wGERrSFdOFUjqJW2fqbC0CQmZzcbL0znlM9nM9D9U5/2h5LxxZYzpRv9lmDMaqEX9kwaymPKeXLuLYBL03JKEN2Y6Nll2/cIvw6bY7MY4YiF0PokTbmY9ciSy+9tA8NQVOw9zZUj9+BXM2IaNqZEtMPprEomUiTlh7BPs2T9iWGtCctohJfzV4HdrR8+dtwFu5fLy7uSCIcxz8cr2ekHXtwYFcYPqOtooDxM/44+d92wm6HkYiR33Weh7+3YppdqMTIOD7pbB4qp/MJrJimUZ/FaB4IxZT7jwoZMTQ/T7+/zTdZotPVZuYli7/Ro7wdbAPSspS3t8d2VdSUNtKZ61E2P4miAMFgAw424qLGrF1Gv5ZsrlDGR3M98JAMGYMPdlQ82f5MoHiJnFwMqrhOBrXf2WXx0axr5ARpVmk7dlDfy/anIq+pXOAhXtYA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert the entire file to use the folio equivalents. Signed-off-by: Matthew Wilcox (Oracle) --- mm/hugetlb.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index a68e0e597a8f..ca9e177b9c54 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -2038,11 +2038,12 @@ static bool prep_compound_gigantic_folio_for_demote(struct folio *folio, */ int PageHuge(struct page *page) { + struct folio *folio; + if (!PageCompound(page)) return 0; - - page = compound_head(page); - return page[1].compound_dtor == HUGETLB_PAGE_DTOR; + folio = page_folio(page); + return folio->_folio_dtor == HUGETLB_PAGE_DTOR; } EXPORT_SYMBOL_GPL(PageHuge); @@ -2052,10 +2053,11 @@ EXPORT_SYMBOL_GPL(PageHuge); */ int PageHeadHuge(struct page *page_head) { - if (!PageHead(page_head)) + struct folio *folio = (struct folio *)page_head; + if (!folio_test_large(folio)) return 0; - return page_head[1].compound_dtor == HUGETLB_PAGE_DTOR; + return folio->_folio_dtor == HUGETLB_PAGE_DTOR; } EXPORT_SYMBOL_GPL(PageHeadHuge); From patchwork Wed Jan 11 14:29:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13096767 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DD770C46467 for ; Wed, 11 Jan 2023 14:29:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5CB6C8E000D; Wed, 11 Jan 2023 09:29:12 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 221F2940008; Wed, 11 Jan 2023 09:29:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CFBA8900005; Wed, 11 Jan 2023 09:29:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id D1F098E000A for ; Wed, 11 Jan 2023 09:29:10 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id AD404403E2 for ; Wed, 11 Jan 2023 14:29:10 +0000 (UTC) X-FDA: 80342750460.20.1019882 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf14.hostedemail.com (Postfix) with ESMTP id 2A8D8100015 for ; Wed, 11 Jan 2023 14:29:08 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=hH0LJ5Py; spf=none (imf14.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673447349; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wNDMpdoIQrLth2wk5ULYVRwIM9k41K1noQJYjxxhx9o=; b=8K3eWM8RJj1zhSCxbnvNqrUGmyeAfMlZFaGq6baaaH04y4oxNSvPwK9n6BklCha64kYxwK ge72kfl4jFfALszkqw+8PDIwxTPqpUvbJtHPe/cH6/CngImel/TeA4+KA3SveSqCWVmUZ7 CM9iz1hqYDFNR12LuVjFDpBwhutQWok= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=hH0LJ5Py; spf=none (imf14.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673447349; a=rsa-sha256; cv=none; b=gMP1BgM3zIFlxs2O0AZUuPrI4j/iC+g8NPXmzVUeX2mUWIp9ROWq3gq/I8rhMHb7HCRQxE NYrn08hVq0SRYtFHfsHC9SAxVc+yF873uO3TgGiKzEKYevEoBxS9YoxShTWlDqXfcRi5fP y71nf4tiPvFIenczQx3Se6o79YPEybs= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=wNDMpdoIQrLth2wk5ULYVRwIM9k41K1noQJYjxxhx9o=; b=hH0LJ5PyTptTHy3HwGc0f9LWFw efvrPlcm7ntOpuyw1x54UUwT2A2nwvTY2lVA2wE9Wz1Sz0TlQOo7L9YOK84Isv9RhPsCKV9mQFKSC hyBJg9IcDcLoT6+VCmdtwn3ORamrYAJqCHua/Ry6aMRRG3ciaRnbmGd303ycMOqSaygbN7qPiEOob 4tyHApgGWc4fQfQfQZt8k6UoC1gTDB2MenRpp2O0VFbYsWukCKPygDmc7O4PadI+whS1veRy3y622 snhGmfu5ecirFhMevFobmpPMcjkfV+h8A85IIrxHYRhzI+DPjy/szMT2tGCz6Rry5DXQLzptVJHZd fDQTInTw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pFc6R-004CaL-Lx; Wed, 11 Jan 2023 14:29:19 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Hugh Dickins Subject: [PATCH 22/28] mm: Remove 'First tail page' members from struct page Date: Wed, 11 Jan 2023 14:29:08 +0000 Message-Id: <20230111142915.1001531-23-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230111142915.1001531-1-willy@infradead.org> References: <20230111142915.1001531-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 2A8D8100015 X-Rspam-User: X-Stat-Signature: fxs6hitucaq497ctqdyuhgusgozdui3i X-HE-Tag: 1673447348-633792 X-HE-Meta: U2FsdGVkX188oXg34Ca7/AKNad4QFAsq2iiZOjz8Yx4TEbx2hbmWEkKBsgG0ogRENJWTcl7p7sEsrnsMc8RWhX5WbyZk0R4hE3XAsX1Kxl4G+N9LZOLQlSOlOnNQiFncQtgBCbKJXsRU/mOPXCoLeVdq4OEqHtRu5M/EHdDCgadP8Ff4jaoW3q6bJIeU8a9WdsFF4Y5kyUMxk44ambZGkMDUyx1+ytPkdhdehcJKU63q6ve705qYchpfYIXG3Nj/CuMyZqyULLO97W94gBAVyTCGn2+Nod/ZR9z88/RHZPlguOM+L8EY+nUAFgIG5uvUFNrybIDL8NSxnZJ+2SiNP8xLqP7oNJvxvaHl1Zz585scKpjkbm+9wplHZGgYTULLPSIJNbXL++s3b29b6R8eZ1VesKsLhI22aC+4TdtEVqiOUc1SRlycbtkpH2MFaVn38dcOfE0UOy3dw7xzBd9sTH+MuzkcS9uvTB4+vdYU2SY1f2S8kTt1Qmsnci4DFGblSj/svRFSBwLVwMTEJWgPIS2ekjO6aCEmnpmWVxWI1FwNngVb+LVwxfDrFO8qav4sdF9pm4MZOTrafCZ/GO6LmG+QkFD/oQNilAkFySOJJ4aOPA2IUT+IUFO5qeuqqvQGMOAZ0XQ5NP5H1vplVbHEFFqojUa3JAJSwBZnZshwBS1ZSN7qCzyBzj0zakHt9vBZExNbOQv0T+aD9O5Os0JdTVhJaUzaOnji9vQF4N1rpowb8J/T+vF21InjiV7Uf4JluGm3tyNWx0FPTdAe1b5/a8Q0eYINTWbueBKlzveViBw0bj32Tpx4SvGIAxfFssm/sjCErxn12OJcCuBi4tY0LtGjEqRgu1mPSDWEX7tz+RNSAKUM0LMhUB6/D9gTCZde7spF3wgMr0E= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: All former users now use the folio equivalents, so remove them from the definition of struct page. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/mm_types.h | 18 ------------------ kernel/crash_core.c | 4 ++-- 2 files changed, 2 insertions(+), 20 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 1cf0fcd99d49..61a2e6b32781 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -140,16 +140,6 @@ struct page { }; struct { /* Tail pages of compound page */ unsigned long compound_head; /* Bit zero is set */ - - /* First tail page only */ - unsigned char compound_dtor; - unsigned char compound_order; - atomic_t compound_mapcount; - atomic_t subpages_mapcount; - atomic_t compound_pincount; -#ifdef CONFIG_64BIT - unsigned int compound_nr; /* 1 << compound_order */ -#endif }; struct { /* Second tail page of transparent huge page */ unsigned long _compound_pad_1; /* compound_head */ @@ -401,14 +391,6 @@ FOLIO_MATCH(memcg_data, memcg_data); offsetof(struct page, pg) + sizeof(struct page)) FOLIO_MATCH(flags, _flags_1); FOLIO_MATCH(compound_head, _head_1); -FOLIO_MATCH(compound_dtor, _folio_dtor); -FOLIO_MATCH(compound_order, _folio_order); -FOLIO_MATCH(compound_mapcount, _entire_mapcount); -FOLIO_MATCH(subpages_mapcount, _nr_pages_mapped); -FOLIO_MATCH(compound_pincount, _pincount); -#ifdef CONFIG_64BIT -FOLIO_MATCH(compound_nr, _folio_nr_pages); -#endif #undef FOLIO_MATCH #define FOLIO_MATCH(pg, fl) \ static_assert(offsetof(struct folio, fl) == \ diff --git a/kernel/crash_core.c b/kernel/crash_core.c index 87ef6096823f..755f5f08ab38 100644 --- a/kernel/crash_core.c +++ b/kernel/crash_core.c @@ -455,8 +455,8 @@ static int __init crash_save_vmcoreinfo_init(void) VMCOREINFO_OFFSET(page, lru); VMCOREINFO_OFFSET(page, _mapcount); VMCOREINFO_OFFSET(page, private); - VMCOREINFO_OFFSET(page, compound_dtor); - VMCOREINFO_OFFSET(page, compound_order); + VMCOREINFO_OFFSET(folio, _folio_dtor); + VMCOREINFO_OFFSET(folio, _folio_order); VMCOREINFO_OFFSET(page, compound_head); VMCOREINFO_OFFSET(pglist_data, node_zones); VMCOREINFO_OFFSET(pglist_data, nr_zones); From patchwork Wed Jan 11 14:29:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13096769 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B0FFEC5479D for ; Wed, 11 Jan 2023 14:29:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C0FF88E0012; Wed, 11 Jan 2023 09:29:12 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 90208940007; Wed, 11 Jan 2023 09:29:12 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1D978940007; Wed, 11 Jan 2023 09:29:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 04B4D900009 for ; Wed, 11 Jan 2023 09:29:11 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id CBA1140C4D for ; Wed, 11 Jan 2023 14:29:10 +0000 (UTC) X-FDA: 80342750460.11.7D5BFBB Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf25.hostedemail.com (Postfix) with ESMTP id 2D73EA000D for ; Wed, 11 Jan 2023 14:29:08 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=acFVmWPY; dmarc=none; spf=none (imf25.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673447349; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=tHWFuBexE/gwqgDbwm1Ra1Jg75D0HvKvTbVZv2jbd88=; b=6chhG3c40B2UYW0LiI0zCsK/xoPlJ5V8p+9WdwHtEJTT2LNGGungNfieoQEJXq/WcRkwFK 9/ia1QYkewtOLomY3r87UUBh2kc7FdNlXJcvf0o5AhAHwKWaqmz9MXkurw/uThXVeXnVTu 2V761UUKTkUWSCz/KI6WXKWbf1aaULc= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=acFVmWPY; dmarc=none; spf=none (imf25.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673447349; a=rsa-sha256; cv=none; b=5dmsiUgbc+Eiv/Yg7A+nEeBe/eD7DuRfpmbdJ7U3UFzOEmZUfFt3+iNZ5HLAwY/3rYgmE5 d7bIPyVyZy/7QIc33e5FLPUDnF4cs+WTQwqHNgCOHMXiiKvvKH7z8xKUi0h6q+R30E2Jbv Qg+fk6kuPLWDHVIcNftA9zZ31R6MtcA= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=tHWFuBexE/gwqgDbwm1Ra1Jg75D0HvKvTbVZv2jbd88=; b=acFVmWPYkuLNp2v5Ek7++JHtB0 cmBrvnt6Xb1ZqWH5Jjyve4OHvmg95UBGuX3YGZTO3n96iBdDmscDc5/dYTWG1l1Yh4aBSEHMbwc5I YU+sTXYqDrUBAlBTRXuaJmOMuevNgX0AN+dQZ32v6xsnVy9Uh/D41LLX/AwtbZV14aYSlfyrcV4MR 7nMAOvv8+Ri9oDjA4MDxsg+6KKhmdSch22ndrQ/XZwlvsR6S3yAnbAc6r7o41G2soUxJVlzdgx34l In86icW9SH/EVdbmf+EXLdE+Xg2D53/cqPPb9qA7cpykcMEQiRQgQaXeCY9Q4Nnq2gXSdh7+eyH8q i1vIxkCw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pFc6R-004CaW-R2; Wed, 11 Jan 2023 14:29:19 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Hugh Dickins Subject: [PATCH 23/28] doc: Correct struct folio kernel-doc Date: Wed, 11 Jan 2023 14:29:09 +0000 Message-Id: <20230111142915.1001531-24-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230111142915.1001531-1-willy@infradead.org> References: <20230111142915.1001531-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 2D73EA000D X-Stat-Signature: n7qypysak77iq8pew1gtcifjxen83foy X-HE-Tag: 1673447348-907766 X-HE-Meta: U2FsdGVkX1+0hnTseuiamHrRqNj3NiF4SWw3WWx807+jKE2xVSrPRCVvYQ556YHrpqGYAII4rxaQgW9dMZB+9T618Rw+B00Z5/CsOLfzdtNcKd2Mr+ECC8rsFIVU0cyl3z8cCD5ouxzEAMd0UGh6DFxd4RjogdstF1l0dZAYKOXnwex+QXBmo9A8uzbIlm+w06UMuD9FKamxH3kxq+ZB0sEjQxebjOQaoDAVrqIkhoGSRsF3/g/SbHYWeU/5kIOJlyhZp9axmziRzMd+V0jIOWFuRgEyjgpURoMGxNNHOUgvfAZqhklSOtMEHSQcfHEF8SbPplxhA8PMAKqxo/iSBBwWSkoFuwCn/nrPhb2rmI4N7kkf7ig4RoQ1UzjNgTpytufbcAoRICXgLqiZlVKrSJDvMImOthwJZq/aiVPFLRDxaRoaMVAPF+IaP9mzDoze/CjZS+I+yHCp2knQNB+4eoEb/jUwztJPS/UtBKJVUVsyqiCljKPVRHZJGGHMVRRwJu+BCtO+AKia2F8PD77gS9N7vCkAchirhmRCgS3nFap2jOErJidMxcAL+yz+s8F5hbbO1mgtH3xqKmcZv0aefiMILQydO2fIAmle4lBVwJHbEOvmWIPXYTq1qhPEnxZ0jd8fnFlMkIrQFrxt+1U93bRoLzFcBGtVKm+w95jgjvwfTDQwmZHYVldL0BIMXImZxbmaj4zsz4VG/8vOxS8K3Ae4rMO7XwsqEEDS0I8dzfGqddkKTPWyp04ER/Rfb0j7vHLGiSvdTJLNi9PoNRtbfB1PRxJBh/fYVSFj6JOOEfteU0MfN0BeOxtvnqcLsH7cn9RXl+I7mZ0AQRBQQv3k9xvLGeFodYOXTpzqgi7T/Sr06zdfcsMyH2Lkp0zT9mXvCAmiD1lJ7pw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Insert appropriate public: and private: markers to make the generated kernel-doc look right. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/mm_types.h | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 61a2e6b32781..4b8aa0f8f9fe 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -292,16 +292,12 @@ static inline struct page *encoded_page_ptr(struct encoded_page *page) * @_refcount: Do not access this member directly. Use folio_ref_count() * to find how many references there are to this folio. * @memcg_data: Memory Control Group data. - * @_flags_1: For large folios, additional page flags. - * @_head_1: Points to the folio. Do not use. * @_folio_dtor: Which destructor to use for this folio. * @_folio_order: Do not use directly, call folio_order(). * @_entire_mapcount: Do not use directly, call folio_entire_mapcount(). * @_nr_pages_mapped: Do not use directly, call folio_mapcount(). * @_pincount: Do not use directly, call folio_maybe_dma_pinned(). * @_folio_nr_pages: Do not use directly, call folio_nr_pages(). - * @_flags_2: For alignment. Do not use. - * @_head_2: Points to the folio. Do not use. * @_hugetlb_subpool: Do not use directly, use accessor in hugetlb.h. * @_hugetlb_cgroup: Do not use directly, use accessor in hugetlb_cgroup.h. * @_hugetlb_cgroup_rsvd: Do not use directly, use accessor in hugetlb_cgroup.h. @@ -348,6 +344,7 @@ struct folio { struct { unsigned long _flags_1; unsigned long _head_1; + /* public: */ unsigned char _folio_dtor; unsigned char _folio_order; atomic_t _entire_mapcount; @@ -356,6 +353,7 @@ struct folio { #ifdef CONFIG_64BIT unsigned int _folio_nr_pages; #endif + /* private: the union with struct page is transitional */ }; struct page __page_1; }; @@ -363,10 +361,12 @@ struct folio { struct { unsigned long _flags_2; unsigned long _head_2; + /* public: */ void *_hugetlb_subpool; void *_hugetlb_cgroup; void *_hugetlb_cgroup_rsvd; void *_hugetlb_hwpoison; + /* private: the union with struct page is transitional */ }; struct page __page_2; }; From patchwork Wed Jan 11 14:29:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13096772 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D1961C46467 for ; Wed, 11 Jan 2023 14:29:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4A266900005; Wed, 11 Jan 2023 09:29:13 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1B38A940007; Wed, 11 Jan 2023 09:29:13 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 72FA6900008; Wed, 11 Jan 2023 09:29:12 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 0AA3190000A for ; Wed, 11 Jan 2023 09:29:11 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id C45EAAB1FD for ; Wed, 11 Jan 2023 14:29:10 +0000 (UTC) X-FDA: 80342750460.20.73FF444 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf03.hostedemail.com (Postfix) with ESMTP id 20A8920009 for ; Wed, 11 Jan 2023 14:29:08 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=aIh1l+UT; dmarc=none; spf=none (imf03.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673447349; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=DqzXVQK7tx6rDR0l+Ej6O1ZL2mvrlzVDMprUi1Asko0=; b=ScjflXJIxDJa585DtyNwIueiCSdS5KejaPoflGSV49tqKz+i5lTwEoE3YtqePIvWZb4U8v x7WEJULzbQ6G6uiHLKQl1e1oUU4WH8Q6tUeNVf9sWOz+k+18w7F5gV8itXYCSKe+jwCV+/ p+4hdzAMRW0MNTBUoSy3oB7RmNqYBlA= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=aIh1l+UT; dmarc=none; spf=none (imf03.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673447349; a=rsa-sha256; cv=none; b=Gbqyh4UBmG2lCzzw5E7wM20UvaQK9DuLycmsWajgLCgV1hZ0jtZ6KkTizNHXFTOD4XGfHp wCqlkSLHorsQi559/+M2g4EiZn4XW95Kep+0OzxIVI7i2ksrpuTkAq17XhdYFyI8f0TRrF QOwMAxwAbFWTxUI/d34E7tPtN+WSbEk= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=DqzXVQK7tx6rDR0l+Ej6O1ZL2mvrlzVDMprUi1Asko0=; b=aIh1l+UT570jexXJDDtA0ez1yV LzjDo/asQASuU7aYe5Tyw873f+YNxxKNE+k3agDCOAq5KKTb0v0+JjlyiUKOnVA1BISKCkeu5jPfB HHKvXB4l+4W8iOnIzOJUHpkmk3yncIwOLj16zSBOqqn/HW+NyHe23IGgjq0388Y3t/6FFCR7yNBU6 UowOq0gqY3rSKHF2LzpB1QA9JiYBEPxgSxbHuQovVOyau8i9yPZE8Mmq6MUK1YRafMAp9PCZzDLLp TDD81QA2kH73g28CgZscOy3AhlElb3/64sQUsyypg3GrzXBSfwNnrY/C7GxzLHsl6pVijKTjqXg67 ynsc7y6g==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pFc6R-004Caj-VO; Wed, 11 Jan 2023 14:29:20 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Hugh Dickins Subject: [PATCH 24/28] mm: Move page->deferred_list to folio->_deferred_list Date: Wed, 11 Jan 2023 14:29:10 +0000 Message-Id: <20230111142915.1001531-25-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230111142915.1001531-1-willy@infradead.org> References: <20230111142915.1001531-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 20A8920009 X-Stat-Signature: 7t5s54kqu3te8bwkwdox7yfkxm83jfw4 X-HE-Tag: 1673447348-669752 X-HE-Meta: U2FsdGVkX1+XUk3BZXR5jHk5+FD8D6LP4CbSziVJrkw6sq3yEFF5z4AY0FkIS8LNJebPxj8CPKUXkM23nl1RYk1WwLubX65pAvhD/sG6jB6dNSAYFWhg+pEaIqXN+nCUTCFkSUxF19lMYRjLirdh2L+khF6DK79MD+/ydN8p/8Qz0XFY/lj7kGN2AlUwL4JnqSIQCiTF6vy7Ye1D8YrKAREMNdCdNwqMv8DSEe65W7xIO1tJsfRAEsfSO8OozHQCltO2PJ0vdm9prab5ZR7gEdW/8Z2Zu7zVaLA07nLsZO28ol+GOucnXwB7ZCjiy17zdofO/ud5KCzlip5ZZphw1yxGZnC7KOomChTeKwNI2hEV11cLBInuB2RvmLwn7omTkDKAwgL4Mph2F5XRuAhJruVAwAm6GM9wqMCoa9tGpup6Fukz+hAJC/coOk/33Vn9PJ139j44RTWLvdXqH9vL1+MUWUlAnJYHsHidVp7SpHBKQwE4md1N6BirVcrtxZKRZJiFpn3xS/CuenADLDTKVlruIpqRDDk/QXx3t5+TmQWyD1EVIkWGV1dFfvZyOmzL+OXMnHvq3DA8xTkJl5qW8DiOiC39gUbrR+KNPCuET98FXm6TuqT/U/QdKJlFcQ06o7sJslMcWZxPtlw2D6qLBP7OiZeUfwF/ckuxUQCPeQqZT1ZPTaNZ9jfqHYMTWK8G3oSc8rSPZJZWs7VPjcFHXrrrLv9Ft1ftWo+qO9Hf6pnFUs84SlfOEM7PQCKIRfRiT5eem9+TKML5jo6QGQ1Oy5Ach0aFjAzfuMsEQ2LmVadkLX4dg49lKoD7hl1fqj4l9zYZ3Nz0VXPVYvnwM0nGgLQcFodX6Ypdn87O1G3t8N97jYNflj6bu1gLOOIZeZ5ikOw/wkT9vZPH121kGGUinuG/76cOGIP96tRSpa1AvrGQwEBezR6A3A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Remove the entire block of definitions for the second tail page, and add the deferred list to the struct folio. This actually moves _deferred_list to a different offset in struct folio because I don't see a need to include the padding. This lets us use list_for_each_entry_safe() in deferred_split_scan() and avoid a number of calls to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/huge_mm.h | 9 ++++----- include/linux/mm_types.h | 14 ++++++++------ mm/huge_memory.c | 32 +++++++++++++++----------------- 3 files changed, 27 insertions(+), 28 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index a1341fdcf666..aacfcb02606f 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -295,11 +295,10 @@ static inline bool thp_migration_supported(void) static inline struct list_head *page_deferred_list(struct page *page) { - /* - * See organization of tail pages of compound page in - * "struct page" definition. - */ - return &page[2].deferred_list; + struct folio *folio = (struct folio *)page; + + VM_BUG_ON_FOLIO(folio_order(folio) < 2, folio); + return &folio->_deferred_list; } #else /* CONFIG_TRANSPARENT_HUGEPAGE */ diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 4b8aa0f8f9fe..c464205cf7ea 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -141,12 +141,6 @@ struct page { struct { /* Tail pages of compound page */ unsigned long compound_head; /* Bit zero is set */ }; - struct { /* Second tail page of transparent huge page */ - unsigned long _compound_pad_1; /* compound_head */ - unsigned long _compound_pad_2; - /* For both global and memcg */ - struct list_head deferred_list; - }; struct { /* Second tail page of hugetlb page */ unsigned long _hugetlb_pad_1; /* compound_head */ void *hugetlb_subpool; @@ -302,6 +296,7 @@ static inline struct page *encoded_page_ptr(struct encoded_page *page) * @_hugetlb_cgroup: Do not use directly, use accessor in hugetlb_cgroup.h. * @_hugetlb_cgroup_rsvd: Do not use directly, use accessor in hugetlb_cgroup.h. * @_hugetlb_hwpoison: Do not use directly, call raw_hwp_list_head(). + * @_deferred_list: Folios to be split under memory pressure. * * A folio is a physically, virtually and logically contiguous set * of bytes. It is a power-of-two in size, and it is aligned to that @@ -366,6 +361,13 @@ struct folio { void *_hugetlb_cgroup; void *_hugetlb_cgroup_rsvd; void *_hugetlb_hwpoison; + /* private: the union with struct page is transitional */ + }; + struct { + unsigned long _flags_2a; + unsigned long _head_2a; + /* public: */ + struct list_head _deferred_list; /* private: the union with struct page is transitional */ }; struct page __page_2; diff --git a/mm/huge_memory.c b/mm/huge_memory.c index bfa960f012fa..a4138daaa0b8 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2756,9 +2756,9 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) /* Prevent deferred_split_scan() touching ->_refcount */ spin_lock(&ds_queue->split_queue_lock); if (folio_ref_freeze(folio, 1 + extra_pins)) { - if (!list_empty(page_deferred_list(&folio->page))) { + if (!list_empty(&folio->_deferred_list)) { ds_queue->split_queue_len--; - list_del(page_deferred_list(&folio->page)); + list_del(&folio->_deferred_list); } spin_unlock(&ds_queue->split_queue_lock); if (mapping) { @@ -2873,8 +2873,8 @@ static unsigned long deferred_split_scan(struct shrinker *shrink, struct pglist_data *pgdata = NODE_DATA(sc->nid); struct deferred_split *ds_queue = &pgdata->deferred_split_queue; unsigned long flags; - LIST_HEAD(list), *pos, *next; - struct page *page; + LIST_HEAD(list); + struct folio *folio, *next; int split = 0; #ifdef CONFIG_MEMCG @@ -2884,14 +2884,13 @@ static unsigned long deferred_split_scan(struct shrinker *shrink, spin_lock_irqsave(&ds_queue->split_queue_lock, flags); /* Take pin on all head pages to avoid freeing them under us */ - list_for_each_safe(pos, next, &ds_queue->split_queue) { - page = list_entry((void *)pos, struct page, deferred_list); - page = compound_head(page); - if (get_page_unless_zero(page)) { - list_move(page_deferred_list(page), &list); + list_for_each_entry_safe(folio, next, &ds_queue->split_queue, + _deferred_list) { + if (folio_try_get(folio)) { + list_move(&folio->_deferred_list, &list); } else { - /* We lost race with put_compound_page() */ - list_del_init(page_deferred_list(page)); + /* We lost race with folio_put() */ + list_del_init(&folio->_deferred_list); ds_queue->split_queue_len--; } if (!--sc->nr_to_scan) @@ -2899,16 +2898,15 @@ static unsigned long deferred_split_scan(struct shrinker *shrink, } spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); - list_for_each_safe(pos, next, &list) { - page = list_entry((void *)pos, struct page, deferred_list); - if (!trylock_page(page)) + list_for_each_entry_safe(folio, next, &list, _deferred_list) { + if (!folio_trylock(folio)) goto next; /* split_huge_page() removes page from list on success */ - if (!split_huge_page(page)) + if (!split_folio(folio)) split++; - unlock_page(page); + folio_unlock(folio); next: - put_page(page); + folio_put(folio); } spin_lock_irqsave(&ds_queue->split_queue_lock, flags); From patchwork Wed Jan 11 14:29:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13096764 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 040A9C54EBC for ; Wed, 11 Jan 2023 14:29:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B5A0D8E000A; Wed, 11 Jan 2023 09:29:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 865E88E000D; Wed, 11 Jan 2023 09:29:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3599590000E; Wed, 11 Jan 2023 09:29:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id A3C458E000C for ; Wed, 11 Jan 2023 09:29:10 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 7E93BAB1FD for ; Wed, 11 Jan 2023 14:29:10 +0000 (UTC) X-FDA: 80342750460.19.485D430 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf26.hostedemail.com (Postfix) with ESMTP id EF053140010 for ; Wed, 11 Jan 2023 14:29:08 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=TGPHDiu2; spf=none (imf26.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673447349; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=I/SUW+Vky3ytrcAhb/+OOcmZgAYmQ4Sl7yFqNgJxV7k=; b=J7lQtTEQQ2KasCx0h1Ru8zIqGcIECPEGz0Uhr/EF25M+9+C1NUt6Ah+wMezIaiABTZ2ARB Sr0sv1OTM7pdCBndDpu2WvXM78dUsKnq/cZQ/FRlefVRAW2k932i2zhnMqXsCjqvQNScw3 NFsXetSdnrRreT3CYouMs+sDdLwf0dU= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=TGPHDiu2; spf=none (imf26.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673447349; a=rsa-sha256; cv=none; b=pVjthFI8vpqF0hnqXWDRk0zr6s/ou9F+1Gu+4h/It1eYMDuFllhOwmxpc87osjNGjN9wEK 01Rns5rLSLMLS2AfeD4rhI1Pe3vo9srr2jpiqCuNYYzdcDJQ4A6mK+W1tVKdBFaxPQZzLO LJSWYVilJyOV2+OBsAj75svGlWXzLiE= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=I/SUW+Vky3ytrcAhb/+OOcmZgAYmQ4Sl7yFqNgJxV7k=; b=TGPHDiu2RSYyqTWCW8qSbEYl3q drAwklNUvzPRIYcm0w/iXQOUSRnYlLEjJKojESLxhVY2viWaIv4EgqW5oTDSm1rr+1mDTOQeXf7bn AgejqPYyrQHwmFpNGjBO81TV7LiMqBWTTJ7LWOhkPiOT8uSpJ2wP94UKlDdd83hsThYSSUCsjfoHU AM4/mJEKAjXfVLvpQT9CV8ZNGb6JQcEHvvoaWq2Ooc0+glNb9m1+cOacK8mgn4rvLRWS+lZ+XWZ0o JbySQwIyoYgB38GGGs00P21FC/R5xtLuGaUF2HVNOZoZHV3bleGPPf33VVjovzQif6U4BDU5Bn4cR rkNhOwtg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pFc6S-004Car-3H; Wed, 11 Jan 2023 14:29:20 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Hugh Dickins Subject: [PATCH 25/28] mm/huge_memory: Remove page_deferred_list() Date: Wed, 11 Jan 2023 14:29:11 +0000 Message-Id: <20230111142915.1001531-26-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230111142915.1001531-1-willy@infradead.org> References: <20230111142915.1001531-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: EF053140010 X-Stat-Signature: iwgosw35g14fofhm6qo36d5pt9hake4g X-Rspam-User: X-HE-Tag: 1673447348-556507 X-HE-Meta: U2FsdGVkX1+pm5Jq45TyQNC5BaM4slhgAxqsFLE8ndoLIWCdxhLiQtltKRwnvyEGxaHgBrDrR01+gzAmBvJFFIUvl8DRn9RM0sRjGUiXI0vmezQvFgg1CZAXYa/wllSVk0xgckkVKNe2EMEf3G6rY6jk9sZO8zda372bRWhtePh0iNCSWGzZ3y410IDI3Togbx3oTAzpK6AAD0qJcxS0UFA6Hrd4HzDC9so9UPdm6wfUaLPnpqsFs5WVfd6cCnEKblIHdq0ymwcqQifvGcgF0Qa62fKFsti840hOKV/6TSSbqWNmKKIta6txVUx2P22aYBDFrCCInviMLCbd0motm68C7H4A9GRGLLYUzXo5O4zt/JZuni+Dci+LPobQ8kbr8t5d3CKPgMLO87BRUV/92nzeT1yWjqTzbc/M3cLFTZHbMZ4lswTHkwlOdxAeaCVl8WDUvij2ac8lZIFWdVUZOLcCaIBMfGFmTlvJ2lkHkptVtlHwMRgSHZO2eVqm4TZ75r5R1HzHUkXfzXpN2AaiKGEDZteTLDPMe8ly1K8IURvXElDqWSiCR5h8O1JwA6i45nfnCQtbkdTaBb7San+UBt2v7kxiBFhXNYkXDbHPTpUtS+aKXcKwNmByr3lFnT1ockQmpM5PQR1QB609FrLbA8rFTaF6KBo/25fPjJ1U4qZ6p6BYiPmRn0vdTFdTrbcwAp2VG1xUMeJ+ozlP7BbLrttbJpcDu3p0nBg5xdZurQEKRq1t06q0K0LWTQQrI3NEb02J1akcs2YxT4AeL+C+mq3ORLndHxHhllkyp6jdjhbRXVQUyX+A/BCbLT+8bmf+KpHN1V60rgElAXQtxrhQFsT1Y73hC59bEQhBvMYgtf9agwoK6BYMnQzvAczOUjhzFa/Hjf4FGogOwW/eX9zMn2pbfwTRd/JlFQFUHtCRwO7cwFqJXdpLrg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use folio->_deferred_list directly. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/huge_mm.h | 8 -------- mm/huge_memory.c | 34 +++++++++++++++++----------------- 2 files changed, 17 insertions(+), 25 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index aacfcb02606f..b9978978a160 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -293,14 +293,6 @@ static inline bool thp_migration_supported(void) return IS_ENABLED(CONFIG_ARCH_ENABLE_THP_MIGRATION); } -static inline struct list_head *page_deferred_list(struct page *page) -{ - struct folio *folio = (struct folio *)page; - - VM_BUG_ON_FOLIO(folio_order(folio) < 2, folio); - return &folio->_deferred_list; -} - #else /* CONFIG_TRANSPARENT_HUGEPAGE */ #define HPAGE_PMD_SHIFT ({ BUILD_BUG(); 0; }) #define HPAGE_PMD_MASK ({ BUILD_BUG(); 0; }) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index a4138daaa0b8..7aedfe7cf5df 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -580,12 +580,10 @@ static inline struct deferred_split *get_deferred_split_queue(struct page *page) void prep_transhuge_page(struct page *page) { - /* - * we use page->mapping and page->index in second tail page - * as list_head: assuming THP order >= 2 - */ + struct folio *folio = (struct folio *)page; - INIT_LIST_HEAD(page_deferred_list(page)); + VM_BUG_ON_FOLIO(folio_order(folio) < 2, folio); + INIT_LIST_HEAD(&folio->_deferred_list); set_compound_page_dtor(page, TRANSHUGE_PAGE_DTOR); } @@ -2802,13 +2800,14 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) void free_transhuge_page(struct page *page) { + struct folio *folio = (struct folio *)page; struct deferred_split *ds_queue = get_deferred_split_queue(page); unsigned long flags; spin_lock_irqsave(&ds_queue->split_queue_lock, flags); - if (!list_empty(page_deferred_list(page))) { + if (!list_empty(&folio->_deferred_list)) { ds_queue->split_queue_len--; - list_del(page_deferred_list(page)); + list_del(&folio->_deferred_list); } spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); free_compound_page(page); @@ -2816,38 +2815,39 @@ void free_transhuge_page(struct page *page) void deferred_split_huge_page(struct page *page) { + struct folio *folio = page_folio(page); struct deferred_split *ds_queue = get_deferred_split_queue(page); #ifdef CONFIG_MEMCG - struct mem_cgroup *memcg = page_memcg(compound_head(page)); + struct mem_cgroup *memcg = folio_memcg(folio); #endif unsigned long flags; - VM_BUG_ON_PAGE(!PageTransHuge(page), page); + VM_BUG_ON_FOLIO(folio_order(folio) < 2, folio); /* * The try_to_unmap() in page reclaim path might reach here too, * this may cause a race condition to corrupt deferred split queue. - * And, if page reclaim is already handling the same page, it is + * And, if page reclaim is already handling the same folio, it is * unnecessary to handle it again in shrinker. * - * Check PageSwapCache to determine if the page is being - * handled by page reclaim since THP swap would add the page into + * Check the swapcache flag to determine if the folio is being + * handled by page reclaim since THP swap would add the folio into * swap cache before calling try_to_unmap(). */ - if (PageSwapCache(page)) + if (folio_test_swapcache(folio)) return; - if (!list_empty(page_deferred_list(page))) + if (!list_empty(&folio->_deferred_list)) return; spin_lock_irqsave(&ds_queue->split_queue_lock, flags); - if (list_empty(page_deferred_list(page))) { + if (list_empty(&folio->_deferred_list)) { count_vm_event(THP_DEFERRED_SPLIT_PAGE); - list_add_tail(page_deferred_list(page), &ds_queue->split_queue); + list_add_tail(&folio->_deferred_list, &ds_queue->split_queue); ds_queue->split_queue_len++; #ifdef CONFIG_MEMCG if (memcg) - set_shrinker_bit(memcg, page_to_nid(page), + set_shrinker_bit(memcg, folio_nid(folio), deferred_split_shrinker.id); #endif } From patchwork Wed Jan 11 14:29:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13096766 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 70C50C54EBC for ; Wed, 11 Jan 2023 14:29:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2E645900007; Wed, 11 Jan 2023 09:29:12 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B4B518E0001; Wed, 11 Jan 2023 09:29:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7A556900005; Wed, 11 Jan 2023 09:29:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id D47568E000D for ; Wed, 11 Jan 2023 09:29:10 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id A79AEAB2D6 for ; Wed, 11 Jan 2023 14:29:10 +0000 (UTC) X-FDA: 80342750460.23.6196F3F Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf22.hostedemail.com (Postfix) with ESMTP id 167F0C0015 for ; Wed, 11 Jan 2023 14:29:08 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=LICi0enL; spf=none (imf22.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673447349; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5k3fLVMrEcfOJt6mTA73s8CScy7I3GVBSY1UgeCs/nY=; b=D55WYPy2ZdmngZ+t6VVMAIeSwoj2dgTMHVUu+rDl+4fj8Ru15XK4hEjTkJAJPaa1FFV4iH iRENsbZ8hI67qnzV4IWJnVaM/HWX8lt3osQHI8y8sUVrZ8NmACrYyOXtQMloLjm24ORuPc dAfR/IMdxog+1fEjidUOVVNC+gRJDwA= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=LICi0enL; spf=none (imf22.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673447349; a=rsa-sha256; cv=none; b=mD9n3Kyd+iZ6eyLdxYY2iwEUHiMyNbamQ1qUUy/dKFM8zYhBzbGYAU8CGUeRZ8KeOj8Usl QQHNisy7l1j0XzbPXks9bv6OLwsQtJ/sjmy2jLaYBix27/xSTc5n1svLzjSHrXShIzPRbx lNmhhB+CfKDgdZwAp5I/M3S2DXbkODM= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=5k3fLVMrEcfOJt6mTA73s8CScy7I3GVBSY1UgeCs/nY=; b=LICi0enLFLtWvn+cjnYivQiNcQ x7RUB64ZXuQ6JNOJgzo2UJdiIwv9rUfzH6F9R+gZz57vmHhjgzUuDPffZo1xElXTGKlAtO5zCJwcy BcFxvYfx2d2ltaXwbDO8IW9gDuuxwdOQU4ot3/VlXf9hi1mg4pXMwZJFhAAsC+8IgIKrXCyxgk6a5 WyNrtmNTqbApHcvgaOgg2K1b+iYNETh426pPI+esEbChm+yy4VBVYMZjW3PE1sxo/PCpvdlZ1iMMI j6JrZ43tjitky5Iue5SfnZ0ISuivp4aEKEl2a52ijJ/0EYzbE/jssOYgmpmqSKPEGl7D//WMoSaez UVkDJ+sA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pFc6S-004Cb2-8y; Wed, 11 Jan 2023 14:29:20 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Hugh Dickins Subject: [PATCH 26/28] mm/huge_memory: Convert get_deferred_split_queue() to take a folio Date: Wed, 11 Jan 2023 14:29:12 +0000 Message-Id: <20230111142915.1001531-27-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230111142915.1001531-1-willy@infradead.org> References: <20230111142915.1001531-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 167F0C0015 X-Stat-Signature: rqzd441hokqy17pqu1ky84h7usmmcj1k X-HE-Tag: 1673447348-506580 X-HE-Meta: U2FsdGVkX18TmswNNcEOCetCX2E04kg320e1LpBl633bbAjJC5VAtslBTooTJOsj/Ksryl1W74qooNjUY3ssJr2LbRI28AGDPix9xgUIzxCSh6w32o/ZdgQ2ee3wxO8JSMsC7SjM6TIpDpcfh8Z3/0U2L9mvvOAorOdy80D8A4JMSDoXHW6z6GMBqbYi0zs8Kwl36vIvr+EsvSW42RRcbxXrpdDD4yq5qQG5DHXEetcJ0+7iBdN4v7pNp8wcP0pyp/muzeiyLbw6jpM7tfVkTBEjRYpRoyJQMJPuedAs2v0q7AEX00aVHVuR5ZufKGdqq5UuiYOP9Zv/LYBge29mEMytfIBsOms03YxmSNpbH+f6hk4omJbKi2zFOnuFc3XMwQB/g+9znNRUfNpJoEMdy14GjtnxV73r6aQeZKaUs2S/llEqetfgiQfIw1rg/EGpFkHlEp0wzcGhB9btznuWo0K35WmnrV2bH5mt9WUPMAZE3SI6AZ1LXJsYOzdWGp9K7G4H3JqOjoavHyzR3UH7ItRlJ+yrljCqV/cGKHIETLpaz5Dchm/BLjdaMJzaAsvFAK4B9WOcicuyxaGSj8e1zx75V1gTV2w37vYQzuQuk0zZXgMiXEsbJGS+0gslcYLMNER49DTD+3aMjefqRhAYH5ujgIfWIVkySvnFBrc04xQXBszel2xhTfbIPit5DG8Rlyfk2ZfE5q3g+RN5xitxkg4AsHUy6VAbSRMJ5veF0Qwoin73mg1Kg0wDQJUQUpSVhost0AJSTVoGHTqQEeby54T7CTbIBr5QONf1Fz0rRcRqm19D2HnjzG8NQ1eXZL9Cb+qu8VY+WBASLV9DYxZkW22OZv62ki47cEZzoSJeAlxfSM1brph55SMfKyzj9vqGr5JFN/VpswZuXy3dKZFUF3dVbOLxT1WwQyniq9svVzFEBU5O5qsrew== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Removes a few calls to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/huge_memory.c | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 7aedfe7cf5df..c23b0e01734b 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -559,10 +559,11 @@ pmd_t maybe_pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma) } #ifdef CONFIG_MEMCG -static inline struct deferred_split *get_deferred_split_queue(struct page *page) +static inline +struct deferred_split *get_deferred_split_queue(struct folio *folio) { - struct mem_cgroup *memcg = page_memcg(compound_head(page)); - struct pglist_data *pgdat = NODE_DATA(page_to_nid(page)); + struct mem_cgroup *memcg = folio_memcg(folio); + struct pglist_data *pgdat = NODE_DATA(folio_nid(folio)); if (memcg) return &memcg->deferred_split_queue; @@ -570,9 +571,10 @@ static inline struct deferred_split *get_deferred_split_queue(struct page *page) return &pgdat->deferred_split_queue; } #else -static inline struct deferred_split *get_deferred_split_queue(struct page *page) +static inline +struct deferred_split *get_deferred_split_queue(struct folio *folio) { - struct pglist_data *pgdat = NODE_DATA(page_to_nid(page)); + struct pglist_data *pgdat = NODE_DATA(folio_nid(folio)); return &pgdat->deferred_split_queue; } @@ -2650,7 +2652,7 @@ bool can_split_folio(struct folio *folio, int *pextra_pins) int split_huge_page_to_list(struct page *page, struct list_head *list) { struct folio *folio = page_folio(page); - struct deferred_split *ds_queue = get_deferred_split_queue(&folio->page); + struct deferred_split *ds_queue = get_deferred_split_queue(folio); XA_STATE(xas, &folio->mapping->i_pages, folio->index); struct anon_vma *anon_vma = NULL; struct address_space *mapping = NULL; @@ -2801,7 +2803,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) void free_transhuge_page(struct page *page) { struct folio *folio = (struct folio *)page; - struct deferred_split *ds_queue = get_deferred_split_queue(page); + struct deferred_split *ds_queue = get_deferred_split_queue(folio); unsigned long flags; spin_lock_irqsave(&ds_queue->split_queue_lock, flags); @@ -2816,7 +2818,7 @@ void free_transhuge_page(struct page *page) void deferred_split_huge_page(struct page *page) { struct folio *folio = page_folio(page); - struct deferred_split *ds_queue = get_deferred_split_queue(page); + struct deferred_split *ds_queue = get_deferred_split_queue(folio); #ifdef CONFIG_MEMCG struct mem_cgroup *memcg = folio_memcg(folio); #endif From patchwork Wed Jan 11 14:29:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13096771 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AC463C63797 for ; Wed, 11 Jan 2023 14:29:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2486C900008; Wed, 11 Jan 2023 09:29:13 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E118C900005; Wed, 11 Jan 2023 09:29:12 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5A8D08E0001; Wed, 11 Jan 2023 09:29:12 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 145CA90000C for ; Wed, 11 Jan 2023 09:29:11 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id C055F160DF2 for ; Wed, 11 Jan 2023 14:29:10 +0000 (UTC) X-FDA: 80342750460.16.FE12B8A Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf04.hostedemail.com (Postfix) with ESMTP id 2EBDC40014 for ; Wed, 11 Jan 2023 14:29:08 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=r22T2BiO; dmarc=none; spf=none (imf04.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673447349; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=tjAbopy+7Q6PAJdQy6rk54Uza9KTpHMnJD/4tJWFpVU=; b=WpecbckKrEDHfNXUMhHc5Kvi8rdF7pHS+p0orgcbHOUBYLgpsHmPkCxEGakkJKobKETq94 QcaLwYwea1+J55+ns5tZYEbRBs7O9tpN3gFquoHZRBvmh6V0CtkpL5hbZyhUGZE+p8DI2o wTYmCrtZzk86i1HQBtAf8N+7MUQEvpY= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=r22T2BiO; dmarc=none; spf=none (imf04.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673447349; a=rsa-sha256; cv=none; b=xk9tItSD4tFy/XPqRW2QS0Tir8EiK+R9wipXbEmZZYYz/80zPdN9E0Qk/v91fy716+Pptr m//7+Z+Q7WpxDIZeT7S5uPP0Cz9oQEW4DFuPkiMdegvuJ6e4Xii6mR/XoMmJYNibgropHW Nqurfyi4KbCAo1f21Cj1AgpnAVQbx2Y= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=tjAbopy+7Q6PAJdQy6rk54Uza9KTpHMnJD/4tJWFpVU=; b=r22T2BiOzS3fqPouvWoMiHtzOT vvpt9rnyw02Jn3OEkh1ILoz7yQMk9gZThuxNE0CXYCCtHMzg9yzsAVurc4cSqjTX6edDcygPZFMOP n4yvixGNK6u+7V5XTCPzspSAOeXruy31v0jGgfmGTudvRrvdXYlhodxcqolM64KxDQai33NPotVH0 3k15FvOue5OJjabDG/6/xcCHdPCixKPUQ6+MAJgD1S11ed4bjIoHI4HxS4R9pSAMZrM8gC7LCUtF9 A+rAI22kCqAuSnOGUo22lU2DyBwN7gCV5NM9pnPp+1LmrlECFcJbMKVMdvWKhH2XY0jtI19Z4sfRi zPQLhT8A==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pFc6S-004CbH-F0; Wed, 11 Jan 2023 14:29:20 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Hugh Dickins Subject: [PATCH 27/28] mm: Convert deferred_split_huge_page() to deferred_split_folio() Date: Wed, 11 Jan 2023 14:29:13 +0000 Message-Id: <20230111142915.1001531-28-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230111142915.1001531-1-willy@infradead.org> References: <20230111142915.1001531-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 2EBDC40014 X-Stat-Signature: uijyyzfgmrof93ssir5i46z8xugtxqh8 X-HE-Tag: 1673447348-400714 X-HE-Meta: U2FsdGVkX186Tz1lmncSjOGK6bZRCDRuyIyJYHXnGkpadk+4VP7tvS0ZfVUr4gHGxGIjhczGseZZ+9HKSVKmaYFuqeF4K1laLXkNKDui6NX+82J50WKo+z12K3RHy+qTUidc7dMqkZzdO4Kp8lXn0prIFZDeW1udpw2MjsOnXktwT4QBgAGtJnEU+y3yghCLFS9e9R20+WRePAQBX+UUwphnmnk5J2jFZrlmFz106LAHgax4kgHEheY6OI4S8/ll4nYv2Yx+WsMaUL3nqkU1emcOx3eBIg6ylkITTmRVJzX7DIvx3pUJqsyCMPNv1B4ukZUAlcjZjbVOzclr/RJsUv7OEOfA49esZzVE/MBDXjTYEKsVxWXIwrYBOHAvtUWUItdMFPkfxnPTkuCXDaTP2AUpdH/TK1adnEISUZCRa7RGrZB20aCcQShF7zNtLRuQKUv3f4oh2DDQew9hD4pJl6RmeBf8Tghin9Ia6q37qBr2h/YKgSAgTkhxa74tTEUo5FDL7PdGIduQKezlZGToOyrYIJuEZc+qY2a+pdsLW21Rghgv+wSDuhOoRf/7WfD/MJ+91ttvDTegCGpwaKGVaD8nDTf+Neb2BxUP06oVg+MyGvU3DAabwg+RiLxF0MVeDRNQz6GwWEE4AIDa3XVLK2s8dBoViSDq7zIEMPpFuW8u4pMI9D2qhHTQejygFpJVluAAqaer0tUHzhrXH04hg824VZ8C9vJiqgWO14UW/49I+LEbCk3tMSyKejTAmvgVMhI6iB8s5OnCTnr6h7U/4DATOcf990SFOizE5EUJmIReXXBj+oFJZ6BH+wO1C/6XWZ1P5SDRuS3R7V8GwlxcW10aiMm0ZKypYZpBuLuCB96PoYm2vDZYUWGVCCiPLD2qsOASqUHbKUTbXDhcudOVfj2wLf0IXuknm9bi1ZyrfvX4JoIoaH5u7w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now that both callers use a folio, pass the folio in and save a call to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- Documentation/mm/transhuge.rst | 6 +++--- include/linux/huge_mm.h | 4 ++-- mm/huge_memory.c | 3 +-- mm/rmap.c | 2 +- 4 files changed, 7 insertions(+), 8 deletions(-) diff --git a/Documentation/mm/transhuge.rst b/Documentation/mm/transhuge.rst index 03bbd0a19041..a9608fe51649 100644 --- a/Documentation/mm/transhuge.rst +++ b/Documentation/mm/transhuge.rst @@ -153,8 +153,8 @@ clear where references should go after split: it will stay on the head page. Note that split_huge_pmd() doesn't have any limitations on refcounting: pmd can be split at any point and never fails. -Partial unmap and deferred_split_huge_page() -============================================ +Partial unmap and deferred_split_folio() +======================================== Unmapping part of THP (with munmap() or other way) is not going to free memory immediately. Instead, we detect that a subpage of THP is not in use @@ -166,6 +166,6 @@ the place where we can detect partial unmap. It also might be counterproductive since in many cases partial unmap happens during exit(2) if a THP crosses a VMA boundary. -The function deferred_split_huge_page() is used to queue a page for splitting. +The function deferred_split_folio() is used to queue a folio for splitting. The splitting itself will happen when we get memory pressure via shrinker interface. diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index b9978978a160..70bd867eba94 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -187,7 +187,7 @@ static inline int split_huge_page(struct page *page) { return split_huge_page_to_list(page, NULL); } -void deferred_split_huge_page(struct page *page); +void deferred_split_folio(struct folio *folio); void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, unsigned long address, bool freeze, struct folio *folio); @@ -340,7 +340,7 @@ static inline int split_huge_page(struct page *page) { return 0; } -static inline void deferred_split_huge_page(struct page *page) {} +static inline void deferred_split_folio(struct folio *folio) {} #define split_huge_pmd(__vma, __pmd, __address) \ do { } while (0) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index c23b0e01734b..868fcccdff72 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2815,9 +2815,8 @@ void free_transhuge_page(struct page *page) free_compound_page(page); } -void deferred_split_huge_page(struct page *page) +void deferred_split_folio(struct folio *folio) { - struct folio *folio = page_folio(page); struct deferred_split *ds_queue = get_deferred_split_queue(folio); #ifdef CONFIG_MEMCG struct mem_cgroup *memcg = folio_memcg(folio); diff --git a/mm/rmap.c b/mm/rmap.c index 462b334d6842..7f76fc40af9a 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1427,7 +1427,7 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, */ if (folio_test_pmd_mappable(folio) && folio_test_anon(folio)) if (!compound || nr < nr_pmdmapped) - deferred_split_huge_page(&folio->page); + deferred_split_folio(folio); } /* From patchwork Wed Jan 11 14:29:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13096778 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A5993C54EBC for ; Wed, 11 Jan 2023 14:29:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E5B5890000E; Wed, 11 Jan 2023 09:29:15 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D958F8E000F; Wed, 11 Jan 2023 09:29:15 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C0F2C8E0010; Wed, 11 Jan 2023 09:29:15 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id A1E228E000F for ; Wed, 11 Jan 2023 09:29:15 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 7A547140878 for ; Wed, 11 Jan 2023 14:29:15 +0000 (UTC) X-FDA: 80342750670.29.211C41F Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf30.hostedemail.com (Postfix) with ESMTP id EE65A8000D for ; Wed, 11 Jan 2023 14:29:13 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=HIzla7wg; spf=none (imf30.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673447354; a=rsa-sha256; cv=none; b=0EC/WlmYs+cgMwYrPIYh2Nso9z6H77D++RTgL83SCon0MOoLNeoXFYhaIMwxsz2I1y9Xlw OWtUe9JflUXYWD3ijJja/xA7aOKsrs1/BEoFUkfwAT/PAPloGoR7mks0CNHTid8ialjVuW m40lZd3dyiEd+uoPQYi/1HBveISnJek= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=HIzla7wg; spf=none (imf30.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673447354; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=bKmmLuHxSB3u7sLoC46CMoIua0qyNhewbdj70P3ETHM=; b=ASE2XDjJfcZhxwrXbRluqtSnRGzc8RQaP9Pb7r/1sEQwEk2H4DC65Un8XVUcFG1iq3VdGG CICz9DBskxpa2YZv4o/lD3tvCzEaE7ri3RSLJMOGuTcIvT5jrod7rYFOf49hMFcloZtL0l yz2RufGvNFHYJudpU6QuH64R2Z2lv2M= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=bKmmLuHxSB3u7sLoC46CMoIua0qyNhewbdj70P3ETHM=; b=HIzla7wg9aXtyVxwFgUkZR5pmR q6ojGe4C2tuIFSqqzQv3bnurfZ/ismaUeJeohP0gufvUPqBGAPVa6FPA4MKR/Hvjc2T6xVBNuPIOT x9mEFnInjU3ZQY8KeMSkI3icKk58en/oxm1zlIm6nmceb5yKvfZTYaFngCzgSpwNCARonvqJ2HVKH DjMkXLjQcvdjt0qoqy/UkQxMI/JdJxoO+4Qhwal462Rau5o+BBQJbQKOv7NGCLtphdR/cYJe/QiRm POvRImoqaOgsi4lq8Z36szE7fu71fItqUwRvjifTSvLHu/e2k/gLsmkNxbenoqPWBYEKv9Um/Lkhp 6X/6vQtA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pFc6S-004CbT-Jo; Wed, 11 Jan 2023 14:29:20 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: Sidhartha Kumar , linux-mm@kvack.org, Hugh Dickins , Matthew Wilcox Subject: [PATCH 28/28] mm: remove the hugetlb field from struct page Date: Wed, 11 Jan 2023 14:29:14 +0000 Message-Id: <20230111142915.1001531-29-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230111142915.1001531-1-willy@infradead.org> References: <20230111142915.1001531-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: EE65A8000D X-Rspamd-Server: rspam01 X-Stat-Signature: 51spmc3fqm85g5tgfyqo9nsj4r1jqihb X-HE-Tag: 1673447353-891649 X-HE-Meta: U2FsdGVkX18syX9hRWMCuHO2VI43n7gNDwnqZ81Bud1tKeqONpx23+O5llSGcMif27r/p2hJ/nvjq7kf0/N9eLbDw6D7s9UmLcAdmj3/occQN0s9a35yHUwisrr8G1rAn/lSseKZB8/bIenTT8AfA+0FA9dFyxLMxlyh7xNf1XAvV+BHxNg2R5wyeV6PVjY4vGFP/VuROhLflJTlf3IU/flBuhEad7TSLK0zyYpRUUFgpSQDjbfA6AVvJKekhJdXFq4kVb/TC1Xai60ldhykHcWb0SgpGz2wmjm07PxH2gPyZzyFJ9joIkQ3fmUvHTMoVdlq4DeK2X93nkhy5EbM/5Lhg+8nip3ne4ALjNVRrmMEKFhcOE4G9IamyC2J/1Id68yWVAzROXrtilAOapsbvu+iMc70CLmUYXO3Yo2NHE+5IP4Gi5k8807xZdm16Rx7wtjp2iC+H24dONR1uEez1qVEjZXdTJOoMhiRTT0htwnpVX5FQ6i903RhEafCmo7zatYKPSgFF/ANxwqa/7AudvGMIA0RJbY/7WgQBjpo6GYg17Id/7s5KtbdgAjJ13JqOT8wxN5zPXKerTnuKO/mMBjpzTg9Uyiv2lcanCHbw3gm42rDPWioOBvxdSK6UjXcuB3asG/deObZGs/ZfRz4Ht5+QEM/2h+SHx0ZlH2qvh3cp56MFjI9DFU/1NzoHP0BQ+Y5KRbZsYOG1/K05qxj5aIuooN7TdhB+uIs5hz5RTr2qE2a023QuZUWo/NQyljMxtHnZFhdhE2lW1cFRhfXxnUpr9lXDgPQR+iYIcwE9wCXjg4328vyg2Tsgcr2kuZXn3JwmMPrksY1V8aRATar0YC3EyiMwfH9jmR0a27ID/hs36WCeKaBDvdN+7vY3+5e X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Sidhartha Kumar commit dad6a5eb5556(mm,hugetlb: use folio fields in second tail page) added a transitional hugetlb field to struct page and struct folio to make room for another int in the first tail of a compound page. Hugetlb folio conversions have changed all page users of this field to use the fields within the folio so struct page no longer needs this hugetlb specific field. Signed-off-by: Sidhartha Kumar Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/mm_types.h | 12 ------------ 1 file changed, 12 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index c464205cf7ea..9932a4cd5b42 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -141,14 +141,6 @@ struct page { struct { /* Tail pages of compound page */ unsigned long compound_head; /* Bit zero is set */ }; - struct { /* Second tail page of hugetlb page */ - unsigned long _hugetlb_pad_1; /* compound_head */ - void *hugetlb_subpool; - void *hugetlb_cgroup; - void *hugetlb_cgroup_rsvd; - void *hugetlb_hwpoison; - /* No more space on 32-bit: use third tail if more */ - }; struct { /* Page table pages */ unsigned long _pt_pad_1; /* compound_head */ pgtable_t pmd_huge_pte; /* protected by page->ptl */ @@ -399,10 +391,6 @@ FOLIO_MATCH(compound_head, _head_1); offsetof(struct page, pg) + 2 * sizeof(struct page)) FOLIO_MATCH(flags, _flags_2); FOLIO_MATCH(compound_head, _head_2); -FOLIO_MATCH(hugetlb_subpool, _hugetlb_subpool); -FOLIO_MATCH(hugetlb_cgroup, _hugetlb_cgroup); -FOLIO_MATCH(hugetlb_cgroup_rsvd, _hugetlb_cgroup_rsvd); -FOLIO_MATCH(hugetlb_hwpoison, _hugetlb_hwpoison); #undef FOLIO_MATCH /*